Remote stream does not always display according to Wifi or mobile network

Hello everyone!

I’m building a video-call app in react native with react-native-webrtc package and I’m facing some problems when I run the app. Here is the situation (all the code used is below). To test my code, I run it on two devices (one android smartphone and one android tablet). Let’s name those “Client 1” and “Client 2”, Client 1 been the one who join a room and wait for Client 2. When Client 2 joins the room, he starts sending ice-candidate and offer to Client 1 who responds with answer. After that they share their stream and see each other’s camera. For connecting the clients, I use a NodeJS server deployed on Heroku on which clients connect through socket io. There are no problems with the connection to the server.

That’s what I want to do. However, it doesn’t succeed all the times. When the two devices are connected to the same wifi, each device can play the role of Client 1 and Client 2. But problems come when they are not on the same wifi or if one device is connected through mobile network. Here are the situation. I tried to be as exhaustive as possible to make you see where the problems come.
When I write “It works”, it means that both clients are connected through WebRTC and can see their local stream and remote stream. When I write “it fails”, the clients are not connected and can not see the remote stream.

  1. If tablet (on wifi) = Client 1 and smartphone (on mobile network 4G) = Client 2 ==> it works

  2. If smartphone (on mobile network 4G) = Client 1 and tablet (on wifi) = Client 2 ==> if fails

  3. If tablet (on wifi 1) = Client 1 (or Client 2) and smartphone (on wifi 1) = Client 2 (or Client 1) ==> it works

  4. If tablet (on wifi 1) = Client 1 and smartphone (on wifi 2) = Client 2 ==> it works

  5. If tablet (on wifi 1) = Client 2 and smartphone (on wifi 2) = Client 1 ==> it fails

I already searched on Google and on this forum. I see that some people had the same problem and resolved it by adding a TURN server but I did it (after signing up on numb) and it’s not working for me:

const peer = new RTCPeerConnection({
      iceServers: [
        {urls: 'stun:stun.l.google.com:19302'},
        {
          url: 'turn:numb.viagenie.ca',
          credential: '************',
          username: '***********@***',
        },
      ],  
    });

This is really weird because even if it fails, I manage to receive the remoteStream and compute the remoteStream.toURL() so I don’t understand why I can’t display the stream.

Here is the react native code:

import React, {useRef, useState, useEffect} from 'react';
import {Text, View, SafeAreaView, Button, StyleSheet, ScrollView} from 'react-native';
import FormButton from '../components/FormButton';
import {RTCPeerConnection, RTCView, mediaDevices, RTCIceCandidate, RTCSessionDescription} from 'react-native-webrtc';
import io from "socket.io-client";

export default function VideoCallScreen({navigation}) {
  const [localStreamCached, setLocalStreamCached] = React.useState();
  const [remoteStream, setRemoteStream] = React.useState();

  const roomID = "*****************";
  let localStream;
  const peerRef = useRef();
  const socketRef = useRef();

  // TMP const pour les afficher dans le DOM et avoir plus facile à comprendre comment tous les messages sont transmis
  const [copySocket, setCopySocket] = useState("");
  const [whichClient, setWhichClient] = useState("");
  const [hasAnyoneInRoomForClient1, setHasAnyoneInRoomForClient1] = useState(false);
  const [hasAnyoneInRoomForClient2, setHasAnyoneInRoomForClient2] = useState(false);
  const [otherUser, setOtherUser] = useState("");
  const [userJoined, setUserJoined] = useState("");

  // Peer connection
  const [receivedOfferFrom, setReceivedOfferFrom] = useState("");
  const [receivedAnswerFrom, setReceivedAnswerFrom] = useState("");
  const [receivedIceCandidate, setReceivedIceCandidate] = useState("");

  
  useEffect(() => {
    // ---------------------
    // Starting local stream
    // ---------------------
    startLocalStream();

    // -----------------
    // SOCKET CONNECTION
    // -----------------
    socketRef.current = io("https://*********.herokuapp.com", {transports: ['websocket']});

    socketRef.current.on("connect", () => {
      setCopySocket(socketRef.current.id);
      console.log("CONNECTION : ", socketRef.current.id);
      console.log(socketRef.current.id, " emit join room to server");
      socketRef.current.emit("join room", roomID);

      socketRef.current.on("other user", userID => {
        setWhichClient("Client 2");
        console.log(" Client 2: (", socketRef.current.id, ") : other user in the room = ", userID);
        setHasAnyoneInRoomForClient2(true);
        setOtherUser(userID);
        // -------------------------------------------------
        // Peer configuration: ice candidate & offer sending
        // -------------------------------------------------
        peerRef.current = peerConfiguration(userID, "Client 2");
      })

      socketRef.current.on("user joined", userID => {
        setWhichClient("Client 1");
        console.log("Client 1 : (", socketRef.current.id, ") : user joined the room = ", userID);
        setHasAnyoneInRoomForClient1(true);
        setUserJoined(userID);
      })

      // Réception de l'ice candidate du socket avec lequel on veut se connecter
      socketRef.current.on("ice-candidate", incoming => {
        handleReceivedIceCandidate(incoming, "Client 1");
      });

      // Réception de l'offre envoyée par l'autre pair
      socketRef.current.on("offer", incoming => {
        console.log("Client 1 : received offer from ", incoming.caller);
        handleReceiveOffer(incoming, "Client 1");
      });

      // Réception de la réponse que l'autre pair nous a envoyée après avoir reçu l'offre envoyée par le socket
      socketRef.current.on("answer", incoming => {
        console.log("Client 2 : received answer from ", incoming.caller);
        handleAnswer(incoming, "Client 2");
      });

    });

    socketRef.current.on('connect_error', (err) => {
      console.log("ERROR connection: ", err);
    })

    socketRef.current.on("disconnect", () => {
      console.log("Disconnected Socket");
    })

  }, []);

      const handleReceivedIceCandidate = (incomingIceCandidate, clientNb) => {
        console.log(clientNb, " : received ice-candidate");
        setReceivedIceCandidate(incomingIceCandidate.candidate);
        const candidate = new RTCIceCandidate(incomingIceCandidate);
        try {
          peerRef.current.addIceCandidate(candidate);
          console.log(clientNb, " : add ice candidate");
        } catch(e) {
          console.error(e);
        }
      }

    const handleReceiveOffer = (incoming, clientNb) => {

    setReceivedOfferFrom(incoming.caller);
    peerRef.current = peerConfiguration(null, clientNb);
    const desc = new RTCSessionDescription(incoming.sdp);
    peerRef.current.setRemoteDescription(desc)
    .then(() => {
      return peerRef.current.createAnswer();
    })
    .catch(err => console.error("Erreur 1: ", err))
    .then(answer => {
      return peerRef.current.setLocalDescription(answer);
    })
    .catch(err => console.error("Erreur 2: ", err))
    .then(() => {
      const payload = {
        target: incoming.caller,
        caller: socketRef.current.id,
        sdp: peerRef.current.localDescription
      }
      socketRef.current.emit("answer", payload);
      console.log(clientNb, " : emit answer to ", incoming.caller);
    })
    .catch(err => console.error("Erreur 3: ", err))
    }

    const handleAnswer = (incomingAnswer, clientNb) => {

     setReceivedAnswerFrom(incomingAnswer.caller);
     const desc = new RTCSessionDescription(incomingAnswer.sdp);
     peerRef.current.setRemoteDescription(desc)
     .catch(e => console.error("Erreur handle answer : ", e));
     console.log(clientNb, " : set remote description");
    }

    const startLocalStream = async () => {

    // isFront will determine if the initial camera should face user or environment
    const isFront = true;
    const devices = await mediaDevices.enumerateDevices();

    const facing = isFront ? 'front' : 'environment';
    const videoSourceId = devices.find(device => device.kind === 'videoinput' && device.facing === facing);
    const facingMode = isFront ? 'user' : 'environment';
    const constraints = {
      audio: true,
      video: {
        mandatory: {
          minWidth: 500, // Provide your own width, height and frame rate here
          minHeight: 300,
          minFrameRate: 30,
        },
        facingMode,
        optional: videoSourceId ? [{sourceId: videoSourceId}] : [],
      },
    };
    const newStream = await mediaDevices.getUserMedia(constraints);
    localStream = newStream;
    setLocalStreamCached(newStream);
  };
  
  const peerConfiguration = (userID, clientNb) => {
    console.log(clientNb, " creates its RTCPeerConnection");
    const peer = new RTCPeerConnection({
      iceServers: [
        {urls: 'stun:stun.l.google.com:19302'},
        {
          url: 'turn:numb.viagenie.ca',
          credential: '************',
          username: '***********@***',
        },
      ],  
    });

    // Ajout du stream local dans le RTCPeerConnection
    peer.addStream(localStream);
    console.log(clientNb, " : add local stream");

    // Ajout du stream remote
    peer.onaddstream = function (e){
      if (e.stream && remoteStream !== e.stream){
        console.log(clientNb, " : add remote stream whose url = ", e.stream.toURL());
        setRemoteStream(e.stream);
      }
    };

    peer.onnegotiationneeded = () => {
      // Création et envoi de l'offre
      console.log("current peerRef = ", peerRef.current);
      peer.createOffer()
        .then(offer => {
          return peer.setLocalDescription(offer)
          .then(() => {
            const payload = {
              target: userID,
              caller: socketRef.current.id,
              sdp: peerRef.current.localDescription
            };
            console.log(clientNb, " : emit offer to ", userID);
            socketRef.current.emit("offer", payload);
          })
          .catch(error => console.error("ERROR set local description", error));
        })
        .catch(error => console.error("ERROR create offer: ", error));
    };

    // Envoi de l'ice candidate
    peer.onicecandidate = function (e) {
      if (userID !== null){
        if (e.candidate) {
          const payload = {
            target: userID,
            candidate: e.candidate,
          }
          console.log(clientNb, " : emit ice-candidate to ", userID);
          socketRef.current.emit("ice-candidate", payload);
        } else {
          console.log(clientNb, " : all ice candidates sent");
        }
      }
    }

    peer.onicecandidateerror = error => {
      console.error("ERROR ice candidate: ", error);
    }

    peer.oniceconnectionstatechange = (val) => {
      console.log("oniceconnectionstatechange : ", val);
    }

    peer.onconnectionstatechange = () => {
      console.log(peer.connectionState);
      if(peer.connectionState === "connected") {
        console.log("PEERS CONNECTED");
      }
    }

    return peer;
  }

  const disconnectSocket = () => {
    socketRef.current.disconnect();
    navigation.navigate("Contact");
  }

return(
    <>
      <Text>Test socket</Text>
      <FormButton buttonTitle="Disconnect" onPress={() => disconnectSocket()}/>
      {copySocket !== "" && (
        <>
          <Text>ID socket : {copySocket}</Text>
          <Text>{whichClient}</Text>
        </>
      )}

      {/*Client 1*/}
      {whichClient === "Client 1" && (
        <>
          {hasAnyoneInRoomForClient1 == false && (
            <Text>No one is in the room for now, Client 1.</Text>
          )}
          {hasAnyoneInRoomForClient1 == true && (
            <Text>User joined: {userJoined}</Text>
          )}
        </>
      )}

      {/*Client 2*/}
      {whichClient === "Client 2" && (
        <>
          {hasAnyoneInRoomForClient2 == false && (
            <Text>No one is in the room for now, Client 2.</Text>
          )}
          {hasAnyoneInRoomForClient2 == true && (
            <Text>Other user : {otherUser}</Text>
          )}
        </>
      )}

      {receivedOfferFrom !== "" && (
        <Text>Offer received from : {receivedOfferFrom}</Text>
      )}
      {receivedIceCandidate !== "" && (
        <Text>Received candidate : {receivedIceCandidate}</Text>
      )}

      <View style={styles.rtcview}>
        <RTCView style={styles.rtc} streamURL={localStreamCached && localStreamCached.toURL()}/>
        {remoteStream && (
          <>
            <Text>{remoteStream.toURL()}</Text>
            <RTCView style={styles.rtc} streamURL = {remoteStream.toURL()}/>
          </>
        )}
      </View>
    </>
   )
 }

Here are the logs for the point 4 above.
Client 1 (tablet on wifi 1) logs:

Client 2 (smartphone on wifi 2) logs:

And here are the logs for the point 5 above.
Client 1 (smartphone on wifi 2) logs:

Client 2 (tablet on wifi 1) logs:

The error Failed to set local offer sdp: Called in wrong state: kHaveRemoteOffer comes everytime I run the app, whether it works or not. It comes from the onnegotiationneeded event but I don’t know why. I guess it’s not the problem I’m looking but if you know how to resolve it, go ahead!

Thansk for you help!