The web does not receive the stream of the phone

Help
1

Hello,

I’ve been having a problem for some time now. I’m developing an application with React Native that communicates via WebRTC to another application in React.js. I can get my video feedback and the stream (audio/video) from the React.js application but I can’t receive the stream from the React Native application to the React.js application, yet there is indeed an AddIceCandidate which is played as well as a RemoteDescription and LocalDescription on both sides but I have the impression that the event onaddstream is not fired on the React.js side whereas it’s on the React Native side, which makes the stream audio/video is well added (I think).

Has either of you ever encountered this problem?

2 Likes
2

Hello again,

I’ve been able to investigate a little more and it looks like the SDP on the phone isn’t okay.
The video tracks would be missing and the recvonly value would be incorrect (I think).

Below is the SDP of the phone application

v=0
o=- 500697800290725000 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1
a=msid-semantic: WMS
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Al2U
a=ice-pwd:SOFYNvVFXkthJl0PEMz0J8KS
a=ice-options:trickle renomination
a=fingerprint:sha-256 67:40:94:45:41:C8:BD:7F:89:CD:10:AD:C3:51:10:33:1C:8E:E2:7F:DF:B1:79:54:8D:1C:3C:F5:1C:DD:C5:F8
a=setup:active
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=recvonly
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 114 115 116
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Al2U
a=ice-pwd:SOFYNvVFXkthJl0PEMz0J8KS
a=ice-options:trickle renomination
a=fingerprint:sha-256 67:40:94:45:41:C8:BD:7F:89:CD:10:AD:C3:51:10:33:1C:8E:E2:7F:DF:B1:79:54:8D:1C:3C:F5:1C:DD:C5:F8
a=setup:active
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:114 red/90000
a=rtpmap:115 rtx/90000
a=fmtp:115 apt=114
a=rtpmap:116 ulpfec/90000

Below is the SDP of the web application

v=0
o=- 4837870407153265079 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1
a=msid-semantic: WMS nf9qrVa8UUSEz58PoYZvxZJxneCBl4xBnauc
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:2rcK
a=ice-pwd:u15OwS/yYm/zFQL4TkszFacx
a=ice-options:trickle
a=fingerprint:sha-256 C1:2E:26:85:74:3E:DF:4A:AC:B6:F3:99:DA:62:AE:9A:02:64:CC:71:BC:5C:A1:28:45:AC:75:81:53:21:34:11
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:nf9qrVa8UUSEz58PoYZvxZJxneCBl4xBnauc 2d37e4d2-6ae6-4d8d-aa15-93e11eaacbae
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:621826625 cname:5zfSbaAEAY8ElXKX
a=ssrc:621826625 msid:nf9qrVa8UUSEz58PoYZvxZJxneCBl4xBnauc 2d37e4d2-6ae6-4d8d-aa15-93e11eaacbae
a=ssrc:621826625 mslabel:nf9qrVa8UUSEz58PoYZvxZJxneCBl4xBnauc
a=ssrc:621826625 label:2d37e4d2-6ae6-4d8d-aa15-93e11eaacbae
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 122 127 121 125 107 108 109 124 120 123 119 114 115 116
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:2rcK
a=ice-pwd:u15OwS/yYm/zFQL4TkszFacx
a=ice-options:trickle
a=fingerprint:sha-256 C1:2E:26:85:74:3E:DF:4A:AC:B6:F3:99:DA:62:AE:9A:02:64:CC:71:BC:5C:A1:28:45:AC:75:81:53:21:34:11
a=setup:actpass
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:nf9qrVa8UUSEz58PoYZvxZJxneCBl4xBnauc 22f2a577-ff62-43ee-8129-79edeabc185e
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 VP9/90000
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=fmtp:100 profile-id=2
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:102 H264/90000
a=rtcp-fb:102 goog-remb
a=rtcp-fb:102 transport-cc
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:122 rtx/90000
a=fmtp:122 apt=102
a=rtpmap:127 H264/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
a=rtpmap:121 rtx/90000
a=fmtp:121 apt=127
a=rtpmap:125 H264/90000
a=rtcp-fb:125 goog-remb
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=125
a=rtpmap:108 H264/90000
a=rtcp-fb:108 goog-remb
a=rtcp-fb:108 transport-cc
a=rtcp-fb:108 ccm fir
a=rtcp-fb:108 nack
a=rtcp-fb:108 nack pli
a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f
a=rtpmap:109 rtx/90000
a=fmtp:109 apt=108
a=rtpmap:124 H264/90000
a=rtcp-fb:124 goog-remb
a=rtcp-fb:124 transport-cc
a=rtcp-fb:124 ccm fir
a=rtcp-fb:124 nack
a=rtcp-fb:124 nack pli
a=fmtp:124 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d0032
a=rtpmap:120 rtx/90000
a=fmtp:120 apt=124
a=rtpmap:123 H264/90000
a=rtcp-fb:123 goog-remb
a=rtcp-fb:123 transport-cc
a=rtcp-fb:123 ccm fir
a=rtcp-fb:123 nack
a=rtcp-fb:123 nack pli
a=fmtp:123 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640032
a=rtpmap:119 rtx/90000
a=fmtp:119 apt=123
a=rtpmap:114 red/90000
a=rtpmap:115 rtx/90000
a=fmtp:115 apt=114
a=rtpmap:116 ulpfec/90000
a=ssrc-group:FID 1894704615 4277299889
a=ssrc:1894704615 cname:5zfSbaAEAY8ElXKX
a=ssrc:1894704615 msid:nf9qrVa8UUSEz58PoYZvxZJxneCBl4xBnauc 22f2a577-ff62-43ee-8129-79edeabc185e
a=ssrc:1894704615 mslabel:nf9qrVa8UUSEz58PoYZvxZJxneCBl4xBnauc
a=ssrc:1894704615 label:22f2a577-ff62-43ee-8129-79edeabc185e
a=ssrc:4277299889 cname:5zfSbaAEAY8ElXKX
a=ssrc:4277299889 msid:nf9qrVa8UUSEz58PoYZvxZJxneCBl4xBnauc 22f2a577-ff62-43ee-8129-79edeabc185e
a=ssrc:4277299889 mslabel:nf9qrVa8UUSEz58PoYZvxZJxneCBl4xBnauc
a=ssrc:4277299889 label:22f2a577-ff62-43ee-8129-79edeabc185e

And this is what the React Native application looks like

import React, { useEffect } from 'react';
import { Dimensions, Text, View, SafeAreaView } from 'react-native';
import { mediaDevices, RTCPeerConnection, RTCView } from 'react-native-webrtc';
import io from 'socket.io-client';

import { getVisionURL } from '../../../config';

const socket = io(getVisionURL(), {
  forceNew: true,
});

let localPeerConnection;
let isCaller = false;
let routingConfig = null;

import styles from './styles';

function App() {
  const [localStream, setLocalStream] = React.useState();
  const [remoteStream, setRemoteStream] = React.useState();
  const [iceConnectionState, setIceConnectionState] = React.useState('new');

  useEffect(() => {
    mediaDevices.enumerateDevices().then(sourceInfos => {
      let isFront = true;

      let videoSourceId;

      for (let i = 0; i < sourceInfos.length; i -= -1) {
        const sourceInfo = sourceInfos[i];

        if (
          sourceInfo.kind === 'videoinput' &&
          sourceInfo.facing === (isFront ? 'front' : 'environment')
        ) {
          videoSourceId = sourceInfo.deviceId;
        }
      }

      captureVideo(videoSourceId);

      socket.on('connect', () => {
        console.log('connect', socket);
      });

      socket.on('disconnect', () => {
        console.log('disconnect', socket);
      });
    });

    return () => stopWebRtc();
  }, []);

  function captureVideo(videoSourceId) {
    let isFront = true;

    mediaDevices
      .getUserMedia({
        audio: true,
        video: {
          mandatory: {
            minWidth: Math.round(Dimensions.get('window').width),
            minHeight: Math.round(Dimensions.get('window').height),
            minFrameRate: 30,
          },
          facingMode: isFront ? 'user' : 'environment',
          optional: videoSourceId ? [{ sourceId: videoSourceId }] : [],
        },
      })
      .then(mediaStream => {
        if (mediaStream) {
          setLocalStream(mediaStream);
          startSignaling(mediaStream);
        }
      })
      .catch();
  }

  function handleConnection(e) {
    socket.emit('peer:message', { type: 'ice', data: e.candidate });
  }

  function handleConnectionChange() {
    setIceConnectionState(localPeerConnection.iceConnectionState);
  }

  function endMessage(consultationId) {
    socket.emit('consultation:leave', consultationId);
  }

  function handleAddTrack(event) {
    setRemoteStream(event.stream);
  }

  function processOffer(description) {
    localPeerConnection
      .setLocalDescription(description)
      .then(() => {
        const sdpData = {
          type: 'offer',
          data: description,
        };

        socket.emit('peer:message', sdpData);
      })
      .catch(error => {
        throw error;
      });
  }

  function processAnswer(description) {
    localPeerConnection
      .setLocalDescription(description)
      .then(() => {
        const sdpData = {
          type: 'anwser',
          data: description,
        };

        socket.emit('peer:message', sdpData);
      })
      .catch(error => {
        throw error;
      });
  }

  function startSignaling(mediaStream) {
    const auth = {
      token: 'token',
      role: 'role',
    };

    socket.emit('user:authenticate', auth);
    socket.once('user:authenticate:status', status => {
      if (status === 'authenticated') {
        socket.emit(
          'consultation:join',
          'consultationId',
        );
      }
    });

    socket.on('consultation:status', status => {
      switch (status) {
        case 'ready':
          startWebRtc();
          break;
        case 'terminated':
          stopWebRtc();
          break;
        case 'waiting_for_peer':
          isCaller = true;

          if (localPeerConnection !== null) {
            localPeerConnection.close();

            restartWebRtc(mediaStream);
          }
          break;
        default:
          break;
      }
    });

    socket.on('peer:message', value => {
      switch (value.type) {
        case 'offer':
          localPeerConnection
            .setRemoteDescription(value.data)
            .then(() => localPeerConnection.createAnswer().then(processAnswer));
          break;
        case 'answer':
          localPeerConnection.setRemoteDescription(value.data);
          break;
        case 'ice':
          if (value.data) {
            localPeerConnection.addIceCandidate(value.data);
          }
          break;
        default:
          break;
      }
    });

    socket.on('routing:config', config => {
      routingConfig = config;
      localPeerConnection = new RTCPeerConnection(config);

      mediaStream
        .getTracks()
        .forEach(track => localPeerConnection.addStream(track, mediaStream));

      localPeerConnection.addEventListener('icecandidate', handleConnection);

      localPeerConnection.addEventListener(
        'iceconnectionstatechange',
        handleConnectionChange,
      );

      localPeerConnection.addEventListener('addstream', handleAddTrack);
    });
  }

  function startWebRtc() {
    if (isCaller) {
      localPeerConnection
        .createOffer()
        .then(processOffer)
        .catch(error => {
          throw error;
        });
    }
  }

  function stopWebRtc() {
    localPeerConnection.close();
    localStream.getTracks().forEach(track => track.stop());
    socket.close();
  }

  function restartWebRtc(mediaStream) {
    localPeerConnection = new RTCPeerConnection(routingConfig);

    localPeerConnection.addEventListener('icecandidate', handleConnection);

    localPeerConnection.addEventListener(
      'iceconnectionstatechange',
      handleConnectionChange,
    );

    localPeerConnection.addEventListener('addstream', handleAddTrack);

    mediaStream
      .getTracks()
      .forEach(track => localPeerConnection.addStream(track, mediaStream));
  }

  return (
    <SafeAreaView style={styles.safeareaview}>
      {localStream && (
        <View style={styles.localStream}>
          <RTCView
            style={styles.RTCView}
            streamURL={localStream.toURL()}
            objectFit="cover"
            zOrder={1}
          />
        </View>
      )}

      {remoteStream && (
        <View style={styles.remoteStream}>
          <RTCView
            style={styles.RTCView}
            streamURL={remoteStream.toURL()}
            objectFit="cover"
          />
        </View>
      )}
    </SafeAreaView>
  );
}

export default App;

I’ve been stuck for a few days now and I can’t find out what I might have missed in the configuration of the React Native application… :pray:

2 Likes
3

Hi, I found a solution to this problem, the bottom line is that the addTrack API is not implemented in react-native-webrtc, and you need to implement it yourself, here is the issue where the solution is written in detail https://github.com/peers/peerjs/issues/612

5

I had same issue before. The problem was - app sdp sent too earlier.
your web sdp has msid with string but your app doesn’t. (example from above code)

your app’s sdp has this (incorrect)
a=msid-semantic: WMS

your web’s sdp has this (correct)
a=msid-semantic: WMS nf9qrVa8UUSEz58PoYZvxZJxneCBl4xBnauc . <— this

Should all offer and answer both have some random strings for msid-semantic.

Try to send your app’s offer/answer sdp AFTER added all tracks on the peerConnection
(After when event.candidate is null in onicecandidate event)

Not sure this is the best solution but I hope it helps you

1 Like