Convert raw RTP packets from mediasoup to frames

I am using mediasoup as a WebRTC server. In mediasoup, you can intercept raw RTP packets on the serverside using the following code:

peer
  .on('newrtpreceiver', (rtpReceiver) => {
    rtpReceiver.on('rtpraw', (packet) => {
      // do something with this packet
    })
  })

These packets are vp8 encoded. I want to pass the packets into FFMPEG and convert them to a stream of frames. I can then send these frames to an OpenCV service for analysis in real-time.

My first attempt at doing this used the following procedure: - Turn the rtpReceiver.on('rtpraw') event into a Readable stream. - Use that readable stream as input into ffmpeg. - Set the output to a writestream.

Here is an example of the code:

import {Readable, Writable} from 'stream'
import * as ffmpeg from 'fluent-ffmpeg'

peer
  .on('newrtpreceiver', (rtpReceiver) => {
    let readStream = new Readable({
      objectMode: false,
      read(size) { return true }
    })

    let writeStream = new Writable({
      objectMode: false,
      write(frame, encoding, done) {
        // send frame somewhere
      }
    })

    let ffmpegStream = ffmpeg(readStream)
      .noAudio()
      .videoCodec('libvpx')
      .size('640x?')
      .format('webm')
      .on('start', (cmdline) => {
        console.log('Command line: ' + cmdline)
      })
      .on('progress', (progress) => {
        console.log('Processing: ' + progress.percent + '% done')
      })
      .on('stderr', (stderrLine) => {
        console.log('Stderr output: ' + stderrLine)
      })
      .on('error', (err, stdout, stderr) => {
        console.log('Cannot process video: ' + err.message)
      })
      .on('end', () => {
        console.log('Finished processing')
      })
      .pipe(writeStream)

    rtpReceiver
      .on('rtpraw', (packet) => {
        readStream.push(packet)
      })
      .on('close', () => { 
        readStream.push(null) 
      })
  })

When I run this, I get the error Invalid data when processing input. Here are the console logs:

Command line: ffmpeg -i pipe:0 -y -an -vcodec libvpx -filter:v scale=w=640:h=trunc(ow/a/2)*2 -f mp4 mymov.mp4
Stderr output: ffmpeg version 3.2.4 Copyright (c) 2000-2017 the FFmpeg developers
Stderr output:   built with Apple LLVM version 8.0.0 (clang-800.0.42.1)
Stderr output:   configuration: --prefix=/usr/local/Cellar/ffmpeg/3.2.4 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-frei0r --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxvid --enable-opencl --disable-lzma --enable-libopenjpeg --disable-decoder=jpeg2000 --extra-cflags=-I/usr/local/Cellar/openjpeg/2.1.2/include/openjpeg-2.1 --enable-nonfree --enable-vda
Stderr output:   libavutil      55. 34.101 / 55. 34.101
Stderr output:   libavcodec     57. 64.101 / 57. 64.101
Stderr output:   libavformat    57. 56.101 / 57. 56.101
Stderr output:   libavdevice    57.  1.100 / 57.  1.100
Stderr output:   libavfilter     6. 65.100 /  6. 65.100
Stderr output:   libavresample   3.  1.  0 /  3.  1.  0
Stderr output:   libswscale      4.  2.100 /  4.  2.100
Stderr output:   libswresample   2.  3.100 /  2.  3.100
Stderr output:   libpostproc    54.  1.100 / 54.  1.100
Stderr output: pipe:0: Invalid data found when processing input
Stderr output: 
Cannot process video: ffmpeg exited with code 1: pipe:0: Invalid data found when processing input

Thank you for all your help!

via user2205763