ffmpeg stream raw video into Unity Texture2D

17 Comments + Add Comment

• 
  Alan Baylis 27/05/2019 at 02:34

  Hi mgear, I did something similar for OpenGL where I chose to render each pixel with a colored quad. You might be interested in the program here at http://www.alsprogrammingresource.com/video.html

  Reply
• 
  bmegli 31/08/2019 at 05:32

  Hi mgear,

  On Windows, have a look at ViveMediaDecoder by HTC
  https://github.com/ViveSoftware/ViveMediaDecoder

  It is Unity Windows software decoder based on FFmpeg. From the code I see that it should support RTSP. If so you should be able to stream directly from FFmpeg or gstreamer.

  On Linux you could take a look at (my) unity-network-hardware-video-decoder

  https://github.com/bmegli/unity-network-hardware-video-decoder

  It will not get you the job done, it uses custom network protocol but you can get the idea how to do it on Linux and other systems (the rendering part is cross platform).

  By combining the rendering part from UNHVD with decoding from Vive you should be able to get cross platform Unity Player.

  By combining the decoding part from UNHVD with Vive, you should be able to get hardware accelerated Unity Player.

  Kind regards

  Reply
• 
  mgear 23/09/2019 at 19:23

  this might work? its for mjpeg:
  Components for video streaming in Unity
  https://github.com/gpvigano/VidStreamComp

  Reply
• 
  mgear 09/10/2019 at 02:26

  Unity Render Streaming webRTC
  https://github.com/Unity-Technologies/UnityRenderStreaming

  Reply
• 
  Ariel 21/11/2019 at 17:00

  Hi, So I’ve tried a lot of things in order to achieve video streaming over UDP…
  bumped into this thread several times:)

  Eventually, salvation came from the attached repo…. it uses AsyncGPUReadbackRequest which is awesome.
  you do need to change the FFmpeg command from inside the code, but it works perfectly and even harness the HW acceleration for decoding.

  anyway enjoy Brother:
  https://github.com/keijiro/FFmpegOut

  and change is this:

  public static FFmpegSession CreateWithOutputPath(
  string outputPath,
  int width, int height, float frameRate,
  FFmpegPreset preset
  )
  {
  return new FFmpegSession(
  “-y -f rawvideo -vcodec rawvideo -pixel_format rgba”
  + ” -colorspace bt709″
  + ” -video_size ” + width + “x” + height
  + ” -framerate ” + frameRate
  + ” -loglevel warning -i – ” + preset.GetOptions()
  //+ ” \”” + outputPath + “\””
  //+ ” -f h264 udp://127.0.0.1:6000″
  + ” -f h264 udp://192.168.1.39:6000″
  );
  }

  Reply
  • 
    Shane 28/09/2021 at 17:34

    Hi Ariel,

    Were you able to use this method to stream video using FFMPEG into Unity over UDP? I have been trying to get something like that working correctly, but can’t seem to get anywhere. I am streaming my desktop using FFMPEG and want it to play in Unity (on a flat plane for example).

    Thanks,
    Shane

    Reply
• 
  Pat 27/10/2020 at 05:07

  @mgear, would you have any advice on how to change this from a desktop stream to a stream from a wifi camera over UDP? I’m trying to get the direct feed of a Tello drone camera into Unity.

  I’m just starting with ffmpeg but this is what I’ve tried (and failed with) so far:

  ffmpeg -i udp://[drone’s IP and port here] -r 1 -f mpegts udp://127.0.0.1:8888

  This (probably unsurprisingly) did not work.

  Reply
  • 
    mgear 27/10/2020 at 11:36

    i’d probably try feeding that camera stream into some standalone streaming server like,
    https://gist.github.com/unitycoder/62e2fca5bd00a3b907cfe0a95d04f62d

    then you can use some video player plugins from asset store to playback those HLS or RTMP streams from that server.

    my test code most certainly would start going out of sync..

    Reply
• 
  mgear 12/04/2021 at 20:22

  Unity network texture sender and receiver for video and frame streaming
  https://github.com/BarakChamo/uTextureSendReceive

  Reply
  • 
    Shane 28/09/2021 at 17:36

    Hi mgear,

    This link you sent uses TCP, have you ever been able to get reliable streaming into Unity over UDP? I am using FFMPEG and want to stream over UDP into Unity. I tried your first method and like you said it works well for several seconds then breaks. Any assistance would be appreciated.

    Thanks,
    Shane

    Reply
    • 
      mgear 30/09/2021 at 15:21

      UDP != reliable, but havent tested further, but i think the issue with my code is that it doesnt have any kind of “sync” method, if even one byte is lost, it goes out of sync. (instead of waiting for something like RTP packaged frame, where you can check frame contents if its corrupt etc.)

      Reply
• 
  mgear 22/06/2022 at 09:55

  Use Mobile-FFmpeg with Unity
  https://sourceexample.com/en/671cf50fddfec13eb5e1/

  Reply
• 
  leo 09/08/2022 at 19:50

  FMETP STREAM 3.0 has Desktop capture & stream over UDP, it’s compatible with PC Mac and Linux.
  https://youtu.be/ZEOzbVy0UYU?t=36

  Reply
• 
  bigzaiko 30/08/2022 at 11:02

  Not the best solution but maybe a hint to reset the buffer.

  if (this.bufferIndex + len > this.bufferSize)
  {
  this.bufferFrameStart = 0;

  var diff = this.bufferSize – this.bufferIndex;
  len -= diff;
  this.bufferIndex = len;
  Array.Copy(this.receivedBytes, 0, this.dumpPixels, this.bufferIndex, len);
  }
  else
  {
  Array.Copy(this.receivedBytes, 0, this.dumpPixels, this.bufferIndex, len);
  this.bufferIndex += len;
  }

  Reply
• 
  bigzaiko 05/09/2022 at 15:12

  TCP receiver:
  https://github.com/unitycoder/ffmpegStreamToUnity/issues/2

  Reply
• 
  mgear 17/01/2024 at 08:23

  https://github.com/fh-joanneum-research-and-design-lab/FFmpegOut/tree/live-streaming

  Reply
• 
  mww 23/07/2024 at 10:56

  Hi mgear,

  The code you provided seems to work but I do have a camera that does RTSP stream and i would like to display that video stream on a quad/plane or maybe on a UI element. Do you have any idea on how to do that? Any help would be appreciated…

  Reply