WebRTC Support

[s-ol]

Using gstreamer because it seems that's mandatory even if you use a more lightweight WebRTC signalling library. For WebRTC, needs the gst-plugins-rs package. Test program in make test/gstreamer shows a PNG at http://localhost:4273/gst-image/.

• basic gstreamer playback into images on DB
• start a gst-webrtc-signalling-server instance
• figure out what happens when a stream fails to start or ends
  • does it just silently retry until it starts working (again)? tie into folk's error system somehow?
  • could use some help with this
• vendor the gstwebrtc-api JS bundle and implement a web route that allows streaming
  

stretch goals:

• gstreamer seeking (for non-live streams)?
• gstreamer publishing (or specifically webrtc publishing)
  • of a region?
• replace a big chunck of camera.folk with gstreamer
  • uridecodebin uri=v4l2:///dev/video0 has strong screen tearing for me, need to investigate

[s-ol]

How to use at the moment:

When the gstreamer pipeline "videotestsrc" frame is /frame/ at /ts/ { ... }

The syntax of the pipeline string is gst-launch-1.0, but gstreamer.folk will automatically add a sink (and videoconvert) at the end.

For webrtc:

• install gst-plugins-rs
• run gst-webrtc-signalling-server
• clone/download this directory https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/tree/main/net/webrtc/gstwebrtc-api
• npm i && npm start
  • npm run build creates a bundle of the library code that can be vendored
• start sharing video in the browser window that opens, take note of peer-id
• use pipeline string uridecodebin uri=gstwebrtc://127.0.0.1:8443?peer-id=$PEER_ID

[osnr]

(slightly concerned about the long-term goal of replacing camera.folk/v4l2 with gstreamer -- it feels like we should try to use the lowest-level reasonable interface in general to avoid latency and give us configuration power. But otherwise this seems good/useful, continuing to look at it now)

[cwervo]

Okay here are some thoughts after thinking about this for a day, seeking :

• @s-ol what do you think about carving this out into its own repo that we vendor? I'm kind of concerned about mixing dependencies that include npm for Gstreamer
• I'm also interested in using just the Gstreamer C core for local video playback
• So, I guess my delineation is: folk.js stays in this repo, everything that depends on npm goes into a separate public repo and is basically Folk WebRTC middleware, does that make sense?

[s-ol]

• @s-ol what do you think about carving this out into its own repo that we vendor? I'm kind of concerned about mixing dependencies that include npm for Gstreamer

Oh, I didn't update the instructions above anymore, npm was necessary to build the gstreamer webrtc demo page for initial testing, but later I compiled the gstwebrtc-api library into a standalone JS bundle and vendored it. Now only the gstreamer distro packages are dependencies and if they aren't detected webrtc.folk errors with a message pointing out what to install.

the flow now is:

• wish for a webrtc feed (or trigger the When when), see test/webrtc.tcl
• open http://localhost:4273/webrtc and click the link for the feed to broadcast to
• start camera or screenshare

[berkan]

Hello @s-ol

I wanted to try this out, and when I do make test/gstreamer.tcl I get:

Warning: Unable to generate getter for `pipeline_t pipeline`: Unrecognized rtype GstElement
Warning: Unable to generate getter for `pipeline_t sink`: Unrecognized rtype GstElement
Warning: Unable to generate getter for `pipeline_t bus`: Unrecognized rtype GstBus
DEBUG(cflags): -I/usr/include/tcl8.6 -ltcl8.6 -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -pthread -I/usr/include/x86_64-linux-gnu -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0
gstreamer-1: Error in match m1:0: couldn't load file "/tmp/cfile_Vzy94y.so": /tmp/cfile_Vzy94y.so: undefined symbol: gst_sample_get_caps
couldn't load file "/tmp/cfile_Vzy94y.so": /tmp/cfile_Vzy94y.so: undefined symbol: gst_sample_get_caps
    while executing
"load [file rootname $cfile][info sharedlibextension] cfile"
    (procedure "::Pipeline::c9::compile" line 34)

(Note that I'm printing the value of $cflags just before the compile step to see if everything looks right)

These problems may be related, but not sure how to proceed:

1. I don't know what to do about Warning: Unable to generate getter for ...

2. It also looks like there is a linking issue: I have the right gstreamer library, but the object file doesn't have that dependency somehow:

nm -D /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so | grep gst_sample_get_caps
00000000000acbd0 T gst_sample_get_caps

ldd /tmp/cfile_Vzy94y.so
	linux-vdso.so.1 (0x00007ffe2268f000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000705ed9600000)
	/lib64/ld-linux-x86-64.so.2 (0x0000705ed9944000)

Any ideas?

[s-ol]

hey @berkan!

1. The rtype warnings are unrelated, probably could be avoided somehow but are safe to ignore.

2. The library was supposed to be included via -l switch in cflags in this line:

    cc cflags {*}[exec pkg-config --cflags --libs gstreamer-1.0]
   

This was the best way I found for making this portable across ArchLinux and Ubuntu in my testing, but folk has also changed since. I immediately have two hypotheses: either cflags is not applied to the linking phase (maybe change to the c interface?) or your pkg-config doesn't produce linking switches (no idea why that would be).

In any case you can probably use some variation of c loadLib / c loadLibLd to explicitly load the library as a workaround.