I am working on a tool to use GL to render frames from a video onto a texture-mapped mesh. I already have a GL app working with a single image (PNG). Now I am trying to use gstreamer to decode the video.
I started with the appsink example.
I have gotten as far as piping the decoded video through glupload into an appsink. Now I need to convert the BufferRef I get from appsink.pull_sample().get_buffer() into a GL texture id (a u32) so I can pass it to GL functions like gl::BindTexture(gl::TEXTURE_2D, tex). I used set_caps() on the appsink to ensure the buffer has feature memory:GLMemory, so it better be a texture and not off-GPU.
How do I extract a GL texture id from a BufferRef using Rust's gstreamer and gstreamer-* crates?
Retrieving the texture from a GstGLMemory in C requires mapping the GstGLMemory itself with a special GST_MAP_GL flag. That specific interface for mapping an OpenGL texture does not currently have an analogue in rust yet. There is some work in a related area within https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/-/merge_requests/581 to help improve the situation with GStreamer OpenGL usage in rust.
If you only need readable access to the texture, there is an extension trait VideoFrameGLExt on VideoFrame that can get you access to the OpenGL texture. There is a usage of VideoFrameGLExt in the glupload example in the gstreamer-rs repository available from https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/-/blob/master/examples/src/bin/glupload.rs. The VideoFrameGLExt trait is currently implemented within https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/-/blob/master/gstreamer-gl/src/gl_video_frame.rs
Something like the following should work for read-only access:
// buffer: gst::Buffer
// info: gst::VideoInfo
if let Ok(frame) = gst_video::VideoFrame::from_buffer_readable_gl(buffer, &info) {
if let Some(texture) = frame.get_texture_id(0) {
// use texture somehow
}
}
If instead you also need to write to the texture, that is currently not exposed and manual bindings would need to be written.
The code I eventually got working was
fn get_gl_memory(bref: &BufferRef, idx: u32) -> Option<*mut GstGLMemory> {
unsafe {
let n = gst_sys::gst_buffer_n_memory(bref.as_ptr() as *mut _);
if idx >= n {
return None;
}
let mem = gst_sys::gst_buffer_peek_memory(bref.as_ptr() as *mut _, idx);
if 0 != gst_gl_sys::gst_is_gl_memory(mem) {
Some(mem as *mut _)
} else {
None
}
}
}
//
let gl_mem = get_gl_memory(buffer, 0).unwrap();
let gl_mem = unsafe { &*gl_mem };
let tex_id = gl_mem.tex_id;
although the solution from ystreet00 works great if you have convenient access to the gst::VideoInfo.
Im trying to rewrite this piece of code to Vala:
gstreamer example
I got stuck at this line:
watch_id = gst_bus_add_watch (bus, message_handler, NULL);
My vala equivalent:
var watch_id = bus.add_watch (Priority.DEFAULT, message_handler);
I haven't got a clue how to format the BusFunc and it's supposed arguments
BusFunc
Complete code so far:
using Gst;
bool Gst.BusFunc message_handler ()
{
return false;
}
void main (string[] args) {
// Initializing GStreamer
Gst.init (ref args);
var caps = Caps.from_string("audio/x-raw,channels=2");
// Creating pipeline and elements
var pipeline = new Pipeline ("my_pipeline");
var bin = new Bin ("my_bin");
var bus = new Bus ();
var src = ElementFactory.make ("autoaudiosrc", "my_src");
var sink = ElementFactory.make ("autoaudiosink", "my_sink");
var convert = ElementFactory.make ("audioconvert", "my_convert");
var level = ElementFactory.make ("level", "my_level");
var fakesink = ElementFactory.make ("fakesink", "my_fakesink");
// Adding elements to pipeline
//pipeline.add_many (src, sink);
bin.add_many (pipeline, src, convert, level, fakesink);
src.link(convert);
convert.link_filtered (level, caps);
level.link(fakesink);
level.set ("post-messages", true);
fakesink.set ("sync", true);
bus = pipeline.get_bus ();
var watch_id = bus.add_watch (Priority.DEFAULT, message_handler);
// Linking source to sink
src.link (sink);
// Set pipeline state to PLAYING
pipeline.set_state (State.PLAYING);
Thanks in advance!
You're almost there. A delegate identifies the function signature: its parameter types and return type. The BusFunc type has the signature: public delegate bool BusFunc (Bus bus, Message message) so your handler will be something like:
bool message_handler (Bus my_bus, Message my_message)
{
print (#"Message type: $(my_message.type.get_name ())\n");
return true;
}
It returns true in this example to keep the handler.
This example is not tested, but should give you the right idea to move forward.
I am trying to query a list of available video capture devices (webcams) on windows using gstreamer 1.0 in c++.
I am using ksvideosrc as source and i am able to capture the video input but i can't query a list of available devices (and their caps).
On gstreamer 0.10 it has been possible through GstPropertyProbe which is removed in gstreamer 1.0. The documentation suggests using GstDeviceMonitor. But i have no luck using that either.
Has anyone succeeded in acquiring a list of device names? Or can you suggests another way of retrieving the available device names and their caps?
You can use GstDeviceMonitor and gst_device_monitor_get_devices () function.
First initialize GstDeviceMonitor by gst_device_monitor_new().
Second start the monitor by gst_device_monitor_start(pMonitor).
Third, get devices list by gst_device_monitor_get_devices(pMonitor).
Code would be like this:
GstDeviceMonitor* monitor= gst_device_monitor_new();
if(!gst_device_monitor_start(monitor)){
printf("WARNING: Monitor couldn't started!!\n");
}
GList* devices = gst_device_monitor_get_devices(monitor);
My references:
https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gstreamer-GstDeviceMonitor.html#gst-device-monitor-get-devices
Although I haven't figured out how to enumerate the device names, I've come up with a workaround to at least get the available ksvideosrc device indexes. Below is the code in Python, but you should be able to port it to C++ fairly easily, thanks to the GObject introspection bindings.
from gi.repository import Gst
def get_ksvideosrc_device_indexes():
device_index = 0
video_src = Gst.ElementFactory.make('ksvideosrc')
state_change_code = None
while True:
video_src.set_state(Gst.State.NULL)
video_src.set_property('device-index', device_index)
state_change_code = video_src.set_state(Gst.State.READY)
if state_change_code != Gst.StateChangeReturn.SUCCESS:
video_src.set_state(Gst.State.NULL)
break
device_index += 1
return range(device_index)
if __name__ == '__main__':
Gst.init()
print get_ksvideosrc_device_indexes()
Note that the video source device-name property is None as of GStreamer version 1.4.5.0 on Windows for the ksvideosrc.
It's very late, but for the future...
The Gst.DeviceMonitor can be used to enumerate devices, and register an addition or removal of a device.
Here's how to get device names in C# with GStreamer 1.14
static class Devices
{
public static void Run(string[] args)
{
Application.Init(ref args);
GtkSharp.GstreamerSharp.ObjectManager.Initialize();
var devmon = new DeviceMonitor();
// to show only cameras
// var caps = new Caps("video/x-raw");
// var filtId = devmon.AddFilter("Video/Source", caps);
var bus = devmon.Bus;
bus.AddWatch(OnBusMessage);
if (!devmon.Start())
{
"Device monitor cannot start".PrintErr();
return;
}
Console.WriteLine("Video devices count = " + devmon.Devices.Length);
foreach (var dev in devmon.Devices)
DumpDevice(dev);
var loop = new GLib.MainLoop();
loop.Run();
}
static void DumpDevice(Device d)
{
Console.WriteLine($"{d.DeviceClass} : {d.DisplayName} : {d.Name} ");
}
static bool OnBusMessage(Bus bus, Message message)
{
switch (message.Type)
{
case MessageType.DeviceAdded:
{
var dev = message.ParseDeviceAdded();
Console.WriteLine("Device added: ");
DumpDevice(dev);
break;
}
case MessageType.DeviceRemoved:
{
var dev = message.ParseDeviceRemoved();
Console.WriteLine("Device removed: ");
DumpDevice(dev);
break;
}
}
return true;
}
}
I need a gstreamer audio sink that outputs integers that
represent volume level of an audio stream. The sampling rate
need not be the same as the incoming audio stream, it can be much
lower, ex.: one value per second would be sufficient.
Does such a sink exist ?
It seems that this one could be modified to do this :
http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-volume.html
But if something already exists I'd prefer to avoid writing one !
there indeed is such an element, it's not a sink though but I don't think you need it to be for that task anyway :)
It is called level (http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good-plugins/html/gst-plugins-good-plugins-level.html), and as you can see there is an "interval" property that you can tweak.
We use this element in our video editor to draw waveforms, here take this simplified script :
from gi.repository import Gst
from gi.repository import GLib
import sys
mainloop = GLib.MainLoop()
def _messageCb(bus, message):
if str(type(message.src)) == "<class '__main__.__main__.GstLevel'>":
s = message.get_structure()
p = None
if s:
p = s.get_value("rms")
if p:
st = s.get_value("stream-time")
print "rms = " + str(p) + "; stream-time = " + str(st)
if message.type == Gst.MessageType.EOS:
mainloop.quit()
elif message.type == Gst.MessageType.ERROR:
bus.disconnect_by_func(_messageCb)
mainloop.quit()
if __name__=="__main__":
global mainloop
Gst.init([])
pipeline = Gst.parse_launch("uridecodebin name=decode uri=" + sys.argv[1] + " ! audioconvert ! level name=wavelevel interval=10000000 post-messages=true ! fakesink qos=false name=faked")
faked = pipeline.get_by_name("faked")
bus = pipeline.get_bus()
bus.add_signal_watch()
bus.connect("message", _messageCb)
pipeline.set_state(Gst.State.PLAYING)
mainloop.run()
pipeline.set_state(Gst.State.NULL)
May I inquire about your use case ?
What is the syntax for caps, specifying media capabilities, in gstreamer? Caps are strings that specify the type of media allowed and look like "audio/x-raw-int,..." but I haven't been able to find good documentation on exactly what is allowed in a caps string.
The syntax is:
<type>[,<property>=<value>]...
Note that the type is not a MIME type, however much it may look like one.
You can find out which caps properties elements support by using gst-inspect. It will proviide "pad templates" for the element's pads, which will specify the ranges of caps supported.
The GStreamer plugin writer's guide also contains a list of defined types which describes properties for common audio, video and image formats.
In Java, for gstreamer-java
final Element videofilter = ElementFactory.make("capsfilter", "flt");
videofilter.setCaps(Caps.fromString("video/x-raw-yuv, width=720, height=576"
+ ", bpp=32, depth=32, framerate=25/1"));
In C, say you want videoscale caps filter
GstElement *videoscale_capsfilter;
GstCaps* videoscalecaps;
...
...
videoscale = gst_element_factory_make ("videoscale", "videoscale");
g_assert (videoscale);
videoscale_capsfilter = gst_element_factory_make ("capsfilter", "videoscale_capsfilter");
g_assert (videoscale_capsfilter);
...
...
then set properties
g_object_set( G_OBJECT ( videoscale_capsfilter ), "caps", videoscalecaps, NULL );
then you could add these to bin and link them the way you have constructed media pipeline using gst-launch
/* Add Elements to the Bin */
gst_bin_add_many (GST_BIN (pipeline),source ,demux ,decoder ,videoscale ,videoscale_capsfilter ,ffmpegcolorspace ,ffmpegcolorspace_capsfilter,autovideosink,NULL);
/* Link confirmation */
if (!gst_element_link_many (demux, decoder,videoscale, videoscale_capsfilter ,ffmpegcolorspace, ffmpegcolorspace_capsfilter, autovideosink, NULL)){
g_warning ("Main pipeline link Fail...");
}
/* Dynamic Pad Creation */
if(! g_signal_connect (source, "pad-added", G_CALLBACK (on_pad_added),demux))
{
g_warning ("Linking Fail...");
}
Here is the format as far as I understand it:
caps = <caps_name>, <field_name>=<field_value>[; <caps>]
<caps_name> = image/jpeg etc
<field_name> = width etc
<field_value> = <fixed_field_value>|<ranged_field_value>|<multi_field_value>
<fixed_field_value> = 800 etc
<ranged_field_value> = [<lower_value>, <upper_value>]
<multi_field_value> = {<fixed_field_value>, <fixed_field_value>, <fixed_field_value>, ...}
I see you are after audio.
I'll just give you the long version, you can drop or change the parts you don't need. It changes between GStreamer 0.10 and GStreamer 1.0 though. I'll give both:
for GStreamer 0.10:
audio/x-raw-int,rate=44100,channels=2,width=16,depth=16,endianness=1234,signed=true
for GStreamer 1.0:
audio/x-raw,format=S16LE,channels=2,layout=interleaved
As you can see, with 1.0, you will need to combine the audio format. S16LE means signed + int + 16 width + little endian (=1234).
This is how i use it in python...HTH
caps = gst.Caps("video/x-raw-yuv,format=(fourcc)AYUV,width=704,height=480")
capsFilter = gst.element_factory_make("capsfilter")
capsFilter.props.caps = caps
I'm unsure due to your question is about syntax, but "list of defined types" may be helpful.
a partial answer, which i'm sure you've worked out already:
"MIMETYPE,PROPERTY1=VALUE1,PROPERTY2=VALUE2,..."
formally, caps are not represented by strings but by a GstCaps object containing an array of GstStructures. see the documentation here.
perhaps if we work out a definitive answer here we could submit a documentation patch for the function gst_caps_from_string()
from x264enc source code: https://github.com/GStreamer/gst-plugins-ugly/blob/master/ext/x264/gstx264enc.c#L693-L704
static GstStaticPadTemplate src_factory = GST_STATIC_PAD_TEMPLATE ("src",
GST_PAD_SRC,
GST_PAD_ALWAYS,
GST_STATIC_CAPS ("video/x-h264, "
"framerate = (fraction) [0/1, MAX], "
"width = (int) [ 1, MAX ], " "height = (int) [ 1, MAX ], "
"stream-format = (string) { avc, byte-stream }, "
"alignment = (string) au, "
"profile = (string) { high-4:4:4, high-4:2:2, high-10, high, main,"
" baseline, constrained-baseline, high-4:4:4-intra, high-4:2:2-intra,"
" high-10-intra }")
);
for this CAPS, when inpstected by gst-inspect, it's like:
Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
video/x-h264
framerate: [ 0/1, 2147483647/1 ]
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
stream-format: { (string)avc, (string)byte-stream }
alignment: au
profile: { (string)high-4:4:4, (string)high-4:2:2, (string)high-10, (string)high, (string)main, (string)baseline, (string)constrained-baseline, (string)high-4:4:4-intra, (string)high-4:2:2-intra, (string)high-10-intra }