I have been tasked with fixing a bug in a medical application that, among other things, can capture snapshots from intra-oral video cameras. It uses a DirectShow SampleGrabber for this task. I must make the disclaimer that I have not worked with DirectShow so I'm trying to get up to speed as I go. I understand basically how the various components work together.
Anyway the bug itself is seemingly trivial but I can't figure out a workaround. Due to the modular nature of this system, the preview window is part of a separate graph than the one created by SampleGrabber (it's a long story but this is due to legacy code to support previous devices). When the camera is active we can take snapshots and everything is happy. When the camera is turned off, the SampleGrabber takes a dark frame but DirectShow is crashing when releasing the IAMStreamConfig interface created in the preview module (access violation). It seems like for some reason the SampleGrabber graph is somehow corrupting the graph built in the preview module. Due to the nature of this application, I cannot show any source here, but essentially here's what I want to accomplish:
I need to be able to detect if the camera is actually on or not. The problem I'm having is that when the camera is plugged in (USB), it seems to look to the system like it is on and returning a video stream, it's just that the stream contains no real data. When I check the state of the capture filter with the GetState method, it claims it is running; also when I check the video format properties it returns the correct properties. It seems to me like the button on the camera simply turns on/off the camera sensor itself, but the device is still returning a blank stream when the camera is off. Something must be different though, because it doesn't crash with the sensor is actually on and returning live video.
Does anybody have an idea of how I could determine if the stream is blank or has live video? IE, are there any exposed interfaces or methods I could call to determine this? I have looked through all of the various interfaces in MSDN's DirectShow documentation but haven't been able to find a way to do this.
If you don't want the callback function of your sample grabber be called, then you may consider adding a special transform filter between the sample grabber and the source filter (or right after the source filter), and what this transform filter does it to check whether the input sample is corrupted and block those corrupted sample. This basically requires you to implement your own Transform() function:
HRESULT CRleFilter::Transform(IMediaSample *pSource, IMediaSample *pDest)
In the filter you connected after the source filter (or the earliest filter you have access to), check the IMediaSample it receives by the receive() function:
HRESULT Receive(IMediaSample *pSample);
In case you're using ISampleGrabber, then you should set its call back function by using ISampleGrabber::SetCallback
HRESULT SetCallback(
ISampleGrabberCB *pCallback,
long WhichMethodToCallback
);
This requires you to implement a class extends ISampleGrabberCB. After that, you can check your received sample in function SampleCB
HRESULT SampleCB(
double SampleTime,
IMediaSample *pSample
);
There is no universal way to tell whether camera is connected or whether the stream is blank or not. You typically have one of the following scenarios:
you stop receiving any samples when camera is off
you receive samples with all pixels zeroed out or, fully blue picture or a sort of this
Some cameras have signal loss notification, but it's model specific as well as notification method.
So in first case you just stop having callback called. And to cover the second one you need to check the frame whether it's filled with solid color entirely. When you capture raw video (uncompressed) this is a fairly simple thing to do.
Related
Background
In order to record the composite-video signal from a variety of analog cameras, I use a basic USB video capture device produced by AverMedia (C039).
I have two analog cameras, one produces a PAL signal, the other produces an NTSC signal:
PAL B, 625 lines, 25 fps
NTSC M, 525 lines, 29.97 fps (i.e. 30/1.001)
Unfortunately, the driver for the AverMedia C039 capture card does not automatically set the correct video standard based on which camera is connected.
Goal
I would like the capture driver to be configured automatically for the correct video standard, either PAL or NTSC, based on the camera that is connected.
Approach
The basic idea is to set one video standard, e.g. PAL, check for signal, and switch to the other standard if no signal is detected.
By cobbling together some examples from the DirectShow documentation, I am able to set the correct video standard manually, from the command line.
So, all I need to do is figure out how to detect whether a signal is present, after switching to PAL or NTSC.
I know it must be possible to auto-detect the type of signal, as described e.g. in the book "Video Demystified".
Moreover, the (commercial) AMCap viewer software actually proves it can be done.
However, despite my best efforts, I have not been able to make this work.
Could someone explain how to detect whether a PAL or NTSC signal is present, using DirectShow in C++?
The world of Windows/COM/DirectShow programming is still new to me, so any help is welcome.
What I tried
Using the IAMAnalogVideoDecoder interface, I can read the current standard (get_TVFormat()), write the standard (put_TVFormat()), read the number of lines, and so on.
The steps I took can be summarized as follows:
// summary of steps used to set the video standard, given a device moniker
// NOTE: declarations, error handling, cleanup, and details of device enumeration are omitted, for brevity
InitCaptureGraphBuilder(&pGraph, &pBuild);
pMoniker->BindToObject(0, 0, IID_IBaseFilter, (void**)&pCap);
pGraph->AddFilter(pCap, L"Capture Filter");
pBuild->FindInterface(&PIN_CATEGORY_ANALOGVIDEOIN, &MEDIATYPE_AnalogVideo, pCap, IID_IAMAnalogVideoDecoder, (void**)&pDecoder);
pDecoder->put_TVFormat(AnalogVideo_PAL_B); // or AnalogVideo_NTSC_M
The steps above work without actually running the graph.
The IAMAnalogVideoDecoder interface also defines a get_HorizontalLocked() method, which returns successful, but the output value does not appear to change, regardless of whether a camera is connected.
I can imagine that it may be necessary to run the graph in order to get updated information regarding e.g. horizontal sync, but that did not seem to make a difference, although I'm not sure my approach was correct.
Some observations
The dialog depicted below is a screenshot from the AMCap viewer software (Options->Video Device->Properties).
This is not the same thing as the AmCap sample for DirectShow, supplied with the Windows SDK (although it maybe based upon that).
The "Signal Detected" value in this dialog changes when I (dis)connect a camera matching the specified standard.
(Although the "Lines detected" value remains the same regardless of whether a camera is connected.)
The "Signal Detected" value is actually what I am looking for.
However, I could not find any mention of it in the DirectShow docs, nor in the property set for analog video decoder devices.
Could this be related to the horizontal sync, somehow?
The dialog looks identical to the dialog that appears when I open the video device using ffmpeg:
ffmpeg -f dshow -show_video_device_dialog true -i video="..." ...
However, in this case the "Signal detected" value does not change when a camera is (dis)connected.
I suppose both programs produce this dialog using the filter property pages.
The AverMedia SDK does define an AVerGetSignalPresence() function.
Not sure if that would do the job, but I would rather not introduce the dependency if it can be done using "pure" DirectShow.
UPDATE
After playing around with the capture device in GraphEdit, I noticed that the "Signal Detected" value is only updated if a (Video) Renderer is connected (and the graph is either running or paused):
The mentioned property page is likely to pull the data using IAMAnalogVideoDecoder and get_HorizontalLocked method in particular. Note that you might be limited in receiving valid status by requirement to have the filter graph in paused or running state, which in turn might require that you connect a renderer to complete the data path (Video Renderer or Null Renderer, or another renderer of your choice).
See also this question on Null Renderer deprecation and source code for the worst case scenario replacement.
Thanks to Roman R.'s answer and comments, I was able to verify that IAMAnalogVideoDecoder->get_HorizontalLocked() can indeed be used to detect the presence of a PAL or NTSC video signal.
In order to obtain a meaningful value from get_HorizontalLocked, it appears to be necessary to:
connect a filter to the capture output pin of the capture filter
run (or pause) the graph
Note that the graph will run successfully even if nothing is connected to the capture output pin, but in that case the value from get_HorizontalLocked remains zero, regardless of signal presence.
Instead of a Video Renderer or Null Renderer, I connected a Smart Tee filter to the output pin of the capture filter. This appears to be sufficient to make get_HorizontalLocked work properly, and it's simpler than including a video renderer or working around the Null Renderer deprecation issue.
Just to illustrate the basic steps, here's what I did, assuming that a camera is already connected:
// assuming we have a pMoniker from device enumeration
IBaseFilter *pCap;
IGraphBuilder *pGraph;
ICaptureGraphBuilder2 *pBuild;
IMediaControl *pControl;
IAMAnalogVideoDecoder *pDecoder;
IBaseFilter *pRender;
long lLocked;
// build graph
InitCaptureGraphBuilder(&pGraph, &pBuild);
pMoniker->BindToObject(0, 0, IID_IBaseFilter, (void**)&pCap);
pGraph->AddFilter(pCap, L"Capture Filter");
AddFilterByCLSID(pGraph, CLSID_SmartTee, &pRender, L"Smart tee");
pBuild->RenderStream(NULL, NULL, pCap, NULL, pRender); // connect the filters
// get interfaces
pBuild->FindInterface(&PIN_CATEGORY_ANALOGVIDEOIN, &MEDIATYPE_AnalogVideo,
pCap, IID_IAMAnalogVideoDecoder, (void**)&pDecoder);
pGraph->QueryInterface(IID_IMediaControl, (void **)&pControl);
// pause graph (or run)
pControl->Pause();
// set standard, check horizontal sync lock
pDecoder->put_TVFormat(AnalogVideo_NTSC_M);
pDecoder->get_HorizontalLocked(&lLocked);
if (!lLocked)
{
pDecoder->put_TVFormat(AnalogVideo_PAL_B);
pDecoder->get_HorizontalLocked(&lLocked);
...
}
Please note: This is not a full working example, just an illustration of the sequence in which the calls are made.
To keep it brief, I omitted device enumeration, error handling, clean-up, etc. I included simplified declarations, just to show which interfaces are used.
Depending on the application, it may be necessary to repeat the get_HorizontalLocked() call for improved robustness.
The DirectShow documentation provides working examples for InitCaptureGraphBuilder, AddFilterByCLSID, and device enumeration.
I want to encode Desktop Duplication API frames to send over the network after encoding them with Media Foundation. I'm stuck with a E_NOTIMPL error when I call IMFTransform::ProcessInput, leaving me a little in the dark.
These are the steps I've done up until now, I'm detailing them because it took me days to gather everything from the scarce scattered info across the web, so if it's resolved it will hopefully help others. Everything below is met with an S_OK:
I'm obtaining the surface through Duplication API, creating an IMFSample from it using MFCreateVideoSampleFromSurface
I'm getting a video encoder using IMFActivate::ActivateObject from an IMFActivate initialized with MFT_CATEGORY_VIDEO_ENCODER and MFVideoFormat_H264
I'm initializing IMFMediaType on the input with bitrate, framerate, aspect ratio, etc.. and most importantly: MFVideoFormat_NV12, seems to be the only one to work with the DXGI_FORMAT_B8G8R8A8_UNORM of Desktop Duplication API.
I'm setting IMFMediatype on the output with the same as the above, aside from MFVideoFormat_H264 as SUB_TYPE.
I'm calling the IMFTransform::SetOutputType then IMFTransform::SetInputType with the 2 above.
I'm setting the IMFSample time at 0 as it seems it's not being set by MFCreateVideaSampleFromSurface. Also setting the SampleDuration with MFFrameRateToAverageTimePerFrame with the input FPS.
After all of this, I call the MFTransform::ProcessInput with the IMFSample created above, and get a "E_NOTIMPL not implemented" as an HRESULT. I've read that I should set an IMFDXGIDeviceManager to my IMFTransform encoder above, so I did that using:
MFCreateDXGIDeviceManager from my ID3D11Device used with Desktop Duplication API and an arbitrary reset token.
Doing an IMFDXGIDeviceManager::ResetDevice with the device and token.
Calling IMFTransform::ProcessMessage(MFT_MESSAGE_SET_D3D_MANAGER, reinterpret_cast(m_pDXDeviceManager).
On this last step I get another "E_NOTIMPL not implemented" on my HRESULT, and that's where I don't know what I'm doing wrong anymore or what needs to be done.
The first thing to do, is to check if your encoder is MF_SA_D3D11_AWARE
I think it does not, reading this : MFT_MESSAGE_SET_D3D_MANAGER
If an MFT does not support this message, it should return E_NOTIMPL
But one thing is strange : DirectX Surface Buffer
Call IMFMediaBuffer::Lock. Generally this is not recommended, because it forces the object to copy memory from the Direct3D surface and then back again.
ProcessInput should have worked.
I am rendering a video file with a FilterGraph consisting of a VMR9 instance. The FilterGraph is created automatically with GraphBuilder->RenderFile(). Basically my setup is described here: http://www.codeproject.com/Articles/9206/Using-the-DirectShow-Video-Mixing-Renderer-filte
The thing is: I would like to detect some video internals like FPS, duration etc. After the call to RenderFile() the video is displayed correctly with MediaControl->StopWhenReady() and plays with Run() and Pause().
To detect the frame rate I try to grab the AM_MEDIA_TYPE struct from the VMR9's input pin:
VRM->FindPin("VMR Input0", pin); // S_OK
pin->ConnectionMediaType(&mt); // VFW_E_NOT_CONNECTED
In my opinion the filter graph should be created correctly with a call to RenderFile() and therefore this pin should be connected to my input stream. Why is it not the case and what is the way to proceed in this matter?
Microsoft provides some functions (http://msdn.microsoft.com/en-us/library/windows/desktop/dd375791%28v=vs.85%29.aspx) to traverse the graph and look for specific interfaces like IID_IAMStreamConfig that would allow the access to AM_MEDIA_TYPE. But these options fail in my implementation. The only pin I can access is the above mentioned.
Thanks in advance!
You are coming from the assumption that filter and pin, interfaces of which you hold, are connected and they are exactly the objects you are interested in. This is not necessarily true and quite a few question in past showed that people incorrectly understand the topologies they create. You need to review the filter graph and ensure you have what you expect to have. See on this: How can I reverse engineer a DirectShow graph?
One you have proper pin connection, indeed you need to use ConnectionMediaType and then go through AM_MEDIA_TYPE to VIDEOINFOHEADER or VIDEOINFOHEADER2 and then AvgTimePerFrame member.
I have a directshow Video renderer redived from CBaseVideoRenderer. The renderer is used in a graph that receives data from a live source (BDA). It looks like the connections are established properly, but the video rendering immediately ends because there is no sample. However, audio Rendering works, ie I can hear the sound while DoRenderSample of my renderer is never called.
Stepping through the code in the debugger, I found out that in CBaseRenderer::StartStreaming, the stream ends immedately, because the member m_pMediaSample is NULL. If I replace my renderer with the EVR renderer, it shows frames, ie the stream is not ending before the first frame for the EVR renderer, but only for my renderer.
Why is that and how can I fix it? I implemented (following the sample from http://www.codeproject.com/Articles/152317/DirectShow-Filters-Development-Part-Video-Render) what I understand as the basic interface (CheckMediaType, SetMediaType and DoRenderSample), so I do not see any possibility to influence what is happening here...
Edit: This is the graph as seen from the ROT:
What I basically try to do is capturing a DVB stream that uses VIDEOINFOHEADER2, which is not supported by the standard sample grabber. Although the channel is a public German TV channel without encryption, could it be that this is a DRM issue?
Edit 2: I have attached my renderer to another source (a Blackmagic Intensity Shuttle). It seams that the source causes the issue, because I get samples in the other graph.
Edit 3: Following Roman's Suggestion, I have created a transform filter. The graph looks like
an has unfortunately the same problem, ie I do not get any sample (Transform is not called).
You supposedly chose wrong path of fetching video frames out of media pipeline. So you are implementing a "network renderer", something that terminates the pipeline to further send data to network.
A renderer which accepts the feed sounds appropriate. Implementing a custom renderer, however, is an untypical task and then there is not so much information around on this. Additionally, a fully featured renderer is typically equipped with sample scheduling part, which end of stream delivery - things relatively easy to break when you customize it through inheriting from base classes. That is, while the approach sounds good, you might want to compare it to another option you have, which is...
A combination of Sample Grabber + Null Renderer, two standard filters, which you can attach your callback to and get frames having the pipeline properly terminated. The problem here is that standard Sample Grabber does not support VIDEOINFOHEADER2. With another video decoder you could possibly have the feed decoded info VIDEOINFOHEADER, which is one option. And then improvement of Sample Grabber itself is another solution: DirectX SDK Extras February 2005 (dxsdk_feb2005_extras.exe) was the SDK which included a filter similar to standard Sample Grabber called Grabber \DirectShow\Samples\C++\DirectShow\Filters\Grabber. It is/was available in source code and provided with a good description text file. It is relatively easy to extend to allow it accept VIDEOINFOHEADER2 and make payload data available to your application this way.
The easiest way to get data out of a DirectShow graph, if youњre not going to use
MultiMedia Streaming, is probably to write your own TransInPlace filter, a sub-variety
of a Transform filter. Then connect this filter to the desired stream of data you wish to
monitor, and then run, pause, seek, or otherwise control the graph. The data, as it passes
through the transform filter, can be manipulated however you want. We call this kind of
filter, a Њsample grabberћ. Microsoft released a limited-functionality sample grabber
with DX8.0. This filter is limited because it doesnњt deal with DV Data or mediatypes
with a format of VideoInfo2. It doesnњt allow the user to receive prerolled samples.
(Whatњs a preroll sample? See the DX8.1 docs) Its ЊOneShotћ mode also has some problems.
To add to this, the Grabber sample is pretty simple itself - perhaps 1000 lines of code all together, including comments.
Looks like your decoder or splitter isn't demuxing the video frames. Look further up the chain to see what filters are supplying your renderer pin with data, chances are its only recognising audio.
Try dropping the file into Graphedit (there's a better one on the web BTW) and see what filters it creates.
Then look at the samples in the DirectShow SDK.
I'm trying to stream a video from a camera ( Sony HVR-Z1E ) over FireWire to my Computer. The incoming pictures/stream shall be processed further by some functions which expect the CVMat format ( from openCV ).
Well my problem is now that I have no idea how to grab the stream. Okay openCV 2.1 offers me some methods ( cvCapturefromCam ) , but no matter which parameter I give him, it always gets the stream from the webcam of the laptop and not from the firewire. I heard I need to switch the primary cam in the DirectShow API ( with the Windows SDK ). But I actually don't know how to do that either.
So any suggestions how to do this?
See my related answer here. OpenCV cannot capture video from Firewire cameras natively. You will either need to use the CMU1394 driver, or the Sony driver (if an SDK is available for it) to capture video from that camera, and then pass it to OpenCV.
Years ago, I've made something like this using DirectShow. The main limitation was fact, that image acquired via DShow was in standard PAL resolution. HD Image grabbing was not possible (it was one of the first pro-consumer HD camcorders from Sony, don't remember exact model now). Good thing was - this method didn't need anything except bare DirectShow - no additional drivers and so on. And it was VERY fast.
In general, method was something like this:
building media render graph (of course, You have to enumerate video devices in that stage)
inserting into it custom class which inherited from ISampleGrabberCB.
How it worked:
it used BufferCB() virtual method from ISampleGrabberCB - which you have to write in your inherited class.
in mentioned method, you have to leave data in global struct, and from main thread - take care of them.
I know, that's a bit fuzzy description, but I hope You'll find Your info (googling for "ISampleGrabberCB" should be a good starting point, there should be a lot of sample code).