C++ web project using OpenCV with Wt - c++

What is a good platform for a web based project that does image processing using OpenCV library? I found Wt ( http://www.webtoolkit.eu/wt ).
Can I use OpenCV with Wt ? Is there any better alternatives to Wt?
Requirements:
A login page GUI to
upload documents,
select areas on image,
handwriting word/line detection using OpenCV

I've used Wt in the past, it is very useful, albeit bulky. It's bloat has to do with having to support a wide variety of web browsers, so in some cases it is a feature.
If you're more of a close-to-metal programmer, I'd recommend PION, and implementing your GUI using some of your web authoring skills:
http://www.pion.org/projects/pion-network-library
You can use OpenCV with pretty much any network library out there. A good review of your choices is available here on StackOverflow:
https://stackoverflow.com/questions/175507/c-c-web-server-library

I think what you ask is possible with Wt. I cannot foresee problems with linking OpenCV in Wt, and the system is definitely interactive enough to provide the functionality you describe. Implement it with server-side actions first, and if required, you can still optimize parts with small bits of client-side JS.

FWIW, this is a simple code to display OpenCV image (possibly changing the image while the app is running):
Wt::WMemoryResource* cvMat2res(const cv::Mat& img){
std::vector<uchar> buf;
cv::imencode(".png",img,buf); // by default, the fastest compression
auto ret=new Wt::WMemoryResource(this);
ret->setMimeType("mime/png");
ret->setData(buf); // data is copied here
return ret;
}
/* ... */
auto img=new Wt::Image();
root()->addWidget(img);
Wt::WMemoryResource* imgRes=nullptr;
/* set image data; this can be done also in event handler and the image updates itself automatically from the new resource */
if(imgRes) delete imgRes;
imgRes=cvMat2res(cvImage);
img->setImageLink(imgRes);

Related

Take high resolution photo from USB camera in Windows (C++)

I am developing an C++ application which should use an USB camera to capture high resolution photos. It should have same behavior as the Camera application in Windows 10. I am trying to use DirectShow for doing it. Now I am only able to take high resolution photo which is delayed or take a photo in time but low resolution. Also I am very confused from MS documentation, lot of things are deprecated and nowhere mentioned what replaces them. I'll describe my hopeless steps awaiting there will be somebody who could be able to show me a way.
Let's start from beginning...
Knowing nothing about video capturing in Window I started by searching suitable library. After some googling I found there are four main libraries for capturing video in Windows.
Video for Windows
DirectShow
Windows Media Foundation
OpenCV
Let's observe:
Video for Windows
This library is unfortunately marked as deprecated but it seems it still works. I have written "unfortunately", because I think this is the only which is easy to use. There are only a few lines of code needed for seeing video from camera. The only think I miss here is a "TakePhoto" function. You can use VFW for capture a video or single frames to an avi file. Or am I missing something?
DirectShow
This is much more complicated library. You need hundreds of lines of code to see a video preview. But you can obtain this code on MS Docs. Ok, now I have a video preview and I need only to take a photo. One would expect this should be just one function call. But where is the function? I did not find it.
You can simply use GetCurrentImage from IVMRWindowlessControl but this takes only one frame from preview with low resolution. If you set a higher resolution for preview the video is not fluent.
Best approach I could achieve is from an article called "Capturing an Image From a Still Image Pin" available here https://learn.microsoft.com/en-us/windows/desktop/directshow/capturing-an-image-from-a-still-image-pin. When I had found this site I thought I won and my task was almost finished. But it wasn't.
The first advice which the article gives you is not to use it: "The recommended way to get still images from the device is to use the Windows Image Acquisition (WIA) APIs. For more information, see "Windows Image Acquisition" in the Platform SDK documentation. However, you can also use DirectShow to capture an image." I tried to explore the WIA. But this stopped to work on Vista. I continued to study the article.
Everything seems to be clear but you need to implement your class which inherits ISampleGrabberCB marked as deprecated here https://learn.microsoft.com/en-us/windows/desktop/directshow/isamplegrabbercb. Why???? Where to find some alternative?
I found an acceptable solution here https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/2ab5c212-5824-419d-b5d9-7f5db82f57cd/qedith-missing-in-current-windows-sdk-v70?forum=windowsdirectshowdevelopment. You need to add header file from elder SDK. (BTW This is an advice almost ten years old.) After I compiled the application with this header I was able to read high resolution picture but I need to wait a few seconds which is unacceptable. I know the problem is not in camera, because in the it works in the Camera application. Furthermore the image is obtained in function SampleCB instead of BufferCB and is in some strange format. I can save it as jpg but it is not compressed enough.
Windows Media Foundation
I think MS doesn't like programmers and that's why it released WMF. I understand nothing. I found this tutorial https://www.dreamincode.net/forums/topic/347938-a-new-webcam-api-tutorial-in-c-for-windows/. It works but it only stores one frame from preview and this is not what I want.
Next I explored some WMF interfaces on MS Docs. IMFCapturePhotoSink interface should do the stuff. But how implement it. The documentation is useless.
OpenCV
During my research I found also this library. But again I'm not able to take a high resolution photo. It only stores one frame from preview.
Could someone tell me what should I focus on? I believe it cannot be so difficult. There are tens and hundreds of applications for webcams. How could other programmers implement them? What's wrong with me? I'd like to find an easy way to implement an easy task. Thank a lot for any help.
You question is not related to the topic - the question must be related to the code - but I faced with the similar problem many years ago and I had found solution:
DirectShow is declared as deprecated for Windows 10 and it has problem with supporting of the USB web cam. In Windows 10 there is USB Video Class which is supported only by Media Foundation.
So, I have wrote a simple C++ wrapper around Media Foundation code which simplify getting of the raw images Capturing Video from Web-camera on Windows 7 and 8 by using Media Foundation
Also, there is project CaptureManager SDK - it is DLL COM component with the simple interfaces, huge functionality and with many demo programs on C++, Python, C#, Java.
Thanks to Evgeny.
Recapitulation:
Download the CaptureEngine video capture sample
Edit CaptureManager::TakePhoto method. Add the code to find highest resolution media type just before CreatePhotoMediaType(pMediaType, &pMediaType2); line
Extra code for setup the photo stream to highest resolution:
DWORD dwMediaTypeIndex = 0;
UINT32 maxSize = 0;
DWORD maxSizeIndex = 0;
while (1) {
IMFMediaType* pMediaType = NULL;
hr = pSource->GetAvailableDeviceMediaType((DWORD)MF_CAPTURE_ENGINE_PREFERRED_SOURCE_STREAM_FOR_PHOTO, dwMediaTypeIndex, &pMediaType);
if (hr == MF_E_NO_MORE_TYPES)
break;
UINT32 w, h;
MFGetAttributeSize(pMediaType, MF_MT_FRAME_SIZE, &w, &h);
UINT32 size = w * h;
if (size > maxSize) {
maxSize = size;
maxSizeIndex = dwMediaTypeIndex;
}
SafeRelease(&pMediaType);
dwMediaTypeIndex++;
}
SafeRelease(&pMediaType);
pSource->GetAvailableDeviceMediaType((DWORD)MF_CAPTURE_ENGINE_PREFERRED_SOURCE_STREAM_FOR_PHOTO, maxSizeIndex, &pMediaType);

Is it okay to execute ffmpeg and do not use their library?

I need to extract audio from video and save it. FFmpeg has command for this purpose. I wonder if it is a right way to execute ffmpeg from my code and not to write code with their API functions.
The lack of this approach is that I use Qt Framework and need cross-platform application. Sometimes (especially in windows, because PATH variable doesn't set up automatically so call ffmpeg won't work) a user will need to indicate path to executable file to run in command line.
So both variants are possible to realize, but which is the best and correct one?
I don't really want to use their API because it is not so easy to understand and will take time to write my own code.
Thanks for any advice!
Using standalone ffmpeg seems to be preferred in your case. You will have to bundle ffmpeg and it's dependencies along with your application. However there is no need to set or use PATH or other environment variables to launch ffmpeg. You should do it by supplying full path to ffmpeg executable.
Using libav API is indeed rather tricky. And I would like to mention that in general (depending on codec) ffmpeg and libav should not be considered stable and you should spawn a separate process to protect main executable from potential crash in this case as well. So complexity of this approach is much higher compared to first one.
Disclaim: I never used Qt with ffmpeg together myself, but have much experience with Qt especially.
Qt tends to try having everything in their library, wrapping many other content for convenience. Most of the time (All those I tested), it is still possible quite easily to use the original library without troubles, but the Qt facilitate integration.
As an example: QOpenGLWidget is a wrapper for OpenGL with their widget system, adding signals and slots, etc. I made some test using normal OpenGL and it worked fine.
In another project, we(my team, not me particularly) used ffmpeg to display video on a QtWidget. It works with limited problems (due to other architectural requirements).
Considering your use case, and especially that you are using ffmpeg for background processing and not for displaying video, you may IMO go ahead with high probability of success.

Running a Qt app over the web

I am writing an application using Qt and want to try and deploy it as a web-application. I want user's to be able to use my application by accessing it through a web browser. I'm guessing that's what a web-application is? What kind of options do I have? I've never looked into doing anything like this but I'd like to learn something new.
EDIT: What if I deployed my application on a Linux server and had users access/run it through a terminal? I think writing web application is going to be more complicated than I had originally thought.
If all you have is a Qt application, then the best you can do is use Qt 5 and run it using a remote visualization package:
Use WebGL streaming, introduced in Qt 5.10. Qt exposes a browser-connectible interface directly, without need for third-party code.
For Qt 5.0-5.9, you can use the vnc platform plugin. Then connect using a web-browser based vnc client.
For many uses it might be sufficient, and certainly it's much less effort than coding up a web app.
You're looking for Wt which provides a different set of drawing routines for many Qt GUI elements, turning them from lines on screen to HTML controls.
http://www.webtoolkit.eu/wt
It also handles websocket calls to provide interactivity. It seems a great idea, let us know how it works in practice.
For the case of QML there is QmlWeb which is a JavaScript library that is able to parse QML-code and create a website out of it using normal HTML/DOM elements and absolute positions within CSS, translating the QML properties into CSS properties.
QmlWeb is a small project by Lauri Paimen that he’s already developing for a few years now. QmlWeb of course doesn’t yet support everything Qt’s implementation of QML does, but it already supports a quite usable subset of it. It supports nearly all of the most basic QML syntax. Moreover it has support for HTML input elements (Button, TextInput, TextArea are currently supported, more to come).
Well, QmlWeb is not finished. I hope Digia help with this project to make it ready with mature features.
Interestingly, it is possible to compile Qt applications to javascript using emscripten-qt. These run fairly fast with Firefox's asm.js interpreter:
http://vps2.etotheipiplusone.com:30176/redmine/projects/emscripten-qt/wiki
Try "Qt for Webassembly".
Webassembly allows the C/C++ code to be compiled and run natively inside majority of the browsers:
WebAssembly (Wasm, WA) is a web standard that defines a binary format and a corresponding assembly-like text format for executable code in Web pages. ... It is executed in a sandbox in the web browser after a verification step. Programs can be compiled from high-level languages into Wasm modules and loaded as libraries from within JavaScript applets ... Its initial aim is to support compilation from C and C++, though support for other source languages such as Rust and .NET languages is also emerging.
To run a Qt application unchanged over the web so users can operate it in a browser, you can compile it for Android using the x86 Android ABI, run it inside an Android emulator on a server and supply the Android Cast videostream to users' browsers. You'll also need to have JavaScript in place that records the keyboard and mouse events on the web clients and relays them back to the server.
I had previously tried Qt WebGL streaming and found it to be good over the local network but too slow over the Internet. A 10 s application startup time is acceptable, but 3 s to show a new screen is rather not. I had the exact same experience with the Qt VNC platform plugin. Compared with that, the Android Cast streaming based appetize.io solution (see below) was much faster, providing a well usable user experience even over my 8 Mbit/s connection.
Existing solutions
Here is an overview of commercial products and open source software components that I found that can help you with this approach:
appetize.io. This is a commercial product to run Android applications over the web for demo and testing purposes. I have just done this with a Qt QML based application and liked the outcome. When choosing an Android 9 / 10 device you can see that the "Screencast" setting is on; which is why I believe that this solution uses the Android Cast technology.
runthatapp.com. This is another commercial offer. Not as sophisticated (yet) as appetize.io, but providing a nice pay-as-you-go scheme.
ScreenStream. An open source Android app that provides a web server to view the screen of one Android device in a web browser, also relying on the Android Cast technology. That Android device could be an emulator running on a web server. And to make this multi-user capable you can employ a small load balancer similar to a technique that I developed for Qt WebGL streaming. The ScreenStream README shows that the application might consume up to 20 Mbit/s per client in short bursts.
Ideas for future improvements
Serving your Qt app as an interactive live video stream seems a promising idea to me, given that I found it already less sluggish than VNC and similar solutions. There are ways to make this even faster, such as using a hardware H.265 video encoder to create a video stream with very little delay. By operating multiple such encoders on a single server, the server could serve multiple clients and still keep its CPU load low. Maybe there are even better video formats for such a purpose, given that user interfaces of programs lend themselves well to lossless compression.
Some hints for appetize.io
Finally: since I used the appetize.io product for a Qt application over the last few days, here are some tips from that experience:
It is necessary to compile your Qt application for the x86 Android ABI. The default armeabi-v7a ABI will not work because most appetize.io devices are actually server-based Android emulators and the only ARM based device ("Nexus 5 Physical") failed to start any Qt application I tried to use with it.
The x86_64 ABI may also work, but you might then have to also compile Qt yourself for it, as not all versions of Qt come pre-compiled for that architecture.
All appetize.io links (both for standalone pages and embeddable iframes) support GET parameters to configure the app presentation format. Especially relevant here is screenOnly=true to show the app without a picture of a phone or tablet around it.
Features that rely on phone hardware (camera, position etc.) will not work or only show dummy data. But if you really wanted, you could create a hybrid application combined with client-side JavaScript. It would run device-dependent code in the user's browser, for example to take a photo with the webcam, and then provide the results to the Qt application via the appetize.io cross-document messaging protocol. The following message types seem suitable to build a simple communication protocol: pasteText(value), keypress(key, shiftKey) and openUrl(value).
In the default appetize.io standalone app demo pages, only the key events of ordinary letter keys are sent to the app, not keyboard shortcuts or function keys like F2 and Esc. This might be possible to fix with JavaScript on an own page embedding the appetize.io iframe, as their cross-document messaging protocol provides the keypress(key, shiftKey) message type.
Qt does not support writing browser based web applications. Unfortunately.
You need to use common web programming technologies for this. There are a lot of ways, but Qt is not one of them.

Audio Signal Processing GUI developement using Qt, How to?

I have some basic effect algorithms (i.e chrous, LP filtering..) which I would like to build a GUI application to be able to use these algorithms.
For example I want to be able to open an audio file, process the audio file in some way with my algorithms and playback the processed file.
Later on I would like to, if possible be able to see the waveforms of the original file and the processed file in the GUI application. This is my objective now.
In the future I want to be able to create a user interface through which users can be able to use my own audio processing algorithms on files of their own.
Is it possible to design such a GUI with the Qt programming framework? If so, could someone point me in the right direction to get started? Right now I have the Qt SDK 1.1 beta running on Windows 7 OS and also using Qt creator. I would really appreciate some guidance.
Qt is a very powerful application framework, but do not expect any extra help with DSP tasks from it. It contains API for some basic and common tasks, like playing an audio/video file, working with audio devices, creating audio effects (search QAudio and Phonon in the Qt's help) etc. You can use some ready-to-use widgets and create your own multimedia player in a few moments.
But in the DSP you are -mostly, on your own. There is, for instance, only a limited audio file format support, so if you want to work with more formats than .wav and .aiff, use some specialized library. I recommend libsndfile (http://www.mega-nerd.com/libsndfile/) which is most powerful free audio file library available. And if you plan your effects to be more universal, use rather VST technology by Steinberg - today's audio plug-in standard, but it is relatively complicated, not suitable for beginners.
There is no built-in widget which can show a waveform, you have to create it yourself, but it is not much complicated. Qt has a really cool drawing functions, brushes, texts, gradients, transformations, antialiasing, even OpenGL wrapper - everything ready and very simple to use.
So the answer is definitely yes. I use Qt in my multimedia applications for three years and now I can't see how I could live without it (using VST GUI and Windows APIs before).
Sure its possible, QT is a framework for writing applications, you can write any application you want using it, you'll probably end up needing to write some custom controls. As an example, here's an Open source QT based application that does pretty much everything you are talking about and much more:
http://qtractor.sourceforge.net/qtractor-index.html

C++: FFMPEG and SDL resources

I'm looking for resources (preferably books, but websites are fine too) for using FFmpeg and/or SDL with C++.
Stuff I'd like to be able to do (eventually):
Decode and play videos in realtime to a QT widget (the QT part isn't a problem)
Overlay text and images on the video (in realtime)
Loop video
Cross-fade from one video to another (in realtime)
Some kind of DVD functionality
LIVE sources? (i.e. webcam, stream, etc.)
So far I've looked at (and consider helpful) the following resources:
FFmpeg and SDL Tutorial by Dranger (extremely helpful)
Stack Overflow: ffmpeg C API documentation/tutorial (not a duplicate, btw)
Sample code bundled with FFMPEG libraries
(I'll try and update this list with anything helpful below)
Thanks for any help...
Also: Operating System is Windows (but maybe one day cross-platform)
Also 2: Resources using alternatives are welcome too... i.e. DirectShow, VFW, etc.
If all you need is to decode and play videos and require overlays I would consider using the Phonon framework, and use QT Graphics View by using a Phonon::VideoWidget inside a QGraphicsProxyWidget. That way you can easily get overlays, cross-fading, animations etc. Phonon in Windows uses DirectShow as a back-end. You can install FFDShow and get the same codecs ffmpeg has to play videos.
There is an FFMPEG C++ wrapper library : FOBS. It is quite simple to use, but what you gain in simplicity you lose in fine grained configuration. It works great but is always a bit outdated.
m2c