I'm trying to write a simple AR app in ReactNative, it should simply see 4 predefined markers and draw a rectangle as a boundary on the live preview of the camera, the thing is I'm trying to do the processing in C++ using opencv so as to have the logic of the app in one place accessible to both Android & IOS.
here's what I've been thinking
write the OS dependent code to open the camera and get permissions in (java/ObjC) & the C++ part to do processing on each frame.
call the C++ code (from within the native code) on each frame, and that should return lets say coordinates for the markers.
draw the rect if 4 markers found on the preview in native code (No idea how to achieve this so far but I think it will be native code).
expose that preview (the live preview with the drawn view) to ReactNative (Not sure about that or how to achieve it)
I've looked at the react native camera component but it doesn't provide access to frames & if that's even possible, I'm not sure if it would be a good idea to send frames over the bridge between JS & java/ObjC.
the problem is that I'm not sure of the performance or if that is even possible.
if you know of any ReactNative library that would be great.
Your steps seem sound. After processing the frame in C++, you will need to set the application properties RCTRootView.appProperties in iOS, and emit an event using RCTDeviceEventEmitter on Android. So, you will need an Objective-C wrapper for your C++ code on iOS and a Java wrapper on Android. In either case, you should be able to use the same React Native code for actually drawing the rectangle on top of the camera preview. You're right that the React Native camera component does not have an API for getting individual frames from the camera, so you'll need to write that code natively for each platform.
Related
I wonder if I can write my own native module, render something with using OpenGL in C++ and finally display rendered picture on react native side ( by simply using component).
If so, can I use that to render an animation in for example 60fps?
My case is that I've got the custom, let's say, game renderer written in OpenGL, and I looking for some fancy solution to create an editor detached from engine code.
I've already analyzed some react-native video libraries and I've discovered that frames are injecting as the texture of components, but I'm not sure is it the best solution (I can't find any documentation of those low-level mechanisms in react native).
Any advice? Thanks in advance!
How can I access pen input with pressure on Windows? I am making a paint program, and had been using wintab, but wintab does not exist on my new computer. Wintab seems to be deprecated, but there are apparently some newer APIs called Windows Ink and Tablet PC.
The problem is that I cannot find documentation or an example of how to actually use a recent pen API. The API needs to be usable from a normal, unmanaged C++ desktop application. Also, I would really rather avoid UWP if possible, because I don't want to deal with "deployment" or "signing".
Where is an SDK I can download that has C/C++ headers and libraries that will give me raw pen input?
I got better results using the Realtimestylus API instead of wintab. Because wintab doesn't seem to return packets frequent enough for smooth lines when drawing.
The windows RTS API gave me more information packets, thus smoother results when drawing, and also more pen information can be accessed such as tilt and pen button presses.
there's a nice small demo app using RealTimeStylus with SDL:
https://backworlds.com/under-pressure/
It includes comments in the implementation, where it explains how to access raw tablet input.
The sample code contains an eventhandler class derived from the interface class IStylusSyncPlugin from rtscom.h.
With this interface you can register pen and touch events using STDMETHODS, for example: STDMETHOD(StylusButtonDown){}.
To get the actual pen position in screen coordinates is a bit difficult. As you first need to get the x,y coordinates as packets, using STDMETHOD(Packets). You will also need to describe which packets you want to receive and in what order using: IRealTimeStylus::GetPacketDescriptionData().
Lastly you need to convert the X, Y input into screen-coordinates depending on the dpi,
using Gdiplus::Graphics::GetDpiX()
X = dpiX * (float)pLastPacket[g_lContexts[iCtx].x] / 2540;
Y = dpiY * (float)pLastPacket[g_lContexts[iCtx].y] / 2540;
All this is provided in the example code by Anders Ekermo.
Hope it helps.
Look at RealTimeStylus and the MS Pointer API (WM_PointerXxx messages). These are two different ways.
So I am trying to figure out how get a video feed (or screenshot feed if I must) of the Desktop using OpenGL in Windows and display that in a 3D environment. I plan to integrate this with ARToolkit to make essentially a virtual screen. The only issue is that I have tried manually getting the pixels in OpenGl, but I have been unable to properly display them in a 3D environment?
I apologize in advance that I do not have minimum runnable code, but due to all the dependencies and whatnot trying to get an ARToolkit code running would be far from minimal. How would I capture the desktop on Windows and display it in ARToolkit?
BONUS: If you can grab each desktop from the 'virtual' desktops in Windows 10, that would be an excellent bonus!
Alternative: If you know another AR library that renders differently, or allows me to achieve the same effect, I would be grateful.
There are 2 different problems here:
a) Make an augmentation that plays video
b) Stream the desktop to somewhere else
For playing video on an augmentation you basically need to have a texture that gets updated on each frame. I recall that ARToolkit for Unity has an example that plays video.However.
Streaming the desktop to the other device is a problem of its own. There are tools that do screen recording, but you probably don't want that.
It sounds to me that what you want to do it to make a VLC viewer and put that into an augmentation. If I am correct, I suggest you to start by looking at existing open source VLC viewers.
I am trying different implementations of how to achieve further processing from the QML camera, I need to pass on the feed to the C++ end so it can then be converted to a cv::Mat image and passed to a function for processing. I have tried setting a QCamera from the C++ end and starting it on a button click but it seems Qt cannot create two instances of the same camera. I have also tried an open CV method but to no avail. What is the best way to do this?
This is a not a fix but a workaround. I used an OpenCv plugin for the camera and made it visible to my QML using qmlRegisterType. I could then easily send the frames from the backend to my other class for processing. For anyone looking to do this I used this plugin: https://github.com/rferrazz/CvCamView
Qt QML Camera to C++ QImage on Android
I just answered this question, I think you were having the same issue.
The basic Idea here is to get the instance of QML camera, access its QMediaObject point and probe it with QVideoProbe. There are other solutions, but AFAIK they arent really easy or fast in Android platforms, but if thats not your case, you probavly should try QAbstractVideoFilter alongside QVideoFilterRunnable classes which were developed specially for post processing qml video feed.
Note that QAbstractVideoFilter and QVideoFilterRunnable are Qt 5.5 classes only.
Simple example - on one side we see camera rendered via standard software rendered "Input" on other hand (labeled "Output") rendered via some directX stuff (at least it seems to me) :
So what function is provided by windows api or DirectX api for capturing such mixed scenes?
TightVNC Server can do it, you may want to look into what they are doing. From a simple glance through their source code it looks like they are creating a virtual screen that mirrors the primary screen.
Specifically though, look into the
CreateCompatibleDC and CreateDIBSection API's
As I known, there is not a direct way to capture DirectX render area, although we can see that on the screen. Because the real render action(aka render instruction) happens in hardware layer. So the API in standard SDK cannot know the finally render result, which lead to the black square.
The only way to do this maybe put your hope on the Render layer(such as DirectX engine) itself can support output interface as well as underlying render action. So I suggest to check some documentation to find if there indeed is.
DirectX can present to a limited subsection of the window that you give it, enabling you to create small regions of DX content in larger windows.