I wonder if I can write my own native module, render something with using OpenGL in C++ and finally display rendered picture on react native side ( by simply using component).
If so, can I use that to render an animation in for example 60fps?
My case is that I've got the custom, let's say, game renderer written in OpenGL, and I looking for some fancy solution to create an editor detached from engine code.
I've already analyzed some react-native video libraries and I've discovered that frames are injecting as the texture of components, but I'm not sure is it the best solution (I can't find any documentation of those low-level mechanisms in react native).
Any advice? Thanks in advance!
Related
I am writing an Application for Android using Qt (with Android Build Kit), Felgo and QtQuick.
The App should be able to display a video which is decoded by libvlc. The problem is, that I do not know, how to display the frames provided by libvlc (format is RGB 24bit). I looked up several approaches in the internet but they were all outdated (e.g. used some OpenGL functions, which are not available in the Kit).
So does somebody knows how to efficiently render the raw rgb-data into a QtQuick Item?
(I know it is possible with a PaintedItem and Painter, but this uses a QImage which makes the process to inefficient.)
(I do not know much about OpenGL, so if your solution includes it, please be so kind and explain the functions you are using.)
I'm trying to write a simple AR app in ReactNative, it should simply see 4 predefined markers and draw a rectangle as a boundary on the live preview of the camera, the thing is I'm trying to do the processing in C++ using opencv so as to have the logic of the app in one place accessible to both Android & IOS.
here's what I've been thinking
write the OS dependent code to open the camera and get permissions in (java/ObjC) & the C++ part to do processing on each frame.
call the C++ code (from within the native code) on each frame, and that should return lets say coordinates for the markers.
draw the rect if 4 markers found on the preview in native code (No idea how to achieve this so far but I think it will be native code).
expose that preview (the live preview with the drawn view) to ReactNative (Not sure about that or how to achieve it)
I've looked at the react native camera component but it doesn't provide access to frames & if that's even possible, I'm not sure if it would be a good idea to send frames over the bridge between JS & java/ObjC.
the problem is that I'm not sure of the performance or if that is even possible.
if you know of any ReactNative library that would be great.
Your steps seem sound. After processing the frame in C++, you will need to set the application properties RCTRootView.appProperties in iOS, and emit an event using RCTDeviceEventEmitter on Android. So, you will need an Objective-C wrapper for your C++ code on iOS and a Java wrapper on Android. In either case, you should be able to use the same React Native code for actually drawing the rectangle on top of the camera preview. You're right that the React Native camera component does not have an API for getting individual frames from the camera, so you'll need to write that code natively for each platform.
For a while I've been using SDL to write my 3D engine,and have recently been implementing an editor that can export an optimized format for the type of engine Im building. Right now the editor is fairly simple, objects can just be moved around and their textures and models can be changed. As of right now, I'm using SDL with OpenGL to render everything, but I want to use Qt for the GUI part of the editor, that way it looks native on every platform. I've got it working great so far, I'm running a QApplication inside of the SDL application, so it basically just opens 2 windows, one that uses SDL and OpenGL, and the other using Qt. Doing a bit of research, I've found that you can manually update a QApplication, which totally removes any threading problems, and everything works. Just in case you're having a hard time visualizing this, heres a picture:
What my goal is to merge these windows into one, because on smaller screens (like my laptop's) it makes it really hard to keep track of all the different windows that I would eventually have. I know theres a way to render to Qt with OpenGL, but can this be integrated with SDL? Am I going to need to move away from using an SDL window and use a QT one if the editor is enabled? Just to clarify, when the engine isn't in editor mode, it won't use and Qt, just SDL, so optimally I wouldn't need to do this.
Drop the SDL part. You have Qt, which does everything SDL does as well. Just subclass a QGLWidget and use that.
You can keep your game and editor separate processes and still make them part of the same app.
Just spawn the window where you want the game to run as part of Qt, and at least in windows, you can then pass the window handle to your game, and make sure when your game is setting up, instead of creating the window yourself in SDL and binding the opengl context to it, you can actually bind to the existing handle.
There are some gotchas with this technique to watch out for such as input focus I believe (I tested with directX, but it might be similar with SDL). The problem is that the foreground mode does some dumb checks on the "root" window which for me was not the window that owned the opengl context, so it failed to initialize. However background mode worked. I think that was for a joystick now that I think about it, but anyway, that's how you can merge everything together.
I plan to embed an unmanaged C++ OpenGL viewer in WPF. The requirements are (apart from robustness and ease of development)
proper resizing
context menu for the OpenGL viewer area (only for viewer related actions)
events (e.g. triggered by selection of an object in the OpenGL viewer)
WPF dialogs should be drawn on top of the OpenGL space
So far I've found some blogs and discussions about using a hosted WinForm control (via C++/CLI wrapper), but it seems to be bit tricky (even a bit tinkery) to get all my above requirements done (if possible).
How about creating an ActiveX control? Despite having no experience with creating an ActiveX control, one advantage would already be that it runs in its own process.
Is it a good idea or would I run into even more tinkering?
EDIT: I should be clearer about the OpenGL viewer. It's an existing app based on an C++ 3D engine which only supports OpenGL. Porting to an DirectX capable engine is considered as last resort.
Although I didn't use ActiveX, I've accomplished what I believe you are trying to achieve. Two pieces of information helped me down this road.
Mixing Managed and Unmanaged code
It turns out, that managed C++ compiles completely differently than unmanaged C++, and data is stored on a completely different heap. Luckily you can mark sections of code as managed or unmanaged by using...
#pragma managed
#pragma unmanaged
See more info at this article
OpenGL context management
I followed a method similar to what's described in this article
wglMakeCurrent and wglShareLists will be your buddy if you need to manage multiple OpenGL windows
Good Luck!
Why you want an activex? There is some managed PInvoke based wrappers as SharpGL doing the job you need easyear.
But also, since you are using WPF and you have a Viewport3d object,why don't use it? If you are familiar with 3d engines you will appreciate the semplicity ( even if there is some drawback in term of speed).
Another point, since as per this question here I was forced to abandon WPF 3d rendering in favor of OpenGl due to the fact that Opengl can render in session 0, by using pure software mode. This is a requirement if you are planning to render something in off-screen bitmaps from a service or an asp.net application.
I'm writing a plugin in firebreath, C++.
I don't have any experience with both, so my question may be very basic.
How do I place a JPEG image inside my plugin window?
Or at least, how do I do it in C++ simple program?
Thanks,
RRR
There are a couple of other questions that may help you better understand this:
How to write a web browser plugin for IE, Firefox and Chrome
Directx control in browser plugin
Basically you'll get a drawing model from FireBreath with the AttachedEvent. Depending on your platform, you will draw to that window using platform-specific drawing APIs. On Windows, for example, you would get the HWND from the PluginWindow (cast it to a PluginWindowWin) and then draw to that. Just make sure you stop drawing when DetachedEvent shows up.
For more information, you'll need to be a lot more specific; but follow those links and do some reading, then you'll know better what questions to ask.
FireBreath 1.5.2 was just released, btw! Good luck!
Good luck!
You can also use OpenGL to display images in plugin. You can get several tutorials to load jpeg image in OpenGL as texture. Same code can be ported into the Firebreath plugin using the already given OpenGL sample plugin for windows. Though OpenGL context creation will vary from one OS to the other. If you want to load jpeg images from web, you'll have to download image before converting it into opengl texture.