Ok so I've used a chessboard image to calibrate my camera. This calibrates both the intrinsic and extrinsic parameters of the scene. Now I want to draw an object to sit on the table in which I had placed the chessboard. My question is do need the rotation+translation vectors of the camera to do this? And if so how can I get these after doing the calibration? Or are these vectors taken into account in the calibrateCamera function?
Basically after I calibrate the camera how can I now draw into the scene on top of a surface?
Thanks!
The calibrateCamera function provides you with the extrinsic (rotation+translation) and intrinsic (camera matrix K and distortion coefficients).
This allows you to project 3D points, expressed in the coordinate system of your chessboard, into an image acquired by the camera, for example using the projectPoints function (link). Using this approach, you can draw wireframe objects directly into the image by projecting the 3D edges using the projectPoints function.
If you would like to render more complex objects (i.e. including textures, etc), this requires using an auxiliary rendering engine since OpenCV does not provide such functionalities. You will have to use a rendering engine, say OpenGL, provide the projection details to it, retrieve the object rendered from the camera viewpoint in a buffer, and overlay this buffer on top of your initial image.
However, notice that the result of this, which is called augmented-reality, may look weird sometimes, because it does not take into account the occlusions between your rendered object and the scene observed in the image... Handling occlusions appropriately would require a 3D model of the scene.
AldourDisciple post is a great answer.
If I can add my 2 cents, this is a problem I worked with, and it is the basic problem of Augmented Reality, and more generally on how to integrate OpenCV and OpenGL (because you'll gonna use OpenGL to draw the 3d stuffs and use a OpenGL texture to show the underline OpenCV images from the video). You can find some (imho) good links, documentation and tutorial and references in my previous answer on that.
Related
I have a very simple OpenGL (3.2) setup, no lighting, perspective projection and a simple shader program (applies projection transformation and uses texture2D to read the color from the texture).
The camera is looking down the negative z-axis and I draw a few walls and pillars on the x-y-plane with a texture (http://i43.tinypic.com/2ryszlz.png).
Now I'm moving the camera in the x-y-plane and this is what it looks like:
http://i.imgur.com/VCrNcly.gif.
My question is now: How do I handle the flickering of the wall texture?
As the camera centers the walls, the view angle onto the texture compresses the texture for the screen, so one pixel on the screen is actually several pixels on the texture, but only one is chosen for display. From the information I have access to in the shaders, I don't see how to perform an operation which interpolates the required color.
As this looks like a problem nearly every 3D application should have, the solution is probably pretty simple (I hope?).
I can't seem to understand the images, but from what you are describing you seem to be looking for MIPMAPPING. Please google it, it's a very easy and very generally used concept. You will be able to use it by adding one or two lines to your program. Good Luck. I'd be more detailed but I am out of time for today.
I need some help in surface area selection on a 3d model rendered in opengl by picking points through mouse. I know how to get a point in world coordinate but cant find a way to select an area. Later I need to remesh that selected area and map an image over it which I know.
Well, OpenGL by itself can't help you there. OpenGL is a drawing API. You draw things, but once the drawing commands have been executed all that's left are pixels in a framebuffer and OpenGL has no recollection about the geometry whatsoever.
You can use OpenGL to implement image based area selection algorithms, for example by drawing each face with a unique index color into an off screen framebuffer. Then by looking at what values can be found therein you know which faces are present in a given area.
Later I need to remesh
This is called topology modification and is completely outside the scope of OpenGL.
that selected area and map an image over it which I know
You can use a image based approach for this again, however you must know in which way you want to make images to faces first. If you want to unwrap the mesh, then OpenGL is of no help. However if you want the user to be able to "directly draw" onto the mesh, this can be done by drawing texture coordinates into another off screen framebuffer and by this reverse mapping screen coordinates to texture coordinates.
I have been brought in on a project where I need to render a 3D volume from a series of images of the volume. The images have been created by a couple of techniques such that they are vertical slices of the object in question.
The data set is similar to this question, but the asker is looking for a Matlab solution.
The goal is to have this drawing be in something near real time (>1Hz update rate), and from my research openGL seems to be the fastest option for drawing. Is there a built in function in openGL render the volume in openGL other than the following psuedocode algorithm.
foreach(Image in Folder)
foreach(Pixel in Image)
pointColour(pixelColour)
pointLocation(Pixel.X,Pixel.Y,Image.Z)
drawPoint
I am not concerned about interpolating between images, the current spacing is small enough that there no need for it.
I'm afraid if you're thinking about volume rendering, you will need to first understand the volume rendering integral because the resultant color of a pixel on the screen is a function of all the voxels that line up with it for the current viewing angle.
There are two methods to render a volume in real-time using conventional graphics hardware.
Render the volume as a set of 2D view-aligned slices that intersect the 3D texture (proxy geometry). Explanation here.
Use a raycaster that uses programmable graphics hardware, tutorial here.
This is not an easy problem to solve - but depending on what you need to do things might be a little simpler. For example: Do you care about having an interactive transfer function? Do you want perspective views, or will orthographic projection suffice? Are you rendering iso-surfaces? Are you using this only for MPR-type views?
I've implemented the volume render using ray-casting in CUDA. Now I need to add other 3D objects (like 3D terrain in my case) in the scene and then make it interact with the volume-render result. For example, when I move the volume-render result overlapping the terrain, I wish to modulate the volume render result such as clipping the overlapping part in the volume render result.
However, the volume render result comes from a ray accumulating color, so it is a 2D picture with no depth. So how to implement the interaction makes me very confuse. Somebody can give me a hint?
First you render your 3D rasterized objects. Then you take the depth buffer and use it as an additional data source in the volume raycaster as additional constraint on the integration limits.
Actually, I think the result of ray-casting is a 2D image, it cannot interact with other 3D objects as the usual way. So my solution is to take the ray-casting 2D image as a texture and blend it in the 3D scene. If I can control the view position and direction, we can map the ray-casting result in the exact place in the 3D scene. I'm still trying to implement this solution, but I think this idea is all right!
Is there a way to extract a point cloud from a rendered 3D Scene (using OPENGL)?
in Detail:
The input should be a rendered 3D Scene.
The output should be e.g a three dimensional array with vertices(x,y,z).
Mission possible or impossible?
Render your scene using an orthographic view so that all of it fits on screen at once.
Use a g-buffer (search for this term or "fat pixel" or "deferred rendering") to capture
(X,Y,Z, R, G, B, A) at each sample point in the framebuffer.
Read back your framebuffer and put the (X,Y,Z,R,G,B,A) tuple at each sample point in a
linear array.
You now have a point cloud sampled from your conventional geometry using OpenGL. Apart from the readback from the GPU to the host, this will be very fast.
Going further with this:
Use depth peeling (search for this term) to generate samples on surfaces that are not
nearest to the camera.
Repeat the rendering from several viewpoints (or equivalently for several rotations
of the scene) to be sure of capturing fragments from a the nooks and crannies of the
scene and append the points generated from each pass into one big linear array.
I think you should take your input data and manually multiply it by your transformation and modelview matrices. No need to use OpenGL for that, just some vector/matrices math.
If I understand correctly, you want to deconstruct a final rendering (2D) of a 3D scene. In general, there is no capability built-in to OpenGL that does this.
There are however many papers describing approaches to analyzing a 2D image to generate a 3D representation. This is for example what the Microsoft Kinect does to some extent. Look at the papers presented at previous editions of SIGGRAPH for a starting point. Many implementations probably make use of the GPU (OpenGL, DirectX, CUDA, etc.) to do their magic, but that's about it. For example, edge-detection filters to identify the visible edges of objects and histogram functions can run on the GPU.
Depending on your application domain, you might be in for something near impossible or there might be a shortcut you can use to identify shapes and vertices.
edit
I think you might have a misunderstanding of how OpenGL rendering works. The application produces and sends to OpenGL the vertices of triangles forming polygons and 3d objects. OpenGL then rasterizes (i.e. converts to pixels) these objects to form a 2d rendering of the 3d scene from a particular point of view with a particular field of view. When you say you want to retrieve a "point cloud" of the vertices, it's hard to understand what you want since you are responsible for producing these vertices in the first place!