Get image pixels using mask - c++

So i have some original image.
I need to get and display some part of this image using specific mask. Mask is not a rectangle or a shape. It contains different polygons and shapes.
Are there any methods or tutorials how to implement that? Or from where to start to make this? Shall i write some small shader or compare appropriate pixels of the images or something like this?
Will be glad for any advises.
Thank you

You can put your image in a UIImageView and add a mask to that image view with something like
myImageView.layer.mask = myMaskLayer;
To draw your custom shapes, myMaskLayer should be an instance of your own subclass of CALayer that implements the drawInContext method. The method should draw the areas to be shown in the image with full alpha and the areas to be hidden with zero alpha (and can also use in between alphas if you like). Here's an example implementation of a CALayer subclass that, when used as a mask on the image view, will cause only an oval area in the top left corner of the image to be shown:
#interface MyMaskLayer : CALayer
#end
#implementation MyMaskLayer
- (void)drawInContext:(CGContextRef)ctx {
static CGColorSpaceRef rgbColorSpace;
static dispatch_once_t onceToken;
dispatch_once(&onceToken, ^{
rgbColorSpace = CGColorSpaceCreateDeviceRGB();
});
const CGRect selfBounds = self.bounds;
CGContextSetFillColorSpace(ctx, rgbColorSpace);
CGContextSetFillColor(ctx, (CGFloat[4]){0.0, 0.0, 0.0, 0.0});
CGContextFillRect(ctx, selfBounds);
CGContextSetFillColor(ctx, (CGFloat[4]){0.0, 0.0, 0.0, 1.0});
CGContextFillEllipseInRect(ctx, selfBounds);
}
#end
To apply such a mask layer to an image view, you might use code like this
MyMaskLayer *maskLayer = [[MyMaskLayer alloc] init];
maskLayer.bounds = self.view.bounds;
[maskLayer setNeedsDisplay];
self.imageView.layer.mask = maskLayer;
Now, if you want to get pixel data from a rendering like this it's best to render into a CGContext backed by a bitmap where you control the pixel buffer. You can inspect the pixel data in your buffer after rendering. Here's an example where I create a bitmap context and render a layer and mask into that bitmap context. Since I supply my own buffer for the pixel data, I can then go poking around in the buffer to get RGBA values after rendering.
const CGRect imageViewBounds = self.imageView.bounds;
const size_t imageWidth = CGRectGetWidth(imageViewBounds);
const size_t imageHeight = CGRectGetHeight(imageViewBounds);
const size_t bytesPerPixel = 4;
// Allocate your own buffer for the bitmap data. You can pass NULL to
// CGBitmapContextCreate and then Quartz will allocate the memory for you, but
// you won't have a pointer to the resulting pixel data you want to inspect!
unsigned char *bitmapData = (unsigned char *)malloc(imageWidth * imageHeight * bytesPerPixel);
CGContextRef context = CGBitmapContextCreate(bitmapData, imageWidth, imageHeight, 8, bytesPerPixel * imageWidth, rgbColorSpace, kCGImageAlphaPremultipliedLast);
// Render the image into the bitmap context.
[self.imageView.layer renderInContext:context];
// Set the blend mode to destination in, pixels become "destination * source alpha"
CGContextSetBlendMode(context, kCGBlendModeDestinationIn);
// Render the mask into the bitmap context.
[self.imageView.layer.mask renderInContext:context];
// Whatever you like here, I just chose the middle of the image.
const size_t x = imageWidth / 2;
const size_t y = imageHeight / 2;
const size_t pixelIndex = y * imageWidth + x;
const unsigned char red = bitmapData[pixelIndex];
const unsigned char green = bitmapData[pixelIndex + 1];
const unsigned char blue = bitmapData[pixelIndex + 2];
const unsigned char alpha = bitmapData[pixelIndex + 3];
NSLog(#"rgba: {%u, %u, %u, %u}", red, green, blue, alpha);

Related

no suitable conversion function from "Magick::Color" to "MagickCore::Quantum" exists

I know already why it gives that error code-wise.
Problem is, I started using the library itself today and following the tutorial I found this.
I installed the "ImageMagick-7.0.9-1-Q16-x64-dll" version of the library, and tried to find the shortest code that gave that error, which is:
#include <Magick++.h>
int main(){
Magick::Quantum result = Magick::Color("black");
}
Given the tutorial(following one), a method that converts from Magick::Color to Magic::Quantum should exist
// Example of using an image pixel cache
Image my_image("640x480", "white"); // we'll use the 'my_image' object in this example
my_image.modifyImage(); // Ensure that there is only one reference to
// underlying image; if this is not done, then the
// image pixels *may* remain unmodified. [???]
Pixels my_pixel_cache(my_image); // allocate an image pixel cache associated with my_image
Quantum* pixels; // 'pixels' is a pointer to a Quantum array
// define the view area that will be accessed via the image pixel cache
int start_x = 10, start_y = 20, size_x = 200, size_y = 100;
// return a pointer to the pixels of the defined pixel cache
pixels = my_pixel_cache.get(start_x, start_y, size_x, size_y);
// set the color of the first pixel from the pixel cache to black (x=10, y=20 on my_image)
*pixels = Color("black");
// set to green the pixel 200 from the pixel cache:
// this pixel is located at x=0, y=1 in the pixel cache (x=10, y=21 on my_image)
*(pixels+200) = Color("green");
// now that the operations on my_pixel_cache have been finalized
// ensure that the pixel cache is transferred back to my_image
my_pixel_cache.sync();
which gives that error ( no suitable conversion function from "Magick::Color" to "MagickCore::Quantum" exists ) at the following lines
*pixels = Color("black");
*(pixels+200) = Color("green");
I believe you are confusing a data-type with a structure. The pixels represents a continuous list of Quantum parts.
Assuming that we're working with RGB colorspace. You would need to set each color part.
Color black("black");
*(pixels + 0) = black.quantumRed();
*(pixels + 1) = black.quantumGreen();
*(pixels + 2) = black.quantumBlue();
To set the 200th pixel, you would need to multiply the offset by the parts-per-pixel count.
Color green("green");
int offset = 199 * 3; // First pixel starts at 0, and 3 parts (Red, Green, Blue)
*(pixels + offset + 0) = green.quantumRed();
*(pixels + offset + 1) = green.quantumGreen();
*(pixels + offset + 2) = green.quantumBlue();

Uploading raw IOS camera data to a texture

We are using AVCaptureDevice on IOS to scan QR codes. We pass the output of the camera using AVCaptureMetadataOutput to the code to recognize the QR code, and currently we also display the camera as a separate view over our Open GL view. However we now want other graphics to appear over the camera preview, so we would like to be able to get the camera data loaded onto one of our Open GL textures.
So, is there a way to get the raw RGB data from the camera
This is the code (below) we're using to initialise the capture device and views.
How could we modify this to access the RGB data so we can load it onto one of our GL textures ? We're using C++/Objective C
Thanks
Shaun Southern
self.captureSession = [[AVCaptureSession alloc] init];
NSError *error;
// Set camera capture device to default and the media type to video.
AVCaptureDevice *captureDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
// Set video capture input: If there a problem initialising the camera, it will give am error.
AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:captureDevice error:&error];
if (!input)
{
NSLog(#"Error Getting Camera Input");
return;
}
// Adding input souce for capture session. i.e., Camera
[self.captureSession addInput:input];
AVCaptureMetadataOutput *captureMetadataOutput = [[AVCaptureMetadataOutput alloc] init];
// Set output to capture session. Initalising an output object we will use later.
[self.captureSession addOutput:captureMetadataOutput];
// Create a new queue and set delegate for metadata objects scanned.
dispatch_queue_t dispatchQueue;
dispatchQueue = dispatch_queue_create("scanQueue", NULL);
[captureMetadataOutput setMetadataObjectsDelegate:self queue:dispatchQueue];
// Delegate should implement captureOutput:didOutputMetadataObjects:fromConnection: to get callbacks on detected metadata.
[captureMetadataOutput setMetadataObjectTypes:[captureMetadataOutput availableMetadataObjectTypes]];
// Layer that will display what the camera is capturing.
self.captureLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:self.captureSession];
[self.captureLayer setVideoGravity:AVLayerVideoGravityResizeAspectFill];
gCameraPreviewView= [[[UIView alloc] initWithFrame:CGRectMake(gCamX1, gCamY1, gCamX2-gCamX1, gCamY2-gCamY1)] retain];
[self.captureLayer setFrame:gCameraPreviewView.layer.bounds];
[gCameraPreviewView.layer addSublayer:self.captureLayer];
UIViewController * lVC = [[[UIApplication sharedApplication] keyWindow] rootViewController];
[lVC.view addSubview:gCameraPreviewView];
You don't need to directly access rgb camera frames to make it as a texture because IOS supports a texture cache which is faster than you.
- (void) writeSampleBuffer:(CMSampleBufferRef)sampleBuffer ofType:(NSString *)mediaType pixel:(CVImageBufferRef)cameraFrame time:(CMTime)frameTime;
in the callback method, you can generate texture using those parameter and functions below
CVOpenGLESTextureCacheCreate(...)
CVOpenGLESTextureCacheCreateTextureFromImage(...)
In the end we converted from the CMSampleBuffer to the raw data (so we could upload to a GL texture) like this. It takes a little time but was fast enough for our purposes.
If there's any improvements, I'd be glad to know :)
shaun
- (void)captureOutput:(AVCaptureFileOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection;
{
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CIImage *ciImage = [CIImage imageWithCVPixelBuffer:imageBuffer];
if(!self.context)
{
self.context = [CIContext contextWithOptions:nil]; //only create this once
}
int xsize= CVPixelBufferGetWidth(imageBuffer);
int ysize= CVPixelBufferGetHeight(imageBuffer);
CGImageRef videoImage = [self.context createCGImage:ciImage fromRect:CGRectMake(0, 0, xsize, ysize)];
UIImage *image = [[UIImage alloc] initWithCGImage:videoImage];
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
if(!colorSpace)
{
return ;
}
size_t bitsPerPixel = 32;
size_t bitsPerComponent = 8;
size_t bytesPerPixel = bitsPerPixel / bitsPerComponent;
size_t bytesPerRow = xsize * bytesPerPixel;
size_t bufferLength = bytesPerRow * ysize;
uint32_t * tempbitmapData = (uint32_t *)malloc(bufferLength);
if(!tempbitmapData)
{
CGColorSpaceRelease(colorSpace);
return ;
}
CGContextRef cgcontext = CGBitmapContextCreate(tempbitmapData, xsize, ysize, bitsPerComponent, bytesPerRow, colorSpace, kCGImageAlphaPremultipliedLast);
if(!cgcontext)
{
free(tempbitmapData);
return;
}
CGColorSpaceRelease(colorSpace);
CGRect rect = CGRectMake(0, 0, xsize, ysize);
CGContextDrawImage(cgcontext, rect, image.CGImage);
unsigned char *bitmapData = (unsigned char *)CGBitmapContextGetData(cgcontext); // Get a pointer to the data
CGContextRelease(cgcontext);
[image release];
CallbackWithData((unsigned int *)bitmapData,xsize,ysize); //send data
free(bitmapData);
CGImageRelease(videoImage);
}

How should I correct this pointer issue, it's giving me an error

Suppose rgbapixel is a class of pixels in rgbapixel.h file, so it has colors like green, blue, red as public members. PNG is a class of images in png.h file, so it has image width and height as private members, then it has two public functions that return width and height.
in my main.cpp, here is the code;
// sets up the output image
PNG * setupOutput(int w, int h)
{
PNG * image = new PNG(w, h);
return image;
}
void sketchify()
{
// Load in.png
PNG * original = new PNG;
original->readFromFile("in.png");
int width = original->width();
int height = original->height();
// Create out.png
// PNG * output; // i change this
PNG * output = new PNG;
setupOutput(width, height);
// Loud our favorite color to color the outline
RGBAPixel * myPixel = myFavoriteColor(192);
// Go over the whole image, and if a pixel differs from that to its upper
// left, color it my favorite color in the output
for (int y = 1; y < height; y++)
{
for (int x = 1; x < width; x++)
{
// Calculate the pixel difference
RGBAPixel * prev = (*original)(x-1, y-1); // previous top lfet
RGBAPixel * curr = (*original)(x , y ); // current
// subtracting to see diffrence between pixels
int diff = abs(curr->red - prev->red ) +
abs(curr->green - prev->green) +
abs(curr->blue - prev->blue );
// If the pixel is an edge pixel,
// color the output pixel with my favorite color
RGBAPixel * currOutPixel = (*output)(x,y);
if (diff > 100)
currOutPixel = myPixel; // something wrong
}
}
// Save the output file
output->writeToFile("out.png");
// Clean up memory
delete myPixel;
delete output;
delete original;
When I execute the code, I get errors like;
[EasyPNG]: Warning: attempted to access non-existent pixel (407, 306);
Truncating request to fit in the range [0,0] x [0,0].
[EasyPNG]: Warning: attempted to access non-existent pixel (408, 306);
Truncating request to fit in the range [0,0] x [0,0].
Where I write "something is wrong", I have been told that there is an error there. I don't see it. 'mypixel' and currout where both declared successfully as pointers, so I don't see how that statement would be wrong. If I try to change it, I get compilation errors. help
The line setupOutput(width, height); does not do anything useful. It create a new PNG on the heap but the returned value is discarded. You end with two problems:
output does not have its width and height properly set.
There is a memory leak.
You can fix that using one of two approaches.
Assign the return value of setupOutput(width, height) to output.
Replace the lines:
PNG * output = new PNG;
setupOutput(width, height);
with
PNG * output = setupOutput(width, height);
Don't use setupOutput(width, height); at all. Create a PNG inline.
Replace the lines:
PNG * output = new PNG;
setupOutput(width, height);
with
PNG * output = new PNG(width, height);
No guarantees that these changes will fix all your problems.
Update, in response to comment by OP
The line
currOutPixel = myPixel; // something wrong
doesn't do anything useful either. It just overrides where the local variable currOutPixel points. Assuming that it is OK to assign objects of type RGBAPixel, you need:
*currOutPixel = *myPixel;

Getting the SDL_Color of a single pixel in a SDL_Texture

I am having some problems finding a solution on how to retrieve a specific color of a pixel on a SDL_Texture...
To be bit more specific: I am trying to calculate the average amount of color used in a given texture. Later on I want to devide for example the number of red pixels by the total amount of pixels. For this task I will need a method, which will get me each pixel color...
I tried to search for some functions, but unfortunately I wasnt able to figure it out..
I saw methods like SDL_RenderReadPixels and SDL_GetPixelFormatName, but none of those helped me out...
Do you have a solution for me?
To access an SDL_Texture's pixels, you must create a blank texture using SDL_CreateTexture() and pass in SDL_TEXTUREACCESS_STREAMING for the access parameter, then copy the pixels of a surface into it. Once that's done, you can use the SDL_LockTexture() function to retrieve a pointer to the pixel data which can then be accessed and modified. To save your changes, you'd call SDL_UnlockTexture(). Try something like this:
SDL_Texture *t;
int main()
{
// Init SDL
SDL_Surface * img = IMG_Load("path/to/file");
SDL_CreateTexture(renderer, SDL_PIXELFORMAT_RGBA8888, SDL_TEXTUREACCESS_STREAMING, img->w, img->h);
void * pixels;
SDL_LockTexture(t, &img->clip_rect, &pixels, img->pitch);
memcpy(pixels, img->pixels, img->w * img->h);
Uint32 * upixels = (Uint32 *) pixels;
// get or modify pixels
SDL_UnlockTexture(t);
return 0;
}
Uint32 get_pixel_at(Uint32 * pixels, int x, int y, int w)
{
return pixels[y * w + x];
}
You can get the colors from a pixel like this:
Uint32 pixel = get_pixel_at(pixels, x, y, img->w);
Uint8 * colors = (Uint8 *) pixel;
// colors[0] is red, 1 is green, 2 is blue, 3 is alpha (assuming you've set the blend mode on the texture to SDL_BLENDMODE_BLEND
If you want more information, then check out these SDL 2.0 tutorials: http://lazyfoo.net/tutorials/SDL/index.php. Tutorial 40 deals specifically with this problem.
Let me know if you have any questions or something is unclear.
Good luck!

Kinect SDK: align depth and color frames

I'm working with Kinect sensor and I'm trying to align depth and color frames so that I can save them as images which "fit" into each other. I've spent a lot of time going through msdn forums and modest documentation of Kinect SDK and I'm getting absolutely nowhere.
Based on this answer: Kinect: Converting from RGB Coordinates to Depth Coordinates
I have the following function, where depthData and colorData are obtained from NUI_LOCKED_RECT.pBits and mappedData is the output containing new color frame, mapped to depth coordinates:
bool mapColorFrameToDepthFrame(unsigned char *depthData, unsigned char* colorData, unsigned char* mappedData)
{
INuiCoordinateMapper* coordMapper;
// Get coordinate mapper
m_pSensor->NuiGetCoordinateMapper(&coordMapper);
NUI_DEPTH_IMAGE_POINT* depthPoints = new NUI_DEPTH_IMAGE_POINT[640 * 480];
HRESULT result = coordMapper->MapColorFrameToDepthFrame(NUI_IMAGE_TYPE_COLOR, NUI_IMAGE_RESOLUTION_640x480, NUI_IMAGE_RESOLUTION_640x480, 640 * 480, reinterpret_cast<NUI_DEPTH_IMAGE_PIXEL*>(depthData), 640 * 480, depthPoints);
if (FAILED(result))
{
return false;
}
int pos = 0;
int* colorRun = reinterpret_cast<int*>(colorData);
int* mappedRun = reinterpret_cast<int*>(mappedData);
// For each pixel of new color frame
for (int i = 0; i < 640 * 480; ++i)
{
// Find the corresponding pixel in original color frame from depthPoints
pos = (depthPoints[i].y * 640) + depthPoints[i].x;
// Set pixel value if it's within frame boundaries
if (pos < 640 * 480)
{
mappedRun[i] = colorRun[pos];
}
}
return true;
}
All I get when running this code is an unchanged color frame with removed (white) all pixels where depthFrame had no information.
With the OpenNI framework there an option call registration.
IMAGE_REGISTRATION_DEPTH_TO_IMAGE – The depth image is transformed to have the same apparent vantage point as the RGB image.
OpenNI 2.0 and Nite 2.0 works very well to capture Kinect information and there a lot of tutorials.
You can have a look to this :
Kinect with OpenNI
And OpenNi have a example in SimplerViewer that merge Depth and Color maybe you can just look on that and try it.
This might not be the quick answer you're hoping for, but this transformation is done successfully within the ofxKinectNui addon for openFrameworks (see here).
It looks like ofxKinectNui delegates to the GetColorPixelCoordinatesFromDepthPixel function defined here.
I think the problem is that you're calling MapColorFrameToDepthFrame, when you should actually call MapDepthFrameToColorFrame.
The smoking gun is this line of code:
mappedRun[i] = colorRun[pos];
Reading from pos and writing to i is backwards, since pos = depthPoints[i] represents the depth coordinates corresponding to the color coordinates at i. You actually want to iterate over writing all depth coordinates and read from the input color image at the corresponding color coordinates.
I think that in your code there are different not correct lines.
First of all, which kind of depth map are you passing to your function?
Depth data is storred using two bytes for each value, that means that the correct type of the pointer that you should use for your depth data
is unsigned short.
Second point is that from what i have understood, you want to map depth frame to color frame, so the correct function that you have
to call from kinect sdk is MapDepthFrameToColorFrame instead of MapColorFrameToDepthFrame.
Finally the function will return a map of point where for each depth data at position [i], you have the position x and position y where that point should
be mapped.
To do this you don't need for colorData pointer.
So your function should be modified as follow:
/** Method used to build a depth map aligned to color frame
#param [in] depthData : pointer to your data;
#param [out] mappedData : pointer to your aligned depth map;
#return true if is all ok : false whene something wrong
*/
bool DeviceManager::mapColorFrameToDepthFrame(unsigned short *depthData, unsigned short* mappedData){
INuiCoordinateMapper* coordMapper;
NUI_COLOR_IMAGE_POINT* colorPoints = new NUI_COLOR_IMAGE_POINT[640 * 480]; //color points
NUI_DEPTH_IMAGE_PIXEL* depthPoints = new NUI_DEPTH_IMAGE_PIXEL[640 * 480]; // depth pixel
/** BE sURE THAT YOU ARE WORKING WITH THE RIGHT HEIGHT AND WIDTH*/
unsigned long refWidth = 0;
unsigned long refHeight = 0;
NuiImageResolutionToSize( NUI_IMAGE_RESOLUTION_640x480, refWidth, refHeight );
int width = static_cast<int>( refWidth ); //get the image width in a right way
int height = static_cast<int>( refHeight ); //get the image height in a right way
m_pSensor>NuiGetCoordinateMapper(&coordMapper); // get the coord mapper
//Map your frame;
HRESULT result = coordMapper->MapDepthFrameToColorFrame( NUI_IMAGE_RESOLUTION_640x480, width * height, depthPoints, NUI_IMAGE_TYPE_COLOR, NUI_IMAGE_RESOLUTION_640x480, width * height, colorPoints );
if (FAILED(result))
return false;
// apply map in terms of x and y (image coordinates);
for (int i = 0; i < width * height; i++)
if (colorPoints[i].x >0 && colorPoints[i].x < width && colorPoints[i].y>0 && colorPoints[i].y < height)
*(mappedData + colorPoints[i].x + colorPoints[i].y*width) = *(depthData + i );
// free your memory!!!
delete colorPoints;
delete depthPoints;
return true;
}
Make sure that your mappedData has been initialized in correct way, for example as follow.
mappedData = (USHORT*)calloc(width*height, sizeof(ushort));
Remember that kinect sdk does not provide an accurate align function between color and depth data.
If you want an accurate alignment between two images you should use a calibration model.
In that case i suggest you to use the Kinect Calibration Toolbox, based on Heikkilä calibration model.
You can find it in the follow link:
http://www.ee.oulu.fi/~dherrera/kinect/.
First of all, you must calibrate your device.
That means, you should calibrate the RGB and the IR sensor and then find the transformation between RGB and IR.
Once you know this information, you can apply the function:
RGBPoint = RotationMatrix * DepthPoint + TranslationVector
Check OpenCV or ROS projects for further details on it.
Extrinsic Calibration
Intrinsic Calibration