Uploading raw IOS camera data to a texture - c++

Question

We are using AVCaptureDevice on IOS to scan QR codes. We pass the output of the camera using AVCaptureMetadataOutput to the code to recognize the QR code, and currently we also display the camera as a separate view over our Open GL view. However we now want other graphics to appear over the camera preview, so we would like to be able to get the camera data loaded onto one of our Open GL textures.
So, is there a way to get the raw RGB data from the camera
This is the code (below) we're using to initialise the capture device and views.
How could we modify this to access the RGB data so we can load it onto one of our GL textures ? We're using C++/Objective C
Thanks
Shaun Southern
self.captureSession = [[AVCaptureSession alloc] init];
NSError *error;
// Set camera capture device to default and the media type to video.
AVCaptureDevice *captureDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
// Set video capture input: If there a problem initialising the camera, it will give am error.
AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:captureDevice error:&error];
if (!input)
{
NSLog(#"Error Getting Camera Input");
return;
}
// Adding input souce for capture session. i.e., Camera
[self.captureSession addInput:input];
AVCaptureMetadataOutput *captureMetadataOutput = [[AVCaptureMetadataOutput alloc] init];
// Set output to capture session. Initalising an output object we will use later.
[self.captureSession addOutput:captureMetadataOutput];
// Create a new queue and set delegate for metadata objects scanned.
dispatch_queue_t dispatchQueue;
dispatchQueue = dispatch_queue_create("scanQueue", NULL);
[captureMetadataOutput setMetadataObjectsDelegate:self queue:dispatchQueue];
// Delegate should implement captureOutput:didOutputMetadataObjects:fromConnection: to get callbacks on detected metadata.
[captureMetadataOutput setMetadataObjectTypes:[captureMetadataOutput availableMetadataObjectTypes]];
// Layer that will display what the camera is capturing.
self.captureLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:self.captureSession];
[self.captureLayer setVideoGravity:AVLayerVideoGravityResizeAspectFill];
gCameraPreviewView= [[[UIView alloc] initWithFrame:CGRectMake(gCamX1, gCamY1, gCamX2-gCamX1, gCamY2-gCamY1)] retain];
[self.captureLayer setFrame:gCameraPreviewView.layer.bounds];
[gCameraPreviewView.layer addSublayer:self.captureLayer];
UIViewController * lVC = [[[UIApplication sharedApplication] keyWindow] rootViewController];
[lVC.view addSubview:gCameraPreviewView];

Answer 1

You don't need to directly access rgb camera frames to make it as a texture because IOS supports a texture cache which is faster than you.
- (void) writeSampleBuffer:(CMSampleBufferRef)sampleBuffer ofType:(NSString *)mediaType pixel:(CVImageBufferRef)cameraFrame time:(CMTime)frameTime;
in the callback method, you can generate texture using those parameter and functions below
CVOpenGLESTextureCacheCreate(...)
CVOpenGLESTextureCacheCreateTextureFromImage(...)

Answer 2

In the end we converted from the CMSampleBuffer to the raw data (so we could upload to a GL texture) like this. It takes a little time but was fast enough for our purposes.
If there's any improvements, I'd be glad to know :)
shaun
- (void)captureOutput:(AVCaptureFileOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection;
{
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CIImage *ciImage = [CIImage imageWithCVPixelBuffer:imageBuffer];
if(!self.context)
{
self.context = [CIContext contextWithOptions:nil]; //only create this once
}
int xsize= CVPixelBufferGetWidth(imageBuffer);
int ysize= CVPixelBufferGetHeight(imageBuffer);
CGImageRef videoImage = [self.context createCGImage:ciImage fromRect:CGRectMake(0, 0, xsize, ysize)];
UIImage *image = [[UIImage alloc] initWithCGImage:videoImage];
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
if(!colorSpace)
{
return ;
}
size_t bitsPerPixel = 32;
size_t bitsPerComponent = 8;
size_t bytesPerPixel = bitsPerPixel / bitsPerComponent;
size_t bytesPerRow = xsize * bytesPerPixel;
size_t bufferLength = bytesPerRow * ysize;
uint32_t * tempbitmapData = (uint32_t *)malloc(bufferLength);
if(!tempbitmapData)
{
CGColorSpaceRelease(colorSpace);
return ;
}
CGContextRef cgcontext = CGBitmapContextCreate(tempbitmapData, xsize, ysize, bitsPerComponent, bytesPerRow, colorSpace, kCGImageAlphaPremultipliedLast);
if(!cgcontext)
{
free(tempbitmapData);
return;
}
CGColorSpaceRelease(colorSpace);
CGRect rect = CGRectMake(0, 0, xsize, ysize);
CGContextDrawImage(cgcontext, rect, image.CGImage);
unsigned char *bitmapData = (unsigned char *)CGBitmapContextGetData(cgcontext); // Get a pointer to the data
CGContextRelease(cgcontext);
[image release];
CallbackWithData((unsigned int *)bitmapData,xsize,ysize); //send data
free(bitmapData);
CGImageRelease(videoImage);
}