Programmatically grab screenshots in OSX - c++

Question

I am going to port some screenshot grabbing code (C++) for linux to osx. The current solution run graphical applications in xvfb and then use xlib to grab screenshots from the display. (That will also support if we are running without xvfb).
So as I understood osx is moving away from X11 so my question is what to use besides xlib to implement it now ? I have found Quartz Display Services. Is that what makes sense to use now ? Will that work with xvfb ?

Answer 1

Yes, you will be able to call functions like CGDisplayCreateImage (documentation linked for you) by linking the Application Services framework to your C++ tool.

Answer 2

I have written an example for capturing the pc display screen and convert to opencv Mat.
#include <iostream>
#include <opencv2/opencv.hpp>
#include <unistd.h>
#include <stdio.h>
#include <ApplicationServices/ApplicationServices.h>
using namespace std;
using namespace cv;
int main (int argc, char * const argv[])
{
size_t width = CGDisplayPixelsWide(CGMainDisplayID());
size_t height = CGDisplayPixelsHigh(CGMainDisplayID());
Mat im(cv::Size(width,height), CV_8UC4);
Mat bgrim(cv::Size(width,height), CV_8UC3);
Mat resizedim(cv::Size(width,height), CV_8UC3);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef contextRef = CGBitmapContextCreate(
im.data, im.cols, im.rows,
8, im.step[0],
colorSpace, kCGImageAlphaPremultipliedLast|kCGBitmapByteOrderDefault);
while (true)
{
CGImageRef imageRef = CGDisplayCreateImage(CGMainDisplayID());
CGContextDrawImage(contextRef,
CGRectMake(0, 0, width, height),
imageRef);
cvtColor(im, bgrim, CV_RGBA2BGR);
resize(bgrim, resizedim,cv::Size(),0.5,0.5);
imshow("test", resizedim);
cvWaitKey(10);
CGImageRelease(imageRef);
}
// CGContextRelease(contextRef);
// CGColorSpaceRelease(colorSpace);
return 0;
}
and then, the result is here.
I had expected my current display would be captured, but only the back wallpaper was captured actually.
What the CGMainDisplayID() refers would be a hint to this problem.
Anyway, I hope this may approach your goal a bit.

Answer 3

void captureScreen(){
CGImageRef image_ref = CGDisplayCreateImage(CGMainDisplayID());
CGDataProviderRef provider = CGImageGetDataProvider(image_ref);
CFDataRef dataref = CGDataProviderCopyData(provider);
size_t width, height; width = CGImageGetWidth(image_ref);
height = CGImageGetHeight(image_ref);
size_t bpp = CGImageGetBitsPerPixel(image_ref) / 8;
uint8 *pixels = malloc(width * height * bpp);
memcpy(pixels, CFDataGetBytePtr(dataref), width * height * bpp);
CFRelease(dataref);
CGImageRelease(image_ref);
FILE *stream = fopen("/Users/username/Desktop/screencap.raw", "w+");
fwrite(pixels, bpp, width * height, stream);
fclose(stream);
free(pixels);
}
or in C#:
// https://stackoverflow.com/questions/1537587/capture-screen-image-in-c-on-osx
// https://github.com/Acollie/C-Screenshot-OSX/blob/master/C%2B%2B-screenshot/C%2B%2B-screenshot/main.cpp
// https://github.com/ScreenshotMonitor/ScreenshotCapture/blob/master/src/Pranas.ScreenshotCapture/ScreenshotCapture.cs
// https://screenshotmonitor.com/blog/capturing-screenshots-in-net-and-mono/
namespace rtaStreamingServer
{
// https://github.com/xamarin/xamarin-macios
// https://qiita.com/shimshimkaz/items/18bcf4767143ea5897c7
public static class OSxScreenshot
{
private const string LIBCOREGRAPHICS = "/System/Library/Frameworks/CoreGraphics.framework/CoreGraphics";
[System.Runtime.InteropServices.DllImport(LIBCOREGRAPHICS)]
private static extern System.IntPtr CGDisplayCreateImage(System.UInt32 displayId);
[System.Runtime.InteropServices.DllImport(LIBCOREGRAPHICS)]
private static extern void CFRelease(System.IntPtr handle);
public static void TestCapture()
{
Foundation.NSNumber mainScreen = (Foundation.NSNumber)AppKit.NSScreen.MainScreen.DeviceDescription["NSScreenNumber"];
using (CoreGraphics.CGImage cgImage = CreateImage(mainScreen.UInt32Value))
{
// https://stackoverflow.com/questions/17334786/get-pixel-from-the-screen-screenshot-in-max-osx/17343305#17343305
// Get byte-array from CGImage
// https://gist.github.com/zhangao0086/5fafb1e1c0b5d629eb76
AppKit.NSBitmapImageRep bitmapRep = new AppKit.NSBitmapImageRep(cgImage);
// var imageData = bitmapRep.representationUsingType(NSBitmapImageFileType.NSPNGFileType, properties: [:])
Foundation.NSData imageData = bitmapRep.RepresentationUsingTypeProperties(AppKit.NSBitmapImageFileType.Png);
long len = imageData.Length;
byte[] bytes = new byte[len];
System.Runtime.InteropServices.GCHandle pinnedArray = System.Runtime.InteropServices.GCHandle.Alloc(bytes, System.Runtime.InteropServices.GCHandleType.Pinned);
System.IntPtr pointer = pinnedArray.AddrOfPinnedObject();
// Do your stuff...
imageData.GetBytes(pointer, new System.IntPtr(len));
pinnedArray.Free();
using (AppKit.NSImage nsImage = new AppKit.NSImage(cgImage, new System.Drawing.SizeF(cgImage.Width, cgImage.Height)))
{
// ImageView.Image = nsImage;
// And now ? How to get the image bytes ?
// https://theconfuzedsourcecode.wordpress.com/2016/02/24/convert-android-bitmap-image-and-ios-uiimage-to-byte-array-in-xamarin/
// https://stackoverflow.com/questions/5645157/nsimage-from-byte-array
// https://stackoverflow.com/questions/53060723/nsimage-source-from-byte-array-cocoa-app-xamarin-c-sharp
// https://gist.github.com/zhangao0086/5fafb1e1c0b5d629eb76
// https://www.quora.com/What-is-a-way-to-convert-UIImage-to-a-byte-array-in-Swift?share=1
// https://stackoverflow.com/questions/17112314/converting-uiimage-to-byte-array
} // End Using nsImage
} // End Using cgImage
} // End Sub TestCapture
public static CoreGraphics.CGImage CreateImage(System.UInt32 displayId)
{
System.IntPtr handle = System.IntPtr.Zero;
try
{
handle = CGDisplayCreateImage(displayId);
return new CoreGraphics.CGImage(handle);
}
finally
{
if (handle != System.IntPtr.Zero)
{
CFRelease(handle);
}
}
} // End Sub CreateImage
} // End Class OSxScreenshot
} // End Namespace rtaStreamingServer