Apparently, Microsoft has changed the way clipping works with Windows update 1809, released in late 2018. Before that update, GetClipBox() returned the full client rectangle of a window, even when it was (partially) offscreen.
After the update, the same function returns a clipped rectangle, only containing the parts that are still onscreen. This leads to the Device Context contents not being updated for the offscreen area, which prevents me from taking screenshots from these windows.
The question is: can I somehow manipulate the clipping region?
I have researched a bit and it seems that the final clipping region is influenced by the window region, the update rectangle, and the system region - as far as I understand the "global clipping region". I've checked the window region with GetWindowRgn() and GetRgnBox(), both return the same values for Windows 1809 and older versions. GetUpdateRect() also returns the full client rectangle, so that cannot be the issue either. I've also tried to hook the BeginPaint() method and see if changing the PAINTSTRUCT.rcPaint does anything, without success.
So what I am left with is trying to adjust the system region, or sometimes called the visible region. However, I have no idea if and how that is possible. MSDN suggests that it's not, but I thought maybe someone does have an idea for a solution!?
EDIT: To make this more clear, I don't think the clipping is done by the application itself, because offscreen screenshots of the same application version work prior to Windows 1809 and don't work with the updated Windows version. Instead, Windows itself seems to clip any offscreen surfaces.
EDIT2: Here's a minimal working code example for taking the screenshot.
// Get the client size.
RECT crect;
GetClientRect(hwnd, &crect);
int width = crect.right - crect.left;
int height = crect.bottom - crect.top;
// Create DC and Bitmap.
HDC windowDC = GetDC(hwnd);
HDC memoryDC = CreateCompatibleDC(windowDC);
BITMAPINFO bitmapInfo;
ZeroMemory(&bitmapInfo, sizeof(BITMAPINFO));
bitmapInfo.bmiHeader.biSize = sizeof(BITMAPINFOHEADER);
bitmapInfo.bmiHeader.biWidth = width;
bitmapInfo.bmiHeader.biHeight = -height;
bitmapInfo.bmiHeader.biPlanes = 1;
bitmapInfo.bmiHeader.biBitCount = 32;
bitmapInfo.bmiHeader.biCompression = BI_RGB;
bitmapInfo.bmiHeader.biSizeImage = width * height * 4;
char* pixels;
HBITMAP bitmap = CreateDIBSection(windowDC, &bitmapInfo, DIB_RGB_COLORS, (void**)&pixels, 0, 0);
HGDIOBJ previousObject = SelectObject(memoryDC, bitmap);
// Take the screenshot. Neither BitBlt nor PrintWindow work.
BitBlt(memoryDC, 0, 0, width, height, windowDC, 0, 0, SRCCOPY);
// ..or..
// PrintWindow(hwnd, memoryDC, PW_CLIENTONLY);
// Save the image.
BITMAPFILEHEADER bitmapFileHeader;
bitmapFileHeader.bfType = 0x4D42;
bitmapFileHeader.bfOffBits = sizeof(BITMAPFILEHEADER) + sizeof(BITMAPINFOHEADER);
std::fstream hFile("./screenshot.bmp", std::ios::out | std::ios::binary);
if(hFile.is_open())
{
hFile.write((char*)&bitmapFileHeader, sizeof(bitmapFileHeader));
hFile.write((char*)&bitmapInfo.bmiHeader, sizeof(bitmapInfo.bmiHeader));
hFile.write(pixels, (((32 * width + 31) & ~31) / 8) * height);
hFile.close();
}
// Free Resources
ReleaseDC(hwnd, windowDC);
SelectObject(memoryDC, previousObject);
DeleteDC(memoryDC);
DeleteObject(bitmap);
You can download a compiled executable from Google Drive here. Usage is Screenshot.exe <HWND>, where HWND is the hex address of the window handle as it is shown in Spy++ for example. It will save a screenshot of the target window in the working directory as screenshot.bmp (make sure you're allowed to write to the directory). The screenshot will work for almost all windows (even if they are hidden behind other windows), but as soon as you partially move the window offscreen, the screenshot will continue to show the old window contents for the offscreen part of the window (resize it while it's offscreen for example, to see the effect). This only happens on Windows 1809, it still shows up-to-date contents on earlier Windows versions.
EDIT3: I did some more research on this. Regarding the AdobeAir application for which the WS_EX_LAYERED style did not work: I found that it uses BitBlt internally do render the back buffer to the window dc. The rendering steps are:
GetDC(hwnd) on the window to obtain hdcWin
CreateCompatibleDC(hdcWin) to create a hdcMem
Call SelectObject(hdcMem, bmp) to select an HBITMAP into hdcMem
BitBlt from hdcMem to hdcWin.
During the BitBlt call, the hdcMem contains valid pixel data even in the offscreen regions, but that data is never copied to the hdcWin.
I looked at the system regions during the BitBlt call. For hdcMem the system region is a NULLREGION, but for the hdcWin the region is always clipped at the screen edges. I also tried to adjust the system region, by replacing all calls to GetDC with GetDCEx(hwnd, hrgn, DCX_CACHE | DCX_INTERSECTRGN) (as mentioned in this article), but that doesn't work and doesn't seem to provide options for extending the region. I really think the secret to solving the problem lies in manipulating the system region for the window dc, but I have no idea how to do that.
If found that the CreateDC function takes a pointer to a DEVMODE struct as the last argument (msdn). That in turn has fields dmPelsWidth, dmPelsHeight and dmPosition. I believe that these make up the system region and maybe if I could manipulate them, the DC would no longer get clipped, but I wasn't able to hook the CreateDC function, yet.
If you have any new ideas based on my new insights, please share them. I'd appreciate any help!
If we take ReactOS as an example, the clipping region is at dc->dclevel.prgnClip and the system region is at dc->prgnVis. When you call BeginPaint on a window, it calls NtUserBeginPaint stub which traps to its kernel counterpart through the win32k SSDT, which calls IntBeginPaint, which passes the window's update region (Window->hrgnUpdate) to UserGetDCEx, which copies this to Dce->hrgnClip and calls DceUpdateVisRgn, which then gets the visible region by calling DceGetVisRgn which calculates the visible region using VIS_ComputeVisibleRegion, which develops a complex region by traversing all child windows, all parent windows and all siblings at each level (a top level window has a parent as the desktop (((PCLIENTINFO)(NtCurrentTeb()->Win32ClientInfo))->pDeskInfo->spwnd) and all top level windows are siblings; the desktop's parent is NULL and removing the parts they cover up – this does not appear to perform any special handling for the desktop window when it gets to it like clipping to the client area, and is treated like any other window in the z order, where only what it is covering is removed). DceGetVisRgn then combines this returned visible region and combines it wil the clipping region Dce->hrgnClip and combines them into RgnVisible using IntGdiCombineRgn(RgnVisible, RgnVisible, RgnClip, RGN_AND), which is then copied into dc->prgnVis using GdiSelectVisRgn(Dce->hDC, RgnVisible). DC is the device context and DCE is the device context entry for the DC in the DC cache. Therefore, the system region of the DC is now the intersection of the visible region and the update region of the window. IntBeginPaint also calls GdiGetClipBox(Ps->hdc, &Ps->rcPaint), which calls REGION_GetRgnBox(pdc->prgnVis, prc) to copy the bound of the region pdc->prgnVis (pdc->prgnVis->rdh.rcBound) to Ps->rcPaint and then GdiGetClipBox calls IntDPtoLP(pdc, (LPPOINT)prc, 2) to convert the bound from physical coordinates to logical coordinates, which DPI-unaware apps use. The paintstruct now contains the smallest logical rectangle that contains the complex intersection of the update region and the visible region.
GetClipRgn calls NtGdiGetRandomRgn, which returns pdc->dclevel.prgnClip when called with CLIPRGN, which is application defined using SetClipRgn
An application-defined clipping region is a clipping region identified by the SelectClipRgn function. It is not a clipping region created when the application calls the BeginPaint function.
There are 2 clipping regions. One is an application defined one created by the application using SelectClipRgn and the pointer is stored in pdc->dclevel.prgnClip, and the other clipping region is system region, after it has been updated to the intersection of the system region and the update region by a BeginPaint call, where it is presented to the application as a logical clipping rectangle in the PAINTSTRUCT.
GetClipBox calls NtGdiGetAppClipBox, which calls GdiGetClipBox, which of course returns the smallest logical rect boundary of the current system region, which may be the visible region if GetDC was used, or it may be the system region intersected with a custom clipping region with GetDCEx, or it may be the system region intersected with the window update region when using BeginPaint. Your issue would imply that the system region, when calculated, is now performing special handling for the desktop window in VIS_ComputeVisibleRegion
To actually access the DC directly, and hence the System region, you'd have to start and interact with a driver to do it from the application.
This seems to be a bug in the relevant versions of Windows which has apparently been fixed in more recent versions.
For anybody sailing through countless google pages, blog posts, SO answers... calling:
SetWindowLong(hwnd, GWL_EXSTYLE, WS_EX_LAYERED);
seems to do the trick. This doesn't answer how to extend the clipping region over the area restricted by windows, but allows to capture screen correctly which is pretty much the goal anyway. This also is a solution to many other issues, like the taskbar thumbnail not updating, parts of window filckering with black when dragging or bugs when capturing to video file. MSDN doesn't specifically explain that anywhere.
One more thing worth pointing out is that Discord is able to stream a partially offscreen window WITHOUT modifying any of window's attributes, so there's probably more to that...
GDI is not the best way for doing screenshots, often it can't get even completely visible window.
Few months ago I found Youtube video with DWM hacking, which allows you to take screenshot of any window.
Here are sources. Personally I, didn't try to compile and run it.
Related
I have a program (we will call it the "virtual screen") that create a full screen window and start arbitrary programs, and with the help of hooks (CBTProc) get handles to windows that started programs create. From those handles I retrieve the content of the windows (using GetDIBits) and displays it in the "virtual screen" window.
Currently, this "virtual screen" copy content of windows and then redraw them, which make it work, sort of like a mirroring software.
Here is how I get the content of a window:
struct WindowContent {
void *pixel;
int width;
int height;
};
WindowContent getWindowContent(HWND hWnd, int height, int width)
{
WindowContent content;
WINDOWINFO windowInfo;
GetWindowInfo(hWnd, &windowInfo);
content.height = windowInfo.rcClient.right - windowInfo.rcClient.left;
content.width = windowInfo.rcClient.bottom - windowInfo.rcClient.top;
HDC hdc = GetDC(hWnd);
HDC captureHdc = CreateCompatibleDC(hdc);
HBITMAP hBitmap = CreateCompatibleBitmap(hdc, content.width, content.height);
HGDIOBJ oldHdc = SelectObject(captureHdc, hBitmap);
BitBlt(captureHdc, 0, 0, content.width, content.height, hdc, 0, 0, SRCCOPY|CAPTUREBLT);
SelectObject(captureHdc, oldHdc);
DeleteDC(captureHdc);
BITMAPINFO outputBitmapInfo = {};
outputBitmapInfo.bmiHeader.biSize = sizeof(BITMAPINFOHEADER);
GetDIBits(hdc, hBitmap, 0, 0, NULL, &outputBitmapInfo, DIB_RGB_COLORS);
content.pixel = (BYTE *)malloc(outputBitmapInfo.bmiHeader.biSizeImage);
outputBitmapInfo.bmiHeader.biCompression = BI_RGB;
outputBitmapInfo.bmiHeader.biBitCount = 32;
GetDIBits(hdc, hBitmap, 0, outputBitmapInfo.bmiHeader.biHeight, content.pixel, &outputBitmapInfo, DIB_RGB_COLORS);
return content;
}
My question is, how do I remove the copying part, how can I make an area on my "virtual screen" the window output for those programs ?
Emphasis on the fact that I'm trying to make created windows be the area on the "virtual screen", I don't want an additional window hidden or living on the desktop.
In my research, I've looked into Windows DWM DLLs and found some undocumented function (SignalRedirectionStartComplete or MilConnection_CreateChannel) which names look linked to what I want to do, but I don't think I should use them, as they are undocumented.
Also, the code is using Win32 API but I don't mind using another Windows API or another language (C#, DX* ...).
Forgot to mention, I have already thought about using the DWM thumbnail stuff, but it's not that reliable enough for what I'm trying to do.
As far as I understand Windows 10 uses DX under the hood for all display output, for GDI, and even for Vulkan / OpenGL programs, and someone used it to make a lib that
gets DX 10 texture from a window (). Is it possible to make something similar, that, for a specific HWND, set its "output" to a texture or some region in
memory (swapchain redirection ?) instead of the screen and then display the output in another program (in my case, on the "virtual screen" window) ?
The DWM was a reasonable place to look. It has some documented functions to get you at least a little closer to what you want.
You can register your window as a "thumbnail" viewer using DwmRegisterThumbnail. Then you (probably) call DwmUpdateThumbnailProperties to tell it how to draw to your window (e.g., set opacity, rectangle to draw to, and whether to draw the whole source window, or just its client area). When you're done, you call DwmUnregisterThumbnail to unhook from displaying it.
This is only a little closer to what you want though--it eliminates your having to copy bitmaps from the source into your own window--but that's about all it does. The target application will still have its own window running elsewhere.
If you want to do more to hide the application, you could create another desktop, and display the application in that desktop. If you don't provide a way to switch desktops, that can keep it pretty well hidden. On the other hand, there are external tools that will let the user change desktops, which would let them see the application directly.
One other avenue to consider (for a subset of applications) would be COM. Part of what COM supports is having a COM server that displays its output inside the frame of some COM client's window. That's no longer nearly the big thing it once was, but (I believe) all the code to support it is still available. But, it only works for applications that are specifically written to support it (which, honestly, isn't a huge number). The code for this isn't exactly trivial either.
I want to write a screencasting program for the Windows platform, but am unsure of how to capture the screen. The only method I'm aware of is to use GDI, but I'm curious whether there are other ways to go about this, and, if there are, which incurs the least overhead? Speed is a priority.
The screencasting program will be for recording game footage, although, if this does narrow down the options, I'm still open for any other suggestions that fall out of this scope. Knowledge isn't bad, after all.
Edit: I came across this article: Various methods for capturing the screen. It has introduced me to the Windows Media API way of doing it and the DirectX way of doing it. It mentions in the Conclusion that disabling hardware acceleration could drastically improve the performance of the capture application. I'm curious as to why this is. Could anyone fill in the missing blanks for me?
Edit: I read that screencasting programs such as Camtasia use their own capture driver. Could someone give me an in-depth explanation on how it works, and why it is faster? I may also need guidance on implementing something like that, but I'm sure there is existing documentation anyway.
Also, I now know how FRAPS records the screen. It hooks the underlying graphics API to read from the back buffer. From what I understand, this is faster than reading from the front buffer, because you are reading from system RAM, rather than video RAM. You can read the article here.
This is what I use to collect single frames, but if you modify this and keep the two targets open all the time then you could "stream" it to disk using a static counter for the file name. - I can't recall where I found this, but it has been modified, thanks to whoever!
void dump_buffer()
{
IDirect3DSurface9* pRenderTarget=NULL;
IDirect3DSurface9* pDestTarget=NULL;
const char file[] = "Pickture.bmp";
// sanity checks.
if (Device == NULL)
return;
// get the render target surface.
HRESULT hr = Device->GetRenderTarget(0, &pRenderTarget);
// get the current adapter display mode.
//hr = pDirect3D->GetAdapterDisplayMode(D3DADAPTER_DEFAULT,&d3ddisplaymode);
// create a destination surface.
hr = Device->CreateOffscreenPlainSurface(DisplayMde.Width,
DisplayMde.Height,
DisplayMde.Format,
D3DPOOL_SYSTEMMEM,
&pDestTarget,
NULL);
//copy the render target to the destination surface.
hr = Device->GetRenderTargetData(pRenderTarget, pDestTarget);
//save its contents to a bitmap file.
hr = D3DXSaveSurfaceToFile(file,
D3DXIFF_BMP,
pDestTarget,
NULL,
NULL);
// clean up.
pRenderTarget->Release();
pDestTarget->Release();
}
EDIT: I can see that this is listed under your first edit link as "the GDI way". This is still a decent way to go even with the performance advisory on that site, you can get to 30fps easily I would think.
From this comment (I have no experience doing this, I'm just referencing someone who does):
HDC hdc = GetDC(NULL); // get the desktop device context
HDC hDest = CreateCompatibleDC(hdc); // create a device context to use yourself
// get the height and width of the screen
int height = GetSystemMetrics(SM_CYVIRTUALSCREEN);
int width = GetSystemMetrics(SM_CXVIRTUALSCREEN);
// create a bitmap
HBITMAP hbDesktop = CreateCompatibleBitmap( hdc, width, height);
// use the previously created device context with the bitmap
SelectObject(hDest, hbDesktop);
// copy from the desktop device context to the bitmap device context
// call this once per 'frame'
BitBlt(hDest, 0,0, width, height, hdc, 0, 0, SRCCOPY);
// after the recording is done, release the desktop context you got..
ReleaseDC(NULL, hdc);
// ..delete the bitmap you were using to capture frames..
DeleteObject(hbDesktop);
// ..and delete the context you created
DeleteDC(hDest);
I'm not saying this is the fastest, but the BitBlt operation is generally very fast if you're copying between compatible device contexts.
For reference, Open Broadcaster Software implements something like this as part of their "dc_capture" method, although rather than creating the destination context hDest using CreateCompatibleDC they use an IDXGISurface1, which works with DirectX 10+. If there is no support for this they fall back to CreateCompatibleDC.
To change it to use a specific application, you need to change the first line to GetDC(game) where game is the handle of the game's window, and then set the right height and width of the game's window too.
Once you have the pixels in hDest/hbDesktop, you still need to save it to a file, but if you're doing screen capture then I would think you would want to buffer a certain number of them in memory and save to the video file in chunks, so I will not point to code for saving a static image to disk.
I wrote a video capture software, similar to FRAPS for DirectX applications. The source code is available and my article explains the general technique. Look at http://blog.nektra.com/main/2013/07/23/instrumenting-direct3d-applications-to-capture-video-and-calculate-frames-per-second/
Respect to your questions related to performance,
DirectX should be faster than GDI except when you are reading from the frontbuffer which is very slow. My approach is similar to FRAPS (reading from backbuffer). I intercept a set of methods from Direct3D interfaces.
For video recording in realtime (with minimal application impact), a fast codec is essential. FRAPS uses it's own lossless video codec. Lagarith and HUFFYUV are generic lossless video codecs designed for realtime applications. You should look at them if you want to output video files.
Another approach to recording screencasts could be to write a Mirror Driver. According to Wikipedia: When video mirroring is active, each time the system draws to the primary video device at a location inside the mirrored area, a copy of the draw operation is executed on the mirrored video device in real-time. See mirror drivers at MSDN: http://msdn.microsoft.com/en-us/library/windows/hardware/ff568315(v=vs.85).aspx.
I use d3d9 to get the backbuffer, and save that to a png file using the d3dx library:
IDirect3DSurface9 *surface ;
// GetBackBuffer
idirect3ddevice9->GetBackBuffer(0, 0, D3DBACKBUFFER_TYPE_MONO, &surface ) ;
// save the surface
D3DXSaveSurfaceToFileA( "filename.png", D3DXIFF_PNG, surface, NULL, NULL ) ;
SAFE_RELEASE( surface ) ;
To do this you should create your swapbuffer with
d3dpps.SwapEffect = D3DSWAPEFFECT_COPY ; // for screenshots.
(So you guarantee the backbuffer isn't mangled before you take the screenshot).
In my Impression, the GDI approach and the DX approach are different in its nature.
painting using GDI applies the FLUSH method, the FLUSH approach draws the frame then clear it and redraw another frame in the same buffer, this will result in flickering in games require high frame rate.
WHY DX quicker?
in DX (or graphics world), a more mature method called double buffer rendering is applied, where two buffers are present, when present the front buffer to the hardware, you can render to the other buffer as well, then after the frame 1 is finished rendering, the system swap to the other buffer( locking it for presenting to hardware , and release the previous buffer ), in this way the rendering inefficiency is greatly improved.
WHY turning down hardware acceleration quicker?
although with double buffer rendering, the FPS is improved, but the time for rendering is still limited. modern graphic hardware usually involves a lot of optimization during rendering typically like anti-aliasing, this is very computation intensive, if you don't require that high quality graphics, of course you can just disable this option. and this will save you some time.
I think what you really need is a replay system, which I totally agree with what people discussed.
I wrote a class that implemented the GDI method for screen capture. I too wanted extra speed so, after discovering the DirectX method (via GetFrontBuffer) I tried that, expecting it to be faster.
I was dismayed to find that GDI performs about 2.5x faster. After 100 trials capturing my dual monitor display, the GDI implementation averaged 0.65s per screen capture, while the DirectX method averaged 1.72s. So GDI is definitely faster than GetFrontBuffer, according to my tests.
I was unable to get Brandrew's code working to test DirectX via GetRenderTargetData. The screen copy came out purely black. However, it could copy that blank screen super fast! I'll keep tinkering with that and hope to get a working version to see real results from it.
For C++ you can use: http://www.pinvoke.net/default.aspx/gdi32/BitBlt.html
This may hower not work on all types of 3D applications/video apps. Then this link may be more useful as it describes 3 different methods you can use.
Old answer (C#):
You can use System.Drawing.Graphics.Copy, but it is not very fast.
A sample project I wrote doing exactly this: http://blog.tedd.no/index.php/2010/08/16/c-image-analysis-auto-gaming-with-source/
I'm planning to update this sample using a faster method like Direct3D: http://spazzarama.com/2009/02/07/screencapture-with-direct3d/
And here is a link for capturing to video: How to capture screen to be video using C# .Net?
You want the Desktop Duplication API (available since Windows 8). That is the officially recommended way of doing it, and it's also the most CPU efficient.
One nice feature it has for screencasting is that it detects window movement, so you can transmit block deltas when windows get moved around, instead of raw pixels. Also, it tells you which rectangles have changed, from one frame to the next.
The Microsoft example code is quite complex, but the API is actually simple and easy to use. I've put together an example project that is much simpler:
Simplified Sample Code
WindowsDesktopDuplicationSample
Microsoft References
Desktop Duplication API
Official example code (my example above is a stripped down version of this)
A few things I've been able to glean: apparently using a "mirror driver" is fast though I'm not aware of an OSS one.
Why is RDP so fast compared to other remote control software?
Also apparently using some convolutions of StretchRect are faster than BitBlt
http://betterlogic.com/roger/2010/07/fast-screen-capture/comment-page-1/#comment-5193
And the one you mentioned (fraps hooking into the D3D dll's) is probably the only way for D3D applications, but won't work with Windows XP desktop capture. So now I just wish there were a fraps equivalent speed-wise for normal desktop windows...anybody?
(I think with aero you might be able to use fraps-like hooks, but XP users would be out of luck).
Also apparently changing screen bit depths and/or disabling hardware accel. might help (and/or disabling aero).
https://github.com/rdp/screen-capture-recorder-program includes a reasonably fast BitBlt based capture utility, and a benchmarker as part of its install, which can let you benchmark BitBlt speeds to optimize them.
VirtualDub also has an "opengl" screen capture module that is said to be fast and do things like change detection http://www.virtualdub.org/blog/pivot/entry.php?id=290
You can try the c++ open source project WinRobot #git, a powerful screen capturer
CComPtr<IWinRobotService> pService;
hr = pService.CoCreateInstance(__uuidof(ServiceHost) );
//get active console session
CComPtr<IUnknown> pUnk;
hr = pService->GetActiveConsoleSession(&pUnk);
CComQIPtr<IWinRobotSession> pSession = pUnk;
// capture screen
pUnk = 0;
hr = pSession->CreateScreenCapture(0,0,1280,800,&pUnk);
// get screen image data(with file mapping)
CComQIPtr<IScreenBufferStream> pBuffer = pUnk;
Support :
UAC Window
Winlogon
DirectShowOverlay
Screen Recording can be done in C# using VLC API. I have done a sample program to demonstrate this. It uses LibVLCSharp and VideoLAN.LibVLC.Windows libraries. You could achieve many more features related to video rendering using this cross platform API.
For API documentation see: LibVLCSharp API Github
using System;
using System.IO;
using System.Reflection;
using System.Threading;
using LibVLCSharp.Shared;
namespace ScreenRecorderNetApp
{
class Program
{
static void Main(string[] args)
{
Core.Initialize();
using (var libVlc = new LibVLC())
using (var mediaPlayer = new MediaPlayer(libVlc))
{
var media = new Media(libVlc, "screen://", FromType.FromLocation);
media.AddOption(":screen-fps=24");
media.AddOption(":sout=#transcode{vcodec=h264,vb=0,scale=0,acodec=mp4a,ab=128,channels=2,samplerate=44100}:file{dst=testvlc.mp4}");
media.AddOption(":sout-keep");
mediaPlayer.Play(media);
Thread.Sleep(10*1000);
mediaPlayer.Stop();
}
}
}
}
This might not be the fastest method, but it is leightweight and easy to use. The image is returned as an integer array containing the RGB colors.
#define WIN32_LEAN_AND_MEAN
#define VC_EXTRALEAN
#include <Windows.h>
int* screenshot(int& width, int& height) {
HDC hdc = GetDC(NULL); // get the desktop device context
HDC cdc = CreateCompatibleDC(hdc); // create a device context to use yourself
height = (int)GetSystemMetrics(SM_CYVIRTUALSCREEN); // get the width and height of the screen
width = 16*height/9; // only capture left monitor for dual screen setups, for both screens use (int)GetSystemMetrics(SM_CXVIRTUALSCREEN);
HBITMAP hbitmap = CreateCompatibleBitmap(hdc, width, height); // create a bitmap
SelectObject(cdc, hbitmap); // use the previously created device context with the bitmap
BITMAPINFOHEADER bmi = { 0 };
bmi.biSize = sizeof(BITMAPINFOHEADER);
bmi.biPlanes = 1;
bmi.biBitCount = 32;
bmi.biWidth = width;
bmi.biHeight = -height; // flip image upright
bmi.biCompression = BI_RGB;
bmi.biSizeImage = 3*width*height;
BitBlt(cdc, 0, 0, width, height, hdc, 0, 0, SRCCOPY); // copy from desktop device context to bitmap device context
ReleaseDC(NULL, hdc);
int* image = new int[width*height];
GetDIBits(cdc, hbitmap, 0, height, image, (BITMAPINFO*)&bmi, DIB_RGB_COLORS);
DeleteObject(hbitmap);
DeleteDC(cdc);
return image;
}
The above code combines this answer and this answer.
Example on how to use it:
int main() {
int width=0, height=0;
int* image = screenshot(width, height);
// access pixel colors for position (x|y)
const int x=0, y=0;
const int color = image[x+y*width];
const int red = (color>>16)&255;
const int green = (color>> 8)&255;
const int blue = color &255;
delete[] image;
}
i myself do it with directx and think it's as fast as you would want it to be. i don't have a quick code sample, but i found this which should be useful. the directx11 version should not differ a lot, directx9 maybe a little more, but thats the way to go
DXGI Desktop Capture
Project that captures the desktop image with DXGI duplication. Saves the captured image to the file in different image formats (*.bmp; *.jpg; *.tif).
This sample is written in C++. You also need some experience with DirectX (D3D11, D2D1).
What the Application Can Do
If you have more than one desktop monitor, you can choose.
Resize the captured desktop image.
Choose different scaling modes.
You can show or hide the mouse icon in the output image.
You can rotate the image for the output picture, or leave it as default.
I realize the following suggestion doesn't answer your question, but the simplest method I have found to capture a rapidly-changing DirectX view, is to plug a video camera into the S-video port of the video card, and record the images as a movie. Then transfer the video from the camera back to an MPG, WMV, AVI etc. file on the computer.
Windows.Graphics.Capture
Enables apps to capture environments, application windows, and displays in a secure, easy to use way with the use of a system picker UI control.
https://blogs.windows.com/windowsdeveloper/2019/09/16/new-ways-to-do-screen-capture/
I'm drawing into a WinAPI-Window by using the SetPixel()-Function.
If I scale the window or lose focus (another window is on top) I lose the whole content that I draw into the window.
I just used
RECT rc;
GetClientRect(hwnd, &rc);
RedrawWindow(hwnd, &rc, NULL, RDW_NOERASE | RDW_NOFRAME | RDW_VALIDATE);
which helped to avoid redrawing the content when I move the window but scaling and losing focus still removes the content.
Does anyone have an idea what I missed?
Draw it to a buffer/bitmap and then draw that to your window.
When a window needs to be repainted it will be sent a WM_PAINT message. At this point you must redraw all of the window, or at least all parts of it which are contained within a clipping region. Windows does some buffering and automatic painting, specifically it will repaint parts of a window which are covered by other windows then uncovered. Once the window has been resized though, or (presumably) invalidated, you're on your own.
As #daniel suggested, if painting is an intensive process and you don't want to do it every time a repaint is required, render your content into a bitmap (which in this case will be an off-screen buffer) and BitBlt (copy) it into the window as necessary.
Grab yourself a copy of Charles Petzold's book "Programming Windows" for information about how you should go about painting. If you are writing a WinAPI app but have used SetPixel I'd recommend reading the entirety of the first few chapters to get an idea of how an old-school Windows programme should be structured.
SetPixel is very slow, you cannot improve your program significantly. Create in-memory bitmap and draw it on the window. For example, you can do this using StretchDIBits function, which draws the whole memory area as bitmap to the window, instead of SetPixel.
The most important StretchDIBits parameters are:
CONST VOID *lpBits - memory array (pixels). You need to fill it in memory instead of SetPixel calls.
CONST BITMAPINFO *lpBitsInfo - BITMAPINFO structure which must describe bitmap structure. For example, if lpBits has BGRW structure (4 bytes per pixel), BITMAPINFO must describe true color bitmap.
You need to draw the content into memory and then draw it to the window when you got WM_PAINT message. There is no way to avoid using memory buffer because window device context does not save what you draw.
Create a DIB surface and draw into it instead. Then redraw the bitmap when redrawing a window.
You're trying to draw with a pre-Windows way in Windows. ;)
I want to write a screencasting program for the Windows platform, but am unsure of how to capture the screen. The only method I'm aware of is to use GDI, but I'm curious whether there are other ways to go about this, and, if there are, which incurs the least overhead? Speed is a priority.
The screencasting program will be for recording game footage, although, if this does narrow down the options, I'm still open for any other suggestions that fall out of this scope. Knowledge isn't bad, after all.
Edit: I came across this article: Various methods for capturing the screen. It has introduced me to the Windows Media API way of doing it and the DirectX way of doing it. It mentions in the Conclusion that disabling hardware acceleration could drastically improve the performance of the capture application. I'm curious as to why this is. Could anyone fill in the missing blanks for me?
Edit: I read that screencasting programs such as Camtasia use their own capture driver. Could someone give me an in-depth explanation on how it works, and why it is faster? I may also need guidance on implementing something like that, but I'm sure there is existing documentation anyway.
Also, I now know how FRAPS records the screen. It hooks the underlying graphics API to read from the back buffer. From what I understand, this is faster than reading from the front buffer, because you are reading from system RAM, rather than video RAM. You can read the article here.
This is what I use to collect single frames, but if you modify this and keep the two targets open all the time then you could "stream" it to disk using a static counter for the file name. - I can't recall where I found this, but it has been modified, thanks to whoever!
void dump_buffer()
{
IDirect3DSurface9* pRenderTarget=NULL;
IDirect3DSurface9* pDestTarget=NULL;
const char file[] = "Pickture.bmp";
// sanity checks.
if (Device == NULL)
return;
// get the render target surface.
HRESULT hr = Device->GetRenderTarget(0, &pRenderTarget);
// get the current adapter display mode.
//hr = pDirect3D->GetAdapterDisplayMode(D3DADAPTER_DEFAULT,&d3ddisplaymode);
// create a destination surface.
hr = Device->CreateOffscreenPlainSurface(DisplayMde.Width,
DisplayMde.Height,
DisplayMde.Format,
D3DPOOL_SYSTEMMEM,
&pDestTarget,
NULL);
//copy the render target to the destination surface.
hr = Device->GetRenderTargetData(pRenderTarget, pDestTarget);
//save its contents to a bitmap file.
hr = D3DXSaveSurfaceToFile(file,
D3DXIFF_BMP,
pDestTarget,
NULL,
NULL);
// clean up.
pRenderTarget->Release();
pDestTarget->Release();
}
EDIT: I can see that this is listed under your first edit link as "the GDI way". This is still a decent way to go even with the performance advisory on that site, you can get to 30fps easily I would think.
From this comment (I have no experience doing this, I'm just referencing someone who does):
HDC hdc = GetDC(NULL); // get the desktop device context
HDC hDest = CreateCompatibleDC(hdc); // create a device context to use yourself
// get the height and width of the screen
int height = GetSystemMetrics(SM_CYVIRTUALSCREEN);
int width = GetSystemMetrics(SM_CXVIRTUALSCREEN);
// create a bitmap
HBITMAP hbDesktop = CreateCompatibleBitmap( hdc, width, height);
// use the previously created device context with the bitmap
SelectObject(hDest, hbDesktop);
// copy from the desktop device context to the bitmap device context
// call this once per 'frame'
BitBlt(hDest, 0,0, width, height, hdc, 0, 0, SRCCOPY);
// after the recording is done, release the desktop context you got..
ReleaseDC(NULL, hdc);
// ..delete the bitmap you were using to capture frames..
DeleteObject(hbDesktop);
// ..and delete the context you created
DeleteDC(hDest);
I'm not saying this is the fastest, but the BitBlt operation is generally very fast if you're copying between compatible device contexts.
For reference, Open Broadcaster Software implements something like this as part of their "dc_capture" method, although rather than creating the destination context hDest using CreateCompatibleDC they use an IDXGISurface1, which works with DirectX 10+. If there is no support for this they fall back to CreateCompatibleDC.
To change it to use a specific application, you need to change the first line to GetDC(game) where game is the handle of the game's window, and then set the right height and width of the game's window too.
Once you have the pixels in hDest/hbDesktop, you still need to save it to a file, but if you're doing screen capture then I would think you would want to buffer a certain number of them in memory and save to the video file in chunks, so I will not point to code for saving a static image to disk.
I wrote a video capture software, similar to FRAPS for DirectX applications. The source code is available and my article explains the general technique. Look at http://blog.nektra.com/main/2013/07/23/instrumenting-direct3d-applications-to-capture-video-and-calculate-frames-per-second/
Respect to your questions related to performance,
DirectX should be faster than GDI except when you are reading from the frontbuffer which is very slow. My approach is similar to FRAPS (reading from backbuffer). I intercept a set of methods from Direct3D interfaces.
For video recording in realtime (with minimal application impact), a fast codec is essential. FRAPS uses it's own lossless video codec. Lagarith and HUFFYUV are generic lossless video codecs designed for realtime applications. You should look at them if you want to output video files.
Another approach to recording screencasts could be to write a Mirror Driver. According to Wikipedia: When video mirroring is active, each time the system draws to the primary video device at a location inside the mirrored area, a copy of the draw operation is executed on the mirrored video device in real-time. See mirror drivers at MSDN: http://msdn.microsoft.com/en-us/library/windows/hardware/ff568315(v=vs.85).aspx.
I use d3d9 to get the backbuffer, and save that to a png file using the d3dx library:
IDirect3DSurface9 *surface ;
// GetBackBuffer
idirect3ddevice9->GetBackBuffer(0, 0, D3DBACKBUFFER_TYPE_MONO, &surface ) ;
// save the surface
D3DXSaveSurfaceToFileA( "filename.png", D3DXIFF_PNG, surface, NULL, NULL ) ;
SAFE_RELEASE( surface ) ;
To do this you should create your swapbuffer with
d3dpps.SwapEffect = D3DSWAPEFFECT_COPY ; // for screenshots.
(So you guarantee the backbuffer isn't mangled before you take the screenshot).
In my Impression, the GDI approach and the DX approach are different in its nature.
painting using GDI applies the FLUSH method, the FLUSH approach draws the frame then clear it and redraw another frame in the same buffer, this will result in flickering in games require high frame rate.
WHY DX quicker?
in DX (or graphics world), a more mature method called double buffer rendering is applied, where two buffers are present, when present the front buffer to the hardware, you can render to the other buffer as well, then after the frame 1 is finished rendering, the system swap to the other buffer( locking it for presenting to hardware , and release the previous buffer ), in this way the rendering inefficiency is greatly improved.
WHY turning down hardware acceleration quicker?
although with double buffer rendering, the FPS is improved, but the time for rendering is still limited. modern graphic hardware usually involves a lot of optimization during rendering typically like anti-aliasing, this is very computation intensive, if you don't require that high quality graphics, of course you can just disable this option. and this will save you some time.
I think what you really need is a replay system, which I totally agree with what people discussed.
I wrote a class that implemented the GDI method for screen capture. I too wanted extra speed so, after discovering the DirectX method (via GetFrontBuffer) I tried that, expecting it to be faster.
I was dismayed to find that GDI performs about 2.5x faster. After 100 trials capturing my dual monitor display, the GDI implementation averaged 0.65s per screen capture, while the DirectX method averaged 1.72s. So GDI is definitely faster than GetFrontBuffer, according to my tests.
I was unable to get Brandrew's code working to test DirectX via GetRenderTargetData. The screen copy came out purely black. However, it could copy that blank screen super fast! I'll keep tinkering with that and hope to get a working version to see real results from it.
For C++ you can use: http://www.pinvoke.net/default.aspx/gdi32/BitBlt.html
This may hower not work on all types of 3D applications/video apps. Then this link may be more useful as it describes 3 different methods you can use.
Old answer (C#):
You can use System.Drawing.Graphics.Copy, but it is not very fast.
A sample project I wrote doing exactly this: http://blog.tedd.no/index.php/2010/08/16/c-image-analysis-auto-gaming-with-source/
I'm planning to update this sample using a faster method like Direct3D: http://spazzarama.com/2009/02/07/screencapture-with-direct3d/
And here is a link for capturing to video: How to capture screen to be video using C# .Net?
You want the Desktop Duplication API (available since Windows 8). That is the officially recommended way of doing it, and it's also the most CPU efficient.
One nice feature it has for screencasting is that it detects window movement, so you can transmit block deltas when windows get moved around, instead of raw pixels. Also, it tells you which rectangles have changed, from one frame to the next.
The Microsoft example code is quite complex, but the API is actually simple and easy to use. I've put together an example project that is much simpler:
Simplified Sample Code
WindowsDesktopDuplicationSample
Microsoft References
Desktop Duplication API
Official example code (my example above is a stripped down version of this)
A few things I've been able to glean: apparently using a "mirror driver" is fast though I'm not aware of an OSS one.
Why is RDP so fast compared to other remote control software?
Also apparently using some convolutions of StretchRect are faster than BitBlt
http://betterlogic.com/roger/2010/07/fast-screen-capture/comment-page-1/#comment-5193
And the one you mentioned (fraps hooking into the D3D dll's) is probably the only way for D3D applications, but won't work with Windows XP desktop capture. So now I just wish there were a fraps equivalent speed-wise for normal desktop windows...anybody?
(I think with aero you might be able to use fraps-like hooks, but XP users would be out of luck).
Also apparently changing screen bit depths and/or disabling hardware accel. might help (and/or disabling aero).
https://github.com/rdp/screen-capture-recorder-program includes a reasonably fast BitBlt based capture utility, and a benchmarker as part of its install, which can let you benchmark BitBlt speeds to optimize them.
VirtualDub also has an "opengl" screen capture module that is said to be fast and do things like change detection http://www.virtualdub.org/blog/pivot/entry.php?id=290
You can try the c++ open source project WinRobot #git, a powerful screen capturer
CComPtr<IWinRobotService> pService;
hr = pService.CoCreateInstance(__uuidof(ServiceHost) );
//get active console session
CComPtr<IUnknown> pUnk;
hr = pService->GetActiveConsoleSession(&pUnk);
CComQIPtr<IWinRobotSession> pSession = pUnk;
// capture screen
pUnk = 0;
hr = pSession->CreateScreenCapture(0,0,1280,800,&pUnk);
// get screen image data(with file mapping)
CComQIPtr<IScreenBufferStream> pBuffer = pUnk;
Support :
UAC Window
Winlogon
DirectShowOverlay
Screen Recording can be done in C# using VLC API. I have done a sample program to demonstrate this. It uses LibVLCSharp and VideoLAN.LibVLC.Windows libraries. You could achieve many more features related to video rendering using this cross platform API.
For API documentation see: LibVLCSharp API Github
using System;
using System.IO;
using System.Reflection;
using System.Threading;
using LibVLCSharp.Shared;
namespace ScreenRecorderNetApp
{
class Program
{
static void Main(string[] args)
{
Core.Initialize();
using (var libVlc = new LibVLC())
using (var mediaPlayer = new MediaPlayer(libVlc))
{
var media = new Media(libVlc, "screen://", FromType.FromLocation);
media.AddOption(":screen-fps=24");
media.AddOption(":sout=#transcode{vcodec=h264,vb=0,scale=0,acodec=mp4a,ab=128,channels=2,samplerate=44100}:file{dst=testvlc.mp4}");
media.AddOption(":sout-keep");
mediaPlayer.Play(media);
Thread.Sleep(10*1000);
mediaPlayer.Stop();
}
}
}
}
This might not be the fastest method, but it is leightweight and easy to use. The image is returned as an integer array containing the RGB colors.
#define WIN32_LEAN_AND_MEAN
#define VC_EXTRALEAN
#include <Windows.h>
int* screenshot(int& width, int& height) {
HDC hdc = GetDC(NULL); // get the desktop device context
HDC cdc = CreateCompatibleDC(hdc); // create a device context to use yourself
height = (int)GetSystemMetrics(SM_CYVIRTUALSCREEN); // get the width and height of the screen
width = 16*height/9; // only capture left monitor for dual screen setups, for both screens use (int)GetSystemMetrics(SM_CXVIRTUALSCREEN);
HBITMAP hbitmap = CreateCompatibleBitmap(hdc, width, height); // create a bitmap
SelectObject(cdc, hbitmap); // use the previously created device context with the bitmap
BITMAPINFOHEADER bmi = { 0 };
bmi.biSize = sizeof(BITMAPINFOHEADER);
bmi.biPlanes = 1;
bmi.biBitCount = 32;
bmi.biWidth = width;
bmi.biHeight = -height; // flip image upright
bmi.biCompression = BI_RGB;
bmi.biSizeImage = 3*width*height;
BitBlt(cdc, 0, 0, width, height, hdc, 0, 0, SRCCOPY); // copy from desktop device context to bitmap device context
ReleaseDC(NULL, hdc);
int* image = new int[width*height];
GetDIBits(cdc, hbitmap, 0, height, image, (BITMAPINFO*)&bmi, DIB_RGB_COLORS);
DeleteObject(hbitmap);
DeleteDC(cdc);
return image;
}
The above code combines this answer and this answer.
Example on how to use it:
int main() {
int width=0, height=0;
int* image = screenshot(width, height);
// access pixel colors for position (x|y)
const int x=0, y=0;
const int color = image[x+y*width];
const int red = (color>>16)&255;
const int green = (color>> 8)&255;
const int blue = color &255;
delete[] image;
}
i myself do it with directx and think it's as fast as you would want it to be. i don't have a quick code sample, but i found this which should be useful. the directx11 version should not differ a lot, directx9 maybe a little more, but thats the way to go
DXGI Desktop Capture
Project that captures the desktop image with DXGI duplication. Saves the captured image to the file in different image formats (*.bmp; *.jpg; *.tif).
This sample is written in C++. You also need some experience with DirectX (D3D11, D2D1).
What the Application Can Do
If you have more than one desktop monitor, you can choose.
Resize the captured desktop image.
Choose different scaling modes.
You can show or hide the mouse icon in the output image.
You can rotate the image for the output picture, or leave it as default.
I realize the following suggestion doesn't answer your question, but the simplest method I have found to capture a rapidly-changing DirectX view, is to plug a video camera into the S-video port of the video card, and record the images as a movie. Then transfer the video from the camera back to an MPG, WMV, AVI etc. file on the computer.
Windows.Graphics.Capture
Enables apps to capture environments, application windows, and displays in a secure, easy to use way with the use of a system picker UI control.
https://blogs.windows.com/windowsdeveloper/2019/09/16/new-ways-to-do-screen-capture/
I'm writing an unmanaged Win32 C++ function that gets a handle to a bitmap, and I need to draw on it.
My problem is that to draw I need to get a device context, but when I do GetDC (NULL), it gives me a device context for the WINDOW! The parameter for GetDC () is a window handle (HWND), but I don't have a window; just a bitmap handle.
How can I draw on this bitmap? Thanks!
In addition to Pavel's answer, the "compatible with the screen" always bugged me too, but, since CreateCompatibleDC(NULL) is universally used for that purpose, I assume it is correct.
I think that the "compatible" thing is related just to DDB (the DC is set up to write on the correct DDB type for the current screen), but does not affect read/writes on DIBs.
So, to be safe, always use DIBs and not DDBs if you need to work on bitmaps that doesn't just have to go temporarily onscreen, nowadays the difference in performance is negligible. See here for more info about DIBs and DDBs.
CreateCompatibleDC() and SelectObject() your bitmap into it.
However, not every bitmap can be selected into any DC.
You might have to play with mapping mode and other options of memory DCs.
The basic win32 paradigm for drawing on a bitmap is that you select the bitmap onto a device context, after which, all drawing operations on that device context are stored in the bitmap. You then use one of the various 'blit' operations (e.g. StretchBlt) to transfer this to a display surface, which is just the device context of a window client area.
Others have provided better detail, this is just the high-level view.
Well, this is a bit outside the box.. I guess.. But I do know that Graphics can return a HDC, and Graphics take a Bitmap as an argument to its ctor . A Bitmap in turn can be created from a HBITMAP and a HPALETTE. The only problem here is that I do not know if the HPALETTE argument can be NULL.
Graphics* g;
Bitmap* bitmap;
HBITMAP _bitmap; // <- this one is yours
bitmap = Bitmap::FromHBITMAP(_bitmap, NULL);
g = new Graphics(bitmap);
HDC hdc = g->GetHDC();
// when done, call g->ReleaseHDC(hdc);
However, I would urge you to receive the HDC as an argument to your function as well.. I do not think that anyone will have a BITMAP and NOT have the DC to it.
If you're having these issues with finding a HDC to a HBITMAP, so will everyone else.