Hanging of XShmPutImage event notification - c++

I am using XShm extension to draw and manipulate images in Linux.
In order to not have screen flickering, I am passing send_event = TRUE to XShmPutImage and then waiting for the event with XIfEvent, immediately after the call to XScmPutImage.
This way, I am making the image drawing blocking in order to not changing the image until it is displayed on the window surface.
Usually everything works fine. But sometimes, when I have intensive image drawing, it seems that the event never comes and the drawing procedure hangs.
Where to see for the problem? Is using XIfEvent appropriate for this task? How can the event dissapear from the message queue?
Is it possible XShmPutImage to not send event (if send_event = TRUE) or to send event different than ShmCompletion on some circumstances? (for example on some internal error or something?)
EDIT:
After some more research, I found that such hangs happens only when the window manager intensively generate events to the window. For example when I resize the window by dragging its corners.
EDIT2:
I tried several ways to solve this problem, but without success. At the end I was forced to use some timeout and to cancel waiting after some time. But of course this is dirty hack and I want to fix it anyway.
So, what can be the reason XShmPutImage to not send event if send_event=TRUE or is it possible this event to disappear from the message queue?
EDIT3:
Here is the questionable code (FASM):
cinvoke XShmPutImage, ......, TRUE
.loop:
lea eax, [.event]
cinvoke XCheckTypedEvent, [Display], [ShmCompletionEvent], eax
test eax, eax
jz .loop ; there is no message
NB: XShmPutImage always return TRUE, regardless whether the event check hangs or not, so I didn't put error check after it.
EDIT4:
Because of request I am posting the whole code of the drawing function. The code uses some macro libraries of FASM, but at least the ideas are clear (I hope)
Notice that this code contains workaround code that limits the event waiting for only 20ms. Without this timeout the waiting loop simply hangs forever. The number of the XShm event is acquired by calling XShmGetEventBase as recommended in the Xshm documentation.
; Draws the image on a OS provided window surface.
proc DrawImageRect, .where, .pImage, .xDst, .yDst, .xSrc, .ySrc, .width, .height
.event XEvent
rb 256
begin
pushad
mov esi, [.pImage]
test esi, esi
jz .exit
mov ebx, [esi+TImage.ximage]
cinvoke XCreateGC, [hApplicationDisplay], [.where], 0, 0
mov edi, eax
cinvoke XShmPutImage, [hApplicationDisplay], [.where], edi, [esi+TImage.ximage], [.xSrc], [.ySrc], [.xDst], [.yDst], [.width], [.height], TRUE
stdcall GetTimestamp
lea esi, [eax+20] ; 20ms timeout
.loop:
lea eax, [.event]
cinvoke XCheckTypedEvent, [hApplicationDisplay], [ShmCompletionEvent], eax
test eax, eax
jnz .finish
stdcall GetTimestamp
cmp eax, esi
jb .loop
.finish:
cinvoke XFreeGC, [hApplicationDisplay], edi
.exit:
popad
return
endp
And here is the code of the main event loop of the application.
The procedure __ProcessOneSystemEvent simply dispatches the events to the GUI objects and ignores all events it does not use. It does not process ShmCompletionEvent at all.
All the windows created in the application have events mask of:
ExposureMask+FocusChangeMask+KeyPressMask+KeyReleaseMask+ButtonPressMask+ButtonReleaseMask+EnterWindowMask+LeaveWindowMask+PointerMotionMask+StructureNotifyMask
proc ProcessSystemEvents
.event XEvent
rb 256
begin
push ebx ecx edx
.event_loop:
; check for quit
get eax, [pApplication], TApplication:MainWindow
test eax, eax
jz .continue
cmp dword [eax], 0
jne .continue
cinvoke XFlush, [hApplicationDisplay]
xor eax, eax
mov [fGlobalTerminate], 1
stc
pop edx ecx ebx
return
.continue:
cinvoke XPending, [hApplicationDisplay]
test eax, eax
jz .noevents
push edi ecx
lea edi, [.event]
mov ecx, sizeof.XEvent/4
xor eax, eax
rep stosd
pop ecx edi
lea ebx, [.event]
cinvoke XNextEvent, [hApplicationDisplay], ebx
stdcall __ProcessOneSystemEvent, ebx
jmp .event_loop
.noevents:
clc
pop edx ecx ebx
return
endp
The full source code is available in the repository but it is a very big project not easy for navigation. The discussed source is in check-in 8453c99b1283def8.
The files: "freshlib/graphics/images.asm" and "freshlib/graphics/Linux/images.asm" are about the image drawing.
The files: "freshlib/gui/Main.asm" and "freshlib/gui/Linux/Main.asm" are about the general events handling in the application.

What is the X server doing?
The X server can and will suppress a ShmCompletionEvent if the parameters passed to XShmPutImage exceed the geometry of the shared memory area attached to the XImage in the call. The server checks X/Y and width/height against the previously stored limits for the given shared area and if the call parameters are out-of-bounds, the server will return BadValue, suppress the drawing operation, and suppress the completion event.
The above is exactly what is happening in your library. Here's how:
The main event dispatcher routine is ProcessSystemEvents. It performs an XEventNext, and based on event type, using a jump table .jump_table dispatches to an event specific handler function.
The event specific function for an Expose event is .expose
The .expose function will, in turn, call DrawImageRect using X/Y and width/height values from the XExposeEvent struct. This is wrong and is the true source of the bug, as we shall see momentarily.
DrawImageRect will pass these values along in a call to XShmPutImage
The handler for XShmPutImage in the X server will examine these parameters and reject if they're out of bounds.
The parameters are being rejected because they come from an exposure event and are related to the geometry of the window and not the geometry of the shared memory attached to the XImage used in the XShmPutImage call.
Specifically, if the window has just be resized (e.g. by the window manager) and has been enlarged, and there has been a prior ConfigureNotify event for resize. Now, with a new Expose event, it will have larger width/height that will exceed the width/height of the shared memory area that the server knows about.
It is the responsibility of the client to field the window resize events [etc.] and teardown/recreate the shared memory area with the enlarged size. This is not being done and is the source of the bug.
NOTE: Just to be completely clear on this, the server can only report on the error and not do anything about it for a few reasons:
The server knows about the window [and its resize].
It knows about the XImage, its shared memory area and size
But they're only associated during the XShmPutImage call [AFAIK]
Even if the server could associate them, it couldn't adjust the shmarea
That's because it has no way to relink the shmarea to the client side
Only the client can do that via XShmDetach/XShmAttach
Below are redacted versions of the relevant source files from the c5c765bc7e commit. They have been cleaned up a bit, so only the most germane parts remain. Some lines have been truncated or wrapped to eliminate horizontal scrolling.
The files have been annotated with NOTE and NOTE/BUG which I did while analyzing them.
gui/Main.asm The top level generic main loop. Nothing to see much to see here.
; FILE: gui/Main.asm
; _____________________________________________________________________________
;| |
;| ..::FreshLib::.. Free, open source. Licensed under "BSD 2-clause" license." |
;|_____________________________________________________________________________|
;
; Description: Main procedure of GUI application library.
;
; Target OS: Any
;
; Dependencies:
;
; Notes: Organize the main message/event loop needed by every GUI engine.
; This file contains only OS independent part and includes OS dependent
; files.
;______________________________________________________________________________
module "Main library"
proc Run
begin
.mainloop:
stdcall ProcessSystemEvents
jc .terminate
mov eax, [pApplication]
test eax, eax
jz .eventok
get ecx, eax, TApplication:OnIdle
jecxz .eventok
stdcall ecx, eax
.eventok:
stdcall WaitForSystemEvent
jmp .mainloop
.terminate:
DebugMsg "Terminate GUI application!"
return
endp
include '%TargetOS%/Main.asm'
endmodule
gui/Linux/Main.asm The event handlers
; FILE: gui/Linux/Main.asm
; _____________________________________________________________________________
;| |
;| ..::FreshLib::.. Free, open source. Licensed under "BSD 2-clause" license." |
;|_____________________________________________________________________________|
;
; Description: Main procedure of GUI application library.
;
; Target OS: Linux
;
; Dependencies:
;
; Notes: Organize the main message/event loop needed by every GUI engine.
;______________________________________________________________________________
body ProcessSystemEvents
; NOTE: this is the storage for the dequeued event -- all dispatch routines
; should use it and process it
.event XEvent
rb 256
begin
push ebx ecx edx
.event_loop:
; check for quit
get eax, [pApplication], TApplication:MainWindow
test eax, eax
jz .continue ; ???????????
cmp dword [eax], 0
jne .continue
cinvoke XFlush, [hApplicationDisplay]
xor eax, eax
mov [fGlobalTerminate], 1
stc
pop edx ecx ebx
return
; NOTE: it is wasteful for the main loop to call WaitForSystemEvent, then call
; us and we do XPending on the first loop -- we already know we have at least
; one event waiting in the queue
.continue:
cinvoke XPending, [hApplicationDisplay]
test eax, eax
jz .noevents
push edi ecx
lea edi, [.event]
mov ecx, sizeof.XEvent/4
xor eax, eax
rep stosd
pop ecx edi
lea ebx, [.event]
cinvoke XNextEvent, [hApplicationDisplay], ebx
stdcall __ProcessOneSystemEvent, ebx
jmp .event_loop
.noevents:
clc
pop edx ecx ebx
return
endp
body WaitForSystemEvent
.event XEvent
begin
push eax ecx edx
lea eax, [.event]
cinvoke XPeekEvent, [hApplicationDisplay], eax
pop edx ecx eax
return
endp
proc __ProcessOneSystemEvent, .linux_event
begin
pushad
mov ebx, [.linux_event]
; mov eax, [ebx+XEvent.type]
; cmp eax, [ShmCompletionEvent]
; je .shm_completion
stdcall _GetWindowStruct, [ebx+XEvent.window]
jc .notprocessed
test eax, eax
jz .notprocessed
mov esi, eax
mov ecx, [ebx+XEvent.type]
cmp ecx, LASTEvent
jae .notprocessed
mov ecx, [.jump_table+4*ecx]
jecxz .notprocessed
jmp ecx
.notprocessed:
popad
stc
return
.finish:
popad
clc
return
;.shm_completion:
; DebugMsg "Put back completion event!"
;
; int3
; cinvoke XPutBackEvent, [hApplicationDisplay], ebx
; jmp .finish
;.........................................................................
; seMove and seResize events.
;-------------------------------------------------------------------------
.moveresize:
; NOTE/BUG!!!!: we must not only process a resize/move request, but we must also
; adjust the size of the shmarea attached to the XImage -- that is _not_ being
; done. (e.g.) if the window is enlarged, the shmarea must be enlarged
cinvoke XCheckTypedWindowEvent, [hApplicationDisplay],
[ebx+XConfigureEvent.window], ConfigureNotify, ebx
test eax, eax
jnz .moveresize
; resize event...
mov eax, [esi+TWindow._width]
mov edx, [esi+TWindow._height]
cmp eax, [ebx+XConfigureEvent.width]
jne .resize
cmp edx, [ebx+XConfigureEvent.height]
je .is_move
.resize:
exec esi, TWindow:EventResize, [ebx+XConfigureEvent.width],
[ebx+XConfigureEvent.height]
; move event...
.is_move:
mov eax, [esi+TWindow._x]
mov edx, [esi+TWindow._y]
cmp eax, [ebx+XConfigureEvent.x]
jne .move
cmp eax, [ebx+XConfigureEvent.y]
je .finish
.move:
exec esi, TWindow:EventMove,
[ebx+XConfigureEvent.x], [ebx+XConfigureEvent.y]
jmp .finish
;.........................................................................
; DestroyNotify handler it invalidates the handle in TWindow structure and
; then destroys TWindow.
.destroy:
test esi, esi
jz .finish
mov [esi+TWindow.handle], 0
destroy esi
jmp .finish
;.........................................................................
; Window paint event
.expose:
get edi, esi, TWindow:ImgScreen
; NOTE:BUG!!!!!
;
; if the window has been resized (e.g. enlarged), these values are wrong!
; they relate to the _window_ but _not_ the shmarea that is attached to the
; XImage
;
; however, DrawImageRect will call XShmPutImage with these values, they
; will exceed the geometry of what the X server knows about the shmarea and
; it will return BadValue and _suppress_ the completion event for XShmPutImage
stdcall DrawImageRect, [esi+TWindow.handle], edi,
[ebx+XExposeEvent.x],[ebx+XExposeEvent.y],
[ebx+XExposeEvent.x], [ebx+XExposeEvent.y],
[ebx+XExposeEvent.width], [ebx+XExposeEvent.height]
jmp .finish
;.........................................................................
; Mouse event handlers
.mousemove:
cinvoke XCheckTypedWindowEvent, [hApplicationDisplay],
[ebx+XConfigureEvent.window], MotionNotify, ebx
test eax, eax
jnz .mousemove
stdcall ServeMenuMouseMove, [ebx+XMotionEvent.window],
[ebx+XMotionEvent.x], [ebx+XMotionEvent.y],
[ebx+XMotionEvent.state]
jc .finish
cinvoke XCheckTypedWindowEvent, [hApplicationDisplay],
[ebx+XMotionEvent.window], MotionNotify, ebx
test eax, eax
jnz .mousemove
mov edi, [__MouseTarget]
test edi, edi
jz .search_target_move
stdcall __GetRelativeXY, edi, [ebx+XMotionEvent.x], [ebx+XMotionEvent.y]
jmp .target_move
.search_target_move:
exec esi, TWindow:ChildByXY, [ebx+XMotionEvent.x],
[ebx+XMotionEvent.y], TRUE
mov edi, eax
.target_move:
cmp edi, [__LastPointedWindow]
je .move_event
cmp [__LastPointedWindow], 0
je .leave_ok
exec [__LastPointedWindow], TWindow:EventMouseLeave
.leave_ok:
mov [__LastPointedWindow], edi
exec edi, TWindow:EventMouseEnter
.move_event:
exec edi, TWindow:EventMouseMove, ecx, edx, [ebx+XMotionEvent.state]
jmp .finish
;.........................................................................
; event jump table
.jump_table dd 0 ; event 0
dd 0 ; event 1
dd .key_press ; KeyPress = 2
dd .key_release ; KeyRelease = 3
dd .mouse_btn_press ; ButtonPress = 4
dd .mouse_btn_release ; ButtonRelease = 5
dd .mousemove ; MotionNotify = 6
dd 0 ; EnterNotify = 7
dd 0 ; LeaveNotify = 8
dd .focusin ; FocusIn = 9
dd .focusout ; FocusOut = 10
dd 0 ; KeymapNotify = 11
dd .expose ; Expose = 12
dd 0 ; GraphicsExpose = 13
dd 0 ; NoExpose = 14
dd 0 ; VisibilityNotify = 15
dd 0 ; CreateNotify = 16
dd .destroy ; DestroyNotify = 17
dd 0 ; UnmapNotify = 18
dd 0 ; MapNotify = 19
dd 0 ; MapRequest = 20
dd 0 ; ReparentNotify = 21
dd .moveresize ; ConfigureNotify = 22
dd 0 ; ConfigureRequest = 23
dd 0 ; GravityNotify = 24
dd 0 ; ResizeRequest = 25
dd 0 ; CirculateNotify = 26
dd 0 ; CirculateRequest = 27
dd 0 ; PropertyNotify = 28
dd 0 ; SelectionClear = 29
dd 0 ; SelectionRequest = 30
dd 0 ; SelectionNotify = 31
dd 0 ; ColormapNotify = 32
dd .clientmessage ; ClientMessage = 33
dd .mapping_notify ; MappingNotify = 34
graphics/Linux/images.asm The image drawing code [including the DrawImageRect function] and the shared memory create/destroy code.
; FILE: graphics/Linux/images.asm
; _____________________________________________________________________________
;| |
;| ..::FreshLib::.. Free, open source. Licensed under "BSD 2-clause" license." |
;|_____________________________________________________________________________|
;
; Description: Memory based images manipulation library.
;
; Target OS: Linux
;
; Dependencies: memory.asm
;
; Notes:
;______________________________________________________________________________
uses libX11, xshm
struct TImage
.width dd ? ; width in pixels.
.height dd ? ; height in pixels.
.pPixels dd ? ; pointer to the pixel memory.
; os dependent data
.ximage dd ?
.shminfo XShmSegmentInfo
ends
body CreateImage
begin
pushad
stdcall GetMem, sizeof.TImage
jc .finish
mov esi, eax
xor eax, eax
inc eax
mov ecx, [.width]
mov edx, [.height]
cmp ecx, 0
cmovle ecx, eax
cmp edx, 0
cmovle edx, eax
mov [esi+TImage.width], ecx
mov [esi+TImage.height], edx
lea eax, [4*ecx]
imul eax, edx
cinvoke shmget, IPC_PRIVATE, eax, IPC_CREAT or 777o
test eax, eax
js .error
mov [esi+TImage.shminfo.ShmID], eax
cinvoke shmat, eax, 0, 0
cmp eax, -1
je .error_free
mov [esi+TImage.shminfo.Addr], eax
mov [esi+TImage.pPixels], eax
mov [esi+TImage.shminfo.fReadOnly], 1
lea ebx, [esi+TImage.shminfo]
cinvoke XShmCreateImage, [hApplicationDisplay], 0, $20, ZPixmap, eax,
ebx, [esi+TImage.width], [esi+TImage.height]
mov [esi+TImage.ximage], eax
cinvoke XShmAttach, [hApplicationDisplay], ebx
clc
mov [esp+4*regEAX], esi
.finish:
popad
return
.error_free:
cinvoke shmctl, [ebx+XShmSegmentInfo.ShmID], IPC_RMID, 0
.error:
stdcall FreeMem, esi
stc
jmp .finish
endp
body DestroyImage
begin
pushad
mov esi, [.ptrImage]
test esi, esi
jz .finish
lea eax, [esi+TImage.shminfo]
cinvoke XShmDetach, [hApplicationDisplay], eax
cinvoke XDestroyImage, [esi+TImage.ximage]
cinvoke shmdt, [esi+TImage.shminfo.Addr]
cinvoke shmctl, [esi+TImage.shminfo.ShmID], IPC_RMID, 0
stdcall FreeMem, esi
.finish:
popad
return
endp
;if used ___CheckCompletionEvent
;___CheckCompletionEvent:
;
;virtual at esp+4
; .display dd ?
; .pEvent dd ?
; .user dd ?
;end virtual
;
;; timeout
; stdcall GetTimestamp
; cmp eax, [.user]
; jbe #f
;
; DebugMsg "Timeout!"
;
; mov eax, 1
; retn
;
;##:
; mov eax, [.pEvent] ;.pEvent
; mov eax, [eax+XEvent.type]
;
; cmp eax, [ShmCompletionEvent]
; sete al
; movzx eax, al
; retn
;end if
body DrawImageRect
.event XEvent
rb 256
begin
pushad
mov esi, [.pImage]
test esi, esi
jz .exit
mov ebx, [esi+TImage.ximage]
; NOTE: is this necessary? it seems wasteful to create and destroy a GC
; repeatedly. Dunno, does this _have_ to be done here, _every_ time?
cinvoke XCreateGC, [hApplicationDisplay], [.where], 0, 0
mov edi, eax
; NOTE/BUG: The return ShmCompletionEvent will be suppressed due to a BadValue
; if the X/Y and width/height parameters given to us by caller exceed the
; geometry/range of the shmarea attached to .ximage
;
; the routine that calls us is .expose and it _is_ giving us bad values. it is
; passing us X/Y width/height related to an exposure event of the .where
; _window_ which we put in the call. The X server will compare these against
; the size of the shmarea of TImage.xmage and complain if we exceed the bounds
cinvoke XShmPutImage, [hApplicationDisplay], [.where], edi,
[esi+TImage.ximage], [.xSrc], [.ySrc], [.xDst], [.yDst],
[.width], [.height], TRUE
; NOTE/BUG: this code should _not_ be looping on XCheckTypedEvent because it
; disrupts the normal event processing. if we want to be "synchronous" on this
; we should loop on the main event dispatcher (ProcessSystemEvents) and let it
; dispatch to a callback we create. we can set a "pending" flag that our [not
; yet existent] dispatch routine can clear
; THIS CODE SOMETIMES CAUSES HANGS!
stdcall GetTimestamp
lea esi, [eax+20]
.loop:
lea eax, [.event]
cinvoke XCheckTypedEvent, [hApplicationDisplay], [ShmCompletionEvent],
eax
test eax, eax
jnz .finish
stdcall GetTimestamp
cmp eax, esi
jb .loop
.finish:
cinvoke XFreeGC, [hApplicationDisplay], edi
.exit:
popad
return
endp
Xext/shm.c The X server code that checks and processes the XShmPutImage call.
// FILE: Xext/shm.c
static int
ProcShmPutImage(ClientPtr client)
{
GCPtr pGC;
DrawablePtr pDraw;
long length;
ShmDescPtr shmdesc;
REQUEST(xShmPutImageReq);
REQUEST_SIZE_MATCH(xShmPutImageReq);
VALIDATE_DRAWABLE_AND_GC(stuff->drawable, pDraw, DixWriteAccess);
VERIFY_SHMPTR(stuff->shmseg, stuff->offset, FALSE, shmdesc, client);
// NOTE: value must be _exactly_ 0/1
if ((stuff->sendEvent != xTrue) && (stuff->sendEvent != xFalse))
return BadValue;
if (stuff->format == XYBitmap) {
if (stuff->depth != 1)
return BadMatch;
length = PixmapBytePad(stuff->totalWidth, 1);
}
else if (stuff->format == XYPixmap) {
if (pDraw->depth != stuff->depth)
return BadMatch;
length = PixmapBytePad(stuff->totalWidth, 1);
length *= stuff->depth;
}
else if (stuff->format == ZPixmap) {
if (pDraw->depth != stuff->depth)
return BadMatch;
length = PixmapBytePad(stuff->totalWidth, stuff->depth);
}
else {
client->errorValue = stuff->format;
return BadValue;
}
// NOTE/BUG: The following block is the "check parameters" code. If the
// given drawing parameters of the request (e.g. X, Y, width, height) [or
// combinations thereof] exceed the geometry/size of the shmarea, the
// BadValue error is being returned here and the code to send a return
// event will _not_ be executed. The bug isn't really here, it's on the
// client side, but it's the client side bug that causes the event to be
// suppressed
/*
* There's a potential integer overflow in this check:
* VERIFY_SHMSIZE(shmdesc, stuff->offset, length * stuff->totalHeight,
* client);
* the version below ought to avoid it
*/
if (stuff->totalHeight != 0 &&
length > (shmdesc->size - stuff->offset) / stuff->totalHeight) {
client->errorValue = stuff->totalWidth;
return BadValue;
}
if (stuff->srcX > stuff->totalWidth) {
client->errorValue = stuff->srcX;
return BadValue;
}
if (stuff->srcY > stuff->totalHeight) {
client->errorValue = stuff->srcY;
return BadValue;
}
if ((stuff->srcX + stuff->srcWidth) > stuff->totalWidth) {
client->errorValue = stuff->srcWidth;
return BadValue;
}
if ((stuff->srcY + stuff->srcHeight) > stuff->totalHeight) {
client->errorValue = stuff->srcHeight;
return BadValue;
}
// NOTE: this is where the drawing takes place
if ((((stuff->format == ZPixmap) && (stuff->srcX == 0)) ||
((stuff->format != ZPixmap) &&
(stuff->srcX < screenInfo.bitmapScanlinePad) &&
((stuff->format == XYBitmap) ||
((stuff->srcY == 0) &&
(stuff->srcHeight == stuff->totalHeight))))) &&
((stuff->srcX + stuff->srcWidth) == stuff->totalWidth))
(*pGC->ops->PutImage) (pDraw, pGC, stuff->depth,
stuff->dstX, stuff->dstY,
stuff->totalWidth, stuff->srcHeight,
stuff->srcX, stuff->format,
shmdesc->addr + stuff->offset +
(stuff->srcY * length));
else
doShmPutImage(pDraw, pGC, stuff->depth, stuff->format,
stuff->totalWidth, stuff->totalHeight,
stuff->srcX, stuff->srcY,
stuff->srcWidth, stuff->srcHeight,
stuff->dstX, stuff->dstY, shmdesc->addr + stuff->offset);
// NOTE: this is where the return event gets sent
if (stuff->sendEvent) {
xShmCompletionEvent ev = {
.type = ShmCompletionCode,
.drawable = stuff->drawable,
.minorEvent = X_ShmPutImage,
.majorEvent = ShmReqCode,
.shmseg = stuff->shmseg,
.offset = stuff->offset
};
WriteEventsToClient(client, 1, (xEvent *) &ev);
}
return Success;
}

Your source code will be the ultimate piece which we can analyse but since I understand Assembly very less, I will give you an answer on macro level. Exact answer is still unknown to me.
I case of too many events only it is creating this issue but not with normal occurrence of events which means you framework is running out of virtual memory or another frame of event is triggered before the previous one releases its memory. In this kind of situations you can do few things
Try to check if there is any memory leak or not.. After one frame of event is over, try to clean up the memory or end that frame object properly before triggering the new one.
You can also develop a mechanism to make the second frame wait for the first frame to get over. In C/C++ we do it using so many synchronization methods like Mutex or select system calls. If you design follows that kind of pattern, then you can do these
If anywhere you have the authority to change the allocated memory given to your window, try to increase it. Because one thing for sure (according to your explanation ) that this is some memory issue.
Replying on Edit 3 It looks like you are calling some method cinvoke. How it is internally handling the even is unknown to me. Why don't you implement it directly in C. I am sure for whatever target that you are working, you will get some Cross-compiler.

Related

Get device encryption support information

I want to detect the device encryption support in my program. This info is available in the System Information program. Please check out the screenshot below:
What kind of Win API functions are used/available to detect the device encryption support? What System Information program uses to detect it? I just need some information.
TL;DR: it uses undocumented functions from fveapi.dll (Windows Bitlocker Drive Encryption API). It seems to rely only on the TPM capabilities.
Note that I only spent like 15 mins on it, but I doubt I missed something crucial, althoug this might be possible.
A bit of Reverse engineering
Typed system information in search bar, saw it spawned msinfo32.exe. Put the binary in a disassembler. It uses a MUI file so I'll have to search for the strings in the MUI file and not the executable.
Searching Device Encryption Support leads to string ID 951 (0x3b7)
STRINGTABLE
LANGUAGE LANG_ENGLISH, SUBLANG_ENGLISH_US
{
951, "Device Encryption Support|%s"
Searching for the contant in the disassembler leads to a function named:
DeviceEncryptionDataPoints(struct CWMIHelper *, struct CPtrList *)
The load ing of the aforementioned string is almost right at the start:
.text:00000001400141E9 mov edx, 3B7h
.text:00000001400141EE lea rcx, [rsp+2C8h+var_280]
.text:00000001400141F3
.text:00000001400141F3 loc_1400141F3:
.text:00000001400141F3 ; try {
.text:00000001400141F3 call cs:__imp_?LoadStringW#CString##QEAAHI#Z ; CString::LoadStringW(uint)
So we are definitely in the right function.
It loads the fveapi.dll module:
.text:0000000140014269 xor edx, edx ; hFile
.text:000000014001426B mov r8d, 800h ; dwFlags
.text:0000000140014271 lea rcx, LibFileName ; "fveapi.dll"
.text:0000000140014278 call cs:__imp_LoadLibraryExW
Gets a pointer on FveQueryDeviceEncryptionSupport:
.text:00000001400142AB lea rdx, aFvequerydevice ; "FveQueryDeviceEncryptionSupport"
.text:00000001400142B2 mov rcx, rdi ; hModule
.text:00000001400142B5 call cs:__imp_GetProcAddress
And immediately calls the function (this is a protected CFG call, but it's here):
.text:00000001400142CA mov [rsp+2C8h+var_254], rbx
.text:00000001400142CF mov [rsp+2C8h+var_260], 14h
.text:00000001400142D7 mov [rsp+2C8h+var_25C], 1
.text:00000001400142DF mov [rsp+2C8h+var_258], 1
.text:00000001400142E7 lea rcx, [rsp+2C8h+var_260]
.text:00000001400142EC call cs:__guard_dispatch_icall_fptr
Return value
If the function fails:
.text:00000001400142EC call cs:__guard_dispatch_icall_fptr
.text:00000001400142F2 mov esi, eax
.text:00000001400142F4 test eax, eax
.text:00000001400142F6 js loc_1400143F0 ; check failure
We land here:
.text:00000001400143F7 mov edx, 2FFh
.text:00000001400143FC lea rcx, [rsp+2C8h+var_288]
.text:0000000140014401 call cs:__imp_?LoadStringW#CString##QEAAHI#Z ; CString::LoadStringW(uint)
The string 0x2FF (767) is:
767, "Elevation Required to View"
If the call succeed, the code checks one of the parameter which is definitly an out parameter:
.text:00000001400142EC call cs:__guard_dispatch_icall_fptr
.text:00000001400142F2 mov esi, eax
.text:00000001400142F4 test eax, eax
.text:00000001400142F6 js loc_1400143F0
.text:00000001400142FC cmp dword ptr [rsp+2C8h+var_254], ebx ; rbx = 0
.text:0000000140014300 jnz short loc_14001431D
.text:0000000140014302 mov edx, 3B8h
.text:0000000140014307 lea rcx, [rsp+2C8h+var_288]
.text:000000014001430C call cs:__imp_?LoadStringW#CString##QEAAHI#Z ; CString::LoadStringW(uint)
If it's 0, the string 0x3b8 (952) is used:
952, "Meets prerequisites"
Otherwise various failure functions are called.
Failure
In case of a failure, the UpdateDeviceEncryptionStateFailureString function is called:
.text:0000000140014325 lea r9, [rsp+2C8h+var_294] ; int *
.text:000000014001432A lea r8, [rsp+2C8h+var_290] ; int *
.text:000000014001432F mov edx, 3C1h ; unsigned int
.text:0000000140014334 lea rcx, [rsp+2C8h+var_288] ; struct CString *
.text:0000000140014339 call ?UpdateDeviceEncryptionStateFailureString##YAXPEAVCString##IPEAH1#Z ; UpdateDeviceEncryptionStateFailureString(CString *,uint,int *,int *)
Its main goal is to fetch some string from the resource file.
One that stands out is 0x3b9:
.text:0000000140013A37 mov edx, 3B9h
.text:0000000140013A3C mov rcx, rbx
.text:0000000140013A3F call cs:__imp_?LoadStringW#CString##QEAAHI#Z ; CString::LoadStringW(uint)
953, "Reasons for failed automatic device encryption"
Which is the case for me since I don't have a TPM.
Other Functions
All of the other functions called from the DeviceEncryptionDataPoints (at least to get the needed results) are all from the fveapi.dll.
There are a lot in a function called PerformIndividualHardwareTests(HINSTANCE hModule, struct CString *, int *, int *):
.text:0000000140013AEF lea rdx, aNgscbcheckisao ; "NgscbCheckIsAOACDevice"
.text:0000000140013AF6 mov [rbp+var_1F], 0
.text:0000000140013AFA mov rdi, r9
.text:0000000140013AFD mov [rbp+var_20], 0
.text:0000000140013B01 mov rsi, r8
.text:0000000140013B04 mov [rbp+var_1E], 0
.text:0000000140013B08 mov rbx, rcx
.text:0000000140013B0B call cs:__imp_GetProcAddress
.text:0000000140013B12 nop dword ptr [rax+rax+00h]
.text:0000000140013B17 mov r12, rax
.text:0000000140013B1A test rax, rax
.text:0000000140013B1D jz loc_140013CB9
.text:0000000140013B23 lea rdx, aNgscbcheckishs ; "NgscbCheckIsHSTIVerified"
.text:0000000140013B2A mov rcx, rbx ; hModule
.text:0000000140013B2D call cs:__imp_GetProcAddress
.text:0000000140013B34 nop dword ptr [rax+rax+00h]
.text:0000000140013B39 mov r15, rax
.text:0000000140013B3C test rax, rax
.text:0000000140013B3F jz loc_140013CB9
.text:0000000140013B45 lea rdx, aNgscbcheckhsti ; "NgscbCheckHSTIPrerequisitesVerified"
.text:0000000140013B4C mov rcx, rbx ; hModule
.text:0000000140013B4F call cs:__imp_GetProcAddress
.text:0000000140013B56 nop dword ptr [rax+rax+00h]
.text:0000000140013B5B mov r13, rax
.text:0000000140013B5E test rax, rax
.text:0000000140013B61 jz loc_140013CB9
.text:0000000140013B67 lea rdx, aNgscbcheckdmas ; "NgscbCheckDmaSecurity"
.text:0000000140013B6E mov rcx, rbx ; hModule
.text:0000000140013B71 call cs:__imp_GetProcAddress
There's also a registry key checked SYSTEM\CurrentControlSet\Control\BitLocker\AutoDE\HSTI:
.text:0000000140013C10 lea r8, ?NGSCB_AUTODE_HSTI_REQUIRED##3QBGB ; "HSTIVerificationRequired"
.text:0000000140013C17 mov [rsp+60h+pcbData], rax ; pcbData
.text:0000000140013C1C lea rdx, ?NGSCB_AUTODE_HSTI_PREREQS##3QBGB ; "SYSTEM\\CurrentControlSet\\Control\\Bit"...
.text:0000000140013C23 lea rax, [rbp+var_1C]
.text:0000000140013C27 mov r9d, 10h ; dwFlags
.text:0000000140013C2D mov [rsp+60h+pvData], rax ; pvData
.text:0000000140013C32 mov rcx, 0FFFFFFFF80000002h ; hkey
.text:0000000140013C39 and [rsp+60h+var_40], 0
.text:0000000140013C3F call cs:__imp_RegGetValueW
and some other functions (NgscbCheckPreventDeviceEncryption, NgscbGetWinReConfiguration, FveCheckTpmCapability, ...) , once again, all from the fveapi.dll module.
So basically the checks are all based on functions from this DLL. It seems that none of them are documented (as far as I can see with a quick search).
I didn't find anything around in the DeviceEncryptionDataPoints caller (which is basically the main() function), since the next calls are dealing with checking the hypervisor capabilities.

How do I call a different subroutine using parameters passed into the current one (for use in recursion)?

I have two functions that take integers x and y read from input.
product returns x * y
power returns x ^ y, however it uses recursion and product to compute this. so x would be "base" and y is "exponent".
They called from C++:
int a, b, x, y;
a = product(x, y);
b = power(x, y);
and here is the asm. I got the product to work, however am having trouble with power because I am not sure of the syntax/method/convention to call product from it (and call itself for the recursion). EDIT: Recursion must be used.
global product
global power
section .text
product:
push ebp
mov ebp, esp
sub esp, 4
push edi
push esi
xor eax, eax
mov edi, [ebp+8]
mov esi, [ebp+12]
mov [ebp-4], edi
product_loop:
add [ebp-4], edi
mov eax, [ebp-4]
sub esi, 1
cmp esi, 1
jne product_loop
product_done:
pop esi
pop edi
mov esp, ebp
pop ebp
ret
power:
push ebp
mov ebp, esp
sub esp, 4
push edi
push esi
push ebx
xor eax, eax
mov edi, [ebp+8]
mov esi, [ebp+12]
;;;
check:
cmp esi, 1 ; if exp < 1
jl power_stop
recursion: ; else (PLEASE HELP!!!!!!!!)
; eax = call product (base, (power(base, exp-1))
power_stop:
mov eax, 1 ; return 1
power_done:
push ebx
pop esi
pop edi
mov esp, ebp
pop ebp
ret
EDIT: My solution!
power:
; Standard prologue
push ebp ; Save the old base pointer
mov ebp, esp ; Set new value of the base pointer
sub esp, 4 ; make room for 1 local variable result
push ebx ; this is exp-1
xor eax, eax ; Place zero in EAX. We will keep a running sum
mov eax, [ebp+12] ; exp
mov ebx, [ebp+8] ; base
cmp eax, 1 ; n >= 1
jge L1 ; if not, go do a recursive call
mov eax, 1 ; otherwise return 1
jmp L2
L1:
dec eax ; exp-1
push eax ; push argument 2: exp-1
push ebx ; push argument 1: base
call power ; do the call, result goes in eax: power(base, exp-1)
add esp, 8 ; get rid of arguments
push eax ; push argument 2: power(base, exponent-1)
push ebx ; push argument 1: base
call product ; product(base, power(base, exponent-1))
L2:
; Standard epilogue
pop ebx ; restore register
mov esp, ebp ; deallocate local variables
pop ebp ; Restore the callers base pointer.
ret ; Return to the caller.
You are using CDECL calling convention, so you have to first push the arguments in the stack in backward direction, then call the function and then clean the stack after the return.
push arg_last
push arg_first
call MyFunction
add esp, 8 ; the argument_count*argument_size
But here are some notes on your code:
Your function product does not return any value. Use mov eax, [ebp-4] immediately after product_done label.
Multiplication is much easy to be made by the instruction mul or imul. Using addition is the slowest possible way.
Computing the power by recursion is not the best idea. Use the following algorithm:
Y = 1;
if N=0 exit.
if N is odd -> Y = Y*x; N=N-1
if N is even -> Y = Y*Y; N=N/2
goto 2
Use SHR instruction in order to divide N by 2. Use test instrction in order to check odd/even number.
This way, you simply don't need to call product from power function.
If you're not sure how to write the assembly, you can generally write it in C++ and assemble it for clues - something like:
int power(int n, int exp)
{
return exp == 0 ? 1 :
exp == 1 ? n :
product(n, power(n, exp - 1));
}
Then you should just be able to use gcc -S or whatever your compiler's equivalent switch for assembly output is, or disassemble the machine code if you prefer.
For example, the function above, thrown in with int product(int x, int y) { return x * y; } and int main() { return product(3, 4); }, compiled with Microsoft's compiler ala cl /Fa power.cc:
; Listing generated by Microsoft (R) Optimizing Compiler Version 15.00.30729.01
TITLE C:\home\anthony\user\dev\power.cc
.686P
.XMM
include listing.inc
.model flat
INCLUDELIB LIBCMT
INCLUDELIB OLDNAMES
PUBLIC ?product##YAHHH#Z ; product
; Function compile flags: /Odtp
_TEXT SEGMENT
_x$ = 8 ; size = 4
_y$ = 12 ; size = 4
?product##YAHHH#Z PROC ; product
; File c:\home\anthony\user\dev\power.cc
; Line 1
push ebp
mov ebp, esp
mov eax, DWORD PTR _x$[ebp]
imul eax, DWORD PTR _y$[ebp]
pop ebp
ret 0
?product##YAHHH#Z ENDP ; product
_TEXT ENDS
PUBLIC ?power##YAHHH#Z ; power
; Function compile flags: /Odtp
_TEXT SEGMENT
tv73 = -8 ; size = 4
tv74 = -4 ; size = 4
_n$ = 8 ; size = 4
_exp$ = 12 ; size = 4
?power##YAHHH#Z PROC ; power
; Line 4
push ebp
mov ebp, esp
sub esp, 8
; Line 7
cmp DWORD PTR _exp$[ebp], 0
jne SHORT $LN5#power
mov DWORD PTR tv74[ebp], 1
jmp SHORT $LN6#power
$LN5#power:
cmp DWORD PTR _exp$[ebp], 1
jne SHORT $LN3#power
mov eax, DWORD PTR _n$[ebp]
mov DWORD PTR tv73[ebp], eax
jmp SHORT $LN4#power
$LN3#power:
mov ecx, DWORD PTR _exp$[ebp]
sub ecx, 1
push ecx
mov edx, DWORD PTR _n$[ebp]
push edx
call ?power##YAHHH#Z ; power
add esp, 8
push eax
mov eax, DWORD PTR _n$[ebp]
push eax
call ?product##YAHHH#Z ; product
add esp, 8
mov DWORD PTR tv73[ebp], eax
$LN4#power:
mov ecx, DWORD PTR tv73[ebp]
mov DWORD PTR tv74[ebp], ecx
$LN6#power:
mov eax, DWORD PTR tv74[ebp]
; Line 8
mov esp, ebp
pop ebp
ret 0
?power##YAHHH#Z ENDP ; power
_TEXT ENDS
PUBLIC _main
; Function compile flags: /Odtp
_TEXT SEGMENT
_main PROC
; Line 11
push ebp
mov ebp, esp
; Line 12
push 4
push 3
call ?power##YAHHH#Z ; power
add esp, 8
; Line 13
pop ebp
ret 0
_main ENDP
_TEXT ENDS
END
To walk you through this:
?power##YAHHH#Z PROC ; power
; Line 4
push ebp
mov ebp, esp
sub esp, 8
The above is the entry code for the power function - just adjusting the stack pointer to jump over the function arguments, which it will access below as _exp$[ebp] (that's exp) and _n$[ebp] (i.e. n).
; Line 7
cmp DWORD PTR _exp$[ebp], 0
jne SHORT $LN5#power
mov DWORD PTR tv74[ebp], 1
jmp SHORT $LN6#power
Basically, if exp is not equal to 0 we'll continue at label $LN5#power below, but if it is 0 then load 1 into the return value location on the stack at tv74[ebp] and jump to the function return instructions at $LN6#power.
$LN5#power:
cmp DWORD PTR _exp$[ebp], 1
jne SHORT $LN3#power
mov eax, DWORD PTR _n$[ebp]
mov DWORD PTR tv73[ebp], eax
jmp SHORT $LN4#power
Similar to the above - if exp is 1 then put n into eax and therefrom into the return value stack memory, then jump to the return instructions.
Now it starts to get interesting...
$LN3#power:
mov ecx, DWORD PTR _exp$[ebp]
sub ecx, 1
push ecx
Subtract 1 from exp and push in onto the stack...
mov edx, DWORD PTR _n$[ebp]
push edx
Also push n onto the stack...
call ?power##YAHHH#Z ; power
Recursively call the power function, which will use the two values pushes above.
add esp, 8
A stack adjustment after the function above returns.
push eax
Put the result of the recursive call - which the power return instructions leave in the eax register - onto the stack...
mov eax, DWORD PTR _n$[ebp]
push eax
Also push n onto the stack...
call ?product##YAHHH#Z ; product
Call the product function to multiple the value returned by the call to power above by n.
add esp, 8
mov DWORD PTR tv73[ebp], eax
Copy the result of product into a temporary address on the stack....
$LN4#power:
mov ecx, DWORD PTR tv73[ebp]
mov DWORD PTR tv74[ebp], ecx
Pick up the value from that tv73 temporary location and copy it into tv74...
$LN6#power:
mov eax, DWORD PTR tv74[ebp]
Finally, move the the product() result from tv74 into the eax register for convenient and fast access after the product call returns.
; Line 8
mov esp, ebp
pop ebp
ret 0
Clean up the stack and return.

In NASM, I want to compare an input keystroke based on its ASCII value

A simple program I am working on (for Homework) requires that I take a keystroke as input and return the categories it falls under (is it a printable charater, decimal, etc..)
I'm using cmp to compare the keystroke against the values of the maximum and/or minimum values in it's category (for example if the ASCII code of the keystroke is above 0x7F then it is a printable character)
However, there is obviously something not working in my comparison since no matter what, i.e. when I use the escape button as input, it is not printing "Control Key".
Could it be that keys need some more processing before they can be compared based on ASCII value?
Here is my code
segment .data
controlKey: db "Control Key", 10
controlLen: equ $-controlKey
printableKey: db "Printable", 10
printableLen: equ $-printableKey
decimalKey: db "Decimal", 10
decimalLen: equ $-decimalKey
segment .bss
key resb 2
segment .text
global main
main:
mov eax, 3 ; system call 3 to get input
mov ebx, 0 ; standart input device
mov ecx, key ; pointer to id
mov edx, 2 ; take in this many bytes
int 0x80
control: ; check if it's a control key
mov ebx, 31 ; highest control key
mov edx, key
cmp edx, ebx
jg printable
mov eax, 4
mov ebx, 1
mov ecx, controlKey
mov edx, controlLen
int 0x80
; jmp exit ; It's obviously not any of the other categories
printable: ; Tell that it's a printable symbol
mov eax, 4
mov ebx, 1
mov ecx, printableKey
mov edx, printableLen
int 0x80
decimal:
mov ebx, 30h ; smallest decimal ASCII
mov edx, key
cmp edx, ebx
jl uppercase
mov ebx, 39h ; test against 9
cmp edx, ebx
jg uppercase
mov eax, 4
mov ebx, 1
mov ecx, decimalKey
mov edx, decimalLen
int 0x80
uppercase:
lowercase:
mov eax, 4 ; system call 4 for output
mov ebx, 1 ; standard output device
mov ecx, key ; move the content into ecx
mov edx, 1 ; tell edx how many bytes
int 0x80 ;
exit:
mov eax, 1
xor ebx, ebx
int 0x80
The Escape key won't be read by your application, since it is - most probably - caught by the terminal that your application runs in. I can see that you're using the read syscall in your code, which is, of course, fine, but you should remember that this function only provides reading from a file descriptor, which doesn't necessarily have to contain all the control signals sent from the keyboard. The file descriptor (stdin) doesn't even have to come from the keyboard, since a file might be redirected to your process as standard input.
I don't know if there's a good way of achieving (capturing keystrokes, not the data that they represent - and this is what you're doing now) what you're trying to do just with system calls in Linux. You could try using some terminal controlling library, for example ncurses or termios, but I guess that isn't a part of your assignment.
I have done this a while back, here is a sample to show how to turn character echo on/off, and canonical mode on/off. When run, when you press a key, the keycode will be displayed on the screen, the program will exit once shift+q is pressed:
terminos.asm
ICANON equ 1<<1
ECHO equ 1<<3
sys_exit equ 1
sys_read equ 3
sys_write equ 4
stdin equ 0
stdout equ 1
global _start
SECTION .bss
lpBufIn resb 2
lpBufOut resb 2
termios resb 36
section .text
_start:
call echo_off
call canonical_off
.GetCode:
call GetKeyCode
movzx esi, byte[lpBufIn]
push esi
call PrintNum
pop esi
cmp esi, 81
jne .GetCode
call echo_on
call canonical_on
mov eax, sys_exit
xor ebx, ebx
int 80H
;~ #########################################
GetKeyCode:
mov eax, sys_read
mov ebx, stdin
mov ecx, lpBufIn
mov edx, 1
int 80h
ret
;~ #########################################
canonical_off:
call read_stdin_termios
; clear canonical bit in local mode flags
mov eax, ICANON
not eax
and [termios+12], eax
call write_stdin_termios
ret
;~ #########################################
echo_off:
call read_stdin_termios
; clear echo bit in local mode flags
mov eax, ECHO
not eax
and [termios+12], eax
call write_stdin_termios
ret
;~ #########################################
canonical_on:
call read_stdin_termios
; set canonical bit in local mode flags
or dword [termios+12], ICANON
call write_stdin_termios
ret
;~ #########################################
echo_on:
call read_stdin_termios
; set echo bit in local mode flags
or dword [termios+12], ECHO
call write_stdin_termios
ret
;~ #########################################
read_stdin_termios:
mov eax, 36h
mov ebx, stdin
mov ecx, 5401h
mov edx, termios
int 80h
ret
;~ #########################################
write_stdin_termios:
mov eax, 36h
mov ebx, stdin
mov ecx, 5402h
mov edx, termios
int 80h
ret
PrintNum:
push lpBufOut
push esi
call dwtoa
mov edi, lpBufOut
call GetStrlen
inc edx
mov ecx, lpBufOut
mov eax, sys_write
mov ebx, stdout
int 80H
ret
;~ #########################################
GetStrlen:
push ebx
xor ecx, ecx
not ecx
xor eax, eax
cld
repne scasb
mov byte [edi - 1], 10
not ecx
pop ebx
lea edx, [ecx - 1]
ret
;~ #########################################
dwtoa:
;~ number to convert = [ebp+8]
;~ pointer to buffer that receives number = [ebp+12]
push ebp
mov ebp, esp
push ebx
push esi
push edi
mov eax, [ebp + 8]
mov edi, [ebp + 12]
test eax, eax
jnz .sign
.zero:
mov word [edi], 30H
jmp .done
.sign:
jns .pos
mov byte [edi], "-"
neg eax
add edi, 1
.pos:
mov ecx, 3435973837
mov esi, edi
.doit:
mov ebx, eax
mul ecx
shr edx, 3
mov eax, edx
lea edx, [edx * 4 + edx]
add edx, edx
sub ebx, edx
add bl, "0"
mov [edi], bl
add edi, 1
cmp eax, 0
jg .doit
mov byte [edi], 0
.fixit:
sub edi, 1
mov al, [esi]
mov ah, [edi]
mov [edi], al
mov [esi], ah
add esi, 1
cmp esi, edi
jl .fixit
.done:
pop edi
pop esi
pop ebx
mov esp, ebp
pop ebp
ret 4 * 2
makefile
APP = terminos
$(APP): $(APP).o
ld -o $(APP) $(APP).o
$(APP).o: $(APP).asm
nasm -f elf $(APP).asm

Call not returning properly [X86_ASM]

This is C++ using x86 inline assembly [Intel syntax]
Function:
DWORD *Call ( size_t lArgs, ... ){
DWORD *_ret = new DWORD[lArgs];
__asm {
xor edx, edx
xor esi, esi
xor edi, edi
inc edx
start:
cmp edx, lArgs
je end
push eax
push edx
push esi
mov esi, 0x04
imul esi, edx
mov ecx, esi
add ecx, _ret
push ecx
call dword ptr[ebp+esi] //Doesn't return to the next instruction, returns to the caller of the parent function.
pop ecx
mov [ecx], eax
pop eax
pop edx
pop esi
inc edx
jmp start
end:
mov eax, _ret
ret
}
}
The purpose of this function is to call multiple functions/addresses without calling them individually.
Why I'm having you debug it?
I have to start school for the day, and I need to have it done by evening.
Thanks alot, iDomo
Thank you for a complete compile-able example, it makes solving problems much easier.
According to your Call function signature, when the stack frame is set up, the lArgs is at ebp+8 , and the pointers start at ebp+C. And you have a few other issues. Here's a corrected version with some push/pop optimizations and cleanup, tested on MSVC 2010 (16.00.40219.01) :
DWORD *Call ( size_t lArgs, ... ) {
DWORD *_ret = new DWORD[lArgs];
__asm {
xor edx, edx
xor esi, esi
xor edi, edi
inc edx
push esi
start:
cmp edx, lArgs
; since you started counting at 1 instead of 0
; you need to stop *after* reaching lArgs
ja end
push edx
; you're trying to call [ebp+0xC+edx*4-4]
; a simpler way of expressing that - 4*edx + 8
; (4*edx is the same as edx << 2)
mov esi, edx
shl esi, 2
add esi, 0x8
call dword ptr[ebp+esi]
; and here you want to write the return value
; (which, btw, your printfs don't produce, so you'll get garbage)
; into _ret[edx*4-4] , which equals ret[esi - 0xC]
add esi, _ret
sub esi, 0xC
mov [esi], eax
pop edx
inc edx
jmp start
end:
pop esi
mov eax, _ret
; ret ; let the compiler clean up, because it created a stack frame and allocated space for the _ret pointer
}
}
And don't forget to delete[] the memory returned from this function after you're done.
I notice that, before calling, you push EAX, EDX, ESI, ECX (in order), but don't pop in the reverse order after returning. If the first CALL returns properly, but subsequent ones don't, that could be the issue.

analyzing assembly code of some c++ code [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
I were asked to analyze an assembly code, which was generated from following c++ code in Visual studio IDE:
here is c++ code:
int plus(int a,int b);
int main()
{
cout<<plus(2,4);
getchar();
return 0;
}
int plus(int a,int b)
{
static int t=2;
return a+b+t;
}
And here is the assembly code (the reduced form):
_main PROC ; COMDAT
; 8 : {
push ebp
mov ebp, esp
sub esp, 192 ; 000000c0H
push ebx
push esi
push edi
lea edi, DWORD PTR [ebp-192]
mov ecx, 48 ; 00000030H
mov eax, -858993460 ; ccccccccH
rep stosd
; 9 : cout<<plus(2,4);
push 4
push 2
call ?plus##YAHHH#Z ; plus
add esp, 8
mov esi, esp
push eax
mov ecx, DWORD PTR __imp_?cout#std##3V?$basic_ostream#DU?$char_traits#D#std###1#A
call DWORD PTR __imp_??6?$basic_ostream#DU?$char_traits#D#std###std##QAEAAV01#H#Z
cmp esi, esp
call __RTC_CheckEsp
; 10 : getchar();
mov esi, esp
call DWORD PTR __imp__getchar
cmp esi, esp
call __RTC_CheckEsp
; 11 : return 0;
xor eax, eax
; 12 : }
pop edi
pop esi
pop ebx
add esp, 192 ; 000000c0H
cmp ebp, esp
call __RTC_CheckEsp
mov esp, ebp
pop ebp
ret 0
_main ENDP
; Function compile flags: /Odtp /RTCsu /ZI
_TEXT ENDS
; COMDAT ?plus##YAHHH#Z
_TEXT SEGMENT
_a$ = 8 ; size = 4
_b$ = 12 ; size = 4
?plus##YAHHH#Z PROC ; plus, COMDAT
; 15 : {
push ebp
mov ebp, esp
sub esp, 192 ; 000000c0H
push ebx
push esi
push edi
lea edi, DWORD PTR [ebp-192]
mov ecx, 48 ; 00000030H
mov eax, -858993460 ; ccccccccH
rep stosd
; 16 : static int t=2;
; 17 : return a+b+t;
mov eax, DWORD PTR _a$[ebp]
add eax, DWORD PTR _b$[ebp]
add eax, DWORD PTR ?t#?1??plus##YAHHH#Z#4HA
; 18 : }
pop edi
pop esi
pop ebx
mov esp, ebp
pop ebp
ret 0
?plus##YAHHH#Z ENDP ; plus
_TEXT ENDS
END
I have to find how does the code deal with stack and how variables stored and retrieved?
Regards.
push 4
push 2
call ?plus##YAHHH#Z
This pushes the values 4 and 2 onto the stack (reverse order to how you'd think of them in C, remember 2 is now on top of 4), then calls plus.
mov eax, DWORD PTR _a$[ebp]
add eax, DWORD PTR _b$[ebp]
add eax, DWORD PTR ?t#?1??plus##YAHHH#Z#4HA
pop edi
pop esi
pop ebx
mov esp, ebp
pop ebp
ret 0
I've ignored some stack fiddling at the top of plus, but this moves a from the stack into eax, adds b to it (which it gets from the stack) then adds t to it (I'm not familiar with MASM at all so I'm not actually sure where it gets t from). You can see the stack offsets of a and b have been stored into _a and _b as 8 and 12 further up the code. This is performed in %eax because this is where you stick the first return value of a function. There's some stack clearing and then the usual function epilogue before returning. The main code then pushes %eax onto the stack and calls the iostream stuff, which will pop it off and output it to screen.
Here is the complete tutorial:
http://www.codeproject.com/KB/cpp/reversedisasm.aspx
Please ask a specific question if you have ? Your original question is too broad.