I made a small debugging application specifically to modify a protected executable, when using the application I can hand off the debugging by suspending the process then calling DebugActiveProcessStop and attaching x64dbg to the process and resuming.
The target I'm debugging performs a checksum on itself in 488 inlined places, with my debugging application I breakpoint all of these with a byte pattern search, when the breakpoint hits I intend to place a hardware breakpoint after the checksum command and then undo all of my memory modifications. When the hardware breakpoint hits I read some pertinent registers, store them in a file and patch that checksum to always pass without performing the checksum calculations the next time it is called.
The issue arrives that when I call ContinueDebugEvent from my debugger to get to the hardware breakpoint the target always crashes after a few steps with an access violation(c0000005). While testing things I found that if I suspend the process instead of calling ContinueDebugEvent and then pass the debugging over to x64dbg and let it ContinueDebugEvent the target runs fine. I debugged x64dbg with x64dbg to see what it's doing differently and the call it makes has what appears to be identical parameters to my call to ContinueDebugEvent, just the processId, the threadID, and 10002 (DBG_CONTINUE)
Does anyone know how/why the target would crash when both of our debuggers are making the same function call?
Related
Currently Im developing Lauterbach CMM scripts to automate test cases for SPC58NG84
As part of Test case:
- Need to reset target system before and after test case.
- Need to read and wrte variable values from C code.
When I run test scripts I got error 'debug port problem' and in 'watch window' all variable values showing BUS ERROR.
Can you please let me know how to debug this issue?.
What are the reasons causing 'debug port problem'?
Error Message in Area winodw:
CO:2 error: CPU suddenly left debug mode (OSR=0x3C1)
CO:0 JTAGID=0x11110041
Warning: CO:1 Core currently in reset. Stopping core on activation.
CMM Script:
Test Pre condition: Reset target
Break.Delete
WAIT 100.ms
SYStem.Mode Down
SYStem.DETECT.CPU
SYStem.Mode Up
B:: Go
WAIT 500.ms
Test case Execution:
--Read and write Variables in software-----
Test Post condition: Reset target
Break
Break.Delete
WAIT 100.ms
SYStem.Mode Down
SYStem.Mode Up
B:: Go
WAIT 1000.ms
The error 'debug port problem' after the Break command usually means that the target application crashed so badly that core does not respond to the debugger's halt command anymore.
In order to debug the problem, make sure that your boot loader sets up the interrupt vector start address (IVPR) as early as possible, and also put branch-to-self instructions to all interrupt handler addresses, unless interrupt handler code already exists.
Once this is done, set program preakpoints to the interrupt handlers typically involved in crashes: machine check, data storage, instruction storage, program interrupt. Doing so should catch the core when the crash occurs, and the SRR0 (CSRR or MCSRR, depending on interrupt type) will show you at which address the problem occurred.
While writing an x86 WinAPI-based debugger, I've encountered a rare condition when the debuggee (which usually works well) suddenly terminates with EXCEPTION_ACCESS_VIOLATION after I attach to it with my native debugger. I can stably reproduce this on any applications it seems (tried on .NET Hello World-styled application and on notepad.exe on multiple Windows 10 machines).
Essentially I've written a simple WaitForDebugEvent loop:
CreateProcessW(L"C:\\Windows\\SYSWOW64\\notepad.exe", […], CREATE_SUSPENDED, […]);
DebugActiveProcess(processId);
DEBUG_EVENT debugEvent = {};
while (WaitForDebugEvent(&debugEvent, INFINITE)) {
switch (debugEvent.dwDebugEventCode) {
// log all the events
}
ContinueDebugEvent(debugEvent.dwProcessId, debugEvent.dwThreadId, DBG_EXCEPTION_NOT_HANDLED);
}
DebugActiveProcessStop(processId);
(here's the full listing: I won't paste it all here, because there's some additional non-essential boilerplate there; the MCVE is 136 lines long)
For the sake of an example, I'll just log all the debugger events and detect whether the debuggee is ready to "proceed normally" or it will terminate due to an exception.
Most of the time, my debugging session looks like that:
CREATE_PROCESS_DEBUG_EVENT (which reports creation of both the process and its initial thread)
LOAD_DLL_DEBUG_EVENT (I was never able to get the name for this DLL, but this is documented in MSDN)
CREATE_THREAD_DEBUG_EVENT (which, I suspect, is a thread injected by debugger)
LOAD_DLL_DEBUG_EVENT […] — after this, many DLLs get loaded into the target process and everything looks okay, the process works as intended
But sometimes (in about 1.5% of all runs), the event sequence changes:
CREATE_PROCESS_DEBUG_EVENT
LOAD_DLL_DEBUG_EVENT
CREATE_THREAD_DEBUG_EVENT
EXCEPTION_DEBUG_EVENT: EXCEPTION_ACCESS_VIOLATION (which I never was able to gather details for: it reports a DEP violation, and the address is empty)
After that, I cannot proceed with debugging, because my debuggee is in exception state and will terminate soon. I was never able to catch notepad.exe crash without my debugger attached (and I doubt it is that bad and will crash for no reason), so I suspect that my debugger causes these exceptions.
One bizarre detail is that I could "fix" the situation by calling Sleep(1) immediately after WaitForDebugEvent. So, this is possibly some sort of race condition, but race condition between what? Between the debugger thread and other threads in the debuggee? Is it a thing? How are we supposed to debug other applications, then? How could actual debuggers work if it is a thing?
I couldn't reproduce the issue with the same code compiled for x64 CPU (and debugging an x64 process).
What could actually cause this erroneous behavior? I've carefully read the documentation about the API functions I call, and checked some other debugger examples online, but still wasn't able to find what's wrong with my debugger: it looks like I follow all the right conventions.
I have tried to debug my debuggee with WinDBG while it is still paused in my debugger, but had no luck doing that. First of all, it's difficult to attach to the debuggee with another debugger (WinDBG only allows to use non-intrusive mode, which is less functional it seems?), and the call stacks for the process' threads aren't usually meaningful.
Steps to reproduce
Checkout this repository, compile with MSVC and then execute in cmd:
Debug\NetRuntimeWaiter.exe > log.txt
It is important to redirect output to the log file and not show it in the terminal: without that, timings for the log writer get changed, and the issue won't reproduce (due to a possible race condition I mentioned earlier?).
Usually the program will start and terminate 1000 notepads in about 10 seconds, and 10-15 of 1000 invocations will hold the error condition (i.e. EXCEPTION_ACCESS_VIOLATION).
the DebugActiveProcess (and undocumented DbgUiDebugActiveProcess which is internally called by DebugActiveProcess) have serious design problem: after calling NtDebugActiveProcess it create remote thread in the target process, via DbgUiIssueRemoteBreakin call - as result new thread in target process is created - DbgUiRemoteBreakin - this thread call DbgBreakPoint and then RtlExitUserThread
all this not documented and explained, only this note from DebugActiveProcess:
After all of this is done, the system resumes all threads in the
process. When the first thread in the process resumes, it executes a
breakpoint instruction that causes an EXCEPTION_DEBUG_EVENT
debugging event to be sent to the debugger.
of course this is wrong. why is DbgUiRemoteBreakin first (??) thread ? and which thread resume first undefined. why not exactly write - we create additional (but not first) thread in process ? and this thread execute breakpoint.
however, when process already running - create this additional thread not create problems. but in case we create process in suspended state, and then just call DebugActiveProcess - the DbgUiRemoteBreakin really became first executing thread in process and process initialization was done on this thread, instead of created first thread. on xp this always lead to fail process initialize at connect to csrss phase. (csrss wait connect to it only on first created thread in process). on later systems this is fixed and process can execute as usual. but can and not, because thread on which it was initialized is exit. it can cause subtle problems.
solution here - not use DebugActiveProcess but NtDebugActiveProcess in it place.
the debug object we can create or via DbgUiConnectToDbg() and then get it via DbgUiGetThreadDebugObject() (system store debug object in thread TEB) or direct by call NtCreateDebugObject
also if we create debuggee process from another process(B) we can do next:
duplicate debug object from debugger process to this B process
call DbgUiSetThreadDebugObject(hDdg) just before call
CreateProcessW with DEBUG_ONLY_THIS_PROCESS or DEBUG_PROCESS
system will be use DbgUiGetThreadDebugObject() for get debug object
from your thread and pass it to low level process create api
remove debug object from your thread via
DbgUiSetThreadDebugObject(0)
really no matter who is create process with debug object. matter who is handle events posted to this debug object.
all undocumented api definitions you can take from ntdbg.h and then link with ntdll.lib or ntdllp.lib
I have a network software that I need to debug. It forks at multiple places and I need to debug one particular function handling one particular request.
Is there any way to setup a global breakpoint that would be caught even when it is in an inferior process?
I cannot use follow-fork-mode child because this will follow the first request, not the one I need to debug.
One way to do this is to have gdb remain attached to all the processes. Then you would set your breakpoint and run the program as usual; the breakpoint would fire in any sub-process that happened to hit that location. You can use breakpoint conditions to try to reduce the number of hits.
To put gdb into multi-inferior mode, I use this:
set detach-on-fork off
set non-stop on
set pagination off
Depending on your version of gdb, you might also need set target-async on.
This mode can be a bit peculiar to work in. For example, when one thread stops, the other keep going. Also, breakpoint stops are reported, but not always obvious; and I think gdb doesn't immediately switch to the stopping thread (this may have changed in gdb git, I forget).
How are SW breakpoints handled (conceptually) by gdb stub or server (I assume client stub and server handle them in pretty much same way)?
I'm interested in a 'bare metal' target where the gdb stub/server runs, and both breakpoints and single stepping use software interrupts.
My actual questions:
When a breakpoint is hit, how is the stored instruction run so that the breakpoint can be 're-installed' and the (saved) machine status (including register contents) is not changed from the moment of hitting the breakpoint?
=>When is the breakpoint re-installed and how? Between breakpoint hit and entering the command interpreter, or during the next single step or coninue?
Also how does single-stepping over breakpoint work such that the original non-breakpoint instruction gets executed, and the breakpoint still remains there after being single-stepped over?
[edit]
Forgot: the document "GDB Internals" seems to be missing that info - and actually the whole subchapter about single stepping in the "Algorithms" chapter.
[edit2]
Ah, I seem to need stronger glasses: The 'Internals'-manual says:
"When the user says to continue, GDB will restore the original instruction, single-step, re-insert the trap, and continue on."
The single stepping over breakpoint, however, is still open question.
The single stepping over breakpoint, however, is still open question.
It's done exactly the same way as continue, except for the last step ("and continue on"). That is:
Process stops. GDB "looks around", discovers that $ip points to one of its breakpoints.
User issues continue, next, step or stepi command.
Restore original instruction (i.e. remove the breakpoint)
Single-step process
Re-insert breakpoint
Continue (this is done for continue but not for next, step or stepi).
For stepi, return control to the user (we are already at the next instruction due to step 4 above). For next, continue single-stepping until we reach a line in source that is not the same line we were on at step 1 above.
I'm currently testing an application that my company wrote. One of the scenarios was to see what happens to the system state if that application was to crash. Is there an application out there that could force crash my application? I'd rather not write a crash into the code itself (ie. null pointer dereference). Using the task manager to kill the process doesn't yield the same results.
On Windows you can attach WinDbg to a process, corrupt some register or memory and detach. For instance you can set instruction pointer to 0 for some active application thread.
windbg -pn notepad.exe
Right after attach, current thread is set to debug thread, so you need to change to app thread to make it crash with RIP register update
0:008> ~0s
0:000> rip=0
0:000> qd
Assuming Windows, see Application Verifier.
It can do fault injection (Low Resource Simulation) that makes various API calls fail, at configurable rates. E.g. Heap allocations, Virtual Alloc, WaitForXxx, Registry APIs, Filesystem APIs, and more.
You can even specify a grace period (in milliseconds) when no faults will be injected during startup.
The best way is to call RaiseException API from windows.h
RaiseException(0x0000DEAD,0,0,0);
Or you can do a runtime linking to KeBugCheckEx() from ntoskrnl.exe and call it in your code.
Example:
#include <windows.h>
#include <iostream>
using namespace std;
int main()
{
HINSTANCE h = LoadLibrary("ntoskrnl.exe");
cout<<h<<endl;
void* a;
a = (void*) GetProcAddress(h,"KeBugCheckEx");
int(*KeBugCheckEx)(ULONG,ULONG_PTR,ULONG_PTR,ULONG_PTR,ULONG_PTR);
KeBugCheckEx = (int(*)(ULONG,ULONG_PTR,ULONG_PTR,ULONG_PTR,ULONG_PTR))a;
cout << a;
KeBugCheckEx(0,0,0,0,0); //crash in module ntoskrnl.exe means that call success!
}
You can use the winapiexec tool for that:
winapiexec64.exe CreateRemoteThread ( OpenProcess 0x1F0FFF 0 1234 ) 0 0 0xDEAD 0 0 0
Replace 1234 with the process id and run the command, the process will crash.
You haven't stated which OS you're running on but, if it's Linux (or another UNIX-like system), you can just kill -9 your process. This signal can't be caught and will result in the rug being pulled out from under your process pretty quickly.
If you're not on a UNIX-like system, I can't help you, sorry, but you may find some useful information here (look for "taskkill").
If the system runs on UNIX/Linux you can send it a signal: SIGQUIT should produce a core-dump, you can also send it SIGSEGV if you want to test it getting a "segmentation fault". Those are signal 3 and 11 respectively.
If the system is Windows I do not know a way to raise a signal in a different application but if you can modify the application to handle a specific Windows message number that will call raise() you can emulate that. raise() causes the signal to be raised without actually having to write code that performs an illegal action. You can then post a message to the application which will have the handler that raises this signal.
You could override the global new operator. Then, you can use a counter and at a specific value you perform a null pointer dereference to force your application to crash. By simply changing the value of when to perform the dereference you can easily vary the time of crash.
Where is this "system state" defined? If this were unix, you could send a signal 9 to the process...
If you really needed to, you could share all the application memory with another process (or thread), and have that thread randomly write random data some unfortunate memory location - I think NASA did this for some of their space projects, but I really couldn't give a reference.
The real question is why you want to do this - what are you /really/ testing?
If this is, for example, some program that controls some medical service that prescribes drugs... Unit test that service instead, analyse the API, and look for flaws.
Make a bufferoverflow yourself.
#include <string.h>
void doSomething(char *Overflow)
{
char Buffer[1];
strcpy(Buffer, Overflow);
}
int main()
{
doSomething("Muhaha");
}
And your program will crash
An alternative would be to run the application in a good debugger, set a breakpoint to a particular line of code, and viola, your application has "crashed". Now, this might not cause all your threads to stop running, depending on the debugger being used. Alternatively, you could run the application in the debugger, and simply "stop" the application after a time.
This doesn't neccessarily result in a crash with the kernel killing the application (and possibly dumping core) but it would probably do what you want regardless.
Call abort() function from your code. Other programs can't reliably "crash" your program - they have their own process context which is isolated from your program's context. You could use something like TerminateProcess() in Windows API or another platform-specific function but that would be more or less the same as using Task Manager.