I have a (complex) multithread-application that produces a crash with some specific data. The crash happens at random positions and it is obvious the position where the crash happens is not the reason.
So at the moment I assume an illegal memory access where data are overwritten by one thread or something like that.
Unfortunately it happens with some larger dataset only, so there is no way to strip down the whole thing to something simple. So to find the root cause of the problem I tried the Intel Inspector 2019 - but it is unusable slow. While the application typically shows the issue after about 15 minutes of intensive calculation with these specific data, with that inspector running I would need to wait more than one day - and it is unsure if the crash happens because the Inspector changes the whole timing dramatically.
So my question: does anybody have a better idea/a better tool to track down the reason for such illegal memory accesses?
Andy idea is welcome!
You can use Application Verifier. select Heaps(with Full) under Basics option and run your program under WinDbg. You can find the exact position where the crash happens.
Notice: it will be slow but it will help.
Related
I have a .net Core web service which seems to slowly increase its cpu usage.
meaning at the first day it won't go past 10%, the second day it can go up to 20% and so on.
Using the TOP command in linux, all my webservices seems to sometime be shown there (probably when a request is made) and afterward disappear.
This specific process after running for a while just stays there constantly consuming cpu even when no request has been made.
the API still working fine, it seems like there are some threads that just keeps hanging and consuming cpu. last time I checked I had 5 threads that consumed 3-4% cpu and didn't die for some reason.
My guess is that in some specific scenario a thread just stays alive consuming cpu.
The app runs on ubuntu machine, my first step was trying to create a dump file with ProcDump so I can analyze those threads and maybe find where they are hanging.
ProcDump generates a huge 21gb file, which trying to analyze with lldb throws out of memory exception. even tried transferring it to a windows machine to debug with windbg , no help there as it couldn't open the file.
As there is no specific exception or anything I can't really share any piece of code as I have no idea where the issue is... just kind a hoping for some suggestion that might help me get to a solution or at least understand where the problem is.
Thanks a lot for reading, cheers
You could try using something like jetBrains’ DotMemory, they also have a fairly high level but helpful guide https://www.jetbrains.com/help/dotmemory/How_to_Find_a_Memory_Leak.html it also worth checking your startup file and double checking the services you’ve registered are used in the correct way ie not added as scoped when they should be transient or even a singleton etc
so iv'e been at it for a while.
Eventually found out that my problem was with HttpClient
Probably some bad mix of static class and creating new instances of HttpClient that causes the issue Iv'e explained above.
Solved it by utilizing HttpClientFactory as explained here -
https://learn.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-2.1
Lesson learned :)
A little late but Procdump for Linux just added .NET Core 3 support that generates much more managable sized core dumps. It automatically detects if the target process is .NET Core and does the right thing (i.e., no need to specify switches).
I have a C++ DLL that is used for authentication that gets loaded by a Windows service for every login.
In that DLL I use the Windows ADSI function ADsOpenObject() to get a user object from Active Directory.
HRESULT hr = ADsOpenObject(L"LDAP://rootDSE",
L"username",
L"password",
m_dwADSFlags,
IID_IDirectorySearch,
(void**)&m_DSSearch);
Generally this works since years. But currently I get the error code
-2147024882 (0x8007000E)
which is OUT_OF_MEMORY. When I restart the service that is using my DLL then it runs fine for weeks but then the errors start occuring.
Now I can't find what is out of memory. The task scheduler looks fine and free memory is plenty.
What can I do to fix that?
which is OUT_OF_MEMORY.
It is E_OUTOFMEMORY, a COM error code. The description isn't very helpful, this error code tends to be returned for any "out of resources" errors by Microsoft code, not just memory. Could by an internal limit being reached, could be a winapi call that fails.
And it is not necessarily limited to the direct software involved. A mishaving device driver that leaks kernel pool memory can be the indirect source of the mishap for example.
You'll be lucky if you can find something back in the Application event log, look both in the machine that reports the error as well as the domain server. Task Manager might give a clue, add the Handles, GDI Objects, USER Object, Commit size, Page pool and NP Pool columns. Pretty hard to give specific advice beyond this. It is no doubt a leak, quacks loudly like one when you have to restart a machine periodically to recover. Good luck hunting it down.
Are you calling Release on m_DSSearch? Also if you are searching you need to call CloseSearchHandle or AbandonSearch. If you are not doing either of those, you could be slowly leaking memory.
Your process could fragment the heap to a point so that ADsOpenObject can't find a consecutive piece of memory that is large enough.
You can use VMMap to profile your memory usage:
http://technet.microsoft.com/en-us/sysinternals/dd535533
You could try enabling low fragmentation heap:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366750(v=vs.85).aspx
If it is now starting to appear and it wasn't earlier, I would suppose one of two possibilities: it has to do with the time (more specifically the year, a milleniumbug of some kind). Another option would be a problem with 32/64 bit architecture.
Now mind that I don't program C++. But I do know a bit about MS errors (I have worked on a MS helpdesk...)
We have a problem for which I am looking for a direction. We have an old MFC based application, and due to various reasons, the application is crashing sometimes intermittently in some weird scenarios. And even the customers who are using our application and getting these crashes are finding difficulty in identifying the pattern of crash. So, I had a thought that, if we can have a mechanism by which we can generate a log whenever the application crashes. Like for example, the call stack or any other information in that log. I know,, we can use the crash dump in this case, but then I feel like having a log is a better option. So any help or information in this regard would be really helpful.
Thank you.
You can find a good implementation of crash reporter in the link here.
When you compile your release build, make sure that both DEBUG and /MAP are enabled. Store your binaries together with your .map files and let your customer run this version until a crash is produced. In the Event Viewer you will then find a crash log with a crash offset. Then debug step into your code (F10) and use the crash offset together with some nifty tricks and tricks to jump (set the EIP register to... well, you have to google this a bit) to the location where the crash occurred. You should now be able to find the error!
I am trying to figure out a "Failed to resume in time" problem. In one of our testers devices (which is an iPhone 4S with the latest OS) it happens very frequently, whereas in my own device it doesn't seem to happen at all.
Anyway, I got a few crashlogs. I am unable to trace the root of the cause though. I understand that the issue might be
1.When a process is holding up the main thread for too long.
2.When there is a memory issue.
I don't think the memory is much of an issue since it seems to happen when the user leaves the main menu and comes back. Nothing much is happening in the main menu so it probably is a task that runs too long.
Here is an excerpt from the crash log:
Can somebody help me or guide me on who I can trace the cause of the issue? Is there anyway to turn off the watchdog timer(probably not huh?) Also, what does highlighted thread refer to?
I have already checked my applicationDidBecomeActive & applicationWillEnterForeground to make sure there is nothing going on there.
To my knowledge there are no synchronous calls being made at this point. Does Reachability use synchronous calls to check for internet? How can I check for that?
I am not making any large data transfers upon resume.
I notice that GameCenter automatically logs in or check for log in upon resuming your app. Is there anyway to prevent this? Could this possibly cause a time out issue?
I tried doing a time profile, but I am not able to understand how to use it to analyze. If you can provide a good resource for that, that would be amazing.
Thanks!!!
You're currently in "trying to find the issue mode". You should switch to "try to find out how much of an issue this really is" mode.
So go find another 4S (actually as many as you can) to rule out that it's a device-specific issue. If it happens on all 4S it should be easier to pinpoint. If not, have someone else look over it, discuss possible causes. The peer programming approach often helps when you're stuck in a dead-end situation.
If the issue is only on that one device, you might want to check if it's broken (or "jailbroken") or might simply need a hard reboot (hold power and home for 10+ seconds).
If it only happens on some devices but not all, try to find what they have in common. This could be language/locale, or dictation, practically any kind of setting the user might have changed. If necessary, write a logger that logs as many settings as possible to your (web) server so you can compare settings one-by-one and quickly discard those that aren't in synch.
If only very few devices are affected, you could also ignore the issue and hope that additional crash logs from users will reveal the key to the issue.
Finally, there's always the option to disable suspend on terminate and instead terminate the app when the home button is pressed (as it was pre iOS 4). Unless of course the app has to run in background.
I am trying to make a program that records a whole bunch of things periodically.
The specific reason is that if it bluescreens, a developer can go back and check a lot of the environment and see what was going on around that time.
My problem, is their a way to cause a bluescreen?
Maybe with a windowsAPI call (ZeroMemory maybe?).
Anywhoo, if you can think of a way to cause a bluescreen on call I would be thankful.
The computer I am testing this on is designed to take stuff like this haha.
by the way the language I am using is C\C++.
Thank you
You can configure a machine to crash on a keystroke (Ctrl-ScrollLock)
http://support.microsoft.com/kb/244139
Since it appears that there are times when that won't work on some systems with USB keyboards, you can also get the Debugging Tools for Windows, install the kernel debugger, and use the ".crash" command to force a bugcheck.
http://www.microsoft.com/whdc/devtools/debugging/default.mspx
In order to cause a BSOD, a driver running in kernel mode needs to cause it. If you really want to do this, you can write a driver which exposes KeBugCheck to usermode.
http://msdn.microsoft.com/en-us/library/ms801640.aspx
Thanks to Andrew below for pointing this utility out:
http://download.sysinternals.com/files/NotMyFault.zip
If you kill the csrss process you'll get a blue-screen rather quickly.
If you want to simulate a hard crash such as a bluescreen, you'd pretty much have to yank the power cord. NOT recommended.
In case of a crash, anything not saved to persistent storage will be lost. If you want to simulate a crash for purposes of logging, write a "kill switch" into your logger, which stops the logging. Now you can simulate a crash by killing the logging and making sure you have the data you would have wanted in case of an actual crash.
First of all, I would advise you to use a Virtual Machine to test this BSOD on. This will allow you to keep a backup just in case the BSOD does some damage to the system. Here's a tip on how to generate a BSOD simply by pressing CTRL+SCROLLLOCK+SCROLLLOCK.
Is there a Windows API to generate one? No, according to this article. Still, if you would call certain API's with invalid data, they could still cause a crash inside the kernel, which would result in your BSOD.
I'm not sure exactly what you'd be testing. Since your program runs periodically, surely it's enough to check that the information is being dumped at the frequency that you specify while the system is running? Are you checking that the information stays around after the blue screen? Depending on how you are dumping it (and whether you are flushing buffers), this may not be necessary.
If you dont want to write code (driver, IOCTL...) you can use DiskCryptor. Note that no disk encrypting is need.
Just need to install the driver:
dcinst.exe -setup
And then generate a bsod using the DC console:
dccon.exe -bsod
Run process as critic and exit http://waleedassar.blogspot.co.uk/2012/03/rtlsetprocessiscritical.html