I am trying to figure out a "Failed to resume in time" problem. In one of our testers devices (which is an iPhone 4S with the latest OS) it happens very frequently, whereas in my own device it doesn't seem to happen at all.
Anyway, I got a few crashlogs. I am unable to trace the root of the cause though. I understand that the issue might be
1.When a process is holding up the main thread for too long.
2.When there is a memory issue.
I don't think the memory is much of an issue since it seems to happen when the user leaves the main menu and comes back. Nothing much is happening in the main menu so it probably is a task that runs too long.
Here is an excerpt from the crash log:
Can somebody help me or guide me on who I can trace the cause of the issue? Is there anyway to turn off the watchdog timer(probably not huh?) Also, what does highlighted thread refer to?
I have already checked my applicationDidBecomeActive & applicationWillEnterForeground to make sure there is nothing going on there.
To my knowledge there are no synchronous calls being made at this point. Does Reachability use synchronous calls to check for internet? How can I check for that?
I am not making any large data transfers upon resume.
I notice that GameCenter automatically logs in or check for log in upon resuming your app. Is there anyway to prevent this? Could this possibly cause a time out issue?
I tried doing a time profile, but I am not able to understand how to use it to analyze. If you can provide a good resource for that, that would be amazing.
Thanks!!!
You're currently in "trying to find the issue mode". You should switch to "try to find out how much of an issue this really is" mode.
So go find another 4S (actually as many as you can) to rule out that it's a device-specific issue. If it happens on all 4S it should be easier to pinpoint. If not, have someone else look over it, discuss possible causes. The peer programming approach often helps when you're stuck in a dead-end situation.
If the issue is only on that one device, you might want to check if it's broken (or "jailbroken") or might simply need a hard reboot (hold power and home for 10+ seconds).
If it only happens on some devices but not all, try to find what they have in common. This could be language/locale, or dictation, practically any kind of setting the user might have changed. If necessary, write a logger that logs as many settings as possible to your (web) server so you can compare settings one-by-one and quickly discard those that aren't in synch.
If only very few devices are affected, you could also ignore the issue and hope that additional crash logs from users will reveal the key to the issue.
Finally, there's always the option to disable suspend on terminate and instead terminate the app when the home button is pressed (as it was pre iOS 4). Unless of course the app has to run in background.
Related
I have a .net Core web service which seems to slowly increase its cpu usage.
meaning at the first day it won't go past 10%, the second day it can go up to 20% and so on.
Using the TOP command in linux, all my webservices seems to sometime be shown there (probably when a request is made) and afterward disappear.
This specific process after running for a while just stays there constantly consuming cpu even when no request has been made.
the API still working fine, it seems like there are some threads that just keeps hanging and consuming cpu. last time I checked I had 5 threads that consumed 3-4% cpu and didn't die for some reason.
My guess is that in some specific scenario a thread just stays alive consuming cpu.
The app runs on ubuntu machine, my first step was trying to create a dump file with ProcDump so I can analyze those threads and maybe find where they are hanging.
ProcDump generates a huge 21gb file, which trying to analyze with lldb throws out of memory exception. even tried transferring it to a windows machine to debug with windbg , no help there as it couldn't open the file.
As there is no specific exception or anything I can't really share any piece of code as I have no idea where the issue is... just kind a hoping for some suggestion that might help me get to a solution or at least understand where the problem is.
Thanks a lot for reading, cheers
You could try using something like jetBrains’ DotMemory, they also have a fairly high level but helpful guide https://www.jetbrains.com/help/dotmemory/How_to_Find_a_Memory_Leak.html it also worth checking your startup file and double checking the services you’ve registered are used in the correct way ie not added as scoped when they should be transient or even a singleton etc
so iv'e been at it for a while.
Eventually found out that my problem was with HttpClient
Probably some bad mix of static class and creating new instances of HttpClient that causes the issue Iv'e explained above.
Solved it by utilizing HttpClientFactory as explained here -
https://learn.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-2.1
Lesson learned :)
A little late but Procdump for Linux just added .NET Core 3 support that generates much more managable sized core dumps. It automatically detects if the target process is .NET Core and does the right thing (i.e., no need to specify switches).
Sometimes (in about 50% of runs), EnumDevices takes 5-10 seconds to return. Normally it is almost instant. I couldn't find any other reports of this kind of behaviour.
When things are this slow, it's ok to profile by watching stdout :) This:
std::cout << "A";
directInput8Interface->EnumDevices(DI8DEVCLASS_GAMECTRL, MyCallback, NULL, DIEDFL_ATTACHEDONLY);
std::cout << "C";
...
BOOL CALLBACK MyCallback(LPCDIDEVICEINSTANCE, LPVOID)
{
std::cout << "B";
return DIENUM_CONTINUE;
}
Seems to hang at a random point through enumerating devices - sometimes it'll be before the callback is called at all, sometimes after a couple, and sometimes it will be after the last call to it.
This is clearly a simplified chunk of code; I'm actually using the OIS input library ( http://sourceforge.net/projects/wgois/ ), so for context, please see the full source here:
http://wgois.svn.sourceforge.net/viewvc/wgois/ois/trunk/src/win32/Win32InputManager.cpp?revision=39&view=markup
There doesn't seem to be anything particularly fruity going on there though, but possibly something in their initialisation could be the cause - I don't know enough about DI8 to spot it.
Any ideas about why it could be so slow will be greatly appreciated!
EDIT:
I've managed to catch the hang in an etl trace file and analysed it in Windows Performance Analyzer. It looks like EnumDevices eventually calls through to DInput8.dll!fGetProductStringFromDevice, which calls HIDUSB.SYS!HumCallUSB, which calls KeWaitForSingleObject and waits. 9 times out of 10 (literally - there are 10 samples in the trace) this returns very quickly (324us each), with the readying callstack containing usbport.sys!USBPORT_Core_iCompleteDoneTransfer followed by HIDUSB.SYS!HumCallUsbComplete, which looks quite normal.
But 1 time in 10, this takes almost exactly 5 seconds to return. On the readying callstack is ntkrnlmp.exe!KiTimerExpiration instead of the HIDUSB.SYS function. I guess all this indicates that the HIDUSB.SYS driver is querying devices asynchronously with a timeout of 5 seconds, and sometimes it fails and hits this timeout.
I don't know whether this failure is associated with any one device in particular (I do have a few USB HIDs) or if it's random - it's hard to test because it doesn't always happen. Again, any information anyone can give me will be appreciated, though I don't hold out any hope for Microsoft fixing this any time soon given the odd situation DirectInput is in!
Perhaps I'll just have to start initialising input earlier, asynchronously, and accept that sometimes there'll be a 5 second delay before user input can happen.
I was running into this too, largely as an end user, but it's been annoying the hell out of me for years. I didn't realize it was this issue until I ran into it on an open source project and was able to debug it.
Turns out it was my USB Headphone DAC (The Objective DAC from Massdrop), it installs the driver: wdma_usb.inf_amd64_134cb113911feba4\wdma_usb.inf for Device Instance ID USB\VID_262A&PID_1048&MI_01\7&F217D4F&0&0001 and then shows up in Device Manager under Sound, video and game controllers as: ODAC-revB USB DAC and, under Human Interface Devices as: USB Input Device and HID-compliant consumer control device.
I have no idea what the HID entries do but... When they are enabled and this DAC is set as the Audio Output device both IDirectInput8_CreateDevice and EnumDevices are painfully slow. Disabling the "USB Input Device" entry seems to cause no negative effects and completely solves my issue.
Changing the Audio output from the DAC to anything else also weirdly solved the issue.
This was so bad that it made the Gamepad Configuration dialog joy.cpl unusable, hanging and eventually crashing.
I was wanting this to just be a comment but I don't have enough rep for it, and this is pretty much the only place on the internet that describes this problem though so hopefully this helps someone else one day!
I had the same issue. I have a Corsair K65 LUX RGB keyboard. I updated CUE and it seems to have fixed the issue
Got same issue when having my Corsair K55 Keyboard. Changing the keyboard of USB port fixes the issue for a while, but then it comes back later on. So it seems to be a buggy drivers issue.
As DaFox has pointed out, a likely cause appears to be certain device drivers being enabled. I contacted JDS Labs support (who sell one device which happens to install one such driver) and they kindly pointed out that the root cause is actually a bug within Windows (not the installed driver), and they actually provide the solution on their troubleshooting page. See Games hang or experience loading delays, which explicitly mentions VID_262. Disabling this driver fixes the issue without apparent side effects (under the condition that that is the only driver triggering the bug). As for what exactly is going wrong within Windows, here there be dragons.
So I guess the go-to solution (for users) is to scrape all the troubleshooting and FAQ pages for all devices which you have ever connected to your system and see if there is a mention of delays/lag caused by a driver.
As a software developer, you will probably want to benchmark the execution time of the affected code and kindly tell the user there is something wrong with their system configuration and where to look for how to fix it in case it is unreasonably long.
Same issue with Corsair K70 Keyboard.
Quickly reconnecting keyboard fixes this, until next time. Usually happens after some DirectInput devices removed from the system or go to sleep.
This has been plaguing me as a developer and my friend as a user for years. All games using DInput, SDL SDL_INIT_JOYSTICK or anything depending on that, took extremely long to initialize.
It was caused by a faulty driver of a DAC, and as pointed out by DaFox, disabling the corresponding USB Input Device resolved the issue. Although it's labeled with a different manufacturer name, the vendor IDs match.
The hardware ID of the device is USB\VID_262A&PID_9023&REV_0001&MI_00.
Same issue appears to happen with a Steelseries Apex 7 keyboard. Unplugging and plugging that keyboard back again got rid of 3 freezes (of 10 seconds each) while enumerating USB devices.
My application is deployed at customer sites, that I can not access, and has no internet connection.
There are complains that in several sites, once in a week or so, the application become unresponsive, so that the operators need to kill and restart it.
We were unable to observe it in our site.
Is there something I can do that may help me find the problem?
It is a VC2008 Win32 MFC applications.
The application is quite complex, and includes many threads, synchronization mechanisms, database access, HMI, communication channels...
Note: The custmer can send us log files.
Note: The application does not crash. It just hangs. Since I don't know what is the nature of the problem, I have no way to know programmatically that something went wrong (or do I?)
I have had great success with ADplus and WinDBG in the past. You may check it out. Especially check out the Hang mode in ADplus.
I would start with some questions - is the CPU hogged during these unresponsive times? Is there a specific process that's hogging it? (You can use PerfMon to get the answers). Depending on the answers I would probably proceed by taking a dump of the process at this stage (ProcDump by sysinternals is great for these purposes) and investigate it offline.
In similar situation on a non-windows platform we have the capability to gather system dumps. Get a thread dump of the entire system for off-site analysis. This enables us to find deadlocks quite easily. For slow problems rather than stop a single dump is not enough. Then we need a sequence of dumps, and some good luck.
Another, rather messier technique is to have enough trace, and enough fine-grained control of trace in the app. Then turn on some trace and hope to spot where the delays are happening.
My experience with finding bugs in installations on the other side of the planet shows three helpful techniques: Logging, logging, and logging.
What do those log files say your customers sent you? If they aren't detailed enough, send them a version that logs more. Use binary approximation to home in on the error.
To know where the process is hung is better to start with the stack trace at that instant.
Now since your program is installed remotely and you can't access it, you can write a monitoring program which can periodically check the stack of your program and log it. This information along with your logging mechanism will make things easier to identify and debug.
Since I am not a windows programmer, i don't know much about such tools availability in windows, however i think you need something similar to this http://www.codeproject.com/KB/threads/StackWalker.aspx
I'm looking for a way to write an application. I use Visual C++ 6.0.
I need to prevent the user from closing this process via task manager.
You can't do it.
Raymond Chen on why this is a bad idea.
You can make an unkillable process, but it won't be able to accomplish anything useful while it's unkillable. For example, one way to make a process unkillable is to have it make synchronous I/O requests to a driver that can never complete (for example, by deliberately writing a buggy driver). The kernel will not allow a process to terminate until the I/O requests finish.
So it's not quite true that you "can't do it" as some people are saying. But you wouldn't want to anyway.
That all depends on who shouldn't be able to kill that process. You usually have one interactively logged-on user. Running the process in that context will alow the user to kill it. It is her process so she can kill it, no surprise here.
If your user has limited privileges you can always start the process as another user. A user can't kill a process belonging to another user (except for the administrator), no surprise here as well.
You can also try to get your process running with Local System privileges where, I think not even an administrator could kill it (even though he could gain permission to do so, iirc).
In general, though, it's a terribly bad idea. Your process does not own the machine, the user does. The only unkillable process on a computer I know is the operating system and rightly so. You have to make sure that you can't hog resources (which can't be released because you're unkillable) and other malicious side-effects. Usually stuff like this isn't the domain of normal applications and they should stay away from that for a reason.
It's a Win32 FAQ for decades. See Google Groups and Und. boards for well-known methods.(hooking cs and others...)
Noobs who answer "You can't do it" know nothing to Win32 programming : you can do everything with Win32 api...
What I've learned from malware:
Create a process that spawns a dozen of itself
Each time you detect that one is missing (it was killed) spawn a dozen more.
Each one should be a unique process name so that a batch process could not easily kill all of them by name
Sequentially close and restart some of the processes to keep the pids changing which would also prevent a batch kill
Depends on the users permission. If you run the program as administrator a normal user will not have enough permissions to kill your process. If an administrator tries to kill the process he will in most cases succeed. If you really want someone not to kill you process you should take a look at windows system services and driver development. In any case, please be aware that if a user cannot kill a process he is stuck with it, even though it behaves abnormally duo to bugs! You will find a huge wealth of these kind of programs/examples on the legal! site rootkit.com. Please respect the user.
I just stumbled upon this post while trying to find a solution to my own (unintentional) unkillable process problem. Maybe my problem will be your solution.
Use jboss Web Native to install a service that will run a batch file (modify service.bat so that it invokes your own batch file)
In your own batch file, invoke a java process that performs whatever task you'd like to persist
Start the service. If you view the process in process explorer, the resulting tree will look like:
jbosssvc.exe -> cmd.exe -> java.exe
use taskkill from an administrative command prompt to kill cmd.exe. Jbosssvc.exe will terminate, and java.exe will be be an orphaned running process that (as far as I can tell) can't be killed. So far, I've tried with Taskmanager, process explorer (running as admin), and taskkill to no avail.
Disclaimer: There are very few instances where doing this is a good idea, as everyone else has said.
There's not a 100% foolproof method, but it should be possible to protect a process this way. Unfortunately, it would require more knowlegde of the Windows security system API than I have right now, but the principle is simple: Let the application run under a different (administrator) account and set the security properties of the process object to the maximum. (Denying all other users the right to close the process, thus only the special administrator account can close it.)
Set up a secondary service and make it run as a process guardian. It should have a lifeline to the protected application and when this lifeline gets cut (the application closes) then it should restart the process again. (This lifeline would be any kind of inter-process communications.)
There are still ways to kill such an unkillable process, though. But that does require knowledge that most users don't really know about, so about 85% of all users won't have a clue to stop your process.
Do keep in mind that there might be legal consequences to creating an application like this. For example, Sony created a rootkit application that installed itself automatically when people inserted a Sony music CD or game CD in their computer. This was part of their DRM solution. Unfortunately, it was quite hard to kill this application and was installed without any warnings to the users. Worse, it had a few weaknesses that would provide hackers with additional ways to get access to those systems and thus to get quite a few of them infected. Sony had to compensate quite a lot of people for damages and had to pay a large fine. (And then I won't even mention the consequences it had on their reputation.)
I would consider such an application to be legal only when you install it on your own computer. If you're planning to sell this application to others, you must tell those buyers how to kill the process, if need be. I know Symantec is doing something similar with their software, which is exactly why I don't use their software anymore. It's my computer, so I should be able to kill any process I like.
The oldest idea in the world, two processes that respawn each other?
I am trying to make a program that records a whole bunch of things periodically.
The specific reason is that if it bluescreens, a developer can go back and check a lot of the environment and see what was going on around that time.
My problem, is their a way to cause a bluescreen?
Maybe with a windowsAPI call (ZeroMemory maybe?).
Anywhoo, if you can think of a way to cause a bluescreen on call I would be thankful.
The computer I am testing this on is designed to take stuff like this haha.
by the way the language I am using is C\C++.
Thank you
You can configure a machine to crash on a keystroke (Ctrl-ScrollLock)
http://support.microsoft.com/kb/244139
Since it appears that there are times when that won't work on some systems with USB keyboards, you can also get the Debugging Tools for Windows, install the kernel debugger, and use the ".crash" command to force a bugcheck.
http://www.microsoft.com/whdc/devtools/debugging/default.mspx
In order to cause a BSOD, a driver running in kernel mode needs to cause it. If you really want to do this, you can write a driver which exposes KeBugCheck to usermode.
http://msdn.microsoft.com/en-us/library/ms801640.aspx
Thanks to Andrew below for pointing this utility out:
http://download.sysinternals.com/files/NotMyFault.zip
If you kill the csrss process you'll get a blue-screen rather quickly.
If you want to simulate a hard crash such as a bluescreen, you'd pretty much have to yank the power cord. NOT recommended.
In case of a crash, anything not saved to persistent storage will be lost. If you want to simulate a crash for purposes of logging, write a "kill switch" into your logger, which stops the logging. Now you can simulate a crash by killing the logging and making sure you have the data you would have wanted in case of an actual crash.
First of all, I would advise you to use a Virtual Machine to test this BSOD on. This will allow you to keep a backup just in case the BSOD does some damage to the system. Here's a tip on how to generate a BSOD simply by pressing CTRL+SCROLLLOCK+SCROLLLOCK.
Is there a Windows API to generate one? No, according to this article. Still, if you would call certain API's with invalid data, they could still cause a crash inside the kernel, which would result in your BSOD.
I'm not sure exactly what you'd be testing. Since your program runs periodically, surely it's enough to check that the information is being dumped at the frequency that you specify while the system is running? Are you checking that the information stays around after the blue screen? Depending on how you are dumping it (and whether you are flushing buffers), this may not be necessary.
If you dont want to write code (driver, IOCTL...) you can use DiskCryptor. Note that no disk encrypting is need.
Just need to install the driver:
dcinst.exe -setup
And then generate a bsod using the DC console:
dccon.exe -bsod
Run process as critic and exit http://waleedassar.blogspot.co.uk/2012/03/rtlsetprocessiscritical.html