I want to modify some variable in my running process and then restart the instance.
in this example i want ot increment maxRetry :
When i pause and start the process, nothing happens.
Thanks for your help
Related
Goal
Block or pause a reboot until my app's files have finished saving. Specifically a reboot initiated from another apps MSI installer.
Summary
The app I'm working on captures and records information during an app install on windows. Occasionally, the other app finishes installing and requires a reboot. When that happens, I need time to finish saving my files before the reboot continues.
What's Working
ShutdownBlockReasonCreate()
ShutdownBlockReasonDestory()
These work as expected; but only if I manually trigger a reboot with Start -> reboot. Windows displays the Shutdown Block message to the user with the 'force shutdown' UI (full screen blue screen)
A program is preventing shutdown.
"shutdown block reason message"
shutdown anyway?
Program Flow
my app : begin watching install process from another app
shutdown block reason create (message)
other app finishes install
You must restart your system for the configuration changes to take effect.
Click 'Yes' to restart now or 'No' if you plan to restart later.
User clicks 'Yes'
WM_QUERYENDSESSION
return false // Do not continue reboot
WM_ENDSESSION
bEnding = True
return false // App is not ready to close
ShutdownBlockReasonDestroy( ) // continue shutdown
Expected Behavior
Same behavior as when I manually start a reboot. When I return false from OnQueryEndSession() the reboot should stop, or at least pause until I have destroyed the shutdown block reason. If there is a current ShutdownBlockReason, Windows should display the shutdown block reason to the user. Windows gives the user the chance to shutdown now, wait for the process to finish, or wait and go back to save their work.
Current Behavior
The Windows Force Shutdown UI is never displayed to the user.
The BlockShutdownReason is never displayed to the user.
The app is not given time to complete the file save.
Windows asks if the shutdown can continue (WM_QUERYENDSESSION) but always follows that up with WM_ENDSESSION, bEnding = True. Windows gives minimal time to complete the file save (3 seconds) and does not give my app a chance to inform the user that a file save is in progress.
The user is not informed that a file save is in progress.
What I've Tried
Create and destroy ShutdownBlockReason
CreateProcess("C:\...\shutdown.exe /a")
AbortSystemShutdown
if I call this before the shutdown has started:
'Error: No shutdown is in progress'
if I call this after the shutdown has begun:
'Error: A shutdown is in progress'
que frustrated confused developer
Set registry key:values to not automatically shut down apps
AutoEndTasks : False
HungAppTimeout : 120_000
WaitToKillAppTimeout : 120_000
Validation
I have ensured the application has valid shutdown permissions
I have set the permissions with the app token
I have double checked that the permissions were set correctly
I have used ShutdownBlockReasonQuery() to validate that the shutdown block reason exists before during and after OnQueryEndSession and OnEndSession.
I have validated that the window is a valid top level window.
References
Windows Official Documentation
ShutdownBlockReasonCreate
ShutdownBlockReasonDestroy
Application Shutdown Changes in Windows Vista
Shutting Down
Example :: How to shut down the system
Force Reboot
Disable Automatic Application Shutdown
Enables the files in use dialog (prior to install)
Does not effect the Force Shutdown (blue screen) UI (at the end of the install)
Restart Dialog Function
ExitWindowsEx
Windows Installer : System Reboots
Blogs and Forums
Now that windows makes it harder for your problem to block shutdown, how do you block shutdown?
Stack Overflow :: Example of ShutdownBlockReasonCreate
Stack Overflow :: ShutdownBlockReasonCreate causes shutdown to be canceled after one minute
Turn off automatic termination of applications
Allows non-window applications to be considered valid when responding to "WM_QUERYENDSESSION"
My application has a top level window and should already be considered valid, as evidenced when the user clicks Start -> Reboot
SO : Is there a way to delay a forced reboot
Only applies to Windows Update reboots
Suppress a MSI Reboot
Suppress reboot from the originating installer, not a 3rd party application.
Four ways to stop a shutdown or reboot
These apply to normal reboots, not a reboot from Windows Installer, which overrides all options.
None of these solutions have provided a way to block a shutdown from the end of an install, when it asks if you want to shutdown.
They all work as expected when the user clicks Start -> shutdown, but otherwise all attempts I have made to block or delay the shutdown have failed with no message to the user, and no effect on the shutdown / reboot.
Is there any "GODLY" method through which i can protect my process from getting killed from:
1>Task manager 2>Command prompt -- forcefully kill
Things i tried : 1. Hooking Open Process and Terminate Process
Result i got: well not much , i could only protect it from task manager but it gets killed through command prompt task kill command
It is not too difficult to modify a process so it cannot be terminated by any other user than an administrator.
Look into:
GetKernelObjectSecurity
SetKernelObjectSecurity
In short: Obtain the security descriptor of the process to be protected, modify its DACL and write it back. Users that you deny access this way get an "Access denied" when trying to kill the process.
What does "Pending Restart" mean? I have stopped and restarted my WebJob numerous times and that doesn't seem to fix it. Does it mean I have to restart my website? What caused my job to get in this state in the first place? Is there any way I can prevent this from happening in the future?
Usually, it means that the job fails to start (an exception?). Look in the jobs dashboard for logs.
Also, make sure that if the job is continuous, you actually have an infinite loop that keeps the process alive.
To add to Victor's answer, the continuous WebJob states are:
Initializing - The site was just started and the WebJob is doing it's initialization process.
Starting - The WebJob is starting up the process/script.
Running - The WebJob's process is running.
PendingRestart - The WebJob's process exited (for any good or bad reason) in less than 2 minutes since it started, for a continuous WebJob it's considered that something was probably not right with it (some exception during start-up probably as mentioned by Victor), at this point the system is waiting for 60 seconds before it'll restart the WebJob process (hence the name "pending restart").
Stopped - The WebJob was stopped (usually from the Azure portal) and is currently not running and will not be running until it is started again, best way to see this is as disabled.
Also, take a look at the webjob log, it should hold cue to what's been happening.
if the JOB is set to run continuously, once the process exits (say you are polling a queue and it's empty) the job shuts down and status changes to "pending restart". Azure Scheduler will typically restart the process in 60 seconds.
Try changing the target framework to .NET 4.5. This same issue was fixed for me when I changed the target framework from 4.6.1 to 4.5.
Had the same problem, found out that i need to keep my webjob alive, so I put a continous loop to keep it alive.
It means that application is failing after start. Check the App Service Application setting might have some problem. .. In my case i am passing date as a configuration and i entered wrong date like 20160431
This is just because Webjobs is failing or giving exception.
Make sure the Webjobs is continuous and you can check that in the log where it failing and can make the changes.
Process went down, waiting for 60 seconds
Status changed to Pending Restart
In my case, we got the deployment package prepared through the Visual Studio folder publish and deployed along with WebApp. Package created through 'Folder publish' lacked 'run.cmd' file in which command to invoke the console application (.exe) is available; this file is automatically created when we directly publish to Azure WebJob from Visual Studio.
After manually adding this to a package folder, the issue got fixed.
I want to use AWS AutoScaling to scaledown a group of instances when SQS queue is short.
These instances do some heavy work that sometimes requires 5-10 minutes to complete. And I want this work to be completed before the instance termination.
I know a lot of people should have faced the same problem. Is it possible on EC2 to handle the AWS termination request and complete all my running processes before the instance is actually terminated? What is the best approach to this?
You could also use Lifecycle hooks. You would need a way to control a specific worker remotely, because AWS will select a particular instance to put in Terminating:Wait state and you need to manage that instance. You would want to take the following actions:
instruct the worker process running on the instance to not accept any more work.
wait for the worker to finish the work it already is handling
call the complete-lifecycle action.
AWS will take care of the rest for you.
ps. if you are using celery to power your workers then you can remotely ask a worker to shutdown gracefully. It won't shutdown unless it finishes with the tasks it had started executing.
Assuming you are using linux, you can create a pre-baked AMI that you use in your Launch Config attached to your Auto Scaling Group.
In the AMI you can put a script under /etc/init.d say /etc/init.d/servicesdown. This script would execute anything that you need to shutdown which would be scripts under /usr/share/services for example.
Here's kind like the gist:
servicesdown
It would always get executed when doing a graceful shutdown.
Then say on Ubuntu/Debian you would do something like this to add it to your shutdown sequence:
/usr/sbin/update-rc.d servicesdown stop 25 0 1 6 .
On CentOS/RedHat you can use the chkconfig command to add it to the right shutdown runlevel.
I stumbled onto this problem because I didn't want to terminate an instance that was doing work. Thought I'd share my findings here. There are two ways to look at this though :
I need to terminate a worker, but I only want to terminate one that's not working
I need to terminate a SPECIFIC worker and I want that specific worker to wait until it's done with the work.
If you're goal is #1, Amazon's new "Instance Protection" looks like it was designed to resolve this.
See the below link for an example, they give this code snippet as an example:
https://aws.amazon.com/blogs/aws/new-instance-protection-for-auto-scaling/
while (true)
{
SetInstanceProtection(False);
Work = GetNextWorkUnit();
SetInstanceProtection(True);
ProcessWorkUnit(Work);
SetInstanceProtection(False);
}
I haven't tested this myself, but I see API calls related to setting the protection, so it appears that this could be integrated into the EC2 Worker App code-base and then when Scaling In, instances shouldn't be terminated if they are protected (currently working).
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/autoscaling/AmazonAutoScaling.html
As far as I know currently there is no option to terminate instance while gracefully shutdown and let process to complete work.
I suggest you to look at http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/as-configure-healthcheck.html.
We implemented it for resque workers while we are moving instance to unhealthy state and than downsizing AS. There is a script which checking constantly health state on each instance. Once instance moved to unhealthy state it stops all services gracefully and sending terminate signal to ec2.
Hope it helps you.
I have a custom django-command that reads and RSS, looks for new feeds and, if any new feed is found, I push it (pusher.com) to my webapp hosted in Heroku (heroku.com). This checking needs to be done as much as possible to be able to get the new feeds as soon as possible, let's say, every second.
The two issues I have are:
As this app will only be used by a few people(2-3), the command must be run only if any of these people are inside the app so I don't overload server jobs.
Once the user left the app (may be they just closed it, or they have certain time of inactivity, i.e. not clicking anything), the command must stop checking RSS.
My questions are,
where should I run the command from? directly from a view, from a signal?
How could I interrupt such command once the user leaves the app?
Thanks in advance for any help :)
You could use request-finished signal. In signal handler you could run celery task, so user hasn't wait the rss server request end