I'm running on windows 8.
Occasionally, when I resume from sleep, my service gets a stop request through the SCM (call to SvcCtrlHandler with SERVICE_CONTROL_STOP). I wasn't able to trace the source of this request. Can it possibly be sent by the OS itself, in some scenario?
My two main suspicions right now:
If the resume event (SERVICE_CONTROL_POWEREVENT of type PBT_APMRESUMEAUTOMATIC) is taking too long, the OS might stop the service (system logs contain logs referring to this specific service: A timeout was reached (30000 milliseconds) while waiting for the [...] The service did not respond to the start or control request in a timely fashion)
The OS stops the service because it has been flagged as a problematic service (system logs contain logs referring to this specific service: service did not shut down properly after receiving a preshutdown control
Related
I'm trying to start a service with StartService method. According to the documentation:
StartService will block for 30 seconds if any service is busy handling a control code.
How can I change this timeout value?
You cannot change the timeout. It is builtin to the SCM, and contractually obligated by the documentation to be 30 seconds only.
UPDATE: Apparently, you can change the timeout after all. But only in the Registry, not in code. And it requires a reboot to take effect. See How do I increase windows service startup timeout on Server Fault.
It is the responsibility of each service to respond to the SCM in a timely manner. During a start request, a service needs to call StartServiceCtrlDispatcher() as soon as possible. If it needs a lengthy startup, it should be starting the dispatcher quickly, then entering a PENDING state and report updated status at regular intervals until ready.
Problem
I have a very basic configuration for a Spring integration mail adapter setup (below is the relevant sample):
<int:channel id="emailChannel">
<int:interceptors>
<int:wire-tap channel="logger"/>
</int:interceptors>
</int:channel>
<mail:inbound-channel-adapter id="popChannel"
store-uri="pop3://user:password#domain.net/INBOX"
channel="emailChannel"
should-delete-messages="true"
auto-startup="true">
<int:poller max-messages-per-poll="1" fixed-rate="30000"/>
</mail:inbound-channel-adapter>
<int:logging-channel-adapter id="logger" level="DEBUG"/>
<int:service-activator input-channel="emailChannel" ref="mailResultsProcessor" method="onMessage" />
This is working fine the majority of the time and I can see the logs showing the polling (and it works fine hooking into my mailResultsProcessor when a mail is there):
2013-08-13 08:19:29,748 [task-scheduler-3] DEBUG org.springframework.integration.mail.Pop3MailReceiver - opening folder [pop3://user:password#fomain.net/INBOX]
2013-08-13 08:19:29,796 [task-scheduler-3] INFO org.springframework.integration.mail.Pop3MailReceiver - attempting to receive mail from folder [INBOX]
2013-08-13 08:19:29,796 [task-scheduler-3] DEBUG org.springframework.integration.mail.Pop3MailReceiver - found 0 new messages
2013-08-13 08:19:29,796 [task-scheduler-3] DEBUG org.springframework.integration.mail.Pop3MailReceiver - Received 0 messages
2013-08-13 08:19:29,893 [task-scheduler-3] DEBUG org.springframework.integration.endpoint.SourcePollingChannelAdapter - Received no Message during the poll, returning 'false'
The problem I have is that the polling stops during the day, with no indication in the logs why it has stopped working. The only reason I can tell is the debug above is not present in the logs and E-Mails build up on the E-Mail account.
Questions
Has anyone seen this before and know how to resolve it?
Is there a change that I can make in my configuration to capture the issue into the log? I thought the logging channel adapter set to debug would have this covered.
Using version 2.2.3.RELEASE of Spring Integration on Tomcat 7, logs output default to catalina.out. Deployed on AWS standard tomcat 7 instance.
Most likely the poller thread is hung someplace upstream. With your configuration, the next poll won't happen until the current poll completes.
You can use jstack or VisualVM to get a thread dump to find out what the thread is doing.
Another possibility is you are suffering from poller thread starvation - if you have a lot of other polled elements in your application, and depending on their configuration. The default taskScheduler bean has only 10 threads.
You can add a task executor to the <poller/> so each poll is handed off to another thread, but be aware that that can result in concurrent polls if a polled task takes longer to execute than the polling rate.
To resolve this problem specifically I used the configuration below:
<mail:inbound-channel-adapter id="popChannel"
store-uri="pop3://***/INBOX"
channel="emailChannel"
should-delete-messages="true"
auto-startup="true">
<int:poller max-messages-per-poll="5" fixed-rate="60000" task-executor="pool"/>
</mail:inbound-channel-adapter>
<task:executor id="pool" pool-size="10" keep-alive="50"/>
Once moving to this approach we saw no further problems, and is with any use of pool the advantage is any Threads that become a problem are cleaned up and recreated.
I write a local service application using C++ and I can't find the correct way of registering for a pre-shut-down notification (for OS later than Windows XP). I believe that SERVICE_CONTROL_PRESHUTDOWN notification has been added since Vista, but when you call SetServiceStatus do we need to specify:
dwServiceStatus.dwControlsAccepted = SERVICE_ACCEPT_PRESHUTDOWN;
or
dwServiceStatus.dwControlsAccepted = SERVICE_ACCEPT_SHUTDOWN | SERVICE_ACCEPT_PRESHUTDOWN;
You cannot accept both a shutdown and a preshutdown if your service is correctly coded. The documentation explicitly states this.
From http://msdn.microsoft.com/en-us/library/windows/desktop/ms683241(v=vs.85).aspx:
Referring to SERVICE_CONTROL_PRESHUTDOWN:
A service that handles this notification blocks system shutdown until the service stops or the preshutdown time-out interval specified through SERVICE_PRESHUTDOWN_INFO expires.
In the same page, the section about SERVICE_CONTROL_SHUTDOWN adds:
Note that services that register for SERVICE_CONTROL_PRESHUTDOWN notifications cannot receive this notification because they have already stopped.
So, the correct way is to set the dwControlsAccepted to include either SERVICE_ACCEPT_SHUTDOWN or SERVICE_ACCEPT_PRESHUTDOWN, depending on your needs, but not to both at the same time.
But do note that you probably want to accept more controls. You should always allow at least SERVICE_CONTROL_INTERROGATE, and almost certainly allow SERVICE_CONTROL_STOP, since without the latter the service cannot be stopped (e.g. in order to uninstall the software) and the process will have to be forcibly terminated (i.e. killed).
As noted by the commenters above, you will need to choose from either SERVICE_ACCEPT_SHUTDOWN or SERVICE_ACCEPT_PRESHUTDOWN (Vista or later). If you are using SERVICE_ACCEPT_PRESHUTDOWN, you will need to register your service with the SCM using RegisterServiceCtrlHandlerEx instead of RegisterServiceCtrlHandler else you will not be receiving the pre-shutdown notifications. The handler prototype also changes from Handler to HandlerEx.
Another point to note is that handling pure shutdown events is limited to 5 seconds in Windows Server 2012 (and presumably Windows 8), 12 seconds in Windows 7 and Windows Server 2008, 20 seconds in Windows XP before your service is killed while stopping. This is the reason why you may need the pre-shutdown notification. You may want to change this at \\HKLM\SYSTEM\CurrentControlSet\Control\WaitToKillServiceTimeout.
In the comment from alexpi there is a key piece of information. I found that the service handling PRESHUTDOWN needs to update the service status with a new checkpoint number (repeatedly) before WaitToKillServiceTimeout has elapsed. My server was configured to 5000 ms and my service only updated every 12000 ms, and the server went into the SHUTDOWN phase, which caused my attempt to stop another service to return the error that the shutdown was in progress.
These two notifications seem to be different as I get it from the documentation. If what you need is really to enable your service to recieve preshutdown notification, you should go with: dwServiceStatus.dwControlsAccepted = SERVICE_ACCEPT_PRESHUTDOWN; But if you also want to enable your service to receive shutdown notifications, you should go with your second option.
I am trying to control a service within an application. Starting the service via StartService (MSDN) works fine, the service needs about 10 seconds to start, but after calling StartService it gives the control back to the main-application immediately.
However, when stopping the service via ControlService (MSDN) - AFAIK there is no StopService - it blocks the main-application for the complete time until the service is stopped, which takes about 10 seconds.
Start: StartServiceW( handle, 0, NULL)
Stop: ControlService( handle, SERVICE_CONTROL_STOP, status )
Is there a way for a non-blocking / asynchronously stopping of a windows service?
I would probably look at stopping the service in a new thread. That will eliminate the blocking of your main thread.
The SCM processes control requests in a serialized manner. If any service is busy processing a control request, ControlService() will be blocked until the SCM can process the new request. This is stated as much in the documentation:
The SCM processes service control notifications in a serial fashion—it
will wait for one service to complete processing a service control
notification before sending the next one. Because of this, a call to
ControlService will block for 30 seconds if any service is busy
handling a control code. If the busy service still has not returned
from its handler function when the timeout expires, ControlService
fails with ERROR_SERVICE_REQUEST_TIMEOUT.
The service is doing its cleanup in its control handler routine. That's OK for a service that will only take a fraction of a second to exit, but a service that's going to take ten seconds should definitely be setting a status of STOP_PENDING and then cleaning up asynchronously.
If this is your own service, you should correct that problem. I'd start by making sure that all of the cleanup is really necessary; for example, there's no need to free memory before stopping (unless the service is sharing a process with other services). If the cleanup really can't be made fast enough, launch a separate thread (or signal your main thread) to perform the service shutdown and set the service status to STOP_PENDING.
If this is someone else's service, the only solution is to issue the stop request from a separate thread or in a subprocess.
Is there a way to kill / re-start a long running task in AWS SWF? Sometimes some of our tasks run for a longer duration and we would like to manually kill a certain task (either via UI or programmatically) and re-start the task if possible. How to achieve this?
Console is option to manually kill workflow.
You can also set timeouts to whole workflow execution time or to individual activities. This can be set when you register your activity or when you start your activity (defaultTaskStartToCloseTimeoutSecond).
It's not clear what language you're using.
If you're using java, then you should look into Exponential Retry in Flow Framework. This make SDK restart your activity if it fails.
Long running activity is expected to heartbeat using RecordActivityTaskHeartbeat. It leads to timeout failure after short hearbeat interval instead of long task execution timeout if the activity process hangs or crashes.
The workflow code (decider) can always request activity cancellation through RequestCancelActivityTask decision. The cancellation request is returned as output of the RecordActivityTaskHeartbeat call. Activity implementation should cancel itself and report back to the service using RespondActivityTaskCanceled API call.
See Error Handling section of AWS Flow Framework Developer Guide for the AWS Flow Framework way of cancelling activities.
Sometimes activity implementation cannot support heartbeating and self cancellation. The solution is to execute another kill activity that terminates the first activity execution. For example under Unix such kill activity could emit "kill -9" command for the process that implements the first one.