Our CF server occasionally stops processing mail. This is problematic, as many of our clients depend on it.
We found suggestions online that mention zero-byte files in the undeliverable folder, so I created a task that removes them every three minutes. However, the stoppage has occurred again.
I am looking for suggestions for diagnosing and fixing this issue.
CF 8 standard
Win2k3
Added:
There are no errors in the mail log at the time the queue fails
We have not tried to run this without using the queue, due to the large amount of mail we send
Added 2:
It does not seem to be a problem with any of the files in the spool folder. When we restart the mail queue, they all seem to process correctly.
Added 3:
We are not using attachments.
What we ended up doing:
I wrote two scheduled tasks. The first checked to see if there were any messages in the queue folder older than n minues (currently set to 30). The second reset the queue every night during low usage.
Unfortunately, we never really discovered why the queue would come off the rails, but it only seems to happen when we use Exchange -- other mail servers we've tried do not have this issue.
Edit: I was asked to post my code, so here's the one to restart when old mail is found:
<cfdirectory action="list" directory="c:\coldfusion8\mail\spool\" name="spool" sort="datelastmodified">
<cfset restart = 0>
<cfif datediff('n', spool.datelastmodified, now()) gt 30>
<cfset restart = 1>
</cfif>
<cfif restart>
<cfset sFactory = CreateObject("java","coldfusion.server.ServiceFactory")>
<cfset MailSpoolService = sFactory.mailSpoolService>
<cfset MailSpoolService.stop()>
<cfset MailSpoolService.start()>
</cfif>
We have not tried to run this without using the queue, due to the large amount of mail we send
Regardless, have you tried turning off spooling? I've seen mail get sent at a rate of 500-600 messages in a half second, and that's on kind of a crappy server. With the standard page timeout at 60 seconds, that would be ~72,000 emails you could send before the page would time out. Are you sending more than 72,000 at a time?
An alternative I used before CFMail was this fast was to build a custom spooler. Instead of sending the emails on the fly, save them to a database table. Then setup a scheduled job to send a few hundred of the messages and reschedule itself for a few minutes later, until the table is empty.
We scheduled the job to run once a day; and it can re-schedule itself to run again in a couple of minutes if the table isn't empty. Never had a problem with it.
Have you tried just bypassing the queue altogether? (In CF Admin, under Mail Spool settings, uncheck "Spool mail messages for delivery.")
I have the same problem sometimes and it isn't due to a zero byte file though that problem did crop up in the past. It seems like one or two files (the oldest ones in the folder) will keep the queue from processing. What I do is move all of the messages to a holding folder, restart the mail queue and copy the messages back in a chunk at a time in reverse chronological order, wait for them to go out and move some more over. The messages which were holding up the queue are put in a separate folder to be examined latter.
You can probably programmatically do this by stopping the queue, moving the oldest file to another folder, then start the mail queue and see if sending begins successfully by checking folder file counts and dates. If removing the oldest file doesn't work, repeat the previous process until all of the offending mail files are moved and sending continues successfully.
I hope the helps.
We have actually an identical setup, 32bit CF8 on Win2K3.
We employed Ben's solution about a year ago, and that certain has helped auto re-queue emails that get stuck.
However recently for no particular reason one of our 7 web servers decided to get into this state with every email attempt.
An exception occurred when setting up mail server parameters.
This exception was caused by:
coldfusion.mail.MailSessionException:
An exception occurred when setting up mail server
parameters..
Each of our web servers are identical clones of each other, so why it was only happening to that one is bizarre.
Another item to note is that we had a script which reboot the machine in the middle of the night due to JRUN's memory management issues. The act of rebooting seemed to initiate the problem. A subsequent restarting of the CF service would then clear it, and the machine would be fine until it rebooted again.
We found that the problem is related to the McAfee virus scanner, after updating it to exclude the c:\ColdFusion8 directory, the problem went away.
Hope that helps.
There is a bug in Ben Doom's code. Thank you anyway ben, the code is great, and we use it now on one of our servers with CF8 installed, but:
if directory (\spool) is empty, the code fails (error: Date value passed to date function DateDiff is unspecified or invalid.) That's because if the query object spool is empty (spool.recordcount EQ 0), the datediff function produces an error.
we used this now:
<!--- check if request for this page is local to prevent "webusers" to request this page over and over, only localhost (server) can get it e.g. by cf scheduled tasks--->
<cfsetting requesttimeout="30000">
<cfset who = CGI.SERVER_NAME>
<cfif find("localhost",who) LT 1>
security restriction, access denied.
<cfabort>
</cfif>
<!--- get spool directory info --->
<cfdirectory action="list" directory="C:\JRun4\servers\cfusion\cfusion-ear\cfusion-war\WEB-INF\cfusion\Mail\Spool\" name="spool" sort="datelastmodified">
<cfset restart = 0>
<cfif spool.recordcount GT 0><!--- content there? --->
<cfif datediff('n', spool.datelastmodified, now()) gt 120>
<cfset restart = 1>
</cfif>
</cfif>
<cfif restart><!--- restart --->
<cfsavecontent variable="liste">
<cfdump var="#list#">
</cfsavecontent>
<!--- info --->
<cfmail to="x#y.com" subject="cfmailqueue restarted by daemon" server="xxx" port="25" from="xxxx" username="xxxx" password="xxx" replyto="xxxx">
1/2 action: ...try to restart. Send another mail if succeeded!
#now()#
Mails:
#liste#
</cfmail>
<cfset sFactory = CreateObject("java","coldfusion.server.ServiceFactory")>
<cfset MailSpoolService = sFactory.mailSpoolService>
<cfset MailSpoolService.stop()>
<cfset MailSpoolService.start()>
<!--- info --->
<cfmail to="x#y.com" subject="cfmailqueue restarted by daemon" server="xxx" port="25" from="xxxx" username="xxxx" password="xxx" replyto="xxxx">
2/2 action: ...succeeded!
#now()#
</cfmail>
</cfif>
There is/was an issue with the mail spooler and messages with attachments in CFMX 8 that was fixed with one of the Hotfixes. Version 8.0.1, at least, should have had that fixed.
Related
I ran into an issue where my mail server only accepts 100 connections to the server every 5 minutes. My current code loops over my database, calling cfmail for each person on the list. I suppose the problem is im opening a new connection each time I use cfmail?
<CFLOOP QUERY="Customer" >
<!---send mail to Admin ----->
<cfmail to = "#cstEmail#"
from = "#FORM.fromAddressEmail#"
subject = "#FORM.subjectEmail#"
server = "#var.mailserver#"
port= "#var.mailport#"
username="#var.mailuser#"
password="#var.mailpass#"
failto="#var.failEmail#
type="html"
>
What I ran into was only 100 mails were being sent at a time, the rest were sent to cf's undelivered folder. I would send them to spool and again 100 would get through..
Now, I've read in older versions of cf there is a checkbox in cf administrator to "maintain connection" -Im running cf9 and dont see this option.
Would using cfmail's query attribute, force cfmail to only connect to the mail server once to send all the emails?
<cfmail query="Customer"
from = "#FORM.fromAddressEmail#"
to = "#cstEmail#"
subject = "#FORM.subjectEmail#">
Im not even sure how to test this without sending a couple hundred emails. Any thoughts if this is a viable solution to the problem?
Thanks for your help!
Biscotti
I ended up compromising by using a scheduled task to move the files every 5 minutes back over to the Spool dir from the Undelivr dir. Im not thrilled with this solution, but it works.
Thanks to Russ's Respooler extension. http://cfrespooler.riaforge.org/
By using the above code to call the QUERY within CFMAIL I only succeeded in speeding up the client side process. The mail server still rejected the mail after the 100th connection - leading me to determine there is no server side benefit to this method over simply looping CFMAIL like in my first example. I seems the only answer is to run the code within the enterprise edition of the cf environment, one that has the "maintain connection" feature enabled.
I need some help with creating an automatic timing script with ColdFusion. My assumption is cfschedule and conditionals will be used to get this done. I am looking for something automatic, not on browser loads.
Anyways, every 60 minutes I would like to see if a page on one of my servers (http://www.mysite.com/page.php) is working or not.
If the page is down when it is checked, then it will check again in 5 minutes to see if the page is back up.
If it is not back up, then I am sent an e-mail to email#mysite.com. If it is back up, then no action is required and we start the 60 minute cycle check again.
Can anybody please help me with this?
Mike,
Not sure what version of CFML you are on but you could certainly set a scheduled task in your cf-administrator (I prefer the admin over cfschedule ... personal preference is all) to run your "site checker" script/page each hour and then, in that script/page, you could do something along the lines of:
<cfhttp url="http://mysite.com/ping-this-mofo.cfm" method="get" >
<cfif cfhttp.statusCode neq "200 OK">
<!--- some code to sleep for 5 minutes or a one-time cfschedule to check the site again --->
<cfschedule action="run" task="my-task-thingy" url="script-that-will-run" interval="once" startDate="today" startTime="5 minutes from now" />
</cfif>
Then, when that "sub-scheduled task" runs, you could check for the server being active and, if it isn't, fire off a an email via cfmail.
Is there any way to restart the CF server through the Application.cfc, when the application times out? As per Adobe documentation, they showed as follows:
<cffunction name="onApplicationEnd">
<cfargument name="ApplicationScope" required=true/>
<cflog file="#This.Name#" type="Information"
text="Application #Arguments.ApplicationScope.applicationname# Ended" >
</cffunction>
What I would like to do is replace the <cflog> above with <cfexecute> as follows:
<cfexecute name = "C:\CFRestart.bat"
outputFile = "C:\output.txt"
timeout = "1">
</cfexecute>
So OnApplicationEnd will run the CFRestart.bat file when the application times out. Is this possible or not?
onApplicationEnd is not likely to be reached unless you have a very quiet application because every time someone access the application the timeout is reset.
I'd be very uncomfortable using an application to restart a coldfusion instance. I can see all sorts of horrible security issues etc looming. To be honest I'm not really sure why you'd want to restart the server if your application end.
Also, according to the docs onApplicationEnd is called when the server is restarted, so if you did get this working, when you restart your server the application would also have a go at restarting your server. This would get very messy.
Don't believe you can call the .bat script from ColdFusion. Because once it stops the service the <cfexecute> will also terminate (think it runs under the CF service), never reaching the restart.
Guessing you have a server that routinely fails because you're hitting an Out of Memory (OOM) exception. To get over the hump in those situations I setup as batch script as a Windows Scheduled Task (see the first answer there for how) that restarts the server periodically, say every 24, 12, or 6 hours. Choose an interval that makes sense for your situation.
Assuming OOM is the root cause, I suggest downloading a Java JDK, configuring ColdFusion to use it (i.e. jvmhome in jvm.config file), and passing parameters to enable a JMX connection. You use this JMX connection to monitor ColdFusion using Visual VM, which comes with the JDK. From there you can generate a heap dump file and/or tell the VM to generate one on OOM. Tehn I've had very good success running that through the Eclipse Memory Analyzer Tool, which has a suspected leaks report that more than once has helped track down the root cause of server OOM crashes.
If that is not your scenario then I suggest enabling snapshots if you're using ColdFusion enterprise, otherwise cfstat is you friend on standard. For either one, you can also setup probes that send a notification when the server is running slowly. This can help you connect to the server in question and generate a dump at the appropriate time or identify if the problem is load related instead.
This may not be your answer, but I use this often to help the garbage collection in JVM memory.
Set this as a scheduled task to run every 5 minutes, and i never get jvm memmory problems anymore.
<cfparam name="url.maxused" default="999">
<cfparam name="url.minfree" default="300">
<cfif NOT isDefined("runtime")>
<Cfset runtime = CreateObject("java","java.lang.Runtime").getRuntime()>
</cfif>
<cfset fm = runtime.freememory()/>
<Cfset fm = int((fm/1024)/1024)/>
<cfset usedmem = 1270-fm/>
<cfoutput>
#Now()#<br>
Before<br>
Free: #fm# megs<br>
Used: #usedmem# megs<br>
</cfoutput>
<br>
<!--- check if we are using too much memory --->
<cfif usedmem gt url.maxused or fm lt url.minfree>
<cfset runtime.gc()>
Released Memory<br>
<cfelse>
No need to release memory using the thresholds you provided<br>
</cfif>
<br>
<cfset fm = runtime.freememory()/>
<Cfset fm = int((fm/1024)/1024)/>
<cfset usedmem = 1270-fm/>
<cfoutput>
After<br>
Free: #fm# megs<br>
Used: #usedmem# megs<br>
</cfoutput>
This has been hanging around unanswered for an age, so I thought I'd help getting it cleared up.
First, this:
"Server Error The server encountered an internal error and was unable to complete your request. Could not connect to JRun Server."
This is NOT an application timeout, this is just the server becoming unresponsive, or running out of memory, or just encountering something it doesn't like. But it's nothing to do with the application timing out.
The application times out when there's been no activity (ie: no page requests... no visitors) on that site for longer than the application timeout period, which by default is two days (or whatever you have set in your Application.cfc).
Now... I can understand why you might want to recover if your server becomes unresponsive, bur you're approaching this from the wrong angle. Intrinsically if the server ain't working, you can't use that server to do anything (like cure itself)! What is generally done here is that some other process checks that the server is responsive, and if that service determines the server is not responsive, issues a restart.
So you should look at some other software which can perform an HTTP request to your CF server, and if the reaction to the HTTP request suggests the CF server is unresponsive, then the monitoring software tells CF to restart.
To add to the answer by Stephen Moretti and may be the possible solution you were looking for to your interesting question above:
Will the OnApplicationEnd run CFRestart BAT file when application is
timeout?
The straight answer is no. Since the OnApplicationEnd() event is a part of the Application's life cycle, so when the application itself is timed out, there is no event that will be called here. This must be clear.
Getting straight to your question though, yes, you can make a custom script file run on the event of an application timeout or end (whatever is the case). You will have to deal with the Serve Scope here.
First up, the application doesn't timeout, a page request does. On request timeout the onApplicationEnd() function is not called. That is only called if the application is shutting down. Here is some info on the CF application life cycle.
Second, in my experience, restarting application servers for whatever reason is probably masking your real problem. If you application is running slow / crashing etc. then I suggest you look into the real reason this is happening rather than restarting it.
However, I can't think of a reason this would not work in principle, but I would suggest you conduct a quick test if this really is what you wish to do.
Hope that helps.
Is there a way to reset the Application, via onApplicationStart, and ensure that all the servers in the cluster have their application restarted?
For example, in the current code, DSN is set in request.
I would like to push it to application scope, but if there is a change that needs to be made, I would like to make sure it is propagated to all the servers in the cluster without starting and stopping CF services.
One suggestion is to have a date-stamped file in your project that each server can read.
Get CF to save the date into the application on start-up and compare what it has against that file in each onRequestStart().
If the date ever changes then CF runs the onApplicationStart() of that server.
We use that technique with our ANT scripts, so as we deploy to each server ANT timestamps the file and the next person to hit our application causes the application reset.
Cheers,
James
We have our applications set up to reset their scopes via URL variable. So you could use code like this to restart the applications on all servers in your cluster via http through a page in your admin:
<cfset serverList = "Server1.domain.com,Server2.domain.com" />
<cfset threadList = "" />
<cfloop list="#serverList#" index="currentServer">
<cfset threadList = ListAppend(threadList, "thread#currentServer#") />
<cfthread
action="run"
name="thread#currentServer#"
url="http://#currentServer#?URLToResetScope"
key="#currentIndex#"
>
<cfhttp method="get" url="#Attributes.URL#" result="thread.cfhttp" />
</cfthread>
</cfloop>
<cfthread action="join" name="#threadList#" timeout="120000" />
<cfloop list="#threadList#" index="currentThread">
<cfset Results &= currenthThread & ": " & threadresult.cfhttp />
</cfloop>
What we're doing is taking a list of servers, then looping through that list, and with cfthread, passing off a request to each server to reset it's own scope. Then, we join all those threads together, and loop through them to get the results of each request. Then we can display those results in the admin, to make sure that all servers in the cluster got updated correctly. If there is any server that didn't get updated correctly, we can then go reset that server individually.
Dan
We are currently experiencing intermittent mail queue stoppages. I'm seeking diagnostic help in another area.
In the meantime, is there a way to restart the CF mail queue without restarting the service as a whole?
CF8 standard
Win2k3
Solution: We are now checking the age of the oldest file in the mail queue. When it exceeds a set age (currently 30 min) the mail queue is restarted.
Yes there is.
<cfset sFactory = CreateObject("java","coldfusion.server.ServiceFactory")>
<cfset MailSpoolService = sFactory.mailSpoolService>
<cfset MailSpoolService.stop()>
<cfset MailSpoolService.start()>