I am currently working on a newsletter application which needs to sent around 20K mails in a single shot. I feel, ColdFusion mail sending capacity will not be suitable for this. I don’t have any metrics with me to prove this.
I searched around in web about this, I found there are lot of complains about this kind of bulk mail sending process using built in CFMAIL. Few of them are,
Java heap size error
IOException while sending message
Mail Spool Lock
Mail Spool Timeout
We are using Adobe ColdFusion 9 standard edition.
Is anyone worked like this requirement? Did you faced any issues? How did you fixed that? Is any other alternative ways are there instead of using regular cfmail tag?
We have a system sending out nearly 500k messages about 2 to 3 times per day. CF does not have a problem with it if you have adequate resources allocated - meaning decent drives and space, adequate heap space, and (especially) a well performing SMTP relay. In our case we set up a dedicated relay server that we could fine tune carefully.
CF Mail has not had significant issues since ColdFusion 7. CF 8 and CF 9 both perform well. There is a level at which I would suggest a different system to be sure - and other requirements might mean an SaaS would serve you better - but from a per performance standpoint 20k messages is fairly small and should not be a problem for an enterprise class server.
I think you may face issue if you are using Standard version, I am not sure but there is some thread limit (I guess it it is 10) for spool for standard version which will delay email. Let's say there are 10 threads for spool and every thread wait for 15 seconds for next run (can be adjust from CFIDE) still it will take around 8.333 hours to send out all emails.
We have been using ColdFusion as a webmail delivery system since version 6. We have mail groups with 20k recipients.
We experienced one issue with it back then, the spooler hung and started retransmitting the same mail over and over again, making recipients receive the mail multiple times.
I think this issue was resolved with version 7.
If I recall well there's a difference between coldfusion standard and enterprise. Enterprise versions has enhanced CFMAIL throughput. At least it used to be in previous versions.
Related
I have two windows computers, both using the Microsoft Edge browser. When I'm typing on the Overleaf website, the connection gets lost every 2-5 minutes. What's worse, some unsynced sentences are gone when the connection resumes. I'm not sure whether this is a network problem since all other websites look good, including Twitter and Gmail. I wondered if this is about the framework of what Overleaf cloud service used. Could anyone give some tips about the issue here?
I have a client who is using the ECM and just dispatched an email blast to approximately 18,000 users. The dispatch is taking quite a while (about 2 hours to process not even half of the users).
Has anyone encountered this issue?
Can the ECM not handle such large lists?
As mentioned elsewhere, ECM can handle that load just fine. In general, throughput on ECM is limited by:
Fragmented indices on the "analytics" database (or just limited capacity on same)
Bandwidth limitations. If each mail is 500KB (lots of images), sending 10 mails per second requires 5Mbit bandwidth
CPU on the server in question
From what you've shared so far, I cannot guess as to which of the above is limiting the throughput on your installation. My personal guess would be capacity and speed of the database.
More information here: http://sdn.sitecore.net/upload/sdn5/products/ecm/200/ecm_tuning_guide_20-a4.pdf
I had a similar issue with a client where it was taking hours to send emails. Check the NumberThreads setting in the Sitecore.EmailCampaign.config file. The default is quite low at "1" and most servers should be able to handle more threads.
Definitely follow the tuning guide that Mark posted. The Performance Measurement Tool can help you get the ideal number of sending threads so that you're not over or under utilizing your server.
An exception occurred when setting up mail server parameters. This
exception was caused by:
coldfusion.mail.MailSpooler$SpoolLockTimeoutException: A timeout
occurred while waiting for the lock on the mail spool directory..
Recently i started to get this nasty exception in my mail.log file. Once this exception shows up, every mail that is sent from that coldfusion instance throws the same exception.
The only thing that seems to work is to restart the coldfusion server. After (usually) a day or two the same exception pops up again and we're back in the same situation.
I am aware of the hotfix to control the mailspool timeout but all it does is increase the timeout from 30 to 60 seconds. Since the mails are sent successfully until i get the exception, i don't think this is my solution.
Also i read the thread in the adobe forum where people have installed the hotfix, but still get the error.
I also tried a script to restart only the mailservice when this exception showed up, but this didn't work for me, as it didn't for others with this problem. This would also not be a concrete solution.
The mails that i send arre simple html mails.
The number of mails sent spreaded over a day is not more then 30.
I've sent mails from
the exact same coldfusion server many times before, but with
<cfmail>. This is the first time i'm sending them in cfscript. I
don't know if this has anything to do with it, but it's only since
i'm using the cfscript equivalent of <cfmail> that i started to get
this exception.
All related blog posts that i could find are all unanswered but also pretty old. I thought that someone might have a solution by now.
Thanks.
(using coldfusion 9.0.1 server on windows 2008 server)
We were also experiencing this mail spool lock issue. After the issue occurred a fourth time in 2 months, we started reviewing these forums and found no solution.
This made me think that perhaps the solution and problem are not really CF at all, so I went into the server's virus protection and excluded the CF mail spool directory so that the virus protection does not touch the spool directory at all. So far, we have not had the problem again.
So I am not sure that this is the permanent fix, but it has worked so far for us. No outside entities create emails within our systems, so the directory should be relatively safe but not having email-outs work is not an option.
this chain from talkingtree might give some light:
http://www.talkingtree.com/blog/index.cfm?mode=entry&entry=67FD4A34-50DA-0559-A042BCA588B4C15B
what they are saying is that it could be an issue with disk activity taking to long. you can increase the mail spool timeout with the jvm argument: -Dcoldfusion.spooltimeout=120
oh.... one more thing. if you're using cfmail to email dumps when an error occurrs, make sure to add 'format="text"' to the cfdump tags. some of the emails can get pretty big and might be causing the error.
(Edited to try to explain better)
We have an agent, written in C++ for Win32. It needs to periodically post information to a server. It must support disconnected operation. That is: the client doesn't always have a connection to the server.
Note: This is for communication between an agent running on desktop PCs, to communicate with a server running somewhere in the enterprise.
This means that the messages to be sent to the server must be queued (so that they can be sent once the connection is available).
We currently use an in-house system that queues messages as individual files on disk, and uses HTTP POST to send them to the server when it's available.
It's starting to show its age, and I'd like to investigate alternatives before I consider updating it.
It must be available by default on Windows XP SP2, Windows Vista and Windows 7, or must be simple to include in our installer.
This product will be installed (by administrators) on a couple of hundred thousand PCs. They'll probably use something like Microsoft SMS or ConfigMgr. In this scenario, "frivolous" prerequisites are frowned upon. This means that, unless the client-side code (or a redistributable) can be included in our installer, the administrator won't be happy. This makes MSMQ a particularly hard sell, because it's not installed by default with XP.
It must be relatively simple to use from C++ on Win32.
Our client is an unmanaged C++ Win32 application. No .NET or Java on the client.
The transport should be HTTP or HTTPS. That is: it must go through firewalls easily; no RPC or DCOM.
It should be relatively reliable, with retries, etc. Protection against replays is a must-have.
It must be scalable -- there's a lot of traffic. Per-message impact on the server should be minimal.
The server end is C#, currently using ASP.NET to implement a simple HTTP POST mechanism.
(The slightly odd one). It must support client-side in-memory queues, so that we can avoid spinning up the hard disk. It must allow flushing to disk periodically.
It must be suitable for use in a proprietary product (i.e. no GPL, etc.).
How is your current solution showing its age?
I would push the logic on to the back end, and make the clients extremely simple.
Messages are simply stored in the file system. Have the client write to c:/queue/{uuid}.tmp. When the file is written, rename it to c:/queue/{uuid}.msg. This makes writing messages to the queue on the client "atomic".
A C++ thread wakes up, scans c:\queue for "*.msg" files, and if it finds one it then checks for the server, and HTTP POSTs the message to it. When it receives the 200 status back from the server (i.e. it has got the message), then it can delete the file. It only scans for *.msg files. The *.tmp files are still being written too, and you'd have a race condition trying to send a msg file that was still being written. That's what the rename from .tmp is for. I'd also suggest scanning by creation date so early messages go first.
Your server receives the message, and here it can to any necessary dupe checking. Push this burden on the server to centralize it. You could simply record every uuid for every message to do duplication elimination. If that list gets too long (I don't know your traffic volume), perhaps you can cull it of items greater than 30 days (I also don't know how long your clients can remain off line).
This system is simple, but pretty robust. If the file sending thread gets an error, it will simply try to send the file next time. The only time you should be getting a duplicate message is in the window between when the client gets the 200 ack from the server and when it deletes the file. If the client shuts down or crashes at that point, you will have a file that has been sent but not removed from the queue.
If your clients are stable, this is a pretty low risk. With the dupe checking based on the message ID, you can mitigate that at the cost of some bookkeeping, but maintaining a list of uuids isn't spectacularly daunting, but again it does depend on your message volume and other performance requirements.
The fact that you are allowed to work "offline" suggests you have some "slack" in your absolute messaging performance.
To be honest, the requirements listed don't make a lot of sense and show you have a long way to go in your MQ learning. Given that, if you don't want to use MSMQ (probably the easiest overall on Windows -- but with [IMO severe] limitations), then you should look into:
qpid - Decent use of AMQP standard
zeromq - (the best, IMO, technically but also requires the most familiarity with MQ technologies)
I'd recommend rabbitmq too, but that's an Erlang server and last I looked it didn't have usuable C or C++ libraries. Still, if you are shopping MQ, take a look at it...
[EDIT]
I've gone back and reread your reqs as well as some of your comments and think, for you, that perhaps client MQ -> server is not your best option. I would maybe consider letting your client -> server operations be HTTP POST or SOAP and allow the HTTP endpoint in turn queue messages on your MQ backend. IOW, abstract away the MQ client into an architecture you have more control over. Then your C++ client would simply be HTTP (easy), and your HTTP service (likely C# / .Net from reading your comments) can interact with any MQ backend of your choice. If all your HTTP endpoint does is spawn MQ messages, it'll be pretty darned lightweight and can scale through all the traditional load balancing techniques.
Last time I wanted to do any messaging I used C# and MSMQ. There are MSMQ libraries available that make using MSMQ very easy. It's free to install on both your servers and never lost a message to this day. It handles reboots etc all by itself. It's a thing of beauty and 100,000's of message are processed daily.
I'm not sure why you ruled out MSMQ and I didn't get point 2.
Quite often for queues we just dump record data into a database table and another process lifts rows out of the table periodically.
How about using Asynchronous Agents library from .NET Framework 4.0. It is still beta though.
http://msdn.microsoft.com/en-us/library/dd492627(VS.100).aspx
Currently using ColdFusion 8 enterprise on 32 bit linux box to send out our mail to a third party provider who do the delivery (relay). Currently we have maintain mail server connections checked in the ColdFusion admin but they'd like us to limit the connections to each one of their servers to 5 and I'm not sure if you can make ColdFusion do that, I'm pretty sure not, at least not officially...
Looking at a max of 4000 / minute being spooled but more likely in the region of 2000 / minute.
Two questions:
Is it possible to "hack" ColdFusion to limit the amount of connections it creates and maintains (in an xml file within lib or SERVER-INF somewhere maybe?)
How much performance loss would you expect from not maintaining the connections?
If anyone has any experience with this it'd be appreciated.
Does "Mail Delivery Threads" on the Mail spool settings not do this?
if the above doesn't give you what you want, then you can always use the javax.mail libraries. Those libraries give you a lot of flexibility and can generally configure it how you want to. Of course, this option works best when your a java developer.