My website runs on gemstone. To provide users with statistical data I need to run periodical tasks. How is this achieved best in gesmtone? From within gemstone or from the operating system? And would be topaz the right way to go?
Using something like cron to fire up a topaz process is the right way to go.
Make sure that you capture stdout of the topaz process and include the purpose and date in the name of the log file.
If the frequency of the task is under say every 5 minutes, then firing up a 'permanent' topaz process (like the maintenance vm) sitting on a Delay is a good way to go.
Some of our customers use CONTROL-M to launch topaz, which includes workflow in it's process scheduling.
Dale
Related
I would like to create a job on Amazon Mechanical Turk that involves two workers engaging in a dialogue. I already have the javascript interface coded up. How would I manage two workers simultaneously, though? I don't want one worker to have to wait around for the second worker to join.
One workaround I was thinking about was having workers sign up for time slots, i.e. the 5 pm slot. Then the two workers signed up for the time slot could both join at the same time, without waiting around. Is this something I can do through Mechanical Turk?
To answer my own question, you can use JATOS for exactly this.
We've got a little java scheduler running on AWS ECS. It's doing what cron used to do on our old monolith. it fires up (fargate) tasks in docker containers. We've got a task that runs every hour and it's quite important to us. I want to know if it crashes or fails to run for any reason (eg the java scheduler fails, or someone turns the task off).
I'm looking for a service that will alert me if it's not notified. I want to call the notification system every time the script runs successfully. Then if the alert system doesn't get the "OK" notification as expected, it shoots off an alert.
I figure this kind of service must exist, and I don't want to re-invent the wheel trying to build it myself. I guess my question is, what's it called? And where can I go to get that kind of thing? (we're using AWS obviously and we've got a pagerDuty account).
We use this approach for these types of problems. First, the task has to write a timestamp to a file in S3 or EFS. This file is the external evidence that the task ran to completion. Then you need an http based service that will read that file and calculate if the time stamp is valid ie has been updated in the last hour. This could be a simple php or nodejs script. This process is exposed to the public web eg https://example.com/heartbeat.php. This script returns a http response code of 200 if the timestamp file is present and valid, or a 500 if not. Then we use StatusCake to monitor the url, and notify us via its Pager Duty integration if there is an incident. We usually include a message in the response so a human can see the nature of the error.
This may seem tedious, but it is foolproof. Any failure anywhere along the line will be immediately notified. StatusCake has a great free service level. This approach can be used to monitor any critical task in same way. We've learned the hard way that critical cron type tasks and processes can fail for any number of reasons, and you want to know before it becomes customer critical. 24x7x365 monitoring of these types of tasks is necessary, and helps us sleep better at night.
Note: We always have a daily system test event that triggers a Pager Duty notification at 9am each day. For the truly paranoid, this assures that pager duty itself has not failed in some way eg misconfiguratiion etc. Our support team knows if they don't get a test alert each day, there is a problem in the notification system itself. The tech on duty has to awknowlege the incident as per SOP. If they do not awknowlege, then it escalates to the next tier, and we know we have to have a talk about response times. It keeps people on their toes. This is the final piece to insure you have robust monitoring infrastructure.
OpsGene has a heartbeat service which is basically a watch dog timer. You can configure it to call you if you don't ping them in x number of minutes.
Unfortunately I would not recommend them. I have been using them for 4 years and they have changed their account system twice and left my paid account orphaned silently. I have to find a new vendor as soon as I have some free time.
I want to create a CFschedule task to run every 30 seconds and was wondering what would happen if a Task tries to fire up while the previous task is still running - does it wait or die or ???
cfschedule doesnt care much about the content of your jobs, think of it like cron daemon on Linux. Its just the scheduling agent.
If you need to ensure only one job of a certain type is running its up to you to implement some kind of busy/locking behavior, probably using some kind of a datastore such as a relational database, plain files (ala the presence of a file indicates in-process, then delete it right before job completion, this would be a good use of the onComplete handler), redis, etc.
Every time a customer completes a transaction a reminder workflow starts for that customer which tries to remind a customer about few actions that he/she has to perform. There are points in the flow where I know for sure that there is no task to be performed. So in this case I want the workflow to go to sleep for some time and come back to life later. I want this sleep feature to avoid database call as my decider does one database query every time it gets a task.
I have gone through the AWS documentation here . But found nothing there (Please point me to document if the feature exists). Does AWS-SWF provide such a feature. If it does not provide a feature of this type then what is smart and clean way of doing this.
A small example of flow I want to create :
1. End of transaction initiates a "simple workflow"
2. Decider gets a task. Decider decides to give it to a Customer
Reminder activity worker or PUT IT TO SLEEP.
3. The decider keeps poling but never gets the workflow till the sleep
time of work flow is over.
4. The sleep time is over so SWF starts giving it the decider which has
been polling all along.
Please tell me if you need any more clarification on this.
Use StartTimerDecision to create a timer.
Refer to the timer documentation.
http://docs.aws.amazon.com/amazonswf/latest/developerguide/swf-dg-timers.html
In my Django app, I need to implement this "timer-based" functionality:
User creates some jobs and for each one defines when (in the same unit the timer works, probably seconds) it will take place.
User starts the timer.
User may pause and resume the timer whenever he wants.
A job is executed when its time is due.
This does not fit a typical cron scenario as time of execution is tied to a timer that the user can start, pause and resume.
What is the preferred way of doing this?
This isn't a Django question. It is a system architecture problem. The http is stateless, so there is no notion of times.
My suggestion is to use Message Queues such as RabbitMQ and use Carrot to interface with it. You can put the jobs on the queue, then create a seperate consumer daemon which will process jobs from the queue. The consumer has the logic about when to process.
If that it too complex a system, perhaps look at implementing the timer in JS and having it call a url mapped to a view that processes a unit of work. The JS would be the timer.
Have a look at Pinax, especially the notifications.
Once created they are pushed to the DB (queue), and processed by the cron-jobbed email-sending (2. consumer).
In this senario you won't stop it once it get fired.
That could be managed by som (ajax-)views, that call system process....
edit
instead of cron-jobs you could use a twisted-based consumer:
write jobs to db with time-information to the db
send a request for consuming (or resuming, pausing, ...) to the twisted server via socket
do the rest in twisted
You're going to end up with separate (from the web server) processes to monitor the queue and execute jobs. Consider how you would build that without Django using command-line tools to drive it. Use Django models to access the the database.
When you have that working, layer on on a web-based interface (using full Django) to manipulate the queue and report on job status.
I think that if you approach it this way the problem becomes much easier.
I used the probably simplest (crudest is more appropriate, I'm afraid) approach possible: 1. Wrote a model featuring the current position and the state of the counter (active, paused, etc), 2. A django job that increments the counter if its state is active, 3. An entry to the cron that executes the job every minute.
Thanks everyone for the answers.
You can always use a client based jquery timer, but remember to initialize the timer with a value which is passed from your backend application, also make sure that the end user didn't edit the time (edit by inspecting).
So place a timer start time (initial value of the timer) and timer end time or timer pause time in the backend (DB itself).
Monitor the duration in the backend and trigger the job ( in you case ).
Hope this is clear.