Q. I have 1 AWS ec2 free instance where I have deployed my website. and now I want to start 1 cronjob (crontab) that will execute 1 PHP file once per minute. Do I have to purchase anything on AWS server or I can run cron every minutes on free instance too ?
Cronjobs have nothing to do with AWS Services. So think of it as a part of your website functionality, what you need to do is to log into your server through ssh and start writing the cronjob you need.
Related
I've been asked to migrate on-premises Python ETL scripts that live on a syslog box over to AWS. These scripts run as cron-jobs and output logs that a Splunk Forwarder parses and sends to our Splunk instance for indexing.
My initial idea was to deploy a Cloudwatch-triggered Lambda function that spins up an EC2 instance, runs the ETL scripts cloned to that instance (30 minutes), and then brings down the instance. Another idea was to containerize the scripts and run them as task definitions. They take approximately 30 minutes to run.
Any help moving forward would be nice; I would like to deploy this in IaaC, preferably in troposphere/boto3.
Another idea was to containerize the scripts and run them as task definitions
This is probably the best approach. You can include the splunk universal forwarder container in your task definition (ensuring both containers are configured to mount the same storage where the logs are held) to get the logs into splunk. You can schedule task execution just like lambda functions or similar. Alternatively to the forwarder container, if you can configure the logs to output to stdout/stderr instead of log files, you can just setup your docker log driver to output directly to splunk.
Assuming you don't already have a cluster with capacity to run the task, you can use a capacity provider for the ASG attached to the ECS cluster to automatically provision instances into the cluster whenever the task needs to run (and scale down after the task completes).
Or use Fargate tasks with EFS storage and you don't have to worry about cluster provisioning at all.
I have some etl python scripts on an EC2 instance which I run in order to fetch, process and load some data in a Postgresql DB that is hosted on the same EC2 instance.
The request was to host the DB on the EC2 instance instead of using RDS in order to save money.
The scripts take 1 hour to complete (because of some heavy processing involved which consumes 90% of the time).
I would like to build a solution to automate this task.
What I would like to do is to:
Turn on the EC2 instance at a given time;
Run the scripts inside the EC2 instance;
Turn off the EC2 instance after the scripts finish their job (which means after 1 hour);
I've seen some solutions with AWS System Manager and AWS Lambda, but they all seem outdated since the AWS interface changed.
I have a full deployment job that takes an ip of a running instance and deploys my system on it.
I currently hold an EC2 instance for automation tests that run every night, but the instance is expensive and im looking for a way to initiate it before the tests and terminate it after the test.
I looked for EC2 plugins that can help and the closest one was this but this is for making slaves and thats not what I want.
I want to be able to launch an EC2 instance, and pass its IP address to the automation tests job, then terminate that instance once done.
I started making a command line bash file for this, but this seems like too much work, and I thought maybe there is something im missing.
Your requirement is valid and amazon knows:
When you stop an instance, we shut it down. We don't charge usage for a stopped instance, or data transfer fees, but we do charge for the storage for any Amazon EBS volumes.
Reference :
- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Stop_Start.html
Here some approaches to start/stop your instaces
Amazon EC2 HTTP API
This is an api rest and you can perform a simple http request to start or stop your instance:
Amazon EC2 API Reference
Start Instance endpoint
https://ec2.amazonaws.com/?Action=StartInstances&...
Stop Instance endpoint
https://ec2.amazonaws.com/?Action=StopInstances&...
You can invoke this api from Jenkins in many ways : simple shell execution,groovy and scripted/declarative, pipelines.
AWS CLI
start instance
stop instance
Here more about how suspend instances using aws cli:
https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-suspend-resume-processes.html
Also with powershell: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Stop_Start.html
You can invoke this api from Jenkins in many ways : simple shell execution,groovy and scripted/declarative, pipelines.
AWS Instance Scheduler
In 2018, AWS launched the AWS Instance Scheduler, a new and improved scheduling solution that enables customers to schedule Amazon EC2 instances.
With this tool you can automatically start and stop the Amazon EC2 and Amazon RDS instances.
Reference :
https://aws.amazon.com/answers/infrastructure-management/instance-scheduler/
With this approach you don't need Jenkins :b
How do I schedule a docker image to be run periodically (hourly) using ECS and without having to use a continually running EC2 instance + cron? I have a docker image containing third party binaries and the python project.
The latter approach is not viable long-term as it's expensive for the instance to be running 24/7, while only being used for a small fraction of the day given invocation of the script only lasts ~3 minutes.
For AWS ECS cluster, it is recommended to have atleast 1 EC2 server running 24x7. Have you looked at AWS Fargate whether it can run your docker container?. Also AWS Batch?. If Fargate and AWS Batch are not possible then for your requirement, I would recommend something like this without ECS.
Build an EC2 AMI with pre-built docker and required softwares and libraries.
Have AWS Instance Scheduler to spin up a EC2 server every hour and as part of user data, start a docker container with image you mentioned.
https://aws.amazon.com/answers/infrastructure-management/instance-scheduler/
If you know your task execution time maybe 5min. After 8 or 10min then bring server down with scheduler.
Above approach will blindly start a EC2 and stop it without knowing whether your python work is done successfully. We can still improve above with Lambda and CloudFormation templates combination. Let me know your thoughts :)
Actually it's possible to schedule the launch directly in CloudWatch defining a rule, as explained in
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduled_tasks.html
This solution is cleaner, because you will not need to worry about the execution time: once finished, the Task will just terminate and a new one will be spawned on the next cycle
I'm currently working on scrapping tool for information analysis and I put a cron job on AWS EC2 instance (Ubuntu 14 TLS Server).
The cron job runs by executing Laravel artisan command.
Following is what I have entered in crontab -e.
0 20 * * * php /var/www/html/artisan data:get
But this doesn't run everyday and I found cron service has stopped for no reason.
Is it possible to have cron job on AWS Ec2 instance?
If not, what's the solution?
Yes, cron should just run on Amazon EC2 instances.
Did you look at the answers to this question ?
Cannot get cron to work on Amazon EC2?