I'm currently working on scrapping tool for information analysis and I put a cron job on AWS EC2 instance (Ubuntu 14 TLS Server).
The cron job runs by executing Laravel artisan command.
Following is what I have entered in crontab -e.
0 20 * * * php /var/www/html/artisan data:get
But this doesn't run everyday and I found cron service has stopped for no reason.
Is it possible to have cron job on AWS Ec2 instance?
If not, what's the solution?
Yes, cron should just run on Amazon EC2 instances.
Did you look at the answers to this question ?
Cannot get cron to work on Amazon EC2?
Related
I want to have my ec2 instances to start running some automation script within the server once they are launched in the auto scaling group. This script will need to run every minute so userdata is probably not a good option. I wonder if there is a way to do this with ssm document. Any ideas would be much appreciated!
Do you need to store the logs of the script on either S3 or Cloudwatch? Then use an EventBridge rule and a SSM Run Command document.
Do you not care about such logs? Then use your operating system native tools: cron on Linux or task scheduler on Windows.
This question already has answers here:
Aws Ec2 run script program at startup
(5 answers)
Closed 1 year ago.
I am trying to run a python script on ec2 instance . The python file is residing on s3.
I am able to run manually from ec2 instance using iam role which allow access to s3 folder and files.
The question is , how can i automate the start and stop of ec2 instance whenever needed and how to invoke /pass a python file to run upon starting the ec2 instance and stop the instance once the python files completes the execution.
Thanks,
Nikhil
Your requirements seem to be:
Schedule an Amazon EC2 instance to start at a specific time every day
The instance should run a Python script after starting
When the Python script has finished running, Stop the instance
Start EC2 instance on a schedule
You can use Amazon EventBridge to trigger an AWS Lambda function on a schedule.
You can code the Lambda function to call StartInstances() on the EC2 instance to Start it.
Run a script on startup
Install a script into the /var/lib/cloud/scripts/per-boot/ directory. This script can download the Python program from S3 and then run it.
When the EC2 instance starts up, it will automatically run any script in that directory.
Stop the instance when the script is finished
At the end of the script, add the command:
shutdown -h now
This will turn off the instance and place it in the Stopped state.
(This assume that the script is running as root. If it is running as another user, it will need to use sudo shutdown -h now.)
EC2 instances use cloudinit which you can customize to run a given script on each boot. You can use use regular os tools from python to shutdown your instance (e.g. shutdown -h now).
Here another alternative could be to use lambda function instead of EC2 instance to run the python script if maximum execution time of script is less than 15 minutes. Go serverless with AWS lambda rather than EC2. just add your script code in AWS lambda and schedule lambda function from AWS event bridge to invoke it.
Q. I have 1 AWS ec2 free instance where I have deployed my website. and now I want to start 1 cronjob (crontab) that will execute 1 PHP file once per minute. Do I have to purchase anything on AWS server or I can run cron every minutes on free instance too ?
Cronjobs have nothing to do with AWS Services. So think of it as a part of your website functionality, what you need to do is to log into your server through ssh and start writing the cronjob you need.
I want to automate a redshift insert query to be run every day.
We actually use Aws environment. I was told using lambda is not the right approach. Which is the best ETL process to automate a query in Redshift.
For automating SQL on Redshift you have 3 options (at least)
Simple - cron
Use a EC2 instance and set up a cron job on that to run your SQL code.
psql -U youruser -p 5439 -h hostname_of_redshift -f your_sql_file
Feature rich - Airflow (Recommended)
If you have a complex schedule to run then it is worth investing time learning and using apache airflow. This also needs to run on a server(ec2) but offers a lot of functionality.
https://airflow.apache.org/
AWS serverless - AWS data pipeline (NOT Recommended)
https://aws.amazon.com/datapipeline/
Cloudwatch->Lambda->EC2 method described below by John Rotenstein
This is a good method when you want to be AWS centric, it will be cheaper than having a dedicated EC2 instance.
One option:
Use Amazon CloudWatch Events on a schedule to trigger an AWS Lambda function
The Lambda function launches an EC2 instance with a User Data script. Configure Shutdown Behavior as Terminate.
The EC2 instance executes the User Data script
When the script is complete, it should call sudo shutdown now -h to shutdown and terminate the instance
The EC2 instance will only be billed per-second.
Redshift now supports scheduled queries natively: https://docs.aws.amazon.com/redshift/latest/mgmt/query-editor-schedule-query.html
You can use boto3 and psycopg2 to run the queries by creating a python script
and scheduling it in cron to be executed daily.
You can also try to convert your queries into Spark jobs and schedule those jobs to run in AWS Glue daily. If you find it difficult, you can also look into Spark SQL and give it a shot. If you are going with Spark SQL, keep in mind the memory usage as Spark SQL is pretty memory intensive.
I'm running a instance in amazon AWS and it runs non-stop everyday. I'm using ubuntu ec2 instance which is running Apache, Mirthconnect tool and LAMP server. I want to run this instance only on particular time duration of a day. I prefer not use any additional AWS services such as cloud-watch . Is there a way we could acheive this?.
The major purpose is for using Mirthconnect fetching data from mysql database
There are 3 solutions.
AWS Data Pipeline - You can schedule the instance start/stop just like cron. It will cost you one hour of t1.micro instance for every start/stop
AWS Lambda - Define a lambda function that gets triggered at a pre defined time. Your lambda function can start/stop instances. Your cost will be very minimal or $0
Write a shell script and run it as a cron job or run it on demand. The script will have AWS CLI command to start and stop the instance.
I used Data Pipeline for a long time before moving to Lambda. Data Pipeline is very trivial. Just paste the AWS CLI commands to stop and start instances. Lambda is more involved.
I guess for that you'll need another machine which is on 24x7. On which you can write cron job in python using boto or any other language like bash.
I don't see how you start a instance in stopped state without using any other machine.
Or you can have a simple raspberry pi on at your home which does the ON-OFF work for you using AWS CLI or simple Python. How about that? ;)