I would like to know how to execute emr jobs & redshift copy commands from a separate ec2 linux instance?
Configuration:
2 eme ec2 running (one for master & one for Task)
1 ec2 linux instance
1 ec2 windows
I am able to execute jobs (s3 file movement, emr script residing in s3, Hard code copy commands to load into redshift]from ec2 windows (donwloaded aws cli).
Regards
Sanjeeb
You can use the EMR CLI to submit a step.
Here's more documentation from the EMR docs on how to do this.
Related
I'm trying to run a EMR cluster with custom AMI because the bootstrap time take 13 minutes. I have created an AMI from a m5.large instances with all of the software installed. This instances is based on amazon linux 2 AMI. I have created the image (clicking in action, image, create image). When I run the cluster it starts the instances on EC2 but the cluster keeps on starting.
How can I solve the problem?
I am not good at writing a shell script/Bootstrap action for EMR. Can I able to use a preconfigured AMI snapshot for creating the cluster?
You can use a preconfigured AMI for EMR. However, there are some restrictions in that you must start with a supported EMR AMI. I have done this many times to create encrypted root volumes for EMR (copying the AMI and enabling encryption).
Amazon EMR now supports launching clusters with custom Amazon Linux AMIs
I need to create a AWS EMR cluster for spark job with one master and 4 core nodes with auto scaling. I need to have different Instance types for master and core with Ubuntu 16.0 installed on it. So do I need to create two AMI's for this master and slave.
Amazon EMR has its own library of AMIs. You can select the AMI version when launching the cluster.
You can create a custom AMI, but it must be based on Amazon Linux.
See: Using a Custom AMI - Amazon EMR
If you wish to launch a Hadoop cluster with your own Ubuntu AMI, you cannot use the Amazon EMR service. You will need to launch and configure it yourself on Amazon EC2 instances.
Is there any way to install exe/MSI agents in AWS EC2 instances in an automated way?? In specific, I am looking for a counterpart of Azure's Custom Script Extension. [Free of cost]
Scenario:
I want to install BigFix and Datadog agents on 1000 Ec2 instances, this is a one time job, so I am not looking for any solution that involves Chef / Puppet, etc.,
Yes, you can pass a script to the instance that will be executed on the first boot (but not thereafter). It is often referred to as a User Data script.
See:
Running Commands on Your Windows Instance at Launch - Amazon Elastic Compute Cloud
Running Commands on Your Linux Instance at Launch - Amazon Elastic Compute Cloud
If you wish to install after the instance has started, use the AWS Systems Manager Run Command.
Background:
Services used: ec2, autoscaling, s3, sqs, cloudwatch
AMI and Environement: Windows 64-bit
Network: IAM and security group attached
Job: Run a script which starts a program (.exe) which is loaded from S3
I have an auto scale option that launches a N number of Instances. The user data script is based on aws CLI and few commands in powershell. I was expecting the instances to execute my script upon their initialization. Note that some of the tasks before the Job is to first download the aws CLI using powershell, because the rest of the script is based on aws commands
What am I missing ? I thought the launch should start the script in user-data.
Note that this script was tested on an instance with the same configurations (VPC, Security Group, etc..)