I would like to run some code whenever my AWS EC2 instance starts. The code should pull down data from Amazon S3, do some work on the data, and then copy the data back to S3. I created a makefile to do this, which works fine if I call it while I'm logged into the instance. I then placed a script in /ect/rc.local (this scripts runs every time the instance starts) that will call the makefile. This scripts successfully runs on instance startup. The problem I'm having is that when the makefile is called from the startup script it does not pull data from or copy data to s3. I read here that setting your access keys solves this problem with a Windows server, but this does not work for me. It looks like the code just stops when it tries to call any aws commands because in the log file the output is always the first line of code from the makefile. Below is what my log file says:
aws s3 sync s3:<s3 bucket to get data from> <location to save data to>
Here is the relelvant code from my makefile:
### Download all data
get_data:
aws s3 sync s3:<s3 bucket to get data from> <location to save data to>
### Copy data back to s3
copy_data_to_s3:
aws s3 sync <location of data to copy to s3> s3:<s3 bucket data is copied to>
Here is my script in /etc/rc.local:
#!/bin/bash
#
# rc.local
#
make -f <location of makefile>/Makefile > <location to save log file>/log.txt
exit 0
Any help would be appreciated.
When you configure the AWS command line, it stores the credentials and region in ~/.aws/.... But when you execute your command on startup, from rc.local, it's not running as you.
The problem you're seeing is the AWS CLI failing to find any credentials.
So, you have a couple of options:
Option 1: (Preferred)
Don't configure locally-stored AWS credentials. Instead, launch your EC2 instance using an IAM role. When you do this, no credentials are required to be store on your instance. The AWS CLI will simply "find" the credentials in the IAM Role.
Option 2: (May work)
From rc.local, run your scripts under your account. This way, the stored credentials may be found.
The more secure way to do what you want to do is using Option 1.
Related
I am a newbie on aws and server in general. Hope someone can help me :)
What I want to do:
I have some image files on remote server and I would like to copy them to aws s3 bucket.
I can ssh this remote server once I ssh intermediate server.
so it looks like
me(local) <-----> server1 <----> server2 where image files on
What I did:
*I first ssh server1.
*I installed aws-cli on that server, and set configure.
*Then I entered aws s3 sync <username-I-use-for-server2>#<server2-IP>:/images/pc s3://<s3 bucket name>
Problem:
What I expected was to get password prompt from server2 then sync the files in images/pc to s3 bucket
but I got the error saying The user-provided path <username-I-use-for-server2>#<server2-IP>:/images/pc does not exist.
can anyone tell me What am I doing wrong?
I've checked that I can ssh server2 and that there's files inside /images/pc as well.
Any help would be appreciated!!
It is expected that this is a local path to the current server.
The available combinations are <LocalPath> <S3Uri> or <S3Uri> <LocalPath> or <S3Uri> <S3Uri>.
You will need to run this command against a local path on the server you're trying to copy the objects from.
Remember that all this command is actually doing is running aws s3api putObject under the hood along with some management to perform operations such as recursion on your local disk. It does not handle any SSH interactions.
If you wanted to share any assets between servers you would need to mount them between servers (for example using an NFS mount) before you are able to sync them up, otherwise run the command on the other server.
I have been searching solution for this task, all I find CLI approaches which I don't want.
I simply want:
I have an S3 Bucket, which has one private file, file can be an image/zip file anything.
And I want when I launch any EC2 instance it should have taken that file from S3 bucket to EC2 instance directory.
And for this, I want to use only EC2 User Data Approach.
The User Data field in Amazon EC2 passes information to the instance that is accessible to applications running on the instance.
Amazon EC2 instances launched with Amazon-provided AMIs (eg Amazon Linux 2) include a program called Cloud-Init that looks at the User Data and, if a script is provided, runs that script the first time that the instance is booted.
Therefore, you can configure a script (passed via User Data) that will run when the instance is first launched. The script will run as the root user. Your script could copy a file from Amazon S3 by using the AWS Command-Line Interface (CLI), like this:
#!
aws s3 cp s3://my-bucket/foo.txt /home/ec2-user/foo.txt
chown ec2-user foo.txt
Please note that you will need to assign an IAM Role to the instance that has permission to access the bucket. The AWS CLI will use these permissions for the file copy.
You mention that you do not wish to use the AWS CLI. You could, instead, write a program that calls the Amazon S3 API using a preferred programming language (eg Python), but using the CLI is much simpler.
My aim is to launch an instance such that a start-up script is triggered on boot-up to download some configuration files stored in AWS S3. Therefore, in the start-up script, I am setting the S3 bucket details and then, triggering a config.sh where aws s3 sync does the actual download. However, the aws command does not work - it is not found for execution.
User data
I have the following user data when launching an EC2 instance:
#!/bin/bash
# Set command from https://stackoverflow.com/a/34206311/919480
set -e -x
export S3_PREFIX='bucket-name/folder-name'
/home/ubuntu/app/sh/config.sh
The AWS CLI was installed with pip as described in the documentation.
Observation
I think, the user data script is run with root user ID. That is why, in the user data I have /home/ubuntu/ because $HOME did not resolve into /home/ubuntu/. In fact, the first command in config.sh is mkdir /home/ubuntu/csv which creates a directory with owner as root!
So, would it be right to conclude that, the user data runs under root user ID?
Resolution
Should I use REST API to download?
Scripts entered as user data are executed as the root user, so do not use the sudo command in the script.
See: Running Commands on Your Linux Instance at Launch
One solution is to set the PATH env variable to include AWS CLI (and add any other required path) before executing AWS CLI.
Solution
Given that, AWS CLI was installed without a sudo pip, the CLI is not available for root. Therefore, to run with ubuntu user, I used the following user data script:
#!/bin/bash
su ubuntu -c '$HOME/app/sh/config.sh default`
In config.sh, the argument default is used to build the full S3 URI before invoking the CLI. However, the invocation was successful only with the full path $HOME/.local/bin/aws despite the fact that aws can be accessed with normal login.
I am trying to get some files from S3 on startup in an EC2 instance by using a User Data script and the command
/usr/bin/aws s3 cp ...
The log tells me that permission was denied and I believe it is due to aws cli not finding any credentials when executing the user data script.
Running the command with sudo after the instance has started works fine.
I have run aws configure both with sudo and without.
I do not want to use cronjob to run something on startup since I am working with an AMI and often need to change the script, therefore it is more convenient for me to change the user data instead of creating a new AMI everytime the script changes.
If possible, I would also like to avoid writing the credentials into the script.
How can I configure awscli in such a way that the credentials are used when running a user data script?
I suggest you remove the AWS credentials from the instance/AMI. Your userdata script will be supplied with temporary credentials when needed by the AWS metadata server.
See: IAM Roles for Amazon EC2
Clear/delete AWS credentials configurations from your instance and create an AMI
Create a policy that has the minimum privileges to run your script
Create a IAM role and attach the policy you just created
Attach the IAM role when you launch the instance (very important)
Have your userdata script call /usr/bin/aws s3 cp ... without supplying credentials explicitly or using credentials file
You can configure your EC2 instance to receive a pre-defined IAM Role that has its credentials "baked-in" to the instance that it fetches upon instantiation, which it turn will allow it to call aws-cli commands in your User Data script without the need to configure credentials at all.
Here's more info on IAM Roles for EC2:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
It's worth noting here that you'll need to attach the appropriate policies to the IAM Role that you assign to your instance in order for the aws-cli commands to succeed. More information on that can be found here:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#working-with-iam-roles
I'm trying to create a reusable delegation set to use as whitelisted nameservers for my domains, using aws cli on Mac OS X. My AWS credentials (those of an IAM profile I created for that purpose with full administrator privileges, an location set to us-east-1) were correctly entered during setup and accepted by the system.
When entering the command
$ aws route53 create-reusable-delegation-set --caller-reference [CALLER-REFERENCE] --hosted-zone-id [HOSTED_ZONE] --generate-cli-skeleton
the request is successful and I get the response:
{
"CallerReference": "",
"HostedZoneId": ""
}
But when I remove --generate-cli-skeleton and enter
aws route53 create-reusable-delegation-set --caller-reference [CALLER-REFERENCE] --hosted-zone-id [HOSTED_ZONE]
I get this:
An error occurred (InvalidClientTokenId) when calling the CreateReusableDelegationSet operation: The security token included in the request is invalid.
I reality, my IAM credentials, despite being valid, and despite the profile I am using (donaldjenkins) having full administrator privileges, are refused systematically in all aws services and for all commands, not just Route53.
I've been unable to pinpoint the cause of this despite extensive research. Any suggestions gratefully receieved.
Deleting my credentials file (Linux, macOS, or Unix: ~/.aws Windows: %UserProfile%\.aws) then running aws configure again worked for me
The solution is to delete existing credentials for the IAM user and issue new ones. For some reason the credentials recorded during the initial setup of aws cli never worked properly, but overwriting them with new ones removed the issue instantly.
I had the same exact issue.
I'm running NodeJS on my local environment, and trying to deploy to Amazon using code deploy and some other aws tools.
What worked for me was to delete the current config and credentials folder, regnerate a new key and use. THis was after i originally installed aws cli and added the keys, had to add the keys again.
Depending on your folder structure, navigate to your home directory.
On mac if you open a new terminal, it should show your current home directory: "/Users/YOURNAME"
cd .aws
rm -rf config
rm -rf credentials
After you do this, go back to your home directory, then run:
"aws configure".
Enter your Key and secret key.
You can find more details here: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration under Quickly Configuring the AWS CLI