I have architecture created using CloudFormation utilizing Windows 2016 EC2 server and S3, written in JSON. I have 7 executables uploaded onto my S3 bucket. I can manually silently install everything from a Powershell for AWS prompt, once I Remote into the EC2. I can do it one at a time, and even have it in a .ps1 file and run it in Powershell for AWS and it runs correctly.
I am now trying to get this to install silently when the EC2 instance is created. I just can't do it and I can't understand why. The JSON code looks correct. As you can see, I first download everything from the S3 bucket, switch to the c:\TEMP directory where they were all downloaded, then run the executables in unattended install mode. I don't get any errors in my CloudFormation template. It runs "successfully." The problem is that nothing happens. Is it a permissions thing? Any help is welcome and appreciated. Thanks!
Under the AWS::EC2::Instance section I have the UserData section looking something like this (I shortened the executable names below):
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
"<powershell>\n",
"copy-S3Object -BucketName mySilentInstallBucket -KeyPrefix * -LocalFolder c:\\TEMP\\",
"\n",
"cd c:\\TEMP\\",
"\n",
"firefox.exe -S ",
"\n",
"notepadpp.exe /S",
"\n",
"Git.exe /SILENT",
"\n",
"</powershell>"
]]}}
This troubleshooting doc will cover the various reasons you may not be able to connect to S3: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-access-s3-bucket/
To connect to your S3 buckets from your EC2 instances, you need to do
the following:
Create an AWS Identity and Access Management (IAM) profile role that grants access to Amazon S3.
Attach the IAM instance profile to the instance.
Validate permissions on your S3 bucket.
Validate network connectivity from the EC2 instance to Amazon S3.
Validate access to S3 buckets.
The CloudFormation template won't fail based on UserData execution exceptions.
Related
I am looking at using jenkins to deploy a war file to an EC2 instance. I have set up similar before. Creating an EC2 instance, a S3 Bucket and a Code Deploy application. The way that worked was that :
1)zip up load the war/jar into a S3 Bucket.
2) Use AWS steps createDeployment to deploy the zip file from the S3 Bucket to the EC2. This would also involve creating a appspec.yml and scripts to set up the environment.
But have been told there is another way. that does not need setting up a code deploy.
I have created an Ec2 instance, set up a docker container inside it, with all the environment settings.
And what I would like to do is load my zip file into the EC2. That I dont need a AWS codedeploy application.
is this correct, is there a AWS CLI command to simply load a zip file into the EC2 instance.
Thank you for any help.
You can copy from an s3 bucket
To copy files from a S3 bucket to EC2 instance,
Create an IAM role with S3 write access or admin access
Map the IAM role to an EC2 instance
Install AWS CLI in EC2 instance
Run the AWS s3 cp command to copy the files from S3 to EC2
To copy the files from S3 to EC2, Keep the source as the bucket URL and the destination to your local directory or filename
To copy the files from S3 to EC2
aws s3 cp s3://<S3BucketName> <Fully Qualified Local filename/Directory>
In the previous command, you can see the difference. Here the source is S3 Bucket URL and the destination is a local file name or directory name.
Background:
I have jenkins installed in AWS Account #1 (account1234) and it has iam Role-jenkins attached to it. There's github configured with Jenkins.
When I click build job in Jenkins, jenkins pulls all the files from github and can be found in
/var/lib/jenkins/workspace/.
There's an application running in AWS Account #2 (acccount5678) in an ec2 instance (i-xyz123) and the project files are in /home/app/all_files/ ; This ec2 instance role has app-role attached to it.
What I'm trying to achieve:
When I click build, I want jenkins to push files from account 1234 to account 5678 by opening an SSM session from Jenkins ,to the ec2 instance on which app is running.
What I tried:
In the jenkins as part of build shell script I added:
aws ssm send-command --region us-east-1 --instance-ids i-xyz123 --document-name AWS-RunShellScript --comment IP config --parameters commands=ifconfig --output text
to test it. (If successful, I want to pass cp var/lib/jenkins/workspace/ /home/app/all_files/ as the command)
Error:
An error occurred (AccessDeniedException) when calling the SendCommand operation: User: arn:aws:sts::account1234:assumed-role/Role-Jenkins/i-01234abcd is not authorized to perform: ssm:SendCommand on resource: arn:aws:ec2:us-east-1:account1234:instance/i-xyz123
Build step 'Execute shell' marked build as failure
Finished: FAILURE
Issue 1: instance/i-xyz123 is in account5678 but error above shows ssm trying to connect to instance in account1234 ( which shouldn't be happening)
Q1: How do I update my command so that it tries to open an ssm session
with instance/i-xyz123 present in account5678 to accomplish what I'm
trying to do.
I believe I would also need to make each role added as a trusted relationship to the other.
(Note I want to do it via sessions manager as I won't have to deal with credentials of any sort)
If I've understood correctly then you're right; to interact with the resources in account5678, there needs to be a trust relationship so that the Jenkins account can assume the relevant role in account5678 and call SSM from there.
Once you've configured the role relationship (ref: IAM cross account roles )
You should be able to achieve what you need by assuming the role first in your shell script and then running the ssm command. That way Jenkins will use the temp creds and execute the command in the correct account (5678).
This site steps through it pretty well :
Tom Gregory - Jenkins Assume Role
If you just cmd/ctrl f on that page ^ and search for 'shell' you should get to the section you need. Hope this somewhat helps.
I am trying to download codedeploy following AWS docs:
Install the CodeDeploy agent for Windows Server
To do it, i need to run following command which basically downloads the file on temporary folder for my region us-east-1:
powershell.exe -Command Read-S3Object -BucketName aws-codedeploy-us-east-1 -Key latest/codedeploy-agent.msi -File c:\temp\codedeploy-agent.msi
On that instance I attached an IAM Role which has AmazonS3FullAccess policy. I'm getting this error when I execute the command:
Read-S3Object : No credentials specified or obtained from persisted/shell defaults.
I'm aware I could fix this adding personal credentials but since this is not considered a good practice I would like to download the client without tackling it this way.
Just in case someone has a similar problem.
This was caused because I previously migrated a custom windows AMI to a different AWS region.
The instance profile metadata server was pointing to old region server. You need to change that to point to the new region:
Import-Module (Join-Path $env:ProgramData 'Amazon\EC2-Windows\Launch\Module\Ec2Launch.psd1')
Add-Routes
I have been searching solution for this task, all I find CLI approaches which I don't want.
I simply want:
I have an S3 Bucket, which has one private file, file can be an image/zip file anything.
And I want when I launch any EC2 instance it should have taken that file from S3 bucket to EC2 instance directory.
And for this, I want to use only EC2 User Data Approach.
The User Data field in Amazon EC2 passes information to the instance that is accessible to applications running on the instance.
Amazon EC2 instances launched with Amazon-provided AMIs (eg Amazon Linux 2) include a program called Cloud-Init that looks at the User Data and, if a script is provided, runs that script the first time that the instance is booted.
Therefore, you can configure a script (passed via User Data) that will run when the instance is first launched. The script will run as the root user. Your script could copy a file from Amazon S3 by using the AWS Command-Line Interface (CLI), like this:
#!
aws s3 cp s3://my-bucket/foo.txt /home/ec2-user/foo.txt
chown ec2-user foo.txt
Please note that you will need to assign an IAM Role to the instance that has permission to access the bucket. The AWS CLI will use these permissions for the file copy.
You mention that you do not wish to use the AWS CLI. You could, instead, write a program that calls the Amazon S3 API using a preferred programming language (eg Python), but using the CLI is much simpler.
I have an EC2 instance and I need to download a file from its D drive through my program. Currently, it's a very annoying process because I can't access the instance directly from my local machine. The way what I am doing now is running a script on the instance and the instance uploads the file I need to S3 and my program access S3 to read the file.
Just wonder whether there is any simple way to access the drive on the instance instead of going through S3?
I have used AWS DataPipeline and its task runner to execute scripts on a remote instance. The taskrunner waits for a pipeline event published to its worker group.
http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-using-task-runner.html
I use it to execute shell script and commands on a schedule. The script to run should be uploaded to S3, and the Data pipeline template specifies the script's path. Works great for periodic tasks. You can do anything you want on the remote box via the script.
You cannot directly download the file from EC2, but via s3( or maybe using scp command) from your remote ec2.
But to simplify this annoying process you can use AWS Systems Manager.
AWS Systems Manager Run Command allows you to remotely and securely run set of commands on EC2 as well on-premise server. Below are high-level steps to achieve this.
Attach Instance IAM role:
The ec2 instance must have IAM role with policy AmazonSSMFullAccess. This role enables the instance to communicate with the Systems Manager API.
Install SSM Agent:
The EC2 instance must have SSM agent installed on it. The SSM Agent process the run command requests & configure the instance as per command.
Execute command :
Example usage via AWS CLI:
Execute the following command to retrieve the services running on the instance. Replace Instance-ID with ec2 instance id.
aws ssm send-command --document-name "AWS-RunShellScript" --comment "listing services" --instance-ids "Instance-ID" --parameters commands="service --status-all" --region us-west-2 --output text
More detailed information: https://www.justdocloud.com/2018/04/01/run-commands-remotely-ec2-instances/