Hi i am trying to install code deploy agent in my ec2 agent but not able to succeed
I m following below steps
sudo apt-get update
sudo apt-get install awscli
sudo apt-get install ruby2.0
cd /home/ubuntu
sudo aws s3 cp s3://bucket-name/latest/install . --region region-name
sudo chmod +x ./install
sudo ./install auto
but ./install file is missing for me .
But I dont think its a problem with AMI as I used same steps with same AMI in different ec2 instance. Any one has any idea. please help me.
You need to fill in the bucket name and region name in sudo aws s3 cp s3://bucket-name/latest/install . --region region-name. If you are in us-east-1 you would use: aws-codedeploy-us-east-1 and us-east-1.
All the buckets follow that pattern so you can fill in another region if you are there instead.
See http://docs.aws.amazon.com/codedeploy/latest/userguide/how-to-set-up-new-instance.html for a complete list of buckets for each region.
Related
new to Terraform, trying somethings to get to know terraform a bit better.
i'm creating my infrastructure via AWS using an EC2 instance.
i managed to create the instance with SG and everything, but i came across some difficulties installing apps (such as docker and so on).
i was wondering if there's a way i can tell the terraform file to pre-install docker, is there any way?
i found some similar issues about the matter here:
Terraform plugins
but i can't figure out if it answers my question fully.
can anyone please advise?
Usually for EC2 instance you would define user_data. User data allows you to:
perform common automated configuration tasks and even run scripts after the instance starts.
Which means that you can write your user data to install any applications you want on the instance, and configure them how you wish. Alternatively, you can create custom AMI and lunch instance using that.
There are many approaches to this, of the somewhat more common and simple you can either:
(1) use a user_data script that will bootstrap the EC2 instance for you
A sample user_data script might look like below. Borrowed from github/gonzaloplaza.
#!/bin/bash
# Install docker
apt-get update
apt-get install -y cloud-utils apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
apt-get update
apt-get install -y docker-ce
usermod -aG docker ubuntu
# Install docker-compose
curl -L https://github.com/docker/compose/releases/download/1.21.0/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
And then embed it in your Terraform EC2 instance definition:
resource "aws_instance" "my-instance" {
ami = "ami-05210ba6bdf75db36" # Ubuntu 20.04LTS eu-central-1
instance_type = "m5.large"
key_name = "<key-path>"
user_data = "${file("install_docker.sh")}"
tags = {
Name = "Terraform"
}
}
(2) or use an AMI (virtual image) that has the requirement already met. AWS Marketplace enables you to use AMIs that other users built. You can check out for example Docker on Ubuntu AWS Marketplace link.
(3) And a more complex approach would be to build your own AMIs with for example Puppet and Packer. You can then upload those AMIs to your AWS Account and use with the instances you create.
References
Ubuntu AMI Locator for the Ubuntu 20.04LTS eu-central-1 AMI
github/gonzaloplaza for the userscript example
I want to be able to use Apache Sedona for distributed GIS computing on AWS EMR. We need the right bootstrap script to have all dependencies.
I tried setting up Geospark using EMR 5.33 using the Jars listed here. It didn't work as some dependencies were still missing.
I then manually set Sedona up on local, found the difference of Jars between Spark 3 and the Sedona setup and came up with following bootstrap script
#!/bin/bash
sudo pip3 install numpy
sudo pip3 install boto3 pandas findspark shapely py4j attrs
sudo pip3 install geospark --no-dependencies
sudo pip3 install apache-sedona
sudo aws s3 cp s3://emr_setup/apache-sedona-1.0.1-incubating-bin/sedona-python-adapter-2.4_2.11-1.0.1-incubating.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/apache-sedona-1.0.1-incubating-bin/sedona-viz-2.4_2.11-1.0.1-incubating.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/geospark_bin/postgresql-42.2.23.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/sedona-core-2.4_2.11-1.0.1-incubating.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/stream-2.7.0.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/orc-core-1.5.5-nohive.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/jersey-media-jaxb-2.22.2.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/hadoop-mapreduce-client-common-2.6.5.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/hadoop-mapreduce-client-shuffle-2.6.5.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/org.w3.xlink-24.0.jar /usr/lib/spark/jars/
sudo aws s3 cp s3:///emr_setup/spark_2.4_2.11_sedona_all_jars/minlog-1.3.0.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/jersey-client-2.22.2.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/xz-1.5.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/pyrolite-4.13.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/hadoop-yarn-common-2.6.5.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/curator-recipes-2.6.0.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/aopalliance-1.0.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/commons-configuration-1.6.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/commons-beanutils-1.7.0.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/gt-metadata-24.0.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/spark-unsafe_2.11-2.4.7.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/objenesis-2.5.1.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/commons-httpclient-3.1.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/stax-api-1.0-2.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/hk2-api-2.4.0-b34.jar /usr/lib/spark/jars/
sudo aws s3 cp s3://emr_setup/spark_2.4_2.11_sedona_all_jars/apacheds-i18n-2.0.0-M15.jar /usr/lib/spark/jars/
The EMR setup starts, but the attached notebooks to the script don't seem to be able to start. The master seems to fail for some reason.
Need help with preparing the right bootstrap script to install Apache Sedona on EMR 6.0.
Here is a complete tutorial of setting up Sedona on EMR EC2.
EMR version: 6.9.0.
Installed applications: Hadoop 3.3.3, JupyterEnterpriseGateway 2.6.0, Livy 0.7.1, Spark 3.3.0
I am using it together EMR Studio (notebooks).
In a S3 bucket, add a script that has the following content:
#!/bin/bash
# EMR clusters only have ephemeral local storage. It does not really matter where we store the jars.
sudo mkdir /jars
# Download Sedona jar
sudo curl -o /jars/sedona-python-adapter-3.0_2.12-1.3.1-incubating.jar "https://repo1.maven.org/maven2/org/apache/sedona/sedona-python-adapter-3.0_2.12/1.3.1-incubating/sedona-python-adapter-3.0_2.12-1.3.1-incubating.jar"
# Download GeoTools jar
sudo curl -o /jars/geotools-wrapper-1.3.0-27.2.jar "https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/1.3.0-27.2/geotools-wrapper-1.3.0-27.2.jar"
# Install necessary python libraries
sudo python3 -m pip install pandas geopandas==0.10.2
sudo python3 -m pip install attrs matplotlib descartes apache-sedona==1.3.1
When you create a EMR cluster, in the bootstrap action, specify the location of this script.
When you create a EMR cluster, in the software configuration, add the following content:
[
{
"Classification":"spark-defaults",
"Properties":{
"spark.yarn.dist.jars": "/jars/sedona-python-adapter-3.0_2.12-1.3.1-incubating.jar,/jars/geotools-wrapper-1.3.0-27.2.jar",
"spark.serializer": "org.apache.spark.serializer.KryoSerializer",
"spark.kryo.registrator": "org.apache.sedona.core.serde.SedonaKryoRegistrator",
"spark.sql.extensions": "org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions"
}
}
]
The key point is to use Sedona 1.3.1-incubating which can search for jars specified in spark.yarn.dist.jars property. spark.jars property is ignored for EMR on EC2 since it uses Yarn to deploy jars. See SEDONA-183
I have to mount the s3 bucket over docker container so that we can store its contents in an s3 bucket.
I found https://www.youtube.com/watch?v=FFTxUlW8_QQ&ab_channel=ValaxyTechnologies video which shows how to do the same process for ec2 instance instead of a docker container.
I am following the same steps as mentioned in the above link. Likewise, I have done the following things on the docker container:
(Install FUSE Packages)
apt-get install build-essential gcc libfuse-dev libcurl4-openssl-dev libxml2-dev mime-support pkg-config libxml++2.6-dev libssl-dev
git clone https://github.com/s3fs-fuse/s3fs-fus...
cd s3fs-fuse
./autogen.sh
./configure
make
make install
(Ensure you have an IAM Role with Full Access to S3)
(Create the Mountpoint)
mkdir -p /var/s3fs-demo-fs
(Target Bucket)
aws s3 mb s3://s3fs-demo-bkt
But when I trying to mount the s3 bucket using
s3fs s3fs-demo-bkt /var/s3fs-demo-fs -o iam_role=
I am getting the following messege
fuse: device not found, try 'modprobe fuse' first
I have looked over several solutions for this problem. But I am not able to resolve this issue. Please let me know how I can solve it.
I encountered the same problem. But later the issue was fixed by adding --privileged when running the docker run command
I am new to AWS and I am trying to deploy using AWS CodeDeploy from Github.
For that, I created my instance named CodeDeployDemo and attached the role and policy to the instance.
Policy ARN arn:aws:iam::378939197253:policy/CE2CodeDeploy9
My policy is:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:Get*",
"s3:List*"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
and also attached policy named AmazonEC2RoleforAWSCodeDeploy
I also installed CodeDeploy agent for my ubuntu step by step as following:
$chmod 400 Code1.pem
$ssh -i "Code1.pem" ubuntu#54.183.22.255
$sudo apt-get update
$sudo apt-get install awscli
$sudo apt-get install ruby2.0
$cd /home/ubuntu
$sudo aws s3 cp s3://aws-codedeploy-us-east-1/latest/install . --region us-east-1
$sudo chmod +x ./install
$sudo ./install auto
and then I create my application and deploy from GitHub to CodeDeploy using CodeDeployDefault.OneAtATime
But at final stage it shows following error:
Deployment failed: Because too many individual instances failed deployment,
too few healthy instances are available for deployment,
or some instances in your deployment group are experiencing problems.
(Error code: HEALTH_CONSTRAINTS)
NOTE: My only one instance is running when my deployment is running.I stopped other instances.
Please help me to find solution for this. THANKS IN ADVANCE.!!
This happens because the codeDeploy checks health of the ec2 instances by hitting instances. Before deployment, you need to run below bash script on the instances and check if the script worked. httpd service must be started. Reboot the instance.
#!/bin/bash
sudo su
apt-get update -y
apt-get install apache2 -y
apt-get install ruby2.0
apt-get install awscli
cd ~
aws s3 cp s3://aws-codedeploy-us-east-1/latest/install . --region us-east-1
chmod +x ./install
./install auto
echo 'hello world' > /var/www/html/index.html
hostname >> /var/www/html/index.html
update-rc.d apache2 defaults
service apache2 start
i have s3cmd and EC2 api pre configured AMI. While creating new instance with user data for downloading files from S3 bucket, i face some following problems.
in user data i have code for
- creating new directory on new instance.
- downloading file from AWS S3 bucket.
Script is
#! /bin/bash
cd /home
mkdir pravin
s3cmd get s3://bucket/usr.sh >> temp.log
But in above script , mkdir pravin creates new directory with name pravin but s3cmd get s3://bucket/usr.sh not downloads file from AWS S3 bucket.
it also creates temp.log but it is empty.
So how i can solve this problem ?
An alternative solution would be to use an instance that has an IAM role assigned to it and the aws-cli, which would require that you have Python installed. All of this could be accomplished by inserting the following in the user-data field for your instance:
#!/bin/bash
apt-get update
apt-get -y install python-pip
apt-get -y install awscli
mkdir pravin
aws s3 cp s3://bucket/usr.sh temp.log --region {YOUR_BUCKET_REGION}
NOTE: The above is applicable for Ubuntu only.
And then for your instances IAM role you would attach a policy like so:
{
"Version":"2012-10-17",
"Statement":[{
"Effect":"Allow",
"Action":"s3:*",
"Resource":"arn:aws:s3:::YourBucketName/*"
}
]
}
I suspect that the user running the user data script lacks a .s3cfg file. You may need to find a way to indicate the location of the file when running this script.