New to AWS so any help would be appreciated.
I'm attempting to run Jenkins through Docker on AWS. I found this article https://docs.aws.amazon.com/aws-technical-content/latest/jenkins-on-aws/containerized-deployment.html
Can anyone share a better step-by-step tutorial to achieve this? the page above seems incomplete.
It talks about "The Dockerfile should also contain the steps to install the Jenkins Amazon ECS plugin" but does not show how to go about installing the plugin using the Dockerfile.
thanks
Please follow below steps:
Launch EC2 cluster according to your needs.
Install docker in you local machine. For example, for ubuntu (sudo apt-get isntall docker.io)
systemctl start docker
Create new folder for our jenkins docker. Create new Dockerfile inside it with following contents.
FROM Jenkins
COPY plugins.txt /usr/share/jenkins/plugins.txt
RUN /usr/local/bin/plugins.sh /usr/share/jenkins/plugins.txt
Create plugins.txt in same folder and add below line
amazon-ecs:1.3
Login to ECR using aws cli. Configure aws first with your credentials.
aws ecr get-login --region <REGION>
Run the output returned from above command to docker login.
sudo docker build -t jenkins_master .
sudo docker tag jenkins_master:latest <AWS ACC ID>.dkr.ecr.<REGION>.amazonaws.com/jenkins_master:latest
Create repository in ECR for this image
aws ecr create-repository --repository-name jenkins_master
Push the image in AWS ECR.
sudo docker push <AWS ACC ID>.dkr.ecr.<REGION>+.amazonaws.com/jenkins_master:latest
Our Jenkins docker image is ready. But data stored by this Jenkins server will not be persistent. To store data permanently, we will create another docker image which will create a volume with mount point. For that, create new directory for this new docker image and inside it create another Dockerfile with below content.
FROM Jenkins
VOLUME ["/var/jenkins_home"]
Again follow same commands to push this new repository to ECR.
sudo docker build -t jenkins_dv .
sudo docker tag jenkins_dv:latest <AWS ACC ID>.dkr.ecr.<REGION>.amazonaws.com/jenkins_dv:latest
aws ecr create-repository --repository-name jenkins_dv
sudo docker push <AWS Account Number>.dkr.ecr.<REGION>.amazonaws.com/jenkins_dv:latest
Now our images are ready. We will use this images to run them as service on our ECS cluster. For that we need to install ecs-cli using below command for linux.
sudo curl -o /usr/local/bin/ecs-cli https://s3.amazonaws.com/amazon-ecs-cli/ecs-cli-linux-amd64-latest
Create a new txt file with below contents which will have jenkins configuration.
jenkins_master:
image: jenkins_master
cpu_shares: 100
mem_limit: 2000M
ports:
- "8080:8080"
- "50000:50000"
volumes_from:
- jenkins_dv
jenkins_dv:
image: jenkins_dv
cpu_shares: 100
mem_limit: 500M
15. Finally push this service using above file to your newly created cluster.
ecs-cli compose --file docker_compose.txt service up --cluster <cluster_name>
Hope this helps!
Related
I am wondering something about PySpark applications. If I container a PySpark program called my_spark_script.py, can I just execute it inside the Docker container? I mean to ask, is a Docker file like this valid:
WORKDIR /app
COPY . .
RUN pip3 install -r requirements.txt
CMD spark-submit --master yarn --deploy-mode cluster--num-executors 2 my_spark_script.py // <-- ???
And I can build it as:
docker build -t my_docker_image .
and then run it as
docker run -d my_docker_image
I am wondering if this can be run on AWS EC2 or AWS EMR or something else like this? Would it work?
I just dont know how the container CMD works in relation to environment like EC2 or EMR. Please help!
Amazon Elastic Container Service (ECS) is a managed AWS service to run Docker containers. ECS provides the Fargate launch type, which is a serverless platform with which a container service is run on Docker containers instead of EC2 instances.
To build the source code into a Docker image you can use AWS CodeBuild service, and AWS CodePipeline for Continuous Integration, please check the following example, here.
I can't find a right solution here.
I got installed Jenkins on EC2 where I added a pipeline with link to the github where I have repository with docker-compose.yml file.
How to install this image of docker into ECR on AWS.
Documentation on pushing to ECR is available here
The steps are:
Authenticate docker with ECR
Build your docker image
tag the docker image with the required ECR format:
docker tag <image>:<version> <aws_account_id>.dkr.ecr.<region>.amazonaws.com/<image>:<version>
docker push <aws_account_id>.dkr.ecr..amazonaws.com/
The simplest way to accomplish this is with a shell script. There are also jenkins plugins that will do this for you.
I got a question - I have a docker image running locally on my Mac. - I'm trying to export that local image and deploy on AWS elasticbean stalk env.
Should I use docker export command which outputs it as a tar file then upload to AWS? or should it be in a different non compressed format?
I already tried the above and docker export it as a tar file but AWS didn't like that so what approach should I take here?
You can create a repository in your aws ECR (Amazon Elastic Container Registry) and push your local image to that repo
aws ecr get-login --no-include-email --region us-east-2
docker tag test-pod:latest 24533xxxxx.dkr.ecr.us-east-2.amazonaws.com/test:latest
docker push 24533xxxxx.dkr.ecr.us-east-2.amazonaws.com/test:latest
I setup a docker registry (ECR) on AWS. From my gitlab repository I'd like to setup a CI to automatically create images and push them to the repository.
I was following the following tutorial to setup everything, but when running the example, I receive the error
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
My yml file looks like this
image: docker:latest
variables:
REPOSITORY_URL: <aws-url>/<registry>/outsite-slackbot
services:
- docker:dind
before_script:
- apk add --no-cache curl jq python py-pip
- pip install awscli
stages:
- build
build:
stage: build
script:
- $(aws ecr get-login --no-include-email --region eu-west-1)
There is no problem with the Dockerfile, you can't be connected to docker daemon by the way. So check these steps:
Are you logged in as a root? (sudo su or sudo -i)
Start Docker service (service docker start)
Then follow the tutorial :)
I'm having trouble pulling docker images from AWS ECR when deploying a stack to my docker swarm cluster that runs in AWS EC2.
If I try to ssh to any of the nodes and authenticate manually and pull an image manually, there are no issues
This works:
root#manager1 ~ # `aws ecr get-login --no-include-email --region us-west-2 `
Login Succeeded
root#manager1 ~ # docker pull *****.dkr.ecr.us-west-2.amazonaws.com/myapp:latest
however, if I try deploying a stack or service:
docker stack deploy --compose-file docker-compose.yml myapp
The image can't be found and on the node that I already authenticated as well as on all other manager/worker nodes.
Error from docker service ps myapp :
"No such image: *****.dkr.ecr.us-west-2.amazonaws.com/myapp:latest"
OS: RHEL 7.3
Docker version: Docker version 1.13.1-cs5, build 21c42d8
Anyone have a solution for this issue?
Try this command
docker login -u Username -p password *****.dkr.ecr.us-west-2.amazonaws.com && docker stack deploy --compose-file docker-compose.yml myapp --with-registry-auth