I was using gcloudbefore and could build a docker image on a GCP machine as follows:
gcloud builds submit ./my-docker-dir/ -t eu.gcr.io/<path>/<component>:<tag> --timeout 30m --machine-type e2-highcpu-32\n
Is there a similar AWS equivalent?
No, there's no such alternative. With AWS you use docker to build & push to ECR.
An example workflow would be:
Create a docker file and build it with:
docker build -t hello-world .
Authenticate your Docker client to the Amazon ECR registry to which you intend to push your image.
aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com
Create an ECR repository
aws ecr create-repository \
--repository-name hello-world \
--image-scanning-configuration scanOnPush=true \
--region region
Tag the docker image you've built.
docker tag hello-world:latest aws_account_id.dkr.ecr.region.amazonaws.com/hello-world:latest
Push it to your ECR repo.
docker push aws_account_id.dkr.ecr.region.amazonaws.com/hello-world:latest
Related
I have a Docker image in AWS ECR and I open my Sagemaker Notebook instance--->go to terminal-->docker run....
This is how I start my Docker container.
Now, I want to automate this process(running my docker image on Sagemaker Notebook Instance) instead of typing the docker run commands.
Can I create a cron job on Sagemaker? or Is there any other approach?
Thanks
For this you can create an inline Bash shell in your SageMaker notebook as follows. This will take your Docker container, create the image, ECR repo if it does not exist and push the image.
%%sh
# Name of algo -> ECR
algorithm_name=your-algo-name
cd container #your directory with dockerfile and other sm components
chmod +x randomForest-Petrol/train #train file for container
chmod +x randomForest-Petrol/serve #serve file for container
account=$(aws sts get-caller-identity --query Account --output text)
# Region, defaults to us-west-2
region=$(aws configure get region)
region=${region:-us-west-2}
fullname="${account}.dkr.ecr.${region}.amazonaws.com/${algorithm_name}:latest"
# If the repository doesn't exist in ECR, create it.
aws ecr describe-repositories --repository-names "${algorithm_name}" > /dev/null 2>&1
if [ $? -ne 0 ]
then
aws ecr create-repository --repository-name "${algorithm_name}" > /dev/null
fi
# Get the login command from ECR and execute it directly
aws ecr get-login-password --region ${region}|docker login --username AWS --password-stdin ${fullname}
# Build the docker image locally with the image name and then push it to ECR
# with the full name.
docker build -t ${algorithm_name} .
docker tag ${algorithm_name} ${fullname}
docker push ${fullname}
I am contributing this on behalf of my employer, AWS. My contribution is licensed under the MIT license. See here for a more detailed explanation
https://aws-preview.aka.amazon.com/tools/stackoverflow-samples-license/
SageMaker Notebook instance lifecycle configuration script can be used to run a script when you create a notebook or at start time. In this script, you access other AWS resources from your notebook at create time or start time, say access your ECR images and automate starting docker container using a shell script. This script show an example of how you can use cron to schedule certain actions, can be modified per your usecase
Refer more lifecycle config samples in this github page
I got a question - I have a docker image running locally on my Mac. - I'm trying to export that local image and deploy on AWS elasticbean stalk env.
Should I use docker export command which outputs it as a tar file then upload to AWS? or should it be in a different non compressed format?
I already tried the above and docker export it as a tar file but AWS didn't like that so what approach should I take here?
You can create a repository in your aws ECR (Amazon Elastic Container Registry) and push your local image to that repo
aws ecr get-login --no-include-email --region us-east-2
docker tag test-pod:latest 24533xxxxx.dkr.ecr.us-east-2.amazonaws.com/test:latest
docker push 24533xxxxx.dkr.ecr.us-east-2.amazonaws.com/test:latest
I'm trying to pull the pre-built docker images for SageMaker. I am able to successfully docker login to ECR (my AWS credentials). When I try to pull the image I get the standard no basic auth credentials.
Maybe I'm misunderstanding... I assumed those ECR URLs were public.
$(aws ecr get-login --region us-west-2 --no-include-email)
docker pull 246618743249.dkr.ecr.us-west-2.amazonaws.com/sagemaker-scikit-learn
As of 29th August 2021, get-login is deprecated and the command in the answer won't work. so, with AWS CLI v2, here's what has worked for me:
You would need to login to AWS CLI on your machine, then pipe the password to your docker login like this:
$ sudo aws ecr get-login-password --region <region> | sudo docker login --username AWS --password-stdin <account-id>.dkr.ecr.<region>.amazonaws.com
find the account IDs of the repo in the aws region nearest to you here; and available images with tags here by region.
Then you should be able pull images like this:
$ sudo docker pull 720646828776.dkr.ecr.ap-south-1.amazonaws.com/sagemaker-scikit-learn:0.23-1-cpu-py3
Could you show your ECR login command and pull command in the question?
For SageMaker pre-built image 520713654638.dkr.ecr.us-west-2.amazonaws.com/sagemaker-mxnet:1.3.0-cpu-py3
What I do is:
Log in ECR
$(aws ecr get-login --no-include-email --registry-ids 520713654638 --region us-west-2)
Pull the image
docker pull 520713654638.dkr.ecr.us-west-2.amazonaws.com/sagemaker-mxnet:1.3.0-cpu-py3
These images are public readable so you can pull them from any AWS account. I guess the reason you failed is that you did not specify --registry-ids in your login. But it's better if you can provide your scripts for others to identify what's wrong.
When I am trying to pull docker image from ECR, I am getting the below error:
Get https://3242344.dkr.ecr.ap-south-1.amazonaws.imagename/latest: no basic auth credentials
Docker service is running fine and I am able to list the repositories.
First, you need to Authenticate your Docker logins to the Amazon ECR:
aws ecr get-login --region <<region>> --no-include-email
Refer below link for Amazon ECR Registries authentication:
https://docs.aws.amazon.com/AmazonECR/latest/userguide/Registries.html#registry_auth
Describe your image within a repository by using below command:
aws ecr describe-images --repository-name amazonlinux
Pull the image by using below command:
docker pull aws_account_id.dkr.ecr.us-west-2.amazonaws.com/amazonlinux:latest
For more information please refer below link:
https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-pull-ecr-image.html
You need to retrieve the docker login command using AWS CLI
$(aws ecr get-login --no-include-email --region <your region>)
More info in Getting Started with Amazon ECR.
Im new to AWS. I want to set up a private docker repository on an AWS ECS container instance. I created a repository named name. The example push commands shown by AWS are working.
aws ecr get-login --region us-west-2
docker build -t name .
docker tag name:latest ############.dkr.ecr.us-west-2.amazonaws.com/name:latest
docker push ############.dkr.ecr.us-west-2.amazonaws.com/name:latest
But with this commands I build and pushed an image named name and I want to build an image named foo. So I altered the commands to:
docker build -t foo .
docker tag foo ###########.dkr.ecr.us-west-2.amazonaws.com/name/foo
docker push ###########.dkr.ecr.us-west-2.amazonaws.com/name/foo
This should work, but it doesn't. After a period of retrys I get the error:
The push refers to a repository [###########.dkr.ecr.us-west-2.amazonaws.com/name/foo]
8cc63cf4528f: Retrying in 1 second
...
name unknown: The repository with name 'name/foo' does not exist in the registry with id '############'
Does AWS really require a dedicated repository for every image i want to push?
The EC2 Container Registry requires an image Repository to be setup for each image "name" or "namespace/name" you want to publish to the registry.
You can publish any :tags you want in each Repository though (The default limit is 100 tags).
I haven't seen anywhere in the AWS documentation that specifically states the repository -> image name mapping but it's implied by Creating a Repository - Section 6d in the ECR User Guide
The Docker Image spec includes it's definition of a Repository
Repository
A collection of tags grouped under a common prefix (the name component before :). For example, in an image tagged with the name
my-app:3.1.4, my-app is the Repository component of the name. A
repository name is made up of slash-separated name components,
optionally prefixed by a DNS hostname. The hostname must comply with
standard DNS rules, but may not contain _ characters. If a hostname is
present, it may optionally be followed by a port number in the format
:8080. Name components may contain lowercase characters, digits, and
separators. A separator is defined as a period, one or two
underscores, or one or more dashes. A name component may not start or
end with a separator.
You need to create a repository for each image name, but the image name can be of the form "mycompanyname/helloworld". So you create mycompanyname/app1, mycompanyname/app2, etc
aws ecr create-repository --repository-name mycompanyname/helloworld
aws ecr create-repository --repository-name mycompanyname/app1
aws ecr create-repository --repository-name mycompanyname/app2
docker tag helloworld:latest xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/mycompanyname/helloworld:latest
docker push xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/mycompanyname/helloworld:latest
docker tag app1:latest xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/mycompanyname/app1:latest
docker push xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/mycompanyname/app1:latest
I tried the following steps and confirmed working for me:
aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com
aws ecr create-repository --repository-name test
docker build -t test .
docker tag test:latest xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test:latest
docker push xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test:latest
Addition to the above answer, I came across here today, as the login command change with aws-cli v2, posting as an answer might help others.
as aws-cli v1 login command no longer work.
V1
$(aws ecr get-login --no-include-email)
To push image to ECR using aws-cli v2 you need
aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 123456789.dkr.ecr.us-west-2.amazonaws.com
Then you are okay to build and push
docker build -t myrepo .
docker tag myrepo:latest 123456789.dkr.ecr.us-west-2.amazonaws.com/myrepo
docker push 123456789.dkr.ecr.us-west-2.amazonaws.com/myrepot
Typically One image per registry is a clean approach, that why AWS increase image per repository and repository per region from 1000 to 10,000.
For this i automated the script that can read your public images from csv file and pull them. After that it will try to create repository in ECR and push to registry.
Prepare CSV file ecr-images.csv
docker.io/amazon/aws-for-fluent-bit,2.13.0
docker.io/couchdb,3.1
docker.io/bitnami/elasticsearch,7.13.1-debian-10-r0
k8s.gcr.io/kube-state-metrics/kube-state-metrics,v2.0.0
k8s.gcr.io/metrics-server-amd64,v0.3.6
--------------------KEEP THIS LINE AT END-------------------------
Automated script ecr.sh that will copy images to ecr
#!/bin/bash
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
assert_value() {
if [ -z "$1" ]; then
echo "No args: $2"
exit 1
fi
}
repository_uri=$1
assert_value "$repository_uri" "repository_uri"
create_repo() {
## try to create & failure will ignored by <|| true>
aws ecr create-repository --repository-name "$1" --output text || true
}
## Copy Docker Images to ECR
COUNTER=0
while IFS=, read -r dockerImage tag; do
outputImage=$(echo "$dockerImage" | sed -E 's/(\w+?\.)+\w+?\///')
outputImageUri="$repository_uri/$outputImage"
# shellcheck disable=SC2219
let COUNTER=COUNTER+1
echo "--------------------------------------------------------------------------"
echo "$COUNTER => $dockerImage:$tag pushing to $outputImageUri:$tag"
echo "--------------------------------------------------------------------------"
docker pull "$dockerImage:$tag"
docker tag "$dockerImage:$tag" "$outputImageUri:$tag"
create_repo "$outputImage"
docker push "$outputImageUri:$tag"
done <"$SCRIPT_DIR/ecr-images.csv"
Run
repository_uri=<ecr_account_id>.dkr.ecr.<ecr_region>.amazonaws.com
aws ecr get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin $repository_uri
./ecr.sh $repository_uri
# Build an image from azure devops pipeline to aws eks
parameters:
- name: succeed
displayName: Succeed or fail
type: boolean
default: false
trigger:
- main
- releases/*
pool:
vmImage: "windows-latest"
stages:
- stage: init
jobs:
- job: init
continueOnError: false
steps:
- task: Docker#2
inputs:
containerRegistry: 'docker'
repository: 'ecr-name'
command: 'build'
Dockerfile: '**/Dockerfile'
tags: 'latest'
- task: ECRPushImage#1
inputs:
awsCredentials: 'aws credentilas'
regionName: 'us-east-1'
imageSource: 'any-name'
sourceImageName: 'ecr-name'
sourceImageTag: 'latest'
repositoryName: 'ecr-name'
pushTag: 'latest'
Create a repo per application:
aws ecr create-repository --repository-name worker --region us-east-1
aws ecr create-repository --repository-name gateway --region us-east-1
Login to registry
The AWS usr name is fixed for all registry logins
aws ecr get-login-password \
--region us-east-1 \
| docker login \
--username AWS \
--password-stdin <aws_12_digit_account_number>.dkr.ecr.us-east-1.amazonaws.com
Push image
docker build -f Dockerfile -t <123456789012>.dkr.ecr.us-east-1.amazonaws.com/worker:v1.0.0
docker push <123456789012>.dkr.ecr.us-east-1.amazonaws.com/worker:v1.0.0