Lets say I have an EKS cluster, an EC2 instance and my local machine, I can pull images from my private ECR without any issues. But when I pull a generic image like nginx, it will come from Docker Hub straight to me. Would it be possible to redirect this pull to enter my ECR first (so that it gets scanned for vulnerabilities, and maybe even for caching purposes perhaps) and then from my ECR to where I pulled from?
If this is not possible, what would be a good alternative?
AWS container team person here. Can you clarify one thing? Would you be ok to point your manifests to ECR (acting as a hub/cache for external registries) or do you want to keep your manifests pointing to DockerHub but somewhat transparently go through ECR for caching? I am asking because we are working on the former scenario.
You can subscribe here to see the progress and leave comments.
It is not possible to redirect your request to pull generic image to ECR and then to Docker Hub.
I understand your concern to pull images from Docker Hub directly. So what you can do what we have done in our projects is:
pull generic image from Docker Hub for one time
Using that image, build your own image with any customisations you may require or not.
Publish the newly created image to your ECR repo.
Going forward use your only ECR repo to pull that image.
In this way, you will have full control on the image you have. Also, it would be more secure to pull it from your ECR repo rather then again and again using Docker Hub. Also, you can do any customisation you want.
Related
Our CD infrastructure deploys Docker images to our ECR repositories. In each repository, there's a latest tag, which is what runs when you (re)start that service.
I'd like to be able to grant IAM users permission to change which deployed image is used by setting the latest tag to another image in the repository, e.g. to roll back to an earlier version after a bad deploy.
The AWS ECR docs for Retagging an image say that you do so by calling aws ecr put-image, but I don't want to enable people to upload arbitrary images of their own devising, just to choose which of the images that came through CD is latest.
It seems like I can grant ecr:PutImage without ecr:InitiateLayerUpload / ecr:CompleteLayerUpload, but it seems like Amazon docs aren't targeting this use case, so I find it tricky to work out what powers this actually grants. What can someone do with the ability to set arbitrary image manifests, if they can't upload images themselves? Or is there some other way to allow people to set tags but do nothing else?
I wonder what are best practices for copying Docker container images from ECR to ECR in AWS.
I have to copy container images periodically between multiple ECR repositories, each placed in separate AWS account - like mirroring but with specific filters for what to copy and what to skip. I wrote a script that does this work by pulling missed images from the 'source' ECR to an EC2 VM, and pushes them to the 'target' ECR.
This works, but I am not satisfied by the performance of doing that in a single thread, and it's not network throughput limiting it but 'expences' to wrap commands, run some necessary calls to AWS, etc.
So I am thinking of rewriting the script to a multi-thread application, but I wonder if I'm inventing a bicycle, and there is some known and better solution for this task.
As it was confirmed by AWS support, there was no 'out-of-the-box' way to do this job, other than direct mirroring of entire repository.
Thus I rewrote the tool for doing it in a smarter and faster way, and published it.
https://github.com/dfad1ripe/aws-ecr-cross-account-clone
To use it, both source and destination AWS accounts should be defined as profiles in ~/.aws/credentials, and the host should be running either Docker Engine or Docker Desktop.
when setting up a job definition in aws batch, I can specify a container image. can i directly specify my image from the docker hub repo ?
I have created a public docker hub repo, but i'm not sure if aws will pull the image directly from there?
Yes, you can specify an image from Docker Hub, there is no such restriction in ECS and AWS Batch Job.
For Container image, choose the Docker image to use for your job. Images in the Docker Hub registry are available by default. You can also specify other repositories with repository-url/image:tag. Up to 255 letters (uppercase and lowercase), numbers, hyphens, underscores, colons, periods, forward slashes, and number signs are allowed. This parameter maps to Image in the Create a container section of the Docker Remote API and the IMAGE parameter of docker run.
If you mentioned image with single name like
nginx:latest
The job will pull image from Docker hub.
Yes, I don't see why it would not work. Since AWS Batch uses ECS, the following options are allowed for Images in ECS (among other things):
Images in official repositories on Docker Hub use a single name (for example, ubuntu or mongo).
Images in other repositories on Docker Hub are qualified with an organization name (for example, amazon/amazon-ecs-agent).
I am launching Apache, MySQL, and memcached docker containers from AWS ECR into an ECS instance. Engineers are able to browse around and make changes as they see fit. These containers expire after a set period of time but they are wanting to save their database changes for use in future containers.
I am looking into seeing if there's a solution I can automate this process to occur before the containers terminate, either with Lambda, aws-cli, or some other utility.
I am looking for a solution that would take the mysql container and create a new image from it. I saw this question and it's mostly what I want:
How to create a new docker image from a running container on Amazon?
But you have to run docker commit from the ECS instance as well as perform the login and push from there. There doesn't appear to be a way to have the committed image pushed to the ECR without having to login with aws ecr get-login --no-include-email and running the output for docker to get the token.
The issue I have with that is if we get to a point where we have multiple ECS instances running it would be difficult to know where the container the engineer is running from, SSHing into that server, and running the docker commit, docker tag, aws ecr login, and docker push commands. To me, that seems kind of hacky and prone to error.
I have the MySQL containers rebuilt and repushed to the ECR every hour so that they have the latest content updates. To launch the containers I am using a combination of ecs-cli and aws-cli to use a docker-compose.yml file to create a task in ECS.
Is there some functionality I can use to commit a running container to ECR with a new name/tag?
The other option I was looking into was starting the MySQL container with persistent storage (EBS/EFS) but am still trying to see if that's doable since I would have to somehow tag the persistent storage so that it will only be used when the engineer launches it that way. Essentially, I would have a unique docker-compose.yml file that is specific to persistent volumes and it would either launch a new container with fresh mysql data or use an existing one if it exists, given a specific name.
I'm trying to set up some infrastructure using AWS ECR to store docker images. I'm just wondering if I have access to the same base images that I do in the docker hub. E.G. FROM node works in my Dockerfile after I log in to ECR. I'm just wondering where this image is getting pulled from. I can't find anything regarding a public ECR repository that stores base images. Thanks.
The name of a Docker image identifies the repository that it comes from. For example:
docker pull aws_account_id.dkr.ecr.us-west-2.amazonaws.com/amazonlinux:latest
The registry is aws_account_id.dkr.ecr.us-west-2.amazonaws.com, the image name is amazonlinux, and the version is latest. The punctuation characters / and : separate these three components.
When you pull from Docker hub, you don't have a registry name, just an image name and version (node:latest).
When you run docker login, it adds credentials to those known by Docker. You can login to as many registries as you want. When you then run docker pull, it looks to see if it has credentials for the specific registry.