when setting up a job definition in aws batch, I can specify a container image. can i directly specify my image from the docker hub repo ?
I have created a public docker hub repo, but i'm not sure if aws will pull the image directly from there?
Yes, you can specify an image from Docker Hub, there is no such restriction in ECS and AWS Batch Job.
For Container image, choose the Docker image to use for your job. Images in the Docker Hub registry are available by default. You can also specify other repositories with repository-url/image:tag. Up to 255 letters (uppercase and lowercase), numbers, hyphens, underscores, colons, periods, forward slashes, and number signs are allowed. This parameter maps to Image in the Create a container section of the Docker Remote API and the IMAGE parameter of docker run.
If you mentioned image with single name like
nginx:latest
The job will pull image from Docker hub.
Yes, I don't see why it would not work. Since AWS Batch uses ECS, the following options are allowed for Images in ECS (among other things):
Images in official repositories on Docker Hub use a single name (for example, ubuntu or mongo).
Images in other repositories on Docker Hub are qualified with an organization name (for example, amazon/amazon-ecs-agent).
Related
Lets say I have an EKS cluster, an EC2 instance and my local machine, I can pull images from my private ECR without any issues. But when I pull a generic image like nginx, it will come from Docker Hub straight to me. Would it be possible to redirect this pull to enter my ECR first (so that it gets scanned for vulnerabilities, and maybe even for caching purposes perhaps) and then from my ECR to where I pulled from?
If this is not possible, what would be a good alternative?
AWS container team person here. Can you clarify one thing? Would you be ok to point your manifests to ECR (acting as a hub/cache for external registries) or do you want to keep your manifests pointing to DockerHub but somewhat transparently go through ECR for caching? I am asking because we are working on the former scenario.
You can subscribe here to see the progress and leave comments.
It is not possible to redirect your request to pull generic image to ECR and then to Docker Hub.
I understand your concern to pull images from Docker Hub directly. So what you can do what we have done in our projects is:
pull generic image from Docker Hub for one time
Using that image, build your own image with any customisations you may require or not.
Publish the newly created image to your ECR repo.
Going forward use your only ECR repo to pull that image.
In this way, you will have full control on the image you have. Also, it would be more secure to pull it from your ECR repo rather then again and again using Docker Hub. Also, you can do any customisation you want.
I'm trying to set up some infrastructure using AWS ECR to store docker images. I'm just wondering if I have access to the same base images that I do in the docker hub. E.G. FROM node works in my Dockerfile after I log in to ECR. I'm just wondering where this image is getting pulled from. I can't find anything regarding a public ECR repository that stores base images. Thanks.
The name of a Docker image identifies the repository that it comes from. For example:
docker pull aws_account_id.dkr.ecr.us-west-2.amazonaws.com/amazonlinux:latest
The registry is aws_account_id.dkr.ecr.us-west-2.amazonaws.com, the image name is amazonlinux, and the version is latest. The punctuation characters / and : separate these three components.
When you pull from Docker hub, you don't have a registry name, just an image name and version (node:latest).
When you run docker login, it adds credentials to those known by Docker. You can login to as many registries as you want. When you then run docker pull, it looks to see if it has credentials for the specific registry.
As AWS CodeBuild provides "Docker" curated images (http://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref.html),
I'd like to reuse one of those to build my custom CodeBuild Image (http://docs.aws.amazon.com/codebuild/latest/userguide/sample-docker-custom-image.html).
I don't know the CodeBuild Registry Url.
How to reference those images (FROM) ?
I could rebuild myself the image I need (https://github.com/aws/aws-codebuild-docker-images) and host it in a private ECR but it's wasting time and storage.
CodeBuild's internal Docker repository used for Curated Images is not publicly accessible. You should build your own base image from the official Dockerfiles.
The AWS CodeBuild provides also a console where you can use these images. I think they use Amazon EC2 Repository to provide these internally, like the one you can use. but there isn't a public registry (or I already haven't found).
You can find these images to create Docker machines using the build spec defined by Amazon
http://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html
http://docs.aws.amazon.com/codebuild/latest/userguide/sample-docker.html
I found this public image but is not official:
https://hub.docker.com/r/sandorzelei/aws-codebuild/
So you can use it as follows:
FROM sandorzelei/aws-codebuild
I am struggling to understand the problems that prevent my task from starting on my Amazon ECS cluster. I have a task with a single container
I am currently getting some weird undocumented STOPPED (CannotPullContainerError: Error: image library/bdf) and I have no idea where to start from.
When I log into my amazon EC2 instance (Amazon linux, the default ecs-optimized image) and I run docker ps -all I only see a amazon/ecs-emptyvolume-base:autogenerated that does not correspond to my image.
I also had to manually install aws-cli and do a aws ecr get-login + a docker pull to retrieve the image that was supposed to be pulled and ran by the task. (I am using Amazon ECR registry to store the image)
Any help on how to debug this ?
STOPPED (CannotPullContainerError: Error: image library/bdf)
That error message indicates that you have a container in your task definition that can't be pulled. Since the error message mentions library/bdf, it indicates that one of the containers in your task definition has the image field set to bdf, meaning that the ECS agent would try to pull an image named bdf from Docker Hub. As you can see here, there is no such image on Docker Hub.
If you're storing your images in Amazon ECR, you need to specify the full name of the image ($registryId.ecr.$region.amazonaws.com/$repository:$tag) in the image field of your task definition.
I am trying to setup a new springboot+docker(microservices) based project. The deployment is targeted on aws. Every service has a Dockerfile associated with it. I am thinking of using amazon container service for deployment, but as far as I see it only pulls images from docker hub. I don't want ECS to pull from docker-hub, rather build the images from docker file and then take over the deploying those containers.Is it possible to do? If yes how.
This is not possible yet with the Amazon EC2 Container Service (ECS) alone - while ECS meanwhile supports private registries (see also the introductory blog post), it doesn't yet offer an image build service (as usual, AWS is expected to add such notable additional features over time, see e.g. the Feature Request: ECS container dream service for more on this).
However, it can already be achieved with AWS Elastic Beanstalk's built in initial support for Single Container Docker Configurations:
Docker uses a Dockerfile to create a Docker image that contains your source bundle. [...] Dockerfile is a plain text file that contains instructions that Elastic Beanstalk uses to build a customized Docker image on each Amazon EC2 instance in your Elastic Beanstalk environment. Create a Dockerfile when you do not already have an existing image hosted in a repository. [emphasis mine]
In an ironic twist, Elastic Beanstalk has now added Multicontainer Docker Environments based on ECS, but this highly desired more versatile Docker deployment option doesn't offer the ability to build images in turn:
Building custom images during deployment with a Dockerfile is not supported by the multicontainer Docker platform on Elastic Beanstalk. Build your images and deploy them to an online repository before creating an Elastic Beanstalk environment. [emphasis mine]
As mentioned above, I would expect this to be added to ECS in a not too distant future due to AWS' well known agility (see e.g. the most recent ECS updates), but they usually don't commit to roadmap details, so it is hard to estimate how long we need to wait on this one.
Meanwhile Amazon has introduced EC2 Container Registry https://aws.amazon.com/ecr/
It is a private docker repository if you do not like docker hub. Nicely integrated with the ECS service.
However it does not build your docker images, so it does not solve the entire problem.
I use a bamboo server for building images (the source is in git repositories in bitbucket). Bamboo pushes the images to Amazons container registry.
I am hoping the Bitbucket Pipelines will make the process more smooth with less configuration of build servers. From the videos I have seen all your build configuration sits right in your repository. It is still in a closed beta so I guess we will have to wait a bit more to see what it ends up being.