I am trying to create a Fargate cluster using Cloud Formation in AWS which uses a bunch of images stored in a private registry behind username/password authentication.
This command
./ecs-cli.exe compose --project-name AdminUI service up --create-log-
groups --cluster-config AdminUIConfig
results in an error
FATA[0302] Deployment has not completed: Running count has not changed
for 5.00 minutes
After investigation it appears the problem is because of the lack of basic auth against the repo which holds the images. How on earth do I pass this? I am currently running on Windows 10 using VS Code, if that matters. It feels like it is not client side, it is the cluster itself which needs to be sending the authentication.
Sorry, new to Docker and AWS
Fargate currently only supports pulling images from an unauthenticated registry (like Docker Hub) or from Amazon ECR.
From the documentation:
The Fargate launch type only supports images in Amazon ECR or public repositories in Docker Hub.
Related
One of the projects has shared its docker-compose.yml file. It contains various services. Each service forms a container. I can easily deploy this image in EC2 and get going. However, I want to use AWS ECS only.
How can I deploy that YML file in AWS ECS?
AWS ECS is little bit different from normal docker environment where you directly start the container.
In ECS you need to create a task with the docker image and then create a service to run that task.
So you cannot directly apply deployment.yaml file over ECS.
Here's how you can do this manually, https://aws.amazon.com/getting-started/hands-on/deploy-docker-containers/
You can always automate this using terraform/ aws cli etc..
I am trying to build on demand AWS jmeter(can be any testing tool like SOAP UI, Selenium ) instance to using Jenkins. Not looking for Server client Jmeter distribution architecture.
This is to provide cost effective solution to the spawn on demand jmeter(Not containerization )instance using Jenkins. New instance need JNLP or jenkins agent to establish connectivity with Jenkins Master.
Can some one provide me any documentation and codes(CLI) to spin up aws instance with or without AMI ?
You can use AWS CLI to manage instances (create, launch, shut down, terminate, etc.)
Example command would be:
aws ec2 run-instances --image-id your_image_id --count how_many_instances_you_want --instance-type desired_EC2_instance_type --key-name your_key_pair --security-groups your_EC2_security_group_name
Make sure that the security group allows the following ports:
the port you define as server_port, by default 1099
the port you define as server.rmi.localport
the port(s) you define as client.rmi.localport
More information:
Remote hosts and RMI configuration
Apache JMeter Properties Customization Guide
Am not sure if your are looking for this kind of setup.
Use terraform, infra as code. You will be able to spawn all the resources that are required for your test. The steps will follow like this,
Create a jmeter Docker image
Push it to ECR
Create a Cluster in ECS
Create a Task definition
Create a service in ECS cluster where it uses the Jmeter image and you can use fargate serverless.
On all the above you can use Jenkins CI/CD where you can trigger you terraform code.
I have the end goal of deploying a docker container on AWS Fargate. As it happens, my dockerfile has no local dependencies and my upload connection is very slow, thus I want to build it in the cloud. What would be the easiest way to build the image on AWS? Creating an EC2 Linux instance, installing docker and aws-cli in it, building the image then uploading to AWS ECR, if that's possible?
The easiest way is by using AWS CodeBuild - it will do everything for you, even push it to AWS ECR.
Basic instructions: here
We have a python docker image which needs to build/publish (CI/CD) into AWS container registry.
At the moment AWS does not support for running docker tasks using docker hub private repositories, therefore we have to use ECR instead of docker hub.
Our CI/CD pipeline uses docker build and push tasks. Docker authentication is done via a Service Endpoint in the VSTS project.
There are few steps we should follow to setup a VSTS service endpoint for ECR. This required to execute AWS CLI command (locally or cloud) to get a user and password for docker client to login, it looks like;
aws ecr get-login --no-include-email
Above command outputs a docker login command with a username (AWS) and a password (token).
The issue with this approach is access token will last only for 12 hours. Therefore CI/CD task requires updating the Service Endpoint every 12 hours, otherwise build fail with unauthorised token exception.
Other option we have is to run some shell commands to execute aws get-login command and run docker build/push commands in the same context. This option required installing aws cli into build agent (we are using public linux agent).
In addition shell command involves awkward task configuration with environment/variables. Otherwise we will be exposing aws application id and secret in the build steps.
Could you please advice if you have solved VSTS CI/CD pipeline using docker with AWS ecr?
Thanks, Mahi
After lot of research, trial and error I found an answer to my own question.
AWS provides an extension to VSTS with build tasks and Service Endpoints. You need to configure AWS service endpoint using an account number, application ID, and secret. Then, in your build/release definition;
build docker image using out of the box docker build task, or shell/bash command (for an example; docker build -t your:tag . )
Then add another build step to push image into AWS registry, for this you can use AWS extension task (Amazon Elastic Container Registry Push Image). Amazon Elastic Container Registry Push Image build task will generate token and login docker client every time you run this build definition. You don't have to worry about updating username/token every 12 hours, AWS extension build task will do that for you.
You are looking for this
Amazon ECR Docker Credential Helper
AWS documentation
This is where Amazon ECR Docker Credential Helper makes it easy for developers to use ECR without the need to use docker login or write logic to refresh tokens and provide transparent access to ECR repositories.
Credential Helper helps developers in a continuous development environment to automate the authentication process to ECR repositories without having to regenerate tokens every 12 hours. In addition, Credential Helper also provides token caching under the hood so you don’t have to worry about getting throttled or writing additional logic
I have most of my app designed and deployed. We have gone for RabbitMQ over SQS for several reasons.
I currently have my consumer running in a docker container and would like to deploy this on AWS. I am fairly familiar with Elastic Beanstalk since our web-tier is running there, but it seems all workers deployed this way have to use SQS?
The other option I am aware of is to use ECS for the docker component, but I do not want to make the image publicly available and don't have access to a private repository.
Is there some basic functionality I am missing, a document describing how to deploy to ECS using a Dockerfile and source code locally, or a way of deploying to EB using a Dockerfile without being locked into SQS?
edit
So I have found the note on the docs for EB which says that Dockerfiles are not supported when building a multi-container environment and that repositories currently have to be used for that purpose, so EB is out for me.