I have a single-instance elastic beanstalk environment which runs a docker image which is hosted as a private image on Dockerhub. This works fine. I am trying to create a new multi-container environment which runs the exact same image (plus one other, not icluded in my code example here). In the multi-container environment, I cannot get elastic beanstalk to launch my docker image, I get the following error:
ECS task stopped due to: Task failed to start. (img1_name: img2_name: CannotPullContainerError: Error response from daemon: pull access denied for user/repo, repository does not exist or may require 'docker login': denied: requested access to the resource is denied)
Here is the dockerrun for my single-instance environment:
{
"AWSEBDockerrunVersion": "1",
"Authentication": {
"Bucket": "my_bucket",
"Key": ".dockercfg"
},
"Image": {
"Name": "user/repo:tag",
"Update": "true"
},
"Ports": [
{
"ContainerPort": 5000,
"HostPort": 443
}
],
"Logging": "/var/log/nginx"
}
And here is the .dockerfcg file:
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "my_token"
}
}
}
Again, the above works fine.
My multi-instance dockerrun file is as follows:
{
"AWSEBDockerrunVersion": "2",
"authentication": {
"bucket": "my_bucket",
"key": ".dockercfg"
},
"containerDefinitions": [
{
"name": "img_name",
"image": "user/repo:tag",
"essential": true,
"memoryReservation": 128,
"portMappings": [
{
"hostPort": 80,
"containerPort": 5000
}
]
}
],
"Logging": "/var/log/nginx"
}
I have ssh-ed into my elastic-beanstalk instance and run the following to check that it is able to access the .dockercfg from my s3 bucket:
aws s3api get-object --bucket mybucket --key dockercfg dockercfg
I have also tried various different formats for the .dockercfg file including...
{
"https://index.docker.io/v1/": {
"auth": "zq212MzEXAMPLE7o6T25Dk0i",
"email": "email#example.com"
}
}
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "zq212MzEXAMPLE7o6T25Dk0i"
}
}
}
I'm tearing my hair out over this, I've found a few similar threads here and on AWS forums, but nothing seems to resolve my issue. Any help greatly appreciated.
Related
I have a basic node app that I've wrapped in a Dockerfile
FROM node:lts-alpine3.15
WORKDIR /app
COPY package.json ./
RUN npm install
COPY . .
EXPOSE 8080
CMD [ "npm", "run", "serve" ]
I push that to Gitlab's container registry. I'm trying to deploy it from there to AWS, but running into problems on the ECS side. In ECS I have:
a cluster (frontend)
a service (frontend)
both of which are configured in terraform
resource "aws_ecs_cluster" "frontend" {
name = "frontend"
setting {
name = "containerInsights"
value = "enabled"
}
}
resource "aws_ecs_service" "frontend" {
name = "frontend"
cluster = aws_ecs_cluster.frontend.id
deployment_controller {
type = "EXTERNAL"
}
tags = {
Name = "WebAppFrontend"
}
}
The web app is in a different repository from the terraform infrastructure. In my .gitlab-ci.yml I'm trying to register a new task definition for the web app I'm trying to register a new task definition with a json file.
I want when there's been changes to the web app I was to perform a rolling update so both the new version and old version are running, but I can't get one version deployed to ecs. My .gitlab-ci.yml is
deploy_ecs:
stage: deploy_ecs
script:
- aws ecs register-task-definition --cli-input-json file://task_definition.json
task_definition.json is:
{
"family": "frontend",
"containerDefinitions": [
{
"name": "frontend",
"image": "registry.gitlab.com/myproject/application/myimage:latest",
"memory": 300,
"portMappings": [
{
"containerPort": 8080,
"hostPort": 80
}
],
"essential": true,
"environment": [
{
"name": "Frontend",
"value": "dev"
}
]
}
]
}
Attempting to create a service from the console I get this error
The selected task definition is not compatible with the selected compute strategy.
Manually on the ec2 instance infrastructure for the ecs cluster I can run
docker run -d -p 80:8080 myimage
which does run the app. Am I able to:
Deploy the task definition file as above and run the service in my cluster
Deploy in a way so that there will be both versions in a rolling update to avoid any downtime
Do both of the above from my .gitlab-ci.yml
The ec2 instance is confirmed to be running the ecs-agent and I can see the container instance showing correctly so I know ecs is running.
I used console and the service was created successfully.
{
"requiresCompatibilities": [
"EC2"
],
"family": "frontend",
"containerDefinitions": [
{
"name": "frontend",
"image": "registry.gitlab.com/myproject/application/myimage:latest",
"memory": 300,
"portMappings": [
{
"containerPort": 8080,
"hostPort": 80
}
],
"essential": true,
"environment": [
{
"name": "Frontend",
"value": "dev"
}
]
}
]
}
The task eventually failed with access denied but the rest everything worked. Plus you need to add the " ecsTaskExecutionRole" for the task to function.
The trouble is that I have an issue when I am trying to deploy the Mailhog and run it with the auth file via AWS EB in the Docker platform. The issue is:
[HTTP] Error reading auth-file: open /app/bin/auth.txt: no such file or directory. Check snapshot logs for details. (ElasticBeanstalk::ExternalInvocationError)
caused by: dcedd77ced7be4bab370671cfe92488db824bd705302ee903ca9aac50232a577
The zip file that I am packaging contains two files:
auth.txt
Dockerrun.aws.json
Below you can find the Dockerrun.aws.json content:
{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "mailhog/mailhog",
"Update": "true"
},
"Ports": [
{
"HostPort": 1025,
"ContainerPort": 1025
},
{
"HostPort": 80,
"ContainerPort": 8025
}
],
"Volumes": [
{
"HostDirectory": "/var/app/mailhog",
"ContainerDirectory": "/var/app/mailhog"
}
],
"Entrypoint": "MailHog",
"Command": "-auth-file=/var/app/auth.txt"
}
So the question is how can I specify the path to my auth file that is stored in a ZIP package?
I am trying to run a dockerised Jenkins and postgres database on AWS elastic beanstalk in a multi-container t2.micro environment:
Dockerrun.aws.json
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"name": "postgres-db",
"image": "postgres:9.5-alpine",
"essential": true,
"memory": 256,
"portMappings": [
{
"hostPort": 5432,
"containerPort": 5432
}
]
},
{
"name": "jenkins-blueocean",
"image": "<account_id>.dkr.ecr.ap-southeast-2.amazonaws.com/<image>:latest",
"essential": true,
"memory": 256,
"mountPoints": [
{
"sourceVolume": "jenkins-data",
"containerPath": "/var/jenkins_home"
}
],
"portMappings": [
{
"hostPort": 80,
"containerPort": 8080
}
],
"links": [
"postgres-db"
]
}
],
"volumes": [
{
"name": "jenkins-data",
"host": {
"sourcePath": "/var/jenkins-data"
}
}
]
}
AWS shows it deploys fine but the logs for jenkins-blueocean container has that error:
/var/log/containers/jenkins-blueocean-7ce78063214b-stdouterr.log
touch: cannot touch '/var/jenkins_home/copy_reference_file.log': Permission denied
Can not write to /var/jenkins_home/copy_reference_file.log. Wrong volume permissions?
Am I missing something to allow jenkins access to the volume?
Thanks in advance!
Not 100% sure if this is the right path but we ended up following the .ebextensions method of running commands to setup the volume path to allow the jenkins user from the jenkins-blueocean container full access to do its thing.
mkdir -p /var/jenkins-data
chmod 777 /var/jenkins-data
This was because the permissions on the location in the docker instance has r-x rights for other users, with root user having rwx.
I'm deploying an ASP.NET Core Web API app as a docker image to AWS ECS, so use a task definition file for that.
It turns out the app only works if I specify environment variable VIRTUAL_HOST with the public DNS of my EC2 instance (as highlighted here: http://docs.servicestack.net/deploy-netcore-docker-aws-ecs), see taskdef.json below:
{
"family": "...",
"networkMode": "bridge",
"containerDefinitions": [
{
"image": "...",
"name": "...",
"cpu": 128,
"memory": 256,
"essential": true,
"portMappings": [
{
"containerPort": 80,
"hostPort": 0,
"protocol": "http"
}
],
"environment": [
{
"name": "VIRTUAL_HOST",
"value": "ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com"
}
]
}
]
}
Once the app is deployed to AWS ECS, I hit the endpoints - eg http://ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com/v1/ping
with the actual public DNS of my EC2 instance in VIRTUAL_HOST all works fine
without the env variable I'm getting "503 Service Temporarily Unavailable" from nginx/1.13.0
and if I put an empty string to VIRTUAL_HOST I'm getting a "502 Bad Gateway" from nginx/1.13.0.
Now, I'd like to avoid specifying virtual host in the taskdef file - is that possible? Is my problem ASP.NET Core related or nginx related?
Amazon ECS have a secret management system using Amazon S3. You have to create a secret in your ECS interface, and then you will be able to reference it in your configuration, as an environment variable.
{
"family": "...",
"networkMode": "bridge",
"containerDefinitions": [
{
"image": "...",
"name": "...",
"cpu": 128,
"memory": 256,
"essential": true,
"portMappings": [
{
"containerPort": 80,
"hostPort": 0,
"protocol": "http"
}
],
"environment": [
{
"name": "VIRTUAL_HOST",
"value": "SECRET_S3_VIRTUAL_HOST"
}
]
}
]
}
Store secrets on Amazon S3, and use AWS Identity and Access Management (IAM) roles to grant access to those stored secrets in ECS.
Full blog post
You could also make your own NGinx Docker image, which will already contain the environment variable.
FROM nginx
LABEL maintainer YOUR_EMAIL
ENV "VIRTUAL_HOST" "ec2-xx-xxx-xxxxxx.compute1.amazonaws.com"
And you would just have to build it, ship it privately and then use it for your configuration.
I am running into issues running docker-compose because an elastic load balancer. Setup is ELB does 443 -> TCP 80 and docker does 0.0.0.0:80->4444/tcp
However the server doesn't seem to be hit and I get DNS_PROBE_FINISHED_NXDOMAIN
Trying to verify if this is a docker setup issue. Docker version 1.12.6 and docker-compose version 1.12.0
is it normal for the bridge config to not have a Gateway defined?
```
[root#loom-server1 ec2-user]# docker network inspect 8f1b234bfb0b
[
{
"Name": "bridge",
"Id": "8f1b234bfb0b6c41962265299871cd8053757ec145f8e3f6b63960b71ceb3690",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.17.0.0/16"
}
]
},
"Internal": false,
"Containers": {
"bbd2b84545a0e3519e37fb4015eea45637b75ccaa1dd362aff68ff41f3118055": {
"Name": "dockercompose_loom_1",
"EndpointID": "b7da2d31ff2503846d4f621bf355b8522afb8dabd1f02ca638c9ef032afefa76",
"MacAddress": "02:42:ac:11:00:02",
"IPv4Address": "172.17.0.2/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
]
```
weird part is that it's able to load assets
ec2-52-53-84-186.us-west-1.compute.amazonaws.com/assets/js/homepage.js
The link may be up or down as I experiment with instances.
This is all running on opsworks.
Any insight or help would be appreciated.