I run a docker compose to start an instance of localstack with the S3 service. It starts normally, but when trying to use the AWS CLI to access it I get the following error:
> aws --endpoint-url=http://localhost:4572 s3api put-bucket-acl --bucket demo-bucket --acl public-read
Connection was closed before we received a valid response from endpoint URL: "http://localhost:4572/demo-bucket?acl".
The docker compose:
version: '3.7'
services:
localstack:
image: localstack/localstack
container_name: localstack_service
ports:
- "4567-4584:4567-4584"
- "8055:8080"
environment:
- SERVICES=s3
- DEBUG=1
- DATA_DIR=/tmp/localstack/data
volumes:
- ./tmp/localstack:/tmp/localstack
- /var/run/docker.sock:/var/run/docker.sock
networks:
default:
name: mock_demo
I think the problem is on localstack, because I can't even access the web page.
I used the version
localstack_demo | LocalStack version: 0.12.1
According to the project documentation on GitHub, in this version all APIs are exposed via a single edge service, which is accessible on http://localhost:4566 by default.
Related
I am trying to connect a aws s3 service from gitlab-ci script but I am keep on getting
Could not connect to the endpoint URL: "http://localhost:4566/". It looks like something has been changed in the latest image of the localstack as it was working fine before.
Below is the docker-compose file
localstack:
container_name: config-localstack
image: localstack/localstack
ports:
- '4566-4599:4566-4599'
environment:
SERVICES: s3
DEFAULT_REGION: 'eu-central-1'
DATA_DIR: /tmp/localstack/data
HOSTNAME_EXTERNAL: localhost
HOSTNAME: localstack
From gitlab-ci
do aws s3api head-bucket --bucket test-bucket --endpoint http://localhost:4566 && echo "Success" && break
Everything was working fine before but somehow it stopped working and it say Could not connect to the endpoint URL: "http://localhost:4566/"
In the docker-compose file when I change the image to some backward version
image: localstack/localstack:1.2.0, it connect successfully.
I want to know if something is changed in the new image and do I need to change something at my end.
I'm getting the following error when I try to run docker compose up to deploy my infrastructure to AWS using Docker's ECS integration. Note that I'm running this on Pop!_OS 21.10, which is based on Ubuntu.
NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Things I've tried, based on an exhaustive search of SO and other sites:
Verified the proper format of my ~/.aws/config and ~/.aws/credentials files are formatted correctly, are in the proper place, and have the correct permissions
Verified that the aws cli works fine
Verify that AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION are all set correctly
Tried copying the config and credentials to /root/.aws
Tried setting AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION in the root user's environment
Created /etc/systemd/system/docker.service.d/aws-credentials.conf and populated it with:
[Service]
Environment="AWS_ACCESS_KEY_ID=********************"
Environment="AWS_SECRET_ACCESS_KEY=****************************************"
Ran docker -l debug compose up (Only extra information it provides is DEBUG deploying on AWS with region="us-east-1"
I'm running out of options. If anyone has any other ideas to try, I'd love to hear it. Thanks!
Update: I've also now tried the following, with no luck:
Tried setting Environment="AWS_SHARED_CREDENTIALS_FILE=/home/kespan/.aws/credentials
Tried setting Environment="AWS_SHARED_CREDENTIALS_FILE=/home/kespan/.aws/credentials in /etc/systemd/system/docker.service.d/override.conf
After remembering my IAM account has MFA enabled, generated a token and added Environment="AWS_SESSION_TOKEN=..." to override.conf
Also to note - each time after I've added/modified files under /etc/systemd/system/docker.service.d/ I've run:
sudo systemctl daemon-reload
sudo systemctl restart docker
Edit:
Here's one of the Dockerfiles (both the scraper and scheduler use an identical Dockerfile):
FROM denoland/deno:alpine
WORKDIR /app
USER deno
COPY deps.ts .
RUN deno cache --unstable --no-check deps.ts
COPY . .
RUN deno cache --unstable --no-check mod.ts
RUN mkdir -p /var/tmp/log
CMD ["run", "--unstable", "--allow-all", "--no-check", "mod.ts"]
Here's my docker-compose (some bits redacted):
version: '3'
services:
grafana:
container_name: grafana
image: grafana/grafana
ports:
- "3000:3000"
volumes:
- grafana:/var/lib/grafana
deploy:
replicas: 1
scheduler:
image: scheduler
x-aws-pull-credentials: "arn..."
container_name: scheduler
environment:
DB_CONNECTION_STRING: "postgres://..."
SQS_URL: "..."
SQS_REGION: "us-east-1"
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
deploy:
replicas: 1
scraper:
image: scraper
x-aws-pull-credentials: "arn..."
container_name: scraper
environment:
DB_CONNECTION_STRING: "postgres://..."
SQS_URL: "..."
SQS_REGION: "us-east-1"
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
deploy:
replicas: 1
volumes:
grafana:
Have you attempted to use the Amazon ECS Local Container Endpoints tool that AWS Labs provides? It allows you to create an override file for you docker-compose configurations, and it will simulate the ECS endpoints and IAM roles you would be using in AWS.
This is done using the local AWS credentials you have on your workstation. More information is available on the AWS Blog.
I'm trying to deploy a docker container with multiple services to ECS. I've been following this article which looks great: https://aws.amazon.com/blogs/containers/deploy-applications-on-amazon-ecs-using-docker-compose/
I can get my container to run locally, and I can connect to the ECS context using the AWS CLI; however in the basic example from the article when I run
docker compose up
In order to deploy the image to ECS, I get the error:
pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Can't seem to make heads or tails of this. My docker is logged in to ECS using
aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com
The default IAM user on my aws CLI has AmazonECS_FullAccess as well as "ecs:ListAccountSettings" and "cloudformation:ListStackResources"
I read here: pull access denied repository does not exist or may require docker login mikemaccana 's answer that after Nov 2020 authentication may be required in your YAML file to allow AWS to pull from hub.docker.io (e.g. give aws your Docker hub username and password) but I can't get the 'auth' syntax to work in my yaml file. This is my YAML file that runs tomcat and mariadb locally:
version: "2"
services:
database:
build:
context: ./tba-database
image: tba-database
# set default mysql root password, change as needed
environment:
MYSQL_ROOT_PASSWORD: password
# Expose port 3306 to host. Not for the application but
# handy to inspect the database from the host machine.
ports:
- "3306:3306"
restart: always
webserver:
build:
context: ./tba-webserver
image: tba-webserver
# mount point for application in tomcat
volumes:
- ./target/testPROJ:/usr/local/tomcat/webapps/ROOT
links:
- database:tba-database
# open ports for tomcat and remote debugging
ports:
- "8080:8080"
- "8000:8000"
restart: always
Author of the blog here (thanks for the kind comment!). I haven't played much with the build side of things but I suspect what's happening here is that when you run docker compose up we ignore the build phase and only leverage the image field. What happens next is that the containers being deployed on ECS/Fargate tries to pull the image tba-database (which is where the deploying seems to be complaining because it doesn't exist). You need extra steps to push your image to either GH or ECR before you could bring it life using docker compose up when in the ecs context.
You also probably need to change the compose version ("2" is very old).
I'm trying to upload some traces on AWS xray using opencensus.
The route of traces is simply this:
client -> opencensus agent -> xray
I'm using a docker-compose.yml with this configuration:
version: '3.7'
services:
#login
login:
build:
context: .
dockerfile: Dockerfile
hostname: login
ports:
- 8080:8080
volumes:
- ./src:/app
#ocagent
ocagent:
image: omnition/opencensus-agent
volumes:
- ./ocagent-config.yaml:/conf/ocagent-config.yaml
#xray
xray:
image: amazon/aws-xray-daemon
volumes:
- ./.aws/:/root/.aws/
command: -o -n eu-west-1 --bind=xray:2000
and my config file for the opencensus exporter to aws is this:
exporters:
aws-xray:
region: "eu-west-1"
version: "latest"
buffer_size: 200
I uploaded credential as environment variables.
when I run docker-compose up my service starts and all works fine but the traces doesn't show up in the console, it seems traces get lost in the route (maybe for a misconfiguration).
Can you help me pls?
My project is a flask project using docker-compose.
And source code is in GitLab.
I wanna auto-deploy to ECS with GitLab CI.
Also, docker images are in ECR.
But I faced following error.
Subnet created: subnet-0ffc4936b92c
Subnet created: subnet-0177c849eeca
Cluster creation succeeded.
WARN[0000] Skipping unsupported YAML option for service... option name=build service name=proxy
WARN[0000] Skipping unsupported YAML option for service... option name=container_name service name=proxy
WARN[0000] Skipping unsupported YAML option for service... option name=restart service name=proxy
WARN[0000] Skipping unsupported YAML option for service... option name=build service name=api
WARN[0000] Skipping unsupported YAML option for service... option name=container_name service name=api
WARN[0000] Skipping unsupported YAML option for service... option name=restart service name=api
WARN[0000] Skipping unsupported YAML option for service... option name=build service name=worker
WARN[0000] Skipping unsupported YAML option for service... option name=container_name service name=worker
WARN[0000] Skipping unsupported YAML option for service... option name=restart service name=worker
INFO[0001] Using ECS task definition TaskDefinition="backend:12"
WARN[0001] No log groups to create; no containers use 'awslogs'
ERRO[0001] Error running tasks error="InvalidParameterException: No Container Instances were found in your cluster." task definition=0xc0005a5ae0
FATA[0001] InvalidParameterException: No Container Instances were found in your cluster.
docker-compose.yml
version: "3.0"
services:
proxy:
container_name: rs-proxy
image: ${REPOSITORY_URL}/proxy
build:
context: proxy/.
dockerfile: Dockerfile
ports:
- 80:80
restart: on-failure
api:
container_name: rs-api
image: ${REPOSITORY_URL}/api
build:
context: api/.
dockerfile: Dockerfile.prod
restart: on-failure
volumes:
- ./api/migrations:/app/migrations
worker:
container_name: rs-worker
image: ${REPOSITORY_URL}/worker
build:
context: .
dockerfile: ./worker/Dockerfile
restart: on-failure
.gitlab-ci.yml
image: tiangolo/docker-with-compose
variables:
PROJECT_NAME: test-project
CONFIG_NAME: $PROJECT_NAME
PROFILE_NAME: $PROJECT_NAME-profile
AWS_ECR_URL: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
REPOSITORY_URL: $AWS_ECR_URL/$PROJECT_NAME
before_script:
- export REPOSITORY_URL=$REPOSITORY_URL
- apk add --no-cache curl jq python3 py-pip
- apk add --update curl
- pip install awscli
- curl -o /usr/local/bin/ecs-cli https://s3.amazonaws.com/amazon-ecs-cli/ecs-cli-linux-amd64-latest
- chmod +x /usr/local/bin/ecs-cli
- echo "Logging in AWS..."
- aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ECR_URL
stages:
- build
- deploy
build:
stage: build
script:
- echo "Building image..."
- docker-compose -f docker-compose.yml build
- echo "Pushing image..."
- docker push ${REPOSITORY_URL}/proxy:latest
- docker push ${REPOSITORY_URL}/api:latest
- docker push ${REPOSITORY_URL}/worker:latest
only:
- master
deploy:
stage: deploy
script:
- echo "Configuring AWS ECS..."
- ecs-cli configure --cluster $CONFIG_NAME --default-launch-type EC2 --config-name $CONFIG_NAME --region $AWS_DEFAULT_REGION
- ecs-cli configure profile --access-key $AWS_ACCESS_KEY_ID --secret-key $AWS_SECRET_ACCESS_KEY --profile-name $PROFILE_NAME
- echo "Updating the service..."
- ecs-cli up --capability-iam --size 1 --instance-type t2.medium --cluster-config $CONFIG_NAME --ecs-profile $PROFILE_NAME --force
- ecs-cli compose --file ./docker-compose.prod.yml up --create-log-groups --cluster-config $CONFIG_NAME --ecs-profile $PROFILE_NAME --force-update
only:
- master
ecs-params.yml
version: 1
task_definition:
task_execution_role:
services:
proxy:
essential: true
api:
essential: true
worker:
essential: true
project structure
I've attached configuration files.
I think, I missed some AWS configurations, but I can't find mistakes.
How can I fix it?