I'm trying to follow this tutorial on AWS ECS integration that mentions the Docker command docker compose convert that is supposed to generate a AWS CloudFormation template.
However, when I run this command, it doesn't appear to exist.
$ docker-compose convert
No such command: convert
#...
$ docker compose convert
docker: 'compose' is not a docker command.
See 'docker --help'
$ docker context create ecs myecscontext
"docker context create" requires exactly 1 argument.
See 'docker context create --help'.
Usage: docker context create [OPTIONS] CONTEXT
Create a context
$ docker --version
Docker version 19.03.13, build 4484c46
$ docker-compose --version
docker-compose version 1.25.5, build unknown
$ docker version
Client:
Version: 19.03.13
API version: 1.40
Go version: go1.13.8
Git commit: 4484c46
Built: Thu Oct 15 18:34:11 2020
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 19.03.11
API version: 1.40 (minimum version 1.12)
Go version: go1.13.12
Git commit: 77e06fd
Built: Mon Jun 8 20:24:59 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit:
docker-init:
Version: 0.18.0
GitCommit: fec3683
$ docker info
Client:
Debug Mode: false
Server:
Containers: 12
Running: 3
Paused: 0
Stopped: 9
Images: 149
Server Version: 19.03.11
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version:
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.8.0-29-generic
Operating System: Ubuntu Core 16
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 7.202GiB
Name: HongLee
ID: GZ5R:KQDD:JHOJ:KCUF:73AE:N3NY:MWXS:ABQ2:2EVY:4ABJ:H375:J64V
Docker Root Dir: /var/snap/docker/common/var-lib-docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Any ideas?
To get the ECS integration, you need to be using an ECS docker context. First, enable the experimental flag in /etc/docker/daemon.json
// /etc/docker/daemon.json
{
"experimental": true
}
Then create the context:
docker context create ecs myecscontext
docker context use myecscontext
$ docker context ls
NAME TYPE DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR
default moby Current DOCKER_HOST based configuration unix:///var/run/docker.sock [redacted] (default) swarm
myecscontext * ecs
Now run convert:
$ docker compose convert
WARN[0000] services.build: unsupported attribute
AWSTemplateFormatVersion: 2010-09-09
Resources:
AdminwebService:
DependsOn:
- AdminwebTCP80Listener
Properties:
Cluster:
...
You're running on Ubuntu. The /usr/bin/docker installed (even with latest docker-ce 20.10.6) does not enable the docker compose subcommand. It is enabled by default on Docker for Desktop Windows or Mac.
See the Linux installation instructions at https://github.com/docker/compose-cli to download and configure so that docker compose works.
There's a curl|bash script for Ubuntu or just download the latest release, put that docker executable into a PATH directory before /usr/bin/ and make sure the original docker is available as com.docker.cli e.g. ln -s /usr/bin/docker ~/bin/com.docker.cli.
Related
I have a c++ grpc server image running on GKE kubernetes and I was trying to profile with heaptrack.
in the docker image I installed heaptrack via apt-get, leaving out unrelated stuff the dockerfile looks like this
FROM ubuntu:20.04 as build
.....
RUN apt-get update && apt install -y software-properties-common && \
apt-get -y --no-install-recommends install \
....
heaptrack
...
ENTRYPOINT ["heaptrack", "./grpc_server"]
this creates a docker image which I store on google container registry
I then deploy the image via a yaml file
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
labels:
app: app
spec:
selector:
matchLabels:
app: app
template:
metadata:
labels:
app: app
spec:
containers:
- name: grpc-server
image: grpc_server_container_repo
resources:
requests:
memory: "54Gi"
cpu: "14"
limits:
memory: "64Gi"
cpu: "16"
ports:
- name: grpc
containerPort: 8080
after I deployed the image I ssh into the container with the command
kubectl exec -it app-6c88bd5854-98dg4 -c grpc-server -- /bin/bash
and I saw the file
heaptrack.grpc_server.1.gz
despite the server still running
I opened this file using heaptrack_gui but it shows total runtime as ~2s, I make a couple request to the server and this file is never updated again. I tried running
ps -aux
in the container and I can see
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 2608 1728 ? Ss 08:49 0:00 /bin/sh /usr/bin/heaptrack ./grpc_server
root 20 0.4 0.0 1817048 31340 ? Sl 08:49 0:06 ./grpc_server
root 113 0.0 0.0 4240 3516 pts/0 Ss 09:08 0:00 /bin/bash
root 125 0.0 0.0 6136 2928 pts/0 R+ 09:12 0:00 ps -aux
seems like I have 2 running instance of the server, one with heaptrack the other don't. I'm not sure what's going on here and was hoping someone can point me some direction to how can I profile a running server on k8s with heaptrack.
During building of a 3rd party library (libtorch, if it matters) in a docker container, I came across an error of a missing include file.
The same process of building worked fine when running the build process from Ubuntu 16.04 host, but when running from an Ubuntu 18.04 host, the file was missing.
After a bit of trace back, I'm now just running the base container from NVidia, and looking for the file.
This is the outputs I get:
Ubuntu 16.04 host:
$ uname -a
Linux ub-carmel 4.15.0-123-generic #126~16.04.1-Ubuntu SMP Wed Oct 21 13:48:05 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ docker --version
Docker version 19.03.13, build 4484c46d9d
$ docker pull nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
11.1-cudnn8-devel-ubuntu18.04: Pulling from nvidia/cuda
Digest: sha256:c5bf5c984998cc18a3f3a741c2bd7187ed860dc6d993b6fb402d0effb9fe6579
Status: Image is up to date for nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
$ docker run -it nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
root#2ecc17248fab:/# ll /usr/lib/gcc/x86_64-linux-gnu/7/include | grep ia32
-rw-r--r-- 1 root root 7817 Dec 4 2019 ia32intrin.h
Ubuntu 18.04 host:
$ uname -a
Linux ub-carmel-18-04 5.4.0-56-generic #62~18.04.1-Ubuntu SMP Tue Nov 24 10:07:50 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ docker --version
Docker version 19.03.14, build 5eb3275d40
$ docker pull nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
11.1-cudnn8-devel-ubuntu18.04: Pulling from nvidia/cuda
Digest: sha256:c5bf5c984998cc18a3f3a741c2bd7187ed860dc6d993b6fb402d0effb9fe6579
Status: Downloaded newer image for nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
$ docker run -it nvcr.io/nvidia/cuda:11.1-cudnn8-devel-ubuntu18.04
root#89f771e82a51:/# ll /usr/lib/gcc/x86_64-linux-gnu/7/include | grep ia32
root#89f771e82a51:/#
As you can see, the sha256 digest of the images is the same (and matches the digest from NVidia's NGC here)
At first I thought that maybe in some hidden way the includes come from the host, but the ia32intrin.h file exists in both hosts
What can cause such issue?
EDIT
Added the docker --version outputs for each host. There's a difference, but I doubt this should cause such issues
EDIT 2
Added the output for uname -a
EDIT 3
Output of docker version:
Ubuntu 16:
$ docker version
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:59 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.13
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:01:30 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.7
GitCommit: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
Ubuntu 18:
$ docker version
Client: Docker Engine - Community
Version: 19.03.14
API version: 1.40
Go version: go1.13.15
Git commit: 5eb3275d40
Built: Tue Dec 1 19:20:17 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.14
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 5eb3275d40
Built: Tue Dec 1 19:18:45 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.9
GitCommit: ea765aba0d05254012b0b9e595e995c09186427f
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
So I tested it on a different Ubuntu machines (EC2 instances) and in that case, for both 18.04 & 16.04 the file exists. so looks like it's a problem on my machine.
Any thoughts of what can cause this?
Best guess is that the pulled layers on the Ubuntu 18.04 host are somehow corrupt. The nuclear option to clean that up is to reset docker. This will delete all images, volumes, containers, logs, networks, everything, so backup anything you want to keep before running this:
sudo -s # these commands need root
systemctl stop docker
rm -rf /var/lib/docker
systemctl start docker
exit # exit sudo
I have an AWS CodeBuild project, and I need to call the SAM CLI inside my CodeBuild container. In the build phase, I added a command to install Linux Homebrew, so that I can install the SAM CLI from the AWS Homebrew tap, per the documentation.
However, upon running this command, I am receiving the error below.
[Container] 2020/01/20 05:29:26 Running command bash -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"
-e:196: warning: Insecure world writable dir /go/bin in PATH, mode 040777
Don't run this as root!
[Container] 2020/01/20 05:29:28 Command did not exit successfully bash -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)" exit status 1
[Container] 2020/01/20 05:29:28 Phase complete: BUILD State: FAILED
[Container] 2020/01/20 05:29:28 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: bash -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)". Reason: exit status 1
I'm using the Ubuntu Standard "3.0" build environment, that AWS provides.
buildspec.yml
version: 0.2
phases:
install:
runtime-versions:
docker: 18
nodejs: 10
python: 3.8
build:
commands:
- echo Installing SAM CLI
- sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"
- brew tap aws/tap
- brew install aws-sam-cli
- sam version
Question: How can I successfully install Linux Homebrew inside an AWS CodeBuild project?
First and recommended option is to bring your own build image with CodeBuild, e.g. use [1] which is an image that includes aws sam cli.
[1] https://hub.docker.com/r/pahud/aws-sam-cli
Second and more difficult option is to install the SAM CLI yourself. Since brew cannot be used as root in any way and the CodeBuild build container is running as root, this gets tricky. Following is a buildspec I have tested and can confirm will install the aws sam cli:
Buildspec:
version: 0.2
phases:
install:
commands:
- curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh > /tmp/install.sh
- cat /tmp/install.sh
- chmod +x /tmp/install.sh
- useradd -m brewuser
- echo "brewuser:brewuser" | chpasswd
- adduser brewuser sudo
- /bin/su -c /tmp/install.sh - brewuser
- /bin/su -c '/home/brewuser/.linuxbrew/bin/brew tap aws/tap' - brewuser
- /bin/su -c '/home/brewuser/.linuxbrew/bin/brew install aws-sam-cli' - brewuser
build:
commands:
- PATH=/home/brewuser/.linuxbrew/bin:$PATH
- sam --version
Note: As per my tests, Python 3.8 does not include sam cli.
Building on #shariqmaws answer, I used a public ECR image that includes AWS SAM and Node.js 10: public.ecr.aws/sam/build-nodejs10.x:latest. You can find out more here: https://gallery.ecr.aws/sam/build-nodejs10.x
CloudFormation template:
CodeBuildIntegrationProject:
Type: AWS::CodeBuild::Project
Properties:
Artifacts:
Type: CODEPIPELINE
Environment:
Type: LINUX_CONTAINER
Image: public.ecr.aws/sam/build-nodejs10.x:latest
ImagePullCredentialsType: CODEBUILD
ComputeType: BUILD_GENERAL1_SMALL
LogsConfig:
CloudWatchLogs:
Status: ENABLED
Name: !Sub ${GitHubRepositoryName}-integration
ServiceRole: !Sub ${CodeBuildRole.Arn}
Source:
Type: CODEPIPELINE
I'm getting this error when I tried to docker-compose build my docker-compose.yml file:
In file './docker-compose.yml' service 'version' doesn't have any configuration options. All top level keys in your docker-compose.yml must map to a dictionary of configuration options.
docker version
Client:
Version: 1.12.6
API version: 1.24
Go version: go1.6.3
Git commit: 78d1802
Built: Tue Jan 31 23:47:34 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Go version: go1.6.3
Git commit: 78d1802
Built: Tue Jan 31 23:47:34 2017
OS/Arch: linux/amd64
docker --version
Docker version 1.12.6, build 78d1802
docker-compose --version
docker-compose version 1.5.2, build unknown
is this because the build unknown?
docker-composer.yml
version: "2"
services:
postgres:
image: postgres:9.6
volumes:
- pgdata:/var/lib/data/postgres
backend:
build: .
command: gosu app bash
volumes:
- .:/app
- pyenv:/python
links:
- postgres:postgres
ports:
- 8000:8000
volumes:
pyenv:
pgdata:
Try upgrading the docker-compose version. Version 2 files are supported by Compose 1.6.0+ and require a Docker Engine of version 1.10.0+.
Install latest "docker-compose" -
$ sudo curl -o /usr/local/bin/docker-compose -L "https://github.com/docker/compose/releases/download/1.15.0/docker-compose-$(uname -s)-$(uname -m)"
$ sudo chmod +x /usr/local/bin/docker-compose
Ref-
https://docs.docker.com/compose/compose-file/compose-versioning/#version-2
You should install docker-compose using the official documentation https://docs.docker.com/compose/install/
If you are using linux, I have found the apt install for docker-compose shows some weird behavior. So uninstall docker-compose and reinstall it using the official documentation above.
sudo apt-get purge docker-compose
I am trying to deploy a django app using Codeship Pro to Elasticbeans talk(using docker of course). Running this step fails when when deploying
codeship-steps.yml:
- name: deployment
tag: aws-docker
service: awsdeployment
command: codeship_aws eb_deploy ./deploy my-project staging my-bucket
docker-compose.yml
services:
app:...
db:...
awsdeployment:
image: codeship/aws-deployment
encrypted_env_file: aws-deployment.env.encrypted
environment:
- AWS_DEFAULT_REGION=eu-central-1
volumes:
- ./:/deploy
Error:
Info: I am trying to setup a CI/CD environment for the project(staging/production env)
UPDATE: in my elasticbeanstalk, I see that the django extension is not found although I have made they are installed in my Dockerfile.(pip install -r requirements.txt)