What's the image name I should use for docker compose? having some trouble getting it all working
Pulling questdb (questdb:)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.
Continue with the new image? [yN]y
Pulling questdb (questdb:)...
ERROR: pull access denied for questdb, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
You are trying to pull questdb but the image needs to be in the format <org>/<image>:
services:
questdb:
image: questdb/questdb
...
Please find an example docker-compose image in the following GitHub repository which shows how to start Grafana and QuestDB networked together
Related
I have a GitHub Action that pushes my image to Artifact Registry. This is the steps that authenticates and then pushes it to my Google Cloud Artifact Registry
- name: Configure Docker Client
run: |-
gcloud auth configure-docker --quiet
gcloud auth configure-docker $GOOGLE_ARTIFACT_HOST_URL --quiet
- name: Push Docker Image to Artifact Registry
run: |-
docker tag $IMAGE_NAME:latest $GOOGLE_ARTIFACT_HOST_URL/$PROJECT_ID/images/$IMAGE_NAME:$GIT_TAG
docker push $GOOGLE_ARTIFACT_HOST_URL/$PROJECT_ID/images/$IMAGE_NAME:$GIT_TAG
Where $GIT_TAG is always 'latest'
I want to add one more command that then purges all except the latest version. In this screenshot below, you see theres 2 images
I would like to remove the one that was 3 days ago as its not the one with the tag 'latest'.
Is there a terminal command to do this?
You may initially check through the filtered list of container images for your specific criteria.
gcloud artifacts docker images list --include-tags
Once you have the view of the images to be deleted, move to the following.
You may use the following command to delete an Artifact Registry container image.
gcloud artifacts docker images delete IMAGE [--async] [--delete-tags] [GCLOUD_WIDE_FLAG …]
A valid container image that can be referenced by tag or digest, has the format of
LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY-ID/IMAGE:tag
This command can fail for the following reasons:
Trying to delete an image by digest when the image is still tagged.
Add --delete-tags to delete the digest and the tags.
Trying to delete an image by tag when the image has other tags. Add
--delete-tags to delete all tags.
A valid repository format was not provided.
The specified image does not exist.
The active account does not have permission to delete images.
It is always recommended to check and reconfirm any deletion operations so you don’t lose any useful artifacts and data items.
Also check this helpful document here for Artifacts docker Image Deletion guidelines and some useful information on Managing Images.
As guillaume blaquiere mentioned you may have a look at this link which may help you.
I'm trying to create a application from a dockerfile in my Private Bitbucket repo, when I go through the web console to make it I've tried using both the SSH and HTTP url's and the ssl/user+pass secrets but its not detecting the Dockerfile in my repo.
I've also tried using the CLI to create it both on Windows and Ubuntu...
oc new-app git#bitbucket.org:mitch-user/myapp.git --context-dir src --source-secret bitbucketssl --strategy docker --name myapp
but I get the following
warning: Cannot check if git requires authentication.
W1126 11:17:56.680242 31748 dockerimagelookup.go:300] container image remote registry lookup failed: you may not have access to the container image "docker.io/library/base:latest"
error: only a partial match was found for "base": "openshift/jenkins-agent-base:latest"
The argument "base" only partially matched the following container image, OpenShift image stream, or template:
* Image stream "jenkins-agent-base" (tag "latest") in project "openshift"
Use --image-stream="openshift/jenkins-agent-base:latest" to specify this image or template
Has anyone seen this issue before? any help would be appreciated.
How do I use a custom builder image in Cloud Build which is stored in a repository in Artifact Registry (instead of Container Registry?)
I have set up a pipeline in Cloud Build where some python code is executed using official python images. As I want to cache my python dependencies, I wanted to create a custom Cloud Builder as shown in the official documentation here.
GCP clearly indicates to switch to Artifact Registry as Container Registry will be replaced by the former. Consequently, I have pushed my docker image to Artifact Registry. I also gave my Cloud Builder Service Account the reader permissions to Artifact Registry.
Using the image in a Cloud Build step like this
steps:
- name: 'europe-west3-docker.pkg.dev/xxxx/yyyy:latest'
id: install_dependencies
entrypoint: pip
args: ["install", "-r", "requirements.txt", "--user"]
throws the following error
Step #0 - "install_dependencies": Pulling image: europe-west3-docker.pkg.dev/xxxx/yyyy:latest
Step #0 - "install_dependencies": Error response from daemon: manifest for europe-west3-docker.pkg.dev/xxxx/yyyy:latest not found: manifest unknown: Requested entity was not found.
"xxxx" is the repository name and "yyyy" the name of my image. The tag "latest" exists.
I can pull the image locally and access the repository.
I could not find any documentation on how to integrate these images from Artifact Registry. There is only this official guide, where the image is built using the Docker image from Container Registry – however this should not be future proof.
It looks like you need to add your Project ID to your image name.
You can use the "$PROJECT_ID" Cloud Build default substitution variable.
So your updated image name would look something like this:
steps:
- name: 'europe-west3-docker.pkg.dev/$PROJECT_ID/xxxx/yyyy:latest'
For more details about substituting variable values in Cloud Build see:
https://cloud.google.com/build/docs/configuring-builds/substitute-variable-values
I'm trying to get docker-compose deployment to AWS Elastic Beanstalk working, in which the docker images are pulled from a private registry hosted by GitLab.
The strange thing is that initial deployment works perfectly; It pulls the image from the private registry and starts the containers using docker-compose, and the webpage (served by Django) is accessible through the host.
Deploying a new version using the same docker-compose and the same docker image will result in an error while pulling the docker image:
2021/03/16 09:28:34.957094 [ERROR] An error occurred during execution of command [app-deploy] - [Run Docker Container]. Stop running the command. Error: failed to run docker containers: Command /bin/sh -c docker-compose up -d failed with error exit status 1. Stderr:Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "current_default" with the default driver
Pulling redis (redis:alpine)...
Pulling mysql (mysql:5.7)...
Pulling project.dockertest(registry.gitlab.com/company/spikes/dockertest:latest)...
Get https://registry.gitlab.com/v2/company/spikes/dockertest/manifests/latest: denied: access forbidden
2021/03/16 09:28:34.957104 [INFO] Executing cleanup logic
Setup
AWS Elastic Beanstalk 64bit Amazon Linux 2/3.2
Gitlab registry credentials are stored within a S3 bucket, with the filename .dockercfg and has the following content:
{
"auths": {
"registry.gitlab.com": {
"auth": "base64 encoded username:personal_access_token"
}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/18.03.1-ce (linux)"
}
}
The repository contains a v3 Dockerrun.aws.json file to refer to the credential file in S3:
{
"AWSEBDockerrunVersion": "3",
"Authentication": {
"bucket": "gitlab-dockercfg",
"key": ".dockercfg"
}
}
Reproduce
Setup docker-compose.yml that uses a service with a private docker image (and can be pulled with the credentials setup in the dockercfg within S3)
Create a new applicatoin that uses the docker-platform.
eb init testapplication --platform=docker --region=eu-west-1
Note: region must be the same as the S3 bucket containing the dockercfg.
Initial deployment (this will succeed)
eb create testapplication-test --branch_default --cname testapplication-test --elb-type=application --instance-types=t2.micro --min-instance=1 --max-instances=4
The initial deployment shows that the image is available and can be started:
2021/03/16 08:58:07.533988 [INFO] save docker tag command: docker tag 5812dfe24a4f redis:alpine
2021/03/16 08:58:07.533993 [INFO] save docker tag command: docker tag f8fcde8b9ae2 mysql:5.7
2021/03/16 08:58:07.533998 [INFO] save docker tag command: docker tag 1dd9b65d6a9f registry.gitlab.com/company/spikes/dockertest:latest
2021/03/16 08:58:07.534010 [INFO] Running command /bin/sh -c docker rm `docker ps -aq`
Without changing anything to the local repository and the remote docker image on the private registry, lets do a redeployment which will trigger the error:
eb deploy testapplication-test
This will fail with the following output:
...
2021-03-16 10:02:28 INFO Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2021-03-16 10:02:29 ERROR Unsuccessful command execution on instance id(s) 'i-0dc445d118ac14b80'. Aborting the operation.
2021-03-16 10:02:29 ERROR Failed to deploy application.
ERROR: ServiceError - Failed to deploy application.
And logs of the instance show (/var/log/eb-engine.log):
Pulling redis (redis:alpine)...
Pulling mysql (mysql:5.7)...
Pulling project.dockertest (registry.gitlab.com/company/spikes/dockertest:latest)...
Get https://registry.gitlab.com/v2/company/spikes/dockertest/manifests/latest: denied: access forbidden
2021/03/16 10:02:25.902479 [INFO] Executing cleanup logic
Steps I've tried to debug or solve the issue
Rename dockercfg to .dockercfg on S3 (somewhere mentioned on the internet as possible solution)
Use the 'old' docker config format instead of the one generated by docker 1.7+. But later on I figured out that Amazon Linux 2-instances are compatible with the new format together with Dockerrun v3
Having an incorrectly formatted dockercfg on S3 will cause an error deployment regarding the misformatted file (so it actually does something with the dockercfg from S3)
Documentation
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/single-container-docker-configuration.html
I'm out of debug options, and I've no idea where to look any further to debug this problem. Perhaps someone can see what is going wrong here?
First of all, the issue describe above is a bug confirmed by Amazon. To get the deployment working on our side, we've contacted Amazon support.
They've a fix in place which should be released this month, so keep an eye on the changelog of the Elastic beanstalk platform: https://docs.aws.amazon.com/elasticbeanstalk/latest/relnotes/relnotes.html
Although the upcoming release should have the fix, there is a workaround available to get the docker-compose deployment working.
Elastic Beanstalk allows hook to be executed within the deployment, which can be used to fetch the .docker.cfg from a S3 bucket to authenticate with against the private registry.
To do so, create the following file and directories from the root of the project:
File location: .platform/hooks/predeploy/docker_login
#!/bin/bash
aws s3 cp s3://{{bucket_name_to_use}}/.dockercfg ~/.docker/config.json
Important: Add execution rights to this file (for example: chmod +x .platform/hooks/predeploy/docker_login)
To support instance configuration changes, please symlink the hooks directory to confighooks:
ln -s .platform/hooks/ .platform/confighooks/
Updating configuration requires the .dockercfg credentials to be fetched too.
This should enable continuous deployments to the same EB-instance without the authentication errors, because the hook will be execute before the docker image pulling.
Some background:
The docker daemon reads credentials from ~/.docker/config by default on traditional linux systems. On the initial deploy this file will exist on the Elastic Beanstalk instance. On the next deployment this file is removed. Unfortunately, on the next deployment the .dockercfg is not refetched, therefor the docker daemon does not have the correct credentials to authenticate with.
I was dealing the same errors while trying to pull images from a privately hosted GitLab instance. I was able to resolve them by including the email address that was associated with the generated token found in the auth field of the .dockercfg file.
The following file format worked for me:
"registry.gitlab.com" {
"auth": "base64 encoded username:personal_access_token",
"email": "email for personal access token"
}
In my case I used a Project Access Token, which has an e-mail address associated with it once it is created.
The file format in the Elastic Beanstalk documentation for the authentication file here, indicates that this is the required file format, though the versions that it says this format is required for are almost certainly outdated, since we are running Docker ^19.
I have a docker image that is 1.46GB on my local machine, but when this is pushed to AWS ECR (either via my local machine or via CicleCI deployment) it is only 537.05MB. I'm pretty new to Docker and to AWS, so any help in figuring out as to why this may be would be appreciated!
I have a feeling that it has not fully uploaded to ECR for whatever reason, as I am trying to use this container for a Batch job, but for some reason the same command which works when used locally does not work when used in the job definition. The command is simply python app.py, but I have also tried with absolute path python /usr/local/src/app/app.py, both of which result in [Errno 2] No such file or directory.
Commands used in my Makefile deployment are as below:
docker build --force-rm=true -t $(EXTRACTOR_IMAGE_NAME) ./extractor
docker tag $(EXTRACTOR_IMAGE_NAME) $(EXTRACTOR_ECR_IMAGE_NAME)
$(shell aws ecr get-login --no-include-email)
docker push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/$(EXTRACTOR_ECR_REPO)
Edit 1:
I think this might be to do with the size of the base image, which is python:2.7 in this case. The base image is 914MB, plus the size of my ECR image 537.05MB = 1451.05MB, i.e. approx 1.46GB. Still not sure what the issue is with the Batch command though...
Edit 2:
I've been mounting code into my container using a volume, which is why this has been working locally. At build time I've forgotten to copy the code into the container, which I assume is the only reason why this is not working in Batch!
That could be due to how docker client acts before it pushes the image to ECR as documented:
Beginning with Docker version 1.9, the Docker client compresses image layers before pushing them to a V2 Docker registry. The output of the docker images command shows the uncompressed image size, so it may return a larger image size than the image sizes shown in the AWS Management Console.
So when you pull an image you will notice that the image layers go through three stages:
Downloading
Extraction
Completion
Regarding this command: python /usr/local/src/app/app.py are executing it while you inside /usr/local/src/app/ ? You might need to ensure this first also have you checked the command inside the container using the image before you push it as the error seems to be code related more than a docker issue
We can read the following in the the AWS ECR documentation:
Note
Beginning with Docker version 1.9, the Docker client compresses image layers before pushing them to a V2 Docker registry. The output of the docker images command shows the uncompressed image size, so it may return a larger image size than the image sizes shown in the AWS Management Console.
I suspect you'd get the sizes you expect, would you use the CLI (docker images) instead of the ECR web console.