Google cloud run does not support the docker registry, therefore I have to manually pull the image, tag it and push it to GCR.
Container image URL should match pattern [region.]gcr.io/repo-path[:tag or #digest]
Is there any simpler way to do this?
Sadly, that's the easiest way to move a Docker image from one container registry to another one.
Just for documentation purposes, I will add the steps for the benefit of the community:
Pull the Docker image using the following command:
docker pull [REPOSITORY-NAME]/[IMAGE]:[TAG]
Then, tag that pulled image using the following command:
docker tag [IMAGE] gcr.io/[PROJECT-ID]/[IMAGE]
Push that image to your gcr repository using the following command:
docker push gcr.io/[PROJECT-ID]/[IMAGE]
I'm afraid in any case, "simpler" won't be a thing. Though, you may try to use Docker web hooks to call a simple Cloud function (pull, tag, push) in order to keep your images in sync in your GCR.
There seems to be some projects to manage that kind of hassle like dregsy but I didn't try them...
I've been working on some tooling called regclient that supports this use case. For copying a single image, the command would be:
regctl image copy ${source} ${target}
e.g.
regctl image copy ubuntu:latest gcr.io/your-project/ubuntu:latest
This checks the digests before copying with a HEAD request to allow the command to be run frequently but only using your quota when the upstream image doesn't match what's on GCR. It also copies multi-platform images which you wouldn't get with a docker pull and docker push (docker dereferences the image to your platform on the pull). And unlike the docker pull, the individual layers are only copied when they don't exist on the target registry.
If you have lots of images to continuously mirror, there's also a regsync command that copies according to a yaml file with a list of images, tags, and schedule to run the copies.
These can run as containers, but they are also available as standalone binaries that don't require docker to run.
Related
I have a GitHub Action that pushes my image to Artifact Registry. This is the steps that authenticates and then pushes it to my Google Cloud Artifact Registry
- name: Configure Docker Client
run: |-
gcloud auth configure-docker --quiet
gcloud auth configure-docker $GOOGLE_ARTIFACT_HOST_URL --quiet
- name: Push Docker Image to Artifact Registry
run: |-
docker tag $IMAGE_NAME:latest $GOOGLE_ARTIFACT_HOST_URL/$PROJECT_ID/images/$IMAGE_NAME:$GIT_TAG
docker push $GOOGLE_ARTIFACT_HOST_URL/$PROJECT_ID/images/$IMAGE_NAME:$GIT_TAG
Where $GIT_TAG is always 'latest'
I want to add one more command that then purges all except the latest version. In this screenshot below, you see theres 2 images
I would like to remove the one that was 3 days ago as its not the one with the tag 'latest'.
Is there a terminal command to do this?
You may initially check through the filtered list of container images for your specific criteria.
gcloud artifacts docker images list --include-tags
Once you have the view of the images to be deleted, move to the following.
You may use the following command to delete an Artifact Registry container image.
gcloud artifacts docker images delete IMAGE [--async] [--delete-tags] [GCLOUD_WIDE_FLAG …]
A valid container image that can be referenced by tag or digest, has the format of
LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY-ID/IMAGE:tag
This command can fail for the following reasons:
Trying to delete an image by digest when the image is still tagged.
Add --delete-tags to delete the digest and the tags.
Trying to delete an image by tag when the image has other tags. Add
--delete-tags to delete all tags.
A valid repository format was not provided.
The specified image does not exist.
The active account does not have permission to delete images.
It is always recommended to check and reconfirm any deletion operations so you don’t lose any useful artifacts and data items.
Also check this helpful document here for Artifacts docker Image Deletion guidelines and some useful information on Managing Images.
As guillaume blaquiere mentioned you may have a look at this link which may help you.
I have an existing image inside an ECR repo with the tag "780" and I wanted to make a copy of it inside the same repo with the tag "781".
I tried executing the below commands which I found from here but that gives a new tag to the same image when given the same repo.
docker login REPO
docker pull REPO/IMAGE:TAG
docker tag REPO/IMAGE:TAG REPO/IMAGE:NEWTAG
docker push REPO/IMAGE:NEWTAG
Is there an API or utility (preferably in python) or any other way using which this can be achieved?
It's not possible to have two Docker images in the same repo with the same SHA256 hash. The Docker repository is saving space by detecting that they are the same image, so it is simply adding the tags to the image that already exists in the repo. This is working as intended.
I'm running a container-optimized compute instance with this startup-script:
#!/bin/bash
mkdir /home/my-app
cd /home/my-app
export HOME=/home/my-app
docker-credential-gcr configure-docker
docker run --rm --security-opt seccomp=./config.json gcr.io/my-project/my-app:latest
This scripts works well when creating a new instance. But when I restart an existing instance it doesnt't pull the latest image.
I've tried to delete all images from the gcr, the instance was able to start anyways, which proves that it doesn't even try to pull the latest image from gcr.
Also, for some reason startup-script logs are not showing up in Cloud Logger.
As per kubernetes, With Docker, if the image already exists, the pull attempt is fast because all image layers are cached and no image download is needed.
As a workaround you can add step 1 and 2 in to your script:
1- docker images // will show you the list of images including (gcr.io/my-project/my-app:latest)
2- docker rmi --force gcr.io/my-project/my-app:latest // will delete local image
3- docker run (rest of your command, it will download the latest image again from the gcr.io)
How can one download files from a GCP Storage bucket to a Container-Optimised OS (COS) on instance startup?
I know of the following solutions:
gcloud compute copy-files
SSH through console
SCP
Yet all of these have to be done manually and externally after an instance is started.
There is also cloud init, yet I can't find any info on how to copy files from a Storage bucket. Examples seem to be suggesting that it's better to include content of files in the cloud init file directly, which is not something I want to do because security. Is it possible to download files from Storge bucket using cloud init?
I considered using a startup script, yet COS lacks CLI tools such as gcloud or gsutil to be able to run any such commands in a startup script.
I know I could copy the files manually and then save the image as a boot disk, but I'm hoping there are solutions that avoid having to do so.
Most of all, I'm assuming I'm not asking for something impossible, given that COS instance setup allows me to specify Docker volumes that I could mount onto the starting container. This seems to suggest I should be able to have some private files on the instance the moment COS will attempt to run my image on startup. But how?
Trying to execute a startup-script with a cloud-sdk image and copying files there as suggested by Guillaume didn't work for me for a while, showing this log. Eventually I realised that the cloud-sdk image is 2.41GB when uncompressed and takes over 2 minutes to complete pulling. I tried again with an empty COS instance and the startup script completed successfully, downloading the data from a Storage bucket.
However, a 2.41GB image and over 2 minutes of boot time sound like a bit of an overkill to download a 2KB file. Don't they?
I'm glad to see a working solution to my question (thanks Guillaume!) although I'm still wondering: isn't there a nicer way to do this? I feel that this method is even less tidy than manually putting the files on the COS instance and then creating a machine image to use in the future.
Based on Guillaume's answer I created and published a gsutil wrapper image, available as voyz/gsutil_wrap. This way I am able to run a startup-script with the following command:
docker run -v /host/path:/container/path \
--entrypoint gsutil voyz/gsutil_wrap \
cp gs://bucket/path /container/path
It's essentially a copy of what Guillaume suggested, except it is using an image containing only a minimum setup required to run gsutil. As a result it weighs 0.22GB and pulls within 10-20 seconds on average - as opposed to 2.41GB and over 2 minutes respectively for the google/cloud-sdk image suggested by Guillaume.
Also, credit to this incredibly useful StackOverflow answer that allows gsutil to use the default service account for authentication.
The startup-script is the correct location to do this. And YES, COS lacks some useful library.
BUT you can run container! And, for example, the Google Cloud SDK container!
So, add this startup-script in the VM metadata:
key -> startup-script
value ->
docker run -v /local/path/to/copy/files:/dummy/container/path \
--entrypoint gsutil google/cloud-sdk \
cp gs://your_bucket/path/to/file /dummy/container/path
Note: the startup script is ran in root mode. Perform a chmod/chown in your startup script if you need to change the file access mode.
Let me know if you need more explanation on this command line
Of course, with a fresh COS image, the startup time is quite long (pull the container image and extract it).
To reduce the startup time, you can "bake" your image. I mean, start with a COS, download/install what you want on it (or only perform a docker pull of the googkle/cloud-sdk container) and create a custom image from this.
Like this, all the required dependencies will be present on the image and the boot start will be quicker.
I have a docker image that is 1.46GB on my local machine, but when this is pushed to AWS ECR (either via my local machine or via CicleCI deployment) it is only 537.05MB. I'm pretty new to Docker and to AWS, so any help in figuring out as to why this may be would be appreciated!
I have a feeling that it has not fully uploaded to ECR for whatever reason, as I am trying to use this container for a Batch job, but for some reason the same command which works when used locally does not work when used in the job definition. The command is simply python app.py, but I have also tried with absolute path python /usr/local/src/app/app.py, both of which result in [Errno 2] No such file or directory.
Commands used in my Makefile deployment are as below:
docker build --force-rm=true -t $(EXTRACTOR_IMAGE_NAME) ./extractor
docker tag $(EXTRACTOR_IMAGE_NAME) $(EXTRACTOR_ECR_IMAGE_NAME)
$(shell aws ecr get-login --no-include-email)
docker push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/$(EXTRACTOR_ECR_REPO)
Edit 1:
I think this might be to do with the size of the base image, which is python:2.7 in this case. The base image is 914MB, plus the size of my ECR image 537.05MB = 1451.05MB, i.e. approx 1.46GB. Still not sure what the issue is with the Batch command though...
Edit 2:
I've been mounting code into my container using a volume, which is why this has been working locally. At build time I've forgotten to copy the code into the container, which I assume is the only reason why this is not working in Batch!
That could be due to how docker client acts before it pushes the image to ECR as documented:
Beginning with Docker version 1.9, the Docker client compresses image layers before pushing them to a V2 Docker registry. The output of the docker images command shows the uncompressed image size, so it may return a larger image size than the image sizes shown in the AWS Management Console.
So when you pull an image you will notice that the image layers go through three stages:
Downloading
Extraction
Completion
Regarding this command: python /usr/local/src/app/app.py are executing it while you inside /usr/local/src/app/ ? You might need to ensure this first also have you checked the command inside the container using the image before you push it as the error seems to be code related more than a docker issue
We can read the following in the the AWS ECR documentation:
Note
Beginning with Docker version 1.9, the Docker client compresses image layers before pushing them to a V2 Docker registry. The output of the docker images command shows the uncompressed image size, so it may return a larger image size than the image sizes shown in the AWS Management Console.
I suspect you'd get the sizes you expect, would you use the CLI (docker images) instead of the ECR web console.