I want to understand the use case of volumes.
eg: i created a docker image using httpd:alipine as my base image. In this images dockerfile i have line COPY ./ui /usr/local/apache2/htdocs.
Whenever i create a new container using this image the images data i.e ./ui folder is added into the new container. So why do i need a volume instead. Even if i use a volume and have data inside it my new container will need a reference image where i use my custom image. So why do i need a volume in any case when the data will be added by the image itself.
eg: docker build -t myimage
docker create container --name mycontainer -p 8080:80 -v my-vol:/htdocs myimage
This new created container will have all the data of the custom image so what is the point of the volume in any case apart from just being a data staging area. Like this any new container created will always carry the data of the reference image used.
Its not recommended to copy application runtime data into docker image. Only configuration data should be added in docker images creation process as portability is one of the main concern in docker. Docker Image size will unnecessarily increase by adding application runtime data. Please follow best practices while creating docker images.
Related
I am trying to run a Custom Training Job in Google Cloud Platform's Vertex AI Training service.
The job is based on a tutorial from Google that fine-tunes a pre-trained BERT model (from HuggingFace).
When I use the gcloud CLI tool to auto-package my training code into a Docker image and deploy it to the Vertex AI Training service like so:
$BASE_GPU_IMAGE="us-docker.pkg.dev/vertex-ai/training/pytorch-gpu.1-7:latest"
$BUCKET_NAME = "my-bucket"
gcloud ai custom-jobs create `
--region=us-central1 `
--display-name=fine_tune_bert `
--args="--job_dir=$BUCKET_NAME,--num-epochs=2,--model-name=finetuned-bert-classifier" `
--worker-pool-spec="machine-type=n1-standard-4,replica-count=1,accelerator-type=NVIDIA_TESLA_V100,executor-image-uri=$BASE_GPU_IMAGE,local-package-path=.,python-module=trainer.task"
... I end up with a Docker image that is roughly 18GB (!) and takes a very long time to upload to the GCP registry.
Granted the base image is around 6.5GB but where do the additional >10GB come from and is there a way for me to avoid this "image bloat"?
Please note that my job loads the training data using the datasets Python package at run time and AFAIK does not include it in the auto-packaged docker image.
The image size shown in the UI is the virtual size of the image. It is the compressed total image size that will be downloaded over the network. Once the image is pulled, it will be extracted and the resulting size will be bigger. In this case, the PyTorch image's virtual size is 6.8 GB while the actual size is 17.9 GB.
Also, when a docker push command is executed, the progress bars show the uncompressed size. The actual amount of data that’s pushed will be compressed before sending, so the uploaded size will not be reflected by the progress bar.
To cut down the size of the docker image, custom containers can be used. Here, only the necessary components can be configured which would result in a smaller docker image. More information on custom containers here.
We're noticing weird SHA1-like tags on our gcr images. The nature of these tags are that
they are the same size as SHA1, i.e. exactly 40 hexadecimal characters
we didn't create them
any image that is tagged by us does not have this weird SHA1-like tag
What are these tagged images and can they be deleted?
The "weird SHA1-like tags" in your Container Registry images is created automatically by Cloud Build. Check this related StackOverflow answer.
As an example, here's an image that is created when I deployed an application on App Engine:
Upon navigating, here's the details:
Yes, there's an option removing or deleting unused image from the container registry.
Just a note, the container images delete command deletes the specified image from the registry, and all associated tags are also deleted.
For example, in order to list all untagged images in a project, first filter digests that lack tags using this command:
gcloud container images list-tags [HOSTNAME]/[PROJECT-ID]/[IMAGE] --filter='-tags:*' --format="get(digest)" --limit=$BIG_NUMBER
Another option is, gcloud container images list-tags [HOSTNAME]/[PROJECT-ID]/[IMAGE] --format=json --limit=unlimited, this will give you an easily consumable json blob of information for images in a repository (such as digests with associated tags).
Then, you can iterate over and delete with this command:
gcloud container images delete [HOSTNAME]/[PROJECT-ID]/[IMAGE]#DIGEST --quiet
I distributed slurm-gcp using Terraform through the GitHub and it was available successfully. Source:
Slurm on Google Cloud Platform
But I want to change the image I use when using node to a custom image.
I am trying to edit /slurm/scripts/config.yaml.
Among the contents of the file:
image: projects/schedmd-slurm-public/global/images/family/schedmd-slurm-20-11-7-hpc-centos-7
I want to edit the part.
How to reroute this part to my custom image?
First you need to create your own image.
Create a new VM with the image you want to modify; make appropriate changes and stop the VM. Then create a new image from the VM's disk.
Next create a custom image from that disk and your path in the config.yaml file can look like this:
image: projects/my-project-name/global/images/your-image-name
You can get exact path to your custom image by running:
wb#cloudshell:~ (wb)$ gcloud compute images describe your-image-name | grep selfLink
selfLink: https://www.googleapis.com/compute/v1/projects/wb/global/images/your-image-name
Quick question and this may be a dumb one. I am attempting to use AWS Code Build with an image I've published to Docker Hub. I selected the option to use a custom image, and the the option to look for the image in another location (an external image repo).
I can't seem to figure out how to reference my image in the appropriate format to use it in the other location field.
Any help would be greatly appreciated.
In the "Other location" text box you can enter the image name from DockerHub. For example, simply give "openjdk" or "openjdk:latest" to use https://hub.docker.com/r/library/openjdk/ as the Docker image for your build. Don't put the "docker pull " prefix for your image name is all.
Note that CodeBuild only supports public Docker images from DockerHub today. Private registries are not supported.
Lets say that you published your image in hub.docker.com, and your repo name is gjackson/myrepo, and you want to grab the image tagged latest, you should populate the other location field with docker.io/gjackson/myrep:latest.
I'm pushing my images to AWS ECR via docker push ... command. The image is tagged with a specific version.
When I actually push two different images with the same tag, this results in two images on the AWS ECR registry, one which become untagged.
0.0.1 sha256:572219f8764b21e5a045bcc1c5eab399e2bc2370a37b23cb1ca9298ea39e233a 138.33 MB
sha256:60d161db0b9cb1345cf7c3e6119b8eba7114bc2dfc44c0b3ed02454803f6ef76 138.21MB
The problem this is causing is that if I continue to push more images with the same tag, the total size of the repository keeps increasing.
What I would like is to "overwrite" the existing tag when pushing an image. Which means that two different sha256 digest with the same tag would result in a single image on the registry (of course multiple when tag version changes).
Is it possible to do so? I would like to avoid an "untagged" pruning technique if possible. For now, my publish script delete the previous same tag if it exists but I feel this should be handled by AWS ECR or docker push directly.
Unfortunately this is not possible. Here is what you can do:
Use 2 different tags for the images that you want to overwrite. I would recommend a tag with the version and another tag with a known prefix and something guaranteed unique e.g. 1.1.1 and SNAPSHOT-hash. The next time you push an image with the same version, the tag 1.1.1 will be removed from the old image and added to the new image. However the SNAPSHOT-* tags will remain in all images.
Configure a lifecycle policy where images starting from SNAPSHOT- will expire after an image count of more than x. This way, old images will automatically expire.