AWS beanstalk docker-pool devmapper running out thin pool - amazon-web-services

Question

Hi I have a few clusters running with AWS beanstalk running docker images. My docker image size is 680MB and I verified that the Docker data volume (/dev/xvdcz) is at capacity of 12GB. I also verified that there are no dangling images / stale containers to prune. Currently it is holding 2 images (so should be roughly 1.4GB).
That said my new application deployments were consistently failing with error:
layer: devmapper: Thin Pool has 1805 free data blocks which is less than minimum required 2425 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior
On future investigation, I found there is a setting in /etc/sysconfig/docker-storage that sets the dm.baseSize to 100GB. Here's the file content:
DOCKER_STORAGE_OPTIONS="--storage-driver devicemapper --storage-opt dm.thinpooldev=/dev/mapper/docker-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true --storage-opt dm.fs=ext4 --storage-opt dm.basesize=100G"
This is coming from default AWS AMI. After that I added an ebextension to increase the docker volume (/dev/xvdcz) to 120GB and things have not failed since.
My question is why do we need baseSize to be 100GB and is there a way I can reduce it down to 10GB so I don't have to attach additional storage for each box? Ideally without building custom AMIs.
Here's the new config after I updated the volume to 120GB.
[ec2-user ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:1 0 8G 0 disk
├─nvme0n1p1 259:2 0 8G 0 part /
└─nvme0n1p128 259:3 0 1M 0 part
nvme1n1 259:0 0 120G 0 disk
└─nvme1n1p1 259:6 0 120G 0 part
├─docker-docker--pool_tdata 253:1 0 118.6G 0 lvm
│ └─docker-docker--pool 253:2 0 118.6G 0 lvm
│ └─docker-259:2-394917-4a52cb647e0a91037aaf4fb64345ae55567f2d8c5ff073048981987e6dcba7b0 253:3 0 100G 0 dm /var/lib/docker/devicemapper/mnt/4a52cb647e0a91037aaf4fb64345ae55567f2d8c5ff073048981987e6dcba7b0
└─docker-docker--pool_tmeta 253:0 0 124M 0 lvm
└─docker-docker--pool 253:2 0 118.6G 0 lvm
└─docker-259:2-394917-4a52cb647e0a91037aaf4fb64345ae55567f2d8c5ff073048981987e6dcba7b0 253:3 0 100G 0 dm /var/lib/docker/devicemapper/mnt/4a52cb647e0a91037aaf4fb64345ae55567f2d8c5ff073048981987e6dcba7b0

Answer 1

By default, EB only gives you 12GB of storage for images.
To expand it to 100GB, Create a .ebextensions/blockdevice-xvdcz.config file with the following contents.
option_settings:
aws:autoscaling:launchconfiguration:
BlockDeviceMappings: /dev/xvdcz=:100:true:gp2
See the "Configuring additional storage volumes" section of https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker.container.console.html#docker-volumes
For improved performance on Amazon Linux AMI, Elastic Beanstalk
configures two Amazon EBS storage volumes for your Docker
environment's Amazon EC2 instances. In addition to the root volume
provisioned for all Elastic Beanstalk environments, a second 12GB
volume named xvdcz is provisioned for image storage on Docker
environments.
If you need more storage space or increased IOPS for Docker images,
you can customize the image storage volume by using the
BlockDeviceMapping configuration option in the
aws:autoscaling:launchconfiguration namespace.
For example, the following configuration file increases the storage
volume's size to 100 GB with 500 provisioned IOPS: