AWS Beanstalk can run applications from Docker containers.
As mentioned in the docs (http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_image.html) it's possible to write directory mappings to the EC2 volume in the Dockerrun.aws.json:
"Volumes": [
{
"HostDirectory": "/var/app/mydb",
"ContainerDirectory": "/etc/mysql"
}
but, is it possible to mount specific EBS volume?
F.e. I need to run db in the Docker container and deploy it with Beanstalk. It's clear that I need to have persistence of the data, backup/restore for db, etc..
You can mount EBS volumes on any Beanstalk environment. This volume will be available on the EC2 instances.
You can do this using ebextensions option settings. Create a file in your app source .ebextensions/01-ebs.config with the following contents:
option_settings:
- namespace: aws:autoscaling:launchconfiguration
option_name: BlockDeviceMappings
value: /dev/sdj=:100,/dev/sdh=snap-51eef269,/dev/sdb=ephemeral0
The format of the mapping is device name=volume where the device mappings are specified as a single string with mappings separated by a comma. This example attaches to all instances in the autoscaling group an empty 100-GB Amazon EBS volume, an Amazon EBS volume with the snapshot ID snap-51eef269, and an instance store volume.
Read more details about this option setting here.
Read more about ebextensions here.
Once you have mounted the EBS volume for your beanstalk environment instances, you can use the volume mapping as above to map directories per your need.
I guess the leg100/docker-ebs-attach Docker container does what you want, i.e. make a particular existing EBS volume available. You can either copy the .py file and relevant Dockerfile statements or create a multi-container EB setup and mount the volume from this container.
BTW I have tried to mount a new EBS volume as proposed by Rohit (+ commands to format and mount it) and it works but Docker does not see the mount until the docker daemon is restarted.
Related
To Clarify my Autoscaling group removes all instances and their root EBS volumes during inactive hours, then once inside active hours recreates and installs all necessary base programs. However I have a smaller EBS volume that is persistent and holds code and data I do not want getting wiped out during down times. I am currently manually attaching via the console and mounting every time I am working inside active hours using the commands below.
sudo mkdir userVolume
sudo mount /dev/xvdf userVolume
How can I automatically attach and mount this volume to a folder? This is all for the sake of minimizing cost and uptime to when I can actually be working on it.
Use this code:
#!/bin/bash
OUTPUT=$(curl http://169.254.169.254/latest/meta-data/instance-id)
aws ec2 attach-volume --volume-id vol-xxxxxxxxxxxx --device /dev/xvdf --instance-id $OUTPUT --region ap-southeast-1
Set your volume ID and region.
Refer this link for further details: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-spot-instance-attach-ebs-volume/
I'm trying to run ElasticSearch on Docker (actually on AWS ECS). If I don't configure the volume it's working correctly, but every time I restart the container I lose all the data.
I can't figure out how to configure the volume.
What I tried:
in the task definition I configured volume "Name=esdata1" and "source path=/usr/share/elasticsearch/data"
inside the container definition in the "storage and logging" section I configured the mount point "source volume= esdata1" and "container path=/usr/share/elasticsearch/data"
Now when I launch the container it fail with error "access denied" when elasticsearch try to write in "/usr/share/elasticsearch/data". So in the section Security I configured "user=ec2-user" but then the container will not even launch (stay in "status=created"). What should I do? I guess the issue is due to the fact that the user of the container must be the same of the one on the host. The user on the host is "ec2-user", I don't know how to proceed.
Edit:
I'm now able to persist data with this configuration:
docker inspect:
"Mounts": [
{
"Name": "elasticsearch_data",
"Source": "/var/lib/docker/volumes/elasticsearch_data/_data",
"Destination": "/usr/share/elasticsearch/data",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": "rprivate"
}
]
Now data persist if I stop the container or I reboot the host. My only last concern is that this folder "/var/lib/docker/volumes/elasticsearch_data/_data" is located on the OS volume and not on the bigger docker volume. From Aws doc:
Amazon ECS-optimized AMIs from version 2015.09.d and later launch with
an 8 GiB volume for the operating system that is attached at /dev/xvda
and mounted as the root of the file system. There is an additional 22
GiB volume that is attached at /dev/xvdcz that Docker uses for image
and metadata storage. The volume is configured as a Logical Volume
Management (LVM) device and it is accessed directly by Docker via the
devicemapper back end.
How can I persist data on /dev/xvdcz?
Thanks very much
Your sourcepath is the path on the host instance where the data is written. In your case elasticsearch_data. You need to point sourcepath to a folder that exists and that is on the disk you want on the EC2 instance.
So attach an EBS disk to the instance. Mount the disk in a place like /data/es and set your source path to that folder.
But remember that to properly run ES you would probably need a cluster of machines that are connected and automated backups. Consider using the managed ES from Amazon if you plan to host critical data. It does not sound like you have a very robust setup here.
I am using the following chef command to create a folder on /mnt
directory '/mnt/node/deploy' do
owner 'ubuntu'
group 'ubuntu'
mode '0755'
recursive true
action :create
end
This is a part of a recipe which is invoked via packer to create an AWS AMI. ubuntu is the user that I use to deploy my code to a provisioned machine.
When I launch an EC2 instance using the AMI, this folder is not created on the machine. What could be the problem? I see no errors when the AMI is created.
Update -1
These are the logs. I tried using root.
`amazon-ebs: * directory[/mnt/node/deploy] action create`
`amazon-ebs: - create new directory /mnt/node/deploy`
`amazon-ebs: - change mode from '' to '0755'`
`amazon-ebs: - change owner from '' to 'root'`
`amazon-ebs: - change group from '' to 'root'`
I see that EC2 is mounting ephemeral storage on /mnt.
I want to create these folders on the ephemeral storage.
I unmounted /mnt, but did not see the folders there.
Packer runs Chef before creating the image. So, if I understand you correctly:
Chef creates the directory on an instance ephemeral storage.
Packer creates the AMI.
You start the AMI and the directory does not exist in the ephemeral storage.
AFAIK that's an expected behavior. The directory is created in a partition that is ephemeral and this kind of partitions are not expected to endure.
Summarizing, when you create an AWS AMI image, it does not include the ephemeral storage. Only the EBS volumes. Ephemeral partitions are always empty at startup. If you want to retain that directory, it must be in a EBS partition.
If you still want to use the /mnt directory, you can avoid mounting the ephemeral storage with the ami_block_device_mappings option:
"ami_block_device_mappings": [
{
"device_name": "/dev/sdb",
"no_device": true
}
],
And the same for the launch_block_device_mappings
Another solution could be to run your Chef cookbook again in the newly created instance.
amazon-ebs is the name of the packer builder:
amazon-ebs - Create EBS-backed AMIs by launching a source AMI and
re-packaging it into a new AMI after provisioning. If in doubt, use
this builder, which is the easiest to get started with.
It runs the whole machine as EBS-backed so it can convert the EBS volume into an AMI later.
This is not related to Chef.
I am running a t2.micro ec2 instance on us-west-2a and instance's state is all green.
When I access my website it stops loading once in a while. Even if I reboot it, the website still doesn't load. When I stop an instance and then relaunch it, it shows 1/2 status checks failed.
ALARM TYPE: awsec2-i-20aaa52c-High-Network-Out
I also faced same type of issue.
EC2 instances were failing Instance Status Checks after a stop/start. I was able to take a look on my side at the System logs available to support and I could confirm that the system was having a kernel panic and was unable to boot from the root volume.
So I launched new EC2 temporary instance so we can attach the EBS root volumes of each EC2 instance . Here we modified the grub configuration file so it can load from a previous kernel.
The following commands:
1. Mount the EBS volume as a secondary volume into mnt folder: $ sudo mount /dev/xvdf1 /mnt
2. Backup the grub.cfg file: sudo cp /mnt/boot/grub2/grub.cfg grub.cfg_backup
3. Edit the grub.cfg file: sudo vim /mnt/boot/grub2/grub.cfg
4. Here we commented # all the lines for the first entry loading the new kernel.
Then you attached the original EBS volumes back to the original EC2 instances and these EC2 instances were able to successfully boot.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstances.html#FilesystemKernel
I ran into a problem the other day while cloning a github repo, and all of a sudden my EC2 Instance (EC2 A) became completely unusable. My question is: how can I re-attach an EBS Volume from an EC2 Instance that I terminated to a new EC2 Instance that I just created?
Step-by-Step of the problem:
0) broke my first EC2 Instance (EC2 A).
1) created a snapshot of the EBS Volume (EBS Volume A) attached to EC2 A.
2) stopped EC2 A.
3) detached EBS Volume A.
4) terminated EC2 A.
Then...
5) created a brand new EC2 Instance (EC2 B) with a new EBS Volume automatically created (EBS Volume B), which is currently attached to EC2 B.
6) set it all up (apache, mysql, php, other plugins, etc...)
7) Now I want to access my data from EBS Volume A. I do not care about anything in EBS Volume B. Please Advise...
Thank you so much for your time!
Yes, you can attach an existing EBS volume to an EC2 instance. There are a number of ways to do this depending on your tools of preference. I prefer the command line tools, so I tend to do something like:
ec2-attach-volume --instance-id i-XXXXXXXX /dev/sdh --device vol-VVVVVVVV
You could also do this in the AWS console:
https://console.aws.amazon.com/ec2/home?#s=Volumes
Right click on the volume, then select [Attach Volume]. Select the instance and enter the device (e.g., /dev/sdh).
After you have attached the volume to the instance, you will want to ssh to the instance and mount the volume with a command like:
sudo mkdir -m000 /vol2
sudo mount /dev/xvdh /vol2
You can then access your old data and configuration under /vol2
Note: The EBS volume and the EC2 instance must be in the same region and in the same availability zone to make the attachment.