unable to create folder on /mnt via chef recipe - amazon-web-services

I am using the following chef command to create a folder on /mnt
directory '/mnt/node/deploy' do
owner 'ubuntu'
group 'ubuntu'
mode '0755'
recursive true
action :create
end
This is a part of a recipe which is invoked via packer to create an AWS AMI. ubuntu is the user that I use to deploy my code to a provisioned machine.
When I launch an EC2 instance using the AMI, this folder is not created on the machine. What could be the problem? I see no errors when the AMI is created.
Update -1
These are the logs. I tried using root.
`amazon-ebs: * directory[/mnt/node/deploy] action create`
`amazon-ebs: - create new directory /mnt/node/deploy`
`amazon-ebs: - change mode from '' to '0755'`
`amazon-ebs: - change owner from '' to 'root'`
`amazon-ebs: - change group from '' to 'root'`
I see that EC2 is mounting ephemeral storage on /mnt.
I want to create these folders on the ephemeral storage.
I unmounted /mnt, but did not see the folders there.

Packer runs Chef before creating the image. So, if I understand you correctly:
Chef creates the directory on an instance ephemeral storage.
Packer creates the AMI.
You start the AMI and the directory does not exist in the ephemeral storage.
AFAIK that's an expected behavior. The directory is created in a partition that is ephemeral and this kind of partitions are not expected to endure.
Summarizing, when you create an AWS AMI image, it does not include the ephemeral storage. Only the EBS volumes. Ephemeral partitions are always empty at startup. If you want to retain that directory, it must be in a EBS partition.
If you still want to use the /mnt directory, you can avoid mounting the ephemeral storage with the ami_block_device_mappings option:
"ami_block_device_mappings": [
{
"device_name": "/dev/sdb",
"no_device": true
}
],
And the same for the launch_block_device_mappings
Another solution could be to run your Chef cookbook again in the newly created instance.

amazon-ebs is the name of the packer builder:
amazon-ebs - Create EBS-backed AMIs by launching a source AMI and
re-packaging it into a new AMI after provisioning. If in doubt, use
this builder, which is the easiest to get started with.
It runs the whole machine as EBS-backed so it can convert the EBS volume into an AMI later.
This is not related to Chef.

Related

How to start/use a Jenkins in data volume attached to AWS EC2

Create an EC2 Instance_A with default root volume VOL_A.
Install Jenkins by yum install, start it, make it auto-started and create many Jenkins jobs.
Because the AMI to start Instance_A is not secure, I need to create a new EC2 Instance_B with default root volume VOL_B.
Attach VOL_A as data volume to Instance_B.
The question is how do I start/use the Jenkins in the data volume?
Following steps may solve your issue.
Create instance_A with root volume vol_A, For reference can use documentation https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/launching-instance.html
Connect to instance_A and install Jenkins as service and create required jobs. For reference documentation is at: https://www.jenkins.io/doc/tutorials/tutorial-for-installing-jenkins-on-AWS/
Launch instance_B with vol_B as root volume by following the steps mentioned on step 1 and install Jenkins by referring step 2
Now attach vol_A to instance_B as /jenkinsmount by using snapshot method or detach from instance_A by stopping instance and attach it to instance_B as add on volume.For reference documentation is at: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-attaching-volume.html
After attaching volume stop Jenkins and change home directory in /etc/default/jenkins as shown below
JENKINS_HOME=/jenkinsmount/var/lib/jenkins
Start Jenkins in instance_B now. The jobs created in instance_A should be shown in instance_B along with job_history.

ElasticSearch with Docker: how to persist data with AWS

I'm trying to run ElasticSearch on Docker (actually on AWS ECS). If I don't configure the volume it's working correctly, but every time I restart the container I lose all the data.
I can't figure out how to configure the volume.
What I tried:
in the task definition I configured volume "Name=esdata1" and "source path=/usr/share/elasticsearch/data"
inside the container definition in the "storage and logging" section I configured the mount point "source volume= esdata1" and "container path=/usr/share/elasticsearch/data"
Now when I launch the container it fail with error "access denied" when elasticsearch try to write in "/usr/share/elasticsearch/data". So in the section Security I configured "user=ec2-user" but then the container will not even launch (stay in "status=created"). What should I do? I guess the issue is due to the fact that the user of the container must be the same of the one on the host. The user on the host is "ec2-user", I don't know how to proceed.
Edit:
I'm now able to persist data with this configuration:
docker inspect:
"Mounts": [
{
"Name": "elasticsearch_data",
"Source": "/var/lib/docker/volumes/elasticsearch_data/_data",
"Destination": "/usr/share/elasticsearch/data",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": "rprivate"
}
]
Now data persist if I stop the container or I reboot the host. My only last concern is that this folder "/var/lib/docker/volumes/elasticsearch_data/_data" is located on the OS volume and not on the bigger docker volume. From Aws doc:
Amazon ECS-optimized AMIs from version 2015.09.d and later launch with
an 8 GiB volume for the operating system that is attached at /dev/xvda
and mounted as the root of the file system. There is an additional 22
GiB volume that is attached at /dev/xvdcz that Docker uses for image
and metadata storage. The volume is configured as a Logical Volume
Management (LVM) device and it is accessed directly by Docker via the
devicemapper back end.
How can I persist data on /dev/xvdcz?
Thanks very much
Your sourcepath is the path on the host instance where the data is written. In your case elasticsearch_data. You need to point sourcepath to a folder that exists and that is on the disk you want on the EC2 instance.
So attach an EBS disk to the instance. Mount the disk in a place like /data/es and set your source path to that folder.
But remember that to properly run ES you would probably need a cluster of machines that are connected and automated backups. Consider using the managed ES from Amazon if you plan to host critical data. It does not sound like you have a very robust setup here.

why does my website stops loading on aws ec2 instance randomly once in a while?

I am running a t2.micro ec2 instance on us-west-2a and instance's state is all green.
When I access my website it stops loading once in a while. Even if I reboot it, the website still doesn't load. When I stop an instance and then relaunch it, it shows 1/2 status checks failed.
ALARM TYPE: awsec2-i-20aaa52c-High-Network-Out
I also faced same type of issue.
EC2 instances were failing Instance Status Checks after a stop/start. I was able to take a look on my side at the System logs available to support and I could confirm that the system was having a kernel panic and was unable to boot from the root volume.
So I launched new EC2 temporary instance so we can attach the EBS root volumes of each EC2 instance . Here we modified the grub configuration file so it can load from a previous kernel.
The following commands:
1. Mount the EBS volume as a secondary volume into mnt folder: $ sudo mount /dev/xvdf1 /mnt
2. Backup the grub.cfg file: sudo cp /mnt/boot/grub2/grub.cfg grub.cfg_backup
3. Edit the grub.cfg file: sudo vim /mnt/boot/grub2/grub.cfg
4. Here we commented # all the lines for the first entry loading the new kernel.
Then you attached the original EBS volumes back to the original EC2 instances and these EC2 instances were able to successfully boot.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstances.html#FilesystemKernel

Mount specific EBS volume to Docker under AWS beanstalk

AWS Beanstalk can run applications from Docker containers.
As mentioned in the docs (http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_image.html) it's possible to write directory mappings to the EC2 volume in the Dockerrun.aws.json:
"Volumes": [
{
"HostDirectory": "/var/app/mydb",
"ContainerDirectory": "/etc/mysql"
}
but, is it possible to mount specific EBS volume?
F.e. I need to run db in the Docker container and deploy it with Beanstalk. It's clear that I need to have persistence of the data, backup/restore for db, etc..
You can mount EBS volumes on any Beanstalk environment. This volume will be available on the EC2 instances.
You can do this using ebextensions option settings. Create a file in your app source .ebextensions/01-ebs.config with the following contents:
option_settings:
- namespace: aws:autoscaling:launchconfiguration
option_name: BlockDeviceMappings
value: /dev/sdj=:100,/dev/sdh=snap-51eef269,/dev/sdb=ephemeral0
The format of the mapping is device name=volume where the device mappings are specified as a single string with mappings separated by a comma. This example attaches to all instances in the autoscaling group an empty 100-GB Amazon EBS volume, an Amazon EBS volume with the snapshot ID snap-51eef269, and an instance store volume.
Read more details about this option setting here.
Read more about ebextensions here.
Once you have mounted the EBS volume for your beanstalk environment instances, you can use the volume mapping as above to map directories per your need.
I guess the leg100/docker-ebs-attach Docker container does what you want, i.e. make a particular existing EBS volume available. You can either copy the .py file and relevant Dockerfile statements or create a multi-container EB setup and mount the volume from this container.
BTW I have tried to mount a new EBS volume as proposed by Rohit (+ commands to format and mount it) and it works but Docker does not see the mount until the docker daemon is restarted.

Having issues adding ephemeral storage to an AWS EBS instance running Ubuntu

I am having problems adding ephemeral storage into my existing EBS backed instance. I have a small instance running on 8GB EBS root-device, and I would like to add ephemeral storage into this instance and run it as a medium instance.
The procedure I have tried which did not work for me:
1) Took a snapshot from the instance EBS volume.
2) Registered new AMI based on the snapshot using ec2-api-tools:
ec2-register -a x86_64 -n "My AMI with ephemeral storage" --kernel <AKI-ID> --root-device-name "/dev/sda1" -b "/dev/sda1=<SNAP-ID>:8:true:standard" -b "/dev/sdc=ephemeral1"
3) Launched new medium instance with the new AMI I just created:
ec2-run-instances <AMI-ID> -t m1.medium --kernel <AKI-ID> -k <MY_KEY_NAME> -g default -b "/dev/sdc=ephemeral1"
4) SSH:ed into my new instance after it started up and the ephemeral storage is nowhere to be found (checked with fdisk -l for example). The root device is fine and correct, but eve nif trying out ephemeral0 instead of 1 did not change anything.
Apparently there is nothing in the API that tells you when you exceed your instance store mappings. A medium instance can only have 1 ephemeral drive. In fact /dev/sdc may only be able to mapped in large instances and up:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/InstanceStorage.html#StorageOnInstanceTypes