I have a EC2 instance which my KOPS cluster is running. I observed that when the instance is stopped and started another day, the cluster starts itself automatically.
Does it mean that when EC2 instance is stopped, it goes into a state like 'Hibernate' Or KOPS has its own mechanism - like disaster recovery - and resilience when the host machine is down and up ?
Instances are just a normal part of the AWS infrastructure. When EBS is used for storage, data is not lost when instance is stopped, hence when you restart your instances they are brought up with the same state stored on EBS drives. This is not an explicit "hibernation" mechanism, nor is it a particularly specific feature of kops, it's just a regular data retention of data stored on AWS EBS.
Related
I need to persist the heap dump when the java process gets OOM and the pod is restarted.
I have following added in the jvm args
-XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/dumps
...and emptydir is mounted on the same path.
But the issue is if the pod gets restarted and if it gets scheduled on a different node, then we are losing the heap dump. How do I persist the heap dump even if the pod is scheduled to a different node?
We are using AWS EKS and we are having more than 1 replica for the pod.
Could anyone help with this, please?
You will have to persists the heap dumps on a shared network location between the pods. In order to achieve this, you will need to provide persistent volume claims and in EKS, this could be achieved using an Elastic File System mounted on different availability zones. You can start learning about it by reading this guide about EFS-based PVCs.
As writing to EFS is too slow in your case, there is another option for AWS EKS - awsElasticBlockStore.
The contents of an EBS volume are persisted and the volume is unmounted when a pod is removed. This means that an EBS volume can be pre-populated with data, and that data can be shared between pods.
Note: You must create an EBS volume by using aws ec2 create-volume or the AWS API before you can use it.
There are some restrictions when using an awsElasticBlockStore volume:
the nodes on which pods are running must be AWS EC2 instances
those instances need to be in the same region and availability zone as the EBS volume
EBS only supports a single EC2 instance mounting a volume
Check the official k8s documentation page on this topic, please.
And How to use persistent storage in EKS.
I am running few experiments on AWS Redshift in the free tier with a single node dc2.large cluster. I keep a snapshot as I do not need it to run the cluster at night and again restoring from that snapshot the next morning.
I can see my EC2 bill is slowly rising up with the utility but not a single documentation or blog I could find to understand if a running Redshift cluster uses EC2 instance or taking and keeping a snapshot of a Redshift cluster does the same.
Can anyone help me understanding the behavior?
Your usage of Amazon Redshift (including running a cluster and creating/keeping snapshots) should not create a charge for Amazon EC2 resources.
It might generate traffic within the VPC depending upon how you are connecting to it (eg cross-AZ traffic).
I am running amazon AWS ECS container which creates one single instance of EC2. I made sure that it is 1 instance when I created ECS.
My issue is that I have another instance running in EC2 and amazon has sent me an email that I am using double of the free quota and will be charged.
But I am not sure why this second EC2 instance is coming from.
I have terminated it many times but it is recreated. When I terminate it, this is the prompt i receive which advises me that it is created from EBS but there is no app in EBS
On an EBS-backed instance, the default action is for the root EBS volume to be deleted when the instance is terminated.
Storage on any local drives will be lost.
This name of the instance is.
ECS Instance - amazon-ecs-cli-setup-ecs-cricketscorer
Please help.
check if you have any Auto Scaling Groups that you do not recognize. It is most probably being created because of it. If not, change your account password and deactivate/delete your existing AccessKeys.
I have received an email about an EC2 instance that is to be retired on the 1/12/2015. I can provide more details on the case if need be. I have gone through this process before (approx a year ago), and it all went smoothly. The only difference this time is in the instructions. Last time, according to the instructions, all I had to do, was just restart my instance. Now, I have to create an AMI from my instance, and launch a new instance with that AMI. Please, note that in both cases my instances were/are EBS backed.
Do I really need to create an AMI and launch a new instance or just restarting the instance would do? Please, keep in mind, that I need to keep my EBS data intact.
Cheers,
Iraklis
From the AWS EC2 documentation:
Managing Instances Scheduled for Retirement
You can perform one of the actions listed below in order to preserve
the data on your retiring instance. It's important that you take this
action before the instance retirement date, to prevent unforeseen
downtime and data loss.
Warning If your instance store-backed instance passes its retirement
date, it's terminated and you cannot recover the instance or any data
that was stored on it. Regardless of the root device of your instance,
the data on instance store volumes is lost when the instance is
retired, even if they are attached to an EBS-backed instance.
An instance is scheduled to be retired when AWS detects irreparable
failure of the underlying hardware hosting the instance. When an
instance reaches its scheduled retirement date, it is stopped or
terminated by AWS. If your instance root device is an Amazon EBS
volume, the instance is stopped, and you can start it again at any
time. Starting the stopped instance migrates it to new hardware. If
your instance root device is an instance store volume, the instance is
terminated, and cannot be used again.
Instance Root Device Type: EBS
Action: Wait for the scheduled retirement date - when the instance is
stopped - or stop the instance yourself before the retirement date.
You can start the instance again at any time. For more information
about stopping and starting your instance, and what to expect when
your instance is stopped, such as the effect on public, private and
Elastic IP addresses associated with your instance, see Stop and
Start Your
Instance.
Instance Root Device Type: EBS
Action: Create an EBS-backed AMI from your instance, and launch a
replacement instance. For more information, see Creating an Amazon
EBS-Backed Linux AMI.
Instance Root Device Type: Instance store
Action: Create an instance store-backed AMI from your instance using
the AMI tools, and launch a replacement instance. For more
information, see Creating an Instance Store-Backed Linux
AMI.
Instance Root Device Type: Instance store
Action: Convert your instance to an EBS-backed instance by
transferring your data to an EBS volume, taking a snapshot of the
volume, and then creating an AMI from the snapshot. You can launch a
replacement instance from your new AMI. For more information, see
Converting your Instance Store-Backed AMI to an Amazon EBS-Backed
AMI.
I want to back up the root device (mounted at /) of my t2.micro instance running Ubuntu. I think the instance is EBS-backed as it is a t2 instance. So I was going to take snapshots of my root device to back it up.
However, it is recommended that I detach the root device before I back it up. There are two problems with this:
I have to use umount to unmount it first, which may cause my instance to crash. What is a safe way to handle this?
I want to run these backups as a cron job on the instance itself, but if my instance's root device is unmounted, will the cron job even run?
A more general question is: what is the best way to do this?
A possible solution might be: use AWS Lambda and execute a Lambda function based on a schedule executing the following commands by the use of the AWS SDK:
Stop EC2 instance
Create EBS snapshot
Start EC2 instance
First, I would confirm that your root device is in fact EBS backed.
Here are the basic steps to confirm:
To determine the root device type of an instance using the console
Open the Amazon EC2 console.
In the navigation pane, click Instances, and select the instance.
Check the value of Root device type in the Description tab as follows:
If the value is ebs, this is an Amazon EBS-backed instance.
If the value is instance store, this is an instance store-backed instance.
(Source: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDeviceStorage.html#display-instance-root-device-type)
AWS states as a best practice is to use snapshots or a backup tool.
Regularly back up your instance using Amazon EBS snapshots or a backup
tool.
(Source: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-best-practices.html)
AWS states in the documentation that root EBS volumes should be shutdown before taking a snapshot.
To create a snapshot for Amazon EBS volumes that serve as root
devices, you should stop the instance before taking the snapshot.
(Source: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-creating-snapshot.html)
So depending on your RPO (Recovery Point Objective), as a general rule it is a good practice to separate your data from your root volume. Store data that you need to keep on a separate EBS volume and take snapshots on the second EBS volume. This way you never have to worry about the instance itself - if it bonks out just launch a new instance and attach your snapshot.
If you have a special case that prevents you from using EBS snapshots, try using a role for your instance(s) that have permissions to read/write data to S3 buckets using your cron job.