I have created a new DataPipeline to stop some instances e.g tagged as auto-stop/auto-start .
My command is sth like this:
aws ec2 describe-instances --region us-west-2 --filter "Name=tag:auto-stop,Values=yes" "Name=instance-state-name,Values=running" --query 'Reservations[*].Instances[*].[InstanceId]' --output text |xargs aws ec2 stop-instances --region us-west-2 --instance-ids
i.e Stop all instances which are running and tagged as auto-stop:yes in given region
Now as soon as scheduler starts an activity, I see a new t1.micro instance is started with a public ip assigned. My question is,is this behavior normal for Data Pipelines ?
Please provide me any link to documentation and how much I ll be charged for this activity.
If the creation of the associated instance is a normal thing,what is the life-cycle of that instance ?
Yes, this behavior is normal. The t1.micro instance is the one executing the command.
An Ec2Resource object is set up in your Data Pipeline. It can be customized, see documentation: http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-ec2resource.html
By default it's a t1.micro instance that terminates after 50 minutes. You are charged for this instance (EC2 pricing: https://aws.amazon.com/ec2/pricing/).
As mentioned in one of EC2 knowledge base article https://aws.amazon.com/premiumsupport/knowledge-center/stop-start-ec2-instances/:
An Amazon EC2 t1.micro instance is started as the host environment for
execution of a data pipeline. EC2 instances started for this purpose
run for a default timeout period of 50 minutes. All resources used to
host execution of a data pipeline are accrued to your account.
Executing pipelines to stop and restart an EC2 instance for 100
minutes or less will use the same amount of resources as would be used
by simply letting an EC2 t1.micro instance continue to run. You should
stop and restart one or more EC2 t1.micro or larger instances for more
than 100 minutes (50 minutes to stop and 50 minutes to start) to
ensure that the method described in this article does not consume more
resources than are conserved.
The life cycle of the Ec2 Resource is tied to the activities that need to run on that resource. Resource is terminated when activities are complete or the 'terminateAfter' timeout is reached.
I believe, as it stands today, AWS Ec2 does not charge for t1.micro instances.
Related
I have multiple Non-Production servers where each have 900GB EBS Volume attached. And currently there is a practice of taking snapshot of these instances every day. I want to detach this volume when ASG scales down and reattach it to them when it scales up thus reducing the size of snapshot. The 900GB volume is the Non-Prod DB Volume so I plan to schedule a weekly snapshot of it separately.
NOTE: Each Non-Production ASG will have only one EC2 spot instance max and the scaling up & down is done via scaling policies.
I am trying to setup Event Bridge notification, triggering a Lambda function whenever ASG performs an action. I want to know if there are any other AWS provided service for this or any better way to do this.
by default an EBS volume gets detached than the ec2 instance is removed. And for mounting it you can run a script in the user data.
/var/awslogs/bin/aws ec2 attach-volume --volume-id `your-ebs-volume-id` --instance-id `your-instance-id` --device /dev/xvdk
How can I remotely turn on the ec2 instance from my computer (ubuntu) and then turn it off (I want to use ec2 only while working on my pc and then hibernate to not pay unnecessarily. Does hibernation mean no cost?
Check out awscli.
You can easily do this and much, much more with it.
See here for download and more details:
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html
Once it's installed, you can do somethind like:
aws ec2 start-instances --instance-ids i-xxxxxxxx
and
aws ec2 stop-instances --instance-ids i-xxxxxxxx
to start and stop instances.
If you stop an instance, you'll stop paying for the ec2 time. But, you'll continue be charged for the EBS volume(s) that you have allocated until/unless you terminate the instance and make sure that all corresponding volumes are deleted.
I had a container service instance i believe running in ECS/EC2. I deleted the EC2 instance and any ECS cluster around 12:30pm yesterday. Now over 24 hours i noticed that the cost of vCPU usage is still going up even though i deleted the EC2 Instance and ECS Cluster under US East 2 (Ohio) region as that what it states on the bill.
But i still see an increase under the following resource AWS Fargate - vCPU - US East 2 (Ohio) when i navigate to the bills area why?
I double checked everywhere under the various regions and i cannot see anything still running or being consumed. I can only see "inactive" Task Definitions.
So how do i prevent from incurring further costs?
Has anyone come across this before?
Whether you are using EC2 or Fargate launch type, you still need a cluster to host your tasks. So if there is no cluster in your account, you should incur no charges. There might be some problem with console UI that is not showing your cluster correctly, it happened to me that I have deleted cluster from console and there was some error which prevented cluster from being completely deleted and I ended up in a weird state where console indicated that the cluster was deleted while it wasn't.
This stuff actually happened to me more than once and not only with ECS so I would always recommend to use CLI to double check your actions.
Run this command
aws ecs list-clusters --region us-east-2
If the output is not empty then you will need to delete that cluster via CLI
aws ecs delete-cluster --cluster <value> --region us-east-2
If the output is empty then there is no cluster running in your Ohio region and those charges were simply delayed a bit.
I am running amazon AWS ECS container which creates one single instance of EC2. I made sure that it is 1 instance when I created ECS.
My issue is that I have another instance running in EC2 and amazon has sent me an email that I am using double of the free quota and will be charged.
But I am not sure why this second EC2 instance is coming from.
I have terminated it many times but it is recreated. When I terminate it, this is the prompt i receive which advises me that it is created from EBS but there is no app in EBS
On an EBS-backed instance, the default action is for the root EBS volume to be deleted when the instance is terminated.
Storage on any local drives will be lost.
This name of the instance is.
ECS Instance - amazon-ecs-cli-setup-ecs-cricketscorer
Please help.
check if you have any Auto Scaling Groups that you do not recognize. It is most probably being created because of it. If not, change your account password and deactivate/delete your existing AccessKeys.
I'm running custom transactional tasks on my EC2 instances. The decision to shutdown or not an instance is taken under many conditions by special process running on this instance. The termination should be done by instance itself, because Autoscaling Group does not know when data processing is finish. Do the following steps are consistent with the philosophy of AWS?
Creates AMI with option: "Shutdown behaviour: Terminate".
Autoscaling group creates a new instance with option "Protect From Scale In".
Custom process on EC2 calls command:
$ sudo shutdown -P now
to terminate an instance in proper time.
Is that correct? Or maybe AWS has some tools to do that, eg. emit special signal to terminate an instance?
Thank you
That process has one issue I believe:
In step 1, the "Shutdown behaviour: Terminate" option is not an AMI level setting. It is a launch time setting, for instances launched outside of an autoscaling group.
Within an Autoscaling Group, there is no option to configure a Launch Configuration with the equivalent of "Shutdown behaviour: Terminate". Presumably, ASG instances must be terminated during scale in events.
The simple approach would be to have the instance call the AWS CLI terminate-instances command:
aws ec2 terminate-instances --instance-ids i-xxxxxxxx
You would need to acquire the instance id from the AWS Metadata in order to run the terminate-instances command.
We have a similar pattern, but getting a working solution seems kludgy. We do it a little differently:
Start with Protect From Scale In
When processing is complete, have the instance turn of it's "Protect From Scale In" flag
Have instance trigger the Scale-In policy, by reducing count by 1.
ASG then terminates this instance and doesn't restart a new one because scale in was called
One of the things I don't like about this is that you end up with creating the IAM Role for the EC2 instances, then create the ASG, then go back and update the IAM Role, to give it permissions to SetInstanceProtection and ExecutePolicy for it's own group. You need to do this because we couldn't figure out how to create a policy that referenced the autoscalegroup that the caller is in.
Did you ever resolve this with a different solution?