Currently I am working to build a data structure ( Python Dictionary stored in mongo) which stores information of vpc's, subnet's and ec2 Instances. I want the data structure to be updated on real time basis whenever some activity which I want to tap into has took place
eg:If a ec2 Instance is added or Deleted or Some Properties of the ec2 Instance is changed or if a new EBS Volume is attached to ec2 Instance etc
you can use CloudWatch a service provided by Amazon to watch all your infrastructure,
see Docs
for more advanced events you can use CloudTrail
here is a use case
Related
When launching an EC2 instance in the AWS Console I see the expected 'RunInstances' calls in CloudTrail, but I do not see 'CreateNetworkInterface' calls that EC2 makes to create the interfaces it attaches.
When I create a new network interface manually I do see the 'CreateNetworkInterface'. Other services like ELB, also record 'CreateNetworkInterface' entries. Lambda, Cognito, etc. Only EC2 instances do not. I can see the network interface it creates.
Does anyone know why these are not recorded and where I can get find this information via CloudTrail? This seems like a security gap.
I am stuck on one point I have created one EC2 Linux based instance in Aws.
Now I want to send the EC2 metrics data to the managed Elasticsearch domain for monitoring purposes in Kiban, I go through the cloud watch console and check the metric is present of instance but didn't get how to connect with the Elasticsearch domain that I have created.
Can anyone please help me with this situation?
There is no build in mechanism for extraction/streaming of metrics data points in real time. You have to develop a custom solution for that. For example, by having a lambda function which is invoked every minute and which reads data points using get_metric_data. The the lambda would inject the points into your ES.
To invoke a lambda function periodically, e.g. every 1 minute you would have to setup CloudWatch Event rule with schedule Expressions. Lambda function would also need to have permissions granted to interact with CloudWatch metrics.
Welcome to SO :)
An alternative to the solution suggested by Marcin is to install metricbeat on the EC2 Instance and configure the metricbeat config file to send metrics to your Managed AWS ES Domain.
This is pretty simple and you should be able to do this fairly quickly.
I am able to monitor a Windows instance's memory usage using custom metrics in CloudWatch.
I have followed the following blog to achieve that :
http://blog.krishnachaitanya.ch/2016/03/monitor-ec2-memory-usage-using-aws.html
Using that, I am able to monitor only one instance. I am now doing the process in every instance launched.
Can I do it at once for all instances instead of changing .json file and enabling cloud watch integration in every instance?
If the instances are already launched, you have to do it for each instance. Else you can take an AMI of the first instance, then launch other instances from that AMI and you do not have to do it for each instance.
If you have to do it manually, consider something like Ansible to do it for you. There is a bit of learning but not difficult.
BTW, adding custom metrics is straightforward for Linux instances. Monitoring Memory and Disk Metrics for Amazon EC2 Linux Instances
For Windows instance: Sending Performance Counters to CloudWatch and Logs to CloudWatch Logs Using Amazon EC2 Simple Systems Manager
If your instances have the appropriate instance profile and are running the SSM agent (which they probably are if you launched from an Amazon provided AMI), you can use SSM run command to run arbitrary powershell against an instance or a set of instances (using tags). There is even a Amazon managed SSM document called AWS-ConfigureCloudWatch that is built specifically for this use case.
See http://docs.aws.amazon.com/systems-manager/latest/userguide/run-command.html
I have created an ec2 instance using data pipeline. TerminateAfter field value was set to 2 hours. How can i verify the created ec2 instance details(like ip, java versions...) using aws ec2 console?
You will see the instance attributes like ip, region etc., in the dashboard. But if you want system specific information like java version, you have to ssh into the created instance. Make sure you are looking for the instance in the correct region.
I created a Amazon AWS EC2 instance under my account and made an website/ftp on it, now a new partner wants to move the instance under his company account so his company can pay the bills.
We can't change the instance IP because banks in the region are communicating with the server.
How can I move the instance to a different account without having to change anything on the configuration?
The short answer is: No, you cannot move an running instance from one account to another unless and ofcourse AWS Technical support has some magic available behind the curtains.
You can However, Create an AMI from this instance and share this AMI with other users/account. refer: http://aws.amazon.com/articles/530
To share or migrate EC2 instances from a source account to a target
account follow these steps:
Create a custom Amazon Machine Image (AMI)
from the instance you want to share or migrate. Be sure to include all
required EBS data volumes in the AMI.
Note: Data stored on instance store volumes isn't preserved in AMIs, and won't be on the instance store volumes of the instances
that you launch from the AMI.
Share the AMI with the target account
using either the EC2 console or the AWS Command Line Interface (CLI).
From the target account, find the AMI
using the EC2 console or the AWS CLI.
Launch a new instance from the shared AMI
on the target account.
Note: The private IP address of VPC instances will be different in the new account, unless you specifically set them during
launch.
Related information
Changing the Encryption State of Your Data
AWS CLI Command Reference (EC2)
Source: Transfer Amazon EC2 Instance
This is not possible.
AWS Support does not have access to copy Amazon EC2 resources or
manipulate any configuration options in AWS accounts. You can't
separate an AWS account from an Amazon.com account or transfer
resources between AWS accounts. It is possible to manually migrate
Amazon EC2 resources from one account to another by completing the
steps described here.
Source : https://aws.amazon.com/premiumsupport/knowledge-center/account-transfer-ec2-instance/
I'm working with several hundreds on EC2 instances in several AWS regions and accounts. You can move an EC2 instance to another AWS account, however, you can't move the Elastic IP and it will take up 16 steps with AWS CLI, if you want to migrate Tags and clone the Security Groups. I wrote a detailed post with the whole process at https://medium.com/#gmusumeci/how-to-move-an-ec2-instance-to-another-aws-account-e5a8f04cef21.
there are more than 10 steps involved in doing the cloud move. I would suggest you use Infrastructure as a Configuration (terraform and CloudFormation) or Infrastructure as a real code (pulumi and CDK)
however if you want to give a go at a nice tool I found called KopiCloud. Please feel welcome to try it and leave your comments below. Is good if you need to move instances on a quick lift and shift scenario.
You can re-think the design of having the banks in the region communicating to your servers via IP.
If the banks communicate using DNS names, you have much more flexibility to move your servers around.
You can also achieve improvements in high availability and resiliency by moving to DNS connections.
So a plan might be
Setup a DNS record for your existing server
Get the banks who connect to your server to connect via the DNS name
Setup your new server in the other account (other answers describe this)
Cut the banks over to your new server in the new account simply by updating the DNS record
I haven't tried load balancing across accounts, but that may be another option, which would give you HA as a bonus. By registering your current instance, and new instance in another account as targets with a load balancer and getting your clients to connect to the load balancer, you could cut over to the other account. The only part I haven't tried is registering targets in different accounts, but looks like this should be possible with an AWS Network Load Balancer