I just want to know if there are other ways to approach this problem:
I have an AWS multi account setup. The EC2's are going to be monitored over all the accounts and when alerts are triggered via sns there is a mail going to be sent. For all EC2's with Windows Server 2016 and later, Amazon Linux and Ubuntu 16.04 and 18.04 the SSM agents come pre-installed. That way I can push the CloudWatch agent via System Manager Run Command to the EC2's per AWS account.
I was wondering is there a more simple way that i can force that CloudWatch Agent is installed with every new EC2 deployed in an AWS account, without installing the agent manually on the instance or via Run Command?
I was thinking working with tags, something like: "IsMonitored" and as value true or false. for example everyday at 17hr a Lambda function will go over all the instances in that account and search for IsMonitored = false, Get that instance ID and with a (boto3?) cript push the agent on that instance. This seemed to complicated so i wanted to check if there is maybe other simple solutions that would do the same.
Thanks in advance,
Iman
To install a cloudwatch agent in each instance particular region you can implement by shell script.
The approach is:
Manual work is create some default configuration file in parameter store for both the type of instance a. for windows b. for linux based
In shell script
For particular region
Get the total number of ec2 instance id list
Check the platform which type of machine is using Windows or Linux based
If the platform is Windows then add Windows type configuration file from parameter store else add Linux configuration file
For getting platform name :
platform=$(aws ec2 describe-instances --instance-ids <instance id> --query 'Reservations[*].Instances[*].[Platform]' --output text)
For installing packages :
aws ssm send-command --instance-ids <instance id> --document-name "AWS-ConfigureAWSPackage" --parameters "name=AmazonCloudWatchAgent,action=Install,installationType=Uninstall and reinstall" --comment "Install CloudWatch Agent on EC2 Windows/Linux machine"
For start CWagent :
aws ssm send-command --instance-ids $one_instance --document-name "AmazonCloudWatch-ManageAgent" --parameters "mode=ec2,optionalRestart=yes,optionalConfigurationSource=ssm,action=configure,optionalConfigurationLocation=AmazonLinuxCloudWatchAgentConfig" --comment "Configure CloudWatch Agent on EC2 Linux machine"
For more reference you can use this link.
One simpler approach could be using prebaked AMI. First, spin up an EC2 with the normal AMI you use. Next, install the CloudWatch agent and create an image. From now on, you can spin up EC2's using the new AMI which has CloudWatch agent preinstalled.
If prebaked AMI doesn't work for you, I recommend using an infrastructure-as-code (IaC) tool like Ansible to automate the installation process.
Related
I have like 15 Lightsail instances created on my AWS account and now I wanted to know when these Lightsail instances were created.
The creation date of the Lightsail instances on which date and time these were created. But is not able to find this information from the AWS Lightsail console.
You can use the AWS Command Line Interface (AWS CLI) to retrieve the creation date of your Lightsail instances. The following AWS CLI command will list all of your Lightsail instances and the creation date for each instance:
aws lightsail get-instances --query "instances[*].{Name:name, CreationDate:createdAt}"
This command uses the get-instances command to retrieve information about all of your Lightsail instances, and the --query option to extract the name and creation date of each instance.
Alternatively, you can also retrieve the creation date of a specific Lightsail instance by using the get-instance command and specifying the instance name:
aws lightsail get-instance --instance-name <instance_name> --query "instance.createdAt"
Replace <instance_name> with the name of the Lightsail instance you want to retrieve information for.
Note: The AWS CLI must be installed and configured on your local machine in order to use these commands.
If you haven't already done so, just follow these steps:
1- Install the AWS CLI:
You can install the AWS CLI on your local machine by following the installation instructions for your operating system. You can find the installation instructions at the following URL: https://aws.amazon.com/cli/
2- Configure the AWS CLI:
After installing the AWS CLI, you need to configure it with your AWS credentials. You can do this by running the following command:
aws configure
This will prompt you for your AWS access key ID, secret access key, default region name, and default output format. You can find your AWS access keys in the AWS Management Console.
3- Verify the configuration:
To verify that your AWS CLI is configured correctly, you can run the following command:
aws lightsail get-instances
This command should list all of your Lightsail instances in your AWS account.
With the AWS CLI installed and configured, you can now use the AWS CLI commands to retrieve information about your Lightsail instances, including the creation date.
I hope my answer helps you
I really need to know about the stopped time of AWS EC2 instances. I have checked with AWS cloudtrail, but its not easy to find the exact stopped EC2 instance. Is possible to see exact time of stopped EC2 instances by aws-cli commands or any boto3 script?
You can get this info from StateTransitionReason in describe-instances AWS CLI when you search for stopped instances:
aws ec2 describe-instances --filter Name=instance-state-name,Values=stopped --query 'Reservations[].Instances[*].StateTransitionReason' --output text
Example output:
User initiated (2020-12-03 07:16:35 GMT)
AWS Config keeps track of the state of resources as they change over time.
From What Is AWS Config? - AWS Config:
AWS Config provides a detailed view of the configuration of AWS resources in your AWS account. This includes how the resources are related to one another and how they were configured in the past so that you can see how the configurations and relationships change over time.
Thus, you could look back through the configuration history of the Amazon EC2 instance and extract times for when the instance changed to a Stopped state.
Sometimes time is missing from StateTransitionReason, you can use CloudTrail and search for Resource Name = instance ID to find out StopInstance(s) API calls.
By default you can track back 90 days, or indefinitely if you create your own trail.
I have an EC2 instance and I need to download a file from its D drive through my program. Currently, it's a very annoying process because I can't access the instance directly from my local machine. The way what I am doing now is running a script on the instance and the instance uploads the file I need to S3 and my program access S3 to read the file.
Just wonder whether there is any simple way to access the drive on the instance instead of going through S3?
I have used AWS DataPipeline and its task runner to execute scripts on a remote instance. The taskrunner waits for a pipeline event published to its worker group.
http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-using-task-runner.html
I use it to execute shell script and commands on a schedule. The script to run should be uploaded to S3, and the Data pipeline template specifies the script's path. Works great for periodic tasks. You can do anything you want on the remote box via the script.
You cannot directly download the file from EC2, but via s3( or maybe using scp command) from your remote ec2.
But to simplify this annoying process you can use AWS Systems Manager.
AWS Systems Manager Run Command allows you to remotely and securely run set of commands on EC2 as well on-premise server. Below are high-level steps to achieve this.
Attach Instance IAM role:
The ec2 instance must have IAM role with policy AmazonSSMFullAccess. This role enables the instance to communicate with the Systems Manager API.
Install SSM Agent:
The EC2 instance must have SSM agent installed on it. The SSM Agent process the run command requests & configure the instance as per command.
Execute command :
Example usage via AWS CLI:
Execute the following command to retrieve the services running on the instance. Replace Instance-ID with ec2 instance id.
aws ssm send-command --document-name "AWS-RunShellScript" --comment "listing services" --instance-ids "Instance-ID" --parameters commands="service --status-all" --region us-west-2 --output text
More detailed information: https://www.justdocloud.com/2018/04/01/run-commands-remotely-ec2-instances/
Does anyone have an set of instructions for creating an AWS AMI from a chef 12, Linux based OpsWorks instance?
AWS publishes instructions on Creating a Custom Linux AMI from an AWS OpsWorks Instance. But it looks like they are out of date for Chef 12, Linux based OpsWorks stacks.
For example, they do not say you should remove the /opt/chef or /var/chef folders.
Here are the current instructions from AWS:
Create a Custom Linux AMI from an AWS OpsWorks Instance
If you want to use a customized AWS OpsWorks Linux instance to create an AMI, you should be aware that every Amazon EC2 instance created by OpsWorks includes a unique identity. If you create a custom AMI from such an instance, it will include that identity and all instances based on the AMI will have the same identity. To ensure that the instances based on your custom AMI have a unique identity, you must remove the identity from the customized instance before creating the AMI.
To create a custom AMI from an AWS OpsWorks instance
Create a Linux stack and add one or more layers to define the configuration of the customized instance. You can use built-in layers, customized as appropriate, as well as fully custom layers. For more information, see Customizing AWS OpsWorks.
Edit the layers and disable AutoHealing.
Add an instance with your preferred Linux distribution to the layer or layers and start it. We recommend using an Amazon EBS-backed instance. Open the instance's details page and record its Amazon EC2 ID for later.
When the instance is online, log in with SSH and run the following commands, in order:
sudo /etc/init.d/monit stop
sudo /etc/init.d/opsworks-agent stop
sudo rm -rf /etc/aws/opsworks/ /opt/aws/opsworks/ /var/log/aws/opsworks/ /var/lib/aws/opsworks/ /etc/monit.d/opsworks-agent.monitrc /etc/monit/conf.d/opsworks-agent.monitrc /var/lib/cloud/
If you are creating an AMI based on Amazon Linux 2014.09, run rpm -e opsworks-agent-ruby to ensure that the agent is running.
If you are creating an AMI based on Ubuntu, run dpkg -r opsworks-agent-ruby to ensure that the agent is running.
This step depends on the instance type:
For an Amazon EBS-backed instance, use the AWS OpsWorks console to stop the instance and create the AMI as described in Creating an Amazon EBS-Backed Linux AMI..
For an instance store-backed instance, create the AMI as described in Creating an Instance Store-Backed Linux AMI and then use the AWS OpsWorks console to stop the instance.
When you create the AMI, be sure to include the certificate files. For example, you can call the ec2-bundle-vol command with the -i argument set to -i $(find /etc /usr /opt -name '.pem' -o -name '.crt' -o -name '*.gpg' | tr '\n' ','). Do not remove the apt public keys when bundling. The default ec2-bundle-vol command handles this task.
Clean up your stack by returning to the AWS OpsWorks console and deleting the instance from the stack.
What's the best way to send logs from Auto scaling groups (of EC2) to Logentries.
I previously used the EC2 platform to create EC2 log monitoring for all of my EC2 instances created by an Autoscaling group. However according to Autoscaling rules, new instance will spin up if a current one is destroyed.
Now how do I create an automation for Logentries to create a new hosts and starting getting logs. I've read this https://logentries.com/doc/linux-agent-with-chef/#updating-le-agent I'm stuck at the override['le']['pull-server-side-config'] = false since I don't know anything about Chef (I just took the training from their site)
For an Autoscaling group, you need to get this baked into an AMI, or scripted to run on startup. You can get an EC2 instance to run commands on startup, after you've figured out which script to run.
The Logentries Linux Agent installation docs has setup instructions for an Amazon AMI (under Installation > Select your distro below > Amazon AMI).
Run the following commands one by one in your terminal:
You will need to provide your Logentries credentials to link the agent to your account.
sudo -s
tee /etc/yum.repos.d/logentries.repo <<EOF
[logentries]
name=Logentries repo
enabled=1
metadata_expire=1d
baseurl=http://rep.logentries.com/amazon\$releasever/\$basearch
gpgkey=http://rep.logentries.com/RPM-GPG-KEY-logentries
EOF
yum update
yum install logentries
le register
yum install logentries-daemon
I recommend trying that script once and seeing if it works properly for you, then you could include it in the user data for your Autoscaling launch configuration.