With the below configuration for AWS cloud watch:
awslogs.conf
[/var/log/messages]
datetime_format = %b %d %H:%M:%S
file = /var/log/messages
buffer_duration = 2500
log_group_name = /var/log/messages
log_stream_name = {cluster}{instance_id}
and below script used with --userdata option for aws ec2 command:
userdata.sh
# Above agentlogs.conf file is copied to /etc/awslogs/awslogs.conf in AWS EC2 instance
# Configure cloudwatch config file
cat > /etc/cloudwatch-logs.ini <<EOF
[/var/log/messages]
datetime_format = %b %d %H:%M:%S
file = /var/log/messages
buffer_duration = 2500
log_stream_name = {cluster}{instance_id}
initial_position = start_of_file
log_group_name = /var/log/messages
EOF
an EC2 is launched from this script(running outside EC2):
spin_up_ec2.sh
# Using AWS CLI, we spin up EC2 instance using userdata.sh,
# Using metadata service How to read values of {cluster} & {instance_id} syntax, shown above:
aws logs describe-log-streams --log-group-name /var/log/messages --log-stream-name-prefix <grab_cluster_name_value><grab_instance_id_value> --region us-east-1
spin_up_ec2.sh is sitting outside EC2, within same VPC, in different subnet. So, am not sure, how to avail EC2 metadata service?
EC2 is running in private subnet.
{cluster} value would be something like clust1
{instance_id} value would be something like i-1a52627268bc
1)
How can a shell script(spin_up_ec2.sh) client talk to EC2 metadata service, to retrieve values of {cluster} & {instance_id}?
2)
Does launching EC2 in public subnet, help? To talk to metadata service
The Amazon EC2 instance metadata is not available outside of an instance.
You could make API calls to AWS services to obtain similar information (eg retrieve the subnet in which an EC2 instance is located).
Related
I am attempting to create my own bastion host for a VPC, and created an auto-scaling group with min/max instances of 1. In my launch configurations, I specify the following for the ec2 user data:
#!
INSTANCE_ID=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/instance-id`
aws ec2 associate-address --instance-id $INSTANCE_ID --allocation-id eipalloc-my-eip-id --allow-reassociation
The goal of this user data is to immediately associate an Elastic IP address with my newly created EC2 instance - I've read from other StackOverflow posts that this must be explicitly done when using ASGs.
However, after the ASG instance spins up and finishes initializing, I still do not see any Elastic IP in my console output for the instance:
I've confirmed that the user data is indeed being used by the instance:
I tried to look inside the system log to see if there were any error messages during the initialization, but I couldn't see anything at first that would suggest that the associate-address command failed (inside /var/log/cloud-init-output).
Edit: Attempt to debug:
However, I then manually associated the Elastic IP with my instance, SSHed, and attempted to run the user data commands above. Interestingly, when I got to the aws ec2 associate-address portion, I ran into
Unable to locate credentials. You can configure credentials by running
"aws configure".
This appears to be at the root of the issue - my AWS profile is not configured. However, I've always been under the impression that a default AWS instance profile is set up for you with access to the AWS CLI when the instance finishes initializing.
Could anyone point me in the direction of why my user data to associate elastic IP addresses might not be executing properly?
Thank you!
Are you running your EC2 instance with the amazon provided AMI?
If so, the profile should be set up indeed.
However, we've noticed the default region is not set in your profile. So when running aws cli commands, you'll either need to give the region using --region <your region> or set the AWS_DEFAULT_REGION environment variable.
We use the following script to provision our servers with an EIP at startup as part of our CloudFormation template:
#!/bin/bash
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
MAXWAIT=3
ALLOC_ID=<your eip allocation id>
AWS_DEFAULT_REGION=<your region>
# Make sure the EIP is free
echo "Checking if EIP with ALLOC_ID[$ALLOC_ID] is free...."
ISFREE=$(aws ec2 describe-addresses --allocation-ids $ALLOC_ID --query Addresses[].InstanceId --output text)
STARTWAIT=$(date +%s)
while [ ! -z "$ISFREE" ]; do
if [ "$(($(date +%s) - $STARTWAIT))" -gt $MAXWAIT ]; then
echo "WARNING: We waited 30 seconds, we're forcing it now."
ISFREE=""
else
echo "Waiting for EIP with ALLOC_ID[$ALLOC_ID] to become free...."
sleep 3
ISFREE=$(aws ec2 describe-addresses --allocation-ids $ALLOC_ID --query Addresses[].InstanceId --output text)
fi
done
# Now we can associate the address
echo Running: aws ec2 associate-address --instance-id $INSTANCE_ID --allocation-id $ALLOC_ID --allow-reassociation}
aws ec2 associate-address --instance-id $INSTANCE_ID --allocation-id $ALLOC_ID --allow-reassociation}
We've added the part where we check if the EIP is free as this is used inside an autoscaling group and we've noticed sometimes the EIP is still in use even though the old instance is already terminated and the new instance is at the point of running the userdata script.
It looks the instance profile attached to this EC2 instance does not have permission to perform the above task.
Referring to https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-policies-ec2-console.html#ex-eip,
Can you make sure your Instance profile has the following action allowed?
ec2:AllocateAddress: To allocate an Elastic IP address.
ec2:ReleaseAddress: To release an Elastic IP address.
ec2:AssociateAddress: To associate an Elastic IP address with an instance or a network interface.
ec2:DescribeNetworkInterfaces and ec2:DescribeInstances: To work with the Associate address screen. The screen displays the available instances or network interfaces to which you can associate an Elastic IP address.
ec2:DisassociateAddress: To disassociate an Elastic IP address from an instance or a network interface.
Sample policy would look like:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeAddresses",
"ec2:AllocateAddress",
"ec2:DescribeInstances",
"ec2:AssociateAddress"
],
"Resource": "*"
}
]
}
Hope this helps.
A User Data script must start with #! to be recognized as a script. For example:
#!
INSTANCE_ID=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/instance-id`
aws ec2 associate-address --instance-id $INSTANCE_ID --allocation-id eipalloc-xxx --allow-reassociation
You can view the resulting log in: /var/log/cloud-init-output.log
I am trying to get the RDS endpoint to use in user data with cli but unable to figure it out.
I need to get the RDS endpoint to inject into a php file but when I try the following I get:
Unable to locate credentials. You can configure credentials by running "aws configure".
I am building the ec2 and vpc using CLI and need to be able to get RDS endpoint as part of the Userdata.
I tried the following on the EC2 instance itself and I get the above error.
aws rds --region ca-central-1 describe-db-instances --query "DBInstances[*].Endpoint.Address"
Even if I am able to resolve that, I need to be able to get the endpoint to pass as part of the userdata. Is that even possible?
The Unable to locate credentials error says that the AWS Command-Line Interface (CLI) does not have any credentials to call the AWS APIs.
You should assign a role to the EC2 instance with sufficient permission to call describe-db-instances on RDS. See: IAM Roles for Amazon EC2
Then, your User Data can include something like:
#!
RDS=`aws rds --region ca-central-1 describe-db-instances --query "DBInstances[*].Endpoint.Address"`
echo >file $RDS
Or pass it as a parameter:
php $RDS
I have it working with this -
mac=curl -s http://169.254.169.254/latest/meta-data/mac
VPC_ID=curl -s http://169.254.169.254/latest/meta-data/network/interfaces/macs/$mac/vpc-id
aws rds describe-db-instances --region us-east-2 | jq -r --arg VPC_ID "VPC_ID" '.DBInstances[] |select (.DBSubnetGroup.VpcId=="'$VPC_ID'") | .Endpoint.Address'
Today, I started working on AWS. I want to create an EC2 instance by using API only. I want to do more actions like copying instance, creating bucket, transforming file from one instance to another etc.
I am not getting how to create the instance using API.
Can anyone assist me?
Here are some examples using the AWS CLI.
aws ec2 run-instances --image-id ami-xxxxxxxx --count 1 --instance-type t2.micro --key-name MyKeyPair --security-group-ids sg-xxxxxxxx --subnet-id subnet-xxxxxxxx
Here's a simple code to create an EC2 instance.
import boto3
client = boto3.client('ec2', region_name='us-east-1')
client.run_instances(ImageId='ami-123456', SubnetId='subnet-12345', InstanceType='t2.micro', MinCount=1, MaxCount=1, KeyName='my-keypair', SecurityGroupIds=['sg-123456'])
Modify the image id: ami-123456, subnet id: subnet-12345, keypair name: my-keypair and security group id: sg-123456. Save this file as run_instances.py (or any name) and execute the script as: python run_instances.py. If you have setup your credentials properly, the instance will be launched in us-east-1.
References:
Setting up environment variables: Environment Variables
Configure profiles: Profiles
Run instances boto3: boto3 run_instances()
What I'm trying to do is to monitor log file through CloudWatch logs agent.
I have installed CloudWatch to my EC2 Linux Instance (EC2 Instance has Instance profile and IAM Role that are connected).
The installation was successful, but when I'm using sudo service awslogs status
I'm having this status massage dead but pid file exists.
In my error log file ( /var/log/awslogs.log) I have only this line that repeats over and over again - 'AccessKeyId'.
How can I fix Cloud Watch logs agent and make it to work?
This means that your AWS Logs agent requires your AWS Access Key/Secret. This can be provided in /etc/awslogs/awscli.conf in following format:
[plugins]
cwlogs = cwlogs
[default]
region = YOUR_INSTANCE_REGION (e.g. us-east-1)
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
Restart the service after making this change:
sudo service awslogs restart
Hope this helps!!!
Is there any way to find out status of AWS EC2 instances, which are running on various different regions, from one EC2 instance which is present in any one of region by using EC2 API tool ?
How this is possible ?
I got the answer :-
ec2-describe-instances instance-ID --region region
Example :-
ec2-describe-instances i-f82d5ca0 --region eu-west-1
Where instance ID is EC2 instance ID which is located in region eu-west-1
Thats all .
Or in the new unified AWS CLI, this is slightly different:
aws ec2 describe-instances --instance-id i-f82d5ca0
You can also change the --output into json, text, or a table