AWS Systems Manager - Instance not showing - amazon-web-services

Could anyone help me investigate an issue with EC2 instance profile? I have create an EC2 instance and I put an IAM role.
But, when I check on the instance I see: No roles attached to instance profile: xxx-instance-profile.
Any idea where I have to look? Because, when I check that instance profile (role), I have this in the trust:
Trusted entities The identity provider(s) ec2.amazonaws.com
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
I have attached one permission policy AmazonSSMManagedInstanceCore
When I go to my instance, I see that no roles are attached. And, in Systems Manager -> Session Manager, I don't see my instances.
I have no clue what I'm doing wrong :(
Any suggestions?
Regards,

I am not sure what you mean by an issue with EC2 instance profile. Instance profiles are permission sets that you grant to an EC2 instance, by defining a policy that contains the permissions required and attaching that policy to a role. The role is attached to an EC2 instance. Because the role will be used by a service it must have a trust relationship to that service.
Which Systems Manager service do you want to use? You can create your own custom policy with specific services and restrictions to specific AWS instances. Or you can use the managed policies.
Here are some example of various policies.
Let's suppose you want a role attached to an EC2 instance so that you can remotely login to that instance using Systems Manager Session Manager.
Let's assume the instance is in a VPC that has a route to the internet, either directly via the Internet Gateway or via a NAT Gateway to the Internet Gateway.
In this case, your instance has a route to the AWS Public Service for Systems Manager Session Manager. The instance must have the Systems Manager Session Manager agent installed. This agent is pre-installed on Amazon Linux 2, Amazon Linux and Ubuntu 16.04, 18.04, 20.04.
Assuming the agent is installed and there is a route to the service, then your instance as you mentioned need rights via IAM to access the service. This is done by granting a role to the EC2 instance.
To do this go to IAM - https://console.aws.amazon.com/iam/.
Select Roles from the navigation panel, create a new role
Select Type of trusted entity as AWS Service
Choose the EC2 option under Common Use cases
Press Next:Permissions
Here you can create a custom policy if you want, I suggest using a managed policy
Select an existing managed policy by searching for AmazonEC2RoleforSSM, there are other SSM managed policies, AmazonEC2RoleforSSM is specific for the management of EC2
select it and press next:tags
press next:review,
give it a name - my-ec2-ssm-role
Now we have a role for the EC2 instance, next we need to add that role to the instance.
Go to EC2 - https://console.aws.amazon.com/ec2
select your instance
from the menu on the top right, select actions, security, modify IAM role.
select the role you just created my-ec2-ssm-role
press save
Now that the role is linked go to Systems Manager Session Manager https://console.aws.amazon.com/systems-manager/session-manager
Press Start session
Your instance should be visible, and you can select it and press start session
If you instance is not visible, it could be that you do not have a route to the AWS Service Endpoints. For example the EC2 instance is not in a public subnet or does not have a route to the internet. In this case you need to add 3 VPC endpoints to your subnet. These endpoints are:
com.amazonaws.[region].ssm
com.amazonaws.[region].ssmmessages
com.amazonaws.[region].ec2messages
You can read how to set it up here.

After attaching the AmazonSSMManagedInstanceCore policy to an existing EC2 role, I had to reboot the EC2 instance before it showed up in Systems Manager. Thanks to #Jason who mentioned this in a comment.

You can run AWSSupport-TroubleshootManagedInstance runbook to check what it is missing in your instance's configuration.
If you need to make any change after the troubleshoot like adding an IAM role make sure to restart, the ssm agent in the ec2 instance in order to make it visible in the registered managed instances.

Answering "Systems Manager -> Session Manager, I don't see my instances" --Do you see your managed instances in Fleet Manager? One reason why Instances are not visible to the Systems manager is if the instance has no ssm agent installed. Eg: Ubuntu comes with ssm pre-installed but RHEL does not have ssm pre-installed. Check this out : https://aws.amazon.com/premiumsupport/knowledge-center/systems-manager-ec2-instance-not-appear/
Systems manager immediately showed my ubuntu instances, for RHEL instances I had to manually install ssm agent. https://docs.aws.amazon.com/systems-manager/latest/userguide/agent-install-rhel.html
This might be the reason why you cant see instances in session manager as well.

I had the same issue with all of my EC2 instances not showing up in Session Manager, even though they had the correct security/networking set up, turns out I had to go to Systems Manager -> Session Manager -> Preferences and Enable KMS encryption.

There are a few scenarios in which ssm can be deployed and break. All of this assumes you have the proper role attached to the vm.
resource aws_iam_role "ssm" {
name = "myssm"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
resource aws_iam_role_policy_attachment "ssm" {
policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
role = aws_iam_role.ssm.id
}
resource aws_iam_instance_profile "ssm" {
name = "myssm"
role = aws_iam_role.ssm.id
}
resource "aws_instance" "private" {
ami = data.aws_ami.amzn2.id
instance_type = "t3.micro"
subnet_id = aws_subnet.private.id
vpc_security_group_ids = [aws_security_group.test-ssm.id]
iam_instance_profile = aws_iam_instance_profile.ssm.name
tags = {
Name = "session-manager-private"
}
}
public subnet with no public ip (internet access)
In this scenario even though you have a vm in a public subnet and outbound via the igw, there's no public ip on the vm so ssm will not work.
public subnet with public ip (internet access)
In this scenario, same as the previous only difference being I've added a public ip to the vm and ssm kicks into life.
private subnet with public ip (internet access)
In this scenario even though the vm is in a private subnet it has outbound internet access via a public nat gateway which in turn has outbound access via the internet gateway. The nat gateway picutred here has a public ip. When a nat gw with a public ip sits infront of a private subnet those vms use that pubic ip for internet outbound, so ssm works.
Well I wonder what's going to happen here? If you guessed absolutely nothing, you'd be right. There's no public ip no route out of any kind and no way in. It's a completely isolated tenant. So how do you get ssm working this scenario? Next diagram.
private subnet with no public ip (no internet access)
In this instance, you need to add vpc endpoints - unsurprisingly to the vpc - and then associate them with the private subnet you want to connect into. Endpoints are created at vpc level and then "associated". The ssm endpoints are of type "interface" so an eni is created in that subnet for each endpoint and a private dns zone is set up so that the vm sends traffic to the local ssm enis and not to the aws fabric globally.
Here's some terraform to do it, sg is allowing 443.
locals {
endpoints= toset([
"com.amazonaws.eu-west-2.ssm",
"com.amazonaws.eu-west-2.ssmmessages",
"com.amazonaws.eu-west-2.ec2messages"
])
}
resource aws_vpc_endpoint "endpoints" {
for_each = local.endpoints
vpc_id = data.aws_vpc.main.id
service_name = each.key
vpc_endpoint_type = "Interface"
security_group_ids = [
aws_security_group.test-ssm.id
]
private_dns_enabled = true
}
resource "aws_vpc_endpoint_subnet_association" "association" {
for_each = local.endpoints
vpc_endpoint_id = aws_vpc_endpoint.endpoints[each.key].id
subnet_id = aws_subnet.completely-private.id
}

In my case, it took about 30 minutes for EC2 instance to appear in Fleet Manager. I had existing EC2 without any attached IAM service role. I created a new IAM role with AmazonSSMManagedInstanceCore and AmazonEC2RoleforSSM permission policies and in about 30 minutes my EC2 popped up in Fleet Manager. BTW, Windows platform EC2 instance also comes with preinstalled SSM Agent.

Related

VPC Endpoint: Specific Services Not Available in Availability Zone

When I attempt to create a VPC Endpoint for the com.amazonaws.us-east-1.lambda (lambda service), the "us-east-1a" Availability Zone is not an option. However, when I choose a different service, like "com.amazonaws.us-east-1.rds", I can choose a subnet in the "us-east-1a" Availability Zone.
I am creating VPC endpoints via CloudFormation template, but also confirmed this occurs when creating via the UI.
I have been reviewing AWS documentation and also previous questions, but I cannot determine why this is occurring and how to fix this so we can select the subnets in that AZ for that VPC endpoint. Any guidance is appreciated.
Screenshot of attempting to create VPC endpoint for lambda with us-east-1a not allowed:
Screenshot of attempting to create VPC endpoint for another service:
You can run the CLI command to check for a service and the Availability Zones which are available to use for creating a VPC endpoint.
aws ec2 describe-vpc-endpoint-services --service-names SERVICE-NAME
Example for Lambda:
aws ec2 describe-vpc-endpoint-services --service-names com.amazonaws.us-east-1.lambda
{
"ServiceDetails": [
{
"ServiceName": "com.amazonaws.us-east-1.lambda",
"AvailabilityZones": [
"us-east-1a",
"us-east-1b",
"us-east-1c"
]....}
Why can’t I select an Availability Zone for my Amazon VPC interface endpoint?
https://aws.amazon.com/premiumsupport/knowledge-center/interface-endpoint-availability-zone/

VPC endpoint unable to connect to S3 in shared AWS account

I am receiving "Could not connect to the endpoint URL: "https://s3.amazonaws.com/" from inside EC2 instance running inside private subnet
Note: We are using our corporate shared AWS account instead of Federated account for this exercise.
Here is a configuration:
Created one VPC with 1 private(Attached to VPC endpoints for S3 and Dynamodb) and 1 public (attached to Internet Gateway) subnet. There is no NAT gateway or instance.
Launched EC2 instance(Amazon Linux AMI) one inside each subnet.
Attached IAM roles to access dynamodb and S3 to both the EC2 instance
Connected to EC2 from terminal. Configured my access keys using aws configure
Policy for S3 VPC endpoint:
"Statement": [
{
"Action": "*",
"Effect": "Allow",
"Resource": "*",
"Principal": "*"
}
]
}
Routing is automatically added to the VPC routing where destination is pl-xxxxxxxx(com.amazonaws.us-east-1.s3) and target is the endpoint created in
Opened all traffic in the outbound rules in Security Group for the private subnet to destination prefix s3 endpoint starting with pl-xxxxxxxx
Now entered following command in private ec2 instance on terminal
aws s3 ls --debug --region us-west-2
I got following error
"ConnectionTimeout: Could not connect to the endpoint URL https://sts.us-west-2.amazonaws.com:443"
I read almost all the resources on google and they follow same steps that I have been following but it is not working out for me.
The only difference is that they are using federated AWS account whereas I am using a shared AWS account.
Same goes for dynamodb access.
Similar stackoverflow issue: Connecting to S3 bucket thru S3 VPC Endpoint inside EC2 instance timing out
But I could not benefit from it much.
Thanks a lot in advance.
Update: I was able to resolve the issue with STS endpoint by creating STS interface endpoint in the private subnet and then accessing the Dynamodb and S3 by assuming role inside the EC2 instance

Create managed instance

I am trying to create a managed instance with AWS.
I followed this page to create the IAM role:https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-instance-profile.html.
So it is AmazonSSMManagedInstanceCore
This is the code I am using to associate the IAM role with the EC2.
# Make EC2s with AWS Ubuntu 20
instances = subnet.create_instances(ImageId='ami-0885b1f6bd170450c',
InstanceType='m1.small',
MaxCount=num,
MinCount=num,
Monitoring={'Enabled': True},
SubnetId=subnet.subnet_id,
KeyName=key_name,
IamInstanceProfile={
'Arn': 'arn goes here',
},)
wait_until_running(instances)
And when I check in the console the role shows up.
But when I do
aws ssm describe-instance-information
I get
{
"InstanceInformationList": []
}
The ultimate goal here is to be able to send a command to the instance.
Based on the comments.
The instance does not have public IP address, which indicates it likely has no access to SSM service.
For SSM to work on your instance, it must be able to connect to the SSM service. This is usually enabled in one of three ways:
Instance is a public subnet and has direct internet access.
Instance is in a private subnet and uses NAT gateway to access internet.
Your VPC uses VPC interface endpoints for SSM service to connect to the SSM service. This does not require internet access and provides private communication between instance and the SSM service.

EKS cluster - api endpoint access - public/private

I have provisioned EKS cluster on AWS with public access to api endpoint. While doing I configured SG with ingress only from specific IP. But I could still run the kubectl get svc against the cluster when accessing it from another IP.
I want to have IP restricted access to EKS cluster.
ref - Terraform - Master cluster SG
If public access is enabled does it mean that anyone who has cluster name can deploy anything?
When you create a new cluster, Amazon EKS creates an endpoint for the managed Kubernetes API server that you use to communicate with your cluster (using Kubernetes management tools such as kubectl as you have done).
By default, this API server endpoint is public to the internet, and access to the API server is secured using a combination of AWS Identity and Access Management (IAM) and native Kubernetes Role Based Access Control (RBAC).
So the public access does not mean that anyone who has the cluster name can deploy anything. You can read more about that in the Amazon EKS Cluster Endpoint Access Control AWS documentation.
If you want to provision EKS with Terraform and manage the network topology it's happened through the VPC (Virtual Private Network). You can check this VPC Terraform Module to get all the proper settings.
Hope it'll help.
As well as Claire Bellivier' answer about how EKS clusters are protected via authentication using IAM and RBAC you can now also configure your EKS cluster to be only accessible from private networks such as the VPC the cluster resides in or any peered VPCs.
This has been added in the (as yet unreleased) 2.3.0 version of the AWS provider and can be configured as part of the vpc_options config of the aws_eks_cluster resource:
resource "aws_eks_cluster" "example" {
name = %[2]q
role_arn = "${aws_iam_role.example.arn}"
vpc_config {
endpoint_private_access = true
endpoint_public_access = false
subnet_ids = [
"${aws_subnet.example.*.id[0]}",
"${aws_subnet.example.*.id[1]}",
]
}
}

EC2 cannot access aws s3 even it has IAM rule to do that

In a VPC, I have two Subnets, one is a public subnet with an EC2 instance, the other is a private subnet with 2 EC2 instances. All 3 EC2 instances have the same IAM role to access S3.
The EC2 instance in the public subnet can access S3 directly if I login and run aws s3 ls. However, both of the EC2 instances in the private subnet cannot. What can be the reasons?
EC2 in the private subnet uses a Security Group that accepts traffic from the whole VPC.
EC2 in the public subnet use a Security Group that accepts traffic from Anywhere.
All 3 EC2 instances use the same routing table, same NACLs, use the same IAM role, with the policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "*"
}
]
}
If I manually create a credential profile in the EC2 in the private Subnet, then it can login and access "aws s3 ls".
Update:
The routing table of the private subnets does have a VPC Endpoint. The routing table is:
dest=10.1.0.0/16 (myVPC)->target=local
dest=0.0.0.0/0, target=iGW
dest=pl-68a54001 (com.amazonaws.us-west-2.s3)->target=vpce-26cf344f
Among them, #3 means the EC2 can access S3 via VPC endpoint. #2 is added because an ELB is in front of the EC2 instance and has to access internet.
Another observation: If I enable Assign Public IP in the private subnet and then launch a new EC2 instance, this EC2 instance can access S3. If I disable Assign Public IP in the private subnet and then launch a new EC2 instance, the new EC2 instance cannot access S3.
BTW, I already have region set as us-west-2 before running terraform:
[ec2-user#ip-XXXXX]$ echo $AWS_DEFAULT_PROFILE
abcdefg
[ec2-user#XXXXX]$ aws configure --profile abcdefg
AWS Access Key ID [****************TRM]:
AWS Secret Access Key [****************nt+]:
Default region name [us-west-2]:
Default output format [None]:
The fact that your solution works when an instance has a Public IP address, but does not work when it does not have a Public IP address, suggests that the instance is actually in a public subnet.
Indeed, looking at "the routing table of the private subnets", you include this line:
2. dest=0.0.0.0/0, target=iGW
This is making the subnet a Public Subnet because it is pointing to an Internet Gateway.
To clarify:
A public subnet is a subnet with a Route Table entry that points to an Internet Gateway
A private subnet is a subnet with a Route Table that does not point to an Internet Gateway
Therefore, you should remove the above entry from the Route Table for your private subnets if you actually want them to be private.
Next, it appears that you aws s3 ls request is not being sent to the VPC Endpoint. This might be because you are not sending traffic to s3.us-west-2.amazonaws.com as listed in your route table. Try this command:
aws s3 ls --region us-west-2
That will send the request to the S3 endpoint that would be routed via the VPC Endpoint. You should direct all of your S3 commands to that region, since the VPC Endpoint only points to the region in which it was configured.
When you place the EC2 in Private Subnet, Network level it doesn't have access to S3 (Not an issue with the IAM policy). To allow outbound access to S3 from EC2 instances in Private Subnet, you have the following options.
VPC endpoints for S3
NAT Gateway
Out of the two approaches, if you plan to allow access only to S3 from EC2 instance in Private Subnet, configure VPC endpoints for S3.