I'm unable to pull Amazon Cloudwatch metrics in an EC2 instance with Grafana. The setup is the following:
EC2 instance with Grafana:
Security group: (ssh,tpc,22,0.0.0.0/0;custom tcp,tpc,3000,0.0.0.0/0;)
Policy attached to the role in the EC2 instance with Grafana:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowReadingMetricsFromCloudWatch",
"Effect": "Allow",
"Action": [
"cloudwatch:DescribeAlarmsForMetric",
"cloudwatch:DescribeAlarmHistory",
"cloudwatch:DescribeAlarms",
"cloudwatch:ListMetrics",
"cloudwatch:GetMetricStatistics",
"cloudwatch:GetMetricData"
],
"Resource": "*"
},
{
"Sid": "AllowReadingLogsFromCloudWatch",
"Effect": "Allow",
"Action": [
"logs:DescribeLogGroups",
"logs:GetLogGroupFields",
"logs:StartQuery",
"logs:StopQuery",
"logs:GetQueryResults",
"logs:GetLogEvents"
],
"Resource": "*"
},
{
"Sid": "AllowReadingTagsInstancesRegionsFromEC2",
"Effect": "Allow",
"Action": [
"ec2:DescribeTags",
"ec2:DescribeInstances",
"ec2:DescribeRegions"
],
"Resource": "*"
},
{
"Sid": "AllowReadingResourcesForTags",
"Effect": "Allow",
"Action": "tag:GetResources",
"Resource": "*"
}
]
}
Grafana query screenshot without retrieving any data:
https://docs.aws.amazon.com/AWSEC2/latest/APIReference/monitor.html
This is an opt-in feature. To enable this feature for your AWS account, contact AWS Support.
Are you sure you have enabled this feature for your AWS account?
Are you sure that InstanceId is the right dimension and that only one
dimension is required?
The best option is to go to the AWS CloudWatch console and generate graph there. Then you can compare generated CloudWatch query with the CloudWatch query generated by Grafana.
I finally could pull some data by changing the namespace to AWS/ElasticBeanstalk and the Dimensions with EnvironmentName to the one I want to target. Before I had to go to the configuration dashboard of the Elasticbeanstalk and add the proper metrics in the CloudWatch Custom Metric in the input select
Related
I have two AWS accounts, Prod and Staging. I need to migrate data from prod elasticache redis to staging. The Redis clusters in prod and staging are both 1 node and 0 shards each. I believe you cant update an already running cluster so I'm trying to
seed a new cluster with the RDB file with the same cluster configuration as already exists in staging, with a view to deleting the original staging cluster after the new one stands up
Problem is every time I go through the console and create a Redis cluster it stands up with 1 shard and 1 node. The original cluster had 0 shards. I selected 0 number of replicas, no multi AZ etc so I'm not sure why its defaulting to sharding. Am I missing an option somewhere, are you able to stand up a 1 node 0 shard cluster via the console?
I also tried creating a cluster via the AWS CLI to see if i get the same behaviour but get the error message:
An error occurred (InvalidParameterValue) when calling the CreateCacheCluster operation: No permission to access S3 object: mybucket/folder/file.rdb
I've set the bucket policy to the following:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::0123456789:root"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::mybucket"
},
{
"Sid": "2",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::0123456789:root"
},
"Action": [
"s3:PutObject",
"s3:GetObject"
],
"Resource": "arn:aws:s3:::mybucket/*"
},
{
"Sid": "ExampleStatement",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::0123456789:user/my-user"
},
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::mybucket",
"arn:aws:s3:::mybucket/*",
"arn:aws:s3:::mybucket/folder/file.rdb"
]
},
{
"Sid": "Stmt15399483",
"Effect": "Allow",
"Principal": {
"Service": "eu-west-1.elasticache-snapshot.amazonaws.com"
},
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:GetBucketAcl"
],
"Resource": [
"arn:aws:s3:::mybucket",
"arn:aws:s3:::mybucket/*",
"arn:aws:s3:::mybucket/folder/file.rdb"
]
}
]
}
Note this policy is in the permissions tab of the top level bucket. when i click on the file there is no area to add/edit a bucket policy only an ACL edit option. I did grant elasticache canonical ID read/write as it suggests in the AWS docs but still get permission denied
I was able to get the cluster configuration (1 node 0 shards) by using the CLI command:
aws elasticache delete-replication-group --replication-group-id my-cluster --retain-primary-cluster
although this requires running the command after cluster creation. I'm hoping theres a way to select this config at creation time
I am trying to create an IAM user that will have least privileges to be able to view enhanced monitoring for a particular RDS database. I have created a ROLE (Enhanced Monitoring) and attached a managed policy to it:'AmazonRDSEnhancedMonitoringRole'. This role is passed to RDS database using the passrole permission. The policy that I am attaching to this IAM user is as below:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData",
"rds:*",
"cloudwatch:GetMetricData",
"iam:ListRoles",
"cloudwatch:GetMetricStatistics",
"cloudwatch:DeleteAnomalyDetector",
"cloudwatch:ListMetrics",
"cloudwatch:DescribeAnomalyDetectors",
"cloudwatch:ListMetricStreams",
"cloudwatch:DescribeAlarmsForMetric",
"cloudwatch:ListDashboards",
"ec2:*",
"cloudwatch:PutAnomalyDetector",
"cloudwatch:GetMetricWidgetImage"
],
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"iam:GetRole",
"iam:PassRole",
"cloudwatch:*"
],
"Resource": [
"arn:aws:cloudwatch:*:accountnumber:insight-rule/*",
"arn:aws:iam::accountnumber:role/Enhanced-Monitoring",
"arn:aws:rds:us-east-1:accountnumber:db:dbidentifier"
]
}
]
}
As you can see,I have given almost every permission to this user, but still I am getting 'Not Authorized' error on the IAM user RDS dashboard for enhanced monitoring, although cloudwatch logs are displaying normally.
I managed to solve the error by modifying the policy. Since Enhanced Monitoring utilizes CloudWatch logs, I added a read-only cloudwatch logs policy(CloudWatchLogsReadOnlyAccess). This policy is mentioned here in AWS documentation. I also added a cloudwatch:GetMetricData permission as suggested on the RDS console. The overall policy is also more compact now:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::account_number:role/Enhanced-Monitoring"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"rds:*",
"logs:Describe*",
"logs:List*",
"logs:StartQuery",
"ec2:*",
"iam:ListRoles",
"logs:StopQuery",
"logs:TestMetricFilter",
"logs:FilterLogEvents",
"logs:Get*",
"cloudwatch:GetMetricData"
],
"Resource": "*"
}
]
}
The setup is working fine now.
I have .ova VM's stored on my S3 bucket, I am trying to create AMI from these OVA.
I was going through this video to Import a VM as an Image Using VM Import/Export to Amazon EC2.
I have created an EC2 Instance which I will use to trigger the necessary CLI commands for Importing.
I have created an IAM Role and attached it to the EC2 Instance.
Please refer to the details of the Role:
Trust Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "vmie.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Inline Policy for Access to S3 and EC2
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"ec2:CopySnapshot",
"s3:ListAccessPointsForObjectLambda",
"s3:GetAccessPoint",
"s3:PutAccountPublicAccessBlock",
"s3:ListAccessPoints",
"ec2:RegisterImage",
"s3:ListJobs",
"s3:PutStorageLensConfiguration",
"s3:ListMultiRegionAccessPoints",
"s3:ListStorageLensConfigurations",
"ec2:Describe*",
"s3:GetAccountPublicAccessBlock",
"ec2:ModifySnapshotAttribute",
"s3:ListAllMyBuckets",
"s3:PutAccessPointPublicAccessBlock",
"s3:CreateJob",
"ec2:ImportImage"
],
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "s3:*",
"Resource": "arn:aws:s3:::vms"
},
{
"Sid": "AllowStsDecode",
"Effect": "Allow",
"Action": "sts:DecodeAuthorizationMessage",
"Resource": "*"
}
]
}
Inline Policy for KMS Decrypt
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "kms:Decrypt",
"Resource": "*"
}
]
}
Also, I have attached the AWSImportExportFullAccess managed policy to the Role.
I am using the following command to Import the VM to AMI:
aws ec2 import-image --description "MY_VM_Image" --disk-containers "file://configuration.json"
Here are the contents of configuration.json
[{
"Description": "Image",
"Format": "ova",
"UserBucket": {
"S3Bucket": "vm",
"S3Key": "xzt.ova"
}
}
]
But I am facing the following error:
An error occurred (InvalidParameter) when calling the ImportImage operation: The service role vmimport provided does not exist or does not have sufficient permissions
I tried to have a look at the Troubleshooting document. It states the following
This error can also occur if the user calling ImportImage has Decrypt permission but the vmimport role does not.
So, I have also disabled the default encryption at S3.
Still no luck.
What else permissions are needed to run the command successfully.
I was facing the same issue and it turned out to be an issue with the clock not being in sync with the NTP servers (it was around 6 minutes off). As soon as the time was synced, the aws ec2 import-image worked as expected.
Here is a link for the importance of Time Synchronization in Kerberos:
https://tldp.org/HOWTO/Kerberos-Infrastructure-HOWTO/time-sync.html#:~:text=If%20you%20allow%20your%20clocks,errors%20and%20refuse%20to%20function.
I've followed a great tutorial by Martin Thwaites outlining the process of logging to AWS CloudWatch using Serilog and .Net Core.
I've got the logging portion working well to text and console, but just can't figure out the best way to authenticate to AWS CloudWatch from my application. He talks about inbuilt AWS authentication by setting up an IAM policy which is great and supplies the JSON to do so but I feel like something is missing. I've created the IAM Policy as per the example with a LogGroup matching my appsettings.json, but nothing comes though on the CloudWatch screen.
My application is hosted on an EC2 instance. Are there more straight forward ways to authenticate, and/or is there a step missing where the EC2 and CloudWatch services are "joined" together?
More Info:
Policy EC2CloudWatch attached to role EC2Role.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
ALL EC2 READ ACTIONS HERE
],
"Resource": "*"
},
{
"Sid": "LogStreams",
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:DescribeLogStreams",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:log-group:cloudwatch-analytics-staging
:log-stream:*"
},
{
"Sid": "LogGroups",
"Effect": "Allow",
"Action": [
"logs:DescribeLogGroups"
],
"Resource": "arn:aws:logs:*:*:log-group:cloudwatch-analytics-staging"
}
]
}
In order to effectively apply the permissions, you need to assign the role to the EC2 instance.
I am using the policy to limit RunIstances only to a specific instance types and a specific region. When I run the launch wizard or simulation under a test user I am getting "implicitly denied" error.
Here is the policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "ec2:RunInstances",
"Resource": [
"arn:aws:ec2:us-east-1::instance/*"
],
"Condition": {
"StringEquals": {
"ec2:InstanceType": [
"t1.micro",
"m1.small"
]
}
}
},
{
"Effect": "Allow",
"Action": "ec2:RunInstances",
"Resource": [
"arn:aws:ec2:us-east-1::image/ami-*",
"arn:aws:ec2:us-east-1::subnet/*",
"arn:aws:ec2:us-east-1::network-interface/*",
"arn:aws:ec2:us-east-1::volume/*",
"arn:aws:ec2:us-east-1::key-pair/*",
"arn:aws:ec2:us-east-1::security-group/*"
]
},
{
"Effect": "Allow",
"Action": [
"ec2:Describe*",
"ec2:CreateSecurityGroup",
"ec2:DeleteSecurityGroup",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:CreateKeyPair"
],
"Resource": [
"*"
]
}
]
}
could somebody point to the issue?
In your resource blocks, insert an asterisk between the two ":" in the arn lines, to specify all accounts, or replace it with your account number.
"arn:aws:ec2:us-east-1:*:instance/*"
"arn:aws:ec2:us-east-1:*:image/ami-*",
"arn:aws:ec2:us-east-1:*:subnet/*",
"arn:aws:ec2:us-east-1:*:network-interface/*",
"arn:aws:ec2:us-east-1:*:volume/*",
"arn:aws:ec2:us-east-1:*:key-pair/*",
"arn:aws:ec2:us-east-1:*:security-group/*"
Please see the IAM policy generator tool. Your code does not look like the right syntax.
First, Allow all actions in EC2. Next, Deny specific actions in EC2.
Example to Allow all actions in EC2:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"NotAction": "ec2:*",
"Resource": "*"
}
]
}
Example to Deny creating resources in specific regions:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DenyTheseActions",
"Effect": "Deny",
"Action": [
"ec2:RunInstances",
"ec2:StartInstances"
],
"Resource": "arn:aws:ec2:us-west-1",
"Resource": "arn:aws:ec2:us-east-1",
"Resource": "arn:aws:ec2:eu-west-1",
"Resource": "arn:aws:ec2:sa-east-1",
"Resource": "arn:aws:ec2:ap-northeast-1",
"Resource": "arn:aws:ec2:ap-southeast-1",
"Resource": "arn:aws:ec2:ap-southeast-2"
}
]
}
Currently the easier way to control access to AWS regions, EC2 and RDS Instances sizes and types would probably be using IAM policies with the Condition (Optional) policy element – lets you specify conditions for when a policy is in effect.
We'll setup an AWS IAM Policy using a Condition statement, which will allow full AWS EC2 and RDS services, strictly for the 3 more cost effective -> AWS regions which are:
us-east-1 (North Virginia, USA)
us-east-2 (Ohio, USA)
us-west-2 (Oregon, USA)
All other regions will not be allowed, this policy also sets up a conditional access control for starting both EC2 and RDS instances. Instead of specifying all of the possible type/class instances to run, we use Deny effect in the statement, which allows for a short and simple policy limiting by instances sizes (micro, small, medium and large). This has the further effect of preventing any other policies from overriding the block.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "MultiServiceFullAccessCustom",
"Effect": "Allow",
"Action": [
"ec2:*",
"rds:*"
],
"Resource": [
"*"
],
"Condition": {
"StringEquals": {
"aws:RequestedRegion": [
"us-east-1",
"us-east-2",
"us-west-2"
]
}
}
},
{
"Sid": "Ec2RunInstanceCustomSize",
"Effect": "Deny",
"Action": "ec2:RunInstances",
"Resource": [
"arn:aws:ec2:*:*:instance/*"
],
"Condition": {
"ForAnyValue:StringNotLike": {
"ec2:InstanceType": [
"*.nano",
"*.micro",
"*.small",
"*.medium",
"*.large"
]
}
}
},
{
"Sid": "RdsFullAccessCustomSize",
"Effect": "Deny",
"Action": [
"rds:CreateDBInstance",
"rds:CreateDBCluster"
],
"Resource": [
"arn:aws:rds:*:*:db:*"
],
"Condition": {
"ForAnyValue:StringNotLike": {
"rds:DatabaseClass": [
"*.micro",
"*.small",
"*.medium",
"*.large"
]
}
}
}
]
}
What does this policy not protect us from?
This does not impose limits on the size of instances deployed though other services - most importantly, through an auto-scaling group or EKS.
We are not limiting in any way the total number of instances spun up by a given user. The normal limits help here, but there isn't a way to determine how many instances are currently running through just policy.
We're not capping costs, just adding some protections. If you want to set alerts for cost overruns or spikes, you need to look into the Billing controls.
terraform-aws-cost-billing-alarm
terraform-aws-cost-budget
terraform-aws-lambda-nuke
terraform-aws-lambda-scheduler-stop-start
We do not require any Multi Factor Authentication (MFA) for the role switch. The assumption is that users logging in are already using some form of second factor for authentication. This is mostly due to the increased complexity around using the CLI with MFA, and the fact that the role change is only to allow an override, not to enable an elevation to another security domain. You might want to add the feature; it is not difficult to add, and can bring reasonable protections.
We are only looking at instance startup, but you may consider who can shutdown or terminate instances, as that may result in downtime or data loss.
Reference links
https://blog.vizuri.com/limiting-allowed-aws-instance-type-with-iam-policy
https://blyx.com/2016/03/24/how-to-restrict-by-regions-and-instance-types-in-aws-with-iam/
https://aws.amazon.com/blogs/security/easier-way-to-control-access-to-aws-regions-using-iam-policies/
https://medium.com/faun/100-days-of-devops-day-10-restricting-user-to-launch-only-t2-instance-509aaaec5aa2
https://aws.amazon.com/blogs/security/back-to-school-understanding-the-iam-policy-grammar/