Spinnaker CloudDriver Pod crashing because of IAM issue - amazon-web-services

When deploying Spinnaker to EKS via hal deploy apply, Spinnaker Clouddriver pod goes to CrashLoopBackOff with the following error,
Caused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/Spinnaker-k8s-Worker-Node-Role/i-yyyyyyyyyyyyyyy is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::xxxxxxxxxxxx:role/Spinnaker-Managed-Role
My Halyard config is like follows,
currentDeployment: default
deploymentConfigurations:
- name: default
version: 1.17.6
providers:
appengine:
enabled: false
accounts: []
aws:
enabled: true
accounts:
- name: my-account
requiredGroupMembership: []
providerVersion: V1
permissions: {}
accountId: 'xxxxxxxxxxxx' # my account id here
regions:
- name: us-east-1
assumeRole: Spinnaker-Clouddriver-Role
lifecycleHooks: []
primaryAccount: my-account
bakeryDefaults:
baseImages: []
defaultKeyPairTemplate: '{{name}}-keypair'
defaultRegions:
- name: us-east-1
defaults:
iamRole: BaseIAMRole
My Spinnaker-Clouddriver-Role IAM role has PowerUserAccess permissions at the moment and has following as the Trust Relationship
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"ec2.amazonaws.com",
"ecs.amazonaws.com",
"application-autoscaling.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::xxxxxxxxxxx:role/Spinnaker-k8s-Worker-Node-Role"
},
"Action": "sts:AssumeRole"
}
]
}
How can I get this resolved?
The full log can be found on https://gist.github.com/agentmilindu/d9d31ee4287c87fb87e5060e0709989d#file-awssecuritytokenserviceexception-log-L3

Take a look at an AWS IAM policy that works flawlessly.
Refer to Armory AWS IAM set up and EC2 provider set up
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:*",
"cloudformation:*",
"ecr:*"
],
"Resource": [
"*"
]
},
{
"Action": "sts:AssumeRole",
"Resource": [
"arn:aws:iam::123456789012:role/SpinnakerManagedRoleAccount1",
"arn:aws:iam::101121314157:role/SpinnakerManagedRoleAccount2",
"arn:aws:iam::202122232425:role/SpinnakerManagedRoleAccount3"
],
"Effect": "Allow"
}
]
}

Related

AWS Role chaning for GitHub Actions

I am pretty new to AWS and struggling with role chaining for my github-actions-role.
I have a role “GithubActionsRole” which has the following Permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"sts:AssumeRole",
"sts:SetSourceIdentity"
],
"Resource": [
"arn:aws:iam::ACCOUNT_NO:role/cdk-contactupd-deploy-role-ACCOUNT_NO-REGION
],
"Effect": "Allow"
}
]
}
And this Trust relationships:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::ACC_NO:oidc-provider/token.actions.githubusercontent.com"
},
"Action": [
"sts:AssumeRoleWithWebIdentity",
"sts:SetSourceIdentity"
],
"Condition": {
"StringLike": {
"token.actions.githubusercontent.com:sub": "repo:my-repo/sub-dir:*”
}
}
}
]
}
So now, the GithubActionsRole should assume the role “cdk-contactupd-deploy-role-ACCOUNT_NO-REGION”. This role is configured like this:
Persmissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"cloudformation:CreateChangeSet",
"cloudformation:DeleteChangeSet",
"cloudformation:DescribeChangeSet",
"cloudformation:DescribeStacks",
"cloudformation:ExecuteChangeSet",
"cloudformation:CreateStack",
"cloudformation:UpdateStack"
],
"Resource": "*",
"Effect": "Allow",
"Sid": "CloudFormationPermissions"
}
]
}
Trust relationships:
{
"Version": "2008-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNT_NO:root"
},
"Action": [
"sts:AssumeRole",
"sts:SetSourceIdentity"
]
}
]
}
In my GitHub-action I execute the following:
…
- name: Set nonlive variables
run: |
echo "STAGE=nonlive" >> $GITHUB_ENV
echo "GITHUB_ROLE=arn:aws:iam::ACC_NO:role/GithubActionsRole" >> $GITHUB_ENV
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials#v1
with:
aws-region: MY_REGION
role-to-assume: ${{ env.GITHUB_ROLE }}
- run: aws sts get-caller-identity
- name: Deploy CDK Stacks
…
Unfortunately the last step is failing with the following Message:
failed: AccessDenied: User: arn:aws:sts::***:assumed-role/GithubActionsRole/GitHubActions is not authorized to perform: cloudformation:DescribeStacks on resource: arn:aws:cloudformation:...:stack/MY_STACK/... because no identity-based policy allows the cloudformation:DescribeStacks action.
But the "cloudformation:DescribeStacks"-policy should be allowed through the role chaining from the ""cdk-contactupd-deploy-role-ACCOUNT_NO-REGION" role.
I appreciate any idea how to fix it. I tried already a lot the last few days :/

IAM role is not authorized to perform ec2messages:GetMessages

I've created a role with an attached Policy "AmazonSSMManagedInstanceCore":
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssm:DescribeAssociation",
"ssm:GetDeployablePatchSnapshotForInstance",
"ssm:GetDocument",
"ssm:DescribeDocument",
"ssm:GetManifest",
"ssm:GetParameter",
"ssm:GetParameters",
"ssm:ListAssociations",
"ssm:ListInstanceAssociations",
"ssm:PutInventory",
"ssm:PutComplianceItems",
"ssm:PutConfigurePackageResult",
"ssm:UpdateAssociationStatus",
"ssm:UpdateInstanceAssociationStatus",
"ssm:UpdateInstanceInformation"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ssmmessages:CreateControlChannel",
"ssmmessages:CreateDataChannel",
"ssmmessages:OpenControlChannel",
"ssmmessages:OpenDataChannel"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2messages:AcknowledgeMessage",
"ec2messages:DeleteMessage",
"ec2messages:FailMessage",
"ec2messages:GetEndpoint",
"ec2messages:GetMessages",
"ec2messages:SendReply"
],
"Resource": "*"
}
]
}
And Trust relationships:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
I've then attached the IAM role to the instance. When I start the SSM agent in the instance I get the following error:
2022-03-16 23:14:49 ERROR [HandleAwsError # awserr.go.49] [ssm-agent-worker] [MessageService] [MDSInteractor] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::XXXX:assumed-role/SSMandCloudWatch/i-YYYYY is not authorized to perform: ec2messages:GetMessages on resource: arn:aws:ssm:eu-central-1:XXXX:* with an explicit deny in a service control policy
status code: 400, request id: zzzz
The call it's complaining about is explicitly allowed in the policy. I've tried restarting the agent but didn't make any difference.
AWS permission evaluation can be complex. I like this AWS diagram below, so it is a good one to follow to track down permissions issues.
So there are a few other things to check or be aware of that could still be limiting access.

Step function unable to trigger ECS task on fargate cluster, permission issue

I am creating and running a task on my ECS fargate cluster.
Task definition (with role) and fargate cluster is already created.
When I use run task step in step function, I am getting following error,
{
"Error": "ECS.AccessDeniedException",
"Cause": "User: arn:aws:sts::xxxxxxxxxx:assumed-role/StepFunctions-my-state-machine-role-xxxxxxxxxx/xxxxxxxxxx is not authorized to perform: iam:PassRole on resource: arn:aws:iam::xxxxxxxxxx:role/my-app-dev-exec because no identity-based policy allows the iam:PassRole action (Service: AmazonECS; Status Code: 400; Error Code: AccessDeniedException; Request ID: xxxxxxxxxx-xxxxxxxxxx-xxxxxxxxxx; Proxy: null)"
}
The role attached to the step function has the following policies (as per the documentation provided by AWS https://docs.aws.amazon.com/step-functions/latest/dg/ecs-iam.html)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecs:RunTask"
],
"Resource": [
"arn:aws:ecs:eu-west-1:xxxxxxxxxx:task-definition/*:*"
]
},
{
"Effect": "Allow",
"Action": [
"ecs:StopTask",
"ecs:DescribeTasks"
],
"Resource": [
"arn:aws:ecs:eu-west-1:xxxxxxxxxx:task/*"
]
},
{
"Effect": "Allow",
"Action": [
"events:PutTargets",
"events:PutRule",
"events:DescribeRule"
],
"Resource": [
"arn:aws:events:eu-west-1:xxxxxxxxxx:rule/StepFunctionsGetEventsForECSTaskRule"
]
},
{
"Effect": "Allow",
"Action": [
"states:DescribeStateMachine",
"states:StartExecution",
"states:ListExecutions",
"states:UpdateStateMachine"
],
"Resource": [
"arn:aws:states:eu-west-1:xxxxxxxxxx:stateMachine:my-state-machine"
]
}
]
}
with following trusted entities
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "states.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Effect": "Allow",
"Principal": {
"Service": "events.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Can someone help with what additional permission I need to give to resolve the above permission issue? From the error, I am not able to figure out what additional permission is required.
If I attach AmazonECS_FullAccess (aws managed) policy to the role, job works perfectly.
Because your task will use an IAM Role, you need to specify the additional permission 'PassRole'.
The best practice is to restrict which roles can be passed. So is recommended to add a condition limiting to only allow to pass roles to ECS tasks.
Try adding this statement to your policy:
{
"Action": "iam:PassRole",
"Effect": "Allow",
"Resource": [
"*"
],
"Condition": {
"StringLike": {
"iam:PassedToService": "ecs-tasks.amazonaws.com"
}
}
}

Cross account IAM roles for Kubernetes service account - s3 bucket

Hey im trying to cross account access for a role. i have 2 accounts: prod and non-prod.
and bucket in prod account, which im trying to write files to there from a non-prod role which is used as a service account in k8s cluster.
in prod account i configured:
a role with the following policy(read write access to the bucket):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListObjectsInBucket",
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::test2"
]
},
{
"Sid": "AllObjectActions",
"Effect": "Allow",
"Action": "s3:*Object",
"Resource": [
"arn:aws:s3:::test2/*"
]
}
]
}
and the following trust:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::non-prod-AccountID:role/name-of-the-non-prod-role"
},
"Action": "sts:AssumeRole",
"Condition": {}
}
]
}
in non prod i configured:
a role with the following policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::prod-Account-ID:role/prod-role-name"
},
"Action": "sts:AssumeRole",
"Condition": {}
}
]
}
and trust as follows:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::non-prod-accountID:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/1111111111111111111"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/1111111111111111111:sub":
"system:serviceaccount:name-space:name-of-the-service-account"
}
}
}
]
}
serviceAccount annotation is:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::non-prod-AccountID:role/non-prod-role-name
when running the command from inside the pod with the service account of the role in non-prod:
aws s3 cp hello.txt s3://test2/hello.txt
im having:
upload failed: ./hello.txt to s3://test2/hello.txt An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
by the way the cluster is in another account (devops account) if its related, surely added OIDC provider identity to both non-prod and prod accounts as identity provider.
If you're getting the error An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: No OpenIDConnect provider found in your account for $oidc_url when trying to cross-account assume roles, but you can assume roles in your cluster account normally, here's some points:
EKS ACCOUNT
Create a ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
name: $sa_name
namespace: $eks_ns
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::$resource_account_id:role/$role_name
Annotate your deployment
spec.template.spec:
serviceAccountName: $sa_name
Get info about your cluster OIDC Provider
aws iam get-open-id-connect-provider --open-id-connect-provider-arn arn:aws:iam::$eks_cluster_account_id:oidc-provider/$oidc_provider
3.1. The output will be like:
{
"Url": "...",
"ClientIDList": ["..."],
"ThumbprintList": ["..."],
"CreateDate": "...",
"Tags": [...]
}
3.2. Take note of the outputs (Url and ThumbprintList specially)
RESOURCE ACCOUNT
Add the provider (if you don`t have it already), using the output from your cluster account
aws iam create-open-id-connect-provider --url $oidc_url --client-id-list sts.amazonaws.com --thumbprint-list $oidc_thumbprint
This should be enought to the mentioned error stop. If you now get An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity, you're problably using the $eks_cluster_account_id on Principal.Federated, instead of $resource_account_id created on the previous step. So, make sure you're using the ARN from the IP that is assigned to the resource account, not the cluster account.
Create a role and a policy to access your resources with following trusted entities policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::$resource_account_id:oidc-provider/$oidc_provider"
},
"Action": "sts:AssumeRoleWithWebIdentity"
}
]
}
Also, there's no need to have two roles. One is enough.

Cannot create only IAM policy with cloudformation

I am having issue with creating IAM policy in cloudformation.But when I run it I get the error that Groups,Roles,Users is required:
Here is my code:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "AWS CloudFormation Template IAM Groups and Policies",
"Resources": {
"PolicyAutoScalingLimitedOperation": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyName": "AutoScaling-Limited-Operation",
"PolicyDocument": {
"Statement": [{
"Effect": "Allow",
"Action": [
"dynamodb:*"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"xray:PutTraceSegments",
"xray:PutTelemetryRecords"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*",
"s3:PutObject"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"logs:PutLogEvents",
"logs:CreateLogStream"
],
"Resource": "arn:aws:logs:*:*:log-group:/aws/elasticbeanstalk*"
},
{
"Effect": "Allow",
"Action": [
"kms:ListAliases",
"kms:ListKeys",
"kms:Encrypt",
"kms:Decrypt"
],
"Resource": "*"
}
]
}
}
}
}
}
Now when I run it I get:
At least one of [Groups,Roles,Users] must be non-empty.
Does that mean I cannot create policy with cloudformation without adding user/role to it?
You probably want to create an AWS::IAM::ManagedPolicy if you just want a standalone policy.
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-managedpolicy.html
From the documentation:
AWS::IAM::ManagedPolicy creates an AWS Identity and Access Management
(IAM) managed policy for your AWS account, which you can use to apply
permissions to IAM users, groups, and roles.
Here's an example:
Resources:
CreateTestDBPolicy:
Type: AWS::IAM::ManagedPolicy
Properties:
Description: "Policy for creating a test database"
Path: "/"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action: "rds:CreateDBInstance"
Resource: "*"
This will resolve your issue.