EC2 instance using the wrong user when executing PHP code - amazon-web-services

Question

I have a ec2 instance with a role attached to it. The role is called webserver and has all the relevant policies attached to it.
I am trying to invoke my lambda function from my PHP code, but I get the following error:
Failed attempt at deleting data/ account: exception 'Aws\Lambda\Exception\LambdaException' with message 'Error executing "Invoke" on "https://lambda.eu-west-2.amazonaws.com/2015-03-31/functions/blahFunction/invocations"; AWS HTTP error: Client error: `POST https://lambda.eu-west-2.amazonaws.com/2015-03-31/functions/blahFunction/invocations` resulted in a `403 Forbidden` response:
{"Message":"User: arn:aws:iam::34234324324342:user/SecretGuy is not authorized to perform: lambda:InvokeFunction on resour (truncated...)
AccessDeniedException (client): User: arn:aws:iam::34234324324342:user/SecretGuy is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:eu-west-2:34234324324342:function:blahFunction because no identity-based policy allows the lambda:InvokeFunction action - {"Message":"User: arn:aws:iam::34234324324342:user/SecretGuy is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:eu-west-2:34234324324342:function:blahFunction because no identity-based policy allows the lambda:InvokeFunction action"}'
Now SecretGuy is a user I created a long time ago, and somehow my ec2 is trying to use that.
I am wondering if anyone can help with this?
Thanks

Answer 1

From my understanding, you are running PHP code on an EC2 instance, and your code invokes the Lambda function.
And this EC2 instance has attached IAM Role with proper permissions to invoke the Lambda function. Then you tried to run your code and faced that the EC2 instance is using an unexpected IAM identity(IAM User named SecretGuy here), not the role you attached.
There's a chance that you might have IAM credentials set by environment variables for that Linux user or static credentials set on your EC2 instance.
AWS SDK client has an order to retrieve credentials on the machine. The official docs of PHP AWS SDK say,
When you initialize a new service client without providing any credential arguments, the SDK uses the default credential provider chain to find AWS credentials. The SDK uses the first provider in the chain that returns credentials without an error.
The default provider chain looks for and uses credentials as follows, in this order:
Use credentials from environment variables.
Setting environment variables is useful if you're doing development work on a machine other than an Amazon EC2 instance.
Use the AWS shared credentials file and profiles.
This credentials file is the same one used by other SDKs and the AWS CLI. If you're already using a shared credentials file, you can use that file for this purpose.
We use this method in most of our PHP code examples.
Assume an IAM role.
IAM roles provide applications on the instance with temporary security credentials to make AWS calls. For example, IAM roles offer an easy way to distribute and manage credentials on multiple Amazon EC2 instances.
To retrieve IAM credentials from the role attached,
You can check which IAM Identity you are using to call AWS API with the below command on the EC2 instance. (as that Linux user, you are running PHP code)
aws sts get-caller-identity
Then it will show result as below,
{
"UserId": "ABCDEFGHIJKLMNOPQRSTU",
"Account": "34234324324342",
"Arn": "arn:aws:iam:: 34234324324342:user/SecretGuy"
}
Then you need to look for environment variables set or static credentials files on that EC2 instance.
My guess is maybe someone used aws CLI on that EC2 before, with SecretGuy credentials, So there would be a file $HOME/.aws/credentials.
If the file exists and is confirmed as a SecretGuy access key, you have to delete that file. (If EC2 runs some critical application, you might want to copy all permissions of SecretGuy to the IAM Role you attached before deleting it to avoid unexpected service outage)
Or, you can look for environment variables.
echo $AWS_ACCESS_KEY_ID
If the above commands return the access key id value, you might have to unset environment variables.
After that, your code will retrieve credentials from IAM Role.