AWS Systems Manager (SSM) fails to UpdateInstanceInformation on SageMaker instance - amazon-web-services

Question

I often get Cloudwatch Authorization alerts because the role attached to my SageMaker instance doesn't seem to have enough SSM (Systems Manager) permissions to UpdateInstanceInformation. My understanding is that the agent amazon-ssm-agent wants to hit an AWS API but fails to do so.
My Role has full SSM permissions:
{
"Action": [
"ssm:*",
"ssmmessages:*"
],
"Resource": "*",
"Effect": "Allow"
}
but the error persists:
{
"eventVersion": "1.05",
"userIdentity": {
"type": "AssumedRole",
"principalId": "XXXXXXXXXXXXX:SageMaker",
"arn": "arn:aws:sts::XXXXXXXXXXXXX:assumed-role/sagemaker_prod_Notebook_Instance_Role/SageMaker",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "XXXXXXXXXXXXX",
"arn": "arn:aws:iam::XXXXXXXXXXXXX:role/sagemaker_prod_Notebook_Instance_Role",
"accountId": "XXXXXXXXXXXXX",
"userName": "sagemaker_prod_Notebook_Instance_Role"
}
},
"invokedBy": "im.amazonaws.com"
},
"eventSource": "ssm.amazonaws.com",
"eventName": "UpdateInstanceInformation",
"sourceIPAddress": "im.amazonaws.com",
"userAgent": "im.amazonaws.com",
"errorCode": "AccessDenied",
"errorMessage": "An unknown error occurred",
"requestParameters": {
"instanceId": "i-045f627a2d2e469b1",
"agentVersion": "2.3.714.0",
"platformType": "Linux",
"agentName": "amazon-ssm-agent"
},
"eventType": "AwsApiCall"
}
Has anyone seen this before ?

Answer 1

This is a bit late but I had a similar issue so I reached out to AWS Support and it seems to be a somewhat of a bug.
I was told that the AWS Sagemaker team has ssm installed by default. The Sagemaker notebook runs in an aws service account, although when a customer assigns Sagemaker a role in their own account the role cannot perform UpdateInstance information via the customer assigned role.
Support suggested I create a lifecycle config and leverage the following code sample to fix it:
https://docs.aws.amazon.com/sagemaker/latest/dg/notebook-lifecycle-config.html
https://github.com/aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples/blob/master/scripts/disable-uninstall-ssm-agent/on-start.sh