AWS Error retrieving credentials from the instance profile metadata server - amazon-web-services

Question

Can someone please give me an insight on this error? Credentials are set via IAM policy. This box is included in auto scaling group, and this is the only one that got the following error.
Error retrieving credentials from the instance profile metadata server. When you are not running inside of Amazon EC2, you must provide your AWS access key ID and secret access key in the \"key\" and \"secret\" options when creating a client or provide an instantiated Aws\Common\Credentials\CredentialsInterface object"
Logs:
CRITICAL
Phalconry\Mvc\Exceptions\ServerException
Extra
"remoteip": "XX.XX.XX.XX, XX.XX.XX.XX",
"userid": "1357416",
"session": "fcke8khsqe4lfo2lj6kdmrd4l7",
"url": "GET:\/manage-competition\/athlete",
"request_identifier": "xxxxxx5c80516bc11532.74367732",
"server": "companydomain.com",
"client_agent": "Mozilla\/5.0 (Linux; Android 9; SM-G965U) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/72.0.3626.121 Mobile Safari\/537.36",
"instance_ip_address": "xx.xx.xx.xx",
"process_id": 29528,
"file": "\/var\/www\/code_deploy\/cfweb\/releases\/20190306195438\/core\/classes\/phalconry\/mvc\/exceptions\/MvcException.php",
"line": 51,
"class": "Phalconry\\Mvc\\Exceptions\\MvcException",
"function": "dispatch"
}```
Context
```{
"Status Code": 500,
"Reason": "Internal Server Error",
"Details": "Array\n(\n [code] => server_error\n [description] => Uncaught Server Error\n [details] => Request could not be processed. Please contact Support.\n)\n",
"Log": "Error retrieving credentials from the instance profile metadata server. When you are not running inside of Amazon EC2, you must provide your AWS access key ID and secret access key in the \"key\" and \"secret\" options when creating a client or provide an instantiated Aws\\Common\\Credentials\\CredentialsInterface object. (Unable to parse response body into JSON: 4)",
"Trace": "#0 [internal function]: Phalconry\\Mvc\\MvcApplication::Phalconry\\Mvc\\{closure}(Object(Phalcon\\Events\\Event), Object(Phalcon\\Mvc\\Dispatcher), Object(Aws\\Common\\Exception\\InstanceProfileCredentialsException))\n#1 [internal function]: Phalcon\\Events\\Manager->fireQueue(Array, Object(Phalcon\\Events\\Event))\n#2 [internal function]: Phalcon\\Events\\Manager->fire('dispatch:before...', Object(Phalcon\\Mvc\\Dispatcher), Object(Aws\\Common\\Exception\\InstanceProfileCredentialsException))\n#3 [internal function]: Phalcon\\Mvc\\Dispatcher->_handleException(Object(Aws\\Common\\Exception\\InstanceProfileCredentialsException))\n#4 [internal function]: Phalcon\\Dispatcher->dispatch()\n#5 \/var\/www\/code_deploy\/cfweb\/releases\/20190306195438\/sites\/games\/lib\/phalcon.php(101): Phalcon\\Mvc\\Application->handle()\n#6 \/var\/www\/code_deploy\/cfweb\/releases\/20190306195438\/sites\/games\/index.php(4): require_once('\/var\/www\/code_d...')\n#7 {main}"
}```

Answer 1

The meta-data of instance-profile credentials is under
http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance
If that is failing it might be an issue with the hypervisor/droplet that your server has been spun up on. This endpoint will give you the last time credentials were refreshed.
'http://169.254.169.254/latest/meta-data/identity-credentials/ec2/info'
If other servers with identical AMI's and availability zones aren't having an issue, I would log a support ticket, terminate and move on.

Answer 2

These credentials come from the instance metadata URL (http://169.254.169.254).
The problem that you are likely having (esp since you are running in an ASG) is that when you create an AMI and then launch it in another AZ, the route to the metadata URL is not updated. What you need to do is force cloud-init to run on next boot.
The simple way to do this is to clear out the cloud-init metadata directory:
sudo rm -f /var/lib/cloud/instances/*/sem/config_scripts_user
After you run that command, then shutdown the machine and create an AMI from it. If you use that AMI for your ASG, cloud-init will do a full run on first boot which will update the routes to the instance metadata URL and your IAM creds should work.