CloudWatch Log group missing although CloudWatch agent is working - amazon-web-services

I can't see the Log group defined by Cloud Watch agent on my EC2 instance
Also, the default log group /var/log/messages is not visible.
I can't see these logs also on root account.
I have other log groups configured and visible.
I have the following setup:
Amazon Linux
AMI managed role attached to instance: CloudWatchAgentServerPolicy
Agent installed via awslogs - https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html
Agent started successfully
No errors in /var/log/awslogs.log. Looks like working normally. Log below.
Configuration done via /etc/awslogs/config/FlaskAppAccessLogs.conf
Instance has outbound access to internet
Instance security groups allows all outbound traffic
Any ideas what to check or what can be missing?
/etc/awslogs/config/FlaskAppAccessLogs.conf:
cat /etc/awslogs/config/FlaskAppAccessLogs.conf
[/var/log/nginx/access.log]
initial_position = start_of_file
file = /var/log/nginx/access.log
datetime_format = %d/%b/%Y:%H:%M:%S %z
buffer_duration = 5000
log_group_name = FlaskApp-Frontends-access-log
log_stream_name = {instance_id}
/var/log/awslogs.log
2019-01-05 17:50:21,520 - cwlogs.push - INFO - 24838 - MainThread - Loading additional configs from /etc/awslogs/config/FlaskAppAccessLogs.conf
2019-01-05 17:50:21,520 - cwlogs.push - INFO - 24838 - MainThread - Missing or invalid value for use_gzip_http_content_encoding config. Defaulting to use gzip encoding.
2019-01-05 17:50:21,520 - cwlogs.push - INFO - 24838 - MainThread - Missing or invalid value for queue_size config. Defaulting to use 10
2019-01-05 17:50:21,520 - cwlogs.push - INFO - 24838 - MainThread - Using default logging configuration.
2019-01-05 17:50:21,544 - cwlogs.push.stream - INFO - 24838 - Thread-1 - Starting publisher for [c17fae93047ac481a4c95b578dd52f94, /var/log/messages]
2019-01-05 17:50:21,550 - cwlogs.push.stream - INFO - 24838 - Thread-1 - Starting reader for [c17fae93047ac481a4c95b578dd52f94, /var/log/messages]
2019-01-05 17:50:21,551 - cwlogs.push.reader - INFO - 24838 - Thread-4 - Start reading file from 0.
2019-01-05 17:50:21,563 - cwlogs.push.stream - INFO - 24838 - Thread-1 - Starting publisher for [8ff79b6440ef7223cc4a59f18e5f3aef, /var/log/nginx/access.log]
2019-01-05 17:50:21,587 - cwlogs.push.stream - INFO - 24838 - Thread-1 - Starting reader for [8ff79b6440ef7223cc4a59f18e5f3aef, /var/log/nginx/access.log]
2019-01-05 17:50:21,588 - cwlogs.push.reader - INFO - 24838 - Thread-6 - Start reading file from 0.
2019-01-05 17:50:27,838 - cwlogs.push.publisher - WARNING - 24838 - Thread-5 - Caught exception: An error occurred (ResourceNotFoundException) when calling the PutLogEvents operation: The specified log group does not exist.
2019-01-05 17:50:27,839 - cwlogs.push.batch - INFO - 24838 - Thread-5 - Creating log group FlaskApp-Frontends-access-log.
2019-01-05 17:50:27,851 - cwlogs.push.publisher - WARNING - 24838 - Thread-3 - Caught exception: An error occurred (ResourceNotFoundException) when calling the PutLogEvents operation: The specified log group does not exist.
2019-01-05 17:50:27,851 - cwlogs.push.batch - INFO - 24838 - Thread-3 - Creating log group /var/log/messages.
2019-01-05 17:50:27,966 - cwlogs.push.batch - INFO - 24838 - Thread-5 - Creating log stream i-0d7e533f67870ff8d.
2019-01-05 17:50:27,980 - cwlogs.push.batch - INFO - 24838 - Thread-3 - Creating log stream i-0d7e533f67870ff8d.
2019-01-05 17:50:28,077 - cwlogs.push.publisher - INFO - 24838 - Thread-5 - Log group: FlaskApp-Frontends-access-log, log stream: i-0d7e533f67870ff8d, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1546688052000, 'start_position': 0L, 'end_position': 161L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1546708885000, 'start_position': 4276L, 'end_position': 4468L}, 'source_id': '8ff79b6440ef7223cc4a59f18e5f3aef', 'num_of_events': 24, 'batch_size_in_bytes': 5068}
Status of awslogs
sudo service awslogs status
awslogs (pid 25229) is running...
IAM role policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData",
"ec2:DescribeTags",
"logs:PutLogEvents",
"logs:DescribeLogStreams",
"logs:DescribeLogGroups",
"logs:CreateLogStream",
"logs:CreateLogGroup"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ssm:GetParameter"
],
"Resource": "arn:aws:ssm:*:*:parameter/AmazonCloudWatch-*"
}
]
}

It's seems that posting a question may quickly help to find an answer.
There is additional configuration in which i have made typo:
sudo cat /etc/awslogs/awscli.conf
[plugins]
cwlogs = cwlogs
[default]
region = us-west-1
As described above the logs are delivered to us-west-1 region.
I was checking us-west-2 :)

Related

UpdateAutoScalingGroup permission error with boto3

i am getting error while running this script :
def set_asg_launch_template_version_latest(asg_name, lt_id):
try:
response = client.update_auto_scaling_group(
AutoScalingGroupName=asg_name,
LaunchTemplate={
'LaunchTemplateId': lt_id,
'Version': '$Latest'
}
)
print("Set launch template: {} version for asg: {} to $Latest".format(lt_id, asg_name))
return response
except ClientError as e:
print('Error setting launch template version to $Latest')
raise e
set_asg_launch_template_version_latest(ASGName,launch_template_id )
============>
ClientError - An error occurred (AccessDenied) when calling the UpdateAutoScalingGroup operation: You are not authorized to use launch template: lt-xxxxxxxxx
hint :
all these permissions are authorized (check last one):
- resource-groups:ListGroupResources
- tag:GetResources
- s3:PutObject
- s3:PutObjectAcl
- s3:List*
- ec2:Describe*
- ec2:CreateSnapshot
- ec2:CreateImage
- kms:CreateGrant
- ec2:StartInstances
- ec2:RunInstances
- ec2:TerminateInstances
- autoscaling:StartInstanceRefresh
- ec2:CreateSecurityGroup
- ec2:AuthorizeSecurityGroupEgress
- ec2:DeleteSecurityGroup
- ec2:RevokeSecurityGroupEgress
- ec2:ModifyLaunchTemplate
- ec2:CreateLaunchTemplateVersion
- autoscaling:Describe*
- ec2:DescribeLaunchTemplateVersions
- ec2:DescribeLaunchTemplates
- autoscaling:UpdateAutoScalingGroup

Is anyone else have issue deleting AWS Glacier archives?

I've been following the instructions on the AWS Docs https://docs.aws.amazon.com/amazonglacier/latest/dev/getting-started-delete-archive-cli.html and it works up until the last command where I delete the archive by its ID.
The command executes fine but the archive still in the vault, is there a time delay between aws glacier delete-archive and the archive actually getting deleted?
aws --debug glacier delete-archive --vault-name test-vault --account-id 1234567890 --archive-id "XIEYSpXAEcTO27fXXREDyb4BssF-nZPi8sIIyvqqfDFXkO4zBdeFfPRBgqmXaRJPGwmpdllkwWpBRtNocrVZv4Q6yJLwCDym6wELMsVw_ZDQq9_WnUE6ufgPT0Le-364s-HGyh47Qw"
2021-02-20 16:37:15,290 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=DeleteArchive) with params: {'url_path': '/1234567890/vaults/test-vault/archives/XIEYSpXAEcTO27fXXREDyb4BssF-nZPi8sIIyvqqfDFXkO4zBdeFfPRBgqmXaRJPGwmpdllkwWpBRtNocrVZv4Q6yJLwCDym6wELMsVw_ZDQq9_WnUE6ufgPT0Le-364s-HGyh47Qw', 'query_string': {}, 'method': 'DELETE', 'headers': {'User-Agent': 'aws-cli/2.0.38 Python/3.7.4 Darwin/20.3.0 exe/x86_64 command/glacier.delete-archive', 'x-amz-glacier-version': '2012-06-01'}, 'body': b'', 'url': ' 'context': {'client_region': 'eu-west-1', 'client_config': <botocore.config.Config object at 0x7fd5e7a9d8d0>, 'has_streaming_input': False, 'auth_type': None}}
2021-02-20 16:37:15,290 - MainThread - botocore.hooks - DEBUG - Event request-created.glacier.DeleteArchive: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fd5e7a9d790>>
2021-02-20 16:37:15,291 - MainThread - botocore.hooks - DEBUG - Event choose-signer.glacier.DeleteArchive: calling handler <function set_operation_specific_signer at 0x7fd5e67118c0>
2021-02-20 16:37:15,291 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2021-02-20 16:37:15,291 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
DELETE
/1234567890/vaults/test-vault/archives/XIEYSpXAEcTO27fXXREDyb4BssF-nZPi8sIIyvqqfDFXkO4zBdeFfPRBgqmXaRJPGwmpdllkwWpBRtNocrVZv4Q6yJLwCDym6wELMsVw_ZDQq9_WnUE6ufgPT0Le-364s-HGyh47Qw
host:glacier.eu-west-1.amazonaws.com
x-amz-date:20210220T163715Z
x-amz-glacier-version:2012-06-01
host;x-amz-date;x-amz-glacier-version
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2021-02-20 16:37:15,291 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20210220T163715Z
20210220/eu-west-1/glacier/aws4_request
3ac54f35e66fba13673a078b721c5db02a9f00c538bd96fc5b2d5b5709926b75
2021-02-20 16:37:15,292 - MainThread - botocore.auth - DEBUG - Signature:
1a09aef04ae7830ae5a7623cedb6ca69bf8cdef7349e3f3493a116967ba03110
2021-02-20 16:37:15,292 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=DELETE, url=https://glacier.eu-west-1.amazonaws.com/1234567890/vaults/test-vault/archives/XIEYSpXAEcTO27fXXREDyb4BssF-nZPi8sIIyvqqfDFXkO4zBdeFfPRBgqmXaRJPGwmpdllkwWpBRtNocrVZv4Q6yJLwCDym6wELMsVw_ZDQq9_WnUE6ufgPT0Le-364s-HGyh47Qw, headers={'User-Agent': b'aws-cli/2.0.38 Python/3.7.4 Darwin/20.3.0 exe/x86_64 command/glacier.delete-archive', 'x-amz-glacier-version': b'2012-06-01', 'X-Amz-Date': b'20210220T163715Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=<Access Key Removed>/20210220/eu-west-1/glacier/aws4_request, SignedHeaders=host;x-amz-date;x-amz-glacier-version, Signature=1a09aef04ae7830ae5a7623cedb6ca69bf8cdef7349e3f3493a116967ba03110', 'Content-Length': '0'}>
2021-02-20 16:37:15,294 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): glacier.eu-west-1.amazonaws.com:443
2021-02-20 16:37:15,510 - MainThread - urllib3.connectionpool - DEBUG - https://glacier.eu-west-1.amazonaws.com:443 "DELETE /1234567890/vaults/test-vault/archives/XIEYSpXAEcTO27fXXREDyb4BssF-nZPi8sIIyvqqfDFXkO4zBdeFfPRBgqmXaRJPGwmpdllkwWpBRtNocrVZv4Q6yJLwCDym6wELMsVw_ZDQq9_WnUE6ufgPT0Le-364s-HGyh47Qw HTTP/1.1" 204 0
2021-02-20 16:37:15,511 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': 'mR_DR8mtULOgCU0FDDv_aec7eYp34-QtXBONWBtWXjH1yts', 'Date': 'Sat, 20 Feb 2021 16:37:15 GMT'}
2021-02-20 16:37:15,511 - MainThread - botocore.parsers - DEBUG - Response body:
b''
2021-02-20 16:37:15,511 - MainThread - botocore.hooks - DEBUG - Event needs-retry.glacier.DeleteArchive: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fd5e7aef190>>
2021-02-20 16:37:15,512 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2021-02-20 16:37:15,512 - MainThread - botocore.hooks - DEBUG - Event after-call.glacier.DeleteArchive: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fd5e7a9dc50>>
2021-02-20 16:37:15,521 - MainThread - awscli.formatter - DEBUG - RequestId: mR_DR8mtULOgCU0FDDv_aec7eYp34-QtXBONWBtWXjH1yts
It's been over an hour since I executed the delete command and the vault still have the archive on it.
Has anyone had this issue? or does anyone know how I can delete a vault fully?
Update
The console allows me to delete but fails with the vault not being empty.
ps: I've posted this on AWS Forum as well but that's always slow at getting answers, so I'm trying my luck here. https://forums.aws.amazon.com/thread.jspa?threadID=335827
Per Deleting an Archive in Amazon S3 Glacier:
After you delete an archive, if you immediately download the vault
inventory, it might include the deleted archive in the list because S3
Glacier prepares vault inventory only about once a day.

Injecting HLS stream into AWS Elemental MediaPackage from WOWZA

I'm following https://github.com/WowzaMediaSystems/wse-example-pushpublish-hls in order to inject an HLS stream from a Wowza into a AWS MediaPackage channel.
My PushPublishProfilesCustom.xml
<?xml version="1.0" encoding="UTF-8"?>
<Root>
<PushPublishProfiles>
<PushPublishProfile>
<Name>cupertino-file</Name>
<Protocol>HTTP</Protocol>
<BaseClass>com.mycompany.wms.example.pushpublish.protocol.cupertino.PushPublishHTTPCupertinoFileHandler</BaseClass>
<Implementation>
<Name>Cupertino File</Name>
</Implementation>
<HTTPConfiguration>
</HTTPConfiguration>
<Properties>
</Properties>
</PushPublishProfile>
<PushPublishProfile>
<Name>cupertino-http</Name>
<Protocol>HTTP</Protocol>
<BaseClass>com.mycompany.wms.example.pushpublish.protocol.cupertino.PushPublishHTTPCupertinoHTTPHandler</BaseClass>
<Implementation>
<Name>Cupertino HTTP</Name>
</Implementation>
<HTTPConfiguration>
</HTTPConfiguration>
<Properties>
</Properties>
</PushPublishProfile>
</PushPublishProfiles>
</Root>
My #APP_NAME#/PushPublishMap.txt (I'm adding EndOfLines to do reading easier)
MediaPackage={
"entryName":"MediaPackage",
"profile":"cupertino-http",
"streamName":"MediaPackageStream",
"destinationName":"MediaPackage0",
"host":"xxxx.mediapackage.eu-west-1.amazonaws.com/in/v2/xxxx/xxxx/channel",
"port":"443",
"sendSSL":"true",
"username":"xxxx,
"password":"xxxx",
"http.path":"hls"
}
When I'm sending data to my wowza ( rtsp://X.X.X.X:1935/#APP_NAME#/MediaPackage ) I start to see logs like this...
WARN server comment 2020-06-02 09:23:49 - - - - - 4325.922 - - - - - - - - PushPublishHTTPCupertinoHTTPHandler.outputSend([MediaPackage] TV/_definst_/MediaPackage->MediaPackageStream) Found 79 segments to send
WARN server comment 2020-06-02 09:23:49 - - - - - 4325.922 - - - - - - - - PushPublishHTTPCupertinoHTTPHandler.outputSend([MediaPackage] TV/_definst_/MediaPackage->MediaPackageStream) Found 76 segments to delete
ERROR server comment 2020-06-02 09:23:49 - - - - - 4325.934 - - - - - - - - PushPublishHTTPCupertinoHTTPHandler.outputSend([MediaPackage] TV/_definst_/MediaPackage->MediaPackageStream) Send media segment. rendition: AUDIOVIDEO chunkId:77 uri:pdmekxw9/media_77.aac result:FAILURE
So, HLS Push Publishing is sending chunks... but without success
I have read https://www.wowza.com/docs/how-to-configure-apple-hls-packetization-cupertinostreaming but I don't know what values I may change.
What am I doing wrong? Any ideas?
EDIT: More logs
2020-06-02 14:32:39 UTC comment server INFO 200 - PushPublishHTTPCupertinoHTTPHandler.createOutputItem([MediaPackage] TV/_definst_/MediaPackage->MediaPackageStream) chunkCount:10, chunkStartIndex:201, lastChunkIndex:209 - - -22856.082 - - - - - - - - - - - - - - - - - - - - - - - - -
2020-06-02 14:32:39 UTC comment server INFO 200 - PushPublishHTTPCupertinoHTTPHandler.createOutputItem([MediaPackage] TV/_definst_/MediaPackage->MediaPackageStream) playlistChunkCount:3, playlistChunkStartIndex:208 - - - 22856.082 - - - - - - - - - - - - - - - - - - - - - - - - -
2020-06-02 14:32:39 UTC comment server INFO 200 - PushPublishHTTPCupertinoHTTPHandler.createOutputItem([MediaPackage] TV/_definst_/MediaPackage->MediaPackageStream) New chunk: chunkRendition:AUDIOVIDEO, chunkId:210, chunkIndex:2 - -- 22856.082 - - - - - - - - - - - - - - - - - - - - - - - - -
2020-06-02 14:32:39 UTC comment server INFO 200 - PushPublishHTTPCupertinoHTTPHandler.createOutputItem([MediaPackage] TV/_definst_/MediaPackage->MediaPackageStream) Marking MediaSegmentModel: pcnod08j/media_207.aac for deletion - -- 22856.083 - - - - - - - - - - - - - - - - - - - - - - - - -
2020-06-02 14:32:39 UTC comment server WARN 200 - PushPublishHTTPCupertinoHTTPHandler.outputSend([MediaPackage] TV/_definst_/MediaPackage->MediaPackageStream) Found 32 segments to send - - - 22856.083 - - -- - - - - - - - - - - - - - - - - - - - - -
2020-06-02 14:32:39 UTC comment server WARN 200 - PushPublishHTTPCupertinoHTTPHandler.outputSend([MediaPackage] TV/_definst_/MediaPackage->MediaPackageStream) Found 29 segments to delete - - - 22856.083 - -- - - - - - - - - - - - - - - - - - - - - - -
2020-06-02 14:32:39 UTC comment server ERROR 500 - PushPublishHTTPCupertinoHTTPHandler.outputSend([MediaPackage] TV/_definst_/MediaPackage->MediaPackageStream) Send media segment. rendition: AUDIOVIDEO chunkId:208 uri:pcnod08j/media_208.aac result:FAILURE - - - 22856.097

How to access S3 with profile on EC2? (sts.amazonaws.com:443 error)

I have EC2 and can access S3 using ec2-user with:
aws s3 ls --profile myprofile s3://
On the same EC2 I created docker container and installed aws cli tool. Created same ~./aws/config file with same content as on host ec2 but this command (created ec2-user to be consistent with host user/setup) from docker container now gives error:
- MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
- MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://sts.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/1.16.190 Python/3.7.3 Linux/3.10.0-957.10.1.el7.x86_64 botocore/1.12.180', 'X-Amz-Date': b'2019Z', 'X-Amz-Security-Token': b'1234=', 'Authorization': b'AWS4-HMAC-SHA256 Credential=abc/2019/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=abc', 'Content-Length': '160'}>
- MainThread - urllib3.util.retry - DEBUG - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
- MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): sts.amazonaws.com:443

AWS CLI on instance with ECR IAM Policy Set

i have an IAM role that i have tested with the simulator to provide AmazonEC2ContainerRegistryReadOnly access. I have launched an ec2 with the role, and i can see in the EC2 console that it is attached to the instance. When i SSH into the EC2, and attempt to run
aws ecr get-authorization-token
I get the message
'AccessKeyId'
I have attempted to do "aws configure" and set the default region and output (leaving the ACCESS and SECRET empty) but still get the same result...
Can anyone help with this please?
Screenshot of 'error'
DEBUG -
[ec2-user#ip-10-0-101-105 ~]$ aws ecr get-authorization-token
'AccessKeyId'
[ec2-user#ip-10-0-101-105 ~]$ aws ecr get-authorization-token --debug
2017-02-01 15:03:00,704 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.11.44 Python/2.7.12 Linux/4.4.41-36.55.amzn1.x86_64 botocore/1.5.7
2017-02-01 15:03:00,704 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['ecr', 'get-authorization-token', '--debug']
2017-02-01 15:03:00,704 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7efd7abe4578>
2017-02-01 15:03:00,704 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7efd7b516c80>
2017-02-01 15:03:00,705 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python2.7/site-packages/botocore/data/ecr/2015-09-21/service-2.json
2017-02-01 15:03:00,712 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.ecr: calling handler <function register_retries_for_service at 0x7efd7be11488>
2017-02-01 15:03:00,712 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: ecr
2017-02-01 15:03:00,713 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ecr: calling handler <function _inject_get_login at 0x7efd7acff1b8>
2017-02-01 15:03:00,713 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ecr: calling handler <function add_waiters at 0x7efd7abe8938>
2017-02-01 15:03:00,716 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'registry-ids', <awscli.arguments.ListArgument object at 0x7efd7a87d9d0>)])
2017-02-01 15:03:00,716 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecr.get-authorization-token: calling handler <function add_streaming_output_arg at 0x7efd7abe4b90>
2017-02-01 15:03:00,716 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecr.get-authorization-token: calling handler <function add_cli_input_json at 0x7efd7b520b90>
2017-02-01 15:03:00,717 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecr.get-authorization-token: calling handler <function unify_paging_params at 0x7efd7ac735f0>
2017-02-01 15:03:00,719 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python2.7/site-packages/botocore/data/ecr/2015-09-21/paginators-1.json
2017-02-01 15:03:00,719 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecr.get-authorization-token: calling handler <function add_generate_skeleton at 0x7efd7ac5d9b0>
2017-02-01 15:03:00,719 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ecr.get-authorization-token: calling handler <bound method CliInputJSONArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7efd7a87da10>>
2017-02-01 15:03:00,719 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ecr.get-authorization-token: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7efd7a85a750>>
2017-02-01 15:03:00,720 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecr.get-authorization-token.registry-ids: calling handler <function uri_param at 0x7efd7b53aaa0>
2017-02-01 15:03:00,720 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecr.get-authorization-token.cli-input-json: calling handler <function uri_param at 0x7efd7b53aaa0>
2017-02-01 15:03:00,720 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecr.get-authorization-token.generate-cli-skeleton: calling handler <function uri_param at 0x7efd7b53aaa0>
2017-02-01 15:03:00,721 - MainThread - botocore.hooks - DEBUG - Event calling-command.ecr.get-authorization-token: calling handler <bound method GenerateCliSkeletonArgument.generate_json_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7efd7a85a750>>
2017-02-01 15:03:00,721 - MainThread - botocore.hooks - DEBUG - Event calling-command.ecr.get-authorization-token: calling handler <bound method CliInputJSONArgument.add_to_call_parameters of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7efd7a87da10>>
2017-02-01 15:03:00,721 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2017-02-01 15:03:00,721 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2017-02-01 15:03:00,721 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2017-02-01 15:03:00,721 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: config-file
2017-02-01 15:03:00,722 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: ec2-credentials-file
2017-02-01 15:03:00,722 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: boto-config
2017-02-01 15:03:00,722 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: container-role
2017-02-01 15:03:00,722 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: iam-role
2017-02-01 15:03:00,725 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): 169.254.169.254
2017-02-01 15:03:00,726 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 11
2017-02-01 15:03:00,727 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): 169.254.169.254
2017-02-01 15:03:00,728 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /latest/meta-data/iam/security-credentials/jenkins-DEV HTTP/1.1" 200 255
2017-02-01 15:03:00,729 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 197, in main
return command_table[parsed_args.command](remaining, parsed_args)
File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 333, in __call__
return command_table[parsed_args.operation](remaining, parsed_globals)
File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 503, in __call__
call_parameters, parsed_globals)
File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 620, in invoke
verify=parsed_globals.verify_ssl)
File "/usr/local/lib/python2.7/site-packages/botocore/session.py", line 825, in create_client
credentials = self.get_credentials()
File "/usr/local/lib/python2.7/site-packages/botocore/session.py", line 449, in get_credentials
'credential_provider').load_credentials()
File "/usr/local/lib/python2.7/site-packages/botocore/credentials.py", line 1083, in load_credentials
creds = provider.load()
File "/usr/local/lib/python2.7/site-packages/botocore/credentials.py", line 488, in load
metadata = fetcher.retrieve_iam_role_credentials()
File "/usr/local/lib/python2.7/site-packages/botocore/utils.py", line 203, in retrieve_iam_role_credentials
'access_key': data[role_name]['AccessKeyId'],
KeyError: 'AccessKeyId'
2017-02-01 15:03:00,735 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255
'AccessKeyId'
EDIT:
Found that the trust policy was set up incorrectly..
Adding an answer for posterity sake so others can see what to do if they hit this error. An EC2 instance profile (an IAM role) needs to allow the EC2 service to assume the role, so you'll need a trust policy like this one:
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}