Below are the steps that I performed for setup :
Created a new windows server 2012 EC2 instance and then connected
with it using the remote desktop connection.
Installed EC2 config service which automatically installs
AmazonSSMAgent service.
Both EC2 and SSM services are running and in error logs, it is throwing some 'no chain provider exception' which is fine because I have not provided aws credentials yet.
Created a new IAM user and give him administrative policy access and
created three environment variables providing access_key,
secret_access_key and region_id.
Then I restarted AmazonSSMAgent service, then the exception changed to below exception:
status code: 400, request id: 400b4c75-eae2-11e7-a120-d33083d55d1f 2017-12-27 08:45:13 ERROR [HandleAwsError # awserr.go.48] [instanceID=i-084646fb32d8a7b6d] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AuthorizationFailureException:
Please suggest what i am doing wrong or any steps that i have missed or any other way to do this.
Related
I am trying to create a default Composer 2 Instance on GCP and get the Errors:
CREATE operation on this environment failed 32 minutes ago with the following error message:
Composer Backend timed out. Currently running tasks are [stage: CP_COMPOSER_AGENT_RUNNING
description: "No agent response published."
...
or
CREATE operation on this environment failed 32 minutes ago with the following error message:
Environment couldn't be created, but no error was surfaced. This can be cause by a lack of
proper permissions. Check if this environment's service account ... .iam.gserviceaccount.com
has the 'roles/composer.worker' role and there is no firewall inhibiting internal
communications set.
I already tried to add the Composer Worker role to the service account and all other required roles (e.g. Cloud Composer v2 API Service Agent Extension) in https://cloud.google.com/composer/docs/composer-2/access-control (for public as well as for private, eventhough instance is public).
I looked into the GKE instance and found the Pod composer-agent failing with:
Traceback (most recent call last): File "composer_agent.py", line 467, in <module> main() File "composer_agent.py", line 292, in main responses = pubsub_subscriber.pull() (...)
oauth2client.client.HttpAccessTokenRefreshError: Failed to retrieve
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/(...)
compute#developer.gserviceaccount.com/token from the Google Compute Enginemetadata service.
Response:
{'date': 'Thu, 17 Feb 2022 10:29:46 GMT', 'status': '403', 'content-length': '668', 'content-
type': 'text/plain; charset=utf-8', 'x-content-type-options': 'nosniff'}
So I assume there is still some permission issue but I can not figure out what, Composer 1 Instances can be created without a problem, as well for a different project Composer 2 Instances with the same permissions on the service accounts.
I also tried to create different than default compute service account with the required permissions but also without success. I also checked that the service account I am adding permissions to is actually the service account sending the request in the composer-agent and is sending the environment creation request to the GKE cluster.
I hope anyone can help, who faced similar issues or knows more about the error occuring in composer-agent, thank you very much!
After being in contact with the Google Support Team, the solution was to manually enable the "IAM Service Account Credentials API". There was no issue in Service Account Rights or Firewall settings.
I am trying to create basic Composer environment:
image version: 1.17.8/2.1.4
using service account with composer.worker permission
my own user has project.owner permission
public ip
All my attempts failed with following error:
Http error status code: 400
Http error message: BAD REQUEST
Errors in: [Web server]; Error messages:
The caller does not have permission
Required 'deploymentmanager.typeProviders.create' permission for 'projects/<my-project>/global/typeProviders/europe-west2-<name-id>-addons-gke-typer'
deploymentmanager.typeProviders.create is covered by Deployment Manager Type Editor, so I added this permission to both my account and service account, but the error remains the same.
Cloud Composer Service Agent account is present in the project without any modifications to its permissions.
Is there anything else I can check or something that I missed during the set up?
For an account (whether User Account or Service Account) to be able to create a Composer Environment, the account must have a composer.environments.create permission.
And according to Google Cloud's documentation on Cloud Composer Access Control,
The Composer Worker role provides the permissions necessary to run a Cloud Composer environment
VM and intended for service accounts.
The Composer Worker role is not intended for creation of environments thus, it does not have the composer.environments.create permission.
If you want your service account to be able to create a Composer environment, you will need to assign the role Composer Administrator and this has the composer.environments.create permission needed.
You may refer to Access Control for Cloud Composer for the complete list of permission for Composer Worker, Composer Administrator and other Composer related roles.
On Execution of command below from AWS Cloud9
copilot app delete
Getting message as :
execute svc delete: delete service: delete stack demoappone-test-lbdemoappone: InvalidClientTokenId: The security token included in the request is invalid
status code: 403, request id: 5001e2d0-3bbc-4821-911c-27713af4f2f9
AWS Cloud9 points to document https://docs.aws.amazon.com/cloud9/latest/user-guide/welcome.html .
Do we know how to solve above 403 error?
This appears to be because Cloud9 has a limited set of allowed actions that're supported for AWS managed temporary credentials. (For the troubleshooting guide: link)
The way that the ECS workshop gets around it is by creating and storing access credentials in the environment: sample solution. (For the Cloud9 guide: link)
I am using boto3 to control my EC2 instances on AWS from a python environment, using ec2 and ssm services. I have created an IAM account, that has access to AmazonSSMFullAccess and AmazonEC2FullAccess policies.
ec2 = boto3.client(
'ec2',
region_name='eu-west-1',
aws_access_key_id='…',
aws_secret_access_key='…/…+…'
)
ssm = boto3.client(
'ssm',
region_name='eu-west-1',
aws_access_key_id='…',
aws_secret_access_key='…/…+…'
)
I ran:
ec2.describe_instances()['Reservations']
Witch returned a list of all my instances.
But when I run:
ssm.describe_instance_information()
I get an empty list, though I have at least one instance running on AWS Linux AMI (ami-ca0135b3), and six others on recent Ubuntu AMIs. They are all in eu-west-1 (Ireland).
They should have SSM Agent preinstalled : (https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-install-ssm-agent.html)
I sshed into the AWS Linux instance, and tried to get the logs for ssm using:
sudo tail -f /var/log/amazon/ssm/amazon-ssm-agent.log
But nothing happens there when I run my python code. A sequence of messages gets displayed from time to time :
HealthCheck reporting agent health.
error when calling AWS APIs. error details - NoCredentialProviders: no valid providers in chain. Deprecated.
I also tried running a command through the web interface, selected ' AWS-RunRemoteScript' but no instance is shown below.
My goal is to run:
ssm.send_command(
DocumentName="AWS-RunShellScript",
Parameters={'commands': [command]},
InstanceIds=[instance_id],
)
But it gives me the following error, probably due to the previous problem.
botocore.errorfactory.InvalidInstanceId: An error occurred (InvalidInstanceId) when calling the SendCommand operation
The agent is pre-installed, but the instance (not just your IAM user) still needs the proper role to communicate with the systems manager. Particularly this step of Configuring Access to Systems Manager.
By default, Systems Manager doesn't have permission to perform actions
on your instances. You must grant access by using an IAM instance
profile. An instance profile is a container that passes IAM role
information to an Amazon EC2 instance at launch.
You should review the whole configuration guide and make sure you have configured all required roles appropriately.
I am trying to use the Amazon EC2 Run Command feature, and basically it says it can't find any instances.
I read all the documentation and followed all the steps that I could find on this relatively new feature, and I cant get it to work for the life of me.
Here is the info:
I go to "EC2 -> Commands -> Command History -> Run A Command"
I click radio box for the "AWS-RunShellScript" command document
I click the "Select Instances" drop down button, and it says "No instances found in this region" as shown in Figure 1 below.
I click the "Where are my instances?" hyperlink, which brings me to the AWS documentation show here:"Troubleshooting SSM Run Command"
The first suggestion for troubleshooting is to make sure that the prerequisites have been met, and provides another hyperlink shown here: "SSM Run Command Prerequisites"
Prerequisites:
Supported OS - Windows 2012 R2 - CHECK
Latest Agent Version - 3.17.1032 - CHECK (Shown In Figure 2)
Access to SSM Run Command - Assign EC2 Instance Role and IAM User Role - CHECK (Shown In Figures 3 and 4)
Internet Access - Outbound Internet Access - CHECK
I followed the AWS documentation for creating an Amazon EC2 Instance role for EC2 Run Command Access, and also to create an IAM User with proper Run Command Access. The documentation is shown here: Delegating Access to SSM Run Command
I used the Amazon CLI from my local machine to run the following command also shown in troubleshooting documentation:
aws ssm describe-instance-information --instance-information-filter-list key=InstanceIds,valueSet=instance-ID
The command returns no information:
{
"InstanceInformationList": []
}
UPDATE 1:
I did find the following errors and warnings in the following places, and have spent the last two hours trying to troubleshoot. I uinstalled and reinstalled the Ec2Config Service. I checked all the Ec2Config files. I Checked the 169.254 addresses, and the meta-data and dynamic data exists, and appears to be correct. Not sure what else to try here.
C:\Program Files\Amazon\Ec2ConfigService\Logs\Ec2ConfigLog - Error/Warning
2016-06-22T23:44:12.663Z: Warning: Unable to Publish to WMI. | System.Management.Instrumentation.WmiProviderInstallationException: Exception of type 'System.Management.Instrumentation.WMIInfraException' was thrown.
at System.Management.Instrumentation.InstrumentationManager.Publish(Object value)
2016-06-22T23:44:16.263Z: Failed to fetch instance metadata http://169.254.169.254/latest/user-data with exception The remote server returned an error: (404) Not Found.
2016-06-22T23:44:16.263Z: Failed to get metadata/user-data The remote server returned an error: (404) Not Found.
Windows Event Viewer Error
2016-06-22 23:46:59,758 [_Worker-2] ERROR [aws:getDocument] - Failed to update instance information., RequestId=9cb8f2dd-38d3-11e6-bc83-19c0650ffecc,ErrorCode=AccessDeniedException,ErrorType=Unknown,StatusCode=BadRequest,Message=Caller
instance identity does not match the given instanceId
UPDATE 2:
Per #praetorian1 's linked stackoverflow article I updated the Ec2Config Services "config.xml" file to enable the "Ec2HandleUserData" setting, rebooted, and still no change.
Per #Kai 's suggestion I also made sure that the "role" attached to the instance
matched the "AmazonEC2RoleforSSM" policy, and specifically permitted "ssm:UpdateInstanceInformation". It did.
I also looked under the AWS EC2 Management Console, at the "System Log" file for the specific instance, which did not contain any errors, and towards the end said that SSM Config: status:Active, iam:Yes. Also noteworthy, it contained: Info EC2Config configuration: status:2; region:us-east-1; iam:1; authz:1
2016/06/23 15:01:29Z: EC2ConfigMonitorState: 0
2016/06/23 15:01:29Z: Windows sysprep configuration complete.
2016/06/23 15:01:32Z: AMI Origin Version: 2016.01.13
2016/06/23 15:01:32Z: AMI Origin Name: Windows_Server-2012-R2_RTM-English-64Bit-SQL_2014_SP1_Web
2016/06/23 15:01:32Z: OS: Microsoft Windows NT 6.3.9600
2016/06/23 15:01:32Z: OsVersion: 6.3
2016/06/23 15:01:32Z: OsProductName: Windows Server 2012 R2 Standard
2016/06/23 15:01:32Z: OsBuildLabEx: 9600.18202.amd64fre.winblue_ltsb.160119-0600
2016/06/23 15:01:32Z: Language: en-US
2016/06/23 15:01:32Z: TimeZone: Eastern Standard Time
2016/06/23 15:01:32Z: Offset: UTC -04:00:00
2016/06/23 15:01:32Z: EC2 Agent: Ec2Config service v3.17.1032
2016/06/23 15:01:32Z: Driver: AWS PV Storage Host Adapter v7.3.2.0
2016/06/23 15:01:32Z: Driver: Intel(R) 82599 Virtual Function v1.0.15.3
2016/06/23 15:01:34Z: Message: Waiting for meta-data accessibility...
2016/06/23 15:01:34Z: Message: Meta-data is now available.
2016/06/23 15:01:37Z: Message: Windows is Ready to use
2016/06/23 15:01:40Z: Amazon EC2 Simple Systems Manager (SSM) is an optional service for custom configuration of instances.
2016/06/23 15:01:40Z: Info EC2Config configuration: status:2; region:us-east-1; iam:1; authz:1
2016/06/23 15:01:40Z: SSM Config: status:Active; iam:Yes
Figure 1:
Figure 2:
Figure 4:
Try checking the application logs in windows on the instance. Might be having issues talking to the SSM service for some reason.
As praetorian1 mentioned, Windows Event Viewer is the place to look for SSM - EC2 Run Command related log messages.
The message you posted indicates that the EC2 instance does not have required permission to call ssm:UpdateInstanceInformation API. Please double check that RunCommandInstance role has AmazonEC2RoleforSSM policy attached. (After attaching the policy, please restart EC2Config service or wait up to 15 min for EC2Config to start reporting instance information.)
For the IAM User in Fig. 4, we do not need policies other than AmazonSSMFullAccess for us to call SSM APIs with awscli.
To dive deeper:
EC2 Role and IAM Policy for the EC2 Instance
SSM on Windows requires an IAM EC2 Role attached to the EC2 instance at launch. Required permission for that Role is provided by an Amazon-managed IAM policy AmazonEC2RoleforSSM.
IAM Policy for IAM User
To make API calls to SSM service with awscli, an IAM user has to be setup with SSM permissions. For testing, Amazon provides a managed IAM policy AmazonSSMFullAccess. With the credentials of this IAM user, we will be able to make API call to describe instance information and also send command to the instance.
DescribeInstanceInformation API
Once the EC2 instance is setup with required permission and EC2Config starts reporting instance information back to SSM, we will be able to use DescribeInstanceInformation API to get a list of EC2 instances that are ready for SSM - EC2 Run Command.