I have created AWS DMS replication instance, replication task and source, target enpoints using terrform.
Now, when i run start replication task from windows aws cli . it throws this SSL error.
Error running command 'aws dms start-replication-task --start-replication-task-type start-replication --replication-task-arn arn:aws:dms:us-west-2:accountnumber:task:xxxxxxx': exit status 254. Output: C:\Program Files\Amazon\AWSCLIV2\urllib3\connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'dms.us-east-1.amazonaws.com'. Adding certificate verification is strongly advised.
My CLI version is The version is aws-cli/2.1.6 Python/3.7.9 Windows/10 exe/AMD64 prompt/off.
There is no proxy configured
Any suggestion on this issue.
Thnaks
Make sure you set your credential correctly where you replication task is living by set [default] on the right one
./aws/credentials
to start the replication task you have to fill two mandatory field
{
"ReplicationTaskArn": "string",
"StartReplicationTaskType": "string"
}
StartReplicationTaskType --> Valid Values: start-replication | resume-processing | reload-target
start-replication only when you create the task so you may need to make it reload-target
Ref:
https://docs.aws.amazon.com/dms/latest/APIReference/API_StartReplicationTask.html
Related
I am unable to call aws services from fargate tasks - secrets manager and sns.
I want these services to be invoked from inside the docker image which is hosted on ECR. When I run the pipeline everything loads and run correctly except when the script inside the docker container is invoked, it throws an error. The script makes a call to either secrets manager or sns. The error thrown is -
Unable to locate credentials. You can configure credentials by running "aws configure".
If I do aws configure then the error goes away and every things works smoothly. But I do not want to store the aws credentials anywhere.
When I open task definitions I can see two roles - pipeline-task and ecsTaskExecutionRole
Although, I have given full administrator rights to both of these roles, the pipeline still throws error. Is there any place missing where I can assign roles/policies etc. I want to completely avoid using aws configure.
If the script with the issue is not a PID 1 process ( used to stop and start the container ), it will not automatically read the Task Role (pipeline-task-role). From your description, this sounds like the case.
Add this to your Dockerfile:
RUN echo 'export $(strings /proc/1/environ | grep AWS_CONTAINER_CREDENTIALS_RELATIVE_URI)' >> /root/.profile
The AWS SDK from the script should know where to pick up the credentials from after that.
I don't know if my problem was the same as yours but I also experienced this kind of problem where I had been set the task role but the container don't get the right permissions. After spending a few days, I discovered that if you set any of the AWS environment variables
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
or AWS_DEFAULT_REGION
into the task definition, the "Credential provider chain" will stop at the "Static credentials" step, so the SDK your script is using will look for the other credentials within the .aws/.credentials file and as it can't find them it throws Unable to locate credentials.
If you want to know further about the Credential provider chain you could read about it in https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html
I am trying to update an ECS service with bamboo and get the following error:
Failed to fetch resource from AWS!
java.lang.RuntimeException: Expected DescribeServiceRequest for
service 'my-service' to return exactly 1 result, but got
'0' at
net.utoolity.atlassian.bamboo.taws.aws.ECS.getSingleService(ECS.java:674)
at
net.utoolity.atlassian.bamboo.taws.ECSServiceTask.executeUpdate(ECSServiceTask.java:311)
at
net.utoolity.atlassian.bamboo.taws.ECSServiceTask.execute(ECSServiceTask.java:133)
at
net.utoolity.atlassian.bamboo.taws.AWSTask.execute(AWSTask.java:164)
at
com.atlassian.bamboo.task.TaskExecutorImpl.lambda$executeTasks$3(TaskExecutorImpl.java:319)
at
com.atlassian.bamboo.task.TaskExecutorImpl.executeTaskWithPrePostActions(TaskExecutorImpl.java:252)
at
com.atlassian.bamboo.task.TaskExecutorImpl.executeTasks(TaskExecutorImpl.java:319)
at
com.atlassian.bamboo.task.TaskExecutorImpl.execute(TaskExecutorImpl.java:112)
at
com.atlassian.bamboo.build.pipeline.tasks.ExecuteBuildTask.call(ExecuteBuildTask.java:73)
at
com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.executeBuildPhase(DefaultBuildAgent.java:203)
at
com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.build(DefaultBuildAgent.java:175)
at
com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.lambda$waitAndPerformBuild$0(BuildAgentControllerImpl.java:129)
at
com.atlassian.bamboo.variable.CustomVariableContextImpl.withVariableSubstitutor(CustomVariableContextImpl.java:185)
at
com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:123)
at
com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent$1.run(DefaultBuildAgent.java:126)
at
com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:48)
at
com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:26)
at
com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:17)
at
com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:41)
at java.lang.Thread.run(Thread.java:745)
I am using the Force new deployment setting.
Any ideas what is the issue?
We have not been able to identify an bug in our code base right away, here's what's seemingly happening:
In order to append progress messages to the Bamboo build log, we need to call the DescribeServices API action before the call to the actual UpdateService API action, and the exception is thrown if and only if the targeted service cannot be found.
So at first glance there may be a subtle configuration issue, which happens to me every now and then when using Bamboo variables to reference resources from a preceding task, where it is easy to accidentally copy and paste the wrong variable name for example.
An incorrect reference in any of the following parameters of the Amazon ECS Service task's Update Service action would yield the resp. task action to fail with the error message at hand, because the DescribeServices API call in itself would succeed, yet fail to identify the target service:
Connector
Region
Service Name
For example, I've just reproduced the problem by using a non existing service name:
24-Oct-2019 17:37:05 Starting task 'Update sample ECS service (w/ ELB) - 2 instances' of type 'net.utoolity.atlassian.bamboo.tasks-for-aws:aws.ecs.service'
24-Oct-2019 17:37:05 Setting maxErrorRetry=7 and awaitTransitionInterval=15000
24-Oct-2019 17:37:05 Using session credentials provided by Identity Federation for AWS app (connector variable: 6f6fc85d-4ea5-43ce-8e70-25aba33a5fda).
24-Oct-2019 17:37:05 Selecting region eu-west-1
24-Oct-2019 17:37:05 Updating service 'NOT-A-SERVICE' on cluster 'TAWS-IT270-100-ubot':
24-Oct-2019 17:37:06 Failed to fetch resource from AWS!
24-Oct-2019 17:37:06 java.lang.RuntimeException: Expected DescribeServiceRequest for service 'NOT-A-SERVICE' to return exactly 1 result, but got '0'
...
Granted, the error message is not exactly helpful here, and we need to think about how to better handle this log pattern across our various tasks - the actual UpdateServiceAPI action would yield the much more appropriate ServiceNotFoundException exception in this scenario.
So assuming 'my-service' has been up and running before calling the 'Update Service' task action, can you please check whether the log from your failing Bamboo build may indicate this particular problem, for example by targeting another region by chance?
I could solve the issue by using a Shell Script Task and wrote a aws-cli command after exporting the keys. This workaround solved the issue:
aws ecs update-service --cluster my-cluster --service my-service --task-definition my-task-definition
So the AWS ECS is working fine and it should be a bug or misconfiguration in the Bamboo module.
But as mentioned in the other answer, the best approach would be to check if the configuration is correct.
We are using the latest version of RabbitMQ, v3.7.2 on a few EC2 instances on AWS. We want to use auto clustering which comes default in the product, Cluster Formation and Peer Discovery.
After we start RabbitMQ it fails/ignores to do this. The only message we see in the log file is:
[info] <0.229.0> Peer discovery backend rabbit_peer_discovery_aws does not support registration, skipping registration.
On our RabbitMQ EC2 instance an IAM role is attached with the coorect policy. The rabbitMQ config is:
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_aws
cluster_formation.aws.region = eu-west-1
cluster_formation.aws.use_autoscaling_group = true
cluster_formation.aws.use_private_ip = true
Did anyone face this issue?
Add the following to your rabbitmq.conf and restart rabbitmq-server
log.file.level = debug
It allows you to see a discovery request to AWS in logs.
Then do this on any rabbitmq node:
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
It'll execute the discovery again. Check rabbitmq logs for 'AWS Request' you'll see corresponding response so that you can check if your ec2 instances were found by specified tags. If no, something is wrong with your tags.
Not an answer (not enough reputation points to comment) but I'm dealing with the same thing. I've double-checked that the security groups are correct, they allow ports 4369, 5672 and 15672 (confirmed via telnet/netcat), and the IAM policies are correct. Debug logging shows nothing else. I'm at a loss how to figure this one out.
Below are the steps that I performed for setup :
Created a new windows server 2012 EC2 instance and then connected
with it using the remote desktop connection.
Installed EC2 config service which automatically installs
AmazonSSMAgent service.
Both EC2 and SSM services are running and in error logs, it is throwing some 'no chain provider exception' which is fine because I have not provided aws credentials yet.
Created a new IAM user and give him administrative policy access and
created three environment variables providing access_key,
secret_access_key and region_id.
Then I restarted AmazonSSMAgent service, then the exception changed to below exception:
status code: 400, request id: 400b4c75-eae2-11e7-a120-d33083d55d1f 2017-12-27 08:45:13 ERROR [HandleAwsError # awserr.go.48] [instanceID=i-084646fb32d8a7b6d] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AuthorizationFailureException:
Please suggest what i am doing wrong or any steps that i have missed or any other way to do this.
I am trying to use the Amazon EC2 Run Command feature, and basically it says it can't find any instances.
I read all the documentation and followed all the steps that I could find on this relatively new feature, and I cant get it to work for the life of me.
Here is the info:
I go to "EC2 -> Commands -> Command History -> Run A Command"
I click radio box for the "AWS-RunShellScript" command document
I click the "Select Instances" drop down button, and it says "No instances found in this region" as shown in Figure 1 below.
I click the "Where are my instances?" hyperlink, which brings me to the AWS documentation show here:"Troubleshooting SSM Run Command"
The first suggestion for troubleshooting is to make sure that the prerequisites have been met, and provides another hyperlink shown here: "SSM Run Command Prerequisites"
Prerequisites:
Supported OS - Windows 2012 R2 - CHECK
Latest Agent Version - 3.17.1032 - CHECK (Shown In Figure 2)
Access to SSM Run Command - Assign EC2 Instance Role and IAM User Role - CHECK (Shown In Figures 3 and 4)
Internet Access - Outbound Internet Access - CHECK
I followed the AWS documentation for creating an Amazon EC2 Instance role for EC2 Run Command Access, and also to create an IAM User with proper Run Command Access. The documentation is shown here: Delegating Access to SSM Run Command
I used the Amazon CLI from my local machine to run the following command also shown in troubleshooting documentation:
aws ssm describe-instance-information --instance-information-filter-list key=InstanceIds,valueSet=instance-ID
The command returns no information:
{
"InstanceInformationList": []
}
UPDATE 1:
I did find the following errors and warnings in the following places, and have spent the last two hours trying to troubleshoot. I uinstalled and reinstalled the Ec2Config Service. I checked all the Ec2Config files. I Checked the 169.254 addresses, and the meta-data and dynamic data exists, and appears to be correct. Not sure what else to try here.
C:\Program Files\Amazon\Ec2ConfigService\Logs\Ec2ConfigLog - Error/Warning
2016-06-22T23:44:12.663Z: Warning: Unable to Publish to WMI. | System.Management.Instrumentation.WmiProviderInstallationException: Exception of type 'System.Management.Instrumentation.WMIInfraException' was thrown.
at System.Management.Instrumentation.InstrumentationManager.Publish(Object value)
2016-06-22T23:44:16.263Z: Failed to fetch instance metadata http://169.254.169.254/latest/user-data with exception The remote server returned an error: (404) Not Found.
2016-06-22T23:44:16.263Z: Failed to get metadata/user-data The remote server returned an error: (404) Not Found.
Windows Event Viewer Error
2016-06-22 23:46:59,758 [_Worker-2] ERROR [aws:getDocument] - Failed to update instance information., RequestId=9cb8f2dd-38d3-11e6-bc83-19c0650ffecc,ErrorCode=AccessDeniedException,ErrorType=Unknown,StatusCode=BadRequest,Message=Caller
instance identity does not match the given instanceId
UPDATE 2:
Per #praetorian1 's linked stackoverflow article I updated the Ec2Config Services "config.xml" file to enable the "Ec2HandleUserData" setting, rebooted, and still no change.
Per #Kai 's suggestion I also made sure that the "role" attached to the instance
matched the "AmazonEC2RoleforSSM" policy, and specifically permitted "ssm:UpdateInstanceInformation". It did.
I also looked under the AWS EC2 Management Console, at the "System Log" file for the specific instance, which did not contain any errors, and towards the end said that SSM Config: status:Active, iam:Yes. Also noteworthy, it contained: Info EC2Config configuration: status:2; region:us-east-1; iam:1; authz:1
2016/06/23 15:01:29Z: EC2ConfigMonitorState: 0
2016/06/23 15:01:29Z: Windows sysprep configuration complete.
2016/06/23 15:01:32Z: AMI Origin Version: 2016.01.13
2016/06/23 15:01:32Z: AMI Origin Name: Windows_Server-2012-R2_RTM-English-64Bit-SQL_2014_SP1_Web
2016/06/23 15:01:32Z: OS: Microsoft Windows NT 6.3.9600
2016/06/23 15:01:32Z: OsVersion: 6.3
2016/06/23 15:01:32Z: OsProductName: Windows Server 2012 R2 Standard
2016/06/23 15:01:32Z: OsBuildLabEx: 9600.18202.amd64fre.winblue_ltsb.160119-0600
2016/06/23 15:01:32Z: Language: en-US
2016/06/23 15:01:32Z: TimeZone: Eastern Standard Time
2016/06/23 15:01:32Z: Offset: UTC -04:00:00
2016/06/23 15:01:32Z: EC2 Agent: Ec2Config service v3.17.1032
2016/06/23 15:01:32Z: Driver: AWS PV Storage Host Adapter v7.3.2.0
2016/06/23 15:01:32Z: Driver: Intel(R) 82599 Virtual Function v1.0.15.3
2016/06/23 15:01:34Z: Message: Waiting for meta-data accessibility...
2016/06/23 15:01:34Z: Message: Meta-data is now available.
2016/06/23 15:01:37Z: Message: Windows is Ready to use
2016/06/23 15:01:40Z: Amazon EC2 Simple Systems Manager (SSM) is an optional service for custom configuration of instances.
2016/06/23 15:01:40Z: Info EC2Config configuration: status:2; region:us-east-1; iam:1; authz:1
2016/06/23 15:01:40Z: SSM Config: status:Active; iam:Yes
Figure 1:
Figure 2:
Figure 4:
Try checking the application logs in windows on the instance. Might be having issues talking to the SSM service for some reason.
As praetorian1 mentioned, Windows Event Viewer is the place to look for SSM - EC2 Run Command related log messages.
The message you posted indicates that the EC2 instance does not have required permission to call ssm:UpdateInstanceInformation API. Please double check that RunCommandInstance role has AmazonEC2RoleforSSM policy attached. (After attaching the policy, please restart EC2Config service or wait up to 15 min for EC2Config to start reporting instance information.)
For the IAM User in Fig. 4, we do not need policies other than AmazonSSMFullAccess for us to call SSM APIs with awscli.
To dive deeper:
EC2 Role and IAM Policy for the EC2 Instance
SSM on Windows requires an IAM EC2 Role attached to the EC2 instance at launch. Required permission for that Role is provided by an Amazon-managed IAM policy AmazonEC2RoleforSSM.
IAM Policy for IAM User
To make API calls to SSM service with awscli, an IAM user has to be setup with SSM permissions. For testing, Amazon provides a managed IAM policy AmazonSSMFullAccess. With the credentials of this IAM user, we will be able to make API call to describe instance information and also send command to the instance.
DescribeInstanceInformation API
Once the EC2 instance is setup with required permission and EC2Config starts reporting instance information back to SSM, we will be able to use DescribeInstanceInformation API to get a list of EC2 instances that are ready for SSM - EC2 Run Command.