I am attempting to have AWS CodeBuild run a Flyway migration. The DB and CodeBuild Project are created via Terraform (the pipeline runs as a GitHub action, if it matters)
That code is here.
I figured this solution would make the difference: AWS CodeBuild fails to interact with RDS instance
When the CodeBuild project is executed by my GitHub workflow (using the aws-actions/aws-codebuild-run-build action), the migration times out:
[Container] 2022/10/07 21:03:56 Running command flyway -user=$DB_USER -password=$DB_PASSWORD -url=jdbc:mariadb://$DB_HOST:$DB_PORT/$DB_NAME -createSchemas=true migrate
ERROR: Unable to obtain connection from database (jdbc:mariadb://***:***/***) for user '***': Could not connect to address=(host=***)(port=***)(type=master) : Socket fail to connect to host:***, port:***. connect timed out
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL State : 08000
Error Code : -1
Message : Could not connect to address=(host=***)(port=***)(type=master) : Socket fail to connect to host:***, port:***. connect timed out
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=***)(port=***)(type=master) : Socket fail to connect to host:***, port:***. connect timed out
Caused by: java.sql.SQLNonTransientConnectionException: Socket fail to connect to host:***, port:***. connect timed out
Caused by: java.net.SocketTimeoutException: connect timed out
This tells me it's some sort of networking problem but I can't put my finger on what route might be missing. No NACLs other than the defaults. Just security groups. I have a similar pipeline in the AWS CDK that works. As near as I can tell, the security groups and IAM permissions are identical, as is the database config itself.
Looking for debugging tips or anything that's missing.
Consider setting the vpc_security_group_ids parameter on your aws_db_instance resource. In that collection should be the security group you associated with your codebuild project. Currently it doesn't appear that your database has an associated security group and so traffic coming from your codebuild project isn't whitelisted and cannot make it through.
See Terrform docs
Related
first time asker.
So I've been trying to implement AWS Cloud Watch to monitor Disk Usage on an EC2 instance running EC2 Linux. I'm interesting in doing this just using the CW Agent and I've installed it according to the how-to found here. The install runs fine and I've made sure I've created an IAM Role for the instance as is described here. Unfortunately whenever I run the amazon-cloudwatch-agent.service it only sends log files and not the custom used_percent measurement specified. I receive this error when I tail the logs.
2021-06-18T15:41:37Z E! WriteToCloudWatch failure, err: RequestError: send request failed
caused by: Post "https://monitoring.us-west-2.amazonaws.com/": dial tcp 172.17.1.25:443: i/o timeout
I've done my best googlefu but gotten nowhere thus far. If you've got any advice it would be appreciated.
Thank you
Belated answer to my own question. I had to create a security group that would accept traffic from that same security group!
Having the same issue, it definitely wasn't a network restriction as I was still able to telnet to the monitoring endpoint.
From AWS docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent.html
One role or user enables CloudWatch agent to be installed on a server
and send metrics to CloudWatch. The other role or user is needed to
store your CloudWatch agent configuration in Systems Manager Parameter
Store. Parameter Store enables multiple servers to use one CloudWatch
agent configuration.
If you're using the default cloudwatchagent configuration wizard, you may require extra policy CloudWatchAgentAdminRole in your role for the agent to connect to the monitoring service.
I am trying to migrate all the data present in my old RDS Aurora PostgreSQL cluster to the new RDS Aurora PostgreSQL cluster using AWS DMS. I have created the source and target endpoints and tested the connection successfully. However when I am trying to create a migration task in DMS, it is continuously failing with the error:
Last Error ODBC general error. Error executing command; Stream component failed at subtask 0,
component st_0_PWDKKAMFPUY2RHV ; Stream component
'st_0_PWDKKAMFPUY2RHV' terminated [reptask/replicationtask.c:3171] [1022502]
Stop Reason RECOVERABLE_ERROR Error Level RECOVERABLE
Even after enabling CloudWatch logs, I am not able to figure out what's missing? What does the error signify or what am I doing wrong?
I had faced the same error and the issue seems related to database user rights for
Replication Client and Replication Slave
I have fixed it by setting the Replication rights using the below statements in SQL
GRANT REPLICATION CLIENT ON *.* to {dbusername}#'%';
GRANT REPLICATION SLAVE ON *.* to {dbusername}#'%';
Note: replacing {dbusername} with the actual database user name which was being used in DMS Endpoint
I have 2 versions of a service that run in the same cluster. I'm using the awslogs driver
The v2 logs works fine however the v1 task fails to start because it can't create a log stream.
The setup is identical between services except for the container being used.
The log group exists and the role has permissions to create a "logstream" and can "putevents" as this is pretty much the same setup for the v2 in a different group.
CannotStartContainerError: Error response from daemon: failed to initialize logging driver: failed to create Cloudwatch log stream: RequestError: send request failed caused by: Post https://logs.eu-west-1-v1.amazonaws.com/: dial tcp: lookup logs.eu-west-1
I've setup a new service and tried to spin it up again but it failed so I thought that this was to do with the container setup.
On the official documentation here it recommends adding this to the environment variables
ECS_AVAILABLE_LOGGING_DRIVERS '["json-file","awslogs"]'
After adding this, it still failed. I've been searching for a while on this and would appreciate any help or preferably guidance.
When running the following command
SELECT transport.import_from_server(%s,5432,'My RDS ADMIN USER',%s,%s,%s,true);
I get the following response from the command:
AWS RDS pg_transport failed to download file data
Both RDS are in the same region, same vpc, both have security groups allowing the connection between them, SG only has inbound for 5432
Unable to find documentation or any further info on possible failure.
Steps followed were: https://aws.amazon.com/blogs/database/migrating-databases-using-rds-postgresql-transportable-databases/
With existing RDS instances, both are running Postgresql 11.5 and custom data instead of the one from the tutorial.
Any advice?
Could you please recheck if your source instance Security group allows connection from destination instance?
Recheck all the parameters that you have set in the source and destination param groups.
Had this before, it seems to be a bug within pg_transport.
Advice from AWS was to use a larger instance class on both the source and target instances. It seems to be stable using db.m5.4xlarge
I am trying to set up AWS CodeDeploy for my PHP web app. I have created a CodeDeploy app and a deployment group on the AWS console. I have created the necessary revision bundle with the appspec yaml file. The revision bundle is stored on Amazon S3.
When I click 'Deploy this revision' button on the AWS console it gives me 'no hosts succeeded' error. I went through the Technical FAQ and could not find any answers. How can I counter this error?
UPDATE: I now understand that this error has something to do with Minimum Healthy Hosts count. But still I am not able to understand how does AWS calculate the healthiness of a host.
Basically what its saying is "The codedeploy service on your ec2 instance is not running"...
For why a deployment failed host health is fairly simple. A host is healthy if that host succeeded in deploying the last deployment to it. A host is unhealthy if it failed. A host is unknown if it was Skipped and had no previous deployment.
There are other aspects of host health that affect what order they are deployed to in the next deployment, but that's not going to affect you deployment failing with "No hosts succeeded".
A host can fail it's individual deployment if any of it's lifecycle events failed. A lifecycle event can fail due to service side timeout waiting for the agent to respond or because the host agent reports an error executing the command. You can check the host agent log for more details in exactly why the host agent reported a failure.
If you are hitting the server side timeouts, you should check that the host agent is running and is able to poll for commands correctly. You might have accidentally restricted access in your VPC configuration or didn't grant appropriate permissions to the instance to poll for commands in the instance profile.
This error message means you are not running CodeDeploy service at the EC2 instance targeted by your deployment group.
1) Download latest version of codedeploy from S3 (choose your region)
PS> Read-S3Object -BucketName aws-codedeploy-eu-west-1 -Key latest/codedeploy-agent.msi -File c:\temp\codedeploy-agent.msi
2) Install codedeploy
cmd> c:\temp\codedeploy-agent.msi /quiet /l c:\temp\host-agent-install-log.txt
3) Start codeploy
PS> Start-Service -Name codedeployagent
AWS CodeDeploy guide: http://docs.aws.amazon.com/codedeploy/latest/userguide/how-to-run-agent.html#how-to-run-agent-install-windows
I just ran into this issue myself. My solution was to run:
ntpdate-debian
If you are running centos it's something like
ntpdate pool.ntp.org
For me the time was off and was causing issues with the codedeploy agent.
Now, if this doesn't solve your problem. First make sure your problem is that your CodeDeploy agent is not registering. I have had this issue before and it was because one of my instances was in a failed state from a botched deployment so be sure to double check. (ELB status, tests, etc)
Then you should enable logging for your CodeDeploy agent by setting log_aws_wire and verbose to true in /etc/codedeploy-agent/conf/codedeployagent.yml and then restart the CodeDeploy. Tail the logs and you should see the reason for your problems.