I am trying to create a connection to AWS Redshift from the Airflow Web UI, but so far it does not work. I always get the error: ('communication error', ConnectionRefusedError(111, 'Connection refused').
My Redshift is in a subnet of a VPC, but I have allowed public access. In its security group I have furthermore added: 0.0.0.0/0 IPV4 incoming access and my own IP as incoming access allowed.
As The input parameters to the airflow connection template, I use:
As the connection-id: An arbitrary name
host: redshift-clusterxxxxxxxxxxxxxxxxxxxxxxxxxx.redshift.amazonaws.com
The other parameters are also correct.
I have build a little python using redshift_connector to test whether I can connect from there but I am always getting a timeout error.
Has anyone got further ideas on how to solve this issue?
Related
I was no longer able to SSH into a Google Cloud Compute Engine VM instance that previously showed no problems.
The error logs show the following
#type: "type.googleapis.com/google.protobuf.Struct" value: {
conditionNotMet: { userVisibleMessage: "Supplied fingerprint does not
match current metadata fingerprint."
Trying SSH through the console showed
Code: 4003 Reason: failed to connect to backend Please ensure that:
your user account has iap.tunnelInstances.accessViaIAP permission
VM has a firewall rule that allows TCP ingress traffic from the IP range XXX.0/20, port: 22
you can make a proper https connection to the IAP for TCP hostname: https://tunnel.cloudproxy.app You may be able to connect without using
the Cloud Identity-Aware Proxy.
The VM instance logs showed the following
Error watching metadata: Get
http://metadata.google.internal/computeMetadata/v1//?recursive=true&alt=json&wait_for_change=true&timeout_sec=60&last_etag=XXX:
net/http: request canceled (Client.Timeout exceeded while awaiting
headers)
After stopping and restarting the instance I was able to ssh again but I would like to understand the reason for the problem in the first place.
The error message you received indicates that the metadata server's response caused the connection to the Google Compute Engine VM instance to time out. This could be because the server was taking too long to respond or there was a problem with the network. You can try to resolve this issue by either increasing the timeout value by using this doc or waiting for the instance to become healthy using the gcloud compute wait command.
The instance was unable to reach the metadata server, as suggested by the timeout error message. This could be a problem with the instance itself or with the network connection. A firewall or network configuration issue could have prevented the instance from connecting to the metadata server, or an issue with the underlying infrastructure could have rendered the instance temporarily unavailable.
To prevent this issue from happening again, you can increase the timeout value or use the gcloud compute wait command to wait for the instance to become healthy.it is recommended that you regularly update the SSH key used to connect to the instance, and check that the instance can reach the metadata server by making an HTTPS request to the IAP for TCP hostname. Additionally, it is important to ensure that your user account has the "iap.tunnelInstances.accessViaIAP" permission, and that the VM has a firewall rule that allows TCP ingress traffic from the IP range XXX.0/20, port: 22.
If you are using windows vm try troubleshooting steps mentioned in this doc.
I set up an private Airflow environment in AWS -v2.2.2-.
Environment and plugins are up and running, I want to connect to Snowflake but I am getting the error below . -whl files in plugins.zip using requirements.txt-
snowflake.connector.vendored.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='......snowflakecomputing.com', port=443): Max retries exceeded with url: /session/v1/login-request?request_id=....... (Caused by ConnectTimeoutError(<snowflake.connector.vendored.urllib3.connection.HTTPSConnection object at >, 'Connection to ........snowflakecomputing.com timed out. (connect timeout=60)'))
Same connection works in public mwaa.
I am adding connection informations into admin-connections tab from the UI
I know private env does not have connection to internet
I am aware i need to add some kind of outbound rule or endpoint but couldn't figure out .
Checked the endpoints and couldn't see anything related to Snowflake.
I will also be connecting to postgres, mysql DB's and few API's, which currently fails also
Is there a 1 click solution like adding some kind of outbound rule or should i be applying everything 1by1, and what would be that ?
If i want to connect to google-api something new, for snowflake something new etc ?
Also private mwaa environment is running on an existing vpc that has igw attachment, but the subnets that mwaa is running doesn't have any igw or nat attachment -as documentation suggests-
I am trying to connect to AWS RDS database using pgAdmin4, I encounter and error:
Unable to connect to server:
could not connect to server: Connection timed out (0x0000274C/10060)
Is the server running on host "database-1.xxxx.eu-west-2.rds.amazonaws.com" (xx.xxx.xxx.xx) and accepting TCP/IP connections on port 5432?
I have tried to Allow Public Accessibility and InBound Rules from my IP. But I am still getting the error? What seems to be the issue?
I was facing the same error but for the MySQL database in rds. What I did was, I
went to http://checkip.amazonaws.com/ and checked the IP shown. I copied that IP and created a new security group by adding two rules, one is your standard all traffic rule, and the second is the one with custom IP which you just copied. Then went to rds and changed VPC security grp to both default and the one I just created. Do this and try to connect again. It worked for me
For a quick test, try opening your inbound rule like this. If it works, then you know the issue is your IP address in your inbound rule:
Once you set this- can you connect?
I have an AWS Lambda function for an API. I have a test function inside of the API that returns a "Success" string so I know that I can hit the API from my machine (Postman).
The Lambda API is inside of my VPC. I have public and private subnets. The API is living in the public subnets. I also have a RDS database that's living in the private subnets. I can connect to the database through a Bastion host so I know the database is connected and working.
When I try an API call that connects to the database, I can see an error in CloudWatch that says:
[Error] Amazon.Lambda.AspNetCoreServer.APIGatewayProxyFunction: Unknown error responding to request: MySqlException:
MySql.Data.MySqlClient.MySqlException (0x80004005): Unable to connect to any of the specified MySQL hosts. ---> System.InvalidOperationException: Sequence contains more than one matching element
There's a lot of references to that error message that talks about the Linq SingleOrDefault() method, but I don't have any references to that method.
I think that error has something to do with the security groups. I have an RDS security group that has an inbound rule on port 3306 from my internal VPC ip addresses.
The security group for my Lambda function should be ok since I can successfully hit the test route.
Finally, I tried connecting to the database through the API in debug on my machine and I get an error "The requested name is valid, but no data of the requested type was found"
Does anyone have any other ideas or places to look?
Can you check your this setting is enable or not for RDS Security group ?
Disclaimer : Allowing inbound traffic on DB from internet is not recommended and should always be avoided
My RDS database (that I was trying to connect to and read from using AWS Lambda) was also a MySQL database.
When I tested the Lambda function I was getting 'Unable to connect to any of the specified MySQL hosts' (after waiting -- like it was timing-out).
I reviewed the answer from #aviboy2006, which helped me. I tried all the Security Group rules shown in that answer. I found I was able to successfully connect-to and read from the RDS MySQL database when I defined both an Inbound and Outbound rule for the associated VPC Security Group as
Type: MYSQL/Aurora
Protocol: TCP
Port Range: 3306
Source: Custom: 0.0.0.0/0
I'm trying to connect from an ec2 micro instance to the following domain: http://www.bnm.gov.my but it seems to be blocked.
I have tried doing a simple request using curl.
curl http://www.bnm.gov.my
which gives me the following error:
curl: (7) Failed to connect to www.bnm.gov.my port 80: Connection timed out
I tried wget
wget http://www.bnm.gov.my
and get the following error:
--2018-10-31 14:33:06-- http://www.bnm.gov.my/
Resolving www.bnm.gov.my (www.bnm.gov.my)... 103.6.236.129
Connecting to www.bnm.gov.my (www.bnm.gov.my)|103.6.236.129|:80... failed: Connection timed out.
Retrying.
Also, a traceroute times out.
I can access the website from my computer as well as websites like downforeveryoneorjustme dot com and uptrends to check if there are any issues with the website but it seems to be my amazon instance.
I have access to internet, I can request other websites using curl... but this particular one isn't working...
I appreciate any ideas you may have as to how to fix this!
Current ACL Configuration for VPC. (The instance is running within a VPC)
Outbound
Rule # Type Protocol Port Range Destination Allow/Deny
100 All IPv4 traffic all all 0.0.0.0/0 ALLOW
* All IPv4 traffic all all 0.0.0.0/0 DENY