I am trying to push logs to Cloudwatch from a RHEL Instance. Originally I was getting the error:
[outputs.cloudwatchlogs] Aws error received when sending logs to LogGroup/LogStream: RequestError: send request failed caused by: Post "https://logs.<region>.amazonaws.com/": dial tcp xx.xx.xx.xx:443: i/o timeout
I tried everything I could think of, I saw some things online that it could be Proxy related. I have a proxy server instance on the AWS account.
I added the following into the common-config.toml for Cloudwatch:
[proxy] http_proxy = "htttp://${PROXY_SERVER}:$PORT" https_proxy = "http://${PROXY_SERVER}:$PORT" no_proxy = "XX.XX.XX.XX"
The error I am getting now is:
[outputs.cloudwatchlogs] Aws error received when sending logs to LogGroup/LogStream: RequestError: send request failed caused by: Post "https://logs.<region>.amazonaws.com/": proxyconnect tcp: dial tcp: lookup http on XX.XX.XX.XX:53: server misbehaving
I am in a private VPC and I can't ping public sites as I get 100% packet loss. I can manually run the AWS Cli command to push an entry into the log stream. For now I am just trying to push /var/log/messages from my instance. Can anyone help with why the CloudWatch logs aren't pushing?
Some thing I have tried that didn't work for the original error:
exporting no_proxy
adding AWS_STS_REGIONAL_ENDPOINTS as an env variable
Ensuring port 443 is open on SG's
Ensuring IAM profile has correct permissions for CW and EC2
Related
I was no longer able to SSH into a Google Cloud Compute Engine VM instance that previously showed no problems.
The error logs show the following
#type: "type.googleapis.com/google.protobuf.Struct" value: {
conditionNotMet: { userVisibleMessage: "Supplied fingerprint does not
match current metadata fingerprint."
Trying SSH through the console showed
Code: 4003 Reason: failed to connect to backend Please ensure that:
your user account has iap.tunnelInstances.accessViaIAP permission
VM has a firewall rule that allows TCP ingress traffic from the IP range XXX.0/20, port: 22
you can make a proper https connection to the IAP for TCP hostname: https://tunnel.cloudproxy.app You may be able to connect without using
the Cloud Identity-Aware Proxy.
The VM instance logs showed the following
Error watching metadata: Get
http://metadata.google.internal/computeMetadata/v1//?recursive=true&alt=json&wait_for_change=true&timeout_sec=60&last_etag=XXX:
net/http: request canceled (Client.Timeout exceeded while awaiting
headers)
After stopping and restarting the instance I was able to ssh again but I would like to understand the reason for the problem in the first place.
The error message you received indicates that the metadata server's response caused the connection to the Google Compute Engine VM instance to time out. This could be because the server was taking too long to respond or there was a problem with the network. You can try to resolve this issue by either increasing the timeout value by using this doc or waiting for the instance to become healthy using the gcloud compute wait command.
The instance was unable to reach the metadata server, as suggested by the timeout error message. This could be a problem with the instance itself or with the network connection. A firewall or network configuration issue could have prevented the instance from connecting to the metadata server, or an issue with the underlying infrastructure could have rendered the instance temporarily unavailable.
To prevent this issue from happening again, you can increase the timeout value or use the gcloud compute wait command to wait for the instance to become healthy.it is recommended that you regularly update the SSH key used to connect to the instance, and check that the instance can reach the metadata server by making an HTTPS request to the IAP for TCP hostname. Additionally, it is important to ensure that your user account has the "iap.tunnelInstances.accessViaIAP" permission, and that the VM has a firewall rule that allows TCP ingress traffic from the IP range XXX.0/20, port: 22.
If you are using windows vm try troubleshooting steps mentioned in this doc.
Iam trying to set up aws cluster. I installed kubectl and configured aws with credentials. When I tried to display pods or any kubectl commands, Iam getting this error
revaa#revaa-Lenovo-E41-25:~$ kubectl get pod
Unable to connect to the server: dial tcp 10.0.12.77:443: i/o timeout
How to resolve this
The ip used is a private, it cannot be accessed outside the aws. Change your cluster's api server endpoint access to public. It worked.
It is a question about AWS.
The following error occurred between ELB and EC2(nginx).
ELB
HTTP 504: Gateway Timeout
EC2(nginx)
HTTP Error 408 Request timeout
Have someone discovered the same phenomenon? and know the cause?
Are you getting 408 in nginx error.log ? If yes, then it has nothing to do with ELB because it shows that request is going from ELB to nginx; and nginx itself is giving "Request timeout".. It is an application issue..Your application is not able to process the request.
One way to test would be to expose EC2 to public i.e. associate an Elastic IP address to EC2 and try to connect to it directly i.e. removing ELB from picture to be double sure.
I had my web app running in EC2 instance(AWS server 1). I have another AWS server 2 where the DB is. I had a verified domain and verified recipient test email address and emails were going out well. One day I did Elastic IP so that AWS 1 talk to AWS 2 for some other purpose. Not sure if that caused the issue. Now I reverted AWS 1 to normal Ipv4 address(removed Elastic IP) and all over my app and for SSH I use normal IPv4 address.
As per other posts, I also contacted AWS and increased the sending
limit.
I also set the outbound rules SMTP and SMTPS. None seems to
be working.
If I run the web app in my localhost with same SES
credentials, emails are sent out. Only when my web app is in Amazon EC2 then emails are not being sent out.
Following is the error that I am getting.
Unable to execute HTTP request: Connect to email.us-west-2.amazonaws.com:443 [email.us-west-2.amazonaws.com/52.94.209.0] failed: connect timed out
It's been 2 days and I am scratching my head to get it resolved. Please help.
PS: As per request, here are the outbound rules
Type : MYSQL/Aurora
Protocol: TCP
Port Range: 3306
Destination : //MyIP
I don't have any other outbound rule.
I deployed an Application to AWS elastic beanstalk. When I try to open the application, I am getting 502 proxy error Saying following message.
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.
Reason: Error reading from remote server
Apache/2.2.31 (Amazon) Server at mehe.us-west-2.elasticbeanstalk.com
Port 80
Strange part is when I run the application from localhost(Still connected to amazon database) the application is working fine, but after deloying it is not working. Here's the link to application
Any ideas how to get rid off it.
The timeout value for HTTPD is lower than the timeout value set for ELB. Change the timeout value in /etc/httpd/conf/httpd.conf
The keep the value between reboots you'll need to either create a custom AMI or use .ebextensions feature.