I have setup AWS SNS setup with a topic say 'A'. I'm doing a subscribe to this SNS topic using Http (tried both manually using AWS console online and using Java Code). All I get is 'pending confirmation' in both cases. However SNS does not send the initial 'SubscriptionConfirmation' to the provided Url.
Note that my endpoint is ready to receive http POST notification. When I manually do a POST from my side I see my servlet processing those Json I send. For some reason I receive nothing from AWS SNS.
Note that my http end point that I used for subscribe is public facing so SNS should have no issue reaching it.
Any inputs is appreciated.
Here is my subscribe function.
public String subscribe(String arn,String url) {
if(arn == null || arn.isEmpty())
arn = topicArn;
SubscribeRequest subRequest = new SubscribeRequest(arn,"http",url);
SubscribeResult result = snsClient.subscribe(subRequest);
//get request id for SubscribeRequest from SNS metadata
if(result != null){
LOGGER.info("SubscribeResult - " + result.toString());
}
LOGGER.info("SubscribeRequest - " + snsClient.getCachedResponseMetadata(subRequest));
return result.toString();
}
You are always going to get "pending confirmation" as the response for the subscriptionArn. The confirmation process is asynchronously as a separate process. To make this even more confusing if you call to get a list of current subscriptions they will show an slightly different subscriptionArn of "PendingConfirmation" so you can not even match it later.
As far as being able to connect, I would try hitting an end point outside of AWS first. By default most AWS elements are very locked down and can not even connect to each other, so there is likely a security setting somewhere that needs to be changed to let SNS connect to your EC2. Which would be why you can connect to the EC2 outside of AWS, but your SNS service can not.
Also check to make sure the SNS and EC2 you are using are in the same region. It is a common cause of connection issues.
If you are using a host name to connect I would try using the direct IP to see if it gets through.
To troubleshoot, you should turn on the "Delivery status" reports in topic actions - https://docs.aws.amazon.com/sns/latest/dg/sns-msg-status.html. Then you will see why the confirmation message failed to be sent from AWS side.
On your EC2 instance side, on network level you must make sure that the port you are listening on is open from outside. There are several things: both making sure the port is open in firewall (Security groups settings), and making sure that the IP is reachable (i.e., make sure your VPC where the machine is located is publicly visible).
I faced the same issue, the region was the problem.
Make sure the SNS, CloudWatch and EC2 are in the same region.
For me disabling encryption on the topic allowed the emails to finally be delivered, albeit to the spam folder.
Related
Is there any way to receive a notification email before an automatic minor version upgrade begins?
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-instances-status-check_sched.html
This page says "AWS sends an email to the email address that's associated with your AWS account prior to the scheduled event", but the root user's email address of the AWS account is not mine.
The page also says "AWS also sends an AWS Health event, which you can monitor and manage using Amazon CloudWatch Events", so I tried to create a rule for RDS scheduledChange event. Under this category there are four types (AWS_RDS_AURORA_HARDWARE_MAINTENANCE_SCHEDULED, AWS_RDS_HARDWARE_MAINTENANCE_SCHEDULED, AWS_RDS_MAINTENANCE_SCHEDULED, AWS_RDS_SYSTEM_UPGRADE_SCHEDULED), but I'm not sure if one of these corresponds to automatic minor version upgrade. I couldn't find any document to confirm that. Does anyone know?
For regular maintenance events, like RDS changes, you can configure notifications for RDS events.
You can see full list of events here.
There are probably two events of interest:
RDS-EVENT-0026 which is sent when the DB begins maintenance.
RDS-EVENT-0155 which is "The DB instance has a DB engine minor version upgrade available"
RDS instances also have a configurable maintenance window, so you can expect maintenance to always occur in that window if there is pending maintenance.
Currently, I have a working modded Minecraft server working running on a C5 EC2 instance. The problem is that I have to manually start and stop the server which can be annoying for my friends. I was wondering if it would be possible to automate the EC2 state so that it runs as soon as a player attempts to join the sever. This would be similar to how Minecraft realms behaves which I heard Mojang was using AWS for:
https://aws.amazon.com/blogs/aws/hosting-minecraft-realms-on-aws/
I have looked up tutorials for this and this is the best I could come across:
https://github.com/trevor-laher/OnDemandMinecraft
The problem with this solution is that it requires to make a separate website to log users in and start the EC2 instance while I want the startup and shutdown to be completely automatic.
I would appreciate any guidance.
If the server is off, it would not be possible to "connect" to the server. Therefore, another mechanism is required that can be used to start the server.
Combine that with your desire to minimise costs and the only real solution is to somehow trigger an AWS Lambda function, which could start the server.
There are a few ways you could have users trigger the AWS Lambda function:
Make a call to API Gateway
Upload an object to Amazon S3
Somehow put a message in an SNS topic or an SQS queue
Trigger an Amazon CloudWatch Alarm (which calls Lambda via SNS)
...and probably other ways
When considering a method to use, you should consider security implications such as:
Whether only authorized users should be able to trigger the Lambda function, or is it okay that anybody (eg a web crawler) might trigger it.
Whether you willing to give your friends AWS credentials (not a good idea) that they could use to start the server directly, or whether it should be an indirect method.
Frankly, I would recommend the following architecture:
Create an AWS Lambda function that turns on the server
Create an API Gateway that triggers the Lambda function
Give a URL to your friends that calls the API Gateway and passes a 'secret' (effectively a password)
The API Gateway will call the Lambda function, passing the secret
The Lambda function confirms that the secret is correct and starts the Amazon EC2 instance with Minecraft installed
Here is a tutorial that shows many of these concepts: Build an API Gateway API with Lambda Integration
The purpose of the secret is to avoid the server from starting if an unauthorized person (or a bot) happens to hit the API Gateway endpoint. They will not provide the secret, so the server will not be started.
Stopping the server after a period of non-use is a different matter. The library you referenced might be able to assist with finding a way to do this. You could have a script running on the Minecraft server that monitors the game and, after a period of inactivity, simply calls the operating system to perform a Shutdown.
You could use a BungeeCord hub server that then allows user to begin a connection to the main server and spin it up via. AWS.
This would require the bungee server to be always up however, but the cost of hosting a small bungee server should be relatively cheap.
I don't think there's any way you could do this without having a small server that receives the initial request to spin up the AWS machine.
I am using AWS IoT Service.
When a device sends a registration message to MQTT broker, I have a rule to store it in a SQS queue.
A Lambda function is triggered, when the message is added to the Queue. The Thing is created for the device and it's certificate is registered.
While carrying out the load testing, I observed that, after some time, the incoming messages are not received on the AWS MQTT broker and are not processed.
I have written some test clients which run on EC2 instances to simulate the MQTT clients.
If I restart the test clients after some time, again I can see the messages coming to AWS IoT.
I am not sure, if this is the issue of MQTT broker or if it is the issue with the clients running on EC2 instances.
I can think of possible issues because of limits on AWS IoT ,
https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_iot
I want to know what are the possible AWS IoT matrices, I need to monitor for this or which IoT specific alarms I need to configure?
Is it a possible issue on EC2 side? ( maybe network out bytes per second, etc.)
There is another load testing scenario, in which I am not doing registration of devices, but just capturing the connect or disconnect events. In this case, I am not observing similar issues.
As you know, there is some limits about AWS IoT.
API Transactions per Second
CreateCertificateFromCsr 15
CreateDynamicThingGroup 5
CreateJob 10
CreatePolicy 10
CreatePolicyVersion 10
CreateRoleAlias 10
CreateThing 15
Generally, AWS API throws Exception when it run over limts.
How about catch Exception?
If you want to check EC2 network issue, use some command ( netstat, tcpdump, ... )
I need to receive notifications whenver my instance in terminated. I know it can be done by cloudtrail and then using sns and sqs to get email for it, if you receive event of termination.
Is there a simpler way to do that ?
Any solution will is appreciated, but I prefer is doing using boto.
While it is not possible to receive a notification directly from Amazon EC2 when an instance is terminated, there are a couple of ways this could be accomplished:
Auto Scaling can send a notification when an instance managed by Auto Scaling is terminated. See: Configure Your Auto Scaling Group to Send Notifications
AWS Config can also be configured to send a Simple Notification Service (SNS) notification when resources change. This would send many notifications, so you would need to inspect and filter the notifications to find the one(s) indicating an instance termination. See the SNS reference in: Set Up AWS Config Using the Console and Example Amazon SNS Notification and Email from AWS Config.
Amazon Simple Notification Service (SNS) can also push a message to Amazon Queueing Service (SQS), which can be easily polled with the boto python SDK.
Receiving notifications via CloudTrail and CloudWatch Logs is somewhat messier, so I'd recommend the AWS Config method.
Now AWS introduced "rules" Under "Events" in AWS CloudWatch. In your case, you can select EC2 as Event Selector and SNS or SQS as Targets.
https://aws.amazon.com/blogs/aws/new-cloudwatch-events-track-and-respond-to-changes-to-your-aws-resources/
According to the AWS doc: Spot Instance Interruptions, it is possible to pool the instance-metadata in order to get an approximation of the termination time. You can build any custom monitoring solution around that.
> curl http://169.254.169.254/latest/meta-data/spot/instance-action
{"action": "stop", "time": "2017-09-18T08:22:00Z"}
If the instance is not scheduled for termination a http:400 will be returned.
We have hosted some apps on Amazon EC2 and are using an Elastic Load Balancer (ELB) to manage several instances of one app. Also, we have set up ELB alarms to get notified about Unhealthy Hosts, i.e. when an instance has gone down.
So far, I could not figure out where to check which instance exactly has gone down when the alarm goes off, except for the ELB status page in the AWS console. However, if the instance comes back to In Service state again, this won't help me either.
The e-mail notification sent out by the ELB does not contain this information; and I couldn't find it anywhere in the alarms history in the console either.
Is there a way to tell which instance an ELB alarm has been triggered for, even if the instance has come back into OK state in the meantime?
Cheers, Alex
Sadly Amazon does not provide a health check log, so its impossible to find out which instance failed the health check afterwards, assuming that the server is no longer unhealthy. You can only use Per-Az metrics to know in which AZ is the instance.
But, you could know which instance is down if you query AWS api during the issue. So, I have thought of a possible workaround:
Set up a new SNS topic, and add an HTTP action to a custom URL that triggers a job that enumerates the instances and send you that info by mail.
Then setup a CloudWatch alarm for UnHealthyHostCount > 0 and setup the action to the SNS topic.
The difficult part is that your URL should handle the SNS subscription & confirmation described here.
The command to know which instance is currently OutOfService is:
elb-describe-instance-health *LoadBalancerName* --region *YourRegion*
You could probably use the AWS SDK gem or other AWS library that can get status. Use it to create a cron task that regularly gets the status of each instance and records it somewhere. Either that will give you what you need or the disappearance of the status for one instance will tell you which one went bad.
We are using the following Lambda function to make up for the lack of Health Check logging:
'use strict';
var AWS = require('aws-sdk');
var elb = new AWS.ELB();
exports.handler = (event, context, callback) => {
var params = {
LoadBalancerName: "<elb_name_here>"
};
elb.describeInstanceHealth(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
};
It does not produce the prettiest logs in CloudWatch, but the data is there. It allows us to see if there is a particular instance which tends to drop more often, etc. It is set up much like Gerardo Grignoli's answer above. I added a CloudWatch alarm to send an SNS message to the Lambda function when the alarm was triggered. It doesn't do anything with the message itself - the message is merely the triggering mechanism for the Lambda function to run and log the instance status.
No. The ELB metrics in CloudWatch do not provide you with that level of details and IMHO from the design perspective they should not. If a host is unhealthy the monitoring on the specific host should report the details for that not the ELB. If a node goes out of service in ELB, it should not be a problem for ELB. Although, in load balancer it makes sense to figure out an alarming state where 3 out of 6 of your machines go into Not In Service state. Take a look at CloudWatch metrics
Go to load balancer and find load balancer associated with you ELB. Then look at instances that OutofService