In my dev box (ColdFusion 2021 Developer edition) I try to setup an ActiveMQ Gateway.
In CF Admin - Event Gateways - Settings the
"Enable ColdFusion Event Gateway Services" is checked (enabled)
and
Event Gateway Processing Threads = 1 (as is the max for developer edition)
Maximum number of events to queue = 2 (the max for developer edition is 10)
In Gateway Instances I added an ActiveMQ instance it was saved successfully, but in the table "Configured ColdFusion Event Gateway Instances" when in Actions column I click on "Start" to enable the new Gateway instance I get a message
==> "Unable to start event gateway instance: Event Gateway Service is not enabled."
I added and another Gateway instance of CFML type and I get the same message.
I rebooted and in eventGateway.log it reports
"Information","Thread-24","06/21/22","14:35:16","","Stopping Event Gateways."
"Information","main","06/21/22","14:35:59","","Starting Event Backend Handlers."
"Information","main","06/21/22","14:35:59","","Initialized EventRequestDispatcher with a Thread Pool size of 1."
"Information","main","06/21/22","14:35:59","","Initializing EventRequestHandler"
"Information","main","06/21/22","14:35:59","","Starting Event Gateways."
After the reboot I tried again to enable the gateway instances and I got the same message.
Do you have any idea what's the problem and how I can start the gateway instances?
Thanks in advance
Related
I'm using API Gateway with Web sockets to implement a chat server. I'm storing all the messages in current session in Redis (AWS Elasticache).
API gateway websocket -> SQS(FIFO) -> Lambda(For processing) -> Elasticache.
I want to implement timeout functionality. If 20 minutes have passed since last message then move the data from redis to sql.
How can I achieve this on AWS?
The approach I would take would be to have a last message key pair stored in your redis server with a timestamp and would update with every message. Something like:
KEY VALUE
---------------------------------------------------
lastmessage 1654371394
I would then have lambda function that checked that value and if the timestamp is over 20 minutes, do the redis to sql logic. I would then create a cloudwatch scheduled event that would trigger the lambda function every minute.
This article explains how:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html
I am using AWS IoT Service.
When a device sends a registration message to MQTT broker, I have a rule to store it in a SQS queue.
A Lambda function is triggered, when the message is added to the Queue. The Thing is created for the device and it's certificate is registered.
While carrying out the load testing, I observed that, after some time, the incoming messages are not received on the AWS MQTT broker and are not processed.
I have written some test clients which run on EC2 instances to simulate the MQTT clients.
If I restart the test clients after some time, again I can see the messages coming to AWS IoT.
I am not sure, if this is the issue of MQTT broker or if it is the issue with the clients running on EC2 instances.
I can think of possible issues because of limits on AWS IoT ,
https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_iot
I want to know what are the possible AWS IoT matrices, I need to monitor for this or which IoT specific alarms I need to configure?
Is it a possible issue on EC2 side? ( maybe network out bytes per second, etc.)
There is another load testing scenario, in which I am not doing registration of devices, but just capturing the connect or disconnect events. In this case, I am not observing similar issues.
As you know, there is some limits about AWS IoT.
API Transactions per Second
CreateCertificateFromCsr 15
CreateDynamicThingGroup 5
CreateJob 10
CreatePolicy 10
CreatePolicyVersion 10
CreateRoleAlias 10
CreateThing 15
Generally, AWS API throws Exception when it run over limts.
How about catch Exception?
If you want to check EC2 network issue, use some command ( netstat, tcpdump, ... )
I want email notification if any specific EC2 windows service entered into the stopped state.
I configured CloudWatch, able to receive logs of all windows services.
Created a lambda function to get notify when any service entered into the stopped state, but the problem is I am receiving alert only when I click on the test function.
I am receiving CloudWatch logs like this:
03:43:02 [System] [INFORMATION] [7036] [Service Control Manager] [mydomain.com] [The Background Intelligent Transfer Service service entered the running state.]
03:43:02 [System] [INFORMATION] [7040] [Service Control Manager] [mydomain.com] [The start type of the Background Intelligent Transfer Service service was changed from demand start to auto start.]
03:43:02 [System] [INFORMATION] [7036] [Service Control Manager] [mydomain.com] [The WinHTTP Web Proxy Auto-Discovery Service service entered the running state.]
03:45:02 [System] [INFORMATION] [7040] [Service Control Manager] [mydomain.com] [The start type of the Background Intelligent Transfer Service service was changed from auto start to demand start.]
This is my lambda function:
import boto3
import time
client = boto3.client('logs')
sns = boto3.client('sns')
instance_name = "Development"
a1 = int(round(time.time() * 1000))
def lambda_handler(event, context):
response = client.get_log_events(
logGroupName = 'Eadev',
logStreamName = 'i-01fe1z56y790cq',
startTime = a1,
startFromHead = False
)
event01 = '[System] [INFORMATION] [7036] [Service Control Manager] [mydomain.com] [The DebtManager-Host service entered the stopped state.]'
event02 = '[System] [INFORMATION] [7036] [Service Control Manager] [mydomain.com] [The DebtManager-Controller service entered the stopped state.]'
for i in response['events']:
if event01 == i['message']:
print(event01)
sns.publish( TargetArn = "arn:aws:sns:us-east-1:3913948:testsns",Message = instance_name +" "+ event01)
if event02 == i['message']:
print(event02)
sns.publish( TargetArn = "arn:aws:sns:us-east-1:3913948:testsns",Message = instance_name +" "+ event02)
I expected email notification from any service stopped, but I am receiving alert only when I clicked on test in Lambda function.
It appears that your desired situation is:
The Amazon CloudWatch agent on the Windows instance sends log data to Amazon CloudWatch Logs
Send a notification when a particular entry is detected in the log file
Rather than triggering a Lambda function for every log message, you can use CloudWatch Logs Filter Metrics to trigger a CloudWatch Alarm:
Collecting Metrics and Logs from Amazon EC2 Instances and On-Premises Servers with the CloudWatch Agent
Searching and Filtering Log Data to detect the desired messages by Creating Metric Filters
This pushes metrics into Amazon CloudWatch Metrics
You can then create a traditional Amazon CloudWatch Alarm on the metric and have it trigger when a certain number of such messages are received
A CloudWatch Alarm can send a notification an Amazon SNS topic
For an end-to-end example, see: Use Amazon CloudWatch Logs Metric Filters to Send Alerts - The IT Hollow
Alternatively, you can use an AWS Lambda function:
Collect Metrics and Logs from Amazon EC2 Instances and On-Premises Servers with the CloudWatch Agent
Use Real-time Processing of Log Data with Subscriptions
It can accept a subscription filter to identify the records of interest
It can then trigger an AWS Lambda function, which you can program to do whatever you wish (eg send a message to an Amazon SNS topic)
I have setup AWS SNS setup with a topic say 'A'. I'm doing a subscribe to this SNS topic using Http (tried both manually using AWS console online and using Java Code). All I get is 'pending confirmation' in both cases. However SNS does not send the initial 'SubscriptionConfirmation' to the provided Url.
Note that my endpoint is ready to receive http POST notification. When I manually do a POST from my side I see my servlet processing those Json I send. For some reason I receive nothing from AWS SNS.
Note that my http end point that I used for subscribe is public facing so SNS should have no issue reaching it.
Any inputs is appreciated.
Here is my subscribe function.
public String subscribe(String arn,String url) {
if(arn == null || arn.isEmpty())
arn = topicArn;
SubscribeRequest subRequest = new SubscribeRequest(arn,"http",url);
SubscribeResult result = snsClient.subscribe(subRequest);
//get request id for SubscribeRequest from SNS metadata
if(result != null){
LOGGER.info("SubscribeResult - " + result.toString());
}
LOGGER.info("SubscribeRequest - " + snsClient.getCachedResponseMetadata(subRequest));
return result.toString();
}
You are always going to get "pending confirmation" as the response for the subscriptionArn. The confirmation process is asynchronously as a separate process. To make this even more confusing if you call to get a list of current subscriptions they will show an slightly different subscriptionArn of "PendingConfirmation" so you can not even match it later.
As far as being able to connect, I would try hitting an end point outside of AWS first. By default most AWS elements are very locked down and can not even connect to each other, so there is likely a security setting somewhere that needs to be changed to let SNS connect to your EC2. Which would be why you can connect to the EC2 outside of AWS, but your SNS service can not.
Also check to make sure the SNS and EC2 you are using are in the same region. It is a common cause of connection issues.
If you are using a host name to connect I would try using the direct IP to see if it gets through.
To troubleshoot, you should turn on the "Delivery status" reports in topic actions - https://docs.aws.amazon.com/sns/latest/dg/sns-msg-status.html. Then you will see why the confirmation message failed to be sent from AWS side.
On your EC2 instance side, on network level you must make sure that the port you are listening on is open from outside. There are several things: both making sure the port is open in firewall (Security groups settings), and making sure that the IP is reachable (i.e., make sure your VPC where the machine is located is publicly visible).
I faced the same issue, the region was the problem.
Make sure the SNS, CloudWatch and EC2 are in the same region.
For me disabling encryption on the topic allowed the emails to finally be delivered, albeit to the spam folder.
I've got an app running on AWS. How do I set up Amazon CloudWatch to notify me when the EC2 instance fails or is no longer responsive?
I went through the CloudWatch screens, and it appears that you can monitor certain statistics, like CPU or disk utilization, but I didn't see a way to monitor an event like "the instance got an http request and took more than X seconds to respond."
Amazon's Route 53 Health Check is the right tool for the job.
Route 53 can monitor the health and performance of your application as well as your web servers and other resources.
You can set up HTTP resource checks in Route 53 that will trigger an e-mail notification if the server is down or responding with an error.
http://eladnava.com/monitoring-http-health-email-alerts-aws/
To monitor an event in CloudWatch you create an Alarm, which monitors a metric against a given threshold.
When creating an alarm you can add an "action" for sending a notification. AWS handles notifications through SNS (Simple Notification Service). You can subscribe to a notification topic and then you'll receive an email for you alarm.
For EC2 metrics like CPU or disk utilization this is the guide from the AWS docs: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/US_AlarmAtThresholdEC2.html
As answered already, use an ELB to monitor HTTP.
This is the list of available metrics for ELB:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/US_MonitoringLoadBalancerWithCW.html#available_metrics
To answer your specific question, for monitoring X seconds for the http response, you would set up an alarm to monitor the ELB "Latency".
CloudWatch monitoring is just like you have discovered. You will be able to infer that one of your instances is frozen by taking a look at the metrics, but CloudWatch won't e.g. send you an email when your app is down or too slow, for example.
If you are looking for some sort of notification when your app or instance is down, I suggest you to use a monitoring service. Pingdom is a good option. You can also set up a new instance on AWS and install a monitoring tool, like Nagios, which would be my preferred option.
Good practices that are always worth, in the long road: using load balancing (Amazon ELB), more than one instance running your app, Autoscaling (when an instance is down, Amazon will automatically start a new one and maintain your SLA), and custom monitoring.
My team has used a custom monitoring script for a long time, and we always knew of failures as soon as they occurred. Basically, if we had two nodes running our app, node 1 sent HTTP requests to node 2 and node 2 to 1. If any request took more than expected, or returned an unexpected HTTP status or response body, the script sent an email to the system admins. Nowadays, we rely on more robust approaches, like Nagios, which can even monitor operating system stuff (threads, etc), application servers (connection pools health, etc) and so on. It's worth every cent invested in setting it up.
CloudWatch recently added "status check" metrics that will answer one of your questions on whether an instance is down or not. It will not do a request to your Web server but rather a system check. As previous answer suggest, use ELB for HTTP health checks.
You could always have another instance for tools/testing, that instance would try the http request based on a schedule and measure the response time, then you could publish that response time with CloudWatch and set an alarm when it goes over a certain threshold.
You could even do that from the instance itself.
As Kurst Ursan mentioned above, using "Status Check" metrics is the way to go. In some cases you won't be able to browse that metrics (i.e if you;re using AWS OpsWorks), so you're going to have to report that custom metric on your own. However, you can set up an alarm built on a metric that always matches (in an OK sate) and have the alarm trigger when the state changes to "INSUFFICIENT DATA" state, this technically means CloudWatch can't tell whether the state is OK or ALARM because it can't reach your instance, AKA your instance is offline.
There are a bunch of ways to get instance health info. Here are a couple.
Watch for instance status checks and EC2 events (planned downtime) in the EC2 API. You can poll those and send to Cloudwatch to create an alarm.
Create a simple daemon on the server which writes to DynamoDB every second (has better granularity than Cloudwatch). Have a second process query the heartbeats and alert when missing.
Put all instances in a load balancer with a dummy port open that that gives a TCP response. Setup TCP health checks on the ELB, and alert on unhealthy instances.
Unless you use a product like Blue Matador (automatically notifies you of production issues), it's actually quite heinous to set something like this up - let alone maintain it. That said, if you're going down the road, and want some help getting started using Cloudwatch (terminology, alerts, logs, etc), start with this blog: How to Monitor Amazon EC2 with CloudWatch
You can use CloudWatch Event Rule to Monitor whenever any EC2 instance goes down. You can create an Event rule from CloudWatch console as following :
In the CLoudWatch Console choose Events -> rule
For Event Pattern, In service Name Choose EC2
For Event Type, Choose EC2 Instance State-change Notification
For Specific States, Choose Stopped
In targets Choose any previously created SNS topic for sending a notification!
Source : Create a Rule - https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CloudWatch-Events-Input-Transformer-Tutorial.html#input-transformer-create-rule
This is not exactly a CloudWatch alarm, however this serves the purpose of monitoring/notification.