In the Riemann stream processing server, when we look at the internal Riemann metrics there is a metric value "Riemann index size"? What causes this value to increase? Is it based on the number of transport listeners configured to accept events (TCP, UDP, graphite)?
Related
I'm trying to design an autoscaling policy to scale out/in the number of consumers listening to a queue. My first instinct was to base the scaling policy off EnqueueTime which if too high should result in scaling out and scale in when low.
However, the way EnqueueTime appears in Cloudwatch does not seem to match my expectations. From the documentation, EnqueueTime is defined as
The end-to-end latency from when a message arrives at a broker until it is delivered to a consumer.
Note:
EnqueueTime does not measure the end-to-end latency from when a message is sent by a producer until it reaches the broker, nor the latency from when a message is received by a broker until it is acknowledged by the broker. Rather, EnqueueTime is the number of milliseconds from the moment a message is received by the broker until it is successfully delivered to a consumer.
I had expected EnqueueTime to represent how long a message will "wait" in the queue until consumed, but from the screenshot, it is not clear to me how the supposed "wait time" is 1.9s despite there being nothing in the queue and no message production (EnqueueCount = 0). I also don't understand why EnqueueTime does not change well after the spike in traffic (the green spike). I expected the value to be close to 0ms after the spike. The metric not changing affects scaling because if the metric does not change, then the policy might erroneously scale out despite there being no traffic.
I'm also new to using ActiveMQ and am not entirely familiar with its operations. I would greatly appreciate it if somebody could somebody explain what's going on here and how to properly interpret EnqueueTime.
EnqueueTime does represent how long messages "wait" in the queue until they are consumed, but it is important to note that it is an average. Therefore, it is unlikely to fit your use-case because the relative "weight" of each message's individual EnqueueTime will change over time. It won't give you a reliably clear picture of the queue's load relative to consumption.
I have developed an application that does simple publish-subscrible messages with AWS IOT Core service.
As per the requirement of AWS IoT SDK, I need to call aws_iot_mqtt_yield() frequently.
Following is description of function aws_iot_mqtt_yield
Called to yield the current thread to the underlying MQTT client. This
time is used by the MQTT client to manage PING requests to monitor the
health of the TCP connection as well as periodically check the socket
receive buffer for subscribe messages. Yield() must be called at a
rate faster than the keepalive interval. It must also be called at a
rate faster than the incoming message rate as this is the only way the
client receives processing time to manage incoming messages. This is
the outer function which does the validations and calls the internal
yield above to perform the actual operation. It is also responsible
for client state changes
I am calling this function at a period of 1 second.
As it is sending PING on tcp connection, it creates too much internet data usage in long run when system is IDLE for most of the time.
My system works on LTE as well and paying more money for IDLE time is not acceptable for us.
I tried to extend period from 1 second to 30 seconds to limit our data usage but it adds 30 seconds latency in receiving messages from cloud.
My requirement is to achieve fast connectivity with low additional data usage in maintaining connection with AWS.
Number Of Messages Received is larger in SQS as compared to Number of messages sent.
Almost from a month my count for both the above metric that is number of messages received and sent are equal but suddenly the number of messages received count started increasing as compared to number of messages sent.
As per my understanding number of messages sent is the count of messages added to the SQS and number of messages received is the count of number of messages received by the consumer. So how the number of received messages can be larger than sent messages.
Also the approximate number of visible messages are growing.
Can anyone please help by explaining how this can happen.
Thanks in advance.
The metric includes retries. From docs:
These metrics are calculated from a service perspective, and can include retries. Don't rely on the absolute values of these metrics, or use them to estimate current queue status.
So your application may be trying to process same messages multiple types.
I want to know when my consumer group fails to handle the amount of events coming in into EventHub. Looking at the metrics, I think the symptom is incoming bytes exceed outgoing bytes.
In Azure portal, I only see alert condition when incoming bytes greater than a static number, which is not what I want. Is it even possible to set up condition like this?
I have checked Azure Monitor docs and alerts don't seem to support metric comparison.
One solution might be to create an Azure Logic App which can store and remember incoming/outgoing byte values and then compare them on a scheduled basis.
For variable support you can look at - https://learn.microsoft.com/en-us/azure/logic-apps/logic-apps-create-variables-store-values
Consider routes containing all the HTTP services
val routes:Route = ...
I wish to throttle number of requests so I used Route.handleFlow(routes) to create flow and called throttle method with finite duration.
Finally, I created HTTP binding using
Http().bindAndHandle(flowObjectAfterThrottling, hostname, port)
When HTTP requests are fired from a loop throttling is not obeyed by akka.
One possibility is that the http requests being "fired from a loop" may be using separate connections. Each incoming connection is throttling at the appropriate rate but the aggregate throughput is higher than expected.
Use Configurations Instead
You don't need to write software to set limiting rates for your Route.
If you are only concerned with consumption of a resource, such as disk or RAM, then you can remove the rate logic and use akka configuration settings instead:
# The maximum number of concurrently accepted connections when using the
# `Http().bindAndHandle` methods.
max-connections = 1024
# The maximum number of requests that are accepted (and dispatched to
# the application) on one single connection before the first request
# has to be completed.
pipelining-limit = 16
This doesn't provide the ability to set a maximum frequency, but it does at least allow for the specification of a maximum concurrent usage which is usually sufficient for resource protection.