I've recently been working on setting up RabbitMQ clusters on Google Computer Engine and AWS connected via federation. So far I've been able to get that working fine although I've encountered an issue that I can't figure out how to solve.
At a certain point, I wanted to see what would happen if I deleted all the VMs in the GCE cluster to then re-create them. I was able to bring the cluster back up, but the AWS cluster exchange that was previously federated, continued to hold the queued messages, even after a new federation link was created from GCE to AWS. All new messages on the AWS cluster were being retrieved via the federation link, but the old queued messages were not being sent also.
How could I get these old messages to also be sent onto the new federation link?
If the messages are already queued in the remote server, then you probably want to use shovel to solve this problem: https://www.rabbitmq.com/shovel.html
Related
Objective: UI test execution takes quite a time and we have a lot of UI test cases, currently we have a grid setup on AWS EC2 but scaling and descaling of resources manualy is time-consuming, so we decided to explore AWS ECS Fargate where we can scale based on CPU and Memory utilization.
Motivation blog: https://aws.amazon.com/blogs/opensource/run-selenium-tests-at-scale-using-aws-fargate/
Problem Statement: Node is initiating registration requests but it is not able to register itself to the hub.
Findings till now: I found a repo on git which is doing what we are trying to achieve except for one thing, that is in version 3.141.59 and we want the version 4.4.0-20220831
What I can achieve: So using this repo I changed the version of Hub and Node to 4.4.0-20220831 and also changed environment variables according to the specific version requirements, on the execution of cloudFormation template Hub is up and running but there was no node connected when I checked the log of hub and node, I found hub service was configured and running as well as the node service was sending registration requests for N times.
This is my first question here so I am not able to show images in question itself, sorry for inconveniance.
HUB Screenshots
Hub environment
Hub service discovery
Hub logs
Node Screenshots
Node environment
Node service discovery
Node logs
Before changing anything everyting is working as expected on V3 but we need V4.
Thank you for gving your valuable time, looking forward for you response.
Thank you once again.
The problem is not with any of these resources, when I allowed ports 4442 and 4443 in my security group it worked.
Thank you everyone for your time and support.
I have created a cluster in AWS EKS with 2 nodes, where it is a web based application and i am using EBS volumes on the Pods. I am able to communicate in initial login page with the server and can proceed with other steps as well. But sometimes for one particular module it is throwing an "internal server error:500" which i used to inspect option whether it is from the development code issue or from the AWS code issue.
Only for one particular Pod(service) the issue is coming and throwing an error whether it is from network related issue, which the communication is not happening or not. from the server the data is unable to pull or not unable to understand.
I have a Redis instance running in GCP Memorystore, and I have enabled notify-keyspace-events on this instance. My ultimate goal is to publish messages from my Redis instance when certain keys expire, and on these events, make a call to a service I have on Cloud Run with the data of the key as input.
How do I think about building this? Only way I can think is to have a thread always running in my Cloud Run instance to check for new messages in Redis Pub/Sub channels. I am afraid this might not work though as Cloud Run is not going to allow background tasks.
I am thinking of a way to generate a POST request to my Cloud Run service when the Redis message is generated, but could not find a way to do this yet.
What I know so far of that can be integrated together is Cloud Pub/Sub with Cloud Run as stated in these guides here and here.
What I don't know for sure is if you will be able to somehow publish events from your GCP Memorystore to a Pub/Sub topic. Maybe, if you are able to read in real time which Redis keys inspire, you could manually publish these events as messages to your Pub/Sub topìc, and then your Cloud Run subscribe to the same topic to receive the messages from it.
Another thing you could consider is using Cloud Background Functions.
As for sending a direct POST request to your Cloud Run service, the following documentation could be useful for you.
first time asker.
So I've been trying to implement AWS Cloud Watch to monitor Disk Usage on an EC2 instance running EC2 Linux. I'm interesting in doing this just using the CW Agent and I've installed it according to the how-to found here. The install runs fine and I've made sure I've created an IAM Role for the instance as is described here. Unfortunately whenever I run the amazon-cloudwatch-agent.service it only sends log files and not the custom used_percent measurement specified. I receive this error when I tail the logs.
2021-06-18T15:41:37Z E! WriteToCloudWatch failure, err: RequestError: send request failed
caused by: Post "https://monitoring.us-west-2.amazonaws.com/": dial tcp 172.17.1.25:443: i/o timeout
I've done my best googlefu but gotten nowhere thus far. If you've got any advice it would be appreciated.
Thank you
Belated answer to my own question. I had to create a security group that would accept traffic from that same security group!
Having the same issue, it definitely wasn't a network restriction as I was still able to telnet to the monitoring endpoint.
From AWS docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent.html
One role or user enables CloudWatch agent to be installed on a server
and send metrics to CloudWatch. The other role or user is needed to
store your CloudWatch agent configuration in Systems Manager Parameter
Store. Parameter Store enables multiple servers to use one CloudWatch
agent configuration.
If you're using the default cloudwatchagent configuration wizard, you may require extra policy CloudWatchAgentAdminRole in your role for the agent to connect to the monitoring service.
I am building some form of a monitoring agent application that is running on AWS EC2 machines.
I need to be able to send commands to the agent running on a specific EC2 instance and only an agent running on that instance should pick it up and act on it. New EC2 instances can come and go at any point in time.
I can use kinesis and push all commands for all instances there and agents can pick up the ones targeted for them. The problem with this is that agents will have to receive a lot of commands that are not for them and filter it out.
I can also use SQS per instance, but then this will require to create/delete SQS every time new instance is being provisioned.
Would like to hear if there are already proven solutions for a similar scenario.
There already is a fully functional feature provided by AWS. I would rather use that one as opposed to reinventing the wheel, as it is a robust, well-integrated, and proven solution that’s being leveraged by thousands of AWS customers to gain operational insights into their instance fleets:
AWS Systems Manager Agent (SSM Agent) is a piece of software that can be installed and configured on an EC2 instance (and it’s pre-installed on many of the default AMIs, including both versions of Amazon Linux, Ubuntu, and various versions of Windows Server). SSM Agent makes it possible to update, manage, and configure these resources. The agent processes requests from the Systems Manager service in the AWS Cloud, and then runs them as specified in the request. SSM Agent then sends status and execution information back to the Systems Manager service by using the Amazon Message Delivery Service.
You can learn more about AWS Systems Manager and the breadth and depth of functionality it provides here.
Have you considered using Simple Notifications Service? Each new EC2 instance could subscribe to a topic using e.g. http, and remove previous subscribers.
That way the topic would stay constant regardless of EC2 rotation.
It might be worth noting that SNS supports subscription filters, so it can decide which messages deliver to which endpoint.
To my observation, AWS SWF could be the option here. Since Amazon SWF is to coordinate work across distributed application components and it provides SDKs for various platforms. Refer to the official FAQs for more in-depth understanding. https://aws.amazon.com/swf/faqs/
Not entirely clear what the volume of the monitoring system messages will be.
But the architecture requirements described sounds to me as follows:
The agents on the EC2 instances are (constantly?) polling some centralized service, which is a poll based architecture
The messages being sent are to a specific predetermined EC2 instance, which is a push based architecture.
To support both options without significant filtering of the messages I suggest you try using an intermediate PubSub system such Kafka, which can be managed on AWS by MSK.
Then to differentiate between the instances, create a Kafka topic named by the EC2 instance ID.
This should give you a unique topic that the instance will easily know to access messages for itself on a topic denoted by it's own instance ID.
You can also send/push Producer messages to a specific EC2 instance by sending messages to the topic in the cluster named by it's EC2 instance ID.
Since there are many EC2 instances coming and going you will end up with many topics. To handle the volume of topics, you can trigger and notify CloudWatch on each EC2 termination event and check CloudWatch to see which EC2 instances were terminated and consequently their topic needs deleting.
Alternatively, you can trigger a Lambda directly on the EC2 termination event event and log it by creating a file denoted by the instance ID to an S3 Bucket, which you can watch using an additional Lambda that will delete old EC2 instance topics from the Kafka cluster when their instance ID's appear there.