I am facing a different issue which is surprised to see. I have created one Event Hub namespace and within that there are 2 (a&b) event hubs with different policies. When I publish some messages to event hub "a" and I am able to subscribe the events from "b" as well. Is it something I missed in configuration? Please help.
Related
I am implementing subscriber in C# for Azure event hub, But it's not working.
There is no error in the logs, What could be the possible reasons for failure?
My code could be found here.
I have found the issues, I was using the event hub namespace key, But I had to use the key and connection string from the Shared Access Policies from the Event Hub.
There was one more issue, in command dapr run --app-id <consumer group name>, app-id should be the consumer group name of the event hub.
I'm currently facing a problem when thinking about a event driven arch using SNS to decouple some applications.
Imagine a SNS Topic, and I have application A producing messages to it and application B will listen and consume messages from this topic.
This application B has a autoscaling group attached to it, so it can scales to more than one instance. How will SNS handle when application B scales? If now I got 2 instances of application B, SNS will send the message for all of them or it can realize that they are the same application and just send the message to one of them?
Think of SNS as a radio broadcast: Everyone who is listening will get your message. Meaning that every single of your subscribed servers will get notified.
SQS, on the other hand, is more like a todo list. Many subscribers can also listen to it, but every message is distributed to at least someone. Meaning that usually, only one server will get triggered.
If that suits you better, then you might consider using SQS instead of SNS.
I'm not sure what your desired outcome is here, so I'm splitting the answer into two parts:
a) You only want to process each message once:
A common pattern in this case is to subscribe an SQS queue to the SNS topic, and then have N application servers polling from this queue. That way, you can make sure that you process each message only once.
b) You want to process each message once on each server:
In this case, you can create one subscription for each server to the SNS topic. Each message published to the topic will be delivered once to each subscription.
I have a process which publish some data(json) onto a queue on Aws-Sqs. Another process reads from this queue. All this is working fine.
Now I want to create a topic which can be listened by mutiple processes and the data is delivered to all the processes. For example Activemq and many other messaging servers have this capability to create a topic. I could not find any such thing on AWS. The closest I could find is AWS SNS.
From what I understand AWS-SNS allows multiple clients to subscribe to a topic. But the type of subscription is either Email, Http, or Sms and so on ... This does not really serve my purpose. I want to recieve json data in all my clients just like Sqs.
Is that achievable? If so how?
You can subscribe multiple SQS into single SNS topic: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-subscribe-queue-sns-topic.html
Then it will be distributed to all of them.
The other option is to use Kinesis - https://aws.amazon.com/kinesis/ but it is more difficult to set up. There you can also read from multiple clients from the stream.
amazon mq is a managed active mq service. maybe this will help with your needs?
I am using Azure Services with node js. I am pushing and getting the messages from service bus. I want to know what is the role that of event grid , event hub. When i create them they want end point what is that.
And how should i proceed
Service Bus, Event Hub, and Event Grid are three different Azure services.
In a nutshell;
Service Bus provides queue and pub/sub messaging typologies to decouple business applications. - https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-messaging-overview
Event Hubs provides a commit log for event driven applications and supports high volume data ingestion architectures and pipelines. - https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about
Event Grid provides solutions for event triggers for reactive apps. - https://learn.microsoft.com/en-us/azure/event-grid/overview
Why do you need Event Hubs or Event Grid? Can you elaborate your design a little more?
I'm trying to use Google Cloud Platform to make a little IoT project.
I've created a registry and a device in the "IoT Core" section and connected the registry to a default topic.
I've also specified three subfolders for that topic: "events", "config" and "status".
Now, I would like to connect a "Cloud Function" for the incoming messages but I can't find how to configure a single subfolder to monitor neither how to publish messages on them in the "Cloud Pub/Sub" section.
All the documentation talks about single topics so.... Am I missing some base concept on how it works?
Need to back up a step. What are you trying to do with the various subfolders? This may be a misunderstanding with how communication works with the device to and from the Cloud.
So, there's 3 MQTT topics, those are the events/config/state (not status). Those DON'T map to Pub/Sub topics at all. They're each handled separately in IoT Core.
Events is device->Cloud, and gets put into the Pub/Sub topic you specify when creating the registry. Setting subfolders is all about splitting telemetry from the same device to multiple places for handling. So, e.g. you have temp data you want to go in one pub/sub topic, and pressure data you want into another. The other way to handle this would be to attach the function to the main pub/sub topic, parse the telemetry, and then re-issue a Pub/Sub message to different places depending on the payload itself.
Config is Cloud->device, and is initiated by calls to the IoT Core Admin SDK. If your device subscribes to the /config/ MQTT topic, then they'll get a callback on the MQTT client's on_message handler (exact code depends on library being used of course) when some external entity sends a config message to IoT Core for your device.
State is device->Cloud but is specially handled, and doesn't go to any Pub/Sub topic. The results can be retrieved by the IoT Core Admin SDK. It's a way for the device to report its state and then external processes/applications to get that status without having to call back down to the device itself. This is particularly useful when you have devices which don't remain connected, for example, but you still want to be able to do things based on the last known state.
So the only one that you can use Cloud Functions with is the /events/ topic. That's done by deploying a Cloud Function and choosing the Pub/Sub publish event hook as the firing mechanism for the function, and specifying the IoT Core's registry Pub/Sub topic as the source of the events. Then anytime your device publishes telemetry to the /events/ MQTT topic, it'll get published to that Pub/Sub topic (confusing I know because we call them both topics) and the Cloud Function will fire.
Hopefully this clarifies what's going on? Or did I totally miss the question? :)
I believe the correct way to set this up would be to set up one pubsub topic for each subfolder. From the cloud iot core documentation:
Devices can publish data to separate Pub/Sub topics by specifying a subfolder in the MQTT topic. The subfolder is the subtopic after {device-id}/events. For example, if the device publishes to the MQTT topic /devices/{device-id}/events/alerts, the subfolder is the string alerts. This subfolder must be configured in the device registry resource's eventNotificationConfigs.subfolderMatches field with a matching Pub/Sub topic in the eventNotificationConfigs.pubsubTopicName field. When data is sent to a subfolder, it is published to the subfolder's matching Pub/Sub topic.
https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#publishing_telemetry_events_to_separate_pubsub_topics