When you send a message to a topic, if there is no subscription with a matching filter, the message gets lost.
How can I avoid this without creating a subscription that basically negates all the existing filters?
I don't want to lose messages, and if one comes without a subscriber, I want someone to look at it to see why that happened. I don't want this logic in the sender because the sender shouldn't be aware of subscribers.
I would like to avoid having to constantly update this "else subscriber" (ex, adding, deleting subscription).
Any ideas?
It would be nice if one could take subscriptions created with their rules, negate the rules and concatenate to the "else" type of subscription you're asking for. Unfortunately, there's no easy negation. And multiple rules are evaluated using OR logic, not AND. So the rules today are based on the accepting criteria only.
This is an interesting feature request you could raise with the Azure Service Bus team here.
`
Related
Consider the example use case as below.
You need to invite a Company as your connection. The sub actions that needs to happen in this situation is.
A Company need to be created by adding an entry to the Company table.
A User account needs to be created for the staff member to login by creating an entry in the User table.
A Staff object is created to ensure that the User has access to the Company by creating an entry in the Staff table.
The invited company is related to the invitee company, so a relation similar to friendship is created to connect the two companies by creating an entry in the Connection table.
An Invitation object is created to store the information as to who invited who onto the system, with other information like invitation time, invite message etc. For this, and entry is created in the Invitation table.
An email needs to be sent to the user to accept invitation and join by setting password.
As you can see, entries are to be made in 5 Tables.
Is it a good practice to do all this in a single API call?
If not, what are the other option.
How do I maintain data integrity if it is to be split into multiple APIs?
If the actions need to be atomic, then it's definitely best to do this in a single API call. Otherwise, you run the risk of someone not completing all the tasks required and leaving the resources in a potentially conflicting state.
That said, you're not updating a single resource, so this isn't a good fit for a single RESTful resource creation call (e.g., POST /companyInvitations) -- as all these other things being created and stitched together might lead to quite a bit of confusion.
If the action you're doing is "inviting a Company", then one option is to use Google's "custom method" syntax (POST /resources/1234:action) as defined in AIP-136. In this case, you might do POST /companies/1234:invite which says "I want to invite Company #1234 to be my connection".
Under the hood, this might atomically upsert (create if resources don't already exist) all the right things that you've listed out.
Something to consider when approaching an API call where multiple things happen when called, is how long those downstream actions take. Leaving the api call blocked isn't the best idea in the world while things are processing in the background.
You could consider (depending on your usecase) taking in the api request, immediately responding with a 200 status, and dropping the request onto an internal queue for processing. When your background service picks up the request it can update whatever needs to be updated and manage the transactions appropriately etc. This also caters for horizontal scaling scenarios where lots of "worker" services can be deployed to process the requests.
As part of this you could consider adding another "status" endpoint where requests can be made to find out how things are going. To avoid lots of polling status requests you could also take in callback details as part of the original api call which then gets called when the background processing is complete. Or you could do both!
Problem:
I know that Google Pub/Sub subscription cannot be patched to update the filter. I am just figuring out other ways to handle updates to filter in production.
Approach I could come up with:
Modify the push config to pull in existing subscription (old filter) so that it accumulates messages
Create a new subscription with latest filter
Transfer the messages from old subscription to a topic using dataflow
Detach the old subscription from the topic
Problems I see with the approach:
As both subscriptions exist at a point of time, I could end up processing duplicate messages
Any suggestions on the best way to handle this?
The timing is important to minimize the duplicates or the message loss.
Firstly, I will deploy a service (Cloud Run for exemple) that save the PubSub message as is somewhere (in a Cloud Storage file, in bigQuery, in Firestore,...)
Then, and in the same time, I will change the push of the old subscription to push to my Cloud Run service; and create the new push subscription with the new filter
Finally detach the subscription.
If you have the capacity, in your REAL app to detect the message already processed, you can remove them from your save location (it's easier with BigQuery for example) and then reprocess only the missing messages. With dataflow, or manually
However, it's recommended to have idempotent processing of your message. Keep in mind that PubSub is at least 1 delivery message and even with the same subscription you could have duplicates.
As you note, a filter expression cannot be changed once a subscription has been created. To effectively change a filter, please do the following steps:
Create a snapshot on the existing subscription (subscription A).
Create a new subscription (subscription B) with the new filter expression.
Seek subscription B to the snapshot created in step 1.
Change your subscribers to use subscription B instead of A.
Please note that during the time subscribers are switching from subscription A to B, there will be a high rate of duplicates delivered as messages are acked independently on the two subscriptions.
If you need to minimize the duplicates processed and can afford downtime, you can stop the all subscribing jobs/servers before step 1 and restart them configured to pull from the new subscription, B, after step 4. All steps must be completed well within the configured message_retention_duration to prevent message loss.
Consider sample chat application where user purchase monthly/annual subscriptions (subscriptions like Amazon Prime, etc).
As soon as the subscription expires, user should not be able to send messages in app.
User can end their subscription before the original subscription end date.
One solution in my mind (Frontend) - to cache the end date in app and before every "send message" operation, compare the end date and current date.
But the problem is - if user ends the subscription early, user will still be able to send the message.
How can I push update the new subscription end date in cache.
Another solution was (Backend) - I have a table in backend storing subscription details like subscription_id, user_id, subscription_enddate. So before any "send message" operation, query the subscription table and compare the dates and then continue/cancel further operations.
Q1. Should I go with Backend solution or can you please share some improvements to frontend method or any best practice for this scenario?
Q2. Also is storing subscription details in separate table best practice or any good design instead. ?
PS- Sample chat app is based on AWS Amplify Datastore
Let me try to breakdown the answer and give my opinion. I would also like to mention solutions to such problems are determined by the scale and various tradeoffs.
Q1-
If sending messages has an adverse effect, you should never rely on the frontend solution only as it is easy to bypass them. You can use a mixture to ensure that the load is not very high on the backend.
Adding a Frontend Cache for subscription will ensure you will be able to filter most of the messages on the frontend if the cache is not tampered with.
Adding a service before the queue, that validates whether the user subscription has expired adds one more layer of security. If the user subscription is valid it pushes the message to Queue else throws an error. This way any bad actor can also not misuse the system.
Q2-
Depending on the use-cases and load, you can have a separate table or a separate micro-service for the subscription itself.
When to have a separate micro-service?
When the subscription data is required from multiple applications in your system and needs to have its own scalability independent of others, it can be beneficial to have a separate micro-service.
When to have a separate table?
In other cases, where you feel adding a service would be overkill. You can keep the data separate in a different table/DB giving you the flexibility to change subscription and even extract it easily in the future.
I am relatively new to GCP platform. I have a system which is publishing messages in pubsub topic. Messages has an attribute named country and based on this attribute different clients would like to subscribe to messages. Client X is interested only in messages for country a. Client Y is interested only in messages of country b.
Will I have to create topic for every country in google pub sub?
or there is a smart way to have subscription on single topic based on attribute value
I am referring to attribute passed by publisher as stated in google docs.
https://cloud.google.com/pubsub/docs/publisher
Thanks
Update June 2020: Filtering is now an available feature in Google Cloud Pub/Sub. When creating a subscription, one can specify a filter that looks at message attributes. If a message does not match the filter, the Pub/Sub service automatically acknowledges the message without delivering it to the subscriber.
In this specific case, you can create a single topic and two different subscriptions, one that is meant to get messages for country a and one that is meant to get messages for country b. The filters for each subscription would be:
attributes.country = a
attributes.country = b
Previous answer:
The feature you are talking about is called filtering: you want the ability for a subscription to specify that it wants to receive a subset of the messages based on the attributes provided in the message. At this time, that feature does not exist in Google Cloud Pub/Sub.
There are two ways to handle this right now:
Filter the messages in the subscribers themselves by looking at the attributes and immediately acking all messages that they are not interested in. This does mean you will pay for delivery of all of the messages to each subscriber which may not be desirable depending on the percentage of messages the subscriber is actually interested in.
Create separate topics and a subscription on each topic, publish messages to those individual topics based on the attributes, and then have subscribers get messages on the subscription for the appropriate topic.
We are exploring ways to add functionality that will make this use case easier in the future.
I know, this is [normally] an RTFM question, except, well, we're talking about Authorize.net so the FM is hardly R-able.
I noticed when looking at the all the fields that get sent during the Silent Post, Authorize.net returns an item called "x_cust_id". It is not addressed in their sample code, and I can't find out where in the XML document that field would get posted. I'd like to use it!
So, how exactly do I add "customer Id" when I create a new ARB subscription?
Furthermore, is there any documentation that actually shows all the information I can send to ARB? I don't even know what my options are!
Silent Post != ARB. They are two different things. Silent Post is triggered whenever a transaction is processed by Authorize.Net. They send the results of each transaction to a script on your server notifying it of the results of the transaction. This allows you to manage subscriptions that fail programatically. It is very similar to Paypal's IPN. x_cust_id will be populated from the <customer/><id/> field in the ARBCreateSubscriptionRequest API call.
FYI, this is explained on pages 20 and 21 of the ARB guide.