If I were to create a client desktop application, I'm trying to find a reliable way to notify client applications of new data that needs to be queried from the server. Would pubsub be a good use for this? Most of the documentation I see for it seems to be focused on server to server communication, and is a bit ambiguous if this would work well for server to client notifications.
If it should work, would I be able to properly authenticate subscribers to limit the topics they could subscribe to? This application would be potentially downloadable by anyone, and I would need to ensure that information intended for one client couldn't end up in the hands of another client.
Cloud Pub/Sub is not going to be a good choice for this use case. First of all, note that each topic and project is limited to 10,000 subscriptions. Therefore, if you intend to have more than that, you will run out of subscriptions. Secondly, note that a subscription only receives messages published after the subscription is created. If you only need messages to be delivered that were published after the user came to the website, this may be okay. However, with these two issues combined, you'll need to consider lifetime of your subscriptions. Do they get deleted when a user logs out? If not, when a user comes back, do you expect them to get all of the messages published since the last time they visited?
Additionally, as discussed in the comments, there is the issue of authentication. Your client-side app would have to have the credentials to subscribe. This would require you to essentially leak those credentials into your client-side code, which could be a vulnerability in your application.
The service designed to deliver notifications of this nature is Firebase Cloud Messaging.
If you want to open the application to anyone on the internet, you can't rely on the IAM service that only works with Google identity -> You can't ask your user to have a Google Account, the user experience will be bad.
Thus, you can't use IAM service to secure the PubSub access, and thus to use PubSub because anyone could access it.
In your use case, the first step is to ask the user to register (create an account, validate email, maybe use payment method,...). Then, you have an identity, but managed by you, not by IAM. You know which messages are for this user and which aren't.
If you want to be notified "in real time", I propose you to use long polling method or streaming to push data to the user. Cloud Run is now capable to do this and I recommend you to have a look on that.
Related
I want to create a button that will open GCP cloud shell and run code that create some resources in the account.
I am trying to use "Open in Cloud Shell" (https://cloud.google.com/shell/docs/open-in-cloud-shell) URL and adding my GIT repo to the URL, but the problem is that my code should get different arguments in every run. There is a way to send arguments with this URL? Or maybe there is another solution for running code with arguments in GCP cloud shell via URL?
This is NOT a direct answer to your original question however it might be useful for an overall answer. If we don't like this answer, simply let me know and we'll delete it.
From you clarification in the comments, what I now sense is that you want to create GCP resources that the user can work with. For example, a PubSub topic. We'll use that as an illustration. The first thing I want to do is disavow us of the notion that there is anything "special" about a resource and the identity that it used to create that resource other than the identity must have authority to create it. For example, if user "john" creates a topic, that doesn't mean that the topic is "owned" by john. A GCP resource "just exists" after it is created. In order for a user to "use" a resource, it (the resource) must authorize the sets of users to work with it. This is where GCP IAM comes into play. Separate your goal into two parts.
Upon request, a new GCP topic is created
Once the GCP topic is created, you grant permissions on the topic to be worked with by named identities (users/groups)
Don't think "The user who creates the topic is immediately the one who can work with it".
For example, you may wish to grant your users the ability to subscribe to a topic but may not want those users to be able to "manipulate" topics such as creation/update/delete.
I am assuming that the solution you are working against is for end users rather than internal developers?
Off the top of my head, I'm tempted to suggest that you review the following very short video:
How to authenticate calls to your Google Cloud Run service
This is just a teaser but it does give us a clue. It alludes to the notion that a request from an authenticated (to Google) user can be received by a Cloud Run instance and Cloud Run can then know who the user is. With that in mind, in the code of your Cloud Run, you can then make a "yes/no" decision as to whether to proceed. If yes to proceed, then Cloud Run (which is indeed running as a single user and we won't change that) creates the topic and then assigns subscription (or publication or other) permissions to the topic on behalf of the identity that came in with the request.
I'm studying GCP and reading about different ways to communicate and manage cloud functions I end up wondering when to use each of the services that offer GCP.
So, I have been reading about GCP Composer, GCP Workflows, Cloud Pub/Sub and I don't see clearly when to use each one, or use simple HTTP calls.
I understand that it depends a lot on the application that you are building, but for example, If I'm building a payment gateway and some functions should be fired after the payment was verified, like sending emails, making not related business logic, adding the purchase to a sales platform. So which one should be the way I manage this flow and in which case would be better to use the others? Should I use events to create an async flow with Pub/Sub, or use complex solutions like composer and workflows? or just simple HTTP calls?
As always, it depends!! Even in your use case, it depends! Ok, after a payment you want to send an email, make business logic, adding the order to your databases,...
But, is all theses actions can be done in parallel, or you need to execute them in a certain order and if a step fails, you stop the process?
In the first case, you can use Cloud PubSub with 1 message published (payment OK) and then a fan out to several functions in parallel. Else, you can use workflow to test the response of the fonction and then to call, or not the following fonctions. With composer you can perform much more checks and actions.
You can also imagine to send another email 24h after to thank the customer for their order, and use Cloud Task to delayed an action.
You talked about Cloud Functions, but you also have other solutions to host code on GCP: App Engine and Cloud Run. Cloud function is, most of the time, single purpose. Sending an email is perfect for a function.
Now, if you have "set of functions" to browse your stock, view the object details, review the price, and book an object (validate an order "books" the order content in your warehouse), the "functions" are all single purpose but related to the same domain: warehouse management. Thus you can create a webserver that propose different path to manage the warehouse (a microservice for the warehouse if you prefer) and host it on CloudRun or App Engine.
Each product has its strength and weakness. You will also see this when you will learn about the storage on GCP. Most of the time, you can achieve things with several product, but if you don't use the right one, it will be slower, or cost much more.
I am working on an app that relies heavily on detecting when users go offline and go back online. I wanted to do this with AWS AppSync, but I can't seem to find a way to do this in the documentation. Is there a way to do it in AppSync?
Thanks for the question. Detecting presence is not currently support out of the box but you can likely build similar features yourself depending on the use case.
For example, a resolver on a subscription field is invoked every time a new device tries to open a subscription. You can use this resolver field to update some data source to tell the rest of your system that some user is currently subscribed. If using something like DynamoDB, you can use a TTL field to have records automatically removed after a certain amount of time and then require a user to "ping" every N minutes to specify that they are still online.
You could also have your application call a mutation when it first starts to register the user as online, then have the application call another mutation when the app closes to register it as offline. You could combine this with TTLs to prevent stale records in situations where the app crashes or something prevents the call to register as offline.
Thanks for the suggestion and hope this helps in the meantime.
I am trying to write a Kafka connector to fetch data from the facebook. The problems are,
How to fetch data from facebook through their API without exceeding the limit of API hit provided by facebook? The connector should call facebook API for data after a specific time interval so that the number of hits won't exceed.
Each user can hit the facebook API with their Access Token so users can't share the same topic partition. So how to handle this scenario. Do we have to create one partition for each user?
I read a few guides and blogs to understand Kafka connect and write a connector.
Confluent- https://docs.confluent.io/current/connect/index.html
Kafka Documentation- https://kafka.apache.org/documentation/#connect
Conceptually It gave me an idea about what is Kafka connect, how it works and what are the important classes to write a Kafka connector. But still, I am confused that practically how to write and run a connector. I tried to find step by step development guide but didn't get.
Any tutorial or pdf If you could suggest which have detailed step by step development guide to write and run Kafka connector.
The only "official guide" is in those links you have
https://docs.confluent.io/current/connect/devguide.html#developing-a-simple-connector
I personally have no experience with the Facebook API, but I assume it uses REST, so you could make start by forking the kafka-connect-rest project, but the simplest answer to not exceed the limit would be to not send more requests than you are allowed within a given time period (add a timer to the code that waits between requests)
Also, one connector would only have one set of access keys. How you create the ConnectRecord objects to ultimately partition the records is up to you, but I don't think having an access key per user will scale very well. It might make more sense to have one key tied to one application, then each user will accept that that application has access to read certain details from their account.
I have integrated my amazon echo device with the amazon portal associated to my account. I was able to create my own custom question with the Alexa Skills Kit and process with an AWS Lambda function to generate a response.
My question is: is it possible to programatically "push" a message to the echo device? For example, I would like for it to speak without having to ask it a question. I'd like it to do something at a specific moment.
If it is possible, could you please share any sample code to achieve this?
It is not currently possible, but it is an oft requested feature on the AWS forums.
http://forums.developer.amazon.com/forums/thread.jspa;jsessionid=EC0D457A400B594DD0F0561EEB43A8FA?messageID=17713䔱
I've not done this myself but it seems using the Alexa Voice Service could do the trick. It allows processing of voice from any type of audio capture and sends it to the Alexa Service. It seems possible you could record the proper phrase into a sound file and send that to AVS, thus triggering the Alexa service.
I know it's capable but Amazon hasn't offered it as a feature yet. If you go to the Echo web site http://alexa.amazon.com/spa/index.html#cards, Settings, Connected Home and select Discover devices, the echo will perform a command triggered from the web site and will speak when completed. I didn't have to say a word.
From what I read on an article about notifications here, they were going to do something that meant you still had to ask Alexa to tell you your notifications, which would build up throughout the day and cause your device to light up and chime to let you know you had a notification to listen to.
Rather than allowing Alexa to randomly blurt stuff out any time she feels like it.
Which kinda seems pointless if you don't get the notification at the exact time it was sent, for example a smart home connected device triggering an alert to let you know you left your door open too long or the app automatically reading you the weather when you get up and turn on the coffee pot in a morning, things like that. If you have to ask for the notifications you may as well just do the speaking and request stuff from the beginning.
Maybe I'm wrong but that's how it reads to me.
Surely we want, with opt-in permission per skill, to allow Alexa to just say whatever she likes whenever she likes from the skills we have set up receiving such commands, without us having to say oh I see I have notifications let me ask what those are.
Just for people who stumble across this question in 2021:
The solution is to use Amazon Proactive Events
Your Skill has to request for the notification permission, subscribe for a specific event and then you can generate a access token and POST events to the amazon API.
Took me some hours to find out because Amazon offers different things which all sound quite similar but some are deprecated (ASK CLI v1) and others are just for Alexa device manufacturers.
Wuhu! Sounds like Amazon may enable push notifications this Fall.
Digital Trends Article
Amazon is expected to establish guidelines for developers and manufacturers so that Alexa remains classy and doesn’t become an interrupting nag.