So we spent almost 6 months in moving our current app stack from AWS to GCP and now we got stuck at the AWS-SNS part.
Apparently there seems to be no service in GCP stack that can simply provide a drop in replacement for SNS or did I miss something?
Right now everything is running perfectly fine on GCP and every call to SNS in the app is still using the old AWS account.
SNS is being used primarily to notify events occurring in the app like "batch completed successfully" or "export failed with xyz error".
What are our options here?
The closest SNS replacement in GCP is probably Cloud Pub/Sub, which provides a similar publish-subscribe messaging pattern.
This document shows how the topics and subscriptions work.
At present, GCP does not have a native / proprietary email or text message service. Instead, the recommendation is to use sendgrid for emails and twilio for messages. See the following articles that describe the usage of these services:
https://cloud.google.com/appengine/docs/standard/php/sms/twilio
https://cloud.google.com/appengine/docs/standard/php/mail/sendgrid
Related
Ref - https://docs.aws.amazon.com/appconfig/latest/userguide/working-with-appconfig-extensions-about-predefined-notification-eventbridge.html
Based on the above documentation, by enabling EventBrdige extension for AWS AppConfig, we can get notifications from AppConfig on actions like ON_DEPLOYMENT_START, ON_DEPLOYMENT_COMPLETE.
I am trying to implement a pub/sub type architecture using EventBridge and SNS Topics. The idea is that services will request for latest configuration from App Config once they receive a ON_DEPLOYMENT_COMPLETE event. It seems straightforward for 'AllAtOnce' deployment strategy that we will get a notification when the deployment is complete.
I would like to know how the notifications work with other strategies like 'AppConfig.Linear50PercentEvery30Seconds' where we incrementally roll out deployments to a certain percentage of hosts.
Can somebody help me with this?
Thanks!
In the strategies like AppConfig.Linear50PercentEvery30Seconds or AppConfig.Canary10Percent20Minutes, we get only one complete notification, same as AllAtOnce
This is not specifically described, but there are sentences like "If no alarms are received in this time, the deployment is complete." in the doc
It seems there is no way to get notified when each growth factor ends.
I'm looking to connect to a public websocket and receive the data into AWS Lambda or SNS.
Looking at similar posts, it seems the only way to do this is via EC2, ECS etc. These posts were from a few years ago so I'd first like to see if anyone has found a way to do this in a serverless manor?
Everything you are thinking of doing in EC2/ECS you should be able to do in AWS Lambda + a Cloudwatch rule to run this websocket data ingestion logic on whatever schedule you need. That said Lambda functions are optimized for running burst workloads and handling non-steady traffic, if you need to maintain a consistent real-time connection to this websocket and dont want to manage servers, your best option will be Fargate, which offers serverless compute for containers that you can use to build an application like yours.
Also, I highly recommend looking into AWS Copilot for a simple way to manage/deploy your application: https://aws.github.io/copilot-cli/
I'd like to get notifications for all standard error logs sent to Google Cloud Logging. Preferably, I'd like to get the notifications through Google Cloud Error Reporting, so I can easily get notifications on my phone through the GCP mobile app.
I've deployed applications to Google Kubernetes Engine that are writing logs to standard error, and GKE is nicely forwarding all the stderr logs to Google Cloud Logging with logName: "projects/projectName/logs/stderr"
I see the logs show up in Google Cloud Logging, but Error Reporting does not pick up on them.
I've tried troubleshooting as described here: https://cloud.google.com/error-reporting/docs/troubleshooting. But the proposed solutions revolve around formatting the logs in a certain way. What if I've deployed applications for which I can't control the log messages?
A (totally ridiculous) option could be to create a "logs-based metric" based on any log sent to stderr, then get notified whenever that metric exceeds 1.
What's the recommended way to get notified for stderr logs?
If Error Reporting is not recognizing the stderr logs from your container it means they are not displayed in the correct format for this API to be able to detect them.
Take a look at this guide on how to setup error reporting for GKE API
There are other ways to do this with third party products like gSlack where basically you will export the stackdriver logs to pub/sub and then send them into the Slack channel with Cloud Functions.
You can also try to do it by using Cloud Build and try to integrate it with GKE container logs.
Still, I think the best and easiest option will be by using the Monitoring Alert.
You can force the error by setting the #type in the context as shown in the docs. For some reason, even if this is the Google library, and it has code to detect that an Exception is thrown, with its stack trace, it won't recognize it as an error worth reporting.
I also added the service array to be able to identify my service in the error reporting.
I have been using the AWS IOT python sdk for developing a cloud based iot software as a part
of a larger solution.However I frequently get the subscribe and publish timeout exceptions after some time on every run. Though i don't see a pattern here i am able to receive the messages published for some time duration.
How do i fix this issue so that i do not get those exceptions?
Is there any parameter that i have to set to prevent the IOt Subscription issue?
Or is there any additional setup needed to be done on the cloud?
Thanks in advance.
I'm new to AWS, started working since last couple of months.
A requirement from the client is,
Get the daily count of `users` table from `AWS RDS` in an alert at 7 am pacific.
I can write a python script to do this and can run it from the aws instace by setting up a CRONJob or A lambda and cloud-watch schedule.
But I've heard from the client that there is something in the AWS (OR AWS RDS) which allows
To run an SQL (or a sequel ;)) Query
And send that query result in an email alert
He added that one of our colleague had done it for some other purposes (And sad part is colleague has left our org now :( ).
So I'm curious what he might have done directly from AWS or from RDS to send an alert notification.
Please suggest if anyone could have any idea on it.
Writing the query in a lambda function and using either SES or SNS to send the notifications is how I would do it - and either of those options would be doing it 'in AWS'.
Depending on the flavor of RDS you are using (SQL Server Aurora, Postgres etc), there may be a vendor specific way as well, but personally I'd still choose the lambda / cloud watch event method.