Is there an easy way to export GCP cloud run logs to Grafana Cloud in anyway?
Perhaps with loki or another exporter tool to output to Grafana...or?
I can't find any guides which even go over this logging exporting process.
GCP Cloud Run logs are sent to GCP Cloud Logging. So, how to get Cloud Logging into Grafana. Grafana documentation shows:
Pull-based subscription: Promtail pulls log entries from a GCP PubSub topic
Push-based subscription: GCP sends log entries to a web server that Promtail listens
The outline of the basic steps is:
Roles and Permission
Setup Pubsub Topic
Setup Log Router
Grant log sink the pubsub publisher role
Create Pubsub subscription for Grafana Loki
Pull
Push
ServiceAccount for Promtail
Operations
Related
I have a template that creates a few resources in GCP, and I want it to either call an HTTP endpoint or publish a message to a topic whenever the deployment completes. I've been checking different services all day, and couldn't find anything about it.
In AWS, it is quite easy to deploy an SNS message that is published to an SNS topic, which is subscribed to SQS Queue, and that triggers a lambda function. In Azure ARM templates, we can use az CLI to invoke a web request and call an endpoint directly.
I couldn't find any similar in GCP. Is there any way of either calling an HTTP endpoint, Cloud Function or perhaps publishing a message to a topic whenever a deployment is finished?
I really appreciate any help.
The best approach in GCP is to Create a Logging Sink using a filter and the Logging query language to only send the Deployment Manager logs to a PubSub topic.
Here is an example of a sink sending deployment manager logs to a PubSub topic previously created:
gcloud logging sinks create $SINK_NAME pubsub.googleapis.com/projects/$YOUR_PROJECT/topics/$TOPIC \
--log-filter='resource.type="deployment"' --description="my sink"
Be careful to Set the destination permissions or will not see the logs in the PubSub topic.
Once you are getting the logs in the PubSub topic, you can configure Cloud Pub/Sub Triggers to fire up an HTTP call based on content of the log.
Does anyone successfully set up GCP Cloud Build trigger via pub/sub notification with Cloud Storage OBJECT_FINALIZE event (new object uploaded)?
I have followed the GCP documentation, but nothing worked. This is what I did:
Create a topic
Create a notification via gsutil as per documentation which sends a message to the topic upon OBJECT_FINALIZE event on Google Cloud Storage (https://cloud.google.com/storage/docs/reporting-changes)
Create the Cloud Build trigger with Pub/Sub subscription to the topic above
I also ensure that Cloud Build API is enabled and the cloud build service account has permissions to read the message from the topic
But when i uploaded new file to the Cloud Storage bucket, the build pipeline was never triggered.
Please help if you already set this up before. Thanks.
Suppose multiple aws services are running (like EC2,S3) and If someone changes the configuration of these services , I want to know immediately and want to pull the updated configuration. So how can i listen to these services to pull configuration immediately . I went through couple of options like aws SNS and all. But i don't want to implement this using any aws services for this.
I am using spring boot application. Any help is appreciated , Thanks in advance.
The two services you should be interested in are:
AWS CloudTrail:
AWS CloudTrail is an AWS service that helps you enable governance, compliance, and operational and risk auditing of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs.
AWS Config:
AWS Config provides a detailed view of the configuration of AWS resources in your AWS account. This includes how the resources are related to one another and how they were configured in the past so that you can see how the configurations and relationships change over time.
You can also configure Amazon CloudWatch Events rules that trigger when a particular API call is made (eg when a user changes some configuration). This could trigger a notification, a Lambda function, etc.
See:
Creating a CloudWatch Events Rule That Triggers on an AWS API Call Using AWS CloudTrail - Amazon CloudWatch Events
How to monitor AWS account activity with Cloudtrail, Cloudwatch Events and Serverless
I am planning to export logging from MuleSoft CloudHub to AWS CloudWatch.
I saw there is one AWS CloudWatch Connector in GitHub:
https://github.com/mulesoft-labs/mule-amazon-cloudwatch-connector/tree/master/mule-cloudwatch-connector
Is there any examples on how to implement this?
And which AWS CloudWatch features is supported in this connector?
I found other export logs method with examples, but not for AWS CloudWatch:
https://help.mulesoft.com/s/question/0D52T00004mXUALSA4/export-log-to-external-system
Thanks.
You could try to create a Mule application that reads logs from CloudHub using CloudHub's API and pushes them to CloudWatch. The readme for the connector doesn't seem to mention an operation to put logs into CloudWatch, but if CloudWatch has a REST API for it -every AWS product has it- you could use it in your application.
This KB article shows how to get the logs from CloudHub: https://help.mulesoft.com/s/article/How-to-get-whole-Application-logs-from-Cloudhub-through-API
When deploying a background Cloud Function for Cloud Pub/Sub via:
gcloud functions deploy function_name --runtime python37 --trigger-topic some_topic
A subscription gets automatically created with a push endpoint (likely App Engine standard endpoint, but those are claimed to be without the need of
domain verification https://cloud.google.com/pubsub/docs/push#other-endpoints).
For the generated subscription/endpoint there doesn't seem like a way to register/verify the domain
(https://www.google.com/webmasters/verification, https://console.cloud.google.com/apis/credentials/domainverification)
As a result the user cannot alter the subscription, for instance:
gcloud alpha pubsub subscriptions update some_subscription --ack-deadline=10
Would yield something like "INVALID_ARGUMENT: The supplied HTTP URL is not registered in the subscription's parent project"
Doing it via the web interface on GCP would also yield an error: "pubsub error INVALID_ARGUMENT"
When connecting to a Cloud Pub/Sub topic via Cloud Functions, a subscription is created on your behalf. This subscription is managed by Cloud Functions and therefore it is not possible to update the acknowledgement deadline.