I am trying to debug a pipeline failing one of its action with an error 403.
I am reaching cloud trail to find more detail about the action, but I can not find the event linked to the request. When I search in cloud trail for the request id, it gives me nothing.
I am in only one region and only one account.
I believe all action are supposed to be log into cloud trail, specially the one with error, am I wrong?
Do I need to configure some service to actually turn on event sent to cloud trail?
Am I missing something?
CloudTrail is not a tool used for troubleshooting. Essentially it's an auditing tool that records changes to resources in your AWS accounts.
What is your pipeline built on? If it's on CodePipeline, you should be able to view the details on the console, and you can configure to send the pipeline logs to CloudWatch. Additionally, CloudTrail should record the API calls made by CodePipeline as well, although it may not be detailed enough to actually troubleshoot an error - https://docs.aws.amazon.com/codepipeline/latest/userguide/monitoring-cloudtrail-logs.html
Related
I have created an alert policy in google cloud platform. I am getting the emails notifications based on the alert policy.
Now I want to configure an external custom monitoring system for this alert. I want to know the REST APIs that this monitoring system can call at after every 10-20 sec and get the status of this alert.
Please help
If you want to use only Cloud Monitoring, and Alerting Policy you won't achieve this.
It is not possible to configure alert policies to notify (repeatedly) while the policy's conditions are met. Alert policies that are created through Google Cloud Console send a notification only when the condition is met. You can also receive a notification when the condition stops being met.
Additional information can be found in Notifications per incident documentation.
In Cloud Monitoring API v3 - Alerting policies, you can find information that only the creation of the incident is sent.
An alerting policy is a configuration resource that describes the criteria for generating incidents and how to notify you when those incidents are created.
In general, if you want to use Notification Channels to send notifications outside you can use Webhooks or PubSub.
Note
Webhooks only support public endpoints. If you need notifications sent to an endpoint that isn't public, then create a Pub/Sub notification channel and configure a subscription to the Pub/Sub topic. For more information, see Webhook notifications fail when configured for a private endpoint.
As you didn't provide more information it's hard to say if you are not using some built-in features in 3rd party software to integrate with GCP Cloud Monitoring. One of the example is Grafana:
Grafana ships with built-in support for Google Cloud Monitoring. Add it as a data source to build dashboards for your Google Cloud Monitoring metrics.
GCP also might use Prometheus features. Maybe this might give you something similar to what you want.
Prometheus is a monitoring tool often used with Kubernetes. If you configure Cloud Operations for GKE and include Prometheus support, then the metrics that are generated by services using the Prometheus exposition format can be exported from the cluster and made visible as external metrics in Cloud Monitoring.
There are some workarounds, however they won't fulfill what you want.
It is possible to create multiple conditions that identify the same issue. Every time a condition is met, a notification will be received.
It is possible to get users notified when a condition is NOT met, however this might cause spam messages.
The last thing I want to mention is that there is already a Feature Request to add multiple notifications until the condition is gone. More details in FR: Repeat Notifications until condition is gone.
Additional Documentation:
Monitoring Alerts in GCP by integrating Cloud Operations with Notification Channels
Conclusion
Alert policies that are created through Google Cloud Console send a notification only when the condition is met. You can also enable notification to get solved notification.
There is Feature Request to add repeatedly notifications - here
To send notifications to other apps/resources you can use Webhooks or PubSub.
I have been exploring to transfer the JsonPayload message field from Logs viewer service (which are syslogs of a service) of GCP to a slack network, but owing to this I am not able to find any predefined services (like alerting policies to transfer Payload) available on Stackdriver. I have been able to create a counter or distribution user-metrics for logs but this will only provide me with some int64 value instead of a string value or the actual message body. Is there a way in GCP to actually send a payload of logs over slack or any email?
We had a similar issue where we wanted to be able to send certain events to slack and for fatal issues trigger an issue with our ops team via VictorOps.
Couldn't find anything out there to fit our needs so we just created our own slack / VictorOps Cloudfunction.
https://github.com/patiently/gcloud-slack-logger
In GCP, you can export logs to Pub/Sub, Cloud Storage, or BigQuery. There is no other way within GCP to export logs at the moment.
As of 2022, I found this can be done as follows:
In GCP Logs viewer (not legacy version) choose the create alert button.. One of the options here is a GCP notification channel, which supports slack. Some points here:
The slack channel can't be private as far as I can tell
Slack channel must be in your correct slack space. If your org has multiple slack spaces, make sure GCP is trying to connect to correct one.
Put in the log query criteria you want. THen go into Monitoring and you will see this in Alerting dropdown.
I am preparing for AWS exam and I found some documentation about AWS CloudTrail and AWS X-RAY where it creates confusion on their usage requirement.
I have came across following question where requirement was to trace and analyse the user request as it travels through Amazon API Gateway APIs to underlying services.
As per my understanding, we can use CludTrail to trace and analyse the user request. But the correct answer was AWS XRAY.
The documents which have referred mentions that, we can use AWS CloudTrail logs for tracing,Security Analysis, Resource Change Tracking and Compliance/Auditing. On the other hand, we can use AWS X-RAY to analyse and debug applications running on distributed micro service architecture.
XRAY and CloudTrail usage both have the term Analyse and trace. So it is quite confusing to which service should we choose under such requirement to trace and analyse the user request
X-Ray is more detailed in the information it provides for the request's flow and state. It scans the request all the way through its lifetime from when it is received in the api gateway to whatever services are called and executed after that. So I imagine that is why it is the preferred option.
In AWS API Gateway I am trying to deploy API to new stage, and im getting error "An unknown error occurred". I am assuming it most likely because of permission. But i have API Gateway administrator permissions.
How do find out exactly which permissions are missing? Does aws api gateway logs any deployment related events? Where do i find whats going on.
Answering your question of Api Gateway deployment events, below are typical steps I do on what's happening.
Any operation you perform on Console or SDK is essentially calling AWS APIs.
For Console failures, you can always check on client side by going to Developer Tools provided by browsers. Typically F12 key --> Network tab, replicate the issue and see which AWS Endpoint gave error or go to Console tab and see any JavaScript errors logged.
If I am still not able to determine the cause, I enable CloudTrail logs and see which user and which API call gave any error responses for which operation. CloudTrail Guide to ApiGateway.
If there is specific role or policy issue having issue, I use Policy Simulator to test policies for the role once permissions are added.
I have an AWS account that use multiple devs and teams [dev/qa/mobile].
I would like to be notified when any change takes place in my AWS account.
For example a dev launches a new instance , or a new open port is added in a security group etc and he forgets to announce it to me or the rest of the team.
I want to be fully informed for these changes in order to apply specific architecture and/or security and people tend to mess with them.
Is there any dashboard or service inside AWS that I can customise it?
Someone suggested that I should take a look in CloudTrail.
Has anyone done something like this?
The easiest way to go is to use cloudtrail with cloudwatch logs. In AWS FAQ:
Q:What are the benefits of CloudTrail integration with CloudWatch Logs?
This integration enables you to receive SNS notifications of API activity captured by CloudTrail. For example, you can create CloudWatch alarms to monitor API calls that create, modify and delete Security Groups and Network ACL’s. For examples, go to the examples section of the user guide.
Based on SNS, you can then send email through SES
I think the easier way is to use Amazon Cloudtrail service.
Cloudtrail logs any API call which is made on your AWS account. Every operation done on AWS is and API call (including instances operations as you have requested)
Here you can find more information about it
http://docs.aws.amazon.com/awscloudtrail/latest/userguide/configure-cloudtrail-to-send-notifications.html
I hope this helps somehow.
You can find logs of your AWS account in S3,
Find below path in S3:
s3://security-logging/AWS_/AWSLogs/AWS Account no./CloudTrail/your region/year
You can also integrate CloudTrail with SQS to send notifications.