I'm using AWS Backup to backup my resources. I would like to get notifications from failed backups, but the only way to check the status of backups is from the AWS Backup service page - there is nothing AWS Backup related on Cloudwatch metrics, I was thinking of creating SNS-topic from Cloudwatch metric but that doesn't seem to be possible now?
Another question - would there be any way to get weekly report from AWS Backup, like "There are 25 resources currently being backed up, and from the last 7 days there is 175 restore points available"?
First of all you should create an SNS topic, add AWS Backup as a trusted entity in the resource-based policy of the SNS topic:
{
"Sid": "__console_pub_0",
"Effect": "Allow",
"Principal": {
"Service": "backup.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:sns:us-west-2:{accountId}:test"
}
then turns on notifications for that topic and add BACKUP_JOB_COMPLETED event by the following AWS documentation:
Using Amazon SNS to Track AWS Backup Events.
Each time when AWS Backup job is completed or failed you will be informed to subscribed email address in SNS topic.
However, I can't find a way to customize notification.
Related
I have RDS SQL server instance and it has the default sqlserver_audit parameter group, but I am not seeing any recent events. What is the issue?
A screen shot of what I am seeing:
Events generated from sqlserver_audit parameter group (HIPAA audit) are not directly visible to you in AWS Console. For more info about HIPAA audit implementation in RDS for SQL Server see this AWS forum post.
When you want to see events from your SQL Server audits, you need to use SQLSERVER_AUDIT option. In that case, RDS will stream data from audits on your RDS instance to your S3 bucket. You can also configure retention time, during which those .sqlaudit files are kept on RDS instance and you can access them by msdb.dbo.rds_fn_get_audit_file. For more info see documentation.
In both cases, "Recent events" will contain only important messages related to your instance, not audited events. So for example, whenever RDS can't access your S3 bucket for writing in order to store your audits, it will tell you so in "Recent events".
Vasek's answer helped me understand why I wasn't seeing logs show up in my s3 bucket and it was because the inline IAM policy attached to my IAM role used to transfer the audit logs was incorrect.
If you use the automated options-group creation wizard to add the SQLSERVER_AUDIT option to your RDS instance, be sure you don't include a trailing slash on your s3 key prefix.
The incorrect IAM policy statement the AWS option group creation wizard created is shown below.
{
"Effect": "Allow",
"Action": [
"s3:ListMultipartUploadParts",
"s3:AbortMultipartUpload",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::my-audit-logs-bucket/audits//*" # <---- INCORRECT
]
}
I changed my SQLSERVER_AUDIT options group to use the bucket's root and changed the IAM policy to the following correct configuration shown below and my audit logs started showing up in my S3 buck
{
"Effect": "Allow",
"Action": [
"s3:ListMultipartUploadParts",
"s3:AbortMultipartUpload",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::my-audit-logs-bucket/*"
]
}
From the docs:
RDS uploads the completed audit logs to your S3 bucket, using the IAM role that you provide. If you enable retention, RDS keeps your audit logs on your DB instance for the configured period of time.
So the log evens will be in S3, assuming all permissions are set correctly, not in the RDS Events console.
I'm trying to ingest Cloudwatch logs in Sumo Logic.
It works for metrics but not for logs. When I try to perform a log search querying
_sourceCategory=aws/cloudwatch
nothing is retrieved.
If I do the same in metrics, it works. So the issue seems to be with the logs.
Here's the context and how I set it up
First I created a role with their template. But since it wasn't working I add an open permission to AWS Cloudwatch and AWS Logs (top of actions):
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"logs:*",
"cloudwatch:*",
"cloudwatch:ListMetrics",
"cloudwatch:GetMetricStatistics",
"tag:GetResources"
],
"Resource": "*",
"Effect": "Allow"
}
]
}
Then, I used the wizard to set up streaming data.
It didn't appear AWS CloudWatch Logs, but it appeared AWS CloudWatch Metrics.
(Could be this related to my issue?)
For source category: aws/cloudwatch
Regions: us-east-1
Namespaces to include:
AWS/Logs
AWS/Lambda
In role, I pasted the ARN of the role created previously.
As I said previously, I can use the metrics and query by metrics, but not query logs. I'm new with both, AWS and Sumo and I don't know what am I missing.
I will appreciate advice.
If you used the wizard, then it makes sense you only get the metrics. Collecting CloudWatch logs is done differently. You will need a Lambda Function to push the logs to Sumo Logic. Sumo Logic cannot pull CloudWatch logs directly.
Grzegorz attached the link to the documentation for collecting CloudWatch logs.
I want a certain HTTPS service to be called every time a file has been uploaded to an S3 bucket.
I have created the S3 bucket and a SNS topic with a verified subscription with the HTTPS service as an endpoint.
I can publish a message on the SNS topic via the AWS UI, and see that the HTTPS service is called as expected.
On the S3 bucket I created an Event, which should link the bucket and the topic. On my first attempt I got an error because the bucket was not allowed to write to the topic, so c.f. the documentation, I changed the topic access policy to:
{
"Version": "2008-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "sns:Publish",
"Resource": "arn:aws:sns:eu-central-1:TOPIC_ID:OrderUpdates",
"Condition": {
"StringEquals": {
"AWS:SourceAccount": "ACCOUNT_ID"
},
"ArnLike": {
"AWS:SourceArn": "arn:aws:s3:*:*:*"
}
}
}
]
}
where TOPIC_ID is the topic owner id which can be seen when the topic is shown in the AWS UI, and the ACCOUNT_ID is the account id shown under account settings in the AWS UI.
This change in the topic access policy allowed me to create the event on the bucket:
When I call the API method getBucketNotificationConfiguration I get:
{
"TopicConfigurations": [
{
"Id": "OrderFulfilled",
"TopicArn": "arn:aws:sns:eu-central-1:TOPIC_ID:OrderUpdates",
"Events": [
"s3:ObjectCreated:*"
]
}
],
"QueueConfigurations": [],
"LambdaFunctionConfigurations": []
}
But the HTTPS service is not called. What am I missing in this setup, that will trigger the HTTPS service to be called by the SNS topic subscription every time a file is uploaded to the S3 bucket?
Thanks,
-Louise
Having the same issue S3 upload event does not trigger sns message even though our sns access policy is correctly set. Turns out we can NOT use the Enable encryption option, since S3 events are triggered via CloudWatch Alarms which do not work with SNS encrypted topics as of now.
Switch back to Disable encryption option, everything works again.
To reproduce this situation, I did the following:
Created an Amazon SNS topic and subscribed my phone via SMS (a good way to debug subscriptions!)
Created an Amazon S3 bucket with an Event pointing to the Amazon SNS topic
I received this error message:
Unable to validate the following destination configurations. Permissions on the destination topic do not allow S3 to publish notifications from this bucket.
I then added the policy you show above (adjusted for my account and SNS ARN)
This allowed the Event to successfully save
Testing
I then tested the event by uploading a file to the S3 bucket.
I received an SMS very quickly
So, it would appear that your configuration should successfully enable a message to be sent via Amazon SNS. This suggests that the problem lies with the HTTPS subscription, either from sending it from SNS or receiving it in the application.
I recommend that you add an Email or SMS subscription to verify whether Amazon SNS is receiving the topic and forwarding it to subscribers. If this works successfully, then you will need to debug the receipt of the message in the HTTPS application.
You must add TopicConfiguration
Read more about enable event notification
I am trying to read message from an encrypted SQS. Objects are landed on an S3 Bucket -> Trigger S3 Event -> Message sent to SQS -> SQS triggers Lambda to Process.
I have got this working using an AWS managed CMK. However, I can't get this working using AWS owned CMK e.g. alias/aws/sqs.
The message just goes into messages in flight and does not invoke the Lambda functions.
As per the the AWS documentation here https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-server-side-encryption.html#sqs-encryption-what-does-sse-encrypt If you don't specify a custom CMK, Amazon SQS uses the AWS managed CMK for Amazon SQS. But we can't attach any policies against AWS owned CMK e.g.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Service": "<<service>>.amazonaws.com"
},
"Action": [
"kms:GenerateDataKey*",
"kms:Decrypt"
],
"Resource": "*"
}]
}
My question is: is it possible to use the AWS owned CMK on an SQS and have Lambda functions be able to read from that queue?
There is a section in the above URL called Enable Compatibility between AWS Services Such as Amazon CloudWatch Events, Amazon S3, and Amazon SNS and Encrypted Queues.
It mentions attaching a policy to the CMK. However, there is an option to use the alias/aws/sqs. I was wondering if I was missing something here.
I spoke with AWS and KMS AWS Managed Key would not work in this scenario. We can't change the key policy for KMS AWS Managed Keys, so wouldn't be possible for scenario: S3 Bucket -> Trigger S3 Event -> Message sent to SQS -> SQS triggers Lambda to Process
I used KMS AWS Customer Managed Key and it worked fine.
I'm simply attempting to setup a budget alert buy am getting the error Invalid SNS topic ARN when attempting to specify an SNS ARN for the alert.
I followed the steps below:
Created a new SNS Topic with SMS as the protocol.
Created a subscription to the topic (a US phone number).
Tested the subscription by sending a text (this worked).
Copied the ARN under the ARN column on the Topics section in AWS SNS.
Checked Notify via Amazon Simple Notification Service (SNS) topic and entered the ARN from #4.
I get the error Invalid SNS topic ARN next to the SNS Topic ARN field, immediately after copying-and-pasting the ARN into the field.
What's up? I know it's the correct ARN because I copied it directly from the console. Also, as far as I understand, Budgets and their alerts aren't tied the a specific region, although SNS Topics are (the topic is in us-east-2 N. Virginia).
Please let me know if you need more specifics.
Make sure you have allowed budget.amazonaws.com to be able to publish messages to this SNS topic in the SNS topic's resource based policy.
Add the following statement to the policy
{
"Sid": "ExampleSid123456789012",
"Effect": "Allow",
"Principal": {
"Service": "budgets.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": "your topic ARN"
}
Hope it works!
I only got it fixed when I created a standard topic (FIFO didn't work).
I got the same problem before, but was able to solved by these steps,
"Please comply with SNS ARN format"
My workaround was
-
1. Create an SNS Topic in US-EAST-1 region
2. Just used Basic Access policy
3. (Optional) publish and subscribe field, I used the option : Only the specified AWS account, pasted my AWS account ID
4. Edit SNS Topic and Enable access policy,by adding the service budgets.amazonaws.com , in SNS:Publish action
just follow and copy the format in the picture below, Then saved
5. After successfully created the SNS topic , I reloaded the configure
alert page, then pasted the SNS topic ARN. No more Errors and can
already confirm budget,
You will receive a confirmation Email.
With this content
Congratulations, you have successfully verified your SNS topic for AWS Budgets! You can now receive your AWS Budgets notifications
through this SNS topic.
I had the same problem. Here is what I experienced.
Started to create AWS billing alert
Tried to use existing SNS topic, got the same error message you did. Interestingly if you have a syntax error in your SNS ARN you will get different error message 'Please comply with SNS ARN format'.
Thought that SNS topic should be in US-EAST region. Created one there just for billing alerts. Still same error message.
Realized that topic must have correct policy attached to it. Added policy statement similar to what #mmal suggested. Still same error message.
But then I just reloaded the page and tried to create billing alert from scratch.
And second time it worked!!
Just as a minor point to this, after following the above steps, it still was not working. I discovered that if you have "Encryption" enabled on the SNS topic it will not work. Disabling this on the topic and refreshing the Budget edit screen fixed it!
You need to add the policy just below "Statement": [ on the existing SNS Access policy.
{
"Sid": "AWSBudgets-notification-1",
"Effect": "Allow",
"Principal": {
"Service": "budgets.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": "SNS-policy-ARN"
},
Makesure not to miss the comma at the end.
SNS-policy-ARN is the ARN of the same SNS topic
If you're running this in a .net application, with your AWS credentials in your ~/.AWS/credentials file, ensure that the Region that is defined in your appsettings.json matches the region in which your SNS Topic exists.
{
"AWS": {
"Profile": "myProfile",
"Region": "eu-west-2"
},
...