Just like in AWS where you can have a SNS topic attached to trigger Cloud Formation and trigger a notification upon completion/failure.
Is there any way we can so something similar in Google Cloud Deployment Manager.
I researched for SNS feature to GCP public documentation, so far this feature is not available from GCP. however the documentation at https://cloud.google.com/deployment-manager/docs/configuration/supported-gcp-types is not updated yet.
Here's the snippet from https://issuetracker.google.com/issues/123013878:
- type: gcp-types/cloudscheduler-v1:projects.locations.jobs
name: <YOUR_JOB_NAME_HERE>
properties:
parent: projects/<YOUR_PROJECT_ID_HERE>/locations/<YOUR_REGION_HERE>
name: <YOUR_JOB_NAME_HERE>
description: <YOUR_JOB_DESCRIPTION_HERE>
schedule: "0 2 * * *" # daily at 2 am
timeZone: "Europe/Amsterdam"
pubsubTarget:
topicName: projects/<YOUR_PROJECT_ID_HERE>/topics/<YOUR_EXPECTED_TOPIC_HERE>
data: aGVsbG8hCg== # base64 encoded "hello!"
Related
I'm creating a rule that should fire every time there is a change in status in a SageMaker batch transform job.
I'm using Serverless Framework but to simplify it even further, here's what I did:
The rule, exported from AWS console:
AWSTemplateFormatVersion: '2010-09-09'
Description: >-
CloudFormation template for EventBridge rule
'sagemaker-transform-status-to-CWL'
Resources:
EventRule0:
Type: AWS::Events::Rule
Properties:
EventBusName: default
EventPattern:
source:
- aws.sagemaker
detail-type:
- SageMaker Training Job State Change
Name: sagemaker-transform-status-to-CWL
State: ENABLED
Targets:
- Id: XXX
Arn: >-
arn:aws:logs:us-east-1:XXX:log-group:/aws/events/sagemaker-notifications
Eventually I want this to trigger a step function or a lambda function, but for now I am configuring the target to be CloudWatch with log group 'sagemaker-notifications'
I expect that everytime I run a batch transform job in SageMaker, this will get notified and the log would show up on cloudwatch.
But I'm not getting any logs, so when I tried to PutEvents manually to test it, I was getting this:
Error. NotAuthorizedForSourceException. Not authorized for the source.
It's probably an issue with roles, but I'm not sure which kind of role to configure, where and who should assume it.
Tried going through AWS tutorials, adding permissions to the default event bus, using serverless framework
See some sample event patterns here - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-events-rule.html#aws-resource-events-rule--examples
Your source should be a custom source, and cannot contain aws. (Reference -https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html)
Original Requirement:
Create a route/path on AWS Api Gateway which connects API Gateway directly to AWS Event Bridge (Cloudwatch Events) and puts/pushes event on an event bus of it.
Was able to create it and executes just fine when done from AWS Console.
Actual Problem:
When writing the AWS Cloudformation script for this API Gateway, it looks like this:
EventsPostMethod:
Type: AWS::ApiGateway::Method
Properties:
ResourceId:
Ref: EventsResource
RestApiId:
Ref: RestAPI
HttpMethod: POST
AuthorizationType: NONE
Integration:
Type: AWS
IntegrationHttpMethod: POST
Uri:
Fn::Sub: arn:aws:apigateway:${AWS::Region}:cloudwatchEvents:action/PutEvents
RequestParameters:
integration.request.header.X-Amz-Target: "'AWSEvents.PutEvents'"
RequestTemplate:
some-script-here...
Notice the Uri value:
"arn:aws:apigateway:${AWS::Region}:cloudwatchEvents:action/PutEvents"
arn:aws:apigateway:{region}:{subdomain.service|service}:path|action/{service_api}
According to AWS Docs the value of uri should be following:
For AWS or AWS_PROXY integrations, the URI is of the form arn:aws:apigateway:{region}:{subdomain.service|service}:path|action/{service_api}. Here, {Region} is the API Gateway region (e.g., us-east-1); {service} is the name of the integrated AWS service (e.g., s3); and {subdomain} is a designated subdomain supported by certain AWS service for fast host-name lookup. action can be used for an AWS service action-based API, using an Action={name}&{p1}={v1}&p2={v2}... query string. The ensuing {service_api} refers to a supported action {name} plus any required input parameters. Alternatively, path can be used for an AWS service path-based API. The ensuing service_api refers to the path to an AWS service resource, including the region of the integrated AWS service, if applicable. For example, for integration with the S3 API of GetObject, the uri can be either arn:aws:apigateway:us-west-2:s3:action/GetObject&Bucket={bucket}&Key={key} or arn:aws:apigateway:us-west-2:s3:path/{bucket}/{key}
You must have noticed that I replaced the service with cloudwatchEvents in the above mentioned uri.
Now, error Given by AWS Cloudformation Console during Publish of API Gateway:
AWS Service of type cloudwatchEvents not supported (Service: AmazonApiGateway; Status Code: 400; Error Code: BadRequestException; Request ID: 07bae22c-d198-4595-8de9-6ea23763eff5; Proxy: null)
Now I have tried replacing service with
cloudwatch
eventBridge
cloudwatchEvent
event-bus
This is the real problem. What should I place in service in uri so that it accepts ?
Based on the comments,
The URI should be something like below for events:
arn:aws:apigateway:${AWS::Region}:events:action/PutEvents
I'm currently using an event rule to publish a message to an SNS topic when a glue job succeeds, like so:
JobCompletedEventRule:
Type: AWS::Events::Rule
Properties:
Description: Sends glue job completed events to SNS topic
EventPattern:
source:
- aws.glue
detail-type:
- Glue Job State Change
detail:
jobName:
- !Ref Job
state:
- SUCCEEDED
State: ENABLED
Targets:
- Arn: !Ref SnsTopic
Id: !GetAtt SnsTopic.TopicName
This works well. However, this job is about to become part of a glue workflow, and I now want the SNS message to be published when the workflow succeeds, instead of the individual job. But I can't find a way to do this.
Automating AWS Glue with CloudWatch Events shows the CloudWatch events that are generated by AWS Glue, but there aren't any for workflows.
I know you asked this question 1 year ago, but I found myself having the same need and I resolved it by adding a "dummy job" (that does nothing) at the end of the workflow and then add a rule similar to yours on SUCCESS of the dummy job.
I have used the boto3 library with the publish() method in the last job of my Glue workflow. That way you can customize the message sent. Useful if you have multiple parallels workflows using the same glue jobs and need to distinguish between them in sns messages.
I have created SNS using cloud formation like this -
MYSNS:
Type: AWS::SNS::Topic
Properties:
DisplayName: "MYSNS"
TopicName: "MYSNS"
Now, I want to publish message to this topic using Java but
snsClient.publish(snsARN, snsEvent);
sns client requires ARN to publish the event. How I can get SNS Topic ARN?
Edit 1: Post deployment we can get the ARN, but don't want this way.
Ex - We can get SQS url by it's Name using SQS Client like this -
sqsClient.getQueueUrl(dlqName).getQueueUrl(); I am looking for similar way in case of SNS.
You can construct the ARN yourself:
arn:aws:sns:<region>:<account>:MYSNS
You can try this in cloudformation template to get the created SNS's ARN
!Ref MYSNS
I've created some CloudFormation templates to deploy Inspector Templates/Targets and associated Lambda functions that parse the outputs and deliver findings to Slack. Is it possible to include in the CF template for Inspector an SNS Topic association as is done when creating a template in the Inspector portal?
It is not an available parameter of AWS::Inspector::AssessmentTemplate. Is this something I will just have to add manually via the portal?
I see the SNS option is available only in the UI and CLI/API, I guess the UI/CLI creates Cloudwatch Events rule for you in the background, you create your own rule using AWS::Events::Rule
Reference: Event Patterns
EventRule:
Type: "AWS::Events::Rule"
Properties:
Description: "EventRule"
EventPattern:
source:
- "aws.inspector"
detail-type:
- "AWS API Call via CloudTrail"
resources:
- arn:aws:inspector:us-west-2:123456789012:target/0-nvgVhaxX/template/0-7sbz2Kz0
detail:
eventSource:
- "inspector.amazonaws.com"
eventName:
- "ASSESSMENT_RUN_COMPLETED"
State: "ENABLED"
Targets:
- arn:aws:sns:us-west-2:123456789012:exampletopic
This is how I did it. I used the cloud formation template to create the assessment target, assessment resource group, and assessment template. Also, included a cloudwatch event rule to trigger assessment runs on a weekly basis.
As of today, there is no support for adding an SNS Topic through the Inspector Assessment template cloud formation resource, I went through the boto3 API for event subscription. Refer the API here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/inspector.html#Inspector.Client.subscribe_to_event
If you refer the above API Doc you will be able to develop a small python lambda function to subscribe your inspector assessment template to the SNS topic. Then call that lambda function using a custom resource as follows in the same template where the assessment template is provisioned or defined.
Custom resource would look something like below:
SubscribeToEvent:
Type: "Custom::<whatever_name>"
Version: "1.0"
Properties:
ServiceToken: !GetAtt <Lambda function logical name>.Arn
AssessmentTemplateArn: !GetAtt <Assessment template logical name>.Arn
topicArn: !Sub arn:aws:sns:${AWS::Region}:${account number}:<Nameofthetopic>
If you are trying to refer a cross-account topic or a topic which exist in another account, in that case, you need to update the topic policy to grant publish topic permissions to AWS Inspector Account. To find the AWS Account numbers refer here : https://docs.aws.amazon.com/inspector/latest/userguide/inspector_assessments.html#sns-topic