I am setting up aws code pipeline based on the git developer branch.
when the developer commits his code, the pipeline will be trigger based on the webhook. Now the idea is when there is a failure in the pipeline, and when the developer triggers a pull-request, the reviewer should know that this is bad branch. He should be able to see the git status of the branch showing there is a failure.
Earlier I have used a build tool called Codeship which has a git-hub app to do this stuff. Now I have gone through git-hub API
https://developer.github.com/v3/checks/runs/#create-a-check-run
But not sure where to start.
To send a notification when stage fails, please follow these steps:
Based on your Cloudwatch events emitted by CodePipeline [0], trigger a lambda function [1].
Lambda function can trigger API call "list-pipeline-executions"[2] and from that you can fetch all the required values like Commit Id, status message etc [3].
Once the values are fetched, you can send same value to SNS by writing lambda function code inside same lambda. Following blogs shows how to publish in SNS using lambda [4][5].
[0] https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-cloudwatch-sns-notifications.html
[1] https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html
[2] https://docs.aws.amazon.com/cli/latest/reference/codepipeline/list-pipeline-executions.html
[3] https://docs.aws.amazon.com/cli/latest/reference/codepipeline/index.html
[4] https://gist.github.com/jeremypruitt/ab70d78b815eae84e037
[5] Can you publish a message to an SNS topic using an AWS Lambda function backed by node.js?
I have done the following to write back the status to gitrepo:
I have made use of gitstatus api:
https://developer.github.com/v3/repos/statuses/
Written a Lambda function to do POST request with these details
state, TargetURL , context
{
"state": "success",
"target_url": "https://example.com/build/status" (Build Tool URL like codebuild or jenkins)
"description": "The build succeeded!",
"context": "continuous-integration/jenkins"
}
the response here should be
"url": "https://api.github.com/repos//-/statuses/6dcb09b5b57875f334f61aebed695e2e4193db5e ",
All these details can be obtained using CLoudwatch event for the pipeline. by using patterns of event detail list :
event.detail.state;event.detail.pipeline;event.region, event ['detail']['execution-id'],
data.pipelineExecution.artifactRevisions[0].revisionUrl
Cheers.
Related
Right now I monitor my submitted jobs on Google AI Platform (formerly ml engine) by polling the job REST API. I don't like this solution for a few reasons:
Awareness of status changes is often delayed or missed altogether if the interval between status changes is smaller than the monitoring polling rate
Lots of unnecessary network traffic
Lots of unnecessary function invocations
I would like to be notified as soon as my training jobs complete. It'd be great if there is some way to assign hooks or callbacks to run when the job status changes.
I've also considered adding calls to cloud functions directly within the training task python package that runs on AI Platform. However, I don't think those function calls will occur in cases where the training job is shutdown unexpectedly, such as when a job is cancelled or forced to end by GCP.
Is there a better way to go about this?
You can use a Stackdriver sink to read the logs and send it to Pub/Sub. From Pub/Sub, you can connect to a bunch of other providers:
1. Set up a Pub/Sub sink
Make sure you have access to the logs and publish rights to the topic you desire before you get started. Follow the instructions for setting up a Stackdriver -> Pub/Sub sink. You’ll want to use this query to limit the events only to Training jobs:
resource.type = "ml_job"
resource.labels.task_name = "service"
Note that Stackdriver can further limit down the query. For example, you can limit to a particular Job by adding a condition like resource.labels.job_id = "..." or to a certain event with a filter like jsonPayload.message : "..."
2. Respond to the Pub/Sub message
In order to tell what changed, the recipient of the Pub/Sub message can either query the job status from the ml.googleapis.com API or read the text of the message
Reading state from ml.googleapis.com
When you receive the message, make a call to https://ml.googleapis.com/v1/<project_id>/jobs/<job_id> to get the Job information, replacing [project_id] and [job_id] in the URL with the values of resource.label.project_id and resource.label.job_id from the Pub/Sub message, respectively.
The returned Job object contains a field state that, naturally, tells the status of the job.
Reading state from the message text
The Pub/Sub message will contain a string telling what happened to the job. You probably want behavior when the job ends. Look for these strings in jsonPayload.message:
"Job completed successfully."
"Job cancelled."
"Job failed."
I implemented a Terraform module as #htappen said. I'm happy if it would help you. But my real hope is that Google updates AI Platform with the same feature.
https://github.com/sfujiwara/terraform-google-ai-platform-notification
I think you can programmatically publish a PubSub message at the end of your training job code. Something like this:
from google.cloud import pubsub_v1
# publish job complete message
client = pubsub_v1.PublisherClient()
topic = client.topic_path(args.gcp_project_id, 'topic-name')
data = {
'ACTION': 'JOB_COMPLETE',
'SAVED_MODEL_DIR': args.job_dir
}
data_bytes = json.dumps(data).encode('utf-8')
client.publish(topic, data_bytes)
Then you can setup a cloud function to be triggered by the same pubsub topic.
You can work around the lack of a callback from the service on a custom TF training job by adding a LamdbaCallback to the fit() call. In the on_epoch method, you could then send yourself a notification on job progress and on_train_end when it finishes.
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/LambdaCallback
I have a cloud function which is configured for slack notification & also have two cloud build trigger against same repository (one is for push activity from git & another one for pull request activity from any branch).
Now when any jobs are getting triggered that cloud function will push the message in slack using pub/sub. But I just want the notification when any process failed or pass activity happen against push activity cloud build trigger, not for the pull request activity trigger. How do I remove the cloud functions for only pull request activity. Can I configure like this?
Thanks in advance. :)
To send the slack notification only according to the build status you can detect the type of the event and choose to send the slack notification or not.
On the event.data variable that you are receiving in the subscribe method you can read the variable status to check the status of the build, so you can send the slack notification only on SUCCESS or FAILURE status.
I do this by adding a substitution variable (either in the build trigger config (UI), or the cloudbuild.yaml itself:
substitutions:
_DISABLE_SLACK: 'true'
Then in your slack webhook code:
// don't send slack messages if we have _DISABLE_SLACK in `cloudbuild.yaml` file -- any value is accepted
if (build && build.substitutions && build.substitutions['_DISABLE_SLACK']) {
return;
}
PS -- I also have another method where you want to disable slack messages for a build, BUT you still want errors... Can post that if you're interested :)
[basically if build.status != 'WORKING', 'QUEUED', 'CANCELLED' && if build.buildTriggerId in [array], then we continue with slack message (overriding DISABLE_SLACK)]
I followed tutorial on creating and invoking step functions
I'm getting output in my GET request of api as
{
"executionArn": "arn:aws:states:ap-northeast-1:123456789012:execution:HelloWorld:MyExecution",
"startDate": 1.486772644911E9
}
But, instead of above response I want my step functions output, which is given by end state as below.
{
"name":"Hellow World"
}
How to achieve this?
Update: You can now use Express Step Functions for synchronous requests.
AWS Step Functions are asynchronous and do not immediately return their results. API Gateway methods are synchronous and have a maximum timeout of 29 seconds.
To get the function output from a Step Function, you have to add a second method in API Gateway which will call the Step Function with the DescribeExecution action. The API Gateway client will have to call this periodically (poll) until the returned status is no longer "RUNNING".
Here's the DescribeExecution documentation
Use Express Step Functions instead. This type of Step Functions can be called synchronously. Go to your API Gateway and in the Integration Request section make sure you have the StartSyncExecution action:
After that, go a bit lower in the same page to the Mapping Templates:
and include the following template for the application/json Content-Type:
#set($input = $input.json('$'))
{
"input": "$util.escapeJavaScript($input)",
"stateMachineArn": "arn:aws:states:us-east-1:your_aws_account_id:stateMachine:your_step_machine_name"
}
After that, go back to the Method Execution and go to the Integration Response and then to the Mapping Templates section:
And use the following template to have a custom response from your lambda:
#set ($parsedPayload = $util.parseJson($input.json('$.output')))
$parsedPayload
My testing Step Function is like this:
And my Lambda Function code is:
Deploy your API Gateway stage.
Now, if you go to Postman and send a POST request with any json body, now you have a response like this:
New Synchronous Express Workflows for AWS Step Functions is the answer:
https://aws.amazon.com/blogs/compute/new-synchronous-express-workflows-for-aws-step-functions/
Amazon API Gateway now supports integration with Step Functions StartSyncExecution for HTTP APIs:
https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-api-gateway-supports-integration-with-step-functions-startsyncexecution-http-apis/
First of all the step functions executes asynchronously and API Gateway is only capable of invoking the step function (Starting a flow) only.
If you are waiting for the results of a step function invocation from a web application, you can use AWS IOT WebSockets for this. The steps are as follows.
Setup AWS IOT topic with WebSockets.
Configure the API Gateway and Step functions invocation.
From the Web Frontend subscribe to the IOT Topic as a WebSocket listener.
At the last step (And in error steps) in the Step Functions workflow use AWS SDK to trigger the IOT Topic which will broadcast the results to the Web App running in the browser using WebSockets.
For more details on WebSockets with AWS IOT refer the medium article Receiving AWS IoT messages in your browser using websockets.
Expanding on what #MikeD at AWS says, if you're certain that the Step Function won't exceed the 30 second timeout, you could create a lambda that executes the step function and then blocks as it polls for the result. Once it has the result, it can return it.
It is a better idea to have the first call return immediately with the execution id, and then pass that id into a second call to retrieve the result, once it's finished.
My Amazon Lambda function (in Python) is called when an object 123456 is created in S3's input_bucket, do a transformation in the object and saves it in output_bucket.
I would like to notify my main application if the request was successful or unsuccessful. For example, a POST http://myapp.com/successful/123456 if the processing is successful and http://myapp.com/unsuccessful/123456 if its not.
One solution I thought is to create a second Amazon Lambda function that is triggered by a put event in output_bucket, and it to do the successful POST request. This solves half of the problem because but I can't trigger the unsuccessful POST request.
Maybe AWS has a more elegant solution using a parameter in Lambda or a service that deals with these types of notifications. Any advice or point in the right direction will be greatly appreciated.
Few possible solutions which I see as elegant
Using SNS Topic: From your transformation lambda, trigger a SNS topic, with success/unsuccess message, where SNS will call a HTTP/HTTPS endpoint with message payload. The advantage here is, your transformation lambda is loosely coupled with endpoint trigger and only connected through messaging.
Using Lambda Step Functions:
You could arrange to run a Lambda function every time a new object is uploaded to an S3 bucket. This function can then kick off a state machine execution by calling StartExecution. The advantage in using step functions is that you can coordinate the components of your application as series of steps in a visual workflow.
I don't think there is any elegant AWS solution, unless you re-architect, something like your lambda sends message to SQS or some intermediatery messaging service with STATUS and then interemdeiatery invokes POST to your application.
If you still want to go with your way of solving, you might need to configure "DeadLetter queue" to do error handling in failure cases (note that use cases described here are not comprehensive, so need to make sure it covers your case) like described here.
I've just created my first (custom) still. I've set the function up in Lambda by uploading a zip file containing my index.js and all the necessary code required, including node_modules and the base Alexa skill that mine is a child of (as per the tutorials). I made sure I zipped up the files and sub-folders, not the folder itself (as I can see this is a common cause of similar errors) but when I create the skill and test in the web harness with a sample utterance I get:
remote endpoint could not be called, or the response it returned was
invalid.
I'm not sure how to debug this as there's nothing logged in CloudWatch.
I can see in the Lambda request that my slot value is translated/parsed successfully and the intentname is correct.
In AWS Lambda I can invoke the function successfully both with a LaunchRequest and another named intent. From the developer console though, I get nothing.
I've tried copying the JSON from the lambda test (that works) to the developer portal and I get the same error. Here is a sample of the JSON I'm putting in the dev portal (that works in Lambda)
{
"session": {
"new": true,
"sessionId": "session1234",
"attributes": {},
"user": {
"userId": null
},
"application": {
"applicationId": "amzn1.echo-sdk-ams.app.149e75a3-9a64-4224-8bcq-30666e8fd464"
}
},
"version": "1.0",
"request": {
"type": "LaunchRequest",
"requestId": "request5678"
}
}
The first step in pursuing this problem is probably to test your lambda separate from your skill configuration.
When looking at your lambda function in the AWS console, note the 'test' button at the top, and next to it there is a drop down with an option to configure a test event. If you select that option you will find that there are preset test events for Alexa. Choose 'alexa start session' and then choose 'save and test' button.
This will give you more detailed feedback about the execution of your lambda.
If your lambda works fine here then the problem probably lies in your skill configuration, so I would go back through whatever tutorial and documentation you were using to configuration your skill and make sure you did it right.
When you write that the lambda request looks fine I assume you are talking about the service simulator, so that's a good start, but there could still be a problem on the configuration tab.
We built a tool for local skill development and testing.
BST Tools
Requests and responses from Alexa will be sent directly to your local server, so that you can quickly code and debug without having to do any deployments. I have found this to be very useful for our own development.
Let me know if you have any questions.
It's open source: https://github.com/bespoken/bst