I am getting this weird result when I try to deploy the lambda step, if I define my lambda step like this
lambda_step = steps.compute.LambdaStep(
"Query Training Results",
parameters={
"FunctionName": execution_input["LambdaFunctionName"],
"Payload": {"TrainingJobName.$": "$.TrainingJobName"},
},
)
For some reason it will automatically just grey out the box meaning the job will be cancelled immediately. If I simply remove the "Payload" part then it will work but the lambda step will still fail because it does not know the training job name that I am trying to pass in the Payload.
I followed this example to the T here. Any suggestions would be greatly appreciated.
Related
I configured aws bridge event rule (via web gui) for running aws batch job - rule is triggered but a I am getting following error after invocation:
shareIdentifier must be specified. (Service: AWSBatch; Status Code: 400; Error Code: ClientException; Request ID: 07da124b-bf1d-4103-892c-2af2af4e5496; Proxy: null)
My job is using scheduling policy and needs shareIdentifier to be set but I don`t know how to set it. Here is screenshot from configuration of rule:
There are no additional settings for subsequent arguments/parameters of job, the only thing I can configure is retries. I also checked aws-cli command for putting rule (https://awscli.amazonaws.com/v2/documentation/api/latest/reference/events/put-rule.html) but it doesn`t seem to have any additional settings. Any suggestions how to solve it? Or working examples?
Edited:
I ended up using java sdk for aws batch: https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-batch. I have a scheduled method that periodically spawns jobs with following peace of code:
AWSBatch client = AWSBatchClientBuilder.standard().withRegion("eu-central-1").build();
SubmitJobRequest request = new SubmitJobRequest()
.withJobName("example-test-job-java-sdk")
.withJobQueue("job-queue")
.withShareIdentifier("default")
.withJobDefinition("job-type");
SubmitJobResult response = client.submitJob(request);
log.info("job spawn response: {}", response);
Have you tried to provide additional settings to your target via the input transformer as referenced in the AWS docs AWS Batch Jobs as EventBridge Targets ?
FWIW I'm running into the same problem.
I had a similar issue, from the CLI and the GUI, I just couldn't find a way to pass ShareIdentifier from an Eventbridge rule. In the end I had to use a state machine (step function) instead:
"States": {
"Batch SubmitJob": {
"Type": "Task",
"Resource": "arn:aws:states:::batch:submitJob.sync",
"Parameters": {
"JobName": <name>,
"JobDefinition": <Arn>,
"JobQueue": <QueueName>,
"ShareIdentifier": <Share>
},
...
You can see it could handle ShareIdentifier fine.
I have a use case which I want to use Step Functions to solve but I can't find a way to solve this problem. Your help would be greatly appreciated.
The problem goes like this: I have an Amazon API Gateway which has a /start endpoint. a POST to this endpoint should start a data processing session and return a URL to an app which the API client can use to capture some data. Once data capture is complete, some processing takes place before the final response is sent to the API client via a callback.
My thinking, as you can see below, is to generate a task token and send it to the Data Capture Service. Then, when the user data capture is complete, the service can send a request to the Step Function API to say that stage is complete. The problem with this is how can I return the URL to the client from within the Step Function? I don't want to use a callback to do this.
One option is to create the data capture session within the 'Step Function Initiator' Lambda but then how do I provide the Data Capture Service with a task token?
Really, what I need is some mechanism of synchronously returning something (either a URL from that call or the task token from the first stage) from within the Step Function to the Lambda which started the execution. Is this possible? How would you solve this?
In step function initiator lambda, you must be doing start-execution which returns an executionArn
Next, you can loop and call get-execution-history api and task token will be part of the 'capture data' task parameters. Since this is the first step, this really should be done with in couple of seconds, so, we can keep running this loop every second until desired step in step function is initiated and task token can be obtained.
Take this example, i am passing the task token to another step function call from current step function.
{
"StartAt":"ChildTask",
"States":{
"ChildTask":{
"End":true,
"Type":"Task",
"Resource":"arn:aws:states:::states:startExecution.waitForTaskToken",
"Parameters":{
"Input":{
"token.$":"$$.Task.Token",
"foo":"bar"
},
"StateMachineArn":"arn:aws:states:us-east-1:110011001100:stateMachine:ChildStateMachine",
"Name":"MyExecutionName"
}
}
}
}
Get Execution history:
aws stepfunctions get-execution-history --execution-arn arn:aws:states:us-east-1:110011001100:execution:ParentStateMachine:667102b3-b19c-b7ab-b119-9ec6cf23e505
Result:
one of the first few entries in execution history and task token is part of the parameters. we can exit the loop, grab that, send it back to Api Gateway.
{
"timestamp": "2021-03-12T13:56:58.097000-05:00",
"type": "TaskScheduled",
"id": 3,
"previousEventId": 2,
"taskScheduledEventDetails": {
"resourceType": "states",
"resource": "startExecution.waitForTaskToken",
"region": "us-east-1",
"parameters": "{\"Input\":{\"foo\":\"bar\",\"token\":\"o6QVQ9gls.......=\"},\"StateMachineArn\":\"arn:aws:states:us-east-1:110011001100:stateMachine:ChildStateMachine\",\"Name\":\"MyExecutionName\"}"
}
}
I am working on Amazon Step functions to leverage the workflow for multiple Batch jobs. The requirement is such that the Batch jobs should be executed sequentially and whenever a job transition from one to another job then send a notification with the execution status of the tasks to a SNS topic. I need to send a notification for SUCCESS and FAILURE of a task.
I have tried the Execution Events using Cloudwatch event rules, but Execution Events only gives information about the State Machine's execution, not about the Tasks execution.
As you have found states aren't in Cloudwatch events need to add this as a separate step, there is no way around this, have a notify step which either executes a lambda, or sync to SNS.
There is also another way to do this as you can compose step functions of step functions. So you have your parent step function and your child step function. Your child step function could be the batch job itself, and then you can make use of Cloudwatch events on the batch-job-step-function step function:
"BatchJob" : {
"Comment": "This snippet is in the parent step function. It will kick off another step function, called: batch-job-step-function",
"Type": "Task",
"Resource": "arn:aws:states:::states:startExecution.sync",
"Parameters": {
"StateMachineArn": "arn:aws:states:us-east-1:TODO:stateMachine:batch-job-step-function",
"Input": {
"batchJobInput.$": "$$.Execution.Input.batchJobInput"
}
},
"End": true | "Next" : "TODO"
}
Now you can put Cloudwatch Event Rules against: arn:aws:states:us-east-1:TODO:stateMachine:batch-job-step-function
We're using a lambda function to respond to the 'User Migration' trigger in AWS Cognito. When something like a syntax error occurs, you can see it in cloud watch logs. However, "Exception during user migration" errors seen on the login page are no where to be found in the cloud watch logs.
Where are we supposed to look for these? I can't find any anything in the documentation and assumed it would have gone to cloud watch.
I can't test it in the lambda interface because one of the parameters being passed into the lambda function will have a function nested within the object and I can't create a test JSON setup that has that. There's also no test trigger for user migration that is pre-built.
Any ideas as to why I can't see this in cloud watch or where the exceptions would be shown would be greatly appreciated.
Unfortunately Cogntio doesn't expose any logs (or metrics, for that matter!).
The closest you can get is to view the lambda's logs in CloudWatch. If you log your response, and watch your lambda's error metric then you should mostly be able to debug issues internal to the lambda.
This does leave a few edge cases:
You won't see anything if the lambda can't be invoked (this would only happen under heavy concurrent loads either on that single lambda, or on all lambdas across your account)
If you return a bad response the lambda will succeed but the trigger action will fail and Cognito will give you a fairly generic message. At this point you're at the mercy of AWS' documentation to work out what's wrong (which can be a bit hit and miss- although StackOverflow always helps!).
You can find an example payload for the lambda in the trigger documentation:
{
"userName": "THE USERNAME",
"request": {
"password": "THE PASSWORD"
},
"response": {
// it is your responsibility to fill this bit in and return the completed object back:
"userAttributes": {
"string": "string",
...
},
"finalUserStatus": "string",
"messageAction": "string",
"desiredDeliveryMediums": [ "string", ... ],
"forceAliasCreation": boolean
}
}
n.b. As an aside, which you might know, but Lambda payloads always have to be in JSON, which does not store functions. So you should always be able to derive a test payload to use in the console.
I've just created my first (custom) still. I've set the function up in Lambda by uploading a zip file containing my index.js and all the necessary code required, including node_modules and the base Alexa skill that mine is a child of (as per the tutorials). I made sure I zipped up the files and sub-folders, not the folder itself (as I can see this is a common cause of similar errors) but when I create the skill and test in the web harness with a sample utterance I get:
remote endpoint could not be called, or the response it returned was
invalid.
I'm not sure how to debug this as there's nothing logged in CloudWatch.
I can see in the Lambda request that my slot value is translated/parsed successfully and the intentname is correct.
In AWS Lambda I can invoke the function successfully both with a LaunchRequest and another named intent. From the developer console though, I get nothing.
I've tried copying the JSON from the lambda test (that works) to the developer portal and I get the same error. Here is a sample of the JSON I'm putting in the dev portal (that works in Lambda)
{
"session": {
"new": true,
"sessionId": "session1234",
"attributes": {},
"user": {
"userId": null
},
"application": {
"applicationId": "amzn1.echo-sdk-ams.app.149e75a3-9a64-4224-8bcq-30666e8fd464"
}
},
"version": "1.0",
"request": {
"type": "LaunchRequest",
"requestId": "request5678"
}
}
The first step in pursuing this problem is probably to test your lambda separate from your skill configuration.
When looking at your lambda function in the AWS console, note the 'test' button at the top, and next to it there is a drop down with an option to configure a test event. If you select that option you will find that there are preset test events for Alexa. Choose 'alexa start session' and then choose 'save and test' button.
This will give you more detailed feedback about the execution of your lambda.
If your lambda works fine here then the problem probably lies in your skill configuration, so I would go back through whatever tutorial and documentation you were using to configuration your skill and make sure you did it right.
When you write that the lambda request looks fine I assume you are talking about the service simulator, so that's a good start, but there could still be a problem on the configuration tab.
We built a tool for local skill development and testing.
BST Tools
Requests and responses from Alexa will be sent directly to your local server, so that you can quickly code and debug without having to do any deployments. I have found this to be very useful for our own development.
Let me know if you have any questions.
It's open source: https://github.com/bespoken/bst