We're using AWS Step Functions in our application. We have one step function we're creating with the use of the CDK as part of a deployment of Application A from Repository A. That step function needs to include a lambda function as one of the steps. The problem we're having is that this lambda function is created and maintained independently in a different repository (Repository B). We're not sure the best way to connect one AWS resource (AWS Lambda) with another AWS resource (AWS Step Functions) when the creation of those two resources is happening independently in two different places.
We'd like to not manually create the lambda or step function (or both) in each environment. It's time consuming, prone to error and we're going to have a lot of these situations occur.
Our best thought at the moment is that we could maybe have Application A create the step function, but have it create and reference an empty lambda. Initially the step function won't be fully functional of course, but then when we deploy Application B it could look for that empty lambda function and upload new code to it.
And, so that we don't have an issue where deploying Application B first results in non-working code. We can also handle the opposite condition: Application B could create the lambda function before uploading the code to it if it doesn't already exist. Application A could then look to see if the lambda function already exists when creating the step function and just reference the lambda function in the step function directly.
Concerns with this approach:
This is extra work and adds a lot of complexity to the deployment, so more potential for failure
I'm not sure I can easily look up a lambda function like this anyway (I guess it would have to be by name since we couldn't know what the ARN would be when we're writing the code). But then we have issues if the name changes too, so maybe there's a pre-defined ID or something we could use to look it up instead.
Potential for code failing in production. If when deploying to QA for testing we deploy Application A, then Application B, we really only know that scenario works. If, then, when going to production we deploy them in the opposite order it might break.
What are some good options for this kind of thing because I can't think of anything great. My best idea involves not using lambda at all but instead having the step function step be queueing something up in SQS, then Application B can just read from that queue no problem. It feels like this is a common enough scenario though that there must be some clean way to do it with lambda and I wouldn't want my decisions on what service type I can use in AWS be stymied by deployment feasibility.
Thanks
You can easily include an existing Lambda function in a new CDK-created Step Function. Use the Function.fromFunctionArn static method to get a read-only reference to the Lambda using its ARN. The CDK uses the ARN to add the necessary lambda:InvokeFunction permissions to the Step Functions' assumed role.
import { aws_stepfunctions_tasks as tasks } from 'aws-cdk-lib';
const importedLambdaTask = new tasks.LambdaInvoke(this, 'ImportedLambdaTask', {
lambdaFunction: lambda.Function.fromFunctionArn(
this,
'ImportedFunc',
'arn:aws:lambda:us-east-1:123456789012:function:My-Lambda5C096DFA-RLhGGzBJSnMN'
),
resultPath: '$.importedLambdaTask',
});
If you prefer not to hard code the Lambda ARN int the CDK stack, save the ARN to a SSM Parameter Store Parameter. Then import it into the stack by name and pass it to fromFunctionArn:
const lambdaArnParam = ssm.StringParameter.fromStringParameterName(
this,
'ArnFromParamStore',
'lambda-arn-saved-as-ssm-param'
);
Edit: Optionally add a Trigger construct to your CDK Application A to confirm the existence of the Application B Lambda dependency before deploying. Triggers are a newish CDK feature that let you run Lambda code during deployments. The Trigger Function should return an error if it cannot find the external Lambda, thereby causing Application A's deployment to fail.
Related
I'm using CodePipeline to deploy my CloudFormation templates that contain Lambda functions as AWS::SAM::Functions.
The CodePipeline is triggered by a commit in my main branch on GitHub.
The Source Stage in the CodePipeline retrieves the source files from GitHub. Zero or more Lambda functions could change in a commit. There are several Lambda Functions in this repository.
I intend on running through taskcat for CloudFormation Templates and Unit Tests for Lambda Python code during a test stage and then deploy the CloudFormation templates and Lambda Functions to production. The problem is, I can't figure out how to differentiate between changed and unchanged Lambda functions or automate the deployment of these Lambda functions.
I would like to only test and deploy new or update changed Lambda functions along with my CloudFormation templates - what is the best practice for this (ideally without Terraform or hacks)?
Regarding testing: Best practice is actually to simply test all lambda code in the repo on push before deploying. You might skip some work for example with github actions that you only test the files that have changed, but it definitely takes some scripting and it hardly ever adds much value. Each testing tool has its own way of dealing with that (sometimes you can simply pass the files you want to test as an argument and then its easy, but sometimes test tools are more of a all-or-nothing approach and it gets quite complicatedreal fast).
Also, personally I'm not a big fan of taskcat since it doesn't really add a lot of value and it's not a very intuitive tool (also relatively outdated IMO). Is there a reason you need to do these types of testing?
Regarding deployment: There are a few considerations when trying to only update lambdas that have changed.
Firstly, cloudformation already does this automatically: as long as the cloudformation resource for the lambda doesn't change, the lambda will not be updated.
However, SAM has a small problem there, since it will re-package the lambda code on every pipeline run and update the CodeUri property of the lambda. And thus the lambda gets updated (even though the code might stay the same).
To work around this, you have several options:
Simply accept that SAM updates your function even though the code might not have changed.
Build SAM locally, and use the --cached and --cache-dir option when deploying in your pipeline. Make sure to push the folder that you set as cache-dir.
Use a different file packaging tool than SAM. Either some custom script that or something else that only pushes your code to s3 when the files have changed.
If you're into programming I'd suggest you take a look into CDK. It's a major upgrade from cloudformation/SAM, and it handles code bundling better (only updates when files have changed). Also the testing options are much wider for CDK.
I am writing a serverless app on AWS.
I have broken up the app into many CloudFormation stacks. I am using CDK (in Python) to create the CF stacks to deploy the app.
A core requirement of my lambda functions, of course, is the ability to log events. To handle this (and all message passing in the app), I have created a custom EventBridge bus in one of my stacks. The name of the event bus is an output of the stack.
Because the logging functionality will be common to many lambda functions, it seems appropriate to put the logging functionality into a lambda layer. I can then have all of my lambda functions implement that layer, and logging will be automagically available to all of my lambda functions.
The problem is that my logging code needs to know which EventBridge event bus to write events to. I don't want to hardcode this, because I may deploy a 'dev' and/or 'test' and/or 'prod' version of the stack simultaneously and the logging layer in each environment needs to log to its environment's logging bus.
Because a lambda layer runs in the context of the parent lambda function, I can't set an environment variable on the layer.
I could set a 'LoggingEventBus' environment variable on all the lambda functions that use this layer. But that seems repetitive and error prone. (Although it's the best solution I can come up with at the moment.)
I could store the event bus in an SMS parameter store. But good Lord, lookup up that parameter every time any function in my app wants to log anything would just be ridiculous. (And I'm back to having to figure out a way for the logger to determine if it's in dev/test/prod to look up the right bus.)
I'm actually considering some build process during 'cdk synth' that modifies the source code of the logging layer and does a string replacement of the event bus name, so that the name actually is hardcoded by the time the code is deployed. But that solution has all kinds of danger signals around it.
Ideally, there would be some kind of 'environment variable' for the layer itself. But no such thing exists, and I acknowledge that such a feature would not be compatible with how most language runtimes work.
How have others solved this? Is there a 'right' answer for putting a setting in a lambda layer?
You are right that there is no concept of environment variables for Lambda Layers.
I could set a 'LoggingEventBus' environment variable on all the
lambda functions that use this layer. But that seems repetitive and
error prone.
We also use Layers for some logic (involving SNS Topics etc) which is common to multiple Lambda functions, and we do it very similar to what you suggested above.
Only difference is that we just set the stage in environment variables, not the values of our SNS Topics (or in your case the EventBridge name). And then let the Layer decide which SNS Topic to use based on the stage set in env.
In our Layer code, we use a config like this -
{
"Prod": {
"SNSTopicARN": "ProdARN",
"OtherConfig": "ProdValue"
},
"Dev": {
"SNSTopicARN": "DevARN",
"OtherConfig": "DevValue"
}
}
Then in the Layer Code, we have
config = configJson.get(env.STAGE)
Not the error-prone I guess :)
And we can add any more number of config variables in the Layer without touching the Lambda functions deployment.
Context, I have a CDK app with two stacks using the following setup:
Stack_A:
StateMachine_A
Lambda_A
S3Bucket_A
IAMRole_A
Stack_B:
StateMachine_B
SageMakerTrainJob_B
IAMRole_B
StateMachine_A runs Lambda_A using execution role IAMRole_A. A separate step in StateMachine_A writes data to S3Bucket_A. StateMachine_B runs SageMakerTrainJob_B using execution role IAMRole_B. Lambda_A's purpose is to start execution of StateMachine_B, whose SageMakerTrainJob_B needs to read from S3Bucket_A. Therefore, we have to configure the following permissions:
IAMRole_A needs startExecution permissions on StateMachine_B.
IAMRole_B needs read permissions on S3Bucket_A.
We tried to model this in CDK by creating a direct dependency in Stack_B on Stack_A, using references to IAMRole_A and S3Bucket_A within Stack_B's definition to grant the needed permissions in code. However, this generated the following error:
Error: 'Stack_B' depends on 'Stack_A' (dependency added using stack.addDependency()). Adding this dependency (Stack_A -> Stack_B/IAMRole_B/Resource.Arn) would create a cyclic reference.
Likewise, trying to model the dependency in the other direction gave the same error:
Error: 'Stack_A' depends on 'Stack_B' (dependency added using stack.addDependency()). Adding this dependency (Stack_B -> Stack_A/S3Bucket_A/Resource.Arn) would create a cyclic reference.
Is there any way around this using code dependencies? Are there any recommended best practices for situations like this? Some options we've considered include:
Using a third stack that depends on both to give Stack_A and Stack_B access to each other's resources.
Creating additional access roles for the necessary resources within each stack and maintaining assumeRole permissions for the Lambda/SageMaker roles somewhere outside of CDK.
Putting them all in one stack. Not great for organization and makes the resources really tightly coupled--we may not want StateMachine_A to be the only entry point to StateMachine_B in the future.
Also , I see there were similar-sounding issues during CDK development with CodeCommit/CodePipeline and APIGateway/Lambda. Is this a related bug, or are we just trying to do something that's not supported?
Circular references are always tricky. This isn't a problem that is unique to the CDK. As you explain the problem logically you see where things start to break down. CloudFormation must create any resources that another resource depends on before it can create the dependent resource. There isn't a one solution fits all approach to this, but I'll give some ideas that work.
Promote shared resources to another stack. In your case the S3 bucket needs to be used by both stacks, so if that is in a stack that runs before both you can create the S3 bucket, use an export/import in stack B to reference the S3 bucket, and use and export/import in stack A to reference both the S3 bucket and the state machine in B.
Use wildcards in permissions. Often you can know the name, or enough of the name, of a resource to use wildcards in your permissions. You want to keep the permissions tightly scoped, but quite often a partial name match is good enough. Use this option with caution, of course. Also keep in mind that many resources can be named by you. Many prefer not to do this ever, and some resources you shouldn't (like and S3 bucket), but I often find it's easier to name things.
Create a custom resource to tie things together. If you have a true circular dependency that cannot be resolved (even in the same stack) you may need to use a custom resource to do the work for you. A prime example of this is S3 bucket events.
In addition to the guidelines by Jason, I'd like to mention a fourth option:
You can also try to decouple the two stacks by moving more towards an event driven architecture. While you can't resolve the dependency of IAMRole_B needs read permissions on S3Bucket_A in this example, you can resolve the dependency IAMRole_A needs startExecution permissions on StateMachine_B. Stack A could introduce an EventBridge where StateMachine_A raises an event as soon as it is finished. StateMachine_B can subscribe to this event and start as soon as it has been risen. The stacks would look like the following:
Stack_A:
StateMachine_A
Event_A
S3Bucket_A
IAMRole_A
Stack_B:
StateMachine_B
SageMakerTrainJob_B
IAMRole_B
You would still have two dependencies:
IAMRole_B needs read permissions on Event_A.
IAMRole_B needs read permissions on S3Bucket_A.
I am mapping Lambda Integrations like this on API Gateway:
${stageVariables.ENV_CHAR}-somelambda
So I can have d-somelambda, s-somelambda, etc. Several versions for environments, all simultaneous. This works fine.
BUT, I am using Custom Authorizers, and I have d-authorizer-jwt and d-authorizer-apikey.
When I deploy the API in DEV stage, it's all ok. But when I deploy to PROD stage, all lambda calls are dynamically pointing properly to *p-lambdas*, except the custom authorizer, which is still pointing to "d" (DEV) and calling dev backend for needed validation (it caches, but sometimes checks the database).
Please note I don't want necessarily to pass the Stage Variables like others are asking, I just want to call the correct Lambda out of a proper configuration like Integration Request offers. By having access to Stage Variables as a final way of solving this, I would need to change my approach and have a single lambda for all envs, and dynamically touch the required backend based on Stage Variables... not that good.
Tks
Solved. It works just as I described. There are some caveats:
a) You need to previously grant access to that lambda
b) You can't test the authorizer due to a UI glitch ... it doesn't ask for the StageVar so you will never reach the lambda
c) You need to deploy the API to get the Authorizers updated on a particular Stage
Cannot tell why it didn't work on my first attempt.
How to manually forcefully discard a aws lambda function in the cluster using aws console or aws cli for development and testing purposes ?
If you redeploy the function it'll terminate all existing containers. It could be as simple as assigning the current date/time to the description of the Lambda function and redeploying. This will allow you to redeploy as many times as you need because something is unique and it will tear down all existing containers each time you do the deployment.
With that said, Lambda functions are supposed to be stateless. You should keep that in mind when you write your code (eg. avoid using global variables, use random file names if creating something temp, etc). From the sounds of things, I think you might have an issue with your design if you require the Lambda container to be torn down.
If you're using the UI, then a simple way to do this is to add or alter an environment variable on the function configuration page.
When you click "Save" the function will be reloaded.
Note: this won't work if you're using the versioned functions feature.