Is it possible to run an external build command as part of a CDK stack sequence? Intention: 1) create a rest API, 2) write rest URL to config file, 3) build and deploy a React app:
import apigateway = require('#aws-cdk/aws-apigateway');
import cdk = require('#aws-cdk/core');
import fs = require('fs')
import s3deployment = require('#aws-cdk/aws-s3-deployment');
export class MyStack extends cdk.Stack {
const restApi = new apigateway.RestApi(this, ..);
fs.writeFile('src/app-config.json',
JSON.stringify({ "api": restApi.deploymentStage.urlForPath('/myResource') }))
// TODO locally run 'npm run build', create 'build' folder incl rest api config
const websiteBucket = new s3.Bucket(this, ..)
new s3deployment.BucketDeployment(this, .. {
sources: [s3deployment.Source.asset('build')],
destinationBucket: websiteBucket
})
}
Unfortunately, it is not possible, as the necessary references are only available after deploy and therefore after you try to write the file (the file will contain cdk tokens).
I personally have solved this problem by telling cdk to output the apigateway URLs to a file and then parse it after the deploy to upload it so a S3 bucket, to do it you need:
deploy with the output file options, for example:
cdk deploy -O ./cdk.out/deploy-output.json
In ./cdk.out/deploy-output.json you will find a JSON object with a key for each stack that produced an output (e.g. your stack that contains an API gateway)
manually parse that JSON to get your apigateway url
create your configuration file and upload it to S3 (you can do it via aws-sdk)
Of course, you have the last steps in a custom script, which means that you have to wrap your cdk deploy. I suggest to do so with a nodejs script, so that you can leverage aws-sdk to upload your file to S3 easily.
Accepting that cdk doesn't support this, I split logic into two cdk scripts, accessed API gateway URL as cdk output via the cli, then wrapped everything in a bash script.
AWS CDK:
// API gateway
const api = new apigateway.RestApi(this, 'my-api', ..)
// output url
const myResourceURL = api.deploymentStage.urlForPath('/myResource');
new cdk.CfnOutput(this, 'MyRestURL', { value: myResourceURL });
Bash:
# deploy api gw
cdk deploy --app (..)
# read url via cli with --query
export rest_url=`aws cloudformation describe-stacks --stack-name (..) --query "Stacks[0].Outputs[?OutputKey=='MyRestURL'].OutputValue" --output text`
# configure React app
echo "{ \"api\" : { \"invokeUrl\" : \"$rest_url\" } }" > src/app-config.json
# build React app with url
npm run build
# run second cdk app to deploy React built output folder
cdk deploy --app (..)
Is there a better way?
I solved a similar issue:
Needed to build and upload react-app as well
Supported dynamic configuration reading from react-app - look here
Released my react-app with specific version (in a separate flow)
Then, during CDK deployment of my app, it took a specific version of my react-app (version retrieved from local configuration) and uploaded its zip file to S3 bucket using CDK BucketDeployment
Then, using AwsCustomResource I generated a configuration file with references to Cognito and API-GW and uploaded this file to S3 as well:
// create s3 bucket for react-app
const uiBucket = new Bucket(this, "ui", {
bucketName: this.stackName + "-s3-react-app",
blockPublicAccess: BlockPublicAccess.BLOCK_ALL
});
let confObj = {
"myjsonobj" : {
"region": `${this.region}`,
"identity_pool_id": `${props.CognitoIdentityPool.ref}`,
"myBackend": `${apiGw.deploymentStage.urlForPath("/")}`
}
};
const dataString = JSON.stringify(confObj, null, 4);
const bucketDeployment = new BucketDeployment(this, this.stackName + "-app", {
destinationBucket: uiBucket,
sources: [Source.asset(`reactapp-v1.zip`)]
});
bucketDeployment.node.addDependency(uiBucket)
const s3Upload = new custom.AwsCustomResource(this, 'config-json', {
policy: custom.AwsCustomResourcePolicy.fromSdkCalls({resources: custom.AwsCustomResourcePolicy.ANY_RESOURCE}),
onCreate: {
service: "S3",
action: "putObject",
parameters: {
Body: dataString,
Bucket: `${uiBucket.bucketName}`,
Key: "app-config.json",
},
physicalResourceId: PhysicalResourceId.of(`${uiBucket.bucketName}`)
}
});
s3Upload.node.addDependency(bucketDeployment);
As others have mentioned, this isn't supported within CDK. So this how we solved it in SST: https://github.com/serverless-stack/serverless-stack
On the CDK side, allow defining React environment variables using the outputs of other constructs.
// Create a React.js app
const site = new sst.ReactStaticSite(this, "Site", {
path: "frontend",
environment: {
// Pass in the API endpoint to our app
REACT_APP_API_URL: api.url,
},
});
Spit out a config file while starting the local environment for the backend.
Then start React using sst-env -- react-scripts start, where we have a simple CLI that reads from the config file and loads them as build-time environment variables in React.
While deploying, replace these environment variables inside a custom resource based on the outputs.
We wrote about it here: https://serverless-stack.com/chapters/setting-serverless-environments-variables-in-a-react-app.html
And here's the source for the ReactStaticSite and StaticSite constructs for reference.
In my case, I'm using the Python language for CDK. I have a Makefile which I invoke directly from my app.py like this:
os.system("make"). I use the make to build up a layer zip file per AWS Docs. Technically you can invoke whatever you'd like. You must import the os package of course. Hope this helps.
Related
I'm building an application with AWS CDK that uses CodePipeline. So there are essentially two stacks, one sets up the code pipeline and the other sets up the application (and it's triggered by the pipeline).
I'm working out of what is built in https://cdkworkshop.com/ so in my project I have a file cdk.json that has an entry app pointing to a specific TypeScript file (example4-be is the application name):
{
"app": "npx ts-node --prefer-ts-exts bin/example4-be.ts",
This file builds the CodePipeline stack:
#!/usr/bin/env node
import * as cdk from "aws-cdk-lib"
import {PipelineStack} from "../lib/pipeline-stack"
const app = new cdk.App()
new PipelineStack(app, "Example4BePipeline")
so when I try to use sam to run the application locally, it fails saying there are no Lambda functions. I believe it's because it's building the CodePipeline stack and not the application stack. If I change exampe4-be.ts to this:
#!/usr/bin/env node
import * as cdk from "aws-cdk-lib"
import {Example4BeStack} from "../lib/example4-be-stack";
const app = new cdk.App()
new Example4BeStack(app, "Example4BePipeline")
it works. Example4BeStack is the application stack. But obviously if I commit this, the CodePipeline will stop working.
How can I have both things working at the same time?
The commands I run to have sam run the application locally are:~
cdk synth --no-staging | out-file template.yaml -encoding utf8
sam local start-api
Create two cdk.App chains in your codebase, one for the pipeline and one for standalone development/testing with sam local or cdk deploy. Your "application" stacks will be part of both chains. Here's a simplified example of the pattern I use:
Pipeline deploy (app-pipeline.ts): ApiStack and DatabaseStack are children of a cdk.Stage, grandchildren of the PipelineStack, and great-granchildren of a cdk.App.
Development deploys (app.ts): ApiStack and DatabaseStack are children of a cdk.App. Use with sam local and cdk deploy for dev and testing.
bin/
app.ts # calls makeAppStacks to add the stacks; runs frequently during development
app-pipeline.ts # adds the PipelineStack to an App
lib/
ApiStack.ts
DatabaseStack.ts
PipelineStack.ts # adds DeployStage to the pipeline
DeployStage.ts # subclasses cdk.Stage; calls makeAppStacks.ts to add the stacks
makeAppStacks.ts # adds the Api and Db stacks to either an App or a Stage
A makeAppStacks wrapper function instantiates the actual stacks.
// makeAppStacks.ts
export const makeAppStacks = (scope: cdk.App | DeployStage, appName: string, account: string, region: string): void => {
const {table} = new DatabaseStack(scope, 'MyDb', ...)
new ApiStack(scope, 'MyApi', {table, ...})
};
makeAppStacks gets called in two places. DeployStage.ts and app.ts are generic and rarely change:
// DeployStage.ts
export class DeployStage extends cdk.Stage {
constructor(scope: Construct, id: string, props: DeployStageProps) {
super(scope, id, props);
makeAppStacks(this, props.appName, props.env.account, props.env.region);
}
}
// app.ts
const app = new cdk.App();
const account = process.env.AWS_ACCOUNT;
makeAppStacks(app, 'MyApp', account, 'us-east-1');
Add some scripts for convenience:
"scripts": {
"---- app (sandbox env) ----": "",
"deploy-sandbox:cdk": "AWS_ACCOUNT=<Sandbox Acct> npx cdk deploy '*' --app 'ts-node ./bin/app.ts' --profile sandbox --outputs-file cdk.outputs.json",
"deploy-sandbox": "build && test && deploy-sandbox:cdk",
"destroy-sandbox": ...,
"synth-sandbox": ...,
"---- app-pipeline (pipeline env) ----": "",
"deploy-pipeline:cdk": "npx cdk deploy '*' --app 'ts-node ./bin/app-pipeline.ts' --profile pipeline",
"deploy-pipeline": "build && deploy-pipeline:cdk",
}
so my goal is to deploy a serverless Dockerized NextJS application on ECS/Fargate.
So when I docker build my project using the command docker build . -f development.Dockerfile --no-cache -t myapp:latest everything is running successfully except Docker build doesn't consider the env file in my project's root directory. Once build finishes, I push the Docker image to Elastic Container Repository(ECR) and my Elastic Container Service(ECS) references that ECR.
So naturally, my built image doesn't have a ENV file(contains the API keys and DB credentials), and as a result my app is deployed but all of the services relying on those credentials are failing because there isn't an ENV file in my container and all of the variables become undefined or null.
To fix this issue I looked at this AWS doc and implemented a solution that stores my .env file in AWS S3 and that S3 ARN gets refrenced in the container service where the .env file is stored. However, that didn't workout and I think it's because of the way I'm setting my
next.config.js to reference my environmental files in my local codebase. I also tried to set my environmental variables manually(very unsecure, screenshot below) when configuring the container in my task defination, and that didn't work either.
My next.confg.js
const dotEnvConfig = { path: `../../${process.env.NODE_ENV}.env` };
require("dotenv").config(dotEnvConfig);
module.exports = {
serverRuntimeConfig: {
// Will only be available on the server side
xyzKey: process.env.xyzSecretKey || "",
},
publicRuntimeConfig: {
// Will be available on both server and client
appUrl: process.env.app_url || "",
},
};
So on my local codebase in the root directory I have two files development.env (local api keys) and production.env(live api keys) and my next.config.js is located in /packages/app/next.config.js
So apparently it was just a plain NextJS's way of handling env variables.
In next.config.js
module.exports = {
env: {
user: process.env.SQL_USER || "",
// add all the env var here
},
};
and to call the environmental variable user in the app all you have to do is call process.env.user and user will reference process.env.SQL_USER in my local .env file where it will be stored as SQL_USER="abc_user"
You should be setting the environment variables in the ECS task definition. In order to prevent storing sensitive values in the task definition you should use AWS Parameter Store, or AWS Secrets Manager, as documented here.
I am trying to debug lambda function locally using SAM cli and AWS CDK. So I am getting error function module not found any idea why so? I have taken this project from github https://github.com/mavi888/cdk-serverless-get-started
function.js:
exports.handler = async function (event) {
console.log("request:", JSON.stringify(event));
// return response back to upstream caller
return sendRes(200, "HELLLOOO");
};
const sendRes = (status, body) => {
var response = {
statusCode: status,
headers: {
"Content-Type": "text/html",
},
body: body,
};
return response;
};
Inside lib folder
// lambda function
const dynamoLambda = new lambda.Function(this, "DynamoLambdaHandler", {
runtime: lambda.Runtime.NODEJS_12_X,
code: lambda.Code.asset("functions"),
handler: "function.handler",
environment: {
HELLO_TABLE_NAME: table.tableName,
},
});
I am using cdk synth > template.yaml command which generates cloud formation template.yaml file. Now I find function name with logicalID eg: myFunction12345678 and then trying to debug it locally using this command sam local invoke myFunction12345678 in my case it is DynamoLambdaHandler function. I get function module not found error. Any idea what I am missing?
Code is available on github: https://github.com/mavi888/cdk-serverless-get-started
The issue is that sam runs a Docker container with a Volume mount from the current directory. So, it's not finding the Lambda code because the path to the code from your CloudFormation template that CDK creates does not include the cdk.out directory in which cdk creates the assets.
You have two options:
Run your sam command with a defined volume mount sam local invoke -v cdk.out
Run the command from within the cdk.out directory and pass the JSON template as an argument since cdk writes a JSON template: sam local invoke -t <StackNameTemplate.json>
I'd recommend the latter because you're working within the framework that CDK creates and not creating additional files.
I'm using AWS CodeBuild to build an application, it is configured to push the build artifacts to an AWS S3 bucket.
On inspecting the artifcats/objects in the S3 bucket I realised that the objects has been encrypted.
Is it possible to disable to encryption on the artifcats/objects?
There is now a checkbox named "Disable artifacts encryption" under the artifacts section which allows you to disable encryption when pushing artifacts to S3.
https://docs.aws.amazon.com/codebuild/latest/APIReference/API_ProjectArtifacts.html
I know this is an old post but I'd like to add my experience in this regard.
My requirement was to get front end assets from a code commit repository, build them and put them in s3 bucket. s3 bucket is further connected with cloudfront for serving the static front end content (written in react in my case).
I found that cloudfront is unable to serve KMS encrypted content as I found KMS.UnrecognizedClientException when I hit the cloudfront Url. I tried to fix that and disabling encryption on aws codebuild artifacts seemed to be the easiest solution when I found this
However, I wanted to manage this using aws-cdk. This code snippet in TypeScript may come handy if you're trying to solve the same issue using aws-cdk
Firstly, get your necessary imports. For this answer it'd be the following:
import * as codecommit from '#aws-cdk/aws-codecommit';
import * as codebuild from '#aws-cdk/aws-codebuild';
Then, I used the following snippet in a class that extends to cdk Stack
Note: The same should work if your class extends to a cdk Construct
// replace these according to your requirement
const frontEndRepo = codecommit.Repository
.fromRepositoryName(this, 'ImportedRepo', 'FrontEnd');
const frontendCodeBuild = new codebuild.Project(this, 'FrontEndCodeBuild', {
source: codebuild.Source.codeCommit({ repository: frontEndRepo }),
buildSpec: codebuild.BuildSpec.fromObject({
version: '0.2',
phases: {
build: {
commands: [
'npm install && npm run build',
],
},
},
artifacts: {
files: 'build/**/*'
}
}),
artifacts: codebuild.Artifacts.s3({
bucket: this.bucket, // replace with s3 bucket object
includeBuildId: false,
packageZip: false,
identifier: 'frontEndAssetArtifact',
name: 'artifacts',
encryption: false // added this to disable the encryption on codebuild
}),
});
Also to ensure that everytime I push a code in the repository, a build is triggered, I added the following snippet in the same class.
// add the following line in your imports if you're using this snippet
// import * as targets from '#aws-cdk/aws-events-targets';
frontEndRepo.onCommit('OnCommit', {
target: new targets.CodeBuildProject(frontendCodeBuild),
});
Note: This may not be a perfect solution, but it's working well for me till now. I'll update this answer if I find a better solution using aws-cdk
Artifact encryption cannot be disabled in AWS CodeBuild
I have tried with following code.
var exec = require('child_process').execFile;
var runCmd = 'java -jar ' + process.env.LAMBDA_TASK_ROOT + '/src/' + 'myjar.jar'
exec(runCmd,
function (err, resp) {
if (err) {
cb(null, { err: err})
} else {
cb(null, { resp: resp})
}
)
Here, I have put my jar file in the root folder and src folder also.
but it is giving my following error. I have already added the.jar file with the code.but i got following error.
"err": {
"code": "ENOENT",
"errno": "ENOENT",
"syscall": "spawn java -jar /var/task/src/myjar.jar",
"path": "java -jar /var/task/src/myjar.jar",
"spawnargs": [],
"cmd": "java -jar /var/task/src/myjar.jar"
}
So How, Can I execute this .jar file in AWS Lambda environment?
Please help me.
With Lambda Layers you can now bring in multiple runtimes.
https://github.com/lambci/yumda and https://github.com/mthenw/awesome-layers both have a lot of prebuilt packages that you can use to create a layer so you have a second runtime available in your environment.
For instance, I'm currently working on a project that uses the Ruby 2.5 runtime on top of a custom layer built from lambci/yumbda to provide Java.
mkdir dependencies
docker run --rm -v "$PWD"/dependencies:/lambda/opt lambci/yumda:1 yum install -y java-1.8.0-openjdk-devel.x86_64
cd dependencies
zip -yr ../javaLayer .
upload javaLayer.zip to aws lambda as a layer
add layer to your function
within your function, java will be located at /opt/lib/jvm/{YOUR_SPECIFIC_JAVA_VERSION}/jre/bin/java
AWS Lambda lets you select a runtime at the time of creation of that lambda function, or later you can change it again.
So, as you are running the Lambda function with NodeJs runtime, the container will not have Java runtime available to it.
You can only have one type of runtime in one container in case of AWS Lambda.
So, Create a separate Lambda with the Jar file that you want to run having Java as the runtime and then you can trigger that lambda function from your current NodeJS lambda function if that's what you ultimately want.
Following is an example of how you can call another Lambda function using NodeJS
var aws = require('aws-sdk');
var lambda = new aws.Lambda({
region: 'put_your_region_here'
});
lambda.invoke({
FunctionName: 'lambda_function_name',
Payload: JSON.stringify(event, null, 2)
}, function(error, data) {
if (error) {
context.done('error', error);
}
if(data.Payload){
context.succeed(data.Payload)
}
});
You can refer to the official documentation for more details.
In addition to the other answers: Since 2020 December, Lambda supports container images: https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/
Ex.: I created a container image using AWS's open-source base image for python, adding a line to install java. One thing my python code did was execute a .jar file using a sys call.