I have created AWS Lambda function to run when new fils in S3 at specific path is created, which works perfectly.
service: redshift
frameworkVersion: '2'
custom:
bucket: extapp
path_prefix: 'xyz'
database: ABC
schema: xyz_dbo
table_prefix: shipmentlog
user: admin
password: "#$%^&*(*&^%$%"
port: 5439
endpoint: "*********.redshift.amazonaws.com"
role: "arn:aws:iam::*****:role/RedshiftFileTransfer"
provider:
name: aws
runtime: python3.8
stage: prod
region: us-west-2
stackName: redshift-prod-copy
stackTags:
Service: "it"
lambdaHashingVersion: 20201221
memorySize: 128
timeout: 900
logRetentionInDays: 14
environment:
S3_BUCKET: ${self:custom.bucket}
S3_BUCKET_PATH_PREFIX: ${self:custom.path_prefix}
REDSHIFT_DATABASE: ${self:custom.database}
REDSHIFT_SCHEMA: ${self:custom.schema}
REDSHIFT_TABEL_PREFIX: ${self:custom.table_prefix}
REDSHIFT_USER: ${self:custom.user}
REDSHIFT_PASSWORD: ${self:custom.password}
REDSHIFT_PORT: ${self:custom.port}
REDSHIFT_ENDPOINT: ${self:custom.endpoint}
REDSHIFT_ROLE: ${self:custom.role}
iam:
role:
name: s3-to-redshift-copy
statements:
- Effect: Allow
Action:
- s3:GetObject
Resource: "arn:aws:s3:::${self:custom.bucket}/*"
functions:
copy:
handler: handler.run
events:
- s3:
bucket: ${self:custom.bucket}
event: s3:ObjectCreated:*
rules:
- prefix: ${self:custom.path_prefix}/
- suffix: .json
existing: true
package:
exclude:
- node_modules/**
- package*.json
- README.md
plugins:
- serverless-python-requirements
But when I deployed this function, there was also another function get deployed with name redshift-prod-custom-resource-existing-s3 which is Node.js function. I want to understand why this second function necessary for triggering primary lambda function when new file is creates in S3 bucket at specific path.
It's the serverless Framework's method of adding the trigger to call the lambda to the S3 bucket via a Custom Resource
I want to create an s3 bucket, and trigger a lambda function whenever some file is uploaded to 'uploads' folder in the bucket. I want to create those resources using serverless framework in aws.
I have defined my s3 bucket configuration in under 'provider.s3', and then I am trying to reference that bucket under functions.hello.events.bucket
However, I am getting following error when i run sls package
Serverless Error ----------------------------------------
MyS3Bucket - Bucket name must conform to pattern (?!^(\d{1,3}\.){3}\d{1,3}$)(^(([a-z0-9]|[a-z0-9][a-z0-9-]*[a-z0-9])\.)*([a-z0-9]|[a-z0-9][a-z0-9-]*[a-z0-9])$). Please check provider.s3.MyS3Bucket and/or s3 events of function "hello".
serverless.yml
service: some-service
frameworkVersion: '2'
useDotenv: true
provider:
name: aws
runtime: python3.8
lambdaHashingVersion: 20201221
s3:
MyS3Bucket:
bucketName: ${env:MY_BUCKET_NAME}
accessControl: Private
lifecycleConfiguration:
Rules:
- Id: ExpireRule
Status: Enabled
ExpirationInDays: '7'
package:
individually: true
functions:
hello:
name: my-lambda-function
handler: function.handler
memorySize: 128
timeout: 900
events:
- s3:
bucket: MyS3Bucket
event: s3:ObjectCreated:*
rules:
- prefix: uploads/
My next try was defining the s3 bucket under 'resources', and using the reference of the bucket in the lambda trigger. I am still getting the warning messages
Serverless: Configuration warning at 'functions.hello.events[0].s3.bucket': should be string
serverless.yml
service: some-service
frameworkVersion: '2'
useDotenv: true
provider:
name: aws
runtime: python3.8
lambdaHashingVersion: 20201221
package:
individually: true
functions:
hello:
name: my-lambda-function
handler: handler.handler
memorySize: 128
timeout: 900
events:
- s3:
bucket:
Ref: MyS3Bucket
event: s3:ObjectCreated:*
rules:
- prefix: uploads/
existing: true
resources:
Resources:
MyS3Bucket:
Type: AWS::S3::Bucket
Properties:
AccessControl: Private
BucketName: 'test.bucket'
OwnershipControls:
Rules:
- ObjectOwnership: ObjectWriter
LifecycleConfiguration:
Rules:
- Id: ExpireRule
Status: Enabled
ExpirationInDays: '7'
You should use your bucket name, not MyS3Bucket:
events:
- s3:
bucket: ${env:MY_BUCKET_NAME}
Create a custom s3 bucket name variable e.g.
custom:
bucket: foo-thumbnails
And use it events
events:
- s3:
bucket: ${self:custom.bucket}
I am using a serverless framework to deploy a serverless stack to AWS. My stack consists of some lambda functions, DynamoDB tables and API Gateway.
I am protected The API Gateway using what's called lambda authorizer. Also, I have a custom standalone self-hosted Auth service that can generate tokens.
So the scenario is that the user can request a token from this service (It's IdentityServer4 hosted on Azure) then the user can send a request to the API Gateway with the bearer token so the API gateway will ask the lambda authorizer to generate iam roles if the token is correct. All of that is valid and works as expected.
Here is an example of the lambda authorizer definition in my serverless.yml and how I use it to protect other API gateway endpoints: (You can see the addUserInfo function has API that protected using the custom authorizer )
functions:
# =================================================================
# API Gateway event handlers
# ================================================================
auth:
handler: api/auth/mda-auth-server.handler
addUserInfo:
handler: api/user/create-replace-user-info.handler
description: Create Or Replace user section
events:
- http:
path: user
method: post
authorizer:
name: auth
resultTtlInSeconds: ${self:custom.resultTtlInSeconds}
identitySource: method.request.header.Authorization
type: token
cors:
origin: '*'
headers: ${self:custom.allowedHeaders}
Now I wanted to extend my APIs so I will allow the user to add images, so I followed this approach. So in this approach, the user will initiate what's called a signed S3 URL and I can put an image to my bucket using this S3 signed URL.
Also, the S3 bucket is not publicly accessible but instead, it's connected to CloudFront distribution. Now I missed the things here, I can't understand how I can protect my images. Is it anyway so I can protect the Images in the CloudFront CDN with my custom Authentication service so the user that has a valid token can just access those resources? How can I protect my CDN (CloudFront) using my Custom Authentication service and configure that using the serverless framework?
This is a bit tricky and it takes from me around a day to get all set.
First we have options here:
Instead of authentication we can sign the URL and return a signed CloudFront URL or signed S3 URL and it's pretty straight forward but obviously that not what I was looking for.
The second option is to use Lambda#Edge to authorize the requests of the CloudFront and that what I followed.
So I ended up create a separate stack to handle all the S3, CloudFront, and Lambda#Edge stuff cause they are all deployed on edges which means that the region doesn't matter but for lambda edge we need to deploy it to the main AWS region ((N. Virginia), us-east-1) So i ended up creating one stack for all of them.
First I have the below code in my auth-service.js (It's just some helpers to allow me to verify my custom jwt):
import * as jwtDecode from 'jwt-decode';
import * as util from 'util';
import * as jwt from 'jsonwebtoken';
import * as jwksClient from 'jwks-rsa';
export function getToken(bearerToken) {
if(bearerToken && bearerToken.startsWith("Bearer "))
{
return bearerToken.replace(/^Bearer\s/, '');
}
throw new Error("Invalid Bearer Token.");
};
export function getDecodedHeader(token) {
return jwtDecode(token, { header: true });
};
export async function getSigningKey(decodedJwtTokenHeader, jwksclient){
const key = await util.promisify(jwksclient.getSigningKey)(decodedJwtTokenHeader.kid);
const signingKey = key.publicKey || key.rsaPublicKey;
if (!signingKey) {
throw new Error('could not get signing key');
}
return signingKey;
};
export async function verifyToken(token,signingKey){
return await jwt.verify(token, signingKey);
};
export function getJwksClient(jwksEndpoint){
return jwksClient({
cache: true,
rateLimit: true,
jwksRequestsPerMinute: 10,
jwksUri: jwksEndpoint
});
};
Then inside the serverless.yml here is my file:
service: mda-app-uploads
plugins:
- serverless-offline
- serverless-pseudo-parameters
- serverless-iam-roles-per-function
- serverless-bundle
custom:
stage: ${opt:stage, self:provider.stage}
resourcesBucketName: ${self:custom.stage}-mda-resources-bucket
resourcesStages:
prod: prod
dev: dev
resourcesStage: ${self:custom.resourcesStages.${self:custom.stage}, self:custom.resourcesStages.dev}
provider:
name: aws
runtime: nodejs12.x
stage: ${opt:stage, 'dev'}
region: us-east-1
versionFunctions: true
functions:
oauthEdge:
handler: src/mda-edge-auth.handler
role: LambdaEdgeFunctionRole
memorySize: 128
timeout: 5
resources:
- ${file(resources/s3-cloudfront.yml)}
Quick points here:
The us-east-1 important here.
It's a bit tricky and not practical to create any lambda edge using the serverless framework so I used it to just configure the function and then inside this cloud formation template resources/s3-cloudfront.yml I added all the needed bits.
Then here is the content of resources/s3-cloudfront.yml:
Resources:
AuthEdgeLambdaVersion:
Type: Custom::LatestLambdaVersion
Properties:
ServiceToken: !GetAtt PublishLambdaVersion.Arn
FunctionName: !Ref OauthEdgeLambdaFunction
Nonce: "Test"
PublishLambdaVersion:
Type: AWS::Lambda::Function
Properties:
Handler: index.handler
Runtime: nodejs12.x
Role: !GetAtt PublishLambdaVersionRole.Arn
Code:
ZipFile: |
const {Lambda} = require('aws-sdk')
const {send, SUCCESS, FAILED} = require('cfn-response')
const lambda = new Lambda()
exports.handler = (event, context) => {
const {RequestType, ResourceProperties: {FunctionName}} = event
if (RequestType == 'Delete') return send(event, context, SUCCESS)
lambda.publishVersion({FunctionName}, (err, {FunctionArn}) => {
err
? send(event, context, FAILED, err)
: send(event, context, SUCCESS, {FunctionArn})
})
}
PublishLambdaVersionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: PublishVersion
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: lambda:PublishVersion
Resource: '*'
LambdaEdgeFunctionRole:
Type: "AWS::IAM::Role"
Properties:
Path: "/"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Sid: "AllowLambdaServiceToAssumeRole"
Effect: "Allow"
Action:
- "sts:AssumeRole"
Principal:
Service:
- "lambda.amazonaws.com"
- "edgelambda.amazonaws.com"
LambdaEdgeFunctionPolicy:
Type: "AWS::IAM::Policy"
Properties:
PolicyName: MainEdgePolicy
PolicyDocument:
Version: "2012-10-17"
Statement:
Effect: "Allow"
Action:
- "lambda:GetFunction"
- "lambda:GetFunctionConfiguration"
Resource: !GetAtt AuthEdgeLambdaVersion.FunctionArn
Roles:
- !Ref LambdaEdgeFunctionRole
ResourcesBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: ${self:custom.resourcesBucketName}
AccessControl: Private
CorsConfiguration:
CorsRules:
- AllowedHeaders: ['*']
AllowedMethods: ['PUT']
AllowedOrigins: ['*']
ResourcesBucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket:
Ref: ResourcesBucket
PolicyDocument:
Statement:
# Read permission for CloudFront
- Action: s3:GetObject
Effect: "Allow"
Resource:
Fn::Join:
- ""
-
- "arn:aws:s3:::"
-
Ref: "ResourcesBucket"
- "/*"
Principal:
CanonicalUser: !GetAtt CloudFrontOriginAccessIdentity.S3CanonicalUserId
- Action: s3:PutObject
Effect: "Allow"
Resource:
Fn::Join:
- ""
-
- "arn:aws:s3:::"
-
Ref: "ResourcesBucket"
- "/*"
Principal:
AWS: !GetAtt LambdaEdgeFunctionRole.Arn
- Action: s3:GetObject
Effect: "Allow"
Resource:
Fn::Join:
- ""
-
- "arn:aws:s3:::"
-
Ref: "ResourcesBucket"
- "/*"
Principal:
AWS: !GetAtt LambdaEdgeFunctionRole.Arn
CloudFrontOriginAccessIdentity:
Type: AWS::CloudFront::CloudFrontOriginAccessIdentity
Properties:
CloudFrontOriginAccessIdentityConfig:
Comment:
Fn::Join:
- ""
-
- "Identity for accessing CloudFront from S3 within stack "
-
Ref: "AWS::StackName"
- ""
# Cloudfront distro backed by ResourcesBucket
ResourcesCdnDistribution:
Type: AWS::CloudFront::Distribution
Properties:
DistributionConfig:
Origins:
# S3 origin for private resources
- DomainName: !Sub '${self:custom.resourcesBucketName}.s3.amazonaws.com'
Id: S3OriginPrivate
S3OriginConfig:
OriginAccessIdentity: !Sub 'origin-access-identity/cloudfront/#{CloudFrontOriginAccessIdentity}'
# S3 origin for public resources
- DomainName: !Sub '${self:custom.resourcesBucketName}.s3.amazonaws.com'
Id: S3OriginPublic
S3OriginConfig:
OriginAccessIdentity: !Sub 'origin-access-identity/cloudfront/#{CloudFrontOriginAccessIdentity}'
Enabled: true
Comment: CDN for public and provate static content.
DefaultRootObject: index.html
HttpVersion: http2
DefaultCacheBehavior:
AllowedMethods:
- DELETE
- GET
- HEAD
- OPTIONS
- PATCH
- POST
- PUT
Compress: true
TargetOriginId: S3OriginPublic
ForwardedValues:
QueryString: false
Headers:
- Origin
Cookies:
Forward: none
ViewerProtocolPolicy: redirect-to-https
CacheBehaviors:
-
PathPattern: 'private/*'
TargetOriginId: S3OriginPrivate
AllowedMethods:
- DELETE
- GET
- HEAD
- OPTIONS
- PATCH
- POST
- PUT
Compress: true
LambdaFunctionAssociations:
-
EventType: viewer-request
LambdaFunctionARN: !GetAtt AuthEdgeLambdaVersion.FunctionArn
ForwardedValues:
QueryString: false
Headers:
- Origin
Cookies:
Forward: none
ViewerProtocolPolicy: redirect-to-https
-
PathPattern: 'public/*'
TargetOriginId: S3OriginPublic
AllowedMethods:
- DELETE
- GET
- HEAD
- OPTIONS
- PATCH
- POST
- PUT
Compress: true
ForwardedValues:
QueryString: false
Headers:
- Origin
Cookies:
Forward: none
ViewerProtocolPolicy: redirect-to-https
PriceClass: PriceClass_200
Some quick points related to this file:
Here I created the S3 bucket that will contain all my private and public resources.
This bucket is private and not accessible and you will find a role who just give the CDN and the lambda edge access to it.
I decided to create a CloudFront (CDN) with two origins public to be pointed to the S3's public folder and private to point it to the S3's private folder and configure the behavior of the CloudFront private origin to use my lambda edge function for the authentication through the viewer-request event type.
You will find also a code to create the function version and another function called PublishLambdaVersion with its role and it helps to give the lambda edge the correct permissions while deploying.
Finally here it the actually code for the lambda edge function used for CDN auth:
import {getJwksClient, getToken, getDecodedHeader, getSigningKey, verifyToken} from '../../../../libs/services/auth-service';
import config from '../../../../config';
const response401 = {
status: '401',
statusDescription: 'Unauthorized'
};
exports.handler = async (event) => {
try{
const cfrequest = event.Records[0].cf.request;
const headers = cfrequest.headers;
if(!headers.authorization) {
console.log("no auth header");
return response401;
}
const jwtValue = getToken(headers.authorization);
const client = getJwksClient(`https://${config.authDomain}/.well-known/openid-configuration/jwks`);
const decodedJwtHeader = getDecodedHeader(jwtValue);
if(decodedJwtHeader)
{
const signingKey = await getSigningKey(decodedJwtHeader, client);
const verifiedToken = await verifyToken(jwtValue, signingKey);
if(verifiedToken)
{
return cfrequest;
}
}else{
throw Error("Unauthorized");
}
}catch(err){
console.log(err);
return response401;
}
};
In case you are interested, I am using IdentityServer4 and hosted it as a docker image in Azure and using it as a custom authorizer.
So the full scenario now that we have an S3 bucket that totally private. It's only accessible through the CloudFront origins. If the request served through the public origin so no authentication needed but if it's served through the private origin so is I am triggering what's called lambda edge to authenticate it and validate the bearer token.
I was totally new to AWS stack before going deep into all of those but AWS is quite easy so I end up configured everything in a perfect way. Please let me know in case there is something not clear or if there are any questions.
Any idea why I’m getting an AccessDenied error when trying to upload to my S3 bucket?
serverless.yml:
service: foo-service
custom:
bucket: my-bucket-name
provider:
name: aws
iamRoleStatements:
- Effect: Allow
Action:
- s3:PutObject
Resource: "arn:aws:s3:::${self:custom.bucket}/*"
functions:
hello:
handler: handler.hello
environment:
BUCKET: ${self:custom.bucket}
I'm trying to add a file to S3 with public-read permissions.
The s3:PutObject permission alone allows you to add an item to the S3 bucket, but if you configure any ACL attributes you'll need the additional permission s3:PutObjectAcl.
It should be like this:
provider:
name: aws
iamRoleStatements:
- Effect: Allow
Action:
- s3:PutObject
- s3:PutObjectAcl
Resource: "arn:aws:s3:::${self:custom.bucket}/*"
I'm using Serverless to create a web application that serves its static content, e.g. a web font, from a S3 bucket. The S3 bucket is configured as a resource in my serverless.yml file. Its CORS configuration has an AllowOrigin set to a wildcard.
I want to change this to have an AllowOrigin with the http endpoint of the service as created by Serverless, e.g. 31alib51b6.execute-api.eu-west-1.amazonaws.com.
I wondered if it's possible to configure this in the serverless.yml file itself.
My example serverless.yml file:
service: example-service
provider:
name: aws
runtime: nodejs4.3
region: eu-west-1
functions:
web:
handler: handler.handler
name: ${self:service}-${self:provider.stage}
description: ${self:service} web application - ${self:provider.stage}
events:
- http:
path: web
method: get
- http:
path: web/{proxy+}
method: get
resources:
Resources:
S3Assets:
Type: AWS::S3::Bucket
Properties:
BucketName: ${self:service}-${self:provider.stage}-assets
CorsConfiguration:
CorsRules:
- AllowedMethods:
- GET
- HEAD
AllowedOrigins:
- "*"
You can define the AllowedOrigin with the following statement:
CorsConfiguration:
CorsRules:
- AllowedMethods:
- GET
- HEAD
AllowedOrigins:
- Fn::Join:
- ""
- - "https://"
- Ref: ApiGatewayRestApi
- ".execute-api.eu-west-1.amazonaws.com"
"Ref: ApiGatewayRestApi" references the internal name of the generated API.