I have a lambda function that writes to a kinesis stream. But now, I want to write to a kinesis stream which belongs to a different AWS account. Assuming I have all the necessary cross account permissions, how can I send data to this stream? How should I change the parameters when I call the kinesis constructor or the putRecord function?
There is the method above which would technically work, however hardcoding creds or even configuring creds into a lambda seems a bit extraneous to me since lambdas themselves require that you have a role. What you need to do is create a cross account trust and assume role using sts.
Create a role in the account with the kinesis stream, and set it to trust your lambda role.
Give that role a policy that allows it to put to the kinesis stream.
In your lambda code use sts to create a session in the account with the kinesis stream and put your record.
Note your lambda will need a policy that allows it to sts into the second account's role.
It is described a bit more clearly here Providing Access to Accounts you Own
First you need to configure the Kinesis instance:
(I chose Javascript for the example)
var kinesis = new AWS.Kinesis({
accessKeyId: 'XXX',
secretAccessKey: 'YYY',
region: 'eu-west-1',
apiVersion: '2013-12-02'
});
For more informations take a look Constructing a Kinesis object
To write/put a record use the following
var params = {
Data: new Buffer('...') || 'STRING_VALUE', /* required */
PartitionKey: 'STRING_VALUE', /* required */
StreamName: 'STRING_VALUE', /* required */
ExplicitHashKey: 'STRING_VALUE',
SequenceNumberForOrdering: 'STRING_VALUE'
};
kinesis.putRecord(params, function (err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
For more informations take a look Calling the putRecord operation
Related
Say I have two accounts 111111111111 and 222222222222 and want to do following.
(Lambda) -> (Kinesis)
111111111111 222222222222
Where Lambda function is a trigger for a data source (could be another Kinesis stream in 111111111111).
exports.handler = async (event, context) => {
// data transformed here
const result = event.records.map(record => {});
return {data: result};
}
I am trying to format the data in 111111111111's Lambda Function and then send it to 22222222222's Kinesis stream, but I couldn't find many resources on this.
I came across this SO post. IAM role aside, it seems like each invocation of the Lambda Function needs to create a session with 22222222222 account and creates a Kinesis instance in order to call PutRecord. This looks like a red flag to me as I was thinking Lambda function could just set up a cross-account destination with resourceArn to send its result data to. What am I missing and is there better alternate to doing this?
This looks like a red flag to me as I was thinking Lambda function could just set up a cross-account destination with resourceArn to send its result data to.
This is not a red flag. The cross-account IAM roles is how it is done for kinesis, because kinesis streams don't have resource-based policies. So you have to assume IAM role from account 2, in your lambda.
I'm not sure which resourceArn are you referring to. The only one I can think of is resourceArn for Kinesis Data Analytics. This does not apply to Kinesis Data Streams.
I faced a problem when a user has signed up but doesn't want to confirm his email. The solution is to delete an unconfirmed user from AWS Cognito.
So as I don't know his password, I am trying to write a Lambda function which I will trigger through API Gateway. This lambda should remove Cognito user.
I wrote this code but it doesn't work.
var cognitoidentityserviceprovider = new AWS.CognitoIdentityServiceProvider({
apiVersion: '2016-03-18',
});
var params = {
UserPoolId: 'us-east-1_123456',
Username: 'user#mail.com' // I want to remove this user
};
cognitoidentityserviceprovider.adminDeleteUser(params, function (err, data) {
if (err) {
callback(err, err.stack);
} else {
callback(data);
}
});
I get an error:
user is not authorized to perform ...
Because of security, I don't want to set my admin credentials on frontend part and I want to do all work in this lambda... How to do it?
Any ideas?
Any solutions to prevent this problem?
You can assign a role to the lambda function and make a call to cognito api without passing any argument to the library you use to access aws services, that way the credential provider would fallback to the assumed role and have the lambda execution role's identity.
Usually roles are the way to go with amazon related authorizations.
Btw, this means that you have to create an iam role, a policy with the right cognito actions allowed and attach it to said role.
const cisp = new CognitoIdentityServiceProvider({ apiVersion: '2016-04-18' })
cisp.adminDeleteUser().promise() //delete current user as admin
From following this question, AWS DynamoDB Stream into Redshift
DynamoDB --> DynamoDBStreams --> Lambda Function --> Kinesis Firehose --> Redshift.
How do I configure my Kinesis function to pick up the Lambda function source?
I created a DynamoDB table (Purchase Sales), and Added DynamoDB Streams. Then I configured the Lambda function to pickup the DynamoDB Stream. My question is how do I configure Kinesis to pick up the Lambda function Source? I know how to configure Lambda Transformation, however would like to pick up as Source. Not sure how to configure the Direct Put Source below.
Thanks,
Performed these steps:
In your case, you would stream the dynamodb to redshift
DynamoDB --> DynamoDBStreams --> Lambda Function --> Kinesis Firehose --> Redshift.
First, you need a lambda function handle the DynamoDBStream. For each DynamoDBStream event, use firehose PutRecord API to send the data to firehose. From the example
var firehose = new AWS.Firehose();
firehose.putRecord({
DeliveryStreamName: 'STRING_VALUE', /* required */
Record: { /* required */
Data: new Buffer('...') || 'STRING_VALUE' /* Strings will be Base-64 encoded on your behalf */ /* required */
}
}, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
Next, we have to know how the data being insert into the RedShift. From the firehose document,
For data delivery to Amazon Redshift, Kinesis Firehose first delivers
incoming data to your S3 bucket in the format described earlier.
Kinesis Firehose then issues an Amazon Redshift COPY command to load
the data from your S3 bucket to your Amazon Redshift cluster.
So, we should know what data format to let the COPY command map the data into RedShift schema. We have to follow the data format requirement for redshift COPY command.
By default, the COPY command expects the source data to be
character-delimited UTF-8 text. The default delimiter is a pipe
character ( | ).
So, you could program the lambda which input dynamodb stream event, transform it to pipe (|) separated line record, and write it to firehose.
var firehose = new AWS.Firehose();
firehose.putRecord({
DeliveryStreamName: 'YOUR_FIREHOSE_NAME',
Record: { /* required */
Data: "RED_SHIFT_COLUMN_1_DATA|RED_SHIFT_COLUMN_2_DATA\n"
}
}, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
remember to add \n as the firehose will not append new line for you.
Can someone tell me how to get access to AWS credentials in an AWS Lambda function?
I've searched the internet thoroughly but still haven't found anything that has helped me.
Im writing the function in Java. I think I should have access to the credentials with the context object in the HandleRequest method.
If it helps, I want to invoke a DynamoDB client and upload a record to the database.
I came into the same problem myself recently.
This certainly is a blind spot in AWS's Lambda documentation for Java, in my opinion.
This snippet in Java should work for you, assuming you're using the AWS SDK for Java Document API :
DynamoDB dynamodb = new DynamoDB(
new AmazonDynamoDBClient(new EnvironmentVariableCredentialsProvider()));
The main takeaway is to use the EnvironmentVariableCredentialsProvider to access the required credentials to access your other AWS resources within the AWS Lambda container. The Lambda containers are shipped with credentials as environment variables, and this is sufficient in retrieving them.
Note: This creates a DynamoDB instance that only sees resources in the default region. To create one for a specific region, use this (assuming you want to access DynamoDB's in the ap-northeast-1 region):
DynamoDB dynamodb = new DynamoDB(
Regions.getRegion(Regions.AP_NORTHEAST_1).createClient(
AmazonDynamoDBClient.class,
new EnvironmentVariableCredentialsProvider(),
new ClientConfiguration()));
Your Lambda function's permissions are controlled by the IAM Role it executes as. Either add Dynamo PutItem permission to the current role, or create a new role for this purpose.
After giving permissions to the Role, you don't need to write special code to handle credentials, just use the AWS SDK normally. For example:
var AWS = require("aws-sdk");
exports.handler = function(event, context) {
var dynamodb = new AWS.DynamoDB();
var putItemParams = {
"TableName": "Scratch",
"Item": {
"Id": {
"S": "foo"
},
"Text": {
"S": "bar"
}
}
};
dynamodb.putItem(putItemParams, function (err, response) {
if (err) {
context.fail("dynamodb.putItem failed: " + err);
} else {
context.succeed("dynamodb.putItem succeeded");
}
});
};
Is sufficient to put an item in a DynamoDB table, with the correct Role permissions.
Adding to #Gordon Tai's answer, using the current api using AmazonDynamoDBClientBuilder this looks like:
AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard()
.withCredentials(new EnvironmentVariableCredentialsProvider())
.withRegion(Regions.US_EAST_1)
.build();
DynamoDB dynamoDB = new DynamoDB(client);
I have an Amazon Lambda instance and an Amazon SNS instance. The Lambda code watches for changes in our database and I would like it to make a call to Amazon SNS to send pushes to our users. For example:
When a user on one of our forums gets a new message, the Lambda code recognizes this change every time it is run (every 10 minutes) and should send a push to the user's smartphone via SNS.
I'm running into a wall when it comes to the documentation; Amazon's docs only talk about how to trigger Lambda code via SNS, but not the reverse. Does anyone have an example of how I can accomplish this?
There is nothing special about pushing SNS notifications in the context of Lambda. I would think of it as just another external service that you interact with.
What you could do is pull in the AWS SDK in your lambda code and after that use the code to make the SNS calls. You will need to inject the right credentials to be able to call the Amazon SNS API (but you probably do something similar for getting the database endpoint and credentials if you are talking to the database)
Yes, you can use AWS Lambda to achieve what you want. You also need to give proper IAM Permissions allowing your Lambda IAM Role to publish messages to you SNS Topic.
Example SNS Publish IAM Policy:
{
"Statement":[ {
"Effect":"Allow",
"Action":"sns:Publish",
"Resource":"arn:aws:sns:*:<your account id>:<your topic id>"
} ]
}
You can use the lambda below to push an SNS message to a user, but you must know what the endpoint ARN is for that user. For example, if in an Android app, when the user logs in you will have the app send a GCM (Google Cloud Messaging) token to your backend (via an API call that triggers a lambda, for example). Your backend, which is connected to GCM, can then use this token to lookup which endpoint ARN corresponds to such user and put that in the lambda below. Alternatively, you can have the app send the endpoint ARN directly to your backend, though I think it might be a bit less secure. Make sure you give IAM permissions to publish to your app via SNS. You can use the lambda below to push the message:
var AWS = require('aws-sdk');
var sns = new AWS.SNS({apiVersion: '2010-03-31'});
exports.handler = (event, context, callback) => {
console.log(JSON.stringify(event))
var payload = {
"default": "The message string.",
"GCM":"{"+
"\"notification\":{"+
"\"body\":\"PUT NOTIFICATION BODY HERE\","+
"\"title\":\"PUT NOTIFICATION TITLE HERE\""+
"}"+
"}"
};
payload = JSON.stringify(payload);
var params = {
TargetArn: 'PUT THE ENDPOINT ARN HERE',
Subject: 'foo2',
MessageStructure: 'json',
Message: payload
}
sns.publish(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
};