AWS Cloudwatch setup with Winston - amazon-web-services

I have been reading various articles/docs and watching some videos on this topic. My issue is that they all conflict in one way or another.
My goal is to use winston to send all console.logs/error messages from my ec2 server to Cloudwatch so that no logs are ever logged on the ec2 terminal itself.
Points of confusion:
If I use winston-aws-cloudwatch or winston-cloudwatch, do I still need to setup an IAM user on AWS or will these auto generate logs within Cloudwatch?
If I setup Cloudwatch as per AWS documentation will that automatically stream any would be console.logs from the EC2 server to Cloudwatch or will it do both? If the first one, then I don't need Winston?
Can I send logs from my local development server to Cloudwatch (just for testing purposes, as soon as it is clear it works, then I would test on staging and finally move it to production) or must it come from an EC2 instance?
I assume the AWS Cloudwatch key is the same as the AWS key I use for the rest of my account?
Present code:
var winston = require('winston'),
CloudWatchTransport = require('winston-aws-cloudwatch');
const logger = new winston.Logger({
transports: [
new (winston.transports.Console)({
timestamp: true,
colorize: true
})
]
});
const cloudwatchConfig = {
logGroupName: 'groupName',
logStreamName: 'streamName',
createLogGroup: false,
createLogStream: true,
awsConfig: {
aws_access_key_id: process.env.AWS_KEY_I_USE_FOR_AWS,
aws_secret_access_key: process.env.AWS_SECRET_KEY_I_USE_FOR_AWS,
region: process.env.REGION_CLOUDWATCH_IS_IN
},
formatLog: function (item) {
return item.level + ': ' + item.message + ' ' + JSON.stringify(item.meta)
}
};
logger.level = 3;
if (process.env.NODE_ENV === 'development') logger.add(CloudWatchTransport, cloudwatchConfig);
logger.stream = {
write: function(message, encoding) {
logger.info(message);
}
};
logger.error('Test log');

Yes
Depends on the transports you configure. If you configure only CloudWatch than it will only end up there. Currently your code has 2 transports, the normal Console one and the CloudWatchTransport so with your current code, both.
As long as you specify your keys as you would normally do with any AWS service (S3, DB, ...) you can push logs from your local/dev device to CloudWatch.
Depends on your IAM user if he has the privileges or not. But it is possible yes.

Related

Is it possible to send emails via AWS SES on AWS LightSail instance?

I have tried sending emails on my LightSail instance via AWS SES using the following Node.js JavaScript AWS SDK code, but it failed with the following error message. The email sending code works fine on my development computer. (The email sending code is from https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/ses-examples-sending-email.html.)
(code here)
import { readFile } from 'fs/promises';
import * as path from 'path';
import { SESClient } from "#aws-sdk/client-ses";
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const awsConfigFileFullName = "aws_config.json";
let awsConfigFileFullPath = path.join(__dirname, awsConfigFileFullName);
const awsConfig = await readFile(awsConfigFileFullPath).then(json => JSON.parse(json)).catch(() => null);
aws_ses_client = new SESClient({ region: awsConfig.region, accessKeyId: awsConfig.accessKeyId, secretAccessKey: awsConfig.secretAccessKey });
const createSendEmailCommand = (toAddress, fromAddress, htmlContent, textContent, emailSubject) => {
return new SendEmailCommand({
Destination: {
/* required */
CcAddresses: [
/* more items */
],
ToAddresses: [
toAddress,
/* more To-email addresses */
],
},
Message: {
/* required */
Body: {
/* required */
Html: {
Charset: "UTF-8",
Data: htmlContent,
},
Text: {
Charset: "UTF-8",
Data: textContent,
},
},
Subject: {
Charset: "UTF-8",
Data: emailSubject,
},
},
Source: emailSenderName + '<' + fromAddress + '>',
ReplyToAddresses: [
/* more items */
],
});
};
const sendEmailCommand = createSendEmailCommand(
recipientEmailAddress,
senderEmailAddress,
htmlEmailContent,
textEmailContent,
emailSubject
);
try {
await aws_ses_client.send(sendEmailCommand);
} catch (e) {
console.error("Failed to send email.", e);
}
(error here)
AccessDenied: User arn:aws:sts::(some number):assumed-role/AmazonLightsailInstanceRole/i-(some alphanumeric number)is not authorized to perform ses:SendEmail' on resource
`arn:aws:ses:us-east-1:(some number):identity/(recipient email
address)'
After doing some search online on the issue, thinking that the error was caused by port 25 restriction on Lightsail instance restriction (https://aws.amazon.com/premiumsupport/knowledge-center/lightsail-port-25-throttle/), I sent a restriction removal request to AWS, but the request was declined, telling me to "consider looking into the Simple Email Service". I sent a reply asking if sending emails via AWS SES was possible on LightSail instance, but got a reply saying "we cannot grant your request"; I feel that the second reply was either automated or wasn't even thoroughly read by the person who reviewed the email.
I have a multisite WordPress installed on my LightSail webserver, which can send emails via AWS SES with WP Mail SMTP plugin, with TLS encryption using SMTP port 587. I think this is a proof that emails can be sent on a LightSail instance via AWS SES. Emails do get sent by my WordPress on my LightSail webserver, everyday, using my AWS SES SMTP credentials; so, maybe the AWS SES SMTP server must be directly contacted in the LightSail instance email sending code to send emails, instead of using the authenticated SES client object in the code?
I'm thinking that maybe the assumed role AmazonLightsailInstanceRole doesn't have AWS SES email sending allowed. I've checked my AWS IAM web console, and there was no role named AmazonLightsailInstanceRole; it doesn't look like I can modify the policy on the assumed role AmazonLightsailInstanceRole.
Can AWS SES email-sending permission be granted to the assumed role AmazonLightsailInstanceRole? If so, how?
Is it possible to send emails via AWS SES on an AWS LightSail instance? If so, how can it be done in Node.js JavaScript AWS SDK code? Any verifications, references and/or directions would be very helpful.
Sounds like an IAM issue - the instance role is the fallback option if you dont pass your own credentials. Lines 6..8 of the code is looking for a file (aws_config.json) which contains an API key and secret and a default region, then on line 9 its passing those values into SESClient. When you finally call aws_ses_client.send(sendEmailCommand) it will use the instance default credentails provided by lightsail which wont have access to resources in your aws account like SES. Check the following:
Does file aws_config.json exist?
Does it contain api key/secret/region for a valid user in your AWS account?
Does that user have a suitable IAM policy including ses:SendEmail and possibly other iam permissions?
FYI catch(() => null) could be hiding an error - you should use console.log in your catch() statements to write an error (at least while your debugging).
UPDATE
While testing my Node.js Express web app on an EC2 instance, to resolve the error I encountered, I came across the following: "Could not load credentials from any providers" while using dynamodb locally in Node.
Because I was using #aws-sdk/ses-client (version 3), not version 2, I should've used the following:
aws_s3_client = new S3Client({ region: awsConfig.region, credentials: {accessKeyId: awsConfig.accessKeyId, secretAccessKey: awsConfig.secretAccessKey }});
aws_ses_client = new SESClient({ region: awsConfig.region, credentials: {accessKeyId: awsConfig.accessKeyId, secretAccessKey: awsConfig.secretAccessKey }});
Using the above code enabled using both AWS SES and S3 from my web app running on a LightSail instance. So, the answer is, yes, it is possible to send emails via AWS SES on AWS LightSail instance.

Possible to trick AWS SDK `defaultProvider` function with a custom environment variable?

Problem: Netlify serverless functions run on AWS Lambda. So AWS_ is a reserved prefix in Netlify, meaning I can't use e.g. AWS_SECRET_ACCESS_KEY for my own environment var that I set in the Netlify admin panel.
But the only way I have been auble to authenticate Nodemailer with AWS SES (the email service) is with #aws/aws-sdk and its defaultProvider function that requires process.env.AWS_SECRET_ACCESS_KEY and process.env.AWS_ACCESS_KEY_ID – spelled exactly like that:
import 'dotenv/config'
import nodemailer from 'nodemailer'
import aws from '#aws-sdk/client-ses'
import { defaultProvider } from '#aws-sdk/credential-provider-node'
const ses = new aws.SES({
apiVersion: '2019-09-29',
region: 'eu-west-1',
defaultProvider,
rateLimit: 1
})
const sesTransporter = nodemailer.createTransport({ SES: { ses, aws } })
When building the function locally with the Netlify CLI, emails are sent.
It fails with 403 and InvalidClientTokenId: The security token included in the request is invalid in the live Netlify environment.
Netlify doesn't have a solution afaik, but mention in a forum post that custom env variables in AWS is a thing. I haven't been able to find anything in searches (they didn't provide any links). The AWS docs are pretty unhelpful as always :/
So the question is, how can this be done?
I thought I was clever when I tried the following, but setting the vars just before creating the SES Transport apparently doesn't help:
// Trick Netlify reserved env vars:
process.env.AWS_ACCESS_KEY_ID = process.env.ACCESS_KEY_ID
process.env.AWS_SECRET_ACCESS_KEY = process.env.SECRET_KEY
console.log('AWS access key id ', process.env.AWS_ACCESS_KEY_ID) // Logs the correct key!
console.log('AWS sec key ', process.env.AWS_SECRET_ACCESS_KEY ) // Logs the correct

AWS CDK Athena Data Source

How I can create an Athena data source in AWS CDK which is a JDBC connection to a MySQL database using the AthenaJdbcConnector?
I believe I can use aws-sam's CfnApplication to create the AthenaJdbcConnector Lambda, but how can I connect it to Athena?
I notice a lot of Glue support in CDK which would transfer to Athena (data catalog), and there are several CfnDataSource types in other modules such as QuickSight, but I'm not seeing anything under Athena in CDK.
See the image and references below.
References:
https://docs.aws.amazon.com/athena/latest/ug/athena-prebuilt-data-connectors-jdbc.html
https://github.com/awslabs/aws-athena-query-federation/tree/master/athena-jdbc
https://serverlessrepo.aws.amazon.com/applications/us-east-1/292517598671/AthenaJdbcConnector
I have been playing with the same issue. Here is what I did to create the Lambda for federated queries (Typescript):
const vpc = ec2.Vpc.fromLookup(this, 'my-project-vpc', {
vpcId: props.vpcId
});
const cluster = new rds.ServerlessCluster(this, 'AuroraCluster', {
engine: rds.DatabaseClusterEngine.AURORA_POSTGRESQL,
parameterGroup: rds.ParameterGroup.fromParameterGroupName(this, 'ParameterGroup', 'default.aurora-postgresql10'),
defaultDatabaseName: 'MyDB',
vpc,
vpcSubnets: {
onePerAz: true
},
scaling: {autoPause: cdk.Duration.seconds(0)} // Optional. If not set, then instance will pause after 5 minutes
});
let password = cluster.secret!.secretValueFromJson('password').toString()
let spillBucket = new Bucket(this, "AthenaFederatedSpill")
let lambdaApp = new CfnApplication(this, "MyDB", {
location: {
applicationId: "arn:aws:serverlessrepo:us-east-1:292517598671:applications/AthenaJdbcConnector",
semanticVersion: "2021.42.1"
},
parameters: {
DefaultConnectionString: `postgres://jdbc:postgresql://${cluster.clusterEndpoint.hostname}/MyDB?user=postgres&password=${password}`,
LambdaFunctionName: "crossref_federation",
SecretNamePrefix: `${cluster.secret?.secretName}`,
SecurityGroupIds: `${cluster.connections.securityGroups.map(value => value.securityGroupId).join(",")}`,
SpillBucket: spillBucket.bucketName,
SubnetIds: vpc.privateSubnets[0].subnetId
}
})
This creates the lambda with a default connection string like you would have it, if you used the AWS Console wizard in Athena to connect to a DataSource. Unfortunately it is NOT possible to add a Athena-catalog specific connection string via CDK. It should be set as an Environment Variable on the Lambda, and I found no way to do that. The Application template simply don't allow it, so this is a post-process by hand. I would sure like to hear from anybody if they have a solution for that!
Also notice that I add the user/password in the jdbc URL directly. I wanted to use SecretsManager, but because the Lambda is deployed in a VPC, it simply refuses to connect to the secretsmanager. I think this might be solvable by added a private VPN connection to SSM. Again - I would like to hear from anybody have tried that.

AWS IoT create things automatically

I am wondering how one creates many "things" in the AWS IoT solution via API without using the AWS web interface since this is not realistic in case I want thousands or millions of things. I guess you could write some script utilizing the "aws" client described here "http://docs.aws.amazon.com/iot/latest/developerguide/thing-registry.html" but thats not optimal if I want to control it from another service.
I assumed there would be a RESTish API to do this but it doesn't seem like it if I read the docs: "You use the AWS IoT console or the AWS CLI to interact with the registry."
Anyone who created thousands/millions of things - how did you interact with AWS IoT?
Ok, so I found it. Here it is possible to manage all the AWS IoT things:
http://docs.aws.amazon.com/iot/latest/apireference/API_Operations.html
anujdeshpande has given some headsup on this area.
"Automatically" means you can look at the following steps:
When do you want to create a thing automatically? What is that triggers that create-thing process?
Check this link for Just In Time registration of the devices
You can create a Lambda function mapped to the AWS API Gateway which can invoke the AWS-IOT SDK to create a thing.
Look at this page where they have created 50+ things using Javascript using aws-sdk library using serverless architecture approach
Another interesting way to do this is to use Just-in-Time registration - basically you upload the CA certificate that you used to sign keys while manufacturing.
This has some awesome advantages -
Your thing won't be created till your device first tries to connect to AWS IoT. So no blanks on your AWS account.
You won't have to mix your AWS IoT thing creation with the manufacturing process. Which is great because you don't want to give your Chinese (most likely scenario) manufacturers direct access to your AWS account
More about this
Link below is an example of creating 50 Iot devices, and implementing a Serverless AWS IoT Backend with AWS Lambda and Amazon DynamoDB. You can vary the devices to the number you like; It is done using the earlier version of AWS Iot Platform. Have a look.
https://aws.amazon.com/blogs/compute/implementing-a-serverless-aws-iot-backend-with-aws-lambda-and-amazon-dynamodb/
Here is the code to create 50 Iot Things
var AWS = require('aws-sdk');
AWS.config.region = 'ap-northeast-1';
var crypto = require('crypto');
var endpoint = "<endpoint prefix>.iot.<region>.amazonaws.com";
var iot = new AWS.Iot();
var iotdata = new AWS.IotData({endpoint: endpoint});
var topic = "registration";
var type = "MySmartIoTDevice"
//Create 50 AWS IoT Things
for(var i = 1; i < 51; i++) {
var serialNumber = "SN-"+crypto.randomBytes(Math.ceil(12/2)).toString('hex').slice(0,15).toUpperCase();
var clientId = "ID-"+crypto.randomBytes(Math.ceil(12/2)).toString('hex').slice(0,12).toUpperCase();
var activationCode = "AC-"+crypto.randomBytes(Math.ceil(20/2)).toString('hex').slice(0,20).toUpperCase();
var thing = "myThing"+i.toString();
var thingParams = {
thingName: thing
};
iot.createThing(thingParams).on('success', function(response) {
//Thing Created!
}).on('error', function(response) {
console.log(response);
}).send();
//Checking all devices were created
iot.listThings().on('success', function(response) {
var things = response.data.things;
var myThings = [];
for(var i = 0; i < things.length; i++) {
if (things[i].thingName.includes("myThing")){
myThings[i]=things[i].thingName;
}
}
if (myThings.length == 50){
console.log("myThing1 to 50 created and registered!");
}
}).on('error', function(response) {
console.log(response);
}).send();
You can do that by using AWS IoT SDKs. You can find supported AWS IoT SDKs here. For doing that you will need proper credentials and policies which the simplest one would be AWS account credentials (Access key along with secret key from IAM or Cognito user credentials, ...).
For creating things automatically, you can use AWS IoT Device SDK for Python which is Boto3. You can find more information about it here. The following codes show the example of creating a thing in AWS IoT automatically using python and AWS credentials:
import boto3
client = boto3.client('iot')
response = client.create_thing(
thingName=[NameOfThing],
thingTypeName=[ThingType],
attributePayload={
'attributes': {
'string': 'string'
},
'merge': True|False
},
billingGroupName='string'
)

Trigger Amazon SNS message via Amazon Lambda

I have an Amazon Lambda instance and an Amazon SNS instance. The Lambda code watches for changes in our database and I would like it to make a call to Amazon SNS to send pushes to our users. For example:
When a user on one of our forums gets a new message, the Lambda code recognizes this change every time it is run (every 10 minutes) and should send a push to the user's smartphone via SNS.
I'm running into a wall when it comes to the documentation; Amazon's docs only talk about how to trigger Lambda code via SNS, but not the reverse. Does anyone have an example of how I can accomplish this?
There is nothing special about pushing SNS notifications in the context of Lambda. I would think of it as just another external service that you interact with.
What you could do is pull in the AWS SDK in your lambda code and after that use the code to make the SNS calls. You will need to inject the right credentials to be able to call the Amazon SNS API (but you probably do something similar for getting the database endpoint and credentials if you are talking to the database)
Yes, you can use AWS Lambda to achieve what you want. You also need to give proper IAM Permissions allowing your Lambda IAM Role to publish messages to you SNS Topic.
Example SNS Publish IAM Policy:
{
"Statement":[ {
"Effect":"Allow",
"Action":"sns:Publish",
"Resource":"arn:aws:sns:*:<your account id>:<your topic id>"
} ]
}
You can use the lambda below to push an SNS message to a user, but you must know what the endpoint ARN is for that user. For example, if in an Android app, when the user logs in you will have the app send a GCM (Google Cloud Messaging) token to your backend (via an API call that triggers a lambda, for example). Your backend, which is connected to GCM, can then use this token to lookup which endpoint ARN corresponds to such user and put that in the lambda below. Alternatively, you can have the app send the endpoint ARN directly to your backend, though I think it might be a bit less secure. Make sure you give IAM permissions to publish to your app via SNS. You can use the lambda below to push the message:
var AWS = require('aws-sdk');
var sns = new AWS.SNS({apiVersion: '2010-03-31'});
exports.handler = (event, context, callback) => {
console.log(JSON.stringify(event))
var payload = {
"default": "The message string.",
"GCM":"{"+
"\"notification\":{"+
"\"body\":\"PUT NOTIFICATION BODY HERE\","+
"\"title\":\"PUT NOTIFICATION TITLE HERE\""+
"}"+
"}"
};
payload = JSON.stringify(payload);
var params = {
TargetArn: 'PUT THE ENDPOINT ARN HERE',
Subject: 'foo2',
MessageStructure: 'json',
Message: payload
}
sns.publish(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
};