My Agents running on various environments/devices are going to drop periodic messages from public network.
These messages will be processed by my AWS Lambda.
The systems are asynchronous.
I am thinking of using SQS to feed the Lambda.
Just that, SQS endpoint will be open to internet.
How can I validate the messages posted on AWS SQS.
Most of the devices/agents pushing messages will be on customer VPN. So, establishing a private-vpn-link is a possible solution.
You shoudn't make an SQS queue public so that anyone without AWS credentials could use it. Its not a good practice.
A better option is to uses API gateway in front of your SQS queue:
Creating an AWS Service Proxy for Amazon SQS
This way you can make your API gateway endpoint public, control its throughput, limits, throttling, access using API keys, and more.
The API gateway would be integrated with your SQS queue which would allow you to trigger your lambda function.
With the use of API keys or lambda authorizes you will be able to control access of your devices/agents to the API gateway, and subsequently, to the SQS.
The Amazon SQS endpoint is on the Internet not in a VPC. There is no benefit to using a VPN connection, unless you wish to create a policy that only allows messages to be sent to the queue if they come via the VPN connection.
Your use-case might be better-suited for AWS IoT rather than Amazon SQS:
It is designed to handle large quantities of data some from various devices, with in-built rules for handling the messages.
If you're looking to minimise your messages traversing the public internet space the you have a few options that are available to you.
Firstly, as you mention you have devices that can connect via a VPN, if this is possible you can create a VPC endpoint for SQS using a custom domain name.
In your application by referring to the custom domain name or a private IP address from one of the ENIs it might be possible to actually interact through the VPC privately.
You could enhance your security checking by adding a proxy layer in front to validate the input to perform any checks against the content. Your application would pass to this validation layer, and then add to the queue if it passes.
One thing worth pointing out is even with VPN the traffic is traversing the internet, however it is encrypted in transit. The best line of security is combining HTTPS alongside private credentials (preferably as an IAM role) to reduce the chance of someone being able to add improper data to your queue.
Related
Trying to turn a containerized Cube.js deployment into a serverless one. I’m getting Continue wait on every API call, and I notice the cubejsProcess lambda is never invoked. I checked SNS reachability by (successfully) publishing a message to it using the AWS SDK in the cubejs lambda. The cubejsProcess does get invoked with this inserted message. What might be preventing cubejs from publishing messages to SNS?
This is similar to this question, however, connectivity to SNS has been verified in this case.
I'd check IAM permissions to see if the cubejs Lambda's role. Does the role allow making requests to SNS?
Another thing that could be causing the issue is if the cubejs Lambda is in a VPC with no Internet access; calling any AWS API endpoint would fail since they all require an Internet connection.
Say you have a bunch of on premise applications (that are your potential publishers).. can we have them (sitting in some private network), publish message on to an AWS eventBus/eventBridge, so that the target subscribers can take it forward from there... Does hybrid AWS help here?
If so, can you share some links on achieving the same
That should be very straight forward - you can use the put events endpoint of any AWS SDK for EventBridge.
You'll need an IAM user with which to call the API via access key/secret key.
From a networking perspective, you'll need to be able to hit the AWS API endpoints.
Here's the method to use in Python.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/events.html#EventBridge.Client.put_events
I am trying to receive an update when a new EC2 security group is created. Haven't had much luck. There have been a lot of resources online however they don't deal with what I'm trying to do. All I need is an SNS notification when a new security group is created for some reason this is proving harder than it should.
Any suggestions on how I may be able to get this to work?
Thanks
The AWS Config service should serve this purpose perfectly.
It allows you to 'listen' for changes on certain (or all) types of resources and perform certain actions in response.
In your situation you could set up listeners for EC2:SecurityGroup events and have these events trigger notifications being sent to an SNS endpoint of your choice.
See some of the docs on streaming AWS Config events to SNS here.
Three are two services of interest:
Amazon CloudWatch Events can use rules to trigger an action when something happens (eg a Security Group is created)
Amazon Config keeps track of historical configurations
For your use-case, I would recommend an Amazon CloudWatch Events Rule:
Service Name: EC2
Event Type: AWS API Call via CloudTrail
Specific Operation: CreateSecurityGroup
Add Target: Specify an SNS Topic to receive the notification
Certificate Pinning on AWS Lambda
Can handler functions in AWS lambda, invoked by API Gateway, reach down into the connection layer of the request at all to access what certificate was used to establish the SSL/TLS connection on the client side? I'd like to implement a certificate pinning strategy for the API I'm building as I've found a dozen ways in which SSL without pinning can be bypassed and the data in my API is not social media posts.
Any Alternatives if not possible?
If Lambda functions don't have access to the connection layer is there a comparable way of further enforcing security and not simply saying "SSL is enough"
Last Resort
If Lambda doesn't give me access to the certificate info of the client connections I'll have to resort to asymmetric encryption.
An AWS Lambda function doesn't have a direct connection with an HTTP request, it can be invoked by a lot of things, e.g. an SNS subscription, or a simple SDK invocation. Or, and that's I guess what you're aiming at, you can invoke a Lambda function via the AWS API Gateway.
So if you want any HTTP or TLS details in your Lambda function, you need to make sure that the API Gateway is somehow injecting that data into the payload that's going to the Lambda function. I'm not sure if that's possible, a deep dive into some API Gateway documentation will probably help you further.
I'm relatively new to AWS, but I am trying to figure out how to get AWS to receive emails. According this post How to configure email accounts like support#xyz.com or feedback#xyz.com on AWS SES only handles outbound email.
What I am hoping to achieve is the ability to filter aliases. For example, if the alias is "xyz12alias", then any email sent to "xyz12alias#mydomain.co", can see the email and process the content appropriately. Which in my case will be storing it in account associated with the filter.
Can anybody direct me to a strategy or service within AWS that would allow me to implement inbound email on Amazon AWS?
https://postmarkapp.com/inbound appears to give me what I want, but is there anything within the AWS framework itself? Are there alternate services to postmarkapp?
Thanks.
Amazon Simple Email Service just introduced incoming e-mail support:
https://aws.amazon.com/about-aws/whats-new/2015/09/amazon-ses-now-supports-inbound-email/
In addition to offering a scalable, cost-effective email-sending
platform, Amazon SES can now accept your incoming emails. You can
configure Amazon SES to deliver your messages to an Amazon S3 bucket,
call your custom code via an AWS Lambda function, or publish
notifications to Amazon SNS. You can also configure Amazon SES to drop
or bounce messages you do not want to receive. If you choose to store
your messages in Amazon S3, Amazon SES can encrypt your mail using AWS
Key Management Service (KMS) before writing it to the bucket.
You configure all of these actions by defining receipt rules, which
you set up by using the Amazon SES console or the Amazon SES API.
Receipt rules enable a single message to trigger multiple actions.
Your rules can be as broad or as specific as you choose because you
can configure them to apply to specific email addresses or entire
domains.
You can also use receipt rules to control which messages Amazon SES
can accept on your behalf. Another filtering method is to set up
custom IP address block lists and allow lists. If you know that you
don’t want to receive mail originating from a particular IP address
range, simply add it to your account's IP address block list. You can
also override block lists by adding IP address ranges to your allow
list, which provides fine-grained control over your inbound email
traffic.
You'd have to set up your own server; that's the way to handle it using AWS. They don't provide anything other than their bulk email delivery service. A few links below:
http://jeffreifman.com/how-to-install-your-own-private-e-mail-server-in-the-amazon-cloud-aws/
http://cerebellumstrategies.com/2012/04/15/amazon-linux-postfix-dovecot/
Update: there is now a solution available in AWS, as referenced in the comments below.
Still doesn't appear to be possible on SES. I'd recommend looking at Mandrill and Sendgrid though.
http://mandrill.com/features/
https://sendgrid.com/docs/API_Reference/Webhooks/parse.html
Here is how to use Amazon and any virtual server to deliver email from SES to a local IMAP account.
This plan is about stable operations: every step is under our control.
Have SES receive emails
And deliver to an S3 bucket and send notification to a SNS topic
Subscribe to that notification with HTTPS protocol
Use aws/aws-php-sns-message-validator and write a small PHP script to reveive notifications from SNS through HTTPS
SNS only sends messaged ID-s, put those in a file
Install incron on your server to start a shell script that downloads those messages from the S3 bucket by s3cmd and delivers to the local IMAP account by sendmail
Use any IMAP server, I use Courier IMAP
Read your messages with e.g. Rainloop webmail
Use SES also for sending outgoing emails
When using AWS SES inbound mail support, main challenge is that, it fits well for programmatic tasks, but hardly human readable since it receives in raw message delivered format.
You you can setup a S3 bucket, SNS Topic (Which could again links to Email, SMS & etc.), Lambda to forward the inbound mail with the same format it receives to take any programmatic actions based on the content of the email.
If you want read the message in human readable format, you need to write your own code to do the formatting. For example using Serverless lambda SES forwarder includes, NodeJS code runs in Lambda to convert raw email to human readable format and forward to recipient email.