I am working on Amazon SES with SQS to receive the bounce list of the email. For security reason, I am only given the information that necessary to connect to the SES and SQS service (host name, API keys, etc), so I am not able to use the AWS console to see the status of the queue. This is reasonable as I don't want to mess with many other services that are under the same account - especially when the services are not free. However, as the job is added to SQS by SES, I would need a way to see what's in SQS, so as to know if the bug is because the job is not inside SQS or simply because my code failed to retrieve the job.
So, are there tools that I can view the SQS status when I don't have access to AWS console?
Yes, you can use the AWS CLI (https://aws.amazon.com/cli/) to view basic information about the queue:
For example:
aws sqs get-queue-attributes --queue-url https://sqs.us-east-1.amazonaws.com/99999999/HBDService-BackgroundTaskQueue --attribute-names All
will show you this:
{
"Attributes": {
"LastModifiedTimestamp": "1522235654",
"ApproximateNumberOfMessages": "7",
"ReceiveMessageWaitTimeSeconds": "20",
"CreatedTimestamp": "1522235629",
"ApproximateNumberOfMessagesDelayed": "0",
"QueueArn": "arn:aws:sqs:us-east-1:999999999:HBDService-BackgroundTaskQueue",
"RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:999999999:HBDService-BackgroundTaskQueue-DLQ\",\"maxReceiveCount\":100}",
"MaximumMessageSize": "262144",
"DelaySeconds": "0",
"ApproximateNumberOfMessagesNotVisible": "0",
"MessageRetentionPeriod": "1209600",
"VisibilityTimeout": "180"
}
}
Related
We have a GCS bucket named 'testfiles' and Pub/Sub topic 'testtopic' with a subscription 'testsubscription'. We have created a notification configuration to receive notifications on the Pub/Sub topic for any event happening on the GCS bucket. When we run the following command to list the notifications on the bucket,
gcloud storage buckets notifications list gs://testfiles
we see the below output,
{
"kind": "storage#notification",
"selfLink": "https://www.googleapis.com/storage/v1/b/testfiles/notificationConfigs/28",
"id": "28",
"topic": "//pubsub.googleapis.com/projects/test-project/topics/testtopic",
"etag": "28",
"payload_format": "JSON_API_V1"
}
Also, we have provided the Cloud Storage Service account the Pub/Sub Publisher role.
Despite these settings, when we upload a file to the bucket 'testfiles', we do not see any JSON messages in the above mentioned topic/subscription (testtopic/testsubscription).
We tried to follow the documentation here
Please advise, if there is something we are missing.
The console (GCP console, console.cloud.google.com) was not displaying the Pub/Sub messages. However, when we connect to Pub/Sub topic's subscription through a listener code, it was able to receive the JSON messages without any issues.
S3 can be configured to invoke lambda when an object arrives in it.
Is it possible to invoke a REST API (endpoint of a microservice running in EKS) when an object arrives in S3?
From November 2021 is possible to integrate S3 with Amazon EventBridge.
So you can create an EventBridge rule which is triggered on bucket object creation and has API destination as a target.
For this, the option Bucket Properties -> Amazon EventBridge -> "Send notifications to Amazon EventBridge for all events in this bucket" should be enabled.
Then, on EventBridge create the rule with an event pattern like this:
{
"source": ["aws.s3"],
"detail-type": ["Object Created"],
"detail": {
"bucket": {
"name": ["<my-bucket-name>"]
}
}
}
And configure the target as API destination endpoint (configure http method, endpoint, authorization).
You can set up an SNS topic as a target for the event from S3. In the SNS topic, you can add an HTTP/s subscriber, which can be your API endpoint.
Have the Lambda hit the REST API for you.
I set an SNS notification to send me an email whenever there is a change regarding the IAM policies. When a change occurs, CloudTrail sends a Log to CloudWatch which triggers an alarm attached to an SNS topic. More details in this link.
Here is an example of what I get by mail:
Alarm Details:
- Name: PolicyAlarm
- Description: This alarm is to monitor IAM Changes
- State Change: INSUFFICIENT_DATA -> ALARM
- Reason for State Change: Threshold Crossed: 1 datapoint [1.0 (31/08/17 09:15:00)] was greater than or equal to the threshold (1.0).
- Timestamp: Thursday 31 August, 2017 09:20:39 UTC
- AWS Account: 00011100000
Threshold:
- The alarm is in the ALARM state when the metric is GreaterThanOrEqualToThreshold 1.0 for 300 seconds.
The only relevant information here is the AWS Account ID. Is there a way to also include the change? Who made it, when and where? Or maybe send little information from the cloudwatch log like the "eventName" ?
There are two ways to trigger notifications from an AWS CloudTrail:
Configure Amazon CloudWatch Logs to look for specific strings. When found, it increments a metric. Then, create an alarm that triggers when the metric exceeds a particular value over a particular period of time. When the notification is sent, only information about the alarm is sent. OR...
Create a rule in Amazon CloudWatch Events to look for the event. Set an Amazon SNS topic as the target. When the notification is sent, full details of the event are passed through.
You should use # 2, since it provides full details of the event.
Here's what I did to test:
Created an Amazon SQS queue in us-east-1 (where all IAM events take place)
Created an Amazon CloudWatch Events rule in us-east-1 with:
Service Name: IAM
Event Type: AWS API Call via CloudTrail
Specific Operations: PutUserPolicy
Edited an IAM policy
Within a short time, the event appeared in SQS:
Here's the relevant bits of the policy that came through:
{
"detail-type": "AWS API Call via CloudTrail",
"source": "aws.iam",
"region": "us-east-1",
"detail": {
"eventSource": "iam.amazonaws.com",
"eventName": "PutUserPolicy",
"awsRegion": "us-east-1",
"requestParameters": {
"policyDocument": "{\n \"Version\": \"2012-10-17\",\n ... }",
"policyName": "my-policy",
"userName": "my-user"
},
"eventType": "AwsApiCall"
}
}
I sent the message to SQS, but you could also send it to SNS to then forward via email.
I set up my Lambda function according to the AWS guides by setting a trigger in the setup stage. (the guide except that the guide is using IoT button and I'm using a rule)
It sets up the trigger rule in the AWS IoT console for me. The thing is setup with a certificate and an "iot:*" policy which gives it full IoT access.
The thing is continuously sending messages to the cloud under a certain topic. The messages can be received if I subscribe to it in the AWS IoT Test console.
My lambda function gets triggered if I publish something under that topic from the AWS IoT Test console.
But the function doesn't trigger from the continuous messages sent by the thing. It only triggers from the IoT Test console.
I didn't add any other policy under certificates for the thing in relation to this trigger. Do I have to do so? What should it be?
I tried changing my topic SQL to SELECT * FROM '*'
Try to change your SQL to SELECT * FROM '#'. With # you get every published topic. When you use *, then you don't get topics e.g. sample/newTopic.
With this SQL statement the Lambdas Function gets invoked for every incoming message. When the AWS IoT Console shows the message and your Lambda Function doesn't do anything, try to look if Lambda did a log in CloudWatch.
If your AWS IoT Thing can't trigger AWS Lambda function, you may have a JSON mapping issue and also to improve your SQL query. In my case, I used the following code to provide Lambda a clean input:
SELECT message.reported.* from "#"
With JSON mapping:
{
"desired": {
"light": "green",
"Temperature": "55",
"timestamp": 1526323886
},
"reported": {
"light": "blue",
"Temperature": "55",
"timestamp": 1526323886
},
"delta": {
"light": "green"
}
}
Then you analyze CloudWatch logs:
Then, check your AWS IoT Console for shadow updates (green below - "Atualizações de sombra") and also Publications (orange)
So, your solution will look like this:
For full details of an end-to-end implementation of AWS IoT using Lambda, please access:
IoT Project - CPU Temperature from Ubuntu to AWS IoT
I have an Amazon Lambda instance and an Amazon SNS instance. The Lambda code watches for changes in our database and I would like it to make a call to Amazon SNS to send pushes to our users. For example:
When a user on one of our forums gets a new message, the Lambda code recognizes this change every time it is run (every 10 minutes) and should send a push to the user's smartphone via SNS.
I'm running into a wall when it comes to the documentation; Amazon's docs only talk about how to trigger Lambda code via SNS, but not the reverse. Does anyone have an example of how I can accomplish this?
There is nothing special about pushing SNS notifications in the context of Lambda. I would think of it as just another external service that you interact with.
What you could do is pull in the AWS SDK in your lambda code and after that use the code to make the SNS calls. You will need to inject the right credentials to be able to call the Amazon SNS API (but you probably do something similar for getting the database endpoint and credentials if you are talking to the database)
Yes, you can use AWS Lambda to achieve what you want. You also need to give proper IAM Permissions allowing your Lambda IAM Role to publish messages to you SNS Topic.
Example SNS Publish IAM Policy:
{
"Statement":[ {
"Effect":"Allow",
"Action":"sns:Publish",
"Resource":"arn:aws:sns:*:<your account id>:<your topic id>"
} ]
}
You can use the lambda below to push an SNS message to a user, but you must know what the endpoint ARN is for that user. For example, if in an Android app, when the user logs in you will have the app send a GCM (Google Cloud Messaging) token to your backend (via an API call that triggers a lambda, for example). Your backend, which is connected to GCM, can then use this token to lookup which endpoint ARN corresponds to such user and put that in the lambda below. Alternatively, you can have the app send the endpoint ARN directly to your backend, though I think it might be a bit less secure. Make sure you give IAM permissions to publish to your app via SNS. You can use the lambda below to push the message:
var AWS = require('aws-sdk');
var sns = new AWS.SNS({apiVersion: '2010-03-31'});
exports.handler = (event, context, callback) => {
console.log(JSON.stringify(event))
var payload = {
"default": "The message string.",
"GCM":"{"+
"\"notification\":{"+
"\"body\":\"PUT NOTIFICATION BODY HERE\","+
"\"title\":\"PUT NOTIFICATION TITLE HERE\""+
"}"+
"}"
};
payload = JSON.stringify(payload);
var params = {
TargetArn: 'PUT THE ENDPOINT ARN HERE',
Subject: 'foo2',
MessageStructure: 'json',
Message: payload
}
sns.publish(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
};