I'm writing to SQS using boto as following.
I got the Message object response from print.
<boto.sqs.message.Message object at 0x102dd4790>
But I'm not seeing that message inside AWS SQS website?
SQS is region specific. So using the following code to initialise sqs fixes this.
sqs = boto.sqs.connect_to_region("ap-southeast-2", aws_access_key_id=AWS_ACCESS_KEY_ID, aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
Related
We have a Lambda function that sends messages to SQS queue.
We are using boto3.
We have built a new environment and Lambda is running in a VPC on a private subnet.
The VPC end point is com.amazonaws.eu-west-2.sqs
Lambda code
sqs = boto3.resource('sqs')
# Get the queue
queue = sqs.get_queue_by_name(QueueName=QueueID)
This gives us the following error
EndpointConnectionError: Could not connect to the endpoint URL: "https://eu-west-2.queue.amazonaws.com/"
We have tried the following change
sqs = boto3.client('sqs')
# Get the queue
queue = sqs.get_queue_url(QueueName=QueueID, QueueOwnerAWSAccountId='xxxxxxxxxxxx')
We get the same error
It is a legacy endpoint issue but we do not know how to use the new endpoints in the Lambda function.
Because you're using a VPC endpoint for SQS you need to override the address that boto3 is using by default.
Something like this:
sqs = boto3.resource('sqs', endpoint_url="https://com.amazonaws.eu-west-2.sqs")
I have a CloudFormation template with an Output section. I would like to have the output sent to an SNS topic, but all I found so far is either the notificationsARNs, which sends the stack events (and not the outputs)
You can use boto3 to get the outputs of cloudformation and send it to sns
cloudformation_client = session.client('cloudformation')
response = cloudformation_client.describe_stacks(
StackName=StackName,
)
outputs_list = response['Stacks'][0]['Outputs']
and then post it to sns
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sns.html
I see this error in my logs:
[INFO] 2019-10-01T14:29:58.303Z e8ad5b4e-119a-48c1-b320-1d855c4efb22 Getting SQS queue url from <some_sqs_queue>...
14:30:16
[CRITICAL] 2019-10-01T14:30:16.743Z e8ad5b4e-119a-48c1-b320-1d855c4efb22 ## Transmission Error Connect timeout on endpoint URL: "https://us-west-2.queue.amazonaws.com/"
and this is the relevant code which works locally but not up on the lambda:
sqs = boto3.client(
'sqs', # region_name="us-west-2",
aws_access_key_id=credentials.access_key,
aws_secret_access_key=credentials.secret_key,
aws_session_token=credentials.token,
config=Config(connect_timeout=6, read_timeout=10, retries={'max_attempts': 2})
)
and
try:
logger.info(f"Getting SQS queue url from {sqs_queue}...")
queue_url = sqs.get_queue_url(QueueName=sqs_queue)['QueueUrl']
# iterate over entries in batches of 10
for batch in [entries[index:index + sqs_batch_limit] for index in range(0, len(entries), sqs_batch_limit)]:
logger.info(f"Sending batch of {len(batch)} records to sqs...")
sqs.send_message_batch(
QueueUrl=queue_url,
Entries=batch
)
logger.info("Sent batch of records to sqs successful")
It doesn't even get the sqs_queue_url but it works locally. What are the possible reasons for this log Transmission Error Connect error?
Few things to try:
Check the permissions on the lambda - You should NOT be putting cli credentials into your lambda. The lambda should be configured to use an IAM Role with the correct access setup, in this case, SQS Read and Write. This is because Lambdas do not inherit permissions the same way as when you run it locally, as they dont have a ".credentials" and ".config" file under your C://users directory which when run locally are automatically picked up.
Check your queue has got the correct permissions to be accessed. You can check this by going to SQS->Pick Your queue -> Then check the permissions tab.
Check your lambda is running where it can reach that endpoint. E.g is it in a locked down subnet or VPC? - for more info on what service endpoints AWS use go here : https://docs.aws.amazon.com/general/latest/gr/rande.html#sqs_region
Hope one of these might help!
I want to run a script to publish message to my sns topic. I followed the documentation and my code is this :
import boto3
client = boto3.client('sns')
response = client.publish(
TopicArn='my topic arn',
Message='ptt message '
)
print("Response: {}".format(response))
It prints the response json, but I don't see any message in my AWS SNS console. I have a lambda function which is pretty basic as a subscriber to this sns.
The lambda code :
import json
def lambda_handler(event, context):
# TODO implement
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
How can I be sure that my message "ptt message" has been published successfully? How can I check it?
For testing of integration you can configure Delivery status logging for your SNS topic. There is a simple setting for this in GUI of SNS. This will export to CloudWatch the logs of your subscription triggers.
Secondly, your Lambda is not doing anything with the event, so your "ptt message stays in the "event" dictionary. You can for example log your event and see the structure where the actual message body comes.
import logging
def lambda_handler(event, context):
logging.info(event)
...
And simply to see that the Lambda was triggered by the subscription, you can open the CloudWatch Logs, find the LogGroup automatically created by AWS for your Lambda (has the name of the function in it) and search the Log Group. There are comfortable filters by "last X minutes".
In order to test this integration you can also publish test messages to SNS Topic from the web interface if this feels more convenient than running a script.
I'm trying to notify an SNS topic from a CloudWatch alarm that's in a different region. The reason is that I want SMS alerting, which isn't available in the region where my services are. If I enter the ARN of the subscription and save the changes in the console, I get "There was an error saving the alarm. Please try again." Trying again does not help. Using a topic in the local region does work, but that's not what I need.
Is there a way to notify a topic in a different region? If not, is there another easy way I can achieve my goal?
Didn't find any docs that explicitly say this can't be done but tried to set an SNS from us-east-1 as an action of an alarm in eu-west-1 using the CLI and I got this:
An error occurred (ValidationError) when calling the PutMetricAlarm operation: Invalid region us-east-1 specified. Only eu-west-1 is supported.
So I'll assume it's not supported.
To get the functionality you need you can use AWS Lambda. Lets say your service is in a region where SMS is not supported, I'll use eu-central-1 as an example.
Setup would go like this:
[us-east-1] Create your SNS topic that can send SMS messages, in the region where SMS is supported.
[eu-central-1 Create a lambda function that sends messages to the SNS topic from step 1 in the region where your service is.
[eu-central-1] Create an SNS topic in the region where your service is.
For the SNS topic configure subscription with AWS Lambda protocol and point it to lambda from step 2.
[eu-central-1] Create your alarm in the region where your service is and put the SNS topic from step 3 as an action.
To add to #Tartaglia's answer, here's the source of such a lambda function using Python 3, cobbled together from various sources because I don't have time to do it properly:
import json
import logging
import boto3
logger = logging.getLogger()
logger.setLevel(logging.INFO)
session = boto3.Session(region_name='eu-west-1') # EU (Ireland)
sns_client = session.client('sns')
def lambda_handler(event, context):
logger.info('Received event: %s', event)
for record in event['Records']:
sns_message = record['Sns']
response = sns_client.publish(
TopicArn='YOUR TOPIC ARN HERE',
Subject=sns_message.get('Subject', None),
Message=sns_message.get('Message', None))
logger.info('Publish response: %s', response)
return 'OK'