I'm trying to pass and then retrieve a message with attributes into AWS SQS.
Even though I can see attributes of a message through Management Console, I can't get them using boto3, always get None. Changing "AttributeNames" doesn't make a difference. Message body can be retrieved OK.
import boto3
sqs = boto3.resource('sqs', region_name = "us-west-2")
queue = sqs.get_queue_by_name(QueueName='test')
queue.send_message(MessageBody = "LastEvaluatedKey",
MessageAttributes ={
'class_number':{
"StringValue":"Value value ",
"DataType":"String"
}
}
)
messages = queue.receive_messages(
MaxNumberOfMessages=1,
AttributeNames=['All']
)
for msg in messages:
print(msg.message_attributes) # returns None
print(msg.body) # returns correct value
Attributes (system generated) and Message Attributes (user defined) are two different kinds of things provided by the back-end API.
You're looking for message attributes, but you're asking the code to fetch attributes, instead.
AttributeNames=['All']`
It seems like you need to be fetching message attributes...
MessageAttributeNames=['All']
...when calling queue.receive_messages(). Or, the specific message attribute names that you want, if known, could be used instead of 'All' (which -- had I designed this API -- would have been called '*', but I digress).
Admittedly, this is only an intuitive guess on my part based on familiarity with the underlying API, but it seems to be consistent with http://boto3.readthedocs.io/en/latest/guide/sqs.html.
Related
I have taken a Flyway lambda from this repo and I am trying to implement a custom resource that invokes the lambda whenever the migration source folder changes. It works fine but occasionally the response object exceeds the 4096 byte limit and the stack fails to deploy. Below is my custom resource code:
flyway_resource = cr.AwsCustomResource(
self,
f"FlywayCustomResource-{construct_id}",
log_retention=RetentionDays.ONE_WEEK,
on_update=cr.AwsSdkCall(
service="Lambda",
action="invoke",
physical_resource_id=cr.PhysicalResourceId.of("FlywayTrigger"),
parameters={
"FunctionName": flyway_lambda_function.function_name,
"InvocationType": "RequestResponse",
"Payload": "{" +
"\"flywayRequest\":{\"flywayMethod\": \"migrate\"}," +
" \"assetHash\": \"" +
# The hash of this asset (i.e., the local migration folder) changes when the content
# of the folder changes and hence an UPDATED life cycle message is produced.
# (i.e., the params of the function have changed)
assets.Asset(self, "hashId", path=migrations_source).asset_hash +
"\"}"
},
# This would avoid lambda telling output size too big when erroring, we must use the right selector(s)
# output_paths=['info', 'errorMessage', 'errorType']
),
policy=cr.AwsCustomResourcePolicy.from_statements([
iam.PolicyStatement(
actions=["lambda:InvokeFunction"],
resources=[flyway_lambda_function.function_arn]
)
])
)
I know that I can use the output_paths parameter to limit the fields in the response that will be actually selected, but I am not quite sure I have the correct field names in the list. Here's a successful example response of the lambda:
For failure I have names such as 'errorMessage', 'errorType' in the root level (the same level that 'info' exists). I have two questions:
Is the commented output_paths parameter correct? I want to verify that the values of this parameter must match the keys in the json response of the lambda. I have a hard time understanding this in the docs. (If I uncomment it works but it might be because the the specified paths don't exist and the response becomes empty and hence does not exceed the limit anymore)
Assuming that 1) is correct how do I verify the response of the custom resource. In the case that the response in erroneous I would like for the deployment to fail. I know that I can do cr.get_response_field so with cr.get_response_field('info') or cr.get_response_field('errorMessage') perhaps I could check which of these keys exists in another lambda backed custom resource. Is there a better way?
I am setting up subscriptions to a topic that a number of different apps publish to. I want to filter the subscription of one particular app so that it doesn't receive back messages that it sends.
I tried adding an attribute with a value identifying the app, eg x-forwarded-by=myapp and set the filter policy as:
{
"x-forwarded-by": [{"anything-but": ["myapp"]}]
}
That correctly prevented the app receiving these however it also didn't receive any messages where there was no x-forwarded-by attribute, nor did it receive any messages whether there were either no attributes or there was an empty attribute object, eg, I would have expected to see messages with:
"MessageAttributes": null
"MessageAttributes": {}
"MessageAttributes": {"someOtherAttribute":{"DataType":"string", "StringValue": "SomeValue"}}
I also tried using an exists filter so that if the app set that attribute, it could filter it out later, eg:
{
"x-forwarded-by-my-app": [{"exists":false}]
}
This only worked if there was at least one other attribute set, and this would require me updating every app that publishes to include a dummy attribute.
I tried combining them, as in:
{
"x-forwarded-by": [
{"exists":false},
{"anything-but":["myapp"]}
}
And this worked slightly better as it handled the case of the x-forwarded-by header either not being present or having a different value. It doesn't however handle the case of no attributes being set.
Is there a way to achieve this with a filtered subscription or do I have to just have the app receive these notifications and have the app check for and ignore them?
It seems there is some issue with "anything-but", I tried this with { {"exists":false},"anything-but":["myapp", ""]} and it works.
In my case I went ahead with: {{"exists":false},"positiveCase"}.
I am using a ServiceBusTrigger to execute code when receiving a message. I would like to use the Singleton attribute to limit which messages can be executed in parallel. This attribute allows specifying a scope bound to properties on the incoming message, such that messages with different values can be executed in parallel but ones with the same value must be done serially.
This works when using top level properties on the incoming message object like CorrelationId.
Example
[Singleton("{CorrelationId}", SingletonScope.Function, Mode = SingletonMode.Function)]
public async Task HandleMessage(
[ServiceBusTrigger("my-topic-name", "my-subscription-name"), ServiceBusAccount("my-account-name")]
Message message,
CancellationToken cancellationToken
)
{
await Task.Yield();
}
What I am struggling to figure out is how to achieve the same behavior with user properties on the message. These are stored in the UserProperties dictionary on the Message object. I'm not seeing a way to refer to these with the binding statement in the Singleton attribute, but it seems like this would be a very common use case when combining Singleton with ServiceBusTrigger
The Service Bus Bindings exposes Message Metadata in binding expressions. So, userProperties.<key> should do the trick.
I want to be able to specify a custom list of valid options for a slot that LEX will either attempt to approximate towards or, in the event that no valid option can be approximated, reject the invalid response.
At first I attempted to do this through custom slot types. And though their examples may lead you to believe these are enumerations, they are not. A user still has the capacity to input whatever value they like.
Their documentation has this to say: https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/migrating-to-the-improved-built-in-and-custom-slot-types#literal
A custom slot type is not the equivalent of an enumeration. Values outside the list may still be returned if recognized by the spoken language understanding system. Although input to a custom slot type is weighted towards the values in the list, it is not constrained to just the items on the list. Your code still needs to include validation and error checking when using slot values.
I am aware that I can validate their submission through a lambda after they have completed their full submission, but by then it's too late. A user has submitted their full intent message. I'm unable to capture it midway and correct them.
Am I missing some way to input slot options or a configuration option for custom slot types? Is there any way to enforce a custom list of options for a slot? (Similar to utterances for intents, or the built in slot types, which will ask the same question again if there is no match.)
Thanks!
I'm unable to capture it midway and correct them.
You can capture the error in lambda without fulfilling the intent and starting over. Here's how I validate input with Python.
If you detect a validation error in lambda, you can elicit the same slot and pass your error message. This allows you to set complex validation rules and have your bot return specific responses to the user.
def validate(input):
if input not in ['foo', 'bar']:
return elicit_slot("Your response must be foo or bar")
def elicit_slot(error_message):
return {
'dialogAction': {
'type': 'ElicitSlot',
'intentName': current_intent,
'slots': current_slots,
'slotToElicit': slot_with_validation_error,
'message': {'contentType': 'PlainText', 'content': error_message }
}
}
I am using Sentry (in a django project), and I'd like to know how I can get the errors to aggregate properly. I am logging certain user actions as errors, so there is no underlying system exception, and am using the culprit attribute to set a friendly error name. The message is templated, and contains a common message ("User 'x' was unable to perform action because 'y'"), but is never exactly the same (different users, different conditions).
Sentry clearly uses some set of attributes under the hood to determine whether to aggregate errors as the same exception, but despite having looked through the code, I can't work out how.
Can anyone short-cut my having to dig further into the code and tell me what properties I need to set in order to manage aggregation as I would like?
[UPDATE 1: event grouping]
This line appears in sentry.models.Group:
class Group(MessageBase):
"""
Aggregated message which summarizes a set of Events.
"""
...
class Meta:
unique_together = (('project', 'logger', 'culprit', 'checksum'),)
...
Which makes sense - project, logger and culprit I am setting at the moment - the problem is checksum. I will investigate further, however 'checksum' suggests that binary equivalence, which is never going to work - it must be possible to group instances of the same exception, with differenct attributes?
[UPDATE 2: event checksums]
The event checksum comes from the sentry.manager.get_checksum_from_event method:
def get_checksum_from_event(event):
for interface in event.interfaces.itervalues():
result = interface.get_hash()
if result:
hash = hashlib.md5()
for r in result:
hash.update(to_string(r))
return hash.hexdigest()
return hashlib.md5(to_string(event.message)).hexdigest()
Next stop - where do the event interfaces come from?
[UPDATE 3: event interfaces]
I have worked out that interfaces refer to the standard mechanism for describing data passed into sentry events, and that I am using the standard sentry.interfaces.Message and sentry.interfaces.User interfaces.
Both of these will contain different data depending on the exception instance - and so a checksum will never match. Is there any way that I can exclude these from the checksum calculation? (Or at least the User interface value, as that has to be different - the Message interface value I could standardise.)
[UPDATE 4: solution]
Here are the two get_hash functions for the Message and User interfaces respectively:
# sentry.interfaces.Message
def get_hash(self):
return [self.message]
# sentry.interfaces.User
def get_hash(self):
return []
Looking at these two, only the Message.get_hash interface will return a value that is picked up by the get_checksum_for_event method, and so this is the one that will be returned (hashed etc.) The net effect of this is that the the checksum is evaluated on the message alone - which in theory means that I can standardise the message and keep the user definition unique.
I've answered my own question here, but hopefully my investigation is of use to others having the same problem. (As an aside, I've also submitted a pull request against the Sentry documentation as part of this ;-))
(Note to anyone using / extending Sentry with custom interfaces - if you want to avoid your interface being use to group exceptions, return an empty list.)
See my final update in the question itself. Events are aggregated on a combination of 'project', 'logger', 'culprit' and 'checksum' properties. The first three of these are relatively easy to control - the fourth, 'checksum' is a function of the type of data sent as part of the event.
Sentry uses the concept of 'interfaces' to control the structure of data passed in, and each interface comes with an implementation of get_hash, which is used to return a hash value for the data passed in. Sentry comes with a number of standard interfaces ('Message', 'User', 'HTTP', 'Stacktrace', 'Query', 'Exception'), and these each have their own implemenation of get_hash. The default (inherited from the Interface base class) is a empty list, which would not affect the checksum.
In the absence of any valid interfaces, the event message itself is hashed and returned as the checksum, meaning that the message would need to be unique for the event to be grouped.
I've had a common problem with Exceptions. Currently our system is capturing only exceptions and I was confused why some of these where merged into a single error, others are not.
With your information above I extraced the "get_hash" methods and tried to find the differences "raising" my errors. What I found out is that the grouped errors all came from a self written Exception type that has an empty Exception.message value.
get_hash output:
[<class 'StorageException'>, StorageException()]
and the multiple errors came from an exception class that has a filled message value (jinja template engine)
[<class 'jinja2.exceptions.UndefinedError'>, UndefinedError('dict object has no attribute LISTza_*XYZ*',)]
Different exception messages trigger different reports, in my case the merge was caused due to the lack of the Exception.message value.
Implementation:
class StorageException(Exception):
def __init__(self, value):
Exception.__init__(self)
self.value = value