I am trying to do a simple dynamoDB scan with a filter expression (documentation here)
This is my expression string:
"attribute_exists("my_db_key") AND ("my_db_key" = 1)"
This simply states:
"If a value for my_db_key exists AND my_db_key EQUALS 1, return it in the results"
However it does not work and I get a this error:
Invalid FilterExpression: Syntax error; token: "1", near: "= 1)
I am aware that I can use an attribute value placeholder for values and then use that in the expression but I do not want to do this. And according to Amazon's documentation it is NOT required.
So how do I do this simple expression? Does anyone have an example or link to documentation? Amazon's documentation is unfortunately of no help.
NOTE: I am implementing this with AWSDynamoDBScanInput on iOS but my issue here is to do with global expression syntax so it should not matter.
Your params need to look something like this (for the Node AWS library):
params = {
"FilterExpression": 'attribute_exists("my_db_key") AND ("my_db_key" = :value)',
"ExpressionAttributeValues": {
":value": 1
},
// ...
};
docClient.scan(params, function(err, data){
// Handle err or process data
})
For some languages, the parameters should look more like this:
{
"FilterExpression": 'attribute_exists("my_db_key") AND ("my_db_key" = :value)',
"ExpressionAttributeValues": {
":value": {"N":1}
},
// ...
};
You have to use a placeholder and pass the value separately. Here's some documentation and a Post from AWS forums
Related
Background:
I am trying to add DynamoDB:GetItem step to my state machine in AWS Step Functions. GetItem API takes input in the following format:
{
"TableName": "MyDynamoDBTable",
"Key": {
"Column": {
"S": "MyEntry"
}
}
}
where "Column" is the primary key name, and "MyEntry" is the primary key value. The issue is that I want to be able to specify both primary key name and value dynamically, using JSON path reference.
Unfortunately, AWS won't allow me to pass value reference for primary key name ("Column"). So I can't do something like
{
"TableName": "MyDynamoDBTable",
"Key.$": {
"$.ColumnName": {
"S": "MyEntry"
}
}
}
Problem:
The only workaround I could think of (albeight a bit ugly) is to use combination of States.StringToJson and States.Format intrinsic functions to first generate stringified version of the input to Key.$ field, and then convert to JSON from string. Something like:
{
"TableName.$": "$.TableName",
"Key.$": "States.StringToJson(States.Format('\{\"{}\":\{\"S.$\":\"{}\"\}\}', $.PrimaryKeyName, $.PrimaryKeyValue))"
}
It should work in theory, but it seems that AWS Step Functions is not happy about escaping double quotes? It's not able to parse the definition above.
So my question is:
Is there a way to make this work? (either by escaping double quotes somehow, or through a totally different approach)
After lots of experimentation, I finally found a way to make dynamic keys work. I am using Pass step with the following parameters defined:
{
"Key.$": "States.StringToJson(States.Format('\\{\"{}\":\\{\"S\":\"{}\"\\}\\}', $.HashKeyName, $.HashKeyValue))"
}
The secret, apparently, was in using double \\ when escaping { and } symbols. Escaping " wasn't a problem after all, even though it's not documented in AWS docs.
The result of this transformation is following:
{
"Key": {
"MyHashKeyName": {
"S": "MyHashKeyValue"
}
}
}
I have a process that has a custom model, similar to the following model(get by calling http://localhost:8080/engine-rest/task/{id}/variables/):
{
"Title": {
"type": "String",
"value": "aaa",
"valueInfo": {
}
},
"247f3af4-36cf-72cc-1a95-601f07640674": {
"type": "String",
"value": "{\"Title\":\"AA\",\"Value\":\"BB\"}",
"valueInfo": {
}
}
}
I want to define a expressions at the gates. How should I do this?
I try these:
${ "247f3af4-36cf-72cc-1a95-601f07640674".Value == "AA"}
Or
${ JSON("247f3af4-36cf-72cc-1a95-601f07640674").prop("Value") == "AA"}
Or
${S(247f3af4-36cf-72cc-1a95-601f07640674).prop("Value").stringValue() == "AA"}
But get following errors:
Unknown property used in expression: ${ "247f3af4-36cf-72cc-1a95-601f07640674".Value == "AA"}. Cause: Could not find property Value in class java.lang.String
Error while evaluating expression: ${ JSON("247f3af4-36cf-72cc-1a95-601f07640674").prop("Value") == "AA"}. Cause: Error invoking function 'JSON'
ENGINE-01009 Error while parsing process. Error parsing '${S(247f3af4-36cf-72cc-1a95-601f07640674).prop("Value").stringValue() == "AA"}': syntax error at position 15, encountered 'c7', expected ')'.
What you are showing is the value of a JSON object stored in a process data, right? What is the name of the process data?
In Java you use JSON(), in the process (JavaScript) use S()
(see https://docs.camunda.org/manual/7.17/reference/spin/json/01-reading-json/)
Place S() around the name of your process data to create the object. Then you can use .prop() to navigate it. ${S(myData).prop("xyz")}.
In this example I used the method to read the JSON response of a REST call and then extract a field:
https://github.com/rob2universe/camunda-http-connector-example
You use JSON() around the name of the process data, then you can access the properties
I finally find answer
I must use something like this:
${S(a247f3af4_36cf_72cc_1a95_601f07640674).prop("Value").stringValue() == "AA"}
we must start variable name with character and do'nt use -.
I am trying to group data in multiple stages.
At the moment my query looks like this:
db.captions.aggregate([
{$project: {
"videoId": "$videoId",
"plainText": "$plainText",
"Group1": {$cond: {if: {$eq: ["plainText", {"$regex": /leave\sa\scomment/i}]},
then: "Yes", else: "No"}}}}
])
I am not sure whether it is actually possible to use the $regex operator within a $cond in the aggregation stage. I would appreciate your help very much!
Thanks in advance
UPDATE: Starting with MongoDB v4.1.11, there finally appears to be a nice solution for your problem which is documented here.
Original answer:
As I wrote in the comments above, $regex does not work inside $cond as of now. There is an open JIRA ticket for that but it's, err, well, open...
In your specific case, I would tend to suggest you solve that topic on the client side unless you're dealing with crazy amounts of input data of which you will always only return small subsets. Judging by your query it would appear like you are always going to retrieve all document just bucketed into two result groups ("Yes" and "No").
If you don't want or cannot solve that topic on the client side, then here is something that uses $facet (MongoDB >= v3.4 required) - it's neither particularly fast nor overly pretty but it might help you to get started.
db.captions.aggregate([{
$facet: { // create two stages that will be processed using the full input data set from the "captions" collection
"CallToActionYes": [{ // the first stage will...
$match: { // only contain documents...
"plainText": /leave\sa\scomment/i // that are allowed by the $regex filter (which could be extended with multiple $or expressions or changed to $in/$nin which accept regular expressions, too)
}
}, {
$addFields: { // for all matching documents...
"CallToAction": "Yes" // we create a new field called "CallsToAction" which will be set to "Yes"
}
}],
"CallToActionNo": [{ // similar as above except we're doing the inverse filter using $not
$match: {
"plainText": { $not: /leave\sa\scomment/i }
}
}, {
$addFields: {
"CallToAction": "No" // and, of course, we set the field to "No"
}
}]
}
}, {
$project: { // we got two arrays of result documents out of the previous stage
"allDocuments" : { $setUnion: [ "$CallToActionYes", "$CallToActionNo" ] } // so let's merge them into a single one called "allDocuments"
}
}, {
$unwind: "$allDocuments" // flatten the "allDocuments" result array
}, {
$replaceRoot: { // restore the original document structure by moving everything inside "allDocuments" up to the top
newRoot: "$allDocuments"
}
}, {
$project: { // include only the two relevant fields in the output (and the _id)
"videoId": 1,
"CallToAction": 1
}
}])
As always with the aggregation framework, it may help to remove individual stages from the end of the pipeline and run the partial query in order to get an understanding of what each individual stage does.
Dynamo DB table
I am trying to append a list to the "Hi" List , but coming up with invalid operator type in the list_append call, which says one of them is a Map but these still look list to me ...
Here is the API call :
response = node_table.update_item(
TableName='xyz',
Key={......},
UpdateExpression='SET #ri = list_append(:vals, #ri)',
ExpressionAttributeNames={'#ri':'Hello.Hi'},
ExpressionAttributeValues={":vals": {"L": [ { "S": "Something" }]}})
Error: Invalid UpdateExpression: Incorrect operand type for operator or function; operator or function: list_append, operand type: M
I referred DynamoDB : SET list_append not working using aws sdk, but could not figure out
Please let me know what am I missing
Looking at your table, Hi is a List of Maps. You're trying to update it with a List of Strings here:
ExpressionAttributeValues={":vals": {"L": [ { "S": "Something" }]}})
That seems to be what the error message is trying to say.
As Matt pointed out the String constraint was infact the error ,
Here is the correct version
response = node_table.update_item(
TableName='xyz',
Key={'..},
UpdateExpression='SET #ri = list_append(:vals, #ri)',
ExpressionAttributeNames={'#ri':'Hello.Hi'},
ExpressionAttributeValues={":vals": ["Something"]})
Please go through this question of mine:
MongoDB $group and explicit group formation with computed column
But this time, I need to compare strings, not numbers.The CASE query must have a LIKE:
CASE WHEN source LIKE '%Web%' THEN 'Web'
I then need to group by source. How to write this in Mongo? I am trying the following but not sure if $regex is supported inside $cond. By the way, is there a list of valid operators inside $cond somewhere? Looks like $cond isn't very fond of me :)
db.Twitter.aggregate(
{ $project: {
"_id":0,
"Source": {
$cond: [
{ $regex:['$source','/.* Android.*/'] },
'Android',
{ $cond: [
{ $eq: ['$source', 'web'] }, 'Web', 'Others'
] }
]
}
} }
);
There're many other values that I need to write in there, doing a deeper nesting. This is just an example with just 'Android' and 'Web' for the sake of brevity. I have tried both with $eq and $regex. Using $regex gives error of invalid operator whereas using $eq doesn't understand the regex expression and puts everything under 'Others'. If this is possible with regex, kindly let me know how to write it for case-insensitive match.
Thanks for any help :-)
Well, it still seems to be not even scheduled to be implemented :(
https://jira.mongodb.org/browse/SERVER-8892
I'm using 2.6 and took a peek on 3.0, but it's just not there.
There's one workaround though, if you can project your problem onto a stable substring. Then you can $substr the field and use multiple nested $cond. It's awkward, but it works.
Maybe you can try it with MapReduce.
var map = function()
{
var reg1=new RegExp("(Android)+");
var reg2=new RegExp("(web)+");
if (reg1.test(this.source)){
emit(this._id,'Android');
}
else if (reg2.test(this.source))
{
emit(this._id,'web');
}
}
var reduce = function (key,value){
var reduced = {
id:key,
source:value
}
return reduced;
}
db.Twitter.mapReduce(map,reduce,{out:'map_reduce_result'});
db.map_reduce_result.find();
You can use JavaScript regular expresions instead of MongoDB $regex.