I am using golang and want to store some data in Amazon DynamoDB in set of strings format SS. It is possible to store data but when I want to pull it back on unmarshaling stage it unmarshal every property except of prop that contain set of strings. Structure that I use to unmarshal in has []string type of that property. I saw that dynamodbattribute have some specific methods for unmarshaling like UnmarshalMap and so one but I didn't find specific method for SS. Can someone give me an advise?
Currently I have created custom unmarshaler following that example:
https://docs.aws.amazon.com/sdk-for-go/api/service/dynamodb/dynamodbattribute/#UnixTime.UnmarshalDynamoDBAttributeValue
but I still think that I am doin smth wrong because it should be easier way to parse set of strings...
Related
I'm storing a list of strings using Array datatype in Datastore(e.g. ["name1", "name2", ...]). As the list grows, I find myself unable to upsert the entry.
INVALID_ARGUMENT: Too many indexed properties
According to https://cloud.google.com/datastore/docs/concepts/entities#array, even if I set the property to be exclude_from_indexes, it gets ignored. The datastore web UI also doesn't have an Index checkmark for me to uncheck.
So the only option I came up with is to convert the Array into a String type and parse to a JSON Object every time I read from DB, and write back stringified.
Was wondering if this is the right approach or if there are better ways to do this I'm not aware of.
Thanks
You should set the exclude_from_indexes on each value of the array. That is what "For a property to be unindexed, the exclude_from_indexes field of Value must be set to true." means.
"order (S)","method (NULL)","time (L)"
"/1553695740/Bar","true","[ { ""N"" : ""1556593200"" }, { ""N"" : ""1556859600"" }]"
"/1556439461/adasd","true","[ { ""N"" : ""1556593200"" }, { ""N"" : ""1556679600"" }]"
"/1556516482/Foobar","cheque","[ { ""N"" : ""1556766000"" }]"
How do I scan or query for that matter on empty "method" attribute values? https://s.natalian.org/2019-04-29/null.mp4
Unfortunately the DynamoDB console offers a simple GUI and assumes the operations you want to perform all have the same type. When you select filters on columns of type "NULL", it only allows you to do exists or not exists. This makes sense since a column containing only NULL datatypes can either exist or not exist.
What you have here is a column that contains multiple datatypes (since NULL is a different datatype than String). There are many ways to filter what you want here but I don't believe they are available to you on the console. Here is an example on how you could filter the dataset via the AWS CLI (note: since your column is a named a reserved word method, you will need to alias it with an expression attribute name):
Using Filter expressions
$ aws dynamodb scan --table-name plocal --filter-expression '#M = :null' --expression-attribute-values '{":null":{"NULL":true}}' --expression-attribute-names '{"#M":"method"}'
An option to consider to avoid this would be to update your logic to write some of sort filler string value instead of a null or empty string when writing your data to the database (i.e. "None" or "N/A"). Then you could solely operate on Strings and search on that value instead.
DynamoDB currently does not allow String values of an empty string and will give you errors if you try and put those items directly. To make this "easier", many of the SDKs have provided mappers/converters for objects to DyanmoDB items and this usually involves converting empty strings to Null types as a way of working around the rule of no empty strings.
If you need to differentiate between null and "", you will need to write some custom logic to marshall/unmarshall empty strings to a unique string value (i.e. "__EMPTY_STRING") when they are stored in DyanmoDB.
I'm pretty sure that there is no way to filter using the console. But I'm guessing that what you really want is to use such a filter in code.
DynamoDB has a very peculiar way of storing NULLs. There is a "NULL" data type which basically represents the concept of null values but it really is sort of like a boolean.
If you have the opportunity to change the data type of that attribute to be a string, or numeric, I strongly recommend doing so. Then you'll be able to create much more powerful queries with filter conditions to match what you want.
If the data already exists and you don't have a significant number of items that need to be updated, I recommend creating a new attribute to represent your data and backfilling.
Just following up on the comments. If you prefer using the mapper, you can customize how it marshals certain attributes that may be null/empty. Have a look at the go sdk encoder implementation for some examples: https://git.codingcafe.org/Mirrors/aws/aws-sdk-go/blob/9b5aaeba7a51edcf3f87bda525a08b04b90d2ef8/service/dynamodb/dynamodbattribute/encode.go
I was able to do this inside a FilterExpression:
attribute_type(MyProperty, :nullType) - Where :nullType is a string with value NULL. This one finds null entries.
attribute_type(MyProperty, :stringType) - Where :stringType is a string with value S. This one finds non-null entries.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.OperatorsAndFunctions.html#Expressions.OperatorsAndFunctions.Syntax
I have been trying to insert key value pairs in database using leveldb and it works fine with simple strings. However if I want to store multiple attributes for a key or for example use JSON encoding , how can it be done in c++ . In Node.js leveldb package it can be done by specifying the encoding . Really can't figure this out
JSON is just a string, so I'm not completely sure where you're coming from here…
If you have some sort of in-memory representation of JSON you'll need to serialize it before writing it to the database and parse it when reading.
I'm looking for the hash equivalent of this question: How to pass array query params to AWS API Gateway?
Basically, I want to set up query parameters that look like this:
example.com?color[background]=yellow&color[foreground]=black
When I try to create a query parameter called color[background] in the API Gateway console, I get the following error message:
Invalid mapping expression specified: Validation Result: warnings : [], errors : [Parameter name should be a non-empty alphanumeric string]
I've also tried setting up a color query param and then passing various "hashes" to it. Here's what I've tried passing into this parameter:
{"background" => 123, "foreground" => "abc"} and removing the spaces
{"background" : 123, "foreground" : "abc"} and removing the spaces
{background:123,foreground:abc}
They all result in a request that is some form of example.com?color=%7Bbackground:123,foreground:abc%7D with the hash that I pass coming after the =.
Any ideas? Is this bad practice for query string parameters anyways, and should I stick with simple params?
Since there isn't a standard defined to pass in complex data structures like arrays or maps via the query string, API Gateway does not attempt to to interpret or parse the query string as anything other than simple key-value string pairs.
If you want to pass in and transform complex types it's best to do so in the body of a POST or PUT request where you can leverage JSON and API Gateway's powerful body mapping templates feature.
Alternatively, if you must stick with query string parameters, then you must either:
Collapse your data structure to be simple key value pairs as suggested by Michael -sqlbot above, or
Passthrough the raw query string to your backend lambda or http integration where it can parsed as you please. See this post for more details on how to do that.
I have a SearchDomain on AWS CloudSearch. I know all the defined facets names.
I would like to build a web query form to use it, but I want to add my categories values (facets) on the side, like it is done on Amazon webstore
The only way I have to get facets values is to make a query (params query) and in the answer will contain facets linked to my query results.
Is there a way to fetch all the facet.FIELD possible values to build the query form ?
If not (as read here), how to design a form using facets ?
You could also use the matchall keyword in a structured query. Also, since you don't need the results you can pass size=0 so you only get the facets which will reduce latency.
/2013-01-01/search?q=matchall&q.parser=structured&size=0&facet.field={sort:'count',size:100}