I have inherited a DynamoDB table with three columns of which one is a key
Id - String - hashKey
ProductID - String
UsageCount - number
I want to run a query/scan that logically equates to the following
select ProductID, UsageCount where ProductID='abcd'
Although ProductID is not a key of any sort, I can use the AmazonDynamoDB console and add a filter
and get just what I want. So I assume I should be able to do the same thing with code (although everyone talks about using hash and range keys for such purposes)
I am trying to do the same thing with RUBY but not getting results...
resp = $dynamodb.scan(
:table_name => PRODUCTTABLE,
:projection_expression => "Id, TenantId, SeatCount",
:filter_expression => 'TenantId = abcd'
)
Could someone help me with the correct syntax for doing this? I tried numerous variations that all throw exceptions except for the above syntax that does not throw an exception but returns zero rows. So Im thinking it must be close :) (or maybe not)
You have to use both filter_expression but make it 'TenantId = :val'
and then pass :val with expression_attribute_names and expression_attribute_values
Even though it's in Java, this doc will help clear things out for you
Related
I am currently testing how to design a query from AWS.DynamoDB.DocumentClient query() call that takes params: DocumentClient.QueryInput, which is used for retrieving data collection from a table in DynamoDB.
Query seems to be simple and working fine while working with indexes of type String or Number only. What I am not able to make is an query, that will use a valid index and filter upon an attribute that is nested (see my data structure please).
I am using FilterExpression, where can be defined logic for filtering - and that seems to be working fine in all cases except cases when trying to do filtering on nested attribute.
Current parameters, I am feeding query with
parameters {
TableName: 'myTable',
ProjectionExpression: 'HashKey, RangeKey, Artist ,#SpecialStatus, Message, Track, Statistics'
ExpressionAttributeNames: { '#SpecialStatus': 'Status' },
IndexName: 'Artist-index',
KeyConditionExpression: 'Artist = :ArtistName',
ExpressionAttributeValues: {
':ArtistName': 'BlindGuadian',
':Track': 'Mirror Mirror'
},
FilterExpression: 'Track = :Track'
}
Data structure in DynamoDB's table:
{
'Artist' : 'Blind Guardian',
..
'Track': 'Mirror Mirror',
'Statistics' : [
{
'Sales': 42,
'WrittenBy' : 'Kursch'
}
]
}
Lets assume we want to filter out all entries from DB, by using Artist in KeyConditionExpression. We can achieve this by feeding Artist with :ArtistName. Now the question, how to retrieve records that I can filter upon WritenBy, which is nested in Statistics?
To best of my knowledge, we are not able to use any other type but String, Number or Binary for purpose of making secondary indexes. I've been experimenting with Secondary Indexes and Sorting Keys as well but without luck.
I've tried documentClient.scan(), same story. Still no luck with accessing nested attributes in List (FilterExpression just won't accept it).
I am aware of possibility to filter result on "application" side, once the records are retrieved (by Artists for instance) but I am interested to filter it out in FilterExpression
If I understand your problem correctly, you'd like to create a query that filters on the value of a complex attribute (in this case, a list of objects).
You can filter on the contents of a list by indexing into the list:
var params = {
TableName: "myTable",
FilterExpression: "Statistics[0].WrittenBy = :writtenBy",
ExpressionAttributeValues: {
":writtenBy": 'Kursch'
}
};
Of course, if you don't know the specific index, this wont really help you.
Alternatively, you could use the CONTAINS function to test if the object exists in your list. The CONTAINS function will require all the attributes in the object to match the condition. In this case, you'd need to provide Sales and WrittenBy, which probably doesn't solve your problem here.
The shape of your data is making your access pattern difficult to implement, but that is often the case with DDB. You are asking DDB to support a query of a list of objects, where the object has a specific attribute with a specific value. As you've seen, this is quote tricky to do. As you know, getting the data model to correctly support your access patterns is critical to your success with DDB. It can also be difficult to get right!
A couple of ideas that would make your access pattern easier to implement:
Move WrittenBy out of the complex attribute and put it alongside the other top-level attributes. This would allow you to use a simple FilterExpression on the WrittenBy attribute.
If the WrittenBy attribute must stay within the Statistics list, make it stand alone (e.g. [{writtenBy: Kursch}, {Sales: 42},...]). This way, you'd be able to use the CONTAINS keyword in your search.
Create a secondary index with the WrittenBy field in either the PK or SK (whichever makes sense for your data model and access patterns).
I'm using Sitecore 8 MVC .net 4.5. I have boolean field Is Sponsored and I need to order all items such as first with field Is Sponsored = true and by relevance. I found if I add order by Is_Sponsored I'm losing relevance order. So my question is: is there a way to include relevance in existing ordering?
the above method will work fine if you have only a limited amount of documents as you must read all documents from the index.
You can sort by the field SCORE to sort by the sort - so your sort order would be isSponsered, SCORE. This should work as far as I remember - but I have not validated it.
else - more to solr and you will have a number of options to solve this.
You could skip the boolean check in your query and run a regular query with the right relevance order coming back. Then filter down that result set by the boolean value into two separate collections.
var results = queryable.Where(predicate).ToList();
Then:
var sponsored = results.Where(i => i.IsSponsored);
var notSponsored = results.Where(i => !i.IsSponsored);
Then join the collections.
I am trying to get a list of users from the database that have ALL the tags in a criteria.
The User entity has a many-to-many association to a Tag entity.
The or version where just one of the tags have to match is working using the following code
$tagIds = array(29,30);
$this->createQueryBuilder('u')
->select('u','t')
->leftJoin('u.tags','t')
->where("t IN(:tagIds)")
->setParameter("tagIds",$tagIds)
;
Can anybody help me with getting it to work so ALL tag ids must match ?
Keep in mind this is a query to get a list of users, not just one user , so i guess every user must be checked to see if they match all the supplied tag ids.
I have tried a bunch of queries but not having any luck so far...
simple bruteforce query:
$tagIds = array(29,30);
$qb = $this->createQueryBuilder('u');
$qb
->select('u')
;
foreach($tagIds as $idx => $tagId)
{
$joinAlias = "t{$idx}";
$qb->leftJoin('u.tags', $joinAlias)
->addSelect($joinAlias)
->andWhere("{$joinAlias}.id = $tagId AND $joinAlias IS NOT NULL")
;
}
this is really bruteforce and costly query, you join each tag as a separate join, if you have lots of users and tags, this will take ages to execute.
since database is the bottleneck of your application, you should make a simple query to the database and then parse the data in your application, so you should use your query and then check which users have those 2 tags in their collections.
Ok... after a lot of searching this seems to work for me :
$tagIds = array(29,30);
$this->createQueryBuilder('u')
->select('u','t')
->leftJoin('u.tags','t')
->where("t IN(:tagIds)")
->groupBy('u.id')
->having('COUNT(DISTINCT t.id) = ' . count($tagIds))
->setParameter("tagIds",$tagIds)
;
I've search hard for an answer to this but haven't found anything that works. I have a NodeJS app, with the Mongoose ORM. I'm trying to query my Mongo database where a result is LIKE the query.
I have tried using a new RegExp to find the results, but it hasn't worked for me. The only time I get a result is when the query is exactly the same as the collection property's value.
Here's what I'm using right now:
var query = "Some Query String.";
var q = new RegExp('^/.*'+ query +'.*/i$');
Quote.find({author: q}, function(err, doc){
cb(doc);
});
If the value of an author property contains something LIKE the query (for instance: 'some. query String'), I need to return the results. Perhaps stripping case, and excluding special characters is all I can do? What is the best way to do this? My RegEx in this example is obviously not working. Thanks!
You likely want to create your RegExp as follows instead as you don't include the / chars when using new RegExp:
var q = new RegExp(query, 'i');
I don't know of a way to ignore periods in the author properties of your docs with a RegExp though. You may want to look at $text queries for more flexible searching like that.
I've read a lot about this and know there are many related questions on here, but I couldn't find a definitive guide for how to go about sanitizing everything. One option is to sanitize on insert, for example I have the following in my model
before_validation :sanitize_content, :on => :create
def sanitize_content
self.content = ActionController::Base.helpers.sanitize(self.content)
end
Do I need to run this on every field in every model? I'm guessing the :on => :create should be removed too so it runs when updates too?
The other option is to sanitize when data is displayed in views, using simple_format, or .html_safe or sanitize(fieldname). SHould I be sanitizing in all my views for every single field, as well as on insert? Having to do this manually everywhere doesn't seem very railsy
Thanks for any help
TL;DR
Regarding user input and queries: Make sure to always use the active record query methods (such as .where), and avoid passing parameters using string interpolation; pass them as hash parameter values, or as parameterized statements.
Regarding rendering potentially unsafe user-generated html / javascript content: As of Rails 3, html/javascript text is automatically properly escaped so that it appears as plain text on the page, rather than interpreted as html/javascript, so you don't need to explicitly sanitize (or use <%= h(potentially_unsafe_user_generated_content)%>
If I understand you correctly, you don't need to worry about sanitizing data in this manner, as long as you use the active record query methods correctly. For example:
Lets say our parameter map looks like this, as a result of a malicious user inputting the following string into the user_name field:
:user_name => "(select user_name from users limit 1)"
The bad way (don't do this):
Users.where("user_name = #{params[:id}") # string interpolation is bad here
The resulting query would look like:
SELECT `users`.* FROM `users` WHERE (user_name = (select user_name from users limit 1))
Direct string interpolation in this manner will place the literal contents of the parameter value with key :user_name into the query without sanitization. As you probably know, the malicious user's input is treated as plain 'ol SQL, and the danger is pretty clear.
The good way (Do this):
Users.where(id: params[:id]) # hash parameters
OR
Users.where("id = ?", params[:id]) # parameterized statement
The resulting query would look like:
SELECT `users`.* FROM `users` WHERE user_name = '(select user_name from users limit 1)'
So as you can see, Rails in fact sanitizes it for you, so long as you pass the parameter in as a hash, or method parameter (depending on which query method you're using).
The case for sanitization of data on creating new model records doesn't really apply, as the new or create methods are expecting a hash of values. Even if you attempt to inject unsafe SQL code into the hash, the values of the hash are treated as plain strings, for example:
User.create(:user_name=>"bobby tables); drop table users;")
Results in the query:
INSERT INTO `users` (`user_name`) VALUES ('bobby tables); drop table users;')
So, same situation as above.
Let me know if I've missed or misunderstood anything.
Edit
Regarding escaping html and javascript, the short version is that ERB "escapes" your string content for you so that it is treated as plain text. You can have it treated like html if you really want, by doing your_string_content.html_safe.
However, simply doing something like <%= your_string_content %> is perfectly safe. The content is treated as a string on the page. In fact, if you examine the DOM using Chrome Developer Tools or Firebug, you should in fact see quotes around that string.
Because I always appreciate when I find the source of knowledge and code on any SO answer, I will provide that for this question.
Both ActiveRecord and ActionController provide methods to sanitize sql input.
Specifically from ActiveRecord::Sanitization::ClassMethods you have sanitize_sql_for_conditions and its two other aliases:
sanitize_conditions and sanitize_sql. The three do literally the exact same thing.
sanitize_sql_for_conditions
Accepts an array, hash, or string of SQL conditions and sanitizes
them into a valid SQL fragment for a WHERE clause.
However, in ActiveRecord you also have
sanitize_sql_for_assignment which
Accepts an array, hash, or string of SQL conditions and sanitizes them
into a valid SQL fragment for a SET clause.
Note that these methods are included in ActiveRecord::Base and therefore are included by default in any ActiveRecord model.
On the other hand, in ActionController you have ActionController::Parameters which allows you to
choose which attributes should be whitelisted for mass updating and
thus prevent accidentally exposing that which shouldn't be exposed.
Provides two methods for this purpose: require and permit.
params = ActionController::Parameters.new(user: { name: 'Bryan', age: 21 })
req = params.require(:user) # will throw exception if user not present
opt = params.permit(:name) # name parameter is optional, returns nil if not present
user = params.require(:user).permit(:name, :age) # user hash is required while `name` and `age` keys are optional
The parameters magic is called Strong Parameters, docs here.
I hope that helps anyone, if only to learn and demystify Rails! :)