This is the pattern I find myself running into:
I start making an app, and I use findAll() to get a list of [something random].
Once the app is being tested with serious data, the number of random resource instances will grow. I need to limit the number of resource instances on screen. I need to start paginating them. For this I need query string support. E.g. page[offset].
So findAll(criteria) is replaced by query(criteria, querystring).
This is a pattern so much that findAll() is starting to look like a development placeholder for query() to be used later.
I'm probably misunderstanding the use for findAll(). Is it true findAll() cannot use pagination at all (without customizing adapter code)? Can someone explain in what cases findAll() should be used?
I personally use the findAll method for fetching data that appears in various drop-downs and short lists that cannot be filtered by the user. I use query and queryRecord for pretty much everything else.
Here are a couple of particularities of findAll that can be misleading:
findAll returns all of the records present in the store along with the data that is fetched using the record's adapter.
The return of findAll is two-fold, firstly you will receive the content of the store and then it will be refreshed with the data fetched using the adapter, this behavior can be overridden using the reload flag.
To expand on Jean's answer, findAll does just that, finds all! If you had entities such as "post types" where you have [advertisement, blog, poem], findall makes sense, because you are pulling these 3 things all the time (for example in a "post creator").
Query is more precise. Say You had an api returning every car you have ever seen.
Say you had a "car" model with properties "color" and "bodyStyle"
You could use:
// find all red cars -> /cars?color=red
store.query('car', {color: 'red'});
// find all cars that are coupes -> /cars?bodyStyle=coupe
store.query('car', {bodyStyle: 'coupe'});
To your question on pagination, this is typically implemented on the API. A popular pattern is to accept/return "page" and "count" properties. These are typically found in an API payload's "meta" property.
So if you wanted to look through all cars you know of/have in your database:
// find first 10 cars -> /cars?count=10&page=1
store.query('car', {count: 10, page: 1});
// on the next page, find the next 10 cars -> /cars?count=10&page=2
store.query('car', {count: 10, page: 2});
It is worth nothing that to further your own research you should look into query parameter binding on controllers to ease the labor needed to implement a solution like this.
https://guides.emberjs.com/release/routing/query-params/
In the examples in that link you can see how you can transition to routes and use the query parameters in your store requests to fetch relevant data.
In short, findAll() is great for finding a finite set of easy to represent information, typically types of entities.
query() is great for any filtered set of results based on a criteria, as you mentioned.
Happy Coding :)
If you want "all" record of a type I would recommend using query + peekAll, this is more or less what findAll does under the hood but without various timing issues / freshness issues that findAll is subject to.
query is generally a much better API because it lets you paginate, and most apps with data of any consequence eventually hit a point they are forced to paginate either for rendering concerns or data size concerns.
Related
I have what I believe to be common but complicated problem to model. I've got a product configurator that has a series of buttons. Every time the user clicks on a button (corresponding to a change in the product configuration), the url will change, essentially creating a bookmarkable state to that configuration. The big caveat: I do not get to know what configuration options or values are until after app initialization.
I'm modeling this using EmberCLI. After much research, I don't think it's a wise idea to try to fold these directly into the path component, and I'm looking into using the new Ember query string additions. That should work for allowing bookmarkability, but I still have the problem of not knowing what those query parameters are until after initialization.
What I need is a way to allow my Ember app to query the server initially for a list of parameters it should accept. On the link above, the documentation uses the parameter 'filteredArticles' for a computed property. Within the associated function, they've hard-coded the value that the computed property should filter by. Is it a good idea to try to extend this somehow to be generalizable, with arguments? Can I even add query parameters on the fly? I was hoping for an assessment of the validity of this approach before I get stuck down the rabbit hole with it.
I dealt with a similar issue when generating a preview popup of a user's changes. The previewed model had a dynamic set of properties that could not be predetermined. The solution I came up with was to base64 encode a set of data and use that as the query param.
Your url would have something like this ?filter=ICLkvaDlpb0iLAogICJtc2dfa3
The query param is bound to a 2-way computed that takes in a base64 string and outputs a json obj,
JSON.parse(atob(serializedPreview));
as well as doing the reverse: take in a json obj and output a base64 string.
serializedPreview = btoa(JSON.stringify(filterParams));
You'll need some logic to prevent empty json objects from being serialized. In that case, you should just set the query param as null, and remove it from your url.
Using this pattern, you can store just about anything you want in your query params and still have the url as shareable. However, the downside is that your url's query params are obfuscated from your users, but I imagine that most users don't really read/edit query params by hand.
I'm using the High Replication Datastore, along with ndb. I have a kind with over 27,000 entities, which isn't that much. Supposedly the datastore is efficient in querying and extracting large amounts of data, but whenever I query over that kind, queries take a long time to finish (I've even got DeadlineExceededErrors).
I have a model where I store keywords and URLs I want to index in Google:
class Keywords(ndb.Model):
keyword = ndb.StringProperty(indexed=True)
url = ndb.StringProperty(indexed=True)
number_articles = ndb.IntegerProperty(indexed=True)
# Some other attributes... All attributes are indexed
My current use cases are to build my Sitemap, and to fetch my top 20 keywords to link from my hope page.
When I fetch many entities, I usually do:
Keywords.query().fetch() # For the sitemap, as I want all of the urls
Keywords.query(Keywords.number_articles > 5).fetch() # For the homepage, I want to link to keywords with more than 5 articles
Is there a better way to extract data?
I've tried to index data into the Search API, and I've seen huge speed gains. Even though this works, I don't think it's ideal to replicate data from the Datastore into Search API with basically the same fields.
Thanks in advance!
I would split this functionality.
For home page you can use your second query, but add, as advised by Bruyere, limit=20 paramater. Such request should run very fast, if you have the right index.
The site map is a bigger issue. Usually, to process large number of entities, you use Map reduce.
It's probably a good idea, but only if you don't have too many requests to sitemap. It can also be the only solution if you update Keywords entities often and want as up to date site map as possible.
Another option can be to generate sitemap in a task, save it as a blob and serve this blob in the request. That is really quick. If your updates to the Keywords entities are not very frequent, then you can run this task after any update. If you have many updates, then you can schedule the task to run periodically in cron. As you have success using search API, then this is probably the best option for you.
Generally speaking I don't think it's a good idea to use datastore to retrieve large amounts of data. I recommend to look at least at Datastore comparison with traditional databases. It's designed to handle large databases, but not necessarily large result sets. I would say that datastore is designed to handle large amounts of small requests.
DB speed is related to the number of results returned, not the number of records in the DB. You say:
to build my Sitemap, and to fetch my top 20 keywords
If thats the case add limit=20 in both fetches. If you do it that way then use run instead as per the docs:
https://developers.google.com/appengine/docs/python/datastore/queryclass#Query_fetch
After reading an article on REST ("Restful Grails"), I have gotten the impression that it is not possible to truly conform to a REST style in a service that demands a lot of parameters. Is this so? All the examples I have seen so far seem to imply that true REST style services are "parameterless". Using parameters would be RPC-ish and not truly RESTful.
To be more specific, say we have a service that returns graph data for stock prices, and this service needs to know the start date, end date, the currency, stock name, and whatever else might be applicable. In any case, at least 4-5 parameters are needed to retrieve the information needed.
I would imagine the URL to be something like this : /stocks/YAHOO?startDate="2008-09-01"&endDate=...
("YAHOO" is here a made-up stock name).
Would this really be REST or is this more RPC-like, what the author of the aforementioned article calls "GETful" (i.e. just low ceremony rpc)?
You can see the querystring as a filter on the resource you are GETing. Here, your resource is the stock prices of yahoo. Doing a GET on that resource give you all the available data, or the most recents. The query string filter the prices you want. Content negociation allow you to change the representation, e.g. a png graph, a csv file, and so on. To add a price, simply POST a representation (e.g. CSV) to the same resource.
The "restfulness" is not realy in the URL itself, since URIs are obscures to client, but in the way you interact with resources themselves identified by their URI
Feel free to use as many parameters as you need to identify the resource you wish to access. REST doesn't care.
Why would you think it is not possible?
Google uses REST for their charts api, and they take alot of params:
http://chart.apis.google.com/chart?cht=bvg&chs=350x300&chd=t:20,35,10&chxr=1,0,40&chds=0,40&chco=FF0000|FFA000|00FF00&chbh=65,0,35&chxt=x,y,x&chxl=0:|High|Medium|Low|2:||Task+Priority||&chxs=2,000000,12&chtt=Tasks+on+my+To+Do+list&chts=000000,20&chg=0,25,5,5
Let's say I have a method that returns a list of customers and as input has a list of states and list of sizes, something like
return customers where state in (NY, CA, TX) and size in (Small, Medium)
What would the best RESTFul URL that I should use? The problem that it is a query and does not point to a specific 'resource'. Here are some options that I am mulling over.
somesite.com/customers?state=NY,CA,TX&size=small,medium (old style)
somesite.com/customers/state/NY,CA,TX/size/small,medium
somesite.com/customers/state=NY,CA,TX/size=small,medium
somesite.com/customers/state(NY,CA,TX)/size(small,medium)
Option 1 - query params are intended for exactly that. Parameters for your query.
You are interested in a list of customers therefore the last "folder" should be "/customers". The fact that you want a subset of these and that that subset is variant depending on input, and in combination leads you to query params acting as filters. (Nothing else would make sense as you see by being compelled to ask the question).
The real question you have is whether the params are going to be inclusive or exclusive by default (i.e. AND or OR). That question has already been asked here if I can just find it...
I think #1 (somesite.com/customers?state=NY,CA,TX&size=small,medium) is the best of the bunch. The customers are the resources, and the query string is just placing restrictions on the resources being requested.
Personally, I'd use the 4th approach, but with the '+' sign instead of parenthesis:
somesite.com/customers/NY+CA+TX/small+medium
RESTful-style your Models are not necessarily all the RESTful Resources you should offer... You can add any number of (artificial) resources as you see fit, even ones that would require a JOINs from your Models.
For what it's worth, URI naming conventions has nothing to do with REST. In fact, if you define a way of constructing your application's URIs out-of-band as part of your API, you are violating a constraint of REST. See: http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven
I'm doing research into a web API for my company, and it's starting to look like we might implement a RESTful one. I've read a couple of books about this now (O'Reilly's "RESTful web services" seeming the most useful) and have come up with the following set of URIs and operations for an object that can be commented on, tagged, and rated.
It doesn't really matter what the object is, as this scenario applies to many things on the net, but for the sake of argument lets say it's a movie.
Some of these seem to fit quite naturally, but others seem a bit forced (rating and tagging particularly) so does anybody have any suggestions about how these could be improved? I'll list them with the URI and then the supported verbs, and what I propose they would do.
/movies
GET = List movies
/movies/5
GET = Get movie 5
/movies/5/comments
GET = List comments on movie 5
POST = Create a new comment on movie 5
/movies/5/comments/8
GET = Get comment 8 on movie 5
POST = Reply to comment 8 on movie 5
PUT = Update comment 8 on movie 5
/movies/5/comments/8/flag
GET = Check whether the movies is flagged as inappropriate (404 if not)
PUT = Flag movie as inappropriate
/movies/5/rating
GET = Get the rating of the movie
POST = Add the user rating of the movie to the overall rating
Edit: My intention is that the movie object would contain its rating as a property, so I wouldn't really expect the GET method to be used here. The URI really exists so that the rating can be an individual resource that can be updated using the POST verb. I'm not sure if this is the best way of doing it, but I can't think of a better one
/movies/5/tags/tagname
GET = Check whether the movies is tagged with tagname (404 if not; but if it is tagged with the tag name should it return the actual tag resource by redirecting to something like /tags/tagname?)
PUT = Add tag tagname to the movie, creating the tag resource /tags/tagname if required
DELETE = Remove tag tagname from the movie, deleting the tag resource tags/tagname if nothing is tagged with it after this removal
Note that these wouldn't be the entire URIs, for example the URI to list the movies would support filtering, paging and sorting. For this I was planning on something like:
/movies/action;90s/rating,desc/20-40
Where:
action;90s is a semi-colon delimited set of filter criteria
rating,desc is the sort order and direction
20-40 is the range of item indices to get
Any comments about this API scheme too?
Edit #1
This post is getting quite long now! After reading some of the answers and comments, this is the changes from above I'm planning on making:
Tags will be handled as a group rather than individually, so they will be at:
/movies/5/tags
GET = List tags
POST = Union of specified tags and existing tags
PUT = Replace any current tags with specified tags
DELETE = Delete all tags
I'm still really not sure how to handle flagging a comment though. One option is that instead of POSTing to a comment replying to it, a comment object will include its parent so it can be POSTed to the general URI, i.e.
/movie/5/comment
POST = Create a new comment (which may be a reply to a comment)
I could then use the POST to a comment to flag it. But this still doesn't feel quite right.
/movie/5/comment/8
POST = Flag comment
Most of what you have looks good. There were just a couple of strange things I saw. When I put my URLs together, I try to follow these four principles.
Peel the onion
If you make the R in REST really be a resource then the resource URL should be able to be peeled back and still be meaningful. If it doesn't make sense you should rethink how to organize the resource. So in the case below, each makes sense. I am either looking at a specific item, or a collection of items.
/movies/horror/10/
/movies/horror/
/movies/
The following seems funny to me because flag isn't a resource, it's a property of the movie.
/movies/5/comments/8/flag -> Funny
/movies/5/comments/8/ -> Gives me all properties of comment including flag
Define the View
The last peice of the URL describes how to show the resource. The URL /movies/horror/ tells me I will have a collection of movies refined by horror. But there might be different ways I want to display that collection.
/movies/horror/simple
/movies/horror/expanded
The simple view might just be the title and an image. The expanded view would give a lot more information like description, synopsis, and ratings.
Helpers
After the resource has been limited and the proper view figured out, query string parameters are used to help the UI with the little stuff. The most common query string parameters I use are
p => Page
n => number of items to display
sortby => field to sort by
asc => sort ascending
So I could end up with a URL like
/movies/horror/default?p=12&n=50&sortby=name
This will give me the list of movies limited to horror movies with the default view; starting on page 12 with 50 movies per page where the movies are sorted by name.
Actions
The last thing needed are your action on the resource. The action are either collection based or item based.
/movies/horror/
GET -> Get resources as a list
POST -> Create, Update
/movies/horror/10/
GET -> Get resource as item
POST -> Update
I hope this helps.
I disagree with the edit. Queries should be defined by querystrings as per Martijn Laarman's post. i.e.:
/movies?genre=action&timeframe=90s&lbound=20&ubound=40&order=desc
Well, the way I see it some of the information you return now as objects could simply be added to the metadata of its parent object.
For instance, rating could be part of the response of /movies/5
<movie>
<title>..</title>
..
<rating url="movies/ratings/4">4</rating>
<tags>
<tag url="movies/tags/creative">creative</tag>
...
Removing a tag simply means posting the above response without that tag.
Also queries should go in URL variables, I believe:
/movies/?startsWith=Forrest%20G&orderBy=DateAdded
Based on my understanding of ROA (I'm only on chapter five of RESTful Web Services) it looks good to me.
This is an awesome initial draft for a spec of a REST API. The next step would to specify expected return codes (like you did with "404 No Tag Available"), acceptable Content-Types, and available content-types (e.g., HTML, JSON). Doing that should expose any additional chinks you'll need to hammer out.
#Nelson LaQuet:
Using the HTTP methods as they are actually defined gives you the safety of knowing that executing a GET on anything on a web site or service won't eat your data or otherwise mangle it. As an example (pointed out in RESTful Web Services) Google's Web Accelerator expects this behaviour -- as stated in the FAQ -- and presumably other services do too.
Also it gets you idempotency for free. That is doing a GET, DELETE, HEAD or PUT on a resource more than once is the same as doing it only once. Thus if your request fails then all you have to do is run it again.
This is not REST.
A REST API must not define fixed resource names or hierarchies (an obvious coupling of client and server). Servers must have the freedom to control their own namespace. Instead, allow servers to instruct clients on how to construct appropriate URIs, such as is done in HTML forms and URI templates, by defining those instructions within media types and link relations. [Failure here implies that clients are assuming a resource structure due to out-of band information, such as a domain-specific standard, which is the data-oriented equivalent to RPC's functional coupling].
http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven