I'm trying to render ember model data which has more than 1000 records. This will take more than 2 min to finish the rendering part.
So, I have a plan to optimize it. I want to load the first 100 records in the first time. Once they go to the end then I need to load the second 100 records.
How can I do that?
Retrieving pages of data
The concept is paging and depends on how your backend handles paging. But in the generic case, something like:
let result = this.store.query('post', {
limit: 10,
offset: 0
});
once processed by the backend would result in a query to a relational database like:
SELECT * FROM post LIMIT 10 OFFSET 0;
So, you will need to keep track of the current page you are showing. Each time you want to fetch a new page, you will simply increment your offset by page * limit where page is the current page index. So the next query when page = 1 would be:
let result = this.store.query('post', {
limit: 10,
offset: 10 // 1 * 10
});
It's probably a good idea for you backend to return the total result count, which you can access via some kind metadata key on your JSON responses (or however you want since it depends on the way your backend speaks collections). This way you know when to stop trying to fetch the next page.
UX around retrieval
You will need to choose whether you want to do simple paging, which supplies a next and previous button that the user clicks on to retrieve the next / previous page. Probably best UX to manage the page with query params so that the forward/back buttons in the browser move pages and refreshing does not lose the page. You should also disable / enable the previous and next buttons when there are no pages to fetch in either direction.
Pros
easy to develop
user never loses their place
since only ever showing one page at a time, no memory / performance concerns
Cons
users have to click buttons and may choose not to / not realize how
Addons
ember-cli-pagination
The alternative would be using infinite scrolling (a la Facebook news feed). You must pay attention to the scrolling position to know when to fetch the next view (which requires math around the sizes of the current items). Alternatively, you evaluate whether item n - 2 or something like that is in the view port and then prefetch the next page.
Pros
users never think about paging and continue seeing new content with ease
Cons
horrible for returning to ones spot on page change / refresh
horrible for searching
requires a great deal of attention to the amount of data you are showing or else you will run into overconsumption of memory and performance degradation.
more difficult to write since you must handle scrolling events, view port detection, etc
addons
ember-in-viewport
vertical-collection
ember-infinity
Related
When using django-tables2, use the paginate parameter to limit the number of rows displayed. For instance, if I have the table my_table, then in views.py I set it to display 10 rows as follows:
RequestConfig(request, paginate={'per_page':10}).configure(my_table)
My problem is that in my app I want the number of rows shown per page to depend on how big the user's screen is. For instance, 10 rows for mobile devices, but 25 rows for desktop screens. I am imagining doing in views.py what you can do with style sheets using the media rule (e.g., #media(max-height: 480px) {do something}).
Unfortunately, my (noob) understanding is that in Django, the view is relatively insulated from low-level parameters like screen size. However, I don't want to implement some scheme like sending information from the template to the view if there is already a simple solution baked into django-tables2 that I am just ignorant of.
Note I am using Bootstrap4 for my front end, if that makes any difference. The point is that I am already importing Javascript and jQuery into the project, at least at the template level.
Related general discussion
Interestingly, I haven't seen a lot of discussion of adapting table row count to media type, but there is obviously tons of more general discussion of responsive media-sensitive design:
Responsive design with media query : screen size?
https://developer.mozilla.org/en-US/docs/Web/CSS/#media
I have no straightforward or complete solution, but I'll share some thoughts:
When (initially) rendering the table, Django-tables2 has no way of knowing the screen size. One of these workarounds might satisfy your needs:
Render less rows
In your initial page view, if your screen size is small:
Hide some rows, then, adjust the offset and number of pages you request for the next pages using JavaScript. This will involve rewriting url parameters of the pagination links.
Reload the page with a smaller number for per_page added to the url.
Default to rendering a smaller number of rows, and allow the user to change the number of rows rendered with a dropdown, which is hidden for mobile users.
Optimize template for responsiveness
Optimize the template for different screen sizes rather than changing the number of rows. An example of how this could work can be found in the table displayed in webpack documentation. You might still want less rows for smaller screens with this technique though.
Client-side table-sorting
If the total number of rows is reasonable, sending everything to the client and deferring the pagination to something like datatables might work.
We need to track all users(anonymous and authenticated) who spent more than 2 minutes on the site? Is it possible through OOB ways of Sitecore or do we need to write any custom pipeline\processor?
If custom way, what's the best pipeline to override(Session End or VisitEnd pipeline of analytics) and also how to calculate the user session duration?which attribute would give the exact time details?
Sitecore will keep track of the time spent for you in the tracker. In Tracker.Current.Interaction you will find the StartDateTime, EndDateTime and even the SaveDateTime if you would need that.
Of course, this measurement is only final when the user ends his visit. So how to handle this information depends on what you actually want to do with it. You say "track", so that is done. The data is in the database..
Concerning how to calculate the user session duration:
Tracker.Current.Session.Interaction.Pages.Sum((Page page) => page.Duration / 1000);
It will return number of seconds.
If your scenario is something like "after user spent 2 minutes on the site we want to show him advertisement" you should use Sitecore rules engine and exactly VisitDuration condition.
I am developing my website in python(webapp2) and google data store in back end.
I have added query cursor for pagination and it's working good but it has only next and previous functions for pagination, the question is that how i will jump to specific page like i am on page 1 and i want to jump to page 3, how i will manage it in Query Cursor?
i have also visited below links but didn't find any solution
https://www.the-swamp.info/blog/pagination-google-app-engine/
https://cloud.google.com/appengine/docs/standard/python/datastore/query-cursors
You can't. Datastore doesn't know the number of results found by your query.
You can however use some tricks to simulate a full pagination.
For example one technique is to generate a certain number of cursors in a loop and generating the "last page" one.
Like in the following pseudo-code:
(results, next_curs, more) = model.query....fetch_page(...)
for p in xrange(5):
# generate cursor for page number 'b'
# by using the next_cursor from previous page
(results, next_curs, more) = model.....fetch_page(cursor=next_cursor,...)
As you don't access the results cursor the performance is somewhat acceptable (depending on your models complexity, query complexity and such).
You can tweak it on your data.
We recently made a shift from relational (MySQL) to NoSQL (couchbase). Basically its a back-end for social mobile game. We were facing a lot of problems scaling our backend to handle increasing number of users. When using MySQL loading a user took a lot of time as there were a lot of joins between multiple tables. We saw a huge improvement after moving to couchbase specially when loading data as most of it is kept in a single document.
On the downside, couchbase also seems to have a lot of limitations as far as querying is concerned. Couchbase alternative to SQL query is views. While we managed to handle most of our queries using map-reduce, we are really having a hard time figuring out how to handle time based queries. e.g. we need to filter users based on timestamp attribute. We only need a user in view if time is less than current time:
if(user.time < new Date().getTime() / 1000)
What happens is that once a user's time is set to some future time, it gets exempted from this view which is the desired behavior but it never gets added back to view unless we update it - a document only gets re-indexed in view when its updated.
Our solution right now is to load first x user documents and then check time in our application. Sorting is done on user.time attribute so we get those users who's time is less than or near to current time. But I am not sure if this is actually going to work in live environment. Ideally we would like to avoid these type of checks at application level.
Also there are times e.g. match making when we need to check multiple time based attributes. Our current strategy doesn't work in such cases and we frequently get documents from view which do not pass these checks when done in application. I would really appreciate if someone who has already tackled similar problems could share their experiences. Thanks in advance.
Update:
We tried using range queries which works for only one key. Like I said in most cases we have multiple time based keys meaning multiple ranges which does not work.
If you use Date().getTime() inside a view function, you'll always get the time when that view was indexed, just as you said "it never gets added back to view unless we update it".
There are two ways:
Bad way (don't do this in production). Query views with stale=false param. That will cause view to update before it will return results. But view indexing is slow process, especially if you have > 1 milllion records.
Good way. Use range requests. You just need to emit your date in map function as a key or a part of complex key and use that range request. You can see one example here or here (also if you want to use DateTime in couchbase this example will be more usefull). Or just look to my example below:
I.e. you will have docs like:
doc = {
"id"=1,
"type"="doctype",
"timestamp"=123456, //document update or creation time
"data"="lalala"
}
For those docs map function will look like:
map = function(){
if (doc.type === "doctype"){
emit(doc.timestamp,null);
}
}
And now to get recently "updated" docs you need to query this view with params:
startKey="dateTimeNowFromApp"
endKey="{}"
descending=true
Note that startKey and endKey are swapped, because I used descending order. Here is also a link to documnetation about key types that couchbase supports.
Also I've found a link to a question that can also help.
I have a couple hundred of image thumbnails, 15k each. I want to display 20 or so on each page.
Would django.core.paginator suffice for the pagination of these pages? I.e., will it return only those images displayed on the current page? (And if not, what would be a good way to do this?) Thank you.
Depends, because there is one big limitation from the RDBMS (which affects all databases, including MySQL, Postgres, etc.).
django.core.paginator takes a QuerySet which represent any kind of SQL query and adds a LIMIT clause to just get a couple of entries from the database. This approach works well for many kinds of applications, but might become a serious problem if you have a lot of entries. The particular problem is, that whenever you access the 800th page, the database will actually fetch 801*20 entries and then drop the first 800*20 entries again to return the last twenty.
Unfortunately, there is no easy way to solve this problem. In a lot of cases, a next/prev button might be enough so you can write your own pagination which does operate on after-keys instead of page numbers. For example, if the last entry currently displayed by the user has the key "D" you show a next button which links to /next?after=D and then use a SQL query like SELECT * FROM objects WHERE key >DORDER BY key LIMIT 20. The advantage of this approach is, that you can add an index on objects.key which speed up things significantly.
The other approach requires, that you add an additional, indexed (!) column page_num to your table. Then you can perform SQL queries like SELECT * FROM objects WHERE page_num=800 ORDER BY key. With that approach, you can still access all pages randomly, but you have to maintain the page_num column. This might be easy if data is mostly appended at the end and is more complicated if you want to delete/insert elements from the middle efficiently.
So, I would start with django.core.paginator because it's just about 1 line of code. But keep an eye on the response times of your paginated views and the slowquery log from your database. If your database server can't handle the load anymore, you will have to choose one of the techniques mentioned above. Choose solution 2 if random page access is an requirement and solution 1 otherwise (because it's much simpler).
PS: And yes, django.core.paginator will work correctly. :)