Looking for a solution to an issue caused by large data sets forcing Ember to lock up the browser while it tries to process the data.
For pagination, I'm using tchak's handy pagination mixin to paginate approximately 13,000+ objects being loaded from a backend API.
The Ember Data objects contain an ID, one text attribute and several number attributes.
The problem is it takes close to a minute before the browser finishes processing the data, rendering the browser unusable in the meantime. Firefox even goes as far as to issue a warning that a script is using up all browser resources and suggests that script be terminated.
I've written my own pagination mixin that requests objects by range, i.e. items 10-25, and it works generally well except for one serious limitation: sorting. To sort the data, I need to make additional requests to the backend and reload the objects even if some of them have already been loaded.
I would love to be able to load all of the content upfront to simplify the process of sorting without doing additional requests to the backend API. I'm looking for guidance on how to tackle this issue but I'm open to an entirely alternative approach.
If nothing else, is it possible to reduce the resource footprint Ember places on the browser as it tries to load all 13k objects into the ArrayController?
I'm using Ember 1.0.0-pre2 with the latest Ember Data (currently at Revision 10).
On the backend is Rails 3.2.8.
Update I sidestepped the issue by loading data into an ArrayController property other than content. This brought the load times down from over a minute to only a few seconds. I then slice the requested number of items and load those into content. This works well for any number of items, at the cost of not being able to easily sort the data.
I suggest you take a look at Ember Table. The demo shows a table with 500 000 records and works very fast. Digging around the source code might help.
Can't you query a view from your db that handles the sorting? Pass in the sort conditions in the query string ?sortBy=name&sortAsc=true
Related
Update: I see that the issue was I had hidden the left sidebar which has all these features.
So just imported a huge collection for postman. Import worked fine. Now what I was expecting is that I could just easily pull out an individual request from the collection and inspect it like any other. I also expect that I can create a request using one in the collection as a template. There must be a way to easily share groups of requests and examine them like others. Inside the collection runner is the only place I am seeing the individual requests. I mean New (Does not allow it to be selected), Import (Already Done That), Collection Runner. So where is the way to pull out the request? Actually I went ahead and ran the collection with just one request by deselecting all except the one. But the way it is inspected is looks much different. And I want to run this request as all the others are run and inspect it just like the others. I mean this should be the most intuitive thing.
Where is the option here to use one of the requests from a collection?
This is right after clicking new
So I decide I want to create a basic one. Now what?
What indicates the collections selected or available during creation of a request?
So what I can't find is:
1) A easy way to select a collection for use application wide.
2) A easy way to select an individual request from the collection
and run it individually.
3) An easy way to just open a collection
One that is not only collection running. Maybe editing or using one of the items as a template for another request. Also when I click on the left hand side of a request I see a menu but nothing at all comes up. I mean collections of requests should not be just for running in collection runner.
This is the main issue. Beyond that what about editing the requests from a collection?
Using a single request from the imported Collection can be down like this:
Select the Collections Tab (If you're not on it already)
Expand the Collection
Select the Request you want to use
Hit Send to use the request
There are a number of ways to use any Request as a template for other Collections. You could make a copy of the whole Collection and rename it or if you just wanted a single Request in a new Collection you can do the following:
On the Request select ...
Select Duplicate to make a copy of the Request
Select the Save As option
From here you can rename the Request and also create a new Collection containing that request.
There are a lot of things that you can do with the app, maybe either looking through the learning centre:
https://learning.getpostman.com/
Or using the in-app Bootcamp will help you understand the app more and what it can offer you:
I have read Ember docs related to Substates etc and I understand how it works. In my current application my loading.hbs and other child loading.hbs templates work fine.
I just want to discuss a use case.
In my route A, in my model function I do fetchAll for my model.
I go to Route A, First time api request is sent and I see my loading screen.
now I navigate to some other route B.
now I come back to first route A, api request is sent again but this
time loading screen is not shown.
I want to develop my understanding here. Now the second time loading screen is not shown which tells us that store had data so there is no sense to put it on loading and after fetch store sent request to sync with backend.
QUESTION
Now I want to know if this is a default behaviour of Ember with Ember-data?
To show this loading screen, will I have to do something manually?
Ideally what I want is, if on second request data was fetched then show it and at the same time show loader to tell user that it is syncing with backend as well.
I know everything can be done manually, I don't want to reinvent the wheel or do things in non-conventional way. So I want to know best optimized solution for this as provided by Ember which an experienced Ember developer can help me understand.
Thanks in advance.
Now I want to know if this is a default behaviour of Ember with
Ember-data?
Yes, that's the default behavior of Ember data when you do a findRecord or findAll where shouldBackgroundReloadRecord or shouldBackgroundReloadAll event of the adapter respectively, is defaulted to true. You can turn this off by returning false and ensuring shouldReloadAll or shouldReloadRecord respectively are set at true to ensure the request always hits the API and not fetches from cache.
Ideally, showing data immediately on the screen is always advisable as it has a better UX in terms of giving the user a feel that data is already there and that some parts of the data is being fetched. Check here
To show this loading screen, will I have to do something manually?
You may also want to read this
To add further details after my own research, I found helpful and relevant details in Ember docs.This is all about caching.
If records were already there then promise will be resolved immediately that's why I don't see loading screen for already loaded record, at the same time Ember-Data syncs with backend as well and re-render the template.
Ember Model Docs
Caching
The store will automatically cache records for you. If a record had already been loaded, asking for it a second time will always return the same object instance. This minimizes the number of round-trips to the server, and allows your application to render its UI to the user as fast as possible.
For example, the first time your application asks the store for a person record with an ID of 1, it will fetch that information from your server.
However, the next time your app asks for a person with ID 1, the store will notice that it had already retrieved and cached that information from the server. Instead of sending another request for the same information, it will give your application the same record it had provided it the first time. This feature—always returning the same record object, no matter how many times you look it up—is sometimes called an identity map.
Using an identity map is important because it ensures that changes you make in one part of your UI are propagated to other parts of the UI. It also means that you don't have to manually keep records in sync—you can ask for a record by ID and not have to worry about whether other parts of your application have already asked for and loaded it.
One downside to returning a cached record is you may find the state of the data has changed since it was first loaded into the store's identity map. In order to prevent this stale data from being a problem for long, Ember Data will automatically make a request in the background each time a cached record is returned from the store. When the new data comes in, the record is updated, and if there have been changes to the record since the initial render, the template is re-rendered with the new information.
I have setup my database in Django in which I have huge amount of data. The task is to download all the data at a time in csv format. The problem which I am facing here is when the data size (in number of table rows) is upto 2000, I am able to download it but when number of rows reaches to more than 5k, it throws an error, "Gateway timeout". How to handle such issue. There is no table indexing as of now.
Also, when there is 2K data available, it takes around 18sec to download. So how this can be optimized.
First, make sure the code that is generating the CSV is as optimized as possible.
Next, the gateway timeout is coming from your front end proxy; so simply increase the timeout there.
However, this is a temporary reprieve - as your data set grows, this timeout will be exhausted and you'll keep getting these errors.
The permanent solution is to trigger a separate process to generate the CSV in the background, and then download it once its finished. You can do this by using celery or rq which are both ways to queue tasks for execution (and then collect the results at a later time).
If you are currently using HttpResponse from django.http then you could try using StreamingHttpResponse instead.
Failing that, you could try querying the database directly. For example, if you use the MySql database backend, these answers might help you:
dump-a-mysql-database-to-a-plaintext-csv-backup-from-the-command-line
As for the speed of the transaction, you could experiment with other database backends. However, if you need to do this often enough for the speed to be a major issue then there may be something else in the larger process which should be optimized instead.
I will have a sidebar that appears on almost every page of my web app. The sidebar will have some drop-downs, which will consist of a total of say, 1000 different options, which are pulled from the db. Rather than doing the query to get these choices on every page load, I think it makes more sense to just do the query once, say in my config.py and store them in a variable that my views have access to. Is this OK? Is there a better way to accomplish this?
You could do that, but then you'd need to restart your Flask server every time you wanted to update the sidebar.
I'd use some other form of caching here. You could use Flask-Cache and memoize your query results. You can pick a nice long cache timeout, and then clear the cached result whenever you update the sidebar.
We are considering moving from Backbone to Ember. There are a few issues through I can't get answers to from the docs.
1) Ember-Data caches it's data. Our application is multi-user so other users need to be able to see new records created by everyone. Is there a way around this? I read on another post that when a query string is passed, ember data does not cache data, is this true? If it is, can I then just always send query string and nothing will be cached?
2) Ember data has a single model in the router that appears to be instantiated at route load time. I can see that your can request data from multiple sources by returning an object with many this.store.find calls. Say I have a select element and when you select an option, another select gets populated with items based on the first select (which requires a call back to the server). How would that work, how can I get model data on demand (not at route load time)?
I'm not sure if it answers your question but you can always call
model.reload()
to refetch data from server so you can work with up to date data.
You may want to consider Faye (http://faye.jcoglan.com/), which would let you have a pub/sub setup that could update your store by listening to topics of interest. This uses WebSocket for the streaming interface. You could then put new objects into the store, remove or update existing objects which the server could publish to the client.