I'm doing some tests with an HTML map in conjunction with Leaflet. Server side I have a Ruby Sinatra app serving json markers fetched by a MySQL table. What are the best practices working with 2k-5k and potentially more markers?
Load all the markers in the first place and then delegate everything to Leaflet.markercluster.
Load the markers every time the map viewport change, sending southWest & northEast points to the server, elaborate the clipping server side and then sync the marker buffer client side with the server-fetched entries (what I'm doing right now).
A mix of the two above approaches.
Thanks,
Luca
A few months have passed since I originally posted the question and I made it through!
As #Brett DeWoody correctly noted the right approach is to be strictly related to the number of DOM elements on the screen (I'm referring mainly to markers). The more the merrier if your device is faster (CPU especially). Since the app I was developing has both desktop and tablet as target devices, CPU was a relevant factor just like the marker density of different geo-areas.
I decided to separate DBase querying/fetching and map representation/displaying. Basically, the user adjusts controls/inputs to filter the whole dataset, afterward records are fetched and Leaflet.markercluster does the job of representation. When a filter is modified the cycle starts over. Users can choose the map zoom level of clusterization depending on their CPU power.
In my particular scenario, the above-mentioned represented the best approach (verified by console.time). I found that viewport optimization was good for lower marker-density areas (a pity).
Hope it may be helpful.
Cheers,
Luca
Try options and optimize when you see issues rather than optimizing early. You can probably get away with just Leaflet.markercluster unless your markers have lots of data attached to them.
Related
I'm using Ember JS 2.3.0, Ember Data 2.3.3 and jQuery 2.1.4 and I'm having an issue when trying to transition away from a page with a HTML table component on it. The table component is populated with data from this Ember Data call:
this.store.query('log', {filter: {
created_at_from: moment().subtract(1, 'month').format('DD/MM/YYYY'),
created_at_to: moment().format('DD/MM/YYYY'),
object: 'IsoApplication'
}})
which having looked in the developer tools "Timeline" and "Network" tabs is resolving in good time. It has no special jQuery events etc attached to it and is simply a plain HTML table made dynamic with Ember.
However, when clicking a "link-to" helper that transitions away from the page with the component on it there is a huge 10 second delay before the destination page renders. During this time nothing seems to happen and having used the developer tools "Timeline" and drilled down it seems that the delay occurs when the "removeFromListeners" function is being called (see attached timeline screenshot). No asynchronous stuff is happening at this point and therefore no loader appears but it seems that tearing down the table is the problem because if I remove the table component from the template the issue disappears. Without the table component the page transitions almost instantly.
I have attached the timeline screenshot to see if anyone can help to pinpoint the possible cause of this problem.
Any suggestions would be welcome.
Thanks.
Ok, after significantly reducing the number of rows in my table I discovered that the issue went away. #runspired provided a detailed and very helpful description of why my issue occurred, which I have posted below to inform others who may have similar issues:
there's a number of things going on here
I've given a few talks on this (including two podcasts on Ember Weekend (episodes 38 and 39))
I wish I had my chicago-ember talk on it recorded but I don't
but basically, comes down to three things as to why this is slow
and none of it is actually because "dom is slow", that argument masks the truth of the matter
1.) don't render the universe, render the scene
easiest analogy is take any video game you can think of, especially one in which you can "pivot" the camera view quickly
that pivot is a lot like scroll
you really only want to render what's visible at the moment, and prepare content that's closer to the camera's view
but it makes no sense to prepare content that's far out of view
this is a lesson that every native SDK and game SDK has baked in, but which no JS framework has yet realized
when people say DOM is slow, it's really because they are trying to render far more things than they ought to
2.) is a bug in most browsers
memory allocated for DOM is always retained by the tab that allocated it
this allocation persists beyond the lifecycle of your page
it's a true leak that hopefully Chrome etc. will fix soon
or basically, the allocation heap size made for DOM can only ever grow
you can reuse allocated memory, but you can't shrink it
this means that if you generate a huge amount of DOM nodes, even once you tear them down there's a high memory overhead for that tab
which causes performance issues in other areas because less memory is available for JS allocations etc.
this problem is bigger on mobile, especially android, than on desktop. But it can affect desktop too.
Something going hand in hand with 1 and 2
is there's a definite limit to how many DOM nodes you can render before the browser starts chugging
I've found that limit to be roughly 40k DOM nodes, varies from as low as 20k on older android devices to > 100k on Safari
but 40k seems to be a limit which if held to as a target works well for all platforms
if you have 700-2000 table rows, and each row has maybe 3 TDs, and each TD has just one link or span or other element, and then in that you have some text
6,300 - 18,000 nodes
or basically, you're dealing with half of your max budget with that setup
just as the content of the rows
so 7002 - 20,002 nodes counting the rows
and it's likely that 9 nodes per row is a very low estimate
if we double that
you're exceeding that max
and 18 nodes sounds more likely
3.) GC events stop the world
[9:28]
even though the memory allocated by the tab is retained
the allocation IS cleaned up
(so it can be reused for more DOM later)(edited)
so you're releasing the memory for at LEAST your max DOM size (likely larger) all at once
that's a lot for the browser to clean up on
it's also likely a lot of arrays and array observers are being torn down in this time.
#runspired is also the author of
https://github.com/runspired/smoke-and-mirrors which is a project that:
focuses on improving initial and re-render performance in high-stress situations by providing components and primitives for performant lists and svelte renders to match a core belief: Don't render the universe, render the scene.
Is it possible to tell bullet that something has happened in the past so it can take this information and adjust internal interpolation to show this change in the present?
There would never be a need to go back in time more than 1-5 seconds, 5 being a very rare occasion, more realistically between 1.5 and 2.5 seconds is where most of the change would occur.
The ability to query an objects position, rotation, and velocities at a specific time in the past would be needed as well, but this can easily be accomplished.
The reason behind all of this would be to facilitate easier synchronization of two physics simulations, specifically in a networked environment.
A server constantly running the simulation in real-time would send position, rotation, and velocity updates to client simulations at periodic intervals. These updates would arrive at the client 'in the past' from the client simulation's perspective due to network latency, thus the need to query the updated objects values in the past to see if they are different arises, if they are different the need to change these values in the past is also necessary. During the next simulation step bullet would take these past-changes into account and update the object accordingly.
Is this ability present in bullet or would it need to be emulated somehow? If emulation is needed could someone point me in the correct direction to getting started on this "rewind and replay" feature?
If you are unfamiliar with "rewind and replay" this article goes into great detail about the theory behind implementation for someone who may be going about creating their own physics library. http://gafferongames.com/game-physics/n ... d-physics/
Imagine to have a Desktop application - could be best described as record keeping where the user inserts/views the records - that relies on a DB back-end which will contain large objects' hierarchies and properties. How should data retrieval be handled?
Should all the data be loaded at start-up and stored in corresponding Classes/Structures for later manipulation or should the data be retrieved only at need, stored in mock-up Classes/Structures and then reused later instead of being asked to the DB again?
As far as I can see the former approach would require a bigger memory portion used and possible waiting time at start-up (not so bad if a splash screen is displayed), while the latter could possibly subject the user to delays during processing due to data retrieval and would require to perform some expensive queries on the database, whose results and/or supporting data structures will most probably serve no purpose once used*.
Something tells me that the solution lies on an in-depth analysis which will lead to a mixture of the two approaches listed above based on data most frequently used, but I am very interested in reading your thoughts, tips and real life experiences on the topic.
For discussion's sake, I'm thinking about C++ and SQLite.
Thanks!
*assuming that you can perform on Classes/Objects faster operations rather than have to perform complicated queries on the DB.
EDIT
Some additional details:
No concurrent access to the data, meaning only 1 user works on the data which is stored locally.
Data is sent back depending on changes made humanly - i.e. with low frequency. This is not necessarily true for reading data from the DB, where I can expect to have few peaks of lots of reads which I'd like to be fast.
What I am most afraid of is the user getting the feeling of slowness when displaying a complex record (because this has to be read in from the DB).
Use Lazy Load and Data Mapper (pg.165) patterns.
I think this question depends on too many variables to be able to give a concrete answer. What you should consider first is how much data you need to read from the database in to your application. Further, how often are you sending that data back to the database and requesting new data? Also, will users be working on the data concurrently? If so, loading the data initially is probably not a good idea.
After your edits I would say it's probably better to leave the data at the database. If you are going to be accessing it with relatively low frequency there is no reason to load up or otherwise try to cache it in your application at launch. Of course, only you know your application best and should decide what bits may be loaded up front to increase performance.
You might consider to user intermediate server (WCF) that will contain cached data from the database in memory, this way users don't have to go every time to the database. Also since it is only one access point to for all users if somebody changes/added record you can update cache as well. Static data can be reloaded every x hours (for example every hour). It still might not the best option, since data needs to be marshaled from Server to the Client, but you can use netTcp binding if you can, which is fast and small.
I have a Windows Phone 7 app that (currently) calls an OData service to get data, and throws the data into a listbox. It is horribly slow right now. The first thing I can think of is because OData returns way more data than I actually need.
What are some suggestions/best practices for speeding up the fetching of data in a Windows Phone 7 app? Anything I could be doing in the app to speed up the retrieval of data and putting into in front of the user faster?
Sounds like you've already got some clues about what to chase.
Some basic things I'd try are:
Make your HTTP requests as small as possible - if possible, only fetch the entities and fields you absolutely need.
Consider using multiple HTTP requests to fetch the data incrementally instead of fetching everything in one go (this can, of course, actually make the app slower, but generally makes the app feel faster)
For large text transfers, make sure that the content is being zipped for transfer (this should happen at the HTTP level)
Be careful that the XAML rendering the data isn't too bloated - large XAML structure repeated in a list can cause slowness.
When optimising, never assume you know where the speed problem is - always measure first!
Be careful when inserting images into a list - the MS MarketPlace app often seems to stutter on my phone - and I think this is caused by the image fetch and render process.
In addition to Stuart's great list, also consider the format of the data that's sent.
Check out this blog post by Rob Tiffany. It discusses performance based on data formats. It was written specifically with WCF in mind but the points still apply.
As an extension to the Stuart's list:
In fact there are 3 areas - communication, parsing, UI. Measure them separately:
Do just the communication with the processing switched off.
Measure parsing of fixed ODATA-formatted string.
Whether you believe or not it can be also the UI.
For example a bad usage of ProgressBar can result in dramatical decrease of the processing speed. (In general you should not use any UI animations as explained here.)
Also, make sure that the UI processing does not block the data communication.
I'm very curious to know how this process works. These sites (http://www.sharkscope.com and http://www.pokertableratings.com) data mine thousands of hands per day from secure poker networks, such as PokerStars and Full Tilt.
Do they have a farm of servers running applications that open hundreds of tables (windows) and then somehow spider/datamine the hands that are being played?
How does this work, programming wise?
There are a few options. I've been researching it since I wanted to implement some of this functionality in a web app I'm working on. I'll use PokerStars for example, since they have, by far, the best security of any online poker site.
First, realize that there is no way for a developer to rip real time information from the PokerStars application itself. You can't access the API. You can, though, do the following:
Screen Scraping/OCR
PokerStars does its best to sabotage screen/text scraping of their application (by doing simple things like pixel level color fluctuations) but with enough motivation you can easily get around this. Google AutoHotkey combined with ImageSearch.
API Access and XML Feeds
PokerStars doesn't offer public access to its API. But it does offer an XML feed to developers who are pre-approved. This XML feed offers:
PokerStars Site Summary - shows player, table, and tournament counts
PokerStars Current Tournament data - files with information about upcoming and active tournaments. The data is provided in two files:
PokerStars Static Tournament Data - provides tournament information that does not change frequently, and
PokerStars Dynamic Tournament Data - provides frequently changing tournament information
PokerStars Tournament Results - provides information about completed tournaments. The data is provided in two files:
PokerStars Tournament Results – provides basic information about completed tournaments, and
PokerStars Tournament Expanded Results – provides expanded information about completed tournaments.
PokerStars Tournament Leaders Board - provides information about top PokerStars players ranked using PokerStars Tournament Ranking System
PokerStars Tournament Leaders Board BOP - provides information about top PokerStars players ranked using PokerStars Battle Of Planets Ranking System
Team PokerStars – provides information about Team PokerStars players and their online activity
It's highly unlikely that these sites have access to the XML feed (or an improved one which would provide all the functionality they need) since PokerStars isn't exactly on good terms with most of these sites.
This leaves two options. Scraping the network connection for said data, which I think is borderline impossible (I don't have experience with this so I'm not sure; I've heard it's highly encrypted and not easy to tinker with, but I'm not sure) and, mentioned above, screen scraping/OCR.
Option #2 is easy enough to implement and, with some work, can avoid detection. From what I've been able to gather, this is the only way they could be doing such massive data mining of PokerStars (I haven't looked into other sites but I've heard security on anything besides PokerStars/Full Tilt is quite horrendous).
[edit]
Reread your question and realized I didn't unambiguously answer it.
Yes, they likely have a massive amount of servers running watching all currently running tables, tournaments, etc. Realize that there is a decent amount of money in what they're doing.
This, for instance, could be how they do it (speculation):
Said bot applications watch the tables and data mine all information that gets "posted" to the chat log. They do this by already having a table of images that correspond to, for example, all letters of the alphabet (since PokerStars doesn't post their text as... text. All text in their software is actually an image). So, the bot then rips an image of the chat log, matches it against the store, converts the data to a format they can work with, and throws it in a database. Done.
[edit]
No, the data isn't sold to them by the poker sites themselves. This would be a PR nightmare if it ever got out, which it would. And it wouldn't account for the functionality of these sites, which appears to be instantaneous. OPR, Sharkscope, etc. There are, without a doubt, applications running that are ripping the data real time from the poker software, likely using the methods I listed.
maybe I can help.
I play poker, run a HUD, look at the stats and am a software developer.
I've seen a few posts on this suggesting it's done by OCR software grabbing the screen. Well, that's really difficult and processor hungry, so a programmer wouldn't choose to do that unless there were no other options.
Also, because you can open multiple windows, the poker window can be hidden or partially obscured by other things on the screen, so you couldn't guarantee to be able to capture the screen.
In short, they read the log files that are output by the poker software.
When you install your HUD like Sharkscope or Jivaro etc, than they run client software on your PC. It reads the log files and updates its own servers with every hand you play.
Most poker software is similar, but lets start with Pokerstars, as thats where I play. The Poker software outputs to local log files for every action you/it makes. It shows your cards, any opponents cards that you see plus what you do. eg. which button you have pressed, how much you/they bet etc. It posts these updates in near real time and timestamps the log file.
You can look at your own files to see this in action.
On a PC do this (not sure what you do on a Mac, but will be similar)
1. Load File Explorer
2. Select VIEW from the menu
3. Select HIDDEN ITEMS so that you can see the hidden data files
4. Goto C:\Users\Dave\AppData\Local\PokerStars.UK (you may not be called DAVE...)
5. Open the PokerStars.log.0 file in NOTEPAD
6. In Notepad, SEARCH for updateMyCard
7. It will show your card numerically
3c for 3 of Clubs
14d for Ace of Diamonds
You can see your opponents cards only where you saw them at the table.
Here is a few example lines from the log file.
OnTableData() round -2
:::TableViewImpl::updateMyCard() 8s (0) [2A0498]
:::TableViewImpl::updateMyCard() 13h (1) [2A0498]
:::TableViewImpl::updatePlayerCard() 7s (0) [2A0498]
:::TableViewImpl::updatePlayerCard() 14s (1) [2A0498]
[2015/12/13 12:19:34]
cheers, hope this helps
Dave
I've thought about this, and have two theories:
The "sniffer" sites have every table open, AND:
Are able to pull the hand data from the network stream. (or:)
Are obtaining the hand data from the GUI (screen scraping, pulling stuff out via the GUI API).
Alternately, they may have developed/modified clients to log everything for them, but I think one of the above solutions is likely simpler.
Well, they have two choices:
they spider/grab the data without consent. Then they risk being shut down anytime. The poker site can easily detect such monitoring at this scale and block it. And even risk a lawsuit for breach of the terms of service, which probably disallow the use of robots.
they pay for getting the data directly. This saves a lot of bandwidth (e.g. not having to load the full pages, extraction, updates with html changes etc.) and makes their business much less risky (legally and technically).
Guess which one they more likely chose; at least if the site has been around for some time without being shut down every now and then.
I'm not sure how it works but I have an application id and a key- which you get as a gold or silver subscriber- sign up for a month and send them an email and you will get access and the API documentation.