I'd like to download all pictures (in small size) for places I've retrieved via a search with type=place via the Facebook API. It works fine getting the pictures (a la graph.facebook.com/[id]/picture?type=small), but it's darn slow (Android via Wi-Fi or 3G). For a set of 25ish pictures it takes up to 10 seconds. Is the only option to go for caching or is there a way to download the pictures faster, possibly together with the other data in a search?
The most time consuming part of any query is the transit time. If you're making single API calls in a loop of some kind, it's going to slow your application down to a crawl. I'd say you're doing really good if you're making 25 API calls in 10 seconds. Most of the time it takes me about 1 sec/call.
If speed is your prime concern, make a single API call that gets everything at once. FQL or batch.request allows you to do this.
Related
I hope the title is chosen well enough to ask this question.
Feel free to edit if not and please accept my apologies.
I am currently laying out an application that is interacting with the web.
Explanation of the basic flow of the program:
The user is entering a UserID into my program, which is then used to access multiple xml-files over the web:
http://example.org/user/userid/?xml=1
This file contains several ID's of products the user owns in a DRM-System. This list is then used to access stats and informations about the users interaction with the product:
http://example.org/user/appid/stats/?xml=1
This also contains links to various images which are specific to that application. And those may change at any time and need to be downloaded for display in the app.
This is where the horror starts, at least for me :D.
1.) How do I store that information on the PC of the user?
I thought about using a directory for the userid, then subfolders with the appid to cache images and the xml-files to load them on demand. I also thought about using a zipfile while using the same structure.
Or would one rather use a local db like sqlite for that?
Average Number of Applications might be around ~100-300 and stats and images per app from basically 5-700.
2.) When should I refresh the content?
The bad thing is, the website from where this data is downloaded, or rather the xmls, do not contain any timestamps when it was refreshed/changed the last time. So I would need to hash all the files and compare them in the moment the user is accessing that data, which can take an inifite amount of time, because it is webbased. Okay, there are timeouts, but I would need to block the access to the content until the data is either downloaded and processed or the timeout occurs. In both cases, the application would not be accessible for a short or maybe even long time and I want to avoid that. I could let the user do the refresh manually when he needs it, but then I hoped there are some better methods for that.
Especially with the above mentioned numbers of apps and stuff.
Thanks for reading and all of that and please feel free to ask if I forgot to explain something.
It's probably worth using a DB since it saves you messing around with file formats for structured data. Remember to delete and rebuild it from time to time (or make sure old stuff is thoroughly removed and compact it from time to time, but it's probably easier to start again, since it's just a cache).
If the web service gives you no clues when to reload, then you'll just have to decide for yourself, but do be sure to check the HTTP headers for any caching instructions as well as the XML data[*]. Decide a reasonable staleness for data (the amount of time a user spends staring at the results is a absolute minimum, since they'll see results that stale no matter what you do). Whenever you download anything, record what date/time you downloaded it. Flush old data from the cache.
To prevent long delays refreshing data, you could:
visually indicate that the data is stale, but display it anyway and replace it once you've refreshed.
allow staler data when the user has a lot of stuff visible, than you do when they're just looking at a small amount of stuff. So, you'll "do nothing" while waiting for a small amount of stuff, but not while waiting for a large amount of stuff.
run a background task that does nothing other than expiring old stuff out of the cache and reloading it. The main app always displays the best available, however old that is.
Or some combination of tactics.
[*] Come to think of it, if the web server is providing reasonable caching instructions, then it might be simplest to forget about any sort of storage or caching in your app. Just grab the XML files and display them, but grab them via a caching web proxy that you've integrated into your app. I don't know what proxies make this easy - you can compile Squid yourself (of course), but I don't know whether you can link it into another app without modifying it yourself.
Are there reasons to go one way or the other?
System.Web.UI.DataVisualization vs. System.Windows.Forms.DataVisualization
I'm building a process for a build server, where it will simply output a chart to an image based on a sql data rows fetched at run-time. I don't have anything vested in either direction, and have experience in both worlds, but none in Ms Charting.
I assume there are good reasons to go either way.
What are the things to consider about going either direction?
We went the WinForms route because it took a long time for our data intensive charts to be generated on the fly. With a service you can have the charts generated slightly ahead of time (5 minute cycle for us) and as a result the webpage loads instantly. Also this means that at most our DB is queried 1 time every five minutes whereas each person who loads the web chart would hit the database individually resulting in a lot more load on the DB.
I have a Windows Phone 7 app that (currently) calls an OData service to get data, and throws the data into a listbox. It is horribly slow right now. The first thing I can think of is because OData returns way more data than I actually need.
What are some suggestions/best practices for speeding up the fetching of data in a Windows Phone 7 app? Anything I could be doing in the app to speed up the retrieval of data and putting into in front of the user faster?
Sounds like you've already got some clues about what to chase.
Some basic things I'd try are:
Make your HTTP requests as small as possible - if possible, only fetch the entities and fields you absolutely need.
Consider using multiple HTTP requests to fetch the data incrementally instead of fetching everything in one go (this can, of course, actually make the app slower, but generally makes the app feel faster)
For large text transfers, make sure that the content is being zipped for transfer (this should happen at the HTTP level)
Be careful that the XAML rendering the data isn't too bloated - large XAML structure repeated in a list can cause slowness.
When optimising, never assume you know where the speed problem is - always measure first!
Be careful when inserting images into a list - the MS MarketPlace app often seems to stutter on my phone - and I think this is caused by the image fetch and render process.
In addition to Stuart's great list, also consider the format of the data that's sent.
Check out this blog post by Rob Tiffany. It discusses performance based on data formats. It was written specifically with WCF in mind but the points still apply.
As an extension to the Stuart's list:
In fact there are 3 areas - communication, parsing, UI. Measure them separately:
Do just the communication with the processing switched off.
Measure parsing of fixed ODATA-formatted string.
Whether you believe or not it can be also the UI.
For example a bad usage of ProgressBar can result in dramatical decrease of the processing speed. (In general you should not use any UI animations as explained here.)
Also, make sure that the UI processing does not block the data communication.
SRS for the system I'm currently working on includes the following non-functional requirement: "the SuD shall be scalable to 200 concurrent users". How can I convert this statement to a more measurable characteristic: "hits per second"?
Assuming you're talking about a web application (based on your desire to estimate "hits" per second), you have to work on a number of assumptions.
- How long will a user spend between interactions? For typical content pages, that might be 10 seconds; for interactive web apps, perhaps only 5 seconds.
- Divide the number of users by the "think time" to get hits per second - 200 concurrent users with a think time of 10 seconds gives you 20 concurrent users on average.
- Then multiply by a "peak multiplier" - most web sites are relatively silent during the night, but really busy around 7PM. So your average number needs to take account of that - typically, I recommend a peak of between 4 and 10 times.
This gives you a peak page requests per second - this is usually the limiting factor for web applications (though by no means always - streaming video is often constrained by bandwidth, for instance).
If you really want to know "hits", you then need to work through the following:
- How many assets on your page? Images, stylesheets, javascript files etc. - "hit" typically refers to any kind of request, not just the HTML page (or ASPX or PHP or whatever). Most modern web apps include dozens of assets.
- How cacheable are your pages and/or assets? Most images, CSS, JS files etc. should be set to cacheable by the browser.
Multiply the page requests by the number of non-cacheable assets. Add to this the number of visitors multiplied by the number of assets if you want to be super precise.
All of this usually means you have to make lots and lots of assumptions - so the final number is an indicator at best. For scalability measurements, I usually spend more time trying to understand the bottlenecks in the system and observing the system under load.
Well that's impossible to answer without knowing anything about your app or what it does. You need to figure out how many hits per second one user is likely to make when using the app, and multiply by 200.
Incidently, hits/second is not the only metric you need to be concerned with. With 200 concurrent users how much memory overhead will that be? How much disk access or open file handles? How many db reads/writes? How much bandwidth (does the app involve streaming media)? Can it all be handled by one machine? etc etc
I've started working on a basic instant search tool.
This is a workflow draft.
User presses a key
Current value gets passed to the function which will make an Ajax call to a web service
Web service will run a select on a database through LINQ-To-SQL and will retrieve a list of values that match my value. I will achieve this by using SQL Like clause
Web service will return data to the function.
Function will populate relative controls through jQuery.
I have the following concerns/considerations:
Problem: Fast typists: I have typed in this sentence within few seconds. This means that on each key press I will send a request to a database. I may have 10 people doing the same thing. Server may return a list of 5 records, or it may return a list of 1000 records. Also I can hold down a key and this will send few hundred requests to a database - this can potentially slow the whole system down.
Possible solutions:
Timer where I will be able to send a request to database once every 2-4 seconds
Do not return any data unless the value is at least 3 characters long
Return a limited number of rows?
Problem: I'm not sure whether LINQ-to-SQL will cope with the potential load.
Solution: I can use stored procedures, but is there any other feasible alternatives?
I'm interested to hear if anybody else is working on a similar project and what things you have considered before implementing it.
Thank you
When to call the web service
You should only call the web service when the user is interested in suggestions. The user will only type fast if he knows what to type. So while he's typing fast, you don't have to provide suggestions to the user.
When a fast typist pauses for a short time, then he's probably interested in search suggestions. That's when you call the web service to retrieve suggestions.
Slow typists will always benefit from search suggestions, because it can save them time typing in the query. In this case you will always have short pauses between the keystrokes. Again, these short pauses are your queue to retrieve suggestions from the web service.
You can use the setTimeout function to call your web service 500 milliseconds after the user has pressed a key. If the user presses a key, you can reset the timeout using clearTimeout. This will result in a call to the web service only when the user is idle for half a second.
Performance of LINQ-to-SQL
If your query isn't too complex, LINQ-to-SQL will probably perform just fine.
To improve performance, you can limit the number of suggestions to about twenty. Most users aren't interested in thousands of suggestions anyway.
Consider using a full text catalog instead of the like clause if you are searching through blocks of text to find specific keywords. Besides being much faster, it can be configured to recognize multiple forms of the same word (like mouse and mice or leaf and leaves).
To really make your search shine, you can correct many common misspellings using the levenshtein distance to compare the search term to a list of similar terms when no matches are found.