We have recently switched over to the new Google People API on our sync platform. We chose to only do this for newly set up connections for now so to determine the impact.
Functionally all seems to work fine, so no actual bugs to be raised.
However we noticed that the performance of the People API seems to be much lower compared to the Contacts API.
More specifically we saw the following differences when summarising all requests of today:
Creation response time: 571 ms (versus 201 ms on the old Contacts API)
Update response time: 735 ms (versus 168 ms on the old Contacts API)
There are also differences in the other operations, but especially the difference in the response time of persists is clear. Even more, the batch persist operation on the old Contacts API was going faster on average (we use a batch size of up to 30 records) than a single create/update operation on the People API.
Is this something that can be improved in the near future?
It is not really blocking our end users, but it does lead to degraded sync performance. Especially users with large contact sets will experience significantly longer syncing times (as we are syncing all historical contact data).
If you like, I can provide you with some example API requests and response calls.
Related
Any recommendation on how to make superset faster?
Cache seems to load full data from the cache, I thought it load only old data from the cache, and real-time data from the database, isn't it like this?
What about some parallel processing?
This answer is valid as of Superset 0.37.0.
At the moment, dashboard performance is affected by a few different factors. I'll enumerate them below along with methods to improve performance:
Database concurrency limits can have an impact on dashboard performance. Dashboards load their information in parallel via concurrent web requests. Make sure that the database user provided allows enough concurrency that queries aren't being queued at the database layer.
Cache performance your caching layer should be able to return multiple results, if not in parallel, extremely quickly. We've had success leveraging S3 for our cache.
Cache hit percentage Superset will hit the cache only for queries that exactly match one that has been run recently. Otherwise the full query will fall through to the underlying analytical DB (Druid in this case). You can reduce the query load on Druid by using a less granular resolution on your dashboard - if it's possible to have it update less frequently, say a couple of times a day rather than in real-time, this can hit cache for all requests other than the first request in the new period under consideration.
Python Web Process Concurrency Limits make sure that your web application server can handle enough parallel requests. The browser will request multiple charts' data at the same time, and the system will need to be able to handle these requests in parallel.
Chart Query Performance As data is frequently requested, especially for real-time data from a database like Druid, optimizing the queries run by the charts can be very useful. I'd take a look at any virtual datasources that are being leveraged to see if they can be materialized or made more efficient.
Web browser concurrent request limits By default most web browsers limit the number of concurrent requests that can be made to the same FQDN. If you have more than 6 charts on the same dashboard, it can be helpful to balance requests across multiple FQDNs running Superset to get around this browser limitation. There's more information on the approach to that in the issue history on Github, but Superset does support this type of configuration.
The community is very interested in improving performance over time, and as such there have been recommendations to move all analytical queries to Celery as well as making other architectural changes to improve performance. I hope this description helps and that something in here will help you track down the issue!
As per AWS Dynamodb pricing
it allows 25 read capacity units which translates to 50 GetItem requests per second ( with eventual consistency and each item being less than 4kb).
Free Tier*
As part of AWS’s Free Tier, AWS customers can get started with Amazon
DynamoDB for free. DynamoDB customers get 25 GB of free storage, as
well as up to 25 write capacity units and 25 read capacity units of
ongoing throughput capacity (enough throughput to handle up to 200
million requests per month) and 2.5 million read requests from
DynamoDB Streams for free.
How does this translate to online web site? If more than 50 users make GET calls at the same time, do the requests gets throttled ? and eventually 400 response is returned? Does it mean free tier practical limits are, during bursts if 100 users log in the site and make GET calls at same time, we may see 400 responses from DynamoDB. Is this valid conclusion?
I understand that there may hot be 100 requests per every second. But if the site has active users of more than 200 at any time, does Dydnamodb free tier still work?
First, read up on DynamoDB burst capacity. Your DynamoDB tables should be able to sustain short bursts of higher throughput without throttling.
Second, the question of how database throughput capacity will "translate to online web site" is way too broad to answer directly. It entirely depends on how many database calls your web application makes per web request. Your question sounds like you are assuming every page load on your website results in exactly one DynamoDB request. That seems like an extremely unrealistic assumption.
You should be using a CDN to prevent as many requests as possible from even hitting your web server. You should be using a cache like Redis to prevent as many data lookups from hitting your database as possible. Then you should perform some benchmarking to determine the database throughput you are going to need, and evaluate that against the DynamoDB free tier.
How often does your web app content change? Do users submit changes to your content and if so, how often? These questions will directly affect how well your site can be cached, which will directly affect how often your app will have to go back to the database to get the latest content.
As for what response is returned to the user in the scenario where the DynamoDB request is throttled, that would be entirely dependent how you capture and handle errors in your web app.
We run a website which heavily relies on the Amazon Product Advertising API (APAA). What happens is that when we experience a sudden spike in users it happens that we hit the rate-limit and all functions relying on the APAA shut down for a while. What can we do so that doesn't happen?
So, obviously we have some basic caching in place, but the APAA doesn't allow us to cache data for a very long time, and APAA queries can vary a lot so there may not be any cached data at all to query.
I think that your only option is to retry the API calls until they work — but do so in a smart way. Unfortunately, that's what everybody that gets throttled does and AWS expects people to handle that themselves.
You can implement an exponential backoff and add jitter to prevent cluster calls. AWS has a great blog post about solutions for this kind of problem: https://www.awsarchitectureblog.com/2015/03/backoff.html
I am using django and tastypie for REST API.
For profiling, I am using django-silk and below is a summary of requests:
How do I profile the complete flow? Time taken except for database queries is (382 - 147) ms on average. How do I figure out the bottleneck and optimize/scale? I did use #silk_profile() for the get_object_list method for this resource, but even this method doesn't seem to be bottleneck.
I used caching for decreasing response time, but that didn't help much, what are the other options?
When testing using loader.io, the peak the server can handle is 1000 requests per 30 secs (which seems very low). Other than caching (which I already tried) what might help?
Here's a bunch of suggestions:
bring the query per request at least below 5 per request (34 per request is really bad)
install django toolbar and have a look where the time is spent
use gunicorn or uwsgi behind a reverse proxy (NGINX)
You have too much queries, even if they are relatively fast you spend
some time to reach database etc. Also if you have external cache
storage (for example, redis) it could take some time to connect
there.
To investigate slow parts of the code you have two options:
Use a profiler - profiling at local PC could make no sense if you have distributed system deployed to several machines
Add tracing points to your code that will record some message and current time (something like https://gist.github.com/dbf256/0f1d5d7d2c9aa70bce89). Deploy this patched code and test it with your load-testing tool and check logs.
I'm building a site with django that lets users move content around between a bunch of photo services. As you can imagine the application does a lot of api hits.
for example: user connects picasa, flickr, photobucket, and facebook to their account. Now we need to pull content from 4 different apis to keep this users data up to date.
right now I have a a function that updates each api and I run them all simultaneously via threading. (all the api's that are not enabled return false on the second line, no it's not much overhead to run them all).
Here is my question:
What is the best strategy for keeping content up to date using these APIs?
I have two ideas that might work:
Update the apis periodically (like a cron job) and whatever we have at the time is what the user gets.
benefits:
It's easy and simple to implement.
We'll always have pretty good data when a user loads their first page.
pitfalls:
we have to do api hits all the time for users that are not active, which wastes a lot of bandwidth
It will probably make the api providers unhappy
Trigger the updates when the user logs in (on a pageload)
benefits:
we save a bunch of bandwidth and run less risk of pissing off the api providers
doesn't require NEARLY the amount of resources on our servers
pitfalls:
we either have to do the update asynchronously (and won't have
anything on first login) or...
the first page will take a very long time to load because we're
getting all the api data (I've measured 26 seconds this way)
edit: the design is very light, the design has only two images, an external css file, and two external javascript files.
Also, the 26 seconds number comes from the firebug network monitor running on a machine which was on the same LAN as the server
Personally, I would opt for the second method you mention. The first time you log in, you can query each of the services asynchronously, showing the user some kind of activity/status bar while the processes are running. You can then populate the page as you get the results back from each of the services.
You can then cache the results of those calls per user so that you don't have to call the apis each time.
That lightens the load on your servers, loads your page fast, and provides the user with some indication of activity (along with incrimental updates to the page as their content loads). I think those add up to the best User Experience you can provide.