We have an superset installation on K8.
We currently have 6 workers and 3 deployment pods and we are working against an Oracle Database.
We test the queries on our database, and we are trying to understand why our UI keeps looping loading graphs, but if we ask the CSV data, in gets downloaded.
We don´t understand why UI keeps "loading" every chart, since the data is loaded. Pods are at 20% CPU and almost the same for memory.
Is there anything else we have to check out to understand why this keeps happening?
We are using Superset 1.5.1
Related
I built a shiny app which works with data that needs frequent update. The app doesn't change, just the data. At the beginning, i used to update my data locally, and republish my app every time the data had been updated, which i quickly found quite annoying.
I then started to store my datasets online (on arcgis online, for various reasons) so that i wouldn't need to republish my shiny app anymore, just need to handle the process of data updates.
The problem is that my app is quite slow as the datasets are very big.
Now i would like to transform my datasets as api's so that the requests coming from shiny could be more targeted.
But i don't know really know to do that. Handling the update of datasets on arcgis online through an R script was fine. But updating the same datasets as hosted feature service, i can't make it work.
Would anyone have an idea?
Or more general question, if moving away from my Arcgis online storage, what would be the best way to store data that needs frequent updates and that feed shiny apps?
You can look into caching data using 'Pins' package
i am using postgres paired with django(python) and application is crawling internet for specific kind of data. As the crawlers find anything of their use they write it to the database. Now as the speed of crawlers is high and they are querying database by get_or_create(in django which checks for if the data is already in the database or not if it is not present then it makes a new row of it) All the cores on my machine are engaged to almost 99%. In that case when i trace the process,the postgres is skipping the write command for some or many instances.What can be the reason? Any recommendations in terms of change in architecture?
traced the crawler procedure manually and the process was printing the data it found but was not added to the postgres. That confirmed the data was getting lost.
In a Python/Django application, is retrieving a value stored in redis slower than retrieving one stored in the request.session dictionary?
Background:
I have a Django app where I use DB-based sessions. I.e., instead of using django.contrib.sessions, I used this nifty little 3rd party library.
I recently ran a benchmark whereby I saved a test value in a local redis instance via the redis-py wrapper (i.e. my_server.set('test','1')). I saved the same test value in request.session['test'].
I then retrieved the test value from each, and compared the time taken. request.session out performed redis by a factor exceeding 2x in this scenario.
Problem:
The application is not distributed in any way, everything is shared and happens on the same machine - very vanilla set up.
The result appears counter-intuitive to me. Why? Because my sessions are DB based, and I thought redis would be faster than whatever Django has to offer. Clearly, I am wrong.
Can an expert explain what's actually going on here? Maybe the python wrapper on redis' core API is slowing things down?
In case you need more information, or are skeptical about how I ran the benchmark, please do ask.
P.s. I simply put the two competing ways in a for loop for 100K iterations and measured the time taken to complete.
The session is stored as a single blob, not as individual keys. It has almost certainly already been loaded and decoded by the time you get into your view, most likely by the auth middleware. Once it is loaded it is stored locally as a dictionary, which is all that your timing tests will measure.
I have a Sitecore website deployed in multi server environment. When I make some changes to Sitecore items sometimes they are shown correctly, but sometimes it shows old data.
I understand that sitecore caches items, but it sometimes showing wrong data and sometime its fine. If its caching it should always be same data at least.
For example:
Sitecore.Globalization.Translate.TextByDomain("MyDictionary", "Category");
Sometimes it returns correct data sometimes it shows wrong data i.e. the one before I changed to item.
I am using Sitecore 8.0
Items get cached on the individual servers in memory, and these are not cleared unless you activate event queues. Further content might be cached in the output cache, which needs to be cleared after you publish.
Here is a guide on how to activate event queues and here is also a good description
Here is how to make your sites clear output cache after publish
Thanks Jens for your help. The links really helped me with my understanding of Sitecore farm.
But the issue turned out to be rather silly. For some reason on one content delivery server Application Pool account didn't had permission on the virtual directory.
We are using Sitecore 8 with Solrcloud and one of the things we have noticed is that anytime several hundred articles are published the web database indexing lags behind. There never seems to be an issue with the master database indexes.
For web database, the strategy being used is publishasync and for master database indexes it is syncmaster.
Are there any configuration changes that can be made to speed up the web indexing. There aren't really that many items that Sitecore needs to get backed up.
We have taken a few steps to rectify the web database situation -
1. Set all CD indexing to manual.
2. Turned off any unnecessary indexes.
Any help is appreciated.
Thanks