I am running django app (wagtail) in kubernetes cluster along with redis. These two pieces are connected by django-redis. This is how my backend configuration look
{
"BACKEND":"django_redis.cache.RedisCache",
"LOCATION":"redis://redis-service:6379/0",
"OPTIONS":{
"CLIENT_CLASS":"django_redis.client.DefaultClient",
"CONNECTION_POOL_KWARGS":{
"max_connections":1000
}
}
}
This works just fine. I can see keys getting created in redis and app is also blazing fast thanks to redis.
Now real issue is every once in a while app slows down for some time. Upong investigation we found out the real issue is a key of size ~90 MB is created in redis. to process this key app takes some time and slows down.
To put a comparison other keys are less than 1 MB always just 1 key gets randomly created on one type of endpoint but not always.
I tried to check the contents of the key after unpickling and it is normal amounts of data just like other keys. but running
redis-cli --bigkeys
gives output something like this
Biggest string found '":1:views.decorators.cache.cache_page.XXXXXXX.GET.9b37c27c4fc5f19eb59d8e8ba7e1999e.83bf0aec533965b875782854d37402b7.en-us.UTC"' has 90709641 bytes
has someone seen similar issue? Anyway to debug the root cause.
django-redis version "==4.12.1"
wagtail version "==2.11.1"
django version "==3.1.3"
Related
I'm using Objectify to access my GCP Datastore set of Entites. I have a full list of around 22000 items that I need to load into the frontend:
List<Record> recs = ofy().load().type(Record.class).order("-sync").list();
The number of records has recently increased and I get an error from the backend:
com.google.apphosting.runtime.HardDeadlineExceededError: This request (00000185caff7b0c) started at 2023/01/19 17:06:58.956 UTC and was still executing at 2023/01/19 17:08:02.545 UTC.
I thought that the move to Cloud Firestore in Datastore mode last year would have fixed this problem.
My only solution is to break down the load() into batches using 2 or 3 calls to my Ofy Service.
Is there a better way to grab all these Entities in one go?
Thanks
Tim
I'm using the NodeJS demo code from here: https://questdb.io/docs/develop/insert-data/ to insert data into QuestDB like this:
setInterval(() => {
run();
}, 3000);
(2 Docker containers on a bridge network)
And I have a browser window open to run
select count(*) from 'trades'
However, the query only runs once and then returns the same result. If I restart the docker containers, the query returns an updated value so I assume the values are successfully getting into the database but they are not reflected in the Postgres queries. I see the same behavior when I use the pg client in Node.
Any explanation or theory that would help me root cause this?
In QDB data is not visible to queries until it's committed .
In case of ILP receiver commits don't happen after each row or even on disconnect . Instead QDB uses a number of properties to determine when to commit efficiently .
In this case the easiest way to reduce insert to commit delay would be to reduce cairo.max.uncommitted.rows to e.g. 10 in conf/server.conf (plus instance/container restart) and then insert 10+ records .
You'll find more details on ILP commits at
https://questdb.io/docs/reference/api/ilp/tcp-receiver/#commit-strategy
We currently have two ColdFusion 10 dedicated servers which we are migrating to a single VPS server. We have many scheduled tasks on each. I have taken each of the neo-cron.xml files and copied the var XML elements, from within the struct type='coldfusion.server.ConfigMap' XML element, and pasted them within that element in the neo-cron.xml file on the new server. Afterward I restarted the ColdFusion service, log into cf admin, and the tasks all show as expected.
My problem is, when I try to update any of the tasks I get the following error when saving:
An error occured scheduling the task. Unable to store Job :
'SERVERSCHEDULETASK#$%^DEFAULT.job_MAKE CATALOGS (SITE CONTROL)',
because one already exists with this identification
Also, when I try to delete a task it tells me a task with that name does not exist. So it seems to me that the task information must also be stored elsewhere. So there when I try to update a task, the record doesn't exist in the secondary location so it tries to add it new to the neo-cron.xml file, which causes an error because it already exists. And when trying to delete, it doesn't exist in the secondary location so it says a task with that name does not exist. That is just a guess though.
Any ideas how I can get this to work without manually re-creating dozens of tasks? From what I've read this should work, but I need to be able to edit the tasks.
Thank you.
After a lot of hair-pulling I was able to figure out the problem. It all boiled down to having parentheses in the scheduled task names. This was causing both the "Unable to store Job : 'SERVERSCHEDULETASK#$%^DEFAULT.job_MAKE CATALOGS (SITE CONTROL)', because one already exists with this identification" error and also causing me to be unable to delete jobs. I believe it has something to do with encoding the parentheses because the actual neo-cron.xml name attribute of the var element encodes the name like so:
serverscheduletask#$%^default#$%^MAKE CATALOGS (SITE CONTROL)
Note that this anomaly did not exist on ColdFusion 10, Update 10, but does exist on Update 13. I'm not sure which update broke it, but there you go.
You will have to copy the neo-cron.xml from C:\ColdFusion10\\lib of one server to another. After that restart the server to make the changes effective. Login to the CF Admin and check the functionality.
This should work.
Note:- Please take a backup of the existing neo-cron.xml, before making the changes.
I am working on doing some simple analytics on a Django webstite (v1.4.1). Seeing as this data will be gathered on pretty much every server request, I figured the right way to do this would be with a piece of custom middleware.
One important metric for the site is how often given images are accessed. Since each image is its own object, I thought about using django-hitcount, but figured that was unnecessary for what I was trying to do. If it proves easier, I may use it though.
The current conundrum I face is that I don't want to query the database and look for a given object for every HttpRequest that occurs. Instead, I would like to wait until a successful response (indicated by an HttpResponse.status of 200 or whatever), and then query the server and update a hit field for the corresponding image. The reason the only way to access the path of the image is in process_request, while the only way to access the status code is in process_response.
So, what do I do? Is it as simple as creating a class variable that can hold the path and then lookup the file once the response code of 200 is returned, or should I just use django-hitcount?
Thanks for your help
Set up a cron task to parse your Apache/Nginx/whatever access logs on a regular basis, perhaps with something like pylogsparser.
You could use memcache to store the counters and then periodically persist them to the database. There are risks that memcache will evict the value before it's been persisted but this could be acceptable to you.
This article provides more information and highlights a risk arising when using hosted memcache with keys distributed over multiple servers. http://bjk5.com/post/36567537399/dangers-of-using-memcache-counters-for-a-b-tests
How can I programmatically dump/query Launch Services database in MacOS (i.e. analog of command lsregister -dump)?
EDIT: I want to get set of associations UTI -> Bundle_IDs. Using LSCopyAllRoleHandlersForContentType - does not always work, here a similar trouble, therefore concluded that the best working method - parsing the output of "lsregister -dump", but the location of lsregister changes from version to version.