I'm using Django with a Postgres database from heroku.
The heroku database has a
"connection limit of 20".
I don't understand what this means.
How is a connection defined? Every time a user visits my site I will get some stuff of the database. Is this already a connection because my website needs to connect to the database to get this data?
And what happens if the total number of connections is reached? Does the website still work?
Every query you execute, makes a connection to database.
For example if you want to get some data out of your database, this is what happens in general:
1 - You execute your query in django level.
2 - Django will translate your query to SQL.
3 - Django creates a connection to your database.
4 - Your query will be executed in database.
5 - Django will receive the result.
6 - Django will close the connection.
That limitation means that you can have only 20 tasks working on step 3 to 6 at the same time.
And if you exceed this limitation, your application will throw an error or your application tries till it can make the connection (there will be limit for number of tries and you will get an error if you exceed the tries too) which leads your application to get slower or break on some requests.
You either have to upgrade your server/get more servers or you have to optimize your code to makes fewer queries (Combining queries for example).
Related
In a long-running management command I'd like to have two connections to the same DB. One connection will hold a transaction to lock a certain row (select for update) and the other connection will record some processing info. If the process crashes, a new run of the management command can use the processing info to skip/simplify some processing steps, hence the need to record it in a different connection.
How do I go about creating a 2nd connection in the same thread? My first thought was to add a default2 entry to DATABASES with the same connection info as default and use .using("default2") in one of the queries, but wasn't sure if that would cause issues in Django
I'm running a periodic task using celery in a django-rest application that pulls data from a large Postgres database with multiple tables, the task starts well and pulls some data for about 50 mins and then fails with this error
client_idle_timeout
server closed the connection unexpectedly, This probably means the server terminated abnormally before or while processing the request.
What could be the issue causing this and how can I go about to fix it?
It most likely means that your PostgreSQL has limit on how long transaction can take (idle in transaction), or how long session can be (session timeout).
This is probably happening because of a typical, incorrect way of dealing with databases (I've seen this done even by senior developers) - process creates a database session, and then starts doing some business logic that may take long time to finish, while DB data has been either partially updated or inserted. Code written in such way is doomed to fail because of timeouts enforced by PostgreSQL.
We are working out a couple of performance issues on one of our web sites, and we have noticed that the command "SET TIME ZONE 'America/Chicago'" is being executed so often, that in a 24 hour period, just under 1 hour (or around 4% of total DB CPU resources) is spent running that command.
Note that the "USE_TZ" setting is False, so based on my understanding, everything should be stored as UTC in the database, and only converted in the UI to the local time zone when necessary.
Do you have any ideas on how we can remove this strain on the database server?
For postgres Django always sets timezone: either server's local (when USE_TZ = False) or UTC (When USE_TZ = True). That way django supports "live switching" of settings.USE_TZ for postgreSQL DB backend.
How have you actually determined that this is a bottle-neck?
Usually SET TIME ZONE is only called during creation of connection to DB. Maybe you should use persistent connections by using settings.DATABASES[...]['CONN_MAX_AGE'] = GREATER_THAN_ZERO (docs). That way connections will be reused and you'll have less calls to SET TIME ZONE. But if you use that approach you should also take closer look at your PostgreSQL configuration:
max_connections should be greater than 1+maximum concurrency of wsgi server + max number of simultaneous cron jobs that use django (if you have them) + maximum concurrency of celery workers (if you have them) + any other potential sources of connections to postgres
if you are running cron job to call pg_terminate_backend then make sure that CONN_MAX_AGE is greater than "idle timeout"
if you are running postgres on VPS, then in some cases there might be
limits on number of open sockets)
if you are using something like pgbouncer then it may already be reusing connections
if you are killing server that serves your django project with sigkill (kill -9) then it may leave some unclosed connections to DB (but I'm not sure)
I think this may also happen if you use django.utils.timezone.activate. But I'm not sure of it... This may happen if you manually call it in your code or when you are using middleware to do this
Other possible explaining: the way youre are "profiling" your requests actually shows you the time of whole transaction
Currently, I am working on a project to integrate mysql with the IOCP server to collect sensor data and verify the collected data from the client.
However, there is a situation where mysql misses a connection.
The query itself is a simple query that inserts a single row of records or gets the average value between date intervals.
The data of each sensor flows into the DB at the same time every 5 seconds. When the messages of the sensors come on occasionally or overlap with the message of the client, the connection is disconnected.
lost connection to mysql server during query
In relation to throwing the above message
max_allowed_packet Numbers changed.
interactive_timeout, net_read_timeout, net_write_timeout, wait_timeout
It seems that if there are overlapping queries, an error occurs.
Please let me know if you know the solution.
I had a similar issue in a MySQL server with very simple queries where the number of concurrent queries were high. I had to disable the query cache to solve the issue. You could try disabling the query cache using following statements.
SET GLOBAL query_cache_size = 0;
SET GLOBAL query_cache_type = 0;
Please note that a server restart will enable the query cache again. Please put the configuration in MySQL configuration file if you need to have it preserved.
Can you run below command and check the current timeouts?
SHOW VARIABLES LIKE '%timeout';
You can change the timeout, if needed -
SET GLOBAL <timeout_variable>=<value>;
Coldfusion 2016
Microsoft Server 2012
Oracle 12
ODBC connection
I turned on profiling and monitoring and now I can see that there are requests that are stuck and cannot be terminated by the CF monitor; Some are over 200k seconds.
I know I can increase the number of simultaneous requests but I want to solve the underlying problem. As I read the stack traces of these “zombie requests” they are getting stuck on and some are in but some are not. I ran the query in my oracle client and they resolve instantly.
Is there a way to terminate these requests or prevent this from happening at all?
EDIT: The server monitor does not treat these requests as slow or hung, the alerts are not triggering for any of these. Honestly, they should have be going off constantly considering how many of these there are.
Also, the execution time is a mere .003 seconds so what happened? Why doesn't ColdFusion know this?
An example of a "zombie"
The active query that is stuck
We have a similar situation with a different database engine - redbrick, which runs on a unix server. We solved it as follows.
We set up a cron job on the database server to run every 5 minutes. This job uses a combination of unix and awk commands.
This job runs a query against the system table that looks for queries that have been running for more than 120 seconds, where the database account is the one used by ColdFusion. Records are outputted to a file. Something like this:
print "alter system cancel user command userName process " $1 ";"
$1 comes from the query and is the process Id we want to stop.
Then we run the file, which executes all those alter system commands.
With a different database engine, and possible different OS for the database server, the details would be different, but the approach should work.
Edit Starts Here
To prevent recurrence, look at the pages that call the ones with the long running queries. If impatient users are able to repeatedly click something because nothing is happening, do something about that. You can use javascript to make the link/button go away. Alternatively, you can go to an intermediate page with a display for the user and something that carries them through to the real page.