I would like to implement a MySQL style named lock like in http://arr.gr/blog/2016/05/mysql-named-locks-in-python-context-managers/ :
Using sqlalchemy Session, I directly execute a GET_LOCK:
lock = session.execute("SELECT GET_LOCK('TEST', 5)")
Then if my lock is OK, I do what I want, especially some stuff with my database using the session. After that I release the lock:
session.execute("SELECT RELEASE_LOCK('TEST)")
My question is the following:
How can I be sure that the session connection to the database for the release is the same as the one at the beginning ?
The sqlalchemy's documentation says:
When the transactional state is completed after a rollback or commit,
the Session releases all Transaction and Connection resources [...]
http://docs.sqlalchemy.org/en/latest/orm/session_transaction.html
I'm in the context of a web application, so every time the session is committed, or if the session is rollbacked, the next queries aren't guaranteed to be on the same connection. For most of my operations, that isn't a problem, but in the case of a MySQL named lock, 'GET_LOCK' and 'RELEASE_LOCK' have to be requested on the same connection.
The only way I found is to have a specific session for the locking : it will keep its connection only for the lock. But is there a way not to use a connection just for this purpose ?
Edit 2017-06-22:
This topic is about the same subject: SQLAlchemy session and connection relationship. But if a scoped_session is used:
Regarding the extract from the documentation above, after a commit or a rollback, the connection for future queries with this session may be different, right ?
A unique session will be used in the thread ?
If that could help someone else, finally I used a scoped_session with a bind when getting the session for the first time in a request:
# Initialization:
Session = scoped_session(sessionmaker(bind=engine))
# ...
# Before processing a request:
connection = engine.connect()
Session(bind=connection)
# ...
# After the processing
Session.remove()
connection.close()
With this configuration, if during my request processing I use GET_LOCK/RELEASE_LOCK, they will apply on the same connection.
Related
In a long-running management command I'd like to have two connections to the same DB. One connection will hold a transaction to lock a certain row (select for update) and the other connection will record some processing info. If the process crashes, a new run of the management command can use the processing info to skip/simplify some processing steps, hence the need to record it in a different connection.
How do I go about creating a 2nd connection in the same thread? My first thought was to add a default2 entry to DATABASES with the same connection info as default and use .using("default2") in one of the queries, but wasn't sure if that would cause issues in Django
i want to know when is dbConnection opened
for exampel
as follow logic
Does wait db Connection to external_api_call
or not?
#transaction.commit_manually
def do_something():
# 1. now db connection open?
# api_call
response= requests.get(~~~~)
# 2. or db connection opened here?
aa= User.objects.get(id=1)
Django internally maintains DB connection. It uses persistent connection approach. But when the connection gets closed after a certain time and you hit a new database query, new connection is made by django and kept it in memory. And the same connection is re-used by multiple request until it expires.
You can read more about it here https://docs.djangoproject.com/en/4.0/ref/databases/
I have a web service that is made using spring, hibernate and c3p0. I also have a service wide cache(which has the results of requests ever made to the service) which can be used to return results when the service isn't able to return(due to whatever reason). The cache might return stale results when the database is out but that's ok.
I recently faced a database outage and my service came to a crashing halt.
I want the clients of my service to survive database outages happening ever again in future.
For that, I need my service to:
Handle new incoming requests like this: quickly say that the database is down and throw some exception(fast-fail).
Requests already being processed: Don't last longer than x seconds. How do I make the thread handling the request be interrupted somehow.
Cache the whole database in memory for read-only purposes(Is this insane?).
There are some observations that I made:
If there is one or more connection(s) with status ESTABLISHED, then an attempt to checkout a new connection is not made. Seems like any one connection with status ESTABLISED is handed over to the thread receiving the request. Now, this thread just hangs till the time the database comes back up.
I would want to make this request fast-fail by knowing before handling over a connection to a thread whether db is up or not. If no, the service should throw exception instead of hanging up.
If there's no connection with status ESTABLISHED, then the request fails in 10 secs with the exception that "Could not checkout a new connection". This is due to my checkout timeout being set for 10s.
If the service was processing some request, now the db goes and then the service makes a call to db, the thread making the call to db gets stuck forever. It resumes execution only after the db comes back.
I would like to interrupt the thread after say x seconds whether or not it was able to complete the request.
Are there ways to accomplish what I seek?
Thanks in advance.
I am trying to put some of the message system to redis. I have a question regarding the connection management towards redis from django. Below is taken from quora:
When talking to Redis from Django (or indeed any other web framework, I imagine) an interesting challenge is deciding when to connect and disconnect.
If you make a new connection for every query to Redis, that's a ton of unnecessary overhead considering a single page request might make hundreds of Redis requests.
If you keep one connection open in the thread / process, you end up with loads of unclosed connections which can lead to problems. I've also seen the Redis client library throw the occasional timeout error, which is obviously bad.
The best result I've had has been from opening a single Redis connection at the start of the request, then closing it at the end - which can be achieved with Django middleware. It feels a bit dirty though having to add a piece of middleware just to get this behaviour.
Does anybody had a chance to create such redis middleware , I am always in favor of not reinventing the wheel but didn't find anything on google related to this topic.
I implemented the middleware :
import redis
from redis_sessions import settings
# Avoid new redis connection on each request
if settings.SESSION_REDIS_URL is not None:
redis_server = redis.StrictRedis.from_url(settings.SESSION_REDIS_URL)
elif settings.SESSION_REDIS_UNIX_DOMAIN_SOCKET_PATH is None:
redis_server = redis.StrictRedis(
host=settings.SESSION_REDIS_HOST,
port=settings.SESSION_REDIS_PORT,
db=settings.SESSION_REDIS_DB,
password=settings.SESSION_REDIS_PASSWORD
)
else:
redis_server = redis.StrictRedis(
unix_socket_path=settings.SESSION_REDIS_UNIX_DOMAIN_SOCKET_PATH,
db=settings.SESSION_REDIS_DB,
password=settings.SESSION_REDIS_PASSWORD,
)
class ReddisMiddleWare(object):
def process_request(self,request):
request.redisserver = redis_server
Then in the view I am just using request.redisserver.get(key) .
I'm running a system with a few workers that's taking jobs from a message queue, all using Djangos ORM.
In one case I'm actually passing a message along from one worker to another in another queue.
It works like this:
Worker1 in queue1 creates an object (MySQL INSERT) and pushes a message to queue2
Worker2 accepts the new message in queue2 and retrieves the object (MySQL SELECT), using Djangos objects.get(pk=object_id)
This works for the first message. But in the second message worker 2 always fails on that it can't find object with id object_id (with Django exception DoesNotExist).
This works seamlessly in my local setup with Django 1.2.3 and MySQL 5.1.66, the problem occurs only in my test environment which runs Django 1.3.1 and MySQL 5.5.29.
If I restart worker2 every time before worker1 pushes a message, it works fine. This makes me believe there's some kind of caching going on.
Is there any caching involved in Django's objects.get() that differs between these versions? If that's the case, can I clear it in some way?
The issue is likely related to the use of MySQL transactions. On the sender's site, the transaction must be committed to the database before notifying the receiver of an item to read. On the receiver's side, the transaction level used for a session must be set such that the new data becomes visible in the session after the sender's commit.
By default, MySQL uses the REPEATABLE READ isolation level. This poses problems where there are more than one process reading/writing to the database. One possible solution is to set the isolation level in the Django settings.py file using a DATABASES option like the following:
'OPTIONS': {'init_command': 'SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED'},
Note however that changing the transaction isolation level may have other side effects, especially when using statement based replication.
The following links provide more useful information:
How do I force Django to ignore any caches and reload data?
Django ticket#13906