Dictionary with safety between readers and writers deadlocking - python-2.7

Question

I'm trying to implement a concurrent dictionary in Python - more specifically, the dictionary would be used by two threads, one that would use its clear and update methods, and the other which would access its values directly (i.e., by its __getitem__ method). The implementation is below:
from threading import Lock, current_thread
class ThreadSafeDict(dict):
def __init__(self, *args, **kwargs):
self._lock = Lock()
super(ThreadSafeDict, self).__init__(*args, **kwargs)
def clear(self, *args, **kwargs):
print("thread {} acquiring clear lock".format(current_thread().ident))
self._lock.acquire()
print("thread {} acquired clear lock".format(current_thread().ident))
super(ThreadSafeDict, self).clear(*args, **kwargs)
print("thread {} releasing clear lock".format(current_thread().ident))
self._lock.release()
print("thread {} released clear lock".format(current_thread().ident))
def __getitem__(self, *args, **kwargs):
print("thread {} acquiring getitem lock".format(current_thread().ident))
self._lock.acquire()
print("thread {} acquired getitem lock".format(current_thread().ident))
val = super(ThreadSafeDict, self).__getitem__(*args, **kwargs)
print("thread {} releasing getitem lock".format(current_thread().ident))
self._lock.release()
print("thread {} released getitem lock".format(current_thread().ident))
return val
def update(self, *args, **kwargs):
print("thread {} acquiring update lock".format(current_thread().ident))
self._lock.acquire()
print("thread {} acquiring update lock".format(current_thread().ident))
super(ThreadSafeDict, self).update(*args, **kwargs)
print("thread {} releasing update lock".format(current_thread().ident))
self._lock.release()
print("thread {} released update lock".format(current_thread().ident))
I'm testing the implementation with this script:
import threading
import random
import time
from threadsafedict import ThreadSafeDict
def reader(tsd):
while True:
try:
val = tsd[1]
except KeyError:
pass
interval = random.random() / 2
time.sleep(interval)
def writer(tsd):
while True:
tsd.clear()
interval = random.random() / 2
time.sleep(interval)
tsd.update({1: 'success'})
def main():
tsd = ThreadSafeDict()
w_worker = threading.Thread(target=writer, args=(tsd,))
r_worker = threading.Thread(target=reader, args=(tsd,))
w_worker.start()
r_worker.start()
w_worker.join()
r_worker.join()
if __name__ == '__main__':
main()
Sample output:
thread 140536098629376 acquiring clear lock
thread 140536098629376 acquired clear lock
thread 140536098629376 releasing clear lock
thread 140536098629376 released clear lock
thread 140536090236672 acquiring getitem lock
thread 140536090236672 acquired getitem lock
thread 140536090236672 acquiring getitem lock
thread 140536098629376 acquiring update lock
What am I doing wrong?
(I realize this concurrency would already be safe in CPython, but I'm trying to be implementation-agnostic)

Answer 1

The problem is that when the super().__getitem__() call in your ThreadSafeDict.__getitem()__ method fails to find an item with the given key, it raises KeyError which causes the remainder of your __getitem__() method to be skipped. That means that the lock will not be released, and any later calls to any of your methods will be blocked forever waiting to obtain a lock that will never be unlocked.
You can see that this is happening by the absence of 'releasing' and 'released' messages after the 'acquired getitem lock' message, which is immediately followed in that excerpt by another attempt to acquire the lock by the read thread. In your test code, your read thread will always hit this condition if it runs in the interval after a clear() has been performed but before an update() has been performed by the write thread.
To fix, catch the KeyError exception in your __getitem__() method, then release the lock, then re-raise the exception. The 'try/finally' construct provides a very straightforward way to do this; in fact this is the perfect situation for the use of 'finally'.
Or you could check that the desired key exists after acquiring the lock and before before calling super().__getitem__(), although that will hurt performance slightly if the usual expectation is that the key will exist.
BTW, it's not a great idea to have your ThreadSafeDict inherit from the dict class. This causes ThreadSafeDict to inherit all of the dict methods (for instance, __setitem__()) and any of those methods that you haven't overridden would bypass your lock if someone used them. If you aren't prepared to override all of those methods then it would be safer to have the underlying dict be an instance member of your class.