accessing multiple tables concurrently in pytable - concurrency

Question

I'm trying to store temporal embedding in pytable. There are 12 tables and each table has more than 130,000 rows, where each table has two columns (word varchar, embedding float(numpy.arry(300,))). What I want is to calculate cosine similarity for a given against all the word in a given table and repeat this for all 12 tables. Presently I'm doing it sequentially by iterating on each table but it takes around 15 minutes to calculate for all 12 tables.
So my question is, is it possible to read all the table concurrently? I used multithreading but I error
Segmentation fault: 11
Below is my code snippet
def synchronized_open_file():
with lock:
return tb.open_file(FILENAME, mode="r", title="embedding DB")
def synchronized_close_file(self, *args, **kwargs):
with lock:
return self.close(*args, **kwargs)
outqueue = queue.Queue()
for table in list_table :
thread = threading.Thread(target=self.top_n_similar, args=(table,))
thread.start()
threads.append(thread)
try:
for _ in range(len(threads)):
result = outqueue.get()
if isinstance(result, Exception):
raise result
else:
top_n_neighbor_per_period[result[0]] = result[1]
finally:
for thread in threads:
thread.join()
def top_n_similar(table_name):
H5FILE = synchronized_open_file()
do work()
outqueue.put(result)
finally :
synchronized_close_file(H5FILE)

Answer 1

Yes, you can access multiple pytable objects simultaneously. A simple example is provided below that creates 3 tables using a (300,2) numpy record array created with random data. It demonstrates you can access all 3 tables as table objects -OR- as numpy arrays (or both).
I have not done multi-threading with pytables, so can't help with that. I suggest you get your code to work in serial before adding multi-threading. Also, review the pytables docs. I know h5py has specific procedures to use mpi4py for multi-threading. Pytables may have similar requirements.
Code Sample
import tables as tb
import numpy as np
h5f = tb.open_file('SO_55445040.h5',mode='w')
mydtype = np.dtype([('word',float),('varchar',float)])
arr = np.random.rand(300,2)
recarr = np.core.records.array(arr,dtype=mydtype)
h5f.create_table('/', 'table1', obj=recarr )
recarr = np.core.records.array(2.*arr,dtype=mydtype)
h5f.create_table('/', 'table2', obj=recarr )
recarr = np.core.records.array(3.*arr,dtype=mydtype)
h5f.create_table('/', 'table3', obj=recarr )
h5f.close()
h5f = tb.open_file('SO_55445040.h5',mode='r')
# Return Table ojbects:
tb1 = h5f.get_node('/table1')
tb2 = h5f.get_node('/table2')
tb3 = h5f.get_node('/table3')
# Return numpy arrays:
arr1 = h5f.get_node('/table1').read
arr2 = h5f.get_node('/table2').read
arr3 = h5f.get_node('/table3').read
h5f.close()