multithreads in python over a large multifasta - python-2.7

Question

my problem is very simple but I can not figure out how to solve it.
I have a list of about of one million sequences and each of them need to be aligned to a sequencing adapter. I`m thinking to do the alignment in python using pairwise2 tools from Biopython. I would like to use this tool because i need to collect all the alignment scores, do some math and select the sequences based on the math. If I run the code below it works but it is slow because only one alignment is run per time.
def align_call(record, adapter):
score = pairwise2.align.localms(record.seq, adapter.seq, 1, -3, -1, -0.99, one_alignment_only=1, score_only=1)
print record.id + " " + record.seq + " " + adapter.id + " " + str(score)
#results.append(line)
return
if __name__ == '__main__':
fastaSeq = argv[1]
threads = argv[2]
fastaAdapt = argv[3]
listSeq = []
adpt = list(SeqIO.parse(fastaAdapt, "fasta"))
for record in SeqIO.parse(fastaSeq, "fasta"):
align_call(record, adpt[0])
Therefore, I was thinking to change the code and use multithreading or multiprocess to speed up process by sending n number of parallel jobs based on the number of threads that the computer has. So i came up to something like this:
results = []
def align_call(record, adapter):
score = pairwise2.align.localms(record.seq, adapter.seq, 1, -3, -1, -0.99, one_alignment_only=1, score_only=1)
line = record.id + " " + record.seq + " " + adapter.id + " " + str(score)
results.append(line)
return results
if __name__ == '__main__':
fastaSeq = argv[1]
threads = argv[2]
fastaAdapt = argv[3]
listSeq = []
adpt = list(SeqIO.parse(fastaAdapt, "fasta"))
for record in SeqIO.parse(fastaSeq, "fasta"):
pool = Pool(processes=1)
result = pool.apply_async(align_call, args= (record, adpt[0]))
print result.get()
The script works but I can not modulate how many sequence need to be send each time and when I get a lot of them, I run out of cores and memory.
Any idea on how I can do this? Suggestions?
I tried by implementing Queue but it did not work
Thanks
Luigi

Answer 1

You might want to look into vsearch (or usearch)
It is pretty fast! and supports multi-threading
https://github.com/torognes/vsearch
vsearch --usearch_global big.fasta -db adapter_seq.fasta --uc outputfile.txt --id 0.8 --iddef 0 --threads 6
--id is the max % difference from the target sequence that you will allow (80 % in this case)
--iddef 0 is the scoring method 0=identity to based on shortest sequence, 2=strip terminal gaps...)
You can then read in this outputfile.txt and grab the alignment scores/matches/gaps/alignment length query name...
for each sequence.
With the desired query names collected, you can use these to pull the relevant sequences from the original fasta file
If you just want the sequences that were greater than x % match to the adapter, you can use --matched, instead of --uc, which will give you a fasta file of the sequences that matched above the --id threshold.
Alternatively, --samout will give you a sam file, with the seq name, alignment length, cigar code for the alignment as well as the query sequence.
https://github.com/torognes/vsearch/releases/download/v2.7.1/vsearch_manual.pdf has the full set of output options and commands

Answer 2

I think the pool should be created before the SeqIO loop. And you need to use a lock or callback to ensure the output is in right order.