Perl program to mimic restriction enzymes using references, hash tables and subs - regex

I'm a student in an intro Perl class. I'm looking for suggestions on how to approach an assignment. My professor encourages forums. The assignment is:
Write a Perl program that will take two files from the command line, an enzyme file and a DNA file. Read the file with restriction enzymes and apply the restriction enzymes to the DNA file.
The output will be fragments of DNA arranged in the order they occur in the dna file. The name of the output files should be constructed by appending the name of the restriction enzyme to the name of the DNA file, with an underscore between them.
For example, if the enzyme is EcoRI and the DNA file is named BC161026, the output file should be named BC161026_EcoRI.
My approach is to create a main program and two subs as follows:
Main:
Not sure how to tie my subs together?
Sub program $DNA:
Take a DNA file and remove any new lines to make a single string
Sub program Enzymes:
Read and store the lines from the enzyme file which is from the command line
Parse the file in a way that it separates the enzyme acronym from the position of the cut.
Store the position of the cut as a regular expression in a hash table
Store the name of the acronym in a hash table
Note on enzyme file format:
The enzyme file follows a format known as Staden. Examples:
AatI/AGG'CCT//
AatII/GACGT'C//
AbsI/CC'TCGAGG//
The enzyme acronym consists of the characters before the first slash (AatI, in the first example. The recognition sequence is everything between the first and second slash (AGG'CCT, again, in the first example). The cut point is denoted by an apostrophe in the recognition sequence
There are standard abbreviations for dna within enzymes as follows:
R = G or A
B = not A (C or G or T)
etc...
Along with a recommendation for a main chunk, do you see any missing pieces that I've omitted? Can you recommend specific tools that you think would be useful in patching this program together?
Example input enzyme: TryII/RRR'TTT//
Example string to read: CCCCCCGGGTTTCCCCCCCCCCCCAAATTTCCCCCCCCCCCCAGATTTCCCCCCCCCCGAGTTTCCCCC
The output should be:
CCCCCCGGG
TTTCCCCCCCCCCCCAAA
TTTCCCCCCCCCCCCAGA
TTTCCCCCCCCCCGAG
TTTCCCCC

Ok, I know I shouldn't just do your homework, but there were some fun tricks to this one, so I played with it. Learn from this, not just copy. I didn't comment very well, so if there is something you don't understand, please ask. There is some slight magic in this that if you haven't covered it in your class, your prof will know, so be sure you understand.
#!/usr/bin/env perl
use strict;
use warnings;
use Getopt::Long;
my ($enzyme_file, $dna_file);
my $write_output = 0;
my $verbose = 0;
my $help = 0;
GetOptions(
'enzyme=s' => \$enzyme_file,
'dna=s' => \$dna_file,
'output' => \$write_output,
'verbose' => \$verbose,
'help' => \$help
);
$help = 1 unless ($dna_file && $enzyme_file);
help() if $help; # exits
# 'Main'
my $dna = getDNA($dna_file);
my %enzymes = %{ getEnzymes($enzyme_file) }; # A function cannot return a hash, so return a hashref and then store the referenced hash
foreach my $enzyme (keys %enzymes) {
print "Applying enzyme " . $enzyme . " gives:\n";
my $dna_holder = $dna;
my ($precut, $postcut) = ($enzymes{$enzyme}{'precut'}, $enzymes{$enzyme}{'postcut'});
my $R = qr/[GA]/;
my $B = qr/[CGT]/;
$precut =~ s/R/${R}/g;
$precut =~ s/B/${B}/g;
$postcut =~ s/R/${R}/g;
$postcut =~ s/B/${B}/g;
print "\tPre-Cut pattern: " . $precut . "\n" if $verbose;
print "\tPost-Cut pattern: " . $postcut . "\n" if $verbose;
#while(1){
# if ($dna_holder =~ s/(.*${precut})(${postcut}.*)/$1/ ) {
# print "\tFound section:" . $2 . "\n" if $verbose;
# print "\tRemaining DNA: " . $1 . "\n" if $verbose;
# unshift #{ $enzymes{$enzyme}{'cut_dna'} }, $2;
# } else {
# unshift #{ $enzymes{$enzyme}{'cut_dna'} }, $dna_holder;
# print "\tNo more cuts.\n" if $verbose;
# print "\t" . $_ . "\n" for #{ $enzymes{$enzyme}{'cut_dna'} };
# last;
# }
#}
unless ($dna_holder =~ s/(${precut})(${postcut})/$1'$2/g) {
print "\tHas no effect on given strand\n" if $verbose;
}
#{ $enzymes{$enzyme}{'cut_dna'} } = split(/'/, $dna_holder);
print "\t$_\n" for #{ $enzymes{$enzyme}{'cut_dna'} };
writeOutput($dna_file, $enzyme, $enzymes{$enzyme}{'cut_dna'}) if $write_output; #Note that $enzymes{$enzyme}{'cut_dna'} is an arrayref already
print "\n";
}
sub getDNA {
my ($dna_file) = #_;
open(my $dna_handle, '<', $dna_file) or die "Cannot open file $dna_file";
my #dna_array = <$dna_handle>;
chomp(#dna_array);
my $dna = join('', #dna_array);
print "Using DNA:\n" . $dna . "\n\n" if $verbose;
return $dna;
}
sub getEnzymes {
my ($enzyme_file) = #_;
my %enzymes;
open(my $enzyme_handle, '<', $enzyme_file) or die "Cannot open file $enzyme_file";
while(<$enzyme_handle>) {
chomp;
if(m{([^/]*)/([^']*)'([^/]*)//}) {
print "Found Enzyme " . $1 . ":\n\tPre-cut: " . $2 . "\n\tPost-cut: " . $3 . "\n" if $verbose;
$enzymes{$1} = {
precut => $2,
postcut => $3,
cut_dna => [] #Added to show the empty array that will hold the cut DNA sections
};
}
}
print "\n" if $verbose;
return \%enzymes;
}
sub writeOutput {
my ($dna_file, $enzyme, $cut_dna_ref) = #_;
my $outfile = $dna_file . '_' . $enzyme;
print "\tSaving data to $outfile\n" if $verbose;
open(my $outfile_handle, '>', $outfile) or die "Cannot open $outfile for writing";
print $outfile_handle $_ . "\n" for #{ $cut_dna_ref };
}
sub help {
my $filename = (split('/', $0))[-1];
my $enzyme_text = <<'END';
AatI/AGG'CCT//
AatII/GACGT'C//
AbsI/CC'TCGAGG//
TryII/RRR'TTT//
Test/AAA'TTT//
END
my $dna_text = <<'END';
CCCCCCGGGTTTCCCCCCC
CCCCCAAATTTCCCCCCCCCCCCAGATTTC
CCCCCCCCCGAGTTTCCCCC
END
print <<END;
Usage:
$filename --enzyme (-e) <enzyme-filename> --dna (-d) <dna-filename> [options] (files may come in either order)
$filename -h (shows this help)
Options:
--verbose (-v) print additional (debugging) information
--output (-o) output final data to files
Files:
The DNA file contains one DNA string which may be broken over many lines. E.G.:
$dna_text
The enzymes file constains enzyme definitions, one per line. E.G.:
$enzyme_text
END
exit;
}
Edit: Added cut_dna initialization explicitly because this is the final result holder for each enzyme, so I thought it would be good to see it more clearly.
Edit 2: Added output routine, call, flag and corresponding help.
Edit 3: Changed main routine to incorporate the best of canavanin's method while removing loops. Now its a global replace to add temporary cut mark (') and then split on cut mark into array. Left old method as comment, new method is the 5 lines following.
Edit 4: Additional test case for writing to multiple files. (Below)
my #names = ('cat','dog','sheep');
foreach my $name (#names) { #$name is lexical, ie dies after each loop
open(my $handle, '>', $name); #open a lexical handle for the file, also dies each loop
print $handle $name; #write to the handle
#$handles closes automatically when it "goes out of scope"
}

Note that in Enzymes, when you store an enzyme in the hash the name of the enzyme should be the key and the site should be the value (since in principle two enzymes could have identical sites).
In the Main routine, you can iterate through the hash; for each enzyme produce one output file. The most direct way is to translate the site to a regex (by means of other regexs) and apply it to the DNA sequence, but there are other ways. (It is probably worth splitting this off into at least one other sub.)

Here is how I have gone about trying to solve the problem (code below).
1) The file names are picked up from the arguments and respective filehandles are created.
2) A new file handle is created for the output file which in the specified format
3) The "cut points" are extracted from the first file
4) The DNA Sequences in the second file are looped over the cut points obtained in step 3.
#!/usr/bin/perl
use strict;
use warnings;
my $file_enzyme=$ARGV[0];
my $file_dna=$ARGV[1];
open DNASEQ, ">$file_dna"."_"."$file_enzyme";
open ENZYME, "$file_enzyme";
open DNA, "$file_dna";
while (<ENZYME>) {
chomp;
if( /'(.*)\/\//) { # Extracts the cut point between ' & // in the enzyme file
my $pattern=$1;
while (<DNA>) {
chomp;
#print $pattern;
my #output=split/$pattern/,;
print DNASEQ shift #output,"\n"; #first recognized sequence being output
foreach (#output) {
print DNASEQ "$pattern$_\n"; #prefixing the remaining sequences with the cut point pattern
}
}
}
}
close DNA;
close ENZYME;
close DNASEQ;

I know there have been several answers already, but hey... I just felt like trying my luck, so here's my suggestion:
#!/usr/bin/perl
use warnings;
use strict;
use Getopt::Long;
my ($enz_file, $dna_file);
GetOptions( "e=s" => \$enz_file,
"d=s" => \$dna_file,
);
if (! $enz_file || ! $dna_file) {
# some help text
print STDERR<<EOF;
Usage: restriction.pl -e enzyme_file -d DNA_file
The enzyme_file should contain one enzyme entry per line.
The DNA_file may contain the sequence on one single or on
several lines; all lines will be concatenated to yield a
single string.
EOF
exit();
}
my %enz_and_patterns; # stores enzyme name and corresponding pattern
open ENZ, "<$enz_file" or die "Could not open file $enz_file: $!";
while (<ENZ>) {
if (m#^(\w+)/([\w']+)//$#) {
my $enzyme = $1; # could also remove those two lines and use
my $pattern = $2; # the match variables directly, but this is clearer
$enz_and_patterns{$enzyme} = $pattern;
}
}
close ENZ;
my $dna_sequence;
open DNA, "<$dna_file" or die "Could not open file $dna_file: $!";
while (my $line = <DNA>) {
chomp $line;
$dna_sequence .= $line; # append the current bit to the sequence
# that has been read so far
}
close DNA;
foreach my $enzyme (keys %enz_and_patterns) {
my $dna_seq_processed = $dna_sequence; # local copy so that we retain the original
# now translate the restriction pattern to a regular expression pattern:
my $pattern = $enz_and_patterns{$enzyme};
$pattern =~ s/R/[GA]/g; # use character classes
$pattern =~ s/B/[^A]/g;
$pattern =~ s/(.+)'(.+)/($1)($2)/; # remove the ', but due to the grouping
# we "remember" its position
$dna_seq_processed =~ s/$pattern/$1\n$2/g; # in effect we are simply replacing
# each ' with a newline character
my $outfile = "${dna_file}_$enzyme";
open OUT, ">$outfile" or die "Could not open file $outfile: $!";
print OUT $dna_seq_processed , "\n";
close OUT;
}
I've tested my code with your TryII example, which worked fine.
As this is a task which can be handled by writing just a few lines of non-repetitive code I didn't feel creating separate subroutines would have been justified. I hope I will be forgiven... :)

Related

Preventing "foo" from matching "foo-bar" with grep -w

I am using grep inside my Perl script and I am trying to grep the exact keyword that I am giving. The problem is that "-w" doesn't recognize the "-" symbol as a separator.
example:
Let's say that I have these two records:
A1BG 0.0767377011073753
A1BG-AS1 0.233775553296782
if I give
grep -w "A1BG"
it returns both of them but I want only the exact one.
Any suggestions?
Many thanks in advance.
PS.
Here is my whole code.
The input file is a two-columns tab separated. So, I want to keep a unique value for each gene. In cases that I have more than one record, I calculate the average.
#!/usr/bin/perl
use strict;
use warnings;
#Find the average fc between common genes
sub avg {
my $total;
$total += $_ foreach #_;
return $total / #_;
}
my #mykeys = `cat G13_T.txt| awk '{print \$1}'| sort -u`;
foreach (#mykeys)
{
my #TSS = ();
my $op1 = 0;
my $key = $_;
chomp($key);
#print "$key\n";
my $command = "cat G13_T.txt|grep -E '([[:space:]]|^)$key([[:space:]]|\$)'";
#my $command = "cat Unique_Genes/G13_T.txt|grep -w $key";
my #belongs= `$command`;
chomp(#belongs);
my $count = scalar(#belongs);
if ($count == 1) {
print "$belongs[0]\n";
}
else {
for (my $i = 0; $i < $count; $i++) {
my #token = split('\t', $belongs[$i]);
my $lfc = $token[1];
push (#TSS, $lfc);
}
$op1 = avg(#TSS);
print $key ."\t". $op1. "\n";
}
}
If I got clarifications in comments right, the objective is to find the average of values (second column) for unique names in the first column. Then there is no need for external tools.
Read the file line by line and add up values for each name. The name uniqueness is granted by using a hash, with names being keys. Along with this also track their counts
use warnings;
use strict;
use feature 'say';
my $file = shift // die "Usage: $0 filename\n";
open my $fh, '<', $file or die "Can't open $file: $!";
my %results;
while (<$fh>) {
#my ($name, $value) = split /\t/;
my ($name, $value) = split /\s+/; # used for easier testing
$results{$name}{value} += $value;
++$results{$name}{count};
}
foreach my $name (sort keys %results) {
$results{$name}{value} /= $results{$name}{count}
if $results{$name}{count} > 1;
say "$name => $results{$name}{value}";
}
After the file is processed each accumulated value is divided by its count and overwritten by that, so by its average (/= divides and assigns), if count > 1 (as a small measure of efficiency).
If there is any use in knowing all values that were found for each name, then store them in an arrayref for each key instead of adding them
while (<$fh>) {
#my ($name, $value) = split /\t/;
my ($name, $value) = split /\s+/; # used for easier testing
push #{$results{$name}}, $value;
}
where now we don't need the count as it is given by the number of elements in the array(ref)
use List::Util qw(sum);
foreach my $name (sort keys %results) {
say "$name => ", sum(#{$results{$name}}) / #{$results{$name}};
}
Note that a hash built this way needs memory comparable to the file size (or may even exceed it), since all values are stored.
This was tested using the shown two lines of sample data, repeated and changed in a file. The code does not test the input in any way, but expects the second field to always be a number.
Notice that there is no reason to ever step out of our program and use external commands.
You may use a POSIX ERE regex with grep like this:
grep -E '([[:space:]]|^)A1BG([[:space:]]|$)' file
To return matches (not matching lines) only:
grep -Eo '([[:space:]]|^)A1BG([[:space:]]|$)' file
Details
([[:space:]]|^) - Group 1: a whitespace or start of line
A1BG - a substring
([[:space:]]|$) - Group 2: a whitespace or end of line

Take an array of phone numbers and search another array for each occurrence of said number and print that matching line and the following line

I have two text files. I'm importing each file into an array. Each value in the numbers array should search the users array for its match. If found echo the matching line and the proceeding line.
So, if the first entry in the numbers array is 1234, search users array for 1234. If found print that line and the next.
numbers.txt looks like:
1234567021
1234566792
users filelooks like:
1234567021#host.com User-Password == "secret"
Framed-IP-Address = 000.000.000.000,
What I have so far:
use strict;
my $users_file = "users";
my $numbers_file = "numbers.txt";
my $phonenumber;
my $numbers;
#### Place phone number into an array ####
open (RESULTS, $numbers_file) or die "Unable to open file: $users_file\n$!";
my #numbers;
#numbers = <NUMBER_RESULTS>;
close(NUMBER_RESULTS);
#### Place users file contents into an array ####
open (RESULTS, $users_file) or die "Unable to open file: $users_file\n$!";
my #users_data;
#users_data = <RESULTS>;
close(RESULTS);
#### Search the array for the string ####
foreach $numbers(#users_data) {
if (index($numbers,$phonenumber) ge 0) {
my #list = grep /\b$numbers\b/, #users_data;
chomp #list;
print "$_\n" foreach #list;
}
}
exit 1;
You are recreating a search for a key when perl has a built-in hash data type that will handle this better and faster than rolling your own. Using this will take a little more work in reading in the data, but it will be worth it.
First, lets switch to a modern version of open where we use a lexically scoped variable for the file handle, and specify a mode.
open (my $results, "<", $users_file) or die "Unable to open file: $users_file\n$!";
From there, we will read the file open line at a time and populate the hash.
my (%users_data, $number, $number_line);
while(<$results>)
{
chomp;
if(defined($number))
{
$user_data{$number} = "$number_line\t$_\n"; #load the line after the number into the hash value.
undef $number;
}else
{
if(/^(\d+)\#/) #match digits between the beginning of the line and the # symbol.
{
$number = $1; #save matched digits from $1.
$number_line = $_;
}
}
Note that this is assuming that the data is well formated. If there are concerns, you can test for proper formatting in the else clause.
Now, for the output we can use the following
for (#numbers)
{
chomp; #since we didn't remove newlines when populating #numbers
if( defined($users_data{$_}) )
{
print $users_data{$_};
}
}
EDIT
Here is a working version. Note use strict and use warnings helped to catch that one variables was declared (RESULTS and %users_file) while another was used later (NUMBER_RESULTS and %user_file), which is why those are so important. Also Data::Dumper was used to print out the contents of the array #numbers and the hash %users_data to see what data actually made it into the data structures.
#!/usr/bin/env perl
use strict;
use warnings;
#use Data::Dumper;
my $users_file = "users";
my $numbers_file = "numbers.txt";
#### Place phone number into an array ####
open (my $results, "<", $numbers_file) or die "Unable to open file: $numbers_file\n$!";
my #numbers;
#numbers = <$results>;
close($results);
#print Dumper \#numbers;
open (my $results, "<", $users_file) or die "Unable to open file: $users_file\n$!";
my (%users_data, $number, $number_line);
while(<$results>)
{
chomp;
if(defined($number))
{
$users_data{$number} = $number_line."\n$_\n"; #load the line after the number into the hash value.
undef $number;
}else
{
if(/^(\d+)\#/) #match digits between the beginning of the line and the # symbol.
{
$number = $1; #save matched digits from $1.
$number_line = $_;
}
}
}
#print Dumper \%users_data;
for (#numbers)
{
chomp; #since we didn't remove newlines when populating #numbers
if( defined($users_data{$_}) )
{
print $users_data{$_};
}
}

Perl extract between start and end

I am aiming to extract a string from start to an end word, (dIonly is start and should be the end workset [including these parenthesis]; furthermore I would like to print the output into a file named report.
I have had problems with lookbehind, as the variable length was not implemented.
Now I reversed the string, to do lookahead. However, something is still not working.
I need to start from dIonly which means I have to reverse the string to circumvent the problem described above, as there are many workset(( in the whole string, which means I can't start from there...
Thank you! I edited the script now. What I need to do is reverse the string. I did that by splitting the string with a space as delimiter into a list, then reversed it, and put it into a string again. Just to split it into a list again at the delimiter 'solution' as my output will have several strings of which I want to extract dIonly to workset (this only works once the string is reversed as otherwise I would encouter worksets that I do not want and extract a different string, as dIonly is a distinct part of the pattern of the solution from which I can work forward to the second workset (which itself is the first workset with 2 parenthesis). Then I want to print it to a new output file. Any suggestions welcome!
This is a sample of the data:
... denotes that it continues after
..... maxRiskC(cA, 3)) c workset((RiskCA(cA, 3), RiskCB(cB, 2), maxRiskC(cA, 3))) c RiskCA(cA, 3) c RiskCB(cB, 2)) ***********
equation (built-in equation for symbol <=) 6 <= 40 ---> true
Solution 4 (state 31) states: 40 rewrites: 8421 in 5357394502ms cpu
(1464ms real) (0) rewrites/second) G:Game --> workset(empty) c playA
c dIonly c
.....
#!/usr/bin/perl
# perl -d ./perl_debugger.pl
use strict;
use Data::Dumper qw(Dumper);
use File::Slurp;
my #a_linesorig;
my #a_out;
my #a_str;
my $line;
my $reversedline;
my #a_linesrev;
my #reversedarray;
my $reversedline;
my $str;
open(my $fh, "<", "data.txt")
or die "cannot open < data.txt: $!";
my $line = read_file('data.txt');
#a_linesorig = split(' ', $line);
#a_linesrev = reverse(#a_linesorig);
$reversedline = join(' ', #a_linesrev); # joins the reversed list to a single string again
#reversedarray = split( /solution/, $reversedline ); # should split huge string into a list from one solution to next
foreach $str (#reversedarray) {
if ($str =~ /\bdIonly:\b(.*?)\bworkset\b/g);
print Dumper \$str;
print (#a_out, "$str");
}
close $fh
or die "can't close file: $!";
open(my $fh, ">", "output.txt")
or die "cannot open > output.txt: $!";
foreach $str (#a_out)
{
print ($fh "$str\n");
}
close $fh
or die "can't close file: $!";
Take off the reverse, it will reverse letters also and not individual words, for that scalar.
You can try it with a greedy match since you are only interested in the last workset:
while (my $line = <$input>) {
chomp $line;
if ($line =~ /.*workset(.*dIonly)/) {
# do something with results
say $fh "'$1'";
}
}
And if you need to reverse before writing to the file, you can do:
while (my $line = <$input>) {
chomp $line;
if ($line =~ /.*workset(.*dIonly)/) {
say $fh join " ",reverse (split / /,$1);
}
}

Using iterated variables with regex

The point of the overall script is to:
step 1) open a single column file and read off first entry.
step 2) open a second file containing lots of rows and columns, read off EACH line one at a time, and find anything in that line that matches the first entry from the first file.
step3) if a match is found, then "do something constructive", and if not, go to the first file and take the second entry and repeat step 2 and step 3, and so on...
here is the script:
#!/usr/bin/perl
use strict; #use warnings;
unless(#ARGV) {
print "\usage: $0 filename\n\n"; # $0 name of the program being executed
exit;
}
my $list = $ARGV[0];
chomp( $list );
unless (open(LIST, "<$list")) {
print "\n I can't open your list of genes!!! \n";
exit;
}
my( #list ) = (<LIST>);
close LIST;
open (CHR1, "<acembly_chr_sorted_by_exon_count.txt") or die;
my(#spreadsheet) = (<CHR1>);
close CHR1;
for (my $i = 0; $i < scalar #list; $i++ ) {
print "$i in list is $list[$i]\n";
for (my $j = 1; $j < scalar #spreadsheet; $j++ ) {
#print "$spreadsheet[$j]\n";
if ( $spreadsheet[$j] ) {
print "will $list[$i] match with $spreadsheet[$j]?\n";
}
else { print "no match\n" };
} #for
} #for
I plan to use a regex in the line if ( $spreadsheet[$j] ) { but am having a problem at this step as it is now. On the first interation, the line print "will $list[$i] match with $spreadsheet[$j]?\n"; prints $list[$i] OK but does not print $spreadsheet[$j]. This line will print both variables correctly on the second and following iterations. I do not see why?
At first glance nothing looks overtly incorrect. As mentioned in the comments the $j = 1 looks questionable but perhaps you are skipping the first row on purpose.
Here is a more perlish starting point that is tested. If it does not work then you have something going on with your input files.
Note the extended trailing whitespace removal. Sometimes if you open a WINDOWS file on a UNIX machine and use chomp, you can have embedded \r in your text that causes weird things to happen to printed output.
#!/usr/bin/perl
use strict; #use warnings;
unless(#ARGV) {
print "\usage: $0 filename\n\n"; # $0 name of the program being executed
exit;
}
my $list = shift;
unless (open(LIST, "<$list")) {
print "\n I can't open your list of genes!!! \n";
exit;
}
open(CHR1, "<acembly_chr_sorted_by_exon_count.txt") or die;
my #spreadsheet = map { s/\s+$//; $_ } <CHR1>;
close CHR1;
# s/\s+$//; is like chomp but trims all trailing whitespace even
# WINDOWS files opened on a UNIX system.
for my $item (<LIST>) {
$item =~ s/\s+$//; # trim all trailing whitespace
print "==> processing '$item'\n";
for my $row (#spreadsheet) {
if ($row =~ /\Q$item\E/) { # see perlre for \Q \E
print "match '$row'\n";
}
else {
print "no match '$row'\n";
}
}
}
close LIST;

regular expression code

I need to find match between two tab delimited files files like this:
File 1:
ID1 1 65383896 65383896 G C PCNXL3
ID1 2 56788990 55678900 T A ACT1
ID1 1 56788990 55678900 T A PRO55
File 2
ID2 34 65383896 65383896 G C MET5
ID2 2 56788990 55678900 T A ACT1
ID2 2 56788990 55678900 T A HLA
what I would like to do is to retrive the matching line between the two file. What I would like to match is everyting after the gene ID
So far I have written this code but unfortunately perl keeps giving me the error:
use of "Use of uninitialized value in pattern match (m//)"
Could you please help me figure out where i am doing it wrong?
Thank you in advance!
use strict;
open (INA, $ARGV[0]) || die "cannot to open gene file";
open (INB, $ARGV[1]) || die "cannot to open coding_annotated.var files";
my #sample1 = <INA>;
my #sample2 = <INB>;
foreach my $line (#sample1) {
my #tab = split (/\t/, $line);
my $chr = $tab[1];
my $start = $tab[2];
my $end = $tab[3];
my $ref = $tab[4];
my $alt = $tab[5];
my $name = $tab[6];
foreach my $item (#sample2){
my #fields = split (/\t/,$item);
if ( $fields[1] =~ m/$chr(.*)/
&& $fields[2] =~ m/$start(.*)/
&& $fields[4] =~ m/$ref(.*)/
&& $fields[5] =~ m/$alt(.*)/
&& $fields[6] =~ m/$name(.*)/
) {
print $line, "\n", $item;
}
}
}
On its surface your code seems to be fine (although I didn't debug it). If you don't have an error I cannot spot, could be that the input data has RE special character, which will confuse the regular expression engine when you put it as is (e.g. if any of the variable has the '$' character). Could also be that instead of tab you have spaces some where, in which case you'll indeed get an error, because your split will fail.
In any case, you'll be better off composing just one regular expression that contains all the fields. My code below is a little bit more Perl Idiomatic. I like using the implicit $_ which in my opinion makes the code more readable. I just tested it with your input files and it does the job.
use strict;
open (INA, $ARGV[0]) or die "cannot open file 1";
open (INB, $ARGV[1]) or die "cannot open file 2";
my #sample1 = <INA>;
my #sample2 = <INB>;
foreach (#sample1) {
(my $id, my $chr, my $start, my $end, my $ref, my $alt, my $name) =
m/^(ID\d+)\s+(\w+)\s+(\w+)\s+(\w+)\s+(\w+)\s+(\w+)\s+(\w+)/;
my $rex = "^ID\\d+\\s+$chr\\s+$start\\s+$end\\s+$ref\\s+$alt\\s+$name\\s+";
#print "$rex\n";
foreach (#sample2) {
if( m/$rex/ ) {
print "$id - $_";
}
}
}
Also, how regular is the input data? Do you have exactly one tab between the fields? If that is the case, there is no point to split the lines into 7 different fields - you only need two: the ID portion of the line, and the rest. The first regex would be
(my $id, my $restOfLine) = m/^(ID\d+)\s+(.*)$/;
And you are searching $restOfLine within the second file in a similar technique as above.
If your files are huge and performance is an issue, you should consider putting the first regular expressions (or strings) in a map. That will give you O(n*log(m)) where n and m are the number of lines in each file.
Finally, I have a similar challenge when I need to compare logs. The logs are supposed to be identical, with the exception of a time mark at the beginning of each line. But more importantly: most lines are the same and in order. If this is what you have, and it make sense for you, you can:
First remove the IDxxx from each line: perl -pe "s/ID\d+ +//" file >cleanfile
Then use BeyondCompare or Windiff to compare the files.
I played a bit with your code. What you wrote there was actually three loops:
one over the lines of the first file,
one over the lines of the second file, and
one over all fields in these lines. You manually unrolled this loop.
The rest of this answer assumes that the files are strictly tab-seperated and that any other whitespace matters (even at the end of fields and lines).
Here is a condensed version of the code (assumes open filehandles $file1, $file2, and use strict):
my #sample2 = <$file2>;
SAMPLE_1:
foreach my $s1 (<$file1>) {
my (undef, #fields1) = split /\t/, $s1;
my #regexens = map qr{\Q$_\E(.*)}, #fields1;
SAMPLE_2:
foreach my $s2 (#sample2) {
my (undef, #fields2) = split /\t/, $s2;
for my $i (0 .. $#regexens) {
$fields2[$i] =~ $regexens[$i] or next SAMPLE_2;
}
# only gets here if all regexes matched
print $s1, $s2;
}
}
I did some optimisations: precompiling the various regexes and storing them in an array, quoting the contents of the fields etc. However, this algorithm is O(n²), which is bad.
Here is an elegant variant of that algorithm that knows that only the first field is different — the rest of the line has to be the same character for character:
my #sample2 = <$file2>;
foreach my $s1 (<$file1>) {
foreach my $s2 (#sample2) {
print $s1, $s2 if (split /\t/, $s1, 2)[1] eq (split /\t/, $s2, 2)[1];
}
}
I just test for string equality of the rest of the line. While this algorithm is still O(n²), it outperforms the first solution roughly by an order of magnitude simply by avoiding braindead regexes here.
Finally, here is an O(n) solution. It is a variant of the previous one, but executes the loops after each other, not inside each other, therefore finishing in linear time. We use hashes:
# first loop via map
my %seen = map {reverse(split /\t/, $_, 2)}
# map {/\S/ ? $_ : () } # uncomment this line to handle empty lines
<$file1>;
# 2nd loop
foreach my $line (<$file2>) {
my ($id2, $key) = split /\t/, $line, 2;
if (defined (my $id1 = $seen{$key})) {
print "$id1\t$key";
print "$id2\t$key";
}
}
%seen is a hash that has the rest of the line as a key and the first field as a value. In the second loop, we retrieve the rest of the line again. If this line was present in the first file, we reconstruct the whole line and print it out. This solution is better than the others and scales well up- and downwards, because of its linear complexity
How about:
#!/usr/bin/perl
use File::Slurp;
use strict;
my ($ina, $inb) = #ARGV;
my #lines_a = File::Slurp::read_file($ina);
my #lines_b = File::Slurp::read_file($inb);
my $table_b = {};
my $ln = 0;
# Store all lines in second file in a hash with every different value as a hash key
# If there are several identical ones we store them also, so the hash values are lists containing the id and line number
foreach (#lines_b) {
chomp; # strip newlines
$ln++; # count current line number
my ($id, $rest) = split(m{[\t\s]+}, $_, 2); # split on whitespaces, could be too many tabs or spaces instead
if (exists $table_b->{$rest}) {
push #{ $table_b->{$rest} }, [$id, $ln]; # push to existing list if we already found an entry that is the same
} else {
$table_b->{$rest} = [ [$id, $ln] ]; # create new entry if this is the first one
}
}
# Go thru first file and print out all matches we might have
$ln = 0;
foreach (#lines_a) {
chomp;
$ln++;
my ($id, $rest) = split(m{[\t\s]+}, $_, 2);
if (exists $table_b->{$rest}) { # if we have this entry print where it is found
print "$ina:$ln:\t\t'$id\t$rest'\n " . (join '\n ', map { "$inb:$_->[1]:\t\t'$_->[0]\t$rest'" } #{ $table_b->{$rest} }) . "\n";
}
}