Extracting nucleotide sequence from a non-standardly formatted text file - regex

I have been given some DNA sequences by collaborators in a word document that I'd like to convert into a series of fasta sequences in one file.
I've made it into a text file and I figured that using regular expressions to extract the gene name and the sequence:
use warnings;
use strict;
die "usage: make_fasta.pl <sequence file>" unless (#ARGV == 1);
my $seq_filename = shift;
my $fasta_db_name = $seq_filename . "_db.fa";
open(my $seq_file, '<', $seq_filename)
or die "can't open file $seq_filename, $!";
open(my $fasta_file, '>', $fasta_db_name)
or die "can't open file $fasta_db_name, $!";
while (my $line = <$seq_file>) {
chomp $line;
if ($line =~ /^[ATCG]+$/) { # if the line is entirely DNA seqence
print $fasta_file "$line\n";
} elsif ($line =~ /Full-length (\w+) cDNA/) { # if the line has gene info
print $fasta_file ">$1\n";
} else {
next;
}
}
But that just gave me the name of the first gene. Clearly I've done something wrong with the DNA regular expression but I can't for the life of me work it out. To my eyes it's exactly the same as other suggested DNA tests I've found on this site and others.
The file I'm trying to parse is configured like so:
Collaborators name
title of gene set
Full-length clock cDNA coding sequence
ATGGTAGGATGTGTAATGCGTACGTGATCGT
Full-length per cDNA coding sequence
ATGCTAGCTACGTACGTAGCTACGTAGTACG
I want the output to be a fasta file so:
>clock
ATGGTAGGATGTGTAATGCGTACGTGATCGT
>per
ATGCTAGCTACGTACGTAGCTACGTAGTACG
The first few lines of the actual input file are:
Dr Lin Zhang (Leicester University 10/2012)
Canonical clock genes
Full-length per cDNA coding seq (3693bp)
ATGGACACAGGAACACCCCATGAAGATGTGCCCTCAGAGGACCACACCTTGGAAGAAGGGGACAGCAAGAACCCCTCGTGCCAGCAAGAGTCAGCCTACGGCTCCCTCGAGTCATCCTCCAATGGACAGTCTCAGAAAAGTTTCGGAGGAAGTGGAAGCAAAAGCTTAAATAGTGGTTCGAGTCACAGCAGCGGCTTTGGGGACCAAAATGATTTCAAGGGTATCCATCTTCACGAAGCGAAACACATAGCGTTGAAGAAGAAGAAAACTGGGAAAGGAGGTGAAAAGGTAGCAGAAATCCCCTTTCAAACTGCCTCTGAGGCAGAACTGTCCTCCAAAGGAAACGAAACAGAAAAGGAGAAAGAAACAAGCCTCGAGGAGTCTCCTGCTGCAAAAGAGGAAGCAATTATCGAAAAGGAGTCTCGTTACATCCACCCGAGGAACT

Kind of hard to answer this question without seeing part of the actual input file.
There is a mis-match between your example input and your REGEX:
# looking for verbatim('Full-length') then <space> then one WORD_WITH_ALPHNUMERICS then <space> and then verbatim 'cDNA'
$line =~ /Full-length (\w+) cDNA/;
Your example input line has 'Full length' without a dash, multiple words for the gene name not just one and no 'cDNA' at the end.
If your input line has 'Full-length gene name with multiple words cDNA', your REGEX can be:
$line=~/Full-length\s+(.*?)\s+cDNA/;

The problem is apparently with your input data. I modified the code you posted to produce the following program:
#!/usr/bin/env perl
use warnings;
use strict;
while (my $line = <DATA>) {
chomp $line;
if ($line =~ /^[ATCG]+$/) { # if the line is entirely DNA seqence
print "$line\n";
} elsif ($line =~ /Full-length (\w+) cDNA/) { # if the line has gene info
print ">$1\n";
}
}
__DATA__
Collaborators name
title of gene set
Full-length clock cDNA coding sequence
ATGGTAGGATGTGTAATGCGTACGTGATCGT
Full-length per cDNA coding sequence
ATGCTAGCTACGTACGTAGCTACGTAGTACG
and it produces the output you specified:
~$ src/tmp/cdna
>clock
ATGGTAGGATGTGTAATGCGTACGTGATCGT
>per
ATGCTAGCTACGTACGTAGCTACGTAGTACG
My modifications were only to make it self-contained and did not change any of the flow control or logic, aside from removing the useless else { next } clause.
Can you find and post a few lines of actual data which fails for you, since the dummy data provided seems to work correctly?

Related

How to capture multiple words using regex on this particular text?

I'm trying to extract the best paying job titles from this sample text:
Data Scientist
#1 in Best Paying Jobs
5,100 Projected Jobs $250,000 Median Salary 0.5% Unemployment Rate
Programmer
#2 in Best Paying Jobs
4,000 Projected Jobs $240,000 Median Salary 1.0% Unemployment Rate
SAP Module Consultant
#3 in Best Paying Jobs
3,000 Projected Jobs $220,000 Median Salary 0.2% Unemployment Rate
by using the following regex and Perl code.
use File::Glob;
local $/ = undef;
my $file = #ARGV[0];
open INPUT, "<", $file
or die "Couldn't open file $!\n";
my $content = <INPUT>;
my $regex = "^\w+(\w+)*$\n\n#(\d+)";
my #arr_found = ($content =~ m/^\w+(\w+)*$\n\n#(\d+)/g);
close (INPUT);
Q1: The regex finds only the one-word titles*. How to make it find the multiple word titles and how to forward (i.e. how to properly capture) those found titles into the Perl array?
Q2: I defined the regex into a Perl variable and tried to use that variable for the regex operation like:
my #arr_found = ($content =~ m/"$regex"/g);
but it gave error. How to make it?
* When I apply the regex ^\w+(\w+)*$\n\n#(\d+) on Sublime Text 2, it finds only the one word titles.
Why not process line-by-line, simple and easy
use warnings;
use strict;
use feature 'say';
my $file = shift || die "Usage: $0 file\n";
open my $fh, '<', $file or die "Can't open $file: $!";
my (#jobs, $prev_line);
while (my $line = <$fh>) {
chomp $line;
next if not $line =~ /\S/;
if ($line =~ /^\s*#[0-9]/) {
push #jobs, $prev_line;
}
$prev_line = $line;
}
say for #jobs;
This relies on the requirement that the #N line is the first non-empty line after the jobs title.
It prints
Data Scientist
Programmer
SAP Module Consultant
The question doesn't say whether rankings are wanted as well but there is a hint in the regex that they may be. Then, assuming that the ordering in the file is "correct" you can iterate over the array indices and print elements (titles) with their indices (rank).
Or, to be certain, capture them in the regex, /^\s*#([0-9]+)/. Then you can directly print both the title and its rank, or perhaps store them in a hash with key-value pairs rank => title.
As for the regex, there are a few needed corrections. To compose a regex ahead of matching, what is a great idea, you want the qr operator. To work with multi-line strings you need the /m modifier. (See perlretut.) The regex itself needs fixing. For example
my $regex = qr/^(.+)?(?:\n\s*)+\n\s*#\s*[0-9]/m;
my #titles = $content =~ /$regex/g
what captures a line followed by at least one empty line and then #N on another line.
If the ranking of titles is needed as well then capture it, too, and store in a hash
my $regex = qr/^(.+)?(?:\n\s*)+\n\s*#\s*([0-9]+)/m;
my %jobs = reverse $content =~ /$regex/g;
or maybe better not push it with reverse-ing the list of matches but iterate through pairs instead
my %jobs;
while ($content =~ /$regex/g) {
$jobs{$2} = $1;
}
since with this we can check our "catch" at each iteration, do other processing, etc. Then you can sort the keys to print in order
say "#$_ $jobs{$_}" for sort { $a <=> $b } keys %jobs;
and just in general pick jobs by their rank as needed.
I think that it's fair to say that the regex here is much more complex than the first program.
Answers for your questions:
you are capturing the second word only and you do not allow for space in between them. That's why it won't match e.g. Data Scientist
use the qr// operator to compile regexes with dynamic content. The error stems from the $ in the middle of the regex which Perl regex compiler assumes you got wrong, because $ should only come at the end of a regex.
The following code should achieve what you want. Note the two-step approach:
Find matching text
beginning of a line (^)
one-or-more words separated by white space (\w+(?:\s+\w+)*, no need to capture match)
2 line ends (\n\n)
# followed by a number (\d+)
apply regex multiple times (/g) and treat strings as multiple lines (/m, i.e. ^ will match any beginning of a line in the input text)
Split match at line ends (\n) and extract the 1st and the 3rd field
as we know $match will contain three lines, this approach is much easier than writing another regex.
#!/usr/bin/perl
use strict;
use warnings;
use feature qw(say);
use File::Slurper qw(read_text);
my $input = read_text($ARGV[0])
or die "slurp: $!\n";
my $regex = qr/^(\w+(?:\s+\w+)*\n\n#\d+)/m;
foreach my $match ($input =~ /$regex/g) {
#say $match;
my($title, undef, $rank) = split("\n", $match);
$rank =~ s/^#//;
say "MATCH '${title}' '${rank}'";
}
exit 0;
Test run over the example text you provided in your question.
$ perl dummy.pl dummy.txt
MATCH 'Data Scientist' '1'
MATCH 'Programmer' '2'
MATCH 'SAP Module Consultant' '3'
UNICODE UPDATE: as suggested by #Jan's answer the code can be improved like this:
my $regex = qr/^(\w+(?:\s+\w+)*\R\R#\d+)/m;
...
my($title, undef, $rank) = split(/\R/, $match);
That is probably the more generic approach, as UTF-8 is the default for File::Slurper::read_text() anyway...
You were not taking whitespaces (as in Data Scientist) into account:
^\w+.*$\R+#(\d+)
See a demo on regex101.com.
\R is equal to (?>\r\n|\n|\r|\f|\x0b|\x85) (matches Unicode newlines sequences).

perl regex: searching thru entire line of file

I'm a regex newbie, and I am trying to use a regex to return a list of dates from a text file. The dates are in mm/dd/yy format, so for years it would be '55' for '1955', for example. I am trying to return all entries from years'50' to '99'.
I believe the problem I am having is that once my regex finds a match on a line, it stops right there and jumps to the next line without checking the rest of the line. For example, I have the dates 12/12/12, 10/10/57, 10/09/66 all on one line in the text file, and it only returns 10/10/57.
Here is my code thus far. Any hints or tips? Thank you
open INPUT, "< dates.txt" or die "Can't open input file: $!";
while (my $line = <INPUT>){
if ($line =~ /(\d\d)\/(\d\d)\/([5-9][0-9])/g){
print "$&\n" ;
}
}
A few points about your code
You must always use strict and use warnings 'all' at the top of all your Perl programs
You should prefer lexical file handles and the three-parameter form of open
If your regex pattern contains literal slashes then it is clearest to use a non-standard delimiter so that they don't need to be escaped
Although recent releases of Perl have fixed the issue, there used to be a significant performance hit when using $&, so it is best to avoid it, at least for now. Put capturing parentheses around the whole pattern and use $1 instead
This program will do as you ask
use strict;
use warnings 'all';
open my $fh, '<', 'dates.txt' or die "Can't open input file: $!";
while ( <$fh> ) {
print $1, "\n" while m{(\d\d/\d\d/[5-9][0-9])}g
}
output
10/10/57
10/09/66
You are printing $& which gets updated whenever any new match is encountered.
But in this case you need to store the all the previous matches and the updated one too, so you can use array for storing all the matches.
while(<$fh>) {
#dates = $_ =~ /(\d\d)\/(\d\d)\/([5-9][0-9])/g;
print "#dates\n" if(#dates);
}
You just need to change the 'if' to a 'while' and the regex will take up where it left off;
open INPUT, "< a.dat" or die "Can't open input file: $!";
while (my $line = <INPUT>){
while ($line =~ /(\d\d)\/(\d\d)\/([5-9][0-9])/g){
print "$&\n" ;
}
}
# Output given line above
# 10/10/57
# 10/09/66
You could also capture the whole of the date into one capture variable and use a different regex delimiter to save escaping the slashes:
while ($line =~ m|(\d\d/\d\d/[5-9]\d)|g) {
print "$1\n" ;
}
...but that's a matter of taste, perhaps.
You can use map also to get year range 50 to 99 and store in array
open INPUT, "< dates.txt" or die "Can't open input file: $!";
#as = map{$_ =~ m/\d\d\/\d\d\/[5-9][0-9]/g} <INPUT>;
$, = "\n";
print #as;
Another way around it is removing the dates you don't want.
$line =~ s/\d\d\/\d\d\/[0-4]\d//g;
print $line;

In perl, how do i use regexes from one file to match FASTA sequences in another file

I have two files, the first (file1) contains several rexeges, while the other(file2) contains FASTA sequences . My intention is to use the regex in file1 to check if they match any Fasta sequences in file2, and print any regexes that match atleast one sequence, with the number of sequences they match. I would have liked to provide my sample code but i couldn't even begin. Please help.
file1 is structured in such a way that each line has an ID, followed by '>>', then the regex;
e.g FGER_HWW_PRT >> ..DW[ALK]..[^P]..[VI]{2,4}
TKAR_GLW_NQW >> [^VKR]{0,2}..FP[D].T.N.Q.
etc...
file2 has an idenfier of a sequence on one line and the sequence on the next line;
e.g >lac9_B: details details
GFVTSDRWPALKMSRWSLEMVWASRGYPLVNDRMWSWSDDDP
>serP_A: otherdetails details2
GFVLSDPPPPALKMSRWSLEMVWASRGYPLVNDPWQRTKRKRKDRTCWASNYIHDRP
etc...
Thanks in advance.
This might get you started. If you think it might be useful for you, let me know and I can explain what's going on:
#!/usr/bin/perl
use warnings;
use strict;
(Using your .fasta file as input):
my $infile = 'in.txt';
open my $input, '<', $infile or die "Can't open to $infile: $!";
my (#head, #seq, %hash);
Set a 'match' variable to test your headers for:
my $match = "details2";
while (<$input>) {
chomp;
push #head, $_ if /^>/;
push #seq, $_ if /^[A-Z]/;
#hash{#head} = #seq;
}
Cycle through the keys (headers) of your hash, and test print the header and sequence if they match your match variable:
foreach my $header (keys %hash){
if ($header =~ /$match/){
print "Name: $header\tcontains: '$match'\nSequence: $hash{$header}\n" ;
}
}
Output:
Name: >serP_A: otherdetails details2 contains: 'details2'
Sequence: GFVLSDPPPPALKMSRWSLEMVWASRGYPLVNDPWQRTKRKRKDRTCWASNYIHDRP

Perl Regex Match Text String and Extract Following Number

I have a giant text data file (~100MB) that is a concatenation of a bunch of data files with various header information then some columns of data. Here's the problem. I want to extract a particular number from the header info before each of these data sets and then append that to another column in the data (and write out that data to a different file).
The header info that I want is of the format ex: BGA 1
Where what I want for that extra data column is the # after word BGA. It will be a number between 1 and maybe 20000. I can write the regex to pull the word BGA, but I don't seem to be able to figure out how to just get the digit after it.
To add EXTRA fun, that text "BGA 1" is repeated in each data section TWICE.
Here's what I have so far, which actually doesn't work... I want it to at least print "BGA" everytime it encounters the word BGA, but it prints nothing.... Any help would be appreciated.
#!/usr/bin/perl
use strict;
use warnings;
my $file = 'alldata.txt';
open my $info, $file or die "Could not open $file: $!";
$_="";
while(my $line = <$info>){
if ($line eq "/BGA/"){
print <>,"\n";
}
}
close $file;
if ($line =~ /BGA\s(\d+)/){
#your code
print "BGA number $1 \n";
#your code
}
And $1 variable will have the number you want
If there is more than one BGA per line, you'll need to allow the regex to match more than once per line:
while (my $line = <$info>) {
while ( $line =~ /BGA\s(\d+)/g ) {
print "$1\n";
}
}
This should print out all the BGA numbers as a single column. Without any further information it's hard to answer this any better.
First, a 100 MB file is not giant. Don't be so defeatist. You could even slurp it into memory:
Let's look at the few critical places in your code:
while(my $line = <$info>) {
if ($line eq "/BGA/") {
Your condition $line eq "/BGA/" tests if the line literally consists of the string "/BGA/". But, that can never be true for the line with at least have the input record separator, i.e. the contents of $/ at the end because you did not chomp it. In any case, what you want is to match lines that contain "BGA" anywhere and the proper Perl syntax to do that is
if ($line =~ /BGA/) {
Now, once you fix that, you are going to run into a problem with the following statement:
print <>,"\n";
What you really want is print $line;. The diamond operator, <>, in list context is going to try to slurp from STDIN or any files specified as arguments on the command line. Not a good idea.
Others have pointed out how to match the string "BGA" followed by a digit. For better answers, you are going to need to show examples of input and expected output.

Perl regex: How to find in a file a word typed by a user

I am writing a script to read a LOG file. I want the user to type a word and then look it up and print the line (from a string) matching the word.
I'm just learning Perl so please be very specific and simple so that I can understand it.
print "Please Enter the word to find: ";
chomp ($userInput = <STDIN>);
while ($line = <INPUT>)
if ($line =~ /userInput/)
print $line;
I know that this is not perfect but I'm just learning.
You were close. You need to expand the variable in the pattern match.
print "Please Enter the word to find: ";
chomp ($userInput = <STDIN>);
while ($line = <INPUT>) {
if ($line =~ /$userInput/) { # note extra dollar sign
print $line;
}
}
Be aware that that is a pattern match, so you are searching with a string that potentially contains wildcards in it. If you want a literal string, put a \Q in front of the variable as you interpolate it: /\Q$userInput/.
Something like .\bWORD\b. might work (thou it is not tested)
print $line if ($line =~ /.*\bWORD\b/)
#NewLearner
\b is for word boundaries
http://www.regular-expressions.info/wordboundaries.html
If you're doing just one loopup, using a while loop is fine. Though of course you'll need to fix your syntax.
You could also use grep:
print grep /$userInput/, <INPUT>;
If you want to do multiple lookups, you can either reopen the file handle (if the file is large), or store it in an array:
print grep /$userInput/, #array;
You'll have meta characters in your input, of course. This can be a good thing, or bad, depending on your users. For example, an experienced user would recognize the option to refine his search by entering a search term such as ^foo(?=bar), whereas other people may get very confused when they can't find the string foo+bar.
A way to escape meta characters is by using quotemeta on your input. Another is to use \Q ... \E inside your regex.
$userInput = quotemeta($userInput);
# or
print grep /\Q$userInput\E/, <INPUT>;
I believe if I were you, I would use a subroutine for the lookup. That way you can perform as many lookups as you like rather handily.
use strict;
use warnings; # ALWAYS use these
print "Please Enter the word to find: ";
chomp (my $userInput = <>); # <> is a more flexible handle
print lookup($userInput);
sub lookup {
my $word = shift;
open my $fh, "<", $inputfile or die $!;
my #hits;
while (<$fh>) {
push #hits, $_ if /\Q$word\E/;
}
return #hits;
}