I'm not sure where to start with this one... my client gets stock figures from his supplier but they are now being sent in a different format, here is a sample snippet:
[["BLK",[["Black","0F1315"]],[["S","813"],["M","1378"],["L","1119"],["XL","1069"],["XXL","412"],["3XL","171"]]],["BOT",[["Bottle","15451A"]],[["S","226"],["M","425"],["L","772"],["XL","509"],["XXL","163"]]],["BUR",[["Burgundy","73002E"]],[["S","402"],["M","530"],["L","356"],["XL","257"],["XXL","79"]]],["DNA",[["Deep Navy","000F33"]],[["S","699"],["M","1161"],["L","1645"],["XL","1032"],["XXL","350"]]],["EME",[["Emerald","0DAB5E"]],[["S","392"],["M","567"],["L","613"],["XL","431"],["XXL","97"]]],["HEA",[["Heather","C0D4D7"]],[["S","374"],["M","447"],["L","731"],["XL","386"],["XXL","115"],["3XL","26"]]],["KEL",[["Kelly","0FFF00"]],[["S","167"],["M","285"],["L","200"],["XL","98"],["XXL","45"]]],["NAV",[["Navy","002466"]],[["S","451"],["M","1389"],["L","1719"],["XL","1088"],["XXL","378"],["3XL","177"]]],["NPU",[["Purple","560D55"]],[["S","347"],["M","553"],["L","691"],["XL","230"],["XXL","101"]]],["ORA",[["Orange","FF4700"]],[["S","125"],["M","273"],["L","158"],["XL","98"],["XXL","98"]]],["RED",[["Red","FF002E"]],[["S","972"],["M","1186"],["L","1246"],["XL","889"],["XXL","184"]]],["ROY",[["Royal","1500CE"]],[["S","1078"],["M","1346"],["L","1102"],["XL","818"],["XXL","135"]]],["SKY",[["Sky","91E3FF"]],[["S","567"],["M","919"],["L","879"],["XL","498"],["XXL","240"]]],["SUN",[["Sunflower","FFC700"]],[["S","843"],["M","1409"],["L","1032"],["XL","560"],["XXL","53"]]],["WHI",[["White","FFFFFF"]],[["S","631"],["M","2217"],["L","1666"],["XL","847"],["XXL","410"],["3XL","74"]]]]
Firstly the inital [ and end ] can be removed
Then it needs be be broken down into segments of colours, i.e.:
["BLK",[["Black","0F1315"]],[["S","813"],["M","1378"],["L","1119"],["XL","1069"],["XXL","412"],["3XL","171"]]]
The BLK is needed here, the next block [["Black","0F1315"]] can be disregarded.
Next I need to take the stock data for each size ["S","813"] etc
Therefore I should have a data such as:
$col = BLK
$size = S
$qty = 813
$col = BLK
$size = M
$qty = 1278
and repeat this segment for every colour seqment in the data.
The amount of colour segments in the data will vary, as will the amount of sizing segements within. Also the amount of sizing segments will vary colour to colour, i.e. there maybe 6 sizes for BLK but only 5 for RED
The data will be written out while in the loop for these so something like print "$col:$size:$qty" will be fine as this would then be in a format ready to be processed.
Sorry for the long message, I just can't seem to get my head round this today!!
Regards,
Stu
This looks like valid JSON to me, why not use a JSON parser instead of trying to solve this with a regex?
use JSON;
my $json_string = '[["BLK",[["Black","0F1315"]],[["S","813"...<snip>';
my $deserialized = from_json( $json_string );
Then you can iterate over the array and extract the pieces of information you need.
Building on Tim Pietzcker's answer:
...
my $deserialized = from_json( $json_string );
foreach my $group ( #$deserialized ) {
my ( $color, undef, $sizes ) = #$group;
print join( ":", $color, #$_ ), "\n" for #$sizes;
}
(And yes, for this particular format, eval should do as well as from_json, although the latter is safer. However, you should really try to find an official spec for the format: is it really JSON or something else?)
Assuming you have your data in $str, then eval(EXPR) (Danger Will Robinson!) and process the resulting data structure:
my $struct = eval $str;
foreach my $cref (#$struct) {
my($color, undef, $sizerefs) = #$cref; # 3 elements in each top level
foreach my $sizeref (#$sizerefs) {
my($size, $qty) = #$sizeref;
print "$color:$size:$qty\n";
}
}
Related
I am trying to remove the units out of a column of cell array data i.e.:
cArray =
time temp
2022-05-10 20:19:43 '167 °F'
2022-05-10 20:19:53 '173 °F'
2022-05-10 20:20:03 '177 °F'
...
2022-06-09 20:18:10 '161 °F'
I have tried str2double but get all NaN.
I have found some info on regexp but don't follow exactly as the example is not the same.
Can anyone help me get the temp column to only read the value i.e.:
cArray =
time temp
2022-05-10 20:19:43 167
2022-05-10 20:19:53 173
2022-05-10 20:20:03 177
...
2022-06-09 20:18:10 161
For some cell array of data
cArray = { ...
1, '123 °F'
2, '234 °F'
3, '345 °F'
};
The easiest option is if we can safely assume the temperature data always starts with numeric values, and you want all of the numeric values. Then we can use regex to match only numbers
temps = regexp( cArray(:,2), '\d+', 'match', 'once' );
The match option causes regexp to return the matching string rather than the index of the match, and once means "stop at the first match" so that we ignore everything after the first non-numeric character.
The pattern '\d+' means "one or more numbers". You could expand it to match numbers with a decimal part using '\d+(\.\d+)?' instead if that's a requirement.
Then if you want to actually output numbers, you should use str2double. You could do this in a loop, or use cellfun which is a compact way of achieving the same thing.
temps = cellfun( #str2double, temps, 'uni', 0 ); % 'uni'=0 to retain cell array
Finally you can override the column in cArray
cArray(:,2) = temps;
Imagine i have a sequence of files, e.g.:
...
segment8_400_av.ts
segment9_400_av.ts
segment10_400_av.ts
segment11_400_av.ts
segment12_400_av.ts
...
When the filenames are known, i can match against the filenames with a regular expression like:
/segment(\d+)_400_av\.ts/
Because i know the incremental pattern.
But what would be a generic approach to this? I mean how can i take two file names out of the list, compare them and find out where in the file name the counting part is, taking into account any other digits that can occur in the filename (the 400 in this case)?
Goal: What i want to do is to run the script against various file sequences to check for example for missing files, so this should be the first step to find out the numbering scheme. File sequences can occur in many different fashions, e.g.:
test_1.jpg (simple counting suffix)
test_2.jpg
...
or
segment9_400_av.ts (counting part inbetween, with other static digits)
segment10_400_av.ts
...
or
01_trees_00008.dpx (padded with zeros)
01_trees_00009.dpx
01_trees_00010.dpx
Edit 2: Probably my problem can be described more simple: With a given set of files, i want to:
Find out, if they are a numbered sequence of files, with the rules below
Get the first file number, get the last file number and file count
Detect missing files (gaps in the sequence)
Rules:
As melpomene summarized in his answer, the file names only differ in one substring, which consists only of digits
The counting digits can occur anywhere in the filename
The digits can be padded with 0's (see example above)
I can do #2 and #3, what i am struggling with is #1 as a starting point.
You tagged this question regex, so here's a regex-based solution:
use strict;
use warnings;
my $name1 = 'segment12_400_av.ts';
my $name2 = 'segment10_400_av.ts';
if (
"$name1\0$name2" =~ m{
\A
( \D*+ (?: \d++ \D++ )* ) # prefix
( \d++ ) # numeric segment 1
( [^\0]* ) # suffix
\0 # separator
\1 # prefix
( \d++ ) # numeric segment 2
\3 # suffix
\z
}xa
) {
print <<_EOT_;
Result of comparing "$name1" and "$name2"
Common prefix: $1
Common suffix: $3
Varying numeric parts: $2 / $4
Position of varying numeric part: $-[2]
_EOT_
}
Output:
Result of comparing "segment12_400_av.ts" and "segment10_400_av.ts"
Common prefix: segment
Common suffix: _400_av.ts
Varying numeric parts: 12 / 10
Position of varying numeric part: 7
It assumes that
the strings are different (guard the condition with $name1 ne $name2 && ... if that's not guaranteed)
there's only one substring that's different between the input strings (otherwise it won't find any match)
the differing substring consists of digits only
all digits surrounding the first point of difference are part of the varying increment (e.g. the example above recognizes segment as the common prefix, not segment1)
The idea is to combine the two names into a single string (separated by NUL, which is unambiguous because filenames can't contain \0), then let the regex engine do the hard work of finding the longest common prefix (using greediness and backtracking).
Because we're in a regex, we can get a bit more fancy than just finding the longest common prefix: We can make sure that the prefix doesn't end with a digit (see the segment1 vs. segment case above) and we can verify that the suffix is also the same.
See if this works for you:
use strict;
use warnings;
sub compare {
my ( $f1, $f2 ) = #_;
my #f1 = split /(\d+)/sxm, $f1;
my #f2 = split /(\d+)/sxm, $f2;
my $i = 0;
my $out1 = q{};
my $out2 = q{};
foreach my $p (#f1) {
if ( $p eq $f2[$i] ) {
$out1 .= $p;
$out2 .= $p;
}
else {
$out1 .= sprintf ' ((%s)) ', $p;
$out2 .= sprintf ' ((%s)) ', $f2[$i];
}
$i++;
}
print $out1 . "\n";
print $out2 . "\n";
return;
}
print "Test1:\n";
compare( 'segment8_400_av.ts', 'segment9_400_av.ts' );
print "\n\nTest2:\n";
compare( 'segment999_8_400_av.ts', 'segment999_9_400_av.ts' );
You basically split strings by starting/ending digits, the loop through the items and compare each of the 'pieces'. If they are equal, you accumulate. If not, then you highlight the differences and accumulate.
Output (I'm using ((number)) for the highlight)
Test1:
segment ((8)) _400_av.ts
segment ((9)) _400_av.ts
Test2:
segment999_ ((8)) _400_av.ts
segment999_ ((9)) _400_av.ts
I assume that only the counter differs across the strings
use warnings;
use strict;
use feature 'say';
my ($fn1, $fn2) = ('segment8_400_av.ts', 'segment12_400_av.ts');
# Collect all numbers from all strings
my #nums = map { [ /([0-9]+)/g ] } ($fn1, $fn2);
my ($n, $pos); # which number in the string, at what position
# Find which differ
NUMS:
for my $j (1..$#nums) { # strings
for my $i (0..$#{$nums[0]}) { # numbers in a string
if ($nums[$j]->[$i] != $nums[0]->[$i]) { # it is i-th number
$n = $i;
$fn1 =~ /($nums[0]->[$i])/g; # to find position
$pos = $-[$i];
say "It is $i-th number in a string. Position: $pos";
last NUMS;
}
}
}
We loop over the array with arrayrefs of numbers found in each string, and over elements of each arrayref (eg [8, 400]). Each number in a string (0th or 1st or ...) is compared to its counterpart in the 0-th string (array element); all other numbers are the same.
The number of interest is the one that differs and we record which number in a string it is ($n-th).
Then its position in the string is found by matching it again and using #- regex variable with (the just established) index $n, so the offset of the start of the n-th match. This part may be unneeded; while question edits helped I am still unsure whether the position may or not be useful.
Prints, with position counting from 0
It is 0-th number in a string. Position: 7
Note that, once it is found that it is the $i-th number, we can't use index to find its position; an number earlier in strings may happen to be the same as the $i-th one, in this string.
To test, modify input strings by adding the same number to each, before the one of interest.
Per question update, to examine the sequence (for missing files for instance), with the above findings you can collect counters for all strings in an array with hashrefs (num => filename)
use Data::Dump qw(dd);
my #seq = map { { $num[$_]->[$n] => $fnames[$_] } } 0..$#fnames;
dd \#seq;
where #fnames contains filenames (like two picked for the example above, $fn1 and $fn2). This assumes that the file list was sorted to begin with, or add the sort if it wasn't
my #seq =
sort { (keys %$a)[0] <=> (keys %$b)[0] }
map { { $num[$_]->[$n] => $fnames[$_] } }
0..$#fnames;
The order is maintained by array.
Adding this to the above example (with two strings) adds to the print
[
{ 8 => "segment8_400_av.ts" },
{ 12 => "segment12_400_av.ts" },
]
With this all goals in "Edit 2" should be straighforward.
I suggest that you build a regex pattern by changing all digit sequences to (\d+) and then see which captured values have changed
For instance, with segment8_400_av.ts and
segment9_400_av.ts you would generate a pattern /segment(\d+)_(\d+)_av\.ts/. Note that s/\d+/(\d+)/g will return the number of numeric fields, which you will need for the subsequent check
The first would capture 8 and 400 which the second would capture 9 and 400. 8 is different from 9, so it is in that region of the string where the number varies
I can't really write much code as you don't say what sort of result you want from this process
I have an Ispell list of english words (nearly 50 000 words), my homework in Perl is to get quickly (like under one minute) list of all strings, that are substrings of some other word. I have tried solution with two foreach cycles comparing all words, but even with some optimalizations, its still too slow. I think, that right solution could be some clever use of regular expressions on array of words. Do you know how to solve this problem quicky (in Perl)?
I have found fast solution, which can find some all these substrings in about 15 seconds on my computer, using just one thread. Basically, for each word, I have created array of every possible substrings (eliminating substrings which differs only in "s" or "'s" endings):
#take word and return list of all valid substrings
sub split_to_all_valid_subwords {
my $word = $_[0];
my #split_list;
my ($i, $j);
for ($i = 0; $i < length($word); ++$i){
for ($j = 1; $j <= length($word) - $i; ++$j){
unless
(
($j == length($word)) or
($word =~ m/s$/ and $i == 0 and $j == length($word) - 1) or
($word =~ m/\'s$/ and $i == 0 and $j == length($word) - 2)
)
{
push(#split_list, substr($word, $i, $j));
}
}
}
return #split_list;
}
Then I just create list of all candidates for substrings and make intersection with words:
my #substring_candidates;
foreach my $word (#words) {
push( #substring_candidates, split_to_all_valid_subwords($word));
}
#make intersection between substring candidates and words
my %substring_candidates=map{$_ =>1} #substring_candidates;
my %words=map{$_=>1} #words;
my #substrings = grep( $substring_candidates{$_}, #words );
Now in substrings I have array of all words, that are substrings of some other words.
Perl regular expressions will optimize patterns like foo|bar|baz into an Aho-Corasick match - up to a certain limit of total compiled regex length. Your 50000 words will probably exceed that length, but could be broken into smaller groups. (Indeed, you probably want to break them up by length and only check words of length N for containing words of length 1 through N-1.)
Alternatively, you could just implement Aho-Corasick in your perl code - that's kind of fun to do.
update
Ondra supplied a beautiful solution in his answer; I leave my post here as an example of overthinking a problem and failed optimisation techniques.
My worst case kicks in for a word that doesn't match any other word in the input. In that case, it goes quadratic. The OPT_PRESORT was a try to advert the worst case for most words. The OPT_CONSECUTIVE was a linear-complexity filter that reduced the total number of items in the main part of the algorithm, but it is just a constant factor when considering the complexity. However, it is still useful with Ondras algorithm and saves a few seconds, as building his split list is more expensive than comparing two consecutive words.
I updated the code below to select ondras algorithm as a possible optimisation. Paired with zero threads and the presort optimisation, it yields maximum performance.
I would like to share a solution I coded. Given an input file, it outputs all those words that are a substring of any other word in the same input file. Therefore, it computes the opposite of ysth's ideas, but I took the idea of optimisation #2 from his answer. There are the following three main optimisations that can be deactivated if required.
Multithreading
The questions "Is word A in list L? Is word B in L?" can be easily parallelised.
Pre-sorting all the words for their length
I create an array that points to the list of all words that are longer than a certain length, for every possible length. For long words, this can cut down the number of possible words dramatically, but it trades quite a lot of space, as one word of length n appears in all lists from length 1 to length n.
Testing consecutive words
In my /usr/share/dict/words, most consecutive lines look quite similar:
Abby
Abby's
for example. As every word that would match the first word also matches the second one, I immediately add the first word to the list of matching words, and only keep the second word for further testing. This saved about 30% of words in my test cases. Because I do that before optimisation No 2, this also saves a lot of space. Another trade-off is that the output will not be sorted.
The script itself is ~120 lines long; I explain each sub before showing it.
head
This is just a standard script header for multithreading. Oh, and you need perl 5.10 or better to run this. The configuration constants define the optimisation behaviour. Add the number of processors of your machine in that field. The OPT_MAX variable can take the number of words you want to process, however this is evaluated after the optimisations have taken place, so the easy words will already have been caught by the OPT_CONSECUTIVE optimisation. Adding anything there will make the script seemingly slower. $|++ makes sure that the status updates are shown immediately. I exit after the main is executed.
#!/usr/bin/perl
use strict; use warnings; use feature qw(say); use threads;
$|=1;
use constant PROCESSORS => 0; # (false, n) number of threads
use constant OPT_MAX => 0; # (false, n) number of words to check
use constant OPT_PRESORT => 0; # (true / false) sorts words by length
use constant OPT_CONSECUTIVE => 1; # (true / false) prefilter data while loading
use constant OPT_ONDRA => 1; # select the awesome Ondra algorithm
use constant BLABBER_AT => 10; # (false, n) print progress at n percent
die q(The optimisations Ondra and Presort are mutually exclusive.)
if OPT_PRESORT and OPT_ONDRA;
exit main();
main
Encapsulates the main logic, and does multi-threading. The output of n words will be matched will be considerably smaller than the number of input words, if the input was sorted. After I have selected all matched words, I print them to STDOUT. All status updates etc. are printed to STDERR, so that they don't interfere with the output.
sub main {
my #matching; # the matching words.
my #words = load_words(\#matching); # the words to be searched
say STDERR 0+#words . " words to be matched";
my $prepared_words = prepare_words(#words);
# do the matching, possibly multithreading
if (PROCESSORS) {
my #threads =
map {threads->new(
\&test_range,
$prepared_words,
#words[$$_[0] .. $$_[1]] )
} divide(PROCESSORS, OPT_MAX || 0+#words);
push #matching, $_->join for #threads;
} else {
push #matching, test_range(
$prepared_words,
#words[0 .. (OPT_MAX || 0+#words)-1]);
}
say STDERR 0+#matching . " words matched";
say for #matching; # print out the matching words.
0;
}
load_words
This reads all the words from the input files which were supplied as command line arguments. Here the OPT_CONSECUTIVE optimisation takes place. The $last word is either put into the list of matching words, or into the list of words to be matched later. The -1 != index($a, $b) decides if the word $b is a substring of word $a.
sub load_words {
my $matching = shift;
my #words;
if (OPT_CONSECUTIVE) {
my $last;
while (<>) {
chomp;
if (defined $last) {
push #{-1 != index($_, $last) ? $matching : \#words}, $last;
}
$last = $_;
}
push #words, $last // ();
} else {
#words = map {chomp; $_} <>;
}
#words;
}
prepare_words
This "blows up" the input words, sorting them after their length into each slot, that has the words of larger or equal length. Therefore, slot 1 will contain all words. If this optimisation is deselected, it is a no-op and passes the input list right through.
sub prepare_words {
if (OPT_ONDRA) {
my $ondra_split = sub { # evil: using $_ as implicit argument
my #split_list;
for my $i (0 .. length $_) {
for my $j (1 .. length($_) - ($i || 1)) {
push #split_list, substr $_, $i, $j;
}
}
#split_list;
};
return +{map {$_ => 1} map &$ondra_split(), #_};
} elsif (OPT_PRESORT) {
my #prepared = ([]);
for my $w (#_) {
push #{$prepared[$_]}, $w for 1 .. length $w;
}
return \#prepared;
} else {
return [#_];
}
}
test
This tests if the word $w is a substring in any of the other words. $wbl points to the data structure that was created by the previous sub: Either a flat list of words, or the words sorted by length. The appropriate algorithm is then selected. Nearly all of the running time is spent in this loop. Using index is much faster than using a regex.
sub test {
my ($w, $wbl) = #_;
my $l = length $w;
if (OPT_PRESORT) {
for my $try (#{$$wbl[$l + 1]}) {
return 1 if -1 != index $try, $w;
}
} else {
for my $try (#$wbl) {
return 1 if $w ne $try and -1 != index $try, $w;
}
}
return 0;
}
divide
This just encapsulates an algorithm that guarantees a fair distribution of $items items into $parcels buckets. It outputs the bounds of a range of items.
sub divide {
my ($parcels, $items) = #_;
say STDERR "dividing $items items into $parcels parcels.";
my ($min_size, $rest) = (int($items / $parcels), $items % $parcels);
my #distributions =
map [
$_ * $min_size + ($_ < $rest ? $_ : $rest),
($_ + 1) * $min_size + ($_ < $rest ? $_ : $rest - 1)
], 0 .. $parcels - 1;
say STDERR "range division: #$_" for #distributions;
return #distributions;
}
test_range
This calls test for each word in the input list, and is the sub that is multithreaded. grep selects all those elements in the input list where the code (given as first argument) return true. It also regulary outputs a status message like thread 2 at 10% which makes waiting for completition much easier. This is a psychological optimisation ;-).
sub test_range {
my $wbl = shift;
if (BLABBER_AT) {
my $range = #_;
my $step = int($range / 100 * BLABBER_AT) || 1;
my $i = 0;
return
grep {
if (0 == ++$i % $step) {
printf STDERR "... thread %d at %2d%%\n",
threads->tid,
$i / $step * BLABBER_AT;
}
OPT_ONDRA ? $wbl->{$_} : test($_, $wbl)
} #_;
} else {
return grep {OPT_ONDRA ? $wbl->{$_} : test($_, $wbl)} #_;
}
}
invocation
Using bash, I invoked the script like
$ time (head -n 1000 /usr/share/dict/words | perl script.pl >/dev/null)
Where 1000 is the number of lines I wanted to input, dict/words was the word list I used, and /dev/null is the place I want to store the output list, in this case, throwing the output away. If the whole file should be read, it can be passed as an argument, like
$ perl script.pl input-file >output-file
time just tells us how long the script ran. Using 2 slow processors and 50000 words, it executed in just over two minutes in my case, which is actually quite good.
update: more like 6–7 seconds now, with the Ondra + Presort optimisation, and no threading.
further optimisations
update: overcome by better algorithm. This section is no longer completely valid.
The multithreading is awful. It allocates quite some memory and isn't exactly fast. This isn't suprising considering the amount of data. I considered using a Thread::Queue, but that thing is slow like $#*! and therefore is a complete no-go.
If the inner loop in test was coded in a lower-level language, some performance might be gained, as the index built-in wouldn't have to be called. If you can code C, take a look at the Inline::C module. If the whole script were coded in a lower language, array access would also be faster. A language like Java would also make the multithreading less painful (and less expensive).
I need to split this string to an array:
$string = "9583526578','9583636523','9673522574','9183556528','9983023378";
Here's how I want my array to look like after splitting:
#foo = [
[9583526578, 9583636523],
[9673522574, 9183556528],
[9983023378]
]
As you might have noticed, I need to split this string into groups of n (2 in this example) but still consider remainder if it doesn't match with n.
How can this be done in Perl?
I've done my research and experimentations but can't seem to get it right after a couple of hours.
Thanks for your time! :)
If you can trust they're all integers, extraction is easy. Just grab all the integers.
my #numbers = $string =~ /(\d+)/g;
Then splitting them into pieces of two...
push #matrix, [splice #numbers, 0, 2] while #numbers;
Not as memory efficient as doing it in place, but simple code (if you grok list processing).
If the only reason you're splitting them into pairs is to process them in pairs, you can destructively iterate through the array...
while( my #pair = splice #numbers, 0, 2 ) {
...
}
Or you can iterate in pairs in one of the rare valid uses of a 3-part for loop in Perl.
for(
my $idx = 0;
my #pair = #numbers[$idx, $idx+1];
$idx += 2;
)
{
...
}
Finally, you can get fancy and use perl5i.
use perl5i::2;
#numbers->foreach( func($first, $second) { ... } );
You can also use List::MoreUtils natatime.
First split on "','" to give you an array, then group the elements as desired.
I would recommend using regex to retrieve the numbers using '([0-9]+)' and just manually building #foo. Or as #MRAB suggested, split is even more straight-forward. Any reason you are aiming at Regex for this?
Tons of different ways. Here's one:
$foo[0] = []; # assuming you really meant an array of arrays of arrays as you showed
while ($string =~ m/([0-9]++)[^0-9]*+([0-9]++)?/g) {
push #{ $foo[0] }, [ $1, $2 // () ];
}
(Did you really mean an array containing just one reference to an array of arrayrefs?)
This should be a reasonably close fit to your desired behavior:
my #foo;
push #foo, [ $1, $2 // () ] while $string =~ / (\d+) (?: \D+ (\d+) ) ? /gx;
Not really getting the point of the map function. Can anyone explain with examples its use?
Are there any performance benefits to using this instead of a loop or is it just sugar?
Any time you want to generate a list based another list:
# Double all elements of a list
my #double = map { $_ * 2 } (1,2,3,4,5);
# #double = (2,4,6,8,10);
Since lists are easily converted pairwise into hashes, if you want a hash table for objects based on a particular attribute:
# #user_objects is a list of objects having a unique_id() method
my %users = map { $_->unique_id() => $_ } #user_objects;
# %users = ( $id => $obj, $id => $obj, ...);
It's a really general purpose tool, you have to just start using it to find good uses in your applications.
Some might prefer verbose looping code for readability purposes, but personally, I find map more readable.
First of all, it's a simple way of transforming an array: rather than saying e.g.
my #raw_values = (...);
my #derived_values;
for my $value (#raw_values) {
push (#derived_values, _derived_value($value));
}
you can say
my #raw_values = (...);
my #derived_values = map { _derived_value($_) } #raw_values;
It's also useful for building up a quick lookup table: rather than e.g.
my $sentence = "...";
my #stopwords = (...);
my #foundstopwords;
for my $word (split(/\s+/, $sentence)) {
for my $stopword (#stopwords) {
if ($word eq $stopword) {
push (#foundstopwords, $word);
}
}
}
you could say
my $sentence = "...";
my #stopwords = (...);
my %is_stopword = map { $_ => 1 } #stopwords;
my #foundstopwords = grep { $is_stopword{$_} } split(/\s+/, $sentence);
It's also useful if you want to derive one list from another, but don't particularly need to have a temporary variable cluttering up the place, e.g. rather than
my %params = ( username => '...', password => '...', action => $action );
my #parampairs;
for my $param (keys %params) {
push (#parampairs, $param . '=' . CGI::escape($params{$param}));
}
my $url = $ENV{SCRIPT_NAME} . '?' . join('&', #parampairs);
you say the much simpler
my %params = ( username => '...', password => '...', action => $action );
my $url = $ENV{SCRIPT_NAME} . '?'
. join('&', map { $_ . '=' . CGI::escape($params{$_}) } keys %params);
(Edit: fixed the missing "keys %params" in that last line)
The map function is used to transform lists. It's basically syntactic sugar for replacing certain types of for[each] loops. Once you wrap your head around it, you'll see uses for it everywhere:
my #uppercase = map { uc } #lowercase;
my #hex = map { sprintf "0x%x", $_ } #decimal;
my %hash = map { $_ => 1 } #array;
sub join_csv { join ',', map {'"' . $_ . '"' } #_ }
See also the Schwartzian transform for advanced usage of map.
It's also handy for making lookup hashes:
my %is_boolean = map { $_ => 1 } qw(true false);
is equivalent to
my %is_boolean = ( true => 1, false => 1 );
There's not much savings there, but suppose you wanted to define %is_US_state?
map is used to create a list by transforming the elements of another list.
grep is used to create a list by filtering elements of another list.
sort is used to create a list by sorting the elements of another list.
Each of these operators receives a code block (or an expression) which is used to transform, filter or compare elements of the list.
For map, the result of the block becomes one (or more) element(s) in the new list. The current element is aliased to $_.
For grep, the boolean result of the block decides if the element of the original list will be copied into the new list. The current element is aliased to $_.
For sort, the block receives two elements (aliased to $a and $b) and is expected to return one of -1, 0 or 1, indicating whether $a is greater, equal or less than $b.
The Schwartzian Transform uses these operators to efficiently cache values (properties) to be used in sorting a list, especially when computing these properties has a non-trivial cost.
It works by creating an intermediate array which has as elements array references with the original element and the computed value by which we want to sort. This array is passed to sort, which compares the already computed values, creating another intermediate array (this one is sorted) which in turn is passed to another map which throws away the cached values, thus restoring the array to its initial list elements (but in the desired order now).
Example (creates a list of files in the current directory sorted by the time of their last modification):
#file_list = glob('*');
#file_modify_times = map { [ $_, (stat($_))[8] ] } #file_list;
#files_sorted_by_mtime = sort { $a->[1] <=> $b->[1] } #file_modify_times;
#sorted_files = map { $_->[0] } #files_sorted_by_mtime;
By chaining the operators together, no declaration of variables is needed for the intermediate arrays;
#sorted_files = map { $_->[0] } sort { $a->[1] <=> $b->[1] } map { [ $_, (stat($_))[8] ] } glob('*');
You can also filter the list before sorting by inserting a grep (if you want to filter on the same cached value):
Example (a list of the files modified in the last 24 hours sorted the last modification time):
#sorted_files = map { $_->[0] } sort { $a->[1] <=> $b->[1] } grep { $_->[1] > (time - 24 * 3600 } map { [ $_, (stat($_))[8] ] } glob('*');
The map function is an idea from the functional programming paradigm. In functional programming, functions are first-class objects, meaning that they can be passed as arguments to other functions. Map is a simple but a very useful example of this. It takes as its arguments a function (lets call it f) and a list l. f has to be a function taking one argument, and map simply applies f to every element of the list l. f can do whatever you need done to every element: add one to every element, square every element, write every element to a database, or open a web browser window for every element, which happens to be a valid URL.
The advantage of using map is that it nicely encapsulates iterating over the elements of the list. All you have to do is say "do f to every element, and it is up to map to decide how best to do that. For example map may be implemented to split up its work among multiple threads, and it would be totally transparent to the caller.
Note, that map is not at all specific to Perl. It is a standard technique used by functional languages. It can even be implemented in C using function pointers, or in C++ using "function objects".
The map function runs an expression on each element of a list, and returns the list results. Lets say I had the following list
#names = ("andrew", "bob", "carol" );
and I wanted to capitalize the first letter of each of these names. I could loop through them and call ucfirst of each element, or I could just do the following
#names = map (ucfirst, #names);
"Just sugar" is harsh. Remember, a loop is just sugar -- if's and goto can do everything loop constructs do and more.
Map is a high enough level function that it helps you hold much more complex operations in your head, so you can code and debug bigger problems.
To paraphrase "Effective Perl Programming" by Hall & Schwartz,
map can be abused, but I think that it's best used to create a new list from an existing list.
Create a list of the squares of 3,2, & 1:
#numbers = (3,2,1);
#squares = map { $_ ** 2 } #numbers;
Generate password:
$ perl -E'say map {chr(32 + 95 * rand)} 1..16'
# -> j'k=$^o7\l'yi28G
You use map to transform a list and assign the results to another list, grep to filter a list and assign the results to another list. The "other" list can be the same variable as the list you are transforming/filtering.
my #array = ( 1..5 );
#array = map { $_+5 } #array;
print "#array\n";
#array = grep { $_ < 7 } #array;
print "#array\n";
It allows you to transform a list as an expression rather than in statements. Imagine a hash of soldiers defined like so:
{ name => 'John Smith'
, rank => 'Lieutenant'
, serial_number => '382-293937-20'
};
then you can operate on the list of names separately.
For example,
map { $_->{name} } values %soldiers
is an expression. It can go anywhere an expression is allowed--except you can't assign to it.
${[ sort map { $_->{name} } values %soldiers ]}[-1]
indexes the array, taking the max.
my %soldiers_by_sn = map { $->{serial_number} => $_ } values %soldiers;
I find that one of the advantages of operational expressions is that it cuts down on the bugs that come from temporary variables.
If Mr. McCoy wants to filter out all the Hatfields for consideration, you can add that check with minimal coding.
my %soldiers_by_sn
= map { $->{serial_number}, $_ }
grep { $_->{name} !~ m/Hatfield$/ }
values %soldiers
;
I can continue chaining these expression so that if my interaction with this data has to reach deep for a particular purpose, I don't have to write a lot of code that pretends I'm going to do a lot more.
It's used anytime you would like to create a new list from an existing list.
For instance you could map a parsing function on a list of strings to convert them to integers.
As others have said, map creates lists from lists. Think of "mapping" the contents of one list into another. Here's some code from a CGI program to take a list of patent numbers and print hyperlinks to the patent applications:
my #patents = ('7,120,721', '6,809,505', '7,194,673');
print join(", ", map { "$_" } #patents);
As others have said, map is most useful for transforming a list. What hasn't been mentioned is the difference between map and an "equivalent" for loop.
One difference is that for doesn't work well for an expression that modifies the list its iterating over. One of these terminates, and the other doesn't:
perl -e '#x=("x"); map { push #x, $_ } #x'
perl -e '#x=("x"); push #x, $_ for #x'
Another small difference is that the context inside the map block is a list context, but the for loop imparts a void context.