Hi I am trying to rename 1000 of files from
xyzPL 7-1-16 page1+2(1).xlsx into 7-1-16.xlsx
xyzPL 12-1-16 page1+2(1).xls into 12-1-16.xls
xyzPL 12-10-16 page1+2(1).xls into 12-10-16.xls
So far I have following for loop
for in *.xls; do echo mv "$f" "${f/_*_/_}"; done
What the expression should I put for ${f/_*_/_}
Thank you!
I'd suggest looking into the rename (or, on some platforms, prename) facility.
It's not part of bash itself but should be available in all the regular distros.
It allows Perl regular expressions to be used to rename files and will almost certainly be a good sight quicker than a bash-based for loop.
By way of example, the following command should handle the three cases you've shown:
rename -n 's/^xyzPL (\d+-\d+-\d+).?*\.(xlsx?)$/$1.$2/' xyzPL*.xls xyzPL*.xlsx
It captures the n-n-n bit into $1 and the file extension into $2 and then performs a simple (as if anything in Perl could be considered simple) substitution.
Note particularly the -n flag, this will print out what the command will do without actually doing it. It's very useful for checking what will happen before actually doing it.
Once you're satisfied it won't screw up everything, just run it again without the -n. Of course, being the paranoid type, I'd tend to back up the entire directory anyway.
A slightly souped-up version of the Perl-based rename command, originally from the 1st Edition of the Camel Book (Programming in Perl, by Larry Wall).
#!/usr/bin/env perl
#
# #(#)$Id: rename.pl,v 1.8 2011/06/03 22:30:22 jleffler Exp $
#
# Rename files using a Perl substitute or transliterate command
use strict;
use warnings;
use Getopt::Std;
my(%opts);
my($usage) = "Usage: $0 [-fnxV] perlexpr [filenames]\n";
my($force) = 0;
my($noexc) = 0;
my($trace) = 0;
die $usage unless getopts('fnxV', \%opts);
if ($opts{V})
{
printf "%s\n", q'RENAME Version $Revision: 1.8 $ ($Date: 2011/06/03 22:30:22 $)';
exit 0;
}
$force = 1 if ($opts{f});
$noexc = 1 if ($opts{n});
$trace = 1 if ($opts{x});
my($op) = shift;
die $usage unless defined $op;
if (!#ARGV) {
#ARGV = <STDIN>;
chop(#ARGV);
}
for (#ARGV)
{
if (-e $_ || -l $_)
{
my($was) = $_;
eval $op;
die $# if $#;
next if ($was eq $_);
if ($force == 0 && -f $_)
{
print STDERR "rename failed: $was - $_ exists\n";
}
else
{
print "+ $was --> $_\n" if $trace;
print STDERR "rename failed: $was - $!\n"
unless ($noexc || rename($was, $_));
}
}
else
{
print STDERR "$_ - $!\n";
}
}
Without using rename utility you can do this in pure BASH:
for file in *.xls*; do
f="${file#* }"
mv "$file" "${f/ *./.}"
done
You could do a subshell with the file name piped to awk...
$(echo "$f" | awk -F '[ .]' '{ print $2 "." $4 }')
Related
My bash script uses a perl command to replace a variable within a file, with the contents of another file. Pretty standard I thought, yet I am really struggling here.
#/bin/bash
display_usage() {
echo -e "\nUsage: This script must be run with an environment parameter (file in directory env/)."
echo -e "Example: ./configureEnv <env parameter>\n"
}
# If no env argument supplied, display usage.
if [ $# -eq 0 ]; then
display_usage
exit 1
fi
# Replace the placeholder of FIREBASE_ENV in index.html with the Firebase env settings.
perl -pi -e 's/%FIREBASE_ENV%/`cat testEnvConfig`/g;' index.html
if [ $? -eq 0 ]; then
echo "Updated Firebase settings based on environment file: $1"
exit 0
else
echo "[Error] Environment settings configuration failed. Please check parameters are correct."
exit 1
fi
As you can see, the key line is:
perl -pi -e 's/%FIREBASE_ENV%/`cat testEnvConfig`/g;' index.html
It should replace the placeholder string of %FIREBASE_ENV% with the contents of the file, but instead it replaces replaces %FIREBASE_ENV% with `cat testEnvConfig`.
The replacement in a substitution interpolates variables like double quotes, but it doesn't interpret backticks. You need to specify the /e modifier to evaluate the replacement as code.
s/%FIREBASE_ENV%/`cat testEnvConfig`/ge
You don't need to shell out for this, though. Perl can read a file and store its contents in a variable:
my $config = do { local( #ARGV, $/ ) = 'testEnvConfig'; <> };
# ...
s/%FIREBASE_ENV%/$config/g;
With a (very useful) Perl module Path::Tiny reading a file takes one statement and we have
perl -MPath::Tiny -i.bak -pe'
BEGIN { $f = path("testEnvConfig")->slurp; chomp $f };
s/%FIREBASE_ENV%/$f/g
' index.html
I've added a backup .bak for your safety while testing, and split code over lines for readability.
I remove the final newline from the file. If you actually want it remove that chomp $f
Why your honest attempt doesn't work has been explained, as you need to evaluate the replacement part as code, with /e. However, once you do that, look at the situation:
out of a bash script call a Perl program (the one-liner)
which goes out to the system (a syscall at least, perhaps another shell!)
where yet another program runs (cat)
with the file content sent back to Perl, which then does its thing.
Huh. Better just read that file in Perl, no?
By default the right-hand side of a substitution (s///) works like a double-quoted string. "`cat whatever`" doesn't do anything special in a string; it doesn't run any commands.
You need to use the /e flag:
s/%FIREBASE_ENV%/`cat testEnvConfig`/eg
This tells Perl that the right-hand side is to be evaluated as a block of code, not a string.
Alternatively you can avoid shelling out to cat by reading the file in Perl:
perl -pi -e 'BEGIN { my $file = "testEnvConfig"; open my $fh, "<", $file or die "$file: $!\n"; local $/; $config = readline $fh } s/%FIREBASE_ENV%/$config/g'
This also avoids having to re-read the file for each occurrence of %FIREBASE_ENV%.
I have a file in which we have entries in following format. I would like to increment the numbers in file names inside this file. So some_v1.png will become some_v2.png. Is there a way with regex OR command line utility to achieve this.
Following is example file (file.config) with file entries as string.
something/some_v1.png
something/some_v4.png
something/some_v3.png
This looks like a great match for awk's "split" function:
awk '{n=split($0,a,"[1-9][0-9]*",s);for(i=1;i<n;++i)printf "%s%d",a[i],s[i]+1;print a[n]}'
The perl one-liner you already found also works great, with one exception: files with leading-zero numbers will lose the zeroes. Here is a fix for that using the magical auto-increment:
perl -pe 's/(\d+)/++($a=$1)/eg'
If you want to rename a bunch of files I'd use an auxiliary directory and a test to see if there is an actual file to rename.
mkdir aux
for i in {1..7} ; do
j=$($i + 1)
[ -f something/some_v${i}.png ] && mv something/some_v${i}.png aux/some_v${j}.png
done
mv aux/* something
rmdir aux
The use of a fixed name for the auxiliary directory could not stand a security review for repeated use in a dynamic production environment but I think it's fine for a one shot use in a controlled environment.
In perl:
#!/usr/bin/env perl
use strict;
use warnings;
foreach
my $filename (
sort { $b =~ s/.*(\d+).*/$1/r <=> $a =~ s/.*(\d+).*/$1/r }
glob "something/some_v*.png" )
{
chomp $filename;
if ( my ($vnum) = $filename =~ m/(\d+)\.png/ ) {
print "mv $filename ", $filename =~ s|\d+\.png|++$vnum.".png"|re,
"\n";
}
}
Note - sorting numerically, to ensure that you're never replacing 5 with 4, before you've renamed 5.
I would like to quickly search and replace with or without regexp in files recursively. In addition, I need to search only in specific files and I do not want to touch the files that do not match my search_pattern otherwise git will think all the parsed files were modified (it what happens with find . --exec sed).
I tried many solutions that I found on internet using find, grep, sed or ack but I don't think they are really good to match specific files only.
Eventually I wrote this perl script:
#!/bin/perl
use strict;
use warnings;
use File::Find;
my $search_pattern = $ARGV[0];
my $replace_pattern = $ARGV[1];
my $file_pattern = $ARGV[2];
my $do_replace = 0;
sub process {
return unless -f;
return unless /(.+)[.](c|h|inc|asm|mac|def|ldf|rst)$/;
open F, $_ or print "couldn't open $_\n" && return;
my $file = $_;
my $i = 0;
while (<F>) {
if (m/($search_pattern)/o) {$i++};
}
close F;
if ($do_replace and $i)
{
printf "found $i occurence(s) of $search_pattern in $file\n";
open F, "+>".$file or print "couldn't open $file\n" && return;
while (<F>)
{
s/($search_pattern)/($replace_pattern)/g;
print F;
}
close F;
}
}
find(\&process, ".");
My question is:
Is there any better solution like this one below (which not exists) ?
`repaint -n/(.+)[.](c|h|inc|asm|mac|def|ldf|rst)$/ s/search/replacement/g .`
Subsidiary questions:
How's my perl script ? Not too bad ? Do I really need to reopen every files that match my search_pattern ?
How people deal with this trivial task ? Almost every good text editor have a "Search and Replace in files" feature, but not vim. How vim users can do this ?
Edit:
I also tried this script ff.pl with ff | xargs perl -pi -e 's/foo/bar/g' but it doesnt work as I expected. It created a backup .bak even though I didn't give anything after the -pi. It seems it is the normal behaviour within cygwin but with this I cannot really use perl -pi -e
#!/bin/perl
use strict;
use warnings;
use File::Find;
use File::Basename;
my $ext = $ARGV[0];
sub process {
return unless -f;
return unless /\.(c|h|inc|asm|mac|def|ldf|rst)$/;
print $File::Find::name."\n" ;
}
find(\&process, ".");
Reedit:
I finally came across this solution (under cygwin I need to remove the backup files)
find . | egrep '\.(c|h|asm|inc)$' | xargs perl -pi.winsucks -e 's/<search>/<replace>/g'
find . | egrep '\.(c|h|asm|inc)\.winsucks$' | xargs rm
The following is a cleaned up version of your code.
Always include use strict; and use warnings at the top of EVERY perl script. If you're doing file processing, include use autodie; as well.
Go ahead and slurp the entire file. That way you only have to read and write optionally write it once.
Consider using File::Find::Rule for cases like this. Your implmentation using File::Find works, and actually is probably the preferred module in this case, but I like the interface for the latter.
I removed the capture groups from the regex. In ones in the RHS were a bug, and the ones in the LHS were superfluous.
And the code:
use strict;
use warnings;
use autodie;
use File::Find;
my $search_pattern = $ARGV[0];
my $replace_pattern = $ARGV[1];
my $file_pattern = $ARGV[2];
my $do_replace = 0;
sub process {
return if !-f;
return if !/[.](?:c|h|inc|asm|mac|def|ldf|rst)$/;
my $data = do {
open my $fh, '<', $_;
local $/;
<$fh>;
};
my $count = $data =~ s/$search_pattern/$replace_pattern/g
or return;
print "found $count occurence(s) of $search_pattern in $_\n";
return if !$do_replace;
open my $fh, '>', $_;
print $fh $data;
close $fh;
}
find(\&process, ".");
Not bad, but several minor notes:
$do_replace is always 0 so it will not replace
in-place open F, "+>" will not work on cygwin + windows
m/($search_pattern)/o /o is good, () is not needed.
$file_pattern is ignored, you overwrite it with your own
s/($search_pattern)/($replace_pattern)/g;
() is unneeded and will actually disturb a counter in the $replace_pattern
/(.+)[.](c|h|inc|asm|mac|def|ldf|rst)$/ should be written as
/\.(c|h|inc|asm|mac|def|ldf|rst)$/ and maybe /i also
Do I really need to reopen every files that match my search_pattern ?
You don't do.
Have no idea about vim, I use emacs, which has several method to accomplish this.
What's wrong with the following command?
:grep foo **/*.{foo,bar,baz}
:cw
It won't cause any problem with any VCS and is pretty basic Vimming.
You are right that Vim doesn't come with a dedicated "Search and Replace in files" feature but there are plugins for that.
why not just:
grep 'pat' -rl *|xargs sed -i 's/pat/rep/g'
or I didn't understand the Q right?
I suggest find2perl if it doesn't work out of the box, you can tweak the code it generates:
find2perl /tmp \! -name ".*?\.(c|h|inc|asm|mac|def|ldf|rst)$" -exec "sed -e s/aaa/bbb/g {}"
it will print the following code to stdout:
#! /usr/bin/perl -w
eval 'exec /usr/bin/perl -S $0 ${1+"$#"}'
if 0; #$running_under_some_shell
use strict;
use File::Find ();
# Set the variable $File::Find::dont_use_nlink if you're using AFS,
# since AFS cheats.
# for the convenience of &wanted calls, including -eval statements:
use vars qw/*name *dir *prune/;
*name = *File::Find::name;
*dir = *File::Find::dir;
*prune = *File::Find::prune;
sub wanted;
sub doexec ($#);
use Cwd ();
my $cwd = Cwd::cwd();
# Traverse desired filesystems
File::Find::find({wanted => \&wanted}, '/tmp');
exit;
sub wanted {
my ($dev,$ino,$mode,$nlink,$uid,$gid);
(($dev,$ino,$mode,$nlink,$uid,$gid) = lstat($_)) &&
! /^\..*.?\\.\(c|h|inc|asm|mac|def|ldf|rst\)\$\z/s &&
doexec(0, 'sed -e s/aaa/bbb/g {}');
}
sub doexec ($#) {
my $ok = shift;
my #command = #_; # copy so we don't try to s/// aliases to constants
for my $word (#command)
{ $word =~ s#{}#$name#g }
if ($ok) {
my $old = select(STDOUT);
$| = 1;
print "#command";
select($old);
return 0 unless <STDIN> =~ /^y/;
}
chdir $cwd; #sigh
system #command;
chdir $File::Find::dir;
return !$?;
}
If you want to execute, you can pipe it to perl:
find2perl /tmp \! -name ".*?\.(c|h|inc|asm|mac|def|ldf|rst)$" -exec "sed -e s/aaa/bbb/g" | perl
You can try this plugin for Vim:
https://github.com/skwp/greplace.vim
Basically, it allows you to type in a search phases (with/without regex) and ask you for the files to search in.
I have a Perl script that strips comments from other Perl scripts:
open (INFILE, $file);
#data = <INFILE>;
foreach $data (#data)
{
$data =~ s/#.*/ /g;
print "$data";
}
The problem is, this code also removes the shebang line:
#!/usr/bin/perl
How can I strip comments except for the shebang?
Writing code to strip comments is not trivial, since the # character can be used in other contexts than just comments. Use perltidy instead:
perltidy --delete-block-comments --delete-side-comments foo
will strip # comments (but not POD) from file foo and write the output to foo.tdy. The shebang is not stripped.
There is a method PPR::decomment() that can be used:
use strict;
use warnings;
use PPR;
my $document = <<'EOF';
print "\n###################################\n";
print '\n###################################\n';
print '\nFollowed by comment \n'; # The comment
return $function && $function !~ /^[\s{}#]/;
EOF
my $res = PPR::decomment( $document );
print $res;
Output:
print "\n###################################\n";
print '\n###################################\n';
print '\nFollowed by comment \n';
return $function && $function !~ /^[\s{}#]/;
perltidy is the method to do this if it's anything but an exercise. There's also PPI for parsing perl. Could use the PPI::Token::Comment token to do something more complicated than just stripping.
However, to answer your direct question, don't try to solve everything in a single regex. Instead, break up your problems into logic pieces of information and logic. In this instead, if you want to skip the first line, do so by using line by line processing which conveniently sets the current line number in $.
use strict;
use warnings;
use autodie;
my $file = '... your file...';
open my $fh, '<', $file;
while (<$fh>) {
if ($. != 1) {
s/#.*//;
}
print;
}
Disclaimer
The approach of using regex's for this problem is definitely flawed as everyone has already said. However, I'm going to give your instructor the benefit of the doubt, and that she/he is aiming to teach by intentionally giving you a problem that is outside of the perview of regex's ability. Good look finding all of those edge cases and figuring out how to do with them.
Whatever you do, don't try to solve them using a single regex. Break your problem up and use lots of if's and elsif's
Since you asked for a regex solution:
'' =~ /(?{
system("perltidy", "--delete-block-comments", "--delete-side-comments", $file);
die "Can't launch perltidy: $!\n" if $? == -1;
die "perltidy killed by signal ".( $? & 0x7F )."\n" if $? & 0x7F;
die "perltidy exited with error ".( $? >> 8 )."\n" if $? >> 8;
});
It seems like you are leaning towards using the following:
#!/usr/bin/perl
while (<>) {
if ($. != 1) {
s/#.*//;
}
print;
}
But it doesn't work on itself:
$ chmod u+x stripper.pl
$ stripper.pl stripper.pl >stripped_stripper.pl
$ chmod u+x stripped_stripper.pl
$ stripped_stripper.pl stripper.pl
Substitution pattern not terminated at ./stripped_stripper.pl line 4.
$ cat stripped_stripper.pl
#!/usr/bin/perl
while (<>) {
if ($. != 1) {
s/
}
print;
}
It also fails to remove comments on the first line:
$ cat >first.pl
# This is my first Perl program!
print "Hello, World!\n";
$ stripper.pl first.pl
# This is my first Perl program!
print "Hello, World!\n";
I've got a few files named stuff like this: file (2).jpg. I'm writing a little Perl script to rename them, but I get errors due to the brackets not being replaced. So. Can someone tell me how to escape all the brackets (and spaces, if they cause a problem) in a string so I can pass it to a command. The script is below:
#Load all jpgs into an array.
#pix = `ls *.JPG`;
foreach $pix (#pix) {
#Let you know it's working
print "Processing photo ".$pix;
$pix2 = $pix;
$pix2 =~ \Q$pix\E; # Problem line
#Use the program exiv2 to rename the file with timestamp
system("exiv2 -r %Y_%m%d_%H%M%S $pix2");
}
The error is this:
Can't call method "Q" without a package or object reference at script.sh line [problem line].
This is my first time with regex, so I'm looking for the answers that explain what to do as well as giving an answer. Thanks for any help.
Why do not use a simple?
find . -name \*.JPG -exec exiv2 -r "%Y_%m%d_%H%M%S" "{}" \;
Ps:
The \Q disabling pattern metacharacters until \E inside the regex.
For example, if you want match a path "../../../somefile.jpg", you can write:
$file =~ m:\Q../../../somefile.jpg\E:;
instead of
$file =~ m:\.\./\.\./\.\./somefile\.jpg:; #e.g. escaping all "dots" what are an metacharacter for regex.
I found this perl renaming script that was written by Larry Wall a while back... it does what you need and so much more. I keep in in my $PATH, and use it daily...
#!/usr/bin/perl -w
use Getopt::Std;
getopts('ht', \%cliopts);
do_help() if( $cliopts{'h'} );
#
# rename script examples from lwall:
# pRename.pl 's/\.orig$//' *.orig
# pRename.pl 'y/A-Z/a-z/ unless /^Make/' *
# pRename.pl '$_ .= ".bad"' *.f
# pRename.pl 'print "$_: "; s/foo/bar/ if <stdin> =~ /^y/i' *
$op = shift;
for (#ARGV) {
$was = $_;
eval $op;
die $# if $#;
unless( $was eq $_ ) {
if( $cliopts{'t'} ) {
print "mv $was $_\n";
} else {
rename($was,$_) || warn "Cannot rename $was to $_: $!\n";
}
}
}
sub do_help {
my $help = qq{
Usage examples for the rename script example from Larry Wall:
pRename.pl 's/\.orig\$//' *.orig
pRename.pl 'y/A-Z/a-z/ unless /^Make/' *
pRename.pl '\$_ .= ".bad"' *.f
pRename.pl 'print "\$_: "; s/foo/bar/ if <stdin> =~ /^y/i' *
CLI Options:
-h This help page
-t Test only, do not move the files
};
die "$help\n";
return 0;
}