How to find and replace special chars within a string in zsh - replace

I'm trying to build a secure copy protocol quick function. When I run the command it will work with a single file OR with the entire directory, but as soon as I put a /* after the local_repo it returns zsh: no matches found: hackingedu/*.
If I put the command scp hackingedu\/\* hackingedu the command works properly. I think I'm on the right track, but can't get it to work.
contains() {
string="$1"
substring="$2"
if test "${string#*$substring}" != "$string"
then
# echo '$substring is in $string'
return 1 # $substring is in $string
else
# echo '$substring is not in $string'
return 0 # $substring is not in $string
fi
}
# Quickly scp files in Workspace to Remote
function scp() {
local_repo="$1"
remote_repo="$2"
# find all the `*` and replace with `/*`
if [ contains $local_repo '*' ]; then
# replace all instances of * with \* <- HOW TO DO
fi
command scp -r $LOCAL_REPOS/$local_repo $ALEX_SERVER_UNAME#$ALEX_SERVER_PORT:$ALEX_REMOTE_ROOT_PATH/$remote_repo
# Description: $1: Local Repo | $2: Remote Repo
# Define ex: scpp local/path/to/file/or/directory/* remote/path/to/file/or/directory/*
# Live ex: scpp alexcory/index.php alexcory/index.php
# Live ex: scpp alexcory/* alexcory/*
#
# This Saves you from having long commands that look like this:
# scp -r ~/Google\ Drive/server/Dev/git\ repositories/hackingedu/* alexander#alexander.com:/home2/alexander/public_html/hackingedu/beta
}
Command trying to execute: scp -r ~/Google\ Drive/server/Dev/git\ repositories/hackingedu/* alexander#alexander.com:/home2/alexander/public_html/hackingedu/beta
Any ideas on how to find and replace an *? If there's a better way to do this please do tell! :)
If you know how to do this in bash I would like your input as well!
References:
How do you tell if a string contains another string in Unix shell scripting?
ZSH Find command replacement
ZSH Find command replacement 2
Using wildcards in commands with zsh

You can either prefix your scp call using noglob (which will turn off globbing for that command, e.g. noglob ls *) or use
autoload -U url-quote-magic
zle -N self-insert url-quote-magic
zstyle -e :urlglobber url-other-schema '[[ $words[1] == scp ]] && reply=("*") || reply=(http https ftp)'
the above should make zsh auto quote * when you use scp.
[...]
BTW, in any case, you should learn that you can easily quote special characters using ${(q)variable_name}, e.g.
% foo='*&$%normal_chars'
% echo $foo
*&$%normal_chars
% echo ${(q)foo}
\*\&\$%normal_chars

Related

bash download the first matching regex on download page

I want to get the newest (first) download link matching a regex.
URL=https://github.com/sharkdp/bat/releases/ # Need to look at /releases/ even though the downloads are under /releases/download/$REL/$BAT
content=$(wget $URL -q -O -)
# Parse $content for string starting 'https://' and ending "_amd64.deb"
# At the moment, that will be: href="/sharkdp/bat/releases/download/v0.18.3/bat_0.18.3_amd64.deb"
# wget -O to specify the name of the file into which wget dumps the page contents, and then - to get the dump onto standard output. -q (quiet) turns off wget output.
Then I need to somehow grep / match strings that starts https:// and ends _amd64. Then I need to just pick the first one in that list.
How do I grep / match / pick first item in this way?
Once I have that, it's then easy for me to download the latest version on the page, with wget -P /tmp/ $DL
With Bash, you can use
rx='href="(/sharkdp/[^"]*_amd64\.deb)"'
if [[ "$content" =~ $rx ]]; then
echo "${BASH_REMATCH[1]}";
else
echo "No match";
fi
# => /sharkdp/bat/releases/download/v0.18.3/bat-musl_0.18.3_amd64.deb
The href="(/sharkdp/[^"]*_amd64\.deb)" regex matches href=", then captures into Group 1 (${BASH_REMATCH[1]}) /shardp/ + zero or more chars other than " + _amd64.deb and then just matches ".
With GNU grep, you can use
> link=$(grep -oP 'href="\K/sharkdp/[^"]*_amd64\.deb' <<< "$content" | head -1)
> echo "$link"
# => /sharkdp/bat/releases/download/v0.18.3/bat-musl_0.18.3_amd64.deb
Here,
href="\K/sharkdp/[^"]*_amd64\.deb - matches href=", then drops this text from the match, then matches /sharkdp/ + any zero or more chars other than " and then _amd_64.deb
head -1 - only keeps the first match.

How to capture the beginning of a filename using a regex in Bash?

I have a number of files in a directory named edit_file_names.sh, each containing a ? in their name. I want to use a Bash script to shorten the file names right before the ?. For example, these would be my current filenames:
test.file.1?twagdsfdsfdg
test.file.2?
test.file.3?.?
And these would be my desired filenames after running the script:
test.file.1
test.file.2
test.file.3
However, I can't seem to capture the beginning of the filenames in my regex to use in renaming the files. Here is my current script:
#!/bin/bash
cd test_file_name_edit/
regex="(^[^\?]*)"
for filename in *; do
$filename =~ $regex
echo ${BASH_REMATCH[1]}
done
At this point I'm just attempting to print off the beginnings of each filename so that I know that I'm capturing the correct string, however, I get the following error:
./edit_file_names.sh: line 7: test.file.1?twagdsfdsfdg: command not found
./edit_file_names.sh: line 7: test.file.2?: command not found
./edit_file_names.sh: line 7: test.file.3?.?: command not found
How can I fix my code to successfully capture the beginnings of these filenames?
Regex as such may not be the best tool for this job. Instead, I'd suggest using bash parameter expansion. For example:
#!/bin/bash
files=(test.file.1?twagdsfdsfdg test.file.2? test.file.3?.?)
for f in "${files[#]}"; do
echo "${f} shortens to ${f%%\?*}"
done
which prints
test.file.1?twagdsfdsfdg shortens to test.file.1
test.file.2? shortens to test.file.2
test.file.3?.? shortens to test.file.3
Here, ${f%%\?*} expands f and trims the longest suffix that matches a ? followed by any characters (the ? must be escaped since it's a wildcard character).
You miss the test command [[ ]] :
for filename in *; do
[[ $filename =~ $regex ]] && echo ${BASH_REMATCH[1]}
done

what is the Regular Expressions to use to find if a line contain a word?

I would like to check in my script if in a line i have the word device i tried this regex
if[$SERIAL =~ /device/] but the execution result commande unkown. this is my script i try to install apk in only devices that present the stat device so not ofline, you find my script below
for SERIAL in $(adb devices | tail -n +2 | cut -sf 1);
do
if [$SERIAL =~ /device/]
#if [$SERIAL = "/^.*/device\b.*$/m"]
then
cd $1
for APKLIST in $(ls *.apk);
do
echo "Installation de $APKLIST on $SERIAL"
adb -s $SERIAL install -r $1/$APKLIST &
#adb bugreport > bug.txt
done
fi
done
try to change your if line into:
if [[ $SERIAL =~ "device" ]]
After the | cut -sf 1 your $SERIAL variable is going to contain just the serial number. So why are you trying to match it against device? It is never going to be true. Why won't you just use
for SERIAL in $(adb devices | grep device$ | cut -sf 1);
instead of your if?
You must always put a space between the square brackets in your if statement and what you're testing. You must also understand there are two types of square brackets, the [ ... ] standard and the expanded [[ ... ]].
The [ is actually a Unix command which is aliased to the test1 command:
$ ls -li /bin/test /bin/[
54008404 -rwxr-xr-x 2 root wheel 18576 Jul 25 2012 /bin/[
54008404 -rwxr-xr-x 2 root wheel 18576 Jul 25 2012 /bin/test
You can see the basic tests by looking at the test manpage.
Both Kornshell (which started it) and BASH give you the expanded test using double square brackets ([[ ... ]]). The [[ ... ]] is mainly used for pattern matching. Pattern matching extends the globbing that the shell can do on the command line -- sort of a poor man's regular expression. Newer versions of BASH actually allow you to use real regular expressions rather than pattern matching.
The important thing to remember is that you must use the double square brackets whenever you are doing a pattern match testing (or regular expression testing in BASH) rather than the single square brackets which can only test strings.
Also remember that regular expressions can match anywhere on a line:
if [[ $SERIAL =~ "device" ]]
could match devices. What you want to do is to add \b to make sure you're matching the word device. The \b is a regular expression word boundary:
if [[ $SERIAL =~ "\bdevice\b" ]]
1. Yes, I know the [ is a shell builtin in BASH and Kornshell, and [ doesn't execute /bin/[. However, the test command is also a shell builtin too.

Regex to remove lines in file(s) that ending with same or defined letters

i need a bash script for mac osx working in this way:
./script.sh * folder/to/files/
#
# or #
#
./script.sh xx folder/to/files/
This script
read a list of files
open each file and read each lines
if lines ended with the same letters ('*' mode) or with custom letters ('xx') then
remove line and RE-SAVE file
backup original file
My first approach to do this:
#!/bin/bash
# ck init params
if [ $# -le 0 ]
then
echo "Usage: $0 <letters>"
exit 0
fi
# list files in current dir
list=`ls BRUTE*`
for i in $list
do
# prepare regex
case $1 in
"*") REGEXP="^.*(.)\1+$";;
*) REGEXP="^.*[$1]$";;
esac
FILE=$i
# backup file
cp $FILE $FILE.bak
# removing line with same letters
sed -Ee "s/$REGEXP//g" -i '' $FILE
cat $FILE | grep -v "^$"
done
exit 0
But it doesn't work as i want....
What's wrong?
How can i fix this script?
Example:
$cat BRUTE02.dat BRUTE03.dat
aa
ab
ac
ad
ee
ef
ff
hhh
$
If i use '*' i want all files that ended with same letters to be clean.
If i use 'ff' i want all files that ended with 'ff' to be clean.
Ah, it's on Mac OSx. Remember that sed is a little different from classical linux sed.
man sed
sed [-Ealn] command [file ...]
sed [-Ealn] [-e command] [-f command_file] [-i extension] [file
...]
DESCRIPTION
The sed utility reads the specified files, or the standard input
if no files are specified, modifying the input as specified by a list
of commands. The
input is then written to the standard output.
A single command may be specified as the first argument to sed.
Multiple commands may be specified by using the -e or -f options. All
commands are applied
to the input in the order they are specified regardless of their
origin.
The following options are available:
-E Interpret regular expressions as extended (modern)
regular expressions rather than basic regular expressions (BRE's).
The re_format(7) manual page
fully describes both formats.
-a The files listed as parameters for the ``w'' functions
are created (or truncated) before any processing begins, by default.
The -a option causes
sed to delay opening each file until a command containing
the related ``w'' function is applied to a line of input.
-e command
Append the editing commands specified by the command
argument to the list of commands.
-f command_file
Append the editing commands found in the file
command_file to the list of commands. The editing commands should
each be listed on a separate line.
-i extension
Edit files in-place, saving backups with the specified
extension. If a zero-length extension is given, no backup will be
saved. It is not recom-
mended to give a zero-length extension when in-place
editing files, as you risk corruption or partial content in situations
where disk space is
exhausted, etc.
-l Make output line buffered.
-n By default, each line of input is echoed to the standard
output after all of the commands have been applied to it. The -n
option suppresses this
behavior.
The form of a sed command is as follows:
[address[,address]]function[arguments]
Whitespace may be inserted before the first address and the
function portions of the command.
Normally, sed cyclically copies a line of input, not including
its terminating newline character, into a pattern space, (unless there
is something left
after a ``D'' function), applies all of the commands with
addresses that select that pattern space, copies the pattern space to
the standard output, append-
ing a newline, and deletes the pattern space.
Some of the functions use a hold space to save all or part of the
pattern space for subsequent retrieval.
anything else?
it's clear my problem?
thanks.
I don't know bash shell too well so I can't evaluate what the failure is.
This is just an observation of the regex as understood (this may be wrong).
The * mode regex looks ok:
^.*(.)\1+$ that ended with same letters..
But the literal mode might not do what you think.
current: ^.*[$1]$ that ended with 'literal string'
This shouldn't use a character class.
Change it to: ^.*$1$
Realize though the string in $1 (before it goes into the regex) should be escaped
incase there are any regex metacharacters contained within it.
Otherwise, do you intend to have a character class?
perl -ne '
BEGIN {$arg = shift; $re = $arg eq "*" ? qr/([[:alpha:]])\1$/ : qr/$arg$/}
/$re/ && next || print
'
Example:
echo "aa
ab
ac
ad
ee
ef
ff" | perl -ne '
BEGIN {$arg = shift; $re = $arg eq "*" ? qr/([[:alpha:]])\1$/ : qr/$arg$/}
/$re/ && next || print
' '*'
produces
ab
ac
ad
ee
ef
A possible issue:
When you put * on the command line, the shell replaces it with the name of all the files in your directory. Your $1 will never equal *.
And some tips:
You can replace replace:
This:
# list files in current dir
list=`ls BRUTE*`
for i in $list
With:
for i in BRUTE*
And:
This:
cat $FILE | grep -v "^$"
With:
grep -v "^$" $FILE
Besides the possible issue, I can't see anything jumping out at me. What do you mean clean? Can you give an example of what a file should look like before and after and what the command would look like?
This is the problem!
grep '\(.\)\1[^\r\n]$' *
on MAC OSX, ( ) { }, etc... must be quoted!!!
Solved, thanks.

How to capture results of regex and replace patterns in bash

Bash scripting does my head in. I have searched for regex assignment, but not really finding answers I understand.
I have files in a directory. I need to loop through the files and check if they fit certain criteria. File names under a certain sequence need to have their sequence increased. Those over a certain sequence need to trigger an alert.
I have pseudo code and need help turning it into correct bash syntax:
#!/bin/sh
function check_file()
{
# example file name "LOG_20101031144515_001.csv"
filename=$1
# attempt to get the sequence (ex. 001) part of file
# if sequence is greater than 003, then raise alert
# else change file name to next sequence (ex. LOG_20101031144515_002.csv)
}
for i in `ls -Ar`; do check_file $i; done;
If PHP were an option, I could do the following:
function check_file($file){
//example file 'LOG_20101031144515_001.csv';
$parts = explode('.',$file);
preg_match('/\d{3}$/', $parts[0], $matches);
if ($matches){
$sequence = $matches[0];
$seq = intval($sequence);
if ($seq > 3){
// do some code to fire alert (ex. email webmaster)
}
else{
// rename file with new sequence
$new_seq = sprintf('%03d', ++$seq);
$new_file = str_replace("_$sequence", "_$new_seq", $file);
rename($file, $new_file);
}
}
}
So long story short, I'm hoping someone can help port the PHP check_file function to the bash equivalent.
Thank you
First of all, your question is tagged [bash], but your shebang is #!/bin/sh. I'm going to assume Bash.
#!/bin/bash
function check_file()
{
# example file name "LOG_20101031144515_001.csv"
filename=$1
# attempt to get the sequence (ex. 001) part of file
seq=${filename%.csv}
seq=${seq##*_}
# if sequence is greater than 003, then raise alert
if (( 10#$seq > 3 ))
then
echo "Alert!"
else
# else change file name to next sequence (ex. LOG_20101031144515_002.csv)
printf -v newseq "%03d" $((seq + 1))
echo "${filename/$seq/$newseq}" # or you could set a variable or do a mv
fi
}
PHP IS an option. If you master PHP, you can run it from shell.
Run
php myfile.php
and get the output right into console. If the PHP file is executable and begins with
#!/path/to/php/executable
then you can run
./myfile.php
I'm no big expert in bash programming, but in order to obtain the list of files that match a certain patter you can use the command
ls -l | grep "pattern_unquoted"
I suggest you to go for the PHP ;-)
A different take on the problem:
#!/bin/sh
YOUR_MAX_SEQ=3
find /path/to/files -maxdepth 1 -name 'LOG_*.csv' -print \
| sed -e 's/\.csv$//' \
| awk -F_ '$3 > SEQ { print }' SEQ=$YOUR_MAX_SEQ
Brief explanation:
Find all files in /path/to/files matching LOG_*.csv
Chop the .csv off the end of each line
Using _ as a separator, print lines where the third field is greater than $YOUR_MAX_SEQ
This will leave you with a list of the files that met your criteria. Optionally, you could pipe the output through sed to stick the .csv back on.
If you're comfortable with PHP, you'd probably be comfortable with Perl, too.