I am trying to create a simple CSV editor in bash,
and I struggle with removing a line. The user passes in the ID of
the line to remove (each row is defined with an ID as the first column).
This is an example file structure:
ID,Name,Surname
0,Mark,Twain
1,Cristopher,Jones
So, having the id saved as a variable and the file name in another variable (say its file.csv) I attempt to remove it from bash with this line:
read -p "Pass the object's ID: " idtoremove
fname=file.csv
sed -i -e "'/^$idtoremove*,/d'" $fname
However, this has no effect on the file. What could be wrong with this line?
Also, how can I replace a line starting with given ID with a string from a variable? This is another problem I will have to face but I have no Idea how to approach this one.
Following script could help you. It asks user to enter an id.
cat script.ksh
echo "Please enter the id to be removed:"
read value
awk -v val="$value" -F, '$1!=val' Input_file
In case you want to save output into Input_file itself append > tmp_file && mv tmp_file Input_file in above awk code.
With sed:
cat script.ksh
echo "Please enter the id to be removed:"
read value
sed "/^$value,/d" Input_file
Kindly use sed -i.bak option in above sed to save output into Input_file itself and have a backup of Input_file(before change) too.
This is best done in awk:
awk -v id="$idtoremove" -F, '$1 != id' file.csv
If you're using gnu awk then you can save in-place also:
awk -i inplace -v id="$idtoremove" -F, '$1 != id' file.csv
For other awk versions use:
awk -v id="$idtoremove" -F, '$1 != id' file.csv > $$.csv &&
mv $$.csv file.csv
Related
My file has lines
Database Name:Mydb
DatabaseServer:DbServer
Password:Example
Username:User1
Database Name:Mydb1
DatabaseServer:DbServer1
Password:Example1
Username:User11
I used sed -i "s/password.//gI" file but that is leaving an empty line(in place of password.) which i don't want.
Desired result:
Database Name:Mydb
DatabaseServer:DbServer
Username:User1
Database Name:Mydb1
DatabaseServer:DbServer1
Username:User11
give this a try:
sed -i '/Password:/d' file
Solution 1st: grep here to help.
grep -v '^Password' Input_file
In case you need to save into same Input_file then you could following.
grep -v '^Password' Input_file > temp && mv temp Input_file
Solution 2nd: Using awk.
awk '!/^Password/' Input_file > temp_file && mv temp_file Input_file
I am having trouble getting a regular expression that will search for an input term in the specified column. If the term is found in that column, then it needs to output that whole line.
These are my variables:
sreg = search word #Example: Adam
file = text file #Example: Contacts.txt
sfield = column number #Example: 1
the text file is in this format with a space being the field seperator, with many contact entries:
First Last Email Phone Category
Adam aster junfmr# 8473847548 word
Jeff Williams 43wadsfddf# 940342221995 friend
JOhn smart qwer#qwer 999999393 enemy
yooun yeall adada 111223123 other
zefir sentr jjdirutk#jd 8847394578 other
I've tried with no success:
grep "$sreg" "$file" | cut -d " " -f"$sfield"-"$sfield"
awk -F, '{ if ($sreg == $sfield) print $0 }' "$file"
awk -v s="$sreg" -v c="$sfield" '$c == s { print $0 }' "$file"
Thanks for any help!
awk may be the best solution for this:
awk -v field="$field" -v name="$name" '$field==name' "$file"
This checks if the field number $field has the value $name. If so, awk automatically prints the full line that contains it.
For example:
$ field=1
$ name="Adam"
$ file="your_file"
$ awk -v field="$field" -v name="$name" '$field==name' "$file"
Adam aster junfmr# 8473847548 word
As you can see, we give the parameters using -v var="$bash_var", so that you can use them inside awk.
Also, the space is the field separator, so you don't need to specify it since it is the default.
This works for me:
awk -v f="$sfield" -v reg="$sreg" '{if ($f ~ reg) {print $0}}' "$file"
Major problem is that you need an indirection from $sfield (ex, "1") to $($sfield) (ex, $1).
I tried using backtricks `, and also using ${!sfield}, but they don't work in awk, as awk does not accept this. Finally I found the way of passing variable into awk, converting to awk internal variabls (using -v).
Within awk, I found you can not even access variables outside. So I had to pass $sreg as well.
Update: I think using "~" instead of "==" is better because the original requirement said matchi==ng a regular expression.
For example,
sreg=Ad
I have a few thousand lines of code spread out across multiple files that I need to update. Right now they are like so:
active_data(COPIED),
I need to replace all instances of COPIED (only in these lines) with the text inside the parenthesis on the previous line. So it total the code currently might look like:
current_data(FOO|BAR|TEXT),
active_data(COPIED),
and I want it to look like:
current_data(FOO|BAR|TEXT),
active_data(FOO|BAR|TEXT),
I can find the lines to be replaced easily enough, and could replace the with some static string with no problem, but I'm not sure how to pull the data from the previous line and use that. I'm sure its pretty simple but can't quite figure it out. Thanks for the help.
(I could see using AWK or something else for this too if sed won't work but I figure sed would be the best solution for a one time change).
sed could work but awk is more natural:
$ awk -F'[()]' '$2 == "COPIED" {sub(/COPIED/, prev)} {prev=$2;} 1' file
current_data(FOO|BAR|TEXT),
active_data(FOO|BAR|TEXT),
-F'[()]'
Use open or claose parens as the field separator.
$2 == "COPIED" {sub("COPIED", prev)}
If the second field is COPIED, then replace it with prev.
prev=$2
Update prev.
1
This is cryptic shorthand which means print the line. It is equivalent to {print $0;}.
How awk sees the fields
$ awk -F'[()]' '{for (i=1;i<=NF;i++)printf "Line %s Field %s=%s\n",NR,i,$i;}' file
Line 1 Field 1=current_data
Line 1 Field 2=FOO|BAR|TEXT
Line 1 Field 3=,
Line 2 Field 1=active_data
Line 2 Field 2=COPIED
Line 2 Field 3=,
Changing in-place all files in a directory
for file in *
do
awk -F'[()]' '$2 == "COPIED" {sub("COPIED", prev)} {prev=$2;} 1' "$file" >tmp$$ && mv tmp$$ "$file"
done
Or, if you have a modern GNU awk:
awk -i inplace -F'[()]' '$2 == "COPIED" {sub("COPIED", prev)} {prev=$2;} 1' *
This might work for you (GNU sed):
sed 'N;s/\((\([^)]*\)).*\n.*(\)COPIED/\1\2/;P;D' file
This keeps a moving window of 2 lines open throughout the length of the file and uses pattern matching to effect the required result.
With GNU awk for FPAT:
$ awk -v FPAT='[(][^)]*[)]|[^)(]*' -v OFS= '$2=="(COPIED)"{$2=prev} {prev=$2; print}' file
current_data(FOO|BAR|TEXT),
active_data(FOO|BAR|TEXT),
With other awks:
$ awk '
match($0,/\([^)]*\)/) {
curr = substr($0,RSTART,RLENGTH)
if (curr == "(COPIED)") {
$0 = substr($0,1,RSTART-1) prev substr($0,RSTART+RLENGTH)
}
else {
prev = curr
}
}
{ print }
' file
current_data(FOO|BAR|TEXT),
active_data(FOO|BAR|TEXT),
I am using an awk command (someawkcommand) that prints these lines (awkoutput):
>Genome1
ATGCAAAAG
CAATAA
and then, I want to use this output (awkoutput) as the input of a sed command. Something like that:
someawkcommand | sed 's/awkoutput//g' file1.txt > results.txt
file1.txt:
>Genome1
ATGCAAAAG
CAATAA
>Genome2
ATGAAAAA
AAAAAAAA
CAA
>Genome3
ACCC
The final objective is to delete all lines in a file (file1.txt) containing the exact pattern found previously by awk.
The file results.txt contains (output of sed):
>Genome2
ATGAAAAA
AAAAAAAA
CAA
>Genome3
ACCC
How should I write the sed command? Is there any simple way that sed will recognize the output of awk as its input?
Using GNU awk for multi-char RS:
$ cat file1
>Genome1
ATGCAAAAG
CAATAA
$ cat file2
>Genome1
ATGCAAAAG
CAATAA
>Genome2
ATGAAAAA
AAAAAAAA
CAA
>Genome3
ACCC
$ gawk -v RS='^$' -v ORS= 'NR==FNR{rmv=$0;next} {sub(rmv,"")} 1' file1 file2
>Genome2
ATGAAAAA
AAAAAAAA
CAA
>Genome3
ACCC
The stuff that might be non-obvious to newcomers but are very common awk idioms:
-v RS='^$' tells awk to read the whole file as one string (instead of it's default one line at a time).
-v ORS= sets the Output Record Separator to the null string (instead of it's default newline) so that when the file is printed as a string awk doesn't add a newline after it.
NR==FNR is a condition that is only true for the first input file.
1 is a true condition invoking the default action of printing the current record.
Here is a possible sed solution:
someawkcommand | sed -n 's_.*_/&/d;_;H;${x;s_\n__g p}' | sed -f - file1.txt
First sed command turns output from someawkcommand into a sed expression.
Concretely, it turns
>Genome1
ATGCAAAAG
CAATAA
into:
/>Genome1/d;/ATGCAAAAG/d;/CAATAA/d;
(in sed language: delete lines containing those patterns; mind that you will have to escape /,[,],*,^,$ in your awk output if there are some, with another substitution for instance).
Second sed command reads it as input expression (-f - reads sed commands from file -, i.e. gets it from pipe) and applies to file file1.txt.
Remark for other readers:
OP wants to use sed, but as notified in comments, it may not be the easiest way to solve this question. Deleting lines with awk could be simpler. Another (easy) solution could be to use grep with -v (invert match) and -f (read patterns from files) options, in this way:
someawkcommand | grep -v -f - file1.txt
Edit: Following #rici's comments, here is a new command that takes output from awk as a single multiline pattern.
Disclaimer: It gets dirty. Kids, don't do it home. Grown-ups are strongly encouraged to consider avoiding sed for that.
someawkcommand | \
sed -n 'H;${x;s_\n__;s_\n_\\n_g;s_.*_H;${x;s/\\n//;s/&//g p}_ p}' | \
sed -n -f - file1.txt
Output from inner sed is:
H;${x;s/\n//;s/>Genome1\nATGCAAAAG\nCAATAA//g p}
Additional drawback: it will add an empty line instead of removed pattern. Can't fix it easily (problems if pattern is at beginning/end of file). Add a substitution to remove it if you really feel like it.
This is can more easily be done in awk, but the usual "eliminate duplicates" code is not correct. As I understand the question, the goal is to remove entire stanzas from the file.
Here's a possible solution which assumes that the first awk script outputs a single stanza:
awk 'NR == FNR {stanza[nstanza++] = $0; next}
$0 == stanza[i] {++i; next}
/^>/ && i == nstanza {i=0; next}
i {for (j=0; j<i; ++j) print stanza[j]; i=0}
{print $0;}
' <(someawkcommand) file1.txt
This might work for you (GNU sed):
sed '1{h;s/.*/:a;$!{N;ba}/p;d};/^>/!{H;$!d};x;s/\n/\\n/g;s|.*|s/&\\n*//g|p;$s|.*|s/\\n*$//|p;x;h;d' file1
sed -f - file2
This builds a script from file1 and then runs it against file2.
The script slurps in file2 and then does a gobal substitution(s) using the contents of file1. Finally it removes any blank lines at the end file caused by the contents deletion.
To see the script produced from file1, remove the pipe and the second sed command.
An alternative way would be to use diff and sed:
diff -e file2 file1 | sed 's/d/p/g' | sed -nf - file2
I would like to replace everything between : if there's a keyword in it
Having
TEXT="/Something/like-this:/How/can-one-replace/text/separated/with/colon/that-includes/a/keyword?:There/may-be/multiple/keywords:/Thanks:/keyword"
with:
sed -e 's/regex here that searches for keyword/\/some\/path/g' <<< $TEXT
To get:
/Something/like-this:/some/path:/some/path:/Thanks:/some/path
P.S.
Another example to make it more clear: How can paths that includes hello be replaced with another path?
/opt/hello/bin:/bin:/home/user/hello:/home/user/bin:/media/hello
=>
/some/path:/bin:/some/path:/home/user/bin:/some/path
My apologies for unclear question.
I think you need this,
$ sed -r 's~^([^:]+):.*:([^:]+):(.*)$~\1:/Replacement:/Replacement:\2:/Replacement~g' file
/Something/like-this:/Replacement:/Replacement:/Thanks:/Replacement
Or something like this,
$ sed -r 's~^([^:]+):.*:([^:]+):(.*)$~\1:/*Replacement*:/*Replacement*:\2:/*Replacement*~g' file
/Something/like-this:/*Replacement*:/*Replacement*:/Thanks:/*Replacement*
Or
it may be like this, if you assign some path to Replacement variable,
$ Replacement=/foo/bar
$ sed -r "s~^([^:]+):.*:([^:]+):(.*)$~\1:/*$Replacement*:/*$Replacement*:\2:/*$Replacement*~g" file
/Something/like-this:/*/foo/bar*:/*/foo/bar*:/Thanks:/*/foo/bar*
Or
You may try this also,
awk -v RS=: -v var=/path -v ORS=: '{sub (/.*hello.*/,var)}1' file
Example:
$ echo '/opt/hello/bin:/bin:/home/user/hello:/home/user/bin:/media/hello' | awk -v RS=: -v var=/foo/bar -v ORS=: '{sub (/.*hello.*/,var)}1'
/foo/bar:/bin:/foo/bar:/home/user/bin:/foo/bar:
Explanation:
Awk inbuilt variable RS(Record seperator) and ORS(Output Record Seperator) are set to :. So awk breaks the string whenever it finds : in the input and treats the text after : would be in the next line.
ORS is set to :, so awk prints the records with : as seperator.
-v var=/foo/bar , Replacement string is assigned to a variable var.
sub (/.*hello.*/,var), if the record matches this regex, it replaces the whole record with the value in the variable var.
1, to print all the records.
My version:
sed 's/:/::/g;s/^/:/;s/$/:/;s/:[^:]*keyword[^:]*:/:REPLACEMENT:/g;s/^://;s/:$//;s/::/:/g'
With bash
IFS=: read -ra arr <<<'/opt/hello/bin:/bin:/home/user/hello:/home/user/bin:/media/hello'
v=$(IFS=:; printf "%s\n" "${arr[*]/*hello*/\/some\/path}")
echo $v
/some/path:/bin:/some/path:/home/user/bin:/some/path