Selectively remove subfields from a CSV file in sed - regex

Question

I have a CSV file my.csv, in which fields are separated by ;. Each field contains arbitrary number (sometimes zero) of subfields separated by |, like this:
aa5|xb1;xc3;ba7|cx2|xx3|da2;ed1
xa2|bx9;ab5;;af2|xb5
xb7;xa6|fc5;fd6|xb5;xc3|ax9
df3;ab5|xc7|de2;da5;ax2|xd8|bb1
I would like to remove all sub-fields (with corresponding |'s) that start from everything but x, i.e. get output like this:
xb1;xc3;xx3;
xa2;;;xb5
xb7;xa6;xb5;xc3
;xc7;;xd8
Now I am doing this in multiple steps with sed:
sed -i 's/^[^;x]*;/;/g' my.csv #In 1st fields without x.
sed -i 's/;[^;x]*;/;;/g' my.csv #In middle field without x.
sed -i 's/;[^;x]*$/;/g' my.csv #In last field without x.
sed -i 's/^[^;x][^;]*|x/x/g' my.csv #In 1st fields with x. before x.
sed -i 's/;[^;x][^;]*|x/;x/g' my.csv #In non-1st fields with x. before x.
sed -i 's/|[^x][^;]*//g' my.csv #In fields with x. after x.
Is there a way to do it one line or at least more simple? I got stuck on the problem how to match "line beginning OR ';'".
In my case it is guaranteed that there is no more than one subfield starting with x. In theory, however, it would be also useful how to solve the problem if it is not the case (e.g., convert field ab1|xa2|bc3|xd4|ex5 to xa2|xd4).

Answer 1

Using sed
sed ':;s/\(^\||\|;\)[^x;|][^;|]*/\1/;t;s/|//g' file
Just loops through removing fields that don't begin with x and then removes the bars.
Output
xb1;xc3;xx3;
xa2;;;xb5
xb7;xa6;xb5;xc3
;xc7;;xd8

Answer 2

You can use this awk:
awk 'BEGIN{FS=OFS=";"} {for (i=1; i<=NF; i++) {
gsub(/(^|\|)[^x][^|]*/, "", $i); sub(/^\|/, "", $i)}} 1' file
xb1;xc3;xx3;
xa2;;;xb5
xb7;xa6;xb5;xc3
;xc7;;xd8
This will also convert ab1|xa2|bc3|xd4|ex5 to xa2|xd4 i.e. multiple fields starting with x.

Answer 3

Consider using Perl:
perl -ple '$_ = join(";", map { join "|", grep /^x/, split /\|/ } split(/;/, $_, -1))'
This starts with split(/;/, $_, -1), splitting the line ($_ at this point) into an array of fields at semicolons. The negative limit parameter makes it so that trailing empty fields (if they exist) are not discarded.
The elements of that array are
transformed in the map expression, and
joined again with semicolons.
The transformation in the map expression is
splitting along |,
grepping for /^x/ (i.e., weeding out those that don't match the regex),
joining with | again.
I believe this structured approach to be more robust than regex wizardry.
Old code that loses empty fields at the end of a line:
perl -F\; -aple '$_=join(";", map { join("|", grep(/^x/, split(/\|/, $_))) } #F)'
This used -a for auto-split, which looked nicer but didn't have the fine-grained control over field splitting that was required.

Answer 4

awk to the rescue!
awk -F";" -vOFS=";" '
{line=sep="";
for(i=1;i<=NF;i++) {
c=split($i,s,"|");
for(j=1;j<=c;j++)
if(s[j]~/^x/) {
line=line sep s[j];
sep=OFS
}
}
print line}'
split each element further for pattern check, combine results in a line, set separator after the first element is set on each line.