Awk to extract and format a highly variable text file - regex

Question

I'm dealing with a text file that's just a mess. It's the service record for a used RV that I'm buying, and it's a regex lover's nightmare
It has both inconsistent field separators and an inconsistent number of fields, with the lines being one of two types:
Type 1 (11 columns):
UNIT Mile GnHr R.O. Ln Service Description Mechanic Hours $ Amt
7-9918;57878 1698 01633 021;0502-00C ENG OIL/ FILTERT IF NEEDED;M02 JOSE A. SANCHEZ;0.80;80.00
Type 2 (10 columns)
UNIT Mile GnHr R.O. Ln Service Description Hours $ Amt
7-9918;55007 1641 [9564 007;ELE-BAT-BAT-0-0AAA;BATTERY AAA ALL BRANDS;2;31.12
I've stripped out all the headings, but put them back here just for reference. In Type 2 lines, the Mechanic field is missing.
I replaced all occurrences of multiple spaces with semicolons, so what I have now is a file where some lines have 10 fields, some lines have 11 fields, and sometimes the field separator is a space, and in other cases it's a semicolon, and some fields have legitimate embedded spaces (Description and Mechanic).
I'm trying to find a way with awk to:
1) Extract each field and be able to print it out with a uniform OFS (semicolon is preferred)
2) If the Mechanic field is missing, insert it and print N/A or -- for the Mechanic
I can deal with column headings and stuff myself, I just can't crack the code for how to deal with the FS problem and variable number of columns in this file. I can grep out specific information that I need, but would be thrilled to get it into a form where I can import it into a spreadsheet or DB.

Answer 1

Your input file's not so bad. Assuming your input file is semi-colon separated:
Replace all blank chars in $2 with a ; to split that up into separate fields for output, then
if there's a blank in $3 then replace the first blank with a ; (since it contains both the service and description so you need to separate them), otherwise
this is a format of line that has no mechanic specified so add the empty-mechanic text after $4 (the description)
and then just print the line:
$ awk 'BEGIN{FS=OFS=";"} {gsub(/ /,OFS,$2)} !sub(/ /,OFS,$3){$4=$4 OFS "N/A"} 1' file
7-9918;57878;1698;01633;021;0502-00C;ENG OIL/ FILTERT IF NEEDED;M02 JOSE A. SANCHEZ;0.80;80.00
7-9918;55007;1641;[9564;007;ELE-BAT-BAT-0-0AAA;BATTERY AAA ALL BRANDS;N/A;2;31.12
and if you'd like to do anything with the individual fields:
$ cat tst.awk
BEGIN { FS=OFS=";" }
{ gsub(/ /,OFS,$2) }
!sub(/ /,OFS,$3) { $4 = $4 OFS "N/A" }
{
$0 = $0
print
for (i=1; i<=NF; i++) {
print NR, i, $i
}
print ""
}
.
$ awk -f tst.awk file
7-9918;57878;1698;01633;021;0502-00C;ENG OIL/ FILTERT IF NEEDED;M02 JOSE A. SANCHEZ;0.80;80.00
1;1;7-9918
1;2;57878
1;3;1698
1;4;01633
1;5;021
1;6;0502-00C
1;7;ENG OIL/ FILTERT IF NEEDED
1;8;M02 JOSE A. SANCHEZ
1;9;0.80
1;10;80.00
7-9918;55007;1641;[9564;007;ELE-BAT-BAT-0-0AAA;BATTERY AAA ALL BRANDS;N/A;2;31.12
2;1;7-9918
2;2;55007
2;3;1641
2;4;[9564
2;5;007
2;6;ELE-BAT-BAT-0-0AAA
2;7;BATTERY AAA ALL BRANDS
2;8;N/A
2;9;2
2;10;31.12

Answer 2

A friend of mine also sent me this solution, done in perl:
#!/usr/bin/env perl -w
use strict;
use warnings;
# 1 1 1 1 1
# 1 2 3 4 5 6 7 8 9 0 1 2 3 4
# 012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
# Type 1:
# 7-9918 55007 1641 [9564 022 0211 INTERIOR MISC. M02 JOSE A. SANCHEZ 0.00 0.00
# Type 2:
# 7-9918 57878 1698 01633 001 FUE-LPG-LPG-S-GAS PROPANE GAS BULK PURCHASE 5 24.00
my $delim="\t";
while (<STDIN>) {
#print $_;
# Both formats are the same at this point
print substr($_, 0, 6) . $delim;
print substr($_, 8, 5) . $delim;
print substr($_, 14, 4) . $delim;
print substr($_, 19, 5) . $delim;
print substr($_, 25, 3) . $delim;
my $qty = substr($_, 109, 11);
$qty =~ s/^\s*//g;
$qty =~ s/\s*$//g;
if ($qty =~ /^\d+\.\d{2}$/) {
# Type 1
print substr($_, 40, 9) . $delim;
print substr($_, 49, 32) . $delim;
# print substr($_, 81, 32) . $delim; # Technician name
print $qty . $delim;
} elsif ($qty =~ /^[-]?\d+$/) {
# Type 2
print substr($_, 40, 23) . $delim;
print substr($_, 63, 46) . $delim;
print $qty . $delim;
}
print sprintf("%.2f", substr($_, 120, 11)) . "\n";
}
1;