How to replace line breaks that are not after a specific character? - regex

Question

I need to process a csv file but one of the fields contains line breaks.
How can I replace all line breaks that are not after the double quote character (") with space? Any solution with awk, perl, sed etc is acceptable.
The file that is in the form:
497,50,2008-08-02T16:56:53Z,469,4,"foo bar
foo
bar"
518,153,2008-08-02T17:42:28Z,469,2,"foo bar
bar"
The desired output is:
497,50,2008-08-02T16:56:53Z,469,4,"foo bar foo bar"
518,153,2008-08-02T17:42:28Z,469,2,"foo bar bar"

Answer 1

I understood your question to be a request to replace intra-field newlines (even if they occur immediately after a ", such as in a field containing ␊foo or foo "bar"␊baz). The following achieves that:
use Text::CSV_XS qw( );
my $qfn_in = ...;
my $qfn_out = ...;
open(my $fh_in, '<', $qfn_in) or die("Can't open \"$qfn_in\": $!\n");
open(my $fh_out, '>', $qfn_out) or die("Can't create \"$qfn_out\": $!\n");
my $csv = Text::CSV_XS->new({ binary => 1, auto_diag => 2 });
while ( my $row = $csv->getline($fh_in) ) {
s/\n/ /g for #$row;
$csv->say($fh_out, $row);
}
I think it would make more sense to use the following:
for (#$row) {
s/^\s+//; # Remove leading whitespace.
s/\s+\z//; # Remove trailing whitespace.
s/\s+/ /g; # Replaces whitespace with a single space.
}

Answer 2

You can try this sed but the question is'nt clear enough to know what to do with a line like
497,50,2008-08-02T16:56:53Z,469,4,"truc biz",test
sed ':A;/[^"]$/{N;bA};y/\n/ /' infile

Answer 3

Its fairly easy to match the fields in csv.
The framework is the stuff between quoted/non-quoted fields
and is either delimiter or end of record tokens.
So the framework is matched as well to validate the fields.
After doing that, it's just a matter of replacing linebreaks in quoted fields.
That can be done in a call back.
The regex ((?:^|,|\r?\n)[^\S\r\n]*)(?:("[^"\\]*(?:\\[\S\s][^"\\]*)*"[^\S\r\n]*(?=$|,|\r?\n))|([^,\r\n]*(?=$|,|\r?\n)))
Here it is in Perl, all in one package.
use strict;
use warnings;
$/ = undef;
sub RmvNLs {
my ($delim, $quote, $non_quote) = #_;
if ( defined $non_quote ) {
return $delim . $non_quote;
}
$quote =~ s/\s*\r?\n/ /g;
return $delim . $quote;
}
my $csv = <DATA>;
$csv =~ s/
( # (1 start), Delimiter (BOS, comma or newline)
(?: ^ | , | \r? \n )
[^\S\r\n]* # Leading optional horizontal whitespaces
) # (1 end)
(?:
( # (2 start), Quoted string field
" # Quoted string
[^"\\]*
(?: \\ [\S\s] [^"\\]* )*
"
[^\S\r\n]* # Trailing optional horizontal whitespaces
(?= $ | , | \r? \n ) # Delimiter ahead (EOS, comma or newline)
) # (2 end)
| # OR
( # (3 start), Non quoted field
[^,\r\n]* # Not comma or newline
(?= $ | , | \r? \n ) # Delimiter ahead (EOS, comma or newline)
) # (3 end)
)
/RmvNLs($1,$2,$3)/xeg;
print $csv;
__DATA__
497,50,2008-08-02T16:56:53Z,469,4,"foo bar
foo
bar"
518,153,2008-08-02T17:42:28Z,469,2,"foo bar
bar"
Output
497,50,2008-08-02T16:56:53Z,469,4,"foo bar foo bar"
518,153,2008-08-02T17:42:28Z,469,2,"foo bar bar"