Ruby Extract Slippery Text Columns - regex

Question

I am needing to get some columnized-text into ruby arrays. They are company names, phone numbers and websites. I've obscured the actual data in order to focus on the parsing as opposed to the nature of the data, which I can deal with.
here is the Gist
As you can see, the nature of the columnar data changes, including:
leading whitespace width changes, from 0 to ~8
some lines are "" or \s+{3,}
column width changes depending on which block it's in (see how line 31 changes from 27)
therefore reliance upon using widths becomes problematic
some lines show empty entries in columns
empty column 1 on line 4 (example)
empty column 2 on line 2 (example)
empty column 3 on line 3 (example)
I'm wanting to get this organized into col1, col2 and col3 as arrays of entries. I can split them later on /\s*/ and choose the first element.
Given the obvious structure of these three columns, I'm thinking there is a pragmatic way of parsing these columns out into arrays of entries, one per line.
Does anybody have any insight into how to parse out the columns? Columns -> arrays col1, col2, col3 is the format which I seek.
Any advice/insight appreciated.

Answer 1

Let's suppose we gulp the file into a string, using IO::read, where the string is as follows.
str=<<~END
aaa bb cccc aaaaaaa aaaa bbb
aaaaaaaa aaaaaaaaa
aaaaa aaaaa bbbb
aaaaa bb cc aaaaaaa
aaa bbb aaaaaa bbb aaaaa bbbbbb
aaaa aaaaaaaaaaaa
aaaaaaaaa
a bb aaaaaaaaa
END
The first step is to divide the string into (two) blocks, which we can do as follows:
a1 = str.split(/\n{2,}/)
#=> ["aaa bb cccc aaaaaaa aaaa bbb\n aaaaaaaa aaaaaaaaa\n aaaaa aaaaa bbbb\naaaaa bb cc aaaaaaa",
# "aaa bbb aaaaaa bbb aaaaa bbbbbb\n aaaa aaaaaaaaaaaa\n aaaaaaaaa\n a bb aaaaaaaaa\n"]
Next, convert each of the two blocks to an array of lines.
a2 = a1.map { |s| s.chomp.split(/\n/) }
#=> [["aaa bb cccc aaaaaaa aaaa bbb",
# " aaaaaaaa aaaaaaaaa",
# " aaaaa aaaaa bbbb",
# "aaaaa bb cc aaaaaaa"],
# ["aaa bbb aaaaaa bbb aaaaa bbbbbb",
# " aaaa aaaaaaaaaaaa",
# " aaaaaaaaa",
# " a bb aaaaaaaaa"]]
We need to now map each each element of a2 (a string) to an array whose "columns" correspond to the columns of the original text.
a3 = a2.flat_map do |group|
indent = group.map { |line| line =~ /\S/ }.min
mx_len = group.map(&:length).max
break_cols = (indent..mx_len-1).each_with_object([]) do |i,cols|
cols << i if group.all? { |line| [" ", nil].include?(line[i]) }
end
b1, b2 = [break_cols.first, break_cols.last]
group.map { |line| [line[0..b1-1], line[b1..b2-1], line[b2..-1]] }
end
#=> [["aaa bb cccc", " aaaaaaa ", " aaaa bbb"],
# [" aaaaaaaa ", " ", " aaaaaaaaa"],
# [" aaaaa ", " ", " aaaaa bbbb"],
# ["aaaaa bb cc", " aaaaaaa", nil],
# ["aaa bbb", " aaaaaa bbb ", " aaaaa bbbbbb"],
# [" aaaa ", " ", " aaaaaaaaaaaa"],
# [" ", " aaaaaaaaa", nil],
# [" a bb ", " ", " aaaaaaaaa"]]
line =~ /\S/ returns the index of the first element of line that contains a character of than a whitespace (the reserved character \S in regular expressions.)
See Enumerable#flat_map.
The following intermediate values were obtained in the calculation of a3.
For group 1:
mx_len = 37
indent = 0
break_cols = [11, 12, 13, 14, 23, 24, 25]
b1 = 11
b2 = 25
For group 2:
mx_len = 38
indent = 0
break_cols = [7, 8, 9, 20, 21, 22]
b1 = 7
b2 = 22
All that remains is to convert nil's to empty strings, strip spaces from the ends of each string and transpose the array.
a3.map { |col| col.map { |s| s.to_s.strip } }.transpose
#=> [["aaa bb cccc", "aaaaaaaa", "aaaaa", "aaaaa bb cc",
# "aaa bbb", "aaaa", "", "a bb"],
# ["aaaaaaa", "", "", "aaaaaaa", "aaaaaa bbb", "",
# "aaaaaaaaa", ""],
# ["aaaa bbb", "aaaaaaaaa", "aaaaa bbbb", "",
# "aaaaa bbbbbb", "aaaaaaaaaaaa", "", "aaaaaaaaa"]]
If desired, we could of course chain the above operations.
str.split(/\n{2,}/).
map { |s| s.chomp.split(/\n/) }.
flat_map do |group|
indent = group.map { |line| line =~ /\S/ }.min
mx_len = group.map(&:length).max
break_cols = (indent..mx_len-1).each_with_object([]) do |i,cols|
cols << i if group.all? { |line| [" ", nil].include?(line[i]) }
end
b1, b2 = [break_cols.first, break_cols.last]
group.map { |line| [line[0..b1-1], line[b1..b2-1], line[b2..-1]] }
end.map { |col| col.map { |s| s.to_s.strip } }.transpose

Answer 2

As Cary has demonstrated, working with widths was painful. That's what tripped me up. I took a new approach at doing a String.gsub(/\s{2,44}/,'•') so it would preserve column widths while inserting delimiters:
col1, col2, col3 = [],[], []
master_data = []
lines = File.open(s, 'r+').read.split("\n")
lines.each do |line|
next if line == "" || line.strip == ""
nline = line.gsub(/\s{2,44}/,'•')
nline[0] = '' if nline.start_with?('•')
nline = nline.split('•')
col1 << nline[0]
col2 << nline[1]
col3 << nline[2]
end
col1.delete_if {|i| i.nil?}
col2.delete_if {|i| i.nil?}
col3.delete_if {|i| i.nil?}
# ap col1
# puts
# ap col2
# puts
# ap col3
counter = 0
col1.each do |i|
next if i.nil?
if i.match?(/^\d{3}-\d{3}-\d{4}/) # matches a phone number, perhaps a big assumption
company = [col1[counter-1], col1[counter], col1[counter+1]]
master_data << company
end
counter += 1
end
# a company is a company name, phone number, and website
# do the same for col2 and col3
ap master_data