Regex substitution over a large file - regex

Question

Given a large file (potentially 1.5GB) with some very long lines, I need to run substitutions. I thought of running a sliding window where I read 4096 bytes off the file, append them to the previous chunk and substitute on that, but of course that doesn't work so well for substitutions where the substitution might be across the fold.
Another thought came to mind that I could start with my first two chunks and s/// on that. if there are no substitutions made I write chunk1 to disk and grab chunk3. Then I s/// on chunk2 and chunk3. If there are substitions I grab chunk4 and append it. Then I keep appending until there's no substitions. At that point I write everything but the latest chunk to disk. Like this:
read( $data, $previous_chunk, 4096 );
while( read( $data, $this_chunk, 4096 ) > 0 ){
my $chunk = $previous_chunk . $this_chunk;
if( 0 == $chunk =~ s/foo/bar/g ){
# There was nothing to substitute, so we'll append all
# the old stuff to our file and just keep the latest chunk.
print OUTPUT $previous_chunk;
$previous_chunk = $this_chunk;
}
else {
# There was a substitution, so we want to keep building
# in case it crossed the fold
$previous_chunk .= $this_chunk;
}
}
Does that sound sane? The only problem I can see is that if the substitution causes a new match in the running $previous_chunk. So we probably need to clear $previous_chunk up to the latest substitution somehow and only keep the clean content that follows it. (Eg, if we had s/foo/foobar/ we'd turn 'foo' into 'foobar' then into 'foobarbar' then into 'foobarbarbar'. Is there a way to avoid that?
Is there a better way to do it?

Answer 1

You ask if there is a better way to do it. "Better" is subjective, but this is what I would do if I were faced with the same problem.
One option is not to use Perl at all but to perform your search-and-replace in a text editor that supports gigantic files.
On Windows, one editor that does so is EditPad Pro, which supports huge files. Not my product, so this is not an ad. I use EPP because it is regex-centric (same author as RegexBuddy).
Quoting from the brochure: Open and edit files of absolutely any size, including files larger than 4 GB, even on a 32-bit system with a modest amount of RAM.
This allows you to offload the whole large file issue.
Other features that may or may not be relevant: you can save your regular expressions, chain them with macros, and invoke tools (such as your own Perl scripts) to manipulate the currently open file (though in that case you are once again at the mercy of the outside tool's memory management).