awk: how to include file names when concatenating files? - regex

Question

Am running GNUwin32 under windows 7.
Have many files in a single directory with file names that look like this:
chem.001.txt
chem.002.b4.txt
chem.003.md6.txt
(more files.txt) ...
In their current form, none of the files includes the file name.
Need to clean these files for further use.
Want to concatenate all files into a single file.
But also need to include the file name at the beginning of concatenated content to later associate the original file with clean data.
For example, the single, concatenated file (new_file.txt) would look like this:
chem.001.txt delimiter (could be a tab or pipe) followed by text from chem.001.txt...
chem.002.b4.txt delimiter followed by text from chem.002.b4.txt ...
chem.003.md6.txt delimiter followed by text from chem.003.md6.txt ...
etc. ...
Will then clean the concatenated file and parse content as needed.
awk - gawk may have a means to associate the file name with ($1), associate the text in the file with ($2) and then, in sequence, print ($1, $2) for each file into 'new_file.txt' but I've not been able to make it work.
How to do this?

Answer 1

Put this in foo.awk:
BEGIN{ RS="^$"; ORS=""; OFS="|" }
{ gsub(/\n[\r]?/," "); print FILENAME, $0 > "new_file.txt" }
and then execute it as
awk -f foo.awk <files>
where <files> is however you provide a list of file names in Windows. It uses GNU awk for multi-char RS to let you read a whole file as a single record.