Command line to remove in-line dupes - regex

Question

What is fast and succinct way to remove dupes from within a line?
I have a file in the following format:
alpha • a | b | c | a | b | c | d
beta • h | i | i | h | i | j | k
gamma •  m | n | o
delta • p | p | q | r | s | q
So there's a headword in column 1, and then various words delimited by pipes, with an unpredictable amount of duplication. The desired output has the dupes removed, as:
alpha • a | b | c | d
beta • h | i | j | k
gamma •  m | n | o
delta • p | q | r | s
My input file is a few thousand lines. The Greek names above correspond to category names (e.g., "baseball"); and the alphabet corresponds English dictionary words (which might contain spaces or accents), e.g. "ball game | batter | catcher | catcher | designated hitter".
This could be programmed many ways, but I suspect there's a smart way to do it. I encounter variations of this scenario a lot, and wonder if there's a concise and elegant way to do this. I am using MacOS, so a few fancy unix options are not available.
Bonus complexity, I often have a comment at the end which should be retained, e.g.,
zeta • x | y | x | z | z ; comment here
P.S. this input is actually the output of a prior StackOverflow question:
Command line to match lines with matching first field (sed, awk, etc.)

Answer 1

BSD awk does not have sort functions builtin where GNU awk does, but I'm not sure they're necessary. The bullet, • (U+2022), causes some grief with awk.
I suggest pre-processing the bullet to a single-byte character. I chose #, but you could use Control-A or something else if you prefer. Your data was in a file data. I note that there was a double space before m in the gamma line; I'm assuming that isn't significant.
sed 's/•/#/' data |
awk -F ' *[#|] *' '
{
delete names
delete comments
delete fields;
if ($NF ~ / *;/) { split($NF, comments, / *; */); $NF=comments[1]; }
j = 1;
for (i = 2; i <= NF; i++)
{
if (names[$i]++ == 0)
fields[j++] = $i;
}
printf("%s", $1);
delim = "•"
for (k = 1; k < j; k++)
{
printf(" %s %s", delim, fields[k]);
delim = "|";
}
if (comments[2])
printf(" ; %s", comments[2]);
printf("\n");
}'
Running this yields:
alpha • a | b | c | d
beta • h | i | j | k
gamma • m | n | o
delta • p | q | r | s
zeta • x | y | z ; comment here

Answer 2

With bash, sort, xargs, sed:
while IFS='•;' read -r a b c; do
IFS="|" read -ra array <<< "$b"
array=( "${array[#]# }" )
array=( "${array[#]% }" )
readarray -t array < <(printf '%s\0' "${array[#]}" | sort -zu | xargs -0n1)
SAVE_IFS="$IFS"; IFS="|"
s="$a• ${array[*]}"
[[ $c != "" ]] && s="$s ;$c"
sed 's/|/ | /g' <<< "$s"
IFS="$SAVE_IFS"
done < file
Output:
alpha • a | b | c | d
beta • h | i | j | k
gamma • m | n | o
delta • p | q | r | s
zeta • x | y | z ; comment here
I suppose the two spaces before "m" are a typo.

Answer 3

This might work for you (GNU sed):
sed 'h;s/.*• \([^;]*\).*/cat <<\\! | sort -u |\1|!/;s/\s*|\s*/\n/2ge;s/\n/ | /g;G;s/^\(.*\)\n\(.*• \)[^;]*/\2\1/;s/;/ &/' file
The sketch of this idea is: to remove the head and tail of each line, morph the data into a mini file, use standard utilities to sort and remove duplicates, then put the line back together again.
Here a copy of the line is held in the hold space. The id and comments removed. The data is munged into a file using cat and the bash here-document syntax and piped through a sort (and uniq if your sort does not come equipped with the -u option). The pattern space is evaluated and the line reassembled by appending the original line to the pattern space and using regexp pattern matching.