Do not print if $previous_line matches $current_line.* [closed] - regex

Question

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I would like to be able to use awk in place of a while loop to remove subdomains from an input string if it also contains the main domain.
Source file:
1234.f.dsfsd.test.com
abc.test.com
ad.sdk.kaffnet.com
amazon.co.uk
analytics.test.dailymail.co.uk
bbc.co.uk
bbc.test.com
dailymail.co.uk
kaffnet.com
sdk.kaffnet.com
sub.test.bbc.co.uk
t.dailymail.co.uk
test.amazon.co.uk
test.bbc.co.uk
test.com
test.dailymail.co.uk
Desired Output:
amazon.co.uk
bbc.co.uk
dailymail.co.uk
kaffnet.com
test.com
Solution: #EdMorton

Answer 1

Check the last part of a domain and see which string is the shortest one among them:
BEGIN{FS="."}
{
ind=$(NF-1) FS $NF;
if (!(ind in size) || (size[ind] > length)) {
size[ind]=length # check the minimum size for this domain
domain[ind]=$0 # store the string with the minimum size on this domain
}
}
END {for (ind in domain) print domain[ind]}
As a one-liner:
$ awk 'BEGIN{FS="."} {ind=$(NF-1) FS $NF; if (!(ind in size) || (size[ind] > length)) { size[ind]=length; domain[ind]=$0}} END {for (ind in domain) print domain[ind]}' file
test.com
bbc.co.uk
Previous approach, that works for top level domains:
Just make use of the field separator and set it to the dot. This way, it is just a matter of storing the penultimate and last one as a string and check how many different ones you find:
$ awk -F. '{a[$(NF-1) FS $NF]} END{for (i in a) print i}' file
test.com
How does this work? a[] is an array to which we keep adding indeces. The index is defined with the penultimate field followed by a dot and the last field. This way, any new bla.test.com will still have the same index and do not add extra info into the array.
With other inputs:
$ cat file
1234.f.dsfsd.test.com
abc.test.com
bbc.test.com
test.com
bla.com
another.bla.com
$ awk -F. '{a[$(NF-1) FS $NF]} END{for (i in a) print i}' file
test.com
bla.com

Answer 2

New answer based on new requirements and new sample input file:
$ cat tst.awk
{ doms[$0] }
END {
for (domA in doms) {
hasSubDom = 0
for (domB in doms) {
if ( index(domA,domB ".") == 1 ) {
hasSubDom = 1
}
}
if ( !hasSubDom ) {
print domA
}
}
}
$ rev file | awk -f tst.awk | rev
bbc.co.uk
dailymail.co.uk
amazon.co.uk
kaffnet.com
test.com
$ rev file | sort |
awk -F'.' 'index($0,prev FS)!=1{ print; prev=$1 FS $2 }' |
rev
bbc.co.uk
test.com
The above just implements the algorithm you described in your question. It reverses the chars on each line and then sorts the result just like you were already doing, then if the previous line was foo.bar.stuff then prev is foo.bar and so if the current line is foo.bar.otherstuff then the call to index WILL find that foo.bar. (note the . at the end - adding that last . to the comparison is important so that foo.bar doesn't falsely match foo.barristers.wig) DOES occur at the start (index position 1) of the current line and so we will NOT print that line and prev will remain as is. If, on the other hand the current line is my.sharona.song then prev (foo.bar) DOES NOT occur at the start of that line and so that line IS printed and prev gets set to my.sharona. Finally it just reverses the chars on each output line back to their original order.

Answer 3

You can test a dynamic regex inside awk if you build a variable with the ~ operator
awk 'NR==1{a=$0} NR>1{if(length(a)>0){regex="^"a;if($0~regex){print a}}a=$0}'
Example (using tac and rev to facilitate the reversion)
The problem with your method is that you need at least 2 lines for the domain because you only display the previous line, but what if you did not have a previous line? Maybe that is not an issue for you if your domains always come with at least 2 lines.

Answer 4

For what it's worth, here is a version that works without requiring reversing and sorting the input.
awk -F. 'BEGIN {
SLDs = "co.uk,gov.uk,add.others" # general-use second-level domains we recognize
split(SLDs, slds, /,/);
for (i in slds) slds[slds[i]] = 1
}
/./ {
tld = $(NF-1) "." $(NF)
if (NF > 2 && tld in slds) tld = $(NF-2) "." tld
lines[NR] = $0
tlds[NR] = tld
if (tld == $0) existing_tlds[tld] = 1
}
END {
for (i = 1; i <= length(lines); i++) {
line = lines[i]; tld = tlds[i]
if (!(tld in existing_tlds) || tld == line) print(line)
}
}' input_file
This goes through the file and builds an array of existing TLDs. In the END block it prints a line only when it is a TLD itself or its TLD does not exist in said array.
When input_file is
1234.f.dsfsd.test.com
abc.test.com
amazon.co.uk
bbc.co.uk
bbc.test.com
sub.test.bbc.co.uk
test.amazon.co.uk
test.bbc.co.uk
test.com
it prints
amazon.co.uk
bbc.co.uk
test.com