sort and uniq

[Asif Ahmed Syed]

Hi there,

I have a file as below

K1|K2|dataA1|K3|dataA2|K4|dataA3
K1|K2|dataB1|K3|dataB2|K4|dataB3


what I need to do is sort on the keys K1,K2,K3,K4 and remove any duplicates on them (only one should remain, could be any thing). How can I do that using sort and uniq.

[wje_lf]

Do the following at the command line.

man sort
man uniq

If man pages are not installed on your system, google for this

linux man sort

and this

man sort uniq

Hope this helps.

[radoulov]

awk -F\| '!_[$1,$2,$4,$6]++' file

This will give you the unique records,
then use sort to sort them as you want.

[wje_lf]

Asif Ahmed Syed, it turns out that radoulov's advice shows you that there are many ways to do things in UNIX/Linux/BSD. I'm not familiar with awk, but radoulov's solution is likely correct.

If, however, you really do wish to do this entirely with sort and uniq, you can do so. In fact, you can do it entirely with sort.

The man page will show you how. :)

[radoulov]

wje_lf is right,
I have to admit that I had to re-read the man pages to find the correct syntax

[scm]

sort has the -u flag to strip out duplicates. Note that uniq requires its input to be already sorted, but the -c flag can be useful to give a count of the number of occurrences of each unique line.