Looking for deduplication in linux.

[Dustspeck]

Do you want to first add a row number to each line, then sort, then fold the identical lines?

so, if your starting data is:
AAAAA
BBBBB
AAAAA
VFVDS
AAAAA

then you'd first transform it to:
AAAAA|1
BBBBB|2
AAAAA|3
VFVDS|4
AAAAA|5

Then you'd sort it to:
AAAAA|1
AAAAA|3
AAAAA|5
BBBBB|2
VFVDS|4

Then you'd fold it to:
AAAAA|1|3|5
BBBBB|2
VFVDS|4

Is that the process you are looking for?

[Syndacate]

Do you want to first add a row number to each line, then sort, then fold the identical lines?

so, if your starting data is:
AAAAA
BBBBB
AAAAA
VFVDS
AAAAA

then you'd first transform it to:
AAAAA|1
BBBBB|2
AAAAA|3
VFVDS|4
AAAAA|5

Then you'd sort it to:
AAAAA|1
AAAAA|3
AAAAA|5
BBBBB|2
VFVDS|4

Then you'd fold it to:
AAAAA|1|3|5
BBBBB|2
VFVDS|4

Is that the process you are looking for?

^ Essentially huffman encoding, eh? I myself am confused to piss by the question.

[Dustspeck]

^ Essentially huffman encoding, eh? I myself am confused to piss by the question.

While it may appear to be Huffman Coding, we don't deal with actual compression of the symbols or likelyhood of the appearance of the symbols. This is more like a linked database with the symbol as a primary key.

I am not entirely clear, either. I hope the OP let's us know if my description is accurate.

[Lakshmipathi]

Do you have any recommendation for a free linux tool?

Have a look at this lessfs | Open source data de-duplication