check for word frequency in 1 column

[foxyg]

Hi, I have a problem I am having trouble to solve.

I have a large tab delimited text file, about 17gb. It only has 6 column. On column number 4, it is all numbers. Ranging from 1-1000. I want to count how many times each number occured. So the output I want is in 2 columns, first one is a number, second column is how many times it occured.

I tried
head -n 1000 coverage | cut -f 4 | uniq -c

Didn't work for me, the first column returned is not unique.

Can anyone help?

Thanks

[nmset]

1. It would help if you post a sample data file.

2. I don't that it's up to bash tools to work out these kinds of tasks and I'm not sure we could easily do it.

3. You should import your huge structured data file in a database and issue a select count command with group by clause.

[drl]

Hi.

The command uniq does not consider items duplicates unless they are adjacent. Try

head -n 1000 coverage | cut -f 4 | sort -n | uniq -c

cheers, drl

Note: 'uniq' does not detect repeated lines unless they are adjacent.
You may want to sort the input first, or use `sort -u' without `uniq'.
-- excerpt from man uniq, q.v.

[nmset]

@foxyg

Ranging from 1-1000.

This is ambiguous. Just for the sake of clarification, did you mean that values in column 4 are in that range ? or that you wish to sample the first 1000 lines of you file ?