[SOLVED] AWK: Join lines if NF is wrong

[jhonan]

Hi,

I have a large CSV file, tab-delimited with CRLF at the end of each line.

Each line should contain 5 fields (i.e. NF == 5) However, there are rogue CRLF characters in the middle of some records, causing records to be split across two lines.

I want to scan each line, check the field count and if it's !=5 then join that line to the following line.

Example input might be;

one two three four five
six seven eight nine ten
eleven tw<CRLF>
elve thirteen fourteen fifteen
sixteen seventeen eighteen nineteen twenty

In the example, I want to merge lines 3 and 4 to read;

eleven twelve thirteen fourteen fifteen

My attempt at the moment is;

awk 'BEGIN { FS = "\t" } ; { if (NF != 5) {saved=$0;next} {print saved,$0} }'

This doesn't work, I'm getting duplicate records inserted. Any help is appreciated (or suggestions on an easier way to do this using awk, sed, or perl)

[markcole]

I would imagine that you need to also remove the truncated line after you save it.

[jhonan]

Thanks for the reply. Here's my solution. I ended up using getline instead of 'next'

awk 'BEGIN { FS = "\t" } ; { if (NF != 5) {saved=$0;getline;print saved$0} else {print $0} }'