[SOLVED] Saving output without control characters

[damianpfister]

I have a text log file which has a copy of everything typed on the console. When I cat this file it will show me the likes of:

server1:>ls
server1:>who
server1:>top

...and so on.

Now if I either cat -v or simply vi this file I get the following:

server1:>ll^Hs^M
server1:>which^H^H^H^H^Hwho^M
server1:>tap^H^Hop^M

Now it is obvious that the ^H is the control character for BACKSPACE and ^M for RETURN - both of which are actually translated by cat so that the final command executed is shown and not all the errors (plus backspaces/enters).

It is easy enough to get rid of the ^M through sed or tr, but how do you get around all those ^H characters?

If I type in:

server1:>grep top outputfile.txt

(where outputfile.txt has the above 3 commands in it), it does not show me the last line of server1:>top since the actual output was server1:>tap^H^Hop^M

This makes it difficult to manipulate the data in this text file since the file is not "true" text but a combination of text and control characters:

server1:>file outputfile.txt
outputfile.txt: ASCII text, with CRLF line terminators, with overstriking

Is there any way I can do a cat of this file and then redirect that to a file, with that file looking the same as it would to STDOUT (console/xterm)?

If I currently try to do:

server1:>cat outputfile.txt > newfile.txt

I am left with all those original control characters, which makes for viewing in vi or parsing with grep difficult. Somewhere between cat and the console those control characters are actually interpreted rather than simply displayed....something I just cannot seem to replicate!

Any suggestions?

[Kieren]

You can remove control characters in vi using the following command:

1,$s/^M//g

To get the ^M character don’t type ^ and M! You need to press Ctrl+v and then Ctrl+m. You should be able to get each control character by changing the M to which ever you need

The 1 at the beginning means start at line 1 and the $ means finish on the final line.

s is the search and replace command. It is used in the format of s/string to search for/string that replaces the search string/

The final g is part of the search and replace command. it makes the command global. If you don’t have the g on the end only the first instance is replaced.

[damianpfister]

The problem is that a simple removal is not what is needed - rather a translation.
Removing ^M is easy enough - it is the ^H that is the problem since it needs to be translated (something cat or the terminal does) rather than deleted/replaced.

When you actually cat the file, it displays the "interpreted" output based on all the control characters, thus "masking" the fact that there was one command typed followed by a bunch of backspaces and then another command, before RETURN was hit.

Converting ^M (Carriage return) is fine, but how do you convert ^H when it is essentially a backspace that signifies the previous character needs to be erased?

Another example
cat file.txt

Hello World!

cat -v file.txt

Hello everyone^H^H^H^H^H^H^H^Hworld!^M

I tried doing a cat of the file (without -v so it shows up as I want it to...without control characters) and then an xsel to "select" the output and put that into the mouse copy buffer (middle-click paste) and then try output that to a file. No joy - get the exact same thing.

Yet if I cat the file, select the text with a mouse and paste it (middle-click) into another file it copies Hello World! rather than Hello everyone^H^H^H^H^H^H^H^Hworld!^M which is what I want (no control characters)....but it is not practical using a mouse (especially when inside of a script).

[drl]

Hi.

I would use:

col -b

see man col for details ... cheers, drl

[damianpfister]

I have tried the col -b option and it appears to work!

cat file.txt
Hello World!

cat -v file.txt
Hello everyone^H^H^H^H^H^H^H^Hworld!^M

col -b < file.txt > newfile.txt
Hello World!

cat -v newfile.txt
Hello World!

So essentially col -b acts as a filter of sorts, "interpreting" the backspace characters and only showing the final output not the initially deleted word nor the backspaces themselves.

Thanks a lot drl! I had wasted weeks on trying to figure this one out....