Compression of a file

[dimpu]

Friends,

I was wondering if some one can help me out with my problem.

I have a C program which outputs a text file test.out which contains hexadecimal addresses. The file size is a little more than 100GB. The problem is I don't have enough space to store this file on my hard drive. The space I have on my drive is only 72GB. Can anyone tell me how to compress this file on the fly so that I can store this on my hard disk?

Again, after I get this file test.out in the compressed format like in .gz or .bz2 I want to use this as an input to a shell script..for example ./simulator < test.out.gz. I will be really grateful if someone helps me out with this as well.

Thanks

[Roxoff]

Use the power of bash...

When you call your C program from the command line (or from a script) pass the output through a compression tool using the pipe '|' command, like this:

yourapp | gzip -9 -c > filename.gzip

[dimpu]

Thanks Rox but unfortunately this command is not working. I've been trying this for a while now. I posted this query because I was unsuccessful implementing this on bash.

I tried the way you said... my application | gzip > file.gzip but that doesn't work either.

The other problem is I want to use this zipped file, for example file.gzip as an input to a script like .....script ........../sim_script < file.gzip. I don't know how to do this either. Can you help me with this too?

Thanks

[Roxoff]

I've just ran that command using 'ls' as the output. Here's what I did:

$ ls | gzip -9 -c > txt.gz

Then I unzipped with

$ unzip txt.gz

and examined the contents of the file 'txt' - it was a simple directory listing. This means the command method is working correctly - something else must be failing.

What happens if you run your application and spool the output to the console - does it produce human-readable text? (i.e. there is no error with the input, is there?)

You said you'd tried this application the way I'd said but then you declared 'that doesn't work either.' What error messages did you encounter? Did it try to write out any kind of file at all? (there isn't a permissions problem with writing the file out, is there? Or maybe even the compressed file is too large? Or maybe the file is being written but gunzip is refusing to unzip *.gzip because it's expecting *.gz?)

[Rubberman]

Try: myapp | gzip -c >data.gz

The -9 option that Roxoff mentioned simply tells gzip to use the best possible compression (probably overkill). The other possibility related to your problem is that your application outputs data to stderr or the named file instead of stdout. In such a case, you will need to either add more disc space, or install an external disc. Does the application have some option to specify the name/path of the output file? Do you have the source code to the application so you can change it?

[Manko10]

The other possibility related to your problem is that your application outputs data to stderr or the named file instead of stdout. In such a case, you will need to either add more disc space, or install an external disc.

Or use

yourapp 2>&1 1>/dev/null | gzip -9 -c > filename.gzip

[Rubberman]

Or use

yourapp 2>&1 1>/dev/null | gzip -9 -c > filename.gzip

Assuming it really is writing to stderr, and not to a named file... Oh well, he has as many options as we can give him, I think.

[Manko10]

I think another problem could be the pipe itself since Linux creates an internal unnamed FIFO buffer for this pipe which might consume some disk space (do they? Or is the producer just paused when the consumer is too slow?). But I don't see a way to work around this instead of pasting output to an external harddrive. Constructs with < also require having a file to get STDIN from.

[Rubberman]

Actually, it does not take up disc space. In Unix/Linux systems, most stuff is represented in the file system, including memory (/dev/mem). That doesn't mean they take up disc space. Pipes and redirected files for input/output, take no permanent (or temporary) disc space.

[Manko10]

Of course not each file actually is a file on disk.
Just wanted to make sure whether unnamed pipes don't take at least swap space when the buffer is full. But yep, they don't seem to do, just wasn't sure about this. Actually if the buffer is full Linux seems to put the producer into a waiting status until the consumer is ready for fetching input again.