Find the answer to your Linux question:
Page 1 of 2 1 2 LastLast
Results 1 to 10 of 13
Friends, I was wondering if some one can help me out with my problem. I have a C program which outputs a text file test.out which contains hexadecimal addresses. The ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Sep 2010
    Posts
    3

    Exclamation Compression of a file


    Friends,

    I was wondering if some one can help me out with my problem.

    I have a C program which outputs a text file test.out which contains hexadecimal addresses. The file size is a little more than 100GB. The problem is I don't have enough space to store this file on my hard drive. The space I have on my drive is only 72GB. Can anyone tell me how to compress this file on the fly so that I can store this on my hard disk?

    Again, after I get this file test.out in the compressed format like in .gz or .bz2 I want to use this as an input to a shell script..for example ./simulator < test.out.gz. I will be really grateful if someone helps me out with this as well.

    Thanks

  2. #2
    Super Moderator Roxoff's Avatar
    Join Date
    Aug 2005
    Location
    Nottingham, England
    Posts
    3,890
    Use the power of bash...

    When you call your C program from the command line (or from a script) pass the output through a compression tool using the pipe '|' command, like this:

    Code:
    yourapp | gzip -9 -c > filename.gzip
    Linux user #126863 - see http://linuxcounter.net/

  3. #3
    Just Joined!
    Join Date
    Sep 2010
    Posts
    3

    Arrow Compression of a file

    Thanks Rox but unfortunately this command is not working. I've been trying this for a while now. I posted this query because I was unsuccessful implementing this on bash.

    I tried the way you said... my application | gzip > file.gzip but that doesn't work either.

    The other problem is I want to use this zipped file, for example file.gzip as an input to a script like .....script ........../sim_script < file.gzip. I don't know how to do this either. Can you help me with this too?

    Thanks

  4. #4
    Super Moderator Roxoff's Avatar
    Join Date
    Aug 2005
    Location
    Nottingham, England
    Posts
    3,890
    I've just ran that command using 'ls' as the output. Here's what I did:

    Code:
    $ ls | gzip -9 -c > txt.gz
    Then I unzipped with
    Code:
    $ unzip txt.gz
    and examined the contents of the file 'txt' - it was a simple directory listing. This means the command method is working correctly - something else must be failing.

    What happens if you run your application and spool the output to the console - does it produce human-readable text? (i.e. there is no error with the input, is there?)

    You said you'd tried this application the way I'd said but then you declared 'that doesn't work either.' What error messages did you encounter? Did it try to write out any kind of file at all? (there isn't a permissions problem with writing the file out, is there? Or maybe even the compressed file is too large? Or maybe the file is being written but gunzip is refusing to unzip *.gzip because it's expecting *.gz?)
    Linux user #126863 - see http://linuxcounter.net/

  5. #5
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,639
    Try: myapp | gzip -c >data.gz

    The -9 option that Roxoff mentioned simply tells gzip to use the best possible compression (probably overkill). The other possibility related to your problem is that your application outputs data to stderr or the named file instead of stdout. In such a case, you will need to either add more disc space, or install an external disc. Does the application have some option to specify the name/path of the output file? Do you have the source code to the application so you can change it?
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

  6. #6
    Linux User Manko10's Avatar
    Join Date
    Sep 2010
    Posts
    250
    The other possibility related to your problem is that your application outputs data to stderr or the named file instead of stdout. In such a case, you will need to either add more disc space, or install an external disc.
    Or use
    Code:
    yourapp 2>&1 1>/dev/null | gzip -9 -c > filename.gzip

  7. #7
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,639
    Quote Originally Posted by Manko10 View Post
    Or use
    Code:
    yourapp 2>&1 1>/dev/null | gzip -9 -c > filename.gzip
    Assuming it really is writing to stderr, and not to a named file... Oh well, he has as many options as we can give him, I think.
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

  8. #8
    Linux User Manko10's Avatar
    Join Date
    Sep 2010
    Posts
    250
    I think another problem could be the pipe itself since Linux creates an internal unnamed FIFO buffer for this pipe which might consume some disk space (do they? Or is the producer just paused when the consumer is too slow?). But I don't see a way to work around this instead of pasting output to an external harddrive. Constructs with < also require having a file to get STDIN from.
    Last edited by Manko10; 09-27-2010 at 03:54 PM.

  9. #9
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,639
    Actually, it does not take up disc space. In Unix/Linux systems, most stuff is represented in the file system, including memory (/dev/mem). That doesn't mean they take up disc space. Pipes and redirected files for input/output, take no permanent (or temporary) disc space.
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

  10. #10
    Linux User Manko10's Avatar
    Join Date
    Sep 2010
    Posts
    250
    Of course not each file actually is a file on disk.
    Just wanted to make sure whether unnamed pipes don't take at least swap space when the buffer is full. But yep, they don't seem to do, just wasn't sure about this. Actually if the buffer is full Linux seems to put the producer into a waiting status until the consumer is ready for fetching input again.

Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •