Find the answer to your Linux question:
Results 1 to 4 of 4
Is there an way to tail gzip files as they are written? I've tried "tail -F | zcat" which results in zcat reporting a corrupt stream. I haven't been able ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    May 2009
    Posts
    2

    Equivalent of "tail -F" for gzip files?


    Is there an way to tail gzip files as they are written? I've tried "tail -F | zcat" which results in zcat reporting a corrupt stream. I haven't been able to find a ztail command which is surprising to me given that zless, zcat, and zgrep are all available.

    In our case we had a process that writes huge output files (~20GB) and was running out of disk space. We modified the process to gzip its ouput files. Unfortunately, I seem to have lost the ability to tail the output files to track the progress of the application.

    I can understand why ztail would be harder to implement than other z* commands. It would have to find a block start near the end of the gzip stream, start decompressing from there, and not abort when it reaches the end of file that is still being written to and doesn't have the gzip suffix. I'm hoping that somebody has done this work, or knows of some other work around.

    I've found other people asking similar questions that haven't gotten very helpful responses, so I'm not holding out too much hope. (Can post link because I'm new!) But I'm hopeful that there is a solution for me.

  2. #2
    Linux Guru
    Join Date
    Nov 2007
    Location
    Córdoba (Spain)
    Posts
    1,513
    Quote Originally Posted by tranmetter View Post
    Is there an way to tail gzip files as they are written? I've tried "tail -F | zcat" which results in zcat reporting a corrupt stream. I haven't been able to find a ztail command which is surprising to me given that zless, zcat, and zgrep are all available.
    If anything, you want to pipe the output of zcat to tail -f, not the other way around

  3. #3
    Linux Enthusiast
    Join Date
    Apr 2004
    Location
    UK
    Posts
    682
    IIRC, gzip has some kind of trailer record meaning the file can't be unzipped until the complete file is present.

    Can you change your process so it spits it's log out to the standard output? If so, you can use tee to output the compressed file and output to stdio again, then feed the stdio into split which will produce lots of short files.

    Finally you'd need another process to tidy up, say, all but two of the files split produced so you could cat what was left to see what was going on.

    Code:
    cat file.txt | tee >(gzip -9 > /tmp/file.txt.gz)|split -l 100 - log
    Alternatively you could skip the split and just have the output from tee scrolling past on the console all the time.

    It looks a bit rough and ready, but it should get you somewhere to what you are after.

    Let us know how you get on,

    Chris...
    To be good, you must first be bad. "Newbie" is a rank, not a slight.

  4. $spacer_open
    $spacer_close
  5. #4
    Just Joined!
    Join Date
    May 2009
    Posts
    2
    That is an interesting solution. I might be able to get something like that to work, but there are a couple hurdles that make it difficult:
    1) its a long running process (several days) that we run via nohup so that it doesn't die when if the ssh connection to the machine is dropped.
    2) It produces multiple files of output. I'm not sure how I'd seperate out the data if it were all written to STDOUT
    3) I'd like to be able to check in on the process from both at home and at work

    If I don't find ztail functionality, I may try to go that route.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •