Find the answer to your Linux question:
Page 1 of 2 1 2 LastLast
Results 1 to 10 of 12
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1

    progress - to monitor progress of commands


    hi all,

    i have found a really good command that monitors the progress of a bunch of commands, its here -

    https://github.com/Xfennec/progress

    once installed you can run this command -

    cp -r /mnt/local/data/call_the_midwife_7_1708/ /mnt/local/data/new/

    then open a new terminal and run this command -

    watch -n 0.5 progress -w

    this will give you this -

    Every 0.5s: progress -w Wed Sep 13 15:05:16 2017

    [12254] cp /mnt/local/data/call_the_midwife_7_1708/Promo/grading_output/for_approval/170818_ctm_7_mipcom_graded_1-1_10bit_422_ycc_f2l_bl_or/192
    0x1080/170818_ctm_7_mipcom_graded_1-1_10bit_422_ycc_f2l_bl_or_V1.mxf
    19.6% (2.3 GiB / 11.9 GiB) 27.4 MiB/s remaining 0:05:55

    has anyone heard of this

    but this gives you details of individual files being copied over, i need something that can give me the ETA and percent of the whole directory copied over and not just individual files?

    rob

  2. #2
    Linux Newbie sarlacii's Avatar
    Join Date
    May 2005
    Location
    South Africa
    Posts
    130
    Hi there

    There are, as with all things Linux, a number of ways to do this sort of thing. Depending on what you want to achieve, look at using commands like "pv" (which pipes std input to std output, so is really useful for simple man-in-the-middle monitoring of progress using pipe redirects) or "rsync" with progress indication turned on, etc.

    With "pv" you can check progress on files easily... but directories are a bit harder. For that you should probably use "rsync --progress". You can of course use pipes (may be slower) and get the total directory size with "du"... like
    Code:
    <command1> | pv -s $(du -sb ${backupItem} | awk '{print $1}') | <command2>
    For ideas, check out this post on stackexchange.
    Last edited by sarlacii; 09-14-2017 at 07:16 AM. Reason: Grammar... i think. :)
    Respectfully... Sarlac II
    ~~
    The moving clock K' appears to K to run slow by the factor (1-v^2/c^2)^(1/2).
    This is the phenomenon of time dilation.
    The faster you run, the younger you look, to everyone but yourself.

  3. #3
    mmm...

    got me thinking can i use the command pv and progress together to get the total ETA/percent of the whole directory instead of an ETA/percent of each individual file in the directory?

  4. $spacer_open
    $spacer_close
  5. #4
    Linux Newbie sarlacii's Avatar
    Join Date
    May 2005
    Location
    South Africa
    Posts
    130
    Quote Originally Posted by robertw View Post
    mmm...

    got me thinking can i use the command pv and progress together to get the total ETA/percent of the whole directory instead of an ETA/percent of each individual file in the directory?
    Use tar to create a stream that you can pass through the pipe, as per the stackexchange link I posted. Like so:
    Code:
    tar c ./sourceDir/ | pv -s $(du -sb ./sourceDir/ | awk '{print $1}') | tar x -C ./destinationDir/
    Respectfully... Sarlac II
    ~~
    The moving clock K' appears to K to run slow by the factor (1-v^2/c^2)^(1/2).
    This is the phenomenon of time dilation.
    The faster you run, the younger you look, to everyone but yourself.

  6. #5
    thanks sarlacii

    tar c sourcedir | pv -s size | tar x -C targetdir

    i thought tar can only handle files not directorys?

  7. #6
    Linux Newbie sarlacii's Avatar
    Join Date
    May 2005
    Location
    South Africa
    Posts
    130
    Ha ha, you can tar basically anything.
    Respectfully... Sarlac II
    ~~
    The moving clock K' appears to K to run slow by the factor (1-v^2/c^2)^(1/2).
    This is the phenomenon of time dilation.
    The faster you run, the younger you look, to everyone but yourself.

  8. #7
    cant believe i said that, what an idiot , sorry of course you can tar directorys and output it as somefilename.tar

    but the code "tar x -C targetdir" wont it create it in a tar file ie like so

    /targetdir/whatever the direcotry is called.tar

    i dont want it to tar it

  9. #8
    i get this -

    [root@robw-linux data]# tar -c call_the_midwife_7_1708/ | pv -s 32455212 | tar -x -C /mnt/local/data/new/
    11.9GiB 0:08:17 [23.3MiB/s] [================================================== ===============================================] 39377% ETA 0:00:00

    i get no ETA and the percent is crazy

  10. #9
    smashed it -

    [root@robw-linux data]# tar -c call_the_midwife_7_1708/ | pv -lep -s 32455212 | tar -x -C /mnt/local/data/new/
    [=> ] 2% ETA 2:34:31

    and to find the dir size i did -

    du -s call_the_midwife_7_1708/

    but doing it via this method takes ages as its creating the tar and extracting the tar, normally doing a normal copy only takes roughly 18 minutes

  11. #10
    just thought of another idea -

    il get the size of the source path -

    du -s /source_path/

    then i will start the copy -

    cp -r /source_path/ /destination_path/

    while im copying i will monitor the progress -

    watch -n 0.5 du -s /destination_path/

    but i want to do this all in a bash script but my issue is it wont watch the destination path while the copy is going on, how do i do both at the same time

    rob

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •