Find the answer to your Linux question:
Results 1 to 3 of 3
Hello, I need to find modification times for a long list of files (around 25000). Network issues aside, what is the fastest way to do that? If I just try ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Jan 2013
    Posts
    1

    what is the fastest way to find modification times for many files?


    Hello,

    I need to find modification times for a long list of files (around 25000). Network issues aside, what is the fastest way to do that?

    If I just try
    >cat list_of_files.txt | xargs stat -c %Y

    it takes around 4 minutes on my machine/my filesystem. Is the above the fastest way?

    Thank you for any insight...

    Mark

  2. #2
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,677
    Your approach is not a bad one. The time mentioned (4 minutes == 240 seconds) is equivalent to 10ms per file to probe. This does not seem bad to me. My guess is that if you were to do the same again right away, it would be a lot faster, given that the directories in question would already be cached in memory - no disc I/O to perform most likely.
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

  3. #3
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    Location
    Munich
    Posts
    3,412
    Hi and welcome

    This should be a bit faster, as there is no need to spawn 25000 stats:
    Code:
    stat -c %Y $(cat list_of_files.txt)
    Note:
    There is a risk to run into an "Argument list too long" error, if the length of arguments in bytes exceeds 1/4 of the stacksize (uname -s).
    So if you need to get these modification times via script/cronjob then either use additional logic to keep the argument length below the maximum
    or simply add e.g. --max-args=100 to your xargs:
    Code:
    cat list_of_files.txt | xargs --max-args=100 stat -c %Y
    Note2: The above is a rule of thumb and fragile for very long filepaths.
    Last edited by Irithori; 01-06-2013 at 10:42 PM.
    You must always face the curtain with a bow.

  4. $spacer_open
    $spacer_close

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •