Find the answer to your Linux question:
Page 1 of 2 1 2 LastLast
Results 1 to 10 of 11
Hello forum, My first post, even though I've had lots of help from this forum in the past by reading I need to put some background to this issue... I ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Sep 2010
    Posts
    7

    Question Slow with reading directory in bash and other languages


    Hello forum,

    My first post, even though I've had lots of help from this forum in the past by reading

    I need to put some background to this issue...

    I had put a webcam in my livingroom that would write a jpeg each second to some directory /home/rob/motion/. Now as you can imagine, this directory became huge after some 2 weeks. Being lazy, I thought i'll write a script later to move them to sub-directories: motion/YYYYMMDD/HH/, each containing some 3600 images.

    Now, with bash I didn't succeed to write this script (it seemed like bash took ages to even start moving files). So I wrote a ruby script instead. And yes, after half a day and a night, all my files were moved from /home/rob/motion/ to /home/rob/motion/YYYYMMDD/HH.

    Now there's only about 14 directories remaining /home/rob/motion/, and no other files.

    Now here's my problem:

    I'd expect, since there's only a few directories left now in motion/, that it should be quick. But when I do 'ls' in that directory, it still takes +16 minutes to display the contents! After all, there are only a few directories remaining..

    It takes the same time no matter what i do (for example, I tried: for i in motion/*; do echo $i; done).

    My question is, why would it still takes so long to read the contents of the directory?

  2. #2
    Linux Guru Cabhan's Avatar
    Join Date
    Jan 2005
    Location
    Seattle, WA, USA
    Posts
    3,252
    This definitely sounds very strange. One problem that jumps immediately to mind is whether you might have any hidden files in that directory. You can run the command "ls -A" to check for them (though you will have to wait for the command to run, sadly ).

    Another possibility might be disk usage. If your disk is heavily fragmented, it might take a long time to read that directory. How close is your disk to being full?

  3. #3
    Just Joined!
    Join Date
    Sep 2010
    Posts
    7
    Thanks for your answer.

    Yes, I did also run a ls -a on the directory. This showed no other hidden files..

    I've been able to resolve the issue by doing:

    mkdir ../motion2
    mv * ../motion2

    This command also took very long to finish. When I did ls in the directory ../motion2, it was just normal speed - fast. Yet, doing ls -a in the empty directory 'motion', still took very long.
    I did a rmdir on motion, this took a few minutes to finish.....

    Pretty strange. I have the feeling it might be some low-level filesystem stuff... maybe '.' in the directory motion is pointing to 'empty' inodes or something? Perhaps it went through a long list of 'empty' inodes? I don't even know if it works that way....

    But this behaviour seemed very strange to me and undesirable. It would mean that when you move or remove files from a large directory, that it could still impact the performance of accessing that directory?? That seems not good to me..

  4. #4
    Just Joined!
    Join Date
    Sep 2010
    Posts
    7
    oh and about the disk size. I still have plenty left on this partition:

    Code:
    /dev/mapper/deimos-home
                          221G   96G  113G  46% /home

  5. #5
    drl
    drl is offline
    Linux Engineer drl's Avatar
    Join Date
    Apr 2006
    Location
    Saint Paul, MN, USA / CentOS, Debian, Slackware, {Free, Open, Net}BSD, Solaris
    Posts
    1,283
    Hi.

    I'd guess that a large number of files might cause indirect and double-indirect blocks to be used to keep track of pointers to the files. See inode pointer structure - Wikipedia, the free encyclopedia for a representation of this. Directories are just special cases of files, the data they contain are pieces of data about other files and directories: Unix Internals . Succinctly:
    A directory is actually implemented as a file that has one line for each item contained within the directory. Each line in a directory file contains only the name of the item, and a numerical reference to the location of the item. The reference is called an i-number, and is an index to a table known as the i-list. The i-list is a complete list of all the storage space available to the file system.

    The UNIX File System
    A directory is never decreased in size, only increased. It needs to be at the most-recently allocated size in case an item is pointed to in one of the indirect blocks. So even if you move or destroy some files, others in the far-flung indirect blocks would still exist and need to be reachable.

    The implication is that to refer to a particular instance of these items the disk may need to be read a number of times just to get to the right place to find out where a particular item is. Many disk reads implies at least some slow-down.

    The disk space remaining on a device may not refer to the allocation of inodes, which in many filesystems are in an area separate from the data area. I don't think the current problem has anything to do with running out of inode space, however.

    For a user that is concerned only with usage and not with low-level design all this detail can be mind-numbing.

    So the usual advice boils down to avoid placing too many files and directories in one directory.

    That's what I recall from the long-ago days of dealing with situations involving large numbers of files at a service organization. Perhaps someone more knowledgeable will stop by with corrections, and a better description and explanation ... cheers, drl
    Welcome - get the most out of the forum by reading forum basics and guidelines: click here.
    90% of questions can be answered by using man pages, Quick Search, Advanced Search, Google search, Wikipedia.
    We look forward to helping you with the challenge of the other 10%.
    ( Mn, 2.6.n, AMD-64 3000+, ASUS A8V Deluxe, 1 GB, SATA + IDE, Matrox G400 AGP )

  6. #6
    Just Joined!
    Join Date
    Sep 2010
    Posts
    7

    Thumbs up

    Quote Originally Posted by drl View Post
    For a user that is concerned only with usage and not with low-level design all this detail can be mind-numbing.
    My reason to post this here is that I see a curious behavior on my system, and want to understand it. So I am actually interested with low-level design so to say

    Thanks a lot for your time. This gives me some pointers to look into.

  7. #7
    Linux Engineer Kloschüssel's Avatar
    Join Date
    Oct 2005
    Location
    Italy
    Posts
    773
    To me it would sound like some serious bogus and I hope I'm not right with the suspect. As drl noted, directories are just files that store inodes that actually are the children of the directory. If this list now is somehow not cleared up properly on move operations, this can have the result that there are still some inodes in the list that need to be processed at the next ls.

    Have you tried to run a fsck? That should clean up inode lists. ls -i may give also some information and the system calls stat(), lstat(), fstat() may also of interest.

  8. #8
    Just Joined!
    Join Date
    Sep 2010
    Posts
    7
    Quote Originally Posted by Kloschüssel View Post
    To me it would sound like some serious bogus and I hope I'm not right with the suspect.
    How do you mean serious bogus?

    As drl noted, directories are just files that store inodes that actually are the children of the directory. If this list now is somehow not cleared up properly on move operations, this can have the result that there are still some inodes in the list that need to be processed at the next ls.

    Have you tried to run a fsck? That should clean up inode lists. ls -i may give also some information and the system calls stat(), lstat(), fstat() may also of interest.
    Unfortunately I've already moved the contents of that directory to a newly created directory, after which problems were solved. Doing ls in the original directory still resulted in very slow processing.

    I would guess that indeed yes it's some "empty" inode list. Wonder if it would qualify as a bug if it would only be cleaned up after a fsck.

  9. #9
    Linux Engineer Kloschüssel's Avatar
    Join Date
    Oct 2005
    Location
    Italy
    Posts
    773
    Bogus is somewhat like a bug, but not really cause it can be a result of wanted behaviour. If it was really a bug, it would have become popular before. Another point of interest would be:

    * is your harddrive intact?
    * what are the specifications of your harddrive?

    One good thing is, if it is really a bogus/bug it should be reproducable:

    * create partition with fs of yours
    * create directory
    * touch files
    * pipe dummy data into the files
    * move files to subdirectory
    * check outcome

    By the way, have you the original bash script at hand that executed so slow? I would like to take a look at it.

  10. #10
    Just Joined!
    Join Date
    Sep 2010
    Posts
    7
    I had been running motion webcam software with the option enabled for writing an image at least every second, whether or not there was motion detected (output_all = on)...

    Motion was saving these files for 2 weeks in /home/rob/motion/.

    Finally I tried to move it using the following bash "script":

    Code:
      167  for i in *; do day=$(echo $i | cut -c4-11); echo $day; hour=$(echo $i | cut -c12-13); if ! [ -d "$day/$hour" ]; then echo "$day/$hour"; mkdir -p "$day/$hour"; fi; mv "$i" "$day/$hour"; done
    I didn't get the feeling that anything worked, so I tried to do this with Ruby instead:

    Code:
    #!/usr/bin/env ruby
    
    require 'fileutils'
    
    Dir.open(ARGV[0]) do|dir|
      while(file = dir.read) do
        hr=file[11,2].to_s
        day=file[3,8].to_s
        dest=day + "/" + hr
        if (File.directory?(dest) == false)
          puts "Creating " + dest + ".."
          FileUtils.mkdir_p(dest)
        end
    
        if (FileTest.exists?(dest + "/" + file) == false)
          FileUtils.mv(file, dest)
        end
      end
    end
    It took some 1,5 days to finish that, on my 1300 Mhz celeron.

    I'm using ext3 filesystem:

    Code:
    /dev/mapper/deimos-home /home           ext3    defaults        0       2

Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •