Find the answer to your Linux question:
Page 1 of 2 1 2 LastLast
Results 1 to 10 of 13
When did Linux get retarded? Or has it always been so, or am I just running in some sort of, 'retarded mode'? I am running simulations which generate thousands of ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Aug 2011
    Posts
    5

    error about 'too many arguments'


    When did Linux get retarded? Or has it always been so, or am I just running in some sort of, 'retarded mode'? I am running simulations which generate thousands of small output files, when I try to process these with even the most trivial commands I get an error about 'too many arguments' eg:

    ls *.txt
    /bin/ls: Argument list too long.

    Seriously, commands that crap themselves when you give them too many objects to operate on. Is this the 1960's ? I thought hard-coded limits embedded in functions had vanished a long time ago.

    Anyone have any thoughts on this ?

    Thanks,

    Usjes

    PS I'm running Red Hat if that's relevant (cat /proc/version
    Linux version 2.6.9-89.0.20.EL.bz501565.11largesmp (mockbuild<AT>x86-002.build.bos.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-11)) #1 SMP Tue Mar 23 04:20:50 EDT 2010
    )

  2. #2
    Trusted Penguin
    Join Date
    May 2011
    Posts
    4,353
    This is a kernel limitation. See here for a good description of the problem and some solutions. In your above example, you might try:
    Code:
    find . -type f -name '*.txt' -exec ls -l {} \;

  3. #3
    Just Joined!
    Join Date
    Aug 2011
    Posts
    5
    Quote Originally Posted by atreyu View Post
    This is a kernel limitation. See here for a good description of the problem and some solutions. In your above example, you might try:
    Code:
    find . -type f -name '*.txt' -exec ls -l {} \;
    Hi atreyu,

    Thanks for your reply, the solution you suggest does work but it is painfully slow. I'm still shocked that this is a hard-coded limit in the kernel, I'm only dealing with thousands of filesm after all. Unfortunately I don't have the option of recompiling the kernel. It is a batch machine where I don't have admin privileges.

  4. #4
    Trusted Penguin
    Join Date
    May 2011
    Posts
    4,353
    Did you can try that article's method 1 for splitting up the list of files?

    e.g.
    Code:
    ls [a-l]*.txt
    ls [m-z]*.txt

  5. #5
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    Location
    Munich
    Posts
    3,384
    The article mentions, that this limit is also meant as a safeguard against local dos.
    Ok, there are other ways to dos a machine, but every bit helps

    Depending on what you actually intend to do (as ls is probably not the command you want), there might be other ways.
    atreyu showed one,
    Another might be to build the filelist first, and then feed this to e.g. GNU Parallel - GNU Project - Free Software Foundation to take advantage of todays multicores.
    This is under the assumption, that
    - the actual processing takes considerably longer than building the filelist
    - the files dont change from one iteration to the next (or the app is aware of that possibility)

    Or maybe the workflow can be changed to immediately process a new file as soon as it arrives? Inotify comes to mind (implemented here for example lsyncd - Lsyncd (Live Syncing Daemon) synchronizes local directories with a remote targets - Google Project Hosting )

    Or your application simply execs whatever comes next with the new filename as parameter.
    You must always face the curtain with a bow.

  6. #6
    Just Joined!
    Join Date
    Sep 2007
    Location
    Silver Spring, MD
    Posts
    95

    File handles

    Quote Originally Posted by Usjes View Post
    When did Linux get retarded? Or has it always been so, or am I just running in some sort of, 'retarded mode'? I am running simulations which generate thousands of small output files, when I try to process these with even the most trivial commands I get an error about 'too many arguments' eg:

    ls *.txt
    /bin/ls: Argument list too long.

    Seriously, commands that crap themselves when you give them too many objects to operate on. Is this the 1960's ? I thought hard-coded limits embedded in functions had vanished a long time ago.

    Anyone have any thoughts on this ?

    Thanks,

    Usjes

    PS I'm running Red Hat if that's relevant (cat /proc/version
    Linux version 2.6.9-89.0.20.EL.bz501565.11largesmp (mockbuild<AT>x86-002.build.bos.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-11)) #1 SMP Tue Mar 23 04:20:50 EDT 2010
    )
    ======================

    One of the things you might want to do is test your kernel settings.

    First look at your kernel settings:

    sysctl -a | egrep -i "file|limit"

    This statement will provide a listing of the kernel's file and limit parameters.

    The items I am interested in changing would be as follows:

    fs.file-nr = 7008 0 282116
    fs.file-max = 282116

    You would change the values like so:

    sysctl -w 'fs.file-nr=7008 0 282116'
    sysctl -w 'fs.file-max=282116'

    I would change the setting, run ls *.txt, then validate if it fixes your problem.

    Also look at inode-max and inode-state

    fs.inode-nr = 25715 2959
    fs.inode-state = 25715 2959 0 0 0 0 0

    Reference: Linux Kernel limits

    Todd

  7. #7
    Linux User
    Join Date
    Nov 2008
    Location
    Tokyo, Japan
    Posts
    260
    Kernel limitations are a part of every operating system. They exist for the sake of security, and also to enforce certain best practices when using the computer. If the OS is limiting the command you entered, you are probably doing something unusual or unexpected.

    The "ls" command is something of a convenience for shell users, and though it is completely stable, it isn't really intended for industrial-strength file operations. The "find" and "mlocate" programs are much better designed for heavy-lifting.

    Try this command:
    Code:
    # (1) this is much faster:
    find . -type f -name '*.txt' ###You dont need -exec ls -l '{}' \;
    
    # The "-exec ls" must execute a sub-process for each file
    # which slows it down by an order of magnitude.
    
    #find also has commands that print information similar to what "ls" prints
    #but using find's built-in directives, like "-printf", is much faster than using "ls".
    
    # (2) prints last-access time and full path
    find . -name '*.txt' -printf '%a %p\n'
    
    # (3) print last-modified time, size in bytes, and full path
    find . -name '*.txt' -printf '%c %s %p\n'
    Check out the "find" man page for more info, especially the "ACTIONS" section of the man page, which has information on the "printf" directive, and all the different file information you can print with it.

    A good power-user technique is to first create a list-file using "find", then use "grep" to narrow-down the list. It is easier to browse a large list of file names with a text editor than it is with the command line:
    Code:
    find . -type f -name '*.txt' | tee /tmp/files.list
    grep filenames-I-like /tmp/files.list | tee /tmp/group1.list
    grep -v filenames-I-dont-like /tmp/files.list | tee /tmp/group2.list
    
    # Or, pipe find's output to the "sort" program to sort the files:
    find . -type f -name '*.txt' -printf '%s %p\n' | sort -n | tee sorted-by-size.list
    #WARNING: "sort" can be very slow when working with large lists.
    You can further narrow-down your lists with "VI" or "Emacs", then use these lists as inputs to other commands.

    If you discover that the brute-force approach is limited by the operating system, you should try finessing it instead. Its all a matter of learning technique, and you have joined exactly the right forums to learn it!

  8. #8
    Just Joined!
    Join Date
    Aug 2006
    Posts
    16
    You might want to use
    Code:
    echo *.txt
    instead of
    Code:
     ls *.txt
    It doesn't have the argument limit, and is faster too.

  9. #9
    Linux User
    Join Date
    Nov 2008
    Location
    Tokyo, Japan
    Posts
    260
    Quote Originally Posted by John Rodkey View Post
    You might want to use
    Code:
    echo *.txt
    instead of
    Code:
     ls *.txt
    It doesn't have the argument limit, and is faster too.
    All programs in Linux have the same limits on the number of bytes you can spend on arguments to programs, and Tdsan's post above indicated where you can find this system set limit.

    The "*.txt" is a "glob" expression interpreted by Bash, or whatever shell program you are using. The shell replaces this expression with every file in the current directory that matches it, which means bash is dumping (possibly) thousands of files into the command line as arguments. Then, the shell program asks the kernel to execute "ls" or "echo" with all of that stuff as the arguments lists data, and the Linux kernel comes back with an error saying there is too much stuff in the arguments list.

    So, if "*.txt" is overflowing "ls", it will overflow "echo" as well, that is unless the "ls" command has its own hard limit smaller than the system-defined limit.

  10. #10
    Just Joined!
    Join Date
    Aug 2006
    Posts
    16
    I'm sure that there are limits to echo *.
    However, my experience is that when ls * gives too many arguments, echo * does not in the same directory.

    Is it possible that echo, being a 'built-in' to the shell, uses a different method for processing its arguments than ls, which is an external program?


    So, if "*.txt" is overflowing "ls", it will overflow "echo" as well, that is unless the "ls" command has its own hard limit smaller than the system-defined limit.

Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •