Find the answer to your Linux question:
Results 1 to 4 of 4
I have a directory that spools files for a faxing application. Once the fax is sent, the files can be deleted after 7 days. This seems like the easiest thing ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Jul 2014
    Posts
    2

    find and delete files older than X days... with a catch


    I have a directory that spools files for a faxing application. Once the fax is sent, the files can be deleted after 7 days. This seems like the easiest thing in the world, but it seems memory use blows up. I have tried the following:

    nice -10 find /uploaded/ -mtime +7 | xargs rm -f

    and

    nice -10 find /uploaded/ -type f -mtime +7 -exec rm -f {} \;

    and

    nice -10 find /uploaded/ -type f -mtime +7 -delete

    and these all seem to eat every bit of RAM on the box.... like 3GB before bombing out.

    To potentially further complicate things, this is an NFS mount. Is there a light weight way to do this? Am I doing this completely wrong? We're talking about 300k+ files per run. This runs via a daily cron.

  2. #2
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    Location
    Munich
    Posts
    3,390
    Hi and welcome

    All three command variations should be fairly memory efficient, with the last being the most efficient one.

    Do you have any RES metrics showing 3GByte?

    Maybe nfs is the problem.
    A short search revealed this memory leak:
    https://bugs.launchpad.net/ubuntu/+s...x/+bug/1047566
    You must always face the curtain with a bow.

  3. #3
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    Location
    Munich
    Posts
    3,390
    In general I am not a big fan of this kind of clean up cronjobs.

    - A cronjob doesnt know application logic.
    There is the assumption, that the files are not needed after x days, but can it be guaranteed?
    e.g.: What if the app introduces a new class of files, that shall stay for 14 days?
    - The cleanup is based on filesystem timestamps.
    a) Which one to choose?
    Probably not atime.
    mtime.. maybe, as a modifikation would reset the cleanup timer which may or may not be intended.
    ctime is also possible, as it shows the "real" age.
    b) the machine might not be time synced.

    The app should track its files and delete them as soon as they are no longer needed.


    As for the current issue and if the app cannot be changed:
    1) Can the fax app be configured to write in a new directory every day? e.g.
    /uploaded-2014-Jul-17
    /uploaded-2014-Jul-16
    /uploaded-2014-Jul-19
    Then you dont need find and you can delete whole directories.

    2) Is this nfs server a unix machine or a closed storage system (emc, netapp)?
    In case of a regular machine, you could run the cleanup job on the local filesystem and not via nfs.
    You must always face the curtain with a bow.

  4. #4
    Just Joined!
    Join Date
    Jul 2014
    Posts
    2
    While I certainly agree with you, the code is the code in this case. Internally built app, resources to fix/change it... non existent.

    1) Would require a code change. Not an option.

    2) This is an Equallogic NFS filer. No local ability.

    I also checked the version of nfs-utils. It is version 1.2.3-39 which is the latest/greatest CentOS 6.x offers and there have been no mention of memory leaks being fixed.

    Neat problem, eh?
    Last edited by Bubbagump210; 07-20-2014 at 10:47 PM. Reason: Engrish

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •