Find the answer to your Linux question:
Page 1 of 2 1 2 LastLast
Results 1 to 10 of 16
HI all, I am working on a solution to a growing problem. I have a realtor MLS database full of listings. Every night, I receive a new table of listings, ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Jul 2006
    Posts
    9

    Bash Script to remove old pics


    HI all,

    I am working on a solution to a growing problem. I have a realtor MLS database full of listings. Every night, I receive a new table of listings, so I delete the old one and load the new one.
    I also receive a photo file every night, but it is an update, not a complete set. In other words, in that photo file are only photos uploaded since the last photo file (which came the night before). I do not get a replacement file.
    All of the photos are stored in a directory. The name of the photo corresponds to a record in the db. I need to periodically remove old photos, or the directory will continue to grow. I need to take each photo name and search for that corresponding record in the db. If it is not found in the db I then I can delete that photo record.
    Any idea how I can accomplish that with a bash script? Oh, and there are probably 20,000 records in the database that I'd have to search through.

    Thanks,
    Carrie

  2. #2
    Linux Engineer Javasnob's Avatar
    Join Date
    Jul 2005
    Location
    Wisconsin
    Posts
    942
    Look into find.
    Flies of a particular kind, i.e. time-flies, are fond of an arrow.

    Registered Linux User #408794

  3. #3
    Linux Guru Cabhan's Avatar
    Join Date
    Jan 2005
    Location
    Seattle, WA, USA
    Posts
    3,252
    I don't know what format the table comes in, but assuming it's plain-text, you can simply take a list of every photo you have (either the find command or fileglobbing will work; I prefer the latter), and then use the grep command to search the table for that file: if it's not listed, the file is just a simple rm away.

  4. $spacer_open
    $spacer_close
  5. #4
    Just Joined!
    Join Date
    Jul 2006
    Posts
    9
    Thank you.

    I need to compare the contents of the photo directory against the contents of a list of fields in the db (mysql). Do you think it can be done with one command?
    That would be great, but I assumed it would take a series of commands.
    The way I started was to load the fields from the db into a text file, and then load the filenames in the directory into a separate textfile. I tried to use diff to compare, but it's not the right command for what I need. I wrote the code below to create a list of all names from file1 not found in file2, but it keeps generating errors.
    What do you think? Is there an easier, faster way?


    Code:
    while read name
    do
       found="no"
       while read line
       do
          echo $line | grep $name >/dev/null
          if [ $? -eq 0 ]
          then
             echo $name found in file
             found="yes"
             break
          fi
       done < file2
       if [ $found = "no" ]
       then
          echo $name not found in file
       fi
    done < file1

    Thanks,
    Carrie

  6. #5
    Linux Guru Cabhan's Avatar
    Join Date
    Jan 2005
    Location
    Seattle, WA, USA
    Posts
    3,252
    I was reading your post, went to take a drink of soda, and realized I still had the cap on <_<.

    Why not something like this:
    Code:
    exec 3< file1
    
    while read line <&3
    do
        if grep -q $line file2
        then
            echo "$line found in file2"
        else
            echo "$line NOT found in file2"
        fi
    done
    As simple as that: I imagine that'll work. Note that file1 is the file containing each picture name (that you have), and file2 contains each db entry.

  7. #6
    Just Joined!
    Join Date
    Jul 2006
    Posts
    9
    I got the script to work, and I think I understand what's going on. What's the 3 for after 'exec' in the first line?

  8. #7
    Just Joined!
    Join Date
    Jul 2006
    Posts
    9
    Hey could you help me with something else?

    I actually have 5 different mysql tables to load the field values from. I tried to write a shell script to connect to the db and run the queries automatically, but within the shell, I can't use backticks (the field name has a space in it). I replaced the backticks with single and double quotes, but instead of getting the field value, I get the field name.
    Do you know how I could write this script and get it to work? Obviously, it makes sense to try to automate this:

    Code:
    #!/bin/bash
    echo starting to get all current mls records
    mysql dbname --user=uname <<**
    SELECT `MLS No` from idx_4 into outfile '/tmp/idx4.txt' ; SELECT `MLS No` from idx_5 into outfile '/tmp/idx5.txt' ; exit
    **
    echo finished getting mls numbers
    from the command prompt, mysql>, the above code will work, but from the shell script it won't. What can I do?

  9. #8
    Linux Guru Cabhan's Avatar
    Join Date
    Jan 2005
    Location
    Seattle, WA, USA
    Posts
    3,252
    Quote Originally Posted by carriehoff
    I got the script to work, and I think I understand what's going on. What's the 3 for after 'exec' in the first line?
    The
    Code:
     exec 3< file1
    opens up a file descriptor for the stream 3 (0 is stdin, 1 is stdout, 2 is stderr), and assigns it to the 'file1'. This way, we can just reference the file using &3, rather than typing out the whole file each time. It's somewhat unnecessary in this example, but I'm in the habit of using it.


    As for the other question, I actually know nothing about MySQL, so that'll need to go to someone else.

  10. #9
    Just Joined!
    Join Date
    Jul 2006
    Posts
    9

    Just files that don't match

    Cabhan,

    I've been using this script and it's been working great. Could you help me modify this script, to get only the files that don't match?

    Code:
    exec 3< file1
    
    while read line <&3
    do
        if grep -q $line file2
        then
            echo "$line found in file2"
        else
            echo "$line NOT found in file2"
        fi
    done
    I thought grep -v would give me all files that don't match, but it's not. I also can't remove the 'echo "$line found in file2" without getting an error. I just want the ones that don't match.

    Thanks,
    Carrie

  11. #10
    Linux Guru Cabhan's Avatar
    Join Date
    Jan 2005
    Location
    Seattle, WA, USA
    Posts
    3,252
    Well hey, I'm glad to see it's going so well for you. I love how such small scripts can really help so much (at NIH, I had little scripts for all sorts of tasks: they're so useful!).

    Anyway, you want only files that don't match? And '-v' doesn't work for you? That's odd...

    Code:
    exec 3< file1
    
    while read line <&3; do
        if grep -q "$line" file2; then
            echo &> /dev/null # basically do nothing
        else
            echo "$line NOT found in file2"
        fi
    done
    This may work...you can't remove the echo line, because then you have if, no statement, then an else. The new echo statement echoes a newline to /dev/null.

    Also note that I've added quotes around the $line in the grep statement. You should add this to the other script as well: it will make the script work even when the value stored in $line is more than one word.

    Give that a shot and let me know how it goes!

Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •