Bash and Python scripts for downloading using wget and cron

[CrowdKiller]

Hi all...

I am writing scripts to download files between 0:00 - 5:00 using wget and cron. At the moment I'm doing the following:

Step 1 --> I have a python script that accepts URLs as input and parses them for use with wget. The commands are written as one line with '&' between each command to allow simultaneous downloading. The line is them printed to a file 'download1.sh'.

print "Create a download bash script..."
print ""
urls = '' 
flag = 'y'
while (flag == 'y') :
  user = raw_input ("Enter URL: \n" )
  logName = ""
  temp = user
  flag1 = "y"
  while flag1 == "y":
    try:
      if (temp.index('/')) >= 0: 
	print temp.index('/')
	temp = temp[(temp.index('/'))+1:] 
	print "temp -->" + temp
	print temp.index('/')
    except ValueError:
      print "in except"
      logName ="/home/user/Downloads/Wget_Downloads/" + temp + ".txt"
      flag1 = "n"
  urls = urls + "wget -c -a '" + logName + "' --directory-prefix='/home/user/Downloads/Wget_Downloads' " + user + " & "
  flag = raw_input ("Would you like to add another URL? [y/n]\n")

print urls
f = open('/home/user/scripts/download1.sh' , 'w')
f.write(urls)

Step 2 --> 'download1.sh' is then called in another bash script 'download.sh' using:

bash home/user/scripts/ 'download1.sh'

Step 3 --> 'download.sh' is called by cron to start the download.

My question is this, how can I remove completed wget commands from the 'download1.sh' script while allowing the other partially completed wget commands to still be run by the next cron job?

Also, I suspect there will be easier ways of doing the same thing, but for educational purposes I'd like to use scripts and wget and not some other GUI downloader.

P.S. --> Don't be too harsh, these are my first python/bash scripts as I'm a very recent linux convert.

[coopstah13]

you could just use urllib in python to download files, read in the file that has the list of files to download, then write back to that file the ones that weren't downloaded