Problem using wget

[dhvl2712]

I downloaded this directory using wget using the command "wget -r -c sitename/directories/mp3/directory-i-want". And it downloaded the directory i wanted, but after it was done it also started downloading all the other directories i.e all files from sitename/directories/mp3/ and a bunch of other files from //sitename/ How can I use wget to download only the directory I want and its subdirectories and nothing else? I am extremely new to linux, I installed openSUSE just about a week ago and I love it.

[Dapper Dan]

Have you tried it without the -r option?

[dhvl2712]

I did actually and it only seemed to download the index or something. I'm going to experiment a bit with -r. Thanks anyways.

[sarlacii]

Hey there... I pulled the following notes from my "LINUX COMMANDS FOR NEWBIES" file that I have compiled over my years of Linux use. I'm not too proud to say that I still use it! LOL
Anyway... look at the following command and then select the switches you need... noting that "-r" is best used with some sort of recursion depth limit "-l <integer no.>" and the "-np" switch which prevents wget from pulling everything - the problem you seem to have experienced?

<snip>
wget -t 40 -r -c -nd -np -l 0 -k [url:// or -i urlfile.txt]
-t <n> : retry 'n' number of times before giving up... 0 = infinite
-i : specify files to be retrived using a text file
-r : recursive web-suck
-c : continue the download if it breaks (depends on server support)
-nd : do not recreate the directory sturcture of the web site being sucked.
-np : no parent, i.e. don't start sucking from the root... without
this, wget tries to suck the entire site!
-l <n> : set the recursion depth to 'n' directories... 0 = don't move out of the specified directory.
-k : convert links after downloading to make them suitable for local browsing.
url:// or -i urlfile.txt : contains the remote directory that you wish to
start the suck from.
<snip>

Good luck!

[MikeTbob]

Hey there... I pulled the following notes from my "LINUX COMMANDS FOR NEWBIES" file that I have compiled over my years of Linux use. I'm not too proud to say that I still use it! LOL
Anyway... look at the following command and then select the switches you need... noting that "-r" is best used with some sort of recursion depth limit "-l <integer no.>" and the "-np" switch which prevents wget from pulling everything - the problem you seem to have experienced?

<snip>
wget -t 40 -r -c -nd -np -l 0 -k [url:// or -i urlfile.txt]
-t <n> : retry 'n' number of times before giving up... 0 = infinite
-i : specify files to be retrived using a text file
-r : recursive web-suck
-c : continue the download if it breaks (depends on server support)
-nd : do not recreate the directory sturcture of the web site being sucked.
-np : no parent, i.e. don't start sucking from the root... without
this, wget tries to suck the entire site!
-l <n> : set the recursion depth to 'n' directories... 0 = don't move out of the specified directory.
-k : convert links after downloading to make them suitable for local browsing.
url:// or -i urlfile.txt : contains the remote directory that you wish to
start the suck from.
<snip>

Good luck!

Man, I'm glad you looked this up, yesterday I started reading the man page for wget and uh well.....I'm still reading and I'll get back to you later. Seriously, have you ever got to the end of man wget? I think that one man page takes up half my partition space!

[sarlacii]

LOL I wanted to download some links one day, and I decided to use my favourite tool, wget. The first time I set it going i noticed that I was happily trying to get the entire website! Off to those man pages... and as you say, have you even seen that many switches? I try to stay away from anything that long... it generally means I've stumbled into some low-level tool for guru's only!
Anyway, I took the time to go through it all, and once I had the formula right, I stuck it in my notes as hard earned knowledge. The thought of having to work it out again someday... scary! LOL
But, that said, I love wget... it is powerful.

[waterhead]

Last year I wanted to download Suse 9.1 from a ftp repository, for a friend. (Don't ask why!) I was having a lot of trouble getting everything to download. One recommendation I got was to use the mget (multiple get) feature found in ftp. From the ftp man pages:

     mget remote-files
                 Expand the remote-files on the remote machine and do a get for each file name thus produced.  See glob
                 for details on the filename expansion.  Resulting file names will then be processed according to case,
                 ntrans, and nmap settings.  Files are transferred into the local working directory, which can be changed
                 with ‘lcd directory’; new local directories can be created with ‘! mkdir directory’.

You would use the * wildcard to download everything.

ftp> mget *
or
ftp> mget *.*

I actually ended up using gftp for my download, so I don't know if mget will actually work.

Edit: Here is a How-To on using the ftp command.

FTP mini-HOWTO