Webcrawler such as teleport pro or superbot for linux?

[pcmanjon]

Does anyone know about a Web Crawler for Linux such as the counterparts for windows (Webwhacker, websnake, teleport pro, black widow and superbot)

I need one, tried these in wine and they crash alot.

Not sure what a webcrawler is?

They're a program that can do something like
# Download all or part of a website to your computer, enabling you to browse the site directly from your hard disk at much greater speeds than if you were to browse the site online
# Create an exact duplicate, or mirror of a website, complete with subdirectory structure and all required files
# Search a website for files of a certain type and size
# Download a list of files at known addresses
# Explore every website linked from a central website
# Search a website for keywords
# Make a list of all pages and files on a website

thanks alot for any replys

[Dolda2000]

Well, it isn't explicitly made for it, but wget has some crawler functionality. To download an entire site, try "wget -rp http://www.site.tld/". It has more options, though. Check "wget --help".

[joesysop]

Wow! You're great! THANKS!

[pcmanjon]

Wow, excellent, it's doing its stuff right now. Thanks a bunch

[pcmanjon]

It doesn't seem to be getting these mp3 files out of the directory listing....

[Dolda2000]

Sorry, but I don't know about that. If noone else is able to provide an answer to it here, try subscribing to wget's mailing list.

[pcmanjon]

got it!

Thanks!

[andutt]

Is it an open site you are trying to use wget against?? wget is following links and syncing webbsites to a local structure. So if the mp3:s doesnt have specific links to them wget will not find them. Or if they are protected behind some kind of memberlogin-mechanism.