Find the answer to your Linux question:
Results 1 to 8 of 8
Does anyone know about a Web Crawler for Linux such as the counterparts for windows (Webwhacker, websnake, teleport pro, black widow and superbot) I need one, tried these in wine ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Jul 2003
    Location
    TEXAS
    Posts
    19

    Webcrawler such as teleport pro or superbot for linux?


    Does anyone know about a Web Crawler for Linux such as the counterparts for windows (Webwhacker, websnake, teleport pro, black widow and superbot)

    I need one, tried these in wine and they crash alot.

    Not sure what a webcrawler is?

    They're a program that can do something like
    # Download all or part of a website to your computer, enabling you to browse the site directly from your hard disk at much greater speeds than if you were to browse the site online
    # Create an exact duplicate, or mirror of a website, complete with subdirectory structure and all required files
    # Search a website for files of a certain type and size
    # Download a list of files at known addresses
    # Explore every website linked from a central website
    # Search a website for keywords
    # Make a list of all pages and files on a website

    thanks alot for any replys

  2. #2
    Linux Guru
    Join Date
    Oct 2001
    Location
    Täby, Sweden
    Posts
    7,578
    Well, it isn't explicitly made for it, but wget has some crawler functionality. To download an entire site, try "wget -rp http://www.site.tld/". It has more options, though. Check "wget --help".

  3. #3
    Just Joined!
    Join Date
    Aug 2003
    Posts
    31
    Wow! You're great! THANKS!

  4. #4
    Just Joined!
    Join Date
    Jul 2003
    Location
    TEXAS
    Posts
    19
    Wow, excellent, it's doing its stuff right now. Thanks a bunch

  5. #5
    Just Joined!
    Join Date
    Jul 2003
    Location
    TEXAS
    Posts
    19
    It doesn't seem to be getting these mp3 files out of the directory listing....

  6. #6
    Linux Guru
    Join Date
    Oct 2001
    Location
    Täby, Sweden
    Posts
    7,578
    Sorry, but I don't know about that. If noone else is able to provide an answer to it here, try subscribing to wget's mailing list.

  7. #7
    Just Joined!
    Join Date
    Jul 2003
    Location
    TEXAS
    Posts
    19
    got it!

    Thanks!

  8. #8
    Linux Engineer
    Join Date
    Apr 2003
    Location
    Sweden
    Posts
    796
    Is it an open site you are trying to use wget against?? wget is following links and syncing webbsites to a local structure. So if the mp3:s doesnt have specific links to them wget will not find them. Or if they are protected behind some kind of memberlogin-mechanism.
    Regards

    Andutt

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •