Bash script: wget a website and save urls containing a specific string

[nata-ban]

Hi,
I've spent hours trying to figure this out without any luck. Would appreciate your help

I need to "crawl" a website and search for a specific string in the html. If a string is found the url to the page containing it is saved to a file.

So the end result should be a file containing a list of urls.

I tried writing a bash script using wget. My knowledge of Linux is very basic. I am using Cygwin for Windows.

[scathefire]

KrazyWorks » Wget examples and scripts

number 5 sounds close to what you are trying to do

[nata-ban]

scathefire, thank you for taking the time to look into it.

The thing is that I don't have a file with urls to loop through. I have the main url to the website (i.e. example.com), and I want wget to crawl it and check every page for my search string.

[scathefire]

does the --spider option not work?

of course, i don't think it will download the page persay. there are other software options out there though.

[nata-ban]

--spider only checks if the page exists, it doesn't get the actual page contents. I need the page contents to search for the string in the html.

My plan B is to download the whole site with wget, and then do the search on my local version:
wget -r -l 2 SITE_URL

But this will not give me the list of URLs.

[scathefire]

why not do something like:

wget -mk SITE_URL

then, with your local copy, filter through the files.

[nata-ban]

Yes, this is what I ended up doing. Thank you for your help!