Stripping links from web page

[ProgramIt]

I am having problems with a script I am working on and was looking for some help. I have a script that grabs the source code from a google search page. I would like to find out how to strip the links from the source as well as the link name

example:
some stuff <a href="www.somedomain.com/webpage.html"> Click Here </a> some more stuff...

I would like to read both the "www.somedomain.com/webpage.html" and "Click Here" part of it out of the source for all links. If you take a look at the source or a google search the page is unformatted and has multiple links per line. I was trying to use a combination of sed and awk to do this but no matter how I go about it I keep getting stuck. Can anyone give me a suggestion on how is the best way to do this?


Thanks,
John

[Ziplock]

Perl should be able to do it for you. The following module is probably a good place to start:

HTML::LinkExtor

There may be others.

[elija]

This regexp should give you a start

<a[^>]*>

[ProgramIt]

Thanks for the replies. I had one more question. Is there a regular expression that matches the first occurrence of something say in

stuff <a href="www.site1.com"> Link1 </a> more content <a href="www.site2.com"> Link 2 </a> and more stuff

is there a way to match any thing before that and then that. something like

sed 's/.*<a href=\"//' -- this doesn't work though I already tried it just a rough example

so that it would output

www.site1.com"> Link1 </a> more content <a href="www.site2.com"> Link 2 </a> and more stuff

Thanks,
John

[Ziplock]

That probably gives you:

www.site2.com">....

The regular expression '.*' is GREEDY (takes as much as it can from the string). A ? character reverses this. What you are looking for is:

sed 's/.*?<a href=\"//'

[drl]

Hi, ziplock.

That probably gives you:

www.site2.com">....

The regular expression '.*' is GREEDY (takes as much as it can from the string). A ? character reverses this. What you are looking for is:

sed 's/.*?<a href=\"//'

Possibly a typo, the minimal *? quantifier works in perl, but not in GNU/sed 4.1.2 & 4.1.5 that I use. Does yours work differently? ... cheers, drl

[Ziplock]

Hi, ziplock.

Possibly a typo, the minimal *? quantifier works in perl, but not in GNU/sed 4.1.2 & 4.1.5 that I use. Does yours work differently? ... cheers, drl

You're totally right drl, sorry about that - I use perl for most things so sorry if I've confused anyone. I had a go at getting this to work in sed without any successs.

The following will work, but is inefficient as it has to start the perl interpreter every time you add a new line:

 <pipe stuff into> | perl -e 'while (<STDIN>) {s/^.*?href=\"//; print $_}'

The other way is to do the whole thing within perl (or any other language): send query, get response, parse output, reformat, output to browser. This is how I would be doing it given a choice