Script to Extract from Website

[jmadero]

Hi All,

This is a bit off topic but I'm trying to create an essential reading list for myself and have come across this site:

Lists of Bests


What I am hoping to do is make a compilation using books that others have in their list, such as this one:

jsherry's "Recommended Reading List for the Well Educated Adult" on Lists of Bests


I've now entered 200 book titles, including author and what not, and it's taking too long. So now the question is, can I make an easy script to extract the book titles (and maybe even the links to amazon that they have) from certain lists? I'm not even sure if this is legal (if it's not, I apologize in advance and don't need to know how to do it )


Thanks in advance

[atreyu]

You mean you want to grab the titles from the page on Amazon? That's perfectly legal. Use wget, something like:

 wget -O books-page1.html http://www.listsofbests.com/list/2366-recommended-reading-list-for-the-well-educated-adult?name=comment_page&page=1

Then parse the output file, "books-page1.html". You can use html2text.py (google for it), to convert the html to plain text and then grep for the titles. A program called "tidy" might help you clean up the html, too. If you wanted to grab the links, then you'd have to parse the html itself, not a plain text version of it.