Keith Devens .com

Tidy isn't infallible - it's possible to give it input which it can't convert to XHTML on its own, at which point it will spit out fatal errors along with tips on how to correct them, This is fine if you're using it as part of a manual document fixing process but is no good at all for random spidering, unless you're sure that the HTML you are crawling is "good enough" for Tidy to be able to convert it on its own.

I would think that using Tidy as in your idea would be a good idea as an experiment. It would allow us to figure out a decent concept on how many sites have terrible (as opposed to just non-compliant) HTML markup. Terrible sites won't be parse by Tidy (and simple error hanlding can catch the return values spit out by Tidy so you can log the pages that couldn't be parsed.

Thus it would give you a chance to write some elegant code to handle the XHTML output from Tidy for the pages that do work, and as we all know, writing elegant well-thought-out code is much better than writing code full of hacks to handle non-compliant pages (not that it can't be done, it just would take a lot more effort... and depending on how much value you put on your free time for pet-projects, it may also not be worth the effort)

The thing that frustrated me about existing parsers was not really anything about the parsers per se, but the fact that for a screenscraping application, parsing is only half the story. Once you've parsed the document you need to extract the part of the document you actually care about, and to do that you have to write a sort of special-purpose tree traversal. I wrote Beautiful Soup partially so I could parse anything that looked like HTML, but mainly to abstract away the tree traversal. It's not as fast as a handwritten traversal, but it's faster to write and (in my experience) more reliable and mantainable.

If you need to handle arbitrary pages that have nothing in common, where you don't get to analyze a representative sample ahead of time, I recommend considering Tidy + an XML parser. For such a project I'd think you'd be doing mostly parsing and very little extraction.

The niche for which I designed Beautiful Soup was where you had an ugly web page full of fascinating information, where if you looked at the HTML code you'd be able to intuit how to get out the good data. It's intended to make it really easy to transform that intuition into code.

Hi Leonard. I'll probably be using BeautifulSoup tomorrow. I'll comment on how it goes Thanks for making it available.

Leonard, your code was a Godsend. Thanks to your library (and thanks to Python) I got stuff done today that I just wouldn't have been able to get done otherwise. Thank you!

You're welcome! I'm glad you found it useful. Would you be willing to give some details about how you used Beautiful Soup so I can add you to my 'real world examples' section?

Hmm... I'll try to write something up and e-mail it to you. It may be a few days before I get to it.

No problem.