Parsley

Parsley is a data extractor for HTML.

ScraperWiki closed?

Huh. The open ScraperWiki forum structure seems to have been closed up. That's a shame. I wonder where people interested in scraping congregate now.  (Well, now it's Big Data and monetized, I guess. Maybe there is no such general-interest forum now that it's getting ramified like that.)

Mailparser.io

Email data extraction as a service.

Scraping project: FictiveKin's recipe project

FictiveKin is trying to restore the glory of Punchfork, a recipe site that unceremoniously shut down without warning or backup, starting with the scraping of publicly available recipes.  If I get some time, I'd like to help. Each publisher needs a separate scraper.

Web scraping with node.js

Nice post on Web scraping, using node.js - but the techniques are pretty universal and really worth a read for any platform.

Spidey

A boilerplate extractor in Ruby to simplify scraper writing.

boilerpipe

When scraping, removal of boilerplate is job #1.  Boilerpipe is a library to do that (one of several, of course).  It provides a sort of "de-boilerplating" step.  (And this is probably a really good way of looking at things.)

On the same topic, a really fantastic overview of web scraping here.

Grepsr

Grepsr appears to be a consulting service slash platform for repeating web scraping.  Pricing appears high.

Bot links

I'm collecting too much stuff in the tabs, so here's a little link dump, some of which is redundant within this blog:

• Scrappy (blog post by the author; it's been rebuilt on Moose)
• WWW::Wikevent::Bot - a useful example of a bot
• Javascript in Perl!  Seems a little static (2010) but it's better than anything I've built yet.
• Example code for Mechanize - good for reimplementation, you see.
• TreeBuilder.  Everything coming into Bot::Page will be parsed.
• RDF::Scutter.  I still don't know what this does, except maybe it's gathering semantics?
• The Spidering Hacks book (2003) from O'Reilly.  Mine it for examples.

Web scraping

And yet I just can't get past thinking about Web scraping as something fun and profitable.

If it could just be simplified, a lot.  So I'm thinking again about declarative means of describing the "shape" of a site in terms of where the useful data is - and I'm coming up empty.  Again.

The only way to get my mind around it is to build some Web scrapers.  Elance is not going to be an interesting place to find challenging scraper specifications, so I'm going to have to look at the ones on ScraperWiki and go from there.

Oh, ho! The Mechanize Cookbook is replete with interesting examples.  I shall start there.

Update: Those seem boring and old.  Instead, I've subscribed to the ScraperWiki mailing list, which involves requests to the masses.  Here's a cool one already: find all the churches in Germany, with lots of links to start with.  So yeah.

Pattern: Python web-mining system

Drool.  Of course, nothing restricts Decl to Perl...

Common Crawl code contest

Cool.  They want volunteers (I don't (yet) qualify as a paid coder there).  I should really look into that.

Web text scraping

Good article on figuring out where the heck the content is on a given Web page. References this blog post elsewhere.

YouTube insult generator

Slick little Webscraping application.

Pastebin harvesting

Here's a fascinating idea. Pastebin has a public section. Concerned about abuse, one Internet citizen scraped it to see what was there - there were lots of .... things there of dubious ethics. Let's say.

Why not scrape it daily and analyze what's there? This seems kind of interesting to me, and I'm not even entirely sure why.

Link dump time: Scrappy

Scrappy is a Perl web scraping module that is starting to look pretty damn nice.

Scraping

I have a spider app in mind. I was DuckDuckGoing on Toonbots, for whatever reason, and ran across a couple of links to "Toonbots forum, being able to avoid spammers and trolls and whatnot." I vaguely recall that conversation.

Well, it turns out that that sentence was incorporated into a whole series of spammy landing pages inserted all over the web, pointing back to e-loan.expert.com via Javascript redirect. This has been a couple of weeks ago, so many of these are getting rolled back up and fixed, but ... it would be absolutely fascinating to make it a statistical project.

I know, I know, I'm a sucker for Web spidering, too. Sigh.

This is how it would work:

• Seed it with one or more of the target sentences.
• Google a sentence.
• Try to find its actual origin.
• Store all the other URLs and text; break the text up into sentences.
• Spread from there.
• Index all the Javascript, in case techniques varied.
• Try to categorize the type of host (different breaking techniques were probably in use).
• Track down everybody making this possible, and fix each and every one of them. Oy.

Wouldn't that be cool?

Seed sentences:

• Toonbots forum, being able to avoid spammers and trolls and whatnot.
• The world is awash in fast money, he said, and it is changing the structure of capital markets.
• When you are pre-approved by spruce mortgage, you will have access to hundreds of loan programs.
• Many sellers would rather have a monthly check than a lump sum settlement when they sell.

Example page: http://cas.ncat.edu/Departments/dance/js/dojo/pbs/ohioautotax.html - hasn't been cleaned up yet! Note that it's inside a Javascript directory for something. This is the kind of thing it would be cool to track down.

Update 11/19/11: That particular link has been cleaned up, but the network as a whole is still there and still forwarding to the same ... actually, just a very similar site. Of course, there may have been multiple sites all along. So this project is still waiting to be done.

Startup tacklebox

A venture (not yet a startup) that I've started with a couple of other guys, cataloging everything we can find about startups and how to run them, in the form of a Wiki.

What would be nice is a spider that would create and maintain a database of all the external links on that site, the date they're added and who added them, and where they appear. A manual effort to categorize them would be next. Finally, we'd try to do some kind of comparison and ranking to determine best practices for Tool X.

The first phase there is the classic textbook case of data scraping. A build-or-buy decision could result in the setting up of a scraper on ScraperWiki or the building of a hosted tool. I'm going to do both (the former for public consumption - saving time - and the latter for my own edification).

Scalable Web crawling

Posted without further comment because I haven't taken the time.

Next up: WWW::Declarative again

I feel the need to start writing scrapers for real. With WWW::Mechanize, HTML::Treebuilder, and Data::Match, I've got most of the heavy lifting ready to go. So that's where I'm looking.