Automating IE with jQuery

This is something I ran across in my shotgun research for IE automation. Neat approach!

Automating IE

Here's the thing. I'm a Windows user - the demands of my main industry require it, and Windows was what there was, when I was getting started in the whole computer thing. And as much as we hates IE, precious, and as much as we likes our Chrome, the fact remains that Windows automation using COM/OLE is a pretty slick setup for integrating GUI stuff.

And yeah, the state of the art for browser automation is Selenium, but for the life of me I can't get Selenium to work with Chrome on my machine, and the documentation is horribly unhelpful in that regard.

The tool of choice has always been Win32::IE::Mechanize, but it has apparently been removed from CPAN - it hadn't been actively developed since 2005, so I suppose that's probably reasonable. Win32::IEAutomation exists but apparently ceased to work with IE7.

I would consider using WWW::Mechanize, obviously, except that for what I want to do, I need the browser visual, and that's not an option with the Mech (which also doesn't support JavaScript, so ....)

So yeah. At the moment, the options are poor. I'm wondering what normal people do. (Probably Selenium, honestly.) I may just get the last active IE::Mech, which is still on CPAN but not indexed; I've asked whether I could adopt it, which might be a mistake.  We'll see.

Update 2013-11-17: Turns out the primary maintainer of Win32::IE:Mechanize is now, ahem, me. Now all I have to do is make it work again, which is sadly no trivial task.

Task: News scraper/tracker

I want to scrape the Reuters news feeds (later, others) into a database for various analytical purposes [eg]. That's going to consist of a daemon on my fileserver that checks the feeds on a period basis and loads things into a database. Then we'll do other analysis on that database. I'm most interested in linking stories and identifying trends.

Yeah, OK, I know this isn't groundbreaking research. It's new for me, though. And it will be a good microcosm of scraping tasks for declaratization as well as a valuable component for all kinds of things. So ... it's a task.

WWW::Mechanize and HTML::TreeBuilder

So I just wrote a little 40-line script to hit my router's HTTP interface and find out the list of DHCP clients currently connected. (This is preliminary to checking each for an rsync server running, and doing a backup on those that do.) To write this script, I used WWW::Mechanize, HTML::TreeBuilder, and maybe an hour and a half of divided attention.

Here are a couple of thoughts that occur.

First, looking at HTML results, then getting HTML::Element's find functions to find the things you're actually looking for, is a painstaking process that ends up with pretty brittle results. The parsley selector language would probably be a far, far better way to look for things in the HTML tree, and has the benefit that it could probably be declarativized pretty easily.

Second, the lack of Javascript capability makes this a particularly error-prone process; my router's forms all use Javascript for preprocessing of form input (which is stupid, but prevalent). I'm not sure that providing an entire Javascript parser is a viable option, though. I guess that would depend on the state of Inline::Javascript, assuming there is one. (If there isn't, well, maybe there should be...)

Third, this script naturally broke down into the Mechanize part and the HTML part. That is, once the document is obtained is when it gets parsed - ... this seems obvious, and maybe I'm still too jetlagged to notice that I've lost my point somewhere.

Anyway, it would probably be most instructive to take some actual scripts like this one, translate them into likely-looking declarative structures, and then implement those.