Text/NLP stuff

Like clockwork, I collect links to interesting NLP stuff.

• GATE: a General Architecture for Text Processing
• textteaser extracts summaries with machine learning
• Dezi is a Lucy/Lucene-alike in Perl

Churnalism

Churnalism is a tool for tracking journalistic plagiarism by scanning a database of existing text for phrases found in a given article. It's a pretty straightforward application of the open-source SuperFastMatch [git] text comparison tool, which I should probably investigate in greater detail.

Memetracker

Considering, just for a moment, that I might take up the practice of following blogs (which would be stupid because I don't even have time for the things I already do right now), I looked at tools for following blogs, and from there to the concept of a memetracker, and from there came to a Drupal module of that name looking for a maintainer.

I could see that.

But yeah.  Meme tracking.

• Memetracker.org - moribund, but follow the authors' links for loads of interesting things.
• Data from Memetracker - for 2008 and 2009 in support of a paper.  As is the whole site, it appears.  I could see doing that, too.

You know where I'm going with this blog, right?

I want to do textual analysis on both the blog and everything it links, then search for similar things so I can just let the blog run itself, more or less.

Instead of letting HackerNews find things for me, I can do it myself and maybe start feeding HackerNews instead of the other way around.

Zemanta

I'm trying out Zemanta, which sits in your blogging editor and analyzes your text on the fly to suggest possible relevant links and images.

It remains to be seen how useful it'll be, but at it's text analysis, I can't help but be interested.  (And they've got an API!)

So far, the only negative is that I have to use the mouse to pull my sidebar down for tags for my posts.  That's not too horrible.

Interesting big data/textual analysis blog

Another Word For It.  The author is interested in topic maps, an automatic semantic structuring technique for large document collections.

My problem: I have never developed the habit or a toolset for consuming news.  If something bubbles up on HNN, that's historically been enough for me (before HNN, it was just whether I encountered something in other online communities; HNN is a lot more efficient in finding things I like, though).

I need to do something about that.

Update 2012-11-06: And then I find CPAN module "TM" (topic maps) today - freaky!

KHC text analysis library

Another.

Bayes classification of HNN posts

I love this meta stuff. I also love the idea of something to read the news for me. Ha.

Stylometry

I remain unconvinced that stylometry can do more than identify "statistical identity trends", but it's still interesting work. [boing boing]

Google Code search algorithm

This is a pretty neat article by the guy that wrote the search engine for Google Code.

Dada Engine

OK, so back in the 90's, Andrew Bulhak came up with a snazzy little engine that interprets a grammar to generate random text. Well, we've all done that, of course, (here's a Tcl translation of one grammar) but his is implemented as an interpreter that takes the grammar as a specification language, and it can use troff to generate some pretty nice output. Like this, famously. And more recently, spam, apparently, which explains a lot. Here's a page that can run it on any grammar you like. (Incidentally, a Google search on "Dada Engine" turns up a paid ad for a code generator.)

So the engine itself is kind of boring, although he's put some really nice features into it. What really gets me going is thinking about that grammar specification language. I know this is Not a New Idea, but his sentence or phrase patterns are templates - syntactic units expressing semantic units, getting back to my Langacker days - that bear research. What he's missing is the semantic pole, although the context his grammar carries along is something in that direction.

What would be interesting would be something that could mine the Web for such patterns. And a statistical analysis of their use and interrelatedness to produce some kind of indication of voice/register for the text. That kind of thing. Not to mention a compact set of "typical error message" patterns, etc. for practical text generation in software. The point being that I think the pattern-based approach could work for both text analysis and text generation (not that this is a new idea).

This is back to my notion of the Lexicon, last looked at seriously in 2005. I really have to take a sabbatical from this damned having-to-earn-money thing.

Text analysis - identification of sources in news articles

So I'm taking this online class about journalism, and one of the exercises is to identify the sources in a news article. By hand, of course, this is easy. Wouldn't it be nice to automate it (even partially)?

Of course, nothing is easy when natural language is concerned. I see two parts to this, clearly. First is taking a page and extracting the news item. Frankly, I don't see any better way to do this than simply to have a bunch of definitions for different news services that could identify the CSS classes used by each of them to mark their payload text. And this is exactly the kind of task that a pattern-matching language would be dandy for.

Which leaves us with the text, and its analysis. Which is hard. I can think of a couple of ways to get some sources out of a given text: "'...' said x" is one obvious pattern. Without language-savvy tools, it would be a series of hacks, but maybe worth the effort. (With language-savvy tools, a lot of this stuff starts to look more amenable to solution, though, doesn't it?)