Local files in JavaScript

Well, this is new. Under HTML5, JavaScript has access to the local filesystem. Wasn't the lack of local access a specific security feature, though?

NLP

I actually started working with natural language this week - by trying my hand at some translation tools at last. First, some links I ran across:

• A list of R packages for NLP, with an intriguing link to Weka, a set of Java implementation of data mining algorithms.
• StackOverflow reference to NLTK and n-gram extraction.
• Note "PMI", point-wise mutual information, cited in the SO link.
• Lucene is NLP for Apache; there is a PyLucene as well. But honestly I think I'm going to have to get my hands dirty in Java, because Java seems inordinately popular in the NLP field.
• JCC is a code generator developed for use in PyLucene.

OK, so I got Text::Aspell working on Windows with MingW32, which was no trivial task (in the end it just required some library setup and a small change to the installation script for the module, but it took a full day to figure that out), and got down to the business of building word lists and checking them. This works almost well, except that it immediately became obvious that I need a better tokenizer.

While prowling the Net for English tokenizers in Perl (note: there aren't any good ones - yet), I found:

• Europarl sentence splitter
• Europarl tokenizer
• A post on sentence splitting options (2007)
• Lingua::EN::Sentence, Text::Sentence
• A tech report on tokenizing for biomedical text indexing

In the end, I resolved to write a proper tokenizer for English; it would recognize certain entities embedded in the text (URLs, numbers, numbers with units, some abbreviations, possibly chemical names, and IDs with capital numbers and digits and dashes and the like), mark punctuation as such, and attempt to deal with quotes in a sane way, including distinguishing between quotes and apostrophes. The result would be an arrayref of words and arrayrefs - same as the other Decl tokenizers, and should work fine not only for English, but for the other European languages as well. I don't yet know enough about non-European languages to know how well I can tokenize them, except for the fact that I know it's non-trivial to tokenize Chinese and Japanese, and presumably Korean.

Finally, searching CPAN for NLP, I found an interface to the Stanford parser (a link on how to use it in Java as part of this book).

It's looking like I'll end up with classes in Xlat for basic translation tool functionality (Xlat::Wordlist and Xlat::Speller), along with probably NLP::Tokenizer if I end up making that a HOP-based pure Perl endeavor.

Update: But as always, note that using a deadline for a job (in this case an editing job) to induce urgency for basic research is generally going to lead to sleep deprivation and depression. I've switched to manually doing this job like a schlub, but hopefully the remembered urgency will get me through the next development cycle on this. I gained some good insights.

Cinder: C++ for creative work

Cinder: Kind of like Processing, but C++. The Medusae project is breathtaking.

A quick note on dates and times

The true depths of date and time calculations are frightening; see Perl's DateTime. So it might be nice for Decl to incorporate that from the get-go, as part of the kitchen-sink philosophy.

XDoclet: "attribute-oriented" programming in Java

Interesting approach to boilerplate generation in Java (of which there is a great deal!) - XDoclet apparently allows you to specify some semantic attributes in comments and then generates all the cruft for you. That's kind of neat.

The philosophy of artificial intelligence

The history of AI (to about the mid-80's) from the point of view of a philosopher. I need to reread this a couple of times.

Decl hits CPAN

For reals this time! I still don't have rights to the Decl namespace (which is why CPAN took its own sweet time indexing the module) so it appears with a honking bold red "** UNAUTHORIZED RELEASE **", but it's up there under my name, which makes me pretty damned happy.

Also nice: except for one test result that I've already fixed for next release, it passes its smoke tests on every system. I love CPAN.

So what's coming up for Decl?

• Traversal: this is hierarchical structure walking (e.g. directory walk) and mapping (e.g. something like XSLT)
• Boilerplate and macros in modules, then release of declarative CSS and HTML modules
• Rewrite Word using some macros (the "select" tag usage is changing) and rerelease it
• Look again at Wx now that macros work, maybe release Wx 0.01
• Look at macros in the PDF context, probably release PDF 0.01
• Database management and access, then release Decl 0.12 with that
• An error management system, finally, which will probably be Decl 0.13
• Literate programming and PHP katas and examples, then release Publisher
• Probably look at Inline next and integrate with Python; I want access to the NLTK.
• Declarative logic somewhere in here, based on AI:Prolog.

After that, I'm not sure. But it will surely be obvious by then - and I'm equally sure this list won't survive contact with the enemy, either. For example, maybe I'll start thinking more about the Lexicon for real by the time I'm halfway through that list.

Two years in

I started this blog on November 5, 2009, with every intention of investigating a specifically semantic framework for programming that might have borne fruit by 2011. It's November 6, 2011, so where do we stand?

I started work on Decl in February of 2010, according to my notes (the first SourceForge checkin was on February 15, but I'd posted on Wx::DefinedUI on the 10th, and honestly I think a Markov-chained snippet from my earlier writing may have triggered the concept in January), and it quickly grew to take over my every waking thought. Essentially, all my progress with semantic programming has been in the implementation of Decl. As I noted on February 10th, my earlier effort in late 2009 foundered on the shoals of syntax. At least that's no longer a problem.

The idea of Decl is to define semantic domains and tags that declare various types of programming construct, then to build programs of those. Eventually, the semantic domains should have enough macro machinery involved that the programs will largely self-construct, but I'm nowhere near that level of detail yet. I just finished the v1.0 macro system last month, after all, and it's by no means clear how to get from point A to point B.

But that's where things stand. I have done some musing about shoehorning my old Hofstadter microdomain work into Decl - not that that would require much shoehorning at all, which is the raison d'etre of Decl in the first place - but haven't really made a serious move in that direction yet.

I'll leave you with this notion: the Decl tag is an instance of a concept. As such, it's a token from a Lexicon. I haven't implemented the actual Lexicon yet - but at least Decl will be a language capable of expressing it right from the start. And that's why Decl is important.

Hmm...

Ping.

So there are others out there!

Javascript pitfall: missing var

A heartrending account of mistaken globality. Killer comment from HNN: jshint, stupid.

So.... Code quality tools in general. I want to build a framework. Gauntlet thrown.

Mulberry: app boilerplate generator

Another boilerplate generator for Web apps.

Underrated Features of PostgreSQL

Another good survey.

Statistical comparison of programming languages

This is a pretty fascinating project - they have lots of different implementations of various algorithms in lots of different programming languages. The article does some comparison between them in terms of expressiveness and speed.

ML at Khan Academy

Very nice article on the application of linear regression.

Clojure DML for SQL

Interesting - this is the conceptual-level kind of thing that fascinates me.

Another CPAN for PHP

Composer and Packagist.

Overview of numerical analysis software

So looking at alternatives to Octave, it turns out - to what should not have been my surprise - that there are a boatload of alternatives:

• Wikipedia has a nice table
• The Octave Wiki recommends Inline::Octave, which I find a little questionable, but hey.
• PDL is probably the best Perl alternative; has direct support for sparse matrices, interestingly.
• The Monks look at some comparisons between R/S, Octave, and PDL.

Again: I'd essentially like to distill the semantics out of this and have a system that knows how to code for a set of alternatives.

Good maxims for consulting programming

Five things to do for programming on a deadline:

• Set up continuous deployment before you start
• Write tests first
• Be transparent
• Maintain daily todo lists
• Do the right thing

Not bad.

Automated freaking writing in the news again

This makes me so envious I could explode. I know it's always the same story. Still.

An aside on machine learning, and open-source contribution

So having forced my brain to code a vectorized cost function in Octave starting from the equation - a task that truly taxed skills that had lain dusty for decades, and involved a brief discussion with my private theoretical physicist - I've started to think maybe I might be capable of learning a new trick or two. This Stanford class just barely scratches the surface, of course, and my mathematical background is essentially nil, so I've got a steep hill to climb.

But. There are open-source machine learning projects out there. Perhaps it might be best to start contributing. So on that note: the mloss.org project database. 334 projects and counting.

And one of the things that caught my eye this week on the software development front is PVS-Studio, a static C/C++ code analyzer that finds common coding errors. There was briefly an article on it listing 91 such errors, but it was deleted. Of course, it would be a hell of a lot more interesting to have an open-source equivalent. If there isn't one, I intend to damn well start one, with a curated set of flags (this may be why the article disappeared, of course...).

Update after reading this: OK, so I'm an idiot. Sometimes it's easy to forget the last twenty years and the Internet and all. [also]

Anyway, the whole concept of static code analysis fits well with my vague idea of a "code understander" set loose on open-source code.