Couponing automation

Probably stupid, but Jeffrey believed that with couponing, he could live on $1 a day for food and have plenty to eat (via MeFi). He blogged it for a month, spending less than $30 and eating rather well - while donating food to a local food bank. Then he kept doing it.

This intrigues me. A lot of the coupon game involves planning and "deal detection" that might be amenable to automation. So ... maybe it's a good target?

If so: SavingAdvice.com coupon database. Jeffrey's FAQ. An explanation of blinkies. There are lots of fora, of course, where heuristics could be gleaned.

Worth thinking about.

MetaOptimize Q&A site

A stack-overflow clone for statistical methods, machine learning, etc.

A great domain for semantic programming.

Javascript local data store

Another interesting component in Javascript. Why, yes, I am in link dump mode, thank you.

Understanding systems

A blog post musing on understanding systems and how it makes us better programmers. This, again, is kind of where I want to be going with a semantic software system.

Intrusion detection

This blog post on intrusion detection probably has no place here, except that if semantic programming can't also include semantically motivated machine learning paradigms like intrusion detection, then I'm probably not thinking about the right thing.

Well. I find it interesting, so it gets bookmarked here.

UTF-8 files and Perl

I ran into some problems trying to load some files with UTF-8 text (German) using Perl. Thing is, these files had a three-byte byte order marker (BOM) of ef bb bf [here is another useful link] and Perl freaks out. You have to check the first line for those three bytes; if present, you toss them and keep a flag. Then for the rest of the file, you have to set the UTF-8 flag on each line read.

The code:

use Encode;

open F, "$d/$file";
$utf8_file = 0;
$firstline = scalar <F>;
if ($firstline) {
  if ($firstline =~ /^\xef\xbb\xbf/) {
     $firstline =~ s/^\xef\xbb\xbf//g;
     $utf8_file = 1;
     Encode::_utf8_on($firstline);
  }
  [ consume $firstline ];
}
while (<F>) {
  Encode::_utf8_on($_) if $utf8_file;
  [ consume $_ ];
}

This is tedious.

I see two approaches to dealing with this. The first is to create Yet Another Module (this includes adding code to Class::Declarative), then always use that module when coding. This is kind of the default, and ultimately it is unsatisfying.

The other approach is some kind of pattern / macro / template system that would include this knowledge and would somehow generate the appropriate code as needed. That's where semantic programming needs to be headed.

Boy, that's vague.

Web app ideas

A list of Web app ideas. Can't have too many. One or two of these look interesting.

Open-source Rails apps

Some example Rails apps. Seems as though a generic semantic platform could establish parallels between different frameworks, and kind of ... port ideas or features back and forth. Or something.

HNN finds another interesting thing

Fabrik, an Apple visual programming toolkit from 1988, has lots of very intriguing features - including bidirectional mapping between objects, and a pervasive use of the dataflow paradigm. (!)

Another thought on levels of abstraction

Maintaining two views of the same structure (i.e. two levels of abstraction) is what I referred to earlier as a dynamic mapping. It's definitely something that has to end up in C::D.

Dataflow programming: Cascading

Cascading is a Java-based dataflow API for Hadoop. Since dataflow is one of the key declarative domains, as I see it, I want to do something more or less like Cascading's feature set.

Levels of abstraction

Here's a concept that crystallized for me today: a programming system that could present code at different levels of abstraction at the same time. Well - clearly not at the same time, but ... a system that had something like a semantic organization of the code could present it at a higher level for a summary view, then get more specific as needed.

This is again something like literate programming, but closer to the code. It would identify larger structures within the source code as having a meaningful relationship with one other, without obscuring the specific source code.

A Lisp macro system is great at producing levels of abstraction, and in many cases this is exactly what's needed; I don't care where the compiler puts variables, so I don't need that low level. But when approaching a large software system, I need to understand what parts of it correspond to specific features, and only look at the features I'm working with at the moment.

Or something. I need to refine this further. But here's kind of where I want to go with this. I've got a venture going with wxPerl, and so I'm developing it in Wx::Declarative (at least as far as I can). I'd like to be able to have a summary description of the overall application, while still having full access to the lower-level specifications of the different panels and so on. I'm not sure I want to put things into a different file, because it's all so compact that I like the idea of having it all in one place (with "business logic" put into a separate module, but with the presentation all being in one place).

Semantically, it would be nice to be able to integrate this summary structure into the structure of the code. I'm really not sure what that would look like, though.

WWW::Mechanize

A good target for declarative coverage.

Data::Match

This might be a decent place to start with some kind of declarative pattern-matching ... thing.

Lisp in Perl

Makes you wonder why you couldn't just write a Lisp that would "compile" to Perl. It would be instructive and fun.

Edit: it's been done in Ruby.

Edit 2: a serendipitous link to the same in Python. Freaky.

What makes Lisp great?

Of all the languages out there, Lisp is the one that seems to have the most proponents who sound believable. Occasionally I think I should learn some of it - then I bounce off what to me are always the stopping points: I have a hard time with the radically different vocabulary, I don't like capital letters, I miss CPAN like childhood innocence, and all this - pathetically - is enough to stop me.

So I decided to check what other people think makes Lisp great, and put that into Class::Declarative if possible. But when you get down to it, Perl already does a lot of what makes Lisp great, it turns out. (The closure epiphany I had in January is what got me started on this path in the first place, after all.)

Paul Graham lists nine new ideas that Lisp embodied: conditionals, first-class functions, recursion, dynamic variables, garbage collection, programs as expressions (i.e. functional programming), a symbol type, a code notation that is a tree of symbols, and the whole language available at all times. Of those, the first three or four are now universal, and the first five unambiguously part of Perl. The sixth is mostly covered by Perl and can be simulated with anonymous subroutines in cases where it's not covered.

That leaves the symbol type, a code notation that is accessible to the program, and the whole language available at all times. I'm not terribly interested in the symbol type, because it's a performance issue (testing for equality using the symbol handles instead of checking string contents); I'm really interested in what makes Lisp more expressive than other languages, at least for those versed in it.

Moving on to several sources, "Lisp is a programmable language", meaning that Lisp can easily write Lisp code in order to provide higher-level semantics for a given domain. I think this is getting closer to the crux of the matter. Due to Lisp's minimal syntax, its control structures are all functions, and so you can easily extend the language to suit your domain. That, plus Lisp's interactive nature - the way a Lisper effectively enters into a dialog with the language while evolving new semantics - make Lisp unique, or at least partly unique. (Here is another good presentation of this notion.)

Python has considerable interactivity, of course, and the introspection that Lispers find so useful. But I've never had much luck with that mode. I find it far more instructive to break things down into unit tests in the CPAN module paradigm, and work out semantics that way - although I can certainly see how an interactive data inspection facility would really speed the process in many cases (actually, it would make some things possible that aren't any other way.)

Perl is supposed to make easy things easy, and hard things possible. An interactive data facility would be a great addition to the language. Well - I suppose the Perl debugger already does this, to a certain extent, but the Perl debugger has always made my brain hurt.

The Lisp macro system and quasiquotation are powerful facilities making it easy to extend the structure of the language; a Lisp macro is a program that writes other programs that perform the ultimate tasks. Class::Declarative is most definitely moving in that direction; my macro system will be equivalent to Lisp's (I think) and nearly as terse as quasiquotation. More on that when I've got something working.

I'm left with the following chief advantages of Lisp: introspection at all levels, and interactivity allowing multiple approaches to new semantics. And I'm forced to admit that at the moment, Class::Declarative is not doing this - but could, if it grows more in the direction of semantic programming.

On paper, I've been exploring some possible approaches to realistic semantic programming; it appears that the key insight is recognition. That is, matching. If I posit a given structure for the world, then allow a matching engine to tell me how the world can be made to match that structure, then I've parsed the world, or recognized some structure in it. That's really what semantics is.

Here's an example. I want to write an invoice. I create an "invoice" structure. The Lexicon looks up what an invoice is, and fills in some structure as follows:

invoice
 customer {{index customer}}
 data items (description, price, unit, subtotal)
 {{.assert count(items) > 0}}
 total "{{select sum(price) from items}}"
 currency "{{index currency default USD}}"
 {{.section comments}}
 comments
    {{@}}
 {{.end}}

This is still pretty crude, but effectively we can scan the template to see whether all the information is available that we need for a full invoice. Some sort of interactive process will allow me to fill in what's missing, and some other specification not shown here will allow the invoice to be stored. That part's probably some sort of workflow environment. Once defined, the invoice can be expressed using a template to produce PDF or a Word file, and attached to an email. These actions are certainly workflow.

The point is that this is semantic programming - I define an invoice, the environment knows what an invoice is, and we interact to define the case. It's fuzzy in my mind, but I'm getting there.

Note that some of the fields in this example (the index fields) are expectations that prevent the object from being fully specified until they are fulfilled. The same applies to the assertion. However, the select field is simply a calculated field that doesn't require - in fact, doesn't support - an interactive assignment of value.

That calculated field gives us something like the capabilities of Excel, by the way. It would be instructive to be able to build a full spreadsheet in a system like this.

Why Perl?

Occasionally I'm beset by the worry that I've chosen Perl as the basis for all this development. Wouldn't some other language be a better choice?

All I can say in my defense is that my brain seems to work better with Perl for some reason. It's not that I've done a great deal of work in Perl - I've written more lines of C and Tcl. It's not that I have no exposure to other languages - I've worked in Python and Scheme. And while I enjoy the joke that Perl is a write-only language as well as the next programmer, still - when I look at my own code written in Perl, I can read it more easily than my own code, let alone that of others, in other languages.

Then there's CPAN. Oh, sure, half the stuff on CPAN is alpha-quality just-off-the-ground stuff, but oddly enough, it's usually enough to get me off the ground on a given project. Then, after I've got something halfway done, I can rewrite the CPAN modules that weren't what I wanted, and get the rest of the way.

At the end of the day, all programming is about semantics. You are taking the inchoate space of possible meaning, and narrowing it down to the meaning that gets you where you want to go. I find it easy to do that in Perl, for whatever reason.

Rebuild complete

I've been doing some kind of neat stuff with unit testing in the Wx::Declarative domain, although timing makes the tests kind of brittle - sometimes you're testing a condition before everything in the tested process has set things up, so you always have to remember to wait a little bit for applications to start, stop, etc. But Wx is working again! So there's that.

And I've been working a little more with it. It occurs to me that you could set up a runtime modification thing using the Wx facilities pretty easily - that and a command line and runtime object editor, and ... you got a lot of what makes Lisp so cool, but in Perl.

Food for thought.

Score one for unit testing

I'll tell you, having extensive unit tests written that the system already passed before this major overhaul makes me ... let's put it this way. It makes it possible to do refactoring like this. I would definitely have lost hope without my unit tests, and this project would be another dead one. Instead, I'm taking the old unit tests, dropping them into the directory, and letting the unit tests quickly and easily find the places in the code that need attention.

I've almost got everything working again. Incroyable!

Parsing works!

OK, so my parsing unit test passes 139 individual tests having to do with all aspects of tokenization, parsing, and building of nodes and nodal structure. I'm independent of Parse::RecDescent and Parse::Indented now, and I'm entirely using Class::Declarative::Parser objects to parse everything.

I still have a little work to adapt the existing classes to their new non-XML::xmlapi existence, but that seems to be going well. So in a couple of days, I guess I'll be starting templates, with a much more satisfactory parse-and-build regime going into the exercise.

Progress!