007, a simple macro language

This is very cool. 007 is a macro-enabled language that is written on top of Perl 6. I really kind of need to look at Perl 6 sooner or later.

LISP macros on JavaScript ASTs

Oh, another project that's all kinds of cool: eslisp is an S-syntax for JavaScript parse trees, with a macro expander that treats any JavaScript function as a perfectly valid macro. That is just brilliant!

Spidermonkey exposes the AST as an API, incidentally. This project makes use of that.

Macropy

Macropy does real live syntactic macros in Python using some manner of magickal tomfoolery.

JDC: keeping languages straight

The ever-thoughtful John Cook opines on keeping similar languages straight, and provides links to some multi-lingual cheat sheets he uses.  Filing under patterns again, although it's not really what other people mean when they use that word.

TXL: Tree Transformation Language

I essentially think of Decl macros or maps or whatever as falling into two categories: first is the text expression of a template, but the other works on tree structures.

Googling on "tree transformation" is a useful exercise, and five minutes perusal finds me two good links: TXL (the tree transformation language) is precisely what I want to be able to do (with a rule-based approach), and then there's an interesting paper ("A Language for Bidirectional Tree Transformations") by the Unison guys at UPenn about reversible lenses that act between tree structures, which is of course my map concept. (I should probably read everything they're doing.)

So those need to be read carefully. I'm sure there's more.

The difficulty of best-practices code patterns

OK, so now that I'm thinking more carefully about macros (in the context of publishing Javascript for Web apps), here's why they're important: because it's too damn hard to understand code when it's written to work right in all the fiddly-bit situations.

Case in point, this article about asynchronous code patterns in node.js. Go ahead, read it.

Now here's the thing. If you read that code, the first version, the synchronous version, is comprehensible. The final version, an asynchronous version that actually works and handles exceptions rationally - well, you can understand it given the presentation, but I defy you to run across that in the wild and tell me what it does, clearly, without the need for fifteen minutes of error-prone thought.

Macros help us get around that. If we think of that final pattern as the compiled version of whatever concept we're trying to express, then it's a lot more palatable. And that's why a universal macro language like Declarative is going to be really useful for a lot of different things.

"Base programs"

Grist for the mill of "types of program" here on HNN, ever the most interesting forum out there.

The list:

basic form
form with multi-valued lines
form with data set (multi-valued parent/children)
form with grid
tabbed form
form with skins
batch (loop thru something)
batch update selected records in a table
batch dump table records --> .txt or .xml file
batch .txt or .xml file --> update table records
batch file --> create & populate database table
cut an email
traverse internal tree function
traverse file index(es)
build html from parameters
build javascript from parameters
build .pdf from parameters
build hp esc sequences from parameters
benchmark a process
how does that syntax work?
which way is faster?
batch string parser
batch source code search
batch source code changer
batch parse source code, identify routine for reuse

A follow-up comment mentions that these should be in a library - but I beg to differ. This is a higher level of organization than the library.  This is where we talk about the semantic purpose of the program.  It fits well with the Unix philosophy of many small utilities, actually.

Food for thought!

Parameterized templates

How about this?

define formatting my-snippet "text"

   parameters (bold)

   nodes

      text "$text"

      text (italic) "$text"

      text "$text"

This would define a parameterized snippet that types its input text three times, in bold, with the middle also italicized. You'd invoke it like this:

document

   para "Some initial text"

   para

      my-snippet "Repeated text here"

By default, parameters would be passed through to the defined object.

You could define-and-implement in place with this:

document

   var text "Here is my variable"

   para "Some initial text"

   <= formatting (bold)

      nodes

         text "$text"

Here, we don't need a separate "parameters" tag because we're just going to use the variables in our event context at runtime.

The reason there's a "nodes" tag in this is that I might want to include other tags as well:

define formatting my-snippet "text"

   parameters (bold)

   do {

      # Set some things up

   }

   nodes

      text ...

We'd want a whole range of tags to denote the parts of a full node: parameters, options, label, parser, code, body, and nodes.

Note that instantiation of a named macro happens at build time, while instantiation of an anonymous macro happens at runtime. To run a named macro at runtime, we'd want to do:

express my-snippet "text here"

I should implement this stuff now, then see whether it handles everything I want. Add some control structures and it could be just about as powerful as you could want.

Further necessaries for Win32::Word::Declarative

So I did the standalone use-WWD thing and I'm polishing up an initial tutorial for the module prior to releasing 0.01 onto CPAN, and there are two things, at this point, that make the module less than perfectly usable in its current form. (This ignores the fact that it covers about 2.7% of Word's functionality; that's just incremental stuff that can be filled in at my leisure.)

First, it's hard to use it from plain Perl. I have a good plan for this: instead of indented strings, accept an arrayref format based more or less on the output from the indentation parser. This allows us to generate nodal structure really easily without worrying about having to format it with indented text.

But the worse thing is that the C::Decl framework still basically supports mere declaration. That is, I can't really specify a mutable data structure that is based on instance data. And that's a severe limitation - which is mildly surprising. A Wx user interface doesn't really need a lot of runtime mutability, but a Word document is an output format. Its natural functionality is to present runtime data, making the lack more glaring.

So really, a high priority for C::Decl has to be mutable structure. Macros. Templates. Whatever the mechanism or mechanisms are called, the use case is this: a script that (1) gathers some information somewhere, then (2) generates a Word document based on that information. We can't really do that right now without dropping into Perl or using Perl to generate the Word-generation script and then running it separately, and that just isn't where I want to be.

Also, a small tweak: right now, C::Decl only gives runtime love to the topmost semantic domain. Instead, I'm pretty sure that it should just scan its children in order and attempt to execute each one. There could be a [norun] option to suppress this explicitly, if necessary. And of course in the case of Wx, execution just won't return after you hit the base frame/dialog - but we don't really care much about that.

But this is necessary in order to gather information in the use case above - and in general, it will be a normal state of affairs to set things up, then instantiate something. The alternative is to allow the macro itself to run code at build time, and that's OK, but in terms of presentation it will often be clearer to do things in two phases that are visually distinct.

Thoughts for the evening

So I tried "adapting" some of the examples for Flash into a pseudocode form, and as usual, everything I do points out the holes in my plan. Then I go walk the dog, and possible solutions occur. Today's harvest:

The sokoban example from the gazbming is rather oddly written; he defines some functions to create tiles and sprites, then creates 200 wall tiles, a bunch of package tiles, and so on. Then to build a map, he moves the tiles in Actionscript (and the maps are encoded in the Actionscript).

So fine. I can imagine writing a macro that expresses the map encodings as Actionscript, but really those 200 wall tiles are a map. I don't want to create them with Perl (though I could), because normally I'm not running Perl at define time, only building code snippets and running them after start. Maybe I could rethink that, but in this case, it seems more elegant to express the 200 wall tiles as a map from a sequence (from=1, to=200) through a template into a series of macro-inserted structures.

Now, this begs the question as to what, really, a map is. I've been going on the presumption that it's a nodal structure thing - but in this case, conceptually, I'm mapping from a sequence onto a series of nodes. I can imagine the same thing with a set of row data. Now, in the case of a sequence (which is immutable) this is a one-way trip. If I map to data, it's ... maybe not a one-way trip, but I probably have to define explicitly how the data should be updated if the nodal structure is changed, and certainly something like this is probably one-way.

But the point is, semantically, this is a map. Just because it's not my two-way nodal structure doesn't mean I shouldn't still call it a map.

Oy. You want to do something simple, define a new programming paradigm and a language to go with it, and revolutionize the world, and then it's just one thing and another.

A thought on code organization

Well, it's a pretty thin thought. Mostly this: These Actionscript examples I'm looking at tonight instead of doing the paying work are a freaking mess. They build stuff, and move stuff, and do all sorts of things in messy imperative ways. Now, obviously I can't write Declarative for Actionscript - but I could write a sort of front-end that would compile to Actionscript.

But if I do that, then, then, well then I've written a front-end that could compile to anything. A sort of Code::Declarative, if you will. And I'm not at all sure I truly understand what that would look like, although to a certain extent it's obviously what this entire venture is fumbling towards.

So maybe I should spend a little thought on just what it means to compile to a text file. I mean, structured text is not such an outré concept.

It's really a macro system again. I wish I could implement just one version of a macro system so I could see how many ways it fails.

Another stab at "invoice"

I now dislike my earlier text-based notion of macros. The macro system should be native, and more importantly, needs to be at a semantic level. That is, we are describing to Class::Declarative what sort of node it should be building.

With this in mind, here's my current notion:

unit invoice
   has customer => customer
   has data items (description, price, unit, subtotal)
   assert usually count(items) > 0
   calc total = sum(price) from items
   has currency => currency default USD
   has comments

I need to find a more specifically macro-oriented structural definition and express it like this.

Mapping

Consider this. A macro expression is a one-way street; given a macro instance, the macro engine expresses it as more ramified code, possibly affected by the environment, or parameters.

A more powerful macro engine could define a mapping between two structures, whereby a change in either could be reflected as a change in the other. For example, a mapping between the graphical boxes on a diagram and the underlying database could be set up that could modify the screen when the database was changed, and write (the significant) changes to the database when the screen image was manipulated.

This would be the equivalent of a mapping in cognitive science, the syntactic/semantic mapping we use to talk about how language expresses concepts. Or the same as the analogies Douglas Hofstadter uses to talk about ... everything.

A mapping in a declarative tree would look superficially a lot like an XSLT program. The only difference is that a mapping would remember where its results were, and could act as the trigger described above.

Another example: we've already got magic variables that look to a GUI field, for example, for reading and writing. An explicit mapping structure could define that in the language instead of just in the code; we could define this sort of mapping in our programs. This could be used to set up Excel-like functions in a spreadsheet. Or not in a spreadsheet if you don't like grids - just set up a dataflow program. (TODO: think about that a little more.)

So the to-do list for macros is:

• !tag defines a one-way map to a virtual tag. It expresses in place during built, and stops.
• !!tag defines an active one-way map; it expresses in place, but rebuilds whenever one of its parameters is changed (or potentially does; we could probably specify some active and some one-time parameters somehow).
• An explicit macro or template tag could also match specific parts of the tree, specify assertions about its structure that must be met, and coordinate expression in multiple points.
• A mapping would define that in two directions in some as-yet-unspecified manner.

The basic ones are going to be necessary to make PDF::Declarative make any sense for real-world use, because they allow us to specify a PDF structure that is parameterizable, say, with variable text. Then we can use PDF::Declarative to build a template that can be used again and again (e.g. as an invoice template, etc.) Do that, and my test case will be complete: I want to use PDF::Declarative to generate my translation invoices.