Link dump: metaclasses in Perl

Since Class::Declarative is technically in the namespace for metaclass programming, at some point I want to be able to define classes in it. Well, and the fact that rendering of PDF content in CAM::PDF requires naming a class to do the rendering instead of proving an object - to use that system, I'm going to have to define a class.

There are other places I need to define classes; I'd like to subclass widgets in Wx, for example. All in all, it's pretty important stuff. And actually not too hard, like anything in Perl.

Class::Struct is the basic template. Essentially, you just need to generate code and run it in your caller's namespace. Here's a good article on it.

Groups of files

Speaking of "unmunge": one logical next step will be to codify the notion of taking a group of files and doing an action on each. To do that, we're going to have to start talking about groups of files and other dynamic structure. How that's all going to play out is unclear.

Mission starting to be accomplished

The whole point of the Class::Declarative effort, to me, is to enable me to write quick scripts in my fields of interest that don't slow me down enough with whatever I'm doing. In other words, when working with Word documents, the idea is to offload all the knowledge, all the lookup work inherent in dealing with OLE (and there's a lot) so I can actually conceive of the need for a script, start it, test it, and use it, within, say, an hour or so. If I go beyond that time, I might as well not have a scripting language - I need to finish the task, not spend a day crafting a way to automate it.

Last night, Win32::Word::Declarative managed to meet that need to a certain extent. Just the ability to remember that I can simply type

use Win32::Word::Declarative;

document (active)

  do {

    ...

  }

is enough to get me over some humps.

Not only that, the declarative framework gives me the start of a way to store knowledge accumulated. My task last night was to iterate over all the stories in a document - not the first time this has come up, of course. And this task is needlessly complicated, but now, for the first time, I have a way to package it up into a function that can easily be called. Now, I can simply say "foreach my $range in (^story_ranges())" and I'm done. It's still not as easy as it could be, but I no longer have to worry about that wacky double loop.

I'm sure I'm not the first programmer ever to be in the position of realizing that his accumulated codebase made things easier for him - it's not even the first time for me (the XMLAPI sure did make working with strings and structures in C much easier) - but it's immensely satisfying knowing that this work is starting to pay off in real terms.

Very rough roadmap

Having just committed the basic database tests, I thought now might be a good time to look up and get my bearings on the immediate future of Class::Declarative.

There are several additions to basic functionality that will go into the next release: these are the "system" tag (configuration), the "application" tag (also configuration), the shell and command line features, and state machines.

A word on the application tag first. Essentially, the definition of an application defines both the basic structure of a given set of programs as well as establishing a context for actions. So the application will also be where we define actions - and actions will inevitably result in the definition of workflow, because workflow is really all about the definition of actions (and who does them, planning for them, and so on). A full workflow engine won't be in the next release, but I think it's important to realize that it's probably going to be in the core functionality, and probably pretty soon.

An application can (1) be a refinement or a modification of another application, and (2) can define any structure at all for the scripts that belong to it. Whatever is defined in the application import structure will end up as macroinserted structure in the running script. This can even go so far as there not being a running script at all - in this case, the file run will be a data file, and the application will have to define how it's handled (most likely as a file tag - still to be written, obviously).

(An aside: it might be useful to go ahead and include filesystem semantics in the next release, but I'm not wedded to that.)

An application will probably be declared something like "application [name]", possibly with some other decoration. Class::Declarative will first try to treat the name as a class [name].pm, then maybe something like App::Declarative::[name].pm, and then whether or not that fails, will look to [name].conf in the current directory, and parent directories up the tree.

The application could also be given on the command line, of course. And it occurs to me that an "include" mechanism would be essentially the same thing, so we'll toss that in as well.

All this gives us multiple ways to define configuration all in one blow. The key is that all the structure found in this way goes into the same bucket as macroinserted tags. (It might be useful to keep track of where structure comes from in this way - and eventually even precompile structure while checking for changes to files each time.)

The state machine mechanism is really only destined for this next release because it's already started (kind of). It's cleaner. But definitely, applications, configuration, command-line invocation, a shell, and probably filesystem access all need to be in this next release, getting us closer to being a mature language.

All that is probably enough to keep me out of trouble for a month or two.

Update 11/19/11: Jeez, I didn't end up doing any of that. Except the shell, kind of.

Milestone of a sort: database access

Just wanted to mark the date that I've got rudimentary database access going and build-time database access to modify built content. Result: I can define a Word document template/script that includes a table defined by a query result.

Kind of neat, and it just makes me think of other things I can include.

Latest idea: generalize the "text" tag as a kind of output. When it fires, it asks its parent how to output its contents, and that propagates up the tree, allowing e.g. a document to define how text gets output, but permitting general output for non-Word areas. Add an "output" tag for alternative output handling, and you've got a pretty neat way to handle text.

Then you give the generalized text tag the ability to do Wiki formatting (for which it would also have to ask its parent how to represent different formatting choices) and you've got a really powerful mechanism. I'm pretty sure it would be most of a templating facility, right there. I need to implement it and see what it can do.

Invoicing again

Here's a little stream-of-consciousness post for ya.

Now that W32::W::Decl is working to create Word files, I need an overall process in which to embed it. What will that look like?

First, it's going to be driven by my invoking something. This "something" is effectively saying, "We are going to run invoices now." And that will involve hitting my prepared ready-to-invoice database in Access, retrieving a query from it, and taking action based on that.

Since everything in my translation business is customer-dependent (each agency has a different workflow for invoicing and, really, for everything else), my first query is simply going to be "which agencies have outstanding finished jobs that should be invoiced?" Then for each of those agencies, I'm going to switch to the agency context (represented in this case by that agency's working directory) and perform "invoicing" in that context.

OK. So here is already an interesting point, because it's workflow and thus I've thought about it for a decade already without satisfactory resolution. Running a script is equivalent to taking an action - a workflow action. But every action happens in a context.

An action and an event are necessarily separate, although in a sense they're the same kind of thing. An event takes place internally to the running program, whereas an action is a long-term, irreversible, action taken in the context of a larger system which includes that running program. An action is workflow.

Thus workflow is going to have to be a part of Class::Declarative, will it or nill it, because running a script is taking action is doing workflow, and in the larger sense we're going to suffer if we don't manage workflow properly. This has to be explicit - well, unless we leave it as a default or something.

So the context. For the time being, I'm going to consider a context to be purely a question of directories. I just can't get my head around much more abstraction than that, although ultimately it's going to end up being necessary, I'm sure.

The context modifies the "system" structure. System structures can be chained, too, allowing us a context hierarchy.

Oh - before I get too much farther here, I need to mention another sort of orthogonal dimension of contextuality, and that's the application. Again, C::Decl needs to take this into consideration right from the start: an application sets up a system context for a set of scripts, too, but stores this configuration in a central location instead of a location relative to the script being run.

The application can be set in one of two ways: first, I can just say "application " in my script. Second, I can specify it on the command line: perl --application= . This allows me to set up extension mappings under windows: .proj might be "perl -MWx::Declarative --application=project ".

The application can monkey with all kinds of things, like importing additional modules and determining the controlling domain. Some of this will be macro-inserted into the current script to be sure it all works out correctly, and that's essentially what the "application" does.

But - and this is key - so does the context. It's just that without an application, we don't actually know what the context is supposed to be named (unless we explicitly name it ourselves).

Oh, which brings me to the third way to invoke a script, which is to associate its extension with a script stored elsewhere...

Well. You can definitely tell this is stream of consciousness. You're lucky to get coherent sentences, even.

Recap:

• Invoke a script in situ
• Invoke a script as a "second language" by giving it -M and other args.
• Invoke a script as an "application language" by giving it -M and --application=
• Invoke a non-script file by assigning it to a script.

I've done the first two successfully, defining ".dpl" on my machine as a declarative Perl script. I have only now really formally acknowledged the last two. I'm not really thinking in terms of Unix yet, am I?

Onwards, to other work. Let this simmer.

Embedding declarative code into regular Perl

It's always when trying to code something that I have the best ideas. So far I've been considering embedding Perl into declarative structures as kind of a one-way street, but it would be useful to close the loop. The point where I realized this was in building a bit of code that could benefit from a state machine.

Well, my current thinking had been that the state machine would be declared out in the tree somewhere, and the Perl code would just call it. That's kind of rigid, though. Wouldn't it be nice to get something like this?

my @tables = $^content->find('table');
my @rows = $tables[1]->find('tr');

declare state-machine process (start=off)
  off {
    => on if $^next->as_HTML =~ something;
  }
  on {
    something
    => off;
  }

$^process->iterate(@rows);

Isn't that neat? And put it together with a "perl mode" that effectively treats the entire file as a do { ... }, and you have a true declarative framework that would be useful without necessarily being in control.

A thought on order of execution

So I've been thinking about the logic way to handle what, of the top-level items in a program, should actually get run. In my thinking, there are three different types of top-level item. First is just "do" and its friends: pure actions. They always run when at the top level. Second is purely declarative stuff. This never runs anywhere, because it doesn't have a go function.

The third class is both fish and fowl. It will run if necessary, but would rather not - these are items that are really more declarative than anything but that still have a default action, such as documents, URLs, GUI definitions, and so on.

The overall regime that makes sense to me is: 1. Run any preliminary code. 2. Skip any ambiguous code. 3. If there is code after the ambiguous items, don't run those items. 4. If the last item in a program is ambiguous, run it.

This gives us the option of setting things up for an ambiguous item that then controls the program (e.g. prints its document or activates its GUI or whatever). But if there is any action below an ambiguous item, it will be considered a declaration, not an action. I think this really captures what makes most sense to me.

Actually, there are places this should hold even within other items. For example, I'm putting a state machine in a URL. Should the state machine run as the code for that URL or act as a function definition for use later? Good question....

A more useful way to include individual declarative classes

It would be nice to be able to say "use Win32::Word::Declarative;" in a conventional Perl program and have it go ahead and set up the Class::Declarative environment. Makes testing setups easier, too.

Hmm...

Getopt::Lucid and Term::Shell

Perl brass tacks: I'm basing my command-line handling on Getopt::Lucid. And in the end, I think I'm going to just end up writing my own Term::Shell replacement that pretty much works like that. The events in an event context can be seen as commands, so just invoking some "shell" conversation against an arbitrary event context will be pretty cool. But Term::Shell is just a little too hardwired to make that entirely useful. So we'll see. Either way, both modules have been open tabs on my browser for a couple of weeks now and it's time to clean up and leave a little nonvolatile state.

Update after looking more closely at the Term::Shell source - it would be folly not to subclass Term::Shell. There's a lot of really cool stuff in there. I just have to have an automagical way of setting up the commands - which is easy - and I'm good to go. This will be very cool!

From looking at the code, it looks as though Term::Shell can run under Tk somehow, but Google isn't being very forthcoming about that. It would be nice to be able to use Term::Shell in a Wx context. Very nice, actually.

Perl: mucking with the symbol table

Suppose you want to be able to build a function on the fly (as a closure) then call it like any other subroutine. Your answer is to modify the symbol table, to wit:

$sub1 = sub {
 print "Hi!\n";
};

$clo2 = sub {
 print "OK?\n";
 local *sub1 = $sub1;
 sub1();
};

$clo2->();


sub1();

Here, the call to $clo2 will print "OK?", then "Hi!", then the second attempt to call sub1() will fail because it's not defined (we localized the typeglob within $clo2).

This is how I've implemented subroutines in Semantics::Code.

Milestone: line parser works!

Just to mark the event.

PDF::Declarative

Man, I started PDF::Declarative two days ago, and nearly have something that can do the PDFs I want. Turns out text layout is hard - who knew? But building declarative wrappers for new domains is relatively easy, given good example code.

So I'm really quite happy about the declarative framework; it's making me a faster, better, more productive programmer, and that's all I really need it to do.

An interesting semantic domain at some point would be the generation of new semantic domains... Speeding up the process of speeding up the process should, um, speed up the process.

Wx::Declarative back where I was

Well, I've successfully factored the Class::Declarative out of Wx::DefinedUI (which is dead, long may it live) and I'm back to having all the stuff work that I had working last week. (!)

Going forward should be way easier now, and the hard stuff is factored out and unit tested so I can't break it as easily, but I feel oddly as though nothing had happened for a week. Except, of course, Class::Declarative is on CPAN now, which is very cool indeed.

So this is what works: code/closures, events (those are the core semantics), buttons, menus, message boxes (hardcoded text only), frames, sizers, dialogs. Next up is the status bar. Then I set out into serious wilderness by subclassing a window for the frame's client. Yikes!

Parse::Indented on CPAN

Bit by bit, I'm establishing a useful toolset here.

I've been spending too much time today in organizing the unit tests for Class::Declarative. Test-driving programming has a higher initial threshold, but once you get the framework all set up, it really starts to take shape.

Anyway, Class::Declarative is parsing properly now; you'd think this would already work, but I spent some time generalizing the semantic framework to make it easy to build new semantic domains. I think this is really going to pay off in the medium term.

Class::Declarative

I'm splitting the declarative-programming part out of Wx::DefinedUI. There are just so many areas I can think of where the declarative paradigm will be so useful!

Data declaration (tables and so forth). Testing! Other messaging frameworks like POE or Moose. And, yeah, semantic programming.

I've literally been thinking about this in one form or another for over twenty years. This is the first time I can see an actual way to get it implemented, dammit, so I can use it.

Anyway, refactoring the code is cleaning it up a lot. I always get the feeling if I just refactored enough everything would be a one-liner.