REBOL

REBOL (Relative Expression Based Object Language, an acronym that somehow evokes the TARDIS to me) seems to be a language that has a lot of the semantic aspirations that Decl does. Interesting!

Exegesis, literate programming, and Decl

The basic outline of Decl's new syntax parser is complete and passing tests, and as always when a milestone is reached and I look around the corner at what's next, I'm a little overwhelmed. It makes me philosophical.

My initial Marpa NLP article was actually a very simple exegetical analysis of a prototype script I wrote myself, and as usual when doing a first stab, I ended up writing a lot of special-case code and syntax to handle things, representing a technical debt that in such situations nearly always strangles its host. Fortunately, I iterated quickly this time, taking the insights from that and putting them into a new Decl-based plan.

Slowly, I'm feeling my way towards a Decl-based system for literate-style transformational exegesis, one that I hope will eventually encompass everything this blog has been about.

The advantage to basing everything on Decl is that parsing is done. I can now parse a very rich, informationally dense data structure that is designed right from the start to group things in more or less the same way natural language does. It's easily extended and configured, easily indexed - in short, it's a way of taking notes about program structure and using them to evolve a program.

So that's where I'm going. Very slowly.

Breadcrumb

Just a note on progress, for later calibration of my past timelines: I've got Decl 2.0 parsing working rather nicely (still a few loose ends) and I have an excellent plan for integrating it into a notes application for literate programming. I think the combination will end up being something pretty powerful.

My first real target for transformational exegesis using this kind of tool will be Melanie Mitchell's Copycat - I want to get it converted to Clojure so it will run on something I actually own. But I'm also exploring various programming sequences and contests as a way of provoking thought about the actual writing of software.

Anyway, that's all pretty jargon-laden but it means something to me. At a later date I hope to circle back around and write about this stuff in more detail, but I'm not in that phase of the cycle at the moment.

Language-oriented programming

Oddly, following the Wikipedia list of different programming paradigms to "language-oriented programming" led me right back to MPS, and a raft of fascinating articles by Martin Fowler about the notion of a "language workbench". [Here], [here], and [here].

Chief among the things that people seem iffy about in LOP is the idea that the stored representation is in fact no longer text (and what that does to version control) - but you know what? Decl is probably an ideal representation language for a DSL-oriented approach (one of the things I was struggling towards in the first iteration) and is entirely text-based.

So that begs further exploration.

Decl and top-heaviness

Man, reading through all the stuff the v0.11 Decl::Node object supports, it's really no wonder I bogged down. It was just doing too much. I really hope that splitting things out into syntactic and semantic poles will make a difference.  (Or really, even more than just the two, given the declarative extraction phase in the middle.)

So yeah, I suppose a post on that is in order.  The new regime is finally getting underway, given that the last update to Decl was in 2011 and it's 2014 now.  I've started coding Decl::Syntax, which is the handling of syntactic nodes.

Note that a syntactic node is used to derive two different sets of semantics. The first is the machine semantics, the second being the human semantics. This is equivalent to the concept of literate programming, except that literate programming also parses the code chunks for indexing, which (initially, and maybe permanently) we will not be doing.

So the surface structure is the indented stuff. To derive the machine semantics, we go through two more phases.  Actually, three.

The first is markup. During markup, a Markdown ruleset is used to convert all Markdown nodes into X-expressions. The ruleset can be specified in the input, or can be one of a few named ones.

After markup comes declarative extraction. Here, we extract a tree of declarative nodes from the syntactic structure. These contain only the "true children" of each tag. X-expressions are converted to tag structures during this phase, and transclusions are resolved. Annotations are inserted into structured parameter values.  Macros might be expressed, I won't know this until I try expressing some things with prototypes.

The result of declarative extraction is a thinner tag structure that contains only machine-meaningful information. Anything explanatory is discarded, although obviously it's still available for examination if there's a need.

After extraction comes semantic mapping. Here, a set of vocabularies map declarative structure onto data structures. A default vocabulary might just map everything into vanilla Perl structures or objects, but more interesting vocabularies will build more interesting objects.

Finally, execution does whatever action is encoded by the semantic structure. This runs code, builds documents, activates the GUI, or whatever.

Keeping these phases strictly separate makes it possible to build all that detailed functionality into this system without losing sight of what's where. Or so I fervently hope.

Analog literals for C++

Cute. This is kind of along the lines of what I'd like to do for layout in Decl.

Decl 2.0 syntax parser

I've been working on a lot of thoughts about the Decl reboot lately, including a ground-up rethinking of the basic way of handling syntax, and I've come to some conclusions.

Indentation is a misleading way of thinking about this. Indentation is just an indication of the two-dimensionality of text. Especially if we look at Markdown and its friends and relatives, we really have to realize that at least the block elements are there to exploit that two-dimensionality, to arrange information vertically as well as horizontally to present and shape information.

In fact, I'm getting a lot closer to just saying that Decl syntax and Markdown are sort of the same thing. And so I want to come up with a parsing language for two-dimensional text that is not a grammar built for one-dimensional sentences. Or at least is only partly a one-dimensional grammar.

Along the way, I hope to start looking at some naturally two-dimensional text items:

• Diagrams
• Musical scores
• Other timing diagrams
• Workflow charts, GANNT charts, etc.
• Page layouts and screen layouts for forms, buttons, etc.

And all that could be directly supported by at least part of the parser. Using indentation and block rules, we can do a "terraced scan", as it were, identifying blocks first and then drilling into them to identify more details.

The combination of Markdown with the Decl parser and interpreter, moreover, gives me a very natural way to implement literate programming tools in a way that finally makes sense to me.

I think this is going to be very fruitful.

Web frameworks and architectural description language

I've been toying with Mojolicious lately, gearing up to get over my fear of complexity and jump in with some simple UI for some little projects.  There's a great Flask megatutorial series I'd like to follow along with, reimplementing the various steps in Mojolicious. (See also this example blogging app in Mojolicious::Lite.)  But "porting" a tutorial begs trips the usual circuit breakers in my mind:

Namely, a Flask webapp and a Mojo webapp are both projections of the same abstract webapp, and what I really want is to describe that abstract webapp in a higher-level language.

I should really just no longer call it Decl.  I should just call it Hylaea right off the bat.  [pause]  Yeah, that name is pretty uncluttered!  There's a guy in Paris calls himself Hylaean (tweets some interesting stuff, not a lot of activity on Github and it shows he prefers the MS ecosystem - a guy worth following, actually) but that's about it outside the Anathem Wiki, and "Hylaea" without the 'n' is free as a bird.

Anyway, there is a semantic abstraction of the notion of the webapp that can be expressed in Mojolicious or in Flask or in, you know, any of the thirty thousand other webapp frameworks out there, am I right?  But the abstraction, at some level, is the same for all of them, it's just that by working in one or another stack or language you might be led in different directions in development due to the affordances of the environment (that's a cogsci term I find particularly apt here).

That stack/language/environment/ecosystem is what we call the architecture on which we're developing a given app - but the way we think about the structure of the app or program or whatever (because this obviously goes way beyond webapps) is some kind of shared semantic structure that I think is amenable to externalization.

Note, of course, that the specific description of a particular architecture or app is a kind of "skeletal form" - because it doesn't encode all the knowledge about a thing, it just identifies it and evokes the library of concepts stored in the (getting back to Hofstadterian research here) the Lexicon.

Meditation prompt: is there a difference between Decl and Hylaea?  I kind of think there is.  Decl is an interpreter in Perl for declarative structures, but remains a pretty Perly thing.  Logic is still represented in Perl, for example.  Hylaea might be the overall programming system or something. Let's consider this for a while.

Macros and washing machines

Well, here's a timely screed about macros and why they make sense - to which I can only say, "Yeah!"

Which brings me to discussion (again, and we'll keep discussing this until it comes out right!) of macros and code generation.

Let us imagine a system of articles and books that describe a codebase.  Some of the codebase may be maintained outside this system; some of it within, because the articles include some literate programming tools that can generate sections of code.  (This way the system can be used to start analyzing an existing codebase and slowly grow to encompass all of it, as needed.)

An article is equivalent to a book section, that is, a book consists of a hierarchical organization of multiple articles, presumably related.  An article may still have hierarchical structure within it, though, because sometimes you just need that for clarity.

In general, though, a single article addresses a single "thing".  That topic could be a feature or a specific function, or it could be a change request touching many different parts of the system.  Ideally the maintenance of a complex system would thus have a narrative made up of multiple articles explaining the thinking at each stage.

OK.  So in that context, let's assume that some of our literate programming-type tools include arbitrary macros that can be reused.  (Literate programming can be seen as writing a number of single-use macros, so generalization of that to reusable macros is no great leap.)  Some languages are easier to macro-ize than others, of course: we have to parse things to make truly effective use of macros without leaving the native syntax.  But by extending the native syntax with a template language, of course (as we do in literate programming, actually) we can build macros for any language.  The key is the code generation, you see.

It might be a good idea, though, if particularly questionable or novel macros were to be given a kind of "half-way existence", where the macro as well as its expansion are shown in the presentation.  Maintenance then has a template or macro to work with, but the full code is shown for clarity.  There are plenty of instances where that makes a lot of sense to me.

New approach to Decl

I came to Decl from the wrong direction, last time.

My original thought was to build wxWidgets stuff in a way that didn't kill the beauty in my soul, and it kinda worked - but there was a lot of work involved, so I got distracted by some other shiny stuff like PDFs and Word documents and went off doing it.  What I ended up with was a clunky interpreter instead of a semantic description language.

What I should have done instead was, well, a semantic description language.  I've been giving some thought to that lately, because spring means it's time to start new things, or bring old ones back to new life.

What occurs to me is:

• The basic parser into a data structure is a good thing.  Let's keep that.  Tags, good.  Parameters, good (maybe drop the distinction between options and parameters because I couldn't keep them straight anyway).  But the basic idea is good: a set of nested tags describing arbitrary structure that can then be mapped onto programming constructs, the tags representing nouns instead of the verbs we see in non-declarative programming.
• The semantics of tags need to be represented more explicitly.  I did a lot of dancing around trying to map tags onto concepts behind the scenes, and it was unmanageable.  Instead, I should explicitly state a mapping - as an appendix or footnote.
• Yeah, you heard me.  We've developed an entire structure of documentation over centuries that is used to address semantic complexities by using out-of-band channels to clarify ambiguity or add information that could be distracting to the flow of presentation.  Why should everything be linear in source code?  Because it's easier for the compiler to understand?  Balderdash.  This is really close to where Knuth is going with literate programming - but Knuth worked with compiled languages, and I don't.  I don't want to respect identifier uniqueness; my compiler should get from context what I mean.  Knuth worked at too low a level.  It's time to kick it up a notch.
• I should be able to use citations, too, to include semantic presentation that bears on a given solution topic.  Citations here are just ("just") libraries - or macros.  Boilerplate and templates.  APIs to external functionality.  CPAN modules.  Anything that has already been worked out to address a solution space, can be a cited reference.
• Decl, although written in Perl, should not be Perl-bound.  Ideally I should be able to use Decl to define a program in any language, or to define it in several languages at once.  I should be able to use the same framework to use NLTK or data science stuff in Python, describe machine learning algorithms in Octave or R, define modules for CPAN, spin out a Web app in JavaScript, or write low-level things in C or assembler. Decl should be an approach, not a chain.  The first time around, it was an interpreted language written in Perl.  That was wrong.  Decl should actually be a compiler - maybe a compiler on the fly, but still a compiler.

How's that for an Easter resolution?

Marpa tutorial

Ooh, Kegler wrote another Marpa example tutorial of a DSL.

Think what this will do when wrapped in Decl!  Maybe I should write the parser tutorial chapter early on, then double back to write more sensible things before it.

Cucumber, and BDD: Behavior-Driven Development

So here's something I ran across (again) today, and really started thinking about what it means: behavior-driven development, BDD.  It's a philosophical refinement of test-driven development, in which you define test scenarios in, essentially, English using some keywords (Feature:, Scenario:, etc.).  This keyworded set of requirements is then used to define tests, and then you do the TDD thing to make all the tests work.

It's pretty cool.  It's been an outgrowth of the Ruby Agile community, and its latest incarnation there is Cucumber.  And yes, there is a Test::BDD::Cucumber on CPAN.

So here we have a very declarative approach to specifying the behavior of a program and using that to drive the development process.  I don't care for the somewhat clunky regexp-driven way you get from the plain text description to the executable code, but the overall shape of the development process is very promising.

Here's the Wiki page for BDD.  It points out that some of the notions in BDD come from domain-driven design.  And all of that needs to be condensed a little into some principles for Decl, perhaps.

Decl is dead. Long live Decl.

I just pulled the plug on the old Decl at Github, replacing it with a tabula rasa generated by module-starter.  The goal at hand is simple: rethink everything from the ground up.  So first things first - our first program to implement is this:

text "Hello, World!"

To make that work, we need a few things:

• The interpreter environment itself (an object of type "Decl").  The environment was itself a node in the last iteration, and I suspect that's a mistake.  There should be a root node for ease of self-printing, but that should be a child of the environment, not the environment itself.
• A "decl" command right from the start that acts like an interpreter.  The environment should be a shell, I think, so we can interact with the environment.  It's not a Python shell - the Python shell builds the environment as the result of a series of verbs, and that's specifically what Decl doesn't want to do.
• Loading code into an environment has to be easier: (1) with a source filter, (2) passing a string in, (3) from a file, and (4) passing some intermediate data structure in - all those have to be supported more transparently than the last version.
• The output handling system has to be in place in at least a rudimentary fashion here.
• Marpa parser.  I'll probably need to refine the grammar as I go.

About that. In the old version, I had sigils to determine how the body of a node would be handled.  I'm eliminating that.  Instead, a trailing quote will mark a text body, and brackets a code body.  Anything else is vanilla nodes. Sigils are simply too hard to remember.

Similarly, the language a code body is in was marked with perl < { } or python < { }.  I doubt that's a good idea. It's clunky and ugly - and I want to be able to use Python with ease and elegance (as well as C).  I'm not yet sure what the solution is.  For multiline code bodies, I can see "{ (perl)" as an override of the default language - the default language being set sometime earlier.  For single-line code bodies, though, I don't see that as sufficiently elegant.

Another thing I've been thinking of.  When using a semantic domain module, I want a much, much more explicit definition of the tags in the domain.  To date this has really sucked.  If a module is used from within a Decl environment, then a lot more information should be provided right at the start.  If the same module is used from Perl, though, it should act like a Perl module - and load Decl itself.

This won't come up for a little while, of course - definition of domains is a little way down the rebooted road.

This might take a while.

Towards a Decl reboot

In a month, it will have been a year since my last active work on Decl.  At this point, I think it's time to reinvent that wheel again, and start clean; I can always steal the good stuff.

So ... what was Decl?  What should Decl be?  I have a few principles I want to adhere to, and maybe a bit of a plan of action as well.

1. Somebody else is doing better parsing than I am (Marpa), so first and foremost, eliminate my HOP-based parser code, which had performance problems anyway.

2. The internals of Decl are pure cruft.  Now that I know how some of this stuff works, I can do a better job.  I can also do a much better job in providing Decl support for definition of Decl, and better syntactic sugar (thanks to some pointers from Dave Rolsky).  So .. redo that stuff.

3. The use cases for Decl are too difficult.  It has to be easy to define a module based on Decl that I can call from regular Perl without worrying about it being Decl.  (I'm looking at you, Word module.)  Not that calling Decl as an independent language has to be avoided (I know how to define that better now, too), but ease of use has to take higher priority now that I know how I want to use it anyway.

4. I need to be able to define something in Decl and compile to non-Decl-dependent Perl data structures.  If possible.  At least I need to be thinking about that at every step.  Lightweight flexibility.

5. The definition of domains has to be more semantically motivated than it is right now.  All my example domains ended up being horrible jumbles of cruft, with helper functions defined haphazardly and without easy ways to self-discover.  About as non-semantic as I could have done it.

6. Similarly, the placement of all tags into a hard directory is a mistake.  Tag discovery has to work by looking at @INC, simple as that.

7. The basic concepts that I started building into Decl were valid.  Here's a list I've come up with right now; I want to refine it and organize it, and motivate things better.  The kitchen-sink approach is .. maybe too much.  On the other hand, the point of Decl is to provide a framework for representing the cognitive structures programmers actually use in thinking about programs, so ... does unification need to be in the core?  Maybe?

• Parser support
• Events - this should be expanded to a Lisp-like condition system for error handling
• Shell support (text commands) at every object level, including the main program
• Templates and filters and macros.  Oh, my.
• Iterators and lazy evaluation.
• Code munging and generation.  This should do more than I was doing, using templates, probably.
• Output handling up the tree, just started scratching the surface there
• Tabular data and DBI support in the box.
• Hierarchical data, the file system, and walking iterators.
• Text as a special thing, with rich text and so on.
• Handling of the system and command line 
• Control flow and workflow - handle the user in a sophisticated manner
• Logic programming, unification, and matching
• Explicitly represented state machines
• Gulp - explicitly represented data types.  When I want to.
• Assertions and explicitly represented behavioral tests?
• Bidirectional maps.
• If I can figure out how to do it, views of code at different levels of detail supported in the basic language.  Perhaps some system of annotation?

That's a heavy list.  Honestly, there's almost nothing I can take out of the core without feeling as though I'm missing something, and in fact there are some things I didn't put in the core the first time that obviously should have been there (error handling, I'm looking at you).  What I need to do is to write a principled primer of Decl, going down this list and writing example code for each and every thing.  Then make it all work.

Write the tutorial first.  Odd thought.  Here's the first chapter; it's the Hello, world for Decl:

text "Hello, world."

That's it.  By default, text's action is output, and by default, output goes to stdout.  From a semantic standpoint, how could this be any different?

File::Find

I've decided to hack out a little script to publish my Vivtek.com site to a directory amenable for Githubbing (a dry run for Heckle development), and in the course of that, ran across File::Find in my ongoing attempt to find a decent paradigm for directory walking in Perl.

File::Find isn't it.

Oh, I'll grant you that it works essentially the same as Unix find, and it's part of the standard distribution, but it's still opaque.  I need a presentation that's not opaque.  There are lots of different directory walkers out there, and I still have a hard time following the code.

So, lame as this may be, directory walking is still an open topic for me.  It will be good to get back to active Decl development.

"Why I am creating a programming language"

Always nice to see people articulate one's own thoughts.  New programming languages represent new ways to see the world.  As such, you can't have too many.  I like it!

Devel::Declare

Here's a little module Dave Rolsky turned me onto - I'm going to have to read it a couple of times, but essentially it takes control of the Perl parser by arcane means and lets you define your own syntax.

I love the idea of getting away from rickety code generation and into rickety on-the-fly code munging, but the real value would be letting the Perl parser do what it does best; trying to fake it with line-by-line regexps is just asking for trouble.

A little Word scripting

As you may know, I'm a technical translator by trade these days, so I work under Windows with Word documents a lot. Word has a scripting mechanism that I've often considered to be Microsoft's way of making it technically possible to program whatever you need in Word, without actually making it sensibly easy (making hard things possible and making easy things hard, thus ensuring grist for a robust consulting mill), but back in a previous life I spent a lot of paid time writing Word macros to do text processing. Profiting from that grist, I suppose.

Anyway, this week I have a huge pile of process definitions to be placed into Word, each consisting of a table on each page with a list of steps in a process to be carried out manually (many with barcodes for acknowledgement and so on). Each step has a unique identifier - by which I mean that each type of step has a unique identifier and there can be any number of tokens of each type of step.

And it's all in German and my job is to translate it to English. And the originals are all scanned PDFs, so I can't actually just use a regular translation tool to find repetitive text, or even cut and paste into Word; it all has to be typed. I do have properly formatted blank tables to start with for each document.

What I've ended up doing is putting each type of text into a table in a dictionary document, then writing a macro that looks up the unique ID I've typed into the given target document's first column and, if it's found, copies the already-translated row into the target document. If that type hasn't been encountered yet, then I translate it, and copy the row into the dictionary by hand.

I'm writing about it here, because at a semantic level that Word dictionary document is a database (a NoSQL database, if you want to be all cool-kids about it). If this macro were expressed in Decl, it would be much easier to get a lot of implications of that construal and work them into new ramifications.

In other words, construing one thing as another (a semantic mapping) is a form of recognition that could explicitly underlie the programming process in a semantic language. Also, I think I like the word "construal" as a technical term for this kind of mapping.

APIs in general

Just had a thought today that might be worth keeping. I've been thinking about Web-based APIs and how to use them in Decl, and it occurred to me that really the way I do databases is a decent way to think of APIs. In other words, once I've defined (and named) an API, I should simply be able to refer to it and have a session automagically created. But if I want to create multiple sessions, then I do that in a separate instantiation step.

Once I've defined an API in a module somewhere, then I should then be able to write quick scripts against it very quickly and easily. I hope this makes sense when I read it later.

Update: Here's a nice example of how a Perl RESTful API class can look: ElasticSearch.

A simpler one is REST::Google::Apps::EmailSettings.

Workflow and the Mechanical Turk

I'm reasonably sure I've made this decision before, but I'm making it again: workflow will be in core Decl.

I've been thinking of a Blogger-based task manager (finding tasks defined in Blogger posts). To consider those native tasks, we again need the concept of a map. The result would be something like:

• Define an API (Blogger) and a filter on it (getting all tasks defined in the blog headers).
• Map those tasks onto a workflow structure by defining how changes to one view of the data changes the other.
• Work with the workflow tools built into the language.

What those tools will end up being is somewhat vague, of course, but essentially workflow will let us define when the system checks for changes to tasks, and what it will do when there are changes found. Any data source can then be mapped onto workflow tasks and treated as human interaction.

Case in point: here's an article comparing oDesk and Mechanical Turk. Imagine if we mapped both oDesk and MTurk onto a common API, then represented the whole thing as a parallel workflow with job queues farmed out to oDesk and MTurk. Fast, scalable tasks would go to the Turk, while longer-term relationships with trained workers could be modeled on oDesk. Quality control for MTurk might be built into the API, or it might be explicitly represented in a workflow module - that's obviously an open question, but hopefully it makes you think about the possibilities inherent in giving your programs human intelligence, quasi built into the language.