Literate programming posts

HNN seems to be posting about litprog again lately.

Here's one about literate Clojure. [hnn] Jacques mentions Leo, which I think I evaluated and didn't care for, once. I don't remember why.

John Shipman responds to the notion that "literate programming [is] considered harmful" (just because you quote the greats doesn't mean your insights are great).

Well, that's just two. Could be coincidence instead of enemy action. But it was enough to make me notice.

Donald Knuth's list of "readable programs"

Literate programming by the master.

Literate programming: contemplative versus exploratory programming styles

A couple of days ago, HN was asked, "Why did literate programming not catch on?" Predictably, answers ranged from "Because it's useless for real requirements" to "What do you mean? We use it all the time by policy." But a rough synthesis of the overall sense of the meeting, as it were, led me to consider that literate programming requires contemplation. And sometimes you just don't have the time, or sufficient knowledge, to contemplate.

In the typical startup environment, code is written quickly to address specific needs, and as the business pivots and refines, it mutates quickly. So many of the responses addressed that: literate styles don't react well to code churn, and you end up with a literate explanation of code that no longer matches the code (which is of course always the objection to documentation of any kind).

Reading actual code to determine its purpose is effectively reverse engineering. Sure, well-written code should be readable and so it feels odd to call that reverse engineering - but so much real-world code is unreadable that I think it's a good default attitude.

Ultimately, the exegetical stance of integrating literate programming with reverse engineering should support a pretty good overall software development style: quick prototypes to sound out a new task, followed by contemplation of lessons learned and a literate presentation of useful tooling. That's the goal.

Exegesis, literate programming, and Decl

The basic outline of Decl's new syntax parser is complete and passing tests, and as always when a milestone is reached and I look around the corner at what's next, I'm a little overwhelmed. It makes me philosophical.

My initial Marpa NLP article was actually a very simple exegetical analysis of a prototype script I wrote myself, and as usual when doing a first stab, I ended up writing a lot of special-case code and syntax to handle things, representing a technical debt that in such situations nearly always strangles its host. Fortunately, I iterated quickly this time, taking the insights from that and putting them into a new Decl-based plan.

Slowly, I'm feeling my way towards a Decl-based system for literate-style transformational exegesis, one that I hope will eventually encompass everything this blog has been about.

The advantage to basing everything on Decl is that parsing is done. I can now parse a very rich, informationally dense data structure that is designed right from the start to group things in more or less the same way natural language does. It's easily extended and configured, easily indexed - in short, it's a way of taking notes about program structure and using them to evolve a program.

So that's where I'm going. Very slowly.

Comparative literate programming

Now here is an article that is bang-on the kind of stuff I want to write: a comparison of JavaScript typing completion code with a newer, cleaner, Clojure one that is literately woven from the article itself. This is transformational exegesis, or at least a first stab in its direction. (The jQuery code isn't actually quoted all that much.)

Decl 2.0 syntax parser

I've been working on a lot of thoughts about the Decl reboot lately, including a ground-up rethinking of the basic way of handling syntax, and I've come to some conclusions.

Indentation is a misleading way of thinking about this. Indentation is just an indication of the two-dimensionality of text. Especially if we look at Markdown and its friends and relatives, we really have to realize that at least the block elements are there to exploit that two-dimensionality, to arrange information vertically as well as horizontally to present and shape information.

In fact, I'm getting a lot closer to just saying that Decl syntax and Markdown are sort of the same thing. And so I want to come up with a parsing language for two-dimensional text that is not a grammar built for one-dimensional sentences. Or at least is only partly a one-dimensional grammar.

Along the way, I hope to start looking at some naturally two-dimensional text items:

• Diagrams
• Musical scores
• Other timing diagrams
• Workflow charts, GANNT charts, etc.
• Page layouts and screen layouts for forms, buttons, etc.

And all that could be directly supported by at least part of the parser. Using indentation and block rules, we can do a "terraced scan", as it were, identifying blocks first and then drilling into them to identify more details.

The combination of Markdown with the Decl parser and interpreter, moreover, gives me a very natural way to implement literate programming tools in a way that finally makes sense to me.

I think this is going to be very fruitful.

Full circle in literate programming (for me)

I think I'm back to proposing literate programming in Word again.  I've been considering approaches to LP that I would actually use - and waaaay back in prehistory I actually wrote the beginnings of a Word-based litprog tool.

Here's the thing.  To somebody who's used it a lot - and God help me, I've used Word a lot - Word is pretty nifty tool for editing human-readable text.  The point is to provide tools that get out of the way of presentation of logic, both to the human and to the computer, right?  So let's consider a document, in this case, to be any structured text that permits sections, headers, internal references, bookmarks, and footnotes.  You could use LaTeX if you wanted, or some kind of Markdown, but for me, editing words comes easiest and most naturally in Word.

Now we map our document structure onto code structure.  Again, I want something that supports macros (expressing macros in whatever language is convenient) and templates that are expressed as code.  I also want widget tools that permit the encoding of some logic as diagrams with graphical editors of some kind.

I want tables embedded in the text to be accessible to macros.  Still easy.  And I want bullet lists and numbered lists to be similarly accessible.

Sections might be standalone invocations of templates.  I'm not sure yet.

References go into a bibliography; the bibliography is literally links to additional libraries, organized on principles I haven't yet thought through.  But as I've said elsewhere, or maybe here (forgive me, the paying work has been glorious lately) - I want to have something amenable to peer review.  Model open coding on scientific publication.

So that's this week's goal.

Macros and washing machines

Well, here's a timely screed about macros and why they make sense - to which I can only say, "Yeah!"

Which brings me to discussion (again, and we'll keep discussing this until it comes out right!) of macros and code generation.

Let us imagine a system of articles and books that describe a codebase.  Some of the codebase may be maintained outside this system; some of it within, because the articles include some literate programming tools that can generate sections of code.  (This way the system can be used to start analyzing an existing codebase and slowly grow to encompass all of it, as needed.)

An article is equivalent to a book section, that is, a book consists of a hierarchical organization of multiple articles, presumably related.  An article may still have hierarchical structure within it, though, because sometimes you just need that for clarity.

In general, though, a single article addresses a single "thing".  That topic could be a feature or a specific function, or it could be a change request touching many different parts of the system.  Ideally the maintenance of a complex system would thus have a narrative made up of multiple articles explaining the thinking at each stage.

OK.  So in that context, let's assume that some of our literate programming-type tools include arbitrary macros that can be reused.  (Literate programming can be seen as writing a number of single-use macros, so generalization of that to reusable macros is no great leap.)  Some languages are easier to macro-ize than others, of course: we have to parse things to make truly effective use of macros without leaving the native syntax.  But by extending the native syntax with a template language, of course (as we do in literate programming, actually) we can build macros for any language.  The key is the code generation, you see.

It might be a good idea, though, if particularly questionable or novel macros were to be given a kind of "half-way existence", where the macro as well as its expansion are shown in the presentation.  Maintenance then has a template or macro to work with, but the full code is shown for clarity.  There are plenty of instances where that makes a lot of sense to me.

Literate programming reprise

So Jeremy Ashkenas has a post on literate CoffeeScript, which is a feature of 1.5, apparently.  And he links to Knuth's own CWEB write-up of ADVENTURE, the original adventure game.

As always when reading Knuth, it makes me think.  One of the thoughts it provoked yesterday was this.  Literate programming as currently conceived has been criticized as not being sufficiently cognizant of the practice of reusability - which is true.  On the other hand, it might actually be nice to track the evolution of subroutines (say) from version to version as one's own skill and knowledge of a given domain grows.

In other words, reusability in the form of a library is also not the end goal.  You can kind of reconstruct the history of a concept in git.  Kind of.  I've never actually done it.  But it might be interesting to have some kind of index of code at (yeah) a semantic or descriptive level that explicitly makes it clear what it is supposed to do and how it reflects increasingly refined knowledge of the domain.

I'm having troubles articulating this.  Hopefully this flailing around will be enough for me to reconstruct the notion later.

Monolithism and literate programming

Interesting story of an example program Donald Knuth wrote and a critique by Doug McIlroy.  Knuth's program, as Knuth's programs tend to be, was monolithic, taking and handling each aspect of the problem in turn and weaving the whole thing into a Pascal program for compilation.  McIlroy's solution was a six-line shell script.

The point here is that Knuth's program did nothing at all to support or to make use of reusability.  And that's a pretty good point.

Declaration of constants

In a graphical coding environment (or something like a literate programming environment), you would have a reference section with the details of constants or starting values. For example, if I have a simple script that works on a list of things, I could put the list of things in a separate file, or define it as a constant list (a local table, in Decl). But if it were a constant list, then it would normally be hidden; you'd just want to be able to click it to manage the data.

Or alternatively, you could specify viewing parameters right on the object or in a separate viewing preferences object in the script file. The more I think about writing my own code editor for Decl, the more I like it - even though I'm reinventing the wheel, as usual.

Raganwald: why literate programming matters

Must be literate programming week.

Some good points here - points that were good twenty years ago but still don't go far enough to really support me to the extent I want to be supported.

Update 2012-06-02: A more careful read reveals that this is a really good article about the semantic issues lurking behind the human activity of programming.  I'm glad some anonymous person searched on "Raganwald twitter" today.  (Still don't know what this has to do with Twitter ... well, of course, now it does.)

Open-source target: Hacker's Diet Online

So I've been using the exercises from the Hacker's Diet, an adaptation of the old 5BX standby we all know and love, and poking around Walker's site, I see that he's put together an online service you can use to track your Hacker's Diet data (scroll down to the bottom for code references). And the fascinating thing about that site is not just that it's open source, but that it is the first actual instance of literate programming using Nuweb I've ever seen in the wild. [Note to self: poke around that site's links some more.]

It deserves praise for that alone.