Literate programming: contemplative versus exploratory programming styles

A couple of days ago, HN was asked, "Why did literate programming not catch on?" Predictably, answers ranged from "Because it's useless for real requirements" to "What do you mean? We use it all the time by policy." But a rough synthesis of the overall sense of the meeting, as it were, led me to consider that literate programming requires contemplation. And sometimes you just don't have the time, or sufficient knowledge, to contemplate.

In the typical startup environment, code is written quickly to address specific needs, and as the business pivots and refines, it mutates quickly. So many of the responses addressed that: literate styles don't react well to code churn, and you end up with a literate explanation of code that no longer matches the code (which is of course always the objection to documentation of any kind).

Reading actual code to determine its purpose is effectively reverse engineering. Sure, well-written code should be readable and so it feels odd to call that reverse engineering - but so much real-world code is unreadable that I think it's a good default attitude.

Ultimately, the exegetical stance of integrating literate programming with reverse engineering should support a pretty good overall software development style: quick prototypes to sound out a new task, followed by contemplation of lessons learned and a literate presentation of useful tooling. That's the goal.

Exegesis, literate programming, and Decl

The basic outline of Decl's new syntax parser is complete and passing tests, and as always when a milestone is reached and I look around the corner at what's next, I'm a little overwhelmed. It makes me philosophical.

My initial Marpa NLP article was actually a very simple exegetical analysis of a prototype script I wrote myself, and as usual when doing a first stab, I ended up writing a lot of special-case code and syntax to handle things, representing a technical debt that in such situations nearly always strangles its host. Fortunately, I iterated quickly this time, taking the insights from that and putting them into a new Decl-based plan.

Slowly, I'm feeling my way towards a Decl-based system for literate-style transformational exegesis, one that I hope will eventually encompass everything this blog has been about.

The advantage to basing everything on Decl is that parsing is done. I can now parse a very rich, informationally dense data structure that is designed right from the start to group things in more or less the same way natural language does. It's easily extended and configured, easily indexed - in short, it's a way of taking notes about program structure and using them to evolve a program.

So that's where I'm going. Very slowly.

Impulse Tracker open-sourced

Impulse Tracker was an influential piece of software in the 90's electronic music scene (and is doubtless still influential today, but isn't used as much, naturally). Its author just open-sourced the whole thing. It would be an interesting code understanding/exegesis target.

Article up, and also: exegesis and code analysis/understanding

My first article kinda-sorta based on an exegetical approach (of my own prototype code) is up on the Vivtek site, and the six-day code rush to write code to build the article from a note database has really whetted my appetite for more of the same. There are all kinds of exegetical efforts I want to make, getting into code reading in a big way.

So I trolled around Google for salient things. Here's a list of interesting things.

• TipsForReadingCode at c2.com was quite helpful as a set of ... well, tips for reading code.
• Code comprehension tools at grok2.com; the vast majority of this class of tool is closed-source and rather expensive. I think this is largely because large codebases are not typical for open source projects, but rather enterprise code, and enterprises have the money to pay for expensive tools. That's my reading, anyway. But:
• cscope is a venerable tool for static analysis of C code, anyway, and has been perverted to handle other large-scale grep-like analyses of large numbers of files. The tool itself may or may not be something I'm interested in, but its approach is probably pretty valid.
• cflow is another flow dependency analysis tool, also open source.
• Perl is, as always, a special case.
• Well, what about static code analysis in general? Here's another list of tools.
• There's a clang-based analyzer.
• This moribund project on code "aspects" looks fascinating.
• Finally, a book on pattern-based OO refactoring, which also looks pretty fascinating.

So, as always, lots of people are doing things tangentially related to what I want to do, but nothing is 100% there - because what I want to do is an extract-then-literate-programming kind of thing. We'll see how much sense this approach makes. Wish me luck.

Exegesis as normal publishing tool

I've been kicking around the notion of a "code exegesis" for a little while, which is the attempt to take some software project (in the simplest case a single file) and to "back-comment" it, that is, explain the author's intent and strategies in the development of the code, as well as possible, and also to focus on different aspects of it in a series of separate articles (or chapters if the whole work is considered a book).

This is an exegesis as classically understood - detailed commentary on the ideas and history behind a given work, often scripture but also e.g. Homer. I call this "interpretive exegesis" to distinguish it from literate programming, which is essentially the same thing except that it independently *generates* the code, so I call it "generative exegesis".

With me so far?

All the publishing I want to do at this point is code-based. So far I had considered doing a Markdown-enabled Perl weaver that is essentially Jekyll in Perl, so I called it Heckle. It was entirely vapor, fortunately - because I'm renaming it "Exegete" instead. I'm going to use exegesis as the basis for all my publishing, because I'm going to be quoting from things all the time anyway. The same document organization tools could be used for anything, not just exegesis, but honestly, it's still a great name.

There are a couple more ideas here.

First is the realization that the same explanatory exegetical structure would be doubly appropriate for binaries, for disassembly and reverse-engineering. Here, instead of a dynamic picture like a conventional disassembly tool (which can be seen as a kind of explorer/browser), we'd explicitly be supporting the writing of articles about a given binary structure, but overall the same principles as IDA or Radare would apply: the identification of small extents that express a given set of actions and ideas.

And then there's the notion of a "derivative work" - a kind of hybrid of interpretation and generation which transforms the original into a new work with changes. This is not going to be a very normal mode for most purposes, because it's not the same as normal maintenance, which is typically done in a more evolutionary fashion. This is definitely intended for those punctuational cases like porting, or reimplementation of archeological finds from the 70's or something. A good term for this would be a "transformational exegesis".

And of course it would be perfect for patching binaries or similar reverse-engineering tasks.

So that's kind of where my thinking is at. Since all this involves the writing of text, probably extensive text, that includes references to and quotations of code objects, it's pretty much ideal for the kind of tech writing I want to do anyway.

Boilerplate

So OK, what about boilerplate?  (Again...)

First, there are a number of modules in CPAN that produce boilerplate of various description, the first being naturally Module::Starter, which I use on a weekly basis. Its structure is surprisingly straightforward, but that's just another way of saying that it is expressing things in Perl that (in my opinion) could better be expressed in a specific boilerplate DSL.

There are others, such as Module::Starter::PBP by Damien Conway, based on "Perl Best Practices". Drupal::Module::Start. A Padre plugin (I did not know this until just now). There's Test::STDmaker, which takes Perl test output and puts it into a boilerplated document to conform to military purchasing standards. HTML::HTML5::Builder, which "erects scaffolding" for Web apps. Even WWW::Mechanize::Boilerplate, which I really would like to look at more closely.

In other words, CPAN contains a lot of knowledge about boilerplate, both Perl-specific and otherwise. (Another reason for a survey, yeah?) But what occurs to me is that I don't just want to look at boilerplate generation. I also want to explore boilerplate degeneration, as it were - extraction of higher-level information from a given text based on recognition and abstraction of boilerplate. This is just phrase-based parsing writ large, using more complex lexical entities, but I think it would be - well. It's a lot of what I expect an exegesis to be, to be honest; an abstract "understanding" of syntactic forms.

So there is actually a boilerplate extractor on CPAN, Text::Identify::Boilerplate, which, given a set of files, will do a line-by-line diff and extract the boilerplate. That's pretty slick!

But first, I propose Text::Boiler, which will take some kind of declarative boilerplate description and build that. Then Text::Unboiler, which will undo boilerplate (perhaps with overrides for changes to the boilerplate itself) and return to you the original record used to create the final files.

Ah. Right. Boilerplate + information = syntax. Boilerplate contains named fields, probably also lists and so forth, and the record contains those fields (which can also have default values if the record omits them). But the record can also override anything in the boilerplate. If the boilerplate also has named sections to make that easier, then unboiling should be pretty flexible indeed!

I think this is going to be a pretty profitable way of looking at things, especially in terms of exegesis, which is kind of a "manual unboiling".