Decl 2.0 syntax parser

I've been working on a lot of thoughts about the Decl reboot lately, including a ground-up rethinking of the basic way of handling syntax, and I've come to some conclusions.

Indentation is a misleading way of thinking about this. Indentation is just an indication of the two-dimensionality of text. Especially if we look at Markdown and its friends and relatives, we really have to realize that at least the block elements are there to exploit that two-dimensionality, to arrange information vertically as well as horizontally to present and shape information.

In fact, I'm getting a lot closer to just saying that Decl syntax and Markdown are sort of the same thing. And so I want to come up with a parsing language for two-dimensional text that is not a grammar built for one-dimensional sentences. Or at least is only partly a one-dimensional grammar.

Along the way, I hope to start looking at some naturally two-dimensional text items:

• Diagrams
• Musical scores
• Other timing diagrams
• Workflow charts, GANNT charts, etc.
• Page layouts and screen layouts for forms, buttons, etc.

And all that could be directly supported by at least part of the parser. Using indentation and block rules, we can do a "terraced scan", as it were, identifying blocks first and then drilling into them to identify more details.

The combination of Markdown with the Decl parser and interpreter, moreover, gives me a very natural way to implement literate programming tools in a way that finally makes sense to me.

I think this is going to be very fruitful.

Afterthought on Markdown: Hoedown

The standard Apache module for Markdown is now Hoedown, which supports lots of extensions, is written in no-dependency C, and is essentially bulletproof. It also separates the parser from the renderer, which is important if you want to index various text pieces. There is a CPAN Text::Markdown::Hoedown module which compiles and passes tests on my box; we'll see how easy it is to use (not much documentation, but that can be fixed with a pull request...)

In poking around and reading the documentation for Markdent, I found a couple of interesting proposals for Markdown extensions from Dave Wheeler: definition lists and better tables with multiline content.

So what I really want is a Hoedown-based parser generator that can add extensions at will. (Maybe later...)  Because honestly, what this is all about is different ways to use simple 2-dimensional, as opposed to 1-dimensional, arrangements of punctuation to delimit different items in text in as general a way as possible, build data structures based on that text, then render presentations based on those data structures. Each level of that process is interesting in and of itself.

Multimarkdown

Multimarkdown is sort of next-generation Markdown; the main project has transitioned out of Perl into C for performance and now addresses all kinds of output.

The output I'm interested in is just the parsed structure - the DOM, if you will. I'm torn between recycling their parsing code (which doesn't actually parse everything I want, but works and works well) and writing my own parser (which usually leads to everything in flames and me losing my hat).

There is, of course, a Text::MultiMarkdown - but it doesn't quite parse correctly and would have to be subclassed to add additional features. And there's Dave Rolsky's Markdent, which does actually provide an event-based Markdown parser but comes with an entire ecosystem of modules and doesn't appear all that easy to extend (unless you're quite familiar with the Moose paradigm).

But overall, you just can't get away from Markdown these days when writing content. So I just keep coming back to it.

Problems with Markdown

Here are some problems with Markdown when you get down to serious formatting with it - but the basic idea I still love.

AsciiDoc

Learned about a new markdown variant today, AsciiDoc [cheat sheet].  It's in Python, but that doesn't necessary mean it's bad.  (ha)  I'm considering using it for my notes application.

Also, there's talk about standardizing Markdown.  Interesting.

Markdent

Well, naturally I want to write my books in Markdown instead of HTML, so I've been prowling around CPAN for proper Markdown processors...  They're all direct-to-HTML converters, except one: Markdent, by the ubiquitous Dave Rolsky.  Markdent is an event-based, configurable Markdown dialect parser.  So that's what I'm going with - because I want to be able to pull out links and the like as I go along my tree, and later I want to be able to output HTML5 article/section/aside tags, especially "aside" for sidebars, and I want to be able to write a markdown block for those.

Actually, I'm probably going to standardize on Markdent in Decl as well, once I get back to it.

Of course, that introduces a Moose dependency, but you know what?  Maybe it's time.