Marpa, German, and ParZu, oh my!

I spent most of May working through my old natural-language tokenizer, adding a vocabulary-driven lexer/lexicon for German, all in preparation for undertaking a Marpa-based German parser. That's looking halfway decent at this point (except I need to do much better stemming), and then I decided to do a general search on German parsers and found ParZu.

The unusual thing about ParZu, among parsers especially, is that it's fully open source. That is, it has a free license, not a free-for-academics-only license - and it's hosted on GitHub. Also, I can try it online. So I fed it some more-or-less hairy sentences from my current translation in progress - and it parsed them perfectly.

So here's the thing. I kind of want to do my own work and come to terms with the hairiness of things myself. And then on the other hand, parsing German by any means would allow me to jump ahead and maybe start doing translation-related tasks directly....

It's a dilemma.

Marpa stuff

As I get further into Marpa, I'm starting to see there's a whole little world of cool stuff out there based on it. Here are a couple of bookmarks for later.

• A fantastic article on using Marpa to convert Excel spreadsheet formulas into Perl using AST transformations.
• Kegler's "Ruby slippers" parsing technique: essentially ways to trick a simple grammar into functioning within a larger whole by using invisible tokens and wishing the language were easier to parse. Marpa is ... well, it's beyond cool and into virgin territory.
• Another Kegler post on mixing declarative and procedural parsing that should come in handy here and there.
• Here's a gist showing a Marpa parser for CSS that uses a tokenizer external to Marpa - the key technique is in the loop starting on line 187, where we pass each individual token to the recognizer. Only after the token stream is complete do we read the value from the recognizer. (So for a series of sentences, do we have to create a new recognizer for each sentence? I think we actually do. That will be something for experimentation later.)

Marpa

So I decided to sit down finally and write the line parser for the new Decl, and since it was parsing, I decided not to unearth my old HOP-inspired parsing code but rather take the plunge and try Marpa, to avoid getting bogged down in parser issues.

I am in love.

It basically looks like Marpa can do anything related to parsing. It can even handle ambiguous parses! One of the test cases is literally "time flies like an arrow"!

But what doesn't yet exist (there's a partial beginning) is a tutorial set, a "Gentle Guide to Marpa". I think I'll write one.

Parsing C

So I'm taking another stab at writing a quasi-literate-programming tool, which, as I am writing things in C with it, requires a credible C parser to find declarations of stuff.

And while Perl has lots of C parsing tools of varying quantity, including a sample with Parse::Eyapp (which is quite fascinating in its own right), none of them are easily adapted - with the exception of the Inline tools.  Inline::C::ParseRegExp, for example, which does exactly what I want it to - find declarations of stuff.

Python, though, has pycparser.  (And of course, Perl has Inline::Python...)

And then, as always, there is Marpa.  I still have a big fat to-do on my list that says "Learn Marpa".  There's a new set of tutorials on Kegler's blog.  I need to work through those.

Update: I realized I was wrong.  I don't actually need a parser - just a tokenizer for C.  This is because all I need to do is cross-reference all identifiers, and the job is done.

Marpa tutorial

Ooh, Kegler wrote another Marpa example tutorial of a DSL.

Think what this will do when wrapped in Decl!  Maybe I should write the parser tutorial chapter early on, then double back to write more sensible things before it.

Decl is dead. Long live Decl.

I just pulled the plug on the old Decl at Github, replacing it with a tabula rasa generated by module-starter.  The goal at hand is simple: rethink everything from the ground up.  So first things first - our first program to implement is this:

text "Hello, World!"

To make that work, we need a few things:

• The interpreter environment itself (an object of type "Decl").  The environment was itself a node in the last iteration, and I suspect that's a mistake.  There should be a root node for ease of self-printing, but that should be a child of the environment, not the environment itself.
• A "decl" command right from the start that acts like an interpreter.  The environment should be a shell, I think, so we can interact with the environment.  It's not a Python shell - the Python shell builds the environment as the result of a series of verbs, and that's specifically what Decl doesn't want to do.
• Loading code into an environment has to be easier: (1) with a source filter, (2) passing a string in, (3) from a file, and (4) passing some intermediate data structure in - all those have to be supported more transparently than the last version.
• The output handling system has to be in place in at least a rudimentary fashion here.
• Marpa parser.  I'll probably need to refine the grammar as I go.

About that. In the old version, I had sigils to determine how the body of a node would be handled.  I'm eliminating that.  Instead, a trailing quote will mark a text body, and brackets a code body.  Anything else is vanilla nodes. Sigils are simply too hard to remember.

Similarly, the language a code body is in was marked with perl < { } or python < { }.  I doubt that's a good idea. It's clunky and ugly - and I want to be able to use Python with ease and elegance (as well as C).  I'm not yet sure what the solution is.  For multiline code bodies, I can see "{ (perl)" as an override of the default language - the default language being set sometime earlier.  For single-line code bodies, though, I don't see that as sufficiently elegant.

Another thing I've been thinking of.  When using a semantic domain module, I want a much, much more explicit definition of the tags in the domain.  To date this has really sucked.  If a module is used from within a Decl environment, then a lot more information should be provided right at the start.  If the same module is used from Perl, though, it should act like a Perl module - and load Decl itself.

This won't come up for a little while, of course - definition of domains is a little way down the rebooted road.

This might take a while.