Project management software and scope creep

Project management is just one of those areas I keep coming back to, from a semantic standpoint. Since it's by definition a tool that every software development project must use (to paraphrase a financial management book I once read, if you have a project, you will manage it - the question is whether you'll manage it well or not), every software developer sooner or later takes a stab at building a project management tool.

As a result, the space is cluttered with three million solutions, all of which do more or less the same thing but expand out to include every aspect of life within an organization. This is scope creep.

Why does it happen? A lack of semantic explicitness.

Case in point - or rather, what's set me off this time: my son is doing a self-study course of AP Macroeconomics. To finish it by the time the AP exam rolls around in May, we have to make a schedule. And as I was doing that by hand with a calendar, of course, I thought: I should be doing this in Perl.

But you know what? CPAN doesn't really have anything that does that. And when I go off to find general project management tools in the open-source space (let's not even talk about the proprietary space!) I find at least a dozen contenders.

They all involve stack lock-in, to a greater or lesser extent, because, well, that's what syntactically specified software does.

And yet we all know roughly what's meant by project management. That is an area rife for a semantic approach. Sadly (or, well, it's not really sad) all my free-time development cycles are taken up by robotics these days. So this might have to stay on the back burner for another few years. But eventually I promise you....

Declarative description of financial contracts

This is something moving rather close to a semantic approach. Very interesting.

Why the semantic level?

For some reason I seem to have gotten off onto a philosophical tangent while walking the dog this evening, with the resulting epiphany: programming is a process of comprehension. And then documentation of that comprehension.

Back in the day, I spent several years working on two large systems (and here and there some other stuff, but the bulk of my programming work was on two systems). The first was a pharmaceutical document management system, and the second was a searchable online machine tool database. The first was for a corporate gig, obviously - pharmaceuticals are not something done in the garage by a startup - and so it was relatively well-managed and the technical debt was relatively well controlled.

The second was a startup, I was the technical lead and entire staff, and over the course of some dozen years I managed to run up a significant technical debt. Due to that debt, working on the system became an increasingly painful chore (every attempt to address one issue simply reminded me of the dozen related issues that were not going to be fixed because the customer had a very limited budget).

The process of leaving that situation and moving to technical translation took many years, and something in me fought it every step of the way. It was the last paid programming I ever did. Since that time, I've been a hanger-on in the startup programming community, but nothing ever gets off the ground, essentially because I have a fear of technical debt.

But why do we have technical debt? I'll tell you: because we commit to specific platforms and solutions during the programming process, and it is very difficult to undo those early decisions later. By programming at the syntactic level (some of which is of course unavoidable) we lock ourselves into low-level structure we can't easily back out of.

Addressing things at the semantic level - were it possible with existing tools - would avoid at least some of that technical debt. If we have semantic structure - if we are defining not software but the concepts behind the software - then the programming itself starts to look more like a compilation process. And just as we can recompile most code onto a new platform (maybe after fiddling with some flags and libraries), we could back out of syntactic-level, stack-level decisions by "recompiling" a set of concepts on a new platform.

Indeed, in a sense a new set of requirements would be a sort of recompilation.

It's a vague ideal, but this is essentially what I see as the promise of semantic-level programming. I earnestly hope I'll be able to make some forward progress on this over the next year.

Semantics and the Windows Registry

In the paying work, as you know, I am a technical translator these days, and that means I work with CAT tools (computer-aided translation - keeps track of stuff you've already done so you don't have to do it twice). The main CAT tool on the market is TRADOS - a name that dates it, as it sounded modern (like DOS!) when it originated in Germany in the 80's. At any rate, TRADOS has some odd ideas about rent-seeking versus more modern openness, because back then, you had to code things down to the metal without any help from anybody and by golly you felt you deserved some money for that. The upshot is that the TRADOS Freelancer version restricts a freelancer to working with five installed languages.

My problem is that I relatively frequently work with seven: English, German, French, Spanish, Italian, Hungarian, and Portuguese.

The TRADOS solution for changing the languages you currently have installed is easy: uninstall the entire tool, then reinstall from scratch; during the reinstallation process the language wizard asks you for your five languages.

Since the TRADOS install process takes twenty minutes, this means considerable effort when switching the cards you have in your hand (in Yugi-Oh terms).

I've been working with a lot of Portuguese and Italian lately for some reason, so this has really been a bother this week - and finally I decided to hit Google and figure it out. Sure enough, those languages are stored in a Registry key (the translation industry runs on Windows). In the older pre-2009 version of the software, the key is relatively understandable, while in post-2009 versions there is an opaque lengthy binary value that I haven't figured out yet, but the key is this:

You can change your language selections by diddling with the Registry.

Well. In Perl, of course, we use Win32::TieRegistry to work with the Registry; it allows you to treat Registry keys as hashes and do the obvious things to modify them. But if I want to distribute a convenient tool to make this available to the unwashed masses - and I do! - then I should probably write that tool in C/C++ with a bog-standard resource-file GUI, because distribution of Perl is not trivial enough to make it worthwhile for a simple utility.

But the model of the Registry, and therefore the model of actions I take against it, should be shared between those two approaches. At a semantic level, we are talking about the same thing. But without a semantic language, I can't really realistically do this.  And so in a certain sense, what we're talking about here is defining a model of the Registry, and then a model of the actions I want to take against it (in some abstract form, like a template or something).  The model of the Registry maps onto either Win32::TieRegistry and Perl, or it maps onto some library code and template-y stuff and C/C++, and the actions then can be expressed in either language in some as-yet-unexamined way.

That's what I'm talking about when I talk about semantic programming.

Now let's take that as a given, OK? Because I want to address an even more abstract concept along these lines.  At a higher level, the Registry falls into a "value store" bucket that could include e.g. XML in an initialization file or other kinds of configuration files. At that higher level, the choice of a specific form of storage is an architectural decision; in other words, the architecture of a program is in a sense independent of the actual program - in that we could take a given implementation of an algorithm and "translate" it from reading an XML config file into reading the Windows Registry, and everyone would agree that is the same program (in a sense). The problem is that this kind of port is essentially a fork; changes made to the syntactic-level program on one branch are very difficult to translate back into the other branch.

In a semantic programming paradigm, they wouldn't be. You'd be working at the level above the architecture, and simply "compile" the specification into specific code in different languages on different architectures with different choices made in terms of protocols, storage locations, Web server back end, database, JavaScript front end, and so on.

And that's also what I'm talking about when I talk about semantic programming.

Anyway, sometime soon I hope to build these little Registry tools. In general, it would be nice to think about the general set of all Registry value utilities and how they could be addressed by a semantic domain in this manner.

Dictionary of Algorithms and Data Structures

Semantic gold mine! A long-running personal project at NIST cataloging data structures and the algorithms that use them.

Summer hiatus

Due to health issues and travel (and it's always fun when those coincide) I have not really done any programming or thinking about programming for about two or three months now.  So I'm coming back to a lot of my old ongoing efforts with a fresh eye, and today I had a strange epiphany:

I'm thinking of the platform for a given piece of software as ephemeral now.

For instance, one of the things I'm working on is a parser of English in order to automate some of the language-quality work I do professionally. I'd like to implement that on my usual machine, but for performance reasons it would be convenient to offload it onto the Parallella platform since I expect it will really benefit from it.

So I can't really write it in Perl because of platform conflict. OK, I know Perl will probably run fine on the managing processors - but the point here is not whether Perl will or won't work, the point is that I really want to develop the algorithms and then "compile" them to Perl or C or whatever, as needs require.

This is what Java purports to address, by the way.  But I'm seeing a lot of new languages that "compile" to various high-level languages, notably JavaScript and C, and maybe this is a new modality.

Maybe what semantic programming is about, I tell myself yet again, is working out the semantic content of an algorithm, expressing it at that level, then having it run in whatever platform is required - and if that means "compiling" to a given language, then in a sense it's really coding in that language. The semantic structure is expressed in C or in Perl, but at some level it's also expressed as a bunch of semantic units that could also be used to express an explanation of the code in English, or even to derive a domain-specific language for intermediate work, a set of macros or something like that.

In other words, what I'm internalizing is that in a semantic programming paradigm the computer should be doing more of the work of coding, at a level that reflects a knowledge of the underlying purpose of each part of the code. That naturally ties back into code understanding to reverse-engineer this kind of semantic structure given existing syntactic expressions, but it's output that should logically come first.

Data modeling and semantics

I've been musing more about the semantics of data modeling lately - or really rather about the fact that data modeling is a form of semantic manipulation.  The thing that makes semantics interesting is how semantic structures can be mapped onto other semantic structures. That is, the mapping, or recognition, of structures is really what semantics buys us.

In the case of accounting (sorry, I do tend to fixate on particular applications for months or years at a time), it would be instructive to gather the various data models used in open-source software (well, and open formats such as QIF used for non-open software) and do a kind of line-by-line comparison. A mapping, in fact - a mapping onto the semantic constructs that accountants use to talk about accounting.

That nexus is where semantic programming resides, in potential anyway.

At any rate, comparison of projects in this manner would allow us to identify certain features of accounting data structures that were incorporated into or absent from different models.  Description of those variants is also part of modeling, and a full description would permit us to auto-generate data migration tools.

And once you can migrate data back and forth between different representations, well, then you have semantic data management, I guess.  Not (semantic data) management, that is, but semantic (data management).  You've started to graduate from data to knowledge.

howdoi

A Python code help tool on the command line.  Neat idea.

Code editors again

So somehow I ended up with a bunch of tabs open with code development environment articles.

• IDEs are important to Java because Java has lousy code arrangement - lots of tiny files with framework-induced nomenclature.  So the IDE is a navigational tool.  That squares with my memories of working with Visual Studio back in the day - vast amounts of boilerplate and Studio was really necessary to find the good stuff.
• Textadept is a programming editor written mostly in Lua, that also uses Scintilla as the editor component.  This makes it kinda like Padre (no doubt why it has no Perl tools).
• Zen Coding is ... typing acceleration for HTML and CSS.

I have some pretty decent thoughts about these links, but it's late and the thoughts are rather inchoate.  Short version: every set of code is a text that is written in a formal language in order to express some carefully defined syntactic structures that can be translated into code or actions.  But - and this is again not a new insight - those syntactic structures are a reflection of the deeper semantic structures in the programmer's mind as she comprehends the problem to be solved.

The actual program may or may not solve that problem (hence the need for good testing), but its intent is to do so.  In reading code, we attempt to discover that intent and reconstruct the deep semantics.  What I'd like to do in an IDE and/or editor is to maintain something approximating those semantics in a structure during editing.  As that toolset improves, you could communicate with the editor on a higher level, interacting with the semantics and letting the toolset manipulate the specific code.

Because you don't actually care about the code any more than you care about assembler.  (Unless you care about assembler, but that's a different point.)  You want to solve your problem.

By looking at competitive programming problems and problem statements, I hope I'll be able to have small enough and abstracted enough snippets of semantics that it will be realistic to think about how the semantic comprehension of the problem statement is translated into program structure.

Casual programming

Here's a very thought-provoking post, generally about the notion of "casual programming", i.e. a language/IDE/whatever that would (as I phrase it) work with you at a semantic level to co-write the software you're working on. It would do this by having context-sensitive assistance for API inclusion. This is a pretty good approach.

Another attempt to express the goal

I'd like to be able to express software in terms of its semantic content - then change it very quickly in response to changes in requirements. Ideally, I'm really after semantic-level (instead of neural-level) machine learning.

In other words, I'm still trying to do exactly what I wanted to do twenty years ago, except now I have better tools.

A couple of posts about hating programming

A post last week, "I want to fix programming," suggests that declarative programming (keeping around the "why" of what you're doing instead of the "how") would be a step up from what we're doing, and adds a couple of neat magic keywords that indicate how it might be done. Color me unconvinced, though, because just saying what you're doing and hoping the compiler is going to get it right might require a little too much from the compiler.

His specific case is a sort algorithm - he just specifies what he wants from the sort, and the compiler would do the sort. Well, I still think the answer there is to tell the compiler: "sort this stuff". Now I don't have to look at his constraints and say, "Well, wait a minute, that's a sort." If I were talking to a colleague, I wouldn't say, "All the elements of this list should match those in the other, with the constraint that each element in the list is less than the one after it." I'd say, "We'll sort this list." Then I'd look up a sort algorithm, and later, down the road, if it turned out that sort was a performance bottleneck, I'd maybe look for a better sort algorithm.

Just specifying the constraints is not enough. You're still losing information if you don't have a machine-readable set of semantics of what you're doing. (I say that as though this was already achievable with today's tools, when of course it mostly isn't, but you see my point.)

This week, we have a response, "Yup, programming is borked," and makes the cogent point that formalization of requirements is the key - and then still goes off the rails with brand-new syntax that's not human language or anything like it. He's got some interesting ideas, and is looking at authoring systems for non-programmers, which I find pretty fascinating, but I think he's still getting lost somewhere along the way.

I remain convinced that a hierarchical set of semantic views, preserved in the final program, is the way to go. Each part of the program not only has formal syntax defining it, but a conceptual framework indicating what it does, why it's done that way, changes that were made along the way, related tests (which can be seen as constraints) or even constraints (which can be seen as tests) - the whole ball of wax is right there and amenable to software manipulation. Changed requirements can then drill down right to the code that must be changed, and with sufficient sophistication, you could imagine that tools could make suggestions or even rewrite the formal syntax to conform to the requirements.

That's the goal.

State machines in Perl

I just did a quick CPAN search and turned up a number of interesting packages:

• FSA::Rules is the package I initially started building a wrapper for. It's actually pretty nice, and has a couple of constructs that my last post probably is missing.
• Parse::FSM builds a parser based on an FSM constructed laboriously by function call.
• State::ML provides a utility for converting XML-encoded state machines into other things or even code. I like the code generation aspect!
• Win32::CtrlGUI::State is a slick little state-machine controller for Win32 GUIs.
• Basset::Machine builds a state machine class in much the same way Term::Shell builds a command line shell.

All pretty cool aspects of the state-machine paradigm. If you really wanted to start getting into the semantic programming approach writ large, you'd think of ways to produce code generators for any of these starting from a Decl state machine description, and ways to organize that kind of code generator family into a semantic domain.

Two years in

I started this blog on November 5, 2009, with every intention of investigating a specifically semantic framework for programming that might have borne fruit by 2011. It's November 6, 2011, so where do we stand?

I started work on Decl in February of 2010, according to my notes (the first SourceForge checkin was on February 15, but I'd posted on Wx::DefinedUI on the 10th, and honestly I think a Markov-chained snippet from my earlier writing may have triggered the concept in January), and it quickly grew to take over my every waking thought. Essentially, all my progress with semantic programming has been in the implementation of Decl. As I noted on February 10th, my earlier effort in late 2009 foundered on the shoals of syntax. At least that's no longer a problem.

The idea of Decl is to define semantic domains and tags that declare various types of programming construct, then to build programs of those. Eventually, the semantic domains should have enough macro machinery involved that the programs will largely self-construct, but I'm nowhere near that level of detail yet. I just finished the v1.0 macro system last month, after all, and it's by no means clear how to get from point A to point B.

But that's where things stand. I have done some musing about shoehorning my old Hofstadter microdomain work into Decl - not that that would require much shoehorning at all, which is the raison d'etre of Decl in the first place - but haven't really made a serious move in that direction yet.

I'll leave you with this notion: the Decl tag is an instance of a concept. As such, it's a token from a Lexicon. I haven't implemented the actual Lexicon yet - but at least Decl will be a language capable of expressing it right from the start. And that's why Decl is important.

Hmm...

Ping.

So there are others out there!

A possible approach

My ultimate goal with Decl is, of course, not only to provide a quick way to bootstrap data-structure-heavy Perl scripts into being, but also to describe software systems at a high level and provide a framework to implement them.

So in that second sense, a semantic domain of "natural language processing" would describe tasks at a high level, describe the algorithms and approaches to take in performing those tasks, and would be amenable to at least some degree of automation in coding the tasks in other languages using various toolkits already available. In other words, the semantic domain encodes at least some of the professional knowledge about that domain that a seasoned programmer would be expected to have; a programmer-in-a-box solution.

To that end, and maybe in NLP to an extent that's a little unusual in comparison with other domains I've wanted to get into, there has to be a means of describing a given toolkit and its basic approach - a theoretical framework if you will - that allows a given task/algorithm to be expressed using it.

Not sure how that's going to happen yet; I just want to throw the gauntlet down here.

It looks like a lot of NLP toolkits are in Java, for whatever reason, with NLTK in Python being a strong contender. Nothing, really, in Perl. Which is why God moved Ingy to create Inline, of course, and Decl will be incorporating Inline very soon.

Something I don't think I blogged at the time

CodeBubbles. I don't even have time to pontificate on it - tl;dr is that it's kind of a mental snapshot of coding for a given issue, and a whole new approach to the IDE. I love it.

Rosetta Code

So there's this site Rosetta Code that shows snippets of how to do Task X in lots of different languages. I find that pretty fascinating from the semantic-programming standpoint; in a certain sense, each set of solutions encodes the same thing, the same meaning.

The link goes to "open a window"; Perl has five ways of doing it, depending on the GUI framework you're using, and I'd like to explore that parallelism in Decl.

Telescopic text

So look here: telescopictext.com. This is a simple story: "I made tea." Click on highlighted words for more detail. A lot more detail. And yet the overall narrative remains that the author made tea. [More detail at telescopictext.org, including a toolset!]

This is more or less what I'm saying about semantic programming. Looking at the high-level specification for a particular action, we see "Make tea". When we consider that action in more detail, we resolve further specifications that were previously invisible. Like varying f-stops that the eye cannot perceive, the mind elides these vast gulfs of detail in order to make sense of the world.

This process is nearly imperceptible to us. As programmers, we're familiar with it - it's the reason for Hoftstader's Law, if nothing else - but programming languages don't take it into consideration (except insofar as some are higher-level, and of course LISP can be made to do some of this, with its fancy macro system to hide cruft at will).

Maybe "telescopic code"?

Data modeling

So here's a thoughtful post about why relational databases aren't always the right thing to do. TL;DR - Oracle's standard order management model has 126 tables, and surely that's sometimes overkill.

This is exactly what I mean with the notion of semantic programming. Semantic structures have sliding levels of detail, and the actual number of tables or location of things in RDBMS or NoSQL or whatever should have no effect whatsoever on the actual semantics of the problem domain.

So: food for thought. It's a good article.