Impulse Tracker open-sourced

Impulse Tracker was an influential piece of software in the 90's electronic music scene (and is doubtless still influential today, but isn't used as much, naturally). Its author just open-sourced the whole thing. It would be an interesting code understanding/exegesis target.

Fuzz testing

Fuzz testing is throwing randomly perturbed inputs at a given piece of software to see what breaks. I was entirely unaware of the state of the art of fuzz testing, though. afl-fuzz is a tool that watches the execution traces of its target while reacting to the input. It can synthesize a legal bash script from nothing, by watching how bash reacts to different byte sequences. (And it discovered that bash vulnerability that made everybody upgrade last month or so.)

To which I can only say: holy Toledo. I have seen the future.

It discovered CDATA sections in XML. Randomly. Against the expectations of its author, who says, "it's an example of the fuzzer defiantly and secretly working around one of its intentional and explicit design limitations". Evolution is weird. Almost magic.

It made a legal JPG from the seed string "Hello" - again, by noticing different execution paths taken in response to different bytes of input. Here are some test sets for graphics. Interesting stuff.

Article up, and also: exegesis and code analysis/understanding

My first article kinda-sorta based on an exegetical approach (of my own prototype code) is up on the Vivtek site, and the six-day code rush to write code to build the article from a note database has really whetted my appetite for more of the same. There are all kinds of exegetical efforts I want to make, getting into code reading in a big way.

So I trolled around Google for salient things. Here's a list of interesting things.

• TipsForReadingCode at c2.com was quite helpful as a set of ... well, tips for reading code.
• Code comprehension tools at grok2.com; the vast majority of this class of tool is closed-source and rather expensive. I think this is largely because large codebases are not typical for open source projects, but rather enterprise code, and enterprises have the money to pay for expensive tools. That's my reading, anyway. But:
• cscope is a venerable tool for static analysis of C code, anyway, and has been perverted to handle other large-scale grep-like analyses of large numbers of files. The tool itself may or may not be something I'm interested in, but its approach is probably pretty valid.
• cflow is another flow dependency analysis tool, also open source.
• Perl is, as always, a special case.
• Well, what about static code analysis in general? Here's another list of tools.
• There's a clang-based analyzer.
• This moribund project on code "aspects" looks fascinating.
• Finally, a book on pattern-based OO refactoring, which also looks pretty fascinating.

So, as always, lots of people are doing things tangentially related to what I want to do, but nothing is 100% there - because what I want to do is an extract-then-literate-programming kind of thing. We'll see how much sense this approach makes. Wish me luck.

Code search engines

An HNN thread listing lots of code search engines.

Post-mortem (literally) of the Knight Capital bug

It was a process error. Worth reading!

Machine learning to find memory leaks

This is pretty cool.

GPG audit

GPG is, of course, an important piece of software in the security world. It's kinda crufty and old. It probably needs an audit. Tptacek on HNN says more than that, it needs some decent code documentation, hence my idea of an exegesis, a deliteralization of sorts.

Anyway, multilevel code understanding and presentation.

cdecl.org

A translator between English descriptions of C type declarations, and the type declarations. Pretty fascinating!

Brilliant vs. insane code

Here's an odd little ditty musing about a line of Python Stavros Korokisthakis (perhaps HNN's StavrosK?) ran across:

def GetContourPoints(self, array):
    """Parses an array of xyz points and returns a array of point dictionaries."""

    return zip(*[iter(array)]*3)

Hmm. Like it says on the label, it takes an iterable of points and returns an iterable of triples in order. But as Stavros notes, it's not at all obvious how it does that. You have to reason your way through it.

It's clockwork, and quite clever - and not the way people think (well, except insofar as people build clockwork and this Python in order to do things like this of course). In terms of code understanding this code is not self-documenting in any way. To determine programmer intent, we have to simulate what it does and see why it does that.

It's kind of like a syntactic artifact of a semantic reasoning process, one that we can recover (hopefully!) with careful reasoning. But the original reasoning is gone.

Interesting.

Anatomy of a buggy random-number generator

Post-mortems are always neat.

Code reviews

Occasionally, large, historically relevant pieces of software get released to open source (games, mostly), and code reviews are then done. Prince of Persia is one. Doom 3 is another. Both reviews by Fabien Sanglard. Good stuff. This is kind of where I want to go with the concept of an exegesis.

GitHub's Linguist

Linguist is what GitHub uses to figure out the (programming) language in a given file.

Code comprehensibility

Here's a paper asking: what makes code hard to understand?

Good question...

Moose for software analysis

Aside from Moose for Perl object-oriented programming, there is also a Moose for the analysis of software. There's a book as well. Moose appears to be about the model-based facilitation of software engineering, especially in the research arena. It's Swiss, meaning that there is this FrancoGerman assumption of underlying ontologies I find nearly incomprehensible, but they appear to be doing a lot of things I want to understand as well.

So I should come back to it. Sometime when I can grok what meta-meta-modelling is supposed to be about.

CommitQ

We want to retire the plain old generic-text diff and replace it with a programming-language aware semantic diff tool.

Sounds good!

Raven Software open-sources code for Star Wars games

This is always cool stuff: a couple of games got open-sourced after Disney's acquisition of Lucas.

I'm posting this under "open source target", but my understanding of what that means seems to have drifted a little. Originally I considered open source targets to be interesting things that could be done for programming using declarative styles and semantic programming. Now I find myself also including things that could be used as existing code for the purpose of exegesis and code understanding.

This is kinda both.

Note 2013-10-10: I just now noticed I didn't link to the post in question, but it doesn't matter. Raven apparently undecided to release, and all trace of their code is gone from SourceForge. That irks me, but there doesn't seem to be anything I can do about it.

Dictionary of Algorithms and Data Structures

Semantic gold mine! A long-running personal project at NIST cataloging data structures and the algorithms that use them.

Gameboy emulation

For Pokemon play, one uses an emulator on the PC or other computer (or, you know, you buy an actual Gameboy, but I'm assuming you're more interesting than that).  Mostly that's VBA or VBA-M, although there are others. On Win64, VBA-M is not working for me with Pokemon Emerald, so I'm using a 32-bit VBA, but the source tree for VBA-M is on SourceForge here.

Clearly, part of this is the emulator itself and part is the UI and associated tools, so we've got to tease those threads apart. But I'm most interested in how the ROM itself works (i.e. its file structure and how all that stuff is defined). It appears to be programmed on this virtual machine in a bytecode; how does that work, and how can we pull it apart to build a new ROM or modified one? I'm pretty sure the emulator code itself is going to tell us that, but the documentation is horrible, all read-the-code-Luke with a few cryptic comments for things the authors found tricky or unexpected, I'm assuming bug fixes mostly.

Anyway, the reason Pokemon is suddenly featuring on this heretofore more general blog is that my son has a truly fascinating idea for a programming project involving emulated Pokemon. More here later if it proves feasible to do what he wants to do. But in the meantime we gotta understand and clean up this codebase, so it's code understanding to the forefront!

Searching Github for SQL injection vulnerabilities

This kind of stuff always makes me drool.

Disassemblers and decompilers

I bookmarked this years ago (the link changed in the meantime) and here it is: a list of disassemblers. I wish I had more time in my days.