Gherkin/behat

Gherkin is a language used for "behavior-driven development", in which a restricted subset of English (or some other natural language) is used to describe the behavior of some software. Behat then tests it using a domain model, which I find interesting. It's part of the PHP ecosystem.

The idea is that you've got a description of usage scenarios that you can agree on with non-technical users. I think that's pretty cool. But also you've got a higher-level description of the program, which arguably is something the programming system should use and/or have access to. The descriptions there are clues to the semantic domains you're interacting with, so analysis of them should result in a reasonable semantic model of the program.

Anyway. Worth thinking about.

Fuzz testing

Fuzz testing is throwing randomly perturbed inputs at a given piece of software to see what breaks. I was entirely unaware of the state of the art of fuzz testing, though. afl-fuzz is a tool that watches the execution traces of its target while reacting to the input. It can synthesize a legal bash script from nothing, by watching how bash reacts to different byte sequences. (And it discovered that bash vulnerability that made everybody upgrade last month or so.)

To which I can only say: holy Toledo. I have seen the future.

It discovered CDATA sections in XML. Randomly. Against the expectations of its author, who says, "it's an example of the fuzzer defiantly and secretly working around one of its intentional and explicit design limitations". Evolution is weird. Almost magic.

It made a legal JPG from the seed string "Hello" - again, by noticing different execution paths taken in response to different bytes of input. Here are some test sets for graphics. Interesting stuff.

Fake name generator

This is very cool - testing purposes, etc.

Huxley for graphical regression testing

This is a good idea (out of Facebook) - take snapshots while testing and compare them at the pixel level for regression.

Mocky

Mocky lets you build test responses for REST APIs. Or something. My brain is a little fried today.

Open Sauce: unlimited free testing for open-source projects

Interesting.  Interesting.

Software development and Romney's loss

A couple of good articles about the Romney campaign's snazzy new GOTV tool, Orca, which went down in a blaze of glory, much like the iconic picture of his blimp, but for entirely predictable reasons: insufficient testing, apparently in part because of a misplaced urge to centralization and secrecy, and way insufficient user training, along with some operational foulups (sending out incorrect passwords to users, etc.)  It was a fiasco in every sense of the term.

It was downright corporate.

Cucumber, and BDD: Behavior-Driven Development

So here's something I ran across (again) today, and really started thinking about what it means: behavior-driven development, BDD.  It's a philosophical refinement of test-driven development, in which you define test scenarios in, essentially, English using some keywords (Feature:, Scenario:, etc.).  This keyworded set of requirements is then used to define tests, and then you do the TDD thing to make all the tests work.

It's pretty cool.  It's been an outgrowth of the Ruby Agile community, and its latest incarnation there is Cucumber.  And yes, there is a Test::BDD::Cucumber on CPAN.

So here we have a very declarative approach to specifying the behavior of a program and using that to drive the development process.  I don't care for the somewhat clunky regexp-driven way you get from the plain text description to the executable code, but the overall shape of the development process is very promising.

Here's the Wiki page for BDD.  It points out that some of the notions in BDD come from domain-driven design.  And all of that needs to be condensed a little into some principles for Decl, perhaps.

Web testing

There is a hugely vast array of Web testing tools, even free ones. Seriously. BadBoy is an example I can't even remember finding. Somebody oughta do a feature comparison list.

Target domain: testing

Unit tests. Something. Testing is probably related to workflow. Yes, I know this is overly terse.

Article: the Pain of JSON API Testing

A weird little article about Ruby JSON API testing and why it's difficult with Cucumber, which is a testing system in Ruby. Might I say: this Cucumber makes me see why Ruby.