Friday, January 16, 2015
Tuesday, January 13, 2015
What happened to Old School NLP?
Answer: statistical methods work OK and they're way easier. 'Struth!
Sunday, January 11, 2015
Flow-Based Programming
Flow-based programming is not a particularly new idea, but it's getting some more attention lately.
Exegesis, literate programming, and Decl
The basic outline of Decl's new syntax parser is complete and passing tests, and as always when a milestone is reached and I look around the corner at what's next, I'm a little overwhelmed. It makes me philosophical.
My initial Marpa NLP article was actually a very simple exegetical analysis of a prototype script I wrote myself, and as usual when doing a first stab, I ended up writing a lot of special-case code and syntax to handle things, representing a technical debt that in such situations nearly always strangles its host. Fortunately, I iterated quickly this time, taking the insights from that and putting them into a new Decl-based plan.
Slowly, I'm feeling my way towards a Decl-based system for literate-style transformational exegesis, one that I hope will eventually encompass everything this blog has been about.
The advantage to basing everything on Decl is that parsing is done. I can now parse a very rich, informationally dense data structure that is designed right from the start to group things in more or less the same way natural language does. It's easily extended and configured, easily indexed - in short, it's a way of taking notes about program structure and using them to evolve a program.
So that's where I'm going. Very slowly.
My initial Marpa NLP article was actually a very simple exegetical analysis of a prototype script I wrote myself, and as usual when doing a first stab, I ended up writing a lot of special-case code and syntax to handle things, representing a technical debt that in such situations nearly always strangles its host. Fortunately, I iterated quickly this time, taking the insights from that and putting them into a new Decl-based plan.
Slowly, I'm feeling my way towards a Decl-based system for literate-style transformational exegesis, one that I hope will eventually encompass everything this blog has been about.
The advantage to basing everything on Decl is that parsing is done. I can now parse a very rich, informationally dense data structure that is designed right from the start to group things in more or less the same way natural language does. It's easily extended and configured, easily indexed - in short, it's a way of taking notes about program structure and using them to evolve a program.
So that's where I'm going. Very slowly.
Sunday, January 4, 2015
Some thoughts on decisions and other things in workflow
So the general areas of functionality of the new wftk (pretty much the same as the old wftk) are: data, actions, events, decisions, agents, workflow, notifications, and schedules. That doesn't match up all that well with the rough functionality outline taken from the chapter headings of the old book, but these are kind of the things that make up business processes.
The data organizer is a nice piece of functionality that breaks off conveniently, defining and naming data in terms of stores, indexes, and documents. My declarative language Decl is taking shape as a document-based language (its syntax and semantics are governed by the metadata of the document in question), and so it's becoming clear to me that a business action is also a document-based kind of thing, which can be expressed in Decl, use various resources, consume input documents, and create a result document (as enactment) and a series of attached output documents.
Events are essentially input queues. From a technical standpoint they're not terribly interesting, but events drive the machinery of a process. An event source pushes flow.
Which brings us to decisions.
A decision in a workflow context can be as simple as a logical test of variables, or as complex as an entire subprocess that involves not only calculation but even the execution of experiments that cost money and the input of human decision-makers. Decisions are arguably one of the most important aspects of a business. They need to be given careful thought.
There are a number of different ways to present decision structures, including simply as logical combinatorics, tables, and flowcharts, and they can be learned as decision trees (categorization is a generalization of decision - a non-binary decision, effectively).
I don't think they're going to rise to the level of being a separate module like the data organizer, but I think it's still valid to put decisions into a separate chapter of discourse, as it were.
The data organizer is a nice piece of functionality that breaks off conveniently, defining and naming data in terms of stores, indexes, and documents. My declarative language Decl is taking shape as a document-based language (its syntax and semantics are governed by the metadata of the document in question), and so it's becoming clear to me that a business action is also a document-based kind of thing, which can be expressed in Decl, use various resources, consume input documents, and create a result document (as enactment) and a series of attached output documents.
Events are essentially input queues. From a technical standpoint they're not terribly interesting, but events drive the machinery of a process. An event source pushes flow.
Which brings us to decisions.
A decision in a workflow context can be as simple as a logical test of variables, or as complex as an entire subprocess that involves not only calculation but even the execution of experiments that cost money and the input of human decision-makers. Decisions are arguably one of the most important aspects of a business. They need to be given careful thought.
There are a number of different ways to present decision structures, including simply as logical combinatorics, tables, and flowcharts, and they can be learned as decision trees (categorization is a generalization of decision - a non-binary decision, effectively).
I don't think they're going to rise to the level of being a separate module like the data organizer, but I think it's still valid to put decisions into a separate chapter of discourse, as it were.
Saturday, January 3, 2015
Friday, January 2, 2015
How vulnerable are Quora posts to writing style analysis?
Verdict: pretty vulnerable. The tools used are of interest.
OCCRP
Organized Crime and Corruption Reporting Project (OCCRP) is a data-journalism kind of thing - mostly journalism, but with some data aspects I find interesting. They seem to have collaborated on an investigative tool, the Investigative Dashboard, which does seem pretty data-oriented.
Sunday, December 28, 2014
Thursday, December 25, 2014
Impulse Tracker open-sourced
Impulse Tracker was an influential piece of software in the 90's electronic music scene (and is doubtless still influential today, but isn't used as much, naturally). Its author just open-sourced the whole thing. It would be an interesting code understanding/exegesis target.
Tuesday, December 23, 2014
Generating mazes and dungeons
Truly fascinating and well-written article on maze/dungeon generation, with interactive JS illustrations. This is the kind of writing about programming I love. [hnn]
Sunday, December 14, 2014
Really teensy ELF executables
Binary data parsing - well, reverse binary data parsing... or something. Cool article, anyway.
Remote controlling browsers
So a significant portion of "business things" that a workflow/business process system has to handle consist of things done in browsers.
Sure, sure, you can automate Web things effectively with a bot, but sometimes what you're controlling is a JavaScript application that, honestly, will only run well in an actual browser. It's a pain, but there you go.
One avenue has traditionally been IEMech (moribund at the moment due to OLE/COM complexity that has changed in later versions), but there are also different remote control solutions available for Firefox and Chrome.
Firefox's FF-Remote-Control is a great little add-on that works quite well. For the time being, therefore, Firefox is going to be my automated browser of choice even though Chrome is currently my actual browser.
For Chrome, the situation is somewhat different, as Chrome's security model doesn't permit an add-on to listen on a port. As a result, the Chromi extension hits a running server on localhost (the Chromix remote-control system). It doesn't seem as flexible as FF-Remote-Control, but I haven't spent much time with it yet.
So: for now, Firefox.
Sure, sure, you can automate Web things effectively with a bot, but sometimes what you're controlling is a JavaScript application that, honestly, will only run well in an actual browser. It's a pain, but there you go.
One avenue has traditionally been IEMech (moribund at the moment due to OLE/COM complexity that has changed in later versions), but there are also different remote control solutions available for Firefox and Chrome.
Firefox's FF-Remote-Control is a great little add-on that works quite well. For the time being, therefore, Firefox is going to be my automated browser of choice even though Chrome is currently my actual browser.
For Chrome, the situation is somewhat different, as Chrome's security model doesn't permit an add-on to listen on a port. As a result, the Chromi extension hits a running server on localhost (the Chromix remote-control system). It doesn't seem as flexible as FF-Remote-Control, but I haven't spent much time with it yet.
So: for now, Firefox.
Sunday, December 7, 2014
Monday, December 1, 2014
Fuzz testing
Fuzz testing is throwing randomly perturbed inputs at a given piece of software to see what breaks. I was entirely unaware of the state of the art of fuzz testing, though. afl-fuzz is a tool that watches the execution traces of its target while reacting to the input. It can synthesize a legal bash script from nothing, by watching how bash reacts to different byte sequences. (And it discovered that bash vulnerability that made everybody upgrade last month or so.)
To which I can only say: holy Toledo. I have seen the future.
It discovered CDATA sections in XML. Randomly. Against the expectations of its author, who says, "it's an example of the fuzzer defiantly and secretly working around one of its intentional and explicit design limitations". Evolution is weird. Almost magic.
It made a legal JPG from the seed string "Hello" - again, by noticing different execution paths taken in response to different bytes of input. Here are some test sets for graphics. Interesting stuff.
To which I can only say: holy Toledo. I have seen the future.
It discovered CDATA sections in XML. Randomly. Against the expectations of its author, who says, "it's an example of the fuzzer defiantly and secretly working around one of its intentional and explicit design limitations". Evolution is weird. Almost magic.
It made a legal JPG from the seed string "Hello" - again, by noticing different execution paths taken in response to different bytes of input. Here are some test sets for graphics. Interesting stuff.
Breadcrumb
Just a note on progress, for later calibration of my past timelines: I've got Decl 2.0 parsing working rather nicely (still a few loose ends) and I have an excellent plan for integrating it into a notes application for literate programming. I think the combination will end up being something pretty powerful.
My first real target for transformational exegesis using this kind of tool will be Melanie Mitchell's Copycat - I want to get it converted to Clojure so it will run on something I actually own. But I'm also exploring various programming sequences and contests as a way of provoking thought about the actual writing of software.
Anyway, that's all pretty jargon-laden but it means something to me. At a later date I hope to circle back around and write about this stuff in more detail, but I'm not in that phase of the cycle at the moment.
My first real target for transformational exegesis using this kind of tool will be Melanie Mitchell's Copycat - I want to get it converted to Clojure so it will run on something I actually own. But I'm also exploring various programming sequences and contests as a way of provoking thought about the actual writing of software.
Anyway, that's all pretty jargon-laden but it means something to me. At a later date I hope to circle back around and write about this stuff in more detail, but I'm not in that phase of the cycle at the moment.
Subscribe to:
Posts (Atom)