Excel and statistics

This spring I worked through an entire statistics class in Excel (well, mostly in Excel). This is mostly because I somehow never actually learned any statistics in college and still have to understand the concepts for translation, but also partly because I've done some other math-oriented stuff in Excel and wanted to learn more about that.

Here's the thing: like many Microsoft tools, Excel can be incredibly powerful - as long as you're using it the way its designers conceived. Go off the reservation and abruptly things get very difficult.

But it's worse. Turns out that specifically for statistics, Excel kind of sucks. Like, wrong-answer-level sucks. The recommendation of the article in the link: use R (or some commercial statistics package).

So here's an interesting idea: wouldn't it be interesting to build a spreadsheet tool that translates to R behind the scenes? Plus has somewhat better handling of tabular data? (That part drives me crazy in Excel: I have to know the shape of my data in advance.)

There might well be such a tool already. If not, it would be fun to build it.

Random walk in Python

An interesting article about the random walk hypothesis (quantitative finance) and an implementation in Python that passes the NIST tests for random algorithms. Very cool.

Stan, a probabilistic programming language

Wow. Stan is a "probabilistic programming language", apparently meaning that it is an engine for executing statistical models directly. My level of statistical naïveté being what it is, I can't actually even understand the manual very well, but it looks fascinating. Writing a set of tutorials for this would be a meaningful endeavor. As always, though, the HNN discussion is valuable, especially in regards to alternatives and books.

Statistics Done Wrong

What the title says. "The woefully complete guide." I'd be willing to bet it's not.

Econophysics

Again. I think. Maybe this isn't - it's really more of a statistics thing.

Statistical cheat sheet for programmers

Nice super-quick recap of the high points of statistics.

List of probability distributions

Statistics is cool.

J, R, and statistical programming

I'm not sure why statisticians are so enamored of single-letter names for their programming systems (makes Googling very difficult), but there you go.  I knew of R, of course, from the existence of CRAN alone, but today I found out about J.

One of J's original authors is Ken Iverson of APL fame.  And yeah, in terms of its compositionality and frank strangeness for imperative programmers like myself, it looks a lot like ... well, like a usable APL, really.  So how does it stack up against R?  Ask StackOverflow: the short answer is ... it's complicated.  R has fantastic third-party library support, but J can run R.  R is more verbose and therefore probably easier to maintain.  On the other hand, for data manipulation, J is succinct and expressive.  So it really depends on what you're doing, which is increasingly the case for programming languages these days.

J has a full-featured runtime environment and IDE, too.  This alone would make it interesting: J is built from the ground up to build things.  Things that will be used by people.  You can package a J app and use J's own installer to distribute your software.  There are GUI and graphics tools built into the language and its basic library system.

In this sense, it would be a great model for some hypothetical standalone Decl system.