6 lesser-known Python data handling libraries

There is just a lot of neat stuff in Pyworld. Data handling, for instance.

The data engineering ecosystem

Interesting. Very interesting, actually. Sort of an architectural mapping approach.

BayesDB

This is cool - a Bayesian query layer on top of tabular data, I think.

Dynatables

Here's a neat jsQuery-based site component: the dynatable.

ETL

Saw a post on the Perl jobs list yesterday for an ETL expert. Apparently this is the name Big Data people give to batch data transformations: Extract, Transform, Load.

What I just call "indexing" in the greater sense.

Cool.

UnQLite

UnQLite is kind of the same thing as SQLite but for NoSQL-type document-based architectures.

Database reconciliation

So a thing that comes up a lot in working with data is the need for reconciliation, where you have two datasources and you need to match them up and see whether everything in one is in the other, and vice versa. (This is overall a part of the whole data quality issue.) So here I am, today, seeing what's out there - the answer is, unless you're buying SAP, very little (there is a Perl module, kind of, for doing table comparison) - and lo! the process of data reconciliation was patented in 1999 by Qwest; later, JPMorgan Chase inherited it and later sold it off.

Can you believe that?  Can you honestly believe that somebody patented the very idea of finding records in one list and matching them with records in another list?  This is the kind of nonsense up with which we really should not put.

Data sharing

Caitlin Rivers deplores the current state of the art in data sharing, and offers some tips. I wonder how much could be done with some kind of semantic "data presentation understanding tool".

Data table editor for jQuery

And another nice drop-in component: a data table viewer/editor built on jQuery.

Flat file module in Perl

This is kind of neat.  I love wheel reinvention - you usually learn something.

Ember table by Addepar

Very good data table browser!  Exactly what I need for ... everything!

Parse's new data browser

This is cool - I mean, it's basically Access (ha!) but still very cool.

I've been thinking again about UI programming.  I need some kind of higher-level UI specification language that can describe the way the UI acts in a platform-independent way.  Then translators into the different UI platforms.

Probably not so impressive, since half the UI platforms already have some kind of similar notion.  Still. From a semantic standpoint, this seems like where I need to be going.

DataWrangler

A data wrangling tool from Stanford.  Data wrangling itself is actually a topic pretty near and dear to my heart.

Druid: open-source real-time analytics store

Druid has just been open sourced.  It appears to be a competitor for Hadoop.

Data handling

I'm lumping a bunch of stuff together into the general rubric of "data handling" that is really kind of poorly defined.  But no matter how poorly defined it is, it appears that people keep writing about it, and a a not insignificant portion of many practical machine learning books is devoted to it.

Anyway, it basically involves all the moving around of files and databases that you wave your hands at, and end up being most of the fiddly work of any practical project.  I'd like to set aside a little time to think about how to do it right (kind of a best-practices semantic domain, as usual).  And I'm getting the occasional link about it.

My fileset module has to do with data handling (as a way to define files that should be subjected to an action).  The Data::Table module is a handy in-memory way to cut off blobs of relational data and manipulate them in handy ways.  Excel is a good place to stash tables like this in a file.  And so on.

A lot of workflow involves "data handling" - grouping things into documents and that kind of thing.  Taking items from this document and summarizing them into that one.  The "bizop" semantic-level language/view I've been musing on is largely a matter of data handling.

DataNitro?

A snippet about options pricing in Excel with Python and DataNitro.