Mail handling in Python

"Envelopes is a wrapper for Python’s email and smtplib modules. It aims to make working with outgoing e-mail in Python simple and fun."

Mailparser.io

Email data extraction as a service.

Gmail: consistent rendering of UI in email

This is a weird one; Google has defined a data schema for defining UI in an email, to be rendered in the mail client. Useful for workflow, I suppose, but it seems a bit, I dunno, hyperfocused.

Email message threading

Jamie Zawinski explains his email threading algorithm here, the one used in Netscape back in the day. I love reading his work.

Email bounce parsing and mapping of modules

The definitive CPAN module for extracting information from email bounces is Mail::DeliveryStatus::BounceParser, and it's still relatively actively maintained - just had an update adding a couple of cases in January.  There's only a couple of problems.  First, there are bugs listed that are years old and (to me) look relatively important - a bounce with multiple bad addresses doesn't get multiple addresses parsed out, that kind of thing.

Second, its report object subclasses Mail::Header.  So sure, that's not horrible, but still - I'd much rather have something that can look at an Email::Abstract object or just the headers of one (Email::Simple::Header) and extract an object that doesn't hook into an alternative mail ecology.

Third, some of the cases might be dubious.  I'd rather have something that uses some kind of tabular organization of filtration cases or something - this is wishy-washier, but it does contribute to my sense of unease in going with this module casually.

But the amount of information in this module is outstandingly valuable - it represents years of tinkering with bounce messages and trying to deal with the weird ones.  So I don't want to just start over, either, and if I port it into a different framework I'd like to be able to keep up with any further updates.

This is a common set of problems in reusable software development.  The source code/module level is not really a fantastic level of granularity for knowledge preservation - it's just better than anything else we've got yet in common use.

Another case I ran across was a translation tool called Anaphraseus - it's written as an open-source replacement for the TRADOS Word tools (essentially a work-alike) but it works only in Open Office.  I use Word, so if I want to use Anaphraseus I'd need to port it - but I'd like to be able to keep in synch with the official release because they do things that I don't think of.

In general I think of these situations as requiring a tool I think of as a "cross-parser"; they take a text in one high-level language and translate it into another, and maybe back.  They allow a continuous mapping of knowledge into two different expressions, in other words.

I need to research this general area of problems.  I'm sure it's old hat to somebody.

Specifically, though, this week, I'm proposing Email::Simple::BounceParser, which would mirror Mail::DeliveryStatus::BounceParser, perhaps through some kind of database representation in the middle.  I have no idea how that would work exactly.

In general, it would be nice to be able to define some kind of abstract module for a specific set of "knowledge" that would be crystallized as specific module instances.  This is essentially what parser parsers already do (Parse::RecDescent does exactly this); it would be nice to formalize this technique with a more visible system of expressing it.  Kind of a code template thing, I guess.

Followup to email

One thing that's actually quite important but is easy to forget is reliability.  When I lose contact with the mail server, I need to fail gracefully.  If I can't contact the server because the machine is offline, I need to take appropriate action.  Under no circumstances should things just crash or die.

There's probably a principled way of approaching this as a design goal, but I don't know precisely what it is.

Email workflow

So here's my current thinking about email handling.

• Incoming mail has to go through various filters and classifiers before humans ever see it.
• Bounces are first - they contain some information no matter what.
• Some filters are database-backed: assignment to customers, assignment to ongoing threads by response header (and thus to tasks).  Since a lot of what I want to do with email is actually workflow, assignment to tasks, projects, and task categories is important and that happens at this stage, mostly.
• Then there are automatic notifications - logs and heartbeats.  That stuff can go into an automatic-processing bucket without much further ado.
• Next comes categorization of undifferentiated stuff.  The first line of defense is spam filtration. Distributed solutions are useful here because it's best to triangulate over a wider recipient base. (Despammed comes in here - and more on Despammed in a bit.)
• Ham can be autofiltered as well, although I'm not as convinced that's terribly useful.  It's an active area of research, though (see a list below).
• Finally, you end up with a little dribble of personal mail and "things that could be new tasks" or related to old ones.
• Except for that last stage, all this happens automatically in the background.
• Once categorized, each incoming mail triggers a system response: this can range from simply marking or putting into a folder, to kicking off an arbitrary program for data storage or statistics, or notifying the user.  In other words, workflow.
• Task mail is intended to be relatively small in scale; when a task is finished, its mail is archived with it, possibly leaving a trace in an index for a while.
• Known task messages have their message IDs stored for detection of follow-ups.
• Known task contacts have their email IDs or domains marked for the same reason.
• Longer-term projects can be archived monthly or when the number of messages gets unmanageable.
• Personal conversations are treated as projects in this sense.
• Responses (outgoing mail) are stored with the conversations responded to.  Gmail does this and it really makes sense.
• Document management of attachments happens in there somewhere, maybe even organization into some kind of versioned track (identical attachments can at least certainly be stored singly and noted as identical or something).

And again, all that could happen on the server or on your machine, all using an IMAP or other client or just autoinvoked upon receipt.  And in fact it might be rather useful to have some subset of this running at Despammed and get serious about that poor old thing.

A couple of promising links I found searching on "email categorization":

• http://people.cs.umass.edu/~ronb/papers/email.pdf looks like a great place to start with machine learning of email categories.
• Implicit Mailroom seems to do a lot of the above, in the Microsoft Exchange context.  Certainly a good idea to glom onto their feature list, anyway.

Anyway, the client is more or less independent of all that and could be anything.  I'm going on the assumption that client-facing folders are going to be virtual (that is, tag index instead of a physical folder) and stored with Email::Store.  Virtual, because right now moving archived folders back and forth to reflect current activity levels of customers takes forever and really is backup-unfriendly.  Thunderbird's method of email management just isn't really cutting it for me any more.  (Gmail's even worse - even apart from the fact that Google is reading my mail.)

So there's my line in the sand.  That's what I want in an email client engine.  Kind of an assistant in a box.

Bounce processing

This bubbled up on my random post bar on the right, and deserves a little reinforcement: MRjob for Postfix bounce processing.

Since I'm looking at email again lately, this was good timing: an autohandler for postmaster bounce notices would be an incredible boon to me, and honestly shouldn't be difficult to write.  So ... I should write it.

A clever rethinking of email's purpose

Here's a pretty cool article written by a guy who's reconceptualized email as being a way to put something on his queue.  You get an autoreply indicating his current probable response time with alternative ways to get in touch with him.

Generalize that to workflow and it makes perfect sense to me.  I like it!

Some IMAP links

IMAP is weird, but I'm starting to get used to it.  I have a bunch of promising links here (mostly Perl-centric).

• Quick overview of IMAP
• Sample IMAP session
• imaputils
• IMAP client example code
• PerlMonks
• Example-code.com?  Interesting - but all examples for one company's libraries...  Still, worth looking at what they have examples for.
• A DeveloperWorks article

IMAP architectural patterns

Here's an interesting concept in terms of architectural patterns: synchronization between multiple IMAP servers with OfflineIMAP.  Again, yeah, structural database application, yadda yadda.

Hmm. Zapier to move data between dozens of Web apps.

IMAP

I'm an idiot.

No, seriously, this is a lesson I just keep learning.  I've been working with email - not as a user, but as a programmer - since 1999, and I've never really taken the trouble to look closely at the IMAP specification.  And because I failed to do that, I've never really fully appreciated just what a perfect synchronization protocol it really is.

Need an email assistant?  Use IMAP.  You can run your usual email client against the server and have a separate assistant application watching the same server.  Delete a mail on one, it gets deleted on the other on the next synch.  You can add messages to the Inbox, even!

For a couple of years, I've toyed with automating Thunderbird to do this kind of assistant task, but automating Thunderbird is needlessly complex and I've just never taken the time to try it.  Now I don't have to!  I can continue to use Thunderbird against my mail server while writing a Perl assistant based on Mail::IMAPClient - problem solved!

So.

I'm an idiot.

Mandrill

This is a neat tool by the folks at MailChimp - handles incoming and outgoing mail and uses Webhooks for everything.  Mandrill.

Openera

Automatically save, organize, and back up files and email attachments.

Testing mail

Nice framework for handling mail in a test environment.  In Rails.

Best practices in email server management, 2012 edition

Best practices are our friends!

Target app: Boomerang

Boomerang is an add-on for Gmail that lets you manage things on a schedule.  You can hide things from your Inbox until a later date, send messages on a schedule, track whether you've gotten a response to an outgoing message, and so on.

Stuff any mail client should actually be able to do.  We've forgotten email; it's been static for 15 years or so.  I blame the damn spammers.

Claws mail

Another possibility for a mail client front end.  I can't tell how automatable it is, but at least it provides a nice list of features to strive for should I want to come up with a reasonable Perl-based mail client.

Somebody else wants a new approach to email

Interesting take on an email client for 2012.

Email::Simple::Markdown

Email::Simple::Markdown, a slick way of generating multipart MIME mail without the cruft of Email::MIME.