Twitter DM Email Parser Script

Twitter DM Email Parser Script00:22:15

1
Aug2

Since I am now my own sysadmin for the various servers I have running, I have forced myself to learn how to setup and administer mailservers on linux boxes. Now that I have that under control, I can also write scripts to parse and interact with incoming/outgoing email. My first such foray into playing with email scripts was to create a Twitter DM parser which strips incoming Direct Message emails from Twitter and re-formats them down to the bare essentials and forwards the resultant information back to my main email address. After that was working, I had the idea to give myself the ability to reply to these emails in order to send a reply DM back to the original sender. This system turns out to be really handy, thus turning Direct Messages into another form of email.

Screenshots and more details follow the jump if you are interested.

It used to be that all Twitter-generated emails were plain text. No HTML markup, no images, just text. All was right with the world. I’m not sure what it is, but I just believe that all email should be plain text. HTML is for the web (this is a poser since I read my email in GMail which is itself a web-based email client…) Outside of GMail I like to use pine (now alpine) which probably explains a lot. Anyway, here is a screen shot of an old format Twitter DM email that I found (note the date):

Some time in early 2009, the Twitter-generated emails changed to HTML formatting:

The whole message of the DM is hello, there jazzychad! which is all of about 23 characters. In order to deliver that Direct Message to my inbox, the email itself is a couple of kilobytes big. Here is the source:

You might say, “A couple of kilobytes? Who cares! Plus, it looks pretty.”

You may be right, except that I am not always at my computer (a shame, I know) and might receive email on my phone (which still uses Edge) and so any extra unneeded bytes (especially the 3 extra images that load) are manifested in extra download time waiting for an email to load:

Whether receiving the Direct Message email on my computer or my phone, if I want to respond I have to leave the email client and use another program/site/twitter client to create and send a reply DM. On my phone this is cumbersome to say the least; on the computer it’s not so bad since I usually have another twitter site/client open, but it does break the flow of reading/responding to my emails. I do a lot of correspondence over DM (maybe more than I should), and the time spent bouncing back and forth adds up.

So, how do I fix this problem? Write a script, of course!

I have setup an email filter which automatically forwards Direct Message emails to a special email account on one of my servers where it then is processed by a PHP script. The original DM email is archived and my script parses the Direct Message email for the important bits (highlighted by red boxes in the above email source screenshot), creates a new email and sends it back to my main email address. The resultant email looks something like this:

Just the facts, ma’am. The source for the email, while not tiny (because of the email headers) is still much smaller than the HTML-ified original:

The email gets straight to the point and also loads quickly on my phone (especially without images).

This was great until I realized that I still had to bounce to another application to reply! What if I could just reply to the email itself and have it deliver the message as a DM to the original sender?

I added a bit more logic to my PHP script so that it knows the difference between an incoming DM and a reply DM and can take action accordingly. Now I can just click “Reply” in the email and send a DM. I have gotten to the point where I know what 140 characters “feels” like (sad, but true), so I’m not worried about going over the limit (though I think the DM limit is unofficially 255 characters).

I click “Send” - et voilà! The reply is delivered.

An added bonus is that I now have a conversation thread saved in my email. As of this writing, there is no way to see the link between DMs on the Twitter site or through the API like there is to see the link between tweets and their replies. Being able to reply to an email gives me context when I need to go back and look at a conversation later.

The one thing I have yet to add to the email script is the ability to originate a DM to somebody. That is, send a DM without replying to somebody who DM’d me first. This would be easy enough to add, I just need to come up with some sort of subject line format which would tell my script to just DM this person. For now, the DM mechanism on TweetGrid allows me to do this easily enough when I’m at my computer. Being able to originate DMs by email would be especially handy on my phone.

Is the script perfect? No, probably not, but it works for me. If you’d like to have a look and play around with it, here is the source code. It requires the PHP Pear mimeDecode module.

If you have a mailserver and it supports a ~/.forward file, you can put the following line in it to pipe the email to your script: