For a fair amount of years now, I've been using Mutt as my primary email client. It's a very nice text-based email client that is permanently running on my server (named drsnuggles). This allows me to connect to my server from anywhere, connect to the running Screen and always get exactly the same, highly customized, mail interface (some people will say that their webmail interfaces will allow for exactly the same, but in my experience webmail is always clumsy and slow compared to a decent, well-customized text-based client when processing a couple of hundreds of emails per day).

Attachment troubles

So I like my mutt / screen setup. However, there has been one particular issue that didn't work quite as efficient: attachments. Whenever I wanted to open an email attachment, I needed to save the attachment within mutt to some place that was shared through HTTP, make the file world-readable (mutt insists on not making your saved attachments world-readable), browse to some url on the local machine and open the attachment. Not quite efficient at all.

Yesterday evening I was finally fed up with all this stuff and decided to hack up a fix. It took a bit of fiddling to get it right (and I had nearly started to spend the day coding a patch for mutt when the folks in #mutt pointed out an easier, albeit less elegant "solution"), but it works now: I can select an attachment in mutt, press "x" and it gets opened on my laptop. Coolness.

How does it work?

Just in case anyone else is interested in this solution, I'll document how it works. The big picture is as follows: When I press "x", a mutt macro is invoked that copies the attachment to my laptop and opens the attachment there. There's a bunch of different steps involved here, which I'll detail below.

How to connect to the laptop?

The first obvious challenge is connecting back to my laptop. Since my laptop moves around a lot and is usually behind a NAT router, drsnuggles can't just connect to a (fixed) IP address.

I've solved this problem the other way around: When I connect from Xanthe (my laptop) to drsnuggles, I set up ssh to do port forwarding. This could work using the following command (my actual setup is a bit more complicated, more on that later).

matthijs@xante:~$ ssh -R 2222:localhost:22 drsnuggles

This tells ssh to open up port 2222 on the remote end (on drsnuggles) and if any connections are made to that port, forward them to localhost:22 on the local end (Xanthe). This allows drsnuggles to connect to Xanthe by connecting to this port 2222.

Alternatively, you can use the RemoteForward directive in your ~/.ssh/config file instead of passing the -R option.

Now, on drsnuggles I added the following bit to my .ssh/config file:

Host xanthe xanthe.stderr.nl
    HostKeyAlias xanthe
    # Incoming ssh connections from xanthe should forward this port back
    # to xanthe, so we can connect back.
    Hostname localhost
    Port 2222

This allows me to connect to Xanthe using the special hostname "xanthe", without having to specify the port number everytime:

matthijs@drsnuggles:~$ ssh xanthe
Linux xanthe.stderr.nl 2.6.34 #8 Sun Jul 4 20:34:08 CEST 2010 x86_64
matthijs@xanthe:~$

It also allows me to copy files to xanthe using scp:

matthijs@drsnuggles:~$ scp /etc/hosts xanthe:
host                                 100%  280     0.3KB/s   00:00    
matthijs@drsnuggles:~$

How to connect to drsnuggles easily?

This part is strictly not related to the attachment-opening, but these are just a few files I'm using to connect to my screen running on drsnuggles. The first script, called ~/bin/inet on Xanthe, allows me to just type "inet" in a terminal to get an SSH connection to drsnuggles with the screen attached. If no connection can be made (or an existing connection fails), the script waits for a press of the enter key and then tries to connect again.

#!/bin/bash

# Open up remote port 2222 and forward to our local ssh port. This
# allows the remote host to connect back to us, even without knowing our
# address.
FORWARD="-R 2222:localhost:22"
# -t forces allocating a pseudo-terminal, so screen can attach.
# ~/bin/attach attaches the proper screen, creating it if it's not there
# yet.
while ! ssh $FORWARD -t drsnuggles ~/bin/attach; do
    if ! read; then # Wait for an enter so error can be read
        echo "Read returned $?. Exiting"
        exit 1;
    fi
    $fixssh
done

Note that this connects to "drsnuggles", which is defined in my ~/.ssh/config as shown in the next section.

After the ssh connection is created, the ~/bin/attach command is run on drsnuggles. This script attaches to the running screen instance, creating it if it's not running yet:

#!/bin/sh

# Source our profile, since ssh doesn't do this when a remote command
# (such as attach) is specified. We need this for the ssh-agent socket
# to be setup properly.
. ~/.bash_profile

exec screen -c ~/.screen/inet -d -R inet

This script tells screen to detach the running screen if it's attached elsewhere (-d) and then attach it here (-R). If the screen is not running yet, it's created using the ~/.screen/inet config file. This file is the final piece of this part of the puzzle:

vbell off
screen 0
screen -t mutt 1 mutt
screen -t TODO 2 vim docs/TODO
screen -t irssi 3 irssi

How to prevent a password prompt every time?

As you might have noticed in the command outputs above, there is no password prompt. It would of course be very inconvenient if I had to type in my Xanthe password for every attachment I want to open (even twice, since we need to copy the file and run a command).

To prevent this, I'm using key-based authentication, together with an ssh-agent running on Xanthe. I won't go into detail on how this works or how to set it up, you should look at this howto for details.

Two things are important here, though. First, I make sure that agent forwarding is enabled on the connection to drsnuggles. This allows the ssh to connect from drsnuggles back to Xanthe to use the key from my ssh-agent for logins. This can be done by passing the -A option to ssh, or by putting ForwardAgent yes in ~/.ssh/config (on Xanthe). I'm doing the latter, since I want to have agent forwarding enabled for all connections to drsnuggles by default:

Host drsnuggles, drsnuggles.stderr.nl
    Hostname drsnuggles.stderr.nl
    ForwardAgent yes

Second, Xanthe must allow for login with an SSH key instead of a password. This is achieved simply by puting the public key in the ~/.ssh/authorized_keys file on Xanthe.

Furthermore, something else is needed to make the forwarded agent work inside a running screen. When I connect to drsnuggles with agent forwarding enabled, ssh sets the $SSH_AUTH_SOCK environment variable to point at a socket somehwere in /tmp. Any ssh command running from drsnuggles uses this information to connect to the the SSH agent through that socket.

Now, every time I connect to drsnuggles, a new socket is created with a different filename. So when I connect, start a screen, detach, reconnect and reattach the screen, the $SSH_AUTH_SOCK variable inside the screens will still point to the old (no longer existing) socket. To make the agent work, we need to point the $SSH_AUTH_SOCK variable inside screen to the new socket on every reconnect.

Before, I used to have some complicated setup that tracked all current forwarded agents and a fixssh script that updated the $SSH_AUTH_SOCK variable every time just before running an ssh command. I think it was inspired by this howto by Sam Row. Anyway, this works, but is a bit complicated and cumbersome.

I've now created a much simpler solution: Just set the $SSH_AUTH_SOCK to always point to ~/.ssh/auth_sock, which is a symlink to the most recently forwarded agent socket. On every SSH login, we replace the symlink to keep it current. So far, it seems to work just fine. The following snippet from my ~/.bashrc takes care of all this:

HOST=`hostname -s`
if [ -n "$SSH_AUTH_SOCK" ]; then
    # If we ssh in, redirect ~/.ssh/auth_sock to the
    # (forwarded) ssh agent socket. Keep a different
    # forwarded socket for each host, since different
    # vservers do not share the same /tmp (and moving the
    # socket to my homedir seems to break the socket...)
    rm -f ~/.ssh/auth_sock-$HOST
    ln -s $SSH_AUTH_SOCK ~/.ssh/auth_sock-$HOST
    echo "Linking SSH auth sock: $SSH_AUTH_SOCK"
fi  
export SSH_AUTH_SOCK="$HOME/.ssh/auth_sock-$HOST"

This script also includes the hostname in the filename, since I use the same homedir on multiple (virtual) hosts. Of course, after I implemented this solution, it turns out I'm not the only one to have thought of this. Others have documented this approach as well, though those are more screen-specific (my approach makes the agent available to all shells, also outside of the screen).

How to run these custom commands from Mutt?

This is actually the least elegant part of all this. I needed to somehow get mutt to save the attachment to some temporary place and run a custom command (i.e., a script containg a few commands) on it. This script would then take care of copying the attachment to Xanthe and opening it there.

An obvious candidate for this is mutt's pipe-entry command, which takes the attachment and pipes it to some user-specified command. However, the main problem with this is that the script will only get the contents of the file, not the filename or filetype (we want to preserve the filename and need the filetype to know how to open the file later on).

Another thing that looks promising is the view-mailcap command. This command uses the mime type or extension of the attachment and the /etc/mailcap file to find out which program to call to view the attachment. It then saves the attachment to a temporary file and runs the pogram to view it. This looks very similar to what we want to do, but there is no way to specify the command to run from within mutt.

However, you can specify what mailcap file mutt should use. So, what I've done is create a mailcap file that maps every filetype to a single command. I've named it ~/.mutt/xanthe-open-mailcap:

text/*; xanthe-open '%s' '%t'; needsterminal
application/*; xanthe-open '%s' '%t'; needsterminal
audio/*; xanthe-open '%s' '%t'; needsterminal
video/*; xanthe-open '%s' '%t'; needsterminal
image/*; xanthe-open '%s' '%t'; needsterminal
message/*; xanthe-open '%s' '%t'; needsterminal
multipart/*; xanthe-open '%s' '%t'; needsterminal

Note that a mailcap file does not allow the */* wilcard, so I've specified all main mime types that I know of manually. What this mailcap file says is to open each filetype using the xanthe-open command, passing in the filename (%s) and the MIME type (%t). Note that it's not entirely clear to me if the %s and %t values should be quoted. Mutt's manual says not to, but the /etc/mailcap on my system does have them quoted.

To make use of this mailcap file, I've created a macro in mutt to call it. The following is from my .muttrc:

macro attach x ":set mailcap_path=~/.mutt/xanthe-open-mailcap\n<view-mailcap>:reset mailcap_path\n"

This enables pressing the "x" in the mutt attachment list to run the xanthe-open script.

How to get the file opened on Xanthe?

The final part of this puzzle is how to get the file opened on Xanthe. For this, there's two pieces: First we need to copy the file, then we need to open it. The script to do this is called xanthe-open. I've saved it in ~/bin, which is is in my $PATH. It contains the following code:

#!/bin/sh

FILENAME=$1
MIMETYPE=$2

REMOTE_HOST=xanthe
REMOTE_DIR=tmp

# Copy the file to the remote host
scp "${FILENAME}" "${REMOTE_HOST}:${REMOTE_DIR}/"

# Set the DISPLAY variable so X appliations can find the display (this
# assumes the display is on :0, but that's usually the case anyway). We
# use run-mailcap to view the file copied, which looks up a program in
# /etc/mailcap. Here we open the single quote for the filename
# argument.
COMMAND="DISPLAY=:0 run-mailcap --action=view '"

if [ -n "${MIMETYPE}" -a "${MIMETYPE}" != "application/octet-stream" ];
then
# If the mimetype is specified and is not the useless
# octet-stream mimetype, prefix the the filename with it.
COMMAND="${COMMAND}${MIMETYPE}:"
fi

BASENAME=$(basename "${FILENAME}")
REMOTE_FILE="${REMOTE_DIR}/${BASENAME}"
# Here we close the single quote for the filename argument.
# Note: Quoting the filename in single quotes won't work well enough
# when there are single quotes in $REMOTE_FILE.
COMMAND="${COMMAND}${REMOTE_FILE}'"

# Run the command in the background. Redirect stdout so ssh terminates
# after running the command (just backgrouding wasn't enough, but stderr
# and stdin don't need redirection apparently). 
COMMAND="${COMMAND} > /dev/null &"
# We wait a short bit so we can see anything that appears on stderr
# directly (like mailcap errors).
COMMAND="${COMMAND} sleep 2"

# Run the command.
ssh xanthe "${COMMAND}"

# vim: set sts=4 sw=4 expandtab:

This script starts out with calling scp to copy the file to Xanthe (using the tunnel and ssh configuration given above). The file is opened by calling the run-mailcap command on Xanthe. This command looks in the /etc/mailcap file to find a program to open the file with.

If a mimetype was given to the script (which eventueally comes from the email headers) and if it is actually a usefule mimetype (a lot of email clients just put in application/octet-stream for all attachments) then the type is passed to run-mailcap as well. Otherwise, run-mailcap will make a guess based on the file extension.

Conclusions

So, it's a fair bit of scripting and configging, but you get a nice setup. A lot of the stuff I've described is applicable more generally as well. The SSH agent stuff is just plain useful in general. The SSH tunnel allows for other cool scripting as well (like Bas did, who used to have the "now playing" part of his blog powered by such a tunnel or perhaps something to auto-load links from IRC into my browser). The mutt stuff even allows me to view ugly HTML email in my browser directly instead of using a text-mode browser for that (though I'm not quite sure what the security implications of this are, imagine an email with javascript that's executed from withtin the "local files" trust zone...).

Have fun with this, and feel free to leave a comment if my stuff was useful for you.

Update: It seems that when using gdm instead of xdm authentication checks are different. Instead of storing auth tokens in ~/.Xauthority, they are stored in some file below /var/lib/gdm3, using the $XAUTHORITY environment variable to point to that file. Since $XAUTHORITY isn't set on the incoming SSH connections, so X programs can't be started to open attachments... To fix this, you'll have to save that $XAUTHORITY value somewhere. I just wrote a blogpost on how to do that, in case you need this. You might notice that this is approach could also be used to handle the $SSH_AUTH_SOCK instead of the link aproach I describe here.

Update: Nieko has posted a blog post showing how to get this same thing working on Windows client.