These are the ramblings of Matthijs Kooijman, concerning the software he hacks on, hobbies he has and occasionally his personal life.
Most content on this site is licensed under the WTFPL, version 2 (details).
Questions? Praise? Blame? Feel free to contact me.
My old blog (pre-2006) is also still available.
See also my Mastodon page.
Sun | Mon | Tue | Wed | Thu | Fri | Sat |
---|---|---|---|---|---|---|
1 | 2 | 3 | 4 | |||
5 | 6 | 7 | 8 | 9 | 10 | 11 |
12 | 13 | 14 | 15 | 16 | 17 | 18 |
19 | 20 | 21 | 22 | 23 | 24 | 25 |
26 | 27 | 28 | 29 | 30 | 31 |
(...), Arduino, AVR, BaRef, Blosxom, Book, Busy, C++, Charity, Debian, Electronics, Examination, Firefox, Flash, Framework, FreeBSD, Gnome, Hardware, Inter-Actief, IRC, JTAG, LARP, Layout, Linux, Madness, Mail, Math, MS-1013, Mutt, Nerd, Notebook, Optimization, Personal, Plugins, Protocol, QEMU, Random, Rant, Repair, S270, Sailing, Samba, Sanquin, Script, Sleep, Software, SSH, Study, Supermicro, Symbols, Tika, Travel, Trivia, USB, Windows, Work, X201, Xanthe, XBee
On a small embedded system, I wanted to run a simple Rails application and have it automatically start up at system boot. The system is running systemd, so a systemd service file seemed appropriate to start the rails service.
Normally, when you run the ruby-on-rails standalone server, it binds on port 3000. Binding on port 80 normally requires root (or a special capability enabled for all of ruby), but I don't want to run the rails server as root. AFAIU, normal deployments using something like Nginx to open port 80 and let it forward requests to the rails server, but I wanted a minimal setup, with just the rails server.
An elegant way to binding port 80 without running as root is to use systemd's socket activation feature. Using socket activation, systemd (running as root) opens up a network port before starting the daemon. It then starts the daemon, which inherits the open network socket file descriptor, with some environment variables to indicate this. Apart from allowing privileged ports without root, this has other advantages such as on-demand starting, easier parallel startup and seamless restarts and upgrades (none of which is really important for my usecase, but it is still nice :-p).
On my mailserver, I'm using Spamassassin with a Bayes filter to detect spam. Such a filter needs to be trained with samples of spam and ham (non-spam) messages to let it learn what spam and ham looks like, but it also needs to be retrained when the spam or ham changes over time. I have some automatic training set up, but since a while I've seen the bayes filter being completely wrong (showing a confident ham score for something that is very clearly spam), so I decided to retrain the filter from scratch, using the spam and ham messages I collected over the last time (I don't really throw away any e-mail).
Since training with all my e-mail is not productive (more than 5,000
messages aren't really helpful AFAIU, and training with old messages is
not representative for current messages), I decided to just take all of
my e-mail and take the last 2,000 spam and ham messages and train with
that. My spam is neatly collected in 2 mailboxes (Spam for obvious spam
and ProbablySpam for messages that need an occasional review to find
false positives), but my ham is sorted out in dozens of different
mailboxes. Hence, I needed some find
magic to get a list of the most
recent spam and ham messages. So, I built these commands:
# find Spam ProbablySpam -type f \( -path '*/cur/*' -o -path '*/new/*' \) -printf "%T@ %p\n"
| sort -n | cut -d' ' -f 2 | tail -n 2000 > spam
# find . -type d \( -path ./Spam -o -path ./ProbablySpam -o -path ./Bulk -o -path ./Sent \) -prune -o \
-type f \( -path '*/cur/*' -o -path '*/new/*' \) -printf "%T@ %p\n" \
| sort -n | cut -d' ' -f 2 | tail -n 2000 > ham
# sa-learn --progress --spam -f spam
# sa-learn --progress --ham -f ham
After retraining with recent spam, the results were a lot better, so I'm not longer spending time every day deleting a couple dozens spam e-mails :-D
While setting up Tika, I stumbled upon a fairly unlikely corner case in the Linux kernel networking code, that prevented some of my packets from being delivered at the right place. After quite some digging through debug logs and kernel source code, I found the cause of this problem in the way the bridge module handles netfilter and iptables.
Just in case someone else actually finds himself in this situation and actually manages to find this blogpost, I'll detail my setup, the problem and it solution here.
In my work as a Debian Maintainer for the OpenTTD and related packages, I occasionally come across platform-specific problems. That is, compiling and running OpenTTD works fine on my own x86 and amd64 systems, but when I my packages to Debian, it turns out there is some problem that only occurs on more obscure platforms like MIPS, S390 or GNU Hurd.
This morning, I saw that my new grfcodec package is not working on a bunch of architectures (it seems all of the failing architectures are big endian). To find out what's wrong, I'll need to have a machine running one of those architectures so I can debug.
In the past, I've requested access to Debian's "porter" machines, which are intended for these kinds of things. But that's always a hassle, which requires other people's time to set up, so I'm using QEMU to set up a virtual machine running the MIPS architecture now.
What follows is essentially an update for this excellent tutorial about running Debian Etch on QEMU/MIPS(EL) by Aurélien Jarno I found. It's probably best to read that tutorial as well, I'll only give the short version, updated for Squeeze. I've also looked at this tutorial on running Squeeze on QEMU/PowerPC by Uwe Hermann.
Finally, note that Aurélien also has pre-built images available for download, for a whole bunch of platforms, including Squeeze on MIPS. I only noticed this after writing this tutorial, might have saved me a bunch of work ;-p
You'll need qemu. The version in Debian Squeeze is sufficient, so
just install the qemu
package:
$ aptitude install qemu
You'll need a virtual disk to install Debian Squeeze on:
$ qemu-img create -f qcow2 debian_mips.qcow2 2G
You'll need a debian-installer kernel and initrd to boot from:
$ wget http://ftp.de.debian.org/debian/dists/squeeze/main/installer-mips/current/images/malta/netboot/initrd.gz
$ wget http://ftp.de.debian.org/debian/dists/squeeze/main/installer-mips/current/images/malta/netboot/vmlinux-2.6.32-5-4kc-malta
Note that in Aurélien's tutorial, he used a "qemu
" flavoured
installer. It seems this is not longer available in Squeeze, just a few
others (malta
, r4k-ip22
, r5k-ip32
, sb1-bcm91250a
). I just picked
the first one and apparently that one works on QEMU.
Also, note that Uwe's PowerPC tutorial suggests downloading a iso cd image and booting from that. I tried that, but QEMU has no BIOS available for MIPS, so this approach didn't work. Instead, you should tell QEMU about the kernel and initrd and let it load them directly.
You just run QEMU, pointing it at the installer kernel and initrd and passing some extra kernel options to keep it in text mode:
$ qemu-system-mips -hda debian_mips.qcow2 -kernel vmlinux-2.6.32-5-4kc-malta -initrd initrd.gz -append "root=/dev/ram console=ttyS0" -nographic
Now, you get a Debian installer, which you should complete normally.
As Aurélien also noted, you can ignore the error about a missing boot loader, since QEMU will be directly loading the kernel anyway.
After installation is completed and the virtual system is rebooting, terminate QEMU:
$ killall qemu-system-mips
(I haven't found another way of terminating a -nographic QEMU...)
Booting the system is very similar to booting the installer, but we leave out the initrd and point the kernel to the real root filesystem instead.
Note that this boots using the installer kernel. If you later upgrade the kernel inside the system, you'll need to copy the kernel out from /boot in the virtual system into the host system and use that to boot. QEMU will not look inside the virtual disk for a kernel to boot automagically.
$ qemu-system-mips -hda debian_mips.qcow2 -kernel vmlinux-2.6.32-5-4kc-malta -append "root=/dev/sda1 console=ttyS0" -nographic
Be sure to check Aurélien's tutorial for some more features, options and details.
A I previously mentioned, Gnome3 is migrating away from the gconf settings storage to the to GSettings settings API (along with the default dconf settings storage backend).
So where you previously used the gconf-editor
program to browse and
edit Gnome settings, you can now use dconf-editor
to browse and
edit settings.
I do wonder if the name actually implies that dconf-editor
is editing
the dconf storage directly, instead of using the fancy new GSettings
API? :-S
For some time, I've been looking for a decent backup solution. Such a solution should:
Up until now I haven't found anything that met my demands. Most backup solutions don't run on (headless Linux) and most generic cloud storage providers are way too expensive (because they offer high-availability, high-performance storage, which I don't really need).
Backblaze seemed interesting when they launched a few years ago. They just took enormous piles of COTS hard disks and crammed a couple dozen of them in a custom designed case, to get a lot of cheap storage. They offered an unlimited backup plan, for only a few euros per month. Ideal, but it only works with their own backup client (no normal FTP/DAV/whatever supported), which (still) does not run on Linux.
Recently, I had another look around and found CrashPlan, which offers an unlimited backup plan for only $5 per month (note that they advertise with $3 per month, but that is only when you pay in advance for four years of subscription, which is a bit much. Given that if you cancel beforehand, you will still get a refund of any remaining months, paying up front might still be a good idea, though). They also offer a family pack, which allows you to run CrashPlan on up to 10 computers for just over twice the price of a single license. I'll probably get one of these, to backup my laptop, Brenda's laptop and my colocated server.
The best part is that the CrashPlan software runs on Linux, and even on a headless Linux server (which is not officially supported, but CrashPlan does document the setup needed). The headless setup is possible because CrashPlan runs a daemon (as root) that takes care of all the actual work, while the GUI connects to the daemon through a TCP port. I still need to double-check what this means for the security though (especially on a multi-user system, I don't want to every user with localhost TCP access to be able to administer my backups), but it seems that CrashPlan can be configured to require the account password when the GUI connects to the daemon.
The CrashPlan software itself is free and allows you to do local backups and backups to other computers running CrashPlan (either running under your own account, or computers of friends running on separate accounts). Another cool feature is that it keeps multiple snapshots of each file in the backup, so you can even get back a previous version of a file you messed up. This part is entirely configurable, but by default it keeps up to one snapshot every 15 minutes for recent changes, and reduces that to one snapshot for every month for snapshots over a year old.
When you pay for a subscription, the software transforms into CrashPlan+ (no reinstall required) and you get extra features such as multiple backup sets, automatic software upgrades and most notably, access to the CrashPlan Central cloud storage.
I've been running the CrashPlan software for a few days now (it comes with a 30-day free trial of the unlimited subscription) and so far, I'm quite content with it. It's been backing up my homedir to a local USB disk and into the cloud automatically, I don't need to check up on it every time.
The CrashPlan runs on Java, which I doesn't usually make me particularly enthousiastic. However, the software seems to run fast and reliable so far, so I'm not complaining. Regarding the software itself, it does seem to me that it's not intended for micromanaging. For example, when my external USB disk is not mounted, the interface shows "Destination unavailable". When I then power on and mount the external disk, it takes some time for Crashplan to find out about this and in the meanwhile, there's no button in the interface to convince CrashPlan to recheck the disk. Also, I can add a list of filenames/path patterns to ignore, but there's not really any way to test these regexes.
Having said that, the software seems to do its job nicely if you just let it do its job in the background. On piece of micromanagement which I do like is that you can manually pause and resume the backups. If you pause the backups, they'll be automatically resumed after 24 hours, which is useful if the backups are somehow bothering you, without the risk that you forget to turn the backups back on.
Of course, sending away backups is nice when I am at home and have 50Mbit fiber available, but when I'm on the road, running on some wifi or even 3G connection, I really don't want to load my connection with the sending of backup data.
Of course I can manually pause the backups, but I don't want to be doing that every time when I pick up my laptop and get moving. Since I'm using a docking station, it makes sense to simply pause backups whenever I undock and resume them when I dock again.
The obvious way to implement this would be to simply stop the CrashPlan daemon when undocking, but when I do that, the CrashPlanDesktop GUI becomes unresponsive (and does not recover when the daemon is started again).
So, I had a look at the "admin console", which offers "command
line" commands, such as pause
and resume
. However, this command line
seems to be available only inside the GUI, which is a bit hard to
script (also note that not all of the commands seem to work for me,
sleep
and help
seem to be unknown commands, which cause the console
to close without an error message, just like when I just type something
random).
It seems that these console commands are really just sent verbatim to the CrashPlan daemon. Googling around a bit more, I found a small script for CrashPlan PRO (the business version of their software), which allows sending commands to the daemon through a shell script. I made some modifications to this script to make it useful for me:
/usr/local/crashplan
in the script instead==
vs =
)-XstartOnFirstThread
argument from java (MacOS only?)$command
but instead
pass "$@" to java directly. This latter prevents bash from splitting
arguments with spaces in them into multiple arguments, which
causes the command "pause 9999" to be interpreted as two commands
instead of one with an argument.I have this script under /usr/local/bin/CrashPlanCommand
:
#!/bin/sh
BASE_DIR=/usr/local/crashplan
if [ "x$@" == "x" ] ; then
echo "Usage: $0 <command> [<command>...]"
exit
fi
hostPort=localhost:4243
echo "Connecting to $hostPort"
echo "Executing $@"
CP=.
for f in `ls $BASE_DIR/lib/*.jar`; do
CP=${CP}:$f
done
java -classpath $CP com.backup42.service.ui.client.ConsoleApp $hostPort "$@"
Now I can run CrashPlanCommand 'pause 9999'
and CrashPlanCommand
resume
to pause and resume the backups (9999 is the number of minutes
to pause, which is about a week, since I might be undocked more than 24
hourse, which is the default pause time).
To make this run automatically on undock, I created a simply udev rules
file as /etc/udev/rules.d/10-local-crashplan-dock.rules
:
ACTION=="change", ATTR{docked}=="0", ATTR{type}=="dock_station", RUN+="/usr/local/bin/CrashPlanCommand 'pause 9999'"
ACTION=="change", ATTR{docked}=="1", ATTR{type}=="dock_station", RUN+="/usr/local/bin/CrashPlanCommand resume"
And voilà! Automatica pausing and resuming on undocking/docking of my laptop!
I upgraded to Gnome3 this week, and after half a day of debugging I got
my (quite non-standard) setup working completely again. One of the
things that got broken was my custom wallpaper on the gdm3 login screen.
This used to be configured in /etc/gdm3/greeter.gconf.defaults
, but
apparently Gnome3 replaced gconf by this new "gsettings" thingy.
Anyway, to change the desktop background in gdm, add the following lines
to /etc/gdm3/greeter.gsettings
:
[org.gnome.desktop.background]
picture-uri='file:///etc/gdm3/thinkpad.jpg'
For reference, I also found some other method, which looks a lot
more complicated. I suspect it also doesn't work in Debian, which runs
gdm as root, not as a separate "gdm
" user. Systems that do use such a
user might need the more complicated method, I guess (which probably
ends up storing the settings somewhere in the homedir of the gdm
user...).
I've been configuring my new laptop (more on that later) and this time I've tried to get the volume hotkeys working properly with Pulseaudio. On a default Debian Squeeze installation, the volume hotkeys are processed by (the media-keys plugin of) gnome-settings-daemon (1). The good news is that Gnome has switched over to using pulseaudio by default (and even removed support for plain ALSA). However, Debian does not want to force users to use pulseaudio. So the bad news is that Debian has disabled this pulseaudio support in gnome-settings-daemon and has a patch to use the ALSA mixer (via GStreamer).
Normally, it shouldn't matter much which mixer you use, as long as they
work. However, I'm using two different sound cards on my laptop: The
builtin one for on the road and an external USB sound card when I'm at
home (to get a S/PDIF output). So I need pulseaudio to route my audio to
the right place, and I want my volume controls to control the same card
as well. Note that gnome-volume-control
, the GUI to control your
volums is installed in two flavours by Debian (Pulseaudio and
GStreamer), and the right one is started by a wrapper script depending
on whether Pulse is running.
Fortunately, the Debian patch is somewhat configurable: You can select a
different mixer device through gconf. To get at that configuration, use
gconf-editor
and browse to /desktop/gnome/sound/default_mixer_device
.
Set this value in the form of "element:device
", where element
selects the gstreamer plugin to use, and device
sets its "device"
property. I initially tried using the "pulsemixer
" element (in the
form
"pulsemixer:alsa_output.usb-0ccd_USB_Audio-00-Aureon51MkII.analog-stereo
"),
but that only allowed me to specify a specific Pulseaudio sink, not
"whatever-is-default").
So, instead, I settled for using the "alsamixer
" gstreamer plugin,
together with the Pulseaudio ALSA plugin (the same one you use to
redirect ALSA applications to Pulseaudio). For this to work, it's
important that you redirect ALSA applications to pulse using the
following in your /etc/asound.conf
or your ~/.asoundrc
:
pcm.!default.type pulse
ctl.!default.type pulse
This makes sure that not just audio streams (pcm
) but also mixer
controls (ctl
) are redirected to Pulseaudio. Now, set the
/desktop/gnome/sound/default_mixer_device
gconf value to the
following:
alsamixer:default
This should make sure that your volume keys work with the device
selected as default in Pulseaudio (through pavucontrol
or
gnome-volume-control
for example). It seems this behaviour relies on
the fact that gnome-settings-daemon only keeps the mixer controls open
for a few seconds, allowing the Pulseaudio ALSA plugin to select the
right pulseaudio sink to control everytime the mixer is reopened (so it
needs a few seconds of not pressing the volume hotkeys after changing
the default device).
By the way, it seems that in the next version of Gnome (and/or Debian) this problem wil probably be fixed out of the box, since the 2.93 packages in Debian experimental have Pulseaudio support enabled (haven't tested them, though).
Hopefully this helps someone else out there struggling with the same problem...
(1): You might have noticed that I'm talking about Gnome here. I case
you wondered, I've actually started to use parts of Gnome for daily use
on my laptop. I'm still using Awesome as my primary window manager
and I'm not using gnome-panel
, so I haven't suddenly become a GUI addict
all of the sudden ;-)
When you use Screen together with Xorg, you'll recognize this: You log in to an X session, start screen and use the terminals within screen to start programs every now and then. Everything works fine so far. Then, you logout and log in again (or X crashes, or whatever). You happily re-attach the still running screen, which allows you to continue whatever you were doing.
But now, whenever you want to start a GUI program, things get wonky. You'll get errors about not being able to find configuration data, connect to gconf or DBUS, or your programs will not start at all, with the ever-informative error message "No protocol specified". You'll also recognize your ssh-agent and gpg-agent to stop working within the screen session...
What is happening here, is that all those programs are using "environment variables" to communicate. In particular, when you log in, various daemons get started (like the DBUS daemon and your ssh-agent). To allow other programs to connect to these daemons, they put their contact info in an environment variable in the login process. Whenever a process starts another process, these environment variables get transfered from the parent process to the child process. Sine these environment variables are set in the X sesssion startup process, which starts everything else, all programs should have access to them.
However, you'll notice that, after logging in a second time, the screen you re-attach to was not started by the current X session. So that means its environment variables still point to the old (no longer runnig) daemons from the previous X session. This includes any shells already running in the screen as well as new shells started within the screen (since the latter inherit the environment variables from the screen process itself).
To fix this, we would like to somehow update the environment of all
processes that are already running when we login, to update them with
the addresses of the new daemons. Unfortunately, we can't change the
environment of other processes (unless we resort to scary stuff
like using gdb
or poking around in /dev/mem
...). So, we'll have to
convice those shells to actually update their own environments.
So, this solution has two parts: First, after login, saving the relevant variables from the environment into a file. Then, we'll need to get our shell to load those variables.
The first part is fairly easy: Just run a script after login that writes
out the values to a file. I have a script called ~/bin/save-env
to do
exactly that. It looks like this (full version here):
#!/bin/sh
# Save a bunch of environment variables. This script should be run just
# after login. The saved variables can then be sourced by every bash
# shell, so long running shells (e.g., in screen) or incoming SSH shells
# can also use these services.
# Save the DBUS sessions address on each login
if [ -n "$DBUS_SESSION_BUS_ADDRESS" ]; then
echo export DBUS_SESSION_BUS_ADDRESS="$DBUS_SESSION_BUS_ADDRESS" > ~/.env.d/dbus
fi
if [ -n "$SSH_AUTH_SOCK" ]; then
echo export SSH_AGENT_PID="$SSH_AGENT_PID" > ~/.env.d/ssh
echo export SSH_AUTH_SOCK="$SSH_AUTH_SOCK" >> ~/.env.d/ssh
fi
# Save other variables here
This script fills the directory ~/.env.d with files containg environment variables, separated by application. I could probably have thrown them all into a single file, but it seemed like a good idea to separate them. Anyway, these files are created in such a way that they can be sourced by a running shell to get the new files.
If you download and install this script, don't forget to make it
executable and create the ~/.env.d directory. You'll need to make sure
it gets run as late as possible after login. I'm running a (stripped
down) Gnome session, so I used gnome-session-properties
to add it
to my list of startup applications. You might call this script from your
.xession, KDE's startup program list, or whatever.
For the second part, we need to set our saved variables in all of our
shells. This sounds easy, just run
for f in ~/.env.d/*; do source "$f"; done
in every shell (Don't be
tempted to do source ~/.env.d/*
, since that sources just the first
file with the other files as arguments!). But, of course we don't want
to do this manually, but let every shell do it automatically.
For this, we'll use a tool completely unintended, but suitable enough
for this job: $PROMPT_COMMAND
. Whenever Bash is about to display a
prompt, it evals whatever is in the variable $PROMPT_COMMAND
. So it
ends up evaluating that command all the time, which makes it a prefect
place to load the saved variables. By setting the $PROMPT_COMMAND
variable in your ~/.bashrc
variable, it will become enabled in every
shell you start (except for login shells, so you might want to source
~/.bashrc
from your ~/.bash_profile
):
# Source some variables at every prompt. This is to make stuff like
# ssh agent, dbus, etc. working in long-running shells (e.g., inside
# screen).
PROMPT_COMMAND='for f in ~/.env.d/*; do source "$f"; done'
You might need to be careful where to place this line, in case
PROMPT_COMMAND already has some other value, like is default on Debian
for example. Here's my full .bashrc file, note the +=
and
starting ;
in the second assignment of $PROMPT_COMMAND
.
The astute reader will have noticed that this will only work for existing shells when a prompt is displayed, meaning you might need to just press enter at an existing prompt (to force a new one) after logging in the second time to get the values loaded. But that's a small enough burden, right?
So, with these two components, you'll be able to optimally use your long-running screen sessions, even when your X sessions are not so stable ;-)
Additionally, this stuff also allows you to use your faithful daemons
when you SSH into the machine. I use this so I can start GUI programs
from another machine (in particular, to open up attachments from my
email client which runs on a server somewhere). See my recent blogpost
about setting that up. However, since running a command through SSH
non-interactively never shows a prompt and thus never evaluates
$PROMPT_COMMAND
, you'll need to manually source the variables at once
in your .bashrc directly. I do this at the top of my ~/.bashrc.
Man, I need to learn how to writer shorter posts...
Only recently, Adobe has started to (finally) support 64 bit Linux with its Flash plugin. I could finally watch Youtube movies (and more importantly, do some Flash development work for Brevidius).
However, this month Adobe has announced that it drops support for 64 bit Linux again. Apparently they "are making significant architectural changes to the 64-bit Linux Flash Player and additional security enhancements" and they can't do that while keeping the old architecture around for stable releases, apparently.
This is particularly nasty, because the latest 10.0 version (which still has amd64 support) has a couple of dozens (!) of security vulnerabilities which are fixed in a 10.1 version only (which does not have Linux amd64 support anymore).
So Adobe is effectively encouraging people on amd64 Linux to either not use their product, or use a version with critical security flaws. Right.