As I already pointed out, I'm not using rdiff-backup anymore. The reason is mainly that it is simply too slow. I'm using a Raspberry Pi as my NAS and it is absolutely not capable of handling larger backups with rdiff-backup. It works for smaller backup sizes, but not for my entire home directory. Even when I pushed the initial full backup directly to the backup disk (not using my Raspberry), all future incremental backups were still unbearably slow. Even when no files changed at all, it took hours over hours for simply comparing all the files I had in my home directory to those on the NAS, whereas a full comparison using rsnapshot is done within five to ten minutes. Now keep this in mind and look at the fact that incomplete backups made with rdiff-backup can't be resumed. You could imagine that in the end you wouldn't have any backup at all. Basically all rdiff-backup would do is to compare and push your files over the day and abort in the evening when you shut down your workstation. And then the next day it would spend all the time reverting the incomplete backup and running another one which might not finish either.

So this is the main reason I stopped my experiments with rdiff-backup. It was a nice time, but I finally moved on. Therefore say hello to our new precious star: rsnapshot!

TL;DR: I have created a GitHub project page with a much more mature version of the scripts I discuss throughout this article. It is linked at the end.

Basically we want to do exactly the same with rsnapshot that we did with rdiff-backup. We want to push our backups from the client to the NAS and we want to keep the privileges separated. That means, there should be a general backup of all global system files run by root and then the backups of the individual home directories run by the users themselves (i.e. with their privileges and UIDs). The whole backup process should be as flexible as possible and each user should decide by himself which of his own files to include in a backup and which to explicitly exclude. Therefore we will again read the list of files and folders which are to be backed up from special files in a) the global system etc folder and b) the users' home directories.

One sticking point we have to solve is that rsnapshot by itself does not support push backups at all. Since it works with hardlinking it needs to operate on a local file system. But fear not, there is a simple yet effective solution, which I (to be honest) borrowed from mad-hacking.net. rsnapshot is based on rsync and rsync by itself is capable of pushing files to another host over SSH by starting a daemon process on the remote side. Furthermore, rsnapshot has the feature to keep the syncing and the rotation process separate, which is a good idea anyways when we push over the network (imagine all the bad things that might happen ). With the sync_first option enabled, rsnapshot uses a directory called .sync as the source for the rotation. This is the directory we will push our files to with rsync. That means we will use rsnapshot only for the rotation, not the real push backup itself.

The server side

Again, let's start with the server side, i.e. the NAS. As already done in the last part I will call the server/NAS bellatrix and the client that pushes its files altair.

The storage which will hold all the backups is mounted under /bkp and that's were our backup users will have their home directories. For each user on each client we will have one user on the server. So first of all create a new group and two new users for backups from altair on the server:

groupadd backup
useradd -G backup -b "/bkp" -m -p '*' -s /bin/sh "altair-root"
useradd -G backup -b "/bkp" -m -p '*' -s /bin/sh "altair-johndoe"

The first account will be for backups run as root on altair, the second one for backups run as the user johndoe.

Next we need to prepare the home directories. I'll do it for altair-root, the process is the same for altair-johndoe and any other user of course.

The backup files should all go to the directory $HOME/files and as described above we will use a folder called .sync inside of it for the actual syncing. Thus we need to create them both:

mkdir -p "/bkp/altair-root/files/.sync"

To enable root on altair to log into the NAS as altair-root we also need to set up an SSH key. But of course, we don't want to grant full shell access but only an rsync daemon instance. Therefore generate a new SSH key on the client and create a folder called .ssh inside /bkp/altair-root on the server. Inside that folder create a file called authorized_keys with the following content:

command="/usr/bin/rsync --server --daemon --config='/bkp/altair-root/rsync.conf' ." ssh-rsa AAAAB3NzaC1yc...

(replace the last part with the proper public key, of course)

Finally protect the file by setting the file owner to root:

chown -R root:root /bkp/altair-root/.ssh

This will ensure that a user that logs into bellatrix with that SSH key can only start an rsync daemon with the specified configuration file /bkp/altair-root/rsync.conf. Now create that file with these contents:

[push]
uid = altair-root
gid = altair-root
path = /bkp/altair-root/files/.sync
use chroot = 0
read only = 0
write only = 1
fake super = 1
max connections = 1
lock file = /bkp/altair-root/rsyncd.lock
post-xfer exec = /usr/local/bin/rs-rotate "/bkp/altair-root/rsnapshot.conf"

[pull]
uid = altair-root
gid = altair-root
path = /bkp/altair-root/files
use chroot = 0
read only = 1
fake super = 1

Save the file and make root own it, too.

The configuration directives above define two basic rsync modules, one named push and one named pull (which is read-only and intended for restoring backups from the NAS). For both modules the config file sets the proper target directories, which is the files/.sync folder for the actual backup process and simply the files directory for restoring backups. Processes connecting via rsync to these modules won't be able to escape these directories easily. One very important option in the push module is the fake super option. Since we're running with basic user privileges, we can't properly set things like ownership. Luckily, rsync is able to save this information to the extended user file attributes (xattrs) when this option is enabled. In the past this required the file system to be mounted with the user_xattr option, but if you're using Ext4, it should also work without since it's enabled by default. If you get a permission denied error, though, you might want to remount the file system with that option.

All the other lines should be more or less self-explanatory, but another very important one is

post-xfer exec = /usr/local/bin/rs-rotate "/bkp/altair-root/rsnapshot.conf"

This tells rsync to run the specified command after it finished the syncing process. We will use this to trigger the rotation. The file /usr/local/bin/rs-rotate (could be any other name and location, too) is a very simple shell script:

#!/bin/sh

if [ "$1" == "" ]; then
        echo "Usage: $(basename $0) <rsnapshot config>"
        exit
fi

if [ "$RSYNC_EXIT_STATUS" == "" ]; then
        echo "This script is intended to be run as rsync post-xfer hook." 2>&1
        exit 1
fi

if [ $RSYNC_EXIT_STATUS -eq 0 ]; then
        rsnapshot -c "$1" push
fi

When the rsync process exited cleanly (i.e. the syncing was entirely successful) it will ask rsnapshot to perform a rotation using the specified configuration file (which we also need to create):

config_version  1.2

cmd_cp			/usr/bin/cp
cmd_rm			/usr/bin/rm
cmd_rsync		/usr/bin/rsync
cmd_logger		/usr/bin/logger

retain			push			2
retain			daily			7
retain			weekly			4
retain			monthly			2

verbose			2
loglevel		3
one_fs			1
sync_first		1

snapshot_root	/bkp/altair-root/files
logfile			/bkp/altair-root/rsnapshot.log
lockfile		/bkp/altair-root/rsnapshot.pid
backup			/bkp/altair-root/files/.sync	./

Save this to /bkp/altair-root/rsnapshot.conf.

The syntax is explained quickly. All entries are separated by tabs (not spaces!) and each line with text contains one configuration directive. The first few lines simply define were to find certain command line tools. More interesting are the retain lines. These specify the different backup levels and how many increments are kept. For the push backups we will keep 2 increments. The second increment will then be the basis for the first daily increment. Since we keep 7 dailies, the seventh daily increment will server as the basis for the first weekly and so on.

Important to note is the sync_first option which needs to be set. Otherwise our whole system would not work. With this option set, though, the syncing would be invoked by running rsnapshot with the sync command which we'll never do. Any other invocation of rsnapshot will simply rotate our files.

Finally the last four lines set the directories rsnapshot should work on. The very last line tells rsnapshot to use the .sync directory as a source and back it up to the current working dir, which is set by snapshot_root. So whichever files are in /bkp/altair-root/files/.sync will be used for rotation.

Side note: you might also want to split the config file and put everything above snapshot_root in a separate file and reference it via a include_conf directive. This would enable you to only write a very minimal config file for each backup user and simply reuse the options which are the same for everyone.

To invoke a rotation for a specific backup level, use rsnapshot <level> which is exactly what we did in the rsync post-xfer hook script above for the push level.

All the other levels can be rotated by a cron script that is run every day. But be careful: since we rely on push backups, you should only perform a rotation via cron when there are enough increments of the preceding level. Otherwise you'd successively delete your older backups. For instance, if we take the configuration above and keep two push increments and seven daily increments, the rotation cron would delete the seventh daily increment, rotate all preceding dailies by one number and then make the second push increment the new daily.0. No problem so far. But if you ran the rotation again without creating a new push increment first, rsnapshot would think that you had deleted all your files. The result would be that the last daily would be deleted again and all the earlier dailies would be rotated by one increment. But there would be no new daily.0 since there is no push.1, only a push.0. So after five more rotations you'd be left with only your push.0 and no dailies at all.

Therefore you should carefully check the number of existing increments of the preceding level first before doing a rotation. To do this you could put something like this in your cron script:

config=$(cat "path/to/the/rsnapshot/config.conf")

# Get number of preceding increments
config=$(echo "${config}" | grep -P '^retain\t')
config=$(echo "${config}" | grep -oPz "retain\t+(\w+)\t+(\d+)\nretain\s+${1}\t+" | sed -n 1p)
preceding_name=$(echo "${config}" | awk '{ print $2 }')
preceding_number=$(($(echo "${config}" | awk ' { print $3 }') - 1))

if [ "${preceding_name}" != "" ] &&
   [ -d "path/to/backup/dir/${preceding_name}.${preceding_number}" ]; then
        # Perform rotation
fi

Of course this also means that “daily” doesn't necessarily main exactly “daily” anymore. Now it rather means something like “backup from the day before the last push backup”.

The client side

We have the server side, now we need the client side, which is simple. Very simple. It is exactly the same as described in the last part of the series, you only need to replace the rdiff-backup command with this:

rsync \
    --rsh=ssh
    --archive \
    --acls \
    --delete \
    --delete-excluded \
    --include-from="${home_dir}/.rsnapshot-backup-filelist"
    --exclude="*" \
    / \
    "$(hostname)-${username}@${BACKUP_HOST}::push"

You do nothing more than rsync-ing your files to the server using the push module we defined in your rsync.conf on the server. Similarly you can pull files from there with rsync pretty much the same way using the pull module:

rsync \
    --rsh=ssh \
    --archive \
    "$(hostname)-${username}@${BACKUP_HOST}::pull/file/on/the/server ./where/the/file/should/go/to"

Of course you can also use the short options -e and -a instead of --rsh and --archive.

One more thing to note: the file that contains the list of files which should be backed up (~/.rsnapshot-backup-filelist in this case) works a little differently. By default all directories are excluded which are not explicitly included. That is also true for parent directories. So to include, e.g., /home/foo/bar/baz and all directories below you'd have to write:

/home
/home/foo
/home/foo/bar
/home/foo/bar/baz/***

Simply writing /home/foo/bar/baz/*** wouldn't be enough since /home is not included explicitly.

Note the slashes: no slash at the end means: “only the directory, not its contents”. A slash at the end, though, means: “the contents of this directory”. The version I used with the three asterisks means “this directory, the files inside it and inside all subdirectories”. So mind the little differences or otherwise your backup might be different from what you intended. For more information about rsync globbing patterns consult the FILTER RULES section in the rsync(1) man page.

For reference: my file currently looks about like this:

- *.swp
- *.tmp
- .directory
- Thumbs.db
- desktop.ini
- Desktop.ini
- .DS_Store
- *~
- .Trash/***

- /home/janek/.Xauthority
- /home/janek/.xsession-errors
- /home/janek/.cache
- /home/janek/.dbus
- /home/janek/.codeintel
- /home/janek/.zsh_history
- /home/janek/.bash_history
- /home/janek/.recently-used
- /home/janek/.pulse-cookie
- /home/janek/.config/pulse/***
- /home/janek/.local/tmp/***
- /home/janek/.dropbox/***
- /home-accel/janek/.kde4/cache-*/***
- /home-accel/janek/.kde4/socket-*/***
- /home-accel/janek/.kde4/tmp-*/***
- /home-accel/janek/.kde4/share/apps/nepomuk/***
- /home-accel/janek/.mozilla/firefox/*/Cache/***
- /home-accel/janek/.thunderbird/*/Cache/***

/home
/home/janek/***
/home-accel
/home-accel/janek/***
/srv
/srv/http
/srv/http/virtual
/srv/http/virtual/janek/***

You see: as with rdiff-backup you can also prefix lines with a - to specify explicit excludes. Lines starting with a + or no sign at all are treated as includes.

Some more polishing

That's pretty much it, but we can still tweak one thing ore two.

Providing read-only SFTP access

Currently we can only restore files from the backup server using rsync. This is okay for the restoring process itself, but it makes it hard to browse your backups. Wouldn't it be convenient if we could also mount the backup folder from the server using SFTP/SSHFS? That's very much possible.

You only need to do two things: first of all modify the authorized_keys file on the server like this:

command="/usr/bin/rs-run-ssh-cmd '/bkp/altair-root'" ssh-rsa AAAAB3NzaC1yc...

and then create the script /usr/bin/rs-run-ssh-cmd:

#!/bin/sh

home_dir=$1

if [ "${SSH_ORIGINAL_COMMAND}" == "internal-sftp" ] || [ "${SSH_ORIGINAL_COMMAND}" == "/usr/lib/ssh/sftp-server" ]; then
    cd "${home_dir}/files"
    exec /usr/lib/ssh/sftp-server -R
else
    exec /usr/bin/rsync --server --daemon --config="${home_dir}/rsync.conf" .
fi

echo "Session failed." >&2

exit 1

This will start an SFTP server when needed, otherwise simply the rsync daemon as before.

Chroot users into /bkp

If you want a little more security, you can also chroot all backup users into /bkp. For this to work you need to add these lines to your /etc/ssh/sshd_config on the server:

Match Group backup 
        ChrootDirectory /bkp/

After you've done that you need to make all files from outside the directory that are necessary for the services to operate, available inside the chroot environment. That means you either need to copy them to /bkp or use bind mounts. Copying has the advantage that you can copy only those files that are really needed, but it also creates a lot of duplicate files. I prefer bind mounts most of the time. Hardlinks are usually not a very good idea.

For rsnapshot you need to bind mount at least /bin, /usr/bin, /lib, /usr/lib and /usr/share/perl5:

mkdir -p "/bkp/"{"bin","usr/bin","lib","usr/lib","/usr/share/perl5"}
mount -o bind "/bin" "/bkp/bin"
mount -o bind "/usr/bin" "/bkp/usr/bin"
mount -o bind "/lib" "/bkp/lib"
mount -o bind "/usr/lib" "/bkp/usr/lib"
mount -o bind "/usr/share/perl5" "/bkp/usr/share/perl5"

If you're using the SFTP server, you also need /dev. Additionally a copy of the /etc/passwd file is necessary for the UID mapping, but you only need to keep those users which should be able to log into the chroot. So for your two backup users altair-root and altair-johndoe the following minimal /bkp/etc/passwd file would be enough:

altair-root:x:1001:1001::/bkp/altair-root:/bin/sh
altair-johndoe:x:1002:1002::/bkp/altair-johndoe:/bin/sh

(the UIDs and GIDs should of course correspond to the real UIDs and GIDs of the users)

Side note: OpenSSH provides an internal-sftp subsystem that works without any bind mounts and additional passwd files. But unfortunately, it is not possible to trigger it from a shell script. So in order to use it you'd either have to remove the command restriction from the authorized_keys file or use a completely different user for SFTP logins.

And finally: The GitHub project page

Because rsnapshot is what I finally settled upon, I needed a very flexible yet reliable backup system that is easy to maintain. So I created a bunch of large shell scripts which do all the work I described above and a lot more in a pretty convenient way.

I uploaded everything to GitHub where you can download it: rs-backup-suite on GitHub

Feel free to test it, modify it and redistribute it if you like.

Performing backups is a tedious process if done wrong. Thus backups need to done automatically in the background without any user intervention. As soon as someone feels the need to do something in order to get his stuff backed up, he will ultimately end up with no backup at all (and probably a bad conscience he only forgets too fast).

Not surprisingly, there are a bunch of tools you can use to crate your backups, but seldom they are just perfect. Often they are simply what I called them: “tools”. You have to know how to use them in order to build a working backup infrastructure on them. There are some plug'n'play backup programs out there such as Apple's Time Machine, but if you're like me and want to have a little more control over your backups then you have to think a little more about it.

In this little two-part article series (Part 2) I will present two tools I've been playing around with a lot and I'll show you how you can use them to set up your own personal NAS with a spare piece of hardware such as a Raspberry Pi. No need for any expensive special storage system.

The two tools are in order: rdiff-backup and rsnapshot. Both have their strengths and weaknesses. Let me give you a list of their pros and cons.

The two candidates

rdiff-backup:

Pros:

• Very small backup sizes since it only stores reverse deltas
• Has a snapshot file system with rdiff-backup-fs making it totally transparent to the user
• Saves all file permissions/attributes separately by default

Cons:

• Very CPU demanding and therefore pretty slow
• Without the rdiff-backup-fs FUSE only the last increment is accessible directly via the file system
• Aborted backups can't be resumed (rdiff-backup will revert to the last successful backup)
• Increments are a little fragile and easily corrupted
• rdiff-backup depends on the very same rdiff-backup version installed on the remote host

rsnapshot:

Pros:

• Stores snapshots in different directories and hardlinks them, so you don't need any special tools to restore older increments
• Very robust, aborted or failed backups can be resumed without further consequences
• A lot faster with less resource consumption due to simpler calculations
• Actually uses rsync making it very flexible

Cons:

• Files that only changed slightly are copied completely resulting in larger backup sizes
• More complex to set up (okay, I admit, it's just rdiff-backup being even simpler)

You see: rdiff-backup has quite a few drawbacks. Additionally to those mentioned above I should also say that rdiff-backup hasn't experienced any visible development since 2009 whereas rsnapshot is still being worked on, which may or may not be an advantage (although the latest official release is from 2008).

To say it directly: I don't use rdiff-backup anymore. I had my experiments with it, but finally I ended up using rsnapshot for reasons I'll explain in the next part. But still I think there are valid uses cases when rdiff-backup is just the right tool for you. For instance, if disk space is a very crucial constraint, you might prefer it over rsnapshot. One thing I also quite liked is that you don't have just files but real timestamped increments. rdiff-backup enables you to restore files based on their backup date, not only the file modification date (which might be wrong). rsnapshot, however, doesn't store this information. The only thing there which lets you guess the backup date is the folder name of the increment.

Push vs. pull backups

In computer networks whose nodes are always on you usually see pull backup systems where the backup servers actively “pull” the backups over from the machines which are to be backed up. This has a major advantage and that is simplicity and therefore reliability and maintainability. You only have one (or a couple) of machines implementing the logic for running backups. All the other machines are simply passive in that regard and can concentrate their resources on their real tasks. That also makes changing things such as the backup destination easier since you don't need to change the settings on each and every machine but only on the backup storage system. It also gives the backup system the opportunity to coordinate its own resources because it can decide itself when it has enough free capacities to process yet another backup in parallel.

But pull backups also have their disadvantages, especially when the machines that should be backed up are not always on. Here it's very difficult for the backup system to get proper backups since some nodes might simply be down when it tries to run a backup. Therefore most consumer-targeted backup systems are “push” backup systems.

Another consideration in favor of push backups is privilege separation. For performing pull backups of non-publicly accessible files and folders the remote backup system needs to log into my machine as root. Although I can still restrict the commands it can run, I don't particularly like the idea of another machine logging into my PC as root in an automatic fashion. When doing push backups, however, I don't need to log in as root anywhere. I still need root privileges on my own machine, but on the remote side I can simply log in as an unprivileged user. Call me paranoid if you don't agree.

To put it in a nutshell: I will concentrate on push backups throughout this series for both rdiff-backup and rsnapshot (which is a little more fiddly in the beginning, but it can be done pretty well).

Getting your hands dirty

Let's start with some basics about how rdiff-backup basically works. Similar to rsync it connects to a remote machine and starts a daemon process (both can of course also perform local backups, but that's not within the scope of this article). The client then sends the file signatures to the daemon process which compares them to the local files. If differences are found that can't be reconstructed from the data already available in the local file, they are transferred over the wire. This requires rdiff-backup to be installed on both systems because it needs to operate on the native file system without having an additional wrapper in between such as SFTP (NFS would work, though). That being said: we can divide our backup model into two parts: the client part (being the machine to be backed up) and the server part (being the backup storage). Let's start with the server part since the client won't work without it.

From now on I will call the client altair and the server bellatrix (which are simply the hostnames I use in my local network for my main workstation and the NAS).

The server part

I'm using a Raspberry Pi as the NAS running Arch Linux for ARM on it, but you can also use any other piece of hardware that can run rdiff-backup and preferably a Linux system.

We could now follow two different approaches: we could either use one directory for each client to be backed up and let the local root push the whole system backup to that folder or we could let each user back up his own files individually. Although the second approach might look more error-prone and less reliable, it's the way I decided to go because it enables me to keep the global system backup and the backup of my personal data separate. That makes it possible that I can simply restore my own data without the need of having root access and I can also run manual backups myself, e.g. when I have worked on something important and don't want to wait until the cron daemon starts the next global backup.

That said we need to find a way to keep separate users for each client and each user on those clients. Therefore I decided to use usernames consisting of the client hostname and the client username. For instance, the backup user for the user janek on the client host altair would be altair-janek. Similarly the global system backup of configuration files and shared resources would be altair-root (both being plainly unprivileged, of course).

The storage (i.e. the backup hard drive or maybe even a RAID and/or LVM system) is mounted to /mnt/storage and contains a folder bkp which is bind mounted to /bkp (mount -o bind /mnt/storage/bkp /bkp) . This is were the home directories of the backup users will go to.

First of all let's create the backup user (we're working on bellatrix):

useradd -b /bkp -m -p '*' -s /bin/sh altair-janek

Next we need to add an SSH key to allow passwordless login to that account (login with a password has been disabled by setting the password hash to * in the step before, but you might also want to disable it entirely in your /etc/ssh/sshd_config). After you created an SSH key on your client, transfer the public part to the server and add it to /bkp/altair-janek/.ssh/authorized_keys.

A word of warning: Don't use your normal SSH key! Create a new one you only use for the backup because it must not be encrypted. Then add a Host entry in ~/.ssh/config to use the newly generated key for your backup host.

This already enables our client to log in, but we don't want to give him full shell access, so we need to restrict the commands he can run. He should be able to log in via SSH, but then only push his backup using rdiff-backup. To make this work, modify the public key entry in /bkp/altair-janek/.ssh/authorized_keys as follows:

command="rdiff-backup --server --restrict '/bkp/altair-janek/files'" ssh-rsa AAAAB3NzaC1yc...

This will allow the client to only start the rdiff-backup daemon which restricts it to push files to and pull them from /bkp/altair-janek/files (that folder needs to exist, of course). Additionally you might want to chroot the user into his home directory or at least into /bkp.

The last thing we need to do is to protect the SSH configuration by assigning ownership to root:

chown -R root:root /bkp/altair-janek/.ssh

Do all this for each client user you want to enable backups for.

The client part

Now that we have set up our server, we need to perform the actual backup, which is quite simple. The client doesn't need to do anything else than running the proper rdiff-backup command. But we want to do it in a little more sophisticated way. We need a script that can be run as either root (and then performs a backup of the whole system, including the home directories) or as an unprivileged user in which case only the own home directory is backed up. Additionally, the backup of a home directory when run as root should be performed under the account of that user (otherwise we would store the backup twice on the server and in a location not accessible by him).

To specify which files should be included in a backup I'll use files containing file matching patterns. All files and directories that are not specified inside those files will not be backed up. For the global system backup I use the file /etc/default/rdiff-backup-filelist which contains something like this:

/etc
/usr/etc
/usr/local
- /srv/http/virtual/**
/srv/http
/root

This will include all files in /usr/etc, /usr/local, /srv/http and /root, but explicitly excludes all files below /srv/http/virtual. For a more detailed description of the format of this file have a look at the rdiff-backup examples page.

Additionally each user has a .rdiff-backup-filelist file inside his home directory which specifies the files which should be backed up under this user. That file could look like this, for instance:

- **.tmp
- **.swp
- **/.directory
- /home/janek/.cache
/home/janek
/srv/http/virtual/janek

Both files will be read from top to bottom and are “short-circuited”, i.e. the first match that is encountered will be used. Therefore /home/janek will be backed up, but without /home/janek/.cache.

Now we need to actually use these files. I wrote four shell functions for this:

BACKUP_HOST="bellatrix"
BACKUP_ROOT="/bkp"

# Back up selected system files
backup_system() {
        cd /root
        rdiff-backup \
                --exclude-other-filesystems \
                --include-symbolic-links \
                --exclude-special-files \
                --create-full-path \
                --ssh-no-compression \
                --include-globbing-filelist /etc/default/rdiff-backup-filelist \
                --exclude '**' \
                / \
                "$(hostname)-root@${BACKUP_HOST}::${BACKUP_ROOT}/$(hostname)-root/files"
}

# Back up single home directory
# Expects the home directory as parameter
backup_single_home_dir() {
        local home_dir=$1
        local passwd_entry
        local username
        local backup_cmd
        
        # Don't create a backup if home directory doesn't belong to a "real" user
        passwd_entry=$(grep ":${home_dir}:[^:]*$" /etc/passwd)
        if [ "$passwd_entry" == "" ]; then
                return
        fi
        
        username=$(echo "${passwd_entry}" | cut -d ':' -f 1)
        
        # Don't back up home directory either, if no files are marked for backup
        if [ ! -e "${home_dir}/.rdiff-backup-filelist" ]; then
                return
        fi
        
        # Also don't create a backup if no SSH key exists
        if [ ! -e "${home_dir}/.ssh/id_rsa" ] && [ ! -e "${home_dir}/.ssh/config" ]; then
                return
        fi
        
        cd "${home_dir}"
        
        backup_cmd="rdiff-backup \
                --create-full-path \
                --ssh-no-compression \
                --include-globbing-filelist \"${home_dir}/.rdiff-backup-filelist\" \
                --exclude '**' \
                / \
                \"$(hostname)-${username}@${BACKUP_HOST}::${BACKUP_ROOT}/$(hostname)-${username}/files\""

        # Lower privileges if running as root
        if [ $(id -u) -eq 0 ]; then
                su - "${username}" -c "${backup_cmd}"
        elif [ "$(id -u ${username})" == "$(id -u)" ]; then
                sh -c "${backup_cmd}"
        fi
}

# Back up all home dirs
backup_home_dirs() {
        local home_dir
        
        for home_dir in /home/*; do
                backup_single_home_dir "${home_dir}"
        done
}

All we still need to do is to trigger the various backup functions depending on whether the user is root or not:

if [ $(id -u) -eq 0 ]; then
        backup_system
        backup_home_dirs
else
        if [ "${HOME}" != "" ]; then
                backup_single_home_dir "${HOME}" "$(id -nu)"
        fi
fi

Now the script backs up all files and folders that are specified in ~/.rdiff-backup-filelist to the corresponding account on the backup host if run as a normal user. If the script is invoked as root, it will first back up all specified system files and then run backups for the home directories under the appropriate user accounts if a ~/.rdiff-backup-filelist file exists.

I have created a tar archive with a little more polished version of the script above (download link at the end of the article). Feel free to use and modify it. The archive also contains a server directory containing scripts to automate the creation of backup users. It also contains a script rb-remove-old-increments which you can run as a cron job on the server from time to time to clean up old increments. Basically it does nothing more than running

rdiff-backup --force --remove-older-than <time> <folder>

for each home directory. The server scripts also have a global configuration file to avoid hardcoding of path names etc. The config file is located in /usr/local/etc/default/rb-server-config. A sample file is included in the tarball.

Last adjustments: spinning down the hard drive

The backup system is basically set up, but there is one thing left you might want to do. Especially when you're performing backups of your local system (which I suppose), you'll probably have the hard drive somewhere near you or in another room you use more often. Unfortunately, hard drives make noise and consume a lot of power, but you won't use it all the time, only a few times a day. So it doesn't need to be running all the time.

Some hard drives have an automatic spin-down timeout configured inside the firmware. If not, you can configure one with hdparm -S timeout /dev/device, but oftentimes that doesn't work. But you can use another tool: sdparm (probably not installed by default). You can use this little script and run it periodically via cron to spin down your hard drive after a given amount of time:

#!/bin/sh
# Check if disk has been used since last check and spin it down if not

if [ "${1}" == "" ]; then
        echo "Usage: $(basename ${0}) "
        exit
fi

last_state_file="/tmp/storage-state-${1}"

touch $last_state_file
chmod 600 $last_state_file

new_storage_state=$(cat /proc/diskstats | grep "$1")
old_storage_state=$(cat $last_state_file)

if [ "$new_storage_state" = "$old_storage_state" ]; then
        sync
        sdparm --flexible --readonly --command=stop /dev/$1 2>&1 > /dev/null
fi

echo "$new_storage_state" > $last_state_file

Use it like this: spin-down-storage sda. It will spin down the hard drive if it has been idle since you last ran the script.

Another option (which I tend to) is to use the hd-idle tool, which does basically the same, only more elegantly (a little short note to my fellow Arch users: I had to modify the systemd unit file as described in the comments on the AUR package page to get it to work). One thing I noticed though: after switching the hard drive off and on again by hand, you sometimes seem to have to restart hd-idle.

Downloads:

• rdiff-backup-scripts

Even more of you might know that KDE has a semantic desktop implementation called Nepomuk and some might agree that, although it's a great thing in general, it has very often caused a lot of issues in the past. Especially the whole file indexing engine was very unstable and even unusable for many people, including me.

Now with KDE 4.10 the file indexer has undergone some major changes which made it pretty usable so I decided to switch it on again. It turned out that the first stage indexing works exceptionally well. It indexed about 60,000 files in my home directory in the blink of an eye.

Unfortunately, I had to realize that the second level indexing does not work so well. I remember Virtuoso often eating up all my CPU in the past. Now Virtuoso keeps quiet, but nepomukindexer let's my workstation fly. It only starts indexing when my PC is idle, but for bigger files it keeps the CPU busy at a level of 100%, which is a pretty bad thing. There is already a Bug report about nepomukindexer consuming too much CPU time on larger files, but I didn't want to wait for a fix.

Long story short: I thought of ways to automatically limit the CPU usage of certain processes (not necessarily only Nepomuk).

To accomplish this there is actually a pretty neat tool out there called cpulimit. It does exactly what I need. I give it a PID and a percentage value and it limits the CPU time available for that process. Unfortunately, it only works for processes that are currently running on the system and Nepomuk spawns a new process every time it starts the indexing process.

Therefore I needed a script that runs permanently in the background and automatically limits the CPU usage if necessary. After googling for some time I found a few, but they were more or less overkill for what I needed. So I decided to write my own. It's a dirty hack, but it works.

#!/bin/zsh

MAXIMUM_ALLOWED_CPU=10
LIMIT_THRESHOLD=80
INTERVAL=10
PROCESSES_TO_WATCH=(nepomukindexer)

while sleep $INTERVAL; do
    for process_name in $PROCESSES_TO_WATCH; do
        pid_list=($(pgrep $process_name))
        
        # Don't do anything if no running process found
        [ ${#pid_list[@]} -le 0 ] && continue
        
        for pid in $pid_list; do
            process_info=$(top -b -n 1 -p $pid | tail -n 1)
            
            # Prevent race condition
            echo $process_info | grep -q "^\s*${pid} " || continue
            
            typeset -i cpu_usage=$(echo $process_info | sed 's/ \+/ /g' | sed 's/^ \+//' | cut -d ' ' -f 9)
            
            if [ $cpu_usage -gt $LIMIT_THRESHOLD ]; then
                echo "Limiting CPU usage of process $pid ($process_name)..."
                cpulimit -p $pid -l $MAXIMUM_ALLOWED_CPU &
            fi
        done
    done
done

I thought I'd share it here, maybe someone of you finds it useful too. The variable $PROCESSES_TO_WATCH is an array of all program names this script should watch.

Once started, the script looks every $INTERVAL seconds for new processes in the list being active on the CPU for more than $LIMIT_THRESHOLD percent of the time and limits them to $MAXIMUM_ALLOWED_CPU percent.

I start this script automatically when I log into KDE to bring Nepomuk to terms when it starts turning my PC into a central heating system again.

Update Mar 5th, 7:10 P.M. UTC+1:

I was asked why I don't use Control Groups, which are in fact the proper way of limiting resources for specific tasks on Linux. Well, the elegant answer would be that cgroups only work on Linux, but not on other *nix systems like, e.g., BSD. But okay, that doesn't really count since this is a Linux blog.

The real answer is that cpulimit is just dead-simple and every user can run the above script. Cgroups, however, require root privileges and probably some deeper understanding of how Linux processes work in general. The cpulimit script can easily be started automatically with your desktop session and all the stuff that comes with it is strictly local. The price is a larger overhead, but as long as you don't use it for too many processes and with too short sleep intervals, you shouldn't notice it. If you need a general solution for coordinating resource usage, though, you should of course use cgroups instead.

Update Mar 7th, 5:00 P.M. UTC+1

I updated the script slightly to prevent race conditions which lead to script termination.

Those of you who have been using the RSS feed don't have to do anything, but as an Atom feed subscriber this means you should check your subscription (most people use the Atom feed). Please make sure the feed URL you have saved to your feed reader is http://www.refining-linux.org/feeds/atom10.xml/ (you may also use HTTPS). Until now that URL was redirecting to http://feeds.feedburner.com/RefiningLinux which has now been deprecated.

The old Feedburner URL will redirect to my Atom feed for the next 30 days and then deliver an error 404, so please make sure you update your subscription within that time.

Thank you and please accept my apologies for the trouble.

The device I'll be using for reference throughout this article is my wireless network router (a FRITZ!Box 7390), but the principles apply to nearly any embedded Linux system. I assume you already have gained root shell access to your device. If you haven't, search for ways to get a shell before you continue. My FRITZ!Box has a built-in telnet server which I can use for that, but your system may be different.

Analysis: what have we here?

The first thing we need to do is to find out some information about the system architecture, the Linux version and the file system structure. Every system is different in this respect, so an analysis is mandatory before continuing with any modification.

Find out the kernel version and architecture

In order to be able to compile programs for our embedded device we need to find out the processor architecture. Only very few embedded devices use a normal x86 or AMD64 architecture. Instead most system rely on more power saving but less powerful architectures such as ARM or MIPS. Finding out what architecture we have is easy. Just run the command

uname -nsrm

This prints the OS name, the kernel version and the processor architecture. In my case the kernel version is 2.6.28.10 and the architecture MIPS.

Determine system endianness

The next thing we need to find out is the byte order of the system, i.e. the system endianness. A system is either big or little endian which means that the most important byte is either the first or the last in order. I always found endianness a little hard to understand, but it is actually quite simple because there is a big analogy to human speech. When we write and read numbers in the English language, we use a big endian format, i.e. the most significant number goes first. That means the number 356 is spoken "three hundred and fifty six", but if English was little endian, it would instead be "six hundred and fifty three" because the least significant number comes first. You see: defining endianness is very important and binaries compiled for the wrong endianness won't be able to run and print some weird error messages of some unexpected symbols or characters.

But how do we test which endianness we have? Well, it's not that simple. You won't find the information somewhere in a /proc file and also not (necessarily) in the uname output (however, if you see something like "mipsel" in the OS name, you have a little endian MIPS architecture). Therefore we must find another way. A clever test I found on serverfault.com is this:

echo -n I | od -to2 | head -n1 | cut -f2 -d" " | cut -c6

It creates some two byte octal numbers and analyzes which byte comes first. So 0 means big endian, 1 means little endian. When you run it on your desktop computer you will most probably get a 1 since the Intel x86 architecture as well as the AMD64 architecture are both little endian. But many embedded systems are big endian. Also most network traffic (such as TCP/IP) is big endian so that might be one reason for manufacturers of embedded systems to choose big endian. But the problem with the snippet above is that it might not be able to run on your system because often there is only a very simple Busybox environment with a very simple Ash shell which doesn't know the commands od or head. Another way would be to compile a little C program which does some int/char conversion to find out the endianness (you find many code examples on the Internet) but because you would most probably need to cross compile it anyway (and therefore already know the endianness) I would suggest a third method: just try. Follow the next section by assuming you have a big endian system, if that doesn't work it should be little endian.

Building the binaries

Now the fun part comes: compiling the stuff you want to have on your embedded device. Normally you would need to build all the binaries on the target machine, but those are mostly too weak and hardly ever have a working build environment. So we need to cross compile our stuff. But because building a working cross compiling environment is a though job, there are some ready-to-use tools for it. One of them is BuildRoot. BuildRoot is a complete cross compiling environment with lots of frequently used applications already included. In most cases you only need to select which package you want to build and then run make. To use BuildRoot, first download it and then extract it somewhere on the hard drive of your working machine (not the embedded system).

Next find out the uClibc version of your embedded system (uClibc is a lightweight replacement for glibc) by running

ls /lib/libuClibc-*

on that machine. Mine is 0.9.32 (in case the system does not use uClibc, you need to use an external toolchain in order to successfully build binaries for your target platform).

Once you have extracted everything, grab a terminal and navigate to that directory. Then run make menuconfig. In Target Architecture set the architecture. For MIPS with big endian select mips, for MIPS with little endian select mipsel (arm, i368, x86_64, powerpc etc should be self-explanatory). If you have a special architecture variant, set it in Target Architecture Variant. For my system mips 32r2 is just fine.

The next thing you need to do is to set the toolchain. In my case I set Toolchain ---> uClibc C library version to uClibc 0.9.32.x. If your uClibc version differs, change the value there. In case you use an external tool chain, you have to set the correct version of your particular C library. Also set the kernel headers to a version appropriate to your target device's kernel version (but it doesn't have to be exact, often the newest kernel headers also work just fine for older kernels)

If you want to tweak the settings a bit further, you can do that. Otherwise just select the packages you want to build under Package Selection for the target and then exit and save. Next thing to do is to run make and grab some coffee.

Important! If you want to speed up the build process by using multiple make jobs, you cannot do that by using the -j parameter. Instead you have to change that setting in Build options ---> Number of jobs to run simultaneously.

Once it's all built, you find the binaries in output/target.

Setting up the overlay

Now that we have built the binaries we need to get them onto our embedded system. But that is not as easy as it seems. You could of course just put them on some connected hard drive or some internal writeable memory and start them from there, but often you need to integrate things into the system's root file system and that's what we're going to do now.

The method we're using is inspired by a great blog post about installing additional software on a FRITZ!Box. I developed the method a little further to automate it and made a huge script out of it which you can download at the end of this article.

These are the things we need to do:

1. Order the binaries we want to integrate in a file system structure similar to the root file system.
2. Bind a second instance of the root file system to a writeable directory.
3. Create a second directory on the same writeable partition and mount a tmpfs to it
4. Recursively symlink chosen directories from the alternate rootfs we created in step 2 and our files from step 1 to that tmpfs. We have to be careful with files that already exist in the alternate rootfs and need to choose which version we want to use: ours or the original one.
5. Bind the directories from the symlink overlay to the respective directories in /
6. Enjoy the modified system

Step 1: Create a directory structure similar to the rootfs

This step is more or less optional, but it helps automating the whole process. What's meant by this is that we put all our binaries (and other files) we want to install into directories which mirror the file system structure of the root file system. For instance, to integrate Dropbear SSH into /usr/sbin, we put the executable in a directory usr/sbin/ somewhere on a hard drive readable by the embedded system. So in the end we have the same structure as the rootfs, but only with the files we want to integrate.

For my FRITZ!Box I put the files in a directory on the internal dedicated data partition that is accessible via FTP. This partition is mounted to /var/media/ftp. The directory containing my file system structure I called /var/media/ftp/fboxmod/rootfs.

Step 2: Bind a second instance of the rootfs to some other location

This step is easy. We only need to create a directory somewhere on a readable and writeable file system and bind / to there. On my FRITZ!Box I'm using /var/_altrootfs as the location for this alternate rootfs where /var is a temporary writeable file system (tmpfs). I suggest you also prefer a temporary file system over a persistent one since it makes no actual modification to the system. Alternatively you can also use an external hard drive that is connected to your device.

To bind the rootfs to the new location run:

mkdir /var/_altrootfs
mount -o bind / /var/_altrootfs

(of course you must change the paths as needed)

Step 3: Create the tmpfs for the overlay

For the actual overlay we create a second directory on the same file system we used for binding the alternate rootfs. I called mine /var/_fboxmod-overlay:

mkdir /var/_fboxmod-overlay
mount -t tmpfs tmpfs /var/_fboxmod-overlay

Step 4: Recursively symlink rootfs and custom files to the tmpfs

This is probably the most complex step as we need to symlink each single file to our new overlay directory. What we basically do is to walk through all files recursively and check for each one whether it is an actual file or a directory. If it is a directory we create another one with the same name in our overlay directory. If it is an actual file we create a symlink instead. We need to do this for the directories we want to modify in our alternate rootfs and our custom files we want to integrate.

To make this process a little easier I have written a little shell function for it:

symlink_dir_recursively() {
    local src_dir="${1}"
    local dest_dir="${2}"
    local dir_contents=$(ls -A "${src_dir}")
    
    mkdir -p ${dest_dir}
    
    for i in ${dir_contents}; do
        # Check if file exists
        if [ -e "${dest_dir}/${i}" ] && [ ! -d "${dest_dir}/${i}" ]; then
            if ! $FORCE_OVERWRITE; then
                continue
            fi
        fi
        
        if [ -h "${src_dir}/${i}" ]; then
            cp -d "${src_dir}/${i}" "${dest_dir}/${i}"
        elif [ -d "${src_dir}/${i}" ]; then
            symlink_dir_recursively "${src_dir}/${i}" "${dest_dir}/${i}"
        elif [ -f "${src_dir}/${i}" ]; then
            ln -sf "${src_dir}/${i}" "${dest_dir}/${i}"
        fi
    done
}

With the variable $FORCE_OVERWRITE you can define whether you want existing files to be overridden or not. Run this function for the alternate rootfs and all our custom files:

# The directories for which we want to create an overlay
dirs_to_overlay="bin etc lib sbin usr/bin usr/sbin usr/lib usr/share var/tmp/root"

# Symlink chosen dirs from alternate root file system to overlay
for i in ${dirs_to_overlay}; do
    if [ -e "/var/_altrootfs/${i}" ]; then
        symlink_dir_recursively "/var/_altrootfs/${i}" "/var/_fboxmod-overlay/${i}"
    fi
done

# Symlink mod file system to overlay
overlay_dirs=$(ls -A "/var/media/ftp/fboxmod/rootfs")
for i in $overlay_dirs; do
    symlink_dir_recursively "/media/ftp/fboxmod/rootfs/${i}" "/var/_fboxmod-overlay/${i}"
done

The reason why we're not just symlink / and /var/media/ftp/fboxmod/rootfs is that we can't just bind /var/_fboxmod-overlay to / in the next step without screwing our system (we would then need to pull the plug since the shell would be completely dead saying "Too many levels of symbolic links") and therefore we also don't need to symlink everything. Better focus on those directories we really need to modify and leave the rest alone. You can even shrink down the $dirs_to_overlay list above and only have the directories there you really want to overlay.

Step 5: Bind the overlay directories to the respective directories in /

The last thing we need to do is to apply the overlay. We now have a symlink directory structure in /var/_fboxmod-overlay consisting of the original files from /var/_altrootfs and our own files from /var/media/ftp/fboxmod/rootfs. Let's apply those to the rootfs.

In order to do this we are again using the bind option of the mount command. We simply do a mount -o bind for each top-level directory in our overlay to the respective directory in /. Again: do NOT just bind /var/_fboxmod-overlay to /!. It will kill the whole system.

# Do this for all directories we defined before
for i in ${dirs_to_overlay}; do
	# Skip non-existing mount points in the rootfs
	# Replace this with the part I commented out below if you
	# want to create them instead (only recommended on non-persistent file systems!)
	if [ ! -e "/${i}" ]; then
		continue
	fi
	#if [ ! -e "/${i}" ]
	#	mkdir  "/${i}" 2> /dev/null || (echo "Creation failed, skipping" && continue)
	#fi
	
	mount -o bind "/var/_fboxmod-overlay/${i}" "/${i}"
done

Step 6: Enjoy!

That's it, we're done. You have modified your embedded system in a completely non-destructive way. Once you reboot it, everything is gone and the system is the same as before (except maybe some mount points you created on non-temporary file systems). And the best thing: this method also works for read-only root file systems.

I used this method to install an SSH daemon and some custom scripts on my FRITZ!Box, but since it is quite a lot of work to do and I don't want to spend so much time on it each time I need to reboot the router (happens not that often, but it happens), I wrote a script for it. You can download it from here. The script is quite complex but also quite flexible. What you basically need to change are the paths defined in the beginning:

OVERLAY_DIR="/var/_fboxmod-overlay"
ALTERNATE_ROOT_DIR="/var/_altrootfs"
MOD_DIR="/var/media/ftp/fboxmod/rootfs"
DIRS_TO_OVERLAY="bin etc lib sbin usr/bin usr/sbin usr/lib usr/share var/tmp/root"

You can also change all the other config variables following them, but you don't need to. You can set them dynamically using command line arguments. To view a list of these either look into the source code or run

./init.sh --help

The script is also able to revert the changes it made at runtime without the need to reboot. Simply run

./init.sh --revert

and your system is clean again.

I hope, you found this long blog post useful and as said above: enjoy!

This blog has now been up for a good one and a half year and nothing has changed much since it started. Now it's time to give it a redesign (if you ask me, this was long overdue). While the main appearance stays the same, the details have changed significantly. Let me walk you through the new goodies.

Responsive design

The most awesome feature first: the whole design is now completely responsive and can be viewed at any size. The old desktop-only design has been replaced by a brand-new flexible design for any kind of device, be it a small smartphone, a bigger tablet or a giant 30" display. Refining Linux has it all.

If you like, you can test this. Just resize your browser window and see how great the new design adapts to the new width. The minimum size I consider looking well is about 240 pixels, but there is no upper limit. It just might start looking a bit ridiculous if you project a tiny page onto your 5k display wall.

Nicer typography

Although Georgia and Verdana are not the worst choices I made in my life, they don't look very pretty. They're old veterans of the web-safe fonts battalion and have gotten a bit long in the tooth. Nothing against these fonts in general, but it was time for something different. I mean, we live in the age of web fonts, don't we?

The new choices are Clarendon Paint and Aller, two fresh fonts for the new look and feel of Refining Linux. I like both and think they support the overall painted appearance.

Improved Header

Not only the body has been refurbished, also the header has been pimped. It was actually the last thing I took care of, but that doesn't mean it's not important. It is. Our little painting Tux has become a bit more lifelike, the background painting a bit more realistic. I also took care of the main menu. It got some more detail, less straight edges and some subtle CSS gradients for the hover effect.

In general I haven't modified much in the header, but again, the details have changed. It's perfectly possible that some things in the header might still be tweaked from time to time, but for now I leave it as that.

Redesigned comments section

The comments section has especially been taken care of. It looks very different now and much better in my opinion. The speech bubbles make a lot more sense this way and the overall look is much cleaner and more pleasing to the eye. I hope you like it as well.

The amount of indentation has grown a bit with the redesign, so I have limited the maximum number of indents to four (plus the root level, i.e. five). Comments already being submitted at a higher nesting level retain their position in the database but their display is linearized at the fifth level.

Also the comments form (and the contact form as well) has been redesigned slightly. It also looks a lot cleaner now and I put in some nice HTML5 form validation features there (more on that in a second).

Better syntax highlighting

The syntax highlighting has much been improved. Instead of SH_JS I'm now using Highlight.js which gives me way more accurate highlighting and particularly more control. Especially for the last Advent series I created a special ZSH highlighting scheme. I couldn't really do that in SH_JS without having to learn GNU Source Highlight (I'm too lazy for that).

The new color scheme for the syntax highlighting is based on Ethan Schoonover's famous Solarized theme. Personally I don't like Solarized too much for my editor (the colors are too muted to my taste), but for the website, I believe, the bright theme is great.

Yes, there are still some glitches here and there in the highlighting, but I guess I will be able to fix them over time. The highlighting as it is now is already tremendously better than it was ever before.

HTML5 rewrite

The whole page has been rewritten in HTML5. This wasn't necessary, but I did it in the process of making the whole thing responsive. It now uses semantic HTML5 markup and of course also goodies like HTML5 input type validation for the comments form, a new Canvas tag cloud (replacing the old and ugly Flash tag cloud) and much more.

The tag cloud, by the way, should be completely accessible as it is based on a normal list of links. Users with very small screens, touch based devices or disabled JavaScript will get that version instead. Also screen readers should be able to access the fallback version.

Better legacy IE support

Although I don't care much, a nice side-effect of the responsive design and its “mobile-first” approach is that the support for IE6, 7 and 8 is much better now. IE9 renders everything fine and just lacks support of some minor things such as gradients without SVG. IE8 and older, however, does horrible things to the layout, especially with the new unknown HTML5 elements which I can only get to work with JavaScript. But with the mobile fallback and a few minor fixes I got a more or less decent but at least working version for old IEs. With JavaScript enabled it looks a little less horrible, with JavaScript turned off a little more. But overall it's all working and the two IE users I have will get a very rough but usable basic version of the website (actually, it's a bit more. I have about 7-8% of IE users here of which about a quarter uses IE 9 or above. In December the IE rate was even as low as 5.7%).

Enjoy!

I hope you enjoy the redesign as much as I do. If you like, you can help me a little by doing some browser testing. Although I have tested this on Linux and Windows in many browsers, there might still be some glitches here and there. If you find some, please let me know.

Have fun!

With this initiative Refining Linux is following the protests against the Stop Online Piracy Act (SOPA) and the PROTECT IP Act (PIPA) proposed by US legislators and the media industry. Many huge Internet companies and organizations such as Reddit and Wikipedia participate in these protests. Also companies such as Google, Amazon, Facebook and of course non-profit organizations such as Mozilla and many smaller groups support the protests against SOPA and PIPA.

Why is this so important?

These two bills have the goal to give law enforcement agencies more power to fight against “rogue websites” and copyright infringements in a way that highly endangers free speech and open communication infrastructure. As supporters of open source initiatives and democratic processes we have to intervene and prevent these bills from becoming applicable law.

Fortunately at least the DNS filtering parts of both these bills have been suspended for now due to massive protest from all over the country and around the world, but many evenly dangerous parts persists and the idea behind the whole proposed law isn't dead at all. In fact, it is very much alive and similar attempts are being made behind closed doors in other countries as well. One example is the Anti-Counterfeiting Trade Agreement (ACTA) which is currently negotiated by many countries in the European Union.

Therefore this is not just a matter of US citizens, it is a matter of people around the world.

Why are so many big companies against SOPA and PIPA?

The simple question is: because the Internet is their business. Other than the entertainment industry, which mainly supports SOPA, these companies depend on the Internet. Should these bills become law, they could be held responsible for any third party content appearing on their websites. That means in an extreme case a single search result pointing to an illegal website could be reason enough to sanction Google. The same applies to any other website with user contributed content such as Wikipedia, Reddit, Facebook, any online community, even your private web forum. All posts made to these website had to be checked and approved manually before they appear publicly. That is impossible to do and a massive restraint of free speech and open communication.

The same applies to ACTA and any similar proposed agreement that abuses legal forces to destroy the freedom of people in the name of fighting online delicts.

You should participate as well

Refining Linux protests for a free Internet and so should you. As Sue Gardener wrote in the Wikimedia's announcement to black out Wikipedia (which I linked above):

“The reality is that we don’t think SOPA is going away, and PIPA is still quite active. Moreover, SOPA and PIPA are just indicators of a much broader problem. All around the world, we're seeing the development of legislation intended to fight online piracy, and regulate the Internet in other ways, that hurt online freedoms. Our concern extends beyond SOPA and PIPA: they are just part of the problem. We want the Internet to remain free and open, everywhere, for everyone.”

This is absolutely true. Even if you're not a US citizen (like me) this affects you. Bills like SOPA, PIPA or ACTA must not pass legislation. They destroy open communication around the world and change the Internet in a way we really don't want it to be.

So please sign the petitions against SOPA or ACTA or call your representatives.

Thank you
Janek

Update 01/19/2012 0100 UTC:

Refining Linux is back. Thanks to all who showed their support.

I have shown you many things about ZSH throughout this series, but there is much more you can do with it than I could cover here. And of course there is also much more to configure, many more options I couldn't tell you about, many more tips and tricks, tweaks and optimizations.

Generally, it's a long way to go before you have your shell set up as you like. Especially ZSH needs a lot of configuration before it becomes very user-friendly. You can do all this configuration by hand or you can use a framework for that. Yes, there are frameworks for ZSH (and for Bash as well, in case you didn't know) and as a completion of this Advent series I'll show you two of them.

oh-my-zsh

Probably the most advanced and mighty ZSH framework is oh-my-zsh. oh-my-zsh provides lots of extra features, special plugins with additional functions and completion definitions for certain command line tools and many, many themes for modifying the appearance of your shell, particularly the prompt.

To use oh-my-zsh, download it to a directory of your choice (e.g. ~/.oh-my-zsh) and load it from within your .zshrc:

# Path to your oh-my-zsh installation (the framework won't work without this setting)
ZSH=$HOME/.oh-my-zsh

# Load oh-my-zsh
source $ZSH/oh-my-zsh.sh

That's basically it. You've successfully loaded the framework. But until now it doesn't do much except changing the default prompt to something a bit more meaningful and configuring some basic stuff such as enabling ls colors. But you can customize the framework. For example to disable ls colors again, write the following in your .zshrc before the line where the framework is loaded:

DISABLE_LS_COLORS="true"

To set another theme modify the parameter $ZSH_THEME:

ZSH_THEME="candy"

Now when you re-source your current shell instance you have the candy theme activated. For a full list of available themes have a look at the themes folder inside your oh-my-zsh folder. An even better overview can be found in the oh-my-zsh wiki on GitHub.

Next let's come to the plugins. By default, no plugins are loaded, but that can be changed. Similar to ZSH's $fpath array, oh-my-zsh has a special $plugins array which contains all the names of all the plugins to load when initializing the framework. For instance:

plugins=(git github perl svn)

for loading the plugins git, github, perl and svn which will provide extra functions and completion features for the corresponding applications. For instance, the git plugins adds many advanced completion features for working with Git repositories and the shell function current_branch as a shorthand for displaying the branch you're currently working on without any additional meta data or formatting stuff.

oh-my-zsh is pretty mighty and you can have a lot of fun with it. Just be aware, that even though ZSH provides an autoloading mechanism for functions, loading too many plugins can slow down your shell. It can be very annoying when you always have to wait 10 seconds after opening a new shell instance.

If you're not quite sure how to start with oh-my-zsh, have a look at the example .zshrc in the templates folder.

zshuery

zshuery is another framework for ZSH which is much smaller. It's more something like a micro-framework. The name is derived from jQuery and the idea behind it is to provide a simple and flexible yet fast framework. zshuery doesn't have lots of plugins. It's just one single file which provides some extra functionality and does the basic ZSH configuration for you. That's it. Compared to oh-my-zsh, zshuery is a stub but that's why I like this framework and prefer it over the more powerful oh-my-zsh. It's just what it wants to be: a simple framework which makes working on the shell easier without blowing you away with feature you probably never need. With some extra configuration in my .zshrc this framework is exactly what I need.

But first things first. Loading zshuery is pretty much straightforward. Just download the framework to an arbitrary folder and reference it in your .zshrc:

# Load zshuery
source ${HOME}/.zshuery/zshuery.sh

That loads the framework. Now to let zshuery set some default options, aliases and autocorrection for you, call the following functions:

load_defaults
load_aliases
load_correction

I told you that zshuery doesn't come with plugins and that is true. However, it provides an easy way to load additional completion functions from the zsh-completions Git repository. These are autocompletion functions which might not yet be in the official ZSH release. To include them, clone the repository (preferably to a directory somwhere inside your zshuery folder) and call the following zshuery function:

load_completion ${HOME}/.zshuery/completion

(provided the path to your copy of the zsh-completion repository is ~/.zshuery/completion)

And finally my Christmas gift for you…

zshuery is a great framework for pimping your Z Shell without overloading it. And because it's my favorite ZSH framework and because Christmas is just around the corner, I give you a commented version of my .zshrc in which I use this framework with some slight modifications:

# Load zshuery
source ${HOME}/.zshuery/zshuery.sh
load_defaults
load_aliases
load_correction

# Colorize ls output
alias ls="ls --color=auto"

# Redefine ls colors and load completion definitions
export LS_COLORS="no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:"
load_completion ${HOME}/.zshuery/completion

# Modify key bindings to make certain keys such as DEL, HOME, END, PAGE UP and PAGE DOWN work
eval "$(sed -n 's/^/bindkey /; s/: / /p' /etc/inputrc)" > /dev/null
bindkey "\e[5~" beginning-of-history    # Page Up
bindkey "\e[6~" end-of-history          # Page Down

# Colorize STDERR (enable if needed, might cause issues with password prompts or escape sequences)
#exec 2>>(while read line; do; print '\e[91m'${(q)line}'\e[0m' > /dev/tty; print -n $'\0'; done)

# Enable menu select
zstyle ':completion:*' menu select

# Enable tree view for kill completion
zstyle ':completion:*:*:kill:*:processes' command 'ps --forest -e -o pid,user,tty,cmd'

# Modify zshuery correction prompt
SPROMPT="Correct $fg[red]%R$reset_color to $fg[green]%r?$reset_color (Yes, No, Abort, Edit) "

# Load fancy prompt (works on Gentoo Linux only)
autoload -U promptinit
promptinit
prompt gentoo

# Non-Gentoo users might configure their prompt manually.
# The default Gentoo prompt from above would look like this:
#PROMPT="%B%{$fg[green]%}%n@%m%k%{$reset_color%} %B%{$fg[blue]%}%1~ %# %b%f%k%{$reset_color%}"

# Set options
setopt complete_in_word
setopt path_dirs

# Load modules
zmodload zsh/regex
zmodload zsh/pcre

# Update terminal CWD once and then on every CWD change
update_terminal_cwd
function chpwd() {
        update_terminal_cwd
}

# Extend $PATH
PATH="${HOME}/.local/bin:${HOME}/.local/opt/bin:${PATH}"

Use it for whatever purpose you want. Modify it, extended it, republish it, whatever comes to your mind. You can also use it together with oh-my-zsh or no framework at all if you like that better. Just alter the lines where the framework is loaded or used.

There are two ZSH modules which allow you to easily work with POSIX extended regular expressions (POSIX ERE) or with Perl compatible regular expressions (PCRE) which are even more advanced than POSIX ERE. These two modules are zsh/regex and zsh/pcre. You can use either one of them or both at the same time. That's entirely up to you. I'll show you both.

First let me illustrate zsh/regex a bit which is the simpler one of both. zsh/regex provides, once loaded, the new conditional expression -regex-match which can be used in combination with the [[ command (e.g. in if conditions or loops):

# Load module
zmodload zsh/regex

# Execute POSIX ERE
[[ "foobar_123" -regex-match "^([a-zA-Z0-9]+)_([0-9]+)$" ]] && echo match || echo no match

The condition returns true if the expressions matches, otherwise false. If there are any matches, the whole matching part is stored in the $MATCH parameter and if there are any substrings in parentheses, these parts will be available in the array $match (here in our example you'd have an array with two elements containing foobar and 123).

Now that I've shown you zsh/regex let's come to the more complex zsh/pcre. zsh/pcre also provides a new conditional expression called -pcre-match which works about the same as -regex-match except that it accepts Perl compatible regular expressions. So we could rewrite our example from above as follows:

# Load module
zmodload zsh/pcre

# Execute PCRE
[[ "foobar_123" -pcre-match "^(\w+)_(\d+)$" ]] && echo match || echo no match

That's a little less to write. But zsh/pcre also provides a few new commands besides the conditional expression -pcre-match. The most important ones to know are pcre_compile and pcre_match. With the first one you compile a regular expression from a string and with the second one you use this compiled regular expression on other strings. That means you always need to use both in combination.

Both commands provide several flag parameters. The most important ones for pcre_compile are -m which will match multi-line patterns, -s which makes the dot pattern (.) match whitespace as well and -i which makes the pattern case-insensitive.

The most important flags for pcre_match are -v and -a which let you set different names for the match variable containing the whole matching part and the match array containing all the substrings from enclosing parentheses (which are again $MATCH and $match by default).

Our example from above with pcre_compile and pcre_match would look like this:

pcre_compile "^(\w+)_(\d+)$"
pcre_match "foobar_123" && echo match || echo no match

Sometimes it may be more to write, but it also gives you some more flexibility due to the arguments both commands can take. For example, the following simple case-insensitive regular expression

pcre_compile -i "^foobar\s+\d+$"
pcre_match "fOoBaR   123" && echo match || echo no match

would need such a monster expression if just performed with -pcre-match:

[[ "fOoBaR   123" -pcre-match "^[fF][oO]{2}[bB][aA][rR]\s+\d+$" ]] && echo match || echo no match

In this case the second variant is not just more to type (yes, that's true, count the characters), the first one is also much easier to read and less error-prone so I'd prefer that one.

Whichever variant you take and whether you prefer POSIX regular expressions or PCRE always depends on the situation. But all of them give you the full power of regular expressions. So use them!

Read more about zsh/regex and zsh/pcre:

• zsh.sf.net: The zsh/regex Module
• zsh.sf.net: The zsh/pcre Module
• zsh.sf.net: Conditional Expressions
• Wikipedia.org: Regular expression

Working on the shell is often working with files and sometimes you need to read or edit their contents. Normally you'd do that with the command line editor of your choice (e.g. nano, vi, vim or emacs), but sometimes you need to write the output of a command or a pipe to a file or feed programs with contents from the hard disk. That's usually done by using the input and output redirection operators, but ZSH gives you one more tool which can sometimes make things easier. This module is called mapfile.

Since mapfile is a ZSH module you need to enable it before you can use it:

zmodload zsh/mapfile

Once the module is loaded, you get the magic associative array $mapfile which gives you direct access to any file when you specify its name as the name of the array key. For example, to echo the contents of the file examplefile run

echo $mapfile[examplefile]

You can also write to files by assigning values to an entry of the array. The value will then be written to disk. If the file does not exist yet, it will be created. To write the string Hello World to our file, run

$mapfile[examplefile]="Hello World"

That's the basics and about everything mapfile can do. How do we make use of it? mapfile can sometimes be a nice thing when you need to work with contents of a file in a very simple way. By using mapfile you can avoid piping the contents through chains of commands or doing stuff like this:

filecontents=$(cat file)

mapfile also enables you to do some basic filtering and editing directly when accessing the file by using ZSH's parameter expansion operators. For example if you need the contents of a file converted to all lowercase, the only thing you need to do do is

echo ${(L)mapfile[file]}

or if you need the contents of a file or a fallback value in case the file doesn't exist, mapfile should be the easiest way to go:

echo ${mapfile[file]:-Fallback value}

or if you need to do some initial search and replace:

echo ${mapfile[file]//search/replace}

Of course, you can also write the edited contents back to disk:

mapfile[file]=${mapfile[file]//search/replace}

No big deal. Another thing you could do is to get the length of a textfile in characters (i.e. bytes):

echo ${#mapfile[file]}

Whatever you want! You can also use mapfile in combination with vared to let the user edit files interactively:

vared mapfile[file]

Of course, you could also open some more advanced editor such as vim, but at least as a fallback, mapfile in combination with vared becomes a valuable tool because no external program is required.

You see, under some circumstances, mapfile can really save your neck. But you should also be aware, that it may also consume a lot of memory, particularly with large files. So use it with caution.

Read more about mapfile:

• zsh.sf.net: The zsh/mapfile Module
• ZSH-LOVERS: zsh/mapfile
• zsh.sf.net: Parameter Expansion