Hacking embedded systems

Posted by | Comment (1) | Trackbacks (0)

Linux is everywhere, not just on desktops. It's on phones, ebook readers, on public terminals, on routers, on electricity meters and many more devices. The key to Linux' success is it's diversity. It is possible to run Linux on nearly every technical device that has a CPU. Many of these are closed systems, so often you don't even notice that Linux is running on that particular device, but there is always a way to gain access to its internals and modify it the way you want. But often you have the problem that heavy modifications might void your warranty or make updates to a more recent firmware version impossible. In this article I want to show you a simple but powerful way to modify such systems in a non-destructive way.

The device I'll be using for reference throughout this article is my wireless network router (a FRITZ!Box 7390), but the principles apply to nearly any embedded Linux system. I assume you already have gained root shell access to your device. If you haven't, search for ways to get a shell before you continue. My FRITZ!Box has a built-in telnet server which I can use for that, but your system may be different.

Analysis: what have we here?

The first thing we need to do is to find out some information about the system architecture, the Linux version and the file system structure. Every system is different in this respect, so an analysis is mandatory before continuing with any modification.

Find out the kernel version and architecture

In order to be able to compile programs for our embedded device we need to find out the processor architecture. Only very few embedded devices use a normal x86 or AMD64 architecture. Instead most system rely on more power saving but less powerful architectures such as ARM or MIPS. Finding out what architecture we have is easy. Just run the command

uname -nsrm

This prints the OS name, the kernel version and the processor architecture. In my case the kernel version is and the architecture MIPS.

Determine system endianness

The next thing we need to find out is the byte order of the system, i.e. the system endianness. A system is either big or little endian which means that the most important byte is either the first or the last in order. I always found endianness a little hard to understand, but it is actually quite simple because there is a big analogy to human speech. When we write and read numbers in the English language, we use a big endian format, i.e. the most significant number goes first. That means the number 356 is spoken "three hundred and fifty six", but if English was little endian, it would instead be "six hundred and fifty three" because the least significant number comes first. You see: defining endianness is very important and binaries compiled for the wrong endianness won't be able to run and print some weird error messages of some unexpected symbols or characters.

But how do we test which endianness we have? Well, it's not that simple. You won't find the information somewhere in a /proc file and also not (necessarily) in the uname output (however, if you see something like "mipsel" in the OS name, you have a little endian MIPS architecture). Therefore we must find another way. A clever test I found on serverfault.com is this:

echo -n I | od -to2 | head -n1 | cut -f2 -d" " | cut -c6

It creates some two byte octal numbers and analyzes which byte comes first. So 0 means big endian, 1 means little endian. When you run it on your desktop computer you will most probably get a 1 since the Intel x86 architecture as well as the AMD64 architecture are both little endian. But many embedded systems are big endian. Also most network traffic (such as TCP/IP) is big endian so that might be one reason for manufacturers of embedded systems to choose big endian. But the problem with the snippet above is that it might not be able to run on your system because often there is only a very simple Busybox environment with a very simple Ash shell which doesn't know the commands od or head. Another way would be to compile a little C program which does some int/char conversion to find out the endianness (you find many code examples on the Internet) but because you would most probably need to cross compile it anyway (and therefore already know the endianness) I would suggest a third method: just try. Follow the next section by assuming you have a big endian system, if that doesn't work it should be little endian.

Building the binaries

Now the fun part comes: compiling the stuff you want to have on your embedded device. Normally you would need to build all the binaries on the target machine, but those are mostly too weak and hardly ever have a working build environment. So we need to cross compile our stuff. But because building a working cross compiling environment is a though job, there are some ready-to-use tools for it. One of them is BuildRoot. BuildRoot is a complete cross compiling environment with lots of frequently used applications already included. In most cases you only need to select which package you want to build and then run make. To use BuildRoot, first download it and then extract it somewhere on the hard drive of your working machine (not the embedded system).

Next find out the uClibc version of your embedded system (uClibc is a lightweight replacement for glibc) by running

ls /lib/libuClibc-*

on that machine. Mine is 0.9.32 (in case the system does not use uClibc, you need to use an external toolchain in order to successfully build binaries for your target platform).

Once you have extracted everything, grab a terminal and navigate to that directory. Then run make menuconfig. In Target Architecture set the architecture. For MIPS with big endian select mips, for MIPS with little endian select mipsel (arm, i368, x86_64, powerpc etc should be self-explanatory). If you have a special architecture variant, set it in Target Architecture Variant. For my system mips 32r2 is just fine.

The next thing you need to do is to set the toolchain. In my case I set Toolchain ---> uClibc C library version to uClibc 0.9.32.x. If your uClibc version differs, change the value there. In case you use an external tool chain, you have to set the correct version of your particular C library. Also set the kernel headers to a version appropriate to your target device's kernel version (but it doesn't have to be exact, often the newest kernel headers also work just fine for older kernels)

If you want to tweak the settings a bit further, you can do that. Otherwise just select the packages you want to build under Package Selection for the target and then exit and save. Next thing to do is to run make and grab some coffee.

Important! If you want to speed up the build process by using multiple make jobs, you cannot do that by using the -j parameter. Instead you have to change that setting in Build options ---> Number of jobs to run simultaneously.

Once it's all built, you find the binaries in output/target.

Setting up the overlay

Now that we have built the binaries we need to get them onto our embedded system. But that is not as easy as it seems. You could of course just put them on some connected hard drive or some internal writeable memory and start them from there, but often you need to integrate things into the system's root file system and that's what we're going to do now.

The method we're using is inspired by a great blog post about installing additional software on a FRITZ!Box. I developed the method a little further to automate it and made a huge script out of it which you can download at the end of this article.

These are the things we need to do:

  1. Order the binaries we want to integrate in a file system structure similar to the root file system.
  2. Bind a second instance of the root file system to a writeable directory.
  3. Create a second directory on the same writeable partition and mount a tmpfs to it
  4. Recursively symlink chosen directories from the alternate rootfs we created in step 2 and our files from step 1 to that tmpfs. We have to be careful with files that already exist in the alternate rootfs and need to choose which version we want to use: ours or the original one.
  5. Bind the directories from the symlink overlay to the respective directories in /
  6. Enjoy the modified system

Step 1: Create a directory structure similar to the rootfs

This step is more or less optional, but it helps automating the whole process. What's meant by this is that we put all our binaries (and other files) we want to install into directories which mirror the file system structure of the root file system. For instance, to integrate Dropbear SSH into /usr/sbin, we put the executable in a directory usr/sbin/ somewhere on a hard drive readable by the embedded system. So in the end we have the same structure as the rootfs, but only with the files we want to integrate.

For my FRITZ!Box I put the files in a directory on the internal dedicated data partition that is accessible via FTP. This partition is mounted to /var/media/ftp. The directory containing my file system structure I called /var/media/ftp/fboxmod/rootfs.

Step 2: Bind a second instance of the rootfs to some other location

This step is easy. We only need to create a directory somewhere on a readable and writeable file system and bind / to there. On my FRITZ!Box I'm using /var/_altrootfs as the location for this alternate rootfs where /var is a temporary writeable file system (tmpfs). I suggest you also prefer a temporary file system over a persistent one since it makes no actual modification to the system. Alternatively you can also use an external hard drive that is connected to your device.

To bind the rootfs to the new location run:

mkdir /var/_altrootfs
mount -o bind / /var/_altrootfs

(of course you must change the paths as needed)

Step 3: Create the tmpfs for the overlay

For the actual overlay we create a second directory on the same file system we used for binding the alternate rootfs. I called mine /var/_fboxmod-overlay:

mkdir /var/_fboxmod-overlay
mount -t tmpfs tmpfs /var/_fboxmod-overlay

Step 4: Recursively symlink rootfs and custom files to the tmpfs

This is probably the most complex step as we need to symlink each single file to our new overlay directory. What we basically do is to walk through all files recursively and check for each one whether it is an actual file or a directory. If it is a directory we create another one with the same name in our overlay directory. If it is an actual file we create a symlink instead. We need to do this for the directories we want to modify in our alternate rootfs and our custom files we want to integrate.

To make this process a little easier I have written a little shell function for it:

symlink_dir_recursively() {
    local src_dir="${1}"
    local dest_dir="${2}"
    local dir_contents=$(ls -A "${src_dir}")
    mkdir -p ${dest_dir}
    for i in ${dir_contents}; do
        # Check if file exists
        if [ -e "${dest_dir}/${i}" ] && [ ! -d "${dest_dir}/${i}" ]; then
            if ! $FORCE_OVERWRITE; then
        if [ -h "${src_dir}/${i}" ]; then
            cp -d "${src_dir}/${i}" "${dest_dir}/${i}"
        elif [ -d "${src_dir}/${i}" ]; then
            symlink_dir_recursively "${src_dir}/${i}" "${dest_dir}/${i}"
        elif [ -f "${src_dir}/${i}" ]; then
            ln -sf "${src_dir}/${i}" "${dest_dir}/${i}"

With the variable $FORCE_OVERWRITE you can define whether you want existing files to be overridden or not. Run this function for the alternate rootfs and all our custom files:

# The directories for which we want to create an overlay
dirs_to_overlay="bin etc lib sbin usr/bin usr/sbin usr/lib usr/share var/tmp/root"

# Symlink chosen dirs from alternate root file system to overlay
for i in ${dirs_to_overlay}; do
    if [ -e "/var/_altrootfs/${i}" ]; then
        symlink_dir_recursively "/var/_altrootfs/${i}" "/var/_fboxmod-overlay/${i}"

# Symlink mod file system to overlay
overlay_dirs=$(ls -A "/var/media/ftp/fboxmod/rootfs")
for i in $overlay_dirs; do
    symlink_dir_recursively "/media/ftp/fboxmod/rootfs/${i}" "/var/_fboxmod-overlay/${i}"

The reason why we're not just symlink / and /var/media/ftp/fboxmod/rootfs is that we can't just bind /var/_fboxmod-overlay to / in the next step without screwing our system (we would then need to pull the plug since the shell would be completely dead saying "Too many levels of symbolic links") and therefore we also don't need to symlink everything. Better focus on those directories we really need to modify and leave the rest alone. You can even shrink down the $dirs_to_overlay list above and only have the directories there you really want to overlay.

Step 5: Bind the overlay directories to the respective directories in /

The last thing we need to do is to apply the overlay. We now have a symlink directory structure in /var/_fboxmod-overlay consisting of the original files from /var/_altrootfs and our own files from /var/media/ftp/fboxmod/rootfs. Let's apply those to the rootfs.

In order to do this we are again using the bind option of the mount command. We simply do a mount -o bind for each top-level directory in our overlay to the respective directory in /. Again: do NOT just bind /var/_fboxmod-overlay to /!. It will kill the whole system.

# Do this for all directories we defined before
for i in ${dirs_to_overlay}; do
	# Skip non-existing mount points in the rootfs
	# Replace this with the part I commented out below if you
	# want to create them instead (only recommended on non-persistent file systems!)
	if [ ! -e "/${i}" ]; then
	#if [ ! -e "/${i}" ]
	#	mkdir  "/${i}" 2> /dev/null || (echo "Creation failed, skipping" && continue)
	mount -o bind "/var/_fboxmod-overlay/${i}" "/${i}"

Step 6: Enjoy!

That's it, we're done. You have modified your embedded system in a completely non-destructive way. Once you reboot it, everything is gone and the system is the same as before (except maybe some mount points you created on non-temporary file systems). And the best thing: this method also works for read-only root file systems.

I used this method to install an SSH daemon and some custom scripts on my FRITZ!Box, but since it is quite a lot of work to do and I don't want to spend so much time on it each time I need to reboot the router (happens not that often, but it happens), I wrote a script for it. You can download it from here. The script is quite complex but also quite flexible. What you basically need to change are the paths defined in the beginning:

DIRS_TO_OVERLAY="bin etc lib sbin usr/bin usr/sbin usr/lib usr/share var/tmp/root"

You can also change all the other config variables following them, but you don't need to. You can set them dynamically using command line arguments. To view a list of these either look into the source code or run

./init.sh --help

The script is also able to revert the changes it made at runtime without the need to reboot. Simply run

./init.sh --revert

and your system is clean again.

I hope, you found this long blog post useful and as said above: enjoy! smile


No Trackbacks for this entry.


There have been 1 comments submitted yet. Add one as well!
iNetGuest wrote on : (permalink)

Very interesting! Thx a lot for your post!

Write a comment:

HTML-Tags will be converted to Entities.
Textile-formatting allowed
Standard emoticons like :-) and ;-) are converted to images.
Design and Code Copyright © 2010-2018 Janek Bevendorff Content on this site is published under the terms of the GNU Free Documentation License (GFDL). You may redistribute content only in compliance with these terms. tweetbackcheck