These are notes how to create a kernel which can be booted in qemu and pass a drive to it which has loadable kernel modules.

The idea

I wanted to play around with the linux kernel and write a minimal filesystem inside the kernel tree. I wanted to boot the kernel in a QEMU, as I do not want to mess with my running kernel.

Here are the steps I did (on a debian VM I run on my universitys infrastructure).

Setup

sudo apt-get install build-essential bc git gcc
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
make x86_64_defconfig
make kvmconfig
make menuconfig

The last step brings up the kernel configuration interface, which I used to enable SATA (Device Drivers -> Serial ATA and Parallel ATA drivers (libata) -> Generic ATA support) support which is needed to be able to mount a device into the qemu instance, where we can put things to play around with in the VM.

I then did a make -j 16 (if you have not 8 cores, use a lower number).

While the kernel compiled, I created a busybox initramfs. I used this blog article for how to do it. I'll dump in the commands from the blog article, for detailed information, go and read it, it's a really good article. I didn't do the speed-optimizing things, as the QEMU boots fast enough for me (~3 seconds).

(I used the latest busybox, but I don't see why it should work with the one from the article, which is about two versions old)

curl http://busybox.net/downloads/busybox-1.23.2.tar.bz2 | tar xjf -
cd busybox-1.23.2
mkdir -pv /tmp/obj/busybox-x86
make O=/tmp/obj/busybox-x86 defconfig
make O=/tmp/obj/busybox-x86 menuconfig # See below why
cd /tmp/obj/busybox-x86
make -j 16

# it does not install in your system, so you can execute this without fear
make install

mkdir -p /tmp/initramfs/x86-busybox
cd /tmp/initramfs/x86-busybox
mkdir -pv {bin,sbin,etc,proc,sys,usr/{bin,sbin}}
cp -av /tmp/obj/busybox-x86/_install/* .

# now edit the init script
vim init

You have to enable busybox build as static binary (with the menuconfig part).

The init script I use:

#!/bin/sh

echo "INIT SCRIPT"

mount -t proc none /proc
mount -t sysfs none /sys
mount -t debugfs none /sys/kernel/debug

mkdir /tmp
mount -t tmpfs none /tmp

mdev -s # We need this to find /dev/sda later

echo -e "\nBoot took $(cut -d' ' -f1 /proc/uptime) seconds\n"

exec /bin/sh

The init script boots into a /bin/sh. We can now build the initramfs with it:

chmod +x init
find . -print0 \
    | cpio --null -ov --format=newc \
    | gzip -9 > /tmp/initramfs-busybox-x86.cpio.gz

Try it out (1)

Now we can try it out, whether the kernel boots and we get out shell:

# In the kernel tree:
qemu-system-x86_64 \
    -kernel arch/x86_64/boot/bzImage \
    -initrd /tmp/initramfs-busybox-x86.cpio.gz \
    -nographic -append "console=ttyS0"

We should see some output like this:

[...]
[    7.170702] Freeing unused kernel memory: 1200K (ffffffff81f55000 - ffffffff82081000)
[    7.171539] Write protecting the kernel read-only data: 14336k
[    7.177328] Freeing unused kernel memory: 816K (ffff880001934000 - ffff880001a00000)
[    7.219360] Freeing unused kernel memory: 1116K (ffff880001ce9000 - ffff880001e00000)
INIT SCRIPT
mount: mounting none on /sys/kernel/config failed: No such file or directory
mount: mounting none on /tmp failed: No such file or directory

Boot took 7.33 seconds

/bin/sh: can't access tty; job control turned off
/ # [    7.630681] input: ImExPS/2 BYD TouchPad as /devices/platform/i8042/serio1/input/input3

/ #

(Yes, the output is slightly messed up... but hey! We just booted a linux kernel in a QEMU!

Load the kernel module

We can now create a kernel module in the linux kernel tree:

# In the kernel tree
git checkout -b my-module
mkdir fs/iamcoolfs/ # because I like filesystems
cat <<EOS > ./fs/iamcoolfs/Kconfig
config COOL_FS
        tristate "COOLFS support"
        help
          Private FS for playing and learning FS programming

          This will definitively eat everything you put into it.

EOS
cat <<EOS > ./fs/iamcoolfs/Makefile
#
# Makefile for the linux mb-filesystem routines.
#

obj-$(CONFIG_MB_FS) += coolfs.o
EOS
cat <<EOS > ./fs/iamcoolfs/coolfs.c
#include <linux/init.h>
#include <linux/module.h>

static int __init coolfs_init(void)
{
        pr_debug("cool module loaded\n");
        return 0;
}

static void __exit coolfs_fini(void)
{
        pr_debug("cool module unloaded\n");
}

module_init(coolfs_init);
module_exit(coolfs_fini);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("<YOUR NAME GOES HERE>");

EOS

This creates all the files we need for a minimal module. Because I like filesystems and I really want to implement a minimal filesystem, I chose to create the files in ./fs, but I guess it shouldn't matter.

Make sure you adapt the fs/Kconfig file to find your new module! Put

source "fs/coolfs/Kconfig"

into it.

Now you can execute make menuconfig and find the configuration option for your new module. We can also simply make M=fs/coolfs to build the module. Now we create a (ext2) disk where we can cp the module to, so we can mount that disk into our QEMU and use the module there:

dd if=/dev/zero of=/tmp/disk bs=4M count=100
sudo mkfs.ext2 /tmp/disk # you could also try ext4, for example
mkdir /tmp/disk-mount
sudo mount /tmp/disk /tmp/disk-mount
cp fs/coolfs/coolfs.ko /tmp/disk-mount/
sudo umount /tmp/disk-mount

Try it out (2)

And when we boot our QEMU now with -hda /tmp/disk, we can find /dev/sda in the booted instance and mount that:

# In the booted VM
mkdir sda
mount /dev/sda ./sda
ls sda

Happy FS dev!

tags: #c #linux #open source #programming #software #testing #kernel

If you think about, the “issue” I ran into is trivial. It really is. Though, I want to write about it because often, these things get too little attention in my opinion.

So what did I do?

I maintain this project task-hookrs. The project itself depends on a few other crates. I updated these dependencies and then released a new version of my crate.

The rewrite included some lines which I had to change, not a big deal actually. One of my dependencies, namingly serde had a breaking change from 0.7.* to 0.8.0 which resulted in a small rewrite of a portion of my code. No functionality was changed, though.

I released 0.2.2 after the dependencies were upgraded. The last version before that version was 0.2.1. And despite me not changing any functionality, the micro-release was a mistake. Not because API changes (which I didn't have) but because of link-time dependencies.

My other project, imag dependes on task-hookrs, but the dependency specification states task-hookrs = "0.2" (note the missing .1 at the end). When the master of imag got rebuild, it failed because imag itself depends on uuid = "0.2", whereas task-hookrs now depended on uuid = "0.3" – the linker failed because the types I passed from imag to task-hookrs were different.

Fuck.

What did I do? Well, I updated uuid in imag as well, everything solved. But that really got me thinking. What I should have done: Updating the minor version (0.2.1 -> 0.3.0).

We talk a lot about breaking changes in APIs and such, in fact this is why we introduced semantic versioning in the Rust community. What we do not talk about is that dependencies are also relevant when updating. Even the semantic versioning website states:

Given a version number MAJOR.MINOR.PATCH, increment the:

1. MAJOR version when you make incompatible API changes,
2. MINOR version when you add functionality in a backwards-compatible manner, and
3. PATCH version when you make backwards-compatible bug fixes.

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

Which clearly states that MAJOR version when you make incompatible API changes – though nobody tells you that updating a dependency of your library might also be a incompatible API change!

After all I'm (kinda) safe here, because the semantic versioning spec also contains the following:

Major version zero (0.y.z) is for initial development. Anything may change at any time. The public API should not be considered stable.

So nobody can blame me. Although this really bugs me and it really shouldn't have happened.

tags: #programming #open source #software #testing

But how to start?

The first steps are easy. You make a new directory, maybe a directory structure. You create some files, such as a README.md or a LICENSE file. You create a MyGreatSoftware.MyFavouriteLanguage file and you open it in your favourite text editor or IDE.

But then you struggle and you don't know what to do first. Let's imagine you want to write a commandline application – do you start with the argument parsing part or do you add some backend abstraction in the first step? Do you build up a great framework around some functionality so you can use it more easily later on or do you implement functionality directly?

I had such problems, too. I started a project for a rather simple commandline application and I really didn't know where to start. It was a ruby project, so Imy initial steps where rather clear: I installed some dependencies I wanted to use, such as an option parsing library and a commandline output helper for pretty formatted output (colored things and so on). I created the directories lib and bin – and then the file lib/app.rb (where “app” is the name of the program.

And then I failed to continue because I didn't know what to implement first. So I ended up implementing the commandline parsing part and the backend abstractions at the same time (on different branches, of course) and it resulted in me deleting the project because things got complicated.

Then, I restarted the project and did something I've never done before: I started to do TDD and I started with the very first line of code.

Disclaimer: I'm not a TDD professionalist. I never had any training in TDD and I don't even know whether I did it right. But it's not the point whether you do TDD right or not here. The point is, you think about simple, neat pieces of code with a lot more focus. You try to imaging what explicit feature you want to have. For example your test says something like this:

If I pass an argument --update a File ~/.app_timestamp gets the current time is appended.

That's a really really compact thing and you can implement this in one single line of Ruby. And that's exactly my point. You start implementing things without building layers and layers of abstraction and losing yourself in the codebase you created yourself for not losing track of things.

I learned that writing tests and implementing just the very functionality that makes the test succeed helps me a damn lot to focus on the functionality I want to build rather than building layers and layers and afterwards having a kind of framework which does nothing (and is mostly even too complicated to use).

Of course, after implementing things you can clean up your codebase and refactor things, so the abstraction layers will be created at some point. But that's after the features have been implemented. And a damn great bonus: You have tests which are there to ensure you don't break your functionality by refactoring and making yourself happier with the codebase.

So that's what I call Test-Driven Project Initialization. You don't need to perform perfect TDD and stick to it all the time. You don't need to test every aspect of your functionality. The tests are there to keep you focused on the functionality, not to ensure there are no bugs. Of course, that's a neat side effect and you can continue and extend your test cases later on to find bugs. But for Test-Driven Project Initialization it is not the point to find bugs.

tags: #programming #software #testing