NetBSD binary kernel modules usable on Linux in rump kernels

Some years ago I wrote about the possibility to load and use standard NetBSD kernel modules in rump kernels on i386 and amd64. With the recent developments in buildrump.sh and the improved ability to host rump kernels on non-NetBSD platforms, I decided to try loading a binary NetBSD kernel module into a rump kernel compiled for and running on Linux. The hypothesis was that the NetBSD kernel modules should just work since both the NetBSD kernel and Linux processes use the ELF calling convention, and all platform details are abstracted by the rump kernel hypercall layer. Sure enough, after two small fixes to the hypervisor I could mount and access a FFS file system on Linux by using ffs.kmod as the driver.

There is an extra twist for having kernel module binaries work in userspace on amd64. The kernel and kernel modules are compiled with -mcmodel=kernel, which produces code with 32bit offsets. When kernel code is loaded into the top 2GB of the 64bit VAS, sign extension of the addresses gives the correct absolute VA at runtime. We work around this limitation by loading the kernel module into the lowest 2GB. The rest of the rump kernel must reside in the lowest 2GB as well or otherwise the kernel module will not be able to make any calls into the rest of the rump kernel, e.g. to allocate memory with kmem_alloc(). In the instructions below, we adhere to the amd64 2GB limitation by linking all rump kernel components statically so that they get loaded with text instead of with solibs (use /proc/*/maps or pmap(1) to check out what a process's memory map looks like).

The following mini-howto explains how to "do try this at home" on a amd64 or i386 Linux host. Note that ffs.kmod is not included in the build process. It is dynamically loaded and linked into the rump kernel at runtime. It is also possible to let ld(1) do the linking of ffs.kmod, though in that case one detail is left as an exercise for the reader. Of course, using the FFS .so rump kernel component produced by buildrump.sh is also possible, and does not impose the 2GB limitation of kernel modules, but there is no adventure there ...

1. Get buildrump.sh and run it to get the rump kernel components built. Note: you will need a NetBSD-current source tree of at least 20121213 1800 UTC.
2. Get the kernel module binary: ffs.kmod is in modules.tgz. You can use a daily snapshot. For example, I used this for amd64. Note that daily snapshots are available only for a few days after being built, so if the example I gave has expired, find a new one starting from the top-level daily snapshot directory. In any case, you'll need modules.tgz with roughly the same timestamp as the source tree you used in step 1 to ensure both contain the same kernel API version number. Extract modules.tgz and copy stand/arch/version/modules/ffs/ffs.kmod into a working directory.
3. Prepare a test FFS image into "ffs.img". You can use either e.g. makefs or you can download the small image I prepared for this demo from here. Note: the image is modified by the test program, so don't use any image you can't risk losing.
4. Copypaste the source code from below into test.c. Modify it if necessary.
5. Compile with (adjust the paths if necessary): cc test.c -Irump/include -Lrump/lib -Wl,-Rrump/lib -Wl,--whole-archive rump/lib/librumpdev_disk.a rump/lib/librumpdev.a rump/lib/librumpvfs.a rump/lib/librump.a -Wl,--no-whole-archive -lrumpuser -lpthread -ldl -Wl,-E -o test -Wall
6. cross fingers and run ./test

The test code follows:

#include <sys/types.h>

#include <err.h>
#include <inttypes.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <rump/rump.h>
#include <rump/rump_syscalls.h>

int
main()
{
        char buf[8192];
        struct rump_modctl_load ml;
        struct rump_ufs_args umnt;
        int fd;

        /* bootstrap rump kernel */
        setenv("RUMP_VERBOSE", "1", 1);
        rump_init();

        /* register host files to rump kernel */
        rump_pub_etfs_register("/ffsmod", "./ffs.kmod", RUMP_ETFS_REG);
        rump_pub_etfs_register("/ffsimg", "./ffs.img", RUMP_ETFS_BLK);

        /* load FFS kernel module into the rump kernel */
        memset(&ml, 0, sizeof(ml));
        ml.ml_filename = "/ffsmod";
        if (rump_sys_modctl(0, &ml) == -1)
                errx(1, "modctl");

        /* mount file system image (r/w) */
        if (rump_sys_mkdir("/mnt", 0755) == -1)
                errx(1, "mkdir /mnt");
        memset(&umnt, 0, sizeof(umnt));
        umnt.fspec = "/ffsimg";
        if (rump_sys_mount("ffs", "/mnt", 0, &umnt, sizeof(umnt)) == -1)
                errx(1, "mount");

        /* check mount works by reading a file that we "guess" exists */
        fd = rump_sys_open("/mnt/a_directory/README", RUMP_O_RDWR);
        if (fd == -1) {
                warnx("open");
                goto reboot;
        }
        memset(buf, 0, sizeof(buf));
        rump_sys_read(fd, buf, sizeof(buf));
        printf("=== Displaying file contents:\n%s=== EOF\n", buf);

        /* be sneaky: replace string */
#define TESTSTR "You already read me.  You need a real hobby instead of this.\n"
        rump_sys_ftruncate(fd, 0);
        rump_sys_pwrite(fd, TESTSTR, strlen(TESTSTR), 0);

        /* reboot the rump kernel.  unmounts and exists */
 reboot:
        rump_sys_reboot(0, NULL);
        return 0; /* compiler whine */
}

Thanks to Zafer Aydogan for providing access to the [very] fast machine on which it was possible to perform the necessary hacking.