Google Summer of Code 2026 Reports: Testing Compat Linux: Syscall testing, part 2

This report was written by Henrique Brito as part of Google Summer of Code 2026.

It is the 2nd part of a series. You can read the 1st part at Google Summer of Code 2026 Reports: Testing Compat Linux: Syscall testing.

Introduction & Description

compat_linux(8) is NetBSD's kernel-level compatibility layer for Linux binaries. It allows running them through a thin, in-kernel translation layer, instead of full emulation. One of its core functions is to simulate Linux syscalls for those programs.

Until now, this functionality was not as well tested as other NetBSD components. The goal of this project was to use Linux's own test suite, LTP (Linux Test Project), to verify the usability and compatibility of compat_linux.

LTP is a well-known and battle-tested framework, already used to validate Linux behavior, so why not use it to also validate NetBSD's compat_linux?

What was done

In this project, I made LTP tests compile under NetBSD, run (as they were designed to be run), documented some of the more relevant syscall emulations, and found some very interesting behaviors involving compat_linux.

Compile tests under compat_linux

This step involved fixing and using suse15_gcc12, which was recently pulled from pkgsrc-wip into pkgsrc. Fixing it involved adding some missing RPMs packages with relevant code, including:

• linux_glibc_devel: adds many Linux headers, essential to compile the low-level LTP tests
• glibc_devel_static: required to achieve static compilation with gcc

Creating a script/package

To be able to correctly run the tests on NetBSD, it was necessary to create a 'manager' script to fill in the gaps between the two operating systems. In order to ship the script and automate the download and compilation of LTP, we decided to create a package. Fortunately, the pkgsrc phases were very similar to what we needed: a fetch to download LTP, a build to compile it, and an install to install it.

The compat_linux_test_project package isn't merged into the official pkgsrc tree yet, but you can find it on my personal fork here in the meantime.

Test Documentation

For the syscalls we considered more fundamental (as in, used more often, by more applications), a more in-depth analysis and testing was made. Those tests were then cataloged and neatly documented in a spreadsheet for future reference. I also added some suggestions for eventual future fixes.

Logs, configuration files and other GSoC project artifacts

All the raw logs and related files used during the project are stored in an online drive, free for anyone who wants to see them.

The final test

To finish with a better understanding of the current state of compat_linux, I ran the full LTP syscall test suite on compat_linux. During this phase, many interesting and relevant results were found. They are detailed in the 'Results' section below.

Challenges

At first, I thought this would be a pretty straightforward project, but it turned out to be more challenging and fun than I had in mind. During this GSoC, I encountered many small and big challenges, some of them are listed here:

The first one was "how to compile LTP". My original suggestion was to not compile the tests on NetBSD and instead compile them statically on a Linux machine and use the resulting binary. For many different reasons, this was not a suitable idea. My mentor, Stephen, then suggested we could use the suse15_gcc12 package. This was a great idea, and a bit funny too: bringing a compiler from Linux and, through compat_linux, using it to compile tests that verify the usability of compat_linux. Fortunately, the package was already in good shape, only requiring the addition of some missing RPMs packages to make it work.

The second one was about "how to convert my script into a package for pkgsrc". My first approach when creating the script was to automate it as much as possible, so it would check the LTP version, download it, compile it, etc. All in one script. When moving to pkgsrc, I realized that many of those things were already implemented inside the framework. So I ended up having to move my logic into pkgsrc. Pkgsrc is a very complete and pleasant package system to work with, so I was able to navigate and use it even though I didn't have any experience with it (with the help of my mentor, of course).

The third one was about "whether to compile statically or dynamically". Dynamic tests are smaller, but require the suse15_base package to run, unlike the static ones, which don't require anything else. So at first, it seemed that static was the way to go with the compilation, but when we checked the package sizes for both of them an issue appeared: the static package was about 5GB, while the dynamic one was 545MB. Not very well suited for a testing package. After some work with gcc flags, I was able to decrease the dynamic package's size to 80MB.

Results

Running the testcases in compat_linux provided many useful insights about its current state. From my perspective, it is very complete and covers most of what is commonly needed/used. It's only when Linux syscalls start to stray from POSIX, with Linux-exclusive functionality or different behaviors, that compat_linux starts to lack coverage, since implementing those functions would basically require 're-creating' the syscall inside compat_linux.

/proc/self/maps difference

Running the mmap04 testcase, I found that it failed due to LTP not being able to read correctly from /proc/self/maps.

This happened because Linux exposes addresses with hex digits of variable length, trimming away the left-most zeros (maintaining a minimum of 8 hex digits), while NetBSD uses a fixed size of 16 hex digits to display the addresses.

Linux:

NetBSD:

According to mount_procfs(8) manpage, maps should be implemented "in a form like the proc filesystem as implemented in Linux", so this is a divergence in behavior.

See thread on the tech-kern@ mailing list.

writev and iovcnt

The emulated syscall writev(2) has a divergence in behavior, but this difference is unlike most of what we have found.

According to POSIX, writev should yield an EINVAL when given an iovcnt of 0. This is followed by NetBSD and, by extension, its compat_linux implementation. On the other hand, Linux displays different behavior, where it doesn't recognize iovcnt = 0 as an error, and instead returns success.

SMAP violation on copy_file_range

This syscall emulation was implemented in GSoC 2024, but only now is being incorporated into the mainstream versions of NetBSD, starting with NetBSD 11. The issue is a simple dereference of a user space pointer inside kernel space, without the proper handling, which triggers a SMAP violation, causing a kernel panic.

This only happens when off_in or off_out is not NULL, an edge case that was not caught during implementation.

Lets take a look at the code responsible for this:

 if ((SCARG(uap, off_in) != NULL && *SCARG(uap, off_in) < 0) ||             <- dereference
     (SCARG(uap, off_out) != NULL && *SCARG(uap, off_out) < 0) ||           <- dereference
     vattr_in.va_type != VREG || vattr_out.va_type != VREG) {
  error = EINVAL;
  DPRINTF("%s: Invalid offset or file type\n", __func__);
  goto out;
 }

Here, *SCARG(uap, off_in/off_out) is attempting to dereference a user-space pointer from kernel space. I think this is a good example of why we should use LTP to verify the compatibility and functionality of new syscall implementations in compat_linux.

As of writing, the copy_file_range fix hasn't landed in the NetBSD tree yet, but you can find the patch here in the meantime.

Killing userland via rt_sigqueueinfo ;)

Perhaps the funniest thing that I have encountered in this project. This was found while testing the rt_sigqueueinfo syscall. I noticed that something strange was happening with it, similar to copy_file_range, but not quite. When running the LTP tests, my SSH session would disconnect, the script would stop, and everything else would halt, only coming back to normal after a reboot. At first I thought this was a kernel panic, but nothing seemed to be happening: nothing in dmesg, no panic screen, nothing. So I had to check the compat_linux implementation of rt_sigqueueinfo and the LTP testcases. The testcase in question that was causing trouble used a tgid of -1, which was meant to represent a tgid that doesn't exist. The compat_linux implementation, unlike Linux's, used kill(2) as part of its logic. Connecting the dots, I noticed that kill(2) has a special use case for -1, which is used to broadcast the signal. It just so happens that the testcase, combined with the syscall emulation, was killing all processes as a side effect. We are still discussing what should be done in this regard.

About the script

As mentioned above, properly running the LTP tests on NetBSD required some sort of 'glue' to work out the differences between both operating systems. This was solved by creating a dedicated script. Installed as compat_linux_test_project, it can run the full syscall suite or a chosen subset, produce structured, per-syscall logs, and compare a fresh run against a baseline set of logs to highlight regressions, fixes, and other changes. Testcases known to hang or panic the kernel are skipped by default, though they can still be run explicitly when needed.

How to install

Since the package isn't merged into the official pkgsrc tree yet, you can build it directly from my fork:

git clone https://github.com/henriquebritoM/pkgsrc.git
cd pkgsrc
git checkout compat_linux_test_project
cd emulators/compat_linux_test_project
make install

How to use

One of the most common use cases is to run the tests for a syscall (or a group of them)

To do this, you can just do

compat_linux_test_project -s readv,writev,open,close -d output_dir

This will run all testcases related to the syscalls readv, writev, open and close and store the resulting logs inside output_dir.

The contents of output dir should be something similar to:

# tree -F output_dir/
output_dir/
|-- close/
|   |-- close01.log
|   `-- close02.log
|-- open/
|   |-- open01.log
|   |-- open02.log
|   |-- open03.log
|   |-- open04.log
|   |-- open06.log
|   |-- open07.log
|   |-- open08.log
|   |-- open09.log
|   |-- open10.log
|   |-- open11.log
|   |-- open12.log
|   |-- open13.log
|   |-- open14.log
|   `-- open15.log
|-- readv/
|   |-- readv01.log
|   `-- readv02.log
`-- writev/
    |-- writev01.log
    |-- writev02.log
    |-- writev03.log
    |-- writev05.log
    |-- writev06.log
    `-- writev07.log

4 directories, 24 files

For more tips, you can use the help option to list the full package use description:

compat_linux_test_project --help

Expected output:

NAME
 compat_linux_test_project - package to run LTP tests on NetBSD

SYNOPSIS
 compat_linux_test_project [options]

DESCRIPTION
 compat_linux_test_project is a package to download, build, and run
 the Linux Test Project (LTP) test suite on NetBSD, intended for
 testing the compat_linux compatibility layer.

 It automates fetching, building, and executing LTP tests, helping
 to identify missing syscalls, behavioral differences, and other
 issues in NetBSD's Linux compatibility subsystem.

 If no --syscall option is passed, the default behavior is to test
 every possible syscall. This uses the LTP runtest/syscalls file to
 determine what to test and in which order. This means that the syscall
 name cannot be correctly-determined sometimes, resulting in suboptimal
 output in the logs. This also takes some time and is not
 recommended, except in some niche cases.

 All tests belong to Linux Testing Project, this package only ports
 them to make them run on NetBSD. Support the Linux Test Project by
 checking their official website:
  https://github.com/linux-test-project/ltp

OPTIONS
 -d, --output-dir output_dir

  Directory where output is written.
  In test-run mode (default): new logs are stored here,
  overwriting any older logs already there.
  In comparison mode (-c): the categorized comparison results
  (compared_logs/) are stored here instead.

 -s, --syscall syscall1[,syscall2,...]
  A comma-separated list of which syscalls to test.
  The script will search for them using the LTP runtest file,
  so the name may be slightly different from the syscall name
  (though this is unusual).

  WARNING: some syscalls are excluded from the default (full)
  run because they are known to cause problems (e.g. kernel
  panics or hangs, see check_for_testcase_issue() in the
  script). If you explicitly list one of these syscalls here,
  the script will still run it, but will print a warning first.

 -r, --reproducible
  Sets the LTP environment variable LTP_REPRODUCIBLE_OUTPUT to 1.
  According to LTP documentation, this "suppress printing TINFO
  and TDEBUG messages and discards the actual content of the
  other messages printed by the test (suitable for a
  reproducible output)."

  This flag should be set if the logs are meant to be compared.

 -c, --compare logs_dir
  Compares logs_dir against a baseline, highlighting
  their differences. For a consistent comparison, both
  sets of logs should have been gathered using the
  'reproducible' mode ('-r' flag).

  The baseline defaults to the reference logs shipped with the
  package, or to whatever is passed via '-b'.

  If '-s' is also passed, the comparison is restricted to the
  syscalls listed there.

  The comparison is done as follows:
  Checks for tests that are new, tests that are no longer
  present, and tests whose result changed. Changed results are
  further split into regressions (e.g. PASS -> FAIL), fixes
  (e.g. FAIL -> PASS) and other changes (e.g. shift in skipped
  or warnings). Some testcases are known to depend on
  functionality with no equivalent in NetBSD and are marked
  as wont_test.

  Each category is stored in its own subdirectory, mirroring the
  syscall/testcase layout of the compared logs:
   compared_logs/
   |-- changed/
   |-- fixed/
   |-- new/
   |-- regressed/
   |-- removed/
   |   |-- syscall_a/
   |       |- testcase01.log
   |-- wont_test/

  Each generated testcase01.log file is self-contained:
  - For 'new'/'removed' testcases, it has a short status line
    followed by the full content of the one log that exists
    (current or reference, respectively).
  - For 'changed'/'fixed'/'regressed' testcases, it has the
    reference and current pass/failed/broken/skipped/warnings
    counts, followed by the full content of both the reference
    and the current log.
  This means you normally do not need to go dig through the
  original sys_logs/ or reference logs directories to
  investigate a specific result; everything relevant is already
  copied into diff_logs/.

  The comparison assumes the testcases output follows the
  standard and new LTP structure. This is not true for every
  testcase, in this case, the comparison has undefined
  behavior.

 -b, --baseline logs_dir
  Directory with the logs to use as baseline. Only meaningful
  together with '-c'. Defaults to the reference logs shipped
  with the package.

 --fail-on-regression
  Only meaningful together with -c.
  When this flag is set, the script exits with a non-zero status
  if any test is found in compared_logs/regressed/. This is
  intended for use by automated testing.
  Has no effect without -c.

 -h, --help
  Shows this message

Examples

Let's say you want to test the read and pipe syscalls:

You can use:

compat_linux_test_project -s read,pipe -d /tmp/pipe_and_read

Expected output (trimmed):

Preparing block device required by some testcases...
Mounting block device vnd0 from image /usr/pkg/libexec/compat_linux_test_project/test.img...
Starting test run. This may take a while...
Output will be stored in: /tmp/pipe_and_read
==> Testing syscall: read
syscall_test: read01

[snip]

Summary:
passed   1
failed   0
broken   0
skipped  0
warnings 0

[snip]

==> Testing syscall: pipe
syscall_test: pipe01

[snip]

Summary:
passed   1
failed   0
broken   0
skipped  0
warnings 0

[snip]

Unmounting block device vnd0...
Test run finished. Logs stored in: /tmp/pipe_and_read

And then you can check out the results of each testcase by navigating through the output dir:

# cat /tmp/pipe_and_read/pipe/pipe01.log
---- compat_linux_test_project log header ----
date: Fri, 4 Sep 2026 00:33:38 +0000
netbsd_version: 11.0
netbsd_build: NetBSD 11.0 (GENERIC) #0: Thu Jul 30 15:23:12 UTC 2026  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/amd64/compile/GENERIC
arch: amd64
ltp_version: 20260529
script_invocation: "/usr/pkg/bin/compat_linux_test_project '-s' 'read,pipe' '-d' '/tmp/pipe_and_read'"
syscall_tested: pipe
reproducible: 0
--------------------------------------------------

[snip]

pipe01.c:48: TPASS: pipe() functionality is correct

Summary:
passed   1
failed   0
broken   0
skipped  0
warnings 0

Conclusion

This GSoC project started with the idea of bringing the LTP testing framework to compat_linux and NetBSD, and along the way grew into a proper pkgsrc package capable of bridging the gaps between both operating systems, an in-depth documentation and analysis of some of the most commonly used syscalls, and a clear view of exactly where compat_linux diverges from a native Linux environment. Two of those divergences turned out to be rather worth-noting issues: a kernel panic in copy_file_range, and a way to kill userland via rt_sigqueueinfo.

With this, I think we have a good reminder of how important a good and thorough test suite really is, even for a compatibility layer that has been around for years.

I'd like to thank my mentor, Stephen Borrill, for his guidance and patience throughout the project, and for pushing me toward better solutions whenever I got stuck. Thanks also to Christos, who helped to fix the copy_file_range panic, to Leonardo for his advice and excellent management of many GSoC-related subjects, and to all the other mentors and students who have been working alongside me and NetBSD during this GSoC.

It's been a genuinely fun summer of digging into two operating systems at once. I've learned a lot about how they really work under the hood, in places where it isn't obvious. And it sure did help me pass my Operating Systems class haha.

[0 comments]