Google Summer of Code 2026 Reports: Improving and Stabilizing the racoon2 IKE Daemon in NetBSD

This report was written by Artem Belan as part of Google Summer of Code 2026.

Introduction & Description

racoon2 is a system to exchange and install security parameters for IPsec. It consists of an IKEv1/IKEv2 key exchange daemon, a security policy management daemon, and a Kerberos-based key exchange daemon. The project lives at zoulasc/racoon2 and is being modernized so that it can serve as a reliable L2TP/IPsec or IKEv2 VPN server on NetBSD and Linux for built-in Windows, iOS and Android VPN clients.

The goal of this GSoC project was to work through that TODO list: implement NAT traversal properly for both IKEv1 and IKEv2 according to the relevant RFCs, make IPv6 works well so racoon2 can be tested with both IPv4 IPv6 properly, clean up the address-macro mess, and build a unit test framework so that the fixes stay fixed. In the end, the daemon can sit behind a NAT device and serve modern VPN clients without any workaround selectors in its configuration, IPv6 works out of the box, and the changes are protected by an automated test suite.

All work was done on the gsoc2026 branch and merged upstream through pull requests #13–#39 between June and September 2026.

What was done

IKE fragmentation (RFC 7383)

The fragmentation code was, in the words of the TODO, "old/incomplete": reassembled messages could be misrecognized by the daemon, and oversized packets could crash it. The result of the project is reliable RFC 7383 fragmentation for both IKEv1/IKEv2 on both the sending and the receiving side: large IKE messages are split and put back together correctly, packets that exceed the allowed fragment size are rejected with a clear log message instead of a crash, and exchanging big messages — certificates, large proposals — between the peers no longer silently fails or takes the daemon down.

NAT-OA payloads in IKEv1 Quick Mode (RFC 3947)

At the beginning of the summer the daemon ignored NAT original address payloads on input and never sent them to the peer. The consequence was that the kernel had no idea which addresses the client actually used behind the NAT, so it had to recompute checksums over entire packets — slow, and not what RFC 3947 prescribes. Now the NAT-OA payloads match the RFC's structure, are constructed and sent in Quick Mode, and are parsed positionally on receipt (local address first, peer address second), with a missing payload from the peer tolerated rather than treated as an error. The original addresses are also passed down to the kernel so it can do incremental checksum fixup instead of full recomputation.

Address substitution in NAT-T transport mode (RFC 7296 §2.23.1)

This was the heart of the project and the direct answer to a long-standing TODO item: when a NAT device was in the path, the addresses the peer proposed in phase 2 did not match the responder's configured selectors, so responders needed extra selectors that were valid only on the initiator's side just to pass the selector check — configurations like transport_ike_natt.conf carried them as a workaround, and connections from real clients stayed fragile. The final behaviour follows RFC 7296 §2.23.1: when NAT-T is enabled and the addresses observed on the wire do not match the configured selectors, the observed addresses are substituted into the negotiated selectors, on both IKEv1 and IKEv2. The workaround selectors are no longer needed, and the daemon now properly handles NAT-T traffic on the dedicated UDP port instead of only the initial one.

Address macros in configurations

The wildcard address macros were, in the TODO's words, "the cause of many configuration-related bugs": what worked in an IKEv1 configuration could silently misbehave in IKEv2, one macro behaved as a wildcard in some code paths and as "unknown, wait until we know" in others, and misconfigurations were ignored rather than reported. The project unified them into a single consistent wildcard notion that behaves the same way everywhere, works for both IKEv1 and IKEv2 configurations, and produces clear diagnostics when an address lookup or expansion fails, instead of failing silently and leaving the user to guess why the policy never got installed.

IPv6 support fixed and enabled by default

IPv6 support had existed in racoon2 for years, but it worked crooked: when the interface was configured to use all addresses, the daemon still behaved as if only IPv4 addresses existed — as though the interface were set to use IPv4 only — and prefix-length edge cases were broken, so in practice racoon2 could not be tested with IPv6 at all. During the project these bugs were fixed — address matching, interface address validation on Linux, prefix-length handling — and IPv6 became what a modern daemon should have been: used by default, with no configuration switch to remember. After the project, the same configuration files simply work with IPv6 addresses, and the TODO item about IPv6 is closed on the configuration side; only end-to-end testing with real IPv6 traffic remains.

Policy and proposal negotiation

Before the project, transport mode policies were generated from the generic SA endpoints rather than the addresses configured for the SA, tunnel mode policies could reference the wrong endpoints, and the daemon assembled the negotiated IPsec proposal itself instead of accepting what the peer actually proposed — a frequent source of failed negotiations with real clients, which all bring their own proposals. Now policies in both transport and tunnel modes are generated from the configured SA addresses, and the negotiated proposal is built from the peer's proposal, so interoperability with stock Windows, iOS and Android clients no longer depends on the local configuration guessing everything right.

A unit test framework for the IKE daemon

The project had no automated tests at all; every change had to be validated by reading code and running the daemon by hand. The last part of the summer went into changing that: a small self-contained test framework with no external dependencies is now integrated into the standard build and its tests run under sanitizers. On top of it, four new test programs cover exactly the logic this project introduced — NAT original address payload handling, traffic selector address substitution, wildcard address handling in configurations, and the address matching used when a peer is purged — and the pre-existing crypto self-test was fixed so the whole suite passes cleanly. This matters beyond the project itself: the framework makes it straightforward to add regression tests for the parts of the daemon that still have none, whoever tackles them next.

Documentation, samples and NEWS

Documentation and sample configurations still described the old defaults, so the manual contradicted the shipped behaviour: it still described the old IPv6 behaviour and the pre-substitution selector workarounds. The project synced the docs and samples with the new defaults — IPv6 used by default, fragmentation handled automatically, tunnel endpoints and wildcard addresses used consistently — and added a NEWS entry summarizing all of the GSoC 2026 work for users and downstream packagers.

How to Use

Everything described above is on the gsoc2026 branch of ssszcmawo/racoon2 (merged upstream into zoulasc/racoon2). A quick tour for anyone who wants to try it:

Build and install. The repository ships only the autotools sources, so the configure script has to be regenerated first — this is the same flow documented in doc/INSTALL and used by the project's CI:

autoreconf -fi
./configure
make
make install

You will need the usual developer toolchain (a C compiler, GNU make, autoconf, automake and libtool, plus flex and bison) and the OpenSSL headers; if you want the kinkd daemon you also need a Kerberos 5 library (MIT krb5 or Heimdal), and the optional packet-dump debug mode additionally wants libpcap. On Debian/Ubuntu the CI installs exactly automake libtool libssl-dev libkrb5-dev libpcap-dev flex bison before running the commands above. Useful configure options are --disable-iked / --disable-kinkd to skip a daemon, --with-krb5=<dir> for a non-standard Kerberos, and --enable-pcap for the packet dumps; --prefix=$NEWDIR moves the installation away from the default /usr/local.

Configure. Everything needed ships in the repository under samples/. The .in files are templates that get their install paths filled in during installation, and the scenario fragments are plain files. For an L2TP/IPsec transport-mode server — the scenario this project was about — the flow is:

1. Take samples/vals.conf.in as your vals.conf and set your environment in it: MY_IPADDRESS and PEERS_IPADDRESS (or the IP_ANY wildcard when the peer's address is not known in advance), MY_PUBLIC_IPADDRESS if you are behind NAT, and the shared-key settings. Generate the key itself with pskgen.

2. Take samples/racoon2.conf.in as your racoon2.conf. It includes vals.conf and default.conf and defines the interfaces; the sample ships with NAT-T commented out, with instructions in place:

   interface
   {
       ike {
           MY_IP port 500;
   #       MY_IP port 4500;   # uncomment to enable NAT-T
       };
       ...
   };
   

3. Uncomment the one include line for the scenario you want — for this case transport_ike.conf. That file holds the remote section (IKEv1 and IKEv2, pre-shared key), the selectors and the policy:

   selector ike_trans_sel_out {
       direction outbound;
       src "${MY_IPADDRESS}" port 1701;
       dst "${PEERS_IPADDRESS}" port any;
       upper_layer_protocol "udp";
       policy_index ike_trans_policy;
   };
   ...
   policy ike_trans_policy {
       action auto_ipsec;
       remote_index ike_trans_remote;
       ipsec_mode transport;
       ipsec_level require;
       ipsec_index { ipsec_esp; };
   };
   

   and, if you are behind NAT, its own commented include line pulls in transport_ike_natt.conf with the additional selectors for common private address ranges.

Two details worth noting: the wildcard address in vals.conf (IP_ANY) now behaves consistently, so a server can accept a peer whose address is unknown in advance; and IPv6 now works in the same files as is — put IPv6 addresses in and they are used, with no switch to flip.

The other samples follow the same pattern: tunnel_ike.conf and tunnel_ike_natt.conf for tunnel mode, transport_kink.conf and tunnel_kink.conf for Kerberos-based key exchange, and local-test.conf for trying two daemons against each other on one machine.

Run it. Start spmd first (it owns the policy channel to the kernel), then iked with your configuration — ./configure puts both in place, and samples/rc.d/ has service scripts for NetBSD-style systems. Once both are up, traffic matching a selector with action auto_ipsec (like the one above) triggers the negotiation automatically, so on a client you just connect with the built-in Windows, iOS or Android VPN client using the server address and the shared key. The daemon logs the NAT detection and the substitution it performs, which makes it easy to see what happened if a client does not come up.

Follow the work. All changes are in pull requests #13–#39 on the gsoc2026 branch, and the NEWS file has a single consolidated entry describing the user-visible changes. The top-level TODO file lists what is still open — see the "What remains" section below.

Challenges

• Old, untested code. racoon2 has been around since the WIDE project days and had no tests. Every change had to be validated mostly by reading code, running the daemon by hand under a debugger, and watching the sanitizers — which is exactly why the test framework became part of the project rather than an afterthought.
• RFC compliance vs. working configurations. Implementing address substitution meant first understanding why responders needed those extra initiator-only selectors before being able to remove the need for them, and getting the substitution to happen at the right point of the IKEv1 negotiation took several attempts.
• Address macros are deceptively deep. The wildcard macros looked like a simple rename, but in some places they mean "the address is not known yet and the kernel policy must wait", while in others they do not. Unifying them without breaking either case required careful tracing through configuration parsing, selector matching and policy updates.
• Build system mistakes. An optional-build patch for the daemons imported from elsewhere looked harmless but broke the build and had to be reverted — a good reminder to actually build the tree, not just read the diff.
• Sanitizer-driven debugging. Building with sanitizers turned up memory-safety bugs in paths that had "worked" for years; fixing them without changing observable behaviour meant carefully tracking ownership of every duplicated buffer.

Results

• All work merged upstream into the gsoc2026 branch through pull requests #13–#39.
• Every item on the TODO list is addressed except racoon2ctl: phase 2 SA management (1), policy generation for IKEv1 (3), IPv6 (4), NAT-OA payloads (5), address substitution (6), fragmentation (7) and the address macro confusion (8) are done; only item 2, the graceful connect/disconnect control tool, remains open.
• A self-contained unit test framework with four new test programs covering the NAT-OA, address-substitution and wildcard-address logic. The full suite — including the pre-existing crypto self-test — passes cleanly under sanitizers: 5 out of 5 test programs, 0 failures.
• A new NEWS entry, and documentation and sample configurations matching the new defaults.

What remains

The following items are recorded in the top-level TODO file and are deliberately left for future work:

• A racoon2ctl tool. There is still no way for an initiator to connect and disconnect gracefully: sending SIGTERM to the daemon does not delete the security associations or send a delete notification to the peer the way an IKEv2 connection would, and for IKEv1 this gap is especially noticeable. A small control tool (and the delete-on-shutdown behaviour behind it) would make racoon2 behave like a proper long-running service instead of something you kill and re-run.
• Tests for the functionality that still has none. This project added tests only for the logic it introduced. The rest of the daemon — the phase 2 SA lifecycle, policy generation, rekeying, fragmentation paths, the configuration parser as a whole — is still only exercised by running it by hand. Extending the framework with tests for those areas is the natural next step, and the purge-related test added here is meant as a starting point for exactly that.
• End-to-end IPv6 testing. IPv6 now works on the configuration side, but it still needs much more thorough testing in real scenarios on NetBSD and Linux: full IPv6 tunnels, NAT-T interactions, and long-running connections on both initiator and responder sides have not been exercised yet.

Conclusion

This project started as "work down the TODO list" and grew into a substantially more robust racoon2: NAT traversal now follows the relevant RFCs for both IKE versions, IPv6 actually works and is used by default, configurations behave consistently, and — perhaps most importantly for the long term — the daemon finally has an automated test suite running under sanitizers.

The open items are collected in the "What remains" section above. I believe the combination of RFC-level fixes and a regression test framework makes racoon2 meaningfully closer to being a dependable VPN server for real clients on NetBSD and Linux.

I'd like to thank my mentor, Christos Zoulas, for reviewing this stream of pull requests, for patiently explaining the history behind this project and quirks like the address-macro mess. Thanks also to the other mentors and students of GSoC 2026 for a pleasant summer of digging into an old codebase and making it a little less scary.