Network Security Audit

As a part of a funded project, I am conducting a security audit of NetBSD’s network stack. The work will end soon, and I would like to briefly present some results.

Fixing, Strengthening, Simplifying

Over the last five months, hundreds of patches were committed to the source tree as a result of this work. Dozens of bugs were fixed, among which a good number of actual, remotely-triggerable vulnerabilities.

Changes were made to strengthen the networking subsystems and improve code quality: reinforce the mbuf API, add many KASSERTs to enforce assumptions, simplify packet handling, and verify compliance with RFCs. This was done in several layers of the NetBSD kernel, from device drivers to L4 handlers.

A lot of cleanup took place, too. For example I managed to remove more than one thousand lines of code in IPsec, while at the same time improving robustness and performance. This kind of cleanup results in a networking code that is much easier to understand and maintain.

The fixes for critical bugs were quickly propagated to the stable branches (NetBSD-6, NetBSD-7) and the NetBSD-8_BETA branch. Along the way, several changes too were discreetly propagated, when they were considered as good mitigations against possible attack vectors.

Fixes in Other Operating Systems

In the course of investigating several bugs discovered in NetBSD, I happened to look at the network stacks of other operating systems, to see whether they had already fixed the issues, and if so how. Needless to say, I found bugs there too.

So far the trophies are:

NetBSD	OpenBSD	FreeBSD
SA2018-003 (1)	6.2-Errata-#005 (2)	SA-18:01.ipsec (2)
SA2018-004 (1)	6.2-Errata-#006 (1)	SA-18:05.ipsec (2)
SA2018-006 (5+)	6.2-Errata-#007 (1)
SA2018-007 (10)	6.2-Errata-#010 (1)
SA2018-008 (2)	6.3-Errata-#006 (2)
	6.3-Errata-#008 (1)

Fig. A: advisory_name (number_of_bugs).

Of course, I am focusing on NetBSD, so it is no surprise the number of bugs found there is higher than in the other OSes.

Also, it is to be noted that FreeBSD hasn’t yet published advisories for several bugs that I reported to them (which they nonetheless fixed pretty quickly).

Some Examples

The IPv6 Buffer Overflow

In January I discovered, in NetBSD’s IPv6 stack, a subtle buffer overflow, which turned out to affect the other BSDs as well.

The overflow allowed an attacker to write one byte of packet-controlled data into ‘packet_storage+off’, where ‘off’ could be approximately controlled too.

This allowed at least a pretty bad remote DoS: by sending specially-crafted packets in a loop, an attacker could overwrite several areas of memory with wrong values, which would eventually lead to undefined behavior and crash.

One way of exploiting this bug was to use a special combination of nested fragments.

Fig. B: A simplified view.

In short, when receiving the last fragment of a packet, the kernel would iterate over the previous IPv6 options of the packet, assuming that everything was located in the first mbuf. It was possible to break this assumption, by sending a fragment nested into another.

Given the nature of the bug there were probably other (and perhaps more direct) ways to trigger it.

This overflow was of course fixed pretty quickly, but in addition the NetBSD kernel was changed to automatically drop nested fragments. This is an example of the many miscellaneous changes made in order to strengthen the network stack.

The IPsec Infinite Loop

When receiving an IPv6-AH packet, the IPsec entry point was not correctly computing the length of the IPv6 suboptions, and this, before authentication. As a result, a specially-crafted IPv6 packet could trigger an infinite loop in the kernel (making it unresponsive). In addition this flaw allowed a limited buffer overflow - where the data being written was however not controllable by the attacker.

The other BSDs too were affected by this vulnerability. In addition they were subject to another buffer overflow, in IPv4-AH this time, which happened to have been already fixed in NetBSD several years earlier.

The IPPROTO Typo

While looking at the IPv6 Multicast code, I stumbled across a pretty simple yet pretty bad mistake: at one point the Pim6 entry point would return IPPROTO_NONE instead of IPPROTO_DONE. Returning IPPROTO_NONE was entirely wrong: it caused the kernel to keep iterating on the IPv6 packet chain, while the packet storage was already freed.

Therefore it was possible to remotely cause a use-after-free if the packet was forged in such a way that the kernel would take the IPPROTO_NONE branch. Generally the kernel would panic shortly afterwards, because it figured out it was double-freeing the same packet storage. (A use-after-free detector is also one of the things that were added in NetBSD to prevent the exploitation of such bugs.)

This typo was found in the Multicast entry code, which is enabled only with a particular, non-default configuration. Therefore it didn’t affect a default install of NetBSD.

While looking for other occurrences of this typo, I found the exact same bug in the exact same place in FreeBSD. Curiously enough, OpenBSD had the same bug too, but in a totally different place: the typo existed in their EtherIP entry point, but there, it was more dangerous, because it was in a branch taken by default, and therefore a default install of OpenBSD was vulnerable.

The PF Signedness Bug

A bug was found in NetBSD’s implementation of the PF firewall, that did not affect the other BSDs. In the initial PF code a particular macro was used as an alias to a number. This macro formed a signed integer.

In NetBSD, however, the macro was defined differently: it contained the sizeof statement. This was a terrible mistake, because it resulted in an unsigned integer.

This was not the intended signedness. Given that the macro in question was used to perform length validation checks in PF’s TCP-SYN entry point when a modulate rule was active, it was easy to cause a remote DoS by sending a malformed packet.

But PF was not a component I was supposed to look at as part of my work. So how did I still manage to find this bug? Well, while closing dozens of reports in NetBSD’s Problem Report database, I stumbled across a PR from 2010, which was briefly saying that PF’s TCP-SYN entry point could crash the kernel if a special packet was received. Looking at the PF code, it was clear, after two minutes, where the problem was.

The NPF Integer Overflow

An integer overflow could be triggered in NPF, when parsing an IPv6 packet with large options. This could cause NPF to look for the L4 payload at the wrong offset within the packet, and it allowed an attacker to bypass any L4 filtering rule on IPv6.

Fig. C: Simplified example of an exploit.

In the example above, NPF allows the packet to enter, based on validation performed on the wrong TCP header (orange, dashed lines). The kernel reads the correct TCP header (red), and delivers it to a socket. NPF was supposed to reject it.

More generally, several problems existed in NPF’s handling of IPv6 packets.

The IPsec Fragment Attack

(A more detailed example)

I noticed some time ago that when reassembling fragments (in either IPv4 or IPv6), the kernel was not removing the M_PKTHDR flag on the secondary mbufs in mbuf chains. This flag is supposed to indicate that a given mbuf is the head of the chain it forms; having the flag on secondary mbufs was suspicious.

Later, deep in the IPsec-ESP handler, I found a function that was assuming that M_PKTHDR was set only in the first mbuf of the chain – assumption that evidently didn’t hold, since reassembled fragments had several M_PKTHDRs. Later in the handler, it resulted in a wrong length being stored in the mbuf header (the packet had become shorter than the actual length registered). The wrong length would in turn trigger a kernel panic shortly afterwards.

This remote DoS was triggerable if the ESP payload was located in a secondary mbuf of the chain, which never is the case in practice. The function in question was called after ESP decryption. Therefore, in order for an attacker to reach this place, it was necessary to send a correct ESP payload – which meant having the ESP key. So at a first glance, this looked like a non-critical bug.

But there was still a theoretical way to exploit the bug. In the case of IPv6 at least, the IP options can be huge, and more importantly, can be located in the unencrypted part of the packet. Let’s say you are MITMing a NetBSD host that is having an ESP conversation with another host. You intercept a packet directed to the NetBSD host. You take the ESP payload as-is (which you can’t decrypt since you don’t have the key), you craft a new two-fragment packet: you put the other IPsec host’s IPv6 address as source, insert a dummy IP option in the first fragment, insert the ESP payload as-is in the second fragment, and send the two fragments to the NetBSD host.

The NetBSD host reassembles the fragments, decrypts the ESP payload correctly, reaches the aforementioned handler, miscomputes the packet length, and crashes.

The other BSDs were affected.

And the rest...

Many security fixes and improvements in different places that I didn’t list here.

What Now

Not all protocols and layers of the network stack were verified, because of time constraints, and also because of unexpected events: the recent x86 CPU bugs, which I was the only one able to fix promptly. A todo list will be left when the project end date is reached, for someone else to pick up. Me perhaps, later this year? We’ll see.

This security audit of NetBSD’s network stack is sponsored by The NetBSD Foundation, and serves all users of BSD-derived operating systems. The NetBSD Foundation is a non-profit organization, and welcomes any donations that help continue funding projects of this kind.

[6 comments]