Supporting "high memory" on NetBSD/sgimips

I purchased an SGI O2 some years back, and over time I have been finding little bits of time to improve support for the hardware on NetBSD. One of the things that has always bothered me is that NetBSD could not use more than 256MB of RAM, so I set out to fix it.

I typically spend most of my time in x86-land, so my knowledge of how MIPS works is weak at best. After speaking with a few people more familiar with the architecture, I learned that the memory size restriction is due to the way NetBSD's MIPS physical address map (pmap(9)) works. There is 512MB available for mapping physical addresses and some of that is typically used for devices. On the O2, the first 256MB is mapped to physical memory and the following 256MB is for memory mapped IO (for CRIME, MACE, PCI, etc).

On the O2, it appears that there are a few ways to access memory. A 256MB "low memory" window is mapped at 0x00000000, another 512MB "high memory" window at 0x20000000, and then a 1GB window at 0x40000000 that provides access to all available memory. Since the "high memory" window is at the 512MB mark, the kernel cannot directly access it. I decided to try to use the full memory window to allow use of all available memory.

As I did not feel comfortable modifying the MIPS pmap, I set out with my evil plan; I figured if I can access the memory at the 0x40000000 mark through a smaller mapping in kernel virtual address space, I should be able to expose this as a memory-backed block device. Using the md(4) driver as a reference, I added support to the crime(4) to expose a block device for any memory above the 256MB mark. Using the mips3_wired_enter_region() kernel function (from src/sys/arch/mips/mips/wired_mem.c), I am able to use 32MB of virtual address space in the block driver's strategy callback to copy in and out of high memory, reprogramming the TLB on demand.

Once this block device was in place, using the memory was as simple as creating a /dev/crime device node and using it as a swap device with higher priority than the swap partition on my hard disk.

With the changes, I see the following debug output at boot on my 448MB O2:

crime0 at mainbus0 addr 0x14000000: rev 1.1 (CRIME_ID: 161)
crime0: himem: 32 MB @ 0x50000000
crime0: himem: 32 MB @ 0x52000000
crime0: himem: 32 MB @ 0x54000000
crime0: himem: 32 MB @ 0x56000000
crime0: himem: 32 MB @ 0x58000000
crime0: himem: 32 MB @ 0x5a000000
crime0: himem: 192 MB memory disk

I then modified /etc/fstab to prioritize swap devices:

/dev/crime none swap sw,priority=0 0 0
/dev/sd0b none swap sw,priority=1 0 0

As a result I can now access the full amount of physical memory on my O2 with a 32-bit MIPS pmap!

A work in progress patch is available on the NetBSD FTP site.