How to develop OS X Kernel Extensions
Introduction
Some time ago I was working on an antivirus Active Protection module, which operates within Mac OS kernel. The most difficult part about it was dealing with kernel panics (this is when you have to restart your machine, potentially losing all your unsaved work, also wasting time waiting until the Mac boots up again) and making it convenient to debug such a project.
Kernel development is different from user mode programming, because almost every mistake costs you a panic.
It can also be hard to debug it. You cannot run and debug it on the same machine (well, you can insert some printf() statements whose output you can read later using the Console.app, but that’s about all you can do). To be able to actually debug the kernel extension using gdb/lldb, you will need two Macs. Or run two OS X systems, one inside a virtual machine.
However It’s not as hard as you may think, once you have everything properly set up.
I would like to share my experience how I managed to make kext development much easier. My goal was to install another instance of OS X on a virtual machine, to be able to quickly resolve panics (in case I got any), without having to wait for the system to reboot, to conveniently attach a debugger to the virtual machine to be able to see the state of my kernel extension at any time. Also I wanted to redirect the printf logging to the development machine (this is where i am running my virtual machine), I even made it use different colors.
Eventually kernel development became almost as easy as a user mode application development.
Installing OS X El Capitan on VirtualBox running on OS X
I chose to use Oracle VirtualBox installation as a second machine. It is free. It is easy to script, and generally works just fine. Installing OS X can get a bit tricky, though. First, “buy” a Mac OS X El Capitan on the Mac App Store (it’s free), download it from the App Store the usual way, like any other app, and make sure you have “Install OS X El Capitan” application appear in your LaunchPad. Now you need to prepare a disk image, suitable for VirtualBox. To do that, first make sure you have enough free disk space available (at least 20 Gb), open Terminal.app, and type in the following:
hdiutil attach "/Applications/Install OS X El Capitan.app/Contents/SharedSupport/InstallESD.dmg" -noverify -nobrowse -mountpoint /Volumes/esd hdiutil create -o ElCapitan3.cdr -size 7316m -layout SPUD -fs HFS+J hdiutil attach ElCapitan3.cdr.dmg -noverify -nobrowse -mountpoint /Volumes/iso asr restore -source /Volumes/esd/BaseSystem.dmg -target /Volumes/iso -noprompt -noverify -erase rm /Volumes/OS\ X\ Base\ System/System/Installation/Packages cp -rp /Volumes/esd/Packages /Volumes/OS\ X\ Base\ System/System/Installation cp -rp /Volumes/esd/BaseSystem.chunklist /Volumes/OS\ X\ Base\ System/ cp -rp /Volumes/esd/BaseSystem.dmg /Volumes/OS\ X\ Base\ System/ hdiutil detach /Volumes/esd hdiutil detach /Volumes/OS\ X\ Base\ System hdiutil convert ElCapitan3.cdr.dmg -format UDTO -o ElCapitan3.iso mv ElCapitan3.iso.cdr ElCapitan3.iso
This will create an ElCapitan3.iso file, which you can use as an installation media.
Now, open VirtualBox, create a new virtual machine of “64-bit Mac OS X” type
Attach your ElCapitan3.iso file
And follow the installation procedure, it is pretty straightforward, you will have to prepare the disk using Disk Utility inside the installer.
Now you have OS X running inside VirtualBox, and another OS X running on your Mac.
Running kernel extension on VirtualBox guest OS X
Disabling System Integrity Protection
To be able to run your kext, you need first to disable “system integrity protection” (otherwise the system allows only signed kexts to be loaded from /Library/Extensions only). SIP can be disabled only from the OS X Recovery. Previously, in Yosemite, there was an option to instead simply execute sudo nvram boot-args=“kext-dev-mode=1” from Terminal to allow unsigned kexts to be loaded, but this option has been removed in ElCapitan. To disable SIP on a real Mac you would reboot into recovery mode, open Terminal and type:
sudo csruril disableand reboot. This command modifies some entries inside NVRAM (Non-volatile random access memory), to allow loading unsigned kexts.
nvram - is a chip on your Mac (a non-volatile part of the RAM), where OS X stores some settings (like, speaker volume, display resolution, some info about last system errors, etc.)
On VirtualBox to get into the Recovery mode you need to reboot into UEFI Shell, by holding F12 key, when VirtualBox restarts:
Select “Boot Manager”, and “EFI Internal Shell”.
Inside this UEFI shell type:
FS2:which will switch you into the FS2 “drive”, and then
cd com.apple.recovery.boot boot.efi
hold on until the OS X recovery boots up.
csrutil disable reboot
Unfortunately, since VirtualBox doesn’t actually support nvram, you will have to do this every time you reboot your system (as this "nvram section" gets erased whenever you reboot your VirtualBox machine). However, you can avoid rebooting your system by saving its state instead, which actually is preferred, see below.
Installing the kernel symbols
You need to install kernel symbols (i.e., all the function names to be able to see them in your debugger) and the development version of the kernel. To do so, download and install the KDK (kernel debug kit) from Developer Apple website (developer.apple.com/downloads) on both development(host) and target(guest) operating systems. Technically this is not required, but it may simplify debugging. The kernel is located in /System/Library/Kernels/ so after having installed the KDK (it installs into /Developer), you need to copy the kernel on your host system:
sudo cp /Library/Developer/KDKs/KDK_version/System/Library/Kernels/kernel.development /System/Library/Kernels
Specifying kernel boot parameters for debugging
This is not all. Now you are able to run unsigned kexts, but what about debugging? The debugging can be set up using kernel boot parameters, which are normally specified inside NVRAM using the command sudo nvram boot-args=“…”
However, as already mentioned, VirtualBox doesn’t have nvram. Luckily, VirtualBox has a command line utility VBoxManage, which lets you specify kernel arguments. So, on your host machine (not the virtual one), open terminal and type
VBoxManage setextradata “your virtual machine name>" "VBoxInternal2/EfiBootArgs" [here go the kernel boot arguments]
in my case, i have the following:
VBoxManage setextradata "ElCap" "VBoxInternal2/EfiBootArgs" "debug=0x14e kcsuffix=development -v serialbaud=921600"
debug - is a set of OR’ed flags (see the table below, took it from the Amit Singh’s book and Apple documentation)
In my case I specified DB_PRT | DB_NMI | DB_KPRT | DB_ARP | DB_LOG_PI_SCRN
The kcsuffix=development argument instructs OS X to run the development version of the kernel with extra logging (in case you installed it from KDK, see the section above).
The -v specified text boot mode, instead of displaying an Apple logo (just a matter of personal preference)
serialbaud=921600 specifies speed of the serial port used for debugging. This is where all your kprintf() output will be sent to (and you can intercept this and print it in a Terminal, for example).
Debugging time!
So you have OS X running on your host and inside VirtualBox. Make sure you have a Bridged network connection set up (instead of the default NAT, in the virtual machine settings), as the debugger doesn't work with NAT:
Now you can connect to the Guest operating system from the debugger.
To do so, open Terminal on your host OS, and type the following:
lldb target create /Library/Developer/KDKs/KDK_10.11.3_15D21.kdk/System/Library/Kernels/kernel.debug kdp-remote 192.168.1.9
Of course, put your own KDK version instead of KDK_10.11.3_15D21, and use the IP address of your virtual machine (you can find it out by running the ifconfig command in your guest machine’s terminal). It is also printed for convenience when the system is halted (see the screenshot)
The last command (kdp-remote) will be waiting for the kernel to allow to attach to itself.
Attaching is easy: just press the rightmost buttons Command+Option+Control+Shift+Esc in your target OS, and it will halt itself, waiting for debugger to be attached:
you can resume the system by typing “c” or “continue” in lldb on your development system, once it is attached to the target system. Or you can set up some breakpoints, for instance, while your system is halted, and it will automatically half again when those breakpoints are hit:
What about logging?
Logging can be very convenient during kernel development. I prefer to use kprintf() call, which is synchronous (so, the next line of code in your kext doesn't get executed until the line is printed). This helps to resolve race conditions, and generally understand how your code is being executed by different threads. The kernel, the way we had set it up previously, prints everything to the serial port. Luckily, Virtualbox allows redirecting serial port into a file:
Once you’ve done so, you can monitor this file constantly using Console.app, but what I personally prefer is just opening another Terminal window, and typing the command:
tail -f /Users/theuser/vboxserialdump.log
Now everything that the Target OS kernel "kprints" shows up in my Terminal window on my development OS immediately. I can even make the kernel extension print in different colors, like so (see here for a complete list of color/text appearance codes:
void kprintfinyellow(const char *fmt, ...) {
va_list args;
va_start(args, fmt);
char destbuf[4096];
vsnprintf(destbuf, sizeof(destbuf), fmt, args);
kprintf("\033[33m%s\033[0m", destbuf);
va_end(args);
}
Just make sure you remove these “kprintf” calls in the release version of your kext, otherwise this kind of logging significantly degrades system performance.
Handling kernel panics
Now that we have two systems and we are able to debug kexts, we can script some things to make it even easier. First of all, VirtualBox allows us to “save machine state” or "take a snapshot" of a running system. This means that instead of rebooting the machine when something goes wrong, we can just restore the previously saved state. If you have SSD - it is much faster than rebooting, since restoring takes only a few seconds. So first you need to save the machine state of a completely set up and running OS X VirtualBox machine, give this saved state a name, say, “Snapshot 1”, and later, whenever you experience a kernel panic, just call the script reopenPanicedVM.sh:
#!/bin/sh echo “Reopening paniced system” VBoxManage controlvm "ElCap" poweroff VBoxManage snapshot "ElCap" restore “Snapshot 1" VBoxManage startvm "ElCap"
Where ElCap - is your Virtualbox machine name, and “Snapshot 1” - is your snapshot name
However, don't be scared of the panics! They actually can be quite helpful. You can find out what the problem is just by reading the panic printout, or you can attach a debugger after the system panic'ed to find out what went wrong in your kext. See this article for details.
Automating kext copying/loading/unloading and other shell operations on the Target machine from the development machine
After easing your pain from kernel panics, lets make it easier to copy the kext from your development machine onto the guest OS, load the kext, unload the kext, etc.
We will use two commands: ssh (to script some operations on the guest machine from the host machine), and scp (to copy files from the host machine to the guest machine).
First of all, lets make ssh/scp not ask for password every time we try to log in onto the guest OS.
To do so, first generate RSA keys using command
ssh-keygen -t rsa -b 2048next, install ssh-copy-id on your machine (you need to have home-brew or MacPorts installed):
brew install ssh-copy-id
now copy the RSA key on your target machine
ssh-copy-id virtual_machine_username@192.168.1.9
and try to login using ssh (it shouldn’t ask for a password)
ssh virtual_machine_username@192.168.1.9
By the way, if you remote machine freezes while you’re logged in using ssh, ssh will also freeze. To forcefully terminate the ssh session (instead of closing the Terminal window), just type “ [enter] ~ .”.
Now you can copy files on your target machine from the development machine without entering password, like so:
scp -r ~/Documents/MyKext/build/Debug/MyKext.kext virtual@$VIRTUALBOX_IP_ADDRESS:/Users/virtual/
Similarly, you can execute commands on the virtual machine, from a script on the development machine this way:
ssh virtual@$VIRTUALBOX_IP_ADDRESS << ENDSSHCOMMANDS
cd /tmp
echo $VIRTUALPASS | sudo -S cp -r /Users/virtual/MyKext.kext /tmp/
echo $VIRTUALPASS | sudo -S kextutil -t MyKext.kext
ENDSSHCOMMANDS
kext-unloading is done the same way. Wrap these calls into a script, that you can even trigger from your Xcode “run script” build phase, and it will be even easier!