Add a syscall to the Linux kernel

I think it’s always a good idea to play around with something to get to know it better. So why not add a syscall to Linx?

To do this, we first need the Linux source code, which we can get from Github.

git clone git@github.com:torvalds/linux.git
cd linux/

If you are using clangd as an LSP, you have to run this script first for “go to definition” to work properly:

./scripts/clang-tools/gen_compile_commands.py

I also had to disable auto-formatting as the formatting used by clang-format does not reflect the format which is used in the source.

Implement Your Own Syscall

Now that we have the source code, we can add the syscall. My plan is to add a syscall named mycall which takes an int as parameter and simply prints a message with that number to the kernel log. Theoretically, you have to add the syscall for each architecture separately. I am on a x86_64 system, so I’m only doing it for this architecture. For x86_64 you have to add it to the file arch/x86/entry/syscalls/syscall_64.tbl. For other you architectures you can look for other .tbl files under arch/:

find arch/ -name "*.tbl"

In the file arch/x86/entry/syscalls/syscall_64.tbl we add the following line for mycall:

...
467	common	mycall			sys_mycall
...

Then you implement the syscall in the file kernel/sys.c, which already contains many other syscalls. To do this we use the SYSCALL_DEFINEX macro, where X is the number for arguments the syscall takes:

SYSCALL_DEFINE1(mycall, int, mynumber)
{
	printk(KERN_INFO "mycall called with %d\n", mynumber);
	return 0;
}

You can also see the changes in this commit.

And voilà, that was already it. Build the new kernel and we have our syscall:

make defconfig
make -j$( nproc )

Test it

That was easy so far. Now we need to run our custom built kernel and execute a program that calls this syscall. Normally we don’t have to call syscalls directly, because this low-level interactions are usually abstracted away by libraries like glibc.

Now we have the problem that there are no abstractions that know our new syscall. Fortunately, glibc provides a macro that allows to call arbitrary syscalls.

The plan is to write a small C program that runs directly as init program. This is the first user space program run by the Linux kernel. Usually init is something like systemd. In init you are not allowed to exit, otherwise you get a kernel panic. So we call reboot to shutdown properly.

init.c:

#include <stdio.h>
#include <sys/reboot.h>
#include <sys/syscall.h>
#include <unistd.h>

#define SYS_MYCALL 467 // We use the number uf our syscall

int main() {
  int arg = 42;
  long result;

  // Invoke the syscall
  result = syscall(SYS_MYCALL, arg);

  // Print the result
  printf("syscall returned: %ld\n", result);

  reboot(RB_POWER_OFF);

  return 0;
}

This program we have to compile statically as we have nothing (no shared libraries) other than this program in our minimal initramfs.

gcc -static -o init init.c

Create the initramfs:

echo ./init | cpio -o -H newc | gzip > initramfs.cpio.gz

Then we can run the whole thing as follows using QEMU:

qemu-system-x86_64 -kernel bzImage -initrd initramfs.cpio.gz -nographic -append "console=ttyS0"