How kernels execute binaries

Execution in an Operating System

We directly execute some of the program on memory to let go of the virtual memory overhead. Here is one place where user mode and kernel mode separation comes into picture.

Kernel sets up a trap table at boot time since boot is in privileged mode. The address of this table is remembered by the hardware. System calls are made via numbers to specify the type of service they need.

Switching between processes

One approach is to assume that the programs running are cooperative. Any time there's a system call, the OS gets the control back and is able to switch between processes. These systems explicitly had a yield() system call that led programs to give control back to the OS. Obviously this can lead to stalls due to ill formed programs.

The other approach is the saner one, and that is to get the OS to take control from the application on its own. This is accomplished using timer interrupts (periodic interrupts as the name suggests). These run a preconfigured handler that is used for context switching.

An example of xv6 can be taken. It has a timer interrupt that triggers the switch() routine in the kernel that saves the registers and other values of the kernel in the kernel stack and the interrupt saves the registers and other info of the process in the kernel stack of that process. This part is done by the hardware. The saving of kernel registers is done by the software.