Memory leak & invalid memory deallocation detection in the Linux kernel for modules using exclusively Operating System Abstraction Layer

In user space there are many open source projects that one can use to identify memory leaks and corruptions such as electric fence [6], its newer fork DUMA [7], and valgrind [8]. For code running in kernel context these tools are not available.

When the problems mentioned above end up as a "kernel panic" as they often do, there is no further ability for the engineer to inspect the offending code other than to observe the stack trace. Also in kernel context, a memory leak is persistent: it will remain existent and most probably keep growing until a reboot, something that is unacceptable.

Kernel development allows direct access to the basic components and resources of the operating system, but debugging is not easy as the use of a debugger is not handy and one has to either use additional logging or use advanced techniques such as using kernel debuggers like kdb [2][3] and kgdb [4] or use kernel patches such as the one provided by Catalin Marinas, described in [1].

In this article we will describe our methodology for identifying memory management problems for kernel modules that use Operating System Abstraction Layer (OSAL) without the need of extra patches or kernel debuggers.

OSAL Fundamentals
The Operating System Abstraction Layer is a software layer that allows engineers to develop applications that are OS agnostic [2]. It reduces the porting complexity by providing an API for OS components such as threads, synchronization and memory management allowing portability to different OSes provided there is implementation of the same OSAL and its API.

Figure 1 below shows the concept of a typical software stack using OSAL inside the kernel.

Figure 1 " The concept of using OSAL

Our  Methodology
This methodology has a a perqusite the presence of an OSAL. As shown in Figure 2 below, the header file needs to be compiled together with the source code of the kernel module so it provides macros that replace the actual memory allocating/deallocating functions with a call to the function itself and a log with useful information about the memory address and size involved. Let us call this header file OsalMemLeak.h.

The post processing script later on will parse all the logs generated by the macros and will analyze the data. It will match the allocations to the deallocations and determine if there is a memory leak or memory corruptions created by code freeing memory buffers that were not allocated properly.

Figure 2 " Proposed Method