A methodology to detect and characterize kernel level rootkit exploits involving redirection of the system call table

Abstract

There is no standardized methodology at present to characterize rootkits that compromise the security of computer systems. The ability to characterize rootkits will provide system administrators with information so that they can take the best possible recovery actions and may also help to detect additional instances and prevent the further installation of the rootkit allowing the security community to react faster to new rootkit exploits. There are limited capabilities at present to detect rootkits, but in most cases these capabilities only indicate that a system is infected without identifying the specific rootkit. We propose a mathematical framework for classifying rootkit exploits as existing, modifications to existing, or entirely new. An indepth analysis of a particular type of kernel rootkit is conducted in order to develop a characterization. As a result of this characterization and analysis, we propose some new methods to detect this particular class of rootkit exploit.