Automated detection of vulnerabilities in privileged programs by execution monitoring

Abstract

Presents a method for detecting exploitations of vulnerabilities in privileged programs by monitoring their execution using audit trails, where the monitoring is with respect to specifications of the security-relevant behavior of the programs. Our work is motivated by the intrusion detection paradigm, but is an attempt to avoid ad hoc approaches to codifying misuse behavior. Our approach is based on the observation that although privileged programs can be exploited (due to errors) to cause security compromises in systems because of the privileges accorded to them, the intended behavior of privileged programs is, of course, limited and benign. The key, then, is to specify the intended behavior (i.e. the program policy) and to detect any action by a privileged program that is outside the intended behavior and that imperils security. We describe a program policy specification language, which is based on simple predicate logic and regular expressions. In addition, we present specifications of privileged programs in Unix, and a prototype execution monitor for analyzing audit trails with respect to these specifications. The program policies are surprisingly concise and clear, and in addition, capable of detecting exploitations of known vulnerabilities in these programs. Although our work has been motivated by the known vulnerabilities in Unix, we believe that by tightly restricting the behavior of all privileged programs, exploitations of unknown vulnerabilities can be detected. As a check on the specifications, work is in progress on verifying them with respect to an abstract security policy.