Testing for Structural Breaks in the Evaluation of Programs

Abstract

A youth homicide reduction initiative in Boston in the mid-1990s poses particular difficulties for program evaluation because it did not have a control group and the exact implementation date is unknown. A standard methodology in program evaluation is to use time series variation to compare pre- and postprogram outcomes. Such an approach is not valid, however, when the timing of program implementation or effect is unknown. To evaluate the Boston initiative, we adapt from the time series literature an unknown-breakpoint test to test for a change in regime. Tests for parameter instability provide a flexible framework for testing a range of hypotheses commonly posed in program evaluation. These tests both pinpoint the timing of maximal break and provide a valid test of statistical significance. We evaluate the results of the estimation using the asymptotic results in the literature and with our own Monte Carlo analyses. We conclude there was a statistically significant discontinuity in youth homicide incidents (on the order of 60%) shortly after the intervention was unveiled. © 2003 President and Fellows of Harvard College and the Massachusetts Institute of Technology.