We have measured third-order nonlinearities in the optical scattering from individual Au nanorods using an interferometric pump-probe technique. The nonlinearity produces changes of as much as 20% in the scattering cross-section over the 20-fs pulse duration. The magnitude of the ultrafast nonlinearity is the same as that due to heating of conduction electrons in the metal, suggesting the increased damping or saturation of strongly driven plasmons.