The electric field dependences of the direct annihilation rates for positrons in helium, neon, and argon are calculated. This is done by using a systematic description of the scattering process. The momentum transfer and direct annihilation cross sections are calculated within the framework of the polarized-orbital method which has worked well for electron–atom scattering. The cross sections are then used in the appropriate diffusion equation to determine the experimentally observable annihilation rates appropriate to the exponential decay region of the spectrum. The resulting annihilation rates are found to be extremely sensitive to the low-energy behavior of the cross sections. Best agreement between calculated and experimental annihilation rates is obtained for helium where fairly rigorous calculations of the cross sections are available. Good agreement between theory and experiment for neon and argon is obtained only by making judicious choices for the components of the distortion included in the calculations. It is thus concluded that the positron–atom scattering process is considerably more sensitive to the details of the mutual distortion interaction than is observed in the corresponding electron–atom collision process.