We show that the zero-point fluctuations of the intrinsic electromagnetic environment limit the phase-coherence time in all mesoscopic systems at low temperatures. We derive this quantum-noise-limited dephasing time and its temperature dependence in the crossover to the thermal regime. Our results agree well with most experiments in one-dimensional systems.