In a superconductor absorption of photons with an energy below the superconducting gap leads to redistribution of quasiparticles over energy and thus induces a strong non-equilibrium quasiparticle energy distribution. We have measured the electrodynamic response, quality factor and resonant frequency, of a superconducting aluminium microwave resonator as a function of microwave power and temperature. Below 200 mK, both the quality factor and resonant frequency decrease with increasing microwave power, consistent with the creation of excess quasiparticles due to microwave absorption. Counterintuitively, above 200 mK, the quality factor and resonant frequency increase with increasing power. We demonstrate that the effect can only be understood by a non-thermal quasiparticle distribution.