Quantum dots of InSb, GaSb, and AlSb were grown on GaAs substrates by molecular beam epitaxy and characterized by atomic force microscopy and Raman spectroscopy. There is a clear correlation between the observation of quantum dots by atomic force microscopy and a phonon mode at an energy a few wavenumbers below the longitudinal optic phonon energy for thick (In,Ga,Al)Sb layers. In the case of nominally AlSb quantum dots, a two-mode behavior is observed and attributed to the segregation of Ga into the AlSb during growth.