Co-citation analysis, a tool initially developed for use in science studies, is entering the science-policy arena where the technical demands placed on data differ. Comparison of co-citation data and a manually generated bibliography of a physics specialty (spin glass) revealed that, at present, co-citation analysis is inconsistent and fragile when used to track national participation in a specialty over time. Specifically, co-citation clusters only emerged some time after the field began, were inconsistent in their coverage over time, were subject to errors in citations, under-represented experimental work and were found to contain a significant subjective element.