Six methods for collecting the judgments of experts concerning the relative value of sets of criteria were compared for their reliability and time efficiency. The methods were ranking, rating, three versions of paired comparisons and a method of successive comparisons suggested by Churchman, Ackoff and Arnoff [Churchman, C. W., R. L. Ackoff, E. L. Arnoff. 1957. Introduction to Operations Research. Wiley, New York.]. The judgment situations used were concerned with the design of a specific air defense system and a general air defense system, and with selecting a personnel subsystem manager for a development program. In each of these three situations six criteria were comparatively evaluated by the judges. The results of these experiments showed that there were no significant differences in the sets of criterion weights derived from collecting the judgment data by any of the methods, but that ranking was by far the most efficient method. A fourth experiment was conducted to develop baseline data on the time required to make comparative judgments vs. number of items to be judged, by the ranking method and by the simplest paired comparisons method. Ranking is increasingly more efficient than paired comparisons as the number of items to be judged increases from six to 30.