All manufacturing systems will, at some time, have to cope with changes and disturbances such as changes in products, changes in demand, machine failures or worker absences. To deal with change, we would like them to be flexible or robust. However, while most people have a vague idea what is meant by flexibility or robustness and think they understand what these concepts mean, it is surprisingly difficult to come up with good operational definitions which enable flexibility or robustness to be measured. If such operational measures could be developed they would help make decisions about the relative merits of systems and compare different designs or different system operating rules. In this paper, we look at a variety of approaches to operational definitions of flexibility and robustness. However, we also show that the resulting measures are open to criticism and it is indeed not at all clear that measuring flexibility is meaningful, thus raising some challenging questions about the usefulness and appropriateness of flexibility and robustness as decision criteria. Nevertheless, thinking about the issues raised by the attempt to come up with measures of flexibility and robustness is very valuable as a means of seeking to improve manufacturing performance.