An empirical evaluation of several methods to select the best system
- 1 October 1999
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Modeling and Computer Simulation
- Vol. 9 (4), 381-407
- https://doi.org/10.1145/352222.352226
Abstract
Simulation is an important tool for comparing the performance of several alternative systems. There is therefore significant interest in procedures that efficiently select the best system, where best is defined by the maximum or minimum expected simulation output. In this paper, we examine both two-stage and sequential procedures that represent three structurally different modeling methodologies for allocating simulation replications to identify the best system, and we evaluate them empirically with respect to several measures of effectiveness. Empirical evidence suggests that sequential procedures perform better than their two-stage counterparts, including a heuristic sequential variation on Rinott's procedure. Further, there appears to be significant benefit to using procedures based on a Bayesian, average-case analysis as opposed to the statistically-conservative indifference-zone formulation.Keywords
This publication has 10 references indexed in Scilit:
- New development of optimal computing budget allocation for discrete event simulationPublished by Association for Computing Machinery (ACM) ,1997
- Bayesian look ahead one-stage sampling allocations for selection of the best populationJournal of Statistical Planning and Inference, 1996
- A gradient approach for smartly allocating computing budget for discrete event simulationPublished by Association for Computing Machinery (ACM) ,1996
- A lower bound for the correct subset-selection probability and its application to discrete-event system simulationsIEEE Transactions on Automatic Control, 1996
- Using Common Random Numbers for Indifference-Zone Selection and Multiple Comparisons in SimulationManagement Science, 1995
- Bayesian TheoryWiley Series in Probability and Statistics, 1994
- A Bayesian Approach to Ranking and Selection of Related Means with Alternatives to Analysis-of-Variance MethodologyJournal of the American Statistical Association, 1988
- A procedure for selecting a subset of size m containing the l best of k independent normal populations, with applications to simulationCommunications in Statistics - Simulation and Computation, 1985
- On two-stage selection procedures and related probability-inequalitiesCommunications in Statistics - Theory and Methods, 1978
- THE SIGNIFICANCE OF THE DIFFERENCE BETWEEN TWO MEANS WHEN THE POPULATION VARIANCES ARE UNEQUALBiometrika, 1938