Extensions of the multiarmed bandit problem: The discounted case
- 1 May 1985
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automatic Control
- Vol. 30 (5), 426-439
- https://doi.org/10.1109/tac.1985.1103989
Abstract
There are N independent machines. Machine i is described by a sequence {X^{i}(s), F^{i}(s)} where X^{i}(s) is the immediate reward and F^{i}(s) is the information available before i is operated for the sth lime. At each time one operates exacfiy one machine; idle machines remain frozen. The problem is to schedule the operation of the machines so as to maximize the expected total discounted sequence of rewards. An elementary proof shows that to each machine is associated an index, and the optimal policy operates the machine with the largest current index. When the machines are completely observed Markov chains, this coincides with the well-known Gittins index rule, and new algorithms are given for calculating the index. A reformulation of the bandit problem yields the tax problem, which includes, as a special case, Klimov's waiting time problem. Using the concept of superprocess, an index rule is derived for the case where new machines arrive randomly. Finally, continuous time versions of these problems are considered for both preemptive and nonpreemptive disciplines.Keywords
This publication has 16 references indexed in Scilit:
- On the evaluation of suboptimal strategies for families of alternative bandit processesJournal of Applied Probability, 1982
- On a sufficient condition for superprocesses due to whittleJournal of Applied Probability, 1982
- Multi-Armed Bandits with Discount Factor Near One: The Bernoulli CaseThe Annals of Statistics, 1981
- On stochastic scheduling with precedence relations and switching costsJournal of Applied Probability, 1980
- Optimal Search for the Best AlternativeEconometrica, 1979
- Time-Sharing Service Systems. IITheory of Probability and Its Applications, 1979
- On the Many-armed Bandit ProblemThe Annals of Probability, 1978
- Bayesian rules for the two-armed bandit problemBiometrika, 1977
- A hamiltonian approach to optimal stochastic resource allocationAdvances in Applied Probability, 1977
- Time-Sharing Service Systems. ITheory of Probability and Its Applications, 1975