SPECIAL INVITED PAPER ON THE CONSISTENCY OF BAYES ESTIMATES

We discuss frequency properties of Bayes rules, paying special attention to consistency. Some new and fairly natural counterexamples are given, involving nonparametric estimates of location. Even the Dirichlet prior can lead to inconsistent estimates if used too aggressively. Finally, we discuss reasons for Bayesians to be interested in frequency properties of Bayes rules. As a part of the discussion we give a subjective equivalent to consistency and compute the derivative of the map taking priors to posteriors. 1. Consistency of Bayes rules. One of the basic problems in statistics can be put this way. Data is collected following a probability model with unknown parameters; the parameters are to be estimated from the data. Often, there is prior information about the parameters, for example, their probable sign or order of magnitude. Many statisticians express such information in the form of a prior probability over the unknown parameters. Estimates based on prior probabilities will be called Bayes estimates in what follows. This paper studies points of contact between frequentist and Bayesian statistics. We derive frequency properties of Bayes estimates and suggest a Bayesian interpretation for some frequentist computations. Our main concern is consistency: as more and more data are collected, will the Bayes estimates converge to the true value for the parameters? If the underlying probability mechanism has only a finite number of possible outcomes (tossing a coin or die) and the prior probability does not exclude the true parameter values as impossible, it has long been known that Bayes estimates are consistent. As will be discussed below, if the underlying mechanism allows an infinite number of possible outcomes (e.g., estimation of an unknown probability on the integers), Bayes estimates can be inconsistent: as more and more data comes in, some Bayesian statisticians will become more and more convinced of the wrong answer. The class of tail-free and Dirichlet priors was introduced to insure consistency in such settings. We present examples showing that mechanical extension of such priors to other very similar settings leads to inconsistent estimates.