Inference, Bayesian

views updated

Inference, Bayesian

BAYES’S FORMULA

STATISTICAL INFERENCE

TECHNICAL NOTES

BIBLIOGRAPHY

Bayesian inference is a collection of statistical methods that are based on a formula devised by the English mathematician Thomas Bayes (1702-1761). Statistical inference is the procedure of drawing conclusions about a population or process based on a sample. Characteristics of a population are known as parameters. The distinctive aspect of Bayesian inference is that both parameters and sample data are treated as random quantities, while other approaches regard the parameters as nonrandom. An advantage of the Bayesian approach is that all inferences can be based on probability calculations, whereas non-Bayesian inference often involves subtleties and complexities. One disadvantage of the Bayesian approach is that it requires both a likelihood function that defines the random process that generates the data, and a prior probability distribution for the parameters. The prior distribution is usually based on a subjective choice, which has been a source of criticism of the Bayesian methodology. From the likelihood and the prior, Bayes’s formula gives a posterior distribution for the parameters, and all inferences are based on this.

BAYES’S FORMULA

There are two interpretations of the probability of an event A, denoted P (A ): (1) the long-run proportion of times that the event A occurs upon repeated sampling; and (2) a subjective belief in how likely it is that the event A will occur. If A and B are two events, and P (B ) > 0, then the conditional probability of A given B is P (A\B ) = P (AB )/P (B ), where AB denotes the event that both A and B occur. The frequency interpretation of P (AǀB ) is the long-run proportion of times that A occurs when we restrict attention to outcomes where B has occurred. The subjective probability interpretation is that P (AǀB ) represents the updated belief of how likely it is that A will occur if we know B has occurred. The simplest version of Bayes’s formula is P (B/A ) = P (A\B ) P (B )/(P (A\B ) P (B ) + P (A \∼B ) P (∼B )), where ∼B denotes the complementary event to B, that is, the event that B does not occur. Thus, starting with the conditional probabilities P (A \B ), P (A \∼B ), and the unconditional probability P (B ) (P (∼B ) = 1 - P (B ) by the laws of probability), we can obtain P (B ǀA ). Most applications require a more advanced version of Bayes’s formula.

Consider the “experiment” of flipping a coin. The mathematical model for the coin flip applies to many other problems, such as survey sampling when the subjects are asked to give a “yes” or “no” response. Let θ denote the probability of heads on a single flip, which we assume is the same for all flips. If we also assume that the flips are statistically independent given θ (i.e., the outcome of one flip is not predictable from other flips), then the probability model for the process is determined by θ and the number of flips. Note that θ can be any number between 0 and 1. Let the random variable X be the number of heads in n flips. Then the probability that X takes a value k is given by

P (X = kǀθ ) = Cn,k θ k (1 – θ )n-k, k = 0, 1, …, n.

Cn, k is a binomial coefficient whose exact form is not needed. This probability distribution is called the binomial distribution. We will denote P (X = kǀθ ) by f(kǀθ), and when we substitute the observed number of heads for k, it gives the likelihood function.

To complete the Bayesian model we specify a prior distribution for the unknown parameter θ. If we have no belief that one value of θ is more likely than another, then a natural choice for the prior is the uniform distribution on the interval of numbers from 0 to 1. This distribution has a probability density function g(θ) which is 1 for 0 ≤ θ ≤ 1 and otherwise equals 0, which means that P (a≤θ ≤b ) = b - a for 0≤a < b ≤ 1.

The posterior density of θ given X = x is given by a version of Bayes’s formula: h(θ ǀx) = K(x)f(xǀθ)g (θ ), where K(x) -1 = ∫ f (x ǀθ )g (θ)dθ is the area under the curve f (xǀθ )g (θ ) when x is fixed at the observed value.

A quarter was flipped n = 25 times and x = 12 heads were observed. The plot of the posterior density h (θ ǀ12) is shown in Figure 1. This represents our updated beliefs about θ after observing twelve heads in twenty-five coin flips. For example, there is little chance that θ ≥.8; in fact, P (θ ≥.8 ǀX = 12) = 0.000135, whereas according to the prior distribution, P (θ ≥.8) = 0.200000.

STATISTICAL INFERENCE

There are three general problems in statistical inference. The simplest is point estimation: What is our best guess for the true value of the unknown parameter θ ? One natural approach is to select the highest point of the posterior density, which is the posterior mode. In this example, the posterior mode is θ Mode = x/n = 12/25 = 0.48. The posterior

mode here is also the maximum likelihood estimate, which is the estimate most non-Bayesian statisticians would use for this problem. The maximum likelihood estimate would not be the same as the posterior mode if we had used a different prior. The generally preferred Bayesian point estimate is the posterior mean : θMEAN = ∫ θ h (θ ǀ x ) d θ = (x + 1)/(n + 2) = 13/27 = 0.4815, almost the same as θMode here.

The second general problem in statistical inference is interval estimation. We would like to find two numbers a < b such that P (a < θ < b ǀX = 12) is large, say 0.95. Using a computer package one finds that P (0.30 < θ < 0.67ǀ X = 12) = 0.950. The interval 0.30 < θ < 0.67 is known as a 95 percent credibility interval. A non-Bayesian 95 percent confidence interval is 0.28 < θ < 0.68, which is very similar, but the interpretation depends on the subtle notion of “confidence.”

The third general statistical inference problem is hypothesis testing: We wish to determine if the observed data support or lend doubt to a statement about the parameter. The Bayesian approach is to calculate the posterior probability that the hypothesis is true. Depending on the value of this posterior probability, we may conclude that the hypothesis is likely to be true, likely to be false, or the result is inconclusive. In our example, we may ask if the coin is biased against heads—that is, is θ < 0.50? We find P (θ < 0.50ǀ X = 12) = 0.58. This probability is not particularly large or small, so we conclude that there is not evidence for a bias for (or against) heads.

Certain problems can arise in Bayesian hypothesis testing. For example, it is natural to ask whether the coin is fair—that is, does θ = 0.50? Because θ is a continuous random variable, P (θ = 0.50ǀ X = 12) = 0. One can perform an analysis using a prior that allows P (θ = 0.50)> 0, but the conclusions will depend on the prior. A non-Bayesian approach would not reject the hypothesis θ = 0.50 since there is no evidence against it (in fact, θ = 0.50 is in the credible interval).

This coin flip example illustrates the fundamental aspects of Bayesian inference, and some of its pros and cons. Leonard J. Savage (1954) posited a simple set of axioms and argued that all statistical inferences should logically be Bayesian. However, most practical applications of statistics tend to be non-Bayesian. There has been more usage of Bayesian statistics since about 1990 because of increasing computing power and the development of algorithms for approximating posterior distributions.

TECHNICAL NOTES

All computations were performed with the R statistical package, which is available from the Comprehensive R Archive Network. The prior and posterior in the example belong to the family of beta distributions, and the R functions dbeta, pbeta, and qbeta were used in the calculations.

SEE ALSO Bayes’ Theorem; Bayesian Econometrics; Bayesian Statistics; Distribution, Uniform; Inference, Statistical; Maximum Likelihood Regression; Probability Distributions; Randomness; Statistics in the Social Sciences

BIBLIOGRAPHY

Berger, James O., and José M. Bernardo. 1992. On the Development of the Reference Prior Method. In Bayesian Statistics 4: Proceedings of the Fourth Valencia International Meeting, eds. José M. Bernardo, James O. Berger, A. P. Dawid, and A. F. M. Smith, 35–60. London: Oxford University Press.

Berger, James O., and Thomas Sellke. 1987. Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence. Journal of the American Statistical Association 82: 112–122.

Department of Statistics and Mathematics, Wirtschaftsuniversität Wien (Vienna University of Economics and Business Administration). Comprehensive R Archive Network. http://cran.r-project.org/.

Gelman, Andrew, John B. Carlin, Hal S. Stern, and Donald B. Rubin. 2004. Bayesian Data Analysis. 2nd ed. Boca Raton, FL: Chapman and Hall/CRC.

O’Hagan, Anthony, and Jonathan Forster. 2004. Kendall’s Advanced Theory of Statistics, vol. 2B: Bayesian Inference. 2nd ed. London: Arnold.

Savage, Leonard J. [1954] 1972. The Foundations of Statistics. 2nd ed. New York: Dover.

Dennis D. Cox

More From encyclopedia.com