INFORMATION THEORY

Among the more interesting trends of the past half-century has been the consolidation of probability, statistics, combinatorial optimization, information theory, and computer science into a single imposing discipline of infomatics.

Minimal Belief Change

Of special philosophical interest is the enrichment of Bayesian inference by a rule of belief change that goes by minimizing the distance between two probability distributions, P = (p ₁, … , p _k and Q = (q ₁, … , q ₂), as measured by the expected log-likelihood ratio:
(1)      
The likelihood ratio, P (e |h ):P (e |k ), is a fundamental index of the support that e accords h over k (see the entry "Foundations of Statistics").

Using the visually transparent Gibbs inequality,
(2)      ln x ≤ x − 1
with equality if and only if (iff) x = 1, in the equivalent form ln x ≥ 1 − 1/x, it follows that H (P, Q ) ≥ 0 with equality iff P = Q. Notice, however, that H (P, Q ) ≠ H (Q, P ).

Alan Turing and his wartime assistant, Irving John Good, used H (P, Q ) in their code-breaking work, but it was not until 1959 that another wartime code breaker, Solomon Kullback, developed its properties systematically in his book Information Theory and Statistics (1959), unleashing a floodtide of applications to classification, contingency tables, pattern recognition, and other topics.

A second line of development began with Claude Shannon's creation of information and coding theory (Shannon and Weaver 1949), whose central concept is a measure of uncertainty,
(3)      
that Shannon dubbed the entropy function. One sees that minimizing H (P, U ) against a uniform distribution, U = (k ⁻¹, … , k ⁻¹), namely,

is equivalent to maximizing the entropy (MAXENT). In view of this relation H (P, Q ) is often called the cross (or relative) entropy of P with respect to Q, and the rule of minimizing it MINXENT.

Shannon characterized H (P ) axiomatically as continuous, strictly increasing in k when P = (k ⁻¹, … , k ⁻¹) is uniform, and by the general form of a consistency requirement exemplified by:

where A (3) = H (⅓, ⅓, ⅓), and so on. Roughly speaking, one's uncertainty about a lottery is the same for an equivalent two-stage form of it, where in the example H (³/₉, ⁴/₉, ²/₉), is one's uncertainty about the first stage.

Exploiting the conceptual link Ludwig Boltzmann established between thermodynamic entropy and Shannon entropy through his celebrated formula, S = k ln W, Edwin T. Jaynes showed how to derive the probability distributions (over microstates) of statistical mechanics by maximizing the (Shannon) entropy subject to mean value constraints given by measured values of macro variables like pressure, volume, or internal energy. Intuitively, the maxent distribution is the most spread out of all those satisfying the constraints, the one that is maximally noncommittal with regard to missing information. Statistical mechanics could thus be seen as a branch of statistical inference operating on physically given constraints. Jaynes quickly realized that MAXENT admits of much

wider application. A collection of his papers (Jaynes 1983) sparked an explosion of applications whose subsequent progress can be followed in the published proceedings of workshops on maximum entropy and Bayesian methods held annually since 1981 (e.g., see Erickson and Smith 1988). Brian Buck and Vincent A. Macaulay's Maximum Entropy in Action (1991) is a sampler of physical applications to such fields as spectroscopy, X-ray crystallography, the structure of macromolecules, magnetic resonance, thermodynamics, and plasma physics. The method is universal, however, and applies equally well to image reconstruction or time series.

Inferring Hidden Causes

Irving John Good (1983) viewed H (P, Q ) as a natural measure of deviation suitable for testing multinomial hypotheses, and like Karl Pearson's better-known chi-squared measure of deviation

the analogous informational measure

is asymptotically distributed as χ ²_{k −1}, the chi-square distribution with k − 1 degrees of freedom. By comparing the current model to the ideally best-fitting model, the resulting psi test sets limits to how much room there is for improving support by moving to a new (possibly more complicated) model (Jaynes 2003). When such a move is indicated, MINXENT helps guide one to plausible better-fitting models, as the following example illustrates:

Seen in the first row of Table 1 are the frequencies with which the six faces of a white die turned up in N = 20,000 tosses recorded by the Swiss astronomer Rudolf Wolf. One sees at a glance that Wolf's die was biased; the question of interest is how to find physically plausible hypotheses that best account for these data. The distributions shown in rows 3 to 5 of the table afford successively better approximations to the empirical distribution, {g _j}, of row 2. Jaynes (1983) found them by moving the initial uniform distribution just enough to satisfy mean value constraints given by the data that correspond to physical imperfections of a die. Among the most plausible are:

(1) A shift in the center-of-mass due to the different amounts of ivory removed from opposite faces

(2) Oblateness, due to the difficulty of machining a perfect cube (Jaynes 1983)

The first implies a mean higher than the 3.5 of an "honest" die. Maximizing the entropy subject to the observed mean, 〈i 〉 = 3.5983, yields the distribution P of row 3. Notice that it assigns higher probability to the 6 spot than the 1 spot, the 5 spot than the 2 spot, and the 4 spot than the 3 spot, with decreasing margins, which is just what the posited physical cause would lead one to expect. But while P improves on the uniform distribution, its fit to the data is still poor, with X ² = 33.4. This points to the existence of another constraint. The lower than expected frequencies of faces 3 and 4 are best explained by oblateness of the die, the 3-4 dimension being longer than the other two. This is reflected in the nonzero mean value, f̄ ₂ = 0.1393, of the oblateness function, f ₂(j ) = 1 for j = 1, 2, 5, 6 and f ₂(j ) = −2 for j = 3, 4. Adding this constraint to the first leads to the maxent distribution Q for both constraints (row 4). The fit to Wolf's data is now fairly good, but there is some slight evidence for a third constraint, a slightly chipped 2-3-5 corner. Maximizing entropy subject to all three constraints then yields the distribution of the last row, whose fit to the data is preternaturally good. Note: It makes no difference in which order the constraints are applied, provided each one is retained in applying the next one.

If Wolf's die still exists, one could actually put the posited physical imperfections to the test of careful measurements. In any case, one would expect other dice to exhibit the first two imperfections. As it happens, Wolf also tossed a red die 20,000 times, and this expectation is realized (Jaynes 1983). To permit such inferences to physical causes, one must assume, of course, that Wolf tossed his dice in a manner that precludes skill.

MAXENT first enters, then, as a technique of mathematical modeling, as a means of generating plausible hypotheses for explaining old data or testing against new data (Good 1983, pp. xvi–xvii, 41, 99–100). Indeed, most of the probability distributions that figure prominently in pure and applied probability are maxent distributions for a suitable set of constraints. For example, the exponential distribution is maxent for a positive random variable whose (finite) first moment is given. (This result depends on extending the cross entropy function to continuous distributions, sums giving way to integrals.) Jaynes (2003, §7.6) even makes a case for thinking the ubiquitous use of the normal law of errors is owing to the fact that it is the maxent distribution having specified values of the first two moments.

When predictions based on MAXENT are verified, one's belief in the completeness of the given constraints is borne out, but one learns most when one's predictions fail. For then one infers the existence of a previously unsuspected cause to which the observed deviations point one. However, for this kind of "learning from error" to be effective requires that those inferences be one's best inferences (Jaynes 2003, p. 326). In what sense, then, are MINXENT or MAXENT inferences best possible?

One can look, first, at how MAXENT operates on a mean value constraint:

With ∑ p _i = 1, the method of Lagrange for constrained extrema yields a solution,
(4)      p _i = Z(λ )⁻¹ exp(λf (x _i )) i = 1, 2, … , n
with. More generally, given m < n such constraints, , a solution is
(5)      p _i = Z (λ₁, …, λ_m )⁻¹ exp[λ₁f ₁(x _i ) + … + λ_m f _m (x _i )]
for i = 1, … , n, where the partition function, Z, is defined by:
(5a)
Moreover, the moments of the resulting maxent distribution are given by the derivatives of ln Z, so that, in particular, the mean values are:
(5b)      
as one can verify. Thus, Z is a kind of (exponential) generating function and is called the partition function for reasons given by Jaynes (2003, pp. 280–282). The parallel minxent distribution is given by:
(6)      
where P ⁰ = (p ^{0 z}, …, p ^{0 n}) is the initial "pre-distribution." (6) reduces to (5) when p ^{0 i} = n ⁻¹ is constant.

For a simple discrete example, let the mean for a die be 〈i 〉 = 4. Here, x _i = i and f (x ) = x, so the partition function becomes (with x = e ^λ ):

Then since dx /dλ = e ^λ = x, the chain rule yields:

whereupon, setting d ln Z /dλ = 4, 3(1 − x )(1 − x ⁶) = 5x ⁷ − 6x ⁶ + x, which is solved numerically for x to yield, x = 1.190804264, or λ = ln x = 0.1746289309. Hence, the maxent distribution of mean 4 is:
(p ₁, … , p ₆) = (.103, .123, .146, .174, .207, .247)
The general formula for mean f̄ works out to (6 − f̄ )x ⁷ − (7 − f̄ )x ⁶ + f̄x − f̄ + 1 = 0, and one seeks a root other than x = 1. Verify that for mean 3.5, the only root is x = 1, so the maxent distribution is uniform. Thus, MINXENT gives back the pre-distribution if it already satisfies the constraint—a redundancy property that should hold generally. Computer programs are available for finding maxent distributions subject to mean value constraints.

The method of Lagrange for constrained extrema is not guaranteed to yield a global maximum, and so to settle this point, one invokes the Gibbs inequality (2) to obtain:

when ∑ u _i = ∑ p _i = 1, whence
−∑ p _i ln p _i ≤ −∑ p _i ln u _i
with equality iff p _i = u _i . Knowing the form (5) of the maxent distribution, one sets
u _i = Z (λ _l , … , λ _m )⁻¹ exp[λ _l f _l (x _i ) + … + λ _m f _m (x _i )]
and the last inequality then specializes to:

with equality iff the p _i are given by (5). This not only shows that (5) is the (one and only) distribution of maximum entropy satisfying the given constraints, but that the right side,
(7)      
is the maximum entropy permitted by those constraints. Any distribution of lower entropy must be importing additional information. Or, as in the example of Wolf's die, if the data distribution is of lower entropy than a hypothesized distribution, an additional condition must be constraining those data to lie in a proper subset of those allowed by the hypothesis in question.

Axiomatic Characterization

Every property of MINXENT (MAXENT) rules out potential rivals. Write P = P ⁰ ○ C for the distribution, P, nearest the pre-distribution, P ⁰, among all those satisfying a constraint, C (of class C ), where for equality or inequality linear constraints the class C is closed and convex and there is a nearest P to P ⁰. The following properties of MINXENT are then easily proved (Shore and Johnson 1980, §4; Williams 1980):

Uniqueness: The minxent distribution is unique

Redundancy P ○ C = P when P satisfies the constraint C [since H (Q, P ) = 0 only if Q = P ]

Chain consistency: The order in which constraints are applied is immaterial, provided each is retained in applying the next one

System independence: The "post-distribution" should not import a dependence between two variates that is neither implied by the constraints nor the pre-distribution

Invariance: P = P ⁰ ○ C should not depend on one's choice of a coordinate system

Subset independence: It should not matter whether one obtains a conditional distribution for disjoint subsets of outcomes by finding the post-distribution nearest the conditional prior for each subset or by obtaining the post-distribution for the whole outcome space and conditioning it on each subset

To illustrate the last property, the even and odd faces in tossing a die may be reported separately. Let the prior be uniform and the mean for both the odd and even faces be 4. By redundancy, the post-distribution, Q, is also uniform on the even faces: Q (2) = Q (4) = Q (6) = ⅓, while for the odd faces, one finds (as above) that Q (1) = 0.1162, Q (3) = 0.2676, and Q (5) = 0.6162. One can also solve the problem by finding the post-distribution for all six states by applying the mean value constraints conjointly obtaining:

To condition on the subsets of odd and even faces, divide each column by its sum and obtain the same conditional distributions found earlier. If these two ways of solving the problem did not agree, MINXENT could be justly deemed inconsistent.

In 1980 John E. Shore and Rodney W. Johnson vindicated Jaynes's conjecture that "deductions made from any other information measure … will eventually lead to contradictions" (1983, p. 9) by deriving MINXENT for mean value constraints from uniqueness and the last three mentioned properties. All these ring changes on the consistency requirement that two ways of doing a calculation must agree, the very condition from which Richard T. Cox derived the basic rules of probability (see the entry "Foundations of Statistics"). Rather than attempt a sketch of their proof, one can illustrate how rival rules violate the Shore-Johnson axioms and lead to contradictions.

Inconsistency of Other Rules

An alternative measure of the spread of a distribution, Σp ²i, dubbed the repeat rate by Turing, is, like entropy, continuous and assumes its extreme values of 1/n and 1 at the extremes of uniformity and concentration. The closely related Gini diversity, , has a well-established place in statistics as a measure of the qualitative diversity of a population, with widespread applications to such fields as genetics, ecology, and linguistics. Table 2 shows how closely the distribution of a die of mean 4 obtained by minimizing Σp ²i, approximates the maxent distribution for this constraint found earlier.

On a superficial examination, one might easily suppose that the repeat rate (RR) rule performs about as well as MAXENT. Still, one knows it must violate one of Shore and Johnson's (1980) consistency postulates, and inconsistency always brings a degradation of performance in its train.

In casting a red and a white die with mean values, E (R ) = 4 and E (W ) = 3, and a uniform prior, the method of Lagrange for the RR-rule leads in this case to three linear equations in three unknowns (the Lagrange multipliers), which are easily solved to yield the joint post-distribution:

from which the marginal distributions for R and W are computed to be:

and

One easily checks that p _i q _i ≠ ¹/₃₆ = π(i,i ). Hence, the RR-rule violates system independence. Its inconsistency shows up even more clearly when one enlarges the problem to the case of symmetric mean values, E (R ) = 3.5 + Δ and E (W ) = 3.5 − Δ. From the joint post-distribution, one can compute a value of Δ, namely, Δ = ⁷/₁₂ at which the RR-rule makes the smallest of the joint probabilities zero, even though no outcome is excluded by the pre-distribution or the constraints. For values of Δ greater than ⁷/₁₂, the RR-rule breaks down completely, making some of the joint probabilities negative.

Consider the more general family of rules that minimize a Csiszar divergence:
H _f (P, Q ) = ∑q _i f (p _i q _i )
with f a convex function. This family includes MINXENT as the special case f (x ) = x ln x, as well as the chi-squared rule that minimizes, given by f (x ) = (x − 1)². Applied to a uniform prior, q _i = 1/n, this is minimized by minimizing Σp ²i, hence, it inherits the inconsistencies of the RR-rule.

Relation to Bayesian Conditioning

The upshot of Shore and Johnson's (1980) axiomatic derivation of MINXENT is to place it on a par with Bayes's rule for revising a probability assignment—the most basic rule of belief change. Hence, it is of interest to see what MINXENT delivers in this case (Williams 1980). For any distribution P satisfying P (B ) = 1:
(8)      H (P, P ⁰) = H (P _B , P ⁰_B ) − ln P ⁰(B )
writing P _B (A ) = P (AB )/P (B ) for the conditional measure. For then using

since P _B (x _i ) = P (x _i )/P (B ) = P (x _i ), which proves (8). Since (8) is clearly a minimum when H (P _B , P ⁰_B) = 0, hence, when (P _B = P ⁰_B), one sees that the distribution P nearest P ⁰ among those for which P (B ) = 1 is the Bayesian posterior distribution, P ⁰_B.

What if the constraint P (B ) = 1 is weakened to P (B ) = q, 0 < q < 1? For any such P one has the following straightforward generalization of (8):
(8a)      
with q̄ = 1 − q. The right side is minimized by making both H (P _B , P ⁰_B) and H (P _B̄ , P ⁰_B) zero, which means that the nearest P to P ⁰ is given by:

a q -weighted average of the conditional distributions for B and its negation. There is an obvious generalization to a partition B _l , … , B _n and the constraint P (B _i ) = q _i , with ∑q _i = 1. This special case of MINXENT is known as Jeffrey conditioning. Indeed, Williams (1980) generalizes

further to the case where the B 's need not be mutually exclusive. The validity of this rule requires that the sole affect of the datum or sensory input is to raise the probability of B to a value q < 1. For conditions under which Jeffrey conditioning is not valid, see Jaynes (2003, §5.6).

Frequency Connections

MAXENT also has frequency connections (Jaynes 1983). Of the k ^N outcomes (i.e., outcome sequences) of N trials, the number that yields category counts (n _l, … , n _k) with ∑n _i = N is given by the multinomial coefficient:

Using Stirling's approximation to the factorial function, one easily proves that
(9)      N ⁻¹ ln W → H (f _l , … , f _k )
Hence, the maxent distribution is W -maximizing, hence realized by the most outcomes. Moreover, given two sets of relative frequencies, {f _i }and {f _i ′},
(10)      
gives the ratio of the number of ways each can be realized, where H = H (f _l , … , f _k ) and H ′ = H (f _l ′, … , f _k ′), (f _i / f _i ′) and . For example, for the two distributions of Table 2, H = 1.7485056 and H ′ = 1.7470082, and so at N = 20,000 trials, the maxent distribution is realized by W /W ′ = 9.86 × 10¹² more outcomes than the similar looking distribution of the RR-rule. The peak is thus enormously sharp. Just how sharp is quantified by Jaynes's concentration theorem (1983), which allows one to compute the fraction of possible outcome sequences whose category counts, f _i , have entropy in the range H _max − ΔH ≤ H (f _l , … , f _k ) ≤ H _max. This gives, in effect, a new kind of significance test for detecting when additional constraints are hidden in one's data.

Not unrelated results concern typical outcomes of a stationary Markov process (Khinchin 1957). Namely, if H is the entropy of a regular Markov chain and s sufficiently large, then almost all s -step outcomes C satisfy:

with η arbitrarily small. That is, almost all s -step sequences of the process deliver information arbitrarily close to the average, H ^(s) = sH. The entropy is defined in the obvious way as the expectation of
H _i = −∑p _ik log p _ik
the one-step uncertainty, so that H = σ_i P _i H _i where P = (P _l , … , P _k ) is the stationary distribution and (p _ik ) is the one-step transition matrix.

Aleksandr Khinchin's prophetic remark, that "the study of entropy will become a permanent part of probability theory" (1957, p. 2), has been borne out not only by the flowering of the maxent method but also by D. S. Ornstein's proof that entropy is a complete invariant of an ergodic Markov chain (for an informal treatment and references to the mathematical literature, see Suppes 2002, §4.5). That is, two ergodic chains (in which any state is reachable from any other) are isomorphic iff they have the same period and the same entropy.

Information Theory and Statistical Mechanics

The frequency implications of MAXENT play their most important role, however, in statistical mechanics. Thus, Ludwig Boltzmann found the famous Maxwell-Boltzmann energy distribution for molecules in a conservative force field by partitioning the 6n-dimensional phase space of position and velocity (or momentum) coordinates into cells, R _k , small enough for the energy to be a constant, E _k , and large enough to contain a sizeable number, N _k , of molecules. Then the total number, N, and the total energy, E, are constants of the molecular motion. Boltzmann argues that the most probable distribution, (N_l , … , N_s ), is the one that is realized by the most microstates among those compatible with the constraints:
∑N _i = N and ∑N _i E _i = E
By virtue of (5), this most probable distribution,
(11)      
is none other than the maxent distribution for the given constraints.

In this derivation, Boltzmann may be said to have launched MAXENT and the information theoretical approach to statistical mechanics. All the canonical distributions J. Willard Gibbs (1902) later derived are simply maxent distributions for other sets of constraints, for example, that for fixed values of the total energy and angular momentum is Gibbs's rotational ensemble. Unfortunately, Gibbs slipped in the logical basis of his derivation so unobtrusively that most readers missed it. Moreover, he provided no clear or compelling rationale, so that in their famous review article of statistical mechanics published a decade later, Paul Ehrenfest and Tatiana Ehrenfest (1912) dismissed Gibbs's method of derivation as "a mere analytic trick" (for the relevant history, see Jaynes 1983, pp. 98ff). Thus began a long siege of confusion and controversy over the justification of Gibbs's formalism that continues to this day, notwithstanding Jaynes's rediscovery of the MAXENT method of Gibbs in 1957 and his clear rationale for using it (1983, chapter 1). In particular, the information theoretical approach dispenses with the ergodic hypothesis.

Jaynes's second great contribution was to extend the Gibbsian (MAXENT) formalism to irreversible processes and nonequilibrium thermodynamics (1983, chapter 10, §D). He writes:

The final breakthrough came in the Christmas vacation period of 1962 when, after all else had failed, I finally had the courage to sit down and work out all the details of the calculations that result from using the Maximum Entropy Principle; and nothing else. Within three days the new formalism was in hand, masses of the known correct results of Onsager, Wiener, Kirkwood, Callen, Kubo, Mori, MacLennon, were pouring out as special cases, just as fast as I could write them down; and it was clear that this was it. (p. 239)

The unbelievably short derivation of (11) as the equilibrium distribution of the energies has seemed too short to many physicists and philosophers. This initial impression is only reinforced when it is seen that (11) implies the familiar barometric formula,
(11a)      ρ (z) = ρ (0) exp(−βmgz )
for the density of the atmosphere at height z, as well as Maxwell's velocity distribution in the form:
(12)      f (v _b ) = B (β)exp(−βmv ²_b /2)
where f (v _b )dv _b is now the probability that a molecule has a velocity in a tiny neighborhood of v _b . Indeed, even more follows, for (12) does not depend on the position of the molecule (an assumption Boltzmann was forced to make in his derivation of the Maxwell distribution from his collision equation). This implies, in turn, the dynamic stability of the distribution. However, the MAXENT derivation seems to ignore the dynamics altogether.

Jaynes responded in several ways. First, the derivation does not ignore the dynamics; it uses conservation of energy as well as the preservation of the volumes of the cells, R _k , under evolution of the system (Liouville's theorem). Jaynes emphasizes that, in addition, one is trying to predict reproducible macrostates. These are ipso facto under the experimenter's control, and so the myriads of details concerning microstates not under his control must needs be irrelevant for prediction. Moreover, reproducible macroscopic properties must be characteristic of the overwhelming preponderance of microstates in the allowed region of the phase space. Given the large number of degrees of freedom entailed, the maxent distribution will be enormously peaked. Hence, predictions of other macro quantities based on their mean values will be correct (within experimental error) with probability close to one (Jaynes 1983). As far as Jaynes is concerned, these considerations fully explain the predictive success of equilibrium thermodynamics as "inferences from the available information."

In particular, there is no need to appeal, as Maxwell and Boltzmann did (but Gibbs did not) to the equality of infinite time averages and averages with respect to the canonical distribution. Even if this could be established from some other easily verified assumption—the program of ergodic theory—it would be nothing to the purpose. For one would have to show, in addition, that the averages over finite time intervals involved in measuring macro quantities closely approximate their infinite time averages, and there are positive reasons to doubt this (Jaynes 1983).

Apart from the clarity, unity, and simplicity the information theoretical approach brings to the foundation of statistical mechanics, David Hestenes (1993) considers Jaynes's greatest merit to lie in his recognition that "in the evolution of statistical mechanics the principles of physics had gotten confused with principles of statistical inference" (1993, p. 153).

Jaynes regards the formalism of quantum mechanics as a similar "nasty omelet" scrambling together properties of physical systems and our information about them in ways that are difficult to unscramble. See the cited article by David Hestenes for one noteworthy attempt to disentangle subjective and objective aspects of the electron wave function, namely, a probability factor and a kinematic factor, using a powerful "universal geometric calculus" based on Hermann Grassmann's Ausdehnumgslehre. Hestenes purports to show, in particular, that the complex "probability amplitudes" of the formalism have nothing to do with probability per se but have, instead, a physical origin in the "Zitterbewegung", or circular dance, of the electrons generating their spin and magnetic moment.

Jaynes's views on quantum mechanics are outlined (with references) in the same volume—a festschrift in his honor—in which the article by Hestenes appears. For other views, see Philosophy of Statistical Mechanics and Quantum Mechanics.

See also Bayes, Bayes's Theorem, Bayesian Approach to Philosophy of Science; Bell, John, and Bell's Theorem; Bohr, Niels; Boltzmann, Ludwig; Copenhagen Interpretation; Einstein, Albert; Gibbs, Josiah; Maxwell, James Clerk; Philosophy of Statistical Mechanics; Probability; Quantum Mechanics; Semantics; Turing, Alan M.

Bibliography

Buck, Brian, and Vincent A. Macaulay, eds. Maximum Entropy in Action. Oxford, U.K.: Clarendon Press, 1991.

Cover, Thomas M., and Joy A. Thomas. Elements of Information Theory. New York: Wiley, 1991.

Erickson, Gary J., and C. Ray Smith, eds. Maximum-Entropy and Bayesian Methods in Science and Engineering. 2 vols. Dordrecht, Netherlands: Kluwer Academic, 1988.

Gibbs, J. Willard. Elementary Principles of Statistical Mechanics. New Haven, CT: Yale University Press, 1902.

Good, Irving John. Good Thinking. Minneapolis: University of Minnesota Press, 1983.

Hestenes, David. "The Kinematic Origin of Complex Wave Functions." In Physics and Probability: Essays in Honor of Edwin T. Jaynes, edited by W. T. Grandy Jr. and P. W. Milonni, 153–160. New York: Cambridge University Press, 1993.

Jaynes, Edwin T. "A Backward Look to the Future." In Physics and Probability: Essays in Honor of Edwin T. Jaynes, edited by W. T. Grandy and P. W. Milonni, 261–275. New York: Cambridge University Press, 1993.

Jaynes, Edwin T. "Clearing up Mysteries—The Original Goal." In Maximum Entropy and Bayesian Methods, edited by John Skilling. Dordrecht, Netherlands: Kluwer Academic, 1989.

Jaynes, Edwin T. Papers on Probability, Statistics, and Statistical Physics, edited by R. D. Rosenkrantz. Dordrecht, Netherlands: D. Reidel, 1983.

Jaynes, Edwin T. Probability Theory: The Logic of Science. New York: Cambridge University Press, 2003.

Khinchin, Aleksandr. Mathematical Foundations of Information Theory. New York: Dover, 1957.

Kullback, Solomon. Information Theory and Statistics. New York: Wiley, 1959.

Shannon, Claude, and Warren Weaver. The Mathematical Theory of Communication. Urbana: University of Illinois Press, 1949.

Shore, John E., and Rodney W. Johnson. "Axiomatic Derivation of Maximum Entropy and Minimum Cross Entropy." IEEE Transactions on Information IT-26 (1980): 26–37.

Suppes, Patrick. Representation and Invariance of Scientific Structures. Stanford, CA: CSLI Publications, 2002.

Williams, Paul M. "Bayesian Conditionalization and the Principle of Minimum Information." British Journal for the Philosophy of Science 31 (1980): 131–144.

Roger D. Rosenkrantz (2005)