From Thomas Bayes, c. 1701–1761
The Bayesians
The most widely held theory of knowledge among scientists and engineers today has no single author. It says that to be rational is to hold beliefs in degrees and to shift them, by one short formula, as the evidence comes in.
The view
Thomas Bayes was a Presbyterian minister in Tunbridge Wells. His essay on a problem in the theory of chances was published after his death, in 1763, by his friend Richard Price. The French mathematician Laplace reached the same result independently and made far more of it. The philosophy now called Bayesianism was built on it in the twentieth century by Frank Ramsey, Bruno de Finetti, Leonard Savage and E. T. Jaynes, among others.
It has three parts. First, belief is not all-or-nothing. It comes in degrees, called credences, which can be written as numbers from 0 to 1. Second, a rational person's credences obey the rules of probability. If they do not, a bookmaker could offer a set of bets, each of which looks fair to that person, on which they are certain to lose overall. Third, when evidence arrives, each credence is updated by a fixed rule.
The probability of a hypothesis given the evidence equals the probability of the evidence given the hypothesis, times the prior probability of the hypothesis, divided by the probability of the evidence. Bayes' theorem
The argument
The theorem itself is a piece of mathematics that nobody disputes, and it corrects real mistakes. Suppose a disease affects one person in a thousand, and a test for it gives a false alarm five times in a hundred. A positive result feels damning. Bayes' theorem says the chance that you have the disease is about two in a hundred, because the healthy people who test positive vastly outnumber the sick.
Bayesians extend this from diseases to theories. A scientist starts with a prior credence in each rival hypothesis. Each observation raises the credence of the hypotheses that predicted it and lowers the rest. Evidence confirms a theory when it raises its probability. With enough evidence, people who began with different priors are drawn together. No theory ever reaches certainty, and none should be given a credence of exactly 0 or 1.
The appeal is plain. It turns the weighing of evidence into arithmetic, it makes room for uncertainty, and it works: Bayesian methods run spam filters, medical trials and much of machine learning.
Where they meet Deutsch
Deutsch has no quarrel with the theorem, or with its use wherever probabilities mean something definite: the frequency of a disease, the reliability of a test, the outcome of a game of chance. Given a fixed list of possibilities and a model of how each produces the data, updating by Bayes' rule is correct.
Bayesians are also fallibilists of a kind. Nothing is certain, every belief is open to revision, and dogmatic conviction, a credence of 1, is a mistake that no evidence can ever undo.
Where they part
As a theory of how knowledge grows, Deutsch regards Bayesianism as inductivism in modern dress: the old idea that evidence supports theories, with numbers attached. He has three main objections.
It cannot create. Updating only moves credence around among hypotheses that are already on the list. The figure above makes the point. No sequence of observations turns the three hypotheses into a fourth. The decisive event in science is the appearance of a theory that nobody had thought of, and about that the formula says nothing.
Good theories are improbable. Popper's argument is that the more a theory says, the more ways it can be false, and so the lower its probability. "Something will happen tomorrow" is nearly certain and worthless. Science looks for the opposite: bold theories with high content, which are improbable for that very reason. A method that told us to prefer the more probable hypothesis would tell us to prefer the emptier one. In 1983 Popper and David Miller published an argument that probabilistic support, where it exists, is never inductive.
Credence is not explanation. We do not accept the tilt theory of the seasons because it is, say, 99.9 per cent probable. We accept it because it is a good explanation, hard to vary, and its rivals are bad ones. A bad explanation is not made better by evidence that agrees with it. And theories are not held or dropped by degrees. A problem is found, and a better theory replaces the old one whole.
The dispute bears on machines. If learning is Bayesian updating, a system that updates on enough data is doing what scientists do. If Deutsch is right, it is doing something else.
| The Bayesians | Deutsch | |
|---|---|---|
| Where does knowledge come from? | Evidence, which shifts degrees of belief. | Conjecture: guesses, improved by criticism and experiment. |
| What does justification do? | A degree of belief is justified when it has been updated correctly by Bayes' theorem. | Nothing. No theory is ever justified, and none needs to be. |
| Can knowledge grow without limit? | Credences can sharpen for ever, but only among hypotheses already thought of. | Yes. Problems are inevitable, and problems are soluble. |
| What is knowledge, physically? | Degrees of belief in an agent: in practice, weights in a brain or a machine. | Information that, once embodied, tends to keep itself in existence. |
Further reading
- Thomas Bayes, "An Essay towards Solving a Problem in the Doctrine of Chances", Philosophical Transactions of the Royal Society 53 (1763).
- Colin Howson and Peter Urbach, Scientific Reasoning: The Bayesian Approach, 3rd edn (Open Court, 2006): the standard philosophical defence.
- E. T. Jaynes, Probability Theory: The Logic of Science (Cambridge, 2003).
- Karl Popper and David Miller, "A Proof of the Impossibility of Inductive Probability", Nature 302 (1983).
- Bayesian Epistemology, Stanford Encyclopedia of Philosophy.