REINFORCEMENT

In its earliest technical usages, the term reinforcement implied strengthening, echoing its colloquial usage. It has been applied to a broad range of phenomena in learning, including the operant or instrumental behavior studied by B. F. Skinner and the respondent or classical conditioning procedures of Ivan Pavlov. The term, which also applies to certain types of machine learning procedures studied by computer scientists and engineers, now refers mostly to cases in which behavior has some consequences and, by virtue of these consequences, comes to occur more often. The term reward, sometimes used as a nontechnical synonym, is not equivalent. For example, one can speak of delivering a reward even without evidence that the reward has an effect on behavior.

As a classical example of reinforcement, imagine a rat in a chamber with a lever and a cup into which food pellets can be delivered. If pressing the lever does nothing, the rat presses only occasionally. If each press produces a food pellet, however, the rat presses the lever more often. The food is called a reinforcer, and the rat's lever press is said to have been reinforced. The response that increases must be the one that produced the consequence. For example, if lever presses produce shock and only the rat's jumping increases, it would be inappropriate to speak of either lever pressing or jumping as reinforced. An ambiguity of usage is that reinforcement sometimes refers to delivery of the reinforcer ("the lever press was reinforced" means that lever presses produced food) and sometimes to the resulting behavior change ("the lever press was reinforced" means that lever presses occurred more often because they produced food). The sense is usually clear from the context.

The Operant Nature of Reinforcement

Edward Thorndike's experiments with animals in problem boxes were early examples of studies of reinforcement. Typically, a food-deprived animal was placed inside a box with food available outside. Sooner or later, typically by chance, the animal operated a device that released the door and freed it from the box. With repetition it operated the device more and more rapidly upon being placed in the box. To describe his findings, Thorndike formulated his law of effect. The law went through many revisions, but its essence was that responses can be made more probable by some consequences and less probable by others; in language closer to Thorndike's, responses with satisfying effects are stamped in, whereas those with annoying effects are stamped out.

Behavior that is modified by its consequences is called operant behavior or, in older and less common usages, instrumental behavior, which is behavior that operates on its environment. It is said to be emitted because it is primarily determined by its consequences; it does not require an eliciting stimulus, as in reflex relations (e.g., as when a puff of air to the eye produces a blink). Operants are classes of responses defined by their environmental effects rather than by their topography (their form or the particular muscle groups involved). For example, a rat might press a lever with its left paw, its right paw, or both paws; it might even depress the lever by biting or sitting on it. But similar movements at the other end of the chamber, far from the lever, would not be lever presses no matter how closely they resembled the responses that operated the lever.

Skinner significantly extended the analysis of reinforcement. One of his critical contributions was to clarify the difference between operant learning and the classical or respondent learning that had been studied by Pavlov (in Pavlov's experiments, the organism's responses had no effect on the stimuli that were presented).

Consider the relation between a red traffic light and a driver's stepping on the brakes. The red light sets the occasion for stepping on the brakes; this response occurs at a red light and not at a green light because it has different consequences in the presence of each stimulus. A stimulus that signals or sets the occasion for different consequences of responding is discriminative. Three terms are involved: the discriminative stimulus, the response, and the consequences of the response in the presence of that stimulus.

For example, imagine a pigeon in a chamber with a feeder; above the feeder is a small disk or key that can be lit from behind with lights of different colors. Suppose that reinforcement of the pigeon's key peck depends on the presence of a green or a red light. If the peck produces food only when the key is green, the pigeon will come to peck in the presence of green but not red. Its pecking in the presence of green is a discriminated operant. Discriminative stimuli correspond to the stimuli colloquially called signals or cues. They do not elicit responses; instead, they set the occasions on which responses have consequences.

Positive and Negative Reinforcement

A variety of consequences can serve as reinforcers, ranging from those of obvious biological significance, such as food or water or a sexual partner, to minor changes in things seen or heard or touched. Consequences are not restricted merely to the production of stimuli. Responses can remove stimuli, as when operating a switch turns off a light; responses can prevent stimuli from occurring, as when unplugging a lamp before repairing it eliminates the possibility of a shock; and responses can even change the consequences of other responses, as when replacing a burned-out light bulb makes the previously ineffective response of operating the light switch an effective response again. Each type of consequence may affect later behavior.

A stimulus is a positive reinforcer if its presentation increases the likelihood of responses that produce it and a negative reinforcer if its removal increases the likelihood of responses that terminate or prevent or postpone it. Negative reinforcers are sometimes called aversive stimuli. Responding that turns off an aversive stimulus (e.g., when a rat turns off a loud noise by pressing a lever) is escape responding. Responding that prevents or postpones an aversive stimulus (e.g., when a rat's lever press prevents or postpones the delivery of an electric shock) is avoidance responding.

Reinforcement and Punishment

Consequences can reduce responding instead of increasing it. For example, if shock is delivered whenever a rat grooms its tail, the rat may groom its tail less often. Responding reduced by its consequences is therefore punished. Punishment, simply the inverse of reinforcement, has had a more controversial history. Thorndike's early versions of the law of effect argued that behavior could be stamped out by annoyers as well as stamped in by satisfiers, but he later withdrew the part that dealt with the stamping out of behavior.

A rat's reinforced lever pressing can be reduced when presses are punished by shock, but the responding recovers to earlier levels once punishment is discontinued. Because punishment suppresses responding only temporarily, some have argued that it is ineffective. Yet reinforcement would also have to be judged ineffective by this criterion. In early studies standards for the effectiveness of punishment were different from those for the effectiveness of reinforcement. Investigations tended to concentrate on the recovery of responding after punishment was discontinued rather than on the reduction of responding during punishment.

Punishment is effective, but that does not recommend it. It is usually an undesirable method for eliminating behavior. Thorndike and his successors were probably right for the wrong reasons. For example, one major problem is that aversive stimuli used as punishers have side effects, such as eliciting aggressive behavior.

Shaping and the Selection of Behavior

Reinforcement does not establish connections or associations between responses and stimuli. Rather, it changes the probability of classes of responses. Some responses become more likely and others less likely. In other words, some responses survive in the organism's behavior, whereas others do not. In this respect, reinforcement works by selection. It operates on populations of responses within the lifetime of the individual organism much as evolutionary selection operates on populations of organisms over successive generations.

The selective nature of reinforcement is best illustrated by shaping, a procedure that creates new responses through the successive reinforcement of other responses that more and more closely approximate it. For example, a rat may rarely if ever press a lever with a force that approximates its own body weight, but it can be induced to do so by selective reinforcement of stronger and stronger presses. Shaping begins with whatever behavior is available. At the start, most presses will probably be weak; but with reinforcement of the strongest ones, the force of pressing will increase a little, and the criterion for reinforcement then moves up to the strongest of the new population of responses. This step in turn produces another increase in force and another change in criterion, and so on, until the rat is pressing with a force that might never have occurred in the absence of the shaping procedure. Reinforcement has had many useful applications, and shaping has figured in a significant way in many of them, including the training of animals and the production of speech in autistic children.

Computational Reinforcement

Learning through reinforcement is studied in computer science and engineering, where the term reinforcement learning is applied to learning algorithms that include some of the principles of reinforcement observed in animal behavior. However, a different type of learning is most commonly studied in these fields. This is called supervised learning (or sometimes "learning by example" or "learning with a teacher") in which a system learns from a collection of examples, where each example consists of an input, usually is a list of numbers, together with a desired, or target, output specifying the response that the system should produce when given that input. The object of the task is for the system to learn a response rule that produces the desired responses for the example inputs and also for novel inputs it may receive in the future.

A reinforcement learning system, in contrast, learns from signals that evaluate the quality of the learning system's behavior without explicitly telling the system what its behavior should be. Evaluative feedback tells the learning system whether or not, and possibly by how much, its behavior has improved or deteriorated; or it provides a measure of the "goodness" of the behavior; or it just provides an indication of success or failure. Evaluative feedback does not directly tell the learner what it should have done or how it should change its behavior. Instead of trying to match a standard of correctness (i.e., match desired outputs), a reinforcement learning system tries to maximize the goodness of behavior as indicated by evaluative feedback. To do this, it has to actively try alternatives, compare the resulting evaluations, and use some kind of selection mechanism to guide behavior toward the better alternatives. Whereas supervised learning is instructional, reinforcement learning is selectional, like reinforcement in animal learning.

Reinforcement learning is useful from a computational perspective because the evaluation component, sometimes called a "critic," requires much less information and knowledge than the "teacher" in a supervised learning system. It is possible to evaluate a system's behavior without knowing what the correct behavior would be. In fact, a critic does not even need access to the learning system's actions; it can evaluate their consequences on some complex process. It does not need to know anything about the mechanism by which the actions produce these consequences. A complication is that actions can have delayed as well as immediate consequences; evaluative feedback generally evaluates the consequences of all of the system's past behavior. Researchers have developed a number of methods enabling a system to learn efficiently under these conditions.

Several technologies have benefited from applications of reinforcement learning. Although inspired by reinforcement phenomena in animal behavior, research in computational reinforcement learning seeks mainly to create learning methods that can make artificial systems more autonomous and adaptable; it does attempt to create accurate models of animal learning.

Conclusion

The principle of reinforcement emerged in experimental psychology to account for increases in an animal's responding that are due to its consequences. Computer scientists and engineers have adapted this principle to create systems that improve their behavior over time by learning from their own experiences.

See also:ALGORITHMS, LEARNING; OPERANT BEHAVIOR

Bibliography

Estes, W. K. (1944). An experimental study of punishment. Psychological Monographs 57, 263.

Pavlov, I. P. (1927). Conditioned reflexes, trans. G. V. Anrep. London: Oxford University Press.

Skinner, B. F. (1938). The behavior of organisms. New York: Appleton-Century-Crofts.

—— (1969). Contingencies of reinforcement. New York: Appleton- Century-Crofts.

—— (1981). Selection by consequences. Science 213, 501-504.

Sutton, R. S., and Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.

Thorndike, E. L. (1898). Animal intelligence: An experimental study of the associative processes in animals. Psychological Review Monograph Supplements 2, 109.

A. CharlesCatania

Revised byAndrew G.Barto