Significance level

A method to describe the reliability of test results.

When researchers measure a behavior, they often compare groups to determine whether they differ on that behavior. The ultimate goal is to determine whether the difference would occur if the measurements were administered a second time, or whether the difference is accidental and not likely to recur. The degree of reliability relates to the concept of significance level. The significance level refers to how likely it is that an error (that is, a wrong decision about whether the groups differ from one another) would be made. Psychologists generally accept a 5 percent error rate as reasonable. In order to decide whether differences are reliable, psychologists conduct statistical tests that provide a measure of confidence in their conclusions. This area of statistics is called inferential statistics because psychologists draw inferences, or conclusions, about what would happen if they made similar measurements with a different set of subjects.

If two similar groups are being measured, then they will produce different scores even though the difference is not particularly meaningful. If a researcher measures how much time students in separate mathematics classes take to solve a similar problem, the average for those two classes is likely to differ somewhat, even if the two classes consist of students with comparable abilities

Further Reading

Berman, Simeon M. Mathematical Statistics: An Introduction Based on the Normal Distribution. Scranton, PA: Intext Educational Publishers, 1971.

Christensen, Larry B. Experimental Methodology, 5th ed. Boston: Allyn and Bacon, 1991.

D'Amato, M. R. Experimental Psychology: Methodology, Psychophysics, and Learning. New York: McGraw-Hill, 1970

Elmes, David G. Research Methods in Psychology, 4th ed. St. Paul: West Publishing Company, 1992.

Kantowitz, Barry H. Experimental Psychology: Understanding Psychological Research, 5th ed. St. Paul: West Publishing Company, 1994.

Martin, David W. Doing Psychology Experiments, 2nd ed. Monterey, CA: Brooks/Cole, 1985.