Reliability
RELIABILITY
The term reliability can be used to indicate a virtue in a person, a feature of scientific knowledge, or the quality of a product, process, or system. Personal unreliability makes an individual difficult to trust. Unreliability in science calls the scientific enterprise into question. Lack of reliability in technology or engineering undermines utility and public confidence and perhaps commercial success. In all cases the pursuit of reliability is a conscious goal.
Scientific Reliability as Replication
Reliability in science takes its primary form as replicability. Research experiments and research must be performed and then communicated in such a way that they can be replicated by others or the results cannot become part of the edifice of science. Both replicability in principle and actual replication by diverse members of the scientific community are central to the processes of science that make the knowledge produced by science uniquely reliable and able to be trusted both within the community and by nonscientists.
Replication is easier to achieve in some scientific domains than in others, but when it fails, the science is judged unreliable. Historically replication was established first in physics and chemistry, and so in the physical sciences especially lack of replicability can become newsworthy. For example, the inability of other scientists to replicate the experiments on which Stanley Pons and Martin Fleischmann based their announcement of the discovery of cold fusion in 1989 doomed the credibility of their claims.
As Harry Collins and Trevor Pinch (1998) have shown in case studies, the replication of particular experiments often depends on the phenomenon of "golden hands." Not all experimenters are equally skilled at setting up and performing experiments, and subtle differences can be more relevant than it is possible to articulate clearly in the methods section of a research article.
In science another version of replicability is associated with peer review. Peer review procedures for scientific publication and for decision making about grants in effect depend on two or more persons coming to the same conclusion about the value of a report or proposal. Assessments must be replicated among independent professionals to support reliable decisions. Several evaluations of the peer review process in various disciplines have been performed (Peters and Ceci 1982). Many of those reports suggest that the system is unreliable because reviewers often fail to agree on the quality of a scientific article. Unreliability in this process undermines the internal quality controls of science, thus hampering progress. It also raises epistemological questions about the constitution of truth.
For instance, even if two reviewers judge a paper to be of high quality, both may be mistaken because they failed to spot a statistical error. In this sense reliability (agreement between reviewers) does not constitute validity (internal consistency or the absence of obvious errors of logic) (Wood, Roberts, and Howell 2004). However, on another level the negotiation of scientific claims within the scientific community is an integral part of determining what is true. Thus, in this sense reliability is a way of making or legitimating truth claims. These issues are made more complex by the role of editors in synthesizing disparate claims by reviewers and the question of whether reliability can be assessed by the metric of agreement between reviewers.
Another example of the issue of replicability in science is associated with the development of the Diagnostic and Statistical Manual of Mental Disorders (DSM) in psychiatry. Before this compendium of standardized descriptions of mental disorders was published, diagnoses of psychological illnesses lacked reliability. For example, if three physicians independently saw a patient with a psychological illness, it was unlikely that they would make the same diagnosis. Indeed, this remained the case through the publication of the original DSM in 1952 and DSM-II in 1968. It was only with the increasing detail and sophistication of DSM-III, published in 1980, that the psychiatric community began to achieve a significant measure of reliability in its diagnostic practices and psychiatry became more respected as a science.
This case suggests the connection between reliability and professionalization (the formation of a specialized academic discipline) because replicability was made possible only after a community of practitioners developed a shared conceptual language and a methodology that were sufficiently nuanced to communicate and establish likes as likes. Reliability as a way of establishing truth through replication thus is a product of both material reality and the way peers conceptualize the world and are able to replicate that conceptualization among themselves.
Functional Reliability in Engineering
Engineering or technological reliability is the probability that a product, process, or system will perform as intended or expected. Issues include the expected level of reliability, the cost-benefit trade-offs in improving reliability, and the consequences of failure. When these issues involve persons other than those inventing or tinkering with the relevant products, processes, or systems, with consequences for public safety, health, or welfare, ethical issues become prominent. Just as in science, reliability, in this case in the form of functional reliability, is a precondition for the integration of a particular technological device into the accepted or trusted edifice of the built environment.
Any technological product, process, or system is designed to perform one or more specified functions. In principle, the performance of the system can be defined mathematically and the demands placed on the system can be specified. Because uncertainties are associated with all aspects of systems in the real world, these descriptors should be defined in terms of uncertainties and reliability should be computed as the probability of intended performance. Because most systems have effects beyond their stated output (radiation, accidents, behavior modification, etc.), a comprehensive model must include all possible outcomes. Because complicated models all are based on extrapolations of the basic principles, the fundamental concepts are described in this entry.
The demands placed on a system include environmental and operational loads, which for simplicity will be designated here as a single demand, D. The capacity of the system to absorb those loads and perform its function is designated C, for capacity. The satisfactory operation of the system simply entails that the capacity be at least as large as the demand. This is expressed mathematically as S = CD 0 in which S represents satisfactory performance. In probability terms this becomes P(S) = P(CD) 0.
Each of these basic quantities can be described probabilistically by its probability function: FD(d) for demand and FC(c) for capacity. It is usually a safe assumption that the capacity (a function of the physical system) and the demand (a function of the operating environment) are statistically independent. In this case the reliability of the system is given by in which fC(x) is the probability density function of the capacity (the derivative of the capacity probability function, FC(x)) if the capacity is a continuous variable and otherwise is the probability mass function of the capacity (analogous to a histogram). In words the preceding equation indicates that one should assign a probability that the capacity is a particular value (fC(x)) and then multiply by the probability that the demand is no greater than that value of capacity (FD(x)). This process then is repeated for all possible values of the demand and the capacity, and the results are added (that is what the integration function does for continuous variables). The integrand of the equation above is shown in the Figure 1.
TIME DEPENDENCY. Most systems are not designed to be used just once but instead to perform over an intended period. In this case, the demand and the capacity become time-dependent variables and the probability of satisfactory performance is interpreted as being over an intended design lifetime. The formulation of the previous section then is interpreted as being at a single point in time, and the results are integrated over the lifetime.
Most technology displays a characteristic failure curve that is relatively steep at the beginning of the design lifetime, during which time initial defects are discovered. The failure rate then decreases to a steady-state value that exists over most of the design lifetime of the technology. As the technology nears the end of its useful lifetime, the failure rate again rises as parts begin to wear out.
When failure is due to relatively rare events such as environmental hazards, unusual parts wear, and abnormal use, simplified time-dependent models can be developed on the basis of the independent occurrence of these unusual events. These models usually are based on the Poisson process model, which is the simplest among the time-dependent processes that are referred to as stochastic processes. The Poisson model assumes that the occurrence of each event is independent of the past history of performance of the technology.
Systems reliability adds another level to this analysis. A system is a technology that is composed of multiple parts. Usually it is necessary that the parts work together properly for the system as a whole to function as desired. Systems theory builds on the theory described above to consider multiple capacities and demands, and many theories and models have been developed to analyze the risks of systems (Haimes 1998). Because systems analysis can be complicated, formalized approaches such as decision tree analysis (Clemen 1996) and event tree and fault tree analysis have been developed (Page 1989). Approximate analyses use the concepts of systems reaching a discrete number of undesirable states that are referred to as limit states. One then evaluates the probability of reaching those states by using approximate analyses such as the first-order, second-moment (FOSM) method, in which the limit state is approximated by a straight line and the full probability descriptors of the demands and capacities are approximated by the first and second moments of the probability function, which usually are the average and the standard deviation (Melchers 1999).
Software reliability can be used to illustrate some of the issues mentioned here. Newly engineered software is notoriously unreliable. After in-house testing and even after beta (user) testing in the field or market, "patches" regularly have to be introduced as new problems arise. Sometimes those problems arise because of a lack of correctness in the underlying code, and at other times because of a lack of robustness in the overall design. Software engineers also can fail to appreciate the ways users may choose to utilize a particular piece of software, and hackers and others may try to exploit weaknesses in ways that undermine reliability. As software illustrates, the pursuit of functional reliability in engineering and technology is a never-ending quest with ethical implications.
Ethics of Reliability
Despite its ethical importance in science and technology reliability has been subject to little extended ethical analysis. With regard to persons, in which case the virtue of reliability manifests itself as trustworthiness, there has been more discussion. However, the following comments on the ethics of reliability in general are only preliminary observations.
First, as has been suggested in this entry, technological reliability is what makes engineered artifice the basis for improved material well-being. It is for this reason that a few technical professional ethics codes include the promotion of reliability as an explicit obligation. For example, in the Code of Ethics (developed 1948) of the American Society for Quality (founded in 1946), the third fundamental principle commits a member to promote "the safety and reliability of products for public use." However, although in some instances unreliability in products may be attributed to a failure of intention, in other cases it is caused by evolutionary changes in nature (e.g., the evolution of antibiotic-resistant bacteria), economic change (as occurs when parts cease to be available for cars or other vehicles) or unintended consequences. Indeed, unintended consequences are one of the most common ways to conceptualize breakdowns in technological reliability as engineered devices bring about unexpected scenarios. This both raises questions about the degree to which reliability can be an ethical obligation and suggests the need for engineers to consider the wider ramifications of technology in their analyses of reliability and to build flexibility into their designs.
Another instance in which reliability has been adopted explicitly as an ethical concept related to technology occurred at a Poynter Journalism Values and Ethics in New Media Conference in 1997. That conference drafted an ethics code that included the following recommended "Online Reliability Statement":
This site strives to provide accurate, reliable information to its users. We pledge to:
Ensure information on our Web site has been edited to a standard equal to our print or broadcast standards.
Notify our online users if newsworthy materials are posted from outside our site and may not have been edited or reviewed to meet our standards for reliability.
Update all our databases for timeliness, accuracy and relevance.
Warn users when they are leaving our site that they may be entering a site that has not embraced the content reliability protocol.
The idea here is that professional standards of reliability in the print media need to be transported consciously into a new technological media framework. Similar statements about the need for commitment to reliability in information delivery related in one way or another to technology have been discussed with regard to both medicine and computers.
With regard to science replicability generally is thought of as a self-regulating process that serves both as a method for epistemological quality control and as a way to prevent scientific misconduct, including fabrication, falsification, and plagiarism. Thus, it is a mechanism for nurturing trust within the scientific community. The dominant perception that scientists deal with absolute certainties often undermines public trust in science when scientists openly communicate uncertainties in their research or when a scientific finding of high public concern is disputed and eventually overturned ("In Science We Trust" 2001).
The notion of reliability as replicability also manifests a certain hierarchy of values or axiology in the pursuit of knowledge. Alvin Weinberg (1971) has noted that physics serves as the ideal science (of which other sciences are more or less distorted images) because of the universalizability and replicability of its findings. It most closely approximates deeply entrenched Western beliefs about truth as timeless and noncontextual. However, this ingrained cultural deference to this ideal of science can lead to misunderstandings of science and unrealistic expectations about its contributions to complex political decisions.
Questions also might be raised about the issue of reliability in ethics itself. The human sciences, including ethical inquiry, proceed by means of dialectical and hermeneutical processes that are different from the models of the engineering construction of reliable artifacts or the scientific construction of reliable knowledge claims. In the popular imagination ethical and other value claims often are treated as matters of religious commitment, subjective preference, or legalistic requirements. However, a more nuanced appreciation of the process of ethical argumentation can point to possibilities for reliability.
Substantive agreement and reliability can be found, for instance, in some common documents, such as the Universal Declaration of Human Rights of 1948. Procedural reliability is manifested in the democratic considerations of ethics and other values that also are able to proceed toward common interest solutions through reasonable argumentation, tolerance, compromise, and openness of mind, procedures not dissimilar to those involved in the pursuit of an always provisional scientific truth.
Thus, the test for reliability in ethics may not be replicability, but it also may not be as distant from the actual workings of science as is maintained by many people. Indeed, when it comes to practical affairs, the desirable trait for both science and ethics may not be replicability so much as something more akin to the functional reliability of technology. That is, reliable science and ethics, much like reliable technologies, help human beings navigate toward common goods within complex situations marked by uncertainties and pluralities.
ROSS B. COROTIS CARL MITCHAM ADAM BRIGGLE
SEE ALSO Uncertainty.
BIBLIOGRAPHY
Clemen, Robert. (1996). Making Hard Decisions: An Introduction to Decision Analysis. Pacific Grove, CA: Duxbury Press.
Collins, Harry, and Trevor Pinch. (1998). The Golem: What You Should Know about Science, 2nd rev. edition. New York: Cambridge University Press. This controversial book, which argues through a series of seven case studies for the influence of social factors in the assessment of experiments, originally was published in 1993.
Haimes, Yacov. (1998). Risk Modeling, Assessment, and Management. New York: Wiley.
"In Science We Trust." (2001). Nature Medicine 7(8): 871.
Melchers, Robert. (1999). Structural Reliability Analysis and Prediction. New York: Wiley.
Page, Lavon. (1989). Probability for Engineering. New York: Computer Science Press.
Peters, Douglas P., and Stephen J. Ceci. (1982). "Peer Review Practices of Psychological Journals: The Fate of Published Articles, Submitted Again." Behavioral and Brain Sciences 5: 187–255. Finds that eight of twelve articles resubmitted after they had been published were rejected by the same journal.
Weinberg, Alvin. (1971). "The Axiology of Science." American Scientist 58(6): 612–617.
Wood, Michael; Martyn Roberts; and Barbara Howell. (2004). "The Reliability of Peer Reviews of Papers on Information Systems." Journal of Information Science 30(1): 2–11.
INTERNET RESOURCES
American Society for Quality. "Code of Ethics." Available from http://www.asq.org. The 1948 code of ethics can be found on their site at http://www.asq.org/join/about/ethics.html
The Poynter Institute. "Online Content Reliability Guidelines." 1997. Includes the "Online Reliability Statement." Available from http://legacy.poynter.org/research/me/nme/me_samprot.htm#relability.
