National Science Foundation: Second Merit Review Criterion
NATIONAL SCIENCE FOUNDATION: SECOND MERIT REVIEW CRITERION
In the early twenty-first century, science finds itself caught in a dilemma that is arguably of its own making: Its very success in terms of understanding and controlling nature means that it has given birth to powers that transcend the traditional boundaries between science and society. Rather than being viewed as essentially neutral in terms of values, society increasingly views scientific knowledge as leading to various types of winners and losers. The review criteria for National Science Foundation proposals offer an instructive case study of this increasingly prominent dynamic.
Established in 1950, the National Science Foundation (NSF) is the only federal agency dedicated to the support of education and basic research across all scientific and engineering disciplines, except for the biomedical sciences (which are handled by the National Institutes of Health). Although no authoritative definition exits, it is generally agreed that basic scientific research is oriented chiefly toward the discovery and creation of new knowledge, without regard for its eventual employment.
In 1993 Congress passed the Government Performance Results Act (GPRA). The purpose of GPRA was to increase the focus of federal agencies on improving and measuring "results," which would in turn provide congressional decision makers with the data they require to assess the effectiveness and efficiency of federally funded programs. In effect, GPRA sent the message that federal funding is contingent on attaining and demonstrating results. Partly in response to such demands for demonstrable results, in 1995 the NSF adopted a new strategic plan: NSF in a Changing World (NSF 95-24). NSF's new strategic plan included among the long-term goals of the foundation the promotion of the discovery of new "knowledge in service to society."
In 1996 the National Science Board (NSB) established the NSB-NSF Task Force on Merit Review to examine and evaluate NSF's generic merit review criteria, which had been in effect since 1981, in light of the new strategic plan. In its "Discussion Report" (NSB/MR-96-15) the task force recommended replacing previous review criteria with two simple questions: (1) What is the intellectual merit of the proposed activity? (2) What are the broader impacts of the proposed activity? The simplification was proposed to help connect NSF investments to societal value while preserving an ability to select proposals on the basis of scientific excellence. Such criteria were more clearly related to the goals and strategies of NSF in a Changing World. NSF published the recommendations of the task force on the web, through press releases, and through direct contact with universities and professional associations, and received around 300 responses from the scientific and engineering community.
In light of these responses, in 1997 the task force published its "Final Recommendations" (NSB/MR-97-05). The responses raised several concerns about the new criteria, including what the task force termed the issue of "weighting" the criteria: Criterion 1 was perceived by respondents as more important than 2, or criterion 2 was perceived as irrelevant, ambiguous, or poorly worded. Moreover, respondents expressed concern that for much of basic research it is impossible to make meaningful statements about the potential usefulness of the research. Ultimately, however, the task force recommended that the new criteria be adopted. Later in 1997, NSF issued Important Notice No. 121, which announced NSB approval of the new merit review criteria, effective October 1.
The NAPA Report
In 1998, and again in 1999, Congress directed NSF to contract with the National Academy of Public Administration (NAPA) to review the effects of the changes in NSF's merit review criteria. NAPA is an independent, nonpartisan organization chartered by Congress to help federal, state, and local governments improve their effectiveness, efficiency, and accountability. In 2000 NSF commissioned the NAPA study.
The NAPA study reviewed relevant legislation, reports by external review committees, interviews with NSF personnel, and interviews with members of the scientific and engineering community. In addition, the NAPA study analyzed sample projects funded under both the old and the new criteria, as well as the intentions of those reviewing proposals using the new criteria. Published in February 2001, the NAPA report provides a history of the development of NSF's new merit review criteria, compares the 1997 criteria to the 1981 criteria, and details many of the challenges faced by the merit review process during the period from 1997 to 2000. The NAPA report offers several recommendations to help NSF improve the merit review process, among which is a recommendation to address the "philosophical issues" raised by the new criteria, in particular criterion 2.
The latter recommendation was based in part on its observation of the diverse interpretations of and reactions to the new merit review criteria among members of the scientific and engineering community. Although the NAPA report fails to delineate explicitly what it considered to be the philosophical issues, it nevertheless provides an excellent source from which those issues can be gleaned. Such issues include:
- whether criterion 2 is inconsistent with criterion 1
- whether criterion 1 is more important than criterion 2
- whether criterion 2 is in need of conceptual clarification
- whether interpretations of criterion 2 are discipline-dependent
- whether reactions to criterion 2 rely on one's conception of scientific inquiry.
These issues are, of course, interrelated: A physicist committed to a strict division between basic and applied scientific research might interpret the criteria as inconsistent, whereas a geologist whose research in plate tectonics might one day lead to predictive capabilities might not; said geologist might nonetheless view criterion 1 as significantly more important than criterion 2.
Moreover, consideration of such issues also raises philosophical issues in the realm of science policy. Is NSF moving away from its emphasis on basic research? If so, is NSF offering a new conception of scientific inquiry? If so, what is this new conception? Is this new conception coherent? If not, should NSF change its merit review criteria? Should criterion 2 be abandoned? If so, must NSF's strategic plan be reconceptualized? What impact would such a reconceptualization have on NSF's compliance with GPRA? Should NSF still receive federal funding? If so, how much and for what?
In attempting to incorporate intellectual merit and broader societal impacts more fully, NSF's 1997 merit review criteria raise a host of philosophical issues. Demands for federal agencies to show results in order to receive funding show no signs of vanishing. It remains to be seen how such issues will be addressed.
J. BRITT HOLBROOK
SEE ALSO National Science Foundation.
National Academy of Public Administration (NAPA). "A Study of the National Science Foundation's Criteria for Project Selection." A report by a panel of the National Academy of Public Administration for the National Science Foundation, issued February 2001. Available from http://www.napawash.org/publications.html.