Executive Summary

This report, made pursuant to the Government Performance and Results Act (1993), covers activities of the National Science Foundation during Fiscal Year 2000. It is substantially more comprehensive than its predecessor, and records an unprecedented level of effort and achievement.

In conducting the FY 2000 assessment, NSF undertook the most rigorous and challenging performance review in its history. Several goals and indicators were added or revised, and evaluation criteria were made much more stringent.

Previously used ratings such as “partially successful” or “minimally effective” were eliminated; all outcomes were judged either “successful” or “not successful.” As a result, some of NSF’s performance scores were actually lower than in FY 1999, despite the fact that outside evaluators cited progress in those particular areas since 1999.

In addition to Foundation staff, about 400 external evaluators participated in the performance assessment, and generated 64 reports covering 78 of the NSF’s 200 programs. NSF engaged an independent outside examiner, PricewaterhouseCoopers LLP, to verify data compilations.

The Foundation aimed extremely high. For example, NSF set itself the goal of processing 70 percent of proposals within six months of submission. That would have been unmatched in the Foundation’s history, and would far exceed the norm at comparable granting agencies. In fact, during FY 2000 well over half of all NSF proposals were fully processed within six months and 71 percent were fully processed within seven months. By most measures, that would be considered extraordinary; yet it was not successful by NSF’s high standards.

Similarly, NSF attempted to develop the technological capability that would permit the electronic review and processing of tens of thousands of competitive proposals each year --making it possible to do so without generating any paperwork within the Foundation. No other research and education funding organization in government has attempted such a feat. NSF did encounter significant technological challenges in trying to realize this goal, but robust progress was made and the agency will initiate pilot projects to demonstrate its electronic review process capability during FY 2001.

Other goals in which NSF was rated “not successful” simply may not be achievable within a short time, such as “improved achievement in mathematics and science skills needed by all Americans”. Performance assessment activities in FY 2000 clearly demonstrated that a goal of reaching all Americans was unrealistic. Consequently, NSF will revise its performance indicators in future years to focus on related aspects more directly within the agency's responsibility and control.

However, even with new and revised goals and more exacting definitions of success, the Foundation met two-thirds of its 28 goals, which are divided into three broad areas: outcome goals, management goals, and investment process goals.

Outcomes

Outcome goals concern the practical, concrete results of NSF grants and programs, as opposed to the procedures and methods whereby the Foundation carries out its work.

NSF achieved 75 percent (6 out of 8) of its goals in the Outcomes category, notably including:

production of “discoveries at and across the frontiers of science and engineering”;
rapid and widespread connections between those discoveries and society as a whole;
ensuring that more than 80 percent of schools participating in education-improvement projects called “systemic initiatives” make substantial progress;
providing intensive professional development programs for at least 65,000 teachers in grades K-12;
prompt compilation and electronic dissemination of essential national data sets; and
development of ways to determine and assure the quality of survey materials.

Two ambitious goals were rated as “not fully successful” (that is, not successful) by reviewers. One was the objective of ensuring that NSF grants contributed to improved scores in “mathematics and science skills needed by all Americans,” as judged by independent external evaluators.

The other was the attempt to produce a “diverse, globally oriented workforce of scientists and engineers.” Both outcomes are extremely difficult to achieve, and neither is completely within the Foundation’s responsibility and control. In NSF's pursuit of both goals, however, evaluators noted progress since FY 1999.

Management

Management performance goals concern the effectiveness and efficiency of the way NSF handles its workload. For FY 2000, the Foundation identified 6 specific management goals, and achieved 5 of them (83 percent).

Goals that were met included:

ensuring that at least 60 percent of full proposals are submitted electronically through the computer-based “FastLane” system;
increasing the total number of science and engineering hires from under-represented groups, as judged against the FY 1997 baseline (NSF achieved a 120 percent increase in female hires and a 27 percent increase in minority hires);
providing FastLane orientation for all NSF staff and insisting on practice in key modules for at least 80 percent of program and support staff;
completing work on all “Y2K” information-technology problems as planned, on schedule and within budget; and
ensuring that at least 85 percent of all project reports are submitted electronically through a new, computer-based Project Reporting System.

Only one management performance goal was not met. NSF had set itself the objective of having the technological capability to move competitive proposals submitted electronically through the entire review and award/decline process without generating any paperwork.

While this may have been extremely ambitious, significant progress was made. By the end of FY 2000, the only significant impediment to attaining a full electronic review capability was the development of a secure “electronic signature”. This issue will continue to be addressed in FY 2001 when the agency will pilot ten all-electronic review projects.

Investment Process

Investment process goals involve the specific procedures whereby NSF makes grants, funds and manages capital projects, and serves its customers in general.

For FY 2000, NSF identified 15 such goals. One was found to be inapplicable. Of the remaining 14, the Foundation clearly achieved 7, or 50 percent. Those included:

allocating at least 90 percent of funds to projects reviewed by external peer groups and selected through merit-based competition;
identifying possible reasons for any customer dissatisfaction with NSF merit review and complaint-management systems;
improving NSF’s overall American Customer Satisfaction Index;
devising systems that require Principal Investigators to integrate educational components into their research proposals, and verifying the outcome;
developing methods of requesting and tracking reviewer answers to NSF merit review criterion - “what are the broader impacts of the proposed activity”;
find ways to increase the number of women and underrepresented minorities in the pool of applicants for grants; and
keep costs of construction and upgrades on facilities within 110 percent of estimates.

NSF did not meet 7 of its Investment Process Goals, even though efforts during FY 2000 often produced remarkable, measurable progress toward achieving those goals.

For example, NSF attempted to ensure that 95 percent of program announcements and solicitations would be available at least three months prior to proposal deadlines or target dates. In fact, 89 percent of announcements and solicitations met that standard - up from 75 percent in the preceding year - and 97 percent of program announcements and solicitations were available within 5 days of the three-month goal.

Another important area involves the goal of making at least 30 percent of competitive research grants to new investigators. In determining its performance on this goal, NSF counted only awards made to new Principal Investigators. During FY 2000, 28 percent of awards were made to new Principal Investigators - up from 27 percent in FY 1999. However, more than 33 percent of FY 2000 awards were made to teams of Investigators where at least one Investigator was new - up from 31 percent in FY 1999. So although the goal was not achieved in its strictest interpretation, the results reported clearly demonstrate an increasing "openness in the system" that the agency is committed to maintain.

Finally, NSF has committed to ensure that external merit reviewers take both NSF criteria fully into account when evaluating proposals. The two generic criteria are, simply put, scientific importance to the individual field, and broader significance to science and society as a whole. Although evaluators noted considerable improvement in this area, they rated NSF’s overall performance as “not fully successful", that is, not successful.

That outcome was not unexpected since the new merit review criteria were implemented only in early FY 1998. In assessing the agency's FY 2000 performance, external evaluators examined proposals considered for funding in FY 1997, FY 1998, FY 1999 and in a few cases, FY 2000. Therefore, only about two thirds of proposals examined actually could have been measured against the new merit review criteria. However, evaluators did note that for those proposals subject to the new review criteria (i.e. those submitted for funding consideration in FY 1998, FY 1999 and FY 2000), NSF staff had been generally successful in employing both criteria in making funding decisions. They also noted that NSF needs to increase its outreach efforts to the research and education community to ensure that proposers and reviewers alike adequately address both criteria in proposals and in the proposal review process.

Realization of this goal is increasingly likely over the next few years as NSF effectively communicates to proposers and reviewers the importance of addressing both criteria in proposals and reviews, and as evaluators examine a full complement of proposals subject to these criteria.

In four other goal areas, NSF did not successfully realize the high standards it set.

In one, NSF had attempted to process 70 percent of proposals within six months of receipt. Although only 54 percent of proposals were processed within that time frame, 71 percent of proposals were processed within seven months. This represents an accomplishment, considering the increasing complexity and multidisciplinarity of proposals and NSF's continuing commitment to external merit review. Nonetheless, the Foundation will take steps to improve its performance in this area in FY 2001 and will again set its sights on processing 70 percent of proposals within six months.

Another goal was to identify the best practices with which NSF staff could explain the merit review process, answer questions, and handle complaints.

During FY 2000, NSF conducted several customer-service surveys in an attempt to understand and improve its performance in these areas. Three concerns identified were: the quality and fairness of proposal review; ease of submitting proposals via FastLane; and trouble finding the right person to contact when questions arise. Models of best practice are still being developed, and staff training is underway.

A third goal involved keeping facilities' construction and upgrades within annual schedules and assuring that total time for completion of each phase did not exceed 110 percent of estimates. This goal is ambitious, although it is one the agency strives to realize. Nonetheless, in many scientific construction projects, unforeseen (and perhaps unforeseeable) delays occur as a consequence of rapidly changing technology and as a result of the fact that many such projects are unique. In those circumstances, it is very difficult to produce exact guidelines or timetables. Even so, of eleven construction and upgrade projects supported by NSF, seven (or 64 percent) of them met the goal.

Finally, NSF had set a goal of holding operating time lost due to unscheduled downtime at NSF-funded facilities to less than 10 percent of total scheduled operating time. Again, that target proved difficult to achieve because of the special problems that crop up in such projects. Nonetheless, of 26 reporting facilities, 22 (85 percent) met the downtime goal, and only four did not.

Based on what was learned from the FY 2000 performance assessment, NSF will pay particular attention to certain areas in FY 2001; these include improving customer service by increasing the percentage of proposals processed within six months and focusing on the effective implementation of the merit review criteria.

Using the experience gained in measuring and assessing performance in FY 2000, the agency will revise its FY 2002 performance indicators to focus more closely on achievements for which it has responsibility and control. The Foundation will continue to set the highest standards for itself and to employ the most stringent criteria for review of its accomplishments.