NSF FY2000 Accountability Report - Performance Results

Agency Profile | Performance Results | Management Integrity | Discussion and Analysis of the Financial Statements

Performance Results

Compared with FY 1999, in FY 2000 NSF was much more rigorous in evaluating goal achievement. Options for grading were limited to either successful or not successful, and full justifications were required for successful grades to be counted for those goals that used qualitative measures. For the Outcome Goals, PricewaterhouseCoopers LLP verified and validated the goal achievement data tables. While NSF was successful in achieving 64% of its goals in FY 2000 as compared with achieving 78% for FY 1999, the results of the second year are very similar to the first. Positive trends were evident in some of the goals, indicating movement in the desirable direction. The areas identified as needing improvement continue to be: (1) use of both merit review criteria by reviewers and applicants; and (2) the customer service goals such as decreasing time to decision on proposals. Both these areas will be focal points in FY 2001.

**FY 2000 Performance Results**
	Number of Goals Achieved

Outcome Goals	6 out of 8 (75%)

Management Goals	5 out of 6 (83%)

Investment Process Goals	7 out of 14 (50%); one goal did not apply

Total	18 out of 28 (64%)

Results for NSF's Outcome Goals
Six of the eight Outcome Goals were achieved in FY 2000. In FY 1999, all Outcome Goals were achieved. Overall, results are similar to those obtained in FY 1999, with trends beginning to appear in this second year of assessment. Reports by external evaluators indicate that NSF successfully achieved the first two Outcome Goals (Goal 1 and Goal 2), and achieved with limited success the second two Outcome Goals (Goal 3 and Goal 4a). FY 2000 evaluators identified the same areas as having limited success and in need of improvement as in FY 1999. In general, programs are showing improvement over FY 1999 performance in the area of increasing diversity through increased participation of underrepresented groups, but reports indicate that the numbers are still lower than expected. The evaluators commented that increasing participation of underrepresented groups is an area needing more attention for NSF. Other areas needing further improvement include: (1) balance of portfolio by taking more risk; and (2) use of the NSF’s merit review criteria by reviewers and applicants. Several reports noted that there are clear indications that use of the merit review criteria is evident in making decisions to fund or not fund applications. Common issues identified in some reports that may result in negative impact on program performance, in general, include workload and delays in processing proposals.

Results for NSF's Management Goals
Five of NSF's six Management Goals were achieved in FY 2000, compared with three out of five in the prior year. Areas identified as improving include orientation and training of NSF staff using FastLane, NSF's electronic system for proposal submission, proposal review, and project reporting; and increasing the use of the new electronic Project Reporting System for project reporting by awardees. The one Management Goal that was not achieved involves the technological capability to submit proposals electronically. The difficulty encountered in FY 2000 which prevented this goal from being achieved was related to the establishing of protocols for electronic signature. NSF piloted two models for electronic certification of proposals and is currently assessing which model will best serve the agency.

Results for NSF's Investment Process Goals
Seven of NSF's Investment Process Goals were achieved in FY 2000; seven were not achieved and, as in FY 1999, one of the facilities management goals did not apply because there were no construction projects completed during the year. In FY 1999, nine Investment Process Goals were achieved, four were not achieved and as previously mentioned, one did not apply. Areas identified as needing improvement include use of the new merit review criteria in some programs; identifying best practices and training for improving customer service; allowing three months time to prepare proposals; decreasing the time to decision; increasing the percentage of awards to new investigators; maintaining facility upgrades and construction on schedule; and keeping operating time lost due to unscheduled downtime to less than 10% of the total scheduled operating time.

The following chart lists NSF’s FY 2000 GPRA goals and results. For a more detailed explanation of these goals and results, see the section on “Performance Results and Related Issues.”

Annual Performance Goals for Outcome Results

Outcome	FY2000 Annual Performance Goals NSF is judged successful when	Aggregated Results

Outcome Goal 1 Discoveries at and across the frontier of science and engineering	Performance Goal 1 NSF awards lead to important discoveries; new knowledge and techniques, both expected and unexpected, within and across traditional disciplinary boundaries; and high-potential links across these boundaries, as judged by independent external experts.	Baseline: Experiments using FY 1997 and FY 1998 information indicated successful achievement. FY 1999: Goal achieved. Judged successful by external experts in all reports. FY 2000: Goal achieved. Reports by external experts indicate NSF is successful in achieving this goal in the aggregate.

Outcome Goal 2 Connections between discoveries and their use in service to society	Performance Goal 2 The results of NSF awards are rapidly and readily available and feed, as appropriate, into education, policy development, or use by other federal agencies or the private sector, as judged by independent external experts.	Baseline: Experiments using FY 1997 and FY 1998 information indicated successful achievement. FY 1999: Goal achieved. Judged successful in the aggregate by external experts who noted improvements could be made in some programs. FY 2000: Goal achieved. Judged successful in the aggregate by external experts who noted improvements could be made in some programs, as in FY 1999.

Outcome Goal 3 A diverse, globally-oriented workforce of scientists and engineers	Performance Goal 3 Participants in NSF activities experience world-class professional practices in research and education, using modern technologies and incorporating international points of reference; when academia, government, business, and industry recognize their quality; and when the science and engineering workforce shows increased participation of underrepresented groups, as judged by independent external experts.	Baseline: Experiments using FY 1997 and FY 1998 information indicated successful achievement. FY 1999: Goal achieved. Judged successful in most areas by external experts. FY 2000: Goal judged successful in the aggregate by external experts with respect to achieving a globally oriented workforce, and not fully successful with respect to achieving diversity or increased participation of underrepresented groups, therefore goal is successful in a limited context but not fully achieved. FY 2000 results indicate improvements over FY 1999 performance, but improvements are still needed in the same areas identified in FY 1999. For FY 2001, this goal has been incorporated into a broader goal that focuses on achieving NSF’s desired outcome of a diverse, internationally competitive and globally engaged workforce of scientists, engineers, and well-prepared citizens.

Outcome Goal 4 Improved achievement in mathematics and science skills needed by all Americans	Performance Goal 4a NSF awards lead to the development, adoption, adaptation, and implementation of effective models, products, and practices that address the needs of all students; well-trained teachers who implement standards-based approaches in their classrooms; and improved student performance in participating schools and districts, as judged by independent external experts.	Baseline: Preliminary pilot efforts did not provide sufficient information to yield a valid baseline. FY 1999: Goal achieved. Judged successful in the aggregate by external experts for programs to which goal applies. FY 2000: Goal judged successful in a limited context in the aggregate by external experts. Where programs did not have funds directed to these objectives, external evaluators were uncertain how to assess performance, resulting in an assessment of less than successful or no assessment. In FY 2001, performance measures/indicators for this goal will be better defined to eliminate confusion by evaluators.

	Performance Goal 4b Over 80% of schools participating in a systemic initiative program will: implement a standards-based curriculum in science and mathematics; further professional development of the instructional workforce; and improve student achievement on a selected battery of tests, after three years of NSF support.	FY 1999: Goal achieved. FY 2000: Goal achieved.

	Performance Goal 4c Through systemic initiatives and related teacher enhancement programs, NSF will provide intensive professional development experiences annually for at least 65,000 pre-college teachers.	FY 1999: Goal achieved. FY 2000: Goal achieved.

Outcome Goal 5 Timely and relevant information on the national and international science and engineering enterprise.	Performance Goal 5a Maintain FY 1999 gains in timeliness for an average of 486 days the time interval between reference period (the time to which the data refer) and reporting of data. FY 1995-96 Baseline 540 days FY 1999-2000 Goal 486 days Actual 461 days	FY 1999: Goal achieved. FY 2000: Goal achieved.

	Performance Goal 5b Establish a standard set of data quality measures for reporting of Science Resource Studies (SRS) products. Prepare reports on these measures for all SRS surveys and publish them in electronic formats to inform users of SRS data quality. New in FY 2000, replacing the FY 1999 goal on relevance. Baseline: None prior to goal setting.	New in FY 2000 FY 2000: Goal achieved.

Annual Performance Goals for Management

New and Emerging Technologies	Critical Factors for Success	Aggregated Results

Electronic proposal submission	Management Goal 1 NSF will receive at least 60% of full proposal submissions electronically through FastLane. FY 1998 Baseline 17% FY 1999 44% FY 2000 Goal 60% FY 2000 Result 81%	FY 1999: Goal achieved. FY 2000: Goal achieved.

Electronic proposal processing	Management Goal 2 By the end of FY 2000, NSF will have the technological capability to take competitive proposals submitted electronically through the entire proposal and award/declination process without generating paper within NSF.	New in FY 2000 FY 2000: Goal not achieved. In FY 2001, NSF will be testing use of an electronic signature for funding approval, the one remaining barrier to a completely electronic processing of awards.

NSF Staff	Critical Factors for Success	Aggregated Results

Diversity	Management Goal 3 In FY 2000, NSF will show an increase over 1997 in the total number of hires to science and engineering (S&E) positions from underrepresented groups. (Revised goal.) FY 1997 Baseline: Of 54 S&E hires, 22% were female and 19% were from underrepresented minority groups. FY 2000 Result: Of 113 S&E hires, 35 were female and 19 were from minority groups. Compared with FY 1997 baseline, this represents a 120% increase in female hires and a 27% increase in minority hires.	FY 1999: Goal achieved. FY 2000: Goal achieved.

Capability in use of information technology	Management Goal 4 By the end of FY 2000, all staff will receive an orientation to FastLane, and at least 80% of program and program support staff will receive practice in using its key modules. Orientation FY 1999 80% FY 2000 Goal 100% FY 2000 Result 100% Training FY 1999 43% FY 2000 Goal 80% FY 2000 Result 90%	FY 1999: Goal not achieved. FY 2000: Goal achieved.

Implementation of Management Reforms	Critical Factors for Success	Aggregated Results

Year 2000 Compliance	Management Goal 5 NSF will complete all activities needed to address the Year 2000 problem for its information systems according to plan, on schedule and within budget. (Revised goal for FY 2000.) FY 2000 Result: All activities needed to address the Year 2000 problem were completed according to plan, on schedule, and within budget.	FY 1999: Goal achieved. FY 2000: Goal achieved.

Project Reporting System	Management Goal 6 During FY 2000, at least 85% of all project reports will be submitted through the new electronic Project Reporting System. FY 1999 59% FY 2000 Goal 85% FY 2000 Result 92%	FY 1999: Goal achieved; target revised for FY 2000. FY 2000: Goal achieved.

Annual Performance Goals for NSF’s Investment Process

Performance Area: Proposal and Award Processes	FY 2000 Annual Investment Process Performance Goals	Aggregated Results

Use of Merit Review	Investment Goal 1 At least 90% of NSF funds will be allocated to projects reviewed by appropriate peers external to NSF and selected through a merit-based competitive process. FY 1998 (Baseline) 95% FY 1999 95% FY 2000 Goal 90% FY 2000 Result 95% During FY 2000, OMB redefined what constitutes a merit-reviewed project and established a new target goal of 70-90%. Revised FY 2000 Goal 80% (est.) FY 2000 Result 87%	FY 1999: Goal achieved. FY 2000: Goal achieved.

Implementation of Merit Review Criteria	Investment Goal 2 NSF performance in implementation of the new merit review criteria is successful when reviewers address the elements of both generic review criteria appropriate to the proposal at hand and when program officers take the information provided into account in their decisions on awards, as judged by external independent experts. Results: About one-third of evaluation reports rated NSF programs as successful in their use of the new merit review criteria. In most cases where NSF was rated not fully successful, reviewers and applicants were not fully addressing the second criterion regarding the broader impacts of the proposed activity.	FY 1999: Goal achieved. FY 2000: Goal not achieved. Full implementation of goal is a priority in FY 2001. A number of measures are being taken to ensure its achievement, e.g., different on-screen pages have been provided in FastLane so reviewers are guided to address each merit review criterion separately; performance data will be collected from the FastLane database; etc.

Customer Service: General	Investment Goal 3 Identify possible reasons for customer dissatisfaction with NSF’s merit review system and with NSF’s complaint system. Results: NSF commissioned surveys in order to ascertain possible reasons for customer dissatisfaction.	New in FY 2000. FY 2000: Goal achieved.

Customer Service: General	Investment Goal 4 Identify best practices and training necessary for NSF staff to conduct merit review and answer questions about the review criteria and process; identify best practices and training necessary for NSF staff to answer questions from the community and to deal with complaints in a forthright manner. Results: Goal underway but not completed in FY 2000; plans to finalize implementation in FY 2001.	New in FY 2000. FY 2000: Goal not achieved. In FY 2001, staff will continue to develop models of best practices and staff training; NSF will pilot the best models at division level and provide specific customer service training to NSF staff.

Customer service: General	Investment Goal 5 Improve NSF's overall American Customer Satisfaction Index (ACSI) compared to the FY 1999 index of 57 (on a scale of 0 to 100). FY 1999 57 FY 2000 Goal > 57 FY 2000 Result 58	New in FY 2000. FY 2000: Goal achieved. Ongoing commitment to improve results; see previous Goals 3&4

Customer service: Time to prepare proposals	Investment Goal 6 95% of program announcements and solicitations will be available at least three months prior to proposal deadlines or target dates. FY 1998 Baseline 66% FY 1999 75% FY 2000 Goal 95% FY 2000 Result 89% Although this goal was not achieved, there is notable improvement from prior year. In FY 2000, 89% of program announcement/solicitations achieved goal; approximately 8% missed the 90-day time limit by fewer than 5 days.	FY 1999: Goal not achieved. FY 2000: Goal not achieved. In FY 2001, staff will limit number of special competitions requiring individual announcements; plan further in advance; initiate clearance process at least 6 months prior to anticipated deadlines; clearance procedures will be reviewed.

Customer service: Time to decision	Investment Goal 7 Maintain the FY 1999 goal to process 70% of proposals within six months of receipt, improving upon the FY 1998 baseline. FY 1998 Baseline 59% FY 1999 58% FY 2000 Goal 70% FY 2000 Result 54% In FY 2000, 54% of proposals were processed within 6 months of receipt and an additional 35% were processed between 6 to 9 months of receipt.	FY 1999: Goal not achieved. FY 2000: Goal not achieved. In FY 2001, staff will make more effective use of electronic mechanisms in conducting reviews; more closely track processing; eliminate overloads/bottlenecks.

Maintaining Openness in the System	Investment Goal 8 The percentage of competitive research grants going to new investigators will be at least 30%. FY 1998 Baseline 27% FY 1999 27% FY 2000 Goal 30% FY 2000 Result 28%	FY 1999: Goal not achieved. FY 2000: Goal not achieved. In FY 2001, NSF staff will pursue outreach efforts to promote awareness of NSF research opportunities; undertake analysis of trends (e.g., whether pool of new investigators is getting smaller, etc.) to determine whether goal needs to be modified.

Performance Area: Integration of Research and Education	FY 2000 Annual Investment Process Performance Goals	Aggregated Results

In Proposals	Investment Goal 9 NSF will develop a plan and system to request that Principal Investigators (PIs) address the integration of research and education in their proposals, and develop and implement a system to verify that PIs have done so. (Revised goal.) No baseline. Result: In FY 2000, NSF implemented an electronic program announcement template clearance process (PAT) that is used by NSF staff to generate announcements and solicitations. Use of the PAT ensures that the integration of research and education is emphasized in all announcements and solicitations for PIs to address in their submissions.	New in FY 2000. FY 2000: Goal achieved.

In Reviews	Investment Goal 10 NSF will develop and implement a system/mechanism to request and track reviewer comments tied to merit review criterion #2, “What are the broader impacts of the proposed activity?” (Revised goal; no baseline.) Result: In FY 2000, screens in FastLane were redesigned so that reviewers can address each merit-review criterion separately. The performance data will be collected from the FastLane database. This will be fully implemented in FY 2001.	New in FY 2000. FY 2000: Goal achieved.

Performance Area:
Diversity

FY 2000 Annual Investment Process Performance Goals

Aggregated Results

NSF Applicants

Investment Goal 11
NSF will identify mechanisms to increase the number of women and underrepresented minorities in the proposal applicant pool, and will identify mechanisms to retain that pool.
(Revised goal; no baseline.)

Result: NSF identified and put into place mechanisms to increase the diversity of NSF applicants.

New in FY 2000.
FY 2000: Goal achieved.

Performance Area: Facilities Oversight	FY 2000 Annual Investment Process Performance Goals	Aggregated Results

Construction and Upgrade	Investment Goal 12 Maintain 1999 goal to keep construction and upgrades within annual expenditure plan, not to exceed 110% of estimates. FY 1999 Result: Majority of facilities were within 110% of annual spending estimates. FY 2000 Result: Of the 11 construction and upgrade projects supported by NSF, all were within annual expenditure plans; most were under budget.	FY 1999: Goal achieved. FY 2000: Goal achieved.

	Investment Goal 13 Maintain 1999 goal to keep construction and upgrades within annual schedule, total time required for major components of the project not to exceed 110% of estimates. FY 1999 Result: Majority of facilities on schedule. FY 2000 Result: Of the 11 construction/upgrade projects supported by NSF, seven were within the annual schedule goal.	FY 1999: Goal achieved. FY 2000: Goal not achieved. NSF program managers will work more closely with project managers to ensure compliance in FY 2001.

	Investment Goal 14 For all construction and upgrade projects initiated after 1996, keep total cost within 110% of estimates made at the initiation of construction.	FY 1999 and FY 2000: There were no completed projects, therefore, this goal did not apply.

Operations	Investment Goal 15 Maintain 1999 goal to keep operating time lost due to unscheduled downtime to less than 10% of the total scheduled operating time. FY 1999 Result: Reporting database under development. FY 2000 Result: Of the 26 reporting facilities, 22 met the goal of keeping unscheduled downtime to below 10% of the total scheduled operating time.	FY 1999: Inconclusive. FY 2000: Goal not achieved. NSF program managers will work more closely with project managers to ensure compliance in FY 2001.

Agency Profile | Performance Results | Management Integrity | Discussion and Analysis of the Financial Statements