CORE QUESTIONS and REPORT TEMPLATE

for

FY 2003 NSF COMMITTEE OF VISITOR (COV) REVIEWS

Guidance to NSF Staff: This document includes the FY 2003 set of Core Questions and the COV Report Template for use by NSF staff when preparing and conducting COVs during FY 2003. Specific guidance for NSF staff describing the COV review process is described in Subchapter 300-Committee of Visitors Reviews (NSF Manual 1, Section VIII) that can be obtained at http://www.inside.nsf.gov/od/gpra/.

NSF relies on the judgment of external experts to maintain high standards of program management, to provide advice for continuous improvement of NSF performance, and to ensure openness to the research and education community served by the Foundation. Committee of Visitor (COV) reviews provide NSF with external expert judgments in two areas: (1) assessments of the quality and integrity of program operations and program-level technical and managerial matters pertaining to proposal decisions; and (2) comments on how the outputs and outcomes generated by awardees have contributed to the attainment of NSF’s mission and strategic outcome goals.

Many of the Core Questions developed for FY 2003 are derived, in part, from the OMB-approved FY 2003 performance goals and apply to the portfolio of activities represented in the program(s) under review. The program(s) under review may include several subactivities as well as NSF-wide activities. The directorate or division may instruct the COV to provide answers addressing a cluster or group of programs – a portfolio of activities integrated as a whole – or to provide answers specific to the subactivities of the program, with the latter requiring more time but providing more detailed information.

The Division or Directorate may choose to add questions relevant to the activities under review. NSF staff should work with the COV members in advance of the meeting to provide them with the report template, organized background materials, and to identify questions/goals that apply to the program(s) under review.

Guidance to the COV: The COV report should provide a balanced assessment of NSF’s performance in two primary areas: (A) the integrity and efficiency of the processes related to proposal review; and (B) the quality of the results of NSF’s investments in the form of outputs and outcomes that appear over time. The COV also explores the relationships between award decisions and program/NSF-wide goals in order to determine the likelihood that the portfolio will lead to the desired results in the future. Discussions leading to answers for Part A of the Core Questions will require study of confidential material such as declined proposals and reviewer comments. COV reports should not contain confidential material or specific information about declined proposals. Discussions leading to answers for Part B of the Core Questions will involve study of non-confidential material such as results of NSF-funded projects. It is important to recognize that the reports generated by COVs are used in assessing agency progress in order to meet government-wide performance reporting requirements, and are made available to the public. Since material from COV reports is used in NSF performance reports, the COV report may be subject to an audit.

We encourage COV members to provide comments to NSF on how to improve in all areas, as well as suggestions for the COV process, format, and questions.

FY 2003 REPORT TEMPLATE FOR

NSF COMMITTEES OF VISITORS (COVs)

Date of COV:	September 10-12, 2003
Program/Cluster:	Geosciences Education and Diversity Programs
Division:	ATM, EAR, OCE and OAD
Directorate:	GEO
Number of actions reviewed by COV : Awards: Declinations: Other:
Total number of actions within Program/Cluster/Division during period being reviewed by COV : Awards: Declinations: Other:
Manner in which reviewed actions were selected:

PART A. INTEGRITY AND EFFICIENCY OF THE PROGRAM’S PROCESSES AND MANAGEMENT

Briefly discuss and provide comments for each relevant aspect of the program's review process and management. Comments should be based on a review of proposal actions (awards, declinations, and withdrawals) that were completed within the past three fiscal years. Provide comments for each program being reviewed and for those questions that are relevant to the program under review. Quantitative information may be required for some questions. Constructive comments noting areas in need of improvement are encouraged. Please do not take time to answer questions if they do not apply to the program.

A.1 Questions about the quality and effectiveness of the program’s use of merit review procedures. Provide comments in the space below the question. Discuss areas of concern in the space provided.

QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES

YES, NO,

DATA NOT AVAILABLE, or NOT APPLICABLE

Is the review mechanism appropriate? (panels, ad hoc reviews, site visits)

Comments:

Is the review process efficient and effective?

Comments:

Are reviews consistent with priorities and criteria stated in the program’s solicitations, announcements, and guidelines?

Comments:

Do the individual reviews (either mail or panel) provide sufficient information for the principal investigator(s) to understand the basis for the reviewer’s recommendation?

Comments:

Do the panel summaries provide sufficient information for the principal investigator(s) to understand the basis for the panel recommendation?

Comments:

Is the documentation for recommendations complete, and does the program officer provide sufficient information and justification for her/his recommendation?

Comments:

Is the time to decision appropriate?

Comments:

Discuss issues identified by the COV concerning the quality and effectiveness of the program’s use of merit review procedures:

A.2 Questions concerning the implementation of the NSF Merit Review Criteria (intellectual merit and broader impacts) by reviewers and program officers. Provide comments in the space below the question. Discuss issues or concerns in the space provided.

IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA

YES, NO,

DATA NOT AVAILABLE, or NOT APPLICABLE

Have the individual reviews (either mail or panel) addressed whether the proposal contributes to both merit review criteria?

Comments:

Have the panel summary reviews addressed whether the proposal contributes to both merit review criteria?

Comments:

Have the review analyses (Form 7s) addressed whether the proposal contributes to both merit review criteria?

Comments:

Discuss any issues or concerns the COV has identified with respect to NSF’s merit review system.

A.3 Questions concerning the selection of reviewers. Provide comments in the space below the question. Discuss areas of concern in the space provided.

Selection of Reviewers

YES , NO,

DATA NOT AVAILABLE,

or NOT APPLICABLE

Did the program make use of an adequate number of reviewers for a balanced review?

Comments:

Did the program make use of reviewers having appropriate expertise and/or qualifications?

Comments:

Did the program make appropriate use of reviewers to reflect balance among characteristics such as geography, type of institution, and underrepresented groups?

Comments:

Did the program recognize and resolve conflicts of interest when appropriate?

Comments:

Discuss any concerns identified that are relevant to selection of reviewers.

A.4 Questions concerning the resulting portfolio of awards under review. Provide comments in the space below the question. Discuss areas of concern in the space provided.

RESULTING PORTFOLIO OF AWARDS

APPROPRIATE,

NOT APPROPRIATE,

OR DATA NOT AVAILABLE

Overall quality of the research and/or education projects supported by the program.

Comments:

Are awards appropriate in size and duration for the scope of the projects?

Comments:

Does the program portfolio have an appropriate balance of:

High Risk Proposals?

Comments:

Does the program portfolio have an appropriate balance of:

Multidisciplinary Proposals?

Comments:

Does the program portfolio have an appropriate balance of:

Innovative Proposals?

Comments:

Does the program portfolio have an appropriate balance of:

Funding for centers, groups and awards to individuals?

Comments:

Does the program portfolio have an appropriate balance of:

Awards to new investigators?

Comments:

Does the program portfolio have an appropriate balance of:

Geographical distribution of Principal Investigators?

Comments:

Does the program portfolio have an appropriate balance of:

Institutional types?

Comments:

Does the program portfolio have an appropriate balance of:

Projects that integrate research and education?

Comments:

Does the program portfolio have an appropriate balance:

Across disciplines and subdisciplines of the activity and of emerging opportunities?

Comments:

Does the program portfolio have appropriate participation of underrepresented groups?

Comments:

Is the program relevant to national priorities, agency mission, relevant fields and other customer needs? Include citations of relevant external reports.

Comments:

Discuss any concerns identified that are relevant to the quality of the projects or the balance of the portfolio.

A.5 Management of the program under review. Please comment on:

Management of the program.

Comments:

Responsiveness of the program to emerging research and education trends.

Comments:

Program planning and prioritization process (internal and external) that guided the development of the portfolio under review.

Comments:

Discuss any concerns identified that are relevant to the management of the program.

PART B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

NSF investments produce results that appear over time. The answers to questions for this section are to be based on the COV’s study of award results, which are direct and indirect accomplishments of projects supported by the program. These projects may be currently active or closed out during the previous three fiscal years. The COV review may also include consideration of significant impacts and advances that have developed since the previous COV review and are demonstrably linked to NSF investments, regardless of when the investments were made. Incremental progress made on results reported in prior fiscal years may also be considered.

The following questions are developed using the NSF outcome goals in the FY 2003 Performance Plan. The COV should look carefully at and comment on (1) noteworthy achievements of the year based on NSF awards; (2) the ways in which funded projects have collectively affected progress toward NSF’s mission and strategic outcomes; and (3) expectations for future performance based on the current set of awards. NSF asks the COV to provide comments on the degree to which past investments in research and education have contributed to NSF’s progress towards its annual strategic outcome goals and to its mission:

To promote the progress of science.

To advance national health, prosperity, and welfare.

To secure the national defense.

And for other purposes.

B. Please provide comments on the activity as it relates to NSF’s Strategic Outcome Goals. Provide examples of outcomes (nuggets) as appropriate. Examples should reference the NSF award number, the Principal Investigator(s) names, and their institutions.

B.1 NSF OUTCOME GOAL for PEOPLE: Developing “a diverse, internationally competitive and globally engaged workforce of scientists, engineers, and well-prepared citizens.”

Comments:

B.2 NSF OUTCOME GOAL for IDEAS: Enabling “discovery across the frontier of science and engineering, connected to learning, innovation, and service to society.”

Comments:

B.3 OUTCOME GOAL for TOOLS: Providing “broadly accessible, state-of-the-art and shared research and education tools.”

Comments:

PART C. OTHER TOPICS

C.1 Please comment on any program areas in need of improvement or gaps (if any) within program areas.

C.2 Please provide comments as appropriate on the program’s performance in meeting program-specific goals and objectives that are not covered by the above questions.

C.3 Please identify agency-wide issues that should be addressed by NSF to help improve the program's performance.

C.4 Please provide comments on any other issues the COV feels are relevant.

C.5 NSF would appreciate your comments on how to improve the COV review process, format and report template.

SIGNATURE BLOCK:

__________________

For the Geosciences Education and Diversity Committee of Visitors

Cheryl Peach

Chair

To be provided by NSF staff.

To be provided by NSF staff.

FINAL

1 –

NSF FY 2003 CORE QUESTIONS FOR COVs

DRAFT O/D STAFF MEMORANDUM

DSCHINDE

2003-08-28

Melissa J. Lane

National Science Foundation