Contract No.:                    DOB-01-33044

MPR Reference No.:         8814-006

Mathematica Logo 

 

 

 


National Science Foundation

Report on Efficiency of Grant Size and Duration

 

Principal Investigator

FY 2001 Grant Award Survey and Institutional Survey

 

 

 

July 2002

 

 

Janice Ballou

Matthew Mishkind

Geraldine Mooney

Welmoet van Kammen

 

 

 

Submitted to:

 

National Science Foundation

Chief Budget and Systems

Operations and Budget Branch

4201 Wilson Boulevard, Room 407N

Arlington, VA  22230

 

Project Officers:

Robert Abel

Norman Bradburn

Bernard McDonald

 

 

Submitted by:

 

Mathematica Policy Research, Inc.

P.O. Box 2393

Princeton, NJ  08543-2393

(609) 799-3535

 

 

Project Director:

Janice Ballou

 


ACKNOWLEDGEMENTS

Mathematica Policy Research, Inc. (MPR) would like to thank the staff at the National Science Foundation (NSF) who provided guidance and resources for this survey.  Specifically, Robert Abel (Contracting Officer’s Technical Representative [COTR]), Norman Bradburn, Bernard McDonald, and Vernon Ross gave their assistance during this project.  At NSF, Kimberly Deutsch and Elizabeth Velo also contributed to the project.  MPR would also like to thank all the Principal Investigator 2001 Grant Award Survey and the Institutional Survey participants.  Without their cooperation and thoughtful responses this report would not be possible.

At MPR Frank Potter was the sampling statistician, and John Hall provided statistical assistance.  Melissa Wood, David Frank, Irene Crawely, and Susan Golden assisted with the data collection and quality assurance.  Programming for the Web survey and data analysis was supervised by Anne Hower; Ron Palanca, Eric French, Scott Reid, Shayne Walcott, Neil deLeon, and Linda Gentzik assisted with the programming and data analysis.  Roy Grisham edited the report and Gloria Gustus and Rosita Turkel prepared the final report.


CONTENTS

 

 

ACKNOWLEDGEMENTS

 

EXECUTIVE SUMMARY

 

I.              INTRODUCTION AND BACKGROUND

 

           A.    INTRODUCTION

 

           B.    BACKGROUND INFORMATION

 

           C.    OVERVIEW OF METHODOLOGY

 

D.      ORGANIZATION OF THIS REPORT

 

           E.    OTHER CONTEXTUAL CONSIDERATIONS

 

II.       PROFILE OF 2001 NSF FUNDED GRANT PROPOSALS

 

           A.  PRINCIPAL INVESTIGATOR CHARACTERISTICS

 

           B.  GRANT CHARACTERISTICS

 

           C.  AMOUNT AND DURATION OF FUNDING

 

           D.  CHANGES IN FUNDING

 

           E.  OTHER FUNDING

 

           F.  PROFILE OF INSTITUTION GRANT MANAGEMENT

 

III.      THE PROPOSAL PROCESS

 

           A.  LEVEL OF PRINCIPAL INVESTIGATOR PROPOSAL EFFORT

 

           B.  OTHER PROPOSAL SUBMISSIONS

 

           C.  LEVEL OF INSTITUTIONAL PROPOSAL EFFORT

 


IV.      MEASURES OF AWARD EFFICIENCY AND EFFECTIVENESS

 

           A.  DIFFERENT MEASURES OF AWARD EFFICIENCY AND

                  EFFECTIVENESS

 

                  1. Award Efficiency and Effectiveness: Deviations from Requested

                  2. Award Efficiency and Effectiveness: Percent of Research Being

                      Funded

                  3. Award Efficiency and Effectiveness: Additional Requirements

                  4. Award Efficiency and Effectiveness: NFS’s Contribution

 

           B.    SUMMARY OF DIFFERENT METHODS TO ESTIMATE AWARD

                  EFFICIENCY AND EFFECTIVENESS

 

           C.   PRIORITY ACTIONS FOR AWARD EFFICIENCY AND

                 EFFECTIVENESS: PI PERSPECTIVE

 

           D.    PRIORITY ACTIONS FOR AWARD EFFICIENCY AND

                  EFFECTIVENESS:  FIELD OF RESEARCH

 

V.       POTENTIAL IMPACT OF INCREASED FUNDING OR DURATION

 

           A.    THE CONSEQUENCES OF REDUCTIONS IN FUNDING OR

                  DURATION

 

           B.    THE POTENTIAL BENEFITS OF ADDITIONAL NSF FUNDING OR

                  DURATION ON THE PI’S CURRENTLY ONGOING BODY OF

                  RESEARCH

 

           C.    THE POTENTIAL BENEFITS OF ADDITIONAL NSF FUNDING OR

                  DURATION ON THE RESEARCH IN GENERAL

 

           D.    THE POTENTIAL IMPACT OF ADDITIONAL NSF FUNDING OR

                  DURATION ON EDUCATIONAL ACTIVITIES

 

           E.    IMPACT OF CHANGES ON INSTITUTIONS

 

           F.    ADDITIONAL COMMENTS ABOUT THE NSF GRANT PROCESS

 

Appendix to the NSF Report on Efficiency of Grant Size and Duration

                 APPENDIX A:   RESEARCH METHODOLOGY

                 APPENDIX B:   ANNOTATED QUESTIONNAIRES

                                            1.  Principal Investigators

                                            2.  Institutions

 

 

                 APPENDIX C:   NON-SAMPLE INSTITUTION SURVEY INFORMATION

                 APPENDIX D:  VERBATIM RESPONSE CODING FRAME

 

                                            1.  Principal Investigators

                                            2.  Institutions

 

                 APPENDIX E:   PRINCIPAL INVESTIGATOR SURVEY CONTACT

                                            INFORMATION

 

                                            1.  Principal Investigators

                                            2.  Institutions

 

                 APPENDIX F:   INSTITUTIONAL SURVEY CONTACT INFORMATION

 

                 APPENDIX G:   STATISTICAL TABULATIONS


 

TABLES

 

                                                                                                                                    

II.1    PROFILE OF FY 2001 NSF GRANT PRINCIPAL INVESTIGATORS

 

II.2.    PROFILE OF FY 2001 NSF GRANT AND OTHER FUNDING

 

II.3    TYPE OF FY 2001 NSF GRANT BY DIRECTORATE

 

II.4    PROFILE OF INSTITUTION LEVEL OF GRANT EFFORT

 

III.1    PI LEVEL OF PROPOSAL EFFORT

 

III.2    TOTAL FUNDING PROFILE

 

III.3    INSTITUTIONAL LEVEL OF EFFORT FOR TYPICAL FY 2002 NSF

          GRANT ACTIVITIES

 

IV.1    PROFILE OF AWARD EFFICIENCY AND EFFECTIVENESS

           ATTRIBUTES

 

IV.2    AWARD EFFICIENCY AND EFFECTIVENESS FUNDING AMOUNT

           TO ACCOMPLISH A 100 PERCENT IN FIVE YEARS

 

IV.3    MOST IMPORTANT NSF AWARD REFORM FOR PI AREA OF

            RESEARCH

 

V.1    NEGATIVE IMPACT OF FUNDING AND/OR DURATION AWARD

           CHANGES

 

V.2    IMPACT OF CHANGE IN AWARD FUNDING AND/OR DURATION

 

V.3    POSITIVE IMPACT OF ADDITIONAL AWARD FUNDING AND/OR

          DURATION

 

V.4    IMPACT OF ADDITIONAL FUNDING AND/OR DURATION

 

V.5    ADDITIONAL NSF AWARD FUNDING AND/OR DURATION:

          POTENTIAL INCREASES IN RESEARCH AND EDUCATIONAL

          ACTIVITIES

 

V.6    ADDITIONAL NSF AWARD FUNDING AND/OR DURATION:

          INCREASED ABILITY FOR RESEARCH AND EDUCATIONAL

          ACTIVITIES

 

V.7    ADDITIONAL NSF AWARD FUNDING AND/OR DURATION:

          POTENTIAL FOR CHANGES IN FIELD OF RESEARCH

 

V.8    INSTITUTION CHANGES IF NSF INCREASED THE AVERAGE

                     DURATION PER GRANT

                                                                                              

V.9    INSTITUTION CHANGES IF NSF INCREASED THE AVERAGE

          DOLLAR AWARD PER GRANT

 

V.10  SUGGESTIONS FOR NSF CHANGES TO REDUCE INSTITUTION

          TIME AND RESOURCES

 

V.11  PRINCIPAL INVESTIGATORS: OTHER COMMENTS ON THE

          NSF GRANT PROCESS

 

V.12  INSTITUTIONS: OTHER COMMENTS ON THE NSF GRANT

          PROCESS

 


        FIGURES

 

                                                                                                                                    

II.1    CURRENT STUDENT SUPERVISION AVERAGE

          NUMBER OF STUDENTS AND POSTDOCTORALS

 

IV.1.   AWARD FUNDING OPTIONS ANNUAL PROJECTED

           AMOUNTS (Means)

 

IV.2   AWARD FUNDING OPTIONS ANNUAL PROJECTED

          AMOUNTS (Means)

 

IV.3   AWARD FUNDING OPTIONS ADDED TO FY 2001

          EXAMPLES OF TOTAL PER YEAR FUNDING

 

IV.4   CURRENT OTHER NSF FUNDING

 

IV.5   CURRENT NON-NSF FUNDING (Means)

 

 


EXECUTIVE SUMMARY

This Executive Summary provides an overview of the key results of two surveys.  Readers are encouraged to read the full report for more comprehensive information and additional details on the survey methodology.

Overview of Reasons for Conducting the Survey

 

The National Science Foundation (NSF) is responsible for advancing science and engineering in the United States.  About 50 percent of the federal nonmedical fundamental research at U.S.  colleges and universities is from NSF’s support of nearly 200,000 scientists, engineers, teachers, and students each year.  Thus, the research productivity and research training of the U.S. scientific and engineering community depend heavily on NSF funding.  However, a key question that needs to be answered is: What constitutes the right size NSF grant?  In addition, there is the related question: What needs to be done to improve NSF award efficiency and effectiveness?

Survey Design

 

The survey was designed to provide an opportunity for two key participants in the NSF grant process to answer these questions: (1) the Principal Investigators (PIs) who receive grants, and (2) the institutions that assist in preparing and administering NSF grants.  In February and March 2002, surveys were conducted with each of these two groups by Mathematica Policy Research,  Inc. (MPR).  The study design was to (1) conduct a census of principal investigators who had received NSF grants in FY 2001, and (2) include a sample of the institutions these recipients represented.  The results of this study describe the key issues NSF is considering from the perspective of these two key groups.  This information will help guide NSF as it makes strategic decisions about the right amount of its grants and their duration.

Overview of Principal Investigator Characteristics

In order to have a context for understanding the answers to the key questions described above, it is important to have some background information about the FY 2001 NSF grant recipients.  While the PIs have numerous diverse characteristics, the following will provide a general profile of who received NSF grants in FY 2001.  The professional age of these PIs—defined as the years since their last degree—is distributed as follows: 10 years ago or less (30%), 11 to 20 years (34%), and more than 20 years ago (36%).  There is variation in the number of graduate students they supervise—with 40 percent currently working with two or fewer graduate students, 25 percent with three to four, and 27 percent with more than four.  The PIs describe the focus of their NSF research project as follows: laboratory (44%), theoretical (37%), field (18%).  PIs currently have various sources to fund their research.  Nineteen percent have only this NSF grant, compared to 9 percent who have an additional NSF grant, 37 percent who have non-NSF grants, and 35 percent who have both an additional NSF grant and non-NSF support.

Level of Proposal Effort

 

One measure of the use of resources that could be improved is the proposal-preparation effort: What is the extent of the resources that PIs and institutions might save if NSF grant awards were more efficient and effective?  A PI who is on 100 percent soft money provided a description of the level of proposal effort needed to get sufficient support:

 

 

“For a soft money researcher needing to cover 12 months of his/her time and current budget standards pushing for 3 months (or less) coverage for senior personnel per proposal (seemingly, this is only communicated obliquely) and an average of three year research duration this results in obtaining four funded proposals per three years.  This would be a limited time sink if success was 100% (4 weeks/year of proposal writing at three weeks/proposal).  At 50% it is eight weeks/year writing proposals.  At 33% success it is 12 week/year, and so on.  It seems that a better balance can be found.  Consider that many programs have funding rates of 10-33 percent.”

 

Specifically, PIs’ estimate of the time it took for them and other people—for example, graduate assistants, budget administrators, and secretaries (not including time spent by institutional personnel)—to prepare their FY 2001 NSF grant submission was, on average, 157 hours, or about 19.5 days.  It should be noted this is the time for just one proposal that was successful.

When PIs lack sufficient funding for their research and educational activities, they need to prepare and submit multiple grants.  For example, 38 percent of the PIs reported that they divide their ongoing body of research and educational activities into several proposals and submit them to NSF; as described earlier, overall, about 8 of 10 PIs currently have funding in addition to the FY 2001 NSF grant identified for this survey.

Award Duration and Funding Efficiency and Effectiveness

 

A key objective of the PI survey was to find out:  What are the award durations and funding amounts that are the most efficient and effective in promoting NSF’s objectives?  The answer to this question is not straightforward.  NSF meets multiple objectives when it awards grants.  These multiple needs increase the challenge of establishing criteria in order to achieve award efficiency and effectiveness.  Adding to this already complex task is the nature of scientific inquiry as described by one PI:

 

“The very nature of research and investigation of hitherto unexplored ideas makes it difficult to know or state future requirements.  More funding generally means more personnel to help perform the intended research.  However, often new and unforeseen opportunities for new research arise as a consequence of new stimulus and ideas in performing the currently-funded research.  This is always difficult to assess ahead of time.  It is what makes research exciting!!”

 

 

Award Duration

 

The information from the PIs suggests a consensus on an increase in award duration.  Currently, the average grant duration for all PIs is three years. When the additional years the PIs suggest they would like are added to each current award, the average is five years.  Five year awards will give PIs the most effective period of time for their research and educational activities.  This period has such benefits as continuity of employment for students, particularly graduate student staff, opportunities to explore other areas of inquiry that may develop as they conduct their research, higher quality of research, and less time spent pursuing additional funding.

 

Award Amount

 

There are multiple ways to consider the appropriate amounts needed to provide efficient and effective grant awards.  The information provided by the PIs can be used by NSF to consider funding needs in a variety of ways.  The following are four examples of possible ways to estimate award efficiency and effectiveness from the PIs’ perspective that have a range from an average of $40,000 to $230,000.

 

Option 1:  Award Effectiveness and Efficiency—Deviations from FY 2001 Grant Request

 

      At the most basic level, award effectiveness and efficiency can be defined as the PIs’ actual experience of a reduction in funding and/or duration—from what they requested to what NSF awarded.  In addition, it should be noted that the funding requests PIs make in their proposals prior to receiving any reductions may already be at the lower limit of what they need for their research and educational activities.  For example, PIs who resubmit proposals that were previously rejected, request less funding than those who are submitting proposals for the first time.

 

      In FY 2001, 51 percent of the PIs had a 5 percent or greater decrease in funding.  If award efficiency and effectiveness are defined as providing all PIs what they request, what would be the additional funding that is needed?  In FY 2001, these PIs requested an overall average amount of $436,000 for an average of three years and received an average of $336,000 over three years.  A calculation for each PI that subtracts the actual award from the request, and averages the differences, finds $40,000 additional annual funding per grant would be required.  The duration of the grant would remain the same at three years using this same type of calculation for award duration.

 


Option 2:  Award Efficiency and Effectiveness—Percent of Research Being Funded

 

      A second way to estimate award efficiency and effectiveness is:  What percentage of the PIs’ ongoing body of research and educational activities is being funded by the    FY 2001 NSF grant, and what is the amount needed to fund 100 percent?  Since PIs view their research and educational activities as being fluid, to provide standard criteria to answer this question, the total of what they would like to accomplish was established as 100 percent, and the time period for this accomplishment was given as five years. Using these criteria, PIs estimate that an overall average of 37 percent of the ongoing research and educational activities that they would like to accomplish in the next five years will be achieved with the FY 2001 NSF grant.  Calculating this for each PI, the average additional funding needed would be $181,000 per year.  Adding this to the FY 2001 average annual amount increases the possible funding to $293,000.

 

      Option 3:  Award Efficiency and Effectiveness—Additional Requirements

 

      A third approach to estimating award efficiency and effectiveness is:  What additional funding do the PIs estimate is needed to accomplish their key goals?  PIs were given the following guidelines to develop their estimates: (1) the additional duration should not include the years of funding they have in the FY 2001 NSF grant, and (2) the additional funding estimate is based on their needs for the next five years and should not include the current funding they have from NSF or from any other sources.  The average funding that PIs project they will need is an additional $230,000 annually.  It is important to emphasize that this amount is not all the PIs require; it is an additional amount they would add to the funding they currently have from NSF and non-NSF sources, which they anticipate will enable them to achieve their five-year goals.  When the current average annual award amount of $112,000 is added to what the PIs would like to have in additional funding, the average annual amount needed is $342,000.

 

      Option 4:  Award Efficiency And Effectiveness: NSF’s Contribution

 

      Another question related to NSF modifications in award efficiency and effectiveness is:  Should NSF be responsible for funding all PIs research and educational activities?  As defined by the focus group participants, NSF grants are distinguished from others because they do not have to be “mission-directed.”  Therefore, PIs may not expect NSF to fund all aspects of their research and educational agendas.  PIs estimate of the amount of additional funding they need in the next five years that is appropriate for NSF to fund is an average of 67 percent.  As described in Option 3, the average annual amount of additional funding PIs require is $230,000.  PIs expect NSF to fund an annual average amount of $135,000.  This amount is calculated by multiplying the additional funding each PI would like over the next five years by the percentage each PI expects NSF to fund.  If this additional amount is added to the current FY 2001 amount, the PIs would require an annual award of $247,000.


It should be noted that, for each option, there can also be variation in the amount needed for award efficiency and effectiveness, depending on the particular characteristics of the PI, such as the type of research being conducted which NSF can consider.  For example, over the next five years, PIs who have theoretical research projects expect to need less funding than those conducting field and laboratory research.

 

Although determining those changes that will best meet the needs of the country’s scientific community is complex, there are clearly benefits to making changes.  As summarized by one of the PIs:

“As I said above, I think NSF is the greatest.  But the size of the individual grants is just too small to really accomplish anything significant in my field.  At [name of institution] 200K per year will get you one post-doc and one student, and some materials and supplies and that is it.  So you need at least three of these grants to run a decent sized group.  This is a lot of grant-writing.  I wish the NSF budget could be multiplied by 10.  Then we might have something to work with.”

 

 

Benefits of Changes in NSF Funding and Duration

 

How might NSF changes in their grant-funding levels and/or duration influence ongoing research and educational activities in this country?  This question can be answered from two perspectives.  First, by assessing the impacts reported by PIs whose FY 2001 NSF grant proposal was funded, but the amount and/or duration was altered; and, second, by PIs’ speculation on how their research and educational activities might be affected if NSF were to increase its grant funding and/or the duration of its grants.

 

Using a list of 18 areas where changes in funding and/or duration potentially could affect a research project, PIs identified what they are experiencing as a result of award cuts.  Following are the key negative impacts they reported:

·   Ability to achieve their research objectives within the specified time (67%)

·   Ability to obtain quality personal (55%)

·     Ability to pursue high-risk ideas (51%)

·   Collaborate with researchers in their area of research (50%)

 

One PI summarized what happened as a result of a decrease in the FY 2001 award:

“A major impact of the shortened funded period was the lack of continuity.  It had a dramatic effect on training students and attracting new students or postdocs.  One cannot plan or attract good students and postdocs if you only have 1.5 yr of funding.  The best training and research is accomplished when you have a dynamic group of people interacting.  The synergisms are incredible.  To do this one needs to be able to constantly recruit new people and this cannot be done without some guarantee of support.”

 

Using the same 18 areas, the PIs speculated on the positive consequences if they were provided additional funding and duration.  The following are the most broadly perceived benefits:

·   Ability to pursue high-risk ideas (96%)

·   Collaboration with researchers in area of research (92%)

·     Ability to achieve research objectives within the specified time (92%)

When asked how they would spend additional funding for securing the resources needed to conduct high-quality research, “student support” is top among 16 resources presented to the PIs.  These areas are where PIs are very likely to expend additional resources:

·   Number and/or months of graduate students (78%)

·   Number of experiments, tests, subjects (54%)

·   Number and/or months of undergraduate students (50%)

·     Number and/or months of postdoctoral associates (48%)

To summarize, support for students is a major focus of what PIs view as a benefit of increases in NSF funding and/or duration.  A change in funding that gives PIs what they request in their grant proposals would provide an additional average amount of $40,000 per year.  At many schools, this difference alone could support at least one additional graduate student.  In addition to improving PI productivity, hiring additional students has the benefit of funding highly qualified students whose talents would be lost if they have to leave the sciences to support themselves in alternative careers.

 

From the PIs’ perspective, it is not only the additional funding and duration that is attractive, it is also NSF’s unique contribution to the development of students.

“Clearly more money helps, but it is the TYPE of funding that NSF provides that is unique.  So many other sources (including federal agencies) have a short-term focus on technological or economic deliverables  More NSF funds opens up a spectrum of possibilities: working on new ideas, allowing students to drive some of the research, building infrastructure and a base from which more funds can be obtained from other sources, more freedom to focus of graduate and undergraduate educational goals rather than "research" goals.  The NSF is different than industry, and needs to stay that way; and it needs to be a bigger part of academic funding.”

 


“My main concern with NSF is the 3 year cap on funding for grants as I mentioned before This 3 year cap is not enough to have a student finish a project.  Even if the project is renewed, then the renewal often does not come in time to support a graduate student’s stipend working on the original project.  As a result, the student has to TA or find a scholarship or just borrow from other grants for obtaining money to finish and graduate.  Consequently, I would be very much in favor of extending grants to 4 years rather than 3.  If this reduces the award amount slightly, so be it.  But I do not think that has to be true since people would just get fewer grants but for longer periods of time.  Since NSF is in the business of education scientists and engineers, I would think that NSF would be most interested in obtaining continuous funding for our hard-working graduate students.  These students often sacrifice personal rewards for the opportunity to do research and we shouldn’t punish them by making them go get more money to finish out their thesis since the NSF grant is out after 3 years.”

Benefits to the PI’s Field of Research

In addition to the individual benefits described above, PIs reported that increased NSF funding or duration had broader implications for their field of scientific research and education.  Among the top four areas most likely to be affected if NSF provides additional resources, two focus on students.  Following  are the key changes expected in the PIs’ field:

 

·   Decrease interruptions in funding (70%)

·   Attract more graduate students (65%)

·   Widen the focus of research in the field (63%)

·     Attract better graduate students (62%)

 

Summary

 

       The results of the surveys indicate that, at the present time, NSF is not meeting the needs of a diverse group of researchers and research and educational activities. PIs underscore the benefits to education that can be achieved by NSF award changes.  In particular, awards of longer duration will provide continuity and avert the problems associated with the disruption of graduate student education.  Additional funding can provide important benefits such as the PIs pursing their more innovative ideas and reducing their need to spend time preparing multiple proposals.  These PIs send a clear message that there is a need to make changes to improve award efficiency and effectiveness.  How that need can be met is more complex.  Just as there is variation in the types of funding NSF currently provides, there are choices about how to maximize award efficiency and effectiveness.


I.  INTRODUCTION AND BACKGROUND

A.   Introduction

In June 2001, the National Science Foundation (NSF) received instructions from the Office of Management and Budget (OMB) to prepare for the FY 2003 Budget by determining the right size grant for “the myriad types of research the agency funds.”[1]  Subsequently, OMB and NSF recognized that the process of meeting OMB’s objective could not be completed in time for consideration in the FY 2003 Budget; so the goal is to provide information that can be used to inform the development of the NSF FY 2004 budget.

To provide information that will help answer OMB’s primary question of what constitutes the right size grant needed, as well as the related issue of what needs to be done to improve award efficiency and effectiveness, NSF contracted with Mathematica Policy Research, Inc. (MPR) to conduct surveys with two key participants in the NSF grant process: (1) the Principal Investigators (PIs) who receive grants, and (2) the institutions that assist in preparing and administering NSF grants.  This report presents the results of those surveys.

B.    BACKGROUND INFORMATION

Through the support of nearly 200,000 scientists, engineers, teachers, and students each year, NSF supports about 50 percent of federal nonmedical fundamental research at U.S. colleges and universities.  Thus, the research productivity and research training of the U.S. scientific and engineering community depend heavily on NSF funding.

The concept of “fully-enabled” investment levels was used in the early discussions about research funding changes that took place in the focus groups and the cognitive pretest conducted for this study.  The three key objectives for fully enabled awards were to: (1) provide researchers and institutions sponsoring the research with resources adequate to complete the work for which the grant was awarded, (2) address the efficiency of the proposal and award processes, and (3) increase the awards’ benefits and outcomes to the nation.  These surveys were commissioned to assist NSF in understanding these objectives from the perspective of the communities that receive funding.

While the objectives of what NSF wants to achieve are clear, identifying a term that summarizes these objectives is more complicated.  The PIs who were selected to assist in the development of the questionnaire agreed that the concept, fully enabled,  did not accurately describe the goals of their grant awards.  These PIs described their research as an “ongoing body of research,” not a finite project that had a specific product or end that could be fully enabled.  In the survey, a PI summarized the difficulty of describing “fully enabled”:

“The very nature of research and investigation of hitherto unexplored ideas makes it difficult to know or state future requirements.  More funding generally means more personnel to help perform the intended research.  However, often new and unforeseen opportunities for new research arise as a consequence of new stimulus and ideas in performing the currently-funded research.  This is always difficult to assess ahead of time.  It is what makes research exciting!!”

 

Since the PIs assisting with the questionnaire rejected the concept of fully enabled, for the purpose of this report, the NSF goal for changes in the funding structure in both size and duration of awards will be referred to as award efficiency and effectiveness.  This description of awards has also been referenced in other NSF reports.  As noted in the Report to the National Science Board on the National Science Foundation’s Merit Review Process Fiscal Year 2001:  “Larger awards increase the efficiency of the system by allowing scientists and engineers to devote a greater portion of their time to actual research rather than proposal writing and other administrative work” (page 8). And, “Longer award terms are important in increasing the effectiveness of principal investigators and graduate students” (page 9).  Although this example uses these concepts to refer to the system, and not to individual grants, it can also be applied to the experiences of individual PIs.  Throughout the report, the term “award efficiency” will be used to summarize the PIs’ experiences that are related to changes in NSF funding and “award effectiveness” that will be used to summarize the experiences related to grant duration.

C.   OVERVIEW OF METHODOLOGY

As noted above, this research project included two surveys: the Principal Investigator FY 2001 Grant Award Survey (PI Survey) and the Institutional Survey.  The research design for the PI Survey called for conducting a census of all PIs included in NSF’s data file of recipients of grants in FY 2001.

A questionnaire for the PIs was developed that uses information from two focus group discussions conducted by MPR in August 2001, with 23 NSF representatives.  Cognitive pretesting of the initial draft questionnaire was completed in December 2001, with eight principal investigators from grants that year.  Two questionnaire modes were developed: a Computerized Self-Administered Questionnaire (CSAQ) for use on the World Wide Web, and a mail questionnaire.  A copy of the questionnaire is included in Appendix B.  Invitations to participate in the survey were sent by e-mail beginning January 30, 2002, and PIs had an opportunity to respond until March 8, 2002.

The total number of PIs invited to participate was 5,793 (PIs with multiple awards were selected for their response to only one grant), and 5,221, or 91 percent, returned a questionnaire. Among these returns, fewer than 1 percent were completed by mail.  The data analysis file includes 4,989 PI questionnaires that met the quality assurance completion criteria.  These


criteria included such standards as providing a response to key questions and having appropriate information about the FY 2001 grant.  Appendix A contains a detailed description of the survey process.

The Institutional Survey research design defined the universe as the 582 institutions that awarded NSF grants to PIs in FY 2001. The questionnaire for this survey was developed from individual interviews conducted with institutional representatives and with cognitive pretesting of the survey instrument.  All institutional representatives were invited to complete a self-administered mail survey.  As part of the study design, among all these institutions, a representative sample of 105 institutions was selected to be targeted for data collection and analysis.  Among all 582 institutions—359, or 62 percent—returned questionnaires.  In the sample—95, or 90 percent—completed questionnaires.  The institutional analysis reported here is based on information from the sample.  It should be noted that the results from the sample institutions have been statistically weighted to ensure that they are representative of the universe of institutions.  The appendices provide a complete description of the survey methodology, the weighting procedure, the standard errors, and a copy of the questionnaire.

Information from the non-sample institutions is not included in the text of the report.  Appendix C has information for the 264 non-sample institutions that participated in the survey.

It should be noted that information from NSF’s grant data file for FY 2001 for the PIs and the institutions is also used in this report.

In addition to this report, readers should be aware that Ruts in “The Royal Road” written by Deborah Shapley, also describes information from the PI survey.  It provides additional information about the PIs’ qualitative responses to the open-ended questions and their perspectives on possible NSF award changes.  It should be noted that the verbatim comments included in both reports are used with permission from the PIs.


D.   ORGANIZATION OF THIS REPORT

The goal of this report is to inform NSF’s effort to improve award efficiency and effectiveness. Chapter II provides a profile of NSF grant receipt characteristics for FY 2001, as well as giving an overview of the grant experience from the perspective of the PI and the institution.  This will give readers a context for the information that is reported in the following chapters.  Chapter III describes the NSF grant proposal process for FY 2001 as experienced by the PIs and institutions.  It provides insights into the level of effort needed to compete for an NSF grant.  The multiple dimensions of award efficiency and effectiveness are outlined in Chapter IV.  Chapter V summarizes the costs and benefits of potential changes in NSF awards.

Following are specific terms that will be used throughout the report:

PI:                                                            Refers to the Principal Investigators who received  NSF grants in FY 2001.

Award efficiency and effectiveness:        As described above, this refers to the broad NSF objective of identifying the changes that are needed to ensure grant awards provide adequate resources for PI research and educational activities.  In addition, for this report, the term focuses specifically on PIs’ perceptions of the resources needed to achieve their key research and educational goals.

 

Institutions:                                              Refers to the institutions that had PIs who received FY 2001 NSF grants.

In addition, it should be noted that the text and the tables in this report that refer to the NSF directorates do not include two that had a small number of PIs in the classification:  (1) Education and Human Resources Directorate (16 PIs) and (2) the directorate (O/DD) that includes primarily the Office of Polar Programs (130 PIs).

Readers should note that percentages may not always add to 100 because they have been rounded to the nearest whole number and means are also rounded to the nearest whole number.  Percentages are reported for all survey participants unless specific subgroups are referenced in the discussion.  In addition, means[2] are used to report numeric results, and dollar amounts are reported to the nearest 1,000.

E.    OTHER CONTEXTUAL CONSIDERATIONS

There are other considerations readers should be aware of as they review the information from these two surveys.  The competitiveness of the grant process, the range of economic and administrative challenges being confronted by academe, and the salaries that need to be supported are examples of the situational factors that have an impact on PIs and institutions as they consider the issues addressed in this survey that are related to NSF reforms in award efficiency and effectiveness.

The information provided in this report is limited to those who received NSF grants in FY 2001.  During FY 2001, NSF took action on 31,942 reviewed proposals and funded approximately 31 percent.  NSF estimates that approximately $1.25 billion was requested for declined proposals that received at least as high as the average rating for an awarded proposal. (Report to the National Science Board on the National Science Foundation’s Merit Review Process Fiscal Year 2001.)

Higher education is experiencing many changes that can have an impact on both faculty and institutional resources.  In particular, reduced budgets can result in institutions providing less support for research and graduate education, as well as an increase in the number of faculty expected to provide substantial portions of their own research funding.  These types of dynamics increase the risk of cutbacks in research that can affect the scientific community.

When considering funding amounts for award efficiency and effectiveness, a key question is:  How many professional staff and students can be supported?  One approach to estimating the personnel costs for research and educational activities is to know the range of typical faculty salaries and fees for graduate students.  According to the College and University Professional Association for Human Resources (CUPA-HR) salary survey for FY2000-FY 2001, the average faculty salary at a private institution is $58,700, and $59,123 at a public institution (Chronicle of Higher Education, July 27, 2001).  However, depending on the specific discipline, it may be lower or higher.  For example, the salary for bioengineering and biomedical engineering is $79,857 (private) and $72,250 (public), chemical engineering is $82,878 (private) and $82,254 (public), and for the social sciences the average faculty salary is $49,894 (private) and $54,560 (public).

Stipends for graduate assistants vary (Chronicle of Higher Education, September 28, 2001).  However, a typical NSF fellowship of $21,000 provides a standard to use when calculating the fees associated with staffing an NSF grant project in FY 2001.  Concerns about having enough funding to support a research staff and the consequences are summarized by a PI:

“As I said above, I think NSF is the greatest.  But the size of the individual grants is just too small to really accomplish anything significant in my field.  At [name of institution], 200K per year will get you one post-doc and one student, and some materials and supplies and that is it.  So you need at least three of these grants to run a decent sized group.  This is a lot of grant-writing.  I wish the NSF budget could be multiplied by 10.  Then we might have something to work with.”


II.  PROFILE OF FY 2001 NSF FUNDED GRANT PROPOSALS

NSF is known for providing awards that meet the needs of principal investigators (PIs) with diverse backgrounds and that support a range of varied research and educational activities.  We use information from the FY 2001 NSF grantee database and from the survey to give a description of the PIs’ characteristics and an overview of the NSF grant awards for FY 2001 and other funding currently supporting PIs’ research.  A discussion of these characteristics will assist in understanding the different experiences that can influence perceptions of award  efficiency and effectiveness.

a.   PRINCIPAL INVESTIGATOR CHARACTERISTICS

PIs have a range of experience in being the primary author on peer-reviewed articles.  Over the past five years, 31 percent have published more than 13 peer-reviewed articles; 30 percent have published 7 to 13 articles; and 36 percent report being the author of 6 or fewer.  PIs conduct their research in the following types of institutions:  25 percent are from one of the top 20 NSF-funded institutions, 26 percent from the top 21 to 50 NSF-funded institutions, and 22 percent from the top 51 to 100 NSF-funded institutions.  It should be noted that these categories of funding are based on the total amount an institution receives from NSF, not just the amount from these individual grant awards.


TABLE II.1

PROFILE OF FY 2001 NSF GRANT PRINCIPAL INVESTIGATORS

(Percentages)

N = 4,989

 

Professional Age:  Date of Last Degree

                      

    10 years or less

                   30

    11 to 20 years

                   34

    More than 20 years

                   36

 

Publication Experience:  Number of Articles in the Past

5 Years

                      

    Low (6 or less)

                   36

    Medium (7-13)

                   30

    High (more than 13)

                   31

PI Student Supervision

                      

    Undergraduates

                      

        Low (1 or less)

                   41

        Medium (2)

                   18

        High (more than 2)

                   24

    Graduates

                      

        Low (2 or less)

                   40

        Medium (3-4)

                   25

        High (more than 4)

                   27

    Postdoctural

                      

        Low (none)

                   33

        Medium (1)

                   25

        High (more than 1)

                   22

Type of Institution

 

    NSF funded top 20

                   25

    NSF funded top 21 to 50

                   26

    NSF funded top 51 to 100

                   22

        Other Ph.D.

                   18

        Non-Ph.D.

                     5

        Non-Academic

                     4

Gender

 

    Male

                   83

    Female

                   17

Race

 

    White

                   85

    Non-white

                   15

 

Most PIs supervise undergraduate and graduate students, as well as postdoctoral fellows.  The average number of undergraduate students assisting PIs with their current research projects is two, graduate students four, and postdoctoral fellows one (Figure II.1).  More than half of the PIs supervise three or more graduate students, compared to 22 percent of PIs who have more than one postdoctoral student working with them.  With respect to individual characteristics, 8 of 10 PIs are white and male.

B.    GRANT CHARACTERISTICS

Seventy-one percent of the FY 2001 awards were reported as first-time submissions of this grant to NSF. Among the NSF directorates, the largest percentage of the awards (26 percent) were funded in Mathematical and Physical Sciences (Table II.2).  We asked the PIs to classify the research for which they were funded in one of the following categories:

Theoretical research can be accomplished with minimal physical resources beyond the investigator’s institutional research library, computing capability, and office space.

 

Laboratory research requires an equipped laboratory, for example, research often found in chemistry, biology or engineering university laboratories requiring research and/or testing equipment, plumbing.

 

Field research requires fieldwork, specimen collection, sample survey, location of sensors, etc. away from the principal investigator’s institution, for example, some science activities in geosciences, biology, social sciences.

 

Among these classifications, 44 percent of the PIs categorized their FY 2001 NSF grant as laboratory research, 37 percent as mostly theoretical, and 18 percent as field research.

Table II.3 shows the proportion of grants in each research category for the different directorates at NSF.  Not surprisingly, more than half of the grants awarded in the Computer and Information Science and Engineering (CISE) directorate and the Mathematical and Physical


Chart of average numbers of students


 


TABLE II.2

PROFILE OF FY 2001 NSF GRANT AND OTHER FUNDING
(Percentages)
N=4,989

Grant Request
First time submission71
Revision of previously declined
   submission
29
 
Type of Research
Theoretical research37
Laboratory research44
Field research18
 
Project Requirements
For National Or International
Research Facility
Yes16
No83
 
Change In 2001 NSF
Grant Amount From
Request to Award
Increase 5% or greater2
All others47
Decrease 5% or greater51
 
Change in 2001 NSF
Grant Duration From Request To
Award
Increase 1 year or greater2
All others88
Decrease 1 year or greater10
FY 2001 Grant Amount
Low ($162,000 or less)33
Medium ($162,000+ to $330,000)34
High (more than $330,000)33
 
NSF Directorate
Biological Sciences16
Computer and Information
   Science and Engineering
12
Engineering13
Geosciences16
Mathematical and Physical Sciences26
Social, Behavioral, and Economic
   Sciences
14
 
Current Funding
Other NSF grants
   Yes44
   No55
Non-NSF Grants
   Yes72
   No27
 
Total Funding Profile
Only 2001 NSF grant19
2001 NSF grant and other NSF
   funding
9
2001 NSF grant and non-NSF
   funding
37
2001 NSF grant, other NSF, and
   non-NSF funding
35

TABLE II.3

TYPE OF FY 2001 NSF GRANT BY DIRECTORATE

(Percentages)

 

 

 

Type of Research

 

Theoretical Research

Laboratory Research

Field Research

Total

(N)

Total

      37

      44

      18

      99

(4,989)

 

Directorate

 

 

 

 

 

Biological Sciences

        7

      76

      17

    100

(819)

Computer and Information Science and Engineering

      62

      32

        5

      99

(602)

Engineering

      35

      61

        3

      99

(646)

Geosciences

      28

      30

      41

      99

(803)

Mathematical and Physical Sciences

      55

      40

        5

    100

(1,290)

Social, Behavioral, and Economic Sciences

      37

      29

      33

      99

(683)

 


Sciences (MPS) directorate were classified as theoretical research.  In the Biological Sciences (BIO), Engineering (ENG), and Education and Human Resources (EHR) directorates, the majority of grants were classified as laboratory research; while in the Geosciences (GEO) and the Social, Behavioral, and Economic Sciences (SBE) directorates, the grants were closer to being equally divided among all three research categories.  When asked about international research facilities—such as an accelerator, light source, ship, major telescope, or supercomputer—about one of five PIs require the use of such facilities.

C.   AMOUNT AND DURATION OF FUNDING

Based on the NSF grantee database, the average NSF grant for the year FY 2001 was $336,000, and the average award duration was three years.  A third of the PIs received total awards of $162,000 or less, another third received awards of more than $162,000 but less than $333,000, and the final 33 percent received more than $333,000 (Table II.2).

D.   CHANGES IN FUNDING AND DURATION

Overall, NSF did not request any changes of 5 percent or greater in the proposed award amount for nearly half (47%) of the funded proposals (Table II.2).  However, 51 percent of the grants had a decrease of 5 percent or greater in the amount of funding awarded, compared to the amount of funding requested, while 2 percent had an increase of this amount.  Chapter IV describes these funding changes in greater detail.

When awarding a grant, NSF is much less likely to change the requested duration of the proposed grant for research and education. Close to 9 of 10 PIs (88%) reported no change in the grant duration they requested for conducting their research and educational activities, compared to 10 percent who reported a decrease of one year or greater and 2 percent who reported an increase of one year or greater in the duration of their award (Table II.2).

E.    OTHER FUNDING

In addition to their FY 2001 NSF grant, most PIs report other funding for their research and educational activities.  For 19 percent of the PIs, the FY 2001 NSF grant is the only source of current funding; 9 percent of the PIs have this grant and other NSF funding, 37 percent have additional non-NSF funding, and 35 percent have other NSF and non-NSF funding (Table II.2).  Among the 44 percent of the PIs who have other NSF grants, based on the resources they currently have (which may not be the total amount and duration of these other grants), the average number of other NSF grants is two, and the current amount is $207,000.  Among the 72 percent who now have non-NSF grants, the current average number is two, and the average amount is $199,000.

F.    PROFILE OF INSTITUTION GRANT MANAGEMENT

The Institutional Survey complements the Principal Investigator FY 2001 Grant Award Survey by providing information on how institutions manage the grant process.  It should be noted that the process of applying for and administering NSF grants varies greatly from institution to institution.  In our discussions with institutions while developing the questionnaire, it was clear that there is no “typical” administrative organization or grant process.  For example, depending on the institution, there is variation in how many or what type of grant activities are centralized, versus their location in various different institutional units and departments.  The responses to this survey are from the contact person nominated by that person’s institution as “the most knowledgeable about the overall grant process from the proposal phase to grant administration. And, who has final administrative responsibility for this process.”

We asked these institutional representatives about the way they are organized in general to handle the various aspects of grant management, and specifically about the use of resources for a “typical” FY 2001 NSF grant.  The representatives provided information on the administrative offices and individuals who are assigned to grant proposals, grant proposal revisions, and the administration of grant awards.  The number of administration offices involved in these three types of grant activities are as follows: grant proposal—an average of 2; grant proposal revisions—an average of 2; and grant administration—an average of 2 (Table II.4).  The individuals assigned to each of these activities are:  grant proposal an average of 6; grant proposal revisions—an average of 5; and grant administration—an average of 8.

As a context for the level of institutional management used for NSF grants, compared to the total number of grants administered in FY 2001, we asked institutions what percentage of the total grant awards and total award dollars are from NSF grants.  For the total number of grants, 16 percent is the average.  This estimate is similar for the total FY 2001 budget, with 18 percent the average percentage of the funding that comes from NSF.  Another dimension of the level of effort expended is the institution’s number of NSF proposal awards and declines.  Using the institutional information from the FY 2001 NSF grant data file, the average number of FY 2001 NSF grant awards is 12, compared to an average of 32 proposals that were declined.  Chapter III goes into detail about the estimated number of hours institutions devoted specifically to FY 2001 NSF grants.

Forty-four percent are PH. D granting institutions that are not among the top 100 funded by NSF in FY 2001.  They also include 5% who are the 20 top FY 2001 NSF-funded institutions, 3% rank in the top 21 to 50, and 9% are in the top 51 to 100 (Table II.4).


TABLE II.4

PROFILE OF INSTITUTION LEVEL OF GRANT EFFORT

N = 95

 

 

 

Mean*

Number of FY 2001 NSF Grant Awards

               12

Number of FY 2001 NSF Grant Declines

               32

 

Percent of FY 2001

   Grants from NSF

               16

Percent of FY 2001 Total Grant

  Dollars from NSF

               18

Individuals Assigned to:

 

  Grant proposals

  Grant proposal revisions

  Grant administration

                 6

                 5

                 8

Administrative Offices Assigned to:

 

  Grant proposals

  Grant proposal revisions

  Grant administration

                 2

                 2

                 2

................................................................................................................................

Type of Institution (Percentages)

 

   NSF funded top 20

                 5

   NSF funded top 21 to 50

                 3

   NSF funded top 51 to 100

                 9

      Other Ph.D.

               44

      Non-Ph.D.

               28

      Non-Academic

               11

 

            * Standard errors for the means are in the Appendix


III.  The Proposal Process

The level of effort the PI expends to submit an application to NSF is an important factor to consider in assessing the funding and duration of NSF grants.  While there may be some benefits to the PI in using the NSF proposal process as an opportunity to clarify research and educational goals and objectives, PIs are more likely to view the process as one in which resources are expended in preparing a proposal that may or may not be funded.  Both the time and the money needed to prepare proposal submissions could be used to support research and educational activities.  It should be noted that the questionnaire did not explicitly ask the PIs to evaluate the costs and benefits of the proposal process.  However, among the PIs who volunteered verbatim comments about the proposal process, those who discussed the cost of time and resources expended were more numerous than those who mentioned the benefits.  The study did, however, obtain information with which to determine the extent of the resources PIs and institutions use in submitting proposals. This information provides an estimate of the amount that could be saved if NSF grant awards were more efficient and effective.  This chapter summarizes the level of effort PIs and institutions use to obtain grant funding.

A.   LEVEL OF PRINCIPAL INVESTIGATOR NSF PROPOSAL EFFORT

One measure of proposal preparation effort is the total hours PIs estimate they needed to prepare their FY 2001 NSF grant submissions.  This estimate includes the time spent by the PI and other people (such as graduate assistants, budget administrators, and secretaries) preparing the submission.  The measure does not include time spent by institutional personnel.  On average, PIs estimate 157 hours, or about 19.5 days (assuming an 8-hour day) for total proposal preparation.  Assuming 235 working days in a full year, on average, about 8 percent of days in a working year is estimated as needed to prepare this one proposal.  Overall, 33 percent of the PIs estimated hours for this proposal preparation in the low range of less than 80 hours; 29 percent are in the medium range of 80 to 150 hours; and 29 percent are in the high range of more than 150 hours used for proposal preparation (Table III.1).  As summarized by one of the PIs, this level of effort can detract from other activities:

“…Raising funding is almost a full time job for me.  If I didn’t have to do that, I could spend more time mentoring women and minority students, and I could do my “dream projects”—those that are either too high risk, or too expensive to do in the current funding environment.”

 

Another PI, who is on 100 percent “soft money” provides a description of the level of proposal effort that is needed to get sufficient support:

“For a soft money researcher needing to cover 12 months of his/her time and current budget standards pushing for 3 months (or less) coverage for senior personnel per proposal (seemingly, this is only communicated obliquely) and an average of three year research duration this results in obtaining four funded proposals per three years.  This would be a limited time sink if success was 100% (4 weeks/year of proposal writing at three weeks/proposal).  At 50% it is eight weeks/year writing proposals.  At 33% success it is 12 week/year and so on.  It seems that a better balance can be found.  Consider that many programs have funding rates of 10-33 percent.  (You tell me, I only hear these numbers, perhaps they are a worst case).”

 

Contrary to what might be expected, PIs who submitted a revision of a previously declined NSF proposal (36%) are more likely than those with a first-time submission (26%) to be in the high range of proposal preparation hours (Table III.1).  In addition, PIs who describe the research funded by this grant as laboratory (35%) and field (33%) are more likely to report hours for proposal preparation in the high range than those who describe their grant as theoretical (19%).

Overall, most grant preparation time is spent on the intellectual content of the proposal, compared to proposal mechanics.  On average, PIs report that 69 percent of the time was devoted to intellectual content, compared to 31 percent on the mechanics of proposal preparation.  The


TABLE III.1

PI LEVEL OF PROPOSAL EFFORT

(Percentages)

 

 

 

Low

80 or Less Hours

Medium

81 to 150 Hours

High

More Than 150 Hours

 

Don’t Know

Total

(N)

Total

         33

         29

         29

        8

      99

(4,989)

 

Grant Request

 

 

 

 

 

 

   First time submission

         36

         28

         26

        9

      99

(3,521)

   Revision of previously      declined submission

 

         25

 

         31

 

         36

 

        8

 

    100

 

(1,449)

 

Type of Research

 

 

 

 

 

 

   Theoretical

         43

         27

         19

      10

      99

(1,863)

   Laboratory

         25

         32

         35

        8

    100

(2,186)

   Field research

         31

         28

         33

        7

      99

  (902)


ranges for the percentage of time spent on preparation of intellectual content are: low, 65 percent or less (35%), medium, 66-80 percent (47%), and high, more than 80 percent (18%).  In comparison, the range for mechanics is: low, 20 percent or less (37%), medium, 21-35 percent (30%), and high, more than 35 percent (32%).  This suggests that a majority of the preparation time is expended by the PIs, since they are the main contributors to the intellectual content of the proposal.

Another level of proposal cost and effort is the PIs interaction with NSF to develop the grant proposal.  Overall, about half of the PIs got advice from NSF about the amount of funding (57%), the substance of the grant (51%), and the grant duration (48%). Consultation with NSF resulted in 27 percent of the PIs decreasing the grant amount requested in their proposal, and 4 percent making an increase.  PIs were less likely to change their duration request.  Six percent decreased, and 3 percent increased, the duration of their grant request.

PIs who do not have sufficient funding for their research and educational activities need to prepare and submit multiple grants.  For example, 38 percent of the PIs reported that they divide their ongoing body of research and educational activities into several proposals and submit them to NSF.  More PIs conducting field research (44%) and laboratory research (40%) than PIs doing theoretical research (32%) reported dividing their work and submitting multiple proposals.

To summarize, just for the grant that the PIs submitted to NSF, proposal writing itself took about 20 days, and the key person in preparing the grant is the PI.  As noted by some PIs, this time they might have used more effectively for research and educational activities.

B.    OTHER PROPOSAL SUBMISSIONS

For this discussion of the level of effort required for proposals, it is important to review the extent to which PIs prepared submissions other than for their FY 2001 NSF proposal that was described in Chapter II.  In addition to submitting multiple grants to NSF, the PIs reported funding from other sources where proposals were submitted.  To summarize, overall, about 8 of 10 PIs currently have funding, in addition to the FY 2001 NSF grant identified for this survey (Table III.2).  It should be noted that depending on characteristics such as their type of research, some PIs spend more time on additional proposals than others.

With some PIs reporting that they have been awarded an average of four additional grants, an estimate can be made of the extent of PI time spent annually on proposals.  Assuming the same number of hours used for each grant proposal that PIs estimated for the FY 2001 NSF grant, there could be an estimated 628 additional hours, or 79 days spent on preparing successful proposals.  However, it should be noted that PIs may also be spending time preparing unsuccessful proposals, and this effort is not included in these estimates.

The level of effort devoted to proposal preparation can have an impact on award efficiency and effectiveness, as described by this PI:

     “The trend of NSF to fund longer time periods is a great and useful change.  It reduces the time spent preparing proposals, increases the time doing research and provides stability for ongoing research programs.”

 

C.   LEVEL OF INSTITUTIONAL PROPOSAL EFFORT

In addition to the hours spent by the PIs and their proposal assistants, there is proposal preparation time expended by the institutions where the PIs do their research and conduct their educational activities.  From the institution’s perspective, this level of effort includes both proposals that are funded and those that are not.  Among the institutions, there is variation in the


TABLE III.2

TOTAL FUNDING PROFILE

(Percentages)

 

 

 

Only      FY 2001    NSF Grant

FY 2001 NSF Grant and Other NSF Funding

FY 2001 NSF Grant and Non-NSF Funding

FY 2001 NSF Grants, Other NSF and Non-NSF Funding

Total

 

 

 

(N)

Total

       19

           9

        37

        35

       100

    (4,989)

 

Type of Research

 

 

 

 

 

 

   Theoretical research

       27

           9

        36

        28

       100

    (1,863)

   Laboratory research

       14

           8

        40

        39

       101

    (2,186)

   Field research

       13

        12

        34

        41

       100

       (902)

 

Additional Funds to Accomplish Goals

 

 

 

 

 

 

   Low ($300,000 or less)

       27

        11

        38

        23

         99

    (1,616)

   Medium ($300,000+ to

      $750,000)

       16

        10

        37

        37

       100

    (1,377)

   High (more than       $750,000)

         6

           7

        37

        50

       100

    (1,352)

   Do not know

       29

           7

        34

        30

       100

       (644)

 

Research Divided Into Several Proposals

 

 

 

 

 

 

   Yes

         8

        12

        23

        56

         99

    (1,895)

   No

       25

           7

        46

        22

       100

    (3,069)

 

Directorate

 

 

 

 

       100

 

   Biological Sciences

       19

           8

        46

        27

       100

       (819)

   Computer and

       Information, Science,

       and Engineering

       16

        12

        31

        41

       100

       (602)

   Engineering

         8

           7

        38

        47

       100

       (646)

   Geosciences

       12

        11

        26

        52

       101

       (803)

   Mathematical and

       Physical Sciences

       29

           8

        39

        24

       100

    (1,290)

   Social, Behavioral, and

       Economic Sciences

       23

           8

        45

        24

       100

       (683)

 


level of institutional proposal involvement.  For the purpose of describing the hours spent on proposal preparation, only those hours spent by the initial office are included.  This may be an underestimate because as noted earlier, for some institutions, more than one office is involved in the proposal effort; but one office is typical.  For a typical FY 2001 NSF proposal, the institutions estimate an average of 6 hours for preparation (Table III.3).  Using the institutional information from the NSF data file, 44 is the average number of accepted (12), and declined (32) NSF proposals; the total average number of hours spent on proposal activity by the institutions can be estimated at 264 hours, or about 33 days (assuming an 8-hour day).

There is an additional level of effort when the institutions are included in the proposal revision process—that is, when NSF does not accept the original proposal, and the institution is involved in preparing the revision.  Overall, from the institutions’ perspective, the average number of hours for each proposal revision is three.

Changes in award efficiency and effectiveness, which reduced the number of proposals needed for preparation, would have multiple benefits as described by one institution:

       “This could change the number of proposals submitted which would require less time for the PI Department and proposal review and submission from the Administrative Offices.  This would help the grant administration as the same amount of time per year would be required, yet there would be additional funds.”


TABLE III.3

INSTITUTIONAL LEVEL OF EFFORT FOR TYPICAL FY 2001 NSF

GRANT ACTIVITIES

(Based on Hours Per Grant)

 

 

 

Mean*

Grant Proposal

 

  Number of hours

                   6

Grant Proposal Revision

 

  Number of hours

                   3

Grant Administration

 

  Number of hours

                 21

Grant Report Requirements

 

  Number of hours

                   6

           

            *Standard errors for the means are in the Appendix.


IV.  MEASURES OF AWARD EFFICIENCY AND EFFECTIVENESS

A key objective of the PI survey is to find out:  What are the award duration and funding amounts that are most efficient and effective in promoting NSF’s objectives?  The answer is not straightforward.  NSF has multiple objectives when it awards grants.  As identified in the focus groups used to develop the survey, NSF grants contribute to the enhancement of the country’s scientific community by: (1) providing the only funding source for some scientific fields; (2) giving researchers the freedom to pursue unique ideas that reflect their own interests and expertise; (3) providing support for fundamental research; (4) playing a central role in establishing the careers of young researchers; (5) supporting the education and training of students; and (6) providing intellectual and scientific benefits through the review process.  PIs who received the FY 2001 NSF grant awards are carrying out these diverse objectives.  This diversity underscores what was said in the initial focus group discussions—that, at NSF, one size does not fit all.  It also increases the challenge of establishing criteria to achieve award efficiency and effectiveness.  This chapter summarizes the information that can be used to meet this challenge.

A.   DIFFERENT MEASURES OF AWARD EFFICIENCY AND EFFECTIVENESS

As described in the PI cognitive group discussion, asking about the extent of additional funding and duration needed to support ongoing research and educational activities is like “trying to measure a dream.”  Because the question of what is the “appropriate” funding amount for award efficiency and effectiveness is critical, the questionnaire used multiple approaches to get the PIs’ perspectives.  In the following, we describe each measure of award efficiency and effectiveness (Table IV.1).  It should be noted that, for the discussion of the amounts of funding


TABLE IV.1

 

PROFILE OF AWARD EFFICIENCY AND EFFECTIVENESS ATTRIBUTES

(Means)

 

 

 

Award Request

 

Duration Request

 

Award Amount

 

Award Duration

Additional Funding for Next 5 Years

Additional Duration for Next 5 Years

Percent of Additional Funding From NSF

Total

$436,000

    3

$336,000

      3

$1,149,000

    3

     67

Type of Research

 

 

 

 

 

 

 

   Theoretical

$373,000

    3

$276,000

      3

$740,000

    2

     71

   Laboratory

$507,000

    3

$390,000

      3

$1,190,000

    3

     63

   Field

$395,000

    3

$331,000

      3

$1,839,000

    3

     71

 

Directorate

 

 

 

 

 

 

 

   Biological

     Sciences

 

$585,000

 

    3

 

$436,000

 

      3

 

$1,135,000

 

    3

 

     67

 

Computer and

     Information

     Science and

     Engineering

 

 

 

$635,000

 

 

 

    3

 

              

 

$447,000

 

 

 

      3

 

               

 

$1,342,,000

 

 

 

    2

 

 

 

     66

 

     Engineering

$378,000

    3

$315,000

      3

$1,255,000

    2

     56

     Geosciences

$309,000

    3

$270,000

      3

$1,871,000

    3

     71

 

Mathematical

    and Physical

    Sciences

 

 

$459,000

 

 

    3

 

              

$350,000

 

 

      3

 

 

$751,000

 

 

    3

 

 

     70

 

Social,

    Behavioral

    and Economic

    Sciences

 

 

 

$229,000

 

 

 

    2

 

              

 

$177,000

 

 

 

      2

 

 

 

$715,000

 

 

 

    3

 

 

 

     71

 

 


and or duration needed for award efficiency and effectiveness, the means include those for single variables and those for constructed variables as described in Appendix A.

Option 1.    Award Effectiveness and Efficiency: Deviations from Requested Award Amount

At the most basic level, award effectiveness and efficiency can be defined as the PIs’ actual experience of a reduction in funding and/or duration from what they requested to what NSF awarded.  In FY 2001, 51 percent of the PIs had a 5 percent or greater decrease in funding, and 10 percent had a one year or greater decrease in duration.  If award efficiency and effectiveness are defined as providing all PIs what they request, what would be the additional funding that is needed?

In FY 2001, these PIs requested an average amount of $436,000 for three years, and received $336,000 over three years.  Defining award efficiency and effectiveness as providing PIs the funding they requested in their original proposal, a calculation for each PI that subtracts the actual award from the request and averages these differences finds $40,000 additional annual funding per grant would be required.  The duration of the grant would remain the same at three years using the same type of calculation (Table IV.1 and Figure IV.1).

Using the same definition, there are variations in award efficiency and effectiveness among different types of PIs.  For example, from the funding perspective, PIs conducting laboratory research and education would require $48,000 additional funding compared to those doing theoretical ($34,000) or field ($29,000) research and education (Figure IV.2).

There may be a caveat when these estimates are used to calculate award effectiveness and efficiency.  The amounts the PIs request in their proposals may already have been adjusted to a lower amount because of advice the PIs receive when they submit or re-submit their budgets or because of perceptions of the amount of funding NSF is expected to support.


       Chart of award funding options annual projected amounts


I               Chart of award funding options by research category


OPTION 2.    Award Efficiency and Effectiveness:  Percent of Research Being Funded

Another way to estimate award efficiency and effectiveness is to consider what percent of PIs’ ongoing body of research and educational activities is being funded by their FY 2001 NSF grant, and the amount needed to fund them 100 percent.  Since PIs view their research and educational activities as being fluid, to provide standard criteria to answer this question, the total of what they would like to accomplish was established as 100 percent, and the time period for this accomplishment was given as the next five years.

Using these criteria, PIs estimated an average of 37 percent of the ongoing research and educational activities they would like to accomplish in the next five years will be achieved with the FY 2001 NSF grant (Table IV.2).  Calculating the amount that would equal 100 per cent and subtracting the PIs current funding, there is an average difference of $904,000 additional funding.  Over a five-year period, this equals $181,000 per year (Figure IV.1).

PIs conducting field research and education (34%) have a lower average estimate of what the current NSF grant can contribute in the next five years, compared to those doing theoretical (39%) and laboratory (37%) research and education.  However, using this definition of award efficiency and effectiveness, the actual funding amounts required are greater for PIs conducting laboratory research and field research than for those doing theoretical research (Figure IV.2).

OPTION 3.    Award Efficiency and Effectiveness:  Additional Requirements

A third approach to estimate award efficiency and effectiveness is:  What additional duration and funding do the PIs estimate is needed to accomplish their key goals?  PIs were given the following guidelines to develop their estimates:  (1) the additional duration should not include the years of funding they have in the FY 2001 NSF grant, and (2) the additional funding estimate is based on their needs for the next five years, and should not include the current funding they have from NSF or from any other sources.  Using these criteria, PIs estimates that they need an


TABLE IV.2

AWARD EFFICIENCY AND EFFECTIVENESS

FUNDING AMOUNT TO ACCOMPLISH A 100 PERCENT IN FIVE YEARS

(Means)

 

Award Dollar Amount

Percent Achieved with FY 2001 Grant

Dollar Amount for 100%*

5-Year Additional Dollar Amount Needed for 100%*

Total

     336,000

        37

     1,240,,000

     904,000

 

Type of Research

 

 

 

 

   Theoretical Research

     276,000

        39

         946,000

     671,000

   Laboratory Research

     390,000

        37

      1,443,000

  1,055,000

   Field Research

     331,000

        34

      1,315,000

     985,000

 

Directorate

 

 

 

 

   Biological Sciences

     436,000

        39

      1,561,000

  1,122,000

   Computer and Information

      Science and Engineering

 

     447,000

 

        36

 

      1,680,000

 

  1,222,000

   Engineering

     315,000

        36

      1,211,000

     900,000

   Geosciences

     270,000

        30

      1,076,000

     812,000

   Mathematical and Physical

      Sciences

 

     350,000

 

        45

 

      1,124,000

 

     782,000

   Social, Behavioral, and

      Economic Sciences

 

     177,000

 

        32

 

         790,000

 

     608,000

 

*Constructed variables created for each PI and then averaged.  See Appendix A for additional information on the calculation.

 


average of three additional award years.  When the additional years the PIs suggest they would like are added to the years for each current award, five years is the average estimate to accomplish their key goals.

The questionnaire asked the PIs to focus on their needs for the next five years.  The additional average funding of PIs’ project that they need in the next five years is $1,149,000 (Table IV.1) or an annual amount of $230,000 (Figure IV.1).  It is important to remember that the $230,000 is not all that the PIs require; they would add this amount to their current funding from NSF and non-NSF sources to enable them to achieve their five-year goals.

The diversity of types of research funded by NSF is underscored by the need for additional duration and funding described by the PIs (Table IV.1).  For example, using the means, PIs conducting laboratory and field research and education want three additional years, compared to those conducting theoretical research and education, who want an average of two years (Table IV.1).  Over the next five years, PIs conducting theoretical research and education expect to need an average of about $148,000 per year, which is less than the $238,000 average for laboratory research and the $368,000 average for field research and education (Figure IV.2).

OPTION 4.    Award Efficiency and Effectiveness:  NSF’s Contribution

Another question related to NSF modifications in award efficiency and effectiveness is: Should NSF be responsible for funding all the PIs’ research and educational activities?  As defined by the focus group participants, NSF grants are distinguished from others because they do not have to be “mission-directed.”  Therefore, PIs may not expect NSF to fund all aspects of their research and educational agendas.  PIs gave an estimate of the amount of additional funding they need in the next five years which they consider appropriate for NSF to fund.

The average percentage of funding that PIs suggest for NSF to fund is 67 percent.  As described in Option 3 above, the annual amount of additional funding that PIs require is $230,000.  The annual average amount each PI expects from NSF is $135,000 (Figure IV.1).  This amount is calculated by multiplying the additional funding each PI would like over the next five years by the percentage that each PI expects NSF to fund and computing the mean.

Not all PIs agree on the percentage of funding appropriate for NSF.  For example, PIs conducting theoretical (71%) and field (71%) research and education, expect NSF to pay a higher average percentage of the additional funds they require over the next five years, than for those doing laboratory (63%) research and education.  These differences probably are based on PIs’ perceptions of NSF’s mission and of the typical type of research project NSF funds.  The PIs conducting theoretical research and education projects ($94,000) expect the lowest amount of additional average annual funding from NSF, compared to laboratory ($140,000) and field ($199,000) research and education (Figure IV.2).

B.    SUMMARY OF DIFFERENT METHODS TO ESTIMATE AWARD EFFICIENCY AND EFFECTIVENESS

These four examples demonstrate the different choices to be made in determining the appropriate funding and duration for improving NSF award efficiency and effectiveness.  Information from the PIs suggests that there is a greater consensus on award duration than on the amount.  An average of five-year awards will, PIs feel, give them the most effective period of time for their research and educational activities.  These awards would have such benefits as continuity of employment for students, particularly graduate student staff, opportunities to explore other areas of inquiry that may develop as they conduct their research, higher quality of research, and less time spent pursuing additional funding.  However, NSF has a greater number of choices in deciding about additional dollars per year needed to fund grants that are efficient and effective.  The previous examples of four possible ways to estimate award efficiency and effectiveness from the PIs’ perspective have averages that range from $40,000 to $230,000 per year (Figure IV.1).  In addition, there is variation in these amounts, depending on particular characteristics of the PI, such as the type of research being conducted.  NSF can use this information to identify which funding approach best meets the needs of the country’s scientific community.  In Chapter V, we go into detail on the benefits of NSF improving award efficiency and effectiveness.

As described above, the requests for additional funding do not include their FY 2001 NSF grant or any other current funding.  Figure IV.3 illustrates the projected total annual amounts when the calculations for the different options are added only to the average FY 2001 NSF grant amount.  Overall, possible changes in annual funding amounts could range from $152,000 to $342,000 for an average NSF grant.

The four approaches to estimating award efficiency and effectiveness, focus on contributions expected from NSF, based on the FY 2001 grant identified for this survey.  However, additional calculations could be made if NSF wanted to explore supplementing or replacing other PI funding sources.  Obtaining the necessary funding from a single or few sources frees up PIs’ time, enabling them to conduct more research and to work more with students.

The survey included information on two additional current funding sources: (1) other NSF funding and (2) current non-NSF funding.  As described in Chapter II, 8 of 10 PIs currently have other funding.  Among those who have other NSF funding (44%), the current annual average amount is $207,000 (Figure IV.4).  Currently 72 percent of the PIs have non-NSF funding that has an annual average of $199,000 (Figure IV.5).  Depending on NSF’s decision on such criteria as the types of proposals they want to fund and what they want to achieve with award efficiency and effectiveness, these are additional estimates of funding that can be used to develop a budget for future awards.


     Chart of award funding options added to FY 2001 award




C.   PRIORITY ACTIONS FOR AWARD EFFICIENCY AND EFFECTIVENESS:         PI PERSPECTIVE

The diversity of needs was reinforced when PIs were asked about their preferences for their own research.  Thinking about what they specifically would like to accomplish, for about half of the PIs, more funding would have the greatest impact on their ongoing body of research and educational activities, while, for one-third, a longer duration would provide the more important resource.  One of 10 PIs did not respond to this question because they did not make a request for a specific amount of additional funding or duration.

d.   PRIORITY ACTIONS FOR AWARD EFFICIENCY AND EFFECTIVENESS:  FIELD OF RESEARCH

As NSF considers ways in which to improve award efficiency and effectiveness, resources may be used in three key ways, to:  (1) increase the amount of funding, (2) increase the length of time per award, and (3) increase the total number of awards per year.  For this question, the PIs were asked to change their focus and think about their general field of research, rather than their own specific grant.  From the PIs’ perspective, there is no consensus as to which of these actions NSF should take if more funding is available.  Among the three choices for their field of research, more PIs consider additional funding (40%) as their first choice, while an almost equal percentage prefer increasing the total number of awards (36%), and 24 percent would increase only the duration of the award (Table IV.3).  The variation in PIs’ selection of choices underscores the multiple research and educational needs that NSF grants meet, which may require different types of resources.  For example, PIs who describe their area of research and education as theoretical (35%) are less likely than those who do field work (39%) or laboratory research (43%) to select an increase in the award amount as their first priority for their field.


 

TABLE IV.3

MOST IMPORTANT NSF AWARD REFORM FOR PI AREA OF RESEARCH

(Percentages)

 

 

 

Increase Only the Amount of Funding Per Award

Increase Only the Length of Time Per Award

Increase Only the Total Number of Awards Per Year

Total

(N)

Total

         40

         24

         36

    100

(4,989)

 

Change in 2001 NSF Grant Funding

 

 

 

 

 

   5% or greater increase

         32

         20

         47

      99

(123)

   5% or greater decrease

         44

         24

         31

      99

(2,533)

   All others

         35

         24

         40

      99

(2,333)

 

Change in 2001 NSF Grant Duration

 

 

 

 

 

   1 year or greater increase

         43

         22

         34

      99

(92)

   1 year or greater decrease

         40

         29

         30

      99

(485)

   All others

         39

         23

         36

      98

(4,412)

 

Type of Research

 

 

 

 

 

   Theoretical

         35

         26

         38

      99

(1,863)

   Laboratory

         43

         25

         31

      99

(2,186)

   Field research

         39

         17

         42

      98

(902)

 

Additional Funds to Accomplish Goals

 

 

 

 

 

   Low (less than $300,000)

         32

         24

         43

      99

(1,616)

   Medium ($300,000 + to $750,000)

         40

         26

         33

      99

(1,377)

   High (greater than $750,000)

         52

         22

         26

    100

(1,352)

   Do not know what amount is needed

         31

         23

         45

      99

(644)

 

Transparent Mathematica logo
 



V.  POTENTIAL IMPACT OF INCREASED FUNDING OR DURATION

How might NSF reforms in its grant funding levels and/or duration influence ongoing research and educational activities in the United States?  We looked at this question from two perspectives. First, we looked at the impacts reported by PIs when the FY2001 NSF grant they had requested was awarded but the funding level or duration had been altered.  Second, we asked the PIs to speculate on how their own research might be affected if NSF increased the funding and/or duration of their grant awards.  The results of these questions are described below.

A.   The CONSEQUENCES OF REDUCTIONS IN FUNDING OR DURATION

As described earlier in this report, among the FY2001 NSF grantees, almost half (47%) were funded within 5 percent of the amount requested.  Of the remaining 53 percent, 51 percent of the grantees had their funding cut by 5 percent or more, while 2 percent had their funding increased by more than 5 percent or more.  Changes to the grant duration were even less frequent, with 88 percent remaining unchanged.  Of the 12 percent with changes, 10 percent had their duration decreased by more than one year, and 2 percent had it increased by more than one year.

The questionnaire listed 18 areas where changes in funding or duration could potentially affect a research project.  PIs whose NSF grants for FY2001 included changes in funding or duration were asked whether, for each area, the change had proven to be positive, of no consequence, or negative.  Not surprisingly, most PIs reported some type of negative impact.  As shown in Table V.1, approximately half (55%) said the change had affected their ability to obtain quality personnel, and another 50 percent said it had limited their ability to collaborate with researchers in their area of research. Close to half (51%) felt that the change had restricted their ability to pursue innovative or “high-risk” ideas.  Again, not surprisingly, about two-thirds (67%)


TABLE V.1

 

NEGATIVE IMPACT OF FUNDING AND/OR DURATION AWARD CHANGES

(Percentages)

 

 

 

Negative Impact

 

 

Total

(N=4,989)

5% or Greater Decrease in Funding

(N = 2,533)

1.

Achieve the research objectives within the specified time

 

            34

 

           67

2.

Obtain quality personnel

            28

           55

3.

Collaborate with researchers in your area of research

            26

           50

4.

Pursue high-risk ideas

            26

           51

5.

Pursue innovative ideas

            23

           45

6.

Collaborate with researchers in different areas of research

 

            21

 

           41

7.

Establish mentoring or other research-based education activities

 

            21

 

           41

8.

Develop instrumentation or other enhancements for the research and education infrastructure

 

            20

 

           40

9.

Integrate research activity into your teaching and training

 

            18

 

           36

10.

Broaden participation of under-represented groups in the research activity

 

            18

 

           34

11.

Access state-of-the-art equipment

            17

           33

12.

Disseminate research findings

            17

           33

13.

Develop partnerships with industry, other educational institutions, or national laboratories

 

            13

 

           25

14.

Improve public understanding of the project

            11

           20

15.

Access facilities

            10

           20

16.

Nurture connections between research activity and its potential for:  health benefits, economic benefits, and national security benefits

 

 

            10

 

 

           20

17.

Develop programs with K‑12 teachers and/or students

 

              7

 

           13

18.

Obtain other funding

              6

           12


indicated that the change had negatively affected their ability to “achieve their research objectives within the specified time.”

PIs were given the opportunity to describe any other impacts they had experienced due to changes in their award.  Overall, 48 percent of those who had a grant change provided a verbatim response. Table V.2 contains a summary of these comments. It should be noted that these percentages are based on the number of responses given, not on the individual PIs.  The key impact described in 35 percent of the responses was the negative impact on building a project team.  Among the responses on this topic, a main concern was the ability to recruit and retain staff (22%).  The second most mentioned  consequence of award changes was the impact on meeting the goals and objectives of what had been proposed.  Nine percent of the comments about goals and objectives were focused on a reduction in the scope of the project.  Thirteen percent of the responses mentioned an impact on aspects of the project process, with 8 percent focused specifically on the increased time spent seeking funding, rather than working on their research and educational activities.

Several PIs summarized their experience resulting from the decreases in their FY2001 awards:

“A major impact of the shortened funded period was the lack of continuity. It had a dramatic effect on training students and attracting new students or postdocs. One cannot plan or attract good students and postdocs if you only have 1.5 yr of funding. The best training and research is accomplished when you have a dynamic group of people interacting. The synergisms is incredible. To do this one needs to be able to constantly recruit new people and this cannot be done without some guarantee of support.”

 

       “The question before addressed well the negative impact that the decrease in funding has on my research program. Having funding for 5 years instead of three would have allowed me to have a broader focus and to be somewhat more ambitious in my long term research goals. Also, a considerable amount of effort goes into the preparation of a grant proposal, and so longer periods between submissions could benefit the research output. It is clear that there is a positive side to the writing of grant proposals, the need


TABLE V.2

IMPACT OF CHANGE IN AWARD FUNDING AND/OR DURATION

(Percentages)

 

 

 

Response Categories

Total Responses

(1,835)

Negative impact on team building – staff, student, collaborators

                        35

 

Reduction in meeting goals and objectives

 

                        27

 

Negative impact on project process

 

                        13

 

No impact or impact not known

 

                          8

 

Limited research tools

 

                          7

 

Comments on positive impact

 

                          6

 

Other comments

 

                          4

 

TOTAL

                      100

 


to clarify one's ideas and to plan ahead with scientific vision, but this is also well achieved through the annual progress reports.”

 

It is useful to contrast the experiences of PIs who had a decrease with those of the small number of PIs who received increases in funding and/or duration.

       “Increased funding level has allowed increased flexibility in how we approach our research, which should have a positive impact on our productivity and which is having a very positive effect on graduate student training.”

B.    THE POTENTIAL BENEFITS OF ADDITIONAL NSF FUNDING OR DURATION ON THE PIs CURRENTLY ONGOING  RESEARCH

Switching from the limitations that a reduction in funding or duration might bring, the PIs were asked to describe the impact of receiving additional funding and/or time on their ongoing body of research and educational activities.  This leads to a key question: If funding and duration increases are implemented, what are the potential benefits?  When PIs who specified a specific additional amount of funding and/or duration were asked this question with respect to 18 areas related to research goals, outcomes, processes or team-building, about half of them indicated a positive impact for all but 1 of the 18 areas (Table V.3).  The most broadly perceived benefit, indicated by 96 percent, was the freedom to “pursue more innovative ideas.”  Moreover, 92 percent also felt that additional support would facilitate greater collaboration among researchers, as well as help them achieve their research objectives within a specified time frame.

About half (47%) of the PIs took the opportunity to describe in their own words the impact of receiving an award that would provide them with what they needed to accomplish their research and educational goals (Table V.4).  Thirty-five percent of the responses described improvements and increases in the applications and outcomes of their research and educational activities; among these 9 percent were a description of a specific scientific advance that would


TABLE V.3

 

POSITIVE IMPACT OF ADDITIONAL AWARD FUNDING AND/OR DURATION

(Percentages)

 

 

 

 

Positive Impact

 

 

 

Total

(N = 4,989)

Gave Funding/

Duration Amount

(N = 4,489)

1.

Pursue innovative ideas

         87

             96

2.

Collaborate with researchers in your area of research

         83

             92

3.

Achieve the research objectives within the specified time

 

         83

 

             92

4.

Collaborate with researchers in different areas of research

 

         76

 

             84

5.

Obtain quality personnel

         76

             85

6.

Pursue high-risk ideas

         76

             85

7.

Disseminate research findings

         74

             82

8.

Integrate research activity into your teaching and training

 

         73

 

             81

9.

Establish mentoring or other research-based education activities

 

         71

 

             79

10.

Broaden participation of under-represented groups in the research activity

 

         62

 

             69

11.

Develop partnerships with industry, other educational institutions, or national laboratories

 

         62

 

             69

12.

Develop instrumentation or other enhancements for the research and education infrastructure

 

         61

 

             67

13.

Access state-of-the-art equipment

         60

             67

14.

Improve public understanding of the project

         58

             64

15.

Obtain other funding

         54

             60

16.

Access facilities

         49

             54

17.

Nurture connections between research activity and its potential for:  health benefits, economic benefits, and national security benefits

 

 

         48

 

 

             54

18.

Develop programs with K‑12 teachers and/or students

 

         32

 

             35


TABLE V.4

IMPACT OF ADDITIONAL FUNDING AND/OR DURATION

(Percentages)

 

 

 

Response Categories

Total Responses

(3,857)

Improved and increased applications and outcomes

                        35

 

Enhanced goals and objectives

 

                        23

 

Improved team building with staff and students

 

                        21

 

Process for funding improved

 

                        11

 

Research tools enhanced

 

                          6

 

No impact

 

                          3

 

Other comments

 

                          2

 


result from having additional resources.  Twenty-three percent of all responses were related to enhanced goals and objectives; these included 13 percent who described the ability to explore new, high-risk ideas and the chance to be more innovative.  Comments on improved team-building (21%) included 13 percent who specifically mentioned enhanced opportunities for students.

When asked how they would spend additional funding for securing the resources needed to conduct high-quality research—personnel, equipment, travel, the size and quality of experiments or tests—the PIs indicated that they would be most likely to spend additional resources on hiring students.  Among the 16 resources listed in Table V.5, three of the top five choices indicated that PIs would spend additional funding on increasing the number and/or time of graduate students, undergraduates, or postdoctoral associates—in that order of preference.  More specifically, 78 percent said they would “very likely” increase the number of graduate students hired or the duration of their students’ work period.  Increasing the quality and number of experiments or tests rounds out the other two top choices.

PIs who indicated that additional funding would benefit their work were asked the extent to which they felt this additional funding would affect the quality, duration, or number of experiments they performed or their ability to recruit the highly qualified labor needed to conduct high-quality research.  As shown in Table V.6, more than half said that additional funding would help them in their efforts to recruit postdoctoral and graduate students “a great deal.”  Close behind, nearly half thought the additional support would improve their research by allowing for a substantial increase in the number of experiments, tests, or subjects.

 


TABLE V.5

 

ADDITIONAL NSF AWARD FUNDING AND/OR DURATION:

POTENTIAL INCREASES IN RESEARCH AND EDUCATIONAL ACTIVITIES

(Percentages)

 

 

 

 

Very Likely

 

 

Total

(N = 4,989)

Gave Funding/

Duration Amount

(N = 4,489)

1.

The number and/or months of graduate students

         70

              78

2.

The number of experiments, tests, subjects

         49

              54

3.

The number and/or months of undergraduate students

         45

              50

4.

The number and/or months of post doctoral associates

 

         43

 

              48

5.

The quality of the experiments or tests

         36

              40

6.

The number of equipment purchases

         33

              36

7.

The size of the experiments or tests

         30

              33

8.

The number and/or months of senior personnel

         29

              32

9.

The number of trips

         29

              32

10.

The quality of equipment purchases

         28

              31

11.

Participant support

         21

              23

12.

Computer/publication costs

         16

              18

13.

The number and/or months of technicians

         15

              17

14.

The number and/or months of programmers

           7

                8

15.

Consultant services

           6

                6

16.

The cost per trip

           3

                4

 


TABLE V.6

 

ADDITIONAL NSF AWARD FUNDING AND/OR DURATION:

INCREASED ABILITY FOR RESEARCH AND EDUCATIONAL ACTIVITIES

(Percentages)

 

 

 

 

Increased a Great Deal

 

 

Total

(N = 4,989)

Gave Funding/

Duration Amount

(N = 4,489)

1.

Recruit graduate students

            56

            62

2.

Recruit post-doctoral associates

            47

            52

3.

Conduct more experiments, tests or subjects

            42

            47

4.

Have higher-quality experiments or tests

            31

            34

5.

Provide adequate support for a graduate student to shorten time to degree

 

            29

 

            32

6.

Recruit undergraduate students

            27

            30

7.

Provide stability for technicians

            17

            19

8.

Duration of experiments

            17

            19

9.

Provide stability for programmers

              8

              8

 


C.   THE POTENTIAL BENEFITS OF ADDITIONAL NSF FUNDING OR DURATION ON RESEARCH IN GENERAL

       As Table V.7 illustrates, these PIs, in addition to the individual benefits they felt their own research would gain, felt that increased NSF funding or duration had broader implications for scientific research.  Overall, more than 60 percent said increased NSF funding and duration would “very likely” “widen the focus” of research in their field, while also attracting “more” and “better graduate students.”  Seventy percent agreed that additional support would “very likely” “decrease interruptions in funding,” and almost half (46 percent) thought that such support would very likely increase the number of proposals containing innovative ideas.

d.   THE POTENTIAL IMPACT OF ADDITIONAL NSF FUNDING OR DURATION ON EDUCATIONAL ACTIVITIES

As described in Chapter II, for their current research projects, PIs support an average of two undergraduates, four graduate students, and one postdoctoral fellow (Figure II.1).  As noted, PIs reported being “very likely” to increase the number or duration of these students and postdoctoral fellows if NSF provided additional grant support.  Moreover, as stated earlier, among those who were awarded an NSF grant in FY2001, the average difference between the level of funding requested and the level awarded was $33,000 a year.  At many schools, being able to provide this difference could itself mean supporting an additional graduate student. In addition to improving PI productivity, hiring additional students has the added benefit of supporting highly qualified students whose talents are lost when they must leave the sciences to support themselves in alternative careers.

From the PIs’ perspective, it is not only the additional funding and duration that is attractive, but  also NSF’s unique contribution to the development of students:


TABLE V.7

 

ADDITIONAL NSF AWARD FUNDING AND/OR DURATION:

POTENTIAL FOR CHANGES IN FIELD OF RESEARCH

(Percentages)

N = 4,989

 

 

 

 

Very Likely

1.

Decrease interruptions in funding

               70

2.

Attract more graduate students

               65

3.

Widen the focus of the research in your field

               63

4.

Attract better graduate students

               62

5.

Increase the number of proposals to NSF with innovative ideas

 

               46

6.

Increase the number of proposals to NSF with high-risk ideas

               37

7.

Attract more established researchers to apply for NSF funding

 

               37

8.

Improve access to facilities and databases

               36

9.

Decrease the amount of time to answer research questions

               31


       “Clearly more money helps, but it is the TYPE of funding that NSF provides that is unique.  So many other sources (including federal agencies) have a short-term focus on technological or economic deliverables.  More NSF funds opens up a spectrum of possibilities: working on new ideas, allowing students to drive some of the research, building infrastructure and a base from which more funds can be obtained from other sources, more freedom to focus of graduate and undergraduate educational goals rather than "research" goals.  The NSF is different than industry, and needs to stay that way; and it needs to be a bigger part of academic funding.”

 

       “The NSF funding provides the "next generation" of technologies that are needed to take this project to its optimal conclusion.  It is amazingly difficult to get funding for this basic research any other way.  Also, the continuity of NSF multi-year funding is critical to the continuity of my students.  Without it, promising funding for a PhD student can be risky, as many of the other grants can have "lag time" between them.”

 

       “My main concern with NSF is the 3 year cap on funding for grants as I mentioned before.  This 3 year cap is not enough to have a student finish a project.  Even if the project is renewed, then the renewal often does not come in time to support a graduate student’s stipend working on the original project.  As a result, the student has to TA or find a scholarship or just borrow from other grants for obtaining money to finish and graduate.  Consequently, I would be very much in favor of extending grants to 4 years rather than 3.  If this reduces the award amount slightly, so be it.  But I do not think that has to be true since people would just get fewer grants but for longer periods of time.  Since NSF is in the business of education scientists and engineers, I would think that NSF would be most interested in obtaining continuous funding for our hard-working graduate students.  These students often sacrifice personal rewards for the opportunity to do research and we shouldn’t punish them by making them go get more money to finish out their thesis since the NSF grant our out after 3 years.

 

E.    IMPACT OF CHANGES ON INSTITUTIONS

The institutional representatives were asked several questions that gave them an opportunity to describe in their own words what they thought would be the most significant change for their institution if NSF increased either the average duration of the awards or the average award amount.   The greatest percentage of comments on possible changes that would occur if NSF increased the average award duration were about how it would impact the institution’s grant process (35%).  Thirteen percent who noted this change said there would be a decrease in time and effort  (Table V.8).  These comments also noted that there would be an improvement in the quality and efficiency of the research (18%), more stable funding (15%), and there would be staffing changes (15%) which included comments on the positive impact on PIs (6%) and more student involvement (4%). 

The institutional representatives had somewhat similar comments about the significant changes that would occur if NSF increased the award amount. More than half of the responses describing the impact of increasing the average dollar amount described improvements in the quality and quantity of the research (27%) and possible staffing changes (24%) such as increased student involvement (16%) (Table V.9).  The impact on the institution’s grant process (28%) was also noted with 9 percent describing a decrease in the institution’s time and effort and 5 percent speculating that there would be an increase in time and effort.

There were also suggestions on how NSF could reduce the amount of time and resources used by the institution to manage NSF grants.  More than half (55%) of these comments described possible changes in the grant process. Specifically, 16 percent described positive experiences with FastLane and 14 percent offered suggestions for possible FastLane improvements.

F.    ADDITIONAL COMMENTS ABOUT THE NSF GRANT PROCESS

 

       At the end of the questionnaire, the PIs were given an opportunity to write in any additional comments they had about the NSF grant process.  Half of them provided  additional information (Table V.11).  Since the question was very broad, the PIs commented on a variety of topics.  Responses related to the review process made up 23 percent of these comments.  Two main themes in this category were the observation that the peer review process is satisfactory and


TABLE V.8

INSTITUTION CHANGES IF NSF INCREASED THE

AVERAGE DURATION PER GRANT

 

 

 

        Total

    Responses*

 

 

Grant Process

     General comments  (18)

     Increase time and effort  (4)

     Decrease time and effort (13)

 

            35

 

 

 

 

Research Changes

     Improved quality/efficiency

 

 

            18

 

 

Award Duration Improvements

     More stable funding; fewer no-cost extensions

 

            15

 

 

Staffing Changes

     General comments (5)

     More student involvement (4)

     Positive PI impact (6)

            15

 

 

 

 

 

No Changes

              6

 

No Comment/No Response

            12

 

TOTAL

          101

 

*Weighted number of responses is 839.

 


TABLE V.9

INSTITUTION CHANGES IF NSF INCREASED THE

AVERAGE DOLLAR AWARD PER GRANT

 

 

 

Total

Responses*

 

Grant Process

     General comments (7)

     Increase time and effort (5)

     Decrease time and effort (9)

     Increase number of applications (7)

            28

 

 

 

 

 

Research Changes

     More conducted, improved quality

 

            27

 

 

Staffing Changes

     General comments (4)

     More student involvement (16)

     More faculty involvement (4)

            24

 

 

 

 

Award Amount

     More stable funding; more budget flexibility

 

 

            11

 

No Changes

              7

No Comment/No Response

              4

TOTAL

          101

 

*Weighted number of responses is 967.

 


TABLE V.10

SUGGESTIONS FOR NSF CHANGES TO REDUCE

INSTITUTION TIME AND RESOURCES

 

 

 

Total

Responses*

Grant Process

            55

     General comments (21)

     Reduce budget revisions, requests (4)

     Comments on FastLane improvements (14)

     Positive experience with FastLane (16)

 

General Comments on Award Amount and

     Duration

              5

 

No Suggestions

              8

 

Experience with NSF Staff

 

No Comments

              4

 

            27

 

TOTAL

            99

 

*Weighted number of responses is 681.

 


TABLE V.11

PRINCIPAL INVESTIGATORS:  OTHER COMMENTS ON THE NSF GRANT PROCESS

(Percentages)

 

 

 

Response Categories

Total Responses

(5,056)

Review process

                        23

 

General award comments

 

                        17

 

Proposal process

 

                        12

 

Award size

 

                        11

 

Award administration

 

                        10

 

Award duration

 

                          8

 

Overall satisfaction with NSF grant

 

                          8

 

Comments on the questionnaire

 

                          3

 

Communication between NSF and principal investigators

 

                          2

 

Other comments

 

                          5

 

TOTAL

 

                        99


award decisions were risk adverse.  Seventeen percent of the comments were general remarks about the grant awards, with many PIs singling out certain aspects of NSF grants—for example, not being able to do the research without NSF, and NSF grants provide more flexibility than other funding organizations. These general comments also covered the proposal process (12%), the award size (11%), the administration of the award (10%), and the award duration (8%).

The institutional representatives were given the opportunity to write in “any other comments they had about their institution’s experiences with the NSF grant process.”  Twenty-three percent of the comments were about FastLane and the use of technology in the grant process (Table V.12); among these comments, 15 percent described positive experiences with FastLane.  There were also comments about working with NSF staff (11%) and the level of effort involved in the grant process (16%).


TABLE V.12

INSTITUTIONS:  OTHER COMMENTS ON THE NSF GRANT PROCESS

 

 

 

Total

Responses*

NSF Staff

            11

     - Positive experiences (9)

     - Other comments (2)

 

Technology/Fast Lane

            23

Level of Effort for Grant Process

            16

 

Award Duration and Amount

              6

 

Other Comments

              3

 

No Comments

            42

 

TOTAL

          101

 

*Weighted number of responses is 666.



[1]“Document the efficiency of the research process.  With the assistance of U.S. academic research institutions, NSF should develop efficiency measures of the research process and determine what is the right grant size for the myriad types of research the agency funds.  These metrics and grant size determinations should be developed for consideration in the FY 2003 Budget”.  June 7, 2001, letter from OMB to Dr. Rita Colwell.

[2] The use of means to describe values was decided after careful consideration.  As a general comparison, the mean uses more information than the median since the exact scores are used in the computation, while the median uses the relative position of the scores.  However, the mean is effected by extreme values while the median is generally unaffected by extremes. For some items there are calculated means which are described in detail in Appendix A.

Return to the Table of Contents