Impact of Proposal and Award Management Mechanisms Working Group (IPAMM) - 2007 Proposer Survey
Initial Survey Results
NSF is very grateful to the thousands of individuals that participated in the survey. Your responses provide valuable feedback that will help us fulfill our goal of supporting excellence in science and engineering research and education through good stewardship, as laid out in our Strategic Plan (https://www.nsf.gov/pubs/2006/nsf0648/NSF-06-48.pdf).
Information from the survey has already helped in the development of recommendations to help promote award balance and flexibility, as detailed in the Impact of Proposal and Award Management Mechanisms report, available here. NSF is also using your feedback to help improve the quality and transparency of the merit review process, including a new web page that explains the process from beginning to end. That web page will be launching soon, and will be accessible from the NSF home page.
Given the extensive nature of the survey data and the thousands of open text responses you provided, detailed analyses of the data are ongoing. Only the aggregated results of all analyses will be provided to NSF, to preserve the respondents' anonymity. Updates on the results of these analyses will be posted on this page in the coming months, as they are completed.
In response to concerns that declining funding rates may be negatively impacting the academic research community and the nation’s science and engineering capacity, NSF established the Impact of Proposal and Award Management Mechanisms (IPAMM) Working Group in March 2006. IPAMM was charged to perform a detailed study of the trends, impacts, and causal factors associated with the recent declines in proposal funding rates and the simultaneous growth in proposal submission rates. As part of this effort, IPAMM and the consulting firm Booz Allen Hamilton prepared a web-based customer satisfaction survey to request feedback from principal investigators on proposal-related topics. The survey focused on the proposal submission process, factors that influence decisions to submit research proposals, and perceptions of funding rates within NSF and other organizations.
The survey instrument was designed for online administration, and included identity protection to ensure that only intended respondents participated. Certain questions in the online version of the instrument were designed to branch or skip, depending on answers provided by respondents. The survey was submitted to OMB for approval under NSF’s generic clearance to assess customer satisfaction of its proposers; OMB granted approval on January 24, 2007 (OMB control number 3145-0157).
After receiving OMB approval, a preliminary email was sent from Dr. Kathie L. Olsen, Deputy Director, on January 25, 2007 to the survey population (all principal investigators who submitted research proposals during fiscal years 2004, 2005 or 2006) introducing the survey and encouraging recipients to participate. The survey population then received an email from Booz Allen on January 29, 2007, asking them to respond, providing them with the web address, and assuring them that their responses would be anonymous to the NSF. The survey remained open until February 16, 2007.
A total of 45,192 PIs submitted research proposals to NSF in FY 2004-2006, and were sent the introductory e-mail from Dr. Olsen. Of these, 43,412 were then sent the survey invitation from Booz Allen (the remaining 1,780 were removed from the list primarily because their e-mail addresses were not functional or, to a much lesser extent, they had requested that their name be removed from the survey list). 24,378 completed the survey, for a response rate of 56%.
NSF provided Booz Allen with demographic information and funding histories for the survey population. Booz Allen generated several variables from this information that were used to characterize the survey population and to analyze the survey data. Several steps were taken to ensure the accuracy and reliability of the data. Prior to sending the survey to the full population, a draft version of the survey instrument was pretested on two focus groups of rotator program officers acting as representatives of the research community. After the survey had been administered, the submitted surveys were checked for completeness—the respondent population included only those that generally answered all of the questions on the survey. Finally, a non-response analysis indicated that the respondent population was representative of the full survey population.
In addition to over 100 multiple choice questions in the survey, there were three open text response questions that generated thousands of responses. 7,862 respondents provided comments for the first open text question (“For what other reasons, if any, do you submit research proposals?”), which followed a series of questions asking the importance of various factors in motivating the respondents to submit proposals to any funding source. At the end of the survey, there were two open text questions; 17,951 responded to the first (“Please describe the single most important thing NSF could do to improve its proposal and review process”), and 8,375 responded to the second (“Please note any other comments you may have”).
For further information concerning the IPAMM working group or the 2007 Proposer Survey, contact:
Dr. Joanne Tornow
Office of the Director
National Science Foundation