Overview 
Survey Design Survey Quality Measures Trend Data Availability of Data
1. Overview (2011 survey cycle) 
a. Purpose
The 2011 Higher Education R&D (HERD) Survey, successor to the Survey of Research and Development Expenditures at Universities and Colleges, collected information from 912 research-performing academic institutions. The survey is the primary source of information on research and development expenditures at higher education institutions in the United States and outlying areas. The HERD survey's predecessor was conducted annually from FY 1972 through FY 2009 and collected information on R&D expenditures by academic field as well as by source of funds. The HERD survey continues to capture core information on R&D expenditures by source of funding and field to retain the historical time series. However, the survey has been revised and expanded to include more detail. The survey was revised to include R&D within non-science and engineering (non-S&E) fields in the totals reported throughout the survey. Previously, non-S&E R&D was reported but not included in the overall totals. The scope of R&D was also broadened to include expenditures on clinical trials and research training grants.
b. Respondents
The HERD survey is a voluntary establishment survey completed by designated contacts at U.S. universities and colleges. The majority of respondents for higher education institutions work in one of the following institutional offices: accounting, grants and contracts, controller, financial, institutional research, or sponsored programs.
c. Key variables
The HERD survey requests data from institutions on their R&D in the following categories. R&D is defined as creative work conducted systematically to increase the stock of knowledge (research) and to use this knowledge to devise new applications (development).
- R&D expenditures by field and source of funds (federal government, state and local government, business, nonprofit, institutional, and other)
- R&D expenditures funded from foreign sources
- R&D expenditures within medical schools
- Clinical trial R&D expenditures (phases I–III)
- R&D expenditures by character of work (basic research, applied research, and development)
- Total and federally funded R&D expenditures passed through to subrecipients or received as a subrecipient
- Federally funded R&D expenditures by field and federal agency
- Federally funded R&D expenditures funded by the American Recovery and Reinvestment Act
- R&D expenditures by cost categories (for example, salaries, software, equipment, indirect costs)
- Total and federally funded R&D equipment expenditures by field
- Headcount of R&D principal investigators and all other R&D personnel
- Headcount of postdocs working on R&D
- Institutional characteristics (highest degree granted, historically black college or university [HBCU], public or private control)
- Geographic location (within the United States)
2. Survey Design
a. Target population and sample frame
Since FY 2010, the target population for the HERD survey has included nonprofit postsecondary institutions with bachelor's or higher degree programs in any field and R&D expenditures of at least $150,000 in any field.
Prior to FY 2010, the population included only institutions with R&D spending and degree programs in science and engineering (S&E) fields. Institutions that performed R&D only in non-S&E fields were excluded from the population. Several other minor changes to the target population and sampling frame were made prior to FY 2004. A complete listing of these changes can be found within the methodology report, available on request from the project officer (see section 5c, below).
Also beginning with FY 2010, each campus headed by a campus-level president or chancellor now completes a separate survey rather than combining its response with other campuses in a university system. As a result of this step, the overall number of academic institutions in the population increased from 711 in FY 2009 to 742 in FY 2010.
The frame for the FY 2011 HERD survey was the list of institutions considered in scope for the FY 2010 survey as well as those who reported over $150,000 on the FY 2010 HERD Short Form (see 3b). Institutions that did not respond to the FY 2010 Short Form and a subset of respondents that reported less than $150,000 on the Short Form were also contacted to determine whether they met the $150,000 expenditure criterion for FY 2011. As a result of the additional institutions found during the Short Form, the FY 2011 population increased to 912 institutions.
b. Sample design
The FY 1997 survey was the last to be conducted as a sample survey. Since 1998 the survey has been a census of all known eligible universities and colleges.
c. Data collection techniques
The FY 2011 survey was conducted by ICF International under contract to the National Center for Science and Engineering Statistics (NCSES). Respondents could choose to print an Adobe Portable Document Format (PDF) questionnaire from the Web and submit a paper survey, upload an Excel version of the questionnaire, or use the Web-based data collection system to respond to the survey.
Respondents were sent survey information and a user ID and password via e-mail. The Web-based system contained instructions and built-in help designed to replicate the printed forms. Each institution's Web survey included comparable data from the 2 preceding years for arithmetical and trend check purposes. Large increases or decreases in expenditures compared with the preceding years were flagged, and respondents were asked to explain and reconcile these differences.
Data collection began in December 2011, with the official survey due date announced as 24 February 2012. Because the desired response rate had not been reached by the survey deadline, a series of reminder e-mail messages and phone calls were used to encourage response from all institutions between March and May of 2012, with priority placed on eliciting responses from the previous year's top 100 institutions in total R&D expenditures. The survey was closed for new submissions by 8 May 2012, and data verification and follow-up took place in May and June of 2012.
d. Estimation techniques
This survey is a census. Imputation was performed for nonresponding institutions in order to make population estimates (see section 3c for more information). Estimates are computed by summing reported and imputed values for institutions with the appropriate characteristics.
3. Survey Quality Measures
a. Sampling variability
Because the FY 2011 survey was a universe survey distributed to all institutions in the universe, there was no sampling error.
b. Coverage
Coverage of the large research institutions is excellent, because they are easily identified using the NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions. However, institutions with smaller amounts of R&D expenditures have been more difficult to find as they often do not receive federal funding for science and engineering R&D.
As part of the expanded HERD survey collection, a special effort was undertaken in 2010 to screen 1,715 four-year and higher postsecondary institutions not currently included in the survey to identify additional institutions meeting the $150,000 threshold. These institutions were given detailed instructions regarding what to include as R&D expenditures and asked to respond to question 1 of the HERD survey, total R&D expenditures by source of funding. This HERD Short Form achieved a 64.4% response rate and identified 187 additional institutions over the $150,000 threshold. These institutions were asked to respond to the full survey for FY 2011. The total R&D expenditures reported by these institutions in FY 2010 was $249 million, or 0.42% of the $61.2 billion reported by the FY 2010 HERD survey respondents. More details on this effort are included in the methodology report and technical notes for the FY 2010 survey.
c. Nonresponse
(1) Unit nonresponse—37 universities and colleges did not respond in FY 2011 out of a total of 912 eligible institutions, for a nonresponse rate of 4.1%.
Information for nonresponding institutions was imputed. Imputation was performed using prior years' figures derived from the data of respondent institutions with similar characteristics, including highest degree granted and type of institutional control (public or private). Imputed values account for less than 1% of the total R&D expenditures included in the survey. When an institution failed to report in a given year but reported in the next year, the data for the missing year were re-imputed to reflect the new information if it appeared likely that the new information would have a significant impact on the original estimate.
(2) Item nonresponse—The item nonresponse rates varied from 0% for Question 1, total R&D expenditures by source of funds, to 17.3% for Question 6, R&D expenditures by character of work (basic research, applied research, and development). The other question with a nonresponse rate over 10% was Question 17, headcount of R&D postdocs (13.0%).
Missing information for nonresponse was imputed for all questions except ARRA-funded expenditures (Question 11), headcount of R&D personnel (Question 16) and headcount of postdocs (Question 17). Imputations were based on average increases, decreases, or proportions derived from data of respondent institutions with similar characteristics, including highest degree granted and type of institutional control (public or private), and total R&D expenditures.
d. Measurement
The most likely source of measurement error is institutional records containing categories different from those requested on the survey. For example, institutions are asked to report all R&D expenditures by field. The NSF-designed fields do not always translate to an institution's departmental structure, and adjustments must be made by the institution in order to complete the survey. A crosswalk between the NSF fields of R&D and the National Center for Education Statistics Classification of Instructional Programs (CIP) codes is provided with the survey in order to mitigate this source of measurement error.
Another example is NSF's category of institutionally financed research. The survey requests that the institutions report discretionary internal funds used for research. NSF discovered through debriefings conducted at the conclusion of the FY 2010 survey that there are varying definitions of what is considered institutionally financed organized research (included on HERD) versus departmental research (excluded on HERD). Some institutions classify all expenditures from separate accounts designated for research as organized research, whereas others only include internal R&D projects that are competitively awarded and have detailed budgets. A workshop was held in the summer of 2012 in order to discuss these differences in definitional interpretations. Based on the findings from the workshop, the FY 2012 survey questionnaire has been modified to clarify that all expenditures designated for research can be included within this category.
The reporting of unreimbursed indirect costs is another known source of measurement error. The survey requests that the total amount of indirect costs associated with a research grant or contract be calculated and reported, including costs that were not reimbursed by the external funding source. Calculating the unreimbursed indirect cost is done by multiplying the institution's negotiated indirect cost rate by the corresponding base and then subtracting the actual indirect cost recovery, preferably on a project by project basis. Previous site visits have determined that some institutions are not able to report the unreimbursed indirect cost amount.
4. Trend Data
Annual data are available for FY 1972–2011, with the exception of FY 1978. That year's survey covered a different population and used different questions than preceding or subsequent surveys and is therefore not comparable to other years. When the review for consistency between each year's data and submissions in prior years reveals discrepancies, it is sometimes necessary to modify prior years' data. This is especially likely to affect trends for certain institutions that fail to report every year, because current year data are used to impute prior year data. Individuals wishing to analyze trends other than those published in NCSES's most recent publication are encouraged to contact the project officer listed in section 5c for more information about comparability of data over time.
5. Availability of Data
a. Publications
The data from this survey are published annually, both in a summary InfoBrief and in full as detailed statistical tables in the series Higher Education Research and Development, or, for FY 2009 and earlier years, its predecessor series, Academic Research and Development Expenditures. Data are available by fiscal year on the NCSES website at http://www.nsf.gov/statistics/herd/. Data for major data elements are available starting in 1972. Information from the survey is also included in Science and Engineering Indicators, National Patterns of Research and Development Resources, Science and Engineering State Profiles, and Academic Institutional Profiles.
b. Electronic access
All publications related to this survey are available on the NCSES website and in the NCSES Integrated Science and Engineering Resources Data System, WebCASPAR. Selected aggregate data are provided in public use data files upon request.
c. Contact for more information
Additional information about this survey may be obtained by contacting:
Ronda Britt
Research and Development Statistics
Program
National Center for Science and Engineering Statistics
National Science Foundation
4201 Wilson Boulevard, Suite 965
Arlington, VA 22230
Phone: (703) 292-7765
E-mail: rbritt@nsf.gov
Last updated: November 26, 2012
