NCSES Home > Survey Descriptions > Survey of Research and Development Expenditures at Universities and Colleges

Higher Education Research and Development (HERD) Survey.

Overview  Survey Design  Survey Quality Measures  Trend Data  Availability of Data

1. Overview (2009 survey cycle)

a. Purpose

The National Science Foundation (NSF) Survey of Research and Development Expenditures at Universities and Colleges (academic R&D expenditures survey) is the primary source of information on separately budgeted research and development (R&D) expenditures by academic institutions in the United States and outlying areas. Conducted annually since fiscal year (FY) 1972, the survey collects information on R&D expenditures by academic field as well as by source of funds. The results of the survey are primarily used to assess trends in R&D expenditures across the fields of science and engineering (S&E). This information is vital for decision making by federal, state, and academic planners regarding future R&D funding priorities.

b. Respondents

The academic R&D expenditures survey is an establishment survey completed by institutional coordinators at U.S. universities and colleges. The majority of respondents for academic institutions work in one of the following institutional offices: accounting, grants and contracts, controller, financial, institutional research, or sponsored programs.

c. Key variables

The academic R&D expenditures survey requests data from institutions on their R&D expenditures in the following categories:

• R&D expenditures by source of funds (federal, state and local, industry, institutional, or other)
• R&D expenditures by character of work (basic research versus applied research and development)
• R&D expenditures passed through to subrecipients
• R&D expenditures received as a subrecipient
• Total and federally funded R&D expenditures by S&E field
• Total and federally funded R&D expenditures by non-S&E field (optional prior to FY 2003)
• Total and federally funded R&D equipment expenditures by S&E field
• Federally funded expenditures by S&E field and federal agency (optional prior to FY 2003)

Other key variables include:

• Academic institution
• Institutional characteristics (highest degree granted, historically black college or university (HBCU), public or private control)
• Geographic location (within the United States)

2. Survey Design

a. Target population and sample frame

The target population includes institutions that have bachelor's or higher programs in S&E and annually perform at least $150,000 in separately budgeted S&E R&D. Prior to FY 2004, all institutions that had doctoral programs in S&E and all HBCUs were included in the survey population, regardless of the level of R&D.

Since FY 1998, the survey has covered all eligible institutions. The frame for the population survey was constructed through the screening of institutions identified as receiving federal funds by the NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, from institutions identified in the NSF-National Institutes of Health Survey of Graduate Students and Postdoctorates in Science and Engineering as having master's or doctoral programs in S&E fields, and institutions listed in the Higher Education Directory. All institutions on the compiled list that are not known to be within the scope of the survey are contacted by phone to determine whether they are in scope.

b. Sample design

The FY 1997 survey was the last to be conducted as a sample survey. Since 1998 the survey has been a census of all eligible universities and colleges.

c. Data collection techniques

The FY 2009 survey was conducted by ICF Macro under contract to the Division of Science Resources Statistics (SRS). Respondents could choose to print an Adobe Portable Document Format questionnaire from the Web and submit a paper survey or use the Web-based data collection system to respond to the survey electronically. Of those who responded to the survey, all but one responded via the Web system.

Forty surveyed institutions did not complete the academic R&D expenditures survey instrument, but instead responded to a pilot version of a new survey instrument, the Higher Education Research and Development (HERD) Survey. A copy of the instrument is available at http://www.nsf.gov/statistics/question.cfm#12. The HERD survey results from an effort by NSF to redesign the current survey. Much of the data requested as part of the HERD survey was the same as requested on the academic R&D expenditures survey. Expenditures collected as part of the HERD pilot are included in the detailed statistical tables (DSTs). When necessary, pilot institutions were asked to provide additional information in order to have comparable data for the DSTs. Institutions in the HERD pilot were not included in the data collection procedures that follow. For more information about the HERD survey pilot and data collection procedures, contact the NSF project manager listed below.

Respondents were sent survey information and a user ID and password via e-mail. The Web-based system contained instructions and built-in help designed to replicate the printed forms. Each institution's Web survey included data from the 2 preceding years for arithmetical and trend check purposes. Large increases or decreases in expenditures compared with the preceding years were automatically flagged by the system, and respondents were asked to explain and reconcile these differences before submitting the survey.

Data collection began on November 9, 2009, with the official survey due date announced as February 4, 2010. Because the desired response rate had not been reached by the survey deadline, a series of reminder e-mail messages and phone calls were used to encourage response from all institutions between February and April of 2010, with priority placed on eliciting responses from the previous year's top 100 institutions in total R&D expenditures. The official survey closeout occurred on May 6, 2010.

d. Estimation techniques

This survey is a census. Imputation was performed for nonresponding institutions in order to make population estimates (see section 3c for more information). No weighting techniques were used.

3. Survey Quality Measures

a. Sampling variability

Since the FY 2009 survey was a universe survey distributed to all institutions in the universe, there was no sampling error.

b. Coverage

Coverage at the institutional level is excellent, given that it is relatively easy to identify schools that grant bachelor's degrees. However, coverage-related issues could arise through incorrect classification of institutions as granting S&E degrees. While not a coverage problem per se, one limitation of the data is the fact that small institutions (those with less than $150,000 in S&E R&D expenditures) are excluded from the population. NSF estimates that these institutions account for less than 1 percent of the Nation's academic S&E R&D expenditures.

c. Nonresponse

(1) Unit nonresponse—17 universities and colleges did not respond in FY 2009 out of a total of 711 eligible institutions, for a nonresponse rate of 2.4 percent.

Information for nonresponding institutions was imputed. Imputation was performed using prior years' figures derived from the data of respondent institutions with similar characteristics, including highest degree granted and type of institutional control (public or private). Imputed values account for less than 1 percent of the total R&D expenditures included in the survey. When an institution failed to report in a given year but reported in the next year, the data for the missing year were re-imputed to reflect the new information if it appeared likely that the new information would have a significant impact on the original estimate.

(2) Item nonresponse—The item nonresponse rates varied from 0 percent for Items 1 and 2, total and federally financed R&D expenditures by source of funds and field of science, to 3.7 percent for Items 1a and 1b, total and federally financed R&D expenditures passed through and received as a subrecipient.

Missing information for nonresponse on Items 1, 2, or 3 was also imputed. Imputations were based on figures derived from data of respondent institutions with similar characteristics, including highest degree granted and type of institutional control (public or private).

d. Measurement

The most likely source of measurement error is institutional records containing categories different from those requested on the survey. For example, institutions are asked to report all R&D expenditures by field of science. The NSF-designed fields do not always translate to an institution's departmental structure, and adjustments must be made by the institution in order to complete the survey. A crosswalk between the NSF fields of science and the National Center for Education Statistics Classification of Instructional Programs (CIP) codes is provided with the survey in order to mitigate this source of measurement error.

Another example is NSF's category of institutionally financed organized research. The survey requests that the institutions report discretionary internal funds used for separately budgeted organized research. Most major academic institutions are able to track expenditures for these projects; however, several large institutions are not able to track institution funding of R&D due to limitations with their current accounting systems and thus report zero institution funding. This contributes to an unknown amount of measurement error.

The reporting of unreimbursed indirect costs is another known source of measurement error. The survey requests that the total amount of indirect costs associated with a research grant or contract be calculated and reported, including costs that were not reimbursed by the external funding source. Calculating the unreimbursed indirect cost is done by multiplying the institution's negotiated indirect cost rate by the corresponding base and then subtracting the actual indirect cost recovery, preferably on a project by project basis. Previous site visits have determined that some institutions are not able to report the unreimbursed indirect cost amount.

In 2007, NSF began a large scale project to redesign and update the survey, which included the aforementioned pilot test of the new HERD survey in the fall of 2009. One of the goals of this redesign was to minimize the sources of measurement error for the survey. As part of the redesign effort, NSF held data user workshops and expert panel meetings, worked with accounting and survey methodology experts, and visited more than 40 institutions to receive input on possible changes to the survey. NSF also held detailed debriefings with each of the respondents from the 40 institutions who participated in the pilot test in order to further refine the survey definitions and instructions.

4. Trend Data

Annual data are available for FY 1972–FY 2009, with the exception of FY 1978. That year's survey covered a different population and used different questions than preceding or subsequent surveys and is therefore not comparable to other years. When the review for consistency between each year's data and submissions in prior years reveals discrepancies, it is sometimes necessary to modify prior years' data. This is especially likely to affect trends for certain institutions that fail to report every year, since current year data are used to impute prior year data. Individuals wishing to analyze trends other than those published in SRS's most recent publication are encouraged to contact the project officer listed in section 5c for more information about comparability of data over time.

5. Availability of Data

a. Publications

The data from this survey are summarized in an InfoBrief and published annually in detailed statistical tables in the series Academic Research and Development Expenditures, available by fiscal year on the SRS Web site. Data for major data elements are available starting in 1972. Information from the survey is also included in Science and Engineering Indicators, National Patterns of Research and Development Resources, Science and Engineering State Profiles, and Academic Institutional Profiles.

b. Electronic access

All data from this survey are available on the SRS Web site and in WebCASPAR. Selected aggregate data are provided in public use data files upon request.

c. Contact for more information

Additional information about this survey can be obtained by contacting:

Ronda Britt
Project Officer
Research and Development Statistics Program
Division of Science Resources Statistics
National Science Foundation
2415 Eisenhower Avenue, Suite W14200
Alexandria, VA 22314
Phone: (703) 292-7765
E-mail: rbritt@nsf.gov