Higher Education Research and Development: Fiscal Year 2010
Appendix A. Technical Notes
The National Science Foundation (NSF) Higher Education Research and Development (HERD) Survey is the primary source of information on separately budgeted research and development expenditures within higher education institutions in the United States and outlying areas. This information is vital for federal, state, and academic planners who make decisions on future R&D funding priorities. The HERD Survey replaces a previous annual collection, the Survey of Research and Development Expenditures at Universities and Colleges (Academic R&D Expenditures Survey), which was conducted from FY 1972 through FY 2009. The revised survey continues to collect information on R&D expenditures, by field of research and source of funds, but additional questions about types of research, cost categories, and R&D performers have been included in the new survey. Survey questions now request data across all academic disciplines, not just science and engineering (S&E) disciplines. Unless otherwise noted, expenditures analyzed in this report refer to R&D activities across all fields of R&D.
Scope of the Survey
The FY 2010 HERD Survey was sent to 747 institutions of higher education in the United States, Guam, Puerto Rico, and the U.S. Virgin Islands. These institutions granted a bachelor's degree or higher, expended at least $150,000 in separately budgeted R&D in FY 2010, and were geographically separate campuses headed by a president or chancellor.
The survey population was reviewed before data collection began to ensure that each institutional classification was accurate. Characteristics of the schools were reviewed before and during the course of the survey to determine whether changes had occurred (e.g., highest degree granted; school openings, closings, or mergers).
After data collection closed, institutions were reviewed to verify that only those reporting at least $150,000 in separately budgeted R&D were included in the population. Of the 747 institutions surveyed, 5 reported total R&D expenditures of less than $150,000. These 5 institutions were excluded from the population, and their data are not included in the FY 2010 detailed statistical tables (DSTs). The total and federally financed R&D expenditures for these 5 institutions are listed in table A-1.
FY 2010 Survey Frame Design
The HERD Survey is a census of the full population of eligible academic institutions. The population of the Academic R&D Expenditures Survey, the predecessor to the HERD Survey, included all institutions that granted a bachelor's degree or higher in S&E fields and had at least $150,000 in separately budgeted R&D expenditures during the targeted fiscal year. These had been the population criteria since the FY 2004 collection. As part of the survey redesign, the survey population was revised to include institutions that granted a bachelor's degree or higher in any field of study. It was further required that, within a university system, each campus headed by a campus-level president or chancellor complete a separate survey rather than combining the response with other campuses in the university system (see appendix C for additional information about this change and for a list of universities now reporting as multiple campuses). As a result of this step, the overall number of academic institutions in the population increased from 711 institutions in FY 2009 to 742 institutions in FY 2010 (table A-2).
In past years, prior to beginning the new survey collection, NSF contacted institutions that met the degree-granting criterion but were not in the previous survey population to determine whether they met the $150,000 expenditure criterion. For the FY 2010 survey, the review of the population of 4-year degree-granting institutions was expanded to collect actual expenditures from each institution and was conducted concurrently to the HERD Survey collection (see appendix D for a description of this population review survey). For this reason, the population of the FY 2010 HERD Survey comprised only institutions that were part of the FY 2009 Academic R&D Expenditures Survey.
The HERD Survey continues to collect much of the same data requested as part of the Academic R&D Expenditures survey; however, several additional questions have been included in the survey, and some of the definitions applicable to the previous survey have been revised. The following section describes the content of the FY 2010 HERD Survey instrument and highlights the differences between the current survey and the Academic R&D Expenditures Survey.
The survey still asks for current fund expenditures for separately budgeted R&D, but total R&D expenditures now include both S&E fields and non-S&E fields such as humanities, education, law, and the arts. Although there was one item in the previous survey concerning non-S&E expenditures, most items only requested information on S&E expenditures. All HERD Survey questions ask about R&D across all fields. Additionally, clinical trials and research training grants are now explicitly included in the definition of R&D.
Analysts should be cautious when examining trend data from FY 2009 to FY 2010. Although many variables are similar across the two surveys, because of changes to the definition of R&D and the inclusion of non-S&E expenditures, some comparisons may be misleading.
Changes to Survey Questions
HERD questions 1, 7–9, 12, and 15 (appendix B) are very similar to items found in the Academic R&D Expenditures Survey; however, all questions were reformatted in some way as part of the survey redesign. For example, in past years respondents were asked to report federal and total expenditures for most questions. On the HERD Survey they are now asked to report federal and nonfederal expenditures, leaving the total to be generated automatically. Questions 2–6, 10, 11, 13, 14, 16, and 17 were not included in the previous survey.
Question 1 is a request for R&D expenditures by source of funds. This question is similar to item 1 from the previous survey, but there are some important differences. In addition to the inclusion of non-S&E expenditures, a separate category has been created for nonprofit organizations. In previous years, these expenditures were reported under the category of "All other sources." A second change is that the "Industry" category has been renamed "Business." A third change is the addition of "Cost sharing" as a separate category under institutional funds.
Question 2 is a request for the amount of R&D expenditures funded by foreign sources.
Question 3 asks respondents to characterize their externally funded R&D expenditures, by type of agreement (i.e., contract vs. grant).
Question 4 requests the amount of R&D expenditures for projects in the institution's medical school, if applicable. For the purposes of this survey, a medical school was defined as a school granting an MD or a DO degree.
Question 5 asks for federally and nonfederally financed R&D expenditures for Phase I, Phase II, and Phase III clinical trials, combined.
Question 5.1 is a request for information on the inclusion of clinical trials in the institution's FY 2009 Academic R&D Expenditures Survey.
Question 6 asks the respondent to characterize federally and nonfederally financed R&D expenditures by character of work: basic research, applied research, or development. In the previous survey, institutions were asked only to report the percentage of S&E R&D expenditures that would be considered basic research. The current survey asks for amounts rather than percentages and was expanded to collect expenditures on applied research and development separately and also to include all fields of R&D.
Question 7 is a request for federally and nonfederally financed expenditures for R&D from funds received by the institution as a subrecipient. This is similar to a question from the previous survey, except that non-S&E R&D expenditures now are included.
Question 8 is a request for federally and nonfederally financed funds for R&D passed through the institution to subrecipients. This is similar to a question from the previous survey, except that non-S&E R&D expenditures now are included.
Question 9 is a request for federally financed expenditures for R&D, by field and federal agency. This is similar to item 2b from the Academic R&D Expenditures Survey, but non-S&E expenditures now are included. Although institutions did report federally financed R&D expenditures in non-S&E fields in the past, they were not asked to report expenditures by agency. When interpreting these data at the detailed discipline level, users should keep in mind that interdisciplinary and multidisciplinary activities are often coded based on the departments of the principal investigators.
Question 10 asks for the agency name and corresponding expenditure amount for the top 10 federal agencies and subagencies included under "Other" federal agencies on question 9. Respondents were not asked to provide details when reporting R&D funded by "Other" agencies in the past.
Question 11 requests the amount of federally financed R&D expenditures funded by the American Recovery and Reinvestment Act of 2009 (ARRA).
Question 12 is a request for nonfederally financed expenditures for separately budgeted R&D activities, by detailed R&D field and source. In past years, institutions have reported nonfederal R&D expenditures, by field, but this is the first year in which they have been asked to report those expenditures, by detailed source (e.g., nonprofit organizations, institutional funds). As with question 9, when interpreting these data at the detailed discipline level, users should keep in mind that fields often are coded based on the departments of the principal investigators.
Question 13 is a request for total expenditures for R&D, by type of cost. The types of costs listed include salaries, software, equipment, pass-throughs, other direct costs, and indirect costs.
Question 14 is a request for the institution's capitalization thresholds for software and equipment.
Question 15 requests the amount of federally and nonfederally financed expenditures that were used for the purchase of R&D equipment. This is similar to item 3 from the Academic R&D Expenditures Survey, but non-S&E expenditures now are included.
Question 16 asks for a headcount of principal investigators and other personnel that were paid from R&D salaries reported in question 13.
Question 17 is a request for the number of personnel reported in question 16 that were categorized as postdocs. For the purposes of this survey, NSF defines a postdoc as someone who (1) holds a recent doctoral degree, generally awarded within the last 5 years; and (2) has a limited-term appointment, generally no more than 5–7 years.
Question 18 is a request for the respondent's contact information, the institution's fiscal year, and the time it took for the institution to complete the survey.
The FY 2010 survey questionnaires were sent by e-mail in December 2010. Respondents could choose to submit an Adobe Portable Document Format questionnaire downloaded from the Web or use a Web-based data collection system to respond to the survey. Every effort was made to maintain close contact with respondents in order to preserve both the consistency and continuity of the resulting data. On receipt, questionnaires were carefully examined for completeness. Computerized facsimiles of the survey data were then prepared for each institution; these compared the current and 2 prior years of data and noted any substantive disparities. Due to the number of new questions included in the FY 2010 survey, there were no prior-year data for many survey items. Respondents were sent personalized e-mail messages asking them to provide any necessary revisions before the final processing and tabulation of data. These e-mail messages included a link to the Higher Education R&D Survey Web-based data collection system, allowing respondents to view and correct their data online.
Respondents were asked to explain significant differences between current-year reporting and established patterns of reporting verified for prior years. They were encouraged to correct prior-year data, if needed. When respondents updated or amended figures from past years, NSF made corresponding changes to trend data in this report and to the underlying microdata database. Similarly, if a respondent institution underwent an organizational change, such as a merger, NSF incorporated the effects of the change into prior years' data.
By the survey's closing date in May 2011, forms had been received from 727 universities and colleges out of the population of 742, a response rate of 98.0%. Responses were received from 99.3% of all doctorate-granting institutions. The R&D expenditures reported by these doctoral institutions constituted 98.3% of the estimated national R&D expenditures for FY 2010. Table A-3 displays a detailed breakdown of response rates, by highest degree granted, and table A-4 displays a breakdown of response rates, by survey item for the academic population.
National Total and Imputation
Missing values from questions that were part of the previous version of the survey—R&D expenditures, by source of funds and R&D field; R&D expenditures passed through to subrecipients; R&D expenditures received as a subrecipient; and R&D equipment expenditures—were estimated based on the previous year's data and the reported data of peer institutions. For the 14 institutions that had not responded by the closing date of the survey and had been included in the FY 2009 population, data for R&D were imputed by applying inflator/deflator factors to the prior years' figures. Imputation factors were derived from the data of responding institutions with similar characteristics, including highest degree granted and type of institutional control (public or private). Data were not imputed for one institution that did not respond. This institution had not responded during previous collections, and, because no previous year's data existed, R&D expenditures could not be imputed. This institution was included in population counts but does not appear in DSTs. Since non-S&E categories were added to several questions this year and the missing non-S&E values from the Academic R&D Expenditure Survey were never imputed, there were typically no past-year data to refer to when distributing imputed values (federal R&D, nonfederal R&D) across non-S&E fields of study. In these cases, the data of peer institutions with similar characteristics were used to estimate the allocation of expenditures across fields. Data for partial nonresponse were imputed using similar techniques.
Questions that were new to the FY 2010 HERD Survey were imputed using a set of core predictors that were highly correlated with most of the survey questions. These included type of institutional control, highest degree granted in S&E, and expenditure size. Several models also included the existence of a medical school in the set of core predictors. In most cases, an imputation class was identified based on predictors (e.g., public, doctoral-granting institutions with total R&D expenditures in the upper 25%), and missing values in that class were imputed as the average reported values in that class.
R&D expenditures from ARRA, capitalization thresholds for software and equipment, and headcounts of personnel and postdocs were not imputed.
The imputed R&D total was $64 million, or 0.1%, of the $61 billion in all separately budgeted R&D expenditures (table A-5). Tables A-6 through A-16 present imputed amounts for each applicable survey variable. The dollar amount imputed is displayed along with the percentage it represents of the national estimate for universities and colleges in a particular field.
A number of surveyed institutions have responded intermittently in past years. For years in which no response was received, data have been imputed as previously described. Although the imputation algorithm accurately reflects national trends, it cannot account for specific trends at individual institutions. For this reason, a separate estimation of institutional data for prior years is also performed, following current-year imputation.
For each institution, formerly imputed key variables for items 1–3 of the Academic R&D Survey were recomputed to ensure that the imputed data accurately represent the growth patterns shown by reported data of the FY 2010 HERD Survey. If, for example, data were reported for FY 2006 and FY 2010 but not for the intervening years, the difference between the reported figures for each item total was calculated and evenly distributed across the intervening years (2007–09). The new figures were spread across disciplines or sources of support on the basis of the most recent reporting pattern. These procedures result in much more consistent reporting trends for individual institutions but have little effect on aggregate figures reflecting national totals.
Question 5 (Clinical Trials) Analysis
Question 5, total R&D expenditures for clinical trials, was completed by 96.7% of the respondents from academic institutions. A total of 104 respondent institutions included clinical trial expenditures in their FY 2009 report, and 48 respondent institutions excluded those expenditures in FY 2009. The total R&D expenditures for clinical trials, $1.4 billion, represent 2.5% of question 5 respondents' total R&D expenditures ($53.5 billion) in FY 2010 (table A-17).
Question 10 (Federal Agency) Analysis
The portion of R&D expenditures financed by federal agencies not explicitly listed in question 9, $3.2 billion, represents 8.5% of all federally financed R&D expenditures. The majority of these expenditures were financed by the Department of Education ($602 million), the Department of Commerce ($460 million), and the Department of Transportation ($330 million) (table A-18).
Question 11 (ARRA) Analysis
Question 11, federally financed R&D expenditures from ARRA, was completed by 98.3% of respondents (table A-19). The portion of federally financed R&D expenditures received from ARRA funding, $2.7 billion (see table 10), represents 7.1% of all survey respondents' separately budgeted higher education R&D expenditures ($37.4 billion) in FY 2010. Doctorate-granting institutions reported a slightly higher portion of funds received from ARRA funding, 7.1% ($2.6 billion), than did nondoctorate-granting institutions, 6.4% ($41 million). Public institutions reported a lower portion of funds received from ARRA funding (6.8%, or $1.6 billion), compared to those from private institutions (7.6%, or $1.1 billion).
Question 14 (Capitalization Thresholds) Analysis
Question 14, capitalization thresholds for software and equipment, was completed by 92.2% of respondents (table A-20). Of these respondents, 84 reported that they do not capitalize software at all. The remaining respondents reported a mean software threshold of $70,000 and a mean equipment threshold of $7,000. Doctorate-granting institutions reported a higher capitalization threshold for software ($91,000) but a lower threshold for equipment ($5,000), compared to nondoctorate-granting institutions ($40,000 and $9,000). Public institutions reported higher thresholds for both software and equipment ($104,000 and $8,000), compared to private institutions ($17,000 and $5,000).
One should be cautious when comparing FY 2010 data to data from previous years. In prior years, the Academic R&D Expenditures Survey collected expenditures for S&E and non-S&E fields separately. Institutions were not always able to provide non-S&E expenditures, and those data were not previously imputed. Also, revisions to the definition of R&D may have influenced reported values to varying degrees, depending on the amounts of clinical trials and research training grants at a particular institution.
Within the University of California (UC) system, UC campuses manage nearly all R&D activities; however, some R&D expenditures are attributable to the UC Office of the President (UCOP) rather than to a particular campus. The reporting of UCOP R&D expenditures has changed over time. In FY 2004, R&D expenditures were first reported separately for UCOP. In FY 2004, when UCOP was reported separately from UC Los Angeles (UCLA), total R&D at UCLA decreased by $77 million, whereas UCOP showed expenditures of $90 million. Thus, the trend changes shown in table 32 for UCLA and UCOP for those years can be attributed to changes in the location of reporting UCOP R&D expenditures.
The Scripps Research Institute (TSRI) in La Jolla, California, was added to the FY 2005 survey population after mistakenly being omitted for several years. During the FY 2005 data collection, TSRI reported R&D expenditures for FY 2005 and FY 2004.
Due to the devastation caused by Hurricane Katrina and the subsequent staff turnovers, records damage, and shortage of resources, there was a higher rate of nonresponse in FY 2005 for universities and colleges in New Orleans, Louisiana. Data were imputed for institutions that were unable to commit the resources necessary to complete the FY 2005 survey. Institutions also reported reduced R&D expenditures in FY 2006 as a result of facility loss and faculty turnovers.
Finally, after the close of data collection and processing for FY 2010, a field classification error was discovered in SUNY Albany's report. SUNY Albany's total of $220 million in engineering R&D expenditures was mistakenly classified as metallurgical/materials engineering. In previous years, the total was reported under "Engineering, not elsewhere classified." The respondent reported that the expenditures are not able to be classified accurately by subfield. Beginning with the FY 2011 report, the engineering total will be imputed across the subfields for 2011 and earlier years in order to improve the accuracy of each subfield's aggregate total. Also, beginning in FY 2011 the SUNY Albany College of Nanoscale Science and Engineering will be reported as a separate entity on the survey.
Data from these and other surveys are available from the National Center for Science and Engineering Statistics (NCSES) at http://www.nsf.gov/statistics/ and through WebCASPAR (Integrated Science and Engineering Resources Data System) at http://ncsesdata.nsf.gov/webcaspar/. Data are available at the institution, state, and national levels and as longitudinal data. WebCASPAR also includes data from the NCSES surveys of universities and colleges. These data include information on earned degrees, opening fall enrollment, tuition, faculty salaries, tenure and fringe benefits, and financial statistics.