State Government Research and Development: Fiscal Year 2009
Appendix A. Technical Notes
The National Science Foundation (NSF) Survey of State Government R&D measures the extent of research and development activity performed and funded by each of the nation's 50 states, the District of Columbia, and Puerto Rico. Results of the survey are useful to a variety of data users interested in R&D performance, including the National Science Board; the Office of Management and Budget; the Office of Science and Technology Policy and other science policy makers; institutional researchers; and private organizations.
Scope of Survey
The survey covered state government departments, agencies, independent commissions, and other entities determined to be state run (defined as falling under the governor's jurisdiction). Several industry-specific state commissions, which are generally chartered by state legislatures but administered independently, were considered state agencies and are included in the survey. The survey excluded state-run colleges and universities, which are included as part of the NSF Survey of R&D Expenditures at Universities and Colleges, as well as state-run laboratories or experiment stations controlled by state universities. It also excluded entities determined to be nonprofit or private, as defined by Census Bureau government classification. Although the universe of interest covered all state-run entities except state-run colleges and universities, evidence showed that some state agencies were not involved in R&D activity and did not need to be surveyed.
Using input from potential state coordinators, agency respondents, and R&D experts, survey staff developed a Web-based survey instrument for both the central state contacts and the state agency respondents. The survey was launched in June 2010 with a letter to governors asking for each state's participation.
Working with Census Bureau staff, governors' offices identified a state coordinator in each state to assist in collecting data from state agencies. Each coordinator was given a list of agencies in his or her state and asked to send the survey to any other state agencies that the coordinator believed could have conducted R&D. The list consisted of agencies that were preselected by the Census Bureau and NSF before the FY 2006 survey, as well as any agencies that had been added by the state coordinator over the previous two survey cycles. For the first time, state coordinators were also able to delete an agency that was preselected. They did so by informing Census Bureau staff that the agency in question funded or performed no R&D and would not in the future; 4 agencies were deleted through this process. Of the final count of 438 agencies for the FY 2009 survey, 392 were agencies that the coordinators provided initially, and 46 were those that coordinators added during the survey.
Before officially submitting the data for their state, coordinators performed a final verification of aggregated agency data. All responses, including the initial agency data submissions and final state coordinator verifications, were received via the Web form or e-mail. Basic logical edit checks, review of respondent comments, and comparisons of data from previous surveys allowed staff to detect data errors and work with state respondents to correct them.
The respondent questionnaire consisted of one screening and five R&D expenditure data questions. The five R&D expenditure questions contained a total of 11 items.
Screening Question: The purpose of this question was twofold: (1) to reduce the burden on agency respondents who did not have qualifying R&D expenditures during FY 2009, and (2) to further clarify the scope of the survey for those agencies with R&D expenditure data to report. Respondents who answered "No" to the screening question were not required to complete the remaining questions.
Question 1: The first seven data items were grouped as one question on the survey form. The purpose of this question was to allow the respondent to report all R&D expenditures for his or her agency, with the exception of R&D plant expenditures. The first three items were the R&D expenditures by the agency's internal performers (employees of the agency), broken out by source of funds (state, federal, and other). The next three items asked the respondent to report expenditures according to three types of external performers (academic performers, companies, and other). The final data item, the agency's total R&D expenditures, was calculated automatically as the respondent entered data, as a sum of the previous six items.
Question 2: This question asked respondents to report the total amount of basic research expenditures for their agency. This response could be less than or equal to the total R&D expenditures reported in Question 1.
Question 3: This question requested total R&D expenditures from federal sources, regardless of whether the R&D was performed internally or externally. This response could be less than or equal to the total expenditures reported in Question 1.
Question 4: This question, new to the survey for FY 2009, asked respondents to distribute their total R&D expenditures among five types of R&D (agriculture, environmental and natural resources, health, transportation, and other).
Question 5: The final question requested that respondents report expenditures on construction or acquisition of R&D facilities during FY 2009. This item was requested as a separate figure from the previous four questions on the survey to avoid skewing the total R&D expenditures amount with large, one-time agency expenditures for construction projects or the purchase of land or buildings.
The instrument also allowed respondents to provide comments for each question.
State coordinators in Iowa, Nevada, New Jersey, New Mexico, Ohio, Oklahoma, Oregon, Rhode Island, and the District of Columbia did not verify the aggregated agency data. At least one agency in each of the following states did not respond to the survey: Iowa (1), Nevada (1), New Jersey (1), New Mexico (4), Ohio (2), Oklahoma (2), Oregon (2), Rhode Island (2), and the District of Columbia (1).
Although every effort was made to exclude out-of-scope amounts, it is likely that some reported data include expenditures for non-R&D activities, such as commercialization, environmental testing, or routine survey work.
The current NSF Survey of State Government R&D has been conducted for FYs 2006, 2007, and 2009. Data presented in trend tables in this report are from the most recently completed survey cycle. References to prior-year data should be restricted to those published in this report for two reasons: (1) when completing the current-year survey, survey respondent may revise their prior-year data, and (2) NSF reviews data for prior years for consistency with current-year responses, and if necessary may revise these data in consultation with respondents.
The previous NSF Survey of State Government R&D collected FY 1995 data. Prior to that effort, surveys were conducted for FYs 1987 and 1988, as well as two efforts in the 1970s. Because of differences in the survey populations, definition of covered R&D activities, and collection methods, the results of those surveys are not comparable with the statistics collected on the FYs 2006, 2007, and 2009 Survey of State Government R&D.
State R&D totals display considerable volatility between the surveys. Although total state agency R&D expenditures increased by 7% between FYs 2007 and 2009, 28 states reported overall increases, and 22 states plus the District of Columbia reported overall declines. Three states (Idaho, New Mexico, and Utah) reported combined agency R&D increases of more than 100% from FY 2007 to FY 2009; 6 states (Maine, Michigan, Vermont, the District of Columbia, West Virginia, and Wyoming) reported combined agency R&D decreases of 50% or more.
State agency expenditures are influenced by several national and state-specific factors, and large changes (positive or negative) are not unusual, especially for discretionary spending items, such as R&D. However, given that states have comparatively little experience in tracking and measuring R&D, it is likely that some portion of the reported changes reflects measurement and coverage errors. Census Bureau/NSF staff conducted follow-up calls for agencies with data changes of plus or minus 50% and at least $1 million between FYs 2007 and 2009 to ensure the accuracy of the FY 2009 survey data.
The data exclude R&D expenditures of states that did not flow through state agencies' budgets. The state totals do not include direct appropriations from state legislatures to colleges and universities. In FY 2009, universities and colleges reported expending $3.6 billion on separately budgeted R&D activities that were funded from all sources of state and local government support. State agencies reported $0.5 billion in expenditures used to support R&D performance by academic institutions. A major factor for the difference between totals reported in the Academic and the State R&D surveys is direct appropriations/grants to state-run universities that are included in the former, but not the latter. Another likely factor is the exclusion of R&D at agricultural experiment stations from the state survey totals.
Two response rates were calculated for the FY 2009 Survey of State Government R&D, one for agency-level responders and one for official data verification at the state level. All states and District of Columbia participated in the survey; however, only 42 out of 51 (82.4%) officially verified final state data (note: Unlike for previous years, Puerto Rico is not included in the FY 2009 survey totals). The eight states and District of Columbia that did not officially verify data had some or all agencies submit data. The final agency response was 422 out of 438 (96.3%). Table A-1 displays final agency response rates, as well as counts of agencies that identified themselves as having qualifying R&D expenditures for FY 2009.
National Totals and Imputations
Statistical methods were not used to account for nonresponding agencies. All state and national totals are aggregates of reported agency data.
Survey respondents were given the following definition of R &D and other important terms:
Research and development is creative work conducted systematically to (1) extend scientific knowledge; or (2) devise new or improved applications, including materials, products, devices, processes, systems, or services.
Sources and examples of R&D funding include the following:
Basic research is conducted primarily to acquire new knowledge without any specific product or process in mind.
Construction and acquisition of facilities used primarily for R&D includes major costs for new construction, major renovations, and purchase of land or buildings to be used primarily as R&D facilities.
Performers—external are those outside the department/agency who perform R&D under the administrative oversight or control of that department/agency. This may include projects for the department/agency, as well as its extramural research programs. External performers include the following:
Performers—internal are the department/agency's own employees who perform R&D and includes R&D performed by those employees and services performed by others in support of an internal R&D project (e.g., lab testing).
Data from this and other NSF R&D surveys are available at http://www.nsf.gov/statistics/.
 The District of Columbia and Puerto Rico are referred to as states in this report.