Survey Quality Measures
Availability of Data
1. Overview (FY 2010–12 Survey Cycle)
The annual Federal Funds Survey is the primary source of information about federal funding for research and development in the United States. It is used by policymakers in the executive and legislative branches of the federal government in determining policies, laws, and regulations affecting science; it is also used by those who follow science trends in every sector of the economy, including university administrators and professors, economic and political analysts, R&D managers inside and outside the government, the science press, and leading members of the science community in the United States and around the world. The results of the survey are also used for budget purposes for four federal programs: the Federal Laboratory Consortium for Technology Transfer, Small Business Innovation Research, Small Business Technology Transfer, and Experimental Program to Stimulate Competitive Research.
The survey is completed by the 15 federal departments their 67 subagencies and the 12 independent agencies listed below:
- Department of Agriculture
- Agricultural Marketing Service
- Agricultural Research Service
- Animal and Plant Health Inspection Service
- Economic Research Service
- Foreign Agricultural Service
- Forest Service
- Grain Inspection Packers and Stockyards Administration
- National Agricultural Statistics Service
- National Institute of Food and Agriculture
- Rural Business-Cooperative Service
- Department of Commerce
- Economic Development Administration
- National Institute of Standards and Technology
- National Oceanic and Atmospheric Administration
- National Telecommunications and Information Administration
- U.S. Census Bureau
- Department of Defense
- Chemical/Biological Defense
- Defense Advanced Research Projects Agency
- Defense Business Transformation Agency
- Defense Contract Management Agency
- Defense Information Systems Agency
- Defense Logistics Agency (includes the Defense Technical Information Center)
- Defense Security Cooperation Agency
- Defense Technical Information Center
- Defense Threat Reduction Agency
- Department of the Air Force
- Department of the Army
- Department of the Navy
- Joint Staff
- Missile Defense Agency
- Office of the Secretary of Defense
- Operational Test and Evaluation
- Special Operations Command
Department of Education
Department of Energy
- Department of Health and Human Services
- Administration for Children and Families
- Administration on Aging
- Agency for Healthcare Research and Quality
- Agency for Toxic Substances and Disease Registry
- Centers for Disease Control and Prevention
- Centers for Medicare & Medicaid Services
- Food and Drug Administration
- Health Resources and Services Administration
- National Institutes of Health
- Office of the Assistant Secretary, Planning and Evaluation
- Department of Homeland Security
- Science and Technology Directorate
- U.S. Coast Guard
Department of Housing and Urban Development
- Department of the Interior
- Bureau of Ocean Energy Management, Regulation and Enforcement
- Bureau of Reclamation
- National Park Service
- U.S. Geological Survey
- Department of Justice
- Bureau of Prisons
- Federal Bureau of Investigation
- Office of Justice Programs
- Department of Labor
- Employee Benefits Security Administration
- Occupational Safety and Health Administration
- Office of the Secretary
- Pension Benefit Guaranty Corporation
Department of State
- Department of Transportation
- Federal Aviation Administration
- Federal Highway Administration
- Federal Motor Carrier Safety Administration
- Federal Railroad Administration
- Federal Transit Administration
- National Highway Traffic Safety Administration
- Office of the Secretary
- Pipeline and Hazardous Materials Safety Administration
- Research and Innovative Technology Administration
- Department of the Treasury
- Bureau of Engraving and Printing
- Internal Revenue Service
- U.S. Mint
Department of Veterans Affairs
- Independent agencies
- Agency for International Development
- Appalachian Regional Commission
- Environmental Protection Agency
- Federal Communications Commission
- Federal Trade Commission
- Library of Congress
- National Aeronautics and Space Administration
- National Archives and Records Administration
- National Science Foundation
- Nuclear Regulatory Commission
- Smithsonian Institution
- Social Security Administration
c. Key variables
The survey provides data on federal obligations by the following key variables:
- Character of work
- Basic research
- Applied research
- Federal agency
- Federally funded research and development centers (FFRDCs)
- Field of science and engineering
- Funding type
- American Recovery and Reinvestment Act (ARRA)
- Geographic location (within the United States and foreign country)
- Performer (type of organization doing the work)
- R&D plant
The survey provides data on federal outlays by the following key variables:
- Character of work
- Basic research
- Applied research
- R&D Plant
Note that the variables "R&D," "character of work," and "R&D plant" in this survey use definitions comparable to those used by the Office of Management and Budget. The variables "geographic location" and "field of science and engineering" are comparable in the Federal Funds Survey and the related NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions.
2. Survey Design
a. Target population and sample frame
The population consists of the federal agencies that conduct R&D programs. In the survey cycle for data collection on FYs 2010–12, 27 federal agencies (15 federal departments and 12 independent agencies) reported R&D data. Because multiple subdivisions of a federal department were in some cases requested to complete the survey, there were 84 individual respondents (5 federal departments, 67 department subdivisions, and 12 independent agencies). The sample frame is obtained from information in the President's Budget submitted to Congress. The Analytical Perspectives volume and the "Detailed Budget Estimates by Agency" section of the appendix to the president's budget identify agencies that receive research and development funding. The survey's population accounts for the agencies that were to receive an aggregate of 99.6% of the R&D funding from the budget and establishes a solid sample frame.
b. Sample design
Five of the 15 federal departments were surveyed at the department level, and 10 were surveyed at the subdivision level, with 67 subdivisions surveyed. All 12 independent agencies were surveyed at the agency level.
c. Data collection techniques
Synectics for Management Decisions, Inc. (Synectics) performed the data collection for volume 60 (FYs 2010–12) under contract with NSF's National Center for Science and Engineering Statistics (NCSES). Agencies were initially contacted by phone to verify the name, email address, and phone number of each survey respondent. A Web-based data collection system was used to collect data for the Federal Funds Survey.
Federal funds data, as collected, span 3 government fiscal years: the fiscal year just completed, the current fiscal year, and the next fiscal year. Actual data are collected for the year just completed; estimates are obtained for the current and next fiscal year.
For Volume 60, actual data (representing completed transactions) were collected for FY 2010 and estimated data were collected for FY 2011 and 2012. Estimated data do not represent final actions. Data collection for the Federal Funds Survey began during the fourth quarter of FY 2010 and continued through the first and second quarters of FY 2012. The amounts reported for FY 2011 reflect congressional appropriation actions as of that period, as well as apportionment and reprogramming decisions as of that time. Data for FY 2012 represent as-yet unimplemented administration budget proposals. Authorization, appropriation, deferral, and apportionment actions completed after these data were collected will be reflected in later surveys in this series.
The Web-based data collection system is used to collect and manage data for the Federal Funds Survey. This Web-based system was designed to help improve survey reporting and reduce data collection and processing costs by offering respondents direct online reporting and editing. The goal is to provide an easier, more cost-effective way both for respondents to complete the survey each year and for the contractor to collect and process the data.
All data collection efforts, data imports, data editing, and trend checking are accomplished using the Web-based data collection system. The Web-based data collection system has a component that allows survey respondents to enter their data online; it also has a component that allows the contractor to monitor support requests, data entry, and data issues. Both components are password-protected, allowing only authorized users to access them.
d. Estimation techniques
This survey is a census of the 15 Federal departments and 12 independent agencies that sponsor R&D. There is no known item nonresponse, so no weighting of responses or imputation used. A submission is considered complete and final when data have been submitted at the agency level and the agency-level respondent has verified the responses.
3. Survey Quality Measures
a. Sampling variability
Because all eligible agencies are included, there is no sampling error.
Because identifying relevant federal agencies is a straightforward task, coverage is assumed to be complete except for activities of the Central Intelligence Agency and other clandestine R&D activities as defined by the president's budget. Agencies are identified through such sources as the president's annual report, Budget of the United States Government, and the respondent agencies themselves. In addition, the Federal Yellow Book and the U.S. Government Manual may be consulted. Because not all agencies issue funding for research and development, the Analytical Perspectives and the Detailed Budget Estimates by Agency in the president's budget are reviewed to identify the subset of all agencies that are allocated R&D funding. Including these agencies for the Federal Funds Survey as well as independent agencies that have a past practice of issuing R&D funding avoids unnecessary contacts with non-R&D-funding agencies.
(1) Unit nonresponse. The response rate for this survey is 100%.
(2) Item nonresponse. Agencies are encouraged to estimate informally when actual data are unavailable. The survey instrument allows respondents to enter data or skip data fields. NSF assumes a blank field is zero for estimation purposes. If a significant number of blank fields are incorrectly skipped by respondents, bias could result.
Some measurement problems are known to exist in the data. More specifically, some agencies are not able to report the full costs of R&D. For example, the Department of Defense (DOD) does not include headquarters costs of planning and administering R&D programs, which are estimated at a fraction of 1 percent of its total cost; DOD has stated that identification of amounts at this level is impracticable.
The National Institutes of Health (NIH) in the Department of Health and Human Services currently has many of its awards in its financial system without any field of science code. NIH therefore uses an alternate source to estimate its research dollars by field of science. NIH uses the name of the academic department (department code) receiving a grant as an approximation for field of science codes. NIH is currently looking at alternatives for improving its reporting by field of science.
The National Aeronautics and Space Administration (NASA) does not include any field of science codes in its financial database, and consequently must estimate what percentage of the agency's research dollars are allocated into the fields of science.
Also, agencies are required to report the ultimate performer of R&D. However, through past workshops, NSF has learned that some agencies do not always track their R&D dollars to the ultimate performer of R&D. This would lead to some degree of misclassification of performers of R&D, but NSF has not determined the extent of the errors in performer misclassification by the reporting agencies.
R&D plant data are underreported to some extent because of the difficulty some agencies, particularly DOD and NASA, encounter in identifying and reporting these data. DOD's respondents report obligations for R&D plant funded under the agency's appropriation for construction, but they are able to identify only a small portion of the R&D plant support that is within R&D contracts funded from DOD's appropriation for research, development, testing, and evaluation. Similarly, NASA respondents cannot separately identify the portions of industrial R&D contracts that apply to R&D plant, since these data are subsumed in the R&D data covering industrial performance. NASA R&D plant data for other performing sectors are reported separately.
4. Trend Data
The information included in this survey has been stable since FY 1973, when federal obligations for research to universities and colleges by agency and detailed S&E field were added to the survey. Many of the other variables are available from the early 1950s on. However, analysts studying trends are encouraged to obtain up-to-date data from the NCSES website, because agencies reclassify their responses for prior years as additional budget data become available.
5. Availability of Data
NCSES publishes the data from this survey annually in detailed statistical tables in the series Federal Funds for Research and Development and the Science and Engineering State Profiles series. Historical information is also available from NSF's Integrated Science and Engineering Resource Data System (WebCASPAR). Through WebCASPAR a user can build various survey data tables including the following:
- Detailed Historical Tables
- Federal Obligations for Research by Agency and Detailed Field of Science and Engineering
- Federal Obligations for Research to Universities and Colleges by Agency and Detailed Field of Science and Engineering
b. Electronic access
Data from this survey are available on the NCSES website at http://www.nsf.gov/statistics/fedfunds/ and in the WebCASPAR data system (http://webcaspar.nsf.gov).
c. NSF contact
For additional information about this survey, contact:
Research and Development Statistics Program
National Center for Science and Engineering Statistics
National Science Foundation
4201 Wilson Boulevard, Suite 965
Arlington, VA 22230
Phone: (703) 292-7815