TITLE: Frequently Asked Questions (FAQs) for [1]NSF 13-126, Common
Guidelines for Education Research and Development
DATE: 9/12/2013
NSF 13-127
Frequently Asked Questions (FAQs) for [1]NSF 13-126, Common Guidelines for
Education Research and Development
1. Do these Guidelines apply to all education projects at NSF?
These Guidelines are most relevant for NSF programs and projects
that undertake education research and development (R and D)
activities. Some solicitations will explicitly reference the
Guidelines. The Guidelines are not developed to be relevant for
programs or projects that conduct outreach activities of various
kinds and scholarship programs are not included.
2. Why are these Guidelines necessary?
There is an interest across the federal government in increasing
coherence and consistency across programs with similar goals. The
education R and D programs in both NSF and the Department of
Education will benefit from the use of a common framework for the
projects that they fund. The Guidelines are intended to help the
PI community share common language to describe key points related
to relevant research literatures, outcomes and external feedback.
Finally, each agency will be able to build on the investments of
the other in more coherent ways.
3. Will these Guidelines preclude projects that are at the cutting
edge of innovation?
The Guidelines should not hurt projects that are at the cutting
edge of innovation. The key point of the Guidelines is to ensure
that projects are explicit about their research questions, methods
and analytic approaches in their proposals. These criteria should
be relevant for all types of education R and D efforts, including
those that may be at the cutting edge of innovation. Further,
these are guidelines intended to help PIs in their proposal
preparation. They should not be viewed as a straitjacket that
hinders creative thinking. Rather, PIs should review them, and
make sensible use of them as they describe the education R and D
activities they propose.
4. How do the guidelines affect NSF's Merit Review criteria?
The Guidelines are not intended to supplant the Merit Review
criteria. One element of the intellectual merit criterion for
proposals is whether the project can advance knowledge and
understanding. In addition, the intellectual merit criterion calls
for a well-reasoned, well organized plan based on a sound
rationale along with a mechanism to assure success. The Guidelines
are consistent with this criterion.
5. Does NSF intend to study the impact of the Guidelines? Are
revisions to the guidelines anticipated?
Conversations are underway about the best approach to studying the
impact of these Guidelines. One example would be to conduct
portfolio analyses of education R and D projects to see how they
fall into the six study types outlined in the Guidelines and to
examine how the distribution changes over time. Another example
would be to analyze how program solicitations change as a result
of implementing the Guidelines.
After the Guidelines have been in place for a while, it is
reasonable to anticipate that changes will be made as a result of
what is learned through the implementation process.
6. Do the Guidelines preclude or privilege any research
methodologies?
The Guidelines don't preclude or favor any research methods, but
they do require that the methods be well described, justified, and
appropriate to the research questions that are posed. They are
consistent with such broad ranging discussion of research methods
as Scientific Research in Education, (NRC, 2002). Both qualitative
and quantitative approaches can be used in all of the six research
genres that are described in the Guidelines. For example, a
small-scale randomized trial might be used in a design and
development study and a qualitative study might be embedded in an
efficacy or effectiveness study.
7. How are the guidelines intended to relate to research in emergent
technologies?
Research in emergent technologies may be concerned with phenomena
that are at scale rather than moving to scale (e.g., MOOCs). Even
so, in studying these phenomena, issues of the quality of the
research must be attended to and the descriptions of the research
questions, methods and analytic approaches must be clearly
described.
8. To what extent will these Guidelines represent a culture shift to
NSF PIs?
For some NSF PIs, these Guidelines will not represent a shift. The
Guidelines essentially codify what is considered to be reasonable
practice for education research and development. For others,
perhaps particularly for those who have been engaged primarily in
development work, the Guidelines will represent more of a shift as
PI teams will have to more explicitly address research questions,
methods and approaches for analysis of data.
9. How will reviewers be informed about the Guidelines?
Reviewers will be informed of the Guidelines through multiple
approaches: the Guidelines will be posted on the NSF website,
referred to in program solicitations, discussed in reviewer
webinars and orientations, and presented at PI and other
professional meetings.
10. What are the implications of the Guidelines for external
evaluation of projects?
The Guidelines include recommendations for all types of studies
that call for external feedback on the work being proposed.
However, they are more expansive than requiring a third-party
evaluation. External feedback can include a number of approaches
including third party evaluation, program officer evaluation,
and/or regular feedback from advisory groups. It will be up to the
proposer to argue for whatever kind of external feedback they
identify as appropriate, aligned with program requirements.
11. What was the input from the field for these guidelines?
The Guidelines were sent out for external review to five experts
through a process organized by the Department of Education's
Institute for Education Sciences. In addition, a Presidential
Invited session on the Guidelines was held at the 2013 American
Education Research Association (AERA) annual meeting in San
Francisco. At that session, there were round table discussions
about the implications and implementation of the Guidelines.
12. What are the plans for future input from the field?
We anticipate that sessions that include discussion of the
Guidelines will occur at 2014 professional meetings of
organizations such as the American Education Research Association
(AERA), National Association for Research on Science Teaching
(NARST), National Council on Teachers of Mathematics (NCTM) and
the Society for Research on Educational Effectiveness (SREE),
among others. The Guidelines will also be discussed at relevant PI
meetings.
13. Do the Guidelines have implications for the inclusion of education
research expertise on education development proposals?
Proposals that focus on education development activities should be
certain to include personnel with appropriate expertise in
research design and methods"as key personnel or advisors.
14. How might these Guidelines help community building among research,
policy and practice community?
Proposers should anticipate the policy and/or practice
significance of the projects they propose in any of the six
research genres. Even if the implications for policy or practice
are distal, projects should describe their potential relevance to
policy or practice. In addition, the Guidelines can help
practitioners and policymakers develop a better understanding of
what various types of education research should be expected to
produce. The Guidelines are intended to support better-informed
decisions based on the levels of evidence provided.
15. Will the use of the Guidelines lead to cookie cutter approaches in
terms of projects that are funded?
We do not anticipate that the use of the Guidelines will lead to
cookie cutter approaches in terms of the research and development
projects that are funded. The Guidelines outline six different
genres of research that call for different kinds of evidence. In
addition, the Guidelines recommend supporting projects that span
the genres. These Guidelines will not restrict new or original
approaches.
16. Do the Guidelines apply to PROGRAMS as well as proposals in both
agencies?
The Guidelines were developed to apply to education research and
development proposals in order to help the agencies make more
strategic decisions on individual projects. They were not designed
to apply to programs. It might be possible to build on the ideas
in the Guidelines to develop a comparable document for program
evaluation, but that would entail undertaking a separate and
different set of activities. We caution others from using the
Guidelines for purposes beyond what is intended.
References
1. http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf13126