Virtual Office Hours: Merit Review and How to Get Involved with NSF

On Monday, June 10^th, 1 – 2pm ET DEB held its Virtual Office Hour: Merit Review and How to Get Involved with NSF. DEB Program Officers discussed what to expect when you are a panelist or reviewer and opportunities to get involved with NSF that don’t involve grant writing. Upcoming DEB Virtual Office Hours are announced ahead of time on DEBrief, so we suggest you also sign up for blog notifications. 

If you couldn't make it to this or any future office hours, don’t worry! Come back to the blog afterwards, as we post recaps and the presentation slides of all office hour sessions. Alternatively, visit our Office Hours homepage for slideshows and recaps of past topics. Slides are made available below. 

Virtual Office Hours are on the second Monday of every other month from 1 – 2pm ET. 

Presentation outline

Division of Environmental Biology NSF staff in attendance today: 

• Jeremy Wojdak (host) - Population and Community Ecology
• Maureen Kearney – Systematics and Biodiversity Science
• Sam Scheiner – Evolutionary Processes
• Jason West – Ecosystem Science

Resources:

• DEB Virtual Office Hour DEB Office Hours: second Monday of each month, 1-2pm Eastern
• DEB Blog posts upcoming topics, registration, and recap posts: https://debblog.nsfbio.com/office-hours/ 
• Sign up for BIO News and Updates
• Volunteer to review DEB: https://www.surveymonkey.com/r/DEBexpertise 
• “The art and science of reviewing proposals” NSF Video
• Find recent BIO funding opportunities

Learning the “culture” of NSF

Formal documentation 

• Proposals & Awards Policy and Procedures Guide (PAPPG)
• Solicitations
• Dear Colleague Letters
• Research.gov/ Fastlane systems 

NSF outreach

• Blogs
• Virtual Office Hours
• Conferences
• Individual meetings with program officers 

Participation

• Proposal submission
• Receiving reviews/ panel Summaries
• Award management including reports, requests, supplements, etc. 
• Writing ad hoc reviews 
• Panelist service

Integrity of the review process

• Conflicts of Interest - Your responsibility to identify COIs, both institutional and individual. Also, even if no official conflict exists, be honest and open if you don’t feel you can be objective. 
• Confidentiality – Do not share proposal information.
• Bias – Many flavors, many origins... but awareness is first step in mitigation.

Reduce cognitive biases 

• Actively reflect on your own thought process
• Think of alternative views 
• Play a devil’s advocate
• Take time with your decision

Merit Review Process Overview

1. Review panelists and ad hoc reviewers 
2. Panel discussion
3. Panel summary and panel recommendation
4. Division directorate review 
5. Business review and processing
6. Notification of award decision

Merit Review Criteria

• Intellectual Merit: The intellectual merit criterion encompasses the potential to advance knowledge;
• Broader Impacts: The broader impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes. 
• Keep in mind: 
  • Is this exciting science? 
  • Will it work? (is the risk/reward appropriate?)
  • Can this team do this work?

Basic Reviewer Guidelines

• Use a structured format, highlighting strengths and weaknesses in intellectual merit and broader impacts
• Focus on the most important aspects... Not every detail like in a manuscript review.
• Make sure the narrative review and proposal rating align: 
  • Poor: Proposal has serious deficiencies
  • Fair: Proposal lacking in one or more critical aspects; key issues need to be addressed 
  • Good: A quality proposal, worthy of support 
  • Very Good: High-quality proposal in nearly all aspects; should be supported if at all possible 
  • Excellent: Outstanding proposal in all respects; deserves highest priority for support 

Advice on Broader Impacts 

• There is no single formula or checklist. Proposals should provide a rationale for chosen broader impacts, describe measurable outcomes, and match the resources (time and money) allocated. Go above and beyond what principal investigators are already paid to do.
• Budget should reflect the proposed broader impacts activities
• Use existing infrastructure, as appropriate, but give, as well as take 
• Assessment of outcomes is ideal though not strictly required. 

Consult Center for Advancing Research Impact in Society (ARIS) for additional resources: https://www.researchinsociety.org/ 

Reviewing Traps 

• Mentally re-writing the proposal yourself 
• Summarizing the proposal
• Giving a pass to underdeveloped plans by great/experienced PIs
• Favoring charismatic systems
• Commenting on things you don’t understand/know about
• Favoring quantity over quality in broader impacts
• Expecting novelty in broader impacts
• Piling on the negatives
• Describing, rather than evaluating

Special criteria

In addition to intellectual merit and broader impacts, sometimes reviewers must consider solicitation-specific criteria:

• Vary by solicitation, but may constrain scope, content, and participants (e.g., RUI, CAREER, OPUS, LTREB). 
• You just want to measure the proposal by the correct standard.
• NSF will provide guidance to reviewers for each solicitation.

As a reviewer, you help shape the community norms… 

NSF defines the “boundaries” for parts of a proposal, while the community helps construct the norms through their perspectives in reviews: 

• Broader impacts – Which are most valuable? How much is “enough”? What are the emergent best practices? How should broader impacts be weighed versus intellectual merit? 
• What constitutes a safe and inclusive field plan (SAIF)? How should data be managed and shared (DMP)? How are postdocs best mentored (PMP)?

Reviewing outside of DEB 

Opportunities for DEB panel and reviewer service are limited by the number of proposals/panels we have.

It is important to look outside of DEB as well, including: 

• Graduate Research Fellowship Program (GRFP) 
• Division of Biological Infrastructure
  • Postdoctoral Research Fellowships in Biology (PRFB)
  • Research and Mentoring for Post-bacs (RaMP)
  • Other infrastructure stuff
  • Postdoctoral Research Fellowships in Biology (PRFB)
• Directorate for STEM Education
  • Historically Black Colleges and Universities - Undergraduate Program (HBCU-UP) 

Q&A Notes

If you were unable to attend, here are some of the questions asked during the Q & A section:

How often do you hold panels? Given no deadlines for many programs, do panels still convene at regular intervals, or do they convene only when they receive a critical mass of reviews/proposals?

We hold frequent and smaller panels once there is a critical mass of proposals. Our goal is to get decisions on proposals out in a timely manner. Therefore, you could be called on to be a reviewer at any point during the year. If you find that you can’t be a reviewer at one time, you may have the opportunity to be a reviewer at a different time.

Is there any good or bad time to submit a proposal?

The best time to submit a proposal is when it is ready. We host 3-4 panels a year per cluster so that whenever you submit there will be an upcoming proposal.

How many people review each proposal?

Within DEB, we strive for 5-6 reviews as a mix of ad hoc and panelists. However, if a proposal is co-reviewed in an additional program there may be more.

For broader impacts, I heard there are some activities that are sort of (informally) expected, such as incorporate findings into graduate education. Is this true?

No. There are no expectations for any particular type of broader impact. However, the most frequent type we see is training of students but that’s not always the case. The most successful broader impacts are context dependent and what the PI is capable of based on their institutions, the research that they can naturally do, or communities that they are tapped into.

Are there any resources for what the community currently constitutes a good Data Management Plan, Post-doc mentoring plan, etc.?

Check out the Biological Sciences Data Management and Sharing Plan Guidance for Proposals and Awards webpage for information regarding those two documents. Additionally, universities sometimes post things online, and you can also speak with your colleagues. 

How long do panelists have to review a proposal?

It depends, but generally about a month.

How does one get selected for panel service? I’m an early-career researcher and I fill out each Survey Monkey for panel service but have not yet been selected despite having done ad hoc reviews several times. Any tips for making oneself more competitive for panel service?

It’s us not you. There are general and specific reasons for this. We strive to ensure that we have a diverse panel including geography, career stages, institution types, gender, race, etc. But also, there needs to be panelists who research areas align with the proposals that come in. The number of panelists is smaller than the number of ad hoc reviewers we use in a year. If you continue to experience this, please reach out to a Program Officer and we’ll try our best to see if we can include you.

How do you decide who will be the ad hoc reviewers for proposals that are resubmissions? Do the same reviewers typically get asked again?

It depends on the proposal and Program Officer. If a reviewer went in-depth or was really excited about the proposal, we may ask for them to review it again if they are available. We have to take into account bias, so it tends to be proposal and context specific.

Do clinicians with extensive clinical experience but without major research papers get to be part of a panel?

The only criterion to be a reviewer is to have a terminal degree in the field of interest. In DEB, this is a Ph.D. You just need to be a critical thinker and able to communicate these thoughts in a written format. Other programs may have specific requirements in terms of career stage, but DEB does not.

For technical merit, could you define the level of excellency?

If asked to be a reviewer, you will be provided with helpful documents and guides on how to write reviewers.  Additionally, the 19-minute video we mentioned in the slides is very helpful.

How are Dear Colleague Letters (DCL) weighed in the decision? Are reviewers asked to comment on this? Say a proposal is submitted to a cluster, and it addresses a DCL, how much weight does it get?

It really depends on what the Dear Colleague Letter is. Reviewers are not required to comment on DCLs as they generally do not have review criteria associated with them. Whether they weigh into the recommendation depends on the specifics either programmatically or what the agency-wide priorities are.

Please reach out to a PO if you have any questions about the proposal submission and review process in DEB programs. NSF has suggested 5 tips on working with Program Officers as part of the NSF 101 series on our Science Matters blog.

Check out the upcoming office hour topics below and be sure to check back here or on the NSF Events Page for information on how to register. We’re taking July off so our next Virtual Office Hour will be on August 12^th from 1pm-2pm Eastern Time. We will be joined by our colleagues in the Graduate Research Fellowship Program (GRFP). This event is suitable for both students looking to apply as well as faculty interested in becoming a reviewer for the GRFP.