WEBVTT

1
00:00:02.159 --> 00:00:14.759
Wendy Nilsen, NSF/CISE: And I'm here with my colleague, Dana Wolf Hughes from the NIH and we're going to talk to you about smart health and biomedical research in the era of artificial intelligence and advanced data science.

2
00:00:15.690 --> 00:00:26.220
Wendy Nilsen, NSF/CISE: Also known as SCH, this is NSF 21-530 and we're going to give you more information on this and we're going to also have time to ask questions.

3
00:00:28.500 --> 00:00:30.390
Wendy Nilsen, NSF/CISE: I will say that in your

4
00:00:33.030 --> 00:00:43.080
Wendy Nilsen, NSF/CISE: In your in your screen, you should see you have a Q&A section so you can ask questions during this talk to my colleagues. NSF staff

5
00:00:43.470 --> 00:00:58.140
Wendy Nilsen, NSF/CISE: will will be answering. So what I'm going to do is introduce my colleagues. Now I'm going to start with Goli Yamini who is central to the program. She's not one of our program stuff, but she is central to this. She is currently a AAAS fellow

6
00:00:59.700 --> 00:01:05.040
Wendy Nilsen, NSF/CISE: The next person I'll introduce is Scott Acton and he's from our CISE CCF program.

7
00:01:06.300 --> 00:01:12.000
Wendy Nilsen, NSF/CISE: Balakrishnan Prabhakaran or Prabha and he's in CISE is

8
00:01:13.530 --> 00:01:15.120
Wendy Nilsen, NSF/CISE: I don't know if Fay is here.

9
00:01:17.040 --> 00:01:33.270
Wendy Nilsen, NSF/CISE: Fay Cobb Payton will be with us at some point, and she's our CISE CNS representative, my colleague way. Wei Ding from CISE is is here, and Georgia Ann-Klutke is our colleague from engineering in CMMI she is not able to join us today, but sends her good wishes.

10
00:01:34.350 --> 00:01:45.510
Wendy Nilsen, NSF/CISE: Tanya Korelsky from CISE, I don't believe will be here today, but she also sends good wishes. Sylvia Spangler from CISE is will be joining us shortly.

11
00:01:46.590 --> 00:02:13.860
Wendy Nilsen, NSF/CISE: Betty Tuller who's from the NSF SBE BCS program, may be joining us. Albert Wang from the engineering ECCS program should also be joining us shortly. Judy Wang from our MPS DMS is on the line with us now, and Johmn Zhang from  from engineering ECCS will also be with us. I'm

12
00:02:17.520 --> 00:02:33.690
Dana Hughes NIH-OBSSR: Dana. Yes. And so as Wendy said I'm Dana Wolf Hughes and I am at NIH and the Office of Behavioral and Social Sciences Research and historically we've led this initiative out of NIH, along with the NSF colleagues.

13
00:02:34.860 --> 00:02:48.300
Dana Hughes NIH-OBSSR: But this year we've expanded the participation across the NIH ICs and in particular are working closely with our colleagues in the office a data science strategy.

14
00:02:49.590 --> 00:03:01.050
Dana Hughes NIH-OBSSR: So on the slot. I'm not gonna name off everybody but we have Institute, centers and offices at the NIH who are now participating and a lot of my colleagues are on the call today.

15
00:03:02.550 --> 00:03:10.080
Dana Hughes NIH-OBSSR: While this will focus a lot on the solicitation that's posted on the NSF site that Wendy just mentioned.

16
00:03:10.830 --> 00:03:29.730
Dana Hughes NIH-OBSSR: There's also an accompanying notice that the link below which is notice-OD-21-011 and that was out for each of these participating institutes and centers and offices, what their specific priorities are as it pertains to smart connected health

17
00:03:30.870 --> 00:03:37.140
Dana Hughes NIH-OBSSR: What we do ask though is that before coming to one of these program officers listed here.

18
00:03:37.380 --> 00:03:53.460
Dana Hughes NIH-OBSSR: About your proposal to start off on the NSF side to determine whether or not the application fits within the scope of the program and what NSF looks for given that the applications are received by them and reviewed on the NSF side as well. Next slide.

19
00:03:55.470 --> 00:04:02.790
Wendy Nilsen, NSF/CISE: So you might ask why why why is NSF and NIH coming together for the smart, smart health program.

20
00:04:03.270 --> 00:04:10.260
Wendy Nilsen, NSF/CISE: And there's many reasons. And we've seen, we've seen computing and engineering, and mathematics and social behavioral sciences.

21
00:04:11.220 --> 00:04:17.820
Wendy Nilsen, NSF/CISE: Really, really are transforming the world. We haven't seen that in health in the same ways

22
00:04:18.180 --> 00:04:26.880
Wendy Nilsen, NSF/CISE: So, I mean, this isn't it. This is a study from 2013 and the Institute of Medicine when you looked at the US compared to the other Western peers.

23
00:04:27.360 --> 00:04:33.900
Wendy Nilsen, NSF/CISE: The US is doing poorly. And if you look at non-communicable diseases. If you look at communicable. This is pre-COVID

24
00:04:34.320 --> 00:04:45.480
Wendy Nilsen, NSF/CISE: I think you can see we were really struggling and so the goal of this initiative in part is to try to bring some of the transformation we've seen in other areas to the area of health.

25
00:04:47.850 --> 00:04:54.780
Wendy Nilsen, NSF/CISE:  And the important thing is here with children. Children are the disadvantages that we see for adults are also in children.

26
00:04:55.050 --> 00:05:10.320
Wendy Nilsen, NSF/CISE: So the US is has the highest probability of a child dying before age five of any of the OECD countries. So if you weren't convinced there's really important reasons why bringing some of the technology we've seen in other areas to health is critical.

27
00:05:13.890 --> 00:05:15.300
Wendy Nilsen, NSF/CISE: Dana, yes.

28
00:05:15.360 --> 00:05:26.580
Dana Hughes NIH-OBSSR: And just about off of what Wendy was saying. We also know that half of the premature deaths in the US are preventable. So if you look at the graph below.

29
00:05:27.390 --> 00:05:39.150
Dana Hughes NIH-OBSSR: Tobacco smoking, as well as diet, physical activity, and activity patterns are really impeding our progress for reducing early deaths. Next slide.

30
00:05:42.990 --> 00:05:43.470
Wendy Nilsen, NSF/CISE: So,

31
00:05:44.670 --> 00:05:54.660
Wendy Nilsen, NSF/CISE: This is just some basic information will be telling you more as we go. But, but to keep in mind this is a cross cutting programs supported from multiple directorates in NSF

32
00:05:55.140 --> 00:06:07.170
Wendy Nilsen, NSF/CISE: and multiple institutes and centers and offices at the National Institute of Health. All proposals are due on February 16 2021 at 5pm in your local time.

33
00:06:07.590 --> 00:06:21.630
Wendy Nilsen, NSF/CISE: Late proposals are not accepted unless there's an extraordinary event: fires, tornadoes, those kind of things, but plan on having your proposal in on time, which usually means submitting getting everything ready early

34
00:06:22.950 --> 00:06:36.150
Wendy Nilsen, NSF/CISE: It's important to note there is one mechanism for this. These are all integrative projects and we'll talk more about that in a moment. They are all $1.2 million in total costs for up to four years.

35
00:06:37.200 --> 00:06:52.170
Wendy Nilsen, NSF/CISE: They are interdisciplinary teams, which we'll talk about more at length, and we have to make a contribution to fundamental science from at least two NSF sciences. And I think what's important is that this is a case of use inspired basic research.

36
00:06:53.940 --> 00:06:57.420
Wendy Nilsen, NSF/CISE: And this is Pasteur's quadrant. For those of you that haven't looked at it.

37
00:06:58.500 --> 00:07:10.230
Wendy Nilsen, NSF/CISE: Don Stokes created this this understanding as a way to think about use inspired basic research and many of you when you think about NSF will think about Niels Bohr.

38
00:07:11.160 --> 00:07:21.960
Wendy Nilsen, NSF/CISE: Basic fundamental physics. And yes, we do that and we're very proud of that. But what's important to note is Louis Pasteur was a microbiologist who did fundamental

39
00:07:22.500 --> 00:07:33.480
Wendy Nilsen, NSF/CISE: Science, but he also was concerned about the output, you know the outcomes in his community. So I think what what you can think of is Louis Pasteur is in the smart health quadrant.

40
00:07:33.960 --> 00:07:46.710
Wendy Nilsen, NSF/CISE: Now that doesn't mean there's anything wrong with being Steve Jobs, who took a lot of the fundamental science, especially that NSF and other Government agency funded and create an applications out of that science.

41
00:07:47.520 --> 00:08:02.010
Wendy Nilsen, NSF/CISE: That's important. But that's not what we're looking for in smart health. What we're looking for is fundamental advances in science that that really address smart health issues so smart, health and biomedical issues.

42
00:08:04.350 --> 00:08:11.430
Wendy Nilsen, NSF/CISE: So our goals here again, like we started with, these are transformative high risk, high reward advances.

43
00:08:11.820 --> 00:08:28.230
Wendy Nilsen, NSF/CISE: We're looking for advances in computer and information science, engineering, mathematics statistics behavioral or cognitive science and we're really trying to make sure that we're answering pressing questions in the biomedical and public health communities.

44
00:08:29.280 --> 00:08:43.200
Wendy Nilsen, NSF/CISE: So what we're looking for is that a that there's a research gap. So when you're thinking about your problem. Think about what is in the fundamental sciences, what gaps are you filling and it fundamental sciences and engineering? What gaps are you filling?

45
00:08:43.980 --> 00:08:53.790
Wendy Nilsen, NSF/CISE: Where is there a gap? It can't be, it's not just, we can do this science, but we've never applied it to this problem before, but the sun is that we have doesn't work in this problem.

46
00:08:54.390 --> 00:09:04.560
Wendy Nilsen, NSF/CISE: You have to have a key health or biomedical problems. So that's gets to having said that gets to the team having an appropriate research expertise in the major areas involved in the work.

47
00:09:04.980 --> 00:09:14.280
Wendy Nilsen, NSF/CISE: Now, we often get a question is can you have, do you need a clinician on these? No, you need the appropriate research expertise. That can be

48
00:09:14.670 --> 00:09:25.110
Wendy Nilsen, NSF/CISE: That can be somebody who's from from biology or public health, it can be a clinical researcher, but what you want is somebody who has the research expertise.

49
00:09:25.500 --> 00:09:31.950
Wendy Nilsen, NSF/CISE: Who knows the research literature in the area that you're doing and to bring that knowledge to your team.

50
00:09:32.520 --> 00:09:49.620
Wendy Nilsen, NSF/CISE: And remember, we're looking at this as something that doesn't, it doesn't duplicate other efforts but it complements those. So if it can go to another program, you should really question whether it should come to smart health, this really is a unique program where we're crossing boundaries.

51
00:09:52.470 --> 00:10:12.780
Dana Hughes NIH-OBSSR: So in terms of smart health research for this iteration, what you'll see here is there are seven different areas. And these are the themes that are outlined in the solicitation and the NIH notice has specifics related to each IC within the themes

52
00:10:14.430 --> 00:10:24.960
Dana Hughes NIH-OBSSR: However, while these are broad themes that we are outlining here for the solicitations purpose, that does not mean that your idea.

53
00:10:26.430 --> 00:10:43.710
Dana Hughes NIH-OBSSR: Necessarily would have to fall into these. So if it does fall outside of this realm that is applicable and you can potentially apply. But again, recommend you talking first to the NSF program officers about your proposal. Next slide.

54
00:10:45.900 --> 00:10:50.130
Dana Hughes NIH-OBSSR: So information infrastructure is one of these themes

55
00:10:51.900 --> 00:11:04.140
Dana Hughes NIH-OBSSR: We have broken that out into sort of three sub buckets. So scalable and interoperable systems. So this could be scalable digital infrastructure languages tools.

56
00:11:04.920 --> 00:11:17.460
Dana Hughes NIH-OBSSR: Fair data sharing and use enhance knowledge representation, such as on technologies and ensuring high confidence security and privacy.

57
00:11:17.940 --> 00:11:32.460
Dana Hughes NIH-OBSSR: So this could be methods for controlling and maintain at maintaining data integrity and security and privacy providing trustworthy patient identification and authentication services as well. Next slide.

58
00:11:34.470 --> 00:11:36.330
Dana Hughes NIH-OBSSR: Another theme that we have

59
00:11:37.530 --> 00:11:56.100
Dana Hughes NIH-OBSSR: Is transformative data science. So this could be fusion and analysis and different computational approaches to do that different visualization and modeling, as well as visual behavioral and contextual related data science research.

60
00:11:57.240 --> 00:11:57.870
Next slide.

61
00:12:00.480 --> 00:12:13.200
Wendy Nilsen, NSF/CISE: And I just want to say, well, we're emphasizing artificial intelligence here. Not all programs, not all proposals, will will focus on artificial intelligence. If you think broadly about it, that is the role of innovative

62
00:12:13.950 --> 00:12:26.640
Wendy Nilsen, NSF/CISE: analytics. It will come in and many places, including the novel multimodal sensor systems. Again, we're looking for integrated sensor systems, not multiple small sensors, but an integrated system.

63
00:12:27.420 --> 00:12:35.760
Wendy Nilsen, NSF/CISE: That can do multimodal intergration and interogating multiple system states. These should be adjustable, just in time, implantable, wearable, or mobile

64
00:12:36.240 --> 00:12:44.970
Wendy Nilsen, NSF/CISE: So they can monitor biomarkers through various mechanisms. And again, we would like multimodal so it isn't just different kinds of electro-chemical signals.

65
00:12:45.270 --> 00:12:54.510
Wendy Nilsen, NSF/CISE: But they could be biochemical, magnetic, acoustic, electrical, all sorts of different ways but bringing them together into a system so we could integrate that data.

66
00:12:54.900 --> 00:13:03.870
Wendy Nilsen, NSF/CISE: And then using sensor data to inform the development of innovative AI, machine learning, math and statistics approaches to build predictive models.

67
00:13:07.980 --> 00:13:19.080
Wendy Nilsen, NSF/CISE: We also have. We also have stressed it's on effective usability. We can create the best technologies around. But if it isn't user- and context-centered,

68
00:13:19.920 --> 00:13:31.320
Wendy Nilsen, NSF/CISE: it won't help us into getting into a new place. So thinking about the user, thinking about the context, it won't help me if I get a message right now it says I should take a walk.

69
00:13:31.830 --> 00:13:41.460
Wendy Nilsen, NSF/CISE: Not during the webinar. So I need to know the context. Then I need this to approach the person in a way. So I wanted to reduce burden and also increase autonomy.

70
00:13:41.910 --> 00:13:53.070
Wendy Nilsen, NSF/CISE: We also want to be able to to address diverse users and effectively support user's across different socioeconomic status. Think about digital, think about it

71
00:13:53.460 --> 00:14:09.240
Wendy Nilsen, NSF/CISE: in broadband and communities without broadband. Communities with different levels of health literacy, gender, ethnicity, you can think of many ways that these are diverse users. But it's important that those are considered when we think about usability.

72
00:14:14.670 --> 00:14:19.380
Wendy Nilsen, NSF/CISE: Okay, automating health if you think of our closed loop systems.

73
00:14:21.330 --> 00:14:31.650
Wendy Nilsen, NSF/CISE: Closed loop systems are really important. And can we think about these in these mission critical systems. So they're closed loop, human in the loop. We know it's been hard to automate health.

74
00:14:32.400 --> 00:14:43.890
Wendy Nilsen, NSF/CISE: Can we put a new human in the loop? These are controls that can automatically do your analysis, set it up that your system is triggered to do specific things at specific times.

75
00:14:44.610 --> 00:14:56.460
Wendy Nilsen, NSF/CISE: You can also automate the systems for novel simulation and modeling. This is important for for many reasons for it for discovery for all sorts of things, but also enhancing digital clinical trials.

76
00:14:56.970 --> 00:15:05.730
Wendy Nilsen, NSF/CISE: And we're also looking for generalizable platforms. Again, this is all foundational science, but can we use them across the range of settings, not just

77
00:15:06.060 --> 00:15:15.090
Wendy Nilsen, NSF/CISE: here's what we'll do with this group in this place. But if you think about some of the great advances in technology, it's because you could use them across settings.

78
00:15:17.490 --> 00:15:21.090
Wendy Nilsen, NSF/CISE: We think about medical and image interpretation.

79
00:15:21.480 --> 00:15:35.940
Wendy Nilsen, NSF/CISE: We've looked a lot at how a I could do this, but we look less at how human pattern recognition, visual searches, perceptual learning, attentional biases, things like that, can really affect how observers are understanding images.

80
00:15:36.420 --> 00:15:49.080
Wendy Nilsen, NSF/CISE: So we really want to understand the context when you're when you're doing an image analysis or when you're looking at an image, what kind of things, effective, both in the context and the environment, including the human's own environment.

81
00:15:50.190 --> 00:16:03.390
Wendy Nilsen, NSF/CISE: But also ways to exploit these this understanding. Can we do a better job at providing images in a way that they help perceptual decision making and this could be in 3D or 40 models.

82
00:16:05.580 --> 00:16:08.460
Dana Hughes NIH-OBSSR: And then also to another theme that

83
00:16:09.570 --> 00:16:17.070
Dana Hughes NIH-OBSSR: Is not mutually exclusive to the other ones is unpacking health disparities and like Wendy mentioned before.

84
00:16:18.030 --> 00:16:28.710
Dana Hughes NIH-OBSSR: Just like some of the other themes. This is not limiting proposals to just the AI area, but can also be more broad, within the mobile health and digital

85
00:16:29.400 --> 00:16:42.870
Dana Hughes NIH-OBSSR: health space. But within unpacking health disparities, this is really more focused on ways to reduce assess and mitigate those disparities, whether that's using data driven approaches and models to address.

86
00:16:43.980 --> 00:16:53.250
Dana Hughes NIH-OBSSR: Issues with like social determinants of health. Strategies to measure and reduce and mitigate the effects of racism

87
00:16:54.000 --> 00:17:08.370
Dana Hughes NIH-OBSSR: and bias and looking at their impacts on health outcomes. As well as novel methods to distinguish between the complex pathways between in and among levels of influence and domains. Next slide.

88
00:17:11.100 --> 00:17:13.680
Dana Hughes NIH-OBSSR: So the review process for smart connected health

89
00:17:14.880 --> 00:17:26.940
Dana Hughes NIH-OBSSR: The applications will come in to NSF and then there is a joint NSF and NIH set of review panels that are that are held

90
00:17:27.420 --> 00:17:48.720
Dana Hughes NIH-OBSSR: And those are open to the NSF program directors, as well as there is an NIH scientific review officer that attends and the applications are scored within the scoring system for both agencies, so it will receive both a NSF score as well as an NIH score.

91
00:17:50.700 --> 00:18:02.340
Dana Hughes NIH-OBSSR: The reviews and summaries are released back to PIs and made available on both the NSF and the NIH side.

92
00:18:03.330 --> 00:18:20.430
Dana Hughes NIH-OBSSR: At that point, there is a set of internal discussions that occur. And at that time, and NIH may make selections of applications to come over to different institutes and centers on the NIH side.

93
00:18:21.690 --> 00:18:39.270
Dana Hughes NIH-OBSSR: If your proposal is selected as one that NIH would like to fund the PIs will be contacted directly and asked to submit your application in the NIH form and you will work closely with one of the program officers.

94
00:18:40.680 --> 00:18:44.670
Dana Hughes NIH-OBSSR: Involved on the NIH side to make those changes.

95
00:18:45.900 --> 00:18:54.840
Dana Hughes NIH-OBSSR: In a timely fashion and then award recommendations are made on both sides and the applications are then funded. Next slide.

96
00:18:55.770 --> 00:18:58.350
Wendy Nilsen, NSF/CISE: One thing I do want to add to this slide too, is

97
00:18:59.370 --> 00:19:08.640
Wendy Nilsen, NSF/CISE: All the PIs will get the NSF reviews and summaries. They do not get NIH scores. Those are not added unless a proposal was going over to NIH.

98
00:19:10.050 --> 00:19:22.020
Wendy Nilsen, NSF/CISE: So our NSF merit review elements. These are the standard information that we have for that were set up by the National Science Board and what's important is for NSF

99
00:19:22.590 --> 00:19:28.830
Wendy Nilsen, NSF/CISE: is that the intellectual merits and broader impacts are the same criteria and you should consider them in a top down way.

100
00:19:29.250 --> 00:19:40.140
Wendy Nilsen, NSF/CISE: But our primary criteria here is what's the potential for the pros proposed activity to advance knowledge and understanding his own field or across different fields.

101
00:19:40.650 --> 00:19:49.770
Wendy Nilsen, NSF/CISE: And what is the benefit to society or advanced desired societal outcome, which is the broader impacts. These two are key to how we look at these proposals.

102
00:19:50.250 --> 00:20:01.050
Wendy Nilsen, NSF/CISE: And I think we've implied it, but I do want to say in this initiative NIH and NSF are looking for the same thing. We're looking for these advances in foundational science.

103
00:20:02.070 --> 00:20:08.400
Wendy Nilsen, NSF/CISE: We're also looking at whether the proposed activities. You know what's creative original and potentially transformative.

104
00:20:09.000 --> 00:20:19.170
Wendy Nilsen, NSF/CISE: Is there a good plan for this? Is that the right team. Do they have institutional support? Are there resources available to the to the PI ad the team that would help them

105
00:20:20.340 --> 00:20:22.770
Wendy Nilsen, NSF/CISE: have a really great and transformative proposal?

106
00:20:24.510 --> 00:20:30.870
Wendy Nilsen, NSF/CISE: Broader impacts are important. They're, they're implicit does you know when you're publishing at the top conferences.

107
00:20:31.290 --> 00:20:37.110
Wendy Nilsen, NSF/CISE: That's that's critical to sharing knowledge. There's explicit and we have technology transfer and you're

108
00:20:37.710 --> 00:20:45.270
Wendy Nilsen, NSF/CISE: Doing results and disseminating, but also what's really important here as broader impacts include research and education.

109
00:20:45.810 --> 00:20:53.130
Wendy Nilsen, NSF/CISE: So what about development of curriculum? Supporting materials? Student involvement in emerging research and technology.

110
00:20:53.580 --> 00:21:00.720
Wendy Nilsen, NSF/CISE: Postdoctoral training. If someone is going to include a postdoc in their program, they need to have a plan, not just

111
00:21:01.140 --> 00:21:07.020
Wendy Nilsen, NSF/CISE: what they're doing, but how are they going to develop that postdoc into an independent researchers

112
00:21:07.620 --> 00:21:16.710
Wendy Nilsen, NSF/CISE: And then the data management plan is required as an all NSF proposals. We're also looking looking for broadening participation underrepresented groups.

113
00:21:17.160 --> 00:21:24.210
Wendy Nilsen, NSF/CISE: Smart health is an excellent way to broaden participation, because you will find that students that were not necessarily interested

114
00:21:24.540 --> 00:21:36.480
Wendy Nilsen, NSF/CISE: in some scientific areas, are really interested in these issues that approach their community. So it's been it's been a wonderful area for to see that broadening participation happen.

115
00:21:39.300 --> 00:21:48.240
Wendy Nilsen, NSF/CISE: So our review criteria also includes a scientific requirement or collaboration management plan and the evaluation plan.

116
00:21:50.010 --> 00:21:58.770
Wendy Nilsen, NSF/CISE: And the science areas. I think this is important, we realized that there are every discipline,  area of science or engineering area.

117
00:22:00.060 --> 00:22:09.120
Wendy Nilsen, NSF/CISE: We are talking, when we're talking about science and engineering, we are talking about the NSF areas so computer and information science,

118
00:22:09.510 --> 00:22:15.660
Wendy Nilsen, NSF/CISE: many different areas database into, you know, artificial intelligence informatics machine learning.

119
00:22:16.200 --> 00:22:25.650
Wendy Nilsen, NSF/CISE: In engineering we have sensor technology, signal processing, which also includes optimization and operations research and closed loop systems,

120
00:22:26.400 --> 00:22:36.450
Wendy Nilsen, NSF/CISE: dynamic systems. All sorts of different ways in mathematics and statistics. Again, these all work in important ways together and strengthen each other.

121
00:22:37.230 --> 00:22:48.540
Wendy Nilsen, NSF/CISE: Math and statistics and stochastic modeling, analysis interpretation, characterizing uncertainty. There's others that are there too. And then the social behavior and cognitive sciences.

122
00:22:49.440 --> 00:22:57.840
Wendy Nilsen, NSF/CISE: There's areas like perception, social psychology, cognition, emotion, but there's also some of the social sciences of economics, ethics.

123
00:22:58.260 --> 00:23:09.360
Wendy Nilsen, NSF/CISE: and then linguistics. So I think that there's this is important that when we talk about two fundamental science or engineering areas we're talking about two out of the NSF science areas.

124
00:23:11.460 --> 00:23:18.000
Dana Hughes NIH-OBSSR: As Wendy mentioned earlier, as well, your proposals will have to include a collaboration plan.

125
00:23:19.560 --> 00:23:24.780
Dana Hughes NIH-OBSSR: And it will be a supplementary document that is no more than two pages.

126
00:23:25.860 --> 00:23:38.850
Dana Hughes NIH-OBSSR: The plan must include a description of the team in their roles, a plan for integration and ways to support that collaboration and the team members individual expertise for the project. And really

127
00:23:39.540 --> 00:23:53.430
Dana Hughes NIH-OBSSR: this is important to make sure that you are describing how your team is going to be working together and how the individual expertise is of those on your team are fitting together

128
00:23:54.180 --> 00:24:02.460
Dana Hughes NIH-OBSSR: to help address your proposal. And , so for example, you know, if you have two

129
00:24:02.880 --> 00:24:17.010
Dana Hughes NIH-OBSSR: sites across the US. You should include information about how you plan to collaborate. Will you do zoom calls are you going to meet up at different conferences? how are you going to actually share knowledge and work together on this?

130
00:24:17.340 --> 00:24:38.040
Dana Hughes NIH-OBSSR: And as Wendy mentioned before, but I will just reiterate this again the expertise that you have on your project should make sense. You don't need to have necessarily a clinician, but you do need to have people who fill the requisite scientific expertise for your proposals. Next slide.

131
00:24:41.850 --> 00:24:51.960
Wendy Nilsen, NSF/CISE: So all proposal's also have an evaluation component. It's important to note that at NSF, we do not require preliminary data that is not a requirement.

132
00:24:52.500 --> 00:25:04.470
Wendy Nilsen, NSF/CISE: Part of your proposal part of your collaboration plan will convince the panel that even without color and even without preliminary data that you will be, you have the skills and ability to do this.

133
00:25:04.830 --> 00:25:13.110
Wendy Nilsen, NSF/CISE: The evaluation will help people understand how you're going to evaluate it. So you can have technical functioning. If you're using millimeter wave

134
00:25:13.530 --> 00:25:20.280
Wendy Nilsen, NSF/CISE: To to assess something or you're you're reading you're using it signal processing to understand 

135
00:25:20.880 --> 00:25:27.030
Wendy Nilsen, NSF/CISE: something. You could say is the algorithms doing what I think it is?

136
00:25:27.600 --> 00:25:38.220
Wendy Nilsen, NSF/CISE: Okay. Is it valid? Is it is it actually measuring what I think it is? Is it reliable? Is it  consistently measuring these things? If you're doing it with with people, if you're doing it with people

137
00:25:38.970 --> 00:25:51.990
Wendy Nilsen, NSF/CISE: who are with users, is it usable? You might pilot this and let some people use your technology. So let's say you were creating new equipment, you might have the users of that equipment trying it.

138
00:25:52.860 --> 00:25:56.970
Wendy Nilsen, NSF/CISE: Also, if you're doing something that you think will have a biomedical or health outcome.

139
00:25:57.540 --> 00:26:05.160
Wendy Nilsen, NSF/CISE: You want to do some of those things. Now remember evaluation doesn't include randomized clinical trials in this initiative.

140
00:26:05.580 --> 00:26:15.180
Wendy Nilsen, NSF/CISE: We know that some of the new guidance from NIH on clinical trials includes many different things. What we're talking about here is randomized trials.

141
00:26:15.780 --> 00:26:28.260
Wendy Nilsen, NSF/CISE: Where we have a control group and a treatment group and they are randomized. These are early stage proposals, they are not ready for randomized clinical trial and they expense and the risks that are associated with it.

142
00:26:34.530 --> 00:26:40.410
Wendy Nilsen, NSF/CISE: So just a few additional comments. What kind of things are not appropriate for the solicitation.

143
00:26:40.920 --> 00:26:57.570
Wendy Nilsen, NSF/CISE: If you are focusing on biological biomedical or public health research and you do not have fundamental science so often I get an email that says I'd like to do research drawing data out of the EHR. We're going to be using. We're going to be using

144
00:26:59.670 --> 00:27:04.800
Wendy Nilsen, NSF/CISE: test data out of the EHR to do predictions of health outcomes.

145
00:27:06.000 --> 00:27:13.440
Wendy Nilsen, NSF/CISE: There's a lot of research now that would allow us to do that. It may not be directly there. But what we were looking for is

146
00:27:13.740 --> 00:27:24.300
Wendy Nilsen, NSF/CISE: Is there something new that you're doing. Are there new ways that you're looking at your analytics. If you're sensing. Is it new ways your multimodal sensor system will be approaching a problem.

147
00:27:25.410 --> 00:27:30.510
Wendy Nilsen, NSF/CISE: Really, what you're doing is it that fundamental science and engineering is really advancing there.

148
00:27:32.730 --> 00:27:37.230
Wendy Nilsen, NSF/CISE: The other thing is if you're if you're saying, nobody's applied this thus far.

149
00:27:38.340 --> 00:27:49.080
Wendy Nilsen, NSF/CISE: If there's a reason why you would think it would work. And you have to, you have to create a new a new system, a new algorithm. If you're looking at how people are perceiving images.

150
00:27:49.740 --> 00:27:59.370
Wendy Nilsen, NSF/CISE: You can't say people perceive all these other images in this way we understand how people perceive things, but when it comes to medical images we think we don't understand it there.

151
00:27:59.880 --> 00:28:06.930
Wendy Nilsen, NSF/CISE: It really want to know more about why that medical was different? Why that knowledge was going to be in a very different way than it was before.

152
00:28:08.070 --> 00:28:19.230
Wendy Nilsen, NSF/CISE: Also, we don't, we're not looking for topics that fit the mission on another agency. So again, if it's, if it's a topics that is driven by a health question, where the advances are for health,

153
00:28:20.040 --> 00:28:27.870
Wendy Nilsen, NSF/CISE: then it should be going to NIH directly. I think if you have a question about this. These are great questions to ask program officers.

154
00:28:30.030 --> 00:28:38.670
Dana Hughes NIH-OBSSR: And no conversation these days would be complete without mentioning what's going on with the pandemic and COVID-19 and I know that

155
00:28:39.870 --> 00:28:57.330
Dana Hughes NIH-OBSSR: Wendy and my NSF colleagues as well as my NIH colleagues have gotten a lot of questions about the applicability of proposals coming in addressing COVID-19. And while you could submit to smart connected health with

156
00:28:58.350 --> 00:29:08.670
Dana Hughes NIH-OBSSR: a proposal that's within that area, you really need to make sure that what it is that you're going to be proposing to do is timely.

157
00:29:09.600 --> 00:29:23.550
Dana Hughes NIH-OBSSR: Given the rapid nature of the response to this, and the duration of a four year award. And again, focusing on that fundamental scientific contribution, 

158
00:29:24.450 --> 00:29:41.760
Dana Hughes NIH-OBSSR: needs to be utmost priority for this. And also, I'll just say, you know, while you could submit specifically to with an idea is for COVID and the solution is addressing some sort of

159
00:29:42.840 --> 00:29:45.750
Dana Hughes NIH-OBSSR: aspect of the response.

160
00:29:46.770 --> 00:29:59.190
Dana Hughes NIH-OBSSR: It may be given the timeline better to couch some of this within a broader pandemic response or on disease response and not necessarily tie it to this use case. Next slide.

161
00:30:01.560 --> 00:30:05.370
Wendy Nilsen, NSF/CISE: So you're going to want to know where to get feedback on your proposal and

162
00:30:06.300 --> 00:30:16.410
Wendy Nilsen, NSF/CISE: You send it,if you send an email to SCH-correspondence@nsf.gov of your idea, send no more than one page. Don't call everybody.

163
00:30:17.040 --> 00:30:26.460
Wendy Nilsen, NSF/CISE: It's hard to get good feedback on the fly. So please send us a one pager that briefly describes what project you're trying to do.

164
00:30:27.300 --> 00:30:36.480
Wendy Nilsen, NSF/CISE: Sorry, what's the problem. What are you doing? The intellectual merit, that this the scientific advances, you expect to make it fundamental science and engineering.

165
00:30:36.960 --> 00:30:47.190
Wendy Nilsen, NSF/CISE: And then the broader impacts. If you send a one pager. We can give you much better feedback. We can also schedule times to talk to you about this. If you feel like if we feel like this is not

166
00:30:48.120 --> 00:30:58.830
Wendy Nilsen, NSF/CISE: going to get you an answer through an email. But again, don't call, send an email with a summary. I often get people that email and say I have a whole bunch of ideas that I want to talk to you about

167
00:30:59.160 --> 00:31:03.840
Wendy Nilsen, NSF/CISE: We can't give you good information if you're asking us about a whole bunch of things on the fly.

168
00:31:05.070 --> 00:31:13.350
Wendy Nilsen, NSF/CISE: My colleagues and I talked to each other all the time. I might send an email to my colleague Scott or Wei Ding or somebody else and say,

169
00:31:14.550 --> 00:31:22.560
Wendy Nilsen, NSF/CISE: Here, I get this proposal, what do you think this is in your area not mind sending that one page or get you much, much better feedback.

170
00:31:25.500 --> 00:31:45.660
Wendy Nilsen, NSF/CISE: So how do you apply? if you look at our website for NSF 21-530 you will read all the instructions. On applying, please make sure you read the instructions. If you don't read the instructions, proposals can be returned without review and we hate to do that.

171
00:31:47.340 --> 00:31:55.980
Wendy Nilsen, NSF/CISE: For NIH please look at the guide that Dana mentioned it's an important document there tells you some of the priorities and NIH.

172
00:31:56.850 --> 00:32:04.440
Wendy Nilsen, NSF/CISE: Also look at the NSF proposals and award policies and procedures guide that is your best friend when you're applying.

173
00:32:05.010 --> 00:32:22.260
Wendy Nilsen, NSF/CISE: And remember you can always send an email to the SCH-correspondence@NSF.gov That will give a, there's a whole group of us that will do that. Don't send individual emails to a group, you will get many different answers from us. We work much better as the smart health team.

174
00:32:27.270 --> 00:32:42.150
Wendy Nilsen, NSF/CISE: Please feel free to join our listserv we put out information all the time or also, we also put out request to look for panelists for these so please feel free to sign up for our listserv. And we will be posting the slides afterwards.

175
00:32:44.010 --> 00:32:52.410
Wendy Nilsen, NSF/CISE: Also feel free to look at the NSF website. I always get people asking me questions about smart health and it's very clear. They haven't looked at the

176
00:32:53.040 --> 00:33:06.030
Wendy Nilsen, NSF/CISE: at the awards that have been made this far, because many of the questions would be answered if you looked at awards, you would say, oh, they are interested in that. So there's many, many things that will be an interest in the NSF website.

177
00:33:08.280 --> 00:33:15.810
Dana Hughes NIH-OBSSR: And just to wrap up really this is not a race. This is a relay and between basic and applied science.

178
00:33:16.890 --> 00:33:26.700
Dana Hughes NIH-OBSSR: And again, just you really need to focus on putting together the team that takes the idea across the finish line from start to end.

179
00:33:29.790 --> 00:33:36.300
Wendy Nilsen, NSF/CISE: So with that we will we will go to our question and answer section. I know my colleagues are here.

180
00:33:36.840 --> 00:33:46.470
Wendy Nilsen, NSF/CISE: I do want to say, I apologize to my colleague Steve Breckler from SBE and the BCS division, because I believe I missed him in my introduction. So

181
00:33:47.460 --> 00:33:59.730
Wendy Nilsen, NSF/CISE: we have wonderful colleagues here from from both NIH and NSF. So, my colleague Goli Yamini will moderate our questions section and we will, we have a some of the questions we will

182
00:34:01.440 --> 00:34:07.830
Wendy Nilsen, NSF/CISE: stop sharing and work on the screen. We can answer from the team because I think there's an amazing team here.

183
00:34:09.150 --> 00:34:10.350
Wendy Nilsen, NSF/CISE: So Goli.

184
00:34:10.860 --> 00:34:22.230
Goli Yamini (NSF/CISE): Yes, hi again. So our panelists. I've been answering questions throughout the presentation. So a lot of the questions have been getting answers, but if you have

185
00:34:22.920 --> 00:34:31.140
Goli Yamini (NSF/CISE): Questions that you would want to, you know, present to them, please put them in the Q&A and I'll read them out loud so so our panelists could answer.

186
00:34:32.190 --> 00:34:36.690
Goli Yamini (NSF/CISE): There are a few that I'm trying to just catch some that haven't been answered between everybody

187
00:34:37.920 --> 00:34:41.280
Goli Yamini (NSF/CISE): Questions. There are a few questions about

188
00:34:42.990 --> 00:34:45.720
Goli Yamini (NSF/CISE): The length of the project, budgetary and

189
00:34:47.190 --> 00:34:57.780
Goli Yamini (NSF/CISE): Some dynamics between NIH and NSF and they can present those and then talk about some of the scientific ones. For example, some people are asking do is a project required to be four years, or can it be shorter

190
00:35:04.380 --> 00:35:14.010
Goli Yamini (NSF/CISE): And also, if a project is selected by NIH, can the budget be increased if anybody wants to answer those. Are we can make him move on to other questions.

191
00:35:14.250 --> 00:35:14.910
Sylvia Spengler: I'll take it.

192
00:35:16.290 --> 00:35:18.840
Sylvia Spengler: I'll take it. Sorry. Sorry for the lack of a camera.

193
00:35:20.310 --> 00:35:24.270
Sylvia Spengler: The project can be less than four years.

194
00:35:25.800 --> 00:35:40.230
Sylvia Spengler: But you want to think about how you're going to design for impact and have time to actually provide for evaluation. Of the other question. And once the budget goes to NIH, it does not get changed.

195
00:35:42.330 --> 00:35:49.560
Goli Yamini (NSF/CISE): Canada teams include international partners on what is a rough estimate of team size that would to be enough.

196
00:35:58.380 --> 00:36:01.050
Wendy Nilsen, NSF/CISE: So maybe I'll pick that up. Sylvia, would you like to go

197
00:36:01.470 --> 00:36:02.760
Sylvia Spengler: I'll start and

198
00:36:04.530 --> 00:36:21.840
Sylvia Spengler: And so I you may have international partners. They cannot be the lead, so that you would have to and and NSF has some some fairly strict rules about where people can come from.

199
00:36:24.300 --> 00:36:48.330
Sylvia Spengler: The a team can be as few as two people. But the reality is the breadth of the science that we're asking you to address and the different skill sets, often mean that you need a third or a fourth person involved and I'm talking about leadership. I'm not talking about students, etc.

200
00:36:49.890 --> 00:37:03.570
Dana Hughes NIH-OBSSR: And Marcus, I'll just add the Sylvia to that some analyses that we've done on the NIH side, it's about an average of three PIs listed per proposal so as an average of what's been funded in the past.

201
00:37:04.800 --> 00:37:10.500
Wendy Nilsen, NSF/CISE: And there can be an international component, but you have to justify your international component

202
00:37:10.980 --> 00:37:26.730
Wendy Nilsen, NSF/CISE: It can't be generally is not a project, all in another country, but it can be it can have international components, but it has to be justified and why what we're learning for the US in that other country. So be, be aware that people will be looking at it through that lens.

203
00:37:28.140 --> 00:37:37.380
Goli Yamini (NSF/CISE): I'm speaking of collaborations. There have been a few questions about whether partnerships or collaborations can be formed with companies developing technologies.

204
00:37:41.970 --> 00:37:49.440
Wendy Nilsen, NSF/CISE: So maybe I deal with most of the companies. So, um, so you can have a sub award in a proposal to a company

205
00:37:50.370 --> 00:38:06.000
Wendy Nilsen, NSF/CISE: That's, that's not unusual. And it may be a good way to get the science out into the community, but all of the applicants have to be academic institutions or nonprofits. So company cannot lead. I will also say if you're partnering with a company,

206
00:38:07.560 --> 00:38:16.260
Wendy Nilsen, NSF/CISE: I would make sure that your intellectual merit isn't all in the company, because that will often require them to keep secrets because of their technology.

207
00:38:17.220 --> 00:38:32.130
Wendy Nilsen, NSF/CISE: That makes it very hard for a panel to be able to assess what exactly is happening in a proposal. So having a having, having a commercial partner is fine, but you may not want it to be the primary driver of what you're doing.

208
00:38:34.800 --> 00:38:48.000
Goli Yamini (NSF/CISE): And then some of other questions are about partnerships between with medical teams. Can half the partnership, you know, half the team be on the medical side?

209
00:38:49.680 --> 00:38:50.130
Goli Yamini (NSF/CISE): And

210
00:38:52.830 --> 00:39:05.820
Sylvia Spengler: So it's this is research. It's an intellectual question you're more you're more we're more concerned about what the contributions of the participants are rather than

211
00:39:06.990 --> 00:39:29.370
Sylvia Spengler: what kind of institution they particularly whether they're in a medical institution or research foundation or in academia, we're less worried about that than we are about the innovation and in the question the interrelationship across the sciences and the advances there

212
00:39:30.960 --> 00:39:33.240
Sylvia Spengler: and how you will evaluate?

213
00:39:37.080 --> 00:39:42.870
Goli Yamini (NSF/CISE): What this question is a follow up to when these answers. Like, what if the company is committed to open source principles.

214
00:39:44.310 --> 00:39:46.290
Wendy Nilsen, NSF/CISE: That helps, that definitely helps.

215
00:39:47.760 --> 00:39:50.580
Goli Yamini (NSF/CISE): Um, can national labs be leads or partners.

216
00:39:52.650 --> 00:40:01.980
Wendy Nilsen, NSF/CISE: So National Labs can be leads National Lab cannot receive salary. So, we, the government, doesn't pay for what the government's already paid for.

217
00:40:02.790 --> 00:40:13.770
Wendy Nilsen, NSF/CISE: If you're on a national lab, I suggest you contact me reach out directly and we can work through the logistics of how it is. We value our colleagues very much and our national labs.

218
00:40:14.520 --> 00:40:21.180
Wendy Nilsen, NSF/CISE: But we also would like to make sure that it works correctly. So please reach out and we'll work that out on a case by case basis.

219
00:40:22.770 --> 00:40:36.720
Goli Yamini (NSF/CISE): And they've also been some questions that keep coming up about the timeline, the date of the proposal, the due date for the proposals that are listed and then approximately how long the review process takes so if you guys just touch upon that.

220
00:40:37.650 --> 00:40:41.220
Sylvia Spengler: NSF is committed, in normal times,

221
00:40:42.510 --> 00:40:59.010
Sylvia Spengler: to responding to more than 80% of the applicants within a six month period. We generally get most of the awards done within that six month period. We certainly

222
00:40:59.700 --> 00:41:14.310
Sylvia Spengler: aim to get the declines done, but not all of the awards can happen. But within six months of the date of submission, but it's certainly more than 80% actually usually more than 95%

223
00:41:16.920 --> 00:41:30.600
Dana Hughes NIH-OBSSR: And I'll say to so the as a reminder for everybody, the proposals are due on February 16 and I believe when you said 5pm local time of where you are submitting from

224
00:41:31.710 --> 00:41:35.460
Dana Hughes NIH-OBSSR: In terms of on the NIH side. If your application is

225
00:41:36.600 --> 00:41:37.800
Dana Hughes NIH-OBSSR: selected for funding, 

226
00:41:39.600 --> 00:41:56.610
Dana Hughes NIH-OBSSR: it usually takes a little bit longer than what Sylvia mentioned. And so from date of submission, probably no more than nine months, probably about eight, it will most likely go to for this round.

227
00:41:57.630 --> 00:42:01.380
Dana Hughes NIH-OBSSR: to a council in the fall and then be funded shortly thereafter.

228
00:42:04.860 --> 00:42:16.620
Goli Yamini (NSF/CISE): I'm a few questions about PI experience can the lead PI be an assistant professor and is there any kind of positives for new PI's.

229
00:42:19.980 --> 00:42:24.450
Wendy Nilsen, NSF/CISE: I think, Scott, why don't you answer that, when you've had experience with this.

230
00:42:27.030 --> 00:42:39.180
Scott Acton NSF signal and image processing/ML: I'm certainly there is no expectation for rank or leadership by a full professor, for example. Assistant professors are more than welcome to contribute and have been successful in the past.

231
00:42:44.280 --> 00:42:56.730
Wendy Nilsen, NSF/CISE: I think that's another one to, if you look at our if you look at the proposals that have been funded, you will see a range of experiences and those teams. So, fabulous PIs are fabulous PIs.

232
00:42:57.990 --> 00:43:01.800
Goli Yamini (NSF/CISE): So, somebody asked, with post docs. I mean, it could a post doc be a PI?

233
00:43:06.600 --> 00:43:07.800
Sylvia Spengler: Oops, sorry, Wendy.

234
00:43:09.570 --> 00:43:16.950
Sylvia Spengler: The ability to be a PI or a PI is something which is established by an institution. It is not a question for

235
00:43:17.700 --> 00:43:42.270
Sylvia Spengler: for NSF, per se. If your institution allows you to be called a PI, then you can be called you can be a PI, you want to think about the time available and possibility of commitment to the organization for the term of the award given the generally tenuous nature of postdoctoral appointments.

236
00:43:45.630 --> 00:44:03.720
Goli Yamini (NSF/CISE): There are also some topic related questions. And those may be more specific to the PI's. But there's some questions about assistive technology and are they within the scope of this solicitation. There are others about how strong is an emphasis on multiple. I mean, or multimodal sensors.

237
00:44:05.790 --> 00:44:08.490
Wendy Nilsen, NSF/CISE: Maybe John could talk about multimodal sensors.

238
00:44:09.810 --> 00:44:16.410
John X.J. Zhang (NSF/ECCS): That's right. Hi everybody, and I replied that a couple questions on the chat. I think in terms of the hardware,

239
00:44:17.100 --> 00:44:20.040
John X.J. Zhang (NSF/ECCS): you're more than welcome to send me email, but just a quick answer.

240
00:44:20.400 --> 00:44:28.170
John X.J. Zhang (NSF/ECCS): Think from the reviewer end, right so if I have a single module on a sensor that's not saying, no, no. But if you propose specific model, 

241
00:44:28.500 --> 00:44:40.050
John X.J. Zhang (NSF/ECCS): Added a multimodal think about making nobody clear and clearly articulate. So from review perspective is competitive side more than happy to you further elaborate if you're sending me email offline.

242
00:44:44.250 --> 00:44:53.010
Goli Yamini (NSF/CISE): Okay, I'm seeing that many new questions there are more topic. Most from before, but like john said a lot of the panelists have been answering those live.

243
00:44:55.650 --> 00:45:14.340
Dana Hughes NIH-OBSSR: So I say one here, Goli, that's probably worthwhile from an NIH perspective about ESI status.  So these ,if this is funded on the NIH side and is brought over, it comes over as an ROI. And so because of that

244
00:45:15.510 --> 00:45:20.280
Dana Hughes NIH-OBSSR: Yes, this will impact the ESI status of the of the investigator.

245
00:45:23.610 --> 00:45:34.590
Goli Yamini (NSF/CISE): Kind of between that NSF and NIH thing if people are asking if the proposals will effectively be reviewed twice. Is that how the process, it would you describe the process that way.

246
00:45:35.730 --> 00:45:48.300
Dana Hughes NIH-OBSSR: No, I wouldn't. So there are reviewed once and they go through the traditional NSF review with participation of the NIH counterparts and so

247
00:45:50.700 --> 00:46:00.930
Dana Hughes NIH-OBSSR: Instead of having separate review for proposals that are selected by NIH, what happens is is that during the review.

248
00:46:01.650 --> 00:46:22.890
Dana Hughes NIH-OBSSR: at NSF, there's an NIH scientific review officer that is assigned to that panel that is there that provides guidance to the reviewers about the NIH scoring and what is necessary for proposals and they sit in. They listen. They take their notes as if

249
00:46:24.240 --> 00:46:42.000
Dana Hughes NIH-OBSSR: They would in a normal study section and NSF conducts the rest of the process. The initial sort of review and scores that are led out as Wendy indicated are all in the NSF format, you will not receive an NIH score.

250
00:46:43.230 --> 00:47:00.570
Dana Hughes NIH-OBSSR: And then if your application is selected to come over to NIH, you will work with the program officer on the NIH side to reformat your application and your application will go into a what we call a dummy panel and so that avoid you having to have your proposal rereviewed.

251
00:47:02.220 --> 00:47:15.840
Dana Hughes NIH-OBSSR: And it's at that point when it's in the NIH system that whatever the score that you received during that NSF review. That will be put on whatever summary statement is released for those proposals.

252
00:47:21.750 --> 00:47:28.500
Goli Yamini (NSF/CISE): Does this program have any correlation, combined with NSF mid career advanced management program.

253
00:47:34.440 --> 00:47:35.670
Wendy Nilsen, NSF/CISE: Sorry Goli, I missed that one.

254
00:47:36.900 --> 00:47:43.440
Goli Yamini (NSF/CISE): Does this program have any correlation or is it combined with the NSF mid career advancement program.

255
00:47:44.250 --> 00:47:59.580
Wendy Nilsen, NSF/CISE: So the mid career, the NSF mid career advancement program, is a separate initiative. So if people are interested in the mid career initiative, they should talk to PDs is listed on that piece and program directors listed on that announcement.

256
00:48:01.230 --> 00:48:06.270
Wendy Nilsen, NSF/CISE: It may have there may be overlap in some teams, but those are completely separate initiatives.

257
00:48:08.400 --> 00:48:13.440
Goli Yamini (NSF/CISE): Would infrastructural projects that enable multiple research studies be interesting?

258
00:48:18.240 --> 00:48:34.020
Wendy Nilsen, NSF/CISE: I think infrastructure studies that are that it can make other projects possible if the infrastructure is not advancing fundamental science or engineering and is making in that, then the science of it isn't other projects, there are better initiatives for that.

259
00:48:35.070 --> 00:48:46.080
Wendy Nilsen, NSF/CISE: If you have a question about that. I suggest you reach out to the team and we can talk to you about where that might be a better fit if their advantage if they're taking their advances required for the infrastructure.

260
00:48:46.860 --> 00:48:58.620
Wendy Nilsen, NSF/CISE: then it would be a fit. But again just, you know, pretty description of your project, the intellectual merit the broader impacts, it makes it much easier for us to be able to give you better information.

261
00:49:01.320 --> 00:49:07.050
Goli Yamini (NSF/CISE): Is the proposed team expected to have previous collaboration, such as co-authorships.

262
00:49:11.070 --> 00:49:21.180
Sylvia Spengler: It strengthens a proposal if there is evidence of previous cooperation and collaboration. We do ask for a collaboration plan.

263
00:49:21.630 --> 00:49:35.010
Sylvia Spengler: But this is much easier, enhances the confidence that the reviewers have, if, for example, you've either publish or presets live together or perhaps talk together. That's another way to get to know somebody.

264
00:49:36.480 --> 00:49:46.860
Goli Yamini (NSF/CISE): Another question is about human subjects on its with the evaluation of the developed technology on human subjects, not clinical trials, be perceived positively?

265
00:49:50.460 --> 00:49:54.450
Wendy Nilsen, NSF/CISE: You repeat that question. I'm looking at Steve's face in the two of us are looking

266
00:49:54.450 --> 00:49:55.380
Wendy Nilsen, NSF/CISE: At going

267
00:49:56.910 --> 00:50:08.370
Goli Yamini (NSF/CISE): With the evaluation of the developed technology on human subjects not clinical trials. Oh, be perceived possible and positively.

268
00:50:10.650 --> 00:50:14.610
Wendy Nilsen, NSF/CISE: So I was thinking human subjects in the other way. So,

269
00:50:15.870 --> 00:50:29.070
Wendy Nilsen, NSF/CISE: so yes, people can do it. When you're doing technology that involves humans, you might be doing a decision support system, you might, there's all sorts of reasons why humans would be in there, usability is an important factor.

270
00:50:30.030 --> 00:50:34.650
Wendy Nilsen, NSF/CISE: You know, if you're thinking about perception. You really want to know what people are perceiving.

271
00:50:35.790 --> 00:50:44.310
Wendy Nilsen, NSF/CISE: You cannot do a randomized clinical trial, but you can do evaluations with people. You might do too small randomization of components.

272
00:50:45.180 --> 00:50:55.830
Wendy Nilsen, NSF/CISE: That's acceptable. There's all sorts of ways you can think about it, but you do want to evaluate your impact with humans, whether it's if you're trying to build something people will use,

273
00:50:56.490 --> 00:51:09.000
Wendy Nilsen, NSF/CISE: if you're trying to do something to improve function, whether it's somebody reading an X ray, whether it's it's human behavior, what you would want to see ifpeople use this, and if so, how does it work?

274
00:51:11.460 --> 00:51:24.990
Goli Yamini (NSF/CISE): Um, there's a couple of questions about the one pager. The one page summary that you mentioned about repeating the email that it needs to go to a site, which I think is on the page. And then if there's a timeframe for that. Like if you should I mean a certain time to be

275
00:51:26.100 --> 00:51:29.190
Goli Yamini (NSF/CISE): seen or it can they submit one anytime

276
00:51:31.170 --> 00:51:47.430
Wendy Nilsen, NSF/CISE: So you can submit one anytime. But if you're if you're waiting till two weeks beforehand, it's a mistake. So probably February 1 is not a great day to first send you know one of my colleagues or I that that one pager. Get them out now.

277
00:51:48.660 --> 00:51:53.460
Wendy Nilsen, NSF/CISE: Will be helpful. The more time you have to think about this. I think the better proposals that you will do

278
00:51:56.610 --> 00:51:57.840
Goli Yamini (NSF/CISE): Um, there were a few

279
00:51:59.190 --> 00:52:10.890
Goli Yamini (NSF/CISE): Questions about different model systems like, does research like maestro SAFA law. There was also a question before about non-human primaate. I mean, other primates.

280
00:52:12.360 --> 00:52:14.130
Goli Yamini (NSF/CISE): Are those of interest in this program?

281
00:52:19.350 --> 00:52:25.620
Wendy Nilsen, NSF/CISE: So I think those are ones, there's a there's a range of responses here we have animals and many of the projects.

282
00:52:26.640 --> 00:52:31.410
Wendy Nilsen, NSF/CISE: I think that's one where it's nice to send a project summary and let us think about it.

283
00:52:32.760 --> 00:52:42.930
Wendy Nilsen, NSF/CISE: Instead of, again, we try to answer it on the fly, but but often it a little thought really helps. And this is a very smart group. So a little thought goes a long way.

284
00:52:48.390 --> 00:52:54.960
Goli Yamini (NSF/CISE): The question of salary caps of with NIH salary caps have been presented a few times.

285
00:52:56.460 --> 00:53:03.960
Goli Yamini (NSF/CISE): Of how they should be included in the proposal and somebody is asking me if we could verbally address that.

286
00:53:06.180 --> 00:53:14.970
Dana Hughes NIH-OBSSR: So Wendy can build off what I'm about to say. But these applications coming to NSF, so you need to address the NSF proposal side of things.

287
00:53:15.330 --> 00:53:34.080
Dana Hughes NIH-OBSSR: And we're not going to get into what the salary caps are and changes in budget because it's $300 total cost and any sorts of modifications to fit within that will, if your proposal is selected to come over it, to NIH will work with the program officer to fit that in within the NIH forms.

288
00:53:36.630 --> 00:53:40.590
Wendy Nilsen, NSF/CISE: I think data is fine is good. These are NSF proposals so

289
00:53:41.640 --> 00:53:57.930
Wendy Nilsen, NSF/CISE: you should follow again, read our proposal guide and follow the criteria there. If NIH picks up your proposal, the program officer, they will definitely work with you. That's a good problem, not a bad way.

290
00:53:59.400 --> 00:54:07.680
Goli Yamini (NSF/CISE): Thinking of budgets. I mean, since we are encoded time someone's asking if they should add an in person PI meeting attendance to the budget or no.

291
00:54:09.810 --> 00:54:21.210
Wendy Nilsen, NSF/CISE: I think we have no idea where we will be this time next year. So, if you do not use that money for the PI meeting, meaning you I'm sure you could find another useful for it so

292
00:54:21.870 --> 00:54:32.670
Wendy Nilsen, NSF/CISE: I would suggest putting it in and then if you don't need it like this year we're doing  our PI meeting virtually, so it won't hurt to have it in there.

293
00:54:34.500 --> 00:54:41.010
Goli Yamini (NSF/CISE): Um, how about research into the ethical issues arising from the technology or those of interest.

294
00:54:43.260 --> 00:54:48.180
Dana Hughes NIH-OBSSR: So they could be of interest. I think, again, as Wendy has said before,

295
00:54:49.200 --> 00:54:54.090
Dana Hughes NIH-OBSSR: without knowing more about the specific proposal and more broadly, what

296
00:54:55.140 --> 00:55:03.060
Dana Hughes NIH-OBSSR: you're aiming to achieve within that area, it's really hard to really say yes or no. This would fit or yes or no.

297
00:55:04.140 --> 00:55:10.230
Dana Hughes NIH-OBSSR: We would be interested. So again, submit your, your one page proposals to the email address that, Wendy.

298
00:55:11.700 --> 00:55:14.490
Dana Hughes NIH-OBSSR: Referenced before and that will be on the slides. They'll be

299
00:55:15.870 --> 00:55:17.640
Dana Hughes NIH-OBSSR: disseminated after this webinar.

300
00:55:23.730 --> 00:55:24.180
Goli Yamini (NSF/CISE): Um,

301
00:55:25.200 --> 00:55:28.350
Goli Yamini (NSF/CISE): There are some questions on the success rate and

302
00:55:32.220 --> 00:55:35.940
Sylvia Spengler: Wait, I had a, I had, this is Sylvia

303
00:55:36.990 --> 00:55:38.250
Sylvia Spengler: I had said.

304
00:55:40.020 --> 00:55:40.830
Sylvia Spengler: There were you

305
00:55:41.850 --> 00:56:09.990
Sylvia Spengler: a requirement for to NSF sciences. How about in NIH institute as one of the two, I, I'm assuming, like your what the question is, is that of the appropriateness of specific domain for a given NIH institute. That's certainly appropriate. That is we're not, we, we don't have a list of only acceptable 

306
00:56:12.120 --> 00:56:13.350
Sylvia Spengler: health issues.

307
00:56:14.940 --> 00:56:19.890
Sylvia Spengler: But the requirement is two NSF sciences

308
00:56:22.350 --> 00:56:23.490
Sylvia Spengler: for advances.

309
00:56:26.460 --> 00:56:31.500
Wendy Nilsen, NSF/CISE: Two science or engineering discipline should be in here, so

310
00:56:33.210 --> 00:56:45.510
Wendy Nilsen, NSF/CISE: There is a question here about Amir and 935, I suggest you whoever that was you should email Dana Wolf Hughes to ask her that because that's beyond our capability in this webinar.

311
00:56:56.370 --> 00:56:58.950
Goli Yamini (NSF/CISE): Or four fundamental science areas.

312
00:57:00.960 --> 00:57:04.260
Goli Yamini (NSF/CISE): Sorry, let me just pull the question.

313
00:57:09.030 --> 00:57:09.600
Wendy Nilsen, NSF/CISE: I'm sorry.

314
00:57:10.170 --> 00:57:26.220
Goli Yamini (NSF/CISE): Oh, sorry. No, I wanted to just hold as the questions moved around four fundamental science areas were listed in the sub topics. Is it okay to to address two sub topics in the same science area, machine learning and databases, for example.

315
00:57:27.210 --> 00:57:40.500
Wendy Nilsen, NSF/CISE: So I think those are those are broad areas of science and engineering across NSF, they are many, many different disciplines, many different areas of interest. So yes, that's entirely possible.

316
00:57:45.300 --> 00:57:47.970
Goli Yamini (NSF/CISE): Should we continue to take questions?

317
00:57:49.590 --> 00:57:51.510
Goli Yamini (NSF/CISE): We are approaching around 4pm but

318
00:58:00.090 --> 00:58:09.720
Wendy Nilsen, NSF/CISE: So I will say that all the questions that we're answering here will be, we will take the themes, out of these and put them in our frequently asked questions.

319
00:58:11.160 --> 00:58:14.850
Wendy Nilsen, NSF/CISE: So that these questions will be available at a later date.

320
00:58:15.960 --> 00:58:24.840
Wendy Nilsen, NSF/CISE: The other thing I want to say there's a question here. It says, considering top down order which was the National Science Board intellectual merit and broader impact criteria.

321
00:58:25.320 --> 00:58:36.480
Wendy Nilsen, NSF/CISE: And it says is that also important? Yes, that is important. Those are all variables that are important to review, but the ones that we start with are the are the intellectual merit.

322
00:58:37.050 --> 00:58:48.630
Wendy Nilsen, NSF/CISE: What is what has happened, what is this doing for science and engineering and the broader impacts? What does this do for society? Those are the two criteria that we start off every conversation about a proposal on.

323
00:58:54.240 --> 00:59:04.950
Dana Hughes NIH-OBSSR: I do know Goli, you had before what the percent of funded applications are historically and historically for both combined NIH and NSF, it's about

324
00:59:06.210 --> 00:59:07.620
Dana Hughes NIH-OBSSR: between 11 and 12%

325
00:59:11.220 --> 00:59:25.440
Goli Yamini (NSF/CISE): Yeah, that came up a few times. And there are a couple of questions also about related ones. For example, if the proposed study involves human participants in the lab. Do you expect an additional COVID-19 progression plan?

326
00:59:27.540 --> 00:59:30.300
Goli Yamini (NSF/CISE): Previously. Somebody else asked also asked that about,

327
00:59:32.790 --> 00:59:47.610
Goli Yamini (NSF/CISE): it might be difficult to collect support letters to recruit subjects such as older adults from retirement community centers assisted living due to COVID-19. Is it okay to involve or mentioned those stakeholders in the proposals without direct support letters.

328
00:59:49.410 --> 00:59:51.120
Wendy Nilsen, NSF/CISE: One of my colleagues on answer this one.

329
00:59:56.520 --> 00:59:58.920
Wendy Nilsen, NSF/CISE: All right, Scott are you going to answer this

330
00:59:59.670 --> 01:00:04.080
Scott Acton NSF signal and image processing/ML: I could from the NSF side, the official policy is that there's

331
01:00:05.340 --> 01:00:15.810
Scott Acton NSF signal and image processing/ML: no requirement to address COVId-19 in a proposal, it's not going to be a positive or a negative. If you do. I'm not sure what the NIH requires

332
01:00:19.980 --> 01:00:21.030
Scott Acton NSF signal and image processing/ML: I dont know if Dana wants to comment.

333
01:00:23.130 --> 01:00:35.250
Dana Hughes NIH-OBSSR: So I am not going to comment right now, but we can add that in the FAQ because my office doesn't oversee the actual awarding and management of grants. And so I don't want to miss misspeak.

334
01:00:35.940 --> 01:00:38.100
Wendy Nilsen, NSF/CISE: All right. And I think Steve had a comment to make.

335
01:00:38.610 --> 01:00:45.990
Steve Breckler (NSF/SBE/BCS): Yeah,  in regard to the part of the question that had to do with letters of support from organizations

336
01:00:47.340 --> 01:00:50.460
Steve Breckler (NSF/SBE/BCS): that may be involved in data collection from human subjects.

337
01:00:51.510 --> 01:00:59.100
Steve Breckler (NSF/SBE/BCS): A couple of things there. One is generally you should know that the more evidence you have that partners are willing to collaborate and to help out the better.

338
01:01:00.150 --> 01:01:07.020
Steve Breckler (NSF/SBE/BCS): But when you do include those letters, the PAPPG, and what Wendy described as your best friend, grant proposal guide.

339
01:01:07.650 --> 01:01:12.390
Steve Breckler (NSF/SBE/BCS): Has very specific instruct guidance on the form of those letters of collaboration.

340
01:01:13.110 --> 01:01:27.060
Steve Breckler (NSF/SBE/BCS): And if you read that it may reassure you that they may not be so difficult to obtain. After all, they can only state that your partner is going to collaborate as described in the proposal. That's it. There are no details so

341
01:01:28.200 --> 01:01:38.700
Steve Breckler (NSF/SBE/BCS): endorsement or recommendation for the project. It simply affirms that that partner is going to do what you described in the proposal. So whether you have that letter of collaboration or not

342
01:01:39.060 --> 01:01:45.300
Steve Breckler (NSF/SBE/BCS): you should describe in the appropriate places the proposal, the nature of that collaboration and what kinds of resources are being pledged.

343
01:01:50.280 --> 01:02:07.260
Goli Yamini (NSF/CISE): A few other questions that we haven't answered yet or are there any. Is there a cap on the number of consultants that can be included and will this solicitation be repeated? It is handled as an NIH A0 and then A1 with an introduction page.

344
01:02:08.340 --> 01:02:27.030
Wendy Nilsen, NSF/CISE: So there are no requirements on whether you have consultants or not and how many you have. Although these are not huge proposals, so you probably don't need a whole lot of consultants, you really would like a few team members who are really going to get to the heart of this.

345
01:02:28.740 --> 01:02:30.900
Wendy Nilsen, NSF/CISE: Goli, now I forgot the other part of the question.

346
01:02:31.950 --> 01:02:41.790
Goli Yamini (NSF/CISE): Sorry, I kind of put two questions in one. It was about the future of the solicitation. Will it be repeated and is it handled as an A1. Okay. Yeah.

347
01:02:41.820 --> 01:02:53.010
Wendy Nilsen, NSF/CISE: So we cannot tell you whether it will be repeated or not, at this point, we've just released this solicitation. Be happy that you got three dates for for this solicitation.

348
01:02:54.120 --> 01:03:04.590
Wendy Nilsen, NSF/CISE: That's as good as it gets. So we so that's good. And this is not treated as like NIH with we do not do resubmissions.

349
01:03:04.920 --> 01:03:16.950
Wendy Nilsen, NSF/CISE: People can revise the proposal and resend it in, but they do not indicate it's been submitted before it does not start with a letter and that will actually hurt you in a panel. If you start with the letter, it says

350
01:03:17.430 --> 01:03:22.620
Wendy Nilsen, NSF/CISE: this was reviewed last time and we made these changes, they should be integrated into the proposal.

351
01:03:23.160 --> 01:03:32.040
Wendy Nilsen, NSF/CISE: If you do not get funded, you should make an appointment to talk to the program officer who managed your proposal and have a discussion about 

352
01:03:32.310 --> 01:03:48.270
Wendy Nilsen, NSF/CISE: what parts of the proposal worked and what parts of the proposal, didn't. We are all willing to talk to you about these things. So please reach out and talk to us before you if you're going to try to revise a proposal and submit. Again, really think about that first.

353
01:03:55.170 --> 01:04:00.960
Goli Yamini (NSF/CISE): Question about scientific areas can an area fundamental advances be physiology.

354
01:04:06.060 --> 01:04:06.780
Wendy Nilsen, NSF/CISE: So,

355
01:04:07.830 --> 01:04:18.630
Wendy Nilsen, NSF/CISE: in NSF sciences physiology is primarily in animals. And that's not consistent with the goal of this initiative. So if you're going to do physiological

356
01:04:19.140 --> 01:04:38.070
Wendy Nilsen, NSF/CISE: research on animals, I suggest you apply to the biology department or you apply to one of the NSF many I'm sure my colleagues in math have many proposals where the application is in complex biological phenomena. BIO has that, CISE has that so

357
01:04:39.330 --> 01:04:43.800
Wendy Nilsen, NSF/CISE: there are many places a proposal like that could go, but this is probably not the initiative for it.

358
01:04:45.120 --> 01:04:47.880
Goli Yamini (NSF/CISE): And is there a limit on the number of PIs and Co-PIs.

359
01:04:49.290 --> 01:04:49.620
Goli Yamini (NSF/CISE): No.

360
01:04:51.690 --> 01:05:07.890
Wendy Nilsen, NSF/CISE: Although I will say that the panel struggle when there is one PI on a proposal. It's the master of everything and the master of everything is hard, given how many how complex these areas are. It's hard to have all the knowledge that you need.

361
01:05:19.110 --> 01:05:29.700
Goli Yamini (NSF/CISE): Following one this comment for submission. Do we need to include the justification for submission submission in the 15 page project description or as a separate document?

362
01:05:32.880 --> 01:05:34.410
Wendy Nilsen, NSF/CISE: What was the start of the question.

363
01:05:35.220 --> 01:05:42.780
Goli Yamini (NSF/CISE): Following your comment about every submission do the PIs need to include a justification. 

364
01:05:43.380 --> 01:05:52.560
Wendy Nilsen, NSF/CISE: I said before that will hurt you in a panel. If you do that, so if you are going to make a few simple if you've applied if you were on a panel with my colleague Wei Ding

365
01:05:52.980 --> 01:05:58.110
Wendy Nilsen, NSF/CISE: who is the program manager of your proposal and it didn't get funded. You're going to want to email

366
01:05:59.010 --> 01:06:05.010
Wendy Nilsen, NSF/CISE: and say would like to set up an appointment to talk to you about my proposal, you're going to integrate that feedback and the reviews.

367
01:06:05.280 --> 01:06:19.290
Wendy Nilsen, NSF/CISE: You're going to integrate that into your proposal, you are not going to mention this as a rsubmission that will hurt you. You will. You don't want to waste your space explaining why you're resubmitting what you really want to do is improve the proposal.

368
01:06:22.320 --> 01:06:28.290
Goli Yamini (NSF/CISE): Is a list of the science and engineering areas on the SCH home page, where can people find that

369
01:06:31.560 --> 01:06:45.510
Wendy Nilsen, NSF/CISE: There is no specific list if you, if you look at the NSF website and you look at it, this is this is Computer and Information Science and Computer Engineering, Engineering, Math and statistics,

370
01:06:46.380 --> 01:06:55.560
Wendy Nilsen, NSF/CISE: and Social and Behavioral Sciences. So if you look at each one of those pages, you will see the broad range of science and each one of those divisions represents

371
01:06:56.520 --> 01:07:01.500
Wendy Nilsen, NSF/CISE: If you have a question, please email our correspondence email and we can also clarify there.

372
01:07:02.160 --> 01:07:11.640
Wendy Nilsen, NSF/CISE: But if you're if you're disciplines are primarily in the better advancing in the NIH area, then you're going to want to have, you're going to want to go to NIH with that award.

373
01:07:12.510 --> 01:07:20.160
Wendy Nilsen, NSF/CISE: If you're advancing something in math and statistics and it's in a health or biomedical area that's when you come here.

374
01:07:23.310 --> 01:07:38.250
Goli Yamini (NSF/CISE): Speaking in a fundamental areas of fundamental science,  can these areas be at like two fundamental science areas be at addressed by one PI and not by the rest of the team. For example, if there are medical doctors.

375
01:07:41.220 --> 01:07:53.430
Dana Hughes NIH-OBSSR: I guess it would depend on what it is that you're proposing to do and what the actual experience is of the one PI with the fundamental science area.

376
01:07:53.940 --> 01:08:05.310
Dana Hughes NIH-OBSSR: As well as the other two PIs. I mean, they also like to say don't have to be medical doctors. Like it could be a PhD and a health-related science.

377
01:08:06.420 --> 01:08:14.700
Dana Hughes NIH-OBSSR: Not always do they have to be clinicians. So again it are you putting together the right team to address the research question that you're proposing.

378
01:08:18.060 --> 01:08:31.500
Wendy Nilsen, NSF/CISE: And so I think we're really at the end of our questions here. And again, people can reach out to us. The one question I want to say is somebody had asked how many of the 10 to 16 funded proposals with NIH focus first versus NSF

379
01:08:32.040 --> 01:08:45.060
Wendy Nilsen, NSF/CISE: And the answer to that is, they will every single proposal will have an NSF fundamental science or engineering advance but the health problem or biomedical issues that they're facing will be different depending on who

380
01:08:45.570 --> 01:08:54.720
Wendy Nilsen, NSF/CISE: which institute or center picks it up, and even if NSF, if you look at what we funded, you will see such a wide range of topics. So

381
01:08:55.500 --> 01:09:13.770
Wendy Nilsen, NSF/CISE: so like we keep saying, the criteria for both agencies is the same for this initiative, it is a fundamental science and engineering driving the questions are there is a there is an application area that has that writing questions and it can be filled find this new science or engineering

382
01:09:16.830 --> 01:09:21.360
Wendy Nilsen, NSF/CISE: So with that, I think, it appears that we were

383
01:09:22.500 --> 01:09:41.160
Wendy Nilsen, NSF/CISE: running to the end of our time. And we've we've answered most of our questions. I think we will, we're going to post an updated FAQ. Please will post the slides to please feel free to email us at SCH-. Oh, can we put that in the Q&A or somewhere. Oh all attendees.

384
01:09:42.180 --> 01:09:46.110
Wendy Nilsen, NSF/CISE: It's sch-correspondence@nsf.gov

385
01:09:48.780 --> 01:09:51.270
I can't spell this sorry

386
01:09:53.280 --> 01:09:58.200
Wendy Nilsen, NSF/CISE: All right there we are, please feel free to email us with any additional questions.

387
01:09:58.800 --> 01:10:05.100
Wendy Nilsen, NSF/CISE: We look forward to talking to you, whether you're whether you want and if you're interested in being a panelist and our review.

388
01:10:05.550 --> 01:10:16.680
Wendy Nilsen, NSF/CISE: Please send who, if you're if you're an engineering send my colleague John in ECCS both your CV and a note saying you're interested.

389
01:10:17.310 --> 01:10:33.180
Wendy Nilsen, NSF/CISE: We will, or you can send it to the correspondence. We love to have new reviewers and we really look forward to seeing your, your exciting science these exciting problems that you're answering and we look forward to you all transforming the world so

390
01:10:34.830 --> 01:10:39.870
Wendy Nilsen, NSF/CISE: And I guess. Happy Holidays everyone because it's a holiday somewhere at this point. So,

391
01:10:45.900 --> 01:10:47.370
John X.J. Zhang (NSF/ECCS): Thank you. Thank you.

392
01:10:47.610 --> 01:10:48.420
John X.J. Zhang (NSF/ECCS): Happy holidays.