Award Abstract # 1734887
NCS-FO: A Computational Theory to Model the Neurobiological Basis of a Visuo-Cognitive Neuroprosthetic
NSF Org: |
SMA
SBE Office of Multidisciplinary Activities
|
Recipient: |
THE RESEARCH FOUNDATION FOR THE STATE UNIVERSITY OF NEW YORK
|
Initial Amendment Date:
|
August 7, 2017 |
Latest Amendment Date:
|
August 7, 2017 |
Award Number: |
1734887 |
Award Instrument: |
Standard Grant |
Program Manager: |
Dwight Kravitz
dkravitz@nsf.gov
(703)292-4502
SMA
SBE Office of Multidisciplinary Activities
SBE
Directorate for Social, Behavioral and Economic Sciences
|
Start Date: |
August 1, 2017 |
End Date: |
July 31, 2022 (Estimated) |
Total Intended Award
Amount: |
$949,928.00 |
Total Awarded Amount to
Date: |
$949,928.00 |
Funds Obligated to Date:
|
FY 2017 = $949,928.00
|
History of Investigator:
|
-
Stephen
Macknik
(Principal Investigator)
macknik@neuralcorrelate.com
-
Susana
Martinez-Conde
(Co-Principal Investigator)
|
Recipient Sponsored Research
Office: |
SUNY Health Science Center at Brooklyn
450 CLARKSON AVE
BROOKLYN
NY
US
11203-2012
(718)270-2680
|
Sponsor Congressional
District: |
09
|
Primary Place of
Performance: |
SUNY Health Science Center at Brooklyn
450 Clarkson Avenue
Brooklyn
NY
US
11203-2012
|
Primary Place of
Performance Congressional District: |
09
|
Unique Entity Identifier
(UEI): |
NJ14V2NZYM68
|
Parent UEI: |
|
NSF Program(s): |
IntgStrat Undst Neurl&Cogn Sys
|
Primary Program Source:
|
01001718DB NSF RESEARCH & RELATED ACTIVIT
|
Program Reference
Code(s): |
8089,
8091,
8551
|
Program Element Code(s):
|
862400
|
Award Agency Code: |
4900
|
Fund Agency Code: |
4900
|
Assistance Listing
Number(s): |
47.075
|
ABSTRACT

Evoking high quality visual perception in a blind person, via direct microstimulation of the brain, poses great difficulties. One major obstacle has been that electrical stimulation of the brain typically affects neuronal populations that are mutually suppressive, which subverts proper neuronal signaling. The visual system has two antagonistic information channels that encode either the perception of lightness, in ON cells, or darkness, in OFF cells. Inappropriate coactivation of these two channels results in nullification of contrast, and deprived visual perception. It follows that high-quality prosthetic stimulation systems must avoid unwanted coactivation of mutually suppressive neurons, just as the natural visual system does. This is a challenge because the antagonistic neurons typically lie within microns of each other in the brain. The project aims at transformative advances in viral transfection and imaging methodology, computational theory, and cortical prosthetic neuroengineering design for the purpose of restoring vision by genetically modifying neurons in the brain and then stimulating them with light, a method called optogenetics. The expected results and methodology will form the scientific basis to build a breakthrough neuroprosthetic, with transformative potential to further brain research in sensory, motor, and cognitive parts of the cortex and to advance human medicine. To promote the development and availability of derived products to the public, the team will disseminate the discoveries to general audiences through public lectures and publications in popular science magazines. The investigators will supervise trainees from underrepresented groups, including postdoctoral fellows, graduate students, undergraduates, and high school students. The investigators are faculty mentors for The Children's Aid Society (CAS) Workforce Development Department Summer Youth Employment Program (SYEP), which provides summer research opportunities to disadvantaged and minority youth in NYC to inspire them to pursue STEM careers.
Recent research has shown that, for any given retinal position, the ON and OFF cell inputs to the brain's visual cortex are purely excitatory, concentrate in a specific layer, and are laid out in a pattern that can be targeted with light from outside the brain. First, the team will modify these neurons genetically, to turn them into a novel type of photoreceptor, embedded within the brain. The team will then target light stimulation to the identified ON and OFF cells, determining the precise balance of activation to either channel to generate high-quality prosthetic vision based on a video camera's signal. This technology can then be used to bypass the eye to stimulate the brain from the camera. The project aims to develop the computational model to drive an optogenetic brain stimulation system that will optimally activate neural responses in the primary visual cortex. By comparing the neuronal responses of sighted nonhuman primates viewing natural visual stimuli to prosthetic responses in the same neurons, the work will optimize stimulation patterns that evoke naturalistic visual perception. The balanced targeting of appropriate ON and OFF inputs at each position in visual space is expected to achieve maximal contrast perception at the highest attainable acuity, with full stereoscopic binocular vision. The team's computational model of spatiotemporal visual inputs into the cortex will also account for the effects of eye movements on early visual responses, a novel approach to visual prosthetics tested here for the first time.
This project is funded by Integrative Strategies for Understanding Neural and Cognitive Systems (NSF-NCS), a multidisciplinary program jointly supported by the Directorates for Computer and Information Science and Engineering (CISE), Education and Human Resources (EHR), Engineering (ENG), and Social, Behavioral, and Economic Sciences (SBE).
PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:
When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external
site maintained by the publisher. Some full text articles may not yet be available without a
charge during the embargo (administrative interval).
Some links on this page may take you to non-federal websites. Their policies may differ from
this site.
(Showing: 1 - 10 of 34)
(Showing: 1 - 34 of 34)
Alexander, Robert and Mintz, Ronald and Custodio, Paul and Macknik, Stephen and Gindina, Sofya and Martinez-Conde, Susana
"Gaze behavior during the averted detection of a simulated faint star"
Journal of Vision
, v.17
, 2017
https://doi.org/10.1167/17.10.1186
Citation
Details
Alexander, Robert and Venkatakrishnan, Ashwin and Chanovas, Jordi and Ferguson, Sophie and Macknik, Stephen and Martinez-Conde, Susana
"Why did Rubens add a parrot to Titians Fall of Man? An eye tracking investigation reveals attentional focus while viewing Italian Renaissance paintings"
Journal of Vision
, v.22
, 2022
https://doi.org/10.1167/jov.22.14.3162
Citation
Details
Alexander, Robert and Waite, Stephen and Bruno, Michael A. and Krupinski, Elizabeth A. and Berlin, Leonard and Macknik, Stephen and Martinez-Conde, Susana
"Mandating Limits on Workload, Duty, and Speed in Radiology"
Radiology
, v.304
, 2022
https://doi.org/10.1148/radiol.212631
Citation
Details
Alexander, Robert G and Macknik, Stephen L and Martinez-Conde, Susana
"Microsaccades in applied environments: Real-world applications of fixational eye movement measurements"
Journal of Eye Movement Research
, v.12
, 2020
https://doi.org/10.16910/jemr.12.6.15
Citation
Details
Alexander, Robert G. and Mintz, Ronald J. and Custodio, Paul J. and Macknik, Stephen L. and Vaziri, Alipasha and Venkatakrishnan, Ashwin and Gindina, Sofya and MartinezConde, Susana
"Gaze mechanisms enabling the detection of faint stars in the night sky"
European Journal of Neuroscience
, v.54
, 2021
https://doi.org/10.1111/ejn.15335
Citation
Details
Alexander, Robert G. and Venkatakrishnan, Ashwin and Chanovas, Jordi and Ferguson, Sophie and Macknik, Stephen L. and Martinez-Conde, Susana
"Why did Rubens add a parrot to Titian's <i>The Fall of Man</i> ? A pictorial manipulation of joint attention"
Journal of Vision
, v.24
, 2024
https://doi.org/10.1167/jov.24.4.1
Citation
Details
Alexander, Robert G. and Venkatakrishnan, Ashwin and Chanovas, Jordi and Macknik, Stephen L. and Martinez-Conde, Susana
"Microsaccade dynamics mediate perceptual alternation in Monets Impression, Sunrise."
Society for Neuroscience Conference 2018
, 2018
Citation
Details
Alexander, Robert G. and Venkatakrishnan, Ashwin and Chanovas, Jordi and Macknik, Stephen L. and Martinez-Conde, Susana
"Microsaccades mediate perceptual alternations in Monets Impression, sunrise"
Scientific Reports
, v.11
, 2021
https://doi.org/10.1038/s41598-021-82222-3
Citation
Details
Alexander, Robert G. and Waite, Stephen and Macknik, Stephen L. and Martinez-Conde, Susana
"What do radiologists look for? Advances and limitations of perceptual learning in radiologic search"
Journal of Vision
, v.20
, 2020
https://doi.org/10.1167/jov.20.10.17
Citation
Details
Alexander, Robert G. and Yazdanie, Fahd and Waite, Stephen and Chaudhry, Zeshan A. and Kolla, Srinivas and Macknik, Stephen L. and Martinez-Conde, Susana
"Visual Illusions in Radiology: Untrue Perceptions in Medical Images and Their Implications for Diagnostic Accuracy"
Frontiers in Neuroscience
, v.15
, 2021
https://doi.org/10.3389/fnins.2021.629469
Citation
Details
Barnhart, Anthony S. and Costela, Francisco M. and Martinez-Conde, Susana and Macknik, Stephen L. and Goldinger, Stephen D.
"Microsaccades reflect the dynamics of misdirected attention in magic"
Journal of Eye Movement Research
, v.12
, 2020
https://doi.org/10.16910/jemr.12.6.7
Citation
Details
Brunet, N and Alexander, R. G and Martinez-Conde, Susana and Macknik, Stephen
"An oculomotor signature as a fraud-resistant tool for biometric verification"
Society for Neuroscience Annual Meeting 2018
, 2018
Citation
Details
Caballero, Olivya and Chanovas, Jordi and Ledo, Manuel and Nandy, Anirvan and Yazdah-Shahmorad, Azadeh and Callaway, Edward and Seidemann, E. and Reynolds, John and Avery, Michael and Li, Peichao and Tang, Shiming and Kolla, S and Martinez-Conde, Susana a
"A Novel pressure regulating brain imaging implant for ultra large field of view microscopic imaging in primates"
Society for Neuroscience 2021
, 2021
Citation
Details
Caballero, Olivya and Ledo, Manuel and Nandy, Anirvan and Yazdah-Shahmorad, Azadeh and Callaway, Edward and Seidemann, E. and Reynolds, John and Avery, Michael and Li, Peichao and Tang, Shiming and Kolla, S. and Martinez-Conde, Susana and Hayes, A and Wai
"A Novel Pressure Regulating Brain Imaging Implant For Ultra-Large Field-of-View Microscopic Imaging in NHPs"
Brain Initiative Investigators Meeting 2019
, 2019
Citation
Details
Caballero, Olivya and Ledo, Manuel and Nandy, Anirvan and Yazdah-Shahmorad, Azadeh and Callaway, Edward and Seidemann, E. and Reynolds, John and Avery, Michael and Li, Peichao and Tang, Shiming and Kolla, S. and Martinez-Conde, Susana and Hayes, A and Wai
"A Novel Pressure Regulating Brain Imaging Implant For Ultra-Large Field-of-View Microscopic Imaging in NHPs"
NYC Life Science Innovation Showcase
, 2018
Citation
Details
Caballero, Olivya and Ledo, Manuel and Nandy, Anirvan and Yazdah-Shahmorad, Azadeh and Callaway, Edward and Seidemann, E. and Reynolds, John and Avery, Michael and Li, Peichao and Tang, Shiming and Kolla, S. and Martinez-Conde, Susana and Hayes, A and Wai
"A Novel Pressure Regulating Brain Imaging Implant For Ultra-Large Field-of-View Microscopic Imaging in NHPs"
Society for Neuroscience 2020
, 2020
Citation
Details
Dunn, Matt J. and Alexander, Robert G. and Amiebenomo, Onyekachukwu M. and Arblaster, Gemma and Atan, Denize and Erichsen, Jonathan T. and Ettinger, Ulrich and Giardini, Mario E. and Gilchrist, Iain D. and Hamilton, Ruth and Hessels, Roy S. and Hodgins, S
"Minimal reporting guideline for research involving eye tracking (2023 edition)"
Behavior Research Methods
, 2023
https://doi.org/10.3758/s13428-023-02187-1
Citation
Details
Holmqvist, Kenneth and Örbom, Saga Lee and Hooge, Ignace T. and Niehorster, Diederick C. and Alexander, Robert G. and Andersson, Richard and Benjamins, Jeroen S. and Blignaut, Pieter and Brouwer, Anne-Marie and Chuang, Lewis L. and Dalrymple, Kirsten A. a
"Eye tracking: empirical foundations for a minimal reporting guideline"
Behavior research methods
, v.55
, 2022
https://doi.org/10.3758/s13428-021-01762-8
Citation
Details
Ju, Niansheng and Jiang, Rundong and Macknik, Stephen L. and Martinez-Conde, Susana and Tang, Shiming and Kohn, Adam
"Long-term all-optical interrogation of cortical neurons in awake-behaving nonhuman primates"
PLOS Biology
, v.16
, 2018
https://doi.org/10.1371/journal.pbio.2005839
Citation
Details
Ju, Niansheng and Li, Yang and Liu, Fang and Jiang, Hongfei and Macknik, Stephen and Martinez-Conde, Susana and Tang, Shiming
"Spatiotemporal functional organization of excitatory synaptic inputs onto macaque V1 neurons"
BioRxiv
, 2019
https://doi.org/10.1101/558163
Citation
Details
Ju, Niansheng and Li, Yang and Liu, Fang and Jiang, Hongfei and Macknik, Stephen L. and Martinez-Conde, Susana and Tang, Shiming
"Spatiotemporal functional organization of excitatory synaptic inputs onto macaque V1 neurons"
Nature Communications
, v.11
, 2020
https://doi.org/10.1038/s41467-020-14501-y
Citation
Details
Leal-Campanario, Rocio and Martinez-Conde, Susana and Macknik, Stephen L.
"In Vivo Fiber-Coupled Pre-Clinical Confocal Laser-scanning Endomicroscopy (pCLE) of Hippocampal Capillaries in Awake Mice"
JoVE
, 2020
https://doi.org/10.3791/57220
Citation
Details
Li, Ming and Ju, Niansheng and Jiang, Rundong and Liu, Fang and Jiang, Hongfei and Macknik, Stephen and Martinez-Conde, Susana and Tang, Shiming
"Perceptual hue, lightness, and chroma are represented in a multidimensional functional anatomical map in macaque V1"
Progress in Neurobiology
, v.212
, 2022
https://doi.org/10.1016/j.pneurobio.2022.102251
Citation
Details
Macknik, Stephen and Caballero, Olivya and Ledo, Manuel and Yazdah-Shahmorad, Azadeh and Cheng, Yu-Ting and Anne Bizimana, Lauri and Nishimura, Nozomi and Schaffer, Chris and Reynolds, John and Tang, Shiming and Chen, Yuzhi and Vaziri, Alipasha and Nobaue
"Advanced Inverse Mapping of Cortical Circuits Using Full-field Optogenetic Stimulation"
The Brain Initiative Meeting 2023
, 2023
Citation
Details
Macknik, Stephen L. and Alexander, Robert G. and Caballero, Olivya and Chanovas, Jordi and Nielsen, Kristina J. and Nishimura, Nozomi and Schaffer, Chris B. and Slovin, Hamutal and Babayoff, Amit and Barak, Ravid and Tang, Shiming and Ju, Niansheng and Ya
"Advanced Circuit and Cellular Imaging Methods in Nonhuman Primates"
The Journal of Neuroscience
, v.39
, 2019
https://doi.org/10.1523/JNEUROSCI.1168-19.2019
Citation
Details
Macknik, Stephen Martinez-Conde
"An Optogenetic Brain System (OBServ) to Restore Visual Perception in the Blind"
Society for Neuroscience (SfN) 2022 Conference
, 2022
Citation
Details
Martinez-Conde, Susana and Alexander, Robert G. and Blum, Deborah and Britton, Noah and Lipska, Barbara K. and Quirk, Gregory J. and Swiss, Jamy Ian and Willems, Roel M. and Macknik, Stephen L.
"The Storytelling Brain: How Neuroscience Stories Help Bridge the Gap between Research and Society"
The Journal of Neuroscience
, v.39
, 2019
https://doi.org/10.1523/jneurosci.1180-19.2019
Citation
Details
Martinez-Conde, Susana and Macknik, Stephen L. and Heeger, David J.
"An Enduring Dialogue between Computational and Empirical Vision"
Trends in Neurosciences
, v.41
, 2018
https://doi.org/10.1016/j.tins.2018.02.005
Citation
Details
Martinez-Conde, Susana and McCamy, Michael B. and Troncoso, Xoana G. and Otero-Millan, Jorge and Macknik, Stephen L.
"Area V1 responses to illusory corner-folds in Vasarelys nested squares and the Alternating Brightness Star illusions"
PLOS ONE
, v.14
, 2019
https://doi.org/10.1371/journal.pone.0210941
Citation
Details
Ojemann, William K.S. and Griggs, Devon J. and Ip, Zachary and Caballero, Olivya and Jahanian, Hesamoddin and Martinez-Conde, Susana and Macknik, Stephen and Yazdan-Shahmorad, Azadeh
"A MRI-Based Toolbox for Neurosurgical Planning in Nonhuman Primates"
Journal of Visualized Experiments
, 2020
https://doi.org/10.3791/61098
Citation
Details
Otero-Millan, Jorge and Langston, Rachel E. and Costela, Francisco and Macknik, Stephen L. and Martinez-Conde, Susana
"Microsaccade generation requires a foveal anchor"
Journal of eye movement research
, v.12
, 2019
https://doi.org/10.16910/jemr.12.6.14
Citation
Details
Waite, Stephen and Farooq, Zerwa and Grigorian, Arkadij and Sistrom, Christopher and Kolla, Srinivas and Mancuso, Anthony and Martinez-Conde, Susana and Alexander, Robert G. and Kantor, Alan and Macknik, Stephen L.
"A Review of Perceptual Expertise in Radiology-How it develops, How we can test it, and Why humans still matter in the era of Artificial Intelligence"
Academic Radiology
, v.27
, 2020
https://doi.org/10.1016/j.acra.2019.08.018
Citation
Details
(Showing: 1 - 10 of 34)
(Showing: 1 - 34 of 34)
PROJECT OUTCOMES REPORT

Disclaimer
This Project Outcomes Report for the General Public is displayed verbatim as submitted by the
Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or
recommendations expressed in this Report are those of the PI and do not necessarily reflect the
views of the National Science Foundation; NSF has not approved or endorsed its content.
We developed an optogenetic technology to stimulate connections within the visual pathways of the brain, in which we genetically turned brain cells within the visual system into photoreceptors, so that we could stimulate them with a video projector to produce artificial vision that did not require input from the retinas. The system included a novel brain activity measurement system using genetically encoded multicolor bioluminescence in V1 cells, suitable for use as a real-time control feedback mechanism to adjust brain stimulation using light for optimized power input. We have designed the initial nanophotonic hardware—based on our empirical findings and simulations—to combine light emission and detection in a novel planar photonic chip for implantation into the brain. Through experimentation and modeling of light scatter in the cortex (modeled as a scattering medium) we have achieved the proof of concept for these tools, and shown that their combined stimulation/detection usage is suitable for implementation as an implantable prosthetic device to restore vision in blind patients. The Optogenetic Brain System (OBServ) is an integrated nanophotonic implantable device that aims to restore foveal vision in the blind by mimicking naturalistic visual input patterns. By transducing brain neurons that normally receive input from the optic nerves--which is unavailable in certain forms of blindness--neurons with light-sensitive proteins, the system can stimulate the connections that enter the visual cortex without unwanted co-activation of inhibitory neurons, theoretically resulting in maximal contrast sensitivity. By targeting these specific localized brain inputs, naturalistic vision can theoretically be restored by providing a pattern of stimulation that mimics natural visual input. This project provided the proofs of concept suggesting that the project can in theory function. OBServ will track eye movements in the blind patients to account for oculomotor effects by adjusting the contemporaneous stimulation of the brain inputs that we artificially stimulate to mimic the effects eye movements. This prosthetic technology does break through the membrane covering the brain and holds the promise of restoring vision in the blind at the highest attainable acuity, with maximal contrast sensitivity while being the most non-invasive implant on the market.
Last Modified: 07/31/2023
Modified by: Stephen L Macknik
Please report errors in award information by writing to: awardsearch@nsf.gov.