Award Abstract # 1302338
HCC: Medium: Combining Crowdsourcing and Computer Vision for Street-level Accessibility

NSF Org: IIS
Division of Information & Intelligent Systems
Recipient: UNIVERSITY OF MARYLAND, COLLEGE PARK
Initial Amendment Date: May 16, 2013
Latest Amendment Date: May 16, 2013
Award Number: 1302338
Award Instrument: Standard Grant
Program Manager: Ephraim Glinert
IIS
 Division of Information & Intelligent Systems
CSE
 Directorate for Computer and Information Science and Engineering
Start Date: May 1, 2013
End Date: April 30, 2019 (Estimated)
Total Intended Award Amount: $1,199,034.00
Total Awarded Amount to Date: $1,199,034.00
Funds Obligated to Date: FY 2013 = $1,199,034.00
History of Investigator:
  • Jon Froehlich (Principal Investigator)
    jonf@cs.washington.edu
  • David Jacobs (Co-Principal Investigator)
Recipient Sponsored Research Office: University of Maryland, College Park
3112 LEE BUILDING
COLLEGE PARK
MD  US  20742-5100
(301)405-6269
Sponsor Congressional District: 04
Primary Place of Performance: University of Maryland College Park
College Park
MD  US  20742-5141
Primary Place of Performance
Congressional District:
04
Unique Entity Identifier (UEI): NPU8ULVAAS23
Parent UEI: NPU8ULVAAS23
NSF Program(s): HCC-Human-Centered Computing
Primary Program Source: 01001314DB NSF RESEARCH & RELATED ACTIVIT
Program Reference Code(s): 7367, 7924
Program Element Code(s): 736700
Award Agency Code: 4900
Fund Agency Code: 4900
Assistance Listing Number(s): 47.070

ABSTRACT

Despite comprehensive civil rights legislation for Americans with disabilities, many city streets, sidewalks, and businesses remain inaccessible. The problem is not just that street-level accessibility affects where and how people travel in cities but also that there are few, if any, mechanisms to determine accessible areas of a city a priori. Traditionally, sidewalk assessment has been conducted via in-person street audits, which are labor intensive and costly, or via citizen call-in reports, which are done on a reactive basis. And while efforts exist for visualizing the walk-ability, bike-ability, and availability of public transport in cities, there are no analogous efforts for accessibility. Thus, wheelchair users, for example, often avoid going to new areas of a city where they don't know about accessible routes. The PI plans to address this problem by means of a two-pronged approach in which he will first develop scalable data collection methods for acquiring sidewalk accessibility information using a combination of crowd-sourcing, computer vision, and online map imagery; he will then use the new data to develop and evaluate a novel set of navigation and map tools for accessibility. To these ends, the PI and his team will collect and analyze interview and survey data both from mobility impaired persons and from ADA streetscape design experts, and will seek to understand how people with mobility impairments can make use of interactive mapping information to enhance mobility. They will study methods for efficiently and effectively crowd-sourcing map labeling tasks, evaluating existing approaches empirically and designing novel, more effective approaches. They will develop new computer vision algorithms for the analysis of street scenes, which will be used to help scale the data collection by focusing human labeling efforts on locations that are most likely to contain significant problems. And they will design, implement and evaluate new accessible-aware map-based tools to aid people with mobility impairments in navigating their cities. As appropriate for each phase of the research, user evaluations will include both lab and field studies.

Broader Impacts: Roughly 30.6 million individuals in the United States have physical disabilities that affect their ambulatory activities, and nearly half of these individuals report using an assistive aid such as a wheelchair, cane, crutches, or walker. The outcomes from this research will have a significant impact on the ability of these Americans to travel independently, by transforming the ways in which accessibility information is collected and visualized for every sidewalk, street, and building façade in America. Project outcomes will include a publicly accessible web site where both the labeled data collected during this work and the new prototype tools developed will be made available for general use. Furthermore, the PI and Co-PI will advise and mentor both graduate and undergraduate students throughout the course of the project, including two PhDs and two MS students who will obtain a cross-disciplinary education in human-computer interaction and computer vision.

PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:  When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

Froehlich, Jon E. and Brock, Anke M. and Caspi, Anat and Guerreiro, Jo\~{a}o and Hara, Kotaro and Kirkham, Reuben and Sch\"{o}ning, Johannes and Tannert, Benjamin "Grand Challenges in Accessible Maps" Interactions , v.26 , 2019 , p.78--81 10.1145/3301657
Hara, Kotaro and Azenkot, Shiri and Campbell, Megan and Bennett, Cynthia L. and Le, Vicki and Pannella, Sean and Moore, Robert and Minckler, Kelly and Ng, Rochelle H. and Froehlich, Jon E. "Improving Public Transit Accessibility for Blind Riders by Crowdsourcing Bus Stop Landmark Locations with Google Street View: An Extended Analysis" ACM Trans. Access. Comput. , v.6 , 2015 , p.5:1--5:23 10.1145/2717513
Jin SunDavid Jacobs "Seeing What Is Not There: Learning Context to Determine Where Objects Are Missing" CVPR , 2017
Kotaro Hara, Christine Chan, Jon E. Froehlich "The Design of Assistive Location-Based Technologies for People With Ambulatory Disabilities: A Formative Study" Proceedings of CHI 2016 , 2016

PROJECT OUTCOMES REPORT

Disclaimer

This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.

Sidewalks significantly impact the mobility and quality of life of millions of Americans; however, there are currently few, if any, mechanisms to determine the locations of sidewalks in the US and their accessibility characteristics (e.g., presence of curb ramps, sidewalk obstructions). To address this problem, our NSF proposal described two interrelated threads of work: first, to develop scalable data collection methods for acquiring street-level accessibility information using a combination of crowdsourcing, machine learning, and online map imagery, and second, to use this data to design, develop, and evaluate a novel set of navigation and map tools for accessibility. Our overarching aim has been--and continues to be--to transform how sidewalk data is collected and visualized in order to: provide increased transparency and accountability about city accessibility, help people with mobility impairments assess and navigate their cities, and assist urban planners and policy makers in decision making about pedestrian infrastructure.

In pursuit of these goals, our cross-disciplinary team has made research contributions to the fields of human-computer interaction, assistive technology, computer vision, and geographical information systems. Below, we first summarize key achievements before describing our research in more detail.

Key Achievements

  • We developed and deployed award-winning interactive software artifacts, including Project Sidewalk (an online crowdsourcing tool for collecting sidewalk accessibility information) and AccessVis (interactive visualizations of urban accessibility). All of our code is open source here: https://github.com/ProjectSidewalk.
  • Our work has produced what we believe to be the largest open-source sidewalk accessibility dataset: over 300,000 geo-located sidewalk accessibility labels across three cities: http://projectsidewalk.io/api
  • Our deployed tools have received substantial interest from government and accessibility organizations (e.g., DDOT, USDS, White House OSTP, AARP) as well as media coverage.
  • We have published twenty scientific publications, including papers at top-tier venues such as CHI, ASSETS, and CVPR--two were honored with Best Paper Awards (Hara et al., ASSETS'13 and Saha et al., CHI'19) and one was selected for the ACM Computing Reviews "Best of Computing 2014."
  • PI Froehlich and Co-PI Jacobs have involved and mentored a diverse set of students from high school to undergrad and graduate students, including Kotaro Hara and Jin Sun's PhD dissertations, Ladan Najafizadeh's MS thesis, and multiple independent study projects with undergraduates. Our students have gone on to top industry positions and academic institutions such as UCLA, Stanford, and Cornell.
  • While our original proposal focused primarily on people with mobility impairments, our work expanded to people who are blind or low-vision (e.g., Hara et al., ASSETS'13 , which received the Best Paper Award and Hara, TACCESS'15)--demonstrating the generalizability of our methods.

Thread-1: Scalable Data Collection

In initial work, we showed how minimally trained remote crowdworkers can accurately label accessibility features in streetscape imagery (Figure 1; Hara et al., ASSETS'12; Hara et al., CHI'13). Later, we demonstrated the scalability of this approach via Project Sidewalk (Saha et al., CHI'19), an online tool where crowdworkers can remotely label pedestrian-related accessibility problems by virtually walking through city streets in Google Street View (Figure 2). In three test deployments, we have collected over 300,000 geo-located sidewalk accessibility labels (Figure 3).

While the above approaches are promising and improve data collection efficiency, they still rely solely on human labor, which limits scalability. Thus, we pursued a parallel thread of work examining how humans + computers could work together to semi-automatically assess sidewalk accessibility in online images. Our initial work focused on identifying curb ramps or missing curb ramps (e.g., Hara et al., HCOMP'13; Hara et al., UIST'14; Sun and Jacobs, CVPR'17). Hara et al., UIST'14, for example, showed that a hybrid solution that incorporates both human labels plus machine learning could detect curb ramps at a rate similar to humans alone but at a 13% reduction in time cost (Figure 4). More recently, enabled by our large Project Sidewalk dataset, we show how a convolutional neural network can significantly improve on previous automated methods and, in some cases, meet or exceed human labeling performance (Weld et al., ASSETS'19).

Thread-2: Accessibility-Aware Mapping Tools

In this thread, we explored how people with mobility impairments assess and evaluate accessibility in the built environment and the role of current and emerging location-based technologies therein (Hara et al., CHI'16), developed an initial set of new accessibility-aware mapping tools enabled by our Project Sidewalk data (Figure 5; Hara et al., CHI'16; Hara and Froehlich, SIGACCESS'15; Li et al., ASSETS'18), and interviewed key stakeholder groups about their reactions to our designs and the idea of crowdsourced accessibility data (Hara et al., CHI'16; Saha et al., CHI'19). We also enumerated and defined a set of 'Grand Challenges' in the area of accessibility and mapping (Froehlich et al., Interactions'19). Our research findings distill key features and data qualities of accessibility-aware mapping tools, help define a design space for future tools, and have been used to produce our own interactive visualizations (e.g., AccessScore and AccessVis).

 


Last Modified: 08/01/2019
Modified by: Jon Froehlich

Please report errors in award information by writing to: awardsearch@nsf.gov.

Print this page

Back to Top of page