In this research project, we developed REIP, a novel Reconfigurable Environmental Intelligence Platform for fast sensor network prototyping. REIP consists of hardware and software components, its most central capability being an open-source software framework (SDK) with a flexible modular API for data collection and analysis using multiple sensing modalities. It was developed to be user-friendly, device-agnostic, and easily extensible, allowing for fast prototyping of heterogeneous sensor networks. REIP is implemented in Python to widen accessibility and encourage future contributions by lowering the learning curve.

The potential and versatility of REIP have been validated through real-world deployments. One such instance involved acquiring the StreetAware dataset, a rich multimodal urban scene collection consisting of high-resolution audio, video, and LiDAR data from three Brooklyn intersections totaling approximately 7 unique hours. The audio and video, which come from multiple REIP devices, are accurately synchronized across the corresponding video and audio inputs, enabling novel applications, such as: (1) discovering and locating occluded objects, (2) associating audio events with their respective visual representations using both video and audio modes, (3) tracking the amount of each type of object in a scene over time, and (4) measuring pedestrian speed using multiple synchronized camera views.

This data enables the study of data-driven solutions for urban problems, and the techniques we are developing can be incorporated into the future built environment to enhance accessibility and equity (e.g., traffic lights that adapt to pedestrians' needs).

As part of our research efforts, we have developed techniques and systems for analyzing, processing, and visualizing large collections of city imagery. We developed Urban Mosaic, a tool for exploring the urban fabric through images that is capable of handling spatially and temporally dense datasets comprising millions of images.  We have also designed CitySurfaces, an active learning-based framework that leverages computer vision techniques for classifying sidewalk materials using street-level images. Sidewalk material information has many practical uses but is seldom available for US cities. The final contribution that we would like to highlight is Tile2Net, an end-to-end open-source tool for extracting sidewalk, crosswalk, and footpath polygons from orthorectified aerial imagery using semantic segmentation.

This project helped to establish the Visualization Lab at New York University, which has contributed to the education of PhD, MS, undergraduate, and even K-12 students. The lab's facilities will continue to serve generations of students to come. The REIP project has given rise to several open-source projects, some of which have already gained substantial external visibility and interest. The project has also produced significant data releases (e.g., StreetAware). Collectively, the REIP hardware and software along with the additional delivered software and data releases have already impacted data-driven urban science, and are likely to continue shaping this rapidly evolving research area.

Last Modified: 02/18/2024
Modified by: Claudio T Silva