The central goal of this project was to develop a suite of tools for audio-based construction site performance monitoring. In this framework, an audio recording system is placed at a jobsite, and once in place, used to track the locations and usage level of various pieces of equipment in real-time. This can provide a beneficial tool by enabling automatic measurements of the amount of accomplished work, downtime, and other factors.  This can enable analyzing the jobsite as a system and optimizing its overall performance. The use of audio as the primary source of input data has crucial advantages including a manageable data stream size, minimal cost and intrusiveness, and operability in both day and night and under harsh conditions (e.g., compared to video-based monitoring).

The first key step involved in this project was to develop a framework for processing, denoising, and isolating the audio components corresponding to individual machines (from recordings which may contain a mixture of various pieces of equipment operating simultaneously). The next key step involved developing classification methods for recognizing various forms of construction activity for each machine from the isolated/processed recordings using machine learning algorithms that learn the classifier from training data. The final key step involved developing higher-level tools for analyzing and visualizing equipment utilization.

We have made significant contributions related to each of the major goals of the project as listed above, including the development of new signal processing techniques for detection and classification of various machinery using audio and that can be utilized in new environments without the need for extensive training data or labels.  Specific activities included:
-Demonstrating, as a proof of concept, that a proposed method for classifying construction equipment activity using a denoised short-time frequency representation and support vector machines can be highly effective.
-Exploring methods of processing to isolate different pieces of equipment when multiple pieces of equipment are simultaneously active. This included exploring a mixture of signal processing tools and hardware modifications to see what is possible, including various forms of blind-source separation combined with microphone array processing and possible the use of equipment-mounted contact microphones. Using multiple microphones, we developed a system that detected the angle and approximate range of a each piece of equipment using data captured at a real-world construction site.
-A weakness of previous approaches to training classifiers arose due to the fact that we must assign an activity label at a very fine level of granularity in time.  This presents difficulty, since in many time periods during different activities, the acoustic signal is virtually indistinguishable (e.g., when a machine is idling).  To address this we have developed more sophisticated techniques which are based on classifying an entire block/window of time, which necessitated the development of new algorithmic approaches. 

The novel approaches to training the classifier are of particular benefit and are a significant technical contribution.  In order to make an audio-based construction monitoring system useful, it must require little to no training in the field.  Any required labeling or other preparation should be robust and simple.  The techniques for training classifiers that were developed as part of this project have the several advantages:
-Only weak labels are required--this means that rather than labeling specific sounds, the system can work with labels that apply to an entire time period, whether or not the machine was active during that entire time period. An example of a weak label is labeling a 30-second sound clip as "machine digging" even if the digging was only intermittent and for a fraction of the entire clip.
-The training data can have background noise and multiple target sounds and it automatically sorts the various sources and noises into "different bins" so that it can learn specific sounds even in the presence of interference.
-The classifier is efficient and quick to train, making it possible to transition the training and operation to an embedded system.

This project provided a proof-of-concept that an audio-based construction site monitoring system is indeed feasible. While there still remains additional development work to turn this into a commercially viable system, we believe that this project provided a foundation for such future work. 

Last Modified: 09/03/2020
Modified by: Mark A Davenport