This project is aimed at introducing, creating and developing an integrated set of cyber-physical methods and tools to sense, understand, characterize, model, and optimize the learning and operations of assembly workers, so as to achieve smart manufacturing with significantly improved efficiency of work training, effectiveness of operations management, and safety of front-line workers.

The project's outcomes in terms of intellectual merit include the following:

• We created a foundation for building multimodal sensor-based action recognition systems by fusing and refining convolutional neural network models. Based on this foundation, we developed a prototype multimodal sensor-based action recognition system and demonstrated using this system for worker activity recognition in human-centered mechanical assembly using data from an inertial measurement unit and a video camera.
• We developed a smart instructional system incorporating augmented reality, with the support of a deep learning network for detection of tools, parts and worker activities in manual assembly. We have demonstrated and evaluated this smart instructional system in assembling a CNC carving machine performed by a human worker.
• We developed a fog computing approach to bring computing power closer to the data source than cloud computing in order to achieve real-time worker assembly action recognition, and based on this approach we demonstrated a transfer learning model?s ability to achieve high recognition accuracy.
• We created a novel video-based human action recognition network which integrates discriminative feature pooling with a video segment attention model. This action recognition network has been shown to outperform the state-of-the-art action recognition networks when evaluated on four widely benchmarked datasets.
• We created a method to develop an individualized system of convolutional neural networks for assembly action recognition using human skeletal data. The system comprises CNN classifiers adapted to any new worker through transfer learning and iterative boosting, followed by an individualized fusion method that integrates the adapted classifiers into a real-time action recognition system. This individualized system of skeletal data-based CNN classifiers for action recognition improves the accuracy of action recognition compared to the CNN classifiers directly built with the skeletal data.
• We introduced a context and structure mining network for video object detection. This network includes an encoding module to encode the spatial-temporal context information in video frames into object features, and an aggregation module to better aggregate structure-based features with temporal information in support frames.
• We introduced a class-aware feature aggregation network for video object detection by putting the video object detection to the edge, and showed this network achieving state-of-the-art performance on the commonly used ImageNet VID dataset without use of any post-processing methods.
• We introduced a convolutional neural network that embeds a novel discriminative feature pooling mechanism and a novel video segment attention model, for video-based human action recognition from both trimmed and untrimmed videos. Based on this method, we developed an action recognition network and demonstrated that this network can be trained using both trimmed videos in a fully supervised way and untrimmed videos in a weakly supervised way.
• We introduced a novel method of weakly-supervised Action Completeness Modeling with Background Aware Networks (ACM-BANets) to address two main challenges in smart manufacturing: (1) how to design and train a weakly-supervised network that can suppress both highly discriminative and ambiguous frames in order to remove the false positives? and (2) how to design a temporal action localization framework in order to discover action instances in both highly discriminative and ambiguous action frames for the complete localization?
• We explored the unique characteristics of human trajectories and introduced a new approach, called reciprocal network learning, for human trajectory prediction. Extensive experimental results obtained using this approach on public benchmark datasets showed that this method outperforms the state-of-the-art human trajectory prediction methods.

The project's outcomes in terms of broader impacts include the following:

• This project has contributed to the smart manufacturing literature through the publication of 11 journal papers, 11 peer-reviewed conference papers, and 1 book chapter. The research results including the developed frameworks, methods, algorithms, and tools for multimodal sensing, data analytics, deep learning, predictive modeling, and augmented reality have contributed significantly to the field of smart manufacturing.
• Three senior investigators, including one female, were involved in this collaborative research project, which provided research training opportunities for 10 Ph.D. students, 1 M.S. student, and 14 undergraduate students over the project duration. The involved faculty and students were from multiple disciplines, and all the project personnel gained valuable experiences in teamwork and convergent research.
• The project has improved the research infrastructure of the participating universities, with laboratories built that enable further research on manufacturing cyber-physical systems involving multimodal sensor fusion, deep learning algorithms for data analytics, and development of augmented reality assistive systems for human-centered intelligent manufacturing.  

Last Modified: 01/03/2022
Modified by: Ming C Leu