The proposed HGRP framework is established based on a cloud computing platform and it is used to enable research capabilities in the following areas: (a) hand-gesture biometrics; (b) cognitive Robotics; and (c) programmable interfaces for gesture-based data processing and visualization. HGRP is composed of the following major components:

• Data collection based on two major types of sensors: (a) motion detection sensors, e.g., wearable sensors such as a watch, wrist band, on figure sensors, data motion gloves, infrared motion detection sensors, etc., and (b) video sensors such as leap motion sensors, video recorders, etc. The detected hands-gesture data is sent to data storage for processing and storing.
• A GPU-based private cloud is established to allow researchers to implement well- known hand-gesture data processing models and establish benchmarking models. HGRP also allows researchers to submit computation tasks for evaluations and comparative studies.
• HGRP provides a web-based data repository that allow researchers remotely to access the hand-gesture data for research.

We organized the outcomes of this project in a more user-friendly way. The generated datasets and tool kits are collectively called FMKit. It includes a code library and data repository for finger motion-based in-air-handwriting analysis. For details of the FMKit project, please refer to the project home page and a 2-minute short video introduction:

• Project homepage: https://github.com/duolu/fmkit
• Introduction video: https://youtu.be/O3Jqq9yqJSE
• Research publication describing the dataset: Duo Lu, Linzhen Luo, Dijiang Huang, Yezhou Yang, "FMKit: An In-Air-Handwriting Analysis Library and Data Repository." CVPR Workshop on Computer Vision for Augmented and Virtual Reality, 2020. (https://github.com/duolu/fmkit/blob/master/papers/fmkit.pdf ) [link] 
• FMKit project documentation: https://duolu-fmkit.github.io/

FMKit contains five datasets of in-air-handwriting signals collected from over 200 participants with two different hand motion capture devices. All datasets are openly and freely available to researchers. For details of the datasets, please refer to:

• FMKits data repository description https://github.com/duolu/fmkit#the-data-repository

To download the data repository for research and evaluation purposes, the project team provides a dataset access procedure by submitting a data acquisition letter to fmkit@googlegroups.com to request access. The datasets are free of access and use for research and evaluation purposes.

• Application letter:  https://docs.google.com/document/d/1AHX3lj1mjm4ZZEZTHNdm3xDmJAAWi7P6bIdLlBNZ8PA/edit?usp=sharing

A new dataset, called the word-210 dataset, can be downloaded directly without needing the data access application. Word-210 dataset includes 10 users? in-air handwriting of 210 English words and 210 Chinese words and each with 5 repetitions: 

• Word-210 dataset: https://www.thothlab.com/getgooglefile/1RXj0t8NMYt_Jr5lW-BIeAIxw4aX5VshV

FMKit also provides a Python code library to process the signals and interface with other Python tools. You can download the FMKit codes at the following links:

• https://github.com/duolu/fmkit/tree/master/code_fmkit

Hand-gesture research is an emerging inter-disciplinary research area. It has been actively involved research areas such as motion detection, visualization, robotics, ML/AI, and cybersecurity. Newly developed models and solutions including behavioral modeling and ML/AI-based solutions to make the hand-gesture applications more adaptive and easier to use.  The developed education materials will significantly promote the research in these directions. 

Last Modified: 06/03/2021
Modified by: Dijiang Huang