This project was primarily focused on the development and assessment of a new type of a patient simulator that could some day be widely used for the training and assessment of nurses and physicians. Specifically the researchers developed what they call a Physical-Virtual Patient Bed (PVPB) that combines the flexibility of computer graphics "virtual" patients with the physicality of conventional medical training manikins. See the included images of their pediatric PVPB prototype system alone and in use. The researchers created multiple PVPB prototypes. The prototypes employed dynamic computer graphics imagery that was projected onto the inside (from underneath) of a human body-shaped shell that was mounted in a metal frame resembling a hospital bed. The resulting system would display a dynamic patient, such as a child, on the body-shaped surface, such that the practitioner could stand next to the "bed" and talk with the patient, while touching their body when and where needed for diagnosis or comfort. The prototypes also included various sensors and actuators. For example they used sound and projected imagery to simulate patient talking and breathing; electronic tactile transducers to create a pulse that the practitioners could feel with their own hands; and forced air heating and cooling from underneath to create a feeling of warmth on the forehead (indicating a fever) or cold on their hands (indicating shock). Some prototypes incorporated cameras, infrared lighting, and novel computer algorithms to detect when a practitioner touched the simulated patient, and to respond by changing the patient imagery, e.g., to allow the practitioner to pull down on the patient's eyelid to check their sclera. The prototypes were able to change race, gender, and various visually-apparent symptoms on the fly. The simulated behavior was created by a remote human who observed the scene and controlled the patient response via a special computer interface. The result was very realistic human emotional intelligence and complexity. The researchers also carried out human subject research designed to evaluate the value and effectiveness of the PVPB paradigm and specific prototypes. The studies employed nursing and medical students who used the PVPB under various circumstances, e.g., stroke diagnosis, cases of the measles, and certain pediatric (children) needs.

During the COVID-19 pandemic the researchers were unable to access the specialized PVPB equipment, or to meet with nursing and medical students in person. Under these circumstances the researchers conceived of a new pandemic-related technology to provide an isolated patient and their remote visitors with a visual interaction augmented by touch --- a perception of being touched for the isolated patient, and a perception of touching for the remote visitors. For example, a loved one might be able to virtually stroke the patient?s arm or head, or even squeeze the patient's hand. The researchers called the approach Tactile Telepresence for Isolated Patients (TTIP). The researchers developed a complete functioning TTIP prototype system employing a "smart tablet" for the family member to hold and a tactile headband for the patient. See the included images of their TTIP "headband" electronics and the headband in use. Imagery of the patient was shown on the tablet, along with indications of areas where one could touch the patient image such that the sensations were transmitted to the patient's forehead via the TTIP headband, which included an array of small vibration devices. The researchers carried out some preliminary (pilot) experiments to evaluate how well the TTIP prototype functioned, and how user's perceived the remote touching. The experiments demonstrated a relatively high rate of recognition of tracing out geometric shapes on a remote person's forehead, and a feeling of not being alone when tested under conditions that simulated real isolation.

Over the life of the project the researchers published and/or presented over 25 relevant articles and several conference posters. There were 12 faculty members involved at various times, and there were four graduate students (with four PhD Degrees) and nine undergraduate students (general undergraduates and NSF REU students) involved.

Last Modified: 11/23/2022
Modified by: Gregory F Welch