Touchless interface interaction by hand tracking with a depth camera and a convolutional neural network

Audrey Leong-Hoi; Sarah Murat; Tanguy Acker; Quentin Sattler; Jean-Pierre Radoux

doi:10.1117/12.2555676

1 April 2020 Touchless interface interaction by hand tracking with a depth camera and a convolutional neural network

Audrey Leong-Hoi, Sarah Murat, Tanguy Acker, Quentin Sattler, Jean-Pierre Radoux

Proceedings Volume 11353, Optics, Photonics and Digital Technologies for Imaging Applications VI; 113531Z (2020) https://doi.org/10.1117/12.2555676
Event: SPIE Photonics Europe, 2020, Online Only

Abstract

In operating rooms (OR), physicians have to work in strict compliance to asepsis rules to not endanger the health of patients. In the case of laparoscopic mini-invasive surgery, where the field of view of the surgeon is restricted, the use of computers is necessary to provide the missing information. Physicians must therefore interact with computers either by directly manipulating mouse and keyboard, by removing their gloves or by using a protection for the devices, or by voicecommanding an assistant to do so. However, in addition to be time-consuming, it may cause hygiene issues in the first case and a lack of precision in the second. The need to have better way of interactions with computers had led to important researches in that area during the last ten years, especially in Touchless Human Machine Interaction (THMI). Indeed, THMI, including gesture recognition, voice recognition and eye-tracking, has a promising future in the medical field, allowing surgeons to interact by themselves with devices, thereby avoiding error-prone process while complying to asepsis rules. In this context, the “Intelligent Touchless Glassless Human-Machine Interface” (ITG-HMI) project aims to provide a new tool for viewing and manipulating 3D objects. In this article, we present how this interface was implemented, through the detection and recognition of hand gestures using Deep Learning, the establishment of a graphical interface to display 3D models and the adaptation of gestures recognized in actions to achieve.

Conference Presentation

Citation Download Citation

Audrey Leong-Hoi, Sarah Murat, Tanguy Acker, Quentin Sattler, and Jean-Pierre Radoux "Touchless interface interaction by hand tracking with a depth camera and a convolutional neural network", Proc. SPIE 11353, Optics, Photonics and Digital Technologies for Imaging Applications VI, 113531Z (1 April 2020); https://doi.org/10.1117/12.2555676

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available