Bi-Perceptive Electronic Skin of Curvature and Magnetism Fusion with Autonomous Learning and Augmented Feedback for Wearable Gesture Interaction

Wearable devices, such as data gloves and electronic skins, can perceive human hand's actions, behaviors and even emotions with the help of knowledge learning and inference. Curvature or magnetism sensing in such devices often lacks comprehensive gesture interactive information, meanwhile, the limited computing power of wearable applications restricts the multi-mode fusion of different sensing data and the deployment of deep learning networks. We present bi-perceptive electronic skins which can detect the changes in magnetization state of magnetostrictive alloy film under external force/magnetic field simultaneously. The sensor realizes fusion of curvature (attitude) and magnetic field (position) information by mapping corresponding signals to the two-directional continuous distribution whose both ends represent contribution of curvature and magnetic field respectively. When installed at the joint of hand, this sensor can perceive hand posture and position simultaneously, ultimately achieving an interactive platform with cross-space perception and manipulation based on gesture recognition algorithm and augmented feedback. We also develop a reinforced knowledge distillation method, for learning and compressing the teacher models by deep reinforcement learning, so as to select knowledge closer to the user's characteristics. By fusing curvature-magnetism dual information sensing and haptic feedback functions, this sensor can be extended for action interaction in virtual environments.