The MoCaBlazer integrates two theremin control boards and four soft antennas for position and gesture detection.
The ability to sense bodily movement unlocks a wealth of potential for wearable, augmented reality, and virtual reality applications. While strides have been made in integrating movement sensing into everyday life — just see the success of step-tracking fitness wearables for evidence of that — there’s work still to be done on making it both comprehensive and comfortable.
A quartet of researchers at the German Research Center for Artificial Intelligence (DFKI) and TU Kaiserslautern claim to have come up with a potential answer to that problem, with the development of a wearable posture- and gesture-detecting system built into a loose-fitting blazer — based, intriguingly, on the core technology behind the theremin musical instrument.
The theremin, then known as the ætherphone, was developed by Russian physicist Lev Sergeyevich Termen in 1919 as part of a government project into proximity sensors. Built using two antennas, one loop-shaped and one upright, the instrument is controlled by waving your hands near — but never touching — both to control the frequency and the amplitude of the synthesized audio.
That the theremin is, in effect, a gesture-sensing instrument is well-known; previously unknown, however, is that the exact same technology can be adapted to wearable gesture-sensing systems — with surprisingly little modification: “We substituted the metal rod [antennas] with soft wires and integrated them inside clothing,” the researchers explain.
“The idea behind our work is that different distances between body parts can describe different postures; thus, appropriately shaped antennas and embedded in garments will result in specific frequency profiles.”
The project’s prototype MoCaBlazer, part of a capacitive sensing platform the team has called MoCapaci, is literally built using theremin technology: Its driving hardware is the OpenTheremin, an open-source controller for home-brew theremin builders, modified only by the replacement of a single capacitor to minimize cross-talk between the two-board four-channel installation inside a Tom Tailor men’s blazer.
As well as the OpenTheremin boards — one per pocket, supporting two channels each — the blazer holds four soft antennas around the chest, shoulders, back, and arms, designed so as not to alter the blazer’s fit, and a Teensy 4.1 development board which samples data from the OpenTheremin boards at 100Hz per channel and transmits it to a Python program running on a host computer.
These data were tied by the researchers to 20 distinct poses and gestures — ranging from leaning and turning in various directions to clapping, shrugging, scratching one’s head in confusion, and even flapping one’s arms like a bird — and fed through deep-learning models to train a network for gesture recognition.
Finalizing on a modified version of the 1D-LeNet5 model, which was fed filtered and normalized data captures cropped and resampled to 400 time-steps to provide a fixed-length input for each gesture, the team saw impressive success.
In leave-recording-out (LRO) training the recognition accuracy hit 97.18 per cent, entirely competitive with state-of-the-art alternatives, while switching to leave-person-out to ensure the system would work regardless of who wore the blazer dropped accuracy to a still impressive 86.25 per cent — with nine of the most distinct gestures recognized with over 95 per cent accuracy.
The MoCaBlazer brings with it a range of advantages over rival tightly-fitted sensing systems, too, with the researchers highlighting its comfort, ease of fit, insensitivity to sweat, and an intrinsic robustness against value-drifting from floating grounds in the capacitive sensing system.
The researchers have also proposed a range of improvements, including adding additional antennas to cover the shoulder blades and to build garments of differing sizes rather than a single one-size-fits-all prototype, and are to investigate miniaturization and sensor fusion with other motion-tracking approaches including the use of radio-frequency identification (RFID) tags.
“One potential use case,” the team notes of its work, “is as a more sophisticated game controller for gesture-based games, such as the Nintendo Wii Rayman Raving Rabbids: TV Party — Shake TV [game] or Just Dance.”
The paper on MoCapaci is available under open-access terms on the ACM Digital Library, following its presentation at the International Symposium on Wearable Computers (ISWC '21).
Hymalai Bello, Bo Zhou, Sungho Suh, and Paul Lukowicz: MoCapaci: Posture and gesture detection in loose garments using textile cables as capacitive antennas, 2021 International Symposium on Wearable Computers (ISWC '21). DOI 10.1145/3460421.3480418.