The ability to sense bodily movement unlocks a wealth of potential for wearable, augmented reality, and virtual reality applications. While strides have been made in integrating movement sensing into everyday life — just see the success of step-tracking fitness wearables for evidence of that — there’s work still to be done on making it both comprehensive and comfortable.

A quartet of researchers at the German Research Center for Artificial Intelligence (DFKI) and TU Kaiserslautern claim to have come up with a potential answer to that problem, with the development of a wearable posture- and gesture-detecting system built into a loose-fitting blazer — based, intriguingly, on the core technology behind the theremin musical instrument.

Music to your limbs

The theremin, then known as the ætherphone, was developed by Russian physicist Lev Sergeyevich Termen in 1919 as part of a government project into proximity sensors. Built using two antennas, one loop-shaped and one upright, the instrument is controlled by waving your hands near — but never touching — both to control the frequency and the amplitude of the synthesized audio.

That the theremin is, in effect, a gesture-sensing instrument is well-known; previously unknown, however, is that the exact same technology can be adapted to wearable gesture-sensing systems — with surprisingly little modification: “We substituted the metal rod [antennas] with soft wires and integrated them inside clothing,” the researchers explain.

“The idea behind our work is that different distances between body parts can describe different postures; thus, appropriately shaped antennas and embedded in garments will result in specific frequency profiles.”

The MoCapaci project uses musical technology and deep learning to detect 20 different positions and gestures.

The project’s prototype MoCaBlazer, part of a capacitive sensing platform the team has called MoCapaci, is literally built using theremin technology: Its driving hardware is the OpenTheremin, an open-source controller for home-brew theremin builders, modified only by the replacement of a single capacitor to minimize cross-talk between the two-board four-channel installation inside a Tom Tailor men’s blazer.

As well as the OpenTheremin boards — one per pocket, supporting two channels each — the blazer holds four soft antennas around the chest, shoulders, back, and arms, designed so as not to alter the blazer’s fit, and a Teensy 4.1 development board which samples data from the OpenTheremin boards at 100Hz per channel and transmits it to a Python program running on a host computer.

Deep-learning for ease of movement

These data were tied by the researchers to 20 distinct poses and gestures — ranging from leaning and turning in various directions to clapping, shrugging, scratching one’s head in confusion, and even flapping one’s arms like a bird — and fed through deep-learning models to train a network for gesture recognition.

Finalizing on a modified version of the 1D-LeNet5 model, which was fed filtered and normalized data captures cropped and resampled to 400 time-steps to provide a fixed-length input for each gesture, the team saw impressive success.

In leave-recording-out (LRO) training the recognition accuracy hit 97.18 per cent, entirely competitive with state-of-the-art alternatives, while switching to leave-person-out to ensure the system would work regardless of who wore the blazer dropped accuracy to a still impressive 86.25 per cent — with nine of the most distinct gestures recognized with over 95 per cent accuracy.

The team has proposed adding antennas over the shoulder blades, as a means of improving the ability to distinguish between certain gestures.

The MoCaBlazer brings with it a range of advantages over rival tightly-fitted sensing systems, too, with the researchers highlighting its comfort, ease of fit, insensitivity to sweat, and an intrinsic robustness against value-drifting from floating grounds in the capacitive sensing system.

The researchers have also proposed a range of improvements, including adding additional antennas to cover the shoulder blades and to build garments of differing sizes rather than a single one-size-fits-all prototype, and are to investigate miniaturization and sensor fusion with other motion-tracking approaches including the use of radio-frequency identification (RFID) tags.

“One potential use case,” the team notes of its work, “is as a more sophisticated game controller for gesture-based games, such as the Nintendo Wii Rayman Raving Rabbids: TV Party — Shake TV [game] or Just Dance.”

The paper on MoCapaci is available under open-access terms on the ACM Digital Library, following its presentation at the International Symposium on Wearable Computers (ISWC '21).

Reference

Hymalai Bello, Bo Zhou, Sungho Suh, and Paul Lukowicz: MoCapaci: Posture and gesture detection in loose garments using textile cables as capacitive antennas, 2021 International Symposium on Wearable Computers (ISWC '21). DOI 10.1145/3460421.3480418.