TapType: Ten-finger text entry on everyday surfaces via Bayesian inference

Problem / Solution

The new types on the keyboards of the latest portable gadgets are often shrunk or sparse to fit their screen sizes. This reduces the comfort, accuracy, and speed of typing compared to a regular-sized keyboard.

TapType is a mobile text entry system that supports full-size touch typing on flat surfaces without the actual QWERTY keyboard. The sensors of the two wristbands register the taps of typing through inertial sensors, and send them to the backend. The system predicts the most likely character sequences by using a Bayesian neural network classifier and an n-gram language model. An average of 19 words per minute with a character error rate of 0.6% is achieved after 30 minutes of training.

Design

Hardware

TapType is composed of a silicone wristband, two accelerometers, and a mainboard with a system on a chip, Bluetooth Low Energy (BLE), and a 3V coin cell battery.

The electronics are cast in the flexible wrist strap made from TFC elastic silicone. The silicone wrist strap firmly and comfortably attaches to the wearer’s skin, and helps couple the tap events to the accelerometers. 

The identification of tapping fingers from mechanical vibrations is recorded using two accelerometer sensors placed near the ulna and radius. The dual-inertial sensor design features two ultra-low power three-axis accelerometers (BMA456) for increased precision by providing higher resolution of 16-bit, ±2G, and higher sampling rates of 1600 Hz. The accelerometers are attached at the ends of the wrist strap, with the mainboard in the middle. 

The mainboard contains the system on a chip (DA14695), Bluetooth Low Energy (BLE), and a 3V coin cell battery (CR2032). The accelerometers measure the vibrations of the typing hands and send the data to the system on a chip. Then, the data is sent to a backend via the BLE antenna. The backend predicts the typed character. TapType consumes 10-15 mW and lasts approximately 30 hours on a 3V coin cell battery.

Bayesian classifier and Text Entry Decoder

The apparatus captures representative accelerometer signals for typing input in order to collect samples for the TapType System's learning-based approach. All hand motions, including touch and tap events, are included in the continuous stream of accelerations. Ground-truth touch events are recorded to label sensor data with the correct finger identity. Four mounted cameras record the motions of participants’ fingertips in midair before and after the type event. The reconstructed finger movements are combined with the recorded events and coordinates to obtain records of finger identities, touch locations, and all IMU streams. Spurious inputs that refer to recorded events in the IMU streams that do not correspond to a touch event are identified. The setup allows for supervised training of the Bayesian classifiers. 

A dataset of a large number of sentences from different sources are used as training data. The n-gram language model is built using Witten-Bell smoothing for characters, and modified Kneser-Ney smoothing for word sequences. 

The text entry decoder predicts the most likely character sequence by cross analyzing the finger probabilities from the Bayesian neural network tap classifier and characters’ prior probabilities from the n-gram language model.