A choreomusical e-textile interactive carpet

This article is a part of our University Technology Exposure Program. The program aims to recognize and reward innovation from engineering students and researchers across the globe. 

When we think of smart fabrics, we usually associate them with wearable and fashion technology, such as clothing that can track your health and physical activity or gloves that can sense tactile information or your finger gestures. Since textiles are ubiquitous in our environments, other realms of textiles, such as flooring and upholstery, will also benefit from functional materials augmentation and technological integration. 

MIT Media Lab researchers have developed a large-scale seamless interactive carpet and established an interdisciplinary project that unites new materials, sensing technologies, and advanced digital fabrication with contemporary dance and music. Tapis Magique project explores the relationship between auditory and physical interaction through an interactive knitted electronic textile interface. The researchers aim to tease the idea of music or an immersive sonic environment that is driven in real-time by movements from dance and choreography and allow dancers to have their own agency over producing and controlling the music. 

Credit Images: Irmandy Wicaksono/MIT Media Lab

The project is motivated by traditional textile's extreme craftsmanship and artistry. Irmandy took some design inspiration from the cultural textiles of Indonesia, such as Javanese and Balinese Batik and Ikat. In these textiles, unique designs and patterns carry specific meanings and wishes. Complementing their traditional dance and music, they unitingly and harmoniously amplify a sense of community, identity, and expression. He hopes that by bringing functionality and interactivity to modern textiles, they will become even more appreciated, treasured, and personal and enhance our interaction with the environments and the digital world. Tapis Magique demonstrates the interplay between art and technology, highlighting the deep emotional link between contemporary textiles, dance, and music through the physical-digital connection. 

As textiles are highly soft, deformable, and palpable materials with a broad spectrum of textures, they are a great candidate for physical interfaces. Irmandy previously knitted a piano-patterned, multimodal musical cloth (KnittedKeyboard II) that simultaneously responds to touch, pressure, stretch, and proximity. This 5-octave soft interface allows intimate and organic finger-fabric interaction. He also collaborated with a composer and keyboardist to demonstrate the multi-dimensionality of this e-textile instrument. With Tapis Magique, he is scaling the knitted e-textile technology and expanding its applications, incorporating footwork and body gestures.

Knitting is an established textile manufacturing technique that has been pervasive in the garment and technical fabrics industry.  In comparison to weaving, knitting has several advantages, including the ability to create intricate patterns and structures. Digital knitting machines now enable users to personalize and rapidly develop their textile design through a specialized visual programming environment and various functional and common yarns. It employs an array of needles or hooks that goes up and down to catch the yarns based on an instruction file. Each yarn is fed to the machine from a cone, passing through a tensioning mechanism towards a yarn carrier. These yarn carriers move sideways as the needles grab the yarn to form new loops. With advances in electronic materials and digital fabrication technologies, we can tune textile physical and at the resolution of a fiber, a loop structure, or a particle coating. These textiles can be fully customized from the micro to macro-scale and computationally designed and fabricated to form a seamless, heterogeneous, and multi-functional skin. This additive manufacturing process of machine-knitting enables the team to engineer patterns, colors, texture, thickness, elasticity, breathability, conductivity in discrete loops through fiber structures or material choices with as little waste as possible. 

Tapis Magique is fabricated by knitting electrically-conducting yarns with other synthetic yarns, such as polyester, mink, thermoplastic, and luminous yarns, and fusing its multi-layer structures through thermoforming. The tapis design is composed of multi-layer knitted textiles. The top and bottom layers are orthogonal conductive line matrices knitted within a single operation using multi-material twisted yarns. The middle layer is a knitted piezoresistive textile, a pressure-sensitive layer that interfaces with the conductive matrices to create a sensing grid. The knitted piezoresistive textile is coated with conductive polymer called polypyrrole (PPy).  Since it is piezoresistive, the resistance of this fabric sensor changes in correlation to the applied force. Like how a force-sensing resistor (FSR) works, a higher pressure compresses the conducting molecules coated into the fabric. These molecules then form a network, allowing more current to flow and reducing the resistance around the contact area.

In Tapis Magique, the researchers carefully consider the textile design from the yarn choices, textures, mechanical properties, to the aesthetics. The dense geometrical patterns of the stars scattered around the brushstroke details in the tapis represent 1800 pressure-sensing pixels and are inspired by the galactic space. Parametric design transformed these patterns into a 3-D spatial illusion to illustrate the multi-dimensionality of the sensor data.  The furry textures from the synthetic mink yarns provide a soft tactility for physical feedback and give the tapis an intimate and comforting feel. The thermoplastic fibers were then steamed to melt the multi-layer knitted textiles into one rigid surface, giving it structural reinforcements. In addition, the outer-facing textile glows in the dark from the luminous yarns, bringing out the starry effects for night performance modes. 

The knitted conductive lines are connected to system hardware consisting of multiplexers, shift-registers, operational amplifiers, and a microcontroller that sequentially reads each pressure sensing pixel and sends it to a computer. These 60x30 pixels correspond to the pressure profile of the feet. They collectively generate continuous 3-D spatiotemporal sensor data correlated to the body posture, movements, and footwork. This data is then mapped into Musical Instrument Digital Interface (MIDI) streams to trigger and control discrete notes, continuous effects, and immersive soundscapes through science-inspired musical tools. MIDI is an industry-standard protocol that connects digital musical instruments, computers, tablets, and smartphones together and is used worldwide to create, produce, and perform music.

The team then designed several musical patches or pieces to invoke various emotions to inspire conversations between the choreographer and the instrument. Behind the scene, runs a digital modular synthesizer made of several patches, each for a different type of performance. The incoming stream of MIDI data is first fed into quantizer modules that align the notes to major, minor, and pentatonic scales, as well as mystic chords. The sounds are then generated by a collection of subtractive, additive, and granular synthesizers and are composed of discrete, melodic sounds with effects and continuously evolving soundscapes. "Venus Sunrise," one of the performance pieces demonstrated by the team, as shown in the video, presents a metaphorical celestial sound of the universe as the dancer is twirling around the stars, traveling through space and time. The team is now exploring other musical patches with different genres and ambiances.

Real-time audio system connected to Tapis Magique

Tapis Magique provides a canvas for dancers and sound artists to program, compose, perform, and even collaborate with their own choreomusical pieces. It could also create an auditory-gestural synesthetic experience that invites and encourages the audiences to move on the carpet, expressing themselves through dance and evokes the bi-directional, intimate relationship between two complex art forms of dance and music. 

Irmandy Wicaksono, an MIT PhD student from the Responsive Environments Group, MIT Media Lab led the project and was involved in the design, fabrication, and development of Tapis Magique with the advice of MIT Professor of Media Arts and Sciences Joe Paradiso. He collaborated with MIT PhD student and sound artist Don Derek Haddad and dancer Loni Landon to demonstrate this interactive e-textile carpet and produce several musical and performance pieces. 

About the University Technology Exposure Program 2022

Wevolver, in partnership with Mouser Electronics and Ansys, is excited to announce the launch of the University Technology Exposure Program 2022. The program aims to recognize and reward innovation from engineering students and researchers across the globe. Learn more about the program here.