Wevolver and<a href="https://www.wevolver.com/profile/syntiant" target="_blank">&nbsp;Syntiant</a> are creating a series that explore the work of innovators and the future of pervasive AI. Syntiant is developing ultra-low-power AI processors. Because they believe in the importance of innovation, Syntiant is engaging in these fireside chats with engineers and designers who are on the cutting edge of their field.In<a href="https://www.wevolver.com/tag/fireside-chat" target="_blank">&nbsp;previous conversations</a> we spoke for example with The Things Industries CEO Wienke Giezeman, Rev.com&rsquo;s Daniel Kokotov, Star Wars Animatronic Designer Gustav Hoegen, and Fashiontech Designer Anouk Wipprecht.&nbsp;<h2 dir="ltr">Bringing more women to voice </h2>While working on her PhD in speech language technologies, Bajorek, after pouring over the datasets that power much of the tools that listen, translate, and talk back, realized something was missing.&nbsp;For all of the technical innovation, she asked, who decides where that innovation goes?&nbsp;&ldquo;When we&#39;re looking at systemic change in these systems, we have to look at what our datasets look like, who is in the room, and who is deciding what this looks like. It&rsquo;s not necessarily technical limitations, it&#39;s about having women and people of color in the room.&rdquo;&nbsp;In 2018 Bajorek launched Women in Voice; a nonprofit that addresses the lack of support for women in voice and conversational AI fields. WiV now has 21 chapters in 15 countries. This past year, WiV launched its first Summit, developed programming with partners like<a href="https://medium.com/women-in-voice/women-in-voice-announces-google-assistant-as-first-corporate-sponsor-73fec986b9b1" target="_blank">&nbsp;Google Assistant</a>, Alexa Education &amp;<a href="https://womeninvoice.org/women-in-voice-amplifies-female-founders-with-amazon-alexa-startups/" target="_blank">&nbsp;Alexa Startups</a>, and<a href="https://womeninvoice.org/founder-investor-happy-hour/" target="_blank">&nbsp;Symbl.ai</a>, and they also debuted a formal<a href="https://womeninvoice.org/career/" target="_blank">&nbsp;Career Accelerator</a>.&nbsp;<h2 dir="ltr">Bias in AI and narrow datasets in enterprise </h2>Racial and gender bias is a well known but consistently pervasive problem in AI, especially with speech recognition technologies. There&#39;s difficulty recognizing female voices, those with accents, jargon and abbreviations, and low resource languages.Despite encouraging research with multilingual transformers and sentence embeddings, a lot of current benchmarking for improvement is narrow in scope - usually focused on early adopter datasets, which can be at least 70% a white, male, English-speaking demographic.&ldquo;From a business perspective as well, if my preferred demographic to work for is going to be LatinX Gen Z, which statistically would be really smart here in the United States, we need to ask the right questions. If I want to do something that performs well for this type of voice, how does the software have to change? How does my dataset need to change? How does the performance of whatever my hyperparameters look like in a multilingual context need to be tweaked? If I&#39;m looking for a multilingual Gen Z Latina dataset of speech samples, does that even exist? We need to recognize the right framework, and also the huge potential in knowing how to optimize and make systems more equitable,&rdquo; says Bajorek.&nbsp;To that end, Bajorek highlights the Mozilla Common Voice team as one example pushing the needle in the right direction, as they&rsquo;re looking at what percentage of gender are currently in different buckets for different languages.<h2 dir="ltr">The future of conversational AI </h2>Automatic Speech Recognition (ASR) has continued to mature - what it looks like today has pivoted many times since the days of speaking each word separately and training models where you have to speak for three hours to adequately capture your voice, to what we see now with deep learning.&nbsp;In the near future, Bajorek sees a world of EdTech that uses really robust language models in virtual reality.&nbsp;&ldquo;You can imagine a world using virtual reality or a similar immersive experience in a language setting that you&rsquo;re learning the language of. For example, if you&#39;re learning Spanish, you could go to a bar and speak Spanish and have an interaction with someone, and the automated speech recognition will pick it up. And if you get it right, you&#39;ll choose different things to order. With this type of EdTech, we&rsquo;re empowering people to experience different cultures and languages without even having to travel.&rdquo;Longer term, the goals are more challenging: major system improvements. &ldquo;I&#39;m really excited about ASR, Text to Speech (TTS), and Natural Language Processing (NLP). What I&rsquo;m even more excited about is how it&#39;s going to transform our world and our use cases for day-to-day things that we barely even pay attention to today. All voices, all sounds, and being able to parse them perfectly in real time with zero latency is a very, very hard problem. I hope to live long enough that we find that solution, but it will be a multi-pronged solution that we probably have not seen yet today.&rdquo;&nbsp;

#8 of our 'Voice of Innovation' fireside chat series: Robotics and AI reporter Rachel Gordon speaks to Dr. Bajorek on her mission to get more women into voice and conversational AI fields.

Fireside chat with CEO and founder of Women in Voice Dr. Joan Palmiter Bajorek

Ever noticed online ads following you that are eerily close to something you&rsquo;ve recently talked about with your friends and family? Microphones are embedded into nearly everything today, from our phones, watches, and televisions to voice assistants, and they are always listening to you. Computers are constantly using neural networks and AI to process your speech, in order to gain information about you. If you wanted to prevent this from happening, how could you go about it?Back in the day, as portrayed in the hit TV show &ldquo;The Americans,&rdquo; you would play music with the volume way up or turn on the water in the bathroom. But what if you didn&rsquo;t want to constantly scream over the music to communicate?&nbsp;<a href="https://www.engineering.columbia.edu/" target="_blank">Columbia Engineering</a> researchers have developed a&nbsp;<a href="https://arxiv.org/pdf/2112.07076.pdf" target="_blank">new system that generates whisper-quiet sounds that you can play in any room, in any situation, to block smart devices from spying on you</a>. And it&rsquo;s easy to implement on hardware like computers and smartphones, giving people agency over protecting the privacy of their voice.&ldquo;A key technical challenge to achieving this was to make it all work fast enough,&rdquo; said&nbsp;<a href="http://www.cs.columbia.edu/~vondrick/" target="_blank">Carl Vondrick</a>, assistant professor of&nbsp;<a href="https://www.engineering.columbia.edu/departments/computer-science" target="_blank">computer science</a>. &ldquo;Our algorithm, which manages to block a rogue microphone from correctly hearing your words 80% of the time, is the fastest and the most accurate on our testbed. It works even when we don&rsquo;t know anything about the rogue microphone, such as the location of it, or even the computer software running on it. It basically camouflages a person&rsquo;s voice over-the-air, hiding it from these listening systems, and without inconveniencing the conversation between people in the room.&rdquo;<h2>Staying Ahead of Conversations</h2>While the team&rsquo;s results in corrupting automatic speech recognition systems have been theoretically known to be possible in AI for a while, achieving them fast enough to use in practical applications has remained a major bottleneck. The problem has been that a sound that breaks a person&rsquo;s speech now&mdash;at this specific moment&mdash;isn&rsquo;t a sound that will break speech a second later. As people talk, their voices constantly change as they say different words and speak very fast. These alterations make it almost impossible for a machine to keep up with the fast pace of a person&rsquo;s speech.&ldquo;Our algorithm is able to keep up by predicting the characteristics of what a person will say next, giving it enough time to generate the right whisper to make,&rdquo; said Mia Chiquier, lead author of the study and a PhD student in Vondrick&rsquo;s lab. &ldquo;So far our method works for the majority of the English language vocabulary, and we plan to apply the algorithm on more languages, as well as eventually make the whisper sound completely imperceptible.&rdquo;<h2>Launching &ldquo;Predictive Attacks&rdquo;</h2>The researchers needed to design an algorithm that could break neural networks in real time, that could be generated continuously as speech is spoken, and applicable to the majority of vocabulary in a language. While earlier work had successfully tackled at least one of these three requirements, none have achieved all three. Chiquier&rsquo;s new algorithm uses what she calls &ldquo;predictive attacks&rdquo;&mdash;a signal that can disrupt any word that automatic speech recognition models are trained to transcribe. In addition, when attack sounds are played over-the-air, they need to be loud enough to disrupt any rogue &ldquo;listening-in&rdquo; microphone that could be far away. The attack sound needs to carry the same distance as the voice.The researchers&rsquo; approach achieves real-time performance by forecasting an attack on the future of the signal, or word, conditioned on two seconds of input speech. The team optimized the attack so it has a volume similar to normal background noise, allowing people in a room to converse naturally and without being successfully monitored by an automatic speech recognition system. The group successfully demonstrated that their method works inside real-world rooms with natural ambient noise and complex scene geometries.<h2>Ethical AI</h2>&ldquo;For many of us in the research community, ethical concerns of AI technology are an essential issue, but it seems to belong to a separate thought process. It is like we are so happy that we finally made a driving car but forgot to design a steering wheel and a brake,&rdquo; says <a href="https://www.grasp.upenn.edu/people/jianbo-shi/" target="_blank">Jianbo Shi</a>, professor of computer and information science at the University of Pennsylvania and a leading researcher in machine learning. &ldquo;As a community, we need to `consciously&#39; think about the human and societal impact of the AI technology we develop from the earliest research design phase. Mia Chiquier and Carl Vondrick&rsquo;s study poses the question: &lsquo;how to use AI to protect us against unintended AI usages?&rsquo; Their work makes many of us think in the following direction: ask not what ethical AI can do for us, but what we can do for ethical AI? Once we believe in this direction, ethical AI research is just as fun and creative.&rdquo;<hr>Chiquier will present her paper on April 25, 2022, at the&nbsp;<a href="https://iclr.cc/" id="isPasted" target="_blank">International Conference for Learning Representations</a>.&nbsp;

Columbia computer scientists have built an algorithm that blocks a rogue microphone from correctly hearing your words—in English so far—80% of the time

Concerned Your Smartphone Is Spying on You?

<html><head></head><body>Wevolver and<a href="https://www.wevolver.com/profile/syntiant" target="_blank">&nbsp;Syntiant</a> are creating a series that explore the work of innovators and the future of pervasive AI. Syntiant is developing ultra-low-power AI processors. Because they believe in the importance of innovation, Syntiant is engaging in these fireside chats with engineers and designers who are on the cutting edge of their field.This video is part of a series of fireside chats in which Wevolver and Syntiant partner to engage with global innovators. In <a href="https://www.wevolver.com/tag/fireside-chat" target="_blank">previous discussions</a> we spoke with The Things Industries CEO Wienke Giezeman, Star Wars Animatronic Designer Gustav Hoegen, and Fashiontech Designer Anouk Wipprecht.&nbsp;<h2 dir="ltr">Using humans and AI to translate and transcribe&nbsp;</h2>When Rev was in its infancy, it was mainly comprised of human freelancers sitting behind their screens, listening with intent to provide true to word accuracy. But in 2016, with the onset of the deep learning revolution, speech-to-text took off at high speeds. &nbsp;“The coupling of hiring intelligent people and working with a dataset that we had created over many years of doing human transcription translated into a very powerful automated speech recognition ecosystem. Now it's humans and machines working together.”&nbsp;Rev’s&nbsp;speech recognition engine is trained on a corpus of human-transcribed text. When a client needs human level quality, a human will translate, but with access to more robust datasets, they’ll actually start with a draft from the speech recognition model.&nbsp;The dataset is fed to the speech recognition engine for training, and the more data it gets from humans, the better the engine becomes. In turn, the “Revvers”, or human freelancers, can save an exorbitant amount of time by starting with a more accurate draft produced by the machine.“You can imagine, if we're doing transcripts for earnings calls, and all those need to be 100 percent accurate, they’re going to want human quality. On the other hand, if you’re doing something that necessitates a quick turnaround, where you might want something within minutes of having available audio, or even live – imagine a Zoom app that could show you a live transcript of our conversation - that will just be AI.”&nbsp;<h2 dir="ltr">Speech-to-text for law enforcement </h2>The ubiquity of the spoken word translated to text has started to feel like a simple extension of how we speak. Digital assistants are bubbling up on our shelves and mantels, telling us the weather or switching on a light. Legal officials transcribe interviews for depositions. And more recently, law enforcement has used speech to text for body camera equipment, a vertical Kokotov is exploring.Rev is working with Axon, a manufacturer of body cam equipment, the largest that leverage software tools for police departments to capture the audio from body cameras, and quickly leverage it for writing up reports for evidence collection. They record, it gets uploaded to Axon, it goes through transcription with&nbsp;Rev,&nbsp;and they don't have to worry about forgetting crucial details when filing reports.“Many of us have seen more body camera footage in the last year and a half than we've ever had before. It’s extremely important that we get the content of those conversations right because they affect a lot of people's lives, and they need to be scalable because officers are constantly in the field.”&nbsp;<h2 dir="ltr">Low resource natural language processing</h2>Speech recognition technologies notoriously favor the English language, and oftentimes fail to recognize different languages, accents, or speech patterns. Stepping even further into multilingual conversations - swinging back and forth between Portuguese and Arabic for example - no speech recognition engine would really handle that very well. However, new techniques, such as using multilingual transformers, are much more amenable to understanding these low resource languages. &nbsp;“I believe this will be one of the biggest challenges and themes over the next five to ten years, in taking speech recognition global, doing better with more languages, and improving these cross language situations. Our goal is to bring what we think is the best speech recognition to as many people as possible. In the service of this vision is understanding every human voice.”</body></html>

#7 of our 'Voice of Innovation' fireside chat series: Robotics and AI reporter Rachel Gordon speaks to Kokotov on his vision to understand every human voice. 

Fireside chat with Rev.com's Daniel Kokotov, VP of Engineering within Speech Technologies

The advancements in computer vision, pattern recognition and signal processing have made visual speech recognition possible for a variety of applications, such as human-computer interaction, speaker recognition, talking heads, sign language recognition and video surveillance. Visual speech recognition deals with several technological breakthroughs, like image processing, artificial intelligence, object detection, and statistical modeling only uses visual signals of the lip movement that produce speech without relying on the audio [1]. In the first generation of VSR, the research could not meet the high accuracy demands of the industry due to factors concerning the size of the audio-visual dataset with limited vocabulary and hand-crafted visual features that are not optimal for VSR model training. &nbsp;The introduction of deep learning and neural networks has made visual speech recognition models more robust than ever before while achieving higher accuracy in unstructured real-life situations. Most recent innovations allow the models to perform much better due to the availability of larger datasets and training models with a huge amount of data. A group of researchers affiliated with the Department of Computing at Imperial College London and Meta AI proposed a novel approach to add prediction-based auxiliary tasks to the VSR model and outperform state-of-the-art methods trained on the publicly available data [2].<h2 dir="ltr">Looking to Listen</h2>In the research paper, “Visual Speech Recognition for Multiple Languages in the Wild,” the team carried out work with the aim to reduce the word error rate (WER) through appropriate data augmentation and hyperparameter optimization of the existing architecture while working on the publicly available datasets that are much smaller in size. Through this, the team points out the fact that the optimization of the models can be brought in through carefully designing the models and not only increasing the size of the training data. Also, the proposed novel method for visual speech recognition not only evaluates English as a sole language but also on Mandarin, Spanish, Italian, French and Portuguese and achieves state-of-the-art performance on all the languages.&nbsp;“We optimize hyperparameters and improve the language model with the aim of squeezing extra performance out of the model,” the team explains. Then, “we introduce time-masking which is a temporal augmentation method and is commonly used in ASR models. Finally, we use a VSR model with auxiliary tasks where the model jointly performs visual speech recognition and prediction of audio and visual representations extracted from pre-trained VSR and ASR models.”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1647384806878-Visual+Speech+Recognition+Results+on+the+LSR2+Dataset.png" style="width: 800px;" class="fr-fic fr-dib" alt="Visual Speech Recognition Results on the LSR2 Dataset">Visual Speech Recognition Results on the LSR2 Dataset [Image Source: Research Paper]<div style='pointer-events: none; overflow: hidden; padding-bottom: 2px; z-index: auto; position: absolute; left: 0px; top: 374.688px; height: 39px; width: 766px; font-style: normal; font-variant: normal; font-weight: 400; font-stretch: 100%; font-size: 14px; font-family: "PT Serif", serif; text-align: center; text-transform: none; letter-spacing: normal; word-spacing: 0px; tab-size: 8;'> </div>Hyperparameter optimization aims to improve the performance of the model by adjusting the values of the parameters that are employed to control the training process and the model architecture. There are several hyperparameters taken into consideration including initial learning rate, learning rate decay parameter, number of layers, size of layers, dropout rate, and the loss function that was performed on the LRS2 dataset achieving reduction in WER on the validation set. The main hyperparameter that had a significant impact on the performance and which might not have been exhaustively optimized was the batch size. The team observed that reduction WER was achieved more when the batch size increased from 8 to 16. Due to GPU memory constraints, the team recognized that certain hyperparameters could not be increased further, but they would bring down the word error rate.&nbsp;<h2 dir="ltr">State-of-the-art performance</h2>The team decided to use all the open-source datasets available for public use, namely LRS2 (an audio-visual English dataset from BBC programs), LRS3 (an audio-visual English dataset taken from TED and TEDx talks), CMLR (a Mandarin dataset collected from Chinese news program), CMU-MOSEAS (contains multiple languages from YouTube videos), Multilingual TEDx (multilingual corpus taken from TEDx talks), and AVSpeech (dataset with over 4700 hours of video in multiple languages). To measure performance, the word error rate is a common performance metric in speech recognition, with an alternative of a character error rate that measures how close the predicted and target character sequences are.&nbsp;For the multilingual TEDxSpanish dataset, the approach results in a 5.6% absolute reduction in the WER and a further reduction of 4.2% can be achieved by using additional training data. While for the Italian corpus, the approach results in an absolute drop of 5.6% in the WER, the same as that for Spanish, but a further reduction of up to 8% for additional training data. “Results on the Multilingual TEDxFrench dataset are shown as a result of a 9.4 % absolute reduction in the WER and a further reduction of 7.6 % can be achieved by using additional training data.” [More details on the additional training data are available in the research article]<iframe width="640" height="360" src="https://www.youtube.com/embed/FIau-6JA9Po?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true"></iframe>The proposed novel method for visual speech recognition performs visual speech recognition and prediction of audio and visual representation on a publicly available dataset to outperform state-of-the-art methods. The research work was published on Cornell University’s research sharing platform, arXiv under <a href="https://arxiv.org/abs/2202.13084" target="_blank">open access terms</a>. The researchers have provided pre-trained networks and testing code on the authors’ <a href="https://github.com/mpc001/Visual_Speech_Recognition_for_Multiple_Languages" target="_blank">GitHub repository</a> for community access.<h2 dir="ltr">References</h2>[1] Ahmad B. A. Hassanat: Visual Speech Recognition. DOI arXiv: <a href="https://arxiv.org/abs/1409.1411" target="_blank">1409.1411 [cs.CV]</a>.[2] Pingchuan Ma, Stavros Petridis, Maja Pantic: Visual Speech Recognition for Multiple Languages in the Wild. DOI arXiv: <a href="https://arxiv.org/abs/2202.13084" target="_blank">2202.13084 [cs.CV]</a>.&nbsp;

Visual Speech Recognition on a Multilingual Dataset that outperforms the state-of-the-art methods [Image Source: Research Paper]

A novel approach to visual speech recognition that carefully designs the VSR model rather than increasing the size of the training data.


Research suggests improvements in visual speech recognition on a multilingual dataset

Researchers at Rutgers University-New Brunswick have&nbsp;<a href="https://dl.acm.org/doi/abs/10.1145/3447993.3483272" target="_blank">published “Face-Mic,” the first work</a> examining how voice command features on virtual reality headsets could lead to major privacy leakages, known as “eavesdropping attacks.”The research shows that hackers could use popular virtual reality (AR/VR) headsets with built in motion sensors to record subtle, speech-associated facial dynamics to steal sensitive information communicated via voice-command, including credit card data and passwords.Common AR/VR systems on the market include the popular brands Oculus Quest 2, HTC Vive Pro, and PlayStation VR.Led by&nbsp;<a href="http://www.winlab.rutgers.edu/~yychen/" target="_blank">Yingying “Jennifer” Chen, associate director of WINLAB and graduate director of Electrical and Computer Engineering at Rutgers University-New Brunswick</a>, the study will be presented at the annual International Conference on Mobile Computing and Networking in March. Other research collaborators include Nitesh Saxena of Texas A&amp;M University and Jian Liu at University of Tennessee at Knoxville.To demonstrate the existence of security vulnerabilities, Chen and her fellow WINLAB researchers developed an eavesdropping attack targeting AR/VR headsets, known as “Face-Mic.”“Face-Mic is the first work that infers private and sensitive information by leveraging the facial dynamics associated with live human speech while using face-mounted AR/VR devices,” said Chen. “Our research demonstrates that Face-Mic can derive the headset wearer’s sensitive information with four mainstream AR/VR headsets, including the most popular ones: Oculus Quest and HTC Vive Pro.”&nbsp;The researchers studied three types of vibrations captured by AR/VR headsets’ motion sensors, including speech-associated facial movements, bone-borne vibrations and airborne vibrations. Chen noted that bone-borne vibrations in particular are richly encoded with detailed gender, identity and speech information.“By analyzing the facial dynamics captured with the motion sensors, we found that both cardboard headsets and high-end headsets suffer security vulnerabilities, revealing a user’s sensitive speech and speaker information without permission,” Chen said.Although vendors usually have policies regarding utilizing the voice access function in headset microphones, Chen’s research found that built-in motion sensors, such as an accelerometer and gyroscope within a VR headset, do not require any permission to access. This security vulnerability can be exploited by malicious actors intent on committing eavesdropping attacks.Eavesdropping attackers can also derive simple speech content, including digits and words, to infer sensitive information, such as credit card numbers, Social Security numbers, phone numbers, PIN numbers, transactions, birth dates and passwords. Exposing such information could lead to identity theft, credit card fraud and confidential and health care information leakage.Chen said once a user has been identified by a hacker, an eavesdropping attack can lead to further exposure of user’s sensitive information and lifestyle, such as AR/VR travel histories, game/video preferences and shopping preferences. Such tracking compromises users’ privacy and can be lucrative for advertising companies.Oculus Quest, for example, supports voice dictation for entering web addresses, controlling the headset and exploring commercial products. Rutgers’ Face-Mic research shows that hackers may leverage these zero-permission sensors to capture sensitive information, leading to severe privacy leakages. &nbsp; &nbsp;Chen said she hopes these findings will raise awareness in the general public about AR/VR security vulnerabilities and encourage manufacturers to develop safer models.“Given our findings, manufacturers of VR headsets should consider additional security measures, such as adding ductile materials in the foam replacement cover and the headband, which may attenuate the speech-associated facial vibrations that would be captured by the built-in accelerometer/gyroscope,” she said.Chen and her WINLAB colleagues are now examining how facial vibration information can authenticate users and improve security, and how AR/VR headsets can capture a user’s breathing and heart rate to measure well-being and mood states unobtrusively. To learn more about ongoing studies at WINLAB,&nbsp;<a href="http://www.winlab.rutgers.edu/" target="_blank">click here</a>.

Rutgers researchers have published the first work examining how hackers could use popular virtual reality headsets to steal sensitive information communicated via voice-command, including credit card data and passwords.

Researchers Discover Security Vulnerabilities in Virtual Reality Headsets

The digital revolution is built on a foundation of invisible 1s and 0s called bits. As decades pass, and more and more of the world’s information and knowledge morph into streams of 1s and 0s, the notion that computers prefer to “speak” in binary numbers is rarely questioned. According to new research from Columbia Engineering, this could be about to change.A <a href="https://openreview.net/pdf?id=MyHwDabUHZm" rel="nofollow" target="_blank">new study</a> from <a href="https://me.columbia.edu/" rel="nofollow" target="_blank">Mechanical Engineering</a> Professor <a href="https://engineering.columbia.edu/faculty/hod-lipson" rel="nofollow" target="_blank">Hod Lipson</a> and his PhD student <a href="http://www.cs.columbia.edu/~bchen/" rel="nofollow" target="_blank">Boyuan Chen</a> proves that artificial intelligence systems might actually reach higher levels of performance if they are programmed with sound files of human language rather than with numerical data labels. The researchers discovered that in a side-by-side comparison, a neural network whose “training labels” consisted of sound files reached higher levels of performance in identifying objects in images, compared to another network that had been programmed in a more traditional manner, using simple binary inputs.“To understand why this finding is significant,” said Lipson, James and Sally Scapa Professor of Innovation and a member of Columbia’s <a href="https://datascience.columbia.edu/" rel="nofollow" target="_blank">Data Science Institute</a>, “It’s useful to understand how neural networks are usually programmed, and why using the sound of the human voice is a radical experiment.”When used to convey information, the language of binary numbers is compact and precise. In contrast, spoken human language is more tonal and analog, and, when captured in a digital file, non-binary. Because numbers are such an efficient way to digitize data, programmers rarely deviate from a numbers-driven process when they develop a neural network.Lipson, a highly regarded roboticist, and Chen, a former concert pianist, had a hunch that neural networks might not be reaching their full potential. They speculated that neural networks might learn faster and better if the systems were “trained” to recognize animals, for instance, by using the power of one of the world’s most highly evolved sounds—the human voice uttering specific words.One of the more common exercises AI researchers use to test out the merits of a new machine learning technique is to train a neural network to recognize specific objects and animals in a collection of different photographs. To check their hypothesis, Chen, Lipson and two students, Yu Li and Sunand Raghupathi, set up a controlled experiment. They created two new neural networks with the goal of training both of them to recognize 10 different types of objects in a collection of 50,000 photographs known as “training images.”<iframe src="https://www.youtube.com/embed/Iq2YjHCAPRQ?&amp;wmode=opaque&amp;rel=0" frameborder="0" allowfullscreen="" class="fr-draggable" style="width: 789px; height: 443px;"></iframe>Video illustrating how a deep neural network that is taught to speak out the answer demonstrates higher performances of learning robust and efficient features. Illustration And Video Credit: Creative Machines Lab/Columbia EngineeringOne AI system was trained the traditional way, by uploading a giant data table containing thousands of rows, each row corresponding to a single training photo. The first column was an image file containing a photo of a particular object or animal; the next 10 columns corresponded to 10 possible object types: cats, dogs, airplanes, etc. A “1” in any column indicates the correct answer, and nine 0s indicate the incorrect answers.The team set up the experimental neural network in a radically novel way. They fed it a data table whose rows contained a photograph of an animal or object, and the second column contained an audio file of a recorded human voice actually voicing the word for the depicted animal or object out loud. There were no 1s and 0s.Once both neural networks were ready, Chen, Li, and Raghupathi trained both AI systems for a total of 15 hours and then compared their respective performance. When presented with an image, the original network spat out the answer as a series of ten 1s and 0s—just as it was trained to do. The experimental neural network, however, produced a clearly discernible voice trying to “say” what the object in the image was. Initially the sound was just a garble. Sometimes it was a confusion of multiple categories, like “cog” for cat and dog. Eventually, the voice was mostly correct, albeit with an eerie alien tone (see example on <a href="https://www.creativemachineslab.com/label-representation.html" rel="nofollow" target="_blank">website</a>).At first, the researchers were somewhat surprised to discover that their hunch had been correct—there was no apparent advantage to 1s and 0s. Both the control neural network and the experimental one performed equally well, correctly identifying the animal or object depicted in a photograph about 92% of the time. To double-check their results, the researchers ran the experiment again and got the same outcome.What they discovered next, however, was even more surprising. To further explore the limits of using sound as a training tool, the researchers set up another side-by-side comparison, this time using far fewer photographs during the training process. While the first round of training involved feeding both neural networks data tables containing 50,000 training images, both systems in the second experiment were fed far fewer training photographs, just 2,500 apiece.It is well known in AI research that most neural networks perform poorly when training data is sparse, and in this experiment, the traditional, numerically trained network was no exception. Its ability to identify individual animals that appeared in the photographs plummeted to about 35% accuracy. In contrast, although the experimental neural network was also trained with the same number of photographs, its performance did twice as well, dropping only to 70% accuracy.Intrigued, Lipson and his students decided to test their voice-driven training method on another classic AI image recognition challenge, that of image ambiguity.This time they set up yet another side-by-side comparison but raised the game a notch by using more difficult photographs that were harder for an AI system to “understand.” For example, one training photo depicted a slightly corrupted image of a dog, or a cat with odd colors. When they compared results, even with more challenging photographs, the voice-trained neural network was still correct about 50% of the time, outperforming the numerically-trained network that floundered, achieving only 20% accuracy.Ironically, the fact their results went directly against the status quo became challenging when the researchers first tried to share their findings with their colleagues in computer science. “Our findings run directly counter to how many experts have been trained to think about computers and numbers; it’s a common assumption that binary inputs are a more efficient way to convey information to a machine than audio streams of similar information ‘richness,’” explained Boyuan Chen, the lead researcher on the study. “In fact, when we submitted this research to a big AI conference, one anonymous reviewer rejected our paper simply because they felt our results were just ‘too surprising and un-intuitive.’”When considered in the broader context of information theory however, Lipson and Chen’s hypothesis actually supports a much older, landmark hypothesis first proposed by the legendary Claude Shannon, the father of information theory. According to Shannon’s theory, the most effective communication “signals” are characterized by an optimal number of bits, paired with an optimal amount of useful information, or “surprise.”“If you think about the fact that human language has been going through an optimization process for tens of thousands of years, then it makes perfect sense, that our spoken words have found a good balance between noise and signal;” Lipson observed. “Therefore, when viewed through the lens of Shannon Entropy, it makes sense that a neural network trained with human language would outperform a neural network trained by simple 1s and 0s.”The study, to be presented at the <a href="https://iclr.cc/" rel="nofollow" target="_blank">International Conference on Learning Representations</a> conference on May 3, 2021, is part of a broader effort at Lipson’s Columbia <a href="https://www.creativemachineslab.com/" rel="nofollow" target="_blank">Creative Machines Lab</a> to create robots that can understand the world around them by interacting with other machines and humans, rather than by being programed directly with carefully preprocessed data.“We should think about using novel and better ways to train AI systems instead of collecting larger datasets,” said Chen. “If we rethink how we present training data to the machine, we could do a better job as teachers.”One of the more refreshing results of computer science research on artificial intelligence has been an unexpected side effect: by probing how machines learn, sometimes researchers stumble upon fresh insight into the grand challenges of other, well-established fields.“One of the biggest mysteries of human evolution is how our ancestors acquired language, and how children learn to speak so effortlessly,” Lipson said. “If human toddlers learn best with repetitive spoken instruction, then perhaps AI systems can, too.”

A deep neural network that is taught to speak out the answer demonstrates higher performances of learning robust and efficient features. This study opens up new research questions on the role of label representations for object recognition. Illustration And Video Credit: Creative Machines Lab/Columbia Engineering

Columbia engineers demonstrate that AI systems might reach higher performance if programmed with sound files of human language rather than with binary data labels

Deep Learning Networks Prefer the Human Voice - Just Like Us

EPFL’s Laboratory for Experimental Museology (eM+), led by Sarah Kenderdine, has received support from Engagement Migros to produce muse, a pioneering audience evaluation app, in collaboration with over 20 Swiss museums. The tool provides visitors with an engaging way to communicate their experiences, while helping museums to consider this feedback when developing exhibits and other attractions. Encouraging audiences to participate in evaluations in large enough numbers to provide useful data is notoriously difficult, and for good reason: respondents may find the process time-consuming or boring. But what if you were invited to communicate your experience at a museum exhibit by making drawings, leaving voice recordings, interacting with animated graphics, or taking photos?&nbsp;This is the aim of muse: a design-led audience evaluation app that helps museums get to know their visitors better. In addition to geographic and demographic information, the tablet-based tool uses some 30 interactive elements to collect qualitative, subjective data about museum visitors’ experiences.<h2>Capturing experiences</h2>Sarah Kenderdine leads the eM+ lab in EPFL’s College of Humanities. She explains that in Switzerland, as in many parts of the world, museums usually collect only basic demographic data from their visitors, such as number of entries and origin. The goal of muse is therefore to give museums access to data analytics tools that can help them focus on many visitors as individuals, and to use the quality of visitor engagement as a method to define, measure, and program for improved museum experiences. “It is the social, political, emotional, educational, and creative elements, in addition to how deeply people are participating in a museum’s programs and exhibitions, that should be the basis for measuring value,” Kenderdine says. Unlike a traditional audience evaluation or survey, visitors use muse to record their experiences as they pass through an exhibit, rather than afterward. “Muse is always meant to be deployed inside the space of experience, and not afterward, so people can reflect in real time on what their reactions to it really are. Simultaneously, museum staff can see aggregated results on a real-time dashboard,” says Kenderdine, who has been developing muse since 2012. She notes a recent example of the success of the muse model, when a previous version of the app was deployed for the Winter at Tantora arts festival in AlUla, Saudi Arabia. It engaged 6,000 respondents across five surveys with 120 data points, resulting in 1,200 pages of reports. “This data provided a phenomenal insight into customer loyalty, future planning, and creative responses to the cultural offering at this new world heritage destination.” With the support of Engagement Migros, and the participation of more than 20 museums across Switzerland, Kenderdine’s goal now is to bring muse to the next level, including more sophisticated analysis driving better public outcomes for museums and their audiences. The project will be officially launched on September 17th, 2020 at EPFL’s ArtLab.<h2>Looking ahead</h2>At first, the muse team will partner with eight ‘core’ museums and institutions for a three-year period, including: the Château de Morges (Morges), the Haus der Elektronischen Künste (Basel), the International Red Cross Museum (Geneva), the Musée d'Éthnographie (Geneva), the Museum für Gestaltung (Zürich), the Rietberg Museum (Zürich), and the Olympic Museum (Lausanne). Each institution will be supported by an administration portal for creating questions and surveys, an app for collecting data, and a dashboard for visualizing the data in real time. A further 16 museums will be invited to join the project through an open call starting in late 2020. Throughout the four-year funding period, the muse team will develop convincing case studies for the Swiss museum landscape. It will also explore options for commercializing the app and its licensing models, and for expanding to include all cultural offerings across Europe and beyond – all in close collaboration with its partners. “With its user-centric approach, muse supports museums in their efforts to continue being active places in society,” says Johanna Muther, Project Manager at Engagement Migros. Kenderdine adds that going forward, muse will also help museums to increasingly embrace their audiences with renewed attention as they adapt to the challenges of the coronavirus pandemic. “Visitors are active participants, and not just consumers, in helping to support museums as polyvocal places. Muse gives an amplifier to those voices,” she says.<hr>About muse Muse is the voice of the visitor. In addition to quantitative data, such as geographic and demographic information, the tool can also be used to collect qualitative findings on the direct impact of an exhibition visit. The focus of the app is on creating playful interactions, and on being easy to use – for museums and visitors alike. Museums can use the findings to take greater account of the perspective of visitors when curating and developing their exhibits, thus helping them to remain firmly established as active places in society, both now and in the future.About Engagement Migros The Engagement Migros development fund enables pioneering projects in the midst of social change, which break new ground and test solutions aimed at the future. This approach is oriented towards making a difference and combines financial support with coaching services in its Pioneerlab. Engagement Migros is made possible by the companies of the Migros Group thanks to annual funding of approximately CHF 10 million; it has been complementing the Migros Culture Percentage since 2012.

The tool provides visitors with an engaging way to communicate their experiences, while helping museums to consider this feedback when developing exhibits and other attractions.

Data-driven app brings 'voice of the visitor' to museum experiences

<h2 class="lw lx el aq ap cc ly lz ma mb mc md me mf mg mh mi" data-selectable-paragraph="" id="31c7">A very brief history of talking to computers</h2>Possibly the first voice activated machine was a 2$ toy called Radio Rex produced in 1920s. It was a spring activated dog that popped out of a little kennel whenever it detected a sound in the 500 Hz range. It wasn&rsquo;t Siri, Cortana or Alexa, but quite impressive for it&rsquo;s time. Later in the 80s, IBM introduced a dictation taking computer which didn&rsquo;t really catch on. It wasn&rsquo;t until recently with the introduction of Apple&rsquo;s Siri that the fiction of Star trek or HAL started approaching reality. A little while before Siri, in the 90s there existed voice dialers which allowed people to dial unto ten different phone numbers by voice. By 2000s, Interactive Voice Response (IVR) systems became fairly commonplace. Apple unveiled Siri in 2011 on the iPhone 4S, but it was the introduction of Amazon&rsquo;s Alexa in 2014 that brought voice interfaces to the forefront. Every major tech company followed suite later.<hr class="mx db hl my mz hy na nb nc nd ne"><section class="jt ju jv jw jx"><div class="bt jy af du dq dr"><blockquote class="nf"><div class="ng nh ht aq kb b ni nj nk nl nm nn mw" data-selectable-paragraph="" id="2bd1">Hey! But I already use my fingers to swipe on Tinder and scroll on Instagram. Why should I care about voice?</div></blockquote><h2 class="lw lx el aq ap cc ly nq ma nr mc ns me nt mg nu mi" data-selectable-paragraph=""> </h2><h2 class="lw lx el aq ap cc ly nq ma nr mc ns me nt mg nu mi" data-selectable-paragraph="" id="31a4">Voice has some clear advantages</h2><ul><li class="mj mk el aq ml b mm mn mo mp mq mr ms mt mu mv mw nv nw nx" data-selectable-paragraph="" id="e914">In many cases, it is faster to say something than perform complex actions to complete a task. It is easier to say &lsquo;set an alarm for 7 AM&rsquo; than finding your alarm app, click on the set alarm button, select 7, then select AM and finally click on the save button.</li><li class="mj mk el aq ml b mm ny mo nz mq oa ms ob mu oc mw nv nw nx" data-selectable-paragraph="" id="0134">In cases where users can&rsquo;t use their hands like driving or cooking, speaking is much safer and practical.</li><li class="mj mk el aq ml b mm ny mo nz mq oa ms ob mu oc mw nv nw nx" data-selectable-paragraph="" id="97fd">There is little or no learning required to interact with machines by voice. Talking is intuitive. Humans already know how to talk. Communication by speech is ubiquitous.</li><li class="mj mk el aq ml b mm ny mo nz mq oa ms ob mu oc mw nv nw nx" data-selectable-paragraph="" id="75f2">Voice can convey much more information than text alone. Speech consists of tone of voice, intonation, rate of speech. Embedded in voice is information about emotions. A computer can respond much better to speech than text alone.</li></ul><h2 class="lw lx el aq ap cc ly lz ma mb mc md me mf mg mh mi" data-selectable-paragraph="" id="d936">That said, it&rsquo;s not always appropriate to use voice as the only medium of communication with computers</h2><ul><li class="mj mk el aq ml b mm mn mo mp mq mr ms mt mu mv mw nv nw nx" data-selectable-paragraph="" id="c1d9">It is downright&nbsp;irritating to operate computers with voice in public spaces. Imagine everyone in an office environment interacting with their computers through voice. It would be chaotic.</li><li class="mj mk el aq ml b mm ny mo nz mq oa ms ob mu oc mw nv nw nx" data-selectable-paragraph="" id="11c0">Sometimes it is preferable to text. Right now, it is&nbsp;difficult to undo what you&rsquo;ve said. It&rsquo;s easier to delete text when communicating something that requires thought and introspection.</li><li class="mj mk el aq ml b mm ny mo nz mq oa ms ob mu oc mw nv nw nx" data-selectable-paragraph="" id="b25a">Privacy is a major concern. You wouldn&rsquo;t want to speak your health issues to a computer in a crowded environment. It would be a violation of privacy if the computer reads out your messages without confirming.</li></ul>The most important thing is to start thinking of voice interface as just another mode of interacting with machines. Talking to your computer in many cases will be cumbersome or downright inappropriate. Saying &lsquo;scroll up&rsquo; every time you want to scroll your facebook feed doesn&rsquo;t sound very intuitive but &lsquo;share my trip photos with my girlfriend&rsquo; is.</div></section>

Possibly the first voice activated machine was a 2$ toy called Radio Rex produced in 1920s. It was a spring activated dog that popped out of a little kennel whenever it detected a sound in the 500 Hz range.

Why should you care about voice interfaces?

MIT computer scientists have developed a system that learns to identify objects within an image, based on a spoken description of the image. Given an image and an audio caption, the model will highlight in real-time the relevant regions of the image being described.Unlike current speech-recognition technologies, the model doesn&rsquo;t require manual transcriptions and annotations of the examples it&rsquo;s trained on. Instead, it learns words directly from recorded speech clips and objects in raw images, and associates them with one another.The model can currently recognize only several hundred different words and object types. But the researchers hope that one day their combined speech-object recognition technique could save countless hours of manual labor and open new doors in speech and image recognition.Speech-recognition systems such as Siri, for instance, require transcriptions of many thousands of hours of speech recordings. Using these data, the systems learn to map speech signals with specific words. Such an approach becomes especially problematic when, say, new terms enter our lexicon, and the systems must be retrained.&ldquo;We wanted to do speech recognition in a way that&rsquo;s more natural, leveraging additional signals and information that humans have the benefit of using, but that machine learning algorithms don&rsquo;t typically have access to. We got the idea of training a model in a manner similar to walking a child through the world and narrating what you&rsquo;re seeing,&rdquo; says David Harwath, a researcher in the Computer Science and Artificial Intelligence Laboratory (CSAIL) and the Spoken Language Systems Group. Harwath co-authored a paper describing the model that was presented at the recent European Conference on Computer Vision.In the paper, the researchers demonstrate their model on an image of a young girl with blonde hair and blue eyes, wearing a blue dress, with a white lighthouse with a red roof in the background. The model learned to associate which pixels in the image corresponded with the words &ldquo;girl,&rdquo; &ldquo;blonde hair,&rdquo; &ldquo;blue eyes,&rdquo; &ldquo;blue dress,&rdquo; &ldquo;white light house,&rdquo; and &ldquo;red roof.&rdquo; When an audio caption was narrated, the model then highlighted each of those objects in the image as they were described.One promising application is learning translations between different languages, without need of a bilingual annotator. Of the estimated 7,000 languages spoken worldwide, only 100 or so have enough transcription data for speech recognition. Consider, however, a situation where two different-language speakers describe the same image. If the model learns speech signals from language A that correspond to objects in the image, and learns the signals in language B that correspond to those same objects, it could assume those two signals &mdash; and matching words &mdash; are translations of one another.&ldquo;There&rsquo;s potential there for a Babel Fish-type of mechanism,&rdquo; Harwath says, referring to the fictitious living earpiece in the &ldquo;Hitchhiker&rsquo;s Guide to the Galaxy&rdquo; novels that translates different languages to the wearer.The CSAIL co-authors are: graduate student Adria Recasens; visiting student Didac Suris; former researcher Galen Chuang; Antonio Torralba, a professor of electrical engineering and computer science who also heads the MIT-IBM Watson AI Lab; and Senior Research Scientist James Glass, who leads the Spoken Language Systems Group at CSAIL.Audio-visual associationsThis work expands on an earlier model developed by Harwath, Glass, and Torralba that correlates speech with groups of thematically related images. In the earlier research, they put images of scenes from a classification&nbsp;<a href="http://places.csail.mit.edu/" target="_blank">database</a> on the crowdsourcing Mechanical Turk platform. They then had people describe the images as if they were narrating to a child, for about 10 seconds. They compiled more than 200,000 pairs of images and audio captions, in hundreds of different categories, such as beaches, shopping malls, city streets, and bedrooms.They then designed a model consisting of two separate convolutional neural networks (CNNs). One processes images, and one processes spectrograms, a visual representation of audio signals as they vary over time. The highest layer of the model computes outputs of the two networks and maps the speech patterns with image data.The researchers would, for instance, feed the model caption A and image A, which is correct. Then, they would feed it a random caption B with image A, which is an incorrect pairing. After comparing thousands of wrong captions with image A, the model learns the speech signals corresponding with image A, and associates those signals with words in the captions. As described in a 2016&nbsp;<a href="http://news.mit.edu/2016/recorded-speech-images-automated-speech-recognition-1206" target="_blank">study</a>, the model learned, for instance, to pick out the signal corresponding to the word &ldquo;water,&rdquo; and to retrieve images with bodies of water.&ldquo;But it didn&rsquo;t provide a way to say, &lsquo;This is exact point in time that somebody said a specific word that refers to that specific patch of pixels,&rsquo;&rdquo; Harwath says.Making a matchmapIn the new paper, the researchers modified the model to associate specific words with specific patches of pixels. The researchers trained the model on the same database, but with a new total of 400,000 image-captions pairs. They held out 1,000 random pairs for testing.In training, the model is similarly given correct and incorrect images and captions. But this time, the image-analyzing CNN divides the image into a grid of cells consisting of patches of pixels. The audio-analyzing CNN divides the spectrogram into segments of, say, one second to capture a word or two.With the correct image and caption pair, the model matches the first cell of the grid to the first segment of audio, then matches that same cell with the second segment of audio, and so on, all the way through each grid cell and across all time segments. For each cell and audio segment, it provides a similarity score, depending on how closely the signal corresponds to the object.The challenge is that, during training, the model doesn&rsquo;t have access to any true alignment information between the speech and the image. &ldquo;The biggest contribution of the paper,&rdquo; Harwath says, &ldquo;is demonstrating that these cross-modal [audio and visual] alignments can be inferred automatically by simply teaching the network which images and captions belong together and which pairs don&rsquo;t.&rdquo;The authors dub this automatic-learning association between a spoken caption&rsquo;s waveform with the image pixels a &ldquo;matchmap.&rdquo; After training on thousands of image-caption pairs, the network narrows down those alignments to specific words representing specific objects in that matchmap.&ldquo;It&rsquo;s kind of like the Big Bang, where matter was really dispersed, but then coalesced into planets and stars,&rdquo; Harwath says. &ldquo;Predictions start dispersed everywhere but, as you go through training, they converge into an alignment that represents meaningful semantic groundings between spoken words and visual objects.&rdquo;&ldquo;It is exciting to see that neural methods are now also able to associate image elements with audio segments, without requiring text as an intermediary,&rdquo; says Florian Metze, an associate research professor at the Language Technologies Institute at Carnegie Mellon University. &ldquo;This is not human-like learning; it&rsquo;s based entirely on correlations, without any feedback, but it might help us understand how shared representations might be formed from audio and visual cues. ... [M]achine [language] translation is an application, but it could also be used in documentation of endangered languages (if the data requirements can be brought down). One could also think about speech recognition for non-mainstream use cases, such as people with disabilities and children.&rdquo;

MIT computer scientists have developed a system that learns to identify objects within an image, based on a spoken description of the image.  Image: Christine Daniloff