human-robot interaction

Smart vending machines powered by high-performance single-board computers can significantly improve customer experience and boost sales with features such as real-time stock monitoring, digital payments, voice/gesture control, and more.


Building Internet-Connected Smart Vending Machines With Powerful Single Board Computers

Article #8 of Innovations in Electric and Autonomous Vehicles Series: Advancements in voice recognition, touch displays, and other HMI technologies of next-gen vehicles are all set to transform the way we interact with our rides. Car dashboards will be safer, non-distracting, and highly intuitive.

Next-Generation Vehicles Need Safe Human Machine Interfaces

<p id="isPasted">On the 24<sup>th</sup> May 2022 in Uvalde, Texas, a gunman killed 19 children and two adults at an elementary school. This is part of the events <a href="https://www.texastribune.org/2022/05/27/uvalde-texas-school-shooting-timeline/" target="_blank">timeline</a>, focused on the police functioning during this event:&nbsp;</p><ul><li><u>11:28 a.m.</u> - Shooter arrives at school</li><li><u>11:30 a.m.</u> - A teacher calls 911</li><li><u>11:35 a.m.</u> - &ldquo;&hellip;Pete Arredondo, the chief of the school district&rsquo;s police department, arrives at the scene around this time. He does not use his radio. <span dir="RTL">)</span><strong>Arredondo wanted both hands on his gun if he encountered the shooter and believed the radio would have slowed him down</strong>, his attorney said.&rdquo;<span dir="RTL">(</span></li><li><u>12:03 p.m.</u> - A student calls 911 from room 112 for a minute and 23 seconds and identifies herself in a whisper. Meanwhile, as many as 19 officers <strong>are positioned in the school&rsquo;s hallway</strong>.</li><li><u>12:46 p.m.</u> - Breach command is given. Arredondo gives his approval to enter the classroom. &ldquo;If y&rsquo;all are ready to do it, you do it,&rdquo; he says, according to a transcript of police body-camera footage.</li><li><u>12:50 p.m.</u> - Forces kill gunman&nbsp;</li></ul><p><strong>It took one hour and twenty minutes from the first 911 call until the gunman was neutralized!</strong></p><p><span class="fr-video fr-deletable fr-fvc fr-dvb fr-draggable" contenteditable="false" draggable="true"><iframe src="https://www.youtube.com/embed/1NqlAcD-OqE?&ab_channel=WFAA&wmode=opaque&rel=0&enablejsapi=1&origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" style="width: 720px; height: 480px;" data-gtm-yt-inspected-11="true" id="559664948" title="Uvalde shooting: Police bodycam footage shows moments when law enforcement entered Robb Elementary"></iframe></span></p><h2><strong>The problem&nbsp;</strong></h2><p>Emergency responders all over the world need an immediate understanding of their surroundings. This critical information makes the difference between lives saved and unfortunately, horrible disasters.<br>All forces involved share the same desire for precise information ASAP in a clear and visual way that will<span dir="RTL">&nbsp;</span>enable them to make smart decisions in real-time.</p><p>From investigating the horrible attack in Texas, we can specify a few points that were handled poorly:</p><h3><strong>Non-unified data management in control rooms led to inefficient management of the forces.&nbsp;</strong></h3><p>Control rooms and dispatches are constructed of multiple sub-systems and&nbsp;multiple&nbsp;screens. These subsystems rely<span dir="RTL">&nbsp;</span>on:</p><ul><li>Extensive operator training</li><li>The misassumption that operators are not suffering from cognitive load&nbsp;</li><li>The ability to communicate with each other simultaneously</li></ul><p>These factors can cause fragmented data, which can easily lead to the loss of important information and in some cases can cost human lives.<span dir="RTL">&nbsp;</span></p><p><a href="https://www.statista.com/statistics/1183457/iot-connected-devices-worldwide/" target="_blank">In a six-year period, the amount of IoT devices in the world is expected to double</a>, which will make the cognitive load even more challenging and almost impossible to process without data fusion systems.</p><p><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjU4MTc1NjY4NTQ3LXJpb19jb250cm9sX3Jvb21fZ2VvcmdlX3NvYXJlcy5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 720px; height: 480px;" class="fr-fic fr-dib"></p><h3><strong>Unclear visual picture and lack of critical elements geolocation in the school.</strong></h3><p>Emergency responders that arrived at the scene were missing relevant visual information and understanding of the current status at the elementary school - in particular, regarding the attacker&rsquo;s and civilians&#39; locations.&nbsp;</p><h3><strong>Operators&rsquo; limitation to free his hands for radio &amp; tech devices (tablets, etc.).</strong></h3><p>A major fact is that Chief Arredondo was only focused on his gun, claiming to be unable to free his hands for the radio, even at the price of being cut off from communication with others.</p><h3><strong>The flow of information from control rooms to the field.</strong></h3><p>It is clear from this timeline that the information <strong>was</strong> <strong>available</strong> to the control room streaming constantly to the 911 dispatch via oral information.&nbsp;</p><p>Moreover, the ability to exploit accessible data such as smartphone devices of the students or even the gunman was possible, yet, unfortunately, unavailable to the emergency responders as a result of poor distribution to the field.&nbsp;</p><h2>The Solution</h2><p>The ultimate solution for this type of scenario will involve taking all the data that was streamed in real-time to the 911 dispatch, combined with any other accessible data such as the students&rsquo; smartphone information, static cameras deployed in the school and other friendly forces&rsquo; locations that were on-site in order to bring to any relevant emergency responder a <strong>comprehensive image</strong> of what&#39;s going on in a way that will <strong>improve efficiency</strong> and drastically increase situational awareness.</p><p>Oversight develops <em><strong>Absorber</strong></em><strong>&nbsp;- A visual, 3D, Command-and-Control (C&amp;C) solution based on augmented and virtual reality tools</strong> enabling information sharing and management on top of Geographic Information System (GIS) between many users simultaneously. The system does not require a physical location to run in and can be managed remotely.</p><h3><strong>Complete data fusion - one platform for all information stitched together in real-time</strong></h3><p>This product has a novel unified and distributed data management system.<strong>&nbsp;</strong>All devices that are map-locatable and can produce valuable data to the system, (hereinafter, data collection agents) - <u>Drones and other unmanned aerial vehicles, robots, IoT sensors, smartphones, XR headsets (extended reality), PTZ cameras or other stationery cameras&nbsp;</u>and other sources that produce localization information or other visual information.</p><h3><strong>XR-based display overlaid on reality image</strong></h3><p>Another breakthrough is connecting information to the actual world by displaying data collection agents in real-time on the user reality image. Seeing things without any physical limitations will open a whole new world for managing data.</p><h3><strong>Easy to operate</strong></h3><p>Controlling information on such a system is done using the operator&rsquo;s <strong>hands, head and eye gestures</strong> in a common and natural way. This will reduce the number of operators and the level of training required. Logics can be in-built in order to eliminate the operator needing to deal with other devices. For example, he will be able to talk to a partner only by focusing on them with his eyes.</p><h3><strong>Decentralized multiple-player system</strong></h3><p>This method for controlling agents is especially suitable for a rear control room operation while the user in the field can also participate by managing data. All data can be shared on one system, making the system mobile and not so dependent on hardware-based devices.</p><h3><strong>Bottom line</strong></h3><p><em>Absorber&nbsp;</em>is a breakthrough all-in-one system that can independently operate throughout a developing situation and communicate information in real-time. <strong>It enables an immediate flow of information and smart decision making</strong></p><h3><span class="fr-video fr-deletable fr-fvc fr-dvb fr-draggable" contenteditable="false" draggable="true"><iframe src="https://www.youtube.com/embed/7WTHrbVi7r0?&t=7s&ab_channel=Oversight&wmode=opaque&rel=0&enablejsapi=1&origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" style="width: 727px; height: 454px;" data-gtm-yt-inspected-11="true" id="283478871" title="Absorber by Oversight"></iframe></span></h3><h2><em>Absorber&nbsp;</em>and Its Components</h2><p>The different data collection agents <em>Absorber&nbsp;</em>will support have the common ability to continuously convey their current location. This is a basic requirement of the system to allow accurate determining of each agent&rsquo;s location, their spatial relation to others and displaying them on a hologram-based 3D digital table-top view board using XR headsets or smartphones.&nbsp;</p><p><em>Absorber&nbsp;</em>will serve as an add-on or replacement for agent management and can enhance operators&rsquo; situational awareness in real-time.&nbsp;</p><h3>Collectors&nbsp;</h3><p><em>Absorber&nbsp;</em>consists of &ldquo;collectors&rdquo; - software utilities for connecting devices to the system. Collectors convert the protocol of the connected device into <em>Absorber&rsquo;s&nbsp;</em>proprietary protocol - that&rsquo;s what makes the system unified, since each device can have different operating systems, SDK, units of measurement or even global datum reference.</p><p>XR devices serve as the system display and also act as data collection agents since most XR devices can report their relative location, produce a real-time map and report device status. By connecting them to a GPS device they can also be a map-locatable device.</p><p>The very use of immersive technology and holograms to represent information will make it possible to build a multi-user system that can share information even from remote locations.</p><h3>Core &nbsp;</h3><p><em>Absorber&rsquo;s&nbsp;</em>core components are media server for managing videos from multiple devices, map server in order to store relevant geocell information for a comprehensive on-premises solution and an interactive simulation engine in order to create events and scenarios for an offline training solution. Data fusion logics for managing multiple agents will be built on top of that collaborative information.</p><h3>Users and SDK</h3><p>On the other side, we have the end-users who can be control room operators or even emergency responders. The system will also deliver SDK to let other users build solutions on top of this collaborative information.</p><p><span class="fr-video fr-deletable fr-fvc fr-dvb fr-draggable" contenteditable="false" draggable="true"><iframe src="https://player.vimeo.com/video/731725270?autoplay=1&loop=1&autopause=0&background=1&muted=1&quality=1080p" width="560" height="315" frameborder="0" allowfullscreen="" class="fr-draggable" data-gtm-yt-inspected-11="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true"></iframe></span><br></p>

Using AR/VR technology, Oversight gives emergency responders the ability to literally SEE THROUGH WALLS and strengthens the connectivity between control rooms and field operators.

Enhancing Emergency Responders Situational Awareness Using XR Tools

<h2 id="isPasted">How industrial robots can help address the growing labor shortage in manufacturing</h2><p>Many manufacturers are struggling to find skilled labor for their manufacturing plants? According to the Society of Manufacturing Engineers, 89% of manufacturers are having difficulty finding skilled workers in the United States today. Moreover, a 2018 study by&nbsp;<a href="https://www.themanufacturinginstitute.org/press-releases/manufacturing-industry-faces-unprecedented-employment-shortfall-2-4-million-skilled-jobs-projected-to-go-unfilled-according-to-deloitte-and-the-manufacturing-institute/" target="_blank">Deloitte and the Manufacturing Institute</a> projected that 2.4 million US manufacturing jobs will go unfilled from 2018-2028 due to the lack of skilled talent<sup>1</sup>. The same study notes that manufacturers who are searching for skilled talent like robot programmers, operations managers, and production managers, expect the skilled labor gap to worsen over the next three years, as the aging baby boomer workforce retires and younger workers decline to enter the manufacturing field.</p><p>Not only is the skilled labor shortage impacting businesses&rsquo; ability to hire, but it&rsquo;s also stunting businesses&rsquo; ability to grow and meet production demands. Manufacturers across industries today are having to turn away new business as a result of not having adequate labor resources. According to a 2018&nbsp;<a href="https://www.kornferry.com/press/korn-ferry-study-reveals-global-talent-shortage-could-threaten-business-growth-around-the-world" target="_blank">study by Korn Ferry</a>, the labor skills gap in the global manufacturing industry could result in more than $607 billion in lost revenue by 2030. In this scenario, the U.S. alone would experience a total of $73 billion in unrealized output.</p><h2>What can be done to address the widening skilled labor gap?</h2><p>One effective solution is to implement robotic automation for repetitive tasks. While there were initial fears that industrial robots would replace human jobs, the reality is that industrial robots are actually helping manufacturers to supplement the low-skilled, repetitive jobs that are otherwise going completely unfilled. According to <a href="https://www2.deloitte.com/us/en/insights/focus/human-capital-trends/2018.html" target="_blank">Deloitte&rsquo;s 2018 Global Human Capital Trends report</a>, 47% of manufacturers are already supplementing their current workforce with automation, with 24% using AI and robotics to perform routine tasks and 16% using automation to augment human skills.</p><p><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1654781735511-shutterstock_681146401.webp" style="width: 766px;" class="fr-fic fr-dib"></p><p id="isPasted">Robotic automation is also helping manufacturers to refocus existing skilled workers on higher-level tasks that require human-only skills. With automation, seasoned human workers are empowered to use their intrinsic knowledge of production best practices to help the manufacturing plant well beyond repetitive or tedious tasks. Refocusing skilled workers on more strategic tasks can create critical competitive advantages for manufacturers, especially as all competitors worldwide struggle to find and retain talent. Additionally, research from the&nbsp;<a href="https://www.weforum.org/agenda/2019/06/industrial-robots-are-bringing-jobs-back-home-but-not-for-low-skilled-workers" target="_blank">Word Economic Forum</a> reveals that increasing robotic automation actually helps to reshore low-skill production jobs from abroad, which in turn helps to increase employment and wages for high-skilled workers.</p><p>Automation doesn&rsquo;t just stop at helping fill open manufacturing jobs. Using industrial robots for tasks that were previously strenuous, or even dangerous, for humans can result in a safer factory environment for all workers. This is especially true for the automotive and electronics industries, where humans were at higher risk for workplace injuries pre-automation; industrial robots in these industries have diverse arrays of computer-vision sensors that help detect when human body parts, like fingers or hands, are in the way of a task being completed. Some industrial robots also have deep machine learning capabilities via artificial intelligence that further help them to learn tasks on the job without pre-programming, all while ensuring that production efforts are being completed in accordance with the highest safety standards.</p><h2>New technologies are enabling automation</h2><p>Advances in industrial automation software are also helping to aid the skilled labor shortage. In the past, manufacturers that purchased industrial robots needed to hire teams of engineer-level programmers to take weeks or months to code the new machines. There are over 30 different manufacturers of industrial robots, each with their own <a href="https://en.wikipedia.org/wiki/Robot_software#Examples_of_programming_languages_for_industrial_robots" target="_blank">unique programming language</a>, meaning that the highly-skilled programmers needed to know how to program the specific industrial robot that was purchased. While these skill requirements made it very difficult in the past for manufacturers to deploy and redeploy robots as quickly as needed, there are now new solutions in automation that make it possible to launch robots without relying on hard-to-find programmers. For example, at READY Robotics, our <a href="https://www.ready-robotics.com/software/forge-os/" target="_blank">Forge/OS</a> software makes it possible for any factory worker to program any robot in minutes via a user-friendly, visual touchscreen &ndash; no coding required. This means that businesses can now rely on existing factory workers to complete the work of rare, high-skilled programmers, which is both more time-efficient and cost-effective.</p><p><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1654781794102-pasted-image-0-3.webp" style="width: 766px;" class="fr-fic fr-dib"></p><p id="isPasted">Human-machine pairing that is enabled by collaborative robots (cobots) will also be helpful for factories that need to maximize their existing human workforce alongside robots. Similar to industrial robots, cobots can also aid with dangerous and repetitive tasks, improve product quality and consistency, and increase overall productivity. A major benefit of cobots over industrial robots is that they allow robots to work within the same factory space as humans, with minimal safety concerns. In some cases, new product designs can be deployed up to 50% faster with cobots than industrial robots, particularly since they often use human-machine interfaces (HMI). By removing the need to have formal robotics or programming experience, HMIs help to address the skilled labor shortage by making it easier and quicker for all workers to learn how to use cobots.</p><p>As the labor shortage situation in the U.S. manufacturing industry continues to unfold in the coming decade, manufacturers should continue to plan ways that they can meet production demands without the availability of highly skilled labor. Though industrial automation cannot fill all roles left empty by the skilled labor shortage, it can help manufacturers to directly address the areas of production that are most repetitive and easiest to complete by robots. It is uplifting to see how automation worldwide is already helping to supplant, and not replace, human workforces. Robotic automation is transforming the factory of the future, where not only is production more efficient, but the available human workforce also benefits from higher-level tasks and safer work environments. It&rsquo;s evident that by working through the implementation of automation, while also empowering factory workers, manufacturers will be in the best position themselves to be successful in the future.</p>

Many manufacturers are struggling to find skilled labor for their manufacturing plants? According to the Society of Manufacturing Engineers, 89% of manufacturers are having difficulty finding skilled workers in the United States today.

Solving the Skilled Labor Shortage with Automation

This article presents a comprehensive technical analysis of CPLD vs FPGA, focusing on their architectural distinctions, performance metrics, design flows, and implementation methodologies. 

CPLD vs FPGA: A Comprehensive Technical Analysis and Implementation Guide

Distributed Control System(DCS) is vital in industrial automation due to its ability to centralize control, monitor processes in real-time, and ensure efficient and safe operations. It enhances process control, improves efficiency, and safeguards operational reliability for maximum productivity


What is Distributed Control System (DCS)?

HMI is a dashboard or user interface that play as the communication bridge between humans and systems, machines, or devices. Read on to know the history, characteristics, advantages, and applications of HMI in different industries.

HMI Technologies: The Ultimate Guide to Human-Machine Interface Innovations

<p id="isPasted">A decade-plus collaboration between MIT’s AgeLab and the Toyota Motor Corporation is recognized as a key contributor to advancements in automotive safety and human-machine interaction. Through the AgeLab at the MIT Center for Transportation and Logistics (CTL), researchers have collected and analyzed vast real-world driving datasets that have helped inform Toyota’s vehicle design and safety systems.</p><p>Toyota recently marked the completion of its 100th project through the Collaborative Safety Research Center (CSRC), celebrating MIT’s role in shaping technologies that enhance driver-assistance features and continue to forge the path for automated mobility. A key foundation for the 100th project is CSRC’s ongoing support for MIT CTL’s Advanced Vehicle Technology (AVT) Consortium.</p><h2><strong>Real-world data, real-world impact</strong></h2><p>“AVT was conceptualized over a decade ago as an academic-industry partnership to promote shared investment in real-world, naturalistic data collection, analysis, and collaboration — efforts aimed at advancing safer, more convenient, and more comfortable automobility,” says Bryan Reimer, founder and co-director of AVT. “Since its founding, AVT has drawn together over 25 organizations — including vehicle manufacturers, suppliers, insurers, and consumer research groups — to invest in understanding how automotive technologies function, how they influence driver behavior, and where further innovation is needed. This work has enabled stakeholders like Toyota to make more-informed decisions in product development and deployment.”</p><p>“CSRC’s 100th project marks a significant milestone in our collaboration,” Reimer adds. “We deeply value CSRC’s sustained investment, and commend the organization’s commitment to global industry impact and the open dissemination of research to advance societal benefit.”</p><p>“Toyota, through its Collaborative Safety Research Center, is proud to be a founding member of the AVT Consortium,” says Jason Hallman, senior manager of Toyota CSRC. “Since 2011, CSRC has collaborated with researchers such as AVT and MIT AgeLab on projects that help inform future products and policy, and to promote a future safe mobility society for all. The AVT specifically has helped us to study the real-world use of several vehicle technologies now available.”</p><p>Among these technologies are lane-centering assistance and adaptive cruise control — widely-used technologies that benefit from an understanding of how drivers interact with automation. “AVT uniquely combines vehicle and driver data to help inform future products and highlight the interplay between the performance of these features and the drivers using them,” says Josh Domeyer, principal scientist at CSRC.</p><h2><strong>Influencing global standards and Olympic-scale innovation</strong></h2><p>Insights from MIT’s pedestrian-driver interaction research with CSRC also helped shape Toyota’s automated vehicle communication systems. “These data helped develop our foundational understanding that drivers and pedestrians use their movements to communicate during routine traffic encounters,” said Domeyer. “This concept informed the deployment of Toyota’s e-Palette at the Tokyo Olympics, and it has been captured as a best practice in an ISO standard for automated driving system communication.”</p><p>The AVT Consortium's naturalistic driving datasets continue to serve as a foundation for behavioral safety strategies. From identifying moments of distraction to understanding how drivers multitask behind the wheel, the work is guiding subtle but impactful design considerations.</p><p>“By studying the natural behaviors of drivers and their contexts in the AVT datasets, we hope to identify new ways to encourage safe habits that align with customer preferences,” Domeyer says. “These can include subtle nudges, or modifications to existing vehicle features, or even communication and education partnerships outside of Toyota that reinforce these safe driving habits.”</p><p>Professor Yossi Sheffi, director of MIT CTL, comments, “This partnership exemplifies the impact of MIT collaborative research on industry to make real, practical innovation possible.” </p><h2><strong>A model for industry-academic collaboration</strong></h2><p>Founded in 2015, the AVT Consortium brings together automotive manufacturers, suppliers, and insurers to accelerate research in driver behavior, safety, and the transition toward automated systems. The consortium’s interdisciplinary approach — integrating engineering, human factors, and data science — has helped generate one of the world’s most unique and actionable real-world driving datasets.</p><p>As Toyota celebrates its research milestone, MIT reflects on a partnership that exemplifies the power of industry-academic collaboration to shape safer, smarter mobility.</p>

Through MIT’s AgeLab and Advanced Vehicle Technology Consortium, researchers have collected and analyzed vast real-world driving datasets that have helped inform Toyota’s vehicle design and safety systems. Photo: Center for Transportation and Logistics

A decade-plus alliance between MIT's AgeLab and Toyota's Collaborative Safety Research Center is recognized as a key contributor to advancements in automotive safety and human-machine interaction.

MIT-Toyota collaboration powers driver assistance in millions of vehicles

<h2 id="isPasted">Programmable Logic Controller (PLC)</h2><p>A PLC is designed to control part or all of a manufacturing process and is simple to use and set up, even for entry-level programmers. A PLC can diagnose faults quickly, and its modular design allows for tailor-made systems to suit every application. A PLC is a highly reliable industrial computer constructed from high-grade, durable materials that can withstand a range of environmental pressures. Its real-time response to changing conditions means operations continue uninterrupted, even when a fault is being resolved.</p><h2>Human Machine Interface (HMI)</h2><p>An HMI is what enables human interaction with the machines that are doing the work, and usually consists of a panel interface that may have keypads, displays and USB connections. While human beings control and operate the machine via the HMI, the machine closely monitors production and simultaneously feeds back information to the operator. This allows quicker and more accurate decision-making and a rapid response to constantly changing production demands.</p><h2>Distributed Control System (DCS)</h2><p>A Distributed Control System is one where a network of autonomous machine controllers is centrally governed by a supervisory operator. Various nodes and control processors are placed at strategic points throughout the manufacturing system, where they can assist in batch-orientated or continuous processes. This means that any problems in the system are localised and only affect that part of the manufacturing process directly controlled by the node, thus reducing overall production downtime.</p><h2>Supervisory Control and Data Acquisition (SCADA)</h2><p>SCADA is a more complex automation concept, referring to a control system architecture which provides an advanced level of supervisory plant management. This architecture comprises a whole network of computers, in one or more locations, combined with a user interface and high speed data communications which allow a distributed network to be accessed remotely.</p><h2>Artificial Neural Network (ANN)</h2><p>An ANN is the basis of Artificial Intelligence and is effectively a computer model based on a biological brain. The brain's neural structure comprises a network of interconnected sensors and synapses, which receive and transmit information to a biological creature and control its responses to received sensory data. The big advantage of a neural network is that it learns by processing and adapting both internal and external information, and an ANN does this in the same way as a biological one. ANNs process data sets and learn from the information there so that they can respond to problems and produce solutions from their own computational capacity.</p>

Automation concepts cover a range of increasingly complex machine initiatives that enhance all areas of 21st century life, and are especially useful for streamlining commercial and production processes.

Five Key Concepts in Automation

<p><strong>How can virtual reality (VR) be experienced haptically, i.e., through the sense of touch? This is one of the fundamental questions that modern VR research is investigating. Computer scientist André Zenner, who is based in Saarbrücken, Germany, has come a significant step closer to answering this question in his doctoral thesis – by inventing new devices and developing software-based techniques inspired by human perception. He has now been awarded the prestigious "Best Dissertation Award" at the world's leading VR conference.</strong></p><p id="isPasted">The award-winning work is about how physical props (technical term: "proxies") can be used to make objects in virtual environments tangible. "Of course, you can't have a proxy for every virtual object, then the approach wouldn't be scalable. In my dissertation, I, therefore, thought about what devices could look like that could be used to simulate the physical properties of several different virtual objects as effectively as possible," explains André Zenner, who completed his doctorate at the Saarbrücken Graduate School of Computer Science at Saarland University and is now conducting research at Saarland University and the German Research Center for Artificial Intelligence.</p><p>This resulted in the prototypes for two special VR controllers, "Shifty" and "Drag:on." VR controllers are devices that can be held in the user’s hand to control or manipulate objects in virtual reality using tracking technology.</p><p><a target="_blank"></a></p><header><h2>Shifty</h2></header><p>"Shifty" is a tubular controller in which a movable weight is installed. The weight can be moved along the lengthwise axis by a motor, changing the center of gravity and inertia of the rod. "In combination with corresponding visualizations in virtual reality, Shifty can be used to create the illusion that a virtual object is getting longer or heavier," explains André Zenner. In experiments, he was able to prove that objects are perceived as lighter or smaller when the weight is close to the user's hand and that, coupled with the corresponding visual input; they are perceived as longer and heavier the further the weight in the rod moves away from the user. "This is mainly due to changes in the inertia of the controller, as the overall weight does not change," explains André Zenner. The research and development department of gaming giant Sony is already experimenting with this concept and cites André Zenner's work in the development of new VR controllers.</p><p><a target="_blank"></a></p><header><h2>Drag:on</h2></header><p>The second controller, "Drag:on," consists of two flamenco fans that can be unfolded using servomotors, thus increasing the air resistance of the controller. This means that the further the fans are unfolded, the more force the user has to exert to move the controller through the air. "Coupled with the right visual stimuli, Drag:on can be used to create the impression that the user is holding a small shovel or a large paddle, for example, or that they are pushing a heavy trolley or are twisting a knob that is difficult to turn," explains André Zenner. Both controllers are basic research and so-called "proof of concepts." This means that the prototypes can be used to show in user experiments that different controller states can improve the perception of different VR objects. Still, specific products using this technology are not yet available on the market.</p><p><a target="_blank"></a></p><header><h2>Similarity and colocation problem</h2></header><p>With the controllers, the Saarbrücken-based computer scientist first addressed the so-called ‘similarity problem.’ The aim here is to ensure that virtual and real objects feel as similar as possible. In the second part of his work, he dealt with the so-called ‘colocation problem,’ i.e., the question of how the proxy can be spatially located in real life where the user sees it in virtual reality. This is particularly challenging as the controllers act as proxies for different virtual objects. Consequently, the user must be given the illusion that they are reaching for various objects, although in reality, they will always grasp the same proxy.</p><p>To achieve this, the researcher made use of the already established method of "hand redirection." As the name suggests, this involves redirecting the movement of the hand in virtual reality so that the user thinks they are reaching to the left, for example, even though they are actually stretching their hand forward. "We conducted experiments to investigate the point at which users realize that their hand has been redirected. Our results showed that this point was reached quickly, so we thought about how we could better conceal the hand redirection," says André Zenner. The solution: he tricked the brain by only redirecting the hand when the brain was blind to visual changes - namely during blinking. Together with a student under his supervision, he developed the appropriate software and used the eye trackers built into many VR headsets. In control studies, the team was then able to show that their new controllers, in combination with hand redirection algorithms, led to more convincing VR perceptions than previously possible.</p><p>This research achievement has now been recognized internationally. The work entitled "Advancing Proxy-Based Haptic Feedback in Virtual Reality" was awarded the "IEEE VGTC Virtual Reality Best Dissertation Award" in mid-March at the "IEEE Conference on Virtual Reality and 3D User Interfaces", which took place this year in Orlando, Florida. According to the IEEE, the award is presented each year to the author of the most outstanding dissertation defended in the previous two calendar years in the research fields of virtual and augmented reality. A total of 16 dissertations were submitted to the 2024 edition of the conference and reviewed by an international committee of experts. The thesis was supervised by Prof. Dr. Antonio Krüger, CEO of the German Research Center for Artificial Intelligence and Head of the Ubiquitous Media Lab at Saarland University.</p><hr><p><strong>Original Publication:&nbsp;</strong>Zenner, A. (2022). Advancing Proxy-Based Haptic Feedback in Virtual Reality. Universität des Saarlandes. <a href="https://dx.doi.org/10.22028/D291-37879" target="_blank" rel="noreferrer">doi:10.22028/D291-37879</a></p>

How can virtual reality (VR) be experienced haptically, i.e., through the sense of touch? This is one of the fundamental questions that modern VR research is investigating.

Human-Robot Interaction (HRI)

Engineers Wiki.

Most Asked Questions.

CPLD vs FPGA: A Comprehensive Technical Analysis and Implementation Guide

What is Distributed Control System (DCS)?

HMI Technologies: The Ultimate Guide to Human-Machine Interface Innovations

TAGGED WITH human machine interaction (hmi)

MIT-Toyota collaboration powers driver assistance in millions of vehicles

CPLD vs FPGA: A Comprehensive Technical Analysis and Implementation Guide

Five Key Concepts in Automation

Tricking the brain: New dimensions of haptics in virtual reality

Latest Posts

Building Internet-Connected Smart Vending Machines With Powerful Single Board Computers

Next-Generation Vehicles Need Safe Human Machine Interfaces

Enhancing Emergency Responders Situational Awareness Using XR Tools

Solving the Skilled Labor Shortage with Automation