How can virtual reality (VR) be experienced haptically, i.e., through the sense of touch? This is one of the fundamental questions that modern VR research is investigating. Computer scientist André Zenner, who is based in Saarbrücken, Germany, has come a significant step closer to answering this question in his doctoral thesis – by inventing new devices and developing software-based techniques inspired by human perception. He has now been awarded the prestigious "Best Dissertation Award" at the world's leading VR conference.The award-winning work is about how physical props (technical term: "proxies") can be used to make objects in virtual environments tangible. "Of course, you can't have a proxy for every virtual object, then the approach wouldn't be scalable. In my dissertation, I, therefore, thought about what devices could look like that could be used to simulate the physical properties of several different virtual objects as effectively as possible," explains André Zenner, who completed his doctorate at the Saarbrücken Graduate School of Computer Science at Saarland University and is now conducting research at Saarland University and the German Research Center for Artificial Intelligence.This resulted in the prototypes for two special VR controllers, "Shifty" and "Drag:on." VR controllers are devices that can be held in the user’s hand to control or manipulate objects in virtual reality using tracking technology.<a target="_blank"></a><header><h2>Shifty</h2></header>"Shifty" is a tubular controller in which a movable weight is installed. The weight can be moved along the lengthwise axis by a motor, changing the center of gravity and inertia of the rod. "In combination with corresponding visualizations in virtual reality, Shifty can be used to create the illusion that a virtual object is getting longer or heavier," explains André Zenner. In experiments, he was able to prove that objects are perceived as lighter or smaller when the weight is close to the user's hand and that, coupled with the corresponding visual input; they are perceived as longer and heavier the further the weight in the rod moves away from the user. "This is mainly due to changes in the inertia of the controller, as the overall weight does not change," explains André Zenner. The research and development department of gaming giant Sony is already experimenting with this concept and cites André Zenner's work in the development of new VR controllers.<a target="_blank"></a><header><h2>Drag:on</h2></header>The second controller, "Drag:on," consists of two flamenco fans that can be unfolded using servomotors, thus increasing the air resistance of the controller. This means that the further the fans are unfolded, the more force the user has to exert to move the controller through the air. "Coupled with the right visual stimuli, Drag:on can be used to create the impression that the user is holding a small shovel or a large paddle, for example, or that they are pushing a heavy trolley or are twisting a knob that is difficult to turn," explains André Zenner. Both controllers are basic research and so-called "proof of concepts." This means that the prototypes can be used to show in user experiments that different controller states can improve the perception of different VR objects. Still, specific products using this technology are not yet available on the market.<a target="_blank"></a><header><h2>Similarity and colocation problem</h2></header>With the controllers, the Saarbrücken-based computer scientist first addressed the so-called ‘similarity problem.’ The aim here is to ensure that virtual and real objects feel as similar as possible. In the second part of his work, he dealt with the so-called ‘colocation problem,’ i.e., the question of how the proxy can be spatially located in real life where the user sees it in virtual reality. This is particularly challenging as the controllers act as proxies for different virtual objects. Consequently, the user must be given the illusion that they are reaching for various objects, although in reality, they will always grasp the same proxy.To achieve this, the researcher made use of the already established method of "hand redirection." As the name suggests, this involves redirecting the movement of the hand in virtual reality so that the user thinks they are reaching to the left, for example, even though they are actually stretching their hand forward. "We conducted experiments to investigate the point at which users realize that their hand has been redirected. Our results showed that this point was reached quickly, so we thought about how we could better conceal the hand redirection," says André Zenner. The solution: he tricked the brain by only redirecting the hand when the brain was blind to visual changes - namely during blinking. Together with a student under his supervision, he developed the appropriate software and used the eye trackers built into many VR headsets. In control studies, the team was then able to show that their new controllers, in combination with hand redirection algorithms, led to more convincing VR perceptions than previously possible.This research achievement has now been recognized internationally. The work entitled "Advancing Proxy-Based Haptic Feedback in Virtual Reality" was awarded the "IEEE VGTC Virtual Reality Best Dissertation Award" in mid-March at the "IEEE Conference on Virtual Reality and 3D User Interfaces", which took place this year in Orlando, Florida. According to the IEEE, the award is presented each year to the author of the most outstanding dissertation defended in the previous two calendar years in the research fields of virtual and augmented reality. A total of 16 dissertations were submitted to the 2024 edition of the conference and reviewed by an international committee of experts. The thesis was supervised by Prof. Dr. Antonio Krüger, CEO of the German Research Center for Artificial Intelligence and Head of the Ubiquitous Media Lab at Saarland University.<hr>Original Publication:&nbsp;Zenner, A. (2022). Advancing Proxy-Based Haptic Feedback in Virtual Reality. Universität des Saarlandes. <a href="https://dx.doi.org/10.22028/D291-37879" target="_blank" rel="noreferrer">doi:10.22028/D291-37879</a>

How can virtual reality (VR) be experienced haptically, i.e., through the sense of touch? This is one of the fundamental questions that modern VR research is investigating.

Tricking the brain: New dimensions of haptics in virtual reality

Accessibility is an undeniably important subject, yet one which is all-too often overlooked in the development of new technologies. Despite the march of progress, the majority of legally blind individuals rely on a simple white cane for navigation — something researchers Manuel Zahn and Armaghan Ahmad Khan are looking to change.The pair, working in the Center for Digital Technology and Management at the Technical University of Munich and in partnership with FRAMOS GmbH, have developed a system which combines a depth-sensing 3D camera with a wearable haptic feedback sleeve — providing, they say, navigation and obstacle avoidance capabilities entirely hands-free, and at a fidelity and range which far exceeds a cane.<h1>Computer vision for the blind</h1>The fully-wearable system proposed by Zahn and Khan is built around an Intel RealSense D415 depth-sensing camera, housed in a 3D-printed pair of glasses worn over the user’s eyes. This is linked to a single-board computer, an AAEON Up Board, which handles processing the data from the camera, and a battery pack for portability.The RealSense Camera is designed around an active infrared stereo imaging system, projecting beams of invisible infrared light and measuring the depth of its surroundings. Compared with a system based on visible light cameras, it has one major advantage: It can operate in absolute darkness, allowing its user to navigate without difficulty.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1644261002540-blind2.jpg" style="width: 300px;" class="fr-fic fr-dib" alt="Two images: the left image shows an Intel RealSense depth-sensing camera mounted in a 3D printed pair of glasses, in place of the lenses; the right image shows a person wearing the camera system.">An off-the-shelf depth-sensing camera combined with a 3D printed frame acts as the input to a hands-free navigation system for the blind. Determining the depth of the wearer’s surroundings is only part of the system’s function, however: That information has to be communicated to the user in some way. Eschewing spoken or otherwise audible feedback, as likely to take away from the user’s ability to easily hear their surroundings, the researchers developed a wearable haptic feedback sleeve made from 25 vibration motors sewn onto fabric and linked to an Adafruit Feather 32U4 microcontroller development board.With this, the system is complete: A depth-sensing camera which can map the wearer’s surroundings onto the 25-“pixel” haptic feedback system, providing a physical response when obstacles are near.<h1>Novel navigation</h1>Exactly what feedback is provided depends on where the user is navigating, the pair explain. For indoor navigation, the system provides vibration of nearby obstacles and walls — the location of the vibration mapped to the direction of the obstacle and the intensity mapped to its proximity. This way, a user can navigate with only the wearable system.An outdoor mode switches from a standalone navigation device to an accessory, designed for use with the traditional cane: The obstacle detection range is extended to beyond two meters (around six and a half feet), providing information about obstacles and other objects outside the range of a traditional cane.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1644261012136-blind3.jpg" style="width: 300px;" class="fr-fic fr-dib" alt="A picture of a custom-built haptic feedback sleeve, comprised of a grid of 25 vibration motors sewn onto fabric. Wires are seen leading to a control box and a battery pack.">Designed to keep the user's hands free, this haptic feedback sleeve proved easy to use. To prove the concept, the pair set up a navigation route which included walkways, doors, and various obstacles — and which had to be traversed in absolute darkness. Impressively, after a period of familiarization with the vibration feedback system, the five volunteers were able to complete the obstacle course perfectly on their very first try.In subsequent attempts, the time taken to traverse the course reduced — showing increasing familiarity with and trust in the system: “Overall the averages for the runs decreased from 320 seconds in the first run to 148 seconds in the third run, two weeks later,” the researchers note — while admitting that one participant did become disoriented for a short period in their last run with the system.The results encourage, with Zahn and Khan pointing to the system’s low cost and lightweight nature as making it of interest for future development — as too is the fact it does not interfere with the wearer’s hearing and leaves both hands free, either for using a traditional navigation aid like cane or for any other task.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1644261019783-blind4.jpg" style="width: 300px;" class="fr-fic fr-dib" alt="A picture of a narrow hall with a grid of colored boxes overlaid; an image to the right shows how each color maps to a strength of vibration on the haptic feedback sleeve.">Varying strength and positional haptic feedback turn obstacles into a feeling on the wearer's arm. “Further developments could include the implementation of an object recognition, based on the RGB [visible light] image of the camera,” the pair add. “This would allow the system to artificially generate bordered paths, solving the problem of orientation in a large open space.”The only sticking point: Intel, the manufacturer of the camera platform used for the prototype, <a href="https://www.crn.com/news/components-peripherals/why-pat-gelsinger-decided-to-wind-down-intel-s-realsense-unit-exclusive" rel="noopener noreferrer" target="_blank">recently confirmed</a> it was to discontinue its RealSense platform — though, at the time of writing, it continued to provide the RealSense D415 for sale.Zahn and Khan's paper is available <a href="https://arxiv.org/abs/2201.04453" rel="noopener noreferrer" target="_blank">on Cornell's arXiv.org preprint server</a>.<h1>References</h1>Manuel Zahn, Armaghan Ahmad Khan: <cite>Obstacle avoidance for blind people using a 3D camera and a haptic feedback sleeve</cite>. DOI <a href="https://arxiv.org/abs/2201.04453" target="_blank">arXiv:2201.04453</a> [cs.HC].Dylan Martin, CRN: Why Pat Gelsinger Decided to Wind Down Intel's RealSense Unit: Exclusive.&nbsp;<a href="https://www.crn.com/news/components-peripherals/why-pat-gelsinger-decided-to-wind-down-intel-s-realsense-unit-exclusive" id="isPasted" rel="noopener noreferrer" target="_blank">https://www.crn.com/news/components-peripherals/why-pat-gelsinger-decided-to-wind-down-intel-s-realsense-unit-exclusive</a>

Using a depth-sensing camera, this wearable navigation platform can alert its wearer to obstacles via haptic feedback.

Built around a custom-made haptic feedback sleeve and an Intel RealSense camera in a 3D-printed glasses housing, this navigation system lets its user find their way even in absolute darkness.

A wearable navigation system, featuring a 3D camera, offers a hands-free navigation solution for the blind

The Robot Teleoperativo is a new robotics result achieved at IIT-Istituto Italiano di Tecnologia and combining the rugged locomotion of HyQReal robot with dexterous and powerful manipulation assured by a new robotic arm, controlled by immersive VR visualization and haptic teleoperation. It represents a significant contribution to the state-of-the-art robotics dedicated to intervention on unstructured, difficult terrain environments, requiring powerful capabilities. The human operator can always be in control, not being replaced by the robot but substituted and assisted by it, in the case of situations where humans may be exposed to risks for their health, such as disaster response or emergency in nuclear, marine, chemical, oil-&amp;amp;-gas, environments.In a new video, researchers show the achieved results: the Robot Teleoperativo is able to open doors, access fire equipment, gather precious items, navigate in the dark, and walk over difficult terrain. The video will be showed during the international conference ICRA 2021, on May 31st.With the effective execution of such tasks, albeit in simulated lab setup scenarios, the three years of developmental research in the Robot Teleoperativo project has shown that although the challenge of substituting human workers is a difficult one, it is certainly not an insurmountable one.The project is a result of the close collaboration between IIT-Istituto Italiano di Tecnologia and INAIL (Italian National Worker’s Compensation Authority). The 3-year project started in 2017 with an aim to enhance occupational safety and health (OSH) in demanding and dangerous occupations. The goal was to keep the worker who is engaged in high-risk interventions in demanding environments, at the core of the development; researchers worked with firefighters to define specific needs and they will further explore the use of Robot Teleoperativo in realistic scenario.Robot Teleoperativo consists of two modules, the field robot and the pilot station. The field robot integrates the versatile locomotion of the HyQReal hydraulic quadruped robot with the dexterity and power of the INAIL-IIT multi-degrees-of-freedom (DOFs) robotic arm. At the pilot station, the 6-DOF haptic teleoperation device REMOTArm, including the 3-finger HEXOTrAc-Plus hand exoskeleton, combines with the 3D virtual reality (VR) based intuitive, information-rich interface for perception, interaction, and teleoperation. The field-pilot&nbsp;combination offers an immersive telepresence experience to the operator (worker), allowing them to project their actions to and perceive the information from the remote environment through the field robot.The HyQReal quadruped, with its 90 cm height, 133 cm length, and 130 kilograms of weight, demonstrated in 2019 being able to develop sufficient force to tow a passenger plane of over three tons. With its advanced locomotion, the HyQReal is capable of navigating difficult terrain, characterized by slopes, stairs, or debris, with a degree of mobility far greater than in the case of wheeled or tracked vehicles. The HyQReal is equipped with a variety of cameras and multimodal sensors for real-time environment perception.INAIL-IIT arm is a robotic manipulator with strength and dexterity. The 5-DOF serial robotic arm provides the ability to interact with, and manipulate the environment with robustness, precision, and power. It is about 1.0m in length, and has a 4 kg payload capacity. Its 5 joints integrate compact and lightweight, torque controlled actuators, which allow the manipulator arm to operate in position and / or torque-control depending on the task. The HERI II robotic hand, with its multi-finger configuration, combines strength and dexterity for grasping heavy as well as delicate objects.REMOTArm is the haptic teleoperation pilot interface for precise tele-manipulation. The REMOTArm is a desk-mounted, 6-DOF, haptic teleoperation device. It tracks the position and orientation of the operator’s wrist with high accuracy, and offers a much larger workspace than similar commercial devices. It provides 3-DOF force feedback at the wrist (&gt; 20N). The 3-finger HEXOTrAc-Plus hand exoskeleton is an ergonomic haptic glove that accurately tracks the operator’s gestures when tele-manipulating the field manipulator. Its actuators can display the remotely sensed interaction forces (&lt; 10 N) to the user at the robotic hand as they happen. The interface is lightweight, easy-to-wear, and includes gravity compensation for user comfort.The operator uses a VR system for an immersive 3D perception of the remote environment. The user interface, using commercial head-mount display devices and VR software, makes for an intuitive and effective real-time projection of the field of action for the remote operator. Using the multimodal vision, depth, and ambient sensing system on-board the field robot, the interface “reconstructs” the remote field context for the operator, providing an information-rich interface for perception, interaction, and control.This successful outcome has encouraged researchers to push the boundaries of technological innovation further. IIT and INAIL continue their close collaboration in this field aiming at expanding the possibilities in remote telerobotics research and its transition from the lab to the real hazardous work environments.<hr>Further informationThe research study was coordinated by Nikhil Deshpande, Researcher at IIT’s Advanced Robotics Line, and involved the research groups led by Claudio Semini (for HyQReal), Nikolaos Tsagarakis (for manipulation) and Ioannis Sarakoglou (for teleoperation) at IIT in Genova (Italy).&nbsp;Website: <a href="https://advr.iit.it/projects/inail-scc/teleoperazione" target="_blank">https://advr.iit.it/projects/inail-scc/teleoperazione</a>

The Robot Teleoperativo is a new robotics result achieved at IIT-Istituto Italiano di Tecnologia and combining the rugged locomotion of HyQReal robot with dexterous and powerful manipulation assured by a new robotic arm, controlled by immersive VR visualization and haptic teleoperation.

haptics

Profiles