The field of robotic dexterous hands is rapidly evolving with technological advancements and innovative research. These agile devices have a potent potential to change industries and day-to-day life and so are the focus of research within the robotics industry. 

However, the engineering challenges posed by dexterous robotic hands are numerous. Complexities like miniaturization, grip and sensing, accurate simulation testing as well as the application of AI and machine learning are hurdles that researchers must overcome to achieve robotic hands that are as capable as human hands.

In this article, we will discuss the complexities of dexterous hand development and offer some insights into how they are being solved using examples from Shadow Robot, a company known for creating one of the most advanced dexterous robotic hands in the world. 

Complexities of Dexterous Robotic Hands

Key areas of robotics development like miniaturization, sensing, and control are particularly complex when integrated with dexterous hands. Miniaturization is a challenge because dexterous hands must be capable of human-like precision and control, which is difficult to achieve in a compact space. Sensing is also a challenge because dexterous hands require a high degree of sensitivity in order to detect and respond to elements in their environment. Control poses yet another challenge, because dexterous hands require sophisticated algorithms and engineering to coordinate the movements of multiple fingers and joints^[1].

Addressing these challenges is important for broader adoption of dexterous hands in both research and industry. 

Improving sensing and grasping methods opens new applications for robotic manipulation. When combined with AI and machine learning, dexterous robotic systems have the potential to apply human-like adaptation and movement to tasks that are repetitive, hazardous, or that require more speed or precision than a human can provide^[2].

Multi-force grip and sensing

The challenge

Dexterous robotic hands must be able to perform intricate tasks and manipulate different types of objects in variable environments. This means that dexterous hands need complex sensor and control systems and must incorporate multi-force gripping that can adapt according to the object and the task. Human grip force is controlled by biological systems, and to replicate this in robotic hands, researchers must develop sensors and control systems that can accurately measure and respond to features of the object and environment, like pressure, interaction forces, and vibrations^[3].

The solution

Sensing capabilities play an important role in dexterous robotic hands. Visual and tactile feedback enables robots to quantify physical properties and is used increasingly in robotic systems. The growing availability of affordable tactile sensors has encouraged exploration of real-world tactile applications, such as the field of agriculture, where grip force is critical in applications like automated harvesting^[4].

Researchers have developed innovative sensors and control systems to enable dexterous manipulation of objects by robotic hands. Examples include:

• MIT’s GelSight Svelte, a finger-shaped camera-based sensor that provides high-resolution tactile sensing over a large area, which enables the hand to perform different types of grasps, including pinch grasps. The sensor has a flexible backbone and uses mirrors to reflect light into a camera, which can see along the length of each finger.^[5]

• The ILDA (integrated linkage-driven dexterous anthropomorphic robotic hand), developed by researchers at Ajou University, with six-axis force and torque sensors that can measure the various forces acting on the robotic fingertips. Researchers integrated algorithms that can determine the contact force parameters, including the contact, location, and the triaxial force applied to the fingertips^[6].

• Shadow Robot’s Dexterous Hand, which has over 100 sensors that operate at up to 1 KHz, which give the hand multiple tactile sensing options and fast response. The hand can be integrated with tactile fingertips, which enable it to perform multiple grasps, like pinch grasps, lateral pinch grasps, and a power grasp that uses the entire sensing area of the fingers^[7].

The Shadow Dexterous Hand gripping a lightbulb. Source: shadowrobot.com

AI and machine learning

The challenge

Artificial intelligence and machine learning play an important role in the development of dexterous robotic hands. When these technologies are applied to sensor data, they can improve perception, object recognition, grasping, and manipulation capabilities. However, deploying AI and ML in robotic hands is a challenge because it can be difficult to produce enough data for effective learning^[8].

The solution

Shadow Robot has integrated AI and ML into their dexterous hand technology to manipulate physical objects and complete tasks in a way that is intuitive for the operator^[9]. The company has worked with companies like OpenAI to advance research in this area. The Dexterous Hand provides abundant sensor and positional data that researchers can apply to machine learning algorithms and develop innovative control systems^[10].

Deep reinforcement learning has also emerged as a promising approach to control dexterous robotic hand systems. Deep reinforcement is model-agnostic, meaning that controls do not rely on a specific model of the system. To overcome the problem of the small sample size for learning data, researchers have combined deep reinforcement learning with human demonstration, which enables learning using sample sizes of just a few hours of robot learning^[11].

The role of simulation testing to overcome challenges

The development of dexterous robotic hands is a complex and challenging task that requires substantial testing and iteration to achieve successful mimicry of the human hand. However, it can quickly become costly, and sometimes hazardous, to perform field testing with physical robots. Instead, simulation testing provides data points on the capabilities of robotic systems using a computer-based simulation of the hardware^[12].

Simulation testing is a practical, safer, and more scalable alternative to field testing and is critical to developing human-like functionality in dexterous hands^[13]. Using simulation testing, developers can evaluate and compare the performance of robotic hands while adjusting design characteristics and environmental factors. Researchers can perform stress tests on the system in less time and without damaging physical hardware or putting the safety of humans at risk^[14].

However, creating realistic simulations for dexterous robotic hands poses another set of challenges. One of the biggest challenges of simulation testing is to create a simulation that is realistic enough to be useful. If the simulation is unable to replicate the kind of environment that the robot will operate in and adapt to, then the data from the simulation cannot reliably be used to improve the robot^[15].

Another challenge of simulation testing is the lack of reproducibility and the resource costs associated with simulators^[16]. Despite these challenges, simulation testing is critical to the ongoing development of dexterous robotic hands. It allows developers to test and evaluate new algorithms and models in a controlled environment before building physical systems^[17].

Shadow Robot’s vision and simulation testing

Simulation testing is a key part of how Shadow Robot is overcoming major challenges in dexterous hand development. Shadow Robot is renowned for pioneering work in advanced dexterous hand development. The company is driven by a passion to create advanced dexterous robots that solve real-world problems and inspire innovation.

Shadow Robot uses several simulators to test innovations in dexterous hand technology:

• The NVIDIA Isaac robotics simulator, which allows robots to be trained efficiently in a simulated universe. This simulation environment provides data that is diverse enough to effectively train the hand to perform challenging tasks.

• The MuJoCo (Multi-Joint dynamics with Contact) physics engine for simulating rigid-body systems and soft-body deformation. This engine provides fast and accurate simulation of how physical bodies interact, with sophisticated replication of kinematics and dynamics.

• Gazebo, an open-source simulator with capabilities to test complex scenarios and algorithms in a 3D environment. Gazebo supports a wide range of sensors and provides real-time data for visualization and analysis.

The company’s Dexterous Hand is the most advanced 5-fingered robotic hand in the world. It can manipulate tools and objects with unprecedented accuracy, opening new avenues for research in robotics and artificial intelligence. The Shadow Hand is designed to replicate the human hand as closely as possible and brings the field of robotics closer to overcoming the challenges of dexterous hand development. Shadow Robot’s advancements in dexterous hand technology have the potential to transform industries and daily life.

Conclusion

The field of dexterous robotic hands is rapidly evolving. Progress in this field is driven by advancements in technology and research, as well as the potential to reshape how humans live and work. In this article, we have shown that the complexities of robot hand development are numerous, but that promising technologies are emerging to overcome these problems. Researchers have found amazing solutions to solve the challenges of simulation testing, miniaturization, gripping and sensing, as well as the integration of AI and ML.

Shadow Robot’s vision encompasses the potential for these advancements with their groundbreaking work in dexterous hand technology. The company is positioned to transform industry and daily life. The journey is challenging, but the potential rewards are immense in a future where robotic hands are integrated into our daily life.

For more information about Shadow Robot, check out their website at www.shadowrobot.com.

You can also read our three-part series on Dexterous Robotic Hands.

References

[1] “Highly dexterous robot hands can operate in the dark — just like humans”. National Science Foundation (June 9, 2023). Accessed from https://new.nsf.gov/news/highly-dexterous-robot-hand-can-operate-dark-just 

[2] Shuang Li, et al. “A hand-arm teleoperation system for robotic dexterous manipulation”. Tactile Sensing, Skill Learning, and Robotic Dexterous Manipulation (2022). Accessed from https://www.sciencedirect.com/topics/computer-science/robotic-hand 

[3] W. Mandil, et. al. “Tactile-Sensing Technologies: Trends, Challenges and Outlook in Agri-Food Manipulation”. Sensors (Basel) (2023). Accessed from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10490130/ 

[4] W. Mandil (2023).

[5] A. Zewe. “Finger-shaped sensor enables more dexterous robots”. MIT News. (4 Oct. 2023). Accessed from https://news.mit.edu/2023/finger-shaped-sensor-enables-more-dexterous-robots-1004 

[6] U. Kim et al. “Integrated linkage-driven dexterous anthropomorphic robotic hand”. Nature Communications, Vol 12. (14 Dec. 2021). Accessed from https://www.nature.com/articles/s41467-021-27261-0 

[7] Shadow Robot. Dexterous Hand Series. Accessed from https://www.shadowrobot.com/dexterous-hand-series/ 

[8] A. Rajeswaran, et. al. “Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations”. Arxiv (2018). Accessed from https://arxiv.org/abs/1709.10087 

[9] Shadow Robot. Dexterous Hand Series. Accessed from https://www.shadowrobot.com/dexterous-hand-series/ 

 [10]Shadow Robot. “The benefits of the Shadow Hand for AI and Machine Learning”. Accessed from https://www.shadowrobot.com/blog/the-benefits-of-the-shadow-hand-for-ai-and-machine-learning/ 

[11] A. Rajeswaran, et. al. “Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations”. Arxiv (2018). Accessed from https://arxiv.org/abs/1709.10087 

 [12]S. Crowe. “10 challenges of using simulators for testing robots”. The Robot Report (May 25, 2020). Accessed from https://www.therobotreport.com/10-challenges-simulators-robotics-testing/ 

[13] S. Crowe (2020).

[14] N. Elangovan, et. al. “An Accessible, Open-Source Dexterity Test: Evaluating the Grasping and Dexterous Manipulation Capabilities of Humans and Robots”. Front Robot AI (2022). Accessed from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9081435/ 

[15] S. Crowe (2020).

[16] S. Crowe (2020).

[17] S. Andersson. “Simulation challenges in robotic grasping”. Umeå University (Spring, 2021). Accessed from https://www.diva-portal.org/smash/get/diva2:1583322/FULLTEXT01.pdf 

[18] Shadow Robot. “Simulating the Shadow Hand”. (1 Aug. 2023). Accessed from https://www.shadowrobot.com/news/press-releases/simulating-the-shadow-hand/