3D scanning of the human back is a modern method that&nbsp;helps noninvasively identify disorders such as scoliosis.&nbsp;Using a 3D scanning approach provides a noninvasive method that can prevent invasive medical procedures&nbsp;and&nbsp;radiation risks&nbsp;from medical X-ray examinations. The Photoneo MotionCam-3D provides&nbsp;accurate and high-resolution 3D scans even in motion, which makes it the ideal choice for scanning the human body.Authors:Mirko Kaiser (1,2), Tobia Brusa (1), Marco Wyss (1), Saša Ćuković (2), Martin Bertsch (1,2), William R. Taylor (2), and Volker M. Koch (1)<ol><li>Biomedical Engineering Lab, <a href="https://www.bfh.ch/en/" target="_blank">Bern University of Applied Sciences</a>, 2501 Biel, Switzerland</li><li><a href="https://movement.ethz.ch/" target="_blank">Laboratory for Movement Biomechanics, ETH Zurich</a>, 8092 Zürich, Switzerland</li></ol>Challenge:Adolescent idiopathic scoliosis (AIS) is an abnormal curvature of the spine that appears in late childhood or adolescence. The early screening of children and adolescents is crucial to identify and prevent the further progression of AIS. Doctors often use tools like scoliometers to estimate the spine’s angle noninvasively.&nbsp;3D scanners are also used to estimate spine deformities but&nbsp;not all scanners are sufficiently good for this purpose. The authors of this pilot study tested multiple 3D scanning technologies on 30 AIS patients and 76 healthy people to identify the minimal resolution required to capture the topography and deformities of the human back.&nbsp;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzEyNzUwNzgzOTQzLTE3MTI3NTA3ODM5NDMucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" alt="3D scanning for scoliosis diagnistics_Workflow" width="774" height="410" srcset="https://www.photoneo.com/wp-content/uploads/2024/03/WORKFLOW_Picture7-400x212.jpg 400w, https://www.photoneo.com/wp-content/uploads/2024/03/WORKFLOW_Picture7-1024x543.jpg 1024w, https://www.photoneo.com/wp-content/uploads/2024/03/WORKFLOW_Picture7-768x407.jpg 768w, https://www.photoneo.com/wp-content/uploads/2024/03/WORKFLOW_Picture7-1536x814.jpg 1536w, https://www.photoneo.com/wp-content/uploads/2024/03/WORKFLOW_Picture7-453x240.jpg 453w, https://www.photoneo.com/wp-content/uploads/2024/03/WORKFLOW_Picture7.jpg 1568w" sizes="(max-width: 774px) 100vw, 774px" class="fr-fic fr-dii">Solution:<a href="https://www.photoneo.com/motioncam-3d/" target="_blank">MotionCam-3D</a>The evaluation is based on a dataset of scoliotic patients, consisting of 30 optical captures of the human back, and a dataset containing 76 healthy subjects.The capturing was carried out using different 3D acquisition systems:a) Photoneo’s patented Parallel Structured Light&nbsp;technology implemented in MotionCam-3D M+b) Structured light with TIDA-00254 (DLP Lightcrafter 4500) and a monochrome 2D camera from HIKROBOT (MV-CA023-10UM)c) Active stereo with Lightcrafter 4500 and two monochrome 2D cameras from HIKROBOTd) Single-shot structured light with the Orbbec Astra Minie) Active stereo with Intel D415&nbsp;<figure><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzEyNzUwNzgzNzU4LTE3MTI3NTA3ODM3NTgucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" alt="3D back shape captures from Photoneo MotionCam-3D and other 3D vision technologies" width="947" height="353" srcset="https://www.photoneo.com/wp-content/uploads/2024/03/Three-dimensional-back-shape-captures-from-a-Photoneo-MotionCam-3D-b-structured-light-with-TIDA-00254-c-active-stereo-with-BoofCV-d-Orbbec-Astra-Mini-e-Intel-D415.-400x149.jpg 400w, https://www.photoneo.com/wp-content/uploads/2024/03/Three-dimensional-back-shape-captures-from-a-Photoneo-MotionCam-3D-b-structured-light-with-TIDA-00254-c-active-stereo-with-BoofCV-d-Orbbec-Astra-Mini-e-Intel-D415.-480x179.jpg 480w, https://www.photoneo.com/wp-content/uploads/2024/03/Three-dimensional-back-shape-captures-from-a-Photoneo-MotionCam-3D-b-structured-light-with-TIDA-00254-c-active-stereo-with-BoofCV-d-Orbbec-Astra-Mini-e-Intel-D415..jpg 742w" sizes="(max-width: 947px) 100vw, 947px" class="fr-fic fr-dii"><figcaption>Picture: 3D back shape captures from: a) Photoneo MotionCam-3D b) Structured light with TIDA-00254 c) Active stereo with BoofCV d) Orbbec Astra Mini e) Intel D415</figcaption></figure>&nbsp;The Photoneo MotionCam-3D is an industrial area-scan 3D camera that is powered by Photoneo’s patented Parallel Structured Light technology. While conventional structured light technologies need to emit several light patterns in a sequence, the Parallel Structured Light&nbsp;technology allows capturing a number of patterns in one laser sweep thanks to the proprietary COMPIS sensor.Thanks to this, MotionCam-3D provides high-resolution and submillimeter-accuracy 3D data even in motion. This makes it an ideal choice for applications involving 3D scanning of the human body.Because of its unique features and unequalled performance level, MotionCam-3D was used as a reference&nbsp;and ground truth for 3D scanning in this pilot study.<figure><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzEyNzUwNzgzODk2LTE3MTI3NTA3ODM4OTYucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" alt="Photoneo MotionCam-3D used for scanning the human back" width="813" height="504" srcset="https://www.photoneo.com/wp-content/uploads/2024/03/Photoneo-MotionCam-3D-used-for-scanning-the-human-back-400x248.png 400w, https://www.photoneo.com/wp-content/uploads/2024/03/Photoneo-MotionCam-3D-used-for-scanning-the-human-back-1024x635.png 1024w, https://www.photoneo.com/wp-content/uploads/2024/03/Photoneo-MotionCam-3D-used-for-scanning-the-human-back-768x476.png 768w, https://www.photoneo.com/wp-content/uploads/2024/03/Photoneo-MotionCam-3D-used-for-scanning-the-human-back-387x240.png 387w, https://www.photoneo.com/wp-content/uploads/2024/03/Photoneo-MotionCam-3D-used-for-scanning-the-human-back.png 1460w" sizes="(max-width: 813px) 100vw, 813px" class="fr-fic fr-dii"><figcaption>Picture: Photoneo MotionCam-3D</figcaption></figure>&nbsp;The scan data was processed using three approaches:1.) The first approach used a frequency analysis of a ground-truth 3D representation to obtain an initial estimate of its contained frequencies. This estimate was used in the second and third approaches as the initial cut-off frequency.&nbsp;2.) The second approach compared the shape of the reduced-quality 3D representation with the ground-truth shape. This approach took a ground-truth 3D representation of the object of interest (in this case, the human back) as the input. The ground-truth 3D representation can be either captured using a very accurate 3D acquisition system (in this case, Photoneo MotionCam-3D, with a point size of 0.52 mm, @z = 900 mm, and accuracy &lt; 0.3 mm) or delivered as a CAD representation (e.g., for artificial objects).3.) The third approach compared the symmetry line of the reduced-quality 3D representation with its benchmark.Discussion:The results of this pilot study suggest that for accurate scans of the human back, a resolution of 2 to 5 millimeters is necessary, and a camera resolution of at least 640 × 480 pixels is recommended for future studies.The optical 3D systems evaluated in this paper are summarized below:<table width="803"><tbody><tr><td>System</td><td>Resolution&nbsp;</td><td>Accuracy</td></tr><tr><td>Photoneo MotionCam-3D M+</td><td>1680 × 1200 resp. 1120 × 800&nbsp;</td><td>error &lt; 0.3 mm at 0.9 m</td></tr><tr><td>SL/AS with TIDA &amp; HIKROBOT</td><td>912 × 1140 resp. 1920 × 1200&nbsp;</td><td>error ~1 mm at 1 m</td></tr><tr><td>Intel D415</td><td>1280 × 720&nbsp;</td><td>error &lt; 2% up to 2 m</td></tr><tr><td>Orbbec Astra Mini</td><td>640 × 480</td><td>error &lt; 3 mm at 1 m</td></tr></tbody></table>All tested systems satisfy the required minimal resolution to capture the human back.The Photoneo MotionCam-3D also satisfies the minimal required spatial resolution. The results&nbsp;confirm that the Photoneo MotionCam-3D is a valid choice as a ground-truth-capturing&nbsp;device&nbsp;because&nbsp;the accuracy error of less than 0.3 mm is three times lower than the required spatial resolution resulting from the first approach (1 mm) and seven to seventeen times lower than the required spatial resolution resulting from the second and third approaches (2 mm to 5 mm).The Photoneo MotionCam-3D is thus a valid option for capturing the human back.The approaches and recommendations in the paper can be used to obtain an approximate estimate of the minimal required spatial resolution and the minimal required camera and projector resolution. However, the authors recommend additional practical measurements with such systems to confirm the requirements for individual applications.&nbsp;The full paper titled “Minimal Required Resolution to Capture the 3D Shape of the Human Back—A Practical Approach” was published by PMC PubMed Central, the National Library of Medicine.You can <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10537399/" target="_blank">access the full paper here</a>.Would you like to discuss your use case with the Photoneo team and how Photoneo technology can help you achieve your desired results? <a href="https://www.photoneo.com/contact-us/?utm_source=Wevolver&utm_medium=Case%20study&utm_campaign=3D%20scanning%20for%20scoliosis%20diagnostics" target="_blank" rel="noopener noreferrer">Contact Photoneo</a>.

Using a 3D scanning approach such as Photoneo MotionCam-3D for scoliosis diagnosis provides a noninvasive method that can prevent invasive medical procedures and radiation risks from medical X-ray examinations.

3D scanning for scoliosis diagnostics

Machine vision represents a fascinating fusion of hardware and software designed to empower machines to 'see' and interpret the world around them. This technology, critical in manufacturing, automates tasks that require visual understanding, enhancing efficiency, accuracy, and safety on production lines. From guiding robots to inspecting products for defects, machine vision systems translate visual inputs into actionable information, fundamentally transforming industrial operations.The evolution of machine vision is closely linked with advancements in artificial intelligence (AI). As AI technologies have progressed, they have significantly enhanced the capabilities of machine vision systems, enabling more complex and nuanced interpretation of visual data. This integration has been pivotal in driving Industry 4.0, the ongoing automation of traditional manufacturing and industrial practices using smart technology.&nbsp;Augmented with AI, machine vision is an enabler of automation and a key player in creating intelligent, adaptable, and efficient manufacturing environments. The convergence of machine vision and AI pushes the boundaries of what's possible in industrial automation, leading to unprecedented productivity and innovation.<h2 dir="ltr">The Core Components of Machine Vision Systems</h2>Machine vision systems are intricate setups that combine hardware and software to enable machines to 'see' and make decisions based on visual inputs. These systems play a pivotal role in various applications, including automated inspection, process control, and guiding robots, significantly contributing to industrial automation and robotics advancements.<h3 dir="ltr">Hardware: The Observational Components</h3>The foundational hardware components of machine vision systems include cameras and sensors, which serve as the 'eyes' of the system. Cameras equipped with image sensors capture visual data from the environment. These cameras range from simple, compact smart cameras for specific applications to more complex multi-camera systems offering greater versatility and data collection capabilities.&nbsp;Sensors are crucial for determining the system's specifications, such as resolution, directly impacting the amount of data collected and the system's ability to discern environmental changes. The resolution, defined by the total number of pixels in rows and columns, affects not just image clarity but also processing speed, with monochrome cameras often facilitating faster processing than colour counterparts due to simpler data.<h3 dir="ltr">Software: The Analytical Brain</h3>On the software side, machine vision systems are powered by image processing algorithms and, increasingly, by deep learning and AI technologies. These software components are tasked with analysing and interpreting the captured visual data. The traditional approach to image processing involves predefined rules for analysing images, which can be rigid and limited in scope.<h3 dir="ltr">The Synergy of Hardware and Software</h3>The interplay between the hardware's data collection capabilities and the software's processing power defines the effectiveness of a machine vision system. Adequate lighting and lens quality are also essential, ensuring the captured images are clear and detailed enough for accurate analysis. For example, the choice between backlighting and front-lighting setups can significantly affect the system's ability to capture usable data in real-world applications.<h2 dir="ltr">Implementing Machine Vision in Manufacturing</h2>Implementing machine vision in manufacturing significantly enhances quality control, defect detection, and production efficiency. This technology integration facilitates a transition from manual to intelligent manufacturing, providing high detection accuracy, speed, and improved coordination.Machine vision technology is pivotal in collecting information, detecting quality, positioning, automatic sorting, and intelligent monitoring within manufacturing. It addresses the challenges of poor quality, low precision, low efficiency, and high labour intensity traditionally associated with manual operations. Adopting machine vision in manufacturing accelerates production, elevates detection accuracy, and minimises the risk of accidents.<h3 dir="ltr">Enabling Algorithms for Object Detection, Identification, and Classification</h3>The backbone of machine vision systems in industrial settings is a suite of sophisticated algorithms that enable object detection, identification, and classification. These algorithms leverage optical tools and computer technology to collect and process data, simulating human visual functions.&nbsp;With the advancement of AI, machine learning, and deep learning technologies, machine vision systems have achieved higher accuracy and broader application scopes. These technologies encourage traditional manufacturing processes to become more intelligent, requiring expertise across various fields, including optical imaging, signal processing, image processing, and deep learning.<h2 dir="ltr">Current Trends and Future Directions</h2>The evolution of machine vision technology is rapidly transforming the landscape of industrial automation, robotics, and beyond. At the forefront of this transformation is integrating 3D imaging and point cloud data, offering enhanced spatial analysis and enabling more complex applications. Let's explore the cutting-edge research and predictions for future advancements in this exciting field.<h3 dir="ltr">Integrating 3D Imaging and Point Cloud Data</h3>In recent years, significant advancements have been made in deep learning techniques for processing 3D point cloud data. This has revolutionised applications in fields such as robotics and autonomous vehicles by allowing for detailed spatial analysis of environments. Techniques exist for various tasks, including 3D object classification, tracking, pose estimation, segmentation, and point cloud completion. Transformative methodologies like Pointr and SnowflakeNet have emerged, highlighting the potential for these technologies to evolve further and impact various domains.<h3 dir="ltr">Predictions for Future Advancements</h3><ul><li dir="ltr">Augmented Reality (AR) Applications: AR's immersive experience can benefit greatly from advancements in machine vision by providing more accurate and real-time interaction with the physical world. This could revolutionise medical imaging, retail, and education by offering more intuitive and interactive ways to engage with information and environments.</li><li dir="ltr">Further Integration with Robotic Process Automation (RPA): Machine vision technology allows robots to perform more complex tasks more efficiently and accurately. Combining machine vision and RPA could lead to significant productivity boosts from manufacturing to logistics by automating tasks that require understanding and interpreting visual data.</li><li dir="ltr">Ease of Use and Accessibility: There's a trend towards making machine vision systems more accessible and easier to use. Efforts are being made to develop "ease-of-use" and "no-code" solutions to democratise the technology, making it more accessible to non-experts.</li><li dir="ltr">Growth in 3D Imaging for Vision Guided Robotics: 3D imaging is no longer emerging but becoming a mature automated imaging offering. From bin picking to product palletizing, these solutions promise to expand the use of VGR by making it more user-friendly and applicable to a broader range of use cases.</li><li dir="ltr">AI and Deep Learning as Automated Inspection Tools: Deep learning has become a valuable tool for segmentation and classification in automated inspection. It enables the identification of images or features within images more intuitively, akin to human perception.</li></ul><h2 dir="ltr">Practical Applications of Machine Vision in Industry</h2>Machine vision technology transforms operations across various sectors, particularly in the automotive and electronics industries, by enhancing product quality and operational efficiency.<h3 dir="ltr">Automotive Industry</h3>Machine vision is essential in inspections and robotic guidance in the automotive sector. Companies like Ford and General Motors (GM) use machine vision to ensure the safety and efficiency of manufacturing processes. Machine vision systems are employed for various applications, including flexible automation, model verification, and quality improvement. The technology offers significant advantages such as adaptable tooling, model verification on flexible lines, and in-station process control for immediate feedback. This enables manufacturers to uphold stringent quality standards and eliminate waste, ultimately contributing to producing a quality product.<h3 dir="ltr">Oil and Gas Industry</h3>Machine vision is critical in inspections and robotic guidance in the oil and gas sector. Companies like Wipro are leveraging machine vision in several cutting-edge ways, including visual inspections of oil rigs, automated quality inspections, and remote monitoring of safety-critical assets. For instance, machine vision enables magnetic crawlers with cameras to inspect difficult-to-reach areas, such as great heights and remote locations within industrial plants. Robots equipped with machine vision can proactively detect internal cracks in gas pipelines, potentially saving millions of dollars by preventing failures. Moreover, these robots can grade the corrosion of pipes based on drone-captured images and videos, facilitating more preventive and cost-effective operations.<h3 dir="ltr">Manufacturing Industry</h3>Machine vision systems are employed for various applications, including flexible automation, model verification, and quality improvement. The technology offers significant advantages such as adaptable tooling, model verification on flexible lines, and in-station process control for immediate feedback. This enables manufacturers to uphold stringent quality standards and eliminate waste, ultimately contributing to producing a quality product.<h3 dir="ltr">Electronics Industry</h3>While the specifics on electronics were not directly covered in the sources, it's evident that the benefits and applications of machine vision in the automotive industry, such as inspection, quality control, and robot guidance, are also pertinent to the electronics sector. High precision and accuracy are essential in electronics manufacturing, where machine vision can identify defects, verify component placement, and ensure product integrity, much like in automotive manufacturing.<h3 dir="ltr">Key Benefits</h3><ul><li dir="ltr">Increased Efficiency: Automation of inspection processes and real-time decision-making capabilities speed up production lines and reduce manual intervention.&nbsp;</li><li dir="ltr">Reduced Error Rates: High-resolution cameras and sophisticated software in modern machine vision systems enable precise defect detection, minimising the risk of errors.</li><li dir="ltr">Enhanced Safety: In the automotive industry, ensuring the proper assembly of over 30,000 parts per vehicle is crucial for driver safety. Machine vision facilitates the flawless fabrication of each component.</li><li dir="ltr">Flexibility and Adaptability:&nbsp;Companies highlight the demand for more flexible automation as a driver for increased use of machine vision. The technology's adaptability allows easy integration into existing production lines, supporting model verification and quality improvements without needing hard-tooled custom fixtures.</li><li dir="ltr">Improved Quality: The ability of machine vision systems to provide feedback for in-station process control plays a vital role in quality assurance. By enabling the detection of defects at the source, these systems help maintain high-quality standards and reduce the cost and complexity of end-of-line repairs.</li></ul>Machine vision technology continues to evolve, offering scalable and versatile solutions that cater to the growing demands of industrial automation.<h2 dir="ltr">Challenges and Considerations</h2>Adopting machine vision technologies has several challenges and considerations that organisations must navigate to leverage their full potential effectively. These challenges include high initial costs, integration complexities, and the need for specialised expertise. By addressing these factors, organisations can better position themselves to take advantage of the numerous benefits that machine vision technologies offer.<h3 dir="ltr">High Initial Costs</h3>Implementing machine vision technologies often requires a significant initial investment. This includes the cost of hardware such as cameras and sensors, software for image processing and analysis, and the infrastructure needed to support these technologies. While the costs of components and engineering have decreased, making the technology more accessible, the initial setup still represents a substantial financial commitment.<h3 dir="ltr">Integration Complexities</h3>Integration involves ensuring compatibility between new and existing systems, configuring machine vision technologies to meet specific operational needs, and optimising performance within the current workflow. This complexity is compounded by the rapid pace of technological advancement, which introduces new standards and capabilities that systems must adapt to.<h3 dir="ltr">Need for Specialized Expertise</h3>Organisations often need professionals skilled in both the technical aspects of machine vision systems and the specific applications within their industry. This expertise is necessary for designing effective solutions, troubleshooting issues, and updating systems to incorporate the latest advancements in the field. The challenge here is finding and retaining talent with the necessary skill set.By proactively addressing these challenges, organisations can harness the power of machine vision technologies to drive innovation and competitive advantage in their operations.<h2 dir="ltr">Conclusion</h2>The transformative impact of machine vision on manufacturing is undeniable. Machine vision has become a cornerstone of modern industrial automation through its ability to automate complex inspection tasks, enhance quality control, and drive efficiencies. Its application spans various sectors, providing unparalleled accuracy and speed in tasks once deemed challenging for automated systems. Machine vision systems minimise downtime, reduce waste, and ensure product quality by enabling more reliable and faster inspections, significantly improving manufacturing.The journey of machine vision from a supportive technology to a pivotal component of industrial automation underscores its transformative impact on manufacturing. As we look to the future, continued innovation, coupled with strong partnerships between technology providers and manufacturers, will be key to unlocking the full potential of machine vision. Embracing these advancements will ensure that industries keep pace with modern production demands and set new standards for efficiency, quality, and innovation.<h2 dir="ltr">References</h2><ol><li dir="ltr"><a href="https://www.automate.org/vision/industry-insights/trends-and-lessons-from-the-2021-machine-vision-market" target="_blank">https://www.automate.org/vision/industry-insights/trends-and-lessons-from-the-2021-machine-vision-market</a></li><li dir="ltr"><a href="https://www.frontiersin.org/articles/10.3389/frobt.2021.627280/full" target="_blank">https://www.frontiersin.org/articles/10.3389/frobt.2021.627280/full</a></li><li dir="ltr"><a href="https://www.cognex.com/what-is/deep-learning/challenges-to-machine-vision" target="_blank">https://www.cognex.com/what-is/deep-learning/challenges-to-machine-vision</a></li><li dir="ltr"><a href="https://www.automate.org/vision/industry-insights/machine-vision-in-the-automotive-industry" target="_blank">https://www.automate.org/vision/industry-insights/machine-vision-in-the-automotive-industry</a></li><li dir="ltr"><a href="https://www.zebra.com/us/en/blog/posts/2021/why-auto-manufacturers-should-take-a-second-look-at-new-machine-vision-systems.html" target="_blank">https://www.zebra.com/us/en/blog/posts/2021/why-auto-manufacturers-should-take-a-second-look-at-new-machine-vision-systems.html</a></li><li dir="ltr"><a href="https://www.automate.org/vision/industry-insights/machine-vision-in-the-automotive-industry" target="_blank">https://www.automate.org/vision/industry-insights/machine-vision-in-the-automotive-industry</a></li><li dir="ltr"><a href="https://www.vision-systems.com/factory/article/14300295/machine-vision-and-imaging-trends-in-automation-a-view-from-the-trenches-for-2024" target="_blank">https://www.vision-systems.com/factory/article/14300295/machine-vision-and-imaging-trends-in-automation-a-view-from-the-trenches-for-2024</a></li><li dir="ltr"><a href="https://news.mit.edu/2019/deep-learning-point-clouds-1021" target="_blank">https://news.mit.edu/2019/deep-learning-point-clouds-1021</a></li><li dir="ltr"><a href="https://pubs.aip.org/aip/acp/article-abstract/2788/1/110005/2903916/An-overview-of-machine-vision-based-quality?redirectedFrom=fulltext" target="_blank">https://pubs.aip.org/aip/acp/article-abstract/2788/1/110005/2903916/An-overview-of-machine-vision-based-quality?redirectedFrom=fulltext</a></li><li dir="ltr"><a href="https://aimagazine.com/articles/machine-vision-enabling-the-factories-of-the-future" target="_blank">https://aimagazine.com/articles/machine-vision-enabling-the-factories-of-the-future</a></li></ol>

Discover the revolutionary impact of machine vision in manufacturing. Learn how integrating AI and machine vision enhances efficiency, quality, and safety in industrial automation, and explore the challenges and future advancements in this cutting-edge technology.

Transforming Manufacturing with Machine Vision Technology

Credit cards embedded chips, national mints printed watermarks, and high-profile locations installed retina scanners all for the same reason—to protect information. As attackers grow smarter, so must defense.<a href="https://engineering.cmu.edu/directory/bios/shen-sheng.html" target="_blank">Sheng Shen</a>, along with collaborators at Penn State University, have developed a pixel-by-pixel approach to visible camouflage with hopes of scaling it for enhanced infrared surveillance, optical security, and anti-forgery protections.“Our collaborators came to us with brochosomes—a ‘magic’ structure leafhoppers produce to create a cloak effect to hide from predators,” said Shen, a professor of <a href="https://www.meche.engineering.cmu.edu/index.html" rel="noopener" target="_blank">mechanical engineeringOpens in new window</a>. “We wanted to understand brochosomes’ optical limitations to see what more we could do with them.”Brochosomes are 3D soccer ball-like objects with nanoscale cavities that internally absorb light rather than reflect it onto outside structures. In nature, biologists suspect that this allows leafhoppers to blend in with their background.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1710755534602-0301-em-mirco-mirage-3.png" style="width: 100%px;" class="fr-fic fr-dib">To test functionality, the team simulated two different versions of the structure, one with cavities for light absorption and one without.“There is a fundamental law in physics that if a structure is a good absorber of energy, it can emit an equal amount of energy,” explained Zhuo Li, Ph.D. candidate at Carnegie Mellon. “We quickly realized that if we put both structures together, one would emit more energy than the other. That would make one appear brighter to an infrared camera than the other.”And so began the development of the world’s smallest QR code.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1710755551949-0301-em-micro-mirage-1.png" style="width: 100%px;" class="fr-fic fr-dib">Using an advanced 3D printing technique, developed by the Penn State collaborators, the team controlled whether each pixel was printed as a structure with or without holes, allowing them to fabricate a QR code readable by infrared camera alone.Sizing up at less than two percent of an inch, the code is only visible under a microscope but both teams plan to explore ways to scale it for commercial use too.<iframe width="640" height="360" src="https://www.youtube.com/embed/x54k-rT5OAA?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;&nbsp;</iframe>“With this technology we are ultimately distorting an objects’ thermal signature,” said Li. “We have the power to disguise how objects are displayed on an infrared camera. Hypothetically, if we laid the brochosome pixels accordingly, we could paint a patrol car to appear as a delivery van to infrared security.”The team’s unique combination of visible camouflage and infrared display creates new opportunities for data encryption and optical security.<blockquote>We’ve taken infrared light and turned it from an energy carrier to an information carrier.<cite>Shen Sheng, Professor, Mechanical Engineering</cite></blockquote>“This is just the start of a new research area my team can explore,” said Sheng. “We’ve taken infrared light and turned it from an energy carrier to an information carrier.”This research was done in collaboration with The Pennsylvania State University researchers Dr. Lin Wang and Professor Tak-Sing Wong. It was first published in <a href="https://www.science.org/doi/10.1126/sciadv.adl4027" rel="noopener" target="_blank">Science AdvancesOpens in new window</a>. Zhuo Li and Lin Wang contributed equally to this work. The authors acknowledge support from the Department of Defense Office of Naval Research.

Researchers fabricate the world’s smallest QR code that, invisible to the naked eye, encrypts information to only be visible by an infrared camera lens.

Micro mirage: the infrared information carrier

An AI-generated depiction of a modern day chipset

Gaining a comprehensive understanding of the distinct capabilities and applications of TPUs and GPUs is crucial for developers and researchers aiming to navigate the complex and rapidly changing terrain of artificial intelligence.

TPU vs GPU in AI: A Comprehensive Guide to Their Roles and Impact on Artificial Intelligence

Smart retail — those retail outlets that use data and automation for better decision-making and store operations — can provide retail managers with insights and help them unlock efficiencies that enable better customer service and improve revenue potential.&nbsp;Smart&nbsp;retail uses data and technology to upgrade customer service and achieve operational excellence in supplier and logistics, assortment and category management, store layout and workforce optimization, customer loyalty and marketing, and selling.&nbsp;&nbsp;<h2>What’s the Difference Between Smart Retail and Traditional Retail?&nbsp;&nbsp;</h2>Retailing is still a labor-intensive industry ripe with opportunities for optimizing the path from manufacturers of a product to that product being purchased by a consumer from a retailer. Worldwide, retailers lose nearly $450 billion in revenue annually because of out-of-stock items, empty shelves, and other in-store inconsistencies.&nbsp;Brick-and-mortar stores have always struggled to find, train, motivate,&nbsp;and schedule personnel to perform a variety of labor-intensive, repetitive tasks of keeping the shelves stocked and forecasting customer demand to keep inventory orders smoothly flowing. This challenge has been made worse by the negative effects of COVID-19 on the labor market and the supply chain.&nbsp;Stocking shelves on time (to avoid out-of-stock situations and sales losses) and keeping price labels accurate is not easy. With around 50,000 stock-keeping unit (SKU) labels in an average-sized&nbsp;hypermarket, the task of efficient inventory availability in traditional&nbsp;retail&nbsp;is typically tracked by cash register checkouts. Price accuracy and inventory solutions mostly rely on manual price audits by store staff.&nbsp;Retailers&nbsp;have been experimenting with some degree of automation and digitization for some time. Some have implemented electronic shelf labels (ESLs) and cameras that monitor store on-shelf inventory, with various success rates, as functionality issues (battery lifespan, tedious device replacement) and vendor lock situations persist.&nbsp;&nbsp;New advances in AI, neural networks, and edge computing made available recently are powering experimentation with autonomous and semi-autonomous robots that promise to make retail even smarter. By helping employees be more efficient and reducing the time for inventory and price accuracy checks by 2/3, retailers can collect digitized data, analyze it with AI with unprecedented accuracy, and apply data analytics for improved insights.&nbsp;<h2>What Are Some Benefits of Smart&nbsp;Retail?&nbsp;</h2>Smart&nbsp;retailers&nbsp;don’t just replace manual processes with automation but implement electronic inventory files, digital price labels, advanced camera solutions, and inventory forecasting software to replace analog store management heuristics and tools. Digital tools are poised to help shift from a regular store to a smart store&nbsp;— one that not only collects siloed data but also analyzes it in near real-time to provide valuable, timely insights.&nbsp;Smart retail technology benefits not only store owners but customers, and potentially even suppliers in the following ways:&nbsp;&nbsp;<h3>Smart&nbsp;Retail&nbsp;Benefits for Store Owners and Management&nbsp;</h3><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1705482602189-ERIS-v2-1-backless-1-e1678797861855-486x1024.png" style="width: 300px;" class="fr-fic fr-dib"> The biggest three operational efficiency wins for the store owners, and management are:1. real-time actionable&nbsp;inventory&nbsp;data;2. significantly improved&nbsp;price&nbsp;label accuracy; and3. better&nbsp;planogram&nbsp;compliance.&nbsp;While most&nbsp;retail&nbsp;inventory forecasting software relies on product checkout data provided by the cash registers, such data will not be available if the product is not on the shelf. Store cameras can help monitor inventory, but only the advanced camera systems combined with powerful AI that can identify store items with 99+% accuracy will be useful. Thankfully, solutions like that have been recently introduced on the market.&nbsp;With up to 200 price labels per square meter on a typical&nbsp;shelf, checking and verifying price labels for accuracy is tedious and error-prone. Luckily, this is just the perfect task for smart automation, powered by the same advanced cameras and AI that help with inventory detection.&nbsp;Even more so, that same hardware can detect SKU shelf placement inconsistencies and direct human operators to make the needed corrections on the spot.&nbsp;To be fair, training and reimagining business processes around newly possible operational tasks do require an upfront investment. Still, the benefits are felt immediately, yet most smart automation projects deliver a positive ROI in twelve months or less. An additional benefit is a reduced likelihood of missed scheduled price and inventory checks, an important consideration given that most retailers can increase revenue by up to 10% with optimal operations efficiency, assuming suppliers can resolve inventory bottlenecks. &nbsp;&nbsp;&nbsp;<h3>Smart&nbsp;Retail&nbsp;Benefits for Suppliers&nbsp;&nbsp;</h3>Retail&nbsp;suppliers potentially have much to benefit from more accurate data on promotion-related sales figures and promotional items being placed appropriately, as they usually have to send representatives to gather some of this data in-store. Automating planogram and inventory status updates have potential, but so far, data sharing is still a contentious issue.&nbsp;Increasing operational efficiency at the&nbsp;store&nbsp;level can also help optimize deliveries and share quantity, frequency, and forecasted demand based on real-time inventory data thanks to decision-making powered by AI algorithms that provide more accurate forecasting and actionable insights.&nbsp;&nbsp;<h3>Smart&nbsp;Retail&nbsp;Benefits for Store Staff&nbsp;&nbsp;</h3>Not to be overlooked are the benefits to the employees at retail stores that deploy AI and robotics to automate previously tedious, time-consuming tasks. Using smart detection of price, inventory, and planogram issues allows store employees to refocus on higher-value tasks that require knowledge and talent. By taking care of drudge work with automation, performing required operational tasks faster, and removing error-prone business process steps such as converting paper notes into digital reports, smart retail stores can allow staff to focus on interacting with customers for a better in-store experience.&nbsp;<h3>Smart Retail Benefits for Customers&nbsp;&nbsp;</h3>While operational improvements (on-shelf inventory availability, price accuracy, planogram compliance) mostly benefit the store owners and managers, bringing retail closer to the goal of consistent, high-quality service is also more feasible with automation. While AI-enabled robotics assist staff on the shop floor, more time and resources will now be available for customers needing attention than without intelligent automation. Robots are better at tedious tasks, and varied, unpredictable customer service work is still better done by human staff.&nbsp;<h2>Smart Retail Operations with ERIS</h2>Supporting efficient, intelligent retail operations practices, ERIS – the semi-autonomous retail robot - delivers more accurate and faster&nbsp;in-store inventory and price accuracy reviews.ERIS is powered by an advanced AI platform, allowing edge computing for intelligent image processing, image recognition, store staff messages, and parallel processing. The semi-autonomous robot can deliver high AI computed performance right when needed to achieve real-time processing and aggregation of large volumes of data.The core computation tool in ERIS allows the semi-autonomous retail robot to process high-resolution images captured on high-performance, industrial-grade camera modules on the device and run image recognition algorithms over neural networks. This edge computing tool, consisting essentially of a series of mini-computers with CPUs and GPUs, enables parallel processing that’s crucial to run efficiently.ERIS runs four depth (3D) cameras and four RGB cameras. With the six modules on board, it can run models for multiple concurrent tasks like image inference, displaying actionable information, for the operator, on the touchscreen, and operating the price label printer. This ultimately provides accurate, real-time snapshots of the store shelves no trained human operator could ever hope to achieve at the speed and the level of detail ERIS can.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1705482619530-Panorama-Shelf-x-edit-768x413.png" style="width: 300px;" class="fr-fic fr-dib">Real-time panorama shelf scan generated by ERIS<h2 id="isPasted">How Can I Make My&nbsp;Brink-and-Mortar Store&nbsp;Smart?&nbsp;&nbsp;</h2>What are the steps supermarkets need to take to start using ERIS? Preparation, time to install, training, best practices?&nbsp;Running a smart store (for example, a hypermarket) requires an entire ecosystem of hardware and software solutions working harmoniously with existing store technology infrastructure and operating processes. To accelerate and improve performance, every application, device, sensor, and AI model in the system must share data and insights across the store and with its network partners.&nbsp;That said, integrating a solution like ERIS takes from one to three months. Shifting the approach to price and inventory management will surely make some people reluctant, but when performance and operational excellence are a priority, those will be the environments that will welcome the coming generation of smart supermarkets.

Smart retail — those retail outlets that use data and automation for better decision-making and store operations — can provide retail managers with insights and help them unlock efficiencies that enable better customer service and improve revenue potential.

What Is Smart Retail?

<h1 dir="ltr" id="isPasted">Introduction</h1>Artificial intelligence (AI) is experiencing a significant transition with the increasing adoption of edge computing, where data processing occurs closer to the data source. The aim is to enable real-time decision-making and reduce dependency on cloud services.&nbsp;This transition is driven by advanced microprocessors, with the Texas Instruments AM68A being a notable example. It offers a combination of efficiency, power, and adaptability, making it well-suited for the requirements of Edge AI applications.&nbsp;In this article, we uncover the capabilities and implications of the Texas Instruments AM68A within the Edge AI landscape. Our exploration will dive into the intricacies of its technological features and spotlight its integration with the&nbsp;<a href="https://edgeimpulse.com/" target="_blank">Edge Impulse</a> platform. We'll also feature some case studies that demonstrate the practical application of the AM68A in Edge AI projects focused on computer vision. <iframe src="https://c8056756.sibforms.com/serve/MUIFAMyA6BGksOUPEokHuBnEmZRHhYgJ4RXb4BV0gvzN98ymNIvQdTcBFQnRVfviT6yZSFW_R2g1hENBu7p7Afnyv9Kt0unAAbfFU5mMQhBwI-8hQnKf2r1mmRKhUgDLmtk8bpVSZ7kZGpleCPqtYRv8_kkvODL7F4neGLNGgdpFDm9pPzoYpwbQkc0BU5GVOFoq8CSmD0DDvDjG" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block; margin-left: auto; margin-right: auto; max-width: 100%; width: 766px; height: 603px;" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><h1 dir="ltr"> </h1><h1 dir="ltr">Overview of the Texas Instruments AM68A</h1>The Texas Instruments AM68A is a microprocessor that delivers great performance and features dedicated AI hardware acceleration. This enhancement is crucial for optimizing AI tasks directly on the device, enhancing efficiency and processing speed.&nbsp;Before exploring its diverse uses and influence in Edge AI, it's essential to understand its core specifications and features. This groundwork will provide a clear perspective on the microprocessor's capabilities and how it can be utilized in various Edge AI contexts.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzA0ODg1MDY3ODI1LWltYWdlMi5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 302px;" class="fr-fic fr-dib" alt="Texas-Instruments-TI-AM68A">Fig. 1: Texas Instruments AM68A development platformHere are some core specifications and features of the AM68A:<div dir="ltr" align="left"><table><tbody><tr><td>Feature</td><td>Description</td></tr><tr><td>Processor Core</td><td>Dual 64-bit ARM Cortex-A72 microprocessor subsystems, up to 2 GHz, with a shared 1MB L2 cache.</td></tr><tr><td>Deep Learning Accelerators</td><td>Deep Learning Accelerator capable of up to 8 TOPS, for efficient AI computations and real-time data processing.</td></tr><tr><td>Vision Processing</td><td>Vision Processing Accelerators, including an Image Signal Processor and multiple vision assist accelerators for handling image and video data.</td></tr><tr><td>Additional Processing Units</td><td>Dual-core ARM Cortex-R5F MCUs for general compute operations and device management.</td></tr><tr><td>Graphics Processing</td><td>IMG BXS-4-64, a high-performance 3D GPU supporting OpenGL ES 3.1 and Vulkan 1.2 for graphic-intensive applications.</td></tr><tr><td>Connectivity Options</td><td>Gigabit ethernet, USB 3.0, CSI 2.0, PCIe gen 3, and various memory interfaces.</td></tr><tr><td>Security Features</td><td>Secure boot and crypto hardware accelerators for robust security.</td></tr></tbody></table></div>Table 1: Specifications and features of Texas Instruments AM68A<h1 dir="ltr">The Role of Deep Learning Accelerators in Edge AI</h1>The Texas Instruments AM68A has deep learning accelerators, which play a crucial role in Edge AI and computer vision applications. These are specialized hardware components designed to efficiently execute the computations required for deep learning algorithms. Unlike general-purpose processors, these accelerators are optimized for specific tasks like matrix multiplications and convolutions, which are fundamental in neural network operations.The significance of deep learning accelerators in edge computing lies in their ability to bring advanced AI functionalities to devices operating at the edge of networks. By handling complex computations locally, these accelerators reduce the need to transmit large amounts of data to the cloud for processing. This cuts down on latency, making real-time processing feasible, and addresses concerns like bandwidth limitations and data privacy.<h1 dir="ltr">Real-World Applications and Case Studies</h1>The Texas Instruments AM68A has been central to a variety of computer vision projects, demonstrating its versatility and capability. These case studies showcase the practical use of the AM68A in diverse Edge AI applications.<h2 dir="ltr">American Sign Language (ASL) Translation</h2>One of the remarkable projects developed using a Texas Instruments board is an American Sign Language (ASL) Translation system. This application utilizes the AM62A, the entry-level alternative to the AM68A, and is designed to translate ASL into text in real-time. By leveraging machine learning algorithms and image processing capabilities, this system enhances communication accessibility for the deaf and hard-of-hearing community.&nbsp;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzA0ODg1MDk3MjkxLWltYWdlNC5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="sign-language">Fig. 2: Training the model to recognize different signsThe project's success on the AM62A highlights the potential advancements that could be achieved with the more powerful AM68A.&nbsp;Read more about this project:&nbsp;<a href="https://docs.edgeimpulse.com/experts/image-projects/asl-byom-ti-am62a" target="_blank">Import a Pretrained Model with "Bring Your Own Model" - Texas Instruments AM62A</a><h2 dir="ltr">Motorcycle Helmet Detection</h2>Another innovative application is the Motorcycle Helmet Detection project. Also developed on the Texas Instruments AM62A, this system utilizes machine learning and computer vision to detect whether motorcyclists are wearing helmets. This application is crucial for enhancing road safety and can be deployed at traffic signals or highway checkpoints.&nbsp;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzA0ODg1MTIwMzY2LWltYWdlNi5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="helmet-detection-edge-ai">Fig. 3: Testing the motorcycle helmet detection system prototypeThe implementation with the AM62A demonstrates the feasibility of such safety applications, which can be further optimized with the AM68A's enhanced capabilities.&nbsp;Explore this project in detail:&nbsp;<a href="https://docs.edgeimpulse.com/experts/image-projects/motorcycle-helmet-detection-smart-light-ti-am62a" target="_blank">Motorcycle Helmet Identification and Traffic Light Control - Texas Instruments AM62A</a><h2 dir="ltr">Posture Detection</h2>The Texas Instruments TDA4VM board, another counterpart of the AM68A with fewer features, has been used in a posture detection project. This application focuses on identifying incorrect postures in real-time to prevent workplace injuries. Using advanced image processing and AI, the system provides immediate feedback for posture correction.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzA0ODg1MTQxOTU2LWltYWdlNS5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="posture-detection-edge-ai">Fig. 4: Texas Instruments AM62A is used to efficiently distinguish between correct and incorrect postures, promoting ergonomic practices and preventing injuries.The AM68A, with its superior processing power, could elevate such health and safety applications to new levels.&nbsp;Learn more about this project:&nbsp;<a href="https://docs.edgeimpulse.com/experts/image-projects/ti-tda4vm-posture-detection" target="_blank">Correct Posture Detection and Enforcement - Texas Instruments TDA4VM</a><h2 dir="ltr">Smart Factory Prototype</h2>Finally, the Smart Factory Prototype project, developed on the Texas Instruments TDA4VM, illustrates the potential of these boards in industrial applications. This system employs AI to monitor manufacturing processes, enhancing efficiency and predicting maintenance needs. The project demonstrates how edge computing can revolutionize industrial operations, and with the advanced features of the AM68A, these applications could achieve even greater efficiency and accuracy.&nbsp;Read more about the Smart Factory Prototype:&nbsp;<a href="https://docs.edgeimpulse.com/experts/image-projects/smart-factory-prototype-ti-tda4vm" target="_blank">Smart Factory Prototype - Texas Instruments TDA4VM</a><h2 dir="ltr">Transitioning to the AM68A</h2>Each of these projects, developed on the AM62A and TDA4VM boards, lays the groundwork for what could be achieved with the AM68A. With its enhanced processing power and deep learning capabilities, the AM68A is poised to elevate these applications, offering more efficient processing, quicker response times, and the ability to handle more complex AI models. This transition represents a significant step forward in the capabilities of edge computing devices and their applications across various industries.<h1 dir="ltr">Conclusion</h1>The AM68A, with its advanced deep learning accelerators and enhanced processing capabilities, stands as a significant player in the rapidly evolving landscape of edge computing. Its integration with the Edge Impulse platform has opened new avenues for AI model development, making advanced AI functionalities more accessible and effective.Throughout the article, we've seen how the AM68A is being utilized in various sectors, from enhancing road safety with motorcycle helmet detection to improving workplace ergonomics through posture detection. These applications, initially developed on the AM62A and TDA4VM, offer a glimpse into the possibilities with the more advanced AM68A. The chip's potential in driving innovations in smart factories, healthcare monitoring, environmental analysis, and beyond, underscores its versatility and efficiency in handling diverse Edge AI applications.<h1 dir="ltr">About the sponsor: Edge Impulse</h1>Edge Impulse is the leading development platform for embedded machine learning, used by over 1,000 enterprises across 200,000 ML projects worldwide. We are on a mission to enable the ultimate development experience for machine learning on embedded devices for sensors, audio, and computer vision, at scale.&nbsp;From getting started in under five minutes to MLOps in production, we enable highly optimized ML deployable to a wide range of hardware from MCUs to CPUs, to custom AI accelerators. With Edge Impulse, developers, engineers, and domain experts solve real problems using machine learning in embedded solutions, speeding up development time from years to weeks. We specialize in industrial and professional applications including predictive maintenance, anomaly detection, human health, wearables, and more.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzA0ODg1MTczMTQ0LWltYWdlMS5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="edge-impulse"><iframe width="540" height="600" src="https://c8056756.sibforms.com/serve/MUIFAMyA6BGksOUPEokHuBnEmZRHhYgJ4RXb4BV0gvzN98ymNIvQdTcBFQnRVfviT6yZSFW_R2g1hENBu7p7Afnyv9Kt0unAAbfFU5mMQhBwI-8hQnKf2r1mmRKhUgDLmtk8bpVSZ7kZGpleCPqtYRv8_kkvODL7F4neGLNGgdpFDm9pPzoYpwbQkc0BU5GVOFoq8CSmD0DDvDjG" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>

High-performance microprocessors with AI hardware acceleration are revolutionizing Edge AI, enabling advanced computer vision capabilities directly on devices for real-time data processing and analysis.

Building Robust Edge AI Computer Vision Applications with High-Performance Microprocessors

Partner:<a href="https://intelligentrobotics.com.au/" target="_blank">Intelligent Robotics Pty Ltd</a>&nbsp;Challenge:Prior to the IR-SCRIBE automation, Beef Scribing was a skilled manual task which required the use of a hazardous circular cutting saw. This operation involved the risk of operator injuries and inaccurate cutting of the high-value beef carcass.Intelligent Robotics Pty Ltd, with the support of AMPC’s investment into advanced manufacturing, developed an AI-driven automated beef scribing system using Photoneo 3D cameras MotionCam-3D, called IR-SCRIBE, being installed at Kilcoy Pastoral Company’s processing plant in Kilcoy, Queensland Australia.&nbsp;Solution:<a href="https://www.photoneo.com/motioncam-3d/" target="_blank">MotionCam-3D</a><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1702554524331-Beef+scribing+by+Intelligent+Robotics+and+Photoneo.jpg" style="width: 300px;" class="fr-fic fr-dib">Before cutting begins, each side of beef passes before a series of Photoneo MotionCam-3D imaging cameras. MotionCam-3D was selected to capture highly accurate 3D point cloud images even during the movement of the beef carcasses on a conveyor and the varying size and shape of the beef sides.Vision processing and AI then identify the key features of interest on the carcass, which define where the robots need to make their incisions.The first robot performs the two spine cuts, before the cameras measure again before the second cuts are made to maintain millimeter-accuracy.The IR-SCRIBE robotic scribing uses Photoneo 3D cameras and AI to find where each cut needs to be made, which is critical for boning room success. This gives&nbsp;downstream process workers&nbsp;the right pattern to follow in order to optimize carcass yields.The end customer said: “It will give us the accuracy we need from a yield perspective, product specifications, and product consistency. Our customers will open a meat carton and notice the difference.”&nbsp;Discussion:MotionCam-3D allowed the end customer to scan carcasses in movement and with millimeter precision, which was a request in order to keep the line speeds and improve the accuracy of the cuts.The IR-SCRIBE system using Photoneo MotionCam-3D cameras provides significant cost savings when compared to alternative X Ray based systems, and results in high accuracy as well as high yield recovery from the beef carcass.&nbsp;The benefits include improving yields, greater consistency in scribing operations, and most importantly – assisting worker safety.&nbsp;Intelligent Robotics Pty Ltd’s electrical engineering manager Jonathan Cook said his company is passionate about engineering novel solutions to add value and improve safety.Would you like to discuss your project with Photoneo? <a href="https://www.photoneo.com/contact-us/" target="_blank" id="isPasted">Contact them here</a>.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1702554636283-IR-SCRIBE+with+logo+%281%29.jpg" style="width: 300px;" class="fr-fic fr-dib">

Novel technology for automated beef carcass scribing

This one-of-a-kind technology developed by Intelligent Robotics uses AI and Photoneo MotionCam-3D cameras to let robots fully automate the process of beef carcass scribing.

Partner:<a href="https://erasystems.ro/" target="_blank">Era Systems</a>Challenge:The customer, an automotive manufacturer, needed an automation solution for the palletization and depalletization of large ABS parts. They decided on a solution where one vision-guided robot operates both processes.Solution:<a href="https://www.photoneo.com/locator-studio/" target="_blank">Locator Studio</a> + <a href="https://www.photoneo.com/phoxi-3d-scanner/" target="_blank">PhoXi 3D Scanner</a>The solution for palletization and depalletization consists of 2 different picking locations with 2 independent production flows:A) Palletization – the ABS parts are taken from an output conveyor and placed on a palletB) Depalletization – the parts are picked from the palletThe FANUC R-2000iC/210F Robot, navigated by Photoneo PhoXi 3D Scanner and Locator Studio, serves both locations. First, it picks the ABS parts from an output conveyor of a thermoformer press and stacks them on a pallet. The pallet is then taken to a cooling area. The height of the full pallet is ~1400-1500 mm from the ground.The cooled parts are then picked from the pallet and fed to a CNC machine for trimming the outer border (the excess material). After trimming, the parts are placed on another pallet.Any of the 2 jobs can be active or inactive at a certain production stage (a few hours or days).The robot can either work in location 1 (the conveyor) or in location 2 (the pallet) or in both locations at the same time. So one robot can serve 2 different robotic cells.Both robotic cells can process 4 part types but only one model during a production cycle.For instance, Heat RT on the conveyor for 7 hours and Heat FT on the pallet for 3 days, then Heat FT – Top/Lid on the conveyor for 2 days and Heat RT on the pallet for 14 hours, and so on.Every time a part model is changed in a location, the operator activates the correct model for that location so the robot knows what to look for.Discussion:This automation solution increased efficiency as one robot can operate two different production flows simultaneously.The robot is navigated by state-of-the-art 3D vision from Photoneo, providing a resolution of up to 3 Million 3D points and an accuracy of 25 – 500 µm across the different models.Would you like to discuss your project with Photoneo?&nbsp;<a href="https://www.photoneo.com/contact-us/" rel="nofollow" target="_blank" id="isPasted">Contact them here</a>.

Palletization and depalletization of large ABS parts

The customer, an automotive manufacturer, needed an automation solution for the palletization and depalletization of large ABS parts. They decided on a solution where one vision-guided robot operates both processes.

Imagine you’re visiting a friend abroad, and you look inside their fridge to see what would make for a great breakfast. Many of the items initially appear foreign to you, with each one encased in unfamiliar packaging and containers. Despite these visual distinctions, you begin to understand what each one is used for and pick them up as needed.Inspired by humans' ability to handle unfamiliar objects, a group from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) designed Feature Fields for Robotic Manipulation (<a href="https://arxiv.org/abs/2308.07931" target="_blank">F3RM</a>), a system that blends 2D images with foundation model features into 3D scenes to help robots identify and grasp nearby items. F3RM can interpret open-ended language prompts from humans, making the method helpful in real-world environments that contain thousands of objects, like warehouses and households.F3RM offers robots the ability to interpret open-ended text prompts using natural language, helping the machines manipulate objects. As a result, the machines can understand less-specific requests from humans and still complete the desired task. For example, if a user asks the robot to “pick up a tall mug,” the robot can locate and grab the item that best fits that description.“Making robots that can actually generalize in the real world is incredibly hard,” says Ge Yang, postdoc at the National Science Foundation AI Institute for Artificial Intelligence and Fundamental Interactions and MIT CSAIL. “We really want to figure out how to do that, so with this project, we try to push for an aggressive level of generalization, from just three or four objects to anything we find in MIT’s Stata Center. We wanted to learn how to make robots as flexible as ourselves, since we can grasp and place objects even though we've never seen them before.”<h2>Learning “what’s where by looking”</h2>The method could assist robots with picking items in large fulfillment centers with inevitable clutter and unpredictability. In these warehouses, robots are often given a description of the inventory that they're required to identify. The robots must match the text provided to an object, regardless of variations in packaging, so that customers’ orders are shipped correctly. For example, the fulfillment centers of major online retailers can contain millions of items, many of which a robot will have never encountered before. To operate at such a scale, robots need to understand the geometry and semantics of different items, with some being in tight spaces. With F3RM’s advanced spatial and semantic perception abilities, a robot could become more effective at locating an object, placing it in a bin, and then sending it along for packaging. Ultimately, this would help factory workers ship customers’ orders more efficiently.“One thing that often surprises people with F3RM is that the same system also works on a room and building scale, and can be used to build simulation environments for robot learning and large maps,” says Yang. “But before we scale up this work further, we want to first make this system work really fast. This way, we can use this type of representation for more dynamic robotic control tasks, hopefully in real-time, so that robots that handle more dynamic tasks can use it for perception.”The MIT team notes that F3RM’s ability to understand different scenes could make it useful in urban and household environments. For example, the approach could help personalized robots identify and pick up specific items. The system aids robots in grasping their surroundings — both physically and perceptively.“Visual perception was defined by David Marr as the problem of knowing ‘what is where by looking,’” says senior author Phillip Isola, MIT associate professor of electrical engineering and computer science and CSAIL principal investigator. “Recent foundation models have gotten really good at knowing what they are looking at; they can recognize thousands of object categories and provide detailed text descriptions of images. At the same time, radiance fields have gotten really good at representing where stuff is in a scene. The combination of these two approaches can create a representation of what is where in 3D, and what our work shows is that this combination is especially useful for robotic tasks, which require manipulating objects in 3D.”<h2>Creating a “digital twin”</h2>F3RM begins to understand its surroundings by taking pictures on a selfie stick. The mounted camera snaps 50 images at different poses, enabling it to build a <a href="https://www.matthewtancik.com/nerf" target="_blank">neural radiance field</a> (NeRF), a deep learning method that takes 2D images to construct a 3D scene. This collage of RGB photos creates a “digital twin” of its surroundings in the form of a 360-degree representation of what’s nearby.In addition to a highly detailed neural radiance field, F3RM also builds a feature field to augment geometry with semantic information. The system uses&nbsp;<a href="https://openai.com/research/clip" target="_blank">CLIP</a>, a vision foundation model trained on hundreds of millions of images to efficiently learn visual concepts. By reconstructing the 2D CLIP features for the images taken by the selfie stick, F3RM effectively lifts the 2D features into a 3D representation.<h2>Keeping things open-ended</h2>After receiving a few demonstrations, the robot applies what it knows about geometry and semantics to grasp objects it has never encountered before. Once a user submits a text query, the robot searches through the space of possible grasps to identify those most likely to succeed in picking up the object requested by the user. Each potential option is scored based on its relevance to the prompt, similarity to the demonstrations the robot has been trained on, and if it causes any collisions. The highest-scored grasp is then chosen and executed. To demonstrate the system’s ability to interpret open-ended requests from humans, the researchers prompted the robot to pick up Baymax, a character from Disney’s “Big Hero 6.” While F3RM had never been directly trained to pick up a toy of the cartoon superhero, the robot used its spatial awareness and vision-language features from the foundation models to decide which object to grasp and how to pick it up. F3RM also enables users to specify which object they want the robot to handle at different levels of linguistic detail. For example, if there is a metal mug and a glass mug, the user can ask the robot for the “glass mug.” If the bot sees two glass mugs and one of them is filled with coffee and the other with juice, the user can ask for the “glass mug with coffee.” The foundation model features embedded within the feature field enable this level of open-ended understanding.“If I showed a person how to pick up a mug by the lip, they could easily transfer that knowledge to pick up objects with similar geometries such as bowls, measuring beakers, or even rolls of tape. For robots, achieving this level of adaptability has been quite challenging,” says MIT PhD student, CSAIL affiliate, and co-lead author William Shen. “F3RM combines geometric understanding with semantics from foundation models trained on internet-scale data to enable this level of aggressive generalization from just a small number of demonstrations.” Shen and Yang wrote the paper under the supervision of Isola, with MIT professor and CSAIL principal investigator Leslie Pack Kaelbling and undergraduate students Alan Yu and Jansen Wong as co-authors. The team was supported, in part, by Amazon.com Services, the National Science Foundation, the Air Force Office of Scientific Research, the Office of Naval Research’s Multidisciplinary University Initiative, the Army Research Office, the MIT-IBM Watson AI Lab, and the MIT Quest for Intelligence. Their work will be presented at the 2023 Conference on Robot Learning.

Feature Fields for Robotic Manipulation (F3RM) enables robots to interpret open-ended text prompts using natural language, helping the machines manipulate unfamiliar objects. The system’s 3D feature fields could be helpful in environments that contain thousands of objects, such as warehouses. Images courtesy of the researchers.

By blending 2D images with foundation models to build 3D feature fields, a new MIT method helps robots understand and manipulate nearby objects with open-ended language prompts.

Using language to give robots a better grasp of an open-ended world

Training robust AI models can be challenging when using crowdsourced or publicly available datasets. The most useful neural networks are ones that replace expert knowledge to solve real problems. One example would be a visual inspection model for the factory floor.<h2>Human level performance</h2>Human performance comes into tiers: general or expert. An expert is someone who has spent a minimum amount of time or has received practical training in the field.The accuracy of your AI model is directly correlated to the quality of data used to train it. When using general datasets for training, the model usually surpasses the knowledge level embedded in the training samples, but it remains well below expert-level performance. A computer is more attentive than an operator performing the same inspection task, so the model trained will probably reach an accuracy that is marginally better than the human labelled dataset.If you think it's a good idea to use services like Amazon Mechanical Turk to label your dataset, let me ask you a question. Would you take a person and after reading a quick instruction manual get them to work inside the factory?That does not sound like a great idea. But it turns out that is what every one of us is doing when leveraging services like these. The bulk of your dataset can be generated in this manner, but is it helping your model?<h2>Practical considerations</h2>Tricky samples may not make it to the dataset because someone was not able to understand your labeling instructions, and that sample may get discarded. While human-level performance is achievable with large crowd labeled datasets, the truth is that the easiest samples will be labeled first.I had seen numerous occasions in which turks were simply targeting the lowest hanging fruit by submitting no labels for the empty samples. In that case, it is a first come first rewarded economy. The difficult samples may end up unlabeled.<h2>Model hardening</h2>Synthetic data allows the possibility to make the task at hand harder than it will be in real life.Variance or noise is good for your model, while generating samples programmatically can make the model more resilient to unseen inputs. Human labeled data is often curated and less noisy than the real inputs.This may work out, in the beginning, to quickly get started with your model. But believe me, you don't want a crowd trained model to be put in production across a factory floor without verifying that its performance surpasses expert level or that it gets as closely as possible to those expectations.<h2>Methods for generating data</h2>There are several methods you can leverage when generating synthetic data.<h3>Simulator data</h3>Use a simulator to generate new data. Custom shaders can provide RGB+D and segmentation labels. Microsoft is now focusing on aerospace-grade data generation because it is open source, AirSim 3D simulator project obtained promising results hence a lot of companies were using it for internal purposes.<iframe width="640" height="360" src="https://www.youtube.com/embed/b59sxe-M-bg?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><h3 id="isPasted">Synthetic augmentation</h3>Simple computer vision enhancements can work as well. Using the OpenCVs inpaint function the price was rewritten as per the examples shown bellow. A mixed approach was implemented here, having the human labelers define the polygon in which the price was contained.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1697620602389-synthetic_inpaint_example-768x376.jpg" style="width: 766px;" class="fr-fic fr-dib"><h2 id="isPasted">Development focus</h2>Validating crowd samples is going to take a similar amount of effort as developing a dataset generator.This is the most essential part of AI and a lot of people get it wrong. If the dataset is poorly curated and not enough time is spent at this stage, your project may end up failing. The dataset should be one of your main priorities, and if you have enough quality data, you can train almost every model to perfection.<h2>Conclusion</h2>The most successful models deployed in production use a mix of the synthetic and real data to achieve great results.<a href="https://www.linkedin.com/company/adapta-robotics" target="_blank">Let us know</a> what other techniques you are using to give your model that extra edge in performance.

Training robust AI models can be challenging when using crowdsourced or publicly available datasets. The most useful neural networks are ones that replace expert knowledge to solve real problems. One example would be a visual inspection model for the factory floor.

Synthetic datasets-Or why you can't train robust models with generic data

This article originally appeared in <a href="https://news.mit.edu/2023/new-tools-available-reduce-energy-that-ai-models-devour-1005" target="_blank" rel="noopener noreferrer">MIT News</a> and has been republished with permission.<hr>When searching for flights on Google, you may have noticed that each flight's carbon-emission estimate is now presented next to its cost. It's a way to inform customers about their environmental impact, and to let them factor this information into their decision-making.A similar kind of transparency doesn't yet exist for the computing industry, despite its carbon emissions <a href="https://www.climatiq.io/blog/measure-greenhouse-gas-emissions-carbon-data-centres-cloud-computing" target="_blank">exceeding</a> those of the entire airline industry. Escalating this energy demand are artificial intelligence models. Huge, popular models like ChatGPT signal a trend of large-scale artificial intelligence, boosting forecasts that predict data centers will draw up to 21 percent of the world's electricity supply by 2030.The MIT <a href="https://www.ll.mit.edu/r-d/cyber-security-and-information-sciences/lincoln-laboratory-supercomputing-center" target="_blank">Lincoln Laboratory Supercomputing Center</a> (LLSC) is developing techniques to help data centers reel in energy use. Their techniques range from simple but effective changes, like power-capping hardware, to adopting novel tools that can stop AI training early on. Crucially, they have found that these techniques have a minimal impact on model performance.In the wider picture, their work is mobilizing green-computing research and promoting a culture of transparency. "Energy-aware computing is not really a research area, because everyone's been holding on to their data," says Vijay Gadepally, senior staff in the LLSC who leads energy-aware research efforts. "Somebody has to start, and we're hoping others will follow."<h2>Curbing power and cooling down</h2>Like many data centers, the LLSC has seen a significant uptick in the number of AI jobs running on its hardware. Noticing an increase in energy usage, computer scientists at the LLSC were curious about ways to run jobs more efficiently. Green computing is a principle of the center, which is powered entirely by carbon-free energy.Training an AI model — the process by which it learns patterns from huge datasets — requires using graphics processing units (GPUs), which are power-hungry hardware. As one example, the GPUs that trained GPT-3 (the precursor to ChatGPT) are estimated to have consumed 1,300 megawatt-hours of electricity, roughly equal to that used by <a href="https://www.eia.gov/tools/faqs/faq.php?id=97&t=3" target="_blank">1,450 average</a> U.S. households per month.While most people seek out GPUs because of their computational power, manufacturers offer ways to limit the amount of power a GPU is allowed to draw. "We studied the effects of capping power and found that we could reduce energy consumption by about <a href="https://arxiv.org/abs/2205.09646" target="_blank">12 percent to 15 percent</a>, depending on the model," Siddharth Samsi, a researcher within the LLSC, says.The trade-off for capping power is increasing task time — GPUs will take about 3 percent longer to complete a task, an increase Gadepally says is "barely noticeable" considering that models are often trained over days or even months. In one of their experiments in which they trained the popular BERT language model, limiting GPU power to 150 watts saw a two-hour increase in training time (from 80 to 82 hours) but saved the equivalent of a U.S. household's week of energy.The team then built software that plugs this power-capping capability into the widely used scheduler system, Slurm. The software lets data center owners set limits across their system or on a job-by-job basis."We can deploy this intervention today, and we've done so across all our systems," Gadepally says.Side benefits have arisen, too. Since putting power constraints in place, the GPUs on LLSC supercomputers have been running about 30 degrees Fahrenheit cooler and at a more consistent temperature, reducing stress on the cooling system. Running the hardware cooler can potentially also increase reliability and service lifetime. They can now consider delaying the purchase of new hardware — reducing the center's "embodied carbon," or the emissions created through the manufacturing of equipment — until the efficiencies gained by using new hardware offset this aspect of the carbon footprint. They're also finding ways to cut down on cooling needs by strategically scheduling jobs to run at night and during the winter months."Data centers can use these easy-to-implement approaches today to increase efficiencies, without requiring modifications to code or infrastructure," Gadepally says.Taking this holistic look at a data center's operations to find opportunities to cut down can be time-intensive. To make this process easier for others, the team — in collaboration with Professor Devesh Tiwari and Baolin Li at Northeastern University — recently developed and <a href="https://arxiv.org/abs/2306.13177" target="_blank">published a comprehensive</a> framework for analyzing the carbon footprint of high-performance computing systems. System practitioners can use this analysis framework to gain a better understanding of how sustainable their current system is and consider changes for next-generation systems. &nbsp;<h2>Adjusting how models are trained and used</h2>On top of making adjustments to data center operations, the team is devising ways to make AI-model development more efficient.When training models, AI developers often focus on improving accuracy, and they build upon previous models as a starting point. To achieve the desired output, they have to figure out what parameters to use, and getting it right can take testing thousands of configurations. This process, called hyperparameter optimization, is one area LLSC researchers have found ripe for cutting down energy waste.&nbsp;"We've developed a model that basically looks at the rate at which a given configuration is learning," Gadepally says. Given that rate, their model predicts the likely performance. Underperforming models are stopped early. "We can give you a very accurate estimate early on that the best model will be in this top 10 of 100 models running," he says.In their studies, this early stopping led to dramatic savings: an <a href="https://www.researchgate.net/publication/362498394_Energy-aware_neural_architecture_selection_and_hyperparameter_optimization" target="_blank">80 percent reduction</a> in the energy used for model training. They've applied this technique to models developed for computer vision, natural language processing, and material design applications."In my opinion, this technique has the biggest potential for advancing the way AI models are trained," Gadepally says.Training is just one part of an AI model's emissions. The largest contributor to emissions over time is model inference, or the process of running the model live, like when a user chats with ChatGPT. To respond quickly, these models use redundant hardware, running all the time, waiting for a user to ask a question.One way to improve inference efficiency is to use the most appropriate hardware. Also with Northeastern University, the team <a href="https://arxiv.org/pdf/2304.09781.pdf" target="_blank" data-lf-fd-inspected-kn9eq4roawkarlvp="true">created an optimizer</a> that matches a model with the most carbon-efficient mix of hardware, such as high-power GPUs for the computationally intense parts of inference and low-power central processing units (CPUs) for the less-demanding aspects. This work recently won the best paper award at the International ACM Symposium on High-Performance Parallel and Distributed Computing.Using this optimizer can decrease energy use by 10-20 percent while still meeting the same "quality-of-service target" (how quickly the model can respond).This tool is especially helpful for cloud customers, who lease systems from data centers and must select hardware from among thousands of options. "Most customers overestimate what they need; they choose over-capable hardware just because they don't know any better," Gadepally says.<h2>Growing green-computing awareness</h2>The energy saved by implementing these interventions also reduces the associated costs of developing AI, often by a one-to-one ratio. In fact, cost is usually used as a proxy for energy consumption. Given these savings, why aren't more data centers investing in green techniques?"I think it's a bit of an incentive-misalignment problem," Samsi says. "There's been such a race to build bigger and better models that almost every secondary consideration has been put aside."They point out that while some data centers buy renewable-energy credits, these renewables aren't enough to cover the growing energy demands. The majority of electricity powering data centers comes from fossil fuels, and water used for cooling is contributing to stressed watersheds.&nbsp;Hesitancy may also exist because systematic studies on energy-saving techniques haven't been conducted. That's why the team has been pushing their research in peer-reviewed venues in addition to open-source repositories. Some big industry players, like Google DeepMind, have applied machine learning to increase data center efficiency but have not made their work available for others to deploy or replicate.&nbsp;Top AI conferences are now pushing for ethics statements that consider how AI could be misused. The team sees the climate aspect as an AI ethics topic that has not yet been given much attention, but this also appears to be slowly changing. Some researchers are now disclosing the carbon footprint of training the latest models, and industry is showing a shift in energy transparency too, as in this <a href="https://ai.meta.com/research/publications/scaling-autoregressive-multi-modal-models-pretraining-and-instruction-tuning/" target="_blank">recent report</a> from Meta AI.They also acknowledge that transparency is difficult without tools that can show AI developers their consumption. Reporting is on the LLSC roadmap for this year. They want to be able to show every LLSC user, for every job, how much energy they consume and how this amount compares to others, similar to home energy reports.Part of this effort requires working more closely with hardware manufacturers to make getting these data off hardware easier and more accurate. If manufacturers can standardize the way the data are read out, then energy-saving and reporting tools can be applied across different hardware platforms. A collaboration is underway between the LLSC researchers and Intel to work on this very problem.Even for AI developers who are aware of the intense energy needs of AI, they can't do much on their own to curb this energy use. The LLSC team wants to help other data centers apply these interventions and provide users with energy-aware options. Their first partnership is with the U.S. Air Force, a sponsor of this research, which operates thousands of data centers. Applying these techniques can make a significant dent in their energy consumption and cost."We're putting control into the hands of AI developers who want to lessen their footprint," Gadepally says. "Do I really need to gratuitously train unpromising models? Am I willing to run my GPUs slower to save energy? To our knowledge, no other supercomputing center is letting you consider these options. Using our tools, today, you get to decide."Visit <a href="https://vijayg.mit.edu/datacenter-ai-energy-reduction" target="_blank" id="isPasted">this webpage</a> to see the group's publications related to energy-aware computing and findings described in this article.

At the Lincoln Laboratory Supercomputing Center, researchers are making changes to cut down on energy use. One of their techniques can reduce the energy of training AI models by 80 percent. Photo: Glen Cooper

Amid the race to make AI bigger and better, Lincoln Laboratory is developing ways to reduce power, train efficiently, and make energy use transparent.

New tools are available to help reduce the energy that AI models devour

<h2 id="isPasted">Its impact on Visual SLAM and beyond</h2>It is clear that Artificial Intelligence (AI) is here to stay and there is a need for an open discussion, to bring some light into the AI black box.At Sevensense Robotics we strongly believe that transparency builds trust, this is why we decided to tackle the elephant in the room and dissect the role that AI plays in our offering.Let’s start by exploring what AI is and the difference between Machine Learning (ML) and Deep Learning (DL).The original definition of AI, coined in 1955 by John McCarthy -one of the founding fathers of the discipline- was really broad: “The science and engineering of making intelligent machines.”Nowadays, we conceive AI as a subset of computer science focused on creating systems that can replicate human intelligence and problem-solving abilities. It involves any technique that enables machines to mimic human intelligence, using logic, if-then rules, decision trees, and machine learning -including deep learning.Machine Learning (ML) is a subset of AI that intersects with statistics. It involves the development of algorithms and statistical models that enable computers to learn and make predictions or decisions without being explicitly programmed; instead, they recognize patterns in the data and make predictions once new data arrives.Deep Learning (DL) is a specialized subset of Machine Learning that analyzes data with a logical structure similar to how a human would draw conclusions.&nbsp;To achieve this, DL applications use a layered structure of algorithms called an Artificial Neural Network (ANN). It is characterized by its deep layers of interconnected nodes or neurons, which allows it to automatically learn hierarchical representations of data, making it well-suited for tasks like image and speech recognition.The three technologies are nested one within the next:<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk3MDIxMTQzOTQ1LURpZmZlcmVuY2VzIGJldHdlZW4gQUksIE1hY2hpbmUgTGVhcm5pbmcgYW5kIERlZXAgTGVhcm5pbmcgKDIpLnBuZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 300px;" class="fr-fic fr-dib" alt="Differences between AI, ML and DL">Visual AI refers to Artificial Intelligence applications that focus on enabling machines to interpret, understand, and process visual information, such as images and videos. Visual AI technologies employ advanced Machine Learning algorithms, Deep Learning models, and computer vision techniques to analyze and extract meaningful insights from visual data.AI-Powered Robot AutonomySevensense uses Visual AI to allow robots to identify, understand, and act on imagery and visual data -captured by cameras- just like humans so that they can navigate autonomously.By processing camera images with AI, the process known as Visual Simultaneous Localization and Mapping (SLAM) builds a rich 3D map of the environment and precisely localizes within it.Sevensense’s Visual SLAM technology comprises proprietary and lightweight AI sensor fusion algorithms to enable robots to precisely estimate their position. This is achieved by extracting distinctive elements from the images, for example, a corner of a window, and using these to build a representative yet compact 3D model of the environment.&nbsp;To deliver robust localization under all conditions, including lighting and environmental changes, Sevensense employs AI -more concretely Deep Convolutional Neural Networks- for feature extraction. Incorporating AI for this purpose makes it possible to overcome a limiting constraint of legacy SLAM technology, which until now could not cope with significant illumination and other environmental changes, leading to failures during autonomous navigation in difficult settings.Sevensense also uses machine learning to maintain the visual maps of dynamic spaces to ensure robust localization and reusability of the initial map of the environment. The solution known as Lifelong Visual SLAM is basically a multi-session map that uses the initial map as the basis and later on, the changes in the environment are added to it to enrich it. All of this, without the need to run Visual SLAM from scratch on each session, increases operational efficiency.The Lifelong Visual SLAM solution enables building and maintaining always-up-to-date maps. These maps incorporate data from multiple conditions of the environment, such as during cloudy- and sunny weather, bright and dark lighting situations, and changes in the elements present in the area, e.g. changes in factory floor layouts, or simply in the warehouse or shop inventory. All of these changes are detected automatically and incorporated into the lifelong map.This allows the maps to be used for visual positioning in environments that change considerably over time.To enable autonomous navigation even in complex and ambiguous environments, a Convolutional Neural Network is employed to perform Semantic Scene Understanding making it possible to detect, classify, determine the position, -and when applicable, predict the movement of obstacles e.g. people and vehicles, and other objects of interest such as pallets, trolleys, floor markings, docking or charging stations.After the robots have gained insights into their environment and the expected behavior of the elements around them, it is time for them to plan their next move. Reinforcement Learning makes it possible for robots to adapt their navigation behavior -stopping, slowing down, or overtaking obstacles, based on real-time perception, allowing them to interact with the environment, for example, docking, picking, or charging, while safely sharing the floor with moving elements like people and other vehicles.All of these processes happen on the Edge, on the multi-camera Alphasense Edge AI device -a GPU-accelerated compute unit- embedded in the robot. By bringing computational power closer to the data source, Sevensense minimizes latency challenges -enabling nearly simultaneous, real-time perception and decision-making- and addresses security and privacy issues typically present in solutions purely based on cloud computing, where visual data needs to be recorded and transmitted to remote servers for processing.&nbsp;AI-Powered Continuous Performance ImprovementTo exceed the ever-increasing performance requirements of the market, both in terms of accuracy and robustness, Sevensense leverages ongoing cutting-edge developments in the fields of AI, computer vision, and robotics.&nbsp;Sevensense uses AI models to enrich its simulation, training, and testing infrastructure.Combining Generative AI and Digital Twins, Sevensense is able to create the most challenging training and testing environments to continuously validate its software stack.Visual AI AnalyticsThe visual data captured by the cameras, apart from providing inputs for best-in-class localization and autonomy performance, also unlocks the potential for providing ground-breaking additional services.The rich visual data captured by the cameras allows Sevensense to offer further Visual AI analytic services focused on providing organized data and insights -beyond Visual SLAM- such as asset tracking, stocktaking, surveillance, and inspection, as an additional safety or reporting data layer.

Visual AI enables autonomous navigation and an array of object-recognition-based applications

Its impact on Visual SLAM and beyond

AI in Robotics Perception and Autonomous Navigation

The perceptual abilities to be aware of the state of the outside world are indispensable for all living things to keep on living. Humans gather information through the five senses of sight, hearing, taste, smell, and touch. We use that information to recognize and understand the situation surrounding us with our superior intelligence. We then perform advanced actions and judgments. Looking at it comprehensively, the perceptual abilities of humans are probably among the best in the living world.However, if we focus on individual perceptual abilities, we find that there are many living things with abilities superior to those of humans. We are no match for the compound eyes of dragonflies when it comes to a sense of sight, whereby the movements of things are perceived in the surroundings. Furthermore, we can't compete with dogs in terms of our senses of hearing and smell. There are even living things that have the ability to capture information that cannot be detected by humans in the first place. Examples of these include migratory birds that determine their direction of flight by perceiving geomagnetism. If we stretch our imaginations further, there may even be perceptive abilities that capture information that living things in existence cannot detect.If we could acquire perception beyond the five senses of humans with the assistance of machines, how would it change our lives and society? We introduce trends in the development and utilization of technologies that realize the super-five senses.<h2>Extending Our Perceptual Abilities Utilizing Evolving Sensors and AI</h2>Sensors are electronic devices with the function to acquire information from the outside world. There are already a wide variety of sensors that can reproduce the five senses of humans. Image sensors, microphones, accelerometers, and temperature sensors are examples of those. There are also many sensors that have performance that exceeds the abilities of humans. For instance, there are infrared ray image sensors capable of detecting the existence of things in the dark. Furthermore, there are also sensors that can acquire information that cannot be detected by humans. Examples include geomagnetic sensors and CO2 sensors.Advances in sensor technologies have made it possible to realize the ability to gather information far beyond the abilities of humans. Nevertheless, even if the range of information we can obtain expands and diversifies, it does not mean it is possible to extend the perceptual abilities of humans. We cannot improve our perceptual abilities without also improving our ability to appropriately process the information we have collected at the same time.&nbsp;For example, facial recognition technology has come to be used for the personal verification of visitors when holding sports and other events. It has also become possible for visitors to gain admission to venues using a face pass without the need for a ticket by registering their image data in advance. Furthermore, in recent years, it has become possible to simultaneously recognize those who can enter and those who have not purchased a ticket by collectively recognizing images of crowds captured by cameras. This has made it possible to smoothly guide people into the venue without lines. In addition, technology has evolved to the point where it is possible to estimate the age and gender of visitors.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1696846836897-super5sense6-01.jpg" style="width: 766px;" class="fr-fic fr-dib">It can be said that such advanced perceptual abilities using machines are the super-five senses beyond the abilities of humans. However, in this example, the super-five senses have not been realized by an evolution in the performance of the sensors that collect the information; or, in other words, the performance of the cameras. The super-five senses have been realized by the evolution of the artificial intelligence (AI), which is utilized in the recognition processing of the acquired images.&nbsp;<h2>Detecting Machine Failures from Noises and Vibrations and Cancer from Exhaled Breath</h2>The advanced perceptual abilities of humans are realized by combining the detection abilities of the sensory organs like the eyes and ears and the cognitive abilities of the brain (Fig. 1). In a similar scheme, the super-five senses will be realized by advancing sensors and AI and then utilizing those technologies in combination. We would now like to introduce just a few examples from among those of the super-five senses already being put into practical use or under development.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1696846855213-super5sense6-02_en.png" style="width: 766px;" class="fr-fic fr-dib">Fig. 1: The perceptual abilities of both humans and machines comprise a combination of detection (information gathering) and cognition (information processing) abilitiesThe super-five senses are already being utilized in many cases in the field of monozukuri. Some factories that produce industrial products employ experienced workers with the ability to sense signs of failure in devices. They do that by identifying the slightest abnormal sounds mixed in with the sounds of those devices operating. This is a personal skill acquired through experience over many years. It can be said that this is an advanced perceptual ability that regular people do not possess. However, high-precision accelerometers are nowadays being mounted to such devices. This has enabled the detection of signs of failure by utilizing AI from the slightest changes in the vibrations of those devices. Moreover, technologies are also being put into practical use that pinpoint the place where abnormal sounds are being emitted on images captured by cameras through catching the slightest abnormal sounds. This is an example of the realization of perceptual abilities not possessed by humans in terms of converting auditory information into visual information.The super-five senses are also starting to be utilized in the field of medicine. A variety of information that reflects the physical and health condition of humans is contained in the things we excrete from our bodies. One of those is the air we exhale when we breathe. It is known that exhaled breath contains components that indicate signs of cancer, appearing as a change in odor. Nevertheless, the change in odor is extremely slight. Accordingly, it is difficult to detect such a change with the human sense of smell. However, there are cases of specially trained dogs called "cancer detecting dogs" being used to detect cancer. That is because dogs have a superior sense of smell that can discern such a change. Many companies and research institutions are carrying out R&amp;D to mechanize this ability by combining a gas sensor that detects the components contained in exhaled breath with AI and other information processing systems. If such a function is installed on a smartphone where exhaled breath is blown during a phone call, it may be possible to achieve a state as though the user was undergoing cancer screening on a daily basis. In addition, attempts are also underway to analyze acetone contained in exhaled breath and to then utilize that analysis in lifestyle-related disease prevention, dieting, and other areas of health.<h2>HSI That Finds Valuable Information from Invisible Light</h2>Furthermore, technologies that acquire information that cannot be obtained from visible light are being developed and utilized. This is done by combining image sensors that detect light with wavelengths that cannot be detected by humans such as ultraviolet rays and infrared rays with AI and other advanced information processing technologies. This technology is called hyperspectral imaging.It may be possible to acquire unexpectedly high-added-value information depending on the method of processing the information obtained from visible light as well. For instance, the nutrient of lycopene contained in tomatoes absorbs blue and green light. Therefore, tomatoes that contain lots of lycopene have a bright red appearance. In other words, we can detect nutrients from color. We can acquire even more diverse information without contact by also applying techniques with similar principles to light other than visible light.For example, it is possible to identify or otherwise recognize the state of pollution in the land, atmosphere, and water and the types of trees growing on the surface from images taken with visible light and infrared rays from an artificial satellite. In addition, it is also possible to discover contamination by foreign substances or abnormal ingredients in food production lines and to evaluate the content of specific amino acids including oleic acid, which is an umami (savory) ingredient. We can also check the health condition of people based on changes in their complexion that cannot be detected by the human eye.There is still room for significant advances in both sensors and AI. As a result, it seems there are infinite possibilities for what kind of super-five senses humans will acquire in the future.<hr><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1696846884169-1696846884169.png" class="fr-fic fr-dii">If you have any questions based on this article that you would like to discuss with an expert at Murata, don't hesitate to reach out. There's an entire team ready to support you with your question.<a href="https://www.murata.com/en-eu/contactform?&Detail=Wevolver%20request" target="_blank" rel="nofollow" class="button-main-action">GET IN TOUCH</a>

Sensors are electronic devices with the function to acquire information from the outside world. There are already a wide variety of sensors that can reproduce the five senses of humans. Image sensors, microphones, accelerometers, and temperature sensors are examples of those.

Super-Five Senses Created by Combining and Utilizing Sensors and AI

<h2>Challenges of self-localization in mixed in- and outdoor environments</h2>Knowing its own location or position within an operating environment (a.k.a. self-localization) is one of the key capabilities any mobile robot must master to be able to get from A to B. Yet, depending on the environment the robot is operating in, self-localization can impose vastly different challenges, rendering the choice of sensor technology and software algorithms far from a trivial task.Indoors, 2D LiDAR technology is widely used, as an example, on AGVs and AMRs for material handling operations. Such a technology allows to measure the geometric structure of the near-by environment (e.g. walls, shelves) to localize in pre-built maps. However, it shows its limitations in environments which do not have unique geometric features (e.g. a long corridor) or are highly dynamic especially if shared with people.&nbsp;On autonomous vehicles for outdoor environments, instead, satellite navigation systems such as GPS or Galileo are often the first choice of sensor technology for self-localization, for good reasons: The GPS receivers are very cheap, and they yield an easily interpretable position in a known global reference coordinate system (e.g., WGS84).&nbsp;However, an accurate GPS position requires unobstructed skies. This is a limitation that is violated in several circumstances like urban surroundings, company campuses, &nbsp;logistic yards and airport tarmac. In fact, these outdoor places often require to operate close to or even inside buildings or tunnels. Due to these restrictions, mobile robots operating outdoors or in mixed indoor-outdoor settings cannot rely solely on GPS for self-localization.&nbsp;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk1MTMxNTk3MzM0LU1peGVkIGluZG9vciBhbmQgb3V0ZG9vciBwb3NpdGlvbmluZy5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib">Mixed indoor and outdoor environments are challenging for GPS and LiDAR solutions aloneEven though 2D LiDAR sensors can be employed outdoors too, using them in the open only works in well-structured spaces of limited extent or requires additional infrastructure such as laser reflectors in order to work better., or to a better extent thanks to the installation of additional infrastructure such as laser reflectors. 3D LiDARs, on the other hand, remain a very expensive sensor gear, despite all efforts in cutting down the costs in recent years.<h3>The strength and challenges of visual localization</h3>As an alternative to GPS and LiDAR sensors, cameras are able to reliably sense the environment both indoors and outdoors. Vision thus has the capability to bridge this gap left uncovered by GPS or LIDAR sensors. It offers highly accurate and reliable self-localization in both indoor and outdoor environments with a very cost effective sensor gear.&nbsp;Visual self-localization does not come without its very own challenges. Cameras naturally provide good images in well illuminated surroundings, while during night time outdoors, or in dim indoor areas, visual self-localization becomes more difficult. However, the selection of camera sensor chips with a high dynamic range &nbsp;has shown to allow proper images to be taken even under challenging illumination conditions.<blockquote>Instead, LiDAR sensors can be employed. However, using 2D LiDARs outdoors only works in well-structured spaces of limited extent. As an alternative to GPS and LiDAR sensors, cameras are able to reliably sense the environment both indoors and outdoors.</blockquote>Further, and in contrast to GPS or LiDAR sensor data, images taken by cameras of the same environment can look vastly different under varying seasonal, weather, or lighting conditions. This change in appearance renders accurate localization difficult. Additionally, geometric characteristics are often unfavorable for visual localization outdoors compared to indoors, as there is often less stable, human-made structure visible, plus it is often located at a considerably farther distance. Therefore, smart algorithms to work with visual data are paramount.<h3>Tackling the challenges of visual localization with AI</h3>To master these challenges, Sevensense employs a smart AI algorithm based on lifelong visual mapping that enables building and maintaining always-up-to-date maps of the environment. These maps incorporate data from multiple appearances of the environment, such as before- and after remodeling of a room, during cloudy- and sunny weather, or bright and dark lighting conditions. This allows the maps to be used for visual localization under practically any appearance condition.&nbsp;At the core of this lifelong mapping process is a software algorithm capable of merging visual data from multiple traversals through the same environment, recorded under potentially very different appearance conditions, into a single, geometrically consistent map. This process is fully automated, resulting in having a multi-appearance map ready for reliable localization after traversing through the environment only a handful of times beforehand.&nbsp;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk1MTMxNzMxMTk3LUxpZmVsb25nIHZpc3VhbCBtYXBwaW5nIGFsZ29yaXRobXMgdXNlZCBieSBTZXZlbnNlbnNlLnBuZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 300px;" class="fr-fic fr-dib">Smart lifelong visual mapping algorithms allow for always-up-to-date mapsFurther, continuously updating this map over time naturally captures structural change in the environment. This way events like remodeling of rooms, construction sites or temporary modifications in street or room layouts don&rsquo;t disrupt everyday operations. These changes are detected automatically and swiftly incorporated into the lifelong map.&nbsp;In particular, no expert knowledge or intervention is necessary at any point in the process. Just as generating the initial map of the environment can be done by anyone without any prior training or expert know-how, updating the maps at any later point is done automatically by Alphasense Position. This considerably lowers the costs of maintaining the system in a long-term horizon.&nbsp;<h3>Vision is the optimal solution for indoor- and outdoor self-localization</h3>By employing this lifelong mapping technique, vision overcomes its limitations and turns out to be the go-to technology for cost-effective, but still reliable and accurate self-localization both indoors and outdoors. In particular, it offers seamless transition from indoor to outdoors and vice-versa, as the same camera sensors and the same map can be used to localize in both types of environments. This is a fundamental advantage over any technology primarily relying on GPS for outdoor localization and navigation that can revolutionize the logistics, delivery, cleaning or supply chain processes.<blockquote>By employing this lifelong mapping technique, vision turns out to be the go-to technology for cost-effective, but still reliable and accurate self-localization both indoors and outdoors.</blockquote>Sevensense has already successfully demonstrated the unique benefits of visual localization in a <a href="https://www.sevensense.ai/blog/kyburz" target="_blank" rel="noopener noreferrer">proof-of-concept study in partnership with KYBURZ Switzerland AG</a>. By equipping a mail delivery robot with Alphasense Position, accurate and reliable self-localization of the vehicle became possible at any time indoors, as well as outdoors, where LiDAR and GPS solutions alone failed.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1695132745402-Reliable+positioning+indoors+and+outdors+cameras+overcome+LiDAR+and+GPS+limitations+Sevensense.jpeg" style="width: 300px;" class="fr-fic fr-dib">Delivery robot used to benchmark Alphasense Position, 3D LiDAR and GPS localization performance[the missing image will go here once the CMS issue is fixed]<blockquote>By equipping a mail delivery robot with Alphasense Position, accurate and reliable self-localization of the vehicle became possible at any time indoors, as well as outdoors, where LiDAR and GPS solutions failed.</blockquote>Along a similar vein, the indoor- and outdoor capabilities of Alphasense Position will open up countless other opportunities for bringing cost-effective autonomy to robotic platforms operating in challenging environments. Delivering goods across multiple buildings on a campus, or material handling in warehouses undergoing frequent layout change have never been simpler before.<h3>With vision towards a fully autonomous future</h3>Self-localization is essential for robots to safely navigate in changing environments and achieve full autonomy. By employing cameras for this task together with smart algorithms for building and maintaining maps, autonomy can be provided at a fraction of the costs - be it for the sensor gear itself, as well as operating costs over the robot&rsquo;s lifetime.&nbsp;At Sevensense we are constantly pushing cutting edge technologies forward to deliver affordable autonomy solutions for new application fields.You can find more informative material about Alphasense Position on its <a href="https://www.sevensense.ai/product/alphasense-position" target="_blank" rel="noopener noreferrer">product page</a>.

Test vehicle benchmarking cameras, LiDAR and GPS systems

Breaking Free from LiDAR and GPS Constraints: Harnessing Vision for Reliable Positioning, Indoors and Outdoors

Why reliable positioning indoor and outdoor requires cameras

<h2 id="isPasted">The assignment</h2>As the Adapta Robotics team is an avid supporter of learning, and wanting to promote it in any way, shape or form, we decided to share with you the experience of one of our interns in the Students&rsquo; Program as he tackles the challenge of making MATT play chess. Let&rsquo;s hear them out:<iframe width="640" height="360" src="https://www.youtube.com/embed/NDs5WkYxWaU?&wmode=opaque&rel=0&enablejsapi=1&origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-gtm-yt-inspected-6="true" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><h2>The planning process</h2>First step in getting familiarized with MATT was to learn its framework. The process was surprisingly easy and, after one week of getting accustomed to the testing robot, I&rsquo;ve got the task of teaching MATT how to play chess.My first though was to search for an online digital chess board but scrolling through websites proved to be a first challenge. For this reason, I decided that the best approach would be to use an app, Chess.com. The choice was not random. Having analyzed multiple apps, I noticed that Chess.com had the option of changing the colors of the board. This feature helped me underline the method I used to complete the task: scanning the board not for the piece that was moved, but by the colored squares it left on the board from its initial position and its move destination (the &lsquo;footprints&rsquo; of executed moves).After some planning, I found it useful to integrate the Stockfish chess engine in my code and use it as storage for the date referring to chess pieces&rsquo; positions and the state of the game. The next step was to design an algorithm that detected the opponent&rsquo;s move. As I mentioned earlier, the option of changing the board&rsquo;s theme was extremely useful since it allowed me to choose the color that was easiest to detect in the given environment and to create a function that was searching for that precise color. Finishing the color detection stage and beginning the calibration and debugging process, I realized that the hard part just began. After every modification, to observe new errors or miscalibrations I started watching one, two or more matches, sometimes as long as 30 minutes each; a very time consuming but necessary operation that was essential in perfecting the way I was solving the given task.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1694083545576-inside-chess-article-2048x1366.jpg" style="width: 766px;" class="fr-fic fr-dib"><h2 id="isPasted">Second iteration, improvements, and challenges</h2>After the first iteration of the code, MATT was able to play only with the white pieces and only with Chess.com bots. Wanting to make MATT game universal, I perfected a new version of the program with a cleaner structure, able to play with any chess piece color and that had the feature to set up the position of the board, making it possible to play on any mode. Furthermore, I upgraded the board scanning algorithm that detected the blue squares (the &lsquo;footprints&rsquo; of executed moves). During this stage, an issue that I&rsquo;ve met was the tap response time of the phone&rsquo;s touchscreen. To solve it, I replaced MATT&rsquo;s simple taps with short holds of 0.2 seconds.After finishing the remaining touchups, I rewrote the board scanning algorithm, which consequently generated faster results, and I started the final tests of the program to find leftover bugs. In the end, the program acted as an intermediate for MATT robot and the Stockfish chess engine, it&rsquo;s job being to scan the table, to check if the opponent&rsquo;s move is correct, to pass the information to the engine and to maneuver MATT&rsquo;s arm in order to move a chess piece.<h2>Conclusion</h2>To sum it up, I can say it was a fascinating experience and a practical way to apply some of the knowledge I&rsquo;ve gained during the training sessions. During the task, I found working with MATT to be very pleasant due to its ability to execute individual simple tasks and being easy to program, that combined led to easiness in carrying out more complex operations.

As the Adapta Robotics team is an avid supporter of learning, and wanting to promote it in any way, shape or form, we decided to share with you the experience of one of our interns in the Students’ Program as he tackles the challenge of making MATT play chess. 

computer vision

Profiles