Matroid makes computer vision accessible, enabling anyone to build, deploy, and iterate on custom visual detectors without writing code. Whether you’re inspecting aluminum extrusions for speed tears or scanning shipping yards for missing containers, Matroid turns complex visual AI into something fast, flexible, and easy to use. However, creating a great detector isn’t a one-and-done task. Much like training a new employee, your detector learns best with clear examples, consistent feedback, and regular check-ins to track progress. In this post, we’ll walk through how to evaluate that progress using the same metrics our platform provides automatically, so you can build detectors that get smarter, sharper, and more reliable over time.<h2>Overview</h2>Matroid’s platform empowers you to create custom “detectors” capable of identifying and classifying virtually anything in visual imagery—whether it’s specific material defects, verifying the correct installation of a component, or detecting human activity in a particular location. If something is visible to the human eye, chances are you can train a detector to recognize it. Think of a detector as a new team member on the factory floor, just one who never sleeps and can review thousands of images per hour. Just like a new employee, your detector needs clear instructions, examples of what to look for, and consistent feedback to improve. The more situations you expose it to, the sharper its judgment becomes.Building an effective detector isn’t a one-and-done task; it is an iterative process. It involves continually refining and retraining your model to improve its performance over time. Much like training a new employee, your detector learns best with clear examples, consistent feedback, and regular check-ins to track progress.But what does it actually mean to “improve” a detector? And how can you tell if a new version is better than the previous one? Matroid simplifies this by automatically calculating and visualizing key performance metrics widely used in machine learning – including precision, recall, F1 score, and mean Average Precision (mAP). Here we discuss what these indicators are demonstrating and how to use them to guide your detector iteration. <h2>Detector Evaluation Metrics </h2><h3>Core Metrics</h3>When evaluating detectors, the two key metrics to focus on are precision and recall. To understand these, it’s essential to know how detection results are categorized and understand how detector confidence thresholds work. If you’re new to computer vision, imagine testing an intern on identifying product defects from photos. Every time they point to a defect, you check: Were they right? Did they miss any? This is essentially what precision and recall measure.Imagine you’ve trained a detector to identify a specific defect in manufactured parts. You test it on new images or video frames and classify the outcomes as follows:True Positive (TP): The detector correctly identifies a fundamental defect. False Positive (FP): The detector flags something as a defect, but it’s actually not. True Negative (TN): The image contains no defects, and the detector correctly makes no detection. False Negative (FN): A genuine defect is present, but the detector fails to identify it.Detection Confidence and Thresholds Before calculating precision and recall for a detector, you need to choose a confidence threshold at which to make that calculation. A Matroid detector will output a confidence score for every detection it makes. This score is between 1% – 100%, which denotes how confident the detector is about the detection. When you choose a “confidence threshold, ” you specify the minimum confidence level required to keep a detection. Say your detector incorrectly flags something as a defect that is not a defect with a confidence score of 45%. If you selected a confidence threshold of 40%, this detection will be recorded as a false positive. If, however, you had chosen a threshold of 50%, this detection would not have surfaced, and the image that contained it would count as a true negative in your calculations. Setting a threshold is like telling your intern: “Only tell me you see a defect if you’re at least 80% sure.” Lower the threshold, and they’ll raise more flags (some incorrect). Raise it, and they’ll speak up less, but might miss something important.Recall Recall measures how well your detector finds all the items it’s supposed to detect. For example, if you run your detector on 100 images containing defects and the detector finds 90 defects,  your recall is 90%. High recall means the detector is good at not missing anything important, but it doesn’t tell you anything about the amount of “false positives”. &lt;figure&gt;&lt;img width="1024" height="261" src="https://www.matroid.com/wp-content/uploads/2025/06/image-4-1024x261.png" alt="" class="fr-fic fr-dii" /&gt;&lt;/figure&gt;Recall is all about completeness. If you’re running safety inspections, high recall ensures the model sees everything, even if some of what it sees turns out not to be an issue.Precision Precision lets you know how well your detector separates real detections from false indicators. If your detector flags 100 defects and 80 of them are real defects (true positives), you have 80% precision. High precision means most of the detector finds are relevant, with fewer false alarms.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1762247371138-image-3-768x172.webp" style="width:100%px" class="fr-fic fr-dib" />Precision, by contrast, is about accuracy. It answers: “When the detector speaks up, is it usually right?” For high-stakes but low-frequency defects, precision can help you avoid panic over false alarms.<h3>Additional Metrics</h3>Precision and recall are often all you need when evaluating detector performance. But in case you want to dive deeper – Matroid also automatically calculates other, more complex metrics that provide different perspectives on performance, such as F1 score and mAP. F1 Score – The F1 score combines precision and recall into a single number. It helps you balance the trade-off between catching everything (recall) and accuracy (precision). A high F1 score means your detector is both thorough and trustworthy. The formula for recall is: <img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1762247377588-image-2-768x159.webp" style="width:100%px" class="fr-fic fr-dib" />If precision and recall are like two employees with different strengths, F1 is your way of hiring the one who’s great at both.mean Average Precision (mAP) – mAP is a more complex, comprehensive measure that looks at how well the detector performs across different confidence levels and object types. It is calculated by averaging the precision values at various recall levels for each label and then averaging across all labels. mAP gives you a big-picture view of overall detector quality. It is often helpful as a quick check when comparing two different detectors since it provides a condensed performance summary across multiple labels and thresholds. mAP is like a final grade across all tests, averaging every subject, every quiz, every effort. It’s useful when comparing students (detectors) across the whole semester, not just one assignment.<h2>Viewing and Interpreting Metrics In Matroid </h2>These metrics we discussed above are automatically calculated as part of the Matroid detector training process. Depending on the size of your dataset, a certain number of images you annotated when creating your detector are set aside for detector validation. The metrics calculated on this validation set can be seen in a trained Matroid detector’s “Analyze” tab (the leftmost tab on the detector page). <img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1762247394981-image-1-1536x289.webp" style="width:100%px" class="fr-fic fr-dib" />You don’t need to be a data scientist to understand your detector’s performance. Matroid presents everything in clean dashboards, so you can make data-driven decisions without needing to crunch formulas.For example, say you have a detector that can detect “Bolts.” You can enter the analyze tab for that detector and use Matroid’s built-in confidence slider to quickly compare the detector’s precision, recall, and F1 score at any threshold. You can use this tool to quickly hone in on the best threshold for deploying that detector in a live environment.This slider lets you simulate real-world decisions. Do you want a detector that catches every bolt, even at the risk of flagging extras? Or one that only calls out bolts that it’s certain about? You get to choose the sweet spot based on your priorities.Your “optimal” threshold choice depends on your goals and tolerances. Oftentimes the goal will be a balance between precision and recall (i.e. maximizing the F1 score). However, perhaps a particular defect is a safety critical issue that can’t be missed, even if it means sifting through more false flags. In that case, you would want to maximize recall even at the expense of precision. The opposite could also be true – perhaps a few slip-throughs are acceptable, but you don’t want to waste operators’ time responding to false flags (in which case you want to optimize for precision). <img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1762247413071-confidence-thresholds-647x1024.webp" style="width:100%px" class="fr-fic fr-dib" />You also can view a confusion matrix for a detector in Matroid’s “advanced metrics” tab. A confusion matrix plots the number of TPs, FPs, FNs, and TNs for each label in a single heatmap, so you can quickly see where the detector is getting “confused” (i.e., where the detector is mixing up or missing detections). <img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1762247429240-image-5-1536x591.webp" style="width:100%px" class="fr-fic fr-dib" />Clicking on a given box in the confusion matrix will display examples from the validation set that fall into that category. The confusion matrix can help provide a better qualitative understanding of what confuses the detector.  This is like a report card showing which test questions your detector struggled with. Did it confuse one defect type with another? Did it miss smaller or darker examples? The confusion matrix doesn’t just show scores; it tells the story of where to improve next.<h2>Running Your Own Analysis</h2>You may want to run a detector against a video or image “test” dataset that you curate yourself and calculate these same detector performance metrics on that dataset. This can be done by receiving detection results using Matroid’s API – either by <a href="https://app.matroid.com/docs/api/documentation">pulling detection results from a Matroid monitoring</a> or by directly running a detector on images programmatically through Matroid’s <a href="https://app.matroid.com/docs/api/index.html#api-Images-Classify">“classify image” API</a>. <h2>When to Tune Thresholds vs Retrain Your Detector </h2>As we have seen here, there are multiple metrics you can use to hone in on the optimal confidence threshold at which to deploy your trained Matroid detector. This confidence threshold is never set in stone – you can adjust it post-deployment to better balance the tradeoff between true positives and false positives based on your evolving needs.If you’ve already optimized the threshold but still aren’t achieving the desired performance, it’s likely time to retrain the detector to improve its baseline accuracy. This can be done through <a href="https://app.matroid.com/product_documentation/improve-detectors/remedies/feedback">providing feedback</a> to the detector, and/or <a href="https://app.matroid.com/product_documentation/build-detectors/dataset-prep">curating a more diverse, higher-quality dataset</a>. Sometimes tweaking the detector’s threshold is enough, like adjusting the sensitivity dial. Other times, you need to go back and re-teach the fundamentals with better examples. That’s when retraining comes in.With each new version of your detector, the same core metrics—precision, recall, etc.—can be used to evaluate progress and compare against previous iterations. These metrics serve as valuable benchmarks to help ensure you’re moving in the right direction during the detector development process.

In this post, we'll walk through how to evaluate that progress using the same metrics our platform provides automatically, so you can build detectors that get smarter, sharper, and more reliable over time.

Evaluating and Refining Computer Vision Detectors

Why it’s still hard to build great products in the age of AI…IntroductionHere are some recent conversations where my voice assistant has struggled:[Sitting down for dinner]Me: “Alexa, play some soft jazz guitar music.”[Silence]Me (more clearly this time):  “ALEXA… play some soft jazz guitar music.”Alexa: “Sorry, that device is offline.”[I stare at my plate, contemplating whether we have five or six music services connected to our Alexa, which is very much still online]——————————————-Me: “Alexa, close the curtains.”Alexa: “There are multiple devices named curtains. Would you like to close living room curtains or curtains 5371?”[Brief pause as I ponder this: we only have one set of curtains, and this command has always worked. I take a guess]Me: “Living room curtains.” [The curtains in our bedroom close] ———————————————-Wife: “Alexa, what are the lyrics to Toxic by Britney Spears?”Alexa (in typical voice assistant monotone):  “Baby can’t you see, that I’m calling…” [completes first verse]Wife: Keep goingAlexa: “Hmm, I’m not sure about that.”——————————————-AI has made incredible recent progress, but voice assistants are still plagued by the rigidity and lack of contextual awareness they launched with years ago. For most, voice remains an interface of last resort, reserved for occasionally pulling up directions or playing a song on a smart speaker. With LLMs capable of nuanced and aware conversations, why are we having so much trouble making everyday language-based products more useful?<h2>Agentic Systems</h2>To understand why voice assistants often stumble, let’s take a step back and look at agentic systems more generally. Automated systems taking actions on behalf of users interact with the world in three steps:<ol><li>Perception: understand the slice of the world relevant to our system </li><li>Planning: enumerate the relevant decision space and select the correct action</li><li>Execution: properly carry out the action chosen in the planning step</li></ol><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1755674995849-Diagram-of-an-automated-system-acting-in-an-environment-1-1.webp" style="width:100%px" class="fr-fic fr-dib" />Diagram of an automated system acting in an environmentFor a simple feedback loop, these steps might be trivial: if the temperature reading is too hot (perception), switch off the power (planning and execution). For a more sophisticated system, like an autonomous vehicle, things are more challenging. The slice of the world to understand is broader: where we are in space relative to other objects, relevant signs and lane lines, the state of traffic lights, the local road laws, etc. Our available action space is large and has more dimensions: we can accelerate or break, continue straight or swerve, turn on our blinker, etc. To execute our actions, we need to communicate with the car’s control systems quickly and accurately.A typical voice assistant also operates in a complex environment. It needs to understand what a user said and then choose and execute an action from a set of available integrations and APIs. There are challenges at each step: picking out the right voice from the background noise, parsing accents correctly, tracking active integrations, choosing the right action, etc. <h2>Improvements with AI</h2>AI is enabling progress in automated systems, especially in the perception and planning phases. On the perception front, AI gives our systems a more flexible and nuanced understanding of the world. Computer vision models, like the ones we work on at Matroid, are increasingly capable of detecting more things in more contexts. LLMs should lend voice assistants near human-level language fluency and comprehension.We’ve also seen impressive progress in planning. Systems can choose correct actions without explicitly programming logic for every possible scenario. A Waymo can navigate the world with steering and acceleration inputs that are smoother and more accurate than most human drivers. If you give some API documentation and a few examples to an LLM, it can often construct a correct API request for a simple task. However, there are still limitations: parts of an environment might be too complex or nuanced for a perception model to understand. A voice assistant can only pull so much signal out of a noisy room. Additionally, complex decision spaces make planning difficult. When using LLMs to do complex stuff with access to sprawling APIs, it’s often easier to first build a layer of abstraction the LLM can target. Reducing hundreds of API endpoints into tens of composable logical blocks greatly condenses the decision space an LLM-based agent must consider, increasing the odds of success. <h2>Execution Challenges</h2>Unlike perception and planning, execution is largely still dependent on existing systems. LLMs are proving themselves capable of tasks like building simple web pages or analyzing data. Rapid progress for use cases like these is possible because we already have the execution part down: we were already really good at executing code cheaply in different environments before LLMs came around.But if tasks extend beyond the digital world – running code, making API calls, automating browser sessions, etc – then we need to first build integrations between our agentic systems and hardware. It wouldn’t matter if AI could generate instructions to fold my laundry because we don’t have a common hardware platform that could carry out those instructions. Voice assistant execution is inherently limited to digital tasks where they have access to proper integrations. Improved perception and planning will help them navigate more advanced integrations and take on more complex tasks, but building more integrations is a prerequisite for progress.<h2>Trust Issues</h2>Even when a voice assistant can handle a task, there’s another adoption barrier: we might not trust it to do a good job. Consider booking a flight, a purely digital task a voice assistant or chatbot could probably handle. When I choose an itinerary on Kayak, it’s based on some blend of price, trip duration, flight timing, airline, and potential rewards use or accrual. It’s a fairly complicated heuristic that would be tricky to accurately describe, and my ultimate choice usually differs from Kayak’s recommended best choice. The stakes are high: a mistake here could cost me hundreds of dollars or make for a miserable travel day. Ordering food or making a dinner reservation poses similar scenarios – I might be able to pick out details on the menu or in pictures that a model might miss. Information-rich aggregators and review sites are still great interfaces for these types of tasks. However, until automated assistants can prove that they will reliably make the same decisions we would, many users will continue to eschew chat and voice for the apps and sites we’re used to. <h2>Navigating Heterogeneous Environments</h2>Another core challenge for AI assistants is that the world they must navigate is a messy one. Revisiting my home voice assistant is illustrative: I’ve got Alexa running on a set of Sonos speakers. Four different music services are connected through the Alexa app, and an additional three are connected through the Sonos app, including two separate Spotify accounts. When I ask Alexa to play music, it has to navigate this mess, somehow selecting one of these services, figuring out the song or playlist that matches my request, and then talking to the API of a speaker made by a company that has <a href="https://www.sonos.com/en-us/blog/update-on-the-sonos-app">publicly apologized</a> for the quality of its software. And this isn’t even a particularly complicated setup by the standards of home entertainment systems.Voice assistants are as ubiquitous as cell phones, but for each user, the set of available integrations, hardware, and accessible data is slightly different. It’s a minor miracle that they work at all. Yet, after getting used to mostly accurate and nuanced conversations with LLMs, how could we not expect more? My entire chat history with Alexa could fit into the context window of a modern LLM, but there’s no evidence of any contextual awareness. I frequently have to repeat myself or rephrase things to get the right result.It isn’t particularly surprising to see failures like the <a href="https://www.google.com/search?q=rabbit+r1+review&amp;oq=rabbit+r1+review&amp;gs_lcrp=EgZjaHJvbWUyCQgAEEUYORiABDIHCAEQABiABDIHCAIQABiABDIICAMQABgWGB4yCAgEEAAYFhgeMggIBRAAGBYYHjIKCAYQABgKGBYYHjIICAcQABgWGB4yCAgIEAAYFhgeMggICRAAGBYYHtIBCDM1ODNqMGoxqAIAsAIA&amp;sourceid=chrome&amp;ie=UTF-8">Rabbit R1</a> from companies new to the assistant space. But even the giants like Apple – a company with a legendary product pedigree that can tightly control the environment its voice assistant operates in – can’t quite figure it out: <a href="https://www.wired.com/story/apple-intelligence-isnt-ready-to-wow-you-yet/">tepid reviews</a> for Siri’s new Apple Intelligence features only underscore the complexity of the challenges. Of course, just because we could enable context for voice assistants or let them step outside a narrow set of pre-defined actions doesn’t mean that we should. Potential unintended consequences abound: contextual awareness could easily become a vector for data leaks, and making untested API calls could wreak all sorts of havoc for users. Heterogeneous deployment environments exacerbate the challenges: it’s a vast surface area against which to test new features safely. <h2>Automated Systems with Computer Vision</h2>So, how do voice assistants relate to the work we’re doing at Matroid? We’re not in the business of assistants, but we work on making automated systems more useful in diverse environments. While voice assistants struggle with the messiness of varied home setups, at Matroid we need to deal with the reality of modern industrial environments. There’s a lack of industry standardization, and the complexity of IT systems with generations of legacy components and Byzantine network architectures quickly dwarfs that of any home entertainment system. Our product largely focuses on perception, enabling our customers to detect safety and quality issues. But we recognize that the best solutions are complete ones, so we’ve also built a number of features to flexibly integrate into diverse environments: customizable APIs, PLC integrations, physical device integrations, and support for human-in-the-loop workflows make it possible for our customers to make decisions and take action with computer vision.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1755675033420-stacklight-1536x1023.jpg" style="width:100%px" class="fr-fic fr-dib" />Deployments will become simpler as AI tooling becomes standard in industrial settings. In the meantime, our talented solutions team works closely with customers to design the hardware footprints and integration points necessary to be successful in any environment. It’s this blend of flexible product features and a semi-tailored approach that enables our customers to maintain quality standards, maximize efficiency, and keep their workers safe. <h2>Looking Forward</h2>Voice assistants will undoubtedly become more accurate and useful as AI improves. Context awareness, hybrid visual / voice input, and more useful actions are likely on the roadmaps for most of the teams working in the space.However, progress in voice assistants and automated systems will depend on more than just model advancements; it will require engineering systems and platforms AI can interact with across a broader range of tasks, bridging the gap between understanding and execution.Until we see that progress? I’ll probably just pull out my phone to pick the dinner music.

AI has made incredible recent progress, but voice assistants are still plagued by the rigidity and lack of contextual awareness they launched with years ago. For most, voice remains an interface of last resort, reserved for occasionally pulling up directions or playing a song on a smart speaker.

If AI is so good, why isn't my voice assistant more useful?

Have you ever wondered how wild animals behave when no one’s watching? Understanding these behaviors is vital for protecting ecosystems—especially as climate change and human expansion alter natural habitats. But collecting this kind of information without interfering has always been tricky.Traditionally, researchers relied on direct observation or sensors strapped to animals—methods that are either disruptive or limited in scope. Camera traps offer a less invasive alternative, but they generate vast amounts of footage that's hard to analyze.<iframe width="640" height="360" src="https://www.youtube.com/embed/5TCZzPb9LjA?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>AI could help, but there's a catch: it needs annotated datasets to learn from. Most current video datasets are either scraped from the internet, missing the authenticity of real wild settings, or are small-scale field recordings lacking detail. And few include the kind of rich context—like multiple camera angles or audio—that’s needed to truly understand complex animal behavior.<h2>Introducing MammAlps</h2>To address this challenge, scientists at EPFL have collected and curated&nbsp;<a href="https://eceo-epfl.github.io/MammAlps/" target="_blank" rel="noopener">MammAlps</a>, the first richly annotated, multi-view, multimodal wildlife behavior dataset in collaboration with the Swiss National Park. MammAlps is designed to train AI models for species and behavior recognition tasks, and ultimately to help researchers understand animal behavior better. This work could make conservation efforts faster, cheaper, and smarter.MammAlps was developed by Valentin Gabeff, a PhD student at EPFL under the supervision of Professors Alexander Mathis and Devis Tuia, together with their respective research teams.<iframe width="640" height="360" src="https://www.youtube.com/embed/hZjktVTozmU?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><h2 id="isPasted">How MammAlps was developed</h2>The researchers set up nine camera traps that recorded&nbsp;more than 43 hours of raw footage over the course of several weeks. The team then meticulously processed it, using AI tools to detect and track individual animals, resulting in&nbsp;8.5 hours of material showing wildlife interaction.They labeled behaviors using a hierarchical approach, categorizing each moment at two levels: high-level activities like foraging or playing, and finer actions like walking, grooming, or sniffing. This structure allows AI models to interpret behaviors more accurately by linking detailed movements to broader behavioral patterns.To provide AI models with richer context,&nbsp;the team supplemented video with audio recordings and captured “reference scene maps” that documented environmental factors like water sources, bushes, and rocks. This addition al data enables better interpretation of habitat-specific behaviors. They also cross-referenced weather conditions and counts of individuals per event to create more complete scene descriptions.“By incorporating other modalities alongside video, we've shown that AI models can better identify animal behavior,” explains Alexander Mathis. “This multi-modal approach gives us a more complete picture of wildlife behavior.”<h3>A new standard to wildlife monitoring</h3>MammAlps brings a new standard to wildlife monitoring: a full sensory snapshot of animal behavior across multiple angles, sounds, and contexts. It also introduces a&nbsp;“long-term event understanding" benchmark, meaning scientists can now study not just isolated behaviors from short clips, but broader ecological scenes over time—like a wolf stalking a deer across several camera views.Research is still ongoing. The team is currently processing data collected in 2024 and carries out more fieldwork in 2025. These additional surveys are necessary to expand the set of recordings for rare species such as alpine hares and lynx and are also useful to develop methods for the temporal analysis of wildlife behavior over multiple seasons.Building more datasets like MammAlps could radically scale up current wildlife monitoring efforts by enabling AI models to identify behaviors of interest from hundreds of hours of video. This would provide wildlife conservationists with timely, actionable insights. Over time, this could make it easier to track how climate change, human encroachment, or disease outbreaks impact wildlife behavior, and help protect vulnerable species.For more information about MammAlps and access to the dataset, visit&nbsp;<a href="https://eceo-epfl.github.io/MammAlps/" target="_blank" rel="noopener">https://eceo-epfl.github.io/MammAlps/</a>.MammAlps was selected to feature as a Highlight at the&nbsp;<a href="https://cvpr.thecvf.com/" target="_blank" rel="noopener">IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR)</a>, a top computer vision conference (11-15 June 2025).<hr>ReferencesValentin Gabeff, Haozhe Qi, Brendan Flaherty, Gencer Sumbül, Alexander Mathis, Devis Tuia. MammAlps: A multi-view video behavior monitoring dataset of wild mammals in the Swiss Alps. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, 2025.&nbsp;<a href="https://arxiv.org/html/2503.18223v1" target="_blank" rel="noopener"></a><a href="https://arxiv.org/html/2503.18223v1" target="_blank" rel="noopener">https://arxiv.org/html/2503.18223v1</a>

two roe deer foraging, with manual annotations for each individual animal. Credit: A. Mathis (EPFL)

Scientists at EPFL have created MammAlps, a multi-view, multi-modal video dataset that captures how wild mammals behave in the Swiss Alps. This new resource could be a game-changer for wildlife monitoring and conservation efforts.

AI monitors wildlife behavior in the Swiss Alps

<h2 id="isPasted">Introduction</h2>Aluminum extrusion is a vital manufacturing process that shapes aluminum into components used in aerospace, automotive, and construction industries. Driven by its lightweight, strength, and recyclability, the global market for aluminum extrusion was valued at $93.4 billion in 2023 and is projected to reach $198.1 billion by 2033, reflecting its growing importance in sustainable and innovative application. However, despite its efficiency, the process faces challenges such as speed tears, which are small fractures that can compromise product integrity and lead to increased costs and material waste.In today's competitive manufacturing landscape, traditional manual inspection methods for aluminum extrusion are increasingly inadequate due to their time-consuming nature, susceptibility to human error, and inability to detect defects like speed tears in real-time. As the demand for high-quality aluminum products grows, manufacturers increasingly turn to advanced technologies like artificial intelligence (AI) and computer vision to streamline operations.Matroid, a leader in AI-powered computer vision solutions, has emerged as a leader in this field. Their automated speed tear detection system offers a cutting-edge solution to one of the most pressing challenges in aluminum extrusion. This article explores how Matroid’s technology transforms quality control in metal manufacturing and why it represents the industry's future.<h2>The Challenges of Quality Control in Aluminum Extrusion</h2>The aluminum extrusion process is complex, with quality influenced by factors such as alloy composition, temperature regulation, extrusion speed, and die design. Maintaining uniform quality is challenging due to defects like surface scratches, die lines, and, most critically, speed tears. These tears are caused by excessive friction or localized stress during extrusion. They are particularly problematic as they can develop unexpectedly and spread rapidly, leading to significant material loss if not identified promptly. The fast pace of production lines and the subtle nature of these defects further complicate detection efforts.Historically, manual inspection has been the primary method for identifying defects. However, this approach is labour-intensive, prone to human error, and inconsistent due to operator fatigue and subjective judgment. Traditional methods also struggle with false positives and lack systematic data collection to correlate defects with process parameters, making it difficult to identify root causes and implement preventative measures. These limitations increase production costs and highlight the need for more robust quality control systems to ensure high-quality aluminum extrusion products while minimizing waste and inefficiencies.<h2>Matroid’s AI-Powered Solution: Redefining Defect Detection for Modern Manufacturing</h2>Matroid, a leader in computer vision solutions, collaborated with a leading aluminum extruder and offered an innovative approach to speed tear detection using artificial intelligence. The AI-powered speed tear detection system is a prime example of how computer vision can address the challenges of aluminum extrusion. The system analyses real-time video footage of the extrusion process, identifying speed tears as they occur. This real-time capability is critical for minimizing waste and identifying defects before they escalate into more significant issues.Optimized lighting techniques were implemented to enhance the system’s effectiveness and improve defect contrast, making it easier for AI to detect even subtle imperfections. Additionally, Matroid collaborated closely with the aluminum extruder to capture operator expertise. Sourcing and defining various speed and hot tear profiles created a plant-wide standard for all packaging operators, metallurgists, and process engineers. This collaboration ensured the AI system was trained on accurate and representative data.One of the standout features of Matroid’s solution is its adaptive AI detector. The system was trained on a diverse dataset encompassing various extrusion profiles, ensuring it could adapt to different shapes and configurations. This adaptability is crucial for manufacturers dealing with various product types. Furthermore, Matroid employed a smart annotation strategy, distinguishing between actual defects and non-defect patterns using newly created “tear definitions.” This approach allowed the detector to quickly differentiate critical tears with minimal training data, significantly speeding up the implementation process.Matroid’s solution's accuracy is one of its key advantages. The system can detect even the most minor defects, ensuring that only high-quality products move forward in the production line. Additionally, the system is highly scalable, making it suitable for manufacturers of all sizes. Matroid’s system is also designed with ease of integration in mind. The solution can be seamlessly incorporated into existing production lines, minimizing disruption and maximizing Region of Interest (ROI). This makes it an attractive option for manufacturers looking to modernize their operations without overhauling their infrastructure. By utilizing Matroid's computer vision technology, manufacturers can improve product quality, achieve substantial cost savings, and enhance operational efficiencies.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1746626166131-1746626166131.png" class="fr-fic fr-dib" /><h2>Conclusion</h2>Matroid’s AI-powered computer vision solution marks a significant advancement in quality control for aluminum extrusion manufacturing. The system’s ability to detect speed tears in real time, combined with optimized lighting techniques and adaptive AI algorithms, addresses long-standing challenges in the industry. Its seamless integration into existing production lines, scalability, and user-friendly design make it a powerful tool for manufacturers seeking to reduce waste, cut costs, and enhance product quality.AI and computer vision are poised to play a transformative role in manufacturing in the coming years. As Industry 4.0 evolves, integrating predictive analytics and Internet of Things (IoT) technologies will significantly enhance defect detection and prevention. These advancements will enable manufacturers to transition from reactive quality control to proactive strategies, effectively minimizing defects before they occur.Additionally, as global emphasis on sustainability increases, technologies like Matroid’s will become indispensable in reducing material waste and optimizing resource efficiency. Leveraging AI-driven solutions allows manufacturers to improve operational efficiency while contributing to a more sustainable and innovative manufacturing ecosystem. Matroid’s pioneering approach is solving current challenges and setting the stage for a smarter, more efficient future in aluminum extrusion.To learn more about how Matroid’s AI-driven solutions can enhance your manufacturing operations, visit <a href="https://www.matroid.com/">Matroid’s website</a>.<hr /><h1 id="isPasted">References:</h1><ol><li><a href="https://simmal.com/the-advantages-of-aluminium-extrusion-in-manufacturing/">https://simmal.com/the-advantages-of-aluminium-extrusion-in-manufacturing/</a></li><li><a href="https://vocal.media/01/aluminum-extrusion-a-vital-manufacturing-process-driving-innovations">https://vocal.media/01/aluminum-extrusion-a-vital-manufacturing-process-driving-innovations</a></li><li><a href="https://www.verifiedmarketresearch.com/product/aluminum-extrusion-market/">https://www.verifiedmarketresearch.com/product/aluminum-extrusion-market/</a></li><li><a href="https://www.alcircle.com/news/tech-trends-driving-the-aluminium-extrusion-industry-111131">https://www.alcircle.com/news/tech-trends-driving-the-aluminium-extrusion-industry-111131</a></li><li><a href="https://www.alliedmarketresearch.com/aluminum-extrusion-market-A09001#:~:text=What%20is%20the%20estimated%20industry,7.8%25%20from%202024%20to%202033">https://www.alliedmarketresearch.com/aluminum-extrusion-market-A09001#:~:text=What%20is%20the%20estimated%20industry,7.8%25%20from%202024%20to%202033</a>.</li><li><a href="https://www.eitmanufacturing.eu/news-events/news/ai-driven-in-line-quality-assurance-for-aluminium-extrusion/">https://www.eitmanufacturing.eu/news-events/news/ai-driven-in-line-quality-assurance-for-aluminium-extrusion/</a></li><li><a href="https://aimlprogramming.com/services/ai-enabled-aluminum-extrusion-monitoring/">https://aimlprogramming.com/services/ai-enabled-aluminum-extrusion-monitoring/</a></li><li><a href="https://www.matroid.com/leveraging-computer-vision-to-improve-efficiency-quality-control-and-workplace-safety-in-metal-forming/">https://www.matroid.com/leveraging-computer-vision-to-improve-efficiency-quality-control-and-workplace-safety-in-metal-forming/</a></li><li><a href="https://www.matroid.com/computer-vision-in-metal-forming-precision-efficiency-and-roi/">https://www.matroid.com/computer-vision-in-metal-forming-precision-efficiency-and-roi/</a></li><li><a href="https://www.matroid.com/case-studies/how-matroids-ai-computer-vision-can-transform-workplace-safety/">https://www.matroid.com/case-studies/how-matroids-ai-computer-vision-can-transform-workplace-safety/</a></li><li><a href="https://aluminumextrusions.net/aluminum-extrusion-quality-control/">https://aluminumextrusions.net/aluminum-extrusion-quality-control/</a></li><li><a href="https://globalaluminium.com/quality-of-aluminium-extrusion-manufacturing/">https://globalaluminium.com/quality-of-aluminium-extrusion-manufacturing/</a></li><li><a href="https://www.chaluminium.com/common-aluminum-extrusion-defects-causes-and-solutions">https://www.chaluminium.com/common-aluminum-extrusion-defects-causes-and-solutions</a></li><li><a href="https://aluphant.com/quality-control-measures-in-aluminum-extrusion-manufacturing-a-comprehensive-overview/">https://aluphant.com/quality-control-measures-in-aluminum-extrusion-manufacturing-a-comprehensive-overview/</a></li></ol>

Explore how advanced computer vision technology transforms quality control in metal manufacturing and why it represents the industry's future.

Transforming Aluminum Extrusion: Advanced Computer Vision Solutions for Enhanced Quality Control and Industry Efficiency

In a broad sense, "machine vision" refers to the utilization of visual information and optical information using digital technologies. However, because it is being introduced to production sites in the manufacturing industry, machine vision generally refers to digital image technologies utilized in automatic inspection and process automation in factories. Machine vision consists of the following:・Imaging section (cameras) ・Interface section (board/general-purpose interface) ・Processing engine section (computer) ・Processing control section (software)etc. The image data obtained by the imaging section is loaded into the processing engine via the interface section. In the processing engine section, the processing control section performs binarization*1, edge detection*2, and other forms of image processing. Next, it outputs the image data that has completed image processing to robots and industrial equipment.*1: Processing that converts the pixel values (threshold values) that are preset for the image to the two values of black and white. *2: Processing that detects the bright and dark areas in an image.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327121894-img-0001-en.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 1: Machine vision configurationMachine vision is used in automatic inspection, automatic recognition, and automatic measurement, etc. For example, automatic inspection and automatic recognition are very fast compared to manual inspection. Moreover, machine vision used in automatic measurement can accurately identify small value differences that humans cannot measure. In addition, linking machine vision to AI functions is expected to produce more intelligent uses going forward.<h2>Differences Between Machine Vision, Computer Vision, and Robot Vision</h2>Computer vision and robot vision are technologies that are similar to machine vision, but they each play different roles. Here, we explain how their roles differ.<h3>Differences Between Machine Vision and Computer Vision</h3>"Computer vision" is a technology that adds equipment control functions to machine vision functions. Computer vision can control various types of equipment based on data output by machine vision. For example, this would make it possible to combine the image data output by machine vision with CAD data generated by the computer to control processors or to adjust the conveyor transport speed based on image data that captured the progress of line work. In addition, computer vision has a broad scope of utilization that naturally includes smart factories as well as referee assistance systems and virtual replays in sports and projection mapping, etc. in entertainment.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327141371-img-0002-en.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 2: Example of computer vision utilization<h3 id="isPasted">Differences Between Machine Vision and Robot Vision</h3>"Robot vision" is a technology that adds machine vision to robot visual functions. Machine vision attaches cameras to various types of equipment to obtain image data, but robot vision attaches cameras to the end of robot arms to obtain image data. This enables the robot to operate based on visual information obtained by itself. For example, a robot that does not use robot vision can only execute certain operations programmed in advance. However, when a robot equipped with robot vision moves its robot arm closer to an object, it then sequentially identifies the target based on image data from the camera to execute operations. Because this type of operational control is possible, a robot equipped with robot vision is capable of detailed, precise, and flexible operations that cannot be specified with a program.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327158790-img-0003-en.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 3: Example of robot vision utilization<h2 id="isPasted">Example of Machine Vision Utilization in a Smart Factory</h2>Machine vision is used in various types of work in a smart factory. Here, we introduce an example of machine vision utilization.<h3>Reading Characters at Ultrahigh-speed – OCR (Optical Character Recognition/Reader)</h3>OCR is a technology that scans handwritten or printed characters with an image scanner or digital camera. Machine vision scans the 2D code, serial number, and date of manufacture, etc. printed on products. The scanned data is utilized in inspections of the print quality or whether the correct characters are printed as well as the screening of the next process, etc. In many cases, the products being scanned by OCR pass through the line at high speeds. Therefore, machine vision uses cameras that can capture images at high speed and image processing systems that are capable of high-speed processing.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327179168-img-0004+%281%29.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 4: OCR on a manufacturing line<h3 id="isPasted">Scanning Shapes in 2D and 3D – Visual Inspections</h3>The utilization of machine vision is considered to be the most effective means of visual inspection on a manufacturing line. Because machine vision-based visual inspection is able to inspect the accuracy of products after cutting and molding and the mounting condition of components, etc. with high speed and accuracy, it is essential for improving production efficiency and maintaining quality. The surfaces of target objects can vary widely in terms of their glossiness and light absorption, etc. The lighting used in machine vision enables products with various surface conditions to be inspected by changing the method of illumination and the color of the light rays. In addition, 3D cameras equipped with a stereophonic*3, time-of-flight*4, and other measurement systems as well as structured light*5 and other imaging systems can also measure the depth.*3: Method that uses two cameras to perform triangulation and measure the distance to an object. *4: Method that measures the distance to an object according to the time it takes for light emitted by a light source to reflect off of the object and return to a light detector. *5: Lighting method that emits light with a specific shading pattern.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327197939-img-0005.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 5: Visual inspection with a 3D camera<h3 id="isPasted">Grabbing Specific Components – Picking</h3>"Picking" on a manufacturing line refers to the task of collecting (pickup) the necessary components and products. Utilizing machine vision makes it possible to identify and pick components with the specified color and shape even if components and products with various colors and shapes are mixed together in one case.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327216910-img-0006.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 6: Picking by a robot<h3 id="isPasted">Providing Visual Functions to Robots – Robot Operational Control</h3>In recent years, industrial robots have also been used in precision work due to their high operating precision. For example, they play an active role in installing, soldering, and welding microcomponents. However, controlling such work with programming requires high-precision positioning devices. With robot vision utilizing machine vision, high-precision positioning devices are not required for the robot to perform precise positioning with the image data from the camera mounted on the end of the robot arm.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327233332-img-0007.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 7: Installing microcomponents with a robot<h2 id="isPasted">Future of Machine Vision – AI Vision</h2>As various types of industrial equipment shift to using AI, one form of machine vision evolution that shows promise is "AI vision."<h3>What is AI Vision?</h3>AI vision is a system that equips lightweight AI chips in smartphones, etc. with cameras and camera functions to make inferences about captured image data. AI vision resembles computer vision in that it uses AI (artificial intelligence), but computer vision mainly makes inferences with a cloud AI service (Figure 8). On the other hand, AI vision equips an AI module in an edge device to make inferences, and the inference results are stored on the edge server. The edge server communicates with the cloud AI only when necessary. (Figure 9).<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327267580-1721327267580.png" class="fr-fic fr-dib">Therefore, AI vision has the following advantages over computer vision: ・Able to decrease the volume of communication with the cloud AI service ・Able to prevent external leaks of data collected by the sensors ・Superior responsiveness Furthermore, the cameras and smartphones used in AI vision are called "edge AI devices" while the method that makes inferences on the edge device is called "edge AI."<h3>What Technologies Are Required to Realize AI Vision?</h3>Although AI vision has the advantages described above, AI modules equipped on edge devices require the following types of functions. ・Superior energy conservation due to restrictions on battery capacity ・Ability to perform high-speed processing due to the responsiveness requirement ・Superior heat dissipation due to installation in a limited space More than anything, it is important that the AI modules be compact.<h2>Summary</h2>This article explained the topic of machine vision and the similar phrases of computer vision and robot vision as well as AI vision. Going forward, it is expected that machine vision and AI will evolve in a close relationship with the evolution in deep learning for image data. In addition, new AI systems and products are expected to be developed through the proper use of cloud AI and edge AI. These trends will likely become essential technologies for smart factory robots, automatic inspection, automatic recognition, and automatic measurement equipment that require visual information that is closer to the human eye and decision-making capabilities that exceed that of humans.

In a broad sense, "machine vision" refers to the utilization of visual information and optical information using digital technologies.

The Eyes of Smart Factories - What is Machine Vision?

Industry 5.0 is a new phase of industrialization that builds upon the fourth industrial revolution (Industry 4.0) by adding a people-centric approach. It is characterized by the extensive use of automation, data exchange, and smart manufacturing technologies in a manner that fosters the well-being of the workers. It bridges the gap of the unfulfilled promise of full automation from Industry 4.0 in processes where a human in the loop is still needed, &nbsp;increasing the competitiveness of the industry and helping attract the best talents.The ultimate goal is to create human-centric, efficient, flexible, and responsive production environments that can quickly adapt to changing market demands.To achieve this goal, this next generation of manufacturing environments leverages the latest available technologies to the entire production process. One of the key operational aspects of manufacturing is intralogistics -understood as the management, control, and optimization of internal material and information flows.&nbsp;Intralogistics 5.0 leverages automated material handling, Internet of Things (IoT), data analytics, and Artificial Intelligence (AI) to create smart, interconnected, data-driven production environments and empower human workers.&nbsp;This intralogistics evolution is not only adding efficiency to industrial processes but is also driving a shift in manufacturing, from large, rigid systems into modular, flexible, and robot-supported self-optimizing ones.<h2>Optimizing Material Handling with Visual AI-Enabled Robotic Vehicles</h2>Autonomous Mobile Robots (AMRs), equipped with Visual AI and orchestrated through a Fleet Management System (FMS) are at the forefront of this transformation. The shape and function of these AMRs range from autonomous forklifts, shuttle systems, tugger trains, and stackers to platform or mouse robots, goods-to-person, and pick-assist robots. Their material transport activities are ideally tracked directly by the Enterprise Resource Planning (ERP) system of the factory.AMRs are designed to navigate with agility and accuracy through complex factory environments using a combination of sensors, cameras, and real-time data processing. The integration of Visual AI enables them to perceive their surroundings, identify objects and obstacles, make informed decisions, and share information in real-time, adding a number of benefits at each step of the process:<ul><li id="isPasted">Enhanced Goods-In: Vehicles equipped with Visual AI contribute to this stage by autonomously transporting goods from receiving areas to designated storage locations reducing manual work and extending operation hours. Additionally, the visual input ensures accurate identification and categorization of incoming goods, feeding the ERP, and laying the foundation for precise inventory management.</li><li>Optimized Storage and Retrieval: Visual AI allows robotic vehicles to analyze available storage space and stack goods efficiently, optimizing storage capacity, ensuring timely retrieval of items, and providing real-time inventory updates.</li><li>Streamlined Production Supply: Visual AI makes autonomous intralogistics vehicles inherently flexible, thus they can optimize routes dynamically, ensuring a continuous and streamlined supply of materials to production lines.&nbsp;</li><li>Accurate Order Picking: Visual AI technology supports workers in order picking, significantly reducing error rates and ensuring accurate order fulfillment.‍</li><li>Efficient Dispatch and Goods-Out: Visual AI enables robotic logistics vehicles to perform order verification and optimize load onto transportation vehicles, adding efficiency to the preparation and organization of finished goods for delivery to customers or distribution centers.</li></ul><h2>Flexible Manufacturing</h2>Besides the optimization across the material handling process, and on account of their inherent reconfigurability, Visual AI AMRs can flexibly support diverse production scenarios, such as batch, mass, or customized production, and different manufacturing layouts, from linear or modular to cellular.&nbsp;Serving as flexible links between individual production stages, these robots are also fundamental in the transition from traditional rigid conveyor-belt-based manufacturing to flexible-cell manufacturing, where production units can be reconfigured and rearranged without downtime.&nbsp;This flexible movement and connection of production cells, create dynamic and modular production environments that can adapt to different product variants and volumes, to respond to changing customer demands and market conditions.<h2>Challenges on the Path Towards the Factories of the Future</h2>Visual AI allows operating this Factory of the Future today. While the technology is already here, its adaption and implementation are only starting to pick up speed. The transition to Intralogistics 5.0 and new levels of automation is mostly gradual and incremental as it requires interoperability, scalability, and human-robot collaboration.<h3>Interoperability</h3>Many manufacturing facilities and warehouses are currently operating with mixed fleets that consist of both manned and unmanned material-handling vehicles, and an initial and crucial challenge in the path toward Intralogistics 5.0 is to integrate both types of shop floor traffic into an all-encompassing FMS and ERP.At this stage, equipping manned vehicles -like forklifts, tuggers, or stackers- with Visual AI-enabled Real Time Location System (RTLS) to track their location and feed the system with the visual data they capture would enable an FMS to orchestrate floor traffic holistically.Adding observability to manned traffic and integrating it into an FMS cancels out typical shortcomings of mixed fleets such as human drivers colliding with AMRs, inefficient manned routing unobservable by the FMS, or inventory compromised by human errors.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzAyOTAzNTU1NTQzLU1hbm5lZF9Vbm1hbm5lZF9UcmFmZmljX0ludGVncmF0aW9uX1NldmVuc2Vuc2VfUm9ib3RpY3MucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib">Visual AI-enabled RTLS feeding manned traffic data into an FMSThe Visual AI-enabled RTLS adds value in manufacturing and warehousing unlocking completely new opportunities in material handling applications, where some operations will continue to be performed using manned vehicles for the next 20+ years.To make the benefits of holistically managed mixed fleets broadly accessible to the market, Sevensense Robotics is developing a Visual AI-enabled RTLS, which is already available to early adopters.<h3>Scalability</h3>In many cases, as part of their automation journey, manufacturers opt for initially commissioning AMRs at a small scale in a dedicated sector of their facilities. However, expanding the AMRs’ operating space from a wing of a plant or to the entire factory has traditionally been time-consuming.Until now, it implied a complex and disruptive endeavor of stopping operations and dedicating specialized professionals to the map expansion. including running a vehicle in manual mode to scan the new areas of the facility where robots will operate, and then manually editing and merging the new maps.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1702909540635-Continous+map+updates+V2.gif" style="width: 302px;" class="fr-fic fr-dib">Lifelong Visual SLAM enables seamless map updatesTo solve this scalability blocker, Sevensense has developed a solution that makes the map expansion seamless. Combining continuous map updates -also known as Lifelong Visual SLAM- with swarm intelligence, robots equipped with this technology can automatically map the new sections of the environment and share the extended map with the rest of the fleet. All they need to start their exploration is an anchor point in the existing map and the expansion is computed on-the-fly.By automatically incorporating relevant changes in the environment, Sevensense’s Lifelong Visual SLAM also offers the key benefit of lifelong robust, high precision localization &lt;5mm (&lt;0.2in) without markers, reflectors, or beacons, allowing the robots to operate in dynamic, ever-changing environments.<h3>Human-Robot Collaboration</h3>An increasing number of companies are incorporating AMRs into their operations. However, the lack of seamless human-machine collaboration solutions poses the need for dedicated robotic operations zones or the introduction of robotic shifts, restricting their flexibility and efficiency while also limiting the ROI.Visual AI AMRs have the built-in capability of understanding the context of the environment, distinguishing between unobstructed paths, obstacles -and types of obstacles-, and adapting their behavior accordingly in real-time.When the robot encounters an obstacle, the system identifies the type of object, then estimates its velocity and predicts its trajectory, calculates the most efficient path, and automatically performs an obstacle avoidance maneuver, rerouting the robot.This navigation mode can boost efficiency, in particular in scenarios when robots share the shop floor with other robots, but it may not be the optimal solution when robots share the shop floor with people.&nbsp;On the one hand, as human behavior is less predictable, robots have less visibility of a person’s movement intent than they have on other mobile robots or RTLS-tracked manned vehicles. On the other hand, having a robot overtake people while walking across a facility may be perceived as unpredictable and unsafe, making human workers feel anxious.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzAyOTAzNzkyNTUyLUh1bWFuX01hY2hpbmVfQ29sbGFib3JhdGlvbl9Sb2JvdF9OYXZpZ3Rpb25fU2V2ZW5zZW5zZV9Sb2JvdGljcy5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib">Visual AI is key to developing collaborative AMRsSolving the human-robot collaboration challenge is key to enabling Industry 5.0 and is motivating Sevensense’s work to solve this issue by combining scene understanding and a people-centric approach, to turn AMRs into Co-AMRs. The solution involves Visual AI perceiving the obstacle as a person and then commanding the robot to perform a people-friendly behavior such as stopping and waiting for the person to continue their movement or complete their task.To learn more about Sevensense Robotics and its Visual AI technology, visit <a href="https://www.sevensense.ai/" target="_blank" rel="noopener noreferrer">sevensense.ai</a>

Visual AI enables Autonomous Mobile Robots to fulfil Industry 5.0 requirements

Visual AI is applied to overcome the obstacles in the path of the Factories of the Future

Autonomous Mobile Robots Powering Industry 5.0

The intralogistics specialist STILL, a subsidiary of the KION Group, leads the German project consortium of the Intelligent Motion Control -IMOCO4.E- research project.The goal of the IMOCO4.E project is to enable intelligent trucks to navigate autonomously in a factory or warehouse.In numerous warehousing and production settings, semi-autonomous trucks are already in use and have already proven to be profitable assets. However, they remain constrained by certain limitations.&nbsp;While these trucks can presently detect obstacles and apply brakes independently, achieving complete self-driving capabilities requires further development. Intelligent trucks should be able to navigate warehouse environments fully autonomously. They should be able to understand their surroundings, avoid obstacles, and find the most efficient routes.The IMOCO4.E project focuses on applying solutions based on Artificial Intelligence (AI) to address challenges in four main scenarios: navigation, collection, and movement of goods, and delivery to their final position.In this context, STILL's iGo neo platform, which already comes very close to the idea of this autonomously driving vehicle due to its 'intelligent' equipment and the resulting capabilities, &nbsp;was equipped with an Alphasense Autonomy unit from Sevensense Robotics.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk5NDQ4MDQ2ODU4LUFwbGhhc2Vuc2UgQXV0b25vbXkgaW5zdGFsbGVkIG9uIGlHTyBOZW8ucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib" alt="Alphasense Autonomy unit installed on a STILL iGo neo platform - Image credits STILL">Alphasense Autonomy leverages Visual AI technology to provide precise navigation and obstacle avoidance capabilities in complex and dynamic warehouse environments -with racking, inbound and outbound areas, and even when transitioning from indoors to outdoors.The Visual AI system maps out the environment and provides precise positioning without using any type of infrastructure.In addition, Sevensense is prototyping the implementation of AI to perform Semantic Scene Understanding, which allows the vehicles to extract information about the objects in their environment, to detect, classify, determine the position, -and when applicable, predict the movement of obstacles e.g. people and vehicles, and other objects of interest such as pallets, docking or charging stations.With this information, the vehicles are able to adapt their navigation behavior -stopping, slowing down, or overtaking obstacles-, based on real-time perception, allowing them to interact with the environment, for example, docking, picking, or charging, while safely sharing the floor with moving elements like people and other vehicles.“In the IMOCO4.E4.E project, STILL is partly refining and extending existing approaches with regard to general interactions of his iGo neo platform in an unknown environment. The high precision and reliability of the localization provided by Sevensense's Visual AI technology is of great benefit here",&nbsp;says Ansgar Bergmann, responsible for the IMOCO4.E project at STILL.The collaboration between Sevensense Robotics and STILL within the IMOCO4.E research project marks a significant stride toward the realization of fully autonomous trucks in warehouse and factory environments.The project is scheduled to be completed in the fourth quarter of 2024.<a href="https://www.sevensense.ai/" target="_blank" rel="noopener noreferrer">Learn more about Sevensense Robotics and its Visual AI technology</a>

STILL's iGo neo platform operating in a warehouse setting - image credits STILL

Sevensense’s Visual AI implemented into STILL's iGo neo platform for precise Autonomous Navigation and Pallet Handling

Still Incorporates Visual AI Technology on IMOCO4.E Research Project

<h2 id="isPasted">Its impact on Visual SLAM and beyond</h2>It is clear that Artificial Intelligence (AI) is here to stay and there is a need for an open discussion, to bring some light into the AI black box.At Sevensense Robotics we strongly believe that transparency builds trust, this is why we decided to tackle the elephant in the room and dissect the role that AI plays in our offering.Let’s start by exploring what AI is and the difference between Machine Learning (ML) and Deep Learning (DL).The original definition of AI, coined in 1955 by John McCarthy -one of the founding fathers of the discipline- was really broad: “The science and engineering of making intelligent machines.”Nowadays, we conceive AI as a subset of computer science focused on creating systems that can replicate human intelligence and problem-solving abilities. It involves any technique that enables machines to mimic human intelligence, using logic, if-then rules, decision trees, and machine learning -including deep learning.Machine Learning (ML) is a subset of AI that intersects with statistics. It involves the development of algorithms and statistical models that enable computers to learn and make predictions or decisions without being explicitly programmed; instead, they recognize patterns in the data and make predictions once new data arrives.Deep Learning (DL) is a specialized subset of Machine Learning that analyzes data with a logical structure similar to how a human would draw conclusions.&nbsp;To achieve this, DL applications use a layered structure of algorithms called an Artificial Neural Network (ANN). It is characterized by its deep layers of interconnected nodes or neurons, which allows it to automatically learn hierarchical representations of data, making it well-suited for tasks like image and speech recognition.The three technologies are nested one within the next:<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk3MDIxMTQzOTQ1LURpZmZlcmVuY2VzIGJldHdlZW4gQUksIE1hY2hpbmUgTGVhcm5pbmcgYW5kIERlZXAgTGVhcm5pbmcgKDIpLnBuZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 300px;" class="fr-fic fr-dib" alt="Differences between AI, ML and DL">Visual AI refers to Artificial Intelligence applications that focus on enabling machines to interpret, understand, and process visual information, such as images and videos. Visual AI technologies employ advanced Machine Learning algorithms, Deep Learning models, and computer vision techniques to analyze and extract meaningful insights from visual data.AI-Powered Robot AutonomySevensense uses Visual AI to allow robots to identify, understand, and act on imagery and visual data -captured by cameras- just like humans so that they can navigate autonomously.By processing camera images with AI, the process known as Visual Simultaneous Localization and Mapping (SLAM) builds a rich 3D map of the environment and precisely localizes within it.Sevensense’s Visual SLAM technology comprises proprietary and lightweight AI sensor fusion algorithms to enable robots to precisely estimate their position. This is achieved by extracting distinctive elements from the images, for example, a corner of a window, and using these to build a representative yet compact 3D model of the environment.&nbsp;To deliver robust localization under all conditions, including lighting and environmental changes, Sevensense employs AI -more concretely Deep Convolutional Neural Networks- for feature extraction. Incorporating AI for this purpose makes it possible to overcome a limiting constraint of legacy SLAM technology, which until now could not cope with significant illumination and other environmental changes, leading to failures during autonomous navigation in difficult settings.Sevensense also uses machine learning to maintain the visual maps of dynamic spaces to ensure robust localization and reusability of the initial map of the environment. The solution known as Lifelong Visual SLAM is basically a multi-session map that uses the initial map as the basis and later on, the changes in the environment are added to it to enrich it. All of this, without the need to run Visual SLAM from scratch on each session, increases operational efficiency.The Lifelong Visual SLAM solution enables building and maintaining always-up-to-date maps. These maps incorporate data from multiple conditions of the environment, such as during cloudy- and sunny weather, bright and dark lighting situations, and changes in the elements present in the area, e.g. changes in factory floor layouts, or simply in the warehouse or shop inventory. All of these changes are detected automatically and incorporated into the lifelong map.This allows the maps to be used for visual positioning in environments that change considerably over time.To enable autonomous navigation even in complex and ambiguous environments, a Convolutional Neural Network is employed to perform Semantic Scene Understanding making it possible to detect, classify, determine the position, -and when applicable, predict the movement of obstacles e.g. people and vehicles, and other objects of interest such as pallets, trolleys, floor markings, docking or charging stations.After the robots have gained insights into their environment and the expected behavior of the elements around them, it is time for them to plan their next move. Reinforcement Learning makes it possible for robots to adapt their navigation behavior -stopping, slowing down, or overtaking obstacles, based on real-time perception, allowing them to interact with the environment, for example, docking, picking, or charging, while safely sharing the floor with moving elements like people and other vehicles.All of these processes happen on the Edge, on the multi-camera Alphasense Edge AI device -a GPU-accelerated compute unit- embedded in the robot. By bringing computational power closer to the data source, Sevensense minimizes latency challenges -enabling nearly simultaneous, real-time perception and decision-making- and addresses security and privacy issues typically present in solutions purely based on cloud computing, where visual data needs to be recorded and transmitted to remote servers for processing.&nbsp;AI-Powered Continuous Performance ImprovementTo exceed the ever-increasing performance requirements of the market, both in terms of accuracy and robustness, Sevensense leverages ongoing cutting-edge developments in the fields of AI, computer vision, and robotics.&nbsp;Sevensense uses AI models to enrich its simulation, training, and testing infrastructure.Combining Generative AI and Digital Twins, Sevensense is able to create the most challenging training and testing environments to continuously validate its software stack.Visual AI AnalyticsThe visual data captured by the cameras, apart from providing inputs for best-in-class localization and autonomy performance, also unlocks the potential for providing ground-breaking additional services.The rich visual data captured by the cameras allows Sevensense to offer further Visual AI analytic services focused on providing organized data and insights -beyond Visual SLAM- such as asset tracking, stocktaking, surveillance, and inspection, as an additional safety or reporting data layer.

Visual AI enables autonomous navigation and an array of object-recognition-based applications

Its impact on Visual SLAM and beyond

AI in Robotics Perception and Autonomous Navigation

<h2>Challenges of self-localization in mixed in- and outdoor environments</h2>Knowing its own location or position within an operating environment (a.k.a. self-localization) is one of the key capabilities any mobile robot must master to be able to get from A to B. Yet, depending on the environment the robot is operating in, self-localization can impose vastly different challenges, rendering the choice of sensor technology and software algorithms far from a trivial task.Indoors, 2D LiDAR technology is widely used, as an example, on AGVs and AMRs for material handling operations. Such a technology allows to measure the geometric structure of the near-by environment (e.g. walls, shelves) to localize in pre-built maps. However, it shows its limitations in environments which do not have unique geometric features (e.g. a long corridor) or are highly dynamic especially if shared with people.&nbsp;On autonomous vehicles for outdoor environments, instead, satellite navigation systems such as GPS or Galileo are often the first choice of sensor technology for self-localization, for good reasons: The GPS receivers are very cheap, and they yield an easily interpretable position in a known global reference coordinate system (e.g., WGS84).&nbsp;However, an accurate GPS position requires unobstructed skies. This is a limitation that is violated in several circumstances like urban surroundings, company campuses, &nbsp;logistic yards and airport tarmac. In fact, these outdoor places often require to operate close to or even inside buildings or tunnels. Due to these restrictions, mobile robots operating outdoors or in mixed indoor-outdoor settings cannot rely solely on GPS for self-localization.&nbsp;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk1MTMxNTk3MzM0LU1peGVkIGluZG9vciBhbmQgb3V0ZG9vciBwb3NpdGlvbmluZy5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib">Mixed indoor and outdoor environments are challenging for GPS and LiDAR solutions aloneEven though 2D LiDAR sensors can be employed outdoors too, using them in the open only works in well-structured spaces of limited extent or requires additional infrastructure such as laser reflectors in order to work better., or to a better extent thanks to the installation of additional infrastructure such as laser reflectors. 3D LiDARs, on the other hand, remain a very expensive sensor gear, despite all efforts in cutting down the costs in recent years.<h3>The strength and challenges of visual localization</h3>As an alternative to GPS and LiDAR sensors, cameras are able to reliably sense the environment both indoors and outdoors. Vision thus has the capability to bridge this gap left uncovered by GPS or LIDAR sensors. It offers highly accurate and reliable self-localization in both indoor and outdoor environments with a very cost effective sensor gear.&nbsp;Visual self-localization does not come without its very own challenges. Cameras naturally provide good images in well illuminated surroundings, while during night time outdoors, or in dim indoor areas, visual self-localization becomes more difficult. However, the selection of camera sensor chips with a high dynamic range &nbsp;has shown to allow proper images to be taken even under challenging illumination conditions.<blockquote>Instead, LiDAR sensors can be employed. However, using 2D LiDARs outdoors only works in well-structured spaces of limited extent. As an alternative to GPS and LiDAR sensors, cameras are able to reliably sense the environment both indoors and outdoors.</blockquote>Further, and in contrast to GPS or LiDAR sensor data, images taken by cameras of the same environment can look vastly different under varying seasonal, weather, or lighting conditions. This change in appearance renders accurate localization difficult. Additionally, geometric characteristics are often unfavorable for visual localization outdoors compared to indoors, as there is often less stable, human-made structure visible, plus it is often located at a considerably farther distance. Therefore, smart algorithms to work with visual data are paramount.<h3>Tackling the challenges of visual localization with AI</h3>To master these challenges, Sevensense employs a smart AI algorithm based on lifelong visual mapping that enables building and maintaining always-up-to-date maps of the environment. These maps incorporate data from multiple appearances of the environment, such as before- and after remodeling of a room, during cloudy- and sunny weather, or bright and dark lighting conditions. This allows the maps to be used for visual localization under practically any appearance condition.&nbsp;At the core of this lifelong mapping process is a software algorithm capable of merging visual data from multiple traversals through the same environment, recorded under potentially very different appearance conditions, into a single, geometrically consistent map. This process is fully automated, resulting in having a multi-appearance map ready for reliable localization after traversing through the environment only a handful of times beforehand.&nbsp;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk1MTMxNzMxMTk3LUxpZmVsb25nIHZpc3VhbCBtYXBwaW5nIGFsZ29yaXRobXMgdXNlZCBieSBTZXZlbnNlbnNlLnBuZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 300px;" class="fr-fic fr-dib">Smart lifelong visual mapping algorithms allow for always-up-to-date mapsFurther, continuously updating this map over time naturally captures structural change in the environment. This way events like remodeling of rooms, construction sites or temporary modifications in street or room layouts don&rsquo;t disrupt everyday operations. These changes are detected automatically and swiftly incorporated into the lifelong map.&nbsp;In particular, no expert knowledge or intervention is necessary at any point in the process. Just as generating the initial map of the environment can be done by anyone without any prior training or expert know-how, updating the maps at any later point is done automatically by Alphasense Position. This considerably lowers the costs of maintaining the system in a long-term horizon.&nbsp;<h3>Vision is the optimal solution for indoor- and outdoor self-localization</h3>By employing this lifelong mapping technique, vision overcomes its limitations and turns out to be the go-to technology for cost-effective, but still reliable and accurate self-localization both indoors and outdoors. In particular, it offers seamless transition from indoor to outdoors and vice-versa, as the same camera sensors and the same map can be used to localize in both types of environments. This is a fundamental advantage over any technology primarily relying on GPS for outdoor localization and navigation that can revolutionize the logistics, delivery, cleaning or supply chain processes.<blockquote>By employing this lifelong mapping technique, vision turns out to be the go-to technology for cost-effective, but still reliable and accurate self-localization both indoors and outdoors.</blockquote>Sevensense has already successfully demonstrated the unique benefits of visual localization in a <a href="https://www.sevensense.ai/blog/kyburz" target="_blank" rel="noopener noreferrer">proof-of-concept study in partnership with KYBURZ Switzerland AG</a>. By equipping a mail delivery robot with Alphasense Position, accurate and reliable self-localization of the vehicle became possible at any time indoors, as well as outdoors, where LiDAR and GPS solutions alone failed.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1695132745402-Reliable+positioning+indoors+and+outdors+cameras+overcome+LiDAR+and+GPS+limitations+Sevensense.jpeg" style="width: 300px;" class="fr-fic fr-dib">Delivery robot used to benchmark Alphasense Position, 3D LiDAR and GPS localization performance[the missing image will go here once the CMS issue is fixed]<blockquote>By equipping a mail delivery robot with Alphasense Position, accurate and reliable self-localization of the vehicle became possible at any time indoors, as well as outdoors, where LiDAR and GPS solutions failed.</blockquote>Along a similar vein, the indoor- and outdoor capabilities of Alphasense Position will open up countless other opportunities for bringing cost-effective autonomy to robotic platforms operating in challenging environments. Delivering goods across multiple buildings on a campus, or material handling in warehouses undergoing frequent layout change have never been simpler before.<h3>With vision towards a fully autonomous future</h3>Self-localization is essential for robots to safely navigate in changing environments and achieve full autonomy. By employing cameras for this task together with smart algorithms for building and maintaining maps, autonomy can be provided at a fraction of the costs - be it for the sensor gear itself, as well as operating costs over the robot&rsquo;s lifetime.&nbsp;At Sevensense we are constantly pushing cutting edge technologies forward to deliver affordable autonomy solutions for new application fields.You can find more informative material about Alphasense Position on its <a href="https://www.sevensense.ai/product/alphasense-position" target="_blank" rel="noopener noreferrer">product page</a>.

Test vehicle benchmarking cameras, LiDAR and GPS systems

Breaking Free from LiDAR and GPS Constraints: Harnessing Vision for Reliable Positioning, Indoors and Outdoors

Why reliable positioning indoor and outdoor requires cameras

Sevensense Robotics builds the eyes and brains of mobile robots. &nbsp;We strive to provide the most flexible and robust Visual AI solutions to give robots the ability to identify, understand, and act on imagery and visual data -captured by cameras- just like humans so that they can navigate autonomously.Our technology enables robots to adapt their navigation behavior based on real-time perception, allowing them to complete their missions, for example, docking, picking, or charging, while sharing the floor with moving elements like people and other vehicles.To ensure everything we ship has the high performance our customers expect, we stress-test our technology in different challenging settings.

Sevensense Robotics pushes the boundaries of Visual SLAM technology

Visual AI

ORGANIZATIONS.

SHAPING THE INDUSTRY.

Sony

Latest Posts