In this episode, we explore how Robuscope is reimagining the future of AI testing, offering groundbreaking solutions for improving the accuracy and reliability of artificial intelligence systems.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/270862db-fef1-42ce-8710-bb98f6e896b3?dark=false" class="fr-draggable" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true"></iframe><hr><h3 id="isPasted">Episode Notes</h3>(0:50) - <a href="https://www.wevolver.com/article/ai-testing-reimagined-with-robuscope" target="_blank">AI Testing Reimagined with Robuscope</a>Become a founding reader of our newsletter: <a href="http://read.thenextbyte.com/" target="_blank">http://read.thenextbyte.com/</a><hr><h2 data-attrid="title" data-local-attribute="d3bn" data-ved="2ahUKEwiBhLnmu4aAAxVmwjgGHcDWC38Q3B0oAXoECEgQEQ" id="isPasted">Transcript</h2>If you've ever seen an AI model, a self-driving car, your phone, something like that, make a mistake, that's okay because AI isn't always ready for the real world. And in today's episode, we're talking about a testing model that you can use today. It's free and it's online to help users and developers submit their AI models and receive detailed reports to understand the strengths and weaknesses of their AI models.I'm Daniel, and I'm Farbod. And this is the NextByte Podcast. Every week, we explore interesting and impactful tech and engineering content from Wevolver.com and deliver it to you in bite sized episodes that are easy to understand, regardless of your background.&nbsp;Daniel:&nbsp;What's up, folks? Today we're talking about AI, but in a little bit of a different way than we've been talking about it in the past. Usually we're talking about, here's AI that's designed for a specific purpose. Here's the problem. Here's how AI helps solve it. And here's the significance. But in today's episode, we're talking about AI in a different twist in that we realize that AI is becoming increasingly integrated into all parts of applications around our lives, a lot of this being critical, autonomous vehicles, medical diagnostics, et cetera, knowing that the more involved AI becomes in our lives, the more crucial it is that they're actually really reliable. So, in today's episode, we're talking all about online tool that's actually free and it's available to help everyone rigorously test the robustness of their own AI model. So, for our listeners who are AI engineers out there or who know them, this is an interesting tool for you to share, to listen to, to evaluate, to provide some feedback on. Because in this case, inadequate testing of AI models in complex real-world scenarios causes bad outcomes. So, our goal here is to reduce AI failures in high-stakes environments and make AI as reliable and safe as we can.Farbod:&nbsp;And just like adding some context there, right? Imagine your doctors become more and more reliant on AI models for analyzing your CAT scans or your MRI scans. Now imagine those models are actually not very robust, meaning that it gives an answer about if you have cancer or not, while it's like maybe 60 or 70% certain, instead of saying, hey, I'm not that sure. And then your doctor just blindly accepts that. Now, I'm not saying this is what happens, I'm just saying envision that future. Well, to get ahead of that, you need to have good testing, good validation of these AI models before they get launched out to the general public, right? And that's what these folks are trying to do.Daniel:&nbsp;No, I'm with you, man. And it's like, it's not just the scenario, I think, that you mentioned where somebody's trusting an AI model that doesn't have 100% certainty. But what if this AI model is 99% certain, but it's wrong. Like that's a potential outcome as well, right? Where AI models have been trained and have really, really high fidelity, high reliability, high accuracy in a sterile scenario with the training data. But then in real world applications, they know that there's a twist to the data or some boundary case that makes the AI model not as reliable as it was before. So, there's a lot of big companies out there that are pushing AI through without testing. We see those things go viral on social media. I think about Facebook, was it, that used AI in their hiring process and it ended up biasing their hiring.Farbod:&nbsp;Amazon.Daniel:&nbsp;Amazon. Yeah. Boom, used AI in their hiring process and it ended up biasing their hiring in a way that it was training off of existing resumes in their employee group. And it ended up furthering the lack of diversity that was already existing in their workforce. So that's just one example. I see stuff all the time, every single day. I don't wanna like take stabs at too many companies because it's a fundamental challenge. It's a fundamental challenge with AI development. And I appreciate the, is it Fraunhofer? Fraunhofer?Farbod: Fraunhofer, yeah.Daniel:&nbsp;Fraunhofer Institute that created this tool that we're going to talk about today called Robuscope. It's a free tool, tests AI in all sorts of tough situations, right, the boundary conditions, a comprehensive testing framework, almost a report card or a health check on your AI model so that you can input data points, test a model. It gives it a comprehensive evaluation in various scenarios to identify potential weaknesses, then gives you a report, a performance report to say like, hey, this is where your AI tool may, or your AI model may be unreliable. I think of it like a, I don't know, webpage loading test that's like this boilerplate test that developers can use to understand whether their page is loading fast enough or not. But in this case, use it with AI models to make sure that they're robust.Farbod:&nbsp;Yeah, definitely. And one thing that came to mind when I saw this is like, I don't do this but the discourse I've seen around AI model testing is very interesting. Essentially, it's a product that you can only do black box testing on. So that leaves a lot of the test engineers in a weird spot of, okay, like how much testing, how much of an in-depth testing can we actually do here? And like you were saying earlier, we've seen a lot of companies just kind of forego the testing process and just shipping products because the competition is so hot, or maybe it's just not worth it. Basically, the manual testing process of these AI models can be quite monotonous, can be quite long, can be quite expensive, which all stand against these tough deadlines for shipping products that our tech companies have become accustomed to. So now you have something that is free, easy to use, quite fast, that at the very minimum gives you, like you're saying, just pulse check that everything is as good as it can be. And you don't even get a yes or no, you get a pretty detailed report card of how it's doing in these different robustness and uncertainty metrics.Daniel: I think it's super sick. Like one of the cool things about it is, I feel like we're like a little bit unorganized today, but it's just because we're excited about it. Two things that I think are really interesting, one of them you're mentioning, it's not just a yes or no. You can go to the website and check out an example report, even without uploading a set of data yourself. It tells you the accuracy of your model, the F score. I'm not sure what F score is. Do you know?Farbod: No. No idea. I saw it and I was like; I'm looking for the explanation. I can't find it.Daniel: Expected calibration error, minimum calibration error, prediction rejection ratio, prediction stability for correct predictions, and prediction stability for incorrect prediction. So, it kind of tells you the likelihood in which your AI model is to get confused, the average confidence you can have for different classes of data. And then it helps interpret each of these things the one that I didn't know. It's got a bunch of descriptors underneath to help explain all the different descript, like all the different factors by which it's grading your AI model. I think that's super cool. The second thing that I think is really interesting, and I think it's even more interesting than the level of detail that it gives you, you don't actually need to upload any proprietary data about your AI model.Farbod: Right!Daniel:&nbsp;All you can do is take the inputs and the outputs and then feed those into Robuscope. And that's how it helps make sure that your model is robust. So, it doesn't need the actual secret sauce, which is funny because that's what this podcast is all about. We got a secret sauce poster on the wall behind me, but you don't actually need to share your secret sauce. In this case, you can just share the inputs and the outputs and keep the secret recipe like Plankton, or no, like Mr. Krabs, trying to keep it locked up in the safe so Plankton doesn't come take it.Farbod:&nbsp;Yeah, yeah. Hey, but before we move on, I did use ChatGPT to tell me what the F score is. It's a metric for analyzing machine learning models that combines the metric for precision and recallability into a single score. So how precise the predictions are and how well it's able to recall, I'm guessing, past information.Daniel:&nbsp;Love it.Farbod:&nbsp;So, combination of those two things. But I totally agree. Not only is it, like you said, is it kind of straightforward in terms of how you test it. You know, you give the ground truth and the predictions that your model came up with. The format in which you provide it is quite amazing as well. It's just a JSON file. For the folks that might not be familiar, JSON is a commonly used format of a file. Think text files or word files they use on your computer. JSON is just another version of it. And inside of it, you usually have hierarchies and structures that are dictionaries. So, you have a key, like your ground truth value. Let's say the right answer is A, or no, no, A's not right, 1. And then you have the predictions from the model, which could be, you know, 0.85, 0.67, et cetera, et cetera. And then that's what you're uploading. So very like in widely used format, very easy to generate. And yeah, super accessible. You can upload, I think files up to four gigabytes, which is quite large. And you can test not just a single grouping of data. You can upload different groupings of data by just customizing your JSON file a little bit.Daniel: Well, and let's talk about like a potential example here that the so what to kind of contextualize what can and would make this so valuable to someone who's a developer. We kind of alluded to this earlier, but an example that they use is folks that are using AI in the field of medical diagnostics, right? Using medical image data such as a CT or an MRI scan to try and diagnose a health issue with a patient. Incorrect diagnoses one way saying, you know, a false positive, let's say, like they tell you, you have breast cancer when you do not. That's a bad outcome. An even worse outcome is a false negative where it tells you you're healthy when you're actually not. So, you always want the AI model results to be both reliable and transparent in terms of the uncertainty associated with a certain outcome. So, Robuscope does a straightforward testing. They actually have focused a lot on the medical realm. They focused on logistics and autonomous driving and essentially you can help developers using this tool understand the degree of uncertainty of predictions for different classes of data. So, in this case, Robuscope would help check how robust and safe the predictions of an AI system, like we said, for breast cancer diagnoses are, and then could help share that with the developer, what types, what classes of data, what types of outcomes need to be escalated to the medical staff using this tool so that data can be reviewed manually. So, it's not saying like, let's sideline your AI model, but you take a bunch of inputs and a bunch of outputs, and this can tell you where it's robust, where it's not, and then can let medical staff know which types of results they have to double check. Maybe it's some certain types of images with a different level of contrast threshold or a different size of image. I'm not sure what those inputs are for a medical diagnostics situation, but the idea is you can understand where the models really, really strong, accurate, reliable, with low certainty, and then also understand the weaknesses to try and help us take a nuanced approach to the rollout and application of AI in very critical realms like self-driving, like medical diagnostics, like logistics, et cetera.Farbod: Totally agree. And as I was reading this, it just really sounds like a win all around. It can be a standardized tool that everyone kind of uses moving forward and it grows and gets better over time, which is great. So, I was struggling to find, because we promised our listeners that we would give a balanced view on every topic that we talk about. So, I was struggling to find a potential downside, but after some reflection, I think being an online only tool might be one of them, like requiring an internet connection. In this day and age, everybody has an internet connection, so I don't know how big that is. But the fact that you can't just download the model, go offline and then use your sensitive data in some cases where you might need to test a sensitive data. I think it could be a potential growth area, not even on downside, that's how I'll phrase it.Daniel: Well, yeah, that's what I was gonna say too, is it would be awesome if, and maybe they already have this available, I don't know, but I can think of a constraint like in my day job where they say like, it doesn't matter if they say it's confidential, do not upload any data to an online tool. Like just as a rule of thumb, right, for IT security. So, I'm with you there. Like if there were some type of air gapped version of this that could help give people a lot of confidence, I think it might help companies, universities, governments, et cetera, that are at the cutting edge of technology feel more confident about using it and help enhance the adoption curve of this tool. Otherwise, I don't, what's the score in Silicon Valley that they like try to achieve the best score.Farbod:&nbsp;The Weissman score?Daniel:&nbsp;The Weissman score. I could see this being like the new Weitzman score for A.I. is like, what's the benchmark for A.I. reliability? What's the, you know, share with investors, share with the public, share with your friends the level of robustness that you get in A.I. development. I could even see professors using this in schools to like help developers learning how to use AI tools. Like someone here listening or someone here recording this podcast that's starting classes next week, like using this like a tool in their school, right? That’s Farbod by the way, not me. I'm not going back to school. You know, like in your master's program starting next week, using something like this to help evaluate the robustness of a machine learning tool that you're developing for a class assignment. Like I could see this becoming the way by which you can self-evaluate your work.Farbod:&nbsp;Well, that's very fitting because my first class is machine learning for stock trading. So, this will come in quite handy.Daniel:&nbsp;Well, yeah, and instead of putting your money where your mouth is first, you can use something like this to like, exactly, check that first.Farbod:&nbsp;Yeah, see if I will go broke or not before I actually do.Daniel:&nbsp;Or to understand the strong areas, the specific stock categories that your model is really robust at so that you can double down on trading in those categories and not trade so much in the others where you would have lost a lot of money.Farbod:&nbsp;I don't know. Knowing myself, my algorithm's just gonna say buy GameStop.Daniel:&nbsp;Probably.Farbod:&nbsp;Yeah, this is not financial advice, by the way.Daniel:&nbsp;Not at all. You don't wanna listen to us for financial advice.Farbod: Or any advice, really. You wanna do a little quick wrap up?Daniel: Yes, sir. Have you ever seen a self-driving car or a smart device or an AI model make a weird mistake? That's okay, it's because AI isn't always as smart and reliable as we think. We always want AI to be super reliable, especially when it's growing. Its importance is growing in our lives. It's helping drive our cars, helping doctors make important decisions. But the problem is traditional development methods, the tests that people use, they don't always catch all the ways that AI can mess up in real life. So, the Fraunhofer Institute created Robuscope. It's this free tool that tests AI in all sorts of tough situations. Helps basically create a report card to make sure that your AI model is ready for anything. Unlike a lot of big companies that push out AI models without doing thorough testing, I'm looking at you, Tesla, Robuscope helps catch all these mistakes before they happen, help you understand where your models are strong and where they're weak, so you implement it in the right way, making AI safer for everyone.Farbod:&nbsp;Wow, a good summary and a jab at Tesla? You're killing it.Daniel:&nbsp;You know me, man. The mic comes up and the gloves come off.Farbod:&nbsp;Oh yeah, that's Daniel for you. Alright everyone, thank you so much for listening. And as always, we'll catch you the next one.Daniel: Peace.<hr id="isPasted">As always, you can find these and other interesting &amp; impactful engineering articles on Wevolver.com.To learn more about this show, please visit our<a href="https://www.wevolver.com/profile/the.next.byte" target="_blank">&nbsp;shows page</a>. By following the page, you will get automatic updates by email when a new show is published. Be sure to give us a follow and review on<a href="https://podcasts.apple.com/nl/podcast/the-next-byte/id1548255861?l=en" target="_blank" rel="nofollow">&nbsp;Apple</a> podcasts,<a href="https://open.spotify.com/show/6lPPsRtegnivsRUEsQ09R7?si=IPMiah7xT_apJtPjqvXMlA" target="_blank" rel="nofollow">&nbsp;Spotify</a>, and most of your favorite podcast platforms!--The Next Byte: We're two engineers on a mission to simplify complex science &amp; technology, making it easy to understand. In each episode of our show, we dive into world-changing tech (such as AI, robotics, 3D printing, IoT, &amp; much more), all while keeping it entertaining &amp; engaging along the way.

In this episode, we explore how Robuscope is reimagining the future of AI testing, offering groundbreaking solutions for improving the accuracy and reliability of artificial intelligence systems. 

Podcast: Here's a Free Tool To Test Your AI

The nationwide rush to integrate AI into manufacturing passed through Pittsburgh in August, bringing representatives from industry, government, and academia together for the <a href="https://www.aimanufacturingconference.com/" rel="noopener" target="_blank">2024 AIMST conferenceOpens in new window</a>. The dual track conference, now in its eighth year, included sessions for AI in Manufacturing and SCADA (Supervisory Control and Data Acquisition) Technology.After being held in Dallas for the past three years, organizers opted to come to Pittsburgh in part due to the strength of manufacturing research and activity at Carnegie Mellon and its <a href="https://engineering.cmu.edu/mfi/mill19/index.html" rel="noopener" target="_blank">Mill 19Opens in new window</a> partner, the <a href="https://arminstitute.org/" rel="noopener" target="_blank">ARM (Advanced Robotics for Manufacturing) InstituteOpens in new window</a>.“We are very happy to be in Pittsburgh,” said conference organizer, Robert Schaudt.When considering the move to Pittsburgh, Schaudt said his first call was to <a href="https://engineering.cmu.edu/directory/bios/fedder-gary.html" target="_blank">Gary Fedder</a>, who is the faculty director of Carnegie Mellon’s <a href="https://engineering.cmu.edu/mfi/index.html" rel="noopener" target="_blank">Manufacturing Futures InstituteOpens in new window</a> and a university professor in <a href="https://www.ece.cmu.edu/" rel="noopener" target="_blank">electrical and computer engineeringOpens in new window</a>. Fedder was also instrumental in establishing the ARM Institute, which is now one of 17 Manufacturing USA institutes.Schaudt said the conference attracts those who want to learn more about new technology and discover what is possible by attending presentations and panel discussions with representatives from organizations that include NIST, NASA, Ford, General Motors, and US Steel.Fedder was one of three keynote speakers during the conference. In his presentation, Research Driving the Future of AI in Manufacturing, he shared examples of how AI and machine learning are being used to advance robotics, additive manufacturing, and digital twin technology.<blockquote>AI is pervasive at Carnegie Mellon. It is aiding nearly all our manufacturing research.<cite>Gary Fedder, Faculty Fedder, Manufacturing Futures Institute</cite></blockquote>“AI is pervasive at Carnegie Mellon. It is aiding nearly all our manufacturing research,” said Fedder.<a href="https://engineering.cmu.edu/directory/bios/tang-pingbo.html" target="_blank">Pingbo Tang</a>, an associate professor in Carnegie Mellon’s department of <a href="https://www.cee.engineering.cmu.edu/index.html" rel="noopener" target="_blank">Civil and Environmental EngineeringOpens in new window</a>, was also invited to be a presenter. During his Human-AI Co-Evolution in Lean Adaptation of Modular Building Production Systems,&nbsp;he discussed how AI agents can learn from production engineers and increasingly are able to help humans identify and resolve time and resource waste in producing modular building products with diverse customizations.The ARM Institute’s Chief Strategy Officer Suzy Teele, who attended the conference, said the conference did an impressive job bringing together experts and practitioners to share actionable information and scenarios for the use of AI in manufacturing, including the panel discussion with ARM Institute consortium members from Boeing and GrayMatter Robotics.<blockquote>There is so much hype and confusion around AI today, and this conference provided foundational knowledge and real-world examples to help attendees plan their own strategies for AI.<cite>Suzy Teele, Chief Strategy Officer, Advanced Robotics for Manufacturing Institute</cite></blockquote>“There is so much hype and confusion around AI today, and this conference provided foundational knowledge and real-world examples to help attendees plan their own strategies for AI,” said Teele.

Manufacturing Futures Institute leaders, researchers, and partners draw national conference to Pittsburgh.

Artificial intelligence in manufacturing

The emergence of Artificial Intelligence (AI) and Machine Learning (ML) is one of the most significant computing trends in recent history. According to research, by 2027, spending on AI software alone will grow to nearly $300B, with a CAGR of 19.1% [1]. And, as the software half of the AI/ML world is growing at meteoric rates, the hardware side is also teeming with innovation. Similar research suggests that the AI accelerator market will increase from $21B in 2024 to $33B by 2028 [2].AI/ML is a software-driven pursuit and can have very different requirements depending on the market and application (i.e., automotive, IoT, cloud, training vs inference, etc). Supporting this fast moving and diverse software market requires a collaborative,industry-wide effort to develop flexible and customized domain-specific hardware solutions optimized for this range of AI workloads. Innovation means that the market should not be limited to a few key players. Rather, the process needs to be democratized to enable everyone, enterprises and startups, to contribute their ideas to the growing AI/ML computing landscape.&nbsp;RISC-V has become a preferred standard for designing the industry’s most performant and efficient AI/ML computing resources. Let’s take a look at what’s happening in the world of RISC-V for AI/ML and discover why RISC-V is a key technology for supporting the industry’s future.&nbsp;<h1 dir="ltr">Why RISC-V for AI/ML?</h1>RISC-V is a compelling architecture for the development of AI/ML systems.The extensible industry standard RISC-V ISA enables a software-focused approach to AI hardware, freeing developers from the restrictions of proprietary compute. RISC-V provides a common language for AI development, as an industry standard ISA, creating a cohesive ecosystem for AI/ML development. This robust and highly capable ecosystem of member companies, organizations, technologists, enthusiasts and academics, brings the expertise and technologies that can deliver future generations of innovative new AI systems.<h2 dir="ltr">Software-Focused Hardware</h2>RISC-V introduces new paradigms to the world of hardware design by enabling the concept of software-focused hardware. AI/ML is software defined, built on frameworks such as TensorFlow, ONNX and OneAPI, and rapidly evolving algorithms. Transformers, the basis of the recent boom in AI were <a href="https://research.google/pubs/attention-is-all-you-need/" target="_blank">only introduced in 2017</a>. New algorithms will emerge, requiring new hardware to run on, and while it is not possible to predict these algorithms, by using RISC-V we can prepare to support them with optimized hardware delivered fast.&nbsp;Historically, software designers have been constrained by the restrictions of proprietary hardware platforms. The development process involved designing applications to the constraints of predefined hardware, ultimately limiting performance and efficiency. Not every AI/ML workload belongs on a GPU. Instead of developing software to fit the constraints of predefined hardware, RISC-V’s modular architecture enables industry leading differentiation, through the development of custom processors and accelerators developed precisely for the requirements of specific AI workloads such as energy efficient IoT and powerful HPC applications.&nbsp;The configurability and openness of RISC-V enables a “software-focused” design paradigm, wherein hardware customization for specific workloads becomes feasible without the constraints imposed by proprietary instruction sets. This is achieved through RISC-V’s modular architecture, which allows developers to define custom extensions and instructions tailored to their unique workloads. For example, in AI/ML applications, developers can implement specialized instructions for matrix multiplications and vector processing—key operations in neural network computations.Using custom instructions, such as Single Instruction, Multiple Data (SIMD) operations, RISC-V processors can execute multiple data points with a single instruction, significantly boosting performance in parallel processing tasks common in AI/ML workloads. Additionally, hardware accelerators tailored to AI tasks can be integrated seamlessly. These accelerators, including tensor processing units (TPUs) or specialized neural processing units (NPUs), are designed to handle the intensive mathematical operations required for deep learning models, such as convolutions and backpropagation.This high degree of customization leads to superior computational efficiency, as the processor is optimized for specific tasks rather than relying on a generic, one-size-fits-all solution. Moreover, RISC-V’s ability to support fine-grained power management techniques reduces power consumption. For instance, designers can implement dynamic voltage and frequency scaling (DVFS) tailored to the specific needs of AI/ML tasks, reducing power usage without compromising performance.Furthermore, the RISC-V standard promotes rapid innovation and cost-effective development. Developers can contribute to and benefit from a global ecosystem of shared knowledge and resources, facilitating rapid innovation and ultimately a faster time-to-market. This collective effort ensures that the latest advancements in AI/ML can be quickly integrated into hardware designs, keeping pace with the fast-evolving demands of the field.<h2 dir="ltr">Creating a Common Language</h2>The RISC-V instruction set architecture (ISA) provides a uniform language for hardware and software developers, creating a cohesive ecosystem for AI/ML development. It allows developers to create tailored domain specific solutions with greater flexibility. Importantly, the controlled way in which this is enabled ensures that the base foundational software elements, such as those running on Linux, remain consistent and interoperable across different RISC-V implementations.Ultimately, RISC-V's unique proposition lies in its ability to foster AI/ML by leading the industry in providing the opportunity to build customized and differentiated solutions from standard and non-standard extensions. This enables domain-specific acceleration while maintaining the productivity benefits of a single, coherent programming model. RISC-V defines a small base feature set for implementers to use along with ratified extensions which they can use as required. Inside a single RISC-V based AI/ML SoC you can deeply customize different RISC-V cores for different jobs such as SoC Control or matrix multiplication, with these different cores all based on the same ISA, making them easier to program.&nbsp;To add to this flexibility, RISC-V permits the addition of vendor-specific extensions to differentiate products with application-specific functionality. RISC-V provides guidance and branding to implementors for interoperability requirements in market segments through the definition of “RISC-V Profiles"Offering a standardized yet flexible platform allows designers to rapidly integrate cutting-edge research into hardware without being hampered by proprietary constraints. Developers can leverage community contributions, such as optimized libraries and toolchains, to expedite the development process and enhance the performance of their AI/ML solutions.Beyond this, the RISC-V ISA presents a significant advantage in providing standardized, modular building blocks for various computing applications. This standardization enables hardware designers to develop and integrate components more seamlessly, reducing the complexity and cost typically associated with custom designs.By making custom designs more accessible, RISC-V makes an easier business case for the creation of optimized platforms for specific use cases, rather than compromise on having to use more generic hardware and software created for other high volume markets.&nbsp;<h2 dir="ltr" id="isPasted">A Highly Capable Ecosystem Delivering AI/ML Solutions on RISC-V </h2>The capabilities of the RISC-V member ecosystem enables AI developers to deliver the innovative software focused AI systems. From processor IP to design services and simulation to software components, the collective knowledge of these companies and organizations covers all the capabilities required to create custom software-focused AI/ML processing, enabling developers to concentrate on their innovation. The range of choice from this healthy market of ecosystem members prevents vendor lock-in and mitigates some of the supply chain challenges faced by other architectures. Alongside the capability to create custom processing, the RISC-V ecosystem also offers a range of pre-designed AI silicon, designed for different workloads.There’s no doubt that this RISC-V ecosystem has taken a foothold in the industry at large. In fact, so far, we estimate that more than 10 billion RISC-V processor cores have shipped [3]. Now, we’re seeing similar adoption in the AI/ML hardware markets, where reports are indicating that RISC-V-based AI SoCs will have a CAGR for units of 73.6% by 2027, while AI SoC unit shipments are projected to reach 25B by then. [4]. RISC-V is quickly becoming the defacto standard for building AI accelerators.&nbsp;Here are some of the highlights of RISC-V’s use in AI/ML.<ul><li dir="ltr">Meta’s MTIA v1 AI Inference Accelerator [5]</li><li dir="ltr">Esperanto ET-SoC-1 [6]</li><li dir="ltr">Google’s TPU [7]</li><li dir="ltr">SiFive Intelligence X390 [8]</li><li dir="ltr">StreamComputing STPC920 [10]</li><li dir="ltr">Ventana Veyron V2 [11]</li><li dir="ltr">Andes Technology QiLai SoC [12]</li><li dir="ltr">Semidynamics ‘All-In-One AI’ IP [13]</li><li dir="ltr">Axelera AI Metis AI Platform [14]</li><li dir="ltr">Project Open Se Cura [15]</li><li dir="ltr">Codeplay oneAPI Construction Kit [16]</li></ul><h1 dir="ltr">Conclusion</h1>As the AI/ML market continues to grow exponentially, RISC-V is poised to be a formidable driver of innovation and change for all. By offering a modular, open, and customizable architecture, RISC-V gives designers the creativity, power, and speed of innovation needed to address the diverse and evolving needs of AI/ML applications. Developers are empowered with the flexibility to tailor solutions to specific requirements while working in a unified and interoperable ecosystem. Finally, the community-driven innovation model of RISC-V accelerates the development of cutting-edge technologies and democratizes access to advanced hardware design for all.For more information on how RISC-V is driving the future of AI, visit <a href="http://www.riscv.org/ai" target="_blank">www.riscv.org/ai.</a><h1 dir="ltr">References</h1>[1]<a href="https://www.gartner.com/en/documents/4925331#:~:text=Summary,2023%20to%2035%25%20by%202027" target="_blank">https://www.gartner.com/en/documents/4925331#:~:text=Summary,2023%20to%2035%25%20by%202027</a>.[2] <a href="https://www.gartner.com/en/newsroom/press-releases/2024-05-29-gartner-forecasts-worldwide-artificial-intelligence-chips-revenue-to-grow-33-percent-in-2024#:~:text=Gartner%20forecasts%20AI%20PC%20shipments,will%20be%20an%20AI%20PC" target="_blank" rel="noopener noreferrer">https://www.gartner.com/en/newsroom/press-releases/2024-05-29-gartner-forecasts-worldwide-artificial-intelligence-chips-revenue-to-grow-33-percent-in-2024#:~:text=Gartner%20forecasts%20AI%20PC%20shipments,will%20be%20an%20AI%20PC</a>.[3]<a href="https://riscv.org/news/2022/07/europe-steps-up-as-risc-v-ships-10bn-cores-nick-flaherty-ee-news-europe/" target="_blank">https://riscv.org/news/2022/07/europe-steps-up-as-risc-v-ships-10bn-cores-nick-flaherty-ee-news-europe/</a>[4] <a href="https://semico.com/content/analyzing-risc-v-cpu-market-sip-socs-ai-and-design-starts" target="_blank" rel="noopener noreferrer">https://semico.com/content/analyzing-risc-v-cpu-market-sip-socs-ai-and-design-starts</a>[5]<a href="https://ai.meta.com/blog/meta-training-inference-accelerator-AI-MTIA/" target="_blank">https://ai.meta.com/blog/meta-training-inference-accelerator-AI-MTIA/</a>[6]<a href="https://www.esperanto.ai/products/" target="_blank">https://www.esperanto.ai/products/</a>[7] <a href="https://riscv.org/wp-content/uploads/2018/12/Antmicro-RISC-V-Summit-Keynote.pdf" target="_blank" data-lf-fd-inspected-kn9eq4roawkarlvp="true">https://riscv.org/wp-content/uploads/2018/12/Antmicro-RISC-V-Summit-Keynote.pdf</a>[8] <a href="https://www.sifive.com/cores/intelligence-x390" target="_blank">https://www.sifive.com/cores/intelligence-x390</a>[9] <a href="https://tenstorrent.com/hardware/wormhole" target="_blank">https://tenstorrent.com/hardware/wormhole</a>&nbsp;[10] <a href="https://www.streamcomputing.com/en/index.php?s=product&c=category&id=2" target="_blank">https://www.streamcomputing.com/en/index.php?s=product&amp;c=category&amp;id=2</a>[11] <a href="https://www.ventanamicro.com/ventana-introduces-veyron-v2/" target="_blank">https://www.ventanamicro.com/ventana-introduces-veyron-v2/</a>[12] <a href="https://www.andestech.com/en/2024/05/30/andes-technology-announced-the-qilai-soc-and/" target="_blank">https://www.andestech.com/en/2024/05/30/andes-technology-announced-the-qilai-soc-and/</a>[13] <a href="https://semidynamics.com/en/resources/press-releases" target="_blank">https://semidynamics.com/en/resources/press-releases</a>[14] <a href="https://www.axelera.ai/blog/the-metis-ai-platform-in-detail" target="_blank">https://www.axelera.ai/blog/the-metis-ai-platform-in-detail</a>[15] <a href="https://opensource.googleblog.com/2023/11/project-open-se-cura-open-source-announcement.html" target="_blank">https://opensource.googleblog.com/2023/11/project-open-se-cura-open-source-announcement.html</a>[16] <a href="https://developer.codeplay.com/products/oneapi/construction-kit/home/" target="_blank" rel="noopener noreferrer">https://developer.codeplay.com/products/oneapi/construction-kit/home/</a>

RISC-V: introducing new paradigms to the world of hardware design with software-focused hardware.

RISC-V Enables Performant and Flexible AI and ML Compute

An autonomous robot – a base on wheels fitted with computer vision technology that snaps images of grapevines in near real-time – rolls down a long row of plants in a vineyard at Cornell AgriTech in Geneva, New York.The <a href="https://news.cornell.edu/stories/2022/11/autonomous-robots-help-modernize-grape-wine-industry" target="_blank">Cornell-developed “PhytoPatholoBot”</a> detects spectral signatures of three kinds of devastating grape diseases – powdery mildew, downy mildew and grapevine leafroll virus – leading to early management of pathogens without needing farmworkers to spot them. Ultimately, data collected by such robots will help to ground-truth NASA satellite data that could one day detect plant diseases from space.This was one of many research projects presented at a technology demonstration on Aug. 20, part of a three-day “Space for Ag Tour” by NASA leaders to better understand the remote sensing needs of Cornell AgriTech researchers, specialty crop growers and local stakeholders. NASA leadership visited Cornell’s main campus, Cornell AgriTech, sites in Niagara county and the Cornell Lake Erie Research and Extension Lab in Portland, New York.The tour was designed to foster two-way dialogue with growers and stakeholders to improve NASA Earth Science’s Earth-observing satellites and instruments for specialty crop agriculture and viticulture and foster a broader conversation about how to effectively translate research into practice.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1724648875297-0822_nasa1_0.jpg" style="width: 100%px;" class="fr-fic fr-dib">Yu Jiang poses with the PhytoPatholoBotas and UV light robot as part of the “Space for Ag Tour” at the Cornell Agritech campus. Ryan Young/Cornell University“My hope is that by having this Space for Ag Tour here, by showing them New York viticulture as well as what we have to offer here at Cornell, it can help grow the specialty crop footprint within NASA as well as their footprint in pest and disease research,” said&nbsp;<a href="https://cals.cornell.edu/kaitlin-gold" target="_blank">Katie Gold</a>, the event’s main organizer and an assistant professor and the Susan Ekert Lynch Faculty Fellow in the Plant Pathology and Plant-Microbe Biology Section of the School of Integrative Plant Science (SIPS) at Cornell AgriTech, which is in the College of Agriculture and Life Sciences.&nbsp;“I think there’s amazing potential for the satellite data that’s being collected both now and in the future with the eventual launch of the Surface Biology and Geology satellite to be applied to specialty crops,” said Gold, who is an expert on ground-based and remote plant disease sensing and applied grape pathology. The meetings were intended to raise awareness among NASA leaders and scientists of the unique challenges facing specialty crops, including pests and diseases, so they can make those connections in their own research and applications, she said. &nbsp;“For NASA leadership, it’s an opportunity as they think long-term about observational needs and priorities that helps them make the case for the observations that are necessary, from a scientific perspective, to make an impact on the different concerns that farmers have,” said Alyssa Whitcraft, a research professor in the Department of Geographical Sciences at the University of Maryland, whose work focuses on satellite remote sensing of agriculture. Whitcraft is the founding executive director of NASA Acres and co-founding deputy director of NASA Harvest, respectively the U.S. and international NASA consortia on food security and agriculture.&nbsp;Gold is the institutional lead for&nbsp;<a href="https://cals.cornell.edu/news/2023/05/bringing-nasa-technology-down-earth-agriculture" target="_blank">NASA Acres at Cornell</a>.As part of the demonstrations,&nbsp;<a href="https://cals.cornell.edu/yun-yang" target="_blank">Yun Yang</a>, assistant professor in SIPS’ Soil and Crop Sciences Section at Cornell AgriTech, described her NASA Acres-funded research to monitor plant water use using remote sensing, and understanding the impacts of agricultural practices on water resource sustainability.One of Yang’s research projects seeks to estimate water loss to evaporation and transpiration in California vineyards. Vineyard managers want to know how much and when to irrigate in spring based on groundwater depletion, which is a global issue, especially in California.&nbsp;Her team built an instrument to better monitor water fluxes on the ground, which provides validation for NASA Landsat satellite data that can be used to estimate water loss to evapotranspiration when farmers irrigate. By knowing how much water is used, based on a simple water balance, they can tell how much water is needed. Yang’s team has been developing a dashboard for users. “With this dashboard, we can really hand the technology we have to the growers,” Yang said. “And when they are in the field, we can easily check the information and help them manage irrigation.” &nbsp; &nbsp;Applied roboticist <a href="https://cals.cornell.edu/yu-jiang" target="_blank">Yu Jiang</a>, assistant professor in SIPS’ Horticulture Section at Cornell AgriTech, demonstrated the PhytoPatholoBot that he and doctoral student Ertai Liu developed with Gold with funding from NASA Acres and U.S. Department of Agriculture National Institute of Food and Agriculture VitisGen3.&nbsp;The PhytoPatholoBot eliminates background noise from cloud cover or sunlight to provide a constant, high-quality image and has autonomous navigation using GPS. The bot provides a readout per row of vines with red areas denoting higher infection and white areas showing lower infection, and coupled with AI, allows for near real-time inferences of disease that growers may use to focus their treatment efforts on plants that need it most.“We can scan and scale all the field, we are also generating the results that are ready to be realized by the growers, and hopefully, while working together to integrate this data stream back to NASA to see if we can train remote sensing models and make a bigger impact for small grower stakeholders,” Jiang said. Additionally, researchers can use data from the robots to train NASA airborne sensing better than humans, Gold said.&nbsp;The hope is to use robots to automate data collection, to make it possible to gather measurements for millions of acres.&nbsp;Following Jiang,&nbsp;<a href="https://cals.cornell.edu/sara-emery" target="_blank">Sarah Emery</a>, assistant professor of entomology at Cornell AgriTech, presented her work to develop risk maps for spotted lanternfly infestations, a growing threat to grape and other growers in New York.&nbsp;“We’ve been trying to understand how this insect pest is going to be moving across the landscape and develop a model to assess the risk to individual vineyards across New York state,” Emery said.&nbsp;In collaboration with the New York State Department of Agriculture and Markets, the model being developed will overlay observations of spotted lanternfly (90,000 sightings have been recorded across the state) with maps of vineyards and locations of its preferred host, tree of heaven. Eventually, they plan to create a public-facing system so that growers can input their information and get back a level of risk.&nbsp;The hope is to use hyperspectral data (which allows for the identification of objects and materials by analyzing their unique spectral signatures) from satellites to identify light reflectance patterns of non-crop plants across the landscape to create risk maps at finer and wider scales. Emery has teamed with Gold and doctoral student Jaclyn Eller for the project.&nbsp;Also,&nbsp;<a href="https://cals.cornell.edu/jason-londo" target="_blank">Jason Londo</a>,&nbsp;associate professor of fruit crop physiology and climate adaptation in SIPS Horticulture Section at Cornell AgriTech, described using hyperspectral and multispectral approaches to understanding plant stress in grapes and apples using drones. He hopes to one day use similar data from satellites, but needs better resolution to be able to detect stress on a tree-by-tree basis, he said.Other visiting NASA personnel included Karen St. Germain, division director of the Earth Science Division in the Science Mission Directorate at NASA.&nbsp;<a href="https://cals.cornell.edu/steven-wolf" target="_blank">Steven Wolf</a>, associate professor of natural resources and the environment in CALS, organized events at Cornell’s Ithaca campus.

Members of Yu Jiang’s team demonstrate the technology. Credits: Ryan Young/Cornell University

An autonomous robot – a base on wheels fitted with computer vision technology that snaps images of grapevines in near real-time – rolls down a long row of plants in a vineyard at Cornell AgriTech in Geneva, New York.

From space to farm: readying NASA satellites to help growers

On a research cruise around Hawaii in 2018, Yuening Zhang SM ’19, PhD ’24 saw how difficult it was to keep a tight ship. The careful coordination required to map underwater terrain could sometimes led to a stressful environment for team members, who might have different understandings of which tasks must be completed in spontaneously changing conditions. During these trips, Zhang considered how a robotic companion could have helped her and her crewmates achieve their goals more efficiently. Six years later, as a research assistant in the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL), Zhang developed what could be considered a missing piece: an AI assistant that communicates with team members to align roles and accomplish a common goal. In a paper presented at the International Conference on Robotics and Automation (ICRA) and&nbsp;<a href="https://ieeexplore.ieee.org/document/10609865" target="_blank">published on IEEE Xplore on Aug. 8</a>, she and her colleagues present a system that can oversee a team of both human and AI agents, intervening when needed to potentially increase teamwork effectiveness in domains like search-and-rescue missions, medical procedures, and strategy video games. The CSAIL-led group has developed a theory of mind model for AI agents, which represents how humans think and understand each other’s possible plan of action when they cooperate in a task. By observing the actions of its fellow agents, this new team coordinator can infer their plans and their understanding of each other from a prior set of beliefs. When their plans are incompatible, the AI helper intervenes by aligning their beliefs about each other, instructing their actions, as well as asking questions when needed. For example, when a team of rescue workers is out in the field to triage victims, they must make decisions based on their beliefs about each other’s roles and progress. This type of epistemic planning could be improved by CSAIL’s software, which can send messages about what each agent intends to do or has done to ensure task completion and avoid duplicate efforts. In this instance, the AI helper may intervene to communicate that an agent has already proceeded to a certain room, or that none of the agents are covering a certain area with potential victims. “Our work takes into account the sentiment that ‘I believe that you believe what someone else believes,’” says Zhang, who is now a research scientist at Mobi Systems. “Imagine you’re working on a team and you ask yourself, ‘What exactly is that person doing? What am I going to do? Does he know what I am about to do?’ We model how different team members understand the overarching plan and communicate what they need to accomplish to help complete their team’s overall goal.” AI to the rescue Even with a sophisticated plan, both human and robotic agents will encounter confusion and even make mistakes if their roles are unclear. This plight looms especially large in search-and-rescue missions, where the objective may be to locate someone in danger despite limited time and a vast area to scan. Thankfully, communication technology augmented with the new robotic assistant could potentially notify the search parties about what each group is doing and where they’re looking. In turn, the agents could navigate their terrain more efficiently.This type of task organization could aid in other high-stakes scenarios like surgeries. In these cases, the nurse first needs to bring the patient to the operation room, then the anesthesiologist puts the patient to sleep before the surgeons begin the operation. Throughout the operation, the team must continuously monitor the patient’s condition while dynamically responding to the actions of each colleague. To ensure that each activity within the procedure remains well-organized, the AI team coordinator could oversee and intervene if confusion about any of these tasks arises.Effective teamwork is also integral to video games like “Valorant,” where players collaboratively coordinate who needs to attack and defend against another team online. In these scenarios, an AI assistant could pop up on the screen to alert individual users about where they’ve misinterpreted which tasks they need to complete.Before she led the development of this model, Zhang designed EPike, a computational model that can act as a team member. In a 3D simulation program, this algorithm controlled a robotic agent that needed to match a container to the drink chosen by the human. As rational and sophisticated as they may be, cases arise where these AI-simulated bots are limited by their misconceptions about their human partners or the task. The new AI coordinator can correct the agents’ beliefs when needed to resolve potential problems, and it consistently intervened in this instance. The system sent messages to the robot about the human’s true intentions to ensure it matched the container correctly.“In our work on human-robot collaboration, we’ve been both humbled and inspired over the years by how fluid human partners can be,” says Brian C. Williams, MIT professor of aeronautics and astronautics, CSAIL member, and senior author on the study. “Just look at a young couple with kids, who work together to get their kids breakfast and off to school. If one parent sees their partner serving breakfast and still in their bathrobe, the parent knows to shower quickly and shuffle the kids off to school, without the need to say a word. Good partners are well in tune with the beliefs and goals of each other, and our work on epistemic planning strives to capture this style of reasoning.”The researchers' method incorporates probabilistic reasoning with recursive mental modeling of the agents, allowing the AI assistant to make risk-bounded decisions. In addition, they focused on modeling agents’ understanding of plans and actions, which could complement previous work on modeling beliefs about the current world or environment. The AI assistant currently infers agents’ beliefs based on a given prior of possible beliefs, but the MIT group envisions applying machine learning techniques to generate new hypotheses on the fly. To apply this counterpart to real-life tasks, they also aim to consider richer plan representations in their work and reduce computation costs further.Dynamic Object Language Labs President Paul Robertson, Johns Hopkins University Assistant Professor Tianmin Shu, and former CSAIL affiliate Sungkweon Hong PhD ’23 join Zhang and Williams on the paper. Their work was supported, in part, by the U.S. Defense Advanced Research Projects Agency (DARPA) Artificial Social Intelligence for Successful Teams (ASIST) program.

A system developed by MIT CSAIL researchers can oversee a team of both human and AI agents, communicating with team members to align roles and accomplish a common goal. Image: Alex Shipps/MIT CSAIL

An AI team coordinator aligns agents’ beliefs about how to achieve a task, intervening when necessary to potentially help with tasks in search and rescue, hospitals, and video games.

AI assistant monitors teamwork to promote effective collaboration

Using a combination of machine learning and powerful X-rays, Cornell researchers have solved a mystery behind the unusual behavior seen in a class of materials with potential for thermoelectric energy conversion and other applications.Researchers have long shown that the cubic phase of germanium telluride (GeTe) exhibits an unexpected rise in lattice thermal conductivity as its temperature increases. But knowing that a property exists and explaining why it exists are two very different things.Zhiting Tian, associate professor at the Sibley School of Mechanical and Aerospace Engineering in Cornell Engineering, is the lead author of a new paper in&nbsp;<a href="https://doi.org/10.1038/s41467-024-51377-8" target="_blank">Nature Communications</a> that provides a convincing explanation for the unexpected behavior of GeTe. More generally, the research deepens researchers’ understanding of thermal transport in a class of materials known as phase-change materials.What Tian and her team of researchers found is that as a sample of GeTe is heated to the point where its phase changes from a rhombohedral structure to a cubic structure, the bonds between second-nearest neighbors of like atoms (Ge-Ge and Te-Te) strengthen considerably. Ge-Ge bond strength increased by 8.3% and the strength of Te-Te bonds increased by a remarkable 103% as the sample’s temperature rose from 693 degres Kelvin to 850 degrees Kelvin.The group used machine learning-assisted first-principles calculations corroborated by X-ray scattering measurements to computationally reproduce the increasing thermal conductivity trend for the first time. They then borrowed a commonly used chemistry technique to perform the bonding analysis, and confirmed that these increasingly strong second-nearest neighbor bonds play a major role in GeTe’s previously unexplained increase in lattice thermal conductivity.“Computationally, it was quite prohibitive to look at temperature effects and to consider higher-order scattering, for example,” Tian said. “But because we were able to leverage machine learning potential, we were able to more efficiently extract the interactions and consider the multiple effects – temperature dependence, four-phonon scattering, and coherence contribution – all at once.”Phase-change materials such as GeTe are valued for their usefulness in a range of optical and electronic applications. Their optical and electrical properties change markedly depending on which of several stable phase states they are in and these phase states can be easily reversed. Moreover, GeTe has been talked about as a replacement for the semiconductor lead telluride as a thermoelectric material, due to lead’s inherent toxicity, according to Tian, who is a faculty fellow at the Cornell Atkinson Center for Sustainability.The work demonstrates an efficient and thorough pathway toward accurate modeling of materials near phase transitions or at high temperatures that have promise for phase change, thermoelectric and other energy applications.“We also identified other materials which showed a similar increase in conductivity, including tin-telluride and tin-selenide,” Tian said. “And so we hope our work will spark interest in looking deeper into the thermal transport behavior of other phase-change materials.”The work was supported by Tian’s National Science Foundation CAREER Award.

Using a combination of machine learning and powerful X-rays, Cornell researchers have solved a mystery behind the unusual behavior seen in a class of materials with potential for thermoelectric energy conversion and other applications.

Machine learning explains material's thermal conductivity

A new cross-disciplinary study by WashU researchers has uncovered an unexpected psychological phenomenon at the intersection of human behavior and artificial intelligence: When told they were training AI to play a bargaining game, participants actively adjusted their own behavior to appear more fair and just, an impulse with potentially important implications for real-world AI developers.“The participants seemed to have a motivation to train AI for fairness, which is encouraging, but other people might have different agendas,” said Lauren Treiman, a PhD student in the Division of Computational &amp; Data Sciences (DCDS) and lead author of the study. “Developers should know that people will intentionally change their behavior when they know it will be used to train AI.”The study, published in PNAS, was supported by a seed grant from the&nbsp;<a href="https://triads.wustl.edu/" target="_blank">Transdisciplinary Institute in Applied Data Sciences (TRIADS)</a>, a signature initiative of the Arts &amp; Sciences Strategic Plan. The co-authors are&nbsp;<a href="https://psych.wustl.edu/people/wouter-kool" target="_blank">Wouter Kool</a>, assistant professor of psychological and brain sciences in Arts &amp; Sciences, and&nbsp;<a href="https://engineering.wustl.edu/faculty/Chien-Ju-Ho.html" target="_blank">Chien-Ju Ho</a>, assistant professor of computer science and engineering in the McKelvey School of Engineering. Kool and Ho are Treiman’s graduate advisors.The study included five experiments, each with roughly 200-300 participants. Subjects were asked to play the “Ultimatum Game,” a challenge that requires them to negotiate small cash payouts (just $1 to $6) with other human players or a computer. In some cases, they were told their decisions would be used to teach an AI bot how to play the game.The players who thought they were training AI were consistently more likely to seek a fair share of the payout, even if such fairness cost them a few bucks. Interestingly, that behavior change persisted even after they were told their decisions were no longer being used to train AI, suggesting the experience of shaping technology had a lasting impact on decision-making. &nbsp;“As cognitive scientists, we’re interested in habit formation,” Kool said. “This is a cool example because the behavior continued even when it was not called for anymore.”Still, the impulse behind the behavior isn’t entirely clear. Researchers didn’t ask about specific motivations and strategies, and Kool explained that participants may not have felt a strong obligation to make AI more ethical. It’s possible, he said, that the experiment simply brought out their natural tendencies to reject offers that seemed unfair. “They may not really be thinking about the future consequences,” he said. “They could just be taking the easy way out.”&nbsp;“The study underscores the important human element in the training of AI,” said Ho, a computer scientist who studies the relationships between human behaviors and machine learning algorithms. “A lot of AI training is based on human decisions,” he said. “If human biases during AI training aren’t taken into account, the resulting AI will also be biased. In the last few years, we’ve seen a lot of issues arising from this sort of mismatch between AI training and deployment.”Some facial recognition software, for example, is less accurate at identifying people of color, Ho said. “That’s partly because the data used to train AI is biased and unrepresentative,” he said.Treiman is now conducting follow-up experiments to get a better sense of the motivations and strategies of people training AI. “It’s very important to consider the psychological aspects of computer science,” she said.&nbsp;<hr>Treiman LS, Ho CJ, Kool W. The consequences of AI training on human decision-making. Proceedings of the National Academy of Sciences (PNAS) Aug. 6, 2024.DOI:&nbsp;<a href="https://doi.org/10.1073/pnas.2408731121" target="_blank">https://doi.org/10.1073/pnas.2408731121</a>&nbsp;

WashU researchers have found humans adjust their own behavior to be more fair when training AI.

McKelvey’s Chien-Ju Ho working with Art & Sciences’ Wouter Kool, DCDS PhD student Lauren Treiman to understand how human behavior changes in training AI

Humans change their own behavior when training AI

With the hype around machine learning (ML) and the rush to transform businesses with it, unsurprisingly, not all ML projects have been a success. Often, a “solution before the problem” mentality leads to poorly defined requirements and goals for the use of ML. Failing to understand why ML should be used, as well as how business metrics will be impacted, may lead to proof of concept (POC) work that takes up valuable time without delivering results. To avoid that, companies seeking to introduce ML into their processes or products should aim to understand their overall goal in doing so and then link that goal to relevant business metrics used to measure a current baseline, setting the stage for assessing ML performance as the POC work progresses. Finally, connecting those indicators to appropriate ML metrics based on the task or use case for the POC and developing a short roadmap for the research and development direction will increase the likelihood of success.<h2>Distilling Goals into Business Metrics</h2>Businesses usually decide to introduce ML into their processes for one of several overarching reasons. Typically, these reasons include improving revenue by increasing the productivity of the team, elevating the success rate of some particular aspect of the business, improving customer outcomes by reducing the time to respond to incoming information, or reducing costs associated with errors or other waste. These reasons tie into more fine-grained business metrics that should be identified when starting an ML project. For example, if the goal is to increase revenue by enhancing the team or company’s capabilities, then relevant metrics for measuring the current state of productivity might relate to sales and marketing metrics such as quota attainment rate, cost per lead, or net sales revenue, among others. For improving customer success, relevant metrics may include churn, customer satisfaction scores, or rate of lost deals. Most importantly, identifying these business needs via correct metrics is crucial when developing projects rather than identifying solutions without understanding the concrete “why” behind their necessity. Deciding to implement an ML process for triaging support tickets or summarizing long documents won’t make sense unless the impact of doing so is effectively measured as a baseline.<h2>Matching Business Metrics to ML Equivalents</h2>Once appropriate business metrics have been chosen for the project based on the overall goals, they should be matched to one or more ML metrics based on the tasks identified for ML to tackle. Typically, a cluster of metrics for the ML side will be needed to capture an improvement on the business side. For example, suppose the goal is to increase productivity by accelerating a team working on a particular process, perhaps quality assurance of software pre-deployment. In that case, the business metrics might be the time from start to end of testing, the number of elements tested, and the number of tests run per session. However, adjacent and equally important metrics include the number of bugs caught pre-deployment and the number of previously passing tests that now fail. When establishing an ML process to handle some of the quality assurance team’s work, the metrics are not just latency or how fast the model can assess expected versus actual output for discrepancies but also model accuracy—specifically, the number of false positives and negatives. Because mistakes that the model makes may need to be assessed by a human, too many flagged issues will add to the team’s workload, and too many missed issues will result in work upstream for the development team. Alongside having a benchmark of the team’s current performance, based on the business metrics discussed, the baseline accuracy is also useful for assessing whether ML performs at least as well to be considered valid.Sometimes, depending on the use case or task and the type of output required, the ML metrics may not be as straightforward as accuracy or error rate. However, some way of measuring how well an ML process is performing usually exists, as long as one can measure how well things work without it. Natural language processing (NLP) is an example of one domain with less direct metrics. For instance, tasks like summarizing text or generating content appear to be difficult to assess on the surface. Still, if some time is spent to build a dataset of input text and examples of the desired output, then metrics such as Recall-Oriented Understudy for Gisting Evaluation (ROUGE) can be used. This and other metrics measure things like the overlap of words between desired and actual output to account for the fact that no single “correct” answer exists; rather, answers vary according to levels of correctness.<h2>Building the ML POC Roadmap</h2>Finally, with the ML metrics defined, the process of building a brief but detailed roadmap of the POC implementation and experimentation approaches can start. This should involve dividing the whole task into smaller, simpler pieces that are quick to validate in terms of success with an ML model, which could include triaging urgent versus non-urgent support tickets instead of a more complicated set of classifications. Additionally, a preliminary literature review or prior art search should be conducted to identify a series of approaches with increasing complexity in case the simpler options fail at the task. Implementing this strategy starts with off-the-shelf models or those available through a third-party application programming interface (if allowable in the business context), then moves to architectures that can be adjusted by fine-tuning or retraining for the particular task, and finishes with those that need to be implemented from scratch with perhaps more complicated training regimes. The costs for each approach can be approximated to provide a fair assessment of the return on investment (ROI) for the ML project based on the expected value. Doing so highlights whether some POC directions will be prohibitively expensive to try and helps set the cutoff point for abandoning the project if the best performance achieved is insufficient.<h2>Conclusion</h2>Introducing ML into a company’s processes or products helps achieve overarching business goals like improving productivity or reducing costs and errors. However, to ensure success, these ML projects need to start on the right foot. Measuring impact via the appropriate business metrics is key, followed by identifying the right ML metrics to show that developed models will provide the expected value because they work as well as required. Once these details are established, building a roadmap to capture the direction of POC implementation will keep the project on track. This roadmap will also ensure that ROI versus effort is tracked so that a sufficient point of abandonment can be identified that balances the two. These best practices are the first of several required to transition from business goals to ML POC to production.Follow this AI blog series to learn the other important stages involved in developing an ML POC in an agile but robust fashion and how to put the resulting outputs into production. Future blogs will cover identifying and establishing a dataset for the project, setting up experimentation tooling, developing resources and approaches for POC-building, including open-source models, creating guidelines and focus points when extending to a production-ready version, as well as considering what to anticipate and monitor post-deployment.

Moving Your ML Proof of Concept to Production Part 1: From Business Goals to ML Metrics

Machine Learning Proof of Concept to Production: Part One

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/270862db-fef1-42ce-8710-bb98f6e896b3?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr><h2>Tl;DR</h2>Fraunhofer Institute's Robuscope provides a comprehensive framework for evaluating the robustness and safety of AI models, ensuring dependable AI applications.<h2>Overview</h2>The integration of AI models into critical applications—from autonomous driving to medical diagnostics—necessitates thorough validation to ensure their robustness and safety.&nbsp;Traditional testing methods are often inadequate to capture the full range of real-world conditions these models will face. The Fraunhofer Institute has introduced Robuscope, a tool designed to rigorously evaluate AI models. This development is crucial for enhancing the reliability of AI systems, which in turn can prevent failures and improve safety in various high-stakes environments.<h2>Fraunhofer Institute</h2>The Fraunhofer Institute for Cognitive Systems IKS is a leading research organization dedicated to advancing AI and cognitive systems.&nbsp;Robuscope is led by key figures including Karsten Roscher, the Department Head of Perception and Imaging, and Anna Guderitz, the Head of Business Development. Their leadership and expertise underscore the institute's dedication to advancing AI safety and reliability. <iframe src="https://c8056756.sibforms.com/serve/MUIFALkfH6sHm0PhgjOdf-A3a5im2wlA4nqaKD9YAAUIzVeXNxunIDvjpxguE0chxmsTCpStIc50Yt6yNbfyjS2pH4r0runZJkLimSEF68pQl2k36X5NK7ob4s89UaGdy192tfsSLKChucCwhS8KSN9_LCHx4aLzkteU_nutPS9IB1r1-o5COxrGtS9fAA4lNCwJPIsbUCE47fCT" frameborder="0" scrolling="no" allowfullscreen="" style="display: block; margin-left: auto; margin-right: auto; max-width: 100%; width: 830px; height: 402px;" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;</iframe><h2>Meet Robuscope</h2>Robuscope is an innovative tool that focuses on the rigorous evaluation of AI models to gauge their robustness. The tool encompasses a comprehensive testing framework that allows users to input test data - with as little as 30 data points - for testing their model without requiring sensitive data. By employing these methods, Robuscope identifies potential weaknesses and failure points that may not be evident in standard testing procedures while offloading the tedious task of manual testing. The tool also offers an online interface for users to submit their AI models and receive detailed reports on their robustness. This allows developers to gain insights into how their models perform under different scenarios and make necessary adjustments to enhance their reliability. The Fraunhofer Institute's approach with Robuscope ensures that AI models are thoroughly vetted before deployment in critical applications like analyzing CAT/MRI scans or autonomous driving.<h2>Impact</h2>The introduction of Robuscope has significant implications for the development and deployment of AI technologies.Positive Impact:<ul><li>Enhanced Safety: Rigorous testing ensures that AI models used in critical applications, such as autonomous vehicles and medical devices, are reliable and less prone to failures, thereby improving safety.</li><li>Increased Trust: Providing detailed evaluations of AI model performance builds trust in AI systems, encouraging broader adoption across various industries.</li><li>Proactive Improvement: Developers can identify and address potential issues early, leading to continuous improvements in AI model robustness.</li></ul>Negative Impact:<ul><li>Online: Users do not have to input sensitive data for testing their models but certain models might require sensitive data for thorough testing and the online nature of this tool would likely be a limiting factor for those use cases.</li></ul>Overall, Robuscope represents a significant step forward in AI testing, offering a robust solution to ensure the reliability and safety of AI applications. This innovation not only improves the performance of AI systems but also builds confidence in their use, facilitating their integration into critical sectors.

Researchers at Fraunhofer Institute have developed a free, online tool for testing the robustness of AI models

AI Testing Reimagined with Robuscope

Picture a robot arriving in a warehouse. It comes packaged with the skills it was trained on, like placing an object, and now it needs to pick items from a shelf it’s not familiar with. At first, the machine struggles with this, since it needs to get acquainted with its new surroundings. To improve, the robot will need to understand which skills within an overall task it needs improvement on, then specialize (or parameterize) that action.A human onsite could program the robot to optimize its performance, but researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and The AI Institute have developed a more effective alternative. Presented at the Robotics: Science and Systems Conference last month, their “Estimate, Extrapolate, and Situate” (EES) algorithm enables these machines to practice on their own, potentially helping them improve at useful tasks in factories, households, and hospitals.&nbsp;<h2>Sizing up the situation</h2>To help robots get better at activities like sweeping floors, EES works with a vision system that locates and tracks the machine’s surroundings. Then, the algorithm estimates how reliably the robot executes an action (like sweeping) and whether it would be worthwhile to practice more. EES forecasts how well the robot could perform the overall task if it refines that particular skill, and finally, it practices. The vision system subsequently checks whether that skill was done correctly after each attempt.EES could come in handy in places like a hospital, factory, house, or coffee shop. For example, if you wanted a robot to clean up your living room, it would need help practicing skills like sweeping. According to Nishanth Kumar SM ’24 and his colleagues, though, EES could help that robot improve without human intervention, using only a few practice trials.“Going into this project, we wondered if this specialization would be possible in a reasonable amount of samples on a real robot,” says Kumar, co-lead author of a&nbsp;<a href="https://arxiv.org/pdf/2402.15025.pdf" target="_blank" data-lf-fd-inspected-kn9eq4roawkarlvp="true">paper</a> describing the work, PhD student in electrical engineering and computer science, and a CSAIL affiliate. “Now, we have an algorithm that enables robots to get meaningfully better at specific skills in a reasonable amount of time with tens or hundreds of data points, an upgrade from the thousands or millions of samples that a standard reinforcement learning algorithm requires.”<h2>See Spot sweep</h2>EES’s knack for efficient learning was evident when implemented on Boston Dynamics’ Spot quadruped during research trials at The AI Institute. The robot, which has an arm attached to its back, completed manipulation tasks after practicing for a few hours. In one demonstration, the robot learned how to securely place a ball and ring on a slanted table in roughly three hours. In another, the algorithm guided the machine to improve at sweeping toys into a bin within about two hours. Both results appear to be an upgrade from previous frameworks, which would have likely taken more than 10 hours per task. “We aimed to have the robot collect its own experience so it can better choose which strategies will work well in its deployment,” says co-lead author Tom Silver SM ’20, PhD ’24, an electrical engineering and computer science (EECS) alumnus and CSAIL affiliate who is now an assistant professor at Princeton University. “By focusing on what the robot knows, we sought to answer a key question: In the library of skills that the robot has, which is the one that would be most useful to practice right now?”EES could eventually help streamline autonomous practice for robots in new deployment environments, but for now, it comes with a few limitations. For starters, they used tables that were low to the ground, which made it easier for the robot to see its objects. Kumar and Silver also 3D printed an attachable handle that made the brush easier for Spot to grab. The robot didn’t detect some items and identified objects in the wrong places, so the researchers counted those errors as failures.<h2>Giving robots homework</h2>The researchers note that the practice speeds from the physical experiments could be accelerated further with the help of a simulator. Instead of physically working at each skill autonomously, the robot could eventually combine real and virtual practice. They hope to make their system faster with less latency, engineering EES to overcome the imaging delays the researchers experienced. In the future, they may investigate an algorithm that reasons over sequences of practice attempts instead of planning which skills to refine. “Enabling robots to learn on their own is both incredibly useful and extremely challenging,” says Danfei Xu, an assistant professor in the School of Interactive Computing at Georgia Tech and a research scientist at NVIDIA AI, who was not involved with this work. “In the future, home robots will be sold to all sorts of households and expected to perform a wide range of tasks. We can't possibly program everything they need to know beforehand, so it’s essential that they can learn on the job. However, letting robots loose to explore and learn without guidance can be very slow and might lead to unintended consequences. The research by Silver and his colleagues introduces an algorithm that allows robots to practice their skills autonomously in a structured way. This is a big step towards creating home robots that can continuously evolve and improve on their own.”“This paper tackles a very challenging and interesting problem: how to enable a robot to improve a repertoire of low-level skills based on interactions with the environment, while sequencing these low-level behaviors into long-horizon plans,” says Florian Shkurti, University of Toronto assistant professor in robotics who was also not involved with the paper. “At the heart of this work is a new method for probabilistically modeling the competence of a skill as a function of the number of data collected, and prioritizing the practice of skills that will lead to the biggest expected improvement in a distribution of tasks. Every active learning algorithm crucially depends on these two design choices, and the paper shows that a simple probabilistic competence model and the expected improvement criterion lead to effective, targeted, and fast exploration.” Silver and Kumar’s co-authors are The AI Institute researchers Stephen Proulx and Jennifer Barry, plus four CSAIL members: Northeastern University PhD student and visiting researcher Linfeng Zhao, MIT EECS PhD student Willie McClinton, and MIT EECS professors Leslie Pack Kaelbling and Tomás Lozano-Pérez. Their work was supported, in part, by The AI Institute, the U.S. National Science Foundation, the U.S. Air Force Office of Scientific Research, the U.S. Office of Naval Research, the U.S. Army Research Office, and MIT Quest for Intelligence, with high-performance computing resources from the MIT SuperCloud and Lincoln Laboratory Supercomputing Center.

The phrase “practice makes perfect” is usually reserved for humans, but it’s also a great maxim for robots newly deployed in unfamiliar environments.

Helping robots practice skills independently to adapt to unfamiliar environments

A team of computer scientists, engineers, mathematicians and cognitive scientists, led by the University of Cambridge, developed an open-source evaluation platform called CheckMate, which allows human users to interact with and evaluate the performance of large language models (LLMs).The researchers tested CheckMate in an experiment where human participants used three LLMs – InstructGPT, ChatGPT and GPT-4 – as assistants for solving undergraduate-level mathematics problems.The team studied how well LLMs can assist participants in solving problems. Despite a generally positive correlation between a chatbot’s correctness and perceived helpfulness, the researchers also found instances where the LLMs were incorrect, but still useful for the participants. However, certain incorrect LLM outputs were thought to be correct by participants. This was most notable in LLMs optimised for chat.The researchers suggest models that communicate uncertainty, respond well to user corrections, and can provide a concise rationale for their recommendations, make better assistants. Human users of LLMs should verify their outputs carefully, given their current shortcomings.The&nbsp;<a href="https://www.pnas.org/doi/10.1073/pnas.2318124121" target="_blank">results</a>, reported in the&nbsp;Proceedings of the National Academy of Sciences (PNAS), could be useful in both informing AI literacy training, and help developers improve LLMs for a wider range of uses.While LLMs are becoming increasingly powerful, they can also make mistakes and provide incorrect information, which could have negative consequences as these systems become more integrated into our everyday lives.“LLMs have become wildly popular, and evaluating their performance in a quantitative way is important, but we also need to evaluate how well these systems work with and can support people,” said co-first author Albert Jiang, from Cambridge’s Department of Computer Science and Technology. “We don’t yet have comprehensive ways of evaluating an LLM’s performance when interacting with humans.”The standard way to evaluate LLMs relies on static pairs of inputs and outputs, which disregards the interactive nature of chatbots, and how that changes their usefulness in different scenarios. The researchers developed CheckMate to help answer these questions, designed for but not limited to applications in mathematics.“When talking to mathematicians about LLMs, many of them fall into one of two main camps: either they think that LLMs can produce complex mathematical proofs on their own, or that LLMs are incapable of simple arithmetic,” said co-first author&nbsp;<a href="https://www.eng.cam.ac.uk/profiles/kmc61" target="_blank">Katie Collins</a>, PhD student&nbsp;from the Department of Engineering. “Of course, the truth is probably somewhere in between, but we wanted to find a way of evaluating which tasks LLMs are suitable for and which they aren’t.”The researchers recruited 25 mathematicians, from undergraduate students to senior professors, to interact with three different LLMs (InstructGPT, ChatGPT, and GPT-4) and evaluate their performance using CheckMate. Participants worked through undergraduate-level mathematical theorems with the assistance of an LLM and were asked to rate each individual LLM response for correctness and helpfulness. Participants did not know which LLM they were interacting with.The researchers recorded the sorts of questions asked by participants, how participants reacted when they were presented with a fully or partially incorrect answer, whether and how they attempted to correct the LLM, or if they asked for clarification. Participants had varying levels of experience with writing effective prompts for LLMs, and this often affected the quality of responses that the LLMs provided.An example of an effective prompt is “what is the definition of X” (X being a concept in the problem) as chatbots can be very good at retrieving concepts they know of and explaining it to the user.“One of the things we found is the surprising fallibility of these models,” said Katie Collins. “Sometimes, these LLMs will be really good at higher-level mathematics, and then they’ll fail at something far simpler. It shows that it’s vital to think carefully about how to use LLMs effectively and appropriately.”However, like the LLMs, the human participants also made mistakes. The researchers asked participants to rate how confident they were in their own ability to solve the problem they were using the LLM for. In cases where the participant was less confident in their own abilities, they were more likely to rate incorrect generations by LLM as correct.“This kind of gets to a big challenge of evaluating LLMs, because they’re getting so good at generating nice, seemingly correct natural language, that it’s easy to be fooled by their responses,” said Albert Jiang. “It also shows that while human evaluation is useful and important, it’s nuanced, and sometimes it’s wrong. Anyone using an LLM, for any application, should always pay attention to the output and verify it themselves.”Based on the results from CheckMate, the researchers say that newer generations of LLMs are increasingly able to collaborate helpfully and correctly with human users on undergraduate-level maths problems, as long as the user can assess the correctness of LLM-generated responses. Even if the answers may be memorised and can be found somewhere on the internet, LLMs have the advantage of being flexible in their inputs and outputs over traditional search engines (though should not replace search engines in their current form).While CheckMate was tested on mathematical problems, the researchers say their platform could be adapted to a wide range of fields. In the future, this type of feedback could be incorporated into the LLMs themselves, although none of the CheckMate feedback from the current study has been fed back into the models.“These kinds of tools can help the research community to have a better understanding of the strengths and weaknesses of these models,” said Katie Collins. “We wouldn’t use them as tools to solve complex mathematical problems on their own, but they can be useful assistants, if the users know how to take advantage of them.”The research was supported in part by the Marshall Commission, the Cambridge Trust, Peterhouse, Cambridge, The Alan Turing Institute, the European Research Council, and the Engineering and Physical Sciences Research Council (EPSRC), part of UK Research and Innovation (UKRI).<hr>Reference: Katherine M. Collins, Albert Q. Jiang, et al. ‘<a href="https://www.pnas.org/doi/10.1073/pnas.2318124121" target="_blank">Evaluating Language Models for Mathematics through Interactions</a>.’ Proceedings of the National Academy of Sciences (2024). DOI: 10.1073/pnas.2318124121

Chatbot concept. Credit: wutzkoh – stock.adobe.com

Researchers have developed a platform for the interactive evaluation of AI-powered chatbots such as ChatGPT. 

New open-source platform allows users to evaluate performance of AI-powered chatbots

Nestled amidst the quaint towns and rolling hills of England lies the fictional town of Anghenfil, a place where whispers of a long-forgotten legend echo through the cobbled streets. However, the legend is about to become a terrifying reality in Every Legend Ends, a captivating fantasy film written and directed by UK-based filmmaker Ryan Garry.A Monster Awakens, a Hero EmergesThe film delves into the heart-pounding tale of a monstrous entity, slumbering for centuries beneath the earth. When the creature awakens, its rage unleashes a wave of destruction upon the unsuspecting town of Anghenfil. As fear and chaos grip the community, an unlikely hero emerges, determined to confront the beast and protect their home.Garry envisioned the film as a high-concept film, drawing inspiration from the epic grandeur of "The Lord of the Rings" and the action-packed thrills of the Marvel Cinematic Universe. However, achieving this level of visual storytelling presented a significant challenge for an independent production.Xsens Mocap: Breathing Life into the Monster<iframe width="640" height="360" src="https://www.youtube.com/embed/QfXj0KFSUI8?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>Behind the scenes: Xsens mocap brings monster vs. soldier fight to life!Creating a believable monster on screen is no easy feat. It requires careful consideration of both storytelling and technical execution. Garry anticipated challenges in syncing the monster's movements with live-action performers and ensuring realistic and fluid monster movements.Garry turned to&nbsp;<a href="https://www.movella.com/products/motion-capture" target="_blank" rel="noopener noreferrer">motion capture</a>, and Xsens became the secret weapon for bringing the film to life. Xsens offered the freedom to deploy the technology in various environments, a crucial factor considering the film's diverse shooting locations. From the damp and cavernous depths of Crank Caverns to the historic brickwork of the Williamson Tunnels, Xsens' lightweight and portable design allowed for seamless filming without technical limitations.The nature of the production needed efficient use of time and resources. With quick and easy setup, Xsens' self-powered system eliminated the need for bulky generators or cumbersome electrical setups, enabling the crew to work efficiently and capture scenes swiftly, even in remote locations.Unlike optical motion capture that requires separate recordings and post-production adjustments, Xsens allowed for simultaneous capture of the monster and live-action actors on set. This resulted in seamless interaction between the characters, enhancing the film's realism. A prime example is the thrilling fight scene between the monster and an armed response soldier, where Xsens' real-time capabilities eliminated the need for extensive post-production work.Capturing the nuances and complexities of the monster's movements, especially during fast-paced fight scenes, was crucial for creating a believable creature. Xsens mocap's accuracy ensured smooth and natural movements.The system's precision minimized the need for extensive post-production adjustments, saving valuable time and resources. The high-fidelity mocap data ensured the monster's movements appeared natural and believable, significantly contributing to the film's immersive experience.Every Legend Ends stands as a testament to the power of imagination and the ability of Xsens mocap technology to empower independent filmmakers. By embracing this cutting-edge technology, Garry has defied the limitations of independent production and crafted a visually stunning and captivating film, proving that even with limited resources, one can breathe life into a captivating story and create a truly immersive cinematic experience.The film is expected to be released later this year. &nbsp;Here is the trailer:&nbsp;<iframe width="640" height="360" src="https://www.youtube.com/embed/_fcgLke8mc0?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>

with Xsens motion capture

Breathing Life Into A Legend With Xsens Motion Capture

<h2 id="isPasted">At the top of many automation wish lists is a particularly time-consuming task: chores.</h2>The moonshot of many roboticists is cooking up the proper hardware and software combination so that a machine can learn “generalist” policies (the rules and strategies that guide robot behavior) that work everywhere, under all conditions. Realistically, though, if you have a home robot, you probably don’t care much about it working for your neighbors. MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) researchers decided, with that in mind, to attempt to find a solution to easily train robust robot policies for very specific environments.“We aim for robots to perform exceptionally well under disturbances, distractions, varying lighting conditions, and changes in object poses, all within a single environment,” says Marcel Torne Villasevil, MIT CSAIL research assistant in the Improbable AI lab and lead author on a recent paper about the work. “We propose a method to create digital twins on the fly using the latest advances in computer vision. With just their phones, anyone can capture a digital replica of the real world, and the robots can train in a simulated environment much faster than the real world, thanks to GPU parallelization. Our approach eliminates the need for extensive reward engineering by leveraging a few real-world demonstrations to jump-start the training process.”<iframe width="640" height="360" src="https://www.youtube.com/embed/w-L90BhfDFo?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><h2 id="isPasted">Taking your robot home</h2>RialTo, of course, is a little more complicated than just a simple wave of a phone and (boom!) home bot at your service. It begins by using your device to scan the target environment using tools like NeRFStudio, ARCode, or Polycam. Once the scene is reconstructed, users can upload it to RialTo’s interface to make detailed adjustments, add necessary joints to the robots, and more.The refined scene is exported and brought into the simulator. Here, the aim is to develop a policy based on real-world actions and observations, such as one for grabbing a cup on a counter. These real-world demonstrations are replicated in the simulation, providing some valuable data for reinforcement learning. “This helps in creating a strong policy that works well in both the simulation and the real world. An enhanced algorithm using reinforcement learning helps guide this process, to ensure the policy is effective when applied outside of the simulator,” says Torne.Testing showed that RialTo created strong policies for a variety of tasks, whether in controlled lab settings or more unpredictable real-world environments, improving 67 percent over imitation learning with the same number of demonstrations. The tasks involved opening a toaster, placing a book on a shelf, putting a plate on a rack, placing a mug on a shelf, opening a drawer, and opening a cabinet. For each task, the researchers tested the system’s performance under three increasing levels of difficulty: randomizing object poses, adding visual distractors, and applying physical disturbances during task executions. When paired with real-world data, the system outperformed traditional imitation-learning methods, especially in situations with lots of visual distractions or physical disruptions.“These experiments show that if we care about being very robust to one particular environment, the best idea is to leverage digital twins instead of trying to obtain robustness with large-scale data collection in diverse environments,” says Pulkit Agrawal, director of Improbable AI Lab, MIT electrical engineering and computer science (EECS) associate professor, MIT CSAIL principal investigator, and senior author on the work.As far as limitations, RialTo currently takes three days to be fully trained. To speed this up, the team mentions improving the underlying algorithms and using foundation models. Training in simulation also has its limitations, and currently it’s difficult to do effortless sim-to-real transfer and simulate deformable objects or liquids.<h2>The next level</h2>So what’s next for RialTo’s journey? Building on previous efforts, the scientists are working on preserving robustness against various disturbances while improving the model’s adaptability to new environments. “Our next endeavor is this approach to using pre-trained models, accelerating the learning process, minimizing human input, and achieving broader generalization capabilities,” says Torne.“We’re incredibly enthusiastic about our 'on-the-fly' robot programming concept, where robots can autonomously scan their environment and learn how to solve specific tasks in simulation. While our current method has limitations — such as requiring a few initial demonstrations by a human and significant compute time for training these policies (up to three days) — we see it as a significant step towards achieving 'on-the-fly' robot learning and deployment,” says Torne. “This approach moves us closer to a future where robots won’t need a preexisting policy that covers every scenario. Instead, they can rapidly learn new tasks without extensive real-world interaction. In my view, this advancement could expedite the practical application of robotics far sooner than relying solely on a universal, all-encompassing policy.”“To deploy robots in the real world, researchers have traditionally relied on methods such as imitation learning from expert data, which can be expensive, or reinforcement learning, which can be unsafe,” says Zoey Chen, a computer science PhD student at the University of Washington who wasn’t involved in the paper. “RialTo directly addresses both the safety constraints of real-world RL [robot learning], and efficient data constraints for data-driven learning methods, with its novel real-to-sim-to-real pipeline. This novel pipeline not only ensures safe and robust training in simulation before real-world deployment, but also significantly improves the efficiency of data collection. RialTo has the potential to significantly scale up robot learning and allows robots to adapt to complex real-world scenarios much more effectively.”"Simulation has shown impressive capabilities on real robots by providing inexpensive, possibly infinite data for policy learning,” adds Marius Memmel, a computer science PhD student at the University of Washington who wasn’t involved in the work. “However, these methods are limited to a few specific scenarios, and constructing the corresponding simulations is expensive and laborious. RialTo provides an easy-to-use tool to reconstruct real-world environments in minutes instead of hours. Furthermore, it makes extensive use of collected demonstrations during policy learning, minimizing the burden on the operator and reducing the sim2real gap. RialTo demonstrates robustness to object poses and disturbances, showing incredible real-world performance without requiring extensive simulator construction and data collection.”Torne wrote this paper alongside senior authors Abhishek Gupta, assistant professor at the University of Washington, and Agrawal. Four other CSAIL members are also credited: EECS PhD student Anthony Simeonov SM ’22, research assistant Zechu Li, undergraduate student April Chan, and Tao Chen PhD ’24. Improbable AI Lab and WEIRD Lab members also contributed valuable feedback and support in developing this project.This work was supported, in part, by the Sony Research Award, the U.S. government, and Hyundai Motor Co., with assistance from the WEIRD (Washington Embodied Intelligence and Robotics Development) Lab.

The moonshot of many roboticists is cooking up the proper hardware and software combination so that a machine can learn “generalist” policies (the rules and strategies that guide robot behavior) that work everywhere, under all conditions.

Precision home robots learn with real-to-sim-to-real

As artificial intelligence models become increasingly prevalent and are integrated into diverse sectors like health care, finance, education, transportation, and entertainment, understanding how they work under the hood is critical. Interpreting the mechanisms underlying AI models enables us to audit them for safety and biases, with the potential to deepen our understanding of the science behind intelligence itself.Imagine if we could directly investigate the human brain by manipulating each of its individual neurons to examine their roles in perceiving a particular object. While such an experiment would be prohibitively invasive in the human brain, it is more feasible in another type of neural network: one that is artificial. However, somewhat similar to the human brain, artificial models containing millions of neurons are too large and complex to study by hand, making interpretability at scale a very challenging task.&nbsp;To address this, MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) researchers decided to take an automated approach to interpreting artificial vision models that evaluate different properties of images. They developed “MAIA” (Multimodal Automated Interpretability Agent), a system that automates a variety of neural network interpretability tasks using a vision-language model backbone equipped with tools for experimenting on other AI systems.“Our goal is to create an AI researcher that can conduct interpretability experiments autonomously. Existing automated interpretability methods merely label or visualize data in a one-shot process. On the other hand, MAIA can generate hypotheses, design experiments to test them, and refine its understanding through iterative analysis,” says Tamar Rott Shaham, an MIT electrical engineering and computer science (EECS) postdoc at CSAIL and co-author on a new&nbsp;<a href="https://arxiv.org/pdf/2404.14394.pdf" target="_blank" data-lf-fd-inspected-kn9eq4roawkarlvp="true">paper about the research</a>. “By combining a pre-trained vision-language model with a library of interpretability tools, our multimodal method can respond to user queries by composing and running targeted experiments on specific models, continuously refining its approach until it can provide a comprehensive answer.”The automated agent is demonstrated to tackle three key tasks: It labels individual components inside vision models and describes the visual concepts that activate them, it cleans up image classifiers by removing irrelevant features to make them more robust to new situations, and it hunts for hidden biases in AI systems to help uncover potential fairness issues in their outputs. “But a key advantage of a system like MAIA is its flexibility,” says Sarah Schwettmann PhD ’21, a research scientist at CSAIL and co-lead of the research. “We demonstrated MAIA’s usefulness on a few specific tasks, but given that the system is built from a foundation model with broad reasoning capabilities, it can answer many different types of interpretability queries from users, and design experiments on the fly to investigate them.”&nbsp;<h2>Neuron by neuron</h2>In one example task, a human user asks MAIA to describe the concepts that a particular neuron inside a vision model is responsible for detecting. To investigate this question, MAIA first uses a tool that retrieves “dataset exemplars” from the ImageNet dataset, which maximally activate the neuron. For this example neuron, those images show people in formal attire, and closeups of their chins and necks. MAIA makes various hypotheses for what drives the neuron’s activity:&nbsp;facial expressions, chins, or neckties. MAIA then uses its tools to design experiments to test each hypothesis individually by generating and editing synthetic images — in one experiment, adding a bow tie to an image of a human face increases the neuron’s response. “This approach allows us to determine the specific cause of the neuron’s activity, much like a real scientific experiment,” says Rott Shaham.MAIA’s explanations of neuron behaviors are evaluated in two key ways. First, synthetic systems with known ground-truth behaviors are used to assess the accuracy of MAIA’s interpretations. Second, for “real” neurons inside trained AI systems with no ground-truth descriptions, the authors design a new automated evaluation protocol that measures how well MAIA’s descriptions predict neuron behavior on unseen data.The CSAIL-led method outperformed baseline methods describing individual neurons in a variety of vision models such as ResNet, CLIP, and the vision transformer DINO. MAIA also performed well on the new dataset of synthetic neurons with known ground-truth descriptions. For both the real and synthetic systems, the descriptions were often on par with descriptions written by human experts.How are descriptions of AI system components, like individual neurons, useful? “Understanding and localizing behaviors inside large AI systems is a key part of auditing these systems for safety before they’re deployed — in some of our experiments, we show how MAIA can be used to find neurons with unwanted behaviors and remove these behaviors from a model,” says Schwettmann. “We’re building toward a more resilient AI ecosystem where tools for understanding and monitoring AI systems keep pace with system scaling, enabling us to investigate and hopefully understand unforeseen challenges introduced by new models.”<h2>Peeking inside neural networks</h2>The nascent field of interpretability is maturing into a distinct research area alongside the rise of “black box” machine learning models. How can researchers crack open these models and understand how they work? Current methods for peeking inside tend to be limited either in scale or in the precision of the explanations they can produce. Moreover, existing methods tend to fit a particular model and a specific task. This caused the researchers to ask: How can we build a generic system to help users answer interpretability questions about AI models while combining the flexibility of human experimentation with the scalability of automated techniques?One critical area they wanted this system to address was bias. To determine whether image classifiers displayed bias against particular subcategories of images, the team looked at the final layer of the classification stream (in a system designed to sort or label items, much like a machine that identifies whether a photo is of a dog, cat, or bird) and the probability scores of input images (confidence levels that the machine assigns to its guesses). To understand potential biases in image classification, MAIA was asked to find a subset of images in specific classes (for example “labrador retriever”) that were likely to be incorrectly labeled by the system. In this example, MAIA found that images of black labradors were likely to be misclassified, suggesting a bias in the model toward yellow-furred retrievers.Since MAIA relies on external tools to design experiments, its performance is limited by the quality of those tools. But, as the quality of tools like image synthesis models improve, so will MAIA. MAIA also shows confirmation bias at times, where it sometimes incorrectly confirms its initial hypothesis. To mitigate this, the researchers built an image-to-text tool, which uses a different instance of the language model to summarize experimental results. Another failure mode is overfitting to a particular experiment, where the model sometimes makes premature conclusions based on minimal evidence.“I think a natural next step for our lab is to move beyond artificial systems and apply similar experiments to human perception,” says Rott Shaham. “Testing this has traditionally required manually designing and testing stimuli, which is labor-intensive. With our agent, we can scale up this process, designing and testing numerous stimuli simultaneously. This might also allow us to compare human visual perception with artificial systems.”“Understanding neural networks is difficult for humans because they have hundreds of thousands of neurons, each with complex behavior patterns. MAIA helps to bridge this by developing AI agents that can automatically analyze these neurons and report distilled findings back to humans in a digestible way,” says Jacob Steinhardt, assistant professor at the University of California at Berkeley, who wasn’t involved in the research. “Scaling these methods up could be one of the most important routes to understanding and safely overseeing AI systems.”Rott Shaham and Schwettmann are joined by five fellow CSAIL affiliates on the paper: undergraduate student Franklin Wang; incoming MIT student Achyuta Rajaram; EECS PhD student Evan Hernandez SM ’22; and EECS professors Jacob Andreas and Antonio Torralba. Their work was supported, in part, by the MIT-IBM Watson AI Lab, Open Philanthropy, Hyundai Motor Co., the Army Research Laboratory, Intel, the National Science Foundation, the Zuckerman STEM Leadership Program, and the Viterbi Fellowship.

The automated, multimodal approach developed by MIT researchers interprets artificial vision models that evaluate the properties of images. Image: iStock

MAIA is a multimodal agent that can iteratively design experiments to better understand various components of AI systems.

Researchers advance automated interpretability in AI models

Even if you’re not very familiar with deep learning, you’ve probably heard about it and how it can, among other things, help automate the driving experience, increase manufacturing efficiency and change the consumer shopping experience. Deep learning is the latest evolution of artificial intelligence (AI) and machine learning. Both machine learning and deep learning are subsets of AI. While traditional machine learning algorithms need to be specifically programmed by people with domain-level expertise, deep learning algorithms utilize neural networks that can be trained by feeding them data. The same network can be used to solve very different problems by training it with different data, removing the requirement of having specific expertise in the problem that needs to be solved. Because of this, deep learning is considered a foundational technology that has the potential to make an impact in many industries.For those new to deep learning, think of it as a technology that can classify things. Many real-world problems can be broken down into a classification problem, for example:<ul><li>Autonomous and semi-autonomous vehicles need to be able to classify roads, other vehicles, people and road signs as shown in Figure 1.</li><li>Smart factories need to be able to classify a product as defective</li><li>Automated retail outlets need to classify if a consumer is removing a product from a store for purchase or just looking at it</li></ul><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1722330537960-Screenshot+2024-07-30+110736.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 1. Classification Example of an Automotive Use Case. Every Object in the Scene Is Determined to Be One of the Five Different Classes.There are two main parts to deep learning: training and inference, as shown in Figure 2. During training, a model of a neural network that has been created to solve a classification problem is fed huge amounts of labeled data, from which it learns how to classify things. In the example scenario shown in Figure 1 of an autonomous vehicle, the model would be fed images of pedestrians, road signs, cars and roads with each of these objects correctly labeled. Real-time performance or power is not an issue during training, so it usually occurs on a desktop or cloud platform.During inference, the trained network that has been deployed into its end application makes a classification decision, such as determining whether or not a part on an assembly line is defective.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1722330524914-Screenshot+2024-07-30+110753.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 2. Typical Development Flow of a Deep Learning Network.For a combination of reasons – including reliability, low latency, privacy, power and cost – it is beneficial for inference to run close to where sensors gather data; that is, at the “edge” of the network. Advanced sensors that can handle some amount of AI are capable of running a simple inference in the sensor itself. For a more complex inference, a separate edge processor is needed.TI Sitara™ processors are great devices for edge processing. With a scalable portfolio integrating Arm® Cortex®-A cores with flexible peripherals, Sitara processors provide industrial-grade solutions with an extensive mix of processing, integration and power efficiency. With TI Deep Learning (TIDL) software, Sitara AM57x processors leverage hardware acceleration to improve the performance of machine learning and deep learning inference at the edge.TIDL software is a set of open-source Linux software packages included on the processor software development kit (SDK) and tools that enable the hardware-accelerated offloading of deep learning inference in an AM57x device. It’s possible to accelerate inference across embedded vision engine (EVE) subsystems, C66x digital signal processor cores or a combination of both, using a set of application programming interfaces built with Open Computing Language (OpenCL).For those who already have an existing neural network framework, TIDL software bridges the gap between the popular Caffe and TensorFlow frameworks with accelerated hardware on the AM57x through an import tool, as shown in Figure 3. To maximize efficiency on the hardware, or for those who don’t have an existing neural network framework, Caffe Jacinto, a TI-developed framework forked from NVIDIA/Caffe and designed for the embedded space, can perform the training part of deep learning.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1722330564360-Screenshot+2024-07-30+110804.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 3. Texas Instruments Deep Learning (TIDL) Software Development Flow.Note that while TI has developed a framework that can be used to train a network model for an embedded application, the training itself still takes place on a desktop or cloud platform. It is the inference that runs on the TI embedded processor.With free software as part of TI’s Processor SDK, training videos and a deep learning TI reference design, whether you’re new to deep learning or a seasoned expert, TI provides the building blocks to get you started quickly and easily.Additional Resources:• <a href="http://www.ti.com/tool/sitara-machine-learning" target="_blank" rel="noopener noreferrer">Visit the TI Sitara Machine Learning page.</a>• <a href="https://e2e.ti.com/blogs_/b/behind_the_wheel/archive/2018/02/08/ai-in-automotive-practical-deep-learning?" target="_blank" rel="noopener noreferrer">Read more about machine learning</a>

Even if you’re not very familiar with deep learning, you’ve probably heard about it and how it can, among
other things, help automate the driving experience, increase manufacturing efficiency and change the consumer
shopping experience.

Bringing the Next Evolution of Machine Learning to the Edge

Matroid-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Matroid-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Matroid-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Matroid-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Matroid-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Matroid

Visio Nerf-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Visio Nerf-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Visio Nerf-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Visio Nerf-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Visio Nerf-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Pulsar Fusion-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Pulsar Fusion-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Pulsar Fusion-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Pulsar Fusion-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Pulsar Fusion-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Fusion Bionic GmbH-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Fusion Bionic GmbH-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Fusion Bionic GmbH-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Fusion Bionic GmbH-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Fusion Bionic GmbH-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Plainsight-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Plainsight-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Plainsight-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Plainsight-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Plainsight-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

AILIVERSE-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

AILIVERSE-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

AILIVERSE-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

AILIVERSE-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

AILIVERSE-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Xsens-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Movella-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Movella-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Movella-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Movella-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Movella-Thu Oct 05 2023 18:51:53 GMT+0300 (Eastern European Summer Time)

Movella-Thu Oct 05 2023 18:51:58 GMT+0300 (Eastern European Summer Time)

Movella-Mon Oct 30 2023 14:34:24 GMT+0200 (Eastern European Standard Time)

Movella-Tue Oct 31 2023 23:59:59 GMT+0200 (Eastern European Standard Time)

Computer Vision

CUDA Cores and Tensor Cores are specialized units within NVIDIA GPUs; the former are designed for a wide range of general GPU tasks, while the latter are specifically optimized to accelerate AI and deep learning through efficient matrix operations.

Tensor Cores vs CUDA Cores: The Powerhouses of GPU Computing from Nvidia

An AI-generated depiction of a modern day chipset

Gaining a comprehensive understanding of the distinct capabilities and applications of TPUs and GPUs is crucial for developers and researchers aiming to navigate the complex and rapidly changing terrain of artificial intelligence.

TPU vs GPU in AI: A Comprehensive Guide to Their Roles and Impact on Artificial Intelligence

Clustering - A Common Unsupervised Learning Technique

Understanding What is Unsupervised Learning, the Mechanisms, Types, and Applications of Various Algorithms and Challenges it presents in Machine Learning

What is Unsupervised Learning: A Comprehensive Guide

Exploring the key concepts related to Unsupervised vs Supervised Learning, understanding the fundamental principles, major algorithms and their real-world applications, and practical distinctions between supervised and unsupervised learning.

Unsupervised vs Supervised Learning: A Comprehensive Comparison

Artificial intelligence (AI) is a wide-ranging tool that enables people to rethink how we integrate information, analyze data, and use the resulting insights to improve decision making
