In this episode, we delve into the ever-evolving world of hybrid AI and generative AI models.

Podcast: The Prius of Artificial Intelligence: Hybrid AI Models

<p>In today&rsquo;s evolving technology landscape, AI is not merely &ldquo;artificial&rdquo;; it is artfully intelligent. But unlike art, AI&rsquo;s practical applicability requires high levels of efficiency, a challenge the industry still grapples with.</p><p>Particularly with deep neural networks, we seem to encounter a challenge. While these networks dazzle us with their ability to recognize faces, translate languages, and make complex decisions, they also come with a voracious appetite for resources. High memory, compute power, and energy consumption have become almost defining characteristics of contemporary AI models, holding AI back from true ubiquity and rendering the need for operational efficiency ever more pressing.</p><p>In this article, we take a deeper look into Qualcomm AI Research&rsquo;s model efficiency research aimed at enhancing power efficiency and boosting AI performance. From the intricacies of Neural Architecture Search (NAS) to the groundbreaking quantization methods, we will discover how Qualcomm AI Research is leading the charge to unlock the full potential of AI.</p><h2 dir="ltr">Qualcomm AI Research&#39;s Focus on Model Efficiency</h2><p dir="ltr"><a href="https://www.qualcomm.com/research/artificial-intelligence/ai-research" target="_blank">Qualcomm AI Research</a> is at the cutting edge of AI, committed to tackling the intricate challenge of AI model efficiency through practical strategies that aim to enhance both power efficiency and performance. With its aim to make AI&rsquo;s core capabilities ubiquitous across devices, Qualcomm AI Research has adopted a holistic approach to AI model efficiency, delving into the practical aspects of full-stack AI optimization.&nbsp;</p><p>Addressing AI&#39;s resource-intensive nature requires an approach that encompasses machine learning hardware, software, and algorithms. As a result, their research efforts aim to ensure that AI systems operate efficiently without compromising functionality, making AI more adaptable and efficient across a diverse range of devices, particularly mobile devices. As Jilei Hou, VP Engineering at Qualcomm AI Research, states, &ldquo;Our holistic systems-approach to full-stack AI is accelerating the pipeline from research to commercialization.&rdquo;</p><p>But what exactly is this holistic approach to AI model efficiency? How is it implemented, and what angles does Qualcomm AI Research tackle the model efficiency challenge from?</p><h2 dir="ltr">A Holistic Approach to AI Model Efficiency</h2><p dir="ltr">Qualcomm AI Research&#39;s mission to unlock the full potential of AI centers around a comprehensive strategy that attacks the model efficiency challenge from multiple angles. This multifaceted approach is driven by the recognition that optimizing AI models requires a combination of techniques.</p><p>Basically, there is no one-size-fits-all solution to AI model efficiency. &nbsp;Just like art, shrinking AI models down and running them efficiently on hardware can and should be approached in multiple ways, including quantization, compression, Neural Architecture Search (NAS), and compilation.</p><p>Qualcomm AI Research is a strong believer in <a href="https://www.qualcomm.com/news/onq/2019/03/heres-why-quantization-matters-ai" target="_blank">quantization</a> as evidenced by their leading research and products in the market. Quantization enables the AI model to run efficiently on dedicated hardware, thus enhancing its performance while minimizing its consumption of power and memory bandwidth. With focus on quantization-aware training and post-training quantization, Qualcomm AI Research&rsquo;s results demonstrate how effective integer inference is in improving the tradeoff between accuracy and on-device latency.</p><p>With compression, Qualcomm AI Research aims at reducing the size of AI models without compromising their functionality by removing parameters. In other words, with no sacrifice to model accuracy, Qualcomm AI Research&rsquo;s compression technique systematically removes activation nodes and connections between nodes, rendering the AI model smaller and more efficient. Quantization and compression of neural networks are supported by Qualcomm Innovation Center&rsquo;s <a href="https://github.com/quic/aimet" target="_blank">AI Model Efficiency Toolkit (AIMET)</a> library.</p><p>Another approach is Neural Architecture Search (NAS). Their research in NAS focuses on automating the design of efficient neural networks that are much smaller than a large state-of-the-art model. Qualcomm AI Research seeks to develop search algorithms and methodologies that automatically create optimal network architectures tailored to specific tasks. This streamlines the process of designing AI models, enhancing their efficiency and performance.</p><p>Furthermore, advanced compilation techniques optimize the execution of AI models on various hardware platforms. By tailoring AI models to the underlying hardware architecture, they ensure that the models run efficiently, achieving the desired performance while conserving resources.</p><p>In the following subsections, we look deeper into quantization and NAS, highlighting how Qualcomm AI Research is enabling these approaches to unlock AI&rsquo;s potential even further.</p><h3 dir="ltr">A Closer Look at Quantization</h3><p dir="ltr">Quantization reduces the number of bits necessary for representing information in the weights and activations of AI models, boosting power efficiency and performance while reducing memory bandwidth and storage. It also helps enable simultaneous use of multiple AI models, a use case found in demanding application areas like mobile, automotive, and XR.</p><p>Qualcomm AI Research has been at the forefront of research in quantization techniques that are revolutionizing power-efficient AI. Their research covers both quantization-aware training and post-training quantization for efficient integer AI inference. Their products support quantization down to 4-bit integers, showcasing their commitment to making AI more power-efficient.&nbsp;</p><p>In fact, some of their recent<a href="https://arxiv.org/abs/2203.11086" target="_blank">&nbsp;findings</a> revealed issues that affect model accuracy, one of which is oscillating weights during model training. With state-of-the-art (SOTA) methods called oscillation dampening and iterative freezing, accuracy is improved even for under 4-bit quantization. Qualcomm AI Research has also explored the role of <a href="https://www.qualcomm.com/news/onq/2023/04/floating-point-arithmetic-for-ai-inference-hit-or-miss" target="_blank">floating-point arithmetic</a> in AI inference, shedding light on the balance between precision and efficiency. This research considers the trade-offs involved in using floating-point or integer representations in AI models, concluding that integer inference is the best approach. Another recent study on post-training quantization (PTQ) and its impact on softmax activation highlights a technique proposed by Qualcomm AI Research, called <a href="https://arxiv.org/abs/2309.01729" target="_blank">offline bias correction</a>. With this method, one can enhance the quantizability of softmax with no additional compute during deployment, thus significantly improving the accuracy for 8-bit quantized softmax.</p><p>Quantization is a highly-effective approach to AI model efficiency. Qualcomm AI Research&#39;s leading quantization research combined with Qualcomm&rsquo;s leading processor solutions with AI acceleration supporting 8-bit or 4-bit integers has resulted in industry leading performance per watt, showing the importance of tailoring AI models to the specific constraints of edge devices.</p><h3 dir="ltr">Neural Architecture Search (NAS)</h3><p dir="ltr">With Neural Architecture Search (NAS), Qualcomm AI Research focuses on automating the design and training of efficient neural networks, aligning them more closely with dedicated tasks and optimizing their overall performance. NAS consists of four components:</p><ul><li dir="ltr"><p dir="ltr">A search space (what types of networks can be searched over)</p></li><li dir="ltr"><p dir="ltr">An accuracy predictor (how accurate a network should be)</p></li><li dir="ltr"><p dir="ltr">A latency predictor (how fast a network will run)</p></li><li dir="ltr"><p dir="ltr">A search algorithm (what the best architecture is for a particular task)</p></li></ul><p>However, NAS still presents some intricate challenges, particularly in defining an expansive search space, where countless network architectures can be explored. This space represents the potential configurations that an AI model could take. The challenge lies in navigating this vast terrain to discover the most efficient and effective neural network for a given task, as well as addressing the high compute cost, unreliable estimates for hardware performances, and inefficient scaling of existing methods.</p><p>That&rsquo;s why Qualcomm AI Research has addressed these challenges with their NAS research called <a href="https://arxiv.org/pdf/2012.08859.pdf" target="_blank" data-lf-fd-inspected-kn9eq4roawkarlvp="true">DONNA (short for Distilling Optimal Neural Network Architectures)</a>.</p><h4 dir="ltr">DONNA: Distilling Optimal Neural Network Architectures</h4><p dir="ltr">An efficient NAS with hardware-in-the-loop optimization, DONNA is a scalable method capable of finding optimal network architectures with high accuracy and minimal latency for virtually any hardware platform at low cost. This revolutionary method tackles the model deployment challenges in real scenarios effectively thanks to its:</p><ul><li dir="ltr"><p dir="ltr">Diverse search space</p></li><li dir="ltr"><p dir="ltr">Low compute cost</p></li><li dir="ltr"><p dir="ltr">Scalability</p></li><li dir="ltr"><p dir="ltr">Reliable direct hardware measurement</p></li></ul><p>Its diverse search space includes the usual variable kernel-size, expansion-rate, depth, channel number, and cell-type, but can also search over activations and attention, a vital parameter to finding optimal architectures. Its hardware predictor is hardware agnostic with a low startup cost, enabling NAS to scale to multiple hardware devices at low costs. Furthermore, DONNA is able to capture the run-time version and hardware architecture, enabling users to find real and accurate latency values rather than simulated ones.</p><p>DONNA also has pre-trained blocks that enable fast fine-tuning of neural networks to fast track reaching full accuracy and is a single scalable solution, applying, for instance, directly to downstream tasks and non-CNN neural architectures with no need for conceptual code changes.</p><h2 dir="ltr">Open-Source AI Model Efficiency Projects</h2><p dir="ltr">Qualcomm AI Research believes that open-source projects are a great way to share knowledge for scaling model-efficient AI to the masses. That&rsquo;s why Qualcomm Innovation Center has provided two open-source GitHub projects based on Qualcomm AI Research&rsquo;s advanced research:</p><ul><li dir="ltr"><p dir="ltr"><a href="https://www.qualcomm.com/news/onq/2020/05/open-sourcing-ai-model-efficiency-toolkit" target="_blank">AI Model Efficiency Toolkit (AIMET)</a></p></li><li dir="ltr"><p dir="ltr"><a href="https://www.qualcomm.com/news/onq/2021/01/aimet-model-zoo-highly-accurate-quantized-ai-models-are-now-available" target="_blank">AIMET Model Zoo</a></p></li></ul><p>AIMET is a toolkit that offers advanced quantization and compression techniques, while AIMET Model Zoo offers pre-trained 8-bit quantized models with high accuracy. With these open-source projects, Qualcomm aims to boost the transition to fixed-point quantized models, which would radically enhance AI applications, particularly those requiring high performance, low latency, and low power consumption, such as mobile platforms.</p><h2 dir="ltr">Advancing the AI Frontier</h2><p dir="ltr">As Qualcomm&rsquo;s OnQ blog series states, &ldquo;<a href="https://www.qualcomm.com/content/dam/qcomm-martech/dm-assets/documents/Whitepaper-The-future-of-AI-is-hybrid-Part-1-Unlocking-the-generative-AI-future-with-on-device-and-hybrid-AI.pdf" target="_blank" data-lf-fd-inspected-kn9eq4roawkarlvp="true">The future of AI is hybrid</a>&rdquo;, AI is evolving in that direction, with AI processing being distributed between edge devices and the cloud. On-device AI provides benefits in cost, power consumption, performance, personalization, privacy, and security. But for all this to happen, model efficiency is key, and that&rsquo;s where Qualcomm is leading the pack as the on-device AI leader.</p><p>Qualcomm AI Research envisions a future where AI processing is highly efficient for all devices across all applications all over the world. With generative AI pushing the computing requirements even higher, Qualcomm&rsquo;s focus on efficiency and performance alongside their open ecosystem is enabling generative AI and other upcoming revolutionary technologies.</p><p>Whether it&#39;s the quantization approach, NAS (and DONNA), compression, or compilation, Qualcomm AI Research will continue to find and develop ways of making AI processing ever more efficient so you and I can enjoy the perks of amazing AI technologies on almost every device you could think of.</p><p>Join Qualcomm&rsquo;s open-source projects (<a href="https://www.qualcomm.com/news/onq/2020/05/open-sourcing-ai-model-efficiency-toolkit" target="_blank">AIMET</a> and <a href="https://www.qualcomm.com/news/onq/2021/01/aimet-model-zoo-highly-accurate-quantized-ai-models-are-now-available" target="_blank">AIMET Model Zoo</a>) and be part of the transformation towards efficient AI processing. Interested in learning more about Qualcomm AI Research&rsquo;s work, check out the following links:</p><ul><li dir="ltr"><p dir="ltr"><a href="https://assets.qualcomm.com/MCC-ALL-Q322-0915-WB-ThefutureofmodelefficiencyforedgeAI_01RegistrationPage.html" target="_blank">The Future of Model Efficiency for Edge AI (on-demand webinar)</a></p></li><li dir="ltr"><p dir="ltr"><a href="https://www.qualcomm.com/content/dam/qcomm-martech/dm-assets/documents/Whitepaper-The-future-of-AI-is-hybrid-Part-1-Unlocking-the-generative-AI-future-with-on-device-and-hybrid-AI.pdf" target="_blank" data-lf-fd-inspected-kn9eq4roawkarlvp="true">The future of edge AI is hybrid (whitepaper)</a></p></li><li dir="ltr"><p dir="ltr"><a href="https://www.qualcomm.com/news/onq/2023/08/5-benefits-of-on-device-generative-ai" target="_blank">5 benefits of on-device generative AI</a></p></li></ul><p><a href="https://assets.qualcomm.com/mobile-computing-newsletter-sign-up.html" target="_blank">Sign up for our What&#39;s Next in AI and Computing newsletter</a>&nbsp;</p>

Live translation devices will benefit from advances in model efficiency. Image credit: TimeKettle

Model efficiency research is enhancing power efficiency and boosting AI performance.

Unlocking the Full Potential of AI: A Deep Dive into Qualcomm's Model Efficiency Research

<p id="isPasted">This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;</p><p><span contenteditable="false" draggable="true" class="fr-video fr-deletable fr-fvc fr-dvb fr-draggable"><iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/50b8e0ea-1cdb-4b34-87e6-4867feefade3?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe></span><br></p><p>The full article will continue below.</p><hr><p><em>This is the second article in a short series. Read the<a href="https://www.wevolver.com/article/balancing-the-cloud-and-the-edge-a-close-look-at-enabling-hybrid-ai" target="_blank" rel="noopener noreferrer">&nbsp;first piec</a>e here.</em></p><h2 dir="ltr" id="isPasted">Introduction</h2><p dir="ltr">Generative AI is a rapidly advancing field of artificial intelligence (AI) focused on creating novel data assets like images, text, or music. Such AI models are trained using extensive datasets, enabling them to create new data resembling what they were trained on. The growth in generative AI's development owes itself to several key factors: availability of large datasets, improved computing hardware capabilities, and the evolution of novel machine learning (ML) algorithms and architectures.</p><p dir="ltr">Large language models–based chatbots like ChatGPT, which can generate coherent and contextually relevant text and even images, have gained the interest and creativity of researchers, businesses, and the public. Generative models have demonstrated remarkable capabilities across a range of applications, from natural language understanding and text generation to creative content creation and even virtual assistants. However, as these models grow in complexity and sophistication, they bring to the forefront an increasingly pressing issue in the form of scalability challenges that apply to both training and inference.&nbsp;</p><p dir="ltr">These challenges encompass the need for significant computational resources, the growing size of models, and increasing costs. Latency and real-time responses primarily relate to the inference phase, while ethical considerations, interoperability, privacy, energy consumption, and cost management are critical factors throughout the entire lifecycle of AI systems, including training and inference.&nbsp;</p><p dir="ltr">One promising solution arises in response to these challenges - on-device AI. Deploying AI models directly on user devices (like smartphones and tablets) offers multiple advantages. It reduces reliance on cloud resources, cutting computational and energy costs. It improves privacy by processing data locally, reducing the need to send sensitive information to the cloud. It also lowers latency, supports offline functionality, and aligns with sustainability goals by reducing energy consumption.</p><h2 dir="ltr">What is Hybrid AI?</h2><p dir="ltr">To properly define Hybrid AI, we need to introduce the terms on-device AI and hybrid AI. On-Device AI, also known as edge AI or local AI, refers to the deployment of AI algorithms and models directly on user devices (smartphones, tablets, IoT devices, autonomous vehicles, etc.), without heavy reliance on cloud-based infrastructure. It enables real-time processing and decision-making on the device itself, reducing the need for constant data transmission to remote servers. Hybrid AI represents a synergy between local devices and cloud infrastructure, strategically dividing AI computations to optimize both user experiences and resource efficiency.</p><p dir="ltr">In certain situations, AI processing predominantly occurs on the device, with the option to delegate tasks to the cloud as needed. In cloud-centric setups, devices will opportunistically offload AI workloads from the cloud, where possible and subject to their capabilities. Employing a hybrid AI architecture, or relying solely on on-device AI, yields numerous advantages including cost savings, improved energy efficiency, enhanced performance, heightened privacy and security, and personalized experiences on a global scale.</p><p dir="ltr">Within a hybrid AI framework, on-device AI processing delivers consistent performance that can rival or surpass cloud-based alternatives, particularly in scenarios characterized by congested cloud servers and network connectivity issues. Instances of high demand for generative AI queries in the cloud can result in lengthy queues and elevated latency, and in extreme cases, lead to service disruptions. These challenges can be mitigated by redistributing computational workloads to edge devices.</p><p dir="ltr">Furthermore, the presence of on-device processing capabilities in a hybrid AI architecture enables generative AI applications to function seamlessly in locations with limited or no connectivity. In a device-centric hybrid AI setup, the device serves as the central point of operation, with cloud resources primarily utilized for tasks that exceed the device's processing capacity. Given that many generative AI models can efficiently run on the device, edge devices can handle the bulk of the processing load. As the development of on-device AI continues, it holds significant potential to revolutionize AI service deployment and user experiences while maintaining a balance between performance and ethical considerations.</p><h2 dir="ltr">Unique Benefits of On-Device AI</h2><p dir="ltr"><a href="https://www.qualcomm.com/news/onq/2023/08/5-benefits-of-on-device-generative-ai" target="_blank">On-device AI offers several unique benefits</a>, including:</p><h3>Cost-efficiency&nbsp;</h3><p>By processing data locally, it reduces the need for extensive cloud infrastructure, which can be expensive to maintain and scale. On-device processing eliminates the necessity of sending data to the cloud, a process that can be both expensive and time-consuming. &nbsp;For instance, on-device AI for image generation, can be employed to generate high-quality images in real-time without relying on cloud-based servers. A mobile photography application equipped with generative AI can apply artistic filters, improve image quality, or create artistic effects directly on the user's device.&nbsp;</p><p>This ensures that users can quickly and seamlessly edit and generate images without uploading them to a remote server, preserving both privacy and reducing latency. Additionally, it allows for offline image generation, enabling users to create and edit images even when they're in locations with poor or no internet connectivity. This level of on-device image generation can significantly improve the user experience and provide powerful tools for content creation without compromising data privacy or speed.&nbsp;</p><h3>Energy Savings&nbsp;</h3><p>On-device AI mitigates the issue of energy consumption by performing computations locally. This is particularly crucial for devices with constrained battery life, like smartphones and wearables. Comparing the power efficiency of running generative AI models on edge devices versus to the cloud, edge devices offer superior performance per watt and reduced energy consumption, potentially resulting in environmental and cost benefits.&nbsp;</p><h3>Performance and Low Latency</h3><p>By processing tasks locally, it minimizes communication and computing delays associated with cloud-based AI. This is particularly important in real-time applications such as autonomous vehicles, medical devices, and interactive content, where even milliseconds of delay can have a significant impact on functionality and user experience.</p><h3>Enhanced Privacy and Security</h3><p>This approach reduces the risk of data breaches and unauthorized access during data transfer and cloud storage. Users can have greater confidence that their personal information remains on their devices, reinforcing privacy and security standards, especially for applications handling sensitive data, like financial transactions and medical information.</p><h3>Personalization and User Experience</h3><p>These models can adapt to the user's context and needs, improving user experiences by recommending relevant products or services based on their preferences and current circumstances. For example, a shopping application could use on-device AI to recommend products similar to past purchases or consider the user's location and weather conditions. Real-time personalization boosts relevance, engagement, and customer satisfaction.</p><h2 dir="ltr">Case Studies: On-Device AI in Action</h2><p dir="ltr">The following case studies illustrate the transformative potential of on-device AI across various domains, showcasing its benefits.</p><h3 dir="ltr" id="isPasted">Automotive</h3><p dir="ltr">In the automotive industry, AI-driven cockpits are revolutionizing the driving experience. These systems, like those in smartphones and PCs, are starting to employ in-vehicle digital assistants powered by generative AI to create highly personalized and hands-free interactions for drivers and passengers.&nbsp;</p><p dir="ltr">These digital assistants leverage personal and vehicle sensor data, including cameras and microphones, to improve the driving journey. Third-party services can also integrate seamlessly, improving navigation, offering proactive assistance, and enabling in-car commerce, such as ordering food. Furthermore, the cockpit experience is improved through individual recognition, customization of media content, and the rise of in-vehicle augmented reality. Vehicle maintenance becomes proactive as AI digital assistants analyze data to predict maintenance needs, to seamlessly communicate with users using natural language, and to provide precise repair guidance. This proactive approach not only enhances reliability but also effectively reduces costs.&nbsp;</p><p dir="ltr">Qualcomm, which leads the field in cockpit and in-vehicle infotainment solutions with numerous global automakers choosing Snapdragon® Cockpit Platforms. These platforms power digital cockpits, with many already in production or design stages. Partners include Honda, Mercedes, Renault, Volvo, and others. The Snapdragon Cockpit Platform's latest iteration focuses on superior in-vehicle user experiences, safety, comfort, and reliability, setting new standards for connected car digital cockpit solutions. Qualcomm's Snapdragon Ride™ Platform complements this, offering scalable automated driving SoC platforms for ADAS, featuring vision perception, parking, and path planning.</p><p dir="ltr"><span class="fr-img-caption fr-fic fr-dib" style="width: 302px;"><span class="fr-img-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk1NjUyNzU0OTg0LXF1YWxjb21tIGNhYmluLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 300px;" class="fr-fic fr-dib"><span class="fr-inner" spellcheck="false">&nbsp;Cockpit platforms are designed to transform in-vehicle experiences, supporting higher levels of compute and computing intelligence needed for advanced capabilities featured in next generation vehicles. Image credit: Qualcomm.</span></span></span></p><h3 dir="ltr" id="isPasted">Extended Reality</h3><p dir="ltr">For Extended Reality (XR), generative AI has the potential to democratize 3D content creation. In the near future, AI rendering tools will enable content creators to generate 3D objects and entire virtual worlds using various inputs like text, speech, images, or video. Furthermore, text-to-text models will facilitate lifelike conversations for avatars, transforming the way we interact with XR content. Despite the exciting possibilities, widespread adoption timelines remain uncertain, though significant progress is expected soon. Immersive XR experiences will benefit from text-to-image models for realistic textures, likely available on smartphones and XR devices within a year.&nbsp;</p><p dir="ltr">These capabilities will involve distributed processing, with headsets handling perception and rendering, while generative AI models run on paired smartphones or in the cloud. In a few years, text-to-3D and image-to-3D models will likely produce high-quality 3D objects, evolving to generate entire scenes and rooms. Eventually, XR may enable users to immerse themselves in virtual worlds constructed from their imagination. Virtual avatars will undergo a similar evolution, utilizing text-to-text models for natural conversations and text-to-image models for outfits. Over time, avatars may become increasingly photorealistic, fully animated, and mass-producible, driven by image-to-3D and encoder/decoder models. This progression promises to revolutionize how we create and experience immersive XR content, with continual advancements anticipated in the years ahead.</p><p dir="ltr"><span class="fr-img-caption fr-fic fr-dib" style="width: 302px;"><span class="fr-img-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk1NjUzMTYyMjIyLUNhbm5lcy1YUi1WaXJ0dWFsLUJyaW5ncy1JbW1lcnNpdmUtQXJ0LXRvLUxpZmUtV2l0aC1WaXJ0dWFsLVJlYWxpdHkucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib"><span class="fr-inner" spellcheck="false">Image credit: Cannes</span></span></span></p><h2 dir="ltr">Challenges and Solutions</h2><p dir="ltr">The technical challenges specific to hybrid AI are being actively addressed through ongoing research and innovation:</p><h3 dir="ltr">Model size</h3><p>Model size is a notable challenge, particularly on resource-constrained devices. As large models are challenging to store and run efficiently, a potential solution involves employing compression and quantization techniques to reduce model size while maintaining accuracy. <a href="https://stackoverflow.blog/2023/08/23/fitting-ai-models-in-your-pocket-with-quantization/" target="_blank">Qualcomm</a> is actively researching ways to enhance <a href="https://www.qualcomm.com/news/onq/2019/03/heres-why-quantization-matters-ai" target="_blank">quantization</a> techniques, specifically aiming to quantize 32-bit floating point parameters to 8-bit or 4-bit integers in neural networks while maintaining accuracy. The research encompasses various model quantization methods, such as Bayesian learning, adaptive rounding, and transformer quantization, as part of <a href="https://www.qualcomm.com/research/artificial-intelligence/ai-research" target="_blank">Qualcomm AI Research</a>'s ongoing efforts.&nbsp;</p><p>For instance, the Qualcomm AI Research team employed the <a href="https://www.forbes.com/sites/davealtavilla/2023/05/15/powering-ai-on-mobile-devices-requires-new-math-and-qualcomm-is-pioneering-it/?sh=54cb40c91a16" target="_blank">AIMET</a> tool (AI Model Efficiency Toolkit) to quantize the 32-bit floating point Stable Diffusion AI model into an 8-bit integer format while preserving almost the same accuracy level. By optimizing and scaling the model for INT8 precision operations instead of FP32, Qualcomm successfully enabled Stable Diffusion to run efficiently on handheld devices without internet connectivity, significantly reducing the model's size while maintaining accuracy and enhancing performance on low-power Snapdragon AI accelerators.</p><h3>Computational power</h3><p>Large AI models may place heavy computational demands, particularly on devices with restricted processing capabilities. A potential solution involves leveraging specialized hardware like AI accelerators to enhance AI model performance on such devices. In 2022, Snapdragon 8 Gen 2 provided groundbreaking AI, integrated across the entire system, powered by their fastest, most advanced Qualcomm AI Engine to date. Users can experience faster natural language processing with multi-language translation or have fun with AI cinematic video capture.&nbsp;</p><p>The latest Hexagon Processor introduced a dedicated power delivery system, which adapts power according to the workload. Special hardware improves group convolution, activation function acceleration, and the performance of the Hexagon Tensor Accelerator. Support for micro-tile inferencing and INT4 hardware acceleration provide even higher performance while reducing power and memory traffic. The transformer acceleration dramatically speeds up inference for multi-head attention that is used throughout generative AI, resulting in AI performance up to 4.35X on certain use cases with MobileBERT.&nbsp;</p><p dir="ltr">These ongoing research efforts are addressing the technical challenges of hybrid AI, with expectations to overcome them successfully as the technology advances and becomes more widely adopted and utilized.</p><h2 dir="ltr">Conclusion</h2><p dir="ltr">On-device AI plays a critical role in addressing the future scalability challenges of generative AI models. It has the potential to revolutionize numerous industries and significantly enhance user experiences. By enabling local processing and real-time personalization, on-device AI reduces the dependence on cloud resources, improves privacy and security as well as the efficiency of AI applications. This approach not only supports the responsible development of AI technologies but also opens new possibilities for scalable and personalized AI solutions across diverse sectors, from healthcare to entertainment, promising a brighter future for both businesses and users alike.</p><p>To stay up to date on the latest from Qualcomm, <a href="https://assets.qualcomm.com/mobile-computing-newsletter-sign-up.html" target="_blank">join the “What’s Next in AI and Computing” newsletter</a> and read their hybrid AI whitepapers (<a href="https://www.qualcomm.com/content/dam/qcomm-martech/dm-assets/documents/Whitepaper-The-future-of-AI-is-hybrid-Part-1-Unlocking-the-generative-AI-future-with-on-device-and-hybrid-AI.pdf" target="_blank" data-lf-fd-inspected-kn9eq4roawkarlvp="true">part 1</a>, <a href="https://www.qualcomm.com/content/dam/qcomm-martech/dm-assets/documents/Whitepaper-The-future-of-AI-is-hybrid-Part-2-Qualcomm-is-uniquely-positioned-to-scale-hybrid-AI.pdf" target="_blank" data-lf-fd-inspected-kn9eq4roawkarlvp="true">part 2</a>)</p>

Hybrid AI

Latest Posts

Podcast: The Prius of Artificial Intelligence: Hybrid AI Models

Unlocking the Full Potential of AI: A Deep Dive into Qualcomm's Model Efficiency Research

The Scalability Challenge: Focusing on Hybrid AI in the Future of Generative AI Models

Profiles