What are AI ac­cel­er­a­tors and how do GPUs, TPUs, NPUs and LPUs differ?

An AI ac­cel­er­a­tor is spe­cial­ized hardware designed to run neural networks faster and more ef­fi­cient­ly than con­ven­tion­al CPUs. While CPUs are built to handle a wide range of tasks flexibly, GPUs, TPUs, NPUs and LPUs are primarily optimized for high-through­put matrix, tensor and multiply–ac­cu­mu­late op­er­a­tions, which are fun­da­men­tal to deep learning.

Key Takeaways ⓘ

The key facts about AI ac­cel­er­a­tors at a glance:

• AI ac­cel­er­a­tors are spe­cial­ized chips for the matrix, tensor and multiply–ac­cu­mu­late op­er­a­tions that con­ven­tion­al CPUs handle only in­ef­fi­cient­ly.
• The GPU is the most versatile class: massively parallel, with tensor cores, high memory bandwidth and a mature software ecosystem around CUDA.
• The TPU is a Google ASIC whose systolic arrays reduce data movement, and it is used mainly through Google Cloud.
• The NPU sits in the system-on-chip of smart­phones and laptops and targets local inference at very low power con­sump­tion.
• Groq’s LPU uses large amounts of on-chip SRAM and compiler-con­trolled execution for par­tic­u­lar­ly low latency in token-by-token gen­er­a­tion.
• What matters is not peak per­for­mance but which stage of the AI lifecycle an ar­chi­tec­ture serves: training, cloud inference, on-device AI or latency-critical language models.

Why matrix op­er­a­tions require spe­cial­ized chips

Modern neural networks generate much of their com­pu­ta­tion­al load through linear algebra, primarily matrix mul­ti­pli­ca­tions and multiply–ac­cu­mu­late op­er­a­tions, in which values are mul­ti­plied and then added together. In trans­former models in par­tic­u­lar, as well as in many con­vo­lu­tion­al networks, this involves very large numbers of similar cal­cu­la­tions that can be processed ef­fi­cient­ly in parallel.

Con­ven­tion­al CPUs are only partly optimized for this type of workload. They are highly flexible, can handle complex program logic and respond quickly to in­di­vid­ual tasks. With AI models, however, a different priority often comes into play: per­form­ing as many simple arith­metic op­er­a­tions as possible in parallel while moving data to the computing units quickly enough. This is where AI ac­cel­er­a­tors come in. Their hardware is designed less for general-purpose program logic and more for high-through­put tensor and matrix op­er­a­tions.

Different classes of AI ac­cel­er­a­tor have emerged for these workloads:

• GPUs excel at massively parallel com­pu­ta­tion and are widely used for both training and inference.
• TPUs are designed specif­i­cal­ly for tensor op­er­a­tions, which are performed in large numbers by modern AI models.
• NPUs are in­creas­ing­ly in­te­grat­ed into smart­phones, laptops and other devices to run AI models directly on the device.
• LPUs are designed to provide par­tic­u­lar­ly fast and pre­dictable inference for large language models.

The dif­fer­ences between these ar­chi­tec­tures therefore go beyond raw computing power. They also lie in which stage of the AI lifecycle they are best suited to, whether that is compute-intensive training, efficient on-device execution or latency-sensitive inference for large language models.

GPU (Graphics Pro­cess­ing Unit)

The GPU is one of the key foun­da­tions of modern AI ac­cel­er­a­tor tech­nol­o­gy. It was orig­i­nal­ly developed for graphics pro­cess­ing, where large numbers of pixels, surfaces and shader programs need to be processed si­mul­ta­ne­ous­ly. GPUs therefore contain a large number of parallel computing units that operate according to the SIMT principle. This allows many threads to run at the same time while applying the same in­struc­tion to different data.

By the mid-2000s, GPUs already offered con­sid­er­able computing power, but using it for general-purpose tasks was cum­ber­some. De­vel­op­ers often had to express cal­cu­la­tions in graphical terms, such as shaders, textures or pixel op­er­a­tions.

With the in­tro­duc­tion of CUDA in 2006/2007, NVIDIA made GPUs much more ac­ces­si­ble for general-purpose computing beyond graphics. This helped turn graphics hardware into a platform for General-Purpose Computing on GPUs, or GPGPU. GPUs sub­se­quent­ly became widely used for training neural networks because they combine two major ad­van­tages: very high parallel computing per­for­mance and a mature software ecosystem built around CUDA, cuDNN, optimized libraries and direct support in frame­works such as PyTorch and JAX.

Modern data centre GPUs no longer consist solely of con­ven­tion­al CUDA cores. They also include spe­cial­ized tensor cores that can execute matrix op­er­a­tions par­tic­u­lar­ly ef­fi­cient­ly. These units support reduced-precision numeric formats such as FP16, FP8 and even FP4. Although these formats offer lower precision than tra­di­tion­al 32-bit floating-point numbers, they can provide sig­nif­i­cant­ly higher through­put and greater energy ef­fi­cien­cy. This is par­tic­u­lar­ly important for large AI models.

The key char­ac­ter­is­tics of GPUs include:

• Massively parallel ar­chi­tec­ture with many computing units designed to perform similar op­er­a­tions on large volumes of data.
• High memory bandwidth, often provided by HBM memory in data centre GPUs.
• Tensor cores for par­tic­u­lar­ly efficient matrix mul­ti­pli­ca­tion using reduced-precision numeric formats.
• Strong software ecosystem built around CUDA, cuDNN, optimized libraries and broad framework support.
• High flex­i­bil­i­ty for training, fine-tuning, sim­u­la­tions and a wide range of inference workloads.

GPUs are among the most versatile AI ac­cel­er­a­tors. They support both inference and compute-intensive model training, combining high through­put and memory bandwidth with a mature software ecosystem.

TPU (Tensor Pro­cess­ing Unit)

The TPU is an ap­pli­ca­tion-specific in­te­grat­ed circuit (ASIC), meaning a chip designed for a specific purpose. Google developed TPUs specif­i­cal­ly for AI workloads, par­tic­u­lar­ly the large matrix op­er­a­tions involved in training and running neural networks.

The main dif­fer­ence compared with a GPU is the degree of spe­cial­iza­tion. Despite its high level of par­al­lelism, a GPU remains a rel­a­tive­ly flexible processor designed to support a wide range of workloads. A TPU is more specif­i­cal­ly optimized for tensor and matrix op­er­a­tions. This makes it less versatile, but allows it to perform certain AI com­pu­ta­tions par­tic­u­lar­ly ef­fi­cient­ly. The main benefits are higher through­put and greater energy ef­fi­cien­cy per com­pu­ta­tion.

At the heart of the TPU ar­chi­tec­ture is the Matrix Multiply Unit (MXU), which uses a systolic array made up of many multiply–ac­cu­mu­late units. Data moves from one unit to the next in step with the clock, reducing the need to re­peat­ed­ly write in­ter­me­di­ate results to memory and read them back. This limits data movement, which can be just as important as raw computing power for AI ac­cel­er­a­tor per­for­mance.

TPUs are closely in­te­grat­ed with the Google ecosystem. Programs for TPUs are compiled into ex­e­cutable code for the hardware using XLA. While early TPUs were closely as­so­ci­at­ed with Ten­sor­Flow, newer gen­er­a­tions are also widely used with JAX and in­creas­ing­ly with PyTorch/XLA. TPU7x (Ironwood), for example, supports JAX and PyTorch but no longer Ten­sor­Flow. For external customers, TPUs are widely available as cloud resources through Google Cloud. Google is also expanding beyond its hosted cloud in­fra­struc­ture to provide TPUs directly to selected en­ter­prise customers for de­ploy­ment in their own data centres.

The key char­ac­ter­is­tics of TPUs include:

• ASIC design for spe­cial­ized AI workloads rather than general-purpose pro­gram­ma­bil­i­ty.
• Systolic arrays for efficient matrix mul­ti­pli­ca­tion with reduced data movement.
• High through­put per watt for suitable tensor and matrix op­er­a­tions.
• Close in­te­gra­tion with the Google Cloud ecosystem through XLA, JAX and PyTorch/XLA.
• Par­tic­u­lar­ly well suited to large-scale training and inference workloads in the cloud.

NPU (Neural Pro­cess­ing Unit)

The NPU brings AI ac­cel­er­a­tion out of the data centre and directly onto the end device. Today, NPUs are in­te­grat­ed into smart­phones, laptops, wearables and many embedded systems, including cameras, sensors and in­dus­tri­al equipment. Their aim is not to provide maximum computing power at any cost, but to enable local inference with the lowest possible power con­sump­tion. In other words, AI functions can run directly on the device without every request having to be sent to a data centre.

The key char­ac­ter­is­tics of NPUs include:

• Spe­cial­ized ac­cel­er­a­tor block within a system-on-chip, alongside the CPU and GPU.
• Optimized for local inference rather than large-scale model training.
• Very low power con­sump­tion, making them suitable for mobile devices, battery-powered operation and ap­pli­ca­tions where low heat gen­er­a­tion is important.
• Typical use cases include image recog­ni­tion, speech pro­cess­ing, live subtitles, camera effects, local assistant functions and smaller gen­er­a­tive models.

Unlike data centre GPUs, NPUs are not primarily focused on peak per­for­mance measured in TFLOPS. A common per­for­mance metric is TOPS, or tera op­er­a­tions per second. This indicates how many trillions of simple arith­metic op­er­a­tions per second an NPU can perform. These figures usually refer to integer-based, highly optimized cal­cu­la­tions, often using INT8 and, depending on the platform, even lower-precision formats.

LPU (Language Pro­cess­ing Unit)

The LPU is the most spe­cial­ized type of ac­cel­er­a­tor. It addresses the challenge of latency-critical inference, which is par­tic­u­lar­ly important for large language models. While training and the model’s prefill stage can be heavily par­al­lelized, text gen­er­a­tion itself is au­tore­gres­sive. The model generates one token at a time, with each new token depending on the preceding context.

Parallel pro­cess­ing still takes place within each token step, as the model’s matrix op­er­a­tions can be dis­trib­uted across many compute units. However, token gen­er­a­tion itself remains se­quen­tial because each new token depends on the tokens generated before it. This shifts the bot­tle­neck away from peak compute per­for­mance alone. For token-by-token gen­er­a­tion, the speed at which weights, in­ter­me­di­ate results and the KV cache move between memory, compute units and, where necessary, multiple chips becomes crucial. In con­ven­tion­al GPU systems, this can create a “memory wall,” where the compute units are fast enough but spend time waiting for data.

Groq is the most prominent example of this approach. In December 2025, NVIDIA entered into a non-exclusive licensing agreement for Groq’s LPU tech­nol­o­gy and hired several members of the Groq team. Groq remained an in­de­pen­dent company, while GroqCloud continued to operate sep­a­rate­ly. In 2026, NVIDIA in­te­grat­ed the Groq 3 LPU into its Vera Rubin platform. The NVIDIA Groq 3 LPX rack combines 256 LPU ac­cel­er­a­tors and is designed to work alongside Vera Rubin NVL72 for latency-sensitive inference workloads. Depending on the con­fig­u­ra­tion, GPUs and LPUs can handle different stages of inference, combining the flex­i­bil­i­ty and through­put of GPUs with the low-latency token gen­er­a­tion of LPUs.

Groq’s LPU uses a very different ar­chi­tec­ture from con­ven­tion­al GPUs. Rather than relying heavily on external HBM memory, large caches and dynamic runtime sched­ul­ing, it uses large amounts of on-chip SRAM and de­ter­min­is­tic, compiler-con­trolled execution. Data flows are planned in advance, making execution highly pre­dictable and enabling in­di­vid­ual requests to be processed with par­tic­u­lar­ly low latency.

The key char­ac­ter­is­tics of LPUs include:

• Optimized for LLM inference rather than general-purpose training.
• Very low and highly pre­dictable latency for token-by-token text gen­er­a­tion.
• Large amounts of on-chip SRAM instead of a strong reliance on external HBM memory.
• De­ter­min­is­tic, compiler-con­trolled execution rather than dynamic runtime sched­ul­ing.
• Per­for­mance metrics such as output tokens per second and time to first token rather than only TFLOPS or TOPS.

The trade-off for this degree of spe­cial­iza­tion is reduced flex­i­bil­i­ty. An LPU is not a general-purpose training platform like a GPU, but is designed primarily for inference. It is par­tic­u­lar­ly useful wherever language models need to respond quickly, such as in chatbots, voice agents, real-time as­sis­tants, agent systems or ap­pli­ca­tions that make many short model calls in rapid suc­ces­sion. It primarily supports open-weight and other openly available models. As the models available on inference platforms can change rapidly, specific model IDs should be checked regularly. At present, Groq focuses par­tic­u­lar­ly on GPT-OSS models and selected Qwen models.

How do GPUs, TPUs, NPUs and LPUs compare?

The four classes of AI ac­cel­er­a­tor cannot be compared solely on the basis of peak per­for­mance. What matters most is what each ar­chi­tec­ture is optimized for, whether that is maximum through­put during training, efficient cloud inference, on-device AI or par­tic­u­lar­ly low latency for language models. The following table sum­ma­rizes the key dif­fer­ences: