What are FP8, FP16, BF16 and FP32 in GPU computing?

Numerical precision describes how ac­cu­rate­ly GPUs represent and process numbers. In deep learning, the chosen floating-point format (such as FP32, FP16, BF16 or FP8) defines the balance between com­pu­ta­tion­al per­for­mance, memory re­quire­ments and model accuracy. It therefore has a sig­nif­i­cant impact on the ef­fi­cien­cy of AI ap­pli­ca­tions.

Key Takeaways ⓘ

The key facts about floating-point formats at a glance:

• Every floating-point number consists of a sign, an exponent and a mantissa: the exponent sets the dynamic range, the mantissa the precision.
• FP32 offers the highest accuracy and remains the reference for numerical stability, but needs the most memory and compute.
• FP16 halves memory re­quire­ments compared with FP32, but is prone to underflow and needs tech­niques such as loss scaling.
• BF16 provides the dynamic range of FP32 at half the memory footprint and is now, alongside FP16, a widely used standard for large language models.
• FP8 reduces memory use and latency con­sid­er­ably further, but requires suitable hardware and careful scaling.
• Modern NVIDIA GPUs ac­cel­er­ate these formats via Tensor Cores: FP8 from the Hopper gen­er­a­tion onwards, even smaller formats from Blackwell.
• In practice, AI frame­works combine several formats at once (mixed precision) to balance ef­fi­cien­cy and accuracy.

What are floating-point numbers?

Floating-point numbers are a way of rep­re­sent­ing real numbers in a computer. Instead of a fixed number of decimal places as with integers, the number is broken down into a kind of sci­en­tif­ic notation. They consist of three central com­po­nents:

• Sign
• Exponent
• Mantissa

The sign bit de­ter­mines whether a number is positive or negative. The exponent defines the range of values that can be rep­re­sent­ed, while the mantissa (also called the sig­nif­i­cand) de­ter­mines the precision — in other words, how ac­cu­rate­ly a number can be rep­re­sent­ed. More exponent bits increase the dynamic range, allowing a wider range of values. More mantissa bits improve the res­o­lu­tion between rep­re­sentable numbers.

A key trade-off is that higher precision produces more accurate results but requires more memory and com­pu­ta­tion­al power. Lower precision, by contrast, reduces resource usage and speeds up cal­cu­la­tions but can increase the risk of rounding errors. This is par­tic­u­lar­ly important for deep learning models, which often contain millions or even billions of pa­ra­me­ters. Depending on the format, special values such as NaN (“Not a Number”) or ±Inf (infinity) may also occur during certain arith­metic op­er­a­tions.

Very small numbers can cause underflow, where values lose accuracy, are stored as subnormal values or are rounded to zero. Con­verse­ly, overflow occurs when numbers exceed the rep­re­sentable range. To balance per­for­mance and stability, AI hardware often combines different floating-point formats in a process known as mixed precision. This approach has become standard in many training pipelines, par­tic­u­lar­ly for large models such as LLMs.

Com­par­i­son of the formats

The choice of floating-point format has a sig­nif­i­cant impact on per­for­mance, memory re­quire­ments and model stability. While FP32 was long con­sid­ered the standard, modern AI systems in­creas­ing­ly rely on reduced-precision formats. FP16, BF16 and, more recently, FP8 have become widely used options, par­tic­u­lar­ly in deep learning ap­pli­ca­tions.

FP32 (single precision)

FP32 is the classic format for sci­en­tif­ic computing and was long con­sid­ered the standard in deep learning. It provides high accuracy and a large dynamic range.

As the name suggests, FP32 uses a total of 32 bits: 1 bit for the sign, 8 bits for the exponent and 23 bits for the mantissa. This allows it to represent both very large and very small numbers with high stability. This level of accuracy is par­tic­u­lar­ly important for complex sim­u­la­tions and sensitive training processes. However, FP32 requires sig­nif­i­cant memory and com­pu­ta­tion­al resources.

In practice, FP32 is now mainly used for critical parts of a model, such as loss cal­cu­la­tion or gradient ac­cu­mu­la­tion. For large models like trans­form­ers, using FP32 ex­clu­sive­ly can be too slow and resource-intensive. Nev­er­the­less, it remains the reference standard for numerical stability.

FP16 (half precision)

FP16 was in­tro­duced to improve computing per­for­mance and reduce memory re­quire­ments. It is a key component of mixed-precision training.

FP16 uses only 16 bits: 1 sign bit, 5 exponent bits and 10 mantissa bits. This halves the memory re­quire­ments compared with FP32. As a result, GPUs can execute FP16 op­er­a­tions sig­nif­i­cant­ly faster, es­pe­cial­ly when using tensor cores. However, its dynamic range is more limited.

One challenge with FP16 is underflow, which can occur with very small values and is par­tic­u­lar­ly prob­lem­at­ic for gradients. To address this issue, training often uses loss scaling — a technique that helps avoid numerical errors by pre­vent­ing small values from being rounded to zero during cal­cu­la­tions. FP16 is well suited for neural networks, but it requires careful im­ple­men­ta­tion to maintain stability.

BF16 (Brain Floating Point)

The BF16 floating-point format was developed specif­i­cal­ly for deep learning and combines the ad­van­tages of FP32 and FP16. It is therefore par­tic­u­lar­ly relevant for large models such as trans­form­ers.

BF16 also uses 16 bits, but dis­trib­utes them dif­fer­ent­ly: 1 sign bit, 8 exponent bits and only 7 mantissa bits. This gives it the same dynamic range as FP32 while still reducing memory re­quire­ments.

The key advantage is that BF16 sig­nif­i­cant­ly reduces underflow problems compared with FP16. This is par­tic­u­lar­ly important when training large language models. As a result, many modern frame­works and hardware platforms favour BF16. However, the smaller mantissa results in lower precision compared with FP32.

FP8 (Ultra Low Precision)

FP8 is a rel­a­tive­ly new format used primarily for inference and optimized training pipelines. Its main goal is to maximize ef­fi­cien­cy.

FP8 uses only 8 bits and is available in different variants, such as E4M3 and E5M2, which use different al­lo­ca­tions of exponent and mantissa bits. This sig­nif­i­cant­ly reduces memory re­quire­ments. Modern GPUs such as the NVIDIA H100 support FP8 natively through Tensor Cores.

FP8 can deliver sig­nif­i­cant speed im­prove­ments, par­tic­u­lar­ly for large models. However, its precision is con­sid­er­ably lower than that of higher-precision formats. For this reason, FP8 is typically combined with other formats to maintain stability. Common use cases include FP8 inference and quantized training.

Direct com­par­i­son of the floating-point formats

FP32 and FP16 are stan­dard­ized by IEEE 754. BF16 and FP8 also follow the floating-point principle but are defined sep­a­rate­ly: BF16 is an industry format derived from FP32, while FP8 is available in variants such as E4M3 and E5M2 according to the OCP-OFP8 spec­i­fi­ca­tion.

Bare Metal Server

Per­for­mance through in­no­va­tion

• Dedicated en­ter­prise hardware
• Intel® Xeon® or AMD proces­sors
• Leading security tech­nolo­gies

See servers

Hardware support

Modern GPUs are specif­i­cal­ly designed to work ef­fi­cient­ly with reduced numerical precision. The NVIDIA Ampere ar­chi­tec­ture expanded Tensor Core support to include formats such as BF16 and TF32, while FP16 had already been supported in earlier gen­er­a­tions. This made mixed-precision training more ac­ces­si­ble and widely adopted in practical AI workflows.

The more recent NVIDIA Hopper Gen­er­a­tion extends this approach to FP8. GPUs such as the NVIDIA H100 include spe­cial­ized hardware units that can execute FP8 cal­cu­la­tions at very high speed.

The NVIDIA Blackwell ar­chi­tec­ture is the successor to Hopper and takes this de­vel­op­ment further with fifth-gen­er­a­tion Tensor Cores and support for ad­di­tion­al low-precision formats such as FP4. These formats are designed to further improve ef­fi­cien­cy for large AI models, par­tic­u­lar­ly in inference and training workflows. FP8 remains an important format for modern AI systems, while newer ultra-low-precision ap­proach­es aim to increase through­put while main­tain­ing model accuracy.

The NVIDIA Ada Lovelace ar­chi­tec­ture is also highly optimized for FP16 and BF16 workloads and supports FP8 in certain models and use cases.

Tensor Cores play a central role in this de­vel­op­ment. They ac­cel­er­ate matrix op­er­a­tions, which are essential for neural networks. Lower precision allows GPUs to process more cal­cu­la­tions in parallel, sig­nif­i­cant­ly improving speed and ef­fi­cien­cy.

The practical benefits depend not only on the hardware but also on the software ecosystem. Frame­works such as PyTorch and Ten­sor­Flow support mixed precision and can often select suitable formats au­to­mat­i­cal­ly.

However, not every GPU supports all formats equally. Older models typically rely mainly on FP32, while modern AI ac­cel­er­a­tors are designed specif­i­cal­ly for FP16, BF16 and FP8 workloads. For large language models, BF16 and FP16 are now widely used standards. FP8 is also gaining adoption for training workflows, such as with DeepSeek-V3, but requires tech­niques such as fine-grained scaling and FP32 ac­cu­mu­la­tion to maintain stability.

How quan­ti­za­tion and down­cast­ing improve ef­fi­cien­cy

In practical ap­pli­ca­tions, reduced floating-point formats have a sig­nif­i­cant impact on the speed, memory re­quire­ments and operating costs of AI systems. Con­vert­ing cal­cu­la­tions from FP32 to FP16 or BF16 can sub­stan­tial­ly reduce memory usage, allowing larger models or higher batch sizes to run within the GPU’s available memory.

Smaller data formats also allow GPUs to perform more cal­cu­la­tions in parallel. This increases through­put and reduces both training and inference times, es­pe­cial­ly for large neural networks with billions of pa­ra­me­ters. A common op­ti­miza­tion technique is down­cast­ing, where data or model pa­ra­me­ters are converted from a larger format such as FP32 to a smaller format such as FP16 or BF16. This reduces memory con­sump­tion and bandwidth re­quire­ments while improving per­for­mance.

Quan­ti­za­tion takes this approach further by con­vert­ing models into even smaller formats, such as FP8 or integer data types. This can further reduce memory re­quire­ments and improve latency, meaning the response time of an AI system. Quan­ti­za­tion is therefore par­tic­u­lar­ly important for pro­duc­tion AI ap­pli­ca­tions and real-time inference.

However, reducing precision always involves a trade-off between ef­fi­cien­cy and accuracy. If precision is reduced too far, rounding errors can increase and neg­a­tive­ly affect model quality. For this reason, AI frame­works often combine multiple formats within the same workflow. Critical cal­cu­la­tions may still use higher precision, while less sensitive op­er­a­tions are performed with FP16, BF16 or FP8.