The Pentium FDIV bug is a hardware bug affecting the floating-point unit (FPU) of the early Intel Pentium processors. Because of the bug, the processor would return incorrect binary floating point results when dividing certain pairs of high-precision numbers. The bug was discovered in 1994 by Thomas R. Nicely, a professor of mathematics at Lynchburg College.^[1] Missing values in a lookup table used by the FPU's floating-point division algorithm led to calculations acquiring small errors. While these errors would in most use-cases only occur rarely and result in small deviations from the correct output values, in certain circumstances the errors can occur frequently and lead to more significant deviations.^[2]

The severity of the FDIV bug is debated. Though rarely encountered by most users (Byte magazine estimated that 1 in 9 billion floating point divides with random parameters would produce inaccurate results),^[3] both the flaw and Intel's initial handling of the matter were heavily criticized by the tech community.

In December 1994, Intel recalled the defective processors in what was the first full recall of a computer chip.^[4] In its 1994 annual report, Intel said it incurred "a $475 million pre-tax charge ... to recover replacement and write-off of these microprocessors."^[5]

In order to improve the speed of floating-point division calculations on the Pentium chip over the 486DX, Intel opted to replace the shift-and-subtract division algorithm with the Sweeney, Robertson, and Tocher (SRT) algorithm. The SRT algorithm can generate two bits of the division result per clock cycle, whereas the 486's algorithm could only generate one. It is implemented using a programmable logic array with 2,048 cells, of which 1,066 cells should have been populated with one of five values: −2, −1, 0, +1, +2. When the original array for the Pentium was compiled, five values were not correctly sent to the equipment that etches the arrays into the chips – thus five of the array cells contained zero when they should have contained +2.^[6]

As a result, calculations that rely on these five cells acquire errors; these errors can accumulate repeatedly owing to the recursive nature of the SRT algorithm. In pathological cases the error can reach the fourth significant digit of the result, although this is rare. The error is usually confined to the ninth or tenth significant digit.^[3]

Only certain combinations of numerator and denominator trigger the bug. One commonly-reported example is dividing 4,195,835 by 3,145,727. Performing this calculation in any software that used the floating-point coprocessor, such as Windows Calculator, would allow users to discover whether their Pentium chip was affected.^[7]

The correct value of the calculation is:

${\displaystyle \textstyle {\dfrac {4{,}195{,}835}{3{,}145{,}727}}=1.333820449136241002}$

When converted to the hexadecimal value used by the processor, 4,195,835 = 0x4005FB and 3,145,727 = 0x2FFFFF. The "5" in 0x4005FB triggers the access to the "empty" array cells. As a result, the value returned by a flawed Pentium processor is incorrect at or beyond four digits:^[8]

${\displaystyle \textstyle {\dfrac {4{,}195{,}835}{3{,}145{,}727}}=1.333{\color {red}{739068902037589}}}$

which is actually the value of 4,195,579/3,145,727 =4,195,835 - 256/3,145,727.

Thomas Nicely, a professor of mathematics at Lynchburg College, had written code to enumerate primes, twin primes, prime triplets, and prime quadruplets. Nicely noticed some inconsistencies in the calculations on June 13, 1994, shortly after adding a Pentium system to his group of computers, but was unable to eliminate other factors (such as programming errors, motherboard chipsets, etc.) until October 19, 1994.^[1] On October 24, 1994, he reported the issue to Intel.^[9] Intel had reportedly become aware of the issue independently by June 1994, and had begun fixing it at this point, but chose not to publicly disclose any details or recall affected CPUs.^[10]

On October 30, 1994, Nicely sent an email describing the bug to various academic contacts, requesting reports of testing for the flaw on 486-DX4s, Pentiums and Pentium clones.^[9] The bug was quickly verified by others, and news of it spread quickly on the Internet. The bug acquired the name "Pentium FDIV bug" from the x86 assembly language mnemonic for floating-point division, the most frequently used instruction affected.^[9]

The story first appeared in the press on November 7, 1994, in an article in Electronic Engineering Times, "Intel fixes a Pentium FPU glitch" by Alexander Wolfe,^[11] and was subsequently picked up by CNN in a segment aired on November 22. It was also reported on by the New York Times and the Boston Globe, making the front page in the latter.^[10][12]

At this point, Intel acknowledged the floating-point flaw, but claimed that it was not serious and would not affect most users. Intel offered to replace processors to users who could prove that they were affected. However, although most independent estimates found that the bug would have a very limited impact on most users, it caused significant negative press for the company. During a 2019 talk, while reflecting on development of Quake, John Romero described how frequently and persistently this bug could be reproduced by describing behavior Michael Abrash spent hours tracking down that would result in parts of a game level appearing unexpectedly when viewed from certain camera angles.^[13] IBM paused the sale of PCs containing Intel CPUs, and Intel's stock price decreased significantly.^[14] The motive behind IBM's decision was questioned by some in the industry; IBM produced the PowerPC CPUs at the time, and potentially stood to benefit from any reputational damage to the Pentium or Intel as a company. However, the decision led to corporate buyers of PC equipment demanding replacements of existing Pentium CPUs, and soon afterwards other PC manufacturers began offering "no questions asked" replacements of flawed Pentium chips.^[4]

The growing dissatisfaction with Intel's response led to the company offering to replace all flawed Pentium processors on request on December 20.^[15] On January 17, 1995, Intel announced a pre-tax charge of $475 million against earnings, ostensibly the total cost associated with replacement of the flawed processors.^[9] This is equivalent to $868 million in 2023.^[16] Intel was criticised for barring resellers and OEMs from participating in the recall program, requiring end-users to replace chips themselves. Intel's justification for this, posted on its support web page, was that "it is the individual decision of the end user to determine if the flaw is affecting their application accuracy".^[14]

A 1995 article in Science describes the value of number theory problems in discovering computer bugs and gives the mathematical background and history of Brun's constant, the problem Nicely was working on when he discovered the bug.^[17]

Intel's response to the FDIV bug has been cited as a case of the public relations impact of a problem eclipsing the practical impact of said problem on customers.^[18] While most users were unlikely to encounter the flaw in their day-to-day computing, the company's initial reaction to not replace chips unless customers could guarantee they were affected caused pushback from a vocal minority of industry experts. The subsequent publicity generated shook consumer confidence in the CPUs, and led to a demand for action even from people unlikely to be affected by the issue. Andy Grove, Intel's CEO at the time was quoted in The Wall Street Journal as saying "I think the kernel of the issue we missed ... was that we presumed to tell somebody what they should or shouldn't worry about, or should or shouldn't do".^[4]

In the aftermath of the bug and subsequent recall, there was a marked increase in the use of formal verification of hardware floating point operations across the semiconductor industry. Prompted by the discovery of the bug, a technique applicable to the SRT algorithm called "word-level model checking" was developed in 1996.^[19] Intel went on to use formal verification extensively in the development of later CPU architectures. In the development of the Pentium 4, symbolic trajectory evaluation and theorem proving were used to find a number of bugs that could have led to a similar recall incident had they gone undetected.^[20] The first Intel microarchitecture to use formal verification as the primary method of validation was Nehalem, developed in 2008.^[21]

The FDIV bug affects the 60 and 66 MHz Pentium P5 800 in stepping levels prior to D1, and the 75, 90, and 100 MHz Pentium P54C 600 in steppings prior to B5. The 120 MHz P54C and P54CQS CPUs are unaffected.^[22][23]

Various software patches were produced by manufacturers to work around the bug. One specific algorithm, outlined in a paper in IEEE Computational Science & Engineering, is to check for divisors that can trigger the access to the programmable logic array cells that erroneously contain zero, and if found, multiply both numerator and denominator by 15/16. This takes them out of the 'buggy' range. This fix does carry a measurable speed penalty - worst case for a program doing nothing but FDIV operations with bad divisors the running time would double since each FDIV would take about 80 instead of 40 clock cycles. With more random divisors the average time per FDIV was approximately 50 clock cycles, i.e. 10 cycles added to check the divisor: Only 5 out of 1024 random divisors would trigger the scaling fixup. Since FDIV is a rare operation in most programs, the normal slowdown with the fix installed was typically a percent or less.^[8]

The main challenge faced by software companies was implementing the fix in pre-existing software, much of which relied on libraries outside their control. Some companies, such as Wolfram Research, opted to directly patch the machine code of existing executables to replace the FDIV opcode with an illegal instruction. This would then trigger an exception that an exception handler (also patched in) would catch. From here, arbitrary code could be executed to work around the bug.^[2]

Microsoft offered operating system level workarounds in versions of Windows up to Windows XP. Utilities were included with the operating system to check for the presence of the bug and disable the FPU if found.^[24][25]

• Pentium F00F bug
• MOS Technology 6502 bugs and quirks
• Accuracy problems in floating point operations
• MaverickCrunch