[LCP]Questions about precision

Chuck Martin nrocinu at myrealbox.com
Wed Oct 9 14:58:01 UTC 2002


First, let me give an example:

#include <stdio.h>

main()
{
    printf("%.2f %.5f %.20f\n", 9.075, 9.075, 9.075);
}

When this is compiled and run, I get this:

9.07 9.07500 9.07499999999999928946

The 9.07 seemed to be an error at first, because it rounded down
instead of up like it seemed it should, until I tried the third one,
with a precision of 20 places to the right of the decimal (or anything
greater than 14).  I understand this is because 9.075 can't be
precisely represented as a double because a double is a binary
floating point format.  However, if I enter the number 9.075 into
either the Lotus 1-2-3 or Excel spreadsheet programs, and set the
output format to display the maximum number of places to the right
of the decimal (15 in Lotus and 30 in Excel), the number displayed
in either case is 9.075 followed by nothing but zeroes.  Both of
these programs also round up, as expected, to 9.08, if set to display
two places after the decimal, instead of down to 9.07.  Does anyone
know how these two programs might be getting that kind of precision?
Are they using something other than binary floating point numbers
for storing and calculating?

Chuck




More information about the linuxCprogramming mailing list