[LCP]Questions about precision

[David Spencer]

It looks like it rounds according to the first undisplayed digit (FUD - 
I got bored typing this phrase over and over).
Using a for loop and %.*f is revealing:

dave at feathers:~/prog/c > cat float.c
main()
{
  int i;
  for (i=0; i<20; i++)
    printf("%.2f %.5f %2d %.*f\n",9.075,9.075,i,9.075);
}

./float
9.07 9.07500  0 9
9.07 9.07500  1 9.1
9.07 9.07500  2 9.07
9.07 9.07500  3 9.075
9.07 9.07500  4 9.0750
9.07 9.07500  5 9.07500
9.07 9.07500  6 9.075000
9.07 9.07500  7 9.0750000
9.07 9.07500  8 9.07500000
9.07 9.07500  9 9.075000000
9.07 9.07500 10 9.0750000000
9.07 9.07500 11 9.07500000000
9.07 9.07500 12 9.075000000000
9.07 9.07500 13 9.0750000000000
9.07 9.07500 14 9.07500000000000
9.07 9.07500 15 9.074999999999999
9.07 9.07500 16 9.0749999999999993
9.07 9.07500 17 9.07499999999999929
9.07 9.07500 18 9.074999999999999289
9.07 9.07500 19 9.0749999999999992895

i=1: The number is displayed as 9.1 as we would expect - the FUD is 7 so 
9.07* rounds up to 9.1.

i=2: Remember your point - that 9.075 cannot be represented exactly in 
binary.  It is converted by the compiler to 9.074* and therefore rounded 
to 2dp DOWN to 9.07.  If it were represented as 9.0750000001 then it 
would be rounded up.

This theory is borne out further by case i=15.  The FUD is a 3, so *9993 
is rounded down to *999.  At i=14 the FUD is 9 so *999 is rounded up to 
*000.

Dealing with exact floating point numbers in C is a bit of a pain.  One 
approach is to store an integer and exponent/scale, so you store 9075 
and 10^3.  When it comes to displaying the represented value, you can 
display I/S (9), a '.', and I-S*floor(I/S), not forgetting to use the 
appropriate format for after the dot, e.g: (I'm storing E and S 
separately for convenience.)

dave at feathers:~/prog/c > !ca
cat float.c
main()
{
  int I=9075, E=3, S=1000;
  int x=I/S;
  printf("Your number is %d.%0*d\n",x,E,I-S*x);
}
dave at feathers:~/prog/c > !cc
cc float.c -o float
dave at feathers:~/prog/c > !./
./float
Your number is 9.075

Replacing the values with those representing 9.0075 (90075, 4, 10000) 
displays 9.0075 with exactly the same printf.

If Excel works to 30 significant digits then it almost certainly doesn't 
use the standard C library.  It might use an alternative FP library, or 
perhaps it stores the numbers differently, like text or BCD.


One way of rounding to an integer is to add a half and round down. 
 Adapting this, and adding .005 could work:

dave at feathers:~/prog/c > !ca
cat float.c
main()
{
  printf("%.20f %.20f %.20f\n",9.075+0.005,9.075,0.005);
}

dave at feathers:~/prog/c > !./
./float
9.08000000000000007105 9.07499999999999928946 0.00500000000000000010

Even if 0.005 were interpreted as 0.0049999999 this would still work to 
2dp as the value would be 9.079*.


Dave.



Chuck Martin wrote:

>First, let me give an example:
>
>#include <stdio.h>
>
>main()
>{
>    printf("%.2f %.5f %.20f\n", 9.075, 9.075, 9.075);
>}
>
>When this is compiled and run, I get this:
>
>9.07 9.07500 9.07499999999999928946
>
>The 9.07 seemed to be an error at first, because it rounded down
>instead of up like it seemed it should, until I tried the third one,
>with a precision of 20 places to the right of the decimal (or anything
>greater than 14).  I understand this is because 9.075 can't be
>precisely represented as a double because a double is a binary
>floating point format.  However, if I enter the number 9.075 into
>either the Lotus 1-2-3 or Excel spreadsheet programs, and set the
>output format to display the maximum number of places to the right
>of the decimal (15 in Lotus and 30 in Excel), the number displayed
>in either case is 9.075 followed by nothing but zeroes.  Both of
>these programs also round up, as expected, to 9.08, if set to display
>two places after the decimal, instead of down to 9.07.  Does anyone
>know how these two programs might be getting that kind of precision?
>Are they using something other than binary floating point numbers
>for storing and calculating?
>  
>