| T O P I C R E V I E W |
| peter.cook |
Posted - 11/10/2003 : 3:04:29 PM
Hi,
Is this a bug or 'expected/normal' behaviour : if the number of decimal digits is set to >14 then $(14.7) returns 14.699999999999999.
Cheers,
Pete
|
| 2 L A T E S T R E P L I E S (Newest First) |
| peter.cook |
Posted - 11/12/2003 : 04:02:06 AM
Thanks for explanation Greg.
Cheers,
Pete
|
| greg |
Posted - 11/10/2003 : 5:33:30 PM
Computers have limited abilities when it comes to storing numbers. In particular, storing numbers in the 'double' format limits precision to about 15 or 16 decimal digits.
When you give a decimal value - like 14.7 - to a computer, that number gets converted to the binary format needed to store that number (as type double). In fact, for certain decimal values, this process yields a binary number that is as close as you can get to the real value, but not exact. One bit 'more' or one bit 'less' would be even further away from the real value.
So this is a fact of computer life that programmers should be aware of and deal with in their code.
|
|
|
