[?] Is my math (a bit) out here? - Printable Version
-Shoutbox (https://shoutbox.menthix.net)
+-- Forum: MsgHelp Archive (/forumdisplay.php?fid=58)
+--- Forum: Messenger Plus! for Live Messenger (/forumdisplay.php?fid=4)
+---- Forum: Scripting (/forumdisplay.php?fid=39)
+----- Thread: [?] Is my math (a bit) out here? (/showthread.php?tid=81924)
[?] Is my math (a bit) out here? by Spunky on 02-26-2008 at 11:30 AM
I have to add 0.1 to a number and keep doing it... but I get some wierd results =/
quote: Script has been stopped
Script is starting
Script is now loaded and ready
(!) 0.1
(!) 0.2
(!) 0.30000000000000004
(!) 0.4
(!) 0.5
(!) 0.6
(!) 0.7
(!) 0.7999999999999999
(!) 0.8999999999999999
(!) 0.9999999999999999
(!) 1.0999999999999998
(!) 1.2
Why is this?
EDIT:
A for loop causes this:
code: for(var x=0;x<10;x+=y){
Debug.Trace;
}
quote: Script has been stopped
Script is starting
Script is now loaded and ready
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
3
3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
3.9
4
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
4.9
5
5.1
5.2
5.3
5.4
5.5
5.6
5.7
5.8
5.9
5.99999999999999
6.09999999999999
6.19999999999999
6.29999999999999
6.39999999999999
6.49999999999999
6.59999999999999
6.69999999999999
6.79999999999999
6.89999999999999
6.99999999999999
7.09999999999999
7.19999999999999
7.29999999999999
7.39999999999999
7.49999999999999
7.59999999999999
7.69999999999999
7.79999999999999
7.89999999999999
7.99999999999999
8.09999999999999
8.19999999999999
8.29999999999999
8.39999999999999
8.49999999999999
8.59999999999999
8.69999999999999
8.79999999999998
8.89999999999998
8.99999999999998
9.09999999999998
9.19999999999998
9.29999999999998
9.39999999999998
9.49999999999998
9.59999999999998
9.69999999999998
9.79999999999998
9.89999999999998
9.99999999999998
RE: [?] Is my math (a bit) out here? by Jesus on 02-26-2008 at 11:59 AM
how is y made up?
maybe y != 0.1 but y == 0.0999999999999999999999
try a Debug.Trace(1000000000000 * y)
just an idea, don't know if it will do any good
edit: though that won't explain the
quote: (!) 0.30000000000000004
RE: [?] Is my math (a bit) out here? by Spunky on 02-26-2008 at 12:04 PM
y is definately 0.1
I'm testing the for loop with static variables
RE: [?] Is my math (a bit) out here? by Adeptus on 02-26-2008 at 12:10 PM
http://en.wikipedia.org/wiki/Floating_point
This is because whatever programming language and data type you use treats your numbers as IEEE floats. Floats are not exact by their nature; the linked article should explain why not. If you are going to use floats you should remember to code your program logic to never do exact comparisons -- only "greater than" or "less than".
If this is not acceptable, there are a few options. Your programming language may offer a built-in BCD (binary coded decimal) type or there may be a suitable free or commercial class available. If you are doing fixed point math, such as dealing with money, you can simply use integer types and represent the values using the smallest base unit (e.g. 115 cents instead of $1.15).
RE: [?] Is my math (a bit) out here? by Spunky on 02-26-2008 at 12:27 PM
quote: Originally posted by Adeptus
If you are doing fixed point math, such as dealing with money, you can simply use integer types and represent the values using the smallest base unit (e.g. 115 cents instead of $1.15).
My best bet I think ty
RE: [?] Is my math (a bit) out here? by Patchou on 02-26-2008 at 10:23 PM
I confirm what Adeptus said.
RE: [?] Is my math (a bit) out here? by Spunky on 02-26-2008 at 11:02 PM
I found away around it now, thanks for everyones help
|