Chosenatrandom Posted November 27, 2019 Posted November 27, 2019 Hi - I’m converting an algorithm from mat lab, and it’s going well, but I’m getting an odd error. When I print the output with ?n[100], I get 0.054, but when I try and compare the value to 0.054, I don’t get the correct evaluation. A working example follows: global n for (private count = 0.004, count <= 4, count += 1/2000) n.append(count) endfor ? n[100] // output is 0.054 // now here is the problem if (n[100] == 0.054) ? “Success” endif Thanks for your help
AzeoTech Posted November 27, 2019 Posted November 27, 2019 OK, this is actually a feature of floating point numbers in computers and has nothing to do specifically with DAQFactory. The short of it is that while computers store integers exactly, the decimal part (right of the decimal place) is almost never stored exactly due to the way floating point values work. DAQFactory uses double precision floating point so the effect is smaller than single precision, but it still means that 0.1, for example, is probably stored as 0.0999999999999998 or something like that. When DAQFactory goes to display the value it shows as 0.1 because of various algorithms to clean up floating values, but internally it is actually slightly off from 0.1. You actually rarely see this problem because the system does some automatic rounding, except for when you do something like what you did where you added a bunch of fractional values together and then compared it to a value that was directly entered. The slight offset of 1/2000 is getting compounded so that by the time it is 0.054 it is actually slightly different. Alas, there is no real easy way to see this. However, try replacing your ? n[100] with: ? n[100]// output is 0.054 ? n[100] == 0.054 ? n[100] < 0.054001 ? n[100] > 0.053999 ? floor(n[100] * 1000) / 1000 == 0.054 What you'll see is: 0.054 0 1 1 1 The 0 is the comparison you did and doesn't compute because n[100] is actually slightly off from 0.054. The next two show that it is just slightly off, and the last one shows how you can simply truncate anything after the 3rd decimal place and do the comparison and it works.
Chosenatrandom Posted November 28, 2019 Author Posted November 28, 2019 Many thanks - I’ll put this to good use today.
Recommended Posts
Archived
This topic is now archived and is closed to further replies.