Odd issue comparing two numbers

Recommended Posts

Hi - I’m converting an algorithm from mat lab, and it’s going well, but I’m getting an odd error. When I print the output with ?n[100], I get 0.054, but when I try and compare the value to 0.054, I don’t get the correct evaluation. A working example follows:

global n

for (private count = 0.004, count <= 4, count += 1/2000)



? n[100] // output is 0.054

// now here is the problem

if (n[100] == 0.054)

   ? “Success”


Thanks for your help


Link to comment
Share on other sites

OK, this is actually a feature of floating point numbers in computers and has nothing to do specifically with DAQFactory.  The short of it is that while computers store integers exactly, the decimal part (right of the decimal place) is almost never stored exactly due to the way floating point values work.  DAQFactory uses double precision floating point so the effect is smaller than single precision, but it still means that 0.1, for example, is probably stored as 0.0999999999999998 or something like that.  When DAQFactory goes to display the value it shows as 0.1 because of various algorithms to clean up floating values, but internally it is actually slightly off from 0.1.  You actually rarely see this problem because the system does some automatic rounding, except for when you do something like what you did where you added a bunch of fractional values together and then compared it to a value that was directly entered.  The slight offset of 1/2000 is getting compounded so that by the time it is 0.054 it is actually slightly different.  Alas, there is no real easy way to see this.  However, try replacing your ? n[100] with:

? n[100]// output is 0.054
? n[100] == 0.054
? n[100] < 0.054001
? n[100] > 0.053999
? floor(n[100] * 1000) / 1000 == 0.054

What you'll see is:


The 0 is the comparison you did and doesn't compute because n[100] is actually slightly off from 0.054.  The next two show that it is just slightly off, and the last one shows how you can simply truncate anything after the 3rd decimal place and do the comparison and it works.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.