unexpected behavior passing large hex as function parameter, etc.


shimh

Recommended Posts

When I run the following code, the result does not look correct.

global x
x= 0x80000001
? Format("0x%08X",test(x))

function test(in)
   return in

The expected result is 0x80000001, but we get 0x7FFFFFFF. Actually we always get 0x7FFFFFFF when x is larger than or equal to 0x80000000.

Now we change the code to the following, which is supposed to be equivalent:

global x
x= 0x1
? Format("0x%08X",test(x))

function test(in)
   in += 0x80000000
   return in

We get 0x80000000 this time, which can be larger than the previous ceiling 0x7FFFFFFF. However, the value is not correct.

Then we change the code again to the following:

global x
x= 0x1
? Format("0x%08X",test(x))

function test(in)
   in += 2147483648
   return in

And we get the expected 0x80000001.

One thing is that when we run ?0x80000000, and we get 2147483647. However, if we convert it in http://www.parkenet.com/apl/HexDecConverter.html, we get 2147483648. We suppose the latter is the proper result, as we see in the third code example. Is it a bug or something? Thanks

Link to comment
Share on other sites

DAQFactory uses 32 signed integers when doing the hex conversions. This means you only get 31 bits of precision with 1 bit used to determine sign. This means that the maximum it can convert is 0x7fffffff. This is talked about in the manual and in this post: (post link) Really if you try and do anything integer related (ie. bit functions) with values > 0x7fffffff you will get unexpected results.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.