# Data presision

## Recommended Posts

Hi, what function do i need to use to reduce my raw input data to 1 Decimal place?

How can i ignore negitive numbers that may come in as raw data.

Sorry if these are simple questions

Andy

##### Share on other sites

DAQFactory stores everything with double precision, meaning 15-16 significant figures. For on screen display, the Precision parameter determines how many decimals are displayed. If you want to round a value to 1 decimal place for everything, then use this conversion:

floor(Value * 10 + 0.5)/10

Eliminate the + 0.5 if you don't want to round, but just truncate. Apply this conversion to the channel. You'll still get 15-16 sig figs, but most of them will be 0's. Please note that floating point values in computers cannot always store exactly 1 decimal place. They can store integers exactly, but sometimes you get 15.39999999999 instead of 15.4000000000. I won't go into the details why. DF will try and fix this, but very occasionally misses.

When you say ignore negative values, what do you then want to do with them? You have to put something there. Perhaps NaN (not a number)? I'd like to remind you of the story of the ozone hole: scientists had been measuring ozone in Antarctica for quite some time by other scientists before the hole was discovered in 1982 or 83. When it was discovered, the other scientists (the ones who had been measuring it, but didn't see a hole), went back and looked at their data and realized that, in order to save disk space, they had thrown out all data < 50ppb, which, of course, is the data that indicated the hole. So, since they were stingy on saving their data, they missed out on one of the biggest discoveries in atmospheric science. For that reason, and especially today when disk space is dirt cheap, I strongly recommend against ignoring any values during data acquisition and logging. Use your analysis tool to ignore the values instead.

##### Share on other sites

Thanks for the reply, love the story.

What is happening is, say one if my data samples coming in is say 0.01796640625 when it is actually 0. this is just noise. and simularly sometimes i get a -0.01796640625 but it all adds up to an hourly count of something but is actually still zero.

so if i can get my data in to 1DP i would be happy.

I tryed your floor(Value * 10 )/10 conversion but didnt seem to get any sense. My conversion is (Abs((Value-5.1)*3.85)-.1) how do i integrate floor(Value * 10 + 0.5)/10 into my conversion?

Am i missing something silly?

i have attached my ctl file just to show you it far far from finished. I love DAQfactory but alas have so much to learn.

Thanks for your time and patients

Andy

##### Share on other sites

Well, true story, and one I tell often, especially when companies (like NI) start releasing hardware and software that "thinks" too much for you. Plug and play is nice, but if you don't understand what the tool is doing to your data to make it plug and play you may be getting something different than you think. Noise comes from many sources, even the timing of your data, and its important to understand possible sources of it, or you may make the conclusions about the data based on something that is actually noise. My favorite example of this is political polls. They generally say they are +/- 4 points, but my guess is that is based on their sample size and therefore doesn't take into account, for example, that all the people that might vote one way are the same ones that would refuse to respond to a poll. I've seen it first hand in the sciences, where a paper was published making a conclusion based on data that was completely wrong because they did not understand their process and made too many assumptions. But alas, I rant and will save that story for another time....

OK, if you want to round the conversion you specified using the formula I specified, you just replace Value in my formula with your whole formula:

floor((Abs((Value-5.1)*3.85)-.1) * 10 + 0.5) / 10

I don't think the attachment went through.

##### Share on other sites

Thanks i will play tonight with the conversion. i will try and attach the file again.

Andy

New_Schamatic.ctl

##### Share on other sites

• 5 years later...

So you're saying all numbers are stored as IEEE 64 bit floats?

i don't recall the allocation of the 64 bits between exponent, mantissa, etc., What's the largest integer that can be stored without rounding error?

##### Share on other sites

A 64 bit floating points is usually 15-16 sig figs depending on what the first sig fig is.  Its 52 bits + 11 bits of exponent and 1 sign bit.  2 raised to the 52'nd power is 4503599627370496, so its +/- that number minus 1.