Min Max from daily records

[RJE]

Now that I have the power of DF to pull in all this data I am trying to figure out what to do with it. Here is where I'm at in the testing stages.

I poll a serial device for data every minute for Relative Humidity values. I get the data, convert it to a meaningful number and then AddValue to a channel. The data is then displayed on a graph which works well.

These channels are set up for 144 History and 3000 Persistance, my idea here is that there will always be enough past history to be displayed when the computer has to be restarted for maintenance, XP updates etc. Also there would be enough history to go back in time to trace a problem. These numbers may be a little large at this stage but you get the idea, that is if I'm on the right track.

So sitting down at the computer and opening the RH page I would see a good history of the RH of certain areas. Now I wouldn't want to keep all that data but may be take the min/max/ave points (with time stamp on min/max) for each day at midnight and put that into a channel(s) and then log that to a file.

So far when trying to get min/max values from a channel on day 16 I just kept getting into infinite loops which makes me thankful for the multiple loop terminate feature in DF.

Thanks for any help.

[AzeoTech]

OK, first, if your history/persist is less than 10,000 and you don't have lots of channels (say more than 100), there is no reason to make the history less than the persist. If they are the same, the data will persist through restart. A history of 10,000 only takes 160K of memory. Even if you had 100 channels, that'd only be 16meg, which is nothing on most modern systems. If you are on a low-memory embedded system, of course, that's another story

To calc daily min/max values its usually easiest to do it from the event for the channel in question. Lets say its called RH:

1) go to the event tab of the channel view for the RH tab

2) enter code like this:

if (formatdatetime("%d",rh.time[0]) != formatdatetime("%d",rh.time[1]))
   // its a new day, take the min/max of previous day:
   private calced = min(rh[rh.time[0]-0.001,rh.time[0] - 86400])
   calced.time = rh.time[0]
   rhmin.addvalue(calced)
   // repeat for max, avg, etc.
endif

That's about it. Notice how I subset rh based on the time of the most recent data point and 86400 seconds before that. This gives us a full day's readings. The if() is just looking at the day of the month and looking for a change.

[RJE]

Hi #63

That cleared up the history/persist ?'s and I'm working on things which seem to be going well.

DF seems to be based heavily on its scripting engine so a sript command manual, detailing all the functions in one spot would be real handy to have. I find myself scrolling up and down through the manual looking for script functions. Just a thought.

A while back you mentioned to keep an eye out for something in February regarding the NI USB-6008 DAQ module. Any revelations yet?

Problem is, all my existing temp, RH, light etc sensors are 0-5 volt output. Using LabJack U3's would require conditioning that signal to feed the U3's input of 0 - 2.44 volt where as using the NI USB-6008, I would only have to change the PGA gain to 2 for the 0 - 10 volt AD input.

Thanks

[AzeoTech]

I don't think I was talking about the NI for february, but rather a new product for doing web based data viewing.

We have an NI driver that will read volts and I can make that available to you, just email us. Its not particularly fast, largely because NI made their NIDAQmx pretty anti- non-LabView. The driver does too much and so makes it difficult for programs like DAQFactory that already do everything to use it.

Yes, a reference manual would be nice, and will likely come with R6 that I keep promising.

[RJE]

Ok, since I already bought a NI USB-6008, I was hoping. Well, LabJack U3 units it will be.

For data every second would you still keep history & persist the same if I want the min/max values for each day?

The machine I'm using now has 2G of ram and I can see me needing an unlimited I/O package as my system grows.

With that in mind, I'm thinking of building a new machine and this time I will go with current hardware.

Thanks

[AzeoTech]

If you are going to do the min/max the way I described using subsetting by time to select what gets analyzed, you can make the persist larger than the history. In order to access data past the history in the persist, you have to be subsetting. This is to prevent you accidently bringing the entire persist into memory. However, 1 second data is only 86400 data points a day, which is only about 1.4 meg of memory, so you could make the history that big provided you don't have 1000 channels.

[RJE]

I've just started and I already have 6 channels with data coming in every second.

So my thinking is this, gather a days worth of data and graph it. This will give me a detailed picture of the day. When midnight comes, pull out the min, max and ave values of that days data and put to another 6 channels which would be logged. Then clear history and start over with the next days worth of data.

So if I follow history & persist right would I not set persist to about 90,000 with the history down low to keep from using up ram.

Be patient little one as DF experience will come

[AzeoTech]

6 channels at 90,000 history is only 8 megabytes. I can assure that is the same as nothing on any modern desktop. The video buffering in DAQFactory uses a lot more than that. Plus, you are likely accessing the persist constantly and pulling it into memory, so there is no advantage. Long persists, with short histories are really only appropriate when you will only occasionally access the data past the history. If you are constantly graphing it, you are bringing it into memory, doing the same thing as history (but slower).

[RJE]

I think I understand now. Use 90,000 history to gather a days worth of data and if I don't care if I loose it if I have to stop DF then persist can be 0. If I want to have the captured data after a restart then persist should be set to 90,000.

Am I still correct about clear history after the daily min/max is done?

Thanks

[AzeoTech]

You don't have to clear the history. It will get pushed out as new data arrives, and if you subset inside the min/max as I showed, the old data won't be picked up. The only reason to clear the history would be if you want your graphs to clear for the new day.