format function and NaN "values"

[ekanderson]

I'm using NaN "values" to mark missing data, so that mean, max, etc. of a channel's data will return the correct values.

To get around format("%3.1f", mychannel[])'s output of 1 followed by all sorts of 0's when passed a "value" of NaN(), I wrote a function returning string values. It's abysmally slow, ~6 seconds to do 240 values.

Format() the same array, is almost instantaneous, BUT returns 100000000000000000000000000 etc.

for NaN values in the channel's data.

Are there any plans to tame format()'s output ???????????????????

Your suggested code of

iif(mychan[0] == NaN(),"NaN",Format("%.3f",mychan[0]))

I can only get to work on individual elements of a channel.

ek

[AzeoTech]

Looks like iif() evaluates only the first array element. I think you'll need to create a second channel to hold the string version of the data, and as you add data to the channel, you put the string rendition of that value in the string channel. It should be device type Test, I/O type String. The easiest way is to put it in the event of the main channel:

private string out
if (mychan[0] == NaN())
   out = "NaN"
else
   out = format("%.3f",mychan[0])
endif
out.time = mychan.time[0]
smychan.addvalue(out)

The extra code (putting it in a variable first, etc) just makes sure that the time stamp stays consistent.

[ekanderson]

I better explain the situation more clearly.

Have 10 or so real channels and about 14 virtual channels.

Data collection, logging, export, ftp'ing goes 24 hours a day.

However, when I have to shut down the data collection system for maintenance, reboot, updates, etc.. I save the *.ctl file with history. Do what ever I have to do.

Then I restart the *.ctl with DF. On a "cold" start. I putch in NaN() into almost all the channels for the data rows for the time that the *.ctl file was NOT collecting valid data. This is my "missing data". By putting the NaN values in, when the hourly and daily statistics (max, mean, min) are calculated, they should be based on only valid data elements.

The problem I'm having with the 1e300 values are when I export the data to file with the "format()" function, i.e., the expression entered box in the "Main" window of export set.

Found an interesting quirk today. The output of format("%3.1f", virtual channel[]) is usuable for my application!!!!! Puts out "1.#" or something similar depending on the # preceeding the "f".

BUT the same format() statement for a real channel puts out "10000000000000000000000" etc.

Doing your suggestion, of:

if (mychan[0] == NaN())

out = "NaN"

else

out = format("%.3f",mychan[0])

endif

out.time = mychan.time[0]

mychan.addvalue(out)

will not work for me because it will causes valid data to be rolled out of my histories prematurely.

and replacing a numeric value with a string value would probably screw up doing the means, etc.

Have no idea why there is the different result output by format() between virtual and "real" channels.

Tested it on several (not all) real and several virtual channels, with similar results.

Sweet mysteries of coding.

Thanks

ek

[AzeoTech]

I'm not getting that format() thing with the v channel output 1.#.

Anyhow, you misunderstand my suggestion. MyChan remains numeric and is what you should do your means, and other calcs on. Out is a string channel that is used only for logging. They are two different channels, one contains the numeric form, the other the formatted string form.

[ekanderson]

your statement

"I'm not getting that format() thing with the v channel output 1.#."

Does the above mean that you don't understand, or that you don't get the result?????

The v.channel[] is the SOURCE of the data (which has numeric values and can contain NaN values) for the format() statement which OUTPUTS the "1.#" values to an export file.

What are your OUTPUTs.??????? with a statement such as

? format("%3.1f", v.channel[])

with v.channel[] containing numeric values and NaN values.

I consistently get similar results with DFexp 5.80 and 5.81 on two seperate machines. It maybe "1.#J",

"1.%", etc depending on the numbers in the format("%#.#f", v.channel[]). in the EXPORT expression slot.

The reason I misunderstood your suggestion:

the "smychan.addvalue", with my eyeballs look like mychan. Sorry about that.

Anyway, with DFexpress that isn't an option for me. # of channels limitation.

[ekanderson]

To clarify:

Example of output from format() for virtual channel containing only NaN values

? format("%3.1f", v.bgr[1,5])

{"1.$", "1.$", "1.$", "1.$", "1.$"}

? format("%3.2f", v.bgr[1,5])

{"1.#J", "1.#J", "1.#J", "1.#J", "1.#J"}

? format("%.2f", v.bgr[1,5])

{"1.#J", "1.#J", "1.#J", "1.#J", "1.#J"}

output from format() for real channel containing NaN

? format("%.6f", nhum[1])

10000000000000000000000000000000000000000000000000000000000000000000000000000000

00000000000000000000000000000000000000000000000000000000000000000000000000000000

00000000000000000000000000000000000000000000000000000000000000000000000000000000

000000000000000000000000000000000000000000000000000000000000000000.000000

[AzeoTech]

I meant I don't get what you are seeing. If I put NaN() into a v. channel and then do format like you did, I still get the big number.

The extra channel thing is my only solution for you at this point unless you want to use the File. functions to write your own logging scheme. I'm afraid version limitations are not a reasonable excuse, especially since you've used that excuse before. I'm not here to help you get around version limitations, especially on a license you got for free.

We will look into the format() bug, but I can't promise when it might be fixed.

[ekanderson]

Odd about the v.channel thing. Could it be connected to any M$ routines. Both of my machines do NOT have anything M$, other than the XP which is @SP3 with latest updates (except Iexplorer).

I'm a person that likes to work with the tools on hand and I assumed that LabJack paid a royalty to you for every copy of DF express they shipped and passed that cost on to their end purchaser. If that is not the case, then I apologize for offending you by trying to save some money, which to me is a very reasonable excuse.

If you don't want to hear anything more from me, just let me know.

I didn't start this topic, or any other, with the specific intent of getting around version limitations. I'm having a hard enough time keeping track of 20+ channels.

With a modicum of contrition.

ek

[AzeoTech]

Yeah it is odd about the V.channel thing, especially since internally, the data is stored the same way in a V channel as a regular channel. Maybe your v.channel data is inadvertantly a string?

No worries on all the posts. We don't mind answering them, its just we can't spend time on items when a valid solution is already available if you just upgrade. So at this point, on this particular topic, I think we are done. We will look into the format() bug, and you have several options to get around it. Feel free to post other questions.

[AzeoTech]

OK, we've fixed the bug and will include it in release 5.82. We aren't ready to do a full release at this point, but you can get the update by going to www.azeotech.com/daqfactory.exe and copying that file over the existing in your DAQFactory directory. Note that this link may not be valid for long, especially after we do the 5.82 release.

[ekanderson]

Thank you.

ek

[ekanderson]

Downloaded the replacement file. 5+Megs.

DAQFactoryExpress.exe original 9+ Megs

Renamed original

copied over replacement

Renamed replacement to the original.

Get a license error, contact tech support.

Just a note: using 5.81 DFexp,

Today the format(), not consistent outputting v.channel, vs real channel as it was yesterday.

Ran tests this morning with slightly different data in the channels. Got mixed results. Some v. channel's results as yesterday, others as yesterdays "real" channels. Some real channels with results as yesterday's v.channels.

ek

[AzeoTech]

Sorry. I forgot you were on express. Go to www.azeotech.com/daqfactoryexpress.exe instead. You can delete the other file.

[ekanderson]

Finally with the help of DFex 5.82 build 1624 have gotten export files to behave they way I want.

If anyone else has been having similar problems the ff two routines may help you to substitute whatever you want instead of "NaN".

The "RdToDig" routine is probably not needed, but I had about 20 export files with 10+ entries that used it when I was trying to get around DF 5.81's problem with "NaN" being 1e300.

// RdToDig() Round to # of digits	  Last Mod   01 July 09
// 10 Jan 09 eka  implement
// 24 Apr 09 eka  return "ND" if Out of Range value passed  Mainly NaN
// 12 May 09 eka  Try for integers, change ND to o.r
// 19 JUN 09 EKA  Guru's code	returns only 1 nd and quits
// 29 Jun 09 eka  Try to filter NaN/s
// 30 Jun 09 eka  MUST RETURN TIME of all to get to work with Export files
// 01 Jul 09 eka  revised to work with non-array as well, rather than external call to OLD code
//				Add alert code
//	   AFTER 6 months, it is finally working as planned!!!!!!!!!!
//
//*** need to get it to work for an array
//
//  variable single or array, number of digits to return
function rdToDig(p, digs)

		 // global for debug
Global r2dig_rows =numrows(p)

	  // all params have a time???
private basetime =p.time[0]
private string ss

//********* Guru's code
//function rdToDig(p, ex)

//if (max(p > 1e20))
  // return("nd")
//endif
//return(floor((p * 10^ex)+0.5) / 10^ex)

//********* end of Guru's code

	  // not an array, just single value
if (r2dig_rows ==1)
  // ? "1"
//   ss =(RdtoDig_save(p, digs))
   if ((p >=10000000000) || (p ==NaN()))
	  ss ="nd"
	  else
	  ss=(format("%3." +doubletostr(digs) +"f", p))
   endif

   ss.time =basetime
   return(ss)
endif

   // Do NOT have a 2nd time, so ff assignment FAILS   p.time[1]

	  // assume all times are at equal interval which our's are
	  // filter_NaN function could be revised to account for other time intervals??
   private time_delta =p.time[0] -p.time[1]

switch
   case (digs ==0)

	  return(filter_NaN(format("%3i", p), basetime, time_delta))

   case (digs >=1)

	  return(filter_NaN(format("%3." +doubleToStr(digs) +"f", p), basetime, time_delta))

   default
		 // all ff added 01 Jul 09
   ? "error in RdToDig: with " +doubleToStr(digs) +" digits."

   page.page_0.BackColor = rgb(0,0,0)	 // show BLACK
   Note1 ="RdToDig Alert "
   Note2 =" Digits = " +doubletoStr(digs)
   beginexport(aHistory)			// log it

   untrapped_string[untrapped_alert_counter] ="RdToDig"
   untrapped_alert_time[untrapped_alert_counter] =sysTime()
   untrapped_alert_counter +=1
   untrapped_alert =1
			// end add 01 Jul 09
endcase

**** end RdToDig

// filter_NaN		Last Mod   29 June 09
// 28 Jun 09 eka  implement
// 29 Jun 09 eka  HAVE TO ASSIGN times, format() has NO times???
//
//  pass string array from format()
// return with "NaN" s replaced with nd

function filter_NaN(string to_filter, basetime, time_delta)
//return(filter_NaN(format("%3i", p), basetime, time_delta))
//return(to_filter)		// same export file results if this done.  i.e. null strings "".

private string filtered_out
private string pp =""

Private no_rows =numrows(to_filter)
private x
//? no_rows
//? time_delta
//? formatdatetime("%H%M", basetime)

for(x =0, x <no_rows, x++)
   pp =to_filter[x]

   if (pp =="NaN")
	  to_filter[x] ="nd"
   endif

   to_filter.time[x] =basetime	  // +(x * time_delta)
   basetime -=time_delta
endfor
//? formatdatetime("%H%M", basetime)

return(to_filter[])

[BeeHay]

Your suggested code of

iif(mychan[0] == NaN(),"NaN",Format("%.3f",mychan[0]))

I wonder if this is what I need to try out for my "NaN" values.

Anytime my counters stop counting, I have many variables that end up being "NaN" instead of their last value.

if(mychan[0] == NaN())

mychan[0] = 0

endif

Would that work?

I didn't know how to say "if NaN, equal this".

[AzeoTech]

I'm not sure, but here are a few things:

iif() only works on scalars, so you can't apply it to a whole array, i.e. iif(mychan == Nan(),...) won't return an array if mychan is an array.

Second, although your if statement is ok, you can't change the value of a channel once the value is in the channel. You can adjust it as it comes in using a conversion, but once its in the channel, its fixed. So, mychan[0] doesn't work. However, myvar[0] does work where myvar is a variable. Variables don't have this limitation.

[bartvandiepen]

Hello,

With the latest version of DAQFactory 5.82, there is a bug fixed in the format() function. Can you give more details about this bug? Could this bug be responsible for a crash or freeze the application DAQFactory?

greetings Bart

Pro Control B.V.

[AzeoTech]

No, the bug was only when formatting NaN values. In DF NaN is actually stored as 1e300, and if NaN value was passed to format() it would actually format the 1e300, so we had to add a check for NaN to just display "NaN" and not internally format it otherwise. It would not cause any problems in DAQFactory. It would only cause a big number to be displayed instead of NaN.