Issue with data logging on a completed test. Will export work?

[robopp]

We recently ran a test for the last 7 days and had issues logging data for one of the tests. DAQFactory is still active and has all the history data. I tried using export to save the data to a file by removing the indexes from each channel (channel_1[0] -> channel_1). When I run the export function, DAQFactory 'freezes' for a couple of hours appearing to be doing something, but no file is actually created. Is it possible to use the export functionality in DAQFactory to save our data to non-volatile storage? 

[AzeoTech]

How much historical data do you have?  Does it do it if you just export one channel at a time?

[cjudson]

I have a similar issue on one, but only one, of running DAQFactory systems. I've verified the setup is the same as our other systems which don't exhibit this issue.

Export randomly stops working and only a restart of the program will get it going again. I believe the sequence that triggers it is working as we get a new row in the file at the scheduled time but when the glitch occurs the rows have no data. 

Fortunately, we also use the log data capability and that's our backup. I prefer Export as Modbus data can be saved directly with the correct decimal value. The Log function has been bulletproof.

Attached is a small file showing the missing data and the result of a restart and a snip of the Export Set (the tank level value is a constant and does not originate as a Modbus value).

MVR-Export.csv

[robopp]

1,000,000 points. I ended up writing a sequence using the File routines to export all data. It was actually a lot easier than I thought it would be. Great job designing your scripting interface and API set! Moving the question to the sequence - I wasn't able to figure out how to programatically list all of the channels and their names. Is this possible?

[AzeoTech]

Yeah, bottom line is that Export and Logging sets are designed for smaller systems with limited flexibility, much like many of the built in features.  But when you need more flexibility you can always drop down into script for that part.

Channel.ListAll() gives you an array of strings containing the channel names.  You can optionally pass in a string with the name of a channel group and get only the channels from that group.

[robopp]

Is there a way to programatically read channel data without knowing the name of the channel? I'd like to do the following, but instead of hard typing the channel name, use the Channel.ListAll to get the channel names and iterate over that list to get each channel's data.

for(i=Left_Pump_Speed.GetHistoryCount()-1, i > 0 , i--)
   Private.strLine = FormatDateTime("%c", Left_Pump_Speed.Time[i]) + ","
   Private.strLine += DoubleToStr(Left_Pump_Speed[i]) + ","

   File.Write(Private.handle, strLine);
   delay(.1)
endfor

 

[AzeoTech]

Sure, use ListAll(), then use evaluate():

private string chans = channel.listAll()
for (private x = 0, x < numrows(chans), x++)
   for(i=evaluate(chans[x] + ".GetHistoryCount()-1"), i > 0 , i--)
      Private.strLine = FormatDateTime("%c", evaluate(chans[x] + ".Time")) + ","
      Private.strLine += DoubleToStr(evaluate(chans[x] + "")) + ","

      File.Write(Private.handle, strLine);
      delay(.1)
   endfor
endfor

Notes:

1) you can do ListAll() and just specify a group of channels by providing the group name as a string
2) the above code will be VERY slow, especially with the delay(0.1) in there
3) I recommend using numrows() instead of GetHistoryCount()

 

 

 

[robopp]

Never mind. I just discovered the 'Evaluate' method. Very useful!

[AzeoTech]

Yup.  Check out execute() too...

[robopp]

I'm seeing some really weird behavior on the export. We have a lot of channels (150+) and the data is stored in persistent files. If I export a single row, but all of the channels at once the channel data is getting corrupted around 75 channels in. If I look at the data in DAQFactory, it looks fine. If I remove the first 70 channels from the export, the channel data is fine, but gets corrupted again 30 channels in. When I say corrupted I mean that the channel data is incorrect and implausible for that channel. Any thoughts of what might be going on? Anything I can do to debug this? Here's my export sequence:

Private.handle = File.Open("C:\Users\Rob\Desktop\export_test\m62_export1.txt", 0, 1, 0, 1)

Private.i = 0
Private.j = 0
Private string strLine = ""

// Get the channel names and write the CSV header
Private string channelNames = Channel.ListAll()
Private numChannelNames = NumRows(channelNames)

strLine = "SystemTime"

for(i = 0, i < numChannelNames, i++)
   strLine += "," + channelNames[i]
endfor

File.Write(Private.handle, strLine);

Private numHistoryPoints = Evaluate(channelNames[0] +".GetHistoryCount()")

for(i = numHistoryPoints-1, i >= 0, i--)
   strLine = FormatDateTime("%c", Evaluate(channelNames[0] +".Time[i]"))
   for(j = 0, j < numChannelNames, j++)
      strLine += "," + DoubleToStr(Evaluate(channelNames[j] + "[i]"))
   endfor
   File.Write(Private.handle, strLine)
   delay(.1)
endfor

File.Close(Private.handle)

 

[AzeoTech]

The function you wrote would only work if all the channels had the exact same number of historical data points, and would only work if no data came in during the time the for() loop runs.

If I wanted to successfully align data and log it, but only after the fact, I would generate each line in-situ, then write it at the end.  I'd do that by creating a Test String channel (with Timing 0 of course), then a sequence that is always running like this:

Private string channelNames = Channel.ListAll()

private string strLine
while(1)
   delay(1)
   try
      strLine = FormatDateTime("%c", systime())
      for(j = 0, j < numrows(ChannelNames), j++)
         strLine += "," + DoubleToStr(Evaluate(channelNames[j] + "[0]"))
      endfor
      myTestStringChannel.addValue(strLine)
   catch()
   endcatch
endwhile

Then when I wanted to write this to disk, I'd write the header like you did, then write the enter strLine history, or whatever part of it I wanted.

 

[robopp]

Thank you for your prompt response!

I'm doing the export because we forgot to start logging and I'd like to recover our test data into a log file.

No data is coming in while I'm running the sequence. This is all being done post test. What I don't understand is how some of the channel data is different for the same channel and the same historical data index. For example, we have a channel called 'Rail_12V'. If I generate a new CSV string using all of the 150 channels. Rail_12V = 0.612 (not correct). However, if run the same sequence, but move the starting index from 0 to 70. 'Rail_12V' now equals 11.956 (correct). If I access Rail_12V using the same historical index it equals 11.956. Literally, everything is the same except the starting channel index. I should not that it's not just Rail_12V that's getting corrupted, but the corrupted channels are always deep into the channel index.

[AzeoTech]

Are you sure some of those channels aren't empty?  Is it off for the entire column or just a few rows?

[robopp]

Some of the channels are empty and it will be off for the entire column.

[AzeoTech]

Empty channels are going to throw your logic.  I'm not sure that you'll get a "," even if the channel is empty.  I'd use isempty() to check.

[robopp]

That was the issue. Two questions:

1 - How do I have empty channels? I know for a fact that these channels had data in them during our test.

2 - What's actually happening in my sequence when empty data is encountered?

Thanks!

[AzeoTech]

1. Well, the channel that has the wrong data isn't the one that is empty.  It is some other channel before it.  Are you sure ALL your channels have data?  They all have to have the same amount of data too.  

2. I would expect it to error out of the loop, but it kind of seems like it is just skipping the line that adds that value and not even putting a comma in.

Do you see fewer columns than expected?