Parsing

jridings · May 7, 2008

I need to create a DAQFactory application from a com port log file. I logged data coming out of a serial port to a text file. Now a need to use this file and do the parsing of data to setup my channels. How can I do this?

AzeoTech · May 7, 2008

I'm guessing that the com port log file is coming from a different application? How to do it depends on the format of the data, especially the frame delimiter. If each frame ends in a carriage return / line feed, its real easy. If not, its only slightly harder but different. Perhaps you can post a sample so I can point you in the right direction?

jridings · May 8, 2008

Here You are.

				BW11  Basis Weight	STANDARDIZE HISTORY				
	ALARM	   AIZ VOLTS	 AIS VOLTS	  SIGMAZ %	  SIGMAB %	  
				DIRT G/M2	TCLEAN DAYS	AVG TEMP C	SRC TEMP C	 
				DET TEMP C   SRC PRG/DG V  DET PRG/DG V				  

2008-04-07 20:42:03.													 
	   1.000		-0.000		 2.606		 0.000		 0.060	 
					-0.540		 4.000		59.980		59.840	 
					59.870		 1.651		 3.048		 0.000	 

2008-04-07 21:42:27.													 
	   1.000		-0.000		 2.603		 0.000		 0.060	 
					-0.490		 4.000		59.980		60.050	 
					59.990		 1.645		 3.063		 0.000	 

2008-04-07 22:41:55.													 
	   1.000		-0.000		 2.601		 0.000		 0.050	 
					-0.460		 4.000		59.980		59.940	 
					59.960		 1.644		 3.063		 0.000	 

2008-04-07 23:42:31.													 
	   1.000		-0.000		 2.601		 0.000		 0.050	 
					-0.450		 4.000		59.980		59.800	 
					60.010		 1.644		 3.070		 0.000	 

2008-04-08 00:42:03.													 
	   1.000		-0.000		 2.602		 0.000		 0.050	 
					-0.460		 4.000		59.980		59.970	 
					60.010		 1.659		 3.056		 0.000	 

2008-04-08 01:42:27.													 
	   1.000		-0.000		 2.601		 0.000		 0.040	 
					-0.450		 4.000		59.980		59.880	 
					59.960		 1.654		 3.051		 0.000	 

2008-04-08 03:42:39.													 
	   1.000		-0.000		 2.602		 0.000		 0.040	 
					-0.470		 4.000		59.980		59.740	 
					59.940		 1.635		 3.046		 0.000	 

2008-04-08 04:42:07.													 
	   1.000		-0.000		 2.602		 0.000		 0.050	 
					-0.470		 4.000		59.980		60.020	 
					59.990		 1.646		 3.050		 0.000	 

2008-04-08 05:42:31.													 
	   1.000		-0.000		 2.601		 0.000		 0.060	 
					-0.450		 4.000		59.980		59.940	 
					60.080		 1.644		 3.037		 0.000	 

2008-04-08 06:42:59.													 
	   1.000		-0.000		 2.600		 0.000		 0.050	 
					-0.420		 4.000		59.980		59.770	 
					59.990		 1.653		 3.053		 0.000	 

2008-04-08 07:42:35.													 
	   1.000		-0.000		 2.598		 0.000		 0.050	 
					-0.390		 4.000		59.980		60.020	 
					59.990		 1.658		 3.056		 0.000	 

2008-04-08 08:42:59.													 
	   1.000		-0.000		 2.598		 0.000		 0.040	 
					-0.420		 5.000		59.980		59.960	 
					59.990		 1.657		 3.058		 0.000	 

2008-04-08 09:38:11.													 
	   1.000		-0.000		 2.598		 0.000		 0.050	 
					-0.420		 5.000		59.980		59.890	 
					60.080		 1.655		 3.051		 0.000	 

2008-04-08 10:38:43.													 
	   1.000		-0.000		 2.596		 0.000		 0.060	 
					-0.380		 5.000		59.980		59.990	 
					59.990		 1.664		 3.072		 0.000	 

2008-04-08 11:05:39.													 
	   1.000		-0.000		 2.594		 0.000		 0.050	 
					-0.340		 5.000		59.980		59.940	 
					59.980		 1.637		 3.044		 0.000	 

2008-04-08 11:38:23.													 
	   1.000		-0.000		 2.593		 0.000		 0.060	 
					-0.330		 5.000		59.980		59.990	 
					59.990		 1.633		 3.038		 0.000	 

2008-04-08 12:14:39.													 
	   1.000		-0.000		 2.593		 0.000		 0.050	 
					-0.330		 5.000		59.980		60.060	 
					60.000		 1.659		 3.056		 0.000	 

2008-04-08 12:19:39.													 
	   1.000		 0.000		 2.592		 0.000		 0.050	 
					-0.300		 5.000		59.980		59.860	 
					60.050		 1.657		 3.053		 0.000	 

2008-04-08 13:20:03.													 
	   1.000		-0.000		 2.595		 0.000		 0.050	 
					-0.370		 5.000		59.980		59.920	 
					59.990		 1.658		 3.059		 0.000	 

2008-04-08 14:19:31.													 
	   1.000		-0.000		 2.595		 0.000		 0.050	 
					-0.370		 5.000		59.980		59.990	 
					59.870		 1.648		 3.043		 0.000	 


END OF REPORT

Logfile_1.txt

AzeoTech · May 9, 2008

OK, this is what is called a fixed-field file. This means that each field / column is a fixed number of characters and spaces are added to the beginning to ensure this. Fixed field is actually one of the easiest to parse. You really only need to use the mid() function. In this case, we the first column is 12 characters, and the rest are 14. But first you have to read the file in. For that you want to use the File. functions. Here's the script to read the file and put the data in the appropriate channels with the specified date in the file. I only did a couple of the channels, but you should be able to figure the rest out. Its just rinse and repeat with the AddValue() calls with different offsets in the mid() function (the second parameter):

private handle = file.Open("c:\logfile-1.txt",1,0,0,1)
private string linein
private thetime
while (1)
   // read a single line looking for 20
   linein = file.Read(handle)
   if (left(linein,3) == "END") // found END OF REPORT, we're done
	  break
   endif
   if (left(linein,2) != "20")  // not 20, so move to next line
	  continue
   endif
   // we now are on the 20xx-xx-xx line, figure out the time:
   thetime = strtotime(linein,"ymdhms")
   // now read next three lines and parse each:
   linein = file.Read(handle)
   AlarmCh.AddValue(inserttime(strtodouble(mid(linein,0,12)),thetime,0))
   AIZVolts.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
   AISVolts.AddValue(inserttime(strtodouble(mid(linein,26,14)),thetime,0))
   // etc. and second line:
   linein = file.Read(handle)
   Dirt.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
   // etc. and third line
   linein = file.Read(handle)
   DetTemp.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
endwhile
file.Close(handle)

jridings · May 9, 2008

Thanks you works great. Now this data is sent out of a serial port. (com 1) How do I create a protocol to capture data. I used HyperTerminal to simulate the com port and sent the text file from com 2 to com 1. I can see the data in the monitor window, but I do not know how to connect the data to a channel. At the end of each line there is a /13. Also, I need to use the skipchar function to ignore /160/114 or use hidehidden.

AzeoTech · May 9, 2008

There is no skipchar function anymore. Where is the /160/114? Can you post a capture of the monitor?

jridings · May 11, 2008

Here you are. I did not see the /160/114 so just help me connect the data to a channel. I do have a /13 at the end of each line

AzeoTech · May 12, 2008

The capture wasn't posted. But I'll assume it looks just like the text file. I'm also going to assume you created a serial device called Basis with a NULL protocol. You'll see the script look very similar to the one reading the file. There is no Open() or Close(), and the Read()s are replaced with ReadUntil() from the device:

private string linein
private thetime
while (1)
   try
	  // read a single line looking for 20
	  linein = device.basis.readuntil(13)
	  if (left(linein,3) == "END") // found END OF REPORT, we're done
		 break
	  endif
	  if (left(linein,2) != "20")  // not 20, so move to next line
		 continue
	  endif
	  // we now are on the 20xx-xx-xx line, figure out the time:
	  thetime = strtotime(linein,"ymdhms")
	  // now read next three lines and parse each:
	  linein = device.basis.readuntil(13)
	  AlarmCh.AddValue(inserttime(strtodouble(mid(linein,0,12)),thetime,0))
	  AIZVolts.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
	  AISVolts.AddValue(inserttime(strtodouble(mid(linein,26,14)),thetime,0))
	  // etc. and second line:
	  linein = device.basis.readuntil(13)
	  Dirt.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
	  // etc. and third line
	  linein = device.basis.readuntil(13)
	  DetTemp.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
   catch() // catch timeout errors
	  ? strLastError
	  delay(0.1)
   endcatch
endwhile

jridings · August 21, 2009

Ok I have it working, but how do I trigger from other text string. You stop at 20 then parsed the next 3 lines. How do I trigger from for example "History"(text from my logfile_1) then drop down 7 line pick a value and add it to a channel. I do not understend the code.

In this line:

AlarmCh.AddValue(inserttime(strtodouble(mid(linein,0,12)),thetime,0))

What does mid do?

What does the first 0 do?

What does the 12 do?

What does the second 0 do?

AzeoTech · August 21, 2009

I recommend reading the Expressions -> Expression Reference -> String functions section of the help as it will cover all the various functions you can use to parse a string. The Mid() function pulls out a section of a string. The first 0 is the first character to get (always numbered from 0, so 0 is the first character of the string), the 12 is the number of characters to get. Technically, this particular Mid() could be replaced with Left(linein,12), but I often just use Mid() since sometimes I need to change where what section I want.

The 0 at the end doesn't do anything in this case. InsertTime() allows you to insert time into an array and that last number determines the spacing, but in this case, linein is not an array, so only one value exists and there is no spacing.

Is "History" right before a carriage return? I can't tell from your post. If so, you can read a line (readuntil(13)) and look for right(linein, 7) == "HISTORY". Dropping down 7 lines is just a matter of doing readuntil(13) seven times. You'll need to figure out if your line has just carriage return, line feed or both at the end. The DF serial monitor will make this obvious (13 is CR, 10 is LF)

jridings · August 27, 2009

Hey Thanks again for your Help

My project is moving along. Now, My serial port does not require polling it is out putting data all the time. These reports are imbedded in this data. Your device call "basis" works great, but when is sees "END" the sequence stops running, missing the next report. How do I trigger the "basis" sequence to start the next time a report runs. I have 6 gauge out putting data on my serial port. One gauge is called basis weight. I have add 5 more sequences one for each gauge. I need to monitor the port and trigger when a report is sent out the port. That's again.

AzeoTech · August 27, 2009

What's your whole sequence look like now?

jridings · September 28, 2009

private string linein
private thetime
while (1)
   try
	  // read a single line looking for Frame 1
	  linein = device.ABB.readuntil(13)
	  if (left(linein,3) == "END") // found END OF REPORT, we're done
		 break
	  endif
	  if (left(linein,6) != "Frame 1")  // not 20, so move to next line
		 continue
	  endif
	  // we now are on the Frame 1, figure out the time:
	  thetime = strtotime(linein,"ymdhms")
	  // now read next three lines and parse each:
	  linein = device.ABB.readuntil(13)
	  AlarmCh.AddValue(inserttime(strtodouble(mid(linein,0,12)),thetime,0))
	  AIZVolts.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
	  AISVolts.AddValue(inserttime(strtodouble(mid(linein,26,14)),thetime,0))
	  // etc. and second line:
	  linein = device.ABB.readuntil(13)
	  Dirt.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
	  // etc. and third line
	  linein = device.ABB.readuntil(13)
	  DetTemp.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
   catch() // catch timeout errors
	  ? strLastError
	  delay(0.1)
   endcatch
endwhile

AzeoTech · September 29, 2009

OK, well you need to have an outer loop that loops for every report and looks for something to indicate the start of a new report. I'll assume BW11 is a good marker:

private string linein
private thetime
while (1)
   try
	  // read a single line looking for BW11
	  linein = device.ABB.readuntil(13)
	  linein = ltrim(linein)
	  if (left(linein,4) == "BW11")
		 while (1)
			try
			   // read a single line looking for Frame 1
			   linein = device.ABB.readuntil(13)
			   if (left(linein,3) == "END") // found END OF REPORT, we're done
				  break
			   endif
			   if (left(linein,6) != "Frame 1")  // not 20, so move to next line
				  continue
			   endif
			   // we now are on the Frame 1, figure out the time:
			   thetime = strtotime(linein,"ymdhms")
			   // now read next three lines and parse each:
			   linein = device.ABB.readuntil(13)
			   AlarmCh.AddValue(inserttime(strtodouble(mid(linein,0,12)),thetime,0))
			   AIZVolts.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
			   AISVolts.AddValue(inserttime(strtodouble(mid(linein,26,14)),thetime,0))
			   // etc. and second line:
			   linein = device.ABB.readuntil(13)
			   Dirt.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
			   // etc. and third line
			   linein = device.ABB.readuntil(13)
			   DetTemp.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
			catch() // catch timeout errors
			   ? strLastError
			   delay(0.1)
			endcatch
		 endwhile
	  endif
   catch() // catch timeout errors
	  ? strLastError // delete this one line once everything works reliably
	  delay(0.1)
   endcatch
endwhile

jridings · September 29, 2009

Thanks for your help. Here is where I'm at. How do I trigger this sequence? In my code I have a statement that stops the seqence from running. If I comment the code out, then I get nan,boggest values ect..., I get something on each line. So I need to read to port all the time then trigger my seqence then the system output at report.

device.basis.Purge()
private string linein
while (1)
   try
	  // read a single line looking for Frame 1
	  linein = device.basis.readuntil(13)
	  if (left(linein,6) == "Frame 1")  // not Frame 1, so move to next line
		 continue
	  endif
	  // we now are on the Frame 1 line:
	  // now read next seven lines and parse each:
	  linein = device.basis.readuntil(13)
	  linein = device.basis.readuntil(13)
	  linein = device.basis.readuntil(13)
	  linein = device.basis.readuntil(13)
	  linein = device.basis.readuntil(13)
	  linein = device.basis.readuntil(13)
	  linein = device.basis.readuntil(13)
	  linein = device.basis.readuntil(13)
	  AlarmCh.AddValue(strtodouble(mid(linein,0,12)))
	  AIZVolts.AddValue(strtodouble(mid(linein,12,14)))
	  AISVolts.AddValue(strtodouble(mid(linein,26,14)))
	  // etc. and second line:
	  linein = device.basis.readuntil(13)
	  Dirt.AddValue(strtodouble(mid(linein,12,14)))
	  // etc. and third line
	  linein = device.basis.readuntil(13)
	  DetTemp.AddValue(strtodouble(mid(linein,12,14)))
	  if (left(linein,3) != "END") // found END OF REPORT, we're done
		 break
	  endif
   catch() // catch timeout errors
	  ? strLastError
	  delay(0.1)
   endcatch
endwhile

AzeoTech · September 30, 2009

You didn't incorporate what I just posted. There needs to be two while() loops. You need that to have the sequence continually loop looking for the BW11 to trigger that a new report was received.

jridings · October 1, 2009

Thanks guys my sequence is working great. You guys are the best:)

private string linein
private thetime
device.abb.Purge()
while (1)
   try
	  // read a single line looking for Frame 1	  
	  linein = device.ABB.readuntil(13)
	  input.AddValue(linein)
	  if (find(linein,"FRAME 1",0) != -1)  // Frame 1, 
		 // we now are on the Frame 1, figure out the time:
		 thetime = systime() //strtotime(linein,"ymdhms")
		 // now read and parse each line:Basis Weight Gauge
		 linein = device.ABB.readuntil(13)
		 Grade.AddValue(mid(linein,0,24))
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 AlarmChBW.AddValue(inserttime(strtodouble(mid(linein,0,12)),thetime,0))
		 AIZVOLTSBW.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
		 AISVolts.AddValue(inserttime(strtodouble(mid(linein,26,14)),thetime,0))
		 SIGMAZBW.AddValue(inserttime(strtodouble(mid(linein,42,11)),thetime,0))
		 SIGMAB.AddValue(inserttime(strtodouble(mid(linein,56,11)),thetime,0))
		 // etc. and next line:
		 linein = device.ABB.readuntil(13)
		 Dirt.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
		 TCLEANDAYS.AddValue(inserttime(strtodouble(mid(linein,26,14)),thetime,0))
		 AVGTEMPC.AddValue(inserttime(strtodouble(mid(linein,42,14)),thetime,0))
		 SRCTEMPC.AddValue(inserttime(strtodouble(mid(linein,56,11)),thetime,0))
		 // etc. and next line
		 linein = device.ABB.readuntil(13)
		 DetTempC.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
		 SRCPRGDGV.AddValue(inserttime(strtodouble(mid(linein,26,14)),thetime,0))
		 DETPRGDGV.AddValue(inserttime(strtodouble(mid(linein,42,14)),thetime,0))
		 // now read and parse each line:Ash Gauge
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 AlarmChAsh.AddValue(inserttime(strtodouble(mid(linein,0,12)),thetime,0))
		 AIZVOLTASH.AddValue(inserttime(strtodouble(mid(linein,12,14)),thetime,0))
		 AISINVOLTS.AddValue(inserttime(strtodouble(mid(linein,26,14)),thetime,0))
		 AISOUTVOLTS.AddValue(inserttime(strtodouble(mid(linein,42,11)),thetime,0))
		 SIGMAZASH.AddValue(inserttime(strtodouble(mid(linein,56,11)),thetime,0))
	  endif
		 // Read next Report
	  if (find(linein,"REEL REPORT",0) != -1)  //Reel Report 
		 // we now are on the Reel Report line, figure out the time:
		 thetime = systime() //strtotime(linein,"ymdhms")
		 // now read and parse line:
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 linein = device.ABB.readuntil(13)
		 PRODUCTIONTONS.AddValue(inserttime(strtodouble(mid(linein,25,12)),thetime,0))
	  endif
   catch() // catch timeout errors
	  delay(0.1)
	  device.abb.Purge()
   endcatch   
endwhile

Parsing

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Archived