Most Efficient Method For Extracting Max, Min From Log File

[SteveMyres]

I have a delimited log file that I can read with File.ReadDelim(). I'm interested in the max/min values for some specific fields (same position in every row across all rows).

Will it be more efficient to crawl the file one line at a time and keep track of the max and min, or extract to a 2D array, and get the max/min of the relevant column(s) in a single call?

Not sure how big the file may get. Will the best method be the same with say, tens of thousands of records in the file? I can set the sequence to low priority and do this task in the background.

[AzeoTech]

Definitely faster to read in big blocks. This is true in DAQFactory and in C++ or any other language. If you have the memory, read the entire file, then use the DAQFactory min/max function to quickly find what you want (possibly in combination with insertTime() and getTime() so you can get the index of the max/min. If you don't have the memory (i.e. the file is gigabytes), its still faster if you split the file in big chunks. Trying to read and process even 10,000 records one at a time will be brutally slow. Doing it with ReadDelim() and the whole file will probably take less than a second.

[SteveMyres]

It's not a DF log file, so there's no time associated, although I can put it in if I need it.

How would I go about splitting the file while using File.ReadDelim(), since File.ReadDelim() leaves the file access pointer in an unpredictable location (likely not at the end of the last requested line)? Would I have to pre-split the file somehow, maybe even creating separate files, and then apply File.ReadDelim() to each?

[AzeoTech]

Well, first, do you really need to split the file? You could probably bring in a good 500 meg data file on most modern systems, and you really shouldn't be making log files that big anyway.

[SteveMyres]

That should be sufficient, I think. Still not really sure how big the file could get, but for now I can proceed on the assumption that I'll be able to read them in one pass.

Thanks!