Software Screen Freeze After About One Month

[raining010]

Hi, Administrator

 

I wrote some daqfactory scada program for site desktop computer. After leaving site, I got the report from customer that, after about one month or a period time, the running software became freeze, they needed to restart computer to solve this.

 

I suspect this is because of the setting of history and persist, but I'm not sure, as I think the setting is not too large. I will send my program to your mailbox, the topic is same as here, thanks

 

James

[raining010]

Hi, could anybody who knows help me please? I think the solution is at persist and history, but what's the exact value?

[raining010]

The customer requests, while using trend to observe each data, it needs to keep at least 1 days data. So I define history as 3600, and persist as 100000, is this right? The quantity of these channels are about 500

[AzeoTech]

At 5 second Timing, you only 17280 data points to hold a day.  I would just set the history and persist both to 17280.  That will make it so you don't lose history if you restart DAQFactory.  100000 gives you more than 5 days.  Either way, that is not the hanging problem.  Even at 17280 for 500 channels, you are only talking 138 meg of memory for the histories.  Unless you are bring in all 500 channels on a single screen for 100000 data points you aren't going to use that much memory.  Are you using 5.90?  Do you have any remote connections?

[raining010]

Hi, Mr Guru

 

Glad to see you. I'm using 5.87a software version, instead of 5.90. The program communicated with two remote devices via one serial port board installed inside the site desktop computer. This serial port board has two serial ports. I sent the program to your mailbox last week, the title is same with this topic.

 

On site, the customer should use the overview screen usually. On this screen, the name of each variable I used was xxxx[0], the latest data and also the only one data. Although variables of this page are more than other screens, I don't think they are too much to cause this problem.

 

Thanks

James

[raining010]

Hi, Attachments are two photos from site. The program used about 1 G ram, the computer available ram is 4G, operating system is windows7

[AzeoTech]

I don't see anything that really jumps out that is wrong in your script, except your J_File_Read function which you do not appear to call from anywhere.   You'll want to chase down the memory leak.  To do so, start DAQFactory, load your doc, and get it to a stable state, looking at the EP, then look at Task Manager and see how much memory it uses.  Write this down, then come back every hour for a few hours and see if it grows.  It will jump around on its own, but if you are leaking it will have a definite upward trend.  If you see this, then try disabling parts of the program, meaning stopping sequences. Do it one at a time until the memory stabilizes.  If stopping all the sequences doesn't do it, try switching to a different page.

 

Do you have a lot of alarms that fire and reset?

 

Also, consider using 5.87b instead of a.  I don't think there are any major changes that would fix this, but you never know.

[raining010]

Hi, Mr Guru

 

Thanks for your reply, I did some tests from last weekend using my own computer. I installed two physical serial ports to the computer, so daqfactory can open and use these communicating ports successfully. The only problem is there is no response on these ports, because they are not linked to any external modbus devices.

 

I did 3 continuous tests, as below

 

First test, show 'EP' page

68.016k at first,

68.932k 1 hour later,

17.732k 4 hours later,

23.712k 1 hour later,

18.320k 4 hours later,

19.968k 2 hours later

 

Second test, show 'EP' page(restarted computer and program after the previous test)

69.732k at first,

69.980k 2 hours later,

70.012k 7 hours later,

69.956k 13 hours later

at last, it turned to 18.***k immediately after my operation(I only minimized the running program, then maximized program), and then stayed at that level(below 20k)

 

Third test, test different pages(restarted computer and program after the previous test)

71.484k at EP page

71.604k at Trend page

71.360k at Login page

71.332k at Control page

71.368k at VI plot page

71.376k at Field page

 

Sequence/Function

About J_File_Read, it's not called and not auto-started, so should not reason

Most my sequences/functions are called by event(customer clicks some special buttons), this program is showing the site equipments status to customer, customer should not operate or touch it, I think these sequences are not the reason. There is only one auto-starting sequence called J_Alarm_Check.

 

Alarms

I don't know how many alarm are fired at site, let's make an assumption half alarms are fired, will they cause program problem?

 

Software version

At site, we used 5.90 at first, but it got freeze soon. Last October we discussed this with you, then we use 5.87a to solve it. I haven't tested 5.87b.

 

I don't understand why the program memory usage sometimes dropped down from 70k to 20k in my computer. I'm not sure if lost of external devices/communication response affects the testing.

So far, in my opinion the only possible points are in channel history/persistence, or logging. Anyway I haven't found the reason.

Thanks looking for your reply

[AzeoTech]

Memory will jump around a lot, so the 70->20meg jump is not an issue.  I'm more concerned with steady growth up approaching 1000 meg.

 

Has it done this before, or is this a one time failure?

[raining010]

Hi, Mr Guru

 

The site has two separate systems, each system has one desktop coumputer running daqfactory program.

Both computers have this issue, happens about one month, recovers after restarting computers. So it's not one time failure  

[raining010]

Hi, Mr Guru

 

I checked sequences in my program again.

Usually there is only one running sequence called J_Alarm_Check, it should be ok

And I wrote a function on EP page for customer, it starts a sequence adding value every minute, it's called J_Mainpower. I global defined a variable, and compared it with 9999999. But I think this should not be the reason.

 

I'm running the program for two days, the memory cost increased very slowly. But why it's increasing?

 

Thanks

James

[AzeoTech]

Its hard to say.  Nothing certainly jumps out.  You'll likely need to repeat the tests you did before, watching the memory for a longer period if that is actually it.  The serial ports, they are on a PCI card?  Did you have to install any drivers for it, or is it using the Windows drivers?

 

This is quite unusual for sure.  Does the system need to run 24x7?  I don't particular like it, but the workaround would be to simply quit DAQFactory and restart every so often.  You can easily automate this from script.

[raining010]

Hi, Mr Guru

 

Thanks for your reply.

The serial ports on site is RS485 serial PCIe card, I had installed driver using CD from card factory.

The system is factory SCADA, so needs 24x7. It's not practical for me to simulate it in the office, as it's urgent I need to update program asap to customer there is no 30 days for me to test; and once the running program screen is minimized by mistake and recovered again its memory cost drops down to 20k, the previous cost time becomes useless.

 

As there is no obvious reason causing this issue, restarting program from script is one way. Usually this is transparent to customer, but once customer has started power accumulation(adding value per minute) in EP screen, program restarted would certainly clear the value. This brings incovenience.

 

I plan to revise channels history from 3600 to 60, remain channels persist at 100000, and update site software from 5.87a to 5.87b. Let customer test it at site, to see if it happens again in one month.

[AzeoTech]

I don't usually like to blame other software, but over the years I have found that non-windows comm port drivers tend to be pretty unreliable.  This is especially true with USB to serial converters, but it wouldn't surprise me to have issues with an RS485 card.  If you had a PCI 232 card, it would likely just use the drivers provided by windows and all would be fine.  But since you had to install drivers to use the card, it means that DAQFactory is subject to any bugs in that comm port driver.  My guess, since your application is quite simple, and you are running 5.87 which is quite stable and has been in public use for several years now by many people, is that the comm port driver is leaking memory.  It may be because of the way DAQFactory is calling it including the fact that it calls the comm port from multiple threads, but it still remains a driver issue as we really don't do anything that weird.  It may be that the manufacturer simply never tested their stuff for a month straight.

 

Anyhow, my recommendation is simply this: buy RS485 ports from another source.  The best solution would be to use an Ethernet to RS485 solution, specifically the ones from SeaLevel.  That is because you can access these without installing any 3rd party driver.  DAQFactory will then communicate with the devices over the standard Windows sockets.  If you really need a PCI solution, however, I would again look at SeaLevel.

 

As I said, I don't usually like to blame other software or hardware, but time and time again if someone is having some weird issue and are using serial ports that require a 3rd party software driver, I'll advise them to replace their serial ports and all will be well.

[raining010]

Hi, I left this topic here for months, because the program had been updated, needed time to check.

 

I revised channel history and channel persist lower. Now after two months, the daqfactory process in site computers remains lower - 15660k, but the physical memeory goes up to 99%. The software version is still 5.87a.

Attached 3 photos are the latest, another 1GB process photo is the first version.

 

I don't know if the 99% memory occupation is related to daqfactory, but from the task manager, there is no process with much memeory occupied. And it's a new computer with win7 os

[AzeoTech]

Is anyone opening and closing applications, such as browsers, etc?

 

The task manager doesn't show everything that has consumed memory.  Many applications leak memory bad, especially, it seems, browsers (all of them, not just IE...).  I have 32 gig on my development machine and hibernate all the time.  Over a few weeks, my base memory usage will climb from like 500meg on boot to 10 gig!  That's just from opening and closing programs, especially browser windows.

[orlee]

Hello Guru,

I suffer badly from memory leaks in DF as well.   I noted that the rate of leaks is directly dependent on errors either in the Device I wrote or scripts.

There are times the leak is barely noted  (Increase size of the task in Task Manager)  and when I disconnect the comm link - the rate of leak increase dramatically.

 

I suspect there is a system error log or buffer that is attached to the DF,  maybe part of the compiler used to compile DF.  May I ask kindly you check where errors are logged?

It looks like a big buffer in the app, and I suspect it is system-compiler by origin.  

 

I get the application grow to huge size when someone disconnect the comm link (it it a tcp/ip USB-Anywhere device)

 

 

Thanks!

[AzeoTech]

That is a definitely possibility, and I appreciate you noticing that and bringing that to our attention.  We'd been having the hardest time trying to reproduce the leak, but maybe that's because we are always connecting to actual devices.  Stay tuned....

[jerzydziewierz]

I don't usually like to blame other software, but over the years I have found that non-windows comm port drivers tend to be pretty unreliable.  This is especially true with USB to serial converters,

 

I concur here, i have recently had a similar issue with a device driver for an USB instrument from one respectable company - took me a week to figure it out. They have not responded to my bug report (!) but I have wrote a workaround to call the offending library call less often.

 

It is possible that they have an USB-to-RSxxx converter internally as they sell the RS232 version of the device too.

 

If you have programming skills - look online, there are generic (non-source-code dependent) tools to hunt down memory leaks; the obviously advertised tools cost money but i have been able to find a combination of free tools to trace the exact .dll and function in it that has the memory leak. Try starting with Microsoft Debug Diagnostic Tool 1.2 from MSDN. Try various versions, different versions have different capability(ugh!)

[AzeoTech]

Yes, the your device probably does have an internal USB to serial converter.  This is kind of standard way for companies with existing 232 devices to make the USB capable, or even those making new devices, as the chips and drivers are readily available. 

 

Finding the memory leak simply requires us to be able to reproduce it.  Now that we know it comes from communication errors, we should be able to find it quickly.  Without being able to reproduce, its next to impossible, no matter what tool you use.  It would be kind of like trying to tell who yelled in a large group of strangers without anyone actually yelling now.  Once you can reproduce it and they yell again, its pretty easy to figure out.

[sadyrbek]

I think it is not the problem of USB converter. Once, I had the same problem with Modbus ethernet.

Finally, I solved using OPC server.

Try to set channels PERSIST 0, then your problem will be solved.