Data Collection Stops Early – Instrument Failure / Communication Error
Answers
-
Hello,
The entries from the Empower message center would be useful. We would like to verify the version of firmware for the Ethernet and IEEE instruments and the version of the ICS if you could provide this information as well that would be good.
Thank you
Dot
0 -
I have the same issue with a brand new stand alone Empower2 workstation installed in about February 2010. Sporadically, several times since IQ/OQ/PQ, the data collection stops and the HPLC runs out of mobile phase over night. Waters Tech Support suggested that we insert an exception in our antivirus software for the Empower2 folders, and set all computer standby and hybernate -type functions to off.
0 -
We frequently experience the same issue with Empower 2. Data acquisition stops, pump continues all night. The only error message we see is' Awaiting next injection' in the Run Samples window but no error messages in the message centre or anywhere else. Waters cannot tell us what is going on. The stop is random and not connect to a system, project, user or LACE
0 -
Hello,
I have discussed this with some team members. We have had similar occurrence. The system may not have been the exact configuration as yours but we do have some suggestions for you.
In the configuration described you are probably using an inject start cable and the signal did not occur. Make sure the contacts for the cable are secure or even better yet change the cable in case it is marginal or bad.
Let us know if this is helpful,
Dot
0 -
I am having the same problem for the past 2 weeks and it drives me crazy… The symptoms are mostly the same as described above.
- Empower stops collecting data in the middle of an injection. Occurrence of this event is completely random within a sample set run but it does stop everytime I tried. My sample set method consists of 6 injections. Empower stops data collection at random point; at 7 min, 30 min, 42 min,etc. and random injection of the sequence; sometimes stops at the first injection, 2nd injection, 4th injection, the sample set run was never completed.
- Once Empower stops data acquisition, Empower becomes non-responsive. I have to reboot the node several times (one rebooting never got Empower working). After this happens, Empower displays “Instrument Failure” in the run sample window and the message center, but doesn’t seem to send out that error signal to instruments, so the shutdown method never kicks in, the instrument continues to wait for the next injection trigger from Empower and eventually mobile phase runs out. I don’t think anything wrong with the instrument itself. When a real instrument failure occurs, Empower always displays which component is at fault. e.g. “Instrument Failure: W2487@4”.
- I checked the cables and didn’t see any issues there. If the cable is bad, I shouldn’t be able to control the instrument and start the sample set in the first place. I do several single injections before I start a sequence, the single injections never had issues (or maybe just a luck?).
- Waters tech suggested that the sample set method somehow corrupted so I created all new instrument method, method set, and sample set method, but that didn’t work out.
- I tried separate project folder with new set of methods, but still sample set didn’t complete.
- Our Empower Administrator removed the system configuration from Empower and put it back on to reset any bad communication. That didn’t help either.
- You have suggested checking the AS trigger cable, but we don’t use that.
Could you please investigate this issue further? I really think this is something to do with Empower side. I’m sure a lot more people are suffering from this than posted here… It really bothers me not knowing what is triggering this error every time I start a sample set run.
We have Empower 2 (Pro) built 2154 service pack D. Instrument components are:
1525µ pump, W2487 detector (rev.1.02), and Wisp717plus autosampler (rev.3.1), all purchased and installed in early 2007. The instrument has not been used that much, but never had this issue before.
All instruments are connected via IEEE 488 cables to a BusLAC/E card (driver version 4.0.0.0)on a node. Only this instrument is connected to this computer.
OS is Win XP pro SP2. Mandatory screen saver kicks in after 15 min. It will lock the computer (user log-in required) but does not hibernate. The error once happened when I was working on the node for something else, so I don’t think it has anything to do with the screen saver kicking in.
Currently this specific error is happening on this instrument only. We have another system with similar configuration (1525 pump, 717AS, 2996PDA), and the maintenance log indicates same thing happened to that system in the past. Since that system has a pump issue, can’t use it for testing.
Thanks for reading and hope you could find a solution for us soon.
0 - Empower stops collecting data in the middle of an injection. Occurrence of this event is completely random within a sample set run but it does stop everytime I tried. My sample set method consists of 6 injections. Empower stops data collection at random point; at 7 min, 30 min, 42 min,etc. and random injection of the sequence; sometimes stops at the first injection, 2nd injection, 4th injection, the sample set run was never completed.
-
Hello,
Checking the cables and connections is always good to insure the problem is not with these components. Creating new methods is an excellent test. There is the chance these might have been corrupted. Now let’s move on to more troubleshooting.
When the Empower run samples stops collecting data in the middle of an injection, does the real time plot go on in a straight line or does it stop right there?
If it continues to plot in a straight line this would indicate possibly an issue with the detector. However, should the plot stop immediately then we need to look for other information. For example, does the message center have entries prior to or after the ‘instrument failure’ entry that are associated with the run and this system? Are there any error or warning entries in the Windows application or system event viewer at or about the time of each failure occurrence?
Let us know does this help.
Dot
0 -
Dot, thanks for your response. When Empower stops collecting data, the real time plot stops right there—no more baseline beyond that point. The rest of plotting area is completely blank. When this happens, say the chromatogram/real time plot stopped at 41 min of my 60 min run, the time counter at the right side bottom of the run sample window keeps going. It keeps counting until 70.0 min (exactly 10 min from my designated run time), that’s when Empower displays “Instrument Failure” message both at the bottom of the run sample window (status bar) and the message center. There was no other message before or after the instrument failure in the message center. This error indication has been consistent. Once Instrument Failure is displayed, Empower becomes non-responsive and needs rebooting. When real plot stopped (but time counter still working), I can’t stop the flow using the stop flow icon but I was able to abort the sample set run. Either way, rebooting was needed eventually.
I checked Windows Event Viewer for both application and system, no entries whatsoever at the time of Empower errors.
I’ve been using a 60 min isocratic run, but the same thing happened with a 60 min gradient run. So I don’t think it’s method related. I’ve been sporadically using this LC system since 2008 and never had such problems. But it had been always short run time (max 15 min, but as many as 16 injections in a sample set method). I wonder if this is anything to do with the longer run time I’m using now?? I also wanted to mention that the project folder has more than enough tablespace so the Empower error was not related to that.
Dot, you’ve mentioned that the same thing has happened at your site as well. Was that single occurance or multiple? I can’t imagine all of them were caused by bad injection trigger cables. What else was done if this has happened multiple times? Thanks so much for your assistance.
0 -
Hello,
I consulted with team internally and there are two other reports similar to yours, IEEE with a system configured with a using the 2487, that are currently being investigated. Both are intermittent. You mentioned there might be another 2487 in your lab. If possible I would suggest replacing the detector with another one in the lab and see if the problem follows the detector. Please let me know if this is an option.
Thank you
Dot
0 -
Thanks, Dot. For GLP reasons, switching/replacing major component of a qualified system is a last resort to us. I don’t mean to disrespect your suggestion, but may I ask what is the reasoning behind it? W2487 seems to be working fine. If the detector stops sending the data to Empower, Empower should detect that with an error message indicating problem receiving signal from W2487. But in my case, the error message has been always non-specific, and Empower gets hung-up everytime, thus I thought the problem was on Empower side. Thoughts?
0 -
Hi
Your environment is important and you should follow the company SOP requirements. We’ve not ruled out that there could be an issue within Empower. The current investigation is such that the focus is on the 2487. I will discuss further with the teams and make sure we include all possibilities for Empower. You’ve verified there are no messages related to the instrument or the system prior to or following the instrument failure entry in the Message Center Also that there are no warnings or errors in the event viewer entries. I would doubt there is a shutdown method but I would like to ask.
Thank you
Dot
0 -
Dot, thank you. I always set a shutdown method whenever I let a sample set run overnight. As I mentioned in my first post, the shutdown method never kicked in when this Empower error happened and the flow kept on going so the mobile phase ran out overnight. It looks the same way for other posters above; if a chromatographer set an overnight run, he/she would have enough mobile phase until the morning without having a shutdown method in place.
I guess I shouldn’t rule out the possibility of faulty detector but for now I need to focus on the pending project work with another instrument. Please keep me posted if anything new you find out. Thanks again!
0 -
Hi
We do not use inject start cables at all. The system set up is Agilent 1100/1200 HPLC's on an Empower 2154 network. Reading through some of the other details for this thread I am including some additional infromation for you
Injections do not stop mid way, an injection is completed and before a new one starts we get the 'Awaiting Next Injection' message
We ruled out corrupt methods as you can simply restart a stopped Sample set with no other interactions
We do have some shutdown methods but the system does not get as far as this
The message board does not give any messages
We cannot find a trend for system, project, user or LACE
Do you have any other suggestions?
SKM
0 -
Hi!
My upgrade from 1154 to 2154 FR5 experienced the same problems as you have. I managed to fix this by :
- Deactivate all but one CPU-core in BIOS
- Latest service pack for Empower and instrument
- Deactivated all powersaving for network card in computer (yes, it's powersaving there as well)
- One of the connected Agilent 1100 had a CAN-cable problem (but had problems on all 4 connected instruments)
- Upgrade of firmware in all A1100 modules (same revision).
We still experience some communication problems, but not that often.
- kjbu
0 -
I've had a lot of very strange and hard to explain problems that were solved by replacing the IEEE cables.
Sometimes they do go bad just sitting there.
My two cents.
Edit to add: We had some dell computers, and the IEEE card would work itself loose sometimes. Open the PC and reseat the IEEE card.
0 -
Data collection stops are not unheard of in our E2 systems either. I'm currently investigating a very recent reoccurrence of an early data collection stop in two separate runs. We previously observed this particular phenomenon about 6 months ago. After eliminating all other possible causes at the time we shelved the LACE and replaced the busLACE card some time later. The LACE has been back in service for a few months and we again have experienced this issue. We're observing that a very limited number of injections collected via busSAT/IN connected instruments are stopping data collection early. Previous and succeeding injections are often collected properly. In our most recent case with 15 min runtimes the first injection collected for 13 minutes (precisely), the second injection for 14 minutes (precisely), and the remaining injections collected successfully for the expected 15 minutes. We've reviewed all available methods and logs and have not seen anything that might be responsible. To my knowledge we have not been experiencing this issue with other systems/LACE's. This "problematic" LACE has been in two different labs now, connected to different SAT/INs and chromatography instruments, and using different projects. I don't see that this is instrument related since the busSAT/IN really doesn't care that an instrument is there or not after data collection is started from my experience. We also have enabled additional logging on the LACE's that tracks the start and stop time as well as other events for each injection. Those logs also indicate that the run times for the shortened injections were indeed 15 minutes. I've exported the raw data and have verified that there is no data after 13 and 14 minutes with the impacted injections. Yet, the database runtimes and all other indicators show that the runtimes were 15 minutes. Lab users have total confidence that their run times were correct. What happened to the data points?
In addition to these collection issues we also have seen LACE's lock resulting in data collection stops alltogether and periodic collection stops mid-run or between injections. While these issues are not rampant and represent a relatively low percentage of our total # of acquisitions they're a disruption to the user community and result in lost time and $. Waters tech support has not been able to explain the isolated incidents when they've occurred. As is often suggested by Waters we've disabled AV scanning of the Empower directories. AV scanning has never been proven in our instances to be the cause of the trouble we've seen.
I welcome any thoughts and responses. Thanks
0 -
Hello GPK,
You’ve described two issues. Let’s discuss one for a bit and then gather information on the second one.
Not all instances of data collection stop occur for the same reason therefore finding the root cause can be quite tedious at times. You have already performed a number of steps to isolate the problem. This is really helpful. It appears in this case the issue may be isolated to the one Lac/e32. Since the buslac/e card has been removed, can I assume you are connecting to an Equinox card? If so, what version of the Equinox driver is installed?
Thank you
Dot
0 -
Hello Dot,
So far, indications are that it's one LAC/E32. I hope that is indeed the case. We received a new busLACE card from Waters and installed that in the suspect LAC/E32. So, we're still using a busLACE card to connect the busSAT/INs in this case. The users have requested that the LAC/E32 be replaced with a spare as they have no confidence that the issue will not arise again. Let me know if you have ideas to troubleshoot this particular box further. Re-imaging is being considered but I'd like to look at all other potential solutions before doing so to try to understand this.
Regards,
Greg
0 -
Hello Greg,
Not seeing any logs and just joining the troubleshooting, replacing the buslac/e was a good suggestion and could be the resolution in this instance. You mentioned previously that additional logging was enabled. Was this the buslac/e diagnostics or some other type of logging? Were the logs and other logs sent to Waters? Also, what is feature release/service pack of Empower 2 that is running at your site?
Thank you
Dot
0 -
Dot,
This has occurred with two busLACE cards so we didn't necessarily have one bad card. For logging there is a registry entry on the LACE's that can be set to dump the instrument server log to a folder for review. I sent that log and all others to Waters for review. We're using FR5 SP G Hotfix 2 now, but this occurred with our previous version FR5 SP F Hotfix 1 as well. An exerpt from the instrument server log showing two impacted injections (id's 20371 and 20374) is shown below. 20371 had 13 minutes of data and 20374 had 14 minutes of data collected. The next injection in the sequence, 20381, had the expected 15 minutes. This log, and the sample set method indicate that all three injections should have collected 15 minutes of data. Very odd.
Thanks
GPK
12/23/10 12:38:45 SetDefaults Controller Msg (Begin) : Code 18.
12/23/10 12:38:45 Connected to Storage Server WAT11, Project *****************.
12/23/10 12:38:45 SetDefaults Controller Msg (End) : Code 18.
12/23/10 12:39:10 SetDefaults Controller Msg (Begin) : Code 18.
12/23/10 12:39:10 Connected to Storage Server WAT11, Project *****************.
12/23/10 12:39:10 SetDefaults Controller Msg (End) : Code 18.
12/23/10 12:39:30 Stop Recording Proxies.
12/23/10 12:39:30 Running SampleSet JG_122310_A42.
12/23/10 12:39:30 Connected to Storage Server WAT11, Project *****************.
12/23/10 12:39:30 Created SampleSet 20369 in Database.
12/23/10 12:39:30 Executing Inject Sample Set Line.
12/23/10 12:39:30 Setup Proxies.
12/23/10 12:39:35 SetupReply Controller Msg (Event Only - SetupProxies) : Code 19.
12/23/10 12:39:35 Creating 1 Vial(s) in Database.
12/23/10 12:39:35 Created Vial 20370 in Database.
12/23/10 12:39:35 Wait Proxies Ready.
12/23/10 12:39:35 ReadyInjectReply Controller Msg (Event Only - WaitProxiesReady) : Code 20.
12/23/10 12:39:35 Creating 1 Injection(s) in Database.
12/23/10 12:39:35 Created Injection 20371 in Database.
12/23/10 12:39:35 Start Chromatograms.
12/23/10 12:39:36 Start Remote Update.
12/23/10 12:39:36 Enable Proxies.
12/23/10 12:39:38 EnableChannelsReply Controller Msg (Event Only - EnableProxies) : Code 21.
12/23/10 12:39:38 Inject Proxies.
12/23/10 12:39:38 Wait Inject Occurred.
12/23/10 12:42:14 InjectOccurred Controller Msg (Event Only - WaitInjectOccurred) : Code 22.
12/23/10 12:42:14 Wait Proxies Complete.
12/23/10 12:57:27 EndOfRun Controller Msg (Event Only - WaitProxiesComplete) : Code 23.
12/23/10 12:57:27 Finish Chromatograms.
12/23/10 12:57:27 Stop Proxy Channels.
12/23/10 12:57:29 Injection 20371 Complete.
12/23/10 12:57:29 Executing Inject Sample Set Line.
12/23/10 12:57:29 Setup Proxies.
12/23/10 12:57:35 SetupReply Controller Msg (Event Only - SetupProxies) : Code 19.
12/23/10 12:57:35 Creating 1 Vial(s) in Database.
12/23/10 12:57:35 Created Vial 20373 in Database.
12/23/10 12:57:35 Wait Proxies Ready.
12/23/10 12:57:35 ReadyInjectReply Controller Msg (Event Only - WaitProxiesReady) : Code 20.
12/23/10 12:57:35 Creating 1 Injection(s) in Database.
12/23/10 12:57:35 Created Injection 20374 in Database.
12/23/10 12:57:35 Start Chromatograms.
12/23/10 12:57:35 Start Remote Update.
12/23/10 12:57:35 Enable Proxies.
12/23/10 12:57:38 EnableChannelsReply Controller Msg (Event Only - EnableProxies) : Code 21.
12/23/10 12:57:38 Inject Proxies.
12/23/10 12:57:38 Wait Inject Occurred.
12/23/10 12:59:06 InjectOccurred Controller Msg (Event Only - WaitInjectOccurred) : Code 22.
12/23/10 12:59:06 Wait Proxies Complete.
12/23/10 13:14:18 EndOfRun Controller Msg (Event Only - WaitProxiesComplete) : Code 23.
12/23/10 13:14:18 Finish Chromatograms.
12/23/10 13:14:18 Stop Proxy Channels.
12...0 -
GPK
Please could you let the forum know of the registry key - this will be useful for all of our troubleshooting activities.
Thanks
0 -
HKEY_LOCAL_MACHINESOFTWAREWATERS
Name: InstrumentServerLogDirectory
Type: REG_SZ
Data: path to the log directory on LACE (e.g. C:Instrument log)
A reboot is required to start it up. It creates a different log for each instrument. The logs will capture data for every injection so you'll need to purge periodically. I've set it up to copy the files out of the primary directory into backup folders upon reboot. It will create a new logfile when the next run starts up.
Note: Waters Tech Support isn't familiar with this log if you mention it on a call.
-GPK
0 -
As SparHawk mentioned, a possible cause of having runs stop is that the PC goes to power saving mode, or hibernation. Due to default IT policies, power saving was on by default in my lab, so you may want to talk to your IT to change that. It is almost impossible to ensure that each new user changes the power settings to "always on" on every acquisition client, especially if the lab is big and/or there is a high staff turnaround.
If you are using an ethernet switch, the ethernet switch will have it own power saving setting, different to the PC. The PCs in my lab would turn off the ethernet switches by default. To avoid this, make sure you untick the option which allows the PC to turn of the switch to save power (in the device manager, from memory).
We also had a few instances where the domain server (different from the Empower server) took too much time to renew the lease of the dynamic IP address the acquisition client was using. That stopped the run too. Change the clients to static IP.
In any case, you may need to review the application/security/system logs on the client controlling the system that stopped to identify the exact cause of the issue. Logs can be accessed by right clicking on MyComputer, then Manage. Logs are kept in System ToolsEvent viewer. Admin privileges on the client are required. Review the logs with a particular attention to the events close to the time of the issue. That time can be determined with Empower, either in the message log, or by checking when the injections stopped.
The most important is: it can be fixed for good. By having IT change the IPs to static and changing the defaul power saving mode, and disabling power saving on the ethernet switches, I eliminated the problem completely.
Hope this helps.
0 -
JayVee,
How did you determine that the domain servers were taking too much time to renew the lease for the acquisition clients?
-GPK
0 -
GPK,
I had to review the client's logs. Those can be found by right clicking on "My Computer" (usually on the desktop), then "Manage". The Application, Security and System logs are in the Event Viewer, inside System tools.
You may not have the privileges to do this though, if you are not a local admin on that PC. In that case, IT will be able to do it for you.
When reviewing the logs, start at the time of the issue, and look at the warnings (yellow) or error (red) messages. There shouold be only few of those. There will be many, many, information messages, and sifting through those can be time consuming.
With a bit of practice, you will be able to tell rapidly what is relevant and what isn't. For example, I know from practice that, for my clients, messages from "b57w2k" relate to the ethernet controller (if you suspect the ethernet controller went to power saving, look at those. See attachment).
I can also identify when the PC goes to power saving by looking at the logs, because there are several information messages about "services entering a paused state".
Similarly, messages from Waters Services, Waters DHCP and so on are usually relevant.
I cannot remember if messages about the IP lease by the domain controler where information, warnings, or errors - but they will be at the time of the issue anyway.
0 -
Hello GPK,
A quick question about your registry key setting. Which version of Empower are you using when you quote this key? We are running Empower version 1 and I tried applying it as you show but no log is activated following a reboot
Our registry tree looks slighly different (Empower subkey exists) so guessed it should go here as in attachment.
Please advise
Thanks,
Kate.
0 -
Kate,
You are correct about the key location, that was a mistake on my part. It belongs in the Empower subkey. The logs won't start writing until you start data acquisition. Create the log directory manually if you haven't already. We're using Empower 2, I'm afraid I don't know if this will work with Empower 1154.
GPK
0 -
Hello,
Try to see in the eventviewer-Application log any error as .NET RUNTIME
Regards
Lakshmi Rathan
0 -
You coul have a look at the time set on the actual autosampler. We had similar issues with injections stopping mid run before the end time set in Empower and it was because the autosampler had an internal time set which was shorter than the Empower time. Since we schronised we have not seen the issue
0
