System Suit graphs over multiple result sets

Empower2018

I use a system suit summary plot to graph data from sample sets, so in a sample set with 50 samples, I need to plot an average value per 10 samples so 5 points on the graph. This works fine with one sample set into one result set but...when I run these samples over 2 or more sample sets and use the report to graph this data, I get a tonne of error messages along the lines of "Was looking for processing method ID 1009 and found 1008 instead, used 1008...was looking for processing method is 1010 and found 1011 instead...etc. I don't have the option to suppress error logs.

Any ideas why I'm getting this over multiple sample sets? I wouldn't have though chart data is only limited to one result set as surely you can use it to track resolution and tailing over multiple runs?

LWood

Hi - yes, you understand correctly  :-)  All results are processed with the PM specified during processing and using the limits set within that method (which are in turn used to fault data within that result).  The limits shown on the plot are for display purposes only and are taken from the PM that is stated in the error on the report.

I have logged this as CRI-1132 and will request that it is posted on waters.com.  This may take a couple weeks to appear.

Thank you for providing this information and for helping Waters improve!

HeatherLongden

How are you doing the averaging? By a custom field?

Empower2018

Hi Heather. Yes the average is a custom field along the lines of SAME.%..AVE(Area) however even when graphing a custom field like Area*CConst1 over more than 1 result set I still get that pile of error messages. Is it the search order causing this?

Empower2018

Bump. 

Anyone any ideas how I can get around this issue? I want to graph a peak custom field over multiple sample sets and I'm getting a big error box saying that Empower is looking for multiple processing methods..surely you can graph values over more than one result set without encountering this? Is there a filter of order by condition to remove this?

Dan_Ramlose

Are these  results from different projects? Is the CF setup to look outside a given result set? 

Empower2018

No, the results are all in the same project but different result sets. The search order is Result Set Only for the CF so that might be whats confusing it. The error message I get at the end of the graph is "Was looking for Proc method ID 2009, instead found Proc Method ID 2088, Proc Method ID 2088 was used instead"...followed by another one "Was looking for Proc Method ID 2011, instead found Proc Method ID 2099, Proc Method ID 2099 was used"..and so on, which suggests it could be an ordering thing as well?

The graph actually does plot the correct data but the following error log is very unsightly.

MJS

I think I see what you are getting at...your plot is fine, but you get the error log at the end of the actual report stating this...not that there are errors preventing the plot from working.  See the attachment to confirm...I grabbed a few random result sets and simply plotted area rather than a custom field even...

I'm not aware of any way to suppress the error log (it plagues me in other scenarios as well).  Maybe the ability to suppress/hide it is a worthy feature improvement idea you can submit?

Empower2018

Yes that's exactly what I get too. Which is misleading when I made the graphs because following some examples from Waters slides, they suggest tracking USP Tailing or Resolution values over the lifetime of the column, which means lots of sample sets which means lots of processing methods id. In their slides they conveniently leave out this error log. Maybe its only suitable for pure development work where you don't generate results and only plot values using raw data. 

There is actually an option in report method properties to "Suppress Wrong Channel and Chrom type errors" but this is viewed with suspicion by auditors and even admin staff who ask "Why do you need that". I tried to explain that these error logs are just a visual message (eg selecting multiple injections of only 1 channel, Empower still looks for other channels) but it didn't make any difference, suppress is a bad word in this industry!

It does still work mind you, I may just live with it, it just sticks out a mile and will prompt any auditor to ask deeper questions.  

HeatherLongden

I’m not sure that the Suppress report errors setting will hide those specific messages, but it’s worth a try. The setting is only about suppressing report errors on a report. I believe they’ll still be in Empower somewhere. If someone felt a need to see them in Empower they could. They aren’t lost, just not on the report...

medicnman

Empower2018 said:

Any ideas why I'm getting this over multiple sample sets? I wouldn't have though chart data is only limited to one result set as surely you can use it to track resolution and tailing over multiple runs?

To answer why.... it is for transparency and traceability. Empower needs to show the associated processing methods to trace the threshold values for UCL/LCL, UWL/LWL, etc.... 

The processing method contains the settings for System Suit charts. Using multiple processing methods (ie:different PM IDs) can be precarious; they may contain different settings and may present data incorrectly. If settings are different between PM IDs, it may be an attempt to present data in a polished way. Empower reports each ID for transparency and traceability. One simply needs to verify the system suit tab settings are consistent across the PM IDs.

QA's tend to look at these as "error messages", I think presenting them as a QA tool for traceable and transparent reporting truly reflects the messages' purpose.

Empower2018

Thanks medicnman. Does that mean then that the first PM ID, and any associated UCL and LCL values will be used to graph the data despite it being spilled over 3 result sets? What I mean is if Empower spots 3 PM IDs then will it revert to all the integration and Limit settings of the first PM ID to calculate and graph the 3 values, including amounts etc? That's what I take from the message "Was looking for Proc Method ID 2999, found 3000 instead. Proc Method ID 2999 will be used to make the report". 

Now I haven't found this to change my results as my result for result set 1 was 98, and the other two were 100 and 103 so even using the original proc id kept the same results, if that's what it does?

HeatherLongden

The Empower team are looking into this, and yes, the likely cause is the reliance on the Processing Method to find limits

Empower2018

Thanks Heather, it will be interesting to find out exactly what Empower does with the data when it encounters multiple PM Ids. As I said, it didn't change any of my values but that doesn't mean it doesn't revert to PM ID1 for all values. 

medicnman

Which PM ID it chooses to use is probably based on the first data point in the sort order. I would test this to be sure, but from a coding perspective, that would make the most sense.

I believe Neil Landers presented on the system suit tab functions not two weeks ago "Continued Performance Verification of Analytical Procedures Using Control Charts from Empower Chromatography Data Software". If you have not seen his blog posts and such, do look them up. 

Empower2018

But wouldn't the points actually be calculated using that result sets PM? 3 x Result Sets and a graph of Amount when you select and report all the results would plot the correct points as calculated by that PM ID, no? Your previous point about the error messages being more about multiple UCL/LCL settings does make sense, but I would still think Empower will plot the point for Amount exactly as it was set in that PM for minimum height, average by etc settings?
If not, that would be a very limited option as it would depend on only plotting a set of data exclusively from one result set. I must test this using different UCL settings but the same PM Settings and see what results I get..

Is that Neil Lander article available as a download do you know? Thanks for the mention. 

ydan1977

try this :

https://www.waters.com/webassets/cms/library/docs/720006581en.pdf

Empower2018

Thanks v much ydan1977, that's an interesting and helpful article, did you get that off the Waters website?

Empower2018

Update on this. So I did some testing on this and I set up a simple peak CF called Average_Amount: SAME.%..AVE(Amount) and a search order of result set only. I tested this cf on one active component and 2 sample sets. Each sample set had a different processing method. In these PMs I set the Target and UCL/LCL slightly different in each so a Target of 100 with a 5% error in one and 90% with a 5% error in the other. Processed both sample sets to get 2 x result sets. 

When I graphed both results using a report with a system suit plot with Average_Amount as the field to plot, the results were correct on the graph ( I added them to the Legend and compared them to the stored results) so I don't think its a case of only one of the processing methods being used to actually calculate the amount and hence the Average_Amount- these values are processed and stored as per the appropriate paramaters of that processing method. The issue was with all the errors such as Was looking for Proc Method ID 29000 and found ID 30000. ID 29000 was used to generate the report. 

What I found was that the processing method linked to the earlier of the 2 sample sets ie earliest acquired, was the one defaulted to graph the data, so therefore any Targets and UCL saved with that PM was the one used to visually graph the data, even if the second processing method had a different Target etc which mine did. I don't know if this is exactly how it works as I only tried it twice but both times the Target and UCLs shown on the graph were from the earliest processing method in terms of acquisition, which is odd as I thought Empower would always use the latest version. But at least the results are correct on the graph, regardless of what layout it used. 

medicnman

@Empower2018

Great testing, and thanks for that information. 

If you were to set the Summary Plot Properties... Order by ...OrderBy List... to sort the Results' Processing Method ID to Descend, I wonder if it would use the thresholds from the most recent processing method instead of the original/first PM.

Empower2018

Good question medicnman, I will try that out on Monday and see what I get in return. In theory though, you should have the Targets and error values set the exact same across multiple PM in a project to prevent the problem of data being graphed as per the earliest PM acquired across a set of result data. 

medicnman

Which PM ID it chooses to use is probably based on the first data point in the sort order. I would test this to be sure, but from a coding perspective, that would make the most sense.

I believe Neil Landers presented on the system suit tab functions not two weeks ago "Continued Performance Verification of Analytical Procedures Using Control Charts from Empower Chromatography Data Software". If you have not seen his blog posts and such, do look them up. Here is a link to the webinar...

Ultimately, these charts and the limit values are used to present the data visually. Now I would like to say, with a great deal of certainty, how this works, but I cannot. Do watch his webinar if you have time. 

medicnman

Which PM ID it chooses to use is probably based on the first data point in the sort order. I would test this to be sure, but from a coding perspective, that would make the most sense.

I believe Neil Landers presented on the system suit tab functions not two weeks ago "Continued Performance Verification of Analytical Procedures Using Control Charts from Empower Chromatography Data Software". If you have not seen his blog posts and such, do look them up. Here is a link to the webinar...

http://t.links.waters.com/r/?id=hd0b5026,20e930bf,20eb73c0&nid=552153279&nrid=86230902&nci=2059264334&sap=IFGN19&sa=CRM&nLink=Button%20Text&xcid=ext7049

Ultimately, these charts and the limit values are used to present the data visually. Now I would like to say, with a great deal of certainty, how this works, but I cannot. Do watch his webinar if you have time. 

medicnman

Which PM ID it chooses to use is probably based on the first data point in the sort order. I would test this to be sure, but from a coding perspective, that would make the most sense.

I believe Neil Landers presented on the system suit tab functions not two weeks ago "Continued Performance Verification of Analytical Procedures Using Control Charts from Empower Chromatography Data Software". If you have not seen his blog posts and such, do look them up. Here is a link to the webinar...

http://t.links.waters.com/r/?id=hd0b5026,20e930bf,20eb73c0&nid=552153279&nrid=86230902&nci=2059264334&sap=IFGN19&sa=CRM&nLink=Button%20Text&xcid=ext7049

Ultimately, these charts and the limit values are used to present the data visually. Now I would like to say, with a great deal of certainty, how this works, but I cannot. Do watch his webinar if you have time. 

HeatherLongden

Great investigative work guys.  But yes, the Amounts and "error limit" flags on the results themselves will depend on the individual Process methods, but the control chart is using a Warning Limit that the results don't use. It is finding that Warning Limit from just one of the processing methods.

LWood

Hi Empower2018 and Medicnman-

You are seeing this issue because the same processing method (PM) was not used with all the data you are taking into the report.  When this sys suit report group is used, Empower looks to the processing method to determine the limits so it can plot them. The error might not be the most graceful way to communicate the problem, but it is basically saying that it found 2 different processing methods and it can only plot the limits from one of them.

Unfortunately also, this error occurs, even if the limits in each of the different PMs are the same.

If it would be helpful, I can have this information posted waters.com so that you can reference the posted information to provide justification as to why the errors are benign.

You will want to confirm that the limits being plotted are indeed correct; e.g. from the desired PM.  I didn't check if ordering the data would change what PM was used for the limits, but that would be easy to do.

Also note: This problem will occur even with standard fields such as area or RT; e.g. this isn't related to your use of a custom field.

Empower2018

Thanks LWood that would be helpful to mention on the website that the error is benign, I imagine a lot of people who graph data come across this error. I presume the data points on the graph are actually correct and come from whatever PM was used to integrate and calculate the original amount/custom field etc? And that its only the limits set in the first PM which are used for the Target , UCL etc? So if you get 15 amounts, 5 calculated from one PM, 5 from a second PM and 5 from a third PM, all these values will be plotted correctly but only the Target, Error and Warning values set in the first PM will be used in the graph?