Difference between precision and rounding in Empower
Hello, I would like to know what is the difference, when using Empower, between precision and rounding? I am testing a custom field which averages the area of my main peak over 6 injections. Thing is, when I compare the calculator results with the custom field its not exactly matching. Its only a small difference, eg I get 109887.176 on the calc but 109887.178 on Empower. I have my custom field set to a precision of 3. Am I getting different results because the stored results are different to that of with a precision of 3.
I don't understand why they would be different, is rounding and precision different on Empower?
I don't understand why they would be different, is rounding and precision different on Empower?
1
Comments
-
How did you calculate this average in Excel? From full precision area from Empower (15-16dp) and then displayed or rounded to 3 dp at the end? Have you tried rounding areas to 3 dp (not just displaying) before averaging?1
-
Comapring results between Empower and Excel is always somewhat of a slippery slope.
I would recommend calculating your result in Empower with much greater of a precision than theee. I typically use a precision of 6-10 depending upon the complexity of the calculation (where precision increases as the difficulty of the calculation increases).
At the end of the day you need to be aware that you are always going to be battling the attached.
1 -
It is likely that Empower is using unrounded numbers, and then rounding the answer only, which is not how you are doing it in Excel. Plus the whitepaper shaunwat attached explains how Empower performs rounding ( on the answer) which again, is likely to be not how Excel does it.2
-
Also - using CFs to do standard average response is like swatting a fly with a Howitzer (apologies to PETA). This is more easily done by creating a component table, filtering by label (typically S0101 if you do 6 injections before assigning S0201 to the next standard) and adding summary statistics to the area (and/or height) columns. If you want to look at responses for all standards, you just make a copy of said table, place it in the report method and change the S0101 filter to S*. If there are unwanted peaks in this table, you can filter them out too (easiest by tagging "main component" as the peak type in the processing method, then filtering for this in the component tables)...1
-
Thanks very much for the comments people, they were very helpful.0
