Rounding out numbers
Options
I have such a huge issue.
Before processing a sequence of injections in Empower, I review one injection of a standard solution and I get an amount of 99.46293750% and if I made it be a 6 decimals number it is rounded out to 99.462938%. This amount if a peak/real/calculated value custom field. Result set first custom fields and it is calculated for all types of samples and peaks. That´s ok, fine.
But if I process the sequence I review and preview the same injection getting the following: an amount of 99.46293750% and if I made it be a 6 decimals number it is not rounded out to 99.462938% but to 99.462937%. So it seems as if it is "cut". Why is it different before and after processing the sequence? Why is it either round out or "cut"?
Thanks
Before processing a sequence of injections in Empower, I review one injection of a standard solution and I get an amount of 99.46293750% and if I made it be a 6 decimals number it is rounded out to 99.462938%. This amount if a peak/real/calculated value custom field. Result set first custom fields and it is calculated for all types of samples and peaks. That´s ok, fine.
But if I process the sequence I review and preview the same injection getting the following: an amount of 99.46293750% and if I made it be a 6 decimals number it is not rounded out to 99.462938% but to 99.462937%. So it seems as if it is "cut". Why is it different before and after processing the sequence? Why is it either round out or "cut"?
Thanks
0
Answers
-
Hi PMJ! This Knowledge Base article does not exactly answer your question, but there is a linked PDF that might.
Let us know if this helps,
Marc0 -
Sounds like the same problem i had some time ago
https://forums.waters.com/discussion/5075/empower-3-round-discrepancies
Empower3 is weird about rounding 5's
I ended up removing any intermediate ROUND functions and whenever possible relying on the Data precision function under "Column properties" or setting data precision in reportmethods, where rounding works as you'd expect. So far i've either fixed the problem, or obfuscated it enough that it's not obvious.
Best of luck and let me know if you find a solution!0 -
Don't forget too that Empower actually calculates to 17 decimal places in the background but only displays 14 decimals as maximum precision, so any discrepancies in rounding can sometimes be attributed to Empower averaging 3 numbers which are actually different from that displayed- for example, 3 amounts of 12.34567888567237, 12.34567998765438 and 12.23456779876548 would typically average to one figure based on the figures you see.
But, the true values when considered to 17 decimal places (Oracle calculates this but as mentioned, only displays 14 decimal places, as this is sufficient for most calculations which require accurate outputs) could actually be 3 different figures which will give a different average.
0 -
So, the "take home lesson" is: Let Empower do what Empower does and set your precision in your report method tables. If you see a discrepency, look another place or two to the right before deciding something is wrong.
If you're checking Empower calculations, export/copy at least 3 places beyond what you'll be reporting in the end and make sure your inputs are EXACTLY what was entered into the sample set (which may require displaying more precision than is defaulted to in the component editor or corresponding report table).0
