How does Empower round this number?

Empower2018
Empower2018
edited December 2018 in Software Products
Calling out for some help on this one before I drive myself crazy thinking of it. If I have a value for amount of 74.49999999999999 and I want it displayed to a precision of 0, does Empower perform a "double rounding" first to 74.5 and then 75? I don't physically want to Round the value as in Round(Amount,0), I just want to know what will be displayed if I choose to set the precision to 0 for that amount. 

I haven't been able to recreate that exact number in Empower, thanks. 

Best Answers

  • Dan_Ramlose
    Dan_Ramlose
    Answer ✓
    From Empower help:

    Method that Empower software uses to round numbers

    Empower software evaluates the digit in the decimal place to the right of the specified precision in the unrounded number.

    • If this digit is smaller than 5, that digit and all digits after it are eliminated, and the preceding digit is unchanged.
    • If this digit is equal to or greater than 5, that digit is eliminated, and the preceding digit is increased by 1. 
  • shaunwat
    shaunwat
    edited December 2018 Answer ✓

    I've tried to pose the white paper before, and the forums don't like it anymore.

    I would strongly encourage you both to look at TECN1851937 (http://www.waters.com/waters/download.htm?lid=1851937&id=1851938&fileName=Differences+in+Rounded+Values+in+Empower+Software&fileUrl=%2fwebassets%2fcms%2fsupport%2fdocs%2fdifferences_in_rounded_values_05.pdf)

    XX.49999999999999999999999999 (etc, etc) is a very hard number for Empower to represent on account of conversions between Base 2 (oracle) and base 10 (Microsoft).


    Pay close attention to example one in the linked document.

    The answer given, though certainly correct, is technically not accurate for this unique case you used in your example. Depending upon where you are in this process (review versus processed result) you could potentially be pulling every last hair our of your head trying to figure out why numbers are not rounding "correctly".


Answers

  • Dan_Ramlose
    Dan_Ramlose
    Drop it in a project as the default value for a CF and then look at it in a report with 0 precision...
    My guess is that it will come up as 74.
  • Empower2018
    Empower2018
    Thanks very much for that Dan, its a good idea to throw it in as a default as well to test that theory, thanks. 
  • Empower2018
    Empower2018
    Thanks very much for the document, shaunwat, that was very helpful. It seems that saved results ie areas and amounts etc are the definitive values because rounding is different before and after saving. Im being asked to set a limit with a minimum value for amount of 75. Going by the logic explained in the handout, I can set 74.5 as the minimum value because Oracle converts 74.5 to 75 when result is saved, and 75 is within spec. 

    I think a more sensible way to check the limits would be "at quantitation only" because this deals with saved results "Always" may display a value which isn't saved and give a false fault. Thanks again for all the help on this for you and Dan. :)

Categories