ROUND and GTE/LTE Not Working

SteSJW1

Hi,

I'm stuck.  I have this boolean peak custom field

GTE(ROUND(<Avg Amount CF>,-1),<Lower Limit CF>)&LTE(ROUND(<Avg Amount CF>,-1),<Upper Limit CF>)

It's looking to round the average amount to 1 dp and compare it to the inputted sample set CFs for lower limit and upper limit so if it sits between for example 17.4 and 20.5 then I get a "Valid" output and if outside an "Invalid" output.

I have an average amount of 17.428 and it's coming up as invalid although to 1dp it is equal to the lower limit of 17.4.

Is this a floating point issue that I've seen in some of the other responses?

Empower2019

I'm not sure why, I've had the same issue using the impurity tab where the precision of my reporting threshold is at odds with the full precision of the value so 2.5 is a fail even if its 2.4999999!. 

I suggest you use CConst1 and CConst2 instead of Lower Limit CF and Upper Limit CF. I've coded dozens of RANGE, GTE LTE Bool formulas and anytime I use the CConst fields for upper and lower values there has never been an issue with the rounding precision so a value of 17.428 will be seen as 17.4 to 1 dp. If you need the custom field to apply for both known and unknown peaks then change the formula to CCalRef1[CConst1] and CCalRef1[CConst2].

Empower2019

You could add a ROUND to the values in Lower Limit/Upper Limit Cf as well so they both align:

GTE(ROUND(<Avg Amount CF>,-1),ROUND(<Lower Limit CF>,-1))&LTE(ROUND(<Avg Amount CF>,-1),ROUND(<Upper Limit CF>,-1))

SteSJW1

Empower2019 said:

You could add a ROUND to the values in Lower Limit/Upper Limit Cf as well so they both align:

GTE(ROUND(<Avg Amount CF>,-1),ROUND(<Lower Limit CF>,-1))&LTE(ROUND(<Avg Amount CF>,-1),ROUND(<Upper Limit CF>,-1))

Thank you, it does fix it which is half the battle but I don't understand why, so I don't make the same mistake elsewhere.  Without rounding the 17.4, it would surely be picked up as 17.40000... which is still at least equal to a 17.428 rounded to 1dp.