hexagon logo

Decimal place rounding vs truncating

Our engineering manager asked me recently if we could change decimal place of certain characteristics on the report to either 3 places or 4 places, but not necessarily have the entire report be one or the other. Like some characteristics would be three places and some might be four places. So I told him that we can certainly do that. Then he asked if when we changed a characteristic from 4 places down to 3 places, does it round or truncate? I told him that I think that it rounds, but I'm not sure. Does anyone have any insight into this?
  • It Truncates.

    To prove it, Create a generic feature with the actuals being close to where you would round. Report it with varying decimal places.
  • It may appear to truncate it, but it does not. +/-0.1mm on a 10mm nominal with a deviation of 0.10001 will show a deviation of 0.100, but it will still show OOT. decimal places will NOT hide OOT for values no longer displayed.
  • The displayed value is rounded however to Matthew's point, the system will flag a value that has trailing data that is out of tolerance.
    So .5996 when set for display precision of 3 will show .600. If .600 is the min shown on the print it will be a red value.

    To truncate the data you would have to pass all the information through variables and then output through generic features. In this case, your report would show .599 and be red.
  • Might be better now, might have been specific to my machine at the time, but a while ago, I did this to a program, it was a longer one.

    I thought 3 places for ± .030" and 2 places for ±5° was more appropriate than 4 decimal places for everything. And it is.

    The program went from taking maybe 60 seconds to load onto the computer to taking 90 seconds, and editting could get painful.

    At that time, I decided it wasn't worth the effort and stopped doing what you suggest, and just leave it at one reporting resolution.
  • 10 inch diameter +-.005 measuring 10.0051 (with three decimal place dimension display precision) will show 10.005 on the report but will be red n thus oot. I've seen people turn off the red before, but thats dirty pool imo.
  • It rounds. DISPLAYPRECISION, toggle back and forth. Note that (at least in most versions) when creating a characteristic, it keeps the number of decimals that was active at the time you created it, adding the command after the fact does nothing.