• iglou@programming.dev
    link
    fedilink
    arrow-up
    3
    ·
    2 days ago

    I would say your biggest issue here is needing precise decimal point computations and using imprecise data types. Any software that requires precision in the decimals needs to use types that are made for precise decimals. No floating point error.

    • moseschrute@lemmy.world
      link
      fedilink
      arrow-up
      1
      ·
      2 days ago

      For this specific project, I need max 2-3 decimal places of precision. So rounding really fixes all the issues. It’s more that we’re preventing the user from seeing awkward decimals. We aren’t doing rocket science. But understanding what metrics need what precision, and sometimes the same metric needs a different precision in different contexts.