Weird decimal behavior in numeric fields

   2870   4   1
User Avatar
Member
26 posts
Joined: June 2016
Offline
Sometimes when I enter a float values I get a very strange precision error. See attached image. Anyone know a solution to this?
Edited by 8BitBeard - Jan. 13, 2017 07:31:33

Attachments:
decimal.png (9.1 KB)

User Avatar
Member
517 posts
Joined: Dec. 2013
Offline
It looks scary but it's not to worry about, if you click the parameter name it should go back to something that looks better

I think thats how the values are actually stored internally?
User Avatar
Member
2038 posts
Joined: Sept. 2015
Offline
It's not a precision error. The solution is to understand why you are seeing what you have.

There are a number of sources that explain it but here is one:

https://docs.python.org/2/tutorial/floatingpoint.html [docs.python.org]
User Avatar
Member
4189 posts
Joined: June 2012
Offline
It's always helpful to know 1 and 0.999… are equal:

https://www.physicsforums.com/insights/why-do-people-say-that-1-and-999-are-equal/ [physicsforums.com]

Edit: For those interested in this:

“9.999… reasons that .999… = 1”
Edited by anon_user_37409885 - Jan. 13, 2017 20:24:14
User Avatar
Member
2038 posts
Joined: Sept. 2015
Offline
Actualy 1 and 0.999…. are not equal and there is no proof, contrary to the assumed proof.

The reason is because one hasn't yet defined 0.999…

By writting 0.999… as is, one is saying it goes on infinitely.

If it's going on infinitely, one has not yet established what that number actually is;

If one doesn't know yet what the number IS, one cannot make a true comparison, whether it is equal or not.

I know I'm being a bit pedantic, but to say both are equal is simply not true and can lead to wrong assumptions about results one gets when writing equations.

1 divided by 3 gives a number that goes on infinitely, one will never get a true finite number from that operation.

It's good to know that there are many instances in which you don't get finite numbers, and in those cases knowing how your software/hardware will handle those situations eg. number of decimal place precision for floating point numbers before they become rounded, or if desired truncated - This, I think is a much more helpful way to look/understand the situation.
Edited by BabaJ - Jan. 13, 2017 15:59:14
  • Quick Links