Either I can't do math or Quantopian can't

Back to Community

edited

So I have this toy algorithm that is a stripped down version of something I am working on and I have noticed a strange behavior. In the month of July 2014 there it calculates a change in unemployment rate of -0.2 and then does a two sets of comparisons, both simple if unr_diff < -0.2 and in one case it results in false and in the other it says true. What is going on?

This is the relevant part of the code:

log.info("unr_diff: %.2f" % unr_diff)  
log.info("unr_diff < -0.2: %s" % unr_diff < -0.2)  
if unr_diff < -0.2:  
    outlook += 0.5  
    log.info("(unr_diff < -0.2) outlook + .5")

This is the relevant part of the log:

2014-07-07trade_unr:52 INFO unr_diff: -0.20  
2014-07-07trade_unr:53 INFO False  
2014-07-07trade_unr:57 INFO (unr_diff < -0.2) outlook + .5  
2014-07-07trade_unr:60 INFO Outlook: 0.5

8 responses

Charles Piché

Hi Andrew, try putting "unr_diff < -0.2" within parentheses in your log statement.

log.info("unr_diff < -0.2: %s" % (unr_diff < -0.2))

Andrew McKenna

You're right I was logging that incorrectly. It still does not explain the strange behavior though. After the correction the log appears as below:

2014-07-07 trade_unr:52 INFO unr_diff: -0.20  
2014-07-07 trade_unr:53 INFO unr_diff < -0.2: True  
2014-07-07 trade_unr:57 INFO (unr_diff < -0.2) outlook + .5  
2014-07-07 trade_unr:60 INFO Outlook: 0.5

Charles Piché

When you are logging the unr_diff variable, you are providing a %.2f format. The real float value probably has something after the first 2 decimals.

For example:

Real value = -0.200008
Logged as "-0.20"
unr_diff < -0.2 == TRUE

I hope I understood your problem correctly! ;) And sorry if not.

Andrew McKenna

The values it is subtracting only go out to a tenth, but I printed out to the hundredth in the logs just to make sure.

To leave no stone un-turned I went ahead and tried printing out to the 20th decimal place and you were right about the float value being different. My question then is why when subtracting two numbers such as 6.1 and 6.3 do I get -0.20000000000000017764? That seems like it could bite some people when they are trying to make decisions on calculations and there are mysterious remainders appearing on the end.

2014-07-07 trade_unr:39 INFO previous UNR: 6.3  
2014-07-07 trade_unr:40 INFO -  
2014-07-07 trade_unr:41 INFO current  UNR: 6.1  
2014-07-07 trade_unr:42 INFO -  
2014-07-07 trade_unr:43 INFO future   UNR: 6.2  
2014-07-07 trade_unr:44 INFO -  
2014-07-07 trade_unr:48 INFO unr delta: -0.20000000000000017764  
2014-07-07 trade_unr:49 INFO -  
2014-07-07 trade_unr:52 INFO unr_diff: -0.20000000000000017764  
2014-07-07 trade_unr:53 INFO unr_diff < -0.2: True  
2014-07-07 trade_unr:57 INFO (unr_diff < -0.2) outlook + .5  
2014-07-07 trade_unr:60 INFO Outlook: 0.5

Andrew McKenna

Alright I figured it out and am going to post an answer to my question in case someone else ends up here. Basically it boils down to the fact that the result of 6.1 - 6.3 is a double precision float which is stored as binary. Due to the constraints of storing a number as binary some numbers can't be stored in their exact values. For more information read this: http://floating-point-gui.de/basic/

So the way to deal with this in the case that you know you only want to deal with a number out to the 10th or 100th decimal place you have to round. See below:

$ ipython
Python 2.7.9 (default, Feb 10 2015, 03:29:19)  
Type "copyright", "credits" or "license" for more information.

IPython 2.3.0 -- An enhanced Interactive Python.  
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.  
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: a = 6.1

In [2]: b = 6.3

In [3]: c = a - b

In [4]: c  
Out[4]: -0.20000000000000018

In [5]: d = round(c,2)

In [6]: d  
Out[6]: -0.2

In [7]: c < -0.2  
Out[7]: True

In [8]: d < -0.2  
Out[8]: False

Simon Thornington

I am not sure rounding is robust either though. You could do something like:

def lt(a,b):  
    return ((a<b) & ~(numpy.isclose(a,b))) > 0

Andrew McKenna

Yeah rounding worked in this case because I know the numbers being processed will not be more precise than a 1/10th, but you're right it is not robust.

I talked to a Python guru friend and he recommended just doing fixed point arithmetic and using the decimal python library. https://docs.python.org/2/library/decimal.html

$ ipython
Python 2.7.9 (default, Feb 10 2015, 03:29:19)  
Type "copyright", "credits" or "license" for more information.

IPython 2.3.0 -- An enhanced Interactive Python.  
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.  
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: from decimal import *

In [2]: getcontext().prec = 4

In [3]: a = Decimal(6.1)

In [4]: a  
Out[4]: Decimal('6.0999999999999996447286321199499070644378662109375')

In [5]: b = Decimal(6.3)

In [6]: a - b  
Out[6]: Decimal('-0.2000')

In [7]: c = a - b

In [8]: c < -.2  
Out[8]: False

Simon Thornington

Yup fixed point works well. Most of the data feeds actually send prices and whatnot as fixed decimal.

You've successfully submitted a support ticket.

Our support team will be in touch soon.