Skip to content

Bug: "ValueError: array must not contain infs or NaNs" #23

@bmorris3

Description

@bmorris3

Some values of input parameters give the following intimidating, cryptic and uninformative error:

(base) tycho:dot bmmorris$ python -c "from dot.tests.test_model import test_model_optimizer as f; f()"
optimizing logp for variables: [dot_R_spot, dot_lat, dot_lon, dot_comp_inc, dot_shear, dot_P_eq, dot_f0]
264it [00:06, 38.29it/s, logp=-6.690179e+04]array: [-0.15787993 -0.52647481 -0.84561983 -2.19530702 -0.25451239 -0.74975759
  0.59741809 -0.52742003 -0.4755868   0.75138306]
point: {'dot_f0_interval__': array(0.75138306), 'dot_P_eq_interval__': array(-0.4755868), 'dot_shear_interval__': array(-0.52742003), 'dot_comp_inc_interval__': array(0.59741809), 'dot_lon_interval__': array([[-0.25451239, -0.74975759]]), 'dot_lat_interval__': array([[-0.84561983, -2.19530702]]), 'dot_R_spot_interval__': array([[-0.15787993, -0.52647481]])}
Traceback (most recent call last):
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/compile/function_module.py", line 903, in __call__
    self.fn() if output_subset is None else\
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/gof/op.py", line 892, in rval
    r = p(n, [x[0] for x in i], o)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/tensor/slinalg.py", line 250, in perform
    A, b, lower=True)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/linalg/basic.py", line 336, in solve_triangular
    b1 = _asarray_validated(b, check_finite=check_finite)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/_lib/_util.py", line 239, in _asarray_validated
    a = toarray(a)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/numpy/lib/function_base.py", line 486, in asarray_chkfinite
    "array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/exoplanet/utils.py", line 197, in objective
    *get_args_for_theano_function(bij.rmap(vec), model=model)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/compile/function_module.py", line 917, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/gof/link.py", line 325, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/compile/function_module.py", line 903, in __call__
    self.fn() if output_subset is None else\
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/gof/op.py", line 892, in rval
    r = p(n, [x[0] for x in i], o)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/tensor/slinalg.py", line 250, in perform
    A, b, lower=True)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/linalg/basic.py", line 336, in solve_triangular
    b1 = _asarray_validated(b, check_finite=check_finite)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/_lib/_util.py", line 239, in _asarray_validated
    a = toarray(a)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/numpy/lib/function_base.py", line 486, in asarray_chkfinite
    "array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs
Apply node that caused the error: Solve{A_structure='lower_triangular', lower=False, overwrite_A=False, overwrite_b=False}(TensorConstant{[[1.287538..6413e-03]]}, Elemwise{Composite{((i0 + i1 + i2) - i3)}}[(0, 2)].0)
Toposort index: 65
Inputs types: [TensorType(float64, matrix), TensorType(float64, col)]
Inputs shapes: [(1418, 1418), (1418, 1)]
Inputs strides: [(8, 11344), (8, 8)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[InplaceDimShuffle{1,0}(Solve{A_structure='lower_triangular', lower=False, overwrite_A=False, overwrite_b=False}.0), Elemwise{mul,no_inplace}(TensorConstant{(1, 1) of 2.0}, TensorConstant{(1, 1) of 0.5}, Solve{A_structure='lower_triangular', lower=False, overwrite_A=False, overwrite_b=False}.0)]]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/distribution.py", line 84, in __new__
    return model.Var(name, dist, data, total_size, dims=dims)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/model.py", line 1118, in Var
    model=self,
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/model.py", line 1744, in __init__
    self.logp_sum_unscaledt = distribution.logp_sum(data)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/distribution.py", line 162, in logp_sum
    return tt.sum(self.logp(*args, **kwargs))
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 328, in logp
    quaddist, logdet, ok = self._quaddist(value)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 113, in _quaddist
    dist, logdet, ok = self._quaddist_cov(delta)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 138, in _quaddist_cov
    return self._quaddist_chol(delta)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 132, in _quaddist_chol
    delta_trans = self.solve_lower(chol_cov, delta.T).T

HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
Traceback (most recent call last):
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/compile/function_module.py", line 903, in __call__
    self.fn() if output_subset is None else\
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/gof/op.py", line 892, in rval
    r = p(n, [x[0] for x in i], o)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/tensor/slinalg.py", line 250, in perform
    A, b, lower=True)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/linalg/basic.py", line 336, in solve_triangular
    b1 = _asarray_validated(b, check_finite=check_finite)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/_lib/_util.py", line 239, in _asarray_validated
    a = toarray(a)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/numpy/lib/function_base.py", line 486, in asarray_chkfinite
    "array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/bmmorris/git/dot/dot/tests/test_model.py", line 24, in test_model_optimizer
    map_soln = m.optimize()
  File "/Users/bmmorris/git/dot/dot/model.py", line 309, in optimize
    map_soln = optimize(start=start, **kwargs)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/exoplanet/utils.py", line 218, in optimize
    info = minimize(objective, x0, **kwargs)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/_minimize.py", line 594, in minimize
    return _minimize_bfgs(fun, x0, args, jac, callback, **options)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/optimize.py", line 1016, in _minimize_bfgs
    old_fval, old_old_fval, amin=1e-100, amax=1e100)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/optimize.py", line 836, in _line_search_wolfe12
    **kwargs)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/linesearch.py", line 101, in line_search_wolfe1
    c1=c1, c2=c2, amax=amax, amin=amin, xtol=xtol)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/linesearch.py", line 175, in scalar_search_wolfe1
    phi1 = phi(stp)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/linesearch.py", line 87, in phi
    return f(xk + s*pk, *args)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/optimize.py", line 326, in function_wrapper
    return function(*(wrapper_args + args))
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/optimize.py", line 64, in __call__
    fg = self.fun(x, *args)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/exoplanet/utils.py", line 197, in objective
    *get_args_for_theano_function(bij.rmap(vec), model=model)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/compile/function_module.py", line 917, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/gof/link.py", line 325, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/compile/function_module.py", line 903, in __call__
    self.fn() if output_subset is None else\
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/gof/op.py", line 892, in rval
    r = p(n, [x[0] for x in i], o)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/tensor/slinalg.py", line 250, in perform
    A, b, lower=True)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/linalg/basic.py", line 336, in solve_triangular
    b1 = _asarray_validated(b, check_finite=check_finite)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/_lib/_util.py", line 239, in _asarray_validated
    a = toarray(a)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/numpy/lib/function_base.py", line 486, in asarray_chkfinite
    "array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs
Apply node that caused the error: Solve{A_structure='lower_triangular', lower=False, overwrite_A=False, overwrite_b=False}(TensorConstant{[[1.287538..6413e-03]]}, Elemwise{Composite{((i0 + i1 + i2) - i3)}}[(0, 2)].0)
Toposort index: 65
Inputs types: [TensorType(float64, matrix), TensorType(float64, col)]
Inputs shapes: [(1418, 1418), (1418, 1)]
Inputs strides: [(8, 11344), (8, 8)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[InplaceDimShuffle{1,0}(Solve{A_structure='lower_triangular', lower=False, overwrite_A=False, overwrite_b=False}.0), Elemwise{mul,no_inplace}(TensorConstant{(1, 1) of 2.0}, TensorConstant{(1, 1) of 0.5}, Solve{A_structure='lower_triangular', lower=False, overwrite_A=False, overwrite_b=False}.0)]]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/distribution.py", line 84, in __new__
    return model.Var(name, dist, data, total_size, dims=dims)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/model.py", line 1118, in Var
    model=self,
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/model.py", line 1744, in __init__
    self.logp_sum_unscaledt = distribution.logp_sum(data)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/distribution.py", line 162, in logp_sum
    return tt.sum(self.logp(*args, **kwargs))
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 328, in logp
    quaddist, logdet, ok = self._quaddist(value)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 113, in _quaddist
    dist, logdet, ok = self._quaddist_cov(delta)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 138, in _quaddist_cov
    return self._quaddist_chol(delta)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 132, in _quaddist_chol
    delta_trans = self.solve_lower(chol_cov, delta.T).T

HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
265it [00:09, 29.26it/s, logp=-6.690179e+04]

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions