Error estimation from nonlinear least squares, singular jacobian

[Clemens]

Hei there,

i have a problem calculating the estimated errors in parameters from a fit:

i fit specific nonlinear function to a dataset. Afterwards i calculate the jacobian of the fitted parameters, however when i want to use nlparci it tells me that the matrix is ill conditioned and taht the matrix is singular to working precision.

Is there any workaround for this? Can i determine the errors from something else, or should i transform, or calculate the jacobian in a specific way?

Best wishes
Clemens

[John D'Errico]

Post by Clemens
Hei there,
i fit specific nonlinear function to a dataset. Afterwards i calculate the jacobian of the fitted parameters, however when i want to use nlparci it tells me that the matrix is ill conditioned and taht the matrix is singular to working precision.
Is there any workaround for this? Can i determine the errors from something else, or should i transform, or calculate the jacobian in a specific way?

A singular matrix here means that you CANNOT solve
your problem in a way that will yield finite confidence
limits. It means that some linear combination of the
parameters is unspecified. Either you have defined the
model poorly, or your data is so poor as to provide no
information.

Sorry, but sometimes there is no magic solution.

John

[Clemens]

Post by John D'Errico

Post by Clemens
Hei there,
i fit specific nonlinear function to a dataset. Afterwards i calculate the jacobian of the fitted parameters, however when i want to use nlparci it tells me that the matrix is ill conditioned and taht the matrix is singular to working precision.
Is there any workaround for this? Can i determine the errors from something else, or should i transform, or calculate the jacobian in a specific way?

A singular matrix here means that you CANNOT solve
your problem in a way that will yield finite confidence
limits. It means that some linear combination of the
parameters is unspecified. Either you have defined the
model poorly, or your data is so poor as to provide no
information.
Sorry, but sometimes there is no magic solution.
John

Dear John,

thank you for your response. Unfortunately i cannot change the model since it is specified by the physics of the process.
However, you seem to know much more about it than i do, could you give me some suggestions what to read for getting a deeper understanding of this error calculation issues (the books i found usually don't deal very strong with estimation of confidence intervals).

thank you in advance.

Best regards
Clemens

[Matt J]

Post by Clemens
thank you for your response. Unfortunately i cannot change the model since it is specified by the physics of the process.

===============

But assuming your data is good, the singularity of the Jacobian is telling you that your model is unfinished. You are not including enough physical information in your model for it to be a good model. Describe the problem mathematically to the community and it may become clear why.

[Bruno Luong]

Post by Matt J
But assuming your data is good, the singularity of the Jacobian is telling you that your model is unfinished.

I disagree with you and John as well. The error estimation by Jacobian (J'*J) is exact for linear model. For non-linear it's only an approximation of the Hessian, and the Hessian alone is in turn not telling the whole story. It "might" indicate the ill-posedness, but not more.

Bruno

[Matt J]

Post by Bruno Luong

Post by Matt J
But assuming your data is good, the singularity of the Jacobian is telling you that your model is unfinished.

I disagree with you and John as well. The error estimation by Jacobian (J'*J) is exact for linear model. For non-linear it's only an approximation of the Hessian, and the Hessian alone is in turn not telling the whole story. It "might" indicate the ill-posedness, but not more.

==========

What else might a singular Hessian indicate?

[Bruno Luong]

Post by Matt J
What else might a singular Hessian indicate?

I don't know. My point is for non-linear model
1) J'*J is NOT the Hessian, and
2) the Hessian alone cannot tell how well or ill the problem is posed

Bruno

[Matt J]

Post by Bruno Luong

Post by Matt J
What else might a singular Hessian indicate?

I don't know. My point is for non-linear model
1) J'*J is NOT the Hessian, and
2) the Hessian alone cannot tell how well or ill the problem is posed

============

That's true, but it is better to have a non-singular Hessian for a variety of reasons, e.g., good convergence, and good modeling should bring this about.

Also, if the Hessian is non-singular, the Jacobian will be as well as the measurement noise goes to zero.

[John D'Errico]

Post by Bruno Luong

Post by Matt J
But assuming your data is good, the singularity of the Jacobian is telling you that your model is unfinished.

I disagree with you and John as well. The error estimation by Jacobian (J'*J) is exact for linear model. For non-linear it's only an approximation of the Hessian, and the Hessian alone is in turn not telling the whole story. It "might" indicate the ill-posedness, but not more.
Bruno

The moon "might" be made of green cheese too,
although all evidence suggests it is not.

Regardless of whence the singularity arises, it DOES
say that no finite estimates will be obtained from the
standard methods. As well, the probability is high (in
my opinion) that there is a problem.

John

[Bruno Luong]

Post by John D'Errico
Regardless of whence the singularity arises, it DOES
say that no finite estimates will be obtained from the
standard methods. As well, the probability is high (in
my opinion) that there is a problem.

Sorry, mathematically, it does NOT.

Consider

f(x) = 1/(x^2+0.1)
y = 10 (= f(0))

The least-squares problem
min ( f(x) - y )

is far from ill-posed. yet f'=0.

Bruno