Next: Bounded minimization (optimize.fminbound) Up: Optimization (optimize) Previous: Hessian product example:

Least-square fitting (minimize.leastsq)

All of the previously-explained minimization procedures can be used to solve a least-squares problem provided the appropriate objective function is constructed. For example, suppose it is desired to fit a set of data \( \left\{ \mathbf{x}_{i},\mathbf{y}_{i}\right\} \) to a known model, \( \mathbf{y}=\mathbf{f}\left( \mathbf{x},\mathbf{p}\right) \) where \( \mathbf{p} \) is a vector of parameters for the model that need to be found. A common method for determining which parameter vector gives the best fit to the data is to minimize the sum of squares of the residuals. The residual is usually defined for each observed data-point as

\begin{displaymath}
e_{i}\left( \mathbf{p},\mathbf{y}_{i},\mathbf{x}_{i}\right) ...
...\mathbf{f}\left( \mathbf{x}_{i},\mathbf{p}\right) \right\Vert .\end{displaymath}

An objective function to pass to any of the previous minization algorithms to obtain a least-squares fit is.

\begin{displaymath}
J\left( \mathbf{p}\right) =\sum _{i=0}^{N-1}e_{i}^{2}\left( \mathbf{p}\right) .\end{displaymath}

The leastsq algorithm performs this squaring and summing of the residuals automatically. It takes as an input argument the vector function \( \mathbf{e}\left( \mathbf{p}\right) \) and returns the value of \( \mathbf{p} \) which minimizes \( J\left( \mathbf{p}\right) =\mathbf{e}^{T}\mathbf{e} \) directly. The user is also encouraged to provide the Jacobian matrix of the function (with derivatives down the columns or across the rows). If the Jacobian is not provided, it is estimated.

An example should clarify the usage. Suppose it is believed some measured data follow a sinusoidal pattern

\begin{displaymath}
y_{i}=A\sin \left( 2\pi kx_{i}+\theta \right) \end{displaymath}

where the parameters \( A, \) \( k \), and \( \theta \) are unknown. The residual vector is

\begin{displaymath}
e_{i}=\left\vert y_{i}-A\sin \left( 2\pi kx_{i}+\theta \right) \right\vert .\end{displaymath}

By defining a function to compute the residuals and (selecting an appropriate starting position), the least-squares fit routine can be used to find the best-fit parameters \( \hat{A},  \hat{k},  \hat{\theta } \). This is shown in the following example and a plot of the results is shown in Figure 1.


>>> x = arange(0,6e-2,6e-2/30)
>>> A,k,theta = 10, 1.0/3e-2, pi/6
>>> y_true = A*sin(2*pi*k*x+theta)
>>> y_meas = y_true + 2*randn(len(x))

>>> def residuals(p, y, x):
        A,k,theta = p
        err = y-A*sin(2*pi*k*x+theta)
        return err

>>> def peval(x, p):
        return p[0]*sin(2*pi*p[1]*x+p[2])

>>> p0 = [8, 1/2.3e-2, pi/3]
>>> print array(p0)
[  8.      43.4783   1.0472]

>>> from scipy.optimize import leastsq
>>> plsq = leastsq(residuals, p0, args=(y_meas, x))
>>> print plsq[0]
[ 10.9437  33.3605   0.5834]

>>> print array([A, k, theta])
[ 10.      33.3333   0.5236]

>>> from scipy.xplt import *    # Only on X-windows systems
>>> plot(x,peval(x,plsq[0]),x,y_meas,'o',x,y_true)
>>> title('Least-squares fit to noisy data')
>>> legend(['Fit', 'Noisy', 'True'])
>>> gist.eps('leastsqfit')   # Make epsi file.

Figure 1: Least-square fitting to noisy data using scipy.optimize.leastsq
\resizebox*{0.5\textwidth}{!}{\includegraphics{leastsqfit.epsi}}



Next: Bounded minimization (optimize.fminbound) Up: Optimization (optimize) Previous: Hessian product example:
2001-07-27