1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
|
<HTML>
<HEAD>
<TITLE>Method</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<P><A NAME="fitcmethod"></A>
<font size="+2" color="green">Method</font>
</P>
<P>
Suppose that you have <code>N</code> data points, <code>y<sub>k</sub></code>, for <code>k = 1,2,...,N</code>,
and the function to be fitted is <code>f(x,p)</code>, where <code>p</code> represents the <code>M</code>
parameters <code><p<sub>1</sub>,p<sub>2</sub>,...,p<sub>M</sub>></code>. Define the likelihood of
the parameters, given the data, as the probability of the data, given the parameters. We fit for the
parameters, <code>p</code>, by finding those values, <code>p<sub>min</sub></code> that
maximize this likelihood. This form of parameter estimation is known as maximum likelihood estimation.</P>
<P>
Good references on this topic include:
<UL>
<LI> <i>Practical Methods of Optimization</i>,
by R. Fletcher, 1980;</li>
<LI> <i>Methods for Unconstrained Optimization Problems</i> by J. Kowalik and M.R. Osborne, 1968;</li>
<LI> <i>Statistical Methods in Experimental Physics</i>, by W.T. Eadie, et.al., 1971;</li>
<LI> <i>Mathematical Statistics</i>, by John E. Freund, 1971;</li>
<LI> <i>Formulae and Methods in Experimental Data Evaluation, Volume 3,
Elements of Probability and Statistics,</i> by Siegmund Brandt, 1984;</li>
<LI> <i>Numerical Recipes - The Art of Scientific Computing</i>, by W.H. Press, et.al. 1986.</li>
</UL></p>
<p>
Consider the likelihood function
<code><IMG SRC="img16.gif">(p)≡P(x<sub>1</sub>,p)*P(x<sub>2</sub>,p)*...*P(x<sub>N</sub>,p)</code>,
where <i>P</i> is the
probability density, which depends on the random variable <i>x</i> and the parameters
<code>p</code>. <IMG SRC="img16.gif"> is a measure for the
probability to observe just the particular sample we have, and is called
an <i>a-posteriori probability</i> since it is computed after the sampling
is done. The best estimates for <code>p</code> are the values which
maximize <IMG SRC="img16.gif">. But maximizing the logarithm of
<IMG SRC="img16.gif"> also maximizes <IMG SRC="img16.gif">, and
maximizing <code>ln(<IMG SRC="img16.gif">)</code> is equivalent to minimizing
<code>-ln(<IMG SRC="img16.gif">)</code>. So, the goal becomes minimizing the log likelihood function:</P>
<P>
<IMG SRC="img19.gif"></P>
<P>
Let <code>p<sup>0</sup></code> be the initial values given for <code>p</code>.
The goal is to find a <code>∇p</code> so that <code>p<sup>1</sup> = p<sup>0</sup>+∇p</code>
is a better approximation to the data. We use the iterative Gauss-Newton method, and the series
<code>p<sup>1</sup>,p<sup>2</sup>,p<sup>3</sup>,...</code> will hopefully converge to the minimum,
<code>p<sub>min</sub></code>.</P>
<P>
Generally, the Gauss-Newton method is locally convergent when χ<sup>2</sup> is
zero at the minimum. Serious difficulties arise when <code>f</code> is sufficiently
nonlinear and χ<sup>2</sup> is large at the minimum. The Gauss-Newton method has the
advantage that linear least squares problems are solved in one iteration.</p>
<P>
Consider the Taylor expansion of <code><i>L</i>(p)</code>:</P>
<P>
<center><IMG SRC="img26.gif"></center></P>
<P>
Define the arrays <IMG SRC="img27.gif">, <IMG SRC="img28.gif"> and <IMG SRC="img29.gif">:</P>
<P>
<center><IMG SRC="img30.gif"></center></P>
<P>
If we linearize, i.e., assume that<br />
<center><IMG SRC="img31.gif"></center><br />
then<br />
<center><IMG SRC="img32.gif"></center><br />
and so<br />
<center><IMG SRC="img33.gif"></center><br />
The problem has reduced to solving the matrix equation</p>
<p>
<center><IMG SRC="img34.gif"></center></P>
<P>
<EM>Note</EM>: The partial derivatives are approximated numerically using a central
difference approximation:</P>
<P>
<IMG SRC="img35.gif"></P>
<P>
<a href="fitform.htm"><img src="../shadow_left.gif">
<font size="+1" color="olive">Graphical user interface</font></a><br />
<a href="tolerance.htm"><img src="../shadow_right.gif">
<font size="+1" color="olive">Tolerance</font></a>
</P>
</body>
</html>
|
