## Newton's Method in n Dimensions

1. Problem: Find n-Dimensional Zero of a Vector-Valued Function of a Vector Argument f(x),

f(z) = 0,

where x = [x i]n×1, z = [z i]n×1 and f(x) = [f i(x)]n×1 (Vectors are indicated by boldfaced variables and functions).

2. Taylor Approximation: Letting the (k+1)th iteration be such that x(k+1) = x(k)+delta_x(k) ~ z, so that

0 = f(x(k+1)) = f(x(k)+delta_x(k)) ~ f(x(k)) + J(k)*delta_x(k),

assuming the change |delta_x(k)| is sufficiently small, where

• x(k) is the kth Newton approximation,
• f(k) = f(x(k)) is the kth value of the n-dim. function f,
• J(k) = ([f i,j]n×n)(k) is the kth Jacobian function,
• and f i,j is the jth partial derivative (with respect to x j) of the ith component of f.
3. Conversion to Linear Algebra Problem:

J(k)*delta_x(k) = - f(k),

can be converted to the Augmented Matrix:

["A" | "b" ] = [ J(k) | - f(k) ].

which can be solved by Forward Gaussian Elimination to get an approximation for delta_x(k) which can be used to get the new Newton n-dimensional iterate,

x(k+1) = x(k) + delta_x(k).

4. Numerical Example:

* Intersection of a circle of radius 2 and center (0,0) with a exponential function function y = 1 - ex. So let the component functions be f1(x1, x2) and f2(x1, x2) or alternately f(x,y) and g(x,y) without the subscripts.

* Vector Function with n=2:

[ f(x,y) ] = [ x2 + y2 - 4 ]
[ g(x,y) ] = [ ex + y  - 1 ]

* 2×2 Jacobian J:

[ J11   J11 ] = [ 2x   2y ]
[ J21   J22 ] = [ ex      1]

* Starting Guess (MATLAB Notation):

[x; y](0) = [0; -2];

* 0th Interate Interation: Initial Augmented Matrix using Forward Gaussian Elimination and Virtual Full Pivoting (pr = row pivot vector and pc = column pivot vector): The maximal full pivot is found in the element (1,2) of J(1):

[J | - f || pr | pc ](0) =

[0   -4 | 0 || 1 | 1] -> [ 0   -4.000      | 0 || 1 | 2 ]
[1   +1 | 2 || 2 | 2] -> [ 1   (-0.25)m | 2 || 2 | 1 ]

The multiplier (1/(-4))m = (-0.25)m is calculated and is not really need at this step, except to do the elimination: J22 -> J22 - (-0.25)*J12 = 0, since the two zeros in the second row due not cause a change in J21 and "J23" = - g(0) = +2

Backsubstitution is trivial due to zeros in the pivot (1st) row, yielding

delta_x(0) = [2; 0] => x(1) = x(0) + delta_x(0) = [0; -2] + [2; 0] = [2; -2];

* 1st Iterate Elimination: The first pivot is found in element (2,1) of J(1) after substiting x(1) = [x(1); y(1)] into the augmented matrix function [J(x,y) | -[f(x,y); g(x,y)] ]:

[J | - f || pr | pc ](1) =(4r)

[4.000   -4.000 | -4.000 || 1 | 1] -> [ (0.5413)m   -4.541 | -1.624 || 1 | 2 ]
[7.389   +1.000 | -4.389 || 2 | 2] -> [ 7.389          +1.000 | -4.389 || 2 | 1 ]

The elimination step used was J1,j = J1,j - (0.5413)*J2,j for j=1, 3, where J1,3 is the element of the RHS (Right Hand Side) in the first row.

Backsubstitution yields

delta_x(0) =(4r) [-0.6424; 0.3576]; =>

x(2) = x(1) + delta_x(1) =(4r) [2; -2] + [-0.6424; 0.3576] =(4r) [1.358; -1.642];

* 2nd Iterate Elimination: The first pivot is found in element (2,1), again, of J(2), after substituting x(2) = [x(2); y(2)] into the augmented matrix function:

[J | - f || pr | pc ](2) =(4r)

[2.716   -3.284 | -0.5903|| 1 | 1] -> [ (0.6986)m   -3.983 | +0.3302 || 2 | 1 ]
[3.888   +1.000 | -1.246 || 2 | 2] -> [ 3.888          +1.000 | -1.246    || 1 | 2 ]

The elimination step used was J1,j = J1,j - (0.6986)*J2,j for j=1, 3, where J1,3 is the element of the RHS (Right Hand Side) in the first row.

Backsubstitution yields

delta_x(1) =(4r) [-0.2992; -0.0.08290] =>

x(2) = x(1) + delta_x(1) =(4r) [1.358; -1.642] + [-0.2992; -0.0.08290] =(4r) [1.059; -1.725];

Two norm measures of the closeness of the iterates that might be used for stopping criteria are the infinity (inf or maximal) norms:

||delta_x(1)|| = ||x(2)-x(1)|| =(4r) 0.2992;

||f(1)|| = ||[f(1); g(1)]|| =(4r) 1.246.

For more terms, this procedure can be implemented in MATLAB.

Remarks:

• These calculation were done in 4 digit rounding exam precision as indicated by the modified equal sign "=(4r)".

• Also, this webpage was done in pure HTML, so the spacing and symbols will not be as good as typeset graphics webapage or a text.

• See also Heath's short and different Section 5.6.2 on N-Dimensional Newton's Method. In addition, Gerald and Wheatley in the Section 2.12 on Systems of Nonlinear Equations treats the same example somewhat differently.

Web Source: http://www.math.uic.edu/~hanson/mcs471/newton-ndim.html

Email Comments or Questions to Professor Hanson