Convergence Analysis of the Secant Method

The convergence of secant method is "superlinear", which is almost quadratic. For the proof of this result, see note in pdf format or (better for printing) postscript version of the note.

Newton's Method

We defined Newton's method as a secant method, where we take the limit of the two previous points colliding. Geometrically, we take the next approximation as the intersection of the x-axis with the tangent line at the current approximation.

Below is a simple MATLAB m-file to solve sin(x) = 0:

function [x,fail] = newton(f,df,x,eps,N)
%
%  Applies Newton's method to the function f, with derivative in df,
%  starting at the point x.  It produces a sequences of points.
%  Stops when the sequence reaches the point x where 
%  |f(x)| < eps or when two consecutive points in the sequence
%  have distance less than eps apart from each other.
%  Failure is reported (fail = 1) when the accuracy requirement
%  is not satisfied in N steps; otherwise fail = 0 on return.
%
%  Example : 
%  >> [x,fail] = newton('sin','cos',pi/4,1e-8,10)
%
fprintf('running the method of Newton...\n');
fx = feval(f,x);
for i = 1:N 
   dx = fx/feval(df,x); 
   x = x - dx;
   fx = feval(f,x);
   fprintf('  x = %e, dx = %e, f(x) = %e\n', x, dx, fx);
   if (abs(fx) < eps) | (abs(dx) < eps)
      fail = 0;
      fprintf('succeeded after %d steps\n', i);
      return;
   end
end
fprintf('failed requirements after %d steps\n', N);
fail = 1;

The output of this algorithm on sin(x) = 0 starting at pi/4 is

>>  [x,fail] = newton('sin','cos',pi/4,1e-8,10)
running the method of Newton...
  x = -2.146018e-01, dx = 1.000000e+00, f(x) = -2.129584e-01
  x = 3.356262e-03, dx = -2.179581e-01, f(x) = 3.356256e-03
  x = -1.260225e-08, dx = 3.356274e-03, f(x) = -1.260225e-08 
  x = 1.654361e-24, dx = -1.260225e-08, f(x) = 1.654361e-24 
succeeded after 4 steps

x =

   1.6544e-24


fail =

     0

Observe the fast convergence.

We can interpret Newton's method as a fixed-point iteration method. This leads to "cobweb pictures". The m-file below produces such a picture with MATLAB:

function [x,fail] = cobweb(g,x,eps,N,xmin,xmax,ymin,ymax)
%
%  On entry :
%    g         function defining the x = g(x);
%    x         start point for the fixed-point iteration;
%    eps       accuracy requirement: stops when two consecutive
%              approximations in the sequence are
%    N         maximal number of allowed iterations;
%    xmin      minimal value for x to view the graph;
%    xmax      maximal value for x to view the graph;
%    ymin      minimal value for g(x) to view the graph;
%    ymax      maximal value for g(x) to view the graph.
%
%  On return :
%    x         last approximation in the sequence;
%    fail      0 means success, 1 means failure.
%
%  Example : 
%    >> cobweb('myfun',0.3,1.0e-6,10,0.1,10,0,10)
%
fprintf('illustration of fixed-point iteration...\n');
xr = xmin:0.1:xmax;           % sample range to plot
yr = feval(g,xr);             % evaluate at the samples
figure;                       % new figure window
axis([xmin xmax ymin ymax]);  % adjust the scale
hold on;                      % overlay plot
plot(xr,yr);                  % plot the function y = g(x)
plot(xr,xr);                  % plot the diagonal y = x
for i = 1:N                   % perform N iterations                
  y = feval(g,x);             % y = g(x)
  plot([x x],[x y],'g--');    % vertical line
  plot([x y],[y y],'r--');    % horizontal line
  fprintf('  x = %e, y = %e, |x-y| = %e\n', x, y, abs(x-y));
  if (abs(x-y) < eps)
     fprintf('succeeded after %d steps\n', i);
     x = y;
     fail = 0;
     return;                  % terminate with success
  end;
  x = y;                    
end;
fail = 1;                     % terminate with failure

where the content of myfun.m is

function y = myfun(x)
%
%  Returns the function value y = (x^2 + 5)/(2*x).
%  The dots before the exponentiation and division operator below
%  enforce the operation to be carried out componentwise.
%  These dots are necessary when x is a vector.
%
y = (x.^2 + 5)./(2*x); 

Note that with myfun is obtained from applying Newton's method on the equation x^2 - 5 = 0.

The nongraphical output is

>> cobweb('myfun',0.3,1.0e-6,10,0.1,10,0,10)
illustration of fixed-point iteration...
  x = 3.000000e-01, y = 8.483333e+00, |x-y| = 8.183333e+00
  x = 8.483333e+00, y = 4.536362e+00, |x-y| = 3.946971e+00
  x = 4.536362e+00, y = 2.819283e+00, |x-y| = 1.717079e+00
  x = 2.819283e+00, y = 2.296392e+00, |x-y| = 5.228916e-01
  x = 2.296392e+00, y = 2.236860e+00, |x-y| = 5.953157e-02
  x = 2.236860e+00, y = 2.236068e+00, |x-y| = 7.921835e-04
  x = 2.236068e+00, y = 2.236068e+00, |x-y| = 1.403255e-07
succeeded after 7 steps

ans =

    2.2361

>> ans^2

ans =

    5.0000

>> 

The picture we see is