Newton’s method


Let 𝐟 be a differentiable function from ℝn to ℝn. Newton’s method consists of starting at an 𝐚𝟎 for the equation 𝐟⁒(𝐱)=𝟎. Then the function is linearized at 𝐚𝟎 by replacing the increment 𝐟⁒(𝐱)-𝐟⁒(𝐚𝟎) by a linear functionMathworldPlanetmath of the increment [𝐃⁒(𝐚𝟎)]⁒(𝐱-𝐚𝟎).

Now we can solve the linear equation 𝐟⁒(𝐚𝟎)+[𝐃⁒(𝐚𝟎)]⁒(𝐱-𝐚𝟎)=𝟎. Since this is a system of n linear equations in n unknowns, [𝐃⁒(𝐚𝟎)]⁒(𝐱-𝐚𝟎)=-𝐟⁒(𝐚𝟎) can be likened to the general linear system A⁒𝐱=𝐛.

Therefore, if [𝐃⁒(𝐚𝟎)] is invertiblePlanetmathPlanetmathPlanetmath, then 𝐱=𝐚𝟎-[𝐃⁒(𝐚𝟎)]-1⁒𝐟⁒(𝐚𝟎). By renaming 𝐱 to 𝐚𝟏, you can reiterate Newton’s method to get an 𝐚𝟐. Thus, Newton’s method states

𝐚n+1=𝐚n-[πƒπŸβ’(𝐚n)]-1⁒𝐟⁒(𝐚n)

Thus we get a series of πšβ€™s that we hope will converge to 𝐱 such that 𝐟⁒(𝐱)=𝟎. When we solve an equation of the form 𝐟⁒(𝐱)=𝟎, we call the solution a root of the equation. Thus, Newton’s method is used to find roots of nonlinear equations.

Unfortunately, Newton’s method does not always converge. There are tests for neighborhoodsMathworldPlanetmath of πšπŸŽβ€™s where Newton’s method will converge however. One such test is Kantorovitch’s theoremMathworldPlanetmath, which combines what is needed into a concise mathematical equation.

Corollary 1: Newton’s Method in one dimensionPlanetmathPlanetmath - The above equation is simplified in one dimension to the well-used

a1=a0-f⁒(a0)f′⁒(a0)

This intuitively cute equation is pretty much the equation of first year calculus. :)

Corollary 2: Finding a square root - So now that you know the equation, you need to know how to use it, as it is an algorithmMathworldPlanetmath. The construction of the primary equation, of course is the important part. Let’s see how you do it if you want to find a square root of a number b.

We want to find a number x (x for unknown), such that x2=b. You might think β€œwhy not find a number such that x=b ?” Well, the problem with that approach is that we don’t have a value for b, so we’d be right back where we started. However, squaring both sides of the equation to get x2=b lets us work with the number we do know, b.) Back to x2=b. With some manipulation, we see this means that x2-b=0 ! Thus we have our f⁒(x)=0 scenario.

We can see that f′⁒(x)=2⁒x thus, f′⁒(a0)=2⁒a0 and f⁒(a0)=a02-b. Now we have all we need to carry out Newton’s method. By renaming x to be a1, we have

a1=a0-12⁒a0⁒(a02-b)=12⁒(a0+ba0)

.

The equation on the far right is also known as the divide and averageMathworldPlanetmath method, for those who have not learned the full Newton’s method, and just want a fast way to find square roots. Let’s see how this works out to find the square root of a number like 2:

Let x2=2

x2-2=0=f⁒(x)

Thus, by Newton’s method,…

a1=a0-a02-22⁒a0

All we did was plug in the expressions f⁒(a0) and f′⁒(a0) where Newton’s method asks for them. Now we have to pick an a0. Hmm, since

1<2<4
1<2<2

let’s pick a reasonable number between 1 and 2 like 1.5

a1=1.5-1.52-22⁒(1.5)
a1=1.41⁒6¯

Looks like our guess was too high. Let’s see what the next iterationMathworldPlanetmath says

a2=1.41⁒6¯-1.41⁒6¯2-22⁒(1.41⁒6¯)
a2=1.414215686⁒…

getting better =) You can use your calculator to find that

2=1.414213562⁒…

Not bad for only two iterations! Of course, the more you iterate, the more decimal places your an will be accurate to. By the way, this is also how your calculator/computer finds square roots!

Geometric InterpretationMathworldPlanetmathPlanetmath: Consider an arbitrary function f:ℝ→ℝ such as f⁒(x)=x2-b. Say you wanted to find a root of this function. You know that in the neighborhood of x=a0, there is a root (Maybe you used Kantorovitch’s theorem or tested and saw that the function changed signs in this neighborhood). We want to use our knowledge a0 to find an a1 that is a better approximation to x0 (in this case, closer to it on the x-axis).

So we know that x0≀a1≀a0 or in another case a0≀a1≀x0. What is an efficient algorithm to bridge the gap between a0 and x0 ? Let’s look at a tangent line to the graph.

Note that the line intercepts the x-axis between a0 and x0, which is exactly what we want. The slope of this tangent line is f′⁒(a0) by definition of the derivativeMathworldPlanetmathPlanetmath at a0, and we know one point on the line is (a1,0), since that is the x-intercept. That is all we need to find the formulaMathworldPlanetmathPlanetmath of the line, and solve for a1.

y-y1 =m⁒(x-x1)
f⁒(a0)-0 =f′⁒(a0)⁒(a0-a1) Substituting
f⁒(a0)f′⁒(a0) =a0-a1
-a1 =-a0+f⁒(a0)f′⁒(a0) Aesthetic change. Flipped the equation around.
a1 =a0-f⁒(a0)f′⁒(a0) Newton’s method!
Title Newton’s method
Canonical name NewtonsMethod
Date of creation 2013-03-22 11:58:05
Last modified on 2013-03-22 11:58:05
Owner alozano (2414)
Last modified by alozano (2414)
Numerical id 28
Author alozano (2414)
Entry type Algorithm
Classification msc 49M15
Synonym Newton-Raphson method
Related topic LipschitzCondition
Related topic KantorovitchsTheorem
Related topic Derivative
Related topic Superconvergence
Related topic FixedPointsOfNormalFunctions