Download Chapter 3: Systems of Linear Equations

Chapter: 3c
System of Linear Equations
Dr. Asaf Varol
1
Pivoting
Some disadvantages of Gaussian elimination are as follows: Since each result
follows and depends on the previous step, for large systems the errors introduced due to
round off (or chop off) errors lead to loss of significant figures and hence to in accurate
results. The error committed in any one step propagates till the final step and it is
amplified. This is especially true for ill-conditioned systems. Of course if any of the
diagonal elements is zero, the method will not work unless the system is rearranged so
to avoid zero elements being on the diagonal. The practice of interchanging rows with
each other so that the diagonal elements are the dominant elements is called partial
pivoting. The goal here is to put the largest possible coefficient along the diagonal by
manipulating the order of the rows. It is also possible to change the order of variables,
i.e. instead of letting the unknown vector
{X}T = (x, y, z) we may let it be {X}T = ( y, z, x)
When this is done in addition to partial pivoting, this practice is called full
pivoting. In this case only the meaning of each variable changes but the system of
equations remain the same.
2
Example E3.4.2
Consider the following set of equations
0.0003 x + 3.0000 y = 2.0001
1.0000 x + 1.0000 y = 1.0000
The exact solution to which is x = 1/3 and y = 2/3
Solving this system by Gaussian elimination with a three significant figure
mantissa yields
x = -3.33, y = 0.667
with four significant figure mantissa yields
x = 0.0000 and y = 0.6667
3
Example E3.4.2 (continued)
Partial pivoting (switch the rows so that the diagonal elements are largest)
1.0000 x + 1.0000 y = 1.0000
0.0003 x + 3.0000 y = 2.0001
The solution of this set of equations using Gaussian elimination gives
y = 0.667 and x = 0.333 with three significant figure arithmetic
y = 0.6667 and x = 0.3333 with four significant figure arithmetic
4
Example E3.4.3
Problem: Apply full pivoting to the following system to achieve a well conditioned
matrix.
3x + 5y - 5z = 3
2x - 4y - z = -3
6x - 5y + z = 2
3 5  5  x   3 

   
2

4

1

  y    3
6  5 1   z   2 
5
Example E3.4.3 (continued)
Solution: First switch the first column with the second column, then switch the
first column with the third column to obtain
-5z + 3x + 5y = 3
- z + 2x - 4y = -3
z + 6x - 5y = 2
Then switch second row with the third row
-5z + 3x + 5y = 3
z + 6x - 5y = 2
-z + 2x - 4y = -3
Yielding finally a well conditioned system given by
 5 3 5   z   3 
 1 6  5 x    2 

   
  1 2  4  y  3
6
Gauss – Jordan Elimination
This method is very similar to Gaussian elimination method. The only difference is
in that the elimination procedure is extended to the upper diagonal elements so that
a backward substitution is no longer necessary. The elimination process begins
with the augmented matrix, and continued until the original matrix turns into an
identity matrix, of course with necessary modifications to the right hand side.
In short our goal is to start with the general augmented system and arrive at the
right side after appropriate algebraic manipulations.
Start
a aa
a
 21
a 31
a 12
a 22
a 32
arrive
a 13  c 1 
 
a 23 c 2 
a 33  c 3 
===>
1
0

0
0
1
0
0c *1 
 
0 c *2 
1 c *3 
The solution can be written at once as
x1  c1* ; x 2  c *2 ; x 3  c *3
7
Pseudo Code for Gauss – Jordan Elimination
do for k = 1 to n
do for j= k+1 to n+1
akj = akj/akk
end do
! Important note
! a(i,n+1) represents the
! the right hand side
do for i = 1 to n ; i is not equal to k
do for j = k+1 to n+1
aij = aij - (aik)(akj)
end do
end do
end do
cc----- The solution vector is saved in a(i,n+1), i=1, to n
8
Example E3.4.4
Problem: Solve the problem of Equations (3.4.3) using Gauss-Jordan elimination
4x1 + x2 + x3 = 6
-x1 - 5x2 + 6x3 = 0
2x1 - 4x2 + x3 = -1
Solution: To do that we start with the augmented the coefficient matrix
 A
(0)
a
4
  1
 2
1
5
1
6
4
1
6

0
1
(i) First multiply the first row by 1/4, then multiply it by -1 and subtract the result from
the second row and replace the result of the subtraction with the second row. Similarly
multiply the first row by 2 and subtract the result from the third row and replace the third
row with the result of the subtraction. These operations lead to
A 
(1)
a
1.0 0.25 0.25 1.5 
  0  4.75 6.25 1.5 


 0  4.50 0.50  4
9
Example E3.4.4 (continued)
(ii) Multiply the second row by -1/4.75, then multiply the second row by 0.25 and subtract the
result from the first row and replace the result with the first row. Similarly multiply the second
row by -4.5 and subtract the result from the third row and replace the result with the third row
to obtain
A 
(2)
a
1 0 0.5789 1.5789 
 0 1  1.3158  0.3158
0 0  5.4211  5.4211
(iii) Multiply the third row by -1/5.4211, then multiply the third row by 0.5789 and subtract the
result from the first row, and then replace the first row by the result. Similarly multiply the third
row by -1.3158 and subtract the result from the second row, and then replace the second row by
the result to finally arrive at
A 
( 3)
a
1

 0
0
0
1
0

0 10000
.

0 10000
.


1 10000
.
Hence we found the expected solution x1=1.0, x2=1.0, and x3 = 1.0. Note, however, that the
solutions of 1.00 were just the solutions. Of course, each matrix will have its own solutions, and 10
not always equal to one!
Finding the Inverse using Gauss-Jordan Elimination
The inverse, [B] of a matrix, [A], is defined such that
[A][B] = [B][A] = [I]
where [I] is the identity matrix. In other words, to find the inverse of a matrix we need to
find the elements of the matrix [B] such that when [A] is multiplied by [B] the result
should equal the identity matrix. If we recall what matrix multiplication is, this amounts to
the same thing as saying the first column of [B] multiplied by [A] must be equal to the first
column of [I] ; the second column of [B] multiplied by [A] must be equal to the second
column of [I] and so on.
For summary, using Gauss-Jordan elimination
 4 1 1 | 1 0 0
 1 0 0 | 0.185 - 0.04854 0.1068 
- 1 - 5 6 | 0 1 0   0 1 0 | 0.1262 0.01942 - 0.2427

 We


obtain
 2 - 4 1 | 0 0 1 
 0 0 1 | 0.1359 0.17476 - 0.1845 
11
LU Decomposition
Lower and Upper (LU) triangular decomposition technique is one of the most widely used
techniques used for solution of linear systems of equations due to its generality and
efficiency. This method calls for first decomposing a given matrix into a product of a
lower and an upper triangular matrices such that
[A] = [L][U]
Given that
[A]{X} = {R}
[U]{X} = {Q}
for an unknown vector {Q} The additional unknown vector {Q} can be found from
[L]{Q} = {R}
The solution procedure (i.e. the algorithm) can now be summarized as
(i)
Determine [L] and [U] for a given system of equations [A]{X} = {R}
(ii)
Calculate {Q} from [L]{Q} = {R} by forward substitution
(iii)
Calculate {X} from [U]{X} = {Q} by backward substitution
12
Example E3.4.5
2
[A] =  1
 3
5
3
4
1
1 ;
2 
12 
 
{R} =  8
 16 
 
To find the solution we start with
2 q1
-q1
3q1
2
[L] =  1
 3
0
1/ 2
7/2
0
0 ;
4
1
[U] = 0
0
5 / 2
1
0
1 / 2
1 
1 
[L]{Q} = {R}; that is
+0
+0
= 12
+ q2/2 + 0
= -8
+ 7q2/2+ 4q3 = 16
We compute {Q} by forward substitution
q1 = 12/2 = 6
q2 = (-8 + q1)/(1/2) = -4
q3 = (16 - 3q1 - 7q2/2)/4 = 3
Hence {Q}T = {6, -4, 3}
Then we solve [U]{X} = {Q} by backward substitution
x1 - (5/2) x2
0
+ x2
0
+ 0
+(1/2) x3
x3
+
x3
x3 = 3
x2 = -4 + x3 = -1
Hence the final solution is
=6
= -4
=3
x1 = 6 + (5/2) x2 -(1/2) x3 = 2
{X}T = { 2, -1, 3 }
13
Crout Decomposition
We illustrate Crout-decomposition in detail for a 3x3 general matrix
[L] [U] = [A]
l 11
l
 21
l 31
0
l 22
l 32
0
0 
l 33 
1
0

0
u 12
1
0
u 13 
a 11

u 23  = a 21
a 31
1 
a 12
a 22
a 32
a 13 
a 23 
a 33 
Multiplying [L] by [U] and equating the corresponding elements of the product with that of the matrix
[A] we get the following equations
l11 = a11 ; l21 = a21 ; l31 = a31 ;
in index notation: li1 = ai1, I=1,2, ... , n (j=1, first column)
(The first column of [L] is equal to the first column of [A]
equation
l11 u12 = a12
l11 u13 = a13
solution
====>
====>
in index notation
l21 u12 + l22 = a22
l31 u12 + l32 = a32
====>
====>
u12 = a12/l11
u13 = a13/l11
u1j = a1j/l11
j=2, 3, ... , n (i=1, first row)
l22 = a22 - l21 u12
l32 = a32 - l31 u12
14
in index notation
li2 = ai2 - li1 u12 ; i=2,3, ... , n (j=2, second column)
Pseudo Code for LU-Decomposition
cc--- Simple coding
do for i = 1 to n
li1 = ai1
end do
c--- Forward substitution
q11 = r1 / l11
do for i = 2 to n
i 1
qi = ri - ( j1l ijq j )/lii
do for j = 2 to n
u1j = a1j/l11
end do
end do
c--- Back substitution
xn = qn
do for j = 2 to n-1
j1
ljj = ajj - k1l jk u kj
do for i = n-1 to 1n
xi = qi - ji1 u ij x j
do for i = j+1 tojn1
lij = aij -  l ik u kj
end do
k 1
j1
uji = aji - (
 l jk u ki
k 1
)/ljj
end do
end do
n 1
lnn = ann -
 l nk u kn
k 1
15
Cholesky Decomposition for Symmetric Matrices
For symmetric matrices the LU-decomposition can be further simplified to take advantage
of the symmetric property that
[A] = [A]T ; or aij = aji
(3.4.9)
Then, it follows that
[A] = [L][U] = [A]T = ([L][U])T = [U]T [L]T
That is
[L] = [U]T ; [U] = [L]T
or in index notation
uij = lji
(3.4.10)
(3.4.11)
It should be noted that the diagonal elements of [U] are no longer, necessarily, equal to
one. Comment: This algorithm should be programmed using complex arithmetic.
16
Tridiagonal Matrix Algorithm (TDMA),
also known as Thomas Algorithm
 b1
a
2
[A] = 
0

0
c1
0
b2
c2
a3
b3
0
a4
0
0

c3 

b4 
If we apply LU-decomposition to this matrix taking into account of its special structure and let
 f1
e
 2
[L] =  0

0
0
0
f2
0
e3
f3
0
e4
0
0 
0

f4 
1
0
[U] = 
0

0
g1
0
1
g2
0
1
0
0
0
0 
g3 

1
By multiplying [L] and [U] and equating the corresponding elements of the product to that of the original matrix
[A] it can be shown that
f1 = b1
g1 = c1/f1
e2 = a2
and for k=2,3, ... , n-1
ek = ak
fk = bk - ek gk-1
gk = ck / fk
further
en = an
fn = bn - en gn-1
17
TDMA Algorithm
c----- Decomposition
c1 = c1/b1
do for k=2 to n-1
bk = bk - ak ck-1
ck = ck/bk
end do
bn = bn - an cn-1
c----- (note here rk ; k=1,2,3, ..., n is the right hand side of the equations)
c----- Forward substitution
r1 = r1/b1
do for k = 2, n
rk = (rk - ak rk-1)/bk
end do
c----- Back substitution
xn = rn
do for k = (n-1) to 1
xk = (rk - ck xk+1)
end do
18
Example E3.4.6
Problem: Determine the LU-decomposition for the following matrix using TDMA
2

[A] = 1
0
2
4
1
0
4
2
Solution: n=3
c1 = c1/b1 = 2/2 = 1
k=2
b2 = b2 - a2 c1 = 4 - (1)(1) = 3
c2 = c2/b2 = 4/3
k=n=3
b3 = b3 - a3 c2 = 2 - (1)(4/3) = 2/3
The other elements remain the same; hence
2

[L] = 1
0
0
3
1
0 
0  ; [U] =
2 / 3
1
0

0
1
1
0
0 
4 / 3
1 
19
It can be verified that the product [L][U] is indeed equal to [A].
Iterative Methods for Solving Linear Systems
Iterative methods are those where an initial guess is made for the
solution vector followed by a correction sequentially until a certain
bound for error tolerance is reached. The concept of iterative
computations was introduced in Chapter 2 for nonlinear equations.
The idea is the same here except that we deal with linear systems of
equations. In this regard the fixed point iteration method presented
in Chapter 2 for two (nonlinear) equations and two unknowns will
be repeated here, because that is precisely what Jacobi iteration is
about.
20
Jacobi Iteration
Two linear equations, in general, can be written as
a11 x1 + a12 x2 = c1
a21 x1 + a22 x2 = c2
We now rearrange these equations such that each unknown is written as a function of the others,
i.e.
x1 = ( c1 - a12 x2 )/ a11 = f(x1,x2)
x2 = ( c2 - a21 x1 )/ a22 = g(x1,x2)
which are in the same form as those used for the fixed point iteration (Chapter 2). The functions
f(x1,x2) and g(x1,x2) are introduced for generality. It is implicitly assumed that the diagonal
elements of the coefficient matrix are not zero. In case there is a zero diagonal element this should
be avoided by changing the order of the equations, i.e. by pivoting. Jacobi iteration calls for
starting with a guess for the unknowns, x1 and x2, and then finding new values using these
equations iteratively. If certain conditions are satisfied this iteration procedure converges to the
right answer as it did in case of fixed point iteration method.
21
Example E3.5.1
Problem: Solve the following system of equations using Jacobi iteration
10 x1 + 2x2 + 3x3 = 23
2x1 - 10x2 + 3x3 = -9
- x1 - x2 + 5x3 = 12
Solution: Rearranging
x1 = ( 23 - 2x2 - 3x3 )/10
x2 = ( -9 - 2x1 - 3x3 )/(-10)
x3 = (12 + x1 + x2 )/5
Starting with a somewhat arbitrary guess, x1 = 0 , x2 = 0 , and x3 = 0. and iterating yield in the values shown in the table
given below
n
ITER
0
1
2
3
4
5
6
7
8
9
10
X1
0
2.300000
1.400000
0.972000
0.952800
0.991520
1.002560
1.001472
1.000101
0.9998797
0.9999606
X2
X3
0
0.900000
2.080000
2.092000
2.023200
1.994400
1.996864
1.999667
2.000260
2.000089
1.999998
0
2.400000
3.040000
3.096000
3.012800
2.995200
2.997184
2.999885
3.000228
3.000072
2.999994

Error Norm, E = i 1
--5.600000
2.720000
4.960001E-01
1.712000E-01
8.512014E-02
1.548803E-02
6.592035E-03
2.306700E-03
5.483031E-04
2.506971E-04
x inew  x old
i
22
Convergence Criteria for Jacobi Iteration
f
f

= a12/a11< 1
 x 1 x 2
g
g

= a21/a22< 1
 x 1 x 2
In general a sufficient (but not necessary) condition for convergence of Jacobi iteration is
(
n

j 1, j  i
a ij ) / a ii  1
In other words, the sum of the absolute value of the off diagonal elements on each row
must be less than the absolute value of the diagonal element of the coefficient matrix.
These matrices are called strictly diagonally dominant matrices. The reader can show
easily that the matrix of example E3.5.1 does satisfy the criteria given.
23
Gauss - Seidel Iteration
This method is essentially the same as the Jacobi iteration method except that the new
values of the variables (i.e. the most recently updated value) is used in subsequent
calculations without waiting for completion of one iteration for all variables. In cases the
convergence criteria is satisfied, Gauss-Seidel iteration takes fewer iteration to achieve the
same error bound compared to Jacobi iteration. To illustrate how this procedure works we
present the first two iterations of the Gauss-Seidel method applied to the example
problem:
x1 = ( 23 - 2x2 - 3x3 )/10
x2 = ( -9 - 2x1 - 3x3 )/(-10)
x3 = (12 + x1 + x2 )/5
ITER=0, x1=0, x2=0, x3=0 (initial guess)
ITER=1,
x1 = ( 23 - 0 -0 )/10 = 2.3
24
Gauss - Seidel Iteration (II)
Now this new value for x1 is used in the second equation as opposed to using x1 = 0 (the
previous value of x1) in Jacobi iteration, that is
x2 = [-9 - (2)(2.3) - 0 )/(-10) = 1.36
Similarly in the third equation the most recent values of x1 and x2 are used. Hence,
x3 = (12 + 2.3 + 1.36 )/5 = 3.132
And so an for the rest of the iterations
ITER=2
x1 = [23 - (2)(1.36) - (3)(3.132) ] /10
x2 = [ -9 - (2)(0.8884) - (3)(3.132) ] /(-10)
x3 = (12 + 0.8884 + 2.10728) /5
= 0.8884
= 2.10728
= 2.998256
25
Example E3.5.2
Problem: Solve the following system of equations (from Example E3.5.1) using Gauss Seidel iteration.
10 x1 + 2x2 + 3x3 = 23
2x1 - 10x2 + 3x3 = -9
- x1 - x2 + 5x3 = 12
Table E3.5.1 Convergence trend of Gauss-Seidel method for Example E3.5.1
ITER
0
1
2
3
4
5
6
7
8
9
10
X1
0
2.300000E+00
1.088400E+00
9.798032E-01
9.999894E-01
1.000243E+00
9.999875E-01
9.999977E-01
1.000000E+00
1.000000E+00
1.000000E+00
X2
0
1.360000E+00
2.057280E+00
2.004701E+00
1.999068E+00
1.999992E+00
2.000012E+00
1.999999E+00
2.000000E+00
2.000000E+00
2.000000E+00
X3
Error Norm, E
0
--3.132000E+00 6.792000E+00
3.029136E+00 2.011744E+00
2.996901E+00 1.934104E-01
2.999811E+00 2.872980E-02
3.000047E+00 1.412988E-03
3.000000E+00 3.223419E-04
3.000000E+00 2.276897E-05
3.000000E+00 3.457069E-06
3.000000E+00 3.576279E-07
3.000000E+00 0.000000E+00
26
Convergence Criteria for Gauss - Seidel Iteration
Convergence criteria for Gauss-Seidel method are more relaxed than the equation for
Jacobi Iteration convergence. Here a sufficient condition is
 n

  a ij  a ii
 j 1, j  i 
 1 for all equations except one.

 1 for at least one equation.
This condition is also known as the Scarborough criterion. We also introduce the
following theorem concerning the convergence of Gauss-Seidel iteration method.
Theorem:
Let [A] be a real symmetric matrix with positive diagonal elements. Then the Gauss-Seidel
method solving [A]{x} = {c} will converge for any choice of initial guess if and only if [A]
is a positive definite matrix.
Definition:
If for all nonzero complex vectors {x}, {x}T[A]{x} > 0 for a real symmetric matrix [A]
then [A] is said to be positive definite matrix. (Note also that [A] is positive definite if and
only if all of its eigenvalues are real and positive.
27
Relaxation Concept
The iterative methods such as Jacobi and Gauss-Seidel iteration are
successively prediction correction methods in that an initial guess is
corrected to compute a new approximate solution, then the new solution is
corrected and so on. The Gauss-Seidel method, for example can be
formulated as
xi k 1  xi k   xi k 
where k denotes the number of iterations and xi(k) is the correction to the
kth solution. Which is obtained from

xi k   xie  xi k 

To interpret and understand this equation better we let
x ie  x inew , x i k   x iold
and write it as
x inew   x inew  1   x iold
28
Example E3.5.4
Problem: Show that the Gauss-Seidel iteration method converges in 50 iterations when
applied to the following system without relaxation (i.e.  = 1)
x1 + 2x2 = -5
x1 – 3x2 = 20
Find an optimum relaxation factor that will minimize the number of iterations to achieve a
solution with the error bound.
|| Ei || < 0.5 x 10-5
2
where || Ei || =
x
new
i
 xiold
i 1
Solution: A few iterations are shown here with  = 0.8
Let x1old = 0 and x2old = 0 (initial guess)
x1new = (-5 - 2x2old) = -5
After a few iterations we reach
Under relax
x2new = (0.8)(-4.853) + (0.2)(-6.4) = -5.162
x2old = -5.162 (switch old and new)
29
Example E3.5.4 (continued)
Table E3.5.4b Number of Iterations versus Relaxation Factor for Example E3.5.4

0.20
0.40
0.60
0.70
0.80
0.90
1.00
1.05
1.10
Number of iterations
62
30
17
14
11
15
50
84
>1000
It is seen that minimum number of iterations is obtained with a relaxation factor of
 = 0.80.
30
Case Study - The Problem of Falling Objects
m1
T1
m2
g - gravity
T2
m3
-m1 a = c1 v2
-m2 a = c2 v2
-m3 a = c3 v2
- m1 g
- m2 g
- m3 g
- T1
- T1 + T2
+ T2
31
Case Study - The Problem of Falling Objects
These three equations can be used to solve for any three of the unknowns parameters: m1,
m2, m3, c1, c2, c3, v, a, T1, and T2. As an example here we set up a problem by
specifying the masses, the drag coefficients, and the acceleration as
m1 = 10 kg, m2 = 6 kg, m3 = 14 kg, c1 = 1.0 kg/m, c2 = 1.5 kg/m, c3 = 2.0 kg/m, and a =
0.5g and find the remaining three unknowns; v, T 1 and T2. The acceleration of gravity
g=9.81 m/sec2.
Substituting these values in the given set of equations and rearranging yield
v2 - T1
= 49.05
2
1.5 v + T1 - T2 = 29.43
2.0 v2
+ T2 = 68.67
v2 = 32.7 (v = 5.72 m/s), T 1 = -16.35 N, T2 = 3.27 N (N=newton=kg.m/sec2)
32
References
1.
2.
3.
4.
5.
6.
Celik, Ismail, B., “Introductory Numerical Methods for Engineering Applications”,
Ararat Books & Publishing, LCC., Morgantown, 2001
Fausett, Laurene, V. “Numerical Methods, Algorithms and Applications”, Prentice
Hall, 2003 by Pearson Education, Inc., Upper Saddle River, NJ 07458
Rao, Singiresu, S., “Applied Numerical Methods for Engineers and Scientists, 2002
Prentice Hall, Upper Saddle River, NJ 07458
Mathews, John, H.; Fink, Kurtis, D., “Numerical Methods Using MATLAB” Fourth
Edition, 2004 Prentice Hall, Upper Saddle River, NJ 07458
Varol, A., “Sayisal Analiz (Numerical Analysis), in Turkish, Course notes, Firat
University, 2001
http://mathonweb.com/help/backgd3e.htm
33