Sect. 4.5 Iterative Methods for Solving Linear System Wenfeng Chen Department of Mathematics and Computer Sciences Laurentian University 1 4.5.0. Introduction and Review 1. Why Iterative Method for Solving Linear System (1). When solving a system Ax = b, because of roundoff errors, direct methods become less efficient than iterative methods when they are applied to large systems, sometimes with as many as 100,000 variables, i.e., matrix size n= 100,000. In these cases, an iterative method is preferable. (2). If the coefficient matrix is sparse, i.e., a matrix that contain a high proportion of zeros, the amount of storage space required for iterative solutions on a 2 4.5.0. Introduction and Review, cont. computer is far less than the one required for direct methods. (3). If A is large and sparse, LU decomposition (Gaussian elimination) may introduce illin. Then direct method may become costly. (4). We want to take advantage when only a rough approximation x’ to x is required. Sometimes a system does not really need an exact solution (5). We want to take advantage when a good x0 3 4.5.0. Introduction and Review, cont. approximating x is known. Sometime there is a pretty good idea of an approximate guess for the solution, this is called warm start. (6). Sometimes A is not explicitly available, only matrix vector products Av for any vector v can be efficiently carried out. 2. Concrete cases that require iterative methods Many linear systems arising in practice involve values defined on a grid, or a mesh, in two or even three space dimensions. 4 4.5.0. Introduction and Review, cont. <1>. Case 1: pixel of 2D computer monitor screen Consider the brightness values at each pixel of a two dimensional computer monitor screen. The collection of such brightness values forms a twodimensional grid function, {ui,j , i = 1, . . . , N, j = 1, . . . ,M}. 5 4.5.0. Introduction and Review, cont. Often there are linear, algebraic relationships between values at neighboring grid nodes. But neighboring locations in a two or threedimensional array do not necessarily correspond to consecutive locations when such a structure is reshaped into a onedimensional array. <2>. Discretization of the Poisson partial differential equation in one and two dimensions (1). Discretization on 1D grid (or lattice) of 1D Poisson equation Consider 1D Poisson equation in the interval [0,1] 6 4.5.0. Introduction and Review, cont. d 2v v ''(t ) 2 g (t ) dt with the homogeneous boundary condition v(0)=v(1)=0 How to do discretization? (1). First, subdivide the interval [0,1] into N equal subintervals of size h, i.e., h=1/N, and define ti =ih, i=0,1,2,…, N. Denote v(ti) = v(ih)=vi . (2). Express the differential derivative as difference derivative 7 4.5.0. Introduction and Review, cont. dv v v(t h) v(t ) 1. v '(t ) lim lim dt t 0 t t 0 h v v(ti h) v(ti ) v (i 1)h v(ih) vi 1 vi v '(ti ) t h h h d 2 v d dv v '(t t ) v '(t ) and then v "(t ) 2 lim dt dt dt t 0 t v '(t h) v '(t ) lim t 0 h v '(ti h) v '(ti ) 1 v ''(ti ) v '(ti 1 ) v '(ti ) h h 1 v v v v 1 i 1 i i i 1 2 vi 1 2vi vi 1 h h h h 8 4.5.0. Introduction and Review, cont. Then the discretized 1D Poisson equation –v’’(t) = g(t) is vi 1 2vi vi 1 2 g (ti ), i 1, 2, , N 1 h To express it in matrix form, we use the Kronecker symbol 1, i j , ij i, j 1, 2, , N 1 0, i j, Then the discretized 1D Poisson equation –v’’(t) = g(t) can be reexpressed as follows: 9 4.5.0. Introduction and Review, cont. N 1 1 h2 j 1 i 1, j 2 i , j i 1, j v j g (ti ) *Remark: In Einstein’s summation notation, repeated index is summed, so the above equation takes the form: 1 2 i 1, j 2 i , j i 1, j v j g (ti ) h v1 g (t1 ) Because of the boundary condition, v g (t ) v(0)=0, v(1)=0, this means after 2 2 v , g discretization,v0 =0, vN =0. v N 2 g (t N 2 ) Denote vN 1 g (t N 1 ) and use the tridiagonal matrix 10 4.5.0. Introduction and Review, cont. (A=1/h2 T) 2 1 0 0 1 2 1 0 1 1 A aij 2 2 ij i 1, j i 1, j 2 h h 0 1 2 1 0 0 1 2 the discrete Poisson equation becomes as a linear system Ax=g. For example, N=5, then h=1/5=0.2, given g(t)= t2, then we have 11 4.5.0. Introduction and Review, cont. v1 g t1 t12 0.22 0.04 v 2 2 g t t 0.4 0.16 v 2 , g 2 22 2 , v3 g t3 t3 0.6 0.36 2 2 v4 g t4 t4 0.8 0.64 and 2 1 0 0 1 2 1 0 A 25 , Av g 0 1 2 1 0 0 2 1 Note that (1). A is symmetric positive definite; 12 4.5.0. Introduction and Review, cont. (2). A is also very sparse when N is large, i.e., ai,j =0 if i>j+1 or j>i+1, so out of (N1)2 elements of A, only fewer than 3(N1) are nonzero. [2]. Discretization of Poisson equation in two dimensions Consider a Poisson PDE defined on an unit square, 0<x,y<1 on xyplane: 2 u 2u 2 2 g ( x, y ) x y 13 4.5.0. Introduction and Review, cont. with homogeneous Dirichlet boundary conditions along the entire boundary of the unit square (i.e., x, y = 0,1): u ( x, 0) u ( x,1) u (0, y ) u (1, y ) 0 where g(x, y) is a given source, u(x, y) is the unknown function sought Discretization: Divide the square 0≤x, y≤1 into N+1 squares of side h=1/(N+1), then u(x,y)u(xi, yj) =u(ih, jh)=ui,j , Hence, 14 4.5.0. Introduction and Review, cont. 2 u ( x, y ) 1 2 ui 1, j 2ui , j ui 1, j , x 2 h 2 u ( x, y ) 1 2 ui , j 1 2ui , j ui , j 1 , y 2 h 1 g( x, y ) g ( xi , y j ) g (ih, jh) 2 bi , j h That is, ui,j is the value of u(x,y) at the (i,j)h nodes of a square (with M=N) planar grid, and ui,j= h2 g(ih,jh) is the value of u(x,y) at the (i,j)h nodes, h=1/(N+1) is the grid width. Usually the value of N can be fairly large, e.g., N=127 or N=1023 to get more precise result. 15 4.5.0. Introduction and Review, cont. The discretized boundary condition becomes ui ,0 ui , N 1 u0, j u N 1, j 0 Then the discretized 2D Poisson equation 1 1 h 2 i 1, j u 2ui, j ui 1, j h 2 i , j 1 u 2ui, j ui , j 1 1 b , 1 i, j N 2 i, j h ui , j 0, Otherwise That is, 16 4.5.0. Introduction and Review, cont. 4ui , j ui 1, j ui 1, j ui , j 1 ui , j 1 u1,1 b1,1 u b N N 2,1 2,1 k 1 l 1 4 ik ; jl ( i 1),k ; j ,l ( i 1),k ; j ,l u N ,1 b N ,1 i ,k ;( j 1),l ik ,( j 1),l uk ,l bi , j , u1,2 b1,2 1 i, j N u u2,2 , b b2,2 ui , j 0, Otherwise u N ,2 bN ,2 Matrix form of the equation: u b 1,3 1,3 I. First denote N2 dimensional vectors u N ,N b N ,N 17 4.5.0. Introduction and Review, cont. II. Second, use the NxN matrices 0 1 0 0 1 0 0 0 0 0 1 0 1 0 I ij , K i 1, j i , j 1 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 L K i 1, j i , j 1 0 1 0 0 0 0 1 0 18 4.5.0. Introduction and Review, cont. Consequently, the matrix form of the equation is 4I I K I L I I K I L u b where the tensor product of two matrices are defined as follows: a11 B a12 B a1n B a B a B a2 n B A B aij bkl aij bkl 21 22 mn p q ( mp )( nq ) am1B am 2 B amn B That is, the discretized Poisson equation in 2D is Au b 19 4.5.0. Introduction and Review, cont. where A is an N2 x N2 J I 0 0 matrix of the form I J I 0 A 0 I J I 0 0 I J I is NXN identity matrix J is the tridiagonal NXN matrix 4 1 0 0 1 0 0 0 1 4 1 0 1 0 I ij , J 0 1 4 1 0 0 1 0 0 1 4 20 4.5.0. Introduction and Review, cont. ● ● ● uN,1 uN,2 uN,N ● ● ● ● ui,j ● ● ● ● u2,1 u2,2 u2,N ● ● ● u1,1 u1,2 u1,N 21 4.5.0. Introduction and Review, cont. For example, in the case N=3 4 1 0 1 0 0 0 0 0 1 4 1 0 1 0 0 0 0 0 1 4 0 0 1 0 0 0 1 0 0 4 1 0 1 0 0 A 0 1 0 1 4 1 0 1 0 0 0 1 0 1 4 0 0 1 0 0 0 1 0 0 4 1 0 0 0 0 0 1 0 1 4 1 0 0 0 0 0 1 0 1 4 22 4.5.0. Introduction and Review, cont. Note that (1). A is nonsingular, and positive definite, (2). u consists of the n = N2 unknowns {ui,j} organized as a vector with N2 components, and b is composed likewise from the values {bi,j} . Remark: Note that n can easily become very large. Even for a simple 2D case with N = 100, the matrix dimension is n = 10, 000. In 3D and the same h we would get n = 106, so A cannot even be stored as a full matrix. 23 4.5.0. Introduction and Review, cont. This is even or the simplest problem of its type! Later, we can solve this equation by iterative method . <3>. Fixed point iteration Here fixed point iteration is a vector version of the fixed point iteration of solving nonlinear equation introduced in Sect.3.2 Brief idea and form: For a given linear system, Ax=b, rewrite it as a vector equation f ( x) b Ax 0 24 4.5.0. Introduction and Review, cont. Then seek an equivalent form of this vector equation: g (x) x from which the fixed point iteration can be defined: x k 1 g x k , k 0,1, 2, starting from an initial guess x0 . 3. Review/Introduction on vector norm To understand convergence of iterative method in solving 25 4.5.0. Introduction and Review, cont. a linear system, we need to know the notions of norm of a vector and norm of a matrix. <1>. Why study norm of vectors and matrices in numerical method As we know, (1). Absolute error and relative error are used to assess the quality of a numerical algorithm (2). Absolute error = distance between approximate solution and true solution 26 4.5.0. Introduction and Review, cont. (3). Solution of linear system = a vector (4). To study errors, we must equip with the notion of distance between vectors and between matrices Norm of a vector or a matrix is just a generalization of the absolute value (or magnitude) of scalar. <2>. Definition of vector norms The norm of a ndimensional vector x, denoted as x, is a scalar function from Rn to R that satisfy 3 requirements: 27 4.5.0. Introduction and Review, cont. 1. x ≥ 0; x = 0 if and only x = 0, 2. αx = α x α R, 3. x + y ≤ x + y , x, y Rn. <3>. Types of norms For a vector x Rn [1]. ℓ2norm (2norm or Euclidean norm) The ℓ2norm of x is just the familiar Euclidean length 12 x1 x 12 n 2 x 2 xT x x1 x2 xn 2 x12 x22 x 2 12 xn n i 1 xn 28 4.5.0. Introduction and Review, cont. [2]. ℓ∞norm (∞norm or max norm) x max xi 1i n [3]. ℓ1norm (1norm or sum norm) n x 1 xi i 1 [4]. The most general one is ℓpnorm (pnorm) 1 p n p x p xi , 1 p i 1 Comparison between ℓ∞norm and ℓ2norm 29 4.5.0. Introduction and Review, cont. Theorem: there exist x Rn (1). x x2 n x (2). x x1n x *Proof: We prove only (1). Let xi be a coordinate of x such that x max xi x p 1i n Then n x p x p xi x 2 2 2 2 2 x 2 i 1 and thus x x 2 n n Further, x 2 x x 2p nx 2p n x 2 2 2 i i 1 i 1 30 4.5.0. Introduction and Review, cont. and hence x2 n x <4>. Distance between two vectors The distance between two vectors is defined as the norm of the difference vector of the two vectors. Example 1: Find the distance between the following two vectors using ℓ1norm, ℓ2norm and ℓ∞norm, respectively 31 4.5.0. Introduction and Review, cont. 11 12 x 12 , y 14 13 16 Solution: Let 12 11 1 z y x 14 12 2 16 13 3 Then the distance is (1). z 1 = 1 + 2 + 3 = 6, (2). z 2 = (1 + 4 + 9)1/2 ≈ 3.7417, (3). z ∞ = 3. 32 4.5.0. Introduction and Review, cont. Because of the distances defined by different norms, an equation describing a curve has distinct shapes. Example 2: Sketch the curve x = 1 for xT=[x, y] with the ℓ1norm, ℓ2norm and ℓ∞norm, respectively Solution: (1). ℓ1norm: x+y=1. Therefore, 1 1st quadrant: x+y=1 y=x+1 1 1 2nd quadrant: x+y=1y=x+1 3rd quadrant: xy=1 y=x1 1 4th quadrant: xy=1 y=x1 33 4.5.0. Introduction and Review, cont. (2). ℓ2norm: 1 1 1 x y 1, i.e., x y 1 2 2 2 2 (3). ℓ∞norm: 1 x 1 1 x max xi 1 i 2 y 1 1 1 1 34 4.5.0. Introduction and Review, cont. <5>. Convergence of a sequence of vectors [1]. Definition of the convergence A sequence {x(k)}∞k=1 of vectors in Rn is said to converge to x with respect to a type norm . (lpnorm or l∞norm etc.) if, given any ε > 0, there exists an integer N(ε) such that x ( k ) x for all k N ( ) [2]. Mathematical results (1). How to check the convergence of a vector sequence with respect to l∞norm 35 <5>. Convergence of a sequence of vectors *Theorem: The sequence of vectors {x(k)} converges to x in Rn with respect to the l∞ norm if and only if lim xi( k ) xi for each i 1, 2, n k (2). Norm equivalence with respect to convergence Theorem: All norms on Rn are equivalent with respect to convergence; that is, if · and ·’ are any two norms on Rn and {x(k)}∞k=1 has the limit x with respect to · , then {x(k)}∞k=1 also has the limit x with respect to ·’ Application of the Theorem: one can choose a most 36 <5>. Convergence of a sequence of vectors convenient normal to discuss convergence of a vector sequence. [3]. Example x1 (k ) 1 (k ) 1 Show that the x ( k ) x2 2 1 k 2 x (k ) 3 k 2 x following vector 3 (k ) k 0 sequence x4 e cos k converge to 0 with respect to the l∞ norm. Solution: Apply the above Theorem 1 lim1 1, lim 2 2, k k k Therefore, x(k) converges to x. 3 lim 2 0, lim e k sin k 0 k k k 37 4.5.0. Introduction and Review, cont. 4. Review/Introduction on matrix norms <1>. What is a norm of a matrix Matrix norm is scalar functions that map a matrix as a scalar (i.e., matrix as an argument). A norm for a matrix can be exactly defined as a norm for a ndimensional vectors in n, since the vector space Mmn of all mxn matrices is isomorphic to Rmn. A matrix norm satisfies an additional property. 38 4.5.0. Introduction and Review, cont. <2>. Definition A matrix norm on the set of all mxn matrices Mmn is a mapping that associates with each mxn matrix A a real number A, called the norm of A, such that the following properties are satisfied for all mxn matrices A, nxp matrices B and all scalars c. 1. A ≥ 0 and A = 0 if and only if A=0. 2. cA=c A 3. A+B≤ A+B 4. AB≤ A B (consistency condition) A matrix norm on Mmn is said to be compatible with a 39 4.5.0. Introduction and Review, cont. There are many ways to define a matrix norm satisfying the above requirements, e.g., the Frobenius norm AF is m n a112 a122 amn ij 12 A F 2 a 2 i 1 j 1 The most useful norm is induced matrix norm (or operator form) of m × n matrix A for a given vector norm x (since it considers the effect of the matrix transformation): Ax A max max x 1 Ax x0 x 40
Enter the password to open this PDF file:











