Microsoft PowerPoint - Sect45WChenSpring20 [Compatibility Mode] - chenw

Please enable JavaScript to view the full PDF

Sect. 4.5 Iterative Methods for Solving Linear System Wenfeng Chen Department of Mathematics and Computer Sciences Laurentian University 1 4.5.0. Introduction and Review 1. Why Iterative Method for Solving Linear System (1). When solving a system Ax = b, because of round-off errors, direct methods become less efficient than iterative methods when they are applied to large systems, sometimes with as many as 100,000 variables, i.e., matrix size n= 100,000. In these cases, an iterative method is preferable. (2). If the coefficient matrix is sparse, i.e., a matrix that contain a high proportion of zeros, the amount of storage space required for iterative solutions on a 2 4.5.0. Introduction and Review, cont. computer is far less than the one required for direct methods. (3). If A is large and sparse, LU decomposition (Gaussian elimination) may introduce ill-in. Then direct method may become costly. (4). We want to take advantage when only a rough approximation x’ to x is required.  Sometimes a system does not really need an exact solution (5). We want to take advantage when a good x0 3 4.5.0. Introduction and Review, cont. approximating x is known.  Sometime there is a pretty good idea of an approximate guess for the solution, this is called warm start. (6). Sometimes A is not explicitly available, only matrix- vector products Av for any vector v can be efficiently carried out. 2. Concrete cases that require iterative methods Many linear systems arising in practice involve values defined on a grid, or a mesh, in two or even three space dimensions. 4 4.5.0. Introduction and Review, cont. <1>. Case 1: pixel of 2-D computer monitor screen Consider the brightness values at each pixel of a two- dimensional computer monitor screen. The collection of such brightness values forms a two-dimensional grid function, {ui,j , i = 1, . . . , N, j = 1, . . . ,M}. 5 4.5.0. Introduction and Review, cont. Often there are linear, algebraic relationships between values at neighboring grid nodes. But neighboring locations in a two- or three-dimensional array do not necessarily correspond to consecutive locations when such a structure is reshaped into a one-dimensional array. <2>. Discretization of the Poisson partial differential equation in one and two dimensions (1). Discretization on 1-D grid (or lattice) of 1-D Poisson equation Consider 1-D Poisson equation in the interval [0,1] 6 4.5.0. Introduction and Review, cont. d 2v v ''(t )   2  g (t ) dt with the homogeneous boundary condition v(0)=v(1)=0 How to do discretization? (1). First, subdivide the interval [0,1] into N equal subintervals of size h, i.e., h=1/N, and define ti =ih, i=0,1,2,…, N. Denote v(ti) = v(ih)=vi . (2). Express the differential derivative as difference derivative 7 4.5.0. Introduction and Review, cont. dv v v(t  h)  v(t ) 1. v '(t )   lim  lim dt t  0 t t 0 h v v(ti  h)  v(ti ) v  (i  1)h   v(ih) vi 1  vi  v '(ti )     t h h h d 2 v d  dv  v '(t  t )  v '(t ) and then v "(t )  2     lim dt dt  dt  t 0 t v '(t  h)  v '(t )  lim t  0 h v '(ti  h)  v '(ti ) 1  v ''(ti )    v '(ti 1 )  v '(ti ) h h 1 v v v v  1   i 1 i  i i 1   2  vi 1  2vi  vi 1  h h h  h 8 4.5.0. Introduction and Review, cont. Then the discretized 1-D Poisson equation –v’’(t) = g(t) is vi 1  2vi  vi 1  2  g (ti ), i  1, 2, , N  1 h To express it in matrix form, we use the Kronecker symbol 1, i  j ,  ij   i, j  1, 2, , N  1 0, i  j, Then the discretized 1-D Poisson equation –v’’(t) = g(t) can be re-expressed as follows: 9 4.5.0. Introduction and Review, cont. N 1 1 h2    j 1 i 1, j  2 i , j   i 1, j  v j  g (ti ) *Remark: In Einstein’s summation notation, repeated index is summed, so the above equation takes the form: 1 2   i 1, j  2 i , j   i 1, j  v j  g (ti ) h  v1   g (t1 )  Because of the boundary condition,  v   g (t )  v(0)=0, v(1)=0, this means after  2   2  v    , g     discretization,v0 =0, vN =0.     v  N 2  g (t  N 2  ) Denote  vN 1   g (t N 1 )  and use the tridiagonal matrix 10 4.5.0. Introduction and Review, cont. (A=1/h2 T) 2 1 0  0   1 2 1  0 1 1  A   aij   2  2 ij   i 1, j   i 1, j   2       h h   0  1 2 1  0  0 1 2  the discrete Poisson equation becomes as a linear system Ax=g. For example, N=5, then h=1/5=0.2, given g(t)= t2, then we have 11 4.5.0. Introduction and Review, cont.  v1   g  t1   t12  0.22  0.04  v     2  2   g  t  t 0.4 0.16 v   2  , g   2    22    2    ,  v3   g  t3   t3  0.6  0.36       2  2   v4   g  t4   t4   0.8  0.64  and  2 1 0 0   1 2 1 0  A  25   , Av  g  0 1 2 1    0 0 2  1 Note that (1). A is symmetric positive definite; 12 4.5.0. Introduction and Review, cont. (2). A is also very sparse when N is large, i.e., ai,j =0 if i>j+1 or j>i+1, so out of (N-1)2 elements of A, only fewer than 3(N-1) are nonzero. [2]. Discretization of Poisson equation in two dimensions Consider a Poisson PDE defined on an unit square, 0<x,y<1 on xy-plane:   2 u  2u    2  2   g ( x, y )  x y  13 4.5.0. Introduction and Review, cont. with homogeneous Dirichlet boundary conditions along the entire boundary of the unit square (i.e., x, y = 0,1): u ( x, 0)  u ( x,1)  u (0, y )  u (1, y )  0 where g(x, y) is a given source, u(x, y) is the unknown function sought Discretization: Divide the square 0≤x, y≤1 into N+1 squares of side h=1/(N+1), then u(x,y)u(xi, yj) =u(ih, jh)=ui,j , Hence, 14 4.5.0. Introduction and Review, cont.  2 u ( x, y ) 1  2  ui 1, j  2ui , j  ui 1, j  , x 2 h  2 u ( x, y ) 1  2  ui , j 1  2ui , j  ui , j 1  , y 2 h 1 g( x, y )  g ( xi , y j )  g (ih, jh)  2 bi , j h That is, ui,j is the value of u(x,y) at the (i,j)h nodes of a square (with M=N) planar grid, and ui,j= h2 g(ih,jh) is the value of u(x,y) at the (i,j)h nodes, h=1/(N+1) is the grid width. Usually the value of N can be fairly large, e.g., N=127 or N=1023 to get more precise result. 15 4.5.0. Introduction and Review, cont. The discretized boundary condition becomes ui ,0  ui , N 1  u0, j  u N 1, j  0 Then the discretized 2-D Poisson equation  1 1    h 2  i 1, j  u  2ui, j  ui 1, j   h 2  i , j 1 u  2ui, j  ui , j 1       1   b , 1  i, j  N 2 i, j  h ui , j  0, Otherwise   That is, 16 4.5.0. Introduction and Review, cont. 4ui , j  ui 1, j  ui 1, j  ui , j 1  ui , j 1  u1,1   b1,1  u  b  N N  2,1   2,1     k 1 l 1 4 ik ; jl   ( i 1),k ; j ,l   ( i 1),k ; j ,l           u  N ,1  b  N ,1    i ,k ;( j 1),l   ik ,( j 1),l uk ,l  bi , j ,  u1,2   b1,2      1  i, j  N u   u2,2  , b   b2,2  ui , j  0, Otherwise            u N ,2   bN ,2  Matrix form of the equation: u  b   1,3   1,3        I. First denote N2 dimensional vectors     u  N ,N  b  N ,N  17 4.5.0. Introduction and Review, cont. II. Second, use the NxN matrices 0 1 0  0 1 0  0 0 0 0 1  0  1  0  I   ij    , K   i 1, j    i , j 1                 0 0  0 1 0 0  1 0 0  0 0  0 0 0  0 1 0 0  0   L  K   i 1, j    i , j 1   0 1         0 0 0 0  1 0  18 4.5.0. Introduction and Review, cont. Consequently, the matrix form of the equation is  4I  I  K  I  L  I  I  K  I  L u  b where the tensor product of two matrices are defined as follows:  a11 B a12 B  a1n B  a B a B  a2 n B  A  B   aij   bkl    aij bkl    21 22 mn p q ( mp )( nq )          am1B am 2 B  amn B  That is, the discretized Poisson equation in 2-D is Au  b 19 4.5.0. Introduction and Review, cont. where A is an N2 x N2 J I  0 0   matrix of the form  I J I  0  A         0  I J I   0  0  I J  I is NXN identity matrix J is the tridiagonal NXN matrix 4 1 0  0 1 0  0   0  1 4 1  0 1  0  I   ij    , J               0  1 4 1 0 0  1  0  0 1 4  20 4.5.0. Introduction and Review, cont. ● ● ● uN,1 uN,2 uN,N ● ● ● ● ui,j ● ● ● ● u2,1 u2,2 u2,N ● ● ● u1,1 u1,2 u1,N 21 4.5.0. Introduction and Review, cont. For example, in the case N=3 4 1 0 1 0 0 0 0 0     1 4 1 0 1 0 0 0 0  0 1 4 0 0 1 0 0 0     1 0 0 4 1 0 1 0 0  A 0 1 0 1 4 1 0 1 0    0 0 1 0 1 4 0 0 1    0 0 0 1 0 0 4 1 0   0 0 0 0 1 0 1 4 1    0 0 0 0 0 1 0 1 4  22 4.5.0. Introduction and Review, cont. Note that (1). A is non-singular, and positive definite, (2). u consists of the n = N2 unknowns {ui,j} organized as a vector with N2 components, and b is composed likewise from the values {bi,j} . Remark: Note that n can easily become very large. Even for a simple 2D case with N = 100, the matrix dimension is n = 10, 000. In 3D and the same h we would get n = 106, so A cannot even be stored as a full matrix. 23 4.5.0. Introduction and Review, cont. This is even or the simplest problem of its type! Later, we can solve this equation by iterative method . <3>. Fixed point iteration Here fixed point iteration is a vector version of the fixed point iteration of solving nonlinear equation introduced in Sect.3.2 Brief idea and form: For a given linear system, Ax=b, rewrite it as a vector equation f ( x)  b  Ax  0 24 4.5.0. Introduction and Review, cont. Then seek an equivalent form of this vector equation: g (x)  x from which the fixed point iteration can be defined: x k 1  g  x k  , k  0,1, 2, starting from an initial guess x0 . 3. Review/Introduction on vector norm To understand convergence of iterative method in solving 25 4.5.0. Introduction and Review, cont. a linear system, we need to know the notions of norm of a vector and norm of a matrix. <1>. Why study norm of vectors and matrices in numerical method As we know, (1). Absolute error and relative error are used to assess the quality of a numerical algorithm (2). Absolute error = distance between approximate solution and true solution 26 4.5.0. Introduction and Review, cont. (3). Solution of linear system = a vector (4). To study errors, we must equip with the notion of distance between vectors and between matrices Norm of a vector or a matrix is just a generalization of the absolute value (or magnitude) of scalar. <2>. Definition of vector norms The norm of a n-dimensional vector x, denoted as ||x||, is a scalar function from Rn to R that satisfy 3 requirements: 27 4.5.0. Introduction and Review, cont. 1. x ≥ 0; x = 0 if and only x = 0, 2. αx = |α| x α R, 3. x + y ≤ x + y , x, y Rn. <3>. Types of norms For a vector x Rn [1]. ℓ2-norm (2-norm or Euclidean norm) The ℓ2-norm of x is just the familiar Euclidean length 12   x1    x   12  n 2 x 2  xT x    x1 x2  xn   2     x12  x22    x 2 12     xn      n  i 1        xn   28 4.5.0. Introduction and Review, cont. [2]. ℓ∞-norm (∞-norm or max norm) x   max xi 1i  n [3]. ℓ1-norm (1-norm or sum norm) n x 1   xi i 1 [4]. The most general one is ℓp-norm (p-norm) 1 p  n p x p    xi  , 1 p    i 1  Comparison between ℓ∞-norm and ℓ2-norm 29 4.5.0. Introduction and Review, cont. Theorem: there exist x Rn (1). x   x2 n x  (2). x   x1n x  *Proof: We prove only (1). Let xi be a coordinate of x such that x   max xi  x p 1i  n Then n  x p  x p   xi  x 2 2 2 2 2 x  2 i 1 and thus x  x  2 n n Further, x 2   x   x 2p  nx 2p  n x 2 2 2 i  i 1 i 1 30 4.5.0. Introduction and Review, cont. and hence x2 n x  <4>. Distance between two vectors The distance between two vectors is defined as the norm of the difference vector of the two vectors. Example 1: Find the distance between the following two vectors using ℓ1-norm, ℓ2-norm and ℓ∞-norm, respectively 31 4.5.0. Introduction and Review, cont. 11 12  x  12  , y  14  13 16  Solution: Let 12  11  1  z  y  x  14   12    2  16  13  3  Then the distance is (1). z 1 = 1 + 2 + 3 = 6, (2). z 2 = (1 + 4 + 9)1/2 ≈ 3.7417, (3). z ∞ = 3. 32 4.5.0. Introduction and Review, cont. Because of the distances defined by different norms, an equation describing a curve has distinct shapes. Example 2: Sketch the curve x = 1 for xT=[x, y] with the ℓ1-norm, ℓ2-norm and ℓ∞-norm, respectively Solution: (1). ℓ1-norm: |x|+|y|=1. Therefore, 1  1st quadrant: x+y=1  y=-x+1 -1 1  2nd quadrant: -x+y=1y=x+1  3rd quadrant: -x-y=1  y=-x-1 -1  4th quadrant: x-y=1  y=x-1 33 4.5.0. Introduction and Review, cont. (2). ℓ2-norm: 1 -1 1 x  y  1, i.e., x  y  1 2 2 2 2 (3). ℓ∞-norm: -1  x  1 1 x  max xi   1 i  2  y  1 -1 1 -1 34 4.5.0. Introduction and Review, cont. <5>. Convergence of a sequence of vectors [1]. Definition of the convergence A sequence {x(k)}∞k=1 of vectors in Rn is said to converge to x with respect to a type norm ||.|| (lp-norm or l∞-norm etc.) if, given any ε > 0, there exists an integer N(ε) such that x ( k )  x   for all k  N ( ) [2]. Mathematical results (1). How to check the convergence of a vector sequence with respect to l∞-norm 35 <5>. Convergence of a sequence of vectors *Theorem: The sequence of vectors {x(k)} converges to x in Rn with respect to the l∞ norm if and only if lim xi( k )  xi for each i  1, 2, n k  (2). Norm equivalence with respect to convergence Theorem: All norms on Rn are equivalent with respect to convergence; that is, if ||·|| and ||·||’ are any two norms on Rn and {x(k)}∞k=1 has the limit x with respect to ||·|| , then {x(k)}∞k=1 also has the limit x with respect to ||·||’ Application of the Theorem: one can choose a most 36 <5>. Convergence of a sequence of vectors convenient normal to discuss convergence of a vector sequence. [3]. Example  x1 (k )   1   (k )    1  Show that the x ( k )   x2    2  1 k  2  x (k )   3 k 2  x  following vector 3  (k )   k 0  sequence  x4  e cos k  converge to  0  with respect to the l∞ norm. Solution: Apply the above Theorem  1 lim1  1, lim  2    2, k  k   k Therefore, x(k) converges to x. 3 lim 2  0, lim e  k sin k  0 k  k k  37 4.5.0. Introduction and Review, cont. 4. Review/Introduction on matrix norms <1>. What is a norm of a matrix  Matrix norm is scalar functions that map a matrix as a scalar (i.e., matrix as an argument).  A norm for a matrix can be exactly defined as a norm for a n-dimensional vectors in n, since the vector space Mmn of all mxn matrices is isomorphic to Rmn.  A matrix norm satisfies an additional property. 38 4.5.0. Introduction and Review, cont. <2>. Definition  A matrix norm on the set of all mxn matrices Mmn is a mapping that associates with each mxn matrix A a real number ||A||, called the norm of A, such that the following properties are satisfied for all mxn matrices A, nxp matrices B and all scalars c. 1. ||A|| ≥ 0 and ||A|| = 0 if and only if A=0. 2. ||cA||=|c| ||A|| 3. ||A+B||≤ ||A||+||B|| 4. ||AB||≤ ||A|| ||B|| (consistency condition)  A matrix norm on Mmn is said to be compatible with a 39 4.5.0. Introduction and Review, cont. There are many ways to define a matrix norm satisfying the above requirements, e.g., the Frobenius norm ||A||F is m n   a112  a122    amn   ij 12 A F 2  a 2 i 1 j 1 The most useful norm is induced matrix norm (or operator form) of m × n matrix A for a given vector norm ||x|| (since it considers the effect of the matrix transformation): Ax   A  max  max  x 1 Ax  x0 x   40