A Statistical Machine Learning Framework for High-Dimensional Partial Differential Equations G14DIS Mathematics 4th Year Dissertation 2019/20 School of Mathematical Sciences University of Nottingham Isaac Oldwood Supervisor: Dr. D Kalise Project code: DK D1 Assessment type: Investigation I have read and understood the School and University guidelines on plagiarism. I confirm that this work is my own, apart from the acknowledged references. Abstract In this paper I will be investigating the use of artificial neural networks to solve high dimensional partial differential equations. I start by introducing neural networks, I pro- ceed by building basic examples to approximate functions. Following an explanation of these, I increase the complexity of the equations and increase the dimensions. I explore how different setups of neural networks alter the performance. Finally I approximate a partial differential equation using a distance function and complex neural network. I use MATLAB for all the computation and include examples of the code I use. Contents 1 Introduction 5 2 Introducing Neural Networks 6 2.1 Topology and notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Forward propagation and activation functions . . . . . . . . . . . . . . . . 7 2.3 Loss and cost functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.4 Optimization problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.5 Gradient descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.6 Backpropagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.7 Stochastic gradient descent . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3 Distance Functions 17 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2 1D example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.3 Distance between sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.4 1D example between a point and set . . . . . . . . . . . . . . . . . . . . . 20 3.5 2D distance function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.6 Approximating distance functions . . . . . . . . . . . . . . . . . . . . . . . 25 3.7 Finding the optimal neural network . . . . . . . . . . . . . . . . . . . . . . 30 4 Partial Differential Equations 51 4.1 The eikonal equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.2 3D example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 5 Conclusions 65 A Levenberg-Marquardt Algorithm 66 B Tables 67 C Code 67 C.1 Code for 3.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 3 C.2 Code for 3.7.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 C.3 Code for neural network and plotting isosurfaces in 4.2 . . . . . . . . . . . 75 4 1 Introduction Machine learning is a field of computer science where algorithm parameters are learned through experience rather than being explicitly programmed. More specifically, machine learning is an approach to data analysis that involves building and adapting models, which allow programs to ”learn” through experience. Machine learning involves the construction of algorithms that adapt their models to improve their ability to make predictions [5]. Artificial neural networks are a specific type of machine learning model inspired by the structure of the brain. The first case of an artificial neural network was in 1943, when neurophysiologist Warren McCulloch and mathematician Walter Pitts wrote a paper about neurons and how they work. They decided to create a model of this using an electrical circuit, and therefore the artificial neural network was born. In 1952, the world saw the first computer program which could learn as it ran. It was a game which played checkers, created by Arthur Samuel [6]. Recently machine learning and specifically neural networks have started growing in popularity. This can be attributed to many factors such as the increase in computing power, as it has allowed us to produce more complicated models than before. Neural networks are able to effectively model highly complex and non-linear relationships which naturally means they are useful in many real life examples. They are highly adaptable, able to be used for many types of tasks such as regression and classification, and can be as simple or as complicated as we desire. This report starts by covering notation and introducing neural networks in section 2. Section 3 contains the bulk of the report. It covers distance functions, examples of neural networks, and performs tests to see how different neural networks perform. The final section (4) brings the report together by combining neural networks, distance functions and partial differential equations. The appendices B and C cover tables and computer code respectively. 5 2 Introducing Neural Networks This section will contain background material on neural networks including how they are designed, how they ”learn” and how they apply this to help us solve real world problems. 2.1 Topology and notation The topology of neural networks can look complex but can be broken down to appear sim- ple. As explained in section 1 they are based on the structure of the brain which is simply made up of neurons connected by synapses. Neural networks replicate this structure of interconnected neurons. Figure 1 shows the basic structure of a neural network. Figure 1: Structure of a basic artificial neural network Each circle represents an artificial neuron and the arrows represent the connections between the neurons. All neural networks contain a single input layer, a single output layer and any number of hidden layers. Each layer can contain any number of neurons however the problem and data being used can give a good indication to the number in certain layers. For example if you are trying to approximate a straight line function then you have one input (x) and one output (y) so you would have one neuron in the input layer and one in the output layer. Intuitively more complex problems require more complex neural networks that contain more layers with more neurons. 6 The first paper I read to spark my interest in this topic was [1], the notation used in the paper is sensible and intuitive to the audience of mathematicians. Since this paper is aimed at the same audience I will use notation consistent with [1]. This paper will use the following notation throughout. a [ l ] j is the output from neuron j at layer l W is a weight matrix b is a vector of biases h is a general activation explained in 2.2 η is the step size / learning rate explained in 2.5 z [ l ] j is the weighted input from neuron j at layer l C is the cost function explained in 2.3 ∂ is the partial derivative σ ( x ) is the sigmoid function explained in 2.2 2.2 Forward propagation and activation functions Forward propagation is the process of feeding input data forward through the neural network. At each layer an input is received, the layer performs some set calculations on the input (governed by the activation function) and passes the output onto the next layer. This process is repeated until the final layer which produces the final output. Let the output of real numbers from one layer of neurons be a vector a Then the output vector from that layer is: h ( W · a + b ) (2.1) Where h is some activation function, W is the weight matrix for that layer, a is the input vector and b is the vector of biases. This equation can be chained together to generate an equation for the whole neural network. Let x be the input data and L be the number of layers, then the output of the neural network F ( x ) can be written as: F ( x ) = h ( W [ L ] h ( W [ L − 1] ...h ( W [2] x + b [2] ) ... + b [ L − 1] ) + b [ L ] ) (2.2) This is the general form of the equation that explains forward propagation. 7 Activation functions are “a function that defines the output of a node (or neuron) given an input”[3]. If the activation functions are omitted in a neural network, then each output would just be a linear function ( W · a + b ). So, a neural network without them is just a linear regression model [4]. Hence, to introduce more complexity and allow neural networks to solve non-trivial problems a non-linear element must be introduced. This is why activation functions are used. Another key characteristic of activation functions is that they should be differentiable, this is to allow the process of backpropagation which is explained in section 2.6, and preferably non-linear. There are many functions that have these properties but some activation functions are better than others. Some take longer to train the neural network but are more efficient computationally. Some functions take more computing power but take less time to train the neural network. A few common activation functions are described below. 2.2.1 Sigmoid function σ ( x ) = 1 1+ e − x Figure 2: Sigmoid Graph The sigmoid function is usually thought of as a smoothed step function. It is easy to understand and apply since the equation is simple with an equally simple derivative, having the form: σ ′ ( x ) = σ ( x )(1 − σ ( x )) The sigmoid function is not as popular as it used to be as it has some problems. For example, its output is not centred around 0, it has slow convergence and it suffers from 8 the vanishing gradient problem. In machine learning the vanishing gradient problem occurs when using gradient-based training methods and backpropagation (see section 2.6) [3]. If the activation function has a portion where the gradient is very small then the change in the weights and biases vanishes and prevents further training. This problem is exacerbated as the gradient is propagated backwards through the layers. To fix this issue, different activation functions have been developed. 2.2.2 Tanh function f ( x ) = 1 − e − 2 x 1+ e − 2 x Figure 3: Tanh Graph This function is an improvement on the sigmoid function since it is centred around 0. However due to its shape it still suffers from the vanishing gradient problem. 9 2.2.3 Rectified Linear Units (ReLU) R ( x ) = max (0 , x ) Figure 4: ReLU Graph Recently this function has gained a lot of popularity and it has been proven to converge six times faster than the Tanh function [23]. Furthermore it is popular because it does not fall prey to the vanishing gradient problem. The reasoning behind this is because the gradient of ReLU is either 1 or 0. This means that as the gradient is propagated backwards it is not diminished [25]. The other main limitation is that it can only be used within the hidden layers. 2.3 Loss and cost functions In order to train and improve the accuracy of a neural network we need to find out how well the model is currently performing. This allows us to evaluate if the model is improving and by how much. The way this is done is using a loss function and a cost function. The loss function of a neural network is the function used to calculate the error of one example. The cost function is the average of the loss functions across all the examples. Therefore the cost function is just a function of all the weights and biases which calculates how wrong the current model is and returns a single value (explained further in section 10 2.5). That single value is the current ”cost” of the model, so the more accurate the neural network, the smaller the cost is. This allows the training of the neural network to become an optimisation problem. Training a neural network can then be condensed down to: what weights and biases give the smallest cost? This problem can be solved by various optimisation methods however the most common is gradient descent and the similar stochastic gradient descent. The problem being solved and the activation functions being used both affect what the loss function should be. Some functions are more effective and common than others for a variety of reasons. One of the simplest and most common loss function is the mean squared error (MSE). This loss function calculates the squared difference between a predicted value and the true value. Since this function involves squaring the difference, larger errors have a much greater affect on the training compared to small errors. Also the value is always positive and therefore the best model would give a value equal to zero. Let ˆ Y be the approximate value produced by a model (or neural network) and Y be the true value of the function, then the equation for the loss function is M SE = ( ˆ Y − Y ) 2 (2.3) By taking the average over all n data points it is possible to find the form of the cost function as M SE = 1 n n ∑ i =1 ( ˆ Y i − Y i ) 2 (2.4) 2.4 Optimization problem We can use the cost function to boil our neural network down to an optimisation problem. Let N N θ ( x ) be our neural network with an input x and a parameter θ such that θ = { W [ L ] , ..., W [2] , b [ L ] , ..., b [2] } containing all the weights and biases. Let our cost function be the MSE described in equation 2.4. Then the optimisation problem is minimise θ J θ = 1 n n ∑ i =1 ( N N θ ( x i ) − Y i ) 2 (2.5) 11 This equation essentially turns training our neural network into an optimisation prob- lem. It means: find the values of θ = { W [ L ] , ..., W [2] , b [ L ] , ..., b [2] } that minimise the error between the neural network’s approximate value and the true value of the function we are trying to approximate. 2.5 Gradient descent Gradient descent is a common optimization method used to find the minimum (or max- imum) of a function. An initialised neural network contains weight matrices of random real numbers and bias vectors of either real numbers or zeroes. This means unless we are very lucky these weights and biases will produce a large cost for the neural network so it is needed to optimise these weights and biases to reduce the cost value. Gradient descent is an iterative method that changes these weights and biases to reduce the cost and train the neural network. 2.5.1 The basic method The main idea of gradient descent is like finding your way down a valley. To do this you need to know two things: which direction to go and how far to go in that direction. You want to find the direction that points directly to the bottom of the valley. You also need to be careful about how far you go on each step; if your steps are too small then it will take a long time to reach the bottom, on the other hand if your steps are too large then you may go too far and miss the bottom of the valley. This method is visualised in figure 5. 12 Figure 5: Gradient Descent [2] Gradient descent determines the direction of travel by using backpropagation which is explained in section 2.6. The gradient of the cost function is represented as ∇ C . After the direction is determined the learning rate η is picked. In the standard gradient descent algorithm described in this section the learning rate η is chosen before the algorithm starts. However there are variations that alter the learning rate η as part of the optimization process such as Adagrad [26]. The learning rate η is proportional to the step size used in the gradient descent method. It is important that a good learning rate η is chosen or calculated. This is because the iterations do not always converge to a minimum for any learning rate η As previously mentioned if the learning rate (and by extension the step size) is too large then the method could overshoot the minimum and the iterations will diverge. On the other hand, if the step size is too small then the method could take too long to converge and would be computationally inefficient. The gradient descent algorithm is: 1. Initialize the weights W and biases b randomly. 2. Calculate the gradients ∇ C of cost function with respect to the parameters using the whole data set. This is done using partial differentiation: ∇ C = ∂J ∂θ The 13 value of the gradient ∇ C depends on the inputs, the current values of the model parameters, and the cost function. 3. Update the weights and biases by an amount proportional to ∇ C , i.e. W = W − η ∇ C 4. Repeat from 2 until the cost J stops reducing, or some other pre-defined termination criteria is met. See [22] for more details. 2.5.2 Convergence of gradient descent for a fixed step size Theorem 2.1 Suppose the function f : R n → R is convex and differentiable, and that its gradient is Lipschitz continuous with constant L > 0 , i.e. we have that || ∇ f ( x ) − ∇ f ( y ) || 2 ≤ L || x − y || 2 for any x, y Then if we run gradient descent for k iterations with a fixed step size t ≤ 1 L , it will yield a solution f ( k ) which satisfies f ( x ( k ) ) − f ( x ∗ ) ≤ || x (0) − x ∗ || 2 2 2 tk (2.6) where f ( x ∗ ) is the optimal value. Intuitively, this means that gradient descent is guar- anteed to converge and that it converges with rate O ( 1 k ) See [24] for more details. Using gradient descent to find the minimum cost assumes that the cost function is a convex function. In the real world the cost function is not a purely convex function. This means that when a minimum is found it is possible that it is a local minimum and not the global minimum. 2.6 Backpropagation Backpropagation is the process used to calculate the partial derivatives of the cost function which can then be used in the gradient descent method to minimise the cost function. First some additional variables need to be explained. 14 z is defined as the weighted input for a given neuron. This means that z is defined by z [ l ] = W [ l ] · a [ l − 1] + b [ l ] ∈ R n l (2.7) Then for completion the vector output of a layer can be written as a [ l ] = σ ( z [ l ] ) (2.8) The partial derivatives calculated are δ [ l ] j = ∂C ∂z [ l ] j (2.9) for 1 ≤ j ≤ n l and 2 ≤ l ≤ L An essential part of backpropagation is the component-wise (Hadamard) product of two vectors. In other words let x, y ∈ R n then x ◦ y ∈ R n is defined as ( x ◦ y ) i = x i y i Using the above and the chain rule, the ensuing results can be proved. Lemma 2.2 δ [ L ] = σ ′ ( z [ L ] ) ◦ ( a L − y ) , (2.10) δ [ l ] = σ ′ ( z [ l ] ) ◦ ( W [ l +1] ) T ∂ [ l +1] , (2.11) for 2 ≤ l ≤ L − 1 , ∂C ∂b [ l ] j = δ [ l ] j , (2.12) for 2 ≤ l ≤ L , ∂C ∂w [ l ] jk = δ [ l ] j a [ l − 1] k , (2.13) for 2 ≤ l ≤ L The proof of the above lemma can be directly calculated using the chain rule but is also available in [1]. 15 2.7 Stochastic gradient descent Gradient descent uses the entire data set to minimise the optimisation problem. Clearly for a large data set this could take a considerable amount of time and computational power so a slightly altered version of the method can be used to reduce this strain on resources. Stochastic gradient descent can be described as ”a little more like a drunk man stumbling aimlessly down a hill, but taking quick steps; rather than a carefully calculating man determining the exact downhill direction of each step before taking a very slow and careful step in that direction.” [7] Stochastic gradient is pretty much the same method but instead of using the entire data set to calculate the perfect gradient it uses a single point to calculate the gradient. Obviously performing the calculations using a single point as opposed to the entire data set takes less time and is less computationally expensive. Figure 6: Stochastic Gradient Descent [8] Therefore the stochastic gradient algorithm is: 1. Initialize the weights W and biases b randomly. 16 2. Randomly select a data point. 3. Calculate the gradient ∇ C of cost function with respect to the parameters using the single data point. This is done using partial differentiation: ∇ C = ∂J ∂θ The value of the gradient ∇ C depends on the inputs, the current values of the model parameters, and the cost function. 4. Update the weights and biases by an amount proportional to ∇ C , i.e. W = W − η ∇ C 5. Repeat from 2 until the cost J stops reducing, or some other pre-defined termination criteria is met. See [22] for more details. 3 Distance Functions The next step is to use neural networks to approximate functions. The functions we are going to approximate are distance functions and their related partial differential equations. The following sections introduce distance functions and give examples of approximating them with neural networks. 3.1 Introduction A metric or distance function is a function d(x,y) that defines the distance between elements of a set as a non-negative real number. If the distance is zero, both elements are equivalent under that specific metric. Distance functions thus provide a way to measure how close two elements are, where elements do not have to be numbers but can also be vectors, matrices or arbitrary objects [11]. A distance function is defined as d : X × X → [0 , inf) (3.1) and for all x, y, z ∈ X the following conditions are satisfied d ( x, y ) ≥ 0 non-negativity (3.2) 17 d ( x, y ) = 0 ⇔ x = y identity of indiscernibles (3.3) d ( x, y ) = d ( y, x ) symmetry (3.4) d ( x, y ) ≤ d ( x, z ) + d ( z, y ) triangle inequality (3.5) 3.2 1D example The most widely known distance function is the Euclidean distance also known as Pythagorean distance. This is used to find the standard straight line distance between two points in Euclidean space. Its more generalised form is the L 2 norm. Let x = ( x 1 , x 2 , ..., x n ) and y = ( y 1 , y 2 , ..., y n ) be two points in Euclidean space of n-dimensions then the Euclidean distance between the two points is d ( x, y ) = √ ( x 1 − y 1 ) 2 + ( x 2 − y 2 ) 2 + ... + ( x n − y n ) 2 = √ √ √ √ n ∑ i =1 ( x i − y i ) 2 (3.6) In one dimension the distance between two points x 1 and x 2 is simply the absolute value | x 1 − x 2 | Or for a more general approach the distance from one point x 1 to a general point is | x − x 1 | which can be plotted to better illustrate this function. 18 Figure 7: 1D distance from x 1 = 2 The graph in figure 7 is showing the distance function for a point x 1 = 2 to a general point on the x axis. For example if we add another point x 2 = 0 then it is clear that d ( x 1 , x 2 ) = 2 which is shown by the red dotted line. 3.3 Distance between sets So far in section 3 we have covered the distance between two points, while this is useful it is much more useful to be able to find the distance between a point and a set of points. To enable this a further condition must be added to the definition. Let X be a discrete set of points such that X = { X 1 , X 2 , ..., X n } and each point X i in n-dimensions has n- coordinates eg X 1 = ( x 1 , x 2 , ..., x n ) as before. To be clear all the sets covered in this report are made up of a number of discrete points and are not continuous sets. Now the distance between a general point p and the set X is d ( p, X ) = min ( d ( p, X 1 ) , d ( p, X 2 ) , ..., d ( p, X n )) ∀ X i ∈ X (3.7) 19 Therefore, this means that the distance between the point p and the set X is the shortest distance between p and any point in the set X. Furthermore this condition can be expanded to the distance between two sets. Let X and Y be defined as two sets of points. Then the distance between the two sets is d ( Y, X ) = min ( d ( Y 1 , X 1 ) , d ( Y 1 , X 2 ) , ..., d ( Y 1 , X n ) , d ( Y 2 , X n ) , ..., d ( Y n , X n )) ∀ X i ∈ X and Y i ∈ Y (3.8) Essentially this equation states that the distance between two sets is the minimum distance between any two points in both sets. 3.4 1D example between a point and set To grasp a better understanding of the distance between a point and a set there is a simple example in 1D to look at. It is similar to the previous 1D example but instead of using the single point x 1 , the set X = { X 1 = 1 , X 2 = 4 } is used along with the general point p ∈ R 1 . Using the previous example and expanding we can plot the absolute value of each point in the set to obtain the graph of the distance function for each point individually. 20