Peter Benner, Stefano Grivet-Talocia, Alfio Quarteroni, Gianluigi Rozza, Wil Schilders, Luís Miguel Silveira (Eds.) Model Order Reduction Also of Interest Model Order Reduction. Volume 1: System- and Data-Driven Methods and Algorithms Peter Benner, Stefano Grivet-Talocia, Alfio Quarteroni, Gianluigi Rozza, Wil Schilders, Luís Miguel Silveira (Eds.), 2020 ISBN 978-3-11-050043-1, e-ISBN (PDF) 978-3-11-049896-7, e-ISBN (EPUB) 978-3-11-049771-7 Model Order Reduction. Volume 2: Snapshot-Based Methods and Algorithms Peter Benner, Stefano Grivet-Talocia, Alfio Quarteroni, Gianluigi Rozza, Wil Schilders, Luís Miguel Silveira (Eds.), 2020 ISBN 978-3-11-067140-7, e-ISBN (PDF) 978-3-11-067149-0, e-ISBN (EPUB) 978-3-11-067150-6 Tensor Numerical Methods in Quantum Chemistry Venera Khoromskaia, Boris N. Khoromskij, 2018 ISBN 978-3-11-037015-7, e-ISBN (PDF) 978-3-11-036583-2, e-ISBN (EPUB) 978-3-11-039137-4 Maxwell’s Equations. Analysis and Numerics Ulrich Langer, Dirk Pauly, Sergey Repin (Eds.), 2019 ISBN 978-3-11-054264-6, e-ISBN (PDF) 978-3-11-054361-2, e-ISBN (EPUB) 978-3-11-054269-1 Computational Intelligence. Theoretical Advances and Advanced Applications Dinesh C.S. Bisht, Mangey Ram (Eds.), 2020 ISBN 978-3-11-065524-7, e-ISBN (PDF) 978-3-11-067135-3, e-ISBN (EPUB) 978-3-11-066833-9 Model Order Reduction | Volume 3: Applications Edited by Peter Benner, Stefano Grivet-Talocia, Alfio Quarteroni, Gianluigi Rozza, Wil Schilders, and Luís Miguel Silveira Editors Prof. Dr. Peter Benner Max Planck Institute for Dynamics of Complex Technical Systems Sandtorstr. 1 39106 Magdeburg Germany benner@mpi-magdeburg.mpg.de Prof. Dr. Stefano Grivet-Talocia Politecnico di Torino Dipartimento di Elettronica Corso Duca degli Abruzzi 24 10129 Turin Italy stefano.grivet@polito.it Prof. Alfio Quarteroni Ecole Polytechnique Fédérale de Lausanne (EPFL) and Politecnico di Milano Dipartimento di Matematica Piazza Leonardo da Vinci 32 20133 Milan Italy alfio.quarteroni@polimi.it Prof. Dr. Gianluigi Rozza Scuola Internazionale Superiore di Studi Avanzati - SISSA Via Bonomea 265 34136 Trieste Italy gianluigi.rozza@sissa.it Prof. Dr. Wil Schilders Technische Universiteit Eindhoven Faculteit Mathematik Postbus 513 5600 MB Eindhoven The Netherlands w.h.a.schilders@tue.nl Prof. Dr. Luís Miguel Silveira INESC ID Lisboa IST Técnico Lisboa Universidade de Lisboa Rua Alves Redol 9 1000-029 Lisbon Portugal lms@inesc-id.pt ISBN 978-3-11-050044-8 e-ISBN (PDF) 978-3-11-049900-1 e-ISBN (EPUB) 978-3-11-049775-5 DOI https://doi.org/10.1515/9783110499001 This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. For details go to http://creativecommons.org/licenses/by-nc-nd/4.0/. Library of Congress Control Number: 2020944453 Bibliographic information published by the Deutsche Nationalbibliothek The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data are available on the Internet at http://dnb.dnb.de. © 2021 with the authors, editing © 2021 Peter Benner, Stefano Grivet-Talocia, Alfio Quarteroni, Gianluigi Rozza, Wil Schilders, Luís Miguel Silveira, published by Walter de Gruyter GmbH, Berlin/Boston. The book is published open access at www.degruyter.com. Cover image: Andrea Manzoni, MOX, Department of Mathematics, Politecnico di Milano Typesetting: VTeX UAB, Lithuania Printing and binding: CPI books GmbH, Leck www.degruyter.com Preface to the third volume of Model Order Reduction The third volume of the Model Order Reduction handbook project offers several re- markable instances of applications of model order reduction (MOR) approaches to the solution of problems arising from the most diverse areas of application. Through these examples, we would like to provide the reader with an overview of the maturity of this emerging field and its readiness to address challenging problems of multifaceted com- plexity. We start with several chapter contributions to classical fields of engineering. The first one, by J. Eason and L. Biegler, is on model reduction in the optimization of a variety of heterogeneous chemical processes. In particular, two case studies are presented on CO 2 capture using nonlinear programming and NLP filter models. The second chapter, by B. Lohmann et al., is on MOR in mechanical engineering. Four applications are discussed, concerning the reduction of a thermo-mechanical machining tool of a car body and driver’s seat, of an elastic crankshaft, and a leaf spring model. The third chapter, by E. Deckers et al., presents several case studies of MOR for acoustics and vibrations in mechanical applications. Two different viewpoints are de- veloped: the application of MOR from a purely mathematical perspective and a con- sideration of expected properties of MOR based on physical arguments from the field of mechanics. Two chapters devoted to microelectronics and electromagnetism, a very classical and successful arena for MOR methods, follow. The first of those, by B. Nouri et al., pursues a twofold goal: to describe the context in which the need for MOR arose in microelectronics, and to present an overview of their applications to address the is- sues of high-speed interconnects in microelectronics at various levels of the design hierarchy. The next chapter, by D. Ioan et al., proposes a computer-aided consistent and ac- curate description of the behavior of electromagnetic devices at various speeds or fre- quencies, and describes procedures to generate compact electrical circuits featuring an approximately equivalent behavior. The chapter by M. Yano is on model reduction in computational aerodynamics. The focus is on techniques that are designed to address nonlinearity, limited stability, limited regularity, and a wide range of scales that have been demonstrated successful for multidimensional large-scale aerodynamic flows. The next two chapters address a somehow less conventional field of applications, that of life sciences. The chapter by B. Karasözen is on MOR in neurosciences, more specifically on the exploitation of models of large-scale neuronal networks to provide an accurate and fast prediction of patterns and their propagation in different areas of the brain. The following chapter, by N. Dal Santo et al., introduces MOR methods to face some of the most challenging processes of the cardiovascular system. Two specific Open Access. © 2021 Peter Benner et al., published by De Gruyter. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. https://doi.org/10.1515/9783110499001-201 VI | Preface to the third volume of Model Order Reduction applications are targeted: the simulation of blood flow past a carotid bifurcation and the computation of activation maps in cardiac electrophysiology. The last five chapters address somewhat more methodological issues arising in various scientific, engineering, societal, and economics applications. The chapter by J.-C. Loiseau aims at bypassing some difficulties of classical proper orthogonal decomposition approaches to the solution of fluid dynamics problems by using feature-based manifold modeling in which the low-dimensional attractor and nonlinear dynamics are characterized from experimental data: time-resolved sensor data and optional nontime-resolved particle image velocimetry snapshots. In the chapter by R. Pulch, MOR is used in the framework of uncertainty quan- tification. Established methods like polynomial chaos, stochastic Galerkin, stochastic collocation, and quadrature sampling are reviewed for dynamical systems consisting of ordinary differential equations or differential algebraic equations. Demonstration of applicability is provided on test examples. The chapter by X. Cheng et al. addresses MOR methods for networks that describe a wide class of complex systems composed of many interacting subsystems. First, clustering-based approaches are reviewed, with the aim of reducing the network scale. Then, methods based on generalized balanced truncation that reduce interconnection structures of a network and the dynamics of each subsystem are discussed. The chapter by D. Hartmann et al. presents use cases where MOR is a key enabler for the realization of digital services and the reduction of simulation times and out- lines the potential of MOR in the context of realizing the digital twin vision. The last chapter, by B. Haasdonk, addresses the issue of software. In the first part, as neither full simulation models nor MOR algorithms are to be reprogrammed, but ideally are reused from existing implementations, the interplay of such packages is discussed. Then an overview of the most popular MOR software libraries is provided. We are confident that the vast set of applications discussed here, combined with the broad variety of numerical techniques and software libraries available, will moti- vate the reader to embrace MOR approaches to successfully address complex applica- tions arising in computational science and engineering. Peter Benner, Stefano Grivet-Talocia, Alfio Quarteroni, Gianluigi Rozza, Wil Schilders, Luìs Miguel Silveira Magdeburg, Germany Torino, Milano, Trieste, Italy Eindhoven, The Netherlands Lisbon, Portugal June 2020 Contents Preface to the third volume of Model Order Reduction | V John P. Eason and Lorenz T. Biegler 1 Model reduction in chemical process optimization | 1 B. Lohmann, T. Bechtold, P. Eberhard, J. Fehr, D. J. Rixen, M. Cruz Varona, C. Lerch, C. D. Yuan, E. B. Rudnyi, B. Fröhlich, P. Holzwarth, D. Grunert, C. H. Meyer, and J. B. Rutzmoser 2 Model order reduction in mechanical engineering | 33 Elke Deckers, Wim Desmet, Karl Meerbergen, and Frank Naets 3 Case studies of model order reduction for acoustics and vibrations | 75 Behzad Nouri, Emad Gad, Michel Nakhla, and Ram Achar 4 Model order reduction in microelectronics | 111 Daniel Ioan, Gabriela Ciuprina, and Wilhelmus H. A. Schilders 5 Complexity reduction of electromagnetic systems | 145 Masayuki Yano 6 Model reduction in computational aerodynamics | 201 Bülent Karasözen 7 Model order reduction in neuroscience | 237 Niccolò Dal Santo, Andrea Manzoni, Stefano Pagani, and Alfio Quarteroni 8 Reduced-order modeling for applications to the cardiovascular system | 251 Jean-Christophe Loiseau, Steven L. Brunton, and Bernd R. Noack 9 From the POD-Galerkin method to sparse manifold models | 279 Roland Pulch 10 Model order reduction in uncertainty quantification | 321 Xiaodong Cheng, Jacquelien M. A. Scherpen, and Harry L. Trentelman 11 Reduced-order modeling of large-scale network systems | 345 Dirk Hartmann, Matthias Herz, Meinhard Paffrath, Joost Rommes, Tommaso Tamarozzi, Herman Van der Auweraer, and Utz Wever VIII | Contents 12 Model order reduction and digital twins | 379 Bernard Haasdonk 13 MOR software | 431 Index | 461 John P. Eason and Lorenz T. Biegler 1 Model reduction in chemical process optimization Abstract: Chemical processes are often described by heterogeneous models that range from algebraic equations for lumped parameter systems to black-box models for PDE systems. The integration, solution, and optimization of this ensemble of process mod- els is often difficult and computationally expensive. As a result, reduction in the form of reduced-order models and data-driven surrogate models is widely applied in chem- ical processes. This chapter reviews the development and application of reduced mod- els (RMs) in this area, as well as their integration to process optimization. Special at- tention is given to the construction of reduced models that provide suitable represen- tations of their detailed counterparts, and a novel trust region filter algorithm with reduced models is described that ensures convergence to the optimum with truth mod- els. Two case studies on CO 2 capture are described and optimized with this trust re- gion filter method. These results demonstrate the effectiveness and wide applicability of the trust region approach with reduced models. Keywords: Model reduction, trust region methods, POD, equation-oriented modeling, glass box, black box, nonlinear programming, NLP filter methods MSC 2010: 35B30, 37M99, 41A05, 65K99, 93A15, 93C05 1.1 Introduction Chemical processes incorporate advanced technologies that need to be modeled, inte- grated, and optimized. To address these needs, state-of-the-art nonlinear optimization algorithms can now solve models with millions of decision variables and constraints. Correspondingly, the computational cost of solving discrete optimization problems has been reduced by several orders of magnitude [14]. Moreover, these algorithmic ad- vances have been realized through software modeling frameworks that link optimiza- tion models to efficient nonlinear programming (NLP) and mixed-integer NLP solvers. On the other hand, these advances are enabled through modeling frameworks that require optimization models to be formulated as well-posed problems with exact first and second derivatives. Despite these advances, multiscale processes still need effective problem formu- lation and modeling environments. At the process optimization level, multiscale in- John P. Eason, Exenity, LLC, Pittsburgh, PA, USA Lorenz T. Biegler, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh, PA, USA Open Access. © 2021 John P. Eason and Lorenz T. Biegler, published by De Gruyter. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. https://doi.org/10.1515/9783110499001-001 2 | J. P. Eason and L. T. Biegler tegration is required to model complex transport and fluid flow phenomena. For in- stance, optimization models for advanced power generation processes, such as in Fig- ure 1.4, comprise a heterogeneous modeling environment with algebraic equation (AE) models, such as heat exchangers, compressors, and expanders, as well as large, non- linear partial differential AE (PDAE) models. These include multiphase reactor models such as fluidized beds, combustors, and gasifiers. Because of the substantial complex- ity of the associated model solvers, computational costs for multiscale process opti- mization are prohibitive. While equation-oriented flowsheet models may take only a few CPU seconds to solve, a PDAE combustion or gasification model alone may require many CPU hours or even days [48]. We denote these prohibitive models as truth mod- els , which require model reduction. These models often follow a “bottom up” model- ing approach, where DAEs or PDAEs derive from fundamental physical laws. Models at this higher fidelity can include transport behavior and detailed reaction kinetics, which require computationally costly simulations. The process flowsheet in Figure 1.4 shows a detailed boiler model, pumps, com- pressors, turbines, heat exchangers, and mixing and splitting junctions, and the re- sulting model comprises equations that connect process units with process streams, conservation laws (mass, momentum, energy) within each unit, constitutive equa- tions that describe physical phenomena, including transport behavior, equilibrium and reaction kinetics, and physical properties for materials and mixtures. This en- semble of PDAEs/DAEs/AEs within a chemical process is typically broader than many PDAE models in other domains, where model reduction can proceed in a more struc- tured manner. Model reduction in chemical process simulation and optimization can be effected in a number of ways. These include: – Simplifying assumptions of physics-based models by removing terms in PDAEs that reflect negligible behavior in time and length scales. Often, these “shortcut” models can only be defined (and validated) over limited domains of applicability. – Time scale reduction, where dynamic behaviors that are either too fast or too slow in the range of interest are eliminated [56]. – Data-driven input–output models which are generally unstructured, require few assumptions, and lead to general-purpose applications [63]. Process optimization with reduced models poses a special challenge as most reduced models are interpolative, while optimization requires extrapolation. Generally, we ex- pect extrapolative capabilities to be captured by physics-based truth models, as they are based on fundamental phenomena and comprise constituent models that have been validated from many domains. To develop and preserve these capabilities, inter- polative reduced models must be reconstructed and recalibrated with truth models, in order to remain consistent over the convergence path of the optimization solver. To develop an integrated optimization approach we expect that an RM-based strategy is allowed to evaluate and compare information from the truth models to cap- 1 Model reduction in chemical process optimization | 3 ture relevant multiscale phenomena such as complex fluid flow, particle mechanics, and dynamic operation within the process optimization. Moreover, general strategies can be applied to create RMs through a number of physics-based or data-driven model reductions. To develop this strategy, this chapter considers the properties needed for the RM-based optimization framework to converge to the optimum of the original system models, as well as the construction of RMs that balance model accuracy with computational cost during the optimization. The next section briefly reviews developments in model reduction that include proper orthogonal decomposition (POD) and data-driven models. Section 1.3 then presents trust algorithms based on reduced models; these depend on whether gra- dients are available from the truth models or not. Section 1.4 then presents two case studies that describe the performance of these methods on large-scale process opti- mization problems. Most importantly, both methods work directly with the RM, and they also guarantee convergence to the optimum of the truth model. Finally, conclu- sions and future work are discussed in Section 1.5. 1.2 Model reduction for simulation Model reduction is a broad topic with contributions from many different research com- munities. There has always been a balance between model fidelity and computational tractability since the earliest use of computers in chemical engineering. For instance, for vapor-liquid equilibrium, which is the basic building block of all process models, reduced physical property models are often constructed through simplifying thermo- dynamic assumptions. In early studies [8, 17, 50] these reduced models proved very effective to accelerate calculations without sacrificing much accuracy. While comput- ing hardware has improved substantially since that time, these early works show how model reduction can be used to solve problems that otherwise may be intractable. Beyond the use of simplifying fundamental assumptions with “shortcut” models, general model reduction strategies can be partitioned into two categories: model order reduction and data-driven model reduction. We classify model order reduction methods as projection-based techniques ap- plied to an accessible state-space description of the truth model, with an explicit projection applied to reduce the state-space dimension [11]. For fully accessible (i. e., equation-based) state-space truth models, system-theoretic model order reduction exploits the specific dynamic system structure and includes balanced truncation and rational interpolation based on Gramians and transfer functions. These usually apply to linear systems, although they have been extended to bilinear systems and quadratic-in-state systems. Moreover, they are widely applied in circuit theory, sig- nal processing, structural mechanics, and linear, optimal control. A comprehensive review of system-theoretic methods can be found in [9]. 4 | J. P. Eason and L. T. Biegler When the truth model is a large-scale system of DAEs, system-theoretic methods can take advantage of that structure. In model-based reduction for chemical engineer- ing, the truth model is also exploited to guide the projection steps [56], but system- theoretic model order reduction is seldom applied to these models. This is mainly because of the high model nonlinearity and limited accessibility of the chemical pro- cess model equations, often embedded within “gray-box” procedures. To handle these truth models, snapshot-based projection methods, such as reduced basis or POD, are applied, with sampled snapshot solutions over the parameter domain and space (or time) domains. Among these, POD is the most generally applicable as it relies only on snapshots of the underlying simulation code. POD has been demonstrated effectively in many areas including fluid dynamics, structural dynamics, thermal modeling, and atmospheric modeling. As a result, it is frequently applied for model-based reduction of large, nonlinear truth models in chemical engineering. 1.2.1 Proper orthogonal decomposition POD, also known as Karhunen–Loève decomposition, can reduce large spatially dis- tributed models to much smaller models. POD models are formulated by projecting the PDAE system onto a set of basis functions, which are themselves generated from the numerical solution of the original equations. Applications are numerous, with ex- amples including [12, 22, 24, 28, 42, 44, 47, 53, 57, 60, 62, 71, 73, 21]. In addition, many studies report the use of POD for optimization. However, the basis functions used in POD are typically determined from a finite set of simulations of the full-scale PDAE system. This greatly reduces the system size, but the accuracy of the POD approxi- mation is inherently local in nature. Therefore optimization may have a tendency to extrapolate far from data or otherwise exploit approximation errors to find artificially improved solutions. Nevertheless, several studies report successful use of model order reduction in optimization and control. Examples in chemical processes include optimization of diffusion–reaction processes [5], transport-reaction processes [10], chemical vapor de- position [66], and thermal processing of foods [6]. As detailed in [67], POD models are constructed from Galerkin projections of the PDAE model onto a set of basis functions. These basis functions are often generated empirically from numerical solutions of the truth model, through the method of snap- shots . The aim is to find a low-dimensional basis that can capture most information of the spatial distribution. To do so, one first gathers snapshot sets which consist of spatial solutions of the original PDAE system at several time instants as determined by numerical simulation. Let the snapshot matrix be given as Z = { z ( ξ , t 1 ), . . . , z ( ξ , t N t ) } , (1.1) 1 Model reduction in chemical process optimization | 5 where each snapshot z ( ξ , t j ) is a column vector representing the (discretized) spatial profile at time t i . There are N t snapshots and N ξ spatial discretization nodes. After gathering a set of snapshots, the singular value decomposition of the snap- shot matrix Z is given as Z = UDV T = ∑ i σ i u i v T i (1.2) The first M vectors { u i } M i = 1 , where M ≤ N ξ , of the orthogonal matrix U represent the desired set of POD basis functions (or basis vectors in the case of discretized spatial dimensions). From this point we refer to these basis functions as φ i ( ξ ) , since each one describes the behavior in the spatial dimensions. To determine the number of basis vectors M , the projection error can be approxi- mated as ε POD ( M ) = N ξ ∑ i = M + 1 σ 2 i (1.3) Then, the interrelation between accuracy and dimension of the POD-based reduced- order models can be balanced by a predetermined threshold. The error bound λ is de- fined as λ ( M ) = 1 − ∑ M i = 1 σ 2 i ∑ N ξ i = 1 σ 2 i (1.4) M is then chosen such that λ ( M ) ≤ λ ∗ for a desired threshold λ ∗ [64]. Typically, M can be chosen rather small compared to N ξ while still keeping λ close to zero (typically < 10 − 3 ). After computing the POD basis set, a reduced-order model is derived by projecting the PDAEs of the system onto the corresponding POD subspace. This means that we seek an approximation of the form z ( ξ , t ) ≈ z POD ( ξ , t ) = M ∑ i = 1 a i ( t ) φ i ( ξ ). (1.5) To demonstrate how the Galerkin approach is applied to determine the coefficients a i ( t ) , consider a PDE in the following form: 𝜕 z 𝜕 t = f ( z , 𝜕 z 𝜕 ξ ) (1.6) Using the POD basis functions as the weighted basis functions for the Galerkin projec- tion, we obtain the system da i dt = ∫ f ( M ∑ j = 1 a j ( t ) φ j ( ξ ), M ∑ j = 1 a j ( t ) dφ j dξ ) φ i ( ξ ) dξ , i = 1 . . . M , (1.7) 6 | J. P. Eason and L. T. Biegler leading to a set of M ordinary differential equations (ODEs). If the spatially discretized system were directly solved with the method of lines, it would consist of N ξ ODEs. Since M is normally much less than N ξ , POD can create a much smaller model that still maintains reasonable accuracy. 1.2.2 Data-driven reduction Data-driven reduction methods have been successfully applied to truth models where projections and basis functions cannot be generated from the model equations, and only input/output responses are available from a black-box code. This black box may be sampled, and regression/interpolation approaches can be used to fit the sampled data. The resulting surrogate model replaces the truth model for simulation, optimiza- tion, or other analysis. There is considerable flexibility in the functional form and fitting methods used for surrogate construction, and this flexibility can be used to customize an approach suitable for a particular problem. Simpson et al. [63] provide a review of the field, which outlines several important steps and existing surrogate modeling frameworks. The main steps of surrogate model construction include experi- mental design, model selection, and model fitting. Several established methodologies suggest combinations of choices for each of these three steps. For example, response surface methodology, typically used in optimization settings, uses central compos- ite designs in combination with quadratic models constructed with least-squares re- gression. The central composite design helps determine curvature information for the quadratic models. A more complete description can be found in Myers and Mont- gomery [59]. In some ways, response surface methodology is a predecessor to the trust region-based methods that will be discussed in Section 1.3. Other surrogate model- ing approaches include Gaussian process regression (including kriging) and artificial neural networks. These methods often perform better with space-filling or sequential experimental designs [45, 36]. Recent work also examines the role of model complexity in surrogate modeling. When simpler functional forms are preferred, best-subset techniques combined with integer programming can be used to fit models [29, 72]. Moreover, recent develop- ments in machine learning have led to a wealth of new methods for model reduction [58, 65]. An example that demonstrates many concepts from data-driven model reduction may be found in [48]. That work proposes a model reduction method for distributed parameter systems based on principal component analysis and neural networks. The reduced model is designed to represent the spatially distributed states z of the sys- tem as functions of the inputs w , including boundary conditions, equipment param- eters, operating conditions, and input stream information. Similar to POD, this PCA approach seeks to represent the states in terms of a set of empirically determined ba- sis functions. First a set of snapshots is determined by running the truth model at 1 Model reduction in chemical process optimization | 7 various input conditions W = { w 1 . . . u n n } , giving the snapshot set Z = { z ( ξ , w 1 ), . . . , z ( ξ , w N ) } , (1.8) where each snapshot z ( ξ , w i ) is a column vector representing the spatial profile at in- put point w i . There are n n snapshots and N ξ spatial discretization nodes. The basis functions are obtained in the same manner as discussed with equations (1.2), (1.3), and (1.4). After obtaining the reduced basis set φ i ( ξ ) (the principal components), the reduced model is expressed as z ( ξ , w ) ≈ z PCA ( ξ , w ) = M ∑ i = 1 a i ( w ) φ i ( ξ ). (1.9) Whereas POD determines the coefficients a i through Galerkin projection of the truth model equations onto the basis set, the PCA-RM approach of [48] uses neural net- works. In other words, each function a i ( w ) is the result of training a neural network to capture the nonlinear relation between the input variables and the states represented with the principal components. This PCA-RM approach was applied to a CFD model of an entrained flow gasifier, embedded within an open-equation advanced power plant model [49]. The truth model was implemented in Fluent and takes around 20 CPU hours to solve. With both high computational cost and the use of commercial tools that may be difficult to use for custom analysis and simulations, this problem has both motivating features for the use of reduced models. There were three input variables for the truth model, including the water concentration in the slurry, oxygen to coal feed ratio, and the ratio of coal injected at the first feed stage. As described in [49], the resulting PCA-based RM had very good accuracy, as validated by leave-one-out cross- validation. 1.3 Process optimization using reduced models Many process engineering applications on reduced modeling involve optimization for- mulations. One of the challenges in this field is the size of the problems created if the system is fully discretized before optimization. In addition, optimization routines are not easily customized to handle a particular problem as simulation. Here, a reduction in problem size can greatly speed solutions to enable real time application. However, despite significant effort in building reduced models, it is known that using an RM in optimization can lead to inaccurate answers. Small errors introduced by the RM approximation can propagate through to large errors in the optimum so- lution. This is worsened by the optimization’s tendency to exploit error to artificially improve the objective function, and hence optimization may terminate in regions of 8 | J. P. Eason and L. T. Biegler poor RM accuracy. The RM can be refined sequentially during optimization, using in- formation from the truth model to improve inaccuracies. However, these iterative im- provements offer no convergence guarantees to the optimum of the high-fidelity opti- mization problem, even if it does converge at a point where the RM matches the truth model. The nonconvergence behavior can be observed through a toy problem in [15, 16], shown as follows: min f ( x ) = ( x ( 1 ) ) 2 + ( x ( 2 ) ) 2 (1.10) s. t. t ( x ) = x ( 2 ) − ( x ( 1 ) ) 3 − ( x ( 1 ) ) 2 − 1 = 0 , where we denote the cubic function t ( x ) as the truth model. As shown in Figure 1.1, the problem (1.10) has a global minimum, ( x ∗ ) = ( 0 , 1 ), f ( x ∗ ) = 1, and a local maximum, ( x ∗ ) = ( − 1 , 1 ), f ( x ∗ ) = 2. Now consider the corresponding RM-based problem given by min ̂ f ( x ) = ( x ( 1 ) ) 2 + ( x ( 2 ) ) 2 (1.11) s. t. r ( x ) = x ( 2 ) − x ( 1 ) − b = 0 , where we denote the linear function r ( x ) as the RM with an adjustable constant b . It is straightforward to show that the solution to (1.11) is given by x ∗ = ( − b / 2 , b / 2 ) , and f ( x ∗ ) = b 2 / 2. Moreover, as shown by the steps in Figure 1.1(a), the RM-based algorithm proceeds at iteration k by choosing b k so that r ( x k ) = t ( x k ) = 0. Then (1.11) is solved to obtain the next iterate x k + 1 . However, this approach does not guarantee convergence to the optimum for the truth model. For instance, if we start from ( 0 , 1 ) , the global minimum solution of (1.10), Figure 1.1(a) shows that the iterates x k actually move away from this solution and eventually converge to a nonoptimal point where t ( ̄ x ) = r ( ̄ x ) , b = 2, and ̄ x = ( − 1 , 1 ) . For this example, it can be seen that this point is actually a local maximum of (1.10). This behavior arises because optimality conditions rely on derivative information, not simple matching in function values. The most common approach to “safe” optimization with reduced models is to use a trust region method. Instead of approximating a black-box function over the entire domain of decision variables, a reduced model is constructed to locally approximate over this trust region. Assuming sufficient data are available, smaller domains can lead to lower absolute error in the reduced model and the choice of functional form becomes less critical with the restricted approximation. Trust region methods exploit this feature by adapting the trust region radius during the optimization process. Most trust region algorithms adopt the following outline. As a general example assume that the goal is to solve the following optimization problem: min x f ( x ) s. t. g ( x ) ≤ 0 (1.12) 1 Model reduction in chemical process optimization | 9 Figure 1.1: Characteristics of toy example. (a) Convergence failure. (b) Convergence with trust region method. The initial point is denoted as x 0 . At each iteration, a trust region method will first construct an approximation of (1.12) that is valid on the trust region at iteration k . In other words, identify ̂ f and ̂ g such that ̂ f ( x ) ≈ f ( x ) and ̂ g ( x ) ≈ g ( x ) for x ∈ B ( x k , Δ k ), where B ( x k , Δ k ) = { x : ‖ x − x k ‖ ≤ Δ k } is the trust region at iteration k . In classical trust region methods for nonlinear optimization, a quadratic approximation of the objec- tive function is constructed, while the constraint functions are linearized. However, alternative forms may be used and the characteristics of the optimization algorithm change depending on the type of approximation (type of RM) and the nature of accu- racy required for the approximation. The second step is to solve the trust region subproblem . This means that the re- duced models are used to optimize the function within the trust region, where suffi- cient accuracy is assumed. For our example problem (1.12), the trust region subprob- lem is min x ̂ f ( x ) s. t. ̂ g ( x ) ≤ 0 , ‖ x − x k ‖ ≤ Δ k (1.13) The trust region constraint helps prevent the algorithm from extrapolating into re- gions where the RM is not accurate, and provides a globalization mechanism to make 10 | J. P. Eason and L. T. Biegler sure the algorithm converges. On the other hand, algorithms vary on how the trust re- gion subproblem is solved, and even the type of subproblem considered (constrained vs. unconstrained, convex or nonconvex). The final step is to evaluate the solution proposed by the trust region subproblem, denoted as ̄ x . Using recourse to the truth model, the solution can be evaluated in terms of accuracy. Depending on the improvement at ̄ x , the algorithm determines that either x k + 1 = ̄ x , in which case we say that the step was accepted, or x k + 1 = x k , in which case we say that the step was rejected. The algorithm also determines the trust region radius for the following iteration Δ k + 1 . There is significant freedom in algorithm design to handle this last step, with various ways to decide on when to accept or reject the step and how to update the trust region radius. Alexandrov et al. [3] applied the trust region concept to general reduced models in engineering. They considered the task of unconstrained minimization using arbi- trary approximation functions in place of the actual function. Reduced models are constructed so that the function and gradient values of the reduced model match those of the truth model at the center of the trust region. The trust region subproblem was the minimization of the reduced model subject to the trust region constraint. Stan- dard methods for unconstrained trust region methods were used to evaluate the step, including the ratio test [26]. Convergence was proved to the optimum with the truth model. These concepts were extended to classes of constrained optimization problems in the DAKOTA package [43]. Moreover, early work in [4, 39] demonstrated the use of POD reduced models with a trust region method. This algorithm was shown to be con- vergent under the assumption that the gradients of the POD model are sufficiently accurate, although this condition may be difficult to verify in practice. To illustrate the RM-based trust region approach, we revisit problem (1.10) but consider the NLP associated with the following RM: min ̂ f ( x ) = ( x ( 1 ) ) 2 + ( x ( 2 ) ) 2 (1.14) s. t. r ( x ) = x ( 2 ) − ax ( 1 ) − b , where the RM has adjustable constants a and b . The corresponding trust region prob- lem is given by min s ( x ( 1 ) k + s ) 2 + ( a k ( x ( 1 ) k + s ) + b k ) 2 (1.15) s. t. ‖ s ‖ ∞ ≤ Δ k , (1.16) and the progress of the trust region algorithm is sketched in Figure 1.1(b). Using Δ 0 = 0 8, the trust region algorithm converges to a tight tolerance after 20 iterations [16]. For the case where gradients are not available from the truth model, Conn et al. [27] discuss sufficient accuracy conditions on the reduced model to guarantee conver- gence. This condition, called the κ -fully linear property, can be verified for data-driven 1 Model reduction in chemical process optimization | 11 reduced models (e. g., polynomial interpolation). The κ -fully linear property (see (1.20) below) dictates how the differences between reduced and truth models must scale di- rectly with the trust region radius. In this way, shrinking the trust region allows first- order optimality to be guaranteed in the limit. These derivative-free trust region ideas were extended by March and Wilcox [55] to consider the use of multifidelity models. In that study, the reduced model was a coarse discretization of the PDE system. Wild, Regis, and Shoemaker also use the framework of κ -fully linear models to develop an al- gorithm with radial basis functions as RMs [70]. March and Wilcox [55] use constrained trust region subproblems with reduced models, with globalization managed with the use of a merit function. More recent extensions of these optimization strategies and applications for PDAE systems are reviewed in [61]. While the RM-based trust region methods guarantee convergence to the truth model-based optimization problem, the RM itself needs to be updated frequently, po- tentially for each trust region subproblem. To address this limitation, recent work on construction of RMs with embedded parameters (e. g., decision variables) appear to be particularly promising [11]. In particular, the idexempirical interpolation method (EIM) and the discrete EIM (DEIM) develop RMs that contain an interpolant for pa- rameter values [7, 25]. This allows fast updates of the RM as part of the optimization process, with much less evaluation of the truth models. Finally, for the optimization of multiscale chemical processes, Caballero and Grossmann [23] use kriging models to represent unit operations for process opti- mization and trust regions were used to shrink the domain of the kriging models, though convergence to local optima was not proved. Agarwal et al. [1] consider the optimization of periodic adsorption processes with POD-based reduced models. For this system they analyze and present convergence results in [2] for constrained sub- problems when derivatives of the truth models are known. For the related simulated moving bed (SMB) process with linear isotherms, Li et al. [52] develop and apply system-theoretic model order reduction methods to accelerate the computation of the cyclic steady states and optimize the SMB system. In a related study, these authors develop and demonstrate an efficient trust region method with surrogate (POD as well as coarse discretization) models to optimize the SMB proc