Principles of Scientific Computing David Bindel and Jonathan Goodman last revised February 2009, last printed March 6, 2009 Preface i ii PREFACE This book grew out of a one semester first course in Scientific Computing for graduate students at New York University It represents our view of how advanced undergraduates or beginning graduate students should start learning the subject, assuming that they will eventually become professionals It is a common foundation that we hope will serve people heading to one of the many areas that rely on computing This generic class normally would be followed by more specialized work in a particular application area We started out to write a book that could be covered in an intensive one semester class The present book is a little bigger than that, but it still benefits or suffers from many hard choices of material to leave out Textbook authors serve students by selecting the few most important topics from very many important ones Topics such as finite element analysis, constrained optimization, algorithms for finding eigenvalues, etc are barely mentioned In each case, we found ourselves unable to say enough about the topic to be helpful without crowding out the material here Scientific computing projects fail as often from poor software as from poor mathematics Well-designed software is much more likely to get the right answer than naive “spaghetti code” Each chapter of this book has a Software section that discusses some aspect of programming practice Taken together, these form a short course on programming practice for scientific computing Included are topics like modular design and testing, documentation, robustness, performance and cache management, and visualization and performance tools The exercises are an essential part of the experience of this book Much important material is there We have limited the number of exercises so that the instructor can reasonably assign all of them, which is what we In particular, each chapter has one or two major exercises that guide the student through turning the ideas of the chapter into software These build on each other as students become progressively more sophisticated in numerical technique and software design For example, the exercise for Chapter draws on an LLt factorization program written for Chapter as well as software protocols from Chapter This book is part treatise and part training manual We lay out the mathematical principles behind scientific computing, such as error analysis and condition number We also attempt to train the student in how to think about computing problems and how to write good software The experiences of scientific computing are as memorable as the theorems – a program running surprisingly faster than before, a beautiful visualization, a finicky failure prone computation suddenly becoming dependable The programming exercises in this book aim to give the student this feeling for computing The book assumes a facility with the mathematics of quantitative modeling: multivariate calculus, linear algebra, basic differential equations, and elementary probability There is some review and suggested references, but nothing that would substitute for classes in the background material While sticking to the prerequisites, we use mathematics at a relatively high level Students are expected to understand and manipulate asymptotic error expansions, to perturbation theory in linear algebra, and to manipulate probability densities iii Most of our students have benefitted from this level of mathematics We assume that the student knows basic C++ and Matlab The C++ in this book is in a “C style”, and we avoid both discussion of object-oriented design and of advanced language features such as templates and C++ exceptions We help students by providing partial codes (examples of what we consider good programming style) in early chapters The training wheels come off by the end We not require a specific programming environment, but in some places we say how things would be done using Linux Instructors may have to help students without access to Linux to some exercises (install LAPACK in Chapter 4, use performance tools in Chapter 9) Some highly motivated students have been able learn programming as they go The web site http://www.math.nyu.edu/faculty/goodman/ScientificComputing/ has materials to help the beginner get started with C++ or Matlab Many of our views on scientific computing were formed during as graduate students One of us (JG) had the good fortune to be associated with the remarkable group of faculty and graduate students at Serra House, the numerical analysis group of the Computer Science Department of Stanford University, in the early 1980’s I mention in particularly Marsha Berger, Petter Bjăorstad, Bill Coughran, Gene Golub, Bill Gropp, Eric Grosse, Bob Higdon, Randy LeVeque, Steve Nash, Joe Oliger, Michael Overton, Robert Schreiber, Nick Trefethen, and Margaret Wright The other one (DB) was fotunate to learn about numerical technique from professors and other graduate students Berkeley in the early 2000s, including Jim Demmel, W Kahan, Beresford Parlett, Yi Chen, Plamen Koev, Jason Riedy, and Rich Vuduc I also learned a tremendous amount about making computations relevant from my engineering colleagues, particularly Sanjay Govindjee, Bob Taylor, and Panos Papadopoulos Colleagues at the Courant Institute who have influenced this book include Leslie Greengard, Gene Isaacson, Peter Lax, Charlie Peskin, Luis Reyna, Mike Shelley, and Olof Widlund We also acknowledge the lovely book Numerical Methods by Germund Dahlquist and ˚ Ake Bjăork [2] From an organizational standpoint, this book has more in common with Numerical Methods and Software by Kahaner, Moler, and Nash [13] iv PREFACE Contents Preface i Introduction Sources of Error 2.1 Relative error, absolute error, and cancellation 2.2 Computer arithmetic 2.2.1 Bits and ints 2.2.2 Floating point basics 2.2.3 Modeling floating point error 2.2.4 Exceptions 2.3 Truncation error 2.4 Iterative methods 2.5 Statistical error in Monte Carlo 2.6 Error propagation and amplification 2.7 Condition number and ill-conditioned problems 2.8 Software 2.8.1 General software principles 2.8.2 Coding for floating point 2.8.3 Plotting 2.9 Further reading 2.10 Exercises 7 8 10 12 13 14 15 15 17 19 19 20 21 22 25 Local Analysis 3.1 Taylor series and asymptotic expansions 3.1.1 Technical points 3.2 Numerical Differentiation 3.2.1 Mixed partial derivatives 3.3 Error Expansions and Richardson Extrapolation 3.3.1 Richardson extrapolation 3.3.2 Convergence analysis 3.4 Integration 3.5 The method of undetermined coefficients 3.6 Adaptive parameter estimation 29 32 33 36 39 41 43 44 45 52 54 v vi CONTENTS 3.7 3.8 3.9 Software 3.7.1 Flexibility and modularity 3.7.2 Error checking and failure reports 3.7.3 Unit testing References and further reading Exercises 57 57 59 61 62 62 Linear Algebra I, Theory and Conditioning 4.1 Introduction 4.2 Review of linear algebra 4.2.1 Vector spaces 4.2.2 Matrices and linear transformations 4.2.3 Vector norms 4.2.4 Norms of matrices and linear transformations 4.2.5 Eigenvalues and eigenvectors 4.2.6 Differentiation and perturbation theory 4.2.7 Variational principles for the symmetric eigenvalue problem 4.2.8 Least squares 4.2.9 Singular values and principal components 4.3 Condition number 4.3.1 Linear systems, direct estimates 4.3.2 Linear systems, perturbation theory 4.3.3 Eigenvalues and eigenvectors 4.4 Software 4.4.1 Software for numerical linear algebra 4.4.2 Linear algebra in Matlab 4.4.3 Mixing C++ and Fortran 4.5 Resources and further reading 4.6 Exercises Linear Algebra II, Algorithms 5.1 Introduction 5.2 Counting operations 5.3 Gauss elimination and LU decomposition 5.3.1 A × example 5.3.2 Algorithms and their cost 5.4 Cholesky factorization 5.5 Least squares and the QR factorization 5.6 Software 5.6.1 Representing matrices 5.6.2 Performance and caches 5.6.3 Programming for performance 5.7 References and resources 5.8 Exercises 67 68 69 69 72 74 76 77 80 82 83 84 87 88 90 90 92 92 93 95 97 98 105 106 106 108 108 110 112 116 117 117 119 121 122 123 CONTENTS vii Nonlinear Equations and Optimization 6.1 Introduction 6.2 Solving a single nonlinear equation 6.2.1 Bisection 6.2.2 Newton’s method for a nonlinear equation 6.3 Newton’s method in more than one dimension 6.3.1 Quasi-Newton methods 6.4 One variable optimization 6.5 Newton’s method for local optimization 6.6 Safeguards and global optimization 6.7 Determining convergence 6.8 Gradient descent and iterative methods 6.8.1 Gauss Seidel iteration 6.9 Resources and further reading 6.10 Exercises 125 126 127 128 128 130 131 132 133 134 136 137 138 138 139 Approximating Functions 7.1 Polynomial interpolation 7.1.1 Vandermonde theory 7.1.2 Newton interpolation formula 7.1.3 Lagrange interpolation formula 7.2 Discrete Fourier transform 7.2.1 Fourier modes 7.2.2 The DFT 7.2.3 FFT algorithm 7.2.4 Trigonometric interpolation 7.3 Software 7.4 References and Resources 7.5 Exercises Dynamics and Differential Equations 8.1 Time stepping and the forward Euler 8.2 Runge Kutta methods 8.3 Linear systems and stiff equations 8.4 Adaptive methods 8.5 Multistep methods 8.6 Implicit methods 8.7 Computing chaos, can it be done? 8.8 Software: Scientific visualization 8.9 Resources and further reading 8.10 Exercises 143 145 145 147 151 151 152 155 159 161 162 162 162 method 165 167 171 173 174 178 180 182 184 189 189 viii Monte Carlo methods 9.1 Quick review of probability 9.1.1 Probabilities and events 9.1.2 Random variables and distributions 9.1.3 Common random variables 9.1.4 Limit theorems 9.1.5 Markov chains 9.2 Random number generators 9.3 Sampling 9.3.1 Bernoulli coin tossing 9.3.2 Exponential 9.3.3 Markov chains 9.3.4 Using the distribution function 9.3.5 The Box Muller method 9.3.6 Multivariate normals 9.3.7 Rejection 9.3.8 Histograms and testing 9.4 Error bars 9.5 Variance reduction 9.5.1 Control variates 9.5.2 Antithetic variates 9.5.3 Importance sampling 9.6 Software: performance issues 9.7 Resources and further reading 9.8 Exercises CONTENTS 195 198 198 199 202 203 204 205 206 206 206 207 208 209 209 210 213 214 215 216 216 217 217 217 218 ... sizes of numbers We should have a feeling for whether 4.5 × 10−6 is a plausible roundoff error if the operands are of the order of magnitude of 50 Is it plausible to compute the inverse of an... models He or she needs the principles of scientific computing to find out what the models predict Scientific computing is challenging partly because it draws on many parts of mathematics and computer... Scientific computing projects fail as often from poor software as from poor mathematics Well-designed software is much more likely to get the right answer than naive “spaghetti code” Each chapter of