Đang tải... (xem toàn văn)
An introduction to statistical signal processing
An Introduction to Statistical Signal Processing Pr(f ∈ F ) = P ({ω : ω ∈ F }) = P (f −1 (F )) f −1 (F ) f F ✲ November 22, 2003 i ii An Introduction to Statistical Signal Processing Robert M. Gray and Lee D. Davisson Information Systems Laboratory Department of Electrical Engineering Stanford University and Department of Electrical Engineering and Computer Science University of Maryland iii iv c 2003 by Cambridge University Press v to our Families vi Contents Preface page xii Glossary xvii 1 Introduction 1 2 Probability 12 2.1 Introduction 12 2.2 Spinning Pointers and Flipping Coins 16 2.3 Probability Spaces 27 2.3.1 Sample Spaces 32 2.3.2 Event Spaces 36 2.3.3 Probability Measures 49 2.4 Discrete Probability Spaces 53 2.5 Continuous Probability Spaces 64 2.6 Independence 81 2.7 Elementary Conditional Probability 82 2.8 Problems 86 3 Random Objects 96 3.1 Introduction 96 3.1.1 Random Variables 96 3.1.2 Random Vectors 101 3.1.3 Random Processes 105 3.2 Random Variables 109 3.3 Distributions of Random Variables 119 3.3.1 Distributions 119 3.3.2 Mixture Distributions 124 vii viii Contents 3.3.3 Derived Distributions 127 3.4 Random Vectors and Random Processes 132 3.5 Distributions of Random Vectors 135 3.5.1 Multidimensional Events 136 3.5.2 Multidimensional Probability Functions 137 3.5.3 Consistency of Joint and Marginal Distri- butions 139 3.6 Independent Random Variables 146 3.7 Conditional Distributions 149 3.7.1 Discrete Conditional Distributions 150 3.7.2 Continuous Conditional Distributions 152 3.8 Statistical Detection and Classification 155 3.9 Additive Noise 158 3.10 Binary Detection in Gaussian Noise 167 3.11 Statistical Estimation 168 3.12 Characteristic Functions 170 3.13 Gaussian Random Vectors 176 3.14 Simple Random Processes 178 3.15 Directly Given Random Processes 182 3.15.1 The Kolmogorov Extension Theorem 182 3.15.2 IID Random Processes 182 3.15.3 Gaussian Random Processes 183 3.16 Discrete Time Markov Processes 184 3.16.1 A Binary Markov Process 184 3.16.2 The Binomial Counting Process 187 3.16.3 Discrete Random Walk 191 3.16.4 The Discrete Time Wiener Process 192 3.16.5 Hidden Markov Models 194 3.17 Nonelementary Conditional Probability 194 3.18 Problems 196 4 Expectation and Averages 213 4.1 Averages 213 4.2 Expectation 217 4.3 Functions of Random Variables 221 4.4 Functions of Several Random Variables 228 4.5 Properties of Expectation 229 Contents ix 4.6 Examples 232 4.6.1 Correlation 232 4.6.2 Covariance 235 4.6.3 Covariance Matrices 236 4.6.4 Multivariable Characteristic Functions 237 4.6.5 Differential Entropy of a Gaussian Vector 240 4.7 Conditional Expectation 241 4.8 Jointly Gaussian Vectors 245 4.9 Expectation as Estimation 248 4.10 Implications for Linear Estimation 256 4.11 Correlation and Linear Estimation 258 4.12 Correlation and Covariance Functions 267 4.13 The Central Limit Theorem 270 4.14 Sample Averages 274 4.15 Convergence of Random Variables 276 4.16 Weak Law of Large Numbers 284 4.17 Strong Law of Large Numbers 287 4.18 Stationarity 292 4.19 Asymptotically Uncorrelated Processes 298 4.20 Problems 302 5 Second-Order Theory 322 5.1 Linear Filtering of Random Processes 324 5.2 Linear Systems I/O Relations 326 5.3 Power Spectral Densities 333 5.4 Linearly Filtered Uncorrelated Processes 335 5.5 Linear Modulation 343 5.6 White Noise 346 5.6.1 Low Pass and Narrow Band Noise 351 5.7 Time-Averages 351 5.8 Mean Square Calculus 355 5.8.1 Mean Square Convergence Revisited 356 5.8.2 Integrating Random Processes 363 5.8.3 Linear Filtering 367 5.8.4 Differentiating Random Processes 368 5.8.5 Fourier Series 373 5.8.6 Sampling 377 x Contents 5.8.7 Karhunen-Loueve Expansion 382 5.9 Linear Estimation and Filtering 387 5.9.1 Discrete Time 388 5.9.2 Continuous Time 403 5.10 Problems 407 6 A Menagerie of Processes 424 6.1 Discrete Time Linear Models 425 6.2 Sums of IID Random Variables 430 6.3 Independent Stationary Increment Processes 432 6.4 Second-Order Moments of ISI Processes 436 6.5 Specification of Continuous Time ISI Processes 438 6.6 Moving-Average and Autoregressive Processes 441 6.7 The Discrete Time Gauss-Markov Process 443 6.8 Gaussian Random Processes 444 6.9 The Poisson Counting Process 445 6.10 Compound Processes 449 6.11 Composite Random Processes 451 6.12 Exponential Modulation 452 6.13 Thermal Noise 458 6.14 Ergodicity 461 6.15 Random Fields 466 6.16 Problems 467 A Preliminaries 479 A.1 Set Theory 479 A.2 Examples of Proofs 489 A.3 Mappings and Functions 492 A.4 Linear Algebra 493 A.5 Linear System Fundamentals 497 A.6 Problems 503 B Sums and Integrals 508 B.1 Summation 508 B.2 Double Sums 511 B.3 Integration 513 B.4 The Lebesgue Integral 515 C Common Univariate Distributions 519 D Supplementary Reading 522 [...]... many years at Stanford University and at the University of Maryland using the book: An Introduction to Statistical Signal Processing Much of the basic content of this course and of the fundamentals of random processes can be viewed as the analysis of statistical signal processing systems: typically one is given a probabilistic description for one random object, which can be considered as an input signal. .. freely available to any who wish to use it provided only that the contents of the entire text remain intact and together Comments, corrections, and suggestions should be sent to rmgray@stanford.edu Every effort will be made to fix typos and take suggestions into an account on at least an annual basis Acknowledgements We repeat our acknowledgements of the original book: to Stanford University and the University... Another goal is to enable the student who might not continue to more advanced courses to be able to read and generally follow the modern literature on applications of random processes to information and communication theory, estimation and detection, control, signal processing, and stochastic systems theory Preface xv Revisions Through the years the original book has continually expanded to roughly double... of the literature in statistical signal processing, communications, control, image and video processing, speech and audio processing, medical signal processing, geophysical signal processing, and classical statistical areas of time series analysis, classification and regression, and pattern recognition show a wide variety of probabilistic models for input processes and for operations on those processes,... between an average engineer and an outstanding engineer is the ability to derive effective models providing a good balance between complexity and accuracy Random processes usually occur in applications in the context of environments or systems which change the processes to produce other processes The intentional operation on a signal produced by one process, an “input signal, ” to produce a new signal, an. .. suddenly changes for the worse Signal processing systems can look for these changes and warn medical personnel when suspicious behavior occurs r Images produced by laser cameras inside elderly North Atlantic pipelines Introduction 3 can be automatically analyzed to locate possible anomolies indicating corrosion by looking for locally distinct random behavior How are these signals characterized? If the signals... original size to include more topics, examples, and problems The material has been significantly reorganized in its grouping and presentation Prerequisites and preliminaries have been moved to the appendices Major additional material has been added on jointly Gaussian vectors, minimum mean squared error estimation, detection and classification, filtering, and, most recently, mean square calculus and its applications... “output signal, ” is generally referred to as signal processing, a topic easily illustrated by examples r A time varying voltage waveform is produced by a human speaking into a microphone or telephone This signal can be modeled by a random process This signal might be modulated for transmission, then it might be digitized and coded for transmission on a digital link Noise in the digital link can cause... through the ever changing versions and provided a stream of comments and corrections Thanks are also due to Richard Blahut and anonymous referees for their careful reading and commenting on the original book Thanks are due xvi Preface to the many readers ho have provided corrections and helpful suggestions through the Internet since the revisions began being posted Particular thanks are due to Yariv Ephraim... dealing with random processes, including the IEEE Transactions on Signal Processing, IEEE Transactions on Image Processing, IEEE Transactions on Speech and Audio Processing, IEEE Transactions on Communications, IEEE Transactions on Control, and IEEE Transactions on Information Theory It also should be mentioned that the authors are electrical engineers and, as such, have written this text with an electrical . An Introduction to Statistical Signal Processing Pr(f ∈ F ) = P ({ω : ω ∈ F }) = P (f −1 (F )) f −1 (F ) f F ✲ November 22, 2003 i ii An Introduction to Statistical Signal Processing Robert. the literature in statistical signal processing, communications, control, image and video processing, speech and audio processing, medical signal processing, geophysical signal processing, and classical. taught for many years at Stanford University and at the University of Maryland using the book: An Introduction to Statistical Signal Processing. Much of the basic content of this course and of the