PRINCIPAL COMPONENT ANALYSIS Edited by Parinya Sanguansat Principal Component Analysis Edited by Parinya Sanguansat Published by InTech Janeza Trdine 9, 51000 Rijeka, Croatia Copyright © 2012 InTech All chapters are Open Access distributed under the Creative Commons Attribution 3.0 license, which allows users to download, copy and build upon published articles even for commercial purposes, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications After this work has been published by InTech, authors have the right to republish it, in whole or part, in any publication of which they are the author, and to make other personal use of the work Any republication, referencing or personal use of the work must explicitly identify the original source As for readers, this license allows users to download, copy and build upon published chapters even for commercial purposes, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications Notice Statements and opinions expressed in the chapters are these of the individual contributors and not necessarily those of the editors or publisher No responsibility is accepted for the accuracy of information contained in the published chapters The publisher assumes no responsibility for any damage or injury to persons or property arising out of the use of any materials, instructions, methods or ideas contained in the book Publishing Process Manager Oliver Kurelic Technical Editor Teodora Smiljanic Cover Designer InTech Design Team First published March, 2012 Printed in Croatia A free online edition of this book is available at www.intechopen.com Additional hard copies can be obtained from orders@intechweb.org Principal Component Analysis, Edited by Parinya Sanguansat p cm ISBN 978-953-51-0195-6 Contents Preface IX Chapter Two-Dimensional Principal Component Analysis and Its Extensions Parinya Sanguansat Chapter Application of Principal Component Analysis to Elucidate Experimental and Theoretical Information 23 Cuauhtémoc Araujo-Andrade, Claudio Frausto-Reyes, Esteban Gerbino, Pablo Mobili, Elizabeth Tymczyszyn, Edgar L Esparza-Ibarra, Rumen Ivanov-Tsonchev and Andrea Gómez-Zavaglia Chapter Principal Component Analysis: A Powerful Interpretative Tool at the Service of Analytical Methodology 49 Maria Monfreda Chapter Subset Basis Approximation of Kernel Principal Component Analysis 67 Yoshikazu Washizawa Chapter Multilinear Supervised Neighborhood Preserving Embedding Analysis of Local Descriptor Tensor 91 Xian-Hua Han and Yen-Wei Chen Chapter Application of Linear and Nonlinear Dimensionality Reduction Methods 107 Ramana Vinjamuri, Wei Wang, Mingui Sun and Zhi-Hong Mao Chapter Acceleration of Convergence of the Alternating Least Squares Algorithm for Nonlinear Principal Components Analysis 129 Masahiro Kuroda, Yuichi Mori, Masaya Iizuka and Michio Sakakihara VI Contents Chapter The Maximum Non-Linear Feature Selection of Kernel Based on Object Appearance 145 Mauridhi Hery Purnomo, Diah P Wulandari, I Ketut Eddy Purnama and Arif Muntasa Chapter FPGA Implementation for GHA-Based Texture Classification 165 Shiow-Jyu Lin, Kun-Hung Lin and Wen-Jyi Hwang Chapter 10 The Basics of Linear Principal Components Analysis Yaya Keho 181 Chapter 11 Robust Density Comparison Using Eigenvalue Decomposition 207 Omar Arif and Patricio A Vela Chapter 12 Robust Principal Component Analysis for Background Subtraction: Systematic Evaluation and Comparative Analysis 223 Charles Guyon, Thierry Bouwmans and El-hadi Zahzah Chapter 13 On-Line Monitoring of Batch Process with Multiway PCA/ICA 239 Xiang Gao Chapter 14 Computing and Updating Principal Components of Discrete and Continuous Point Sets 263 Darko Dimitrov Preface It is more than a century since Karl Pearson invented the concept of Principal Component Analysis (PCA) Nowadays, it is a very useful tool in data analysis in many fields PCA is the technique of dimensionality reduction, which transforms data in the high-dimensional space to space of lower dimensions The advantages of this subspace are numerous First of all, the reduced dimension has the effect of retaining the most of the useful information while reducing noise and other undesirable artifacts Secondly, the time and memory that used in data processing are smaller Thirdly, it provides a way to understand and visualize the structure of complex data sets Furthermore, it helps us identify new meaningful underlying variables Indeed, PCA itself does not reduce the dimension of the data set It only rotates the axes of data space along lines of maximum variance The axis of the greatest variance is called the first principal component Another axis, which is orthogonal to the previous one and positioned to represent the next greatest variance, is called the second principal component, and so on The dimension reduction is done by using only the first few principal components as a basis set for the new space Therefore, this subspace tends to be small and may be dropped with minimal loss of information Originally, PCA is the orthogonal transformation which can deal with linear data However, the real-world data is usually nonlinear and some of it, especially multimedia data, is multilinear Recently, PCA is not limited to only linear transformation There are many extension methods to make possible nonlinear and multilinear transformations via manifolds based, kernel-based and tensor-based techniques This generalization makes PCA more useful for a wider range of applications In this book the reader will find the applications of PCA in many fields such as image processing, biometric, face recognition and speech processing It also includes the core concepts and the state-of-the-art methods in data analysis and feature extraction X Preface Finally, I would like to thank all recruited authors for their scholarly contributions and also to InTech staff for publishing this book, and especially to Mr.Oliver Kurelic, for his kind assistance throughout the editing process Without them this book could not be possible On behalf of all the authors, we hope that readers will benefit in many ways from reading this book Parinya Sanguansat Faculty of Engineering and Technology, Panyapiwat Institute of Management Thailand 274 12 Principal Component Analysis Will-be-set-by-IN-TECH Deleting points Let the new convex hull be obtained by deleting nd tetrahedra from and added n a tetrahedra to the old convex hull If the interior point o (needed for a tetrahedronization of a convex polytope), after several deletions, lies inside the new convex hull, then the same formulas and time complexity, as by adding points, follow If o lie outside the new convex hull, then, we need to choose a new interior point o , and recompute the new tetrahedra associated with it Thus, we need in total O(n) time to update the principal components Under certain assumptions, we can recompute the new principal components faster: • If we know that a certain point of the polyhedron will never be deleted, we can choose o to be that point In that case, we also have the same closed-formed solution as for adding a point • Let the facets of the convex polyhedron have similar (uniformly distributed) area We choose o to be the center of gravity of the polyhedron Then, we can expect that after deleting a point, o will remain in the new convex hull However, after several deletions, o could lie outside the convex hull, and then we need to recompute it and the tetrahedra associated with it Note that in the case when we consider boundary of a convex polyhedron (Subsection 6.1 and Subsection 6.3), we not need an interior point o and the same time complexity holds for both adding and deleting points Conclusion In this chapter, we have presented closed-form solutions for computing and updating the principal components of (dynamic) discrete and continuous point sets The new principal components can be computed in constant time, when a constant number of points are added or deleted from the point set This is a significant improvement of the commonly used approach, when the new principal components are computed from scratch, which takes linear time The advantages of some of the theoretical results were verified and presented in the context of computing dynamic PCA bounding boxes in Dimitrov, Holst, Knauer & Kriegel (2009); Dimitrov et al (2011) An interesting open problem is to find a closed-form solutions for dynamical point sets different from convex polyhedra, for example, implicit surfaces or B-splines An implementation of computing principal components in a dynamic, continuous setting could be a useful practical extension of the results presented here regarding continuous point sets Applications of the results presented here in other fields, like computer vision or visualization, are of high interest Appendix: Computing and updating the principal components efficiently continuous case 6.1 Continuous PCA over the boundary of a polyhedron in R3 Let X be a polyhedron in R3 We assume that the boundary of X is triangulated (if it is not, we can triangulate it in a preprocessing), containing n triangles The k-th triangle, with vertices x1,k , x2,k , x3,k , can be represented in a parametric form by Tk (s, t) = x1,k + s ( x2,k − x1,k ) + 275 13 Computing and Updating Principal Components of Discrete and Continuous Point Sets of Discrete and Continuous Point Sets Computing and Updating Principal Components t ( x3,k − x1,k ), for ≤ s, t ≤ 1, and s + t ≤ For ≤ i ≤ 3, we denote by xi,j,k the i-th coordinate of the vertex x j of the triangle Tk The center of gravity of the k-th triangle is 1− s Ti (s, t) dt ds 0 1− s dt ds 0 μk = = x1,k + x2,k + x3,k The contribution of each triangle to the center of gravity of the triangulated surface is proportional to its area The area of the k-th triangle is ak = area( Tk ) = |( x2,k − x1,k )| × |( x3,k − x1,k )| We introduce a weight to each triangle that is proportional to its area, define as ak a = k, n a ∑ i =1 a k wk = where a is the area of X Then, the center of gravity of the boundary of X is μ= n ∑ wk μk k =1 The covariance matrix of the k-th triangle is 1− s 0 Σk = = 12 ( Tk (s, t) − μ) ( Tk (s, t) − μ) T dt ds 1− s 0 dt ds 3 ∑ ∑ (x j,k − μ)(xh,k − μ)T + ∑ (x j,k − μ)(x j,k − μ)T j =1 h =1 j =1 The (i, j)-th element of Σk , i, j ∈ {1, 2, 3}, is σij,k = 12 3 ∑ ∑ (xi,l,k − μi )(x j,h,k − μ j ) + ∑ (xi,l,k − μi )(x j,l,k − μ j ) l =1 h =1 , l =1 with μ = (μ1 , μ2 , μ3 ) Finally, the covariance matrix of the boundary of X is Σ= n ∑ wk Σk k =1 Adding points We add points to X Let X be the new convex hull We assume that X is obtained from X by deleting nd , and adding n a tetrahedra Then the sum of the areas of all triangles is a = n n+na k =1 k = n +1 ∑ ak + ∑ ak − n+n a +nd ∑ k = n + n a +1 ak = a + n+na ∑ k = n +1 ak − n+n a +nd ∑ k = n + n a +1 ak 276 14 Principal Component Analysis Will-be-set-by-IN-TECH The center of gravity of X is μ = = n n+na k =1 k = n +1 ∑ wk μk + ∑ a aμ + n+na ∑ k = n +1 wk μk − ak μk − n+n a +nd ∑ k = n + n a +1 n+n a +nd ∑ k = n + n a +1 wk μk = ak μk n n+na k =1 a k = n +1 ∑ ak μk + ∑ ak μk − n+n a +nd ∑ k = n + n a +1 ak μk (29) Let n+na n+n a +nd ∑ ak μk , and μd = a ∑ ak μk a k = n +1 k = n + n a +1 Then, we can rewrite (29) as a μ = μ + μ a − μd (30) a The i-th component of μ a and μd , ≤ i ≤ 3, is denoted by μi,a and μi,d , respectively The (i, j)-th component, σij , ≤ i, j ≤ 3, of the covariance matrix Σ of X is μa = σij = 12 12 n 3 n ∑ ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ) + ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ) k =1 l =1 h =1 n+na 3 k =1 l =1 n+na + ∑ ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ) + ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ) − k = n +1 l =1 h =1 n+n a +nd ∑ k = n +1 l =1 3 ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ) − k = n + n a +1 l =1 h =1 n+n a +nd ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ) k = n + n a +1 l =1 Let σij = (σ + σij,12 + σij,21 + σij,22 − σij,31 − σij,32 ), 12 ij,11 where σij,11 = n ∑ ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ), (31) k =1 l =1 h =1 σij,12 = 1.5mmσij,21 = n ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ), (32) k =1 l =1 n+na 3 ∑ ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ), (33) k = n +1 l =1 h =1 1.5mmσij,22 = n+na ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ), k = n +1 l =1 (34) Computing and Updating Principal Components of Discrete and Continuous Point Sets of Discrete and Continuous Point Sets Computing and Updating Principal Components σij,31 = n+n a +nd ∑ 277 15 ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ), (35) k = n + n a +1 l =1 h =1 σij,32 = n+n a +nd ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ) (36) k = n + n a +1 l =1 Plugging-in the values of μi and μ j in (31), we obtain: σij,11 = = n 3 a ∑ ∑ ∑ wk (xi,l,k − a k =1 l =1 h =1 n μi − μi,a + μi,d )( x j,h,k − a ∑ ∑ ∑ wk (xi,l,k − μi + μi (1 − a k =1 l =1 h =1 ( x j,h,k − μ j + μ j (1 − = n a μ − μ j,a + μ j,d ) a j ) − μi,a + μi,d ) · a ) − μ j,a + μ j,d ) a ∑ ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ) + k =1 l =1 h =1 n 3 a ∑ ∑ ∑ wk (xi,l,k − μi )(μ j (1 − a k =1 l =1 h =1 n 3 a ) − μi,a + μi,d )( x j,h,k − μ j ) + a ) − μi,a + μi,d )(μ j (1 − ∑ ∑ ∑ w k ( μ i (1 − a k =1 l =1 h =1 n 3 ∑ ∑ ∑ w k ( μ i (1 − a k =1 l =1 h =1 ) − μ j,a + μ j,d ) + a ) − μ j,a + μ j,d ) a (37) Since ∑n=1 ∑3=1 wk ( xi,l,k − μi ) = 0, ≤ i ≤ 3, we have k l σij,11 = n 3 ∑ a (x − μi )(x j,h,k − μ j ) + a k∑ l∑ h=1 k i,l,k =1 =1 n 3 a a ∑ a (μ (1 − a ) − μi,a + μi,d )(μ j (1 − a ) − μ j,a + μ j,d ) a k∑ l∑ h=1 k i =1 =1 = n 3 ∑ a (x − μi )(x j,h,k − μ j ) + a k∑ l∑ h=1 k i,l,k =1 =1 a a a (μ (1 − ) − μi,a + μi,d )(μ j (1 − ) − μ j,a + μ j,d ) a i a a Plugging-in the values of μi and μ j in (32), we obtain: (38) 278 16 Principal Component Analysis Will-be-set-by-IN-TECH σij,12 = = n a ∑ ∑ wk (xi,l,k − a k =1 l =1 n μi − μi,a + μi,d )( x j,h,k − a ∑ ∑ wk (xi,l,k − μi + μi (1 − a k =1 l =1 ( x j,h,k − μ j + μ j (1 − = n a μ − μ j,a + μ j,d ) a j ) − μi,a + μi,d ) · a ) − μ j,a + μ j,d ) a ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ) + k =1 l =1 n a ∑ ∑ wk (xi,l,k − μi )(μ j (1 − a k =1 l =1 n a ) − μi,a + μi,d )( x j,h,k − μ j ) + a ) − μi,a + μi,d )(μ j (1 − ∑ ∑ w k ( μ i (1 − a k =1 l =1 n ∑ ∑ w k ( μ i (1 − a k =1 l =1 ) − μ j,a + μ j,d ) + a ) − μ j,a + μ j,d ) a (39) Since ∑n=1 ∑3=1 wk ( xi,l,k − μi ) = 0, ≤ i ≤ 3, we have k l σij,12 = n a (x − μi )( x j,h,k − μ j ) + a k∑ l∑ k i,l,k =1 =1 n a a a (μ (1 − ) − μi,a + μi,d )(μ j (1 − ) − μ j,a + μ j,d ) a k∑ l∑ k i a a =1 =1 = n a (x − μi )( x j,h,k − μ j ) + a k∑ l∑ k i,l,k =1 =1 a a a (μi (1 − ) − μi,a + μi,d )(μ j (1 − ) − μ j,a + μ j,d ) a a a (40) From (39) and (40), we obtain σij,1 = σij,11 + σij,12 a a a = σij + 12 (μi (1 − ) − μi,a + μi,d )(μ j (1 − ) − μ j,a + μ j,d ) a a a (41) Note that σij,1 can be computed in O(1) time The components σij,21 and σij,22 can be computed in O(n a ) time, while O(nd ) time is needed to compute σij,31 and σij,32 Thus, μ and σij = 1 (σ + σij,12 + σij,21 + σij,22 + σij,31 + σij,32 ) = (σ + σij,21 + σij,22 + σij,31 + σij,32 ) + 12 ij,11 12 ij a a a (μ (1 − ) − μi,a + μi,d )(μ j (1 − ) − μ j,a + μ j,d ) (42) a i a a can be computed in O(n a + nd ) time 279 17 Computing and Updating Principal Components of Discrete and Continuous Point Sets of Discrete and Continuous Point Sets Computing and Updating Principal Components Deleting points Let the new convex hull be obtained by deleting nd tetrahedra from and added n a tetrahedra to the old convex hull Consequently, the same formulas and time complexity, as by adding points, follow 6.2 Continuous PCA over a polygon in R2 We assume that the polygon X is triangulated (if it is not, we can triangulate it in a preprocessing), and the number of triangles is n The k-th triangle, with vertices x1,k , x2,k , x3,k = o, can be represented in a parametric form by Ti (s, t) = x3,k + s ( x1,k − x3,k ) + t ( x2,k − x3,k ), for ≤ s, t ≤ 1, and s + t ≤ The center of gravity of the k-th triangle is 1− s Ti (s, t) dt ds 0 1− s dt ds 0 μi = = x1,k + x2,k + x3,k The contribution of each triangle to the center of gravity of X is proportional to its area The area of the i-th triangle is |( x2,k − x1,k )| × |( x3,k − x1,k )| , ak = area( Tk ) = where × denotes the vector product proportional to its area, define as wk = We introduce a weight to each triangle that is ak a = k, a ∑n=1 ak k where a is the area of X.Then, the center of gravity of X is μ= n ∑ wk μk k =1 The covariance matrix of the k-th triangle is 1− s 0 Σk = = 12 ( Tk (s, t) − μ) ( Tk (s, t) − μ) T dt ds 1− s 0 dt ds 3 ∑ ∑ (x j,k − μ)(xh,k − μ)T + ∑ (x j,k − μ)(x j,k − μ)T j =1 h =1 j =1 The (i, j)-th element of Σk , i, j ∈ {1, 2}, is σij,k = 12 3 ∑ ∑ (xi,l,k − μi )(x j,h,k − μ j ) + ∑ (xi,l,k − μi )(x j,l,k − μ j ) l =1 h =1 l =1 with μ = (μ1 , μ2 ) The covariance matrix of X is Σ= n ∑ wk Σk k =1 , 280 18 Principal Component Analysis Will-be-set-by-IN-TECH Adding points We add points to X Let X be the new convex hull We assume that X is obtained from X by deleting nd , and adding n a triangles Then the sum of the areas of all triangles is a = n n+na k =1 k = n +1 ∑ ak + ∑ ak − n+n a +nd ∑ k = n + n a +1 ak = a + n+na ∑ k = n +1 ak − n+n a +nd ∑ k = n + n a +1 ak The center of gravity of X is μ = = n n+na k =1 k = n +1 ∑ wk μk + ∑ a aμ + n+na ∑ k = n +1 wk μk − ak μk − n+n a +nd ∑ k = n + n a +1 n+n a +nd ∑ k = n + n a +1 wk μk = ak μk n n+na k =1 a k = n +1 ∑ ak μk + ∑ ak μk − n+n a +nd ∑ k = n + n a +1 ak μk (43) Let n+na n+n a +nd ∑ ak μk , and μd = a ∑ ak μk a k = n +1 k = n + n a +1 Then, we can rewrite (43) as a μ = μ + μ a − μd (44) a The i-th component of μ a and μd , ≤ i ≤ 2, is denoted by μi,a and μi,d , respectively The (i, j)-th component, σij , ≤ i, j ≤ 2, of the covariance matrix Σ of X is μa = σij = 12 12 n 3 n ∑ ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ) + ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ) k =1 l =1 h =1 n+na 3 k =1 l =1 n+na + ∑ ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ) + ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ) − k = n +1 l =1 h =1 n+n a +nd ∑ k = n +1 l =1 ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ) − k = n + n a +1 l =1 h =1 n+n a +nd ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ) k = n + n a +1 l =1 Let σij = (σ + σij,12 + σij,21 + σij,22 − σij,31 − σij,32 ), 12 ij,11 where σij,11 = (45) k =1 l =1 h =1 σij,12 = σij,21 = n ∑ ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ), n ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ), (46) k =1 l =1 n+na 3 ∑ ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ), k = n +1 l =1 h =1 (47) Computing and Updating Principal Components of Discrete and Continuous Point Sets of Discrete and Continuous Point Sets Computing and Updating Principal Components σij,22 = σij,31 = n+na 281 19 ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ), (48) k = n +1 l =1 n+n a +nd ∑ 3 ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ), (49) k = n + n a +1 l =1 h =1 σij,32 = n+n a +nd ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ) (50) k = n + n a +1 l =1 Plugging-in the values of μi and μ j in (45), we obtain: σij,11 = = n 3 a ∑ ∑ ∑ wk (xi,l,k − a k =1 l =1 h =1 n μi − μi,a + μi,d )( x j,h,k − a ∑ ∑ ∑ wk (xi,l,k − μi + μi (1 − a k =1 l =1 h =1 ( x j,h,k − μ j + μ j (1 − = n a μ − μ j,a + μ j,d ) a j ) − μi,a + μi,d ) · a ) − μ j,a + μ j,d ) a ∑ ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ) + k =1 l =1 h =1 n 3 a ∑ ∑ ∑ wk (xi,l,k − μi )(μ j (1 − a k =1 l =1 h =1 n 3 a ) − μi,a + μi,d )( x j,h,k − μ j ) + a ) − μi,a + μi,d )(μ j (1 − ∑ ∑ ∑ w k ( μ i (1 − a k =1 l =1 h =1 n 3 ∑ ∑ ∑ w k ( μ i (1 − a k =1 l =1 h =1 ) − μ j,a + μ j,d ) + a ) − μ j,a + μ j,d ) a (51) Since ∑n=1 ∑3=1 wk ( xi,l,k − μi ) = 0, ≤ i ≤ 2, we have k l σij,11 = n 3 ∑ a (x − μi )(x j,h,k − μ j ) + a k∑ l∑ h=1 k i,l,k =1 =1 n 3 a a ∑ a (μ (1 − a ) − μi,a + μi,d )(μ j (1 − a ) − μ j,a + μ j,d ) a k∑ l∑ h=1 k i =1 =1 = n 3 ∑ a (x − μi )(x j,h,k − μ j ) + a k∑ l∑ h=1 k i,l,k =1 =1 a a a (μi (1 − ) − μi,a + μi,d )(μ j (1 − ) − μ j,a + μ j,d ) a a a (52) 282 20 Principal Component Analysis Will-be-set-by-IN-TECH Plugging-in the values of μi and μ j in (46), we obtain: σij,12 = = n a ∑ ∑ wk (xi,l,k − a k =1 l =1 n μi − μi,a + μi,d )( x j,h,k − a ∑ ∑ wk (xi,l,k − μi + μi (1 − a k =1 l =1 ( x j,h,k − μ j + μ j (1 − = n a μ − μ j,a + μ j,d ) a j ) − μi,a + μi,d ) · a ) − μ j,a + μ j,d ) a ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ) + k =1 l =1 n a ∑ ∑ wk (xi,l,k − μi )(μ j (1 − a k =1 l =1 n a ) − μi,a + μi,d )( x j,h,k − μ j ) + a ) − μi,a + μi,d )(μ j (1 − ∑ ∑ w k ( μ i (1 − a k =1 l =1 n ) − μ j,a + μ j,d ) + ∑ ∑ w k ( μ i (1 − a k =1 l =1 a ) − μ j,a + μ j,d ) a (53) Since ∑n=1 ∑3=1 wk ( xi,l,k − μi ) = 0, ≤ i ≤ 2, we have k l σij,12 = n a (x − μi )( x j,h,k − μ j ) + a k∑ l∑ k i,l,k =1 =1 n a a a (μ (1 − ) − μi,a + μi,d )(μ j (1 − ) − μ j,a + μ j,d ) a k∑ l∑ k i a a =1 =1 = n a (x − μi )( x j,h,k − μ j ) + a k∑ l∑ k i,l,k =1 =1 a a a (μi (1 − ) − μi,a + μi,d )(μ j (1 − ) − μ j,a + μ j,d ) a a a (54) From (53) and (54), we obtain σij,1 = σij,11 + σij,12 = σij + 12 a a a (μ (1 − ) − μi,a + μi,d )(μ j (1 − ) − μ j,a + μ j,d ) a i a a (55) Note that σij,1 can be computed in O(1) time The components σij,21 and σij,22 can be computed in O(n a ) time, while O(nd ) time is needed to compute σij,31 and σij,32 Thus, μ and (σ + σij,12 + σij,21 + σij,22 + σij,31 + σij,32 ) 12 ij,11 = (σ + σij,21 + σij,22 + σij,31 + σij,32 ) + 12 ij a a a (μ (1 − ) − μi,a + μi,d )(μ j (1 − ) − μ j,a + μ j,d ) a i a a σij = can be computed in O(n a + nd ) time (56) 283 21 Computing and Updating Principal Components of Discrete and Continuous Point Sets of Discrete and Continuous Point Sets Computing and Updating Principal Components Deleting points Let the new convex hull be obtained by deleting nd tetrahedra from and added n a tetrahedra to the old convex hull If the interior point o, after deleting points, lies inside the new convex hull, then the same formulas and time complexity, as by adding points, follow However, o could lie outside the new convex hull Then, we need to choose a new interior point o , and recompute the new tetrahedra associated with it Thus, we need in total O(n) time to update the principal components 6.3 Continuous PCA over the boundary of a polygon R2 Let X be a polygon in R2 We assume that the boundary of X is comprised of n line segments The k-th line segment, with vertices x1,k , x2,k , can be represented in a parametric form by Sk (t) = x1,k + t ( x2,k − x1,k ) Since we assume that the mass density is constant, the center of gravity of the k-th line segment is Sk (t) dt x + x2,k μk = = 1,k dt The contribution of each line segment to the center of gravity of the boundary of a polygon is proportional to the length of the line segment The length of the k-th line segment is sk = length(Sk ) = || x2,k − x1,k || We introduce a weight to each line segment that is proportional to its length, define as wk = sk n ∑ k =1 s k = sk , s where s is the perimeter of X Then, the center of gravity of the boundary of X is μ= n ∑ wk μk k =1 The covariance matrix of the k-th line segment is Σk = = (Sk (t) − μ) (Sk (t) − μ) T dt 1 dt 2 ∑ ∑ (x j,k − μ)(xh,k − μ)T + ∑ (x j,k − μ)(x j,k − μ)T j =1 h =1 j =1 The (i, j)-th element of Σk , i, j ∈ {1, 2}, is σij,k = with μ = (μ1 , μ2 ) 2 ∑ ∑ (xi,l,k − μi )(x j,h,k − μ j ) + ∑ (xi,l,k − μi )(x j,l,k − μ j ) l =1 h =1 l =1 , 284 22 Principal Component Analysis Will-be-set-by-IN-TECH The covariance matrix of the boundary of X is Σ= n ∑ wk Σk k =1 Adding points We add points to X Let X be the new convex hull We assume that X is obtained from X by deleting nd , and adding n a line segments Then the sum of the lengths of all line segments is s = n n+na k =1 k = n +1 ∑ lk + ∑ sk − n+n a +nd ∑ k = n + n a +1 sk = s + n+na ∑ k = n +1 sk − n+n a +nd ∑ k = n + n a +1 sk The center of gravity of X is μ = n n+na k =1 k = n +1 ∑ wk μk + ∑ = s = s Let μa = wk μk − n n+na k =1 k = n +1 ∑ sk μk + ∑ sμ + n+na ∑ k = n +1 n+na ∑ sμ, s k = n +1 k k n+n a +nd ∑ k = n + n a +1 sk μk − sk μk − n+n a +nd ∑ k = n + n a +1 n+n a +nd ∑ k = n + n a +1 and μd = wk μk sk μk sk μk (57) n+n a +nd s μ s k=n∑ +1 k k +na Then, we can rewrite (57) as s μ + μ a − μd (58) s The i-th component of μ a and μd , ≤ i ≤ 2, is denoted by μi,a and μi,d , respectively The (i, j)-th component, σij , ≤ i, j ≤ 2, of the covariance matrix Σ of X is μ = σij = 6 n 2 n ∑ ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ) + ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ) k =1 l =1 h =1 n+na k =1 l =1 n+na 2 + ∑ ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ) + ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ) − k = n +1 l =1 h =1 n+n a +nd ∑ k = n +1 l =1 ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ) − k = n + n a +1 l =1 h =1 n+n a +nd ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ) k = n + n a +1 l =1 Computing and Updating Principal Components of Discrete and Continuous Point Sets of Discrete and Continuous Point Sets Computing and Updating Principal Components Let σij = 285 23 (σ + σij,12 + σij,21 + σij,22 − σij,31 − σij,32 ), ij,11 where σij,11 = n (59) ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ), (60) k =1 l =1 n+na 2 ∑ ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ), (61) k = n +1 l =1 h =1 σij,22 = σij,31 = k =1 l =1 h =1 σij,12 = σij,21 = n ∑ ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ), n+na ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ), (62) k = n +1 l =1 n+n a +nd ∑ 2 ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ), (63) k = n + n a +1 l =1 h =1 σij,32 = n+n a +nd ∑ ∑ wk (xi,l,k − μi )(x j,l,k − μ j ) (64) k = n + n a +1 l =1 Plugging-in the values of μi and μ j in (59), we obtain: σij,11 = = n 2 s ∑ ∑ ∑ wk (xi,l,k − s k =1 l =1 h =1 n μi − μi,a + μi,d )( x j,h,k − s ∑ ∑ ∑ wk (xi,l,k − μi + μi (1 − s k =1 l =1 h =1 ( x j,h,k − μ j + μ j (1 − = n s μ − μ j,a + μ j,d ) s j ) − μi,a + μi,d ) · s ) − μ j,a + μ j,d ) s ∑ ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ) + k =1 l =1 h =1 n 2 s ∑ ∑ ∑ wk (xi,l,k − μi )(μ j (1 − s k =1 l =1 h =1 n 2 s ) − μi,a + μi,d )( x j,h,k − μ j ) + s ) − μi,a + μi,d )(μ j (1 − ∑ ∑ ∑ w k ( μ i (1 − s k =1 l =1 h =1 n 2 ∑ ∑ ∑ w k ( μ i (1 − s k =1 l =1 h =1 ) − μ j,a + μ j,d ) + s ) − μ j,a + μ j,d ) s (65) 286 24 Principal Component Analysis Will-be-set-by-IN-TECH Since ∑n=1 ∑2=1 wk ( xi,l,k − μi ) = 0, ≤ i ≤ 2, we have l k σij,11 = n 2 ∑ s (x − μi )(x j,h,k − μ j ) + s k∑ l∑ h=1 k i,l,k =1 =1 n 2 s s ∑ s (μ (1 − s ) − μi,a + μi,d )(μ j (1 − s ) − μ j,a + μ j,d ) s k∑ l∑ h=1 k i =1 =1 = n 2 ∑ s (x − μi )(x j,h,k − μ j ) + s k∑ l∑ h=1 k i,l,k =1 =1 s s s (μi (1 − ) − μi,a + μi,d )(μ j (1 − ) − μ j,a + μ j,d ) s s s (66) Plugging-in the values of μi and μ j in (60), we obtain: σij,12 = = n s ∑ ∑ wk (xi,l,k − s k =1 l =1 n μi − μi,a + μi,d )( x j,h,k − s ∑ ∑ wk (xi,l,k − μi + μi (1 − s k =1 l =1 ( x j,h,k − μ j + μ j (1 − = n s μ − μ j,a + μ j,d ) s j ) − μi,a + μi,d ) · s ) − μ j,a + μ j,d ) s ∑ ∑ wk (xi,l,k − μi )(x j,h,k − μ j ) + k =1 l =1 n s ∑ ∑ wk (xi,l,k − μi )(μ j (1 − s k =1 l =1 n s ) − μi,a + μi,d )( x j,h,k − μ j ) + s ) − μi,a + μi,d )(μ j (1 − ∑ ∑ w k ( μ i (1 − s k =1 l =1 n ∑ ∑ w k ( μ i (1 − s k =1 l =1 ) − μ j,a + μ j,d ) + s ) − μ j,a + μ j,d ) s (67) Since ∑n=1 ∑2=1 wk ( xi,l,k − μi ) = 0, ≤ i ≤ 2, we have l k σij,12 = n s (x − μi )( x j,h,k − μ j ) + s k∑ l∑ k i,l,k =1 =1 n s s s (μ (1 − ) − μi,a + μi,d )(μ j (1 − ) − μ j,a + μ j,d ) s k∑ l∑ k i s s =1 =1 = n s (x − μi )( x j,h,k − μ j ) + s k∑ l∑ k i,l,k =1 =1 s s s (μi (1 − ) − μi,a + μi,d )(μ j (1 − ) − μ j,a + μ j,d ) s s s From (67) and (68), we obtain (68) Computing and Updating Principal Components of Discrete and Continuous Point Sets of Discrete and Continuous Point Sets Computing and Updating Principal Components σij,1 = σij,11 + σij,12 s s s = σij + (μi (1 − ) − μi,a + μi,d )(μ j (1 − ) − μ j,a + μ j,d ) s s s 287 25 (69) Note that σij,1 can be computed in O(1) time The components σij,21 and σij,22 can be computed in O(n a ) time, while O(nd ) time is needed to compute σij,31 and σij,32 Thus, μ and (σ + σij,12 + σij,21 + σij,22 + σij,31 + σij,32 ) ij,11 = (σij + σij,21 + σij,22 + σij,31 + σij,32 ) + s s s (μ (1 − ) − μi,a + μi,d )(μ j (1 − ) − μ j,a + μ j,d ) s i s s σij = (70) can be computed in O(n a + nd ) time Deleting points Let the new convex hull be obtained by deleting nd tetrahedra from and added n a tetrahedra to the old convex hull Consequently, the same formulas and time complexity, as by adding points, follow References Chan, T F., Golub, G H & LeVeque, R J (1979) Updating formulae and a pairwise algorithm for computing sample variances, Technical Report STAN-CS-79-773, Department of Computer Science, Stanford University Cheng, S.-W & Y Wang, Z W (2008) Provable dimension detection using principal component analysis, Int J Comput Geometry Appl 18: 415–440 Dimitrov, D., Holst, M., Knauer, C & Kriegel, K (2009) Closed-form solutions for continuous PCA and bounding box algorithms, A Ranchordas et al (Eds.): VISIGRAPP 2008, CCIS, Springer 24: 26–40 Dimitrov, D., Holst, M., Knauer, C & Kriegel, K (2011) Efficient dynamical computation of principal components, Proceedings of International Conference on Computer Graphics Theory and Applications - GRAPP, pp 85–93 Dimitrov, D., Knauer, C., Kriegel, K & Rote, G (2009) Bounds on the quality of the PCA bounding boxes, Computational Geometry 42: 772–789 Duda, R., Hart, P & Stork, D (2001) Pattern classification, John Wiley & Sons, Inc., 2nd ed Gottschalk, S., Lin, M C & Manocha, D (1996) OBBTree: A hierarchical structure for rapid interference detection, Computer Graphics 30: 171–180 Jolliffe, I (2002) Principal Component Analysis, Springer-Verlag, New York, 2nd ed Knuth, D E (1998) The art of computer programming, volume 2: seminumerical algorithms, Addison-Wesley, Boston, 3rd ed Parlett, B N (1998) The symmetric eigenvalue problem, Society of Industrial and Applied Mathematics (SIAM), Philadelphia, PA 288 26 Principal Component Analysis Will-be-set-by-IN-TECH Pébay, P P (2008) Formulas for robust, one-pass parallel computation of covariances and arbitrary-order statistical moments, Technical Report SAND2008-6212, Sandia National Laboratories Press, W H., Teukolsky, S A., Veterling, W T & Flannery, B P (1995) Numerical recipes in C: the art of scientific computing, Cambridge University Press, New York, USA, 2nd ed Vrani´ , D V., Saupe, D & Richter, J (2001) Tools for 3D-object retrieval: Karhunen-Loeve c transform and spherical harmonics, IEEE 2001 Workshop Multimedia Signal Processing, pp 293–298 Welford, B P (1962) Note on a method for calculating corrected sums of squares and products, Technometrics 4: 419–420 West, D H D (1979) Updating mean and variance estimates: an improved method, Communications of the ACM 22: 532–535 ... Two-Dimensional Principal Component Analysis and Its Extensions Two-Dimensional Principal Component Analysis and Its Extensions Fig The samples of shifted images on the ORL database 17 17 18 18 Principal Component. .. Chapter Principal Component Analysis: A Powerful Interpretative Tool at the Service of Analytical Methodology 49 Maria Monfreda Chapter Subset Basis Approximation of Kernel Principal Component Analysis. .. Linear Principal Components Analysis Yaya Keho 181 Chapter 11 Robust Density Comparison Using Eigenvalue Decomposition 207 Omar Arif and Patricio A Vela Chapter 12 Robust Principal Component Analysis