EDITOR-IN-CHIEF Peter W Hawkes CEMES-CNRS Toulouse, France VOLUME ONE HUNDRED AND EIGHTY SEVEN ADVANCES IN IMAGING AND ELECTRON PHYSICS Edited by PETER W HAWKES CEMES-CNRS, Toulouse, France AMSTERDAM • BOSTON • HEIDELBERG • LONDON NEW YORK • OXFORD • PARIS • SAN DIEGO SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO Academic Press is an imprint of Elsevier Cover photo credit: Ahmed Elgammal, Homeomorphic Manifold Analysis (HMA): Untangling Complex Manifolds Advances in Imaging and Electron Physics (2015) 187, pp 1-82 Academic Press is an imprint of Elsevier 125, London Wall, EC2Y 5AS 525 B Street, Suite 1800, San Diego, CA 92101-4495, USA 225 Wyman Street, Waltham, MA 02451, USA The Boulevard, Langford Lane, Kidlington, Oxford OX5 1GB, UK First edition 2015 Copyright Ó 2015 Elsevier Inc All rights reserved No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher Details on how to seek permission, further information about the Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein) Notices Knowledge and best practice in this field are constantly changing As new research and experience broaden our understanding, changes in research methods, professional practices, or medical treatment may become necessary Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information, methods, compounds, or experiments described herein In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Library of Congress Cataloging-in-Publication Data A catalog record for this book is available from the Library of Congress ISBN: 978-0-12-802255-9 ISSN: 1076-5670 For information on all Academic Press publications visit our website at http://store.elsevier.com/ PREFACE The first of the two chapters that make up this volume deals with spinpolarized scanning electron microscopy, a technique that is not new but is today of the highest interest Teruo Kohashi has been using this approach for more than 20 years and his chapter is therefore an authoritative account of the subject He first explains the principle behind spin-polarization detection for the study of magnetic domains He then describes at length the instrumental aspects The chapter concludes with a wide range of applications This lucid and knowledgeable text will surely be much appreciated In the second chapter, Ahmed Elgammal explores a very different topic: many problems in computer vision, and almost all tasks in human vision, involve analysis of image data in high-dimensional spaces The human case is very striking, for we are often able to recognize objects whatever the viewpoint, scale, lighting, and orientation The process that originally generated the image with which the computer or human is confronted is, however, frequently governed by a relatively small number of variables and the data are often assumed to lie on a low-dimensional manifold In this account of the subject, A Elgammal first surveys the problems arising from these vision tasks and then presents homeomorphic manifold analysis in detail This long chapter forms a monograph on the subject and will, I am sure, be of great value to readers in this active area of research As always, I thank the authors for the trouble they have taken to make their subjects understandable by readers from other subject areas Peter Hawkes vii j FUTURE CONTRIBUTIONS H.-W Ackermann Electron micrograph quality S Ando Gradient operators and edge and corner detection J Angulo Mathematical morphology for complex and quaternion-valued images D Batchelor Soft x-ray microscopy E Bayro Corrochano Quaternion wavelet transforms C Beeli Structure and microscopy of quasicrystals M Berz, P.M Duxbury, K Makino and C.-Y Ruan (Vol 190) Femtosecond electron imaging and spectroscopy C Bobisch and R M€ oller Ballistic electron microscopy F Bociort Saddle-point methods in lens design K Bredies Diffusion tensor imaging A Broers A retrospective R.E Burge (Vol 190) A scientific autobiography A Carroll (Vol 189) Reflective electron beam lithography N Chandra and R Ghosh Quantum entanglement in electron optics A Cornejo Rodriguez and F Granados Agustin Ronchigram quantification N de Jonge, Ed (Vol 189) CISCEM 2014 L.D Duffy and A Dragt Eigen-emittance J Elorza Fuzzy operators ix j x Future Contributions A.R Faruqi, G McMullan and R Henderson (Vol 190) Direct detectors M Ferroni Transmission microscopy in the scanning electron microscope R.G Forbes Liquid metal ion sources P Gai and E.D Boyes Aberration-corrected environmental electron microscopy J Grotemeyer and T Muskat (Vol 189) Time-of-flight mass spectrometry V.S Gurov, A.O Saulebekov and A.A Trubitsyn Analytical, approximate analytical and numerical methods for the design of energy analyzers M Haschke Micro-XRF excitation in the scanning electron microscope R Herring and B McMorran Electron vortex beams M.S Isaacson Early STEM development K Ishizuka Contrast transfer and crystal images K Jensen, D Shiffler and J Luginsland Physics of field emission cold cathodes M Jourlin Logarithmic image processing, the LIP model Theory and applications U Kaiser The sub-Ångstr€ om low-voltage electron microcope project (SALVE) C.T Koch In-line electron holography O.L Krivanek Aberration-corrected STEM M Kroupa The Timepix detector and its applications B Lencova Modern developments in electron optical calculations H Lichte New developments in electron holography M Matsuya Calculation of aberration coefficients using Lie algebra Future Contributions J.A Monsoriu Fractal zone plates L Muray Miniature electron optics and applications M.A O’Keefe Electron image simulation V Ortalan Ultrafast electron microscopy D Paganin, T Gureyev and K Pavlov Intensity-linear methods in inverse imaging M Pap (Vol 189) A special voice transform, analytic wavelets and Zernike functions N Papamarkos and A Kesidis The inverse Hough transform Q Ramasse and R Brydson The SuperSTEM laboratory B Rieger and A.J Koster Image formation in cryo-electron microscopy P Rocca and M Donelli Imaging of dielectric objects J Rodenburg Lensless imaging J Rouse, H.-n Liu and E Munro The role of differential algebra in electron optics J Sanchez Fisher vector encoding for the classification of natural images P Santi Light sheet fluorescence microscopy C.J.R Sheppard, S.S Kou and J Lin (Vol 189) The Hankel transform in n-dimensions, and its applications in optical propagation and imaging R Shimizu, T Ikuta and Y Takai Defocus image modulation processing in real time T Soma Focus-deflection systems and their applications I.F Spivak-Lavrov Analytical methods of calculation and simulation of new schemes of static and time-of-flight mass spectrometers xi xii P Sussner and M.E Valle Fuzzy morphological associative memories J Valdés Recent developments concerning the Systeme International (SI) G Wielgoszewski (Vol 190) Scanning thermal microscopy and related techniques Future Contributions CONTRIBUTORS Ahmed Elgammal Department of Computer Science, Rutgers University, 110 Frelinghuysen Rd., Piscataway, NJ 08854 Teruo Kohashi Central Research Laboratory, Hitachi, Ltd., Hatoyama, Saitama, Japan xiii j CHAPTER ONE Homeomorphic Manifold Analysis (HMA): Untangling Complex Manifolds Ahmed Elgammal* Department of Computer Science, Rutgers University, 110 Frelinghuysen Rd., Piscataway, NJ 08854 E-mail: elgammal@cs.rutgers.edu Contents Introduction Motivating Scenarios 2.1 Case Example I: Modeling the View-Object Manifold 2.2 Case Example II: Modeling the Visual Manifold of Biological Motion 2.3 Biological Motivation Framework Overview Manifold Factorization 4.1 Style Setting 4.2 Manifold Parameterization 4.3 Style Factorization 6 11 13 16 16 17 18 4.3.1 One-Style-Factor Model 4.3.2 Multifactor Model 18 19 4.4 Content Manifold Embedding 21 4.4.1 Nonlinear Dimensionality Reduction from Visual Data 4.4.2 Topological Conceptual Manifold Embedding 22 24 Inference 5.1 Solving for One Style Factor 25 26 5.1.1 Iterative Solution 5.1.2 Sampling-based Solution 26 28 5.2 Solving for Multiple Style Factors Given a Whole Sequence 5.3 Solving for Body Configuration and Style Factors from a Single Image Applications of Homomorphism on 1-D Manifolds 6.1 A Single-Style-Factor Model for Gait 6.1.1 Style-Dependent Shape Interpolation 6.1.2 Style-Preserving Posture-Preserving Reconstruction 6.1.3 Shape and Gait Synthesis 28 29 30 31 32 33 34 6.2 A Multifactor Model for Gait 6.3 A Multifactor Model for Facial Expression Analysis 37 41 * This work was funded by NSF award IIS-0328991 and NSF CAREER award IIS-0546372 Advances in Imaging and Electron Physics, Volume 187 ISSN 1076-5670 http://dx.doi.org/10.1016/bs.aiep.2014.12.002 © 2015 Elsevier Inc All rights reserved j PLATE Linear interpolation in the shape space between two subjects and the resulting shapes at eight different points of the gait cycle Moving along the curve generates different postures of the walking cycle, while linear interpolation of the style vectors on the top generates intermediate shape styles (Figure 13 on page 37 of this Volume) (a) (b) PLATE 10 Example of a multifactor model: Multiple views and multiple people generative model for gait (a) Examples of training data from different views, (b) examples of training data for multiple people from a side view (Figure 14 on page 38 of this Volume) (a) (c) (b) (d) PLATE 11 Multifactor model: (a) Style subspace: each person’s cycles have the same label (b) Unit circle embedding for three cycles (c) Mean style vectors for each person’s cluster in the style space (d) Viewpoint vectors (Lee & Elgammal, 2005b, Ó Springer) (Figure 15 on page 39 of this Volume) PLATE 12 Example posture recovery From top to bottom: Input shapes, implicit function, recovered 3-D pose (Lee & Elgammal, 2005b, Ó Springer) (Figure 16 on page 40 of this Volume) Style Weight (a) Style weights 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 10 15 20 25 30 35 40 35 40 Frame Number View Weight (b) View weights 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 view view view view 10 15 20 25 30 Frame Number (c) Style and viewpoint estimation PLATE 13 Estimated weights during a cycle (a) Style weights, (b) view weights, (c) iterative style and view estimations for each frame Below left: Error; center: style weights; right: view weights (Lee & Elgammal, 2005b, Ó Springer) (Figure 17 on page 41 of this Volume) PLATE 14 Examples of pose recovery and view classification for three people (Lee & Elgammal, 2005b, Ó Springer) (Figure 18 on page 42 of this Volume) (b) expression vectors (a) style vectors 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 −1 (c) Style plotting in 3-D 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 −1 (d) Expression plotting in 3-D 0.8 0.6 0.4 0.5 0.2 0 −0.5 −0.2 −1 −0.48 −0.4 −0.46 −0.6 −0.8 0.5 −0.44 −0.42 −0.4 −0.5 −1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55 −0.38 −0.4 −0.2 0.2 0.4 0.6 0.8 PLATE 15 Facial expression analysis for Cohn-Kanade data set for subjects with expressions, and their 3-D space plotting (a) the style vectors of the subjects (b) the expression vectors for the facial expressions (c) A visualization of the style vectors in a 3D Euclidean embedding space (d) A visualization of the expression vectors in a 3D Euclidean embedding space (Lee & Elgammal, 2005a, Ó Springer) (Figure 19 on page 43 of this Volume) (a) Joy (4, 8, 12, 16, 20th frames) (b) Surprise (2, 5, 8, 11, 14th frames) Surprise sequence 0.9 0.8 0.8 0.7 0.7 Weight values Weight values Smile sequence 0.9 0.6 0.5 0.4 0.3 0.5 0.4 0.3 0.2 0.2 0.1 0.6 0.1 10 15 Frame number 20 25 0 Frame number 10 12 14 PLATE 16 Estimated expression weights using frame-based estimation Top: sample frames from each sequence Bottom: estimated expression weights at each frame (Lee & Elgammal, 2005a, Ó Springer) (Figure 20 on page 45 of this Volume) (a) (b) (c) (d) (e) PLATE 17 Data-driven view and body configuration manifolds: a) Examples of sample data with view and configuration variations Rows: Body pose at 0; T; T; T; T, where T is a walking cycle period Cols.: View 0, 30, 60, /, 330 (b) Intrinsic configuration manifold when view 5 5 angle is 0, 60, 120, 180, and 240 (c) View manifold for five different fixed body poses (d) (e) Combined view and body configuration manifold by LLE and Isomap (Elgammal & Lee, 2009, Ó IEEE) (Figure 23 on page 48 of this Volume) PLATE 18 Torus representation for continuous view-variant dynamic human motion: (a) Three different trajectories on the torus manifold according to the view and configuration changes shown in (b), (c), and (d) (b) Syntheses of body posture variations with a fixed view (m ¼ 0:25; n : 0/1) (c) Syntheses of view variations with a fixed body configuration m : 0/1; n ¼ 0:3 (d) Syntheses of both view and body configuration variation: m : 0/1; n : 0:25/0:75 (Elgammal & Lee, 2009, Ó IEEE) (Figure 25 on page 53 of this Volume) (a) (b) (c) (d) (e) Manual Estimated 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 (f) 100 200 300 1.2 400 500 Manual Estimated 0.8 0.6 0.4 0.2 −0.2 100 200 300 400 500 PLATE 19 Reconstruction of 3-D body posture: (a) Input silhouettes (b) Input as implicit functions used in the estimation (c) Reconstructed silhouettes from the maximum a posterior (MAP) estimate on the torus (d) Reconstructed 3-D posture shown from a fixed viewpoint (e) Estimated values for the view parameter (m) (f) Estimated values for the body configuration parameter (n) (Elgammal & Lee, 2009, Ó IEEE) (Figure 26 on page 57 of this Volume) PLATE 20 Sample results for posture recovery (Figure 27 on page 58 of this Volume) PLATE 21 The trajectory of the estimated configuration and view parameters on the torus from the particle filter: MAP estimation (green), expected values (blue), and mode values (red) (Elgammal & Lee, 2009, Ó IEEE) (Figure 28 on page 58 of this Volume) PLATE 22 Evaluation of view-variant gait tracking from real data: (a) Sample input frames (b) Input silhouettes (c) The estimated body configuration parameter values (d) The estimated view parameter values (e) The distributions of the particles on the torus (f) The recovered shape from the estimated configuration and view (Elgammal & Lee, 2009, Ó IEEE) (Figure 29 on page 59 of this Volume) PLATE 23 Edge-based tracking: (a,b) A gait sequence tracking: (a) Estimated shape contours; (b) view and configuration particle distributions on the torus (c,d) Golf swing tracking: (c) Estimated shape contours; (d) view and configuration particle distributions on the torus (Elgammal & Lee, 2009, Ó IEEE) (Figure 30 on page 60 of this Volume) PLATE 24 Sample factorized mode for human motion with three latent spaces: viewinvariant posture representation, posture-invariant view representation (from a viewing circle), and person shape style representation (not shown) (Figure 31 on page 60 of this Volume) (a) (c) −1 (b) −2 −3 −2 −2 PLATE 25 Example of a complex motion from different views (a) Example postures from a ballet motion The 8th, 16th,/, 360th frames are shown from a sequence (b) Sampled shapes from different views and postures Rows: different views (30o, 90o, /, 330o) Columns: body postures at the 25th, 50th, /, 375th frames (c) Visual manifold embedding using LLE, combining the view and body configuration variations (Lee & Elgammal, 2007, Ó IEEE) (Figure 32 on page 61 of this Volume) (a) (b) 280 80 270 240 340 100 60 20 70 130 30 140 210 120 110 250 170 50 150 200 10 220 180 330 310 230 160 40 90 350 380 370 260 300 320 290 360 190 (c) (d) 30 0.9 0.8 210 0.2 360 240 180 0.7 60 0.6 0.5 −0.2 270 0.4 90 150 330 0.3 −0.4 120 0.8 0.6 0.4 0.2 −0.2 −0.4 0.2 0.6 300 0.4 0.1 0.2 (e) 0.2 0.4 0.6 0.8 (f) 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2 0.4 0.6 0.8 PLATE 26 Configuration and view manifolds for a ballet motion: (a)(b) Embedded kinematic manifold in 2-D (c) 1-D configuration-invariant view manifold embedding (the first three dimensions are shown here) (d)(e) Velocity field and its interpolation on the configuration manifold (f) Prior probabilistic distribution of body configuration on the kinematic embedding (Lee & Elgammal, 2007, Ó IEEE) (Figure 33 on page 63 of this Volume) (a) (b) (c) (d) (e) 0.35 1.5 0.3 0.5 0.25 330 300 0.2 0.15 −0.5 270 360 90 0.1 −0.4 −1 −2 0.2 −1.5 −1 −0.5 0.5 1.5 0.4 0.3 0.4 0.3 0.25 t θ −0.3 −0.4 −0.1 −0.2 0.35 0.2 0.15 −1 0.1 −2 0.05 0.1 0.2 (g) Estimated view configuration (f) 210 180 240 120 30 −0.2 −1.5 150 60 −3 10 15 20 25 30 35 Frame number 40 45 50 −4 −4 −3 −2 −1 PLATE 27 Catch/throw motion (evaluation): (a) Rendered image sequence (frames 3, 25, 47, 69, /, 333) (b) A test sequence with a moving camera (c) Estimated shapes after view and configuration estimation (d) 2-D configuration manifold embedding and selected basis points (e) Configuration-invariant view manifold in a 3-D space (f) Estimated view (g) Motion flow field on the embedding space (Lee & Elgammal, 2007, Ó IEEE) (Figure 34 on page 67 of this Volume) (a) (b) (c) (d) (e) (f) 35 30 25 20 15 10 (g) 10 20 30 40 Frame number 50 70 60 Estimated view θt 350 300 θt 250 200 150 100 50 10 20 (h) 30 Frame number 40 50 60 40 50 60 Error in view estimation 180 160 140 120 100 80 60 40 20 10 20 30 Frame number PLATE 28 A ballet motion: (a) A test input sequence (rendered) (b) A test image sequence (silhouette) (c) Estimated silhouette (generated from MAP estimation) (d) Ground truth 3-D body posture (in body-centered coordinates) (e) Estimated 3-D body posture (generated from the estimated body configuration) (f) Average error in the joint location estimation for each frame (g) Ground truth body rotation (from rotation of the root in the motion-captured data), estimated view coordinates (with body rotation measured by view rotation in the opposite direction), and absolute error between the true and estimated rotation (Lee & Elgammal, 2010a, Ó IEEE) (Figure 35 on page 68 of this Volume) (a) (b) (c) (d) 30 150 120 180 0.5 0.6 210 90 60 240 −0.5 360 330 300 0.4 270 0.2 −1 0.6 0.4 0.2 −0.2 −0.4 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2 0.4 0.6 0.8 (e) (f) (g) θ t (h) (i) 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 Estimated view θt True Estimated 50 100 150 200 250 Frame number 35 30 25 20 15 10 0 50 100 150 200 250 300 PLATE 29 Dancing sequence evaluation with a fixed-view camera (a) Input frames (rendered) (b) Manifold embedding of a dancing sequence (c) View manifold representation (d) Dynamic model (e) Input silhouettes for testing from a fixed view (f) Ground truth 3-D body posture (g) Reconstructed silhouettes (h) Estimated view parameters (i) Average location error for all joints (Lee & Elgammal, 2007, Ó IEEE) (Figure 36 on page 70 of this Volume) (a) (b) θt (c) Estimated view θt 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 True Estimated 50 100 150 200 250 200 250 Frame number (d) 30 25 20 15 10 0 50 100 150 300 PLATE 30 Dancing sequence evaluation with a camera rotation (a) Silhouettes for a rotating view (b) Reconstructed silhouettes (c) Estimated view parameters (d) Average location error for all joints (Lee & Elgammal, 2007, Ó IEEE) (Figure 37 on page 71 of this Volume) ... experience and knowledge in evaluating and using any information, methods, compounds, or experiments described herein In using such information or methods they should be mindful of their own safety and. .. S.S Kou and J Lin (Vol 189) The Hankel transform in n-dimensions, and its applications in optical propagation and imaging R Shimizu, T Ikuta and Y Takai Defocus image modulation processing in real... award IIS-0328991 and NSF CAREER award IIS-0546372 Advances in Imaging and Electron Physics, Volume 187 ISSN 1076-5670 http://dx.doi.org/10.1016/bs.aiep.2014.12.002 © 2015 Elsevier Inc All rights