SPATIO-TEMPORAL APPROACHES TO DENOISING AND FEATURE EXTRACTION IN RAPID IMAGE TRIAGE YU KE (B.Eng., Zhejiang University) A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY DEPARTMENT OF BIOENGINEERING NATIONAL UNIVERSITY OF SINGAPORE 2012 i Acknowledgments I would like to express my sincere gratefulness to my supervisor Professor Li Xiaoping, Director of Neuroengineering Laboratories, for his generous sharing, encouraging attitude, insightful vision and enlightening guidance. It is my pleasure to enjoy a wonderful 4-year study with so many amazing lab mates, Dr. Shen Kaiquan, Dr. Ng Wu Chun, Dr. Fan Jie, Dr. Ning Ning, Dr. Shao Shiyun, Dr. Wu Xiang, Mr. Khoa Wei Long Geoffrey, Mr. Wu Ji, Mr. Rohit Tyagi, Mr. Bui Ha Duc, Miss Wang Yue, Miss Ye Yan and Mr. Wu Tiecheng. I benefit a lot from their selfless support and valuable suggestions, and would like to take this opportunity to show my deep thankfulness. Special acknowledgments are given to my parents. Their love accompanies me whenever and wherever I am. Last but not the least, I am very grateful to the National University of Singapore for granting me the financial support, with which I can endeavor myself to this doctoral research. NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE ii Table of Contents Acknowledgments i Summary vii List of Tables xi List of Figures xvii List of Symbols xviii Acronyms xxi Introduction 1.1 A Snapshot of Image Screening Strategies . . . . . . . . . . . . 1.1.1 Artificial intelligence based . . . . . . . . . . . . . . . 1.1.2 Human intelligence oriented . . . . . . . . . . . . . . . 1.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE TABLE OF CONTENTS Literature Review 2.1 2.2 2.3 2.4 2.5 iii 10 EEG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.1.1 Physiological background . . . . . . . . . . . . . . . . 11 2.1.2 Technical background . . . . . . . . . . . . . . . . . . 12 2.1.3 Event-related potentials . . . . . . . . . . . . . . . . . 17 Brain Computer Interface . . . . . . . . . . . . . . . . . . . . . 23 2.2.1 Invasive BCI . . . . . . . . . . . . . . . . . . . . . . . 23 2.2.2 Noninvasive BCI . . . . . . . . . . . . . . . . . . . . . 25 EEG Signal Processing Methods . . . . . . . . . . . . . . . . . 29 2.3.1 Signal modeling . . . . . . . . . . . . . . . . . . . . . 30 2.3.2 Denoising . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.3.3 Feature extraction . . . . . . . . . . . . . . . . . . . . . 36 2.3.4 Classification . . . . . . . . . . . . . . . . . . . . . . . 39 Rapid Image Triage . . . . . . . . . . . . . . . . . . . . . . . . 41 2.4.1 Rationale of RIT . . . . . . . . . . . . . . . . . . . . . 41 2.4.2 Past work on RIT . . . . . . . . . . . . . . . . . . . . . 43 Mathematical Supplement . . . . . . . . . . . . . . . . . . . . 50 2.5.1 Common spatial pattern . . . . . . . . . . . . . . . . . 50 2.5.2 Weighted support vector machine . . . . . . . . . . . . 52 Rapid Image Triage System NATIONAL UNIVERSITY OF SINGAPORE 59 SINGAPORE TABLE OF CONTENTS 3.1 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . 60 3.2 RIT for Online Broad-area Imagery Screening . . . . . . . . . . 61 3.3 iv 3.2.1 Image preparation . . . . . . . . . . . . . . . . . . . . 61 3.2.2 Experimental procedure . . . . . . . . . . . . . . . . . 64 3.2.3 Data processing . . . . . . . . . . . . . . . . . . . . . . 67 3.2.4 Broad-area imagery screening . . . . . . . . . . . . . . 67 Real-life experiments . . . . . . . . . . . . . . . . . . . . . . . 68 3.3.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . 69 3.3.2 Experimental paradigm . . . . . . . . . . . . . . . . . . 69 3.3.3 Data acquisition . . . . . . . . . . . . . . . . . . . . . 70 A Spatio-Temporal Filtering Approach to Denoising of Single-Trial ERP in Rapid Image Triage 72 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 4.2 Proposed Spatio-Temporal Filtering Approach . . . . . . . . . . 76 4.3 4.2.1 Background . . . . . . . . . . . . . . . . . . . . . . . . 76 4.2.2 Spatial filtering . . . . . . . . . . . . . . . . . . . . . . 77 4.2.3 Spatio-temporal filtering . . . . . . . . . . . . . . . . . 78 4.2.4 Estimating the ERP latency difference between channels 79 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 4.3.1 Simulation tests . . . . . . . . . . . . . . . . . . . . . . 81 NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE TABLE OF CONTENTS 4.3.2 4.4 4.5 v Real-life RIT experiments . . . . . . . . . . . . . . . . 82 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . 84 4.4.1 Simulation test I . . . . . . . . . . . . . . . . . . . . . 84 4.4.2 Simulation test II . . . . . . . . . . . . . . . . . . . . . 85 4.4.3 Simulation test III . . . . . . . . . . . . . . . . . . . . 88 4.4.4 Real RIT experiments . . . . . . . . . . . . . . . . . . 90 4.4.5 Future work . . . . . . . . . . . . . . . . . . . . . . . . 95 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . 96 Common Spatio-Temporal Pattern for Single-Trial Detection of ERP in Rapid Image Triage 97 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 5.2 Single-Trial ERP Detection . . . . . . . . . . . . . . . . . . . . 100 5.3 5.4 5.2.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . 100 5.2.2 Common spatio-temporal pattern method . . . . . . . . 101 5.2.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 105 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 5.3.1 Filters and patterns . . . . . . . . . . . . . . . . . . . . 106 5.3.2 Classification . . . . . . . . . . . . . . . . . . . . . . . 109 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 5.4.1 Scenario I . . . . . . . . . . . . . . . . . . . . . . . . . 111 NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE TABLE OF CONTENTS 5.4.2 5.5 Scenario II . . . . . . . . . . . . . . . . . . . . . . . . 115 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . 116 Bilinear Common Spatial Pattern for Single-Trial Detection of ERP in Rapid Image Triage vi 117 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 6.2 Proposed Method . . . . . . . . . . . . . . . . . . . . . . . . . 119 6.2.1 Common spatial pattern . . . . . . . . . . . . . . . . . 119 6.2.2 Bilinear common spatial pattern . . . . . . . . . . . . . 121 6.2.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 125 6.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 6.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 6.5 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . 132 Conclusions and Recommendations 134 7.0.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . 134 7.0.2 Recommendations . . . . . . . . . . . . . . . . . . . . 137 Author’s Publications 139 Bibliography 141 NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE vii Summary Nowadays, along with the advances in imaging and storage technology, there has been an accentuated contradiction between the fast increasing sets of largevolume imagery and limited number of skilled image analysts. Rapid image triage (RIT) which leverages split-second human perceptual judgement via the interpretation of electroencephalogram (EEG) signals, can effectively improve the efficiency of imagery screening. This thesis is mainly concerned with developing novel single-trial EEG signal processing methods which are the backbone of RIT system, to augment visual target object detection. These novel singletrial methods are characterized by explicitly exploiting the spatio-temporal propagation of event related potentials (ERP) across the scalp, which are particularly informative for ERP detection. Improvements regarding the RIT protocol are also taken into account. The measured scalp EEG signals are always contaminated by physiological artifacts and environmental artifacts. These artifacts are of much stronger amplitude than the EEG signals and thus significantly deteriorate the decoding of informative cerebral signals. In this work, a non-sophisticated and highly effective denoising approach is put forward to strengthen the signal-to-noise ratio (SNR). NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE SUMMARY viii The approach performs spatial smoothing in a temporally adjusted space, in which noises become less correlated and can be easily suppressed, while retaining inherent signals. The results from simulation experiments and real-life experiments indicate that the proposed approach is well suited for RIT. Single-trial feature extraction serves as an important mean of counteracting “curse of dimensionality” which refers to the situation that a large volume of data is required to achieve statistical significant result in a high-dimensional space. By solely preserving underlying meaningful features, the RIT system is less vulnerable to irrelevant and misleading information. Hence the optimization problem in RIT can be greatly simplified. This work implements two single-trial feature extraction methods, extending the common spatial pattern analysis (CSP) to accommodate additional temporal structures. The incorporation of discriminative temporal information has been proven to be meaningful and very effective as demonstrated in the comparison with competing methods on real-life experiments. The real-life experiments were conducted on the developed near real-time RIT system, which is primarily designed for online broad-area imagery screening. The RIT system integrates software platform with hardware devices. It streamlines the procedures and is characterized by an image analyst-centric protocol: 1) centering visual objects for convenient observation; 2) maintaining the spatial information flow of imagery so as to avoid eliciting interfering brain signals. The present work enriches conventional EEG signal processing toolbox with several novel single-trial spatio-temporal denoising and feature extraction approaches, which lend RIT system significant discriminating capability. Further NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 159 Magliero, A., Bashore, T. R., Coles, M. G., and Donchin, E. 1984. On the dependence of P300 latency on stimulus evaluation processes. Psychophysiology, 21(2):171–186. Makeig, S., Bell, A. J., Jung, T. P., and Sejnowski, T. J. 1996. Independent component analysis of electroencephalographic data. Advances in neural information processing systems, 8:145–151. Makeig, S., Delorme, A., Westerfield, M., Jung, T.-P., Townsend, J., Courchesne, E., and Sejnowski, T. J. 2004. Electroencephalographic brain dynamics following manually responded visual targets. PLoS Biology, 2(6):E176. Mason, J. A. and Herrmann, K. R. 1998. Universal infant hearing screening by automated auditory brainstem response measurement. Pediatrics, 101(2):221–228. Massimini, M., Huber, R., Ferrarelli, F., Hill, S., and Tononi, G. 2004. The sleep slow oscillation as a traveling wave. The Journal of Neuroscience, 24(31):6862–6870. Mathan, S., Erdogmus, D., Huang, Y., Pavel, M., Ververs, P., Carciofini, J., Dorneich, M., and Whitlow, S. 2008. Rapid image analysis using neural signals. In: CHI ’08 extended abstracts on human factors in computing systems, New York, NY, USA. ACM, pages 3309–3314. Mathan, S., Whitlow, S., Erdogmus, D., Pavel, M., Ververs, P., and Dorneich, M. 2006. Neurophysiologically driven image triage: a pilot study. In: CHI ’06 extended abstracts on human factors in computing systems, New York, NY, USA. ACM, pages 1085–1090. Mauricio, J. A. 1995. Exact maximum likelihood estimation of stationary vector ARMA models. Journal of the American Statistical Association, 90:282– 291. NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 160 Miller, K. J., Schalk, G., Fetz, E. E., den Nijs, M., Ojemann, J. G., and Rao, R. P. N. 2010. Cortical activity during motor execution, motor imagery, and imagery-based online feedback. Proceedings of the National Academy of Sciences, 107(9):4430–4435. Miwakeichi, F., Mart´ınez-Montes, E., Vald´es-Sosa, P. A., Nishiyama, N., Mizuhara, H., and Yamaguchi, Y. 2004. Decomposing EEG data into space-time-frequency components using parallel factor analysis. NeuroImage, 22(3):1035 – 1045. Model, D. and Zibulevsky, M. 2006. Learning subject-specific spatial and temporal filters for single-trial EEG classification. NeuroImage, 32(4):1631 – 1641. Molholm, S., Ritter, W., Javitt, D. C., and Foxe, J. J. 2004. Multisensory visualcauditory object recognition in humans: a high-density electrical mapping study. Cerebral Cortex, 14(4):452–465. Mørup, M., Hansen, L. K., Arnfred, S. M., Lim, L.-H., and Madsen, K. H. 2008. Shift-invariant multilinear decomposition of neuroimaging data. NeuroImage, 42(4):1439 – 1450. M¨uller-Gerking, J., Pfurtscheller, G., and Flyvbjerg, H. 1999. Designing optimal spatial filters for single-trial EEG classification in a movement task. Clinical Neurophysiology, 110(5):787 – 798. Nadenau, M. J., Winkler, S., Alleysson, D., and Kunt, M. 2000. Human vision models for perceptually optimized image processing – a review. In: Proceedings of the IEEE. Natarajan, P., Saleem, S., Prasad, R., MacRostie, E., and Subramanian, K. 2008. Multi-lingual offline handwriting recognition using hidden Markov models: A script-independent approach. In: Doermann, D. and Jaeger, S. (Editors), Arabic and Chinese Handwriting Recognition, volume 4768 of NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 161 Lecture Notes in Computer Science, pages 231–250. Springer Berlin / Heidelberg. Nelson, D. B. 1990. Stationarity and persistence in the GARCH(1,1) model. Econometric Theory, 6(03):318–334. Neumann, N., K¨ubler, A., Kaiser, J., Hinterberger, T., and Birbaumer, N. 2003. Conscious perception of brain states: mental strategies for brain computer communication. Neuropsychologia, 41(8):1028 – 1036. Neuper, C., M¨uller, G., K¨ubler, A., Birbaumer, N., and Pfurtscheller, G. 2003. Clinical application of an EEG-based brain-computer interface: A case study in a patient with severe motor impairment. Clinical Neurophysiology, 114(3):399 – 409. Ng, W., Seet, H., Lee, K., Ning, N., Tai, W., Sutedja, M., Fuh, J., and Li, X. 2009. Micro-spike EEG electrode and the vacuum-casting technology for mass production. Journal of Materials Processing Technology, 209(9):4434 – 4438. Niedermeyer, E. and Lopes, d. S. F. 2004. Electroencephalography: Basic Principles, Clinical Applications, and Related Fields. Lippincot Williams & Wilkins. Nieuwenhuis, S., Ridderinkhof, K. R., Blom, J., Band, G. P., and Kok, A. 2001. Error-related brain potentials are differentially related to awareness of response errors: Evidence from an antisaccade task. Psychophysiology, 38(5):752–760. Nøaøatøanen, R. and Picton, T. W. 1986. N2 and automatic versus controlled processes. Electroencephalography and Clinical Neurophysiology. Supplement, 38:169–186. NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 162 Novi, Q., Guan, C., Dat, T. H., and Xue, P. 2007. Sub-band common spatial pattern (SBCSP) for brain-computer interface. In: Neural Engineering, 2007 3rd International IEEE/EMBS Conference on. pages 204 –207. Nunez, P. L. and Pilgreen, K. L. 1991. The spline-laplacian in clinical neurophysiology: A method to improve EEG spatial resolution. Journal of Clinical Neurophysiology, 8(4):397–413. Nuwer, M. R., Comi, G., Emerson, R., Fuglsang-Frederiksen, A., Gu´erit, J. M., Hinrichs, H., Ikeda, A., Luccas, F. J., and Rappelsburger, P. 1998. IFCN standards for digital recording of clinical EEG. Electroencephalography and clinical Neurophysiology, 106(3):259–261. Olejniczak, P. 2006. Neurophysiologic basis of EEG. Journal of Clinical Neurophysiology, 23(3):186–9. Onitsuka, T., Ninomiya, H., Sato, E., Yamamoto, T., and Tashiro, N. 2000. The effect of interstimulus intervals and between-block rests on the auditory evoked potential and magnetic field: is the auditory P50 in humans an overlapping potential? Clinical Neurophysiology, 111(2):237 – 245. Oostenveld, R. and Praamstra, P. 2001. The five percent electrode system for high-resolution EEG and ERP measurements. Clinical Neurophysiology, 112(4):713 – 719. Osuna, E. E., Freund, R., and Girosi, F. 1997. Support vector machines: Training and applications. Technical report memo 1602, MIT. Pakes, A. and McGuire, P. 2001. Stochastic algorithms, symmetric Markov perfect equilibrium, and the ’curse’ of dimensionality. Econometrica, 69(5):1261–1281. Park, S.-Y. and Lee, J.-J. 2008. EEG feature extraction and classification using data dimension reduction. In: Industrial Informatics, 2008 6th IEEE International Conference on. pages 355 –358. NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 163 Parra, L., Christoforou, C., Gerson, A., Dyrholm, M., Luo, A., Wagner, M., Philiastides, M., and Sajda, P. 2008. Spatiotemporal linear decoding of brain state. Signal Processing Magazine, IEEE, 25(1):107 –115. Parra, L. C., Spence, C. D., Gerson, A. D., and Sajda, P. 2005. Recipes for the linear analysis of EEG. NeuroImage, 28(2):326 – 341. Patel, S. H. and Azzam, P. N. 2005. Characterization of N200 and P300: Selected studies of the event-related potential. International Journal of Medical Science, 2:147–154. Pelosi, L., Holly, M., Slade, T., Hayward, M., Barrett, G., and Blumhardt, L. D. 1992. Event-related potential (ERP) correlates of performance of intelligence tests. Electroencephalography and Clinical Neurophysiol- ogy/Evoked Potentials Section, 84(6):515 – 520. Pereira, F., Mitchell, T., and Botvinick, M. 2009. Machine learning classifiers and fMRI: A tutorial overview. NeuroImage, 45(1, Supplement 1):S199 – S209. Pfurtscheller, G. and da Silva, F. L. 1999. Event-related EEG/MEG synchronization and desynchronization: basic principles. Clinical Neurophysiology, 110(11):1842 – 1857. Piccione, F., Giorgi, F., Tonin, P., Priftis, K., Giove, S., Silvoni, S., Palmas, G., and Beverina, F. 2006. P300-based brain computer interface: reliability and performance in healthy and paralysed participants. Clinical Neurophysiology, 117(3):531–7. Picton, T., Alain, C., Otten, L., Ritter, W., and Achim, A. 2000a. Mismatch negativity: Different water in the same river. Audiology Neurotology, 5:111– 139. Picton, T., Bentin, S., Berg, P., Donchin, E., Hillyard, S., Johnson, R., Miller, G., Ritter, W., Ruchkin, D., Rugg, M., and Taylor, M. 2000b. Guidelines for NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 164 using human event-related potentials to study cognition: Recording standards and publication criteria. Psychophysiology, 37(2):127–152. Pinto, N., Cox, D. D., and DiCarlo, J. J. 2008. Why is real-world visual object recognition hard? PLoS Computional Biology, 4(1):E27. Piritchard, W. S., Shappell, S. A., and Brandt, M. E. 1991. Psychophysiology of N200/N400: A review and classification scheme. Advances in Psychophysiology, 4:43–106. Polich, J. 2007. Updating p300: An integrative theory of P3a and P3b. Clinical Neurophysiology, 118(10):2128 – 2148. Polich, J. and Bondurant, T. 1997. P300 sequence effects, probability, and interstimulus interval. Physiology & Behaviour, 61(6):843 – 849. Polich, J. and Criado, J. R. 2006. Neuropsychology and neuropharmacology of P3a and P3b. International Journal of Psychophysiology, 60(2):172 – 185. Poolman, P., Frank, R. M., Luu, P., Pederson, S. M., and Tucker, D. M. 2008. A single-trial analytic framework for EEG analysis and its application to target detection and classification. NeuroImage, 42(2):787 – 798. Rakshit, S. and Basu, S. 2010. Recognition of handwritten roman script using tesseract open source OCR engine. In: Proceedings of National Conference on NAQC-2008. pages 141–145. Ramoser, H., M¨uller-Gerking, J., and Pfurtscheller, G. 2000. Optimal spatial filtering of single trial EEG during imagined hand movement. Rehabilitation Engineering, IEEE Transactions on, 8(4):441 –446. Regan, D. 1989. Human Brain Electrophysiology: Evoked Potentials and Evoked Magnetic Fields in Science and Medicine. Elsevier. NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 165 Ritter, W., Simson, R., and Vaughan, H. G. 1983. Event-related potential correlates of two stages of information processing in physical and semantic discrimination tasks. Psychophysiology, 20(2):168–179. Rossion, B. and Jacques, C. 2008. Does physical interstimulus variance account for early electrophysiological face sensitive responses in the human brain? ten lessons on the N170. NeuroImage, 39(4):1959 – 1979. Rugg, M., Milner, A., Lines, C., and Phalp, R. 1987. Modulation of visual event-related potentials by spatial and non-spatial visual selective attention. Neuropsychologia, 25(1, Part 1):85 – 96. Saerens, M., Fouss, F., Yen, L., and Dupont, P. 2004. The principal components analysis of a graph, and its relationships to spectral clustering. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., and Pedreschi, D. (Editors), Machine Learning: ECML 2004, volume 3201 of Lecture Notes in Computer Science, pages 371–383. Springer Berlin / Heidelberg. Sajda, P., Pohlmeyer, E., Wang, J., Parra, L., Christoforou, C., Dmochowski, J., Hanna, B., Bahlmann, C., Singh, M., and Chang, S.-F. 2010. In a blink of an eye and a switch of a transistor: Cortically coupled computer vision. Proceedings of the IEEE, 98(3):462 –478. Salman, A., Turovets, S., Malony, A., Eriksen, J., and Tucker, D. 2005. Computational modeling of human head conductivity. In: Sunderam, V., van Albada, G., Sloot, P., and Dongarra, J. (Editors), Computational Science - ICCS 2005, volume 3514 of Lecture Notes in Computer Science, pages 3–92. Springer Berlin / Heidelberg. Sams, M., Alho, K., and Ntnen, R. 1983. Sequential effects on the ERP in discriminating two stimuli. Biological Psychology, 17(1):41 – 58. Sanei, S. and Chambers, J. 2007. EEG signal processing. John Wiley & Sons, Chichester ; Hoboken, NJ. NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 166 Santucci, D. M., Kralik, J. D., Lebedev, M. A., and Nicolelis, M. A. L. 2005. Frontal and parietal cortical ensembles predict single-trial muscle activity during reaching movements in primates. European Journal of Neuroscience, 22(6):1529–1540. S¨arel¨a, J. and Valpola, H. 2005. Denoising source separation. Journal of Machine Learning Research, 6:233–272. Schendan, H. E. and Kutas, M. 2007. Neurophysiological evidence for the time course of activation of global shape, part, and local contour representations during visual object categorization and memory. Journal of cognitive neuroscience, 19:734–749. Schmidt, E., McIntosh, J., Durelli, L., and Bak, M. 1978. Fine control of operantly conditioned firing patterns of cortical neurons. Experimental Neurology, 61(2):349 – 369. Schmitt, U., Louis, A., Darvas, F., Buchner, H., and Fuchs, M. 2001. Numerical aspects of spatio-temporal current density reconstruction from EEG/MEG-data. Medical Imaging, IEEE Transactions on, 20(4):314 –324. Serby, H., Yom-Tov, E., and Inbar, G. 2005. An improved P300-based braincomputer interface. Neural Systems and Rehabilitation Engineering, IEEE Transactions on, 13(1):89 –98. Serruya, M. D., Hatsopoulos, N. G., Paninski, L., Fellows, M. R., and Donoghue, J. P. 2002. Brain-machine interface: Instant neural control of a movement signal. Nature, 416:141–142. Shao, S.-Y., Shen, K.-Q., Ong, C. J., Wilder-Smith, E., and Li, X.-P. 2009. Automatic EEG artifact removal: A weighted support vector machine approach with error correction. Biomedical Engineering, IEEE Transactions on, 56(2):336 –344. NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 167 Shen, K. Q., Shao, S., Yu, K., Kwok, K., Boopathi, V. G., and Li, X. 2009. Single-Trial ERP Detection for Rapid Image Triage, pages 59–62. Kyoto University Press. Shenoy, P., Miller, K., Ojemann, J., and Rao, R. 2008. Generalized features for electrocorticographic BCIs. Biomedical Engineering, IEEE Transactions on, 55(1):273 –280. Sitaram, R., Weiskopf, N., Caria, A., Veit, R., Erb, M., and Birbaumer, N. 2008. fMRI brain-computer interfaces. Signal Processing Magazine, IEEE, 25(1):95 –106. Smeulders, A., Worring, M., Santini, S., Gupta, A., and Jain, R. 2000. Contentbased image retrieval at the end of the early years. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 22(12):1349 –1380. Smulders, F. T. Y., Kenemans, J. L., and Kok, A. 1996. Effects of task variables on measures of the mean onset latency of LRP depend on the scoring method. Psychophysiology, 33(2):194–205. Squires, N. K., Squires, K. C., and Hillyard, S. A. 1975. Two varieties of longlatency positive waves evoked by unpredictable auditory stimuli in man. Electroencephalography and Clinical Neurophysiology, 38(4):387 – 401. Stolarova, M., Keil, A., and Moratti, S. 2006. Modulation of the C1 visual eventrelated component by conditioned stimuli: Evidence for sensory plasticity in early affective perception. Cerebral Cortex, 16(6):876–887. Strik, W. K., Dierks, T., B¨oning, J., Osterheider, M., Caspari, A., and K¨orber, J. 1992. Disorders of smooth pursuit eye movement and auditory N100 in schizophrenic patients. Psychiatry Research, 41(3):227 – 235. Su, T.-H., Zhang, T.-W., Guan, D.-J., and Huang, H.-J. 2009. Off-line recognition of realistic chinese handwriting using segmentation-free strategy. Pattern Recognition, 42(1):167 – 182. NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 168 Subasi, A. and Gursoy, M. I. 2010. EEG signal classification using PCA, ICA, LDA and support vector machines. Expert Systems with Applications, 37(12):8659 – 8666. Tallgren, P., Vanhatalo, S., Kaila, K., and Voipio, J. 2005. Evaluation of commercially available electrodes and gels for recording of slow EEG potentials. Clinical Neurophysiology, 116(4):799 – 806. Tao, Q., Wu, G.-W., Wang, F.-Y., and Wang, J. 2005. Posterior probability support vector machines for unbalanced data. Neural Networks, IEEE Transactions on, 16(6):1561 –1573. Taylor, D. M., Tillery, S. I. H., and Schwartz, A. B. 2002. Direct cortical control of 3D neuroprosthetic devices. Science, 296(5574):1829–1832. Tejada, J. and Martinez-Echevarria, J. 2007. Support vector machines. In: Andina, D. and Pham, D. T. (Editors), Computational Intelligence, pages 147– 191. Springer US. Teli, M. and Anderson, C. 2009. Nonlinear dimensionality reduction of electroencephalogram (EEG) for brain computer interfaces. In: Engineering in Medicine and Biology Society, 2009 Annual International Conference of the IEEE. pages 2486 –2489. Teplan, M. 2002. Fundamentals of EEG measurement. Measurement Science Review, 2:1–11. Thakor, N. and Tong, S. 2004. Advances in quantitative electroencephalogram analysis methods. Annual Review of Biomedical Engineering, 6(9):1–43. Thomas, K., Guan, C., Lau, C. T., Vinod, A., and Ang, K. K. 2009. A new discriminative common spatial pattern method for motor imagery brain - computer interfaces. Biomedical Engineering, IEEE Transactions on, 56(11):2730 –2733. NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 169 Thorpe, S., Fize, D., and Marlot, C. 1996. Speed of processing in the human visual system. Nature, 381(6582):520–2. Tome, A., Teixeira, A., Lang, E., and da Silva, A. 2007. Greedy kernel PCA applied to single-channel EEG recordings. In: Engineering in Medicine and Biology Society, 2007 29th Annual International Conference of the IEEE. pages 5441 –5444. Tomioka, R., Dornhege, G., Nolte, G., Blankertz, B., Aihara, K., and M¨uller, K.R. 2006. Spectrally weighted common spatial pattern algorithm for single trial EEG classification. Technical report, Univ. Tokyo. Tomioka, R. and M¨uller, K.-R. 2010. A regularized discriminative framework for EEG analysis with application to brain-computer interface. NeuroImage, 49(1):415 – 432. Towle, V. L., Bola˜nos, J., Suarez, D., Tan, K., Grzeszczuk, R., Levin, D. N., Cakmur, R., Frank, S. A., and Spire, J.-P. 1993. The spatial location of EEG electrodes: Locating the best-fitting sphere relative to cortical anatomy. Electroencephalography and Clinical Neurophysiology, 86(1):1 – 6. Townsend, G., Graimann, B., and Pfurtscheller, G. 2004. Continuous EEG classification during motor imagery-simulation of an asynchronous BCI. Neural Systems and Rehabilitation Engineering, IEEE Transactions on, 12(2):258 –265. Truccolo, W., Knuth, K. H., Shah, A., Bressler, S. L., Schroeder, C. E., and Ding, M. 2003. Estimation of single-trial multicomponent ERPs: Differentially variable component analysis (dVCA). Biological Cybernetics, 89:426–438. Tyraskis, P. and Jensen, O. 1985. Multichannel linear prediction and maximumentropy spectral analysis using least squares modeling. Geoscience and Remote Sensing, IEEE Transactions on, GE-23(2):101 –109. NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 170 Urrestarazu, E., Iriarte, J., Alegre, M., Valencia, M., Viteri, C., and Artieda, J. 2004. Independent component analysis removing artifacts in ictal recordings. Epilepsia, 45:1071–1078. Van de Borne, P., Nguyen, H., Biston, P., Linkowski, P., and Degaute, J. P. 1994. Effects of wake and sleep stages on the 24-h autonomic control of blood pressure and heart rate in recumbent men. American Journal of Physiology - Heart and Circulatory Physiology, 266(2):H548–H554. van Schie, H. T., Mars, R. B., Coles, M. G., and Bekkering, H. 2004. Modulation of activity in medial frontal and motor cortices during error observation. Nature Neuroscience, 7:549 – 554. van Straaten, H. L. M., Groote, M. E., and Oudesluys-Murphy, A. M. 1996. Evaluation of an automated auditory brainstem response infant hearing screening method in at risk neonates. European Journal of Pediatrics, 155:702–705. Vaughan, H. G., Costa, L. D., and Ritter, W. 1968. Topography of the human motor potential. Electroencephalography and Clinical Neurophysiology, 25(1):1 – 10. Verleger, R. 1988. Event-related potentials and cognition: A critique of the context updating hypothesis and an alternative interpretation of P3. Behavioral and Brain Sciences, 11(03):343–356. Verleysen, M. and Francois, D. 2005. The curse of dimensionality in data mining and time series prediction. In: Cabestany, J., Prieto, A., and Sandoval, F. (Editors), Computational Intelligence and Bioinspired Systems, volume 3512 of Lecture Notes in Computer Science, pages 85–125. Springer Berlin / Heidelberg. Vig´ario, R. N. 1997. Extraction of ocular artefacts from EEG using independent NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 171 component analysis. Electroencephalography and Clinical Neurophysiology, 103(3):395 – 404. Visani, M., Garcia, C., and Jolion, J.-M. 2005. Normalized radial basis function networks and bilinear discriminant analysis for face recognition. In: Advanced Video and Signal Based Surveillance, 2005 IEEE Conference on, Los Alamitos, CA, USA. IEEE Computer Society, pages 342–347. Vogel, E. K. and Luck, S. J. 2000. The visual N1 component as an index of a discrimination process. Psychophysiology, 37(02):190–203. Vroomen, J. and Stekelenburg, J. J. 2010. Visual anticipatory information modulates multisensory interactions of artificial audiovisual stimuli. Journal of Cognitive Neuroscience, 22:7:1583–1596. Vuckovic, A., Radivojevic, V., Chen, A. C., and Popovic, D. 2002. Automatic recognition of alertness and drowsiness from EEG by an artificial neural network. Medical Engineering & Physics, 24(5):349 – 360. Vˇera and K˘rkov´a 1992. Kolmogorov’s theorem and multilayer neural networks. Neural Networks, 5(3):501 – 506. Wallstrom, G. L., Kass, R. E., Miller, A., Cohn, J. F., and Fox, N. A. 2004. Automatic correction of ocular artifacts in the EEG: a comparison of regression-based and component-based methods. International Journal of Psychophysiology, 53(2):105–19. Walter, W., Cooper, R., Aldridge, V., McCallum, W., and Winter, A. 1964. Contingent negative variation: An electric sign of sensorimotor association and expectancy. Nature, 203:380–384. Wang, C., Xia, B., Li, J., Yang, W., Xiao, D., Velez, A., and Yang, H. 2011. Motor imagery BCI-based robot arm system. In: Natural Computation (ICNC), 2011 7th International Conference on. pages 181 –184. NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 172 Wang, J., Pohlmeyer, E., Hanna, B., Jiang, Y.-G., Sajda, P., and Chang, S.-F. 2009. Brain state decoding for rapid image retrieval. In: Proceedings of the 17th ACM international conference on Multimedia, New York, NY, USA. ACM, pages 945–954. Wang, Y., Gao, S., and Gao, X. 2005. Common spatial pattern method for channel selelction in motor imagery based brain-computer interface. In: Engineering in Medicine and Biology Society, 2005 27th Annual International Conference of. pages 5392 –5395. Wascher, E., Hoffmann, S., Snger, J., and Grosjean, M. 2009. Visuo-spatial processing and the N1 component of the ERP. Psychophysiology, 46(6):1270– 1277. Watt, R. and Morgan, M. 1985. A theory of the primitive spatial code in human vision. Vision Research, 25(11):1661 – 1674. Webster, J. G. (Editor) 1997. Medical Instrumentation: Application and Design. Wiley, 3rd edition. Wessberg, J., Stambaugh, C. R., Kralik, J. D., Beck, P. D., Laubach, M., Chapin, J. K., Kim, J., Biggs, S. J., Srinivasan, M. A., and Nicolelis, M. A. L. 2000. Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature, 408:361–365. Widrow, B., Glover, J.R., J., McCool, J., Kaunitz, J., Williams, C., Hearn, R., Zeidler, J., Eugene Dong, J., and Goodlin, R. 1975. Adaptive noise cancelling: Principles and applications. Proceedings of the IEEE, 63(12):1692 – 1716. Wolfgang and Klimesch 1999. EEG alpha and theta oscillations reflect cognitive and memory performance: A review and analysis. Brain Research Reviews, 29(2-3):169 – 195. NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 173 Woody, C. 1967. Characterization of an adaptive filter for the analysis of variable latency neuroelectric signals. Medical and Biological Engineering and Computing, 5:539–554. Wu, W., Gao, X., Hong, B., and Gao, S. 2008. Classifying single-trial EEG during motor imagery by iterative spatio-spectral patterns learning (ISSPL). Biomedical Engineering, IEEE Transactions on, 55(6):1733 –1743. Yoo, S.-S., Fairneny, T., Chen, N.-K., Choo, S.-E., Panych, L. P., Park, H., Lee, S.-Y., and Jolesz, F. A. 2004. Brain-computer interface using fMRI: spatial navigation by thoughts. NeuroReport, 15(10):1591–1595. Yordanova, J., Kolev, V., and Polich, J. 2001. P300 and alpha event-related desynchronization (ERD). Psychophysiology, 38(1):143–152. Zeng, J. and Liu, Z.-Q. 2008. Type-2 fuzzy Markov random fields and their application to handwritten chinese character recognition. Fuzzy Systems, IEEE Transactions on, 16(3):747 –760. Zhang, D., Maye, A., Gao, X., Hong, B., Engel, A. K., and Gao, S. 2010. An independent brain-computer interface using covert non-spatial visual selective attention. Journal of Neural Engineering, 7(1):016010. Zhang, J., Sudre, G., Li, X., Wang, W., Weber, D., and Bagic, A. 2011. Clustering linear discriminant analysis for MEG-based brain computer interfaces. Neural Systems and Rehabilitation Engineering, IEEE Transactions on, 19(3):221 –231. Zhao, Q., Zhang, L., and Cichocki, A. 2009. Multilinear generalization of common spatial pattern. In: Proceedings of the 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, Washington, DC, USA. pages 525–528. Zietsch, B. P., Hansen, J. L., Hansell, N. K., Geffen, G. M., Martin, N. G., and Wright, M. J. 2007. Common and specific genetic influences on EEG NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE BIBLIOGRAPHY 174 power bands delta, theta, alpha, and beta. Biological Psychology, 75(2):154 – 164. NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE [...]... effective triage techniques for rapid high-volume imagery screening during recent years Triage is a preliminary process that identifies a subset of images containing mostly target objects and a few false positive images, which will be followed-up by further inspection Among those triage techniques, the brain NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE 1.1 A Snapshot of Image Screening Strategies 2 computer integrated... exploring and utilizing the discriminative temporal patterns of ERP for robust feature extraction; (3) managing the unbalanced classification problem • System-level (1) easing the difficulties of observing target objects during RIT experiments; (2) minimizing the interfering EEG responses elicited by distractors This doctoral research contributes to the development of single-trial EEG signal processing... automatically comparing his/her fingerprint with those in the database The screening method depends on extraction and comparison of individually unique biometric (Jain et al., 1999), such as the characteristics of ridges (Jain et al., 2007; Ji and Yi, 2008) and minutia points (Jea and Govindaraju, 2005; Jianjiang and Feng, 2008) Handwriting recognition is another well explored area, which makes sense in applications... evaluated and compared to some competing methods in simulated experiments and real-life RIT experiments Chapter 5 presents a new single-trial EEG feature extraction algorithm which acquires discriminative temporal information besides spatial information in reallife RIT experiments The relationship between the temporal information and spatial information is discussed in details NATIONAL UNIVERSITY OF SINGAPORE... intelligence in terms of serial processing speed, human intelligence is a marvel at other aspects, e.g learning and generalizing Human beings are born with the abilities of conscious thinking, associating and reasoning as well as sub-conscious feeling In addition, they are able to grasp and recognize the gist of an image just at a brief glance For these reasons, human vision system has always been a meaningful... concerned with developing novel single-trial EEG processing methods which are capable of granting the brain computer integrated rapid image triage system sound ERP detection performance The specific objectives could be grouped into two parts: NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE 1.2 Objectives 7 • Algorithm-level (1) designing a novel denoising approach for enhancing the SNR in the context of RIT... order of image preparation Target is the image containing point of interest (POI) 63 3.4 Ordinary image preparation The rectangular in color stands for the image boundary 63 NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE LIST OF FIGURES 3.5 xii Image preparation with overlapping Adjacent images share a portion of imagery 64 3.6 Checking impedance... multivariate autoregressive PARAFAC parallel factor analysis PCA principle component analysis POI point of interest RIT rapid image triage RSVP rapid serial visual presentation SCP shifted CANDECOMP/PARAFAC model SNR signal -to- noise ratio SVM support vector machine WSVM weighted support vector machine NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE ACRONYMS 2D-G xxiii 2-dimensional Gaussian smoothing NATIONAL... integrated triage system which leverages human vision is very promising and is being intensively investigated, due to its unmatchable generalizing capabilities Since the human vision guided triage system heavily relies on the decoding of brain signals, i.e electroencephalogram (EEG), the key issue is to develop novel signal processing methods tailored for triage task 1.1 A Snapshot of Image Screening Strategies... processing capability of human vision, the screening of images under great NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE 1.1 A Snapshot of Image Screening Strategies 5 variations such as in scale, lighting and pose (bottlenecks for computer vision), is narrowed down and simplified to a binary discrimination problem: only two types of distinctive brain signals are to be differentiated, i.e event-related potentials . SPATIO-TEMPORAL APPROACHES TO DENOISING AND FEATURE EXTRACTION IN RAPID IMAGE TRIAGE YU KE (B.Eng., Zhejiang University) A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY DEPARTMENT. 70 4 A Spatio-Temporal Filtering Approach to Denoising of Single-Trial ERP in Rapid Image Triage 72 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 4.2 Proposed Spatio-Temporal. analyst-centric protocol: 1) centering visual objects for convenient observation; 2) maintaining the spatial information flow of imagery so as to avoid eliciting interfering brain signals. The present work