Neural Networks and Intellect: Using Model-Based Concepts Leonid I. Perlovsky OXFORD UNIVERSITY PRESS NEURAL NETWORKS AND INTELLECT This page intentionally left blank NEURAL NETWORKS AND INTELLECT Using Model-Based Concepts Leonid I. Perlovsky New York • Oxford OXFORD UNIVERSITY PRESS 2001 Oxford University Press Oxford New York Athens Auckland Bangkok Bogot ´ a Buenos Aires Calcutta Cape Town Chennai Dar es Salaam Delhi Florence Hong Kong Istanbul Karachi Kuala Lumpur Madrid Melbourne Mexico City Mumbai Nairobi Paris S ˜ ao Paulo Shanghai Singapore Taipei Tokyo Toronto Warsaw and associated companies in Berlin Ibadan Copyright © 2001 by Oxford University Press, Inc. Published by Oxford University Press, Inc., 198 Madison Avenue, New York, New York, 10016 http://www.oup-usa.org Oxford is a registered trademark of Oxford University Press All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior permission of Oxford University Press. Library of Congress Cataloging-in-Publication Data Perlovsky, Leonid I. Neural networks and intellect : using model-based concepts / Leonid I. Perlovsky. p. cm. Includes bibliographical references and index. ISBN 0-19-511162-1 1. Neural networks (computer science) 2. Mathematical models. I. Title. QA76.87.P435 2000 006.3'2—dc21 00-026297 Printing (last digit):987654321 Printed in the United States of America on acid-free paper CONDENSED TABLE OF CONTENTS PREFACE xix PART ONE: OVERVIEW: 2300 YEARS OF PHILOSOPHY, 100 YEARS OF MATHEMATICAL LOGIC, AND 50 YEARS OF COMPUTATIONAL INTELLIGENCE 1 Introduction: Concepts of Intelligence 3 2 Mathematical Concepts of Mind 51 3 Mathematical versus Metaphysical Concepts of Mind 125 PART TWO: MODELING FIELD THEORY: NEW MATHEMATICAL THEORY OF INTELLIGENCE WITH ENGINEERING APPLICATIONS 4 Modeling Field Theory 153 5 MLANS: Maximum Likelihood Adaptive Neural System for Grouping and Recognition 206 6 Einsteinian Neural Network 263 7 Prediction, Tracking, and Dynamic Models 289 8 Quantum Modeling Field Theory (QMFT) 321 9 Fundamental Limitations on Learning 329 10 Intelligent System Organization: MFT, Genetic Algorithms, and Kant 356 PART THREE: FUTURISTIC DIRECTIONS: FUN STUFF: MIND—PHYSICS + MATHEMATICS + CONJECTURES 11 Gödel Theorems, Mind, and Machine 383 12 Toward Physics of Consciousness 391 LIST OF SYMBOLS 425 DEFINITIONS 429 BIBLIOGRAPHY 447 INDEX 461 v This page intentionally left blank CONTENTS PREFACE xix PART ONE: OVERVIEW: 2300 YEARS OF PHILOSOPHY, 100 YEARS OF MATHEMATICAL LOGIC, AND 50 YEARS OF COMPUTATIONAL INTELLIGENCE 1 Introduction: Concepts of Intelligence 3 1.1 CONCEPTS OF INTELLIGENCE IN MATHEMATICS, PSYCHOLOGY, AND PHILOSOPHY 3 1.1.1 What Is Intelligence? 3 1.1.2 Plato, Occam, and Neural Networks 4 1.1.3 Rule-Based Artificial Intelligence, Complexity, and Aristotle 6 1.1.4 Philosophy vs. Architecture of Intelligent Tracker 8 1.1.5 Summary 12 1.2 PROBABILITY, HYPOTHESIS CHOICE, PATTERN RECOGNITION, AND COMPLEXITY 13 1.2.1 Prerequisite: Basic Notions of the Theory of Probability 13 1.2.2 Classical Hypotheses Choice Paradigms and Definitions 20 1.2.3 Pattern Recognition 22 1.2.4 A Priori Information and Adaptation 24 1.2.5 Mathematical Formulation of Model-Based Recognition 27 1.2.6 Conundrum of Combinatorial Complexity 29 1.3 PREDICTION, TRACKING, AND DYNAMIC MODELS 29 1.3.1 Linear Regression 30 1.3.2 Regression as an Expectation 32 1.3.3 Autoregression 33 1.3.4 Tracking 35 vii viii Contents 1.3.5 Association Problem 37 1.4 PREVIEW: INTELLIGENCE, INTERNAL MODEL, SYMBOL, EMOTIONS, AND CONSCIOUSNESS 42 Notes 45 Bibliographical Notes 46 Problems 47 2 Mathematical Concepts of Mind 51 2.1 COMPLEXITY, ARISTOTLE, AND FUZZY LOGIC 52 2.1.1 Conundrum of Combinatorial Complexity 52 2.1.2 Adaptivity, Apriority, and Complexity 53 2.1.3 Fuzzy Logic and Complexity 55 2.2 NEAREST NEIGHBORS AND DEGENERATE GEOMETRIES 58 2.2.1 The Nearest Neighbor Concept 58 2.2.2 Mathematical Formulation 59 2.2.3 What Constitutes Simple and Complex Classification Problems? 59 2.2.4 Degenerate Geometry of Classification Spaces 60 2.3 GRADIENT LEARNING, BACK PROPAGATION, AND FEEDFORWARD NEURAL NETWORKS 62 2.3.1 Concept of Discriminating Surfaces and Gradient Learning 62 2.3.2 Mathematical Formulation 64 2.3.3 Learning Disability 67 2.4 RULE-BASED ARTIFICIAL INTELLIGENCE 68 2.4.1 Minsky, Apriority, and Adaptivity 68 2.4.2 Soar Production System 70 2.5 CONCEPT OF INTERNAL MODEL 73 2.5.1 Prolegomena: Parametric vs. Nonparametric Estimation 73 2.5.2 Model-Based Vision (MBV) 74 2.5.3 Adaptivity and MBV 75 2.6 ABDUCTIVE REASONING 76 2.6.1 Deduction, Induction, and Abduction 76 2.6.2 Abductive Reasoning Trees and Bayesian Networks 77 2.7 STATISTICAL LEARNING THEORY AND SUPPORT VECTOR MACHINES 79 2.7.1 Model Complexity: Risk Minimization vs. PDF Estimation 79 Contents ix 2.7.2 Consistency of ERM and VC Dimension 81 2.7.3 Support Vector Machines (SVM) 82 2.8 AI DEBATES PAST AND FUTURE 85 2.8.1 Arguments and Disagreements: An Overview 85 2.8.2 Can a Machine Think? 87 2.8.3 Rule-Based AI vs. Connectivism 90 2.8.4 Emerging Debates 91 2.9 SOCIETY OF MIND 94 2.9.1 Society of Agents 94 2.9.2 Types of Agents 95 2.9.3 Frames and Unity of Apperception 96 2.9.4 Limitations and What Is Next 96 2.10 SENSOR FUSION AND JDL MODEL 97 2.10.1 Sensor Fusion and Origins of JDL Model 97 2.10.2 Definitions, Issues, and Types of Fusion Problems 98 2.10.3 Sensor Fusion Levels 99 2.10.4 Hierarchy of JDL Model Organization 100 2.11 HIERARCHICAL ORGANIZATION 100 2.12 SEMIOTICS 104 2.13 EVOLUTIONARY COMPUTATION, GENETIC ALGORITHMS, AND CAS 106 2.13.1 Complex Adaptive Systems (CAS) 107 2.13.2 CAS: Complexity vs. Fuzziness 109 2.14 NEURAL FIELD THEORIES 110 2.14.1 Grossberg’s Method: Physics of Mind 110 2.14.2 ART Neural Network 111 2.14.3 Illusions and A Priori Contents of Vision 114 2.14.4 Motor Coordination and Sensorimotor Control 115 2.14.5 Emotions and Learning 116 2.14.6 Quantum Neurodynamics 118 2.14.7 Modeling Field Theory 119 2.15 INTELLIGENCE, LEARNING, AND COMPUTABILITY 120 2.15.1 Computability: Turing vs. Physics 120 2.15.2 Computational Methods of Intelligence: Summary 121 Notes 121 Bibliographical Notes 122 Problems 124 [...]... undergraduate Chapter 5 4 Model-Based Neural Networks: Dynamic Models Internal models of the world are considered an essential part of intelligence in AI, cognitive sciences, and psychology The course describes how to design neural networks with internal models Model-based neural networks combine domain knowledge with learning and adaptivity of neural networks Prerequisites: probability and signal processing... 7 Or, use your favorite problems 3 Model-Based Neural Networks: Statistical Models Internal models of the world are considered an essential part of intelligence in AI, cognitive sciences, and psychology The course describes how to design neural networks with internal models Model-based neural networks combine domain knowledge with learning and adaptivity of neural networks Prerequisites: probability... concept called modeling field theory; demonstrates applications of neural networks based on this theory to a variety of problems; and analyzes relationships among mathematics, computational concepts in neural networks, and concepts of mind in psychology and philosophy Deep philosophical questions are discussed and related in detail to mathematics and the engineering of intelligence The book is directed toward... book selectively Concepts Chapters and Sections General Philosophical Concepts of Intellect and Their Relationships to Mathematical Concepts Ch 1, Sec 1.1 Ch 2, Sec 2.1, 2.8 Ch 3 Ch 9, Sec 9.5 Ch 10 through 12 Overview of Basic Mathematical Concepts of Modeling Intelligence and Their Relationships to Philosophical Concepts Ch 1, Sec 1.2, 1.3, 1.4 Ch 2 Ch 9 through 12 Mathematical Concepts and Techniques... graduate or advanced undergraduate Chapters 6 and 7 5 Relationships between Philosophical and Mathematical Concepts of Mind (for students with hard-science background) Relationships between contemporary mathematical concepts of intelligence and 2300-year-old philosophical concepts of mind are much closer than is generally recognized Specific mathematical concepts and debates are related to specific philosophical... debates of apriority and adaptivity of mind in mathematics and philosophy 1.1.2 Plato, Occam, and Neural Networks A contemporary direction in the theory of intellect is based on modeling neural structures of the brain It was founded by McCulloch and co-workers beginning in the early 1940s McCulloch intended to create a mathematical theory of intellect on the basis of complicated a priori neural structures... in data and imagery sequences, using a variety of sensors and information sources, and the problems of sensor and information fusion The first three chapters review mathematical and philosophical concepts of intelligence and mind Chapter 1, the introduction, begins with the discussion of mathematical approaches to intelligence during the past 50 years and their relationships to philosophical concepts. .. mathematics, engineering, philosophy, psychology, and linguistics and analyze fundamental computational concepts of major algorithmic and neural network paradigms This analysis provides continuity to a large variety of seemingly disparate techniques and establishes relationships between contemporary computational concepts of modeling intellect and concepts of mind discussed over 2300 years I found... 256 5.7 6 254 MLANS, PHYSICS, BIOLOGY, AND OTHER NEURAL NETWORKS 257 Note 260 Bibliographical Notes 260 Problems 261 Einsteinian Neural Network 6.1 225 263 IMAGES, SIGNALS, AND SPECTRA 263 6.1.1 Definitions, Notations, and Simple Signal Models 263 6.1.2 Frequency Components, Spectrum, and Spectral Models 6.1.3 Model-Based Spectrum Estimation 269 265 Contents 6.2 6.3 NEURAL DYNAMICS OF ENN 271 6.3.1 Shannon’s... course in AI, neural networks, pattern recognition, signal processing, or control Level: graduate or undergraduate Chapter 2, Chapter 3, Sec 3.1, Chapters 10 through 12 6 Relationships between Philosophical and Mathematical Concepts of Mind (for students without hard-science background) Relationships between contemporary mathematical concepts of intelligence and 2300-year-old philosophical concepts of . Neural Networks and Intellect: Using Model-Based Concepts Leonid I. Perlovsky OXFORD UNIVERSITY PRESS NEURAL NETWORKS AND INTELLECT This page intentionally left blank NEURAL NETWORKS AND INTELLECT Using. Cataloging-in-Publication Data Perlovsky, Leonid I. Neural networks and intellect : using model-based concepts / Leonid I. Perlovsky. p. cm. Includes bibliographical references and index. ISBN 0-1 9-5 1116 2-1 1 of neural networks based on this theory to a variety of problems; and analyzes relationships among mathematics, computational concepts in neural networks, and concepts of mind in psychology and