1 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #1 NEURAL NETWORKS Lecturer: Primož Potočnik University of Ljubljana Faculty of Mechanical Engineering Laboratory of Synergetics www.neural.si primoz.potocnik@fs.uni-lj.si +386-1-4771-167 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #2 TABLE OF CONTENTS 0. Organization of the Study 1. Introduction to Neural Networks 2. Neuron Model – Network Architectures – Learning 3. Perceptrons and linear filters 4. Backpropagation 5. Dynamic Networks 6. Radial Basis Function Networks 7. Self-Organizing Maps 8. Practical Considerations 2 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #3 0. Organization of the Study 0.1 Objectives of the study 0.2 Teaching methods 0.3 Assessment 0.4 Lecture plan 0.5 Books 0.6 SLO books 0.7 E-Books 0.8 Online resources 0.9 Simulations 0.10 Homeworks © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #4 1. Objectives of the study • Objectives – Introduce the principles and methods of neural networks (NN) – Present the principal NN models – Demonstrate the process of applying NN • Learning outcomes – Understand the concept of nonparametric modelling by NN – Explain the most common NN architectures • Feedforward networks • Dynamic networks • Radial Basis Function Networks • Self-organized networks – Develop the ability to construct NN for solving real-world problems • Design proper NN architecture • Achieve good training and generalization performance • Implement neural network solution 3 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #5 2. Teaching methods • Teaching methods: 1. Lectures 4 hours weekly, clasical & practical (MATLAB) • Tuesday 9:15 - 10:45 • Friday 9:15 - 10:45 2. Homeworks home projects 3. Consultations with the lecturer • Organization of the study – Nov – Dec: lectures – Jan: homework presentations – Jan: exam • Location – Institute for Sustainable Innovative Technologies, (Pot za Brdom 104, Ljubljana) © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #6 3. Assessment • ECTS credits: – EURHEO (II): 6 ECTS • Final mark: – Homework 50% final mark – Written exam 50% final mark • Important dates – Homework presentations: Tue, 8 Jan 2013 Fri, 11 Jan 2013 – Written exam: Fri, 18 Jan 2013 4 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #7 4. Lecture plan (1/5) 1. Introduction to Neural Networks 1.1 What is a neural network? 1.2 Biological neural networks 1.3 Human nervous system 1.4 Artificial neural networks 1.5 Benefits of neural networks 1.6 Brief history of neural networks 1.7 Applications of neural networks 2. Neuron Model, Network Architectures and Learning 2.1 Neuron model 2.2 Activation functions 2.3 Network architectures 2.4 Learning algorithms 2.5 Learning paradigms 2.6 Learning tasks 2.7 Knowledge representation 2.8 Neural networks vs. statistical methods © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #8 4. Lecture plan (2/5) 3. Perceptrons and Linear Filters 3.1 Perceptron neuron 3.2 Perceptron learning rule 3.3 Adaline 3.4 LMS learning rule 3.5 Adaptive filtering 3.6 XOR problem 4. Backpropagation 4.1 Multilayer feedforward networks 4.2 Backpropagation algorithm 4.3 Working with backpropagation 4.4 Advanced algorithms 4.5 Performance of multilayer perceptrons 5 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #9 5. Dynamic Networks 5.1 Historical dynamic networks 5.2 Focused time-delay neural network 5.3 Distributed time-delay neural network 5.4 NARX network 5.5 Layer recurrent network 5.6 Computational power of dynamic networks 5.7 Learning algorithms 5.8 System identification 5.9 Model reference adaptive control 4. Lecture plan (3/5) © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #10 6. Radial Basis Function Networks 6.1 RBFN structure 6.2 Exact interpolation 6.3 Commonly used radial basis functions 6.4 Radial Basis Function Networks 6.5 RBFN training 6.6 RBFN for pattern recognition 6.7 Comparison with multilayer perceptron 6.8 RBFN in Matlab notation 6.9 Probabilistic networks 6.10 Generalized regression networks 4. Lecture plan (4/5) 6 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #11 7. Self-Organizing Maps 7.1 Self-organization 7.2 Self-organizing maps 7.3 SOM algorithm 7.4 Properties of the feature map 7.5 Learning vector quantization 8. Practical considerations 8.1 Designing the training data 8.2 Preparing data 8.3 Selection of inputs 8.4 Data encoding 8.5 Principal component analysis 8.6 Invariances and prior knowledge 8.7 Generalization 4. Lecture plan (5/5) © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #12 5. Books 1. Neural Networks and Learning Machines, 3/E Simon Haykin (Pearson Education, 2009) 2. Neural Networks: A Comprehensive Foundation, 2/E Simon Haykin (Pearson Education, 1999) 3. Neural Networks for Pattern Recognition Chris M. Bishop (Oxford University Press, 1995) 4. Practical Neural Network Recipes in C++ Timothy Masters (Academic Press, 1993) 5. Advanced Algorithms for Neural Networks Timothy Masters (John Wiley and Sons, 1995) 6. Signal and Image Processing with Neural Networks Timothy Masters (John Wiley and Sons, 1994) 7 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #13 6. SLO Books 1. Nevronske mreže Andrej Dobnikar, (Didakta 1990) 2. Modeliranje dinamičnih sistemov z umetnimi nevronskimi mrežami in sorodnimi metodami Juš Kocijan, (Založba Univerze v Novi Gorici, 2007) © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #14 7. E-Books (1/2) List of links at www.neural.si – An Introduction to Neural Networks Ben Krose & Patrick van der Smagt, 1996 – Neural Networks - Methodology and Applications Gerard Dreyfus, 2005 – Metaheuristic Procedures for Training Neural Networks Enrique Alba & Rafael Marti (Eds.), 2006 – FPGA Implementations of Neural Networks Amos R. Omondi & Mmondi J.C. Rajapakse (Eds.), 2006 – Trends in Neural Computation Ke Chen & Lipo Wang (Eds.), 2007 Recommended as an easy introduction 8 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #15 7. E-Books (2/2) – Neural Preprocessing and Control of Reactive Walking Machines Poramate Manoonpong, 2007 – Artificial Neural Networks for the Modelling and Fault Diagnosis of Technical Processes Krzysztof Patan, 2008 – Speech, Audio, Image and Biomedical Signal Processing using Neural Networks [only two chapters], Bhanu Prasad & S.R. Mahadeva Prasanna (Eds.), 2008 – MATLAB Neural Networks Toolbox 7 User's Guide, 2010 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #16 8. Online resources List of links at www.neural.si • Neural FAQ – by Warren Sarle, 2002 • How to measure importance of inputs – by Warren Sarle, 2000 • MATLAB Neural Networks Toolbox (User's Guide) – latest version • Artificial Neural Networks on Wikipedia.org • Neural Networks – online book by StatSoft • Radial Basis Function Networks – by Mark Orr • Principal components analysis on Wikipedia.org • libsvm – Support Vector Machines library 9 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #17 9. Simulations • Recommended computing platform – MATLAB R2010b (or later) & Neural Network Toolbox 7 http://www.mathworks.com/products/neuralnet/ Acceptable older MATLAB release: – MATLAB 7.5 & Neural Network Toolbox 5.1 (Release 2007b) • Introduction to Matlab – Get familiar with MATLAB M-file programming – Online documentation: Getting Started with MATLAB • Freeware computing platform – Stuttgart Neural Network Simulator http://www.ra.cs.uni-tuebingen.de/SNNS/ © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #18 10. Homeworks • EURHEO students (II) 1. Practical oriented projects 2. Based on UC Irvine Machine Learning Repository data http://archive.ics.uci.edu/ml/ 3. Select data set and discuss with lecturer 4. Formulate problem 5. Develop your solution (concept & Matlab code) 6. Describe solution in a short report 7. Submit results (report & Matlab source code) 8. Present results and demonstrate solution • Presentation (~10 min) • Demonstration (~20 min) 10 Video links • Robots with Biological Brains: Issues and Consequences Kevin Warwick, University of Reading http://videolectures.net/icannga2011_warwick_rbbi/ • Computational Neurogenetic Modelling: Methods, Systems, Applications Nikola Kasabov, University of Auckland http://videolectures.net/icannga2011_kasabov_cnm/ © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #19 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #20 [...]...1 Introduction to Neural Networks 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 © 2012 Primož Potočnik What is a neural network? Biological neural networks Human nervous system Artificial neural networks Benefits of neural networks Brief history of neural networks Applications of neural networks List of symbols NEURAL NETWORKS (1) Introduction to Neural Networks #21 1.1 What is a neural network?... blocks of all neural networks Similar NN architecture for various tasks: pattern recognition, regression, time series forecasting, control applications, © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #37 www.stanford.edu/group/brainsinsilicon/ © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #38 19 1.6 Brief history of neural networks (1/2) -1940... feedforward networks (1/2) © 2012 Primož Potočnik NEURAL NETWORKS (2) Neuron Model, Network Architectures and Learning #57 Multi-layer feedforward networks (2/2) • Data flow strictly feedforward: input  output • No feedback  Static network, easy learning © 2012 Primož Potočnik NEURAL NETWORKS (2) Neuron Model, Network Architectures and Learning #58 29 Recurrent networks (1/2) • Also called “Dynamic networks ... pathways, topographic maps Central nervous system final level of complexity © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #32 16 1.4 Artificial neural networks • Neuron model • Network of neurons © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #33 What NN can do? • In principle – NN can compute any computable function (everything a normal digital... neural networks are made up of real biological neurons that are connected or functionally-related in the peripheral nervous system or the central nervous system • Artificial neurons – Simple mathematical approximations of biological neurons © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #22 11 What is a neural network? (2/2) • Artificial neural networks – – – – – Networks. .. Hopfield networks 1982 Kohonen – Developed the Self-Organising Maps 1980s Rumelhart and McClelland – Backpropagation rediscovered, re-emergence of neural networks field – Books, conferences, courses, funding in USA, Europe, Japan 1990s Radial Basis Function Networks were developed 2000s The power of Ensembles of Neural Networks and Support Vector Machines becomes apparent © 2012 Primož Potočnik NEURAL NETWORKS. .. layer) 2 Multi-layer feedforward networks • • One or more hidden layers Can extract higher-order statistics 3 Recurrent networks • • Contains at least one feedback loop Powerfull temporal learning capabilities © 2012 Primož Potočnik NEURAL NETWORKS (2) Neuron Model, Network Architectures and Learning #55 Single-layer feedforward networks © 2012 Primož Potočnik NEURAL NETWORKS (2) Neuron Model, Network... crude approximations of small parts of biological brain Implemented as software or hardware By “Neural Networks we usually mean Artificial Neural Networks Neurocomputers, Connectionist networks, Parallel distributted processors, © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #23 Neural network definitions • Haykin (1999) – A neural network is a massively parallel distributed... or glands 0.1 mm © 2012 Primož Potočnik This complex network forms the nervous system, which relays information through the body NEURAL NETWORKS (1) Introduction to Neural Networks #25 Biological neuron © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #26 13 Interaction of neurons • • These currents depolarize the membrane at its axon, provoking an action potential • © 2012... and Hoff – Perceptron, ADALINE – First practical networks and learning rules 1969 Minsky and Papert – Published book Perceptrons, generalised the limitations of single layer perceptrons to multilayered systems – Neural Network field went into hibernation © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #39 Brief history of neural networks (2/2) 1974 Werbos – Developed back-propagation . Explain the most common NN architectures • Feedforward networks • Dynamic networks • Radial Basis Function Networks • Self-organized networks – Develop the ability to construct NN for solving. Human nervous system 1.4 Artificial neural networks 1.5 Benefits of neural networks 1.6 Brief history of neural networks 1.7 Applications of neural networks 2. Neuron Model, Network Architectures. Neural Networks Toolbox (User's Guide) – latest version • Artificial Neural Networks on Wikipedia.org • Neural Networks – online book by StatSoft • Radial Basis Function Networks