Abstract: |
The goal of supervised learning is to adjust parameters of a neural network so that it approximates with a desired accuracy a functional relationship between inputs and outputs by learning from a set of examples in such a way that the network has a generalization capability, i.e., it can be used for processing new data that were not used for learning. To guarantee generalization, one needs some global knowledge of the desired input/output functional relationship, such as smoothness and lack of high frequency oscillations. The lecture will present various approaches to modelling of learning with generalization capability based on regularization methods. Learning as a regularized optimization problem will be studied for a special class of function spaces called reproducing kernel Hilbert spaces, in which many types of oscillations and smoothness conditions can be formally described. It will be shown how methods developed for treating inverse problem related to differential equations from physics can be used as tools in mathematical theory of learning. There will be described properties and relationships of important types of regularization techniques (Ivanov's regularization based on a restriction of the space of input/output functions, Tychonov's one adding to an empirical error enforcing fitting to empirical data a term penalizing undesired properties of input/output function and Miller's and Philips' combining Ivanov's and Tychonov's method). Various versions of the Representer Theorem describing the form of the unique solution of the learning problem will be derived. Algorithms based on such theorems will be discussed and compared with typical neural network algorithms designed for networks with limited model complexity. |
ICSC
