Multi-step learning and underlying structure in statistical models

Speaker: Maia Fraser (uOttawa)

Time: 2:30-3:30, Nov 11, 2016

Room: KED B004

Title: Multi-step learning and underlying structure in statistical models

Abstract: 

In semi-supervised learning, or multi-step learning more generally, the intuition is that successive learning tasks transform the initial data into representations more and more "suited" to the final learning task. A related principle arises in transfer learning where Baxter (2000) proposed a theoretical framework to study how learning multiple tasks transforms the inductive bias of a learner. Analysis of the benefit of unlabeled data in a semi-supervised approach began with the work of Castelli-Cover (1996) and has been revisited by several authors (Balcan-Blum, 2005; Niyogi, 2008; Ben-David et al, 2008; Urner et al, 2011). One immediate and easy observation is that without some link between concept class and unlabeled data distribution, unlabeled data is of no benefit for a final, labeled, task. This means the classic PAC learning framework is not suited to study semi-supervised learning situations. I will describe a novel framework, partly inspired by Baxter's, which can be used quite fruitfully to analyse two key semi-supervised settings: manifold learning and invariant-feature leaning. This framework also allows one to consider smooth statistical models, something not addressed by PAC learning. As part of this I will describe a notion of complexity of statistical models which can be viewed as a generalization of VC dimension, a tool commonly used to prove lower- and upper-bounds in the PAC framework.