Deep Convolutional Networks are Hierarchical Kernel Machines

TitleDeep Convolutional Networks are Hierarchical Kernel Machines
Publication TypeCBMM Memo
Year of Publication2015
AuthorsAnselmi F., Rosasco L., Tan C., Poggio T.
NumberCBMM Memo No. 035
Date Published06/2015

We extend i-theory to incorporate not only pooling but also rectifying nonlinearities in an extended HW module (eHW) designed for supervised learning. The two operations roughly correspond to invariance and selectivity, respectively. Under the assumption of normalized inputs, we show that appropriate linear combinations of rectifying nonlinearities are equivalent to radial kernels. If pooling is present an equivalent kernel also exist. Thus present-day DCNs (Deep Convolutional Networks) can be exactly equivalent to a hierarchy of kernel machines with pooling and non-pooling layers. Finally, we describe a conjecture for theoretically understanding hierarchies of such modules. A main consequence of the conjecture is that hierarchies of eHW modules minimize memory requirements while computing a selective and invariant representation.

Citation Key12

CBMM Memo No.: