Pragmatic Apparatus – Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks

Cited by Lee Sonogan

Kernel of error | R-bloggers

Abstract by Abdulkadir Canatar,Blake Bordelon &Cengiz Pehlevan 

A theoretical understanding of generalization remains an open problem for many machine learning models, including deep networks where overparameterization leads to better performance, contradicting the conventional wisdom from classical statistics. Here, we investigate generalization error for kernel regression, which, besides being a popular machine learning method, also describes certain infinitely overparameterized neural networks. We use techniques from statistical mechanics to derive an analytical expression for generalization error applicable to any kernel and data distribution. We present applications of our theory to real and synthetic datasets, and for many kernels including those that arise from training deep networks in the infinite-width limit. We elucidate an inductive bias of kernel regression to explain data with simple functions, characterize whether a kernel is compatible with a learning task, and show that more data may impair generalization when noisy or not expressible by the kernel, leading to non-monotonic learning curves with possibly many peaks.

Publication: Nature Communications (Peer-Reviewed Journal)

Pub Date: 18 May 2021 Doi: (Plenty more sections, figures and references in this article)

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.