Definition
An inductive bias is an idea of what we are looking for: a preference for some functions over others that fit the data equally well. Learning is impossible without such a bias.
- Examples: the function is periodic, the function is βsimpleβ, similar inputs lead to similar outputs.
- Finitely many points are fit perfectly by infinitely many functions, so the data alone cannot decide between them.
- The loss function, the function class and the algorithm all contribute to the bias. If an algorithm works, it has a bias, even when it is hard to say which one.
- Animals have inductive biases too (Garcia effect: rats link taste, but not sounds, to sickness).
Appears in
- Lecture 1.1, why machine learning works at all
- Lecture 1.1, Garcia effect
- Lecture 1.1, different losses give different inductive biases
- Lecture 1.2, No Free Lunch: learning without an inductive bias is impossible