Tabular Foundation Models
A Short And Opinionated Guide
Tabular foundation models like TabPFN are weird. On the surface, using them is the same as any machine learning algorithm: fit, then predict. Looking deeper, they work very differently, and they may transform tabular machine learning as we know it.
Where To Read
The book is free to read online. All planned chapters are written; it’s currently in beta while I fix errors and polish.
Summary
Tabular foundation models are pretrained on millions of mostly simulated datasets. To make predictions, you don’t train the model on your specific task. Instead, prediction is based on in-context learning: you provide the training data during inference. There are no hyperparameters to tune, and no model weights to update.
I found models like TabPFN puzzling at first. Now I’m convinced they may transform tabular machine learning, and that their transformative potential is not communicated enough. This is a short and opinionated guide to help you:
- Understand how tabular foundation models work.
- Apply these models to your own data.
- Form your own opinion on whether to bet on them.
Who This Book Is For
I strongly feel everyone in tabular machine learning should learn about tabular foundation models. If you know why we split data into train and test, and have at least once in your life written the line “.predict()”, this book is for you. Fair warning: the book is short and opinionated, and that writing style won’t suit everyone.
What’s In The Book
- Introduction
Understanding tabular foundation models
- First Look
- Prior-Data Fitted Networks
- In-Context Learning
- Pretraining
Applying tabular foundation models
- Classification
- Regression
- Quantile Regression
- Time Series Forecasting
Betting on tabular foundation models
- Tabular’s GPT Moment
- Inheriting the LLM Stack
- Roadmap and Challenges