Inference

The three layers you have to get right

A generative model is trained once and then serves for the rest of its life. This chapter is about that second phase: what serving actually involves, and the three layers of engineering it takes to do well.

6 sections

0.1Two phases, two disciplines
0.2The three layers
0.3Six techniques that define the runtime
0.4Scale changes the problem
0.5Where to put the abstraction
0.6A map of what follows