Chapter 6 · Modalities
Beyond text: vision, speech, embeddings, pixels
Chapter 6pp. 153–176

Modalities

Beyond text: vision, speech, embeddings, pixels

Most non-text models are transformer variants, which means the toolkit transfers almost intact. Image and video generation are the exception, with a different architecture and different bottlenecks.

Vision-language models, multimodal embeddings, speech recognition, and speech synthesis all adapt LLM architectures. That is genuinely good news: the same engines serve them, the same quantization and batching and caching apply, and the intuitions from Chapter 2 hold.

What changes is the metric. Time to first token is meaningless for a model whose output is an image, and the first 20 milliseconds of audio from a TTS model is not useful on its own. Each modality measures something different, and knowing which number matters is most of the work.

6 sections

  1. 6.1Vision language modelsVideo processing for vision language models · Omni-modal models
  2. 6.2Embedding modelsEmbedding model architecture · Embedding model inference
  3. 6.3ASR modelsSingle-chunk latency optimization · Long file latency optimization · Diarization
  4. 6.4TTS modelsStreaming real-time text to speech · Speech-to-speech models
  5. 6.5Image generation modelsImage generation kernel optimization · One weird trick for faster image generation
  6. 6.6Video generation modelsAttention optimization and quantization · Context parallelism