modality

Definition

A type of input signal — text, image, audio, video, depth, sensor streams. Multimodal means handling more than one; cross-modal means transferring between them. The engineering problem is that modalities differ in structure, sampling rate, and statistics, so most designs turn on how to project them into a shared representation space and how to keep that space aligned.

Explainers using this term

• Paper Deep-Dive: The 'Physics' of Multimodal Pretraining — Which Way Does Knowledge Actually Flow?Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes