NEO 1.5: Identity-Aware Talking-Head Performance

NEO 1.5 keeps the two-stage pipeline, the diffusion foundation, and the performance data from NEO. The change is that speaker identity now sits inside the attention pass rather than around it.

In NEO, audio and motion were the two streams that reasoned together. Identity shaped the result from outside the attention pass, as a conditioning signal. NEO 1.5 extends that joint attention from two streams to three, so identity becomes a stream of its own. It co-evolves with audio and motion inside the same pass rather than steering them from the side.

The clearest way to see the difference is to watch a speaker hold a half-second pause. In NEO, the face goes still because nothing in the attention pass knows whose face it is. In NEO 1.5, the eyes refocus and the breath comes back, because identity sits in the same pass as audio and motion.

In NEO, identity entered as a side conditioner around the attention pass. Audio and motion reasoned together; identity influenced the result from outside. In NEO 1.5, identity enters the attention pass itself as a third stream alongside audio and motion.

Putting three streams in one attention pass is the headline change. Two further refinements make that pass behave well at depth. Attention residuals act across the stack of layers, keeping early information reachable late. Differential attention works inside a single attention step, sharpening which tokens actually matter.

Attention Residuals: A Skip Across Depth

A standard deep transformer is a strictly sequential stack. Useful early features, like a clean audio onset or a stable identity signature, can get progressively overwritten through depth. Vanilla residuals add every layer's contribution with equal weight, so the model cannot decide which depth's representation matters most for a given input.

Attention residuals replace that uniform accumulation with a learned, input-dependent blend across depth. The simplest way to think about it: attention, but over layers instead of over tokens. At every block boundary the network blends all cached states with softmax weights rather than forwarding only the most recent one. The weights shift with the input, so the model can favour shallow or deep representations as needed.

In the unified design, that gives every stream a skip-across-depth highway. Motion can recover an early kinematic estimate if later layers over-smooth it. Audio can preserve sharp phonetic timing that deep mixing would blur. Identity can hold its speaker signature steady rather than drifting.

Differential Attention: Cancelling Attention Noise

Softmax attention almost never assigns exactly zero weight to irrelevant tokens. Summed over a long sequence, that attention noise dilutes the signal. In a multimodal field it is worse, because motion, audio, and identity tokens all compete in one place, so motion queries can leak attention onto unrelated audio frames and the reverse.

Differential attention cancels that common-mode noise, by analogy with a differential amplifier that rejects noise shared between two signal lines. Each head is split into two sub-heads that compute two independent softmax maps over the same tokens, and the output uses their difference. If both maps place spurious weight on the same irrelevant tokens, subtracting them cancels that shared mode, while the genuine signal survives.

The variant is compute neutral: splitting each head into two halves means there are half as many differential heads, each computing two maps, so the total cost matches standard attention.

VOCAL 1.5 stays a diffusion-based generator, and at inference time it still supports guided generation across the signals that define a performance. You tune how strongly each one steers the result.

Some clips want stronger audio responsiveness, others want stricter identity preservation or steadier reference behaviour. Try the balance below.

The table below summarises the five dimensions Unified Joint Attention moves. The pillars and architecture above explain why each one moves.

NEO 1.5 is an architectural change. The dataset, the diffusion foundation, and the reinforcement-learning recipe carry over from NEO. It runs on the same dataset that grounds NEO, so the quality gain comes from how the attention pass is wired, not from more data. NEO 1.5 sits between NEO and our full-body successor, NEO 2.