Continuous causal attention operator (CCT)

structures / Linear Structures / Continuous causal attention operator (CCT)

TOOLoperatornonlinearintegral-attentionadvancedhow it's classified →

A generalization of discrete causal self-attention that replaces the sum over token positions with an integral over a continuous time variable, applied to a continuous-time embedding function x(t) rather than a discrete token sequence — using the same pretrained weight matrices unchanged, and proven to reduce exactly to standard discrete attention when the continuous function is stepwise-constant with unit-duration steps.

Replicationcomputed from the corpus — never hand-assigned

1 paper1 architecture class1 domain5 model families · one study

Filled = two or more values reported by papers that share no author — replication. Outlined = two or more values, but all from a single study — breadth, not replication. Grey = a single value. Derived from paper authorship and each model's architecture class, domain and family; it updates itself when a paper is added.

Definition

Assume a discrete sentence $\{x_1,\ldots,x_T\}$ is really an integer sampling of an underlying continuous-time function $x(t)$ on $[0,T]$ , taken in the simplest case as stepwise-constant: $x(t)=\sum_{s=1}^T x_s\cdot\mathbb{1}_{[a_s,b_s]}(t)$ , subject to the intervals partitioning $[0,T]$ and $s\in[a_s,b_s]$ ; each token’s duration is $d_s:=b_s-a_s$ . The Continuous Causal Transformer (CCT) replaces discrete causal attention with an integral over this continuous function:

$y(t)=\int_0^t \frac{1}{Z_t}\exp\!\Big(\frac{q(t)^\top k(s)}{\sqrt d}\Big)\, v(s)\, ds$

where $q(t)=W^{(q)}x(t)$ , $k(t)=W^{(k)}x(t)$ , $v(t)=W^{(v)}x(t)$ use the same, unchanged pretrained weight matrices — the operator is a generalization of the attention mechanism applied to a richer input space, not a retrained model. Dropping the constraint that $x(t)$ takes values only in the discrete embedding lattice $\mathcal X=\mathrm{Rg}(\mathcal E)\subset\mathbb R^d$ (so arbitrary points of $\mathbb R^d$ , e.g. linear interpolations between two token embeddings, are valid continuous “sentence” values) is the companion space-continuity extension.

Exact reduction to discrete attention

Under the stepwise-constant assumption, the integral is proven (not just argued) to reduce exactly to $y(t)=\sum_k \frac{1}{Z_t}\exp\big(q_{\bar t}^\top k_k/\sqrt d\big)v_k\,d_k$ — standard discrete causal attention exactly recovered when every duration $d_k=1$ . A companion equivalence shows Euler-discretizing the integral is the same as standard attention with duration-weighted coefficients. CCT is therefore not a new architecture requiring retraining, but a strict mathematical generalization of the operator already implemented by any pretrained Transformer’s weights.

Key evidence

Marro, Evangelista, Huang, La Malfa, Lombardi & Wooldridge (2025, ICLR 2025) run pretrained Llama2-13B-Chat, Llama3-8B, Phi-3-Medium-4k-Instruct, Gemma-1-7B, Gemma-2-9B and Mistral-7B through the CCT operator with two kinds of off-lattice continuous input intervention, in both cases directly manipulating the input and measuring the behavioral output effect:

Time continuity — continuously shrinking a repeated token’s duration (e.g. “apple” repeated $n$ times, duration factor swept over $[0.1,1]$ ) and recording which count the model outputs: a genuinely discrete model should output a fixed count almost everywhere in the sweep (one “peak”), but every model tested instead outputs multiple distinct counts as duration varies continuously — normalized peak frequency averages 0.74 observed vs. 0.26 expected under the discrete-model counterfactual for single-token duration shrinking (up to 3.60x more distinct outputs than the discrete baseline, e.g. Llama2-13B 0.26 $\to$ 0.92), and 0.67 vs. 0.22 (3.28x) for full-sentence event-duration shrinking on 200 how-to tutorials — replicated across all six models.

Space continuity — linearly interpolating between two token embeddings (200 records, 40 interpolation steps) and measuring output smoothness via a normalized max-derivative (empirical Lipschitz) estimate (global average 6.214, range 4.96-9.28 across models) and an excursion statistic $m_{\max}$ (how far the interpolated output probability strays outside the range spanned by the two endpoint outputs; global average 0.034, with 20.79% of records exceeding a 0.05 threshold). Off-lattice embedding points — which map to no real token — still produce smoothly-varying, semantically consistent next-token distributions rather than noise (e.g. an “apple”-“banana” interpolation passing through an intermediate point whose top prediction is “green,” a non-trivial semantic composition of the two fruits’ colors, not a simple average). A translation/scaling ablation (Appendix D) confirms CCT output is empirically shift-invariant (shifting all positions by up to 10 has negligible effect) but not scale-invariant (uniformly scaling durations changes output substantially), consistent with RoPE’s known translation-invariance; the effect is weaker, but still present, in GPT-2 (learned positional embeddings, no RoPE).

See shrinking-token-duration-or-interpolating-embeddings-produces-multiple-distinct-smoothly-varying-outputs-from-pretrained-llms-under-a-provably-attention-equivalent-continuous-time-operator.

Relative to manifolds-1d-continuum and linear-direction

Unlike 1D continuum manifold or Linear Direction, which describe a learned geometric shape found by probing already-trained representations, this operator is a architecture-level mathematical generalization applied at inference time — the “continuity” claim is about the function the (unmodified) weights compute on an enlarged, continuous input domain, not about a shape discovered in the model’s own activations.