T3Time: Tri-Modal Time Series Forecasting via Adaptive Multi-Head Alignment and Residual Fusion

Three ways to read one series

Multivariate forecasting takes the last \(L\) observations of \(N\) variables and predicts the next \(L_p\). Architectures have split along the representation they trust. Time-domain Transformers such as PatchTST and iTransformer learn from raw values; frequency-domain models such as FEDformer model periodicity; and a newer line prompts a pretrained language model, either by reprogramming it (Time-LLM) or by aligning series and prompt embeddings (TimeCMA).

Each of the \(\lfloor 96/2 \rfloor + 1 = 49\) bins becomes one token for the frequency encoder. Noise spreads across every bin, but a cycle stays in one, which is why a few spectral tokens can summarise periodic structure that would take attention over the whole window to find in the time domain. The series is z-scored first, as the model's input is.

Encode three views, gate by horizon, align, and keep a residual path

Figure 2. The T3Time architecture. Three encoding branches (frequency, time series, and a frozen LLM over text prompts), a horizon-aware gate that fuses time and frequency, multi-head cross-modal alignment with per-head importance weights, and a channel-wise residual connection before the decoder.

Tri-modal encoding

For a normalised input \(X \in \mathbb{R}^{B \times N \times L}\), the frequency branch takes the real FFT along time and keeps the magnitude. Each of the \(L_f\) bins is a token: projected, encoded by a one-layer Transformer, and pooled with learned attention weights over the bins.

The time branch embeds each variable's whole window, \(Z_t = X W_t\), and encodes it with a Transformer into \(\tilde Z_t\). The language branch writes one prompt per variable, runs it through a frozen GPT-2, and keeps the last token's embedding as \(Z_{\text{LLM}}\):

# prompt template, one per variable and window
From [t1] to [t2], the values were value1, ..., valuen every f. The total trend value was T.

Horizon-aware gating

Short forecasts should lean on local, time-domain detail and long ones on global periodicity. A small MLP sees the pooled time encoding and the normalised prediction length, and outputs a per-channel gate that mixes the two views:

Adaptive multi-head cross-modal alignment

Following TimeCMA, the fused temporal–spectral features query the prompt embeddings through cross-attention, but with \(H\) independent heads instead of one. A gating network then scores the heads for each variable and takes their convex combination, so different variables can rely on different alignments:

where \(U\) concatenates the head outputs. The model uses \(H = 4\); a sweep over 1 to 16 heads finds four best on average, with more heads fragmenting the attention rather than helping.

Channel-wise residual fusion

Alignment should add context, not overwrite what the series already says. A learned coefficient per latent channel, \(\gamma_c \in [0,1]\), decides how much of each channel comes from the aligned features and how much from the gated temporal–spectral features, before a Transformer decoder and a linear head produce the forecast:

Long-term forecasting on eight benchmarks

Following TimeCMA's protocol, the input window is 96 steps and each score averages four horizons, \(\{96, 192, 336, 720\}\) (for ILI, a 36-step input and \(\{24, 36, 48, 60\}\)). The eight datasets cover electricity transformers (ETTh1/2, ETTm1/2), electricity consumption (ECL), weather, influenza-like illness (ILI) and exchange rates. Each result averages three seeds.

Negative is better. The bar runs from zero on a shared −12% to +4% scale; it is coloured where T3Time has the lower error.

With a tenth or a twentieth of the data

Following Time-LLM's few-shot setup, models train on only 10% or 5% of the training time steps, with a 512-step input window. This is where pretrained language priors are supposed to pay off, and where the gap to non-LLM baselines grows: TimesNet, DLinear and PatchTST trail by more than 15% in MSE.

Every component helps; the residual path most

Removing one component at a time under the long-term setup, the full model is best, or tied for best, in all 14 dataset–metric pairs of the design study.

What the three views learn

Figure 3. t-SNE of each modality's embeddings, coloured by dataset. Time-series and especially frequency embeddings separate the datasets cleanly, since each has its own periodicities. Prompt embeddings are more mixed, reflecting how similar the textual descriptions are. The forecast embeddings cluster much like the time-series ones.

Scope: all results use standard public benchmarks and a frozen GPT-2 for the prompt branch. Larger-scale pretraining and richer representations for each modality are left for future work.

Citation

If this work is useful to you, please cite it as:

@inproceedings{chowdhury2026t3time,
  title     = {{T3Time}: Tri-Modal Time Series Forecasting via Adaptive
               Multi-Head Alignment and Residual Fusion},
  author    = {Chowdhury, Abdul Monaf and Akter, Rabeya and Arib, Safaeid Hossain},
  booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
  volume    = {40},
  number    = {25},
  pages     = {20597--20605},
  year      = {2026},
  doi       = {10.1609/aaai.v40i25.39196}
}