Haris Riaz ·
Hyungji Kim ·
Mihai Surdeanu
Computational Language Understanding (CLU) Lab, University of Arizona
Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to syntactic structure. We introduce Syntax-informed Positional Embeddings (SiPE), which learns a lightweight syntactic prior from dependency parses during pretraining and injects it across all three dominant PE families (absolute, relative, rotary), for both encoders and decoders, leaving self-attention and the rest of the architecture untouched. We isolate where and how the prior should enter the model, and find it depends on the architecture: for autoregressive decoders that use relative PE, the prior is strongest when coupled multiplicatively with the relative-position term of the attention score, outperforming injection into the input embeddings, into self-attention, or into the positional and attention terms jointly — while for encoders it is best added directly to the input embeddings, composing with each encoder's native positional mechanism. We find that models pre-trained with SiPE improve on the SyntaxGym benchmark by up to 10.3% while simultaneously reducing perplexity by 9.0% over a base model with no syntactic supervision — a metric nearly every existing syntax-injection method instead degrades. Crucially, these gains extend beyond syntactic generalization: SiPE also improves real-world language understanding, raising scores on the GLUE benchmark by up to 8.2% over a model trained without it. Unlike existing syntactic language models that marginalize over many parses at inference or discard syntax at runtime, SiPE conditions on a single parse, establishing a new Pareto frontier between syntactic supervision and inference cost.
Overview of the approach. Every sequence the model sees — during pretraining, fine-tuning, and inference — is first hexatagged by a fast dependency parser. Left: the dependency parse is converted to a binary head tree (BHT) and read off as per-word hexatags: a terminal tag τ (2 values, the word's attachment direction) and a non-terminal tag ν (4 values from the BHT, plus an EOS tag introduced by the left shift). Right: the simplest injection, for absolute positional embeddings — each word's first subword adds one row from each tiny tag table (Eτ: 2×D, Eν: 5×D, ~7·D parameters in total) to its input embedding, and three prediction heads recover the token and both tags at masked positions (L = L_MLM + L_T + L_NT). For decoders, next-token prediction replaces MLM. For relative-PE decoders, the strongest variant instead multiplies the position term of the attention score by a syntax alignment: Ã = AC + (1 + c)·BD.
git clone https://github.com/hriaz17/SiPE.git && cd SiPE
pip install -e .Python 3.10, PyTorch 2.5.1 (CUDA 12.1) and Transformers 4.56.2 are what the
experiments were run with; requirements.txt pins the rest.
Transformer-XL was removed from Transformers in the 4.4x series, so a fork of it
lives under src/sipe/models/transfo_xl/ with its Apache-2.0 headers intact.
src/sipe/
tagging/ hexatag vocabularies (2 terminal, 5 non-terminal, 40 deprel)
models/ transfo_xl/ (decoder), roberta.py, modernbert.py
data/ SPM encoding, collators, sequence packing
training/ trainer, callbacks, checkpoint averaging
scripts/ train_txl.py, train_mlm.py, eval_*.py
configs/ one YAML per variant reported in the paper
All the injection sites we study live in one model. You pick between them with
--embedding-type, so switching from one variant to another means changing a
flag rather than swapping in a different file:
--embedding-type |
Injection site | Paper |
|---|---|---|
NONE |
no syntactic supervision | Table 2 baseline |
ADD |
input residual | Eq. 4 |
PE_BIAS |
the position term of the attention score | Eq. 5–7 |
ATTN_BIAS |
Shaw-style key/value biases | Eq. 8–9 |
PE_BIAS+ATTN_BIAS |
position and attention together | §3.3.5 |
ADD_AND_ATTN_BIAS |
input and attention together | Table 2 |
Under PE_BIAS, --pe-bias-mode multiplicative is the main result,
à = AC + (1 + c)·BD; additive and disentangled are the ablations.
# best decoder setting (SyntaxGym 80.60, perplexity 16.95)
python scripts/train_txl.py --config configs/decoder_pp_sipe.yaml \
--output-dir runs/pp_sipe
# encoder
python scripts/train_mlm.py --config configs/encoder_roberta_add.yaml \
--output-dir runs/roberta_addpython scripts/eval_syntaxgym.py --model_name_or_path runs/pp_sipe/final \
--tokenizer_name tokenizer/spm_bllip.model --output_path results/sg.json
python scripts/eval_blimp.py --model_name_or_path runs/pp_sipe/final \
--tokenizer_name tokenizer/spm_bllip.model
python scripts/eval_perplexity.py --model_name_or_path runs/pp_sipe/final \
--tokenizer_name tokenizer/spm_bllip.model --output_path results/ppl.json
python scripts/eval_glue.py --arch roberta --checkpoint_dir runs/roberta_add/final \
--output_dir results/glue --integration_mode ADD_T_NTSyntaxGym scores a suite item as correct only when every prediction inequality holds, and the headline number is the macro average over suites; surprisals are in bits with an equality tolerance of 1.0. BLiMP and perplexity use sentence log-likelihood.
Hexatagged pretrained checkpoints and their GLUE fine-tuned counterparts are on the Hub: SiPE model collection.
| Model | Pretrained on | Variant |
|---|---|---|
| Transformer-XL + SiPE | BLLIP-LG (hexatagged) | positional pathway, multiplicative |
| Transformer-XL | BLLIP-LG (hexatagged) | no syntactic supervision |
| RoBERTa-base / large + SiPE | WikiText-103 (hexatagged) | input pathway |
| DeBERTa-v3-base + SiPE | WikiText-103 (hexatagged) | input pathway |
| ModernBERT-base / large + SiPE | WikiText-103 (hexatagged) | input pathway |
Each encoder is released both as a pretrained checkpoint and fine-tuned on the eight GLUE tasks, alongside the matching no-prior baseline so the two sides of the comparison are available together.
Every sequence is hexatagged before it reaches the model, at training and at inference. The tagged corpora are on the Hub:
| Dataset | Used for |
|---|---|
hriaz/bllip-lg-hexatagged |
decoder pretraining, perplexity |
hriaz/syntaxgym-hexatagged |
SyntaxGym |
hyungjikim/blimp-with-hexatags |
BLiMP |
hyungjikim/wikitext-tags-roberta-v3 |
RoBERTa pretraining |
hyungjikim/wikitext-tags-modernbert-v3 |
ModernBERT pretraining |
hriaz/wikitext-tags-deberta-v3 |
DeBERTa-v3 pretraining |
hyungjikim/glue-with-hexatags_v3 |
GLUE fine-tuning |
Tags come from Hexatagger (Amini et al., 2023); we use a DeBERTa-v3-base tagger.
Encoder pretraining uses a 50M-token slice of WikiText-103, which is the
default for --token-limit. scripts/prepare_packed_data.py and
--pack-sequences concatenate examples into full-length blocks if you want
larger effective batches.
@misc{riaz2026sequenceordersyntaxinformedpositional,
title={Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers},
author={Haris Riaz and Hyungji Kim and Mihai Surdeanu},
year={2026},
eprint={2608.06111},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2608.06111},
}