[flyingdoubleG]

Description

XGLM adds a sinusoidal position table to the scaled token embedding before the first block:

# XGLMModel.forward
hidden_states = inputs_embeds + self.embed_positions(position_ids, past_key_values_length)

The adapter did not bridge model.embed_positions. There was no hook_pos_embed, so hook_embed != blocks.0.hook_resid_pre, and every residual decomposition missed the positional term. On a tiny random XGLM (3 layers), the max abs error of each sum against the residual stream, relative to its max abs value:

	dev	this PR
decompose_resid() (every layer)	1.0	0
get_full_resid_decomposition() (with and without neurons)	1.0	4e-7
hook_embed + hook_pos_embed vs blocks.0.hook_resid_pre	no hook_pos_embed	0

Fix. Bridge model.embed_positions as pos_embed with the existing PosEmbedBridge. It already handles sinusoidal modules that keep their table in a weights buffer (as on M2M100), so W_pos is the table, and HF's (position_ids, past_key_values_length) call passes through unchanged. Logits, compatibility mode and greedy generate still match HF. supports_center_writing_weights stays False: the table is a non-persistent buffer, not a weight.

Type of change

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• I have commented my code, particularly in hard-to-understand areas

• I have made corresponding changes to the documentation

• My changes generate no new warnings

• I have added tests that prove my fix is effective or that my feature works

• New and existing unit tests pass locally with my changes

• I have not rewritten tests relating to key interfaces which would affect backward compatibility

• New tests/integration/model_bridge/test_xglm_pos_embed.py builds a tiny random XGLMForCausalLM locally and checks that:

  • logits match HF;
  • hook_embed + hook_pos_embed == blocks.0.hook_resid_pre;
  • hook_pos_embed is the offset sinusoidal table, and W_pos is the table;
  • decompose_resid sums to the residual stream at every layer;
  • get_full_resid_decomposition sums to the final residual stream and has a pos_embed row;
  • greedy generate matches HF.

• test_xglm_adapter.py: the top-level mapping now includes pos_embed, a PosEmbedBridge on model.embed_positions.

On dev, 10 of these tests fail.

Local runs (Windows, CPU, torch 2.9.1, transformers 5.20.2, --jaxtyping-packages=transformer_lens,beartype.beartype, PYTHONUTF8=1):

• The two files above: 34 passed.
• XGLM acceptance tests (test_symbolic_bridge_hooks.py, the XGLM case of test_inspect_provider.py): pass; the inspect one skips without inspect_ai.
• tests/unit -m "not slow": 8174 passed, 14 failed. The same 14 fail on dev in my environment (ten Gemma 3n adapter tests need the optional timm, three test_sparse_probing.py tests check optimizer stop reasons, and test_faithfulness_recovery_is_graded_not_binary is [Bug Report] test_faithfulness_recovery_is_graded_not_binary passes or fails on the sign of float32 rounding #1993).

black (the locked 23.12.1), isort --check-only and mypy on the changed files pass.

This was prepared with AI assistance (Claude); I reviewed and tested the change.