Yang Xu

on the atlas — 3

• 2502.110893 savers
• A Telescope, Pointed at the Stars | Tilde3 savers
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learning23 savers

highlights — 3

• NSA reduces per-query computation by organizing keys and values into temporal blocks and processing them through three attention paths: compressed coarse-grained tokens, selectively retained fine-grained tokens, and sliding windows for local contextual information.

  2502.11089

• By contrast, there are several lines of work within the sequence modeling and architecture research literature which do exemplify the “development through understanding” approach outlined above (despite not typically being classified as “interpretability” papers). In particular: Fu et al.'s H3 builds on Olsson et al.'s work on induction heads by identifying a synthetic task where two state-space models underperform. This insight motivates the design of the H3 layer, which introduces induction-like mechanisms into state-space models[15]. Zhu et. al's Transformers without Normalization identifie…

  A Telescope, Pointed at the Stars | Tilde

• To assess the effect of dataset correlations on the interpretability of feature activations, we run dictionary learning on a version of our one-layer model with random weights.The resulting features are here, and contain many single-token features (such as "span", "file", ".", and "nature") and some other features firing on seemingly arbitrary subsets of different broadly recognizable contexts (such as LaTeX or code). However, we are unable to construct interpretations for the non-single-token features that make much sense and invite the reader to examine feature visualizations from the model …

  Towards Monosemanticity: Decomposing Language Models With Dictionary Learning