← @bgSponge

The Emergent Symbolic Structure of Artificial Neural Networks

Summary

Researchers show that neural networks implicitly realize symbolic structure through linearly-transformed Tensor Product Representations (TPRs), a mechanism they identify using the DISCOVER analysis method.

  1. DISCOVER approximates neural encodings.

    The method trains a model structured as a linearly-transformed TPR to reconstruct black-box representations, providing an interpretable, closed-form equation for complex, nonlinear encodings.

    DISCOVER produces an interpretable closed-form equation for the representations of these models.

  2. LLMs prioritize linear order.

    In period-unpacking tasks, bidirectional role schemes outperform syntactic schemes, suggesting LLM representations capture sequential information rather than formal syntactic structure.

    suggesting that the period encodings mainly capture the sentence’s linear order syntactic

  3. GPT-OSS uses task-specific TPRs.

    The 'all' role scheme provides the most accurate approximation for GPT-OSS, indicating the model constructs token representations that encode specific information about the preceding context.

    the findings support the view that the way GPT-OSS performs these symbolic tasks is by constructing a TPR

  4. Constituent surgery proves causality.

    By subtracting and adding role-filler pairs, researchers achieved 0.903 average accuracy in causal interventions, demonstrating that identified structures are implicated in model behavior.

    This result shows that GPT-OSS is indeed sensitive to the constituent structure 0.903

  5. Generalization proves systematic binding.

    DISCOVER's ability to generalize to novel role-filler pairs confirms that target models encode systematic pairings rather than just atomic concepts.

    the learned representations of neural networks do not solely encode atomic concepts but rather

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →