The Emergent Symbolic Structure of Artificial Neural Networks
Summary
Researchers show that neural networks implicitly realize symbolic structure through linearly-transformed Tensor Product Representations (TPRs), a mechanism they identify using the DISCOVER analysis method.
DISCOVER approximates neural encodings.
The method trains a model structured as a linearly-transformed TPR to reconstruct black-box representations, providing an interpretable, closed-form equation for complex, nonlinear encodings.
DISCOVER produces an interpretable closed-form equation for the representations of these models.
LLMs prioritize linear order.
In period-unpacking tasks, bidirectional role schemes outperform syntactic schemes, suggesting LLM representations capture sequential information rather than formal syntactic structure.
suggesting that the period encodings mainly capture the sentence’s linear order syntactic
GPT-OSS uses task-specific TPRs.
The 'all' role scheme provides the most accurate approximation for GPT-OSS, indicating the model constructs token representations that encode specific information about the preceding context.
the findings support the view that the way GPT-OSS performs these symbolic tasks is by constructing a TPR
Constituent surgery proves causality.
By subtracting and adding role-filler pairs, researchers achieved 0.903 average accuracy in causal interventions, demonstrating that identified structures are implicated in model behavior.
This result shows that GPT-OSS is indeed sensitive to the constituent structure 0.903
Generalization proves systematic binding.
DISCOVER's ability to generalize to novel role-filler pairs confirms that target models encode systematic pairings rather than just atomic concepts.
the learned representations of neural networks do not solely encode atomic concepts but rather
Read the original ↗Save to your Sponge
Summarized with Sponge. Quotes are the author’s own words.