Energy Landscapes
Every polynomial Hopfield layer defines an energy landscape — a surface over the state space where local minima are stored memory patterns (basins).
State Space
Each layer operates in ℝ^d_model. A state is a vector x of dimension d_model. The energy function assigns a scalar value to every point in this space.
Energy Function
E(x) = -1/d · |x|^d + 0.5 · x^T · W · f(x)- First term: drives states toward the origin
- Second term: the weight matrix W creates local minima (basins)
f(x) = sign(x) · |x|^(d-1): polynomial interaction
Basins
A basin is a local minimum in the energy landscape — a point where the energy is lower than all nearby points. States near a basin will converge to it under the dynamics.
Discovery
find_basins() discovers basins by:
- Starting from 200 random initializations
- Running Krotov-Hopfield dynamics (annealed τ + tanh(βh)) to convergence
- Clustering converged states by cosine similarity (threshold: 0.9)
Properties
Each basin has:
- Energy: depth of the minimum (lower = more stable)
- Population: how many random initializations converged to it (wider = more reachable)
- Location: the d_model-dimensional state vector
Dynamics
The convergence dynamics that find basins:
h = W · f(x)
x_new = (1 - τ) · x + τ · tanh(β · h)- τ (tau): annealed from 0.9 to 0.1 over iterations
- β (beta): inverse temperature, controls basin sharpness
- Convergence: when
‖x_new - x‖ / ‖x‖ < ε
What Surgery Does
Surgery modifies W, which reshapes the energy landscape:
- Inject: creates a new local minimum
- Remove: fills in (raises) an existing minimum
- Move: fills one minimum and creates another
The landscape visualization tools let you see these changes in real time.