How is memory structured (from a modeling perspective)? My understanding is that memory is neither purely associationist (fully connected graph with weights) nor purely hierarchical (tree). What structure does biological evidence point towards? I'm interested in the mathematical representation.
The structure of memory depends on what you're trying to model. Since there are many types of memory, this question is almost too broad. So instead of giving you a systematic review, I'll give a quick tour of some models I know which should give you a starting point.

List memorization appears to be modeled by a combination of neuron weight updates and integrators saving symbols modeled as vectors. See "An Integrated Model of Context, Short-Term, and Long-Term Memory" by Jan Gosmann.

Task/procedural memory has been modeled as a stack-like data-structure while still representing the symbols as vectors. See "A scaleable spiking neural model of action planning" by Blouw et al.

Spatial memory is weirder and seems to be best modeled as a function space. See "Vector-based navigation using grid-like representations in artificial agents" by Banino et al.

Visual memory has some nice models too, but I don't know them that well.