How can I model cellular membranes in terms of topology?
As Buck Thorn and MaxW already said in the comments, there is less to it than the textbook excerpt might suggest. The use of the term topology in this context is non-mathematical, similar to when someone says there has been a quantum leap in product development.

The actual topology of most proteins is the same as that of a sphere, and the topology of the typical cartoon channel is that of a donut. Having hydrophilic residues in the channel or outside of the channel is not an inversion because those parts of the surface are topologically equivalent.


  [OP] I can't seem to state this membrane chemistry in terms of the mathematical formalism.


That's because it is non-mathematical. There is some real topology associated with membranes, and that is the concept of inside and outside. Membrane proteins insert in one direction, and they don't "flip" directions over time, even though they are perfectly mobile to move around in the membrane (or have lipids move around them) while pointing in one direction or the other one. During endocytosis and exocytosis, the relative orientation of membrane proteins is conserved but inverted, so that parts that are inside organelles (such as glycosylation) end up on the outside of the plasma membrane.