Why doesn't Costco simply raise prices and nix the membership fee?
Because there is not a single successful model of actually doing business. Costco's model is pretty text-book level, but not economics textbook level, but rather marketing textbook level:  


Differentiate ("In Costco we do things differently").
Make the customer feel special (hey, here is "members only"!).  
Induce loyalty or at least "loyalty" ("now that I paid the executive fee I might as well buy from Costco to get part of it back")


Whether it will indeed be successful or not, depends on time, place, culture it operates in, as well as a few boring economic facts like, whether Costco's prices are indeed lower, etc.  

Economics analyzes firm behavior at a much more abstract level, "losing sight" of all these different business strategies because Economics is ultimately interested in market outcomes (and then economy-wide outcomes), not individual-firm outcomes.