A publisher is a person who won't take your phone calls. A literary agent is a person who can get a publisher on the phone. There's more of course, like negotiating, but access is the essential element. Instead of dealing with thousands of would-be authors, the publisher can limit contact to a handful of agents.