Is it ethical for a journal to request substantial payment for publishing a paper?
Well someone's got to pay. If it's the reader, then howls go up about Open Access. So that leaves the authors.

The costs of publishing

A high-quality journal needs high-quality editors.  An awful lot of literature still gets printed and distributed around the world and that's expensive. Remember, reading this, we are the ones who are most active online, and so our paper-reading habits are much more likely to be skewed towards predominantly (or even exclusively) online access, so are not fully representative. High-quality journals often also get involved in conference sponsorship, publicity, and so on. The peer-reviews get co-ordinated; there is often some vetting of suitable reviewers; their responses need to be interpreted correctly; special issues get co-ordinated; someone ensures a good balance of articles covering the journal's remit.

So, there's a lot of costs, and someone's got to cover them.  If we want to publish at almost-zero overheads, we write a blog.  A journal is much much more than a blog - and some of the things that make it much more than a blog, cost money.

Not necessarily an efficient market

That's not to say that in certain sectors, there isn't a market failure: some sectors do have suspiciously high profit margins. And market regulators should be looking at removing barriers to entry.  Those barriers certainly aren't insurmountable, as notable new scientific publishers have emerged recently, with new business models: PLOS being an obvious example, founded in 2000 to support Open Access, and becoming a publisher in its own right in 2003; and now a serious player, using the author-pays model.

Author beware

There are also plenty of scammers using the author-pays model. So be careful out there. Talk to colleagues about who's reputable and who's not. Read the journals. And there are lists online of the disreputable publishers. Read more about that on this site at How do you judge the quality of a journal?