When I speak, I don't use notes. This probably because all the early speakers and preachers I admired didn't use notes. They didn't read. When you don't read, as Seth Godin points out, your speech becomes better. It's more natural. Audiences connect with people that appear natural instead of stiff. We all like conversations more than we like lectures.
I'm lucky because I have a great memory. I can glance at something for a while and recall most of it instantly. This was great when cramming in college.
Write it down. Taking notes is useful because you can look up information later. But it's also useful because you're involving another sense (touch) with the material at hand. Write out or type your speech (or at least the key points). This will increase the amount of time you're spending with your subject matter. You'll also be able to weed out the points that aren't interesting when you can look over the speech as a whole.
Get intimate. I'm keynoting a conference for small business owners this May in Florida. Even thought we're still 90 days out, I'm already thinking about the big concept I'd like to leave them with. I'm already making notes and lists of stories and analogies to use with them that are different than when I speak to nonprofits or colleges. Thinking about my talk (all or in part) for three months buries it deep in my brain so I can draw upon that content when on stage, sans paper.
Say it out loud. A lot. If you were to sneak into my house or hotel room the days before a speech, you'd see quite a rant. This is where and when I "perform." Especially for a new talk, I actually give the entire talk while pacing throughout my house or around my hotel room. I inflect where needed. I say it out loud. I time myself. I check my notes when I'm done to see if I forgot something important in this latest rendition. This way, when I step on stage, it's like I've already given the talk a dozen times.
The key to speaking without notes is to be very familiar with the topic at hand. If you're not speaking about something you spend hours a week studying, then you'll need to find ways to increase exposure to your content so that it easily flows from your lips and is as natural as talking about your kids, the weather, or your hobbies.
Again, I can't recommend Moonwalking With Einstein by Joshua Foer enough. He gets very scientific (and humorous) about the science of memory. In short, the more we can associate information with places, we're more likely we'll be able to recall it. This is why I write stuff down (so I can remember physically where I saw that point on paper) and walk through my house when talking (a key point I wanted to make might be associated with my guest bedroom).
Memorizing can be easy. This will make all of your speeches great.