How can I listen to a voice memo in background like iPod and open some other note taking app on iPhone 4?
Since Voice Memos both plays and records audio, and recording can't be done in the background as far as I know, then the whole app does not work in the background. That sounds more like a design decision to me, rather than a technical limitation. In theory, Apple could have made the playing audio part of Voice Memos to work in the background. 

A very long-winded way to get to do what you need is to import the voice memos to your iTunes library. In iTunes, select your iPhone from the Devices list. One of the tabs (probably Music or Info) has a checkbox that will enable transferring voice memos into iTunes. When this is enabled and you sync your device, the voice memos get imported and a playlist titled "Voice Memos" is created. 

At this point, those audio files and the "Voice Memos" playlist should act like any other. You could, for instance, get to the audio files from the "Voice Memos" playlist and then set the metadata for each file by right-clicking it and selecting the Get Info dialog. Useful voice memos can be set to a Media Type of Audiobook--which makes them available to you as audiobooks in the iPod app.

If you make them Audiobooks or basically do anything that sends them to the iPod app; then you could play them in the background while you open your note-taking app of choice. 

You do not need to set them as audiobooks, though. Once they're in the voice memos playlist; then  you can sync the playlist as you would any other. The caveat is that without metadata, it becomes hard to find the files in the iPod app. In iTunes, for instance, they show either with a blank artist name or with an artist name of "Untitled". That is why I strongly suggest adding metadata ASAP.

I am just reminded of Dragon Naturally Speaking and Dragon Dictate. I do not have them, but I have heard nothing but great things about them. You may try their apps (one of them is free, I believe). They may offer you better features over the included Voice Memos app. I've seen screenshots where it looks like in addition to having a record option, the UI also has a text field for you to type on. 

Hope this helps.