Troubleshooting Blender's Memory Use (Memory Statistics)
To fully investigate the problem you may need debugging tools (debug builds etc), but as a user, here are some things you can do.

The first thing to check on is Blenders memory statistics in the top header and compare this to what the operating system reports Blender is using, to see if the memory is being tracked by Blender or not.

If Blender's header doesn't show a large difference in memory then its possible there is a memory leak, or more likely - one of the 3rd party libraries Blender uses is holding onto memory.

If Blender also reports a jump in memory then you can use an operator which reports details about memory use.

Reporting Memory Use

Start Blender in a terminal: See manual for instructions.

Before rendering run the operator "Memory Statistics" from the space-bar search menu. Copy and paste this text into a file.

Render the scene and re-run "Memory Statistics" copying the output into a different file.

The text will look like this:

total memory len: 9.319 MB
peak memory len: 18.868 MB
slop memory len: 5.043 MB
 ITEMS TOTAL-MiB AVERAGE-KiB TYPE
  1993 (   3.051     1.568) BLI_Mempool Chunk
   210 (   0.972     4.740) Data from ME
  2544 (   0.612     0.246) PropertyRNA
    20 (   0.584    29.922) Chunk buffer
  2296 (   0.438     0.195) StructRNA
  3025 (   0.369     0.125) IDProperty array dup
  2209 (   0.322     0.149) dupli_alloc
  1723 (   0.302     0.180) keymap entry




--- snip ---

Now you can compare the results, it may help to use a visual diff tool. search online - many free applications exist.

There may be a perfectly logical reason that Blender is using more memory but if not, then best report a bug on the tracker so a developer can investigate,