Monitoring my program's Virtual Bytes usage while it is running showed that by doing some kind operations, the virtual bytes usage goes up by about 1GB in about 5 minutes.
The program deals with tcp sockets and high data transfer throughput between them (~800Mbps).

Loading a dump file of the program in windbg showed that the reason for the very high and fast memory usage is about 1GB of "free" objects.
Indeed, when I call the garbage collector (gen 0, 1 & 2) from the console screen of the program (after getting it to this state) it frees up about 1GB of memory usage.

I'm trying to understand what exactly are these free objects and why aren't they garbage collected by the garbage collector automatically.

**Edit:** One suggestion was that I may be creating the objects in the Large Object Heap and it becomes fragmanted but this is not the case as I've seen that all of the "free" objects sits in Gen 2 Heap.

Other suggestion was that maybe Gen 2 Heap gets fragmented because of pinned objects but if this was the case, GC.Collect wouldn't fix the problem but it actually does so I believe this is not the case as well.

What I suspect from the discussion with Paul is that the memory does gets freed but is from some reason returned to the OS rarely or only when I manually call GC.Collect.