Reduce File Size of Dxtory Video Without Causing Lag
The problem with compression is that it is a double edged sword. It can reduce the space required to store video, however it also takes processing time to perform the compression.

When you are running a game, however, the CPU and GPU are already busy keeping the game running smoothly. In order to shrink the file size, you must either reduce the quality or increase the compression. If you increase the compression, the computer has to take more away from the game and you start seeing lag.

The only ways around this are to:


a) reduce the resolution of the video you are recording.  Throwing
away information is free and will also make applying further
compression less processor intensive.  
b) move the encoding to another system.  Some recorders can send the data to another computer and have that computer perform the encoding.  I am not sure if Dxtory supports this or not though.
c) Continue to use the uncompressed files and transcode the video after you record a clip. 


Of these options, c is probably the easiest way to maintain maximum quality.  You can use a program like Handbrake to take the large file and compress it down to something of reasonable size, just so long as you don't have to record more than you have room for at a given time.  Note that this process can take quite some time depending on the quality level you choose and how fast your computer is.