How to connect faces between 2 edge loops?
In your particular situation, it looks like the bridge tool is what you want. Select two or more edge loops with AltShiftRMB, and then connect them with W&gt;Bridge Edge Loops:

Fill
To do exactly what it looks like you are trying to do in your screenshot (connecting two vertices with an edge), select the two vertices and press F to create edge/face.

If you have the bundled F2 addon enabled you can create a new face from the selected edge and the surrounding vertices by pressing F again. The side on which the new face is created is controlled by the cursor position.
Knife
The knife tool is for cutting edges into existing faces: