what is this current source
When we say an ideal current source has infinite resistance, we are not talking about series resistance, we are talking about shunting resistance. The model of a non-ideal current source looks like this:







simulate this circuit &ndash; Schematic created using CircuitLab

In this diagram I labelled the resistor as "G" instead of "R" to emphasize that this is the dual of a voltage source, so it has a shunt conductor instead of a series resistor.

When we say an ideal current source has infinite resistance, we really should say it has 0 conductance, the dual of the ideal voltage source having 0 resistance.

If we add a load resistor we have this circuit:







simulate this circuit

And indeed, even if G is 0, there is a complete circuit. Current flows through the ideal current source, through the load, and back through the connection to the other terminal of the source.

If you are concerned that current can't flow through the current source because of Ohm's law, remember that Ohm's law isn't a law for all circuit elements, it's just a description of one type of element: the resistor. Current can also flow through other elements that have nothing to do with Ohm's law like capacitors, inductors, and sources.

Edit

This is to explain what I mean about circuit duality and why I bring it up.

First, in circuit theory neither the voltage source nor the current source is more fundamental than the other. Understanding both is necessary in order to model all circuits. For example, when doing transient analysis of circuits containing capacitors and inductors, the model of the capacitor at each timestep contains an ideal voltage source and the model of an inductor at each timestep contains an ideal current source. Even among practical circuits there are sources that are more straightforward to model as current sources than voltage sources: current mirrors and certain switching power supply configurations come to mind. We also use current sources to model the collector-emitter branch of a BJT in many cases.

Now, how does duality apply to this question?

You are probably already familiar with the voltage source, which looks like this:



And this model leads you to ask, if an ideal current source has infinite resistance, how can any current at all flow through it?

What you are missing is that when you talk about current sources, you are talking about the dual of the voltage source. And when you convert to the dual, you should convert series resistance to shunt conductance. This gives the current source model I showed above:



Notice that I didn't just replace the resistor with a conductor. I changed the circuit topology from series to parallel. 

As you can see, in this dual device, when G goes to 0 (when the internal resistance is infinite) it does nothing at all to prevent current flowing through the source. It simply allows the ideal current source to deliver all of its value to the load, rather than having some shunted by its non-ideality element.  

Just like how, by duality, in the voltage source when the non-ideality element (resistor) goes to 0, it allows all of the voltage to be delivered to the load.