Where did the continents come from?
The answer is that we don't know. I could give you a list of all the considerations that go into evaluating that question, but it will probably be incomplete and outdated very soon. There were several important papers published in the recent years, including some very readable reviews that you might be interested in (1 2 3). But, there isn't a consensus. A friend just had a paper accepted in which she shows Yet-Another-Breakthrough™ in this field.

However, the general idea is that most of the continental lithosphere formed early on (Hadean to Archean) and since then the rate has slowed down.

Continental crust forms, as you said, by formation of "lighter stuff". Melting ultramafic mantle rocks leads to formation of basalt, and further melting of basalt leads to granites (and other related rocks) which are indeed lighter in colour, and also lighter in density so they end up at the top, and then "float" as the continental lithosphere on top of the mantle. The problem is that you can also remelt over and over again the continental lithosphere, recycling the already buoyant rocks and not increasing the net volume of the continents. It's this balance which is unknown. How much lithosphere is "juvenile" (i.e. formed by melting of mantle) and how much is recycled.

To make matters worse, continental lithosphere can also be destroyed. The most obvious case is erosion from the continents which then settles as a thin sedimentary cover on the oceanic crust and then lost to the mantle during subduction. Another process is delamination. Some rocks such as basalt are normally less dense than the mantle rocks. However, when metamorphosed at high pressure to become eclogite, they paradoxically become denser than the mantle, leading to the bottom part of the lithosphere to be "ripped off" and sink down into the mantle, thus destroying the continental lithosphere from the bottom. Here's a nice visualisation of this process:



Source for the image.