Where to start for a Topics Class
One concept/approach, which is what I try to keep in mind in addressing a heterogenous audience in a not-required more-advanced course, is exactly to not try to be "systematic", much less "complete". As noted in the question, it would be easy to use all the available time for "general theory", without ever getting around to the points of interest that motivated the course in the first place. So, instead of "laying a systematic foundation...", I try to get to interesting phenomena as directly as possible, and then back-fill "general theory" as relevant. For that matter, "general theory" is more widely available in textbooks and on-line notes than are interesting examples (since the latter tend to be written in "research papers" aiming at more-expert audiences), so students can more easily do further back-fill themselves if desired.

For that matter, some "general theory" is somehow over-hyped as a big deal, when, in fact, the introductory parts of it are really very elementary. For example, in an introduction to modular/automorphic forms (my interest...), one could position the thing as having prerequisites of "Lie theory", "algebraic number theory", "representation theory", and/or "algebraic geometry". It is certainly true that all these things are relevant and useful, but (I claim) it is not the case that one must have "systematically" studied all these things prior... For example, "Lie theory" at the entry level can be simply a study of behavior of two-by-two real and complex matrices. "Algebraic number theory" at a basic level is just about integers, or perhaps Gaussian integers, and so on. "Representation theory" (infinite-dimensional unitaries, etc.) relevant to automorphic forms will not even be found in "basic" representation theory books (which tend to do finite-dimensional and more combinatorial things), but can be approached in a down-to-earth way (for real rank-one groups) as Bargmann and Wigner did, by looking at asymptotics of solutions of second-order ordinary differential equations. (Indeed, the later work of HarishChandra and Casselman-Milicic might be construed as adapting those classical asymptotic techniques via Deligne's PDE version of the old ODE asymptotic results, as in the appendix to the C-M paper about it.) 

In particular, to my perception, far too often students think that they have some moral-professional obligation to be "completely expert-prepared" in the alleged background material before starting the next stage... which is (to my perception) substantially misguided, since that way one has no inkling of what the use of the "background" might be. Trying to short-circuit this impulse a bit by directly drawing attention and explicating the applications is a good anti-dote.

And, to repeat, one has no obligation to "prove everything in class"... That's a recipe for not getting anywhere.