Playing samples in a loop, without clicks or pops I'm trying to build an audio player that has this specific feature: when you scroll to x sample time, [x to x + 4096] samples are played in a loop until you move the mouse to another place.

The samples are taken from the backing audio. The problem is, I fail to play them back as a continuous note: the looping is not seamless.

So far I tried to start and end the loop at zero crossings. This slightly improved it, but I can still hear the seamless looping. 

One idea I had was to go to the frequency domain, remove all, but the max magnitude bin and do an inverse FFT afterwards. This, sort of works, the looping is better, not perfect, but better, however, removing all the other freq components from the loop leaves it really "dry", not resembling the original (missing the harmonics)

Is there a way to handle this in the time domain? Where should I look? Are there any filters/algorithms that could help me?
The popping comes from discontinuities in the signal or its derivative.  Even if you match the starting sample value with the ending one, you will still have a "corner" in most cases.  Imagine a pure sine wave of one frequency at the start and a pure sine wave of another at the end. Discontinuities tend to produce a lot of high frequency noise.  Your second solution using the low pass filter eliminated that noise (it smoothed out the signal), but at the cost of eliminating the signal too.

The simplest solution to this is just to crossfade the end into the start.  Admittedly, at typical playback rates (22KHz) you are looping about 200ms which gives very little time to fade in/out.  At this point, you may be closer to doing an (overlapped) window function.  Doing an abrupt crossfade would be roughly akin to a Tukey window.