Which is better for PhD application: research-based BS thesis somewhat related to interests, or extra courses + standard thesis in area of interest?
If your plans to pursue a PhD eventually are serious, I would definitely recommend the research-based option. You'll see this advice in many other places, but acquiring a PhD is very much the process of learning how to do research and proving that you can indeed do it. Even if the project is not directly in an area that you're very interested in, you'll have the opportunity to start practicing your research skills. The prospect of a publication is also something to keep in mind; an entering PhD student who has already published is viewed as already having proved to some degree that they're capable of quality research, and I've never met a faculty member anywhere who didn't appreciate that.

You're exactly right in that the research experience will be valuable to you even if you do not end up with a publication. Why? Primarily, if you do good work you've given your faculty collaborators some very good material to put in the reference letters you're going to want them to write when you apply to PhD programs. In fact, in your position I'd probably be thinking more about coming out of the experience with a strong reference than a publication.