Specify which CPU threads are used using cpuminer
Don't worry about it, just let your OS take care of the scheduling. As long as you're not trying to run more CPU-bound threads than you have cores, you'll be running as efficiently as you can. Even a single threaded program will bounce around between cores (you can see this by looking at a CPU usage graph, and your single threaded program will probably use 25% of each of a quad core system on average).