LM2678 Output Voltage Drop Under Load
At the higher current point use a scope to check if the circuit is oscillating or ringing on the input or output pins.  

Be sure the feedback line is far enough away from the toroid so as not to pick up noise from it, (per the data sheet). While a good commercially made toroid should not radiate much magnetic noise a hand made part may not be so perfect.