Powering a small DC motor off of 18650s
i belive that you are using something like this diagram in your wiring ,

since you`re using 2 channel relays ,you have 4 states for the relays,
1- ch1 on ch 2 on =&gt; this will connect both terminals of motor to the dc which will result in hard breaking,
2- ch1 on ch2 off =&gt; this will run the motor clockwise/clock side based on your connection
3- ch1 is off ch2 is on =&gt; this will reverse the motor direction,
4- ch1 is off ch2 is off =&gt; this will turn off motor but it will not break, its like Neutral [N] gear in the car,

to move from state 2 to state 3 you should go through either state 1 or 4 , as flipping the status of relays is not instant, a voltage (Ve) will be genetrated in motor also due to mechanical movement, this will be addup voltage to the source (Vs) and so on it will be Vs+Ve and a spike will activate the protection in the MT3608 converter.
if your replaced the relays with a transistors (4 of them, 2 n type,2 p-type) you should make sure to let time gap between on and of state for each side so that the transistor leaking current get dumped, else as @transistor said , it will short the system,