the way to do the cross validation
It depends what is your end goal. K-Fold CV is used for finding model hyperparameters.
After this phase you may change your validation dataset and train your model with it.
If you want to harness as much data as you can (performing predictions) it might be a good idea to train N models on N different folds and ensemble their predictions. This one is similar to boostrap, all in all your ensemble saw all the data yet it didn't overfit. This approach is N times more computationally intensive though, so it still comes down to your goals.
Finally, you should get better results with fitting different models to your folds instead of a single one, but this would require separate hyperparameter space for each algorithm.
Let say I have fold1, fold2 , fold3.
I trained fold1,fold2,fold3 with  modelA. 


A) modelA(fold1) -> modelA(fold2) -> modelA(fold3) 
B) modelA(fold1) -> saved weight modelA(fold1) -> modelA(fold2)->
  saved weight modelA(fold2) -> modelA(fold3)-> saved weight
  modelA(fold3) -> ensemble 3 weight 

which way is the right way to do the k-fold cross validation and why?