Python recursive function failing
The issue that I am having is a really strange issue. 

What I am trying to accomplish is the following: I am training a neural network using pytorch, and I want to restart my training function if the training loss doesn't decrease, so as to re-initialize the neural network with a different set of weights. The training function is presented below:

def __train__(dp, i, j, net, restarts, epoch=0):
    if net == '2CH': model = TwoChannelCNN().cuda()
    elif net == 'Siam' : model = SiameseCNN().cuda()
    elif net == 'Trad' : model = TraditionalCNN().cuda()
    ls_fn = torch.nn.MSELoss(reduce=True)
    optim = torch.optim.SGD(model.parameters(),  lr=1e-6, momentum=0.9)
    epochs = np.arange(100)
    eloss = []
    for epoch in epochs:
        model.train()
        train_loss = []
        tr_batches = np.array_split(dp.train_set, int(len(dp.train_set)/8))
        for tr_batch in tr_batches:
            if net == '2CH': loaded_batch = dp.__load2CH__(tr_batch)
            elif net == 'Siam': loaded_batch = dp.__loadSiam__(tr_batch)
            elif net == 'Trad' : loaded_batch = dp.__load__(tr_batch, i)
            for x_batch, y_batch in loaded_batch:
                x_var, y_var = Variable(x_batch.cuda()), Variable(y_batch.cuda())
                y_pred = torch.clamp(model(x_var), 0, 1)
                loss = ls_fn(y_pred, y_var)
                train_loss.append(abs(loss.item()))
                optim.zero_grad()
                loss.backward()
                optim.step()
        eloss.append(np.mean(train_loss))
        print(epoch, np.mean(train_loss))
        if epoch == 10 and np.mean(train_loss) &gt; 0.2:
            restarts += 1
            print('Number of restarts for client {} and fold {}: {}'.format(i,j,restarts))
            __train__(dp, i, j, net, restarts, epoch=0)

    __plotLoss__(epochs, eloss, 'train', str(i), str(j))
    torch.save(model.state_dict(), "Output/client_{}_fold_{}.pt".format(i, j))


So the restarting based on if epoch == 10 and np.mean(train_loss) &gt; 0.2: works, but only sometimes, which is beyond my comprehension. Here is an example of the output:

0 0.5000133737921715
1 0.4999906486272812
2 0.464298670232296
3 0.2727506290078163
4 0.2628978116512299
5 0.2588871221542358
6 0.25728522151708605
7 0.25630473804473874
8 0.2556223524808884
9 0.25522999209165576
10 0.25467908215522767
Number of restarts for client 5 and fold 1: 3
0 0.10957609283713009
1 0.02840371729924134
2 0.021477583368030594
3 0.017759160268232682
4 0.015173796122947827
5 0.013349939693290782
6 0.011949078906879265
7 0.010810676779671655
8 0.00987362345259362
9 0.009110640348696108
10 0.008239036202623808
11 0.007680381585537574
12 0.007171026876221333
13 0.006765962297888837
14 0.006428168776848068
15 0.006133011780953467
16 0.005819878347673745
17 0.005572605537395361
18 0.00535818950227004
19 0.005159409143814457
20 0.0049763926251294235
21 0.004738794513338235
22 0.004578812885309958
23 0.004428663117960554
24 0.004282198464788351
25 0.004145324644400691
26 0.004018862769889626
27 0.0039044404603504573
28 0.0037960831121495744
29 0.0036947361258523586
30 0.0035982220717533267
31 0.0035018146670104723
32 0.0034150678806059887
33 0.0033372560733512698
34 0.003261332974241583
35 0.00318166259540763
36 0.003108531899014735
37 0.0030385089141125848
38 0.002977990984523103
39 0.0029195284016142937
40 0.002870084639441188
41 0.0028180573325994373
42 0.0027717544270049643
43 0.002719321814503495
44 0.0026704726860933194
45 0.0026204266263459316
46 0.002570544072460258
47 0.0025225681523167224
48 0.0024814611543610746
49 0.0024358948737413116
50 0.002398673941639636
51 0.0023606415423654587
52 0.002330436484101057
53 0.0022891738560574027
54 0.002260655496376241
55 0.002227568955708719
56 0.002191826719741698
57 0.0021609061182290058
58 0.0021279943092100666
59 0.0020966088490456513
60 0.002066195117003474
61 0.0020381672924407895
62 0.002009863329306995
63 0.001986304977759602
64 0.0019564831849032487
65 0.0019351609173580756
66 0.0019077356409993626
67 0.0018875047204855945
68 0.0018617453310780547
69 0.001839518720600381
70 0.001815563331498197
71 0.0017149778925132932
72 0.0016894878409248121
73 0.0016652211918212743
74 0.0016422999463582074
75 0.0016183732903472788
76 0.0015962369183098418
77 0.0015757764620279887
78 0.0015542267022799728
79 0.0015323152910759318
80 0.0014337954093957706
81 0.001410489170542867
82 0.0013871921329466962
83 0.0013641994057461773
84 0.001345829172682187
85 0.001322142209181493
86 0.00130379223035348
87 0.001282231878045458
88 0.001263879886683956
89 0.001243419097817167
90 0.0012279346547037929
91 0.001206978429649382
92 0.0011871445969959496
93 0.001172510546330841
94 0.0011529557384797045
95 0.0011350733004023273
96 0.001118382818282214
97 0.001103347793609089
98 0.0010848538354748599
99 0.0010698940242660911
11 0.2542190085053444
12 0.2538975296020508


So here you can see that the restarting is correct from the 3rd restart, but then, since the network converges, the training should be complete, but the function restarts AGAIN after the 99th epoch (for an unknown reason), and somehow starts at the 11th epoch, which also makes no sense as I am explicitly specifying epoch = 0 whenever the function starts or restarts. I should also add that, SOMETIMES, the function completes correctly after the epoch 99, when convergence has been achieved, and does not restart. 

So my question is, why does this piece of code produce inconsistent results and outcomes? What am I missing here? Thanks in advance for any suggestions.

You are restarting the training by calling  __train__ a second time in the case if epoch == 10 and np.mean(train_loss) &gt; 0.2: but you never terminate the first loop.
So, after the second training has converged, the outer loop continues at epoch 11.

What you need is a break statement after the inner call to __train__.
