Getting ModuleNotFoundError while the module is next to the calling file
I am in a weird situation where in order for my package to be called just fine, one single file (models.py) needs to be next to the calling script and also inside the package folder where it is used.

To make it a bit more clear, this is how the package organization looks like : 

-FV_dir
 ---__init__.py
 ---F_V.py
 ---models.py
 ---utils.py
 ---service_utils.py
 ---subModule1_dir
    ----__init__.py
    ----detector.py
    ----utils.py
    ----subdir1
        --- etc
    ----subdir2
        --- etc
    ----subdir3


This whole package is placed in the site-packages, so it's usable system-wide. And there is a user script that uses this package like this: 
service_client.py:  

from FV.service_utils import ServiceCore
from FV.utils import a_helper_function

def run():
  service = ServiceCore()
  service.run()

if __name__ == "__main__":
   run()



The ServiceCore itself uses F_V.py which is the main module here. The F_V module itself uses the models.py and utils.py next to it like  this :

F_V.py :

from FV.utils import func1, func2  
from FV.models import model1, model2, model3
...


Now the problem is, if the models.py is not next to the client code (service_client.py) it just complains that the module is not found : here is an example error I get when this is the case: 

           └─19146 /home/user1/anaconda3/bin/python3 /home/user1/Documents/service_client.py

Mar 17 19:22:54 ubuntu python3[19146]:     self.fv = FaceVerification(**cfg['Face_Verification']['ARGS'])
Mar 17 19:22:54 ubuntu python3[19146]:   File "/home/user1/anaconda3/lib/python3.7/site-packages/FV/F_V.py", line 58, in __init__
Mar 17 19:22:54 ubuntu python3[19146]:     self._init_model()
Mar 17 19:22:54 ubuntu python3[19146]:   File "/home/user1/anaconda3/lib/python3.7/site-packages/FV/F_V.py", line 80, in _init_model
Mar 17 19:22:54 ubuntu python3[19146]:     checkpoint = torch.load(self.model_checkpoint_path, map_location=torch.device('cpu'))
Mar 17 19:22:54 ubuntu python3[19146]:   File "/home/user1/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 529, in 
Mar 17 19:22:54 ubuntu python3[19146]:     return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
Mar 17 19:22:54 ubuntu python3[19146]:   File "/home/user1/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 702, in 
Mar 17 19:22:54 ubuntu python3[19146]:     result = unpickler.load()
Mar 17 19:22:54 ubuntu python3[19146]: ModuleNotFoundError: No module named 'models'


If I remove the models.py next to the F_V.py obviously F_V.py would complain, as it's directly using it : 


           └─19216 /home/user1/anaconda3/bin/python3 /home/user1/Documents/fv_service_linux.py

Mar 17 19:27:33 ubuntu systemd[1532]: Started FV Service.
Mar 17 19:27:34 ubuntu python3[19216]: Traceback (most recent call last):
Mar 17 19:27:34 ubuntu python3[19216]:   File "/home/user1/Documents/fv_service_linux.py", line 83, in &lt;module&gt;
Mar 17 19:27:34 ubuntu python3[19216]:     from FV.service_utils import ServiceCore
Mar 17 19:27:34 ubuntu python3[19216]:   File "/home/user1/anaconda3/lib/python3.7/site-packages/FV/service_utils.py", line 21, in &lt;mod
Mar 17 19:27:34 ubuntu python3[19216]:     from FV.F_V import FaceVerification
Mar 17 19:27:34 ubuntu python3[19216]:   File "/home/user1/anaconda3/lib/python3.7/site-packages/FV/F_V.py", line 17, in &lt;module&gt;
Mar 17 19:27:34 ubuntu python3[19216]:     from FV.models import resnet18, resnet50, resnet101
Mar 17 19:27:34 ubuntu python3[19216]: ModuleNotFoundError: No module named 'FV.models'



So the only way to get this to work is to have models.py next to the client code as well. I can't understand why this is happening, as the client code doesn't even directly interacts with models.py. What am I missing here?  

Update:

In the F_V.py I use Pytorch and load a pretrained model. I thought this was unrelated and I was doing something wrong with regards to the packaging in Python, however, it turns out this was indeed the culprit. read the answer for more information.  

This actually happened to be a Pytorch related issue, in which this whole mess has been caused by this simple command when loading the model inside F_V.py : 

if self.device == 'cpu':
    checkpoint = torch.load(self.model_checkpoint_path, map_location=torch.device('cpu'))
else:
    checkpoint = torch.load(self.model_checkpoint_path)
# the culprit!
self.model = checkpoint['model'].module


This way of storing and loading the model is very bad, the pretrained models were initially wrapper around by a nn.DataParallel and the person who saved them, did it like this : 

def save_checkpoint(epoch, epochs_since_improvement, model, metric_fc, optimizer, acc, is_best):
    print('saving checkpoint ...')
    state = {'epoch': epoch,
             'epochs_since_improvement': epochs_since_improvement,
             'acc': acc,
             'model': model,
             'metric_fc': metric_fc,
             'optimizer': optimizer}
    # filename = 'checkpoint_' + str(epoch) + '_' + str(loss) + '.tar'
    filename = 'checkpoint.tar'
    torch.save(state, filename)
    # If this checkpoint is the best so far, store a copy so it doesn't get overwritten by a worse checkpoint
    if is_best:
        torch.save(state, 'BEST_checkpoint.tar')



As you can see, he used the whole model ('model': model) in the state_dict and save that away. He should have used the state_dict().
This is bad since the same files/dir structure hierarchy /everything must be retained all the time, everywhere that this model is going to be used.
and we were affected by this as well as you can see. All the client services
relied on models.py and needed it to be next to them while they didn't even use it.
Initially I thought therefore in order to solve the issue, we have to instantiate the models ourselves and then load the weights manually.     

if self.model_name == 'r18':
    self.model = resnet18(pretrained=False, use_se=use_se)
elif self.model_name == 'r50':
    self.model = resnet50(pretrained=False, use_se=use_se)
elif self.model_name == 'r101':
    self.model = resnet101(pretrained=False, use_se=use_se)
else:
    raise Exception(f"Model name: '{self.model_name}' is not recognized.")

# load the model weights
self.model.load_state_dict(checkpoint['model'].module.state_dict())


Note that since the model was initially a nn.DataParallel model, in order to access the model itself, we use the .module property and then use the models state_dict() to initialized the model and hopefully this solves the issue.
However, it seems this is not the case, and since the model is saved that way, it seems there is no way to get rid of such dependencies this way.
instead convert your model into torch script and then save the model.
This way you can get rid of all the nuisance . 

Solution 1:

Try converting your model into torch script and then use that instead:

def convert_model(model, input=torch.tensor(torch.rand(size=(1,3,112,112)))):
        model = torch.jit.trace(self.model, input)
        torch.jit.save(model,'/home/Rika/Documents/models/model.tjm')


and then loaded this version instead: 

# load the model 
self.model = torch.jit.load('/home/Rika/Documents/models/model.tjm')


Solution 2:

simply save the model's state_dict() again and use  that instead : 
I myself ended up doing : 

self.model = checkpoint['model'].module
# create the new checkpoint based on what you need 
torch.save({'state_dict' : self.model.state_dict(), 'use_se':True},
            '/home/Rika/Documents/BEST_checkpoint_r18_2.tar')

and started using the new checkpoint and so far everything has been good 

