Pytorch C++ RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select
I am calculating word similarity using torch::Embedding module by pretrained wordvector (glove.300d) on Ubuntu 18.04LTS PyTorch C++ (1.5.1, CUDA 10.1). I believe I have moved everything I can to the GPU, but when I execute it, it still says (full error log on the end of the question):
Expected object of device type cuda but got device type cpu for
  argument #1 'self' in call to _th_index_select
  (checked_dense_tensor_unwrap at /pytorch/aten/src/ATen/Utils.h:72)

I have checked my model initialization method in main.cpp, and it is okay if I only do initialization.
SimilarityModel simiModel(args, 400000, 300);
simiModel.to(device);

//model forward
torch::Tensor data = ids.index({Slice(i*batch_size, (i+1)*batch_size), Slice()}).to(torch::kInt64).to(device);        //take a batch
tie(score, indice) = simiModel.forward(data);   //forward and transfer score, indice to cpu for further calculation

and this is how I define SimilarityModel in Similarity.h:
class SimilarityModel : public torch::nn::Module {
    public:
        int64_t topk;       // num of top words;
        Dictionary dict;
        int64_t vocab_size;
        int64_t embedding_dim;
        torch::nn::Embedding embedding{nullptr};
        vector&lt;vector&lt;float&gt; &gt; vec_embed;

        SimilarityModel(unordered_map&lt;string, string&gt; args, int64_t vocab_size, int64_t embed_dim);
        tuple&lt;torch::Tensor, torch::Tensor&gt; forward(torch::Tensor x);
};

At the same time I have done embedding initialization in SimilarityModel function in Similarity.cpp:
SimilarityModel::SimilarityModel(unordered_map&lt;string, string&gt; args, int64_t vocab_size, int64_t embed_dim)
        :embedding(vocab_size, embed_dim) {      //Embedding initialize
    
    this-&gt;topk = stoi(args[&quot;topk&quot;]);
    vector&lt;vector&lt;float&gt; &gt; pre_embed;
    tie(pre_embed, dict) = loadwordvec(args);       //load pretrained wordvec from txt file

    this-&gt;vocab_size = int64_t(dict.size());
    this-&gt;embedding_dim = int64_t(pre_embed[0].size());
    this-&gt;vec_embed = pre_embed;
    this-&gt;dict = dict;

    vector&lt;float&gt; temp_embed;
    for(const auto&amp; i : pre_embed)      //faltten to 1-d
        for(const auto&amp; j : i)
            temp_embed.push_back(j);
    torch::Tensor data = torch::from_blob(temp_embed.data(), {this-&gt;vocab_size, this-&gt;embedding_dim}, torch::TensorOptions().dtype(torch::kFloat32)).clone();   //vector to tensor    
    register_module(&quot;embedding&quot;, embedding);      
    this-&gt;embedding = embedding.from_pretrained(data, torch::nn::EmbeddingFromPretrainedOptions().freeze(true));
}

and forward function in Similarity.cpp:
tuple&lt;torch::Tensor, torch::Tensor&gt; SimilarityModel::forward(torch::Tensor x) {     

    auto cuda_available = torch::cuda::is_available();      //copy to gpu
    torch::Device device(cuda_available ? torch::kCUDA : torch::kCPU);
    
    torch::Tensor wordvec;
    wordvec = this-&gt;embedding-&gt;forward(x).to(device);      //python:embedding(x)
    torch::Tensor similarity_score = wordvec.matmul(this-&gt;embedding-&gt;weight.transpose(0, 1)).to(device);
    torch::Tensor score, indice;
    tie(score, indice) = similarity_score.topk(this-&gt;topk, -1, true, true);        //Tensor.topk(int64_t k, int64_t dim, bool largest = true, bool sorted = true)

    score = score.to(device);
    indice = indice.to(device);
    score.slice(1, 1, score.size(1));       //Tensor.slice(int64_t dim, int64_t start, int64_t end, int64_t step)
    indice.slice(1, 1, indice.size(1));
    return {score.cpu(), indice.cpu()};   //transfer to cpu for further calculation
}

As for intermediate variables in forward() have been put to GPU too. However, I totally have no idea which one is left in CPU, and the error log does not help so much. I have tried the method in Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select to do SimilarityModel().to(device), but that does not work. I am still having a hard time reading this error log and would like some instructions on how to debug such questions.
Error log:
terminate called after throwing an instance of 'c10::Error'
  what():  Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select (checked_dense_tensor_unwrap at /pytorch/aten/src/ATen/Utils.h:72)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x46 (0x7fb566a27536 in /home/switchsyj/Downloads/libtorch/lib/libc10.so)
frame #1: &lt;unknown function&gt; + 0x101a80b (0x7fb520fa380b in /home/switchsyj/Downloads/libtorch/lib/libtorch_cuda.so)
frame #2: &lt;unknown function&gt; + 0x105009c (0x7fb520fd909c in /home/switchsyj/Downloads/libtorch/lib/libtorch_cuda.so)
frame #3: &lt;unknown function&gt; + 0xf9d76b (0x7fb520f2676b in /home/switchsyj/Downloads/libtorch/lib/libtorch_cuda.so)
frame #4: &lt;unknown function&gt; + 0x10c44e3 (0x7fb558d224e3 in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #5: at::native::embedding(at::Tensor const&amp;, at::Tensor const&amp;, long, bool, bool) + 0x2e2 (0x7fb558870712 in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #6: &lt;unknown function&gt; + 0x114ef9d (0x7fb558dacf9d in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #7: &lt;unknown function&gt; + 0x1187b4d (0x7fb558de5b4d in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #8: &lt;unknown function&gt; + 0x2bfe42f (0x7fb55a85c42f in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #9: &lt;unknown function&gt; + 0x1187b4d (0x7fb558de5b4d in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #10: &lt;unknown function&gt; + 0x32b63a9 (0x7fb55af143a9 in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #11: torch::nn::EmbeddingImpl::forward(at::Tensor const&amp;) + 0x71 (0x7fb55af127b1 in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #12: SimilarityModel::forward(at::Tensor) + 0xa9 (0x55c96b8e5793 in ./demo)
frame #13: main + 0xaba (0x55c96b8bfe5c in ./demo)
frame #14: __libc_start_main + 0xe7 (0x7fb51edf5b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #15: _start + 0x2a (0x55c96b8bd74a in ./demo)

Aborted (core dumped)


Based on the error message, one of the two following Tensors are not in the GPU when you're running SimilarityModel::forward():

this-&gt;embedding-&gt;weight
x

Given that the error points to the argument #1, I'd say that weight is the one on the CPU.
Here's the call for index.select:
Tensor embedding(const Tensor &amp; weight, const Tensor &amp; indices,
                 int64_t padding_idx, bool scale_grad_by_freq, bool sparse) {
  auto indices_arg = TensorArg(indices, &quot;indices&quot;, 1);
  checkScalarType(&quot;embedding&quot;, indices_arg, kLong);

  // TODO: use tensor.index() after improving perf
  if (indices.dim() == 1) {
    return weight.index_select(0, indices);
  }

  auto size = indices.sizes().vec();
  for (auto d : weight.sizes().slice(1)) {
    size.push_back(d);
  }
  return weight.index_select(0, indices.reshape(-1)).view(size);
}

First, try to directly move the weight to the GPU. If it works, it means that when you called TORCH_MODULE(SimilarityModel), and moved the model to the device, it should have worked too. Remember that you have to change the name to SimilarityModelImpl (Name+Impl) in this case. Otherwise, it won't work as well.
