In python3: strange behaviour of list(iterables)
I have a specific question regarding the behaviour of iterables in python. My iterable is a custom built Dataset class in pytorch:

import torch
from torch.utils.data import Dataset
class datasetTest(Dataset):
    def __init__(self, X):
        self.X = X

    def __len__(self):
        return len(self.X)

    def __getitem__(self, x):
        print('***********')
        print('getitem x = ', x)
        print('###########')
        y = self.X[x]
        print('getitem y = ', y)
        return y


The weird behaviour now comes about when I initialize a specific instance of that datasetTest class. Depending on what data structure I pass as an argument X, it behaves differently when I call list(datasetTestInstance). In particular, when passing a torch.tensor as argument there is no problem, however when passing a dict as argument it will throw a KeyError. The reason for this is that list(iterable) not just calls i=0, ..., len(iterable)-1, but it calls i=0, ..., len(iterable). That is, it will iterate until (inclusive) the index equal to the length of the iterable. Obviously, this index is not definied in any python datastructure, as the last element has always the index len(datastructure)-1 and not len(datastructure). If X is a torch.tensor or a list, no error will be risen, even though I think the should be an error. It will still call getitem even for the (non-existent) element with index len(datasetTestinstance) but it will not compute y=self.X[len(datasetTestInstance]. Does anyone know if pytorch handels this somehow gracefully internally?

When passing a dict as data it will throw an error in the last iteration, when x=len(datasetTestInstance). This is actually the expected behaviour I guess. But why does this only happen for a dict and not for a list or torch.tensor?

if __name__ == "__main__":
    a = datasetTest(torch.randn(5,2))
    print(len(a))
    print('++++++++++++')
    for i in range(len(a)):
        print(i)
        print(a[i])
    print('++++++++++++')
    print(list(a))

    print('++++++++++++')
    b = datasetTest({0: 12, 1:35, 2:99, 3:27, 4:33})
    print(len(b))
    print('++++++++++++')
    for i in range(len(b)):
        print(i)
        print(b[i])
    print('++++++++++++')
    print(list(b))


You could try out that snippet of code if you want to understand better what I have observed.

My questions are:

1.) Why does list(iterable) iterate until (including) the len(iterable)? A for loop doesnt do that.

2.) In case of a torch.tensor or a list passed as data X: Why does it not throw an error even when calling the getitem method for the index len(datasetTestInstance) which should actually be out of range since it is not defined as an index in the tensor/list? Or, in other words, when having reached the index len(datasetTestInstance) and then going into the getitem method, what happens exactly? It obviously doesnt make the call 'y = self.X[x]' anymore (otherwiese there would be an IndexError) but it DOES enter the getitem method which I can see as it prints the index x from within the getitem method. So what happens in that method? And why does it behave different depending on whether having a torch.tensor/list or a dict?

This isn't really a pytorch specific issue, it's a general python question.

You're constructing a list using list(iterable) where an iterable class is one which implements sequence semantics.

Take a look here at the expected behavior of __getitem__ for sequence types (most relevant parts are in bold)


  object.__getitem__(self, key)
  
  Called to implement evaluation of
  self[key]. For sequence types, the accepted keys should be integers
  and slice objects. Note that the special interpretation of negative
  indexes (if the class wishes to emulate a sequence type) is up to the
  __getitem__() method. If key is of an inappropriate type, TypeError may be raised; if of a value outside the set of indexes for the
  sequence (after any special interpretation of negative values),
  IndexError should be raised. For mapping types, if key is missing (not
  in the container), KeyError should be raised.
  
  Note: for loops expect that an IndexError will be raised for illegal
  indexes to allow proper detection of the end of the sequence.


The problem here is that for sequence types python expects an IndexError in the case where __getitem__ is invoked with an invalid index. It appears the list constructor relies on this behavior. In your example when X is a dict, attempting to access an invalid key causes __getitem__ to raise KeyError instead which isn't expected so isn't caught and causes the construction of the list to fail.



Based on this information you could do something like the following

class datasetTest:
    def __init__(self):
        self.X = {0: 12, 1:35, 2:99, 3:27, 4:33}

    def __len__(self):
        return len(self.X)

    def __getitem__(self, index):
        if index &lt; 0 or index &gt;= len(self):
            raise IndexError
        return self.X[index]

d = datasetTest()
print(list(d))


I can't recommend doing this in practice since it relies on your dictionary X containing only the integer keys 0, 1, ..., len(X)-1 which means it end up behaving just like a list for most cases, so you're probably better off just using a list.
