Reading h5py files into tensors
So I have a training set and a test set both in h5py format. I also have a data_load function that loads the files and returns NumPy arrays. The main problem is I don't need NumPy as I am working with Tensors. I am expecting to have an x&amp;y tensor of size N(batch size) and D_in(input size for each image) and D_out(Output size of each tensor).

The problem:

x&amp;y do not get converted to tensors of dimensions mentioned below.If anything their types remain to be numpy.ndarray. Any help is appreciated.

def load_data(train_file, test_file):
# Load the training data

train_dataset =h5py.File(train_file, 'r')


# Separate features(x) and labels(y) for training set
train_set_x_orig =np.array(train_dataset["train_set_x"][:])
train_set_y_orig =np.array(train_dataset["train_set_y"][:])

# Load the test data
test_dataset =h5py.File(test_file,'r') 

# Separate features(x) and labels(y) for training set
test_set_x_orig =np.array(test_dataset["test_set_x"][:]) 
test_set_y_orig =np.array(test_dataset["test_set_y"][:])

classes = np.array(test_dataset["list_classes"][:]) # the list of classes

train_set_y_orig = torch.from_numpy(train_set_y_orig.reshape((1, train_set_y_orig.shape[0])))
test_set_y_orig = torch.from_numpy(test_set_y_orig.reshape((1, test_set_y_orig.shape[0])))

return train_set_x_orig, train_set_y_orig, test_set_x_orig, test_set_y_orig, classes






x = torch.Tensor(N, D_in)
y = torch.Tensor(N, D_out)
train_file="data/train_catvnoncat.h5"
test_file="data/test_catvnoncat.h5"
x,y,_,_,_=load_data(train_file,test_file)


Because you did not convert train_set_x_orig to a torch tensor before returning.
Either use torch.from_numpy() on train_set_x_orig before returning as you do with train_set_y_orig or cast it to a tensor before assigning to x.

However, y should be of type torch.tensor.



Below is a demonstration that explains the issue:

# some sample tensor
In [27]: x = torch.Tensor(3, 2)    

# check its type
In [28]: type(x)   
Out[28]: torch.Tensor

# some sample ndarray
In [29]: arrx = np.arange(6).reshape(3, -1)   

# assign array to tensor
# note that now the object `x` refers to the numpy array object
In [30]: x = arrx     

# see that the type() of `x` is now numpy ndarray
In [31]: type(x)  
Out[31]: numpy.ndarray




Also, as hpaulj pointed out in the comments, there is no need to wrap the sliced objects from h5py in np.array() since the sliced objects are already of type numpy ndarrays. So, you can just get rid of them and the code will look more cleaner!
