Python Numpy - Matrix memory error and limits
Question: after I have been searching this topic on the web, I saw that I was not the
only one to have this problem but I can't understand if there is a way to
overcome the problem or not.

I have 5036 text files and a word list of 15985 words. For each word of the
word list found on the text file, I would like to put a 1 on my matrix. But I
get the error : _MemoryError_.

I tried also just to create the matrix and make a print (in case of a bug in
my python code).. I got the same error. Any suggestions?

    
    
    matrix = np.zeros(shape=(5036,15985))
    

Edit: That's my code maybe there are some errors.. It should work in this way:

  1. create the dictionary from a file (each word on the text file is names as "1_word1 2_word2 " etc, so splitting each line of the text file I will have in split_[0] the position in matrix, in split_[1] the word itself)
  2. for each text file it saves the number of the file in order to put the right document on the matrix (each text file is named “1_1A_out.txt 2_1A_out.txt etc”)
  3. finally it prints the matrix.
    
        import os
    import re
    import fileinput
    import numpy as np
    
    matrix = np.zeros(shape=(6000,16000))
    
    def dictionary_creation (filepath):
        fileopen = open(filepath, "r")
        dictionary = fileopen.read().split('\n')
        fileopen.close()
        return dictionary
    
    
    def find_doc_matrix_position (filename):
        regex = re.compile('(\d)_(.*)')
        find_regex = regex.search(file)
        if find_regex:
            pos_doc = int(find_regex.group(1))-1
        return pos_doc
    
    def put_nbdoc_nbword_in_matrix (filename, dictionary, nb_file):
        for line in fileinput.input([filename]):
                line = line.replace("\n", "")
                for w in range(0,len(dictionary)-1):
                    split_ = dictionary[w].split('_',1)              
                    if line == split_[1]:
    #                    print ("nb_file is: "+str(nb_file))                    
    #                    print ("nb_word is : "+str(split_[0]))
    #                    print ("line is: "+line+" word is: "+split_[1])
    #                    print '####'
                        matrix[nb_file,split_[0]] = 1
    
    dictionary = dictionary_creation('C:\\Users\\KP\\Desktop\\FSC_lemmes_sort.txt')
    
    for file in os.listdir('C:\Users\KP\Desktop\FSC_Treetag\out'):
            fin = open(file, 'r')
            filename = file
            nb_file = find_doc_matrix_position(file)
            put_nbdoc_nbword_in_matrix(filename, dictionary, nb_file)
    
    print "this is the final matrix\n"
    print matrix
    


Answer: You are getting a memory error probably because your matrix is too big, or you
don't have enough available RAM memory in your computer.

You can try to iterate across all your text files, so you just need to create
a matrix like:

    
    
    matrix = np.zeros(shape=(15985))
    

and then save the result to a file for every of your texts.

