   [java] *sigh* java.lang.OutOfMemoryError : Java heap space
  Hello everybody,this isnt really a game related question but since im lurking around these forums for some years now and its about java i thought my question might be at the right place here anyways.Here it comes.I need to readout a rather large txt file containing genomic data, and convert these data into a n*m int matrix. To do so i read the file using a FileReader and a BufferedReader. Then im going to tokenize the file using the StreamTokenizerExample Data:rs3000004 A/G chr14 19166795 - ncbi_b35 perlegen urn:lsid:perlegen.hapmap.org:Protocol:Genotyping_1.0.0:2 urn:lsid:perlegen.hapmap.org:Assay:25761.5787732:1 urn:lsid:dcc.hapmap.org:Panel:CEPH-30-trios:1 QC+ GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG NN GG GG GG GG GG GG GG NN GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GGrs8013379 C/G chr14 19217196 + ncbi_b35 perlegen urn:lsid:perlegen.hapmap.org:Protocol:Genotyping_1.0.0:2 urn:lsid:perlegen.hapmap.org:Assay:25760.7547817:1 urn:lsid:dcc.hapmap.org:Panel:CEPH-30-trios:1 QC+ CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC NN CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC NN CC CC CC CC CC NN CC CC CC CC CC CC CC CC CC CC CC CC NN CC CC CC CC CC CC CC CC CC CC NNI basicly just check what kind of nucelotide combination i have found and then set my matrix to the corresponding value (int). However if my file exceeds ~7MB i encounter a java.lang.OutOfMemoryError : Java heap space. What really puzzels me is that i can read files up to 100MB without any problem if i only print them back on screen for example, aswell as manipulating a 200000 * 900 matrix isnt a problem either. Im not creating any references to the Strings returned by StreamTokenizer aswell :/Your help will be much appreciated!-Kai Kratzp.s. sorry for any grammar/spelling errors i made, bad english mmmh kay ;)   
  This isn't a proper solution, but by default the VM starts with a maximum of only 64Mb. Theres a command line switch you can use to increase the amount available (eg. -Xmx256m for 256Mb) which might be good enough for a quick hacky workaround.   ;  Hi,Two remarks:- I think some source code could help us.- Have you checked the bug database?Regards,  EmmetjeGee   ;  Using...java -Xmx128m MyProgram.java...will give you more heap space. However, it seems you're holding references that you shouldn't while parsing your file. I know you mentioned you're not using any references to the captured String, but maybe one of the API classes you're using may be holding unwanted stuff.As the above post suggests, post some code so that we can try to spot what's causing this memory abuse.Son Of Cain;  Ok here is the sorcecode of my nasty little method.But before you start to tear it appart i think i neet to explain how it works a little further.As you saw in my previous post i was extracting data from txt files.Each line started with rsxxxx then some other stuff and finaly what im really intrested in the GG AA TT etc...However, when i translate these into my n*m matrix im assining values according to what combination came first in this line.The first pair of identical chars gets a 0 in my int matrix.The secong pair of identical chars gets a 1The pairs of non-identical chars get a 2if my sample data looks like thisGG GG AA ATGG GG GG GGAA TT AA ATit will be translated to: (rows in the flatfile represent columns in the matrix)0 0 00 0 11 0 02 0 2Conversion into my matrix isnt any problem(well at least for files smaller than ~7MB) its tested and working well, but since i share the code with you i want to give you all information to fully understand wtf i am doing there ^^.Used Methods:setTaxaIDs <- placeholder Method does nothing yetmatrix.setStateMatrixLegendColumnInfo <- is supposed to store some information about the line found in the flatfile, currently disabledmatrix.setStateMatrixLegendColumnInfo <- disabled for debugging aswellmatrix.setState <- sets a position in my n*m matrix to a valueAbout GenotypeMatrix:Well as you might see im using this line to set my int:matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,0);The last 3 are pretty self explanatory its n and m of my matrix and what value.The first parameter however is a enum used to determin wich matrix in Genotypematrix im accessing. So far it holds only one matrix but is planed to hold several (not yet implemented). I tested with disabling all above methods step by step and i didnt get any increase in performance, thats why i didnt include them in the sorce code (seems like the problem is not in there)I honestly admit my code is a bit messy in between but i didnt come up with a better way of reading my file and writing my matrix at the same time, since the coding is dependant on what pair of chars i found first in the currently read line.My first approach was to store all information in a List but i didnt programm it out after seeing that my list might be a little longer than 200000*90 Strings :PSo i thougt a direct conversion to the corresponding int would be better.HapMapIO.javapublic static GenotypeMatrix readHapmapFlatFile(String filename){try{int[] dimension = getDimension(filename);matrix = new GenotypeMatrix(dimension[0],dimension[1]);setTaxaIDs(filename, matrix);fr = new FileReader(filename);br = new BufferedReader(fr);st = new StreamTokenizer(br);st.ordinaryChar('/');st.ordinaryChar(':');st.ordinaryChar('#');int currentColumn = -1;int currentTaxon = 0;String rsID = "";char firstHomozygot = '-';char secondHomozygot = '-';char firstHeterozygot = '-';char secondHeterozygot = '-';boolean foundFirstHomozygot = false;boolean foundSecondHomozygot = false;boolean foundHeterozygot = false;do{if(st.nextToken() == StreamTokenizer.TT_WORD && st.sval.startsWith("rs") && st.lineno()>1){if(currentColumn >= 0)matrix.setStateMatrixLegendColumnInfo(currentColumn, rsID, firstHomozygot, secondHomozygot,  firstHeterozygot, secondHeterozygot);currentColumn++;rsID = st.sval;firstHomozygot = '-';secondHomozygot = '-';firstHeterozygot = '-';secondHeterozygot = '-';foundFirstHomozygot = false;foundSecondHomozygot = false;foundHeterozygot = false;currentTaxon = 0;}if(st.ttype == StreamTokenizer.TT_WORD && tokenIsHomozygotGenotypeInfo(st.sval)){if(foundFirstHomozygot && foundSecondHomozygot){if(st.sval.charAt(0) == firstHomozygot){matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,0);currentTaxon++;}if(st.sval.charAt(0) == secondHomozygot){matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,1);currentTaxon++;}}else{if(foundFirstHomozygot && !foundSecondHomozygot){if(st.sval.charAt(0) == firstHomozygot){matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,0);currentTaxon++;}else{foundSecondHomozygot = true;secondHomozygot = st.sval.charAt(0);matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,1);currentTaxon++;}}if(!foundFirstHomozygot){foundFirstHomozygot = true;firstHomozygot = st.sval.charAt(0);matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,0);currentTaxon++;}}}else{if(st.ttype == StreamTokenizer.TT_WORD && tokenIsHeterozygotGenotypeInfo(st.sval)){if(foundHeterozygot){if((st.sval.charAt(0) == firstHeterozygot && st.sval.charAt(1) == secondHeterozygot)||   (st.sval.charAt(1) == firstHeterozygot && st.sval.charAt(0) == secondHeterozygot)){matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,2);currentTaxon++;}}else{firstHeterozygot = st.sval.charAt(0);secondHeterozygot = st.sval.charAt(1);foundHeterozygot = true;matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,2);currentTaxon++;}}else{if(st.ttype == StreamTokenizer.TT_WORD && tokenIsUnknownGenotypeInfo(st.sval)){matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,GenotypeMatrix.MISSING_DATA);currentTaxon++;}}}if(st.ttype == StreamTokenizer.TT_EOL && st.lineno()>1)matrix.setStateMatrixLegendColumnInfo(currentColumn, rsID, firstHomozygot, secondHomozygot, firstHeterozygot, secondHeterozygot);}while(st.ttype != StreamTokenizer.TT_EOF);}catch(IOException e){// Nothing yet.}return matrix;}I did look into the StreamTokenizer sorce aswell but couldnt find any varaible holding old references.Ah and before i forget, your help is much appreciated!   ;  Are you sure the rsIDs are not getting stored any where?  1 per line isn't really a whole lot only 9 bytes and 1000 lines would only be ~1k, and it would take over 100,000 lines of stored rsIDs even to amount to 1mb...But a 200,000x2 int matrix >1mb.L-   ;  Hi,A remark:Different strings are always remembered/reused in java. They are stored in the string static pool. They are reused if you create the same string twice.I just ran your program with a file of 2GB (consisting of the two lines you gave) and commented out the matrix.setState(...) and tokenIs..GenotypeInfo(...) methods.Unfortunately I don't get a "out of memory exception" (I have 1GB). But then again, I am still missing parts of your program. I can't see a "memory consumer" in the part given, but I can give you some advice:- Look carefull at the stack trace when the exception occurs;- Strip your program down to the essential that still crashes;- Post your whole (but small) program again.You can also try some profilers (Look at the memory footprint while your application runs).Regards,  EmmetjeGee   