title: C program: [Done] exited with code=3221225477 in 2.322 seconds (Calloc / Free / Segmenation Error?)
body: <p>I'm working on a C program to manipulate large volumes of CSV data. It was running fine in develoment with a smallish test file. But when the size of the file increases, it starts to fail. Depending on whether I compile it with gcc or minGW's gcc, it fails with segmentation error or 3221225477 / 0xC0000005 at different places, always either :</p>
  <pre><code>if (fclose(fp)) { 
      printf(&quot;Error closing file: %s, %s, %d.\n&quot;, fileName, __func__, __LINE__);
      exit(300); 
  }
  </code></pre>
  <p>Note it doesn't get past the fclose(). Or one of these:</p>
  <pre><code>data_PE_T12         = calloc(width_T*dataDepthDay, sizeof(*data_PE_T12));
  </code></pre>
  <p>It's long, so I'll try to show the relevant parts. First the Main function:</p>
  <pre><code>#include &lt;stdio.h&gt;
  #include &lt;string.h&gt; // strtok
  #include &lt;stdlib.h&gt; // atoi &amp; atof
  #include &lt;time.h&gt;   // time functions
  #include &lt;math.h&gt;   // expf()
  
  
  ...
  // Array Sizes
  static int dataDepth, dataDepthDay;                                                                                             
  static int fromTime, toTime;                                                                                                    
  static int width, width_T, width_H, width_C;
  
  
  // Array Pointers
  static int                      *timeArray,         *timeArrayDay,      *timeArrayPE;                                          
  static struct sensorHeader_t    *headerArray,       *headerArray_T,     *headerArray_H,     *headerArray_C;                 
  
  // of depth dataDepthDay
  static float                    *data_E_T25,        *data_E_T30;        
  static float                    *data_E_T12,        *data_E_T18,        *data_E_H60,        *data_E_H70,        *data_E_C1500;
  static float                    *data_PE_T12,       *data_PE_T18,       *data_PE_H60,       *data_PE_H70,       *data_PE_C1500;
  ... plus loads more.
  
  // functions
  void  grabDepth(void);                  // OK
  void  grabPayload(void);                // OK
  ... plus loads more.
  
  
  int main(int argc, char **argv)
  {
  
  // Grab Input File Name 
      if (argc == 2) {
          strcpy(rawFile, &quot;in/&quot;);
          strcat(rawFile, argv[1]);
      } else { // dev
          strcpy(rawFile, &quot;in/sensor_report.csv&quot;);
      }
  
      printf(&quot;size max = %d&quot;, __SIZE_MAX__);
  
  // Parse and Copy File
      grabDepth();
      grabPayload();
  
  // Run functions
      genRawData();       // Raw T, H &amp; C files   
      genExposureE();     // 
      genExposureAPE();   // 
  
  
      return 0;
  }
  </code></pre>
  <p>Next the first function that is called. This one opens the main input file and pulls out a number of array widths and depths that are used to calloc for arrays already declared as static pointers. The idea is that this will make the memory handling nice and flexible as the file size increases...</p>
  <pre><code>void grabDepth(void)
  {
  // 1. Open File
      FILE *fp = fopen(rawFile, &quot;r&quot;);
      char buf[15000]; // Big enough to deal with lots of devices.
      
      if (!fp) {
          printf(&quot;Can't open the file: %s: %s, %d.\n&quot;, rawFile, __func__, __LINE__);
          exit(100);
      }
      
      while (fgets (buf, sizeof(buf), fp)) {
          int lineLen = strlen(buf);
          int colNum = 1;
          char *field = strtok(buf, &quot;,&quot;);
  
          if (field &amp;&amp; strcmp(field, &quot;From&quot;) == 0) {
              // printf(&quot;\n\n*** row 2 ***\n\n&quot;);
              // int fromTime, toTime = 0;
              
              while (field) {
                  if (colNum == 2) {
                      fromTime = atof(field);
                  }
                  
                  if (colNum == 4) {
                      toTime = atof(field);
                  }
                  field = strtok(NULL, &quot;,&quot;);
                  colNum++;
              }
              
              // printf(&quot;FromTime = %d. ToTime = %d.\n&quot;, fromTime, toTime);
              dataDepth = ( toTime - fromTime )/900;
              // printf(&quot;dataDepth = %d.\n&quot;, dataDepth);
              continue; // to next iteration. 
               
          }
          
  // 3. Grab file width from line 10 (commsType) Check if buf Overruns too
          if (field &amp;&amp; strcmp(field, &quot;TimeStamp&quot;) == 0) {
              // First Check Line is long enough!
              if (lineLen == sizeof(buf)-1) { // buf has overrun!
                  printf(&quot;File Read-Line Overrun: %s, %d.\n&quot;, rawFile, __func__, __LINE__);
                  exit(200);
              }
              // printf(&quot;Line Length = %d\n&quot;, lineLen);
              // printf(&quot;Buf Size    = %d\n&quot;, sizeof(buf));
              width = -2; // ignore timestamps : I ballsed up the commas in csv file (-2 instead of -1)
              while (field) {
                  if(field = strtok(NULL, &quot;,&quot;)) {
                      width ++;
                  }
              }
              break; // out of loop!
          }
      }
      
      //dataDepthDay = dataDepth/96 + (dataDepth % 96 !=0); // round up?!
      dataDepthDay = dataDepth/96;                        // round down?!
      printf(&quot;\n 1. grabDepth() Results\n&quot;);
      printf(  &quot;------------------------\n&quot;);
      printf(&quot;Raw Data Width     = %d\n&quot;, width);
      printf(&quot;Raw Data Depth     = %d\n&quot;, dataDepth);
      printf(&quot;dataDepthDay Depth = %d\n\n&quot;, dataDepthDay);
  
      if (fclose(fp)) { 
          printf(&quot;Error closing file: %s, %d.\n&quot;, rawFile, __func__, __LINE__);
          exit(300); 
      }
  }
  </code></pre>
  <p>After that, it's just calling one function after another, all of which follow the general pattern of:</p>
  <pre><code>void _genRawData(char* sensorType, struct sensorHeader_t *sensorHeader, float *dataArray, int *timeArray, size_t dataDepth, size_t width) {
      FILE *fp;
      strcpy(fileName, &quot;out/&quot;);
      strcat(fileName, sensorType);
      strcat(fileName, &quot;_raw.csv&quot;);
      fp = fopen(fileName, &quot;w&quot;);
  
      // check file opened OK. 
      if (fp == NULL) {
          printf(&quot;Error! Couldn't Create file: %s\n&quot;, fileName);
          return;
      }
      printf(&quot;building file : %s\n&quot;, fileName);
      
  
      // Allocate Memory
      timeArrayDay    = calloc(dataDepthDay, sizeof(*timeArrayDay));
      timeArrayPE     = calloc(dataDepthDay, sizeof(*timeArrayPE)); // xxxx same array as day time array!?
      data_E_T12      = calloc(width_T*dataDepthDay, sizeof(*data_E_T12));
      data_E_T18      = calloc(width_T*dataDepthDay, sizeof(*data_E_T18));
      data_E_H60      = calloc(width_H*dataDepthDay, sizeof(*data_E_H60));
      data_E_H70      = calloc(width_H*dataDepthDay, sizeof(*data_E_H70));
      
      // do stuff and build new arrays up and put into files...
      
      
      
          if (fclose(fp)) { 
          printf(&quot;Error closing file: %s, %d.\n&quot;, rawFile, __func__, __LINE__);
          exit(300); 
      }
  }
  </code></pre>
  <p>I've only called calloc once on each 2-D array, and for the sake of debugging I've removed the free() calls.</p>
  <p>I figure I'm doing something wrong with memory management, which is biting me when the array sizes grow past a certain point, but I can't figure out what's wrong. I've tried to make sure the memory I access has been allocated correctly and working on a big powerful actual computer (I'm an embedded person usually), I wouldn't expect any issues with OS handing out data? Isn't there plenty to go around!?</p>