title: In case of integer overflows what is the result of (unsigned int) * (int) ? unsigned or int?
body: <p>In case of integer overflows what is the result of <code>(unsigned int) * (int)</code> ? <code>unsigned</code> or <code>int</code>? What type does the array index operator (<code>operator[]</code>) take for <code>char*</code>: <code>int</code>, <code>unsigned int</code> or something else?</p>
  
  <p>I was auditing the following function, and suddenly this question arose. The function has a vulnerability at line 17.</p>
  
  <pre><code>// Create a character array and initialize it with init[] 
  // repeatedly. The size of this character array is specified by 
  // w*h.
  char *function4(unsigned int w, unsigned int h, char *init)
  {
      char *buf;
      int i;
  
      if (w*h &gt; 4096)
          return (NULL);
  
      buf = (char *)malloc(4096+1);
      if (!buf)
          return (NULL);
  
      for (i=0; i&lt;h; i++)
          memcpy(&amp;buf[i*w], init, w);  // line 17
  
      buf[4096] = '\0';
  
      return buf;
  }
  </code></pre>
  
  <p>Consider both <code>w</code> and <code>h</code> are very large unsigned integers. The multiplication at line 9 have a chance to pass the validation. </p>
  
  <p>Now the problem is at line 17. Multiply <code>int i</code> with <code>unsigned int w</code>: if the result is <code>int</code>, it is possible that the product is negative, resulting in accessing a position that is before <code>buf</code>. If the result is <code>unsigned int</code>, the product will always be positive, resulting in accessing a position that is after <code>buf</code>.</p>
  
  <p>It's hard to write code to justify this: <code>int</code> is too large. Does anyone has ideas on this?</p>
  
  <p>Is there any documentation that specifies the type of the product? I have searched for it, but so far haven't found anything.</p>
  
  <p>I suppose that as far as the vulnerability is concerned, whether <code>(unsigned int) * (int)</code> produces <code>unsigned int</code> or <code>int</code> doesn't matter, because in the compiled object file, they are just bytes. The following code works the same no matter the type of the product:</p>
  
  <pre><code>unsigned int x = 10;
  int y = -10;
  
  printf("%d\n", x * y);  // print x * y in signed integer
  printf("%u\n", x * y);  // print x * y in unsigned integer
  </code></pre>
  
  <p>Therefore, it does not matter what type the multiplication returns. It matters that whether the consumer function takes <code>int</code> or <code>unsigned</code>.</p>
  
  <p>The question here is <strong>not</strong> how bad the function is, or how to improve the function to make it better. The function undoubtedly has a vulnerability. The question is about the exact behavior of the function, based on the prescribed behavior from the standards.</p>