Calculating an Integer Overflow

I was playing an exploit game yesterday, and had to compute an exact value for an integer overflow, which made me think (when I’ve run into this before, I’ve just had to get ‘close enough’). In the binary, it compares some user input to the integer 9, which it must be “less than”

call _atoi
mov [ebp+var_C], eax
cmp [ebp+var_C], 9
jle short loc_8 ; process input and reach overflow

n is then multiplied by 4 to make room for 9 ints

shl eax, 2

var_c is then used as the n parameter in memcpy

void *memcpy(void *dest, const void *src, size_t n);

The vulnerability is possible (at least in part) to the shl, which can be used to wrap the integer and bypass the jle check. It’s fairly obvious there is an integer overflow here, and in fact, calculating n to be an exact value is also not difficult. So in my case I wanted n in the memcpy call to equal exactly 80.

The very first thing I did was to look at this http://en.wikipedia.org/wiki/Two’s_complement, which I remember having to do in school. It’s not complicated, but once you start throwing algebra in… anyway, so instead of using math I just wrote a wrapper program on the same machine.

#include <limits.h>
#include <stdio.h>

void main()
{
  //this should be 80. Sanity check
  int y =  -INT_MAX - INT_MAX + 78;
  printf("%dn", y); 

  printf("%dn", INT_MAX);
}

which prints

2147483647
80

Then just plop this in a calculator. Remember to divide by 4 to undo the multiply

>>> (-2147483647*2 + 78)/4.0
-1073741804.0

I entered this in the appropriate place, and set a breakpoint on the call to memcpy.

(gdb) x/d $esp+8
0xbffff2b8: 80

Success, we’ve managed to set n to 80. This one took more time to write out than to solve, but hey, maybe it will be useful for someone. Plus I needed a filler today… I have some cool stuff I’m working on, but it won’t be ready until at least next post, or maybe the post after :)