Calculating an Integer Overflow

I was playing an exploit game yesterday, and had to compute an exact value for an integer overflow, which made me think (when I’ve run into this before, I’ve just had to get ‘close enough’). In the binary, it compares some user input to the integer 9, which it must be “less than”

call _atoi
mov [ebp+var_C], eax
cmp [ebp+var_C], 9
jle short loc_8 ; process input and reach overflow

n is then multiplied by 4 to make room for 9 ints

shl eax, 2

var_c is then used as the n parameter in memcpy

void *memcpy(void *dest, const void *src, size_t n);

The vulnerability is possible (at least in part) to the shl, which can be used to wrap the integer and bypass the jle check. It’s fairly obvious there is an integer overflow here, and in fact, calculating n to be an exact value is also not difficult. So in my case I wanted n in the memcpy call to equal exactly 80.

The very first thing I did was to look at this’s_complement, which I remember having to do in school. It’s not complicated, but once you start throwing algebra in… anyway, so instead of using math I just wrote a wrapper program on the same machine.

#include <limits.h>
#include <stdio.h>

void main()
  //this should be 80. Sanity check
  int y =  -INT_MAX - INT_MAX + 78;
  printf("%dn", y); 

  printf("%dn", INT_MAX);

which prints


Then just plop this in a calculator. Remember to divide by 4 to undo the multiply

>>> (-2147483647*2 + 78)/4.0

I entered this in the appropriate place, and set a breakpoint on the call to memcpy.

(gdb) x/d $esp+8
0xbffff2b8: 80

Success, we’ve managed to set n to 80. This one took more time to write out than to solve, but hey, maybe it will be useful for someone. Plus I needed a filler today… I have some cool stuff I’m working on, but it won’t be ready until at least next post, or maybe the post after :)

gcc security tips

Here are some flags that may help vulnerable code from being executed.


This should get rid of some buffer overflows that can be analyzed statically and some obvious ones (strcpying input, format string vulnerabilities).

More information can be found here:


From the man page:

Emit extra code to check for buffer overflows, such as stack smashing attacks.  This is done by adding a guard variable to functions with vulnerable objects.  This includes functions that call alloca, and functions with buffers larger than 8 bytes. The guards are initialized when a function is entered and then checked when the function exits.  If a guard check fails, an error message is printed and the program exits.

(this is enabled by default in recent versions of Ubuntu)