Chapter 12: Integer Overflows
Mathematical integers are unbounded. Machine integers are not: an int on our
platform holds exactly 32 bits, a short 16, a char 8. When a computation produces
a result that does not fit, C does not raise an error. Depending on the types
involved, the result silently wraps around, gets truncated, changes sign, or — in the
case of signed overflow — is undefined, which permits the compiler to assume it never
happens.
An integer overflow is rarely dangerous by itself. It becomes a vulnerability when the wrong number is then used for something that matters: the size of a buffer, the length of a copy, an array index, or a security check. A bad multiplication produces a tiny allocation; the following loop writes the “intended” amount of data into it; the heap is overflowed. This chapter shows the three ways integers go wrong in C, how attackers exploit each one, and how to write code that does not.
Learning objectives. After this chapter you should be able to:
- Explain two’s-complement representation and the range of common C integer types.
- Recognize the three classes of integer bugs: truncation, signedness errors, and arithmetic overflow.
- Trace how an integer bug turns into a memory-safety bug (an undersized buffer, an oversized copy, or a negative index) or a broken security check.
- Explain why integer overflow is hard to detect in C, including the role of undefined behavior in compiler optimization.
- Write overflow-safe size computations and use compiler flags and sanitizers to catch overflows.
12.1 A Two’s-Complement Refresher
The CPU stores an n-bit integer as n raw bits. Whether those bits mean a signed or
an unsigned number is decided by the instructions that use them, not by the bits
themselves. For signed integers, x86 (and every mainstream CPU) uses
two’s complement: the most significant bit has weight −2n−1, and all
other bits have their usual positive weights.
For a 32-bit int:
| Bits (hex) | As unsigned int | As int |
|---|---|---|
0x00000000 | 0 | 0 |
0x00000001 | 1 | 1 |
0x7fffffff | 2,147,483,647 | 2,147,483,647 (INT_MAX) |
0x80000000 | 2,147,483,648 | −2,147,483,648 (INT_MIN) |
0xffffffff | 4,294,967,295 (UINT_MAX) | −1 |
Two facts follow and will matter shortly:
- Negation is “invert all bits and add 1.” The x86
neginstruction replaces its operand with its two’s complement (equivalently, subtracts it from 0). - The range is asymmetric. There is one more negative value than positive values:
−2,147,483,648 has no positive counterpart in 32 bits. Negating
INT_MINyieldsINT_MINagain:0x80000000→ invert →0x7fffffff→ add 1 →0x80000000.
The common types on 32-bit x86 Linux (and, except where noted, on x86-64 Linux) are:
| Type | Bits | Range |
|---|---|---|
char / signed char | 8 | −128 … 127 |
unsigned char | 8 | 0 … 255 |
short | 16 | −32,768 … 32,767 |
unsigned short | 16 | 0 … 65,535 |
int | 32 | −231 … 231−1 |
unsigned int | 32 | 0 … 232−1 |
long, size_t, pointers | 32 (64 on x86-64) |
12.2 Truncation
Truncation happens when a value is assigned to a narrower type. The high bits are discarded:
int i = 0x12345678;
short s = i; /* s == 0x5678 = 22,136 */
char c = i; /* c == 0x78 = 120 ('x') */
i : 12 34 56 78 (32 bits)
s : 56 78 (low 16 bits kept)
c : 78 (low 8 bits kept)
Converting an out-of-range value to a signed narrower type is implementation-defined in C; GCC and Clang simply keep the low bits, as above. Converting to an unsigned narrower type is defined to wrap modulo 2n.
Truncation becomes a bug when a check is done on one width and the use on another. A classic shape:
void copy_record(char *src, unsigned int len)
{
char buf[256];
unsigned short n = len; /* truncated to 16 bits */
if (n >= sizeof(buf)) /* check uses the truncated value */
return;
memcpy(buf, src, len); /* copy uses the full value */
}
With len = 0x10010 (65,552), n is 0x0010 = 16, which passes the check, and
memcpy then copies 65,552 bytes into a 256-byte buffer. The fix is to check and use
the same variable, of a type wide enough for every possible input.
12.3 Signedness Bugs
Signedness bugs arise when a value is interpreted as signed in one place and as unsigned in another. C’s usual arithmetic conversions make this easy to do by accident: when a signed and an unsigned operand of the same rank meet in an expression, the signed one is converted to unsigned.
Signed checks, unsigned uses
The most common pattern is a length that is checked as a signed number but used as an unsigned one:
#define MAX 64
void handle(char *src, int size) /* size comes from the attacker */
{
char buf[MAX];
if (size < MAX) /* signed comparison: -1 < 64 is true */
memcpy(buf, src, size); /* memcpy takes size_t: -1 -> 0xffffffff */
}
A negative size passes the check, because −1 is indeed less than 64. But memcpy’s
third parameter has type size_t, so −1 is converted to 4,294,967,295 and memcpy
copies until it crashes or corrupts everything above buf. From the attacker’s point
of view, size is very big as an unsigned number yet still passes the check as a
signed one.
Interestingly, if the check had been written as if (size < sizeof(buf)), the
comparison would have been done in unsigned arithmetic (because sizeof yields a
size_t), −1 would have become huge, and the check would have rejected it. Correctness
here depends on the type of a constant several lines away — which is precisely why
these bugs are so common. Write the intent explicitly: reject negative values, or make
the parameter size_t from the start.
Negative sizes to allocators
int size = read_int_from_packet(); /* attacker sends -1 */
void *p = malloc(size); /* malloc(size_t): 0xffffffff bytes */
malloc receives an enormous request and returns NULL. If the program does not
check the return value, the next write dereferences NULL (a crash) or, if an offset
is added first, writes to a low address. In other variants, size + 1 or size * 2
wraps a negative value back to a small positive one, and the allocation succeeds
with the wrong size.
Assuming non-negativity
int table[100];
void set(int x, int value)
{
if (x < 100) /* forgets that x may be negative */
table[x] = value; /* x = -10 writes 40 bytes before table */
}
The check enforces only the upper bound. A negative index writes below the array, into whatever precedes it in memory. Array indices and sizes should be unsigned, or both bounds must be checked.
Case study: an “absolute value” password check
The following example looks as if no input could succeed:
int absolute(int i)
{
if (i < 0)
return -i;
else
return i;
}
void pw(void)
{
int passwd;
printf("Password: ");
scanf("%d", &passwd);
if (absolute(passwd) < 0) { /* an absolute value is never negative... */
printf("Password OK :)\n");
} else {
printf("Invalid Pw!\n");
}
}
The programmer reasons that absolute() never returns a negative number, so the
“Password OK” branch is dead code. On the machine, -i compiles to a neg
instruction, and Section 12.1 showed that negating INT_MIN gives INT_MIN:
input -2147483648 -> 0x80000000
neg 0x80000000 -> 0x7fffffff + 1 -> 0x80000000 (still INT_MIN)
absolute() returns -2147483648, which is < 0 -> "Password OK :)"
$ gcc -m32 -O0 pw.c -o pw
$ ./pw
Password: -2147483648
Password OK :)
In C, negating INT_MIN is undefined behavior, so the outcome is not even
guaranteed. With optimizations enabled, the compiler may reason that absolute()
can never be negative and delete the branch. GCC 9.4 on Ubuntu 20.04 does exactly
that:
$ gcc -m32 -O2 pw.c -o pw && echo -2147483648 | ./pw
Password: Invalid Pw!
$ gcc -m32 -ftrapv pw.c -o pw && echo -2147483648 | ./pw
Password: Aborted (core dumped)
The same source is “exploitable” at -O0, “safe” at -O2, and a crash with
-ftrapv (Section 12.6). The program’s security now depends on compiler flags,
which is a good sign that the code itself is wrong.
12.4 Arithmetic Overflow
Arithmetic overflow occurs when the result of +, -, or * does not fit in
the result type:
unsigned int sum = a + b; /* wraps if a + b >= 2^32 */
unsigned int product = a * b; /* wraps if a * b >= 2^32 */
unsigned int difference = a - b; /* wraps to a huge value if b > a */
For unsigned types the result is defined as the true result modulo 232, so
there is no crash and no warning — just a wrong number. Subtraction underflow is
especially common in length calculations: len - header_size with len smaller
than the header becomes a value close to 4 billion.
Case study: OpenSSH challenge-response (CVE-2002-0639)
Versions of OpenSSH from 2002 contained this code in the challenge-response authentication path:
nresp = packet_get_int(); /* attacker-controlled */
if (nresp > 0) {
response = xmalloc(nresp * sizeof(char *));
for (i = 0; i < nresp; i++)
response[i] = packet_get_string(NULL); /* writes nresp pointers */
}
On a 32-bit system sizeof(char *) is 4. If the attacker sends
nresp = 0x40000000, then
nresp * 4 = 0x40000000 * 4 = 0x1_0000_0000 -> truncated to 32 bits -> 0
and the allocation size wraps to 0. A slightly larger value, such as 0x40000001,
wraps to 4: a tiny but perfectly valid allocation. The loop then tries to store about
a billion pointers into that buffer, overflowing the heap with data (pointers to
attacker-supplied strings) and corrupting neighboring heap objects. This bug allowed
remote code execution in sshd — before authentication.
The pattern malloc(count * size) with an attacker-controlled count is one of
the most frequently exploited integer bugs, and the reason allocation APIs such as
calloc check the multiplication internally.
12.5 Why Integer Bugs Are Hard to Catch in C
Machine code has no types. Registers and memory hold raw bits. The same
0xffffffff is −1 to a signed comparison (jl, jg) and 4,294,967,295 to an
unsigned one (jb, ja). The CPU does record overflow in the flags — the carry
flag (CF) for unsigned, the overflow flag (OF) for signed — but C provides no portable
way to read them, so the information is discarded.
Signed overflow is undefined behavior. The C standard says nothing about what a signed overflow produces, and compilers exploit that freedom: they may assume signed overflow never happens. A well-intentioned check like this one
int a = get_input();
if (a + 100 < a) { /* "did a + 100 overflow?" */
abort();
}
may be compiled to nothing at -O2, because if overflow is impossible, a + 100 < a
is always false. The check must be written so that it never overflows in the first
place: if (a > INT_MAX - 100).
Implicit conversions are invisible. Integer promotions, the usual arithmetic
conversions, and conversions at function-call boundaries (int to size_t in
memcpy) happen without any syntax at the point of use. The bug in Section 12.3 is
not visible on the line where it happens.
Testing rarely hits the edges. Overflows usually require values near 231 or 232, and coverage-guided fuzzers (Chapter 9) explore branches, not values: the overflowing multiplication executes the same instructions as a benign one, so the fuzzer sees no new coverage. Unless a sanitizer turns the overflow itself into a crash, the bug shows up only if the corrupted size later causes a detectable memory error. The same blind spot affects human and LLM-based code review: the arithmetic looks reasonable, and the dangerous range of inputs is not apparent from the code. This is one reason integer bugs, like the race conditions discussed later in the course, keep appearing in mature, heavily fuzzed software.
12.6 Defending Against Integer Overflow
Use the right types
- Sizes and counts: use
size_t, which is unsigned and wide enough for any object size. Never store a size in anint. - Fixed-width data (network fields, file formats): use
uint8_t,uint16_t,uint32_t,uint64_tfrom<stdint.h>, and convert tosize_twith an explicit range check. - Integers that hold pointers: use
intptr_t/uintptr_t, neverint(which is narrower than a pointer on x86-64). - Reject negative values as soon as untrusted input is parsed.
Check before you compute
Test for overflow using operations that cannot themselves overflow:
#include <stdint.h>
/* a + b for size_t */
if (b > SIZE_MAX - a) goto too_big;
total = a + b;
/* count * sizeof(T) */
if (count > SIZE_MAX / sizeof(T)) goto too_big;
p = malloc(count * sizeof(T));
Better still, let the library do it. calloc(count, size) checks the multiplication
and fails rather than wrapping, and glibc’s reallocarray(ptr, count, size) does the
same for resizing.
GCC and Clang provide checked-arithmetic builtins that compute the mathematically correct result and report whether it fit:
size_t bytes;
if (__builtin_mul_overflow(nresp, sizeof(char *), &bytes))
fatal("too many responses");
response = xmalloc(bytes);
__builtin_add_overflow and __builtin_sub_overflow work the same way, for signed and
unsigned types alike. C23 standardizes this interface as ckd_add, ckd_sub, and
ckd_mul in <stdckdint.h>.
Compiler flags and sanitizers
| Flag | Effect |
|---|---|
-fwrapv | Define signed overflow to wrap (two’s complement). Removes the undefined behavior, so overflow checks are not optimized away, but still produces wrong values. |
-ftrapv | Trap (abort) on signed overflow in +, -, *. Historically buggy on older compilers and with constant expressions. |
-fsanitize=undefined | UBSan: report signed overflow, invalid shifts, and other undefined behavior at run time. Not available in very old compilers. |
-fsanitize=signed-integer-overflow | Only the signed-overflow check of UBSan. |
-fsanitize=unsigned-integer-overflow | (Clang only) Also report unsigned wraparound, which is legal C but often a bug. |
A demonstration:
/* int.c */
#include <stdio.h>
#include <limits.h>
int main(void)
{
int a = INT_MAX;
int b = a + 1;
printf("%d\n", b);
return 0;
}
$ gcc int.c -o int && ./int
-2147483648
$ gcc -ftrapv int.c -o safe-int && ./safe-int
Aborted (core dumped)
$ gcc -fsanitize=undefined int.c -o ubsan-int && ./ubsan-int
int.c:7:15: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int'
-2147483648
(The exact line:column and wording vary slightly between GCC and Clang.) UBSan
reports and continues by default; add -fno-sanitize-recover=undefined to make
it abort, which is what you want when fuzzing so that every overflow becomes a crash
the fuzzer records (Chapter 9).
Memory-safe languages
Rust makes the choice explicit. In debug builds, arithmetic overflow panics
(“attempt to multiply with overflow”); in release builds it wraps unless
overflow-checks is enabled. Code that handles untrusted numbers should not rely on
either behavior and should instead say what it means:
#![allow(unused)] fn main() { let total = count.checked_mul(elem_size)?.checked_add(HEADER)?; // None on overflow let wrapped = a.wrapping_add(b); // explicit modular arithmetic let clamped = a.saturating_sub(b); // stops at 0 instead of wrapping }
A Rust integer overflow can still produce a wrong size, but the subsequent buffer access is bounds-checked, so the result is a panic or an error — a denial of service at worst — rather than heap corruption.
Going deeper. Integer bugs are often only the first step: the wrapped size produces an undersized heap buffer, and the real exploitation happens through heap metadata or adjacent heap objects. Heap exploitation is the subject of a later chapter. Dietz, Li, Regehr, and Adve’s study “Understanding Integer Overflow in C/C++” (ICSE 2012), which built the checker that became Clang’s integer sanitizers, found intentional and unintentional overflows in a large fraction of the widely used programs they tested.
Check Your Understanding
1. What values do s and c hold after int i = 0x12345678; short s = i; char c = i;?
Answer
s keeps the low 16 bits: 0x5678 = 22,136. c keeps the low 8 bits: 0x78 = 120,
the character 'x'.
2. Why does entering -2147483648 make absolute(passwd) < 0 true in the
password example (compiled without optimization)?
Answer
-2147483648 is INT_MIN = 0x80000000. It is negative, so absolute returns -i,
compiled as neg. Two’s-complement negation inverts the bits (0x7fffffff) and adds 1,
giving 0x80000000 again, because +2,147,483,648 is not representable in 32 bits. The
function returns INT_MIN, which is less than 0. (In C this is undefined behavior, so an
optimizing compiler may behave differently.)
3. In the handle() function of Section 12.3, which value of size overflows
buf, and what are two different ways to fix the function?
Answer
Any negative value, such as −1: it passes the signed check size < 64, then is
converted to a huge size_t by memcpy. Fixes: (1) declare size as size_t, so
negative inputs cannot exist and the comparison is unsigned; or (2) keep int but
check both bounds, if (size >= 0 && size < MAX).
4. In the OpenSSH bug, why does nresp = 0x40000000 lead to a heap overflow on a
32-bit system but not on a 64-bit system with the same code?
Answer
On 32-bit, sizeof(char *) is 4 and the product 0x40000000 × 4 = 232 is
truncated to 0 (a value just above 0x40000000 gives a tiny allocation), while the loop
still writes nresp pointers. On 64-bit, the multiplication is done in 64-bit size_t
arithmetic with sizeof(char *) = 8, giving 8 GiB without wrapping: the allocation
either fails or is genuinely that large, so the loop does not overflow it. (A large
enough nresp type on 64-bit could still wrap; the robust fix is a checked
multiplication.)
5. Why might the check if (a + 100 < a) abort(); disappear from an optimized
binary, and how should it be written?
Answer
For signed a, overflow is undefined behavior, so the compiler may assume a + 100
never overflows; under that assumption a + 100 < a is always false and the branch
is removed. Write a check that cannot overflow: if (a > INT_MAX - 100) abort();, or
use __builtin_add_overflow(a, 100, &r).
6. Rewrite p = malloc(count * sizeof(struct item)); so that it cannot allocate a
wrapped size.
Answer
Any of: p = calloc(count, sizeof(struct item)); (which also zeroes the memory);
p = reallocarray(NULL, count, sizeof(struct item));; or check first:
if (count > SIZE_MAX / sizeof(struct item)) return NULL;. Always check for a NULL
result afterwards.
7. Why do coverage-guided fuzzers often miss integer overflows, and what makes them better at finding them?
Answer
The overflowing operation executes the same instructions and branches as a benign one,
so the fuzzer gets no new coverage feedback for reaching the dangerous value, and the
overflow itself does not crash the program. Compiling the target with UBSan
(-fsanitize=undefined, -fno-sanitize-recover) or -ftrapv turns each overflow into
a crash, and comparison-splitting or value-profile features in modern fuzzers help them
reach boundary values.
Summary
- C integers have fixed widths. Signed integers use two’s complement, whose range
is asymmetric:
-INT_MINis not representable and evaluates toINT_MINon x86. - Truncation: assigning to a narrower type drops high bits; checking one width and using another is a bug.
- Signedness bugs: a negative value that passes a signed check becomes huge when
converted to
size_t; negative indices bypass upper-bound checks. - Arithmetic overflow: unsigned arithmetic wraps silently;
malloc(count * size)with attacker-controlledcount(as in OpenSSH, CVE-2002-0639) yields an undersized buffer and a heap overflow. - Integer bugs are hard to see because machine code has no types, conversions are implicit, signed overflow is undefined behavior that compilers optimize around, and fuzzers get no coverage signal for bad values.
- Defenses:
size_tand fixed-width types, checks that cannot overflow,calloc/reallocarray,__builtin_*_overflow(C23ckd_*), compiler options-fwrapv,-ftrapv, and-fsanitize=undefined, and languages such as Rust with explicitchecked_*/wrapping_*arithmetic.
Connection to the Labs
Optional Lab: Rust Memory Safety, Part B, asks you to
implement alloc_size(count, elem_size) without overflow — the Rust version of
Section 12.6’s checked allocation size — and to explain what the equivalent C
expression count * elem_size + HEADER would produce for usize::MAX / 4 and 4.
The same part covers Heartbleed, a close relative of this chapter’s bugs: an
attacker-supplied length that the program trusted without checking it against the
real data.
Further Reading
- CVE-2002-0639 (OpenSSH challenge-response integer overflow): https://nvd.nist.gov/vuln/detail/CVE-2002-0639
- Will Dietz, Peng Li, John Regehr, and Vikram Adve. “Understanding Integer Overflow in C/C++.” 34th International Conference on Software Engineering (ICSE), 2012.
- x86
neginstruction reference: https://www.felixcloutier.com/x86/neg - Two’s complement: https://en.wikipedia.org/wiki/Two%27s_complement
- GCC manual, “Built-in Functions to Perform Arithmetic with Overflow Checking,” and
the
-fsanitize=undefineddocumentation.