Chapter 3: A Machine-Level Primer: x86, the Stack, and GDB

The attacks in the next several chapters do not happen in C. They happen in the machine the C program is compiled to. A C programmer sees a 16-byte array; the CPU sees a range of addresses on the stack that happens to sit a few bytes below a saved return address. A C programmer sees a function call; the CPU sees a call instruction that pushes an address onto memory any buggy store can overwrite. To understand how a buffer overflow hijacks a program — and later, how defenses stop it — you need to see the program the way the CPU does.

This chapter is a focused refresher, not a full course in assembly. It covers just enough of 32-bit x86 to read the disassembly of a vulnerable function, draw its stack frame, and watch it run under a debugger. We point out the 64-bit differences along the way, because they matter for exploitation.

Learning objectives. After this chapter you should be able to:

  • Explain little-endian byte order and predict how a multi-byte value is laid out in memory.
  • Name the x86 general-purpose registers and the roles of esp, ebp, and eip (and their 64-bit counterparts).
  • Read AT&T-syntax assembly and translate it to Intel syntax and back.
  • Explain exactly what push, pop, call, ret, and leave do to the stack.
  • Draw the stack frame of a 32-bit cdecl function, including where arguments, the return address, the saved frame pointer, and locals live.
  • Use GDB with pwndbg to set breakpoints, step, and examine registers and memory.
  • Generate raw binary input for a program with Python 3.
  • Describe the memory layout of a Linux process.

3.1 Why a Machine-Level View?

C gives you an abstract machine where arrays have sizes and functions return to their callers. Neither is enforced at run time. An array is a starting address; nothing stops a store from running past its end. A function’s “return to the caller” is a ret instruction that jumps to whatever address happens to be stored at the top of the stack. Memory-corruption attacks live exactly in the gap between the abstraction and the machine.

We use 32-bit x86 for most examples in this part of the book, following the labs: the stack frames are simpler (arguments live on the stack), addresses are four bytes and usually contain no zero bytes, and the 32-bit environment is still easy to build on a modern 64-bit Linux system with gcc -m32. Where 64-bit x86 (also called x86-64, AMD64, or x64) behaves differently, we say so.

Note. You need an Intel or AMD x86-64 machine (or VM) for this material. An ARM machine, including an Apple Silicon Mac running an ARM Linux VM, cannot natively run the 32-bit x86 binaries used here. On Ubuntu, install the 32-bit toolchain with sudo apt install gcc-multilib; without it, gcc -m32 fails with fatal error: bits/libc-header-start.h: No such file or directory.

3.2 Bytes, Words, and Endianness

Memory is an array of bytes, each with an address. A 32-bit value (a dword, “double word,” in Intel’s terminology; a word is 16 bits) occupies four consecutive bytes. Which byte goes first is the machine’s endianness:

  • Little endian (x86, and ARM in its usual configuration): the least significant byte is stored at the lowest address.
  • Big endian (network byte order; some older RISC machines): the most significant byte is stored at the lowest address.

Storing the dword 0x12345678 at address 0x1000:

 address:        0x1000  0x1001  0x1002  0x1003
 little endian:    78      56      34      12      <- x86
 big endian:       12      34      56      78

This matters constantly in exploitation. When you overwrite a return address with 0x080491a6 by writing bytes into a buffer, you must write them in memory order: a6 91 04 08. In Python:

>>> (0x080491a6).to_bytes(4, byteorder='little')
b'\xa6\x91\x04\x08'

3.3 Registers

Registers are the CPU’s small set of named, fast storage locations. 32-bit x86 has:

  • 8 general-purpose registers (GPRs): eax, ebx, ecx, edx, esi, edi, ebp, esp;
  • the instruction pointer eip;
  • the flags register eflags, holding condition codes set by arithmetic and comparison instructions;
  • 6 segment registers (cs, ds, ss, es, fs, gs), mostly vestigial in modern flat-memory systems, though gs/fs point to thread-local storage — which, as Chapter 6 shows, is where the stack canary lives;
  • control registers (cr0, cr3, …), debug registers, and vector registers for floating-point and SIMD (x87, SSE, AVX), which we will not need.

x86-64 widens the GPRs to 64 bits, renames them with an r prefix (rax, …, rsp), renames the instruction pointer rip, and adds eight more GPRs, r8–r15. Thanks to backward compatibility, the low parts of each register can be named separately:

  63                31            15     7     0
 +-------------------+-------------+------+------+
 |                  rax (64 bits)                 |
 |                   |     eax (32 bits)          |
 |                   |             |   ax (16)    |
 |                   |             |  ah  |  al   |
 +-------------------+-------------+------+------+
64-bit32-bit16-bit8-bitConventional role
raxeaxaxah/alaccumulator; function return value; syscall number
rbxebxbxbh/blbase; callee-saved
rcxecxcxch/clcounter
rdxedxdxdh/dldata
rsiesisisilsource index
rdiedididildestination index
rbpebpbpbplbase (frame) pointer
rspespspsplstack pointer
r8–r15r8d–r15dr8w–r15wr8b–r15b64-bit mode only

(The 8-bit sil, dil, bpl, spl names exist only in 64-bit mode.)

Three registers deserve special attention because attacks target them:

  • esp/rsp always holds the address of the top of the stack.
  • ebp/rbp conventionally holds the base of the current function’s stack frame; locals and arguments are addressed relative to it.
  • eip/rip holds the address of the next instruction to execute. You cannot mov into it directly; it changes only through control-flow instructions — jmp, conditional jumps, call, ret. Controlling eip means controlling the program, and the attacks in Part II are all, at bottom, ways to get an attacker-chosen value into it.

3.4 Reading Assembly: AT&T and Intel Syntax

Two notations exist for the same instructions. GNU tools (gcc -S, objdump, GDB by default) use AT&T syntax; Intel’s manuals, most Windows tools, and pwndbg (by default) use Intel syntax. You must be able to read both.

AT&TIntel
Operand orderop src, dstop dst, src
Registers%eaxeax
Immediates$0x100x10
Sizesuffix on mnemonic: movl, movbfrom operands, or DWORD PTR
Memorydisp(base,index,scale)[base+index*scale+disp]

The same three instructions both ways:

 AT&T                         Intel
 movl  $0x0, -0xc(%ebp)       mov   DWORD PTR [ebp-0xc], 0x0
 lea   -0x1c(%ebp), %eax      lea   eax, [ebp-0x1c]
 add   $0x10, %esp            add   esp, 0x10

The memory form disp(base,index,scale) means address base + index*scale + disp. For example, 0x8(%ebp) is the four bytes at ebp+8, and (%eax,%ecx,4) is eax + ecx*4 — the ecx-th element of an int array starting at eax.

This book writes hand-written assembly in AT&T syntax, to match gcc and the labs. In GDB you can switch with set disassembly-flavor intel or set disassembly-flavor att.

3.5 Essential Instructions

You can read most compiler output with a dozen instructions.

Data movement.

  • mov src, dst — copy. At most one operand may be memory.
  • lea mem, reg — load effective address: compute the address of a memory operand and put the address itself (not the memory contents) in reg. Compilers use it to take addresses (&buf) and for cheap arithmetic.
  • push src — decrement esp by 4, then store src at (%esp).
  • pop dst — load (%esp) into dst, then increment esp by 4.

Arithmetic and logic. add, sub, imul, inc, dec, neg, and, or, xor, shl, shr, sar. Each sets flags in eflags. xor %eax, %eax is the idiomatic way to set a register to zero — a fact that will matter in Chapter 5.

Comparison. cmp b, a computes a - b (AT&T order!) and sets the flags without storing the result; test b, a computes a & b likewise. A conditional jump then reads the flags:

 cmpb  $0xa, (%ebx)     # compare the byte at address ebx with 10
 je    loop             # jump to 'loop' if they were equal

Jumps.

  • jmp target — unconditional. jmp label is a direct jump (target encoded in the instruction); jmp *%eax is an indirect jump (target computed at run time). The distinction is central to Control-Flow Integrity (Chapter 8).
  • jcc target — conditional jumps: je/jz (equal / zero), jne/jnz, jg, jge, jl, jle (signed comparisons), and ja, jae, jb, jbe (unsigned comparisons). Signed-vs-unsigned confusion in comparisons reappears in Chapter 12.

Calls and returns.

  • call target — push the address of the next instruction (the return address), then jump to target. Equivalent to push %eip_next; jmp target. Like jmp, it has direct (call vul) and indirect (call *%eax, call *0x8(%ebp)) forms.
  • ret — pop the top of the stack into eip. Equivalent to pop %eip.
  • leave — tear down a frame: mov %ebp, %esp; pop %ebp.

Note what ret does not do: it does not check that the value it pops was put there by a call. It trusts the stack. Hold on to that thought.

System calls. int $0x80 (32-bit) and syscall (64-bit) enter the kernel. Chapter 5 covers them.

3.6 The Run-Time Stack

The stack is a region of memory used for function calls: return addresses, saved registers, arguments, and local variables. On x86 it starts at a high address and grows toward lower addresses. esp points at the most recently pushed value.

 push %eax                         pop %ebx

 high                              high
 |  ...  |                         |  ...  |
 |   A   |                         |   A   |
 |   B   | <- esp (before)         |   B   |
 | eax   | <- esp (after: esp-4)   |  eax  | <- esp (before);  ebx := eax value
 low                               low        esp (after) = esp+4, back at B

Popping does not erase anything; the old value stays in memory below esp until something overwrites it. That is why uninitialized local variables can contain stale data from earlier calls.

3.7 Calling Conventions and Stack Frames

A calling convention is an agreement between caller and callee about where arguments go, where the return value goes, and which registers each side must preserve. The traditional 32-bit Linux convention, cdecl, says:

  • Arguments are pushed on the stack right to left, so the first argument ends up at the lowest address, nearest the return address.
  • The caller removes the arguments after the call returns.
  • The return value is in eax.
  • ebx, esi, edi, and ebp are callee-saved: a function that uses them must restore them before returning. eax, ecx, edx are caller-saved (scratch).

Prologue and epilogue

A function compiled with frame pointers starts with a prologue and ends with an epilogue:

 vul:
     push  %ebp              # save caller's frame pointer
     mov   %esp, %ebp        # ebp := base of our frame
     sub   $N, %esp          # reserve N bytes for locals
     ...                     # body
     leave                   # mov %ebp,%esp ; pop %ebp
     ret                     # pop return address into eip

After the prologue, the frame has a fixed shape relative to ebp:

                 high addresses
            +-------------------+
            |       arg 2       |  ebp+12
            |       arg 1       |  ebp+8      <- arguments: positive offsets
            |  return address   |  ebp+4      <- pushed by 'call'
  ebp ----> |   saved ebp       |  ebp+0      <- pushed by the prologue
            |   local var A     |  ebp-4      <- locals: negative offsets
            |   local var B     |  ebp-8
            |       ...         |
            | outgoing args     |
  esp ----> |  (for callees)    |  esp+0, esp+4, ...
            +-------------------+
                 low addresses
  • Negative offsets from ebp are local variables.
  • Positive offsets from ebp are the saved frame pointer (+0), the return address (+4), and this function’s arguments (+8, +12, …).
  • Non-negative offsets from esp are the arguments this function is preparing to pass to the functions it calls.

The saved ebp values form a linked list through all active frames, which is how a debugger prints a backtrace. Compilers may omit the frame pointer (-fomit-frame-pointer, the default at -O1 and above on many targets) and address everything relative to esp; the labs compile without optimization, so you will see the classic layout.

A worked example

Here is the vulnerable function used throughout the next chapters:

#define SIZE 16

int vul(char *input)
{
    int flag = 0;
    char buf[SIZE];
    strcpy(buf, input);
    if (flag)
        printf("Flag was changed!\n");
    else
        printf("Flag was not changed. Try it again!\n");
    return 0;
}

Compile it for 32-bit with the protections that would get in the way of learning turned off (Chapter 6 explains each flag):

$ gcc -m32 -g -fno-stack-protector -z execstack -no-pie -fno-pic -o stack vul.c
$ objdump -d --no-show-raw-insn stack | sed -n '/<vul>:/,/ret/p'

On the SEED Ubuntu 20.04 VM (GCC 9.4) the beginning of the output looks like this (exact offsets depend on the compiler version and flags, so always check your own binary):

 <vul>:
   endbr32                          # CET landing pad (Chapter 8); a no-op otherwise
   push   %ebp
   mov    %esp,%ebp
   sub    $0x28,%esp                # reserve space for locals + outgoing args
   movl   $0x0,-0xc(%ebp)           # flag = 0               (flag at ebp-0xc)
   sub    $0x8,%esp                 # keep esp 16-byte aligned for the call
   pushl  0x8(%ebp)                 # push arg 2 of strcpy: input  (our arg 1)
   lea    -0x1c(%ebp),%eax          # eax = &buf             (buf at ebp-0x1c)
   push   %eax                      # push arg 1 of strcpy: &buf
   call   strcpy
   add    $0x10,%esp                # caller pops strcpy's arguments
   cmpl   $0x0,-0xc(%ebp)           # if (flag) ...
   je     ...

From this we can draw the frame at the moment strcpy is called:

                  high addresses
              +--------------------+
   ebp+8      |   input (char *)   |  vul's argument
   ebp+4      |   return address   |  where vul returns in main
   ebp  ----> |    saved ebp       |
   ebp-4      |    (padding)       |
   ebp-8      |    (padding)       |
   ebp-0xc    |       flag         |  4 bytes: ebp-0xc .. ebp-0x9
   ebp-0x10   |                    |
   ebp-0x14   |      buf[16]       |  16 bytes: ebp-0x1c .. ebp-0xd
   ebp-0x18   |                    |
   ebp-0x1c   |                    |  <- &buf
              |       ...          |
              |       input        |  strcpy's 2nd argument (src)
   esp  ----> |       &buf         |  strcpy's 1st argument (dest)
              +--------------------+
                  low addresses

Read the picture the way an attacker would. strcpy writes upward from &buf (ebp-0x1c) toward higher addresses until it copies a zero byte. The 16 bytes of buf are followed immediately by flag, then padding and the saved ebp, then the return address. The distance from the start of buf to the return address is 0x1c + 4 = 32 bytes. An input longer than 16 bytes changes flag; an input of 32 bytes followed by four more bytes replaces the return address. Chapter 4 does exactly that.

Note. Do not assume locals appear in the order they were declared, or that buf is exactly SIZE bytes from anything. Compilers reorder locals, add alignment padding, and (with stack protection on) deliberately place arrays above scalars. Always read the disassembly or measure in a debugger.

64-bit differences

The System V AMD64 ABI, used by Linux on x86-64, changes the picture in ways that matter for later chapters:

  • The first six integer/pointer arguments are passed in registers: rdi, rsi, rdx, rcx, r8, r9. Only further arguments go on the stack. (For system calls, the fourth argument uses r10 instead of rcx.) So to call system("/bin/sh") in a 64-bit exploit, you must get the string’s address into rdi — you cannot just place it on the stack (Chapter 7).
  • Return values are in rax. Callee-saved registers are rbx, rbp, r12–r15.
  • Stack slots and addresses are 8 bytes. User-space addresses fit in 47 bits, so they look like 0x00007ffff7a52000 — the top two bytes are zero. A strcpy-based overflow stops at the first zero byte, which makes writing a full 64-bit address harder.
  • The stack must be 16-byte aligned at each call; some libc functions crash (typically on a movaps instruction) if it is not.
  • Leaf functions may use the 128-byte red zone below rsp without moving rsp.

3.8 GDB and pwndbg

The debugger is the most important tool in this part of the course. GDB, the GNU Debugger, lets you stop a program, step one instruction at a time, and inspect every register and byte of memory. Plain GDB is spartan; pwndbg is a GDB plug-in for exploit development and reverse engineering that shows registers, disassembly, the stack, and a backtrace every time the program stops.

Install it:

$ git clone https://github.com/pwndbg/pwndbg ~/.pwndbg
$ cd ~/.pwndbg && ./setup.sh

Running a program under GDB

Two equivalent ways to pass command-line arguments:

$ gdb --args ./stack aaaaaaaa
pwndbg> run
$ gdb ./stack
pwndbg> run aaaaaaaa

To feed standard input from a file, use run < input.bin. To pass binary arguments, use shell command substitution inside GDB: run $(python3 -c '...').

Essential commands

CommandMeaning
break vul / b *0x080491c2breakpoint at a function / at an address
run [args] / rstart the program
continue / cresume until the next breakpoint
next / n, step / snext source line, stepping over / into calls
nexti / ni, stepi / sinext instruction, over / into calls
finishrun until the current function returns
disassemble vuldisassemble a function
info registers / i rshow registers
info frameshow the current frame, including saved eip and ebp locations
btbacktrace (walks the saved-ebp chain)
p &buf, p $ebp+4print an expression
x/16wx $espexamine memory: 16 words in hex starting at esp
x/s 0x0804a008examine memory as a C string
x/5i $eipexamine memory as 5 instructions

The x (examine) command takes a count, a format (x hex, d decimal, s string, i instruction), and a unit size (b byte, h halfword, w 4-byte word, g 8-byte giant). Note that GDB’s “word” is 4 bytes, unlike Intel’s.

pwndbg adds commands such as context (redraw the full display), stack 20 or telescope $esp 20 (show stack slots and follow pointers), vmmap (show the memory map), checksec (show which protections a binary has; Chapter 6), and cyclic (generate patterns to find offsets; Chapter 4).

A session

$ gdb --args ./stack AAAAAAAAAAAAAAAA
pwndbg> break vul
Breakpoint 1 at 0x80491b6: file vul.c, line 5.
pwndbg> run
...
pwndbg> p &buf
$1 = (char (*)[16]) 0xffffcf9c
pwndbg> p $ebp
$2 = (void *) 0xffffcfb8
pwndbg> p/d (char*)$ebp + 4 - (char*)&buf
$3 = 32
pwndbg> info frame
Stack level 0, frame at 0xffffcfc0:
 eip = 0x80491b6 in vul (vul.c:5); saved eip = 0x8049232
 ...
 Saved registers:
  ebx at 0xffffcfb4, ebp at 0xffffcfb8, eip at 0xffffcfbc
pwndbg> next              # execute strcpy
pwndbg> x/12wx &buf
0xffffcf9c:  0x41414141  0x41414141  0x41414141  0x41414141
0xffffcfac:  0x00000000  ...         ...         ...

The arithmetic $ebp + 4 - &buf is the most useful single expression in this part of the course: it is the number of bytes from the start of the buffer to the saved return address. Here it confirms the 32 we derived from the disassembly. The 0x41414141 words are our As (ASCII 0x41); the 0x00000000 right after is flag, whose low byte was just overwritten by the string’s terminating zero — harmless here, because flag was already zero.

Note. Addresses inside GDB can differ by a few bytes from addresses when the same program runs outside GDB, because GDB sets environment variables such as LINES and COLUMNS and runs the program by its absolute path; both change how much data sits at the top of the stack (Section 2.9). Exploits that hard-code stack addresses must account for this — one reason for the NOP sled in Chapter 5.

3.9 Generating Input with Python

Exploit inputs contain arbitrary bytes, including non-printable ones. You cannot type \xa6\x91\x04\x08 at the keyboard, so you generate input with a short program. Python 3 is the usual choice:

# 32 'A's as a command-line argument
$ ./stack $(python3 -c "print('A'*32)")

# raw bytes: 32 'A's followed by a little-endian address
$ ./stack $(python3 -c "import sys; sys.stdout.buffer.write(b'A'*32 + b'\xa6\x91\x04\x08')")

# the same input on standard input, for programs that use gets()/read()
$ python3 -c "import sys; sys.stdout.buffer.write(b'A'*32 + b'\xa6\x91\x04\x08')" | ./prog

Two traps:

  • Do not use print() for raw bytes. In Python 3, print('\xa6') writes the character U+00A6, which is UTF-8 encoded as the two bytes c2 a6. Use sys.stdout.buffer.write() with a bytes object (b'...') instead.
  • Command-line arguments cannot contain zero bytes (C strings end at the first \x00), and unquoted $(...) output is split at whitespace bytes (\x09, \x0a, \x20). Quote the substitution, "$(python3 ...)", when your payload might contain spaces or tabs.

For anything longer than a line, write a script that builds the payload in a bytearray and writes it to a file; the labs give you skeletons in this style. The pwntools library (pip3 install pwntools) provides helpers such as p32(0x080491a6) for packing addresses and cyclic(100) for offset-finding patterns.

3.10 The Layout of a Process in Memory

Finally, where does everything live? A running 32-bit Linux process has an address space like this:

 0xffffffff  +---------------------------+
             |  kernel space             |  (not accessible from user mode)
             +---------------------------+  ~0xffffe000 for 32-bit processes on a 64-bit kernel
             |  stack                    |  argv, envp, frames; grows down
             |            |              |
             |            v              |
             |                           |
             |  shared libraries         |  libc, ld.so  (e.g. 0xf7xxxxxx)
             |                           |
             |            ^              |
             |            |              |
             |  heap                     |  malloc(); grows up
             +---------------------------+
             |  .bss                     |  uninitialized globals (zeroed)
             |  .data                    |  initialized globals
             |  .rodata                  |  string literals, constants
             |  .text                    |  program code  (0x08048000 for 32-bit non-PIE)
 0x00000000  +---------------------------+

You can see the real map of any process in /proc/<pid>/maps, or with vmmap in pwndbg:

$ cat /proc/self/maps        # the map of the 'cat' process itself

Each line shows an address range, its permissions (r, w, x, and p for private), and what is mapped there. Notice that, on a modern system, the stack and heap are mapped rw-p — readable and writable, but not executable. Chapter 6 explains why; the labs turn this protection off with -z execstack so that you can first see the attack it prevents.

Also notice that if you run cat /proc/self/maps twice, the stack, heap, and library addresses change. That is Address Space Layout Randomization (ASLR), also covered in Chapter 6. For the early exercises we disable it so that addresses are predictable:

$ sudo sysctl -w kernel.randomize_va_space=0     # 2 restores the default

With that, you have everything you need: you can read the function, draw its frame, predict what an overlong input will overwrite, and verify the prediction in a debugger. The next chapter puts it to use.

Check Your Understanding

1. On x86, the 4-byte value 0xdeadbeef is stored at address 0x2000. What byte is at address 0x2000? At 0x2003?

Answer

x86 is little endian, so the least significant byte comes first: 0x2000 holds 0xef, and 0x2003 holds 0xde. The bytes in memory are ef be ad de.

2. Translate mov 0x8(%ebp), %eax into Intel syntax and explain what it does.

Answer

mov eax, DWORD PTR [ebp+0x8]. It loads the 4-byte value at address ebp+8 into eax. In a cdecl frame, ebp+8 is the function’s first argument.

3. What is the difference between lea -0x1c(%ebp), %eax and mov -0x1c(%ebp), %eax?

Answer

lea puts the address ebp-0x1c into eax (for example, &buf). mov loads the 4 bytes stored at that address into eax (for example, the first four characters of buf).

4. Describe precisely what call 0x8049186 and ret do to esp, eip, and memory.

Answer

call: esp := esp - 4; the address of the instruction after the call is stored at the new (%esp); eip := 0x8049186. ret: eip := (%esp) (the 4 bytes at the top of the stack); esp := esp + 4. ret does not verify that the value it loads was written by a call.

5. In a 32-bit cdecl function with a standard prologue, a local buffer starts at ebp-0x48. How many bytes of input must a strcpy write before it begins to overwrite the saved return address? What about the saved ebp?

Answer

The saved ebp is at ebp+0, so it starts 0x48 = 72 bytes after the buffer start. The return address is at ebp+4, so it starts 0x48 + 4 = 76 bytes after the buffer start: bytes 76–79 of the input overwrite it. (76 is the offset you will see in the s2CTF exercises of Chapter 4.)

6. Why is the following command wrong for passing the address 0x080491a6, and how do you fix it? ./prog $(python3 -c "print('A'*32 + '\xa6\x91\x04\x08')")

Answer

Python 3 print writes text encoded as UTF-8, so '\xa6' and '\x91' (both above 0x7f) each become two bytes (c2 a6, c2 91), corrupting the address and shifting everything after it. Write raw bytes instead: python3 -c "import sys; sys.stdout.buffer.write(b'A'*32 + b'\xa6\x91\x04\x08')".

7. In a 64-bit program, how is the first argument to system() passed? Why does that matter to an attacker who controls only stack memory?

Answer

In register rdi (System V AMD64 ABI). An attacker who can only write to the stack cannot simply place the argument next to a fake return address, as in 32-bit cdecl; they must first execute some existing code that loads rdi from the stack (such as a pop %rdi; ret gadget) — the idea behind return-oriented programming in Chapter 7.

Summary

  • x86 is little endian: the least significant byte of a value is at the lowest address. Exploit payloads write addresses in that byte order.
  • esp is the top of the stack, ebp the base of the current frame, and eip the next instruction. Controlling eip means controlling the program; it changes only through jmp, jcc, call, and ret.
  • AT&T syntax is op src, dst with % and $; Intel syntax is op dst, src. GNU tools default to AT&T, pwndbg to Intel.
  • The stack grows down. call pushes a return address; ret pops into eip and trusts whatever it finds.
  • In a 32-bit cdecl frame: arguments at ebp+8 and up, return address at ebp+4, saved ebp at ebp, locals below. A buffer at ebp-N is N+4 bytes from the return address. In x86-64, the first six arguments travel in registers and addresses contain zero bytes.
  • GDB with pwndbg lets you verify every one of these facts on your own binary; the expression $ebp + 4 - &buf gives the offset to the return address.
  • Generate binary input with sys.stdout.buffer.write(), never print().
  • A process’s address space contains code, data, heap, shared libraries, and stack. Modern systems make data regions non-executable and randomize their locations.

Connection to the Labs

This chapter has no lab of its own; it is the toolkit for everything that follows. Before starting Lab 2: Return-to-libc Attack, read its appendix on the function-call mechanism, which applies Section 3.7 to the lab’s retlib.c, and practice the GDB commands in Section 3.8 on the lab binary. The same skills are needed for Lab 3, where the stack you must reason about belongs to printf. The s2CTF playground, https://github.com/sysec-uic/s2ctf-playground, is a good place to practice the frame-drawing exercise from Section 3.7 on small binaries.

Further Reading

  • Intel Corporation. Intel 64 and IA-32 Architectures Software Developer’s Manuals, Volume 2 (instruction set reference). An unofficial HTML version of the instruction reference is at https://www.felixcloutier.com/x86/.
  • x86 Assembly Wikibook, “X86 Architecture”: https://en.wikibooks.org/wiki/X86_Assembly/X86_Architecture
  • Randal E. Bryant and David R. O’Hallaron. Computer Systems: A Programmer’s Perspective, 3rd edition, Chapter 3 (machine-level representation of programs).
  • pwndbg: https://github.com/pwndbg/pwndbg
  • System V Application Binary Interface, AMD64 Architecture Processor Supplement, for the 64-bit calling convention.