Chapter 1: The Security Mindset

Most of computer science is about making systems do what they are supposed to do. Security is about making sure they do nothing else — even when someone is working hard to make them misbehave. That one change of perspective, from “does it work?” to “how could it be made to fail?”, is what this chapter is about. Everything else in the book, from buffer overflows to public-key certificates, is an application of it.

Learning objectives. After this chapter you should be able to:

  • Explain what an adversary is and why security differs from ordinary correctness.
  • Write down a threat model for a system: who attacks, with what capabilities, and what they are after.
  • Classify a security requirement as confidentiality, integrity, or availability.
  • Name the main families of countermeasures and the costs each one carries.
  • Recognize a handful of classic security design principles when you see them applied (or violated) in real code.

1.1 Why Security Is Different

A program that sorts numbers is correct if it sorts every input correctly. Testing it against random inputs is a reasonable way to gain confidence, because bugs that hide behind one-in-a-billion inputs rarely matter: nature does not choose inputs to hurt you.

An attacker does. Security engineering assumes that somewhere out there is an adversary — an intelligence that actively tries to make the system misbehave and that gets to pick the worst possible input, the worst possible timing, and the worst possible environment. A bug that is triggered by one specially shaped 200-byte string is irrelevant to a sorting routine and catastrophic for a network server, because the adversary will find that string and send it.

This is why the field is fun. To defend a system you have to understand how it can be broken, so a large part of this course is spent breaking things: hijacking the control flow of a program, turning a printf call into an arbitrary memory write, opening a shell on a remote machine. The same skills are rewarded in Capture the Flag (CTF) competitions, in bug-bounty programs that pay for vulnerability reports, and in large-scale challenges such as DARPA’s Cyber Grand Challenge (2016), where fully automated systems found, exploited, and patched bugs with no human in the loop, and its successor, the AI Cyber Challenge (AIxCC), where LLM-driven systems did the same on real open-source software.

It is also why the field is hard. As the following chapters show, defending means closing every hole, while attacking means finding one.

1.2 The Systems We Secure

A modern computer is a stack of layers, each built on the one below and each trusting the one below:

   +---------------------------------------------------+
   |  Applications   (browser, server, your program)   |   application security
   +---------------------------------------------------+
   |  Runtime        (libc, dynamic linker, JVM, ...)  |
   +---------------------------------------------------+
   |  Operating system kernel                          |   OS security
   +---------------------------------------------------+
   |  Hardware       (CPU, memory, devices)            |   hardware security
   +---------------------------------------------------+
            \___________ network ___________/              network security

“All problems in computer science can be solved by another level of indirection.” — attributed to Butler Lampson

Layering is how we manage complexity, and it shapes security in two directions. A lower layer can protect the layers above it: the kernel enforces file permissions that no application can bypass, and the CPU enforces page permissions that the kernel relies on. But a lower layer is also part of everyone’s trusted computing base: a bug in the dynamic linker (Chapter 2) or a CPU side channel affects every program running on top, no matter how carefully that program was written.

Roughly, this book follows the stack. Part I starts with the boundary between users and privileged programs on a Linux system, then spends several chapters on the memory-safety bugs in C programs that let an attacker take over a process. Later parts cover cryptography, network security, OS and hardware security, and the security of — and security with — AI systems.

1.3 Adversaries and Threat Models

“Is this system secure?” has no answer by itself. Secure against whom, protecting what? A threat model makes those questions explicit. Writing one forces you to answer three groups of questions:

  1. Who are the attackers? What are their capabilities (can they run code on the machine, or only send network packets?), their access (insider or outsider, physical or remote?), and their motivation (money, espionage, vandalism, curiosity)?
  2. What are we trying to protect? Data, money, availability of a service, the integrity of a device, user privacy?
  3. What properties must hold? For example: “no app can read another app’s files” or “only the account owner can transfer money.”

A threat model is also a statement of what you are not defending against. A password manager may assume the operating system is not compromised; a smart-card design may explicitly consider attackers with oscilloscopes. Neither is wrong, but a system deployed outside its threat model is not secure, whatever its documentation says.

Example: a mobile operating system

Imagine you are on the security team for a mobile OS such as Android.

  • Potential attackers: external hackers sending malicious network traffic or messages; malicious app developers who publish apps that try to escape their sandbox; thieves who steal the physical phone; users with limited security awareness who can be tricked into granting permissions or installing apps from untrusted sources.
  • Assets: the user’s personal data (photos, messages, location, credentials), other apps’ private data, payment information, the integrity of the OS itself, and the device’s availability.
  • Required properties: apps are isolated from each other; an app gets only the permissions the user granted; the OS boots only signed code; a locked phone does not reveal its data to whoever holds it.

Notice that the threat model already suggests the architecture: per-app sandboxes, a permission system, verified boot, and storage encryption are each direct answers to one line above.

1.4 Security Objectives: Confidentiality, Integrity, Availability

Security requirements are traditionally sorted into three objectives, abbreviated CIA.

Confidentiality — prevent or detect improper disclosure of information.

  • Data confidentiality: sensitive information is available only to authorized parties.
  • Privacy: individuals control what information about them is collected and stored, and to whom it is disclosed.

Integrity — prevent or detect improper modification of information.

  • Data integrity: information and software can be changed only in a specified and authorized manner.
  • System integrity: the system performs its intended function, unimpaired by unauthorized manipulation.

Availability — prevent or detect improper denial of access to services that the system provides.

Most real requirements map cleanly onto one of the three:

RequirementObjective
You should not learn your classmates’ scores in this course.Confidentiality
You should not be able to change your score after the semester ends.Integrity
Your scores should always be viewable in the student portal.Availability

Attacks map the same way. Most of the memory-corruption attacks in the next chapters are integrity attacks first — the attacker overwrites a return address or a function pointer — that are then leveraged into confidentiality breaches (reading a secret key) or full system compromise. A program crash is an availability attack. Keep the mapping in mind: when you design a defense, you should be able to say which objective it protects.

Note. Other properties are often added to the triad: authenticity (you know who sent a message), non-repudiation (the sender cannot later deny sending it), and accountability (actions can be traced to whoever performed them). We will meet them in the cryptography chapters.

1.5 How We Achieve the Objectives

There is no single mechanism that “adds security.” Systems combine many countermeasures, each covering part of the threat model.

Technical countermeasures include:

  • Access control — the OS decides which user or process may read, write, or execute which resource (Chapter 2).
  • Memory-safety checks — compiler- and hardware-enforced defenses that make memory corruption harder to exploit (Chapters 6 and 8), or languages such as Rust that rule it out by construction.
  • Cryptography — encryption for confidentiality, MACs and signatures for integrity and authenticity.
  • Firewalls and intrusion detection — filter and monitor network traffic.
  • Malware scanners and software checking mechanisms such as code signing and static analysis.

Non-technical countermeasures are just as important: laws and regulations, institutional policies, procedures, user training, auditing, and incentives. A perfectly encrypted database does not help if an employee can be phoned up and talked into reading records aloud.

Why security is hard

If the mechanisms exist, why are systems still broken so often?

  • People do not perceive the value of security investment — until a failure occurs, such as a breach of stored credit-card numbers.
  • Security gets in the way. Users see security measures as obstacles to getting their work done, and they will route around them (sticky-note passwords, disabled warnings). Good mechanisms are transparent to normal use.
  • Retrofitting is difficult. Adding security to an existing system, such as a memory-safety check to a large C code base, is far harder than designing it in.

Security has a cost

No security mechanism is free. Direct costs include design, implementation, runtime overhead, and enforcement — including the cost of false positives that block legitimate users. Indirect costs include added complexity (itself a source of bugs) and lost productivity. Throughout the book you will see defenses that were known for years but deployed only once their overhead dropped below a few percent; Control-Flow Integrity (Chapter 8) is a prime example. Security is always a trade-off against performance, usability, and money.

1.6 Security Design Principles

Some ideas recur so often that they have names. Here are the ones we will use most.

  • Least privilege (separation of privilege). Every component should run with the minimum privilege it needs, for the minimum time. A program that needs root to open one file should open it and then drop root (Chapter 2).
  • Isolation and encapsulation. Keep components that do not need to interact apart, so a compromise of one does not spread. Processes, virtual machines, containers, and app sandboxes are all isolation mechanisms. One special case appears again and again in this book: keep code and data separate. Many attacks — command injection, SQL injection, cross-site scripting, and buffer-overflow code injection — happen because attacker-supplied data ends up being interpreted as code.
  • Usability (psychological acceptability). Mechanisms should be transparent to normal users and easy to use correctly; otherwise they will be bypassed.
  • Layering (defense in depth). Do not rely on a single mechanism. If a stack canary is bypassed, a non-executable stack should still stop the shellcode; if that is bypassed, ASLR should make the addresses unknown.
  • Fail-safe defaults. Base access decisions on explicit permission, not on exclusion. When in doubt, deny.
  • Economy of mechanism. Keep security-critical code small and simple enough that it can actually be reviewed and verified.

These principles come from a long tradition, most famously Saltzer and Schroeder’s 1975 paper listed at the end of this chapter. You will find it a useful exercise, whenever you read about an attack in later chapters, to name the principle that the victim violated.

1.7 Thinking Like an Attacker

A useful habit is to look at every program as an attacker would. Consider this fragment, which appears (in some form) in the next few chapters:

int check(char *input)
{
    char buf[16];
    strcpy(buf, input);      /* copies until the first '\0' byte */
    return 0;
}

A programmer reads it as “copy the input into a buffer.” An attacker asks different questions:

  • Who controls input? If it comes from a command-line argument, a file, or the network, the attacker does.
  • What happens if it is longer than 16 bytes? strcpy keeps writing past the end of buf into whatever lies next to it in memory.
  • What lies next to buf? On the stack, among other things, the address this function will return to.
  • Who runs this program, with what privilege? If it is a root-owned Set-UID program (Chapter 2), whatever the attacker achieves, they achieve as root.

Each of those questions turns out to have an answer that leads to a full exploit. That chain — attacker-controlled input → memory corruption → control of execution → privilege — is the backbone of the first half of this book.

A note on ethics. The techniques in this book are real attack techniques. Use them only inside your own virtual machine or containers, or against systems you have explicit written permission to test. Using them against university infrastructure or anyone else’s machine is a violation of university policy and, very likely, of the law.

Check Your Understanding

1. In your own words, what distinguishes a security bug from an ordinary bug?

Answer

An ordinary bug matters in proportion to how often natural inputs trigger it. A security bug matters if any input an adversary can supply triggers it, because the adversary will deliberately choose that input. Security is correctness against a worst-case, intelligent, adaptive opponent rather than against random inputs.

2. Classify each requirement as confidentiality (C), integrity (I), or availability (A): (a) a ransomware attack encrypts a hospital’s records; (b) an attacker changes the destination account of a wire transfer in transit; (c) a contractor copies a customer database onto a USB stick.

Answer

(a) A — the records still exist and were not disclosed (in the basic version of the attack), but the hospital can no longer access them. Modern “double extortion” ransomware also steals the data first, adding a C violation. (b) I — data was modified without authorization. (c) C — data was disclosed to an unauthorized party.

3. Write a short threat model (attackers, assets, required properties) for the course’s online grade book.

Answer

One reasonable answer. Attackers: students who want to change their own grade or see others’ grades; outside attackers who want student records (personal data); a careless or malicious TA account. Assets: grades and submissions, student identities and personal data, instructor credentials. Properties: only the instructor and TAs can modify grades, and every change is logged (integrity, accountability); a student sees only their own grades (confidentiality); the grade book is reachable during the term, especially around deadlines (availability). An explicit non-goal might be “we assume the university’s single-sign-on system is not compromised.”

4. A program runs as root, opens a protected log file, and then keeps running as root for the rest of its lifetime even though it never needs root again. Which design principle does it violate, and why does that matter?

Answer

Least privilege. If any later part of the program has a bug — for example, a buffer overflow in its input parser — the attacker who exploits it inherits root privilege. Had the program dropped privilege right after opening the file, the same bug would give the attacker only an ordinary user’s privilege. (Chapter 2 shows that dropping privilege has its own pitfalls: the open file descriptor can still leak.)

5. Command injection, SQL injection, and stack-based code injection look very different. What do they have in common?

Answer

All three let attacker-supplied data be interpreted as code: shell syntax in a command string, SQL syntax in a query string, or machine instructions in a stack buffer to which the program later jumps. They all violate the principle of keeping code and data separate, and the robust fixes all restore that separation (execve with an argument vector, parameterized queries, non-executable data pages).

Summary

  • Security is correctness in the presence of an adversary who chooses inputs, timing, and environment to make the system misbehave.
  • A threat model states who the attackers are, what they can do, what they want, and what properties the system must preserve — and, implicitly, what it does not defend against.
  • Security objectives are classified as confidentiality, integrity, and availability.
  • Countermeasures are technical (access control, memory safety, cryptography, firewalls) and non-technical (law, policy, training). All of them cost something, so security is always a trade-off.
  • Design principles such as least privilege, isolation (especially of code from data), defense in depth, fail-safe defaults, and economy of mechanism explain why attacks succeed and how defenses are built.

Further Reading

  • Jerome H. Saltzer and Michael D. Schroeder. “The Protection of Information in Computer Systems.” Proceedings of the IEEE, 63(9), 1975. The origin of most of the design principles above.
  • Wenliang Du. Computer & Internet Security: A Hands-on Approach, 3rd edition. The recommended textbook for this course.
  • CTF101, an introduction to Capture-the-Flag competitions: https://ctf101.org/
  • DARPA AI Cyber Challenge: https://aicyberchallenge.com/