When OpenAI released the GPT-3.5-turbo API in March 2023 at a price of $0.002 per 1,000 tokens, developer sign-ups surged by hundreds of thousands within days. The company's rate-limit infrastructure buckled under load. Startups building on the API — including Snap's My AI feature, which launched the same month — had to implement exponential back-off logic overnight simply to keep their products functional. The incident made one thing sharply clear: calling an AI API is not pressing a button; it is entering a contract with a system that has its own throughput, pricing, and failure modes.
An Application Programming Interface (API) is a defined contract between two software systems. One system exposes a set of endpoints — addressable URLs with specified input formats and output formats — and any other system that knows that contract can send requests and receive structured responses. REST APIs, which dominate the web, use standard HTTP verbs (GET, POST, PUT, DELETE) and return data in JSON.
AI APIs follow this same pattern but the payload is richer. Instead of querying a database record or triggering a webhook, you are sending natural language text (and increasingly images, audio, or documents) to a model that runs inference — a computationally expensive forward pass through billions of parameters — and returns generated text, embeddings, or structured data.
A typical call to a modern chat completion API (OpenAI, Anthropic, Google Gemini) involves three layers of structure:
The Authorization header carries your API key — a credential that identifies your account for billing and rate limiting. The messages array encodes the conversation history in turn order. The model sees the entire array on every call; there is no server-side memory between separate API requests. max_tokens caps spending; temperature controls output randomness (0 = deterministic, 1+ = creative).
The server returns a JSON object containing the generated text nested inside choices[0].message.content, plus a usage block reporting prompt tokens, completion tokens, and total tokens consumed. Your code must parse this JSON and handle cases where finish_reason equals length (truncated due to max_tokens) versus stop (natural end).
GitHub Copilot, which uses OpenAI APIs under the hood, sends the entire visible file buffer plus surrounding context on every keystroke-triggered completion. At scale — GitHub reported 1 million paid subscribers in June 2023 — this generated enormous token throughput, which is why GitHub negotiated enterprise-tier rate limits and a dedicated capacity arrangement with OpenAI rather than using the public API tier.
By default, AI API calls are synchronous: your client waits, sometimes 5–30 seconds for long outputs, before receiving any data. Most providers also support streaming via Server-Sent Events (SSE), where the model sends tokens as they are generated. Streaming reduces perceived latency dramatically — users see words appearing rather than a blank screen — but requires your code to handle chunked responses and partial JSON fragments. ChatGPT's own interface uses streaming for exactly this reason.
Before writing a single line of code using an AI API, confirm three things: your API key is stored in an environment variable (never hardcoded), you understand the model's token pricing and have set a billing cap, and you have written a handler for non-200 HTTP status codes (429 rate limit, 503 overload, 400 bad request). Skipping any of these produces applications that either leak credentials, generate surprise invoices, or fail silently in production.
You are building a small app that uses an AI API. Use this lab to get hands-on with API concepts: ask about structuring a request, interpreting a response object, handling a 429 rate-limit error, or calculating token costs. Engage in at least 3 exchanges to complete the lab.
In January 2023, security researchers at GitGuardian reported that over 10 million secrets were exposed in public GitHub repositories in 2022 — a 67 percent increase year over year. OpenAI API keys were among the most common credentials found. One widely circulated case involved a developer who accidentally committed a .env file containing an active OpenAI key to a public repository; within 72 hours, unauthorized parties had consumed over $3,000 in API credits. OpenAI's abuse team confirmed they detected anomalous usage and revoked the key, but by then the charges had accrued. The incident became a recurring teaching example in security documentation from both OpenAI and Anthropic.
AI API providers implement authentication via long-lived secret keys — typically 40–60 character random strings prefixed with a provider identifier (e.g., sk- for OpenAI). When you include this key in the Authorization header, the provider's gateway server validates it against its credential store, looks up your account, applies rate limits and quotas for that account tier, and routes the request to model infrastructure. There is no secondary factor by default; possession of the key is sufficient for full API access.
This design prioritizes developer convenience — machine-to-machine calls cannot do browser-based OAuth flows — but creates severe risk if keys are mishandled. Unlike a password, an API key does not require a username; it alone is sufficient to consume your quota and generate charges.
API keys are typically exposed through five failure modes, in roughly descending order of frequency:
The industry-standard approach separates secrets from code entirely. Environment variables — set in the deployment environment and read at runtime via os.environ (Python), process.env (Node.js), or equivalent — keep keys out of the codebase. Secret management services like AWS Secrets Manager, HashiCorp Vault, or Vercel environment variables extend this by encrypting secrets at rest and providing audit logs of access.
GitGuardian scanned over 1 billion new GitHub commits in 2022 and found that generic high-entropy strings (likely API keys) represented the single largest category of detected secrets at 45% of all exposures. They noted that even private repositories carry risk because employees leave companies, forks get made public, and repository visibility settings can be changed accidentally.
Several providers now offer project-scoped keys — credentials that can only access specific models, have per-month spending caps, and are revocable without affecting other keys on the same account. OpenAI introduced project API keys with configurable spending limits in late 2023. Anthropic allows key revocation and creation per workspace. Google Cloud's Vertex AI uses IAM-based service accounts with per-API-method permissions. The principle of least privilege — giving each key only the permissions it actually needs — limits blast radius when a key is compromised.
A leaked key should be revoked immediately, before investigating how it leaked. Most provider dashboards offer one-click revocation. After revocation, audit usage logs (available in the provider dashboard) for the period of suspected exposure to assess what was consumed. Then rotate: generate a new key, update all deployment environments, verify the service is working, and only then investigate the root cause. The common mistake is investigating first and revoking second — which extends the exposure window.
Add .env and *.env to your .gitignore before writing the first line of code in any project that uses API keys. Use a pre-commit hook (like git-secrets or truffleHog) that scans diffs for high-entropy strings before commits are made. Set a spending cap in the provider dashboard the day you generate your key — not after the first unexpected bill arrives.
You are a developer setting up an application that calls the OpenAI API. Your lab assistant will help you think through secure key management — storage, exposure risks, spending caps, and rotation procedures. Engage in at least 3 exchanges to complete the lab.
When Notion launched its AI writing assistant in private alpha in November 2022, demand exceeded every projection. The product required a live OpenAI API call for every AI action — summarize, fix grammar, continue writing — and the volume of concurrent users immediately collided with OpenAI's per-organization rate limits. Notion's engineering team publicly described implementing request queuing and exponential back-off within the first 72 hours of launch to prevent cascading 429 errors from crashing the feature entirely. They also negotiated an elevated rate-limit tier directly with OpenAI. The incident became a widely cited example in developer communities of why rate-limit handling is not optional scaffolding — it is a core feature of any production AI integration.
AI API providers enforce two principal rate limits simultaneously. Requests per minute (RPM) caps how many API calls your application can make in a 60-second rolling window. Tokens per minute (TPM) caps the total input plus output tokens across all those calls. Both limits apply concurrently — you can hit the TPM cap while still under the RPM cap if each of your requests is large.
Limits vary dramatically by account tier. OpenAI's free tier offered 3 RPM on GPT-4 as of 2024; a Tier 5 (enterprise) account gets 10,000 RPM and 2,000,000 TPM on GPT-4o. The practical implication: a prototype that works fine on a personal account can fail catastrophically the moment it faces real user traffic, because production volume pushes past free-tier limits within seconds.
The provider rejects your request because you have exceeded RPM or TPM. The response includes a Retry-After header indicating when you may send the next request. Your code must catch this and wait — not retry immediately in a tight loop, which makes the problem worse.
The provider is overloaded (distinct from your personal rate limit). This is a transient infrastructure issue. Exponential back-off — waiting 1s, then 2s, then 4s, then 8s before retrying — is the standard mitigation. Most provider client libraries implement this automatically.
Exponential back-off means doubling the wait time between retries: attempt 1 waits 1 second, attempt 2 waits 2 seconds, attempt 3 waits 4 seconds, and so on up to a maximum (typically 60–120 seconds). Jitter adds a small random component to prevent the thundering herd problem — if 1,000 clients all hit a rate limit simultaneously and all retry at exactly the same interval, they generate another synchronized spike. Adding a random offset (e.g., wait_time + random(0, 1) seconds) spreads the retry load.
Token pricing as of mid-2024 ranged from $0.15 per million tokens (GPT-4o mini) to $15 per million tokens (GPT-4o input). A 500-word document is roughly 650 tokens. At GPT-4o pricing, summarizing 10,000 such documents costs approximately $42.25 — affordable for a batch job, catastrophic if triggered inadvertently in a production loop.
The open-source library tiktoken (released by OpenAI) allows exact token counting before making a call, enabling cost estimation at request time. Set max_tokens conservatively for each use case. Use smaller, cheaper models (GPT-4o mini, Claude Haiku) for tasks that do not require full model capability — classification, formatting, short Q&A — and reserve large models for complex reasoning.
Multiple developers have publicly reported accidentally running infinite or near-infinite loops against AI APIs — typically a retry loop missing a success condition check, or a webhook that re-triggered itself. In one widely discussed 2023 incident on the OpenAI developer forum, a developer reported $47,000 in API charges accrued over a weekend from a misconfigured automation. OpenAI's soft spending limit alerts (configurable in the dashboard) existed but had not been set. The lesson: set a monthly spending hard limit on day one, not after the invoice arrives.
Application-level caching stores API responses for identical or near-identical inputs, serving them without a new API call. This is appropriate for static queries — e.g., a product description that 10,000 users will all request — but not for personalized or dynamic content. Anthropic introduced prompt caching in 2024, where repeated system prompts or large context blocks (minimum 1,024 tokens) are cached server-side for up to 5 minutes, reducing cost on those tokens by 90% for cache hits. OpenAI added a similar feature (Prompt Caching) for inputs over 1,024 tokens in late 2024, automatically discounting repeated prefixes by 50%.
Before deploying an AI feature to production, answer these four questions: What is the worst-case token consumption per user action? Have you set a monthly spending hard cap in the provider dashboard? Does your retry logic implement exponential back-off with jitter? Have you tested what happens to user experience when a 429 error occurs? If any answer is "not yet," that is a production blocker, not a nice-to-have.
Your application is scaling up and you're starting to hit rate limits. Work with the lab assistant to think through: exponential back-off implementation, estimating token costs for a feature, choosing between model tiers, and structuring requests efficiently. Engage in at least 3 exchanges to complete the lab.
In 2023, Stripe built an AI-powered documentation assistant that answered developer questions about its API. The team described in a public engineering post how their initial prototype — using a simple user message with no system prompt — produced answers that confidently cited non-existent Stripe API parameters and gave outdated deprecation dates. The solution was a layered approach: a system prompt that constrained the model to only information in the retrieved document chunks, an instruction to say "I don't know" rather than speculate, and a post-processing step that cross-referenced cited parameter names against the live API reference before displaying the answer. This architecture — prompt constraint plus output validation — became a template for Stripe's subsequent AI feature builds.
The system prompt is the developer's primary tool for shaping model behavior. It is processed before any user input and persists for the entire conversation. A well-engineered system prompt specifies: the model's role and persona, the scope of topics it should and should not address, the output format expected (plain text, JSON, markdown), the tone and length constraints, and explicit instructions for edge cases (what to do when uncertain, when asked about competitors, when asked for personal advice).
Weak system prompts — one sentence descriptions — produce unpredictable behavior in edge cases because the model fills unspecified behaviors with its training defaults, which may not match product requirements. Strong system prompts are often 200–800 words and are treated as a critical engineering artifact, version-controlled and reviewed like code.
Parsing free-text AI responses in production is fragile — natural language output can vary in structure between calls even with identical inputs. Two approaches enforce parseable output. JSON mode (available in GPT-4o, Claude with proper prompting) instructs the model to return only valid JSON. The response can then be parsed with standard JSON libraries without regex heuristics. Function calling (OpenAI) and tool use (Anthropic) go further: you define a JSON schema describing a function signature, and the model returns a structured object conforming to that schema — guaranteed parseable and type-validated.
Even with JSON mode or function calling, production code must validate model output before acting on it. Models can hallucinate field values, return null for required fields, or produce structurally valid but semantically wrong content. The engineering pattern is: parse → validate schema → validate business rules → act. For high-stakes outputs (medical, financial, legal), a second AI call asking the model to verify its own answer — or a rule-based cross-check against known data — provides an additional safety layer.
String outputs require similar care. Before displaying model-generated text to users: check length bounds (a runaway model can generate thousands of tokens if max_tokens is unset), sanitize for XSS if rendering HTML, and check for policy-violating content if your system prompt's guardrails could be bypassed by adversarial user inputs.
In September 2023, researchers demonstrated prompt injection attacks against Bing Chat (powered by GPT-4) where malicious instructions embedded in web pages the model was asked to summarize caused it to relay those instructions as if they were its own. Kevin Liu and others published public demonstrations where text hidden in white-on-white font in a webpage overrode the system prompt. Microsoft patched mitigations across multiple updates. The incident established prompt injection as a first-class security concern for any AI feature that processes user-controlled or external content.
AI API calls are slow by web standards: a GPT-4o call generating 500 tokens takes roughly 2–5 seconds on average, with occasional spikes to 20+ seconds during high-load periods. Production applications must set explicit HTTP timeouts (typically 30–60 seconds), implement graceful degradation — a non-AI fallback when the API is slow or unavailable — and use streaming where the UX requires responsiveness. Stripe, Notion, and GitHub Copilot all use streaming as a core architectural choice, not an optional enhancement, because synchronous waits of 5+ seconds violate user experience expectations established by instant web responses.
Traditional unit tests assert deterministic outputs, but AI outputs are probabilistic. The emerging practice uses eval suites — collections of representative inputs with expected output properties (not exact strings) — run against the model to detect prompt regressions. When you change a system prompt, run the eval suite and compare pass rates. Companies including Anthropic, OpenAI, and Brex have published internal evals frameworks. The open-source promptfoo library provides a structured way to define test cases and assertions against AI outputs as part of a CI/CD pipeline.
Treat your system prompt as production code: version control it, test changes against an eval suite before deploying, and document why each instruction exists. Use function calling or JSON mode for any output your code will parse programmatically — never parse free-text responses with regex in production. Set HTTP timeouts, implement streaming for user-facing features, and always have a graceful degradation path for when the AI API is unavailable.
You are building an AI feature for a real product. Work with the lab assistant to design effective system prompts, choose between free-text and structured JSON output, think through output validation logic, and plan for edge cases including prompt injection risks. Engage in at least 3 exchanges to complete the lab.