Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 3 of 24

HTTP/REST/JSON

0 of 24 complete

Why it matters

Every hosted model provider exposes an HTTP API. So do the embedding service, the vector database, the tool gateway and the observability backend. When one of those calls fails at 2am, the failure is expressed in HTTP vocabulary: a status code, a header, a body that may or may not match the schema you expected.

Engineers who treat the protocol as boilerplate write clients that hang forever, retry a validation error four times, and parse an error page as if it were JSON. Engineers who understand it get retry behaviour, timeouts, caching and rate limiting almost for free, because the protocol already defines where those decisions live.

REST is the design style layered on top: resources addressed by URL, manipulated with a small set of methods, representations transferred as JSON. It is not a standard you can fail, and most model APIs are only loosely RESTful — a completion is a procedure, not a resource. That is fine. What matters is that the semantics of the method and status are not decorative.

Mental model

A postal service with receipts. The envelope carries addressing and handling instructions (headers), the contents are the payload (body), and the postmark tells you what happened (status code). You can tell a lot about a delivery from the postmark without opening anything.

The analogy breaks on repetition. Post a letter twice and two letters arrive. Send an HTTP request twice and the effect depends on the method: reading is harmless, creating an order is not. That is why the protocol separates safe methods from idempotent ones, and why the client — not the server — has to know which it is using.

How it works

429 waits, 422 stopsInteractive

One POST, one retry, one refusal. The gateway enforces auth, rate limits and the request shape; the provider produces the completion. The client reads status and content type before it parses anything.

Client

retry policy

Gateway

auth + limits

Provider

model API

Step 1/4 · send
1 · POST /v1/generatecall · Client → Gateway
POST /v1/generate HTTP/1.1
Host: api.example
Authorization: Bearer <token>
Content-Type: application/json
Accept: application/json
Idempotency-Key: req_7f3c

{ "input": "Draft a refund reply",
  "max_output_tokens": 512 }

Step 1: send. Hover, tap or tab to any message to inspect its payload.

Figure. The status line decides the client's next action: a 429 from the gateway is a wait that honours Retry-After and repeats the idempotency key, while a 422 is a stop that returns the failing pointer. The body matters only after the status and the content type, and the retried POST is safe only because the key repeats.

What to notice. The failure is retrying the wrong class: 429 is a 4xx that means later, while 422 means the request will fail the same way every time. The write is retryable only because the idempotency key repeats; a 200 with an HTML body is the other trap and is not drawn. Cache validators, streaming and connection reuse are omitted; ids, headers and limits are illustrative.

Text version

Sequence across Client, Gateway and Provider. Step 1, send: Client to Gateway POST /v1/generate with a bearer token, JSON content type, Accept application/json, Idempotency-Key req_7f3c, and a prompt plus max_output_tokens 512. Step 2, throttled: Gateway to Client 429 Too Many Requests with Retry-After 2, RateLimit-Limit 60, RateLimit-Remaining 0, RateLimit-Reset 2 and a problem+json body; an in-process Client row classifies 429 as retryable, waits max of Retry-After and 2 s, keeps the idempotency key and has two attempts left. Step 3, retry: Client to Gateway the same body and key after 2 s; Gateway to Provider the request with Authorization, Idempotency-Key and body unchanged, with no retry of its own; Provider to Gateway 200 with application/json, X-Request-Id and no-store, and a generated id; Gateway to Client 200 with the same X-Request-Id and RateLimit-Remaining 57. Step 4, rejected: Client to Gateway a new request with max_output_tokens as the string 512; Gateway to Client 422 Unprocessable Content with problem+json and an errors pointer /max_output_tokens expecting an integer; an in-process Client row stops, surfaces the pointer, and does not retry. The gateway enforces auth, rate limits and the request schema at the edge, so the invalid body never reaches the provider.

  1. Compose the request. A method (GET, POST, PUT, PATCH, DELETE), a URL, headers, and optionally a body. GET and HEAD are safe: they must not change state. PUT and DELETE are idempotent: repeating them has the same intended effect as sending once. POST is neither.
  2. Send it over a connection. HTTP/1.1 keeps connections alive between requests; HTTP/2 and HTTP/3 multiplex many requests over one connection. The practical consequence is that a client without a connection pool pays handshake cost on every call, which is visible in tail latency.
  3. Let the server classify the outcome. 2xx success, 3xx redirect or cache validation, 4xx the request cannot succeed as sent, 5xx the server failed to fulfil an apparently valid request. 429 is a 4xx that means "later", not "wrong".
  4. Read headers before parsing the body. content-type tells you whether the body is JSON or an HTML error page. retry-after tells you how long to wait on 429 or 503. Rate-limit headers describe remaining budget. etag enables conditional requests and 304 responses.
  5. Parse the representation defensively. JSON over the wire is untyped. Validate it into a schema at the edge, tolerate unknown fields, and fail on missing ones you depend on rather than on shape changes you do not.
  6. Return errors in a consistent shape. A machine-readable error type, a human message and a correlation ID. RFC 9457 standardises this as application/problem+json, which is worth adopting even internally so clients parse one error format everywhere.
  7. Decide the retry and timeout policy at the client. Per-attempt timeout, total deadline, which status codes are retryable, and whether the operation is idempotent. The server cannot make that decision for you.
Safe methods
GET, HEAD, OPTIONS, TRACE
Idempotent methods
Safe methods plus PUT and DELETE
Retryable by default
Connection errors, 429, most 5xx — if the operation is safe to repeat
Never retry blindly
400, 401, 403, 404, 422
Cost of no timeout
A held connection per stuck request, until the pool is empty

A client that behaves

import httpx
 
timeout = httpx.Timeout(30.0, connect=5.0)  # total and connect budgets differ
 
resp = httpx.post(
    f"{BASE_URL}/generate",
    headers={
        "authorization": f"Bearer {api_key}",
        "content-type": "application/json",
        "idempotency-key": request_id,          # only if the server honours it
    },
    json={"input": prompt, "max_output_tokens": 512},
    timeout=timeout,
)
 
if resp.status_code == 429:
    delay = float(resp.headers.get("retry-after", "1"))
    raise RateLimited(delay)
if resp.status_code >= 500:
    raise UpstreamError(resp.status_code, resp.headers.get("request-id"))
resp.raise_for_status()
data = resp.json()

The three lines that matter are the timeout, the 429 branch and the correlation ID. Everything else is schema work.

Trade-offs

InterfaceGood forCostFails when
REST + JSONBroad compatibility, easy debugging, cacheable readsVerbose payloads, no schema by defaultYou need strict contracts or high-throughput binary transport
gRPC + protobufTyped contracts, streaming, compact payloadsTooling, proxies, harder debugging by handClients are browsers or third parties without generated stubs
GraphQLClient-shaped queries over a graph of resourcesQuery complexity, caching difficultyThe data model is a single procedural call
Async job + polling or webhookLong-running work, provider-side queuesExtra state machine, delivery guaranteesThe user expects tokens on screen within a second
Server-sent streamingToken-by-token output over one responseDisconnect handling, proxy bufferingThe client needs bidirectional messages mid-stream
Trade-off mapInteractive

Good for

Broad compatibility, easy debugging, cacheable reads

Cost

Verbose payloads, no schema by default

Fails when

You need strict contracts or high-throughput binary transport

  1. You need strict contracts or high-throughput binary transport
  2. Clients are browsers or third parties without generated stubs
  3. The data model is a single procedural call
  4. The user expects tokens on screen within a second
  5. The client needs bidirectional messages mid-stream

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: REST + JSON is good for Broad compatibility, easy debugging, cacheable reads, costs Verbose payloads, no schema by default, and fails when You need strict contracts or high-throughput binary transport. gRPC + protobuf is good for Typed contracts, streaming, compact payloads, costs Tooling, proxies, harder debugging by hand, and fails when Clients are browsers or third parties without generated stubs. GraphQL is good for Client-shaped queries over a graph of resources, costs Query complexity, caching difficulty, and fails when The data model is a single procedural call. Async job + polling or webhook is good for Long-running work, provider-side queues, costs Extra state machine, delivery guarantees, and fails when The user expects tokens on screen within a second. Server-sent streaming is good for Token-by-token output over one response, costs Disconnect handling, proxy buffering, and fails when The client needs bidirectional messages mid-stream

Failure modes

Retrying a deterministic error. A 400 from a malformed JSON body is retried until the budget is gone. Debug: log status class with every attempt and alert when retryable-class retries exceed the number of failed requests.

No timeout, or one timeout for everything. The connect timeout is fine but the read timeout is infinite, so a stalled response pins a worker. Debug: inject a server that accepts and never responds; if the client does not give up, the timeout is not configured.

Ignoring Retry-After. Clients retry immediately and turn a provider's overload signal into a self-inflicted flood. Debug: compare retry timestamps against the header value in a captured trace.

Parsing errors as success. A gateway returns HTML with status 200, resp.json() throws deep inside rendering, and the user sees a blank page. Debug: assert content-type before parsing, and map every non-2xx to one internal error type.

In production

Track latency by status class and by endpoint, not just overall. A rising 429 rate is a capacity or budgeting problem; a rising 5xx rate is a provider problem; both look like "slow" on a dashboard that only shows p95. Set a total deadline per user request and pass the remaining budget down to each hop, keep a connection pool sized to your concurrency, and put a request ID in every call so a provider support ticket has something to match. Finally, distinguish "the call failed" from "the operation may have happened" — that distinction is the whole subject of idempotency.

Common misconceptions

People think: A 200 status means the request worked.

Actually: It means the server accepted and answered. The body can still be an error payload, a rate-limit notice, or a truncated response. Check content type and schema.

People think: POST is idempotent because I sent the same JSON.

Actually: Identical payloads are not the same request. Each POST may create a new record, charge a card or send an email. Only the server can make a POST safely repeatable, via an idempotency key.

People think: REST requires a specific URL shape.

Actually: REST is a set of constraints about stateless interaction and uniform interfaces, not a naming convention. A /generate endpoint is fine; what matters is method semantics and cache behaviour.

Interview angle

Practical and system design rounds probe this indirectly: "the provider returns 429 — what does your client do?" A strong answer names the timeout hierarchy, the retryable classes, the Retry-After honour, the idempotency requirement, and the metric that would have caught the problem. Weak answers reach for a retry decorator without asking whether the operation is safe to repeat.

See practical coding and production questions.

Sources