Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 1 of 24

Python for AI

0 of 24 complete

Why it matters

Python dominates AI work for one reason: the ecosystem. Training frameworks, tokenisers, vector clients, evaluation libraries and orchestration tools all speak it first. That does not mean the language is incidental. The code around a model is where most defects live, and Python's flexibility makes those defects easy to write and hard to see.

A model call returns text. Text has no schema, no invariants and no error type. In a notebook you eyeball it. In a service, that text flows into a database, a UI, a billing calculation and sometimes an automated action. The engineering job is to narrow the type at each boundary so that a malformed response fails in one predictable place instead of corrupting six downstream assumptions.

Mental model

Python is both the lab bench and the delivery van. On the bench you want speed of thought: ad-hoc exploration, printing shapes, rewriting cells. On the road you want a vehicle with brakes, mirrors and a manifest. Same engine, incompatible habits.

The analogy breaks because the two modes share one file system and often one repository. There is no physical separation forcing you to switch, which is why reproducibility has to be enforced by tooling — lockfiles, tests, containers — rather than by discipline alone.

How it works

Mechanism, step by stepInteractive
  1. 1Declare types at every boundary. Function signatures, request and response models, and configuration objects. Type hints do not enforce anything at runtime, so pair them with a validator where data enters from outside: HTTP bodies, model output, files, environment variables.
  2. 2Validate model output rather than trusting it. Parse into a model with required fields and constraints. On failure, raise or return a typed error — never a partially populated dict. Every downstream consumer then has one shape to handle.
  3. 3Use async for I/O, threads for blocking calls. Model calls, database queries and HTTP requests are waits, not computations. One event loop can hold many in flight. A blocking library (a synchronous SDK, time.sleep, heavy CPU work) has to go to a worker thread or it stalls every request on the loop.
  4. 4Stream with iterators, not string concatenation. Expose token or chunk streams as generators so the HTTP layer can forward them as they arrive and the caller can cancel by closing the iterator.
  5. 5Pin the environment. A virtual environment per project plus a lockfile that records exact versions, including transitive ones. pyproject.toml declares intent; the lockfile makes it reproducible. Containerise when the deployment target differs from the developer machine.
  6. 6Test without the network. Unit tests replace model clients with fakes, assert on the assembled request, and check parsing of recorded responses. A separate, explicitly tagged suite hits the real provider.

Step 1 of 6

Declare types at every boundary. Function signatures, request and response models, and configuration objects. Type hints do not enforce anything at runtime, so pair them with a validator where data enters from outside: HTTP bodies, model output, files, environment variables.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Declare types at every boundary. Function signatures, request and response models, and co… 2. Validate model output rather than trusting it. Parse into a model with required fields an… 3. Use async for I/O, threads for blocking calls. Model calls, database queries and HTTP req… 4. Stream with iterators, not string concatenation. Expose token or chunk streams as generat… 5. Pin the environment. A virtual environment per project plus a lockfile that records exact… 6. Test without the network. Unit tests replace model clients with fakes, assert on the asse…

The subset of Python that carries weight in AI systems is small and stable.

  1. Declare types at every boundary. Function signatures, request and response models, and configuration objects. Type hints do not enforce anything at runtime, so pair them with a validator where data enters from outside: HTTP bodies, model output, files, environment variables.
  2. Validate model output rather than trusting it. Parse into a model with required fields and constraints. On failure, raise or return a typed error — never a partially populated dict. Every downstream consumer then has one shape to handle.
  3. Use async for I/O, threads for blocking calls. Model calls, database queries and HTTP requests are waits, not computations. One event loop can hold many in flight. A blocking library (a synchronous SDK, time.sleep, heavy CPU work) has to go to a worker thread or it stalls every request on the loop.
  4. Stream with iterators, not string concatenation. Expose token or chunk streams as generators so the HTTP layer can forward them as they arrive and the caller can cancel by closing the iterator.
  5. Pin the environment. A virtual environment per project plus a lockfile that records exact versions, including transitive ones. pyproject.toml declares intent; the lockfile makes it reproducible. Containerise when the deployment target differs from the developer machine.
  6. Test without the network. Unit tests replace model clients with fakes, assert on the assembled request, and check parsing of recorded responses. A separate, explicitly tagged suite hits the real provider.
  7. Keep configuration out of code. Keys, model names, base URLs and limits come from the environment or a secret store, never from a literal in a module that gets committed.

Validating a model response

from pydantic import BaseModel, Field, ValidationError
 
class Citation(BaseModel):
    doc_id: str = Field(min_length=1)
    quote: str = Field(min_length=1)
 
class Answer(BaseModel):
    text: str
    citations: list[Citation]
 
def parse_answer(raw: str) -> Answer:
    """Raise a typed error instead of returning a half-valid object."""
    try:
        return Answer.model_validate_json(raw)
    except ValidationError as exc:
        raise InvalidModelOutput(exc.error_count()) from exc

Two properties matter here. The parse is total — either you get an Answer or an exception, never a dict with missing keys. And the failure is local: the caller decides whether to retry, repair or abstain, instead of discovering the problem during rendering. Pydantic v2 exposes model_validate_json and error_count() for exactly this use.

Trade-offs

ChoiceGood forCostFails when
Notebook for explorationFast iteration, visible outputHidden state, no reproducibilityThe notebook becomes the deployment
Typed models at boundariesPredictable failures, safe refactorsBoilerplate and mapping codeThe schema changes every week and nobody owns it
Async throughoutHigh concurrency on I/O-bound workHarder debugging, blocking calls become outagesThe workload is CPU-bound or libraries are sync-only
Sync with worker threadsSimple code, sync SDKsThread pools and context switchingConcurrency requirements outgrow the pool
Thin dependenciesSmall images, fast cold starts, fewer CVEsRebuilding utilities you could importA library genuinely solves a hard problem
Heavy frameworkBatteries-included orchestrationUpgrade churn, opaque control flowYou need to debug inside the abstraction
Trade-off mapInteractive

Good for

Fast iteration, visible output

Cost

Hidden state, no reproducibility

Fails when

The notebook becomes the deployment

  1. The notebook becomes the deployment
  2. The schema changes every week and nobody owns it
  3. The workload is CPU-bound or libraries are sync-only
  4. Concurrency requirements outgrow the pool
  5. A library genuinely solves a hard problem

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Notebook for exploration is good for Fast iteration, visible output, costs Hidden state, no reproducibility, and fails when The notebook becomes the deployment. Typed models at boundaries is good for Predictable failures, safe refactors, costs Boilerplate and mapping code, and fails when The schema changes every week and nobody owns it. Async throughout is good for High concurrency on I/O-bound work, costs Harder debugging, blocking calls become outages, and fails when The workload is CPU-bound or libraries are sync-only. Sync with worker threads is good for Simple code, sync SDKs, costs Thread pools and context switching, and fails when Concurrency requirements outgrow the pool. Thin dependencies is good for Small images, fast cold starts, fewer CVEs, costs Rebuilding utilities you could import, and fails when A library genuinely solves a hard problem

Failure modes

Unpinned dependencies. A minor release changes a tokeniser or client default and quality moves with no code change. Debug: diff the lockfile between deploys and bisect the dependency, not your prompt.

Blocking call inside async code. One synchronous HTTP client serialises every request on the loop. Debug: log event-loop lag; a p99 that tracks total concurrency rather than work done is the signature.

Notebook state as source of truth. Results cannot be reproduced because the variables came from cells run in an unknown order. Debug: restart the kernel and run top to bottom; if the result changes, the notebook was never the artefact.

Swallowed validation errors. A broad except around parsing returns None, and None becomes an empty answer in the UI. Debug: assert that every parse failure increments a counter with a reason label, and alert on the rate.

In production

Two operational notes dominate. First, memory: importing a large numerical stack costs tens to hundreds of megabytes before your code runs, which matters for cold starts and container limits. Second, process model: in a standard CPython build the GIL means CPU-heavy preprocessing does not scale by adding threads, so use processes for compute and a single event loop for I/O. Log with structured fields, set an explicit flush strategy so container logs are not lost on crash, and pin the interpreter version in the image as well as the dependencies.

Common misconceptions

People think: Type hints make Python type-safe.

Actually: Hints are checked by external tools, not at runtime. Data entering from a model or a network call needs an explicit validator; annotations alone will not stop a string where an integer belongs.

People think: Async makes code faster.

Actually: Async improves throughput for waiting. It does not speed up computation, and it makes blocking calls more damaging because they stall every concurrent request.

Interview angle

Practical rounds rarely test exotic language features. They test whether you validate untrusted data, whether you can structure a small service, and whether you notice that a synchronous call in an async handler is a production bug. Say the words "parse at the boundary" and show the failure path — that is the signal.

See practical coding questions for the format, and testing pyramid for the surrounding discipline.

Sources