Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 1 of 33

Neural networks

0 of 33 complete

Why it matters

Most real inputs — pixels, tokens, tabular rows — have no useful closed-form mapping to a label. A neural network replaces hand-written rules with a parameterised function whose internal layers are adjusted from examples. The engineering problem is that the mapping is unknown, the data is noisy, and the only signal is a scalar loss, so the architecture must be expressive enough to represent the mapping while staying trainable by local gradient information.

Mental model

A network is a factory line where each station reshapes the material and the final station reports how wrong the product is. Training nudges every station's tooling by its share of the error. The analogy breaks because stations are not sequential specialists: one parameter affects every later output, so credit is shared across the whole line rather than assigned locally.

How it works

Two layers · composed, then gatedInteractive

hover or focus a cell

−2.00+2.00midpoint 0
layer output ↓layer input →
x₁x₂x₃
y₁
y₂
y₃

Hover or focus a cell to read its exact value. Shade and printed number encode the same quantity.

Step 1 of 4

Layer 1 · W₁

Each row is one hidden unit and each column one input. The forward pass computes h = W₁x, so a row and a column meet in one multiply-add; the grid is the whole first layer's mixing, not one example's activations.

Next: Layer 2 · W₂

Step 1/4 · Layer 1 · W₁

Figure. Two 3 × 3 layers with a ReLU between them, illustrative weights. Rows are each layer's output units and columns its input units, so the hidden layer is W₁'s output row and W₂'s input column. The first two frames print W₁ and W₂, the third their product W₂W₁: with no nonlinearity the stack is one 3 × 3 map, and nine effective weights describe both layers. The last frame prints the derivative at one input, x = (1, 0, 1), where ReLU switches two hidden units off, so the local derivative becomes a rank-one outer product. Shade is one signed scale from −2 to +2 across all four frames.

What to notice. The product is the arithmetic the prose cannot show. The top-left cell is 0.5×0.8 + 0.9×(−0.5) + (−0.3)×0.2 = −0.11: one output cell sums one path through each hidden unit. The gated frame is a local derivative, not a second architecture. At x = (1, 0, 1) the hidden pre-activations are 1.4, −0.2 and −0.7, so only the first unit passes ReLU and every path through the other two vanishes; at a different input a different set survives. That input dependence is exactly what a linear stack cannot do, and it is why depth alone adds no representational power without the nonlinearity.

Text version

Four 3 × 3 matrices. Rows are each layer's output units and columns its input units: in W₁ the rows are the hidden units, in W₂ the columns are. The first frame prints W₁: 0.80, −0.40, 0.60; −0.50, 1.10, 0.30; 0.20, 0.70, −0.90. The second prints W₂: 0.50, 0.90, −0.30; −0.60, 0.40, 1.20; 0.80, −0.20, 0.50. The third prints their product: −0.11, 0.58, 0.84; −0.44, 1.52, −1.32; 0.84, −0.19, −0.03. The fourth prints the gated derivative at x = (1, 0, 1): 0.40, −0.20, 0.30; −0.48, 0.24, −0.36; 0.64, −0.32, 0.48. Shade runs teal below zero through zero to accent above, on one scale from −2 to +2. Steps highlight layer one, layer two, the composed top-left cell and the gated frame; the readout reports the layer output, the layer input and the printed weight.

  1. Represent one example as a numeric array. a feature vector or a sequence of token ids — and fix the input shape the first layer expects.
  2. Compose a layer as a matrix multiply that mixes channels, a bias that shifts the result, and a nonlinearity that stops consecutive matrices from merging.
  3. Turn the prediction and the label into one scalar with a loss: squared error for a continuous target, cross-entropy for class probabilities.
  4. Run the forward pass, then backpropagate through the same graph to obtain a gradient for every weight and bias.
  5. Apply an optimiser step, evaluate on data the model never trained on, and stop when that held-out loss stops improving.
  6. Add width or depth only after a small model underfits, because each layer adds parameters, activation memory and inference latency.

Trade-offs

ChoiceGood forCostFails when
Deep narrow stackCompositional features with fewer parameters per layerVanishing gradients, delicate initialisationNo residual path or normalisation to carry signal back
Wide single hidden layerSmooth approximation, simple to trainParameter count grows quickly with input sizeThe target needs hierarchical composition
ReLU-family activationCheap, sparse gradients, fast convergenceDead units and unbounded pre-activationsLearning rate is high or inputs are unnormalised
Small network on more dataBetter generalisation per parameterMay underfit a genuinely complex mappingInputs are high-dimensional and labels are scarce
Trade-off mapInteractive

Good for

Compositional features with fewer parameters per layer

Cost

Vanishing gradients, delicate initialisation

Fails when

No residual path or normalisation to carry signal back

  1. No residual path or normalisation to carry signal back
  2. The target needs hierarchical composition
  3. Learning rate is high or inputs are unnormalised
  4. Inputs are high-dimensional and labels are scarce

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Deep narrow stack is good for Compositional features with fewer parameters per layer, costs Vanishing gradients, delicate initialisation, and fails when No residual path or normalisation to carry signal back. Wide single hidden layer is good for Smooth approximation, simple to train, costs Parameter count grows quickly with input size, and fails when The target needs hierarchical composition. ReLU-family activation is good for Cheap, sparse gradients, fast convergence, costs Dead units and unbounded pre-activations, and fails when Learning rate is high or inputs are unnormalised. Small network on more data is good for Better generalisation per parameter, costs May underfit a genuinely complex mapping, and fails when Inputs are high-dimensional and labels are scarce

Failure modes

  • Loss pinned at chance. The stack learns nothing because a layer saturates or gradients vanish before reaching the first weights. Debug: print per-layer gradient norms after one step; a near-zero norm at the input layer localises the break.
  • Memorisation instead of fitting. Training loss keeps falling while held-out loss climbs, so the network stores examples rather than the mapping. Debug: plot both curves on one axis and note whether the gap opens before or after augmentation is disabled.
  • Silent feature reordering. Columns arrive in a different order than at training time and predictions degrade with no error raised. Debug: assert feature names and dtypes at the model boundary rather than trusting tensor shapes alone.

In production

Log loss and gradient norm per step, not only per epoch, and keep a frozen evaluation batch so runs stay comparable. Size the network against the latency budget before tuning architecture, since depth is the hardest thing to remove later.

Common misconceptions

People think: More layers always mean a better model.

Actually: Depth adds representational power only when gradients reach the early layers; without residual paths and normalisation, extra depth trains worse than a shallow model.

People think: A neural network discovers features from raw data with no assumptions.

Actually: Architecture, preprocessing and loss encode strong assumptions about which invariances matter, and those choices usually decide success more than the optimiser does.

Interview angle

A strong answer frames the network as a parameterised function fitted by a scalar loss, then separates capacity, conditioning and generalisation: depth and width, initialisation and normalisation, data and regularisation. It should explain why the nonlinearity is required and what backpropagation actually computes. Weak answers list layer names without saying what degrades when depth grows or why a linear stack is useless.

Sources