Landmark 1 stop to get here · leads to 35

Neural Network

A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

Your route here

1 stop · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network · you are here ✓ understood

Picture it

InputHiddenHiddenOutput
01/05
0:00 / 0:23

💡 For Software Engineers

Think of a neural network like a pipeline of function calls:

// Conceptual pseudocode
function neuralNetwork(input) {
  let h1 = relu(matmul(input, W1) + b1);
  let h2 = relu(matmul(h1, W2) + b2);
  let output = softmax(matmul(h2, W3) + b3);
  return output;
}

The weights (W1, W2, W3) and biases (b1, b2, b3) are the learnable parameters. Training means finding the values that minimize prediction errors.

Key Concepts

Neuron (Node)

A computation unit that receives inputs, multiplies by weights, adds a bias, applies an activation function, and outputs a value.

Layer

A collection of neurons that operate in parallel. Input layer receives data, hidden layers process it, output layer produces predictions.

Weights

Learnable parameters on connections. Higher weight = stronger influence. These are what "training" adjusts.

Activation Function

Non-linear function applied after weighted sum. Common ones: ReLU (max(0,x)), Sigmoid (0-1 output), Softmax (probabilities).

🔥 PyTorch Implementation

import torch
import torch.nn as nn

class SimpleNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        # Define layers
        self.layer1 = nn.Linear(input_size, hidden_size)
        self.layer2 = nn.Linear(hidden_size, hidden_size)
        self.layer3 = nn.Linear(hidden_size, output_size)
        self.relu = nn.ReLU()

    def forward(self, x):
        # Forward pass - data flows through layers
        x = self.relu(self.layer1(x))
        x = self.relu(self.layer2(x))
        x = self.layer3(x)  # No activation on output
        return x

# Create a network: 3 inputs → 4 hidden → 4 hidden → 2 outputs
model = SimpleNN(3, 4, 2)

nn.Linear creates a fully-connected layer with weights and biases. The forward method defines how data flows through the network.

What's Next?

Prerequisites Covered

You now understand: layers, neurons, weights, activation functions, forward pass

Leads To

Next: Loss Functions, Gradient Descent, Backpropagation - how networks actually learn

💡 Key Takeaways

  • 1.Neural networks are just matrix multiplications + non-linear functions, stacked in layers.
  • 2.Weights are the "memory" of the network - they encode what patterns matter for making predictions.
  • 3.Activation functions add non-linearity. Without them, any depth of layers would just collapse to a single linear transformation.
  • 4.The magic isn't in the architecture - it's in finding the right weights through training.

A neural network is a function built by stacking simple pieces. Each unit takes a weighted sum of its inputs, adds a bias, and passes the result through an activation function. Units are arranged in layers: the first reads the raw input, the last produces the prediction, and the hidden layers in between turn simple features into richer ones.

The weights are what the network learns. Training nudges millions or billions of them so the output moves closer to the right answer, using backpropagation to work out which way each weight should move. The name is only loosely borrowed from neuroscience; modern networks are guided by math and engineering, not by how brains work.

Why layers and nonlinearity matter

Without the activation function, stacking layers gains nothing: a chain of weighted sums is still one weighted sum. The nonlinearity between layers is what lets depth build up curved, intricate decision boundaries.

That’s also the idea behind the universal approximation theorem: a network with even one hidden layer can, in principle, approximate a very wide class of functions if it’s wide enough. The textbook’s caveat is the important part: the theorem promises such a network exists, not that training will find it, or how large it must be. In practice, depth tends to be the more efficient way to get there.

From perceptron to today

Frank Rosenblatt’s perceptron (1958) was a single trainable unit. Training networks with hidden layers became practical once backpropagation was popularized in 1986. Today’s architectures are specialized arrangements of the same parts: a CNN shares weights across an image, and a transformer lets every token attend to every other.

The catch

A trained network is a very good pattern-matcher over the data it saw, and it offers no built-in guarantee outside that range. Its knowledge is spread across millions of numbers, which makes it hard to inspect, and big networks need a lot of data and compute to train. Stacking many layers is what’s called deep learning.

Where it sits

Before this

Machine Learning
Neural Network

Leads to

Activation Function Attention Mechanism Backpropagation Convolutional Neural Network Deep Learning Diffusion Model

+ 29 more

Explore nearby

Neural Networks Deep Learning A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data. Neural Networks Activation Function A non-linear function applied to neuron outputs that introduces non-linearity, enabling networks to learn complex patterns. Training Backpropagation The algorithm for computing gradients of the loss with respect to network weights, enabling training through gradient descent. Neural Networks Layer A collection of neurons/operations that process data together, neural networks are composed of stacked layers. Neural Networks Weight Learnable parameters connecting neurons in neural networks, determining the strength of connections.

In the research

All papers →

6 papers that build on Neural Network ; showing 5, canon first.

Canon · 1958 The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain It is the first learning neural network: the idea that weights, not rules, should hold the knowledge starts here. Canon · 1986 Learning Representations by Back-Propagating Errors Backpropagation is how essentially every neural network, from LeNet to today’s LLMs, is trained. Canon · 1998 Gradient-Based Learning Applied to Document Recognition It is the blueprint for the convolutional network, the architecture that later cracked computer vision. Canon · 2014 Generative Adversarial Networks GANs launched the modern era of generated images and dominated image synthesis until diffusion arrived. Canon · 2015 Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift It made very deep networks practical to train, and normalization layers became a standard part of the recipe.

Sources

  1. Goodfellow, Bengio & Courville, Deep Learning, ch. 6 "Deep Feedforward Networks" . MIT Press, 2016
  2. Frank Rosenblatt, "The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain" . Psychological Review, 1958
  3. Rumelhart, Hinton & Williams, "Learning Representations by Back-Propagating Errors" . Nature, 1986