Understanding Neural Networks

Imagine you’re designing a system that needs to drive a car. You could hardcode a ton of if-else rules, but that would be a nightmare—handling every edge case manually is impractical. Instead, you need something that can learn from examples. That’s where neural networks come in.

At their core, neural networks are just layers of mathematical functions that transform input data into meaningful outputs.

Basic Structure

1. Input Layer – This is where the raw data comes in. If you’re working with images, these are pixel values. If you’re processing financial data, these might be stock prices.

2. Hidden Layers – This is where the actual "thinking" happens. Each layer applies transformations using weights (which control how much influence each input has) and activation functions (which decide whether a neuron should fire or not).

3. Output Layer – After passing through all the hidden layers, we get a final prediction. Maybe it’s a probability that an image is a cat, or a forecast of tomorrow’s stock price.

In a fully connected network, each neuron in one layer connects to every neuron in the next. Each connection has a weight, and every neuron (except input neurons) has a bias. The network adjusts these parameters through training, gradually improving its ability to make accurate predictions.

What’s a bias? It’s a setting that adjusts the output of a neuron, even when the inputs are zero. Weights determine how strong a connection is, but biases give the network flexibility—allowing neurons to fire even if the weighted sum is too low. Without biases, the network would be far more rigid and struggle to learn complex patterns.

In my simple neural network whose picture is attached, it's structured like this:

3 input neurons

2 hidden layers, each with 4 neurons

1 output neuron

What’s happening inside?

Nodes represent neurons—each performs a calculation.

Edges represent connections (weights)—they define how much influence one neuron has on another.

The output neuron produces the final result, like a classification or a prediction.

As you increase the number of neurons and layers, the complexity of what it can model grows. That’s both the power and the challenge of deep learning—you can build arbitrarily complex models, but the cost in training time and hardware resources skyrockets.

How Many Parameters Does This Network Have?

One of the best ways to understand a neural network’s complexity is by calculating its trainable parameters—the weights and biases it learns during training.

Layer-by-layer Breakdown

1. Input → Hidden Layer 1 (3 → 4)

Weights: 3 x 4 = 12

Biases: 4

Total: 16

2. Hidden Layer 1 → Hidden Layer 2 (4 → 4)

Weights: 4 x 4 = 16

Biases: 4

Total: 20

3. Hidden Layer 2 → Output Layer (4 → 1)

Weights: 4 x 1 = 4

Biases: 1

Total: 5

Total Parameters: 16 + 20 + 5 = 41

This means our simple neural network has 41 trainable parameters—small enough to run on practically any modern processor.

What Can Our 41-Parameter Model Actually Do?

A network this small isn’t going to generate human-like text or drive a self-driving car. But it’s still powerful within its scope. With proper training, a 41-parameter model could:

Classify basic patterns – Think of a simple spam detector or recognizing handwritten digits from a limited dataset.

Control basic game mechanics – A simple AI in a game could decide if an enemy should chase the player or stand still based on input conditions.

Small networks like this paved the way for modern deep learning—before we had billion-parameter models, researchers were solving real problems with networks just a bit larger than this.

To put this in perspective, Meta’s large-scale AI model:

LLama 2-7B → 7 billion parameters

LLama 2-13B → 13 billion parameters

LLama 2-65B → 65 billion parameters

That’s billions of parameters, compared to our 41. The difference? Large models operate on vastly larger datasets, requiring thousands of GPUs and weeks of training to reach high accuracy. Meanwhile, our simple 41-parameter model could be trained in seconds on a laptop.

But the core principles remain the same. Whether it’s a small neural network recognizing digits or a billion-parameter model generating human-like text, it’s all just weighted sums, activation functions, and optimization. The difference is scale.

Every large AI model started with something simple. If you get how this 41-parameter network works, you’ve already got the foundation to understand the billion-parameter giants shaping modern AI.