Building a Neural Network from scratch

I'm trying to improve my knowledge of backpropagation. I feel like my current understanding relies too much on "black-box" abstractions. So the next best step? Build a simple NN from scratch.
My code on GitHub
Technical specifics on Notion

Graph vs Matrix implementation

You can express the layered model of a neural network as a graph (nodes and weighted edges that are inter-connected) or represent it as a matrix. Matrix implementations require less compute by leveraging matrix operations, but they limit the modifications you can make to how backpropagation adjusts weights, etc. For now I'll be using the matrix implementation.

Forward propagation

Forward prop is how the network generates a probability vector for your given inputs. The input layer is just the data you're passing in, and every layer after that is a combination of a bias value, the previous layer's value and a connection weight between the nodes.
To put this into an equation, A_L = ReLU(A_L-1 * W_L + B_L). There's a detailed visual on the Notion page for reference.

Backpropagation

The specifics of this are a little complex, so there's detailed info on the Notion page. But basically, we are trying to reduce the loss function of the network using gradient descent: an iterative approach that adjusts weights based on the difference between generated and expected outputs, using derivatives to find the minimum of a function.

Overall thoughts

Today I finished all parts of the network, including backpropagation. Unfortunately, its performance is not great; you can review specifics, including the loss progression, on the Notion page. I'm going to experiment and improve performance over the next few weeks. I suspect something might be wrong with how the network is ingesting data. But now I'm going to focus on the OS project. Thanks for joining the journey!