Implementing Simple Neural Network Backpropagation from Scratch | by Siq Sun | Mar, 2024

Solving XOR gate problem— using just NumPy, then compare with PyTorch implementation.

Outline

・Introduction to the XOR Gate Problem

・Constructing a 2-Layer Neural Network

・Forward Propagation

・Chain Rules for Backpropagation

・Implementation with NumPy

・Comparing Results with PyTorch

・Summary

・References

Introduction to the XOR Gate Problem

The XOR (exclusive OR) gate problem is considered simple for a neural network because it involves learning a simple pattern of relationships between inputs and outputs that a properly designed network can capture, even though it is not linearly separable (meaning you can’t draw a single straight line to separate the outputs into two groups based on inputs). Neural networks, particularly those with hidden layers, are capable of learning non-linear patterns.
Let’s look at the inputs and outputs of XOR Gate. Here is our 4 training data.