The importance of building things from scratch: A neural networks story

One of the best ways to learn machine learning is to build stuff yourself!

1. A Neural Networks story:

I was a mechanical engineer till 2017 and then I transitioned to machine learning. I clearly remember one moment during my first machine learning class at MIT:

The professor had drawn a schematic of a neural network on the blackboard as shown in the above figure.

I was diligently making notes.

I even solved the neural network take home assignment: using neural networks to solve a classification problem.

I thought that I had understood neural networks.

Last year, I wanted to revisit backpropagation. I thought of really testing myself.

I sat down with a pen and paper and decide to write the backpropagation step for a single neuron myself.

That’s when I realized that when I was sitting in that classroom at MIT, I did not really “understand” neural networks.

I merely listened to a professor explaining what neural networks are and then I ran a Google Colab notebook.

As I proceeded to write the backpropagation step for a single neuron, a lot of my concepts were getting clarified along the way.

It took some time, but after I wrote the backpropagation step, I felt a sense of satisfaction.

That’s the same satisfaction you feel when you build an aeroplane from LEGO blocks.

That’s the same satisfaction you feel when you create a car shape from just clay mould.

That’s the same satisfaction you feel when you make an entire cooking recipe from the bare, raw ingredients.

It is the satisfaction of building things from scratch.

2. Getting power

Now that I had written the backpropagation step on a piece of paper, I thought to myself: what would it be like to build the entire neural network from scratch.

For the next 3 months, I did exactly that.

Starting from one neuron to multiple neurons to a chain of neurons, I coded the forward pass and then the backward pass completely from scratch.

When I ran training on this neural network built from scratch, I felt powerful.

I felt like Popeye after he eats his Spinach!

I felt that deep learning is actually simple.

I felt that I “really understood” neural networks now and I can now easily apply them to any problem.

Concepts like overfitting, early stopping, regularization and dropout start making so much sense after you build the neural network yourself.

These concepts are not abstract anymore.

3. ML engineer-research skills

What I noticed was that after I had built a neural network from scratch, my machine learning engineering skills also improved.

What are these skills:

• Performing hyperparameter optimization

• Reducing overfitting

• Improving testing and validation accuracy

• Obtaining interpretable solutions

• Explaining ML solution to stakeholders

When I read neural network research papers, I started to understand many things.

I realized that students-industry professionals-researchers who never build a neural network from scratch always have difficulty understanding research papers.

4. Teaching

I went one step further and documented my entire learning journey.

I made a set of 35 videos teaching you how to build a neural network from scratch.

Here is the playlist: Link

Here is the first video for you to get started:

5. Conclusion

1. Get your hands dirty and start building from scratch

2. Don’t just do toy projects on Kaggle and Github

3. Building ML pipelines from scratch has long term benefits. It also makes you a better ML engineer and researcher.