LeNet implementation

activatedgeek/LeNet-5

Good packaging! Tested on Ubuntu 22.10:

git clone https://github.com/activatedgeek/LeNet-5
cd LeNet-5
git checkout 95b55a838f9d90536fd3b303cede12cf8b5da47f
virtualenv -p python3 .venv
. .venv/bin/activate

# Their requirements.txt uses >= and some == are incompatible with our Ubuntu.
pip install
  Pillow==6.2.0 \
  numpy==1.24.2 \
  onnx==1.13.1 \
  torch==2.0.0 \
  torchvision==0.15.1 \
  visdom==0.2.4 \
;

time python run.py

This throws a billion exceptions because we didn't start the visdom server, but never mind that.

The scrip does a fixed 15 epochs.

Output on P51:

real    2m10.262s
user    11m9.771s
sys     0m26.368s

The run also produces a lenet.onnx ONNX file, which is pretty neat, and allows us for example to visualize it on Netron:

Figure 1.
Netron visualization of the activatedgeek/LeNet-5 ONNX output
. From this we can see the bifurcation on the computational graph as done in the code at:
`output = self.c1(img) x = self.c2_1(output) output = self.c2_2(output) output += x output = self.c3(output)`
This doesn't seem to conform to the original LeNet-5 however?

activatedgeek/LeNet-5 use ONNX for inference

 0  0 

Now let's try and use the trained ONNX file for inference on some manually drawn images on GIMP:

Note that the images must be drawn with white on black. If you use black on white, it the accuracy becomes terrible. This is a good very example of brittleness in AI systems!

Figure 1.
Number 9 drawn with mouse on GIMP by Ciro Santilli (2023)

We can try the code adapted from thenewstack.io/tutorial-using-a-pre-trained-onnx-model-for-inferencing/ at python/onnx_cheat/infer_mnist.py:

cd python/onnx_cheat
./infer_mnist.py lenet.onnx infer_mnist_9.png

and it works pretty well! The protram outputs:

as desired.

We can also try with images directly from Extract MNIST images.

for f in /home/ciro/git/mnist_png/out/testing/1/*.png; do echo $f; infer.py $f ; done

and the accuracy is great as expected.

activatedgeek/LeNet-5 run on GPU

 0  0 

By default, the setup runs on CPU only, not GPU, as could be seen by running htop. But by the magic of PyTorch, modifying the program to run on the GPU is trivial:

cat << EOF | patch
diff --git a/run.py b/run.py
index 104d363..20072d1 100644
--- a/run.py
+++ b/run.py
@@ -24,7 +24,8 @@ data_test = MNIST('./data/mnist',
 data_train_loader = DataLoader(data_train, batch_size=256, shuffle=True, num_workers=8)
 data_test_loader = DataLoader(data_test, batch_size=1024, num_workers=8)

-net = LeNet5()
+device = 'cuda'
+net = LeNet5().to(device)
 criterion = nn.CrossEntropyLoss()
 optimizer = optim.Adam(net.parameters(), lr=2e-3)

@@ -43,6 +44,8 @@ def train(epoch):
     net.train()
     loss_list, batch_list = [], []
     for i, (images, labels) in enumerate(data_train_loader):
+        labels = labels.to(device)
+        images = images.to(device)
         optimizer.zero_grad()

         output = net(images)
@@ -71,6 +74,8 @@ def test():
     total_correct = 0
     avg_loss = 0.0
     for i, (images, labels) in enumerate(data_test_loader):
+        labels = labels.to(device)
+        images = images.to(device)
         output = net(images)
         avg_loss += criterion(output, labels).sum()
         pred = output.detach().max(1)[1]
@@ -84,7 +89,7 @@ def train_and_test(epoch):
     train(epoch)
     test()

-    dummy_input = torch.randn(1, 1, 32, 32, requires_grad=True)
+    dummy_input = torch.randn(1, 1, 32, 32, requires_grad=True).to(device)
     torch.onnx.export(net, dummy_input, "lenet.onnx")

     onnx_model = onnx.load("lenet.onnx")
EOF

and leads to a faster runtime, with less user as now we are spending more time on the GPU than CPU:

real    1m27.829s
user    4m37.266s
sys     0m27.562s

Residual neural network (2015, ResNet)

 0  0 

Interesting layer skip architecture thing.

Apparently destroyed ImageNet 2015 and became very very famous as such.

ResNet implementation

 0  0 

torchvision ResNet

ResNet variant

 0  0 

ResNet v1 vs v1.5

 0  0 

catalog.ngc.nvidia.com/orgs/nvidia/resources/resnet_50_v1_5_for_pytorch explains:

The difference between v1 and v1.5 is that, in the bottleneck blocks which requires downsampling, v1 has stride = 2 in the first 1x1 convolution, whereas v1.5 has stride = 2 in the 3x3 convolution.
This difference makes ResNet50 v1.5 slightly more accurate (~0.5% top1) than v1, but comes with a smallperformance drawback (~5% imgs/sec).

Convolutional neural network (CNN)

 1  0 

CNN convolution kernels are also learnt

 0  0 

CNN convolution kernels are not hardcoded. They are learnt and optimized via backpropagation. You just specify their size! Example in PyTorch you'd do just:

nn.Conv2d(1, 6, kernel_size=(5, 5))

as used for example at: activatedgeek/LeNet-5.

This can also be inferred from: stackoverflow.com/questions/55594969/how-to-visualise-filters-in-a-cnn-with-pytorch where we see that the kernels are not perfectly regular as you'd expected from something hand coded.