Ciro Santilli @cirosantilli 34

 Incoming links: COCO dataset

MLperf  Updated 2024-12-15  +Created 1970-01-01

mlcommons.org/en/ Their homepage is not amazingly organized, but it does the job.

Benchmark focused on deep learning. It has two parts:

training: produces a trained network
inference: uses the trained network

Furthermore, a specific network model is specified for each benchmark in the closed category: so it goes beyond just specifying the dataset.

Results can be seen e.g. at:

training: mlcommons.org/en/training-normal-21/
inference: mlcommons.org/en/inference-datacenter-21/

And there are also separate repositories for each:

E.g. on mlcommons.org/en/training-normal-21/ we can see what the the benchmarks are:

Dataset	Model
ImageNet	ResNet
KiTS19	3D U-Net
OpenImages	RetinaNet
COCO dataset	Mask R-CNN
LibriSpeech	RNN-T
Wikipedia	BERT
1TB Clickthrough	DLRM
Go	MiniGo

 Read the full article

MLperf v2.1 ResNet  Updated 2024-12-15  +Created 1970-01-01

 View more

Instructions at:

Ubuntu 22.10 setup with tiny dummy manually generated ImageNet and run on ONNX:

sudo apt install pybind11-dev

git clone https://github.com/mlcommons/inference
cd inference
git checkout v2.1

virtualenv -p python3 .venv
. .venv/bin/activate
pip install numpy==1.24.2 pycocotools==2.0.6 onnxruntime==1.14.1 opencv-python==4.7.0.72 torch==1.13.1

cd loadgen
CFLAGS="-std=c++14" python setup.py develop
cd -

cd vision/classification_and_detection
python setup.py develop
wget -q https://zenodo.org/record/3157894/files/mobilenet_v1_1.0_224.onnx
export MODEL_DIR="$(pwd)"
export EXTRA_OPS='--time 10 --max-latency 0.2'

tools/make_fake_imagenet.sh
DATA_DIR="$(pwd)/fake_imagenet" ./run_local.sh onnxruntime mobilenet cpu --accuracy

Last line of output on P51, which appears to contain the benchmark results

TestScenario.SingleStream qps=58.85, mean=0.0138, time=0.136, acc=62.500%, queries=8, tiles=50.0:0.0129,80.0:0.0137,90.0:0.0155,95.0:0.0171,99.0:0.0184,99.9:0.0187

where presumably qps means queries per second, and is the main results we are interested in, the more the better.

Running:

tools/make_fake_imagenet.sh

produces a tiny ImageNet subset with 8 images under fake_imagenet/.

fake_imagenet/val_map.txt contains:

val/800px-Porsche_991_silver_IAA.jpg 817
val/512px-Cacatua_moluccensis_-Cincinnati_Zoo-8a.jpg 89
val/800px-Sardinian_Warbler.jpg 13
val/800px-7weeks_old.JPG 207
val/800px-20180630_Tesla_Model_S_70D_2015_midnight_blue_left_front.jpg 817
val/800px-Welsh_Springer_Spaniel.jpg 156
val/800px-Jammlich_crop.jpg 233
val/782px-Pumiforme.JPG 285

where the numbers are the category indices from ImageNet1k. At gist.github.com/yrevar/942d3a0ac09ec9e5eb3a see e.g.:

817: 'sports car, sport car',
89: 'sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita',

and so on, so they are coherent with the image names. By quickly looking at the script we see that it just downloads from Wikimedia and manually creates the file.

TODO prepare and test on the actual ImageNet validation set, README says:

Prepare the imagenet dataset to come.

Since that one is undocumented, let's try the COCO dataset instead, which uses COCO 2017 and is also a bit smaller. Note that his is not part of MLperf anymore since v2.1, only ImageNet and open images are used. But still:

wget https://zenodo.org/record/4735652/files/ssd_mobilenet_v1_coco_2018_01_28.onnx
DATA_DIR_BASE=/mnt/data/coco
export DATA_DIR="${DATADIR_BASE}/val2017-300"
mkdir -p "$DATA_DIR_BASE"
cd "$DATA_DIR_BASE"
wget http://images.cocodataset.org/zips/val2017.zip
wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip
unzip val2017.zip
unzip annotations_trainval2017.zip
mv annotations val2017
cd -
cd "$(git-toplevel)"
python tools/upscale_coco/upscale_coco.py --inputs "$DATA_DIR_BASE" --outputs "$DATA_DIR" --size 300 300 --format png
cd -

Now:

./run_local.sh onnxruntime mobilenet cpu --accuracy

fails immediately with:

No such file or directory: '/path/to/coco/val2017-300/val_map.txt

The more plausible looking:

./run_local.sh onnxruntime mobilenet cpu --accuracy --dataset coco-300

first takes a while to preprocess something most likely, which it does only one, and then fails:

Traceback (most recent call last):
  File "/home/ciro/git/inference/vision/classification_and_detection/python/main.py", line 596, in <module>
    main()
  File "/home/ciro/git/inference/vision/classification_and_detection/python/main.py", line 468, in main
    ds = wanted_dataset(data_path=args.dataset_path,
  File "/home/ciro/git/inference/vision/classification_and_detection/python/coco.py", line 115, in __init__
    self.label_list = np.array(self.label_list)
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 2 dimensions. The detected shape was (5000, 2) + inhomogeneous part.

TODO!

 Read the full article

Open Images dataset  Updated 2024-12-15  +Created 1970-01-01

 View more

storage.googleapis.com/openimages/web/index.html

TODO vs COCO dataset.

As of v7:

~9M images
600 object classes
bounding boxes
visual relatoinships are really hard: storage.googleapis.com/openimages/web/factsfigures_v7.html#visual-relationships e.g. "person kicking ball": storage.googleapis.com/openimages/web/visualizer/index.html?type=relationships&set=train&c=kick
google.github.io/localized-narratives/ localized narratives is ludicrous, you can actually hear the (Indian women mostly) annotators describing the image while hovering their mouses to point what they are talking about). They are clearly bored out of their minds the poor people!

The images and annotations are both under CC BY, with Google as the copyright holder.

 Read the full article

torchvision ResNet  Updated 2024-12-15  +Created 1970-01-01

 View more

pytorch.org/vision/0.13/models.html has a minimal runnable example adapted to python/pytorch/resnet_demo.py.

That example uses a ResNet pre-trained on the COCO dataset to do some inference, tested on Ubuntu 22.10:

cd python/pytorch
wget -O resnet_demo_in.jpg https://upload.wikimedia.org/wikipedia/commons/thumb/6/60/Rooster_portrait2.jpg/400px-Rooster_portrait2.jpg
./resnet_demo.py resnet_demo_in.jpg resnet_demo_out.jpg

This first downloads the model, which is currently 167 MB.

We know it is COCO because of the docs: pytorch.org/vision/0.13/models/generated/torchvision.models.detection.fasterrcnn_resnet50_fpn_v2.html which explains that

FasterRCNN_ResNet50_FPN_V2_Weights.DEFAULT

is an alias for:

FasterRCNN_ResNet50_FPN_V2_Weights.COCO_V1

The runtime is relatively slow on P51, about 4.7s.

After it finishes, the program prints the recognized classes:

['bird', 'banana']

so we get the expected bird, but also the more intriguing banana.

By looking at the output image with bounding boxes, we understand where the banana came from!

Figure 2.
python/pytorch/resnet_demo_out.jpg
. The beak was of course a banana, not a beak!

 Read the full article

YOLO model  Updated 2024-12-15  +Created 1970-01-01

 View more

Object detection model.

You can get some really sweet pre-trained versions of this, typically trained on the COCO dataset.

 Read the full article