From 0794ad3dba78d98614e6dc31dd4a8d3f90722acc Mon Sep 17 00:00:00 2001 From: Bharath Ramaswamy Date: Fri, 16 Dec 2022 12:21:43 -0800 Subject: [PATCH] Added model tasks to README table (#74) * Added model tasks to README table * Added model task for DeepSpeech2 * Updated task names based on feedback * Grouped models of the same task category together Signed-off-by: Bharath Ramaswamy --- .githooks/pre-commit | 3 +- README.md | 136 ++++++++++++++++++----------- zoo_torch/bert/Bert.md | 2 +- zoo_torch/distilbert/DistilBert.md | 2 +- zoo_torch/minilm/MiniLM.md | 2 +- zoo_torch/mobilebert/MobileBert.md | 2 +- 6 files changed, 93 insertions(+), 54 deletions(-) diff --git a/.githooks/pre-commit b/.githooks/pre-commit index 4820c04..2b7bdcf 100755 --- a/.githooks/pre-commit +++ b/.githooks/pre-commit @@ -42,10 +42,11 @@ else fi echo "Running Pylint - using $prj_root/.pylintrc" - export PYTHONPATH=$prj_root/zoo_tensorflow/examples:$prj_root/zoo_torch/examples + export PYTHONPATH=$prj_root/zoo_tensorflow:$prj_root/zoo_torch # Verify that each of the source path directories exist for path in ${PYTHONPATH//:/ }; do if [[ ! -d $path ]]; then + echo "ERROR: Path $path does NOT exist" exit 1 fi done diff --git a/README.md b/README.md index 201de3b..82125ac 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,7 @@ An original FP32 source model is quantized either using post-training quantizati ## PyTorch Models + @@ -30,12 +31,14 @@ An original FP32 source model is quantized either using post-training quantizati + + @@ -46,6 +49,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -56,6 +60,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -66,6 +71,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -76,6 +82,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -86,16 +93,29 @@ An original FP32 source model is quantized either using post-training quantizati - - - - - - - - + + + + + + + + + + + + + + + + + + + + + @@ -106,6 +126,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -122,6 +143,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -138,6 +160,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -148,16 +171,7 @@ An original FP32 source model is quantized either using post-training quantizati - - - - - - - - - - + @@ -166,6 +180,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -174,6 +189,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -182,6 +198,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -190,6 +207,18 @@ An original FP32 source model is quantized either using post-training quantizati + + + + + + + + + + + + @@ -200,6 +229,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -210,6 +240,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -220,6 +251,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -228,6 +260,18 @@ An original FP32 source model is quantized either using post-training quantizati + + + + + + + + + + + + @@ -247,6 +291,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -266,6 +311,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -285,6 +331,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -298,6 +345,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -316,26 +364,6 @@ An original FP32 source model is quantized either using post-training quantizati - - - - - - - - - - - - - - - - - - - -
Task Network[1] Model Source[2] Floating Pt (FP32) Model [3] Metric FP32 W8A8[6] W4A8[7]
Image Classification MobileNetV2 GitHub Repo Pretrained ModelTBD
Image Classification Resnet18 Pytorch Torchvision Pytorch Torchvision 69.1%
Image Classification Resnet50 Pytorch Torchvision Pytorch Torchvision 75.63%
Image Classification Regnet_x_3_2gf Pytorch Torchvision Pytorch Torchvision 77.70%
Image Classification EfficientNet-lite0 GitHub Repo Pretrained Model74.46%
DeepLabV3+GitHub RepoPretrained ModelQuantized Model(PascalVOC) mIOU72.91%72.44%72.18%Image ClassificationViTRepoPrepared Models See Example (ImageNet dataset) Accuracy81.3281.57TBD
Image ClassificationMobileViTRepoPrepared Models See Example (ImageNet dataset) Accuracy78.4677.59TBD
Object Detection MobileNetV2-SSD-Lite GitHub Repo Pretrained ModelTBD
Pose Estimation Pose Estimation Based on Ref. Based on Ref.TBD
Pose Estimation HRNET-Posenet Based on Ref. FP32 Model0.791
Super Resolution SRGAN GitHub Repo Pretrained Model (older version from here)TBD
DeepSpeech2GitHub RepoPretrained ModelSee Example(Librispeech Test Clean) WER9.92%10.22%TBD
Super Resolution Anchor-based Plain Net (ABPN) Based on Ref. See TarballsTBD
Super Resolution Extremely Lightweight Quantization Robust Real-Time Single-Image Super Resolution (XLSR) Based on Ref. See TarballsTBD
Super Resolution Super-Efficient Super Resolution (SESR) Based on Ref. See TarballsTBD
Super Resolution QuickSRNet - See TarballsTBD
Semantic SegmentationDeepLabV3+GitHub RepoPretrained ModelQuantized Model(PascalVOC) mIOU72.91%72.44%72.18%
Semantic Segmentation HRNet-W48 GitHub Repo Original model weight not available 80.07%
Semantic Segmentation InverseForm (HRNet-16-Slim-IF) GitHub Repo Pretrained ModelTBD
Semantic Segmentation InverseForm (OCRNet-48) GitHub Repo Pretrained ModelTBD
Semantic Segmentation FFNets Github Repo Prepared Models (5 in total)TBD
Speech RecognitionDeepSpeech2GitHub RepoPretrained ModelSee Example(Librispeech Test Clean) WER9.92%10.22%TBD
NLP / NLU Bert Repo Prepared Models Detailed Results
NLP / NLU MobileBert Repo Prepared Models Detailed Results
NLP / NLU MiniLM Repo Prepared Models Detailed Results
NLP / NLU Roberta Repo Prepared Models Detailed Results
NLP / NLU DistilBert Repo Prepared Models
Detailed Results
ViTRepoPrepared Models See Example (ImageNet dataset) Accuracy81.3281.57TBD
MobileViTRepoPrepared Models See Example (ImageNet dataset) Accuracy78.4677.59TBD
@@ -351,6 +379,7 @@ An original FP32 source model is quantized either using post-training quantizati ## Tensorflow Models + @@ -364,12 +393,14 @@ An original FP32 source model is quantized either using post-training quantizati + + @@ -381,6 +412,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -392,6 +424,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -403,6 +436,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -414,6 +448,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -425,6 +460,19 @@ An original FP32 source model is quantized either using post-training quantizati + + + + + + + + + + + + + @@ -442,6 +490,7 @@ An original FP32 source model is quantized either using post-training quantizati + @@ -452,17 +501,6 @@ An original FP32 source model is quantized either using post-training quantizati - - - - - - - - - - -
Task Network [1] Model Source [2] Floating Pt (FP32) Model [3] Metric FP32 W8A8[6] W4A8[7]
Image Classification ResNet-50 (v1) GitHub Repo Pretrained ModelTBD
Image Classification MobileNet-v2-1.4 GitHub Repo Pretrained ModelTBD
Image Classification EfficientNet Lite GitHub Repo Pretrained Model TBD
Object Detection SSD MobileNet-v2 GitHub Repo Pretrained ModelTBD
Object Detection RetinaNet GitHub Repo Pretrained ModelTBD
Object DetectionMobileDet-EdgeTPUGitHub RepoPretrained ModelSee Example2.4(COCO) Mean Avg. Precision (mAP)0.2810.279TBD
Pose Estimation Pose Estimation Based on Ref. Based on Ref.TBD
Super Resolution SRGAN GitHub Repo Pretrained Model24.78 / 0.628 25.41 / 0.666 (INT8W / INT16Act.)
MobileDet-EdgeTPUGitHub RepoPretrained ModelSee Example2.4(COCO) Mean Avg. Precision (mAP)0.2810.279TBD
*[1]* Model usage documentation diff --git a/zoo_torch/bert/Bert.md b/zoo_torch/bert/Bert.md index 7ecfab4..e569128 100644 --- a/zoo_torch/bert/Bert.md +++ b/zoo_torch/bert/Bert.md @@ -1,5 +1,5 @@ # PyTorch Transformer model Bert-base-uncased for Natural Language Classifier and Question Answering -This document describes evaluation of optimized checkpoints for transformer models Bert-base-uncased for NL Classification tasks and Question Anwering tasks. +This document describes evaluation of optimized checkpoints for transformer models Bert-base-uncased for NL Classification tasks and Question Answering tasks. ## AIMET installation and setup Please [install and setup AIMET](https://github.com/quic/aimet/blob/release-aimet-1.23/packaging/install.md) (*Torch GPU* variant) before proceeding further. diff --git a/zoo_torch/distilbert/DistilBert.md b/zoo_torch/distilbert/DistilBert.md index eb0e635..55905a9 100644 --- a/zoo_torch/distilbert/DistilBert.md +++ b/zoo_torch/distilbert/DistilBert.md @@ -1,5 +1,5 @@ # PyTorch Transformer model DistilBert-base-uncased for Natural Language Classifier and Question Answering -This document describes evaluation of optimized checkpoints for transformer models DistilBert-base-uncased for NL Classification tasks and Question Anwering tasks. +This document describes evaluation of optimized checkpoints for transformer models DistilBert-base-uncased for NL Classification tasks and Question Answering tasks. ## AIMET installation and setup Please [install and setup AIMET](https://github.com/quic/aimet/blob/release-aimet-1.23/packaging/install.md) (*Torch GPU* variant) before proceeding further. diff --git a/zoo_torch/minilm/MiniLM.md b/zoo_torch/minilm/MiniLM.md index 19159c4..c18ad9d 100644 --- a/zoo_torch/minilm/MiniLM.md +++ b/zoo_torch/minilm/MiniLM.md @@ -1,5 +1,5 @@ # PyTorch Transformer model MiniLM-L12-H384-uncased for Natural Language Classifier and Question Answering -This document describes evaluation of optimized checkpoints for transformer models MiniLM-L12-H384-uncased for NL Classification tasks and Question Anwering tasks. +This document describes evaluation of optimized checkpoints for transformer models MiniLM-L12-H384-uncased for NL Classification tasks and Question Answering tasks. ## AIMET installation and setup Please [install and setup AIMET](https://github.com/quic/aimet/blob/release-aimet-1.23/packaging/install.md) (*Torch GPU* variant) before proceeding further. diff --git a/zoo_torch/mobilebert/MobileBert.md b/zoo_torch/mobilebert/MobileBert.md index ba02f9c..c0d4fe9 100644 --- a/zoo_torch/mobilebert/MobileBert.md +++ b/zoo_torch/mobilebert/MobileBert.md @@ -1,5 +1,5 @@ # PyTorch Transformer model Mobilebert-uncased for Natural Language Classifier and Question Answering -This document describes evaluation of optimized checkpoints for transformer models Mobilebert-uncased for NL Classification tasks and Question Anwering tasks. +This document describes evaluation of optimized checkpoints for transformer models Mobilebert-uncased for NL Classification tasks and Question Answering tasks. ## AIMET installation and setup Please [install and setup AIMET](https://github.com/quic/aimet/blob/release-aimet-1.23/packaging/install.md) (*Torch GPU* variant) before proceeding further.