feat(ansible): tvm artifacts download (#3916)

* feat(ansible): add tvm artifacts download to artifacts role Signed-off-by: Alexey Panferov <lexavtanke@gmail.com> * feat(ansible): add tvm artifacts download instruction to readme Signed-off-by: Alexey Panferov <lexavtanke@gmail.com> * feat(ansible): add tvm artifacts sha256 checksum Signed-off-by: Alexey Panferov <lexavtanke@gmail.com> * style(pre-commit): autofix * feat(ansible): fix ansible-lint errors, add file permission Signed-off-by: Alexey Panferov <lexavtanke@gmail.com> --------- Signed-off-by: Alexey Panferov <lexavtanke@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
autowarefoundation · Oct 23, 2023 · b92953c · b92953c
1 parent 255ddab
commit b92953c
Show file tree

Hide file tree

Showing 3 changed files with 158 additions and 6 deletions.
diff --git a/ansible/roles/artifacts/README.md b/ansible/roles/artifacts/README.md
@@ -2,11 +2,9 @@
 
 The Autoware perception stack uses models for inference. These models are automatically downloaded if using `ansible`, but they can also be downloaded manually.
 
-## ONNX model files
+## Download instructions
 
-### Download instructions
-
-The ONNX model files are stored in a common location, hosted by Web.Auto
+The artifacts files are stored in a common location, hosted by Web.Auto
 
 Any tool that can download files from the web (e.g. `wget` or `curl`) is the only requirement for downloading these files:
 
@@ -100,11 +98,64 @@ $ mkdir -p ~/autoware_data/traffic_light_ssd_fine_detector/
 $ wget -P ~/autoware_data/traffic_light_ssd_fine_detector/ \
        https://awf.ml.dev.web.auto/perception/models/mb2-ssd-lite-tlr.onnx \
        https://awf.ml.dev.web.auto/perception/models/voc_labels_tl.txt
+
+
+# tvm_utility
+
+$ mkdir -p ~/autoware_data/tvm_utility/models/yolo_v2_tiny
+$ wget -P ~/autoware_data/tvm_utility/ \
+       https://autoware-modelzoo.s3.us-east-2.amazonaws.com/models/3.0.0-20221221/yolo_v2_tiny-x86_64-llvm-3.0.0-20221221.tar.gz
+
+
+# lidar_centerpoint_tvm
+
+$ mkdir -p ~/autoware_data/lidar_centerpoint_tvm/models/centerpoint_encoder
+$ mkdir -p ~/autoware_data/lidar_centerpoint_tvm/models/centerpoint_backbone
+$ wget -P ~/autoware_data/lidar_centerpoint_tvm/ \
+       https://autoware-modelzoo.s3.us-east-2.amazonaws.com/models/3.0.0-20221221/centerpoint_encoder-x86_64-llvm-3.0.0-20221221.tar.gz \
+       https://autoware-modelzoo.s3.us-east-2.amazonaws.com/models/3.0.0-20221221/centerpoint_backbone-x86_64-llvm-3.0.0-20221221.tar.gz
+
+
+# lidar_apollo_segmentation_tvm
+
+$ mkdir -p ~/autoware_data/lidar_apollo_segmentation_tvm/models/baidu_cnn
+$ wget -P ~/autoware_data/lidar_apollo_segmentation_tvm/ \
+      https://autoware-modelzoo.s3.us-east-2.amazonaws.com/models/3.0.0-20221221/baidu_cnn-x86_64-llvm-3.0.0-20221221.tar.gz
 ```
 
 After downloading you can check integrity of the files with `sha256sum`:
 
 ```console
-cd ~/autoware_data/
-wget -q -O - https://raw.githubusercontent.com/autowarefoundation/autoware/main/ansible/roles/artifacts/SHA256SUMS | sha256sum -c
+#
+$ cd ~/autoware_data/
+$ wget -q -O - https://raw.githubusercontent.com/autowarefoundation/autoware/main/ansible/roles/artifacts/SHA256SUMS | sha256sum -c
+```
+
+Extracting files:
+
+```console
+# yabloc_pose_initializer
+
+$ tar -xf ~/autoware_data/yabloc_pose_initializer/resources.tar.gz \
+       -C ~/autoware_data/yabloc_pose_initializer/
+
+
+# tvm_utility
+
+$ tar -xf ~/autoware_data/tvm_utility/yolo_v2_tiny-x86_64-llvm-3.0.0-20221221.tar.gz \
+       -C ~/autoware_data/tvm_utility/models/yolo_v2_tiny/
+
+
+# lidar_centerpoint_tvm
+
+$ tar -xf ~/autoware_data/lidar_centerpoint_tvm/centerpoint_encoder-x86_64-llvm-3.0.0-20221221.tar.gz \
+       -C ~/autoware_data/lidar_centerpoint_tvm/models/centerpoint_encoder
+$ tar -xf ~/autoware_data/lidar_centerpoint_tvm/centerpoint_backbone-x86_64-llvm-3.0.0-20221221.tar.gz \
+       -C ~/autoware_data/lidar_centerpoint_tvm/models/centerpoint_backbone
+
+
+# lidar_apollo_segmentation_tvm
+
+$ tar -xf ~/autoware_data/lidar_apollo_segmentation_tvm/baidu_cnn-x86_64-llvm-3.0.0-20221221.tar.gz \
+       -C ~/autoware_data/lidar_apollo_segmentation_tvm/models/baidu_cnn
 ```
diff --git a/ansible/roles/artifacts/SHA256SUMS b/ansible/roles/artifacts/SHA256SUMS
@@ -3,10 +3,13 @@
 86348d8c4bced750f54288b01cc471c0d4f1ec9c693466169ef19413731e6ecc  ./lidar_apollo_instance_segmentation/hdl-64.onnx
 eec521ebad7553d0ea2c90472a293aecb7499ab592632f0e100481c8196eb421  ./lidar_apollo_instance_segmentation/vlp-16.onnx
 95ef950bb694bd6de91b7e47f5d191d557e92a7f5e2a6bdf655a8b5eed4075cc  ./lidar_apollo_instance_segmentation/vls-128.onnx
+4293e6196ec937d2cd5ec658e5ce70933647d2d38633a1805febb36cafd684e3  ./lidar_apollo_segmentation_tvm/baidu_cnn-x86_64-llvm-3.0.0-20221221.tar.gz
 3fe7e128955646740c41a25be0c8f141d5a94594fe79d7405fe2a859e391542e  ./lidar_centerpoint/pts_backbone_neck_head_centerpoint.onnx
 9bb0b634f3664bd098ce7d6a3d8a9fb7cc8d9b8252b27f302c71e43316bab551  ./lidar_centerpoint/pts_backbone_neck_head_centerpoint_tiny.onnx
 dc1a876580d86ee7a341d543f8ade2ede7f43bd032dc5b44155b1f0175405764  ./lidar_centerpoint/pts_voxel_encoder_centerpoint.onnx
 2c53465715c1fd2e9dc5727ef3fca74f4cdf0538f74286b0946e219d0ca5693b  ./lidar_centerpoint/pts_voxel_encoder_centerpoint_tiny.onnx
+3840b6b3590984e8115d66b12061aea3a2cfaed70b4e8d59457f04b0d6f6a1fc  ./lidar_centerpoint_tvm/centerpoint_backbone-x86_64-llvm-3.0.0-20221221.tar.gz
+41e19de43f30549c325968aee06b4ad0d9701220be819e79d37efdfa86b918d0  ./lidar_centerpoint_tvm/centerpoint_encoder-x86_64-llvm-3.0.0-20221221.tar.gz
 634a1132eb33f8091d60f2c346ababe8b905ae08387037aed883953b7329af84  ./tensorrt_yolo/coco.names
 61e922f76918dd3d8e0abdc5fb7406f390609e08bd8ab9e5d3b97afb00f30f8c  ./tensorrt_yolo/yolov3.onnx
 0e877c716fbf8a2b431ee3e57f6c7411a6741319b52c32c6dafc53c7e1b17027  ./tensorrt_yolo/yolov4-tiny.onnx
@@ -34,4 +37,5 @@ b3c6e00acc6ff547d165469684ffb620a9a6330e9d591d445f50c4cf5cb4e292  ./traffic_ligh
 2824d4c5b7ab5f6bfd41e43e82747107c53e1c727b1cf1dd6746bc49e6749128  ./traffic_light_fine_detector/tlr_yolox_s_batch_6.onnx
 e29e6ee68751a270fb285fd037713939ca7f61a897b4c3a7ab22b0d6a9a21ddf  ./traffic_light_ssd_fine_detector/mb2-ssd-lite-tlr.onnx
 a41e6e3324e32c30b3b2fe38908eaf3471e2bfdaeb9e14ca0c1c3bc0275119c6  ./traffic_light_ssd_fine_detector/voc_labels_tl.txt
+66b3ca668e577393b657fbe1ed626538d89ca3adccd5862de6c7fa190238dbca  ./tvm_utility/yolo_v2_tiny-x86_64-llvm-3.0.0-20221221.tar.gz
 1f660e15f95074bade32b1f80dbf618e9cee1f0b9f76d3f4671cb9be7f56eb3a  ./yabloc_pose_initializer/resources.tar.gz
diff --git a/ansible/roles/artifacts/tasks/main.yaml b/ansible/roles/artifacts/tasks/main.yaml
@@ -361,3 +361,100 @@
     dest: "{{ data_dir }}/traffic_light_ssd_fine_detector/voc_labels_tl.txt"
     mode: "644"
     checksum: sha256:a41e6e3324e32c30b3b2fe38908eaf3471e2bfdaeb9e14ca0c1c3bc0275119c6
+
+# tvm_utility
+- name: Create tvm_utility/models directory inside {{ data_dir }}
+  ansible.builtin.file:
+    path: "{{ data_dir }}/tvm_utility/models"
+    mode: "755"
+    state: directory
+
+- name: Download yolo_v2_tiny-x86_64-llvm-3.0.0-20221221.tar.gz
+  become: true
+  ansible.builtin.get_url:
+    url: https://autoware-modelzoo.s3.us-east-2.amazonaws.com/models/3.0.0-20221221/yolo_v2_tiny-x86_64-llvm-3.0.0-20221221.tar.gz
+    dest: "{{ data_dir }}/tvm_utility/yolo_v2_tiny-x86_64-llvm-3.0.0-20221221.tar.gz"
+    mode: "644"
+    checksum: sha256:66b3ca668e577393b657fbe1ed626538d89ca3adccd5862de6c7fa190238dbca
+
+- name: Create yolo_v2_tiny folder in tvm_utility/models of {{ data_dir }}
+  ansible.builtin.file:
+    path: "{{ data_dir }}/tvm_utility/models/yolo_v2_tiny"
+    mode: "755"
+    state: directory
+
+- name: Extract yolo_v2_tiny-x86_64-llvm-3.0.0-20221221.tar.gz
+  ansible.builtin.unarchive:
+    src: "{{ data_dir }}/tvm_utility/yolo_v2_tiny-x86_64-llvm-3.0.0-20221221.tar.gz"
+    dest: "{{ data_dir }}/tvm_utility/models/yolo_v2_tiny"
+
+# lidar_centerpoint_tvm
+- name: Create lidar_centerpoint_tvm/models directory inside {{ data_dir }}
+  ansible.builtin.file:
+    path: "{{ data_dir }}/lidar_centerpoint_tvm/models"
+    mode: "755"
+    state: directory
+
+- name: Download centerpoint_encoder-x86_64-llvm-3.0.0-20221221.tar.gz
+  become: true
+  ansible.builtin.get_url:
+    url: https://autoware-modelzoo.s3.us-east-2.amazonaws.com/models/3.0.0-20221221/centerpoint_encoder-x86_64-llvm-3.0.0-20221221.tar.gz
+    dest: "{{ data_dir }}/lidar_centerpoint_tvm/centerpoint_encoder-x86_64-llvm-3.0.0-20221221.tar.gz"
+    mode: "644"
+    checksum: sha256:41e19de43f30549c325968aee06b4ad0d9701220be819e79d37efdfa86b918d0
+
+- name: Create centerpoint_encoder folder in lidar_centerpoint_tvm/models of {{ data_dir }}
+  ansible.builtin.file:
+    path: "{{ data_dir }}/lidar_centerpoint_tvm/models/centerpoint_encoder"
+    mode: "755"
+    state: directory
+
+- name: Extract centerpoint_encoder-x86_64-llvm-3.0.0-20221221.tar.gz
+  ansible.builtin.unarchive:
+    src: "{{ data_dir }}/lidar_centerpoint_tvm/centerpoint_encoder-x86_64-llvm-3.0.0-20221221.tar.gz"
+    dest: "{{ data_dir }}/lidar_centerpoint_tvm/models/centerpoint_encoder"
+
+- name: Download centerpoint_backbone-x86_64-llvm-3.0.0-20221221.tar.gz
+  become: true
+  ansible.builtin.get_url:
+    url: https://autoware-modelzoo.s3.us-east-2.amazonaws.com/models/3.0.0-20221221/centerpoint_backbone-x86_64-llvm-3.0.0-20221221.tar.gz
+    dest: "{{ data_dir }}/lidar_centerpoint_tvm/centerpoint_backbone-x86_64-llvm-3.0.0-20221221.tar.gz"
+    mode: "644"
+    checksum: sha256:3840b6b3590984e8115d66b12061aea3a2cfaed70b4e8d59457f04b0d6f6a1fc
+
+- name: Create centerpoint_backbone folder in lidar_centerpoint_tvm/models of {{ data_dir }}
+  ansible.builtin.file:
+    path: "{{ data_dir }}/lidar_centerpoint_tvm/models/centerpoint_backbone"
+    mode: "755"
+    state: directory
+
+- name: Extract centerpoint_backbone-x86_64-llvm-3.0.0-20221221.tar.gz
+  ansible.builtin.unarchive:
+    src: "{{ data_dir }}/lidar_centerpoint_tvm/centerpoint_backbone-x86_64-llvm-3.0.0-20221221.tar.gz"
+    dest: "{{ data_dir }}/lidar_centerpoint_tvm/models/centerpoint_backbone"
+
+# lidar_apollo_segmentation_tvm
+- name: Create lidar_apollo_segmentation_tvm/models directory inside {{ data_dir }}
+  ansible.builtin.file:
+    path: "{{ data_dir }}/lidar_apollo_segmentation_tvm/models"
+    mode: "755"
+    state: directory
+
+- name: Download baidu_cnn-x86_64-llvm-3.0.0-20221221.tar.gz
+  become: true
+  ansible.builtin.get_url:
+    url: https://autoware-modelzoo.s3.us-east-2.amazonaws.com/models/3.0.0-20221221/baidu_cnn-x86_64-llvm-3.0.0-20221221.tar.gz
+    dest: "{{ data_dir }}/lidar_apollo_segmentation_tvm/baidu_cnn-x86_64-llvm-3.0.0-20221221.tar.gz"
+    mode: "644"
+    checksum: sha256:4293e6196ec937d2cd5ec658e5ce70933647d2d38633a1805febb36cafd684e3
+
+- name: Create baidu_cnn folder in lidar_apollo_segmentation_tvm/models of {{ data_dir }}
+  ansible.builtin.file:
+    path: "{{ data_dir }}/lidar_apollo_segmentation_tvm/models/baidu_cnn"
+    mode: "755"
+    state: directory
+
+- name: Extract baidu_cnn-x86_64-llvm-3.0.0-20221221.tar.gz
+  ansible.builtin.unarchive:
+    src: "{{ data_dir }}/lidar_apollo_segmentation_tvm/baidu_cnn-x86_64-llvm-3.0.0-20221221.tar.gz"
+    dest: "{{ data_dir }}/lidar_apollo_segmentation_tvm/models/baidu_cnn"