- 
-
- -
Kinetics-400 上的动作识别
-
- 
-
+ -
NTURGB+D-120 上的基于人体姿态的动作识别
-
+
+ Kinetics-400 数据集行为识别结果（左） 和 NTU-RGB+D-120 数据集基于骨架的行为识别结果（右）
Kinetics-400 上的动作识别
-NTURGB+D-120 上的基于人体姿态的动作识别
-
+
+ Kinetics-400 数据集行为识别结果（左） 和 NTU-RGB+D-120 数据集基于骨架的行为识别结果（右）
Kinetics-400 上的基于 skeleton 的时空动作检测和动作识别
+Kinetics-400 数据集基于骨骼点的时空行为检测及视频行为识别结果
AVA-2.1 上的时空动作检测
+AVA-2.1 数据集时空行为检测结果
| 行为识别方法 | +行为识别 | ||||||||
| C3D (CVPR'2014) | @@ -148,138 +174,166 @@ pip install -v -e .MultiModality: Audio (ArXiv'2020) | TANet (ArXiv'2020) | TimeSformer (ICML'2021) | +ActionCLIP (ArXiv'2021) | VideoSwin (CVPR'2022) | -VideoMAE (NeurIPS'2022) | |||
| VideoMAE (NeurIPS'2022) | MViT V2 (CVPR'2022) | UniFormer V1 (ICLR'2022) | UniFormer V2 (Arxiv'2022) | -- | + | VideoMAE V2 (CVPR'2023) | |||
| 时序动作检测方法 | +时序动作定位 | ||||||||
| SSN (ICCV'2017) | BSN (ECCV'2018) | BMN (ICCV'2019) | +TCANet (CVPR'2021) | ||||||
| 时空动作检测方法 | +时空行为检测 | ||||||||
| ACRN (ECCV'2018) | SlowOnly+Fast R-CNN (ICCV'2019) | SlowFast+Fast R-CNN (ICCV'2019) | LFB (CVPR'2019) | -+ | VideoMAE (NeurIPS'2022) | ||||
| 基于骨骼点的动作识别方法 | +基于骨骼点的行为识别 | ||||||||
| ST-GCN (AAAI'2018) | 2s-AGCN (CVPR'2019) | PoseC3D (CVPR'2022) | STGCN++ (ArXiv'2022) | +CTRGCN (CVPR'2021) | +|||||
| MSG3D (CVPR'2020) | ++ | + | + | + | |||||
| 视频检索 | +|||||||||
| CLIP4Clip (ArXiv'2022) | ++ | + | |||||||
| 动作识别数据集 | +行为识别 | ||||||
| HMDB51 (Homepage) (ICCV'2011) | -UCF101 (Homepage) (CRCV-IR-12-01) | -ActivityNet (Homepage) (CVPR'2015) | -Kinetics-[400/600/700] (Homepage) (CVPR'2017) | +HMDB51 (官网) (ICCV'2011) | +UCF101 (官网) (CRCV-IR-12-01) | +ActivityNet (官网) (CVPR'2015) | +Kinetics-[400/600/700] (官网) (CVPR'2017) |
| SthV1 (ICCV'2017) | -SthV2 (Homepage) (ICCV'2017) | -Diving48 (Homepage) (ECCV'2018) | -Jester (Homepage) (ICCV'2019) | +SthV2 (官网) (ICCV'2017) | +Diving48 (官网) (ECCV'2018) | +Jester (官网) (ICCV'2019) | |
| Moments in Time (Homepage) (TPAMI'2019) | -Multi-Moments in Time (Homepage) (ArXiv'2019) | -HVU (Homepage) (ECCV'2020) | -OmniSource (Homepage) (ECCV'2020) | +Moments in Time (官网) (TPAMI'2019) | +Multi-Moments in Time (官网) (ArXiv'2019) | +HVU (官网) (ECCV'2020) | +OmniSource (官网) (ECCV'2020) |
| FineGYM (Homepage) (CVPR'2020) | -+ | FineGYM (官网) (CVPR'2020) | +Kinetics-710 (官网) (Arxiv'2022) | ||||
| 时序动作检测数据集 | +时序动作定位 | ||||||
| THUMOS14 (Homepage) (THUMOS Challenge 2014) | -ActivityNet (Homepage) (CVPR'2015) | -+ | THUMOS14 (官网) (THUMOS Challenge 2014) | +ActivityNet (官网) (CVPR'2015) | +HACS (官网) (ICCV'2019) | ||
| 时空动作检测数据集 | +时空行为检测 | +||||||
| UCF101-24* (官网) (CRCV-IR-12-01) | +JHMDB* (官网) (ICCV'2015) | +AVA (官网) (CVPR'2018) | +AVA-Kinetics (官网) (Arxiv'2020) | ||||
| UCF101-24* (Homepage) (CRCV-IR-12-01) | -JHMDB* (Homepage) (ICCV'2015) | -AVA (Homepage) (CVPR'2018) | -AVA-Kinetics (Homepage) (Arxiv'2020) | +基于骨架的行为识别 | |||
| 基于骨骼点的动作识别数据集 | +PoseC3D-FineGYM (官网) (ArXiv'2021) | +PoseC3D-NTURGB+D (官网) (ArXiv'2021) | +PoseC3D-UCF101 (官网) (ArXiv'2021) | +PoseC3D-HMDB51 (官网) (ArXiv'2021) | |||
| PoseC3D-FineGYM (Homepage) (ArXiv'2021) | -PoseC3D-NTURGB+D (Homepage) (ArXiv'2021) | -PoseC3D-UCF101 (Homepage) (ArXiv'2021) | -PoseC3D-HMDB51 (Homepage) (ArXiv'2021) | +视频检索 | +|||
| MSRVTT (官网) (CVPR'2016) | ++ | + | |||||
+
+
+
++ The page you are looking for cannot be found. +
++ If you just switched documentation versions, it is likely that the page you were on is moved. You can look for it in + the content table left, or go to the homepage. +
++ If you cannot find documentation you want, please open an issue to tell us! +
+ +{% endblock %} diff --git a/docs/en/advanced_guides/customize_dataset.md b/docs/en/advanced_guides/customize_dataset.md index 690ce3859c..f5ab6254ae 100644 --- a/docs/en/advanced_guides/customize_dataset.md +++ b/docs/en/advanced_guides/customize_dataset.md @@ -9,7 +9,7 @@ In this tutorial, we will introduce some methods about how to customize your own ## General understanding of the Dataset in MMAction2 -MMAction2 provides task-specific `Dataset` class, e.g. `VideoDataset`/`RawframeDataset` for action recognition, `AVADataset` for spatio-temporal action detection, `PoseDataset` for skeleton-based action recognition. These task-specific datasets only require the implementation of `load_data_list(self)` for generating a data list from the annotation file. The remaining functions are automatically handled by the superclass (i.e., `BaseActionDataset` and `BaseDataset`). The following table shows the inherent relationship and the main method of the modules. +MMAction2 provides task-specific `Dataset` class, e.g. `VideoDataset`/`RawframeDataset` for action recognition, `AVADataset` for spatio-temporal action detection, `PoseDataset` for skeleton-based action recognition. These task-specific datasets only require the implementation of `load_data_list(self)` for generating a data list from the annotation file. The remaining functions are automatically handled by the superclass (i.e., `BaseActionDataset` and `BaseDataset`). The following table shows the inheritance relationship and the main method of the modules. | Class Name | Class Method | | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/docs/en/advanced_guides/customize_logging.md b/docs/en/advanced_guides/customize_logging.md index ccbbeeafed..c097995daf 100644 --- a/docs/en/advanced_guides/customize_logging.md +++ b/docs/en/advanced_guides/customize_logging.md @@ -9,7 +9,7 @@ MMAction2 produces a lot of logs during the running process, such as loss, itera ## Flexible Logging System -The MMAction2 logging system is configured by the `LogProcessor` in [default_runtime](/configs/_base_/default_runtime.py) by default, which is equivalent to: +The MMAction2 logging system is configured by the `LogProcessor` in [default_runtime](https://github.com/open-mmlab/mmaction2/tree/main/configs/_base_/default_runtime.py) by default, which is equivalent to: ```python log_processor = dict(type='LogProcessor', window_size=20, by_epoch=True) diff --git a/docs/en/advanced_guides/customize_models.md b/docs/en/advanced_guides/customize_models.md index 3d8c0e1d4e..37e4ca399f 100644 --- a/docs/en/advanced_guides/customize_models.md +++ b/docs/en/advanced_guides/customize_models.md @@ -1 +1,3 @@ # Customize Models + +coming soon... diff --git a/docs/en/conf.py b/docs/en/conf.py index 6623d99b45..14a19907db 100644 --- a/docs/en/conf.py +++ b/docs/en/conf.py @@ -42,8 +42,17 @@ def get_version(): # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ - 'sphinx.ext.autodoc', 'sphinx.ext.napoleon', 'sphinx.ext.viewcode', - 'sphinx_markdown_tables', 'sphinx_copybutton', 'myst_parser' + 'sphinx.ext.autodoc', + 'sphinx.ext.autosummary', + 'sphinx.ext.intersphinx', + 'sphinx.ext.napoleon', + 'sphinx.ext.viewcode', + 'myst_parser', + 'sphinx_markdown_tables', + 'sphinx_copybutton', + 'sphinx_tabs.tabs', + 'notfound.extension', + 'sphinxcontrib.jquery', ] # numpy and torch are required @@ -90,26 +99,35 @@ def get_version(): { 'name': 'Upstream', - 'children': [ - { - 'name': 'MMCV', - 'url': 'https://github.com/open-mmlab/mmcv', - 'description': 'Foundational library for computer vision' - }, - { - 'name': - 'MMClassification', - 'url': - 'https://github.com/open-mmlab/mmclassification', - 'description': - 'Open source image classification toolbox based on PyTorch' - }, - { - 'name': 'MMDetection', - 'url': 'https://github.com/open-mmlab/mmdetection', - 'description': 'Object detection toolbox and benchmark' - }, - ] + 'children': [{ + 'name': + 'MMCV', + 'url': + 'https://github.com/open-mmlab/mmcv', + 'description': + 'Foundational library for computer vision' + }, { + 'name': + 'MMPreTrain', + 'url': + 'https://github.com/open-mmlab/mmpretrain', + 'description': + 'Open source pre-training toolbox based on PyTorch' + }, { + 'name': + 'MMDetection', + 'url': + 'https://github.com/open-mmlab/mmdetection', + 'description': + 'Object detection toolbox and benchmark' + }, { + 'name': + 'MMPose', + 'url': + 'https://github.com/open-mmlab/mmpose', + 'description': + 'Open-source toolbox for pose estimation based on PyTorch.' + }] }, ], # Specify the language of shared menu @@ -121,15 +139,29 @@ def get_version(): master_doc = 'index' html_static_path = ['_static'] -html_css_files = ['css/readthedocs.css'] +html_css_files = [ + 'https://cdn.datatables.net/v/bs4/dt-1.12.1/datatables.min.css', + 'css/readthedocs.css' +] +html_js_files = [ + 'https://cdn.datatables.net/v/bs4/dt-1.12.1/datatables.min.js', + 'js/custom.js' +] myst_enable_extensions = ['colon_fence'] myst_heading_anchors = 3 +# The not found page +notfound_template = '404.html' + def builder_inited_handler(app): - subprocess.run(['bash', './merge_docs.sh']) - subprocess.run(['python', './stat.py']) + if subprocess.run(['python', './stat.py']).returncode != 0: + raise RuntimeError('Failed to run the script `stat.py`.') + if subprocess.run(['python', './project_zoo.py']).returncode != 0: + raise RuntimeError('Failed to run the script `project_zoo.py`.') + if subprocess.run(['python', './dataset_zoo.py']).returncode != 0: + raise RuntimeError('Failed to run the script `dataset_zoo.py`.') def setup(app): diff --git a/docs/en/dataset_zoo.py b/docs/en/dataset_zoo.py new file mode 100644 index 0000000000..ce89980477 --- /dev/null +++ b/docs/en/dataset_zoo.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python +import re +from pathlib import Path + +from utils import replace_link + +DATASETS_ROOT = Path('dataset_zoo') # Path to save generated paper pages. +MODELZOO_TEMPLATE = """\ +# Dataset Zoo Summary + +In this page, we list [all datasets](#all-supported-datasets) we support. You can click the link to jump to the corresponding dataset pages. + +## All supported datasets + +* Number of datasets: {num_datasets} +{dataset_msg} + +""" # noqa: E501 + + +def generate_datasets_pages(): + dataset_list = Path('../../tools/data').glob('*/README.md') + num_datasets = 0 + dataset_msgs = [] + + for file in dataset_list: + num_datasets += 1 + + copy = DATASETS_ROOT / file.parent.with_suffix('.md').name + + with open(file, 'r') as f: + content = f.read() + + title = re.match(r'^# Preparing (.*)', content).group(1) + content = replace_link(r'\[([^\]]+)\]\(([^)]+)\)', '[{}]({})', content, + file) + content = replace_link(r'\[([^\]]+)\]: (.*)', '[{}]: {}', content, + file) + dataset_msgs.append(f'\t - [{title}]({copy})') + + with open(copy, 'w') as f: + f.write(content) + + dataset_msg = '\n'.join(dataset_msgs) + + modelzoo = MODELZOO_TEMPLATE.format( + num_datasets=num_datasets, + dataset_msg=dataset_msg, + ) + + with open('datasetzoo_statistics.md', 'w') as f: + f.write(modelzoo) + + +DATASETS_ROOT.mkdir(exist_ok=True) +generate_datasets_pages() diff --git a/docs/en/docutils.conf b/docs/en/docutils.conf new file mode 100644 index 0000000000..0c00c84688 --- /dev/null +++ b/docs/en/docutils.conf @@ -0,0 +1,2 @@ +[html writers] +table_style: colwidths-auto diff --git a/docs/en/get_started/contribution_guide.md b/docs/en/get_started/contribution_guide.md index 3748329873..5e28623a77 100644 --- a/docs/en/get_started/contribution_guide.md +++ b/docs/en/get_started/contribution_guide.md @@ -5,12 +5,12 @@ All kinds of contributions are welcome, including but not limited to the followi - Fixes (typo, bugs) - New features and components - Add documentation or translate the documentation into other languages -- Add new project (Recommended) about video understanding algorithm with less restriction, refer to [here](/projects/README.md) for details +- Add new project (Recommended) about video understanding algorithm with less restriction, refer to [here](../projectzoo.md) for details ## Workflow 1. Fork and pull the latest mmaction2 -2. Checkout a new branch with a meaningful name (do not use master branch for PRs) +2. Checkout a new branch with a meaningful name (do not use main branch for PRs) 3. Commit your changes 4. Create a PR diff --git a/docs/en/get_started/faq.md b/docs/en/get_started/faq.md index 45d1b6889a..08679a8349 100644 --- a/docs/en/get_started/faq.md +++ b/docs/en/get_started/faq.md @@ -12,7 +12,7 @@ We list some common issues faced by many users and their corresponding solutions - [Testing](#testing) Feel free to enrich the list if you find any frequent issues and have ways to help others to solve them. -If the contents here do not cover your issue, please create an issue using the [provided templates](/.github/ISSUE_TEMPLATE/error-report.md) and make sure to fill in all required information in the template. +If the contents here do not cover your issue, please create an issue using the [provided templates](https://github.com/open-mmlab/mmaction2/tree/main/.github/ISSUE_TEMPLATE/error-report.md) and make sure to fill in all required information in the template. ## Installation @@ -88,7 +88,7 @@ If the contents here do not cover your issue, please create an issue using the [ - **How to set `load_from` value in config files to finetune models?** - In MMAction2, We set `load_from=None` as default in `configs/_base_/default_runtime.py` and owing to [inheritance design](/docs/en/user_guides/config.md), + In MMAction2, We set `load_from=None` as default in `configs/_base_/default_runtime.py` and owing to [inheritance design](https://github.com/open-mmlab/mmaction2/tree/main/docs/en/user_guides/config.md), users can directly change it by setting `load_from` in their configs. - **How to use `RawFrameDataset` for training?** diff --git a/docs/en/get_started/installation.md b/docs/en/get_started/installation.md index 18b0545f64..8cc64b7798 100644 --- a/docs/en/get_started/installation.md +++ b/docs/en/get_started/installation.md @@ -4,7 +4,7 @@ In this section we demonstrate how to prepare an environment with PyTorch. -MMAction2 works on Linux, Windows and macOS. It requires Python 3.7+, CUDA 9.2+ and PyTorch 1.6+. +MMAction2 works on Linux, Windows and macOS. It requires Python 3.7+, CUDA 10.2+ and PyTorch 1.8+. ```{note} If you are experienced with PyTorch and have already installed it, just skip this part and jump to the [next section](#installation). Otherwise, you can follow these steps for the preparation. @@ -185,7 +185,7 @@ We provide a [Dockerfile](https://github.com/open-mmlab/mmaction2/blob/main/dock to build an image. Ensure that your [docker version](https://docs.docker.com/engine/install/) >=19.03. ```shell -# build an image with PyTorch 1.6.0, CUDA 10.1, CUDNN 7. +# build an image with PyTorch 1.8.1, CUDA 10.2, CUDNN 7. # If you prefer other versions, just modified the Dockerfile docker build -f ./docker/Dockerfile --rm -t mmaction2 . ``` diff --git a/docs/en/get_started/overview.md b/docs/en/get_started/overview.md index 498b093830..3eae38e089 100644 --- a/docs/en/get_started/overview.md +++ b/docs/en/get_started/overview.md @@ -77,8 +77,8 @@ We have prepared a wealth of documents to meet your various needs:+ 未找到你要打开的页面。 +
++ 如果你是从旧版本文档跳转至此，可能是对应的页面被移动了。请从左侧的目录中寻找新版本文档，或者跳转至首页。 +
++ 如果你找不到希望打开的文档，欢迎在 Issue 中告诉我们！ +
+ +{% endblock %} diff --git a/docs/zh_cn/advanced_guides/customize_dataset.md b/docs/zh_cn/advanced_guides/customize_dataset.md new file mode 100644 index 0000000000..7749a484a3 --- /dev/null +++ b/docs/zh_cn/advanced_guides/customize_dataset.md @@ -0,0 +1,126 @@ +# 自定义数据集 + +在本教程中，我们将介绍如何通过在线转换来自定义你的数据集。 + +- [自定义数据集](#自定义数据集) + - [MMAction2 数据集概述](#mmaction2-数据集概述) + - [定制新的数据集](#定制新的数据集) + - [为 PoseDataset 自定义关键点格式](#为-posedataset-自定义关键点格式) + +## MMAction2 数据集概述 + +MMAction2 提供了任务特定的 `Dataset` 类，例如用于动作识别的 `VideoDataset`/`RawframeDataset`，用于时空动作检测的 `AVADataset`，用于基于骨骼的动作识别的`PoseDataset`。这些任务特定的数据集只需要实现 `load_data_list(self)` 来从注释文件生成数据列表。剩下的函数由超类（即 `BaseActionDataset` 和 `BaseDataset`）自动处理。下表显示了模块的继承关系和主要方法。 + +| 类名 | 类方法 | +| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `MMAction2::VideoDataset` | `load_data_list(self)`|
+ 行为识别 |
+ + 基于骨架的行为识别 |
+
|
+ 时空动作检测 |
+ + 时空动作检测 |
+
+
+
+| 旧版本 | ++ +```python +data = dict( + videos_per_gpu=32, + workers_per_gpu=2, + train=dict(...), + val=dict(...), + test=dict(...), +) +``` + + | +
| 新版本 | ++ +```python +train_dataloader = dict( + batch_size=32, + num_workers=2, + dataset=dict(...), + sampler=dict(type='DefaultSampler', shuffle=True) # 必要 +) + +val_dataloader = dict( + batch_size=32, + num_workers=2, + dataset=dict(...), + sampler=dict(type='DefaultSampler', shuffle=False) # 必要 +) + +test_dataloader = val_dataloader +``` + + | +
| 旧版本 | ++ +```python + +train_pipeline = [ + dict(type='DecordInit'), + dict(type='SampleFrames', clip_len=1, frame_interval=1, num_clips=8), + dict(type='DecordDecode'), + dict(type='Resize', scale=(-1, 256)), + dict( + type='MultiScaleCrop', + input_size=224, + scales=(1, 0.875, 0.75, 0.66), + random_crop=False, + max_wh_scale_gap=1), + dict(type='Resize', scale=(224, 224), keep_ratio=False), + dict(type='Flip', flip_ratio=0.5), + dict(type='Normalize', **img_norm_cfg), + dict(type='FormatShape', input_format='NCHW'), + dict(type='Collect', keys=['imgs', 'label'], meta_keys=[]), + dict(type='ToTensor', keys=['imgs', 'label']) +] +``` + + | +
| 新版本 | ++ +```python +model.data_preprocessor = dict( + mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=False) + +train_pipeline = [ + dict(type='DecordInit'), + dict(type='SampleFrames', clip_len=1, frame_interval=1, num_clips=5), + dict(type='DecordDecode'), + dict(type='Resize', scale=(-1, 256)), + dict( + type='MultiScaleCrop', + input_size=224, + scales=(1, 0.875, 0.75, 0.66), + random_crop=False, + max_wh_scale_gap=1), + dict(type='Resize', scale=(224, 224), keep_ratio=False), + dict(type='Flip', flip_ratio=0.5), + dict(type='FormatShape', input_format='NCHW'), + dict(type='PackActionInputs') +] +``` + + | +
| 旧版本 | ++ +```python +evaluation = dict( + interval=5, + metrics=['top_k_accuracy', 'mean_class_accuracy']) +``` + + | +
| 新版本 | ++ +```python +val_evaluator = dict( + type='AccMetric', + metric_list=('top_k_accuracy', 'mean_class_accuracy')) +test_evaluator = val_evaluator +``` + + | +
| 旧版本 | ++ +```python +optimizer = dict( + type='AdamW', + lr=0.0015, + weight_decay=0.3, + paramwise_cfg = dict( + norm_decay_mult=0.0, + bias_decay_mult=0.0, + )) + +optimizer_config = dict(grad_clip=dict(max_norm=1.0)) +``` + + | +
| 新版本 | ++ +```python +optim_wrapper = dict( + optimizer=dict(type='AdamW', lr=0.0015, weight_decay=0.3), + paramwise_cfg = dict( + norm_decay_mult=0.0, + bias_decay_mult=0.0, + ), + clip_gard=dict(max_norm=1.0), +) +``` + + | +
| 旧版本 | ++ +```python +lr_config = dict( + policy='CosineAnnealing', + min_lr=0, + warmup='linear', + warmup_iters=5, + warmup_ratio=0.01, + warmup_by_epoch=True) +``` + + | +
| 新版本 | ++ +```python +param_scheduler = [ + # 学习率预热 + dict( + type='LinearLR', + start_factor=0.01, + by_epoch=True, + end=5, + # 在每个迭代后更新学习率。 + convert_to_iter_based=True), + # 主要的学习率策略 + dict(type='CosineAnnealingLR', by_epoch=True, begin=5), +] +``` + + | +
| 旧版本 | ++ +```python +runner = dict(type='EpochBasedRunner', max_epochs=100) +``` + + | +
| 新版本 | ++ +```python +# `val_interval` 是原 `evaluation.interval`。 +train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=100, val_begin=1, val_interval=1) +val_cfg = dict(type='ValLoop') # 使用默认验证循环。 +test_cfg = dict(type='TestLoop') # 使用默认测试循环。 +``` + + | +
| 旧版本 | ++ +```python +log_config = dict( + interval=100, + hooks=[ + dict(type='TextLoggerHook'), + dict(type='TensorboardLoggerHook'), + ]) +``` + + | +
| 新版本 | ++ +```python +default_hooks = dict( + ... + logger=dict(type='LoggerHook', interval=100), +) + +visualizer = dict( + type='ActionVisualizer', + vis_backends=[dict(type='LocalVisBackend'), dict(type='TensorboardVisBackend')], +) +``` + + | +
+![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ff6iCPqqooZp"
+ },
+ "source": [
+ "# Spatio-temporal action detection with MMAction2\n",
+ "Welcome to MMAction2! This is a tutorial on how to use MMAction2 for spatio-temporal action detection. In this tutorial, we will use the MultiSports dataset as an example, and provide a complete step-by-step guide for spatio-temporal action detection, including\n",
+ "- Prepare spatio-temporal action detection dataset\n",
+ "- Train detection model\n",
+ "- Prepare AVA format dataset\n",
+ "- Train spatio-temporal action detection model\n"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xQlffdn7ooZq"
+ },
+ "source": [
+ "## 0. Install MMAction2 and MMDetection"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "4vWjBJI-ooZr",
+ "outputId": "1c852c24-eb40-407d-e1c4-72d4b43385a3"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
+ "Collecting openmim\n",
+ " Downloading openmim-0.3.7-py2.py3-none-any.whl (51 kB)\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m51.3/51.3 kB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25hRequirement already satisfied: Click in /usr/local/lib/python3.10/dist-packages (from openmim) (8.1.3)\n",
+ "Collecting colorama (from openmim)\n",
+ " Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n",
+ "Collecting model-index (from openmim)\n",
+ " Downloading model_index-0.1.11-py3-none-any.whl (34 kB)\n",
+ "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from openmim) (1.5.3)\n",
+ "Requirement already satisfied: pip>=19.3 in /usr/local/lib/python3.10/dist-packages (from openmim) (23.1.2)\n",
+ "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from openmim) (2.27.1)\n",
+ "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from openmim) (13.3.4)\n",
+ "Requirement already satisfied: tabulate in /usr/local/lib/python3.10/dist-packages (from openmim) (0.8.10)\n",
+ "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from model-index->openmim) (6.0)\n",
+ "Requirement already satisfied: markdown in /usr/local/lib/python3.10/dist-packages (from model-index->openmim) (3.4.3)\n",
+ "Collecting ordered-set (from model-index->openmim)\n",
+ " Downloading ordered_set-4.1.0-py3-none-any.whl (7.6 kB)\n",
+ "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->openmim) (2.8.2)\n",
+ "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->openmim) (2022.7.1)\n",
+ "Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.10/dist-packages (from pandas->openmim) (1.22.4)\n",
+ "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->openmim) (1.26.15)\n",
+ "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->openmim) (2022.12.7)\n",
+ "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->openmim) (2.0.12)\n",
+ "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->openmim) (3.4)\n",
+ "Requirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->openmim) (2.2.0)\n",
+ "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->openmim) (2.14.0)\n",
+ "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py<3.0.0,>=2.2.0->rich->openmim) (0.1.2)\n",
+ "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas->openmim) (1.16.0)\n",
+ "Installing collected packages: ordered-set, colorama, model-index, openmim\n",
+ "Successfully installed colorama-0.4.6 model-index-0.1.11 openmim-0.3.7 ordered-set-4.1.0\n",
+ "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
+ "Looking in links: https://download.openmmlab.com/mmcv/dist/cu118/torch2.0.0/index.html\n",
+ "Collecting mmengine\n",
+ " Downloading mmengine-0.7.4-py3-none-any.whl (374 kB)\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m374.3/374.3 kB\u001b[0m \u001b[31m17.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25hCollecting addict (from mmengine)\n",
+ " Downloading addict-2.4.0-py3-none-any.whl (3.8 kB)\n",
+ "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from mmengine) (3.7.1)\n",
+ "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from mmengine) (1.22.4)\n",
+ "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from mmengine) (6.0)\n",
+ "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from mmengine) (13.3.4)\n",
+ "Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from mmengine) (2.3.0)\n",
+ "Collecting yapf (from mmengine)\n",
+ " Downloading yapf-0.40.0-py3-none-any.whl (250 kB)\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m250.3/250.3 kB\u001b[0m \u001b[31m28.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25hRequirement already satisfied: opencv-python>=3 in /usr/local/lib/python3.10/dist-packages (from mmengine) (4.7.0.72)\n",
+ "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (1.0.7)\n",
+ "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (0.11.0)\n",
+ "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (4.39.3)\n",
+ "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (1.4.4)\n",
+ "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (23.1)\n",
+ "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (8.4.0)\n",
+ "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (3.0.9)\n",
+ "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (2.8.2)\n",
+ "Requirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->mmengine) (2.2.0)\n",
+ "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->mmengine) (2.14.0)\n",
+ "Collecting importlib-metadata>=6.6.0 (from yapf->mmengine)\n",
+ " Downloading importlib_metadata-6.6.0-py3-none-any.whl (22 kB)\n",
+ "Collecting platformdirs>=3.5.1 (from yapf->mmengine)\n",
+ " Downloading platformdirs-3.5.3-py3-none-any.whl (15 kB)\n",
+ "Requirement already satisfied: tomli>=2.0.1 in /usr/local/lib/python3.10/dist-packages (from yapf->mmengine) (2.0.1)\n",
+ "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata>=6.6.0->yapf->mmengine) (3.15.0)\n",
+ "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py<3.0.0,>=2.2.0->rich->mmengine) (0.1.2)\n",
+ "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib->mmengine) (1.16.0)\n",
+ "Installing collected packages: addict, platformdirs, importlib-metadata, yapf, mmengine\n",
+ " Attempting uninstall: platformdirs\n",
+ " Found existing installation: platformdirs 3.3.0\n",
+ " Uninstalling platformdirs-3.3.0:\n",
+ " Successfully uninstalled platformdirs-3.3.0\n",
+ "Successfully installed addict-2.4.0 importlib-metadata-6.6.0 mmengine-0.7.4 platformdirs-3.5.3 yapf-0.40.0\n",
+ "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
+ "Looking in links: https://download.openmmlab.com/mmcv/dist/cu118/torch2.0.0/index.html\n",
+ "Collecting mmcv\n",
+ " Downloading https://download.openmmlab.com/mmcv/dist/cu118/torch2.0.0/mmcv-2.0.0-cp310-cp310-manylinux1_x86_64.whl (74.4 MB)\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m74.4/74.4 MB\u001b[0m \u001b[31m8.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25hRequirement already satisfied: addict in /usr/local/lib/python3.10/dist-packages (from mmcv) (2.4.0)\n",
+ "Requirement already satisfied: mmengine>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from mmcv) (0.7.4)\n",
+ "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from mmcv) (1.22.4)\n",
+ "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from mmcv) (23.1)\n",
+ "Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from mmcv) (8.4.0)\n",
+ "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from mmcv) (6.0)\n",
+ "Requirement already satisfied: yapf in /usr/local/lib/python3.10/dist-packages (from mmcv) (0.40.0)\n",
+ "Requirement already satisfied: opencv-python>=3 in /usr/local/lib/python3.10/dist-packages (from mmcv) (4.7.0.72)\n",
+ "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from mmengine>=0.2.0->mmcv) (3.7.1)\n",
+ "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from mmengine>=0.2.0->mmcv) (13.3.4)\n",
+ "Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from mmengine>=0.2.0->mmcv) (2.3.0)\n",
+ "Requirement already satisfied: importlib-metadata>=6.6.0 in /usr/local/lib/python3.10/dist-packages (from yapf->mmcv) (6.6.0)\n",
+ "Requirement already satisfied: platformdirs>=3.5.1 in /usr/local/lib/python3.10/dist-packages (from yapf->mmcv) (3.5.3)\n",
+ "Requirement already satisfied: tomli>=2.0.1 in /usr/local/lib/python3.10/dist-packages (from yapf->mmcv) (2.0.1)\n",
+ "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata>=6.6.0->yapf->mmcv) (3.15.0)\n",
+ "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine>=0.2.0->mmcv) (1.0.7)\n",
+ "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine>=0.2.0->mmcv) (0.11.0)\n",
+ "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine>=0.2.0->mmcv) (4.39.3)\n",
+ "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine>=0.2.0->mmcv) (1.4.4)\n",
+ "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine>=0.2.0->mmcv) (3.0.9)\n",
+ "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine>=0.2.0->mmcv) (2.8.2)\n",
+ "Requirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->mmengine>=0.2.0->mmcv) (2.2.0)\n",
+ "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->mmengine>=0.2.0->mmcv) (2.14.0)\n",
+ "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py<3.0.0,>=2.2.0->rich->mmengine>=0.2.0->mmcv) (0.1.2)\n",
+ "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib->mmengine>=0.2.0->mmcv) (1.16.0)\n",
+ "Installing collected packages: mmcv\n",
+ "Successfully installed mmcv-2.0.0\n",
+ "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
+ "Looking in links: https://download.openmmlab.com/mmcv/dist/cu118/torch2.0.0/index.html\n",
+ "Collecting mmdet\n",
+ " Downloading mmdet-3.0.0-py3-none-any.whl (1.7 MB)\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m17.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from mmdet) (3.7.1)\n",
+ "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from mmdet) (1.22.4)\n",
+ "Requirement already satisfied: pycocotools in /usr/local/lib/python3.10/dist-packages (from mmdet) (2.0.6)\n",
+ "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from mmdet) (1.10.1)\n",
+ "Requirement already satisfied: shapely in /usr/local/lib/python3.10/dist-packages (from mmdet) (2.0.1)\n",
+ "Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from mmdet) (1.16.0)\n",
+ "Collecting terminaltables (from mmdet)\n",
+ " Downloading terminaltables-3.1.10-py2.py3-none-any.whl (15 kB)\n",
+ "Requirement already satisfied: mmcv<2.1.0,>=2.0.0rc4 in /usr/local/lib/python3.10/dist-packages (from mmdet) (2.0.0)\n",
+ "Requirement already satisfied: mmengine<1.0.0,>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from mmdet) (0.7.4)\n",
+ "Requirement already satisfied: addict in /usr/local/lib/python3.10/dist-packages (from mmcv<2.1.0,>=2.0.0rc4->mmdet) (2.4.0)\n",
+ "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from mmcv<2.1.0,>=2.0.0rc4->mmdet) (23.1)\n",
+ "Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from mmcv<2.1.0,>=2.0.0rc4->mmdet) (8.4.0)\n",
+ "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from mmcv<2.1.0,>=2.0.0rc4->mmdet) (6.0)\n",
+ "Requirement already satisfied: yapf in /usr/local/lib/python3.10/dist-packages (from mmcv<2.1.0,>=2.0.0rc4->mmdet) (0.40.0)\n",
+ "Requirement already satisfied: opencv-python>=3 in /usr/local/lib/python3.10/dist-packages (from mmcv<2.1.0,>=2.0.0rc4->mmdet) (4.7.0.72)\n",
+ "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from mmengine<1.0.0,>=0.7.1->mmdet) (13.3.4)\n",
+ "Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from mmengine<1.0.0,>=0.7.1->mmdet) (2.3.0)\n",
+ "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmdet) (1.0.7)\n",
+ "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmdet) (0.11.0)\n",
+ "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmdet) (4.39.3)\n",
+ "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmdet) (1.4.4)\n",
+ "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmdet) (3.0.9)\n",
+ "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmdet) (2.8.2)\n",
+ "Requirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->mmengine<1.0.0,>=0.7.1->mmdet) (2.2.0)\n",
+ "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->mmengine<1.0.0,>=0.7.1->mmdet) (2.14.0)\n",
+ "Requirement already satisfied: importlib-metadata>=6.6.0 in /usr/local/lib/python3.10/dist-packages (from yapf->mmcv<2.1.0,>=2.0.0rc4->mmdet) (6.6.0)\n",
+ "Requirement already satisfied: platformdirs>=3.5.1 in /usr/local/lib/python3.10/dist-packages (from yapf->mmcv<2.1.0,>=2.0.0rc4->mmdet) (3.5.3)\n",
+ "Requirement already satisfied: tomli>=2.0.1 in /usr/local/lib/python3.10/dist-packages (from yapf->mmcv<2.1.0,>=2.0.0rc4->mmdet) (2.0.1)\n",
+ "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata>=6.6.0->yapf->mmcv<2.1.0,>=2.0.0rc4->mmdet) (3.15.0)\n",
+ "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py<3.0.0,>=2.2.0->rich->mmengine<1.0.0,>=0.7.1->mmdet) (0.1.2)\n",
+ "Installing collected packages: terminaltables, mmdet\n",
+ "Successfully installed mmdet-3.0.0 terminaltables-3.1.10\n",
+ "Cloning into 'mmaction2'...\n",
+ "remote: Enumerating objects: 22869, done.\u001b[K\n",
+ "remote: Counting objects: 100% (1491/1491), done.\u001b[K\n",
+ "remote: Compressing objects: 100% (800/800), done.\u001b[K\n",
+ "remote: Total 22869 (delta 855), reused 1176 (delta 686), pack-reused 21378\u001b[K\n",
+ "Receiving objects: 100% (22869/22869), 82.81 MiB | 15.42 MiB/s, done.\n",
+ "Resolving deltas: 100% (15954/15954), done.\n",
+ "/content/mmaction2\n",
+ "Using pip 23.1.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)\n",
+ "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
+ "Obtaining file:///content/mmaction2\n",
+ " Running command python setup.py egg_info\n",
+ " running egg_info\n",
+ " creating /tmp/pip-pip-egg-info-d1y_zlo6/mmaction2.egg-info\n",
+ " writing /tmp/pip-pip-egg-info-d1y_zlo6/mmaction2.egg-info/PKG-INFO\n",
+ " writing dependency_links to /tmp/pip-pip-egg-info-d1y_zlo6/mmaction2.egg-info/dependency_links.txt\n",
+ " writing requirements to /tmp/pip-pip-egg-info-d1y_zlo6/mmaction2.egg-info/requires.txt\n",
+ " writing top-level names to /tmp/pip-pip-egg-info-d1y_zlo6/mmaction2.egg-info/top_level.txt\n",
+ " writing manifest file '/tmp/pip-pip-egg-info-d1y_zlo6/mmaction2.egg-info/SOURCES.txt'\n",
+ " reading manifest file '/tmp/pip-pip-egg-info-d1y_zlo6/mmaction2.egg-info/SOURCES.txt'\n",
+ " reading manifest template 'MANIFEST.in'\n",
+ " warning: no files found matching 'mmaction/.mim/model-index.yml'\n",
+ " warning: no files found matching '*.py' under directory 'mmaction/.mim/configs'\n",
+ " warning: no files found matching '*.yml' under directory 'mmaction/.mim/configs'\n",
+ " warning: no files found matching '*.sh' under directory 'mmaction/.mim/tools'\n",
+ " warning: no files found matching '*.py' under directory 'mmaction/.mim/tools'\n",
+ " adding license file 'LICENSE'\n",
+ " writing manifest file '/tmp/pip-pip-egg-info-d1y_zlo6/mmaction2.egg-info/SOURCES.txt'\n",
+ " Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
+ "Collecting decord>=0.4.1 (from mmaction2==1.0.0)\n",
+ " Downloading decord-0.6.0-py3-none-manylinux2010_x86_64.whl (13.6 MB)\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.6/13.6 MB\u001b[0m \u001b[31m71.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25hCollecting einops (from mmaction2==1.0.0)\n",
+ " Downloading einops-0.6.1-py3-none-any.whl (42 kB)\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.2/42.2 kB\u001b[0m \u001b[31m4.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from mmaction2==1.0.0) (3.7.1)\n",
+ "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from mmaction2==1.0.0) (1.22.4)\n",
+ "Requirement already satisfied: opencv-contrib-python in /usr/local/lib/python3.10/dist-packages (from mmaction2==1.0.0) (4.7.0.72)\n",
+ "Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from mmaction2==1.0.0) (8.4.0)\n",
+ "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from mmaction2==1.0.0) (1.10.1)\n",
+ "Requirement already satisfied: torch>=1.3 in /usr/local/lib/python3.10/dist-packages (from mmaction2==1.0.0) (2.0.1+cu118)\n",
+ "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.3->mmaction2==1.0.0) (3.12.0)\n",
+ "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch>=1.3->mmaction2==1.0.0) (4.5.0)\n",
+ "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.3->mmaction2==1.0.0) (1.11.1)\n",
+ "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.3->mmaction2==1.0.0) (3.1)\n",
+ "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.3->mmaction2==1.0.0) (3.1.2)\n",
+ "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.3->mmaction2==1.0.0) (2.0.0)\n",
+ "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.3->mmaction2==1.0.0) (3.25.2)\n",
+ "Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.3->mmaction2==1.0.0) (16.0.5)\n",
+ "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmaction2==1.0.0) (1.0.7)\n",
+ "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmaction2==1.0.0) (0.11.0)\n",
+ "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmaction2==1.0.0) (4.39.3)\n",
+ "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmaction2==1.0.0) (1.4.4)\n",
+ "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmaction2==1.0.0) (23.1)\n",
+ "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmaction2==1.0.0) (3.0.9)\n",
+ "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmaction2==1.0.0) (2.8.2)\n",
+ "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib->mmaction2==1.0.0) (1.16.0)\n",
+ "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.3->mmaction2==1.0.0) (2.1.2)\n",
+ "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.3->mmaction2==1.0.0) (1.3.0)\n",
+ "Installing collected packages: einops, decord, mmaction2\n",
+ " Running setup.py develop for mmaction2\n",
+ " Running command python setup.py develop\n",
+ " running develop\n",
+ " /usr/local/lib/python3.10/dist-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.\n",
+ " !!\n",
+ "\n",
+ " ********************************************************************************\n",
+ " Please avoid running ``setup.py`` and ``easy_install``.\n",
+ " Instead, use pypa/build, pypa/installer, pypa/build or\n",
+ " other standards-based tools.\n",
+ "\n",
+ " See https://github.com/pypa/setuptools/issues/917 for details.\n",
+ " ********************************************************************************\n",
+ "\n",
+ " !!\n",
+ " easy_install.initialize_options(self)\n",
+ " /usr/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.\n",
+ " !!\n",
+ "\n",
+ " ********************************************************************************\n",
+ " Please avoid running ``setup.py`` directly.\n",
+ " Instead, use pypa/build, pypa/installer, pypa/build or\n",
+ " other standards-based tools.\n",
+ "\n",
+ " See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.\n",
+ " ********************************************************************************\n",
+ "\n",
+ " !!\n",
+ " self.initialize_options()\n",
+ " running egg_info\n",
+ " creating mmaction2.egg-info\n",
+ " writing mmaction2.egg-info/PKG-INFO\n",
+ " writing dependency_links to mmaction2.egg-info/dependency_links.txt\n",
+ " writing requirements to mmaction2.egg-info/requires.txt\n",
+ " writing top-level names to mmaction2.egg-info/top_level.txt\n",
+ " writing manifest file 'mmaction2.egg-info/SOURCES.txt'\n",
+ " reading manifest file 'mmaction2.egg-info/SOURCES.txt'\n",
+ " reading manifest template 'MANIFEST.in'\n",
+ " adding license file 'LICENSE'\n",
+ " writing manifest file 'mmaction2.egg-info/SOURCES.txt'\n",
+ " running build_ext\n",
+ " Creating /usr/local/lib/python3.10/dist-packages/mmaction2.egg-link (link to .)\n",
+ " Adding mmaction2 1.0.0 to easy-install.pth file\n",
+ "\n",
+ " Installed /content/mmaction2\n",
+ "Successfully installed decord-0.6.0 einops-0.6.1 mmaction2-1.0.0\n",
+ "/content/mmaction2/projects/stad_tutorial\n"
+ ]
+ }
+ ],
+ "source": [
+ "%pip install -U openmim\n",
+ "!mim install mmengine\n",
+ "!mim install mmcv\n",
+ "!mim install mmdet\n",
+ "\n",
+ "!git clone https://github.com/open-mmlab/mmaction2.git\n",
+ "\n",
+ "%cd mmaction2\n",
+ "%pip install -v -e .\n",
+ "%cd projects/stad_tutorial"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Ox0TM64FooZt"
+ },
+ "source": [
+ "## 1. Prepare spatio-temporal action detection dataset\n",
+ "\n",
+ "Similar to detection tasks that require bounding box annotations, spatio-temporal action detection tasks require temporal and spatial localization, so more complex tube annotations are required. Taking the MultiSports dataset as an example, the `gttubes` field provides all the target action annotations in the video, and the following is an annotation fragment:\n",
+ "\n",
+ "```\n",
+ " 'gttubes': {\n",
+ " 'aerobic_gymnastics/v_aqMgwPExjD0_c001': # video_key\n",
+ " {\n",
+ " 10: # label index\n",
+ " [\n",
+ " array([[ 377., 904., 316., 1016., 584.], # 1st tube of class 10\n",
+ " [ 378., 882., 315., 1016., 579.], # shape (n, 5): n frames，each annotation includes (frame idx，x1，y1, x2, y2)\n",
+ " ...\n",
+ " [ 398., 861., 304., 954., 549.]], dtype=float32)，\n",
+ "\n",
+ " array([[ 399., 881., 308., 955., 542.], # 2nd tube of class 10\n",
+ " [ 400., 862., 303., 988., 539.],\n",
+ " [ 401., 853., 292., 1000., 535.],\n",
+ " ...])\n",
+ " ...\n",
+ "\n",
+ " ] ,\n",
+ " 9: # label index\n",
+ " [\n",
+ " array(...), # 1st tube of class 9\n",
+ " array(...), # 2nd tube of class 9\n",
+ " ...\n",
+ " ]\n",
+ " ...\n",
+ " }\n",
+ " }\n",
+ "```\n",
+ "\n",
+ "The annotation file also needs to provide other field information, and the complete ground truth file includes the following information:\n",
+ "\n",
+ "```\n",
+ "{\n",
+ " 'labels': # label list\n",
+ " ['aerobic push up', 'aerobic explosive push up', ...],\n",
+ " 'train_videos': # training video list\n",
+ " [\n",
+ " [\n",
+ " 'aerobic_gymnastics/v_aqMgwPExjD0_c001',\n",
+ " 'aerobic_gymnastics/v_yaKOumdXwbU_c019',\n",
+ " ...\n",
+ " ]\n",
+ " ]\n",
+ " 'test_videos': # test video list\n",
+ " [\n",
+ " [\n",
+ " 'aerobic_gymnastics/v_crsi07chcV8_c004',\n",
+ " 'aerobic_gymnastics/v_dFYr67eNMwA_c005',\n",
+ " ...\n",
+ " ]\n",
+ " ]\n",
+ " 'n_frames': # dict provides frame number of each video\n",
+ " {\n",
+ " 'aerobic_gymnastics/v_crsi07chcV8_c004': 725,\n",
+ " 'aerobic_gymnastics/v_dFYr67eNMwA_c005': 750,\n",
+ " ...\n",
+ " }\n",
+ " 'resolution': # dict provides resolution of each video\n",
+ " {\n",
+ " 'aerobic_gymnastics/v_crsi07chcV8_c004': (720, 1280),\n",
+ " 'aerobic_gymnastics/v_dFYr67eNMwA_c005': (720, 1280),\n",
+ " ...\n",
+ " }\n",
+ " 'gt_tubes': # dict provides bouding boxes of each tube\n",
+ " {\n",
+ " ... # refer to above description\n",
+ " }\n",
+ "}\n",
+ "```\n",
+ "\n",
+ "The subsequent experiments are based on MultiSports-tiny, we extracted a small number of videos from MultiSports for demonstration purposes."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "n5AzsRvdooZv",
+ "outputId": "a6cad83b-4613-43cc-8c09-86ac79242656"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "--2023-06-15 06:00:15-- https://download.openmmlab.com/mmaction/v1.0/projects/stad_tutorial/multisports-tiny.tar\n",
+ "Resolving download.openmmlab.com (download.openmmlab.com)... 163.181.82.215, 163.181.82.216, 163.181.82.218, ...\n",
+ "Connecting to download.openmmlab.com (download.openmmlab.com)|163.181.82.215|:443... connected.\n",
+ "HTTP request sent, awaiting response... 200 OK\n",
+ "Length: 82780160 (79M) [application/x-tar]\n",
+ "Saving to: ‘data/multisports-tiny.tar’\n",
+ "\n",
+ "multisports-tiny.ta 100%[===================>] 78.95M 13.3MB/s in 44s \n",
+ "\n",
+ "2023-06-15 06:01:00 (1.78 MB/s) - ‘data/multisports-tiny.tar’ saved [82780160/82780160]\n",
+ "\n",
+ "multisports-tiny/multisports/\n",
+ "multisports-tiny/multisports/test/\n",
+ "multisports-tiny/multisports/test/aerobic_gymnastics/\n",
+ "multisports-tiny/multisports/test/aerobic_gymnastics/v_7G_IpU0FxLU_c001.mp4\n",
+ "multisports-tiny/multisports/annotations/\n",
+ "multisports-tiny/multisports/annotations/multisports_GT.pkl\n",
+ "multisports-tiny/multisports/trainval/\n",
+ "multisports-tiny/multisports/trainval/aerobic_gymnastics/\n",
+ "multisports-tiny/multisports/trainval/aerobic_gymnastics/v__wAgwttPYaQ_c001.mp4\n",
+ "multisports-tiny/multisports/trainval/aerobic_gymnastics/v__wAgwttPYaQ_c003.mp4\n",
+ "multisports-tiny/multisports/trainval/aerobic_gymnastics/v__wAgwttPYaQ_c002.mp4\n",
+ "Reading package lists...\n",
+ "Building dependency tree...\n",
+ "Reading state information...\n",
+ "The following NEW packages will be installed:\n",
+ " tree\n",
+ "0 upgraded, 1 newly installed, 0 to remove and 46 not upgraded.\n",
+ "Need to get 43.0 kB of archives.\n",
+ "After this operation, 115 kB of additional disk space will be used.\n",
+ "Get:1 http://archive.ubuntu.com/ubuntu focal/universe amd64 tree amd64 1.8.0-1 [43.0 kB]\n",
+ "Fetched 43.0 kB in 1s (43.0 kB/s)\n",
+ "Selecting previously unselected package tree.\n",
+ "(Reading database ... 122541 files and directories currently installed.)\n",
+ "Preparing to unpack .../tree_1.8.0-1_amd64.deb ...\n",
+ "Unpacking tree (1.8.0-1) ...\n",
+ "Setting up tree (1.8.0-1) ...\n",
+ "Processing triggers for man-db (2.9.1-1) ...\n",
+ "\u001b[01;34mdata\u001b[00m\n",
+ "├── \u001b[01;34mmultisports\u001b[00m\n",
+ "│ ├── \u001b[01;34mannotations\u001b[00m\n",
+ "│ │ └── \u001b[01;32mmultisports_GT.pkl\u001b[00m\n",
+ "│ ├── \u001b[01;34mtest\u001b[00m\n",
+ "│ │ └── \u001b[01;34maerobic_gymnastics\u001b[00m\n",
+ "│ │ └── \u001b[01;32mv_7G_IpU0FxLU_c001.mp4\u001b[00m\n",
+ "│ └── \u001b[01;34mtrainval\u001b[00m\n",
+ "│ └── \u001b[01;34maerobic_gymnastics\u001b[00m\n",
+ "│ ├── \u001b[01;32mv__wAgwttPYaQ_c001.mp4\u001b[00m\n",
+ "│ ├── \u001b[01;32mv__wAgwttPYaQ_c002.mp4\u001b[00m\n",
+ "│ └── \u001b[01;32mv__wAgwttPYaQ_c003.mp4\u001b[00m\n",
+ "└── \u001b[01;31mmultisports-tiny.tar\u001b[00m\n",
+ "\n",
+ "6 directories, 6 files\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Download dataset\n",
+ "!wget -P data -c https://download.openmmlab.com/mmaction/v1.0/projects/stad_tutorial/multisports-tiny.tar\n",
+ "!tar -xvf data/multisports-tiny.tar --strip 1 -C data\n",
+ "!apt-get -q install tree\n",
+ "!tree data"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_u69LHscooZw"
+ },
+ "source": [
+ "## 2. Train detection model\n",
+ "\n",
+ "In the SlowOnly + Det paradigm, we need to train a human detector first, and then predict actions based on the detection results. In this section, we train a detection model based on the annotation format in the previous section and the MMDetection algorithm library.\n",
+ "\n",
+ "### 2.1 Build detection dataset annotation (COCO format)\n",
+ "\n",
+ "Based on the annotation information of the spatio-temporal action detection dataset, we can build a COCO format detection dataset for training the detection model. We provide a script to convert the MultiSports format annotation, if you need to convert from other formats, you can refer to the [custom dataset](https://mmdetection.readthedocs.io/zh_CN/latest/advanced_guides/customize_dataset.html) document provided by MMDetection."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "e8fu9VtRooZw",
+ "outputId": "3e7a7053-a08d-4c32-9d66-a362b3de164d"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\u001b[01;34mdata/multisports/annotations\u001b[00m\n",
+ "├── multisports_det_anno_train.json\n",
+ "├── multisports_det_anno_val.json\n",
+ "└── \u001b[01;32mmultisports_GT.pkl\u001b[00m\n",
+ "\n",
+ "0 directories, 3 files\n"
+ ]
+ }
+ ],
+ "source": [
+ "!python tools/generate_mmdet_anno.py data/multisports/annotations/multisports_GT.pkl data/multisports/annotations/multisports_det_anno.json\n",
+ "!tree data/multisports/annotations"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "HJAb8EwwooZx",
+ "outputId": "1c82387c-c731-484c-a4cc-8c255b3f2e62"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Will generate 3 rgb dir for aerobic_gymnastics.\n",
+ "Generate v__wAgwttPYaQ_c003 rgb dir successfully.\n",
+ "Generate v__wAgwttPYaQ_c002 rgb dir successfully.\n",
+ "Generate v__wAgwttPYaQ_c001 rgb dir successfully.\n"
+ ]
+ }
+ ],
+ "source": [
+ "!python tools/generate_rgb.py"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "9xIOk_XkooZx"
+ },
+ "source": [
+ "### 2.2 Modify config file\n",
+ "\n",
+ "We use faster-rcnn_x101-64x4d_fpn_1x_coco as the base configuration, and make the following modifications to train on the MultiSports dataset. The following parts need to be modified:\n",
+ "- Number of model categories\n",
+ "- Learning rate adjustment strategy\n",
+ "- Optimizer configuration\n",
+ "- Dataset/annotation file path\n",
+ "- Evaluator configuration\n",
+ "- Pre-trained model\n",
+ "\n",
+ "For more detailed tutorials, please refer to the [prepare configuration file](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/train.html#id9) document provided by MMDetection."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "Ad1QLNM8ooZy",
+ "outputId": "55f95e91-8fdf-40fa-dd08-5fa980444b6f"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "# Copyright (c) OpenMMLab. All rights reserved.\n",
+ "_base_ = './faster-rcnn_r50-caffe_fpn_ms-1x_coco.py'\n",
+ "model = dict(roi_head=dict(bbox_head=dict(num_classes=1)))\n",
+ "\n",
+ "# take 2 epochs as an example\n",
+ "train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=2, val_interval=1)\n",
+ "\n",
+ "# learning rate\n",
+ "param_scheduler = [\n",
+ " dict(type='ConstantLR', factor=1.0, by_epoch=False, begin=0, end=500)\n",
+ "]\n",
+ "\n",
+ "# optimizer\n",
+ "optim_wrapper = dict(\n",
+ " type='OptimWrapper',\n",
+ " optimizer=dict(type='SGD', lr=0.0050, momentum=0.9, weight_decay=0.0001))\n",
+ "\n",
+ "dataset_type = 'CocoDataset'\n",
+ "# modify metainfo\n",
+ "metainfo = {\n",
+ " 'classes': ('person', ),\n",
+ " 'palette': [\n",
+ " (220, 20, 60),\n",
+ " ]\n",
+ "}\n",
+ "\n",
+ "# specify metainfo, dataset path\n",
+ "data_root = 'data/multisports/'\n",
+ "\n",
+ "train_dataloader = dict(\n",
+ " dataset=dict(\n",
+ " data_root=data_root,\n",
+ " ann_file='annotations/multisports_det_anno_train.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " metainfo=metainfo))\n",
+ "\n",
+ "val_dataloader = dict(\n",
+ " dataset=dict(\n",
+ " data_root=data_root,\n",
+ " ann_file='annotations/multisports_det_anno_val.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " metainfo=metainfo))\n",
+ "\n",
+ "test_dataloader = dict(\n",
+ " dataset=dict(\n",
+ " data_root=data_root,\n",
+ " ann_file='annotations/ms_infer_anno.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " metainfo=metainfo))\n",
+ "\n",
+ "# specify annotaition file path, modify metric items\n",
+ "val_evaluator = dict(\n",
+ " ann_file='data/multisports/annotations/multisports_det_anno_val.json',\n",
+ " metric_items=['mAP_50', 'AR@100'],\n",
+ " iou_thrs=[0.5],\n",
+ ")\n",
+ "\n",
+ "test_evaluator = dict(\n",
+ " ann_file='data/multisports/annotations/ms_infer_anno.json',\n",
+ " metric_items=['mAP_50', 'AR@100'],\n",
+ " iou_thrs=[0.5],\n",
+ ")\n",
+ "\n",
+ "# specify pretrain checkpoint\n",
+ "load_from = 'https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929-d022e227.pth' # noqa: E501\n"
+ ]
+ }
+ ],
+ "source": [
+ "!cat configs/faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person.py"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "W40JO80nooZ0"
+ },
+ "source": [
+ "### 2.3 Train detection model"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Oc1LWr4AooZ0"
+ },
+ "source": [
+ "By using MIM, you can directly train MMDetection models in the current directory. Here is the simplest example of training on a single GPU. For more training commands, please refer to the MIM [tutorial](https://github.com/open-mmlab/mim#command)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "QpxCbvr2ooZ0",
+ "outputId": "ffe7b420-c359-4e5a-a1b1-3a75e923046d"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Training command is /usr/bin/python3 /usr/local/lib/python3.10/dist-packages/mmdet/.mim/tools/train.py configs/faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person.py --launcher none --work-dir work_dirs/det_model. \n",
+ "06/15 06:02:09 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - \n",
+ "------------------------------------------------------------\n",
+ "System environment:\n",
+ " sys.platform: linux\n",
+ " Python: 3.10.12 (main, Jun 7 2023, 12:45:35) [GCC 9.4.0]\n",
+ " CUDA available: True\n",
+ " numpy_random_seed: 503128501\n",
+ " GPU 0: Tesla T4\n",
+ " CUDA_HOME: /usr/local/cuda\n",
+ " NVCC: Cuda compilation tools, release 11.8, V11.8.89\n",
+ " GCC: x86_64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0\n",
+ " PyTorch: 2.0.1+cu118\n",
+ " PyTorch compiling details: PyTorch built with:\n",
+ " - GCC 9.3\n",
+ " - C++ Version: 201703\n",
+ " - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications\n",
+ " - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n",
+ " - OpenMP 201511 (a.k.a. OpenMP 4.5)\n",
+ " - LAPACK is enabled (usually provided by MKL)\n",
+ " - NNPACK is enabled\n",
+ " - CPU capability usage: AVX2\n",
+ " - CUDA Runtime 11.8\n",
+ " - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n",
+ " - CuDNN 8.7\n",
+ " - Magma 2.6.1\n",
+ " - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n",
+ "\n",
+ " TorchVision: 0.15.2+cu118\n",
+ " OpenCV: 4.7.0\n",
+ " MMEngine: 0.7.4\n",
+ "\n",
+ "Runtime environment:\n",
+ " cudnn_benchmark: False\n",
+ " mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}\n",
+ " dist_cfg: {'backend': 'nccl'}\n",
+ " seed: 503128501\n",
+ " Distributed launcher: none\n",
+ " Distributed training: False\n",
+ " GPU number: 1\n",
+ "------------------------------------------------------------\n",
+ "\n",
+ "06/15 06:02:09 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Config:\n",
+ "model = dict(\n",
+ " type='FasterRCNN',\n",
+ " data_preprocessor=dict(\n",
+ " type='DetDataPreprocessor',\n",
+ " mean=[103.53, 116.28, 123.675],\n",
+ " std=[1.0, 1.0, 1.0],\n",
+ " bgr_to_rgb=False,\n",
+ " pad_size_divisor=32),\n",
+ " backbone=dict(\n",
+ " type='ResNet',\n",
+ " depth=50,\n",
+ " num_stages=4,\n",
+ " out_indices=(0, 1, 2, 3),\n",
+ " frozen_stages=1,\n",
+ " norm_cfg=dict(type='BN', requires_grad=False),\n",
+ " norm_eval=True,\n",
+ " style='caffe',\n",
+ " init_cfg=dict(\n",
+ " type='Pretrained',\n",
+ " checkpoint='open-mmlab://detectron2/resnet50_caffe')),\n",
+ " neck=dict(\n",
+ " type='FPN',\n",
+ " in_channels=[256, 512, 1024, 2048],\n",
+ " out_channels=256,\n",
+ " num_outs=5),\n",
+ " rpn_head=dict(\n",
+ " type='RPNHead',\n",
+ " in_channels=256,\n",
+ " feat_channels=256,\n",
+ " anchor_generator=dict(\n",
+ " type='AnchorGenerator',\n",
+ " scales=[8],\n",
+ " ratios=[0.5, 1.0, 2.0],\n",
+ " strides=[4, 8, 16, 32, 64]),\n",
+ " bbox_coder=dict(\n",
+ " type='DeltaXYWHBBoxCoder',\n",
+ " target_means=[0.0, 0.0, 0.0, 0.0],\n",
+ " target_stds=[1.0, 1.0, 1.0, 1.0]),\n",
+ " loss_cls=dict(\n",
+ " type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),\n",
+ " loss_bbox=dict(type='L1Loss', loss_weight=1.0)),\n",
+ " roi_head=dict(\n",
+ " type='StandardRoIHead',\n",
+ " bbox_roi_extractor=dict(\n",
+ " type='SingleRoIExtractor',\n",
+ " roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0),\n",
+ " out_channels=256,\n",
+ " featmap_strides=[4, 8, 16, 32]),\n",
+ " bbox_head=dict(\n",
+ " type='Shared2FCBBoxHead',\n",
+ " in_channels=256,\n",
+ " fc_out_channels=1024,\n",
+ " roi_feat_size=7,\n",
+ " num_classes=1,\n",
+ " bbox_coder=dict(\n",
+ " type='DeltaXYWHBBoxCoder',\n",
+ " target_means=[0.0, 0.0, 0.0, 0.0],\n",
+ " target_stds=[0.1, 0.1, 0.2, 0.2]),\n",
+ " reg_class_agnostic=False,\n",
+ " loss_cls=dict(\n",
+ " type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),\n",
+ " loss_bbox=dict(type='L1Loss', loss_weight=1.0))),\n",
+ " train_cfg=dict(\n",
+ " rpn=dict(\n",
+ " assigner=dict(\n",
+ " type='MaxIoUAssigner',\n",
+ " pos_iou_thr=0.7,\n",
+ " neg_iou_thr=0.3,\n",
+ " min_pos_iou=0.3,\n",
+ " match_low_quality=True,\n",
+ " ignore_iof_thr=-1),\n",
+ " sampler=dict(\n",
+ " type='RandomSampler',\n",
+ " num=256,\n",
+ " pos_fraction=0.5,\n",
+ " neg_pos_ub=-1,\n",
+ " add_gt_as_proposals=False),\n",
+ " allowed_border=-1,\n",
+ " pos_weight=-1,\n",
+ " debug=False),\n",
+ " rpn_proposal=dict(\n",
+ " nms_pre=2000,\n",
+ " max_per_img=1000,\n",
+ " nms=dict(type='nms', iou_threshold=0.7),\n",
+ " min_bbox_size=0),\n",
+ " rcnn=dict(\n",
+ " assigner=dict(\n",
+ " type='MaxIoUAssigner',\n",
+ " pos_iou_thr=0.5,\n",
+ " neg_iou_thr=0.5,\n",
+ " min_pos_iou=0.5,\n",
+ " match_low_quality=False,\n",
+ " ignore_iof_thr=-1),\n",
+ " sampler=dict(\n",
+ " type='RandomSampler',\n",
+ " num=512,\n",
+ " pos_fraction=0.25,\n",
+ " neg_pos_ub=-1,\n",
+ " add_gt_as_proposals=True),\n",
+ " pos_weight=-1,\n",
+ " debug=False)),\n",
+ " test_cfg=dict(\n",
+ " rpn=dict(\n",
+ " nms_pre=1000,\n",
+ " max_per_img=1000,\n",
+ " nms=dict(type='nms', iou_threshold=0.7),\n",
+ " min_bbox_size=0),\n",
+ " rcnn=dict(\n",
+ " score_thr=0.05,\n",
+ " nms=dict(type='nms', iou_threshold=0.5),\n",
+ " max_per_img=100)))\n",
+ "dataset_type = 'CocoDataset'\n",
+ "data_root = 'data/multisports/'\n",
+ "backend_args = None\n",
+ "train_pipeline = [\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='RandomChoiceResize',\n",
+ " scales=[(1333, 640), (1333, 672), (1333, 704), (1333, 736),\n",
+ " (1333, 768), (1333, 800)],\n",
+ " keep_ratio=True),\n",
+ " dict(type='RandomFlip', prob=0.5),\n",
+ " dict(type='PackDetInputs')\n",
+ "]\n",
+ "test_pipeline = [\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='Resize', scale=(1333, 800), keep_ratio=True),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='PackDetInputs',\n",
+ " meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',\n",
+ " 'scale_factor'))\n",
+ "]\n",
+ "train_dataloader = dict(\n",
+ " batch_size=2,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=True),\n",
+ " batch_sampler=dict(type='AspectRatioBatchSampler'),\n",
+ " dataset=dict(\n",
+ " type='CocoDataset',\n",
+ " data_root='data/multisports/',\n",
+ " ann_file='annotations/multisports_det_anno_train.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " filter_cfg=dict(filter_empty_gt=True, min_size=32),\n",
+ " pipeline=[\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='RandomChoiceResize',\n",
+ " scales=[(1333, 640), (1333, 672), (1333, 704), (1333, 736),\n",
+ " (1333, 768), (1333, 800)],\n",
+ " keep_ratio=True),\n",
+ " dict(type='RandomFlip', prob=0.5),\n",
+ " dict(type='PackDetInputs')\n",
+ " ],\n",
+ " backend_args=None,\n",
+ " metainfo=dict(classes=('person', ), palette=[(220, 20, 60)])))\n",
+ "val_dataloader = dict(\n",
+ " batch_size=1,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " drop_last=False,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=False),\n",
+ " dataset=dict(\n",
+ " type='CocoDataset',\n",
+ " data_root='data/multisports/',\n",
+ " ann_file='annotations/multisports_det_anno_val.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " test_mode=True,\n",
+ " pipeline=[\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='Resize', scale=(1333, 800), keep_ratio=True),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='PackDetInputs',\n",
+ " meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',\n",
+ " 'scale_factor'))\n",
+ " ],\n",
+ " backend_args=None,\n",
+ " metainfo=dict(classes=('person', ), palette=[(220, 20, 60)])))\n",
+ "test_dataloader = dict(\n",
+ " batch_size=1,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " drop_last=False,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=False),\n",
+ " dataset=dict(\n",
+ " type='CocoDataset',\n",
+ " data_root='data/multisports/',\n",
+ " ann_file='annotations/ms_infer_anno.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " test_mode=True,\n",
+ " pipeline=[\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='Resize', scale=(1333, 800), keep_ratio=True),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='PackDetInputs',\n",
+ " meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',\n",
+ " 'scale_factor'))\n",
+ " ],\n",
+ " backend_args=None,\n",
+ " metainfo=dict(classes=('person', ), palette=[(220, 20, 60)])))\n",
+ "val_evaluator = dict(\n",
+ " type='CocoMetric',\n",
+ " ann_file='data/multisports/annotations/multisports_det_anno_val.json',\n",
+ " metric='bbox',\n",
+ " format_only=False,\n",
+ " backend_args=None,\n",
+ " metric_items=['mAP_50', 'AR@100'],\n",
+ " iou_thrs=[0.5])\n",
+ "test_evaluator = dict(\n",
+ " type='CocoMetric',\n",
+ " ann_file='data/multisports/annotations/ms_infer_anno.json',\n",
+ " metric='bbox',\n",
+ " format_only=False,\n",
+ " backend_args=None,\n",
+ " metric_items=['mAP_50', 'AR@100'],\n",
+ " iou_thrs=[0.5])\n",
+ "train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=2, val_interval=1)\n",
+ "val_cfg = dict(type='ValLoop')\n",
+ "test_cfg = dict(type='TestLoop')\n",
+ "param_scheduler = [\n",
+ " dict(type='ConstantLR', factor=1.0, by_epoch=False, begin=0, end=500)\n",
+ "]\n",
+ "optim_wrapper = dict(\n",
+ " type='OptimWrapper',\n",
+ " optimizer=dict(type='SGD', lr=0.005, momentum=0.9, weight_decay=0.0001))\n",
+ "auto_scale_lr = dict(enable=False, base_batch_size=16)\n",
+ "default_scope = 'mmdet'\n",
+ "default_hooks = dict(\n",
+ " timer=dict(type='IterTimerHook'),\n",
+ " logger=dict(type='LoggerHook', interval=50),\n",
+ " param_scheduler=dict(type='ParamSchedulerHook'),\n",
+ " checkpoint=dict(type='CheckpointHook', interval=1),\n",
+ " sampler_seed=dict(type='DistSamplerSeedHook'),\n",
+ " visualization=dict(type='DetVisualizationHook'))\n",
+ "env_cfg = dict(\n",
+ " cudnn_benchmark=False,\n",
+ " mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n",
+ " dist_cfg=dict(backend='nccl'))\n",
+ "vis_backends = [dict(type='LocalVisBackend')]\n",
+ "visualizer = dict(\n",
+ " type='DetLocalVisualizer',\n",
+ " vis_backends=[dict(type='LocalVisBackend')],\n",
+ " name='visualizer')\n",
+ "log_processor = dict(type='LogProcessor', window_size=50, by_epoch=True)\n",
+ "log_level = 'INFO'\n",
+ "load_from = 'https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929-d022e227.pth'\n",
+ "resume = False\n",
+ "metainfo = dict(classes=('person', ), palette=[(220, 20, 60)])\n",
+ "launcher = 'none'\n",
+ "work_dir = 'work_dirs/det_model'\n",
+ "\n",
+ "06/15 06:02:17 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.\n",
+ "06/15 06:02:17 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Hooks will be executed in the following order:\n",
+ "before_run:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "before_train:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_train_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) DistSamplerSeedHook \n",
+ " -------------------- \n",
+ "before_train_iter:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_train_iter:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "after_train_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_val_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "before_val_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_val_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) DetVisualizationHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_val_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "after_train:\n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_test_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "before_test_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_test_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) DetVisualizationHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_test_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_run:\n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "loading annotations into memory...\n",
+ "Done (t=0.01s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "loading annotations into memory...\n",
+ "Done (t=0.00s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "loading annotations into memory...\n",
+ "Done (t=0.00s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "06/15 06:02:18 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - load model from: open-mmlab://detectron2/resnet50_caffe\n",
+ "06/15 06:02:18 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Loads checkpoint by openmmlab backend from path: open-mmlab://detectron2/resnet50_caffe\n",
+ "Downloading: \"https://download.openmmlab.com/pretrain/third_party/resnet50_msra-5891d200.pth\" to /root/.cache/torch/hub/checkpoints/resnet50_msra-5891d200.pth\n",
+ "100% 89.9M/89.9M [00:02<00:00, 34.8MB/s]\n",
+ "06/15 06:02:21 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - The model and loaded state dict do not match exactly\n",
+ "\n",
+ "unexpected key in source state_dict: conv1.bias\n",
+ "\n",
+ "Loads checkpoint by http backend from path: https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929-d022e227.pth\n",
+ "Downloading: \"https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929-d022e227.pth\" to /root/.cache/torch/hub/checkpoints/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929-d022e227.pth\n",
+ "100% 158M/158M [00:04<00:00, 37.4MB/s]\n",
+ "06/15 06:02:26 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Load checkpoint from https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929-d022e227.pth\n",
+ "06/15 06:02:26 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - \"FileClient\" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io\n",
+ "06/15 06:02:26 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - \"HardDiskBackend\" is the alias of \"LocalBackend\" and the former will be deprecated in future.\n",
+ "06/15 06:02:26 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Checkpoints will be saved to /content/mmaction2/projects/stad_tutorial/work_dirs/det_model.\n",
+ "06/15 06:02:57 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][ 50/118] lr: 5.0000e-03 eta: 0:01:56 time: 0.6273 data_time: 0.0111 memory: 3414 loss: 0.5456 loss_rpn_cls: 0.0070 loss_rpn_bbox: 0.0167 loss_cls: 0.1887 acc: 93.2617 loss_bbox: 0.3332\n",
+ "06/15 06:03:22 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][100/118] lr: 5.0000e-03 eta: 0:01:16 time: 0.5041 data_time: 0.0078 memory: 3414 loss: 0.4017 loss_rpn_cls: 0.0027 loss_rpn_bbox: 0.0130 loss_cls: 0.1313 acc: 94.8242 loss_bbox: 0.2547\n",
+ "06/15 06:03:31 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person_20230615_060208\n",
+ "06/15 06:03:31 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 1 epochs\n",
+ "06/15 06:03:39 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][ 50/120] eta: 0:00:08 time: 0.1196 data_time: 0.0059 memory: 3414 \n",
+ "06/15 06:03:45 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][100/120] eta: 0:00:02 time: 0.1234 data_time: 0.0082 memory: 679 \n",
+ "06/15 06:03:47 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Evaluating bbox...\n",
+ "Loading and preparing results...\n",
+ "DONE (t=0.00s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "Running per image evaluation...\n",
+ "Evaluate annotation type *bbox*\n",
+ "DONE (t=0.05s).\n",
+ "Accumulating evaluation results...\n",
+ "DONE (t=0.01s).\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 0.872\n",
+ " Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= small | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area=medium | maxDets=1000 ] = 0.709\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= large | maxDets=1000 ] = 0.886\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 0.964\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=300 ] = 0.964\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=1000 ] = 0.964\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= small | maxDets=1000 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area=medium | maxDets=1000 ] = 1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= large | maxDets=1000 ] = 0.963\n",
+ "06/15 06:03:47 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - bbox_mAP_copypaste: 0.872 -1.000 -1.000 -1.000 0.709 0.886\n",
+ "06/15 06:03:47 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][120/120] coco/bbox_mAP_50: -1.0000 coco/bbox_AR@100: 0.9640 data_time: 0.0067 time: 0.1212\n",
+ "06/15 06:04:14 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][ 50/118] lr: 5.0000e-03 eta: 0:00:37 time: 0.5316 data_time: 0.0094 memory: 3414 loss: 0.3385 loss_rpn_cls: 0.0012 loss_rpn_bbox: 0.0111 loss_cls: 0.1119 acc: 95.4102 loss_bbox: 0.2143\n",
+ "06/15 06:04:40 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][100/118] lr: 5.0000e-03 eta: 0:00:09 time: 0.5152 data_time: 0.0078 memory: 3414 loss: 0.3152 loss_rpn_cls: 0.0017 loss_rpn_bbox: 0.0109 loss_cls: 0.1050 acc: 94.7266 loss_bbox: 0.1977\n",
+ "06/15 06:04:49 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person_20230615_060208\n",
+ "06/15 06:04:49 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 2 epochs\n",
+ "06/15 06:04:57 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][ 50/120] eta: 0:00:08 time: 0.1237 data_time: 0.0080 memory: 3414 \n",
+ "06/15 06:05:03 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][100/120] eta: 0:00:02 time: 0.1202 data_time: 0.0062 memory: 679 \n",
+ "06/15 06:05:06 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Evaluating bbox...\n",
+ "Loading and preparing results...\n",
+ "DONE (t=0.00s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "Running per image evaluation...\n",
+ "Evaluate annotation type *bbox*\n",
+ "DONE (t=0.04s).\n",
+ "Accumulating evaluation results...\n",
+ "DONE (t=0.01s).\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 0.907\n",
+ " Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= small | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area=medium | maxDets=1000 ] = 0.762\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= large | maxDets=1000 ] = 0.910\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 0.960\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=300 ] = 0.960\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=1000 ] = 0.960\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= small | maxDets=1000 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area=medium | maxDets=1000 ] = 1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= large | maxDets=1000 ] = 0.960\n",
+ "06/15 06:05:06 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - bbox_mAP_copypaste: 0.907 -1.000 -1.000 -1.000 0.762 0.910\n",
+ "06/15 06:05:06 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][120/120] coco/bbox_mAP_50: -1.0000 coco/bbox_AR@100: 0.9600 data_time: 0.0066 time: 0.1214\n",
+ "\u001b[32mTraining finished successfully. \u001b[0m\n"
+ ]
+ }
+ ],
+ "source": [
+ "!mim train mmdet configs/faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person.py \\\n",
+ " --work-dir work_dirs/det_model"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IxlO927KooZ1"
+ },
+ "source": [
+ "### 2.4 Generating Proposal BBoxes\n",
+ "\n",
+ "During the training of the spatiotemporal action detection model, we need to rely on proposals generated by the detection model, rather than annotated detection boxes. Therefore, we need to use a trained detection model to perform inference on the entire dataset and convert the resulting proposals into the required format for subsequent training.\n",
+ "\n",
+ "#### 2.4.1 Converting the Dataset to Coco Format\n",
+ "\n",
+ "We provide a script to convert the MultiSports dataset into an annotation format without ground truth, which is used for inference."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "e6C7D2DSooZ1",
+ "outputId": "878015d1-0fc7-4eb6-af77-4f61aefcf2b2"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[>>] 2350/2350, 2053.0 task/s, elapsed: 1s, ETA: 0s\n",
+ "save json file: data/multisports/rawframes/../annotations/ms_infer_anno.json\n"
+ ]
+ }
+ ],
+ "source": [
+ "!echo 'person' > data/multisports/annotations/label_map.txt\n",
+ "!python tools/images2coco.py \\\n",
+ " data/multisports/rawframes \\\n",
+ " data/multisports/annotations/label_map.txt \\\n",
+ " ms_infer_anno.json"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "fGL3t4MEooZ1"
+ },
+ "source": [
+ "#### 2.4.2 Inference for Generating Proposal Files\n"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "gerYk6q6ooZ1"
+ },
+ "source": [
+ "The inference of MMDetection models is also based on MIM. For more testing commands, please refer to the MIM [tutorial](GitHub - open-mmlab/mim: MIM Installs OpenMMLab Packages).\n",
+ "\n",
+ "After the inference is completed, the results will be saved in 'data/multisports/ms_proposals.pkl'."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "lutiaqzpooZ1",
+ "outputId": "b05db6e8-04de-4e1e-8d99-32f4c952d633"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Testing command is /usr/bin/python3 /usr/local/lib/python3.10/dist-packages/mmdet/.mim/tools/test.py configs/faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person.py work_dirs/det_model/epoch_2.pth --launcher none --out data/multisports/annotations/ms_det_proposals.pkl. \n",
+ "06/15 06:05:16 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - \n",
+ "------------------------------------------------------------\n",
+ "System environment:\n",
+ " sys.platform: linux\n",
+ " Python: 3.10.12 (main, Jun 7 2023, 12:45:35) [GCC 9.4.0]\n",
+ " CUDA available: True\n",
+ " numpy_random_seed: 1289054678\n",
+ " GPU 0: Tesla T4\n",
+ " CUDA_HOME: /usr/local/cuda\n",
+ " NVCC: Cuda compilation tools, release 11.8, V11.8.89\n",
+ " GCC: x86_64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0\n",
+ " PyTorch: 2.0.1+cu118\n",
+ " PyTorch compiling details: PyTorch built with:\n",
+ " - GCC 9.3\n",
+ " - C++ Version: 201703\n",
+ " - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications\n",
+ " - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n",
+ " - OpenMP 201511 (a.k.a. OpenMP 4.5)\n",
+ " - LAPACK is enabled (usually provided by MKL)\n",
+ " - NNPACK is enabled\n",
+ " - CPU capability usage: AVX2\n",
+ " - CUDA Runtime 11.8\n",
+ " - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n",
+ " - CuDNN 8.7\n",
+ " - Magma 2.6.1\n",
+ " - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n",
+ "\n",
+ " TorchVision: 0.15.2+cu118\n",
+ " OpenCV: 4.7.0\n",
+ " MMEngine: 0.7.4\n",
+ "\n",
+ "Runtime environment:\n",
+ " cudnn_benchmark: False\n",
+ " mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}\n",
+ " dist_cfg: {'backend': 'nccl'}\n",
+ " seed: 1289054678\n",
+ " Distributed launcher: none\n",
+ " Distributed training: False\n",
+ " GPU number: 1\n",
+ "------------------------------------------------------------\n",
+ "\n",
+ "06/15 06:05:16 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Config:\n",
+ "model = dict(\n",
+ " type='FasterRCNN',\n",
+ " data_preprocessor=dict(\n",
+ " type='DetDataPreprocessor',\n",
+ " mean=[103.53, 116.28, 123.675],\n",
+ " std=[1.0, 1.0, 1.0],\n",
+ " bgr_to_rgb=False,\n",
+ " pad_size_divisor=32),\n",
+ " backbone=dict(\n",
+ " type='ResNet',\n",
+ " depth=50,\n",
+ " num_stages=4,\n",
+ " out_indices=(0, 1, 2, 3),\n",
+ " frozen_stages=1,\n",
+ " norm_cfg=dict(type='BN', requires_grad=False),\n",
+ " norm_eval=True,\n",
+ " style='caffe',\n",
+ " init_cfg=dict(\n",
+ " type='Pretrained',\n",
+ " checkpoint='open-mmlab://detectron2/resnet50_caffe')),\n",
+ " neck=dict(\n",
+ " type='FPN',\n",
+ " in_channels=[256, 512, 1024, 2048],\n",
+ " out_channels=256,\n",
+ " num_outs=5),\n",
+ " rpn_head=dict(\n",
+ " type='RPNHead',\n",
+ " in_channels=256,\n",
+ " feat_channels=256,\n",
+ " anchor_generator=dict(\n",
+ " type='AnchorGenerator',\n",
+ " scales=[8],\n",
+ " ratios=[0.5, 1.0, 2.0],\n",
+ " strides=[4, 8, 16, 32, 64]),\n",
+ " bbox_coder=dict(\n",
+ " type='DeltaXYWHBBoxCoder',\n",
+ " target_means=[0.0, 0.0, 0.0, 0.0],\n",
+ " target_stds=[1.0, 1.0, 1.0, 1.0]),\n",
+ " loss_cls=dict(\n",
+ " type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),\n",
+ " loss_bbox=dict(type='L1Loss', loss_weight=1.0)),\n",
+ " roi_head=dict(\n",
+ " type='StandardRoIHead',\n",
+ " bbox_roi_extractor=dict(\n",
+ " type='SingleRoIExtractor',\n",
+ " roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0),\n",
+ " out_channels=256,\n",
+ " featmap_strides=[4, 8, 16, 32]),\n",
+ " bbox_head=dict(\n",
+ " type='Shared2FCBBoxHead',\n",
+ " in_channels=256,\n",
+ " fc_out_channels=1024,\n",
+ " roi_feat_size=7,\n",
+ " num_classes=1,\n",
+ " bbox_coder=dict(\n",
+ " type='DeltaXYWHBBoxCoder',\n",
+ " target_means=[0.0, 0.0, 0.0, 0.0],\n",
+ " target_stds=[0.1, 0.1, 0.2, 0.2]),\n",
+ " reg_class_agnostic=False,\n",
+ " loss_cls=dict(\n",
+ " type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),\n",
+ " loss_bbox=dict(type='L1Loss', loss_weight=1.0))),\n",
+ " train_cfg=dict(\n",
+ " rpn=dict(\n",
+ " assigner=dict(\n",
+ " type='MaxIoUAssigner',\n",
+ " pos_iou_thr=0.7,\n",
+ " neg_iou_thr=0.3,\n",
+ " min_pos_iou=0.3,\n",
+ " match_low_quality=True,\n",
+ " ignore_iof_thr=-1),\n",
+ " sampler=dict(\n",
+ " type='RandomSampler',\n",
+ " num=256,\n",
+ " pos_fraction=0.5,\n",
+ " neg_pos_ub=-1,\n",
+ " add_gt_as_proposals=False),\n",
+ " allowed_border=-1,\n",
+ " pos_weight=-1,\n",
+ " debug=False),\n",
+ " rpn_proposal=dict(\n",
+ " nms_pre=2000,\n",
+ " max_per_img=1000,\n",
+ " nms=dict(type='nms', iou_threshold=0.7),\n",
+ " min_bbox_size=0),\n",
+ " rcnn=dict(\n",
+ " assigner=dict(\n",
+ " type='MaxIoUAssigner',\n",
+ " pos_iou_thr=0.5,\n",
+ " neg_iou_thr=0.5,\n",
+ " min_pos_iou=0.5,\n",
+ " match_low_quality=False,\n",
+ " ignore_iof_thr=-1),\n",
+ " sampler=dict(\n",
+ " type='RandomSampler',\n",
+ " num=512,\n",
+ " pos_fraction=0.25,\n",
+ " neg_pos_ub=-1,\n",
+ " add_gt_as_proposals=True),\n",
+ " pos_weight=-1,\n",
+ " debug=False)),\n",
+ " test_cfg=dict(\n",
+ " rpn=dict(\n",
+ " nms_pre=1000,\n",
+ " max_per_img=1000,\n",
+ " nms=dict(type='nms', iou_threshold=0.7),\n",
+ " min_bbox_size=0),\n",
+ " rcnn=dict(\n",
+ " score_thr=0.05,\n",
+ " nms=dict(type='nms', iou_threshold=0.5),\n",
+ " max_per_img=100)))\n",
+ "dataset_type = 'CocoDataset'\n",
+ "data_root = 'data/multisports/'\n",
+ "backend_args = None\n",
+ "train_pipeline = [\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='RandomChoiceResize',\n",
+ " scales=[(1333, 640), (1333, 672), (1333, 704), (1333, 736),\n",
+ " (1333, 768), (1333, 800)],\n",
+ " keep_ratio=True),\n",
+ " dict(type='RandomFlip', prob=0.5),\n",
+ " dict(type='PackDetInputs')\n",
+ "]\n",
+ "test_pipeline = [\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='Resize', scale=(1333, 800), keep_ratio=True),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='PackDetInputs',\n",
+ " meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',\n",
+ " 'scale_factor'))\n",
+ "]\n",
+ "train_dataloader = dict(\n",
+ " batch_size=2,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=True),\n",
+ " batch_sampler=dict(type='AspectRatioBatchSampler'),\n",
+ " dataset=dict(\n",
+ " type='CocoDataset',\n",
+ " data_root='data/multisports/',\n",
+ " ann_file='annotations/multisports_det_anno_train.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " filter_cfg=dict(filter_empty_gt=True, min_size=32),\n",
+ " pipeline=[\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='RandomChoiceResize',\n",
+ " scales=[(1333, 640), (1333, 672), (1333, 704), (1333, 736),\n",
+ " (1333, 768), (1333, 800)],\n",
+ " keep_ratio=True),\n",
+ " dict(type='RandomFlip', prob=0.5),\n",
+ " dict(type='PackDetInputs')\n",
+ " ],\n",
+ " backend_args=None,\n",
+ " metainfo=dict(classes=('person', ), palette=[(220, 20, 60)])))\n",
+ "val_dataloader = dict(\n",
+ " batch_size=1,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " drop_last=False,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=False),\n",
+ " dataset=dict(\n",
+ " type='CocoDataset',\n",
+ " data_root='data/multisports/',\n",
+ " ann_file='annotations/multisports_det_anno_val.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " test_mode=True,\n",
+ " pipeline=[\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='Resize', scale=(1333, 800), keep_ratio=True),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='PackDetInputs',\n",
+ " meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',\n",
+ " 'scale_factor'))\n",
+ " ],\n",
+ " backend_args=None,\n",
+ " metainfo=dict(classes=('person', ), palette=[(220, 20, 60)])))\n",
+ "test_dataloader = dict(\n",
+ " batch_size=1,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " drop_last=False,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=False),\n",
+ " dataset=dict(\n",
+ " type='CocoDataset',\n",
+ " data_root='data/multisports/',\n",
+ " ann_file='annotations/ms_infer_anno.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " test_mode=True,\n",
+ " pipeline=[\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='Resize', scale=(1333, 800), keep_ratio=True),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='PackDetInputs',\n",
+ " meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',\n",
+ " 'scale_factor'))\n",
+ " ],\n",
+ " backend_args=None,\n",
+ " metainfo=dict(classes=('person', ), palette=[(220, 20, 60)])))\n",
+ "val_evaluator = dict(\n",
+ " type='CocoMetric',\n",
+ " ann_file='data/multisports/annotations/multisports_det_anno_val.json',\n",
+ " metric='bbox',\n",
+ " format_only=False,\n",
+ " backend_args=None,\n",
+ " metric_items=['mAP_50', 'AR@100'],\n",
+ " iou_thrs=[0.5])\n",
+ "test_evaluator = dict(\n",
+ " type='CocoMetric',\n",
+ " ann_file='data/multisports/annotations/ms_infer_anno.json',\n",
+ " metric='bbox',\n",
+ " format_only=False,\n",
+ " backend_args=None,\n",
+ " metric_items=['mAP_50', 'AR@100'],\n",
+ " iou_thrs=[0.5])\n",
+ "train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=2, val_interval=1)\n",
+ "val_cfg = dict(type='ValLoop')\n",
+ "test_cfg = dict(type='TestLoop')\n",
+ "param_scheduler = [\n",
+ " dict(type='ConstantLR', factor=1.0, by_epoch=False, begin=0, end=500)\n",
+ "]\n",
+ "optim_wrapper = dict(\n",
+ " type='OptimWrapper',\n",
+ " optimizer=dict(type='SGD', lr=0.005, momentum=0.9, weight_decay=0.0001))\n",
+ "auto_scale_lr = dict(enable=False, base_batch_size=16)\n",
+ "default_scope = 'mmdet'\n",
+ "default_hooks = dict(\n",
+ " timer=dict(type='IterTimerHook'),\n",
+ " logger=dict(type='LoggerHook', interval=50),\n",
+ " param_scheduler=dict(type='ParamSchedulerHook'),\n",
+ " checkpoint=dict(type='CheckpointHook', interval=1),\n",
+ " sampler_seed=dict(type='DistSamplerSeedHook'),\n",
+ " visualization=dict(type='DetVisualizationHook'))\n",
+ "env_cfg = dict(\n",
+ " cudnn_benchmark=False,\n",
+ " mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n",
+ " dist_cfg=dict(backend='nccl'))\n",
+ "vis_backends = [dict(type='LocalVisBackend')]\n",
+ "visualizer = dict(\n",
+ " type='DetLocalVisualizer',\n",
+ " vis_backends=[dict(type='LocalVisBackend')],\n",
+ " name='visualizer')\n",
+ "log_processor = dict(type='LogProcessor', window_size=50, by_epoch=True)\n",
+ "log_level = 'INFO'\n",
+ "load_from = 'work_dirs/det_model/epoch_2.pth'\n",
+ "resume = False\n",
+ "metainfo = dict(classes=('person', ), palette=[(220, 20, 60)])\n",
+ "launcher = 'none'\n",
+ "work_dir = './work_dirs/faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person'\n",
+ "\n",
+ "06/15 06:05:19 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.\n",
+ "06/15 06:05:19 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Hooks will be executed in the following order:\n",
+ "before_run:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "before_train:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_train_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) DistSamplerSeedHook \n",
+ " -------------------- \n",
+ "before_train_iter:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_train_iter:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "after_train_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_val_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "before_val_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_val_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) DetVisualizationHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_val_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "after_train:\n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_test_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "before_test_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_test_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) DetVisualizationHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_test_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_run:\n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "loading annotations into memory...\n",
+ "Done (t=0.00s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "loading annotations into memory...\n",
+ "Done (t=0.00s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "06/15 06:05:20 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - The prefix is not set in metric class DumpDetResults.\n",
+ "Loads checkpoint by local backend from path: work_dirs/det_model/epoch_2.pth\n",
+ "06/15 06:05:20 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Load checkpoint from work_dirs/det_model/epoch_2.pth\n",
+ "06/15 06:05:28 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 50/2350] eta: 0:05:50 time: 0.1523 data_time: 0.0084 memory: 512 \n",
+ "06/15 06:05:34 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 100/2350] eta: 0:05:05 time: 0.1191 data_time: 0.0042 memory: 512 \n",
+ "06/15 06:05:40 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 150/2350] eta: 0:04:45 time: 0.1178 data_time: 0.0023 memory: 512 \n",
+ "06/15 06:05:46 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 200/2350] eta: 0:04:36 time: 0.1255 data_time: 0.0074 memory: 512 \n",
+ "06/15 06:05:52 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 250/2350] eta: 0:04:26 time: 0.1205 data_time: 0.0031 memory: 512 \n",
+ "06/15 06:05:58 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 300/2350] eta: 0:04:19 time: 0.1238 data_time: 0.0063 memory: 512 \n",
+ "06/15 06:06:04 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 350/2350] eta: 0:04:11 time: 0.1206 data_time: 0.0046 memory: 512 \n",
+ "06/15 06:06:10 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 400/2350] eta: 0:04:03 time: 0.1178 data_time: 0.0030 memory: 512 \n",
+ "06/15 06:06:16 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 450/2350] eta: 0:03:56 time: 0.1212 data_time: 0.0058 memory: 512 \n",
+ "06/15 06:06:22 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 500/2350] eta: 0:03:48 time: 0.1165 data_time: 0.0031 memory: 512 \n",
+ "06/15 06:06:28 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 550/2350] eta: 0:03:41 time: 0.1202 data_time: 0.0061 memory: 512 \n",
+ "06/15 06:06:34 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 600/2350] eta: 0:03:34 time: 0.1179 data_time: 0.0044 memory: 512 \n",
+ "06/15 06:06:40 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 650/2350] eta: 0:03:27 time: 0.1156 data_time: 0.0024 memory: 512 \n",
+ "06/15 06:06:46 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 700/2350] eta: 0:03:21 time: 0.1212 data_time: 0.0058 memory: 512 \n",
+ "06/15 06:06:52 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 750/2350] eta: 0:03:14 time: 0.1161 data_time: 0.0025 memory: 512 \n",
+ "06/15 06:06:58 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 800/2350] eta: 0:03:08 time: 0.1200 data_time: 0.0058 memory: 512 \n",
+ "06/15 06:07:04 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 850/2350] eta: 0:03:02 time: 0.1203 data_time: 0.0053 memory: 512 \n",
+ "06/15 06:07:09 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 900/2350] eta: 0:02:55 time: 0.1177 data_time: 0.0030 memory: 512 \n",
+ "06/15 06:07:16 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 950/2350] eta: 0:02:50 time: 0.1233 data_time: 0.0076 memory: 512 \n",
+ "06/15 06:07:21 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1000/2350] eta: 0:02:43 time: 0.1172 data_time: 0.0025 memory: 512 \n",
+ "06/15 06:07:27 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1050/2350] eta: 0:02:37 time: 0.1202 data_time: 0.0053 memory: 512 \n",
+ "06/15 06:07:34 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1100/2350] eta: 0:02:31 time: 0.1208 data_time: 0.0059 memory: 512 \n",
+ "06/15 06:07:39 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1150/2350] eta: 0:02:25 time: 0.1167 data_time: 0.0030 memory: 512 \n",
+ "06/15 06:07:45 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1200/2350] eta: 0:02:19 time: 0.1212 data_time: 0.0053 memory: 512 \n",
+ "06/15 06:07:51 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1250/2350] eta: 0:02:12 time: 0.1163 data_time: 0.0027 memory: 512 \n",
+ "06/15 06:07:57 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1300/2350] eta: 0:02:06 time: 0.1188 data_time: 0.0046 memory: 512 \n",
+ "06/15 06:08:03 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1350/2350] eta: 0:02:00 time: 0.1201 data_time: 0.0056 memory: 512 \n",
+ "06/15 06:08:09 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1400/2350] eta: 0:01:54 time: 0.1161 data_time: 0.0024 memory: 512 \n",
+ "06/15 06:08:15 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1450/2350] eta: 0:01:48 time: 0.1234 data_time: 0.0079 memory: 512 \n",
+ "06/15 06:08:21 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1500/2350] eta: 0:01:42 time: 0.1165 data_time: 0.0024 memory: 512 \n",
+ "06/15 06:08:27 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1550/2350] eta: 0:01:36 time: 0.1191 data_time: 0.0043 memory: 512 \n",
+ "06/15 06:08:33 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1600/2350] eta: 0:01:30 time: 0.1219 data_time: 0.0071 memory: 512 \n",
+ "06/15 06:08:39 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1650/2350] eta: 0:01:24 time: 0.1166 data_time: 0.0026 memory: 512 \n",
+ "06/15 06:08:45 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1700/2350] eta: 0:01:18 time: 0.1224 data_time: 0.0067 memory: 512 \n",
+ "06/15 06:08:51 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1750/2350] eta: 0:01:12 time: 0.1175 data_time: 0.0032 memory: 512 \n",
+ "06/15 06:08:57 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1800/2350] eta: 0:01:06 time: 0.1186 data_time: 0.0041 memory: 512 \n",
+ "06/15 06:09:03 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1850/2350] eta: 0:01:00 time: 0.1227 data_time: 0.0067 memory: 512 \n",
+ "06/15 06:09:09 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1900/2350] eta: 0:00:54 time: 0.1220 data_time: 0.0070 memory: 512 \n",
+ "06/15 06:09:15 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1950/2350] eta: 0:00:48 time: 0.1229 data_time: 0.0081 memory: 512 \n",
+ "06/15 06:09:21 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2000/2350] eta: 0:00:42 time: 0.1173 data_time: 0.0029 memory: 512 \n",
+ "06/15 06:09:27 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2050/2350] eta: 0:00:36 time: 0.1184 data_time: 0.0037 memory: 512 \n",
+ "06/15 06:09:33 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2100/2350] eta: 0:00:30 time: 0.1216 data_time: 0.0066 memory: 512 \n",
+ "06/15 06:09:39 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2150/2350] eta: 0:00:24 time: 0.1166 data_time: 0.0026 memory: 512 \n",
+ "06/15 06:09:45 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2200/2350] eta: 0:00:18 time: 0.1213 data_time: 0.0052 memory: 512 \n",
+ "06/15 06:09:51 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2250/2350] eta: 0:00:12 time: 0.1180 data_time: 0.0033 memory: 512 \n",
+ "06/15 06:09:57 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2300/2350] eta: 0:00:06 time: 0.1173 data_time: 0.0032 memory: 512 \n",
+ "06/15 06:10:03 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2350/2350] eta: 0:00:00 time: 0.1203 data_time: 0.0048 memory: 512 \n",
+ "06/15 06:10:03 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Evaluating bbox...\n",
+ "Loading and preparing results...\n",
+ "DONE (t=0.01s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "Running per image evaluation...\n",
+ "Evaluate annotation type *bbox*\n",
+ "DONE (t=0.36s).\n",
+ "Accumulating evaluation results...\n",
+ "DONE (t=0.28s).\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= small | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area=medium | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= large | maxDets=1000 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=300 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=1000 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= small | maxDets=1000 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area=medium | maxDets=1000 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= large | maxDets=1000 ] = -1.000\n",
+ "06/15 06:10:04 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - bbox_mAP_copypaste: -1.000 -1.000 -1.000 -1.000 -1.000 -1.000\n",
+ "06/15 06:10:04 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Results has been saved to data/multisports/annotations/ms_det_proposals.pkl.\n",
+ "06/15 06:10:04 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2350/2350] coco/bbox_mAP_50: -1.0000 coco/bbox_AR@100: -1.0000 data_time: 0.0047 time: 0.1202\n",
+ "\u001b[32mTesting finished successfully.\u001b[0m\n"
+ ]
+ }
+ ],
+ "source": [
+ "!mim test mmdet configs/faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person.py \\\n",
+ " --checkpoint work_dirs/det_model/epoch_2.pth \\\n",
+ " --out data/multisports/annotations/ms_det_proposals.pkl"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jzWhc7ClooZ1"
+ },
+ "source": [
+ "## 3. Training the Spatio-temporal Action Detection Model\n",
+ "The provided annotation files and the proposal files generated by MMDetection need to be converted to the required format for training the spatiotemporal action detection model. We have provided relevant script to generate the specified format."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "W3slJsWHooZ2",
+ "outputId": "42a4b7be-91f8-4443-b693-ab40b743a14f"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "loading test result...\n",
+ "[>>] 2350/2350, 3799.7 task/s, elapsed: 1s, ETA: 0s\n",
+ "\u001b[01;34mdata/multisports/annotations\u001b[00m\n",
+ "├── label_map.txt\n",
+ "├── ms_det_proposals.pkl\n",
+ "├── ms_infer_anno.json\n",
+ "├── multisports_det_anno_train.json\n",
+ "├── multisports_det_anno_val.json\n",
+ "├── \u001b[01;32mmultisports_GT.pkl\u001b[00m\n",
+ "├── multisports_proposals_train.pkl\n",
+ "├── multisports_proposals_val.pkl\n",
+ "├── multisports_train.csv\n",
+ "└── multisports_val.csv\n",
+ "\n",
+ "0 directories, 10 files\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Convert annotation files\n",
+ "!python ../../tools/data/multisports/parse_anno.py\n",
+ "\n",
+ "# Convert proposal files\n",
+ "!python tools/convert_proposals.py\n",
+ "\n",
+ "!tree data/multisports/annotations"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yRSSHmw0ooZ2"
+ },
+ "source": [
+ "### 3.2 Training the Spatio-temporal Action Detection Model\n",
+ "\n",
+ "MMAction2 already supports training on the MultiSports dataset. You just need to modify the path to the proposal file. For detailed configurations, please refer to the [config](configs/slowonly_k400_multisports.py) file. Since the training data is limited, the configuration uses a pre-trained model trained on the complete MultiSports dataset. When training with a custom dataset, you don't need to specify the `load_from` configuration."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "vwaay7NvooZ2",
+ "outputId": "add60ddd-2a40-4356-b120-1e7940043778"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Training command is /usr/bin/python3 /content/mmaction2/mmaction/.mim/tools/train.py configs/slowonly_k400_multisports.py --launcher none --work-dir work_dirs/stad_model/. \n",
+ "06/15 06:10:18 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - \n",
+ "------------------------------------------------------------\n",
+ "System environment:\n",
+ " sys.platform: linux\n",
+ " Python: 3.10.12 (main, Jun 7 2023, 12:45:35) [GCC 9.4.0]\n",
+ " CUDA available: True\n",
+ " numpy_random_seed: 1735696538\n",
+ " GPU 0: Tesla T4\n",
+ " CUDA_HOME: /usr/local/cuda\n",
+ " NVCC: Cuda compilation tools, release 11.8, V11.8.89\n",
+ " GCC: x86_64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0\n",
+ " PyTorch: 2.0.1+cu118\n",
+ " PyTorch compiling details: PyTorch built with:\n",
+ " - GCC 9.3\n",
+ " - C++ Version: 201703\n",
+ " - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications\n",
+ " - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n",
+ " - OpenMP 201511 (a.k.a. OpenMP 4.5)\n",
+ " - LAPACK is enabled (usually provided by MKL)\n",
+ " - NNPACK is enabled\n",
+ " - CPU capability usage: AVX2\n",
+ " - CUDA Runtime 11.8\n",
+ " - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n",
+ " - CuDNN 8.7\n",
+ " - Magma 2.6.1\n",
+ " - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n",
+ "\n",
+ " TorchVision: 0.15.2+cu118\n",
+ " OpenCV: 4.7.0\n",
+ " MMEngine: 0.7.4\n",
+ "\n",
+ "Runtime environment:\n",
+ " cudnn_benchmark: False\n",
+ " mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}\n",
+ " dist_cfg: {'backend': 'nccl'}\n",
+ " seed: 1735696538\n",
+ " diff_rank_seed: False\n",
+ " deterministic: False\n",
+ " Distributed launcher: none\n",
+ " Distributed training: False\n",
+ " GPU number: 1\n",
+ "------------------------------------------------------------\n",
+ "\n",
+ "06/15 06:10:19 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Config:\n",
+ "default_scope = 'mmaction'\n",
+ "default_hooks = dict(\n",
+ " runtime_info=dict(type='RuntimeInfoHook', _scope_='mmaction'),\n",
+ " timer=dict(type='IterTimerHook', _scope_='mmaction'),\n",
+ " logger=dict(\n",
+ " type='LoggerHook', interval=20, ignore_last=False, _scope_='mmaction'),\n",
+ " param_scheduler=dict(type='ParamSchedulerHook', _scope_='mmaction'),\n",
+ " checkpoint=dict(\n",
+ " type='CheckpointHook',\n",
+ " interval=1,\n",
+ " save_best='auto',\n",
+ " _scope_='mmaction'),\n",
+ " sampler_seed=dict(type='DistSamplerSeedHook', _scope_='mmaction'),\n",
+ " sync_buffers=dict(type='SyncBuffersHook', _scope_='mmaction'))\n",
+ "env_cfg = dict(\n",
+ " cudnn_benchmark=False,\n",
+ " mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n",
+ " dist_cfg=dict(backend='nccl'))\n",
+ "log_processor = dict(\n",
+ " type='LogProcessor', window_size=20, by_epoch=True, _scope_='mmaction')\n",
+ "vis_backends = [dict(type='LocalVisBackend', _scope_='mmaction')]\n",
+ "visualizer = dict(\n",
+ " type='ActionVisualizer',\n",
+ " vis_backends=[dict(type='LocalVisBackend')],\n",
+ " _scope_='mmaction')\n",
+ "log_level = 'INFO'\n",
+ "load_from = 'https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-a1ca5e76.pth'\n",
+ "resume = False\n",
+ "url = 'https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-e7b65fad.pth'\n",
+ "num_classes = 66\n",
+ "model = dict(\n",
+ " type='FastRCNN',\n",
+ " _scope_='mmdet',\n",
+ " init_cfg=dict(\n",
+ " type='Pretrained',\n",
+ " checkpoint=\n",
+ " 'https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-e7b65fad.pth'\n",
+ " ),\n",
+ " backbone=dict(\n",
+ " type='mmaction.ResNet3dSlowOnly',\n",
+ " depth=50,\n",
+ " pretrained=None,\n",
+ " pretrained2d=False,\n",
+ " lateral=False,\n",
+ " num_stages=4,\n",
+ " conv1_kernel=(1, 7, 7),\n",
+ " conv1_stride_t=1,\n",
+ " pool1_stride_t=1,\n",
+ " spatial_strides=(1, 2, 2, 1)),\n",
+ " roi_head=dict(\n",
+ " type='AVARoIHead',\n",
+ " bbox_roi_extractor=dict(\n",
+ " type='SingleRoIExtractor3D',\n",
+ " roi_layer_type='RoIAlign',\n",
+ " output_size=8,\n",
+ " with_temporal_pool=True),\n",
+ " bbox_head=dict(\n",
+ " type='BBoxHeadAVA',\n",
+ " in_channels=2048,\n",
+ " num_classes=66,\n",
+ " multilabel=False,\n",
+ " dropout_ratio=0.5)),\n",
+ " data_preprocessor=dict(\n",
+ " type='mmaction.ActionDataPreprocessor',\n",
+ " mean=[123.675, 116.28, 103.53],\n",
+ " std=[58.395, 57.12, 57.375],\n",
+ " format_shape='NCTHW'),\n",
+ " train_cfg=dict(\n",
+ " rcnn=dict(\n",
+ " assigner=dict(\n",
+ " type='MaxIoUAssignerAVA',\n",
+ " pos_iou_thr=0.9,\n",
+ " neg_iou_thr=0.9,\n",
+ " min_pos_iou=0.9),\n",
+ " sampler=dict(\n",
+ " type='RandomSampler',\n",
+ " num=32,\n",
+ " pos_fraction=1,\n",
+ " neg_pos_ub=-1,\n",
+ " add_gt_as_proposals=True),\n",
+ " pos_weight=1.0)),\n",
+ " test_cfg=dict(rcnn=None))\n",
+ "dataset_type = 'AVADataset'\n",
+ "data_root = 'data/multisports/trainval'\n",
+ "anno_root = 'data/multisports/annotations'\n",
+ "ann_file_train = 'data/multisports/annotations/multisports_train.csv'\n",
+ "ann_file_val = 'data/multisports/annotations/multisports_val.csv'\n",
+ "gt_file = 'data/multisports/annotations/multisports_GT.pkl'\n",
+ "proposal_file_train = 'data/multisports/annotations/multisports_proposals_train.pkl'\n",
+ "proposal_file_val = 'data/multisports/annotations/multisports_proposals_val.pkl'\n",
+ "file_client_args = dict(io_backend='disk')\n",
+ "train_pipeline = [\n",
+ " dict(type='DecordInit', io_backend='disk', _scope_='mmaction'),\n",
+ " dict(\n",
+ " type='SampleAVAFrames',\n",
+ " clip_len=4,\n",
+ " frame_interval=16,\n",
+ " _scope_='mmaction'),\n",
+ " dict(type='DecordDecode', _scope_='mmaction'),\n",
+ " dict(type='RandomRescale', scale_range=(256, 320), _scope_='mmaction'),\n",
+ " dict(type='RandomCrop', size=256, _scope_='mmaction'),\n",
+ " dict(type='Flip', flip_ratio=0.5, _scope_='mmaction'),\n",
+ " dict(\n",
+ " type='FormatShape',\n",
+ " input_format='NCTHW',\n",
+ " collapse=True,\n",
+ " _scope_='mmaction'),\n",
+ " dict(type='PackActionInputs', _scope_='mmaction')\n",
+ "]\n",
+ "val_pipeline = [\n",
+ " dict(type='DecordInit', io_backend='disk', _scope_='mmaction'),\n",
+ " dict(\n",
+ " type='SampleAVAFrames',\n",
+ " clip_len=4,\n",
+ " frame_interval=16,\n",
+ " test_mode=True,\n",
+ " _scope_='mmaction'),\n",
+ " dict(type='DecordDecode', _scope_='mmaction'),\n",
+ " dict(type='Resize', scale=(-1, 256), _scope_='mmaction'),\n",
+ " dict(\n",
+ " type='FormatShape',\n",
+ " input_format='NCTHW',\n",
+ " collapse=True,\n",
+ " _scope_='mmaction'),\n",
+ " dict(type='PackActionInputs', _scope_='mmaction')\n",
+ "]\n",
+ "train_dataloader = dict(\n",
+ " batch_size=2,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=True, _scope_='mmaction'),\n",
+ " dataset=dict(\n",
+ " type='AVADataset',\n",
+ " ann_file='data/multisports/annotations/multisports_train.csv',\n",
+ " pipeline=[\n",
+ " dict(type='DecordInit', io_backend='disk'),\n",
+ " dict(type='SampleAVAFrames', clip_len=4, frame_interval=16),\n",
+ " dict(type='DecordDecode'),\n",
+ " dict(type='RandomRescale', scale_range=(256, 320)),\n",
+ " dict(type='RandomCrop', size=256),\n",
+ " dict(type='Flip', flip_ratio=0.5),\n",
+ " dict(type='FormatShape', input_format='NCTHW', collapse=True),\n",
+ " dict(type='PackActionInputs')\n",
+ " ],\n",
+ " num_classes=66,\n",
+ " proposal_file=\n",
+ " 'data/multisports/annotations/multisports_proposals_train.pkl',\n",
+ " data_prefix=dict(img='data/multisports/trainval'),\n",
+ " timestamp_start=1,\n",
+ " start_index=0,\n",
+ " use_frames=False,\n",
+ " fps=1,\n",
+ " _scope_='mmaction'))\n",
+ "val_dataloader = dict(\n",
+ " batch_size=1,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=False, _scope_='mmaction'),\n",
+ " dataset=dict(\n",
+ " type='AVADataset',\n",
+ " ann_file='data/multisports/annotations/multisports_val.csv',\n",
+ " pipeline=[\n",
+ " dict(type='DecordInit', io_backend='disk'),\n",
+ " dict(\n",
+ " type='SampleAVAFrames',\n",
+ " clip_len=4,\n",
+ " frame_interval=16,\n",
+ " test_mode=True),\n",
+ " dict(type='DecordDecode'),\n",
+ " dict(type='Resize', scale=(-1, 256)),\n",
+ " dict(type='FormatShape', input_format='NCTHW', collapse=True),\n",
+ " dict(type='PackActionInputs')\n",
+ " ],\n",
+ " num_classes=66,\n",
+ " proposal_file=\n",
+ " 'data/multisports/annotations/multisports_proposals_val.pkl',\n",
+ " data_prefix=dict(img='data/multisports/trainval'),\n",
+ " test_mode=True,\n",
+ " timestamp_start=1,\n",
+ " start_index=0,\n",
+ " use_frames=False,\n",
+ " fps=1,\n",
+ " _scope_='mmaction'))\n",
+ "test_dataloader = dict(\n",
+ " batch_size=1,\n",
+ " num_workers=8,\n",
+ " persistent_workers=True,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=False, _scope_='mmaction'),\n",
+ " dataset=dict(\n",
+ " type='AVADataset',\n",
+ " ann_file='data/multisports/annotations/multisports_val.csv',\n",
+ " pipeline=[\n",
+ " dict(type='DecordInit', io_backend='disk'),\n",
+ " dict(\n",
+ " type='SampleAVAFrames',\n",
+ " clip_len=4,\n",
+ " frame_interval=16,\n",
+ " test_mode=True),\n",
+ " dict(type='DecordDecode'),\n",
+ " dict(type='Resize', scale=(-1, 256)),\n",
+ " dict(type='FormatShape', input_format='NCTHW', collapse=True),\n",
+ " dict(type='PackActionInputs')\n",
+ " ],\n",
+ " num_classes=66,\n",
+ " proposal_file=\n",
+ " 'data/multisports/annotations/multisports_dense_proposals_val.recall_96.13.pkl',\n",
+ " data_prefix=dict(img='data/multisports/trainval'),\n",
+ " test_mode=True,\n",
+ " timestamp_start=1,\n",
+ " start_index=0,\n",
+ " use_frames=False,\n",
+ " fps=1,\n",
+ " _scope_='mmaction'))\n",
+ "val_evaluator = dict(\n",
+ " type='MultiSportsMetric',\n",
+ " ann_file='data/multisports/annotations/multisports_GT.pkl',\n",
+ " _scope_='mmaction')\n",
+ "test_evaluator = dict(\n",
+ " type='MultiSportsMetric',\n",
+ " ann_file='data/multisports/annotations/multisports_GT.pkl',\n",
+ " _scope_='mmaction')\n",
+ "train_cfg = dict(\n",
+ " type='EpochBasedTrainLoop',\n",
+ " max_epochs=8,\n",
+ " val_begin=1,\n",
+ " val_interval=1,\n",
+ " _scope_='mmaction')\n",
+ "val_cfg = dict(type='ValLoop', _scope_='mmaction')\n",
+ "test_cfg = dict(type='TestLoop', _scope_='mmaction')\n",
+ "param_scheduler = [\n",
+ " dict(\n",
+ " type='LinearLR',\n",
+ " start_factor=0.1,\n",
+ " by_epoch=True,\n",
+ " begin=0,\n",
+ " end=5,\n",
+ " _scope_='mmaction'),\n",
+ " dict(\n",
+ " type='MultiStepLR',\n",
+ " begin=0,\n",
+ " end=8,\n",
+ " by_epoch=True,\n",
+ " milestones=[6, 7],\n",
+ " gamma=0.1,\n",
+ " _scope_='mmaction')\n",
+ "]\n",
+ "optim_wrapper = dict(\n",
+ " optimizer=dict(\n",
+ " type='SGD',\n",
+ " lr=0.01,\n",
+ " momentum=0.9,\n",
+ " weight_decay=1e-05,\n",
+ " _scope_='mmaction'),\n",
+ " clip_grad=dict(max_norm=5, norm_type=2))\n",
+ "launcher = 'none'\n",
+ "work_dir = 'work_dirs/stad_model/'\n",
+ "randomness = dict(seed=None, diff_rank_seed=False, deterministic=False)\n",
+ "\n",
+ "06/15 06:10:23 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.\n",
+ "06/15 06:10:23 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Hooks will be executed in the following order:\n",
+ "before_run:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "before_train:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_train_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) DistSamplerSeedHook \n",
+ " -------------------- \n",
+ "before_train_iter:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_train_iter:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "after_train_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) SyncBuffersHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_val_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) SyncBuffersHook \n",
+ " -------------------- \n",
+ "before_val_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_val_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_val_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "after_train:\n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_test_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "before_test_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_test_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_test_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_run:\n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "06/15 06:10:24 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - 236 out of 236 frames are valid.\n",
+ "06/15 06:10:24 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - 120 out of 120 frames are valid.\n",
+ "06/15 06:10:25 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - load model from: https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-e7b65fad.pth\n",
+ "06/15 06:10:25 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Loads checkpoint by http backend from path: https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-e7b65fad.pth\n",
+ "Downloading: \"https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-e7b65fad.pth\" to /root/.cache/torch/hub/checkpoints/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-e7b65fad.pth\n",
+ "100% 124M/124M [00:01<00:00, 103MB/s]\n",
+ "06/15 06:10:28 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - The model and loaded state dict do not match exactly\n",
+ "\n",
+ "unexpected key in source state_dict: cls_head.fc_cls.weight, cls_head.fc_cls.bias\n",
+ "\n",
+ "missing keys in source state_dict: roi_head.bbox_head.fc_cls.weight, roi_head.bbox_head.fc_cls.bias\n",
+ "\n",
+ "Loads checkpoint by http backend from path: https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-a1ca5e76.pth\n",
+ "Downloading: \"https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-a1ca5e76.pth\" to /root/.cache/torch/hub/checkpoints/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-a1ca5e76.pth\n",
+ "100% 122M/122M [00:03<00:00, 36.1MB/s]\n",
+ "06/15 06:10:32 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Load checkpoint from https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-a1ca5e76.pth\n",
+ "06/15 06:10:32 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - \"FileClient\" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io\n",
+ "06/15 06:10:32 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - \"HardDiskBackend\" is the alias of \"LocalBackend\" and the former will be deprecated in future.\n",
+ "06/15 06:10:32 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Checkpoints will be saved to /content/mmaction2/projects/stad_tutorial/work_dirs/stad_model.\n",
+ "06/15 06:10:40 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][ 20/118] lr: 1.0000e-03 eta: 0:06:07 time: 0.3982 data_time: 0.0431 memory: 1383 grad_norm: 13.0844 loss: 1.3834 recall@thr=0.5: 0.5385 prec@thr=0.5: 0.5385 recall@top3: 0.8462 prec@top3: 0.2821 recall@top5: 0.8462 prec@top5: 0.1692 loss_action_cls: 1.3834\n",
+ "06/15 06:10:46 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][ 40/118] lr: 1.0000e-03 eta: 0:05:32 time: 0.3383 data_time: 0.0732 memory: 1383 grad_norm: 4.6786 loss: 0.6001 recall@thr=0.5: 0.9444 prec@thr=0.5: 0.9444 recall@top3: 0.9444 prec@top3: 0.3148 recall@top5: 0.9444 prec@top5: 0.1889 loss_action_cls: 0.6001\n",
+ "06/15 06:10:52 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][ 60/118] lr: 1.0000e-03 eta: 0:04:59 time: 0.2784 data_time: 0.0300 memory: 1383 grad_norm: 2.9446 loss: 0.5144 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.5144\n",
+ "06/15 06:10:57 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][ 80/118] lr: 1.0000e-03 eta: 0:04:36 time: 0.2646 data_time: 0.0144 memory: 1383 grad_norm: 1.7695 loss: 0.4988 recall@thr=0.5: 0.6923 prec@thr=0.5: 0.6923 recall@top3: 0.6923 prec@top3: 0.2308 recall@top5: 0.6923 prec@top5: 0.1385 loss_action_cls: 0.4988\n",
+ "06/15 06:11:04 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][100/118] lr: 1.0000e-03 eta: 0:04:35 time: 0.3502 data_time: 0.0839 memory: 1383 grad_norm: 2.4095 loss: 0.3218 recall@thr=0.5: 0.9333 prec@thr=0.5: 0.9333 recall@top3: 0.9333 prec@top3: 0.3111 recall@top5: 0.9333 prec@top5: 0.1867 loss_action_cls: 0.3218\n",
+ "06/15 06:11:09 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_061017\n",
+ "06/15 06:11:09 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][118/118] lr: 1.0000e-03 eta: 0:04:20 time: 0.2563 data_time: 0.0102 memory: 1383 grad_norm: 1.8156 loss: 0.3895 recall@thr=0.5: 0.8125 prec@thr=0.5: 0.8125 recall@top3: 0.9375 prec@top3: 0.3125 recall@top5: 0.9375 prec@top5: 0.1875 loss_action_cls: 0.3895\n",
+ "06/15 06:11:09 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 1 epochs\n",
+ "06/15 06:11:14 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][ 20/120] eta: 0:00:16 time: 0.1669 data_time: 0.1073 memory: 466 \n",
+ "06/15 06:11:18 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][ 40/120] eta: 0:00:13 time: 0.1698 data_time: 0.1145 memory: 466 \n",
+ "06/15 06:11:20 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][ 60/120] eta: 0:00:09 time: 0.1428 data_time: 0.0896 memory: 466 \n",
+ "06/15 06:11:22 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][ 80/120] eta: 0:00:05 time: 0.0998 data_time: 0.0504 memory: 466 \n",
+ "06/15 06:11:25 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][100/120] eta: 0:00:02 time: 0.1122 data_time: 0.0612 memory: 466 \n",
+ "06/15 06:11:27 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][120/120] eta: 0:00:00 time: 0.1031 data_time: 0.0528 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 59.66\n",
+ "aerobic split jump 30.80\n",
+ "aerobic scissors leap 88.34\n",
+ "aerobic turn 98.48\n",
+ "mAP 69.32\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 25.00\n",
+ "aerobic split jump 20.00\n",
+ "aerobic scissors leap 80.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 56.25\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 25.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 50.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 43.75\n",
+ "06/15 06:11:27 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][120/120] mAP/frameAP: 69.3181 mAP/v_map@0.2: 56.2500 mAP/v_map@0.5: 43.7500 mAP/v_map_0.05:0.45: 55.1389 mAP/v_map_0.10:0.90: 41.2500 mAP/v_map_0.50:0.95: 28.1750 data_time: 0.0793 time: 0.1324\n",
+ "06/15 06:11:29 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - The best checkpoint with 69.3181 mAP/frameAP at 1 epoch is saved to best_mAP_frameAP_epoch_1.pth.\n",
+ "06/15 06:11:40 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][ 20/118] lr: 3.2500e-03 eta: 0:04:10 time: 0.2884 data_time: 0.0401 memory: 1383 grad_norm: 1.3823 loss: 0.3596 recall@thr=0.5: 0.6923 prec@thr=0.5: 0.6923 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3596\n",
+ "06/15 06:11:46 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][ 40/118] lr: 3.2500e-03 eta: 0:04:00 time: 0.2728 data_time: 0.0204 memory: 1383 grad_norm: 1.2185 loss: 0.5274 recall@thr=0.5: 0.9333 prec@thr=0.5: 0.9333 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.5274\n",
+ "06/15 06:11:52 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][ 60/118] lr: 3.2500e-03 eta: 0:03:56 time: 0.3296 data_time: 0.0699 memory: 1383 grad_norm: 1.7120 loss: 0.3599 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3599\n",
+ "06/15 06:11:57 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][ 80/118] lr: 3.2500e-03 eta: 0:03:46 time: 0.2584 data_time: 0.0120 memory: 1383 grad_norm: 1.7462 loss: 0.2598 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2598\n",
+ "06/15 06:12:03 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][100/118] lr: 3.2500e-03 eta: 0:03:39 time: 0.2858 data_time: 0.0263 memory: 1383 grad_norm: 0.8975 loss: 0.3959 recall@thr=0.5: 0.7692 prec@thr=0.5: 0.7692 recall@top3: 0.9231 prec@top3: 0.3077 recall@top5: 0.9231 prec@top5: 0.1846 loss_action_cls: 0.3959\n",
+ "06/15 06:12:09 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_061017\n",
+ "06/15 06:12:09 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][118/118] lr: 3.2500e-03 eta: 0:03:35 time: 0.3381 data_time: 0.0807 memory: 1383 grad_norm: 0.5466 loss: 0.4871 recall@thr=0.5: 0.8333 prec@thr=0.5: 0.8333 recall@top3: 0.8333 prec@top3: 0.2778 recall@top5: 0.8333 prec@top5: 0.1667 loss_action_cls: 0.4871\n",
+ "06/15 06:12:09 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 2 epochs\n",
+ "06/15 06:12:13 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][ 20/120] eta: 0:00:12 time: 0.1230 data_time: 0.0693 memory: 466 \n",
+ "06/15 06:12:15 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][ 40/120] eta: 0:00:09 time: 0.1138 data_time: 0.0632 memory: 466 \n",
+ "06/15 06:12:18 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][ 60/120] eta: 0:00:07 time: 0.1214 data_time: 0.0672 memory: 466 \n",
+ "06/15 06:12:21 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][ 80/120] eta: 0:00:05 time: 0.1539 data_time: 0.1001 memory: 466 \n",
+ "06/15 06:12:24 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][100/120] eta: 0:00:02 time: 0.1488 data_time: 0.0936 memory: 466 \n",
+ "06/15 06:12:26 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][120/120] eta: 0:00:00 time: 0.1030 data_time: 0.0539 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 39.91\n",
+ "aerobic split jump 29.66\n",
+ "aerobic scissors leap 90.70\n",
+ "aerobic turn 96.92\n",
+ "mAP 64.30\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 20.00\n",
+ "aerobic scissors leap 100.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 55.00\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 36.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 34.00\n",
+ "06/15 06:12:27 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][120/120] mAP/frameAP: 64.2982 mAP/v_map@0.2: 55.0000 mAP/v_map@0.5: 34.0000 mAP/v_map_0.05:0.45: 53.8889 mAP/v_map_0.10:0.90: 34.5833 mAP/v_map_0.50:0.95: 19.1250 data_time: 0.0744 time: 0.1270\n",
+ "06/15 06:12:32 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [3][ 20/118] lr: 5.5000e-03 eta: 0:03:28 time: 0.2786 data_time: 0.0358 memory: 1383 grad_norm: 1.0935 loss: 0.3780 recall@thr=0.5: 0.8667 prec@thr=0.5: 0.8667 recall@top3: 0.8667 prec@top3: 0.2889 recall@top5: 0.8667 prec@top5: 0.1733 loss_action_cls: 0.3780\n",
+ "06/15 06:12:39 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [3][ 40/118] lr: 5.5000e-03 eta: 0:03:22 time: 0.3217 data_time: 0.0573 memory: 1383 grad_norm: 1.4278 loss: 0.3261 recall@thr=0.5: 0.8750 prec@thr=0.5: 0.8750 recall@top3: 0.9375 prec@top3: 0.3125 recall@top5: 0.9375 prec@top5: 0.1875 loss_action_cls: 0.3261\n",
+ "06/15 06:12:44 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [3][ 60/118] lr: 5.5000e-03 eta: 0:03:15 time: 0.2823 data_time: 0.0358 memory: 1383 grad_norm: 0.6230 loss: 0.4514 recall@thr=0.5: 0.9286 prec@thr=0.5: 0.9286 recall@top3: 0.9286 prec@top3: 0.3095 recall@top5: 0.9286 prec@top5: 0.1857 loss_action_cls: 0.4514\n",
+ "06/15 06:12:49 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [3][ 80/118] lr: 5.5000e-03 eta: 0:03:08 time: 0.2561 data_time: 0.0115 memory: 1383 grad_norm: 0.1768 loss: 0.3241 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3241\n",
+ "06/15 06:12:56 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [3][100/118] lr: 5.5000e-03 eta: 0:03:02 time: 0.3094 data_time: 0.0422 memory: 1383 grad_norm: 0.4979 loss: 0.4081 recall@thr=0.5: 0.8333 prec@thr=0.5: 0.8333 recall@top3: 0.8333 prec@top3: 0.2778 recall@top5: 0.8333 prec@top5: 0.1667 loss_action_cls: 0.4081\n",
+ "06/15 06:13:01 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_061017\n",
+ "06/15 06:13:01 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [3][118/118] lr: 5.5000e-03 eta: 0:02:56 time: 0.2776 data_time: 0.0266 memory: 1383 grad_norm: 0.7488 loss: 0.4131 recall@thr=0.5: 0.6667 prec@thr=0.5: 0.6667 recall@top3: 0.6667 prec@top3: 0.2222 recall@top5: 0.6667 prec@top5: 0.1333 loss_action_cls: 0.4131\n",
+ "06/15 06:13:01 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 3 epochs\n",
+ "06/15 06:13:05 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [3][ 20/120] eta: 0:00:11 time: 0.1182 data_time: 0.0691 memory: 466 \n",
+ "06/15 06:13:07 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [3][ 40/120] eta: 0:00:09 time: 0.1132 data_time: 0.0628 memory: 466 \n",
+ "06/15 06:13:10 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [3][ 60/120] eta: 0:00:07 time: 0.1542 data_time: 0.0996 memory: 466 \n",
+ "06/15 06:13:13 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [3][ 80/120] eta: 0:00:05 time: 0.1479 data_time: 0.0937 memory: 466 \n",
+ "06/15 06:13:15 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [3][100/120] eta: 0:00:02 time: 0.1232 data_time: 0.0726 memory: 466 \n",
+ "06/15 06:13:17 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [3][120/120] eta: 0:00:00 time: 0.1029 data_time: 0.0529 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 29.65\n",
+ "aerobic split jump 20.83\n",
+ "aerobic scissors leap 90.63\n",
+ "aerobic turn 97.10\n",
+ "mAP 59.55\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 100.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 50.00\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 36.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 34.00\n",
+ "06/15 06:13:18 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [3][120/120] mAP/frameAP: 59.5538 mAP/v_map@0.2: 50.0000 mAP/v_map@0.5: 34.0000 mAP/v_map_0.05:0.45: 50.0000 mAP/v_map_0.10:0.90: 32.9167 mAP/v_map_0.50:0.95: 19.1250 data_time: 0.0750 time: 0.1264\n",
+ "06/15 06:13:24 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [4][ 20/118] lr: 7.7500e-03 eta: 0:02:50 time: 0.3089 data_time: 0.0514 memory: 1383 grad_norm: 0.2046 loss: 0.3238 recall@thr=0.5: 0.9091 prec@thr=0.5: 0.9091 recall@top3: 0.9091 prec@top3: 0.3030 recall@top5: 0.9091 prec@top5: 0.1818 loss_action_cls: 0.3238\n",
+ "06/15 06:13:32 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [4][ 40/118] lr: 7.7500e-03 eta: 0:02:46 time: 0.3790 data_time: 0.0937 memory: 1383 grad_norm: 0.7468 loss: 0.4123 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.4123\n",
+ "06/15 06:13:37 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [4][ 60/118] lr: 7.7500e-03 eta: 0:02:39 time: 0.2685 data_time: 0.0171 memory: 1383 grad_norm: 0.1904 loss: 0.4407 recall@thr=0.5: 0.6667 prec@thr=0.5: 0.6667 recall@top3: 0.6667 prec@top3: 0.2222 recall@top5: 0.6667 prec@top5: 0.1333 loss_action_cls: 0.4407\n",
+ "06/15 06:13:42 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [4][ 80/118] lr: 7.7500e-03 eta: 0:02:32 time: 0.2546 data_time: 0.0100 memory: 1383 grad_norm: 0.1966 loss: 0.4266 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.4266\n",
+ "06/15 06:13:49 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [4][100/118] lr: 7.7500e-03 eta: 0:02:27 time: 0.3283 data_time: 0.0548 memory: 1383 grad_norm: 0.3165 loss: 0.3308 recall@thr=0.5: 0.8000 prec@thr=0.5: 0.8000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3308\n",
+ "06/15 06:13:53 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_061017\n",
+ "06/15 06:13:53 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [4][118/118] lr: 7.7500e-03 eta: 0:02:21 time: 0.2671 data_time: 0.0151 memory: 1383 grad_norm: 0.1487 loss: 0.3003 recall@thr=0.5: 0.8333 prec@thr=0.5: 0.8333 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3003\n",
+ "06/15 06:13:53 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 4 epochs\n",
+ "06/15 06:13:58 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [4][ 20/120] eta: 0:00:12 time: 0.1273 data_time: 0.0729 memory: 466 \n",
+ "06/15 06:14:00 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [4][ 40/120] eta: 0:00:10 time: 0.1306 data_time: 0.0797 memory: 466 \n",
+ "06/15 06:14:03 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [4][ 60/120] eta: 0:00:08 time: 0.1539 data_time: 0.0979 memory: 466 \n",
+ "06/15 06:14:06 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [4][ 80/120] eta: 0:00:05 time: 0.1355 data_time: 0.0815 memory: 466 \n",
+ "06/15 06:14:08 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [4][100/120] eta: 0:00:02 time: 0.1132 data_time: 0.0646 memory: 466 \n",
+ "06/15 06:14:10 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [4][120/120] eta: 0:00:00 time: 0.1050 data_time: 0.0553 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 23.92\n",
+ "aerobic split jump 19.60\n",
+ "aerobic scissors leap 91.02\n",
+ "aerobic turn 96.05\n",
+ "mAP 57.64\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 100.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 50.00\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 36.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 34.00\n",
+ "06/15 06:14:11 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [4][120/120] mAP/frameAP: 57.6444 mAP/v_map@0.2: 50.0000 mAP/v_map@0.5: 34.0000 mAP/v_map_0.05:0.45: 50.0000 mAP/v_map_0.10:0.90: 32.9167 mAP/v_map_0.50:0.95: 18.3250 data_time: 0.0753 time: 0.1274\n",
+ "06/15 06:14:17 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [5][ 20/118] lr: 1.0000e-02 eta: 0:02:14 time: 0.2810 data_time: 0.0329 memory: 1383 grad_norm: 0.6113 loss: 0.4312 recall@thr=0.5: 0.8182 prec@thr=0.5: 0.8182 recall@top3: 0.8182 prec@top3: 0.2727 recall@top5: 0.8182 prec@top5: 0.1636 loss_action_cls: 0.4312\n",
+ "06/15 06:14:23 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [5][ 40/118] lr: 1.0000e-02 eta: 0:02:09 time: 0.3316 data_time: 0.0732 memory: 1383 grad_norm: 0.2282 loss: 0.3932 recall@thr=0.5: 0.8182 prec@thr=0.5: 0.8182 recall@top3: 0.8182 prec@top3: 0.2727 recall@top5: 0.8182 prec@top5: 0.1636 loss_action_cls: 0.3932\n",
+ "06/15 06:14:29 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [5][ 60/118] lr: 1.0000e-02 eta: 0:02:03 time: 0.2738 data_time: 0.0286 memory: 1383 grad_norm: 0.2938 loss: 0.3828 recall@thr=0.5: 0.8571 prec@thr=0.5: 0.8571 recall@top3: 0.8571 prec@top3: 0.2857 recall@top5: 0.8571 prec@top5: 0.1714 loss_action_cls: 0.3828\n",
+ "06/15 06:14:34 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [5][ 80/118] lr: 1.0000e-02 eta: 0:01:56 time: 0.2756 data_time: 0.0192 memory: 1383 grad_norm: 0.1112 loss: 0.3722 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3722\n",
+ "06/15 06:14:41 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [5][100/118] lr: 1.0000e-02 eta: 0:01:51 time: 0.3193 data_time: 0.0573 memory: 1383 grad_norm: 0.6399 loss: 0.4427 recall@thr=0.5: 0.8000 prec@thr=0.5: 0.8000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.4427\n",
+ "06/15 06:14:45 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_061017\n",
+ "06/15 06:14:45 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [5][118/118] lr: 1.0000e-02 eta: 0:01:45 time: 0.2535 data_time: 0.0093 memory: 1383 grad_norm: 0.0985 loss: 0.2719 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2719\n",
+ "06/15 06:14:45 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 5 epochs\n",
+ "06/15 06:14:50 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [5][ 20/120] eta: 0:00:13 time: 0.1329 data_time: 0.0774 memory: 466 \n",
+ "06/15 06:14:53 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [5][ 40/120] eta: 0:00:12 time: 0.1787 data_time: 0.1259 memory: 466 \n",
+ "06/15 06:14:56 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [5][ 60/120] eta: 0:00:08 time: 0.1363 data_time: 0.0829 memory: 466 \n",
+ "06/15 06:14:58 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [5][ 80/120] eta: 0:00:05 time: 0.1012 data_time: 0.0513 memory: 466 \n",
+ "06/15 06:15:00 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [5][100/120] eta: 0:00:02 time: 0.1095 data_time: 0.0593 memory: 466 \n",
+ "06/15 06:15:02 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [5][120/120] eta: 0:00:00 time: 0.1033 data_time: 0.0536 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 14.21\n",
+ "aerobic split jump 15.37\n",
+ "aerobic scissors leap 91.25\n",
+ "aerobic turn 91.43\n",
+ "mAP 53.06\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 100.00\n",
+ "aerobic turn 80.00\n",
+ "mAP 45.00\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 36.00\n",
+ "aerobic turn 20.00\n",
+ "mAP 14.00\n",
+ "06/15 06:15:03 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [5][120/120] mAP/frameAP: 53.0627 mAP/v_map@0.2: 45.0000 mAP/v_map@0.5: 14.0000 mAP/v_map_0.05:0.45: 40.0000 mAP/v_map_0.10:0.90: 22.4444 mAP/v_map_0.50:0.95: 7.0250 data_time: 0.0749 time: 0.1267\n",
+ "06/15 06:15:09 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [6][ 20/118] lr: 1.0000e-02 eta: 0:01:39 time: 0.3193 data_time: 0.0634 memory: 1383 grad_norm: 0.5229 loss: 0.3929 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3929\n",
+ "06/15 06:15:15 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [6][ 40/118] lr: 1.0000e-02 eta: 0:01:33 time: 0.2972 data_time: 0.0439 memory: 1383 grad_norm: 0.4621 loss: 0.2891 recall@thr=0.5: 0.7692 prec@thr=0.5: 0.7692 recall@top3: 0.9231 prec@top3: 0.3077 recall@top5: 0.9231 prec@top5: 0.1846 loss_action_cls: 0.2891\n",
+ "06/15 06:15:20 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [6][ 60/118] lr: 1.0000e-02 eta: 0:01:27 time: 0.2567 data_time: 0.0127 memory: 1383 grad_norm: 0.2534 loss: 0.3438 recall@thr=0.5: 0.9333 prec@thr=0.5: 0.9333 recall@top3: 0.9333 prec@top3: 0.3111 recall@top5: 0.9333 prec@top5: 0.1867 loss_action_cls: 0.3438\n",
+ "06/15 06:15:27 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [6][ 80/118] lr: 1.0000e-02 eta: 0:01:21 time: 0.3277 data_time: 0.0645 memory: 1383 grad_norm: 0.0856 loss: 0.1859 recall@thr=0.5: 0.8571 prec@thr=0.5: 0.8571 recall@top3: 0.8571 prec@top3: 0.2857 recall@top5: 0.8571 prec@top5: 0.1714 loss_action_cls: 0.1859\n",
+ "06/15 06:15:33 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [6][100/118] lr: 1.0000e-02 eta: 0:01:15 time: 0.2995 data_time: 0.0503 memory: 1383 grad_norm: 0.3619 loss: 0.3205 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3205\n",
+ "06/15 06:15:37 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_061017\n",
+ "06/15 06:15:37 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [6][118/118] lr: 1.0000e-02 eta: 0:01:10 time: 0.2619 data_time: 0.0190 memory: 1383 grad_norm: 0.3812 loss: 0.3911 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3911\n",
+ "06/15 06:15:37 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 6 epochs\n",
+ "06/15 06:15:43 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [6][ 20/120] eta: 0:00:17 time: 0.1739 data_time: 0.1178 memory: 466 \n",
+ "06/15 06:15:46 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [6][ 40/120] eta: 0:00:13 time: 0.1519 data_time: 0.1032 memory: 466 \n",
+ "06/15 06:15:48 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [6][ 60/120] eta: 0:00:08 time: 0.1031 data_time: 0.0536 memory: 466 \n",
+ "06/15 06:15:50 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [6][ 80/120] eta: 0:00:05 time: 0.0998 data_time: 0.0505 memory: 466 \n",
+ "06/15 06:15:52 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [6][100/120] eta: 0:00:02 time: 0.1126 data_time: 0.0620 memory: 466 \n",
+ "06/15 06:15:54 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [6][120/120] eta: 0:00:00 time: 0.0995 data_time: 0.0506 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 10.49\n",
+ "aerobic split jump 14.53\n",
+ "aerobic scissors leap 90.24\n",
+ "aerobic turn 87.53\n",
+ "mAP 50.70\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 100.00\n",
+ "aerobic turn 40.00\n",
+ "mAP 35.00\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 36.00\n",
+ "aerobic turn 40.00\n",
+ "mAP 19.00\n",
+ "06/15 06:15:55 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [6][120/120] mAP/frameAP: 50.6970 mAP/v_map@0.2: 35.0000 mAP/v_map@0.5: 19.0000 mAP/v_map_0.05:0.45: 35.0000 mAP/v_map_0.10:0.90: 20.7778 mAP/v_map_0.50:0.95: 8.4000 data_time: 0.0724 time: 0.1229\n",
+ "06/15 06:16:02 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [7][ 20/118] lr: 1.0000e-03 eta: 0:01:04 time: 0.3578 data_time: 0.0847 memory: 1383 grad_norm: 0.5369 loss: 0.3628 recall@thr=0.5: 0.9167 prec@thr=0.5: 0.9167 recall@top3: 0.9167 prec@top3: 0.3056 recall@top5: 0.9167 prec@top5: 0.1833 loss_action_cls: 0.3628\n",
+ "06/15 06:16:07 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [7][ 40/118] lr: 1.0000e-03 eta: 0:00:58 time: 0.2652 data_time: 0.0202 memory: 1383 grad_norm: 0.1603 loss: 0.2293 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2293\n",
+ "06/15 06:16:13 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [7][ 60/118] lr: 1.0000e-03 eta: 0:00:52 time: 0.2710 data_time: 0.0178 memory: 1383 grad_norm: 0.3857 loss: 0.2737 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2737\n",
+ "06/15 06:16:20 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [7][ 80/118] lr: 1.0000e-03 eta: 0:00:46 time: 0.3420 data_time: 0.0698 memory: 1383 grad_norm: 0.1271 loss: 0.2149 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2149\n",
+ "06/15 06:16:25 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [7][100/118] lr: 1.0000e-03 eta: 0:00:40 time: 0.2673 data_time: 0.0232 memory: 1383 grad_norm: 0.0990 loss: 0.2749 recall@thr=0.5: 0.8571 prec@thr=0.5: 0.8571 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2749\n",
+ "06/15 06:16:30 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_061017\n",
+ "06/15 06:16:30 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [7][118/118] lr: 1.0000e-03 eta: 0:00:34 time: 0.2612 data_time: 0.0156 memory: 1383 grad_norm: 0.1387 loss: 0.3211 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3211\n",
+ "06/15 06:16:30 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 7 epochs\n",
+ "06/15 06:16:35 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [7][ 20/120] eta: 0:00:16 time: 0.1657 data_time: 0.1063 memory: 466 \n",
+ "06/15 06:16:38 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [7][ 40/120] eta: 0:00:11 time: 0.1164 data_time: 0.0654 memory: 466 \n",
+ "06/15 06:16:40 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [7][ 60/120] eta: 0:00:07 time: 0.1053 data_time: 0.0546 memory: 466 \n",
+ "06/15 06:16:42 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [7][ 80/120] eta: 0:00:04 time: 0.1005 data_time: 0.0511 memory: 466 \n",
+ "06/15 06:16:44 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [7][100/120] eta: 0:00:02 time: 0.1035 data_time: 0.0533 memory: 466 \n",
+ "06/15 06:16:47 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [7][120/120] eta: 0:00:00 time: 0.1382 data_time: 0.0850 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 11.65\n",
+ "aerobic split jump 15.62\n",
+ "aerobic scissors leap 89.83\n",
+ "aerobic turn 93.96\n",
+ "mAP 52.77\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 100.00\n",
+ "aerobic turn 80.00\n",
+ "mAP 45.00\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 38.67\n",
+ "aerobic turn 20.00\n",
+ "mAP 14.67\n",
+ "06/15 06:16:48 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [7][120/120] mAP/frameAP: 52.7652 mAP/v_map@0.2: 45.0000 mAP/v_map@0.5: 14.6667 mAP/v_map_0.05:0.45: 40.6944 mAP/v_map_0.10:0.90: 22.6389 mAP/v_map_0.50:0.95: 6.6833 data_time: 0.0691 time: 0.1213\n",
+ "06/15 06:16:54 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [8][ 20/118] lr: 1.0000e-04 eta: 0:00:29 time: 0.3243 data_time: 0.0649 memory: 1383 grad_norm: 0.1808 loss: 0.3648 recall@thr=0.5: 0.8571 prec@thr=0.5: 0.8571 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3648\n",
+ "06/15 06:16:59 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [8][ 40/118] lr: 1.0000e-04 eta: 0:00:23 time: 0.2578 data_time: 0.0117 memory: 1383 grad_norm: 0.0784 loss: 0.2355 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2355\n",
+ "06/15 06:17:06 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [8][ 60/118] lr: 1.0000e-04 eta: 0:00:17 time: 0.3075 data_time: 0.0490 memory: 1383 grad_norm: 0.1707 loss: 0.3776 recall@thr=0.5: 0.9333 prec@thr=0.5: 0.9333 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3776\n",
+ "06/15 06:17:12 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [8][ 80/118] lr: 1.0000e-04 eta: 0:00:11 time: 0.3092 data_time: 0.0576 memory: 1383 grad_norm: 0.1387 loss: 0.3873 recall@thr=0.5: 0.8182 prec@thr=0.5: 0.8182 recall@top3: 0.8182 prec@top3: 0.2727 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3873\n",
+ "06/15 06:17:17 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [8][100/118] lr: 1.0000e-04 eta: 0:00:05 time: 0.2578 data_time: 0.0100 memory: 1383 grad_norm: 0.2137 loss: 0.3337 recall@thr=0.5: 0.8462 prec@thr=0.5: 0.8462 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3337\n",
+ "06/15 06:17:22 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_061017\n",
+ "06/15 06:17:22 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [8][118/118] lr: 1.0000e-04 eta: 0:00:00 time: 0.2755 data_time: 0.0148 memory: 1383 grad_norm: 0.0712 loss: 0.2038 recall@thr=0.5: 0.9091 prec@thr=0.5: 0.9091 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2038\n",
+ "06/15 06:17:22 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 8 epochs\n",
+ "06/15 06:17:27 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [8][ 20/120] eta: 0:00:11 time: 0.1180 data_time: 0.0649 memory: 466 \n",
+ "06/15 06:17:29 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [8][ 40/120] eta: 0:00:09 time: 0.1168 data_time: 0.0667 memory: 466 \n",
+ "06/15 06:17:31 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [8][ 60/120] eta: 0:00:06 time: 0.1026 data_time: 0.0535 memory: 466 \n",
+ "06/15 06:17:33 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [8][ 80/120] eta: 0:00:04 time: 0.1017 data_time: 0.0533 memory: 466 \n",
+ "06/15 06:17:36 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [8][100/120] eta: 0:00:02 time: 0.1444 data_time: 0.0915 memory: 466 \n",
+ "06/15 06:17:39 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [8][120/120] eta: 0:00:00 time: 0.1496 data_time: 0.0962 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 11.34\n",
+ "aerobic split jump 12.82\n",
+ "aerobic scissors leap 90.68\n",
+ "aerobic turn 90.47\n",
+ "mAP 51.33\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 100.00\n",
+ "aerobic turn 80.00\n",
+ "mAP 45.00\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 72.00\n",
+ "aerobic turn 20.00\n",
+ "mAP 23.00\n",
+ "06/15 06:17:40 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [8][120/120] mAP/frameAP: 51.3281 mAP/v_map@0.2: 45.0000 mAP/v_map@0.5: 23.0000 mAP/v_map_0.05:0.45: 40.0000 mAP/v_map_0.10:0.90: 24.4444 mAP/v_map_0.50:0.95: 9.7250 data_time: 0.0704 time: 0.1216\n",
+ "\u001b[32mTraining finished successfully. \u001b[0m\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Train the model using MIM\n",
+ "!mim train mmaction2 configs/slowonly_k400_multisports.py \\\n",
+ " --work-dir work_dirs/stad_model/"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yVjHqupPooZ2"
+ },
+ "source": [
+ "## 4. Inferring the Spatiotemporal Action Detection Model\n",
+ "\n",
+ "After training the detection model and the spatiotemporal action detection model, we can use the spatiotemporal action detection demo for inference and visualize the model's performance.\n",
+ "\n",
+ "Since the tutorial uses a limited training dataset, the model's performance is not optimal, so a pre-trained model is used for visualization."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "NQF1yrEhooZ3",
+ "outputId": "5331fbb6-7075-415c-f6f0-ec41c4b584a4"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "ALSA lib confmisc.c:767:(parse_card) cannot find card '0'\n",
+ "ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory\n",
+ "ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings\n",
+ "ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory\n",
+ "ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name\n",
+ "ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory\n",
+ "ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory\n",
+ "ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM default\n",
+ "ALSA lib confmisc.c:767:(parse_card) cannot find card '0'\n",
+ "ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory\n",
+ "ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings\n",
+ "ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory\n",
+ "ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name\n",
+ "ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory\n",
+ "ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory\n",
+ "ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM default\n",
+ "Loads checkpoint by local backend from path: work_dirs/det_model/epoch_2.pth\n",
+ "Performing Human Detection for each frame\n",
+ "[>>] 99/99, 7.0 task/s, elapsed: 14s, ETA: 0s\n",
+ "Loads checkpoint by http backend from path: https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-a1ca5e76.pth\n",
+ "Performing SpatioTemporal Action Detection for each clip\n",
+ "[>>] 99/99, 17.1 task/s, elapsed: 6s, ETA: 0sPerforming visualization\n",
+ "Moviepy - Building video data/demo_spatiotemporal_det.mp4.\n",
+ "Moviepy - Writing video data/demo_spatiotemporal_det.mp4\n",
+ "\n",
+ "Moviepy - Done !\n",
+ "Moviepy - video ready data/demo_spatiotemporal_det.mp4\n"
+ ]
+ }
+ ],
+ "source": [
+ "!python ../../demo/demo_spatiotemporal_det.py \\\n",
+ " data/multisports/test/aerobic_gymnastics/v_7G_IpU0FxLU_c001.mp4 \\\n",
+ " data/demo_spatiotemporal_det.mp4 \\\n",
+ " --config configs/slowonly_k400_multisports.py \\\n",
+ " --checkpoint https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-a1ca5e76.pth \\\n",
+ " --det-config configs/faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person.py \\\n",
+ " --det-checkpoint work_dirs/det_model/epoch_2.pth \\\n",
+ " --det-score-thr 0.85 \\\n",
+ " --action-score-thr 0.8 \\\n",
+ " --label-map ../../tools/data/multisports/label_map.txt \\\n",
+ " --predict-stepsize 8 \\\n",
+ " --output-stepsize 1 \\\n",
+ " --output-fps 24"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 741
+ },
+ "id": "9JmeIkh5ooZ3",
+ "outputId": "7fc38469-d8c4-4a02-81e7-ff93b88a62b2"
+ },
+ "outputs": [],
+ "source": [
+ "# Show Video\n",
+ "import moviepy.editor\n",
+ "moviepy.editor.ipython_display(\"data/demo_spatiotemporal_det.mp4\")"
+ ]
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "gpuType": "T4",
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "ipy_stad",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.0"
+ },
+ "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/projects/stad_tutorial/demo_stad_zh_CN.ipynb b/projects/stad_tutorial/demo_stad_zh_CN.ipynb
new file mode 100644
index 0000000000..30df0f5624
--- /dev/null
+++ b/projects/stad_tutorial/demo_stad_zh_CN.ipynb
@@ -0,0 +1,4107 @@
+{
+ "cells": [
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "B74HkZjCxQ_6"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "MwmrGv9exRAH"
+ },
+ "source": [
+ "# 基于 MMAction2 进行时空行为检测任务\n",
+ "欢迎使用 MMAction2! 这是一篇关于如何使用 MMAction2 进行时空行为检测的教程。在此教程中，我们会以 MultiSports 数据集为例，提供时空行为检测的完整步骤教程，包括\n",
+ "- 准备时空行为检测数据集\n",
+ "- 训练检测模型\n",
+ "- 准备 AVA 格式的数据集\n",
+ "- 训练时空行为检测模型\n"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "em5lgDTUxRAI"
+ },
+ "source": [
+ "## 0. 安装 MMAction2 和 MMDetection"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "bBM9DCrsxRAJ",
+ "outputId": "b310311f-f05e-4a5c-b6e5-8e6ee7e0dfae"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
+ "Collecting openmim\n",
+ " Downloading openmim-0.3.7-py2.py3-none-any.whl (51 kB)\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m51.3/51.3 kB\u001b[0m \u001b[31m3.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25hRequirement already satisfied: Click in /usr/local/lib/python3.10/dist-packages (from openmim) (8.1.3)\n",
+ "Collecting colorama (from openmim)\n",
+ " Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n",
+ "Collecting model-index (from openmim)\n",
+ " Downloading model_index-0.1.11-py3-none-any.whl (34 kB)\n",
+ "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from openmim) (1.5.3)\n",
+ "Requirement already satisfied: pip>=19.3 in /usr/local/lib/python3.10/dist-packages (from openmim) (23.1.2)\n",
+ "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from openmim) (2.27.1)\n",
+ "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from openmim) (13.3.4)\n",
+ "Requirement already satisfied: tabulate in /usr/local/lib/python3.10/dist-packages (from openmim) (0.8.10)\n",
+ "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from model-index->openmim) (6.0)\n",
+ "Requirement already satisfied: markdown in /usr/local/lib/python3.10/dist-packages (from model-index->openmim) (3.4.3)\n",
+ "Collecting ordered-set (from model-index->openmim)\n",
+ " Downloading ordered_set-4.1.0-py3-none-any.whl (7.6 kB)\n",
+ "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->openmim) (2.8.2)\n",
+ "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->openmim) (2022.7.1)\n",
+ "Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.10/dist-packages (from pandas->openmim) (1.22.4)\n",
+ "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->openmim) (1.26.15)\n",
+ "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->openmim) (2022.12.7)\n",
+ "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->openmim) (2.0.12)\n",
+ "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->openmim) (3.4)\n",
+ "Requirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->openmim) (2.2.0)\n",
+ "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->openmim) (2.14.0)\n",
+ "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py<3.0.0,>=2.2.0->rich->openmim) (0.1.2)\n",
+ "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas->openmim) (1.16.0)\n",
+ "Installing collected packages: ordered-set, colorama, model-index, openmim\n",
+ "Successfully installed colorama-0.4.6 model-index-0.1.11 openmim-0.3.7 ordered-set-4.1.0\n",
+ "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
+ "Looking in links: https://download.openmmlab.com/mmcv/dist/cu118/torch2.0.0/index.html\n",
+ "Collecting mmengine\n",
+ " Downloading mmengine-0.7.4-py3-none-any.whl (374 kB)\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m374.3/374.3 kB\u001b[0m \u001b[31m8.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25hCollecting addict (from mmengine)\n",
+ " Downloading addict-2.4.0-py3-none-any.whl (3.8 kB)\n",
+ "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from mmengine) (3.7.1)\n",
+ "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from mmengine) (1.22.4)\n",
+ "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from mmengine) (6.0)\n",
+ "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from mmengine) (13.3.4)\n",
+ "Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from mmengine) (2.3.0)\n",
+ "Collecting yapf (from mmengine)\n",
+ " Downloading yapf-0.40.0-py3-none-any.whl (250 kB)\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m250.3/250.3 kB\u001b[0m \u001b[31m29.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25hRequirement already satisfied: opencv-python>=3 in /usr/local/lib/python3.10/dist-packages (from mmengine) (4.7.0.72)\n",
+ "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (1.0.7)\n",
+ "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (0.11.0)\n",
+ "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (4.39.3)\n",
+ "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (1.4.4)\n",
+ "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (23.1)\n",
+ "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (8.4.0)\n",
+ "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (3.0.9)\n",
+ "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine) (2.8.2)\n",
+ "Requirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->mmengine) (2.2.0)\n",
+ "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->mmengine) (2.14.0)\n",
+ "Collecting importlib-metadata>=6.6.0 (from yapf->mmengine)\n",
+ " Downloading importlib_metadata-6.6.0-py3-none-any.whl (22 kB)\n",
+ "Collecting platformdirs>=3.5.1 (from yapf->mmengine)\n",
+ " Downloading platformdirs-3.5.3-py3-none-any.whl (15 kB)\n",
+ "Requirement already satisfied: tomli>=2.0.1 in /usr/local/lib/python3.10/dist-packages (from yapf->mmengine) (2.0.1)\n",
+ "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata>=6.6.0->yapf->mmengine) (3.15.0)\n",
+ "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py<3.0.0,>=2.2.0->rich->mmengine) (0.1.2)\n",
+ "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib->mmengine) (1.16.0)\n",
+ "Installing collected packages: addict, platformdirs, importlib-metadata, yapf, mmengine\n",
+ " Attempting uninstall: platformdirs\n",
+ " Found existing installation: platformdirs 3.3.0\n",
+ " Uninstalling platformdirs-3.3.0:\n",
+ " Successfully uninstalled platformdirs-3.3.0\n",
+ "Successfully installed addict-2.4.0 importlib-metadata-6.6.0 mmengine-0.7.4 platformdirs-3.5.3 yapf-0.40.0\n",
+ "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
+ "Looking in links: https://download.openmmlab.com/mmcv/dist/cu118/torch2.0.0/index.html\n",
+ "Collecting mmcv\n",
+ " Downloading https://download.openmmlab.com/mmcv/dist/cu118/torch2.0.0/mmcv-2.0.0-cp310-cp310-manylinux1_x86_64.whl (74.4 MB)\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m74.4/74.4 MB\u001b[0m \u001b[31m9.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25hRequirement already satisfied: addict in /usr/local/lib/python3.10/dist-packages (from mmcv) (2.4.0)\n",
+ "Requirement already satisfied: mmengine>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from mmcv) (0.7.4)\n",
+ "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from mmcv) (1.22.4)\n",
+ "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from mmcv) (23.1)\n",
+ "Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from mmcv) (8.4.0)\n",
+ "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from mmcv) (6.0)\n",
+ "Requirement already satisfied: yapf in /usr/local/lib/python3.10/dist-packages (from mmcv) (0.40.0)\n",
+ "Requirement already satisfied: opencv-python>=3 in /usr/local/lib/python3.10/dist-packages (from mmcv) (4.7.0.72)\n",
+ "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from mmengine>=0.2.0->mmcv) (3.7.1)\n",
+ "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from mmengine>=0.2.0->mmcv) (13.3.4)\n",
+ "Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from mmengine>=0.2.0->mmcv) (2.3.0)\n",
+ "Requirement already satisfied: importlib-metadata>=6.6.0 in /usr/local/lib/python3.10/dist-packages (from yapf->mmcv) (6.6.0)\n",
+ "Requirement already satisfied: platformdirs>=3.5.1 in /usr/local/lib/python3.10/dist-packages (from yapf->mmcv) (3.5.3)\n",
+ "Requirement already satisfied: tomli>=2.0.1 in /usr/local/lib/python3.10/dist-packages (from yapf->mmcv) (2.0.1)\n",
+ "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata>=6.6.0->yapf->mmcv) (3.15.0)\n",
+ "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine>=0.2.0->mmcv) (1.0.7)\n",
+ "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine>=0.2.0->mmcv) (0.11.0)\n",
+ "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine>=0.2.0->mmcv) (4.39.3)\n",
+ "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine>=0.2.0->mmcv) (1.4.4)\n",
+ "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine>=0.2.0->mmcv) (3.0.9)\n",
+ "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmengine>=0.2.0->mmcv) (2.8.2)\n",
+ "Requirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->mmengine>=0.2.0->mmcv) (2.2.0)\n",
+ "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->mmengine>=0.2.0->mmcv) (2.14.0)\n",
+ "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py<3.0.0,>=2.2.0->rich->mmengine>=0.2.0->mmcv) (0.1.2)\n",
+ "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib->mmengine>=0.2.0->mmcv) (1.16.0)\n",
+ "Installing collected packages: mmcv\n",
+ "Successfully installed mmcv-2.0.0\n",
+ "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
+ "Looking in links: https://download.openmmlab.com/mmcv/dist/cu118/torch2.0.0/index.html\n",
+ "Collecting mmdet\n",
+ " Downloading mmdet-3.0.0-py3-none-any.whl (1.7 MB)\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m23.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from mmdet) (3.7.1)\n",
+ "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from mmdet) (1.22.4)\n",
+ "Requirement already satisfied: pycocotools in /usr/local/lib/python3.10/dist-packages (from mmdet) (2.0.6)\n",
+ "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from mmdet) (1.10.1)\n",
+ "Requirement already satisfied: shapely in /usr/local/lib/python3.10/dist-packages (from mmdet) (2.0.1)\n",
+ "Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from mmdet) (1.16.0)\n",
+ "Collecting terminaltables (from mmdet)\n",
+ " Downloading terminaltables-3.1.10-py2.py3-none-any.whl (15 kB)\n",
+ "Requirement already satisfied: mmcv<2.1.0,>=2.0.0rc4 in /usr/local/lib/python3.10/dist-packages (from mmdet) (2.0.0)\n",
+ "Requirement already satisfied: mmengine<1.0.0,>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from mmdet) (0.7.4)\n",
+ "Requirement already satisfied: addict in /usr/local/lib/python3.10/dist-packages (from mmcv<2.1.0,>=2.0.0rc4->mmdet) (2.4.0)\n",
+ "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from mmcv<2.1.0,>=2.0.0rc4->mmdet) (23.1)\n",
+ "Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from mmcv<2.1.0,>=2.0.0rc4->mmdet) (8.4.0)\n",
+ "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from mmcv<2.1.0,>=2.0.0rc4->mmdet) (6.0)\n",
+ "Requirement already satisfied: yapf in /usr/local/lib/python3.10/dist-packages (from mmcv<2.1.0,>=2.0.0rc4->mmdet) (0.40.0)\n",
+ "Requirement already satisfied: opencv-python>=3 in /usr/local/lib/python3.10/dist-packages (from mmcv<2.1.0,>=2.0.0rc4->mmdet) (4.7.0.72)\n",
+ "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from mmengine<1.0.0,>=0.7.1->mmdet) (13.3.4)\n",
+ "Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from mmengine<1.0.0,>=0.7.1->mmdet) (2.3.0)\n",
+ "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmdet) (1.0.7)\n",
+ "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmdet) (0.11.0)\n",
+ "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmdet) (4.39.3)\n",
+ "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmdet) (1.4.4)\n",
+ "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmdet) (3.0.9)\n",
+ "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmdet) (2.8.2)\n",
+ "Requirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->mmengine<1.0.0,>=0.7.1->mmdet) (2.2.0)\n",
+ "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->mmengine<1.0.0,>=0.7.1->mmdet) (2.14.0)\n",
+ "Requirement already satisfied: importlib-metadata>=6.6.0 in /usr/local/lib/python3.10/dist-packages (from yapf->mmcv<2.1.0,>=2.0.0rc4->mmdet) (6.6.0)\n",
+ "Requirement already satisfied: platformdirs>=3.5.1 in /usr/local/lib/python3.10/dist-packages (from yapf->mmcv<2.1.0,>=2.0.0rc4->mmdet) (3.5.3)\n",
+ "Requirement already satisfied: tomli>=2.0.1 in /usr/local/lib/python3.10/dist-packages (from yapf->mmcv<2.1.0,>=2.0.0rc4->mmdet) (2.0.1)\n",
+ "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata>=6.6.0->yapf->mmcv<2.1.0,>=2.0.0rc4->mmdet) (3.15.0)\n",
+ "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py<3.0.0,>=2.2.0->rich->mmengine<1.0.0,>=0.7.1->mmdet) (0.1.2)\n",
+ "Installing collected packages: terminaltables, mmdet\n",
+ "Successfully installed mmdet-3.0.0 terminaltables-3.1.10\n",
+ "Cloning into 'mmaction2'...\n",
+ "remote: Enumerating objects: 22869, done.\u001b[K\n",
+ "remote: Counting objects: 100% (1491/1491), done.\u001b[K\n",
+ "remote: Compressing objects: 100% (801/801), done.\u001b[K\n",
+ "remote: Total 22869 (delta 854), reused 1171 (delta 685), pack-reused 21378\u001b[K\n",
+ "Receiving objects: 100% (22869/22869), 82.81 MiB | 27.92 MiB/s, done.\n",
+ "Resolving deltas: 100% (15952/15952), done.\n",
+ "/content/mmaction2\n",
+ "Using pip 23.1.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)\n",
+ "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
+ "Obtaining file:///content/mmaction2\n",
+ " Running command python setup.py egg_info\n",
+ " running egg_info\n",
+ " creating /tmp/pip-pip-egg-info-x4x7terp/mmaction2.egg-info\n",
+ " writing /tmp/pip-pip-egg-info-x4x7terp/mmaction2.egg-info/PKG-INFO\n",
+ " writing dependency_links to /tmp/pip-pip-egg-info-x4x7terp/mmaction2.egg-info/dependency_links.txt\n",
+ " writing requirements to /tmp/pip-pip-egg-info-x4x7terp/mmaction2.egg-info/requires.txt\n",
+ " writing top-level names to /tmp/pip-pip-egg-info-x4x7terp/mmaction2.egg-info/top_level.txt\n",
+ " writing manifest file '/tmp/pip-pip-egg-info-x4x7terp/mmaction2.egg-info/SOURCES.txt'\n",
+ " reading manifest file '/tmp/pip-pip-egg-info-x4x7terp/mmaction2.egg-info/SOURCES.txt'\n",
+ " reading manifest template 'MANIFEST.in'\n",
+ " warning: no files found matching 'mmaction/.mim/model-index.yml'\n",
+ " warning: no files found matching '*.py' under directory 'mmaction/.mim/configs'\n",
+ " warning: no files found matching '*.yml' under directory 'mmaction/.mim/configs'\n",
+ " warning: no files found matching '*.sh' under directory 'mmaction/.mim/tools'\n",
+ " warning: no files found matching '*.py' under directory 'mmaction/.mim/tools'\n",
+ " adding license file 'LICENSE'\n",
+ " writing manifest file '/tmp/pip-pip-egg-info-x4x7terp/mmaction2.egg-info/SOURCES.txt'\n",
+ " Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
+ "Collecting decord>=0.4.1 (from mmaction2==1.0.0)\n",
+ " Downloading decord-0.6.0-py3-none-manylinux2010_x86_64.whl (13.6 MB)\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.6/13.6 MB\u001b[0m \u001b[31m98.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25hCollecting einops (from mmaction2==1.0.0)\n",
+ " Downloading einops-0.6.1-py3-none-any.whl (42 kB)\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.2/42.2 kB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from mmaction2==1.0.0) (3.7.1)\n",
+ "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from mmaction2==1.0.0) (1.22.4)\n",
+ "Requirement already satisfied: opencv-contrib-python in /usr/local/lib/python3.10/dist-packages (from mmaction2==1.0.0) (4.7.0.72)\n",
+ "Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from mmaction2==1.0.0) (8.4.0)\n",
+ "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from mmaction2==1.0.0) (1.10.1)\n",
+ "Requirement already satisfied: torch>=1.3 in /usr/local/lib/python3.10/dist-packages (from mmaction2==1.0.0) (2.0.1+cu118)\n",
+ "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.3->mmaction2==1.0.0) (3.12.0)\n",
+ "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch>=1.3->mmaction2==1.0.0) (4.5.0)\n",
+ "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.3->mmaction2==1.0.0) (1.11.1)\n",
+ "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.3->mmaction2==1.0.0) (3.1)\n",
+ "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.3->mmaction2==1.0.0) (3.1.2)\n",
+ "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.3->mmaction2==1.0.0) (2.0.0)\n",
+ "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.3->mmaction2==1.0.0) (3.25.2)\n",
+ "Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.3->mmaction2==1.0.0) (16.0.5)\n",
+ "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmaction2==1.0.0) (1.0.7)\n",
+ "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmaction2==1.0.0) (0.11.0)\n",
+ "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmaction2==1.0.0) (4.39.3)\n",
+ "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmaction2==1.0.0) (1.4.4)\n",
+ "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmaction2==1.0.0) (23.1)\n",
+ "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmaction2==1.0.0) (3.0.9)\n",
+ "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->mmaction2==1.0.0) (2.8.2)\n",
+ "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib->mmaction2==1.0.0) (1.16.0)\n",
+ "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.3->mmaction2==1.0.0) (2.1.2)\n",
+ "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.3->mmaction2==1.0.0) (1.3.0)\n",
+ "Installing collected packages: einops, decord, mmaction2\n",
+ " Running setup.py develop for mmaction2\n",
+ " Running command python setup.py develop\n",
+ " running develop\n",
+ " /usr/local/lib/python3.10/dist-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.\n",
+ " !!\n",
+ "\n",
+ " ********************************************************************************\n",
+ " Please avoid running ``setup.py`` and ``easy_install``.\n",
+ " Instead, use pypa/build, pypa/installer, pypa/build or\n",
+ " other standards-based tools.\n",
+ "\n",
+ " See https://github.com/pypa/setuptools/issues/917 for details.\n",
+ " ********************************************************************************\n",
+ "\n",
+ " !!\n",
+ " easy_install.initialize_options(self)\n",
+ " /usr/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.\n",
+ " !!\n",
+ "\n",
+ " ********************************************************************************\n",
+ " Please avoid running ``setup.py`` directly.\n",
+ " Instead, use pypa/build, pypa/installer, pypa/build or\n",
+ " other standards-based tools.\n",
+ "\n",
+ " See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.\n",
+ " ********************************************************************************\n",
+ "\n",
+ " !!\n",
+ " self.initialize_options()\n",
+ " running egg_info\n",
+ " creating mmaction2.egg-info\n",
+ " writing mmaction2.egg-info/PKG-INFO\n",
+ " writing dependency_links to mmaction2.egg-info/dependency_links.txt\n",
+ " writing requirements to mmaction2.egg-info/requires.txt\n",
+ " writing top-level names to mmaction2.egg-info/top_level.txt\n",
+ " writing manifest file 'mmaction2.egg-info/SOURCES.txt'\n",
+ " reading manifest file 'mmaction2.egg-info/SOURCES.txt'\n",
+ " reading manifest template 'MANIFEST.in'\n",
+ " adding license file 'LICENSE'\n",
+ " writing manifest file 'mmaction2.egg-info/SOURCES.txt'\n",
+ " running build_ext\n",
+ " Creating /usr/local/lib/python3.10/dist-packages/mmaction2.egg-link (link to .)\n",
+ " Adding mmaction2 1.0.0 to easy-install.pth file\n",
+ "\n",
+ " Installed /content/mmaction2\n",
+ "Successfully installed decord-0.6.0 einops-0.6.1 mmaction2-1.0.0\n",
+ "/content/mmaction2/projects/stad_tutorial\n"
+ ]
+ }
+ ],
+ "source": [
+ "%pip install -U openmim\n",
+ "!mim install mmengine\n",
+ "!mim install mmcv\n",
+ "!mim install mmdet\n",
+ "\n",
+ "!git clone https://github.com/open-mmlab/mmaction2.git\n",
+ "\n",
+ "%cd mmaction2\n",
+ "%pip install -v -e .\n",
+ "%cd projects/stad_tutorial"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4M1PQASJxRAM"
+ },
+ "source": [
+ "## 1. 准备时空行为检测数据集\n",
+ "\n",
+ "类似于检测任务需要提供检测框标注，时空行为检测任务需要对时间和空间同时定位，所以需要提供更复杂的 tube 标注。以 MultiSports 数据集的标注为例，`gttubes` 字段提供了视频中所有的目标动作标注，以下为一个标注片段：\n",
+ "\n",
+ "```\n",
+ " 'gttubes': {\n",
+ " 'aerobic_gymnastics/v_aqMgwPExjD0_c001': # video_key\n",
+ " {\n",
+ " 10: # 类别标号\n",
+ " [\n",
+ " array([[ 377., 904., 316., 1016., 584.], # 类别 10 的第 1 个 tube,\n",
+ " [ 378., 882., 315., 1016., 579.], # shape (n, 5): 表示 n 帧，每帧标注中包括 (帧号，x1，y1, x2, y2)\n",
+ " ...\n",
+ " [ 398., 861., 304., 954., 549.]], dtype=float32)，\n",
+ "\n",
+ " array([[ 399., 881., 308., 955., 542.], # 类别 10 的第 2 个 tube\n",
+ " [ 400., 862., 303., 988., 539.],\n",
+ " [ 401., 853., 292., 1000., 535.],\n",
+ " ...])\n",
+ " ...\n",
+ "\n",
+ " ] ,\n",
+ " 9: # 类别标号\n",
+ " [\n",
+ " array(...), # 类别 9 的第 1 个 tube\n",
+ " array(...), # 类别 9 的第 2 个 tube\n",
+ " ...\n",
+ " ]\n",
+ " ...\n",
+ " }\n",
+ " }\n",
+ "```\n",
+ "\n",
+ "标注文件中还需要提供其他字段的信息，完整的真值文件包括以下信息：\n",
+ "```\n",
+ "{\n",
+ " 'labels': # 标签列表\n",
+ " ['aerobic push up', 'aerobic explosive push up', ...],\n",
+ " 'train_videos': # 训练视频列表\n",
+ " [\n",
+ " [\n",
+ " 'aerobic_gymnastics/v_aqMgwPExjD0_c001',\n",
+ " 'aerobic_gymnastics/v_yaKOumdXwbU_c019',\n",
+ " ...\n",
+ " ]\n",
+ " ]\n",
+ " 'test_videos': # 测试视频列表\n",
+ " [\n",
+ " [\n",
+ " 'aerobic_gymnastics/v_crsi07chcV8_c004',\n",
+ " 'aerobic_gymnastics/v_dFYr67eNMwA_c005',\n",
+ " ...\n",
+ " ]\n",
+ " ]\n",
+ " 'n_frames': # dict 文件，提供各个视频的帧数信息\n",
+ " {\n",
+ " 'aerobic_gymnastics/v_crsi07chcV8_c004': 725,\n",
+ " 'aerobic_gymnastics/v_dFYr67eNMwA_c005': 750,\n",
+ " ...\n",
+ " }\n",
+ " 'resolution': # dict 文件，提供各个视频的分辨率信息\n",
+ " {\n",
+ " 'aerobic_gymnastics/v_crsi07chcV8_c004': (720, 1280),\n",
+ " 'aerobic_gymnastics/v_dFYr67eNMwA_c005': (720, 1280),\n",
+ " ...\n",
+ " }\n",
+ " 'gt_tubes': # dict 文件，提供 tube 的检测框信息\n",
+ " {\n",
+ " ... # 格式参考上述说明\n",
+ " }\n",
+ "}\n",
+ "```\n",
+ "后续的实验基于 MultiSports-tiny 进行，我们从 MultiSports 中抽取了少量视频，用于演示整个流程。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "fiJPDuR9xRAQ",
+ "outputId": "8b3d8719-a9c0-4a59-d220-a3626fa34d3b"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "--2023-06-15 06:41:29-- https://download.openmmlab.com/mmaction/v1.0/projects/stad_tutorial/multisports-tiny.tar\n",
+ "Resolving download.openmmlab.com (download.openmmlab.com)... 8.48.85.214, 8.48.85.207, 8.48.85.208, ...\n",
+ "Connecting to download.openmmlab.com (download.openmmlab.com)|8.48.85.214|:443... connected.\n",
+ "HTTP request sent, awaiting response... 200 OK\n",
+ "Length: 82780160 (79M) [application/x-tar]\n",
+ "Saving to: ‘data/multisports-tiny.tar’\n",
+ "\n",
+ "multisports-tiny.ta 100%[===================>] 78.95M 27.9MB/s in 2.8s \n",
+ "\n",
+ "2023-06-15 06:41:32 (27.9 MB/s) - ‘data/multisports-tiny.tar’ saved [82780160/82780160]\n",
+ "\n",
+ "multisports-tiny/multisports/\n",
+ "multisports-tiny/multisports/test/\n",
+ "multisports-tiny/multisports/test/aerobic_gymnastics/\n",
+ "multisports-tiny/multisports/test/aerobic_gymnastics/v_7G_IpU0FxLU_c001.mp4\n",
+ "multisports-tiny/multisports/annotations/\n",
+ "multisports-tiny/multisports/annotations/multisports_GT.pkl\n",
+ "multisports-tiny/multisports/trainval/\n",
+ "multisports-tiny/multisports/trainval/aerobic_gymnastics/\n",
+ "multisports-tiny/multisports/trainval/aerobic_gymnastics/v__wAgwttPYaQ_c001.mp4\n",
+ "multisports-tiny/multisports/trainval/aerobic_gymnastics/v__wAgwttPYaQ_c003.mp4\n",
+ "multisports-tiny/multisports/trainval/aerobic_gymnastics/v__wAgwttPYaQ_c002.mp4\n",
+ "Reading package lists...\n",
+ "Building dependency tree...\n",
+ "Reading state information...\n",
+ "The following NEW packages will be installed:\n",
+ " tree\n",
+ "0 upgraded, 1 newly installed, 0 to remove and 46 not upgraded.\n",
+ "Need to get 43.0 kB of archives.\n",
+ "After this operation, 115 kB of additional disk space will be used.\n",
+ "Get:1 http://archive.ubuntu.com/ubuntu focal/universe amd64 tree amd64 1.8.0-1 [43.0 kB]\n",
+ "Fetched 43.0 kB in 0s (253 kB/s)\n",
+ "Selecting previously unselected package tree.\n",
+ "(Reading database ... 122541 files and directories currently installed.)\n",
+ "Preparing to unpack .../tree_1.8.0-1_amd64.deb ...\n",
+ "Unpacking tree (1.8.0-1) ...\n",
+ "Setting up tree (1.8.0-1) ...\n",
+ "Processing triggers for man-db (2.9.1-1) ...\n",
+ "\u001b[01;34mdata\u001b[00m\n",
+ "├── \u001b[01;34mmultisports\u001b[00m\n",
+ "│ ├── \u001b[01;34mannotations\u001b[00m\n",
+ "│ │ └── \u001b[01;32mmultisports_GT.pkl\u001b[00m\n",
+ "│ ├── \u001b[01;34mtest\u001b[00m\n",
+ "│ │ └── \u001b[01;34maerobic_gymnastics\u001b[00m\n",
+ "│ │ └── \u001b[01;32mv_7G_IpU0FxLU_c001.mp4\u001b[00m\n",
+ "│ └── \u001b[01;34mtrainval\u001b[00m\n",
+ "│ └── \u001b[01;34maerobic_gymnastics\u001b[00m\n",
+ "│ ├── \u001b[01;32mv__wAgwttPYaQ_c001.mp4\u001b[00m\n",
+ "│ ├── \u001b[01;32mv__wAgwttPYaQ_c002.mp4\u001b[00m\n",
+ "│ └── \u001b[01;32mv__wAgwttPYaQ_c003.mp4\u001b[00m\n",
+ "└── \u001b[01;31mmultisports-tiny.tar\u001b[00m\n",
+ "\n",
+ "6 directories, 6 files\n"
+ ]
+ }
+ ],
+ "source": [
+ "# 下载数据集\n",
+ "!wget -P data -c https://download.openmmlab.com/mmaction/v1.0/projects/stad_tutorial/multisports-tiny.tar\n",
+ "!tar -xvf data/multisports-tiny.tar --strip 1 -C data\n",
+ "!apt-get -q install tree\n",
+ "!tree data"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "XjG0dEE8xRAS"
+ },
+ "source": [
+ "## 2. 训练检测模型\n",
+ "\n",
+ "在 SlowOnly + Det 的范式中，需要先训练人体检测器，再基于检测结果来预测行为。这一节中，我们基于上一节中的标注格式和 MMDetection 算法库训练检测模型。\n",
+ "\n",
+ "### 2.1 构建检测数据集标注（COCO 格式）\n",
+ "\n",
+ "基于时空行为检测数据集的标注信息，我们可以构建一个 COCO 格式的检测数据集，用于训练检测模型。我们提供了一个工具脚本对 MultiSports 格式的标注进行转换，如果需要基于其他格式转换，可以参考 MMDetection 提供的[自定义数据集](https://mmdetection.readthedocs.io/zh_CN/latest/advanced_guides/customize_dataset.html)文档。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "inBtClMIxRAV",
+ "outputId": "3ac5199b-562f-48c4-da27-819d34069213"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\u001b[01;34mdata/multisports/annotations\u001b[00m\n",
+ "├── multisports_det_anno_train.json\n",
+ "├── multisports_det_anno_val.json\n",
+ "└── \u001b[01;32mmultisports_GT.pkl\u001b[00m\n",
+ "\n",
+ "0 directories, 3 files\n"
+ ]
+ }
+ ],
+ "source": [
+ "!python tools/generate_mmdet_anno.py data/multisports/annotations/multisports_GT.pkl data/multisports/annotations/multisports_det_anno.json\n",
+ "!tree data/multisports/annotations"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "TkPONRezxRAZ",
+ "outputId": "0f8075a1-47fb-490d-9c88-4904f45363fb"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Will generate 3 rgb dir for aerobic_gymnastics.\n",
+ "Generate v__wAgwttPYaQ_c003 rgb dir successfully.\n",
+ "Generate v__wAgwttPYaQ_c002 rgb dir successfully.\n",
+ "Generate v__wAgwttPYaQ_c001 rgb dir successfully.\n"
+ ]
+ }
+ ],
+ "source": [
+ "!python tools/generate_rgb.py"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "MP-umqqnxRAa"
+ },
+ "source": [
+ "### 2.2 修改 config 文件\n",
+ "\n",
+ "我们以 faster-rcnn_x101-64x4d_fpn_1x_coco 为基础配置，做如下修改，在 MultiSports 数据集上进行训练。需要修改以下几个部分：\n",
+ "- 模型的类别数量\n",
+ "- 学习率调整策略\n",
+ "- 优化器配置\n",
+ "- 数据集/标注文件路径\n",
+ "- 评测器配置\n",
+ "- 预训练模型\n",
+ "\n",
+ "更详细的教程可以参考 MMDetection 提供的[准备配置文件](https://mmdetection.readthedocs.io/zh_CN/latest/user_guides/train.html#id9)文档。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "yMw9MrI0xRAc",
+ "outputId": "1f5ee99a-d4cb-45b0-df71-f0209a9b6275"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "# Copyright (c) OpenMMLab. All rights reserved.\n",
+ "_base_ = './faster-rcnn_r50-caffe_fpn_ms-1x_coco.py'\n",
+ "model = dict(roi_head=dict(bbox_head=dict(num_classes=1)))\n",
+ "\n",
+ "# take 2 epochs as an example\n",
+ "train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=2, val_interval=1)\n",
+ "\n",
+ "# learning rate\n",
+ "param_scheduler = [\n",
+ " dict(type='ConstantLR', factor=1.0, by_epoch=False, begin=0, end=500)\n",
+ "]\n",
+ "\n",
+ "# optimizer\n",
+ "optim_wrapper = dict(\n",
+ " type='OptimWrapper',\n",
+ " optimizer=dict(type='SGD', lr=0.0050, momentum=0.9, weight_decay=0.0001))\n",
+ "\n",
+ "dataset_type = 'CocoDataset'\n",
+ "# modify metainfo\n",
+ "metainfo = {\n",
+ " 'classes': ('person', ),\n",
+ " 'palette': [\n",
+ " (220, 20, 60),\n",
+ " ]\n",
+ "}\n",
+ "\n",
+ "# specify metainfo, dataset path\n",
+ "data_root = 'data/multisports/'\n",
+ "\n",
+ "train_dataloader = dict(\n",
+ " dataset=dict(\n",
+ " data_root=data_root,\n",
+ " ann_file='annotations/multisports_det_anno_train.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " metainfo=metainfo))\n",
+ "\n",
+ "val_dataloader = dict(\n",
+ " dataset=dict(\n",
+ " data_root=data_root,\n",
+ " ann_file='annotations/multisports_det_anno_val.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " metainfo=metainfo))\n",
+ "\n",
+ "test_dataloader = dict(\n",
+ " dataset=dict(\n",
+ " data_root=data_root,\n",
+ " ann_file='annotations/ms_infer_anno.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " metainfo=metainfo))\n",
+ "\n",
+ "# specify annotaition file path, modify metric items\n",
+ "val_evaluator = dict(\n",
+ " ann_file='data/multisports/annotations/multisports_det_anno_val.json',\n",
+ " metric_items=['mAP_50', 'AR@100'],\n",
+ " iou_thrs=[0.5],\n",
+ ")\n",
+ "\n",
+ "test_evaluator = dict(\n",
+ " ann_file='data/multisports/annotations/ms_infer_anno.json',\n",
+ " metric_items=['mAP_50', 'AR@100'],\n",
+ " iou_thrs=[0.5],\n",
+ ")\n",
+ "\n",
+ "# specify pretrain checkpoint\n",
+ "load_from = 'https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929-d022e227.pth' # noqa: E501\n"
+ ]
+ }
+ ],
+ "source": [
+ "!cat configs/faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person.py"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "S3Ux8echxRAe"
+ },
+ "source": [
+ "### 2.3 训练检测模型"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "MYtjYFU5xRAf"
+ },
+ "source": [
+ "利用 MIM 可以在当前路径直接训练 MMDetection 模型，这里提供最简单的单卡训练示例，更多训练命令可以参考 MIM [教程](https://github.com/open-mmlab/mim#command)。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "56m--2T8xRAg",
+ "outputId": "d47ceca0-e930-4063-e25d-739a44410b86"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Training command is /usr/bin/python3 /usr/local/lib/python3.10/dist-packages/mmdet/.mim/tools/train.py configs/faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person.py --launcher none --work-dir work_dirs/det_model. \n",
+ "06/15 06:42:40 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - \n",
+ "------------------------------------------------------------\n",
+ "System environment:\n",
+ " sys.platform: linux\n",
+ " Python: 3.10.12 (main, Jun 7 2023, 12:45:35) [GCC 9.4.0]\n",
+ " CUDA available: True\n",
+ " numpy_random_seed: 1318688827\n",
+ " GPU 0: Tesla T4\n",
+ " CUDA_HOME: /usr/local/cuda\n",
+ " NVCC: Cuda compilation tools, release 11.8, V11.8.89\n",
+ " GCC: x86_64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0\n",
+ " PyTorch: 2.0.1+cu118\n",
+ " PyTorch compiling details: PyTorch built with:\n",
+ " - GCC 9.3\n",
+ " - C++ Version: 201703\n",
+ " - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications\n",
+ " - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n",
+ " - OpenMP 201511 (a.k.a. OpenMP 4.5)\n",
+ " - LAPACK is enabled (usually provided by MKL)\n",
+ " - NNPACK is enabled\n",
+ " - CPU capability usage: AVX2\n",
+ " - CUDA Runtime 11.8\n",
+ " - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n",
+ " - CuDNN 8.7\n",
+ " - Magma 2.6.1\n",
+ " - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n",
+ "\n",
+ " TorchVision: 0.15.2+cu118\n",
+ " OpenCV: 4.7.0\n",
+ " MMEngine: 0.7.4\n",
+ "\n",
+ "Runtime environment:\n",
+ " cudnn_benchmark: False\n",
+ " mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}\n",
+ " dist_cfg: {'backend': 'nccl'}\n",
+ " seed: 1318688827\n",
+ " Distributed launcher: none\n",
+ " Distributed training: False\n",
+ " GPU number: 1\n",
+ "------------------------------------------------------------\n",
+ "\n",
+ "06/15 06:42:40 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Config:\n",
+ "model = dict(\n",
+ " type='FasterRCNN',\n",
+ " data_preprocessor=dict(\n",
+ " type='DetDataPreprocessor',\n",
+ " mean=[103.53, 116.28, 123.675],\n",
+ " std=[1.0, 1.0, 1.0],\n",
+ " bgr_to_rgb=False,\n",
+ " pad_size_divisor=32),\n",
+ " backbone=dict(\n",
+ " type='ResNet',\n",
+ " depth=50,\n",
+ " num_stages=4,\n",
+ " out_indices=(0, 1, 2, 3),\n",
+ " frozen_stages=1,\n",
+ " norm_cfg=dict(type='BN', requires_grad=False),\n",
+ " norm_eval=True,\n",
+ " style='caffe',\n",
+ " init_cfg=dict(\n",
+ " type='Pretrained',\n",
+ " checkpoint='open-mmlab://detectron2/resnet50_caffe')),\n",
+ " neck=dict(\n",
+ " type='FPN',\n",
+ " in_channels=[256, 512, 1024, 2048],\n",
+ " out_channels=256,\n",
+ " num_outs=5),\n",
+ " rpn_head=dict(\n",
+ " type='RPNHead',\n",
+ " in_channels=256,\n",
+ " feat_channels=256,\n",
+ " anchor_generator=dict(\n",
+ " type='AnchorGenerator',\n",
+ " scales=[8],\n",
+ " ratios=[0.5, 1.0, 2.0],\n",
+ " strides=[4, 8, 16, 32, 64]),\n",
+ " bbox_coder=dict(\n",
+ " type='DeltaXYWHBBoxCoder',\n",
+ " target_means=[0.0, 0.0, 0.0, 0.0],\n",
+ " target_stds=[1.0, 1.0, 1.0, 1.0]),\n",
+ " loss_cls=dict(\n",
+ " type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),\n",
+ " loss_bbox=dict(type='L1Loss', loss_weight=1.0)),\n",
+ " roi_head=dict(\n",
+ " type='StandardRoIHead',\n",
+ " bbox_roi_extractor=dict(\n",
+ " type='SingleRoIExtractor',\n",
+ " roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0),\n",
+ " out_channels=256,\n",
+ " featmap_strides=[4, 8, 16, 32]),\n",
+ " bbox_head=dict(\n",
+ " type='Shared2FCBBoxHead',\n",
+ " in_channels=256,\n",
+ " fc_out_channels=1024,\n",
+ " roi_feat_size=7,\n",
+ " num_classes=1,\n",
+ " bbox_coder=dict(\n",
+ " type='DeltaXYWHBBoxCoder',\n",
+ " target_means=[0.0, 0.0, 0.0, 0.0],\n",
+ " target_stds=[0.1, 0.1, 0.2, 0.2]),\n",
+ " reg_class_agnostic=False,\n",
+ " loss_cls=dict(\n",
+ " type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),\n",
+ " loss_bbox=dict(type='L1Loss', loss_weight=1.0))),\n",
+ " train_cfg=dict(\n",
+ " rpn=dict(\n",
+ " assigner=dict(\n",
+ " type='MaxIoUAssigner',\n",
+ " pos_iou_thr=0.7,\n",
+ " neg_iou_thr=0.3,\n",
+ " min_pos_iou=0.3,\n",
+ " match_low_quality=True,\n",
+ " ignore_iof_thr=-1),\n",
+ " sampler=dict(\n",
+ " type='RandomSampler',\n",
+ " num=256,\n",
+ " pos_fraction=0.5,\n",
+ " neg_pos_ub=-1,\n",
+ " add_gt_as_proposals=False),\n",
+ " allowed_border=-1,\n",
+ " pos_weight=-1,\n",
+ " debug=False),\n",
+ " rpn_proposal=dict(\n",
+ " nms_pre=2000,\n",
+ " max_per_img=1000,\n",
+ " nms=dict(type='nms', iou_threshold=0.7),\n",
+ " min_bbox_size=0),\n",
+ " rcnn=dict(\n",
+ " assigner=dict(\n",
+ " type='MaxIoUAssigner',\n",
+ " pos_iou_thr=0.5,\n",
+ " neg_iou_thr=0.5,\n",
+ " min_pos_iou=0.5,\n",
+ " match_low_quality=False,\n",
+ " ignore_iof_thr=-1),\n",
+ " sampler=dict(\n",
+ " type='RandomSampler',\n",
+ " num=512,\n",
+ " pos_fraction=0.25,\n",
+ " neg_pos_ub=-1,\n",
+ " add_gt_as_proposals=True),\n",
+ " pos_weight=-1,\n",
+ " debug=False)),\n",
+ " test_cfg=dict(\n",
+ " rpn=dict(\n",
+ " nms_pre=1000,\n",
+ " max_per_img=1000,\n",
+ " nms=dict(type='nms', iou_threshold=0.7),\n",
+ " min_bbox_size=0),\n",
+ " rcnn=dict(\n",
+ " score_thr=0.05,\n",
+ " nms=dict(type='nms', iou_threshold=0.5),\n",
+ " max_per_img=100)))\n",
+ "dataset_type = 'CocoDataset'\n",
+ "data_root = 'data/multisports/'\n",
+ "backend_args = None\n",
+ "train_pipeline = [\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='RandomChoiceResize',\n",
+ " scales=[(1333, 640), (1333, 672), (1333, 704), (1333, 736),\n",
+ " (1333, 768), (1333, 800)],\n",
+ " keep_ratio=True),\n",
+ " dict(type='RandomFlip', prob=0.5),\n",
+ " dict(type='PackDetInputs')\n",
+ "]\n",
+ "test_pipeline = [\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='Resize', scale=(1333, 800), keep_ratio=True),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='PackDetInputs',\n",
+ " meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',\n",
+ " 'scale_factor'))\n",
+ "]\n",
+ "train_dataloader = dict(\n",
+ " batch_size=2,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=True),\n",
+ " batch_sampler=dict(type='AspectRatioBatchSampler'),\n",
+ " dataset=dict(\n",
+ " type='CocoDataset',\n",
+ " data_root='data/multisports/',\n",
+ " ann_file='annotations/multisports_det_anno_train.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " filter_cfg=dict(filter_empty_gt=True, min_size=32),\n",
+ " pipeline=[\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='RandomChoiceResize',\n",
+ " scales=[(1333, 640), (1333, 672), (1333, 704), (1333, 736),\n",
+ " (1333, 768), (1333, 800)],\n",
+ " keep_ratio=True),\n",
+ " dict(type='RandomFlip', prob=0.5),\n",
+ " dict(type='PackDetInputs')\n",
+ " ],\n",
+ " backend_args=None,\n",
+ " metainfo=dict(classes=('person', ), palette=[(220, 20, 60)])))\n",
+ "val_dataloader = dict(\n",
+ " batch_size=1,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " drop_last=False,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=False),\n",
+ " dataset=dict(\n",
+ " type='CocoDataset',\n",
+ " data_root='data/multisports/',\n",
+ " ann_file='annotations/multisports_det_anno_val.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " test_mode=True,\n",
+ " pipeline=[\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='Resize', scale=(1333, 800), keep_ratio=True),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='PackDetInputs',\n",
+ " meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',\n",
+ " 'scale_factor'))\n",
+ " ],\n",
+ " backend_args=None,\n",
+ " metainfo=dict(classes=('person', ), palette=[(220, 20, 60)])))\n",
+ "test_dataloader = dict(\n",
+ " batch_size=1,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " drop_last=False,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=False),\n",
+ " dataset=dict(\n",
+ " type='CocoDataset',\n",
+ " data_root='data/multisports/',\n",
+ " ann_file='annotations/ms_infer_anno.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " test_mode=True,\n",
+ " pipeline=[\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='Resize', scale=(1333, 800), keep_ratio=True),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='PackDetInputs',\n",
+ " meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',\n",
+ " 'scale_factor'))\n",
+ " ],\n",
+ " backend_args=None,\n",
+ " metainfo=dict(classes=('person', ), palette=[(220, 20, 60)])))\n",
+ "val_evaluator = dict(\n",
+ " type='CocoMetric',\n",
+ " ann_file='data/multisports/annotations/multisports_det_anno_val.json',\n",
+ " metric='bbox',\n",
+ " format_only=False,\n",
+ " backend_args=None,\n",
+ " metric_items=['mAP_50', 'AR@100'],\n",
+ " iou_thrs=[0.5])\n",
+ "test_evaluator = dict(\n",
+ " type='CocoMetric',\n",
+ " ann_file='data/multisports/annotations/ms_infer_anno.json',\n",
+ " metric='bbox',\n",
+ " format_only=False,\n",
+ " backend_args=None,\n",
+ " metric_items=['mAP_50', 'AR@100'],\n",
+ " iou_thrs=[0.5])\n",
+ "train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=2, val_interval=1)\n",
+ "val_cfg = dict(type='ValLoop')\n",
+ "test_cfg = dict(type='TestLoop')\n",
+ "param_scheduler = [\n",
+ " dict(type='ConstantLR', factor=1.0, by_epoch=False, begin=0, end=500)\n",
+ "]\n",
+ "optim_wrapper = dict(\n",
+ " type='OptimWrapper',\n",
+ " optimizer=dict(type='SGD', lr=0.005, momentum=0.9, weight_decay=0.0001))\n",
+ "auto_scale_lr = dict(enable=False, base_batch_size=16)\n",
+ "default_scope = 'mmdet'\n",
+ "default_hooks = dict(\n",
+ " timer=dict(type='IterTimerHook'),\n",
+ " logger=dict(type='LoggerHook', interval=50),\n",
+ " param_scheduler=dict(type='ParamSchedulerHook'),\n",
+ " checkpoint=dict(type='CheckpointHook', interval=1),\n",
+ " sampler_seed=dict(type='DistSamplerSeedHook'),\n",
+ " visualization=dict(type='DetVisualizationHook'))\n",
+ "env_cfg = dict(\n",
+ " cudnn_benchmark=False,\n",
+ " mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n",
+ " dist_cfg=dict(backend='nccl'))\n",
+ "vis_backends = [dict(type='LocalVisBackend')]\n",
+ "visualizer = dict(\n",
+ " type='DetLocalVisualizer',\n",
+ " vis_backends=[dict(type='LocalVisBackend')],\n",
+ " name='visualizer')\n",
+ "log_processor = dict(type='LogProcessor', window_size=50, by_epoch=True)\n",
+ "log_level = 'INFO'\n",
+ "load_from = 'https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929-d022e227.pth'\n",
+ "resume = False\n",
+ "metainfo = dict(classes=('person', ), palette=[(220, 20, 60)])\n",
+ "launcher = 'none'\n",
+ "work_dir = 'work_dirs/det_model'\n",
+ "\n",
+ "06/15 06:42:49 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.\n",
+ "06/15 06:42:49 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Hooks will be executed in the following order:\n",
+ "before_run:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "before_train:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_train_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) DistSamplerSeedHook \n",
+ " -------------------- \n",
+ "before_train_iter:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_train_iter:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "after_train_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_val_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "before_val_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_val_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) DetVisualizationHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_val_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "after_train:\n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_test_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "before_test_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_test_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) DetVisualizationHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_test_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_run:\n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "loading annotations into memory...\n",
+ "Done (t=0.00s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "loading annotations into memory...\n",
+ "Done (t=0.00s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "loading annotations into memory...\n",
+ "Done (t=0.00s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "06/15 06:42:50 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - load model from: open-mmlab://detectron2/resnet50_caffe\n",
+ "06/15 06:42:50 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Loads checkpoint by openmmlab backend from path: open-mmlab://detectron2/resnet50_caffe\n",
+ "Downloading: \"https://download.openmmlab.com/pretrain/third_party/resnet50_msra-5891d200.pth\" to /root/.cache/torch/hub/checkpoints/resnet50_msra-5891d200.pth\n",
+ "100% 89.9M/89.9M [00:03<00:00, 31.4MB/s]\n",
+ "06/15 06:42:53 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - The model and loaded state dict do not match exactly\n",
+ "\n",
+ "unexpected key in source state_dict: conv1.bias\n",
+ "\n",
+ "Loads checkpoint by http backend from path: https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929-d022e227.pth\n",
+ "Downloading: \"https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929-d022e227.pth\" to /root/.cache/torch/hub/checkpoints/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929-d022e227.pth\n",
+ "100% 158M/158M [00:06<00:00, 24.4MB/s]\n",
+ "06/15 06:43:00 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Load checkpoint from https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco-person/faster_rcnn_r50_fpn_1x_coco-person_20201216_175929-d022e227.pth\n",
+ "06/15 06:43:00 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - \"FileClient\" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io\n",
+ "06/15 06:43:00 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - \"HardDiskBackend\" is the alias of \"LocalBackend\" and the former will be deprecated in future.\n",
+ "06/15 06:43:00 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Checkpoints will be saved to /content/mmaction2/projects/stad_tutorial/work_dirs/det_model.\n",
+ "06/15 06:43:33 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][ 50/118] lr: 5.0000e-03 eta: 0:02:00 time: 0.6468 data_time: 0.0127 memory: 3419 loss: 0.4823 loss_rpn_cls: 0.0063 loss_rpn_bbox: 0.0151 loss_cls: 0.1676 acc: 95.0195 loss_bbox: 0.2933\n",
+ "06/15 06:43:57 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][100/118] lr: 5.0000e-03 eta: 0:01:17 time: 0.4922 data_time: 0.0077 memory: 3419 loss: 0.4234 loss_rpn_cls: 0.0031 loss_rpn_bbox: 0.0134 loss_cls: 0.1394 acc: 91.9922 loss_bbox: 0.2676\n",
+ "06/15 06:44:07 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person_20230615_064239\n",
+ "06/15 06:44:07 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 1 epochs\n",
+ "06/15 06:44:15 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][ 50/120] eta: 0:00:08 time: 0.1269 data_time: 0.0112 memory: 3419 \n",
+ "06/15 06:44:21 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][100/120] eta: 0:00:02 time: 0.1159 data_time: 0.0032 memory: 682 \n",
+ "06/15 06:44:23 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Evaluating bbox...\n",
+ "Loading and preparing results...\n",
+ "DONE (t=0.00s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "Running per image evaluation...\n",
+ "Evaluate annotation type *bbox*\n",
+ "DONE (t=0.04s).\n",
+ "Accumulating evaluation results...\n",
+ "DONE (t=0.01s).\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 0.913\n",
+ " Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= small | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area=medium | maxDets=1000 ] = 0.817\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= large | maxDets=1000 ] = 0.908\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 0.960\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=300 ] = 0.960\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=1000 ] = 0.960\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= small | maxDets=1000 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area=medium | maxDets=1000 ] = 1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= large | maxDets=1000 ] = 0.960\n",
+ "06/15 06:44:23 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - bbox_mAP_copypaste: 0.913 -1.000 -1.000 -1.000 0.817 0.908\n",
+ "06/15 06:44:23 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][120/120] coco/bbox_mAP_50: -1.0000 coco/bbox_AR@100: 0.9600 data_time: 0.0065 time: 0.1205\n",
+ "06/15 06:44:49 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][ 50/118] lr: 5.0000e-03 eta: 0:00:37 time: 0.5233 data_time: 0.0099 memory: 3419 loss: 0.3250 loss_rpn_cls: 0.0025 loss_rpn_bbox: 0.0107 loss_cls: 0.1116 acc: 95.2148 loss_bbox: 0.2002\n",
+ "06/15 06:45:16 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][100/118] lr: 5.0000e-03 eta: 0:00:09 time: 0.5354 data_time: 0.0083 memory: 3419 loss: 0.3042 loss_rpn_cls: 0.0013 loss_rpn_bbox: 0.0105 loss_cls: 0.0946 acc: 94.9219 loss_bbox: 0.1978\n",
+ "06/15 06:45:26 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person_20230615_064239\n",
+ "06/15 06:45:26 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 2 epochs\n",
+ "06/15 06:45:34 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][ 50/120] eta: 0:00:08 time: 0.1237 data_time: 0.0050 memory: 3419 \n",
+ "06/15 06:45:40 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][100/120] eta: 0:00:02 time: 0.1225 data_time: 0.0058 memory: 682 \n",
+ "06/15 06:45:43 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Evaluating bbox...\n",
+ "Loading and preparing results...\n",
+ "DONE (t=0.00s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "Running per image evaluation...\n",
+ "Evaluate annotation type *bbox*\n",
+ "DONE (t=0.07s).\n",
+ "Accumulating evaluation results...\n",
+ "DONE (t=0.01s).\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 0.912\n",
+ " Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= small | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area=medium | maxDets=1000 ] = 0.747\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= large | maxDets=1000 ] = 0.916\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = 0.955\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=300 ] = 0.955\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=1000 ] = 0.955\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= small | maxDets=1000 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area=medium | maxDets=1000 ] = 1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= large | maxDets=1000 ] = 0.954\n",
+ "06/15 06:45:43 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - bbox_mAP_copypaste: 0.912 -1.000 -1.000 -1.000 0.747 0.916\n",
+ "06/15 06:45:43 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][120/120] coco/bbox_mAP_50: -1.0000 coco/bbox_AR@100: 0.9550 data_time: 0.0052 time: 0.1228\n",
+ "\u001b[32mTraining finished successfully. \u001b[0m\n"
+ ]
+ }
+ ],
+ "source": [
+ "!mim train mmdet configs/faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person.py \\\n",
+ " --work-dir work_dirs/det_model"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "-pf9MnuUxRAh"
+ },
+ "source": [
+ "### 2.4 生成 proposal bboxes\n",
+ "\n",
+ "在时空行为检测模型训练时，需要基于检测模型推理得到的 proposal，而不能基于标注的检测框。因此我们需要利用训练好的检测模型对整个数据集进行推理，得到 proposal 后转换为需要的格式，用于后续训练。\n",
+ "\n",
+ "#### 2.4.1 将数据集转换为 Coco 格式\n",
+ "\n",
+ "我们提供了脚本将 MultiSports 数据集转换成没有真值的标注格式，用于推理。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "nL2n0AKJxRAi",
+ "outputId": "51907af1-7da3-4713-8e90-a61b052000aa"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[>>] 2350/2350, 1606.7 task/s, elapsed: 1s, ETA: 0s\n",
+ "save json file: data/multisports/rawframes/../annotations/ms_infer_anno.json\n"
+ ]
+ }
+ ],
+ "source": [
+ "!echo 'person' > data/multisports/annotations/label_map.txt\n",
+ "!python tools/images2coco.py \\\n",
+ " data/multisports/rawframes \\\n",
+ " data/multisports/annotations/label_map.txt \\\n",
+ " ms_infer_anno.json"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_REQniysxRAj"
+ },
+ "source": [
+ "#### 2.4.2 推理生成 proposal file"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ShnTsjs1xRAk"
+ },
+ "source": [
+ "MMDetection 模型的推理同样基于 MIM，更多测试命令请参考 MIM [教程](https://github.com/open-mmlab/mim#command)。\n",
+ "\n",
+ "推理完成后，会将推理结果保存在 'data/multisports/ms_proposals.pkl'。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "DXnT4aArxRAm",
+ "outputId": "565faf02-4b7f-49ab-f30f-b20e7eb09085"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Testing command is /usr/bin/python3 /usr/local/lib/python3.10/dist-packages/mmdet/.mim/tools/test.py configs/faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person.py work_dirs/det_model/epoch_2.pth --launcher none --out data/multisports/annotations/ms_det_proposals.pkl. \n",
+ "06/15 06:45:52 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - \n",
+ "------------------------------------------------------------\n",
+ "System environment:\n",
+ " sys.platform: linux\n",
+ " Python: 3.10.12 (main, Jun 7 2023, 12:45:35) [GCC 9.4.0]\n",
+ " CUDA available: True\n",
+ " numpy_random_seed: 1403639615\n",
+ " GPU 0: Tesla T4\n",
+ " CUDA_HOME: /usr/local/cuda\n",
+ " NVCC: Cuda compilation tools, release 11.8, V11.8.89\n",
+ " GCC: x86_64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0\n",
+ " PyTorch: 2.0.1+cu118\n",
+ " PyTorch compiling details: PyTorch built with:\n",
+ " - GCC 9.3\n",
+ " - C++ Version: 201703\n",
+ " - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications\n",
+ " - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n",
+ " - OpenMP 201511 (a.k.a. OpenMP 4.5)\n",
+ " - LAPACK is enabled (usually provided by MKL)\n",
+ " - NNPACK is enabled\n",
+ " - CPU capability usage: AVX2\n",
+ " - CUDA Runtime 11.8\n",
+ " - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n",
+ " - CuDNN 8.7\n",
+ " - Magma 2.6.1\n",
+ " - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n",
+ "\n",
+ " TorchVision: 0.15.2+cu118\n",
+ " OpenCV: 4.7.0\n",
+ " MMEngine: 0.7.4\n",
+ "\n",
+ "Runtime environment:\n",
+ " cudnn_benchmark: False\n",
+ " mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}\n",
+ " dist_cfg: {'backend': 'nccl'}\n",
+ " seed: 1403639615\n",
+ " Distributed launcher: none\n",
+ " Distributed training: False\n",
+ " GPU number: 1\n",
+ "------------------------------------------------------------\n",
+ "\n",
+ "06/15 06:45:52 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Config:\n",
+ "model = dict(\n",
+ " type='FasterRCNN',\n",
+ " data_preprocessor=dict(\n",
+ " type='DetDataPreprocessor',\n",
+ " mean=[103.53, 116.28, 123.675],\n",
+ " std=[1.0, 1.0, 1.0],\n",
+ " bgr_to_rgb=False,\n",
+ " pad_size_divisor=32),\n",
+ " backbone=dict(\n",
+ " type='ResNet',\n",
+ " depth=50,\n",
+ " num_stages=4,\n",
+ " out_indices=(0, 1, 2, 3),\n",
+ " frozen_stages=1,\n",
+ " norm_cfg=dict(type='BN', requires_grad=False),\n",
+ " norm_eval=True,\n",
+ " style='caffe',\n",
+ " init_cfg=dict(\n",
+ " type='Pretrained',\n",
+ " checkpoint='open-mmlab://detectron2/resnet50_caffe')),\n",
+ " neck=dict(\n",
+ " type='FPN',\n",
+ " in_channels=[256, 512, 1024, 2048],\n",
+ " out_channels=256,\n",
+ " num_outs=5),\n",
+ " rpn_head=dict(\n",
+ " type='RPNHead',\n",
+ " in_channels=256,\n",
+ " feat_channels=256,\n",
+ " anchor_generator=dict(\n",
+ " type='AnchorGenerator',\n",
+ " scales=[8],\n",
+ " ratios=[0.5, 1.0, 2.0],\n",
+ " strides=[4, 8, 16, 32, 64]),\n",
+ " bbox_coder=dict(\n",
+ " type='DeltaXYWHBBoxCoder',\n",
+ " target_means=[0.0, 0.0, 0.0, 0.0],\n",
+ " target_stds=[1.0, 1.0, 1.0, 1.0]),\n",
+ " loss_cls=dict(\n",
+ " type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),\n",
+ " loss_bbox=dict(type='L1Loss', loss_weight=1.0)),\n",
+ " roi_head=dict(\n",
+ " type='StandardRoIHead',\n",
+ " bbox_roi_extractor=dict(\n",
+ " type='SingleRoIExtractor',\n",
+ " roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0),\n",
+ " out_channels=256,\n",
+ " featmap_strides=[4, 8, 16, 32]),\n",
+ " bbox_head=dict(\n",
+ " type='Shared2FCBBoxHead',\n",
+ " in_channels=256,\n",
+ " fc_out_channels=1024,\n",
+ " roi_feat_size=7,\n",
+ " num_classes=1,\n",
+ " bbox_coder=dict(\n",
+ " type='DeltaXYWHBBoxCoder',\n",
+ " target_means=[0.0, 0.0, 0.0, 0.0],\n",
+ " target_stds=[0.1, 0.1, 0.2, 0.2]),\n",
+ " reg_class_agnostic=False,\n",
+ " loss_cls=dict(\n",
+ " type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),\n",
+ " loss_bbox=dict(type='L1Loss', loss_weight=1.0))),\n",
+ " train_cfg=dict(\n",
+ " rpn=dict(\n",
+ " assigner=dict(\n",
+ " type='MaxIoUAssigner',\n",
+ " pos_iou_thr=0.7,\n",
+ " neg_iou_thr=0.3,\n",
+ " min_pos_iou=0.3,\n",
+ " match_low_quality=True,\n",
+ " ignore_iof_thr=-1),\n",
+ " sampler=dict(\n",
+ " type='RandomSampler',\n",
+ " num=256,\n",
+ " pos_fraction=0.5,\n",
+ " neg_pos_ub=-1,\n",
+ " add_gt_as_proposals=False),\n",
+ " allowed_border=-1,\n",
+ " pos_weight=-1,\n",
+ " debug=False),\n",
+ " rpn_proposal=dict(\n",
+ " nms_pre=2000,\n",
+ " max_per_img=1000,\n",
+ " nms=dict(type='nms', iou_threshold=0.7),\n",
+ " min_bbox_size=0),\n",
+ " rcnn=dict(\n",
+ " assigner=dict(\n",
+ " type='MaxIoUAssigner',\n",
+ " pos_iou_thr=0.5,\n",
+ " neg_iou_thr=0.5,\n",
+ " min_pos_iou=0.5,\n",
+ " match_low_quality=False,\n",
+ " ignore_iof_thr=-1),\n",
+ " sampler=dict(\n",
+ " type='RandomSampler',\n",
+ " num=512,\n",
+ " pos_fraction=0.25,\n",
+ " neg_pos_ub=-1,\n",
+ " add_gt_as_proposals=True),\n",
+ " pos_weight=-1,\n",
+ " debug=False)),\n",
+ " test_cfg=dict(\n",
+ " rpn=dict(\n",
+ " nms_pre=1000,\n",
+ " max_per_img=1000,\n",
+ " nms=dict(type='nms', iou_threshold=0.7),\n",
+ " min_bbox_size=0),\n",
+ " rcnn=dict(\n",
+ " score_thr=0.05,\n",
+ " nms=dict(type='nms', iou_threshold=0.5),\n",
+ " max_per_img=100)))\n",
+ "dataset_type = 'CocoDataset'\n",
+ "data_root = 'data/multisports/'\n",
+ "backend_args = None\n",
+ "train_pipeline = [\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='RandomChoiceResize',\n",
+ " scales=[(1333, 640), (1333, 672), (1333, 704), (1333, 736),\n",
+ " (1333, 768), (1333, 800)],\n",
+ " keep_ratio=True),\n",
+ " dict(type='RandomFlip', prob=0.5),\n",
+ " dict(type='PackDetInputs')\n",
+ "]\n",
+ "test_pipeline = [\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='Resize', scale=(1333, 800), keep_ratio=True),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='PackDetInputs',\n",
+ " meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',\n",
+ " 'scale_factor'))\n",
+ "]\n",
+ "train_dataloader = dict(\n",
+ " batch_size=2,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=True),\n",
+ " batch_sampler=dict(type='AspectRatioBatchSampler'),\n",
+ " dataset=dict(\n",
+ " type='CocoDataset',\n",
+ " data_root='data/multisports/',\n",
+ " ann_file='annotations/multisports_det_anno_train.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " filter_cfg=dict(filter_empty_gt=True, min_size=32),\n",
+ " pipeline=[\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='RandomChoiceResize',\n",
+ " scales=[(1333, 640), (1333, 672), (1333, 704), (1333, 736),\n",
+ " (1333, 768), (1333, 800)],\n",
+ " keep_ratio=True),\n",
+ " dict(type='RandomFlip', prob=0.5),\n",
+ " dict(type='PackDetInputs')\n",
+ " ],\n",
+ " backend_args=None,\n",
+ " metainfo=dict(classes=('person', ), palette=[(220, 20, 60)])))\n",
+ "val_dataloader = dict(\n",
+ " batch_size=1,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " drop_last=False,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=False),\n",
+ " dataset=dict(\n",
+ " type='CocoDataset',\n",
+ " data_root='data/multisports/',\n",
+ " ann_file='annotations/multisports_det_anno_val.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " test_mode=True,\n",
+ " pipeline=[\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='Resize', scale=(1333, 800), keep_ratio=True),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='PackDetInputs',\n",
+ " meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',\n",
+ " 'scale_factor'))\n",
+ " ],\n",
+ " backend_args=None,\n",
+ " metainfo=dict(classes=('person', ), palette=[(220, 20, 60)])))\n",
+ "test_dataloader = dict(\n",
+ " batch_size=1,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " drop_last=False,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=False),\n",
+ " dataset=dict(\n",
+ " type='CocoDataset',\n",
+ " data_root='data/multisports/',\n",
+ " ann_file='annotations/ms_infer_anno.json',\n",
+ " data_prefix=dict(img='rawframes/'),\n",
+ " test_mode=True,\n",
+ " pipeline=[\n",
+ " dict(type='LoadImageFromFile', backend_args=None),\n",
+ " dict(type='Resize', scale=(1333, 800), keep_ratio=True),\n",
+ " dict(type='LoadAnnotations', with_bbox=True),\n",
+ " dict(\n",
+ " type='PackDetInputs',\n",
+ " meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',\n",
+ " 'scale_factor'))\n",
+ " ],\n",
+ " backend_args=None,\n",
+ " metainfo=dict(classes=('person', ), palette=[(220, 20, 60)])))\n",
+ "val_evaluator = dict(\n",
+ " type='CocoMetric',\n",
+ " ann_file='data/multisports/annotations/multisports_det_anno_val.json',\n",
+ " metric='bbox',\n",
+ " format_only=False,\n",
+ " backend_args=None,\n",
+ " metric_items=['mAP_50', 'AR@100'],\n",
+ " iou_thrs=[0.5])\n",
+ "test_evaluator = dict(\n",
+ " type='CocoMetric',\n",
+ " ann_file='data/multisports/annotations/ms_infer_anno.json',\n",
+ " metric='bbox',\n",
+ " format_only=False,\n",
+ " backend_args=None,\n",
+ " metric_items=['mAP_50', 'AR@100'],\n",
+ " iou_thrs=[0.5])\n",
+ "train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=2, val_interval=1)\n",
+ "val_cfg = dict(type='ValLoop')\n",
+ "test_cfg = dict(type='TestLoop')\n",
+ "param_scheduler = [\n",
+ " dict(type='ConstantLR', factor=1.0, by_epoch=False, begin=0, end=500)\n",
+ "]\n",
+ "optim_wrapper = dict(\n",
+ " type='OptimWrapper',\n",
+ " optimizer=dict(type='SGD', lr=0.005, momentum=0.9, weight_decay=0.0001))\n",
+ "auto_scale_lr = dict(enable=False, base_batch_size=16)\n",
+ "default_scope = 'mmdet'\n",
+ "default_hooks = dict(\n",
+ " timer=dict(type='IterTimerHook'),\n",
+ " logger=dict(type='LoggerHook', interval=50),\n",
+ " param_scheduler=dict(type='ParamSchedulerHook'),\n",
+ " checkpoint=dict(type='CheckpointHook', interval=1),\n",
+ " sampler_seed=dict(type='DistSamplerSeedHook'),\n",
+ " visualization=dict(type='DetVisualizationHook'))\n",
+ "env_cfg = dict(\n",
+ " cudnn_benchmark=False,\n",
+ " mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n",
+ " dist_cfg=dict(backend='nccl'))\n",
+ "vis_backends = [dict(type='LocalVisBackend')]\n",
+ "visualizer = dict(\n",
+ " type='DetLocalVisualizer',\n",
+ " vis_backends=[dict(type='LocalVisBackend')],\n",
+ " name='visualizer')\n",
+ "log_processor = dict(type='LogProcessor', window_size=50, by_epoch=True)\n",
+ "log_level = 'INFO'\n",
+ "load_from = 'work_dirs/det_model/epoch_2.pth'\n",
+ "resume = False\n",
+ "metainfo = dict(classes=('person', ), palette=[(220, 20, 60)])\n",
+ "launcher = 'none'\n",
+ "work_dir = './work_dirs/faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person'\n",
+ "\n",
+ "06/15 06:45:55 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.\n",
+ "06/15 06:45:55 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Hooks will be executed in the following order:\n",
+ "before_run:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "before_train:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_train_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) DistSamplerSeedHook \n",
+ " -------------------- \n",
+ "before_train_iter:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_train_iter:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "after_train_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_val_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "before_val_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_val_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) DetVisualizationHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_val_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "after_train:\n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_test_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "before_test_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_test_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) DetVisualizationHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_test_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_run:\n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "loading annotations into memory...\n",
+ "Done (t=0.00s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "loading annotations into memory...\n",
+ "Done (t=0.00s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "06/15 06:45:56 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - The prefix is not set in metric class DumpDetResults.\n",
+ "Loads checkpoint by local backend from path: work_dirs/det_model/epoch_2.pth\n",
+ "06/15 06:45:57 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Load checkpoint from work_dirs/det_model/epoch_2.pth\n",
+ "06/15 06:46:04 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 50/2350] eta: 0:05:46 time: 0.1507 data_time: 0.0046 memory: 512 \n",
+ "06/15 06:46:10 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 100/2350] eta: 0:05:06 time: 0.1217 data_time: 0.0059 memory: 512 \n",
+ "06/15 06:46:16 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 150/2350] eta: 0:04:47 time: 0.1193 data_time: 0.0022 memory: 512 \n",
+ "06/15 06:46:22 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 200/2350] eta: 0:04:34 time: 0.1197 data_time: 0.0023 memory: 512 \n",
+ "06/15 06:46:29 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 250/2350] eta: 0:04:27 time: 0.1258 data_time: 0.0073 memory: 512 \n",
+ "06/15 06:46:35 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 300/2350] eta: 0:04:19 time: 0.1215 data_time: 0.0026 memory: 512 \n",
+ "06/15 06:46:41 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 350/2350] eta: 0:04:12 time: 0.1242 data_time: 0.0046 memory: 512 \n",
+ "06/15 06:46:47 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 400/2350] eta: 0:04:04 time: 0.1218 data_time: 0.0029 memory: 512 \n",
+ "06/15 06:46:53 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 450/2350] eta: 0:03:58 time: 0.1229 data_time: 0.0042 memory: 512 \n",
+ "06/15 06:46:59 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 500/2350] eta: 0:03:51 time: 0.1229 data_time: 0.0048 memory: 512 \n",
+ "06/15 06:47:05 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 550/2350] eta: 0:03:44 time: 0.1193 data_time: 0.0020 memory: 512 \n",
+ "06/15 06:47:12 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 600/2350] eta: 0:03:37 time: 0.1234 data_time: 0.0060 memory: 512 \n",
+ "06/15 06:47:17 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 650/2350] eta: 0:03:30 time: 0.1184 data_time: 0.0025 memory: 512 \n",
+ "06/15 06:47:23 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 700/2350] eta: 0:03:24 time: 0.1200 data_time: 0.0041 memory: 512 \n",
+ "06/15 06:47:30 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 750/2350] eta: 0:03:17 time: 0.1216 data_time: 0.0046 memory: 512 \n",
+ "06/15 06:47:35 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 800/2350] eta: 0:03:11 time: 0.1184 data_time: 0.0024 memory: 512 \n",
+ "06/15 06:47:42 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 850/2350] eta: 0:03:04 time: 0.1234 data_time: 0.0064 memory: 512 \n",
+ "06/15 06:47:48 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 900/2350] eta: 0:02:58 time: 0.1196 data_time: 0.0028 memory: 512 \n",
+ "06/15 06:47:54 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [ 950/2350] eta: 0:02:52 time: 0.1217 data_time: 0.0046 memory: 512 \n",
+ "06/15 06:48:00 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1000/2350] eta: 0:02:45 time: 0.1220 data_time: 0.0046 memory: 512 \n",
+ "06/15 06:48:06 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1050/2350] eta: 0:02:39 time: 0.1203 data_time: 0.0028 memory: 512 \n",
+ "06/15 06:48:12 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1100/2350] eta: 0:02:33 time: 0.1231 data_time: 0.0055 memory: 512 \n",
+ "06/15 06:48:18 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1150/2350] eta: 0:02:27 time: 0.1207 data_time: 0.0033 memory: 512 \n",
+ "06/15 06:48:24 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1200/2350] eta: 0:02:21 time: 0.1217 data_time: 0.0049 memory: 512 \n",
+ "06/15 06:48:30 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1250/2350] eta: 0:02:14 time: 0.1211 data_time: 0.0038 memory: 512 \n",
+ "06/15 06:48:36 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1300/2350] eta: 0:02:08 time: 0.1242 data_time: 0.0070 memory: 512 \n",
+ "06/15 06:48:43 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1350/2350] eta: 0:02:02 time: 0.1249 data_time: 0.0077 memory: 512 \n",
+ "06/15 06:48:49 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1400/2350] eta: 0:01:56 time: 0.1181 data_time: 0.0022 memory: 512 \n",
+ "06/15 06:48:55 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1450/2350] eta: 0:01:50 time: 0.1219 data_time: 0.0055 memory: 512 \n",
+ "06/15 06:49:01 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1500/2350] eta: 0:01:44 time: 0.1198 data_time: 0.0034 memory: 512 \n",
+ "06/15 06:49:07 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1550/2350] eta: 0:01:37 time: 0.1194 data_time: 0.0028 memory: 512 \n",
+ "06/15 06:49:13 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1600/2350] eta: 0:01:31 time: 0.1228 data_time: 0.0059 memory: 512 \n",
+ "06/15 06:49:19 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1650/2350] eta: 0:01:25 time: 0.1193 data_time: 0.0026 memory: 512 \n",
+ "06/15 06:49:25 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1700/2350] eta: 0:01:19 time: 0.1232 data_time: 0.0060 memory: 512 \n",
+ "06/15 06:49:31 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1750/2350] eta: 0:01:13 time: 0.1199 data_time: 0.0028 memory: 512 \n",
+ "06/15 06:49:37 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1800/2350] eta: 0:01:07 time: 0.1205 data_time: 0.0035 memory: 512 \n",
+ "06/15 06:49:43 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1850/2350] eta: 0:01:01 time: 0.1237 data_time: 0.0067 memory: 512 \n",
+ "06/15 06:49:49 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1900/2350] eta: 0:00:54 time: 0.1190 data_time: 0.0024 memory: 512 \n",
+ "06/15 06:49:55 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [1950/2350] eta: 0:00:48 time: 0.1238 data_time: 0.0069 memory: 512 \n",
+ "06/15 06:50:01 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2000/2350] eta: 0:00:42 time: 0.1183 data_time: 0.0020 memory: 512 \n",
+ "06/15 06:50:07 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2050/2350] eta: 0:00:36 time: 0.1212 data_time: 0.0049 memory: 512 \n",
+ "06/15 06:50:13 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2100/2350] eta: 0:00:30 time: 0.1212 data_time: 0.0044 memory: 512 \n",
+ "06/15 06:50:19 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2150/2350] eta: 0:00:24 time: 0.1180 data_time: 0.0019 memory: 512 \n",
+ "06/15 06:50:25 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2200/2350] eta: 0:00:18 time: 0.1233 data_time: 0.0062 memory: 512 \n",
+ "06/15 06:50:31 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2250/2350] eta: 0:00:12 time: 0.1186 data_time: 0.0021 memory: 512 \n",
+ "06/15 06:50:37 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2300/2350] eta: 0:00:06 time: 0.1227 data_time: 0.0064 memory: 512 \n",
+ "06/15 06:50:43 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2350/2350] eta: 0:00:00 time: 0.1196 data_time: 0.0033 memory: 512 \n",
+ "06/15 06:50:44 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Evaluating bbox...\n",
+ "Loading and preparing results...\n",
+ "DONE (t=0.01s)\n",
+ "creating index...\n",
+ "index created!\n",
+ "Running per image evaluation...\n",
+ "Evaluate annotation type *bbox*\n",
+ "DONE (t=0.37s).\n",
+ "Accumulating evaluation results...\n",
+ "DONE (t=0.28s).\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= small | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area=medium | maxDets=1000 ] = -1.000\n",
+ " Average Precision (AP) @[ IoU=0.50:0.50 | area= large | maxDets=1000 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=100 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=300 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= all | maxDets=1000 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= small | maxDets=1000 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area=medium | maxDets=1000 ] = -1.000\n",
+ " Average Recall (AR) @[ IoU=0.50:0.50 | area= large | maxDets=1000 ] = -1.000\n",
+ "06/15 06:50:45 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - bbox_mAP_copypaste: -1.000 -1.000 -1.000 -1.000 -1.000 -1.000\n",
+ "06/15 06:50:45 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Results has been saved to data/multisports/annotations/ms_det_proposals.pkl.\n",
+ "06/15 06:50:45 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(test) [2350/2350] coco/bbox_mAP_50: -1.0000 coco/bbox_AR@100: -1.0000 data_time: 0.0042 time: 0.1219\n",
+ "\u001b[32mTesting finished successfully.\u001b[0m\n"
+ ]
+ }
+ ],
+ "source": [
+ "!mim test mmdet configs/faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person.py \\\n",
+ " --checkpoint work_dirs/det_model/epoch_2.pth \\\n",
+ " --out data/multisports/annotations/ms_det_proposals.pkl"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "1zErF-nsxRAo"
+ },
+ "source": [
+ "## 3. 训练时空行为检测模型\n",
+ "\n",
+ "### 3.1 转换标注文件以及 proposal 文件\n",
+ "\n",
+ "MultiSports 数据集提供的标注文件，以及 MMDetection 推理生成的 proposal 都需要进行格式转换，才能用于时空行为检测模型的训练。我们已经提供了相关的脚本工具，执行后即可生成指定格式"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "naAfcO4QxRAo",
+ "outputId": "2a309bef-241f-44fc-8276-b2ea4735e37d"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "loading test result...\n",
+ "[>>] 2350/2350, 3582.6 task/s, elapsed: 1s, ETA: 0s\n",
+ "\u001b[01;34mdata/multisports/annotations\u001b[00m\n",
+ "├── label_map.txt\n",
+ "├── ms_det_proposals.pkl\n",
+ "├── ms_infer_anno.json\n",
+ "├── multisports_det_anno_train.json\n",
+ "├── multisports_det_anno_val.json\n",
+ "├── \u001b[01;32mmultisports_GT.pkl\u001b[00m\n",
+ "├── multisports_proposals_train.pkl\n",
+ "├── multisports_proposals_val.pkl\n",
+ "├── multisports_train.csv\n",
+ "└── multisports_val.csv\n",
+ "\n",
+ "0 directories, 10 files\n"
+ ]
+ }
+ ],
+ "source": [
+ "# 转换 anno 文件\n",
+ "!python ../../tools/data/multisports/parse_anno.py\n",
+ "\n",
+ "# 转换 proposal 文件\n",
+ "!python tools/convert_proposals.py\n",
+ "\n",
+ "!tree data/multisports/annotations"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "balpcJEbxRAp"
+ },
+ "source": [
+ "### 3.2 训练时空行为检测模型\n",
+ "\n",
+ "MMAction2 中已经支持训练 MultiSports 数据集，这里只需要修改 proposal 文件的路径即可, 详细配置可以参考 [config](configs/slowonly_k400_multisports.py) 文件。由于训练数据较少，配置中将在完整 MultiSports 数据集上训练得到的模型作为预训练模型，使用自定义数据集训练时不需要指定 `load_from` 配置。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "cIuQTmnuxRAq",
+ "outputId": "253d7f08-3c89-4e31-c5f4-3880aed5d817"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Training command is /usr/bin/python3 /content/mmaction2/mmaction/.mim/tools/train.py configs/slowonly_k400_multisports.py --launcher none --work-dir work_dirs/stad_model/. \n",
+ "06/15 06:50:58 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - \n",
+ "------------------------------------------------------------\n",
+ "System environment:\n",
+ " sys.platform: linux\n",
+ " Python: 3.10.12 (main, Jun 7 2023, 12:45:35) [GCC 9.4.0]\n",
+ " CUDA available: True\n",
+ " numpy_random_seed: 546414243\n",
+ " GPU 0: Tesla T4\n",
+ " CUDA_HOME: /usr/local/cuda\n",
+ " NVCC: Cuda compilation tools, release 11.8, V11.8.89\n",
+ " GCC: x86_64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0\n",
+ " PyTorch: 2.0.1+cu118\n",
+ " PyTorch compiling details: PyTorch built with:\n",
+ " - GCC 9.3\n",
+ " - C++ Version: 201703\n",
+ " - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications\n",
+ " - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n",
+ " - OpenMP 201511 (a.k.a. OpenMP 4.5)\n",
+ " - LAPACK is enabled (usually provided by MKL)\n",
+ " - NNPACK is enabled\n",
+ " - CPU capability usage: AVX2\n",
+ " - CUDA Runtime 11.8\n",
+ " - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n",
+ " - CuDNN 8.7\n",
+ " - Magma 2.6.1\n",
+ " - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n",
+ "\n",
+ " TorchVision: 0.15.2+cu118\n",
+ " OpenCV: 4.7.0\n",
+ " MMEngine: 0.7.4\n",
+ "\n",
+ "Runtime environment:\n",
+ " cudnn_benchmark: False\n",
+ " mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}\n",
+ " dist_cfg: {'backend': 'nccl'}\n",
+ " seed: 546414243\n",
+ " diff_rank_seed: False\n",
+ " deterministic: False\n",
+ " Distributed launcher: none\n",
+ " Distributed training: False\n",
+ " GPU number: 1\n",
+ "------------------------------------------------------------\n",
+ "\n",
+ "06/15 06:50:59 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Config:\n",
+ "default_scope = 'mmaction'\n",
+ "default_hooks = dict(\n",
+ " runtime_info=dict(type='RuntimeInfoHook', _scope_='mmaction'),\n",
+ " timer=dict(type='IterTimerHook', _scope_='mmaction'),\n",
+ " logger=dict(\n",
+ " type='LoggerHook', interval=20, ignore_last=False, _scope_='mmaction'),\n",
+ " param_scheduler=dict(type='ParamSchedulerHook', _scope_='mmaction'),\n",
+ " checkpoint=dict(\n",
+ " type='CheckpointHook',\n",
+ " interval=1,\n",
+ " save_best='auto',\n",
+ " _scope_='mmaction'),\n",
+ " sampler_seed=dict(type='DistSamplerSeedHook', _scope_='mmaction'),\n",
+ " sync_buffers=dict(type='SyncBuffersHook', _scope_='mmaction'))\n",
+ "env_cfg = dict(\n",
+ " cudnn_benchmark=False,\n",
+ " mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n",
+ " dist_cfg=dict(backend='nccl'))\n",
+ "log_processor = dict(\n",
+ " type='LogProcessor', window_size=20, by_epoch=True, _scope_='mmaction')\n",
+ "vis_backends = [dict(type='LocalVisBackend', _scope_='mmaction')]\n",
+ "visualizer = dict(\n",
+ " type='ActionVisualizer',\n",
+ " vis_backends=[dict(type='LocalVisBackend')],\n",
+ " _scope_='mmaction')\n",
+ "log_level = 'INFO'\n",
+ "load_from = 'https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-a1ca5e76.pth'\n",
+ "resume = False\n",
+ "url = 'https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-e7b65fad.pth'\n",
+ "num_classes = 66\n",
+ "model = dict(\n",
+ " type='FastRCNN',\n",
+ " _scope_='mmdet',\n",
+ " init_cfg=dict(\n",
+ " type='Pretrained',\n",
+ " checkpoint=\n",
+ " 'https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-e7b65fad.pth'\n",
+ " ),\n",
+ " backbone=dict(\n",
+ " type='mmaction.ResNet3dSlowOnly',\n",
+ " depth=50,\n",
+ " pretrained=None,\n",
+ " pretrained2d=False,\n",
+ " lateral=False,\n",
+ " num_stages=4,\n",
+ " conv1_kernel=(1, 7, 7),\n",
+ " conv1_stride_t=1,\n",
+ " pool1_stride_t=1,\n",
+ " spatial_strides=(1, 2, 2, 1)),\n",
+ " roi_head=dict(\n",
+ " type='AVARoIHead',\n",
+ " bbox_roi_extractor=dict(\n",
+ " type='SingleRoIExtractor3D',\n",
+ " roi_layer_type='RoIAlign',\n",
+ " output_size=8,\n",
+ " with_temporal_pool=True),\n",
+ " bbox_head=dict(\n",
+ " type='BBoxHeadAVA',\n",
+ " in_channels=2048,\n",
+ " num_classes=66,\n",
+ " multilabel=False,\n",
+ " dropout_ratio=0.5)),\n",
+ " data_preprocessor=dict(\n",
+ " type='mmaction.ActionDataPreprocessor',\n",
+ " mean=[123.675, 116.28, 103.53],\n",
+ " std=[58.395, 57.12, 57.375],\n",
+ " format_shape='NCTHW'),\n",
+ " train_cfg=dict(\n",
+ " rcnn=dict(\n",
+ " assigner=dict(\n",
+ " type='MaxIoUAssignerAVA',\n",
+ " pos_iou_thr=0.9,\n",
+ " neg_iou_thr=0.9,\n",
+ " min_pos_iou=0.9),\n",
+ " sampler=dict(\n",
+ " type='RandomSampler',\n",
+ " num=32,\n",
+ " pos_fraction=1,\n",
+ " neg_pos_ub=-1,\n",
+ " add_gt_as_proposals=True),\n",
+ " pos_weight=1.0)),\n",
+ " test_cfg=dict(rcnn=None))\n",
+ "dataset_type = 'AVADataset'\n",
+ "data_root = 'data/multisports/trainval'\n",
+ "anno_root = 'data/multisports/annotations'\n",
+ "ann_file_train = 'data/multisports/annotations/multisports_train.csv'\n",
+ "ann_file_val = 'data/multisports/annotations/multisports_val.csv'\n",
+ "gt_file = 'data/multisports/annotations/multisports_GT.pkl'\n",
+ "proposal_file_train = 'data/multisports/annotations/multisports_proposals_train.pkl'\n",
+ "proposal_file_val = 'data/multisports/annotations/multisports_proposals_val.pkl'\n",
+ "file_client_args = dict(io_backend='disk')\n",
+ "train_pipeline = [\n",
+ " dict(type='DecordInit', io_backend='disk', _scope_='mmaction'),\n",
+ " dict(\n",
+ " type='SampleAVAFrames',\n",
+ " clip_len=4,\n",
+ " frame_interval=16,\n",
+ " _scope_='mmaction'),\n",
+ " dict(type='DecordDecode', _scope_='mmaction'),\n",
+ " dict(type='RandomRescale', scale_range=(256, 320), _scope_='mmaction'),\n",
+ " dict(type='RandomCrop', size=256, _scope_='mmaction'),\n",
+ " dict(type='Flip', flip_ratio=0.5, _scope_='mmaction'),\n",
+ " dict(\n",
+ " type='FormatShape',\n",
+ " input_format='NCTHW',\n",
+ " collapse=True,\n",
+ " _scope_='mmaction'),\n",
+ " dict(type='PackActionInputs', _scope_='mmaction')\n",
+ "]\n",
+ "val_pipeline = [\n",
+ " dict(type='DecordInit', io_backend='disk', _scope_='mmaction'),\n",
+ " dict(\n",
+ " type='SampleAVAFrames',\n",
+ " clip_len=4,\n",
+ " frame_interval=16,\n",
+ " test_mode=True,\n",
+ " _scope_='mmaction'),\n",
+ " dict(type='DecordDecode', _scope_='mmaction'),\n",
+ " dict(type='Resize', scale=(-1, 256), _scope_='mmaction'),\n",
+ " dict(\n",
+ " type='FormatShape',\n",
+ " input_format='NCTHW',\n",
+ " collapse=True,\n",
+ " _scope_='mmaction'),\n",
+ " dict(type='PackActionInputs', _scope_='mmaction')\n",
+ "]\n",
+ "train_dataloader = dict(\n",
+ " batch_size=2,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=True, _scope_='mmaction'),\n",
+ " dataset=dict(\n",
+ " type='AVADataset',\n",
+ " ann_file='data/multisports/annotations/multisports_train.csv',\n",
+ " pipeline=[\n",
+ " dict(type='DecordInit', io_backend='disk'),\n",
+ " dict(type='SampleAVAFrames', clip_len=4, frame_interval=16),\n",
+ " dict(type='DecordDecode'),\n",
+ " dict(type='RandomRescale', scale_range=(256, 320)),\n",
+ " dict(type='RandomCrop', size=256),\n",
+ " dict(type='Flip', flip_ratio=0.5),\n",
+ " dict(type='FormatShape', input_format='NCTHW', collapse=True),\n",
+ " dict(type='PackActionInputs')\n",
+ " ],\n",
+ " num_classes=66,\n",
+ " proposal_file=\n",
+ " 'data/multisports/annotations/multisports_proposals_train.pkl',\n",
+ " data_prefix=dict(img='data/multisports/trainval'),\n",
+ " timestamp_start=1,\n",
+ " start_index=0,\n",
+ " use_frames=False,\n",
+ " fps=1,\n",
+ " _scope_='mmaction'))\n",
+ "val_dataloader = dict(\n",
+ " batch_size=1,\n",
+ " num_workers=2,\n",
+ " persistent_workers=True,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=False, _scope_='mmaction'),\n",
+ " dataset=dict(\n",
+ " type='AVADataset',\n",
+ " ann_file='data/multisports/annotations/multisports_val.csv',\n",
+ " pipeline=[\n",
+ " dict(type='DecordInit', io_backend='disk'),\n",
+ " dict(\n",
+ " type='SampleAVAFrames',\n",
+ " clip_len=4,\n",
+ " frame_interval=16,\n",
+ " test_mode=True),\n",
+ " dict(type='DecordDecode'),\n",
+ " dict(type='Resize', scale=(-1, 256)),\n",
+ " dict(type='FormatShape', input_format='NCTHW', collapse=True),\n",
+ " dict(type='PackActionInputs')\n",
+ " ],\n",
+ " num_classes=66,\n",
+ " proposal_file=\n",
+ " 'data/multisports/annotations/multisports_proposals_val.pkl',\n",
+ " data_prefix=dict(img='data/multisports/trainval'),\n",
+ " test_mode=True,\n",
+ " timestamp_start=1,\n",
+ " start_index=0,\n",
+ " use_frames=False,\n",
+ " fps=1,\n",
+ " _scope_='mmaction'))\n",
+ "test_dataloader = dict(\n",
+ " batch_size=1,\n",
+ " num_workers=8,\n",
+ " persistent_workers=True,\n",
+ " sampler=dict(type='DefaultSampler', shuffle=False, _scope_='mmaction'),\n",
+ " dataset=dict(\n",
+ " type='AVADataset',\n",
+ " ann_file='data/multisports/annotations/multisports_val.csv',\n",
+ " pipeline=[\n",
+ " dict(type='DecordInit', io_backend='disk'),\n",
+ " dict(\n",
+ " type='SampleAVAFrames',\n",
+ " clip_len=4,\n",
+ " frame_interval=16,\n",
+ " test_mode=True),\n",
+ " dict(type='DecordDecode'),\n",
+ " dict(type='Resize', scale=(-1, 256)),\n",
+ " dict(type='FormatShape', input_format='NCTHW', collapse=True),\n",
+ " dict(type='PackActionInputs')\n",
+ " ],\n",
+ " num_classes=66,\n",
+ " proposal_file=\n",
+ " 'data/multisports/annotations/multisports_dense_proposals_val.recall_96.13.pkl',\n",
+ " data_prefix=dict(img='data/multisports/trainval'),\n",
+ " test_mode=True,\n",
+ " timestamp_start=1,\n",
+ " start_index=0,\n",
+ " use_frames=False,\n",
+ " fps=1,\n",
+ " _scope_='mmaction'))\n",
+ "val_evaluator = dict(\n",
+ " type='MultiSportsMetric',\n",
+ " ann_file='data/multisports/annotations/multisports_GT.pkl',\n",
+ " _scope_='mmaction')\n",
+ "test_evaluator = dict(\n",
+ " type='MultiSportsMetric',\n",
+ " ann_file='data/multisports/annotations/multisports_GT.pkl',\n",
+ " _scope_='mmaction')\n",
+ "train_cfg = dict(\n",
+ " type='EpochBasedTrainLoop',\n",
+ " max_epochs=8,\n",
+ " val_begin=1,\n",
+ " val_interval=1,\n",
+ " _scope_='mmaction')\n",
+ "val_cfg = dict(type='ValLoop', _scope_='mmaction')\n",
+ "test_cfg = dict(type='TestLoop', _scope_='mmaction')\n",
+ "param_scheduler = [\n",
+ " dict(\n",
+ " type='LinearLR',\n",
+ " start_factor=0.1,\n",
+ " by_epoch=True,\n",
+ " begin=0,\n",
+ " end=5,\n",
+ " _scope_='mmaction'),\n",
+ " dict(\n",
+ " type='MultiStepLR',\n",
+ " begin=0,\n",
+ " end=8,\n",
+ " by_epoch=True,\n",
+ " milestones=[6, 7],\n",
+ " gamma=0.1,\n",
+ " _scope_='mmaction')\n",
+ "]\n",
+ "optim_wrapper = dict(\n",
+ " optimizer=dict(\n",
+ " type='SGD',\n",
+ " lr=0.01,\n",
+ " momentum=0.9,\n",
+ " weight_decay=1e-05,\n",
+ " _scope_='mmaction'),\n",
+ " clip_grad=dict(max_norm=5, norm_type=2))\n",
+ "launcher = 'none'\n",
+ "work_dir = 'work_dirs/stad_model/'\n",
+ "randomness = dict(seed=None, diff_rank_seed=False, deterministic=False)\n",
+ "\n",
+ "06/15 06:51:03 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.\n",
+ "06/15 06:51:03 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Hooks will be executed in the following order:\n",
+ "before_run:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "before_train:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_train_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) DistSamplerSeedHook \n",
+ " -------------------- \n",
+ "before_train_iter:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_train_iter:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "after_train_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) SyncBuffersHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_val_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(NORMAL ) SyncBuffersHook \n",
+ " -------------------- \n",
+ "before_val_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_val_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_val_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ "(LOW ) ParamSchedulerHook \n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "after_train:\n",
+ "(VERY_LOW ) CheckpointHook \n",
+ " -------------------- \n",
+ "before_test_epoch:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "before_test_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ " -------------------- \n",
+ "after_test_iter:\n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_test_epoch:\n",
+ "(VERY_HIGH ) RuntimeInfoHook \n",
+ "(NORMAL ) IterTimerHook \n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "after_run:\n",
+ "(BELOW_NORMAL) LoggerHook \n",
+ " -------------------- \n",
+ "06/15 06:51:05 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - 236 out of 236 frames are valid.\n",
+ "06/15 06:51:05 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - 120 out of 120 frames are valid.\n",
+ "06/15 06:51:07 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - load model from: https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-e7b65fad.pth\n",
+ "06/15 06:51:07 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Loads checkpoint by http backend from path: https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-e7b65fad.pth\n",
+ "Downloading: \"https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-e7b65fad.pth\" to /root/.cache/torch/hub/checkpoints/slowonly_imagenet-pretrained-r50_8xb16-4x16x1-steplr-150e_kinetics400-rgb_20220901-e7b65fad.pth\n",
+ "100% 124M/124M [00:05<00:00, 25.9MB/s]\n",
+ "06/15 06:51:12 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - The model and loaded state dict do not match exactly\n",
+ "\n",
+ "unexpected key in source state_dict: cls_head.fc_cls.weight, cls_head.fc_cls.bias\n",
+ "\n",
+ "missing keys in source state_dict: roi_head.bbox_head.fc_cls.weight, roi_head.bbox_head.fc_cls.bias\n",
+ "\n",
+ "Loads checkpoint by http backend from path: https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-a1ca5e76.pth\n",
+ "Downloading: \"https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-a1ca5e76.pth\" to /root/.cache/torch/hub/checkpoints/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-a1ca5e76.pth\n",
+ "100% 122M/122M [00:04<00:00, 29.7MB/s]\n",
+ "06/15 06:51:17 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Load checkpoint from https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-a1ca5e76.pth\n",
+ "06/15 06:51:17 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - \"FileClient\" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io\n",
+ "06/15 06:51:17 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - \"HardDiskBackend\" is the alias of \"LocalBackend\" and the former will be deprecated in future.\n",
+ "06/15 06:51:17 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Checkpoints will be saved to /content/mmaction2/projects/stad_tutorial/work_dirs/stad_model.\n",
+ "06/15 06:51:26 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][ 20/118] lr: 1.0000e-03 eta: 0:07:06 time: 0.4613 data_time: 0.0472 memory: 1381 grad_norm: 17.8613 loss: 1.1505 recall@thr=0.5: 0.6667 prec@thr=0.5: 0.6667 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 1.1505\n",
+ "06/15 06:51:31 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][ 40/118] lr: 1.0000e-03 eta: 0:05:28 time: 0.2655 data_time: 0.0204 memory: 1381 grad_norm: 6.8642 loss: 0.5417 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.5417\n",
+ "06/15 06:51:38 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][ 60/118] lr: 1.0000e-03 eta: 0:05:06 time: 0.3121 data_time: 0.0505 memory: 1381 grad_norm: 5.3190 loss: 0.6625 recall@thr=0.5: 0.9000 prec@thr=0.5: 0.9000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.6625\n",
+ "06/15 06:51:43 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][ 80/118] lr: 1.0000e-03 eta: 0:04:44 time: 0.2771 data_time: 0.0255 memory: 1381 grad_norm: 3.0057 loss: 0.6646 recall@thr=0.5: 0.9231 prec@thr=0.5: 0.9231 recall@top3: 0.9231 prec@top3: 0.3077 recall@top5: 0.9231 prec@top5: 0.1846 loss_action_cls: 0.6646\n",
+ "06/15 06:51:48 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][100/118] lr: 1.0000e-03 eta: 0:04:26 time: 0.2625 data_time: 0.0130 memory: 1381 grad_norm: 1.8442 loss: 0.5711 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.5711\n",
+ "06/15 06:51:54 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_065057\n",
+ "06/15 06:51:54 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [1][118/118] lr: 1.0000e-03 eta: 0:04:18 time: 0.2930 data_time: 0.0322 memory: 1381 grad_norm: 2.5183 loss: 0.6887 recall@thr=0.5: 0.6923 prec@thr=0.5: 0.6923 recall@top3: 0.6923 prec@top3: 0.2308 recall@top5: 0.6923 prec@top5: 0.1385 loss_action_cls: 0.6887\n",
+ "06/15 06:51:54 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 1 epochs\n",
+ "06/15 06:51:59 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][ 20/120] eta: 0:00:14 time: 0.1446 data_time: 0.0853 memory: 466 \n",
+ "06/15 06:52:01 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][ 40/120] eta: 0:00:10 time: 0.1124 data_time: 0.0612 memory: 466 \n",
+ "06/15 06:52:03 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][ 60/120] eta: 0:00:07 time: 0.1016 data_time: 0.0505 memory: 466 \n",
+ "06/15 06:52:05 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][ 80/120] eta: 0:00:04 time: 0.1083 data_time: 0.0581 memory: 466 \n",
+ "06/15 06:52:08 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][100/120] eta: 0:00:02 time: 0.1650 data_time: 0.1102 memory: 466 \n",
+ "06/15 06:52:11 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][120/120] eta: 0:00:00 time: 0.1410 data_time: 0.0866 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 47.41\n",
+ "aerobic split jump 30.01\n",
+ "aerobic scissors leap 88.94\n",
+ "aerobic turn 98.43\n",
+ "mAP 66.20\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 25.00\n",
+ "aerobic split jump 20.00\n",
+ "aerobic scissors leap 80.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 56.25\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 45.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 36.25\n",
+ "06/15 06:52:12 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [1][120/120] mAP/frameAP: 66.1965 mAP/v_map@0.2: 56.2500 mAP/v_map@0.5: 36.2500 mAP/v_map_0.05:0.45: 50.4167 mAP/v_map_0.10:0.90: 37.7963 mAP/v_map_0.50:0.95: 26.8167 data_time: 0.0753 time: 0.1288\n",
+ "06/15 06:52:13 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - The best checkpoint with 66.1965 mAP/frameAP at 1 epoch is saved to best_mAP_frameAP_epoch_1.pth.\n",
+ "06/15 06:52:23 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][ 20/118] lr: 3.2500e-03 eta: 0:04:11 time: 0.3098 data_time: 0.0484 memory: 1381 grad_norm: 1.1745 loss: 0.4384 recall@thr=0.5: 0.7857 prec@thr=0.5: 0.7857 recall@top3: 0.9286 prec@top3: 0.3095 recall@top5: 0.9286 prec@top5: 0.1857 loss_action_cls: 0.4384\n",
+ "06/15 06:52:30 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][ 40/118] lr: 3.2500e-03 eta: 0:04:06 time: 0.3245 data_time: 0.0667 memory: 1381 grad_norm: 1.0271 loss: 0.3960 recall@thr=0.5: 0.9333 prec@thr=0.5: 0.9333 recall@top3: 0.9333 prec@top3: 0.3111 recall@top5: 0.9333 prec@top5: 0.1867 loss_action_cls: 0.3960\n",
+ "06/15 06:52:35 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][ 60/118] lr: 3.2500e-03 eta: 0:03:55 time: 0.2572 data_time: 0.0111 memory: 1381 grad_norm: 0.8150 loss: 0.3958 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3958\n",
+ "06/15 06:52:41 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][ 80/118] lr: 3.2500e-03 eta: 0:03:47 time: 0.2843 data_time: 0.0167 memory: 1381 grad_norm: 1.4691 loss: 0.4575 recall@thr=0.5: 0.9333 prec@thr=0.5: 0.9333 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.4575\n",
+ "06/15 06:52:47 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][100/118] lr: 3.2500e-03 eta: 0:03:41 time: 0.3118 data_time: 0.0559 memory: 1381 grad_norm: 1.9420 loss: 0.5529 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.5529\n",
+ "06/15 06:52:52 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_065057\n",
+ "06/15 06:52:52 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [2][118/118] lr: 3.2500e-03 eta: 0:03:33 time: 0.2532 data_time: 0.0082 memory: 1381 grad_norm: 1.6790 loss: 0.4253 recall@thr=0.5: 0.7500 prec@thr=0.5: 0.7500 recall@top3: 0.8333 prec@top3: 0.2778 recall@top5: 0.8333 prec@top5: 0.1667 loss_action_cls: 0.4253\n",
+ "06/15 06:52:52 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 2 epochs\n",
+ "06/15 06:52:56 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][ 20/120] eta: 0:00:15 time: 0.1515 data_time: 0.0968 memory: 466 \n",
+ "06/15 06:53:00 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][ 40/120] eta: 0:00:12 time: 0.1679 data_time: 0.1143 memory: 466 \n",
+ "06/15 06:53:02 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][ 60/120] eta: 0:00:08 time: 0.1134 data_time: 0.0631 memory: 466 \n",
+ "06/15 06:53:04 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][ 80/120] eta: 0:00:05 time: 0.0961 data_time: 0.0459 memory: 466 \n",
+ "06/15 06:53:06 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][100/120] eta: 0:00:02 time: 0.1063 data_time: 0.0549 memory: 466 \n",
+ "06/15 06:53:08 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][120/120] eta: 0:00:00 time: 0.1017 data_time: 0.0522 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 42.09\n",
+ "aerobic split jump 27.71\n",
+ "aerobic scissors leap 90.02\n",
+ "aerobic turn 95.76\n",
+ "mAP 63.89\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 20.00\n",
+ "aerobic scissors leap 100.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 55.00\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 36.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 34.00\n",
+ "06/15 06:53:08 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [2][120/120] mAP/frameAP: 63.8934 mAP/v_map@0.2: 55.0000 mAP/v_map@0.5: 34.0000 mAP/v_map_0.05:0.45: 51.8889 mAP/v_map_0.10:0.90: 34.0278 mAP/v_map_0.50:0.95: 18.7250 data_time: 0.0710 time: 0.1226\n",
+ "06/15 06:53:17 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [3][ 20/118] lr: 5.5000e-03 eta: 0:03:34 time: 0.4330 data_time: 0.1493 memory: 1381 grad_norm: 0.4795 loss: 0.5049 recall@thr=0.5: 0.8462 prec@thr=0.5: 0.8462 recall@top3: 0.8462 prec@top3: 0.2821 recall@top5: 0.8462 prec@top5: 0.1692 loss_action_cls: 0.5049\n",
+ "06/15 06:53:23 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [3][ 40/118] lr: 5.5000e-03 eta: 0:03:27 time: 0.2948 data_time: 0.0370 memory: 1381 grad_norm: 0.8584 loss: 0.4820 recall@thr=0.5: 0.6154 prec@thr=0.5: 0.6154 recall@top3: 0.6154 prec@top3: 0.2051 recall@top5: 0.6154 prec@top5: 0.1231 loss_action_cls: 0.4820\n",
+ "06/15 06:53:28 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [3][ 60/118] lr: 5.5000e-03 eta: 0:03:19 time: 0.2622 data_time: 0.0118 memory: 1381 grad_norm: 1.1041 loss: 0.2944 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2944\n",
+ "06/15 06:53:35 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [3][ 80/118] lr: 5.5000e-03 eta: 0:03:13 time: 0.3111 data_time: 0.0470 memory: 1381 grad_norm: 0.8394 loss: 0.3393 recall@thr=0.5: 0.9091 prec@thr=0.5: 0.9091 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3393\n",
+ "06/15 06:53:40 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [3][100/118] lr: 5.5000e-03 eta: 0:03:06 time: 0.2989 data_time: 0.0417 memory: 1381 grad_norm: 0.2155 loss: 0.4345 recall@thr=0.5: 0.8182 prec@thr=0.5: 0.8182 recall@top3: 0.8182 prec@top3: 0.2727 recall@top5: 0.8182 prec@top5: 0.1636 loss_action_cls: 0.4345\n",
+ "06/15 06:53:45 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_065057\n",
+ "06/15 06:53:45 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [3][118/118] lr: 5.5000e-03 eta: 0:02:59 time: 0.2576 data_time: 0.0112 memory: 1381 grad_norm: 0.2509 loss: 0.4634 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.4634\n",
+ "06/15 06:53:45 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 3 epochs\n",
+ "06/15 06:53:50 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [3][ 20/120] eta: 0:00:18 time: 0.1815 data_time: 0.1180 memory: 466 \n",
+ "06/15 06:53:53 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [3][ 40/120] eta: 0:00:13 time: 0.1451 data_time: 0.0905 memory: 466 \n",
+ "06/15 06:53:55 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [3][ 60/120] eta: 0:00:08 time: 0.1020 data_time: 0.0510 memory: 466 \n",
+ "06/15 06:53:57 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [3][ 80/120] eta: 0:00:05 time: 0.1008 data_time: 0.0528 memory: 466 \n",
+ "06/15 06:54:00 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [3][100/120] eta: 0:00:02 time: 0.1072 data_time: 0.0569 memory: 466 \n",
+ "06/15 06:54:02 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [3][120/120] eta: 0:00:00 time: 0.1018 data_time: 0.0536 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 37.09\n",
+ "aerobic split jump 27.98\n",
+ "aerobic scissors leap 89.41\n",
+ "aerobic turn 95.67\n",
+ "mAP 62.54\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 20.00\n",
+ "aerobic scissors leap 100.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 55.00\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 36.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 34.00\n",
+ "06/15 06:54:02 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [3][120/120] mAP/frameAP: 62.5361 mAP/v_map@0.2: 55.0000 mAP/v_map@0.5: 34.0000 mAP/v_map_0.05:0.45: 51.2222 mAP/v_map_0.10:0.90: 34.1389 mAP/v_map_0.50:0.95: 18.7250 data_time: 0.0704 time: 0.1229\n",
+ "06/15 06:54:10 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [4][ 20/118] lr: 7.7500e-03 eta: 0:02:55 time: 0.3717 data_time: 0.0993 memory: 1381 grad_norm: 0.2139 loss: 0.3119 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3119\n",
+ "06/15 06:54:15 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [4][ 40/118] lr: 7.7500e-03 eta: 0:02:48 time: 0.2730 data_time: 0.0230 memory: 1381 grad_norm: 0.6102 loss: 0.4782 recall@thr=0.5: 0.9375 prec@thr=0.5: 0.9375 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.4782\n",
+ "06/15 06:54:21 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [4][ 60/118] lr: 7.7500e-03 eta: 0:02:41 time: 0.2895 data_time: 0.0311 memory: 1381 grad_norm: 0.4057 loss: 0.3422 recall@thr=0.5: 0.9474 prec@thr=0.5: 0.9474 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3422\n",
+ "06/15 06:54:27 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [4][ 80/118] lr: 7.7500e-03 eta: 0:02:36 time: 0.3170 data_time: 0.0490 memory: 1381 grad_norm: 0.3051 loss: 0.3628 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3628\n",
+ "06/15 06:54:32 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [4][100/118] lr: 7.7500e-03 eta: 0:02:29 time: 0.2633 data_time: 0.0131 memory: 1381 grad_norm: 0.1671 loss: 0.3691 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3691\n",
+ "06/15 06:54:37 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_065057\n",
+ "06/15 06:54:37 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [4][118/118] lr: 7.7500e-03 eta: 0:02:23 time: 0.2721 data_time: 0.0181 memory: 1381 grad_norm: 0.1954 loss: 0.3076 recall@thr=0.5: 0.8571 prec@thr=0.5: 0.8571 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3076\n",
+ "06/15 06:54:37 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 4 epochs\n",
+ "06/15 06:54:43 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [4][ 20/120] eta: 0:00:14 time: 0.1431 data_time: 0.0854 memory: 466 \n",
+ "06/15 06:54:45 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [4][ 40/120] eta: 0:00:10 time: 0.1086 data_time: 0.0584 memory: 466 \n",
+ "06/15 06:54:47 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [4][ 60/120] eta: 0:00:07 time: 0.1056 data_time: 0.0552 memory: 466 \n",
+ "06/15 06:54:49 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [4][ 80/120] eta: 0:00:04 time: 0.0922 data_time: 0.0399 memory: 466 \n",
+ "06/15 06:54:51 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [4][100/120] eta: 0:00:02 time: 0.1166 data_time: 0.0671 memory: 466 \n",
+ "06/15 06:54:54 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [4][120/120] eta: 0:00:00 time: 0.1468 data_time: 0.0927 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 25.62\n",
+ "aerobic split jump 28.75\n",
+ "aerobic scissors leap 89.02\n",
+ "aerobic turn 93.30\n",
+ "mAP 59.17\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 20.00\n",
+ "aerobic scissors leap 80.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 50.00\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 45.00\n",
+ "aerobic turn 100.00\n",
+ "mAP 36.25\n",
+ "06/15 06:54:55 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [4][120/120] mAP/frameAP: 59.1749 mAP/v_map@0.2: 50.0000 mAP/v_map@0.5: 36.2500 mAP/v_map_0.05:0.45: 46.9444 mAP/v_map_0.10:0.90: 28.9352 mAP/v_map_0.50:0.95: 14.6667 data_time: 0.0663 time: 0.1186\n",
+ "06/15 06:55:01 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [5][ 20/118] lr: 1.0000e-02 eta: 0:02:17 time: 0.3090 data_time: 0.0513 memory: 1381 grad_norm: 0.2988 loss: 0.3067 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3067\n",
+ "06/15 06:55:06 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [5][ 40/118] lr: 1.0000e-02 eta: 0:02:10 time: 0.2584 data_time: 0.0142 memory: 1381 grad_norm: 0.6702 loss: 0.3996 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3996\n",
+ "06/15 06:55:13 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [5][ 60/118] lr: 1.0000e-02 eta: 0:02:04 time: 0.3286 data_time: 0.0617 memory: 1381 grad_norm: 0.4347 loss: 0.4374 recall@thr=0.5: 0.8462 prec@thr=0.5: 0.8462 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.4374\n",
+ "06/15 06:55:19 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [5][ 80/118] lr: 1.0000e-02 eta: 0:01:58 time: 0.2774 data_time: 0.0247 memory: 1381 grad_norm: 0.4373 loss: 0.3679 recall@thr=0.5: 0.7500 prec@thr=0.5: 0.7500 recall@top3: 0.8750 prec@top3: 0.2917 recall@top5: 0.8750 prec@top5: 0.1750 loss_action_cls: 0.3679\n",
+ "06/15 06:55:24 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [5][100/118] lr: 1.0000e-02 eta: 0:01:51 time: 0.2603 data_time: 0.0108 memory: 1381 grad_norm: 0.2507 loss: 0.3226 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3226\n",
+ "06/15 06:55:30 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_065057\n",
+ "06/15 06:55:30 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [5][118/118] lr: 1.0000e-02 eta: 0:01:46 time: 0.3256 data_time: 0.0497 memory: 1381 grad_norm: 0.0940 loss: 0.2914 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2914\n",
+ "06/15 06:55:30 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 5 epochs\n",
+ "06/15 06:55:34 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [5][ 20/120] eta: 0:00:11 time: 0.1166 data_time: 0.0625 memory: 466 \n",
+ "06/15 06:55:36 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [5][ 40/120] eta: 0:00:09 time: 0.1119 data_time: 0.0618 memory: 466 \n",
+ "06/15 06:55:38 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [5][ 60/120] eta: 0:00:06 time: 0.1012 data_time: 0.0504 memory: 466 \n",
+ "06/15 06:55:40 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [5][ 80/120] eta: 0:00:04 time: 0.1017 data_time: 0.0537 memory: 466 \n",
+ "06/15 06:55:44 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [5][100/120] eta: 0:00:02 time: 0.1766 data_time: 0.1239 memory: 466 \n",
+ "06/15 06:55:46 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [5][120/120] eta: 0:00:00 time: 0.1421 data_time: 0.0884 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 17.82\n",
+ "aerobic split jump 20.05\n",
+ "aerobic scissors leap 89.00\n",
+ "aerobic turn 91.20\n",
+ "mAP 54.52\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 80.00\n",
+ "aerobic turn 60.00\n",
+ "mAP 35.00\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 45.00\n",
+ "aerobic turn 26.67\n",
+ "mAP 17.92\n",
+ "06/15 06:55:47 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [5][120/120] mAP/frameAP: 54.5189 mAP/v_map@0.2: 35.0000 mAP/v_map@0.5: 17.9167 mAP/v_map_0.05:0.45: 31.2037 mAP/v_map_0.10:0.90: 19.0741 mAP/v_map_0.50:0.95: 9.5833 data_time: 0.0733 time: 0.1249\n",
+ "06/15 06:55:53 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [6][ 20/118] lr: 1.0000e-02 eta: 0:01:40 time: 0.2867 data_time: 0.0385 memory: 1381 grad_norm: 0.1572 loss: 0.3008 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3008\n",
+ "06/15 06:55:58 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [6][ 40/118] lr: 1.0000e-02 eta: 0:01:34 time: 0.2720 data_time: 0.0167 memory: 1381 grad_norm: 0.0803 loss: 0.2377 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2377\n",
+ "06/15 06:56:05 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [6][ 60/118] lr: 1.0000e-02 eta: 0:01:28 time: 0.3423 data_time: 0.0840 memory: 1381 grad_norm: 0.3120 loss: 0.2442 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2442\n",
+ "06/15 06:56:10 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [6][ 80/118] lr: 1.0000e-02 eta: 0:01:22 time: 0.2580 data_time: 0.0112 memory: 1381 grad_norm: 0.5726 loss: 0.3794 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3794\n",
+ "06/15 06:56:16 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [6][100/118] lr: 1.0000e-02 eta: 0:01:16 time: 0.2949 data_time: 0.0347 memory: 1381 grad_norm: 0.1732 loss: 0.3004 recall@thr=0.5: 0.8750 prec@thr=0.5: 0.8750 recall@top3: 0.8750 prec@top3: 0.2917 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3004\n",
+ "06/15 06:56:22 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_065057\n",
+ "06/15 06:56:22 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [6][118/118] lr: 1.0000e-02 eta: 0:01:10 time: 0.3258 data_time: 0.0625 memory: 1381 grad_norm: 0.3709 loss: 0.3439 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3439\n",
+ "06/15 06:56:22 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 6 epochs\n",
+ "06/15 06:56:26 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [6][ 20/120] eta: 0:00:11 time: 0.1169 data_time: 0.0624 memory: 466 \n",
+ "06/15 06:56:28 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [6][ 40/120] eta: 0:00:09 time: 0.1131 data_time: 0.0631 memory: 466 \n",
+ "06/15 06:56:30 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [6][ 60/120] eta: 0:00:06 time: 0.1064 data_time: 0.0553 memory: 466 \n",
+ "06/15 06:56:33 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [6][ 80/120] eta: 0:00:04 time: 0.1401 data_time: 0.0862 memory: 466 \n",
+ "06/15 06:56:36 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [6][100/120] eta: 0:00:02 time: 0.1519 data_time: 0.0982 memory: 466 \n",
+ "06/15 06:56:38 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [6][120/120] eta: 0:00:00 time: 0.0986 data_time: 0.0486 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 19.05\n",
+ "aerobic split jump 22.20\n",
+ "aerobic scissors leap 85.83\n",
+ "aerobic turn 79.04\n",
+ "mAP 51.53\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 80.00\n",
+ "aerobic turn 0.00\n",
+ "mAP 20.00\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 45.00\n",
+ "aerobic turn 0.00\n",
+ "mAP 11.25\n",
+ "06/15 06:56:38 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [6][120/120] mAP/frameAP: 51.5300 mAP/v_map@0.2: 20.0000 mAP/v_map@0.5: 11.2500 mAP/v_map_0.05:0.45: 18.0556 mAP/v_map_0.10:0.90: 11.8519 mAP/v_map_0.50:0.95: 6.9167 data_time: 0.0688 time: 0.1209\n",
+ "06/15 06:56:44 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [7][ 20/118] lr: 1.0000e-03 eta: 0:01:04 time: 0.2819 data_time: 0.0331 memory: 1381 grad_norm: 0.2811 loss: 0.2776 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2776\n",
+ "06/15 06:56:50 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [7][ 40/118] lr: 1.0000e-03 eta: 0:00:58 time: 0.3114 data_time: 0.0473 memory: 1381 grad_norm: 0.1573 loss: 0.2043 recall@thr=0.5: 0.8182 prec@thr=0.5: 0.8182 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2043\n",
+ "06/15 06:56:56 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [7][ 60/118] lr: 1.0000e-03 eta: 0:00:52 time: 0.2903 data_time: 0.0342 memory: 1381 grad_norm: 0.1343 loss: 0.3411 recall@thr=0.5: 0.8667 prec@thr=0.5: 0.8667 recall@top3: 0.8667 prec@top3: 0.2889 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3411\n",
+ "06/15 06:57:01 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [7][ 80/118] lr: 1.0000e-03 eta: 0:00:46 time: 0.2623 data_time: 0.0128 memory: 1381 grad_norm: 0.1026 loss: 0.2895 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2895\n",
+ "06/15 06:57:08 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [7][100/118] lr: 1.0000e-03 eta: 0:00:40 time: 0.3206 data_time: 0.0503 memory: 1381 grad_norm: 0.1911 loss: 0.3552 recall@thr=0.5: 0.7333 prec@thr=0.5: 0.7333 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3552\n",
+ "06/15 06:57:13 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_065057\n",
+ "06/15 06:57:13 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [7][118/118] lr: 1.0000e-03 eta: 0:00:35 time: 0.2884 data_time: 0.0335 memory: 1381 grad_norm: 0.1274 loss: 0.4391 recall@thr=0.5: 0.8571 prec@thr=0.5: 0.8571 recall@top3: 0.8571 prec@top3: 0.2857 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.4391\n",
+ "06/15 06:57:13 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 7 epochs\n",
+ "06/15 06:57:17 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [7][ 20/120] eta: 0:00:11 time: 0.1193 data_time: 0.0693 memory: 466 \n",
+ "06/15 06:57:19 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [7][ 40/120] eta: 0:00:09 time: 0.1188 data_time: 0.0670 memory: 466 \n",
+ "06/15 06:57:23 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [7][ 60/120] eta: 0:00:08 time: 0.1645 data_time: 0.1114 memory: 466 \n",
+ "06/15 06:57:25 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [7][ 80/120] eta: 0:00:05 time: 0.1391 data_time: 0.0850 memory: 466 \n",
+ "06/15 06:57:27 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [7][100/120] eta: 0:00:02 time: 0.1104 data_time: 0.0585 memory: 466 \n",
+ "06/15 06:57:30 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [7][120/120] eta: 0:00:00 time: 0.1025 data_time: 0.0512 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 20.79\n",
+ "aerobic split jump 20.11\n",
+ "aerobic scissors leap 84.84\n",
+ "aerobic turn 78.58\n",
+ "mAP 51.08\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 80.00\n",
+ "aerobic turn 20.00\n",
+ "mAP 25.00\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 45.00\n",
+ "aerobic turn 0.00\n",
+ "mAP 11.25\n",
+ "06/15 06:57:30 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [7][120/120] mAP/frameAP: 51.0794 mAP/v_map@0.2: 25.0000 mAP/v_map@0.5: 11.2500 mAP/v_map_0.05:0.45: 22.5000 mAP/v_map_0.10:0.90: 14.0741 mAP/v_map_0.50:0.95: 6.9167 data_time: 0.0735 time: 0.1255\n",
+ "06/15 06:57:36 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [8][ 20/118] lr: 1.0000e-04 eta: 0:00:29 time: 0.2894 data_time: 0.0322 memory: 1381 grad_norm: 0.1227 loss: 0.3286 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3286\n",
+ "06/15 06:57:44 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [8][ 40/118] lr: 1.0000e-04 eta: 0:00:23 time: 0.4105 data_time: 0.1257 memory: 1381 grad_norm: 0.1948 loss: 0.3202 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3202\n",
+ "06/15 06:57:50 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [8][ 60/118] lr: 1.0000e-04 eta: 0:00:17 time: 0.3095 data_time: 0.0537 memory: 1381 grad_norm: 0.7997 loss: 0.2428 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2428\n",
+ "06/15 06:57:56 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [8][ 80/118] lr: 1.0000e-04 eta: 0:00:11 time: 0.2918 data_time: 0.0330 memory: 1381 grad_norm: 0.8157 loss: 0.3045 recall@thr=0.5: 1.0000 prec@thr=0.5: 1.0000 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.3045\n",
+ "06/15 06:58:03 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [8][100/118] lr: 1.0000e-04 eta: 0:00:05 time: 0.3443 data_time: 0.0786 memory: 1381 grad_norm: 0.0966 loss: 0.2605 recall@thr=0.5: 0.9375 prec@thr=0.5: 0.9375 recall@top3: 1.0000 prec@top3: 0.3333 recall@top5: 1.0000 prec@top5: 0.2000 loss_action_cls: 0.2605\n",
+ "06/15 06:58:08 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: slowonly_k400_multisports_20230615_065057\n",
+ "06/15 06:58:08 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [8][118/118] lr: 1.0000e-04 eta: 0:00:00 time: 0.2611 data_time: 0.0148 memory: 1381 grad_norm: 0.3034 loss: 0.2694 recall@thr=0.5: 0.9231 prec@thr=0.5: 0.9231 recall@top3: 0.9231 prec@top3: 0.3077 recall@top5: 0.9231 prec@top5: 0.1846 loss_action_cls: 0.2694\n",
+ "06/15 06:58:08 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 8 epochs\n",
+ "06/15 06:58:12 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [8][ 20/120] eta: 0:00:14 time: 0.1433 data_time: 0.0869 memory: 466 \n",
+ "06/15 06:58:15 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [8][ 40/120] eta: 0:00:12 time: 0.1664 data_time: 0.1160 memory: 466 \n",
+ "06/15 06:58:18 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [8][ 60/120] eta: 0:00:08 time: 0.1269 data_time: 0.0772 memory: 466 \n",
+ "06/15 06:58:20 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [8][ 80/120] eta: 0:00:05 time: 0.0951 data_time: 0.0455 memory: 466 \n",
+ "06/15 06:58:22 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [8][100/120] eta: 0:00:02 time: 0.1144 data_time: 0.0630 memory: 466 \n",
+ "06/15 06:58:24 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [8][120/120] eta: 0:00:00 time: 0.1028 data_time: 0.0530 memory: 466 \n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluate aerobic kick jump\n",
+ "do not evaluate aerobic off axis jump\n",
+ "do not evaluate aerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluate aerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluate basketball save\n",
+ "do not evaluate basketball jump ball\n",
+ "frameAP_0.5\n",
+ "\n",
+ "aerobic straight jump 15.29\n",
+ "aerobic split jump 20.74\n",
+ "aerobic scissors leap 86.38\n",
+ "aerobic turn 80.98\n",
+ "mAP 50.85\n",
+ "\u001b[2Klinking tubes... \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
+ "\u001b[?25hno such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.2\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 80.00\n",
+ "aerobic turn 20.00\n",
+ "mAP 25.00\n",
+ "no such label 0 aerobic push up\n",
+ "no such label 1 aerobic explosive push up\n",
+ "no such label 2 aerobic explosive support\n",
+ "no such label 3 aerobic leg circle\n",
+ "no such label 4 aerobic helicopter\n",
+ "no such label 5 aerobic support\n",
+ "no such label 6 aerobic v support\n",
+ "no such label 7 aerobic horizontal support\n",
+ "no such label 9 aerobic illusion\n",
+ "no such label 10 aerobic bent leg(s) jump\n",
+ "no such label 11 aerobic pike jump\n",
+ "no such label 12 aerobic straddle jump\n",
+ "do not evaluateaerobic kick jump\n",
+ "do not evaluateaerobic off axis jump\n",
+ "do not evaluateaerobic butterfly jump\n",
+ "no such label 18 aerobic split\n",
+ "do not evaluateaerobic balance turn\n",
+ "no such label 21 volleyball serve\n",
+ "no such label 22 volleyball block\n",
+ "no such label 23 volleyball first pass\n",
+ "no such label 24 volleyball defend\n",
+ "no such label 25 volleyball protect\n",
+ "no such label 26 volleyball second pass\n",
+ "no such label 27 volleyball adjust\n",
+ "no such label 28 volleyball save\n",
+ "no such label 29 volleyball second attack\n",
+ "no such label 30 volleyball spike\n",
+ "no such label 31 volleyball dink\n",
+ "no such label 32 volleyball no offensive attack\n",
+ "no such label 33 football shoot\n",
+ "no such label 34 football long pass\n",
+ "no such label 35 football short pass\n",
+ "no such label 36 football through pass\n",
+ "no such label 37 football cross\n",
+ "no such label 38 football dribble\n",
+ "no such label 39 football trap\n",
+ "no such label 40 football throw\n",
+ "no such label 41 football diving\n",
+ "no such label 42 football tackle\n",
+ "no such label 43 football steal\n",
+ "no such label 44 football clearance\n",
+ "no such label 45 football block\n",
+ "no such label 46 football press\n",
+ "no such label 47 football aerial duels\n",
+ "no such label 48 basketball pass\n",
+ "no such label 49 basketball drive\n",
+ "no such label 50 basketball dribble\n",
+ "no such label 51 basketball 3-point shot\n",
+ "no such label 52 basketball 2-point shot\n",
+ "no such label 53 basketball free throw\n",
+ "no such label 54 basketball block\n",
+ "no such label 55 basketball offensive rebound\n",
+ "no such label 56 basketball defensive rebound\n",
+ "no such label 57 basketball pass steal\n",
+ "no such label 58 basketball dribble steal\n",
+ "no such label 59 basketball interfere shot\n",
+ "no such label 60 basketball pick-and-roll defensive\n",
+ "no such label 61 basketball sag\n",
+ "no such label 62 basketball screen\n",
+ "no such label 63 basketball pass-inbound\n",
+ "do not evaluatebasketball save\n",
+ "do not evaluatebasketball jump ball\n",
+ "VideoAP_0.5\n",
+ "\n",
+ "aerobic straight jump 0.00\n",
+ "aerobic split jump 0.00\n",
+ "aerobic scissors leap 45.00\n",
+ "aerobic turn 20.00\n",
+ "mAP 16.25\n",
+ "06/15 06:58:25 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [8][120/120] mAP/frameAP: 50.8487 mAP/v_map@0.2: 25.0000 mAP/v_map@0.5: 16.2500 mAP/v_map_0.05:0.45: 23.0556 mAP/v_map_0.10:0.90: 15.1852 mAP/v_map_0.50:0.95: 8.4167 data_time: 0.0732 time: 0.1244\n",
+ "\u001b[32mTraining finished successfully. \u001b[0m\n"
+ ]
+ }
+ ],
+ "source": [
+ "# 使用 MIM 训练模型\n",
+ "!mim train mmaction2 configs/slowonly_k400_multisports.py \\\n",
+ " --work-dir work_dirs/stad_model/"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "HCg6C9HYxRAt"
+ },
+ "source": [
+ "## 4. 时空行为检测模型推理\n",
+ "\n",
+ "训练得到检测模型和时空行为检测模型后，我们可以利用时空行为检测 demo 进行推理，可视化模型效果。\n",
+ "\n",
+ "由于 tutorial 中使用的训练数据较少，模型性能较差，所以可视化时使用预先训练好的模型。"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WW5-IJ7IxRAu"
+ },
+ "source": [
+ "###"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "FofW_5RoxRAu",
+ "outputId": "91217660-946d-48ab-f663-b0f7f2d6a6f6"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "ALSA lib confmisc.c:767:(parse_card) cannot find card '0'\n",
+ "ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory\n",
+ "ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings\n",
+ "ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory\n",
+ "ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name\n",
+ "ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory\n",
+ "ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory\n",
+ "ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM default\n",
+ "ALSA lib confmisc.c:767:(parse_card) cannot find card '0'\n",
+ "ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory\n",
+ "ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings\n",
+ "ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory\n",
+ "ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name\n",
+ "ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory\n",
+ "ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory\n",
+ "ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM default\n",
+ "Loads checkpoint by local backend from path: work_dirs/det_model/epoch_2.pth\n",
+ "Performing Human Detection for each frame\n",
+ "[>>] 99/99, 6.8 task/s, elapsed: 15s, ETA: 0s\n",
+ "Loads checkpoint by http backend from path: https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-a1ca5e76.pth\n",
+ "Performing SpatioTemporal Action Detection for each clip\n",
+ "[>>] 99/99, 16.6 task/s, elapsed: 6s, ETA: 0sPerforming visualization\n",
+ "Moviepy - Building video data/demo_spatiotemporal_det.mp4.\n",
+ "Moviepy - Writing video data/demo_spatiotemporal_det.mp4\n",
+ "\n",
+ "Moviepy - Done !\n",
+ "Moviepy - video ready data/demo_spatiotemporal_det.mp4\n"
+ ]
+ }
+ ],
+ "source": [
+ "!python ../../demo/demo_spatiotemporal_det.py \\\n",
+ " data/multisports/test/aerobic_gymnastics/v_7G_IpU0FxLU_c001.mp4 \\\n",
+ " data/demo_spatiotemporal_det.mp4 \\\n",
+ " --config configs/slowonly_k400_multisports.py \\\n",
+ " --checkpoint https://download.openmmlab.com/mmaction/v1.0/detection/slowonly/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb/slowonly_kinetics400-pretrained-r50_8xb16-4x16x1-8e_multisports-rgb_20230320-a1ca5e76.pth \\\n",
+ " --det-config configs/faster-rcnn_r50-caffe_fpn_ms-1x_coco_ms_person.py \\\n",
+ " --det-checkpoint work_dirs/det_model/epoch_2.pth \\\n",
+ " --det-score-thr 0.85 \\\n",
+ " --action-score-thr 0.8 \\\n",
+ " --label-map ../../tools/data/multisports/label_map.txt \\\n",
+ " --predict-stepsize 8 \\\n",
+ " --output-stepsize 1 \\\n",
+ " --output-fps 24"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 741
+ },
+ "id": "677FUWFRxRAv",
+ "outputId": "f702d544-3492-494c-af81-9e90f43d6b6c"
+ },
+ "outputs": [],
+ "source": [
+ "# Show Video\n",
+ "import moviepy.editor\n",
+ "moviepy.editor.ipython_display(\"data/demo_spatiotemporal_det.mp4\")"
+ ]
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "gpuType": "T4",
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.0"
+ },
+ "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/projects/stad_tutorial/tools/convert_proposals.py b/projects/stad_tutorial/tools/convert_proposals.py
new file mode 100644
index 0000000000..4e43c6dd89
--- /dev/null
+++ b/projects/stad_tutorial/tools/convert_proposals.py
@@ -0,0 +1,59 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+import argparse
+
+import numpy as np
+from mmengine import dump, load, track_iter_progress
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ '--det_test_result',
+ default='data/multisports/annotations/ms_det_proposals.pkl')
+ parser.add_argument(
+ '--stad_gt',
+ help='spatio-temporal action detection ground truth file',
+ default='data/multisports/annotations/multisports_GT.pkl')
+ parser.add_argument(
+ '--out_result',
+ default='data/multisports/annotations/multisports_proposals.pkl')
+ args = parser.parse_args()
+ return args
+
+
+def dump_det_result(args):
+ print('loading test result...')
+ det_result = load(args.det_test_result)
+ stad_gt = load(args.stad_gt)
+ train_list = stad_gt['train_videos'][0]
+ val_list = stad_gt['test_videos'][0]
+ train_bbox_result = {}
+ val_bbox_result = {}
+ for sample in track_iter_progress(det_result):
+ bboxes = sample['pred_instances']['bboxes']
+ scores = sample['pred_instances']['scores']
+ h, w = sample['ori_shape']
+ bboxes[:, ::2] /= w
+ bboxes[:, 1::2] /= h
+ img_path = sample['img_path']
+ frm_key_list = img_path.split('.jpg')[0].split('/')
+ frm_key = ','.join([
+ f'{frm_key_list[-3]}/{frm_key_list[-2]}.mp4',
+ f'{int(frm_key_list[-1]):04d}'
+ ])
+ bbox = np.concatenate([bboxes, scores[:, None]], axis=1)
+
+ vid_key = '/'.join(frm_key_list[-3:-1])
+ if vid_key in train_list:
+ train_bbox_result[frm_key] = bbox
+ elif vid_key in val_list:
+ val_bbox_result[frm_key] = bbox
+ else:
+ raise KeyError(vid_key)
+ dump(train_bbox_result, args.out_result[:-4] + '_train.pkl')
+ dump(val_bbox_result, args.out_result[:-4] + '_val.pkl')
+
+
+if __name__ == '__main__':
+ args = parse_args()
+ dump_det_result(args)
diff --git a/projects/stad_tutorial/tools/generate_mmdet_anno.py b/projects/stad_tutorial/tools/generate_mmdet_anno.py
new file mode 100644
index 0000000000..7b4ba62994
--- /dev/null
+++ b/projects/stad_tutorial/tools/generate_mmdet_anno.py
@@ -0,0 +1,74 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+import argparse
+from collections import defaultdict
+
+from mmengine import dump, load
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ 'stad_anno', help='spatiotemporal action detection anno path')
+ parser.add_argument('det_path', help='output detection anno path')
+ args = parser.parse_args()
+ return args
+
+
+def generate_mmdet_coco_anno(args):
+ ori_anno = load(args.stad_anno)
+ train_videos = ori_anno['train_videos']
+ val_videos = ori_anno['test_videos']
+ videos = {'train': train_videos, 'val': val_videos}
+ for split in ['train', 'val']:
+ img_id = 0
+ bbox_id = 0
+ img_list = []
+ anno_list = []
+ for vid in videos[split][0]:
+ vid_tubes = ori_anno['gttubes'][vid]
+ height, width = ori_anno['resolution'][vid]
+ frm2bbox = defaultdict(list)
+ for label_idx, tube_list in vid_tubes.items():
+ for tube in tube_list:
+ for frm_anno in tube:
+ frm_idx, bbox = frm_anno[0], frm_anno[1:]
+ frm2bbox[frm_idx].append({'label': 0, 'bbox': bbox})
+ for frm_idx, frm_bboxes in frm2bbox.items():
+ img_path = f'{vid}/{int(frm_idx):05d}.jpg'
+ img_instance = {
+ 'file_name': img_path,
+ 'height': height,
+ 'width': width,
+ 'id': img_id
+ }
+ img_list.append(img_instance)
+
+ for bbox_info in frm_bboxes:
+ label = bbox_info['label']
+ x1, y1, x2, y2 = bbox_info['bbox']
+ bbox = [x1, y1, x2 - x1, y2 - y1]
+ anno_instance = {
+ 'area': bbox[2] * bbox[3],
+ 'image_id': img_id,
+ 'bbox': bbox,
+ 'category_id': label,
+ 'iscrowd': 0,
+ 'id': bbox_id
+ }
+ anno_list.append(anno_instance)
+ bbox_id += 1
+ img_id += 1
+ total_anno = {
+ 'images': img_list,
+ 'annotations': anno_list,
+ 'categories': [{
+ 'id': 0,
+ 'name': 'person'
+ }],
+ }
+ dump(total_anno, args.det_path[:-5] + f'_{split}' + args.det_path[-5:])
+
+
+if __name__ == '__main__':
+ args = parse_args()
+ generate_mmdet_coco_anno(args)
diff --git a/projects/stad_tutorial/tools/generate_rgb.py b/projects/stad_tutorial/tools/generate_rgb.py
new file mode 100644
index 0000000000..61952f4b21
--- /dev/null
+++ b/projects/stad_tutorial/tools/generate_rgb.py
@@ -0,0 +1,40 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+import os
+import os.path as osp
+
+import cv2
+
+src_dir = 'data/multisports/trainval'
+target_dir = 'data/multisports/rawframes'
+
+sport_list = ['aerobic_gymnastics']
+for sport in sport_list:
+ video_root = osp.join(src_dir, sport)
+ if not osp.exists(video_root):
+ print('No {} video dir to generate rgb images.'.format(video_root))
+ continue
+ print('Will generate {} rgb dir for {}.'.format(
+ len(os.listdir(video_root)), osp.basename(sport)))
+ for clip_name in os.listdir(video_root):
+ mp4_path = osp.join(video_root, clip_name)
+ save_dir = osp.join(target_dir, sport, clip_name[:-4])
+ if not osp.exists(save_dir):
+ os.makedirs(save_dir)
+ cap = cv2.VideoCapture(mp4_path)
+ fps = cap.get(cv2.CAP_PROP_FPS)
+ size = (int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)),
+ int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)))
+ fourcc = cv2.VideoWriter_fourcc(*'I420')
+ ii = 1
+ while (cap.isOpened()):
+ ret, frame = cap.read()
+ aa = str(ii)
+ s = aa.zfill(5)
+ image_name = osp.join(save_dir + '/' + s + '.jpg')
+ if ret is True:
+ cv2.imwrite(image_name, frame)
+ else:
+ break
+ ii = ii + 1
+ cap.release()
+ print('Generate {} rgb dir successfully.'.format(clip_name[:-4]))
diff --git a/projects/stad_tutorial/tools/images2coco.py b/projects/stad_tutorial/tools/images2coco.py
new file mode 100644
index 0000000000..18a2e4bb0d
--- /dev/null
+++ b/projects/stad_tutorial/tools/images2coco.py
@@ -0,0 +1,111 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+import argparse
+import os
+
+from mmengine.fileio import dump, list_from_file
+from mmengine.utils import mkdir_or_exist, scandir, track_parallel_progress
+from PIL import Image
+
+
+def parse_args():
+ parser = argparse.ArgumentParser(
+ description='Convert images to coco format without annotations')
+ parser.add_argument('img_path', help='The root path of images')
+ parser.add_argument(
+ 'classes', type=str, help='The text file name of storage class list')
+ parser.add_argument(
+ 'out',
+ type=str,
+ help='The output annotation json file name, The save dir is in the '
+ 'same directory as img_path')
+ parser.add_argument(
+ '-e',
+ '--exclude-extensions',
+ type=str,
+ nargs='+',
+ help='The suffix of images to be excluded, such as "png" and "bmp"')
+ args = parser.parse_args()
+ return args
+
+
+def get_img_info(args):
+ path, image_path, exclude_extensions = args
+ if exclude_extensions is None or (
+ exclude_extensions is not None
+ and not image_path.lower().endswith(exclude_extensions)):
+ # image_path =
+ img_pillow = Image.open(os.path.join(path, image_path))
+ img_info = {
+ 'filename': image_path,
+ 'width': img_pillow.width,
+ 'height': img_pillow.height,
+ }
+ return img_info
+
+
+def collect_image_infos(path, exclude_extensions=None):
+ img_infos = []
+
+ images_generator = scandir(path, recursive=True)
+
+ img_infos = track_parallel_progress(
+ get_img_info, [(path, image_path, exclude_extensions)
+ for image_path in images_generator],
+ nproc=64)
+
+ return img_infos
+
+
+def cvt_to_coco_json(img_infos, classes):
+ image_id = 0
+ coco = dict()
+ coco['images'] = []
+ coco['type'] = 'instance'
+ coco['categories'] = []
+ coco['annotations'] = []
+ image_set = set()
+
+ for category_id, name in enumerate(classes):
+ category_item = dict()
+ category_item['supercategory'] = str('none')
+ category_item['id'] = int(category_id)
+ category_item['name'] = str(name)
+ coco['categories'].append(category_item)
+
+ for img_dict in img_infos:
+ file_name = img_dict['filename']
+ assert file_name not in image_set
+ image_item = dict()
+ image_item['id'] = int(image_id)
+ image_item['file_name'] = str(file_name)
+ image_item['height'] = int(img_dict['height'])
+ image_item['width'] = int(img_dict['width'])
+ coco['images'].append(image_item)
+ image_set.add(file_name)
+
+ image_id += 1
+ return coco
+
+
+def main():
+ args = parse_args()
+ assert args.out.endswith(
+ 'json'), 'The output file name must be json suffix'
+
+ # 1 load image list info
+ img_infos = collect_image_infos(args.img_path, args.exclude_extensions)
+
+ # 2 convert to coco format data
+ classes = list_from_file(args.classes)
+ coco_info = cvt_to_coco_json(img_infos, classes)
+
+ # 3 dump
+ save_dir = os.path.join(args.img_path, '..', 'annotations')
+ mkdir_or_exist(save_dir)
+ save_path = os.path.join(save_dir, args.out)
+ dump(coco_info, save_path)
+ print(f'save json file: {save_path}')
+
+
+if __name__ == '__main__':
+ main()
diff --git a/requirements/docs.txt b/requirements/docs.txt
index 828ae1f8e2..cc4b6aedb1 100644
--- a/requirements/docs.txt
+++ b/requirements/docs.txt
@@ -1,10 +1,14 @@
-docutils==0.16.0
+docutils==0.18.1
einops
+modelindex
myst-parser
opencv-python
-e git+https://github.com/open-mmlab/pytorch_sphinx_theme.git#egg=pytorch_sphinx_theme
scipy
-sphinx==4.0.2
+sphinx==6.1.3
+sphinx-notfound-page
+sphinx-tabs
sphinx_copybutton
sphinx_markdown_tables
-sphinx_rtd_theme==0.5.2
+sphinxcontrib-jquery
+tabulate
diff --git a/requirements/optional.txt b/requirements/optional.txt
index d2204c6441..ff4985ca37 100644
--- a/requirements/optional.txt
+++ b/requirements/optional.txt
@@ -1,10 +1,10 @@
av>=9.0
future
-fvcore
imgaug
librosa
lmdb
moviepy
+openai-clip
packaging
pims
PyTurboJPEG
diff --git a/setup.cfg b/setup.cfg
index 19a6462ca7..c1cc13df6e 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -19,3 +19,7 @@ known_first_party = mmaction
known_third_party = cv2,decord,einops,joblib,matplotlib,mmcv,numpy,pandas,pytest,pytorch_sphinx_theme,scipy,seaborn,titlecase,torch,webcolors
no_lines_before = STDLIB,LOCALFOLDER
default_section = THIRDPARTY
+
+[flake8]
+per-file-ignores =
+ mmaction/configs/*:F401,F403,F405
diff --git a/setup.py b/setup.py
index c8a8f8e0f2..4776e54145 100644
--- a/setup.py
+++ b/setup.py
@@ -119,7 +119,7 @@ def add_mim_extension():
else:
return
- filenames = ['tools', 'configs', 'model-index.yml']
+ filenames = ['tools', 'configs', 'model-index.yml', 'dataset-index.yml']
repo_path = osp.dirname(__file__)
mim_path = osp.join(repo_path, 'mmaction', '.mim')
os.makedirs(mim_path, exist_ok=True)
diff --git a/tests/apis/test_inference.py b/tests/apis/test_inference.py
index 6ecd89cb67..1b004943f7 100644
--- a/tests/apis/test_inference.py
+++ b/tests/apis/test_inference.py
@@ -1,13 +1,17 @@
# Copyright (c) OpenMMLab. All rights reserved.
import os.path as osp
from pathlib import Path
+from tempfile import TemporaryDirectory
from unittest import TestCase
import torch
+from mmengine.testing import assert_dict_has_keys
from parameterized import parameterized
-from mmaction.apis import inference_recognizer, init_recognizer
+from mmaction.apis import (detection_inference, inference_recognizer,
+ init_recognizer, pose_inference)
from mmaction.structures import ActionDataSample
+from mmaction.utils import frame_extract, get_str_type
class TestInference(TestCase):
@@ -55,7 +59,7 @@ def test_inference_recognizer(self, config, video_path, devices):
model = init_recognizer(config_file, device=device)
for ops in model.cfg.test_pipeline:
- if ops['type'] in ('TenCrop', 'ThreeCrop'):
+ if get_str_type(ops['type']) in ('TenCrop', 'ThreeCrop'):
# Use CenterCrop to reduce memory in order to pass CI
ops['type'] = 'CenterCrop'
@@ -63,3 +67,89 @@ def test_inference_recognizer(self, config, video_path, devices):
self.assertIsInstance(result, ActionDataSample)
self.assertTrue(result.pred_scores.item.shape, (400, ))
+
+ def test_detection_inference(self):
+ from mmdet.apis import init_detector
+ from mmdet.structures import DetDataSample
+
+ for device in ('cpu', 'cuda'):
+ if device == 'cuda' and not torch.cuda.is_available():
+ # Skip the test if cuda is required but unavailable
+ continue
+ project_dir = osp.abspath(osp.dirname(osp.dirname(__file__)))
+ project_dir = osp.join(project_dir, '..')
+ det_config = 'demo/demo_configs/faster-rcnn_r50_fpn_2x_coco_infer.py' # noqa: E501
+ det_ckpt = 'http://download.openmmlab.com/mmdetection/' \
+ 'v2.0/faster_rcnn/faster_rcnn_r50_fpn_2x_coco/faster_rcnn_r50_fpn_2x_coco_bbox_mAP-0.384_20200504_210434-a5d8aa15.pth' # noqa: E501
+ video_path = 'demo/demo_skeleton.mp4'
+ video_path = osp.join(project_dir, video_path)
+ config_file = osp.join(project_dir, det_config)
+ with TemporaryDirectory() as tmpdir:
+ frm_paths, _ = frame_extract(video_path, out_dir=tmpdir)
+ # skip remaining frames to speed up ut
+ frm_paths = frm_paths[:10]
+ results, data_samples = detection_inference(
+ config_file, det_ckpt, frm_paths, device=device)
+ self.assertTrue(results[0].shape, (4, ))
+ self.assertIsInstance(data_samples[0], DetDataSample)
+ # test with_score
+ results, data_samples = detection_inference(
+ config_file,
+ det_ckpt,
+ frm_paths,
+ with_score=True,
+ device=device)
+ self.assertTrue(results[0].shape, (5, ))
+ # test inference with model object
+ model = init_detector(
+ config=det_config, checkpoint=det_ckpt, device=device)
+ results, data_samples = detection_inference(
+ model, None, frm_paths, device=device)
+ self.assertTrue(results[0].shape, (4, ))
+ self.assertIsInstance(data_samples[0], DetDataSample)
+
+ def test_pose_inference(self):
+ from mmpose.apis import init_model
+ from mmpose.structures import PoseDataSample
+
+ for device in ('cpu', 'cuda'):
+ if device == 'cuda' and not torch.cuda.is_available():
+ # Skip the test if cuda is required but unavailable
+ continue
+ project_dir = osp.abspath(osp.dirname(osp.dirname(__file__)))
+ project_dir = osp.join(project_dir, '..')
+ det_config = 'demo/demo_configs/faster-rcnn_r50_fpn_2x_coco_infer.py' # noqa: E501
+ det_ckpt = 'http://download.openmmlab.com/mmdetection/' \
+ 'v2.0/faster_rcnn/faster_rcnn_r50_fpn_2x_coco/faster_rcnn_r50_fpn_2x_coco_bbox_mAP-0.384_20200504_210434-a5d8aa15.pth' # noqa: E501
+ pose_config = 'demo/demo_configs/' \
+ 'td-hm_hrnet-w32_8xb64-210e_coco-256x192_infer.py'
+ pose_ckpt = 'https://download.openmmlab.com/mmpose/top_down/' \
+ 'hrnet/hrnet_w32_coco_256x192-c78dce93_20200708.pth'
+ video_path = 'demo/demo_skeleton.mp4'
+ video_path = osp.join(project_dir, video_path)
+ pose_config = osp.join(project_dir, pose_config)
+ with TemporaryDirectory() as tmpdir:
+ frm_paths, _ = frame_extract(video_path, out_dir=tmpdir)
+ # skip remaining frames to speed up ut
+ frm_paths = frm_paths[:10]
+ det_results, _ = detection_inference(
+ det_config, det_ckpt, frm_paths, device=device)
+
+ results, data_samples = pose_inference(
+ pose_config,
+ pose_ckpt,
+ frm_paths,
+ det_results,
+ device=device)
+ assert_dict_has_keys(results[0], ('keypoints', 'bbox_scores',
+ 'bboxes', 'keypoint_scores'))
+ self.assertIsInstance(data_samples[0], PoseDataSample)
+
+ # test inference with model object
+ model = init_model(
+ config=pose_config, checkpoint=pose_ckpt, device=device)
+ results, data_samples = pose_inference(
+ model, None, frm_paths, det_results, device=device)
+ assert_dict_has_keys(results[0], ('keypoints', 'bbox_scores',
+ 'bboxes', 'keypoint_scores'))
+ self.assertIsInstance(data_samples[0], PoseDataSample)
diff --git a/tests/data/annotations/video_text_test_list.json b/tests/data/annotations/video_text_test_list.json
new file mode 100644
index 0000000000..99e968fe88
--- /dev/null
+++ b/tests/data/annotations/video_text_test_list.json
@@ -0,0 +1 @@
+{"test.mp4": ["A person is cleaning a swimming pool", "A person is using a cleaning machine to clean the swimming pool"]}
\ No newline at end of file
diff --git a/tests/data/eval_multisports/data_samples.pkl b/tests/data/eval_multisports/data_samples.pkl
new file mode 100644
index 0000000000..f1341c9bdf
Binary files /dev/null and b/tests/data/eval_multisports/data_samples.pkl differ
diff --git a/tests/data/eval_multisports/gt.pkl b/tests/data/eval_multisports/gt.pkl
new file mode 100644
index 0000000000..13bc68d0a1
Binary files /dev/null and b/tests/data/eval_multisports/gt.pkl differ
diff --git a/tests/data/multisports_dataset/multisports_proposals_sample.pkl b/tests/data/multisports_dataset/multisports_proposals_sample.pkl
new file mode 100644
index 0000000000..524b854599
Binary files /dev/null and b/tests/data/multisports_dataset/multisports_proposals_sample.pkl differ
diff --git a/tests/data/multisports_dataset/multisports_sample.csv b/tests/data/multisports_dataset/multisports_sample.csv
new file mode 100644
index 0000000000..d6bbe51e8a
--- /dev/null
+++ b/tests/data/multisports_dataset/multisports_sample.csv
@@ -0,0 +1,9 @@
+aerobic_gymnastics/v_aqMgwPExjD0_c001.mp4,377,0.706,0.439,0.794,0.811,11,0
+aerobic_gymnastics/v_aqMgwPExjD0_c001.mp4,378,0.689,0.438,0.794,0.804,11,0
+aerobic_gymnastics/v_aqMgwPExjD0_c001.mp4,379,0.672,0.419,0.802,0.797,11,0
+aerobic_gymnastics/v_aqMgwPExjD0_c001.mp4,380,0.680,0.361,0.791,0.783,11,0
+aerobic_gymnastics/v_yaKOumdXwbU_c019.mp4,443,0.109,0.669,0.345,0.768,1,0
+aerobic_gymnastics/v_yaKOumdXwbU_c019.mp4,444,0.112,0.668,0.347,0.767,1,0
+aerobic_gymnastics/v_yaKOumdXwbU_c019.mp4,445,0.115,0.663,0.350,0.761,1,0
+aerobic_gymnastics/v_yaKOumdXwbU_c019.mp4,446,0.117,0.644,0.352,0.757,1,0
+aerobic_gymnastics/v_yaKOumdXwbU_c019.mp4,447,0.118,0.636,0.352,0.754,1,0
\ No newline at end of file
diff --git a/tests/datasets/base.py b/tests/datasets/base.py
index 3748925c8d..d859d896e4 100644
--- a/tests/datasets/base.py
+++ b/tests/datasets/base.py
@@ -44,6 +44,8 @@ def setup_class(cls):
'video_test_list.txt')
cls.video_ann_file_multi_label = osp.join(
cls.ann_file_prefix, 'video_test_list_multi_label.txt')
+ cls.video_text_ann_file = osp.join(cls.ann_file_prefix,
+ 'video_text_test_list.json')
cls.pose_ann_file = osp.join(cls.ann_file_prefix, 'sample.pkl')
# pipeline configuration
@@ -140,6 +142,17 @@ def setup_class(cls):
dict(type='OpenCVDecode')
]
+ cls.video_text_pipeline = [
+ dict(type='OpenCVInit'),
+ dict(
+ type='SampleFrames',
+ clip_len=32,
+ frame_interval=2,
+ num_clips=1),
+ dict(type='OpenCVDecode'),
+ dict(type='CLIPTokenize')
+ ]
+
cls.hvu_categories = [
'action', 'attribute', 'concept', 'event', 'object', 'scene'
]
diff --git a/tests/datasets/test_ava_dataset.py b/tests/datasets/test_ava_dataset.py
index a75dd512b6..1ae4833858 100644
--- a/tests/datasets/test_ava_dataset.py
+++ b/tests/datasets/test_ava_dataset.py
@@ -4,7 +4,7 @@
import mmengine
import numpy as np
from mmengine.testing import assert_dict_has_keys
-from numpy.testing import assert_array_equal
+from numpy.testing import assert_array_almost_equal, assert_array_equal
from mmaction.datasets import AVADataset, AVAKineticsDataset
from mmaction.utils import register_all_modules
@@ -23,7 +23,7 @@ def setup_class(cls):
cls.proposal_file = osp.join(cls.data_prefix,
'ava_proposals_sample.pkl')
cls.pipeline = [
- dict(dict(type='SampleAVAFrames', clip_len=32, frame_interval=2))
+ dict(type='SampleAVAFrames', clip_len=32, frame_interval=2)
]
cls.proposal = mmengine.load(cls.proposal_file)
@@ -31,8 +31,8 @@ def test_ava_dataset(self):
register_all_modules()
ava_dataset = AVADataset(
self.ann_file,
- self.exclude_file,
self.pipeline,
+ self.exclude_file,
self.label_file,
data_prefix={'img': self.data_prefix},
proposal_file=self.proposal_file)
@@ -40,8 +40,8 @@ def test_ava_dataset(self):
# custom classes
ava_dataset = AVADataset(
self.ann_file,
- self.exclude_file,
self.pipeline,
+ self.exclude_file,
label_file=self.label_file,
custom_classes=[17, 79],
num_classes=3,
@@ -55,16 +55,16 @@ def test_ava_dataset(self):
ava_dataset = AVADataset(
self.ann_file,
- None,
self.pipeline,
+ None,
self.label_file,
data_prefix={'img': self.data_prefix},
proposal_file=self.proposal_file)
ava_dataset = AVADataset(
self.ann_file,
- None,
self.pipeline,
+ None,
self.label_file,
test_mode=True,
data_prefix={'img': self.data_prefix},
@@ -84,8 +84,8 @@ def test_ava_pipeline(self):
ava_dataset = AVADataset(
self.ann_file,
- self.exclude_file,
self.pipeline,
+ self.exclude_file,
self.label_file,
data_prefix={'img': self.data_prefix},
proposal_file=self.proposal_file)
@@ -94,7 +94,7 @@ def test_ava_pipeline(self):
assert result['filename_tmpl'] == 'img_{:05}.jpg'
assert result['modality'] == 'RGB'
- assert result['start_index'] == 0
+ assert result['start_index'] == 1
assert result['timestamp_start'] == 900
assert result['timestamp_end'] == 1800
assert_array_equal(result['proposals'],
@@ -107,8 +107,8 @@ def test_ava_pipeline(self):
ava_dataset = AVADataset(
self.ann_file,
- None,
self.pipeline,
+ None,
self.label_file,
test_mode=True,
data_prefix={'img': self.data_prefix},
@@ -117,11 +117,130 @@ def test_ava_pipeline(self):
result = ava_dataset[0]
assert result['filename_tmpl'] == 'img_{:05}.jpg'
assert result['modality'] == 'RGB'
- assert result['start_index'] == 0
+ assert result['start_index'] == 1
assert result['timestamp_start'] == 900
assert result['timestamp_end'] == 1800
+class TestMultiSportsDataset:
+
+ @classmethod
+ def setup_class(cls):
+ cls.data_prefix = osp.normpath(
+ osp.join(
+ osp.dirname(__file__), './../data', 'multisports_dataset'))
+ cls.ann_file = osp.join(cls.data_prefix, 'multisports_sample.csv')
+ cls.proposal_file = osp.join(cls.data_prefix,
+ 'multisports_proposals_sample.pkl')
+ cls.pipeline = [
+ dict(type='DecordInit'),
+ dict(type='SampleAVAFrames', clip_len=32, frame_interval=2),
+ dict(type='DecordDecode')
+ ]
+ cls.proposal = mmengine.load(cls.proposal_file)
+
+ def test_multisports_dataset(self):
+ register_all_modules()
+ ava_dataset = AVADataset(
+ self.ann_file,
+ self.pipeline,
+ data_prefix={'img': self.data_prefix},
+ proposal_file=self.proposal_file,
+ use_frames=False,
+ timestamp_start=1,
+ start_index=0,
+ multilabel=False,
+ fps=1)
+
+ ava_dataset = AVADataset(
+ self.ann_file,
+ self.pipeline,
+ test_mode=True,
+ data_prefix={'img': self.data_prefix},
+ proposal_file=self.proposal_file,
+ use_frames=False,
+ timestamp_start=1,
+ start_index=0,
+ multilabel=False,
+ fps=1)
+
+ del ava_dataset
+
+ def test_ava_pipeline(self):
+ register_all_modules()
+ target_keys = [
+ 'filename', 'video_id', 'timestamp', 'img_key', 'shot_info', 'fps',
+ 'filename_tmpl', 'modality', 'start_index', 'timestamp_start',
+ 'timestamp_end', 'proposals', 'scores', 'frame_inds', 'clip_len',
+ 'frame_interval', 'gt_labels', 'gt_bboxes', 'entity_ids'
+ ]
+
+ def mock_video_reader(filename):
+ from unittest.mock import MagicMock
+ container = MagicMock()
+ container.__len__.return_value = 100
+ container.get_avg_fps.return_value = 24
+ frame_batch = MagicMock()
+ frame_batch.asnumpy.return_value = np.zeros((32, 720, 1280, 3))
+ container.get_batch.return_value = frame_batch
+ return container
+
+ ava_dataset = AVADataset(
+ self.ann_file,
+ self.pipeline,
+ data_prefix={'img': self.data_prefix},
+ proposal_file=self.proposal_file,
+ use_frames=False,
+ timestamp_start=1,
+ start_index=0,
+ multilabel=False,
+ fps=1)
+
+ # Mock a decord Container
+ ava_dataset.pipeline.transforms[
+ 0]._get_video_reader = mock_video_reader
+ result = ava_dataset[0]
+ assert assert_dict_has_keys(result, target_keys)
+
+ assert result['modality'] == 'RGB'
+ assert result['fps'] == 1
+ assert result['start_index'] == 0
+
+ h, w = result['imgs'][0].shape[:2]
+ scale_factor = np.array([w, h, w, h])
+ gt_bboxes = np.array([[0.71097612, 0.44144461, 0.79291363, 0.80873633],
+ [0.19915699, 0.40121613, 0.29834411,
+ 0.79667876]])
+ assert_array_almost_equal(
+ result['proposals'], gt_bboxes * scale_factor, decimal=4)
+ assert_array_almost_equal(result['scores'],
+ np.array([0.994165, 0.9902001]))
+
+ assert result['clip_len'] == 32
+ assert result['frame_interval'] == 2
+ assert len(result['frame_inds']) == 32
+
+ ava_dataset = AVADataset(
+ self.ann_file,
+ self.pipeline,
+ test_mode=True,
+ data_prefix={'img': self.data_prefix},
+ proposal_file=self.proposal_file,
+ use_frames=False,
+ timestamp_start=1,
+ start_index=0,
+ multilabel=False,
+ fps=1)
+ # Mock a decord Container
+ ava_dataset.pipeline.transforms[
+ 0]._get_video_reader = mock_video_reader
+ # Try to get a sample
+ result = ava_dataset[0]
+ assert result['modality'] == 'RGB'
+ assert result['fps'] == 1
+ assert result['start_index'] == 0
+
+
class TestAVAKineticsDataset:
@classmethod
diff --git a/tests/datasets/test_video_text_dataset.py b/tests/datasets/test_video_text_dataset.py
new file mode 100644
index 0000000000..43cd26ae0a
--- /dev/null
+++ b/tests/datasets/test_video_text_dataset.py
@@ -0,0 +1,46 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+from mmengine.testing import assert_dict_has_keys
+
+from mmaction.datasets import VideoTextDataset
+from mmaction.utils import register_all_modules
+from .base import BaseTestDataset
+
+
+class TestVideoTextDataset(BaseTestDataset):
+ register_all_modules()
+
+ def test_video_dataset(self):
+ video_dataset = VideoTextDataset(
+ self.video_text_ann_file,
+ self.video_text_pipeline,
+ data_prefix={'video': self.data_prefix},
+ start_index=3)
+ assert len(video_dataset) == 2
+ assert video_dataset.start_index == 3
+
+ video_dataset = VideoTextDataset(
+ self.video_text_ann_file,
+ self.video_text_pipeline,
+ data_prefix={'video': self.data_prefix})
+ assert video_dataset.start_index == 0
+
+ def test_video_pipeline(self):
+ target_keys = ['filename', 'text', 'start_index', 'modality', 'imgs']
+
+ # VideoTextDataset not in test mode
+ video_dataset = VideoTextDataset(
+ self.video_text_ann_file,
+ self.video_text_pipeline,
+ data_prefix={'video': self.data_prefix},
+ test_mode=False)
+ result = video_dataset[0]
+ assert assert_dict_has_keys(result, target_keys)
+
+ # VideoTextDataset in test mode
+ video_dataset = VideoTextDataset(
+ self.video_text_ann_file,
+ self.video_text_pipeline,
+ data_prefix={'video': self.data_prefix},
+ test_mode=True)
+ result = video_dataset[0]
+ assert assert_dict_has_keys(result, target_keys)
diff --git a/tests/datasets/transforms/test_formating.py b/tests/datasets/transforms/test_formating.py
index 8e741c24e5..4668732746 100644
--- a/tests/datasets/transforms/test_formating.py
+++ b/tests/datasets/transforms/test_formating.py
@@ -5,7 +5,7 @@
import numpy as np
import pytest
import torch
-from mmengine.structures import InstanceData, LabelData
+from mmengine.structures import InstanceData
from mmengine.testing import assert_dict_has_keys
from numpy.testing import assert_array_equal
@@ -22,62 +22,77 @@
class TestPackActionInputs(unittest.TestCase):
def test_transform(self):
+ # none input
+ with self.assertRaises(ValueError):
+ results = PackActionInputs()(dict())
+
# keypoint input
- results = dict(keypoint=np.random.randn(1, 2, 300, 17, 3), label=1)
+ results = dict(keypoint=np.random.randn(2, 300, 17, 3), label=1)
transform = PackActionInputs()
results = transform(results)
self.assertIn('inputs', results)
self.assertIn('data_samples', results)
self.assertIsInstance(results['inputs'], torch.Tensor)
+ self.assertEqual(results['inputs'].shape, (2, 300, 17, 3))
self.assertEqual(results['data_samples'].gt_labels.item,
torch.LongTensor([1]))
- # audio input
- data = dict(
- audios=np.random.randn(3, 1, 128, 80),
- label=[1],
- filename='test.txt')
+ # heatmap_imgs input
+ results = dict(heatmap_imgs=np.random.randn(2, 17, 56, 56), label=1)
+ transform = PackActionInputs()
+ results = transform(results)
+ self.assertIn('inputs', results)
+ self.assertIn('data_samples', results)
+ self.assertIsInstance(results['inputs'], torch.Tensor)
+ self.assertEqual(results['inputs'].shape, (2, 17, 56, 56))
+ self.assertEqual(results['data_samples'].gt_labels.item,
+ torch.LongTensor([1]))
- cfg = dict(type='PackActionInputs')
- transform = TRANSFORMS.build(cfg)
- results = transform(copy.deepcopy(data))
+ # audios input
+ results = dict(audios=np.random.randn(3, 1, 128, 80), label=[1])
+ transform = PackActionInputs()
+ results = transform(results)
self.assertIn('inputs', results)
+ self.assertIn('data_samples', results)
+ self.assertEqual(results['inputs'].shape, (3, 1, 128, 80))
self.assertIsInstance(results['inputs'], torch.Tensor)
- # img input with label
+
+ # text input
+ results = dict(text=np.random.randn(77))
+ transform = PackActionInputs()
+ results = transform(results)
+ self.assertIn('inputs', results)
+ self.assertIn('data_samples', results)
+ self.assertEqual(results['inputs'].shape, (77, ))
+ self.assertIsInstance(results['inputs'], torch.Tensor)
+
+ # imgs input with label
data = dict(
- imgs=np.random.randn(256, 256, 3),
+ imgs=np.random.randn(2, 256, 256, 3),
label=[1],
filename='test.txt',
original_shape=(256, 256, 3),
img_shape=(256, 256, 3),
- pad_shape=(256, 256, 3),
flip_direction='vertical')
- cfg = dict(type='PackActionInputs')
- transform = TRANSFORMS.build(cfg)
+ transform = PackActionInputs()
results = transform(copy.deepcopy(data))
self.assertIn('inputs', results)
- self.assertIsInstance(results['inputs'], torch.Tensor)
self.assertIn('data_samples', results)
+ self.assertIsInstance(results['inputs'], torch.Tensor)
self.assertIsInstance(results['data_samples'], ActionDataSample)
- self.assertIn('img_shape', results['data_samples'].metainfo_keys())
- self.assertIsInstance(results['data_samples'].gt_labels, LabelData)
+ self.assertEqual(results['data_samples'].img_shape, (256, 256, 3))
+ self.assertEqual(results['data_samples'].gt_labels.item,
+ torch.LongTensor([1]))
# Test grayscale image
data['imgs'] = data['imgs'].mean(-1)
results = transform(copy.deepcopy(data))
self.assertIn('inputs', results)
self.assertIsInstance(results['inputs'], torch.Tensor)
- self.assertEqual(results['inputs'].shape, (256, 256))
+ self.assertEqual(results['inputs'].shape, (2, 256, 256))
- # Test without `img` and `gt_label`
- del data['imgs']
- del data['label']
- with self.assertRaises(ValueError):
- results = transform(copy.deepcopy(data))
- self.assertNotIn('gt_labels', results['data_samples'])
-
- # img input with gt_bboxes
+ # imgs input with gt_bboxes
data = dict(
imgs=np.random.randn(256, 256, 3),
gt_bboxes=np.array([[0, 0, 340, 224]]),
@@ -85,8 +100,7 @@ def test_transform(self):
proposals=np.array([[0, 0, 340, 224]]),
filename='test.txt')
- cfg = dict(type='PackActionInputs')
- transform = TRANSFORMS.build(cfg)
+ transform = PackActionInputs()
results = transform(copy.deepcopy(data))
self.assertIn('inputs', results)
self.assertIsInstance(results['inputs'], torch.Tensor)
@@ -96,6 +110,18 @@ def test_transform(self):
InstanceData)
self.assertIsInstance(results['data_samples'].proposals, InstanceData)
+ # imgs and text input
+ data = dict(
+ imgs=np.random.randn(2, 256, 256, 3), text=np.random.randn(77))
+
+ transform = PackActionInputs(collect_keys=('imgs', 'text'))
+ results = transform(copy.deepcopy(data))
+ self.assertIn('inputs', results)
+ self.assertIn('data_samples', results)
+ self.assertIsInstance(results['inputs'], dict)
+ self.assertEqual(results['inputs']['imgs'].shape, (2, 256, 256, 3))
+ self.assertEqual(results['inputs']['text'].shape, (77, ))
+
def test_repr(self):
cfg = dict(
type='PackActionInputs', meta_keys=['flip_direction', 'img_shape'])
@@ -114,7 +140,7 @@ def test_transform(self):
gt_bbox=np.array([[0.1, 0.3], [0.375, 0.625]]),
filename='test.txt')
- cfg = dict(type='PackLocalizationInputs', keys='gt_bbox')
+ cfg = dict(type='PackLocalizationInputs', keys=('gt_bbox', ))
transform = TRANSFORMS.build(cfg)
results = transform(copy.deepcopy(data))
self.assertIn('inputs', results)
diff --git a/tests/datasets/transforms/test_sampling.py b/tests/datasets/transforms/test_sampling.py
index f4a5e457bd..364972ea39 100644
--- a/tests/datasets/transforms/test_sampling.py
+++ b/tests/datasets/transforms/test_sampling.py
@@ -561,7 +561,7 @@ def test_untrim_sample_frames(self):
label=1)
video_result = copy.deepcopy(self.video_results)
- config = dict(clip_len=1, frame_interval=16) # , start_index=0)
+ config = dict(clip_len=1, clip_interval=16) # , start_index=0)
sample_frames = UntrimmedSampleFrames(**config)
sample_frames_results = sample_frames(frame_result)
assert assert_dict_has_keys(sample_frames_results, target_keys)
@@ -570,9 +570,10 @@ def test_untrim_sample_frames(self):
np.array([8, 24, 40, 56, 72, 88]))
assert repr(sample_frames) == (f'{sample_frames.__class__.__name__}('
f'clip_len={1}, '
- f'frame_interval={16})')
+ f'clip_interval={16}, '
+ f'frame_interval={1})')
- config = dict(clip_len=1, frame_interval=16) # , start_index=0)
+ config = dict(clip_len=1, clip_interval=16) # , start_index=0)
sample_frames = UntrimmedSampleFrames(**config)
sample_frames_results = sample_frames(video_result)
assert assert_dict_has_keys(sample_frames_results, target_keys)
@@ -581,9 +582,10 @@ def test_untrim_sample_frames(self):
assert_array_equal(sample_frames_results['frame_inds'], frame_inds)
assert repr(sample_frames) == (f'{sample_frames.__class__.__name__}('
f'clip_len={1}, '
- f'frame_interval={16})')
+ f'clip_interval={16}, '
+ f'frame_interval={1})')
- config = dict(clip_len=1, frame_interval=16)
+ config = dict(clip_len=1, clip_interval=16)
sample_frames = UntrimmedSampleFrames(**config)
frame_result_ = copy.deepcopy(frame_result)
frame_result_['start_index'] = 1
@@ -594,9 +596,10 @@ def test_untrim_sample_frames(self):
np.array([8, 24, 40, 56, 72, 88]) + 1)
assert repr(sample_frames) == (f'{sample_frames.__class__.__name__}('
f'clip_len={1}, '
- f'frame_interval={16})')
+ f'clip_interval={16}, '
+ f'frame_interval={1})')
- config = dict(clip_len=3, frame_interval=16) # , start_index=0)
+ config = dict(clip_len=3, clip_interval=16) # , start_index=0)
sample_frames = UntrimmedSampleFrames(**config)
sample_frames_results = sample_frames(frame_result)
assert assert_dict_has_keys(sample_frames_results, target_keys)
@@ -609,7 +612,25 @@ def test_untrim_sample_frames(self):
]))
assert repr(sample_frames) == (f'{sample_frames.__class__.__name__}('
f'clip_len={3}, '
- f'frame_interval={16})')
+ f'clip_interval={16}, '
+ f'frame_interval={1})')
+
+ config = dict(
+ clip_len=3, clip_interval=16, frame_interval=4) # , start_index=0)
+ sample_frames = UntrimmedSampleFrames(**config)
+ sample_frames_results = sample_frames(frame_result)
+ assert assert_dict_has_keys(sample_frames_results, target_keys)
+ assert len(sample_frames_results['frame_inds']) == 18
+ assert_array_equal(
+ sample_frames_results['frame_inds'],
+ np.array([
+ 4, 8, 12, 20, 24, 28, 36, 40, 44, 52, 56, 60, 68, 72, 76, 84,
+ 88, 92
+ ]))
+ assert repr(sample_frames) == (f'{sample_frames.__class__.__name__}('
+ f'clip_len={3}, '
+ f'clip_interval={16}, '
+ f'frame_interval={4})')
def test_sample_ava_frames(self):
target_keys = [
diff --git a/tests/datasets/transforms/test_text_transforms.py b/tests/datasets/transforms/test_text_transforms.py
new file mode 100644
index 0000000000..f8018bfc50
--- /dev/null
+++ b/tests/datasets/transforms/test_text_transforms.py
@@ -0,0 +1,15 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+import torch
+
+from mmaction.datasets.transforms import CLIPTokenize
+
+
+class TestTextTransforms:
+
+ @staticmethod
+ def test_clip_tokenize():
+ results = {'text': 'Hello, MMAction2 2.0!'}
+ clip_tokenize = CLIPTokenize()
+ results = clip_tokenize(results)
+ assert results['text'].shape[0] == 77
+ assert results['text'].dtype == torch.int32
diff --git a/tests/evaluation/metrics/test_acc_metric.py b/tests/evaluation/metrics/test_acc_metric.py
index 295c301ab6..aeb6fb2cb0 100644
--- a/tests/evaluation/metrics/test_acc_metric.py
+++ b/tests/evaluation/metrics/test_acc_metric.py
@@ -1,10 +1,16 @@
# Copyright (c) OpenMMLab. All rights reserved.
+import os.path as osp
+import platform
from unittest import TestCase
import numpy as np
+import pytest
import torch
+from mmengine import load
+from numpy.testing import assert_array_almost_equal
-from mmaction.evaluation import AccMetric, ConfusionMatrix
+from mmaction.evaluation import AccMetric, ConfusionMatrix, MultiSportsMetric
+from mmaction.evaluation.functional import ava_eval
from mmaction.registry import METRICS
from mmaction.structures import ActionDataSample
@@ -54,6 +60,37 @@ def test_acc_metric():
assert eval_results['mmit_mean_average_precision'] == 1.0
+@pytest.mark.skipif(platform.system() == 'Windows', reason='Multiprocess Fail')
+def test_ava_detection():
+ data_prefix = osp.normpath(
+ osp.join(osp.dirname(__file__), '../../data/eval_detection'))
+
+ gt_path = osp.join(data_prefix, 'gt.csv')
+ result_path = osp.join(data_prefix, 'pred.csv')
+ label_map = osp.join(data_prefix, 'action_list.txt')
+
+ # eval bbox
+ detection = ava_eval(result_path, 'mAP', label_map, gt_path, None)
+ assert_array_almost_equal(detection['overall'], 0.09385522)
+
+
+def test_multisport_detection():
+ data_prefix = osp.normpath(
+ osp.join(osp.dirname(__file__), '../../data/eval_multisports'))
+
+ gt_path = osp.join(data_prefix, 'gt.pkl')
+ result_path = osp.join(data_prefix, 'data_samples.pkl')
+
+ result_datasamples = load(result_path)
+ metric = MultiSportsMetric(gt_path)
+ metric.process(None, result_datasamples)
+ eval_result = metric.compute_metrics(metric.results)
+ assert eval_result['frameAP'] == 83.6506
+ assert eval_result['v_map@0.2'] == 37.5
+ assert eval_result['v_map@0.5'] == 37.5
+ assert eval_result['v_map_0.10:0.90'] == 29.1667
+
+
class TestConfusionMatrix(TestCase):
def test_evaluate(self):
diff --git a/tests/evaluation/metrics/test_retrieval_metric.py b/tests/evaluation/metrics/test_retrieval_metric.py
new file mode 100644
index 0000000000..cb1f1c72ba
--- /dev/null
+++ b/tests/evaluation/metrics/test_retrieval_metric.py
@@ -0,0 +1,49 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+import pytest
+import torch
+
+from mmaction.evaluation.metrics import RetrievalMetric
+
+
+def generate_data(num_samples=5, feat_dim=10, random_label=False):
+ data_batch = []
+ data_samples = []
+ for i in range(num_samples):
+ if random_label:
+ video_feature = torch.randn(feat_dim)
+ text_feature = torch.randn(feat_dim)
+ else:
+ video_feature = torch.randn(feat_dim)
+ text_feature = video_feature.clone()
+
+ data_sample = dict(
+ features=dict(
+ video_feature=video_feature, text_feature=text_feature))
+ data_samples.append(data_sample)
+ return data_batch, data_samples
+
+
+def test_acc_metric():
+ with pytest.raises(ValueError):
+ RetrievalMetric(metric_list='R100')
+
+ num_samples = 20
+ metric = RetrievalMetric()
+ data_batch, predictions = generate_data(
+ num_samples=num_samples, random_label=True)
+ metric.process(data_batch, predictions)
+ eval_results = metric.compute_metrics(metric.results)
+ assert 0.0 <= eval_results['R1'] <= eval_results['R5'] <= eval_results[
+ 'R10'] <= 100.0
+ assert 0.0 <= eval_results['MdR'] <= num_samples
+ assert 0.0 <= eval_results['MnR'] <= num_samples
+
+ metric.results.clear()
+
+ data_batch, predictions = generate_data(
+ num_samples=num_samples, random_label=False)
+ metric.process(data_batch, predictions)
+ eval_results = metric.compute_metrics(metric.results)
+ assert eval_results['R1'] == eval_results['R5'] == eval_results[
+ 'R10'] == 100.0
+ assert eval_results['MdR'] == eval_results['MnR'] == 1.0
diff --git a/tests/models/backbones/test_mobilenet_v2.py b/tests/models/backbones/test_mobilenet_v2.py
index 4ddb8e8091..d9d862e8f8 100644
--- a/tests/models/backbones/test_mobilenet_v2.py
+++ b/tests/models/backbones/test_mobilenet_v2.py
@@ -30,8 +30,8 @@ def is_block(modules):
with pytest.raises(TypeError):
# pretrained must be a string path
- model = MobileNetV2(pretrained=0)
- model.init_weights()
+ model = MobileNetV2()
+ model.init_weights(pretrained=0)
with pytest.raises(ValueError):
# frozen_stages must in range(1, 9)
diff --git a/tests/models/backbones/test_mobilenet_v2_tsm.py b/tests/models/backbones/test_mobilenet_v2_tsm.py
index 34ad6b86a9..b0637be1d5 100644
--- a/tests/models/backbones/test_mobilenet_v2_tsm.py
+++ b/tests/models/backbones/test_mobilenet_v2_tsm.py
@@ -16,7 +16,7 @@ def test_mobilenetv2_tsm_backbone():
imgs = generate_backbone_demo_inputs(input_shape)
# mobilenetv2_tsm with width_mult = 1.0
- mobilenetv2_tsm = MobileNetV2TSM()
+ mobilenetv2_tsm = MobileNetV2TSM(pretrained='mmcls://mobilenet_v2')
mobilenetv2_tsm.init_weights()
for cur_module in mobilenetv2_tsm.modules():
if isinstance(cur_module, InvertedResidual) and \
@@ -33,13 +33,13 @@ def test_mobilenetv2_tsm_backbone():
assert feat.shape == torch.Size([8, 1280, 2, 2])
# mobilenetv2 with widen_factor = 0.5 forword
- mobilenetv2_tsm_05 = MobileNetV2TSM(widen_factor=0.5)
+ mobilenetv2_tsm_05 = MobileNetV2TSM(widen_factor=0.5, pretrained2d=False)
mobilenetv2_tsm_05.init_weights()
feat = mobilenetv2_tsm_05(imgs)
assert feat.shape == torch.Size([8, 1280, 2, 2])
# mobilenetv2 with widen_factor = 1.5 forword
- mobilenetv2_tsm_15 = MobileNetV2TSM(widen_factor=1.5)
+ mobilenetv2_tsm_15 = MobileNetV2TSM(widen_factor=1.5, pretrained2d=False)
mobilenetv2_tsm_15.init_weights()
feat = mobilenetv2_tsm_15(imgs)
assert feat.shape == torch.Size([8, 1920, 2, 2])
diff --git a/tests/models/backbones/test_resnet_tsm.py b/tests/models/backbones/test_resnet_tsm.py
index ddac948354..db22aaf3cb 100644
--- a/tests/models/backbones/test_resnet_tsm.py
+++ b/tests/models/backbones/test_resnet_tsm.py
@@ -1,113 +1,130 @@
# Copyright (c) OpenMMLab. All rights reserved.
import copy
+from unittest import TestCase
import pytest
import torch
import torch.nn as nn
from mmaction.models import ResNetTSM
-from mmaction.models.backbones.resnet_tsm import NL3DWrapper
+from mmaction.models.backbones.resnet import Bottleneck
+from mmaction.models.backbones.resnet_tsm import NL3DWrapper, TemporalShift
from mmaction.testing import generate_backbone_demo_inputs
-def test_resnet_tsm_backbone():
- """Test resnet_tsm backbone."""
- with pytest.raises(NotImplementedError):
- # shift_place must be block or blockres
- resnet_tsm_50_block = ResNetTSM(50, shift_place='Block')
+class Test_ResNet_TSM(TestCase):
+
+ def setUp(self):
+ input_shape = (8, 3, 64, 64)
+ self.imgs = generate_backbone_demo_inputs(input_shape)
+
+ def test_init(self):
+ with pytest.raises(NotImplementedError):
+ # shift_place must be block or blockres
+ resnet_tsm_50_block = ResNetTSM(50, shift_place='Block')
+ resnet_tsm_50_block.init_weights()
+
+ def test_init_from_scratch(self):
+ resnet_tsm_50 = ResNetTSM(50, pretrained=None, pretrained2d=False)
+ resnet_tsm_50.init_weights()
+
+ def test_resnet_tsm_temporal_shift_blockres(self):
+ # resnet_tsm with depth 50
+ resnet_tsm_50 = ResNetTSM(50, pretrained='torchvision://resnet50')
+ resnet_tsm_50.init_weights()
+ for layer_name in resnet_tsm_50.res_layers:
+ layer = getattr(resnet_tsm_50, layer_name)
+ blocks = list(layer.children())
+ for block in blocks:
+ assert isinstance(block.conv1.conv, TemporalShift)
+ assert block.conv1.conv.num_segments == resnet_tsm_50.num_segments # noqa: E501
+ assert block.conv1.conv.shift_div == resnet_tsm_50.shift_div
+ assert isinstance(block.conv1.conv.net, nn.Conv2d)
+ feat = resnet_tsm_50(self.imgs)
+ assert feat.shape == torch.Size([8, 2048, 2, 2])
+
+ def test_resnet_tsm_temporal_shift_block(self):
+ # resnet_tsm with depth 50, no pretrained, shift_place is block
+ resnet_tsm_50_block = ResNetTSM(
+ 50, shift_place='block', pretrained='torchvision://resnet50')
resnet_tsm_50_block.init_weights()
-
- from mmaction.models.backbones.resnet import Bottleneck
- from mmaction.models.backbones.resnet_tsm import TemporalShift
-
- input_shape = (8, 3, 64, 64)
- imgs = generate_backbone_demo_inputs(input_shape)
-
- # resnet_tsm with depth 50
- resnet_tsm_50 = ResNetTSM(50)
- resnet_tsm_50.init_weights()
- for layer_name in resnet_tsm_50.res_layers:
- layer = getattr(resnet_tsm_50, layer_name)
- blocks = list(layer.children())
- for block in blocks:
- assert isinstance(block.conv1.conv, TemporalShift)
- assert block.conv1.conv.num_segments == resnet_tsm_50.num_segments
- assert block.conv1.conv.shift_div == resnet_tsm_50.shift_div
- assert isinstance(block.conv1.conv.net, nn.Conv2d)
-
- # resnet_tsm with depth 50, no pretrained, shift_place is block
- resnet_tsm_50_block = ResNetTSM(50, shift_place='block')
- resnet_tsm_50_block.init_weights()
- for layer_name in resnet_tsm_50_block.res_layers:
- layer = getattr(resnet_tsm_50_block, layer_name)
- blocks = list(layer.children())
- for block in blocks:
- assert isinstance(block, TemporalShift)
- assert block.num_segments == resnet_tsm_50_block.num_segments
- assert block.num_segments == resnet_tsm_50_block.num_segments
- assert block.shift_div == resnet_tsm_50_block.shift_div
- assert isinstance(block.net, Bottleneck)
-
- # resnet_tsm with depth 50, no pretrained, use temporal_pool
- resnet_tsm_50_temporal_pool = ResNetTSM(50, temporal_pool=True)
- resnet_tsm_50_temporal_pool.init_weights()
- for layer_name in resnet_tsm_50_temporal_pool.res_layers:
- layer = getattr(resnet_tsm_50_temporal_pool, layer_name)
- blocks = list(layer.children())
-
- if layer_name == 'layer2':
- assert len(blocks) == 2
- assert isinstance(blocks[1], nn.MaxPool3d)
- blocks = copy.deepcopy(blocks[0])
-
- for block in blocks:
- assert isinstance(block.conv1.conv, TemporalShift)
- if layer_name == 'layer1':
- assert block.conv1.conv.num_segments == \
- resnet_tsm_50_temporal_pool.num_segments
- else:
- assert block.conv1.conv.num_segments == \
- resnet_tsm_50_temporal_pool.num_segments // 2
- assert block.conv1.conv.shift_div == resnet_tsm_50_temporal_pool.shift_div # noqa: E501
- assert isinstance(block.conv1.conv.net, nn.Conv2d)
-
- # resnet_tsm with non-local module
- non_local_cfg = dict(
- sub_sample=True,
- use_scale=False,
- norm_cfg=dict(type='BN3d', requires_grad=True),
- mode='embedded_gaussian')
- non_local = ((0, 0, 0), (1, 0, 1, 0), (1, 0, 1, 0, 1, 0), (0, 0, 0))
- resnet_tsm_nonlocal = ResNetTSM(
- 50, non_local=non_local, non_local_cfg=non_local_cfg)
- resnet_tsm_nonlocal.init_weights()
- for layer_name in ['layer2', 'layer3']:
- layer = getattr(resnet_tsm_nonlocal, layer_name)
- for i, _ in enumerate(layer):
- if i % 2 == 0:
- assert isinstance(layer[i], NL3DWrapper)
-
- resnet_tsm_50_full = ResNetTSM(
- 50,
- non_local=non_local,
- non_local_cfg=non_local_cfg,
- temporal_pool=True)
- resnet_tsm_50_full.init_weights()
-
- # TSM forword
- feat = resnet_tsm_50(imgs)
- assert feat.shape == torch.Size([8, 2048, 2, 2])
-
- # TSM with non-local forward
- feat = resnet_tsm_nonlocal(imgs)
- assert feat.shape == torch.Size([8, 2048, 2, 2])
-
- # TSM with temporal pool forward
- feat = resnet_tsm_50_temporal_pool(imgs)
- assert feat.shape == torch.Size([4, 2048, 2, 2])
-
- # TSM with temporal pool + non-local forward
- input_shape = (16, 3, 32, 32)
- imgs = generate_backbone_demo_inputs(input_shape)
- feat = resnet_tsm_50_full(imgs)
- assert feat.shape == torch.Size([8, 2048, 1, 1])
+ for layer_name in resnet_tsm_50_block.res_layers:
+ layer = getattr(resnet_tsm_50_block, layer_name)
+ blocks = list(layer.children())
+ for block in blocks:
+ assert isinstance(block, TemporalShift)
+ assert block.num_segments == resnet_tsm_50_block.num_segments
+ assert block.num_segments == resnet_tsm_50_block.num_segments
+ assert block.shift_div == resnet_tsm_50_block.shift_div
+ assert isinstance(block.net, Bottleneck)
+
+ def test_resnet_tsm_temporal_pool(self):
+ # resnet_tsm with depth 50, no pretrained, use temporal_pool
+ resnet_tsm_50_temporal_pool = ResNetTSM(
+ 50, temporal_pool=True, pretrained='torchvision://resnet50')
+ resnet_tsm_50_temporal_pool.init_weights()
+ for layer_name in resnet_tsm_50_temporal_pool.res_layers:
+ layer = getattr(resnet_tsm_50_temporal_pool, layer_name)
+ blocks = list(layer.children())
+
+ if layer_name == 'layer2':
+ assert len(blocks) == 2
+ assert isinstance(blocks[1], nn.MaxPool3d)
+ blocks = copy.deepcopy(blocks[0])
+
+ for block in blocks:
+ assert isinstance(block.conv1.conv, TemporalShift)
+ if layer_name == 'layer1':
+ assert block.conv1.conv.num_segments == \
+ resnet_tsm_50_temporal_pool.num_segments
+ else:
+ assert block.conv1.conv.num_segments == \
+ resnet_tsm_50_temporal_pool.num_segments // 2
+ assert block.conv1.conv.shift_div == resnet_tsm_50_temporal_pool.shift_div # noqa: E501
+ assert isinstance(block.conv1.conv.net, nn.Conv2d)
+
+ feat = resnet_tsm_50_temporal_pool(self.imgs)
+ assert feat.shape == torch.Size([4, 2048, 2, 2])
+
+ def test_resnet_tsm_non_local(self):
+ # resnet_tsm with non-local module
+ non_local_cfg = dict(
+ sub_sample=True,
+ use_scale=False,
+ norm_cfg=dict(type='BN3d', requires_grad=True),
+ mode='embedded_gaussian')
+ non_local = ((0, 0, 0), (1, 0, 1, 0), (1, 0, 1, 0, 1, 0), (0, 0, 0))
+ resnet_tsm_nonlocal = ResNetTSM(
+ 50,
+ non_local=non_local,
+ non_local_cfg=non_local_cfg,
+ pretrained='torchvision://resnet50')
+ resnet_tsm_nonlocal.init_weights()
+ for layer_name in ['layer2', 'layer3']:
+ layer = getattr(resnet_tsm_nonlocal, layer_name)
+ for i, _ in enumerate(layer):
+ if i % 2 == 0:
+ assert isinstance(layer[i], NL3DWrapper)
+
+ feat = resnet_tsm_nonlocal(self.imgs)
+ assert feat.shape == torch.Size([8, 2048, 2, 2])
+
+ def test_resnet_tsm_full(self):
+ non_local_cfg = dict(
+ sub_sample=True,
+ use_scale=False,
+ norm_cfg=dict(type='BN3d', requires_grad=True),
+ mode='embedded_gaussian')
+ non_local = ((0, 0, 0), (1, 0, 1, 0), (1, 0, 1, 0, 1, 0), (0, 0, 0))
+ resnet_tsm_50_full = ResNetTSM(
+ 50,
+ pretrained='torchvision://resnet50',
+ non_local=non_local,
+ non_local_cfg=non_local_cfg,
+ temporal_pool=True)
+ resnet_tsm_50_full.init_weights()
+
+ input_shape = (16, 3, 32, 32)
+ imgs = generate_backbone_demo_inputs(input_shape)
+ feat = resnet_tsm_50_full(imgs)
+ assert feat.shape == torch.Size([8, 2048, 1, 1])
diff --git a/tests/models/data_preprocessors/test_data_preprocessor.py b/tests/models/data_preprocessors/test_data_preprocessor.py
index a4a3d851d7..5fe3e8f663 100644
--- a/tests/models/data_preprocessors/test_data_preprocessor.py
+++ b/tests/models/data_preprocessors/test_data_preprocessor.py
@@ -95,3 +95,8 @@ def test_data_preprocessor():
(raw_3d_data['inputs'][0] - psr.mean) / psr.std)
assert_array_equal(data[1]['inputs'][1],
(raw_3d_data['inputs'][1] - psr.mean) / psr.std)
+
+ raw_data = generate_dummy_data(2, (77, ))
+ psr = ActionDataPreprocessor(to_float32=False)
+ data = psr(raw_data)
+ assert data['inputs'].dtype == raw_data['inputs'][0].dtype
diff --git a/tests/models/heads/test_feature_head.py b/tests/models/heads/test_feature_head.py
new file mode 100644
index 0000000000..932ed87133
--- /dev/null
+++ b/tests/models/heads/test_feature_head.py
@@ -0,0 +1,138 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+from unittest import TestCase
+
+import pytest
+import torch
+
+from mmaction.models import FeatureHead
+from mmaction.registry import MODELS
+from mmaction.structures import ActionDataSample
+from mmaction.testing import get_recognizer_cfg
+from mmaction.utils import register_all_modules
+
+
+class TestFeatureHead(TestCase):
+
+ def test_2d_recognizer(self):
+ register_all_modules()
+ config = get_recognizer_cfg(
+ 'tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py' # noqa: E501
+ )
+ config.model['backbone']['pretrained'] = None
+ config.model['cls_head'] = dict(
+ type='FeatureHead', average_clips='score')
+
+ recognizer = MODELS.build(config.model)
+
+ input_shape = [3, 3, 32, 32]
+ data_batch = {
+ 'inputs': [torch.randint(0, 256, input_shape)],
+ 'data_samples': [ActionDataSample().set_gt_labels(2)]
+ }
+ feat = recognizer.test_step(data_batch)
+ assert isinstance(feat, torch.Tensor)
+ assert feat.shape == torch.Size([1, 2048])
+
+ def test_3d_recognizer(self):
+ register_all_modules()
+ config = get_recognizer_cfg(
+ 'slowonly/slowonly_r50_8xb16-4x16x1-256e_kinetics400-rgb.py')
+ config.model['backbone']['pretrained'] = None
+ config.model['backbone']['pretrained2d'] = False
+ config.model['cls_head'] = dict(
+ type='FeatureHead', average_clips='score')
+
+ recognizer = MODELS.build(config.model)
+ input_shape = [1, 3, 4, 32, 32]
+ data_batch = {
+ 'inputs': [torch.randint(0, 256, input_shape)],
+ 'data_samples': [ActionDataSample().set_gt_labels(2)]
+ }
+ feat = recognizer.test_step(data_batch)
+ assert isinstance(feat, torch.Tensor)
+ assert feat.shape == torch.Size([1, 2048])
+
+ def test_3d_backbone(self):
+ with pytest.raises(NotImplementedError):
+ head = FeatureHead(spatial_type='test')
+
+ head = FeatureHead(average_clips='score')
+ x = torch.rand(1, 64, 2, 7, 7)
+ feat = head(x)
+ assert feat.shape == torch.Size([1, 64])
+
+ head = FeatureHead(spatial_type=None, average_clips='score')
+ feat = head(x)
+ assert feat.shape == torch.Size([1, 64, 7, 7])
+
+ head = FeatureHead(temporal_type=None, average_clips='score')
+ feat = head(x)
+ assert feat.shape == torch.Size([1, 64, 2])
+
+ head = FeatureHead(
+ spatial_type=None, temporal_type=None, average_clips='score')
+ feat = head(x)
+ assert feat.shape == torch.Size([1, 64, 2, 7, 7])
+
+ def test_slowfast_backbone(self):
+ head = FeatureHead(backbone_name='slowfast', average_clips='score')
+ x_slow = torch.rand(1, 64, 2, 7, 7)
+ x_fast = torch.rand(1, 32, 6, 7, 7)
+ x = (x_slow, x_fast)
+ feat = head(x)
+ assert feat.shape == torch.Size([1, 96])
+
+ head = FeatureHead(
+ backbone_name='slowfast', spatial_type=None, average_clips='score')
+ feat = head(x)
+ assert feat.shape == torch.Size([1, 96, 7, 7])
+
+ with pytest.raises(AssertionError):
+ head = FeatureHead(
+ backbone_name='slowfast',
+ temporal_type=None,
+ average_clips='score')
+ feat = head(x)
+
+ def test_2d_backbone(self):
+ head = FeatureHead(average_clips='score')
+ x = torch.rand(2, 64, 7, 7)
+ with pytest.raises(AssertionError):
+ feat = head(x)
+
+ feat = head(x, num_segs=2)
+ assert feat.shape == torch.Size([1, 64])
+
+ x = torch.rand(2, 64, 7, 7)
+ head = FeatureHead(spatial_type=None, average_clips='score')
+ feat = head(x, num_segs=2)
+ assert feat.shape == torch.Size([1, 64, 7, 7])
+
+ head = FeatureHead(temporal_type=None, average_clips='score')
+ feat = head(x, num_segs=2)
+ assert feat.shape == torch.Size([1, 2, 64])
+
+ def test_tsm_backbone(self):
+ head = FeatureHead(backbone_name='tsm', average_clips='score')
+ x = torch.rand(2, 64, 7, 7)
+ with pytest.raises(AssertionError):
+ feat = head(x)
+ with pytest.raises(AssertionError):
+ feat = head(x, num_segs=2)
+
+ head = FeatureHead(num_segments=2, average_clips='score')
+ feat = head(x, num_segs=2)
+ assert feat.shape == torch.Size([1, 64])
+
+ x = torch.rand(2, 64, 7, 7)
+ head = FeatureHead(
+ num_segments=2, spatial_type=None, average_clips='score')
+ feat = head(x, num_segs=2)
+ assert feat.shape == torch.Size([1, 64, 7, 7])
+
+ def test_gcn_backbone(self):
+ # N, M, C, T, V
+ head = FeatureHead(backbone_name='gcn', average_clips='score')
+ x = torch.rand(1, 5, 64, 2, 7)
+ feat = head(x)
+ assert feat.shape == torch.Size([1, 64])
diff --git a/tests/models/recognizers/test_recognizer2d.py b/tests/models/recognizers/test_recognizer2d.py
index 773bc0806f..b40398755b 100644
--- a/tests/models/recognizers/test_recognizer2d.py
+++ b/tests/models/recognizers/test_recognizer2d.py
@@ -1,44 +1,74 @@
# Copyright (c) OpenMMLab. All rights reserved.
import platform
+from unittest.mock import MagicMock
import pytest
import torch
+from mmengine.utils import digit_version
from mmaction.registry import MODELS
-from mmaction.testing import (generate_recognizer_demo_inputs,
- get_recognizer_cfg)
+from mmaction.structures import ActionDataSample
+from mmaction.testing import get_recognizer_cfg
from mmaction.utils import register_all_modules
+def train_test_step(cfg, input_shape):
+ recognizer = MODELS.build(cfg.model)
+ num_classes = cfg.model.cls_head.num_classes
+ batch_size = input_shape[0]
+ input_shape = input_shape[1:]
+ data_batch = {
+ 'inputs':
+ [torch.randint(0, 256, input_shape) for i in range(batch_size)],
+ 'data_samples':
+ [ActionDataSample().set_gt_labels(2) for i in range(batch_size)]
+ }
+
+ # test train_step
+ optim_wrapper = MagicMock()
+ loss_vars = recognizer.train_step(data_batch, optim_wrapper)
+ assert 'loss' in loss_vars
+ assert 'loss_cls' in loss_vars
+ optim_wrapper.update_params.assert_called_once()
+
+ # test test_step
+ with torch.no_grad():
+ predictions = recognizer.test_step(data_batch)
+ score = predictions[0].pred_scores.item
+ assert len(predictions) == batch_size
+ assert score.shape == torch.Size([num_classes])
+ assert torch.min(score) >= 0
+ assert torch.max(score) <= 1
+
+ # test twice sample + 3 crops
+ num_views = input_shape[0] * 2 * 3
+ input_shape = (num_views, *input_shape[1:])
+ data_batch['inputs'] = [torch.randint(0, 256, input_shape)]
+ with torch.no_grad():
+ predictions = recognizer.test_step(data_batch)
+ score = predictions[0].pred_scores.item
+ assert len(predictions) == batch_size
+ assert score.shape == torch.Size([num_classes])
+
+ return loss_vars, predictions
+
+
def test_tsn():
register_all_modules()
config = get_recognizer_cfg(
'tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py')
config.model['backbone']['pretrained'] = None
- recognizer = MODELS.build(config.model)
-
input_shape = (1, 3, 3, 32, 32)
- demo_inputs = generate_recognizer_demo_inputs(input_shape)
-
- imgs = demo_inputs['imgs']
- gt_labels = demo_inputs['gt_labels']
+ train_test_step(config, input_shape)
- losses = recognizer(imgs, gt_labels)
- assert isinstance(losses, torch.Tensor)
- # Test forward test
- with torch.no_grad():
- img_list = [img[None, :] for img in imgs]
- for one_img in img_list:
- recognizer(one_img, None, return_loss=False)
-
- # Test forward gradcam
- recognizer(imgs, gradcam=True)
- for one_img in img_list:
- recognizer(one_img, gradcam=True)
- """
- TODO
+def test_tsn_mmcls_backbone():
+ register_all_modules()
+ config = get_recognizer_cfg(
+ 'tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py')
+ config.model['backbone']['pretrained'] = None
+ # test mmcls backbone
mmcls_backbone = dict(
type='mmcls.ResNeXt',
depth=101,
@@ -49,109 +79,85 @@ def test_tsn():
style='pytorch')
config.model['backbone'] = mmcls_backbone
- recognizer = MODELS.build(config.model)
+ input_shape = (1, 3, 3, 32, 32)
+ train_test_step(config, input_shape)
+
+ from mmcls.models import ResNeXt
+ mmcls_backbone['type'] = ResNeXt
+ config.model['backbone'] = mmcls_backbone
input_shape = (1, 3, 3, 32, 32)
- demo_inputs = generate_recognizer_demo_inputs(input_shape)
+ train_test_step(config, input_shape)
+
+
+def test_tsn_timm_backbone():
+ # test tsn from timm
+ register_all_modules()
+ config = get_recognizer_cfg(
+ 'tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py')
+ timm_backbone = dict(type='timm.efficientnet_b0', pretrained=False)
+ config.model['backbone'] = timm_backbone
+ config.model['cls_head']['in_channels'] = 1280
- imgs = demo_inputs['imgs']
- gt_labels = demo_inputs['gt_labels']
+ input_shape = (1, 3, 3, 32, 32)
+ train_test_step(config, input_shape)
+ import timm
+ if digit_version(timm.__version__) <= digit_version('0.6.7'):
+ feature_shape = 'NLC'
+ else:
+ feature_shape = 'NHWC'
- losses = recognizer(imgs, gt_labels)
- assert isinstance(losses, torch.Tensor)
+ timm_swin = dict(
+ type='timm.swin_base_patch4_window7_224',
+ pretrained=False,
+ feature_shape=feature_shape)
+ config.model['backbone'] = timm_swin
+ config.model['cls_head']['in_channels'] = 1024
- # Test forward test
- with torch.no_grad():
- img_list = [img[None, :] for img in imgs]
- for one_img in img_list:
- recognizer(one_img, None, return_loss=False)
+ input_shape = (1, 3, 3, 224, 224)
+ train_test_step(config, input_shape)
- # test mixup forward
- # TODO
+def test_tsn_tv_backbone():
+ register_all_modules()
config = get_recognizer_cfg(
- 'tsn/tsn_r50_video_mixup_1x1x8_100e_kinetics400_rgb.py')
+ 'tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py')
config.model['backbone']['pretrained'] = None
- recognizer = MODELS.build(config.model)
- input_shape = (2, 8, 3, 32, 32)
- demo_inputs = generate_recognizer_demo_inputs(input_shape)
- imgs = demo_inputs['imgs']
- gt_labels = demo_inputs['gt_labels']
- losses = recognizer(imgs, gt_labels)
- assert isinstance(losses, torch.Tensor)
-
- # test torchvision backbones
- # TODO
+ # test tv backbone
tv_backbone = dict(type='torchvision.densenet161', pretrained=True)
config.model['backbone'] = tv_backbone
config.model['cls_head']['in_channels'] = 2208
- recognizer = MODELS.build(config.model)
-
input_shape = (1, 3, 3, 32, 32)
- demo_inputs = generate_recognizer_demo_inputs(input_shape)
-
- imgs = demo_inputs['imgs']
- gt_labels = demo_inputs['gt_labels']
+ train_test_step(config, input_shape)
- losses = recognizer(imgs, gt_labels)
- assert isinstance(losses, torch.Tensor)
+ from torchvision.models import densenet161
+ tv_backbone = dict(type=densenet161, pretrained=True)
+ config.model['backbone'] = tv_backbone
+ config.model['cls_head']['in_channels'] = 2208
- # Test forward test
- with torch.no_grad():
- img_list = [img[None, :] for img in imgs]
- for one_img in img_list:
- recognizer(one_img, None, return_loss=False)
- """
+ input_shape = (1, 3, 3, 32, 32)
+ train_test_step(config, input_shape)
def test_tsm():
register_all_modules()
config = get_recognizer_cfg(
- 'tsm/tsm_imagenet-pretrained-mobilenetv2_8xb16-1x1x8-50e_kinetics400-rgb.py' # noqa: E501
+ 'tsm/tsm_imagenet-pretrained-mobilenetv2_8xb16-1x1x8-100e_kinetics400-rgb.py' # noqa: E501
)
config.model['backbone']['pretrained'] = None
+ config.model['backbone']['pretrained2d'] = None
- recognizer = MODELS.build(config.model)
- recognizer.init_weights()
+ input_shape = (1, 8, 3, 32, 32)
+ train_test_step(config, input_shape)
config = get_recognizer_cfg(
'tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_kinetics400-rgb.py')
config.model['backbone']['pretrained'] = None
-
- recognizer = MODELS.build(config.model)
- recognizer.init_weights()
+ config.model['backbone']['pretrained2d'] = None
input_shape = (1, 8, 3, 32, 32)
- demo_inputs = generate_recognizer_demo_inputs(input_shape)
-
- imgs = demo_inputs['imgs']
- gt_labels = demo_inputs['gt_labels']
-
- losses = recognizer(imgs, gt_labels)
- assert isinstance(losses, torch.Tensor)
-
- # Test forward test
- with torch.no_grad():
- img_list = [img[None, :] for img in imgs]
- for one_img in img_list:
- recognizer(one_img, None, return_loss=False)
-
- # test twice sample + 3 crops
- input_shape = (2, 48, 3, 32, 32)
- demo_inputs = generate_recognizer_demo_inputs(input_shape)
- imgs = demo_inputs['imgs']
-
- # Test forward test
- with torch.no_grad():
- img_list = [img[None, :] for img in imgs]
- for one_img in img_list:
- recognizer(one_img, None, return_loss=False)
-
- # Test forward gradcam
- recognizer(imgs, gradcam=True)
- for one_img in img_list:
- recognizer(one_img, gradcam=True)
+ train_test_step(config, input_shape)
def test_trn():
@@ -160,38 +166,8 @@ def test_trn():
'trn/trn_imagenet-pretrained-r50_8xb16-1x1x8-50e_sthv1-rgb.py')
config.model['backbone']['pretrained'] = None
- recognizer = MODELS.build(config.model)
-
input_shape = (1, 8, 3, 32, 32)
- demo_inputs = generate_recognizer_demo_inputs(input_shape)
-
- imgs = demo_inputs['imgs']
- gt_labels = demo_inputs['gt_labels']
-
- losses = recognizer(imgs, gt_labels)
- assert isinstance(losses, torch.Tensor)
-
- # Test forward test
- with torch.no_grad():
- img_list = [img[None, :] for img in imgs]
- for one_img in img_list:
- recognizer(one_img, None, return_loss=False)
-
- # test twice sample + 3 crops
- input_shape = (2, 48, 3, 32, 32)
- demo_inputs = generate_recognizer_demo_inputs(input_shape)
- imgs = demo_inputs['imgs']
-
- # Test forward test
- with torch.no_grad():
- img_list = [img[None, :] for img in imgs]
- for one_img in img_list:
- recognizer(one_img, None, return_loss=False)
-
- # Test forward gradcam
- recognizer(imgs, gradcam=True)
- for one_img in img_list:
- recognizer(one_img, gradcam=True)
+ train_test_step(config, input_shape)
@pytest.mark.skipif(platform.system() == 'Windows', reason='Windows mem limit')
@@ -201,30 +177,8 @@ def test_tpn():
'tpn/tpn-tsm_imagenet-pretrained-r50_8xb8-1x1x8-150e_sthv1-rgb.py')
config.model['backbone']['pretrained'] = None
- recognizer = MODELS.build(config.model)
-
- input_shape = (1, 8, 3, 32, 32)
- demo_inputs = generate_recognizer_demo_inputs(input_shape)
-
- imgs = demo_inputs['imgs']
- gt_labels = demo_inputs['gt_labels']
-
- losses = recognizer(imgs, gt_labels)
-
- if not isinstance(losses, torch.Tensor):
- for i in losses:
- assert isinstance(i, torch.Tensor)
-
- # Test forward test
- with torch.no_grad():
- img_list = [img[None, :] for img in imgs]
- for one_img in img_list:
- recognizer(one_img, None, return_loss=False)
-
- # Test forward gradcam
- recognizer(imgs, gradcam=True)
- for one_img in img_list:
- recognizer(one_img, gradcam=True)
+ input_shape = (1, 8, 3, 64, 64)
+ train_test_step(config, input_shape)
def test_tanet():
@@ -233,63 +187,5 @@ def test_tanet():
'dense-1x1x8-100e_kinetics400-rgb.py')
config.model['backbone']['pretrained'] = None
- recognizer = MODELS.build(config.model)
-
input_shape = (1, 8, 3, 32, 32)
- demo_inputs = generate_recognizer_demo_inputs(input_shape)
-
- imgs = demo_inputs['imgs']
- gt_labels = demo_inputs['gt_labels']
-
- losses = recognizer(imgs, gt_labels)
- assert isinstance(losses, torch.Tensor)
-
- # Test forward test
- with torch.no_grad():
- img_list = [img[None, :] for img in imgs]
- for one_img in img_list:
- recognizer(one_img, None, return_loss=False)
-
- # test twice sample + 3 crops
- input_shape = (2, 48, 3, 32, 32)
- demo_inputs = generate_recognizer_demo_inputs(input_shape)
- imgs = demo_inputs['imgs']
-
- # Test forward test
- with torch.no_grad():
- img_list = [img[None, :] for img in imgs]
- for one_img in img_list:
- recognizer(one_img, None, return_loss=False)
-
- # Test forward gradcam
- recognizer(imgs, gradcam=True)
- for one_img in img_list:
- recognizer(one_img, gradcam=True)
-
-
-def test_timm_backbone():
- # test tsn from timm
- register_all_modules()
- config = get_recognizer_cfg(
- 'tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py')
- config.model['backbone']['pretrained'] = None
- timm_backbone = dict(type='timm.efficientnet_b0', pretrained=False)
- config.model['backbone'] = timm_backbone
- config.model['cls_head']['in_channels'] = 1280
-
- recognizer = MODELS.build(config.model)
-
- input_shape = (1, 3, 3, 32, 32)
- demo_inputs = generate_recognizer_demo_inputs(input_shape)
-
- imgs = demo_inputs['imgs']
- gt_labels = demo_inputs['gt_labels']
-
- losses = recognizer(imgs, gt_labels)
- assert isinstance(losses, torch.Tensor)
-
- # Test forward test
- with torch.no_grad():
- img_list = [img[None, :] for img in imgs]
- for one_img in img_list:
- recognizer(one_img, None, return_loss=False)
+ train_test_step(config, input_shape)
diff --git a/tests/models/similarity/test_adapters.py b/tests/models/similarity/test_adapters.py
new file mode 100644
index 0000000000..56e44f2310
--- /dev/null
+++ b/tests/models/similarity/test_adapters.py
@@ -0,0 +1,44 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+import pytest
+import torch
+
+from mmaction.models import SimpleMeanAdapter, TransformerAdapter
+
+
+def test_transformer_adapter():
+ """Test transformer adapter."""
+ with pytest.raises(RuntimeError):
+ num_segs_model = 8
+ num_segs_features = 9
+ adapter = TransformerAdapter(
+ num_segs=num_segs_model,
+ transformer_width=64,
+ transformer_heads=8,
+ transformer_layers=2)
+ features = torch.randn(2, num_segs_features, 64)
+ adapter(features)
+
+ num_segs = 8
+ adapter = TransformerAdapter(
+ num_segs=num_segs,
+ transformer_width=64,
+ transformer_heads=8,
+ transformer_layers=2)
+ adapter.init_weights()
+ features = torch.randn(2, num_segs, 64)
+ adapted_features = adapter(features)
+ assert adapted_features.shape == torch.Size([2, 64])
+
+
+def test_simple_mean_adapter():
+ """Test simple mean adapter."""
+
+ adapter = SimpleMeanAdapter(dim=1)
+ features = torch.randn(2, 8, 64)
+ adapted_features = adapter(features)
+ assert adapted_features.shape == torch.Size([2, 64])
+
+ adapter = SimpleMeanAdapter(dim=(1, 2))
+ features = torch.randn(2, 8, 2, 64)
+ adapted_features = adapter(features)
+ assert adapted_features.shape == torch.Size([2, 64])
diff --git a/tests/models/similarity/test_clip_similarity.py b/tests/models/similarity/test_clip_similarity.py
new file mode 100644
index 0000000000..9afa158243
--- /dev/null
+++ b/tests/models/similarity/test_clip_similarity.py
@@ -0,0 +1,82 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+from unittest.mock import MagicMock
+
+import torch
+
+from mmaction.registry import MODELS
+from mmaction.structures import ActionDataSample
+from mmaction.testing import get_similarity_cfg
+from mmaction.utils import register_all_modules
+
+
+def test_clip_similarity():
+ register_all_modules()
+ cfg = get_similarity_cfg(
+ 'clip4clip/'
+ 'clip4clip_vit-base-p32-res224-clip-pre_8xb16-u12-5e_msrvtt-9k-rgb.py')
+ cfg.model.frozen_layers = -1 # no frozen layers
+ model = MODELS.build(cfg.model)
+ model.train()
+
+ data_batch = {
+ 'inputs': {
+ 'imgs': [torch.randint(0, 256, (2, 3, 224, 224))],
+ 'text': [torch.randint(0, 49408, (77, ))]
+ },
+ 'data_samples': [ActionDataSample()]
+ }
+
+ # test train_step
+ optim_wrapper = MagicMock()
+ loss_vars = model.train_step(data_batch, optim_wrapper)
+ assert 'loss' in loss_vars
+ assert 'sim_loss_v2t' in loss_vars
+ assert 'sim_loss_t2v' in loss_vars
+ optim_wrapper.update_params.assert_called_once()
+
+ # test test_step
+ with torch.no_grad():
+ predictions = model.test_step(data_batch)
+ features = predictions[0].features
+ assert len(predictions) == 1
+ assert features.video_feature.size() == (512, )
+ assert features.text_feature.size() == (512, )
+
+ # test frozen layers
+ def check_frozen_layers(mdl, frozen_layers):
+ if frozen_layers >= 0:
+ top_layers = [
+ 'ln_final', 'text_projection', 'logit_scale', 'visual.ln_post',
+ 'visual.proj'
+ ]
+ mid_layers = [
+ 'visual.transformer.resblocks', 'transformer.resblocks'
+ ]
+
+ for name, param in mdl.clip.named_parameters():
+ if any(name.find(n) == 0 for n in top_layers):
+ assert param.requires_grad is True
+ elif any(name.find(n) == 0 for n in mid_layers):
+ layer_n = int(name.split('.resblocks.')[1].split('.')[0])
+ if layer_n >= frozen_layers:
+ assert param.requires_grad is True
+ else:
+ assert param.requires_grad is False
+ else:
+ assert param.requires_grad is False
+ else:
+ assert all([p.requires_grad for p in mdl.clip.parameters()])
+
+ check_frozen_layers(model, -1)
+
+ model.frozen_layers = 0
+ model.train()
+ check_frozen_layers(model, 0)
+
+ model.frozen_layers = 6
+ model.train()
+ check_frozen_layers(model, 6)
+
+ model.frozen_layers = 12
+ model.train()
+ check_frozen_layers(model, 12)
diff --git a/tests/models/utils/test_gradcam.py b/tests/models/utils/test_gradcam.py
index b53d37886f..e9568531c5 100644
--- a/tests/models/utils/test_gradcam.py
+++ b/tests/models/utils/test_gradcam.py
@@ -138,6 +138,7 @@ def test_slowfast():
_do_test_3D_models(recognizer, target_layer_name, input_shape)
+@pytest.mark.skipif(platform.system() == 'Windows', reason='Windows mem limit')
def test_tsm():
config = get_recognizer_cfg(
'tsm/tsm_imagenet-pretrained-r50_8xb16-1x1x8-50e_kinetics400-rgb.py')
@@ -158,6 +159,7 @@ def test_tsm():
_do_test_2D_models(recognizer, target_layer_name, input_shape)
+@pytest.mark.skipif(platform.system() == 'Windows', reason='Windows mem limit')
def test_csn():
config = get_recognizer_cfg(
'csn/ipcsn_ig65m-pretrained-r152-bnfrozen_32x2x1-58e_kinetics400-rgb.py' # noqa: E501
diff --git a/tools/analysis_tools/confusion_matrix.py b/tools/analysis_tools/confusion_matrix.py
index 224b8364bc..696537ea4a 100644
--- a/tools/analysis_tools/confusion_matrix.py
+++ b/tools/analysis_tools/confusion_matrix.py
@@ -1,5 +1,6 @@
# Copyright (c) OpenMMLab. All rights reserved.
import argparse
+import logging
import tempfile
import torch
@@ -83,6 +84,7 @@ def main():
classes = runner.test_loop.dataloader.dataset.metainfo.get(
'classes')
cm = runner.test()['confusion_matrix/result']
+ logging.shutdown()
else:
predictions = load(args.ckpt_or_result)
evaluator = Evaluator(ConfusionMatrix())
diff --git a/tools/analysis_tools/eval_metric.py b/tools/analysis_tools/eval_metric.py
index 08b6da31e2..9e61ddc843 100644
--- a/tools/analysis_tools/eval_metric.py
+++ b/tools/analysis_tools/eval_metric.py
@@ -5,6 +5,7 @@
from mmengine import Config, DictAction
from mmengine.evaluator import Evaluator
from mmengine.registry import init_default_scope
+from rich import print
def parse_args():
diff --git a/tools/analysis_tools/report_accuracy.py b/tools/analysis_tools/report_accuracy.py
index 2008f9fb46..c361f644de 100644
--- a/tools/analysis_tools/report_accuracy.py
+++ b/tools/analysis_tools/report_accuracy.py
@@ -1,11 +1,13 @@
# Copyright (c) OpenMMLab. All rights reserved.
import argparse
+import numpy as np
from mmengine import load
from scipy.special import softmax
from mmaction.evaluation.functional import (get_weighted_score,
mean_class_accuracy,
+ mmit_mean_average_precision,
top_k_accuracy)
@@ -23,6 +25,10 @@ def parse_args():
help='coefficients of each score file',
default=[1.0, 1.0])
parser.add_argument('--apply-softmax', action='store_true')
+ parser.add_argument(
+ '--multi-label',
+ action='store_true',
+ help='whether the task is multi label classification')
args = parser.parse_args()
return args
@@ -37,9 +43,16 @@ def main():
sample['pred_scores']['item'].numpy() for sample in data_samples
]
score_list.append(scores)
- labels = [
- sample['gt_labels']['item'].item() for sample in data_sample_list[0]
- ]
+
+ if args.multi_label:
+ labels = [
+ sample['gt_labels']['item'] for sample in data_sample_list[0]
+ ]
+ else:
+ labels = [
+ sample['gt_labels']['item'].item()
+ for sample in data_sample_list[0]
+ ]
if args.apply_softmax:
@@ -49,11 +62,16 @@ def apply_softmax(scores):
score_list = [apply_softmax(scores) for scores in score_list]
weighted_scores = get_weighted_score(score_list, args.coefficients)
- mean_class_acc = mean_class_accuracy(weighted_scores, labels)
- top_1_acc, top_5_acc = top_k_accuracy(weighted_scores, labels, (1, 5))
- print(f'Mean Class Accuracy: {mean_class_acc:.04f}')
- print(f'Top 1 Accuracy: {top_1_acc:.04f}')
- print(f'Top 5 Accuracy: {top_5_acc:.04f}')
+ if args.multi_label:
+ mean_avg_prec = mmit_mean_average_precision(
+ np.array(weighted_scores), np.stack([t.numpy() for t in labels]))
+ print(f'MMit Average Precision: {mean_avg_prec:.04f}')
+ else:
+ mean_class_acc = mean_class_accuracy(weighted_scores, labels)
+ top_1_acc, top_5_acc = top_k_accuracy(weighted_scores, labels, (1, 5))
+ print(f'Mean Class Accuracy: {mean_class_acc:.04f}')
+ print(f'Top 1 Accuracy: {top_1_acc:.04f}')
+ print(f'Top 5 Accuracy: {top_5_acc:.04f}')
if __name__ == '__main__':
diff --git a/tools/analysis_tools/report_map.py b/tools/analysis_tools/report_map.py
index 2aa46a1c50..26d1bc0f9f 100644
--- a/tools/analysis_tools/report_map.py
+++ b/tools/analysis_tools/report_map.py
@@ -3,10 +3,10 @@
import os
import os.path as osp
-import mmcv
+import mmengine
import numpy as np
-from mmaction.core import ActivityNetLocalization
+from mmaction.evaluation import ActivityNetLocalization
args = None
@@ -17,9 +17,9 @@ def cuhk17_top1():
if not osp.exists('cuhk_anet17_pred.json'):
os.system('wget https://download.openmmlab.com/'
'mmaction/localization/cuhk_anet17_pred.json')
- proposal = mmcv.load(args.proposal)
+ proposal = mmengine.load(args.proposal)
results = proposal['results']
- cuhk_pred = mmcv.load('cuhk_anet17_pred.json')['results']
+ cuhk_pred = mmengine.load('cuhk_anet17_pred.json')['results']
def get_topk(preds, k):
preds.sort(key=lambda x: x['score'])
@@ -36,7 +36,7 @@ def get_topk(preds, k):
new_value.append(x)
results[k] = new_value
proposal['results'] = results
- mmcv.dump(proposal, args.det_output)
+ mmengine.dump(proposal, args.det_output)
cls_funcs = {'cuhk17_top1': cuhk17_top1}
diff --git a/tools/data/activitynet/README.md b/tools/data/activitynet/README.md
index f3286f6fc1..17daef6acd 100644
--- a/tools/data/activitynet/README.md
+++ b/tools/data/activitynet/README.md
@@ -78,7 +78,7 @@ For this case, the downloading scripts update the annotation file after download
### Step 3. Extract RGB and Flow
-Before extracting, please refer to [install.md](/docs/install.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
+Before extracting, please refer to [install.md](/docs/en/get_started/installation.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
Use following scripts to extract both RGB and Flow.
@@ -87,7 +87,7 @@ bash extract_frames.sh
```
The command above can generate images with new short edge 256. If you want to generate images with short edge 320 (320p), or with fix size 340x256, you can change the args `--new-short 256` to `--new-short 320` or `--new-width 340 --new-height 256`.
-More details can be found in [data_preparation](/docs/data_preparation.md)
+More details can be found in [prepare dataset](/docs/en/user_guides/prepare_dataset.md)
### Step 4. Generate File List for ActivityNet Finetuning
@@ -108,13 +108,29 @@ Both RGB models and Flow models are supported.
After finetuning TSN on ActivityNet, you can use it to extract both RGB and Flow feature.
```shell
-python tsn_feature_extraction.py --data-prefix ../../../data/ActivityNet/rawframes --data-list ../../../data/ActivityNet/anet_train_video.txt --output-prefix ../../../data/ActivityNet/rgb_feat --modality RGB --ckpt /path/to/rgb_checkpoint.pth
-
-python tsn_feature_extraction.py --data-prefix ../../../data/ActivityNet/rawframes --data-list ../../../data/ActivityNet/anet_val_video.txt --output-prefix ../../../data/ActivityNet/rgb_feat --modality RGB --ckpt /path/to/rgb_checkpoint.pth
-
-python tsn_feature_extraction.py --data-prefix ../../../data/ActivityNet/rawframes --data-list ../../../data/ActivityNet/anet_train_video.txt --output-prefix ../../../data/ActivityNet/flow_feat --modality Flow --ckpt /path/to/flow_checkpoint.pth
-
-python tsn_feature_extraction.py --data-prefix ../../../data/ActivityNet/rawframes --data-list ../../../data/ActivityNet/anet_val_video.txt --output-prefix ../../../data/ActivityNet/flow_feat --modality Flow --ckpt /path/to/flow_checkpoint.pth
+python ../../misc/clip_feature_extraction.py tsn_extract_rgb_feat_config.py \
+ /path/to/rgb_checkpoint.pth ../../../data/ActivityNet/rgb_tarin_feat.pkl \
+ --video-list ../../../data/ActivityNet/anet_train_video.txt \
+ --video-root ../../../data/ActivityNet/rawframes \
+ --dump-score
+
+python ../../misc/clip_feature_extraction.py tsn_extract_rgb_feat_config.py \
+ path/to/rgb_checkpoint.pth ../../../data/ActivityNet/rgb_val_feat.pkl \
+ --video-list ../../../data/ActivityNet/anet_val_video.txt \
+ --video-root ../../../data/ActivityNet/rawframes \
+ --dump-score
+
+python ../../misc/clip_feature_extraction.py tsn_extract_flow_feat_config.py \
+ /path/to/flow_checkpoint.pth ../../../data/ActivityNet/flow_tarin_feat.pkl \
+ --video-list ../../../data/ActivityNet/anet_train_video.txt \
+ --video-root ../../../data/ActivityNet/rawframes \
+ --dump-score
+
+python ../../misc/clip_feature_extraction.py tsn_extract_flow_feat_config.py \
+ /path/to/flow_checkpoint.pth ../../../data/ActivityNet/flow_val_feat.pkl \
+ --video-list ../../../data/ActivityNet/anet_val_video.txt \
+ --video-root ../../../data/ActivityNet/rawframes \
+ --dump-score
```
After feature extraction, you can use our post processing scripts to concat RGB and Flow feature, generate the `100-t X 400-d` feature for Action Detection.
@@ -168,4 +184,4 @@ mmaction2
```
-For training and evaluating on ActivityNet, please refer to [getting_started.md](/docs/getting_started.md).
+For training and evaluating on ActivityNet, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).
diff --git a/tools/data/activitynet/README_zh-CN.md b/tools/data/activitynet/README_zh-CN.md
index 7687b948db..83969041c2 100644
--- a/tools/data/activitynet/README_zh-CN.md
+++ b/tools/data/activitynet/README_zh-CN.md
@@ -78,7 +78,7 @@ bash download_bsn_videos.sh
### 步骤 3. 抽取 RGB 帧和光流
-在抽取视频帧和光流之前，请参考 [安装指南](/docs_zh_CN/install.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
+在抽取视频帧和光流之前，请参考 [安装指南](/docs/zh_cn/get_started/installation.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
可使用以下命令抽取视频帧和光流。
@@ -87,7 +87,7 @@ bash extract_frames.sh
```
以上脚本将会生成短边 256 分辨率的视频。如果用户想生成短边 320 分辨率的视频（即 320p），或者 340x256 的固定分辨率，用户可以通过改变参数由 `--new-short 256` 至 `--new-short 320`，或者 `--new-width 340 --new-height 256` 进行设置
-更多细节可参考 [数据准备指南](/docs_zh_CN/data_preparation.md)
+更多细节可参考 [数据准备指南](/docs/zh_cn/user_guides/prepare_dataset.md)
### 步骤 4. 生成用于 ActivityNet 微调的文件列表
@@ -107,13 +107,29 @@ python generate_rawframes_filelist.py
在 ActivityNet 上微调 TSN 模型之后，用户可以使用该模型进行 RGB 特征和光流特征的提取。
```shell
-python tsn_feature_extraction.py --data-prefix ../../../data/ActivityNet/rawframes --data-list ../../../data/ActivityNet/anet_train_video.txt --output-prefix ../../../data/ActivityNet/rgb_feat --modality RGB --ckpt /path/to/rgb_checkpoint.pth
-
-python tsn_feature_extraction.py --data-prefix ../../../data/ActivityNet/rawframes --data-list ../../../data/ActivityNet/anet_val_video.txt --output-prefix ../../../data/ActivityNet/rgb_feat --modality RGB --ckpt /path/to/rgb_checkpoint.pth
-
-python tsn_feature_extraction.py --data-prefix ../../../data/ActivityNet/rawframes --data-list ../../../data/ActivityNet/anet_train_video.txt --output-prefix ../../../data/ActivityNet/flow_feat --modality Flow --ckpt /path/to/flow_checkpoint.pth
-
-python tsn_feature_extraction.py --data-prefix ../../../data/ActivityNet/rawframes --data-list ../../../data/ActivityNet/anet_val_video.txt --output-prefix ../../../data/ActivityNet/flow_feat --modality Flow --ckpt /path/to/flow_checkpoint.pth
+python ../../misc/clip_feature_extraction.py tsn_extract_rgb_feat_config.py \
+ /path/to/rgb_checkpoint.pth ../../../data/ActivityNet/rgb_tarin_feat.pkl \
+ --video-list ../../../data/ActivityNet/anet_train_video.txt \
+ --video-root ../../../data/ActivityNet/rawframes \
+ --dump-score
+
+python ../../misc/clip_feature_extraction.py tsn_extract_rgb_feat_config.py \
+ path/to/rgb_checkpoint.pth ../../../data/ActivityNet/rgb_val_feat.pkl \
+ --video-list ../../../data/ActivityNet/anet_val_video.txt \
+ --video-root ../../../data/ActivityNet/rawframes \
+ --dump-score
+
+python ../../misc/clip_feature_extraction.py tsn_extract_flow_feat_config.py \
+ /path/to/flow_checkpoint.pth ../../../data/ActivityNet/flow_tarin_feat.pkl \
+ --video-list ../../../data/ActivityNet/anet_train_video.txt \
+ --video-root ../../../data/ActivityNet/rawframes \
+ --dump-score
+
+python ../../misc/clip_feature_extraction.py tsn_extract_flow_feat_config.py \
+ /path/to/flow_checkpoint.pth ../../../data/ActivityNet/flow_val_feat.pkl \
+ --video-list ../../../data/ActivityNet/anet_val_video.txt \
+ --video-root ../../../data/ActivityNet/rawframes \
+ --dump-score
```
在提取完特征后，用户可以使用后处理脚本整合 RGB 特征和光流特征，生成 `100-t X 400-d` 维度的特征用于时序动作检测。
@@ -166,4 +182,4 @@ mmaction2
```
-关于对 ActivityNet 进行训练和验证，可以参考 [基础教程](/docs_zh_CN/getting_started.md).
+关于对 ActivityNet 进行训练和验证，可以参考 [训练教程](/docs/zh_cn/user_guides/train_test.md).
diff --git a/tools/data/activitynet/activitynet_feature_postprocessing.py b/tools/data/activitynet/activitynet_feature_postprocessing.py
index 8dcd7bfe26..e1d8369886 100644
--- a/tools/data/activitynet/activitynet_feature_postprocessing.py
+++ b/tools/data/activitynet/activitynet_feature_postprocessing.py
@@ -6,7 +6,7 @@
import numpy as np
import scipy.interpolate
-from mmcv import dump, load
+from mmengine import dump, load
args = None
@@ -88,9 +88,13 @@ def merge_feat(name):
def main():
global args
args = parse_args()
- rgb_feat = os.listdir(args.rgb)
- flow_feat = os.listdir(args.flow)
+ rgb_feat = [file for file in os.listdir(args.rgb) if file.endswith('.pkl')]
+ flow_feat = [
+ file for file in os.listdir(args.flow) if file.endswith('.pkl')
+ ]
assert set(rgb_feat) == set(flow_feat)
+ # for feat in rgb_feat:
+ # merge_feat(feat)
pool = multiprocessing.Pool(32)
pool.map(merge_feat, rgb_feat)
diff --git a/tools/data/activitynet/convert_proposal_format.py b/tools/data/activitynet/convert_proposal_format.py
index f2f8613eb4..6f5992c75a 100644
--- a/tools/data/activitynet/convert_proposal_format.py
+++ b/tools/data/activitynet/convert_proposal_format.py
@@ -4,16 +4,16 @@
P-GCN, not including TSN, I3D etc.)."""
import argparse
-import mmcv
+import mmengine
import numpy as np
-from mmaction.core import pairwise_temporal_iou
+from mmaction.evaluation import pairwise_temporal_iou
def load_annotations(ann_file):
"""Load the annotation according to ann_file into video_infos."""
video_infos = []
- anno_database = mmcv.load(ann_file)
+ anno_database = mmengine.load(ann_file)
for video_name in anno_database:
video_info = anno_database[video_name]
video_info['video_name'] = video_name
@@ -143,7 +143,7 @@ def parse_args():
video_infos = load_annotations(args.ann_file)
ground_truth = import_ground_truth(video_infos, activity_index)
proposal, num_proposals = import_proposals(
- mmcv.load(args.proposal_file)['results'])
+ mmengine.load(args.proposal_file)['results'])
video_idx = 0
for video_info in video_infos:
diff --git a/tools/data/activitynet/download.py b/tools/data/activitynet/download.py
index 1d1bf41a2d..4e614e83b7 100644
--- a/tools/data/activitynet/download.py
+++ b/tools/data/activitynet/download.py
@@ -7,7 +7,7 @@
import ssl
import subprocess
-import mmcv
+import mmengine
from joblib import Parallel, delayed
ssl._create_default_https_context = ssl._create_unverified_context
@@ -101,7 +101,7 @@ def parse_activitynet_annotations(input_csv, is_bsn_case=False):
# YoutubeIDs do not have prefix `v_`
youtube_ids = [x.split(',')[0][2:] for x in lines]
else:
- data = mmcv.load(anno_file)['database']
+ data = mmengine.load(anno_file)['database']
youtube_ids = list(data.keys())
return youtube_ids
@@ -125,15 +125,15 @@ def main(input_csv, output_dir, anno_file, num_jobs=24, is_bsn_case=False):
for index in youtube_ids)
# Save download report.
- mmcv.dump(status_list, 'download_report.json')
- annotation = mmcv.load(anno_file)
+ mmengine.dump(status_list, 'download_report.json')
+ annotation = mmengine.load(anno_file)
downloaded = {status[0]: status[1] for status in status_list}
annotation = {k: v for k, v in annotation.items() if downloaded[k]}
if is_bsn_case:
anno_file_bak = anno_file.replace('.json', '_bak.json')
os.rename(anno_file, anno_file_bak)
- mmcv.dump(annotation, anno_file)
+ mmengine.dump(annotation, anno_file)
if __name__ == '__main__':
diff --git a/tools/data/activitynet/tsn_extract_flow_feat_config.py b/tools/data/activitynet/tsn_extract_flow_feat_config.py
new file mode 100644
index 0000000000..e09c3f99c1
--- /dev/null
+++ b/tools/data/activitynet/tsn_extract_flow_feat_config.py
@@ -0,0 +1,43 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+_base_ = [
+ 'mmaction::_base_/models/tsn_r50.py', 'mmaction::_base_/default_runtime.py'
+]
+
+clip_len = 5
+model = dict(
+ backbone=dict(in_channels=2 * clip_len),
+ data_preprocessor=dict(mean=[128], std=[128]))
+
+# dataset settings
+dataset_type = 'RawframeDataset'
+data_root_val = 'data/kinetics400/rawframes_val'
+ann_file_val = 'data/kinetics400/kinetics400_val_list_rawframes.txt'
+
+file_client_args = dict(io_backend='disk')
+
+test_pipeline = [
+ dict(type='UntrimmedSampleFrames', clip_len=clip_len, clip_interval=16),
+ dict(type='RawFrameDecode', **file_client_args),
+ dict(type='Resize', scale=(-1, 256)),
+ dict(type='CenterCrop', crop_size=256),
+ dict(type='FormatShape', input_format='NCHW_Flow'),
+ dict(type='PackActionInputs')
+]
+
+test_dataloader = dict(
+ batch_size=1,
+ num_workers=8,
+ persistent_workers=True,
+ sampler=dict(type='DefaultSampler', shuffle=False),
+ dataset=dict(
+ type=dataset_type,
+ ann_file=ann_file_val,
+ data_prefix=dict(img=data_root_val),
+ pipeline=test_pipeline,
+ filename_tmpl='{}_{:05d}.jpg',
+ modality='Flow',
+ test_mode=True))
+
+test_evaluator = []
+
+test_cfg = dict(type='TestLoop')
diff --git a/tools/data/activitynet/tsn_extract_rgb_feat_config.py b/tools/data/activitynet/tsn_extract_rgb_feat_config.py
new file mode 100644
index 0000000000..803e031935
--- /dev/null
+++ b/tools/data/activitynet/tsn_extract_rgb_feat_config.py
@@ -0,0 +1,36 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+_base_ = [
+ 'mmaction::_base_/models/tsn_r50.py', 'mmaction::_base_/default_runtime.py'
+]
+
+# dataset settings
+dataset_type = 'RawframeDataset'
+data_root_val = 'data/kinetics400/rawframes_val'
+ann_file_val = 'data/kinetics400/kinetics400_val_list_rawframes.txt'
+
+file_client_args = dict(io_backend='disk')
+
+test_pipeline = [
+ dict(type='UntrimmedSampleFrames', clip_len=1, clip_interval=16),
+ dict(type='RawFrameDecode', **file_client_args),
+ dict(type='Resize', scale=(-1, 256)),
+ dict(type='CenterCrop', crop_size=256),
+ dict(type='FormatShape', input_format='NCHW'),
+ dict(type='PackActionInputs')
+]
+
+test_dataloader = dict(
+ batch_size=1,
+ num_workers=8,
+ persistent_workers=True,
+ sampler=dict(type='DefaultSampler', shuffle=False),
+ dataset=dict(
+ type=dataset_type,
+ ann_file=ann_file_val,
+ data_prefix=dict(img=data_root_val),
+ pipeline=test_pipeline,
+ test_mode=True))
+
+test_evaluator = []
+
+test_cfg = dict(type='TestLoop')
diff --git a/tools/data/activitynet/tsn_extract_video_feat_config.py b/tools/data/activitynet/tsn_extract_video_feat_config.py
new file mode 100644
index 0000000000..ab815c1318
--- /dev/null
+++ b/tools/data/activitynet/tsn_extract_video_feat_config.py
@@ -0,0 +1,37 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+_base_ = [
+ 'mmaction::_base_/models/tsn_r50.py', 'mmaction::_base_/default_runtime.py'
+]
+
+# dataset settings
+dataset_type = 'VideoDataset'
+data_root_val = 'data/kinetics400/videos_val'
+ann_file_val = 'data/kinetics400/kinetics400_val_list_videos.txt'
+
+file_client_args = dict(io_backend='disk')
+
+test_pipeline = [
+ dict(type='DecordInit', **file_client_args),
+ dict(type='UntrimmedSampleFrames', clip_len=1, clip_interval=16),
+ dict(type='DecordDecode'),
+ dict(type='Resize', scale=(-1, 256)),
+ dict(type='CenterCrop', crop_size=256),
+ dict(type='FormatShape', input_format='NCHW'),
+ dict(type='PackActionInputs')
+]
+
+test_dataloader = dict(
+ batch_size=1,
+ num_workers=8,
+ persistent_workers=True,
+ sampler=dict(type='DefaultSampler', shuffle=False),
+ dataset=dict(
+ type=dataset_type,
+ ann_file=ann_file_val,
+ data_prefix=dict(video=data_root_val),
+ pipeline=test_pipeline,
+ test_mode=True))
+
+test_evaluator = []
+
+test_cfg = dict(type='TestLoop')
diff --git a/tools/data/activitynet/tsn_feature_extraction.py b/tools/data/activitynet/tsn_feature_extraction.py
deleted file mode 100644
index db8318a533..0000000000
--- a/tools/data/activitynet/tsn_feature_extraction.py
+++ /dev/null
@@ -1,149 +0,0 @@
-# Copyright (c) OpenMMLab. All rights reserved.
-import argparse
-import os
-import os.path as osp
-import pickle
-
-import mmcv
-import numpy as np
-import torch
-
-from mmaction.datasets.transforms import Compose
-from mmaction.models import build_model
-
-
-def parse_args():
- parser = argparse.ArgumentParser(description='Extract TSN Feature')
- parser.add_argument('--data-prefix', default='', help='dataset prefix')
- parser.add_argument('--output-prefix', default='', help='output prefix')
- parser.add_argument(
- '--data-list',
- help='video list of the dataset, the format should be '
- '`frame_dir num_frames output_file`')
- parser.add_argument(
- '--frame-interval',
- type=int,
- default=16,
- help='the sampling frequency of frame in the untrimed video')
- parser.add_argument('--modality', default='RGB', choices=['RGB', 'Flow'])
- parser.add_argument('--ckpt', help='checkpoint for feature extraction')
- parser.add_argument(
- '--part',
- type=int,
- default=0,
- help='which part of dataset to forward(alldata[part::total])')
- parser.add_argument(
- '--total', type=int, default=1, help='how many parts exist')
- args = parser.parse_args()
- return args
-
-
-def main():
- args = parse_args()
- args.is_rgb = args.modality == 'RGB'
- args.clip_len = 1 if args.is_rgb else 5
- args.input_format = 'NCHW' if args.is_rgb else 'NCHW_Flow'
- rgb_norm_cfg = dict(
- mean=[123.675, 116.28, 103.53],
- std=[58.395, 57.12, 57.375],
- to_bgr=False)
- flow_norm_cfg = dict(mean=[128, 128], std=[128, 128])
- args.img_norm_cfg = rgb_norm_cfg if args.is_rgb else flow_norm_cfg
- args.f_tmpl = 'img_{:05d}.jpg' if args.is_rgb else 'flow_{}_{:05d}.jpg'
- args.in_channels = args.clip_len * (3 if args.is_rgb else 2)
- # max batch_size for one forward
- args.batch_size = 200
-
- # define the data pipeline for Untrimmed Videos
- data_pipeline = [
- dict(
- type='UntrimmedSampleFrames',
- clip_len=args.clip_len,
- frame_interval=args.frame_interval,
- start_index=0),
- dict(type='RawFrameDecode'),
- dict(type='Resize', scale=(-1, 256)),
- dict(type='CenterCrop', crop_size=256),
- dict(type='Normalize', **args.img_norm_cfg),
- dict(type='FormatShape', input_format=args.input_format),
- dict(type='Collect', keys=['imgs'], meta_keys=[]),
- dict(type='ToTensor', keys=['imgs'])
- ]
- data_pipeline = Compose(data_pipeline)
-
- # define TSN R50 model, the model is used as the feature extractor
- model_cfg = dict(
- type='Recognizer2D',
- backbone=dict(
- type='ResNet',
- depth=50,
- in_channels=args.in_channels,
- norm_eval=False),
- cls_head=dict(
- type='TSNHead',
- num_classes=200,
- in_channels=2048,
- spatial_type='avg',
- consensus=dict(type='AvgConsensus', dim=1)),
- test_cfg=dict(average_clips=None))
- model = build_model(model_cfg)
- # load pretrained weight into the feature extractor
- state_dict = torch.load(args.ckpt)['state_dict']
- model.load_state_dict(state_dict)
- model = model.cuda()
- model.eval()
-
- data = open(args.data_list).readlines()
- data = [x.strip() for x in data]
- data = data[args.part::args.total]
-
- # enumerate Untrimmed videos, extract feature from each of them
- prog_bar = mmcv.ProgressBar(len(data))
- if not osp.exists(args.output_prefix):
- os.system(f'mkdir -p {args.output_prefix}')
-
- for item in data:
- frame_dir, length, _ = item.split()
- output_file = osp.basename(frame_dir) + '.pkl'
- frame_dir = osp.join(args.data_prefix, frame_dir)
- output_file = osp.join(args.output_prefix, output_file)
- assert output_file.endswith('.pkl')
- length = int(length)
-
- # prepare a pseudo sample
- tmpl = dict(
- frame_dir=frame_dir,
- total_frames=length,
- filename_tmpl=args.f_tmpl,
- start_index=0,
- modality=args.modality)
- sample = data_pipeline(tmpl)
- imgs = sample['imgs']
- shape = imgs.shape
- # the original shape should be N_seg * C * H * W, resize it to N_seg *
- # 1 * C * H * W so that the network return feature of each frame (No
- # score average among segments)
- imgs = imgs.reshape((shape[0], 1) + shape[1:])
- imgs = imgs.cuda()
-
- def forward_data(model, data):
- # chop large data into pieces and extract feature from them
- results = []
- start_idx = 0
- num_clip = data.shape[0]
- while start_idx < num_clip:
- with torch.no_grad():
- part = data[start_idx:start_idx + args.batch_size]
- feat = model.forward(part, return_loss=False)
- results.append(feat)
- start_idx += args.batch_size
- return np.concatenate(results)
-
- feat = forward_data(model, imgs)
- with open(output_file, 'wb') as fout:
- pickle.dump(feat, fout)
- prog_bar.update()
-
-
-if __name__ == '__main__':
- main()
diff --git a/tools/data/ava/README.md b/tools/data/ava/README.md
index a416eb2632..ed5089b7d1 100644
--- a/tools/data/ava/README.md
+++ b/tools/data/ava/README.md
@@ -64,7 +64,7 @@ bash cut_videos.sh
## Step 4. Extract RGB and Flow
-Before extracting, please refer to [install.md](/docs/install.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
+Before extracting, please refer to [install.md](/docs/en/get_started/installation.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
If you have plenty of SSD space, then we recommend extracting frames there for better I/O performance. And you can run the following script to soft link the extracted frames.
@@ -141,7 +141,7 @@ mmaction2
| │ │ │ ├── ...
```
-For training and evaluating on AVA, please refer to [getting_started](/docs/getting_started.md).
+For training and evaluating on AVA, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).
## Reference
diff --git a/tools/data/ava/README_zh-CN.md b/tools/data/ava/README_zh-CN.md
index 5a7b96da88..5a712798e5 100644
--- a/tools/data/ava/README_zh-CN.md
+++ b/tools/data/ava/README_zh-CN.md
@@ -56,7 +56,7 @@ bash cut_videos.sh
## 4. 提取 RGB 帧和光流
-在提取之前，请参考 [安装教程](/docs_zh_CN/install.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
+在提取之前，请参考 [安装指南](/docs/zh_cn/get_started/installation.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
如果用户有足够的 SSD 空间，那么建议将视频抽取为 RGB 帧以提升 I/O 性能。用户可以使用以下脚本为抽取得到的帧文件夹建立软连接：
@@ -131,4 +131,4 @@ mmaction2
| │ │ │ ├── ...
```
-关于 AVA 数据集上的训练与测试，请参照 [基础教程](/docs_zh_CN/getting_started.md)。
+关于 AVA 数据集上的训练与测试，请参照 [训练教程](/docs/zh_cn/user_guides/train_test.md)。
diff --git a/tools/data/ava/download_videos_parallel.py b/tools/data/ava/download_videos_parallel.py
index 7be4b1b883..524cf37d2e 100644
--- a/tools/data/ava/download_videos_parallel.py
+++ b/tools/data/ava/download_videos_parallel.py
@@ -3,7 +3,7 @@
import os.path as osp
import subprocess
-import mmcv
+import mmengine
from joblib import Parallel, delayed
URL_PREFIX = 'https://s3.amazonaws.com/ava-dataset/trainval/'
@@ -36,7 +36,7 @@ def download_video(video_url, output_dir, num_attempts=5):
def main(source_file, output_dir, num_jobs=24, num_attempts=5):
- mmcv.mkdir_or_exist(output_dir)
+ mmengine.mkdir_or_exist(output_dir)
video_list = open(source_file).read().strip().split('\n')
video_list = [osp.join(URL_PREFIX, video) for video in video_list]
@@ -49,7 +49,7 @@ def main(source_file, output_dir, num_jobs=24, num_attempts=5):
delayed(download_video)(video, output_dir, num_attempts)
for video in video_list)
- mmcv.dump(status_list, 'download_report.json')
+ mmengine.dump(status_list, 'download_report.json')
if __name__ == '__main__':
diff --git a/tools/data/ava_kinetics/prepare_annotation.py b/tools/data/ava_kinetics/prepare_annotation.py
index 00b7669d49..2e53916540 100644
--- a/tools/data/ava_kinetics/prepare_annotation.py
+++ b/tools/data/ava_kinetics/prepare_annotation.py
@@ -60,7 +60,7 @@ def filter_train_list(kinetics_anotation_file, lookup):
'--avakinetics_anotation',
type=str,
default='./ava_kinetics_v1_0',
- help='the directory to ava-kinetics anotations')
+ help='the directory to ava-kinetics annotations')
p.add_argument(
'--num_workers',
type=int,
diff --git a/tools/data/diving48/README.md b/tools/data/diving48/README.md
index 588cddd173..02d19d5a62 100644
--- a/tools/data/diving48/README.md
+++ b/tools/data/diving48/README.md
@@ -15,11 +15,28 @@
```
For basic dataset information, you can refer to the official dataset [website](http://www.svcl.ucsd.edu/projects/resound/dataset.html).
-Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/diving48/`.
+
+`````{tabs}
+
+````{group-tab} Download by MIM
+MIM supports downloading from OpenDataLab and preprocessing Diving48 dataset with one command line.
+```Bash
+# install OpenDataLab CLI tools
+pip install -U opendatalab
+# log in OpenDataLab
+odl login
+# download and preprocess by MIM
+mim download mmaction2 --dataset diving48
+```
+
+````
+
+````{group-tab} Download form Official Source
## Step 1. Prepare Annotations
You can run the following script to download annotations (considering the correctness of annotation files, we only download V2 version here).
+Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/diving48/`.
```shell
bash download_annotations.sh
@@ -39,7 +56,7 @@ This part is **optional** if you only want to use the video loader.
The frames provided in official compressed file are not complete. You may need to go through the following extraction steps to get the complete frames.
-Before extracting, please refer to [install.md](/docs/install.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
+Before extracting, please refer to [install.md](/docs/en/get_started/installation.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
If you have plenty of SSD space, then we recommend extracting frames there for better I/O performance.
@@ -81,7 +98,10 @@ bash generate_videos_filelist.sh
bash generate_rawframes_filelist.sh
```
-## Step 5. Check Directory Structure
+````
+`````
+
+### Check Directory Structure
After the whole data process for Diving48 preparation,
you will get the rawframes (RGB + Flow), videos and annotation files for Diving48.
@@ -97,7 +117,7 @@ mmaction2
│ ├── diving48
│ │ ├── diving48_{train,val}_list_rawframes.txt
│ │ ├── diving48_{train,val}_list_videos.txt
-│ │ ├── annotations
+│ │ ├── annotations (optinonal)
│ | | ├── Diving48_V2_train.json
│ | | ├── Diving48_V2_test.json
│ | | ├── Diving48_vocab.json
@@ -105,7 +125,7 @@ mmaction2
│ | | ├── _8Vy3dlHg2w_00000.mp4
│ | | ├── _8Vy3dlHg2w_00001.mp4
│ | | ├── ...
-│ | ├── rawframes
+│ | ├── rawframes (optional)
│ | | ├── 2x00lRzlTVQ_00000
│ | | | ├── img_00001.jpg
│ | | | ├── img_00002.jpg
@@ -120,4 +140,4 @@ mmaction2
│ | | ├── ...
```
-For training and evaluating on Diving48, please refer to [getting_started.md](/docs/getting_started.md).
+For training and evaluating on Diving48, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).
diff --git a/tools/data/diving48/README_zh-CN.md b/tools/data/diving48/README_zh-CN.md
index e91f8729a5..825344039e 100644
--- a/tools/data/diving48/README_zh-CN.md
+++ b/tools/data/diving48/README_zh-CN.md
@@ -15,11 +15,26 @@
```
用户可参考该数据集的 [官网](http://www.svcl.ucsd.edu/projects/resound/dataset.html)，以获取数据集相关的基本信息。
-在数据集准备前，请确保命令行当前路径为 `$MMACTION2/tools/data/diving48/`。
+`````{tabs}
+
+````{group-tab} 使用 MIM 下载
+# MIM 支持下载 Diving48 数据集。用户可以通过一行命令，从 OpenDataLab 进行下载，并进行预处理。
+```Bash
+# 安装 OpenDataLab CLI 工具
+pip install -U opendatalab
+# 登录 OpenDataLab
+odl login
+# 通过 MIM 进行数据集下载，预处理。注意这将花费较长时间
+mim download mmaction2 --dataset diving48
+```
+
+````
+
+````{group-tab} 从官方源下载
## 步骤 1. 下载标注文件
-用户可以使用以下命令下载标注文件（考虑到标注的准确性，这里仅下载 V2 版本）。
+用户可以使用以下命令下载标注文件（考虑到标注的准确性，这里仅下载 V2 版本）。在数据集准备前，请确保命令行当前路径为 `$MMACTION2/tools/data/diving48/`。
```shell
bash download_annotations.sh
@@ -39,7 +54,7 @@ bash download_videos.sh
官网提供的帧压缩包并不完整。若想获取完整的数据，可以使用以下步骤解帧。
-在抽取视频帧和光流之前，请参考 [安装指南](/docs_zh_CN/install.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
+在抽取视频帧和光流之前，请参考 [安装指南](/docs/zh_cn/get_started/installation.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
如果拥有大量的 SSD 存储空间，则推荐将抽取的帧存储至 I/O 性能更优秀的 SSD 中。
@@ -81,7 +96,10 @@ bash generate_videos_filelist.sh
bash generate_rawframes_filelist.sh
```
-## 步骤 5. 检查文件夹结构
+````
+`````
+
+### 检查文件夹结构
在完成所有 Diving48 数据集准备流程后，
用户可以获得对应的 RGB + 光流文件，视频文件以及标注文件。
@@ -97,7 +115,7 @@ mmaction2
│ ├── diving48
│ │ ├── diving48_{train,val}_list_rawframes.txt
│ │ ├── diving48_{train,val}_list_videos.txt
-│ │ ├── annotations
+│ │ ├── annotations（可选）
│ | | ├── Diving48_V2_train.json
│ | | ├── Diving48_V2_test.json
│ | | ├── Diving48_vocab.json
@@ -105,7 +123,7 @@ mmaction2
│ | | ├── _8Vy3dlHg2w_00000.mp4
│ | | ├── _8Vy3dlHg2w_00001.mp4
│ | | ├── ...
-│ | ├── rawframes
+│ | ├── rawframes（可选）
│ | | ├── 2x00lRzlTVQ_00000
│ | | | ├── img_00001.jpg
│ | | | ├── img_00002.jpg
@@ -120,4 +138,4 @@ mmaction2
│ | | ├── ...
```
-关于对 Diving48 进行训练和验证，可以参考 [基础教程](/docs_zh_CN/getting_started.md)。
+关于对 Diving48 进行训练和验证，请参考 [训练和测试教程](/docs/en/user_guides/train_test.md)。
diff --git a/tools/data/diving48/preprocess.sh b/tools/data/diving48/preprocess.sh
new file mode 100644
index 0000000000..10d9e42044
--- /dev/null
+++ b/tools/data/diving48/preprocess.sh
@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+
+DOWNLOAD_DIR=$1
+DATA_ROOT=$2
+
+cat $DOWNLOAD_DIR/diving48/raw/*.tar.gz.* | tar -xvz -C $(dirname $DATA_ROOT)
+tar -xvf $DATA_ROOT/diving48.tar -C $(dirname $DATA_ROOT)
+rm $DATA_ROOT/diving48.tar
diff --git a/tools/data/gym/README.md b/tools/data/gym/README.md
index a39eda6fd4..84386c012e 100644
--- a/tools/data/gym/README.md
+++ b/tools/data/gym/README.md
@@ -55,7 +55,7 @@ python trim_subaction.py
This part is **optional** if you only want to use the video loader for RGB model training.
-Before extracting, please refer to [install.md](/docs/install.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
+Before extracting, please refer to [install.md](/docs/en/get_started/installation.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
Run the following script to extract both rgb and flow using "tvl1" algorithm.
@@ -106,4 +106,4 @@ mmaction2
| | └── subaction_frames
```
-For training and evaluating on GYM, please refer to [getting_started](/docs/getting_started.md).
+For training and evaluating on GYM, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).
diff --git a/tools/data/gym/README_zh-CN.md b/tools/data/gym/README_zh-CN.md
index cb3a796ec7..0b3e54aeb1 100644
--- a/tools/data/gym/README_zh-CN.md
+++ b/tools/data/gym/README_zh-CN.md
@@ -55,7 +55,7 @@ python trim_subaction.py
如果用户仅使用 video loader，则可以跳过本步。
-在提取之前，请参考 [安装教程](/docs_zh_CN/install.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
+在提取之前，请参考 [安装指南](/docs/zh_cn/get_started/installation.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
用户可使用如下脚本同时抽取 RGB 帧和光流（提取光流时使用 tvl1 算法）：
@@ -106,4 +106,4 @@ mmaction2
| | └── subaction_frames
```
-关于 GYM 数据集上的训练与测试，请参照 [基础教程](/docs_zh_CN/getting_started.md)。
+关于 GYM 数据集上的训练与测试，请参照 [训练教程](/docs/zh_cn/user_guides/train_test.md)。
diff --git a/tools/data/hacs/README-CN.md b/tools/data/hacs/README-CN.md
new file mode 100644
index 0000000000..245d6313fe
--- /dev/null
+++ b/tools/data/hacs/README-CN.md
@@ -0,0 +1,119 @@
+# 准备 HACS Segments
+
+## 简介
+
+
+
+```BibTeX
+@inproceedings{zhao2019hacs,
+ title={Hacs: Human action clips and segments dataset for recognition and temporal localization},
+ author={Zhao, Hang and Torralba, Antonio and Torresani, Lorenzo and Yan, Zhicheng},
+ booktitle={Proceedings of the IEEE International Conference on Computer Vision},
+ pages={8668--8678},
+ year={2019}
+}
+```
+
+### 0. 下载视频
+
+在我们开始准备数据集之前，请按照[官方代码库](https://github.com/hangzhaomit/HACS-dataset)的指令下载HACS Segments数据集中的视频。如果有视频缺失，您可以向HACS数据集存储库的维护者提交请求以获取缺失的视频。但是如果一些视频缺失，您仍然可以为MMAction2准备数据集。
+
+在下载完数据集后，请将数据集文件夹移动到(或者使用软链接)`$MMACTION2/tools/data/hacs/`。文件夹结构应该如下所示：
+
+```
+mmaction2
+├── mmaction
+├── data
+├── configs
+├── tools
+│ ├── hacs
+│ │ ├── slowonly_feature_infer.py
+│ │ ├── ..
+│ │ ├── data
+│ │ │ ├── Applying_sunscreen
+│ │ │ │ ├── v_0Ch__DqMPwA.mp4
+│ │ │ │ ├── v_9CTDjFHl8WE.mp4
+│ │ │ │ ├── ..
+
+
+```
+
+在开始之前，请确保您位于`$MMACTION2/tools/data/hacs/`路径下。
+
+### 1. 提取特征
+
+以下是使用[SlowOnly ResNet50 8x8](/configs/recognition/slowonly/slowonly_imagenet-pretrained-r50_16xb16-4x16x1-steplr-150e_kinetics700-rgb.py)在Kinetics700数据集上预训练的模型，从HACS视频中提取特征。对于每个视频，我们均匀采样100个视频片段，并提取700维输出（softmax之前）作为特征，即特征形状为100x700。
+
+首先，我们使用如下命令生成数据集的视频列表：
+
+```
+python generate_list.py
+```
+
+这将生成一个位于`$MMACTION2/tools/data/hacs/`的`hacs_data.txt`文件，其内容格式如下：
+
+```
+Horseback_riding/v_Sr2BSq_8FMw.mp4 0
+Horseback_riding/v_EQb6OKoqz3Q.mp4 1
+Horseback_riding/v_vYKUV8TRngg.mp4 2
+Horseback_riding/v_Y8U0X1F-0ck.mp4 3
+Horseback_riding/v_hnspbB7wNh0.mp4 4
+Horseback_riding/v_HPhlhrT9IOk.mp4 5
+```
+
+接下来，我们使用[slowonly_feature_infer.py](/tools/data/hacs/slowonly_feature_infer.py) 配置文件来提取特征：
+
+```
+# 指定提取特征的GPU数量
+NUM_GPUS=8
+
+# 下载预训练模型权重
+wget https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_16xb16-8x8x1-steplr-150e_kinetics700-rgb/slowonly_imagenet-pretrained-r50_16xb16-8x8x1-steplr-150e_kinetics700-rgb_20221013-15b93b10.pth
+
+bash ../mmaction2/tools/dist_test.sh \
+ slowonly_feature_infer.py \
+ slowonly_imagenet-pretrained-r50_16xb16-8x8x1-steplr-150e_kinetics700-rgb_20221013-15b93b10.pth \
+ $NUM_GPUS --dump result.pkl
+```
+
+我们将得到一个名为 `result.pkl` 的文件，其中包含每个视频的大小为100x700的特征。我们将特征重写为csv格式，并保存在 `$MMACTION2/data/HACS/` 目录下。
+
+```
+＃确保您位于$ $MMACTION2/tools/data/hacs/
+python write_feature_csv.py
+```
+
+### 2. 准备标注文件
+
+我们首先从官方仓库下载标注文件：
+
+```
+wget https://github.com/hangzhaomit/HACS-dataset/raw/master/HACS_v1.1.1.zip
+unzip HACS_v1.1.1.zip
+```
+
+解压缩后，应该有一个名为`HACS_v1.1.1`的文件夹，其中包含一个名为`HACS_segments_v1.1.1.json`的文件。
+
+我们在`$MMACTION2/data/HACS/`目录下生成`hacs_anno_train.json`、`hacs_anno_val.json`和`hacs_anno_test.json`文件：
+
+```
+python3 generate_anotations.py
+```
+
+完成这两个步骤后，HACS Segments数据集的文件夹结构应该如下所示：
+
+```
+mmaction2
+├── mmaction
+├── data
+│ ├── HACS
+│ │ ├── hacs_anno_train.json
+│ │ ├── hacs_anno_val.json
+│ │ ├── hacs_anno_test.json
+│ │ ├── slowonly_feature
+│ │ │ ├── v_008gY2B8Pf4.csv
+│ │ │ ├── v_0095rqic1n8.csv
+├── configs
+├── tools
+
+```
diff --git a/tools/data/hacs/README.md b/tools/data/hacs/README.md
new file mode 100644
index 0000000000..d567b60277
--- /dev/null
+++ b/tools/data/hacs/README.md
@@ -0,0 +1,119 @@
+# Preparing HACS Segments
+
+## Introduction
+
+
+
+```BibTeX
+@inproceedings{zhao2019hacs,
+ title={Hacs: Human action clips and segments dataset for recognition and temporal localization},
+ author={Zhao, Hang and Torralba, Antonio and Torresani, Lorenzo and Yan, Zhicheng},
+ booktitle={Proceedings of the IEEE International Conference on Computer Vision},
+ pages={8668--8678},
+ year={2019}
+}
+```
+
+### Step 0. Download Videos
+
+Before we start preparing the dataset, please following the official [repository](https://github.com/hangzhaomit/HACS-dataset) to download videos from the HACS Segments dataset. You can submit a request for missing videos to the maintainer of the HACS dataset repository. But you can still prepare the dataset for MMAction2 if some videos are missing.
+
+After you finish downloading the dataset, please move the dataset folder to `$MMACTION2/tools/data/hacs/` or use a soft link. The the folder structure should look like:
+
+```
+mmaction2
+├── mmaction
+├── data
+├── configs
+├── tools
+│ ├── hacs
+│ │ ├── slowonly_feature_infer.py
+│ │ ├── ..
+│ │ ├── data
+│ │ │ ├── Applying_sunscreen
+│ │ │ │ ├── v_0Ch__DqMPwA.mp4
+│ │ │ │ ├── v_9CTDjFHl8WE.mp4
+│ │ │ │ ├── ..
+
+
+```
+
+Before we start, make sure you are at `$MMACTION2/tools/data/hacs/`.
+
+### Step 1. Extract Features
+
+We extract features from the HACS videos using [SlowOnly ResNet50 8x8](/configs/recognition/slowonly/slowonly_imagenet-pretrained-r50_16xb16-4x16x1-steplr-150e_kinetics700-rgb.py) pretrained on Kinetics700 dataset. For each video, we uniformly sample 100 video clips and extract the 700-dimensional output (before softmax) as the feature, i.e., the feature shape is 100x700.
+
+First, we generate a video list of the dataset:
+
+```
+python generate_list.py
+```
+
+It will generate an `hacs_data.txt` file located at `$MMACTION2/tools/data/hacs/` which looks like:
+
+```
+Horseback_riding/v_Sr2BSq_8FMw.mp4 0
+Horseback_riding/v_EQb6OKoqz3Q.mp4 1
+Horseback_riding/v_vYKUV8TRngg.mp4 2
+Horseback_riding/v_Y8U0X1F-0ck.mp4 3
+Horseback_riding/v_hnspbB7wNh0.mp4 4
+Horseback_riding/v_HPhlhrT9IOk.mp4 5
+```
+
+Next we use the [slowonly_feature_infer.py](/tools/data/hacs/slowonly_feature_infer.py) config to extract features:
+
+```
+# number of GPUs to extract feature
+NUM_GPUS=8
+
+# download the pretraining checkpoint
+wget https://download.openmmlab.com/mmaction/v1.0/recognition/slowonly/slowonly_imagenet-pretrained-r50_16xb16-8x8x1-steplr-150e_kinetics700-rgb/slowonly_imagenet-pretrained-r50_16xb16-8x8x1-steplr-150e_kinetics700-rgb_20221013-15b93b10.pth
+
+bash ../mmaction2/tools/dist_test.sh \
+ slowonly_feature_infer.py \
+ slowonly_imagenet-pretrained-r50_16xb16-8x8x1-steplr-150e_kinetics700-rgb_20221013-15b93b10.pth \
+ $NUM_GPUS --dump result.pkl
+```
+
+We will get a `result.pkl` that contains the 100x700 feature for each video. We re-write the features into csv format at `$MMACTION2/data/HACS/`:
+
+```
+# Make sure you are at $MMACTION2/tools/data/hacs/
+python write_feature_csv.py
+```
+
+### Step 2. Prepare Annotations
+
+We first download the original annotations from the official repository:
+
+```
+wget https://github.com/hangzhaomit/HACS-dataset/raw/master/HACS_v1.1.1.zip
+unzip HACS_v1.1.1.zip
+```
+
+After unzipping, there should be an `HACS_v1.1.1` folder with an `HACS_segments_v1.1.1.json` file in it.
+
+We generate `hacs_anno_train.json`, `hacs_anno_val.json` and `hacs_anno_test.json` files at `$MMACTION2/data/HACS/`:
+
+```
+python3 generate_anotations.py
+```
+
+After the two steps finished, the folder structure of the HACS Segments dataset should look like:
+
+```
+mmaction2
+├── mmaction
+├── data
+│ ├── HACS
+│ │ ├── hacs_anno_train.json
+│ │ ├── hacs_anno_val.json
+│ │ ├── hacs_anno_test.json
+│ │ ├── slowonly_feature
+│ │ │ ├── v_008gY2B8Pf4.csv
+│ │ │ ├── v_0095rqic1n8.csv
+├── configs
+├── tools
+
+```
diff --git a/tools/data/hacs/generate_anotations.py b/tools/data/hacs/generate_anotations.py
new file mode 100644
index 0000000000..6bcc5bf047
--- /dev/null
+++ b/tools/data/hacs/generate_anotations.py
@@ -0,0 +1,58 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+import json
+import multiprocessing
+import os
+
+import decord
+
+with open('HACS_v1.1.1/HACS_segments_v1.1.1.json') as f:
+ all_annotations = json.load(f)['database']
+
+
+def parse_anno(key):
+ anno = {}
+ anno['duration_second'] = float(all_annotations[key]['duration'])
+ anno['annotations'] = all_annotations[key]['annotations']
+ anno['subset'] = all_annotations[key]['subset']
+
+ labels = set([i['label'] for i in anno['annotations']])
+ num_frames = int(anno['duration_second'] * 30)
+ for label in labels:
+ path = f'data/{label}/v_{key}.mp4'
+ if os.path.isfile(path):
+ vr = decord.VideoReader(path)
+ num_frames = len(vr)
+ break
+
+ anno['feature_frame'] = anno['duration_frame'] = num_frames
+ anno['key'] = f'v_{key}'
+ return anno
+
+
+pool = multiprocessing.Pool(16)
+video_list = list(all_annotations)
+outputs = pool.map(parse_anno, video_list)
+
+train_anno = {}
+val_anno = {}
+test_anno = {}
+
+for anno in outputs:
+ key = anno.pop('key')
+ subset = anno.pop('subset')
+ if subset == 'training':
+ train_anno[key] = anno
+ elif subset == 'validation':
+ val_anno[key] = anno
+ else:
+ test_anno[key] = anno
+
+outdir = '../../../data/HACS'
+with open(f'{outdir}/hacs_anno_train.json', 'w') as f:
+ json.dump(train_anno, f)
+
+with open(f'{outdir}/hacs_anno_val.json', 'w') as f:
+ json.dump(val_anno, f)
+
+with open(f'{outdir}/hacs_anno_test.json', 'w') as f:
+ json.dump(test_anno, f)
diff --git a/tools/data/hacs/generate_list.py b/tools/data/hacs/generate_list.py
new file mode 100644
index 0000000000..f659c7f29a
--- /dev/null
+++ b/tools/data/hacs/generate_list.py
@@ -0,0 +1,17 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+import os
+
+data_root = './data'
+
+video_list = []
+idx = 0
+for folder in os.listdir(data_root):
+ path = f'{data_root}/{folder}'
+ for video in os.listdir(path):
+ line = f'{folder}/{video} {idx}\n'
+ idx += 1
+ video_list.append(line)
+
+with open('hacs_data.txt', 'w') as f:
+ for line in video_list:
+ f.write(line)
diff --git a/tools/data/hacs/slowonly_feature_infer.py b/tools/data/hacs/slowonly_feature_infer.py
new file mode 100644
index 0000000000..3fce06add4
--- /dev/null
+++ b/tools/data/hacs/slowonly_feature_infer.py
@@ -0,0 +1,77 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+default_hooks = dict(
+ runtime_info=dict(type='RuntimeInfoHook'),
+ timer=dict(type='IterTimerHook'),
+ logger=dict(type='LoggerHook', interval=20, ignore_last=False),
+ param_scheduler=dict(type='ParamSchedulerHook'),
+ checkpoint=dict(type='CheckpointHook', interval=4, save_best='auto'),
+ sampler_seed=dict(type='DistSamplerSeedHook'),
+ sync_buffers=dict(type='SyncBuffersHook'))
+
+env_cfg = dict(
+ cudnn_benchmark=False,
+ mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
+ dist_cfg=dict(backend='nccl'))
+
+log_processor = dict(type='LogProcessor', window_size=20, by_epoch=True)
+vis_backends = [dict(type='LocalVisBackend')]
+visualizer = dict(
+ type='ActionVisualizer', vis_backends=[dict(type='LocalVisBackend')])
+log_level = 'INFO'
+
+model = dict(
+ type='Recognizer3D',
+ backbone=dict(
+ type='ResNet3dSlowOnly',
+ depth=50,
+ lateral=False,
+ conv1_kernel=(1, 7, 7),
+ conv1_stride_t=1,
+ pool1_stride_t=1,
+ inflate=(0, 0, 1, 1),
+ norm_eval=False),
+ cls_head=dict(
+ type='I3DHead',
+ in_channels=2048,
+ num_classes=700,
+ spatial_type='avg',
+ dropout_ratio=0.5,
+ average_clips=None),
+ data_preprocessor=dict(
+ type='ActionDataPreprocessor',
+ mean=[123.675, 116.28, 103.53],
+ std=[58.395, 57.12, 57.375],
+ format_shape='NCTHW'))
+
+data_root = './data'
+ann_file = 'hacs_data.txt'
+
+test_pipeline = [
+ dict(type='DecordInit', io_backend='disk'),
+ dict(
+ type='SampleFrames',
+ clip_len=8,
+ frame_interval=8,
+ num_clips=100,
+ test_mode=True),
+ dict(type='DecordDecode'),
+ dict(type='Resize', scale=(-1, 256)),
+ dict(type='CenterCrop', crop_size=256),
+ dict(type='FormatShape', input_format='NCTHW'),
+ dict(type='PackActionInputs')
+]
+
+test_dataloader = dict(
+ batch_size=1,
+ num_workers=8,
+ persistent_workers=True,
+ sampler=dict(type='DefaultSampler', shuffle=False),
+ dataset=dict(
+ type='VideoDataset',
+ ann_file=ann_file,
+ data_prefix=dict(video=data_root),
+ pipeline=test_pipeline,
+ test_mode=True))
+
+test_evaluator = dict(type='DumpResults', out_file_path='result.pkl')
+test_cfg = dict(type='TestLoop')
diff --git a/tools/data/hacs/write_feature_csv.py b/tools/data/hacs/write_feature_csv.py
new file mode 100644
index 0000000000..9fb098b4bf
--- /dev/null
+++ b/tools/data/hacs/write_feature_csv.py
@@ -0,0 +1,20 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+import mmengine
+
+features = mmengine.load('result.pkl')
+video_list = mmengine.list_from_file('hacs_data.txt')
+feature_dir = '../../../data/HACS/slowonly_feature'
+mmengine.mkdir_or_exist(feature_dir)
+
+head = ','.join([f'f{i}' for i in range(700)]) + '\n'
+
+for feature, video in zip(features, video_list):
+ video_id = video.split()[0].split('/')[1]
+ csv_file = video_id.replace('mp4', 'csv')
+ feat = feature['pred_scores']['item'].numpy()
+ feat = feat.tolist()
+ csv_path = f'{feature_dir}/{csv_file}'
+ with open(csv_path, 'w') as f:
+ f.write(head)
+ for line in feat:
+ f.write(str(line)[1:-1] + '\n')
diff --git a/tools/data/hmdb51/README.md b/tools/data/hmdb51/README.md
index 206b548764..4993f6705b 100644
--- a/tools/data/hmdb51/README.md
+++ b/tools/data/hmdb51/README.md
@@ -41,7 +41,7 @@ bash download_videos.sh
This part is **optional** if you only want to use the video loader.
-Before extracting, please refer to [install.md](/docs/install.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
+Before extracting, please refer to [install.md](/docs/en/get_started/installation.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
If you have plenty of SSD space, then we recommend extracting frames there for better I/O performance.
@@ -122,4 +122,4 @@ mmaction2
```
-For training and evaluating on HMDB51, please refer to [getting_started.md](/docs/getting_started.md).
+For training and evaluating on HMDB51, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).
diff --git a/tools/data/hmdb51/README_zh-CN.md b/tools/data/hmdb51/README_zh-CN.md
index a34c4b9ce9..2da13751cd 100644
--- a/tools/data/hmdb51/README_zh-CN.md
+++ b/tools/data/hmdb51/README_zh-CN.md
@@ -39,7 +39,7 @@ bash download_videos.sh
如果用户只想使用视频加载训练，则该部分是 **可选项**。
-在抽取视频帧和光流之前，请参考 [安装指南](/docs_zh_CN/install.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
+在抽取视频帧和光流之前，请参考 [安装指南](/docs/zh_cn/get_started/installation.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
如果用户有大量的 SSD 存储空间，则推荐将抽取的帧存储至 I/O 性能更优秀的 SSD 上。
用户可使用以下命令为 SSD 建立软链接。
@@ -118,4 +118,4 @@ mmaction2
```
-关于对 HMDB51 进行训练和验证，可以参照 [基础教程](/docs_zh_CN/getting_started.md)。
+关于对 HMDB51 进行训练和验证，可以参照 [训练教程](/docs/zh_cn/user_guides/train_test.md)。
diff --git a/tools/data/hvu/README.md b/tools/data/hvu/README.md
index 6bcc73f862..dd7952922c 100644
--- a/tools/data/hvu/README.md
+++ b/tools/data/hvu/README.md
@@ -43,7 +43,7 @@ bash download_videos.sh
This part is **optional** if you only want to use the video loader.
-Before extracting, please refer to [install.md](/docs/install.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
+Before extracting, please refer to [install.md](/docs/en/get_started/installation.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
You can use the following script to extract both RGB and Flow frames.
@@ -52,7 +52,7 @@ bash extract_frames.sh
```
By default, we generate frames with short edge resized to 256.
-More details can be found in [data_preparation](/docs/data_preparation.md)
+More details can be found in [prepare_dataset](/docs/en/user_guides/prepare_dataset.md)
## Step 4. Generate File List
@@ -120,4 +120,4 @@ mmaction2
```
-For training and evaluating on HVU, please refer to [getting_started](/docs/getting_started.md).
+For training and evaluating on HVU, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).
diff --git a/tools/data/hvu/README_zh-CN.md b/tools/data/hvu/README_zh-CN.md
index 5b3ffa1ea3..cbb9613d8e 100644
--- a/tools/data/hvu/README_zh-CN.md
+++ b/tools/data/hvu/README_zh-CN.md
@@ -43,7 +43,7 @@ bash download_videos.sh
如果用户仅使用 video loader，则可以跳过本步。
-在提取之前，请参考 [安装教程](/docs_zh_CN/install.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
+在提取之前，请参考 [安装指南](/docs/zh_cn/get_started/installation.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
用户可使用如下脚本同时抽取 RGB 帧和光流：
@@ -51,7 +51,7 @@ bash download_videos.sh
bash extract_frames.sh
```
-该脚本默认生成短边长度为 256 的帧，可参考 [数据准备](/docs_zh_CN/data_preparation.md) 获得更多细节。
+该脚本默认生成短边长度为 256 的帧，可参考 [数据准备](/docs/zh_cn/user_guides/prepare_dataset.md) 获得更多细节。
## 4. 生成文件列表
@@ -107,4 +107,4 @@ mmaction2
```
-关于 HVU 数据集上的训练与测试，请参照 [基础教程](/docs_zh_CN/getting_started.md)。
+关于 HVU 数据集上的训练与测试，请参照 [训练教程](/docs/zh_cn/user_guides/train_test.md)。
diff --git a/tools/data/hvu/download.py b/tools/data/hvu/download.py
index 2ab18e8434..29fcd5c3b2 100644
--- a/tools/data/hvu/download.py
+++ b/tools/data/hvu/download.py
@@ -11,7 +11,7 @@
import subprocess
import uuid
-import mmcv
+import mmengine
from joblib import Parallel, delayed
ssl._create_default_https_context = ssl._create_unverified_context
@@ -173,7 +173,7 @@ def main(input_csv,
# Clean tmp dir.
shutil.rmtree(tmp_dir)
# Save download report.
- mmcv.dump(status_lst, 'download_report.json')
+ mmengine.dump(status_lst, 'download_report.json')
if __name__ == '__main__':
diff --git a/tools/data/hvu/generate_file_list.py b/tools/data/hvu/generate_file_list.py
index 83e99b1482..ecc4063ffa 100644
--- a/tools/data/hvu/generate_file_list.py
+++ b/tools/data/hvu/generate_file_list.py
@@ -5,7 +5,7 @@
import os
import os.path as osp
-import mmcv
+import mmengine
annotation_root = '../../data/hvu/annotations'
tag_file = 'hvu_tags.json'
@@ -109,7 +109,7 @@ def parse_args():
if __name__ == '__main__':
args = parse_args()
- tag_cates = mmcv.load(tag_file)
+ tag_cates = mmengine.load(tag_file)
tag2category = {}
for k in tag_cates:
for tag in tag_cates[k]:
@@ -149,4 +149,4 @@ def parse_line(line):
]
elif args.mode == 'videos':
result = [dict(filename=k[0] + '.mp4', label=k[1]) for k in data_list]
- mmcv.dump(result, args.output)
+ mmengine.dump(result, args.output)
diff --git a/tools/data/hvu/generate_sub_file_list.py b/tools/data/hvu/generate_sub_file_list.py
index 8313a9b3c9..ee14b82f6b 100644
--- a/tools/data/hvu/generate_sub_file_list.py
+++ b/tools/data/hvu/generate_sub_file_list.py
@@ -2,7 +2,7 @@
import argparse
import os.path as osp
-import mmcv
+import mmengine
def main(annotation_file, category):
@@ -10,7 +10,7 @@ def main(annotation_file, category):
'action', 'attribute', 'concept', 'event', 'object', 'scene'
]
- data = mmcv.load(annotation_file)
+ data = mmengine.load(annotation_file)
basename = osp.basename(annotation_file)
dirname = osp.dirname(annotation_file)
basename = basename.replace('hvu', f'hvu_{category}')
@@ -24,7 +24,7 @@ def main(annotation_file, category):
item['label'] = label[category]
result.append(item)
- mmcv.dump(data, target_file)
+ mmengine.dump(data, target_file)
if __name__ == '__main__':
diff --git a/tools/data/hvu/parse_tag_list.py b/tools/data/hvu/parse_tag_list.py
index 0871491ef8..b4d0d699bf 100644
--- a/tools/data/hvu/parse_tag_list.py
+++ b/tools/data/hvu/parse_tag_list.py
@@ -1,5 +1,5 @@
# Copyright (c) OpenMMLab. All rights reserved.
-import mmcv
+import mmengine
tag_list = '../../../data/hvu/annotations/hvu_categories.csv'
@@ -13,4 +13,4 @@
for k in tag_categories:
tag_categories[k].sort()
-mmcv.dump(tag_categories, 'hvu_tags.json')
+mmengine.dump(tag_categories, 'hvu_tags.json')
diff --git a/tools/data/jester/README.md b/tools/data/jester/README.md
index e90841a850..616a29a9a4 100644
--- a/tools/data/jester/README.md
+++ b/tools/data/jester/README.md
@@ -64,7 +64,7 @@ data = dict(
This part is **optional** if you only want to use RGB frames.
-Before extracting, please refer to [install.md](/docs/install.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
+Before extracting, please refer to [install.md](/docs/en/get_started/installation.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
If you have plenty of SSD space, then we recommend extracting frames there for better I/O performance.
@@ -140,4 +140,4 @@ mmaction2
```
-For training and evaluating on Jester, please refer to [getting_started.md](/docs/getting_started.md).
+For training and evaluating on Jester, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).
diff --git a/tools/data/jester/README_zh-CN.md b/tools/data/jester/README_zh-CN.md
index 7660c23ae7..22b4680a7d 100644
--- a/tools/data/jester/README_zh-CN.md
+++ b/tools/data/jester/README_zh-CN.md
@@ -64,7 +64,7 @@ data = dict(
如果用户只想使用 RGB 帧训练，则该部分是 **可选项**。
-在抽取视频帧和光流之前，请参考 [安装指南](/docs_zh_CN/install.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
+在抽取视频帧和光流之前，请参考 [安装指南](/docs/zh_cn/get_started/installation.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
如果拥有大量的 SSD 存储空间，则推荐将抽取的帧存储至 I/O 性能更优秀的 SSD 中。
@@ -140,4 +140,4 @@ mmaction2
```
-关于对 jester 进行训练和验证，可以参考 [基础教程](/docs_zh_CN/getting_started.md)。
+关于对 jester 进行训练和验证，请参考 [训练和测试教程](/docs/en/user_guides/train_test.md)。
diff --git a/tools/data/kinetics/README.md b/tools/data/kinetics/README.md
index 0df8f8634f..257c650eba 100644
--- a/tools/data/kinetics/README.md
+++ b/tools/data/kinetics/README.md
@@ -15,8 +15,7 @@
}
```
-For basic dataset information, please refer to the official [website](https://deepmind.com/research/open-source/open-source-datasets/kinetics/). The scripts can be used for preparing kinetics400, kinetics600, kinetics700. To prepare different version of kinetics, you need to replace `${DATASET}` in the following examples with the specific dataset name. The choices of dataset names are `kinetics400`, `kinetics600` and `kinetics700`.
-Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/${DATASET}/`.
+For basic dataset information, please refer to the official [website](https://deepmind.com/research/open-source/open-source-datasets/kinetics/).
:::{note}
Because of the expirations of some YouTube links, the sizes of kinetics dataset copies may be different. Here are the sizes of our kinetics dataset copies that used to train all checkpoints.
@@ -29,8 +28,38 @@ Because of the expirations of some YouTube links, the sizes of kinetics dataset
:::
+`````{tabs}
+
+````{group-tab} Download by MIM
+:::{note}
+All experiments on Kinetics in MMAction2 are based on this version, we recommend users to try this version.
+:::
+
+MIM supports downloading from OpenDataLab and preprocessing Kinetics-400/600/700 dataset with one command line.
+
+```Bash
+# install OpenDataLab CLI tools
+pip install -U opendatalab
+# log in OpenDataLab
+odl login
+# download and preprocess Kinetics-400 by MIM. Note that this might take a long time.
+mim download mmaction2 --dataset kinetics400
+# download and preprocess Kinetics-600 by MIM. Note that this might take a long time.
+mim download mmaction2 --dataset kinetics600
+# download and preprocess Kinetics-700 by MIM. Note that this might take a long time.
+mim download mmaction2 --dataset kinetics700
+
+```
+
+````
+
+````{group-tab} Download form Official Source
+
## Step 1. Prepare Annotations
+The scripts can be used for preparing kinetics400, kinetics600, kinetics700. To prepare different version of kinetics, you need to replace `${DATASET}` in the following examples with the specific dataset name. The choices of dataset names are `kinetics400`, `kinetics600` and `kinetics700`.
+Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/${DATASET}/`.
+
First of all, you can run the following script to prepare annotations by downloading from the official [website](https://deepmind.com/research/open-source/open-source-datasets/kinetics/).
```shell
@@ -48,15 +77,6 @@ bash download_backup_annotations.sh ${DATASET}
## Step 2. Prepare Videos
-### Option 1: Download from OpenDataLab
-
-**Recommend**: [OpenDataLab](https://opendatalab.com/) provides the Kinetics dataset ([Kinetics400](https://opendatalab.com/Kinetics-400), [Kinetics600](https://opendatalab.com/Kinetics600), [Kinetics700](https://opendatalab.com/Kinetics_700)), users can download Kinetics dataset with short edge 320 pixels from here.
-
-:::{note}
-All experiments on Kinetics in MMAction2 are based on this version, we recommend users to try this version.
-
-### Option 2: Download from Other Source
-
you can run the following script to prepare videos.
The codes are adapted from the [official crawler](https://github.com/activitynet/ActivityNet/tree/master/Crawler/Kinetics). Note that this might take a long time.
@@ -83,7 +103,7 @@ You can also download from [Academic Torrents](https://academictorrents.com/) ([
This part is **optional** if you only want to use the video loader.
-Before extracting, please refer to [install.md](/docs/install.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
+Before extracting, please refer to [install.md](/docs/en/get_started/installation.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
If you have plenty of SSD space, then we recommend extracting frames there for better I/O performance. And you can run the following script to soft link the extracted frames.
@@ -114,7 +134,7 @@ bash extract_frames.sh ${DATASET}
```
The commands above can generate images with new short edge 256. If you want to generate images with short edge 320 (320p), or with fix size 340x256, you can change the args `--new-short 256` to `--new-short 320` or `--new-width 340 --new-height 256`.
-More details can be found in [data_preparation](/docs/data_preparation.md)
+More details can be found in [prepare dataset](/docs/en/user_guides/prepare_dataset.md).
## Step 4. Generate File List
@@ -126,7 +146,10 @@ bash generate_videos_filelist.sh ${DATASET}
bash generate_rawframes_filelist.sh ${DATASET}
```
-## Step 5. Folder Structure
+````
+`````
+
+### Folder Structure
After the whole data pipeline for Kinetics preparation.
you can get the rawframes (RGB + Flow), videos and annotation files for Kinetics.
@@ -153,9 +176,9 @@ mmaction2
│ │ │ ├── wrapping_present
│ │ │ ├── ...
│ │ │ ├── zumba
-│ │ ├── rawframes_train
-│ │ ├── rawframes_val
+│ │ ├── rawframes_train (optional)
+│ │ ├── rawframes_val (optional)
```
-For training and evaluating on Kinetics, please refer to [getting_started](/docs/getting_started.md).
+For training and evaluating on Kinetics, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).
diff --git a/tools/data/kinetics/README_zh-CN.md b/tools/data/kinetics/README_zh-CN.md
index 86cb65239e..a0d2e858e1 100644
--- a/tools/data/kinetics/README_zh-CN.md
+++ b/tools/data/kinetics/README_zh-CN.md
@@ -15,8 +15,7 @@
}
```
-请参照 [官方网站](https://deepmind.com/research/open-source/open-source-datasets/kinetics/) 以获取数据集基本信息。此脚本用于准备数据集 kinetics400，kinetics600，kinetics700。为准备 kinetics 数据集的不同版本，用户需将脚本中的 `${DATASET}` 赋值为数据集对应版本名称，可选项为 `kinetics400`，`kinetics600`， `kinetics700`。
-在开始之前，用户需确保当前目录为 `$MMACTION2/tools/data/${DATASET}/`。
+请参照 [官方网站](https://deepmind.com/research/open-source/open-source-datasets/kinetics/) 以获取数据集基本信息。
:::{note}
由于部分 YouTube 链接失效，爬取的 Kinetics 数据集大小可能与原版不同。以下是我们所使用 Kinetics 数据集的大小：
@@ -26,9 +25,36 @@
| Kinetics400 | 240436 | 19796 |
| Kinetics600 | 383393 | 27910 |
| Kinetics700 | 542357 | 34824 |
+| ::: | | |
+
+`````{tabs}
+
+````{group-tab} 使用 MIM 下载
+:::{note}
+MMAction2 代码仓库中提供的 Kinetics 实验性能，都是基于这个版本的数据得到的。我们建议用户使用这个版本的 Kinetics 数据集进行实验。
+:::
+
+# MIM 支持下载 Kinetics-400/600/700 数据集。用户可以通过一行命令，从 OpenDataLab 进行下载，并进行预处理。
+```Bash
+# 安装 OpenDataLab CLI 工具
+pip install -U opendatalab
+# 登录 OpenDataLab
+odl login
+# 通过 MIM 进行 Kinetics-400 数据集下载，预处理。注意这将花费较长时间
+mim download mmaction2 --dataset kinetics400
+# 通过 MIM 进行 Kinetics-600 数据集下载，预处理。注意这将花费较长时间
+mim download mmaction2 --dataset kinetics600
+# 通过 MIM 进行 Kinetics-700 数据集下载，预处理。注意这将花费较长时间
+mim download mmaction2 --dataset kinetics700
+```
+````
+
+````{group-tab} 从官方源下载
## 1. 准备标注文件
+此脚本用于准备数据集 kinetics400，kinetics600，kinetics700。为准备 kinetics 数据集的不同版本，用户需将脚本中的 `${DATASET}` 赋值为数据集对应版本名称，可选项为 `kinetics400`，`kinetics600`， `kinetics700`。
+在开始之前，用户需确保当前目录为 `$MMACTION2/tools/data/${DATASET}/`。
首先，用户可以使用如下脚本从 [Kinetics 数据集官网](https://deepmind.com/research/open-source/open-source-datasets/kinetics/)下载标注文件并进行预处理：
```shell
@@ -45,15 +71,6 @@ bash download_backup_annotations.sh ${DATASET}
## 2. 准备视频
-### 选项 1: 从 OpenDataLab 下载
-
-**推荐**：[OpenDataLab](https://opendatalab.com/) 提供了 Kinetics 数据集 ([Kinetics400](https://opendatalab.com/Kinetics-400), [Kinetics600](https://opendatalab.com/Kinetics600), [Kinetics700](https://opendatalab.com/Kinetics_700)), 用户可以从这里下载短边长度为 320 的 Kinetics 数据集。
-
-:::{note}
-MMAction2 代码仓库中提供的 Kinetics 实验性能，都是基于这个版本的数据得到的。我们建议用户使用这个版本的 Kinetics 数据集进行实验。
-
-### 选项 2：从其他数据源下载
-
用户可以使用以下脚本准备视频，视频准备代码修改自 [官方爬虫](https://github.com/activitynet/ActivityNet/tree/master/Crawler/Kinetics)。注意这一步骤将花费较长时间。
```shell
@@ -78,7 +95,7 @@ python ../resize_videos.py ../../../data/${DATASET}/videos_train/ ../../../data/
如果用户仅使用 video loader，则可以跳过本步。
-在提取之前，请参考 [安装教程](/docs_zh_CN/install.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
+在提取之前，请参考 [安装指南](/docs/zh_cn/get_started/installation.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
如果用户有足够的 SSD 空间，那么建议将视频抽取为 RGB 帧以提升 I/O 性能。用户可以使用以下脚本为抽取得到的帧文件夹建立软连接：
@@ -109,7 +126,7 @@ bash extract_frames.sh ${DATASET}
```
以上的命令生成短边长度为 256 的 RGB 帧和光流帧。如果用户需要生成短边长度为 320 的帧 (320p)，或是固定分辨率为 340 x 256 的帧，可改变参数 `--new-short 256` 为 `--new-short 320` 或 `--new-width 340 --new-height 256`。
-更多细节可以参考 [数据准备](/docs_zh_CN/data_preparation.md)。
+更多细节可以参考 [数据准备](/docs/zh_cn/user_guides/prepare_dataset.md)。
## 4. 生成文件列表
@@ -121,7 +138,10 @@ bash generate_videos_filelist.sh ${DATASET}
bash generate_rawframes_filelist.sh ${DATASET}
```
-## 5. 目录结构
+````
+`````
+
+### 目录结构
在完整完成 Kinetics 的数据处理后，将得到帧文件夹（RGB 帧和光流帧），视频以及标注文件。
@@ -136,7 +156,7 @@ mmaction2
│ ├── ${DATASET}
│ │ ├── ${DATASET}_train_list_videos.txt
│ │ ├── ${DATASET}_val_list_videos.txt
-│ │ ├── annotations
+│ │ ├── annotations（可选）
│ │ ├── videos_train
│ │ ├── videos_val
│ │ │ ├── abseiling
@@ -146,9 +166,9 @@ mmaction2
│ │ │ ├── wrapping_present
│ │ │ ├── ...
│ │ │ ├── zumba
-│ │ ├── rawframes_train
-│ │ ├── rawframes_val
+│ │ ├── rawframes_train（可选）
+│ │ ├── rawframes_val（可选）
```
-关于 Kinetics 数据集上的训练与测试，请参照 [基础教程](/docs_zh_CN/getting_started.md)。
+关于 Kinetics 数据集上的训练与测试，请参照 [训练教程](/docs/zh_cn/user_guides/train_test.md)。
diff --git a/tools/data/kinetics/preprocess_k400.sh b/tools/data/kinetics/preprocess_k400.sh
new file mode 100644
index 0000000000..9f07885095
--- /dev/null
+++ b/tools/data/kinetics/preprocess_k400.sh
@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+
+set -x
+
+DOWNLOAD_DIR=$1
+DATA_ROOT=$2
+
+cat $DOWNLOAD_DIR/Kinetics-400/raw/*.tar.gz* | tar -xvz -C $(dirname $DATA_ROOT)
+mv $(dirname $DATA_ROOT)/Kinetics-400 $DATA_ROOT
diff --git a/tools/data/kinetics/preprocess_k600.sh b/tools/data/kinetics/preprocess_k600.sh
new file mode 100644
index 0000000000..438297a620
--- /dev/null
+++ b/tools/data/kinetics/preprocess_k600.sh
@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+
+set -x
+
+DOWNLOAD_DIR=$1
+DATA_ROOT=$2
+
+cat $DOWNLOAD_DIR/Kinetics600/raw/*.tar.gz* | tar -xvz -C $(dirname $DATA_ROOT)
+mv $(dirname $DATA_ROOT)/Kinetics600 $DATA_ROOT
diff --git a/tools/data/kinetics/preprocess_k700.sh b/tools/data/kinetics/preprocess_k700.sh
new file mode 100644
index 0000000000..930bf8577a
--- /dev/null
+++ b/tools/data/kinetics/preprocess_k700.sh
@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+
+set -x
+
+DOWNLOAD_DIR=$1
+DATA_ROOT=$2
+
+cat $DOWNLOAD_DIR/Kinetics_700/raw/*.tar.gz* | tar -xvz -C $(dirname $DATA_ROOT)
+mv $(dirname $DATA_ROOT)/Kinetics_700 $DATA_ROOT
diff --git a/tools/data/kinetics710/README.md b/tools/data/kinetics710/README.md
new file mode 100644
index 0000000000..76a239fbf5
--- /dev/null
+++ b/tools/data/kinetics710/README.md
@@ -0,0 +1,91 @@
+# Preparing Kinetics-710
+
+## Introduction
+
+
+
+```BibTeX
+@misc{li2022uniformerv2,
+ title={UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer},
+ author={Kunchang Li and Yali Wang and Yinan He and Yizhuo Li and Yi Wang and Limin Wang and Yu Qiao},
+ year={2022},
+ eprint={2211.09552},
+ archivePrefix={arXiv},
+ primaryClass={cs.CV}
+}
+```
+
+For basic dataset information, please refer to the [paper](https://arxiv.org/pdf/2211.09552.pdf). The scripts can be used for preparing kinetics-710. MMAction2 supports Kinetics-710
+dataset as a concat dataset, which means only provides a list of annotation files, and makes use of the original data of Kinetics-400/600/700 dataset. You could refer to the [config](/configs/recognition/uniformerv2/uniformerv2-base-p16-res224_clip_u8_kinetics710-rgb.py)
+for details, which also provides a template config about how to use concat dataset in MMAction2.
+Before we start, please make sure that the directory is located at `$MMACTION2`.
+
+## Step 1. Download Kinetics 400/600/700
+
+Kinetics-710 is a video benchmark based on Kinetics-400/600/700, which merges the training set of these Kinetics datasets, and deletes the repeated videos according to Youtube IDs. MMAction2 provides an annotation file based on the Kinetics-400/600/700 on [OpenDataLab](https://opendatalab.com/). So we suggest you download Kinetics-400/600/700 first from OpenDataLab by [MIM](https://github.com/open-mmlab/mim).
+
+```shell
+# install OpenDataLab CLI tools
+pip install -U opendatalab
+# log in OpenDataLab
+odl login
+# download Kinetics-400/600/700, note that this might take a long time.
+mim download mmaction2 --dataset kinetics400
+mim download mmaction2 --dataset kinetics600
+mim download mmaction2 --dataset kinetics700
+
+```
+
+## Step 2. Download Kinetics-710 Annotations
+
+We provide the annotation list of Kinetics-710 corresponding to OpenDataLab version Kinetics, you could download it from aliyun and unzip it to the `$MMACTION2/data/`
+
+```shell
+wget -P data https://download.openmmlab.com/mmaction/dataset/kinetics710/annotations.zip
+cd data && unzip annotations.zip && cd ..
+
+```
+
+## Step 3. Folder Structure
+
+After the whole data pipeline for Kinetics preparation.
+you can get the videos and annotation files for Kinetics-710.
+
+In the context of the whole project (for Kinetics only), the *minimal* folder structure will look like:
+(*minimal* means that some data are not necessary: for example, you may want to evaluate kinetics using the original video format.)
+
+```
+mmaction2
+├── mmaction
+├── tools
+├── configs
+├── data
+│ ├── kinetics400
+│ │ ├── videos_train
+│ │ ├── videos_val
+│ │ │ ├── jf7RDuUTrsQ.mp4
+│ │ │ ├── ...
+│ ├── kinetics600
+│ │ ├── videos
+│ │ │ ├── vol_00
+│ │ │ │ ├── -A5JFdMXB_k_000018_000028.mp4
+│ │ │ │ ├── ...
+│ │ │ ├── ...
+│ │ │ ├── vol63
+│ ├── kinetics700
+│ │ ├── videos
+│ │ │ ├── vol_00
+│ │ │ │ ├── -Paa0R0tQ1w_000009_000019.mp4
+│ │ │ │ ├── ...
+│ │ │ ├── ...
+│ │ │ ├── vol63
+│ ├── kinetics710
+│ │ ├── k400_train_list_videos.txt
+│ │ ├── k400_val_list_videos.txt
+│ │ ├── k600_train_list_videos.txt
+│ │ ├── k600_val_list_videos.txt
+│ │ ├── k700_train_list_videos.txt
+│ │ ├── k700_val_list_videos.txt
+```
+
+For training and evaluating on Kinetics, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).
diff --git a/tools/data/kinetics710/README_zh-CN.md b/tools/data/kinetics710/README_zh-CN.md
new file mode 100644
index 0000000000..b428be43df
--- /dev/null
+++ b/tools/data/kinetics710/README_zh-CN.md
@@ -0,0 +1,89 @@
+# 准备 Kinetics-710
+
+## 介绍
+
+
+
+```BibTeX
+@misc{li2022uniformerv2,
+ title={UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer},
+ author={Kunchang Li and Yali Wang and Yinan He and Yizhuo Li and Yi Wang and Limin Wang and Yu Qiao},
+ year={2022},
+ eprint={2211.09552},
+ archivePrefix={arXiv},
+ primaryClass={cs.CV}
+}
+```
+
+关于基本数据集信息，请参考 [论文](https://arxiv.org/pdf/2211.09552.pdf)。这些脚本可以用于准备 kinetics-710。MMAction2 以 Concat Daataset 的形式支持了 Kinetics-710 数据集，我们只提供一个注释文件列表，并利用 Kinetics-400/600/700 数据集的原始数据。你可以参考 [配置](/configs/recognition/uniformerv2/uniformerv2-base-p16-res224_clip_u8_kinetics710-rgb.py) 了解详情，它也提供了一个模板配置，说明了如何在 MMAction2 中使用 Concat Dataset。
+在我们开始之前，请确保目录位于 `$MMACTION2`。
+
+## 第一步：下载 Kinetics 400/600/700
+
+Kinetics-710 是基于 Kinetics-400/600/700 的视频数据集，它合并了这些 Kinetics 数据集的训练集，并根据 Youtube ID 删除了重复的视频。MMAction2 提供了一个基于 Kinetics-400/600/700 的 OpenDataLab 版本的标注文件，你可以通过 [MIM](https://github.com/open-mmlab/mim) 从 OpenDataLab 下载。
+
+```shell
+# 安装 OpenDataLab CLI 工具
+pip install -U opendatalab
+# 登录 OpenDataLab
+odl login
+# 下载 Kinetics-400/600/700，注意这可能需要很长时间。
+mim download mmaction2 --dataset kinetics400
+mim download mmaction2 --dataset kinetics600
+mim download mmaction2 --dataset kinetics700
+
+```
+
+## 第二步：下载 Kinetics-710 标注文件
+
+我们提供了与 OpenDataLab 版本 Kinetics 相对应的 Kinetics-710 标注文件列表，你可以从阿里云下载它，并将其解压到 `$MMACTION2/data/`
+
+```shell
+wget -P data https://download.openmmlab.com/mmaction/dataset/kinetics710/annotations.zip
+cd data && unzip annotations.zip && cd ..
+
+```
+
+## 第三步：文件夹结构
+
+完成 Kinetics 准备的整个数据流程后。
+你可以得到 Kinetics-710 的视频和注释文件。
+
+在整个项目目录下（仅针对 Kinetics），*最小*的文件夹结构如下：
+（*最小*意味着一些数据是不必要的：例如，你可能想要使用原始视频格式评估 kinetics。）
+
+```
+mmaction2
+├── mmaction
+├── tools
+├── configs
+├── data
+│ ├── kinetics400
+│ │ ├── videos_train
+│ │ ├── videos_val
+│ │ │ ├── jf7RDuUTrsQ.mp4
+│ │ │ ├── ...
+│ ├── kinetics600
+│ │ ├── videos
+│ │ │ ├── vol_00
+│ │ │ │ ├── -A5JFdMXB_k_000018_000028.mp4
+│ │ │ │ ├── ...
+│ │ │ ├── ...
+│ │ │ ├── vol63
+│ ├── kinetics700
+│ │ ├── videos
+│ │ │ ├── vol_00
+│ │ │ │ ├── -Paa0R0tQ1w_000009_000019.mp4
+│ │ │ │ ├── ...
+│ │ │ ├── ...
+│ │ │ ├── vol63
+│ ├── kinetics710
+│ │ ├── k400_train_list_videos.txt
+│ │ ├── k400_val_list_videos.txt
+│ │ ├── k600_train_list_videos.txt
+│ │ ├── k600_val_list_videos.txt
+│ │ ├── k700_train_list_videos.txt
+│ │ ├── k700_val_list_videos.txt
+```
+
+关于在 Kinetics 上进行训练和评估，请参考 [训练和测试教程](/docs/en/user_guides/train_test.md)。
diff --git a/tools/data/kinetics710/label_map_k710.txt b/tools/data/kinetics710/label_map_k710.txt
new file mode 100644
index 0000000000..5b3f53cedd
--- /dev/null
+++ b/tools/data/kinetics710/label_map_k710.txt
@@ -0,0 +1,710 @@
+abseiling
+air drumming
+answering questions
+applauding
+applying cream
+archery
+arm wrestling
+arranging flowers
+assembling computer
+auctioning
+baby waking up
+baking cookies
+inflating balloons
+bandaging
+barbequing
+bartending
+beatboxing
+bee keeping
+belly dancing
+bench pressing
+bending back
+bending metal
+biking through snow
+blasting sand
+blowing glass
+blowing leaves
+blowing nose
+blowing out candles
+bobsledding
+bookbinding
+bouncing on trampoline
+bowling
+braiding hair
+breading or breadcrumbing
+breakdancing
+brush painting
+brushing hair
+brushing teeth
+building cabinet
+building shed
+bungee jumping
+busking
+canoeing or kayaking
+capoeira
+carrying baby
+cartwheeling
+carving pumpkin
+catching fish
+catching or throwing baseball
+catching or throwing frisbee
+catching or throwing softball
+celebrating
+changing oil
+changing wheel (not on bike)
+checking tires
+cheerleading
+chopping wood
+clapping
+clay pottery making
+clean and jerk
+cleaning floor
+cleaning gutters
+cleaning pool
+cleaning shoes
+cleaning toilet
+cleaning windows
+climbing a rope
+climbing ladder
+climbing tree
+contact juggling
+cooking chicken
+cooking egg
+cooking on campfire
+cooking sausages (not on barbeque)
+counting money
+country line dancing
+cracking neck
+crawling baby
+crossing river
+crying
+curling hair
+cutting nails
+cutting pineapple
+cutting watermelon
+dancing ballet
+dancing charleston
+dancing gangnam style
+dancing macarena
+deadlifting
+decorating the christmas tree
+digging
+dining
+disc golfing
+diving cliff
+dodgeball
+doing aerobics
+doing laundry
+doing nails
+drawing
+dribbling basketball
+sipping cup
+drinking beer
+drinking shots
+driving car
+driving tractor
+drop kicking
+drumming fingers
+dunking basketball
+dyeing hair
+eating burger
+eating cake
+eating carrots
+eating chips
+eating doughnuts
+eating hotdog
+eating ice cream
+eating spaghetti
+eating watermelon
+egg hunting
+exercising arm
+exercising with an exercise ball
+extinguishing fire
+faceplanting
+feeding birds
+feeding fish
+feeding goats
+filling eyebrows
+finger snapping
+fixing hair
+flipping pancake
+flying kite
+folding clothes
+folding napkins
+folding paper
+front raises
+frying vegetables
+person collecting garbage
+gargling
+getting a haircut
+getting a tattoo
+giving or receiving award
+golf chipping
+golf driving
+golf putting
+grinding meat
+grooming dog
+grooming horse
+gymnastics tumbling
+hammer throw
+headbanging
+headbutting
+high jump
+high kick
+hitting baseball
+hockey stop
+holding snake
+hopscotch
+hoverboarding
+hugging (not baby)
+hula hooping
+hurdling
+hurling (sport)
+ice climbing
+ice fishing
+ice skating
+ironing
+javelin throw
+jetskiing
+jogging
+juggling balls
+juggling fire
+juggling soccer ball
+jumping into pool
+jumpstyle dancing
+kicking field goal
+kicking soccer ball
+kissing
+kitesurfing
+knitting
+krumping
+laughing
+laying bricks
+long jump
+lunge
+making a cake
+making a sandwich
+making the bed
+making jewelry
+making pizza
+making snowman
+making sushi
+making tea
+marching
+massaging back
+massaging feet
+massaging legs
+massaging person's head
+milking cow
+mopping floor
+motorcycling
+moving furniture
+mowing lawn
+news anchoring
+opening bottle (not wine)
+opening present
+paragliding
+parasailing
+parkour
+passing American football (in game)
+passing American football (not in game)
+peeling apples
+peeling potatoes
+petting animal (not cat)
+petting cat
+picking apples
+planting trees
+plastering
+playing accordion
+playing badminton
+playing bagpipes
+playing basketball
+playing bass guitar
+playing cards
+playing cello
+playing chess
+playing clarinet
+playing controller
+playing cricket
+playing cymbals
+playing didgeridoo
+playing drums
+playing flute
+playing guitar
+playing harmonica
+playing harp
+playing ice hockey
+playing keyboard
+playing kickball
+playing monopoly
+playing organ
+playing paintball
+playing piano
+playing poker
+playing recorder
+playing saxophone
+playing squash or racquetball
+playing tennis
+playing trombone
+playing trumpet
+playing ukulele
+playing violin
+playing volleyball
+playing xylophone
+pole vault
+presenting weather forecast
+pull ups
+pumping fist
+pumping gas
+punching bag
+punching person (boxing)
+push up
+pushing car
+pushing cart
+pushing wheelchair
+reading book
+reading newspaper
+recording music
+riding a bike
+riding camel
+riding elephant
+riding mechanical bull
+riding mountain bike
+riding mule
+riding or walking with horse
+riding scooter
+riding unicycle
+ripping paper
+robot dancing
+rock climbing
+rock scissors paper
+roller skating
+running on treadmill
+sailing
+salsa dancing
+sanding floor
+scrambling eggs
+scuba diving
+setting table
+shaking hands
+shaking head
+sharpening knives
+sharpening pencil
+shaving head
+shaving legs
+shearing sheep
+shining shoes
+shooting basketball
+shooting goal (soccer)
+shot put
+shoveling snow
+shredding paper
+shuffling cards
+side kick
+sign language interpreting
+singing
+situp
+skateboarding
+ski jumping
+skiing mono
+skiing crosscountry
+skiing slalom
+skipping rope
+skydiving
+slacklining
+slapping
+sled dog racing
+smoking
+smoking hookah
+snatch weight lifting
+sneezing
+smelling feet
+snorkeling
+snowboarding
+snowkiting
+snowmobiling
+somersaulting
+spinning poi
+spray painting
+spraying
+springboard diving
+squat
+sticking tongue out
+stomping grapes
+stretching arm
+stretching leg
+strumming guitar
+surfing crowd
+surfing water
+sweeping floor
+swimming backstroke
+swimming breast stroke
+swimming butterfly stroke
+swing dancing
+swinging legs
+swinging on something
+sword fighting
+tai chi
+taking a shower
+tango dancing
+tap dancing
+tapping guitar
+tapping pen
+tasting beer
+tasting food
+testifying
+texting
+throwing axe
+throwing ball
+throwing discus
+tickling
+tobogganing
+tossing coin
+tossing salad
+training dog
+trapezing
+trimming or shaving beard
+trimming trees
+triple jump
+tying bow tie
+tying knot (not on a tie)
+tying necktie
+unboxing
+unloading truck
+using computer
+using remote controller (not gaming)
+using segway
+vault
+waiting in line
+walking the dog
+washing dishes
+washing feet
+washing hair
+washing hands
+water skiing
+water sliding
+watering plants
+waxing back
+waxing chest
+waxing eyebrows
+waxing legs
+weaving basket
+welding
+whistling
+windsurfing
+wrapping present
+wrestling
+writing
+yawning
+yoga
+zumba
+poaching eggs
+playing nose flute
+entering church
+closing door
+helmet diving
+doing sudoku
+coughing
+seasoning food
+peeling banana
+eating nachos
+waxing armpits
+shouting
+silent disco
+polishing furniture
+taking photo
+dealing cards
+putting wallpaper on wall
+uncorking champagne
+curling eyelashes
+brushing floor
+pulling espresso shot
+playing american football
+grooming cat
+playing checkers
+moving child
+stacking cups
+squeezing orange
+opening coconuts
+rolling eyes
+picking blueberries
+playing road hockey
+carving wood with a knife
+slicing onion
+saluting
+letting go of balloon
+breaking glass
+carrying weight
+mixing colours
+moving baby
+blending fruit
+pouring milk
+surveying
+making slime
+sieving
+walking with crutches
+flipping bottle
+playing billiards
+arresting
+listening with headphones
+spinning plates
+carving marble
+cutting cake
+shoot dance
+being excited
+petting horse
+splashing water
+filling cake
+stacking dice
+checking watch
+treating wood
+laying decking
+shooting off fireworks
+pouring wine
+pretending to be a statue
+steering car
+playing rounders
+looking in mirror
+jumping sofa
+lighting candle
+walking on stilts
+crocheting
+playing piccolo
+vacuuming car
+high fiving
+playing shuffleboard
+chasing
+pulling rope (game)
+being in zero gravity
+sanding wood
+decoupage
+using megaphone
+making latte art
+ski ballet
+playing oboe
+bouncing ball (not juggling)
+playing mahjong
+herding cattle
+swimming with sharks
+milking goat
+swimming with dolphins
+metal detecting
+playing slot machine
+polishing metal
+throwing tantrum
+lawn mower racing
+laying stone
+cutting orange
+skipping stone
+pouring beer
+making bubbles
+jaywalking
+leatherworking
+card stacking
+putting on eyeliner
+card throwing
+chewing gum
+falling off bike
+repairing puncture
+dumpster diving
+tiptoeing
+sleeping
+using circular saw
+cracking knuckles
+pinching
+chiseling wood
+playing rubiks cube
+weaving fabric
+fencing (sport)
+sword swallowing
+lighting fire
+vacuuming floor
+combing hair
+building lego
+playing pinball
+fly tying
+playing lute
+opening door
+waving hand
+rolling pastry
+chiseling stone
+threading needle
+playing dominoes
+opening wine bottle
+playing with trains
+steer roping
+playing field hockey
+separating eggs
+sewing
+talking on cell phone
+needle felting
+pushing wheelbarrow
+using a paint roller
+playing netball
+lifting hat
+massaging neck
+blowing bubble gum
+walking through snow
+docking boat
+clam digging
+marriage proposal
+packing
+sausage making
+licking
+scrapbooking
+flint knapping
+lock picking
+putting on lipstick
+sawing wood
+playing hand clapping games
+geocaching
+looking at phone
+making cheese
+poking bellybutton
+contorting
+fixing bicycle
+using a microscope
+using a wrench
+doing jigsaw puzzle
+making horseshoes
+cooking scallops
+square dancing
+getting a piercing
+playing ocarina
+making paper aeroplanes
+playing scrabble
+visiting the zoo
+crossing eyes
+jumping bicycle
+throwing water balloon
+bodysurfing
+pirouetting
+luge
+spelunking
+watching tv
+attending conference
+curling (sport)
+directing traffic
+swimming front crawl
+ice swimming
+battle rope training
+putting on mascara
+bouncing on bouncy castle
+smoking pipe
+pillow fight
+putting on sari
+calligraphy
+roasting pig
+cracking back
+shopping
+burping
+using bagging machine
+staring
+shucking oysters
+blowdrying hair
+smashing
+playing laser tag
+wading through mud
+rope pushdown
+preparing salad
+making balloon shapes
+tagging graffiti
+adjusting glasses
+using a power drill
+trimming shrubs
+popping balloons
+playing pan pipes
+using puppets
+arguing
+backflip (human)
+riding snow blower
+hand washing clothes
+calculating
+gospel singing in church
+standing on hands
+tasting wine
+shaping bread dough
+wading through water
+falling off chair
+throwing snowballs
+building sandcastle
+land sailing
+tying shoe laces
+jumping jacks
+wood burning (art)
+putting on foundation
+putting on shoes
+cumbia
+archaeological excavation
+mountain climber (exercise)
+assembling bicycle
+head stand
+cutting apple
+shuffling feet
+bottling
+breathing fire
+using inhaler
+historical reenactment
+hugging baby
+mushroom foraging
+delivering mail
+laying tiles
+using atm
+chopping meat
+tightrope walking
+mosh pit dancing
+photobombing
+coloring in
+huddling
+playing gong
+laying concrete
+breaking boards
+acting in play
+base jumping
+tie dying
+using a sledge hammer
+playing ping pong
+photocopying
+winking
+waking up
+swinging baseball bat
+twiddling fingers
+playing polo
+longboarding
+ironing hair
+bathing dog
+moon walking
+playing marbles
+embroidering
+playing beer pong
+home roasting coffee
+gold panning
+karaoke
+changing gear in car
+raising eyebrows
+yarn spinning
+scrubbing face
+fidgeting
+planing wood
+cosplaying
+capsizing
+tackling
+shining flashlight
+dyeing eyebrows
+drooling
+alligator wrestling
+playing blackjack
+carving ice
+playing maracas
+opening refrigerator
+throwing knife
+putting in contact lenses
+passing soccer ball
+casting fishing line
+sucking lolly
+installing carpet
+bulldozing
+roasting marshmallows
+playing darts
+chopping vegetables
+bull fighting
diff --git a/tools/data/mit/README.md b/tools/data/mit/README.md
index e67ca45335..69828024d9 100644
--- a/tools/data/mit/README.md
+++ b/tools/data/mit/README.md
@@ -34,7 +34,7 @@ python ../resize_videos.py ../../../data/mit/videos/ ../../../data/mit/videos_25
This part is **optional** if you only want to use the video loader.
-Before extracting, please refer to [install.md](/docs/install.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
+Before extracting, please refer to [install.md](/docs/en/get_started/installation.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
If you have plenty of SSD space, then we recommend extracting frames there for better I/O performance. And you can run the following script to soft link the extracted frames.
@@ -125,4 +125,4 @@ mmaction2
```
-For training and evaluating on Moments in Time, please refer to [getting_started.md](/docs/getting_started.md).
+For training and evaluating on Moments in Time, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).
diff --git a/tools/data/mit/README_zh-CN.md b/tools/data/mit/README_zh-CN.md
index 74a3d0c247..1eaa6eafb2 100644
--- a/tools/data/mit/README_zh-CN.md
+++ b/tools/data/mit/README_zh-CN.md
@@ -36,7 +36,7 @@ python ../resize_videos.py ../../../data/mit/videos/ ../../../data/mit/videos_25
如果用户只想使用视频加载训练，则该部分是 **可选项**。
-在抽取视频帧和光流之前，请参考 [安装指南](/docs_zh_CN/install.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
+在抽取视频帧和光流之前，请参考 [安装指南](/docs/zh_cn/get_started/installation.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
如果用户有大量的 SSD 存储空间，则推荐将抽取的帧存储至 I/O 性能更优秀的 SSD 上。
用户可使用以下命令为 SSD 建立软链接。
@@ -127,4 +127,4 @@ mmaction2
```
-关于对 Moments in Times 进行训练和验证，可以参照 [基础教程](/docs_zh_CN/getting_started.md)。
+关于对 Moments in Times 进行训练和验证，可以参照 [训练教程](/docs/zh_cn/user_guides/train_test.md)。
diff --git a/tools/data/mmit/README.md b/tools/data/mmit/README.md
index 5deedf71d0..cadd8ab705 100644
--- a/tools/data/mmit/README.md
+++ b/tools/data/mmit/README.md
@@ -32,7 +32,7 @@ python ../resize_videos.py ../../../data/mmit/videos/ ../../../data/mmit/videos_
This part is **optional** if you only want to use the video loader.
-Before extracting, please refer to [install.md](/docs/install.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
+Before extracting, please refer to [install.md](/docs/en/get_started/installation.md) for installing [denseflow](https://github.com/open-mmlab/denseflow).
First, you can run the following script to soft link SSD.
@@ -110,4 +110,4 @@ mmaction2/
└── ...
```
-For training and evaluating on Multi-Moments in Time, please refer to [getting_started.md](/docs/getting_started.md).
+For training and evaluating on Multi-Moments in Time, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).
diff --git a/tools/data/mmit/README_zh-CN.md b/tools/data/mmit/README_zh-CN.md
index e070505e34..231a5aac68 100644
--- a/tools/data/mmit/README_zh-CN.md
+++ b/tools/data/mmit/README_zh-CN.md
@@ -34,7 +34,7 @@ python ../resize_videos.py ../../../data/mmit/videos/ ../../../data/mmit/videos_
如果用户只想使用视频加载训练，则该部分是 **可选项**。
-在抽取视频帧和光流之前，请参考 [安装指南](/docs_zh_CN/install.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
+在抽取视频帧和光流之前，请参考 [安装指南](/docs/zh_cn/get_started/installation.md) 安装 [denseflow](https://github.com/open-mmlab/denseflow)。
如果用户有大量的 SSD 存储空间，则推荐将抽取的帧存储至 I/O 性能更优秀的 SSD 上。
用户可使用以下命令为 SSD 建立软链接。
@@ -112,4 +112,4 @@ mmaction2/
└── ...
```
-关于对 Multi-Moments in Time 进行训练和验证，可以参照 [基础教程](/docs_zh_CN/getting_started.md)。
+关于对 Multi-Moments in Time 进行训练和验证，可以参照 [训练教程](/docs/zh_cn/user_guides/train_test.md)。
diff --git a/tools/data/multisports/README.md b/tools/data/multisports/README.md
new file mode 100644
index 0000000000..98fe25698f
--- /dev/null
+++ b/tools/data/multisports/README.md
@@ -0,0 +1,111 @@
+# Preparing Multisports
+
+## Introduction
+
+
+
+```BibTeX
+@inproceedings{li2021multisports,
+ title={Multisports: A multi-person video dataset of spatio-temporally localized sports actions},
+ author={Li, Yixuan and Chen, Lei and He, Runyu and Wang, Zhenzhi and Wu, Gangshan and Wang, Limin},
+ booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
+ pages={13536--13545},
+ year={2021}
+}
+```
+
+For basic dataset information, please refer to the official [project](https://deeperaction.github.io/datasets/multisports.html) and the [paper](https://arxiv.org/abs/2105.07404).
+Before we start, please make sure that the directory is located at `$MMACTION2/tools/data/multisports/`.
+
+## Step 1. Prepare Annotations
+
+First of all, you have to download annotations and videos to `$MMACTION2/data/multisports` on the official [website](https://github.com/MCG-NJU/MultiSports), please also download the Person Boxes and put it to `$MMACTION2/data/multisports`.
+
+## Step 2. Prepare Videos
+
+Before this step, please make sure the folder structure looks like:
+
+```
+mmaction2
+├── mmaction
+├── tools
+├── configs
+├── data
+│ ├── multisports
+│ | ├── MultiSports_box.zip
+│ | ├── trainval
+│ | | ├── aerobic_gymnastics.zip
+│ | | ├── basketball.zip
+│ | | ├── multisports_GT.pkl
+│ | | ├──...
+│ | ├── test
+│ | | ├── aerobic_gymnastics.zip
+│ | | ├── basketball.zip
+│ | | ├──...
+```
+
+Then, you can use the following command to uncompress.
+
+```shell
+cd $MMACTION2/data/multisports/
+unzip MultiSports_box.zip
+cd $MMACTION2/data/multisports/trainval
+find . -name '*.zip' -exec unzip {} \;
+cd $MMACTION2/data/multisports/test
+find . -name '*.zip' -exec unzip {} \;
+cd $MMACTION2/tools/data/multisports/
+```
+
+## Step 3. Convert Annotations
+
+you can run the following script to convert annotations and proposals as we need.
+
+```shell
+cd $MMACTION2/tools/data/multisports/
+python parse_anno.py
+```
+
+## Step 5. Check Directory Structure
+
+After the whole data process, you will get the videos and annotation files for MultiSports.
+
+In the context of the whole project (for MultiSports only), the folder structure will look like:
+
+```
+mmaction2
+├── mmaction
+├── tools
+├── configs
+├── data
+│ ├── multisports
+│ | ├── annotations
+| │ | ├── multisports_dense_proposals_test.recall_96.13.pkl
+| │ | ├── multisports_dense_proposals_train.recall_96.13.pkl
+| │ | ├── multisports_dense_proposals_val.recall_96.13.pkl
+| │ | ├── multisports_GT.pkl
+| │ | ├── multisports_train.csv
+| │ | ├── multisports_val.csv
+│ | ├── trainval
+│ | | ├── aerobic_gymnastics
+| │ | | ├── v__wAgwttPYaQ_c001.mp4
+| │ | | ├── v__wAgwttPYaQ_c002.mp4
+| │ | | ├── ...
+│ | | ├── basketball
+| │ | | ├── v_-6Os86HzwCs_c001.mp4
+| │ | | ├── v_-6Os86HzwCs_c002.mp4
+| │ | | ├── ...
+│ | | ├── multisports_GT.pkl
+│ | | ├──...
+│ | ├── test
+│ | | ├── aerobic_gymnastics
+| │ | | ├── v_2KroSzspz-c_c001.mp4
+| │ | | ├── v_2KroSzspz-c_c002.mp4
+| │ | | ├── ...
+│ | | ├── basketball
+| │ | | ├── v_1tefH1iPbGM_c001.mp4
+| │ | | ├── v_1tefH1iPbGM_c002.mp4
+│ | | ├──...
+```
+
+We don't need the zip files under the project, you can handle them as you want.
+For training and evaluating on MultiSports, please refer to [Training and Test Tutorial](/docs/en/user_guides/train_test.md).
diff --git a/tools/data/multisports/README_zh-CN.md b/tools/data/multisports/README_zh-CN.md
new file mode 100644
index 0000000000..3e0470f12e
--- /dev/null
+++ b/tools/data/multisports/README_zh-CN.md
@@ -0,0 +1,111 @@
+# 准备 MultiSports
+
+## 介绍
+
+
+
+```BibTeX
+@inproceedings{li2021multisports,
+ title={Multisports: A multi-person video dataset of spatio-temporally localized sports actions},
+ author={Li, Yixuan and Chen, Lei and He, Runyu and Wang, Zhenzhi and Wu, Gangshan and Wang, Limin},
+ booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
+ pages={13536--13545},
+ year={2021}
+}
+```
+
+关于基本数据集信息，请参考官方 [项目](https://deeperaction.github.io/datasets/multisports.html) 和 [论文](https://arxiv.org/abs/2105.07404)。
+在我们开始之前，请确保目录位于 `$MMACTION2/tools/data/multisports/`。
+
+## 第一步：准备标注
+
+首先，你必须从官方 [网站](https://github.com/MCG-NJU/MultiSports) 下载标注和视频到 `$MMACTION2/data/multisports`，请同时下载人物检测框并将其放到 `$MMACTION2/data/multisports`。
+
+## 第二步：准备视频
+
+在这一步之前，请确保文件夹结构如下：
+
+```
+mmaction2
+├── mmaction
+├── tools
+├── configs
+├── data
+│ ├── multisports
+│ | ├── MultiSports_box.zip
+│ | ├── trainval
+│ | | ├── aerobic_gymnastics.zip
+│ | | ├── basketball.zip
+│ | | ├── multisports_GT.pkl
+│ | | ├──...
+│ | ├── test
+│ | | ├── aerobic_gymnastics.zip
+│ | | ├── basketball.zip
+│ | | ├──...
+```
+
+然后，你可以使用以下命令进行解压。
+
+```shell
+cd $MMACTION2/data/multisports/
+unzip MultiSports_box.zip
+cd $MMACTION2/data/multisports/trainval
+find . -name '*.zip' -exec unzip {} \;
+cd $MMACTION2/data/multisports/test
+find . -name '*.zip' -exec unzip {} \;
+cd $MMACTION2/tools/data/multisports/
+```
+
+## 第三步：转换标注文件
+
+你可以运行以下脚本来转换我们需要的标注文件和候选框。
+
+```shell
+cd $MMACTION2/tools/data/multisports/
+python parse_anno.py
+```
+
+## 第五步：检查目录结构
+
+完成整个数据处理后，你将得到 MultiSports 数据集的视频和标注文件。
+
+在整个项目的目录中（仅针对 MultiSports），文件夹结构如下：
+
+```
+mmaction2
+├── mmaction
+├── tools
+├── configs
+├── data
+│ ├── multisports
+│ | ├── annotations
+| │ | ├── multisports_dense_proposals_test.recall_96.13.pkl
+| │ | ├── multisports_dense_proposals_train.recall_96.13.pkl
+| │ | ├── multisports_dense_proposals_val.recall_96.13.pkl
+| │ | ├── multisports_GT.pkl
+| │ | ├── multisports_train.csv
+| │ | ├── multisports_val.csv
+│ | ├── trainval
+│ | | ├── aerobic_gymnastics
+| │ | | ├── v__wAgwttPYaQ_c001.mp4
+| │ | | ├── v__wAgwttPYaQ_c002.mp4
+| │ | | ├── ...
+│ | | ├── basketball
+| │ | | ├── v_-6Os86HzwCs_c001.mp4
+| │ | | ├── v_-6Os86HzwCs_c002.mp4
+| │ | | ├── ...
+│ | | ├── multisports_GT.pkl
+│ | | ├──...
+│ | ├── test
+│ | | ├── aerobic_gymnastics
+| │ | | ├── v_2KroSzspz-c_c001.mp4
+| │ | | ├── v_2KroSzspz-c_c002.mp4
+| │ | | ├── ...
+│ | | ├── basketball
+| │ | | ├── v_1tefH1iPbGM_c001.mp4
+| │ | | ├── v_1tefH1iPbGM_c002.mp4
+│ | | ├──...
+```
+
+我们不需要项目下的 zip 文件，你可以按照自己的意愿处理它们。
+关于在 MultiSports 上进行训练和评估，请参考 [训练和测试教程](/docs/en/user_guides/train_test.md)。
diff --git a/tools/data/multisports/format_det_result.py b/tools/data/multisports/format_det_result.py
new file mode 100644
index 0000000000..84fd78811e
--- /dev/null
+++ b/tools/data/multisports/format_det_result.py
@@ -0,0 +1,62 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+from argparse import ArgumentParser
+
+import numpy as np
+from mmengine import dump, load
+from rich.progress import track
+
+from mmaction.evaluation import link_tubes
+
+
+def parse_args():
+ parser = ArgumentParser()
+ parser.add_argument('test-result', help='path of dumped reuslts')
+ parser.add_argument(
+ '--anno-path',
+ default='data/multisports/videos/trainval/multisports_GT.pkl')
+ parser.add_argument(
+ '--frm_out_path',
+ default=None,
+ help='frame-level detection results output path')
+ parser.add_argument(
+ '--tube_out_path',
+ default=None,
+ help='tube-level detection results output path')
+ args = parser.parse_args()
+ if not args.frm_out_path:
+ args.frm_out_path = args.test_result[:-4] + '-formated.pkl'
+ if not args.tube_out_path:
+ args.tube_out_path = args.test_result[:-4] + '_vid_dets.pkl'
+ return args
+
+
+def format_det_result():
+ """convert test results to specified format in MultiSports competition."""
+ test_results = load(args.test_result)
+ annos = load(args.anno_path)
+ test_videos = annos['test_videos'][0]
+ resolutions = annos['resolution']
+ frm_dets = []
+ for pred in track(test_results, description='formating...'):
+ video_key = pred['video_id'].split('.mp4')[0]
+ frm_num = pred['timestamp']
+ bboxes = pred['pred_instances']['bboxes']
+ cls_scores = pred['pred_instances']['scores']
+ for bbox, cls_score in zip(bboxes, cls_scores):
+ video_idx = test_videos.index(video_key)
+ pred_label = np.argmax(cls_score)
+ score = cls_score[pred_label]
+ h, w = resolutions[video_key]
+ bbox *= np.array([w, h, w, h])
+ instance_result = np.array(
+ [video_idx, frm_num, pred_label, score, *bbox])
+ frm_dets.append(instance_result)
+ frm_dets = np.array(frm_dets)
+ video_tubes = link_tubes(annos, frm_dets, K=1)
+ dump(frm_dets, args.frm_out_path)
+ dump(video_tubes, args.tube_out_path)
+
+
+if __name__ == '__main__':
+ args = parse_args()
+ format_det_result()
diff --git a/tools/data/multisports/label_map.txt b/tools/data/multisports/label_map.txt
new file mode 100644
index 0000000000..b710721362
--- /dev/null
+++ b/tools/data/multisports/label_map.txt
@@ -0,0 +1,66 @@
+0: aerobic_push_up
+1: aerobic_explosive_push_up
+2: aerobic_explosive_support
+3: aerobic_leg_circle
+4: aerobic_helicopter
+5: aerobic_support
+6: aerobic_v_support
+7: aerobic_horizontal_support
+8: aerobic_straight_jump
+9: aerobic_illusion
+10: aerobic_bent_leg(s)_jump
+11: aerobic_pike_jump
+12: aerobic_straddle_jump
+13: aerobic_split_jump
+14: aerobic_scissors_leap
+15: aerobic_kick_jump
+16: aerobic_off_axis_jump
+17: aerobic_butterfly_jump
+18: aerobic_split
+19: aerobic_turn
+20: aerobic_balance_turn
+21: volleyball_serve
+22: volleyball_block
+23: volleyball_first_pass
+24: volleyball_defend
+25: volleyball_protect
+26: volleyball_second_pass
+27: volleyball_adjust
+28: volleyball_save
+29: volleyball_second_attack
+30: volleyball_spike
+31: volleyball_dink
+32: volleyball_no_offensive_attack
+33: football_shoot
+34: football_long_pass
+35: football_short_pass
+36: football_through_pass
+37: football_cross
+38: football_dribble
+39: football_trap
+40: football_throw
+41: football_diving
+42: football_tackle
+43: football_steal
+44: football_clearance
+45: football_block
+46: football_press
+47: football_aerial_duels
+48: basketball_pass
+49: basketball_drive
+50: basketball_dribble
+51: basketball_3-point_shot
+52: basketball_2-point_shot
+53: basketball_free_throw
+54: basketball_block
+55: basketball_offensive_rebound
+56: basketball_defensive_rebound
+57: basketball_pass_steal
+58: basketball_dribble_steal
+59: basketball_interfere_shot
+60: basketball_pick-and-roll_defensive
+61: basketball_sag
+62: basketball_screen
+63: basketball_pass-inbound
+64: basketball_save
+65: basketball_jump_ball
diff --git a/tools/data/multisports/parse_anno.py b/tools/data/multisports/parse_anno.py
new file mode 100644
index 0000000000..bb8ed66316
--- /dev/null
+++ b/tools/data/multisports/parse_anno.py
@@ -0,0 +1,96 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+import csv
+import os
+import os.path as osp
+from argparse import ArgumentParser
+
+import numpy as np
+from mmengine import dump, list_dir_or_file, load
+
+
+def parse_args():
+ parser = ArgumentParser()
+ parser.add_argument(
+ '--data-root',
+ default='data/multisports',
+ help='the directory to multisports annotations')
+ parser.add_argument(
+ '--out-root',
+ default='data/multisports',
+ help='output directory of output annotation files')
+ parser.add_argument('--dump-proposals', action='store_true')
+ args = parser.parse_args()
+ return args
+
+
+def parse_anno(args):
+ if not osp.exists(args.out_root):
+ os.makedirs(osp.join(args.out_root, 'annotations'))
+
+ anno_path = osp.join(args.data_root, 'annotations/multisports_GT.pkl')
+ annos = load(anno_path)
+
+ # convert key in proposal file to filename
+ key2filename = {
+ video.split('/')[1]: video + '.mp4'
+ for video in annos['nframes'].keys()
+ }
+ test_videos = [
+ file for file in list_dir_or_file(
+ osp.join(args.data_root, 'test'), recursive=True)
+ if file.endswith('.mp4')
+ ]
+ key2filename.update(
+ {video.split('/')[1][:-4]: video
+ for video in test_videos})
+ # convert proposal bboxes
+ if args.dump_proposals:
+ proposals_path = osp.join(args.data_root,
+ 'annotations/MultiSports_box')
+ for proposals in os.listdir(proposals_path):
+ proposal_info = load(osp.join(proposals_path, proposals))
+ proposal_out = dict()
+ for key in proposal_info.keys():
+ key_split = key.split(',')
+ if key_split[0] in key2filename.keys():
+ new_key = \
+ f'{key2filename[key_split[0]]},{int(key_split[1]):04d}'
+ proposal_out[new_key] = proposal_info[key]
+ target_path = osp.join(args.out_root, 'annotations',
+ 'multisports_dense_proposals_' + proposals)
+ dump(proposal_out, target_path)
+ # dump train and val list
+ for split in ['train', 'val']:
+ out_anno_path = osp.join(args.out_root, 'annotations',
+ f'multisports_{split}.csv')
+ with open(out_anno_path, 'w') as csv_f:
+ writer = csv.writer(csv_f)
+ if split == 'train':
+ video_list = annos['train_videos'][0]
+ elif split == 'val':
+ video_list = annos['test_videos'][0]
+ gt_tubes = annos['gttubes']
+ resolutions = annos['resolution']
+ for video_id in video_list:
+ vid_tubes = gt_tubes[video_id]
+ h, w = resolutions[video_id]
+ for label, tubes in vid_tubes.items():
+ entity_id = 0
+ for tube in tubes:
+ for frame_anno in tube:
+ frame_stamp = int(frame_anno[0])
+ entity_box = frame_anno[1:]
+ entity_box /= np.array([w, h, w, h])
+ entity_box = [f'{num:.3f}' for num in entity_box]
+ filename = video_id + '.mp4'
+ anno_line = [
+ filename, frame_stamp, *entity_box, label,
+ entity_id
+ ]
+ writer.writerow(anno_line)
+ entity_id += 1
+
+
+if __name__ == '__main__':
+ args = parse_args()
+ parse_anno(args)
diff --git a/tools/data/omnisource/trim_raw_video.py b/tools/data/omnisource/trim_raw_video.py
index 81aef77140..6f31103755 100644
--- a/tools/data/omnisource/trim_raw_video.py
+++ b/tools/data/omnisource/trim_raw_video.py
@@ -4,7 +4,7 @@
import sys
from subprocess import check_output
-import mmcv
+import mmengine
def get_duration(vid_name):
@@ -28,7 +28,7 @@ def trim(vid_name):
# We output 10-second clips into the folder `name`
dest = name
- mmcv.mkdir_or_exist(dest)
+ mmengine.mkdir_or_exist(dest)
command_tmpl = ('ffmpeg -y loglevel error -i {} -ss {} -t {} -crf 18 '
'-c:v libx264 {}/part_{}.mp4')
diff --git a/tools/data/skeleton/README.md b/tools/data/skeleton/README.md
index 10244d23a1..2f55a2021e 100644
--- a/tools/data/skeleton/README.md
+++ b/tools/data/skeleton/README.md
@@ -58,7 +58,7 @@ Each pickle file corresponds to an action recognition dataset. The content of a
## Visualization
-For skeleton data visualization, you need also to prepare the RGB videos. Please refer to [visualize_heatmap_volume](/demo/visualize_heatmap_volume.ipynb) for detailed process. Here we provide some visualization examples from NTU-60 and FineGYM.
+For skeleton data visualization, you need also to prepare the RGB videos. Please refer to \[visualize_heatmap_volume\] for detailed process. Here we provide some visualization examples from NTU-60 and FineGYM.