diff --git a/research/cv/m2det/README.md b/research/cv/m2det/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..189ae2688439e875896b2ecd8f7ea4cbefd69626
--- /dev/null
+++ b/research/cv/m2det/README.md
@@ -0,0 +1,299 @@
+# Contents
+
+- [Contents](#contents)
+ - [M2Det Description](#m2det-description)
+ - [Model Architecture](#model-architecture)
+ - [Dataset](#dataset)
+ - [Dataset used: COCO](#dataset-used-ms-coco)
+ - [Dataset organize way](#dataset-organize-way)
+ - [Environment Requirements](#environment-requirements)
+ - [Quick Start](#quick-start)
+ - [Script Description](#script-description)
+ - [Script and Sample Code](#script-and-sample-code)
+ - [Parameter configuration](#parameter-configuration)
+ - [Training Process](#training-process)
+ - [Training](#training)
+ - [Run M2Det on GPU](#run-m2det-on-gpu)
+ - [Evaluation Process](#evaluation-process)
+ - [Evaluation](#evaluation)
+ - [Model Description](#model-description)
+ - [Performance](#performance)
+ - [Training Performance](#training-performance)
+ - [Evaluation Performance](#evaluation-performance)
+ - [Description of Random Situation](#description-of-random-situation)
+ - [ModelZoo Homepage](#modelzoo-homepage)
+
+## [M2Det Description](#contents)
+
+M2Det (Multi-Level Multi-Scale Detector) is an end-to-end one-stage object detection model. It uses Multi-Level Feature Pyramid Network (MLFPN) to extract features from input image, and then produces dense bounding boxes and category scores.
+
+[Paper](https://qijiezhao.github.io/imgs/m2det.pdf): Q. Zhao, T. Sheng, Y.Wang, Zh. Tang, Y. Chen, L. Cai, H. Ling. M2Det: A Single-Shot Object Detector base on Multi-Level Feature Pyramid Network.
+
+## [Model Architecture](#contents)
+
+M2Det consists of several modules. Feature Fusion Module (FFM) rescales and concatenates features from several backbone feature layers (VGG, ResNet, etc.) to produce base feature for further modules. Thinned U-shape Modules (TUMs) use encoder-decoder architecture to produce multi-level multi-scale features, which afterwards aggregated by Scale-wise Aggregation Module (SFAM). Resulting Multi-Level Feature Pyramid is used by prediction layers to achieve local bounding box regression and classification.
+
+## [Dataset](#contents)
+
+Note that you can run the scripts based on the dataset mentioned in original paper or widely used in relevant domain/network architecture. In the following sections, we will introduce how to run the scripts using the related dataset below.
+
+### Dataset used: [MS-COCO](https://cocodataset.org/#download)
+
+COCO is a large-scale object detection, segmentation, and captioning dataset. The COCO train, validation, and test sets, containing more than 200,000 images and 80 object categories. All object instance are annotated with bounding boxes and detailed segmentation mask.
+
+For training the M2Det model, download the following files:
+
+- 2014 Train images [83K / 13GB]
+- 2014 Val images [41K / 6GB]
+- 2015 Test images [81K / 12GB]
+- 2014 Train/Val annotations [241MB]
+- 2014 Testing Image info [1MB]
+- 2015 Testing Image info [2MB]
+
+#### Dataset organize way
+
+```text
+.
+└─ coco
+ ├─ annotations
+ ├── captions_train2014.json
+ ├── captions_val2014.json
+ ├── image_info_test2014.json
+ ├── image_info_test2015.json
+ ├── image_info_test-dev2015.json
+ ├── instances_minival2014.json
+ ├── instances_train2014.json
+ ├── instances_val2014.json
+ └── instances_valminusminival2014.json
+ ├─images
+ ├── test2015
+ └── COCO_test2015_*.jpg
+ ├── train2014
+ └── COCO_train2014_*.jpg
+ └── val2014
+ └── COCO_val2014_*.jpg
+
+...
+```
+
+You can find instances_minival2014.json and instances_valminusminival2014.json here: http://datasets.d2.mpi-inf.mpg.de/hosang17cvpr/coco_minival2014.tar.gz
+
+## [Environment Requirements](#contents)
+
+- Hardware（GPU）
+ - Prepare hardware environment with a GPU processor.
+- Framework
+ - [MindSpore](https://www.mindspore.cn/install/en)
+- For more information, please check the resources below：
+ - [MindSpore Tutorials](https://www.mindspore.cn/tutorials/en/master/index.html)
+ - [MindSpore Python API](https://www.mindspore.cn/docs/api/en/master/index.html)
+
+## [Quick Start](#contents)
+
+After installing MindSpore via the official website, specify dataset location in `src/config.py` file.
+Run Soft-NMS building script with the following command:
+
+```bash
+bash ./scripts/make.sh
+```
+
+Download pretrained VGG-16 backbone from https://s3.amazonaws.com/amdegroot-models/vgg16_reducedfc.pth
+Convert pretrained VGG-16 backbone to Mindspore format with the following command:
+
+```bash
+bash ./scripts/convert_vgg.sh /path/to/vgg16_reducedfc.pth
+```
+
+A converted checkpoint will be in the same directory as the original file, but with ".ckpt" extension.
+
+You can start training and evaluation as follows:
+
+- Training
+
+ For GPU training, set `device = 'GPU'` in `src/config.py`.
+
+ ```bash
+ # Single GPU training
+ bash ./scripts/run_standalone_train.sh [DEVICE_ID] [PRETRAINED_BACKBONE] [DATASET_PATH]
+
+ # Multi-GPU training
+ bash ./scripts/run_distributed_train_gpu.sh [RANK_SIZE] [DEVICE_START] [PRETRAINED_BACKBONE] [DATASET_PATH]
+ ```
+
+ Example：
+
+ ```bash
+ # Single GPU training
+ bash ./scripts/run_standalone_train.sh 0 /path/to/vgg16_reducedfc.ckpt /path/to/COCO/
+
+ # Multi-GPU training
+ bash ./scripts/run_distributed_train_gpu.sh 8 0 /path/to/vgg16_reducedfc.ckpt /path/to/COCO/
+ ```
+
+- Evaluation：
+
+ ```bash
+ bash ./scripts/run_eval.sh [DEVICE_ID] [CHECKPOINT_PATH] [DATASET_PATH]
+ ```
+
+ Example：
+
+ ```bash
+ bash ./scripts/run_eval.sh 0 /path/to/checkpoint /path/to/COCO/
+ ```
+
+## [Script Description](#contents)
+
+### [Script and Sample Code](#contents)
+
+```text
+|-- README.md # English README
+|-- convert.py # Script for pretrained VGG backbone conversion
+|-- eval.py # Evaluation
+|-- export.py # MINDIR model export
+|-- requirements.txt # pip dependencies
+|-- scripts
+| |-- convert_vgg.sh # Script for pretrained VGG backbone conversion
+| |-- make.sh # Script for building Soft-NMS function
+| |-- run_distributed_train_gpu.sh # GPU distributed training script
+| |-- run_eval.sh # Evaluation script
+| |-- run_export.sh # MINDIR model export script
+| `-- run_standalone_train.sh # Single-device training script
+|-- src
+| |-- nms
+ `-- cpu_nms.pyx # Soft-NMS algorithm
+| |-- box_utils.py # Function for bounding boxes processing
+| |-- build.py # Script for building Soft-NMS function
+| |-- callback.py # Custom callback functions
+| |-- coco_utils.py # COCO dataset functions
+| |-- config.py # Configuration file
+| |-- dataset.py # Dataset loader
+| |-- detector.py # Bounding box detector
+| |-- loss.py # Multibox loss function
+| |-- lr_scheduler.py # Learning rate scheduler utilities
+| |-- model.py # M2Det model architecture
+| |-- priors.py # SSD prior boxes definition
+| `-- utils.py # General utilities
+`-- train.py # Training
+```
+
+### [Parameter configuration](#contents)
+
+Parameters for both training and evaluation can be set in `src/config.py`.
+
+```python
+random_seed = 1
+experiment_tag = 'm2det512_vgg16_lr_7.5e-4'
+
+train_cfg = dict(
+ lr = 7.5e-4,
+ warmup = 5,
+ per_batch_size = 7,
+ gamma = [0.5, 0.2, 0.1, 0.1],
+ lr_epochs = [90, 110, 130, 150, 160],
+ total_epochs = 160,
+ print_epochs = 10,
+ num_workers = 3,
+ )
+
+test_cfg = dict(
+ cuda = True,
+ topk = 0,
+ iou = 0.45,
+ soft_nms = True,
+ score_threshold = 0.1,
+ keep_per_class = 50,
+ save_folder = 'eval'
+ )
+
+optimizer = dict(
+ type='SGD',
+ momentum=0.9,
+ weight_decay=0.00005,
+ loss_scale=1,
+ dampening=0.0,
+ clip_grad_norm=5.)
+```
+
+### [Training Process](#contents)
+
+#### Training
+
+##### Run M2Det on GPU
+
+For GPU training, set `device = 'GPU'` in `src/config.py`.
+
+- Training using single device (1p)
+
+ ```bash
+ bash ./scripts/run_standalone_train.sh 0 /path/to/vgg16_reducedfc.ckpt /path/to/COCO/
+ ```
+
+- Distributed Training (8p)
+
+ ```bash
+ bash ./scripts/run_distributed_train_gpu.sh 8 0 /path/to/vgg16_reducedfc.ckpt /path/to/COCO/
+ ```
+
+Checkpoints will be saved in `./checkpoints/[EXPERIMENT_TAG]` folder. Checkpoint filename format: `[MODEL.M2DET_CONFIG.BACKBONE]_[MODEL.INPUT_SIZE]-[EPOCH]_[ITERATION].ckpt`. Final checkpoint filename format: `[MODEL.M2DET_CONFIG.BACKBONE]_[MODEL.INPUT_SIZE]-final.ckpt`
+
+### [Evaluation Process](#contents)
+
+#### Evaluation
+
+To start evaluation, run the following command:
+
+```bash
+bash ./scripts/run_eval.sh [DEVICE_ID] [CHECKPOINT_PATH] [DATASET_PATH]
+
+# Example:
+bash ./scripts/run_eval.sh 0 /path/to/checkpoint /path/to/COCO/
+```
+
+## [Model Description](#contents)
+
+### [Performance](#contents)
+
+#### Training Performance
+
+Training performance in the following tables is obtained by the M2Det-512-VGG16 model based on the COCO dataset:
+
+| Parameters | M2Det-512-VGG16 (8GPU) |
+| ------------------- | -------------------|
+| Model Version | M2Det-512-VGG16 |
+| Resource | Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz, 8x V100-PCIE |
+| Uploaded Date | 2022-06-29 |
+| MindSpore version | 1.5.2 |
+| Dataset | COCO |
+| Training Parameters | seed=1；epoch=160；batch_size = 7；lr=1e-3；weight_decay = 5e-5, clip_by_global_norm = 4.0 |
+| Optimizer | SGD |
+| Loss Function | Multibox MSE loss |
+| Outputs | Bounding boxes and class scores |
+| Loss value | 2.299 |
+| Average checkpoint (.ckpt file) size | 507 Mb |
+| Speed | 707 ms/step, 1493 s/epoch |
+| Total time | 2 days 18 hours 16 minutes |
+| Scripts | [M2Det training script](https://gitee.com/mindspore/models/tree/master/research/cv/m2det/train.py) |
+
+#### Evaluation Performance
+
+- Evaluation performance in the following tables is obtained by the M2Det-512-VGG16 model based on the COCO dataset:
+
+| Parameters | M2Det-512-VGG16 (8GPU) |
+| ------------------- | ------------------- |
+| Model Version | M2Det-512-VGG16 |
+| Resource | Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz, 8x V100-PCIE |
+| Uploaded Date | 2022-06-29 |
+| MindSpore version | 1.5.2 |
+| Dataset | COCO |
+| Loss Function | Multibox MSE loss |
+| AP | 36.2 |
+| Scripts | [M2Det evaluation script](https://gitee.com/mindspore/models/tree/master/research/cv/m2det/eval.py) |
+
+## [Description of Random Situation](#contents)
+
+Global training random seed is fixed in `src/config.py` with `random_seed` parameter. 'None' value will execute training without dataset shuffle.
+
+## [ModelZoo Homepage](#contents)
+
+Please check the official [homepage](https://gitee.com/mindspore/models).
diff --git a/research/cv/m2det/convert.py b/research/cv/m2det/convert.py
new file mode 100644
index 0000000000000000000000000000000000000000..063ef65e707cea8dacb6a85dccf8598e874fab7d
--- /dev/null
+++ b/research/cv/m2det/convert.py
@@ -0,0 +1,88 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""Convert the VGG backbone from the PyTorch format to the MindSpore format"""
+import argparse
+from pathlib import Path
+
+import torch
+from mindspore import Tensor
+from mindspore import context
+from mindspore import save_checkpoint
+
+from src.model import VGG as ms_vgg
+
+_VGG_PARAM = [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'C', 512, 512, 512, 'M', 512, 512, 512]
+_IN_CHANNELS = 3
+
+
+def pytorch_vgg(cfg, i, batch_norm=False):
+ """Build VGG model"""
+ layers = []
+ in_channels = i
+ for v in cfg:
+ if v == 'M':
+ layers += [torch.nn.MaxPool2d(kernel_size=2, stride=2)]
+ elif v == 'C':
+ layers += [torch.nn.MaxPool2d(kernel_size=2, stride=2, ceil_mode=True)]
+ else:
+ conv2d = torch.nn.Conv2d(in_channels, v, kernel_size=3, padding=1)
+ if batch_norm:
+ layers += [conv2d, torch.nn.BatchNorm2d(v), torch.nn.ReLU(inplace=True)]
+ else:
+ layers += [conv2d, torch.nn.ReLU(inplace=True)]
+ in_channels = v
+ pool5 = torch.nn.MaxPool2d(kernel_size=3, stride=1, padding=1)
+ conv6 = torch.nn.Conv2d(512, 1024, kernel_size=3, padding=6, dilation=6)
+ conv7 = torch.nn.Conv2d(1024, 1024, kernel_size=1)
+ layers += [pool5, conv6,
+ torch.nn.ReLU(inplace=True), conv7, torch.nn.ReLU(inplace=True)]
+ return layers
+
+
+def convert():
+ """Convert the checkpoint from PyTorch to the MindSpore format"""
+ context.set_context(mode=context.PYNATIVE_MODE, device_target="CPU")
+
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--backbone_path", type=str, required=True, help="Path to pre-trained VGG backbone")
+ args = parser.parse_args()
+
+ backbone_path = Path(args.backbone_path).resolve()
+
+ if not backbone_path.exists():
+ raise ValueError('Pre-trained VGG backbone does not exist')
+
+ ms_model = ms_vgg(_VGG_PARAM, _IN_CHANNELS, batch_norm=False, pretrained=None)
+ pt_model = torch.nn.ModuleList(pytorch_vgg(_VGG_PARAM, _IN_CHANNELS, batch_norm=False))
+ weights = torch.load(args.backbone_path)
+ pt_model.load_state_dict(weights)
+
+ pt_names = {}
+ for name, param in pt_model.named_parameters():
+ pt_names['layers.' + name] = param.cpu().detach().numpy()
+
+ for param in ms_model.trainable_params():
+ param.set_data(Tensor(pt_names[param.name]))
+
+ converted_ckpt_path = backbone_path.with_suffix('.ckpt')
+
+ print(args.backbone_path)
+ print(converted_ckpt_path)
+ save_checkpoint(ms_model, str(converted_ckpt_path))
+ print(f'Succesfully converted VGG backbone. Path to checkpoint: {converted_ckpt_path}')
+
+
+if __name__ == '__main__':
+ convert()
diff --git a/research/cv/m2det/eval.py b/research/cv/m2det/eval.py
new file mode 100644
index 0000000000000000000000000000000000000000..84399e6858a7b8ec43364ef02a42f06fbc68dcd8
--- /dev/null
+++ b/research/cv/m2det/eval.py
@@ -0,0 +1,145 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+""" M2Det evaluation """
+
+import argparse
+import os
+import pickle
+
+from mindspore import context
+from mindspore import load_checkpoint
+from tqdm import tqdm
+
+from src import config as cfg
+from src.dataset import BaseTransform
+from src.dataset import get_dataset
+from src.detector import Detect
+from src.model import get_model
+from src.priors import PriorBox
+from src.priors import anchors
+from src.utils import Timer
+from src.utils import image_forward
+from src.utils import nms_process
+
+parser = argparse.ArgumentParser()
+parser.add_argument('--device_id', help="DEVICE_ID", type=int, default=0)
+parser.add_argument("--train_url", type=str, default='./checkpoints/', help="Storage path of training results.")
+parser.add_argument("--checkpoint_path", type=str, default=None, help="Path to checkpoint to evaluate")
+parser.add_argument("--dataset_path", type=str, default=None, help="Path to dataset root folder")
+args = parser.parse_args()
+
+
+def _print_results(images_num, total_detect_time, total_nms_time):
+ print(f'Detect time per image: {total_detect_time / (images_num - 1):.3f}s')
+ print(f'Nms time per image: {total_nms_time / (images_num - 1):.3f}s')
+ print(f'Total time per image: {(total_detect_time + total_nms_time) / (images_num - 1):.3f}s')
+ print(f'FPS: {(images_num - 1) / (total_detect_time + total_nms_time):.3f} fps')
+
+
+def test_network(
+ save_folder,
+ network,
+ detector,
+ test_dataset,
+ transform,
+ priors,
+ max_per_image=300,
+ threshold=0.005,
+):
+ images_number = len(test_dataset)
+ print(f'=> Total {images_number} images to test.')
+
+ if not os.path.exists(save_folder):
+ os.mkdir(save_folder)
+
+ classes_number = cfg.model['m2det_config']['num_classes']
+ all_boxes = [[[] for _ in range(images_number)] for _ in range(classes_number)]
+
+ in_detect_timer = Timer()
+ misc_timer = Timer()
+ det_file = os.path.join(save_folder, 'detections.pkl')
+ tot_detect_time, tot_nms_time = 0, 0
+ print('Begin evaluating')
+ print(images_number)
+
+ for image_index in tqdm(range(images_number)):
+ image = test_dataset.pull_image(image_index)
+ # 1: detection
+ in_detect_timer.tic()
+ boxes, scores = image_forward(image, network, priors, detector, transform)
+ detect_time = in_detect_timer.toc()
+ # 2: Post-processing
+ misc_timer.tic()
+ nms_process(classes_number, image_index, scores, boxes, cfg, threshold, all_boxes, max_per_image)
+ nms_time = misc_timer.toc()
+
+ tot_detect_time += detect_time if image_index > 0 else 0
+ tot_nms_time += nms_time if image_index > 0 else 0
+
+ with open(det_file, 'wb') as file:
+ pickle.dump(all_boxes, file, pickle.HIGHEST_PROTOCOL)
+
+ print('===> Evaluating detections')
+ test_dataset.evaluate_detections(all_boxes, save_folder)
+ print('Done')
+ _print_results(images_number, tot_detect_time, tot_nms_time)
+
+
+def main():
+ local_train_url = args.train_url
+
+ if args.checkpoint_path:
+ last_checkpoint = args.checkpoint_path
+ else:
+ model_name = cfg.model['m2det_config']['backbone'] + '_' + str(cfg.model['input_size'])
+ last_checkpoint = os.path.join(local_train_url, cfg.experiment_tag, f"{model_name}-final.ckpt")
+
+ if args.dataset_path:
+ cfg.COCOroot = args.dataset_path
+
+ save_dir = os.path.join(local_train_url, cfg.experiment_tag)
+
+ context.set_context(mode=context.GRAPH_MODE, device_target=cfg.device, device_id=args.device_id)
+ cfg.model['m2det_config']['checkpoint_path'] = None
+ net = get_model(cfg.model['m2det_config'], cfg.model['input_size'], test=True)
+ print(f'Loading checkpoint from {last_checkpoint}')
+ load_checkpoint(last_checkpoint, net=net)
+ net.set_train(False)
+
+ priorbox = PriorBox(cfg)
+ priors = priorbox.forward()
+ _, generator = get_dataset(
+ cfg,
+ 'COCO',
+ priors.asnumpy(),
+ 'eval_sets',
+ )
+
+ detector = Detect(cfg.model['m2det_config']['num_classes'], anchors(cfg))
+ _preprocess = BaseTransform(cfg.model['input_size'], cfg.model['rgb_means'], (2, 0, 1))
+ test_network(
+ save_dir,
+ net,
+ detector,
+ generator,
+ transform=_preprocess,
+ priors=priors,
+ max_per_image=cfg.test_cfg['topk'],
+ threshold=cfg.test_cfg['score_threshold'],
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/research/cv/m2det/export.py b/research/cv/m2det/export.py
new file mode 100644
index 0000000000000000000000000000000000000000..3c322275c9b912f9f54a7b662cbf99e34e34a475
--- /dev/null
+++ b/research/cv/m2det/export.py
@@ -0,0 +1,71 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""Export to MINDIR."""
+import argparse
+from pathlib import Path
+
+import numpy as np
+from mindspore import Tensor
+from mindspore import context
+from mindspore import dtype as mstype
+from mindspore import load_checkpoint
+from mindspore.train.serialization import export
+
+from src import config as default_config
+from src.model import get_model
+
+
+def run_export(config, ckpt_url):
+ """
+ Export model to MINDIR.
+
+ Args:
+ config (any): Config parameters.
+ ckpt_url (str): Path to the trained model checkpoint.
+ """
+ # Init the model
+ model = get_model(config.model['m2det_config'], config.model['input_size'], test=True)
+ load_checkpoint(ckpt_url, model)
+ model.set_train(False)
+
+ # Correctly process input
+ model_input = np.ones([3, config.model['input_size'], config.model['input_size']]) # CxWxH
+ model_input = Tensor(np.expand_dims(model_input, 0), mstype.float32)
+
+ name = Path(ckpt_url).stem
+ path = Path(ckpt_url).resolve().parent
+ save_path = str(Path(path, name))
+
+ export(model, model_input, file_name=save_path, file_format='MINDIR')
+ print('Model exported successfully!')
+ print(f'Path to exported model {save_path}.mindir')
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser(description="Export model to MINDIR.")
+ parser.add_argument('--device_id', help="device_id", type=int, default=0)
+ parser.add_argument("--ckpt_url", type=str, default='checkpoints/model.ckpt', help="Trained model ckpt.")
+ args = parser.parse_args()
+
+ if not Path(args.ckpt_url).is_file():
+ raise FileNotFoundError(f"Can not find checkpoint by --ckpt_url path: {args.ckpt_url}")
+
+ context.set_context(
+ mode=context.GRAPH_MODE,
+ device_target=default_config.device,
+ device_id=args.device_id,
+ )
+
+ run_export(default_config, args.ckpt_url)
diff --git a/research/cv/m2det/requirements.txt b/research/cv/m2det/requirements.txt
new file mode 100644
index 0000000000000000000000000000000000000000..4a3d4fdea4f4b677850678e993914a0cba162076
--- /dev/null
+++ b/research/cv/m2det/requirements.txt
@@ -0,0 +1,4 @@
+tqdm
+addict
+pycocotools
+opencv-python-headless
diff --git a/research/cv/m2det/scripts/convert_vgg.sh b/research/cv/m2det/scripts/convert_vgg.sh
new file mode 100644
index 0000000000000000000000000000000000000000..d9f9b4b49d999cdbb0c1d65b680394e4bef7b0e6
--- /dev/null
+++ b/research/cv/m2det/scripts/convert_vgg.sh
@@ -0,0 +1,22 @@
+#!/bin/bash
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+if [[ $# -ne 1 ]]; then
+ echo "Usage: bash ./scripts/convert_vgg.sh [BACKBONE_PATH]"
+ exit 1
+fi
+
+python convert.py --backbone_path "$1"
diff --git a/research/cv/m2det/scripts/make.sh b/research/cv/m2det/scripts/make.sh
new file mode 100644
index 0000000000000000000000000000000000000000..96f07265f17a8e0f8aaf6279f3b847345723ea4b
--- /dev/null
+++ b/research/cv/m2det/scripts/make.sh
@@ -0,0 +1,21 @@
+#!/bin/bash
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+cd ./src/
+
+python build.py build_ext --inplace
+
+cd ..
\ No newline at end of file
diff --git a/research/cv/m2det/scripts/run_distributed_train.sh b/research/cv/m2det/scripts/run_distributed_train.sh
new file mode 100644
index 0000000000000000000000000000000000000000..5fd23730d14c9f1944a795f6ce0ef6d6149fae45
--- /dev/null
+++ b/research/cv/m2det/scripts/run_distributed_train.sh
@@ -0,0 +1,38 @@
+#!/bin/bash
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+
+if [[ $# -ne 4 ]]; then
+ echo "Usage: bash ./scripts/run_distributed_train_gpu.sh [RANK_SIZE] [DEVICE_START] [PRETRAINED_BACKBONE]" \
+ "[DATASET_PATH]"
+exit 1
+fi
+
+device_start=$2
+rank_size=$1
+device_end=$(($device_start+$rank_size-1))
+gpus=""
+for i in $(eval echo {$device_start..$device_end})
+ do
+ gpus="$gpus,$i"
+done
+gpus="${gpus:1}"
+export CUDA_VISIBLE_DEVICES=$gpus
+rm -rf logs
+mkdir ./logs
+nohup mpirun -n $rank_size --allow-run-as-root python train.py --run_distribute True \
+--pretrained_backbone $3 --dataset_path $4 > ./logs/train.log 2>&1 &
+echo $! > ./logs/train.pid
diff --git a/research/cv/m2det/scripts/run_eval.sh b/research/cv/m2det/scripts/run_eval.sh
new file mode 100644
index 0000000000000000000000000000000000000000..436aaed922f9ad5c5c641814eb131f7b44e60984
--- /dev/null
+++ b/research/cv/m2det/scripts/run_eval.sh
@@ -0,0 +1,28 @@
+#!/bin/bash
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+
+if [[ $# -ne 3 ]]; then
+ echo "Usage: bash ./scripts/run_eval.sh [DEVICE_ID] [CHECKPOINT_PATH] [DATASET_PATH]"
+exit 1
+fi
+
+if [ ! -d "logs" ]; then
+ mkdir logs
+fi
+
+nohup python -u eval.py --device_id=$1 --checkpoint_path $2 --dataset_path $3> ./logs/eval.log 2>&1 &
+echo "Evaluation started on device $1 ! PID: $!"
diff --git a/research/cv/m2det/scripts/run_standalone_train.sh b/research/cv/m2det/scripts/run_standalone_train.sh
new file mode 100644
index 0000000000000000000000000000000000000000..5cdc5209023c856214fc96845b799a6538055a3d
--- /dev/null
+++ b/research/cv/m2det/scripts/run_standalone_train.sh
@@ -0,0 +1,29 @@
+#!/bin/bash
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+
+if [[ $# -ne 3 ]]; then
+ echo "Usage: bash ./scripts/run_standalone_train.sh [DEVICE_ID] [PRETRAINED_BACKBONE] [DATASET_PATH]"
+exit 1
+fi
+
+if [ ! -d "logs" ]; then
+ mkdir logs
+fi
+
+nohup python -u train.py --device_id=$1 --pretrained_backbone $2 --dataset_path $3 > ./logs/train.log 2>&1 &
+echo "Training started on device $1 ! PID: $!"
+echo $! > ./logs/train.pid
diff --git a/research/cv/m2det/src/box_utils.py b/research/cv/m2det/src/box_utils.py
new file mode 100644
index 0000000000000000000000000000000000000000..85e074bde71975a730040f880dc38b895e987beb
--- /dev/null
+++ b/research/cv/m2det/src/box_utils.py
@@ -0,0 +1,208 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+import numpy as np
+from mindspore import ops
+
+
+def point_form(boxes):
+ """ Convert prior_boxes to (xmin, ymin, xmax, ymax)
+ representation for comparison to point form ground truth data.
+ Args:
+ boxes: (tensor) center-size default boxes from priorbox layers.
+ Return:
+ boxes: (tensor) Converted xmin, ymin, xmax, ymax form of boxes.
+ """
+ return np.concatenate((boxes[:, :2] - boxes[:, 2:] / 2, # xmin, ymin
+ boxes[:, :2] + boxes[:, 2:] / 2), 1) # xmax, ymax
+
+
+def center_size(boxes):
+ """ Convert prior_boxes to (cx, cy, w, h)
+ representation for comparison to center-size form ground truth data.
+ Args:
+ boxes: (tensor) point_form boxes
+ Return:
+ boxes: (tensor) Converted xmin, ymin, xmax, ymax form of boxes.
+ """
+ return np.concatenate((boxes[:, 2:] + boxes[:, :2]) / 2, # cx, cy
+ boxes[:, 2:] - boxes[:, :2], 1) # w, h
+
+
+def intersect(box_a, box_b):
+ """ We resize both tensors to [A,B,2] without new malloc:
+ [A,2] -> [A,1,2] -> [A,B,2]
+ [B,2] -> [1,B,2] -> [A,B,2]
+ Then we compute the area of intersect between box_a and box_b.
+ Args:
+ box_a: (tensor) bounding boxes, Shape: [A,4].
+ box_b: (tensor) bounding boxes, Shape: [B,4].
+ Return:
+ (tensor) intersection area, Shape: [A,B].
+ """
+ A = box_a.shape[0]
+ B = box_b.shape[0]
+ max_xy = np.minimum(np.broadcast_to(np.expand_dims(box_a[:, 2:], 1), (A, B, 2)),
+ np.broadcast_to(np.expand_dims(box_b[:, 2:], 0), (A, B, 2)))
+
+ min_xy = np.maximum(np.broadcast_to(np.expand_dims(box_a[:, :2], 1), (A, B, 2)),
+ np.broadcast_to(np.expand_dims(box_b[:, :2], 0), (A, B, 2)))
+
+ diff = max_xy - min_xy
+ inter = np.clip(diff, 0, None)
+ return inter[:, :, 0] * inter[:, :, 1]
+
+
+def jaccard(box_a, box_b):
+ """Compute the jaccard overlap of two sets of boxes. The jaccard overlap
+ is simply the intersection over union of two boxes. Here we operate on
+ ground truth boxes and default boxes.
+ E.g.:
+ A ∩ B / A ∪ B = A ∩ B / (area(A) + area(B) - A ∩ B)
+ Args:
+ box_a: (tensor) Ground truth bounding boxes, Shape: [num_objects,4]
+ box_b: (tensor) Prior boxes from priorbox layers, Shape: [num_priors,4]
+ Return:
+ jaccard overlap: (tensor) Shape: [box_a.size(0), box_b.size(0)]
+ """
+ inter = intersect(box_a, box_b)
+ area_a = np.expand_dims(((box_a[:, 2]-box_a[:, 0]) *
+ (box_a[:, 3]-box_a[:, 1])), 1) # [A,B]
+ area_b = np.expand_dims(((box_b[:, 2]-box_b[:, 0]) *
+ (box_b[:, 3]-box_b[:, 1])), 0) # [A,B]
+ area_a = np.broadcast_to(area_a, inter.shape)
+ area_b = np.broadcast_to(area_b, inter.shape)
+ union = area_a + area_b - inter
+ return inter / union # [A,B]
+
+
+def matrix_iou(a, b):
+ """
+ return iou of a and b, numpy version for data augenmentation
+ """
+ lt = np.maximum(a[:, np.newaxis, :2], b[:, :2])
+ rb = np.minimum(a[:, np.newaxis, 2:], b[:, 2:])
+
+ area_i = np.prod(rb - lt, axis=2) * (lt < rb).all(axis=2)
+ area_a = np.prod(a[:, 2:] - a[:, :2], axis=1)
+ area_b = np.prod(b[:, 2:] - b[:, :2], axis=1)
+ return area_i / (area_a[:, np.newaxis] + area_b - area_i)
+
+
+def match(threshold, truths, priors, variances, labels):
+ """Match each prior box with the ground truth box of the highest jaccard
+ overlap, encode the bounding boxes, then return the matched indices
+ corresponding to both confidence and location preds.
+ Args:
+ threshold: (float) The overlap threshold used when matching boxes.
+ truths: (tensor) Ground truth boxes, Shape: [num_obj, num_priors].
+ priors: (tensor) Prior boxes from priorbox layers, Shape: [n_priors,4].
+ variances: (tensor) Variances corresponding to each prior coord,
+ Shape: [num_priors, 4].
+ labels: (tensor) All the class labels for the image, Shape: [num_obj].
+ loc_t: (tensor) Tensor to be filled w/ encoded location targets.
+ conf_t: (tensor) Tensor to be filled w/ matched indices for conf preds.
+ idx: (int) current batch index
+ Return:
+ The matched indices corresponding to 1)location and 2)confidence preds.
+ """
+ # jaccard index
+ overlaps = jaccard(
+ truths,
+ point_form(priors)
+ )
+ # (Bipartite Matching)
+ # [1,num_objects] best prior for each ground truth
+ best_prior_idx = np.expand_dims(np.argmax(overlaps, axis=1), 1)
+ best_prior_overlap = np.expand_dims(np.max(overlaps, axis=1), 1)
+
+ # [1,num_priors] best ground truth for each prior
+ best_truth_idx = np.expand_dims(np.argmax(overlaps, axis=0), 0)
+ best_truth_overlap = np.expand_dims(np.max(overlaps, axis=0), 0)
+
+ if best_truth_idx.shape[0] == 1:
+ best_truth_idx = best_truth_idx.squeeze(0)
+ if best_truth_overlap.shape[0] == 1:
+ best_truth_overlap = best_truth_overlap.squeeze(0)
+ if best_prior_idx.shape[1] == 1:
+ best_prior_idx = best_prior_idx.squeeze(1)
+ if best_prior_overlap.shape[1] == 1:
+ best_prior_overlap = best_prior_overlap.squeeze(1)
+
+ best_truth_overlap[best_prior_idx] = np.broadcast_to(np.array(2.), best_prior_idx.size)
+
+ for j in range(best_prior_idx.shape[0]):
+ best_truth_idx[best_prior_idx[j]] = j
+ matches = truths[best_truth_idx] # Shape: [num_priors,4]
+ conf = labels.astype(int)[best_truth_idx] # Shape: [num_priors]
+ conf[best_truth_overlap < threshold] = 0 # label as background
+ loc = encode(matches, priors, variances)
+ return loc, conf
+
+
+def encode(matched, priors, variances):
+ """Encode the variances from the priorbox layers into the ground truth boxes
+ we have matched (based on jaccard overlap) with the prior boxes.
+ Args:
+ matched: (tensor) Coords of ground truth for each prior in point-form
+ Shape: [num_priors, 4].
+ priors: (tensor) Prior boxes in center-offset form
+ Shape: [num_priors,4].
+ variances: (list[float]) Variances of priorboxes
+ Return:
+ encoded boxes (tensor), Shape: [num_priors, 4]
+ """
+
+ # dist b/t match center and prior's center
+ g_cxcy = (matched[:, :2] + matched[:, 2:]) / 2 - priors[:, :2]
+ # encode variance
+ g_cxcy /= (variances[0] * priors[:, 2:])
+ # match wh / prior wh
+ g_wh = (matched[:, 2:] - matched[:, :2]) / priors[:, 2:]
+ g_wh = np.log(g_wh + 1e-10) / variances[1]
+ # return target for smooth_l1_loss
+ return np.concatenate([g_cxcy, g_wh], 1) # [num_priors,4]
+
+
+def decode(loc, priors, variances):
+ """Decode locations from predictions using priors to undo
+ the encoding we did for offset regression at train time.
+ Args:
+ loc (tensor): location predictions for loc layers,
+ Shape: [num_priors,4]
+ priors (tensor): Prior boxes in center-offset form.
+ Shape: [num_priors,4].
+ variances: (list[float]) Variances of priorboxes
+ Return:
+ decoded bounding box predictions
+ """
+
+ boxes = ops.Concat(axis=1)((
+ priors[:, :2] + loc[:, :2] * variances[0] * priors[:, 2:],
+ priors[:, 2:] * ops.Exp()(loc[:, 2:] * variances[1])))
+ boxes[:, :2] -= boxes[:, 2:] / 2
+ boxes[:, 2:] += boxes[:, :2]
+ return boxes
+
+
+def log_sum_exp(x):
+ """Utility function for computing log_sum_exp while determining
+ This will be used to determine unaveraged confidence loss across
+ all examples in a batch.
+ Args:
+ x (Variable(tensor)): conf_preds from conf layers
+ """
+ x_max = x.max()
+ return ops.Log()(ops.ReduceSum(keep_dims=True)(ops.Exp()(x-x_max), 1)) + x_max
diff --git a/research/cv/m2det/src/build.py b/research/cv/m2det/src/build.py
new file mode 100644
index 0000000000000000000000000000000000000000..e2faa88e47cfb5214d5ee2206b2d38ab0c11cc76
--- /dev/null
+++ b/research/cv/m2det/src/build.py
@@ -0,0 +1,34 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+from distutils.core import setup
+from distutils.extension import Extension
+
+import numpy as np
+from Cython.Distutils import build_ext
+
+_EXTENSION_NAME = 'nms.cpu_nms'
+_SOURCES = ["nms/cpu_nms.pyx"]
+_EXTA_ARGS = ["-Wno-cpp", "-Wno-unused-function"]
+
+# Obtain the numpy include directory. This logic works across numpy versions.
+try:
+ numpy_include = np.get_include()
+except AttributeError:
+ numpy_include = np.get_numpy_include()
+
+modules = [Extension(_EXTENSION_NAME, _SOURCES, extra_compile_args=_EXTA_ARGS, include_dirs=[numpy_include])]
+
+setup(name='mot_utils', ext_modules=modules, cmdclass={'build_ext': build_ext})
diff --git a/research/cv/m2det/src/callback.py b/research/cv/m2det/src/callback.py
new file mode 100644
index 0000000000000000000000000000000000000000..0d32681a431ef63f30c9cee615eef9b23405a1bc
--- /dev/null
+++ b/research/cv/m2det/src/callback.py
@@ -0,0 +1,90 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""Training callbacks."""
+
+import time
+import numpy as np
+
+from mindspore.train.callback import Callback
+from mindspore import Tensor
+
+
+class TimeLossMonitor(Callback):
+ """
+ Monitor loss and time.
+
+ Args:
+ lr_init (numpy array): train lr
+
+ Returns:
+ None
+
+ Examples:
+ >>> TimeLossMonitor(100,lr_init=Tensor([0.05]*100).asnumpy())
+ """
+
+ def __init__(self, lr_init=None):
+ super().__init__()
+ self.lr_init = lr_init
+ self.losses = []
+ self.epoch_time = 0
+ self.step_time = 0
+ self.steps_made = 0
+
+ def begin(self, run_context):
+ print('Training start')
+
+ def epoch_begin(self, run_context):
+ """Epoch begin."""
+ self.losses = []
+ self.epoch_time = time.time()
+
+ def epoch_end(self, run_context):
+ """Epoch end."""
+ print('start epoch evaluation')
+ cb_params = run_context.original_args()
+
+ cur_epoch = cb_params.cur_epoch_num
+ tot_epoch = cb_params.epoch_num
+ epoch_seconds = (time.time() - self.epoch_time)
+ batch_num = cb_params.batch_num
+ per_step_mseconds = epoch_seconds / cb_params.batch_num * 1000
+ mean_loss = np.mean(self.losses)
+ cur_lr = self.lr_init[cb_params.cur_step_num - 1]
+ print(f"epoch: [{cur_epoch:3d}/{tot_epoch:3d}], epoch time: {epoch_seconds:5.1f} s, "
+ f"steps: {batch_num:5d}, per step time: {per_step_mseconds:5.3f} ms, "
+ f"avg loss: {mean_loss:.5f}, lr: {cur_lr:8.6f}",
+ flush=True)
+
+ def step_begin(self, run_context):
+ """Step begin."""
+ self.step_time = time.time()
+ self.steps_made += 1
+
+ def step_end(self, run_context):
+ """step end"""
+ cb_params = run_context.original_args()
+ step_loss = cb_params.net_outputs
+
+ if isinstance(step_loss, (tuple, list)) and isinstance(step_loss[0], Tensor):
+ step_loss = step_loss[0]
+ if isinstance(step_loss, Tensor):
+ step_loss = np.mean(step_loss.asnumpy())
+
+ # Step time measurement, works only in dataset_sink_mode=False. Uncomment for debugging
+ step_time = (time.time() - self.step_time) * 1000
+ print(f'Step: {self.steps_made}, Loss: {step_loss}, step_time: {step_time} ms')
+
+ self.losses.append(step_loss)
diff --git a/research/cv/m2det/src/coco_utils.py b/research/cv/m2det/src/coco_utils.py
new file mode 100644
index 0000000000000000000000000000000000000000..b157cd9a92d387beaf57175be095517b9318810e
--- /dev/null
+++ b/research/cv/m2det/src/coco_utils.py
@@ -0,0 +1,256 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+import os
+import pickle
+import json
+import cv2
+import numpy as np
+from pycocotools.coco import COCO
+from pycocotools.cocoeval import COCOeval
+
+
+class COCODetection:
+
+ def __init__(self, root, image_sets, preproc=None, target_transform=None,
+ dataset_name='COCO'):
+ self.root = root
+ self.cache_path = os.path.join(self.root, 'coco_cache')
+ os.makedirs(self.cache_path, exist_ok=True)
+ self.image_set = image_sets
+ self.preproc = preproc
+ self.target_transform = target_transform
+ self.name = dataset_name
+ self.ids = list()
+ self.annotations = list()
+ self._view_map = {
+ 'minival2014': 'val2014', # 5k val2014 subset
+ 'valminusminival2014': 'val2014', # val2014 \setminus minival2014
+ 'test-dev2015': 'test2015',
+ }
+
+ for (year, image_set) in image_sets:
+ coco_name = image_set+year
+ data_name = (self._view_map[coco_name]
+ if coco_name in self._view_map
+ else coco_name)
+ annofile = self._get_ann_file(coco_name)
+ _COCO = COCO(annofile)
+ self._COCO = _COCO
+ self.coco_name = coco_name
+ cats = _COCO.loadCats(_COCO.getCatIds())
+ self._classes = tuple(['__background__'] + [c['name'] for c in cats])
+ self.num_classes = len(self._classes)
+ self._class_to_ind = dict(zip(self._classes, range(self.num_classes)))
+ self._class_to_coco_cat_id = dict(zip([c['name'] for c in cats],
+ _COCO.getCatIds()))
+ indexes = _COCO.getImgIds()
+ self.image_indexes = indexes
+ self.ids.extend([self.image_path_from_index(data_name, index) for index in indexes])
+ if image_set.find('test') != -1:
+ print('test set will not load annotations!')
+ else:
+ self.annotations.extend(self._load_coco_annotations(coco_name, indexes, _COCO))
+
+ def image_path_from_index(self, name, index):
+ """
+ Construct an image path from the image's "index" identifier.
+ """
+ # Example image path for index=119993:
+ # images/train2014/COCO_train2014_000000119993.jpg
+ file_name = ('COCO_' + name + '_' +
+ str(index).zfill(12) + '.jpg')
+ image_path = os.path.join(self.root, 'images',
+ name, file_name)
+ assert os.path.exists(image_path), 'Path does not exist: {}'.format(image_path)
+ return image_path
+
+ def _get_ann_file(self, name):
+ prefix = 'instances' if name.find('test') == -1 \
+ else 'image_info'
+ return os.path.join(self.root, 'annotations',
+ prefix + '_' + name + '.json')
+
+ def _load_coco_annotations(self, coco_name, indexes, _COCO):
+ cache_file = os.path.join(self.cache_path, coco_name + '_gt_roidb.pkl')
+ if os.path.exists(cache_file):
+ with open(cache_file, 'rb') as fid:
+ roidb = pickle.load(fid)
+ print('{} gt roidb loaded from {}'.format(coco_name, cache_file))
+ return roidb
+
+ gt_roidb = [self._annotation_from_index(index, _COCO)
+ for index in indexes]
+ with open(cache_file, 'wb') as fid:
+ pickle.dump(gt_roidb, fid, pickle.HIGHEST_PROTOCOL)
+ print('wrote gt roidb to {}'.format(cache_file))
+ return gt_roidb
+
+ def _annotation_from_index(self, index, _COCO):
+ """
+ Loads COCO bounding-box instance annotations. Crowd instances are
+ handled by marking their overlaps (with all categories) to -1. This
+ overlap value means that crowd "instances" are excluded from training.
+ """
+ im_ann = _COCO.loadImgs(index)[0]
+ width = im_ann['width']
+ height = im_ann['height']
+
+ annIds = _COCO.getAnnIds(imgIds=index, iscrowd=None)
+ objs = _COCO.loadAnns(annIds)
+ # Sanitize bboxes -- some are invalid
+ valid_objs = []
+ for obj in objs:
+ x1 = np.max((0, obj['bbox'][0]))
+ y1 = np.max((0, obj['bbox'][1]))
+ x2 = np.min((width - 1, x1 + np.max((0, obj['bbox'][2] - 1))))
+ y2 = np.min((height - 1, y1 + np.max((0, obj['bbox'][3] - 1))))
+ if obj['area'] > 0 and x2 >= x1 and y2 >= y1:
+ obj['clean_bbox'] = [x1, y1, x2, y2]
+ valid_objs.append(obj)
+ objs = valid_objs
+ num_objs = len(objs)
+
+ res = np.zeros((num_objs, 5))
+
+ # Lookup table to map from COCO category ids to our internal class
+ # indices
+ coco_cat_id_to_class_ind = {self._class_to_coco_cat_id[cls]: self._class_to_ind[cls]
+ for cls in self._classes[1:]}
+
+ for ix, obj in enumerate(objs):
+ cls = coco_cat_id_to_class_ind[obj['category_id']]
+ res[ix, 0:4] = obj['clean_bbox']
+ res[ix, 4] = cls
+
+ return res
+
+ def __getitem__(self, index):
+ img_id = self.ids[index]
+ target = self.annotations[index]
+ img = cv2.imread(img_id, cv2.IMREAD_COLOR)
+
+ if self.target_transform is not None:
+ target = self.target_transform(target)
+
+ if self.preproc is not None:
+ img, target = self.preproc(img, target)
+
+ return img.astype(np.float32), target.astype(np.float32)
+
+ def __len__(self):
+ return len(self.ids)
+
+ def pull_image(self, index):
+ """Returns the original image object at index in PIL form
+
+ Note: not using self.__getitem__(), as any transformations passed in
+ could mess up this functionality.
+
+ Argument:
+ index (int): index of img to show
+ Return:
+ PIL img
+ """
+ img_id = self.ids[index]
+ return cv2.imread(img_id, cv2.IMREAD_COLOR)
+
+ def _print_detection_eval_metrics(self, coco_eval):
+ IoU_lo_thresh = 0.5
+ IoU_hi_thresh = 0.95
+
+ def _get_thr_ind(coco_eval_, thr):
+ ind = np.where((coco_eval_.params.iouThrs > thr - 1e-5) &
+ (coco_eval_.params.iouThrs < thr + 1e-5))[0][0]
+ iou_thr = coco_eval_.params.iouThrs[ind]
+ assert np.isclose(iou_thr, thr)
+ return ind
+
+ ind_lo = _get_thr_ind(coco_eval, IoU_lo_thresh)
+ ind_hi = _get_thr_ind(coco_eval, IoU_hi_thresh)
+ # precision has dims (iou, recall, cls, area range, max dets)
+ # area range index 0: all area ranges
+ # max dets index 2: 100 per image
+ precision = \
+ coco_eval.eval['precision'][ind_lo:(ind_hi + 1), :, :, 0, 2]
+ ap_default = np.mean(precision[precision > -1])
+ print('~~~~ Mean and per-category AP @ IoU=[{:.2f},{:.2f}] '
+ '~~~~'.format(IoU_lo_thresh, IoU_hi_thresh))
+ print('{:.1f}'.format(100 * ap_default))
+ aps = list()
+ for cls_ind, cls in enumerate(self._classes):
+ if cls == '__background__':
+ continue
+ # minus 1 because of __background__
+ precision = coco_eval.eval['precision'][ind_lo:(ind_hi + 1), :, cls_ind - 1, 0, 2]
+ ap = np.mean(precision[precision > -1])
+ aps.append(100 * ap)
+ print('~~~~ Summary metrics ~~~~')
+ coco_eval.summarize()
+
+ def _do_detection_eval(self, res_file, output_dir):
+ ann_type = 'bbox'
+ coco_dt = self._COCO.loadRes(res_file)
+ coco_eval = COCOeval(self._COCO, coco_dt)
+ coco_eval.params.useSegm = (ann_type == 'segm')
+ coco_eval.evaluate()
+ coco_eval.accumulate()
+ self._print_detection_eval_metrics(coco_eval)
+ eval_file = os.path.join(output_dir, 'detection_results.pkl')
+ with open(eval_file, 'wb') as fid:
+ pickle.dump(coco_eval, fid, pickle.HIGHEST_PROTOCOL)
+ print('Wrote COCO eval results to: {}'.format(eval_file))
+
+ def _coco_results_one_category(self, boxes, cat_id):
+ results = []
+ for im_ind, index in enumerate(self.image_indexes):
+ dets = boxes[im_ind].astype(np.float)
+ if dets == []:
+ continue
+ scores = dets[:, -1]
+ xs = dets[:, 0]
+ ys = dets[:, 1]
+ ws = dets[:, 2] - xs + 1
+ hs = dets[:, 3] - ys + 1
+ results.extend(
+ [{'image_id': index,
+ 'category_id': cat_id,
+ 'bbox': [xs[k], ys[k], ws[k], hs[k]],
+ 'score': scores[k]} for k in range(dets.shape[0])])
+ return results
+
+ def _write_coco_results_file(self, all_boxes, res_file):
+ results = []
+ print('Collecting Results......')
+ for cls_ind, cls in enumerate(self._classes):
+ if cls == '__background__':
+ continue
+ coco_cat_id = self._class_to_coco_cat_id[cls]
+ results.extend(self._coco_results_one_category(all_boxes[cls_ind],
+ coco_cat_id))
+
+ print('Writing results json to {}'.format(res_file))
+ with open(res_file, 'w') as fid:
+ json.dump(results, fid)
+
+ def evaluate_detections(self, all_boxes, output_dir):
+ res_file = os.path.join(output_dir, ('detections_' +
+ self.coco_name +
+ '_results'))
+ res_file += '.json'
+ self._write_coco_results_file(all_boxes, res_file)
+
+ if self.coco_name.find('test') == -1:
+ self._do_detection_eval(res_file, output_dir)
diff --git a/research/cv/m2det/src/config.py b/research/cv/m2det/src/config.py
new file mode 100644
index 0000000000000000000000000000000000000000..9d91f8cd6c1afde1b2922aafb5a70d034dd5a93d
--- /dev/null
+++ b/research/cv/m2det/src/config.py
@@ -0,0 +1,102 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+device = 'GPU'
+random_seed = 1
+experiment_tag = 'm2det512_vgg16'
+checkpoint_name = None
+start_epoch = 0
+
+if checkpoint_name:
+ checkpoint_path = '/workdir/m2det-mindspore/checkpoints/' + experiment_tag + '/' + checkpoint_name
+else:
+ checkpoint_path = None
+
+
+model = dict(
+ type='m2det',
+ input_size=512,
+ init_net=True,
+ m2det_config=dict(
+ backbone='vgg16',
+ net_family='vgg',
+ base_out=[22, 34], # [22,34] for vgg, [2,4] or [3,4] for res families
+ planes=256,
+ num_levels=8,
+ num_scales=6,
+ sfam=False,
+ smooth=True,
+ num_classes=81,
+ checkpoint_path='/workdir/m2det-mindspore/checkpoints/vgg16_reducedfc.ckpt'
+ ),
+ rgb_means=(104, 117, 123),
+ p=0.6,
+ anchor_config=dict(
+ step_pattern=[8, 16, 32, 64, 128, 256],
+ size_pattern=[0.06, 0.15, 0.33, 0.51, 0.69, 0.87, 1.05],
+ ),
+ checkpoint_interval=10,
+ weights_save='weights/'
+)
+
+train_cfg = dict(
+ lr=1e-3,
+ warmup=5,
+ per_batch_size=7,
+ gamma=[0.5, 0.2, 0.1, 0.1],
+ lr_epochs=[90, 110, 130, 150, 160],
+ total_epochs=160,
+ print_epochs=10,
+ num_workers=3,
+ )
+
+test_cfg = {
+ 'cuda': True,
+ 'topk': 0,
+ 'iou': 0.45,
+ 'soft_nms': True,
+ 'score_threshold': 0.1,
+ 'keep_per_class': 50,
+ 'save_folder': 'eval',
+}
+
+loss = {
+ 'overlap_thresh': 0.5,
+ 'prior_for_matching': True,
+ 'bkg_label': 0,
+ 'neg_mining': True,
+ 'neg_pos': 3,
+ 'neg_overlap': 0.5,
+ 'encode_target': False,
+}
+
+optimizer = {
+ 'type': 'SGD',
+ 'momentum': 0.9,
+ 'weight_decay': 0.00005,
+ 'dampening': 0.0,
+ 'clip_grad_norm': 4.,
+}
+
+dataset = {
+ 'COCO': {
+ 'train_sets': [('2014', 'train'), ('2014', 'valminusminival')],
+ 'eval_sets': [('2014', 'minival')],
+ 'test_sets': [('2015', 'test-dev')],
+ }
+}
+
+# Dataset root folder. Used then no dataset_path argument specified for training script
+COCOroot = '/workdir/datasets/coco2014/'
diff --git a/research/cv/m2det/src/dataset.py b/research/cv/m2det/src/dataset.py
new file mode 100644
index 0000000000000000000000000000000000000000..19f7b24a48834e94704a40ff24f75a331fe686d0
--- /dev/null
+++ b/research/cv/m2det/src/dataset.py
@@ -0,0 +1,304 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+import math
+import random
+
+import cv2
+import numpy as np
+from mindspore import Tensor
+from mindspore import dataset as de
+from mindspore import dtype as mstype
+from mindspore import set_seed
+from mindspore.communication import get_group_size
+from mindspore.communication import get_rank
+
+from src.box_utils import match
+from src.box_utils import matrix_iou
+from src.coco_utils import COCODetection
+
+
+def _crop(image, boxes, labels):
+ height, width, _ = image.shape
+
+ if boxes.size == 0:
+ return image, boxes, labels
+
+ while True:
+ mode = random.choice((
+ None,
+ (0.1, None),
+ (0.3, None),
+ (0.5, None),
+ (0.7, None),
+ (0.9, None),
+ (None, None),
+ ))
+
+ if mode is None:
+ return image, boxes, labels
+
+ min_iou, max_iou = mode
+ if min_iou is None:
+ min_iou = float('-inf')
+ if max_iou is None:
+ max_iou = float('inf')
+
+ for _ in range(50):
+ scale = random.uniform(0.3, 1.)
+ min_ratio = max(0.5, scale * scale)
+ max_ratio = min(2, 1. / scale / scale)
+ ratio = math.sqrt(random.uniform(min_ratio, max_ratio))
+ w = int(scale * ratio * width)
+ h = int((scale / ratio) * height)
+
+ left = random.randrange(width - w)
+ top = random.randrange(height - h)
+ roi = np.array((left, top, left + w, top + h))
+
+ iou = matrix_iou(boxes, roi[np.newaxis])
+
+ if not (min_iou <= iou.min() and iou.max() <= max_iou):
+ continue
+
+ image_t = image[roi[1]:roi[3], roi[0]:roi[2]]
+
+ centers = (boxes[:, :2] + boxes[:, 2:]) / 2
+ mask = np.logical_and(roi[:2] < centers, centers < roi[2:]) \
+ .all(axis=1)
+ boxes_t = boxes[mask].copy()
+ labels_t = labels[mask].copy()
+ if boxes_t.size == 0:
+ continue
+
+ boxes_t[:, :2] = np.maximum(boxes_t[:, :2], roi[:2])
+ boxes_t[:, :2] -= roi[:2]
+ boxes_t[:, 2:] = np.minimum(boxes_t[:, 2:], roi[2:])
+ boxes_t[:, 2:] -= roi[:2]
+
+ return image_t, boxes_t, labels_t
+
+
+def _distort(image):
+ def _convert(image_, alpha=1, beta=0):
+ tmp_ = image_.astype(float) * alpha + beta
+ tmp_[tmp_ < 0] = 0
+ tmp_[tmp_ > 255] = 255
+ image_[:] = tmp_
+
+ image = image.copy()
+
+ if random.randrange(2):
+ _convert(image, beta=random.uniform(-32, 32))
+
+ if random.randrange(2):
+ _convert(image, alpha=random.uniform(0.5, 1.5))
+
+ image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
+
+ if random.randrange(2):
+ tmp = image[:, :, 0].astype(int) + random.randint(-18, 18)
+ tmp %= 180
+ image[:, :, 0] = tmp
+
+ if random.randrange(2):
+ _convert(image[:, :, 1], alpha=random.uniform(0.5, 1.5))
+
+ image = cv2.cvtColor(image, cv2.COLOR_HSV2BGR)
+
+ return image
+
+
+def _expand(image, boxes, fill, p):
+ if random.random() > p:
+ return image, boxes
+
+ height, width, depth = image.shape
+ for _ in range(50):
+ scale = random.uniform(1, 4)
+
+ min_ratio = max(0.5, 1. / scale / scale)
+ max_ratio = min(2, scale*scale)
+ ratio = math.sqrt(random.uniform(min_ratio, max_ratio))
+ ws = scale * ratio
+ hs = scale / ratio
+ if ws < 1 or hs < 1:
+ continue
+ w = int(ws * width)
+ h = int(hs * height)
+
+ left = random.randint(0, w - width)
+ top = random.randint(0, h - height)
+
+ boxes_t = boxes.copy()
+ boxes_t[:, :2] += (left, top)
+ boxes_t[:, 2:] += (left, top)
+
+ expand_image = np.empty(
+ (h, w, depth),
+ dtype=image.dtype)
+ expand_image[:, :] = fill
+ expand_image[top:top + height, left:left + width] = image
+ image = expand_image
+
+ return image, boxes_t
+
+
+def _mirror(image, boxes):
+ _, width, _ = image.shape
+ if random.randrange(2):
+ image = image[:, ::-1]
+ boxes = boxes.copy()
+ boxes[:, 0::2] = width - boxes[:, 2::-2]
+ return image, boxes
+
+
+def preproc_for_test(image, insize, mean):
+ interp_methods = [cv2.INTER_LINEAR, cv2.INTER_CUBIC, cv2.INTER_AREA, cv2.INTER_NEAREST, cv2.INTER_LANCZOS4]
+ interp_method = interp_methods[random.randrange(5)]
+ image = cv2.resize(image, (insize, insize), interpolation=interp_method)
+ image = image.astype(np.float32)
+ image -= mean
+ return image.transpose(2, 0, 1)
+
+
+class preproc:
+
+ def __init__(self, resize, rgb_means, p):
+ self.means = rgb_means
+ self.resize = resize
+ self.p = p
+
+ def __call__(self, image, targets):
+ boxes = targets[:, :-1].copy()
+ labels = targets[:, -1].copy()
+ if boxes.size == 0:
+ targets = np.zeros((1, 5))
+ image = preproc_for_test(image, self.resize, self.means)
+ return image, targets
+
+ image_o = image.copy()
+ targets_o = targets.copy()
+ height_o, width_o, _ = image_o.shape
+ boxes_o = targets_o[:, :-1]
+ labels_o = targets_o[:, -1]
+ boxes_o[:, 0::2] /= width_o
+ boxes_o[:, 1::2] /= height_o
+ labels_o = np.expand_dims(labels_o, 1)
+ targets_o = np.hstack((boxes_o, labels_o))
+
+ image_t, boxes, labels = _crop(image, boxes, labels)
+ image_t = _distort(image_t)
+ image_t, boxes = _expand(image_t, boxes, self.means, self.p)
+ image_t, boxes = _mirror(image_t, boxes)
+
+ height, width, _ = image_t.shape
+ image_t = preproc_for_test(image_t, self.resize, self.means)
+ boxes = boxes.copy()
+ boxes[:, 0::2] /= width
+ boxes[:, 1::2] /= height
+ b_w = (boxes[:, 2] - boxes[:, 0])*1.
+ b_h = (boxes[:, 3] - boxes[:, 1])*1.
+ mask_b = np.minimum(b_w, b_h) > 0.01
+ boxes_t = boxes[mask_b]
+ labels_t = labels[mask_b].copy()
+
+ if boxes_t.size == 0:
+ image = preproc_for_test(image_o, self.resize, self.means)
+ return image, targets_o
+
+ labels_t = np.expand_dims(labels_t, 1)
+ targets_t = np.hstack((boxes_t, labels_t))
+
+ return image_t, targets_t
+
+
+class BaseTransform:
+ """Defines the transformations that should be applied to test PIL image
+ for input into the network
+
+ dimension -> tensorize -> color adj
+
+ Arguments:
+ resize (int): input dimension to SSD
+ rgb_means ((int,int,int)): average RGB of the dataset
+ (104,117,123)
+ swap ((int,int,int)): final order of channels
+ Returns:
+ transform (transform) : callable transform to be applied to test/val
+ data
+ """
+
+ def __init__(self, resize, rgb_means, swap=(2, 0, 1)):
+ self.means = rgb_means
+ self.resize = resize
+ self.swap = swap
+
+ # assume input is cv2 img for now
+ def __call__(self, img):
+
+ interp_methods = [cv2.INTER_LINEAR, cv2.INTER_CUBIC, cv2.INTER_AREA, cv2.INTER_NEAREST, cv2.INTER_LANCZOS4]
+ interp_method = interp_methods[0]
+ img = cv2.resize(np.array(img), (self.resize,
+ self.resize), interpolation=interp_method).astype(np.float32)
+ img -= self.means
+ img = img.transpose(self.swap)
+ return Tensor(img, dtype=mstype.float32)
+
+
+def target_preprocess(img, annotation, cfg, priors):
+ loc, conf = match(cfg.loss['overlap_thresh'],
+ annotation[:, :-1],
+ priors,
+ [0.1, 0.2],
+ annotation[:, -1])
+ return img, loc, conf.astype(np.int32)
+
+
+def get_dataset(cfg, dataset, priors, setname='train_sets', random_seed=None, distributed=False):
+ _preproc = preproc(cfg.model['input_size'], cfg.model['rgb_means'], cfg.model['p'])
+ Dataloader_function = {'COCO': COCODetection}
+ _Dataloader_function = Dataloader_function[dataset]
+ shuffle = False
+ if random_seed is not None:
+ set_seed(random_seed)
+ shuffle = True
+
+ if setname == 'train_sets':
+ generator = _Dataloader_function(cfg.COCOroot if dataset == 'COCO' else cfg.VOCroot,
+ cfg.dataset[dataset][setname], _preproc)
+ else:
+ generator = _Dataloader_function(cfg.COCOroot if dataset == 'COCO' else cfg.VOCroot,
+ cfg.dataset[dataset][setname], None)
+ if distributed:
+ rank_id = get_rank()
+ rank_size = get_group_size()
+ ds = de.GeneratorDataset(source=generator,
+ column_names=['img', 'annotation'],
+ num_parallel_workers=cfg.train_cfg['num_workers'],
+ shuffle=shuffle,
+ num_shards=rank_size,
+ shard_id=rank_id)
+ else:
+ ds = de.GeneratorDataset(source=generator,
+ column_names=['img', 'annotation'],
+ num_parallel_workers=cfg.train_cfg['num_workers'],
+ shuffle=shuffle)
+ target_preprocess_function = (lambda img, annotation: target_preprocess(img, annotation, cfg, priors))
+ ds = ds.map(operations=target_preprocess_function, input_columns=['img', 'annotation'],
+ output_columns=['img', 'loc', 'conf'], column_order=['img', 'loc', 'conf'])
+ ds = ds.batch(cfg.train_cfg['per_batch_size'], drop_remainder=True)
+
+ return ds, generator
diff --git a/research/cv/m2det/src/detector.py b/research/cv/m2det/src/detector.py
new file mode 100644
index 0000000000000000000000000000000000000000..d15d1c648d995538a7f1db6b9bef5df61b27eba8
--- /dev/null
+++ b/research/cv/m2det/src/detector.py
@@ -0,0 +1,66 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+import mindspore
+from mindspore import Tensor
+from mindspore import nn
+from mindspore import ops
+
+from src.box_utils import decode
+
+
+class Detect(nn.Cell):
+ """At test time, Detect is the final layer of SSD. Decode location preds,
+ apply non-maximum suppression to location predictions based on conf
+ scores and threshold to a top_k number of output predictions for both
+ confidence score and locations.
+ """
+ def __init__(self, num_classes, cfg):
+ super().__init__()
+ self.num_classes = num_classes
+ self.variance = cfg['variance']
+ self.zero_float = Tensor(0, dtype=mindspore.float32)
+
+ def construct(self, predictions, prior):
+ """
+ Args:
+ predictions: Predictions
+ prior: Prior boxes and variances from priorbox layers
+ """
+
+ loc, conf = predictions
+
+ bs = loc.shape[0] # batch size
+ priors_number = prior.shape[0]
+ boxes = ops.BroadcastTo((1, priors_number, 4))(self.zero_float)
+ scores = ops.BroadcastTo((1, priors_number, self.num_classes))(self.zero_float)
+
+ if bs == 1:
+ conf_preds = ops.ExpandDims()(conf, 0) # Shape [bs, cls_num, priors_num]
+
+ else:
+ conf_preds = conf.view(bs, priors_number, self.num_classes)
+ boxes = ops.BroadcastTo((bs, priors_number, 4))(boxes)
+ scores = ops.BroadcastTo((bs, priors_number, self.num_classes))(scores)
+
+ # Decode predictions into bounding boxes.
+ for sample_index in range(bs):
+ decoded_boxes = decode(loc[sample_index], prior, self.variance)
+ conf_scores = conf_preds[sample_index].copy()
+
+ boxes[sample_index] = decoded_boxes
+ scores[sample_index] = conf_scores
+
+ return boxes, scores
diff --git a/research/cv/m2det/src/loss.py b/research/cv/m2det/src/loss.py
new file mode 100644
index 0000000000000000000000000000000000000000..8ea95e0a3334b9bf9ae148f3c1143b6a2cc65bb7
--- /dev/null
+++ b/research/cv/m2det/src/loss.py
@@ -0,0 +1,94 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""Loss function"""
+
+import mindspore
+from mindspore import Tensor
+from mindspore import nn
+from mindspore import ops
+
+from src.box_utils import log_sum_exp
+
+
+class MultiBoxLoss(nn.Cell):
+ """SSD Weighted Loss Function"""
+
+ def __init__(self,
+ num_classes,
+ neg_pos):
+ super().__init__()
+ self.num_classes = num_classes
+ self.negpos_ratio = neg_pos
+
+ self.zero_float = Tensor(0, dtype=mindspore.float32)
+ self.one_float = Tensor(1, dtype=mindspore.float32)
+
+ def construct(self, predictions, loc_t, conf_t):
+ """Forward pass"""
+
+ loc, conf = predictions
+ num = loc.shape[0]
+
+ unsqueeze = ops.ExpandDims()
+ pos = ops.repeat_elements(unsqueeze(conf_t, conf_t.ndim), rep=loc.shape[-1], axis=conf_t.ndim)
+
+ # Localization Loss (Smooth L1)
+ # Shape: [batch,num_priors,4]
+ loc_p = ops.Select()(pos > 0, loc, ops.ZerosLike()(loc))
+ loc_t = ops.Select()(pos > 0, loc_t, ops.ZerosLike()(loc))
+ loss_l = ops.SmoothL1Loss()(loc_p, loc_t).sum()
+
+ # Compute max conf across batch for hard negative mining
+ batch_conf = conf.view(-1, self.num_classes)
+ loss_cls = log_sum_exp(batch_conf) - ops.GatherD()(batch_conf, 1, conf_t.view(-1, 1))
+
+ # Hard Negative Mining
+ loss_cls = ops.Select()(conf_t.view(-1, 1) > 0, ops.ZerosLike()(loss_cls), loss_cls) # filter out pos boxes
+ loss_cls = loss_cls.view(num, -1)
+ _, loss_idx = ops.Sort(axis=1, descending=True)(loss_cls)
+ _, idx_rank = ops.Sort(axis=1)(loss_idx.astype('float32'))
+ pos_number = (conf_t > 0).astype('int32').sum(1, keepdims=True)
+ negpos_numpos = (self.negpos_ratio * pos_number).astype('float32')
+ num_neg = ops.clip_by_value(negpos_numpos, clip_value_min=negpos_numpos.min(), clip_value_max=pos.shape[1] - 1)
+ neg = idx_rank < num_neg.expand_as(idx_rank)
+
+ # Confidence Loss Including Positive and Negative Examples
+ pos_idx = unsqueeze((conf_t > 0).astype('int32'), 2).expand_as(conf)
+ neg_idx = unsqueeze(neg.astype('int32'), 2).expand_as(conf)
+ conf_p = ops.Select()(
+ ops.Greater()(pos_idx + neg_idx, 0),
+ conf,
+ ops.ZerosLike()(conf),
+ ).view(-1, self.num_classes)
+ valid_targets = ops.Greater()((conf_t > 0).astype('int32') + neg.astype('int32'), 0)
+ targets_weighted = ops.Select()(
+ valid_targets,
+ conf_t,
+ ops.ZerosLike()(conf_t),
+ ).view(-1).astype('int32')
+ depth, on_value, off_value = self.num_classes, self.one_float, self.zero_float
+ loss_cls, _ = ops.SoftmaxCrossEntropyWithLogits()(
+ conf_p,
+ ops.OneHot()(targets_weighted, depth, on_value, off_value),
+ )
+ loss_cls = ops.Select()(valid_targets.view(-1), loss_cls, ops.ZerosLike()(loss_cls))
+ loss_cls = loss_cls.sum()
+
+ # Sum of losses: L(x,c,l,g) = (Lconf(x, c) + αLloc(x,l,g)) / N
+
+ N = ops.Maximum()(pos_number.sum(), self.one_float)
+ loss_l /= N
+ loss_cls /= N
+ return loss_l + loss_cls
diff --git a/research/cv/m2det/src/lr_scheduler.py b/research/cv/m2det/src/lr_scheduler.py
new file mode 100644
index 0000000000000000000000000000000000000000..023dcd8cd7753fbc3bfdf78ce7d9a69748dcfc25
--- /dev/null
+++ b/research/cv/m2det/src/lr_scheduler.py
@@ -0,0 +1,75 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""Learning rate scheduler."""
+
+from collections import Counter
+
+import numpy as np
+
+
+def linear_warmup_lr(current_step, warmup_steps, base_lr, init_lr):
+ """Linear learning rate."""
+ lr_inc = (float(base_lr) - float(init_lr)) / float(warmup_steps)
+ lr = float(init_lr) + lr_inc * current_step
+ return lr
+
+
+def warmup_step_lr(lr, lr_epochs, steps_per_epoch, warmup_epochs, max_epoch, gamma=0.1):
+ """Warmup step learning rate."""
+ base_lr = lr
+ warmup_init_lr = 0
+ total_steps = int(max_epoch * steps_per_epoch)
+ warmup_steps = int(warmup_epochs * steps_per_epoch)
+ if lr_epochs[-1] != max_epoch:
+ lr_epochs = lr_epochs[:-1]
+ lr_epochs.append(max_epoch)
+ milestones = lr_epochs
+ milestones_steps = []
+ for milestone in milestones:
+ milestones_step = milestone * steps_per_epoch
+ milestones_steps.append(milestones_step)
+
+ lr_each_step = []
+ lr = base_lr
+ milestones_steps_counter = Counter(milestones_steps)
+ if isinstance(gamma, list):
+ gamma_per_milestone = [1]
+ for reduction in gamma:
+ gamma_per_milestone.append(reduction * gamma_per_milestone[-1])
+ current_milestone = 0
+ for i in range(total_steps):
+ if i < warmup_steps:
+ lr = linear_warmup_lr(i + 1, warmup_steps, base_lr, warmup_init_lr)
+ elif isinstance(gamma, list):
+ if milestones_steps_counter[i] == 1:
+ current_milestone += milestones_steps_counter[i]
+ lr = base_lr * gamma_per_milestone[current_milestone]
+ else:
+ lr = lr * gamma**milestones_steps_counter[i]
+ lr_each_step.append(lr)
+
+ return np.array(lr_each_step).astype(np.float32)
+
+
+def get_lr(cfg, steps_per_epoch):
+ """generate learning rate."""
+ lr = warmup_step_lr(cfg['lr'],
+ cfg['lr_epochs'],
+ steps_per_epoch,
+ cfg['warmup'],
+ cfg['total_epochs'],
+ gamma=cfg['gamma'],
+ )
+ return lr
diff --git a/research/cv/m2det/src/model.py b/research/cv/m2det/src/model.py
new file mode 100644
index 0000000000000000000000000000000000000000..db7610dba97b29ad9efafd2b262575598b63cb32
--- /dev/null
+++ b/research/cv/m2det/src/model.py
@@ -0,0 +1,348 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""Network modules and utilities"""
+
+from mindspore import load_checkpoint
+from mindspore import nn
+from mindspore import ops
+from mindspore.common import initializer
+
+
+class VGG(nn.Cell):
+
+ def __init__(self, cfg, i, batch_norm=False, pretrained=None):
+ super().__init__()
+ self.layers = self.make_layers(cfg, i, batch_norm=batch_norm)
+ if pretrained:
+ print('Loading pretrained VGG16...')
+ load_checkpoint(pretrained, self)
+
+ def make_layers(self, cfg, i, batch_norm=False):
+ layers = []
+ in_channels = i
+ for v in cfg:
+ if v == 'M':
+ layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
+ elif v == 'C':
+ layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
+ else:
+ conv2d = nn.Conv2d(in_channels, v, kernel_size=3, pad_mode='pad', padding=1, has_bias=True)
+ if batch_norm:
+ layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU()]
+ else:
+ layers += [conv2d, nn.ReLU()]
+ in_channels = v
+ pool5 = nn.MaxPool2d(kernel_size=3, stride=1, pad_mode='same')
+ conv6 = nn.Conv2d(512, 1024, kernel_size=3, pad_mode='pad', padding=6, dilation=6, has_bias=True)
+ conv7 = nn.Conv2d(1024, 1024, kernel_size=1, has_bias=True)
+ layers += [pool5, conv6,
+ nn.ReLU(), conv7, nn.ReLU()]
+ return nn.CellList(layers)
+
+ def construct(self, x, out_inds):
+ out = []
+ for i, layer in enumerate(self.layers):
+ x = layer(x)
+ if i in out_inds:
+ out.append(x)
+ return out
+
+
+class BasicConv(nn.Cell):
+
+ def __init__(self, in_planes, out_planes, kernel_size, stride=1, padding=0, dilation=1,
+ groups=1, relu=True, bn=True, bias=False):
+ super().__init__()
+ self.out_channels = out_planes
+ self.conv = nn.Conv2d(in_planes, out_planes, kernel_size=kernel_size,
+ stride=stride, padding=padding, dilation=dilation, group=groups, has_bias=bias,
+ pad_mode='pad')
+ self.bn = nn.BatchNorm2d(out_planes, eps=1e-5, momentum=0.99, affine=True) if bn else None
+ self.relu = nn.ReLU() if relu else None
+
+ def construct(self, x):
+ x = self.conv(x)
+ if self.bn is not None:
+ x = self.bn(x)
+ if self.relu is not None:
+ x = self.relu(x)
+ return x
+
+
+class TUM(nn.Cell):
+
+ def __init__(self, first_level=True, input_planes=128, is_smooth=True, side_channel=512, scales=6):
+ super().__init__()
+ self.is_smooth = is_smooth
+ self.side_channel = side_channel
+ self.input_planes = input_planes
+ self.planes = 2 * self.input_planes
+ self.first_level = first_level
+ self.scales = scales
+ self.in1 = input_planes + side_channel if not first_level else input_planes
+ self.concat = ops.Concat(axis=1)
+
+ layers = [BasicConv(self.in1, self.planes, 3, 2, 1)]
+ for i in range(self.scales - 2):
+ if not i == self.scales - 3:
+ layers.append(BasicConv(self.planes, self.planes, 3, 2, 1))
+ else:
+ layers.append(BasicConv(self.planes, self.planes, 3, 1, 0))
+ self.layers = nn.CellList(layers)
+ self.n_layers = len(layers)
+ self.toplayer = nn.CellList([BasicConv(self.planes, self.planes, 1, 1, 0)])
+
+ latlayer = []
+ for i in range(self.scales - 2):
+ latlayer.append(BasicConv(self.planes, self.planes, 3, 1, 1))
+ latlayer.append(BasicConv(self.in1, self.planes, 3, 1, 1))
+ self.latlayer = nn.CellList(latlayer)
+
+ if self.is_smooth:
+ smooth = []
+ for i in range(self.scales - 1):
+ smooth.append(BasicConv(self.planes, self.planes, 1, 1, 0))
+ self.smooth = nn.CellList(smooth)
+
+ def _upsample_add(self, x, y):
+ H, W = y.shape[-2:]
+ out = ops.ResizeNearestNeighbor((H, W))(x) + y
+ return out
+
+ def construct(self, x, y):
+ if not self.first_level:
+ x = self.concat([x, y])
+ conved_feat = [x]
+ for i in range(len(self.layers)):
+ x = self.layers[i](x)
+ conved_feat.append(x)
+
+ deconved_feat = [self.toplayer[0](conved_feat[-1])]
+ for i in range(len(self.latlayer)):
+ deconved_feat.append(
+ self._upsample_add(
+ deconved_feat[i], self.latlayer[i](conved_feat[len(self.layers) - 1 - i])
+ )
+ )
+ if self.is_smooth:
+ smoothed_feat = [deconved_feat[0]]
+ for i in range(len(self.smooth)):
+ smoothed_feat.append(
+ self.smooth[i](deconved_feat[i + 1])
+ )
+ return smoothed_feat
+ return deconved_feat
+
+
+class DynamicUpscale(nn.Cell):
+
+ def __init__(self, scale_factor=1, mode='nearest'):
+ super().__init__()
+ self.scale_factor = scale_factor
+ self.mode = mode
+
+ def construct(self, x):
+ shape = x.shape[-2:]
+ if self.mode == 'nearest':
+ operation = ops.ResizeNearestNeighbor((shape[0]*self.scale_factor, shape[1]*self.scale_factor))(x)
+ else:
+ operation = nn.ResizeBilinear()(x, size=(shape[0]*self.scale_factor, shape[1]*self.scale_factor))
+ return operation
+
+
+class M2Det(nn.Cell):
+
+ def __init__(self, phase, size, config=None):
+ """M2Det: Multi-level Multi-scale single-shot object Detector"""
+ super().__init__()
+ self.model_phase = phase
+ self.size = size
+ self.init_params(config)
+ self.construct_modules()
+ self.concat = ops.Concat(axis=1)
+ self.upscale = DynamicUpscale(scale_factor=2, mode='nearest')
+
+ def init_params(self, config=None): # Directly read the config
+ assert config is not None, 'Error: no config'
+ for key, value in config.items():
+ setattr(self, key, value)
+
+ def construct_modules(self):
+ # construct tums
+ for i in range(self.num_levels):
+ if i == 0:
+ setattr(self,
+ 'unet{}'.format(i + 1),
+ TUM(first_level=True,
+ input_planes=self.planes // 2,
+ is_smooth=self.smooth,
+ scales=self.num_scales,
+ side_channel=512)) # side channel isn't fixed.
+ else:
+ setattr(self,
+ 'unet{}'.format(i + 1),
+ TUM(first_level=False,
+ input_planes=self.planes // 2,
+ is_smooth=self.smooth,
+ scales=self.num_scales,
+ side_channel=self.planes))
+ self.unets = []
+ for i in range(self.num_levels):
+ self.unets.append(getattr(self, 'unet{}'.format(i + 1)))
+
+ # construct base features
+ if 'vgg' in self.net_family:
+ if self.backbone == 'vgg16':
+ vgg_param = [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'C', 512, 512, 512, 'M', 512, 512, 512]
+ self.base = VGG(vgg_param, 3, batch_norm=False, pretrained=self.checkpoint_path)
+ shallow_in, shallow_out = 512, 256
+ deep_in, deep_out = 1024, 512
+ else:
+ print(f'Backbone {self.backbone} not implemented')
+ else:
+ print(f'Net family {self.net_family} not implemented')
+ self.reduce = BasicConv(shallow_in, shallow_out, 3, stride=1, padding=1)
+ self.up_reduce = BasicConv(deep_in, deep_out, 1, stride=1)
+
+ # construct others
+ if self.model_phase == 'test':
+ self.softmax = nn.Softmax()
+ self.Norm = nn.BatchNorm2d(256 * 8)
+ self.leach = nn.CellList([BasicConv(
+ deep_out + shallow_out,
+ self.planes // 2,
+ kernel_size=(1, 1), stride=(1, 1))] * self.num_levels)
+
+ # construct localization and recognition layers
+ loc_ = list()
+ conf_ = list()
+ for i in range(self.num_scales):
+ loc_.append(nn.Conv2d(in_channels=self.planes * self.num_levels,
+ out_channels=4 * 6, # 4 is coordinates, 6 is anchors for each pixels,
+ kernel_size=3,
+ stride=1,
+ padding=1,
+ pad_mode='pad',
+ has_bias=True,
+ weight_init='uniform'))
+ conf_.append(nn.Conv2d(in_channels=self.planes * self.num_levels,
+ out_channels=self.num_classes * 6, # 6 is anchors for each pixels,
+ kernel_size=3,
+ stride=1,
+ padding=1,
+ pad_mode='pad',
+ has_bias=True,
+ weight_init='uniform'))
+ self.loc = nn.CellList(loc_)
+ self.conf = nn.CellList(conf_)
+
+ def construct(self, x):
+ loc, conf, base_feats = [], [], []
+ base_feats = self.base(x, self.base_out)
+ base_feature = self.concat(
+ (self.reduce(base_feats[0]), self.upscale(self.up_reduce(base_feats[1]))))
+
+ # tum_outs is the multi-level multi-scale feature
+ tum_outs = [self.unets[0](self.leach[0](base_feature), 'none')]
+ for i in range(1, self.num_levels, 1):
+ tum_outs.append(
+ self.unets[i](
+ self.leach[i](base_feature), tum_outs[i - 1][-1]
+ )
+ )
+ # concat with same scales
+ sources = []
+ for i in range(self.num_scales, 0, -1):
+ _fx_list = []
+ for j in range(self.num_levels):
+ _fx_list.append(tum_outs[j][i - 1])
+ sources.append(self.concat(_fx_list))
+
+ sources[0] = self.Norm(sources[0])
+
+ for (k, l, c) in zip(sources, self.loc, self.conf):
+ loc.append(l(k).transpose(0, 2, 3, 1))
+ conf.append(c(k).transpose(0, 2, 3, 1))
+
+ loc_list = []
+ conf_list = []
+ for i in range(self.num_scales):
+ loc_list.append(loc[i].view(loc[i].shape[0], -1))
+ conf_list.append(conf[i].view(conf[i].shape[0], -1))
+ loc = self.concat(loc_list)
+ conf = self.concat(conf_list)
+
+ if self.model_phase == "test":
+ output = (
+ loc.view(loc.shape[0], -1, 4), # loc preds
+ self.softmax(conf.view(-1, self.num_classes)), # conf preds
+ )
+ else:
+ output = (
+ loc.view(loc.shape[0], -1, 4),
+ conf.view(conf.shape[0], -1, self.num_classes),
+ )
+ return output
+
+ def init_model(self):
+ def weights_init(m):
+ for _, cell in m.cells_and_names():
+ if isinstance(cell, nn.Conv2d):
+ cell.weight.set_data(initializer.initializer(initializer.Normal(sigma=0.001),
+ cell.weight.shape,
+ cell.weight.dtype))
+ if cell.has_bias:
+ cell.bias.set_data(initializer.initializer(0,
+ cell.bias.shape,
+ cell.bias.dtype))
+ elif isinstance(cell, nn.BatchNorm2d):
+ cell.gamma.set_data(initializer.initializer(1,
+ cell.gamma.shape,
+ cell.gamma.dtype))
+ cell.beta.set_data(initializer.initializer(0,
+ cell.beta.shape,
+ cell.beta.dtype))
+
+ print('Initializing weights for [tums, reduce, up_reduce, leach, loc, conf]...')
+ for i in range(self.num_levels):
+ weights_init(self.unets[i])
+ weights_init(self.reduce)
+ weights_init(self.up_reduce)
+ weights_init(self.leach)
+ weights_init(self.loc)
+ weights_init(self.conf)
+
+
+class M2DetWithLoss(nn.Cell):
+
+ def __init__(self, model, loss):
+ super().__init__()
+ self.model = model
+ self.loss = loss
+
+ def construct(self, img, loc, conf):
+ output = self.model(img)
+ return self.loss(output, loc, conf)
+
+
+def get_model(cfg, input_size, test=False):
+ if test:
+ phase = 'test'
+ else:
+ phase = 'train'
+ model = M2Det(phase, input_size, config=cfg)
+ model.init_model()
+
+ return model
diff --git a/research/cv/m2det/src/nms/__init__.py b/research/cv/m2det/src/nms/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
diff --git a/research/cv/m2det/src/nms/cpu_nms.pyx b/research/cv/m2det/src/nms/cpu_nms.pyx
new file mode 100644
index 0000000000000000000000000000000000000000..8f6180d822562e389f5b2d56da33d0daf39de770
--- /dev/null
+++ b/research/cv/m2det/src/nms/cpu_nms.pyx
@@ -0,0 +1,171 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+import numpy as np
+cimport numpy as np
+
+cdef inline np.float32_t max(np.float32_t a, np.float32_t b):
+ return a if a >= b else b
+
+cdef inline np.float32_t min(np.float32_t a, np.float32_t b):
+ return a if a <= b else b
+
+def cpu_nms(np.ndarray[np.float32_t, ndim=2] dets, np.float thresh):
+ cdef np.ndarray[np.float32_t, ndim=1] x1 = dets[:, 0]
+ cdef np.ndarray[np.float32_t, ndim=1] y1 = dets[:, 1]
+ cdef np.ndarray[np.float32_t, ndim=1] x2 = dets[:, 2]
+ cdef np.ndarray[np.float32_t, ndim=1] y2 = dets[:, 3]
+ cdef np.ndarray[np.float32_t, ndim=1] scores = dets[:, 4]
+
+ cdef np.ndarray[np.float32_t, ndim=1] areas = (x2 - x1 + 1) * (y2 - y1 + 1)
+ cdef np.ndarray[np.int_t, ndim=1] order = scores.argsort()[::-1]
+
+ cdef int ndets = dets.shape[0]
+ cdef np.ndarray[np.int_t, ndim=1] suppressed = \
+ np.zeros((ndets), dtype=np.int)
+
+ # nominal indices
+ cdef int _i, _j
+ # sorted indices
+ cdef int i, j
+ # temp variables for box i's (the box currently under consideration)
+ cdef np.float32_t ix1, iy1, ix2, iy2, iarea
+ # variables for computing overlap with box j (lower scoring box)
+ cdef np.float32_t xx1, yy1, xx2, yy2
+ cdef np.float32_t w, h
+ cdef np.float32_t inter, ovr
+
+ keep = []
+ for _i in range(ndets):
+ i = order[_i]
+ if suppressed[i] == 1:
+ continue
+ keep.append(i)
+ ix1 = x1[i]
+ iy1 = y1[i]
+ ix2 = x2[i]
+ iy2 = y2[i]
+ iarea = areas[i]
+ for _j in range(_i + 1, ndets):
+ j = order[_j]
+ if suppressed[j] == 1:
+ continue
+ xx1 = max(ix1, x1[j])
+ yy1 = max(iy1, y1[j])
+ xx2 = min(ix2, x2[j])
+ yy2 = min(iy2, y2[j])
+ w = max(0.0, xx2 - xx1 + 1)
+ h = max(0.0, yy2 - yy1 + 1)
+ inter = w * h
+ ovr = inter / (iarea + areas[j] - inter)
+ if ovr >= thresh:
+ suppressed[j] = 1
+
+ return keep
+
+def cpu_soft_nms(np.ndarray[float, ndim=2] boxes, float sigma=0.5, float Nt=0.3, float threshold=0.001, unsigned int method=0):
+ cdef unsigned int N = boxes.shape[0]
+ cdef float iw, ih, box_area
+ cdef float ua
+ cdef int pos = 0
+ cdef float maxscore = 0
+ cdef int maxpos = 0
+ cdef float x1,x2,y1,y2,tx1,tx2,ty1,ty2,ts,area,weight,ov
+
+ for i in range(N):
+ maxscore = boxes[i, 4]
+ maxpos = i
+
+ tx1 = boxes[i,0]
+ ty1 = boxes[i,1]
+ tx2 = boxes[i,2]
+ ty2 = boxes[i,3]
+ ts = boxes[i,4]
+
+ pos = i + 1
+ # get max box
+ while pos < N:
+ if maxscore < boxes[pos, 4]:
+ maxscore = boxes[pos, 4]
+ maxpos = pos
+ pos = pos + 1
+
+ # add max box as a detection
+ boxes[i,0] = boxes[maxpos,0]
+ boxes[i,1] = boxes[maxpos,1]
+ boxes[i,2] = boxes[maxpos,2]
+ boxes[i,3] = boxes[maxpos,3]
+ boxes[i,4] = boxes[maxpos,4]
+
+ # swap ith box with position of max box
+ boxes[maxpos,0] = tx1
+ boxes[maxpos,1] = ty1
+ boxes[maxpos,2] = tx2
+ boxes[maxpos,3] = ty2
+ boxes[maxpos,4] = ts
+
+ tx1 = boxes[i,0]
+ ty1 = boxes[i,1]
+ tx2 = boxes[i,2]
+ ty2 = boxes[i,3]
+ ts = boxes[i,4]
+
+ pos = i + 1
+ # NMS iterations, note that N changes if detection boxes fall below threshold
+ while pos < N:
+ x1 = boxes[pos, 0]
+ y1 = boxes[pos, 1]
+ x2 = boxes[pos, 2]
+ y2 = boxes[pos, 3]
+ s = boxes[pos, 4]
+
+ area = (x2 - x1 + 1) * (y2 - y1 + 1)
+ iw = (min(tx2, x2) - max(tx1, x1) + 1)
+ if iw > 0:
+ ih = (min(ty2, y2) - max(ty1, y1) + 1)
+ if ih > 0:
+ ua = float((tx2 - tx1 + 1) * (ty2 - ty1 + 1) + area - iw * ih)
+ ov = iw * ih / ua #iou between max box and detection box
+
+ if method == 1: # linear
+ if ov > Nt:
+ weight = 1 - ov
+ else:
+ weight = 1
+ elif method == 2: # gaussian
+ weight = np.exp(-(ov * ov)/sigma)
+ else: # original NMS
+ if ov > Nt:
+ weight = 0
+ else:
+ weight = 1
+
+ boxes[pos, 4] = weight*boxes[pos, 4]
+
+ # if box score falls below threshold, discard the box by swapping with last box
+ # update N
+ if boxes[pos, 4] < threshold:
+ boxes[pos,0] = boxes[N-1, 0]
+ boxes[pos,1] = boxes[N-1, 1]
+ boxes[pos,2] = boxes[N-1, 2]
+ boxes[pos,3] = boxes[N-1, 3]
+ boxes[pos,4] = boxes[N-1, 4]
+ N = N - 1
+ pos = pos - 1
+
+ pos = pos + 1
+
+ keep = [i for i in range(N)]
+ return keep
diff --git a/research/cv/m2det/src/priors.py b/research/cv/m2det/src/priors.py
new file mode 100644
index 0000000000000000000000000000000000000000..86c3ae77dcfa7c96ab6eae905e8728d69cf3d46c
--- /dev/null
+++ b/research/cv/m2det/src/priors.py
@@ -0,0 +1,111 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+import math
+from itertools import product
+
+import mindspore
+
+_NUM_REGLAYER = 6
+
+
+def reglayer_scale(size, num_layer, size_the):
+ reg_layer_size = []
+ for i in range(num_layer + 1):
+ size = math.ceil(size / 2.)
+ if i >= 2:
+ reg_layer_size += [size]
+ if i == num_layer and size_the != 0:
+ reg_layer_size += [size - size_the]
+ return reg_layer_size
+
+
+def get_scales(size, size_pattern):
+ size_list = []
+ for x in size_pattern:
+ size_list += [round(x * size, 2)]
+ return size_list
+
+
+def prepare_aspect_ratio(num):
+ as_ra = []
+ for _ in range(num):
+ as_ra += [[2, 3]]
+ return as_ra
+
+
+def mk_anchors(size, multiscale_size, size_pattern, step_pattern):
+ cfg = {
+ 'feature_maps': reglayer_scale(size, _NUM_REGLAYER, 2),
+ 'min_dim': size,
+ 'steps': step_pattern,
+ 'min_sizes': get_scales(multiscale_size, size_pattern[:-1]),
+ 'max_sizes': get_scales(multiscale_size, size_pattern[1:]),
+ 'aspect_ratios': prepare_aspect_ratio(_NUM_REGLAYER),
+ 'variance': [0.1, 0.2],
+ 'clip': True,
+ }
+ return cfg
+
+
+def anchors(cfg):
+ input_size = cfg.model['input_size']
+ size_pattern = cfg.model['anchor_config']['size_pattern']
+ step_pattern = cfg.model['anchor_config']['step_pattern']
+ return mk_anchors(input_size, input_size, size_pattern, step_pattern)
+
+
+class PriorBox:
+ """Computing prior boxes coordinates"""
+
+ def __init__(self, config):
+ cfg = anchors(config)
+ self.image_size = cfg['min_dim']
+ # number of priors for feature map location (either 4 or 6)
+ self.num_priors = len(cfg['aspect_ratios'])
+ self.variance = cfg['variance'] or [0.1]
+ self.feature_maps = cfg['feature_maps']
+ self.min_sizes = cfg['min_sizes']
+ self.max_sizes = cfg['max_sizes']
+ self.steps = cfg['steps']
+ self.aspect_ratios = cfg['aspect_ratios']
+ self.clip = cfg['clip']
+ for v in self.variance:
+ if v <= 0:
+ raise ValueError('Variances must be greater than 0')
+
+ def forward(self):
+ mean = []
+ for k, f in enumerate(self.feature_maps):
+ f_k = self.image_size / self.steps[k]
+ s_k = self.min_sizes[k] / self.image_size
+ s_k_prime = math.sqrt(s_k * (self.max_sizes[k] / self.image_size))
+
+ for i, j in product(range(f), repeat=2):
+ cx = (j + 0.5) / f_k
+ cy = (i + 0.5) / f_k
+
+ mean += [cx, cy, s_k, s_k]
+ mean += [cx, cy, s_k_prime, s_k_prime] # aspect_ratio: 1
+
+ # rest of aspect ratios
+ for ar in self.aspect_ratios[k]:
+ mean += [cx, cy, s_k * math.sqrt(ar), s_k / math.sqrt(ar)]
+ mean += [cx, cy, s_k / math.sqrt(ar), s_k * math.sqrt(ar)]
+
+ output = mindspore.Tensor(mean, dtype=mindspore.float32).view(-1, 4)
+ if self.clip:
+ output = mindspore.ops.clip_by_value(output, clip_value_max=1, clip_value_min=0)
+ return output
diff --git a/research/cv/m2det/src/utils.py b/research/cv/m2det/src/utils.py
new file mode 100644
index 0000000000000000000000000000000000000000..7740ff96ffacbed97baada32b5e0c0ef4aaa65f5
--- /dev/null
+++ b/research/cv/m2det/src/utils.py
@@ -0,0 +1,267 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+import os.path as osp
+import sys
+import time
+from argparse import ArgumentParser
+from collections import Iterable
+from importlib import import_module
+
+import numpy as np
+from addict import Dict
+import mindspore
+
+from src.nms.cpu_nms import cpu_soft_nms
+
+
+def get_param_groups(network):
+ """Param groups for optimizer."""
+ decay_params = []
+ no_decay_params = []
+ for x in network.trainable_params():
+ parameter_name = x.name
+ if parameter_name.endswith('.bias'):
+ # all bias not using weight decay
+ no_decay_params.append(x)
+ elif parameter_name.endswith('.gamma'):
+ # bn weight bias not using weight decay, be carefully for now x not include BN
+ no_decay_params.append(x)
+ elif parameter_name.endswith('.beta'):
+ # bn weight bias not using weight decay, be carefully for now x not include BN
+ no_decay_params.append(x)
+ else:
+ decay_params.append(x)
+
+ return [{'params': no_decay_params, 'weight_decay': 0.0}, {'params': decay_params}]
+
+
+class ConfigDict(Dict):
+
+ def __missing__(self, name):
+ raise KeyError(name)
+
+ def __getattr__(self, name):
+ try:
+ value = super(ConfigDict, self).__getattr__(name)
+ except KeyError:
+ ex = AttributeError("'{}' object has no attribute '{}'".format(
+ self.__class__.__name__, name))
+ else:
+ return value
+ raise ex
+
+
+def add_args(parser, cfg, prefix=''):
+ for k, v in cfg.items():
+ if isinstance(v, str):
+ parser.add_argument('--' + prefix + k)
+ elif isinstance(v, int):
+ parser.add_argument('--' + prefix + k, type=int)
+ elif isinstance(v, float):
+ parser.add_argument('--' + prefix + k, type=float)
+ elif isinstance(v, bool):
+ parser.add_argument('--' + prefix + k, action='store_true')
+ elif isinstance(v, dict):
+ add_args(parser, v, k + '.')
+ elif isinstance(v, Iterable):
+ parser.add_argument('--' + prefix + k, type=type(v[0]), nargs='+')
+ else:
+ print('connot parse key {} of type {}'.format(prefix + k, type(v)))
+ return parser
+
+
+def check_file_exist(filename, msg_tmpl='file "{}" does not exist'):
+ if not osp.isfile(filename):
+ raise FileNotFoundError(msg_tmpl.format(filename))
+
+
+class Config:
+ """A facility for config and config files.
+ It supports common file formats as configs: python/json/yaml. The interface
+ is the same as a dict object and also allows access config values as
+ attributes.
+ Example:
+ >>> cfg = Config(dict(a=1, b=dict(b1=[0, 1])))
+ >>> cfg.a
+ 1
+ >>> cfg.b
+ {'b1': [0, 1]}
+ >>> cfg.b.b1
+ [0, 1]
+ >>> cfg = Config.fromfile('tests/data/config/a.py')
+ >>> cfg.filename
+ "/home/kchen/projects/mmcv/tests/data/config/a.py"
+ >>> cfg.item4
+ 'test'
+ >>> cfg
+ "Config [path: /home/kchen/projects/mmcv/tests/data/config/a.py]: "
+ "{'item1': [1, 2], 'item2': {'a': 0}, 'item3': True, 'item4': 'test'}"
+ """
+
+ @staticmethod
+ def fromfile(filename):
+ filename = osp.abspath(osp.expanduser(filename))
+ check_file_exist(filename)
+ if filename.endswith('.py'):
+ module_name = osp.basename(filename)[:-3]
+ if '.' in module_name:
+ raise ValueError('Dots are not allowed in config file path.')
+ config_dir = osp.dirname(filename)
+ sys.path.insert(0, config_dir)
+ mod = import_module(module_name)
+ sys.path.pop(0)
+ cfg_dict = {
+ name: value
+ for name, value in mod.__dict__.items()
+ if not name.startswith('__')
+ }
+ elif filename.endswith(('.yaml', '.json')):
+ import mmcv
+ cfg_dict = mmcv.load(filename)
+ else:
+ raise IOError('Only py/yaml/json type are supported now!')
+ return Config(cfg_dict, filename=filename)
+
+ @staticmethod
+ def auto_argparser(description=None):
+ """Generate argparser from config file automatically (experimental)
+ """
+ partial_parser = ArgumentParser(description=description)
+ partial_parser.add_argument('config', help='config file path')
+ cfg_file = partial_parser.parse_known_args()[0].config
+ cfg = Config.from_file(cfg_file)
+ parser = ArgumentParser(description=description)
+ parser.add_argument('config', help='config file path')
+ add_args(parser, cfg)
+ return parser, cfg
+
+ def __init__(self, cfg_dict=None, filename=None):
+ if cfg_dict is None:
+ cfg_dict = dict()
+ elif not isinstance(cfg_dict, dict):
+ raise TypeError('cfg_dict must be a dict, but got {}'.format(
+ type(cfg_dict)))
+
+ super(Config, self).__setattr__('_cfg_dict', ConfigDict(cfg_dict))
+ super(Config, self).__setattr__('_filename', filename)
+ if filename:
+ with open(filename, 'r') as f:
+ super(Config, self).__setattr__('_text', f.read())
+ else:
+ super(Config, self).__setattr__('_text', '')
+
+ @property
+ def filename(self):
+ return self._filename
+
+ @property
+ def text(self):
+ return self._text
+
+ def __repr__(self):
+ return 'Config (path: {}): {}'.format(self.filename,
+ self._cfg_dict.__repr__())
+
+ def __len__(self):
+ return len(self._cfg_dict)
+
+ def __getattr__(self, name):
+ return getattr(self._cfg_dict, name)
+
+ def __getitem__(self, name):
+ return self._cfg_dict.__getitem__(name)
+
+ def __setattr__(self, name, value):
+ if isinstance(value, dict):
+ value = ConfigDict(value)
+ self._cfg_dict.__setattr__(name, value)
+
+ def __setitem__(self, name, value):
+ if isinstance(value, dict):
+ value = ConfigDict(value)
+ self._cfg_dict.__setitem__(name, value)
+
+ def __iter__(self):
+ return iter(self._cfg_dict)
+
+
+def image_forward(img, net, priors, detector, transform):
+ w, h = img.shape[1], img.shape[0]
+ scale = mindspore.Tensor([w, h, w, h], dtype=mindspore.float32)
+ x = mindspore.ops.ExpandDims()(transform(img), 0)
+ out = net(x)
+ boxes, scores = detector.construct(out, priors)
+ boxes = (boxes[0] * scale).asnumpy()
+ scores = scores[0].asnumpy()
+ return boxes, scores
+
+
+def nms_process(num_classes, image_index, scores, boxes, cfg, min_thresh, all_boxes, max_per_image):
+ for cls_index in range(1, num_classes): # ignore the background (category_id=0)
+ inds = np.where(scores[:, cls_index] > min_thresh)[0]
+ if inds.size == 0:
+ all_boxes[cls_index][image_index] = np.empty([0, 5], dtype=np.float32)
+ continue
+ cls_bboxes = boxes[inds]
+ cls_scores = scores[inds, cls_index]
+ cls_dets = np.hstack((cls_bboxes, cls_scores[:, None])).astype(np.float32, copy=False)
+
+ keep = cpu_soft_nms(cls_dets, cfg.test_cfg['iou'], method=1)
+ keep = keep[:cfg.test_cfg['keep_per_class']] # keep only the highest boxes
+ cls_dets = cls_dets[keep, :]
+ all_boxes[cls_index][image_index] = cls_dets
+
+ if max_per_image > 0:
+ image_scores = np.hstack([all_boxes[cls_index][image_index][:, -1] for cls_index in range(1, num_classes)])
+ if len(image_scores) > max_per_image:
+ image_thresh = np.sort(image_scores)[-max_per_image]
+ for cls_index in range(1, num_classes):
+ keep = np.where(all_boxes[cls_index][image_index][:, -1] >= image_thresh)[0]
+ all_boxes[cls_index][image_index] = all_boxes[cls_index][image_index][keep, :]
+
+
+class Timer:
+ """A simple timer."""
+
+ def __init__(self):
+ self.total_time = 0.
+ self.calls = 0
+ self.start_time = 0.
+ self.diff = 0.
+ self.average_time = 0.
+
+ def tic(self):
+ # using time.time instead of time.clock because time time.clock
+ # does not normalize for multi-threading
+ self.start_time = time.time()
+
+ def toc(self, average=True):
+ self.diff = time.time() - self.start_time
+ self.total_time += self.diff
+ self.calls += 1
+ self.average_time = self.total_time / self.calls
+ if average:
+ out = self.average_time
+ else:
+ out = self.diff
+ return out
+
+ def clear(self):
+ self.total_time = 0.
+ self.calls = 0
+ self.start_time = 0.
+ self.diff = 0.
+ self.average_time = 0.
diff --git a/research/cv/m2det/train.py b/research/cv/m2det/train.py
new file mode 100644
index 0000000000000000000000000000000000000000..b5ba6c2f7bac2411a71326b7c2a79954224d3c39
--- /dev/null
+++ b/research/cv/m2det/train.py
@@ -0,0 +1,224 @@
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+""" M2Det training """
+import argparse
+import ast
+import os
+
+from mindspore import Model
+from mindspore import Tensor
+from mindspore import context
+from mindspore import load_checkpoint
+from mindspore import nn
+from mindspore import ops
+from mindspore import save_checkpoint
+from mindspore.common import set_seed
+from mindspore.communication import get_group_size
+from mindspore.communication import get_rank
+from mindspore.communication import init
+from mindspore.context import ParallelMode
+from mindspore.train.callback import CheckpointConfig
+from mindspore.train.callback import ModelCheckpoint
+
+from src import config as cfg
+from src.callback import TimeLossMonitor
+from src.dataset import get_dataset
+from src.loss import MultiBoxLoss
+from src.lr_scheduler import get_lr
+from src.model import M2DetWithLoss
+from src.model import get_model
+from src.priors import PriorBox
+from src.utils import get_param_groups
+
+
+class CustomTrainOneStepCell(nn.Cell):
+ """Custom TrainOneStepCell with global gradients clipping"""
+
+ def __init__(self, network, optimizer, max_grad_norm):
+ super().__init__()
+ self.network = network
+ self.network.set_grad()
+ self.optimizer = optimizer
+ self.weights = self.optimizer.parameters
+ self.grad = ops.GradOperation(get_by_list=True)
+ self.reducer_flag = False
+ self.grad_reducer = None
+ self.max_grad_norm = max_grad_norm
+ self.parallel_mode = context.get_auto_parallel_context("parallel_mode")
+ if self.parallel_mode in [ParallelMode.DATA_PARALLEL, ParallelMode.HYBRID_PARALLEL]:
+ self.reducer_flag = True
+ mean = context.get_auto_parallel_context("gradients_mean")
+ degree = get_group_size()
+ self.grad_reducer = nn.DistributedGradReducer(optimizer.parameters, mean, degree)
+
+ def construct(self, *inputs):
+ """construct"""
+ pred = self.network(*inputs)
+ grads = self.grad(self.network, self.weights)(*inputs)
+ if self.max_grad_norm:
+ grads = ops.clip_by_global_norm(grads, clip_norm=self.max_grad_norm)
+ if self.reducer_flag:
+ grads = self.grad_reducer(grads)
+ pred = ops.depend(pred, self.optimizer(grads))
+ return pred
+
+
+def parse_args():
+ """Get arguments from command-line."""
+ parser = argparse.ArgumentParser(description="Mindspore HRNet Training Configurations.")
+ parser.add_argument("--train_url", type=str, default='./checkpoints/', help="Storage path of training results.")
+ parser.add_argument("--run_distribute", type=ast.literal_eval, default=False,
+ help="Use one card or multiple cards training.")
+ parser.add_argument("--device_id", type=int, default=0)
+ parser.add_argument("--pretrained_backbone", type=str, default=None,
+ help="Path to pretrained backbone checkpoint")
+ parser.add_argument("--dataset_path", type=str, default=None,
+ help="Path to dataset root folder")
+
+ return parser.parse_args()
+
+
+def print_config():
+ """Print the configuration file"""
+ print('CONFIG_FILE')
+ for cfg_item in dir(cfg):
+ if cfg_item[0] != '_':
+ item = getattr(cfg, cfg_item)
+ print(f'{cfg_item} = {item}')
+ print('CONFIG_FILE_END')
+
+
+def set_device(device):
+ """Set device"""
+ if device == 'GPU':
+ context.set_context(mode=context.GRAPH_MODE, device_target="GPU")
+ elif device == 'Ascend':
+ raise ValueError('Ascend training not implemented')
+ else:
+ raise ValueError(f'Unknown device type: {device}. Only "GPU" device type implemented')
+
+
+def get_optimizer(net, lr, config):
+ """Get optimizer"""
+ opt = nn.SGD(get_param_groups(net),
+ learning_rate=Tensor(lr),
+ momentum=config.optimizer['momentum'],
+ weight_decay=config.optimizer['weight_decay'],
+ dampening=config.optimizer['dampening'])
+ return opt
+
+
+def main():
+ """Training process."""
+ set_seed(1)
+ args = parse_args()
+
+ local_train_url = args.train_url
+ model_name = cfg.model['m2det_config']['backbone'] + '_' + str(cfg.model['input_size'])
+
+ if args.pretrained_backbone:
+ cfg.model['m2det_config']['checkpoint_path'] = args.pretrained_backbone
+
+ if args.dataset_path:
+ cfg.COCOroot = args.dataset_path
+
+ device_id = args.device_id
+ device = cfg.device
+ set_device(device)
+
+ # Print configuration to log
+ print_config()
+
+ # Create dataset
+ if args.run_distribute:
+ init('nccl')
+ parallel_mode = ParallelMode.DATA_PARALLEL
+ context.set_auto_parallel_context(parallel_mode=parallel_mode, gradients_mean=True)
+ priorbox = PriorBox(cfg)
+ priors = priorbox.forward()
+ loader, _ = get_dataset(cfg, 'COCO', priors.asnumpy(), 'train_sets',
+ random_seed=cfg.random_seed, distributed=True)
+ else:
+ context.set_context(device_id=device_id)
+ priorbox = PriorBox(cfg)
+ priors = priorbox.forward()
+ loader, _ = get_dataset(cfg, 'COCO', priors.asnumpy(), 'train_sets',
+ random_seed=cfg.random_seed, distributed=False)
+
+ loss = MultiBoxLoss(cfg.model['m2det_config']['num_classes'], cfg.loss['neg_pos'])
+
+ # Create network
+ backbone = get_model(cfg.model['m2det_config'], cfg.model['input_size'])
+ if not cfg.start_epoch:
+ cfg.start_epoch = 0
+ if cfg.checkpoint_path is not None:
+ print(f'Loading checkpoint for epoch {cfg.start_epoch}')
+ print(f'Checkpoint filename: {cfg.checkpoint_path}')
+ cfg.model['m2det_config']['checkpoint_path'] = None
+ net = M2DetWithLoss(backbone, loss)
+ load_checkpoint(cfg.checkpoint_path, net=net)
+ else:
+ net = M2DetWithLoss(backbone, loss)
+
+ net.set_train()
+
+ steps_per_epoch = loader.get_dataset_size()
+
+ # Learning rate adjustment with linear scaling rule
+ if args.run_distribute:
+ n_gpus = get_group_size()
+ lr_default = cfg.train_cfg['lr']
+ print(f'Adjusting learning rate (default = {lr_default}) to {n_gpus} GPUs')
+ cfg.train_cfg['lr'] = cfg.train_cfg['lr'] * n_gpus
+
+ # Learning rate schedule construction
+ lr = get_lr(cfg.train_cfg, steps_per_epoch)
+ if cfg.start_epoch > 0:
+ lr = lr[int(cfg.start_epoch * steps_per_epoch):]
+
+ # Optimizer
+ opt = get_optimizer(net, lr, cfg)
+
+ # Create model
+ net_loss_opt = CustomTrainOneStepCell(net, opt, cfg.optimizer['clip_grad_norm'])
+ model = Model(net_loss_opt)
+
+ # Callbacks
+ ckpt_config = CheckpointConfig(save_checkpoint_steps=steps_per_epoch * cfg.model['checkpoint_interval'],
+ keep_checkpoint_max=10)
+ ckpt_cb = ModelCheckpoint(prefix=model_name,
+ directory=os.path.join(local_train_url, cfg.experiment_tag),
+ config=ckpt_config)
+
+ if args.run_distribute:
+ if get_rank() == 0:
+ callbacks = [TimeLossMonitor(lr_init=lr), ckpt_cb]
+ else:
+ callbacks = [TimeLossMonitor(lr_init=lr)]
+ else:
+ callbacks = [TimeLossMonitor(lr_init=lr), ckpt_cb]
+
+ # Run the training process
+ model.train(cfg.train_cfg['total_epochs'] - cfg.start_epoch, loader, callbacks=callbacks, dataset_sink_mode=False)
+
+ last_checkpoint = os.path.join(local_train_url, cfg.experiment_tag, f"{model_name}-final.ckpt")
+ if args.run_distribute & (get_rank() == 0):
+ save_checkpoint(net, last_checkpoint)
+ elif not args.run_distribute:
+ save_checkpoint(net, last_checkpoint)
+
+
+if __name__ == "__main__":
+ main()