diff --git a/research/cv/glore_res200/README_CN.md b/research/cv/glore_res200/README_CN.md
deleted file mode 100644
index cd0bf75fa9600e1c1c49eaeca3e03e65ea8d6d90..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/README_CN.md
+++ /dev/null
@@ -1,329 +0,0 @@
-
-# 目录
-
-<!-- TOC -->
-
-- [目录](#目录)
-- [Glore_resnet200描述](#glore_resnet200描述)
- - [概述](#概述)
- - [论文](#论文)
-- [模型架构](#模型架构)
-- [数据集](#数据集)
-- [特性](#特性)
- - [混合精度](#混合精度)
-- [环境要求](#环境要求)
-- [快速入门](#快速入门)
-- [脚本说明](#脚本说明)
- - [脚本及样例代码](#脚本及样例代码)
- - [脚本参数](#脚本参数)
- - [训练过程](#训练过程)
- - [用法](#用法)
- - [Ascend处理器环境运行](#ascend处理器环境运行)
- - [GPU处理器环境运行](#gpu处理器环境运行)
- - [训练结果](#训练结果)
- - [推理过程](#推理过程)
- - [用法](#用法-1)
- - [Ascend处理器环境运行](#ascend处理器环境运行-1)
- - [GPU处理器环境运行](#gpu处理器环境运行-1)
- - [推理结果](#推理结果)
-- [模型描述](#模型描述)
- - [性能](#性能)
- - [训练性能](#训练性能)
- - [ImageNet2012上的Glore_resnet200](#imagenet2012上的glore_resnet200)
- - [推理性能](#推理性能)
- - [ImageNet2012上的Glore_resnet200](#imagenet2012上的glore_resnet200-1)
-- [随机情况说明](#随机情况说明)
-- [ModelZoo主页](#modelzoo主页)
-
-<!-- /TOC -->
-
-# Glore_resnet200描述
-
-## 概述
-
-卷积神经网络擅长提取局部关系，但是在处理全局上的区域间关系时显得低效，且需要堆叠很多层才可能完成，而在区域之间进行全局建模和推理对很多计算机视觉任务有益。为了进行全局推理，facebook research、新加坡国立大学和360 AI研究所提出了基于图的全局推理模块-Global Reasoning Unit，可以被插入到很多任务的网络模型中。glore_res200是在ResNet200的Stage2, Stage3中分别均匀地插入了2和3个全局推理模块的用于图像分类任务的网络模型。
-
-## 论文
-
-1.[论文](https://arxiv.org/abs/1811.12814):Yunpeng Chenyz, Marcus Rohrbachy, Zhicheng Yany, Shuicheng Yanz, Jiashi Fengz, Yannis Kalantidisy
-
-# 模型架构
-
-网络模型的backbone是ResNet200, 在Stage2, Stage3中分别均匀地插入了了2个和3个全局推理模块。全局推理模块在Stage2和Stage 3中插入方式相同.
-
-# 数据集
-
-使用的数据集：[ImageNet2012](http://www.image-net.org/)
-
-- 数据集大小：共1000个类、224*224彩色图像
- - 训练集：共1,281,167张图像
- - 测试集：共50,000张图像
-- 数据格式：JPEG
- - 注：数据在dataset.py中处理。
-- 下载数据集，目录结构如下:
-
-```text
-└─dataset
- ├─train # 训练数据集
- └─val # 评估数据集
-```
-
-# 特性
-
-## 混合精度
-
-采用[混合精度](https://www.mindspore.cn/tutorials/experts/zh-CN/master/others/mixed_precision.html)的训练方法使用支持单精度和半精度数据来提高深度学习神经网络的训练速度，同时保持单精度训练所能达到的网络精度。混合精度训练提高计算速度、减少内存使用的同时，支持在特定硬件上训练更大的模型或实现更大批次的训练。
-以FP16算子为例，如果输入数据类型为FP32，MindSpore后台会自动降低精度来处理数据。用户可打开INFO日志，搜索“reduce precision”查看精度降低的算子。
-
-# 环境要求
-
-- 硬件(Ascend/GPU)
- - 准备Ascend或GPU处理器搭建硬件环境。
-- 框架
- - [MindSpore](https://www.mindspore.cn/install)
-- 如需查看详情，请参见如下资源：
- - [MindSpore教程](https://www.mindspore.cn/tutorials/zh-CN/master/index.html)
- - [MindSpore Python API](https://www.mindspore.cn/docs/api/zh-CN/master/index.html)
-
-# 快速入门
-
-通过官方网站安装MindSpore后，您可以按照如下步骤进行训练和评估：
-
-- Ascend处理器环境运行
-
-```python
-# 分布式训练
-用法:bash run_distribute_train.sh [DATASET_PATH] [RANK_TABLE]
-
-# 单机训练
-用法:bash run_standalone_train.sh [DATASET_PATH] [DEVICE_ID]
-
-# 运行评估示例
-用法:bash run_eval.sh [DATASET_PATH] [DEVICE_ID] [CHECKPOINT_PATH]
-```
-
-- GPU处理器环境运行
-
-```python
-# 分布式训练
-用法:bash run_distribute_train_gpu.sh [RANK_SIZE] [DATASET_PATH]
-
-# 单机训练
-用法:bash run_standalone_train_gpu.sh [DATASET_PATH]
-
-# 运行评估示例
-用法:bash run_eval_gpu.sh [DATASET_PATH] [CHECKPOINT_PATH]
-```
-
- 对于分布式训练，需要提前创建JSON格式的hccl配置文件。
-
- 请遵循以下链接中的说明：
-
- <https://gitee.com/mindspore/models/tree/master/utils/hccl_tools.>
-
-# 脚本说明
-
-## 脚本及样例代码
-
-```shell
-.
-└──Glore_resnet200
- ├── README.md
- ├── script
- ├── run_distribute_train.sh # 启动Ascend分布式训练（8卡）
- ├── run_distribute_train_gpu.sh # 启动GPU分布式训练（8卡）
- ├── run_eval.sh # 启动Ascend推理（单卡）
- ├── run_eval_gpu.sh # 启动GPU推理（单卡）
- ├── run_standalone_train.sh # 启动Ascend单机训练（单卡）
- └── run_standalone_train_gpu.sh # 启动GPU单机训练（单卡）
- ├── src
- ├── _init_.py
- ├── config.py #参数配置
- ├── dataset.py # 加载数据集
- ├── lr_generator.py # 学习率策略
- ├── glore_resnet200.py # glore_resnet200网络
- ├── transform.py # 数据增强
- └── transform_utils.py # 数据增强
- ├── eval.py # 推理脚本
- ├── export.py # 将checkpoint导出
- └── train.py # 训练脚本
-```
-
-## 脚本参数
-
-- 配置Glore_resnet200在ImageNet2012数据集参数(Ascend)。
-
-```text
-"class_num":1000, # 数据集类数
-"batch_size":80, # 输入张量的批次大小
-"loss_scale":1024, # 损失等级
-"momentum":0.08, # 动量优化器
-"weight_decay":0.0002, # 权重衰减
-"epoch_size":150, # 此值仅适用于训练；应用于推理时固定为1
-"pretrain_epoch_size":0, # 加载预训练检查点之前已经训练好的模型的周期大小；实际训练周期大小等于epoch_size减去pretrain_epoch_size
-"save_checkpoint":True, # 是否保存检查点
-"save_checkpoint_epochs":5, # 两个检查点之间的周期间隔；默认情况下，最后一个检查点将在最后一个周期完成后保存
-"keep_checkpoint_max":10, # 只保存最后一个keep_checkpoint_max检查点
-"save_checkpoint_path":"./", # 检查点相对于执行路径的保存路径
-"warmup_epochs":0, # 热身周期数
-"lr_decay_mode":"poly", # 用于生成学习率的衰减模式
-"lr_init":0.1, # 初始学习率
-"lr_max":0.4, # 最大学习率
-"lr_end":0.0, # 最小学习率
-```
-
-- 配置Glore_resnet200在ImageNet2012数据集参数(GPU)。
-
-```text
-"class_num":1000, # 数据集类数
-"batch_size":64, # 输入张量的批次大小
-"loss_scale":1024, # 损失等级
-"momentum":0.08, # 动量优化器
-"weight_decay":0.0002, # 权重衰减
-"epoch_size":150, # 此值仅适用于训练；应用于推理时固定为1
-"pretrain_epoch_size":0, # 加载预训练检查点之前已经训练好的模型的周期大小；实际训练周期大小等于epoch_size减去pretrain_epoch_size
-"save_checkpoint":True, # 是否保存检查点
-"save_checkpoint_epochs":5, # 两个检查点之间的周期间隔；默认情况下，最后一个检查点将在最后一个周期完成后保存
-"keep_checkpoint_max":10, # 只保存最后一个keep_checkpoint_max检查点
-"save_checkpoint_path":"./", # 检查点相对于执行路径的保存路径
-"warmup_epochs":0, # 热身周期数
-"lr_decay_mode":"poly", # 用于生成学习率的衰减模式
-"lr_init":0.1, # 初始学习率
-"lr_max":0.4, # 最大学习率
-"lr_end":0.0, # 最小学习率
-```
-
-更多配置细节请参考脚本`config.py`。
-
-## 训练过程
-
-### 用法
-
-#### Ascend处理器环境运行
-
-```text
-# 分布式训练
-用法:bash run_distribute_train.sh [DATASET_PATH] [RANK_TABLE]
-
-# 单机训练
-用法:bash run_standalone_train.sh [DATASET_PATH] [DEVICE_ID]
-
-# 运行推理示例
-用法:bash run_eval.sh [DATASET_PATH] [DEVICE_ID] [CHECKPOINT_PATH]
-```
-
-分布式训练需要提前创建JSON格式的HCCL配置文件。
-
-具体操作，参见[hccn_tools](https://gitee.com/mindspore/models/tree/master/utils/hccl_tools)中的说明。
-
-训练结果保存在示例路径中，文件夹名称以“train”或“train_parallel”开头。您可在此路径下的日志中找到检查点文件以及结果，如下所示。
-
-#### GPU处理器环境运行
-
-```text
-# 分布式训练
-用法:bash run_distribute_train_gpu.sh [RANK_SIZE] [DATASET_PATH]
-
-# 单机训练
-用法:bash run_standalone_train_gpu.sh [DATASET_PATH]
-
-# 运行推理示例
-用法:bash run_eval_gpu.sh [DATASET_PATH] [CHECKPOINT_PATH]
-```
-
-## 训练结果
-
-- 使用ImageNet2012数据集训练Glore_resnet200（8 pcs）
-
-```text
-# 分布式训练结果（8P）
-epoch:1 step:1251, loss is 6.0563216
-epoch:2 step:1251, loss is 5.3812423
-epoch:3 step:1251, loss is 4.782114
-epoch:4 step:1251, loss is 4.4079633
-epoch:5 step:1251, loss is 4.080069
-...
-```
-
-## 推理过程
-
-### 用法
-
-#### Ascend处理器环境运行
-
-```bash
-# 推理
-Usage: bash run_eval.sh [DATASET_PATH] [DEVICE_ID] [CHECKPOINT_PATH]
-```
-
-```bash
-# 推理示例
-bash run_eval.sh ~/Imagenet 0 ~/glore_resnet200-150_1251.ckpt
-```
-
-#### GPU处理器环境运行
-
-```bash
-# 推理
-Usage: bash run_eval_gpu.sh [DATASET_PATH] [CHECKPOINT_PATH]
-```
-
-```bash
-# 推理示例
-bash run_eval.sh ~/Imagenet ~/glore_resnet200-150_2502.ckpt
-```
-
-## 推理结果
-
-```text
-result:{'top_1 acc':0.802303685897436}
-```
-
-# 模型描述
-
-## 性能
-
-### 训练性能
-
-#### ImageNet2012上的Glore_resnet200
-
-| 参数 | Ascend 910 | GPU |
-| -------------------------- | -------------------------------------- |------------------------------------|
-| 模型版本 | Glore_resnet200 |Glore_resnet200 |
-| 资源 | Ascend 910；CPU：2.60GHz，192核；内存：2048G |GPU-V100(SXM2) |
-| 上传日期 | 2021-03-34 |2021-05-25 |
-| MindSpore版本 | 1.3.0 |1.2.0 |
-| 数据集 | ImageNet2012 | ImageNet2012 |
-| 训练参数 | epoch=150, steps per epoch=2001, batch_size = 80 |epoch=150, steps per epoch=2502, batch_size = 64 |
-| 优化器 | NAG | NAG |
-| 损失函数 | SoftmaxCrossEntropyExpand |SoftmaxCrossEntropyExpand |
-| 输出 | 概率 |概率 |
-| 损失 |0.8068262 |0.55614954 |
-| 速度 | 400.343毫秒/步（8卡） |912.211 毫秒/步（8卡） |
-| 总时长 | 33时35分钟 |94时08分 |
-| 参数(M) | 70.6 |70.6
-| 微调检查点| 807.57M（.ckpt文件） |808.28(.ckpt)
-| 脚本 | [链接](https://gitee.com/mindspore/models/tree/master/research/cv/glore_res200) |
-
-### 推理性能
-
-#### ImageNet2012上的Glore_resnet200
-
-| 参数 | Ascend | GPU
-| ------------------- | ----------------------|------------------------------|
-| 模型版本 | Glore_resnet200 | Glore_resnet200 |
-| 资源 | Ascend 910 | GPU |
-| 上传日期 | 2021-3-24 |2021-05-25 |
-| MindSpore版本 | 1.3.0 |1.2.0 |
-| 数据集 | 120万张图像 |120万张图像 |
-| batch_size | 80 |64 |
-| 输出 | 概率 |概率 |
-| 准确性 | 8卡: 80.23% |8卡：80.603% |
-
-# 随机情况说明
-
-transform_utils.py中使用数据增强时采用了随机选择策略，train.py中使用了随机种子。
-
-# ModelZoo主页
-
- 请浏览官网[主页](https://gitee.com/mindspore/models)
diff --git a/research/cv/glore_res200/__init__.py b/research/cv/glore_res200/__init__.py
deleted file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000
diff --git a/research/cv/glore_res200/ascend310_infer/CMakeLists.txt b/research/cv/glore_res200/ascend310_infer/CMakeLists.txt
deleted file mode 100644
index ee3c85447340e0449ff2b70ed24f60a17e07b2b6..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/ascend310_infer/CMakeLists.txt
+++ /dev/null
@@ -1,14 +0,0 @@
-cmake_minimum_required(VERSION 3.14.1)
-project(Ascend310Infer)
-add_compile_definitions(_GLIBCXX_USE_CXX11_ABI=0)
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O0 -g -std=c++17 -Werror -Wall -fPIE -Wl,--allow-shlib-undefined")
-set(PROJECT_SRC_ROOT ${CMAKE_CURRENT_LIST_DIR}/)
-option(MINDSPORE_PATH "mindspore install path" "")
-include_directories(${MINDSPORE_PATH})
-include_directories(${MINDSPORE_PATH}/include)
-include_directories(${PROJECT_SRC_ROOT})
-find_library(MS_LIB libmindspore.so ${MINDSPORE_PATH}/lib)
-file(GLOB_RECURSE MD_LIB ${MINDSPORE_PATH}/_c_dataengine*)
-
-add_executable(main src/main.cc src/utils.cc)
-target_link_libraries(main ${MS_LIB} ${MD_LIB} gflags)
diff --git a/research/cv/glore_res200/ascend310_infer/build.sh b/research/cv/glore_res200/ascend310_infer/build.sh
deleted file mode 100644
index 285514e19f2a1878a7bf8f0eed3c99fbc73868c4..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/ascend310_infer/build.sh
+++ /dev/null
@@ -1,29 +0,0 @@
-#!/bin/bash
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-if [ -d out ]; then
- rm -rf out
-fi
-
-mkdir out
-cd out || exit
-
-if [ -f "Makefile" ]; then
- make clean
-fi
-
-cmake .. \
- -DMINDSPORE_PATH="`pip3.7 show mindspore-ascend | grep Location | awk '{print $2"/mindspore"}' | xargs realpath`"
-make
diff --git a/research/cv/glore_res200/ascend310_infer/inc/utils.h b/research/cv/glore_res200/ascend310_infer/inc/utils.h
deleted file mode 100644
index f8ae1e5b473d869b77af8d725a280d7c7665527c..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/ascend310_infer/inc/utils.h
+++ /dev/null
@@ -1,35 +0,0 @@
-/**
- * Copyright 2021 Huawei Technologies Co., Ltd
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-#ifndef MINDSPORE_INFERENCE_UTILS_H_
-#define MINDSPORE_INFERENCE_UTILS_H_
-
-#include <sys/stat.h>
-#include <dirent.h>
-#include <vector>
-#include <string>
-#include <memory>
-#include "include/api/types.h"
-
-std::vector<std::string> GetAllFiles(std::string_view dirName);
-DIR *OpenDir(std::string_view dirName);
-std::string RealPath(std::string_view path);
-mindspore::MSTensor ReadFileToTensor(const std::string &file);
-int WriteResult(const std::string& imageFile, const std::vector<mindspore::MSTensor> &outputs);
-std::vector<std::string> GetAllFiles(std::string dir_name);
-std::vector<std::vector<std::string>> GetAllInputData(std::string dir_name);
-
-#endif
diff --git a/research/cv/glore_res200/ascend310_infer/src/main.cc b/research/cv/glore_res200/ascend310_infer/src/main.cc
deleted file mode 100644
index 1b1f2a91f01ef99b8dc63ef5d201fa3219e5d141..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/ascend310_infer/src/main.cc
+++ /dev/null
@@ -1,152 +0,0 @@
-/**
- * Copyright 2021 Huawei Technologies Co., Ltd
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-#include <sys/time.h>
-#include <gflags/gflags.h>
-#include <dirent.h>
-#include <iostream>
-#include <string>
-#include <algorithm>
-#include <iosfwd>
-#include <vector>
-#include <fstream>
-#include <sstream>
-
-#include "include/api/model.h"
-#include "include/api/context.h"
-#include "include/api/types.h"
-#include "include/api/serialization.h"
-#include "include/dataset/vision_ascend.h"
-#include "include/dataset/execute.h"
-#include "include/dataset/transforms.h"
-#include "include/dataset/vision.h"
-#include "inc/utils.h"
-
-using mindspore::Context;
-using mindspore::Serialization;
-using mindspore::Model;
-using mindspore::Status;
-using mindspore::ModelType;
-using mindspore::GraphCell;
-using mindspore::kSuccess;
-using mindspore::MSTensor;
-using mindspore::dataset::Execute;
-using mindspore::dataset::vision::Decode;
-using mindspore::dataset::vision::Resize;
-using mindspore::dataset::vision::CenterCrop;
-using mindspore::dataset::vision::Normalize;
-using mindspore::dataset::vision::HWC2CHW;
-
-
-DEFINE_string(mindir_path, "", "mindir path");
-DEFINE_string(dataset_name, "imagenet2012", "['cifar10', 'imagenet2012']");
-DEFINE_string(input0_path, ".", "input0 path");
-DEFINE_int32(device_id, 0, "device id");
-
-int load_model(Model *model, std::vector<MSTensor> *model_inputs, std::string mindir_path, int device_id) {
- if (RealPath(mindir_path).empty()) {
- std::cout << "Invalid mindir" << std::endl;
- return 1;
- }
-
- auto context = std::make_shared<Context>();
- auto ascend310 = std::make_shared<mindspore::Ascend310DeviceInfo>();
- ascend310->SetDeviceID(device_id);
- context->MutableDeviceInfo().push_back(ascend310);
- mindspore::Graph graph;
- Serialization::Load(mindir_path, ModelType::kMindIR, &graph);
-
- Status ret = model->Build(GraphCell(graph), context);
- if (ret != kSuccess) {
- std::cout << "ERROR: Build failed." << std::endl;
- return 1;
- }
-
- *model_inputs = model->GetInputs();
- if (model_inputs->empty()) {
- std::cout << "Invalid model, inputs is empty." << std::endl;
- return 1;
- }
- return 0;
-}
-
-int main(int argc, char **argv) {
- gflags::ParseCommandLineFlags(&argc, &argv, true);
-
- Model model;
- std::vector<MSTensor> model_inputs;
- load_model(&model, &model_inputs, FLAGS_mindir_path, FLAGS_device_id);
-
- std::map<double, double> costTime_map;
- struct timeval start = {0};
- struct timeval end = {0};
- double startTimeMs;
- double endTimeMs;
-
- auto input0_files = GetAllInputData(FLAGS_input0_path);
- if (input0_files.empty()) {
- std::cout << "ERROR: no input data." << std::endl;
- return 1;
- }
- size_t size = input0_files.size();
- for (size_t i = 0; i < size; ++i) {
- for (size_t j = 0; j < input0_files[i].size(); ++j) {
- std::vector<MSTensor> inputs;
- std::vector<MSTensor> outputs;
- std::cout << "Start predict input files:" << input0_files[i][j] <<std::endl;
- auto decode = Decode();
- auto resize = Resize({256, 256});
- auto centercrop = CenterCrop({224, 224});
- auto normalize = Normalize({123.675, 116.28, 103.53}, {58.395, 57.12, 57.375});
- auto hwc2chw = HWC2CHW();
-
- Execute SingleOp({decode, resize, centercrop, normalize, hwc2chw});
- auto imgDvpp = std::make_shared<MSTensor>();
- SingleOp(ReadFileToTensor(input0_files[i][j]), imgDvpp.get());
- inputs.emplace_back(model_inputs[0].Name(), model_inputs[0].DataType(), model_inputs[0].Shape(),
- imgDvpp->Data().get(), imgDvpp->DataSize());
- gettimeofday(&start, nullptr);
- Status ret = model.Predict(inputs, &outputs);
- gettimeofday(&end, nullptr);
- if (ret != kSuccess) {
- std::cout << "Predict " << input0_files[i][j] << " failed." << std::endl;
- return 1;
- }
- startTimeMs = (1.0 * start.tv_sec * 1000000 + start.tv_usec) / 1000;
- endTimeMs = (1.0 * end.tv_sec * 1000000 + end.tv_usec) / 1000;
- costTime_map.insert(std::pair<double, double>(startTimeMs, endTimeMs));
- WriteResult(input0_files[i][j], outputs);
- }
- }
- double average = 0.0;
- int inferCount = 0;
-
- for (auto iter = costTime_map.begin(); iter != costTime_map.end(); iter++) {
- double diff = 0.0;
- diff = iter->second - iter->first;
- average += diff;
- inferCount++;
- }
- average = average / inferCount;
- std::stringstream timeCost;
- timeCost << "NN inference cost average time: "<< average << " ms of infer_count " << inferCount << std::endl;
- std::cout << "NN inference cost average time: "<< average << "ms of infer_count " << inferCount << std::endl;
- std::string fileName = "./time_Result" + std::string("/test_perform_static.txt");
- std::ofstream fileStream(fileName.c_str(), std::ios::trunc);
- fileStream << timeCost.str();
- fileStream.close();
- costTime_map.clear();
- return 0;
-}
diff --git a/research/cv/glore_res200/ascend310_infer/src/utils.cc b/research/cv/glore_res200/ascend310_infer/src/utils.cc
deleted file mode 100644
index d71f388b83d23c2813d8bfc883dbcf2e7e0e4ef0..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/ascend310_infer/src/utils.cc
+++ /dev/null
@@ -1,185 +0,0 @@
-/**
- * Copyright 2021 Huawei Technologies Co., Ltd
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-#include <fstream>
-#include <algorithm>
-#include <iostream>
-#include "inc/utils.h"
-
-using mindspore::MSTensor;
-using mindspore::DataType;
-
-
-std::vector<std::vector<std::string>> GetAllInputData(std::string dir_name) {
- std::vector<std::vector<std::string>> ret;
-
- DIR *dir = OpenDir(dir_name);
- if (dir == nullptr) {
- return {};
- }
- struct dirent *filename;
- /* read all the files in the dir ~ */
- std::vector<std::string> sub_dirs;
- while ((filename = readdir(dir)) != nullptr) {
- std::string d_name = std::string(filename->d_name);
- // get rid of "." and ".."
- if (d_name == "." || d_name == ".." || d_name.empty()) {
- continue;
- }
- std::string dir_path = RealPath(std::string(dir_name) + "/" + filename->d_name);
- struct stat s;
- lstat(dir_path.c_str(), &s);
- if (!S_ISDIR(s.st_mode)) {
- continue;
- }
-
- sub_dirs.emplace_back(dir_path);
- }
- std::sort(sub_dirs.begin(), sub_dirs.end());
-
- (void)std::transform(sub_dirs.begin(), sub_dirs.end(), std::back_inserter(ret),
- [](const std::string &d) { return GetAllFiles(d); });
-
- return ret;
-}
-
-
-std::vector<std::string> GetAllFiles(std::string dir_name) {
- struct dirent *filename;
- DIR *dir = OpenDir(dir_name);
- if (dir == nullptr) {
- return {};
- }
-
- std::vector<std::string> res;
- while ((filename = readdir(dir)) != nullptr) {
- std::string d_name = std::string(filename->d_name);
- if (d_name == "." || d_name == ".." || d_name.size() <= 3) {
- continue;
- }
- res.emplace_back(std::string(dir_name) + "/" + filename->d_name);
- }
- std::sort(res.begin(), res.end());
-
- return res;
-}
-
-
-std::vector<std::string> GetAllFiles(std::string_view dirName) {
- struct dirent *filename;
- DIR *dir = OpenDir(dirName);
- if (dir == nullptr) {
- return {};
- }
- std::vector<std::string> res;
- while ((filename = readdir(dir)) != nullptr) {
- std::string dName = std::string(filename->d_name);
- if (dName == "." || dName == ".." || filename->d_type != DT_REG) {
- continue;
- }
- res.emplace_back(std::string(dirName) + "/" + filename->d_name);
- }
- std::sort(res.begin(), res.end());
- for (auto &f : res) {
- std::cout << "image file: " << f << std::endl;
- }
- return res;
-}
-
-
-int WriteResult(const std::string& imageFile, const std::vector<MSTensor> &outputs) {
- std::string homePath = "./result_Files";
- for (size_t i = 0; i < outputs.size(); ++i) {
- size_t outputSize;
- std::shared_ptr<const void> netOutput;
- netOutput = outputs[i].Data();
- outputSize = outputs[i].DataSize();
- int pos = imageFile.rfind('/');
- std::string fileName(imageFile, pos + 1);
- fileName.replace(fileName.find('.'), fileName.size() - fileName.find('.'), '_' + std::to_string(i) + ".bin");
- std::string outFileName = homePath + "/" + fileName;
- FILE *outputFile = fopen(outFileName.c_str(), "wb");
- fwrite(netOutput.get(), outputSize, sizeof(char), outputFile);
- fclose(outputFile);
- outputFile = nullptr;
- }
- return 0;
-}
-
-mindspore::MSTensor ReadFileToTensor(const std::string &file) {
- if (file.empty()) {
- std::cout << "Pointer file is nullptr" << std::endl;
- return mindspore::MSTensor();
- }
-
- std::ifstream ifs(file);
- if (!ifs.good()) {
- std::cout << "File: " << file << " is not exist" << std::endl;
- return mindspore::MSTensor();
- }
-
- if (!ifs.is_open()) {
- std::cout << "File: " << file << "open failed" << std::endl;
- return mindspore::MSTensor();
- }
-
- ifs.seekg(0, std::ios::end);
- size_t size = ifs.tellg();
- mindspore::MSTensor buffer(file, mindspore::DataType::kNumberTypeUInt8, {static_cast<int64_t>(size)}, nullptr, size);
-
- ifs.seekg(0, std::ios::beg);
- ifs.read(reinterpret_cast<char *>(buffer.MutableData()), size);
- ifs.close();
-
- return buffer;
-}
-
-
-DIR *OpenDir(std::string_view dirName) {
- if (dirName.empty()) {
- std::cout << " dirName is null ! " << std::endl;
- return nullptr;
- }
- std::string realPath = RealPath(dirName);
- struct stat s;
- lstat(realPath.c_str(), &s);
- if (!S_ISDIR(s.st_mode)) {
- std::cout << "dirName is not a valid directory !" << std::endl;
- return nullptr;
- }
- DIR *dir;
- dir = opendir(realPath.c_str());
- if (dir == nullptr) {
- std::cout << "Can not open dir " << dirName << std::endl;
- return nullptr;
- }
- std::cout << "Successfully opened the dir " << dirName << std::endl;
- return dir;
-}
-
-std::string RealPath(std::string_view path) {
- char realPathMem[PATH_MAX] = {0};
- char *realPathRet = nullptr;
- realPathRet = realpath(path.data(), realPathMem);
- if (realPathRet == nullptr) {
- std::cout << "File: " << path << " is not exist.";
- return "";
- }
-
- std::string realPath(realPathMem);
- std::cout << path << " realpath is: " << realPath << std::endl;
- return realPath;
-}
diff --git a/research/cv/glore_res200/eval.py b/research/cv/glore_res200/eval.py
deleted file mode 100644
index 528df69cd63a06dac71506fc6a1062d552428f42..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/eval.py
+++ /dev/null
@@ -1,88 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""
-##############test glore_resnet200 example on Imagenet2012#################
-python eval.py
-"""
-import random
-import argparse
-import ast
-import numpy as np
-from mindspore import context
-from mindspore import dataset as de
-from mindspore.train.model import Model
-from mindspore.train.serialization import load_checkpoint, load_param_into_net
-from src.config import config1, config2
-from src.glore_resnet200 import glore_resnet200
-from src.dataset import create_dataset_ImageNet as ImageNet
-from src.loss import SoftmaxCrossEntropyExpand
-
-parser = argparse.ArgumentParser(description='Image classification with glore_resnet50')
-parser.add_argument('--use_glore', type=ast.literal_eval, default=True, help='Enable GloreUnit')
-parser.add_argument('--data_url', type=str, default=None, help='Dataset path')
-parser.add_argument('--train_url', type=str, help='Train output in modelarts')
-parser.add_argument('--device_target', type=str, default='Ascend', help='Device target')
-parser.add_argument('--device_id', type=int, default=0)
-parser.add_argument('--ckpt_path', type=str, default=None)
-parser.add_argument('--isModelArts', type=ast.literal_eval, default=True)
-parser.add_argument('--parameter_server', type=ast.literal_eval, default=False, help='Run parameter server train')
-args_opt = parser.parse_args()
-
-if args_opt.isModelArts:
- import moxing as mox
-if args_opt.device_target == "Ascend":
- config = config1
-elif args_opt.device_target == "GPU":
- config = config2
-
-random.seed(1)
-np.random.seed(1)
-de.config.set_seed(1)
-
-
-if __name__ == '__main__':
- target = args_opt.device_target
- # init context
- device_id = args_opt.device_id
- context.set_context(mode=context.GRAPH_MODE, device_target=target, save_graphs=False,
- device_id=device_id)
-
- # dataset
- eval_dataset_path = args_opt.data_url
- if args_opt.isModelArts:
- mox.file.copy_parallel(src_url=args_opt.data_url, dst_url='/cache/dataset')
- eval_dataset_path = '/cache/dataset/'
- predict_data = ImageNet(dataset_path=eval_dataset_path,
- do_train=False,
- repeat_num=1,
- batch_size=config.batch_size,
- target=target)
- step_size = predict_data.get_dataset_size()
- if step_size == 0:
- raise ValueError("Please check dataset size > 0 and batch_size <= dataset size")
-
- # define net
- net = glore_resnet200(class_num=config.class_num, use_glore=args_opt.use_glore)
-
- # load checkpoint
- param_dict = load_checkpoint(args_opt.ckpt_path)
- load_param_into_net(net, param_dict)
-
- # define loss, model
- loss = SoftmaxCrossEntropyExpand(sparse=True)
- model = Model(net, loss_fn=loss, metrics={'top_1_accuracy', 'top_5_accuracy'})
- print("============== Starting Testing ==============")
- acc = model.eval(predict_data)
- print("==============Acc: {} ==============".format(acc))
diff --git a/research/cv/glore_res200/export.py b/research/cv/glore_res200/export.py
deleted file mode 100644
index ff82411ab0dbccb12001adf89be04465c56da776..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/export.py
+++ /dev/null
@@ -1,50 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""
-##############export checkpoint file into air, onnx, mindir models#################
-python export.py
-"""
-import argparse
-import numpy as np
-
-from mindspore import Tensor, load_checkpoint, load_param_into_net, export, context
-import mindspore.common.dtype as ms
-
-from src.config import config1 as config
-from src.glore_resnet200 import glore_resnet200
-
-parser = argparse.ArgumentParser(description='Classification')
-parser.add_argument("--device_id", type=int, default=0, help="Device id")
-parser.add_argument("--batch_size", type=int, default=1, help="batch size")
-parser.add_argument("--file_name", type=str, default="glore_resnet200", help="output file name.")
-parser.add_argument('--file_format', type=str, choices=["AIR", "ONNX", "MINDIR"], default='AIR', help='file format')
-parser.add_argument("--device_target", type=str, choices=["Ascend", "GPU", "CPU"], default="Ascend",
- help="device target")
-parser.add_argument("--ckpt_path", type=str, default=None)
-
-args = parser.parse_args()
-
-context.set_context(mode=context.GRAPH_MODE, device_target=args.device_target)
-if args.device_target == "Ascend":
- context.set_context(device_id=args.device_id)
-
-if __name__ == '__main__':
- net = glore_resnet200(class_num=config.class_num)
- assert args.ckpt_path is not None, "arg.ckpt_path is None."
- param_dict = load_checkpoint(args.ckpt_path)
- load_param_into_net(net, param_dict)
-
- input_arr = Tensor(np.ones([args.batch_size, 3, 224, 224]), ms.float32)
- export(net, input_arr, file_name=args.file_name, file_format=args.file_format)
diff --git a/research/cv/glore_res200/infer/convert/aipp.cfg b/research/cv/glore_res200/infer/convert/aipp.cfg
deleted file mode 100644
index 45ad83ca378eee7b0c076efec5013902c6daab81..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/infer/convert/aipp.cfg
+++ /dev/null
@@ -1,17 +0,0 @@
-aipp_op {
- aipp_mode: static
- input_format : RGB888_U8
-
- rbuv_swap_switch : true
-
- mean_chn_0 : 0
- mean_chn_1 : 0
- mean_chn_2 : 0
- min_chn_0 : 123.675
- min_chn_1 : 116.28
- min_chn_2 : 103.53
- var_reci_chn_0 : 0.0171247538316637
- var_reci_chn_1 : 0.0175070028011204
- var_reci_chn_2 : 0.0174291938997821
-}
-
diff --git a/research/cv/glore_res200/infer/convert/convert_om.sh b/research/cv/glore_res200/infer/convert/convert_om.sh
deleted file mode 100644
index d27c462efa2d95d60a6b00f7dc679edb61198974..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/infer/convert/convert_om.sh
+++ /dev/null
@@ -1,45 +0,0 @@
-#!/bin/bash
-
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-if [ $# -ne 3 ]
-then
- echo "Wrong parameter format."
- echo "Usage:"
- echo " bash $0 [INPUT_AIR_PATH] [AIPP_PATH] [OUTPUT_OM_PATH_NAME]"
- echo "Example: "
- echo " bash convert_om.sh xxx.air ./aipp.cfg xx"
-
- exit 1
-fi
-
-input_air_path=$1
-aipp_cfg_file=$2
-output_om_path=$3
-
-echo "Input AIR file path: ${input_air_path}"
-echo "Output OM file path: ${output_om_path}"
-
-atc --input_format=NCHW \
- --framework=1 \
- --model="${input_air_path}" \
- --input_shape="actual_input_1:1,3,224,224" \
- --output="${output_om_path}" \
- --insert_op_conf="${aipp_cfg_file}" \
- --enable_small_channel=1 \
- --log=error \
- --soc_version=Ascend310 \
- --op_select_implmode=high_precision
-
diff --git a/research/cv/glore_res200/infer/data/config/glore_res200.cfg b/research/cv/glore_res200/infer/data/config/glore_res200.cfg
deleted file mode 100644
index 581fc76d3d75445323ea9a387f7152a72bedd1d3..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/infer/data/config/glore_res200.cfg
+++ /dev/null
@@ -1,3 +0,0 @@
-CLASS_NUM=1000
-SOFTMAX=false
-TOP_K=5
diff --git a/research/cv/glore_res200/infer/data/config/glore_res200.pipeline b/research/cv/glore_res200/infer/data/config/glore_res200.pipeline
deleted file mode 100644
index c435e564c0e1899e6c3ff91c604d099a0e44aedb..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/infer/data/config/glore_res200.pipeline
+++ /dev/null
@@ -1,73 +0,0 @@
-{
- "glore_res200": {
- "stream_config": {
- "deviceId": "0"
- },
- "appsrc1": {
- "props": {
- "blocksize": "409600"
- },
- "factory": "appsrc",
- "next": "mxpi_imagedecoder0"
- },
- "mxpi_imagedecoder0": {
- "props": {
- "handleMethod": "opencv"
- },
- "factory": "mxpi_imagedecoder",
- "next": "mxpi_imageresize0"
- },
- "mxpi_imageresize0": {
- "props": {
- "handleMethod": "opencv",
- "resizeType": "Resizer_Stretch",
- "resizeHeight": "256",
- "resizeWidth": "256"
- },
- "factory": "mxpi_imageresize",
- "next": "mxpi_opencvcentercrop0"
- },
- "mxpi_opencvcentercrop0": {
- "props": {
- "dataSource": "mxpi_imageresize0",
- "cropHeight": "224",
- "cropWidth": "224"
- },
- "factory": "mxpi_opencvcentercrop",
- "next": "mxpi_tensorinfer0"
- },
- "mxpi_tensorinfer0": {
- "props": {
- "dataSource": "mxpi_opencvcentercrop0",
- "modelPath": "../model/glore_res200_1012.om",
- "waitingTime": "2000",
- "outputDeviceId": "-1"
- },
- "factory": "mxpi_tensorinfer",
- "next": "mxpi_classpostprocessor0"
- },
- "mxpi_classpostprocessor0": {
- "props": {
- "dataSource": "mxpi_tensorinfer0",
- "postProcessConfigPath": "./glore_res200.cfg",
- "labelPath": "../../labelset/imagenet1000_clsidx_to_labels.names",
- "postProcessLibPath": "libresnet50postprocess.so"
- },
- "factory": "mxpi_classpostprocessor",
- "next": "mxpi_dataserialize0"
- },
- "mxpi_dataserialize0": {
- "props": {
- "outputDataKeys": "mxpi_classpostprocessor0"
- },
- "factory": "mxpi_dataserialize",
- "next": "appsink0"
- },
- "appsink0": {
- "props": {
- "blocksize": "4096000"
- },
- "factory": "appsink"
- }
- }
-}
diff --git a/research/cv/glore_res200/infer/docker_start_infer.sh b/research/cv/glore_res200/infer/docker_start_infer.sh
deleted file mode 100644
index fa39d8c7f3b20132262a7e310acb82de9233ce1d..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/infer/docker_start_infer.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/usr/bin/env bash
-
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-docker_image=$1
-data_path=$2
-
-function show_help() {
- echo "Usage: docker_start.sh docker_image data_path"
-}
-
-function param_check() {
- if [ -z "${docker_image}" ]; then
- echo "please input docker_image"
- show_help
- exit 1
- fi
-
- if [ -z "${data_path}" ]; then
- echo "please input data_path"
- show_help
- exit 1
- fi
-}
-
-param_check
-
-docker run -it \
- --device=/dev/davinci0 \
- --device=/dev/davinci_manager \
- --device=/dev/devmm_svm \
- --device=/dev/hisi_hdc \
- -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
- -v ${data_path}:${data_path} \
- ${docker_image} \
- /bin/bash
\ No newline at end of file
diff --git a/research/cv/glore_res200/infer/mxbase/CMakeLists.txt b/research/cv/glore_res200/infer/mxbase/CMakeLists.txt
deleted file mode 100644
index c7f859497a06dc0cf9cb8e264cdbbd17d096ae9f..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/infer/mxbase/CMakeLists.txt
+++ /dev/null
@@ -1,47 +0,0 @@
-cmake_minimum_required(VERSION 3.14.0)
-project(glore_res200)
-set(TARGET glore_res200)
-
-add_definitions(-DENABLE_DVPP_INTERFACE)
-add_definitions(-D_GLIBCXX_USE_CXX11_ABI=0)
-add_definitions(-Dgoogle=mindxsdk_private)
-add_compile_options(-std=c++11 -fPIE -fstack-protector-all -fPIC -Wall)
-add_link_options(-Wl,-z,relro,-z,now,-z,noexecstack -s -pie)
-
-# Check environment variable
-if(NOT DEFINED ENV{ASCEND_HOME})
- message(FATAL_ERROR "please define environment variable:ASCEND_HOME")
-endif()
-if(NOT DEFINED ENV{ASCEND_VERSION})
- message(WARNING "please define environment variable:ASCEND_VERSION")
-endif()
-if(NOT DEFINED ENV{ARCH_PATTERN})
- message(WARNING "please define environment variable:ARCH_PATTERN")
-endif()
-set(ACL_INC_DIR $ENV{ASCEND_HOME}/$ENV{ASCEND_VERSION}/$ENV{ARCH_PATTERN}/acllib/include)
-set(ACL_LIB_DIR $ENV{ASCEND_HOME}/$ENV{ASCEND_VERSION}/$ENV{ARCH_PATTERN}/acllib/lib64)
-
-set(MXBASE_ROOT_DIR $ENV{MX_SDK_HOME})
-set(MXBASE_INC ${MXBASE_ROOT_DIR}/include)
-set(MXBASE_LIB_DIR ${MXBASE_ROOT_DIR}/lib)
-set(MXBASE_POST_LIB_DIR ${MXBASE_ROOT_DIR}/lib/modelpostprocessors)
-set(MXBASE_POST_PROCESS_DIR ${MXBASE_ROOT_DIR}/include/MxBase/postprocess/include)
-set(OPENSOURCE_DIR ${MXBASE_ROOT_DIR}/opensource)
-
-include_directories(${ACL_INC_DIR})
-include_directories(${OPENSOURCE_DIR}/include)
-include_directories(${OPENSOURCE_DIR}/include/opencv4)
-
-include_directories(${MXBASE_INC})
-include_directories(${MXBASE_POST_PROCESS_DIR})
-
-link_directories(${ACL_LIB_DIR})
-link_directories(${OPENSOURCE_DIR}/lib)
-link_directories(${MXBASE_LIB_DIR})
-link_directories(${MXBASE_POST_LIB_DIR})
-
-add_executable(${TARGET} src/main.cpp src/GloreRes200ClassifyOpencv.cpp)
-
-target_link_libraries(${TARGET} glog cpprest mxbase resnet50postprocess opencv_world stdc++fs)
-
-install(TARGETS ${TARGET} RUNTIME DESTINATION ${PROJECT_SOURCE_DIR}/)
diff --git a/research/cv/glore_res200/infer/mxbase/build.sh b/research/cv/glore_res200/infer/mxbase/build.sh
deleted file mode 100644
index 8eadc9593de1607482289d5d202a945598b4a23c..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/infer/mxbase/build.sh
+++ /dev/null
@@ -1,55 +0,0 @@
-#!/bin/bash
-
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-path_cur=$(dirname $0)
-
-function check_env()
-{
- # set ASCEND_VERSION to ascend-toolkit/latest when it was not specified by user
- if [ ! "${ASCEND_VERSION}" ]; then
- export ASCEND_VERSION=nnrt/latest
- echo "Set ASCEND_VERSION to the default value: ${ASCEND_VERSION}"
- else
- echo "ASCEND_VERSION is set to ${ASCEND_VERSION} by user"
- fi
-
- if [ ! "${ARCH_PATTERN}" ]; then
- # set ARCH_PATTERN to ./ when it was not specified by user
- export ARCH_PATTERN=.
- echo "ARCH_PATTERN is set to the default value: ${ARCH_PATTERN}"
- else
- echo "ARCH_PATTERN is set to ${ARCH_PATTERN} by user"
- fi
-}
-
-function build_glore_res50()
-{
- cd $path_cur
- rm -rf build
- mkdir -p build
- cd build
- cmake ..
- make
- ret=$?
- if [ ${ret} -ne 0 ]; then
- echo "Failed to build glore_res200."
- exit ${ret}
- fi
- make install
-}
-
-check_env
-build_glore_res50
diff --git a/research/cv/glore_res200/infer/mxbase/src/GloreRes200ClassifyOpencv.cpp b/research/cv/glore_res200/infer/mxbase/src/GloreRes200ClassifyOpencv.cpp
deleted file mode 100644
index 4f3ed81f6a5fa269a6c0a9e264a9d097efb1b83e..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/infer/mxbase/src/GloreRes200ClassifyOpencv.cpp
+++ /dev/null
@@ -1,219 +0,0 @@
-/*
- * Copyright (c) 2021. Huawei Technologies Co., Ltd. All rights reserved.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-#include "GloreRes200ClassifyOpencv.h"
-#include "MxBase/DeviceManager/DeviceManager.h"
-#include "MxBase/DvppWrapper/DvppWrapper.h"
-#include "MxBase/Log/Log.h"
-
-using MxBase::DeviceManager;
-using MxBase::TensorBase;
-using MxBase::MemoryData;
-using MxBase::ClassInfo;
-
-APP_ERROR GloreRes200ClassifyOpencv::Init(const InitParam &initParam) {
- deviceId_ = initParam.deviceId;
- APP_ERROR ret = DeviceManager::GetInstance()->InitDevices();
- if (ret != APP_ERR_OK) {
- LogError << "Init devices failed, ret=" << ret << ".";
- return ret;
- }
- ret = MxBase::TensorContext::GetInstance()->SetContext(initParam.deviceId);
- if (ret != APP_ERR_OK) {
- LogError << "Set context failed, ret=" << ret << ".";
- return ret;
- }
- model_ = std::make_shared<MxBase::ModelInferenceProcessor>();
- ret = model_->Init(initParam.modelPath, modelDesc_);
- if (ret != APP_ERR_OK) {
- LogError << "ModelInferenceProcessor init failed, ret=" << ret << ".";
- return ret;
- }
- MxBase::ConfigData configData;
- const std::string softmax = initParam.softmax ? "true" : "false";
- const std::string checkTensor = initParam.checkTensor ? "true" : "false";
-
- configData.SetJsonValue("CLASS_NUM", std::to_string(initParam.classNum));
- configData.SetJsonValue("TOP_K", std::to_string(initParam.topk));
- configData.SetJsonValue("SOFTMAX", softmax);
- configData.SetJsonValue("CHECK_MODEL", checkTensor);
-
- auto jsonStr = configData.GetCfgJson().serialize();
- std::map<std::string, std::shared_ptr<void>> config;
- config["postProcessConfigContent"] = std::make_shared<std::string>(jsonStr);
- config["labelPath"] = std::make_shared<std::string>(initParam.labelPath);
-
- post_ = std::make_shared<MxBase::Resnet50PostProcess>();
- ret = post_->Init(config);
- if (ret != APP_ERR_OK) {
- LogError << "Resnet50PostProcess init failed, ret=" << ret << ".";
- return ret;
- }
- return APP_ERR_OK;
-}
-
-APP_ERROR GloreRes200ClassifyOpencv::DeInit() {
- model_->DeInit();
- post_->DeInit();
- MxBase::DeviceManager::GetInstance()->DestroyDevices();
- return APP_ERR_OK;
-}
-
-void GloreRes200ClassifyOpencv::ReadImage(const std::string &imgPath, cv::Mat *imageMat) {
- *imageMat = cv::imread(imgPath, cv::IMREAD_COLOR);
-}
-
-void GloreRes200ClassifyOpencv::ResizeImage(const cv::Mat &srcImageMat, cv::Mat *dstImageMat) {
- static constexpr uint32_t resizeHeight = 256;
- static constexpr uint32_t resizeWidth = 256;
-
- cv::resize(srcImageMat, *dstImageMat, cv::Size(resizeWidth, resizeHeight));
-}
-
-void GloreRes200ClassifyOpencv::CenterCropImage(const cv::Mat &img, cv::Mat *crop_im) {
- static cv::Rect rectOfImg(16, 16, 224, 224);
- *crop_im = img(rectOfImg).clone();
-}
-
-APP_ERROR GloreRes200ClassifyOpencv::CVMatToTensorBase(const cv::Mat &imageMat, TensorBase *tensorBase) {
- const uint32_t dataSize = imageMat.cols * imageMat.rows * MxBase::YUV444_RGB_WIDTH_NU;
- LogInfo << "image size after crop" << imageMat.cols << " " << imageMat.rows;
- MemoryData memoryDataDst(dataSize, MemoryData::MEMORY_DEVICE, deviceId_);
- MemoryData memoryDataSrc(imageMat.data, dataSize, MemoryData::MEMORY_HOST_MALLOC);
-
- APP_ERROR ret = MxBase::MemoryHelper::MxbsMallocAndCopy(memoryDataDst, memoryDataSrc);
- if (ret != APP_ERR_OK) {
- LogError << GetError(ret) << "Memory malloc failed.";
- return ret;
- }
-
- std::vector<uint32_t> shape = {imageMat.rows * MxBase::YUV444_RGB_WIDTH_NU, static_cast<uint32_t>(imageMat.cols)};
- *tensorBase = TensorBase(memoryDataDst, false, shape, MxBase::TENSOR_DTYPE_UINT8);
- return APP_ERR_OK;
-}
-
-APP_ERROR GloreRes200ClassifyOpencv::Inference(const std::vector<TensorBase> &inputs,
- std::vector<TensorBase> *outputs) {
- auto dtypes = model_->GetOutputDataType();
- for (size_t i = 0; i < modelDesc_.outputTensors.size(); ++i) {
- std::vector<uint32_t> shape = {};
- for (size_t j = 0; j < modelDesc_.outputTensors[i].tensorDims.size(); ++j) {
- shape.push_back((uint32_t)modelDesc_.outputTensors[i].tensorDims[j]);
- }
- TensorBase tensor(shape, dtypes[i], MemoryData::MemoryType::MEMORY_DEVICE, deviceId_);
- APP_ERROR ret = TensorBase::TensorBaseMalloc(tensor);
- if (ret != APP_ERR_OK) {
- LogError << "TensorBaseMalloc failed, ret=" << ret << ".";
- return ret;
- }
- outputs->push_back(tensor);
- }
- MxBase::DynamicInfo dynamicInfo = {};
- dynamicInfo.dynamicType = MxBase::DynamicType::STATIC_BATCH;
- auto startTime = std::chrono::high_resolution_clock::now();
- APP_ERROR ret = model_->ModelInference(inputs, *outputs, dynamicInfo);
- auto endTime = std::chrono::high_resolution_clock::now();
- double costMs = std::chrono::duration<double, std::milli>(endTime - startTime).count(); // save time
- inferCostTimeMilliSec += costMs;
- if (ret != APP_ERR_OK) {
- LogError << "ModelInference failed, ret=" << ret << ".";
- return ret;
- }
- return APP_ERR_OK;
-}
-
-APP_ERROR GloreRes200ClassifyOpencv::PostProcess(const std::vector<TensorBase> &inputs,
- std::vector<std::vector<ClassInfo>> *clsInfos) {
- APP_ERROR ret = post_->Process(inputs, *clsInfos);
- if (ret != APP_ERR_OK) {
- LogError << "Process failed, ret=" << ret << ".";
- return ret;
- }
- return APP_ERR_OK;
-}
-
-APP_ERROR GloreRes200ClassifyOpencv::SaveResult(const std::string &imgPath,
- const std::vector<std::vector<ClassInfo>> &batchClsInfos) {
- LogInfo << "image path" << imgPath;
- std::string fileName = imgPath.substr(imgPath.find_last_of("/") + 1);
- size_t dot = fileName.find_last_of(".");
- std::string resFileName = "result/" + fileName.substr(0, dot) + "_1.txt";
- LogInfo << "file path for saving result" << resFileName;
-
- std::ofstream outfile(resFileName);
- if (outfile.fail()) {
- LogError << "Failed to open result file: ";
- return APP_ERR_COMM_FAILURE;
- }
-
- uint32_t batchIndex = 0;
- for (auto clsInfos : batchClsInfos) {
- std::string resultStr;
- for (auto clsInfo : clsInfos) {
- LogDebug << " className:" << clsInfo.className << " confidence:" << clsInfo.confidence <<
- " classIndex:" << clsInfo.classId;
- resultStr += std::to_string(clsInfo.classId) + " ";
- }
-
- outfile << resultStr << std::endl;
- batchIndex++;
- }
- outfile.close();
- return APP_ERR_OK;
-}
-
-APP_ERROR GloreRes200ClassifyOpencv::Process(const std::string &imgPath) {
- cv::Mat imageMat;
- ReadImage(imgPath, &imageMat);
-
- cv::Mat resizeImg;
- ResizeImage(imageMat, &resizeImg);
-
- cv::Mat cropImg;
- CenterCropImage(resizeImg, &cropImg);
-
- std::vector<TensorBase> inputs = {};
- std::vector<TensorBase> outputs = {};
-
- TensorBase tensorBase;
- APP_ERROR ret = CVMatToTensorBase(cropImg, &tensorBase);
- if (ret != APP_ERR_OK) {
- LogError << "CVMatToTensorBase failed, ret=" << ret << ".";
- return ret;
- }
- inputs.push_back(tensorBase);
-
-
- ret = Inference(inputs, &outputs);
- if (ret != APP_ERR_OK) {
- LogError << "Inference failed, ret=" << ret << ".";
- return ret;
- }
- std::vector<std::vector<ClassInfo>> batchClsInfos;
- ret = PostProcess(outputs, &batchClsInfos);
- if (ret != APP_ERR_OK) {
- LogError << "PostProcess failed, ret=" << ret << ".";
- return ret;
- }
-
- ret = SaveResult(imgPath, batchClsInfos);
- if (ret != APP_ERR_OK) {
- LogError << "Save infer results into file failed. ret = " << ret << ".";
- return ret;
- }
-
- return APP_ERR_OK;
-}
diff --git a/research/cv/glore_res200/infer/mxbase/src/GloreRes200ClassifyOpencv.h b/research/cv/glore_res200/infer/mxbase/src/GloreRes200ClassifyOpencv.h
deleted file mode 100644
index 35ad837099770dcf46c188c3412857fe72c552dc..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/infer/mxbase/src/GloreRes200ClassifyOpencv.h
+++ /dev/null
@@ -1,67 +0,0 @@
-/*
- * Copyright (c) 2021. Huawei Technologies Co., Ltd. All rights reserved.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-#ifndef MXBASE_GLORERES50CLASSIFYOPENCV_H
-#define MXBASE_GLORERES50CLASSIFYOPENCV_H
-
-#include <map>
-#include <memory>
-#include <string>
-#include <vector>
-#include <opencv2/opencv.hpp>
-#include "MxBase/ModelInfer/ModelInferenceProcessor.h"
-#include "ClassPostProcessors/Resnet50PostProcess.h"
-#include "MxBase/Tensor/TensorContext/TensorContext.h"
-
-struct InitParam {
- uint32_t deviceId;
- std::string labelPath;
- uint32_t classNum;
- uint32_t topk;
- bool softmax;
- bool checkTensor;
- std::string modelPath;
-};
-
-class GloreRes200ClassifyOpencv {
- public:
- APP_ERROR Init(const InitParam &initParam);
- APP_ERROR DeInit();
- void ReadImage(const std::string &imgPath, cv::Mat *imageMat);
- void ResizeImage(const cv::Mat &srcImageMat, cv::Mat *dstImageMat);
- void CenterCropImage(const cv::Mat &img, cv::Mat *crop_im);
- APP_ERROR CVMatToTensorBase(const cv::Mat &imageMat, MxBase::TensorBase *tensorBase);
- APP_ERROR Inference(const std::vector<MxBase::TensorBase> &inputs, std::vector<MxBase::TensorBase> *outputs);
- APP_ERROR PostProcess(const std::vector<MxBase::TensorBase> &inputs,
- std::vector<std::vector<MxBase::ClassInfo>> *clsInfos);
- APP_ERROR Process(const std::string &imgPath);
- // get infer time
- double GetInferCostMillisec() const {return inferCostTimeMilliSec;}
-
- private:
- APP_ERROR SaveResult(const std::string &imgPath,
- const std::vector<std::vector<MxBase::ClassInfo>> &batchClsInfos);
-
- private:
- std::shared_ptr<MxBase::ModelInferenceProcessor> model_;
- std::shared_ptr<MxBase::Resnet50PostProcess> post_;
- MxBase::ModelDesc modelDesc_;
- uint32_t deviceId_ = 0;
- // infer time
- double inferCostTimeMilliSec = 0.0;
-};
-
-#endif
diff --git a/research/cv/glore_res200/infer/mxbase/src/main.cpp b/research/cv/glore_res200/infer/mxbase/src/main.cpp
deleted file mode 100644
index cf3df11e9e79e9fdae35641d3d46aa11092b9dc4..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/infer/mxbase/src/main.cpp
+++ /dev/null
@@ -1,86 +0,0 @@
-/*
- * Copyright (c) 2021. Huawei Technologies Co., Ltd. All rights reserved.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-#include <dirent.h>
-#include "GloreRes200ClassifyOpencv.h"
-#include "MxBase/Log/Log.h"
-
-namespace {
-const uint32_t CLASS_NUM = 1000;
-} // namespace
-
-APP_ERROR scan_images(const std::string &path, std::vector<std::string> *imgFiles) {
- DIR *dirPtr = opendir(path.c_str());
- if (dirPtr == nullptr) {
- LogError << "opendir failed. dir:" << path;
- return APP_ERR_INTERNAL_ERROR;
- }
- dirent *direntPtr = nullptr;
- while ((direntPtr = readdir(dirPtr)) != nullptr) {
- std::string fileName = direntPtr->d_name;
- if (fileName == "." || fileName == "..") {
- continue;
- }
-
- imgFiles->emplace_back(path + "/" + fileName);
- }
- closedir(dirPtr);
- return APP_ERR_OK;
-}
-
-int main(int argc, char* argv[]) {
- if (argc <= 1) {
- LogWarn << "Please input image path, such as './gloreres50 image_dir'.";
- return APP_ERR_OK;
- }
-
- InitParam initParam = {};
- initParam.deviceId = 0;
- initParam.classNum = CLASS_NUM;
- initParam.labelPath = "../../labelset/imagenet1000_clsidx_to_labels.names";
- initParam.topk = 5;
- initParam.softmax = false;
- initParam.checkTensor = true;
- initParam.modelPath = "../model/glore_res200_1012.om";
- auto glore_res200 = std::make_shared<GloreRes200ClassifyOpencv>();
- APP_ERROR ret = glore_res200->Init(initParam);
- if (ret != APP_ERR_OK) {
- LogError << "GloreRes200Classify init failed, ret=" << ret << ".";
- return ret;
- }
-
- std::string imgPath = argv[1];
- std::vector<std::string> imgFilePaths;
- ret = scan_images(imgPath, &imgFilePaths);
- if (ret != APP_ERR_OK) {
- return ret;
- }
- auto startTime = std::chrono::high_resolution_clock::now();
- for (auto &imgFile : imgFilePaths) {
- ret = glore_res200->Process(imgFile);
- if (ret != APP_ERR_OK) {
- LogError << "GloreRes200Classify process failed, ret=" << ret << ".";
- glore_res200->DeInit();
- return ret;
- }
- }
- auto endTime = std::chrono::high_resolution_clock::now();
- glore_res200->DeInit();
- double costMilliSecs = std::chrono::duration<double, std::milli>(endTime - startTime).count();
- double fps = 1000.0 * imgFilePaths.size() / glore_res200->GetInferCostMillisec();
- LogInfo << "[Process Delay] cost: " << costMilliSecs << " ms\tfps: " << fps << " imgs/sec";
- return APP_ERR_OK;
-}
diff --git a/research/cv/glore_res200/infer/sdk/main.py b/research/cv/glore_res200/infer/sdk/main.py
deleted file mode 100644
index f43c9246a3bb61931543e3e7f13e017ec1cb78aa..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/infer/sdk/main.py
+++ /dev/null
@@ -1,93 +0,0 @@
-# coding=utf-8
-
-"""
-Copyright 2021 Huawei Technologies Co., Ltd
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-"""
-
-import datetime
-import json
-import os
-import sys
-
-from StreamManagerApi import StreamManagerApi
-from StreamManagerApi import MxDataInput
-
-
-if __name__ == '__main__':
- # init stream manager
- stream_manager_api = StreamManagerApi()
- ret = stream_manager_api.InitManager()
- if ret != 0:
- print("Failed to init Stream manager, ret=%s" % str(ret))
- exit()
-
- # create streams by pipeline config file
- with open("./glore_res200.pipeline", 'rb') as f:
- pipelineStr = f.read()
- ret = stream_manager_api.CreateMultipleStreams(pipelineStr)
-
- if ret != 0:
- print("Failed to create Stream, ret=%s" % str(ret))
- exit()
-
- # Construct the input of the stream
- data_input = MxDataInput()
-
- dir_name = sys.argv[1]
- res_dir_name = sys.argv[2]
- file_list = os.listdir(dir_name)
- if not os.path.exists(res_dir_name):
- os.makedirs(res_dir_name)
-
- for file_name in file_list:
- file_path = os.path.join(dir_name, file_name)
- if not (file_name.lower().endswith(".jpg") or file_name.lower().endswith(".jpeg")):
- continue
-
- with open(file_path, 'rb') as f:
- data_input.data = f.read()
- empty_data = []
- stream_name = b'glore_res200'
- in_plugin_id = 0
- unique_id = stream_manager_api.SendData(stream_name, in_plugin_id, data_input)
- if unique_id < 0:
- print("Failed to send data to stream.")
- exit()
- # Obtain the inference result by specifying streamName and uniqueId.
- start_time = datetime.datetime.now()
- infer_result = stream_manager_api.GetResult(stream_name, unique_id)
- end_time = datetime.datetime.now()
- print('sdk run time: {}'.format((end_time - start_time).microseconds))
- if infer_result.errorCode != 0:
- print("GetResultWithUniqueId error. errorCode=%d, errorMsg=%s" % (
- infer_result.errorCode, infer_result.data.decode()))
- exit()
- # print the infer result
- infer_res = infer_result.data.decode()
- print("process img: {}, infer result: {}".format(file_name, infer_res))
- load_dict = json.loads(infer_result.data.decode())
- if load_dict.get('MxpiClass') is None:
- with open(res_dir_name + "/" + file_name[:-5] + '.txt', 'w') as f_write:
- f_write.write("")
- continue
- res_vec = load_dict.get('MxpiClass')
-
- with open(res_dir_name + "/" + file_name[:-5] + '_1.txt', 'w') as f_write:
- res_list = [str(item.get("classId")) + " " for item in res_vec]
- f_write.writelines(res_list)
- f_write.write('\n')
-
- # destroy streams
- stream_manager_api.DestroyAllStreams()
diff --git a/research/cv/glore_res200/infer/sdk/run.sh b/research/cv/glore_res200/infer/sdk/run.sh
deleted file mode 100644
index 157dad8c91a37c4817b2be98e874549b24e2a68a..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/infer/sdk/run.sh
+++ /dev/null
@@ -1,35 +0,0 @@
-#!/bin/bash
-
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-image_path=$1
-result_dir=$2
-
-set -e
-
-#CUR_PATH=$(cd "$(dirname "$0")" || { warn "Failed to check path/to/run.sh" ; exit ; } ; pwd)
-
-# Simple log helper functions
-info() { echo -e "\033[1;34m[INFO ][MxStream] $1\033[1;37m" ; }
-warn() { echo >&2 -e "\033[1;31m[WARN ][MxStream] $1\033[1;37m" ; }
-
-export LD_LIBRARY_PATH=${MX_SDK_HOME}/lib:${MX_SDK_HOME}/opensource/lib:${MX_SDK_HOME}/opensource/lib64:/usr/local/Ascend/ascend-toolkit/latest/acllib/lib64:${LD_LIBRARY_PATH}
-export GST_PLUGIN_SCANNER=${MX_SDK_HOME}/opensource/libexec/gstreamer-1.0/gst-plugin-scanner
-export GST_PLUGIN_PATH=${MX_SDK_HOME}/opensource/lib/gstreamer-1.0:${MX_SDK_HOME}/lib/plugins
-
-#to set PYTHONPATH, import the StreamManagerApi.py
-export PYTHONPATH=$PYTHONPATH:${MX_SDK_HOME}/python
-
-python3.7 main.py $image_path $result_dir
-exit 0
diff --git a/research/cv/glore_res200/infer/util/eval_by_sdk.py b/research/cv/glore_res200/infer/util/eval_by_sdk.py
deleted file mode 100644
index a4fbfda29a90deb84ad6492ca063e67dc0e65cfc..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/infer/util/eval_by_sdk.py
+++ /dev/null
@@ -1,172 +0,0 @@
-# coding = utf-8
-"""
-Copyright 2021 Huawei Technologies Co., Ltd
-
-Licensed under the BSD 3-Clause License (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-https://opensource.org/licenses/BSD-3-Clause
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-"""
-
-
-import os
-import sys
-import json
-import numpy as np
-
-np.set_printoptions(threshold=sys.maxsize)
-
-def gen_file_name(img_name):
- """
- :param filename: image file name
- :return: a list contains image name and ext
- """
- full_name = img_name.split('/')[-1]
- return os.path.splitext(full_name)
-
-
-def cre_groundtruth_dict_fromjson(gtfile_path):
- """
- :param filename: json file contains the imagename and label number
- :return: dictionary key imagename, value is label number
- """
- img_gt_dict = {}
- with open(gtfile_path, 'r') as f:
- gt = json.load(f)
- for key, value in gt.items():
- img_gt_dict[gen_file_name(key)[0]] = value
- return img_gt_dict
-
-
-def cre_groundtruth_dict_fromtxt(gtfile_path):
- """
- :param filename: file contains the imagename and label number
- :return: dictionary key imagename, value is label number
- """
- img_gt_dict = {}
- with open(gtfile_path, 'r')as f:
- for line in f.readlines():
- temp = line.strip().split(" ")
- img_name = temp[0].split(".")[0]
- img_lab = temp[1]
- img_gt_dict[img_name] = img_lab
- return img_gt_dict
-
-
-def load_statistical_predict_result(filepath):
- """
- function:
- the prediction esult file data extraction
- input:
- result file:filepath
- output:
- n_label:numble of label
- data_vec: the probabilitie of prediction in the 1000
- :return: probabilities, numble of label
- """
- with open(filepath, 'r')as f:
- data = f.readline()
- temp = data.strip().split(" ")
- n_label = len(temp)
- data_vec = np.zeros((n_label), dtype=np.float32)
- if n_label != 0:
- for ind, cls_ind in enumerate(temp):
- data_vec[ind] = np.int64(cls_ind)
- return data_vec, n_label
-
-
-def create_visualization_statistical_result(prediction_file_path,
- result_store_path, json_file_path,
- img_gt_dict, topn=5):
- """
- :param prediction_file_path:
- :param result_store_path:
- :param json_file_name:
- :param img_gt_dict:
- :param topn:
- :return:
- """
- writer = open(os.path.join(result_store_path, json_file_path), 'w')
- table_dict = {}
- table_dict["title"] = "Overall statistical evaluation"
- table_dict["value"] = []
-
- count = 0
- res_cnt = 0
- n_labels = ""
- count_hit = np.zeros(topn)
- for tfile_name in os.listdir(prediction_file_path):
- count += 1
- img_name = tfile_name.split('.')[0][:-2]
- filepath = os.path.join(prediction_file_path, tfile_name)
-
- ret = load_statistical_predict_result(filepath)
- prediction = ret[0]
- n_labels = ret[1]
-
- gt = img_gt_dict[img_name]
- if n_labels == 1000:
- real_label = int(gt)
- elif n_labels == 1001:
- real_label = int(gt) + 1
- else:
- real_label = int(gt)
-
- res_cnt = min(len(prediction), topn)
- for i in range(res_cnt):
- if str(real_label) == str(int(prediction[i])):
- count_hit[i] += 1
- break
- if 'value' not in table_dict.keys():
- print("the item value does not exist!")
- else:
- table_dict["value"].extend(
- [{"key": "Number of images", "value": str(count)},
- {"key": "Number of classes", "value": str(n_labels)}])
- if count == 0:
- accuracy = 0
- else:
- accuracy = np.cumsum(count_hit) / count
- for i in range(res_cnt):
- table_dict["value"].append({"key": "Top" + str(i + 1) + " accuracy",
- "value": str(
- round(accuracy[i] * 100, 2)) + '%'})
- json.dump(table_dict, writer)
- writer.close()
-
-
-if __name__ == '__main__':
- try:
- # prediction txt files path
- folder_davinci_target = sys.argv[1]
- # annotation files path, "imagenet2012_label.json"
- annotation_file_path = sys.argv[2]
- # the path to store the results json path
- result_json_path = sys.argv[3]
- # result json file name
- json_file_name = sys.argv[4]
- except IndexError:
- print("Please enter prediction files folder, ground truth label json file, result json file folder, "
- "result json file name\n")
- print("\teg: python3.7 eval_by_sdk.py ./preds ../data/config/imagenet_label.json ./result result.json")
- exit(1)
-
- if not os.path.exists(folder_davinci_target):
- print("Target file folder does not exist.")
-
- if not os.path.exists(annotation_file_path):
- print("Ground truth file does not exist.")
-
- if not os.path.exists(result_json_path):
- print("Result folder doesn't exist.")
-
- img_label_dict = cre_groundtruth_dict_fromjson(annotation_file_path)
- create_visualization_statistical_result(folder_davinci_target, result_json_path,
- json_file_name, img_label_dict, topn=5)
diff --git a/research/cv/glore_res200/modelart/start.py b/research/cv/glore_res200/modelart/start.py
deleted file mode 100644
index 78f8e519ceeb4ac5c6f57508908c4f8fc5f74bdb..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/modelart/start.py
+++ /dev/null
@@ -1,260 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""
-#################train glore_resnet200 on Imagenet2012########################
-python train.py
-"""
-import os
-import glob
-import random
-import argparse
-import ast
-import numpy as np
-from mindspore import Tensor
-from mindspore import context
-from mindspore import export
-from mindspore import dataset as de
-from mindspore.context import ParallelMode
-from mindspore.train.model import Model
-from mindspore.train.callback import ModelCheckpoint, CheckpointConfig, LossMonitor, TimeMonitor
-from mindspore.train.loss_scale_manager import FixedLossScaleManager
-from mindspore.train.serialization import load_checkpoint, load_param_into_net
-from mindspore.communication.management import init, get_group_size, get_rank
-import mindspore.nn as nn
-import mindspore.common.dtype as mstype
-from mindspore.common import set_seed
-import mindspore.common.initializer as weight_init
-from glore_res200.src.lr_generator import get_lr
-from glore_res200.src.config import config1, config2
-from glore_res200.src.glore_resnet200 import glore_resnet200
-from glore_res200.src.dataset import create_dataset_ImageNet as get_dataset
-from glore_res200.src.loss import SoftmaxCrossEntropyExpand
-
-
-parser = argparse.ArgumentParser(
- description='Image classification with glore_resnet200')
-parser.add_argument('--use_glore', type=ast.literal_eval,
- default=True, help='Enable GloreUnit')
-parser.add_argument('--cast_fp16', type=ast.literal_eval,
- default=True, help='GloreUnit cast to fp16 for accelerating')
-parser.add_argument('--run_distribute', type=ast.literal_eval,
- default=True, help='Run distribute')
-parser.add_argument('--data_url', type=str, default=None, help='Dataset path')
-parser.add_argument('--train_url', type=str)
-parser.add_argument('--device_target', type=str,
- default='Ascend', help='Device target')
-parser.add_argument('--device_num', type=int,
- default=1, help='Number of device')
-parser.add_argument('--device_id', type=int, default=0)
-parser.add_argument('--pre_trained', type=ast.literal_eval, default=False)
-parser.add_argument('--pre_ckpt_path', type=str, default='',
- help='Ckpt file name(when modelarts on), full path for ckpt file(whem not modelarts) \
- REQUIRED when pre_trained is enabled.')
-parser.add_argument('--parameter_server', type=ast.literal_eval,
- default=False, help='Run parameter server train')
-parser.add_argument('--isModelArts', type=ast.literal_eval, default=True)
-parser.add_argument('--export', type=ast.literal_eval, default=False,
- help="Export air | mindir model.")
-parser.add_argument("--export_batch_size", type=int,
- default=1, help="batch size")
-parser.add_argument("--export_file_name", type=str,
- default="glore_res200", help="output file name.")
-parser.add_argument('--export_file_format', type=str,
- choices=["AIR", "ONNX", "MINDIR"], default='AIR', help='file format')
-parser.add_argument('--export_ckpt_url', type=str, default=None,
- help='Ckpt file name(when modelarts on), full path for ckpt file(whem not modelarts) \
- REQUIRED when export is enabled.')
-
-args_opt = parser.parse_args()
-
-if args_opt.isModelArts:
- import moxing as mox
-if args_opt.device_target == "Ascend":
- config = config1
-elif args_opt.device_target == "GPU":
- config = config2
- args_opt.cast_fp16 = False
-
-random.seed(1)
-np.random.seed(1)
-de.config.set_seed(1)
-set_seed(1)
-
-if __name__ == '__main__':
-
- target = args_opt.device_target
- ckpt_save_dir = config.save_checkpoint_path
- # init context
- context.set_context(mode=context.GRAPH_MODE,
- device_target=target, save_graphs=False)
- if args_opt.run_distribute:
- if target == "Ascend":
- device_id = int(os.getenv('DEVICE_ID'))
- context.set_context(device_id=device_id)
- context.set_auto_parallel_context(parallel_mode=ParallelMode.DATA_PARALLEL,
- gradients_mean=True,
- auto_parallel_search_mode="recursive_programming")
- init()
- elif target == "GPU":
- init()
- context.set_auto_parallel_context(device_num=get_group_size(),
- parallel_mode=ParallelMode.DATA_PARALLEL,
- gradients_mean=True)
- else:
- if target == "Ascend":
- device_id = args_opt.device_id
- context.set_context(mode=context.GRAPH_MODE, device_target=target, save_graphs=False,
- device_id=device_id)
-
- train_dataset_path = args_opt.data_url + '/train'
- if args_opt.isModelArts:
- # download dataset from obs to cache
- root = '/cache/dataset/device_' + os.getenv('DEVICE_ID')
- mox.file.copy_parallel(
- src_url=args_opt.data_url, dst_url='/cache/dataset/device_' + os.getenv('DEVICE_ID'))
- train_dataset_path = '/cache/dataset/device_' + os.getenv('DEVICE_ID') + '/train'
- if args_opt.pre_trained:
- args_opt.pre_ckpt_path = root + '/' + args_opt.pre_ckpt_path
- if args_opt.export:
- args_opt.export_ckpt_url = root + '/' + args_opt.export_ckpt_url
- args_opt.export_file_name = '/cache/train_output/' + args_opt.export_file_name
-
- # define net
-
- net = glore_resnet200(cast_fp16=args_opt.cast_fp16,
- class_num=config.class_num, use_glore=args_opt.use_glore)
-
- if args_opt.export:
- param_dict = load_checkpoint(args_opt.export_ckpt_url)
- load_param_into_net(net, param_dict)
-
- input_arr = Tensor(
- np.ones([args_opt.export_batch_size, 3, 224, 224]), mstype.float32)
- export(net, input_arr, file_name=args_opt.export_file_name,
- file_format=args_opt.export_file_format)
- if args_opt.isModelArts:
- mox.file.copy_parallel(
- src_url='/cache/train_output', dst_url=args_opt.train_url)
- else:
- # create dataset
- dataset = get_dataset(dataset_path=train_dataset_path, do_train=True, use_randaugment=True, repeat_num=1,
- batch_size=config.batch_size, target=target)
- step_size = dataset.get_dataset_size()
-
- # init weight
- if args_opt.pre_trained:
- param_dict = load_checkpoint(args_opt.pre_ckpt_path)
- load_param_into_net(net, param_dict)
- else:
- for _, cell in net.cells_and_names():
- if isinstance(cell, nn.Conv2d):
- cell.weight.default_input = weight_init.initializer(weight_init.XavierUniform(),
- cell.weight.shape,
- cell.weight.dtype)
- if isinstance(cell, nn.Dense):
- cell.weight.default_input = weight_init.initializer(weight_init.TruncatedNormal(),
- cell.weight.shape,
- cell.weight.dtype)
-
- # init lr
- lr = get_lr(lr_init=config.lr_init,
- lr_end=config.lr_end,
- lr_max=config.lr_max,
- warmup_epochs=config.warmup_epochs,
- total_epochs=config.epoch_size,
- steps_per_epoch=step_size,
- lr_decay_mode=config.lr_decay_mode)
- lr = Tensor(lr)
-
- #
- # define opt
- decayed_params = []
- no_decayed_params = []
- for param in net.trainable_params():
- if 'beta' not in param.name and 'gamma' not in param.name and 'bias' not in param.name:
- decayed_params.append(param)
- else:
- no_decayed_params.append(param)
-
- group_params = [{'params': decayed_params, 'weight_decay': config.weight_decay},
- {'params': no_decayed_params},
- {'order_params': net.trainable_params()}]
- net_opt = nn.SGD(group_params, learning_rate=lr, momentum=config.momentum, weight_decay=config.weight_decay,
- loss_scale=config.loss_scale, nesterov=True)
-
- # define loss, model
- loss = SoftmaxCrossEntropyExpand(sparse=True)
- loss_scale = FixedLossScaleManager(
- config.loss_scale, drop_overflow_update=False)
- model = Model(net, loss_fn=loss, optimizer=net_opt,
- loss_scale_manager=loss_scale)
-
- # define callbacks
- time_cb = TimeMonitor(data_size=step_size)
- loss_cb = LossMonitor()
- cb = [time_cb, loss_cb]
- if config.save_checkpoint:
- config_ck = CheckpointConfig(save_checkpoint_steps=config.save_checkpoint_epochs * step_size,
- keep_checkpoint_max=config.keep_checkpoint_max)
- if args_opt.isModelArts:
- save_checkpoint_path = '/cache/train_output/device_' + \
- os.getenv('DEVICE_ID') + '/'
- if args_opt.device_num == 1:
- ckpt_cb = ModelCheckpoint(prefix='glore_resnet200',
- directory=save_checkpoint_path,
- config=config_ck)
- cb += [ckpt_cb]
- if args_opt.device_num > 1 and get_rank() % 8 == 0:
- ckpt_cb = ModelCheckpoint(prefix='glore_resnet200',
- directory=save_checkpoint_path,
- config=config_ck)
- cb += [ckpt_cb]
- else:
- save_checkpoint_path = config.save_checkpoint_path
- if not os.path.isdir(save_checkpoint_path):
- os.makedirs(save_checkpoint_path)
-
- if args_opt.device_num == 1:
- ckpt_cb = ModelCheckpoint(prefix='glore_resnet200',
- directory=save_checkpoint_path,
- config=config_ck)
- cb += [ckpt_cb]
- if args_opt.device_num > 1 and get_rank() % 8 == 0:
- ckpt_cb = ModelCheckpoint(prefix='glore_resnet200',
- directory=save_checkpoint_path,
- config=config_ck)
- cb += [ckpt_cb]
-
- model.train(config.epoch_size - config.pretrain_epoch_size, dataset,
- callbacks=cb, dataset_sink_mode=True)
- ckpt_list = glob.glob('/cache/train_output/device_' + \
- os.getenv('DEVICE_ID') + '/*.ckpt')
- if not ckpt_list:
- print("ckpt file not generated.")
-
- ckpt_list.sort(key=os.path.getmtime)
- ckpt_model = ckpt_list[-1]
- print("checkpoint path", ckpt_model)
- ckpt_param_dict = load_checkpoint(ckpt_model)
- input_arr = Tensor(np.zeros([1, 3, 224, 224], np.float32))
- # frozen to ait file
- export_net = glore_resnet200(cast_fp16=args_opt.cast_fp16,
- class_num=config.class_num, use_glore=args_opt.use_glore)
- load_param_into_net(export_net, ckpt_param_dict)
- export(export_net, input_arr, file_name='/cache/train_output/device_' + \
- os.getenv('DEVICE_ID') + '/glore_res200', file_format="AIR")
- if args_opt.isModelArts:
- mox.file.copy_parallel(
- src_url='/cache/train_output', dst_url=args_opt.train_url)
diff --git a/research/cv/glore_res200/postprocess.py b/research/cv/glore_res200/postprocess.py
deleted file mode 100644
index 4144a3ee71e41825744a5651d01e04983513e355..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/postprocess.py
+++ /dev/null
@@ -1,48 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""postprocess for 310 inference"""
-import os
-import argparse
-import json
-import numpy as np
-from mindspore.nn import Top1CategoricalAccuracy, Top5CategoricalAccuracy
-parser = argparse.ArgumentParser(description="postprocess")
-label_path = "./preprocess_Result/cifar10_label_ids.npy"
-parser.add_argument("--result_dir", type=str, default="./result_Files", help="result files path.")
-parser.add_argument('--dataset_name', type=str, default="imagenet2012")
-parser.add_argument("--label_dir", type=str, default=label_path, help="image file path.")
-args = parser.parse_args()
-
-def calcul_acc(lab, preds):
- return sum(1 for x, y in zip(lab, preds) if x == y) / len(lab)
-if __name__ == '__main__':
- batch_size = 1
- top1_acc = Top1CategoricalAccuracy()
- rst_path = args.result_dir
- label_list = []
- pred_list = []
- file_list = os.listdir(rst_path)
- top5_acc = Top5CategoricalAccuracy()
- with open('./preprocess_Result/imagenet_label.json', "r") as label:
- labels = json.load(label)
- for f in file_list:
- label = f.split("_0.bin")[0] + ".JPEG"
- label_list.append(labels[label])
- pred = np.fromfile(os.path.join(rst_path, f), np.float32)
- pred = pred.reshape(batch_size, int(pred.shape[0] / batch_size))
- top1_acc.update(pred, [labels[label],])
- top5_acc.update(pred, [labels[label],])
- print("Top1 acc: ", top1_acc.eval())
- print("Top5 acc: ", top5_acc.eval())
diff --git a/research/cv/glore_res200/preprocess.py b/research/cv/glore_res200/preprocess.py
deleted file mode 100644
index da3a7f8f5ebf3b7645f493246bb5181bbe5990dd..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/preprocess.py
+++ /dev/null
@@ -1,48 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""preprocess"""
-import os
-import argparse
-import json
-parser = argparse.ArgumentParser('preprocess')
-parser.add_argument('--dataset_name', type=str, choices=["cifar10", "imagenet2012"], default="imagenet2012")
-parser.add_argument('--data_path', type=str, default='', help='eval data dir')
-parser.add_argument('--result_path', type=str, default='./preprocess_Result/', help='result path')
-def create_label(result_path, dir_path):
- """create label json for imagenet"""
- print("[WARNING] Create imagenet label. Currently only use for Imagenet2012!")
- dirs = os.listdir(dir_path)
- file_list = []
- for file in dirs:
- file_list.append(file)
- file_list = sorted(file_list)
-
- total = 0
- img_label = {}
- for i, file_dir in enumerate(file_list):
- files = os.listdir(os.path.join(dir_path, file_dir))
- for f in files:
- img_label[f] = i
- total += len(files)
-
- json_file = os.path.join(result_path, "imagenet_label.json")
- with open(json_file, "w+") as label:
- json.dump(img_label, label)
-
- print("[INFO] Completed! Total {} data.".format(total))
-
-args = parser.parse_args()
-if __name__ == "__main__":
- create_label('./preprocess_Result/', args.data_path)
diff --git a/research/cv/glore_res200/scripts/docker_start.sh b/research/cv/glore_res200/scripts/docker_start.sh
deleted file mode 100644
index a033c3e3d5f6d0032397f027c2e2d9427429fa3d..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/scripts/docker_start.sh
+++ /dev/null
@@ -1,35 +0,0 @@
-#!/bin/bash
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.mitations under the License.
-
-docker_image=$1
-data_dir=$2
-model_dir=$3
-
-docker run -it --ipc=host \
- --device=/dev/davinci0 \
- --device=/dev/davinci1 \
- --device=/dev/davinci2 \
- --device=/dev/davinci3 \
- --device=/dev/davinci4 \
- --device=/dev/davinci5 \
- --device=/dev/davinci6 \
- --device=/dev/davinci7 \
- --device=/dev/davinci_manager \
- --device=/dev/devmm_svm --device=/dev/hisi_hdc \
- -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
- -v /usr/local/Ascend/add-ons/:/usr/local/Ascend/add-ons/ \
- -v ${model_dir}:${model_dir} \
- -v ${data_dir}:${data_dir} \
- -v /root/ascend/log:/root/ascend/log ${docker_image} /bin/bash
diff --git a/research/cv/glore_res200/scripts/run_310_infer.sh b/research/cv/glore_res200/scripts/run_310_infer.sh
deleted file mode 100644
index db8af43d84a62d54a95066943115f15906c5d070..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/scripts/run_310_infer.sh
+++ /dev/null
@@ -1,125 +0,0 @@
-#!/bin/bash
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-
-if [[ $# -lt 2 || $# -gt 3 ]]; then
- echo "Usage: bash run_infer_310.sh [MINDIR_PATH] [DATASET_PATH] [DEVICE_ID]
- DEVICE_ID is optional, it can be set by environment variable device_id, otherwise the value is zero"
-exit 1
-fi
-
-get_real_path(){
- if [ "${1:0:1}" == "/" ]; then
- echo "$1"
- else
- echo "$(realpath -m $PWD/$1)"
- fi
-}
-model=$(get_real_path $1)
-dataset_name='imagenet2012'
-dataset_path=$(get_real_path $2)
-need_preprocess='y'
-device_id=0
-if [ $# == 3 ]; then
- device_id=$3
-fi
-
-echo "mindir name: "$model
-echo "dataset name: "$dataset_name
-echo "dataset path: "$dataset_path
-echo "need preprocess: "$need_preprocess
-echo "device id: "$device_id
-
-export ASCEND_HOME=/usr/local/Ascend/
-if [ -d ${ASCEND_HOME}/ascend-toolkit ]; then
- export PATH=$ASCEND_HOME/fwkacllib/bin:$ASCEND_HOME/fwkacllib/ccec_compiler/bin:$ASCEND_HOME/ascend-toolkit/latest/fwkacllib/ccec_compiler/bin:$ASCEND_HOME/ascend-toolkit/latest/atc/bin:$PATH
- export LD_LIBRARY_PATH=$ASCEND_HOME/fwkacllib/lib64:/usr/local/lib:$ASCEND_HOME/ascend-toolkit/latest/atc/lib64:$ASCEND_HOME/ascend-toolkit/latest/fwkacllib/lib64:$ASCEND_HOME/driver/lib64:$ASCEND_HOME/add-ons:$LD_LIBRARY_PATH
- export TBE_IMPL_PATH=$ASCEND_HOME/ascend-toolkit/latest/opp/op_impl/built-in/ai_core/tbe
- export PYTHONPATH=$ASCEND_HOME/fwkacllib/python/site-packages:${TBE_IMPL_PATH}:$ASCEND_HOME/ascend-toolkit/latest/fwkacllib/python/site-packages:$PYTHONPATH
- export ASCEND_OPP_PATH=$ASCEND_HOME/ascend-toolkit/latest/opp
-else
- export ASCEND_HOME=/usr/local/Ascend/latest/
- export PATH=$ASCEND_HOME/fwkacllib/bin:$ASCEND_HOME/fwkacllib/ccec_compiler/bin:$ASCEND_HOME/atc/ccec_compiler/bin:$ASCEND_HOME/atc/bin:$PATH
- export LD_LIBRARY_PATH=$ASCEND_HOME/fwkacllib/lib64:/usr/local/lib:$ASCEND_HOME/atc/lib64:$ASCEND_HOME/acllib/lib64:$ASCEND_HOME/driver/lib64:$ASCEND_HOME/add-ons:$LD_LIBRARY_PATH
- export PYTHONPATH=$ASCEND_HOME/fwkacllib/python/site-packages:$ASCEND_HOME/atc/python/site-packages:$PYTHONPATH
- export ASCEND_OPP_PATH=$ASCEND_HOME/opp
-fi
-export ASCEND_HOME=/usr/local/Ascend
-export PATH=$ASCEND_HOME/fwkacllib/ccec_compiler/bin:$ASCEND_HOME/fwkacllib/bin:$ASCEND_HOME/toolkit/bin:$PATH
-export LD_LIBRARY_PATH=/usr/local/lib/:/usr/local/fwkacllib/lib64:$ASCEND_HOME/driver/lib64:$ASCEND_HOME/add-ons:/usr/local/Ascend/toolkit/lib64:$LD_LIBRARY_PATH
-export PYTHONPATH=$ASCEND_HOME/fwkacllib/python/site-packages
-export PATH=/usr/local/python375/bin:$PATH
-export NPU_HOST_LIB=/usr/local/Ascend/acllib/lib64/stub
-export ASCEND_OPP_PATH=/usr/local/Ascend/opp
-export ASCEND_AICPU_PATH=/usr/local/Ascend
-export LD_LIBRARY_PATH=/usr/local/lib64/:$LD_LIBRARY_PATH
-function preprocess_data()
-{
- if [ -d preprocess_Result ]; then
- rm -rf ./preprocess_Result
- fi
- mkdir preprocess_Result
- python3.7 ../preprocess.py --dataset_name=$dataset_name --data_path=$dataset_path
-}
-
-function compile_app()
-{
- cd ../ascend310_infer/ || exit
- bash build.sh &> build.log
-}
-
-function infer()
-{
- cd - || exit
- if [ -d result_Files ]; then
- rm -rf ./result_Files
- fi
- if [ -d time_Result ]; then
- rm -rf ./time_Result
- fi
- mkdir result_Files
- mkdir time_Result
-
- ../ascend310_infer/out/main --mindir_path=$model --dataset_name=$dataset_name --input0_path=$dataset_path --device_id=$device_id &> infer.log
-
-}
-
-function cal_acc()
-{
- python3.7 ../postprocess.py --dataset_name=$dataset_name &> acc.log
-}
-
-if [ $need_preprocess == "y" ]; then
- preprocess_data
- if [ $? -ne 0 ]; then
- echo "preprocess dataset failed"
- exit 1
- fi
-fi
-compile_app
-if [ $? -ne 0 ]; then
- echo "compile app code failed"
- exit 1
-fi
-infer
-if [ $? -ne 0 ]; then
- echo " execute inference failed"
- exit 1
-fi
-cal_acc
-if [ $? -ne 0 ]; then
- echo "calculate accuracy failed"
- exit 1
-fi
diff --git a/research/cv/glore_res200/scripts/run_distribute_train.sh b/research/cv/glore_res200/scripts/run_distribute_train.sh
deleted file mode 100644
index 8468b34efc6e5a484f7a6e35db01c2c30b8df9b1..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/scripts/run_distribute_train.sh
+++ /dev/null
@@ -1,80 +0,0 @@
-#!/bin/bash
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-
-echo "=============================================================================================================="
-echo "Please run the script as: "
-echo "bash run_distribute_train.sh DATA_PATH RANK_SIZE"
-echo "For example: bash run_distribute_train.sh /path/dataset /path/rank_table"
-echo "It is better to use the absolute path."
-echo "=============================================================================================================="
-set -e
-get_real_path(){
- if [ "${1:0:1}" == "/" ]; then
- echo "$1"
- else
- echo "$(realpath -m $PWD/$1)"
- fi
-}
-DATA_PATH=$(get_real_path $1)
-export DATA_PATH=${DATA_PATH}
-RANK_TABLE=$(get_real_path $2)
-export RANK_TABLE_FILE=${RANK_TABLE}
-export RANK_SIZE=8
-
-echo "$EXEC_PATH"
-
-export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
-
-for((i=1;i<8;i++))
-do
- rm -rf device$i
- mkdir device$i
- cd ./device$i
- mkdir src
- cd ../
- cp ../*.py ./device$i
- cp ../src/*.py ./device$i/src
- cd ./device$i
- export DEVICE_ID=$i
- export RANK_ID=$i
- echo "start training for device $i"
- env > env$i.log
- python3 train.py --data_url $1 --isModelArts False --run_distribute True > train$i.log 2>&1 &
- echo "$i finish"
- cd ../
-done
-rm -rf device0
-mkdir device0
-cd ./device0
-mkdir src
-cd ../
-cp ../*.py ./device0
-cp ../src/*.py ./device0/src
-cd ./device0
-export DEVICE_ID=0
-export RANK_ID=0
-echo "start training for device 0"
-env > env0.log
-python3 train.py --data_url $1 --isModelArts False --run_distribute True > train0.log 2>&1 &
-
-if [ $? -eq 0 ];then
- echo "training success"
-else
- echo "training failed"
- exit 2
-fi
-echo "finish"
-cd ../
diff --git a/research/cv/glore_res200/scripts/run_distribute_train_gpu.sh b/research/cv/glore_res200/scripts/run_distribute_train_gpu.sh
deleted file mode 100644
index 35c7da610eb2cd46bb43f0644287069da21d660d..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/scripts/run_distribute_train_gpu.sh
+++ /dev/null
@@ -1,45 +0,0 @@
-#!/bin/bash
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-
-echo "=============================================================================================================="
-echo "Please run the script as: "
-echo "bash run_distribute_train.sh DATA_PATH RANK_SIZE"
-echo "For example: bash run_distribute_train.sh /path/dataset 8"
-echo "It is better to use the absolute path."
-echo "=============================================================================================================="
-set -e
-DEVICE_NUM=$1
-DATA_PATH=$2
-export DATA_PATH=${DATA_PATH}
-export DEVICE_NUM=$1
-export RANK_SIZE=$1
-export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
-
-cd ../
-rm -rf ./train_parallel
-mkdir ./train_parallel
-cd ./train_parallel
-mkdir src
-cd ../
-cp *.py ./train_parallel
-cp src/*.py ./train_parallel/src
-cd ./train_parallel
-env > env.log
-echo "start training"
- mpirun -n $1 --allow-run-as-root --output-filename log_output --merge-stderr-to-stdout \
- python3 train.py --device_num=$1 --data_url=$2 --isModelArts=False --run_distribute=True \
- --device_target="GPU" > train.log 2>&1 &
-
diff --git a/research/cv/glore_res200/scripts/run_eval.sh b/research/cv/glore_res200/scripts/run_eval.sh
deleted file mode 100644
index 6ee62ba2c98bb4d36a89aecbd68166671b1ae804..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/scripts/run_eval.sh
+++ /dev/null
@@ -1,47 +0,0 @@
-#!/bin/bash
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-
-echo "=============================================================================================================="
-echo "Please run the script as: "
-echo "bash run_eval.sh DATA_PATH DEVICE_ID CKPT_PATH"
-echo "For example: bash run_eval.sh /path/dataset 0 /path/ckpt"
-echo "It is better to use the absolute path."
-echo "=============================================================================================================="
-set -e
-DATA_PATH=$1
-DEVICE_ID=$2
-export DATA_PATH=${DATA_PATH}
-
-EXEC_PATH=$(pwd)
-
-echo "$EXEC_PATH"
-
-export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
-
-cd ../
-export DEVICE_ID=$2
-export RANK_ID=$2
-env > env0.log
-python3 eval.py --data_url $1 --isModelArts False --device_id $2 --ckpt_path $3> eval.log 2>&1
-
-if [ $? -eq 0 ];then
- echo "testing success"
-else
- echo "testing failed"
- exit 2
-fi
-echo "finish"
-cd ../
diff --git a/research/cv/glore_res200/scripts/run_eval_gpu.sh b/research/cv/glore_res200/scripts/run_eval_gpu.sh
deleted file mode 100644
index 2b75b1590caf73989b0ca59bb732f186bc767101..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/scripts/run_eval_gpu.sh
+++ /dev/null
@@ -1,44 +0,0 @@
-#!/bin/bash
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-
-echo "=============================================================================================================="
-echo "Please run the script as: "
-echo "bash run_eval.sh DATA_PATH DEVICE_ID CKPT_PATH"
-echo "For example: bash run_eval.sh /path/dataset 0 /path/ckpt"
-echo "It is better to use the absolute path."
-echo "=============================================================================================================="
-set -e
-DATA_PATH=$1
-export DATA_PATH=${DATA_PATH}
-export CUDA_VISIBLE_DEVICES=0
-EXEC_PATH=$(pwd)
-
-echo "$EXEC_PATH"
-
-export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
-
-cd ../
-env > env0.log
-python3 eval.py --data_url $1 --isModelArts False --ckpt_path $2 --device_target="GPU"> eval.log 2>&1 &
-
-if [ $? -eq 0 ];then
- echo "testing success"
-else
- echo "testing failed"
- exit 2
-fi
-echo "finish"
-cd ../
diff --git a/research/cv/glore_res200/scripts/run_standalone_train.sh b/research/cv/glore_res200/scripts/run_standalone_train.sh
deleted file mode 100644
index a60db9896508c736be84c9a2d8b36f1ca4f0645f..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/scripts/run_standalone_train.sh
+++ /dev/null
@@ -1,47 +0,0 @@
-#!/bin/bash
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-
-echo "=============================================================================================================="
-echo "Please run the script as: "
-echo "bash run_standalone_train.sh DATA_PATH DEVICE_ID"
-echo "For example: bash run_standalone_train.sh /path/dataset 0"
-echo "It is better to use the absolute path."
-echo "=============================================================================================================="
-set -e
-DATA_PATH=$1
-DEVICE_ID=$2
-export DATA_PATH=${DATA_PATH}
-
-EXEC_PATH=$(pwd)
-
-echo "$EXEC_PATH"
-
-export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
-
-cd ../
-export DEVICE_ID=$2
-export RANK_ID=$2
-env > env0.log
-python3 train.py --data_url $1 --isModelArts False --run_distribute False --device_id $2 > train.log 2>&1
-
-if [ $? -eq 0 ];then
- echo "training success"
-else
- echo "training failed"
- exit 2
-fi
-echo "finish"
-cd ../
diff --git a/research/cv/glore_res200/scripts/run_standalone_train_gpu.sh b/research/cv/glore_res200/scripts/run_standalone_train_gpu.sh
deleted file mode 100644
index d88d3a2c28f8a4fa6905d831dd504676776a61c9..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/scripts/run_standalone_train_gpu.sh
+++ /dev/null
@@ -1,49 +0,0 @@
-#!/bin/bash
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-
-echo "=============================================================================================================="
-echo "Please run the script as: "
-echo "bash run_standalone_train.sh DATA_PATH DEVICE_ID"
-echo "For example: bash run_standalone_train.sh /path/dataset 0"
-echo "It is better to use the absolute path."
-echo "=============================================================================================================="
-set -e
-
-ulimit -u unlimited
-export DEVICE_NUM=1
-export RANK_SIZE=1
-DATA_PATH=$1
-export DATA_PATH=${DATA_PATH}
-export CUDA_VISIBLE_DEVICES=0
-
-EXEC_PATH=$(pwd)
-
-echo "$EXEC_PATH"
-
-export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
-
-cd ../
-env > env0.log
-python3 train.py --data_url=$1 --isModelArts=False --run_distribute=False --device_target="GPU" > train.log 2>&1
-
-if [ $? -eq 0 ];then
- echo "training success"
-else
- echo "training failed"
- exit 2
-fi
-echo "finish"
-cd ../
diff --git a/research/cv/glore_res200/src/__init__.py b/research/cv/glore_res200/src/__init__.py
deleted file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000
diff --git a/research/cv/glore_res200/src/config.py b/research/cv/glore_res200/src/config.py
deleted file mode 100644
index ce2fe8bc2494a6f10f0dc132d1fbb8bd88a0c8a6..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/src/config.py
+++ /dev/null
@@ -1,55 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""
-network config setting, will be used in train.py
-"""
-from easydict import EasyDict
-config1 = EasyDict({
- "class_num": 1000,
- "batch_size": 128,
- "loss_scale": 1024,
- "momentum": 0.08,
- "weight_decay": 0.0002,
- "epoch_size": 150,
- "pretrain_epoch_size": 0,
- "save_checkpoint": True,
- "save_checkpoint_epochs": 5,
- "keep_checkpoint_max": 10,
- "save_checkpoint_path": "./",
- "warmup_epochs": 0,
- "lr_decay_mode": "poly",
- "lr_init": 0.1,
- "lr_end": 0,
- "lr_max": 0.4
-})
-
-config2 = EasyDict({
- "class_num": 1000,
- "batch_size": 64,
- "loss_scale": 1024,
- "momentum": 0.08,
- "weight_decay": 0.0002,
- "epoch_size": 150,
- "pretrain_epoch_size": 0,
- "save_checkpoint": True,
- "save_checkpoint_epochs": 5,
- "keep_checkpoint_max": 10,
- "save_checkpoint_path": "./",
- "warmup_epochs": 0,
- "lr_decay_mode": "poly",
- "lr_init": 0.1,
- "lr_end": 0,
- "lr_max": 0.4
-})
diff --git a/research/cv/glore_res200/src/dataset.py b/research/cv/glore_res200/src/dataset.py
deleted file mode 100644
index 02427c807cf07056430df7fca4a74acf5be40522..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/src/dataset.py
+++ /dev/null
@@ -1,125 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""
-Data operations, will be used in train.py and eval.py
-"""
-import os
-import mindspore.common.dtype as mstype
-import mindspore.dataset.engine as de
-import mindspore.dataset.vision.c_transforms as C
-import mindspore.dataset.transforms.c_transforms as C2
-from mindspore.communication.management import init, get_rank, get_group_size
-from mindspore.dataset.vision import Inter
-from src.transform import RandAugment
-
-
-def create_dataset_ImageNet(dataset_path, do_train, use_randaugment=False, repeat_num=1, batch_size=32,
- target="Ascend"):
- """
- create a train or eval imagenet2012 dataset for resnet50
-
- Args:
- dataset_path(string): the path of dataset.
- do_train(bool): whether dataset is used for train or eval.
- use_randaugment(bool): enable randAugment.
- repeat_num(int): the repeat times of dataset. Default: 1
- batch_size(int): the batch size of dataset. Default: 32
- target(str): the device target. Default: Ascend
-
- Returns:
- dataset
- """
- if target == "Ascend":
- device_num, rank_id = _get_rank_info()
- elif target == "GPU":
- init("nccl")
- rank_id = get_rank()
- device_num = get_group_size()
-
- if device_num == 1:
- ds = de.ImageFolderDataset(dataset_path, num_parallel_workers=8, shuffle=True)
- else:
- ds = de.ImageFolderDataset(dataset_path, num_parallel_workers=8, shuffle=True,
- num_shards=device_num, shard_id=rank_id)
-
- image_size = 224
- mean = [0.485 * 255, 0.456 * 255, 0.406 * 255]
- std = [0.229 * 255, 0.224 * 255, 0.225 * 255]
-
- # define map operations
-
- if do_train:
- if use_randaugment:
- trans = [
- C.Decode(),
- C.RandomResizedCrop(size=(image_size, image_size),
- scale=(0.08, 1.0),
- ratio=(3. / 4., 4. / 3.),
- interpolation=Inter.BICUBIC),
- C.RandomHorizontalFlip(prob=0.5),
- ]
- else:
- trans = [
- C.RandomCropDecodeResize(image_size, scale=(0.08, 1.0), ratio=(0.75, 1.333)),
- C.RandomHorizontalFlip(prob=0.5),
- C.Normalize(mean=mean, std=std),
- C.HWC2CHW()
- ]
-
- else:
- use_randaugment = False
- trans = [
- C.Decode(),
- C.Resize(256),
- C.CenterCrop(image_size),
- C.Normalize(mean=mean, std=std),
- C.HWC2CHW()
- ]
-
- type_cast_op = C2.TypeCast(mstype.int32)
-
- ds = ds.map(input_columns="image", num_parallel_workers=8, operations=trans)
- ds = ds.map(input_columns="label", num_parallel_workers=8, operations=type_cast_op)
-
- # apply batch operations
- if use_randaugment:
- efficient_rand_augment = RandAugment()
- ds = ds.batch(batch_size,
- per_batch_map=efficient_rand_augment,
- input_columns=['image', 'label'],
- num_parallel_workers=2,
- drop_remainder=True)
- else:
- ds = ds.batch(batch_size, drop_remainder=True)
- # apply dataset repeat operation
- ds = ds.repeat(repeat_num)
-
- return ds
-
-
-def _get_rank_info():
- """
- get rank size and rank id
- """
- rank_size = int(os.environ.get("RANK_SIZE", 1))
-
- if rank_size > 1:
- rank_size = int(os.environ.get("RANK_SIZE"))
- rank_id = int(os.environ.get("RANK_ID"))
- else:
- rank_size = 1
- rank_id = 0
-
- return rank_size, rank_id
diff --git a/research/cv/glore_res200/src/glore_resnet200.py b/research/cv/glore_res200/src/glore_resnet200.py
deleted file mode 100644
index 5271a965c2cf83539e1cbee4f6d5a3e590f09952..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/src/glore_resnet200.py
+++ /dev/null
@@ -1,343 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""glore_resnet200"""
-from collections import OrderedDict
-import numpy as np
-import mindspore.nn as nn
-import mindspore.ops as ops
-from mindspore.ops import operations as P
-import mindspore.common.dtype as mstype
-from mindspore.common.tensor import Tensor
-
-
-def _weight_variable(shape, factor=0.01):
- init_value = np.random.randn(*shape).astype(np.float32) * factor
- return Tensor(init_value)
-
-
-def _conv3x3(in_channel, out_channel, stride=1):
- weight_shape = (out_channel, in_channel, 3, 3)
- weight = _weight_variable(weight_shape)
- return nn.Conv2d(in_channel, out_channel,
- kernel_size=3, stride=stride, padding=0, pad_mode='same', weight_init=weight, has_bias=False)
-
-
-def _conv1x1(in_channel, out_channel, stride=1):
- weight_shape = (out_channel, in_channel, 1, 1)
- weight = _weight_variable(weight_shape)
- return nn.Conv2d(in_channel, out_channel,
- kernel_size=1, stride=stride, padding=0, pad_mode='same', weight_init=weight, has_bias=False)
-
-
-def _conv7x7(in_channel, out_channel, stride=1):
- weight_shape = (out_channel, in_channel, 7, 7)
- weight = _weight_variable(weight_shape)
- return nn.Conv2d(in_channel, out_channel,
- kernel_size=7, stride=stride, padding=3, pad_mode='pad', weight_init=weight, has_bias=False)
-
-
-def _bn(channel):
- return nn.BatchNorm2d(channel, eps=1e-4, momentum=0.92,
- gamma_init=1, beta_init=0, moving_mean_init=0, moving_var_init=1)
-
-
-def _bn_last(channel):
- return nn.BatchNorm2d(channel, eps=1e-4, momentum=0.92,
- gamma_init=0, beta_init=0, moving_mean_init=0, moving_var_init=1)
-
-
-def _fc(in_channel, out_channel):
- weight_shape = (out_channel, in_channel)
- weight = _weight_variable(weight_shape)
- return nn.Dense(in_channel, out_channel, has_bias=True, weight_init=weight, bias_init=0)
-
-
-class BN_AC_Conv(nn.Cell):
- """
- Basic convolution block.
- """
- def __init__(self,
- in_channel,
- out_channel,
- kernel=1,
- pad=0,
- pad_mode='same',
- stride=1,
- groups=1,
- has_bias=False):
- super(BN_AC_Conv, self).__init__()
- self.bn = _bn(in_channel)
- self.relu = nn.ReLU()
- self.conv = nn.Conv2d(in_channel, out_channel,
- pad_mode=pad_mode,
- padding=pad,
- kernel_size=kernel,
- stride=stride,
- has_bias=has_bias,
- group=groups)
-
- def construct(self, x):
- out = self.bn(x)
- out = self.relu(out)
- out = self.conv(out)
- return out
-
-
-class GCN(nn.Cell):
- """
- Graph convolution unit (single layer)
- """
-
- def __init__(self, num_state, num_mode, bias=False):
- super(GCN, self).__init__()
- self.conv1 = nn.Conv1d(num_mode, num_mode, kernel_size=1)
- self.relu2 = nn.ReLU()
- self.conv2 = nn.Conv1d(num_state, num_state, kernel_size=1, has_bias=bias)
- self.transpose = ops.Transpose()
- self.add = P.Add()
-
- def construct(self, x):
- """construct"""
- identity = x
- # (n, num_state, num_node) -> (n, num_node, num_state)
- # -> (n, num_state, num_node)
- out = self.transpose(x, (0, 2, 1))
- out = self.conv1(out)
- out = self.transpose(out, (0, 2, 1))
- out = self.add(out, identity)
- out = self.relu2(out)
- out = self.conv2(out)
- return out
-
-
-class GloreUnit(nn.Cell):
- """
- Graph-based Global Reasoning Unit
- Parameter:
- 'normalize' is not necessary if the input size is fixed
- Args:
- num_in: Input channel
- num_mid:
- """
-
- def __init__(self, num_in, num_mid, cast_fp16=True,
- normalize=False):
- super(GloreUnit, self).__init__()
- self.normalize = normalize
- self.cast_fp16 = cast_fp16
- self.num_s = int(2 * num_mid) # 512 num_in = 1024
- self.num_n = int(1 * num_mid) # 256
- # reduce dim
- self.conv_state = nn.SequentialCell([_bn(num_in),
- nn.ReLU(),
- _conv1x1(num_in, self.num_s, stride=1)])
- # projection map
- self.conv_proj = nn.SequentialCell([_bn(num_in),
- nn.ReLU(),
- _conv1x1(num_in, self.num_n, stride=1)])
-
- self.gcn = GCN(num_state=self.num_s, num_mode=self.num_n)
-
- self.conv_extend = nn.SequentialCell([_bn_last(self.num_s),
- nn.ReLU(),
- _conv1x1(self.num_s, num_in, stride=1)])
-
- self.reshape = ops.Reshape()
- self.matmul = ops.BatchMatMul()
- self.transpose = ops.Transpose()
- self.add = P.Add()
- self.cast = P.Cast()
-
- def construct(self, x):
- """construct"""
- n = x.shape[0]
- identity = x
- # (n, num_in, h, w) --> (n, num_state, h, w)
- # --> (n, num_state, h*w)
- x_conv_state = self.conv_state(x)
- x_state_reshaped = self.reshape(x_conv_state, (n, self.num_s, -1))
-
- # (n, num_in, h, w) --> (n, num_node, h, w)
- # --> (n, num_node, h*w)
- x_conv_proj = self.conv_proj(x)
- x_proj_reshaped = self.reshape(x_conv_proj, (n, self.num_n, -1))
-
- # (n, num_in, h, w) --> (n, num_node, h, w)
- # --> (n, num_node, h*w)
- x_rproj_reshaped = x_proj_reshaped
-
- # projection: coordinate space -> interaction space
- # (n, num_state, h*w) x (n, num_node, h*w)T --> (n, num_state, num_node)
- x_proj_reshaped = self.transpose(x_proj_reshaped, (0, 2, 1))
-
- if self.cast_fp16:
- # 提高速度
- x_state_reshaped_fp16 = self.cast(x_state_reshaped, mstype.float16)
- x_proj_reshaped_fp16 = self.cast(x_proj_reshaped, mstype.float16)
- x_n_state_fp16 = self.matmul(x_state_reshaped_fp16, x_proj_reshaped_fp16)
- x_n_state = self.cast(x_n_state_fp16, mstype.float32)
- else:
- x_n_state = self.matmul(x_state_reshaped, x_proj_reshaped)
-
- if self.normalize:
- x_n_state = x_n_state * (1. / x_state_reshaped.shape[2])
-
- # reasoning: (n, num_state, num_node) -> (n, num_state, num_node)
- x_n_rel = self.gcn(x_n_state)
-
- # reverse projection: interaction space -> coordinate space
- # (n, num_state, num_node) x (n, num_node, h*w) --> (n, num_state, h*w)
- if self.cast_fp16:
- x_n_rel_fp16 = self.cast(x_n_rel, mstype.float16)
- x_rproj_reshaped_fp16 = self.cast(x_rproj_reshaped, mstype.float16)
- x_state_reshaped_fp16 = self.matmul(x_n_rel_fp16, x_rproj_reshaped_fp16)
- x_state_reshaped = self.cast(x_state_reshaped_fp16, mstype.float32)
- else:
- x_state_reshaped = self.matmul(x_n_rel, x_rproj_reshaped)
-
- # (n, num_state, h*w) --> (n, num_state, h, w)
- x_state = self.reshape(x_state_reshaped, (n, self.num_s, identity.shape[2], identity.shape[3]))
-
- # (n, num_state, h, w) -> (n, num_in, h, w)
- x_conv_extend = self.conv_extend(x_state)
- out = self.add(x_conv_extend, identity)
- return out
-
-
-class Residual_Unit(nn.Cell):
- """
- Residual unit used in Resnet
- """
- def __init__(self,
- in_channel,
- mid_channel,
- out_channel,
- groups=1,
- stride=1,
- first_block=False):
- super(Residual_Unit, self).__init__()
- self.first_block = first_block
- self.BN_AC_Conv1 = BN_AC_Conv(in_channel, mid_channel, kernel=1, pad=0)
- self.BN_AC_Conv2 = BN_AC_Conv(mid_channel, mid_channel, kernel=3, pad_mode='pad', pad=1, stride=stride,
- groups=groups)
- self.BN_AC_Conv3 = BN_AC_Conv(mid_channel, out_channel, kernel=1, pad=0)
- if self.first_block:
- self.BN_AC_Conv_w = BN_AC_Conv(in_channel, out_channel, kernel=1, pad=0, stride=stride)
- self.add = P.Add()
-
- def construct(self, x):
- identity = x
- out = self.BN_AC_Conv1(x)
- out = self.BN_AC_Conv2(out)
- out = self.BN_AC_Conv3(out)
- if self.first_block:
- identity = self.BN_AC_Conv_w(identity)
-
- out = self.add(out, identity)
- return out
-
-
-class ResNet(nn.Cell):
- """
- Resnet architecture
- """
- def __init__(self,
- layer_nums,
- num_classes,
- use_glore=False,
- cast_fp16=True):
- super(ResNet, self).__init__()
- self.layer1 = nn.SequentialCell(OrderedDict([
- ('conv', _conv7x7(3, 64, stride=2)),
- ('bn', _bn(64),),
- ('relu', nn.ReLU(),),
- ('maxpool', nn.MaxPool2d(kernel_size=3, stride=2, pad_mode="same"))
- ]))
-
- num_in = [64, 256, 512, 1024]
- num_mid = [64, 128, 256, 512]
- num_out = [256, 512, 1024, 2048]
- self.layer2 = nn.SequentialCell(OrderedDict([
- ("Residual_Unit{}".format(i), Residual_Unit(in_channel=(num_in[0] if i == 1 else num_out[0]),
- mid_channel=num_mid[0],
- out_channel=num_out[0],
- stride=1,
- first_block=(i == 1))) for i in range(1, layer_nums[0] + 1)
- ]))
-
- blocks_layer3 = []
- for i in range(1, layer_nums[1] + 1):
- blocks_layer3.append(
- ("Residual_Unit{}".format(i), Residual_Unit(in_channel=(num_in[1] if i == 1 else num_out[1]),
- mid_channel=num_mid[1],
- out_channel=num_out[1],
- stride=(2 if i == 1 else 1),
- first_block=(i == 1))))
- if use_glore and i in [12, 18]:
- blocks_layer3.append(("Residual_Unit{}_GloreUnit".format(i), GloreUnit(num_out[1], num_mid[1],
- cast_fp16)))
- self.layer3 = nn.SequentialCell(OrderedDict(blocks_layer3))
-
- blocks_layer4 = []
- for i in range(1, layer_nums[2] + 1):
- blocks_layer4.append(
- ("Residual_Unit{}".format(i), Residual_Unit(in_channel=(num_in[2] if i == 1 else num_out[2]),
- mid_channel=num_mid[2],
- out_channel=num_out[2],
- stride=(2 if i == 1 else 1),
- first_block=(i == 1))))
- if use_glore and i in [16, 24, 32]:
- blocks_layer4.append(("Residual_Unit{}_GloreUnit".format(i), GloreUnit(num_out[2], num_mid[2],
- cast_fp16)))
- self.layer4 = nn.SequentialCell(OrderedDict(blocks_layer4))
-
- self.layer5 = nn.SequentialCell(OrderedDict([
- ("Residual_Unit{}".format(i), Residual_Unit(in_channel=(num_in[3] if i == 1 else num_out[3]),
- mid_channel=num_mid[3],
- out_channel=num_out[3],
- stride=(2 if i == 1 else 1),
- first_block=(i == 1))) for i in range(1, layer_nums[3] + 1)
- ]))
-
- self.tail = nn.SequentialCell(OrderedDict([
- ('bn', _bn(num_out[3])),
- ('relu', nn.ReLU())
- ]))
-
- self.mean = ops.ReduceMean(keep_dims=True)
- self.flatten = nn.Flatten()
- self.classifier = _fc(num_out[3], num_classes)
- self.print = ops.Print()
-
- def construct(self, x):
- """construct"""
- c1 = self.layer1(x)
- c2 = self.layer2(c1)
- c3 = self.layer3(c2)
- c4 = self.layer4(c3)
- c5 = self.layer5(c4)
-
- out = self.tail(c5)
- out = self.mean(out, (2, 3))
- out = self.flatten(out)
- out = self.classifier(out)
- return out
-
-
-def glore_resnet200(cast_fp16=True, class_num=1000, use_glore=True):
- return ResNet(layer_nums=[3, 24, 36, 3],
- num_classes=class_num,
- use_glore=use_glore,
- cast_fp16=cast_fp16)
diff --git a/research/cv/glore_res200/src/loss.py b/research/cv/glore_res200/src/loss.py
deleted file mode 100644
index ff9c650cb6d4b2fde8e03059f6fad3ffd1898c52..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/src/loss.py
+++ /dev/null
@@ -1,53 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""define loss function for network"""
-from mindspore import dtype as mstype
-from mindspore import Tensor
-import mindspore.nn as nn
-import mindspore.ops as ops
-
-
-class SoftmaxCrossEntropyExpand(nn.Cell): # pylint: disable=missing-docstring
- def __init__(self, sparse=False):
- super(SoftmaxCrossEntropyExpand, self).__init__()
- self.exp = ops.Exp()
- self.sum = ops.ReduceSum(keep_dims=True)
- self.onehot = ops.OneHot()
- self.on_value = Tensor(1.0, mstype.float32)
- self.off_value = Tensor(0.0, mstype.float32)
- self.div = ops.RealDiv()
- self.log = ops.Log()
- self.sum_cross_entropy = ops.ReduceSum(keep_dims=False)
- self.mul = ops.Mul()
- self.mul2 = ops.Mul()
- self.mean = ops.ReduceMean(keep_dims=False)
- self.sparse = sparse
- self.max = ops.ReduceMax(keep_dims=True)
- self.sub = ops.Sub()
- self.eps = Tensor(1e-24, mstype.float32)
-
- def construct(self, logit, label): # pylint: disable=missing-docstring
- logit_max = self.max(logit, -1)
- exp = self.exp(self.sub(logit, logit_max))
- exp_sum = self.sum(exp, -1)
- softmax_result = self.div(exp, exp_sum)
- if self.sparse:
- label = self.onehot(label, ops.shape(logit)[1], self.on_value, self.off_value)
-
- softmax_result_log = self.log(softmax_result + self.eps)
- loss = self.sum_cross_entropy((self.mul(softmax_result_log, label)), -1)
- loss = self.mul2(ops.scalar_to_array(-1.0), loss)
- loss = self.mean(loss, -1)
- return loss
diff --git a/research/cv/glore_res200/src/lr_generator.py b/research/cv/glore_res200/src/lr_generator.py
deleted file mode 100644
index ee35b35264049bb35e3c64809a0a80e85589b79d..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/src/lr_generator.py
+++ /dev/null
@@ -1,128 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""learning rate generator"""
-import math
-import numpy as np
-
-
-def get_lr(lr_init, lr_end, lr_max, warmup_epochs, total_epochs, steps_per_epoch, lr_decay_mode):
- """
- generate learning rate array
-
- Args:
- lr_init(float): init learning rate
- lr_end(float): end learning rate
- lr_max(float): max learning rate
- warmup_epochs(int): number of warmup epochs
- total_epochs(int): total epoch of training
- steps_per_epoch(int): steps of one epoch
- lr_decay_mode(string): learning rate decay mode, including steps, poly or default
-
- Returns:
- np.array, learning rate array
- """
- lr_each_step = []
- total_steps = steps_per_epoch * total_epochs
- warmup_steps = steps_per_epoch * warmup_epochs
- if lr_decay_mode == 'steps':
- decay_epoch_index = [0.3 * total_steps, 0.6 * total_steps, 0.8 * total_steps]
- for i in range(total_steps):
- if i < decay_epoch_index[0]:
- lr = lr_max
- elif i < decay_epoch_index[1]:
- lr = lr_max * 0.1
- elif i < decay_epoch_index[2]:
- lr = lr_max * 0.01
- else:
- lr = lr_max * 0.001
- lr_each_step.append(lr)
- elif lr_decay_mode == 'poly':
- if warmup_steps != 0:
- inc_each_step = (float(lr_max) - float(lr_init)) / float(warmup_steps)
- else:
- inc_each_step = 0
- for i in range(total_steps):
- if i < warmup_steps:
- lr = float(lr_init) + inc_each_step * float(i)
- else:
- base = (1.0 - (float(i) - float(warmup_steps)) / (float(total_steps) - float(warmup_steps)))
- lr = float(lr_max) * base * base
- if lr < 0.0:
- lr = 0.0
- lr_each_step.append(lr)
- elif lr_decay_mode == 'cosine':
- decay_steps = total_steps - warmup_steps
- for i in range(total_steps):
- if i < warmup_steps:
- lr_inc = (float(lr_max) - float(lr_init)) / float(warmup_steps)
- lr = float(lr_init) + lr_inc * (i + 1)
- else:
- linear_decay = (total_steps - i) / decay_steps
- cosine_decay = 0.5 * (1 + math.cos(math.pi * 2 * 0.47 * i / decay_steps))
- decayed = linear_decay * cosine_decay + 0.00001
- lr = lr_max * decayed
- lr_each_step.append(lr)
- else:
- for i in range(total_steps):
- if i < warmup_steps:
- lr = lr_init + (lr_max - lr_init) * i / warmup_steps
- else:
- lr = lr_max - (lr_max - lr_end) * (i - warmup_steps) / (total_steps - warmup_steps)
- lr_each_step.append(lr)
-
- lr_each_step = np.array(lr_each_step).astype(np.float32)
-
- return lr_each_step
-
-
-def linear_warmup_lr(current_step, warmup_steps, base_lr, init_lr):
- lr_inc = (float(base_lr) - float(init_lr)) / float(warmup_steps)
- lr = float(init_lr) + lr_inc * current_step
- return lr
-
-
-def warmup_cosine_annealing_lr(lr, steps_per_epoch, warmup_epochs, max_epoch=120, global_step=0):
- """
- generate learning rate array with cosine
-
- Args:
- lr(float): base learning rate
- steps_per_epoch(int): steps size of one epoch
- warmup_epochs(int): number of warmup epochs
- max_epoch(int): total epochs of training
- global_step(int): the current start index of lr array
- Returns:
- np.array, learning rate array
- """
- base_lr = lr
- warmup_init_lr = 0
- total_steps = int(max_epoch * steps_per_epoch)
- warmup_steps = int(warmup_epochs * steps_per_epoch)
- decay_steps = total_steps - warmup_steps
-
- lr_each_step = []
- for i in range(total_steps):
- if i < warmup_steps:
- lr = linear_warmup_lr(i + 1, warmup_steps, base_lr, warmup_init_lr)
- else:
- linear_decay = (total_steps - i) / decay_steps
- cosine_decay = 0.5 * (1 + math.cos(math.pi * 2 * 0.47 * i / decay_steps))
- decayed = linear_decay * cosine_decay + 0.00001
- lr = base_lr * decayed
- lr_each_step.append(lr)
-
- lr_each_step = np.array(lr_each_step).astype(np.float32)
- learning_rate = lr_each_step[global_step:]
- return learning_rate
diff --git a/research/cv/glore_res200/src/transform.py b/research/cv/glore_res200/src/transform.py
deleted file mode 100644
index cba6ea73a65fe5d5e86318557250ffb140476c1f..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/src/transform.py
+++ /dev/null
@@ -1,51 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""
-random augment class
-"""
-import numpy as np
-import mindspore.dataset.vision.py_transforms as P
-from src import transform_utils
-
-IMAGENET_DEFAULT_MEAN = (0.485, 0.456, 0.406)
-IMAGENET_DEFAULT_STD = (0.229, 0.224, 0.225)
-
-class RandAugment:
- """
- random augment
- """
- # config_str belongs to str
- # hparams belongs to dict
- def __init__(self, config_str="rand-m9-mstd0.5", hparams=None):
- hparams = hparams if hparams is not None else {}
- self.config_str = config_str
- self.hparams = hparams
-
- def __call__(self, imgs, labels, batchInfo):
- # assert the imgs object are pil_images
- ret_imgs = []
- ret_labels = []
- py_to_pil_op = P.ToPIL()
- to_tensor = P.ToTensor()
- normalize_op = P.Normalize(IMAGENET_DEFAULT_MEAN, IMAGENET_DEFAULT_STD)
- rand_augment_ops = transform_utils.rand_augment_transform(self.config_str, self.hparams)
- for i, image in enumerate(imgs):
- img_pil = py_to_pil_op(image)
- img_pil = rand_augment_ops(img_pil)
- img_array = to_tensor(img_pil)
- img_array = normalize_op(img_array)
- ret_imgs.append(img_array)
- ret_labels.append(labels[i])
- return np.array(ret_imgs), np.array(ret_labels)
diff --git a/research/cv/glore_res200/src/transform_utils.py b/research/cv/glore_res200/src/transform_utils.py
deleted file mode 100644
index c8a1301fca1b607422b53a0b959d93701d29daa4..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/src/transform_utils.py
+++ /dev/null
@@ -1,594 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""
-augment operation
-"""
-import math
-import random
-import re
-import numpy as np
-import PIL
-from PIL import Image, ImageEnhance, ImageOps
-
-_PIL_VER = tuple([int(x) for x in PIL.__version__.split('.')[:2]])
-_FILL = (128, 128, 128)
-_MAX_LEVEL = 10.
-_HPARAMS_DEFAULT = dict(translate_const=250, img_mean=_FILL)
-_RAND_TRANSFORMS = [
- 'Distort',
- 'Zoom',
- 'Blur',
- 'Skew',
- 'AutoContrast',
- 'Equalize',
- 'Invert',
- 'Rotate',
- 'PosterizeTpu',
- 'Solarize',
- 'SolarizeAdd',
- 'Color',
- 'Contrast',
- 'Brightness',
- 'Sharpness',
- 'ShearX',
- 'ShearY',
- 'TranslateXRel',
- 'TranslateYRel',
-]
-_RANDOM_INTERPOLATION = (Image.BILINEAR, Image.BICUBIC)
-_RAND_CHOICE_WEIGHTS_0 = {
- 'Rotate': 0.3,
- 'ShearX': 0.2,
- 'ShearY': 0.2,
- 'TranslateXRel': 0.1,
- 'TranslateYRel': 0.1,
- 'Color': .025,
- 'Sharpness': 0.025,
- 'AutoContrast': 0.025,
- 'Solarize': .005,
- 'SolarizeAdd': .005,
- 'Contrast': .005,
- 'Brightness': .005,
- 'Equalize': .005,
- 'PosterizeTpu': 0,
- 'Invert': 0,
- 'Distort': 0,
- 'Zoom': 0,
- 'Blur': 0,
- 'Skew': 0,
-}
-
-
-def _interpolation(kwargs):
- interpolation = kwargs.pop('resample', Image.BILINEAR)
- if isinstance(interpolation, (list, tuple)):
- return random.choice(interpolation)
- return interpolation
-
-
-def _check_args_tf(kwargs):
- if 'fillcolor' in kwargs and _PIL_VER < (5, 0):
- kwargs.pop('fillcolor')
- kwargs['resample'] = _interpolation(kwargs)
-
-# define all kinds of functions
-
-
-def _randomly_negate(v):
- return -v if random.random() > 0.5 else v
-
-
-def shear_x(img, factor, **kwargs):
- _check_args_tf(kwargs)
- return img.transform(img.size, Image.AFFINE, (1, factor, 0, 0, 1, 0), **kwargs)
-
-
-def shear_y(img, factor, **kwargs):
- _check_args_tf(kwargs)
- return img.transform(img.size, Image.AFFINE, (1, 0, 0, factor, 1, 0), **kwargs)
-
-
-def translate_x_rel(img, pct, **kwargs):
- pixels = pct * img.size[0]
- _check_args_tf(kwargs)
- return img.transform(img.size, Image.AFFINE, (1, 0, pixels, 0, 1, 0), **kwargs)
-
-
-def translate_y_rel(img, pct, **kwargs):
- pixels = pct * img.size[1]
- _check_args_tf(kwargs)
- return img.transform(img.size, Image.AFFINE, (1, 0, 0, 0, 1, pixels), **kwargs)
-
-
-def translate_x_abs(img, pixels, **kwargs):
- _check_args_tf(kwargs)
- return img.transform(img.size, Image.AFFINE, (1, 0, pixels, 0, 1, 0), **kwargs)
-
-
-def translate_y_abs(img, pixels, **kwargs):
- _check_args_tf(kwargs)
- return img.transform(img.size, Image.AFFINE, (1, 0, 0, 0, 1, pixels), **kwargs)
-
-
-def rotate(img, degrees, **kwargs):
- """
- rotate operation
- """
- kwargs_new = kwargs
- kwargs_new.pop('resample')
- kwargs_new['resample'] = Image.BICUBIC
- if _PIL_VER >= (5, 2):
- return img.rotate(degrees, **kwargs_new)
- if _PIL_VER >= (5, 0):
- w, h = img.size
- post_trans = (0, 0)
- rotn_center = (w / 2.0, h / 2.0)
- angle = -math.radians(degrees)
- matrix = [
- round(math.cos(angle), 15),
- round(math.sin(angle), 15),
- 0.0,
- round(-math.sin(angle), 15),
- round(math.cos(angle), 15),
- 0.0,
- ]
-
- def transform(x, y, matrix):
- (a, b, c, d, e, f) = matrix
- return a * x + b * y + c, d * x + e * y + f
-
- matrix[2], matrix[5] = transform(
- -rotn_center[0] - post_trans[0], -rotn_center[1] - post_trans[1], matrix
- )
- matrix[2] += rotn_center[0]
- matrix[5] += rotn_center[1]
- return img.transform(img.size, Image.AFFINE, matrix, **kwargs_new)
- return img.rotate(degrees, resample=kwargs['resample'])
-
-
-def auto_contrast(img, **__):
- return ImageOps.autocontrast(img)
-
-
-def invert(img, **__):
- return ImageOps.invert(img)
-
-
-def equalize(img, **__):
- return ImageOps.equalize(img)
-
-
-def solarize(img, thresh, **__):
- return ImageOps.solarize(img, thresh)
-
-
-def solarize_add(img, add, thresh=128, **__):
- """
- add solarize operation
- """
- lut = []
- for i in range(256):
- if i < thresh:
- lut.append(min(255, i + add))
- else:
- lut.append(i)
- if img.mode in ("L", "RGB"):
- if img.mode == "RGB" and len(lut) == 256:
- lut = lut + lut + lut
- return img.point(lut)
- return img
-
-
-def posterize(img, bits_to_keep, **__):
- if bits_to_keep >= 8:
- return img
- return ImageOps.posterize(img, bits_to_keep)
-
-
-def contrast(img, factor, **__):
- return ImageEnhance.Contrast(img).enhance(factor)
-
-
-def color(img, factor, **__):
- return ImageEnhance.Color(img).enhance(factor)
-
-
-def brightness(img, factor, **__):
- return ImageEnhance.Brightness(img).enhance(factor)
-
-
-def sharpness(img, factor, **__):
- return ImageEnhance.Sharpness(img).enhance(factor)
-
-
-def _rotate_level_to_arg(level, _hparams):
- # range [-30, 30]
- level = (level / _MAX_LEVEL) * 30.
- level = _randomly_negate(level)
- return (level,)
-
-
-def _enhance_level_to_arg(level, _hparams):
- # range [0.1, 1.9]
- return ((level / _MAX_LEVEL) * 1.8 + 0.1,)
-
-
-def _shear_level_to_arg(level, _hparams):
- # range [-0.3, 0.3]
- level = (level / _MAX_LEVEL) * 0.3
- level = _randomly_negate(level)
- return (level,)
-
-
-def _translate_abs_level_to_arg(level, hparams):
- translate_const = hparams['translate_const']
- level = (level / _MAX_LEVEL) * float(translate_const)
- level = _randomly_negate(level)
- return (level,)
-
-
-def _translate_rel_level_to_arg(level, _hparams):
- # range [-0.45, 0.45]
- level = (level / _MAX_LEVEL) * 0.45
- level = _randomly_negate(level)
- return (level,)
-
-
-def _posterize_original_level_to_arg(level, _hparams):
- # As per original AutoAugment paper description
- # range [4, 8], 'keep 4 up to 8 MSB of image'
- return (int((level / _MAX_LEVEL) * 4) + 4,)
-
-
-def _posterize_research_level_to_arg(level, _hparams):
- # As per Tensorflow models research and UDA impl
- # range [4, 0], 'keep 4 down to 0 MSB of original image'
- return (4 - int((level / _MAX_LEVEL) * 4),)
-
-
-def _posterize_tpu_level_to_arg(level, _hparams):
- # As per Tensorflow TPU EfficientNet impl
- # range [0, 4], 'keep 0 up to 4 MSB of original image'
- return (int((level / _MAX_LEVEL) * 4),)
-
-
-def _solarize_level_to_arg(level, _hparams):
- # range [0, 256]
- return (int((level / _MAX_LEVEL) * 256),)
-
-
-def _solarize_add_level_to_arg(level, _hparams):
- # range [0, 110]
- return (int((level / _MAX_LEVEL) * 110),)
-
-
-def _distort_level_to_arg(level, _hparams):
- return (int((level / _MAX_LEVEL) * 10 + 10),)
-
-
-def _zoom_level_to_arg(level, _hparams):
- return ((level / _MAX_LEVEL) * 0.4,)
-
-
-def _blur_level_to_arg(level, _hparams):
- level = (level / _MAX_LEVEL) * 0.5
- level = _randomly_negate(level)
- return (level,)
-
-
-def _skew_level_to_arg(level, _hparams):
- level = (level / _MAX_LEVEL) * 0.3
- level = _randomly_negate(level)
- return (level,)
-
-
-def distort(img, v, **__):
- """
- distort operation
- """
- w, h = img.size
- horizontal_tiles = int(0.1 * v)
- vertical_tiles = int(0.1 * v)
-
- width_of_square = int(math.floor(w / float(horizontal_tiles)))
- height_of_square = int(math.floor(h / float(vertical_tiles)))
- width_of_last_square = w - (width_of_square * (horizontal_tiles - 1))
- height_of_last_square = h - (height_of_square * (vertical_tiles - 1))
- dimensions = []
-
- for vertical_tile in range(vertical_tiles):
- for horizontal_tile in range(horizontal_tiles):
- if vertical_tile == (vertical_tiles - 1) and horizontal_tile == (horizontal_tiles - 1):
- dimensions.append([horizontal_tile * width_of_square,
- vertical_tile * height_of_square,
- width_of_last_square + (horizontal_tile * width_of_square),
- height_of_last_square + (height_of_square * vertical_tile)])
- elif vertical_tile == (vertical_tiles - 1):
- dimensions.append([horizontal_tile * width_of_square,
- vertical_tile * height_of_square,
- width_of_square + (horizontal_tile * width_of_square),
- height_of_last_square + (height_of_square * vertical_tile)])
- elif horizontal_tile == (horizontal_tiles - 1):
- dimensions.append([horizontal_tile * width_of_square,
- vertical_tile * height_of_square,
- width_of_last_square + (horizontal_tile * width_of_square),
- height_of_square + (height_of_square * vertical_tile)])
- else:
- dimensions.append([horizontal_tile * width_of_square,
- vertical_tile * height_of_square,
- width_of_square + (horizontal_tile * width_of_square),
- height_of_square + (height_of_square * vertical_tile)])
- last_column = []
- for i in range(vertical_tiles):
- last_column.append((horizontal_tiles - 1) + horizontal_tiles * i)
-
- last_row = range((horizontal_tiles * vertical_tiles) - horizontal_tiles, horizontal_tiles * vertical_tiles)
-
- polygons = []
- for x1, y1, x2, y2 in dimensions:
- polygons.append([x1, y1, x1, y2, x2, y2, x2, y1])
-
- polygon_indices = []
- for i in range((vertical_tiles * horizontal_tiles) - 1):
- if i not in last_row and i not in last_column:
- polygon_indices.append([i, i + 1, i + horizontal_tiles, i + 1 + horizontal_tiles])
-
- for a, b, c, d in polygon_indices:
- dx = v
- dy = v
-
- x1, y1, x2, y2, x3, y3, x4, y4 = polygons[a]
- polygons[a] = [x1, y1,
- x2, y2,
- x3 + dx, y3 + dy,
- x4, y4]
-
- x1, y1, x2, y2, x3, y3, x4, y4 = polygons[b]
- polygons[b] = [x1, y1,
- x2 + dx, y2 + dy,
- x3, y3,
- x4, y4]
-
- x1, y1, x2, y2, x3, y3, x4, y4 = polygons[c]
- polygons[c] = [x1, y1,
- x2, y2,
- x3, y3,
- x4 + dx, y4 + dy]
-
- x1, y1, x2, y2, x3, y3, x4, y4 = polygons[d]
- polygons[d] = [x1 + dx, y1 + dy,
- x2, y2,
- x3, y3,
- x4, y4]
-
- generated_mesh = []
- for idx, i in enumerate(dimensions):
- generated_mesh.append([dimensions[idx], polygons[idx]])
- return img.transform(img.size, PIL.Image.MESH, generated_mesh, resample=PIL.Image.BICUBIC)
-
-
-def zoom(img, v, **__):
- #assert 0.1 <= v <= 2
- w, h = img.size
- image_zoomed = img.resize((int(round(img.size[0] * v)),
- int(round(img.size[1] * v))),
- resample=PIL.Image.BICUBIC)
- w_zoomed, h_zoomed = image_zoomed.size
-
- return image_zoomed.crop((math.floor((float(w_zoomed) / 2) - (float(w) / 2)),
- math.floor((float(h_zoomed) / 2) - (float(h) / 2)),
- math.floor((float(w_zoomed) / 2) + (float(w) / 2)),
- math.floor((float(h_zoomed) / 2) + (float(h) / 2))))
-
-
-def erase(img, v, **__):
- """
- distort operation
- """
- #assert 0.1<= v <= 1
- w, h = img.size
- w_occlusion = int(w * v)
- h_occlusion = int(h * v)
- if len(img.getbands()) == 1:
- rectangle = PIL.Image.fromarray(np.uint8(np.random.rand(w_occlusion, h_occlusion) * 255))
- else:
- rectangle = PIL.Image.fromarray(np.uint8(np.random.rand(w_occlusion, h_occlusion, len(img.getbands())) * 255))
-
- random_position_x = random.randint(0, w - w_occlusion)
- random_position_y = random.randint(0, h - h_occlusion)
- img.paste(rectangle, (random_position_x, random_position_y))
- return img
-
-
-def skew(img, v, **__):
- """
- skew operation
- """
- #assert -1 <= v <= 1
- w, h = img.size
- x1 = 0
- x2 = h
- y1 = 0
- y2 = w
- original_plane = [(y1, x1), (y2, x1), (y2, x2), (y1, x2)]
- max_skew_amount = max(w, h)
- max_skew_amount = int(math.ceil(max_skew_amount * v))
- skew_amount = max_skew_amount
- new_plane = [(y1 - skew_amount, x1), # Top Left
- (y2, x1 - skew_amount), # Top Right
- (y2 + skew_amount, x2), # Bottom Right
- (y1, x2 + skew_amount)]
- matrix = []
- for p1, p2 in zip(new_plane, original_plane):
- matrix.append([p1[0], p1[1], 1, 0, 0, 0, -p2[0] * p1[0], -p2[0] * p1[1]])
- matrix.append([0, 0, 0, p1[0], p1[1], 1, -p2[1] * p1[0], -p2[1] * p1[1]])
-
- A = np.matrix(matrix, dtype=np.float)
- B = np.array(original_plane).reshape(8)
- perspective_skew_coefficients_matrix = np.dot(np.linalg.pinv(A), B)
- perspective_skew_coefficients_matrix = np.array(perspective_skew_coefficients_matrix).reshape(8)
-
- return img.transform(img.size, PIL.Image.PERSPECTIVE, perspective_skew_coefficients_matrix,
- resample=PIL.Image.BICUBIC)
-
-
-def blur(img, v, **__):
- #assert -3 <= v <= 3
- return img.filter(PIL.ImageFilter.GaussianBlur(v))
-
-
-def rand_augment_ops(magnitude=10, hparams=None, transforms=None):
- hparams = hparams or _HPARAMS_DEFAULT
- transforms = transforms or _RAND_TRANSFORMS
- return [AutoAugmentOp(name, prob=0.5, magnitude=magnitude, hparams=hparams) for name in transforms]
-
-
-def _select_rand_weights(weight_idx=0, transforms=None):
- transforms = transforms or _RAND_TRANSFORMS
- assert weight_idx == 0 # only one set of weights currently
- rand_weights = _RAND_CHOICE_WEIGHTS_0
- probs = [rand_weights[k] for k in transforms]
- probs /= np.sum(probs)
- return probs
-
-
-def rand_augment_transform(config_str, hparams):
- """
- rand selcet transform operation
- """
- magnitude = _MAX_LEVEL # default to _MAX_LEVEL for magnitude (currently 10)
- num_layers = 2 # default to 2 ops per image
- weight_idx = None # default to no probability weights for op choice
- config = config_str.split('-')
- assert config[0] == 'rand'
- config = config[1:]
- for c in config:
- cs = re.split(r'(\d.*)', c)
- if len(cs) < 2:
- continue
- key, val = cs[:2]
- if key == 'mstd':
- # noise param injected via hparams for now
- hparams.setdefault('magnitude_std', float(val))
- elif key == 'm':
- magnitude = int(val)
- elif key == 'n':
- num_layers = int(val)
- elif key == 'w':
- weight_idx = int(val)
- else:
- assert False, 'Unknown RandAugment config section'
- ra_ops = rand_augment_ops(magnitude=magnitude, hparams=hparams)
- choice_weights = None if weight_idx is None else _select_rand_weights(weight_idx)
-
- final_result = RandAugment(ra_ops, num_layers, choice_weights=choice_weights)
- return final_result
-
-
-LEVEL_TO_ARG = {
- 'Distort': _distort_level_to_arg,
- 'Zoom': _zoom_level_to_arg,
- 'Blur': _blur_level_to_arg,
- 'Skew': _skew_level_to_arg,
- 'AutoContrast': None,
- 'Equalize': None,
- 'Invert': None,
- 'Rotate': _rotate_level_to_arg,
- 'PosterizeOriginal': _posterize_original_level_to_arg,
- 'PosterizeResearch': _posterize_research_level_to_arg,
- 'PosterizeTpu': _posterize_tpu_level_to_arg,
- 'Solarize': _solarize_level_to_arg,
- 'SolarizeAdd': _solarize_add_level_to_arg,
- 'Color': _enhance_level_to_arg,
- 'Contrast': _enhance_level_to_arg,
- 'Brightness': _enhance_level_to_arg,
- 'Sharpness': _enhance_level_to_arg,
- 'ShearX': _shear_level_to_arg,
- 'ShearY': _shear_level_to_arg,
- 'TranslateX': _translate_abs_level_to_arg,
- 'TranslateY': _translate_abs_level_to_arg,
- 'TranslateXRel': _translate_rel_level_to_arg,
- 'TranslateYRel': _translate_rel_level_to_arg,
-}
-
-NAME_TO_OP = {
- 'Distort': distort,
- 'Zoom': zoom,
- 'Blur': blur,
- 'Skew': skew,
- 'AutoContrast': auto_contrast,
- 'Equalize': equalize,
- 'Invert': invert,
- 'Rotate': rotate,
- 'PosterizeOriginal': posterize,
- 'PosterizeResearch': posterize,
- 'PosterizeTpu': posterize,
- 'Solarize': solarize,
- 'SolarizeAdd': solarize_add,
- 'Color': color,
- 'Contrast': contrast,
- 'Brightness': brightness,
- 'Sharpness': sharpness,
- 'ShearX': shear_x,
- 'ShearY': shear_y,
- 'TranslateX': translate_x_abs,
- 'TranslateY': translate_y_abs,
- 'TranslateXRel': translate_x_rel,
- 'TranslateYRel': translate_y_rel,
-}
-
-
-class AutoAugmentOp:
- """
- AutoAugmentOp class
- """
- def __init__(self, name, prob=0.5, magnitude=10, hparams=None):
- hparams = hparams or _HPARAMS_DEFAULT
- self.aug_fn = NAME_TO_OP[name]
- self.level_fn = LEVEL_TO_ARG[name]
- self.prob = prob
- self.magnitude = magnitude
- self.hparams = hparams.copy()
- self.kwargs = dict(
- fillcolor=hparams['img_mean'] if 'img_mean' in hparams else _FILL,
- resample=hparams['interpolation'] if 'interpolation' in hparams else _RANDOM_INTERPOLATION,
- )
- self.magnitude_std = self.hparams.get('magnitude_std', 0)
-
- def __call__(self, img):
- if random.random() > self.prob:
- return img
- magnitude = self.magnitude
- if self.magnitude_std and self.magnitude_std > 0:
- magnitude = random.gauss(magnitude, self.magnitude_std)
- level_args = self.level_fn(magnitude, self.hparams) if self.level_fn is not None else tuple()
- return self.aug_fn(img, *level_args, **self.kwargs)
-
-
-class RandAugment:
- """
- rand augment class
- """
- def __init__(self, ops, num_layers=2, choice_weights=None):
- self.ops = ops
- self.num_layers = num_layers
- self.choice_weights = choice_weights
-
- def __call__(self, img):
- ops = np.random.choice(
- self.ops, self.num_layers, replace=self.choice_weights is None, p=self.choice_weights)
- for op in ops:
- img = op(img)
- return img
diff --git a/research/cv/glore_res200/train.py b/research/cv/glore_res200/train.py
deleted file mode 100644
index 3768a652baaff29d8a7e5427433ed887a693af79..0000000000000000000000000000000000000000
--- a/research/cv/glore_res200/train.py
+++ /dev/null
@@ -1,194 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""
-#################train glore_resnet200 on Imagenet2012########################
-python train.py
-"""
-import os
-import random
-import argparse
-import ast
-import numpy as np
-from mindspore import Tensor
-from mindspore import context
-from mindspore import dataset as de
-from mindspore.context import ParallelMode
-from mindspore.train.model import Model
-from mindspore.train.callback import ModelCheckpoint, CheckpointConfig, LossMonitor, TimeMonitor
-from mindspore.train.loss_scale_manager import FixedLossScaleManager
-from mindspore.train.serialization import load_checkpoint, load_param_into_net
-from mindspore.communication.management import init, get_group_size, get_rank
-import mindspore.nn as nn
-from mindspore.common import set_seed
-import mindspore.common.initializer as weight_init
-from src.lr_generator import get_lr
-from src.config import config1, config2
-from src.glore_resnet200 import glore_resnet200
-from src.dataset import create_dataset_ImageNet as get_dataset
-from src.loss import SoftmaxCrossEntropyExpand
-
-
-parser = argparse.ArgumentParser(description='Image classification with glore_resnet200')
-parser.add_argument('--use_glore', type=ast.literal_eval, default=True, help='Enable GloreUnit')
-parser.add_argument('--cast_fp16', type=ast.literal_eval, default=True, help='GloreUnit cast to fp16 for accelerating')
-parser.add_argument('--run_distribute', type=ast.literal_eval, default=True, help='Run distribute')
-parser.add_argument('--data_url', type=str, default=None, help='Dataset path')
-parser.add_argument('--train_url', type=str)
-parser.add_argument('--device_target', type=str, default='Ascend', help='Device target')
-parser.add_argument('--device_num', type=int, default=1, help='Number of device')
-parser.add_argument('--device_id', type=int, default=0)
-parser.add_argument('--pre_trained', type=ast.literal_eval, default=False)
-parser.add_argument('--pre_ckpt_path', type=str, default='')
-parser.add_argument('--parameter_server', type=ast.literal_eval, default=False, help='Run parameter server train')
-parser.add_argument('--isModelArts', type=ast.literal_eval, default=True)
-args_opt = parser.parse_args()
-
-if args_opt.isModelArts:
- import moxing as mox
-if args_opt.device_target == "Ascend":
- config = config1
-elif args_opt.device_target == "GPU":
- config = config2
- args_opt.cast_fp16 = False
-
-random.seed(1)
-np.random.seed(1)
-de.config.set_seed(1)
-set_seed(1)
-
-if __name__ == '__main__':
-
- target = args_opt.device_target
- ckpt_save_dir = config.save_checkpoint_path
- # init context
- context.set_context(mode=context.GRAPH_MODE, device_target=target, save_graphs=False)
- if args_opt.run_distribute:
- if target == "Ascend":
- device_id = int(os.getenv('DEVICE_ID'))
- context.set_context(device_id=device_id)
- context.set_auto_parallel_context(parallel_mode=ParallelMode.DATA_PARALLEL,
- gradients_mean=True,
- auto_parallel_search_mode="recursive_programming")
- init()
- elif target == "GPU":
- init()
- context.set_auto_parallel_context(device_num=get_group_size(),
- parallel_mode=ParallelMode.DATA_PARALLEL,
- gradients_mean=True)
- else:
- if target == "Ascend":
- device_id = args_opt.device_id
- context.set_context(mode=context.GRAPH_MODE, device_target=target, save_graphs=False,
- device_id=device_id)
-
- train_dataset_path = args_opt.data_url
- if args_opt.isModelArts:
- # download dataset from obs to cache
- mox.file.copy_parallel(src_url=args_opt.data_url, dst_url='/cache/dataset/device_' + os.getenv('DEVICE_ID'))
- train_dataset_path = '/cache/dataset/device_' + os.getenv('DEVICE_ID')
- # create dataset
- dataset = get_dataset(dataset_path=train_dataset_path, do_train=True, use_randaugment=True, repeat_num=1,
- batch_size=config.batch_size, target=target)
- step_size = dataset.get_dataset_size()
-
- # define net
-
- net = glore_resnet200(cast_fp16=args_opt.cast_fp16, class_num=config.class_num, use_glore=args_opt.use_glore)
-
- # init weight
- if args_opt.pre_trained:
- param_dict = load_checkpoint(args_opt.pre_ckpt_path)
- load_param_into_net(net, param_dict)
- else:
- for _, cell in net.cells_and_names():
- if isinstance(cell, nn.Conv2d):
- cell.weight.default_input = weight_init.initializer(weight_init.XavierUniform(),
- cell.weight.shape,
- cell.weight.dtype)
- if isinstance(cell, nn.Dense):
- cell.weight.default_input = weight_init.initializer(weight_init.TruncatedNormal(),
- cell.weight.shape,
- cell.weight.dtype)
-
- # init lr
- lr = get_lr(lr_init=config.lr_init,
- lr_end=config.lr_end,
- lr_max=config.lr_max,
- warmup_epochs=config.warmup_epochs,
- total_epochs=config.epoch_size,
- steps_per_epoch=step_size,
- lr_decay_mode=config.lr_decay_mode)
- lr = Tensor(lr)
-
- #
- # define opt
- decayed_params = []
- no_decayed_params = []
- for param in net.trainable_params():
- if 'beta' not in param.name and 'gamma' not in param.name and 'bias' not in param.name:
- decayed_params.append(param)
- else:
- no_decayed_params.append(param)
-
- group_params = [{'params': decayed_params, 'weight_decay': config.weight_decay},
- {'params': no_decayed_params},
- {'order_params': net.trainable_params()}]
- net_opt = nn.SGD(group_params, learning_rate=lr, momentum=config.momentum, weight_decay=config.weight_decay,
- loss_scale=config.loss_scale, nesterov=True)
-
- # define loss, model
- loss = SoftmaxCrossEntropyExpand(sparse=True)
- loss_scale = FixedLossScaleManager(config.loss_scale, drop_overflow_update=False)
- model = Model(net, loss_fn=loss, optimizer=net_opt, loss_scale_manager=loss_scale)
-
- # define callbacks
- time_cb = TimeMonitor(data_size=step_size)
- loss_cb = LossMonitor()
- cb = [time_cb, loss_cb]
- if config.save_checkpoint:
- config_ck = CheckpointConfig(save_checkpoint_steps=config.save_checkpoint_epochs * step_size,
- keep_checkpoint_max=config.keep_checkpoint_max)
- if args_opt.isModelArts:
- save_checkpoint_path = '/cache/train_output/device_' + os.getenv('DEVICE_ID') + '/'
- if args_opt.device_num == 1:
- ckpt_cb = ModelCheckpoint(prefix='glore_resnet200',
- directory=save_checkpoint_path,
- config=config_ck)
- cb += [ckpt_cb]
- if args_opt.device_num > 1 and get_rank() % 8 == 0:
- ckpt_cb = ModelCheckpoint(prefix='glore_resnet200',
- directory=save_checkpoint_path,
- config=config_ck)
- cb += [ckpt_cb]
- else:
- save_checkpoint_path = config.save_checkpoint_path
- if not os.path.isdir(save_checkpoint_path):
- os.makedirs(save_checkpoint_path)
-
- if args_opt.device_num == 1:
- ckpt_cb = ModelCheckpoint(prefix='glore_resnet200',
- directory=save_checkpoint_path,
- config=config_ck)
- cb += [ckpt_cb]
- if args_opt.device_num > 1 and get_rank() % 8 == 0:
- ckpt_cb = ModelCheckpoint(prefix='glore_resnet200',
- directory=save_checkpoint_path,
- config=config_ck)
- cb += [ckpt_cb]
-
- model.train(config.epoch_size - config.pretrain_epoch_size, dataset,
- callbacks=cb, dataset_sink_mode=True)
- if args_opt.isModelArts:
- mox.file.copy_parallel(src_url='/cache/train_output', dst_url=args_opt.train_url)
diff --git a/research/cv/glore_res50/README.md b/research/cv/glore_res50/README.md
deleted file mode 100644
index bc80ce7d18f2c1fff629fabf858c86e47dcfbd61..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/README.md
+++ /dev/null
@@ -1,239 +0,0 @@
-# 目录
-
-<!-- TOC -->
-
-- [glore_res50描述](#glore_res50描述)
-- [模型架构](#模型架构)
-- [数据集](#数据集)
-- [特性](#特性)
- - [混合精度](#混合精度)
-- [环境要求](#环境要求)
-- [快速入门](#快速入门)
-- [脚本说明](#脚本说明)
- - [脚本及样例代码](#脚本及样例代码)
- - [脚本参数](#脚本参数)
- - [训练过程](#训练过程)
- - [评估过程](#评估过程)
-- [模型描述](#模型描述)
- - [性能](#性能)
- - [评估性能](#评估性能)
-- [随机情况说明](#随机情况说明)
-- [ModelZoo主页](#ModelZoo主页)
-
-<!-- /TOC -->
-
-# glore_res描述
-
-## 概述
-
-卷积神经网络擅长提取局部关系，但是在处理全局上的区域间关系时显得低效，且需要堆叠很多层才可能完成，而在区域之间进行全局建模和推理对很多计算机视觉任务有益。为了进行全局推理，facebook research、新加坡国立大学和360 AI研究所提出了基于图的全局推理模块-Global Reasoning Unit，可以被插入到很多任务的网络模型中。glore_res200是在ResNet200的Stage2, Stage3中分别均匀地插入了2和3个全局推理模块的用于图像分类任务的网络模型。
-
-如下为MindSpore使用ImageNet2012数据集对glore_res50进行训练的示例。glore_res50可参考[论文1](https://arxiv.org/pdf/1811.12814v1.pdf)
-
-## 论文
-
-1. [论文](https://arxiv.org/pdf/1811.12814v1.pdf)：Yupeng Chen, Marcus Rohrbach, Zhicheng Yan, Shuicheng Yan,
- Jiashi Feng, Yannis Kalantidis."Deep Residual Learning for Image Recognition"
-
-# 模型架构
-
-glore_res的总体网络架构如下：
-[链接](https://arxiv.org/pdf/1811.12814v1.pdf)
-
-# 数据集
-
-使用的数据集：[ImageNet2012](http://www.image-net.org/)
-
-- 数据集大小：共1000个类、224*224彩色图像
- - 训练集：共1,281,167张图像
- - 测试集：共50,000张图像
-- 数据格式：JPEG
- - 注：数据在dataset.py中处理。
-- 下载数据集，目录结构如下：
-
- ```text
-└─imagenet_original
- ├─train # 训练数据集
- └─val # 评估数据集
-```
-
-# 特性
-
-## 混合精度
-
-采用[混合精度](https://www.mindspore.cn/tutorials/experts/zh-CN/master/others/mixed_precision.html)的训练方法使用支持单精度和半精度数据来提高深度学习神经网络的训练速度，同时保持单精度训练所能达到的网络精度。混合精度训练提高计算速度、减少内存使用的同时，支持在特定硬件上训练更大的模型或实现更大批次的训练。
-以FP16算子为例，如果输入数据类型为FP32，MindSpore后台会自动降低精度来处理数据。用户可打开INFO日志，搜索“reduce precision”查看精度降低的算子。
-
-# 环境要求
-
-- 硬件(Ascend)
-- 框架
- - [MindSpore](https://www.mindspore.cn/install)
-- 如需查看详情，请参见如下资源：
- - [MindSpore教程](https://www.mindspore.cn/tutorials/zh-CN/master/index.html)
- - [MindSpore Python API](https://www.mindspore.cn/docs/api/zh-CN/master/index.html)
-
-# 快速入门
-
-通过官方网站安装MindSpore后，您可以按照如下步骤进行训练和评估：
-
-- Ascend处理器环境运行
-
-```text
-# 分布式训练
-用法：bash run_distribute_train.sh [DATA_PATH] [DEVICE_NUM]
-
-# 单机训练
-用法：bash run_standalone_train.sh [DATA_PATH] [DEVICE_ID]
-
-# 运行评估示例
-用法：bash run_eval.sh [DATA_PATH] [DEVICE_ID] [CKPT_PATH]
-```
-
-# 脚本说明
-
-## 脚本及样例代码
-
-```shell
-.
-└──glore_res50
- ├── README.md
- ├── scripts
- ├── run_distribute_train.sh # 启动Ascend分布式训练（8卡）
- ├── run_eval.sh # 启动Ascend评估
- ├── run_standalone_train.sh # 启动Ascend单机训练（单卡）
- ├── src
- ├── __init__.py
- ├── autoaugment.py # AutoAugment组件与类
- ├── config.py # 参数配置
- ├── dataset.py # 数据预处理
- ├── glore_res50.py # glore_res50网络定义
- ├── loss.py # ImageNet2012数据集的损失定义
- ├── save_callback.py # 训练时推理并保存最优精度下的参数
- └── lr_generator.py # 生成每个步骤的学习率
- ├── eval.py # 评估网络
- └── train.py # 训练网络
-```
-
-## 脚本参数
-
-在config.py中可以同时配置训练参数和评估参数。
-
-- 配置glore_res50和ImageNet2012数据集。
-
-```text
-"class_num":1000, # 数据集类数
-"batch_size":128, # 输入张量的批次大小
-"loss_scale":1024, # 损失等级
-"momentum":0.9, # 动量优化器
-"weight_decay":1e-4, # 权重衰减
-"epoch_size":120, # 此值仅适用于训练；应用于推理时固定为1
-"pretrained": False, # 加载预训练权重
-"pretrain_epoch_size": 0, # 加载预训练检查点之前已经训练好的模型的周期大小；实际训练周期大小等于epoch_size减去pretrain_epoch_size
-"save_checkpoint":True, # 是否保存检查点
-"save_checkpoint_epochs":5, # 两个检查点之间的周期间隔；默认情况下，最后一个检查点将在最后一个周期完成后保存
-"keep_checkpoint_max":10, # 只保存最后一个keep_checkpoint_max检查点
-"save_checkpoint_path":"./", # 检查点相对于执行路径的保存路径
-"warmup_epochs":0, # 热身周期数
-"lr_decay_mode":"Linear", # 用于生成学习率的衰减模式
-"use_label_smooth":True, # 标签平滑
-"label_smooth_factor":0.05, # 标签平滑因子
-"weight_init": "xavier_uniform", # 权重初始化方式,可选"he_normal", "he_uniform", "xavier_uniform"
-"use_autoaugment": True, # 是否应用AutoAugment方法
-"lr_init":0, # 初始学习率
-"lr_max":0.8, # 最大学习率
-"lr_end":0.0, # 最小学习率
-```
-
-## 训练过程
-
-### 用法
-
-#### Ascend处理器环境运行
-
-```text
-# 分布式训练
-用法：bash run_distribute_train.sh [DATA_PATH] [DEVICE_NUM]
-
-# 单机训练
-用法：bash run_standalone_train.sh [DATA_PATH] [DEVICE_ID]
-
-```
-
-分布式训练需要提前创建JSON格式的HCCL配置文件。
-
-具体操作，参见[hccn_tools](https://gitee.com/mindspore/models/tree/master/utils/hccl_tools)中的说明。
-
-### 结果
-
-- 使用ImageNet2012数据集训练glore_res50
-
-```text
-# 分布式训练结果（8P）
-epoch:1 step:1251, loss is 5.721338
-epoch:2 step:1251, loss is 4.8941164
-epoch:3 step:1251, loss is 4.3002024
-epoch:4 step:1251, loss is 3.862403
-epoch:5 step:1251, loss is 3.5204496
-...
-```
-
-## 评估过程
-
-### 用法
-
-#### Ascend处理器环境运行
-
-```bash
-# 评估
-用法：bash run_eval.sh [DATA_PATH] [DEVICE_ID] [CKPT_PATH]
-```
-
-```bash
-# 评估示例
-bash run_eval.sh ~/dataset/imagenet 0 ~/ckpt/glore_res50_120-1251.ckpt
-```
-
-### 结果
-
-评估结果保存在示例路径中，文件夹名为“eval”。您可在此路径下的日志找到如下结果：
-
-- 使用ImageNet2012数据集评估glore_res50
-
-```text
-{'Accuracy': 0.7844638020833334}
-```
-
-# 模型描述
-
-## 性能
-
-### 评估性能
-
-#### ImageNet2012上的glore_res50
-
-| 参数 | Ascend 910
-| -------------------------- | -------------------------------------- |
-| 模型版本 | glore_res50
-| 资源 | Ascend 910；CPU：2.60GHz，192核；内存：755G |
-| 上传日期 | 2021-03-21 |
-| MindSpore版本 | r1.1 |
-| 数据集 | ImageNet2012 |
-| 训练参数 | epoch=120, steps per epoch=1251, batch_size = 128 |
-| 优化器 | Momentum |
-| 损失函数 | Softmax交叉熵 |
-| 输出 | 概率 |
-| 损失 | 1.8464266 |
-| 速度 | 263.483毫秒/步（8卡）|
-| 总时长 | 10.98小时 |
-| 参数(M) | 30.5 |
-| 微调检查点| 233.46M（.ckpt文件）|
-| 脚本 | [链接](https://gitee.com/mindspore/models/tree/master/research/cv/glore_res50) |
-
-# 随机情况说明
-
-使用了train.py中的随机种子。
-
-# ModelZoo主页
-
- 请浏览官网[主页](https://gitee.com/mindspore/models)。
diff --git a/research/cv/glore_res50/__init__.py b/research/cv/glore_res50/__init__.py
deleted file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000
diff --git a/research/cv/glore_res50/ascend310_infer/CMakeLists.txt b/research/cv/glore_res50/ascend310_infer/CMakeLists.txt
deleted file mode 100644
index ee3c85447340e0449ff2b70ed24f60a17e07b2b6..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/ascend310_infer/CMakeLists.txt
+++ /dev/null
@@ -1,14 +0,0 @@
-cmake_minimum_required(VERSION 3.14.1)
-project(Ascend310Infer)
-add_compile_definitions(_GLIBCXX_USE_CXX11_ABI=0)
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O0 -g -std=c++17 -Werror -Wall -fPIE -Wl,--allow-shlib-undefined")
-set(PROJECT_SRC_ROOT ${CMAKE_CURRENT_LIST_DIR}/)
-option(MINDSPORE_PATH "mindspore install path" "")
-include_directories(${MINDSPORE_PATH})
-include_directories(${MINDSPORE_PATH}/include)
-include_directories(${PROJECT_SRC_ROOT})
-find_library(MS_LIB libmindspore.so ${MINDSPORE_PATH}/lib)
-file(GLOB_RECURSE MD_LIB ${MINDSPORE_PATH}/_c_dataengine*)
-
-add_executable(main src/main.cc src/utils.cc)
-target_link_libraries(main ${MS_LIB} ${MD_LIB} gflags)
diff --git a/research/cv/glore_res50/ascend310_infer/build.sh b/research/cv/glore_res50/ascend310_infer/build.sh
deleted file mode 100644
index 285514e19f2a1878a7bf8f0eed3c99fbc73868c4..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/ascend310_infer/build.sh
+++ /dev/null
@@ -1,29 +0,0 @@
-#!/bin/bash
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-if [ -d out ]; then
- rm -rf out
-fi
-
-mkdir out
-cd out || exit
-
-if [ -f "Makefile" ]; then
- make clean
-fi
-
-cmake .. \
- -DMINDSPORE_PATH="`pip3.7 show mindspore-ascend | grep Location | awk '{print $2"/mindspore"}' | xargs realpath`"
-make
diff --git a/research/cv/glore_res50/ascend310_infer/inc/utils.h b/research/cv/glore_res50/ascend310_infer/inc/utils.h
deleted file mode 100644
index f8ae1e5b473d869b77af8d725a280d7c7665527c..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/ascend310_infer/inc/utils.h
+++ /dev/null
@@ -1,35 +0,0 @@
-/**
- * Copyright 2021 Huawei Technologies Co., Ltd
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-#ifndef MINDSPORE_INFERENCE_UTILS_H_
-#define MINDSPORE_INFERENCE_UTILS_H_
-
-#include <sys/stat.h>
-#include <dirent.h>
-#include <vector>
-#include <string>
-#include <memory>
-#include "include/api/types.h"
-
-std::vector<std::string> GetAllFiles(std::string_view dirName);
-DIR *OpenDir(std::string_view dirName);
-std::string RealPath(std::string_view path);
-mindspore::MSTensor ReadFileToTensor(const std::string &file);
-int WriteResult(const std::string& imageFile, const std::vector<mindspore::MSTensor> &outputs);
-std::vector<std::string> GetAllFiles(std::string dir_name);
-std::vector<std::vector<std::string>> GetAllInputData(std::string dir_name);
-
-#endif
diff --git a/research/cv/glore_res50/ascend310_infer/src/main.cc b/research/cv/glore_res50/ascend310_infer/src/main.cc
deleted file mode 100644
index 1b1f2a91f01ef99b8dc63ef5d201fa3219e5d141..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/ascend310_infer/src/main.cc
+++ /dev/null
@@ -1,152 +0,0 @@
-/**
- * Copyright 2021 Huawei Technologies Co., Ltd
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-#include <sys/time.h>
-#include <gflags/gflags.h>
-#include <dirent.h>
-#include <iostream>
-#include <string>
-#include <algorithm>
-#include <iosfwd>
-#include <vector>
-#include <fstream>
-#include <sstream>
-
-#include "include/api/model.h"
-#include "include/api/context.h"
-#include "include/api/types.h"
-#include "include/api/serialization.h"
-#include "include/dataset/vision_ascend.h"
-#include "include/dataset/execute.h"
-#include "include/dataset/transforms.h"
-#include "include/dataset/vision.h"
-#include "inc/utils.h"
-
-using mindspore::Context;
-using mindspore::Serialization;
-using mindspore::Model;
-using mindspore::Status;
-using mindspore::ModelType;
-using mindspore::GraphCell;
-using mindspore::kSuccess;
-using mindspore::MSTensor;
-using mindspore::dataset::Execute;
-using mindspore::dataset::vision::Decode;
-using mindspore::dataset::vision::Resize;
-using mindspore::dataset::vision::CenterCrop;
-using mindspore::dataset::vision::Normalize;
-using mindspore::dataset::vision::HWC2CHW;
-
-
-DEFINE_string(mindir_path, "", "mindir path");
-DEFINE_string(dataset_name, "imagenet2012", "['cifar10', 'imagenet2012']");
-DEFINE_string(input0_path, ".", "input0 path");
-DEFINE_int32(device_id, 0, "device id");
-
-int load_model(Model *model, std::vector<MSTensor> *model_inputs, std::string mindir_path, int device_id) {
- if (RealPath(mindir_path).empty()) {
- std::cout << "Invalid mindir" << std::endl;
- return 1;
- }
-
- auto context = std::make_shared<Context>();
- auto ascend310 = std::make_shared<mindspore::Ascend310DeviceInfo>();
- ascend310->SetDeviceID(device_id);
- context->MutableDeviceInfo().push_back(ascend310);
- mindspore::Graph graph;
- Serialization::Load(mindir_path, ModelType::kMindIR, &graph);
-
- Status ret = model->Build(GraphCell(graph), context);
- if (ret != kSuccess) {
- std::cout << "ERROR: Build failed." << std::endl;
- return 1;
- }
-
- *model_inputs = model->GetInputs();
- if (model_inputs->empty()) {
- std::cout << "Invalid model, inputs is empty." << std::endl;
- return 1;
- }
- return 0;
-}
-
-int main(int argc, char **argv) {
- gflags::ParseCommandLineFlags(&argc, &argv, true);
-
- Model model;
- std::vector<MSTensor> model_inputs;
- load_model(&model, &model_inputs, FLAGS_mindir_path, FLAGS_device_id);
-
- std::map<double, double> costTime_map;
- struct timeval start = {0};
- struct timeval end = {0};
- double startTimeMs;
- double endTimeMs;
-
- auto input0_files = GetAllInputData(FLAGS_input0_path);
- if (input0_files.empty()) {
- std::cout << "ERROR: no input data." << std::endl;
- return 1;
- }
- size_t size = input0_files.size();
- for (size_t i = 0; i < size; ++i) {
- for (size_t j = 0; j < input0_files[i].size(); ++j) {
- std::vector<MSTensor> inputs;
- std::vector<MSTensor> outputs;
- std::cout << "Start predict input files:" << input0_files[i][j] <<std::endl;
- auto decode = Decode();
- auto resize = Resize({256, 256});
- auto centercrop = CenterCrop({224, 224});
- auto normalize = Normalize({123.675, 116.28, 103.53}, {58.395, 57.12, 57.375});
- auto hwc2chw = HWC2CHW();
-
- Execute SingleOp({decode, resize, centercrop, normalize, hwc2chw});
- auto imgDvpp = std::make_shared<MSTensor>();
- SingleOp(ReadFileToTensor(input0_files[i][j]), imgDvpp.get());
- inputs.emplace_back(model_inputs[0].Name(), model_inputs[0].DataType(), model_inputs[0].Shape(),
- imgDvpp->Data().get(), imgDvpp->DataSize());
- gettimeofday(&start, nullptr);
- Status ret = model.Predict(inputs, &outputs);
- gettimeofday(&end, nullptr);
- if (ret != kSuccess) {
- std::cout << "Predict " << input0_files[i][j] << " failed." << std::endl;
- return 1;
- }
- startTimeMs = (1.0 * start.tv_sec * 1000000 + start.tv_usec) / 1000;
- endTimeMs = (1.0 * end.tv_sec * 1000000 + end.tv_usec) / 1000;
- costTime_map.insert(std::pair<double, double>(startTimeMs, endTimeMs));
- WriteResult(input0_files[i][j], outputs);
- }
- }
- double average = 0.0;
- int inferCount = 0;
-
- for (auto iter = costTime_map.begin(); iter != costTime_map.end(); iter++) {
- double diff = 0.0;
- diff = iter->second - iter->first;
- average += diff;
- inferCount++;
- }
- average = average / inferCount;
- std::stringstream timeCost;
- timeCost << "NN inference cost average time: "<< average << " ms of infer_count " << inferCount << std::endl;
- std::cout << "NN inference cost average time: "<< average << "ms of infer_count " << inferCount << std::endl;
- std::string fileName = "./time_Result" + std::string("/test_perform_static.txt");
- std::ofstream fileStream(fileName.c_str(), std::ios::trunc);
- fileStream << timeCost.str();
- fileStream.close();
- costTime_map.clear();
- return 0;
-}
diff --git a/research/cv/glore_res50/ascend310_infer/src/utils.cc b/research/cv/glore_res50/ascend310_infer/src/utils.cc
deleted file mode 100644
index d71f388b83d23c2813d8bfc883dbcf2e7e0e4ef0..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/ascend310_infer/src/utils.cc
+++ /dev/null
@@ -1,185 +0,0 @@
-/**
- * Copyright 2021 Huawei Technologies Co., Ltd
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-#include <fstream>
-#include <algorithm>
-#include <iostream>
-#include "inc/utils.h"
-
-using mindspore::MSTensor;
-using mindspore::DataType;
-
-
-std::vector<std::vector<std::string>> GetAllInputData(std::string dir_name) {
- std::vector<std::vector<std::string>> ret;
-
- DIR *dir = OpenDir(dir_name);
- if (dir == nullptr) {
- return {};
- }
- struct dirent *filename;
- /* read all the files in the dir ~ */
- std::vector<std::string> sub_dirs;
- while ((filename = readdir(dir)) != nullptr) {
- std::string d_name = std::string(filename->d_name);
- // get rid of "." and ".."
- if (d_name == "." || d_name == ".." || d_name.empty()) {
- continue;
- }
- std::string dir_path = RealPath(std::string(dir_name) + "/" + filename->d_name);
- struct stat s;
- lstat(dir_path.c_str(), &s);
- if (!S_ISDIR(s.st_mode)) {
- continue;
- }
-
- sub_dirs.emplace_back(dir_path);
- }
- std::sort(sub_dirs.begin(), sub_dirs.end());
-
- (void)std::transform(sub_dirs.begin(), sub_dirs.end(), std::back_inserter(ret),
- [](const std::string &d) { return GetAllFiles(d); });
-
- return ret;
-}
-
-
-std::vector<std::string> GetAllFiles(std::string dir_name) {
- struct dirent *filename;
- DIR *dir = OpenDir(dir_name);
- if (dir == nullptr) {
- return {};
- }
-
- std::vector<std::string> res;
- while ((filename = readdir(dir)) != nullptr) {
- std::string d_name = std::string(filename->d_name);
- if (d_name == "." || d_name == ".." || d_name.size() <= 3) {
- continue;
- }
- res.emplace_back(std::string(dir_name) + "/" + filename->d_name);
- }
- std::sort(res.begin(), res.end());
-
- return res;
-}
-
-
-std::vector<std::string> GetAllFiles(std::string_view dirName) {
- struct dirent *filename;
- DIR *dir = OpenDir(dirName);
- if (dir == nullptr) {
- return {};
- }
- std::vector<std::string> res;
- while ((filename = readdir(dir)) != nullptr) {
- std::string dName = std::string(filename->d_name);
- if (dName == "." || dName == ".." || filename->d_type != DT_REG) {
- continue;
- }
- res.emplace_back(std::string(dirName) + "/" + filename->d_name);
- }
- std::sort(res.begin(), res.end());
- for (auto &f : res) {
- std::cout << "image file: " << f << std::endl;
- }
- return res;
-}
-
-
-int WriteResult(const std::string& imageFile, const std::vector<MSTensor> &outputs) {
- std::string homePath = "./result_Files";
- for (size_t i = 0; i < outputs.size(); ++i) {
- size_t outputSize;
- std::shared_ptr<const void> netOutput;
- netOutput = outputs[i].Data();
- outputSize = outputs[i].DataSize();
- int pos = imageFile.rfind('/');
- std::string fileName(imageFile, pos + 1);
- fileName.replace(fileName.find('.'), fileName.size() - fileName.find('.'), '_' + std::to_string(i) + ".bin");
- std::string outFileName = homePath + "/" + fileName;
- FILE *outputFile = fopen(outFileName.c_str(), "wb");
- fwrite(netOutput.get(), outputSize, sizeof(char), outputFile);
- fclose(outputFile);
- outputFile = nullptr;
- }
- return 0;
-}
-
-mindspore::MSTensor ReadFileToTensor(const std::string &file) {
- if (file.empty()) {
- std::cout << "Pointer file is nullptr" << std::endl;
- return mindspore::MSTensor();
- }
-
- std::ifstream ifs(file);
- if (!ifs.good()) {
- std::cout << "File: " << file << " is not exist" << std::endl;
- return mindspore::MSTensor();
- }
-
- if (!ifs.is_open()) {
- std::cout << "File: " << file << "open failed" << std::endl;
- return mindspore::MSTensor();
- }
-
- ifs.seekg(0, std::ios::end);
- size_t size = ifs.tellg();
- mindspore::MSTensor buffer(file, mindspore::DataType::kNumberTypeUInt8, {static_cast<int64_t>(size)}, nullptr, size);
-
- ifs.seekg(0, std::ios::beg);
- ifs.read(reinterpret_cast<char *>(buffer.MutableData()), size);
- ifs.close();
-
- return buffer;
-}
-
-
-DIR *OpenDir(std::string_view dirName) {
- if (dirName.empty()) {
- std::cout << " dirName is null ! " << std::endl;
- return nullptr;
- }
- std::string realPath = RealPath(dirName);
- struct stat s;
- lstat(realPath.c_str(), &s);
- if (!S_ISDIR(s.st_mode)) {
- std::cout << "dirName is not a valid directory !" << std::endl;
- return nullptr;
- }
- DIR *dir;
- dir = opendir(realPath.c_str());
- if (dir == nullptr) {
- std::cout << "Can not open dir " << dirName << std::endl;
- return nullptr;
- }
- std::cout << "Successfully opened the dir " << dirName << std::endl;
- return dir;
-}
-
-std::string RealPath(std::string_view path) {
- char realPathMem[PATH_MAX] = {0};
- char *realPathRet = nullptr;
- realPathRet = realpath(path.data(), realPathMem);
- if (realPathRet == nullptr) {
- std::cout << "File: " << path << " is not exist.";
- return "";
- }
-
- std::string realPath(realPathMem);
- std::cout << path << " realpath is: " << realPath << std::endl;
- return realPath;
-}
diff --git a/research/cv/glore_res50/eval.py b/research/cv/glore_res50/eval.py
deleted file mode 100644
index 8fedc3aa027330425eb2bd378b46e576ab2cc10b..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/eval.py
+++ /dev/null
@@ -1,88 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""
-################################eval glore_resnet50################################
-python eval.py
-"""
-import os
-import ast
-import random
-import argparse
-import numpy as np
-
-from mindspore import context
-from mindspore import dataset as de
-from mindspore.train.model import Model
-from mindspore.train.serialization import load_checkpoint, load_param_into_net
-
-from src.glore_resnet50 import glore_resnet50
-from src.dataset import create_eval_dataset
-from src.loss import CrossEntropySmooth, SoftmaxCrossEntropyExpand
-from src.config import config
-
-parser = argparse.ArgumentParser(description='Image classification with glore_resnet50')
-parser.add_argument('--use_glore', type=ast.literal_eval, default=True, help='Enable GloreUnit')
-parser.add_argument('--data_url', type=str, default=None, help='Dataset path')
-parser.add_argument('--train_url', type=str, help='Train output in modelarts')
-parser.add_argument('--device_target', type=str, default='Ascend', help='Device target')
-parser.add_argument('--device_id', type=int, default=0)
-parser.add_argument('--ckpt_url', type=str, default=None)
-parser.add_argument('--is_modelarts', type=ast.literal_eval, default=True)
-parser.add_argument('--parameter_server', type=ast.literal_eval, default=False, help='Run parameter server train')
-args_opt = parser.parse_args()
-
-if args_opt.is_modelarts:
- import moxing as mox
-
-random.seed(1)
-np.random.seed(1)
-de.config.set_seed(1)
-
-if __name__ == '__main__':
- target = args_opt.device_target
- # init context
- device_id = args_opt.device_id
- context.set_context(mode=context.GRAPH_MODE, device_target=target, save_graphs=False,
- device_id=device_id)
-
- # dataset
- eval_dataset_path = os.path.join(args_opt.data_url, 'val')
- if args_opt.is_modelarts:
- mox.file.copy_parallel(src_url=args_opt.data_url, dst_url='/cache/dataset')
- eval_dataset_path = '/cache/dataset/'
- predict_data = create_eval_dataset(dataset_path=eval_dataset_path, repeat_num=1, batch_size=config.batch_size)
- step_size = predict_data.get_dataset_size()
- if step_size == 0:
- raise ValueError("Please check dataset size > 0 and batch_size <= dataset size")
-
- # define net
- net = glore_resnet50(class_num=config.class_num, use_glore=args_opt.use_glore)
-
- # load checkpoint
- param_dict = load_checkpoint(args_opt.ckpt_url)
- load_param_into_net(net, param_dict)
-
- # define loss, model
- if config.use_label_smooth:
- loss = CrossEntropySmooth(sparse=True, reduction="mean", smooth_factor=config.label_smooth_factor,
- num_classes=config.class_num)
- else:
- loss = SoftmaxCrossEntropyExpand(sparse=True)
- model = Model(net, loss_fn=loss, metrics={'top_1_accuracy', 'top_5_accuracy'})
- print("============== Starting Testing ==============")
- print("ckpt path : {}".format(args_opt.ckpt_url))
- print("data path : {}".format(eval_dataset_path))
- acc = model.eval(predict_data)
- print("==============Acc: {} ==============".format(acc))
diff --git a/research/cv/glore_res50/export.py b/research/cv/glore_res50/export.py
deleted file mode 100644
index 933304d2ec43458446c51cd5b429bccb2850e190..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/export.py
+++ /dev/null
@@ -1,50 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""
-##############export checkpoint file into air, onnx, mindir models#################
-python export.py
-"""
-import argparse
-import numpy as np
-
-import mindspore.common.dtype as mstype
-from mindspore import Tensor, load_checkpoint, load_param_into_net, export, context
-
-from src.config import config
-from src.glore_resnet50 import glore_resnet50
-
-parser = argparse.ArgumentParser(description='Classification')
-parser.add_argument("--device_id", type=int, default=0, help="Device id")
-parser.add_argument("--batch_size", type=int, default=1, help="batch size")
-parser.add_argument("--file_name", type=str, default="googlenet", help="output file name.")
-parser.add_argument('--file_format', type=str, choices=["AIR", "ONNX", "MINDIR"], default='AIR', help='file format')
-parser.add_argument("--device_target", type=str, choices=["Ascend", "GPU", "CPU"], default="Ascend",
- help="device target")
-parser.add_argument("--ckpt_url", type=str, default=None)
-
-args = parser.parse_args()
-
-context.set_context(mode=context.GRAPH_MODE, device_target=args.device_target)
-if args.device_target == "Ascend":
- context.set_context(device_id=args.device_id)
-
-if __name__ == '__main__':
- net = glore_resnet50(class_num=config.class_num)
-
- param_dict = load_checkpoint(args.ckpt_url)
- load_param_into_net(net, param_dict)
-
- input_arr = Tensor(np.ones([args.batch_size, 3, 224, 224]), mstype.float32)
- export(net, input_arr, file_name=args.file_name, file_format=args.file_format)
diff --git a/research/cv/glore_res50/infer/convert/aipp.cfg b/research/cv/glore_res50/infer/convert/aipp.cfg
deleted file mode 100644
index 45ad83ca378eee7b0c076efec5013902c6daab81..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/infer/convert/aipp.cfg
+++ /dev/null
@@ -1,17 +0,0 @@
-aipp_op {
- aipp_mode: static
- input_format : RGB888_U8
-
- rbuv_swap_switch : true
-
- mean_chn_0 : 0
- mean_chn_1 : 0
- mean_chn_2 : 0
- min_chn_0 : 123.675
- min_chn_1 : 116.28
- min_chn_2 : 103.53
- var_reci_chn_0 : 0.0171247538316637
- var_reci_chn_1 : 0.0175070028011204
- var_reci_chn_2 : 0.0174291938997821
-}
-
diff --git a/research/cv/glore_res50/infer/convert/convert_om.sh b/research/cv/glore_res50/infer/convert/convert_om.sh
deleted file mode 100644
index 634608cf26bdf185d6c07bc01635fede3ff57f6b..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/infer/convert/convert_om.sh
+++ /dev/null
@@ -1,31 +0,0 @@
-#!/bin/bash
-
-if [ $# -ne 3 ]
-then
- echo "Wrong parameter format."
- echo "Usage:"
- echo " bash $0 [INPUT_AIR_PATH] [AIPP_PATH] [OUTPUT_OM_PATH_NAME]"
- echo "Example: "
- echo " bash convert_om.sh xxx.air ./aipp.cfg xx"
-
- exit 1
-fi
-
-input_air_path=$1
-aipp_cfg_file=$2
-output_om_path=$3
-
-echo "Input AIR file path: ${input_air_path}"
-echo "Output OM file path: ${output_om_path}"
-
-atc --input_format=NCHW \
- --framework=1 \
- --model="${input_air_path}" \
- --input_shape="actual_input_1:1,3,224,224" \
- --output="${output_om_path}" \
- --insert_op_conf="${aipp_cfg_file}" \
- --enable_small_channel=1 \
- --log=error \
- --soc_version=Ascend310 \
- --op_select_implmode=high_precision
-
diff --git a/research/cv/glore_res50/infer/data/config/glore_res50.cfg b/research/cv/glore_res50/infer/data/config/glore_res50.cfg
deleted file mode 100644
index 581fc76d3d75445323ea9a387f7152a72bedd1d3..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/infer/data/config/glore_res50.cfg
+++ /dev/null
@@ -1,3 +0,0 @@
-CLASS_NUM=1000
-SOFTMAX=false
-TOP_K=5
diff --git a/research/cv/glore_res50/infer/data/config/glore_res50.pipeline b/research/cv/glore_res50/infer/data/config/glore_res50.pipeline
deleted file mode 100644
index 4720187585ac73f940dbca5520e199d550645383..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/infer/data/config/glore_res50.pipeline
+++ /dev/null
@@ -1,73 +0,0 @@
-{
- "glore_res50": {
- "stream_config": {
- "deviceId": "0"
- },
- "appsrc1": {
- "props": {
- "blocksize": "409600"
- },
- "factory": "appsrc",
- "next": "mxpi_imagedecoder0"
- },
- "mxpi_imagedecoder0": {
- "props": {
- "handleMethod": "opencv"
- },
- "factory": "mxpi_imagedecoder",
- "next": "mxpi_imageresize0"
- },
- "mxpi_imageresize0": {
- "props": {
- "handleMethod": "opencv",
- "resizeType": "Resizer_Stretch",
- "resizeHeight": "256",
- "resizeWidth": "256"
- },
- "factory": "mxpi_imageresize",
- "next": "mxpi_opencvcentercrop0"
- },
- "mxpi_opencvcentercrop0": {
- "props": {
- "dataSource": "mxpi_imageresize0",
- "cropHeight": "224",
- "cropWidth": "224"
- },
- "factory": "mxpi_opencvcentercrop",
- "next": "mxpi_tensorinfer0"
- },
- "mxpi_tensorinfer0": {
- "props": {
- "dataSource": "mxpi_opencvcentercrop0",
- "modelPath": "../model/glore_resnet50.om",
- "waitingTime": "2000",
- "outputDeviceId": "-1"
- },
- "factory": "mxpi_tensorinfer",
- "next": "mxpi_classpostprocessor0"
- },
- "mxpi_classpostprocessor0": {
- "props": {
- "dataSource": "mxpi_tensorinfer0",
- "postProcessConfigPath": "./glore_res50.cfg",
- "labelPath": "../data/config/imagenet1000_clsidx_to_labels.names",
- "postProcessLibPath": "../../../lib/modelpostprocessors/libresnet50postprocess.so"
- },
- "factory": "mxpi_classpostprocessor",
- "next": "mxpi_dataserialize0"
- },
- "mxpi_dataserialize0": {
- "props": {
- "outputDataKeys": "mxpi_classpostprocessor0"
- },
- "factory": "mxpi_dataserialize",
- "next": "appsink0"
- },
- "appsink0": {
- "props": {
- "blocksize": "4096000"
- },
- "factory": "appsink"
- }
- }
-}
diff --git a/research/cv/glore_res50/infer/docker_start_infer.sh b/research/cv/glore_res50/infer/docker_start_infer.sh
deleted file mode 100644
index 072b0819ae7edbe63d62c98a039c43469157f792..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/infer/docker_start_infer.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/usr/bin/env bash
-
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-docker_image=$1
-data_dir=$2
-
-function show_help() {
- echo "Usage: docker_start.sh docker_image data_dir"
-}
-
-function param_check() {
- if [ -z "${docker_image}" ]; then
- echo "please input docker_image"
- show_help
- exit 1
- fi
-
- if [ -z "${data_dir}" ]; then
- echo "please input data_dir"
- show_help
- exit 1
- fi
-}
-
-param_check
-
-docker run -it \
- --device=/dev/davinci0 \
- --device=/dev/davinci_manager \
- --device=/dev/devmm_svm \
- --device=/dev/hisi_hdc \
- -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
- -v ${data_dir}:${data_dir} \
- ${docker_image} \
- /bin/bash
diff --git a/research/cv/glore_res50/infer/mxbase/CMakeLists.txt b/research/cv/glore_res50/infer/mxbase/CMakeLists.txt
deleted file mode 100644
index a4e5dcf91d1f3694283abb6e29310c335be11ff6..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/infer/mxbase/CMakeLists.txt
+++ /dev/null
@@ -1,47 +0,0 @@
-cmake_minimum_required(VERSION 3.14.0)
-project(glore_res50)
-set(TARGET glore_res50)
-
-add_definitions(-DENABLE_DVPP_INTERFACE)
-add_definitions(-D_GLIBCXX_USE_CXX11_ABI=0)
-add_definitions(-Dgoogle=mindxsdk_private)
-add_compile_options(-std=c++11 -fPIE -fstack-protector-all -fPIC -Wall)
-add_link_options(-Wl,-z,relro,-z,now,-z,noexecstack -s -pie)
-
-# Check environment variable
-if(NOT DEFINED ENV{ASCEND_HOME})
- message(FATAL_ERROR "please define environment variable:ASCEND_HOME")
-endif()
-if(NOT DEFINED ENV{ASCEND_VERSION})
- message(WARNING "please define environment variable:ASCEND_VERSION")
-endif()
-if(NOT DEFINED ENV{ARCH_PATTERN})
- message(WARNING "please define environment variable:ARCH_PATTERN")
-endif()
-set(ACL_INC_DIR $ENV{ASCEND_HOME}/$ENV{ASCEND_VERSION}/$ENV{ARCH_PATTERN}/acllib/include)
-set(ACL_LIB_DIR $ENV{ASCEND_HOME}/$ENV{ASCEND_VERSION}/$ENV{ARCH_PATTERN}/acllib/lib64)
-
-set(MXBASE_ROOT_DIR $ENV{MX_SDK_HOME})
-set(MXBASE_INC ${MXBASE_ROOT_DIR}/include)
-set(MXBASE_LIB_DIR ${MXBASE_ROOT_DIR}/lib)
-set(MXBASE_POST_LIB_DIR ${MXBASE_ROOT_DIR}/lib/modelpostprocessors)
-set(MXBASE_POST_PROCESS_DIR ${MXBASE_ROOT_DIR}/include/MxBase/postprocess/include)
-set(OPENSOURCE_DIR ${MXBASE_ROOT_DIR}/opensource)
-
-include_directories(${ACL_INC_DIR})
-include_directories(${OPENSOURCE_DIR}/include)
-include_directories(${OPENSOURCE_DIR}/include/opencv4)
-
-include_directories(${MXBASE_INC})
-include_directories(${MXBASE_POST_PROCESS_DIR})
-
-link_directories(${ACL_LIB_DIR})
-link_directories(${OPENSOURCE_DIR}/lib)
-link_directories(${MXBASE_LIB_DIR})
-link_directories(${MXBASE_POST_LIB_DIR})
-
-add_executable(${TARGET} src/main.cpp src/GloreRes50ClassifyOpencv.cpp)
-
-target_link_libraries(${TARGET} glog cpprest mxbase resnet50postprocess opencv_world stdc++fs)
-
-install(TARGETS ${TARGET} RUNTIME DESTINATION ${PROJECT_SOURCE_DIR}/)
diff --git a/research/cv/glore_res50/infer/mxbase/build.sh b/research/cv/glore_res50/infer/mxbase/build.sh
deleted file mode 100644
index 011ff536bc1bd71f82801cc8886240939152589f..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/infer/mxbase/build.sh
+++ /dev/null
@@ -1,55 +0,0 @@
-#!/bin/bash
-
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-path_cur=$(dirname $0)
-
-function check_env()
-{
- # set ASCEND_VERSION to ascend-toolkit/latest when it was not specified by user
- if [ ! "${ASCEND_VERSION}" ]; then
- export ASCEND_VERSION=nnrt/latest
- echo "Set ASCEND_VERSION to the default value: ${ASCEND_VERSION}"
- else
- echo "ASCEND_VERSION is set to ${ASCEND_VERSION} by user"
- fi
-
- if [ ! "${ARCH_PATTERN}" ]; then
- # set ARCH_PATTERN to ./ when it was not specified by user
- export ARCH_PATTERN=.
- echo "ARCH_PATTERN is set to the default value: ${ARCH_PATTERN}"
- else
- echo "ARCH_PATTERN is set to ${ARCH_PATTERN} by user"
- fi
-}
-
-function build_glore_res50()
-{
- cd $path_cur
- rm -rf build
- mkdir -p build
- cd build
- cmake ..
- make
- ret=$?
- if [ ${ret} -ne 0 ]; then
- echo "Failed to build glore_res50."
- exit ${ret}
- fi
- make install
-}
-
-check_env
-build_glore_res50
diff --git a/research/cv/glore_res50/infer/mxbase/src/GloreRes50ClassifyOpencv.cpp b/research/cv/glore_res50/infer/mxbase/src/GloreRes50ClassifyOpencv.cpp
deleted file mode 100644
index 6e43dfc05dcdc0fcb8b914545d846305752948f5..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/infer/mxbase/src/GloreRes50ClassifyOpencv.cpp
+++ /dev/null
@@ -1,245 +0,0 @@
-/*
- * Copyright 2021 Huawei Technologies Co., Ltd
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-#include "GloreRes50ClassifyOpencv.h"
-#include "MxBase/DeviceManager/DeviceManager.h"
-#include "MxBase/Log/Log.h"
-
-using MxBase::DeviceManager;
-using MxBase::TensorBase;
-using MxBase::MemoryData;
-using MxBase::ClassInfo;
-
-namespace {
- const uint32_t YUV_BYTE_NU = 3;
- const uint32_t YUV_BYTE_DE = 2;
- const uint32_t VPC_H_ALIGN = 2;
-}
-
-APP_ERROR GloreRes50ClassifyOpencv::Init(const InitParam &initParam) {
- deviceId_ = initParam.deviceId;
- APP_ERROR ret = DeviceManager::GetInstance()->InitDevices();
- if (ret != APP_ERR_OK) {
- LogError << "Init devices failed, ret=" << ret << ".";
- return ret;
- }
- ret = MxBase::TensorContext::GetInstance()->SetContext(initParam.deviceId);
- if (ret != APP_ERR_OK) {
- LogError << "Set context failed, ret=" << ret << ".";
- return ret;
- }
- dvppWrapper_ = std::make_shared<MxBase::DvppWrapper>();
- ret = dvppWrapper_->Init();
- if (ret != APP_ERR_OK) {
- LogError << "DvppWrapper init failed, ret=" << ret << ".";
- return ret;
- }
- model_ = std::make_shared<MxBase::ModelInferenceProcessor>();
- ret = model_->Init(initParam.modelPath, modelDesc_);
- if (ret != APP_ERR_OK) {
- LogError << "ModelInferenceProcessor init failed, ret=" << ret << ".";
- return ret;
- }
- MxBase::ConfigData configData;
- const std::string softmax = initParam.softmax ? "true" : "false";
- const std::string checkTensor = initParam.checkTensor ? "true" : "false";
-
- configData.SetJsonValue("CLASS_NUM", std::to_string(initParam.classNum));
- configData.SetJsonValue("TOP_K", std::to_string(initParam.topk));
- configData.SetJsonValue("SOFTMAX", softmax);
- configData.SetJsonValue("CHECK_MODEL", checkTensor);
-
- auto jsonStr = configData.GetCfgJson().serialize();
- std::map<std::string, std::shared_ptr<void>> config;
- config["postProcessConfigContent"] = std::make_shared<std::string>(jsonStr);
- config["labelPath"] = std::make_shared<std::string>(initParam.labelPath);
-
- post_ = std::make_shared<MxBase::Resnet50PostProcess>();
- ret = post_->Init(config);
- if (ret != APP_ERR_OK) {
- LogError << "Resnet50PostProcess init failed, ret=" << ret << ".";
- return ret;
- }
- return APP_ERR_OK;
-}
-
-APP_ERROR GloreRes50ClassifyOpencv::DeInit() {
- dvppWrapper_->DeInit();
- model_->DeInit();
- post_->DeInit();
- DeviceManager::GetInstance()->DestroyDevices();
- return APP_ERR_OK;
-}
-
-APP_ERROR GloreRes50ClassifyOpencv::ReadImage(const std::string &imgPath, cv::Mat *imageMat) {
- *imageMat = cv::imread(imgPath, cv::IMREAD_COLOR);
- return APP_ERR_OK;
-}
-
-APP_ERROR GloreRes50ClassifyOpencv::ResizeImage(const cv::Mat &srcImageMat, cv::Mat *dstImageMat) {
- static constexpr uint32_t resizeHeight = 256;
- static constexpr uint32_t resizeWidth = 256;
-
- cv::resize(srcImageMat, *dstImageMat, cv::Size(resizeWidth, resizeHeight));
- return APP_ERR_OK;
-}
-
-APP_ERROR GloreRes50ClassifyOpencv::CenterCropImage(const cv::Mat &img, cv::Mat *crop_im) {
- static cv::Rect rectOfImg(16, 16, 224, 224);
- *crop_im = img(rectOfImg).clone();
- return APP_ERR_OK;
-}
-
-APP_ERROR GloreRes50ClassifyOpencv::CvmatToTensorBase(const cv::Mat &imageMat, MxBase::TensorBase *tensorBase) {
- const uint32_t dataSize = imageMat.cols * imageMat.rows * MxBase::YUV444_RGB_WIDTH_NU;
- LogInfo << "image size after crop" << imageMat.cols << " " << imageMat.rows;
- MxBase::MemoryData memoryDataDst(dataSize, MemoryData::MEMORY_DEVICE, deviceId_);
- MxBase::MemoryData memoryDataSrc(imageMat.data, dataSize, MemoryData::MEMORY_HOST_MALLOC);
-
- APP_ERROR ret = MxBase::MemoryHelper::MxbsMallocAndCopy(memoryDataDst, memoryDataSrc);
- if (ret != APP_ERR_OK) {
- LogError << GetError(ret) << "Memory malloc failed.";
- return ret;
- }
-
- std::vector<uint32_t> shape = {imageMat.rows * MxBase::YUV444_RGB_WIDTH_NU, static_cast<uint32_t>(imageMat.cols)};
- *tensorBase = TensorBase(memoryDataDst, false, shape, MxBase::TENSOR_DTYPE_UINT8);
- return APP_ERR_OK;
-}
-
-APP_ERROR GloreRes50ClassifyOpencv::Inference(const std::vector<TensorBase> &inputs,
- std::vector<TensorBase> *outputs) {
- auto dtypes = model_->GetOutputDataType();
- for (size_t i = 0; i < modelDesc_.outputTensors.size(); ++i) {
- std::vector<uint32_t> shape = {};
- for (size_t j = 0; j < modelDesc_.outputTensors[i].tensorDims.size(); ++j) {
- shape.push_back((uint32_t)modelDesc_.outputTensors[i].tensorDims[j]);
- }
- TensorBase tensor(shape, dtypes[i], MemoryData::MemoryType::MEMORY_DEVICE, deviceId_);
- APP_ERROR ret = TensorBase::TensorBaseMalloc(tensor);
- if (ret != APP_ERR_OK) {
- LogError << "TensorBaseMalloc failed, ret=" << ret << ".";
- return ret;
- }
- outputs->push_back(tensor);
- }
- MxBase::DynamicInfo dynamicInfo = {};
- dynamicInfo.dynamicType = MxBase::DynamicType::STATIC_BATCH;
- auto startTime = std::chrono::high_resolution_clock::now();
- APP_ERROR ret = model_->ModelInference(inputs, *outputs, dynamicInfo);
- auto endTime = std::chrono::high_resolution_clock::now();
- double costMs = std::chrono::duration<double, std::milli>(endTime - startTime).count(); // save time
- inferCostTimeMilliSec += costMs;
- if (ret != APP_ERR_OK) {
- LogError << "ModelInference failed, ret=" << ret << ".";
- return ret;
- }
- return APP_ERR_OK;
-}
-
-APP_ERROR GloreRes50ClassifyOpencv::PostProcess(const std::vector<TensorBase> &inputs,
- std::vector<std::vector<ClassInfo>> *clsInfos) {
- APP_ERROR ret = post_->Process(inputs, *clsInfos);
- if (ret != APP_ERR_OK) {
- LogError << "Process failed, ret=" << ret << ".";
- return ret;
- }
- return APP_ERR_OK;
-}
-
-APP_ERROR GloreRes50ClassifyOpencv::SaveResult(const std::string &imgPath,
- const std::vector<std::vector<ClassInfo>> &batchClsInfos) {
- LogInfo << "image path" << imgPath;
- std::string fileName = imgPath.substr(imgPath.find_last_of("/") + 1);
- size_t dot = fileName.find_last_of(".");
- std::string resFileName = "result/" + fileName.substr(0, dot) + "_1.txt";
- LogInfo << "file path for saving result" << resFileName;
-
- std::ofstream outfile(resFileName);
- if (outfile.fail()) {
- LogError << "Failed to open result file: ";
- return APP_ERR_COMM_FAILURE;
- }
-
- uint32_t batchIndex = 0;
- for (auto clsInfos : batchClsInfos) {
- std::string resultStr;
- for (auto clsInfo : clsInfos) {
- LogDebug << " className:" << clsInfo.className << " confidence:" << clsInfo.confidence <<
- " classIndex:" << clsInfo.classId;
- resultStr += std::to_string(clsInfo.classId) + " ";
- }
-
- outfile << resultStr << std::endl;
- batchIndex++;
- }
- outfile.close();
- return APP_ERR_OK;
-}
-
-APP_ERROR GloreRes50ClassifyOpencv::Process(const std::string &imgPath) {
- cv::Mat imageMat;
- APP_ERROR ret = ReadImage(imgPath, &imageMat);
- if (ret != APP_ERR_OK) {
- LogError << "ReadImage failed, ret=" << ret << ".";
- return ret;
- }
-
- cv::Mat resizeImg;
- ret = ResizeImage(imageMat, &resizeImg);
- if (ret != APP_ERR_OK) {
- LogError << "ResizeImage failed, ret=" << ret << ".";
- return ret;
- }
-
- cv::Mat cropImg;
- ret = CenterCropImage(resizeImg, &cropImg);
- if (ret != APP_ERR_OK) {
- LogError << "CropImage failed, ret=" << ret << ".";
- return ret;
- }
-
- std::vector<MxBase::TensorBase> inputs = {};
- std::vector<MxBase::TensorBase> outputs = {};
-
- MxBase::TensorBase tensorBase;
- ret = CvmatToTensorBase(cropImg, &tensorBase);
- if (ret != APP_ERR_OK) {
- LogError << "CVMatToTensorBase failed, ret=" << ret << ".";
- return ret;
- }
- inputs.push_back(tensorBase);
-
- ret = Inference(inputs, &outputs);
- if (ret != APP_ERR_OK) {
- LogError << "Inference failed, ret=" << ret << ".";
- return ret;
- }
- std::vector<std::vector<MxBase::ClassInfo>> batchClsInfos = {};
- ret = PostProcess(outputs, &batchClsInfos);
- if (ret != APP_ERR_OK) {
- LogError << "PostProcess failed, ret=" << ret << ".";
- return ret;
- }
-
- ret = SaveResult(imgPath, batchClsInfos);
- if (ret != APP_ERR_OK) {
- LogError << "Save infer results into file failed. ret = " << ret << ".";
- return ret;
- }
-
- return APP_ERR_OK;
-}
diff --git a/research/cv/glore_res50/infer/mxbase/src/GloreRes50ClassifyOpencv.h b/research/cv/glore_res50/infer/mxbase/src/GloreRes50ClassifyOpencv.h
deleted file mode 100644
index 99c23a124b5136ed0afbf6416202d8f8ab394630..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/infer/mxbase/src/GloreRes50ClassifyOpencv.h
+++ /dev/null
@@ -1,70 +0,0 @@
-/*
- * Copyright 2021 Huawei Technologies Co., Ltd
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-#ifndef MXBASE_GLORERES50CLASSIFYOPENCV_H
-#define MXBASE_GLORERES50CLASSIFYOPENCV_H
-
-#include <map>
-#include <memory>
-#include <string>
-#include <vector>
-#include <opencv2/opencv.hpp>
-#include "MxBase/DvppWrapper/DvppWrapper.h"
-#include "MxBase/ModelInfer/ModelInferenceProcessor.h"
-#include "ClassPostProcessors/Resnet50PostProcess.h"
-#include "MxBase/Tensor/TensorContext/TensorContext.h"
-
-struct InitParam {
- uint32_t deviceId;
- std::string labelPath;
- uint32_t classNum;
- uint32_t topk;
- bool softmax;
- bool checkTensor;
- std::string modelPath;
-};
-
-class GloreRes50ClassifyOpencv {
- public:
- APP_ERROR Init(const InitParam &initParam);
- APP_ERROR DeInit();
- APP_ERROR ReadImage(const std::string &imgPath, cv::Mat *imageMat);
- APP_ERROR ResizeImage(const cv::Mat &srcImageMat, cv::Mat *dstImageMat);
- APP_ERROR CenterCropImage(const cv::Mat &img, cv::Mat *crop_im);
- APP_ERROR CvmatToTensorBase(const cv::Mat &imageMat, MxBase::TensorBase *tensorBase);
- APP_ERROR Inference(const std::vector<MxBase::TensorBase> &inputs, std::vector<MxBase::TensorBase> *outputs);
- APP_ERROR PostProcess(const std::vector<MxBase::TensorBase> &inputs,
- std::vector<std::vector<MxBase::ClassInfo>> *clsInfos);
- APP_ERROR Process(const std::string &imgPath);
- // get infer time
- double GetInferCostMillisec() const {return inferCostTimeMilliSec;}
-
- private:
- APP_ERROR SaveResult(const std::string &imgPath,
- const std::vector<std::vector<MxBase::ClassInfo>> &batchClsInfos);
-
- private:
- std::shared_ptr<MxBase::DvppWrapper> dvppWrapper_;
- std::shared_ptr<MxBase::ModelInferenceProcessor> model_;
- std::shared_ptr<MxBase::Resnet50PostProcess> post_;
- MxBase::ModelDesc modelDesc_;
- uint32_t deviceId_ = 0;
- // infer time
- double inferCostTimeMilliSec = 0.0;
-};
-
-
-#endif
diff --git a/research/cv/glore_res50/infer/mxbase/src/main.cpp b/research/cv/glore_res50/infer/mxbase/src/main.cpp
deleted file mode 100644
index cfa6cb7c862407ec87b8b38ae943a6dfdae00bcf..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/infer/mxbase/src/main.cpp
+++ /dev/null
@@ -1,88 +0,0 @@
-/*
- * Copyright 2021 Huawei Technologies Co., Ltd
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-#include <dirent.h>
-#include "GloreRes50ClassifyOpencv.h"
-#include "MxBase/Log/Log.h"
-
-namespace {
-const uint32_t CLASS_NUM = 1000;
-} // namespace
-
-APP_ERROR scan_images(const std::string &path, std::vector<std::string> *imgFiles) {
- DIR *dirPtr = opendir(path.c_str());
- if (dirPtr == nullptr) {
- LogError << "opendir failed. dir:" << path;
- return APP_ERR_INTERNAL_ERROR;
- }
- dirent *direntPtr = nullptr;
- while ((direntPtr = readdir(dirPtr)) != nullptr) {
- std::string fileName = direntPtr->d_name;
- if (fileName == "." || fileName == "..") {
- continue;
- }
-
- imgFiles->emplace_back(path + "/" + fileName);
- }
- closedir(dirPtr);
- return APP_ERR_OK;
-}
-
-
-int main(int argc, char* argv[]) {
- if (argc <= 1) {
- LogWarn << "Please input image path, such as './gloreres50 image_dir'.";
- return APP_ERR_OK;
- }
-
- InitParam initParam = {};
- initParam.deviceId = 0;
- initParam.classNum = CLASS_NUM;
- initParam.labelPath = "../../labelset/imagenet1000_clsidx_to_labels.names";
- initParam.topk = 5;
- initParam.softmax = false;
- initParam.checkTensor = true;
- initParam.modelPath = "../model/glore_resnet50.om";
- auto glore_res50 = std::make_shared<GloreRes50ClassifyOpencv>();
- APP_ERROR ret = glore_res50->Init(initParam);
- if (ret != APP_ERR_OK) {
- LogError << "GloreRes50Classify init failed, ret=" << ret << ".";
- return ret;
- }
-
- std::string imgPath = argv[1];
- std::vector<std::string> imgFilePaths;
- ret = scan_images(imgPath, &imgFilePaths);
- if (ret != APP_ERR_OK) {
- return ret;
- }
- auto startTime = std::chrono::high_resolution_clock::now();
- for (auto &imgFile : imgFilePaths) {
- ret = glore_res50->Process(imgFile);
- if (ret != APP_ERR_OK) {
- LogError << "GloreRes50Classify process failed, ret=" << ret << ".";
- glore_res50->DeInit();
- return ret;
- }
- }
- auto endTime = std::chrono::high_resolution_clock::now();
- glore_res50->DeInit();
- double costMilliSecs = std::chrono::duration<double, std::milli>(endTime - startTime).count();
- double unit_convert = 1000.0; // 1s = 1000 ms
- double fps = unit_convert * imgFilePaths.size() / glore_res50->GetInferCostMillisec();
- LogInfo << "[Process Delay] cost: " << costMilliSecs << " ms\tfps: " << fps << " imgs/sec";
- return APP_ERR_OK;
-}
diff --git a/research/cv/glore_res50/infer/sdk/main.py b/research/cv/glore_res50/infer/sdk/main.py
deleted file mode 100644
index 596e6bce492a3f55f143d3d66b717ce984269e0d..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/infer/sdk/main.py
+++ /dev/null
@@ -1,93 +0,0 @@
-# coding=utf-8
-
-"""
-Copyright 2021 Huawei Technologies Co., Ltd
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-"""
-
-import datetime
-import json
-import os
-import sys
-
-from StreamManagerApi import StreamManagerApi
-from StreamManagerApi import MxDataInput
-
-
-if __name__ == '__main__':
- # init stream manager
- stream_manager_api = StreamManagerApi()
- ret = stream_manager_api.InitManager()
- if ret != 0:
- print("Failed to init Stream manager, ret=%s" % str(ret))
- exit()
-
- # create streams by pipeline config file
- with open("./glore_res50.pipeline", 'rb') as f:
- pipelineStr = f.read()
- ret = stream_manager_api.CreateMultipleStreams(pipelineStr)
-
- if ret != 0:
- print("Failed to create Stream, ret=%s" % str(ret))
- exit()
-
- # Construct the input of the stream
- data_input = MxDataInput()
-
- dir_name = sys.argv[1]
- res_dir_name = sys.argv[2]
- file_list = os.listdir(dir_name)
- if not os.path.exists(res_dir_name):
- os.makedirs(res_dir_name)
-
- for file_name in file_list:
- file_path = os.path.join(dir_name, file_name)
- if not (file_name.lower().endswith(".jpg") or file_name.lower().endswith(".jpeg")):
- continue
-
- with open(file_path, 'rb') as f:
- data_input.data = f.read()
- empty_data = []
- stream_name = b'glore_res50'
- in_plugin_id = 0
- unique_id = stream_manager_api.SendData(stream_name, in_plugin_id, data_input)
- if unique_id < 0:
- print("Failed to send data to stream.")
- exit()
- # Obtain the inference result by specifying streamName and uniqueId.
- start_time = datetime.datetime.now()
- infer_result = stream_manager_api.GetResult(stream_name, unique_id)
- end_time = datetime.datetime.now()
- print('sdk run time: {}'.format((end_time - start_time).microseconds))
- if infer_result.errorCode != 0:
- print("GetResultWithUniqueId error. errorCode=%d, errorMsg=%s" % (
- infer_result.errorCode, infer_result.data.decode()))
- exit()
- # print the infer result
- infer_res = infer_result.data.decode()
- print("process img: {}, infer result: {}".format(file_name, infer_res))
- load_dict = json.loads(infer_result.data.decode())
- if load_dict.get('MxpiClass') is None:
- with open(res_dir_name + "/" + file_name[:-5] + '.txt', 'w') as f_write:
- f_write.write("")
- continue
- res_vec = load_dict.get('MxpiClass')
-
- with open(res_dir_name + "/" + file_name[:-5] + '_1.txt', 'w') as f_write:
- res_list = [str(item.get("classId")) + " " for item in res_vec]
- f_write.writelines(res_list)
- f_write.write('\n')
-
- # destroy streams
- stream_manager_api.DestroyAllStreams()
diff --git a/research/cv/glore_res50/infer/sdk/run.sh b/research/cv/glore_res50/infer/sdk/run.sh
deleted file mode 100644
index 157dad8c91a37c4817b2be98e874549b24e2a68a..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/infer/sdk/run.sh
+++ /dev/null
@@ -1,35 +0,0 @@
-#!/bin/bash
-
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-image_path=$1
-result_dir=$2
-
-set -e
-
-#CUR_PATH=$(cd "$(dirname "$0")" || { warn "Failed to check path/to/run.sh" ; exit ; } ; pwd)
-
-# Simple log helper functions
-info() { echo -e "\033[1;34m[INFO ][MxStream] $1\033[1;37m" ; }
-warn() { echo >&2 -e "\033[1;31m[WARN ][MxStream] $1\033[1;37m" ; }
-
-export LD_LIBRARY_PATH=${MX_SDK_HOME}/lib:${MX_SDK_HOME}/opensource/lib:${MX_SDK_HOME}/opensource/lib64:/usr/local/Ascend/ascend-toolkit/latest/acllib/lib64:${LD_LIBRARY_PATH}
-export GST_PLUGIN_SCANNER=${MX_SDK_HOME}/opensource/libexec/gstreamer-1.0/gst-plugin-scanner
-export GST_PLUGIN_PATH=${MX_SDK_HOME}/opensource/lib/gstreamer-1.0:${MX_SDK_HOME}/lib/plugins
-
-#to set PYTHONPATH, import the StreamManagerApi.py
-export PYTHONPATH=$PYTHONPATH:${MX_SDK_HOME}/python
-
-python3.7 main.py $image_path $result_dir
-exit 0
diff --git a/research/cv/glore_res50/infer/util/eval_by_sdk.py b/research/cv/glore_res50/infer/util/eval_by_sdk.py
deleted file mode 100644
index 83fa25359dcc10cec1a4405a4dbd9c2bff1b9b4f..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/infer/util/eval_by_sdk.py
+++ /dev/null
@@ -1,179 +0,0 @@
-#coding = utf-8
-
-"""
-Copyright 2021 Huawei Technologies Co., Ltd
-
-Licensed under the BSD 3-Clause License (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-https://opensource.org/licenses/BSD-3-Clause
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-"""
-
-import os
-import sys
-import json
-import numpy as np
-
-
-np.set_printoptions(threshold=sys.maxsize)
-
-def gen_file_name(img_name):
- """
- :param filename: image file name
- :return: a list contains image name and ext
- """
- full_name = img_name.split('/')[-1]
- return os.path.splitext(full_name)
-
-
-def cre_groundtruth_dict_fromjson(gtfile_path):
- """
- :param filename: json file contains the imagename and label number
- :return: dictionary key imagename, value is label number
- """
- img_gt_dict = {}
- with open(gtfile_path, 'r') as f:
- gt = json.load(f)
- for key, value in gt.items():
- img_gt_dict[gen_file_name(key)[0]] = value
- return img_gt_dict
-
-
-def cre_groundtruth_dict_fromtxt(gtfile_path):
- """
- :param filename: file contains the imagename and label number
- :return: dictionary key imagename, value is label number
- """
- img_gt_dict = {}
- with open(gtfile_path, 'r')as f:
- for line in f.readlines():
- temp = line.strip().split(" ")
- img_name = temp[0].split(".")[0]
- img_lab = temp[1]
- img_gt_dict[img_name] = img_lab
- return img_gt_dict
-
-
-def load_statistical_predict_result(filepath):
- """
- function:
- the prediction esult file data extraction
- input:
- result file:filepath
- output:
- n_label:numble of label
- data_vec: the probabilitie of prediction in the 1000
- :return: probabilities, numble of label, in_type, color
- """
- with open(filepath, 'r')as f:
- data = f.readline()
- temp = data.strip().split(" ")
- n_label = len(temp)
- data_vec = np.zeros((n_label), dtype=np.float32)
- in_type = ''
- color = ''
- if n_label == 0:
- in_type = f.readline()
- color = f.readline()
- else:
- for ind, cls_ind in enumerate(temp):
- data_vec[ind] = np.int64(cls_ind)
- return data_vec, n_label, in_type, color
-
-
-def create_visualization_statistical_result(prediction_file_path,
- result_store_path, json_file_path,
- img_gt_dict, topn=5):
- """
- :param prediction_file_path:
- :param result_store_path:
- :param json_file_name:
- :param img_gt_dict:
- :param topn:
- :return:
- """
- writer = open(os.path.join(result_store_path, json_file_path), 'w')
- table_dict = {}
- table_dict["title"] = "Overall statistical evaluation"
- table_dict["value"] = []
-
- count = 0
- res_cnt = 0
- n_labels = ""
- count_hit = np.zeros(topn)
- for tfile_name in os.listdir(prediction_file_path):
- count += 1
- img_name = tfile_name.split('.')[0][:-2]
- filepath = os.path.join(prediction_file_path, tfile_name)
-
- ret = load_statistical_predict_result(filepath)
- prediction = ret[0]
- n_labels = ret[1]
-
- gt = img_gt_dict[img_name]
- if n_labels == 1000:
- real_label = int(gt)
- elif n_labels == 1001:
- real_label = int(gt) + 1
- else:
- real_label = int(gt)
-
- res_cnt = min(len(prediction), topn)
- for i in range(res_cnt):
- if str(real_label) == str(int(prediction[i])):
- count_hit[i] += 1
- break
- if 'value' not in table_dict.keys():
- print("the item value does not exist!")
- else:
- table_dict["value"].extend(
- [{"key": "Number of images", "value": str(count)},
- {"key": "Number of classes", "value": str(n_labels)}])
- if count == 0:
- accuracy = 0
- else:
- accuracy = np.cumsum(count_hit) / count
- for i in range(res_cnt):
- table_dict["value"].append({"key": "Top" + str(i + 1) + " accuracy",
- "value": str(
- round(accuracy[i] * 100, 2)) + '%'})
- json.dump(table_dict, writer)
- writer.close()
-
-
-if __name__ == '__main__':
- try:
- # prediction txt files path
- folder_davinci_target = sys.argv[1]
- # annotation files path, "imagenet2012_label.json"
- annotation_file_path = sys.argv[2]
- # the path to store the results json path
- result_json_path = sys.argv[3]
- # result json file name
- json_file_name = sys.argv[4]
- except IndexError:
- print("Please enter prediction files folder, ground truth label json file, result json file folder, "
- "result json file name\n")
- print("\teg: python3.7 eval_by_sdk.py ./preds ../data/config/imagenet_label.json ./result result.json")
- exit(1)
-
- if not os.path.exists(folder_davinci_target):
- print("Target file folder does not exist.")
-
- if not os.path.exists(annotation_file_path):
- print("Ground truth file does not exist.")
-
- if not os.path.exists(result_json_path):
- print("Result folder doesn't exist.")
-
- img_label_dict = cre_groundtruth_dict_fromjson(annotation_file_path)
- create_visualization_statistical_result(folder_davinci_target, result_json_path,
- json_file_name, img_label_dict, topn=5)
-
\ No newline at end of file
diff --git a/research/cv/glore_res50/modelarts/start.py b/research/cv/glore_res50/modelarts/start.py
deleted file mode 100644
index 4810d9d871c771c1986142491ad089e02edfe777..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/modelarts/start.py
+++ /dev/null
@@ -1,249 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""
-################################train glore_resnet50################################
-python train.py
-"""
-import os
-import glob
-import random
-import argparse
-import ast
-import numpy as np
-
-import mindspore.nn as nn
-from mindspore import context
-from mindspore import Tensor, export
-from mindspore import dataset as de
-from mindspore.nn.metrics import Accuracy
-import mindspore.common.dtype as mstype
-from mindspore.communication.management import init
-import mindspore.common.initializer as weight_init
-from mindspore.nn.optim.momentum import Momentum
-from mindspore.context import ParallelMode
-from mindspore.train.model import Model
-from mindspore.train.loss_scale_manager import FixedLossScaleManager
-from mindspore.train.serialization import load_checkpoint, load_param_into_net
-from mindspore.train.callback import ModelCheckpoint, CheckpointConfig, LossMonitor, TimeMonitor
-
-from glore_res50.src.config import config
-from glore_res50.src.lr_generator import get_lr
-from glore_res50.src.dataset import create_train_dataset, create_eval_dataset, _get_rank_info
-from glore_res50.src.save_callback import SaveCallback
-from glore_res50.src.glore_resnet50 import glore_resnet50
-from glore_res50.src.loss import SoftmaxCrossEntropyExpand, CrossEntropySmooth
-from glore_res50.src.autoaugment import autoaugment
-
-parser = argparse.ArgumentParser(
- description='Image classification with glore_resnet50')
-parser.add_argument('--use_glore', type=ast.literal_eval,
- default=True, help='Enable GloreUnit')
-parser.add_argument('--run_distribute', type=ast.literal_eval,
- default=True, help='Run distribute')
-parser.add_argument('--data_url', type=str, default=None,
- help='Dataset path')
-parser.add_argument('--train_url', type=str)
-parser.add_argument('--device_target', type=str,
- default='Ascend', help='Device target')
-parser.add_argument('--device_id', type=int, default=0)
-parser.add_argument('--is_modelarts', type=ast.literal_eval, default=True)
-parser.add_argument('--pretrained_ckpt', type=str,
- default=None, help='Pretrained ckpt path \
- Ckpt file name(when modelarts on), full path for ckpt file(whem not modelarts)')
-parser.add_argument('--parameter_server', type=ast.literal_eval,
- default=False, help='Run parameter server train')
-parser.add_argument('--export', type=ast.literal_eval, default=False,
- help="Export air | mindir model.")
-parser.add_argument("--export_batch_size", type=int,
- default=1, help="batch size")
-parser.add_argument("--export_file_name", type=str,
- default="glore_res50", help="output file name.")
-parser.add_argument('--export_file_format', type=str,
- choices=["AIR", "ONNX", "MINDIR"], default='AIR', help='file format')
-parser.add_argument('--export_ckpt_url', type=str, default=None,
- help='Ckpt file name(when modelarts on), full path for ckpt file(whem not modelarts) \
- REQUIRED when export is enabled.')
-args_opt = parser.parse_args()
-
-if args_opt.is_modelarts:
- import moxing as mox
-
-random.seed(1)
-np.random.seed(1)
-de.config.set_seed(1)
-
-if __name__ == '__main__':
- target = args_opt.device_target
-
- # init context
- context.set_context(mode=context.GRAPH_MODE,
- device_target=target, save_graphs=False)
- if args_opt.run_distribute:
- if target == "Ascend":
- device_id = int(os.getenv('DEVICE_ID'))
- context.set_context(device_id=device_id)
- context.set_auto_parallel_context(
- parallel_mode=ParallelMode.DATA_PARALLEL, gradients_mean=True)
- init()
- else:
- if target == "Ascend":
- device_id = args_opt.device_id
- context.set_context(
- mode=context.GRAPH_MODE, device_target=target, save_graphs=False, device_id=device_id)
- # create dataset
- train_dataset_path = os.path.join(args_opt.data_url, 'train')
- eval_dataset_path = os.path.join(args_opt.data_url, 'val')
-
- # download dataset from obs to cache if train on ModelArts
- root = '/cache/dataset/device_' + os.getenv('DEVICE_ID')
- if args_opt.is_modelarts:
- mox.file.copy_parallel(
- src_url=args_opt.data_url, dst_url=root)
- train_dataset_path = root + '/train'
- eval_dataset_path = root + '/val'
- if config.pretrained:
- args_opt.pretrained_ckpt = root + '/' + args_opt.pretrained_ckpt
- if args_opt.export:
- args_opt.export_ckpt_url = root + '/' + args_opt.export_ckpt_url
- args_opt.export_file_name = '/cache/train_output/' + args_opt.export_file_name
-
- # define net
-
- net = glore_resnet50(class_num=config.class_num,
- use_glore=args_opt.use_glore)
-
- if args_opt.export:
- param_dict = load_checkpoint(args_opt.export_ckpt_url)
- load_param_into_net(net, param_dict)
-
- input_arr = Tensor(
- np.ones([args_opt.export_batch_size, 3, 224, 224]), mstype.float32)
- export(net, input_arr, file_name=args_opt.export_file_name,
- file_format=args_opt.export_file_format)
- if args_opt.is_modelarts:
- mox.file.copy_parallel(
- src_url='/cache/train_output', dst_url=args_opt.train_url)
- else:
- if config.use_autoaugment:
- print("===========Use autoaugment==========")
- train_dataset = autoaugment(dataset_path=train_dataset_path, repeat_num=1,
- batch_size=config.batch_size, target=target)
- else:
- train_dataset = create_train_dataset(dataset_path=train_dataset_path, repeat_num=1,
- batch_size=config.batch_size, target=target)
-
- eval_dataset = create_eval_dataset(
- dataset_path=eval_dataset_path, repeat_num=1, batch_size=config.batch_size)
-
- step_size = train_dataset.get_dataset_size()
- # init weight
- if config.pretrained:
- param_dict = load_checkpoint(args_opt.pretrained_ckpt)
- load_param_into_net(net, param_dict)
- else:
- for _, cell in net.cells_and_names():
- if isinstance(cell, (nn.Conv2d, nn.Conv1d)):
- if config.weight_init == 'xavier_uniform':
- cell.weight.default_input = weight_init.initializer(weight_init.XavierUniform(),
- cell.weight.shape,
- cell.weight.dtype)
- elif config.weight_init == 'he_uniform':
- cell.weight.default_input = weight_init.initializer(weight_init.HeUniform(),
- cell.weight.shape,
- cell.weight.dtype)
- else: # config.weight_init == 'he_normal' or the others
- cell.weight.default_input = weight_init.initializer(weight_init.HeNormal(),
- cell.weight.shape,
- cell.weight.dtype)
-
- if isinstance(cell, nn.Dense):
- cell.weight.default_input = weight_init.initializer(weight_init.TruncatedNormal(),
- cell.weight.shape,
- cell.weight.dtype)
-
- # init lr
- lr = get_lr(lr_init=config.lr_init, lr_end=config.lr_end, lr_max=config.lr_max,
- warmup_epochs=config.warmup_epochs, total_epochs=config.epoch_size,
- steps_per_epoch=step_size, lr_decay_mode=config.lr_decay_mode)
- lr = Tensor(lr)
-
- #
- # define opt
- decayed_params = []
- no_decayed_params = []
- for param in net.trainable_params():
- if 'beta' not in param.name and 'gamma' not in param.name and 'bias' not in param.name:
- decayed_params.append(param)
- else:
- no_decayed_params.append(param)
-
- group_params = [{'params': decayed_params, 'weight_decay': config.weight_decay},
- {'params': no_decayed_params},
- {'order_params': net.trainable_params()}]
- net_opt = Momentum(group_params, lr, config.momentum,
- loss_scale=config.loss_scale)
- # define loss, model
- if config.use_label_smooth:
- loss = CrossEntropySmooth(sparse=True, reduction="mean", smooth_factor=config.label_smooth_factor,
- num_classes=config.class_num)
- else:
- loss = SoftmaxCrossEntropyExpand(sparse=True)
- loss_scale = FixedLossScaleManager(
- config.loss_scale, drop_overflow_update=False)
- model = Model(net, loss_fn=loss, optimizer=net_opt,
- loss_scale_manager=loss_scale, metrics={"Accuracy": Accuracy()})
-
- # define callbacks
- time_cb = TimeMonitor(data_size=step_size)
- loss_cb = LossMonitor()
- cb = [time_cb, loss_cb]
- device_num, device_id = _get_rank_info()
- if config.save_checkpoint:
- if args_opt.is_modelarts:
- save_checkpoint_path = '/cache/train_output/device_' + \
- os.getenv('DEVICE_ID') + '/'
- else:
- save_checkpoint_path = config.save_checkpoint_path
- config_ck = CheckpointConfig(save_checkpoint_steps=config.save_checkpoint_epochs * step_size,
- keep_checkpoint_max=config.keep_checkpoint_max)
- ckpt_cb = ModelCheckpoint(
- prefix="glore_resnet50", directory=save_checkpoint_path, config=config_ck)
- save_cb = SaveCallback(model, eval_dataset, save_checkpoint_path)
- cb += [ckpt_cb]
-
- # train model
- print("=======Training Begin========")
- model.train(config.epoch_size - config.pretrain_epoch_size,
- train_dataset, callbacks=cb, dataset_sink_mode=True)
- ckpt_list = glob.glob('/cache/train_output/device_' + \
- os.getenv('DEVICE_ID') + '/*.ckpt')
- if not ckpt_list:
- print("ckpt file not generated.")
-
- ckpt_list.sort(key=os.path.getmtime)
- ckpt_model = ckpt_list[-1]
- print("checkpoint path", ckpt_model)
- ckpt_param_dict = load_checkpoint(ckpt_model)
- input_arr = Tensor(np.zeros([1, 3, 224, 224], np.float32))
- # frozen to ait file
- export_net = glore_resnet50(class_num=config.class_num,
- use_glore=args_opt.use_glore)
- load_param_into_net(export_net, ckpt_param_dict)
- export(export_net, input_arr, file_name='/cache/train_output/device_' + \
- os.getenv('DEVICE_ID') + '/glore_res50', file_format="AIR")
- # copy train result from cache to obs
- if args_opt.is_modelarts:
- mox.file.copy_parallel(
- src_url='/cache/train_output', dst_url=args_opt.train_url)
diff --git a/research/cv/glore_res50/postprocess.py b/research/cv/glore_res50/postprocess.py
deleted file mode 100644
index 4144a3ee71e41825744a5651d01e04983513e355..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/postprocess.py
+++ /dev/null
@@ -1,48 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""postprocess for 310 inference"""
-import os
-import argparse
-import json
-import numpy as np
-from mindspore.nn import Top1CategoricalAccuracy, Top5CategoricalAccuracy
-parser = argparse.ArgumentParser(description="postprocess")
-label_path = "./preprocess_Result/cifar10_label_ids.npy"
-parser.add_argument("--result_dir", type=str, default="./result_Files", help="result files path.")
-parser.add_argument('--dataset_name', type=str, default="imagenet2012")
-parser.add_argument("--label_dir", type=str, default=label_path, help="image file path.")
-args = parser.parse_args()
-
-def calcul_acc(lab, preds):
- return sum(1 for x, y in zip(lab, preds) if x == y) / len(lab)
-if __name__ == '__main__':
- batch_size = 1
- top1_acc = Top1CategoricalAccuracy()
- rst_path = args.result_dir
- label_list = []
- pred_list = []
- file_list = os.listdir(rst_path)
- top5_acc = Top5CategoricalAccuracy()
- with open('./preprocess_Result/imagenet_label.json', "r") as label:
- labels = json.load(label)
- for f in file_list:
- label = f.split("_0.bin")[0] + ".JPEG"
- label_list.append(labels[label])
- pred = np.fromfile(os.path.join(rst_path, f), np.float32)
- pred = pred.reshape(batch_size, int(pred.shape[0] / batch_size))
- top1_acc.update(pred, [labels[label],])
- top5_acc.update(pred, [labels[label],])
- print("Top1 acc: ", top1_acc.eval())
- print("Top5 acc: ", top5_acc.eval())
diff --git a/research/cv/glore_res50/preprocess.py b/research/cv/glore_res50/preprocess.py
deleted file mode 100644
index da3a7f8f5ebf3b7645f493246bb5181bbe5990dd..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/preprocess.py
+++ /dev/null
@@ -1,48 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""preprocess"""
-import os
-import argparse
-import json
-parser = argparse.ArgumentParser('preprocess')
-parser.add_argument('--dataset_name', type=str, choices=["cifar10", "imagenet2012"], default="imagenet2012")
-parser.add_argument('--data_path', type=str, default='', help='eval data dir')
-parser.add_argument('--result_path', type=str, default='./preprocess_Result/', help='result path')
-def create_label(result_path, dir_path):
- """create label json for imagenet"""
- print("[WARNING] Create imagenet label. Currently only use for Imagenet2012!")
- dirs = os.listdir(dir_path)
- file_list = []
- for file in dirs:
- file_list.append(file)
- file_list = sorted(file_list)
-
- total = 0
- img_label = {}
- for i, file_dir in enumerate(file_list):
- files = os.listdir(os.path.join(dir_path, file_dir))
- for f in files:
- img_label[f] = i
- total += len(files)
-
- json_file = os.path.join(result_path, "imagenet_label.json")
- with open(json_file, "w+") as label:
- json.dump(img_label, label)
-
- print("[INFO] Completed! Total {} data.".format(total))
-
-args = parser.parse_args()
-if __name__ == "__main__":
- create_label('./preprocess_Result/', args.data_path)
diff --git a/research/cv/glore_res50/requirements.txt b/research/cv/glore_res50/requirements.txt
deleted file mode 100644
index 5a619e015a5ba164371d32865c44aa47a3aa416c..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/requirements.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-numpy
-easydict
diff --git a/research/cv/glore_res50/scripts/docker_start.sh b/research/cv/glore_res50/scripts/docker_start.sh
deleted file mode 100644
index a033c3e3d5f6d0032397f027c2e2d9427429fa3d..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/scripts/docker_start.sh
+++ /dev/null
@@ -1,35 +0,0 @@
-#!/bin/bash
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.mitations under the License.
-
-docker_image=$1
-data_dir=$2
-model_dir=$3
-
-docker run -it --ipc=host \
- --device=/dev/davinci0 \
- --device=/dev/davinci1 \
- --device=/dev/davinci2 \
- --device=/dev/davinci3 \
- --device=/dev/davinci4 \
- --device=/dev/davinci5 \
- --device=/dev/davinci6 \
- --device=/dev/davinci7 \
- --device=/dev/davinci_manager \
- --device=/dev/devmm_svm --device=/dev/hisi_hdc \
- -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
- -v /usr/local/Ascend/add-ons/:/usr/local/Ascend/add-ons/ \
- -v ${model_dir}:${model_dir} \
- -v ${data_dir}:${data_dir} \
- -v /root/ascend/log:/root/ascend/log ${docker_image} /bin/bash
diff --git a/research/cv/glore_res50/scripts/run_310_infer.sh b/research/cv/glore_res50/scripts/run_310_infer.sh
deleted file mode 100644
index db8af43d84a62d54a95066943115f15906c5d070..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/scripts/run_310_infer.sh
+++ /dev/null
@@ -1,125 +0,0 @@
-#!/bin/bash
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-
-if [[ $# -lt 2 || $# -gt 3 ]]; then
- echo "Usage: bash run_infer_310.sh [MINDIR_PATH] [DATASET_PATH] [DEVICE_ID]
- DEVICE_ID is optional, it can be set by environment variable device_id, otherwise the value is zero"
-exit 1
-fi
-
-get_real_path(){
- if [ "${1:0:1}" == "/" ]; then
- echo "$1"
- else
- echo "$(realpath -m $PWD/$1)"
- fi
-}
-model=$(get_real_path $1)
-dataset_name='imagenet2012'
-dataset_path=$(get_real_path $2)
-need_preprocess='y'
-device_id=0
-if [ $# == 3 ]; then
- device_id=$3
-fi
-
-echo "mindir name: "$model
-echo "dataset name: "$dataset_name
-echo "dataset path: "$dataset_path
-echo "need preprocess: "$need_preprocess
-echo "device id: "$device_id
-
-export ASCEND_HOME=/usr/local/Ascend/
-if [ -d ${ASCEND_HOME}/ascend-toolkit ]; then
- export PATH=$ASCEND_HOME/fwkacllib/bin:$ASCEND_HOME/fwkacllib/ccec_compiler/bin:$ASCEND_HOME/ascend-toolkit/latest/fwkacllib/ccec_compiler/bin:$ASCEND_HOME/ascend-toolkit/latest/atc/bin:$PATH
- export LD_LIBRARY_PATH=$ASCEND_HOME/fwkacllib/lib64:/usr/local/lib:$ASCEND_HOME/ascend-toolkit/latest/atc/lib64:$ASCEND_HOME/ascend-toolkit/latest/fwkacllib/lib64:$ASCEND_HOME/driver/lib64:$ASCEND_HOME/add-ons:$LD_LIBRARY_PATH
- export TBE_IMPL_PATH=$ASCEND_HOME/ascend-toolkit/latest/opp/op_impl/built-in/ai_core/tbe
- export PYTHONPATH=$ASCEND_HOME/fwkacllib/python/site-packages:${TBE_IMPL_PATH}:$ASCEND_HOME/ascend-toolkit/latest/fwkacllib/python/site-packages:$PYTHONPATH
- export ASCEND_OPP_PATH=$ASCEND_HOME/ascend-toolkit/latest/opp
-else
- export ASCEND_HOME=/usr/local/Ascend/latest/
- export PATH=$ASCEND_HOME/fwkacllib/bin:$ASCEND_HOME/fwkacllib/ccec_compiler/bin:$ASCEND_HOME/atc/ccec_compiler/bin:$ASCEND_HOME/atc/bin:$PATH
- export LD_LIBRARY_PATH=$ASCEND_HOME/fwkacllib/lib64:/usr/local/lib:$ASCEND_HOME/atc/lib64:$ASCEND_HOME/acllib/lib64:$ASCEND_HOME/driver/lib64:$ASCEND_HOME/add-ons:$LD_LIBRARY_PATH
- export PYTHONPATH=$ASCEND_HOME/fwkacllib/python/site-packages:$ASCEND_HOME/atc/python/site-packages:$PYTHONPATH
- export ASCEND_OPP_PATH=$ASCEND_HOME/opp
-fi
-export ASCEND_HOME=/usr/local/Ascend
-export PATH=$ASCEND_HOME/fwkacllib/ccec_compiler/bin:$ASCEND_HOME/fwkacllib/bin:$ASCEND_HOME/toolkit/bin:$PATH
-export LD_LIBRARY_PATH=/usr/local/lib/:/usr/local/fwkacllib/lib64:$ASCEND_HOME/driver/lib64:$ASCEND_HOME/add-ons:/usr/local/Ascend/toolkit/lib64:$LD_LIBRARY_PATH
-export PYTHONPATH=$ASCEND_HOME/fwkacllib/python/site-packages
-export PATH=/usr/local/python375/bin:$PATH
-export NPU_HOST_LIB=/usr/local/Ascend/acllib/lib64/stub
-export ASCEND_OPP_PATH=/usr/local/Ascend/opp
-export ASCEND_AICPU_PATH=/usr/local/Ascend
-export LD_LIBRARY_PATH=/usr/local/lib64/:$LD_LIBRARY_PATH
-function preprocess_data()
-{
- if [ -d preprocess_Result ]; then
- rm -rf ./preprocess_Result
- fi
- mkdir preprocess_Result
- python3.7 ../preprocess.py --dataset_name=$dataset_name --data_path=$dataset_path
-}
-
-function compile_app()
-{
- cd ../ascend310_infer/ || exit
- bash build.sh &> build.log
-}
-
-function infer()
-{
- cd - || exit
- if [ -d result_Files ]; then
- rm -rf ./result_Files
- fi
- if [ -d time_Result ]; then
- rm -rf ./time_Result
- fi
- mkdir result_Files
- mkdir time_Result
-
- ../ascend310_infer/out/main --mindir_path=$model --dataset_name=$dataset_name --input0_path=$dataset_path --device_id=$device_id &> infer.log
-
-}
-
-function cal_acc()
-{
- python3.7 ../postprocess.py --dataset_name=$dataset_name &> acc.log
-}
-
-if [ $need_preprocess == "y" ]; then
- preprocess_data
- if [ $? -ne 0 ]; then
- echo "preprocess dataset failed"
- exit 1
- fi
-fi
-compile_app
-if [ $? -ne 0 ]; then
- echo "compile app code failed"
- exit 1
-fi
-infer
-if [ $? -ne 0 ]; then
- echo " execute inference failed"
- exit 1
-fi
-cal_acc
-if [ $? -ne 0 ]; then
- echo "calculate accuracy failed"
- exit 1
-fi
diff --git a/research/cv/glore_res50/scripts/run_distribute_train.sh b/research/cv/glore_res50/scripts/run_distribute_train.sh
deleted file mode 100644
index 9a34e873c802fdd563d88929dcfe13a6e7efcd40..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/scripts/run_distribute_train.sh
+++ /dev/null
@@ -1,79 +0,0 @@
-#!/bin/bash
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-echo "=============================================================================================================="
-echo "Please run the script as: "
-echo "bash run.sh DATA_PATH RANK_SIZE"
-echo "For example: bash run.sh /path/dataset 8"
-echo "It is better to use the absolute path."
-echo "=============================================================================================================="
-set -e
-DATA_PATH=$1
-export DATA_PATH=${DATA_PATH}
-RANK_SIZE=$2
-
-EXEC_PATH=$(pwd)
-
-echo "$EXEC_PATH"
-
-test_dist_8pcs()
-{
- export RANK_TABLE_FILE=${EXEC_PATH}/rank_table_8pcs.json
- export RANK_SIZE=8
-}
-
-test_dist_2pcs()
-{
- export RANK_TABLE_FILE=${EXEC_PATH}/rank_table_2pcs.json
- export RANK_SIZE=2
-}
-
-test_dist_${RANK_SIZE}pcs
-
-export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
-
-for((i=1;i<${RANK_SIZE};i++))
-do
- rm -rf device$i
- mkdir device$i
- cd ./device$i
- mkdir src
- cd ../
- cp ../*.py ./device$i
- cp ../src/*.py ./device$i/src
- cd ./device$i
- export DEVICE_ID=$i
- export RANK_ID=$i
- echo "start training for device $i"
- env > env$i.log
- python train.py --data_url $1 --is_modelarts False --run_distribute True > train$i.log 2>&1 &
- echo "$i finish"
- cd ../
-done
-rm -rf device0
-mkdir device0
-cd ./device0
-mkdir src
-cd ../
-cp ../*.py ./device0
-cp ../src/*.py ./device0/src
-cd ./device0
-export DEVICE_ID=0
-export RANK_ID=0
-echo "start training for device 0"
-env > env0.log
-python train.py --data_url $1 --is_modelarts False --run_distribute True > train0.log 2>&1 &
-
-echo "training in the background."
diff --git a/research/cv/glore_res50/scripts/run_eval.sh b/research/cv/glore_res50/scripts/run_eval.sh
deleted file mode 100644
index 1f6a66e7128f886274c4fed1c0ab84082593ac20..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/scripts/run_eval.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/bin/bash
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-echo "=============================================================================================================="
-echo "Please run the script as: "
-echo "bash run.sh DATA_PATH DEVICE_ID CKPT_PATH"
-echo "For example: bash run.sh /path/dataset 0 /path/ckpt"
-echo "It is better to use the absolute path."
-echo "=============================================================================================================="
-set -e
-DATA_PATH=$1
-DEVICE_ID=$2
-export DATA_PATH=${DATA_PATH}
-
-EXEC_PATH=$(pwd)
-
-echo "$EXEC_PATH"
-
-export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
-
-cd ../
-rm -rf eval/
-mkdir eval
-export DEVICE_ID=$2
-export RANK_ID=$2
-env > env0.log
-python3 eval.py --data_url $1 --is_modelarts False --device_id $2 --ckpt_url $3> ./eval/eval.log 2>&1
-
-if [ $? -eq 0 ];then
- echo "evaling success"
-else
- echo "evaling failed"
- exit 2
-fi
-echo "finish"
-cd ../
diff --git a/research/cv/glore_res50/scripts/run_standalone_train.sh b/research/cv/glore_res50/scripts/run_standalone_train.sh
deleted file mode 100644
index 6707037ed27b958f037f571b98f7cf2396e8aba0..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/scripts/run_standalone_train.sh
+++ /dev/null
@@ -1,43 +0,0 @@
-#!/bin/bash
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-echo "=============================================================================================================="
-echo "Please run the script as: "
-echo "bash run.sh DATA_PATH DEVICE_ID"
-echo "For example: bash run.sh /path/dataset 0"
-echo "It is better to use the absolute path."
-echo "=============================================================================================================="
-set -e
-DATA_PATH=$1
-DEVICE_ID=$2
-export DATA_PATH=${DATA_PATH}
-EXEC_PATH=$(pwd)
-echo "$EXEC_PATH"
-export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
-cd ../
-rm -rf train
-mkdir train
-export DEVICE_ID=$2
-export RANK_ID=$2
-env > env0.log
-echo "Standalone train begin."
-python3 train.py --data_url $1 --is_modelarts False --run_distribute False --device_id $2 > ./train/train_alone.log 2>&1
-if [ $? -eq 0 ];then
- echo "training success"
-else
- echo "training failed"
- exit 2
-fi
-echo "finish"
diff --git a/research/cv/glore_res50/src/__init__.py b/research/cv/glore_res50/src/__init__.py
deleted file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000
diff --git a/research/cv/glore_res50/src/autoaugment.py b/research/cv/glore_res50/src/autoaugment.py
deleted file mode 100644
index 12a5565794ec1d74913cf0657d6c32c77ea765b0..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/src/autoaugment.py
+++ /dev/null
@@ -1,191 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""define autoaugment"""
-import os
-import mindspore.dataset.engine as de
-import mindspore.dataset.transforms.c_transforms as c_transforms
-import mindspore.dataset.vision.c_transforms as c_vision
-from mindspore import dtype as mstype
-from mindspore.communication.management import init, get_rank, get_group_size
-
-# define Auto Augmentation operators
-PARAMETER_MAX = 10
-
-
-def float_parameter(level, maxval):
- return float(level) * maxval / PARAMETER_MAX
-
-
-def int_parameter(level, maxval):
- return int(level * maxval / PARAMETER_MAX)
-
-
-def shear_x(level):
- v = float_parameter(level, 0.3)
- return c_transforms.RandomChoice(
- [c_vision.RandomAffine(degrees=0, shear=(-v, -v)), c_vision.RandomAffine(degrees=0, shear=(v, v))])
-
-
-def shear_y(level):
- v = float_parameter(level, 0.3)
- return c_transforms.RandomChoice(
- [c_vision.RandomAffine(degrees=0, shear=(0, 0, -v, -v)), c_vision.RandomAffine(degrees=0, shear=(0, 0, v, v))])
-
-
-def translate_x(level):
- v = float_parameter(level, 150 / 331)
- return c_transforms.RandomChoice(
- [c_vision.RandomAffine(degrees=0, translate=(-v, -v)), c_vision.RandomAffine(degrees=0, translate=(v, v))])
-
-
-def translate_y(level):
- v = float_parameter(level, 150 / 331)
- return c_transforms.RandomChoice([c_vision.RandomAffine(degrees=0, translate=(0, 0, -v, -v)),
- c_vision.RandomAffine(degrees=0, translate=(0, 0, v, v))])
-
-
-def color_impl(level):
- v = float_parameter(level, 1.8) + 0.1
- return c_vision.RandomColor(degrees=(v, v))
-
-
-def rotate_impl(level):
- v = int_parameter(level, 30)
- return c_transforms.RandomChoice(
- [c_vision.RandomRotation(degrees=(-v, -v)), c_vision.RandomRotation(degrees=(v, v))])
-
-
-def solarize_impl(level):
- level = int_parameter(level, 256)
- v = 256 - level
- return c_vision.RandomSolarize(threshold=(0, v))
-
-
-def posterize_impl(level):
- level = int_parameter(level, 4)
- v = 4 - level
- return c_vision.RandomPosterize(bits=(v, v))
-
-
-def contrast_impl(level):
- v = float_parameter(level, 1.8) + 0.1
- return c_vision.RandomColorAdjust(contrast=(v, v))
-
-
-def autocontrast_impl(level):
- return c_vision.AutoContrast()
-
-
-def sharpness_impl(level):
- v = float_parameter(level, 1.8) + 0.1
- return c_vision.RandomSharpness(degrees=(v, v))
-
-
-def brightness_impl(level):
- v = float_parameter(level, 1.8) + 0.1
- return c_vision.RandomColorAdjust(brightness=(v, v))
-
-
-# define the Auto Augmentation policy
-imagenet_policy = [
- [(posterize_impl(8), 0.4), (rotate_impl(9), 0.6)],
- [(solarize_impl(5), 0.6), (autocontrast_impl(5), 0.6)],
- [(c_vision.Equalize(), 0.8), (c_vision.Equalize(), 0.6)],
- [(posterize_impl(7), 0.6), (posterize_impl(6), 0.6)],
- [(c_vision.Equalize(), 0.4), (solarize_impl(4), 0.2)],
-
- [(c_vision.Equalize(), 0.4), (rotate_impl(8), 0.8)],
- [(solarize_impl(3), 0.6), (c_vision.Equalize(), 0.6)],
- [(posterize_impl(5), 0.8), (c_vision.Equalize(), 1.0)],
- [(rotate_impl(3), 0.2), (solarize_impl(8), 0.6)],
- [(c_vision.Equalize(), 0.6), (posterize_impl(6), 0.4)],
-
- [(rotate_impl(8), 0.8), (color_impl(0), 0.4)],
- [(rotate_impl(9), 0.4), (c_vision.Equalize(), 0.6)],
- [(c_vision.Equalize(), 0.0), (c_vision.Equalize(), 0.8)],
- [(c_vision.Invert(), 0.6), (c_vision.Equalize(), 1.0)],
- [(color_impl(4), 0.6), (contrast_impl(8), 1.0)],
-
- [(rotate_impl(8), 0.8), (color_impl(2), 1.0)],
- [(color_impl(8), 0.8), (solarize_impl(7), 0.8)],
- [(sharpness_impl(7), 0.4), (c_vision.Invert(), 0.6)],
- [(shear_x(5), 0.6), (c_vision.Equalize(), 1.0)],
- [(color_impl(0), 0.4), (c_vision.Equalize(), 0.6)],
-
- [(c_vision.Equalize(), 0.4), (solarize_impl(4), 0.2)],
- [(solarize_impl(5), 0.6), (autocontrast_impl(5), 0.6)],
- [(c_vision.Invert(), 0.6), (c_vision.Equalize(), 1.0)],
- [(color_impl(4), 0.6), (contrast_impl(8), 1.0)],
- [(c_vision.Equalize(), 0.8), (c_vision.Equalize(), 0.6)],
-]
-
-
-def autoaugment(dataset_path, repeat_num=1, batch_size=32, target="Ascend"):
- """
- define dataset with autoaugment
- """
- if target == "Ascend":
- device_num, rank_id = _get_rank_info()
- else:
- init("nccl")
- rank_id = get_rank()
- device_num = get_group_size()
-
- if device_num == 1:
- ds = de.ImageFolderDataset(dataset_path, num_parallel_workers=8, shuffle=True)
- else:
- ds = de.ImageFolderDataset(dataset_path, num_parallel_workers=8, shuffle=True,
- num_shards=device_num, shard_id=rank_id)
-
- image_size = 224
- mean = [0.485 * 255, 0.456 * 255, 0.406 * 255]
- std = [0.229 * 255, 0.224 * 255, 0.225 * 255]
- trans = [
- c_vision.RandomCropDecodeResize(image_size, scale=(0.08, 1.0), ratio=(0.75, 1.333)),
- ]
-
- post_trans = [
- c_vision.RandomHorizontalFlip(prob=0.5),
- c_vision.Normalize(mean=mean, std=std),
- c_vision.HWC2CHW()
- ]
- dataset = ds.map(operations=trans, input_columns="image")
- dataset = dataset.map(operations=c_vision.RandomSelectSubpolicy(imagenet_policy), input_columns=["image"])
- dataset = dataset.map(operations=post_trans, input_columns="image")
-
- type_cast_op = c_transforms.TypeCast(mstype.int32)
- dataset = dataset.map(operations=type_cast_op, input_columns="label")
- # apply the batch operation
- dataset = dataset.batch(batch_size, drop_remainder=True)
- # apply the repeat operation
- dataset = dataset.repeat(repeat_num)
-
- return dataset
-
-
-def _get_rank_info():
- """
- get rank size and rank id
- """
- rank_size = int(os.environ.get("RANK_SIZE", 1))
-
- if rank_size > 1:
- rank_size = int(os.environ.get("RANK_SIZE"))
- rank_id = int(os.environ.get("RANK_ID"))
- else:
- rank_size = 1
- rank_id = 0
-
- return rank_size, rank_id
diff --git a/research/cv/glore_res50/src/config.py b/research/cv/glore_res50/src/config.py
deleted file mode 100644
index 331e9a0f6bb30c0d2218c24adeb3ff2bcc6d3847..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/src/config.py
+++ /dev/null
@@ -1,40 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""config for train or evaluation"""
-from easydict import EasyDict
-
-config = EasyDict({
- "class_num": 1000,
- "batch_size": 128,
- "loss_scale": 1024,
- "momentum": 0.9,
- "weight_decay": 1e-4,
- "epoch_size": 120,
- "pretrained": False,
- "pretrain_epoch_size": 0,
- "save_checkpoint": True,
- "save_checkpoint_epochs": 5,
- "keep_checkpoint_max": 5,
- "save_checkpoint_path": "./",
- "warmup_epochs": 5,
- "lr_decay_mode": "poly",
- "use_label_smooth": True,
- "use_autoaugment": True,
- "label_smooth_factor": 0.1,
- "weight_init": "xavier_uniform",
- "lr_init": 0,
- "lr_max": 0.6,
- "lr_end": 0.0
-})
diff --git a/research/cv/glore_res50/src/dataset.py b/research/cv/glore_res50/src/dataset.py
deleted file mode 100644
index e6fb0865a7dd3db3cebc5d74a91d6457f3ae6766..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/src/dataset.py
+++ /dev/null
@@ -1,194 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""
-create train or eval dataset.
-"""
-import os
-import mindspore.common.dtype as mstype
-import mindspore.dataset as ds
-import mindspore.dataset.vision.c_transforms as C
-import mindspore.dataset.transforms.c_transforms as C2
-from mindspore.communication.management import init, get_rank, get_group_size
-
-
-def cifar10(dataset_path, do_train, repeat_num=1, batch_size=32, target="Ascend", distribute=False):
- """
- create a train or evaluate cifar10 dataset for resnet50
- Args:
- dataset_path(string): the path of dataset.
- do_train(bool): whether dataset is used for train or eval.
- repeat_num(int): the repeat times of dataset. Default: 1
- batch_size(int): the batch size of dataset. Default: 32
- target(str): the device target. Default: Ascend
- distribute(bool): data for distribute or not. Default: False
-
- Returns:
- dataset
- """
- if target == "Ascend":
- device_num, rank_id = _get_rank_info()
- else:
- if distribute:
- init()
- rank_id = get_rank()
- device_num = get_group_size()
- else:
- device_num = 1
- if device_num == 1:
- data_set = ds.Cifar10Dataset(dataset_path, num_parallel_workers=8, shuffle=True)
- else:
- data_set = ds.Cifar10Dataset(dataset_path, num_parallel_workers=8, shuffle=True,
- num_shards=device_num, shard_id=rank_id)
-
- # define map operations
- trans = []
- if do_train:
- trans += [
- C.RandomCrop((32, 32), (4, 4, 4, 4)),
- C.RandomHorizontalFlip(prob=0.5)
- ]
-
- trans += [
- C.Resize((224, 224)),
- C.Rescale(1.0 / 255.0, 0.0),
- C.Normalize([0.4914, 0.4822, 0.4465], [0.2023, 0.1994, 0.2010]),
- C.HWC2CHW()
- ]
-
- type_cast_op = C2.TypeCast(mstype.int32)
-
- data_set = data_set.map(operations=type_cast_op, input_columns="label", num_parallel_workers=8)
- data_set = data_set.map(operations=trans, input_columns="image", num_parallel_workers=8)
-
- # apply batch operations
- data_set = data_set.batch(batch_size, drop_remainder=True)
- # apply dataset repeat operation
- data_set = data_set.repeat(repeat_num)
-
- return data_set
-
-
-def create_train_dataset(dataset_path, repeat_num=1, batch_size=32, target="Ascend"):
- """
- create a train or eval imagenet2012 dataset for resnet50
-
- Args:
- dataset_path(string): the path of dataset.
- repeat_num(int): the repeat times of dataset. Default: 1
- batch_size(int): the batch size of dataset. Default: 32
- target(str): the device target. Default: Ascend
- distribute(bool): data for distribute or not. Default: False
-
- Returns:
- dataset
- """
- if target == "Ascend":
- device_num, rank_id = _get_rank_info()
-
- if device_num == 1:
- data_set = ds.ImageFolderDataset(dataset_path, num_parallel_workers=8, shuffle=True)
- else:
- data_set = ds.ImageFolderDataset(dataset_path, num_parallel_workers=8, shuffle=True,
- num_shards=device_num, shard_id=rank_id)
-
- image_size = 224
- mean = [0.485 * 255, 0.456 * 255, 0.406 * 255]
- std = [0.229 * 255, 0.224 * 255, 0.225 * 255]
-
- # std = [127.5, 127.5, 127.5]
- # mean = [127.5, 127.5, 127.5]
-
- # define map operations
-
- trans = [
- C.RandomCropDecodeResize(image_size, scale=(0.08, 1.0), ratio=(0.75, 1.333)),
- C.RandomHorizontalFlip(prob=0.5),
- C.Normalize(mean=mean, std=std),
- C.HWC2CHW()
- ]
-
- type_cast_op = C2.TypeCast(mstype.int32)
-
- data_set = data_set.map(operations=trans, input_columns="image", num_parallel_workers=8)
- data_set = data_set.map(operations=type_cast_op, input_columns="label", num_parallel_workers=8)
-
- # apply batch operations
- data_set = data_set.batch(batch_size, drop_remainder=True)
-
- # apply dataset repeat operation
- data_set = data_set.repeat(repeat_num)
-
- return data_set
-
-
-def create_eval_dataset(dataset_path, repeat_num=1, batch_size=32, target="Ascend"):
- """
- create a train or eval imagenet2012 dataset for resnet50
-
- Args:
- dataset_path(string): the path of dataset.
- repeat_num(int): the repeat times of dataset. Default: 1
- batch_size(int): the batch size of dataset. Default: 32
- target(str): the device target. Default: Ascend
- Returns:
- dataset
- """
- data_set = ds.ImageFolderDataset(dataset_path, num_parallel_workers=8, shuffle=True)
-
- image_size = 224
- mean = [0.485 * 255, 0.456 * 255, 0.406 * 255]
- std = [0.229 * 255, 0.224 * 255, 0.225 * 255]
-
- # std = [127.5, 127.5, 127.5]
- # mean = [127.5, 127.5, 127.5]
-
- # define map operations
-
- trans = [
- C.Decode(),
- C.Resize(256),
- C.CenterCrop(image_size),
- C.Normalize(mean=mean, std=std),
- C.HWC2CHW()
- ]
-
- type_cast_op = C2.TypeCast(mstype.int32)
-
- data_set = data_set.map(operations=trans, input_columns="image", num_parallel_workers=8)
- data_set = data_set.map(operations=type_cast_op, input_columns="label", num_parallel_workers=8)
-
- # apply batch operations
- data_set = data_set.batch(batch_size, drop_remainder=True)
-
- # apply dataset repeat operation
- data_set = data_set.repeat(repeat_num)
-
- return data_set
-
-
-def _get_rank_info():
- """
- get rank size and rank id
- """
- rank_size = int(os.environ.get("RANK_SIZE", 1))
-
- if rank_size > 1:
- rank_size = get_group_size()
- rank_id = get_rank()
- else:
- rank_size = 1
- rank_id = 0
-
- return rank_size, rank_id
diff --git a/research/cv/glore_res50/src/glore_resnet50.py b/research/cv/glore_res50/src/glore_resnet50.py
deleted file mode 100644
index 39e7029526091eda267bab097d41bd829cdc5d69..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/src/glore_resnet50.py
+++ /dev/null
@@ -1,394 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""
-create glore_resnet50
-"""
-from collections import OrderedDict
-import numpy as np
-import mindspore.nn as nn
-import mindspore.common.dtype as mstype
-from mindspore.ops import operations as P
-from mindspore.common.tensor import Tensor
-
-
-
-def _weight_variable(shape, factor=0.01):
- init_value = np.random.randn(*shape).astype(np.float32) * factor
- return Tensor(init_value)
-
-
-def _conv3x3(in_channel, out_channel, stride=1):
- weight_shape = (out_channel, in_channel, 3, 3)
- weight = _weight_variable(weight_shape)
- return nn.Conv2d(in_channel, out_channel,
- kernel_size=3, stride=stride, padding=0, pad_mode='same', weight_init=weight)
-
-
-def _conv1x1(in_channel, out_channel, stride=1):
- weight_shape = (out_channel, in_channel, 1, 1)
- weight = _weight_variable(weight_shape)
- return nn.Conv2d(in_channel, out_channel,
- kernel_size=1, stride=stride, padding=0, pad_mode='same', weight_init=weight)
-
-
-def _conv7x7(in_channel, out_channel, stride=1):
- weight_shape = (out_channel, in_channel, 7, 7)
- weight = _weight_variable(weight_shape)
- return nn.Conv2d(in_channel, out_channel,
- kernel_size=7, stride=stride, padding=0, pad_mode='same', weight_init=weight)
-
-
-def _bn(channel):
- return nn.BatchNorm2d(channel, eps=1e-4, momentum=0.08,
- gamma_init=1, beta_init=0, moving_mean_init=0, moving_var_init=1)
-
-
-def _bn_last(channel):
- return nn.BatchNorm2d(channel, eps=1e-4, momentum=0.08,
- gamma_init=0, beta_init=0, moving_mean_init=0, moving_var_init=1)
-
-
-def _fc(in_channel, out_channel):
- weight_shape = (out_channel, in_channel)
- weight = _weight_variable(weight_shape)
- return nn.Dense(in_channel, out_channel, has_bias=True, weight_init=weight, bias_init=0)
-
-
-class GCN(nn.Cell):
- """
- Graph convolution unit (single layer)
- """
-
- def __init__(self, num_state, num_mode, bias=False):
- super(GCN, self).__init__()
- self.conv1 = nn.Conv1d(num_mode, num_mode, kernel_size=1)
- self.relu = nn.ReLU()
- self.conv2 = nn.Conv1d(num_state, num_state, kernel_size=1, has_bias=bias)
- self.transpose = P.Transpose()
- self.add = P.Add()
-
- def construct(self, x):
- """construct GCN"""
- identity = x
- # (n, num_state, num_node) -> (n, num_node, num_state)
- # -> (n, num_state, num_node)
- out = self.transpose(x, (0, 2, 1))
- out = self.conv1(out)
- out = self.transpose(out, (0, 2, 1))
- out = self.add(out, identity)
- out = self.relu(out)
- out = self.conv2(out)
- return out
-
-
-class GloreUnit(nn.Cell):
- """
- Graph-based Global Reasoning Unit
- Parameter:
- 'normalize' is not necessary if the input size is fixed
- Args:
- num_in: Input channel
- num_mid:
- """
-
- def __init__(self, num_in, num_mid,
- normalize=False):
- super(GloreUnit, self).__init__()
- self.normalize = normalize
- self.num_s = int(2 * num_mid) # 512 num_in = 1024
- self.num_n = int(1 * num_mid) # 256
- # reduce dim
- self.conv_state = nn.SequentialCell([_bn(num_in),
- nn.ReLU(),
- _conv1x1(num_in, self.num_s, stride=1)])
- # projection map
- self.conv_proj = nn.SequentialCell([_bn(num_in),
- nn.ReLU(),
- _conv1x1(num_in, self.num_n, stride=1)])
-
- self.gcn = GCN(num_state=self.num_s, num_mode=self.num_n)
-
- self.conv_extend = nn.SequentialCell([_bn_last(self.num_s),
- nn.ReLU(),
- _conv1x1(self.num_s, num_in, stride=1)])
-
- self.reshape = P.Reshape()
- self.matmul = P.BatchMatMul()
- self.transpose = P.Transpose()
- self.add = P.Add()
- self.cast = P.Cast()
-
- def construct(self, x):
- """construct Graph-based Global Reasoning Unit"""
- n = x.shape[0]
- identity = x
- # (n, num_in, h, w) --> (n, num_state, h, w)
- # --> (n, num_state, h*w)
- x_conv_state = self.conv_state(x)
- x_state_reshaped = self.reshape(x_conv_state, (n, self.num_s, -1))
-
- # (n, num_in, h, w) --> (n, num_node, h, w)
- # --> (n, num_node, h*w)
- x_conv_proj = self.conv_proj(x)
- x_proj_reshaped = self.reshape(x_conv_proj, (n, self.num_n, -1))
-
- # (n, num_in, h, w) --> (n, num_node, h, w)
- # --> (n, num_node, h*w)
- x_rproj_reshaped = x_proj_reshaped
-
- # projection: coordinate space -> interaction space
- # (n, num_state, h*w) x (n, num_node, h*w)T --> (n, num_state, num_node)
- x_proj_reshaped = self.transpose(x_proj_reshaped, (0, 2, 1))
-
- x_state_reshaped_fp16 = self.cast(x_state_reshaped, mstype.float16)
- x_proj_reshaped_fp16 = self.cast(x_proj_reshaped, mstype.float16)
- x_n_state_fp16 = self.matmul(x_state_reshaped_fp16, x_proj_reshaped_fp16)
- x_n_state = self.cast(x_n_state_fp16, mstype.float32)
-
- if self.normalize:
- x_n_state = x_n_state * (1. / x_state_reshaped.shape[2])
-
- # reasoning: (n, num_state, num_node) -> (n, num_state, num_node)
- x_n_rel = self.gcn(x_n_state)
-
- # reverse projection: interaction space -> coordinate space
- # (n, num_state, num_node) x (n, num_node, h*w) --> (n, num_state, h*w)
- x_n_rel_fp16 = self.cast(x_n_rel, mstype.float16)
- x_rproj_reshaped_fp16 = self.cast(x_rproj_reshaped, mstype.float16)
- x_state_reshaped_fp16 = self.matmul(x_n_rel_fp16, x_rproj_reshaped_fp16)
- x_state_reshaped = self.cast(x_state_reshaped_fp16, mstype.float32)
-
- # (n, num_state, h*w) --> (n, num_state, h, w)
- x_state = self.reshape(x_state_reshaped, (n, self.num_s, identity.shape[2], identity.shape[3]))
-
- # (n, num_state, h, w) -> (n, num_in, h, w)
- x_conv_extend = self.conv_extend(x_state)
- out = self.add(x_conv_extend, identity)
- return out
-
-
-class ResidualBlock(nn.Cell):
- """
- ResNet V1 residual block definition.
-
- Args:
- in_channel (int): Input channel.
- out_channel (int): Output channel.
- stride (int): Stride size for the first convolutional layer. Default: 1.
-
- Returns:
- Tensor, output tensor.
-
- Examples:
- >>> ResidualBlock(3, 256, stride=2)
- """
- expansion = 4
-
- def __init__(self,
- in_channel,
- out_channel,
- stride=1):
- super(ResidualBlock, self).__init__()
- self.stride = stride
- channel = out_channel // self.expansion
- self.conv1 = _conv1x1(in_channel, channel, stride=1)
- self.bn1 = _bn(channel)
-
- self.conv2 = _conv3x3(channel, channel, stride=stride)
- self.bn2 = _bn(channel)
-
- self.conv3 = _conv1x1(channel, out_channel, stride=1)
- self.bn3 = _bn_last(out_channel)
- self.relu = nn.ReLU()
- self.down_sample = False
- if stride != 1 or in_channel != out_channel:
- self.down_sample = True
- self.down_sample_layer = None
-
- if self.down_sample:
- self.down_sample_layer = nn.SequentialCell(
- [
- nn.AvgPool2d(kernel_size=stride, stride=stride),
- _conv1x1(in_channel, out_channel, stride=1),
- _bn(out_channel)
- ])
- self.add = P.Add()
-
- def construct(self, x):
- """construct ResidualBlock"""
- identity = x
-
- out = self.conv1(x)
- out = self.bn1(out)
- out = self.relu(out)
-
- out = self.conv2(out)
- out = self.bn2(out)
- out = self.relu(out)
- out = self.conv3(out)
- out = self.bn3(out)
-
- if self.down_sample:
- identity = self.down_sample_layer(identity)
-
- out = self.add(out, identity)
- out = self.relu(out)
-
- return out
-
-
-class ResNet(nn.Cell):
- """
- ResNet architecture.
-
- Args:
- block (Cell): Block for network.
- layer_nums (list): Numbers of block in different layers.
- in_channels (list): Input channel in each layer.
- out_channels (list): Output channel in each layer.
- strides (list): Stride size in each layer.
- num_classes (int): The number of classes that the training images are belonging to.
- Returns:
- Tensor, output tensor.
-
- Examples:
- >>> ResNet(ResidualBlock,
- >>> [3, 4, 6, 3],
- >>> [64, 256, 512, 1024],
- >>> [256, 512, 1024, 2048],
- >>> [1, 2, 2, 2],
- >>> 10)
- """
-
- def __init__(self,
- block,
- layer_nums,
- in_channels,
- out_channels,
- strides,
- num_classes,
- use_glore=False):
- super(ResNet, self).__init__()
-
- if not len(layer_nums) == len(in_channels) == len(out_channels) == 4:
- raise ValueError("the length of layer_num, in_channels, out_channels list must be 4!")
-
- self.conv1 = nn.SequentialCell(OrderedDict([
- ('conv_1', _conv3x3(3, 32, stride=2)),
- ('bn1', _bn(32)),
- ('relu1', nn.ReLU()),
- ('conv_2', _conv3x3(32, 32, stride=1)),
- ('bn2', _bn(32)),
- ('relu2', nn.ReLU()),
- ('conv_3', _conv3x3(32, 64, stride=1)),
- ]))
- self.bn1 = _bn(64)
- self.relu = P.ReLU()
- self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, pad_mode="same")
- self.layer1 = self._make_layer(block,
- layer_nums[0],
- in_channel=in_channels[0],
- out_channel=out_channels[0],
- stride=strides[0])
- self.layer2 = self._make_layer(block,
- layer_nums[1],
- in_channel=in_channels[1],
- out_channel=out_channels[1],
- stride=strides[1])
- self.layer3 = self._make_layer(block,
- layer_nums[2],
- in_channel=in_channels[2],
- out_channel=out_channels[2],
- stride=strides[2],
- use_glore=use_glore)
- self.layer4 = self._make_layer(block,
- layer_nums[3],
- in_channel=in_channels[3],
- out_channel=out_channels[3],
- stride=strides[3])
- self.mean = P.ReduceMean(keep_dims=True)
- self.flatten = nn.Flatten()
- self.end_point = _fc(out_channels[3], num_classes)
-
- def _make_layer(self, block, layer_num, in_channel, out_channel, stride,
- use_glore=False, glore_pos=None):
- """
- Make stage network of ResNet.
-
- Args:
- block (Cell): Resnet block.
- layer_num (int): Layer number.
- in_channel (int): Input channel.
- out_channel (int): Output channel.
- stride (int): Stride size for the first convolutional layer.
- Returns:
- SequentialCell, the output layer.
-
- Examples:
- >>> _make_layer(ResidualBlock, 3, 128, 256, 2)
- """
- if use_glore and glore_pos is None:
- glore_pos = [1, 3, 5]
-
- layers = []
- for i in range(1, layer_num + 1):
- resnet_block = block(in_channel=(in_channel if i == 1 else out_channel),
- out_channel=out_channel,
- stride=(stride if i == 1 else 1))
- layers.append(resnet_block)
- if use_glore and i in glore_pos:
- glore_unit = GloreUnit(out_channel, int(out_channel / 4))
- layers.append(glore_unit)
- return nn.SequentialCell(layers)
-
- def construct(self, x):
- """construct ResNet"""
- x = self.conv1(x)
- x = self.bn1(x)
- x = self.relu(x)
- c1 = self.maxpool(x)
-
- c2 = self.layer1(c1)
- c3 = self.layer2(c2)
- c4 = self.layer3(c3)
- c5 = self.layer4(c4)
-
- out = self.mean(c5, (2, 3))
- out = self.flatten(out)
- out = self.end_point(out)
-
- return out
-
-
-def glore_resnet50(class_num=1000, use_glore=True):
- """
- Get ResNet50 with GloreUnit neural network.
-
- Args:
- class_num (int): Class number.
- use_glore (bool)
- Returns:
- Cell, cell instance of ResNet50 neural network.
-
- Examples:
- >>> net = glore_resnet50(10)
- """
- return ResNet(ResidualBlock,
- [3, 4, 6, 3],
- [64, 256, 512, 1024],
- [256, 512, 1024, 2048],
- [1, 2, 2, 2],
- class_num,
- use_glore=use_glore)
diff --git a/research/cv/glore_res50/src/loss.py b/research/cv/glore_res50/src/loss.py
deleted file mode 100644
index ee5906a02b4dbedce4c79cc6844b8cf0d3e8f1ea..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/src/loss.py
+++ /dev/null
@@ -1,75 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""define loss for glore_resnet50"""
-import mindspore.nn as nn
-from mindspore import Tensor
-from mindspore.common import dtype as mstype
-from mindspore.nn.loss.loss import LossBase
-from mindspore.ops import functional as F
-from mindspore.ops import operations as P
-import mindspore.ops as ops
-
-
-class SoftmaxCrossEntropyExpand(nn.Cell): # pylint: disable=missing-docstring
- def __init__(self, sparse=False):
- super(SoftmaxCrossEntropyExpand, self).__init__()
- self.exp = ops.Exp()
- self.sum = ops.ReduceSum(keep_dims=True)
- self.onehot = ops.OneHot()
- self.on_value = Tensor(1.0, mstype.float32)
- self.off_value = Tensor(0.0, mstype.float32)
- self.div = ops.RealDiv()
- self.log = ops.Log()
- self.sum_cross_entropy = ops.ReduceSum(keep_dims=False)
- self.mul = ops.Mul()
- self.mul2 = ops.Mul()
- self.mean = ops.ReduceMean(keep_dims=False)
- self.sparse = sparse
- self.max = ops.ReduceMax(keep_dims=True)
- self.sub = ops.Sub()
- self.eps = Tensor(1e-24, mstype.float32)
-
- def construct(self, logit, label): # pylint: disable=missing-docstring
- logit_max = self.max(logit, -1)
- exp = self.exp(self.sub(logit, logit_max))
- exp_sum = self.sum(exp, -1)
- softmax_result = self.div(exp, exp_sum)
- if self.sparse:
- label = self.onehot(label, ops.shape(logit)[1], self.on_value, self.off_value)
-
- softmax_result_log = self.log(softmax_result + self.eps)
- loss = self.sum_cross_entropy((self.mul(softmax_result_log, label)), -1)
- loss = self.mul2(ops.scalar_to_array(-1.0), loss)
- loss = self.mean(loss, -1)
-
- return loss
-
-
-class CrossEntropySmooth(LossBase):
- """CrossEntropy"""
-
- def __init__(self, sparse=True, reduction='mean', smooth_factor=0., num_classes=1000):
- super(CrossEntropySmooth, self).__init__()
- self.onehot = P.OneHot()
- self.sparse = sparse
- self.on_value = Tensor(1.0 - smooth_factor, mstype.float32)
- self.off_value = Tensor(1.0 * smooth_factor / (num_classes - 1), mstype.float32)
- self.ce = nn.SoftmaxCrossEntropyWithLogits(reduction=reduction)
-
- def construct(self, logit, label):
- if self.sparse:
- label = self.onehot(label, F.shape(logit)[1], self.on_value, self.off_value)
- loss = self.ce(logit, label)
- return loss
diff --git a/research/cv/glore_res50/src/lr_generator.py b/research/cv/glore_res50/src/lr_generator.py
deleted file mode 100644
index ee35b35264049bb35e3c64809a0a80e85589b79d..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/src/lr_generator.py
+++ /dev/null
@@ -1,128 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""learning rate generator"""
-import math
-import numpy as np
-
-
-def get_lr(lr_init, lr_end, lr_max, warmup_epochs, total_epochs, steps_per_epoch, lr_decay_mode):
- """
- generate learning rate array
-
- Args:
- lr_init(float): init learning rate
- lr_end(float): end learning rate
- lr_max(float): max learning rate
- warmup_epochs(int): number of warmup epochs
- total_epochs(int): total epoch of training
- steps_per_epoch(int): steps of one epoch
- lr_decay_mode(string): learning rate decay mode, including steps, poly or default
-
- Returns:
- np.array, learning rate array
- """
- lr_each_step = []
- total_steps = steps_per_epoch * total_epochs
- warmup_steps = steps_per_epoch * warmup_epochs
- if lr_decay_mode == 'steps':
- decay_epoch_index = [0.3 * total_steps, 0.6 * total_steps, 0.8 * total_steps]
- for i in range(total_steps):
- if i < decay_epoch_index[0]:
- lr = lr_max
- elif i < decay_epoch_index[1]:
- lr = lr_max * 0.1
- elif i < decay_epoch_index[2]:
- lr = lr_max * 0.01
- else:
- lr = lr_max * 0.001
- lr_each_step.append(lr)
- elif lr_decay_mode == 'poly':
- if warmup_steps != 0:
- inc_each_step = (float(lr_max) - float(lr_init)) / float(warmup_steps)
- else:
- inc_each_step = 0
- for i in range(total_steps):
- if i < warmup_steps:
- lr = float(lr_init) + inc_each_step * float(i)
- else:
- base = (1.0 - (float(i) - float(warmup_steps)) / (float(total_steps) - float(warmup_steps)))
- lr = float(lr_max) * base * base
- if lr < 0.0:
- lr = 0.0
- lr_each_step.append(lr)
- elif lr_decay_mode == 'cosine':
- decay_steps = total_steps - warmup_steps
- for i in range(total_steps):
- if i < warmup_steps:
- lr_inc = (float(lr_max) - float(lr_init)) / float(warmup_steps)
- lr = float(lr_init) + lr_inc * (i + 1)
- else:
- linear_decay = (total_steps - i) / decay_steps
- cosine_decay = 0.5 * (1 + math.cos(math.pi * 2 * 0.47 * i / decay_steps))
- decayed = linear_decay * cosine_decay + 0.00001
- lr = lr_max * decayed
- lr_each_step.append(lr)
- else:
- for i in range(total_steps):
- if i < warmup_steps:
- lr = lr_init + (lr_max - lr_init) * i / warmup_steps
- else:
- lr = lr_max - (lr_max - lr_end) * (i - warmup_steps) / (total_steps - warmup_steps)
- lr_each_step.append(lr)
-
- lr_each_step = np.array(lr_each_step).astype(np.float32)
-
- return lr_each_step
-
-
-def linear_warmup_lr(current_step, warmup_steps, base_lr, init_lr):
- lr_inc = (float(base_lr) - float(init_lr)) / float(warmup_steps)
- lr = float(init_lr) + lr_inc * current_step
- return lr
-
-
-def warmup_cosine_annealing_lr(lr, steps_per_epoch, warmup_epochs, max_epoch=120, global_step=0):
- """
- generate learning rate array with cosine
-
- Args:
- lr(float): base learning rate
- steps_per_epoch(int): steps size of one epoch
- warmup_epochs(int): number of warmup epochs
- max_epoch(int): total epochs of training
- global_step(int): the current start index of lr array
- Returns:
- np.array, learning rate array
- """
- base_lr = lr
- warmup_init_lr = 0
- total_steps = int(max_epoch * steps_per_epoch)
- warmup_steps = int(warmup_epochs * steps_per_epoch)
- decay_steps = total_steps - warmup_steps
-
- lr_each_step = []
- for i in range(total_steps):
- if i < warmup_steps:
- lr = linear_warmup_lr(i + 1, warmup_steps, base_lr, warmup_init_lr)
- else:
- linear_decay = (total_steps - i) / decay_steps
- cosine_decay = 0.5 * (1 + math.cos(math.pi * 2 * 0.47 * i / decay_steps))
- decayed = linear_decay * cosine_decay + 0.00001
- lr = base_lr * decayed
- lr_each_step.append(lr)
-
- lr_each_step = np.array(lr_each_step).astype(np.float32)
- learning_rate = lr_each_step[global_step:]
- return learning_rate
diff --git a/research/cv/glore_res50/src/save_callback.py b/research/cv/glore_res50/src/save_callback.py
deleted file mode 100644
index 885507d14d35833417f72c6cef16fdd5153d0b6a..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/src/save_callback.py
+++ /dev/null
@@ -1,43 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""define savecallback, save best model while training."""
-from mindspore import save_checkpoint
-from mindspore.train.callback import Callback
-
-
-class SaveCallback(Callback):
- """
- define savecallback, save best model while training.
- """
- def __init__(self, model_save, eval_dataset_save, save_file_path):
- super(SaveCallback, self).__init__()
- self.model = model_save
- self.eval_dataset = eval_dataset_save
- self.acc = 0.78
- self.save_path = save_file_path
-
- def step_end(self, run_context):
- """
- eval and save model while training.
- """
- cb_params = run_context.original_args()
-
- result = self.model.eval(self.eval_dataset)
- print(result)
- if result['Accuracy'] > self.acc:
- self.acc = result['Accuracy']
- file_name = self.save_path + str(self.acc) + ".ckpt"
- save_checkpoint(save_obj=cb_params.train_network, ckpt_file_name=file_name)
- print("Save the maximum accuracy checkpoint,the accuracy is", self.acc)
diff --git a/research/cv/glore_res50/train.py b/research/cv/glore_res50/train.py
deleted file mode 100644
index 3231baf4265d2ef5cf2c59629bca168b812aba93..0000000000000000000000000000000000000000
--- a/research/cv/glore_res50/train.py
+++ /dev/null
@@ -1,182 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""
-################################train glore_resnet50################################
-python train.py
-"""
-import os
-import random
-import argparse
-import ast
-import numpy as np
-
-import mindspore.nn as nn
-from mindspore import context
-from mindspore import Tensor
-from mindspore import dataset as de
-from mindspore.nn.metrics import Accuracy
-from mindspore.communication.management import init
-import mindspore.common.initializer as weight_init
-from mindspore.nn.optim.momentum import Momentum
-from mindspore.context import ParallelMode
-from mindspore.train.model import Model
-from mindspore.train.loss_scale_manager import FixedLossScaleManager
-from mindspore.train.serialization import load_checkpoint, load_param_into_net
-from mindspore.train.callback import ModelCheckpoint, CheckpointConfig, LossMonitor, TimeMonitor
-
-from src.config import config
-from src.lr_generator import get_lr
-from src.dataset import create_train_dataset, create_eval_dataset, _get_rank_info
-from src.save_callback import SaveCallback
-from src.glore_resnet50 import glore_resnet50
-from src.loss import SoftmaxCrossEntropyExpand, CrossEntropySmooth
-from src.autoaugment import autoaugment
-
-parser = argparse.ArgumentParser(description='Image classification with glore_resnet50')
-parser.add_argument('--use_glore', type=ast.literal_eval, default=True, help='Enable GloreUnit')
-parser.add_argument('--run_distribute', type=ast.literal_eval, default=True, help='Run distribute')
-parser.add_argument('--data_url', type=str, default=None,
- help='Dataset path')
-parser.add_argument('--train_url', type=str)
-parser.add_argument('--device_target', type=str, default='Ascend', help='Device target')
-parser.add_argument('--device_id', type=int, default=0)
-parser.add_argument('--is_modelarts', type=ast.literal_eval, default=True)
-parser.add_argument('--pretrained_ckpt', type=str, default=None, help='Pretrained ckpt path')
-parser.add_argument('--parameter_server', type=ast.literal_eval, default=False, help='Run parameter server train')
-args_opt = parser.parse_args()
-
-if args_opt.is_modelarts:
- import moxing as mox
-
-random.seed(1)
-np.random.seed(1)
-de.config.set_seed(1)
-
-if __name__ == '__main__':
- target = args_opt.device_target
-
- # init context
- context.set_context(mode=context.GRAPH_MODE, device_target=target, save_graphs=False)
- if args_opt.run_distribute:
- if target == "Ascend":
- device_id = int(os.getenv('DEVICE_ID'))
- context.set_context(device_id=device_id)
- context.set_auto_parallel_context(parallel_mode=ParallelMode.DATA_PARALLEL, gradients_mean=True)
- init()
- else:
- if target == "Ascend":
- device_id = args_opt.device_id
- context.set_context(mode=context.GRAPH_MODE, device_target=target, save_graphs=False, device_id=device_id)
- # create dataset
- train_dataset_path = os.path.join(args_opt.data_url, 'train')
- eval_dataset_path = os.path.join(args_opt.data_url, 'val')
-
- # download dataset from obs to cache if train on ModelArts
- if args_opt.is_modelarts:
- mox.file.copy_parallel(src_url=args_opt.data_url, dst_url='/cache/dataset/device_' + os.getenv('DEVICE_ID'))
- train_dataset_path = '/cache/dataset/device_' + os.getenv('DEVICE_ID') + '/train'
- eval_dataset_path = '/cache/dataset/device_' + os.getenv('DEVICE_ID') + '/val'
- if config.use_autoaugment:
- print("===========Use autoaugment==========")
- train_dataset = autoaugment(dataset_path=train_dataset_path, repeat_num=1,
- batch_size=config.batch_size, target=target)
- else:
- train_dataset = create_train_dataset(dataset_path=train_dataset_path, repeat_num=1,
- batch_size=config.batch_size, target=target)
-
- eval_dataset = create_eval_dataset(dataset_path=eval_dataset_path, repeat_num=1, batch_size=config.batch_size)
-
- step_size = train_dataset.get_dataset_size()
-
- # define net
-
- net = glore_resnet50(class_num=config.class_num, use_glore=args_opt.use_glore)
-
- # init weight
- if config.pretrained:
- param_dict = load_checkpoint(args_opt.pretrained_ckpt)
- load_param_into_net(net, param_dict)
- else:
- for _, cell in net.cells_and_names():
- if isinstance(cell, (nn.Conv2d, nn.Conv1d)):
- if config.weight_init == 'xavier_uniform':
- cell.weight.default_input = weight_init.initializer(weight_init.XavierUniform(),
- cell.weight.shape,
- cell.weight.dtype)
- elif config.weight_init == 'he_uniform':
- cell.weight.default_input = weight_init.initializer(weight_init.HeUniform(),
- cell.weight.shape,
- cell.weight.dtype)
- else: # config.weight_init == 'he_normal' or the others
- cell.weight.default_input = weight_init.initializer(weight_init.HeNormal(),
- cell.weight.shape,
- cell.weight.dtype)
-
- if isinstance(cell, nn.Dense):
- cell.weight.default_input = weight_init.initializer(weight_init.TruncatedNormal(),
- cell.weight.shape,
- cell.weight.dtype)
-
- # init lr
- lr = get_lr(lr_init=config.lr_init, lr_end=config.lr_end, lr_max=config.lr_max, warmup_epochs=config.warmup_epochs,
- total_epochs=config.epoch_size, steps_per_epoch=step_size, lr_decay_mode=config.lr_decay_mode)
- lr = Tensor(lr)
-
- #
- # define opt
- decayed_params = []
- no_decayed_params = []
- for param in net.trainable_params():
- if 'beta' not in param.name and 'gamma' not in param.name and 'bias' not in param.name:
- decayed_params.append(param)
- else:
- no_decayed_params.append(param)
-
- group_params = [{'params': decayed_params, 'weight_decay': config.weight_decay},
- {'params': no_decayed_params},
- {'order_params': net.trainable_params()}]
- net_opt = Momentum(group_params, lr, config.momentum, loss_scale=config.loss_scale)
- # define loss, model
- if config.use_label_smooth:
- loss = CrossEntropySmooth(sparse=True, reduction="mean", smooth_factor=config.label_smooth_factor,
- num_classes=config.class_num)
- else:
- loss = SoftmaxCrossEntropyExpand(sparse=True)
- loss_scale = FixedLossScaleManager(config.loss_scale, drop_overflow_update=False)
- model = Model(net, loss_fn=loss, optimizer=net_opt, loss_scale_manager=loss_scale, metrics={"Accuracy": Accuracy()})
-
- # define callbacks
- time_cb = TimeMonitor(data_size=step_size)
- loss_cb = LossMonitor()
- cb = [time_cb, loss_cb]
- device_num, device_id = _get_rank_info()
- if config.save_checkpoint:
- if args_opt.is_modelarts:
- save_checkpoint_path = '/cache/train_output/device_' + os.getenv('DEVICE_ID') + '/'
- else:
- save_checkpoint_path = config.save_checkpoint_path
- config_ck = CheckpointConfig(save_checkpoint_steps=config.save_checkpoint_epochs * step_size,
- keep_checkpoint_max=config.keep_checkpoint_max)
- ckpt_cb = ModelCheckpoint(prefix="glore_resnet50", directory=save_checkpoint_path, config=config_ck)
- save_cb = SaveCallback(model, eval_dataset, save_checkpoint_path)
- cb += [ckpt_cb, save_cb]
-
- # train model
- print("=======Training Begin========")
- model.train(config.epoch_size - config.pretrain_epoch_size, train_dataset, callbacks=cb, dataset_sink_mode=True)
-
- # copy train result from cache to obs
- if args_opt.is_modelarts:
- mox.file.copy_parallel(src_url='/cache/train_output', dst_url=args_opt.train_url)