!2607 [西安交通大学][高校贡献][Mindspore][TernaryBert]-310推理

Merge pull request !2607 from slyang/ternarybert

research/nlp/ternarybert/README.md

+
+# Contents
+
+- [Contents](#contents)
+- [TernaryBERT Description](#ternarybert-description)
+- [Model Architecture](#model-architecture)
+- [Dataset](#dataset)
+- [Environment Requirements](#environment-requirements)
+- [Quick Start](#quick-start)
+- [Script Description](#script-description)
+    - [Script and Sample Code](#script-and-sample-code)
+    - [Script Parameters](#script-parameters)
+        - [Train](#train)
+        - [Eval](#eval)
+    - [Options and Parameters](#options-and-parameters)
+        - [Parameters](#parameters)
+    - [Training Process](#training-process)
+        - [Training](#training)
+    - [Evaluation Process](#evaluation-process)
+        - [Evaluation](#evaluation)
+            - [evaluation on STS-B dataset](#evaluation-on-sts-b-dataset)
+    - [Model Description](#model-description)
+    - [Performance](#performance)
+        - [training Performance](#training-performance)
+        - [Inference Performance](#inference-performance)
+- [Description of Random Situation](#description-of-random-situation)
+- [ModelZoo Homepage](#modelzoo-homepage)
+
+# [TernaryBERT Description](#contents)
+
+[TernaryBERT](https://arxiv.org/abs/2009.12812) ternarizes the weights in a fine-tuned [BERT](https://arxiv.org/abs/1810.04805) or [TinyBERT](https://arxiv.org/abs/1909.10351) model and achieves competitive performances in natural language processing tasks. TernaryBERT outperforms the other BERT quantization methods, and even achieves comparable performance as the full-precision model while being 14.9x smaller
+
+[Paper](https://arxiv.org/abs/2009.12812): Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang and Qun Liu. [TernaryBERT: Distillation-aware Ultra-low Bit BERT](https://arxiv.org/abs/2009.12812). arXiv preprint arXiv:2009.12812.
+
+# [Model Architecture](#contents)
+
+The backbone structure of TernaryBERT is transformer, the transformer contains six encoder modules, one encoder contains one self-attention module and one self-attention module contains one attention module. The pretrained teacher model and student model are provided [here](https://download.mindspore.cn/model_zoo/research/nlp/ternarybert/).
+
+# [Dataset](#contents)
+
+- Download glue dataset for task distillation. Convert dataset files from json format to tfrecord format, please refer to data_transfer.py which in [BERT](https://gitee.com/slyang2021/bert-multi-gpu#convert-sts_b-dataset-to-tf_record) repository.
+
+- Note that when the parameter is passed in, the directory is the upper level directory of sts-b
+
+```text
+
+└─data_dir
+  ├─sts-b
+    ├─eval.tf_record
+    ├─predict.tf_record
+    ├─train.tf_record
+
+```
+
+# [Environment Requirements](#contents)
+
+- Hardware（Ascend or GPU）
+    - Prepare hardware environment with GPU processor or Ascend processor.
+- Framework
+    - [MindSpore](https://gitee.com/mindspore/mindspore)
+- For more information, please check the resources below：
+    - [MindSpore Tutorials](https://www.mindspore.cn/tutorials/en/master/index.html)
+    - [MindSpore Python API](https://www.mindspore.cn/docs/api/en/master/index.html)
+- Software：
+    - sklearn
+
+# [Quick Start](#contents)
+
+After installing MindSpore via the official website, you can start training and evaluation as follows:
+
+```bash
+
+# run training example
+
+bash scripts/run_standalone_train_ascend.sh [TASK_NAME] [DEVICE_TARGET] [TEACHER_MODEL_DIR] [STUDENT_MODEL_DIR] [DATA_DIR]
+
+Before running the shell script, please set the `task_name`, `device_target`, `teacher_model_dir`, `student_model_dir` and `data_dir` in the run_standalone_train_ascend.sh file first.
+
+# run evaluation example
+
+bash scripts/run_standalone_eval_ascend.sh [TASK_NAME] [DEVICE_TARGET] [MODEL_DIR] [DATA_DIR]
+
+Before running the shell script, please set the `task_name`, `device_target`, `model_dir` and `data_dir` in the run_standalone_eval_ascend.sh file first.
+```
+
+# [Script Description](#contents)
+
+## [Script and Sample Code](#contents)
+
+```text
+
+.
+└─ternarybert
+  ├─ascend310_infer
+  ├─README.md
+  ├─scripts
+    ├─run_standalone_train_ascend.sh                  # shell script for training phase
+    ├─run_standalone_eval_ascend.sh                   # shell script for evaluation phase
+    ├─run_infer_310.sh              # shell script for 310infer
+  ├─src
+    ├─__init__.py
+    ├─assessment_method.py          # assessment method for evaluation
+    ├─cell_wrapper.py               # cell for training
+    ├─config.py                     # parameter configuration for training and evaluation phase
+    ├─dataset.py                    # data processing
+    ├─quant.py                      # function for quantization
+    ├─tinybert_model.py             # backbone code of network
+    ├─utils.py                      # util function
+  ├─train.py                        # train net for task distillation
+  ├─eval.py                         # evaluate net after task distillation
+  ├─export.py                       # export scripts
+  ├─preprocess.py                   # 310推理前处理脚本
+  ├─postprocess.py                  # 310推理后处理脚本
+  ├─mindspore_hub_conf.py           # Mindspore Hub接口
+```
+
+## [Script Parameters](#contents)
+
+### Train
+
+```text
+
+usage: train.py    [--h] [--device_target GPU] [--do_eval {true,false}] [--epoch_size EPOCH_SIZE]
+                   [--device_id DEVICE_ID] [--do_shuffle {true,false}] [--enable_data_sink {true,false}] [--save_ckpt_step SAVE_CKPT_STEP]
+                   [--max_ckpt_num MAX_CKPT_NUM] [--data_sink_steps DATA_SINK_STEPS]
+                   [--teacher_model_dir TEACHER_MODEL_DIR] [--student_model_dir STUDENT_MODEL_DIR] [--data_dir DATA_DIR]
+                   [--output_dir OUTPUT_DIR] [--task_name {sts-b,qnli,mnli}] [--dataset_type DATASET_TYPE] [--seed SEED]
+                   [--train_batch_size TRAIN_BATCH_SIZE] [--eval_batch_size EVAL_BATCH_SIZE]
+
+options:
+    --device_target                 Device where the code will be implemented: "Ascend"
+    --do_eval                       Do eval task during training or not: "true" | "false", default is "true"
+    --epoch_size                    Epoch size for train phase: N, default is 5
+    --device_id                     Device id: N, default is 0
+    --do_shuffle                    Enable shuffle for train dataset: "true" | "false", default is "true"
+    --enable_data_sink              Enable data sink: "true" | "false", default is "true"
+    --save_ckpt_step                If do_eval is false, the checkpoint will be saved every save_ckpt_step: N, default is 50
+    --max_ckpt_num                  The number of checkpoints will not be larger than max_ckpt_num: N, default is 50
+    --data_sink_steps               Sink steps for each epoch: N, default is 1
+    --teacher_model_dir             The checkpoint directory of teacher model: PATH, default is ""
+    --student_model_dir             The checkpoint directory of student model: PATH, default is ""
+    --data_dir                      Data directory: PATH, default is ""
+    --output_dir                    The output checkpoint directory: PATH, default is "./"
+    --task_name                     The name of the task to train: "sts-b" | "qnli" | "mnli", default is "sts-b"
+    --dataset_type                  The name of the task to train: "tfrecord" | "mindrecord", default is "tfrecord"
+    --seed                          The random seed: N, default is 1
+    --train_batch_size              Batch size for training: N, default is 16
+    --eval_batch_size               Eval Batch size in callback: N, default is 32
+    --file_name                     The output filename of export, default is "ternarybert"
+    --file_format                   The output format of export, default is "MINDIR"
+    --enable_modelarts              Do modelarts or not. (Default: False)
+    --data_url                      Real input file path
+    --train_url                     Real output file path include .ckpt and .air
+    --modelarts_data_dir            Modelart input path
+    --modelarts_result_dir          Modelart output path
+    --result_dir                    Output path
+```
+
+### Eval
+
+```text
+
+usage: eval.py    [--h] [--device_target GPU] [--device_id DEVICE_ID] [--model_dir MODEL_DIR] [--data_dir DATA_DIR]
+                  [--task_name {sts-b,qnli,mnli}] [--dataset_type DATASET_TYPE] [--batch_size BATCH_SIZE]
+
+options:
+    --device_target                 Device where the code will be implemented: "GPU"
+    --device_id                     Device id: N, default is 0
+    --model_dir                     The checkpoint directory of model: PATH, default is ""
+    --data_dir                      Data directory: PATH, default is ""
+    --task_name                     The name of the task to train: "sts-b" | "qnli" | "mnli", default is "sts-b"
+    --dataset_type                  The name of the task to train: "tfrecord" | "mindrecord", default is "tfrecord"
+    --batch_size                    Batch size for evaluating: N, default is 32
+
+```
+
+## Parameters
+
+`config.py`contains parameters of glue tasks, train, optimizer, eval, teacher BERT model and student BERT model.
+
+```text
+
+Parameters for glue task:
+    num_labels                      the numbers of labels: N.
+    seq_length                      length of input sequence: N
+    task_type                       the type of task: "classification" | "regression"
+    metrics                         the eval metric for task: Accuracy | F1 | Pearsonr | Matthews
+
+Parameters for train:
+    batch_size                      batch size of input dataset: N, default is 16
+    loss_scale_value                initial value of loss scale: N, default is 2^16
+    scale_factor                    factor used to update loss scale: N, default is 2
+    scale_window                    steps for once updatation of loss scale: N, default is 50
+
+Parameters for optimizer:
+    learning_rate                   value of learning rate: Q, default is 5e-5
+    end_learning_rate               value of end learning rate: Q, must be positive, default is 1e-14
+    power                           power: Q, default is 1.0
+    weight_decay                    weight decay: Q, default is 1e-4
+    eps                             term added to the denominator to improve numerical stability: Q, default is 1e-6
+    warmup_ratio                    the ratio of warmup steps to total steps: Q, default is 0.1
+
+Parameters for eval:
+    batch_size                      batch size of input dataset: N, default is 32
+
+Parameters for teacher bert network:
+    seq_length                      length of input sequence: N, default is 128
+    vocab_size                      size of each embedding vector: N, must be consistent with the dataset you use. Default is 30522
+    hidden_size                     size of bert encoder layers: N
+    num_hidden_layers               number of hidden layers: N
+    num_attention_heads             number of attention heads: N, default is 12
+    intermediate_size               size of intermediate layer: N
+    hidden_act                      activation function used: ACTIVATION, default is "gelu"
+    hidden_dropout_prob             dropout probability for BertOutput: Q
+    attention_probs_dropout_prob    dropout probability for BertAttention: Q
+    max_position_embeddings         maximum length of sequences: N, default is 512
+    save_ckpt_step                  number for saving checkponit: N, default is 100
+    max_ckpt_num                    maximum number for saving checkpoint: N, default is 1
+    type_vocab_size                 size of token type vocab: N, default is 2
+    initializer_range               initialization value of Normal: Q, default is 0.02
+    use_relative_positions          use relative positions or not: True | False, default is False
+    dtype                           data type of input: mstype.float16 | mstype.float32, default is mstype.float32
+    compute_type                    compute type in BertTransformer: mstype.float16 | mstype.float32, default is mstype.float32
+
+Parameters for student bert network:
+    seq_length                      length of input sequence: N, default is 128
+    vocab_size                      size of each embedding vector: N, must be consistent with the dataset you use. Default is 30522
+    hidden_size                     size of bert encoder layers: N
+    num_hidden_layers               number of hidden layers: N
+    num_attention_heads             number of attention heads: N, default is 12
+    intermediate_size               size of intermediate layer: N
+    hidden_act                      activation function used: ACTIVATION, default is "gelu"
+    hidden_dropout_prob             dropout probability for BertOutput: Q
+    attention_probs_dropout_prob    dropout probability for BertAttention: Q
+    max_position_embeddings         maximum length of sequences: N, default is 512
+    save_ckpt_step                  number for saving checkponit: N, default is 100
+    max_ckpt_num                    maximum number for saving checkpoint: N, default is 1
+    type_vocab_size                 size of token type vocab: N, default is 2
+    initializer_range               initialization value of Normal: Q, default is 0.02
+    use_relative_positions          use relative positions or not: True | False, default is False
+    dtype                           data type of input: mstype.float16 | mstype.float32, default is mstype.float32
+    compute_type                    compute type in BertTransformer: mstype.float16 | mstype.float32, default is mstype.float32
+    do_quant                        do activation quantilization or not: True | False, default is True
+    embedding_bits                  the quant bits of embedding: N, default is 2
+    weight_bits                     the quant bits of weight: N, default is 2
+    cls_dropout_prob                dropout probability for BertModelCLS: Q
+    activation_init                 initialization value of activation quantilization: Q, default is 2.5
+    is_lgt_fit                      use label ground truth loss or not: True | False, default is False
+
+```
+
+## [Training Process](#contents)
+
+### Training
+
+Before running the command below, please check `teacher_model_dir`, `student_model_dir` and `data_dir` has been set. Please set the path to be the absolute full path, e.g:"/home/xxx/model_dir/".
+
+```text
+
+python
+    python train.py --task_name='sts-b' --device_target="Ascend" --teacher_model_dir='/home/xxx/model_dir/' --student_model_dir='/home/xxx/model_dir/' --data_dir='/home/xxx/data_dir/'
+shell
+    bash scripts/run_standalone_train_ascend.sh [TASK_NAME] [DEVICE_TARGET] [DEVICE_ID] [TEACHER_MODEL_DIR] [STUDENT_MODEL_DIR] [DATA_DIR]
+
+```
+
+The shell command above will run in the background, you can view the results the file log.txt. The python command will run in the console, you can view the results on the interface. After training, you will get some checkpoint files under the script folder by default. The eval metric value will be achieved as follows:
+
+```text
+
+train dataset size: 359
+eval dataset size: 47
+epoch: 1 step: 359, loss is 3.5354042053222656
+epoch time: 2039458.960 ms, per step time: 5680.944 ms
+epoch: 2 step: 359, loss is 1.4011192321777344
+epoch time: 1955723.881 ms, per step time: 5447.699 ms
+epoch: 3 step: 359, loss is 1.2592418193817139
+epoch time: 1955666.337 ms, per step time: 5447.539 ms
+epoch: 4 step: 359, loss is 0.7391554713249207
+epoch time: 1955738.087 ms, per step time: 5447.738 ms
+epoch: 5 step: 359, loss is 0.5966147184371948
+epoch time: 1955702.814 ms, per step time: 5447.640 ms
+===========training success================
+===========Done!!!!!================
+
+```
+
+### Training on ModelArts
+
+Upload weight file and sts-b file (data.zip), the directory structure is as follows
+
+```text
+.
+└─data
+  ├─sts-b
+    ├─eval.tf_record
+    ├─predict.tf_record
+    ├─train.tf_record
+  ├─weight
+    ├─student_model
+        ├─sts-b
+            ├─eval_model.ckpt
+    ├─teacher_model
+        ├─sts-b
+            ├─eval_model.ckpt
+```
+
+Select startup file train.py, dataset data.zip, training parameters is as follows
+
+```text
+enable_modelarts         True
+data_dir                 data
+student_model_dir        data/weight/student_model
+teacher_model_dir        data/weight/teacher_model
+```
+
+## [Evaluation Process](#contents)
+
+### Evaluation
+
+If you want to after running and continue to eval.
+
+#### evaluation on STS-B dataset
+
+```text
+
+python
+    python eval.py --task_name='sts-b' --device_target="Ascend" --model_dir='/home/xxx/model_dir/' --data_dir='/home/xxx/data_dir/'
+shell
+    bash scripts/run_standalone_eval_ascend.sh [TASK_NAME] [DEVICE_TARGET] [DEVICE_ID] [MODEL_DIR] [DATA_DIR]
+
+
+```
+
+The shell command above will run in the background, you can view the results the file log.txt. The python command will run in the console, you can view the results on the interface. The metric value of the test dataset will be as follows:
+
+```text
+
+eval step: 0, Pearsonr: 96.91109003302263
+eval step: 1, Pearsonr: 95.6800637493701
+eval step: 2, Pearsonr: 94.23823082886167
+...
+The best Pearsonr: 87.58388835685437
+
+```
+
+## [Evaluation on Ascend 310](#contents)
+
+### Export MINDIR
+
+```text
+python export.py --task_name [TASK_NAME] --file_name [FILE_NAME] --ckpt_file [CKPT_FILE]
+#example
+python export.py --task_name sts-b --file_name ternarybert --ckpt_file ./output/sts-b/eval_model.ckpt
+```
+
+### Evaluation
+
+Before performing inference, the mindir file must be exported through the export.py script. The following shows an example of using the mindir model to perform inference.
+
+```text
+bash run_infer_310.sh [MINDIR_PATH] [DATASET_PATH] [DATASET_TYPE] [TASK_NAME] [ASSESSMENT_METHOD] [NEED_PREPROCESS] [DEVICE_ID]
+# example
+bash run_infer_310.sh ../ternarybert.mindir ../data/sts-b/eval.tf_record tfrecord sts-b pearsonr y 0
+
+```
+
+## [Model Description](#contents)
+
+## [Performance](#contents)
+
+### training Performance
+
+| Parameters                                                                    | Ascend                       |
+| -------------------------- | ------------------------- |
+| Model Version              | TernaryBERT                           |
+| Resource                   | Ascend 910, ARM CPU 2.60GHz, cores 192, mem 755G, os Euler2.8         |
+| Date              | 2021-6-10      |
+| MindSpore Version          | 1.7.0                     |
+| Dataset                    | STS-B              |
+| batch_size                    | 16              |
+| Metric value                 | 87.5                       |
+
+# [Description of Random Situation](#contents)
+
+In train.py, we set do_shuffle to shuffle the dataset.
+
+In config.py, we set the hidden_dropout_prob, attention_pros_dropout_prob and cls_dropout_prob to dropout some network node.
+
+# [ModelZoo Homepage](#contents)
+
+Please check the official [homepage](https://gitee.com/mindspore/models).

research/nlp/ternarybert/ascend310_infer/CMakeLists.txt

+cmake_minimum_required(VERSION 3.14.1)
+project(Ascend310Infer)
+add_compile_definitions(_GLIBCXX_USE_CXX11_ABI=0)
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O0 -g -std=c++17 -Werror -Wall -fPIE -Wl,--allow-shlib-undefined")
+set(PROJECT_SRC_ROOT ${CMAKE_CURRENT_LIST_DIR}/)
+option(MINDSPORE_PATH "mindspore install path" "")
+include_directories(${MINDSPORE_PATH})
+include_directories(${MINDSPORE_PATH}/include)
+include_directories(${PROJECT_SRC_ROOT})
+find_library(MS_LIB libmindspore.so ${MINDSPORE_PATH}/lib)
+file(GLOB_RECURSE MD_LIB ${MINDSPORE_PATH}/_c_dataengine*)
+
+add_executable(main src/main.cc src/utils.cc)
+target_link_libraries(main ${MS_LIB} ${MD_LIB} gflags)

research/nlp/ternarybert/ascend310_infer/build.sh

+#!/bin/bash
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+if [ -d out ]; then
+    rm -rf out
+fi
+
+mkdir out
+cd out || exit
+
+if [ -f "Makefile" ]; then
+  make clean
+fi
+
+cmake .. \
+    -DMINDSPORE_PATH="`pip show mindspore-ascend | grep Location | awk '{print $2"/mindspore"}' | xargs realpath`"
+make

research/nlp/ternarybert/ascend310_infer/inc/utils.h

+/**
+ * Copyright 2022 Huawei Technologies Co., Ltd
+ * 
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef MINDSPORE_INFERENCE_UTILS_H_
+#define MINDSPORE_INFERENCE_UTILS_H_
+
+#include <sys/stat.h>
+#include <dirent.h>
+#include <vector>
+#include <string>
+#include <memory>
+#include "include/api/types.h"
+
+std::vector<std::string> GetAllFiles(std::string_view dirName);
+DIR *OpenDir(std::string_view dirName);
+std::string RealPath(std::string_view path);
+mindspore::MSTensor ReadFileToTensor(const std::string &file);
+int WriteResult(const std::string& imageFile, const std::vector<mindspore::MSTensor> &outputs);
+#endif

research/nlp/ternarybert/ascend310_infer/src/main.cc

+/**
+ * Copyright 2022 Huawei Technologies Co., Ltd
+ * 
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <sys/time.h>
+#include <gflags/gflags.h>
+#include <dirent.h>
+#include <iostream>
+#include <string>
+#include <algorithm>
+#include <iosfwd>
+#include <vector>
+#include <fstream>
+#include <sstream>
+
+#include "include/api/model.h"
+#include "include/api/context.h"
+#include "include/api/types.h"
+#include "include/api/serialization.h"
+#include "include/dataset/execute.h"
+#include "include/dataset/vision.h"
+#include "inc/utils.h"
+
+using mindspore::Context;
+using mindspore::Serialization;
+using mindspore::Model;
+using mindspore::Status;
+using mindspore::MSTensor;
+using mindspore::dataset::Execute;
+using mindspore::ModelType;
+using mindspore::GraphCell;
+using mindspore::kSuccess;
+
+DEFINE_string(mindir_path, "", "mindir path");
+DEFINE_string(input0_path, "", "input0 path");
+DEFINE_string(input1_path, "", "input1 path");
+DEFINE_string(input2_path, "", "input2 path");
+DEFINE_int32(device_id, 0, "device id");
+
+int main(int argc, char **argv) {
+  gflags::ParseCommandLineFlags(&argc, &argv, true);
+  if (RealPath(FLAGS_mindir_path).empty()) {
+    std::cout << "Invalid mindir" << std::endl;
+    return 1;
+  }
+
+  std::cout << "begin infer!!!!" << std::endl;
+
+  auto context = std::make_shared<Context>();
+  auto ascend310 = std::make_shared<mindspore::Ascend310DeviceInfo>();
+  ascend310->SetDeviceID(FLAGS_device_id);
+  ascend310->SetPrecisionMode("allow_fp32_to_fp16");
+  ascend310->SetOpSelectImplMode("high_precision");
+  context->MutableDeviceInfo().push_back(ascend310);
+  mindspore::Graph graph;
+  Serialization::Load(FLAGS_mindir_path, ModelType::kMindIR, &graph);
+  std::cout << "load model!!!!" << std::endl;
+  Model model;
+  Status ret = model.Build(GraphCell(graph), context);
+  if (ret != kSuccess) {
+    std::cout << "ERROR: Build failed." << std::endl;
+    return 1;
+  }
+
+  std::vector<MSTensor> model_inputs = model.GetInputs();
+  if (model_inputs.empty()) {
+    std::cout << "Invalid model, inputs is empty." << std::endl;
+    return 1;
+  }
+
+  auto input0_files = GetAllFiles(FLAGS_input0_path);
+  auto input1_files = GetAllFiles(FLAGS_input1_path);
+  auto input2_files = GetAllFiles(FLAGS_input2_path);
+
+  if (input0_files.empty() || input1_files.empty() || input2_files.empty()) {
+    std::cout << "ERROR: input data empty." << std::endl;
+    return 1;
+  }
+
+  std::map<double, double> costTime_map;
+  size_t size = input0_files.size();
+
+  for (size_t i = 0; i < size; ++i) {
+    struct timeval start = {0};
+    struct timeval end = {0};
+    double startTimeMs;
+    double endTimeMs;
+    std::vector<MSTensor> inputs;
+    std::vector<MSTensor> outputs;
+    std::cout << "Start predict input files:" << input0_files[i] << std::endl;
+
+    auto input0 = ReadFileToTensor(input0_files[i]);
+    auto input1 = ReadFileToTensor(input1_files[i]);
+    auto input2 = ReadFileToTensor(input2_files[i]);
+    inputs.emplace_back(model_inputs[0].Name(), model_inputs[0].DataType(), model_inputs[0].Shape(),
+                        input0.Data().get(), input0.DataSize());
+    inputs.emplace_back(model_inputs[1].Name(), model_inputs[1].DataType(), model_inputs[1].Shape(),
+                        input1.Data().get(), input1.DataSize());
+    inputs.emplace_back(model_inputs[2].Name(), model_inputs[2].DataType(), model_inputs[2].Shape(),
+                        input2.Data().get(), input2.DataSize());
+
+    gettimeofday(&start, nullptr);
+    ret = model.Predict(inputs, &outputs);
+    gettimeofday(&end, nullptr);
+    if (ret != kSuccess) {
+      std::cout << "Predict " << input0_files[i] << " failed." << std::endl;
+      return 1;
+    }
+    startTimeMs = (1.0 * start.tv_sec * 1000000 + start.tv_usec) / 1000;
+    endTimeMs = (1.0 * end.tv_sec * 1000000 + end.tv_usec) / 1000;
+    costTime_map.insert(std::pair<double, double>(startTimeMs, endTimeMs));
+    WriteResult(input0_files[i], outputs);
+  }
+  double average = 0.0;
+  int inferCount = 0;
+
+  for (auto iter = costTime_map.begin(); iter != costTime_map.end(); iter++) {
+    double diff = 0.0;
+    diff = iter->second - iter->first;
+    average += diff;
+    inferCount++;
+  }
+  average = average / inferCount;
+  std::stringstream timeCost;
+  timeCost << "NN inference cost average time: "<< average << " ms of infer_count " << inferCount << std::endl;
+  std::cout << "NN inference cost average time: "<< average << "ms of infer_count " << inferCount << std::endl;
+  std::string fileName = "./time_Result" + std::string("/test_perform_static.txt");
+  std::ofstream fileStream(fileName.c_str(), std::ios::trunc);
+  fileStream << timeCost.str();
+  fileStream.close();
+  costTime_map.clear();
+  return 0;
+}

research/nlp/ternarybert/ascend310_infer/src/utils.cc

+/**
+ * Copyright 2022 Huawei Technologies Co., Ltd
+ * 
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <fstream>
+#include <algorithm>
+#include <iostream>
+#include "inc/utils.h"
+
+using mindspore::MSTensor;
+using mindspore::DataType;
+
+std::vector<std::string> GetAllFiles(std::string_view dirName) {
+  struct dirent *filename;
+  DIR *dir = OpenDir(dirName);
+  if (dir == nullptr) {
+    return {};
+  }
+  std::vector<std::string> res;
+  while ((filename = readdir(dir)) != nullptr) {
+    std::string dName = std::string(filename->d_name);
+    if (dName == "." || dName == ".." || filename->d_type != DT_REG) {
+      continue;
+    }
+    res.emplace_back(std::string(dirName) + "/" + filename->d_name);
+  }
+  std::sort(res.begin(), res.end());
+  for (auto &f : res) {
+    std::cout << "image file: " << f << std::endl;
+  }
+  return res;
+}
+
+int WriteResult(const std::string& imageFile, const std::vector<MSTensor> &outputs) {
+  std::cout << "===============>>>  Begin WriteResult  <<<===============" << std::endl;
+  std::cout << "output size is " << outputs.size() << std::endl;
+  std::string homePath = "./result_Files";
+  for (size_t i = 0; i < outputs.size(); ++i) {
+    size_t outputSize;
+    std::shared_ptr<const void> netOutput;
+    netOutput = outputs[i].Data();
+    outputSize = outputs[i].DataSize();
+    int pos = imageFile.rfind('/');
+    std::string fileName(imageFile, pos + 1);
+    fileName.replace(fileName.find('.'), fileName.size() - fileName.find('.'), '_' + std::to_string(i) + ".bin");
+    std::string outFileName = homePath + "/" + fileName;
+    FILE * outputFile = fopen(outFileName.c_str(), "wb");
+    fwrite(netOutput.get(), outputSize, sizeof(char), outputFile);
+
+    std::cout << i << " " <<outputs[i].DataSize() << " " << outFileName << std::endl;
+
+    fclose(outputFile);
+    outputFile = nullptr;
+  }
+
+  std::cout << "===============>>>  END  <<<===============" << std::endl;
+  return 0;
+}
+
+mindspore::MSTensor ReadFileToTensor(const std::string &file) {
+  if (file.empty()) {
+    std::cout << "Pointer file is nullptr" << std::endl;
+    return mindspore::MSTensor();
+  }
+
+  std::ifstream ifs(file);
+  if (!ifs.good()) {
+    std::cout << "File: " << file << " is not exist" << std::endl;
+    return mindspore::MSTensor();
+  }
+
+  if (!ifs.is_open()) {
+    std::cout << "File: " << file << "open failed" << std::endl;
+    return mindspore::MSTensor();
+  }
+
+  ifs.seekg(0, std::ios::end);
+  size_t size = ifs.tellg();
+  mindspore::MSTensor buffer(file, mindspore::DataType::kNumberTypeUInt8, {static_cast<int64_t>(size)}, nullptr, size);
+
+  ifs.seekg(0, std::ios::beg);
+  ifs.read(reinterpret_cast<char *>(buffer.MutableData()), size);
+  ifs.close();
+
+  return buffer;
+}
+
+
+DIR *OpenDir(std::string_view dirName) {
+  if (dirName.empty()) {
+    std::cout << " dirName is null ! " << std::endl;
+    return nullptr;
+  }
+  std::string realPath = RealPath(dirName);
+  struct stat s;
+  lstat(realPath.c_str(), &s);
+  if (!S_ISDIR(s.st_mode)) {
+    std::cout << "dirName is not a valid directory !" << std::endl;
+    return nullptr;
+  }
+  DIR *dir;
+  dir = opendir(realPath.c_str());
+  if (dir == nullptr) {
+    std::cout << "Can not open dir " << dirName << std::endl;
+    return nullptr;
+  }
+  std::cout << "Successfully opened the dir " << dirName << std::endl;
+  return dir;
+}
+
+std::string RealPath(std::string_view path) {
+  char realPathMem[PATH_MAX] = {0};
+  char *realPathRet = nullptr;
+  realPathRet = realpath(path.data(), realPathMem);
+
+  if (realPathRet == nullptr) {
+    std::cout << "File: " << path << " is not exist.";
+    return "";
+  }
+
+  std::string realPath(realPathMem);
+  std::cout << path << " realpath is: " << realPath << std::endl;
+  return realPath;
+}

research/nlp/ternarybert/eval.py

+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+"""eval standalone script"""
+
+import os
+import re
+import argparse
+
+from mindspore import context
+from mindspore.train.serialization import load_checkpoint, load_param_into_net
+from src.dataset import create_dataset
+from src.config import eval_cfg, student_net_cfg, task_cfg
+from src.tinybert_model import BertModelCLS
+
+
+def parse_args():
+    """
+    parse args
+    """
+    parser = argparse.ArgumentParser(description='ternarybert evaluation')
+    parser.add_argument('--device_target', type=str, default='Ascend', choices=['Ascend', 'GPU'],
+                        help='Device where the code will be implemented. (Default: GPU)')
+    parser.add_argument('--device_id', type=int, default=0, help='Device id. (Default: 0)')
+    parser.add_argument('--model_dir', type=str, default='', help='The checkpoint directory of model.')
+    parser.add_argument('--data_dir', type=str, default='', help='Data directory.')
+    parser.add_argument('--task_name', type=str, default='sts-b', choices=['sts-b', 'qnli', 'mnli'],
+                        help='The name of the task to train. (Default: sts-b)')
+    parser.add_argument('--dataset_type', type=str, default='tfrecord', choices=['tfrecord', 'mindrecord'],
+                        help='The name of the task to train. (Default: tfrecord)')
+    parser.add_argument('--batch_size', type=int, default=32, help='Batch size for evaluating')
+    parser.add_argument('--data_name', type=str, default='eval.tf_record', help='')
+    return parser.parse_args()
+
+
+def get_ckpt(ckpt_file):
+    lists = os.listdir(ckpt_file)
+    lists.sort(key=lambda fn: os.path.getmtime(ckpt_file + '/' + fn))
+    return os.path.join(ckpt_file, lists[-1])
+
+
+def do_eval_standalone(args_opt):
+    """
+    do eval standalone
+    """
+    ckpt_file = os.path.join(args_opt.model_dir, args_opt.task_name)
+    ckpt_file = get_ckpt(ckpt_file)
+    print('ckpt file:', ckpt_file)
+    task = task_cfg[args_opt.task_name]
+    student_net_cfg.seq_length = task.seq_length
+    eval_cfg.batch_size = args_opt.batch_size
+    eval_data_dir = os.path.join(args_opt.data_dir, args_opt.task_name, args_opt.data_name)
+
+    context.set_context(mode=context.GRAPH_MODE, device_target=args_opt.device_target, device_id=args.device_id)
+
+    eval_dataset = create_dataset(batch_size=eval_cfg.batch_size,
+                                  device_num=1,
+                                  rank=0,
+                                  do_shuffle=False,
+                                  data_dir=eval_data_dir,
+                                  data_type=args_opt.dataset_type,
+                                  seq_length=task.seq_length,
+                                  task_type=task.task_type,
+                                  drop_remainder=False)
+    print('eval dataset size:', eval_dataset.get_dataset_size())
+    print('eval dataset batch size:', eval_dataset.get_batch_size())
+
+    eval_model = BertModelCLS(student_net_cfg, False, task.num_labels, 0.0, phase_type='student')
+    param_dict = load_checkpoint(ckpt_file)
+    new_param_dict = {}
+    for key, value in param_dict.items():
+        new_key = re.sub('tinybert_', 'bert_', key)
+        new_key = re.sub('^bert.', '', new_key)
+        new_param_dict[new_key] = value
+    load_param_into_net(eval_model, new_param_dict)
+    eval_model.set_train(False)
+
+    columns_list = ["input_ids", "input_mask", "segment_ids", "label_ids"]
+    callback = task.metrics()
+    for step, data in enumerate(eval_dataset.create_dict_iterator()):
+        input_data = []
+        for i in columns_list:
+            input_data.append(data[i])
+        input_ids, input_mask, token_type_id, label_ids = input_data
+        _, _, logits, _ = eval_model(input_ids, token_type_id, input_mask)
+        callback.update(logits, label_ids)
+        print('eval step: {}, {}: {}'.format(step, callback.name, callback.get_metrics()))
+    metrics = callback.get_metrics()
+    print('The best {}: {}'.format(callback.name, metrics))
+
+
+if __name__ == '__main__':
+    args = parse_args()
+    do_eval_standalone(args)

research/nlp/ternarybert/export.py

+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ===========================================================================
+
+"""export checkpoint file into model"""
+
+import argparse
+import re
+import numpy as np
+
+from mindspore import Tensor, context
+from mindspore.train.serialization import load_checkpoint, load_param_into_net, export
+
+from src.config import student_net_cfg, task_cfg, cfg_cfg
+from src.tinybert_model import BertModelCLS
+
+parser = argparse.ArgumentParser(description="TernaryBert export model")
+parser.add_argument("--device_target", type=str, default="Ascend", choices=["Ascend", "GPU"],
+                    help="device where the code will be implemented. (Default: Ascend)")
+parser.add_argument("--task_name", type=str, default="sts-b", choices=["sts-b", "QNLI", "SST-2"],
+                    help="The name of the task to eval.")
+parser.add_argument("--file_name", type=str, default="ternarybert", help="The name of the output file.")
+parser.add_argument("--file_format", type=str, default="MINDIR", choices=["AIR", "MINDIR"],
+                    help="output model type")
+parser.add_argument("--ckpt_file", type=str, default="", help="pretrained checkpoint file")
+args = parser.parse_args()
+
+context.set_context(mode=context.GRAPH_MODE, device_target=args.device_target)
+
+DEFAULT_NUM_LABELS = cfg_cfg.DEFAULT_NUM_LABELS
+DEFAULT_SEQ_LENGTH = cfg_cfg.DEFAULT_SEQ_LENGTH
+DEFAULT_BS = cfg_cfg.DEFAULT_BS
+
+
+class Task:
+    """
+    Encapsulation class of get the task parameter.
+    """
+
+    def __init__(self, task_name):
+        self.task_name = task_name
+
+    @property
+    def num_labels(self):
+        if self.task_name in task_cfg and "num_labels" in task_cfg[self.task_name]:
+            return task_cfg[self.task_name]["num_labels"]
+        return DEFAULT_NUM_LABELS
+
+    @property
+    def seq_length(self):
+        if self.task_name in task_cfg and "seq_length" in task_cfg[self.task_name]:
+            return task_cfg[self.task_name]["seq_length"]
+        return DEFAULT_SEQ_LENGTH
+
+
+if __name__ == "__main__":
+    task = Task(args.task_name)
+    student_net_cfg.seq_length = task.seq_length
+    student_net_cfg.batch_size = DEFAULT_BS
+    student_net_cfg.do_quant = False
+
+    eval_model = BertModelCLS(student_net_cfg, False, task.num_labels, 0.0, phase_type='student')
+    param_dict = load_checkpoint(args.ckpt_file)
+    new_param_dict = {}
+    for key, value in param_dict.items():
+        new_key = re.sub('tinybert_', 'bert_', key)
+        new_key = re.sub('^bert.', '', new_key)
+        new_param_dict[new_key] = value
+    load_param_into_net(eval_model, new_param_dict)
+    eval_model.set_train(False)
+    input_ids = Tensor(np.zeros((student_net_cfg.batch_size, task.seq_length), np.int32))
+    token_type_id = Tensor(np.zeros((student_net_cfg.batch_size, task.seq_length), np.int32))
+    input_mask = Tensor(np.zeros((student_net_cfg.batch_size, task.seq_length), np.int32))
+
+    input_data = [input_ids, token_type_id, input_mask]
+    export(eval_model, *input_data, file_name=args.file_name, file_format=args.file_format)

research/nlp/ternarybert/mindspore_hub_conf.py

+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+"""Bert hub interface for bert base"""
+
+from src.tinybert_model import BertModel
+from src.tinybert_model import BertConfig
+import mindspore.common.dtype as mstype
+
+tinybert_student_net_cfg = BertConfig(
+    seq_length=128,
+    vocab_size=30522,
+    hidden_size=768,
+    num_hidden_layers=6,
+    num_attention_heads=12,
+    intermediate_size=3072,
+    hidden_act="gelu",
+    hidden_dropout_prob=0.1,
+    attention_probs_dropout_prob=0.1,
+    max_position_embeddings=512,
+    type_vocab_size=2,
+    initializer_range=0.02,
+    use_relative_positions=False,
+    dtype=mstype.float32,
+    compute_type=mstype.float32,
+    do_quant=True,
+    embedding_bits=2,
+    weight_bits=2,
+    weight_clip_value=3.0,
+    cls_dropout_prob=0.1,
+    activation_init=2.5,
+    is_lgt_fit=False
+)
+
+
+def create_network(name, *args, **kwargs):
+    """
+    Create tinybert network.
+    """
+    if name == "ternarybert":
+        if "seq_length" in kwargs:
+            tinybert_student_net_cfg.seq_length = kwargs["seq_length"]
+        is_training = kwargs.get("is_training", False)
+        return BertModel(tinybert_student_net_cfg, is_training, *args)
+    raise NotImplementedError(f"{name} is not implemented in the repo")

research/nlp/ternarybert/postprocess.py

+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+"""postprocess"""
+
+import os
+import argparse
+import numpy as np
+from mindspore import Tensor
+from src.assessment_method import Accuracy, F1, Pearsonr, Matthews
+from src.config import task_cfg, cfg_cfg
+
+
+parser = argparse.ArgumentParser(description='postprocess')
+parser.add_argument("--task_name", type=str, default="sts-b",
+                    choices=['sts-b', "SST-2", "QNLI", "MNLI", "TNEWS", "CLUENER"],
+                    help="The name of the task to train.")
+parser.add_argument("--assessment_method", type=str, default="pearsonr",
+                    choices=["accuracy", "f1", 'pearsonr', 'matthews'],
+                    help="assessment_method include: [accuracy, bf1, mf1], default is accuracy")
+parser.add_argument("--result_path", type=str, default="", help="result path")
+parser.add_argument("--label_path", type=str, default="", help="label path")
+args_opt = parser.parse_args()
+
+DEFAULT_NUM_LABELS = cfg_cfg.DEFAULT_NUM_LABELS
+DEFAULT_SEQ_LENGTH = cfg_cfg.DEFAULT_SEQ_LENGTH
+DEFAULT_BS = cfg_cfg.DEFAULT_BS
+
+
+class Task:
+    """
+    Encapsulation class of get the task parameter.
+    """
+
+    def __init__(self, task_name):
+        self.task_name = task_name
+
+    @property
+    def num_labels(self):
+        if self.task_name in task_cfg and "num_labels" in task_cfg[self.task_name]:
+            return task_cfg[self.task_name]["num_labels"]
+        return DEFAULT_NUM_LABELS
+
+    @property
+    def seq_length(self):
+        if self.task_name in task_cfg and "seq_length" in task_cfg[self.task_name]:
+            return task_cfg[self.task_name]["seq_length"]
+        return DEFAULT_SEQ_LENGTH
+
+
+task = Task(args_opt.task_name)
+
+
+def eval_result_print(assessment_method="pearsonr", callback=None):
+    """print eval result"""
+    if assessment_method == "accuracy":
+        print("accuracy is {}".format(callback.get_metrics()))
+    elif assessment_method == "f1":
+        print("F1 {:.6f} ".format(callback.get_metrics()))
+    elif assessment_method == "matthews":
+        print("matthews {:.6f} ".format(callback.get_metrics()))
+    elif assessment_method == 'pearsonr':
+        print("pearson {:.6f} ".format(callback.get_metrics()))
+    else:
+        raise ValueError("Assessment method not supported, support: [accuracy, f1]")
+
+def get_acc():
+    """
+    calculate accuracy
+    """
+    if args_opt.assessment_method == "accuracy":
+        callback = Accuracy()
+    elif args_opt.assessment_method == "f1":
+        callback = F1()
+    elif args_opt.assessment_method == "matthews":
+        callback = Matthews()
+    elif args_opt.assessment_method == "pearsonr":
+        callback = Pearsonr()
+    else:
+        raise ValueError("Assessment method not supported, support: [accuracy, f1, matthews, pearsonr]")
+    labels = np.load(args_opt.label_path)
+    file_num = len(os.listdir(args_opt.result_path))
+    for i in range(int(file_num/15)):
+        f_name = "tinybert_bs" + str(DEFAULT_BS) + "_" + str(i) + "_13.bin"
+        logits = np.fromfile(os.path.join(args_opt.result_path, f_name), np.float32)
+        label_ids = labels[i]
+        logits = logits.reshape(DEFAULT_BS, task.num_labels)
+        callback.update(Tensor(logits), Tensor(label_ids))
+    print("==============================================================")
+    eval_result_print(args_opt.assessment_method, callback)
+    print("==============================================================")
+
+
+if __name__ == '__main__':
+    get_acc()

research/nlp/ternarybert/preprocess.py

+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+"""preprocess"""
+
+import os
+import argparse
+import numpy as np
+from src.dataset import create_dataset, DataType
+
+
+parser = argparse.ArgumentParser(description='preprocess')
+parser.add_argument("--eval_data_dir", type=str, default="", help="Data path, it is better to use absolute path")
+parser.add_argument("--dataset_type", type=str, default="tfrecord",
+                    help="dataset type tfrecord/mindrecord, default is tfrecord")
+parser.add_argument("--result_path", type=str, default="", help="result path")
+args_opt = parser.parse_args()
+
+BATCH_SIZE = 32
+
+if args_opt.dataset_type == "tfrecord":
+    dataset_type = DataType.TFRECORD
+elif args_opt.dataset_type == "mindrecord":
+    dataset_type = DataType.MINDRECORD
+else:
+    raise Exception("dataset format is not supported yet")
+
+
+def get_bin():
+    """
+    generate bin files.
+    """
+    input_ids_path = os.path.join(args_opt.result_path, "00_input_ids")
+    token_type_id_path = os.path.join(args_opt.result_path, "01_token_type_id")
+    input_mask_path = os.path.join(args_opt.result_path, "02_input_mask")
+    label_ids_path = os.path.join(args_opt.result_path, "label_ids.npy")
+
+    if not os.path.exists(input_ids_path):
+        os.makedirs(input_ids_path)
+    if not os.path.exists(token_type_id_path):
+        os.makedirs(token_type_id_path)
+    if not os.path.exists(input_mask_path):
+        os.makedirs(input_mask_path)
+
+    eval_dataset = create_dataset(batch_size=BATCH_SIZE,
+                                  device_num=1, rank=0, do_shuffle=False,
+                                  data_dir=args_opt.eval_data_dir,
+                                  data_type=dataset_type)
+    columns_list = ["input_ids", "input_mask", "segment_ids", "label_ids"]
+    label_list = []
+    for j, data in enumerate(eval_dataset.create_dict_iterator(output_numpy=True, num_epochs=1)):
+        file_name = "tinybert_bs" + str(BATCH_SIZE) + "_" + str(j) + ".bin"
+        input_data = []
+        for i in columns_list:
+            input_data.append(data[i])
+        input_ids, input_mask, token_type_id, label_ids = input_data
+        input_ids.tofile(os.path.join(input_ids_path, file_name))
+        input_mask.tofile(os.path.join(input_mask_path, file_name))
+        token_type_id.tofile(os.path.join(token_type_id_path, file_name))
+        label_list.append(label_ids)
+    np.save(label_ids_path, label_list)
+    print("=" * 20, 'export files finished', "=" * 20)
+
+
+if __name__ == '__main__':
+    get_bin()

research/nlp/ternarybert/requirements.txt

+numpy

research/nlp/ternarybert/scripts/run_infer_310.sh

+#!/bin/bash
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+if [[ $# -lt 6 || $# -gt 7 ]]; then
+    echo "Usage: bash run_infer_310.sh [MINDIR_PATH] [DATASET_PATH] [SCHEMA_DIR] [DATASET_TYPE] [TASK_NAME] [ASSESSMENT_METHOD] [NEED_PREPROCESS] [DEVICE_ID]
+    NEED_PREPROCESS means weather need preprocess or not, it's value is 'y' or 'n'.
+    DEVICE_ID is optional, it can be set by environment variable device_id, otherwise the value is zero"
+exit 1
+fi
+
+get_real_path(){
+    if [ "${1:0:1}" == "/" ]; then
+        echo "$1"
+    else
+        echo "$(realpath -m $PWD/$1)"
+    fi
+}
+
+model=$(get_real_path $1)
+dataset_path=$(get_real_path $2)
+dataset_type=$3
+task_name=$4
+assessment_method=$5
+
+
+if [ "$6" == "y" ] || [ "$6" == "n" ];then
+    need_preprocess=$6
+else
+  echo "weather need preprocess or not, it's value must be in [y, n]"
+  exit 1
+fi
+
+device_id=0
+if [ $# == 7 ]; then
+    device_id=$7
+fi
+
+echo "mindir name: "$model
+echo "dataset path: "$dataset_path
+echo "dataset_type: "$dataset_type
+echo "task_name: "$task_name
+echo "assessment_method: "$assessment_method
+echo "need preprocess: "$need_preprocess
+echo "device id: "$device_id
+
+export ASCEND_HOME=/usr/local/Ascend/
+if [ -d ${ASCEND_HOME}/ascend-toolkit ]; then
+    export PATH=$ASCEND_HOME/fwkacllib/bin:$ASCEND_HOME/fwkacllib/ccec_compiler/bin:$ASCEND_HOME/ascend-toolkit/latest/fwkacllib/ccec_compiler/bin:$ASCEND_HOME/ascend-toolkit/latest/atc/bin:$PATH
+    export LD_LIBRARY_PATH=$ASCEND_HOME/fwkacllib/lib64:/usr/local/lib:$ASCEND_HOME/ascend-toolkit/latest/atc/lib64:$ASCEND_HOME/ascend-toolkit/latest/fwkacllib/lib64:$ASCEND_HOME/driver/lib64:$ASCEND_HOME/add-ons:$LD_LIBRARY_PATH
+    export TBE_IMPL_PATH=$ASCEND_HOME/ascend-toolkit/latest/opp/op_impl/built-in/ai_core/tbe
+    export PYTHONPATH=$ASCEND_HOME/fwkacllib/python/site-packages:${TBE_IMPL_PATH}:$ASCEND_HOME/ascend-toolkit/latest/fwkacllib/python/site-packages:$PYTHONPATH
+    export ASCEND_OPP_PATH=$ASCEND_HOME/ascend-toolkit/latest/opp
+else
+    export ASCEND_HOME=/usr/local/Ascend/latest/
+    export PATH=$ASCEND_HOME/fwkacllib/bin:$ASCEND_HOME/fwkacllib/ccec_compiler/bin:$ASCEND_HOME/atc/ccec_compiler/bin:$ASCEND_HOME/atc/bin:$PATH
+    export LD_LIBRARY_PATH=$ASCEND_HOME/fwkacllib/lib64:/usr/local/lib:$ASCEND_HOME/atc/lib64:$ASCEND_HOME/acllib/lib64:$ASCEND_HOME/driver/lib64:$ASCEND_HOME/add-ons:$LD_LIBRARY_PATH
+    export PYTHONPATH=$ASCEND_HOME/fwkacllib/python/site-packages:$ASCEND_HOME/atc/python/site-packages:$PYTHONPATH
+    export ASCEND_OPP_PATH=$ASCEND_HOME/opp
+fi
+
+function preprocess_data()
+{
+    if [ -d preprocess_Result ]; then
+        rm -rf ./preprocess_Result
+    fi
+    mkdir preprocess_Result
+    python ../preprocess.py --eval_data_dir=$dataset_path  --dataset_type=$dataset_type --result_path=./preprocess_Result/
+}
+
+function compile_app()
+{
+    cd ../ascend310_infer || exit
+    bash build.sh &> build.log
+}
+
+function infer()
+{
+    cd - || exit
+    if [ -d result_Files ]; then
+        rm -rf ./result_Files
+    fi
+    if [ -d time_Result ]; then
+        rm -rf ./time_Result
+    fi
+    mkdir result_Files
+    mkdir time_Result
+
+    ../ascend310_infer/out/main --mindir_path=$model --input0_path=./preprocess_Result/00_input_ids --input1_path=./preprocess_Result/01_token_type_id --input2_path=./preprocess_Result/02_input_mask --device_id=$device_id &> infer.log
+
+}
+
+function cal_acc()
+{
+    python ../postprocess.py --task_name=$task_name --assessment_method=$assessment_method --result_path=./result_Files --label_path=./preprocess_Result/label_ids.npy &> acc.log
+}
+
+if [ $need_preprocess == "y" ]; then
+    preprocess_data
+    if [ $? -ne 0 ]; then
+        echo "preprocess dataset failed"
+        exit 1
+    fi
+fi
+compile_app
+if [ $? -ne 0 ]; then
+    echo "compile app code failed"
+    exit 1
+fi
+infer
+if [ $? -ne 0 ]; then
+    echo " execute inference failed"
+    exit 1
+fi
+cal_acc
+if [ $? -ne 0 ]; then
+    echo "calculate accuracy failed"
+    exit 1
+fi

research/nlp/ternarybert/scripts/run_standalone_eval_ascend.sh

+#!/bin/bash
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+if [ $# != 4 ]
+then
+    echo "============================================================================================================"
+    echo "Please run the script as: "
+    echo "bash scripts/run_eval.sh [TASK_NAME] [DEVICE_TARGET] [MODEL_DIR] [DATA_DIR]"
+    echo "============================================================================================================"
+exit 1
+fi
+
+echo "===============================================start evaling================================================"
+
+task_name=$1
+device_target=$2
+device_id=$3
+model_dir=$4
+data_dir=$5
+device_id=$6
+
+mkdir -p ms_log
+PROJECT_DIR=$(cd "$(dirname "$0")" || exit; pwd)
+CUR_DIR=`pwd`
+export GLOG_log_dir=${CUR_DIR}/ms_log
+export GLOG_logtostderr=0
+python ${PROJECT_DIR}/../eval.py \
+    --task_name=$task_name \
+    --device_target=$device_target \
+    --device_id=$device_id \
+    --model_dir=$model_dir \
+    --data_dir=$data_dir > eval_log.txt 2>&1 &

research/nlp/ternarybert/scripts/run_standalone_train_ascend.sh

+#!/bin/bash
+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+echo "===============================================start training==============================================="
+
+task_name=$1
+device_target=$2
+device_id=$3
+teacher_model_dir=$4
+student_model_dir=$5
+data_dir=$6
+
+mkdir -p ms_log
+PROJECT_DIR=$(cd "$(dirname "$0")" || exit; pwd)
+CUR_DIR=`pwd`
+export GLOG_log_dir=${CUR_DIR}/ms_log
+export GLOG_logtostderr=0
+python ${PROJECT_DIR}/../train.py \
+    --task_name=$task_name \
+    --device_target=$device_target \
+    --device_id=$device_id \
+    --teacher_model_dir=$teacher_model_dir \
+    --student_model_dir=$student_model_dir \
+    --output_dir=$task_name \
+    --data_dir=$data_dir> train_log.txt 2>&1 &
+    

research/nlp/ternarybert/src/assessment_method.py

+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+"""assessment methods"""
+
+import numpy as np
+
+
+class Accuracy:
+    """Accuracy"""
+    def __init__(self):
+        self.acc_num = 0
+        self.total_num = 0
+        self.name = 'Accuracy'
+
+    def update(self, logits, labels):
+        labels = labels.asnumpy()
+        labels = np.reshape(labels, -1)
+        logits = logits.asnumpy()
+        logit_id = np.argmax(logits, axis=-1)
+        self.acc_num += np.sum(labels == logit_id)
+        self.total_num += len(labels)
+
+    def get_metrics(self):
+        return self.acc_num / self.total_num * 100.0
+
+
+class F1:
+    """F1"""
+    def __init__(self):
+        self.logits_array = np.array([])
+        self.labels_array = np.array([])
+        self.name = 'F1'
+
+    def update(self, logits, labels):
+        labels = labels.asnumpy()
+        labels = np.reshape(labels, -1)
+        logits = logits.asnumpy()
+        logits = np.argmax(logits, axis=1)
+        self.labels_array = np.concatenate([self.labels_array, labels]).astype(np.bool)
+        self.logits_array = np.concatenate([self.logits_array, logits]).astype(np.bool)
+
+    def get_metrics(self):
+        if len(self.labels_array) < 2:
+            return 0.0
+        tp = np.sum(self.labels_array & self.logits_array)
+        fp = np.sum(self.labels_array & (~self.logits_array))
+        fn = np.sum((~self.labels_array) & self.logits_array)
+        p = tp / (tp + fp)
+        r = tp / (tp + fn)
+        return 2.0 * p * r / (p + r) * 100.0
+
+
+class Pearsonr:
+    """Pearsonr"""
+    def __init__(self):
+        self.logits_array = np.array([])
+        self.labels_array = np.array([])
+        self.name = 'Pearsonr'
+
+    def update(self, logits, labels):
+        labels = labels.asnumpy()
+        labels = np.reshape(labels, -1)
+        logits = logits.asnumpy()
+        logits = np.reshape(logits, -1)
+        self.labels_array = np.concatenate([self.labels_array, labels])
+        self.logits_array = np.concatenate([self.logits_array, logits])
+
+    def get_metrics(self):
+        if len(self.labels_array) < 2:
+            return 0.0
+        x_mean = self.logits_array.mean()
+        y_mean = self.labels_array.mean()
+        xm = self.logits_array - x_mean
+        ym = self.labels_array - y_mean
+        norm_xm = np.linalg.norm(xm)
+        norm_ym = np.linalg.norm(ym)
+        return np.dot(xm / norm_xm, ym / norm_ym) * 100.0
+
+
+class Matthews:
+    """Matthews"""
+    def __init__(self):
+        self.logits_array = np.array([])
+        self.labels_array = np.array([])
+        self.name = 'Matthews'
+
+    def update(self, logits, labels):
+        labels = labels.asnumpy()
+        labels = np.reshape(labels, -1)
+        logits = logits.asnumpy()
+        logits = np.argmax(logits, axis=1)
+        self.labels_array = np.concatenate([self.labels_array, labels]).astype(np.bool)
+        self.logits_array = np.concatenate([self.logits_array, logits]).astype(np.bool)
+
+    def get_metrics(self):
+        if len(self.labels_array) < 2:
+            return 0.0
+        tp = np.sum(self.labels_array & self.logits_array)
+        fp = np.sum(self.labels_array & (~self.logits_array))
+        fn = np.sum((~self.labels_array) & self.logits_array)
+        tn = np.sum((~self.labels_array) & (~self.logits_array))
+        return (tp * tn - fp * fn) / np.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn)) * 100.0

research/nlp/ternarybert/src/config.py

+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+"""config script"""
+
+from easydict import EasyDict as edict
+import mindspore.common.dtype as mstype
+from .tinybert_model import BertConfig
+from .assessment_method import Accuracy, F1, Pearsonr, Matthews
+
+
+gradient_cfg = edict({
+    'clip_type': 1,
+    'clip_value': 1.0
+})
+
+task_cfg = edict({
+    "sst-2": edict({"num_labels": 2, "seq_length": 64, "task_type": "classification", "metrics": Accuracy}),
+    "qnli": edict({"num_labels": 2, "seq_length": 128, "task_type": "classification", "metrics": Accuracy}),
+    "mnli": edict({"num_labels": 3, "seq_length": 128, "task_type": "classification", "metrics": Accuracy}),
+    "cola": edict({"num_labels": 2, "seq_length": 64, "task_type": "classification", "metrics": Matthews}),
+    "mrpc": edict({"num_labels": 2, "seq_length": 128, "task_type": "classification", "metrics": F1}),
+    "sts-b": edict({"num_labels": 1, "seq_length": 128, "task_type": "regression", "metrics": Pearsonr}),
+    "qqp": edict({"num_labels": 2, "seq_length": 128, "task_type": "classification", "metrics": F1}),
+    "rte": edict({"num_labels": 2, "seq_length": 128, "task_type": "classification", "metrics": Accuracy})
+})
+
+train_cfg = edict({
+    'batch_size': 16,
+    'loss_scale_value': 2 ** 16,
+    'scale_factor': 2,
+    'scale_window': 50,
+    'optimizer_cfg': edict({
+        'AdamWeightDecay': edict({
+            'learning_rate': 5e-5,
+            'end_learning_rate': 1e-14,
+            'power': 1.0,
+            'weight_decay': 1e-4,
+            'eps': 1e-6,
+            'decay_filter': lambda x: 'layernorm' not in x.name.lower() and 'bias' not in x.name.lower(),
+            'warmup_ratio': 0.1
+        }),
+    }),
+})
+
+eval_cfg = edict({
+    'batch_size': 32,
+})
+
+teacher_net_cfg = BertConfig(
+    seq_length=128,
+    vocab_size=30522,
+    hidden_size=768,
+    num_hidden_layers=6,
+    num_attention_heads=12,
+    intermediate_size=3072,
+    hidden_act="gelu",
+    hidden_dropout_prob=0.0,
+    attention_probs_dropout_prob=0.0,
+    cls_dropout_prob=0.0,
+    max_position_embeddings=512,
+    type_vocab_size=2,
+    initializer_range=0.02,
+    use_relative_positions=False,
+    dtype=mstype.float32,
+    compute_type=mstype.float32,
+    do_quant=False
+)
+
+student_net_cfg = BertConfig(
+    seq_length=128,
+    vocab_size=30522,
+    hidden_size=768,
+    num_hidden_layers=6,
+    num_attention_heads=12,
+    intermediate_size=3072,
+    hidden_act="gelu",
+    hidden_dropout_prob=0.0,
+    attention_probs_dropout_prob=0.0,
+    max_position_embeddings=512,
+    type_vocab_size=2,
+    initializer_range=0.02,
+    use_relative_positions=False,
+    dtype=mstype.float32,
+    compute_type=mstype.float32,
+    do_quant=True,
+    embedding_bits=2,
+    weight_bits=2,
+    weight_clip_value=3.0,
+    cls_dropout_prob=0.0,
+    activation_init=2.5,
+    is_lgt_fit=False
+)
+
+cfg_cfg = edict({
+    'WEIGHTS_NAME': 'eval_model.ckpt',
+    'EVAL_DATA_NAME': 'eval.tf_record',
+    'TRAIN_DATA_NAME': 'train.tf_record',
+    'DEFAULT_NUM_LABELS': 2,
+    'DEFAULT_SEQ_LENGTH': 128,
+    'DEFAULT_BS': 32
+})

research/nlp/ternarybert/src/dataset.py

+# Copyright 2022 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+"""create tinybert dataset"""
+
+from enum import Enum
+import mindspore.common.dtype as mstype
+# import mindspore.dataset.engine.datasets as de
+import mindspore.dataset as de
+import mindspore.dataset.transforms.c_transforms as C
+
+
+class DataType(Enum):
+    """Enumerate supported dataset format"""
+    TFRECORD = 1
+    MINDRECORD = 2
+
+
+def create_dataset(batch_size=32, device_num=1, rank=0, do_shuffle=True, data_dir=None,
+                   data_type='tfrecord', seq_length=128, task_type=mstype.int32, drop_remainder=True):
+    """create tinybert dataset"""
+    if isinstance(data_dir, list):
+        data_files = data_dir
+    else:
+        data_files = [data_dir]
+
+    columns_list = ["input_ids", "input_mask", "segment_ids", "label_ids"]
+
+    if data_type == 'mindrecord':
+        ds = de.MindDataset(data_files, columns_list=columns_list, shuffle=do_shuffle, num_shards=device_num,
+                            shard_id=rank)
+    else:
+        ds = de.TFRecordDataset(data_files, columns_list=columns_list, shuffle=do_shuffle, num_shards=device_num,
+                                shard_id=rank, shard_equal_rows=(device_num != 1))
+
+    if device_num == 1 and do_shuffle is True:
+        ds = ds.shuffle(10000)
+
+    type_cast_op = C.TypeCast(mstype.int32)
+    slice_op = C.Slice(slice(0, seq_length, 1))
+    label_type = mstype.int32 if task_type == 'classification' else mstype.float32
+    ds = ds.map(operations=[type_cast_op, slice_op], input_columns=["segment_ids"])
+    ds = ds.map(operations=[type_cast_op, slice_op], input_columns=["input_mask"])
+    ds = ds.map(operations=[type_cast_op, slice_op], input_columns=["input_ids"])
+    ds = ds.map(operations=[C.TypeCast(label_type), slice_op], input_columns=["label_ids"])
+    # apply batch operations
+    ds = ds.batch(batch_size, drop_remainder=drop_remainder)
+
+    return ds