* Add diag op (#4344)

* diag算子提交

* diag算子提交

* diag算子提交

* diag op commit

* commit op diag

* commit op diag

* diag op commit

* diag op commit

* diag_op.cpp

* diag_kernel commit

* test_diag.py modified

* diag commit

* code format

* commit gpu_kernel code

* Solve the problems mentioned in the issue

* Solve the problems mentioned in the issue

* Solve the problems mentioned in the issue

* Modify test_diag.py

Co-authored-by: JasenWangLab <junwei_w95@163.com>

* Remove batch axis

* Update diag kernel

* Update

* add blob object id and eager blob object interface

* use maybe for blob object id

* minor fix

* add registry for data type, modified dropout as example

* revert origin data type infer

* add infer data type entry

* minor fix infer data type

* Add infer data type fn

* Update diag test script

* Fix sbp

* Update oneflow_api

Co-authored-by: JasenWangLab <junwei_w95@163.com>
Co-authored-by: poohRui <yuruil@qq.com>
Co-authored-by: Yurui Li <32978179+poohRui@users.noreply.github.com>
Co-authored-by: MARD1NO <359521840@qq.com>
Co-authored-by: ZZK <42901638+MARD1NO@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* op & kernel

* python api

* mv bgemm to self_att*.cu

* test case

* fix bugs

* update test

* add op to amp gray list

* mv to white list

* rename bgemm to BatchedGemm

* refine transpose

* oneflow_api is deprecated

* format

* reshape ops infer parallel_distribution

* refine

* refine

* test case

* test case

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Accumulate partial grad to acc tensor promptly.

* Accumulate partial grad to acc tensor promptly.

* Fix stateful grad function to be stateless.

* Fix

Co-authored-by: hjchen2 <hjchen2>

* add instruction to release tensor

* add detail implementation for cpu and cuda instruction

* minor fix

* fix cpu compile error

* minor fix

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* PhyInstrOperand

* CHECK_NOTNULL

* LocalCallOpKernelUtil

* implement LocalCallOpKernelUtil

* fix WithOpInferContext/WithComputeInferContext

* fix tensor->blob_object() to tensor->eager_blob_object()

* init commit

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* init more ctx in constructor

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* test

Signed-off-by: daquexian <daquexian566@gmail.com>

* set parallel_desc according to scope

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* LogicalRun -> PhysicalRun

* refine stateful op kernel

* refine

* refine

* build eager blob object list before calling builder, rename TensorsPtr

Signed-off-by: daquexian <daquexian566@gmail.com>

* Fix device CHECK_EQ

* update

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

* refine

* code style updates

* add const quantifiers

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix comments

* update tests

Signed-off-by: daquexian <daquexian566@gmail.com>

* revert api/python/symbol/placement_symbol.cpp

Signed-off-by: daquexian <daquexian566@gmail.com>

* update ForEachOutputTensor, replace auto with const auto&

Signed-off-by: daquexian <daquexian566@gmail.com>

* add local dep objects in local opkernel

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: lixinqi <lixinqi0703106@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix a bug in exporting qat model

* update

* update

* update

* update

* update

* update

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* use bitset

* refactor using vector

* refine

* refine

* rename

* refine

* address review

* address review

* refine

* refine

* address review

* smaller BITSET_SIZE

* refine

* refine

* refine

* refine nameing

* refine

* refine

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* set larger threshold

* fix format

Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Remove outdate use_memory_allocation_algorithm_v2

* fix format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Fix spelling error: ture -> true

* fix more spell err in transport.cpp note

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(Autograd): export autograd py-interface and support dynamic accumulate node

* style(*): refine codes

* style(*): format code

* Speed up autograd.grad (#4696)

* refactor(AutogradEngine): speed up autograd.grad

* style(*): format

* refactor(*): add tensor_tuple_util

* style(*): format

* fix(*): user oneflow._oneflow_internal instead of oneflow_api

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* support -m oneflow --doctor

* add in script

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* relax tolerance

* Update oneflow/python/test/ops/test_matmul.py

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* remove_user_op_conf_in_kernel_compute_ctx

* remove_user_op_conf_in_kernel_infer_ctx

* remove_user_op_conf_in_kernel_init_ctx

* remove_user_op_conf_in_kernel_create_ctx

* remove user_op_conf in UserKernelOpInferContext

* solve KernelComputeContext in nvtx_range_kernel.c

* remove user_op_conf in InferContext and slove bug after pull master

* del useless code

* remove attrs_ to derived class

* optimize

* del useless code

* use Attr() instead of attr()

* remove attr() in ctx

* minor fix

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* impl pack in transpose util

* fix transpose sbp

* rm pack in transpose kernel

* fix bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* remove oneflow_api

* refine

* rm

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: hjchen2 <hjchen2>

* op & grad def

* kernel

* update py api

* fix typo

* add test case

* update test

* op & grad def

* kernel

* update py api

* fix typo

* add test case

* update test

* address review comment

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* prune amp_white_identity

* rm useless

* refine

* fix

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>

* Refactor op expr.

* Refine

* Fix a merge mistake

* Add more grad funcs.

* Refine

* Refine

* Mark override

Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Refactor op expr.

* Refine

* Fix a merge mistake

* Refine

* Mark override

Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refactor register function

* revert bug fix for tensor

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* create a source file when running cmake

* refine

* address review

* refine

* refine

* add expand op implementation

* merge master

* fix format

* remove comments

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>