* op & kernel

* python api

* mv bgemm to self_att*.cu

* test case

* fix bugs

* update test

* add op to amp gray list

* mv to white list

* rename bgemm to BatchedGemm

* refine transpose

* oneflow_api is deprecated

* format

* reshape ops infer parallel_distribution

* refine

* refine

* test case

* test case

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Accumulate partial grad to acc tensor promptly.

* Accumulate partial grad to acc tensor promptly.

* Fix stateful grad function to be stateless.

* Fix

Co-authored-by: hjchen2 <hjchen2>

* add instruction to release tensor

* add detail implementation for cpu and cuda instruction

* minor fix

* fix cpu compile error

* minor fix

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* PhyInstrOperand

* CHECK_NOTNULL

* LocalCallOpKernelUtil

* implement LocalCallOpKernelUtil

* fix WithOpInferContext/WithComputeInferContext

* fix tensor->blob_object() to tensor->eager_blob_object()

* init commit

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* init more ctx in constructor

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* test

Signed-off-by: daquexian <daquexian566@gmail.com>

* set parallel_desc according to scope

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* LogicalRun -> PhysicalRun

* refine stateful op kernel

* refine

* refine

* build eager blob object list before calling builder, rename TensorsPtr

Signed-off-by: daquexian <daquexian566@gmail.com>

* Fix device CHECK_EQ

* update

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

* refine

* code style updates

* add const quantifiers

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix comments

* update tests

Signed-off-by: daquexian <daquexian566@gmail.com>

* revert api/python/symbol/placement_symbol.cpp

Signed-off-by: daquexian <daquexian566@gmail.com>

* update ForEachOutputTensor, replace auto with const auto&

Signed-off-by: daquexian <daquexian566@gmail.com>

* add local dep objects in local opkernel

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: lixinqi <lixinqi0703106@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix a bug in exporting qat model

* update

* update

* update

* update

* update

* update

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* use bitset

* refactor using vector

* refine

* refine

* rename

* refine

* address review

* address review

* refine

* refine

* address review

* smaller BITSET_SIZE

* refine

* refine

* refine

* refine nameing

* refine

* refine

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* set larger threshold

* fix format

Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Remove outdate use_memory_allocation_algorithm_v2

* fix format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Fix spelling error: ture -> true

* fix more spell err in transport.cpp note

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(Autograd): export autograd py-interface and support dynamic accumulate node

* style(*): refine codes

* style(*): format code

* Speed up autograd.grad (#4696)

* refactor(AutogradEngine): speed up autograd.grad

* style(*): format

* refactor(*): add tensor_tuple_util

* style(*): format

* fix(*): user oneflow._oneflow_internal instead of oneflow_api

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* support -m oneflow --doctor

* add in script

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* relax tolerance

* Update oneflow/python/test/ops/test_matmul.py

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* remove_user_op_conf_in_kernel_compute_ctx

* remove_user_op_conf_in_kernel_infer_ctx

* remove_user_op_conf_in_kernel_init_ctx

* remove_user_op_conf_in_kernel_create_ctx

* remove user_op_conf in UserKernelOpInferContext

* solve KernelComputeContext in nvtx_range_kernel.c

* remove user_op_conf in InferContext and slove bug after pull master

* del useless code

* remove attrs_ to derived class

* optimize

* del useless code

* use Attr() instead of attr()

* remove attr() in ctx

* minor fix

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* impl pack in transpose util

* fix transpose sbp

* rm pack in transpose kernel

* fix bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* remove oneflow_api

* refine

* rm

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: hjchen2 <hjchen2>

* op & grad def

* kernel

* update py api

* fix typo

* add test case

* update test

* op & grad def

* kernel

* update py api

* fix typo

* add test case

* update test

* address review comment

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* prune amp_white_identity

* rm useless

* refine

* fix

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>

* Refactor op expr.

* Refine

* Fix a merge mistake

* Add more grad funcs.

* Refine

* Refine

* Mark override

Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Refactor op expr.

* Refine

* Fix a merge mistake

* Refine

* Mark override

Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refactor register function

* revert bug fix for tensor

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* create a source file when running cmake

* refine

* address review

* refine

* refine

* add expand op implementation

* merge master

* fix format

* remove comments

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* NCCL logical refine timeshape

* Insert nccl ops after acc interface

* Inser NCCL ops after acc implement; need refine or add new acc_tick_op

* deadlock

* speed up and run

* add acc tick fix deadlocak ; and add nccl comm debug log

* refine log: rm cc_debug_log and cclog

* use reference for speed up

* refine code for review

* fix for review

Co-authored-by: Juncheng <liujuncheng1022@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Juncheng <liujuncheng1022@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>