Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add greater_less_argmax module

* fix comment

* add test_case

* fix conflict

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* format file

Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* model_io_v2 process multi variable

* refine

* reserve

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* b21 boxing add ctrl_edge

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: cheng cheng <472491134@qq.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add memory detect info

* small fix in opattrref optimize

* use bitset

* refactor using vector

* refine

* refine

* rename

* refine

* address review

* address review

* refine

* refine

* address review

* smaller BITSET_SIZE

* refine

* refine

* refine

* refine nameing

* refine

* refine

* refine

* update

* delete swp file

* small update

* format fix

* format modify

* format modify

* Update compiler.cpp

fix for comment

* Update reshape_user_op_util.cpp

bug about reshape is fixed

Co-authored-by: jackalcooper <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add exp_tanh_gelu module

* fix comment

* fix comment

* fix comment

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* need to be reformat

* reformat

* add docstring and refine test case

* add test case

* refine according to comments of wyg

* refine

* add TODO for asymmetric padding

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix tmp_buffer in stateful local kernel

Signed-off-by: daquexian <daquexian566@gmail.com>

* move tmp_buffer_view_ != nullptr first

Signed-off-by: daquexian <daquexian566@gmail.com>

* Move tmp_buffer check to bottom

* add specific tmp_buffer not null check

Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix summary_graph bug

* modify test_summary

Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>

* remove useless register func

* Update oneflow/python/framework/tensor.py

Co-authored-by: daquexian <daquexian566@gmail.com>

* format

Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add case

* mv files

* refine

* refine

* rm unnecessary skip

* refine

* refine

* fix order

* reorder

* refine

* refine

* rename cuda_new_interface

* fix module load_state_dict test

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: daquexian <daquexian566@gmail.com>

* add linear module

* add _calculate_fan_in_and_fan_out func

* fix comment

* fix linear module test bug

* fix linear module test bug

* fix comment

* fix comment and add docs for linear module

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Support dynamic attrs in local kernel (not tested)

* Support updating UserOpConfWrapper by make methods virtual

* Update attrs in instructions instead of main thread

* call clear_attr()

* Set dynamic attrs based on a based op_conf

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Rename namespace eager to vm

* Merge the same namespace

* Fix conflict

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Rename InferXXXFn as XXXInferFn

* rename interface_name

* modify infer_xxx_fn to xxx_infer_fn

Co-authored-by: binbinHan <han_binbin@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* mark todo

* inline _Run to PhysicalRun and Logical

* construct instruction_list's shared_ptr and input nullptr to id_generator

* non-allocated space construction function

* modify eager_symbol_list

* modify vm::InstructionMsgList*

* change std::shared_ptr<vm::InstructionMsgList> to vm::InstructionMsgList*

* del istr_list and symbol_list in session

* optimize

* minor fix

* use InstructionsBuilder*

* fix eager run bug

* optimize

* fix bug with update master

* fix bug

* make of_format

Co-authored-by: wanghongsheng <2496533749@qq.com>
Co-authored-by: clackhan <han_binbin@163.com>

remove SourceIntruction, add ResettingIdToObjectMap, fix bug in ForEachMutMirroredObject4MutPhyInstrOperand (#4734)

* Add NeedsRunInAdvance()

* Rename to ResettingIdToObjectMap

* rename IsImmediateOperandsOnly -> HasImmediateOperandsOnly

Signed-off-by: daquexian <daquexian566@gmail.com>

* Fix ForEachMut2MirroredObject -> ForEachMutMirroredObject in ForEachMutMirroredObject4MutPhyInstrOperand

Signed-off-by: daquexian <daquexian566@gmail.com>

* split CHECK(..&&..) to two CHECK(..)

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* release tensor by instructions, update shut_down_util, skip tensor releasing when exiting

Signed-off-by: daquexian <daquexian566@gmail.com>

* Captures shared_ptr instread of raw pointer

Co-authored-by: lixinqi <lixinqi0703106@163.com>

* startup

* empty op finished

* add sbp signature

* refine test case for fp16

* try to fix sbp problem

* refien sbpGetFn

* add sbp config attr

* refine

* refine

* add balancedSpliter and add parallel check on py

* refine

* add partialSum parallel support

* unexported empty and rm its test case

* rm python wrapper for empty op

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fused_cast_scale_pass fuse scalar_mul

* test case

* refine

* fix

Co-authored-by: Juncheng <liujuncheng1022@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* use bitset

* refactor using vector

* refine

* refine

* rename

* refine

* address review

* address review

* refine

* refine

* address review

* smaller BITSET_SIZE

* refine

* refine

* refine

* refine nameing

* refine

* refine

* move full plan when pushing plan

* refine

* refine

* move tasks

* rm const

* refine

* refine

* refine

* only insert bn parallel_conf if different from op

* refine

* address review

* address review

* refine

* fix for worker

* rename

* refine

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* skip test_gpt_data_loader in eager mode

* change_GetOpAttributes_to_GetInterfaceOpAttributes

* checkout change of test_gpt_data_loader.py

* fix empty parallel desc

* refactor device

* format

* minor fix

* format

* add default environment

* python interface for default environment

* remove mutex

* move hash value to device

* refine

* remove env global object scope

* minor fix

* format

* refine device

* recursively delete log file in default mode

* add directory for default log

* revert

* fix duplicate methods

* try to fix multiprocess faill

* add default init env

* minor fix

* rename

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* PhyInstrOperand

* CHECK_NOTNULL

* LocalCallOpKernelUtil

* implement LocalCallOpKernelUtil

* fix WithOpInferContext/WithComputeInferContext

* fix tensor->blob_object() to tensor->eager_blob_object()

* init commit

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* init more ctx in constructor

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* test

Signed-off-by: daquexian <daquexian566@gmail.com>

* set parallel_desc according to scope

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* LogicalRun -> PhysicalRun

* refine stateful op kernel

* refine

* refine

* build eager blob object list before calling builder, rename TensorsPtr

Signed-off-by: daquexian <daquexian566@gmail.com>

* Fix device CHECK_EQ

* update

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

* refine

* code style updates

* wait on tensor.shape

Signed-off-by: daquexian <daquexian566@gmail.com>

* wait on .numpy(), build tensor directly by eager blob object

* add const quantifiers

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix comments

* update tests

Signed-off-by: daquexian <daquexian566@gmail.com>

* revert api/python/symbol/placement_symbol.cpp

Signed-off-by: daquexian <daquexian566@gmail.com>

* update ForEachOutputTensor, replace auto with const auto&

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* make indexed_in/output_pairs_ shared_ptr

Signed-off-by: daquexian <daquexian566@gmail.com>

* add local dep objects in local opkernel

Signed-off-by: daquexian <daquexian566@gmail.com>

* refine python code

Signed-off-by: daquexian <daquexian566@gmail.com>

* refine

Signed-off-by: daquexian <daquexian566@gmail.com>

* MakeTensor -> MakeEagerTensor

Signed-off-by: daquexian <daquexian566@gmail.com>

* add InferAccessBlobByCallback

Signed-off-by: daquexian <daquexian566@gmail.com>

* replace InferAccessBlobByCallback with AccessTensorShapeByCallback

Signed-off-by: daquexian <daquexian566@gmail.com>

* refine

Signed-off-by: daquexian <daquexian566@gmail.com>

* remove unused header and forward declaration

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix typo and AccessTensorShapeByCallback bug

Signed-off-by: daquexian <daquexian566@gmail.com>

* rename access_tensor_shape -> read_tensor_shape, skip compute callback

Signed-off-by: daquexian <daquexian566@gmail.com>

* remove unused OfBlob::mut_blob()

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix control stream type

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix InferStreamType<ControlStreamType> specialization

Signed-off-by: daquexian <daquexian566@gmail.com>

* Add ForeignLock, make mirrored default

Signed-off-by: daquexian <daquexian566@gmail.com>

* init is_mirrored_strategy_enabled_stack_ with {true}

Signed-off-by: daquexian <daquexian566@gmail.com>

* restore shape() return type to reference

Signed-off-by: daquexian <daquexian566@gmail.com>

* add check in PopMirroredStrategyEnabled

Signed-off-by: daquexian <daquexian566@gmail.com>

* add const in WithScopedRelease

Signed-off-by: daquexian <daquexian566@gmail.com>

* rename ForeignLock names

Signed-off-by: daquexian <daquexian566@gmail.com>

* restore session_util.cpp changes and make mirrored default only in op interpreter

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: lixinqi <lixinqi0703106@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Add diag op (#4344)

* diag算子提交

* diag算子提交

* diag算子提交

* diag op commit

* commit op diag

* commit op diag

* diag op commit

* diag op commit

* diag_op.cpp

* diag_kernel commit

* test_diag.py modified

* diag commit

* code format

* commit gpu_kernel code

* Solve the problems mentioned in the issue

* Solve the problems mentioned in the issue

* Solve the problems mentioned in the issue

* Modify test_diag.py

Co-authored-by: JasenWangLab <junwei_w95@163.com>

* Remove batch axis

* Update diag kernel

* Update

* add blob object id and eager blob object interface

* use maybe for blob object id

* minor fix

* add registry for data type, modified dropout as example

* revert origin data type infer

* add infer data type entry

* minor fix infer data type

* Add infer data type fn

* Update diag test script

* Fix sbp

* Update oneflow_api

Co-authored-by: JasenWangLab <junwei_w95@163.com>
Co-authored-by: poohRui <yuruil@qq.com>
Co-authored-by: Yurui Li <32978179+poohRui@users.noreply.github.com>
Co-authored-by: MARD1NO <359521840@qq.com>
Co-authored-by: ZZK <42901638+MARD1NO@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* op & kernel

* python api

* mv bgemm to self_att*.cu

* test case

* fix bugs

* update test

* add op to amp gray list

* mv to white list

* rename bgemm to BatchedGemm

* refine transpose

* oneflow_api is deprecated

* format

* reshape ops infer parallel_distribution

* refine

* refine

* test case

* test case

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Accumulate partial grad to acc tensor promptly.

* Accumulate partial grad to acc tensor promptly.

* Fix stateful grad function to be stateless.

* Fix

Co-authored-by: hjchen2 <hjchen2>

* add instruction to release tensor

* add detail implementation for cpu and cuda instruction

* minor fix

* fix cpu compile error

* minor fix

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* PhyInstrOperand

* CHECK_NOTNULL

* LocalCallOpKernelUtil

* implement LocalCallOpKernelUtil

* fix WithOpInferContext/WithComputeInferContext

* fix tensor->blob_object() to tensor->eager_blob_object()

* init commit

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* init more ctx in constructor

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* test

Signed-off-by: daquexian <daquexian566@gmail.com>

* set parallel_desc according to scope

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* LogicalRun -> PhysicalRun

* refine stateful op kernel

* refine

* refine

* build eager blob object list before calling builder, rename TensorsPtr

Signed-off-by: daquexian <daquexian566@gmail.com>

* Fix device CHECK_EQ

* update

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

* refine

* code style updates

* add const quantifiers

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix comments

* update tests

Signed-off-by: daquexian <daquexian566@gmail.com>

* revert api/python/symbol/placement_symbol.cpp

Signed-off-by: daquexian <daquexian566@gmail.com>

* update ForEachOutputTensor, replace auto with const auto&

Signed-off-by: daquexian <daquexian566@gmail.com>

* add local dep objects in local opkernel

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: lixinqi <lixinqi0703106@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix a bug in exporting qat model

* update

* update

* update

* update

* update

* update

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* use bitset

* refactor using vector

* refine

* refine

* rename

* refine

* address review

* address review

* refine

* refine

* address review

* smaller BITSET_SIZE

* refine

* refine

* refine

* refine nameing

* refine

* refine

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>