* Pipeline Parallelism: checkpointing insert identity buffer op

* fix complier err

* identity buffer op custom out regst num

* fix bug and runnable

* Chain merge divide fw/bw; MemChain ignore merge; copyhd regst num hack

* Pipeline buffer pass

* Pipeline runnable

* rollback NOT merge mem chain hack

* pipeline_stage_id_hint and rollback checkpointing buffer

* Pipeline buffer only. test pass.

* rollback repeat hack

* Remove CopyHd Hack; Add buffer cross label loader and loss

* refine code for review & fix for new dtype infer

* add note

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add tmp.rst

* fix bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix typo

* create ref for more nodes

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Return index for SaveTensorForBackward.

* Fix layer_norm gradient function.

Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add reshape_expand_repeat module

* fix comment

* fix comment

* update docs

* fix comment

* fix comment

* fix comment

* fix bug

* fix bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add relu module and test case

* add maxpool2d module and test case

* update according to pr4746

* add batchnorm module

* update sample code

* format

* refine according to reviewer's comment

* refine batchnorm test case

* fix asserttation

* refine

* add maxpool2d testcase

* update date of license

* add constant modules

* refine

* add sin cos log module

* add more test cases

* add maxpool2d test case

* refine

* fix tmp_buffer

Signed-off-by: daquexian <daquexian566@gmail.com>

* refine constant module

* refine according to comments

* fix according comments

* recover license date

* remove todo item

* remove useless codes

* remove useless import

* refine

* update constant modules

Signed-off-by: daquexian <daquexian566@gmail.com>

* refine docs

* remove device param

* add test case

* update pooling test case

* add batchnorm eval test

* refine according to comments

* rename var and remove useless codes

* refine batchnorm test case

* refine according to comments

* Update stateful_local_opkernel.cpp

* fix pooling module

* fix according to comments

* refine

* refine

* refine param

* refine api docs

* flake format

* format

Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: Ldpe2G <liangdepeng@gmail.com>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Refactor

* Draft

* Refactor

* Add MutableCfgAttrValueMap

* implement AttrValueMap and ComposedAttrValueMap (#4767)

* Attr value util (#4773)

* implement AttrValueMap and ComposedAttrValueMap

* AttrValueUtil::ToProtoAttrValue

* Fix compile

* Fix compilation

* Rename AttrValueMap by AttrMap.

Co-authored-by: lixinqi <lixinqi0703106@163.com>
Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* move default log under log dir

* remove default env log from proto

* refine

Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* NcclLogialOpAllGatherNoncontinuous S1 to B

* delete useless file

* fix user op attr err

* fix tmp_buffer

* Update oneflow/user/kernels/nccl_logical_kernels.cpp

* fix data size check in 2d s1-b

Co-authored-by: guo-ran <360112263@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Juncheng <liujuncheng1022@gmail.com>

* add flatten module and unit test

* add changes according to review

* fix docs

* use nn.init

* add modification according to review

* remove useless code

* add more tests & fix doc

* fix default parameters for module and function

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add greater_less_argmax module

* fix comment

* add test_case

* fix conflict

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* format file

Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* model_io_v2 process multi variable

* refine

* reserve

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* b21 boxing add ctrl_edge

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: cheng cheng <472491134@qq.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add memory detect info

* small fix in opattrref optimize

* use bitset

* refactor using vector

* refine

* refine

* rename

* refine

* address review

* address review

* refine

* refine

* address review

* smaller BITSET_SIZE

* refine

* refine

* refine

* refine nameing

* refine

* refine

* refine

* update

* delete swp file

* small update

* format fix

* format modify

* format modify

* Update compiler.cpp

fix for comment

* Update reshape_user_op_util.cpp

bug about reshape is fixed

Co-authored-by: jackalcooper <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add exp_tanh_gelu module

* fix comment

* fix comment

* fix comment

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* need to be reformat

* reformat

* add docstring and refine test case

* add test case

* refine according to comments of wyg

* refine

* add TODO for asymmetric padding

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix tmp_buffer in stateful local kernel

Signed-off-by: daquexian <daquexian566@gmail.com>

* move tmp_buffer_view_ != nullptr first

Signed-off-by: daquexian <daquexian566@gmail.com>

* Move tmp_buffer check to bottom

* add specific tmp_buffer not null check

Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix summary_graph bug

* modify test_summary

Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>

* remove useless register func

* Update oneflow/python/framework/tensor.py

Co-authored-by: daquexian <daquexian566@gmail.com>

* format

Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add case

* mv files

* refine

* refine

* rm unnecessary skip

* refine

* refine

* fix order

* reorder

* refine

* refine

* rename cuda_new_interface

* fix module load_state_dict test

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: daquexian <daquexian566@gmail.com>

* add linear module

* add _calculate_fan_in_and_fan_out func

* fix comment

* fix linear module test bug

* fix linear module test bug

* fix comment

* fix comment and add docs for linear module

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Support dynamic attrs in local kernel (not tested)

* Support updating UserOpConfWrapper by make methods virtual

* Update attrs in instructions instead of main thread

* call clear_attr()

* Set dynamic attrs based on a based op_conf

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Rename namespace eager to vm

* Merge the same namespace

* Fix conflict

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Rename InferXXXFn as XXXInferFn

* rename interface_name

* modify infer_xxx_fn to xxx_infer_fn

Co-authored-by: binbinHan <han_binbin@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* mark todo

* inline _Run to PhysicalRun and Logical

* construct instruction_list's shared_ptr and input nullptr to id_generator

* non-allocated space construction function

* modify eager_symbol_list

* modify vm::InstructionMsgList*

* change std::shared_ptr<vm::InstructionMsgList> to vm::InstructionMsgList*

* del istr_list and symbol_list in session

* optimize

* minor fix

* use InstructionsBuilder*

* fix eager run bug

* optimize

* fix bug with update master

* fix bug

* make of_format

Co-authored-by: wanghongsheng <2496533749@qq.com>
Co-authored-by: clackhan <han_binbin@163.com>

remove SourceIntruction, add ResettingIdToObjectMap, fix bug in ForEachMutMirroredObject4MutPhyInstrOperand (#4734)

* Add NeedsRunInAdvance()

* Rename to ResettingIdToObjectMap

* rename IsImmediateOperandsOnly -> HasImmediateOperandsOnly

Signed-off-by: daquexian <daquexian566@gmail.com>

* Fix ForEachMut2MirroredObject -> ForEachMutMirroredObject in ForEachMutMirroredObject4MutPhyInstrOperand

Signed-off-by: daquexian <daquexian566@gmail.com>

* split CHECK(..&&..) to two CHECK(..)

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* release tensor by instructions, update shut_down_util, skip tensor releasing when exiting

Signed-off-by: daquexian <daquexian566@gmail.com>

* Captures shared_ptr instread of raw pointer

Co-authored-by: lixinqi <lixinqi0703106@163.com>

* startup

* empty op finished

* add sbp signature

* refine test case for fp16

* try to fix sbp problem

* refien sbpGetFn

* add sbp config attr

* refine

* refine

* add balancedSpliter and add parallel check on py

* refine

* add partialSum parallel support

* unexported empty and rm its test case

* rm python wrapper for empty op

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fused_cast_scale_pass fuse scalar_mul

* test case

* refine

* fix

Co-authored-by: Juncheng <liujuncheng1022@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* use bitset

* refactor using vector

* refine

* refine

* rename

* refine

* address review

* address review

* refine

* refine

* address review

* smaller BITSET_SIZE

* refine

* refine

* refine

* refine nameing

* refine

* refine

* move full plan when pushing plan

* refine

* refine

* move tasks

* rm const

* refine

* refine

* refine

* only insert bn parallel_conf if different from op

* refine

* address review

* address review

* refine

* fix for worker

* rename

* refine

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* skip test_gpt_data_loader in eager mode

* change_GetOpAttributes_to_GetInterfaceOpAttributes

* checkout change of test_gpt_data_loader.py

* fix empty parallel desc

* refactor device

* format

* minor fix

* format

* add default environment

* python interface for default environment

* remove mutex

* move hash value to device

* refine

* remove env global object scope

* minor fix

* format

* refine device

* recursively delete log file in default mode

* add directory for default log

* revert

* fix duplicate methods

* try to fix multiprocess faill

* add default init env

* minor fix

* rename

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>