* add python interface

* minor fix

* change name

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* ArgTuple

* refactor BlobRegister::GetObject4BlobName

* refactor the type of ArgTuple::arg_name2bn_index2tensor_tuple_index_

* add GetRank and GetWordSize interface

* minor fix

* add dropout module

* update identity module and test case

* add std module

* add layernorm module

* format

* refine according to comments

* add sqrt square module

* update

* restructure d ro pdropout module

* format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Fw/Bw support double compute stream

* NCCL comm create by stream id

* 2D NCCL logical kernel support BW independent stream

* StreamIndex: NcclComputeStream for each subgraph insert nccl logical.

* refactor code

* refine code for review

* Add WITH_CUDA in DoJobPass(InsertNcclLogicalOpPass)

* Pipeline Parallelism: checkpointing insert identity buffer op

* fix complier err

* identity buffer op custom out regst num

* fix bug and runnable

* Chain merge divide fw/bw; MemChain ignore merge; copyhd regst num hack

* Pipeline buffer pass

* Pipeline runnable

* rollback NOT merge mem chain hack

* pipeline_stage_id_hint and rollback checkpointing buffer

* Pipeline buffer only. test pass.

* rollback repeat hack

* Remove CopyHd Hack; Add buffer cross label loader and loss

* InsertNcclLogicalOps support multi-subgraph by placement group

* Implement of InsertNcclLogicalOpPass support multi-subgraph and batch acc and pipeline parallel

* Pipeline + 2D-SBP runnable

* remote note

* WARNING to INFO

* refine code for review & fix for new dtype infer

* add note

* Hotfix ib qp max_send_wr

* Pipeline Debug: hack copy hd/commnet regst num to 2 and increase buffer regst num +1

* add buffer op for copyD2H pipe

* fix bug of insert buffer...

* dont use import to determine if export onnx

* fix

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Pipeline Parallelism: checkpointing insert identity buffer op

* fix complier err

* identity buffer op custom out regst num

* fix bug and runnable

* Chain merge divide fw/bw; MemChain ignore merge; copyhd regst num hack

* Pipeline buffer pass

* Pipeline runnable

* rollback NOT merge mem chain hack

* pipeline_stage_id_hint and rollback checkpointing buffer

* Pipeline buffer only. test pass.

* rollback repeat hack

* Remove CopyHd Hack; Add buffer cross label loader and loss

* InsertNcclLogicalOps support multi-subgraph by placement group

* Implement of InsertNcclLogicalOpPass support multi-subgraph and batch acc and pipeline parallel

* Pipeline + 2D-SBP runnable

* remote note

* WARNING to INFO

* refine code for review & fix for new dtype infer

* add note

* collection reserve for saving rehash cost

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add 163 centos mirror

* fix cuda version

* update dockerfile

* add gdb

* refine

* refine

* add one api mkl path

* refine

* refine path

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add logsoftmax module

* add sigmoid module

* add maskedfill module

* format

* add unsquuze module

* add eq module

* add arange module

* fix arange module

* add softmax module

* refine

* update logsoftmax module

* update eq module

* update eq module

* refine

* refine according to comments

* refine softmax test case

* fix unsqueeze module

* refine according to comments

* refine

* format

* format

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add modules:

- sum
- mean
- add
- sub
- mul
- div
- reciprocal

* add +, -, *, / for tensor

* add test case for add,div,mean,mul,reciprocal,sub,sum

* refine test cases

* refine sum module

* remove useless oneflow_export for mean and sum

* add tensor op test for mean and sum

* add CrossEntropyLoss module

* refine Sum

* sum axis attr not found

* refine sum

* refine math ops

* refien cross entropy loss

* refine (not finished)

* fix scalar div, sub

* rename test Module name and refine test case

* rename test Module name and refine test case

* add neg method for Tensor

* remove useless improt items

* refine

* reformat

* remove useless dynamic attr

* fix math_ops and loss

* rename test

* fix format

* delete name

* refine codes and format

* style(*): format codes

* fix(MathOp): fix merge bug

Co-authored-by: simonJJJ <821898965@qq.com>
Co-authored-by: wyg1997 <wyg19970408@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add exp_tanh_gelu module

* fix comment

* fix comment

* fix comment

* change docs location

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add conv2d module

* support bias

* add conv2d test

* update conv2d test

* return nullptr instead of LOG(FATAL) when arg is not found

* format

* format & update conv2d test

* update test

* add reset_parameters for conv2d

* fix a bug

Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

This reverts commit 32e439ba.

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add concat_matmul module

* add concat_matmul module

* fix comment

* fix comment

* fix comment

* fix comment

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Pipeline Parallelism: checkpointing insert identity buffer op

* fix complier err

* identity buffer op custom out regst num

* fix bug and runnable

* Chain merge divide fw/bw; MemChain ignore merge; copyhd regst num hack

* Pipeline buffer pass

* Pipeline runnable

* rollback NOT merge mem chain hack

* pipeline_stage_id_hint and rollback checkpointing buffer

* Pipeline buffer only. test pass.

* rollback repeat hack

* Remove CopyHd Hack; Add buffer cross label loader and loss

* refine code for review & fix for new dtype infer

* add note

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add tmp.rst

* fix bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix typo

* create ref for more nodes

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Return index for SaveTensorForBackward.

* Fix layer_norm gradient function.

Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add reshape_expand_repeat module

* fix comment

* fix comment

* update docs

* fix comment

* fix comment

* fix comment

* fix bug

* fix bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add relu module and test case

* add maxpool2d module and test case

* update according to pr4746

* add batchnorm module

* update sample code

* format

* refine according to reviewer's comment

* refine batchnorm test case

* fix asserttation

* refine

* add maxpool2d testcase

* update date of license

* add constant modules

* refine

* add sin cos log module

* add more test cases

* add maxpool2d test case

* refine

* fix tmp_buffer

Signed-off-by: daquexian <daquexian566@gmail.com>

* refine constant module

* refine according to comments

* fix according comments

* recover license date

* remove todo item

* remove useless codes

* remove useless import

* refine

* update constant modules

Signed-off-by: daquexian <daquexian566@gmail.com>

* refine docs

* remove device param

* add test case

* update pooling test case

* add batchnorm eval test

* refine according to comments

* rename var and remove useless codes

* refine batchnorm test case

* refine according to comments

* Update stateful_local_opkernel.cpp

* fix pooling module

* fix according to comments

* refine

* refine

* refine param

* refine api docs

* flake format

* format

Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: Ldpe2G <liangdepeng@gmail.com>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Refactor

* Draft

* Refactor

* Add MutableCfgAttrValueMap

* implement AttrValueMap and ComposedAttrValueMap (#4767)

* Attr value util (#4773)

* implement AttrValueMap and ComposedAttrValueMap

* AttrValueUtil::ToProtoAttrValue

* Fix compile

* Fix compilation

* Rename AttrValueMap by AttrMap.

Co-authored-by: lixinqi <lixinqi0703106@163.com>
Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* move default log under log dir

* remove default env log from proto

* refine

Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* NcclLogialOpAllGatherNoncontinuous S1 to B

* delete useless file

* fix user op attr err

* fix tmp_buffer

* Update oneflow/user/kernels/nccl_logical_kernels.cpp

* fix data size check in 2d s1-b

Co-authored-by: guo-ran <360112263@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Juncheng <liujuncheng1022@gmail.com>

* add flatten module and unit test

* add changes according to review

* fix docs

* use nn.init

* add modification according to review

* remove useless code

* add more tests & fix doc

* fix default parameters for module and function

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add greater_less_argmax module

* fix comment

* add test_case

* fix conflict

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* format file

Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* model_io_v2 process multi variable

* refine

* reserve

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* b21 boxing add ctrl_edge

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: cheng cheng <472491134@qq.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add memory detect info

* small fix in opattrref optimize

* use bitset

* refactor using vector

* refine

* refine

* rename

* refine

* address review

* address review

* refine

* refine

* address review

* smaller BITSET_SIZE

* refine

* refine

* refine

* refine nameing

* refine

* refine

* refine

* update

* delete swp file

* small update

* format fix

* format modify

* format modify

* Update compiler.cpp

fix for comment

* Update reshape_user_op_util.cpp

bug about reshape is fixed

Co-authored-by: jackalcooper <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add exp_tanh_gelu module

* fix comment

* fix comment

* fix comment

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>