* refactor(Optim): refine optimizer codes

* docs(SGD): add document for SGD

* docs(SGD): fix code

* test(Adam): fix test_optim_adam bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add scalar input support

* format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Rewrite sparse softmax cross entropy gradient func.

* Rewrite reduce ops gradient funcs.

* Add crossentropyloss grad unittest.

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: hjchen2 <hjchen2>

* infer shape in main thread and fix multi-thread bug

Signed-off-by: daquexian <daquexian566@gmail.com>

* void -> Maybe<void>

Signed-off-by: daquexian <daquexian566@gmail.com>

* rename op_infer_ctx_1/2

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix comments

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix typo

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add broadcast matmul support

* refine

* add batch matmul support

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add ones_like and zeros_like op with tests

* docs(*): add documents for ones_like and zeros_like op

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* device_infer_fn

* Device::local_call_instruction_name

* implement UserOpExprDeviceInferContext

* merge master

* refactor UserOpConfWrapper

* fix a ci bug

* Refine (#4825)

* fix segmentation fault bug

* add cpu only device_infer_fn

* optimize code

* del blank line

* minor fix

* override Device::New

* minor fix

Co-authored-by: lixinqi <lixinqi0703106@163.com>
Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: Houjiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* rebase master

* refine according to review

* fix format

* fix add, sub, div module unit tests

* fix unit test dtype

* fix format

* make changes according to review

* remove logs

* merge master

* little changes

* revert flow.Tensor default dtype

* remove print

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* initial commit

* format code

* add test case for align_corners

* add compare_with_numpy

Co-authored-by: Xuefei Jiang <sc.xfjiang@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Serialize proto in binary rather than text

* move del ops out from loop

* refine

* Skip GenCollectiveBoxingPlan if no CollectiveBoxingTaskNode

* multi core to proto

* copy pointers explicitly

* make toproto const method

* reorder

* larger tol

* Update test_layers_conv1d.py

* fix deadlock

* remove ForeignCallBack in Operator::ToOpAttribute

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: liujuncheng <liujuncheng1022@gmail.com>
Co-authored-by: clackhan <han_binbin@163.com>

* add pow module

* update

* update

* move pow module into new file

* refine doc

* format

* update pow and its test

* format

Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* device_infer_fn

* Device::local_call_instruction_name

* implement UserOpExprDeviceInferContext

* merge master

* refactor UserOpConfWrapper

* fix a ci bug

* Refine (#4825)

* fix segmentation fault bug

Co-authored-by: Houjiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: clackhan <han_binbin@163.com>

* feat(Optim): add sgd

* add adam optimizer

* add adam optimizer

* add adam optimize module

* add adam optimizer module

* fix adam optimizer module bug

* add adam optimizer

* file format

* add adam optimizer docs

* fix comment

* fix comment

* style(*): format codes

Co-authored-by: wyg1997 <wyg19970408@gmail.com>

* fix_cpu_only_complie_bug_with_gcc_7_3_0

* move -lrt to oneflow.cmake

* minor fix

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Serialize proto in binary rather than text

* move del ops out from loop

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(Optim): add sgd

* fix(*): add no_grad guard in optimizer

* fix(*): fix bug in optimizer test

* style(*): format codes

* style(Optimizer): refine codes

* style(Optimizer): refine codes

* fix(Optimizer): fix bug in ParamGroup

* fix(*): add __init__.py

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Accumulate partial grad to acc tensor promptly.

* refactor(Tensor): add AutogradMeta and delete retain_grad in constructor

* refactor(AutogradEngine): function node just cache autograd_meta and
                          output tensor_infos

* fix(*): fix bugs and add zeros op

* Capture detached output tensors.

* fix(*): fix merge bugs

* style(*): refine code

* fix(*): refine FunctionNode release

* fix(Autograd): save input tensors which requires_grad is true

* fix(*): fix bug

* format code

* chore(*): add head file in cmake

Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Yurui Li <32978179+poohRui@users.noreply.github.com>

Co-authored-by: ZZK <42901638+MARD1NO@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix crossentropy_loss bug

* fix comment

* fix comment

* format test_crossentropyloss.py

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add negative and nllloss module

* add negative and nllloss module

* add negative and nllloss module

* add negative and nllloss module

* add nllloss impleted, relay on other module, do not merge now

* fix nllloss bug

* support 2d nllloss bug

* fix comment

* add flow.neg api

* support bert nllloss

* fix nllloss 2d bug

* add docs

* fix comment

* fix comment

* fix comment

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Serialize proto in binary rather than text

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add python interface

* minor fix

* change name

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* ArgTuple

* refactor BlobRegister::GetObject4BlobName

* refactor the type of ArgTuple::arg_name2bn_index2tensor_tuple_index_

* add GetRank and GetWordSize interface

* minor fix

* add dropout module

* update identity module and test case

* add std module

* add layernorm module

* format

* refine according to comments

* add sqrt square module

* update

* restructure d ro pdropout module

* format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Fw/Bw support double compute stream

* NCCL comm create by stream id

* 2D NCCL logical kernel support BW independent stream

* StreamIndex: NcclComputeStream for each subgraph insert nccl logical.

* refactor code

* refine code for review

* Add WITH_CUDA in DoJobPass(InsertNcclLogicalOpPass)

* Pipeline Parallelism: checkpointing insert identity buffer op

* fix complier err

* identity buffer op custom out regst num

* fix bug and runnable

* Chain merge divide fw/bw; MemChain ignore merge; copyhd regst num hack

* Pipeline buffer pass

* Pipeline runnable

* rollback NOT merge mem chain hack

* pipeline_stage_id_hint and rollback checkpointing buffer

* Pipeline buffer only. test pass.

* rollback repeat hack

* Remove CopyHd Hack; Add buffer cross label loader and loss

* InsertNcclLogicalOps support multi-subgraph by placement group

* Implement of InsertNcclLogicalOpPass support multi-subgraph and batch acc and pipeline parallel

* Pipeline + 2D-SBP runnable

* remote note

* WARNING to INFO

* refine code for review & fix for new dtype infer

* add note

* Hotfix ib qp max_send_wr

* Pipeline Debug: hack copy hd/commnet regst num to 2 and increase buffer regst num +1

* add buffer op for copyD2H pipe

* fix bug of insert buffer in backward pass stages

* rollback buffer hack

* buffer size = total stage num

* Hack buffer size of loader data/label

* Double buffer

* fix bug of check

* hack buffer size

* Fix buffer connected bug. and refine buffer size

* collection reserve for saving rehash cost

* refine code and note

* set dst buffer size to max

* refine dst buffer size

* Refine Backward buffer size

Co-authored-by: liujuncheng <liujuncheng1022@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* dont use import to determine if export onnx

* fix

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Pipeline Parallelism: checkpointing insert identity buffer op

* fix complier err

* identity buffer op custom out regst num

* fix bug and runnable

* Chain merge divide fw/bw; MemChain ignore merge; copyhd regst num hack

* Pipeline buffer pass

* Pipeline runnable

* rollback NOT merge mem chain hack

* pipeline_stage_id_hint and rollback checkpointing buffer

* Pipeline buffer only. test pass.

* rollback repeat hack

* Remove CopyHd Hack; Add buffer cross label loader and loss

* InsertNcclLogicalOps support multi-subgraph by placement group

* Implement of InsertNcclLogicalOpPass support multi-subgraph and batch acc and pipeline parallel

* Pipeline + 2D-SBP runnable

* remote note

* WARNING to INFO

* refine code for review & fix for new dtype infer

* add note

* collection reserve for saving rehash cost

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add 163 centos mirror

* fix cuda version

* update dockerfile

* add gdb

* refine

* refine

* add one api mkl path

* refine

* refine path

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add logsoftmax module

* add sigmoid module

* add maskedfill module

* format

* add unsquuze module

* add eq module

* add arange module

* fix arange module

* add softmax module

* refine

* update logsoftmax module

* update eq module

* update eq module

* refine

* refine according to comments

* refine softmax test case

* fix unsqueeze module

* refine according to comments

* refine

* format

* format

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add modules:

- sum
- mean
- add
- sub
- mul
- div
- reciprocal

* add +, -, *, / for tensor

* add test case for add,div,mean,mul,reciprocal,sub,sum

* refine test cases

* refine sum module

* remove useless oneflow_export for mean and sum

* add tensor op test for mean and sum

* add CrossEntropyLoss module

* refine Sum

* sum axis attr not found

* refine sum

* refine math ops

* refien cross entropy loss

* refine (not finished)

* fix scalar div, sub

* rename test Module name and refine test case

* rename test Module name and refine test case

* add neg method for Tensor

* remove useless improt items

* refine

* reformat

* remove useless dynamic attr

* fix math_ops and loss

* rename test

* fix format

* delete name

* refine codes and format

* style(*): format codes

* fix(MathOp): fix merge bug

Co-authored-by: simonJJJ <821898965@qq.com>
Co-authored-by: wyg1997 <wyg19970408@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add exp_tanh_gelu module

* fix comment

* fix comment

* fix comment

* change docs location

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>