* feat(Optim): support custom parameters for optimizer

* feat(Adam): adam support custom parameters

* feat(Adamw): adamw support custom parameters

* feat(RMSprop): rmsprop support custom parameters

* style(Optim): refine adam and adamw

Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>

* expose local tensor

Signed-off-by: daquexian <daquexian566@gmail.com>

* mt19937 -> minstd_rand

Signed-off-by: daquexian <daquexian566@gmail.com>

* revert unnecessary diff

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix comments

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* support expand and repeat op int datatype

* code format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* align dim size funtion

* fix dim usage

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Fix reduce sum grad func.

* Fix zeros op

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* copy user op

* add to module and tensor.to interface

* remove unnecessary code

* backward for tensor.to

* remove capture of input

* support cpu only tensor

* module to (#4858)

* remove backward kernel and op

* friendly deal with when tensor.grad is None

* minor fix

* minor fix

* revert

* suport 1m1d only

* skip test normalization

* skip test normalization

* skip conv

* support construct device using string

* minor fix

* minor fix

* use maybe

* fix device id type for device infer ctx

* skip batchnorm

* skip some tensor test case

Co-authored-by: Xiaoyu Xu <xiaoyulink@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* check in naive struct

* refine

* refine

* refine

* refine

* add functions

* refine

* refine

* refine

* fmt

* refine

* refine

* refine

* refine

* refine

* refine

* add note

* refine

* refine

* refine

* refine

* refine

* refine

* refine

* refine

* rm include

* revert cmakelist changes

* refine

* address review

* rename

* address review

* address review

* remove glog dependency

* fix

* refine

* refine

* print lib path in stdout

* address review

* address review

* fix

* support ONEFLOW_LIBIBVERBS_PATH

* add case

* update init_cluster_env.py for ONEFLOW_LIBIBVERBS_PATH

* fix comment

* address review

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Device::compute_dep_object_

* sequantialize instructions in the same stream.

* adjust atexit sort

Co-authored-by: Houjiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: clackhan <han_binbin@163.com>

* supoort nllloss 3dim

* supoort nllloss 3dim

* merge conflict

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* support crossentropy loss 3dim

* merge conflict

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix interpreter determin output leaf and grad

* fix GradMode get

* simplify

* add test for no_grad

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* align mean module

* allow negative dim param

* support tuple of negative dim param

* refine

* format

Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix export experimental docs bug

* fix export experimental docs bug

* fix export experimental docs bug

Co-authored-by: Yao Chi <later@usopp.net>

* feat(PySize): support negative index and add test

* style(*): refine code

* format code

* use composed attr map

Signed-off-by: daquexian <daquexian566@gmail.com>

* move implementation to .cpp

Signed-off-by: daquexian <daquexian566@gmail.com>

* OpExpr::New returns Maybe

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix GroupNorm and modify test case

* add grad op for reshape_like op

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix adam weight decay

* fix adam weight decay

* fix comment

* fix comment

* fix commnet

* fix commnet

* fix commnet

* fix bug

* fix(Adam): fix Adam test bug

* revert adam test threshold to 1e-3

* fix(Adam): fix adam test bug and adjust param to increase error

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: wyg1997 <wyg19970408@gmail.com>

* align squeeze module with torch

* fix comment

* fix argmax bug

* fix bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* numpy: create np arr in python and copy in c++, use busy loop to wait

Signed-off-by: daquexian <daquexian566@gmail.com>

* reformat

Signed-off-by: daquexian <daquexian566@gmail.com>

* add CopyBetweenMirroredTensorAndNumpy

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* use less event records

* more comments

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Do not save inputs in function nodes even if requires_grad is true.

* Allocate raw memory with actual size.

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add temp parallel ctx for single card

Signed-off-by: daquexian <daquexian566@gmail.com>

* add TODO comment

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add broadcast_like module

* add where module, still has bug

* fix where module bug

* fix where module bug

* fix bug and add where module

* fix where module commnet

* code format

* fix where module

* code format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* A more efficient implementation of NLL Loss

* A more efficient implementation of NLL Loss

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

remove MakeParallelDescByDevice, fix the missing setting of parallel_desc in InstructionMsg copy constructor (#4850)

Signed-off-by: daquexian <daquexian566@gmail.com>

* add experimental apis

* merge master fix conflict

* revert flow._oneflow_internal.dtype to flow.dtype

* refine

* fix test optimizer

* update module docs

* fix unit tests

* fix matmul module test

* fix adamw and rmsprop tests

* fix crossentropy loss grad

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* init adamw optimizer

* fix adamw optimizer bug

* fix comment

* fix comment

* code format

* fix comment

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add rmsprop optimizer

* fix rmsprop optimizer bug

* fix rmsprop optimizer bug

* add rmsprop optimizer docs

* add rmsprop docs

* fix comment

* fix comment

* fix comment

* fix comment

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add broadcast matmul support

* refine

* add batch matmul support

* remove redundant test case

* linear module support high dimension input

* format

* fix linear

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix eager with unknow symbol id

* minor fix

* fix conflict

* remove unnnecessary function

* remove unnecessary header

* remove unnecessary methods

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refactor(Optim): refine optimizer codes

* docs(SGD): add document for SGD

* docs(SGD): fix code

* test(Adam): fix test_optim_adam bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add scalar input support

* format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Rewrite sparse softmax cross entropy gradient func.

* Rewrite reduce ops gradient funcs.

* Add crossentropyloss grad unittest.

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: hjchen2 <hjchen2>

* infer shape in main thread and fix multi-thread bug

Signed-off-by: daquexian <daquexian566@gmail.com>

* void -> Maybe<void>

Signed-off-by: daquexian <daquexian566@gmail.com>

* rename op_infer_ctx_1/2

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix comments

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix typo

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>