Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add test on add input to graph

* add var into graph

* LazyInterpreter for FetchOutputOpExpr and set op parallel_distribution

* refine input var build

* split file

* rename

* mini refine

* add complete_graph_config

* add sgd

* fix style

* Add note

* LazyInterpret::ApplyImpl for UserOpExpr

* refine test scripts

* add output to graph

* format

* tensor.backward to lazy_add_loss(tensor)

* Fix bug of LazyInterpret UserOpExpr for change output lbns

* Add test user op expr test

* fix note mistake

* add userop and test

* rename

* add idea on l2 to variable op

* add scale in sgd

* refactor add loss

* revert change on tensor_name_scope

* fix typo

* check is lazy in tensor.backward

* rm useless state_tensortuple in nn.Graph

Co-authored-by: chengtbf <472491134@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* added flatten backward

* flatten and softmax backward

* fix bug for not supporting all kinds of iterables in optimizers

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refactor(ModelUpdate): SGDUpdate and MomentumUpdate use optional input
                       for learning rate

* fix(*): fix bugs

* style(*): refine code

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(xxxLikeOp): ones_like and zeros_like use user op

* fix(Optim): fix learning rate device error bug

* style(*): format codes

* style(*): use int instead of np.int

* test(Optim): add optimizer gpu test (#4908)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(Optim): support custom parameters for optimizer

* feat(Adam): adam support custom parameters

* feat(Adamw): adamw support custom parameters

* feat(RMSprop): rmsprop support custom parameters

* style(Optim): refine adam and adamw

Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>

* add experimental apis

* merge master fix conflict

* revert flow._oneflow_internal.dtype to flow.dtype

* refine

* fix test optimizer

* update module docs

* fix unit tests

* fix matmul module test

* fix adamw and rmsprop tests

* fix crossentropy loss grad

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refactor(Optim): refine optimizer codes

* docs(SGD): add document for SGD

* docs(SGD): fix code

* test(Adam): fix test_optim_adam bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(Optim): add sgd

* add adam optimizer

* add adam optimizer

* add adam optimize module

* add adam optimizer module

* fix adam optimizer module bug

* add adam optimizer

* file format

* add adam optimizer docs

* fix comment

* fix comment

* style(*): format codes

Co-authored-by: wyg1997 <wyg19970408@gmail.com>

* feat(Optim): add sgd

* fix(*): add no_grad guard in optimizer

* fix(*): fix bug in optimizer test

* style(*): format codes

* style(Optimizer): refine codes

* style(Optimizer): refine codes

* fix(Optimizer): fix bug in ParamGroup

* fix(*): add __init__.py

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>