* Multi-Client LogicalRun degenerate to PhysicalRun

* Add Logical->Physical in Multi-Client for IdGenerator and InstructionsBuilder

* multi client test

Signed-off-by: daquexian <daquexian566@gmail.com>

* auto format by CI

* fix wrong std::shared_ptr

Signed-off-by: daquexian <daquexian566@gmail.com>

* run module api test by test_multi_client/generic_test.sh

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix test script

Signed-off-by: daquexian <daquexian566@gmail.com>

* use oneflow.distributed.launch -m unittest discover instead of bash for loop to speed up the test

Signed-off-by: daquexian <daquexian566@gmail.com>

* update ci/test_multi_client/generic_test.sh

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: chengtbf <472491134@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* mkdir compatible/single_client

* mv to compatible/singe_client/python

* mv to ./python/compatible/

* use oneflow.compatible.single_client

* back up

* backup

* refactor oneflow.xxx to oneflow.compatible.single_client.xxx

* mv to single_client

* refactor import oneflow to oneflow.compatible.single_client

* backup

* refine

* refine

* make single_clint ci test copy from correct path

* refine

* refine

* delete ioconf and split single-clint ci test

* auto format by CI

* delete default session

* fix method of erase sess id

* delete string multi client in test.yml

* fix spell error

* delete space

* refine

* mv single client python file to compatible_single_client_python

* refine

* fix ci error

* fix bug

* fix bug

* refine

* refine

* delete skip if in test/(modules/tensor)/* because of default eager env

* fix bug

* synchronization oneflow/compatible_single_client and oneflow/python

* fix distributed_run bug

* fix test_multi_process dead lock

* f...

* feat(xxxLikeOp): ones_like and zeros_like use user op

* fix(Optim): fix learning rate device error bug

* style(*): format codes

* style(*): use int instead of np.int

* test(Optim): add optimizer gpu test (#4908)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(Optim): support custom parameters for optimizer

* feat(Adam): adam support custom parameters

* feat(Adamw): adamw support custom parameters

* feat(RMSprop): rmsprop support custom parameters

* style(Optim): refine adam and adamw

Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>

* add experimental apis

* merge master fix conflict

* revert flow._oneflow_internal.dtype to flow.dtype

* refine

* fix test optimizer

* update module docs

* fix unit tests

* fix matmul module test

* fix adamw and rmsprop tests

* fix crossentropy loss grad

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refactor(Optim): refine optimizer codes

* docs(SGD): add document for SGD

* docs(SGD): fix code

* test(Adam): fix test_optim_adam bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(Optim): add sgd

* fix(*): add no_grad guard in optimizer

* fix(*): fix bug in optimizer test

* style(*): format codes

* style(Optimizer): refine codes

* style(Optimizer): refine codes

* fix(Optimizer): fix bug in ParamGroup

* fix(*): add __init__.py

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>