* improve ofrecord unit test

* remove no_grad

* fix codes according to review

* fix format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add upsample module

* add upsample2d unittest

* add upsample2d unittest

* add docs

* add UpsamplingNearest2d and UpsamplingBilinear2d module

* code format and add docs

* add more unit_test

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* copy user op

* add to module and tensor.to interface

* remove unnecessary code

* backward for tensor.to

* remove capture of input

* support cpu only tensor

* module to (#4858)

* remove backward kernel and op

* friendly deal with when tensor.grad is None

* minor fix

* minor fix

* revert

* suport 1m1d only

* skip test normalization

* skip test normalization

* skip conv

* support construct device using string

* minor fix

* minor fix

* use maybe

* fix device id type for device infer ctx

* skip batchnorm

* skip some tensor test case

* startup of add backward

* startup of add gpu test

* refine

* add cuda test for Linear module

* refine after sum fixed

* gpu backward

* gpu backward crashed

* retain grad

* refine according to comments of WangYinggang

* refine: construct specified device tensor

* refine testcase

* refine: specifiy device when construct in test case

* refien testcase for linear

* refine

* refien to_device

* refine import statement

* refine import path

* remove useless _to_device fun

Co-authored-by: poohRui <yuruil@qq.com>
Co-authored-by: Yurui Li <32978179+poohRui@users.noreply.github.com>
Co-authored-by: Xiaoyu Xu <xiaoyulink@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refactor Tensor

* Export ConsistentTensor::is_cuda

* minor fix

* minor fix

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add hardsigmoid module

* refine docs

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* naive impl

* refine

* refine

* add log

* refine

* refine

* refine

* refine

* add todo

* refine

* refine

* sync dynamic libs

* refine

* fix docker cmd

* fix rank

* refine

* refine

* add callbacks simple rpc

* refine

* refine

* fix

* refine

* refine

* refine

* fix conn

* support tradional mode

* refine

* refine

* refine

* rm

* refine

* refine

* refine

* refine todo

* refine

* refine

* rm unused

* rm todo

* revert

* refine

* add log

* refine

* refine

* fix order

* refine

* refine

* refine

* refine

* refine

* refine

* refine

* rm

* rename

* add comment

* refine

* rm

* refine

* refine

* refine

* refine

* add todo

* add info

* refine

* refine

* refine

* add back some legacy code

* refine

* refine

* refine

* refine

* refine

* rm oneflow_worker exe

* rm log

* fix bug

* support --cmd

* add check

* refine

* fix

* fmt

* add hard swish module

* fix bug

* fix bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* support cast in to

* refactor to interface

* refine doc

* refine doc

* refine kwargs

* add test case support tensor

* minor fix and test case

* format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add leakyrelu module

* code format

* update docs

* update docs

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(LrScheduler): add ConsineScheduler

* feat(LrScheduler): update cosine_scheduler and add test

* feat(LrScheduler): refine codes

* style(*): format codes

* docs(LrScheduler): add document

* docs(LrScheduler): refine documents

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add flow.tensor

* small fix

* fix dtype

* Update oneflow/python/framework/tensor.py

Co-authored-by: daquexian <daquexian566@gmail.com>

* deal with multi-dimension list or tuple

* remove list

* Update oneflow/python/framework/tensor.py

Co-authored-by: daquexian <daquexian566@gmail.com>

* add unit test case

* format

Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add rsqrt moduel

* add batchnorm module cpu support

* refine

* update

* fix param ini

* add reduce series modules

* add batchnorm,layernorm modules and test cases

* refine

* update .rst

* refine according to comments

* refine

* update

* fix layernorm bug

* refine

* remove additional license

Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* try to merge eager ofrecord to master branch

* refine

* temp fix

* try to add seed but fails

* try to add seed but failsclear

* use global function to init mirror/conssitent flag

* fix test

* add modules

* fix record modules

* fix destruction order

* fix mirror gen seed

* skip record unit test

* remove TODO

Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: mosout <mosout@qq.com>
Co-authored-by: Ldpe2G <liangdepeng@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* support expand and repeat op int datatype

* add prelu module

* add prelu module

* add prelu module

* fix comments

* fix comment

* fix comment

* add backward test

* fix comment

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(xxxLikeOp): ones_like and zeros_like use user op

* fix(Optim): fix learning rate device error bug

* style(*): format codes

* style(*): use int instead of np.int

* test(Optim): add optimizer gpu test (#4908)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* skip test_gpt_data_loader in eager mode

* 1_node_fix_egaer_test_bug

* remove useless head file

* skip tensor and module

* skip 2-D sbp in eager mode

* fix error

* fix bug and remove some skip under eager

* fix error

* del oneflow_api

* rm test_tensor.py

* skip test_summary in eager mode

* skip test_stateful_local_kernel under cpu only mode

* add class AsyncCudaStreamType

* fix bug

* import os

* remove BlobObject::is_python_shutting_down_

* fix error

* sikp 2d sbp

* minor fix

* refine comment

* make of_format

Co-authored-by: lixinqi <lixinqi0703106@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* remove useless code in expand module test

* add conv2d docs

* support expand and repeat op int datatype

* fix modules docs bug

* fix docs bug

* fix docstring bug

* fix docs bug

* fix docs bug

* add permute module

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* support create cpu tensor

* add empty op

* remove skip tensor test case:

* remove skip tensor test case

* remove TODO

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add gather module

* add gather module

* add test case

* add embedding module

* fix comments

* update embedding module and test case

* refine

* fix comment

* fix comment

* fix comment

Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* style(Optim): add warning when no param update

* style(Optim): add TODO

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* support expand and repeat op int datatype

* fix modules docs bug

* fix docs bug

* fix docstring bug

* fix docs bug

* fix docs bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix and check to on forwad and backward

* add todo

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Disable infer instruction if instruction type has physical operand

Signed-off-by: daquexian <daquexian566@gmail.com>

* remove more infer instructions

Signed-off-by: daquexian <daquexian566@gmail.com>

* raise UNIMPLEMENTED() in infer

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix hanging on exit

Signed-off-by: daquexian <daquexian566@gmail.com>

* reformat

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix typo

Signed-off-by: daquexian <daquexian566@gmail.com>

* wrap results by Tensor() in .to()

Signed-off-by: daquexian <daquexian566@gmail.com>

* set need_check_mem_case to false for copy op

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add class AsyncCudaStreamType

* fix bug

* remove useless headfile

Co-authored-by: lixinqi <lixinqi0703106@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* support expand and repeat op int datatype

* support conv cpu module

* support conv cpu module

* support conv cpu module

* support conv cpu module

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Add ones like op.

Conflicts:
	oneflow/api/foreign_lock_helper.h
	oneflow/api/python/autograd/autograd.cpp
	oneflow/api/python/framework/tensor.cpp
	oneflow/core/framework/op_interpreter/op_interpreter.cpp
	oneflow/core/framework/op_interpreter/op_interpreter_util.cpp
	oneflow/core/framework/tensor.cpp
	oneflow/core/framework/tensor_impl.cpp
	oneflow/core/framework/tensor_impl.h

* Add ones_like unittest.

* Use SwithCase

* Fix typo

* undef

* Bugfix

* Fix merge conflicits

Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* align transpose module with pytorch

* fix comment

* align tranpose module

* support expand and repeat op int datatype

* fix bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(Optim): support custom parameters for optimizer

* feat(Adam): adam support custom parameters

* feat(Adamw): adamw support custom parameters

* feat(RMSprop): rmsprop support custom parameters

* style(Optim): refine adam and adamw

Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>

* expose local tensor

Signed-off-by: daquexian <daquexian566@gmail.com>

* mt19937 -> minstd_rand

Signed-off-by: daquexian <daquexian566@gmail.com>

* revert unnecessary diff

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix comments

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* support expand and repeat op int datatype

* code format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>