* enhance_local_to_consistent

* refine

* minor fix

* refine

* refine

* minor fix

* refine

* refine

* refine

* fix test_file bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Delete part of the interface

* auto format by CI

* delete comment

* auto format by CI

* delete enable_if.unique(*****)

* pr 5758 combined

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: binbinHan <han_binbin@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add flow.randn

* auto format by CI

* refine

* add mean and std as attr to normal_op/kernel/distribution

* refine

* refine

* keep module, fix docstring to pass CI

* refine

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* support_tensor_to/to_local

* export consistent_tensor.to_local()

* refine code

* export tensor.to()...

* refine code

* refine code

* optimize code

* refine code

* refine

* back up

* add tensor.to func

* make of_format

* remove to in pyTensor

* sync gpu data

* refine

* refine

* refine

* refine

* refine

* refine

* refine

* refine

* refine

* backup

* refine

* rebase

* check in gen py

* merge master and fix bugs

* address pr comments

* eager boxing

* address pr comments

* fix b2p error

* auto format by CI

* remove boxing

* export sbp

* add tensor to_consistent

* /minor fix

* minor fix

* refine

* remove useless head file

* Fix optional

* remove to in tensor.cpp

* update

* Support symbol placement type in functional.

* add sbp and sbp list arg

* refine

* use functional

* refactor CastConsistentOpExpr

* to_consistent(flow.B) backward

* Cache op expr

* add EagerNcclOpKernelState

* refine

* refine

* refine

* refine

* refine

* refine

* minor fix

* capture OpInterpContext

* unimplemented apply

* add GetNdSbp

* add mutex

* refine

* merge EagerConsistentTensorImpl::NewWithPhyTensor and EagerConsistentTensorImpl::NewWithoutPhyTensor into EagerConsistentTensorImpl::New

* rename functiona SyncData to SyncMetaAndData

* fix function yml

* refine

* refine

* refine collective boxing

* make of_format

* of_format

* add to_local to pybind

* refactor EagerBoxingInterpreter

* minor fix

* optimize CastParallelDistribution

* add placement_sbp_util

* minor fix

* eager boxing backward

* minor fix

* sync shape and data when tensor_to_local

* fix rpc_token bugs

* fix p2s backward bug

* refactor AsyncRpcCtx

* set logical_shape correctly

* simplify implementation of consistent_tensor.to_local

* refine

* initialize rpc_token with zero

* refactor grad functions of to_consistent/to_local

* refine

* reformat and address pr comment

* reformat

* add check_meta_consistency in consistent2sonsistent

* refactor eager_nccl_reduce lernel

* refine

* refine to_consistent api

* ban_non_pod_data_in_eager_boxing

* refine

* refine

* refine

* backup code

* THREAD_LOCAL_CACHED

* Delete thread_local_cache.h

* bugfix: DeviceId4ParallelId -> MachineId4ParallelId

* optimize

* use inplace in grad of hierarchical_parallel_cas

Co-authored-by: tsai <jackalcooper@gmail.com>
Co-authored-by: Xinqi Li <lixinqi0703106@163.com>
Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* collective_boxing_executor add_plan

* new delete in env_scope

* CollectiveBoxingExecutorPlanToken

* refine

* refine

* address review

* refine

* tidy code

* minor fix

* auto format by CI

Co-authored-by: leaves-zwx <kunta0932@gmail.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: cheng cheng <472491134@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* avoid input tensor null

* add todo

* auto format by CI

Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Delete part of the interface

* auto format by CI

* delete comment

* delet comment

* auto format by CI

* delete enable_if.unique(***)

* delete get_current_machine_id

* auto format by CI

* Add init empty function

* Add blank line

* Fix the last line

* auto format by CI

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Delete part of the interface

* auto format by CI

* Delete comment

* Delete comment

* delete oneflow.rst advanced

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* init fakequantization op

* add backward op for fake_quantization

* add min_max_observer op and refine fake_quantization

* fix min_max_observer bug

* add moving_average_min_max_observer op

* add test example for min_max_observer

* add min_max_observer test

* add moving_average_min_max_observer op test

* fake quantization has bug

* try to fix zero_point calc bug

* fix test bug

* fix quantize op bug

* add docs for min_max_observer_op

* add docs for moving_average_min_max_observer_op

* refine doctest

* build docs for quantization

* refine doctest

* fix moving_average_min_max_observer op

* fix comments

* add post quantization op

* fix quantization op bug

* add doctest

* fix comments

* fix comments

* auto format by CI

* fix bug

* fix bn ci bug

* fix doc bug

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* gather mean, sum

* abs, acos, acosh, slice

* abs, acosh, argwhere, bmm, chunk, concat, slice

* constant, diag, eq, exp, expand, eye, flip, floor, gathr_nd, >=

* greater, less, less_equal, maksed_fill, masked_select

* asin, asinh, sin, cos, atan

* std and related ops

* matmul, ne, negative, permute, repeat, ceil, expm1

* reshape, round, sign, sinh, squeeze, stack, tile etc.

unsqueeze, where, transpose, triu

* fix argwhere default arg

* fix squeeze x not found bug

* remove Atanh()(input)

* refine modules(bmm, cast, diag, expm1)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample nearest bug

* fix upsample nearest bug (#5347)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bilinear bug

* fix export bug

* add api

* add api

* add api

* fix comments

* fix complie bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Yao Chi <later@usopp.net>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* export functors of tan, erfc, scatterNd

* refine log1p, erfc, tan, scatter_nd

* rename log1p method

* Refactor functional matmul and add apis.

* Export batch matmul and fix python module

* Check inplace valid in C++

* Support scalar add tensor.

* Support inplace when broadcasting add

* Refactor functional sub.

* dev scatter in cpp

* add docstr for flow.F.scatter

* refine scatter signature

* add check type

* refine check

* add PyClamp

* refine scalar

* remove scalar reference

* add flow.F.pow in cpp

* Dev refine scatter py2cpp (#5776)

* Adjust the python functions order.

* Refine

* fix PyScalar(max) error

Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* replace ConsistentTensorInferResult::input_parallel_distribution_ with ConsistentTensorInferResult::input_tensor_metas_

* fix compiler complains

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* skip dataloader test

* add test but no assure acc

* break after test few batches

* refine

* auto format by CI

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* better repr of nn.Graph for debug

* add sbp placement repr

* fix

* restore compile return

* repr block io

* trac module input output to repr

* add graph build repr

* auto format by CI

* fix for None return value

* Update lazy_op_interpreter.cpp

* skip dataloader test

* add test but no assure acc

* break after test few batches

* refine

* auto format by CI

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: cheng cheng <472491134@qq.com>
Co-authored-by: Flowingsun007 <flowingsun007@163.com>

* add first batch of nn functional methods

* fix CLA

* remove #doctest

Co-authored-by: tsai <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add concat

* rm scatter add functor in gather files

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix bugs in shareing EagerBlobObject::blob_desc_.shape and EagerBlobObject::blob_.shape

* feat(EmptyOp): add flow.empty

* docs(EmptyOp): add doctest and refine document

* docs(EmptyOp): refine document

* refactor(Tensor): Tensor constructor use empty_op

* refactor(Tensor): remove useless code

* feat(EmptyOp): support construct in given device and add
               consistent_empty op

* feat(EmptyOp): support unpacked tuple shape

* refine array functor code

* docs(EmptyOp): update empty op document

* refine code

* docs(EmptyOp): add test and document for consistent empty op

* update document

* fix merge bugs

* fix(*): fix infer distribution

* test(EmptyOp): fix ConsistentEmptyOp CPU_ONLY test bug

* fix(*): init shape when InitBlob

* fix(*): Constant and Empty Op use broadcast sbp

* fix(indexing): replace MakeTensor with functional::Empty

* fix(*): fix compile bug

* refine code

* fix(nnGraph): make eager tensor

* auto format by CI

* fix(Stride): infer stride before initializing shape

Co-authored-by: Xinqi Li <lixinqi0703106@163.com>
Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* change flip_code to scalar

* add print info

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add script

* refein

* refien

* refein

* refein

* skip dataloader test

* add test but no assure acc

Co-authored-by: Flowingsun007 <flowingsun007@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bug

* fix bn ci eroor

Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>