* export functors of tan, erfc, scatterNd

* refine log1p, erfc, tan, scatter_nd

* rename log1p method

* Refactor functional matmul and add apis.

* Export batch matmul and fix python module

* Check inplace valid in C++

* Support scalar add tensor.

* Support inplace when broadcasting add

* Refactor functional sub.

* dev scatter in cpp

* add docstr for flow.F.scatter

* refine scatter signature

* add check type

* refine check

* add PyClamp

* refine scalar

* remove scalar reference

* add flow.F.pow in cpp

* Dev refine scatter py2cpp (#5776)

* Adjust the python functions order.

* Refine

* fix PyScalar(max) error

Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* replace ConsistentTensorInferResult::input_parallel_distribution_ with ConsistentTensorInferResult::input_tensor_metas_

* fix compiler complains

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* skip dataloader test

* add test but no assure acc

* break after test few batches

* refine

* auto format by CI

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* better repr of nn.Graph for debug

* add sbp placement repr

* fix

* restore compile return

* repr block io

* trac module input output to repr

* add graph build repr

* auto format by CI

* fix for None return value

* Update lazy_op_interpreter.cpp

* skip dataloader test

* add test but no assure acc

* break after test few batches

* refine

* auto format by CI

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: cheng cheng <472491134@qq.com>
Co-authored-by: Flowingsun007 <flowingsun007@163.com>

* add first batch of nn functional methods

* fix CLA

* remove #doctest

Co-authored-by: tsai <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add concat

* rm scatter add functor in gather files

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix bugs in shareing EagerBlobObject::blob_desc_.shape and EagerBlobObject::blob_.shape

* feat(EmptyOp): add flow.empty

* docs(EmptyOp): add doctest and refine document

* docs(EmptyOp): refine document

* refactor(Tensor): Tensor constructor use empty_op

* refactor(Tensor): remove useless code

* feat(EmptyOp): support construct in given device and add
               consistent_empty op

* feat(EmptyOp): support unpacked tuple shape

* refine array functor code

* docs(EmptyOp): update empty op document

* refine code

* docs(EmptyOp): add test and document for consistent empty op

* update document

* fix merge bugs

* fix(*): fix infer distribution

* test(EmptyOp): fix ConsistentEmptyOp CPU_ONLY test bug

* fix(*): init shape when InitBlob

* fix(*): Constant and Empty Op use broadcast sbp

* fix(indexing): replace MakeTensor with functional::Empty

* fix(*): fix compile bug

* refine code

* fix(nnGraph): make eager tensor

* auto format by CI

* fix(Stride): infer stride before initializing shape

Co-authored-by: Xinqi Li <lixinqi0703106@163.com>
Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* change flip_code to scalar

* add print info

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add script

* refein

* refien

* refein

* refein

* skip dataloader test

* add test but no assure acc

Co-authored-by: Flowingsun007 <flowingsun007@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bug

* fix bn ci eroor

Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Delete assingn_op.py

* delete categorical_ordinal_encode_op.py constant_op.py

* Delete  get_variable.py

* auto format by CI

* Delete   import

* add get_variable.py

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* support_tensor_to/to_local

* export consistent_tensor.to_local()

* refine code

* export tensor.to()...

* refine code

* refine code

* optimize code

* refine code

* refine

* back up

* add tensor.to func

* make of_format

* remove to in pyTensor

* sync gpu data

* refine

* refine

* refine

* refine

* refine

* refine

* refine

* refine

* refine

* backup

* refine

* rebase

* check in gen py

* merge master and fix bugs

* address pr comments

* eager boxing

* address pr comments

* fix b2p error

* auto format by CI

* remove boxing

* export sbp

* add tensor to_consistent

* /minor fix

* minor fix

* refine

* remove useless head file

* Fix optional

* remove to in tensor.cpp

* update

* Support symbol placement type in functional.

* add sbp and sbp list arg

* refine

* use functional

* refactor CastConsistentOpExpr

* to_consistent(flow.B) backward

* Cache op expr

* add EagerNcclOpKernelState

* refine

* refine

* refine

* refine

* refine

* refine

* minor fix

* capture OpInterpContext

* unimplemented apply

* add GetNdSbp

* add mutex

* refine

* merge EagerConsistentTensorImpl::NewWithPhyTensor and EagerConsistentTensorImpl::NewWithoutPhyTensor into EagerConsistentTensorImpl::New

* rename functiona SyncData to SyncMetaAndData

* fix function yml

* refine

* refine

* refine collective boxing

* make of_format

* of_format

* add to_local to pybind

* refactor EagerBoxingInterpreter

* minor fix

* optimize CastParallelDistribution

* add placement_sbp_util

* minor fix

* eager boxing backward

* minor fix

* sync shape and data when tensor_to_local

* fix rpc_token bugs

* fix p2s backward bug

* refactor AsyncRpcCtx

* set logical_shape correctly

* simplify implementation of consistent_tensor.to_local

* refine

* initialize rpc_token with zero

* refactor grad functions of to_consistent/to_local

* refine

* reformat and address pr comment

* reformat

* add check_meta_consistency in consistent2sonsistent

* refactor eager_nccl_reduce lernel

* refine

* refine to_consistent api

* ban_non_pod_data_in_eager_boxing

* refine

* refine

* refine

* backup code

* THREAD_LOCAL_CACHED

* Delete thread_local_cache.h

* bugfix: DeviceId4ParallelId -> MachineId4ParallelId

* optimize

* minor fix

Co-authored-by: tsai <jackalcooper@gmail.com>
Co-authored-by: Xinqi Li <lixinqi0703106@163.com>
Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Rename CpuStreamType to AsyncCpuStreamType

* Rename CpuStreamType to AsyncCpuStreamType

* backup code

* cpu broadcast

* backup code

* inplace call broadcast when current rank is root

* address pr comments

* refactor  to

* Fix compilation warning for old version gcc (#5757)

Co-authored-by: Houjiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* use default_generator.seed() as random_seed in init

Signed-off-by: daquexian <daquexian566@gmail.com>

* auto format by CI

* fix sampler bug

Signed-off-by: daquexian <daquexian566@gmail.com>

* relax test threshold, add dataloader test data to .gitignore

Signed-off-by: daquexian <daquexian566@gmail.com>

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Fix batch norm if not track_running_stats

* test(BatchNorm2d): add autotest (#5734)

* Fix

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

* rename rpc_token to TransportToken

* minor fix

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Refactor functional matmul and add apis.

* Export batch matmul and fix python module

* Check inplace valid in C++

* Support scalar add tensor.

* Support inplace when broadcasting add

* Refactor functional sub.

* Refactor mul and div

* Refactor functional sub.

* Fix div

* Fix add

* Fix add

* refacotr_maximum_minimum__py2cpp

* refine

* refine

* minor fix

* refactor

* auto format by CI

Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>
Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* comm_net_sequence_number

* remove piece_id

* Remove IsAllowedActor

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* beautify stack frame

* make stack frame python style

* display more word

* refine

* add error type info

* add error type name

* refine and add summary_msg, msg info

* refine

* delete useless

* delete useless

* refine

* refine

* refine

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* pass test on linear with training

* Refactor RuntimeCtx for multi-runtime

* refactor inplace to support nn graph

* block support iterator

* block iter add check

* fix scalar_mul op conf build

* deal with inplace after merge master

* add alexnet graph test

* add cpu test and format

* cout to glog

* deal with Job run finish bug

* refactor lazy deal with inplace

* deal with 0D tensor

* update data path

* address review

* deal with lazy default attr

* mv according to ci

* merge master

* fix for ci

* fix for ci limit

* block proxy func

* support module custom func and refacotr get attr of block

* auto format by CI

Co-authored-by: chengtbf <472491134@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* Add env ONEFLOW_THREAD_LOCAL_MESSAGE_QUEUE_ENABLE

* refine GetGlobalWorkStreamId

* refine name

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Refactor functional matmul and add apis.

* Export batch matmul and fix python module

* Check inplace valid in C++

* Support scalar add tensor.

* Support inplace when broadcasting add

* Refactor mul and div

* Refactor functional sub.

* Fix div

* Fix add

* Fix add

Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* LazyInterpret::ApplyImpl for SourceUserOp and Copy

* Fix bug of CropMirrorNormalize ops; output ops check

* add test

* Add test case for tensor.to in nn.Graph

* remove useless code and refine error catch

* refine note

* Support CoinFlip module with consistent and device

* using const&

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Dev logical_and/or modules

* Add docs

* Add autotest

* Add single_client api

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* consistent_tensor_to_api

* add check_meta_consistency

* Update to.py

* refine

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add avgpool 2d module

* pooling cpp

* make format

* fix format

* add avgpool1d but backward still have bug

* fix avgpool1d backward

* add avgpool3d cpu

* add avgpool series GPU kernel

* fix docs

* fix output grad size

* fix path

* fix comment

* fix comment

* fix comment

* remove get sequence

* fix doctest

* remove doctest part

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add changes for multi dev demo

Signed-off-by: daquexian <daquexian566@gmail.com>

* add part of backward hook

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* add naive init_with_env

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* support_multi_client

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* Remove unused code

* Fix multi client launch

* fix __main__ bug

* update abcd op

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix multi client sync, make nccl instr ordered

Signed-off-by: daquexian <daquexian566@gmail.com>

* temp changes

Signed-off-by: daquexian <daquexian566@gmail.com>

* Use functional api instead of op_expr_helper::XXXOp.

* align with latest master, remove unused code

Signed-off-by: daquexian <daquexian566@gmail.com>

* local rank returns 0 when no env var, save is_multi_client in EnvDesc

Signed-off-by: daquexian <daquexian566@gmail.com>

* move is_multi_client to ProcessCtx, rename cuda_d2d device to nccl, remove unused code

Signed-off-by: daquexian <daquexian566@gmail.com>

* abcd -> return_first_input op

Signed-off-by: daquexian <daquexian566@gmail.com>

* remove launch.py for now

Signed-off-by: daquexian <daquexian566@gmail.com>

* refine

Signed-off-by: daquexian <daquexian566@gmail.com>

* update IsMultiClient in env_util.py

Signed-off-by: daquexian <daquexian566@gmail.com>

* rm multi_dev_demo.py

Signed-off-by: daquexian <daquexian566@gmail.com>

* remove exported functions in env_util.py

Signed-off-by: daquexian <daquexian566@gmail.com>

* remove unused op expr helper func

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix bug

Signed-off-by: daquexian <daquexian566@gmail.com>

* add DevVmDepObjectConsumeMode and set it as NONE in backward

Signed-off-by: daquexian <daquexian566@gmail.com>

* move return_first_input op from math_ops.py to tensor_ops.py

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix compile error

Signed-off-by: daquexian <daquexian566@gmail.com>

* refine

Signed-off-by: daquexian <daquexian566@gmail.com>

* add comments

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix exit bug in init.py

Signed-off-by: daquexian <daquexian566@gmail.com>

* align with master

Signed-off-by: daquexian <daquexian566@gmail.com>

* update device ctor

Signed-off-by: daquexian <daquexian566@gmail.com>

* default dev id = local rank % gpu num

Signed-off-by: daquexian <daquexian566@gmail.com>

* assert single machine

Signed-off-by: daquexian <daquexian566@gmail.com>

* reformat

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix consume mode, implement eager_nccl_allreduce by process ranks

Signed-off-by: daquexian <daquexian566@gmail.com>

* fill sorted_ranks field in old code, reformat

Signed-off-by: daquexian <daquexian566@gmail.com>

* set default val for op conf, align with master

Signed-off-by: daquexian <daquexian566@gmail.com>

* impl return_first_input as functional api, impl allreduce as module

Signed-off-by: daquexian <daquexian566@gmail.com>

* add more tests

Signed-off-by: daquexian <daquexian566@gmail.com>

* reformat

Signed-off-by: daquexian <daquexian566@gmail.com>

* align with master

Signed-off-by: daquexian <daquexian566@gmail.com>

* rename ddp to flow.nn.parallel.DistributedDataParallel

Signed-off-by: daquexian <daquexian566@gmail.com>

* refine eager nccl comm

Signed-off-by: daquexian <daquexian566@gmail.com>

* refine eager nccl comm, divide grad by group size

Signed-off-by: daquexian <daquexian566@gmail.com>

* rename reversed_param_list -> ddp_state_for_reversed_params

Signed-off-by: daquexian <daquexian566@gmail.com>

* make return_first_input inplace

Signed-off-by: daquexian <daquexian566@gmail.com>

* restore eager allreduce

Signed-off-by: daquexian <daquexian566@gmail.com>

* add static all zero tensor and select first

Signed-off-by: daquexian <daquexian566@gmail.com>

* refine

Signed-off-by: daquexian <daquexian566@gmail.com>

* add functional allreduce op and use current rank group

Signed-off-by: daquexian <daquexian566@gmail.com>

* meterialize StaticAllZeroTensor in allreduce, support it in scalar mul

Signed-off-by: daquexian <daquexian566@gmail.com>

* materialize static zeros tensor in set_acc_grad

Signed-off-by: daquexian <daquexian566@gmail.com>

* rename

Signed-off-by: daquexian <daquexian566@gmail.com>

* auto format by CI

Co-authored-by: clackhan <han_binbin@163.com>
Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* pass test on linear with training

* Refactor RuntimeCtx for multi-runtime

* refactor inplace to support nn graph

* block support iterator

* block iter add check

* fix scalar_mul op conf build

* deal with inplace after merge master

* add alexnet graph test

* add cpu test and format

* cout to glog

* deal with Job run finish bug

* refactor lazy deal with inplace

* deal with 0D tensor

* update data path

* address review

* deal with lazy default attr

* mv according to ci

* merge master

* fix for ci

* fix for ci limit

Co-authored-by: chengtbf <472491134@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Refactor functional matmul and add apis.

* Export batch matmul and fix python module

* Check inplace valid in C++

* Support scalar add tensor.

* Support inplace when broadcasting add

* Fix add

Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* debug

* construct job plan name as key

* rm debug code

* mod comment

* chang plan name to pubsub format

* correct comment syntax

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Rename CpuStreamType to AsyncCpuStreamType

* Rename CpuStreamType to AsyncCpuStreamType

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>