* replace ForeignJobInstance using JobInstance

* LazyJobStreamType

* NNGraphIf

* NNGraph -> NNGraphIf

* fix compile bugs

* add unit tests for instruction RunLazyJob

* GetInputBufferName/GetOutputBufferName

* multi-client push pull

* refactor wait_and_send_ids_kernel to support multi-client mode

* support multi-client mode return_op

Co-authored-by: liufengwei <2472937968@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* core: GlobalMultiClientEnv; refine EagerExecution

* move IsMultiClient to EnvDesc

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Multi-Client LogicalRun degenerate to PhysicalRun

* Add Logical->Physical in Multi-Client for IdGenerator and InstructionsBuilder

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Modified the OutputArgModifyFn interface

* maybe error stack from CheckAndConstructOp to OutputArgModifier callback function

* maybe error stack from CheckAndConstructOp to OutputArgModifier callback function

* OutputArgModifier return maybe part_1

* maybe error stack from CheckAndConstructOp to OutputArgModifier callback function

* add JUST for hander in ForEachOperator

Co-authored-by: aishangjj <702572275@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* replace ForeignJobInstance using JobInstance

* LazyJobStreamType

* NNGraphIf

* NNGraph -> NNGraphIf

* fix compile bugs

* add unit tests for instruction RunLazyJob

* GetInputBufferName/GetOutputBufferName

* address pr comments

* add cuda complie switch

* refine

Co-authored-by: liufengwei <2472937968@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: clackhan <han_binbin@163.com>

* support inplace forward

* support inplace backward

* add test case

* add test case for clone

* inplace is not support for leaf nodes

* refine clone

* add checks

* refine

* forbid clone with no grad

* Separate autograd meta to tensor (#5267)

* separate autograd meta

* minor fix

* fix acc_grad interface

* fix acc_grad with null

* minor fix

* inplace without clone

* refine

* minor fix

* remove maybe from constructor

* change from create to set

* fix merge bugs

* fix merge bug

* remove inplace flag in local_call_opkernel_phy_instr_operand

* remove out-date codes

* refine code

* add JUST

* fix merge master bug

* revert autograd engine input_grad check

* fix bug in tensor_hook

Co-authored-by: wyg1997 <wyg19970408@gmail.com>
Co-authored-by: Houjiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add random generator

* reformat

* refactor: allow auto generator

* refactor: remove kAUTO, update python api and test

* refactor: use member initializer lists, fix build issue when cpu only

* handle exeception given invalid device

* add dropout functor; add OpExprInterpContext; refactor random_mask_like based on random_generator

* refactor random generator

* disable generator's copyandmove constructor

* reformat

* fix bad merge

* refine

* fix cpu only build

* auto format by CI

* refactor

* use global generator when no generator specified in functional api

* refine

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Houjiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* update cmake

* check in fix

* add default for apple

* skip and generate_api for in mac ci

* fix yml

* fix has_grad template

* update flatbuffer

* fix align

* revert

* revert

* refine

* add FLATBUFFERS_BUILD_TESTS=OFF

* revert

* refine

* add note

* Update flatbuffers.cmake

* refine

* rm log

* FLATBUFFERS_BUILD_TESTS=OFF

* Update flatbuffers.cmake

* restore

* refine

* fix ndarry std::min

* restore

* refine

* update test config

* run doctest in cpu

* fix vm::TestResourceDescScope

* dont run doctest because it seems to involve more

* add simple ci

* refine

* refine

* add ubuntu

* refine

* refine

* add todo

* refine

* refine cache

* build oneflow if linux

* refine

* refine

* fix cache order

* refine cache key

* refine

* rm skip

* add instruction on readme

* refine

* refine

* revert

* refine

* refine

* refine

* Update CMakeLists.txt

* add

* Fix protobuf debug postfix

* refine

* fix

* refine

* refine

* refine

* refine

* add config in matrix

* refine

* refine

* refine

* fix

* add badge

* refine

* refine

* refine

* opencv diable intel ITT

* rm RelWithDebInfo

* refine

* refine

* rm intel ITT

* dont use realpath to prevent failure

* refine

* fix

* don't build Ninja with Release to reduce product

* revert

* refine order

* refine order

* refine order

* refine order

* less deps

* refine

* rm mkl note

* refine

* refine

* link py libs if building shared lib on mac

* refine

* add handy name

* refine

* refine

* refine

* refine

* refine

* refine

* quick fix

* refine

* refine

* refine

* refine

Co-authored-by: strint <xiaoyulink@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: liujuncheng <liujuncheng1022@gmail.com>

* Make the blob header memory aligned.

* Aligned to 64 bytes.

* Refine

* Rename kAlignSize

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix has_grad template

* update flatbuffer

* fix align

* revert

* add FLATBUFFERS_BUILD_TESTS=OFF

* revert

* refine

* add note

* Update flatbuffers.cmake

* FLATBUFFERS_BUILD_TESTS=OFF

* Update flatbuffers.cmake

* update test config

* run doctest in cpu

* fix vm::TestResourceDescScope

* dont run doctest because it seems to involve more

* refine

* fix cpu only errors

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix bin dir

* fix

* fix

* rm transport test main (#5215)

* refine

* refine

* refine

Co-authored-by: strint <xiaoyulink@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: daquexian <daquexian566@gmail.com>

* refactor SbpXXX to cfg::SbpXXX

* modify ParallelDistributionHint4InputArgNameAndIndex to be const function

* fix sbp to cfg::sbp in job_pass

* fix bug ToProto, InitFromProto and pb passed to cfg

* auto format by CI

* fix gpt segment fault

* fix xla

* tmp commit

* tmp commit

* fix xla compile error

* [fix bug] return tmp in model_io_v2

* auto format by CI

Co-authored-by: lixinqi <lixinqi0703106@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: poohRui <yuruil@qq.com>

* fix data race about composed attr map

Signed-off-by: daquexian <daquexian566@gmail.com>

* move ResetPrior before ChooseOpKernel

Signed-off-by: daquexian <daquexian566@gmail.com>

* delete vm before others

Signed-off-by: daquexian <daquexian566@gmail.com>

* revert deletion order change, sync by atexit

Signed-off-by: daquexian <daquexian566@gmail.com>

* add comments

Signed-off-by: daquexian <daquexian566@gmail.com>

* rename

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix multi machine bug

Signed-off-by: daquexian <daquexian566@gmail.com>

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* add tensor.zeros_() and soft sync stream instr

Signed-off-by: daquexian <daquexian566@gmail.com>

* separate cpu and gpu version of SoftSyncStream

* Remove SyncAutoMemset

* fix compile error

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix wrong parallel_desc()

Signed-off-by: daquexian <daquexian566@gmail.com>

* remove unused code

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* try to merge eager ofrecord to master branch

* refine

* temp fix

* try to add seed but fails

* try to add seed but failsclear

* use global function to init mirror/conssitent flag

* fix test

* add modules

* fix record modules

* fix destruction order

* fix mirror gen seed

* skip record unit test

* remove TODO

Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: mosout <mosout@qq.com>
Co-authored-by: Ldpe2G <liangdepeng@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Disable infer instruction if instruction type has physical operand

Signed-off-by: daquexian <daquexian566@gmail.com>

* remove more infer instructions

Signed-off-by: daquexian <daquexian566@gmail.com>

* raise UNIMPLEMENTED() in infer

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix hanging on exit

Signed-off-by: daquexian <daquexian566@gmail.com>

* reformat

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix typo

Signed-off-by: daquexian <daquexian566@gmail.com>

* wrap results by Tensor() in .to()

Signed-off-by: daquexian <daquexian566@gmail.com>

* set need_check_mem_case to false for copy op

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add class AsyncCudaStreamType

* fix bug

* remove useless headfile

Co-authored-by: lixinqi <lixinqi0703106@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* copy user op

* add to module and tensor.to interface

* remove unnecessary code

* backward for tensor.to

* remove capture of input

* support cpu only tensor

* module to (#4858)

* remove backward kernel and op

* friendly deal with when tensor.grad is None

* minor fix

* minor fix

* revert

* suport 1m1d only

* skip test normalization

* skip test normalization

* skip conv

* support construct device using string

* minor fix

* minor fix

* use maybe

* fix device id type for device infer ctx

* skip batchnorm

* skip some tensor test case

Co-authored-by: Xiaoyu Xu <xiaoyulink@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Device::compute_dep_object_

* sequantialize instructions in the same stream.

* adjust atexit sort

Co-authored-by: Houjiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: clackhan <han_binbin@163.com>

* use less event records

* more comments

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* infer shape in main thread and fix multi-thread bug

Signed-off-by: daquexian <daquexian566@gmail.com>

* void -> Maybe<void>

Signed-off-by: daquexian <daquexian566@gmail.com>

* rename op_infer_ctx_1/2

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix comments

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix typo

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* device_infer_fn

* Device::local_call_instruction_name

* implement UserOpExprDeviceInferContext

* merge master

* refactor UserOpConfWrapper

* fix a ci bug

* Refine (#4825)

* fix segmentation fault bug

Co-authored-by: Houjiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: clackhan <han_binbin@163.com>

* ArgTuple

* refactor BlobRegister::GetObject4BlobName

* refactor the type of ArgTuple::arg_name2bn_index2tensor_tuple_index_

* Refactor

* Draft

* Refactor

* Add MutableCfgAttrValueMap

* implement AttrValueMap and ComposedAttrValueMap (#4767)

* Attr value util (#4773)

* implement AttrValueMap and ComposedAttrValueMap

* AttrValueUtil::ToProtoAttrValue

* Fix compile

* Fix compilation

* Rename AttrValueMap by AttrMap.

Co-authored-by: lixinqi <lixinqi0703106@163.com>
Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Support dynamic attrs in local kernel (not tested)

* Support updating UserOpConfWrapper by make methods virtual

* Update attrs in instructions instead of main thread

* call clear_attr()

* Set dynamic attrs based on a based op_conf

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Rename namespace eager to vm

* Merge the same namespace

* Fix conflict

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* mark todo

* inline _Run to PhysicalRun and Logical

* construct instruction_list's shared_ptr and input nullptr to id_generator

* non-allocated space construction function

* modify eager_symbol_list

* modify vm::InstructionMsgList*

* change std::shared_ptr<vm::InstructionMsgList> to vm::InstructionMsgList*

* del istr_list and symbol_list in session

* optimize

* minor fix

* use InstructionsBuilder*

* fix eager run bug

* optimize

* fix bug with update master

* fix bug

* make of_format

Co-authored-by: wanghongsheng <2496533749@qq.com>
Co-authored-by: clackhan <han_binbin@163.com>

* release tensor by instructions, update shut_down_util, skip tensor releasing when exiting

Signed-off-by: daquexian <daquexian566@gmail.com>

* Captures shared_ptr instread of raw pointer

Co-authored-by: lixinqi <lixinqi0703106@163.com>

* PhyInstrOperand

* CHECK_NOTNULL

* LocalCallOpKernelUtil

* implement LocalCallOpKernelUtil

* fix WithOpInferContext/WithComputeInferContext

* fix tensor->blob_object() to tensor->eager_blob_object()

* init commit

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* init more ctx in constructor

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* test

Signed-off-by: daquexian <daquexian566@gmail.com>

* set parallel_desc according to scope

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* LogicalRun -> PhysicalRun

* refine stateful op kernel

* refine

* refine

* build eager blob object list before calling builder, rename TensorsPtr

Signed-off-by: daquexian <daquexian566@gmail.com>

* Fix device CHECK_EQ

* upda...

* add instruction to release tensor

* add detail implementation for cpu and cuda instruction

* minor fix

* fix cpu compile error

* minor fix

Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* PhyInstrOperand

* CHECK_NOTNULL

* LocalCallOpKernelUtil

* implement LocalCallOpKernelUtil

* fix WithOpInferContext/WithComputeInferContext

* fix tensor->blob_object() to tensor->eager_blob_object()

* init commit

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* init more ctx in constructor

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* test

Signed-off-by: daquexian <daquexian566@gmail.com>

* set parallel_desc according to scope

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* LogicalRun -> PhysicalRun

* refine stateful op kernel

* refine

* refine

* build eager blob object list before calling builder, rename TensorsPtr

Signed-off-by: daquexian <daquexian566@gmail.com>

* Fix device CHECK_EQ

* update

* update

Signed-off-by: daquexian <daquexian566@gmail.com>

* update

* refine

* code style updates

* add const quantifiers

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix comments

* update tests

Signed-off-by: daquexian <daquexian566@gmail.com>

* revert api/python/symbol/placement_symbol.cpp

Signed-off-by: daquexian <daquexian566@gmail.com>

* update ForEachOutputTensor, replace auto with const auto&

Signed-off-by: daquexian <daquexian566@gmail.com>

* add local dep objects in local opkernel

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: lixinqi <lixinqi0703106@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Remove RtBlobDesc

* refine code for RuntimeBlobShapeInferHelper::BlobDesc4BnInOp

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add stateful opkernel

Signed-off-by: daquexian <daquexian566@gmail.com>

* set need_check_mem_case_ to false, avoid copy for GetIndex

* address comments, remove job_desc

Signed-off-by: daquexian <daquexian566@gmail.com>

* reuse codes by ZeroCopyBaseContext

* rename files and variables

* Fix comments

* remove unused headers

* Fix TryInitOpKernelState bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add write blob callback intr

* minor fix

* add ofblob arg phy instr operand

* add eager blob object interface, modified instruction

* minor fix

* value copy to reference copy

* fix typo

* add instruction dependence

* minor fix

* modified interface

* minor fix

* remove unnecessary code

* refactor PhyInstrOperand

* using enum string rather than bool

* using new phy instr interface

* using access rather than write and read

* minor fix

* user emplace back rather than push

Co-authored-by: lixinqi <lixinqi0703106@163.com>
Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* replace pushkv with new Transport in transport_blob_instruction

* add Global<EnvGlobalObjectsScope>::Delete() in worker process

* fix GetDefaultResource bug

* use GlobalProcessCtx::NodeSize()

* add sys_ptrace

* fix dead lock bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: jackalcooper <jackalcooper@gmail.com>

* Remove Pod and header in BlobDesc

* mv make_shared to cpp; fix bug of flow2onnx use blob proto

* fix Blob::CopyHeaderFrom check

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>

* allow more streams using scheduling thread

* reset last error to cudaSuccess if it's a tolerable error

* enable sharing shape between BlobDescs

* add class TensorStorage

* add missed #include <functional>

* rename releaser_ to releaser_hook_

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add sequantial callback instruction

* add a test_case for sequential instruction type

* refactor RunLogicalInstruction/RunPhysicalInstruction

* refactor RunLogicalInstruction/RunPhysicalInstruction