* Test 'nn.Linear' and 'nn.Identity' with random data

* Test 'nn.PixelShuffle' with random data

* Test 'greater' and 'less' with random data

* Test 'flow.Tensor.flatten' with random data

* Test 'greater' and 'less' with random-sized data

* Test 'nn.CrossEntropyLoss' with random data

* Replace multiple return entries with the 'oneof' generator

* Add 'nothing()' test cases for parameters with default values

* Remove the hard-coded test cases of `nn.CrossEntropyLoss`

* auto format by CI

* Skip the test of 'nn.CrossEntropyLoss' temporarily

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* fix-abs

* fmod-init

* broadcastfloormod_back

* vstorch

* modify_test

* fix_doctest_flag

* add-fmod-kernel

* fix-marco4binaryop

* split-marco

* fix-format

* add-autotest

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* LazyInterpret build LocalTensor if input is local

* refine graph default scope

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refine

* allow post result to fail

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample nearest bug

* fix upsample nearest bug (#5347)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bilinear bug

* add new autotest case

* add more tests

* fix bug

* auto format by CI

* add rtol and atol

* add rtol and atol

* add nothing

* auto format by CI

* fix export bug

* fix bug

* fix bug

* fix bug

* fix bug

* auto format by CI

* fix bug

* fix bug

* fix ci error

* fix bug

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* refactor job_pass by maybe_system

* refactor AutoSourceAndSinkTick to SingleClientAutoSourceAndSinkTick

* remove useless files

* add job_pass MultiClientAutoSourceAndSinkTick

* address review

* fix check

* fix OpConf4OpName return

Co-authored-by: leaves-zwx <kunta0932@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add test on add input to graph

* add var into graph

* LazyInterpreter for FetchOutputOpExpr and set op parallel_distribution

* refine input var build

* split file

* rename

* mini refine

* add complete_graph_config

* add sgd

* fix style

* Add note

* LazyInterpret::ApplyImpl for UserOpExpr

* refine test scripts

* add output to graph

* format

* tensor.backward to lazy_add_loss(tensor)

* Fix bug of LazyInterpret UserOpExpr for change output lbns

* Add test user op expr test

* fix note mistake

* add userop and test

* rename

* add idea on l2 to variable op

* add scale in sgd

* refactor add loss

* revert change on tensor_name_scope

* fix typo

* check is lazy in tensor.backward

* rm useless state_tensortuple in nn.Graph

Co-authored-by: chengtbf <472491134@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* NNGraph interface and implement for CompileAndRuntime

* fix compile

* RunLazyNNGraph

* New/CloseRuntimeBuffers and RunLazyJob impl

* using job_conf.concurrency_width for Source/Sink BufferSize

* Add note for buffer size

* NNGraph interface and implement for CompileAndRuntime

* fix compile

* RunLazyNNGraph

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* prevent upload logs

* refine

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Generalize 'AdaptiveAvgPool' for 1~3 dims usage

* Implement GPU version

* Enable pooling with larger 'output_size' which is supported by PyTorch

* Add docs for 'AdaptiveAvgPool1d' and 'AdaptiveAvgPool3d'

* Add functional API for 'AdaptiveAvgPoolXd'

* Add 'flow.adaptive_avg_poolxd'

* Add test cases for 'flow.adaptive_avg_poolxd'

* auto format by CI

* Avoid using 'Shape::Count' in for loop

* Change names of compute functions

* Change 'AdaptivePoolNdGradOp' to functor

* Register integer types for CUDA 'adaptive_avg_poolxd' kernel

* Integrate 'adaptive_avg_poolxd' into 'nn.functional.interpolate'

* Correct wrong 'AdaptiveAvgPoolXd' entries

* Add missing 'not None' assertions for 'output_size'

* Support tuple input for 'AdaptiveAvgPool1d'

* Mark TODO for auto testing 'AdaptiveAvgPoolXd'

* Denote return types for 'BackwardOpConfGenFn'

* Combine test classes of 'AdaptiveAvgPoolXd' into one

* Rename 'AvgXXXCompute'

* auto format by CI

* Simplify output shape inference

* Delete the wrong 'if' block

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* update automated test framework

Signed-off-by: daquexian <daquexian566@gmail.com>

* rename func -> generator

Signed-off-by: daquexian <daquexian566@gmail.com>

* reformat

Signed-off-by: daquexian <daquexian566@gmail.com>

* update more powerful conv test

Signed-off-by: daquexian <daquexian566@gmail.com>

* new imperative autotest api

Signed-off-by: daquexian <daquexian566@gmail.com>

* update automated test framework

Signed-off-by: daquexian <daquexian566@gmail.com>

* add some tests and refine code

Signed-off-by: daquexian <daquexian566@gmail.com>

* update test_matmul.py

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix test_matmul.py

Signed-off-by: daquexian <daquexian566@gmail.com>

* add atol and rtol, fix test_matmul.py

Signed-off-by: daquexian <daquexian566@gmail.com>

* auto format by CI

* reformat

Signed-off-by: daquexian <daquexian566@gmail.com>

* tricky export

* tricky export

* fix comments

* auto format by CI

* fix comments

* auto format by CI

* fix code

* dirtyfix

* fixname

* fix

* fix bug

* fix comments

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: tsai <jackalcooper@gmail.com>

* add test on add input to graph

* add var into graph

* LazyInterpreter for FetchOutputOpExpr and set op parallel_distribution

* refine input var build

* split file

* rename

* mini refine

* Add note

* LazyInterpret::ApplyImpl for UserOpExpr

* refine test scripts

* add output to graph

* format

* Fix bug of LazyInterpret UserOpExpr for change output lbns

* Add test user op expr test

* fix note mistake

* add userop and test

* address review

* address review

* save i/o/s op_name and tensor for c_nn_graph

* address review

* adjust test

* refine build_graph_state

Co-authored-by: chengtbf <472491134@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add speed test in ci

Signed-off-by: daquexian <daquexian566@gmail.com>

* use actions/checkout@v2

Signed-off-by: daquexian <daquexian566@gmail.com>

* add more input size

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix arg order

Signed-off-by: daquexian <daquexian566@gmail.com>

* update test.yml to use oneflow-inc/models master

Signed-off-by: daquexian <daquexian566@gmail.com>

* only clone models repo in cuda_new_interface

Signed-off-by: daquexian <daquexian566@gmail.com>

* update test_speed.sh to check rel speed

Signed-off-by: daquexian <daquexian566@gmail.com>

* update yml and test_speed.sh to post speed stats

Signed-off-by: daquexian <daquexian566@gmail.com>

* escpae newline for github actions

Signed-off-by: daquexian <daquexian566@gmail.com>

* use correct escaping

Signed-off-by: daquexian <daquexian566@gmail.com>

* replace '\n' to \n in js

Signed-off-by: daquexian <daquexian566@gmail.com>

* collapse speed stats

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix comments

Signed-off-by: daquexian <daquexian566@gmail.com>

* set fixed commit and refine msg

Signed-off-by: daquexian <daquexian566@gmail.com>

* test 30 times instead of 20, relax the threshold

Signed-off-by: daquexian <daquexian566@gmail.com>

* update oneflow-models

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix(*): remove inplace broadcast_add

* fix(BroadcastLike): fix axes bug

* refactor job_pass by maybe_system

* refactor AutoSourceAndSinkTick to SingleClientAutoSourceAndSinkTick

* remove useless files

Co-authored-by: leaves-zwx <kunta0932@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* align torch maxpool

* remove redundant params

* add torch style functional

* remove cout code

* align pooling backward

* fix ceil mode

* add dataformat back

* add annotation

* add maxpool1d functor

* fix farward to forward

* add getwindowedOutputShape function

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix return type

* auto format by CI

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refine

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* LazyInterpret::ApplyImpl for UserOpExpr

* Fix bug of LazyInterpret UserOpExpr for change output lbns

* Add test user op expr test

* fix note mistake

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(Stride): add Stride class

* feat(Tensor): support stride and storage_offset interface

* feat(Tensor): add is_contiguous interface

* remove test declaration

* feat(TensorMeta): add hash and compare for stride

* refine code

* refine IsContiguous function

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample nearest bug

* fix upsample nearest bug (#5347)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bilinear bug

* init flip op

* add flip op register

* add flip cpu kernel forward

* add flip kernel impl

* add flip op functor and gradient_funcs

* add test, still need fix bug

* fix segmentfault bug

* add docs

* fix comments

* fix comments

* fix comments

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* implementation of constantpad-3d op

* format

* refine

* rename files

* refine

* refine test case

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample nearest bug

* fix upsample nearest bug (#5347)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bilinear bug

* restruct upsample module

* auto format by CI

* change docs

* fix bug

* fix bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* del default env init

* refine

* fix bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* prevent endless loop

* add timeout 2 hours to prevent exhaust machine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* LazyInterpreter for FetchOutputOpExpr and set op parallel_distribution

* Add note

* refine test scripts

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* replace ForeignJobInstance using JobInstance

* LazyJobStreamType

* NNGraphIf

* NNGraph -> NNGraphIf

* fix compile bugs

* add unit tests for instruction RunLazyJob

* GetInputBufferName/GetOutputBufferName

* multi-client push pull

* refactor wait_and_send_ids_kernel to support multi-client mode

* support multi-client mode return_op

Co-authored-by: liufengwei <2472937968@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix single client env reinit

* addnote

* Multi-Client LogicalRun degenerate to PhysicalRun

* Add Logical->Physical in Multi-Client for IdGenerator and InstructionsBuilder

* multi client test

Signed-off-by: daquexian <daquexian566@gmail.com>

* auto format by CI

* fix wrong std::shared_ptr

Signed-off-by: daquexian <daquexian566@gmail.com>

* run module api test by test_multi_client/generic_test.sh

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix test script

Signed-off-by: daquexian <daquexian566@gmail.com>

* use oneflow.distributed.launch -m unittest discover instead of bash for loop to speed up the test

Signed-off-by: daquexian <daquexian566@gmail.com>

* update ci/test_multi_client/generic_test.sh

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: chengtbf <472491134@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* modified SetInputArgModifyFn

* Delete the CHECK changes in the assign_op.cpp file

* Format

* Modified the OutputArgModifyFn interface

* add return

* maybe error stack from CheckAndConstructOp to OutputArgModifier callback function

* maybe error stack from CheckAndConstructOp to OutputArgModifier callback function

* OutputArgModifier return maybe part_1

* maybe error stack from CheckAndConstructOp to OutputArgModifier callback function

* input_arg_modifier return maybe

* gen_bw_fn return maybe

* bw_gen_fn return maybe

* registry_callback_fn return maybe

* fix bug after merge master

* fix bug

Co-authored-by: aishangjj <702572275@qq.com>

* edit cmake cache and protobuf

* edit more cmakes

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* modified SetInputArgModifyFn

* Delete the CHECK changes in the assign_op.cpp file

* Format

* Modified the OutputArgModifyFn interface

* add return

* maybe error stack from CheckAndConstructOp to OutputArgModifier callback function

* maybe error stack from CheckAndConstructOp to OutputArgModifier callback function

* OutputArgModifier return maybe part_1

* maybe error stack from CheckAndConstructOp to OutputArgModifier callback function

* input_arg_modifier return maybe

* gen_bw_fn return maybe

* bw_gen_fn return maybe

* fix bug: return Maybe without JUST

Co-authored-by: aishangjj <702572275@qq.com>

* fix TryClose

* fix addressing watch callback

* modified SetInputArgModifyFn

* Delete the CHECK changes in the assign_op.cpp file

* Format

* Modified the OutputArgModifyFn interface

* add return

* maybe error stack from CheckAndConstructOp to OutputArgModifier callback function

* maybe error stack from CheckAndConstructOp to OutputArgModifier callback function

* OutputArgModifier return maybe part_1

* maybe error stack from CheckAndConstructOp to OutputArgModifier callback function

* input_arg_modifier return maybe

* gen_bw_fn return maybe

* add MakeGenBackwardOpConf because ofstatement-expression not allowed outside function resulting in JUST failed in lambda

* add maybe after merge master

* fix bug: JUST in lambda

Co-authored-by: aishangjj <702572275@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>