* fix return type error on gcc 4.8.5

Signed-off-by: daquexian <daquexian566@gmail.com>

* auto format by CI

* fix return type error in xrt

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* Use function interface

* Delete blank lines

* modefied scalar add

* elementwise add

* broadcase add

* modified  ScalarAddByTensor

* modified Addmm

* Delete the old implementation

* fix _sub errot

* auto format by CI

* modified  _inplaceable

Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample nearest bug

* fix upsample nearest bug (#5347)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bilinear bug

* fix export bug

* add conv autotest

* add dead loop error raise

* add floor auto_test

* fix random tensor generator bug

* restruct math ops test

* add atan autotest

* restruct math_ops

* fix bug

* fix conv test bug

* fix commnets

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* disable eager_op consistent mode

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* add ibverbs_qp.h

* modify the qp 2021-07-14

* optimize the qp_init_attr.cap.max_send_wr
when we send  too many message which are greater then qp_init_attr.cap.max_send_wr  it may have some problem
we use the msg_pendding_list_ to store the extra messages when the message is greater than qp_init_attr.cap.max_send_wr

* optimize the qp_init_attr.cap.max_send_wr
    when we send  too many message which are greater then qp_init_attr.cap.max_send_wr  it may have some proble
    we use the msg_pendding_list_ to store the extra messages when the message is greater than qp_init_attr.cap and clear todo

* optimize the qp_init_attr.cap.max_send_wr
    when we send  too many message which are greater then qp_init_attr.cap.max_send_wr  it may have some proble
    we use the msg_pendding_list_ to store the extra messages when the message is greater than qp_init_attr.cap and clear todo  and clear extra line

* optimize the qp_init_attr.cap.max_send_wr
    when we send  too many message which are greater then qp_init_attr.cap.max_send_wr  it may have some proble
    we use the msg_pendding_list_ to store the extra messages when the message is greater than qp_init_attr.cap and clear todo  and clear extra line and license

* optimize the qp_init_attr.cap.max_send_wr
    when we send  too many message which are greater then qp_init_attr.cap.max_send_wr  it may have some proble
    we use the msg_pendding_list_ to store the extra messages when the message is greater than qp_init_attr.cap and clear todo  and clear extra line and license and todo

* optimize the qp_init_attr.cap.max_send_wr
    when we send  too many message which are greater then qp_init_attr.cap.max_send_wr  it may have some proble
    we use the msg_pendding_list_ to store the extra messages when the message is greater than qp_init_attr.cap -2021-07-15-20:00

* fix the review and pass the test

* increase the kMaxSendWr 07-19

* fix the deconstruct function 2021-07-19

* test the first pr and make it precise

* fix the format and check the logic and remove the using_pendding_list_

* 2021-07-21

* fix the code  according to the  review

* fix the

* rewrite the pr and now we test it

* optimize the IB
when send too many message whose number is bigger then qp_init_attr.cap.max_send_wr,
I use the msg_pendding_list to store the extra message and wait for some time to send these message of the msg_pendding_list

* clear the duplicate license

* use pendding_list to optimize qp_init_attr.cap.max_send_wr of IB module

* change the format of the IB

* commit the format

* add new line 2021-07-27

* add new line 2021-07-27-19:30

* add new line 2021-07-27-19:33

* what is the problem of the format

* change what

* optimize the format

* win10 changed

* runtime

* runtime change again

* optimize the format of runtime

* optimize the format of acor_message.h

* optimize the format ofcpp

* optimize the format ofcpp  again

* optimize the format ofcpp  version0.56

* remove some head file

* fix the name of some paremeter

* merge master to the lambda7 and push it

* cleat the newline of 2021-0728

* why crlf

* optimze the format

* use the make format and optimization

* make the  EnqueuePostSendReadWR/PostPenddingSendWR be private

* use the ending_send_wr_lock_ as lock

* use the pending instead of the ending

* use the pending instead of the ending and

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add activation

* rename swish to silu

* add selu

* add four activation op

* add softsign test

* add silu mish selu softsign

* Add softsign docs

* Add functional impl

* small fix for softsign backward

* remove flow.mish test

* add silu module test

* add selu test

* fix docs

* fix softsign docs

* fix format

* fix static cast

* merge master functional api yaml

* add torch style unittest

* Remove assert and add torch unittest

* add tensor def

* remove softsign test temporary

* add return maybe ok

* migrate nn ops to single_client dir

* migrate unittest

* remove lazy unittest

* add unittest

* fix to new directory

* Remove useless docs and single client test

* add doc

* fix docs

* add docs in oneflow and tensor namespace

* add torch autotest

* fix to new autotest

* remove outdated python code

* remove useless docs

* enlarge unittest tolerance

* Add static cast for const value

* skip softsign unittest

* skip tensor softsign

Co-authored-...

* fix upsample nearest bug

* fix upsample nearest bug (#5347)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bilinear bug

* fix export bug

* fix flip bug

* fix clip bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Feat: memory shared between EagerTensor with VariableRegst

* Fix BUG of Maybe<void> no return

* Add args of Runtime; Add Test Scripts and Runnable

* refine note

* refine test scripts

* refine code for review

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample nearest bug

* fix upsample nearest bug (#5347)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bilinear bug

* fix export bug

* support expand flow.Size and tuple input

* delete tmp file

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* add valid optimizer case to pass job complete

* refactor and fix optimizer

* support l2 in variable op

* merge master

* format

* fix merge

* rm useless include

* address review

* auto format by CI

* rm internal tensor check after merge master

* restore add loss after merge master

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* fix upsample nearest bug

* fix upsample nearest bug (#5347)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bilinear bug

* fix export bug

* support flow.Size

* support flow.Size

* format

* auto format by CI

* fix repeat module bug

* fix repeat module bug

* fix repeat module bug

* fix repeat module bug

* auto format by CI

* fix repeat module bug

* fix conflict

* fix conflict

* auto format by CI

* support tuple parameter

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* Adapt to new structure

* auto format by CI

* fix bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* IBVerbsCommNet add knobs

* clean

* move tensor to c++

Signed-off-by: daquexian <daquexian566@gmail.com>

* align with master and fix export error

Signed-off-by: daquexian <daquexian566@gmail.com>

* reformat

Signed-off-by: daquexian <daquexian566@gmail.com>

* rename some functions

Signed-off-by: daquexian <daquexian566@gmail.com>

* refine

Signed-off-by: daquexian <daquexian566@gmail.com>

* remove unused header

Signed-off-by: daquexian <daquexian566@gmail.com>

* restore device repr and str

Signed-off-by: daquexian <daquexian566@gmail.com>

* upgrade pybind11 to 2.7.0 for https://github.com/pybind/pybind11/pull/2956



Signed-off-by: daquexian <daquexian566@gmail.com>

* impl parameter in c++, remove determine() call

Signed-off-by: daquexian <daquexian566@gmail.com>

* let parameter derive from TensorIf to own a separate grad_fn_node, add missing get_device method, remove out-dated methods in docs

Signed-off-by: daquexian <daquexian566@gmail.com>

* add test_parameter.py, fix requires_grad_ and get_de...

* LazyInterpret gen Variable ParallelDistribution

* LazyInterpret NewScopeWithParallelDescByTensor

* Debug and pass test for  input/output op expr lazy tensor infer device

* Add normal infer user op scope impl; add test user op expr for cuda

* fix for review

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* nn.Graph call and launch impl

* Fix bug of 1.pybind11 list->vector; 2.GlobalJobDesc in NNGraph Compile

* Fix bug: AutoTick Multi-Client callback notifier lbn

* Fix bug: callback notifier op check in multi-client

* Fix bug: CheckOpGraph in multi-client

* Fix bug of MultiClientSessionContext new ProfileConf

* Fix bug of create NONE type kernel; add note; refine PlanUtil::PopulateOpAttibute

* Fix bug: CallbackNotifier kernel dtype with uint8; change tick dtype; Feat: add Kernel create fault-tolerant

* inputs tensor list

* rollback of hack

* Using TensorTuple for RunLazyNNGraph

* Fix bug: WaitAndSendIdsKernel handle for Multi-Client

* skip graph runnable test

* refine

* skip test graph optimizer for nn.Graph train job completer error

* debug code

* rm InputKernel Forward overwrite

* rm debug code

* revert useless change

* revert debug code in graph

* rm skip in test

* revert source tick op

* rm useless import

* mv test_graph_relu.py

* correct break

* change the code of finding tick

Co-authored-by: chengtbf <472491134@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Support symbol placement type in functional.

* add sbp and sbp list arg

* refine

Co-authored-by: clackhan <han_binbin@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample nearest bug

* fix upsample nearest bug (#5347)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bilinear bug

* fix export bug

* fix conflict

* auto format by CI

* fix license

* fix ci error

* Delete version.py

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refine and add test case

* support ellipsis type slice

* refine

* refine

* support slice assign ellipsis type

* refine

* register fn to localtensor

* basic implemetation of dataloader

* format

* refine as comments

* fix comments

* remove useless code

* add test case into unnitest

* refine ascomments

* refine as comments

* register uint8 dtypeto support dataloader

* add cifer mnist dataset support

* format

* refine

* refine

* use F.interpolation

* export dataloader dataset

* export more api

* refine

* refine

* add oneflow_test_cache_dir

* refien

* refine

* refine

* refine

* update datasets links

* refine

* fix comment

* unset export

* auto format by CI

* add requirement

* always reisntall test deps

* update requirement

* add pillow as oneflow dependency

* auto format by CI

* add sessiong context debug str

* remove debug str

* remove experimental
...

* refein

* rm

* fix

* refein

* log

* add ignore

* add ignore

* refein

* refein

* refein

* Support numpy advanced indexing.

* Finally adjust if continuos subspace.

* Finish integer array indexing and combining indexing.

* Finish index array parsing.

* Add tensor indexing unittest

* Add check

* Refine

* Refine tensor getitem unittest

* Support int64_t cuda atomicAdd.

* Not use PyObjectPtr since PySequence_Fast_GET_ITEM returns borrowed reference.

* Remove experimental in indexing unittest.

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add argmax test

* fix ci error

* fix docstring warning

* fix tensor greater and less bug

* fix conflict

* add test_flow_xxx_against_pytorch func

* fix bn autograd bug

* fix bn autograd bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refine

* change abs.py for test

* refine

* refine

* refine

* refine

* refine

* refine

* add

* refeom

* refine

* refine

* refien

* refine

* refine

* refine

* refien

* add check

* refine

* refine

* refine

* skip license

* refine

* refien

* refine

* refien

* refien

* refine

* refine

* refine acos

* refien

* remove enable_eager_execution() in oneflow/python

remove @experimental_api decorator for apis, except that ones
having same name with single client api.

* remove all import oneflow.experimental in doctest

* replace oneflow.experimental with oneflow in docstr

* reformat

* auto format by CI

* refine

* refien

* refine

* refien

* refine

* refine

* refine

* refein

* refine

* remove signle client twice api

* refine

* refine

* refine

* refien

* refien

* refine

* refien

* refien

* refien

* refine

* r...

* refine oneflow/python/ops/array_ops.py

	modified:   ../oneflow/python/nn/modules/masked_select.py
	modified:   ../oneflow/python/ops/array_ops.py

* refine

* refine

* rm reduce_ops.py and summary files

* refine layers etc.

Co-authored-by: binbinHan <han_binbin@163.com>

* refine and add test case

* support ellipsis type slice

* refine

* refine

* support slice assign ellipsis type

* refine

* register fn to localtensor

* add constantpad1d kernel

* add constantpad1d kernel

* implementation of fuctional api/gradients/test case

* refine test case

* refine

* format docstring

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* nn.Graph call and launch impl

* Fix bug of 1.pybind11 list->vector; 2.GlobalJobDesc in NNGraph Compile

* Fix bug: AutoTick Multi-Client callback notifier lbn

* Fix bug: callback notifier op check in multi-client

* Fix bug: CheckOpGraph in multi-client

* Fix bug of MultiClientSessionContext new ProfileConf

* Fix bug of create NONE type kernel; add note; refine PlanUtil::PopulateOpAttibute

* Fix bug: CallbackNotifier kernel dtype with uint8; change tick dtype; Feat: add Kernel create fault-tolerant

* inputs tensor list

* rollback of hack

* Using TensorTuple for RunLazyNNGraph

* Fix bug: WaitAndSendIdsKernel handle for Multi-Client

* skip graph runnable test

* refine

* skip test graph optimizer for nn.Graph train job completer error

* Skip SetDefaultVariableConf JobPass in Multi-Client

* Fix bug: register_variable_op_names_and_tensors args with TensorTuple

* refine test graphs

* rm experimental in new scripts

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample nearest bug

* fix upsample nearest bug (#5347)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bilinear bug

* fix export bug

* add new_autotest_part1

* add new_autotest_part1

* add squeeze test

* add permute test

* add hardswish test

* add prelu

* fix bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* change doctest code in abs.py

* del >>> flow.enable_eager_execution() in  modules/abs.py

* del @experimental_api

* del @experimental_api

* slove doctest not pass

* change abs.py for test

* refine acos

* remove enable_eager_execution() in oneflow/python

remove @experimental_api decorator for apis, except that ones
having same name with single client api.

* remove all import oneflow.experimental in doctest

* replace oneflow.experimental with oneflow in docstr

* reformat

* auto format by CI

* remove signle client twice api

* refine

* remove experimental of nn

* okokok

* remove experimental in test/modules and test/tensor

* remove experimental in test/dataloader

* remove experimental call in nn/modules

* remove experimental from nn/ framework/ test_utils

* remove experimental in functionalpy

* remove experimental of new merge

* skip fmod grad test

* remove experimental of fmod

* fix error when do multi case test on graph

* remove outdated rst files

* refine and add new rst files

* format

* revert test_fmod.py

* remove all experimental in python/nn dir (done!)

* remove experimental in test/graph

* add linagl

* refine rst

* refine scatter_nd

* refine oneflow.rst

* refine load_minst's docstring

add image.rst module.rst

refine rst files

* reformat

* remove Graph api

* fix file name

* fix index

Co-authored-by: YaoChi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: strint <xiaoyulink@gmail.com>
Co-authored-by: tsai <jackalcooper@gmail.com>

* refine oneflow/python/ops/array_ops.py

	modified:   ../oneflow/python/nn/modules/masked_select.py
	modified:   ../oneflow/python/ops/array_ops.py

* refine

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix error when do multi case test on graph

* format

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Test 'nn.Linear' and 'nn.Identity' with random data

* Test 'nn.PixelShuffle' with random data

* Test 'greater' and 'less' with random data

* Test 'flow.Tensor.flatten' with random data

* Test 'greater' and 'less' with random-sized data

* Test 'nn.CrossEntropyLoss' with random data

* Replace multiple return entries with the 'oneof' generator

* Add 'nothing()' test cases for parameters with default values

* Remove the hard-coded test cases of `nn.CrossEntropyLoss`

* auto format by CI

* Skip the test of 'nn.CrossEntropyLoss' temporarily

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>