* split vector-matrix norm

* fix_vector_norm

* fix_docstring

* fix_matrix_norm

* fix-default

* fix-doctest

* auto format by CI

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample nearest bug

* fix upsample nearest bug (#5347)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bilinear bug

* fix upsample bilinear bug

* recover code

* align with pytorch

* redesign upsample bilinear

* fix align corner bug

* align bilinear with pytorch

* fix conflict

* fix bug

* code reuse

* fix commets

* add upsample_linear_1d_user_op

* add upsample_linear_1d_user_op

* add linear1d interp cpu kernel

* add linear1d interp gpu kernel

* rename kernel name

* add upsample nearest_1d kernel

* add upsample nearest_1d cuda kernel

* add upsample nearest_1d op

* fix complie bug

* add bicubic interp cpu kernel forward

* add bicubic2d interp cpu kernel backward impl

* add bicubic2d gpu kernel impl

* fix bicubic kernel compile error

* add bicubic2d kernel register

* fix compile bug

* nearest3d interp cpu kernel

* add nearest3d gpu kernel

* add nearest3d kernel registry

* add nearest3d cpu kernel

* add nearest3d cpu kernel

* add nearest3d cpu kernel

* fix upsample nearest interp bug

* fix upsample nearest interp bug

* add trilinear3d cpu kernel forward

* add trilinear3d cpu kernel backward

* add trilinear gpu kernel impl

* add trilinear op registry

* init interpolate nn function

* add upsample functors

* add interolate module

* add interpolate test_case

* restruct upsample_nearest2d and upsample_bilinear2d

* add upsample nearest2d and bilinear2d op registry

* add upsample nearest2d and bilinear2d op registry

* add upsample kernel grad functor

* add upsample_nearest_2d backward

* add upsample_linear_1d backward

* add upsample_nearest_1d backward

* add upsample_bicubic_2d backward

* fix backward impl bug

* fix nearest1d backward bug

* try to fix backward segmentfault

* add data_format parammeter

* fix upsample kernel backward bug

* fix cuda kernel impl bug

* fix upsample nearest bug

* fix align_corner attr error

* align linear1d impl with pytorch

* fix trilinear register bug

* fix trilinear bug

* add trilinear aligncorners test

* add test for bicubic, still has bug

* reimpl bicubic2d kernel

* align cubic interpolate with pytorch

* add bicubic cuda kernel

* fix bug

* align api with pytorch

* add doctest for interpolate

* update interpolate docs

* fix comments

* fix check_of_return comment

* fix comment

* fix comments

* fix commnet

* fix comment

* fix comments

* fix comments

* fix comments

* fix comments

* fix comment

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Luyang <flowingsun007@163.com>

* add gather_nd and scatter_nd

* add testcase

* stable_api

* add docs and doctest

* add scatter_nd grad

* gradfunc

* delete some code comments

* add grad test

* add docs

* update words usage

* delete some import

* update

* update

* auto format by CI

Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* init nn.image.flip

* clean code

* add unit test.

* revert python.cmake

* auto format by CI

* resolve comments

* add functor

* fix image_flip

* fix testcase

* fix warning:  Unexpected indentation

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: simonJJJ <821898965@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* init of op diag

* init of op diag

* modify  op diag

* merge testcase

* delete no use tensor diag

* amend diag api docs

* resolve conficts

* resolve confict

* resolve confict

* resolve confict

* auto format by CI

* add tensor.diag

* auto format by CI

* amend API description

* add non_square test

* amend diag API description

* amend diag API description

* compact test code

* compact code

* auto format by CI

* amend API description

* amend API description

* motify functional API

* motify test_diag

* amend diag functional API

* amend diag docstring

* auto format by CI

* amend code standards

* auto format by CI

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Add in_top_k function

* auto format by CI

* fix bug

* Update oneflow/python/ops/math_ops.py

Co-authored-by: Yao Chi <later@usopp.net>

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* startup: use python, not C extension

* rst changes test

* refine

* add F.rst

* add bernoulli add docstr example

* add doctest for functional method

* del docstr utils immediately in init.py

* refine test case

* use sin, cos instead of add, mul

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(add_): support inplace add

* fix(add_): fix push bug

* Add InplaceableBinaryFunctor

* feat(add_): add tensor += other

* docs(add_): fix docs

* fix relu module bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* implement smoothl1loss

* add smoothl1loss  test

* fix bug

* fix bug

* remove swp file

* fix bug

* refine

* refine_grpc_cmake_for_conda

* refine

* refine

* refine

* rm

* refine

* refine

* refine

* refine

* also fix opencv

* refine

* reorder

* refine gflags and glog

* rm useless

* rm useless

* rm useless

* refine

* rm var only available in protobuf

* add not-suppported check for reduce and size_average

* add test for datatype of int32

* auto format by CI

* implement backward and rebuild Factor

* rebuild Factor for smooth_l1_loss_grad

* auto format by CI

* fix bug

Co-authored-by: jackalcooper <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* add bernoulli module

* fix doc test

* add bernoulli functor

* make changes according to review

* refine (#5415)

* fix

Co-authored-by: Bowen Chen <bob2420083992@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample nearest bug

* Enhance the difficulty of test samples

* fix

* fix

* fix

* fix upsample nearest bug (#5347)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bilinear bug

* fix upsample bilinear bug

* fix test

* recover code

* align with pytorch

* redesign upsample bilinear

* fix align corner bug

* align bilinear with pytorch

* enchaned 4-dim samples

* fix conflict

* add backward test

* fix bug

* complete grad test

* fix merge conflict

* delete handwritten interpolate

* bypass pytorch interpolate bug in test

* add tensor grad assignment

* add test

* revert upsample changes

* fix TypeError: set_grad(): incompatible function arguments.  bug

* add zero_grad() docstring

* formmat comment

* fix docstring

* fix suggestions

* Update oneflow/python/test/tensor/test_tensor.py

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

* Update oneflow/api/python/framework/tensor.cpp

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

* enhance grad setter

* refine zero_grad() docstring and add slice docstring declaration to rst

* adjust optim.rst order

* refine comment

* refine zero_grad() docstring

* fix bug

Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

* chunk forward

* chunk forward

* forward nn.chunk

* chunk forward

* chunk forward

* complete module flow.chunk

* auto format by CI

* flow.chunk/tensor.chunk

* resolve confict

* auto format by CI

* functional api

* auto format by CI

* doctest.testmod(raise_on_error=True)

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add flow.view

* clean code

* clean code

* clean code

* clean code

* clean code

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add masked_select module impl

* add masked_select module test

* add masked_select module test

* add masked_select module test

* fix bug

* fix comment

* fix comment

* fix comment

* init gather_nd backward

* fix backward functor bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Add partial unary and math functional apis.

* Revert elementwise pow.

* auto format by CI

* Support add with large number of inputs.

* Update oneflow/python/nn/modules/math_ops.py

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

* Refine

* Migrate binary and activation ops.

* Migrate array ops.

* Add or refactor activation grad funcs.

* Add or refactor activation grad funcs.

* Revert unpack all

* Fix masked fill

* Refine

* Add nn ops.

* Refine

* Refine

* Migrate conv op

* Fix functional normalization.

* auto format by CI

* unfinished

* Refine code style

* align Torch params

* align Torch params

* develop unfinish

* add conv1d

* add conv1d docs rst

* add conv1d module and docs

* fix bias add error

* fix groups bug

* add test case

* Support optional parameter.

* fix group bug

* add new test case

* add more test case

* small fix

* add torch reference

* add conv base functor

* remove useless print

* reorganize code structure

* fix name and vector size

* fix pushback to at

* small fix for deconv docs

Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: Luyang <flowingsun007@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add not_equal, greater_equal and less_equal module

* fix docs

* add hash method to Tensor

* remove __eq__ and __ne__ override

* fix doc

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refine and add test case

* support ellipsis type slice

* refine

* refine

* support slice assign ellipsis type

* refine

* register fn to localtensor

* add type_as module

* add long module

* add more test

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add new ones module

* add newones module test

* add newones module test

* add new_ones module doctest

* add test_case

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* add bmm module

* add bmm module test

* add doctest

* add doctest

* fix docstring bug

* fix commets

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix conflict

* add tensor_buffer_ops eager

* add testcase

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Add 'nn.Mish' module and corresponding functions

* Improve the docs and unit test

* auto format by CI

* Fix the indent problems

Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* add zeropad2d and testcase

* add doctest

* doctest edit

* delete h,w if-else

* add docs

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add tile module

* add backward testcase

* add docstring

* parameters consistent with pytorch

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* delete onnx

* Update test.yml

Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Add GroupNorm

* Fix bugs

* Add test groupnorm

* Add docstring

* auto format by CI

* Fix bug

* Adjust test function name

* Remove useless codes

* auto format by CI

Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add avgpool1d and avgpool3d

* add test case

* test padding value

* avgpool1d raise NotImplementedError

* add test case

* numpy_out is different of module_out

* numpy grad error

* add docs and doctest

* use torch out

* add docs

* delete backward function and commented code

* adjust docs format

* auto format by CI

* adjust code

Co-authored-by: Flowingsun007 <flowingsun007@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* add reflecctionpad2d

* add reflecctionpad2d

* backward unittest passed

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* module BCELoss

* auto format by CI

* module bce_loss bce_loss.py

* complete develop bce_loss forward with bce_loss.py

* module bce_loss

* testCase -numpy_BCELoss_BP

* modify testCase

* match args

* auto format by CI

* move bce_loss to loss.py

* auto format by CI

* resolve ci error

* resolve license_duplicated

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* init-norm

* checkpoint

* update order 0

* first_push

* refine-norm

* delete-tripletloss-part

* fix-conlict

* add-tensor-norm

* fix-docstring

* fix-docstring

* auto format by CI

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* first commit of op floor

* first commit of op acos

* split acos from math_ops and add doctest

* resolve conflicting files

* fix format bug

* fix bug in forward

* add test_bcewithlogitsloss

* add test_bcewithlogitsloss fix bugs

* modify doctest

* modify doctest

* add reduce and size_average arguments

* rebuild test case and modify bugs in docstring

* solve conficts

* auto format by CI

* rectify args explanation in docstring

* remove redundant check of pos_weight length

* resolve conflicts

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* modules CTCLoss

* merge master

* Rewrite ctc_loss gradient funcs

* int64_t -> int32_t

* modify int to int32_t

* add Reference

* add requires_grad

Co-authored-by: YaoChi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* rename CosineScheduler to CosineAnnealingLR

* Rename cosine_scheduler.py to cosine_annealing_lr.py

* add reference

* clean code

* clean code

* auto format by CI

* clean code

* auto format by CI

Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add argmax test

* fix ci error

* fix docstring warning

* fix tensor greater and less bug

* add meshgrid module

* add meshgrid module

* fix meshgrid module bug

* add docstring for meshgrid module

* add doctest for meshgrid module

* auto format by CI

* fix commnet

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add ReplicationPad2d

* add replicationpad2d

* update

* update

* update replicationpad2d

* update docstring

* auto format by CI

* Update test_replicationpad2d.py

update test

* bug fix about value

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* add kldivloss module

* support for larger range of target values

* add doctest

* separate testcase

* delete deprecated argument

* fix zeros device bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* first commit of floor

* Added function floor_op_tensor

* fixed a copy paste bug

* added test for backward_cpu

* Added more test cases, added doctest

* fix conflicts

* changed doctest testmod, eliminated the print commands in doctest

* made sure the math_op.py is the same as in the latest master branch

* added an ending newline

* auto format by CI

* fix test_tensor python format

* auto format by CI

* fix license

* fix docstring

* fix doc test format

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* add ConstantPad2d

* add ConstantPad2d

* add ConstantPad2d

* add ContantPad2d

* update constantpad2d

* update doctest

* update doctest

* Update ConstantPad2d.py

update line

* update name

* update

* review

* auto format by CI

* update docstring

Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>

* fix bert backward bug

* fix argwhere module bug

* align dropout module with torch

* add adaptive_avg_pool2d ops register

* add adaptive_avg_pool2d ops register

* add adaptive_avg_pool2d ops register

* add adaptive_avg_pool2d forward kernel

* add adaptive_avg_pool2d backward kernel

* export flow.adaptive_avg_pool2d and test

* export flow.adaptive_avg_pool2d and test

* recover code

* fix op register bug

* get segment fault

* adaptive avgpool2d module

* delete useless user_op code

* add adaptive_avg_pool2d module test

* refine test_adaptive_avgpool2d module

* fix test_adaptive_avgpool2d op bug

* fix adaptive_avgpool2d_module bug

* fix adaptive_avgpool2d_module bug

* fix adaptive_avgpool2d_module bug

* add adaptive avg_pool2d module

* fix adaptive avg pool2d module backward bug

* fix adaptive avg pool2d module backward bug

* fix comments

* fix comment

* fix comment

* fix comment

* add adaptive_avg_pool_gpu_kernel

* add adaptive_avg_pool2d gpu k...

* init instancenorm.

* fix bert backward bug

* set time limit

* add time limit

* add instancenorm and unittests

* add to rst

* clean code.

* add references to pytorch

* add tests

* auto format by CI

Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: jackalcooper <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* first commit of op floor

* first commit of op acos

* split acos from math_ops and add doctest

* resolve conflicting files

* fix format bug

* auto format by CI

* add check license

* synchronize local with remote

* auto format by CI

* first commit of op acos

* Update tensor.cpp

* solve conflicts

* add of_softmax_use_fast_math (#4979)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* auto format by CI

* ci check license

Co-authored-by: Luyang <flowingsun007@163.com>
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: JiaojiaoYe1994 <jiaojiaoye@tum.de>