* fix upsample nearest bug

* fix upsample nearest bug (#5347)

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix upsample bilinear bug

* fix upsample bilinear bug

* recover code

* align with pytorch

* redesign upsample bilinear

* fix align corner bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* check in files

* refine

* fix yml

* fix path

* refine

* refine

* refine

* refine

* refine cache

* refine

* refine

* check out oneflow first

* refine

* setup clang as compiler

* add ONEFLOW_MAYBE_CHECK_ONLY_FN

* add llvm libs to LD_LIBRARY_PATH

* refine

* refine

* refine

* refine

* update yml to use apt to install clang

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix wrong --target

Signed-off-by: daquexian <daquexian566@gmail.com>

* add oneflow deps

* refine

* use debug to run faster

* add note

* easy to see

* refine

* fix

* refine

* refine

* refine

* add missing JUST

Signed-off-by: daquexian <daquexian566@gmail.com>

* skip .cfg.cpp

Signed-off-by: daquexian <daquexian566@gmail.com>

* improve ONEFLOW_MAYBE_CHECK_ONLY_FN

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix cmake wrong message and fix env var

Signed-off-by: daquexian <daquexian566@gmail.com>

* remove temp log

Signed-off-by: daquexian <daquexian566@gmail.com>

* add missing JUST in master

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Register no gradient user op, and check failed if the gradient func was missing for the op that needs gradient.

* Remove the default gradient func.

* Make broadcast logical ops no grad.

* Add identity op grad func

Co-authored-by: cheng cheng <472491134@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Device::compute_dep_object_

* sequantialize instructions in the same stream.

* refactor AttrMap

* refactor Tensor

* Export ConsistentTensor::is_cuda

* remove ConsistentTensor::blob_object

* refactor TensorImpl

* minor fix

* fix compiler' complains

* Implements EagerConsistentTensorImpl::New

* minor fix

* fix compiler complains

* remove unused code

* skip test_creating_consistent_tensor

* backup code

* Symbol::shared_from_symbol

* remove redundant header file includes

* fix bug in Symbol::shared_from_symbol

* symbolize ParallelDesc and ParallelDistribution

* symbolize Scope::GetParallelDesc()

* IsScalarType

* fix compiler complains

* InputConsistentTensorMeta

* refactor Scope with PlacementScope

* fix bug in exporting Scope to python

* backup code

* refactor DType

* fix compiler complains

* backup code

* DType is only allowed to be used in python code

* backup code

* dtype api bugfix

* fix error on exiting

Signed-off-by: daquexian <daquexian566@gmail.com>

* lazily get rank

Signed-off-by: daquexian <daquexian566@gmail.com>

* Export const DType* into python

* minor fix

* fix bug

* refine

* refactor signature of OpExpr::InferLogicalShapeAndDtype

* fix bug

* backup_code

* fix bug

* refactor SbpXXX to cfg::SbpXXX

* merge refactor_sbp_to_cfg_sbp

* fix bug

* Infer ConsistentTensorMeta

* Implement EagerConsistentInterpret::ApplyImpl

* 1) move XXXTensorMeta into the new file tensor_meta.h; 2) add new Class ConsistentTensorInferCache

* add class ConsistentTensorInferResult

* remove unused OpArgMutConsistentTensorMeta::parallel_distribution_

* fix stack-overflow bug in Tensor::mut_eager_mirrored_tensor_impl

* ignore empty parallel distribution constaint

* fix bug

* add explicit of cfg

* fix xla compile bug

* auto format by CI

* fix according comment

* fix bug

* refine

* refine

Co-authored-by: lixinqi <lixinqi0703106@163.com>
Co-authored-by: Li Xinqi <lixinqi2010@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* add changes

* fix

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add triu

* add triu cuda kernel

* add triu gradient func

* add tensor test for triu

* add triu op

* add hard testcase for triu

* fix format

* add space

* add complicate test case

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Device::compute_dep_object_

* sequantialize instructions in the same stream.

* refactor AttrMap

* refactor Tensor

* Export ConsistentTensor::is_cuda

* remove ConsistentTensor::blob_object

* refactor TensorImpl

* minor fix

* fix compiler' complains

* Implements EagerConsistentTensorImpl::New

* minor fix

* fix compiler complains

* remove unused code

* skip test_creating_consistent_tensor

* backup code

* Symbol::shared_from_symbol

* remove redundant header file includes

* fix bug in Symbol::shared_from_symbol

* symbolize ParallelDesc and ParallelDistribution

* symbolize Scope::GetParallelDesc()

* IsScalarType

* fix compiler complains

* InputConsistentTensorMeta

* refactor Scope with PlacementScope

* fix bug in exporting Scope to python

* backup code

* refactor DType

* fix compiler complains

* backup code

* DType is only allowed to be used in python code

* backup code

* dtype api bugfix

* fix error on exiting

Signed-off-by: daquexian <daquexian566@gmail.com>

* lazily get rank

Signed-off-by: daquexian <daquexian566@gmail.com>

* Export const DType* into python

* minor fix

* fix bug

* refine

* refactor signature of OpExpr::InferLogicalShapeAndDtype

* fix bug

* backup_code

* fix bug

* refactor SbpXXX to cfg::SbpXXX

* merge refactor_sbp_to_cfg_sbp

* fix bug

* Infer ConsistentTensorMeta

* Implement EagerConsistentInterpret::ApplyImpl

* 1) move XXXTensorMeta into the new file tensor_meta.h; 2) add new Class ConsistentTensorInferCache

* add class ConsistentTensorInferResult

* remove unused OpArgMutConsistentTensorMeta::parallel_distribution_

* fix stack-overflow bug in Tensor::mut_eager_mirrored_tensor_impl

* ignore empty parallel distribution constaint

* fix bug

* add explicit of cfg

* fix xla compile bug

* auto format by CI

* fix according comment

* fix bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: clackhan <han_binbin@163.com>
Co-authored-by: daquexian <daquexian566@gmail.com>
Co-authored-by: Shenghang Tsai <jackalcooper@gmail.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* Add pad grad func.

* Fix constant pad grad func.

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Houjiang Chen <chenhoujiangcug@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add missing JUST

Signed-off-by: daquexian <daquexian566@gmail.com>

* remove redundant header

Signed-off-by: daquexian <daquexian566@gmail.com>

* add missing JUST in master

Signed-off-by: daquexian <daquexian566@gmail.com>

* fix compile error on gcc5

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* check in updates

* refine

* refine

* fix case of LinkedIn

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refine pow module and its test

* simplify test & add scalar-pow backward test

* refine pow module doc

Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: guo ran <360112263@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* added flatten backward

* flatten and softmax backward

* fix bug for not supporting all kinds of iterables in optimizers

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(GraphAutogradEngine): add GraphAutogradEngine

* feat(GraphEngine): support backward interface

* feat(GraphEngine): support autograd.backward()

* fix(GraphEngine): some grad_fn in next_functions but can not apply

* feat(GraphEngine): support autograd.grad interface

* style(*): add JUST

* fix(GraphEngine): fix autograd.grad bugs

* test(Autograd): add flow.autograd test

* fix(GraphEngine): fix autograd.grad bug

* refine codes

* style(*): remove comments

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Dev functional interface.

* Remove repeated function_traits

* Implement add and add_scalar static functional op.

* Remove unused code

* Refactor

* Refine

* Generate and export functional apis.

* Refine

* Refine

* Refine

* Generate functional api and pybind cpp when building the project.

* Refine code style and implement normalization functor.

* Fix cmake

* Add PyYAML requirement.

* Add JUST

* Fix scalar IsIntegral and IsSigned

* Add scalar add grad func.

* Fix norm grad func to support dynamic attrs.

* Refactor math modules

* Refactor activation modules.

* Support DataType.

* Add functional range.

* Support DataType.

* Add functional argmax and flatten.

* Add functional argwhere.

* Add functional broadcast_like

* Add functional cast, zeros_like and ones_like.

* Recursive determine input tensors.

* Add functional concat, bias_add and conv2d.

* Recursive determine input tensors.

* Add functional conv2d, bias_add, eq, exp, expand, gather, dim_gather, greater, less, matmul and broadcast_matmul.

* Fix scalar_div_by_tensor grad.

* Update generate_functional_api.py

* Update generate_functional_api.py

* Add functional activation, argmax, eq, layer norm etc.

* Fix dynamic sparse_softmax_cross_entropy.

* Add functional expand_dims and where.

* Fix conversion from python object to dtype.

* Fix conversion from python object to dtype.

* Access device_infer_func dynamically since this func maybe has not been registered while the op expr was constructed.

* Add more functional apis, and fix bugs.

* Fix crash since function has been cast with wrong function signature, the converted function behavior is undefined.

* Check and throw error.

* Refine

* Use Maybe instead of throwing exception directly.

* Reformat

* Fix conv module, refactor bias_add grad func to support dynamic attrs.

* Use composed attrs when creating kernel state.

* Enable static conv op.

* Update generated functional files only when different.

* Use default target other than custom command.

* Create new kernel state if in eager mode

* make conv_kernels state less

* fix typo

* add conv grad functor; make conv_gpu_kernel stateless

* Remove unused code.

* Add partial unary and math functional apis.

* Revert elementwise pow.

* auto format by CI

* move conv grad functor to grad_functor; add poolNdGrad functor, fix pool in gradient_funcs to use functional api

* Revert "Access device_infer_func dynamically since this func maybe has not been registered while the op expr was constructed."

This reverts commit 07cc4a59dc62ee5bf885f3c0402cfdc3a2674721.

* Lazy construct functor to make sure that the operators has already been registered.

* fix minor bug in pool grad

* make pool_gpu_kernel stateless

* Refine function library

* Refine code style.

* refactor code

* Support add with large number of inputs.

* Support concat with any number of inputs.

* Support add with large number of inputs.

* Update oneflow/python/nn/modules/math_ops.py

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

* Refine

* Reformat

* Refine code style.

* make constant_kernel stateless (#5242)

* Migrate binary and activation ops.

* Migrate array ops.

* Add or refactor activation grad funcs.

* Add or refactor activation grad funcs.

* Revert maxpool test.

* Revert unpack all

* Fix masked fill

* Refine

* Add nn ops.

* Refine

* Refine

* Migrate conv op

* Revert changes.

* Refine code style

* Fix pooling kernel

* auto format by CI

Co-authored-by: VertexC <bob2420083992@gmail.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: Luyang <flowingsun007@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Add BUILD_BYPRODUCTS for ExternalProject_Add

* build in one step if it is ninja

* fix yml syntax

* limit max-parallel

* limit max-parallel

* rm useless

* refine

* add --shm-size=8g

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refine and add test case

* support ellipsis type slice

* refine

* refine

* support slice assign ellipsis type

* refine

* register fn to localtensor

* add type_as module

* add long module

* add more test

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* tensor slice assign supports broadcasting

Signed-off-by: daquexian <daquexian566@gmail.com>

* auto format by CI

Co-authored-by: Luyang <flowingsun007@163.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add new ones module

* add newones module test

* add newones module test

* add new_ones module doctest

* add test_case

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* refine and add test case

* support ellipsis type slice

* refine

* refine

* support slice assign ellipsis type

* refine

* register fn to localtensor

* make sure input's dtype == update's dtype

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* use_symbol_device

* use SymbolOf

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* hard_ware acc

* generate roi

* jpeg decoder fallback to cpu decoder when not jpeg

* CreateDecodeHandleFactory add target_width target_height

* refine

* fix

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add bmm module

* add bmm module test

* add doctest

* add doctest

* fix docstring bug

* fix commets

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add image decode module

* add gen_tensor_buffer module

* add testcase

* convert PIL to cv2 in testcase

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* build nightly for py39

* refine

* rfine

* revert

* refine

* address review

* refine

* max-parallel: 3

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix file is not exit

* minor fix

* minor fix

* print errno

* add test case

* refine

Co-authored-by: liufengwei <2472937968@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix_bug_in_make_parallel

* optimize

* optimize

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add files

* refine

* refine

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* optimize softmax warp impl

* fix

* add block size conf

* use ComputeType as template

* refine dispatch Smem block size

* rename fetch to load

* refine

Co-authored-by: Juncheng <liujuncheng1022@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* replace train_conf, job_conf using cfg::xx

* auto format by CI

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* fix conflict

* add tensor_buffer_ops eager

* add testcase

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Add 'nn.Mish' module and corresponding functions

* Improve the docs and unit test

* auto format by CI

* Fix the indent problems

Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* Add partial unary and math functional apis.

* Revert elementwise pow.

* auto format by CI

* Support add with large number of inputs.

* Update oneflow/python/nn/modules/math_ops.py

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

* Refine

* Migrate binary and activation ops.

* Migrate array ops.

* Add or refactor activation grad funcs.

* Add or refactor activation grad funcs.

* Revert unpack all

* Fix masked fill

* Refine

* Add nn ops.

* Refine

* Refine

* Migrate conv op

* Fix functional normalization.

* auto format by CI

* Refine code style

* align Torch params

* fix bias add error

* Support optional parameter.

* fix group bug

* add new test case

* remove useless state

* Move optional storage into namespace internal.

* add check

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: Luyang <flowingsun007@163.com>
Co-authored-by: MARD1NO <359521840@qq.com>
Co-au...