* added flatten backward

* flatten and softmax backward

* fix bug for not supporting all kinds of iterables in optimizers

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* feat(GraphAutogradEngine): add GraphAutogradEngine

* feat(GraphEngine): support backward interface

* feat(GraphEngine): support autograd.backward()

* fix(GraphEngine): some grad_fn in next_functions but can not apply

* feat(GraphEngine): support autograd.grad interface

* style(*): add JUST

* fix(GraphEngine): fix autograd.grad bugs

* test(Autograd): add flow.autograd test

* fix(GraphEngine): fix autograd.grad bug

* refine codes

* style(*): remove comments

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Dev functional interface.

* Remove repeated function_traits

* Implement add and add_scalar static functional op.

* Remove unused code

* Refactor

* Refine

* Generate and export functional apis.

* Refine

* Refine

* Refine

* Generate functional api and pybind cpp when building the project.

* Refine code style and implement normalization functor.

* Fix cmake

* Add PyYAML requirement.

* Add JUST

* Fix scalar IsIntegral and IsSigned

* Add scalar add grad func.

* Fix norm grad func to support dynamic attrs.

* Refactor math modules

* Refactor activation modules.

* Support DataType.

* Add functional range.

* Support DataType.

* Add functional argmax and flatten.

* Add functional argwhere.

* Add functional broadcast_like

* Add functional cast, zeros_like and ones_like.

* Recursive determine input tensors.

* Add functional concat, bias_add and conv2d.

* Recursive determine input tensors.

* Add functional conv2d, bias_add, eq, exp, expand, gather, dim_gather, greater, less, matmul and broadcast_matmul.

* Fix scalar_div_by_tensor grad.

* Update generate_functional_api.py

* Update generate_functional_api.py

* Add functional activation, argmax, eq, layer norm etc.

* Fix dynamic sparse_softmax_cross_entropy.

* Add functional expand_dims and where.

* Fix conversion from python object to dtype.

* Fix conversion from python object to dtype.

* Access device_infer_func dynamically since this func maybe has not been registered while the op expr was constructed.

* Add more functional apis, and fix bugs.

* Fix crash since function has been cast with wrong function signature, the converted function behavior is undefined.

* Check and throw error.

* Refine

* Use Maybe instead of throwing exception directly.

* Reformat

* Fix conv module, refactor bias_add grad func to support dynamic attrs.

* Use composed attrs when creating kernel state.

* Enable static conv op.

* Update generated functional files only when different.

* Use default target other than custom command.

* Create new kernel state if in eager mode

* make conv_kernels state less

* fix typo

* add conv grad functor; make conv_gpu_kernel stateless

* Remove unused code.

* Add partial unary and math functional apis.

* Revert elementwise pow.

* auto format by CI

* move conv grad functor to grad_functor; add poolNdGrad functor, fix pool in gradient_funcs to use functional api

* Revert "Access device_infer_func dynamically since this func maybe has not been registered while the op expr was constructed."

This reverts commit 07cc4a59dc62ee5bf885f3c0402cfdc3a2674721.

* Lazy construct functor to make sure that the operators has already been registered.

* fix minor bug in pool grad

* make pool_gpu_kernel stateless

* Refine function library

* Refine code style.

* refactor code

* Support add with large number of inputs.

* Support concat with any number of inputs.

* Support add with large number of inputs.

* Update oneflow/python/nn/modules/math_ops.py

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

* Refine

* Reformat

* Refine code style.

* make constant_kernel stateless (#5242)

* Migrate binary and activation ops.

* Migrate array ops.

* Add or refactor activation grad funcs.

* Add or refactor activation grad funcs.

* Revert maxpool test.

* Revert unpack all

* Fix masked fill

* Refine

* Add nn ops.

* Refine

* Refine

* Migrate conv op

* Revert changes.

* Refine code style

* Fix pooling kernel

* auto format by CI

Co-authored-by: VertexC <bob2420083992@gmail.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: Luyang <flowingsun007@163.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Add BUILD_BYPRODUCTS for ExternalProject_Add

* build in one step if it is ninja

* fix yml syntax

* limit max-parallel

* limit max-parallel

* rm useless

* refine

* add --shm-size=8g

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* refine and add test case

* support ellipsis type slice

* refine

* refine

* support slice assign ellipsis type

* refine

* register fn to localtensor

* add type_as module

* add long module

* add more test

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Signed-off-by: daquexian <daquexian566@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* tensor slice assign supports broadcasting

Signed-off-by: daquexian <daquexian566@gmail.com>

* auto format by CI

Co-authored-by: Luyang <flowingsun007@163.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add new ones module

* add newones module test

* add newones module test

* add new_ones module doctest

* add test_case

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* refine and add test case

* support ellipsis type slice

* refine

* refine

* support slice assign ellipsis type

* refine

* register fn to localtensor

* make sure input's dtype == update's dtype

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* use_symbol_device

* use SymbolOf

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* hard_ware acc

* generate roi

* jpeg decoder fallback to cpu decoder when not jpeg

* CreateDecodeHandleFactory add target_width target_height

* refine

* fix

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add bmm module

* add bmm module test

* add doctest

* add doctest

* fix docstring bug

* fix commets

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add image decode module

* add gen_tensor_buffer module

* add testcase

* convert PIL to cv2 in testcase

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* build nightly for py39

* refine

* rfine

* revert

* refine

* address review

* refine

* max-parallel: 3

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix file is not exit

* minor fix

* minor fix

* print errno

* add test case

* refine

Co-authored-by: liufengwei <2472937968@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix_bug_in_make_parallel

* optimize

* optimize

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* add files

* refine

* refine

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* optimize softmax warp impl

* fix

* add block size conf

* use ComputeType as template

* refine dispatch Smem block size

* rename fetch to load

* refine

Co-authored-by: Juncheng <liujuncheng1022@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* replace train_conf, job_conf using cfg::xx

* auto format by CI

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* fix conflict

* add tensor_buffer_ops eager

* add testcase

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Add 'nn.Mish' module and corresponding functions

* Improve the docs and unit test

* auto format by CI

* Fix the indent problems

Co-authored-by: Yao Chi <later@usopp.net>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* Add partial unary and math functional apis.

* Revert elementwise pow.

* auto format by CI

* Support add with large number of inputs.

* Update oneflow/python/nn/modules/math_ops.py

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

* Refine

* Migrate binary and activation ops.

* Migrate array ops.

* Add or refactor activation grad funcs.

* Add or refactor activation grad funcs.

* Revert unpack all

* Fix masked fill

* Refine

* Add nn ops.

* Refine

* Refine

* Migrate conv op

* Fix functional normalization.

* auto format by CI

* Refine code style

* align Torch params

* fix bias add error

* Support optional parameter.

* fix group bug

* add new test case

* remove useless state

* Move optional storage into namespace internal.

* add check

Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>
Co-authored-by: Luyang <flowingsun007@163.com>
Co-authored-by: MARD1NO <359521840@qq.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: ZZK <42901638+MARD1NO@users.noreply.github.com>

* update cmake

* check in fix

* add default for apple

* skip and generate_api for in mac ci

* fix yml

* fix has_grad template

* update flatbuffer

* fix align

* revert

* revert

* refine

* add FLATBUFFERS_BUILD_TESTS=OFF

* revert

* refine

* add note

* Update flatbuffers.cmake

* refine

* rm log

* FLATBUFFERS_BUILD_TESTS=OFF

* Update flatbuffers.cmake

* restore

* refine

* fix ndarry std::min

* restore

* refine

* update test config

* run doctest in cpu

* fix vm::TestResourceDescScope

* dont run doctest because it seems to involve more

* add simple ci

* refine

* refine

* add ubuntu

* refine

* refine

* add todo

* refine

* refine cache

* build oneflow if linux

* refine

* refine

* fix cache order

* refine cache key

* refine

* rm skip

* add instruction on readme

* refine

* refine

* revert

* refine

* refine

* refine

* Update CMakeLists.txt

* add

* Fix protobuf debug postfix

* refine

* fix

* refine

* refine

* refine

* refine

* add config in matrix

* refine

* refine

* refine

* fix

* add badge

* refine

* refine

* refine

* opencv diable intel ITT

* rm RelWithDebInfo

* refine

* refine

* rm intel ITT

* dont use realpath to prevent failure

* refine

* fix

* don't build Ninja with Release to reduce product

* revert

* refine order

* refine order

* refine order

* refine order

* less deps

* refine

* rm mkl note

* refine

* refine

* link py libs if building shared lib on mac

* refine

* add handy name

* refine

* refine

* refine

* refine

* refine

* refine

* quick fix

* refine

* refine

* refine

* refine

Co-authored-by: strint <xiaoyulink@gmail.com>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: liujuncheng <liujuncheng1022@gmail.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix conflict

* add sort module

* add sort testcase

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix bceloss bug about weight

* fix bceloss bug in default reduction

* fix

* fix

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>

* fix bert backward bug

* fix argwhere module bug

* align dropout module with torch

* fix acosh ci bug

* fix where module bug

* fix comments

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Remove ONNX from setup py

* refine

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* remove redundant assert and check

* split to 3 ops file

* remove useless header file

* merge ops into single file

* remove useless params

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Don't build test by default

* refine

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* fix bert backward bug

* fix argwhere module bug

* align dropout module with torch

* fix acosh ci bug

* fix l1loss bug

* fix test bug

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* Don't build test by default

* refine

* Retry distributed run for 3 times to prevent failure

* refine

* revert changes unwanted

* address review

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>

* move btob to op

* auto format by CI

Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>