[mlir] Overhaul C/Python registration APIs to properly scope registration/loading activities.Since the very first commits, the Python and C MLIR APIs have had mis-placed registration/load functiona
[mlir] Overhaul C/Python registration APIs to properly scope registration/loading activities.Since the very first commits, the Python and C MLIR APIs have had mis-placed registration/load functionality for dialects, extensions, etc. This was done pragmatically in order to get bootstrapped and then just grew in. Downstreams largely bypass and do their own thing by providing various APIs to register things they need. Meanwhile, the C++ APIs have stabilized around this and it would make sense to follow suit.The thing we have observed in canonical usage by downstreams is that each downstream tends to have native entry points that configure its installation to its preferences with one-stop APIs. This patch leans in to this approach with `RegisterEverything.h` and `mlir._mlir_libs._mlirRegisterEverything` being the one-stop entry points for the "upstream packages". The `_mlir_libs.__init__.py` now allows customization of the environment and Context by adding "initialization modules" to the `_mlir_libs` package. If present, `_mlirRegisterEverything` is treated as such a module. Others can be added by downstreams by adding a `_site_initialize_{i}.py` module, where '{i}' is a number starting with zero. The number will be incremented and corresponding module loaded until one is not found. Initialization modules can:* Perform load time customization to the global environment (i.e. registering passes, hooks, etc).* Define a `register_dialects(registry: DialectRegistry)` function that can extend the `DialectRegistry` that will be used to bootstrap the `Context`.* Define a `context_init_hook(context: Context)` function that will be added to a list of callbacks which will be invoked after dialect registration during `Context` initialization.Note that the `MLIRPythonExtension.RegisterEverything` is not included by default when building a downstream (its corresponding behavior was prior). For downstreams which need the default MLIR initialization to take place, they must add this back in to their Python CMake build just like they add their own components (i.e. to `add_mlir_python_common_capi_library` and `add_mlir_python_modules`). It is perfectly valid to not do this, in which case, only the things explicitly depended on and initialized by downstreams will be built/packaged. If the downstream has not been set up for this, it is recommended to simply add this back for the time being and pay the build time/package size cost.CMake changes:* `MLIRCAPIRegistration` -> `MLIRCAPIRegisterEverything` (renamed to signify what it does and force an evaluation: a number of places were incidentally linking this very expensive target)* `MLIRPythonSoure.Passes` removed (without replacement: just drop)* `MLIRPythonExtension.AllPassesRegistration` removed (without replacement: just drop)* `MLIRPythonExtension.Conversions` removed (without replacement: just drop)* `MLIRPythonExtension.Transforms` removed (without replacement: just drop)Header changes:* `mlir-c/Registration.h` is deleted. Dialect registration functionality is now in `IR.h`. Registration of upstream features are in `mlir-c/RegisterEverything.h`. When updating MLIR and a couple of downstreams, I found that proper usage was commingled so required making a choice vs just blind S&R.Python APIs removed: * mlir.transforms and mlir.conversions (previously only had an __init__.py which indirectly triggered `mlirRegisterTransformsPasses()` and `mlirRegisterConversionPasses()` respectively). Downstream impact: Remove these imports if present (they now happen as part of default initialization). * mlir._mlir_libs._all_passes_registration, mlir._mlir_libs._mlirTransforms, mlir._mlir_libs._mlirConversions. Downstream impact: None expected (these were internally used).C-APIs changed: * mlirRegisterAllDialects(MlirContext) now takes an MlirDialectRegistry instead. It also used to trigger loading of all dialects, which was already marked with a TODO to remove -- it no longer does, and for direct use, dialects must be explicitly loaded. Downstream impact: Direct C-API users must ensure that needed dialects are loaded or call `mlirContextLoadAllAvailableDialects(MlirContext)` to emulate the prior behavior. Also see the `ir.c` test case (e.g. ` mlirContextGetOrLoadDialect(ctx, mlirStringRefCreateFromCString("func"));`). * mlirDialectHandle* APIs were moved from Registration.h (which now is restricted to just global/upstream registration) to IR.h, arguably where it should have been. Downstream impact: include correct header (likely already doing so).C-APIs added: * mlirContextLoadAllAvailableDialects(MlirContext): Corresponds to C++ API with the same purpose.Python APIs added: * mlir.ir.DialectRegistry: Mapping for an MlirDialectRegistry. * mlir.ir.Context.append_dialect_registry(MlirDialectRegistry) * mlir.ir.Context.load_all_available_dialects() * mlir._mlir_libs._mlirAllRegistration: New native extension that exposes a `register_dialects(MlirDialectRegistry)` entry point and performs all upstream pass/conversion/transforms registration on init. In this first step, we eagerly load this as part of the __init__.py and use it to monkey patch the Context to emulate prior behavior. * Type caster and capsule support for MlirDialectRegistryThis should make it possible to build downstream Python dialects that only depend on a subset of MLIR. See: https://github.com/llvm/llvm-project/issues/56037Here is an example PR, minimally adapting IREE to these changes: https://github.com/iree-org/iree/pull/9638/files In this situation, IREE is opting to not link everything, since it is already configuring the Context to its liking. For projects that would just like to not think about it and pull in everything, add `MLIRPythonExtension.RegisterEverything` to the list of Python sources getting built, and the old behavior will continue.Reviewed By: mehdi_amini, ftynseDifferential Revision: https://reviews.llvm.org/D128593
show more ...
[mlir] Fix python bindings build on Windows in DebugCurrently, building mlir with the python bindings enabled on Windows in Debug is broken because pybind11, python and cmake don't like to play tog
[mlir] Fix python bindings build on Windows in DebugCurrently, building mlir with the python bindings enabled on Windows in Debug is broken because pybind11, python and cmake don't like to play together. This change normalizes how the three interact, so that the builds can now run and succeed.The main issue is that python and cmake both make assumptions about which libraries are needed in a Windows build based on the flavor.- cmake assumes that a debug (or a debug-like) flavor of the build will always require pythonX_d.lib and provides no option/hint to tell it to use a different library. cmake does find both the debug and release versions, but then uses the debug library.- python (specifically pyconfig.h and by extension python.h) hardcodes the dependency on pythonX_d.lib or pythonX.lib depending on whether `_DEBUG` is defined. This is NOT transparent - it does not show up anywhere in the build logs until the link step fails with `pythonX_d.lib is missing` (or `pythonX.lib is missing`)- pybind11 tries to "fix" this by implementing a workaround - unless Py_DEBUG is defined, `_DEBUG` is explicitly undefined right before including python headers. This also requires some windows headers to be included differently, so while clever, this is a non-trivial workaround.mlir itself includes the pybind11 headers (which contain the workaround) AS WELL AS python.h, essentially always requiring both pythonX.lib and pythonX_d.lib for linking. cmake explicitly only adds one or the other, so the build fails.This change does a couple of things:- In the cmake files, explicitly add the release version of the python library on Windows builds regardless of flavor. Since Py_DEBUG is not defined, pybind11 will always require release and it will be satisfied- To satisfy python as well, this change removes any explicit inclusions of Python.h on Windows instead relying on the fact that pybind11 headers will bring in what is neededThere are a few additional things that we could do but I rejected as unnecessary at this time:- define Py_DEBUG based on the CMAKE_BUILD_TYPE - this will *mostly* work, we'd have to think through multiconfig generators like VS, but it's possible. There doesn't seem to be a need to link against debug python at the moment, so I chose not to overcomplicate the build and always default to release- similar to above, but define Py_DEBUG based on the CMAKE_BUILD_TYPE *as well as* the presence of the debug python library (`Python3_LIBRARY_DEBUG`). Similar to above, this seems unnecessary right now. I think it's slightly better than above because most people don't actually have the debug version of python installed, so this would prevent breaks in that case.- similar to the two above, but add a cmake variable to control the logic- implement the pybind11 workaround directly in mlir (specifically in Interop.h) so that Python.h can still be included directly. This seems prone to error and a pain to maintain in lock step with pybind11- reorganize how the pybind11 headers are included and place at least one of them in Interop.h directly, so that the header has all of its dependencies included as was the original intention. I decided against this because it really doesn't need pybind11 logic and it's always included after pybind11 is, so we don't necessarily need the python includesReviewed By: stellaraccidentDifferential Revision: https://reviews.llvm.org/D125284
Apply clang-tidy fixes for bugprone-macro-parentheses in Interop.h (NFC)
[mlir][python] Makes C++ extension code relocatable by way of a macro.* Resolves a TODO by making this configurable by downstreams.* This seems to be the last thing allowing full use of the Python
[mlir][python] Makes C++ extension code relocatable by way of a macro.* Resolves a TODO by making this configurable by downstreams.* This seems to be the last thing allowing full use of the Python bindings as a library within another project (i.e. be embedding them).Differential Revision: https://reviews.llvm.org/D108523
[MLIR][Python] Add capsule methods for pybind11 to PyValue.Add the `getCapsule()` and `createFromCapsule()` methods to thePyValue class, as well as the necessary interoperability.Reviewed By: st
[MLIR][Python] Add capsule methods for pybind11 to PyValue.Add the `getCapsule()` and `createFromCapsule()` methods to thePyValue class, as well as the necessary interoperability.Reviewed By: stellaraccidentDifferential Revision: https://reviews.llvm.org/D101090
Add basic JIT Python BindingsThis offers the ability to create a JIT and invoke a function by passingctypes pointers to the argument and the result.Differential Revision: https://reviews.llvm.or
Add basic JIT Python BindingsThis offers the ability to create a JIT and invoke a function by passingctypes pointers to the argument and the result.Differential Revision: https://reviews.llvm.org/D97523
[mlir] Add Python bindings for IntegerSetThis follows up on the introduction of C API for the same object and is similarto AffineExpr and AffineMap.Reviewed By: stellaraccidentDifferential Rev
[mlir] Add Python bindings for IntegerSetThis follows up on the introduction of C API for the same object and is similarto AffineExpr and AffineMap.Reviewed By: stellaraccidentDifferential Revision: https://reviews.llvm.org/D95437
[mlir] Add Python bindings for AffineExprThis adds the Python bindings for AffineExpr and a couple of utility functionsto the C API. AffineExpr is a top-level context-owned object and is modeleds
[mlir] Add Python bindings for AffineExprThis adds the Python bindings for AffineExpr and a couple of utility functionsto the C API. AffineExpr is a top-level context-owned object and is modeledsimilarly to attributes and types. It is required, e.g., to build layout mapsof the built-in memref type.Reviewed By: mehdi_aminiDifferential Revision: https://reviews.llvm.org/D94225
[mlir][Python] Initial Affine Map Python Bindings.- Add `PyAffineMap` to wrap around `MlirAffineMap`.- Add `mlirPythonAffineMapToCapsule` and `mlirPythonCapsuleToAffineMap` to interoperate with py
[mlir][Python] Initial Affine Map Python Bindings.- Add `PyAffineMap` to wrap around `MlirAffineMap`.- Add `mlirPythonAffineMapToCapsule` and `mlirPythonCapsuleToAffineMap` to interoperate with python capsule.- Add and test some simple bindings of `PyAffineMap`.Differential Revision: https://reviews.llvm.org/D93200
[mlir][Python] Python API cleanups and additions found during code audit.* Add capsule get/create for Attribute and Type, which already had capsule interop defined.* Add capsule interop and get/cr
[mlir][Python] Python API cleanups and additions found during code audit.* Add capsule get/create for Attribute and Type, which already had capsule interop defined.* Add capsule interop and get/create for Location.* Add Location __eq__.* Use get() and implicit cast to go from PyAttribute, PyType, PyLocation to MlirAttribute, MlirType, MlirLocation (bundled with this change because I didn't want to continue the pattern one more time).Differential Revision: https://reviews.llvm.org/D92283
[mlir][Python] Add additional capsule interop types.Differential Revision: https://reviews.llvm.org/D92144
[mlir][Python] Allow PassManager to interop with the capsule APIs.* Used in npcomp to cast Python objects via the C-API.Differential Revision: https://reviews.llvm.org/D91232
Switch from C-style comments `/* ... */` to C++ style `//` (NFC)This is mostly a scripted update, it may not be perfect.function replace() { FROM=$1 TO=$2 git grep "$FROM" $REPO_PATH |cut -
Switch from C-style comments `/* ... */` to C++ style `//` (NFC)This is mostly a scripted update, it may not be perfect.function replace() { FROM=$1 TO=$2 git grep "$FROM" $REPO_PATH |cut -f 1 -d : | sort -u | \ while read file; do sed -i "s#$FROM#$TO#" $file ; done}replace '|\*===----------------------------------------------------------------------===\*|$' '//===----------------------------------------------------------------------===//'replace '^/\* =' '//=='replace '^/\*=' '//='replace '^\\\*=' '//='replace '^|\*' '//'replace ' \*|$' ''replace '=\*\\$' '=//'replace '== \*/$' '===//'replace '==\*/$' '==//'replace '^/\*\*\(.*\)\*/$' '///\1'replace '^/\*\(.*\)\*/$' '//\1'replace '//============================================================================//' '//===----------------------------------------------------------------------===//'Differential Revision: https://reviews.llvm.org/D90732
[mlir][Python] Add missing capsule->module and Context.create_module.* Extends Context/Operation interning to cover Module as well.* Implements Module.context, Attribute.context, Type.context, and
[mlir][Python] Add missing capsule->module and Context.create_module.* Extends Context/Operation interning to cover Module as well.* Implements Module.context, Attribute.context, Type.context, and Location.context back-references (facilitated testing and also on the TODO list).* Adds method to create an empty Module.* Discovered missing in npcomp.Differential Revision: https://reviews.llvm.org/D89294
Adds MLIR C-API for marshaling Python capsules.* Providing stable, C-accessible definitions for bridging MLIR Python<->C APIs, we eliminate inter-extension dependencies (i.e. they can all share a d
Adds MLIR C-API for marshaling Python capsules.* Providing stable, C-accessible definitions for bridging MLIR Python<->C APIs, we eliminate inter-extension dependencies (i.e. they can all share a diamond dependency on the MLIR C-API).* Just provides accessors for context and module right now.* Needed in NPComp in ~a week or so for high level Torch APIs.Differential Revision: https://reviews.llvm.org/D88426