[mlir] Refactor the Parser library in preparation for an MLIR binary format

The current Parser library is solely focused on providing API for
the textual MLIR format, but MLIR will soon also provide

[mlir] Refactor the Parser library in preparation for an MLIR binary format

The current Parser library is solely focused on providing API for
the textual MLIR format, but MLIR will soon also provide a binary
format. This commit renames the current Parser library to AsmParser to
better correspond to what the library is actually intended for. A new
Parser library is added which will act as a unified parser interface
between both text and binary formats. Most parser clients are
unaffected, given that the unified interface is essentially the same as
the current interface. Only clients that rely on utilizing the
AsmParserState, or those that want to parse Attributes/Types need to be
updated to point to the AsmParser library.

Differential Revision: https://reviews.llvm.org/D129605

show more ...

[mlir] Overhaul C/Python registration APIs to properly scope registration/loading activities.

Since the very first commits, the Python and C MLIR APIs have had mis-placed registration/load functiona

[mlir] Overhaul C/Python registration APIs to properly scope registration/loading activities.

Since the very first commits, the Python and C MLIR APIs have had mis-placed registration/load functionality for dialects, extensions, etc. This was done pragmatically in order to get bootstrapped and then just grew in. Downstreams largely bypass and do their own thing by providing various APIs to register things they need. Meanwhile, the C++ APIs have stabilized around this and it would make sense to follow suit.

The thing we have observed in canonical usage by downstreams is that each downstream tends to have native entry points that configure its installation to its preferences with one-stop APIs. This patch leans in to this approach with `RegisterEverything.h` and `mlir._mlir_libs._mlirRegisterEverything` being the one-stop entry points for the "upstream packages". The `_mlir_libs.__init__.py` now allows customization of the environment and Context by adding "initialization modules" to the `_mlir_libs` package. If present, `_mlirRegisterEverything` is treated as such a module. Others can be added by downstreams by adding a `_site_initialize_{i}.py` module, where '{i}' is a number starting with zero. The number will be incremented and corresponding module loaded until one is not found. Initialization modules can:

* Perform load time customization to the global environment (i.e. registering passes, hooks, etc).
* Define a `register_dialects(registry: DialectRegistry)` function that can extend the `DialectRegistry` that will be used to bootstrap the `Context`.
* Define a `context_init_hook(context: Context)` function that will be added to a list of callbacks which will be invoked after dialect registration during `Context` initialization.

Note that the `MLIRPythonExtension.RegisterEverything` is not included by default when building a downstream (its corresponding behavior was prior). For downstreams which need the default MLIR initialization to take place, they must add this back in to their Python CMake build just like they add their own components (i.e. to `add_mlir_python_common_capi_library` and `add_mlir_python_modules`). It is perfectly valid to not do this, in which case, only the things explicitly depended on and initialized by downstreams will be built/packaged. If the downstream has not been set up for this, it is recommended to simply add this back for the time being and pay the build time/package size cost.

CMake changes:
* `MLIRCAPIRegistration` -> `MLIRCAPIRegisterEverything` (renamed to signify what it does and force an evaluation: a number of places were incidentally linking this very expensive target)
* `MLIRPythonSoure.Passes` removed (without replacement: just drop)
* `MLIRPythonExtension.AllPassesRegistration` removed (without replacement: just drop)
* `MLIRPythonExtension.Conversions` removed (without replacement: just drop)
* `MLIRPythonExtension.Transforms` removed (without replacement: just drop)

Header changes:
* `mlir-c/Registration.h` is deleted. Dialect registration functionality is now in `IR.h`. Registration of upstream features are in `mlir-c/RegisterEverything.h`. When updating MLIR and a couple of downstreams, I found that proper usage was commingled so required making a choice vs just blind S&R.

Python APIs removed:
* mlir.transforms and mlir.conversions (previously only had an __init__.py which indirectly triggered `mlirRegisterTransformsPasses()` and `mlirRegisterConversionPasses()` respectively). Downstream impact: Remove these imports if present (they now happen as part of default initialization).
* mlir._mlir_libs._all_passes_registration, mlir._mlir_libs._mlirTransforms, mlir._mlir_libs._mlirConversions. Downstream impact: None expected (these were internally used).

C-APIs changed:
* mlirRegisterAllDialects(MlirContext) now takes an MlirDialectRegistry instead. It also used to trigger loading of all dialects, which was already marked with a TODO to remove -- it no longer does, and for direct use, dialects must be explicitly loaded. Downstream impact: Direct C-API users must ensure that needed dialects are loaded or call `mlirContextLoadAllAvailableDialects(MlirContext)` to emulate the prior behavior. Also see the `ir.c` test case (e.g. ` mlirContextGetOrLoadDialect(ctx, mlirStringRefCreateFromCString("func"));`).
* mlirDialectHandle* APIs were moved from Registration.h (which now is restricted to just global/upstream registration) to IR.h, arguably where it should have been. Downstream impact: include correct header (likely already doing so).

C-APIs added:
* mlirContextLoadAllAvailableDialects(MlirContext): Corresponds to C++ API with the same purpose.

Python APIs added:
* mlir.ir.DialectRegistry: Mapping for an MlirDialectRegistry.
* mlir.ir.Context.append_dialect_registry(MlirDialectRegistry)
* mlir.ir.Context.load_all_available_dialects()
* mlir._mlir_libs._mlirAllRegistration: New native extension that exposes a `register_dialects(MlirDialectRegistry)` entry point and performs all upstream pass/conversion/transforms registration on init. In this first step, we eagerly load this as part of the __init__.py and use it to monkey patch the Context to emulate prior behavior.
* Type caster and capsule support for MlirDialectRegistry

This should make it possible to build downstream Python dialects that only depend on a subset of MLIR. See: https://github.com/llvm/llvm-project/issues/56037

Here is an example PR, minimally adapting IREE to these changes: https://github.com/iree-org/iree/pull/9638/files In this situation, IREE is opting to not link everything, since it is already configuring the Context to its liking. For projects that would just like to not think about it and pull in everything, add `MLIRPythonExtension.RegisterEverything` to the list of Python sources getting built, and the old behavior will continue.

Reviewed By: mehdi_amini, ftynse

Differential Revision: https://reviews.llvm.org/D128593

show more ...

[MLIR] Add block detach func to CAPI and use it in Python bindings

Adds `mlirBlockDetach` to the CAPI to remove a block from its parent
region. Use it in the Python bindings to implement
`Block.appe

[MLIR] Add block detach func to CAPI and use it in Python bindings

Adds `mlirBlockDetach` to the CAPI to remove a block from its parent
region. Use it in the Python bindings to implement
`Block.append_to(region)`.

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D123165

show more ...

[mlir][capi] Add external pass creation to MLIR C-API

Adds the ability to create external passes using the C-API. This allows passes
to be written in C or languages that use the C-bindings.

Differe

[mlir][capi] Add external pass creation to MLIR C-API

Adds the ability to create external passes using the C-API. This allows passes
to be written in C or languages that use the C-bindings.

Differential Revision: https://reviews.llvm.org/D121866

show more ...

Update more `parseSourceString()` call sites.

Change to non-deprecated function template (see D121075).

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D121102

[mlir][NFC] Move Parser.h to Parser/

There is no reason for this file to be at the top-level, and
its current placement predates the Parser/ folder's existence.

Differential Revision: https://revie

[mlir][NFC] Move Parser.h to Parser/

There is no reason for this file to be at the top-level, and
its current placement predates the Parser/ folder's existence.

Differential Revision: https://reviews.llvm.org/D121024

show more ...

[mlir][capi] Add DialectRegistry to MLIR C-API

Exposes mlir::DialectRegistry to the C API as MlirDialectRegistry along with
helper functions. A hook has been added to MlirDialectHandle that inserts

[mlir][capi] Add DialectRegistry to MLIR C-API

Exposes mlir::DialectRegistry to the C API as MlirDialectRegistry along with
helper functions. A hook has been added to MlirDialectHandle that inserts
the dialect into a registry.

A future possible change is removing mlirDialectHandleRegisterDialect in
favor of using mlirDialectHandleInsertDialect, which it is now implemented with.

Differential Revision: https://reviews.llvm.org/D118293

show more ...

Replace OwningModuleRef with OwningOpRef<ModuleOp>

This addresses a TODO in BuiltinOps.h.

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D118574

[mlir] Make locations required when adding/creating block arguments

BlockArguments gained the ability to have locations attached a while ago, but they
have always been optional. This goes against th

[mlir] Make locations required when adding/creating block arguments

BlockArguments gained the ability to have locations attached a while ago, but they
have always been optional. This goes against the core tenant of MLIR where location
information is a requirement, so this commit updates the API to require locations.

Fixes #53279

Differential Revision: https://reviews.llvm.org/D117633

show more ...

[mlir] Finish removing Identifier from the C++ API

There have been a few API pieces remaining to allow for a smooth transition for
downstream users, but these have been up for a few months now. Afte

[mlir] Finish removing Identifier from the C++ API

There have been a few API pieces remaining to allow for a smooth transition for
downstream users, but these have been up for a few months now. After this only
the C API will have reference to "Identifier", but those will be reworked in a followup.

The main updates are:
* Identifier -> StringAttr
* StringAttr::get requires the context as the first parameter
- i.e. `Identifier::get("...", ctx)` -> `StringAttr::get(ctx, "...")`

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D116626

show more ...

Fix clang-tidy issues in mlir/ (NFC)

Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D115956

[mlir][python] Implement more SymbolTable methods.

* set_symbol_name, get_symbol_name, set_visibility, get_visibility, replace_all_symbol_uses, walk_symbol_tables
* In integrations I've been doing,

[mlir][python] Implement more SymbolTable methods.

* set_symbol_name, get_symbol_name, set_visibility, get_visibility, replace_all_symbol_uses, walk_symbol_tables
* In integrations I've been doing, I've been reaching for all of these to do both general IR manipulation and module merging.
* I don't love the replace_all_symbol_uses underlying APIs since they necessitate SYMBOL_COUNT walks and have various sharp edges. I'm hoping that whatever emerges eventually for this can still retain this simple API as a one-shot.

Differential Revision: https://reviews.llvm.org/D114687

show more ...

[mlir] Convert NamedAttribute to be a class

NamedAttribute is currently represented as an std::pair, but this
creates an extremely clunky .first/.second API. This commit
converts it to a class, with

[mlir] Convert NamedAttribute to be a class

NamedAttribute is currently represented as an std::pair, but this
creates an extremely clunky .first/.second API. This commit
converts it to a class, with better accessors (getName/getValue)
and also opens the door for more convenient API in the future.

Differential Revision: https://reviews.llvm.org/D113956

show more ...

[mlir] Refactor AbstractOperation and OperationName

The current implementation is quite clunky; OperationName stores either an Identifier
or an AbstractOperation that corresponds to an operation. Th

[mlir] Refactor AbstractOperation and OperationName

The current implementation is quite clunky; OperationName stores either an Identifier
or an AbstractOperation that corresponds to an operation. This has several problems:

* OperationNames created before and after an operation are registered are different
* Accessing the identifier name/dialect/etc. from an OperationName are overly branchy
- they need to dyn_cast a PointerUnion to check the state

This commit refactors this such that we create a single information struct for every
operation name, even operations that aren't registered yet. When an OperationName is
created for an unregistered operation, we only populate the name field. When the
operation is registered, we populate the remaining fields. With this we now have two
new classes: OperationName and RegisteredOperationName. These both point to the
same underlying operation information struct, but only RegisteredOperationName can
assume that the operation is actually registered. This leads to a much cleaner API, and
we can also move some AbstractOperation functionality directly to OperationName.

Differential Revision: https://reviews.llvm.org/D114049

show more ...

[mlir][NFC] Replace references to Identifier with StringAttr

This is part of the replacement of Identifier with StringAttr.

Differential Revision: https://reviews.llvm.org/D113953

[mlir] Replace usages of Identifier with StringAttr

Identifier and StringAttr essentially serve the same purpose, i.e. to hold a string value. Keeping these seemingly identical pieces of functionali

[mlir] Replace usages of Identifier with StringAttr

Identifier and StringAttr essentially serve the same purpose, i.e. to hold a string value. Keeping these seemingly identical pieces of functionality separate has caused problems in certain situations:

* Identifier has nice accessors that StringAttr doesn't
* Identifier can't be used as an Attribute, meaning strings are often duplicated between Identifier/StringAttr (e.g. in PDL)

The only thing that Identifier has that StringAttr doesn't is support for caching a dialect that is referenced by the string (e.g. dialect.foo). This functionality is added to StringAttr, as this is useful for StringAttr in generally the same ways it was useful for Identifier.

Differential Revision: https://reviews.llvm.org/D113536

show more ...

[mlir-c] Add Region iterators matching Block & Operation ones

Enables using the same iterator interface to these even though underlying storage is different.

Differential Revision: https://reviews.

[mlir-c] Add Region iterators matching Block & Operation ones

Enables using the same iterator interface to these even though underlying storage is different.

Differential Revision: https://reviews.llvm.org/D113512

show more ...

[mlir] provide C API and Python bindings for symbol tables

Symbol tables are a largely useful top-level IR construct, for example, they
make it easy to access functions in a module by name instead o

[mlir] provide C API and Python bindings for symbol tables

Symbol tables are a largely useful top-level IR construct, for example, they
make it easy to access functions in a module by name instead of traversing the
list of module's operations to find the corresponding function.

Depends On D112886

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D112821

show more ...

[mlir][python] allow for detaching operations from a block

Provide support for removing an operation from the block that contains it and
moving it back to detached state. This allows for the operati

[mlir][python] allow for detaching operations from a block

Provide support for removing an operation from the block that contains it and
moving it back to detached state. This allows for the operation to be moved to
a different block, a common IR manipulation for, e.g., module merging.

Also fix a potential one-past-end iterator dereference in Operation::moveAfter
discovered in the process.

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D112700

show more ...

[mlir][python] Add 'loc' property to ops

Add a read-only `loc` property to Operation and OpView

Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D111972

[mlir][capi] Add TypeID to MLIR C-API

Exposes mlir::TypeID to the C API as MlirTypeID along with various accessors
and helper functions.

Differential Revision: https://reviews.llvm.org/D110897

[mlir-c] Add getting fused loc

For creating a fused loc using array of locations and metadata.

Differential Revision: https://reviews.llvm.org/D110022

[mlir][capi] Add NameLoc

Add method to get NameLoc. Treat null child location as unknown to avoid
needing to create UnknownLoc in C API where child loc is not needed.

Differential Revision: https:/

[mlir][capi] Add NameLoc

Add method to get NameLoc. Treat null child location as unknown to avoid
needing to create UnknownLoc in C API where child loc is not needed.

Differential Revision: https://reviews.llvm.org/D108678

show more ...

[mlir][python] Extend C/Python API to be usable for CFG construction.

* It is pretty clear that no one has tried this yet since it was both incomplete and broken.
* Fixes a symbol hiding issues keep

[mlir][python] Extend C/Python API to be usable for CFG construction.

* It is pretty clear that no one has tried this yet since it was both incomplete and broken.
* Fixes a symbol hiding issues keeping even the generic builder from constructing an operation with successors.
* Adds ODS support for successors.
* Adds CAPI `mlirBlockGetParentRegion`, `mlirRegionEqual` + tests (and missing test for `mlirBlockGetParentOperation`).
* Adds Python property: `Block.region`.
* Adds Python methods: `Block.create_before` and `Block.create_after`.
* Adds Python property: `InsertionPoint.block`.
* Adds new blocks.py test to verify a plausible CFG construction case.

Differential Revision: https://reviews.llvm.org/D108898

show more ...

Surface clone APIs in CAPI

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D102987