[mlir][tablegen] Generate default attr values in Python bindings

When specifying an op attribute with a default value (via DefaultValuedAttr), the default value is a string of C++ code. In the gener

[mlir][tablegen] Generate default attr values in Python bindings

When specifying an op attribute with a default value (via DefaultValuedAttr), the default value is a string of C++ code. In the general case, the default value of such an attribute cannot be translated to Python when generating the bindings. However, we can hard-code default Python values for frequently-used C++ default values.

This change adds a Python default value for empty ArrayAttrs.

Differential Revision: https://reviews.llvm.org/D127750

show more ...

Apply clang-tidy fixes for llvm-else-after-return in OpPythonBindingGen.cpp (NFC)

[mlir] provide Python bindings for the Transform dialect

Python bindings for extensions of the Transform dialect are defined in separate
Python source files that can be imported on-demand, i.e., tha

[mlir] provide Python bindings for the Transform dialect

Python bindings for extensions of the Transform dialect are defined in separate
Python source files that can be imported on-demand, i.e., that are not imported
with the "main" transform dialect. This requires a minor addition to the
ODS-based bindings generator. This approach is consistent with the current
model for downstream projects that are expected to bundle MLIR Python bindings:
such projects can include their custom extensions into the bundle similarly to
how they include their dialects.

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D126208

show more ...

Apply clang-tidy fixes for llvm-else-after-return in OpPythonBindingGen.cpp (NFC)

[mlir][tblgen][ods][python] Use keyword-only arguments for optional builder arguments in generated Python bindings

This diff modifies `mlir-tblgen` to generate Python Operation class `__init__()`
fu

[mlir][tblgen][ods][python] Use keyword-only arguments for optional builder arguments in generated Python bindings

This diff modifies `mlir-tblgen` to generate Python Operation class `__init__()`
functions that use Python keyword-only arguments.

Previously, all `__init__()` function arguments were positional. Python code to
create MLIR Operations was required to provide values for ALL builder arguments,
including optional arguments (attributes and operands). Callers that did not
provide, for example, an optional attribute would be forced to provide `None`
as an argument for EACH optional attribute. Proposed changes in this diff use
`tblgen` record information (as provided by ODS) to generate keyword arguments
for:
- optional operands
- optional attributes (which includes unit attributes)
- default-valued attributes

These `__init__()` function keyword arguments have default `None` values (i.e.
the argument form is `optionalAttr=None`), allowing callers to create Operations
more easily.

Note that since optional arguments become keyword-only arguments (since they are
placed after the bare `*` argument), this diff will require ALL optional
operands and attributes to be provided using explicit keyword syntax. This may,
in the short term, break any out-of-tree Python code that provided values via
positional arguments. However, in the long term, it seems that requiring
keywords for optional arguments will be more robust to operation changes that
add arguments.

Tests were modified to reflect the updated Operation builder calling convention.

This diff partially addresses the requests made in the github issue below.

https://github.com/llvm/llvm-project/issues/54932

Reviewed By: stellaraccident, mikeurbach

Differential Revision: https://reviews.llvm.org/D124717

show more ...

Apply clang-tidy fixes for performance-for-range-copy to MLIR (NFC)

Differential Revision: https://reviews.llvm.org/D116248

[mlir][Python] Fix generation of accessors for Optional

Previously, in case there was only one `Optional` operand/result within
the list, we would always return `None` from the accessor, e.g., for a

[mlir][Python] Fix generation of accessors for Optional

Previously, in case there was only one `Optional` operand/result within
the list, we would always return `None` from the accessor, e.g., for a
single optional result we would generate:

```
return self.operation.results[0] if len(self.operation.results) > 1 else None
```

But what we really want is to return `None` only if the length of
`results` is smaller than the total number of element groups (i.e.,
the optional operand/result is in fact missing).

This commit also renames a few local variables in the generator to make
the distinction between `isVariadic()` and `isVariableLength()` a bit
more clear.

Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D113855

show more ...

[mlir][python] fix constructor generation for optional operands in presence of segment attribute

The ODS-based Python op bindings generator has been generating incorrect
specification of the operand

[mlir][python] fix constructor generation for optional operands in presence of segment attribute

The ODS-based Python op bindings generator has been generating incorrect
specification of the operand segment in presence if both optional and variadic
operand groups: optional groups were treated as variadic whereas they require
separate treatement. Make sure it is the case. Also harden the tests around
generated op constructors as they could hitherto accept the code for both
optional and variadic arguments.

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D113259

show more ...

[mlir][python] Infer result types in generated constructors whenever possible

In several cases, operation result types can be unambiguously inferred from
operands and attributes at operation constru

[mlir][python] Infer result types in generated constructors whenever possible

In several cases, operation result types can be unambiguously inferred from
operands and attributes at operation construction time. Stop requiring the user
to provide these types as arguments in the ODS-generated constructors in Python
bindings. In particular, handle the SameOperandAndResultTypes and
FirstAttrDerivedResultType traits as well as InferTypeOpInterface using the
recently added interface support. This is a significant usability improvement
for IR construction, similar to what C++ ODS provides.

Depends On D111656

Reviewed By: gysit

Differential Revision: https://reviews.llvm.org/D111811

show more ...

[mlir] fix region property generation in python bindings

[mlir][python] Better support for variadic regions in Python bindings

Improve support for variadic regions in ODS-generated operation view classes.
In particular, make generated constructors take an

[mlir][python] Better support for variadic regions in Python bindings

Improve support for variadic regions in ODS-generated operation view classes.
In particular, make generated constructors take an extra argument that
specifies the number of variadic regions if the operation has them. Previously,
there was no mechanism to specify a non-zero number of variadic regions. Also
generate named accessors to regions.

Reviewed By: gysit

Differential Revision: https://reviews.llvm.org/D111783

show more ...

[mlir][python] support taking ops instead of values in op constructors

Introduce support for accepting ops instead of values when constructing ops. A
single-result op can be used instead of a value,

[mlir][python] support taking ops instead of values in op constructors

Introduce support for accepting ops instead of values when constructing ops. A
single-result op can be used instead of a value, including in lists of values,
and any op can be used instead of a list of values. This is similar to, but
more powerful, than the C++ API that allows for implicitly casting an OpType to
Value if it is statically known to have a single result - the cast in Python is
based on the op dynamically having a single result, and also handles the
multi-result case. This allows to build IR in a more concise way:

op = dialect.produce_multiple_results()
other = dialect.produce_single_result()
dialect.consume_multiple_results(other, op)

instead of having to access the results manually

op = dialect.produce.multiple_results()
other = dialect.produce_single_result()
dialect.consume_multiple_results(other.result, op.operation.results)

The dispatch is implemented directly in Python and is triggered automatically
for autogenerated OpView subclasses. Extension OpView classes should use the
functions provided in ods_common.py if they want to implement this behavior.
An alternative could be to implement the dispatch in the C++ bindings code, but
it would require to forward opaque types through all Python functions down to a
binding call, which makes it hard to inspect them in Python, e.g., to obtain
the types of values.

Reviewed By: gysit

Differential Revision: https://reviews.llvm.org/D111306

show more ...

[mlir][python] Extend C/Python API to be usable for CFG construction.

* It is pretty clear that no one has tried this yet since it was both incomplete and broken.
* Fixes a symbol hiding issues keep

[mlir][python] Extend C/Python API to be usable for CFG construction.

* It is pretty clear that no one has tried this yet since it was both incomplete and broken.
* Fixes a symbol hiding issues keeping even the generic builder from constructing an operation with successors.
* Adds ODS support for successors.
* Adds CAPI `mlirBlockGetParentRegion`, `mlirRegionEqual` + tests (and missing test for `mlirBlockGetParentOperation`).
* Adds Python property: `Block.region`.
* Adds Python methods: `Block.create_before` and `Block.create_after`.
* Adds Python property: `InsertionPoint.block`.
* Adds new blocks.py test to verify a plausible CFG construction case.

Differential Revision: https://reviews.llvm.org/D108898

show more ...

[mlir] Set the namespace of the BuiltinDialect to 'builtin'

Historically the builtin dialect has had an empty namespace. This has unfortunately created a very awkward situation, where many utilities

[mlir] Set the namespace of the BuiltinDialect to 'builtin'

Historically the builtin dialect has had an empty namespace. This has unfortunately created a very awkward situation, where many utilities either have to special case the empty namespace, or just don't work at all right now. This revision adds a namespace to the builtin dialect, and starts to cleanup some of the utilities to no longer handle empty namespaces. For now, the assembly form of builtin operations does not require the `builtin.` prefix. (This should likely be re-evaluated though)

Differential Revision: https://reviews.llvm.org/D105149

show more ...

[MLIR] [Python ODS] Use @builtins.property for cases where 'property' is already defined

In cases where an operation has an argument or result named 'property', the
ODS-generated python fails on imp

[MLIR] [Python ODS] Use @builtins.property for cases where 'property' is already defined

In cases where an operation has an argument or result named 'property', the
ODS-generated python fails on import because the `@property` resolves to the
`property` operation argument instead of the builtin `@property` decorator. We
should always use the fully qualified decorator name.

Reviewed By: mikeurbach

Differential Revision: https://reviews.llvm.org/D106106

show more ...

[mlir][python] Reorganize MLIR python into namespace packages.

* Only leaf packages are non-namespace packages. This allows most of the top levels to be split into different directories or deploymen

[mlir][python] Reorganize MLIR python into namespace packages.

* Only leaf packages are non-namespace packages. This allows most of the top levels to be split into different directories or deployment packages. In the previous state, the presence of __init__.py files at each level meant that the entire tree could only ever exist in one physical directory on the path.
* This changes the API usage slightly: `import mlir` will no longer do a deep import of `mlir.ir`, etc. This may necessitate some client code changes.
* Dialect gen code was restructured so that the user is responsible for providing the `my_dialect.py` file, which then must import its peer `_my_dialect_ops_gen`. This gives complete control of the dialect namespace to the user instead of to tablegen code, allowing further dialect-specific python APIs.
* Correspondingly, the previous extension modules `_my_dialect.py` are now `_my_dialect_ops_ext.py`.
* Now that the `linalg` namespace is open, moved the `linalg_opdsl` tool into it.
* This may require some corresponding downstream adjustments to npcomp, circt, et al:
* Probably some shallow imports need to be converted to deep imports (i.e. not `import mlir` brings in the world).
* Each tablegen generated dialect now needs an explicit `foo.py` which does a `from ._foo_ops_gen import *`. This is similar to the way that generated code operates in the C++ world.
* If providing dialect op extensions, those need to be moved from `_foo.py` -> `_foo_ops_ext.py`.

Differential Revision: https://reviews.llvm.org/D98096

show more ...

[mlir][Python] Roll up of python API fixes.

* As discussed, fixes the ordering or (operands, results) -> (results, operands) in various `create` like methods.
* Fixes a syntax error in an ODS access

[mlir][Python] Roll up of python API fixes.

* As discussed, fixes the ordering or (operands, results) -> (results, operands) in various `create` like methods.
* Fixes a syntax error in an ODS accessor method.
* Removes the linalg example in favor of a test case that exercises the same.
* Fixes FuncOp visibility to properly use None instead of the empty string and defaults it to None.
* Implements what was documented for requiring that trailing __init__ args `loc` and `ip` are keyword only.
* Adds a check to `InsertionPoint.insert` so that if attempting to insert past the terminator, an exception is raised telling you what to do instead. Previously, this would crash downstream (i.e. when trying to print the resultant module).
* Renames `_ods_build_default` -> `build_generic` and documents it.
* Removes `result` from the list of prohibited words and for single-result ops, defaults to naming the result `result`, thereby matching expectations and what is already implemented on the base class.
* This was intended to be a relatively small set of changes to be inlined with the broader support for ODS generating the most specific builder, but it spidered out once actually testing various combinations, so rolling up separately.

Differential Revision: https://reviews.llvm.org/D95320

show more ...

Add Python bindings for the builtin dialect

This includes some minor customization for FuncOp and ModuleOp.

Differential Revision: https://reviews.llvm.org/D95022

[mlir][python] Add facility for extending generated python ODS.

* This isn't exclusive with other mechanisms for more ODS centric op definitions, but based on discussions, we feel that we will alway

[mlir][python] Add facility for extending generated python ODS.

* This isn't exclusive with other mechanisms for more ODS centric op definitions, but based on discussions, we feel that we will always benefit from a python escape hatch, and that is the most natural way to write things that don't fit the mold.
* I suspect this facility needs further tweaking, and once it settles, I'll document it and add more tests.
* Added extensions for linalg, since it is unusable without them and continued to evolve my e2e example.

Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D94752

show more ...

[mlir][python] Factor out standalone OpView._ods_build_default class method.

* This allows us to hoist trait level information for regions and sized-variadic to class level attributes (_ODS_REGIONS,

[mlir][python] Factor out standalone OpView._ods_build_default class method.

* This allows us to hoist trait level information for regions and sized-variadic to class level attributes (_ODS_REGIONS, _ODS_OPERAND_SEGMENTS, _ODS_RESULT_SEGMENTS).
* Eliminates some splicey python generated code in favor of a native helper for it.
* Makes it possible to implement custom, variadic and region based builders with one line of python, without needing to manually code access to the segment attributes.
* Needs follow-on work for region based callbacks and support for SingleBlockImplicitTerminator.
* A follow-up will actually add ODS support for generating custom Python builders that delegate to this new method.
* Also includes the start of an e2e sample for constructing linalg ops where this limitation was discovered (working progressively through this example and cleaning up as I go).

Differential Revision: https://reviews.llvm.org/D94738

show more ...

Fix a few GCC compiler warnings (NFC)

[mlir][python] Aggressively avoid name collisions in generated python ODS code.

* When porting npcomp to use these bindings, I ran into enough patterns of collisions that I decided to be somewhat dr

[mlir][python] Aggressively avoid name collisions in generated python ODS code.

* When porting npcomp to use these bindings, I ran into enough patterns of collisions that I decided to be somewhat draconian about not polluting the namespace.
* With these changes all of the npcomp dialects generate and pass what tests we have.

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D93920

show more ...

[mlir] Make attributes mutable in Python bindings

Attributes represent additional data about an operation and are intended to be
modifiable during the lifetime of the operation. In the dialect-speci

[mlir] Make attributes mutable in Python bindings

Attributes represent additional data about an operation and are intended to be
modifiable during the lifetime of the operation. In the dialect-specific Python
bindings, attributes are exposed as properties on the operation class. Allow
for assigning values to these properties. Also support creating new and
deleting existing attributes through the generic "attributes" property of an
operation. Any validity checking must be performed by the op verifier after the
mutation, similarly to C++. Operations are not invalidated in the process: no
dangling pointers can be created as all attributes are owned by the context and
will remain live even if they are not used in any operation.

Introduce a Python Test dialect by analogy with the Test dialect and to avoid
polluting the latter with Python-specific constructs. Use this dialect to
implement a test for the attribute access and mutation API.

Reviewed By: stellaraccident, mehdi_amini

Differential Revision: https://reviews.llvm.org/D91652

show more ...

[mlir] Add a _get_default_loc_context utility to Python bindings

This utility function is helpful for dialect-specific builders that need
to access the context through location, and the location its

[mlir] Add a _get_default_loc_context utility to Python bindings

This utility function is helpful for dialect-specific builders that need
to access the context through location, and the location itself may be
either provided as an argument or expected to be recovered from the
implicit location stack.

Reviewed By: stellaraccident

Differential Revision: https://reviews.llvm.org/D91623

show more ...

[mlir] Add basic support for attributes in ODS-generated Python bindings

In ODS, attributes of an operation can be provided as a part of the "arguments"
field, together with operands. Such attribute

[mlir] Add basic support for attributes in ODS-generated Python bindings

In ODS, attributes of an operation can be provided as a part of the "arguments"
field, together with operands. Such attributes are accepted by the op builder
and have accessors generated.

Implement similar functionality for ODS-generated op-specific Python bindings:
the `__init__` method now accepts arguments together with operands, in the same
order as in the ODS `arguments` field; the instance properties are introduced
to OpView classes to access the attributes.

This initial implementation accepts and returns instances of the corresponding
attribute class, and not the underlying values since the mapping scheme of the
value types between C++, C and Python is not yet clear. Default-valued
attributes are not supported as that would require Python to be able to parse
C++ literals.

Since attributes in ODS are tightely related to the actual C++ type system,
provide a separate Tablegen file with the mapping between ODS storage type for
attributes (typically, the underlying C++ attribute class), and the
corresponding class name. So far, this might look unnecessary since all names
match exactly, but this is not necessarily the cases for non-standard,
out-of-tree attributes, which may also be placed in non-default namespaces or
Python modules. This also allows out-of-tree users to generate Python bindings
without having to modify the bindings generator itself. Storage type was
preferred over the Tablegen "def" of the attribute class because ODS
essentially encodes attribute _constraints_ rather than classes, e.g. there may
be many Tablegen "def"s in the ODS that correspond to the same attribute type
with additional constraints

The presence of the explicit mapping requires the change in the .td file
structure: instead of just calling the bindings generator directly on the main
ODS file of the dialect, it becomes necessary to create a new file that
includes the main ODS file of the dialect and provides the mapping for
attribute types. Arguably, this approach offers better separability of the
Python bindings in the build system as the main dialect no longer needs to know
that it is being processed by the bindings generator.

Reviewed By: stellaraccident

Differential Revision: https://reviews.llvm.org/D91542

show more ...