[mlir] (NFC) run clang-format on all files

[mlir][NFC] Split the non-templated bits out of IROperand into a base class

This removes the need to define the derived Operand class before the derived
Value class. The major benefit of this refact

[mlir][NFC] Split the non-templated bits out of IROperand into a base class

This removes the need to define the derived Operand class before the derived
Value class. The major benefit of this refactoring is that we no longer need
the OpaqueValue class, as OpOperand can now be defined after Value. As part of
this refactoring the BlockOperand and OpOperand classes are moved out of
UseDefLists.h and to more suitable locations in BlockSupport and Value. After
this change, UseDefLists.h is almost entirely composed of generic use def utilities.

Differential Revision: https://reviews.llvm.org/D103353

show more ...

[IR] Add a Location to BlockArgument

This adds the ability to specify a location when creating BlockArguments.
Notably Value::getLoc() will return this correctly, which makes diagnostics
more precis

[IR] Add a Location to BlockArgument

This adds the ability to specify a location when creating BlockArguments.
Notably Value::getLoc() will return this correctly, which makes diagnostics
more precise (e.g. the example in test-legalize-type-conversion.mlir).

This is currently optional to avoid breaking any existing code - if
absent, the BlockArgument defaults to using the location of its enclosing
operation (preserving existing behavior).

The bulk of this change is plumbing location tracking through the parser
and printer to make sure it can round trip (in -mlir-print-debuginfo
mode). This is complete for generic operations, but requires manual
adoption for custom ops.

I added support for function-like ops to round trip their argument
locations - they print correctly, but when parsing the locations are
dropped on the floor. I intend to fix this, but it will require more
invasive plumbing through "function_like_impl" stuff so I think it
best to split it out to its own patch.

This is a reapply of the patch here: https://reviews.llvm.org/D102567
with an additional change: we now never defer block argument locations,
guaranteeing that we can round trip correctly.

This isn't required in all cases, but allows us to hill climb here and
works around unrelated bugs like https://bugs.llvm.org/show_bug.cgi?id=50451

Differential Revision: https://reviews.llvm.org/D102991

show more ...

Revert "[IR] Add a Location to BlockArgument." and follow-on commit
"[mlir] Speed up Lexer::getEncodedSourceLocation"

This reverts commit 3043be9d2db4d0cdf079adb5e1bdff032405e941 and commit
861d69a5

Revert "[IR] Add a Location to BlockArgument." and follow-on commit
"[mlir] Speed up Lexer::getEncodedSourceLocation"

This reverts commit 3043be9d2db4d0cdf079adb5e1bdff032405e941 and commit
861d69a5259653f60d59795597493a7939b794fe.

This change resulted in printing textual MLIR that can't be parsed; see
review thread https://reviews.llvm.org/D102567 for details.

show more ...

[IR] Add a Location to BlockArgument.

This adds the ability to specify a location when creating BlockArguments.
Notably Value::getLoc() will return this correctly, which makes diagnostics
more preci

[IR] Add a Location to BlockArgument.

This adds the ability to specify a location when creating BlockArguments.
Notably Value::getLoc() will return this correctly, which makes diagnostics
more precise (e.g. the example in test-legalize-type-conversion.mlir).

This is currently optional to avoid breaking any existing code - if
absent, the BlockArgument defaults to using the location of its enclosing
operation (preserving existing behavior).

The bulk of this change is plumbing location tracking through the parser
and printer to make sure it can round trip (in -mlir-print-debuginfo
mode). This is complete for generic operations, but requires manual
adoption for custom ops.

I added support for function-like ops to round trip their argument
locations - they print correctly, but when parsing the locations are
dropped on the floor. I intend to fix this, but it will require more
invasive plumbing through "function_like_impl" stuff so I think it
best to split it out to its own patch.

Differential Revision: https://reviews.llvm.org/D102567

show more ...

[mlir][NFC] Add helper for common pattern of replaceAllUsesExcept

This covers the extremely common case of replacing all uses of a Value
with a new op that is itself a user of the original Value.

T

[mlir][NFC] Add helper for common pattern of replaceAllUsesExcept

This covers the extremely common case of replacing all uses of a Value
with a new op that is itself a user of the original Value.

This should also be a little bit more efficient than the
`SmallPtrSet<Operation *, 1>{op}` idiom that was being used before.

Differential Revision: https://reviews.llvm.org/D102373

show more ...

[mlir][IR] Refactor the internal implementation of Value

The current implementation of Value involves a pointer int pair with several different kinds of owners, i.e. BlockArgumentImpl*, Operation *,

[mlir][IR] Refactor the internal implementation of Value

The current implementation of Value involves a pointer int pair with several different kinds of owners, i.e. BlockArgumentImpl*, Operation *, TrailingOpResult*. This design arose from the desire to save memory overhead for operations that have a very small number of results (generally 0-2). There are, unfortunately, many problematic aspects of the current implementation that make Values difficult to work with or just inefficient.

Operation result types are stored as a separate array on the Operation. This is very inefficient for many reasons: we use TupleType for multiple results, which can lead to huge amounts of memory usage if multi-result operations change types frequently(they do). It also means that simple methods like Value::getType/Value::setType now require complex logic to get to the desired type.

Value only has one pointer bit free, severely limiting the ability to use it in things like PointerUnion/PointerIntPair. Given that we store the kind of a Value along with the "owner" pointer, we only leave one bit free for users of Value. This creates situations where we end up nesting PointerUnions to be able to use Value in one.

As noted above, most of the methods in Value need to branch on at least 3 different cases which is both inefficient, possibly error prone, and verbose. The current storage of results also creates problems for utilities like ValueRange/TypeRange, which want to efficiently store base pointers to ranges (of which Operation* isn't really useful as one).

This revision greatly simplifies the implementation of Value by the introduction of a new ValueImpl class. This class contains all of the state shared between all of the various derived value classes; i.e. the use list, the type, and the kind. This shared implementation class provides several large benefits:

* Most of the methods on value are now branchless, and often one-liners.

* The "kind" of the value is now stored in ValueImpl instead of Value
This frees up all of Value's pointer bits, allowing for users to take full advantage of PointerUnion/PointerIntPair/etc. It also allows for storing more operation results as "inline", 6 now instead of 2, freeing up 1 word per new inline result.

* Operation result types are now stored in the result, instead of a side array
This drops the size of zero-result operations by 1 word. It also removes the memory crushing use of TupleType for operations results (which could lead up to hundreds of megabytes of "dead" TupleTypes in the context). This also allowed restructured ValueRange, making it simpler and one word smaller.

This revision does come with two conceptual downsides:
* Operation::getResultTypes no longer returns an ArrayRef<Type>
This conceptually makes some usages slower, as the iterator increment is slightly more complex.
* OpResult::getOwner is slightly more expensive, as it now requires a little bit of arithmetic

From profiling, neither of the conceptual downsides have resulted in any perceivable hit to performance. Given the advantages of the new design, most compiles are slightly faster.

Differential Revision: https://reviews.llvm.org/D97804

show more ...

Revert "Remove use of tuple for multiresult type storage"

This reverts commit 08f0764ff551c5aa2486c40871453e1ff40fb679.

Remove use of tuple for multiresult type storage

Move the results in line with the op instead. This results in each
operation having its own types recorded vs single tuple type, but comes
at benefit

Remove use of tuple for multiresult type storage

Move the results in line with the op instead. This results in each
operation having its own types recorded vs single tuple type, but comes
at benefit that every mutation doesn't incurs uniquing. Ran into cases
where updating result type of operation led to very large memory usage.

Differential Revision: https://reviews.llvm.org/D97652

show more ...

[AsmPrinter] Make OpAsmPrinter::printFunctionalType be resilient to null values.

A previous patch made Value::getType() be resilient to null values which was
considered to be too sweeping. This is

[AsmPrinter] Make OpAsmPrinter::printFunctionalType be resilient to null values.

A previous patch made Value::getType() be resilient to null values which was
considered to be too sweeping. This is a more targeted change which requires
deabstracting some templates.

A middle ground would be to make ValueTypeIterator be tolerant to null values.

Differential Revision: https://reviews.llvm.org/D93908

show more ...

[IR] Make Value::getType() work better with invalid IR.

The asmprinter would crash when dumping IR objects that had their
operands dropped. With this change, we now get this output, which
makes op-

[IR] Make Value::getType() work better with invalid IR.

The asmprinter would crash when dumping IR objects that had their
operands dropped. With this change, we now get this output, which
makes op->dump() style debugging more useful.

%5 = "firrtl.eq"(<<NULL>>, <<NULL>>) : (<<NULL TYPE>>, <<NULL TYPE>>) -> !firrtl.uint<1>

Previously the asmprinter would crash getting the types of the null operands.

Differential Revision: https://reviews.llvm.org/D93869

show more ...

[mlir][IR][NFC] Move context/location parameters of builtin Type::get methods to the start of the parameter list

This better matches the rest of the infrastructure, is much simpler, and makes it eas

[mlir][IR][NFC] Move context/location parameters of builtin Type::get methods to the start of the parameter list

This better matches the rest of the infrastructure, is much simpler, and makes it easier to move these types to being declaratively specified.

Differential Revision: https://reviews.llvm.org/D93432

show more ...

[mlir][Types][NFC] Move all of the builtin Type classes to BuiltinTypes.h

This is part of a larger refactoring the better congregates the builtin structures under the BuiltinDialect. This also remov

[mlir][Types][NFC] Move all of the builtin Type classes to BuiltinTypes.h

This is part of a larger refactoring the better congregates the builtin structures under the BuiltinDialect. This also removes the problematic "standard" naming that clashes with the "standard" dialect, which is not defined within IR/. A temporary forward is placed in StandardTypes.h to allow time for downstream users to replaced references.

Differential Revision: https://reviews.llvm.org/D92435

show more ...

[mlir][DialectConversion] Enable deeper integration of type conversions

This revision adds support for much deeper type conversion integration into the conversion process, and enables auto-generatin

[mlir][DialectConversion] Enable deeper integration of type conversions

This revision adds support for much deeper type conversion integration into the conversion process, and enables auto-generating cast operations when necessary. Type conversions are now largely automatically managed by the conversion infra when using a ConversionPattern with a provided TypeConverter. This removes the need for patterns to do type cast wrapping themselves and moves the burden to the infra. This makes it much easier to perform partial lowerings when type conversions are involved, as any lingering type conversions will be automatically resolved/legalized by the conversion infra.

To support this new integration, a few changes have been made to the type materialization API on TypeConverter. Materialization has been split into three separate categories:
* Argument Materialization: This type of materialization is used when converting the type of block arguments when calling `convertRegionTypes`. This is useful for contextually inserting additional conversion operations when converting a block argument type, such as when converting the types of a function signature.
* Source Materialization: This type of materialization is used to convert a legal type of the converter into a non-legal type, generally a source type. This may be called when uses of a non-legal type persist after the conversion process has finished.
* Target Materialization: This type of materialization is used to convert a non-legal, or source, type into a legal, or target, type. This type of materialization is used when applying a pattern on an operation, but the types of the operands have not yet been converted.

Differential Revision: https://reviews.llvm.org/D82831

show more ...

[mlir] Add support for merging identical blocks during canonicalization

This revision adds support for merging identical blocks, or those with the same operations that branch to the same successors.

[mlir] Add support for merging identical blocks during canonicalization

This revision adds support for merging identical blocks, or those with the same operations that branch to the same successors. Operands that mismatch between the different blocks are replaced with new block arguments added to the merged block.

Differential Revision: https://reviews.llvm.org/D79134

show more ...

[mlir][DialectConversion] Add support for properly tracking replaceUsesOfBlockArgument

The current implementation of this method performs the replacement directly, and thus doesn't support proper ba

[mlir][DialectConversion] Add support for properly tracking replaceUsesOfBlockArgument

The current implementation of this method performs the replacement directly, and thus doesn't support proper back tracking.

Differential Revision: https://reviews.llvm.org/D78790

show more ...

[mlir] Give each OpResult its own use list

This revision removes the multi use-list to ensure that each result gets its own. This decision was made by doing some extensive benchmarking of programs t

[mlir] Give each OpResult its own use list

This revision removes the multi use-list to ensure that each result gets its own. This decision was made by doing some extensive benchmarking of programs that actually use multiple results. This results in a size increase of 1-word per result >1, but the common case of 1-result remains unaffected. A side benefit is that 0-result operations now shrink by 1-word.

Differential Revision: https://reviews.llvm.org/D78701

show more ...

[MLIR] Move `replaceAllUsesExcept` from LoopUtil.h to Value.h.

Differential Revision: https://reviews.llvm.org/D78426

[mlir] Optimize OpResult use case for single result operations.

Summary:
Operation represents all of the uses of each result with one use list, so manipulating the use list of a specific result requ

[mlir] Optimize OpResult use case for single result operations.

Summary:
Operation represents all of the uses of each result with one use list, so manipulating the use list of a specific result requires filtering the main use list. This revision adds an optimization for the case of single result operations to avoid this filtering.

Differential Revision: https://reviews.llvm.org/D73430

show more ...

Mass update the MLIR license header to mention "Part of the LLVM project"

This is an artifact from merging MLIR into LLVM, the file headers are
now aligned with the rest of the project.

[mlir] NFC: Remove Value::operator* and Value::operator-> now that Value is properly value-typed.

Summary: These were temporary methods used to simplify the transition.

Reviewed By: antiagainst

Di

[mlir] NFC: Remove Value::operator* and Value::operator-> now that Value is properly value-typed.

Summary: These were temporary methods used to simplify the transition.

Reviewed By: antiagainst

Differential Revision: https://reviews.llvm.org/D72548

show more ...

[mlir] Rewrite the internal representation of OpResult to be optimized for memory.

Summary:
This changes the implementation of OpResult to have some of the results be represented inline in Value, vi

[mlir] Rewrite the internal representation of OpResult to be optimized for memory.

Summary:
This changes the implementation of OpResult to have some of the results be represented inline in Value, via a pointer int pair of Operation*+result number, and the rest being trailing objects on the main operation. The full details of the new representation is detailed in the proposal here:
https://groups.google.com/a/tensorflow.org/g/mlir/c/XXzzKhqqF_0/m/v6bKb08WCgAJ

The only difference between here and the above proposal is that we only steal 2-bits for the Value kind instead of 3. This means that we can only fit 2-results inline instead of 6. This allows for other users to steal the final bit for PointerUnion/etc. If necessary, we can always steal this bit back in the future to save more space if 3-6 results are common enough.

Reviewed By: jpienaar

Differential Revision: https://reviews.llvm.org/D72020

show more ...

Fix some corner cases missed by D71955

* replaceAllUsesWith may be supplied with a null value.
* some compilers fail to implicitly convert single result operations to
OpaqueValue, so add an explicit

Fix some corner cases missed by D71955

* replaceAllUsesWith may be supplied with a null value.
* some compilers fail to implicitly convert single result operations to
OpaqueValue, so add an explicit OpOperand::set(Value) method.

show more ...

[mlir] Refactor operation results to use a single use list for all results of the operation.

Summary: A new class is added, IRMultiObjectWithUseList, that allows for representing an IR use list that

[mlir] Refactor operation results to use a single use list for all results of the operation.

Summary: A new class is added, IRMultiObjectWithUseList, that allows for representing an IR use list that holds multiple sub values(used in this case for OpResults). This class provides all of the same functionality as the base IRObjectWithUseList, but for specific sub-values. This saves a word per operation result and is a necessary step in optimizing the layout of operation results. For now the use list is placed on the operation itself, so zero-result operations grow by a word. When the work for optimizing layout is finished, this can be moved back to being a trailing object based on memory/runtime benchmarking.

Reviewed By: jpienaar

Differential Revision: https://reviews.llvm.org/D71955

show more ...

[mlir] Merge the successor operand count into BlockOperand.

Summary: The successor operand counts are directly tied to block operands anyways, and this simplifies the trailing objects of Operation(i

[mlir] Merge the successor operand count into BlockOperand.

Summary: The successor operand counts are directly tied to block operands anyways, and this simplifies the trailing objects of Operation(i.e. one less computation to perform).

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D71949

show more ...