[mlir][tensor][bufferize] Fix deallocation of GenerateOp/FromElementsOp

Both ops allocate a buffer. There were cases in which the buffer was not deallocated.

Differential Revision: https://reviews.

[mlir][tensor][bufferize] Fix deallocation of GenerateOp/FromElementsOp

Both ops allocate a buffer. There were cases in which the buffer was not deallocated.

Differential Revision: https://reviews.llvm.org/D130469

show more ...

[mlir][linalg][NFC] Cleanup: Drop linalg.inplaceable attribute

bufferization.writable is used in most cases instead. All remaining test cases are updated. Some code that is no longer needed is delet

[mlir][linalg][NFC] Cleanup: Drop linalg.inplaceable attribute

bufferization.writable is used in most cases instead. All remaining test cases are updated. Some code that is no longer needed is deleted.

Differential Revision: https://reviews.llvm.org/D129739

show more ...

[mlir][sparse] Switch to One-Shot Bufferize

This change removes the partial bufferization passes from the sparse compilation pipeline and replaces them with One-Shot Bufferize. One-Shot Analysis (an

[mlir][sparse] Switch to One-Shot Bufferize

This change removes the partial bufferization passes from the sparse compilation pipeline and replaces them with One-Shot Bufferize. One-Shot Analysis (and TensorCopyInsertion) is used to resolve all out-of-place bufferizations, dense and sparse. Dense ops are then bufferized with BufferizableOpInterface. Sparse ops are still bufferized in the Sparsification pass.

Details:
* Dense allocations are automatically deallocated, unless they are yielded from a block. (In that case the alloc would leak.) All test cases are modified accordingly. E.g., some funcs now have an "out" tensor argument that is returned from the function. (That way, the allocation happens at the call site.)
* Sparse allocations are *not* automatically deallocated. They must be "released" manually. (No change, this will be addressed in a future change.)
* Sparse tensor copies are not supported yet. (Future change)
* Sparsification no longer has to consider inplacability. If necessary, allocations and/or copies are inserted during TensorCopyInsertion. All tensors are inplaceable by the time Sparsification is running. Instead of marking a tensor as "not inplaceable", it can be marked as "not writable", which will trigger an allocation and/or copy during TensorCopyInsertion.

Differential Revision: https://reviews.llvm.org/D129356

show more ...

[mlir] Use has_value instead of hasValue (NFC)

[mlir][bufferization][NFC] Move more unknown type conversion logic into BufferizationOptions

The `unknownTypeConversion` bufferization option (enum) is now a type converter function option. Some log

[mlir][bufferization][NFC] Move more unknown type conversion logic into BufferizationOptions

The `unknownTypeConversion` bufferization option (enum) is now a type converter function option. Some logic of `getMemRefType` is now handled by that function.

This change makes type conversion more controllable. Previously, there were only two options when generating memref types for non-bufferizable ops: Static identity layout or fully dynamic layout. With this change, users of One-Shot Bufferize can provide a function with custom logic.

Differential Revision: https://reviews.llvm.org/D129273

show more ...

[mlir][bufferize] Infer memory space in all bufferization patterns

This change updates all remaining bufferization patterns (except for scf.while) and the remaining bufferization infrastructure to i

[mlir][bufferize] Infer memory space in all bufferization patterns

This change updates all remaining bufferization patterns (except for scf.while) and the remaining bufferization infrastructure to infer the memory space whenever possible instead of falling back to "0". (If a default memory space is set in the bufferization options, we still fall back to that value if the memory space could not be inferred.)

Differential Revision: https://reviews.llvm.org/D128423

show more ...

[mlir][bufferize][NFC] Change signature of allocateTensorForShapedValue

Add a failure return value and bufferization options argument. This is to keep a subsequent change smaller.

Differential Revi

[mlir][bufferize][NFC] Change signature of allocateTensorForShapedValue

Add a failure return value and bufferization options argument. This is to keep a subsequent change smaller.

Differential Revision: https://reviews.llvm.org/D128278

show more ...

[mlir][bufferization][NFC] Add error handling to getBuffer

This is in preparation of adding memory space support.

Differential Revision: https://reviews.llvm.org/D128277

[mlir][bufferization][NFC] Bufferize with PostOrder traversal

This is useful because the result type of an op can sometimes be inferred from its body (e.g., `scf.if`). This will be utilized in subse

[mlir][bufferization][NFC] Bufferize with PostOrder traversal

This is useful because the result type of an op can sometimes be inferred from its body (e.g., `scf.if`). This will be utilized in subsequent changes.

Also introduces a new `getBufferType` interface method on BufferizableOpInterface. This method is useful for computing a bufferized block argument type with respect to OpOperand types of the parent op.

Differential Revision: https://reviews.llvm.org/D128420

show more ...

[mlir][bufferization][NFC] Change signature of getMemRefType

These functions now accep unsigned attributes for address spaces instead of Attributes.

Differential Revision: https://reviews.llvm.org/

[mlir][bufferization][NFC] Change signature of getMemRefType

These functions now accep unsigned attributes for address spaces instead of Attributes.

Differential Revision: https://reviews.llvm.org/D128275

show more ...

[mlir][bufferization][NFC] Make `escape` a dialect attribute

All bufferizable ops that bufferize to an allocation receive a `bufferization.escape` attribute during TensorCopyInsertion.

Differential

[mlir][bufferization][NFC] Make `escape` a dialect attribute

All bufferizable ops that bufferize to an allocation receive a `bufferization.escape` attribute during TensorCopyInsertion.

Differential Revision: https://reviews.llvm.org/D128137

show more ...

[mlir][bufferization] Set emitAccessorPrefix dialect flag

Generate get/set accessors on all bufferization ops. Also update all internal uses.

Differential Revision: https://reviews.llvm.org/D128057

[mlir][bufferize][NFC] Remove BufferizationState

With the recent refactorings, this class is no longer needed. We can use BufferizationOptions in all places were BufferizationState was used.

Differ

[mlir][bufferize][NFC] Remove BufferizationState

With the recent refactorings, this class is no longer needed. We can use BufferizationOptions in all places were BufferizationState was used.

Differential Revision: https://reviews.llvm.org/D127653

show more ...

[mlir][bufferize] Bufferize after TensorCopyInsertion

This change changes the bufferization so that it utilizes the new TensorCopyInsertion pass. One-Shot Bufferize no longer calls the One-Shot Anal

[mlir][bufferize] Bufferize after TensorCopyInsertion

This change changes the bufferization so that it utilizes the new TensorCopyInsertion pass. One-Shot Bufferize no longer calls the One-Shot Analysis. Instead, it relies on the TensorCopyInsertion pass to make the entire IR fully inplacable. The `bufferize` implementations of all ops are simplified; they no longer have to account for out-of-place bufferization decisions. These were already materialized in the IR in the form of `bufferization.alloc_tensor` ops during the TensorCopyInsertion pass.

Differential Revision: https://reviews.llvm.org/D127652

show more ...

[MLIR][Bufferization] Assume alias if no information is available

- Post (minor) fix after: https://reviews.llvm.org/D127301

Reviewed By: springerm

Differential Revision: https://reviews.llvm.org/

[MLIR][Bufferization] Assume alias if no information is available

- Post (minor) fix after: https://reviews.llvm.org/D127301

Reviewed By: springerm

Differential Revision: https://reviews.llvm.org/D127868

show more ...

[mlir][bufferize] Better implementation of AnalysisState::isTensorYielded

If `create-deallocs=0`, mark all bufferization.alloc_tensor ops as escaping. (Unless they already have an `escape` attribute

[mlir][bufferize] Better implementation of AnalysisState::isTensorYielded

If `create-deallocs=0`, mark all bufferization.alloc_tensor ops as escaping. (Unless they already have an `escape` attribute.) In the absence of analysis information, check SSA use-def chains to see if the value may be yielded.

Differential Revision: https://reviews.llvm.org/D127302

show more ...

[mlir][bufferize][NFC] Merge AlwaysCopyAnalysisState into AnalysisState

`AnalysisState` now has default implementations of all virtual functions.

Differential Revision: https://reviews.llvm.org/D12

[mlir][bufferize][NFC] Merge AlwaysCopyAnalysisState into AnalysisState

`AnalysisState` now has default implementations of all virtual functions.

Differential Revision: https://reviews.llvm.org/D127301

show more ...

[mlir][bufferize] Avoid tensor copies when the data is not read

There are various shortcuts in `BufferizationState::getBuffer` that avoid a buffer copy when we just need an allocation (and no initia

[mlir][bufferize] Avoid tensor copies when the data is not read

There are various shortcuts in `BufferizationState::getBuffer` that avoid a buffer copy when we just need an allocation (and no initialization). This change adds those shortcuts to the TensorCopyInsertion pass, so that `getBuffer` can be simplified in a subsequent change.

Differential Revision: https://reviews.llvm.org/D126821

show more ...

[mlir][bufferize] Improve resolveConflicts for ExtractSliceOp

It is sometimes better to make a copy of the OpResult instead of making a copy of the OpOperand. E.g., when bufferizing tensor.extract_s

[mlir][bufferize] Improve resolveConflicts for ExtractSliceOp

It is sometimes better to make a copy of the OpResult instead of making a copy of the OpOperand. E.g., when bufferizing tensor.extract_slice.

This implementation will eventually make parts of extract_slice's `bufferize` implementation obsolete (and simplify it). It will only need to handle in-place OpOperands.

Differential Revision: https://reviews.llvm.org/D126819

show more ...

[mlir][bufferization][NFC] Put inplacability conflict resolution in op interface

The TensorCopyInsertion pass resolves out-of-place bufferization decisions by inserting explicit `bufferization.alloc

[mlir][bufferization][NFC] Put inplacability conflict resolution in op interface

The TensorCopyInsertion pass resolves out-of-place bufferization decisions by inserting explicit `bufferization.alloc_tensor` ops. This change moves that functionality into a new BufferizableOpInterface method, so that it can be overridden by op implementations. Some op bufferizations must insert additional `alloc_tensor` ops to make sure that certain aliasing invariants are not violated (e.g., scf::ForOp). This will be addressed in a subsequent change.

Differential Revision: https://reviews.llvm.org/D126817

show more ...

[mlir][bufferization] Decouple buffer-deallocation from One-Shot Bufferize

The buffer deallocation pass must now be run explicitly when `allow-return-alloc` is set.

This results in a few extra buff

[mlir][bufferization] Decouple buffer-deallocation from One-Shot Bufferize

The buffer deallocation pass must now be run explicitly when `allow-return-alloc` is set.

This results in a few extra buffer copies in unoptimized test cases. The proper way to avoid such copies is to relax the OpOperand/OpResult aliasing contract on ops such as scf.for. Some of these copies can also be avoided by improving the buffer deallocation pass.

Differential Revision: https://reviews.llvm.org/D126252

show more ...

[mlir][bufferization][NFC] Move OpFilter out of BufferizationOptions

Differential Revision: https://reviews.llvm.org/D126568

[mlir][bufferization][NFC] Remove dead code

There were two copies of AlwaysCopyAnalysisState. (Must have been a merge conflict mistake...)

Differential Revision: https://reviews.llvm.org/D126414

[mlir][bufferize] Remove hoisting functionality from One-Shot Bufferize

The same functionality is already provided by `-buffer-hoisting` and `-buffer-loop-hoisting`.

Differential Revision: https://

[mlir][bufferize] Remove hoisting functionality from One-Shot Bufferize

The same functionality is already provided by `-buffer-hoisting` and `-buffer-loop-hoisting`.

Differential Revision: https://reviews.llvm.org/D126251

show more ...

[mlir][SCF] Fix scf.while bufferization

Before this fix, the bufferization implementation made the incorrect assumption that the values yielded from the "before" region must match with the values yi

[mlir][SCF] Fix scf.while bufferization

Before this fix, the bufferization implementation made the incorrect assumption that the values yielded from the "before" region must match with the values yielded from the "after" region.

Differential Revision: https://reviews.llvm.org/D125835

show more ...