81dd4d4d - dmaengine: idxd: Add IDXD performance monitor support

Sat, 24 Apr 2021 15:04:15 +0000

dmaengine: idxd: Add IDXD performance monitor supportImplement the IDXD performance monitor capability (named 'perfmon' inthe DSA (Data Streaming Accelerator) spec [1]), which supports thecollection of information about key events occurring during DSA andIAX (Intel Analytics Accelerator) device execution, to assist inperformance tuning and debugging.The idxd perfmon support is implemented as part of the IDXD driver andinterfaces with the Linux perf framework. It has several features incommon with the existing uncore pmu support: - it does not support sampling - does not support per-thread countingHowever it also has some unique features not present in the core anduncore support: - all general-purpose counters are identical, thus no event constraints - operation is always system-wideWhile the core perf subsystem assumes that all counters are by defaultper-cpu, the uncore pmus are socket-scoped and use a cpu mask torestrict counting to one cpu from each socket. IDXD counters use asimilar strategy but expand the scope even further; since IDXDcounters are system-wide and can be read from any cpu, the IDXD perfdriver picks a single cpu to do the work (with cpu hotplug notifiersto choose a different cpu if the chosen one is taken off-line).More specifically, the perf userspace tool by default opens a counterfor each cpu for an event. However, if it finds a cpumask fileassociated with the pmu under sysfs, as is the case with the uncorepmus, it will open counters only on the cpus specified by the cpumask.Since perfmon only needs to open a single counter per event for agiven IDXD device, the perfmon driver will create a sysfs cpumask filefor the device and insert the first cpu of the system into it. When auser uses perf to open an event, perf will open a single counter onthe cpu specified by the cpu mask. This amounts to the defaultsystem-wide rather than per-cpu counting mentioned previously forperfmon pmu events. In order to keep the cpu mask up-to-date, thedriver implements cpu hotplug support for multiple devices, as IDXDusually enumerates and registers more than one idxd device.The perfmon driver implements basic perfmon hardware capabilitydiscovery and configuration, and is initialized by the IDXD driver'sprobe function. During initialization, the driver retrieves the totalnumber of supported performance counters, the pmu ID, and the devicetype from idxd device, and registers itself under the Linux perfframework.The perf userspace tool can be used to monitor single or multipleevents depending on the given configuration, as well as event groups,which are also supported by the perfmon driver. The user configuresevents using the perf tool command-line interface by specifying theevent and corresponding event category, along with an optional set offilters that can be used to restrict counting to specific work queues,traffic classes, page and transfer sizes, and engines (See [1] forspecifics).With the configuration specified by the user, the perf tool issues asystem call passing that information to the kernel, which uses it toinitialize the specified event(s). The event(s) are opened andstarted, and following termination of the perf command, they'restopped. At that point, the perfmon driver will read the latest countfor the event(s), calculate the difference between the latest countervalues and previously tracked counter values, and display the finalincremental count as the event count for the cycle. An overflowhandler registered on the IDXD irq path is used to account for counteroverflows, which are signaled by an overflow interrupt.Below are a couple of examples of perf usage for monitoring DSA events.The following monitors all events in the 'engine' category. Becuuseno filters are specified, this captures all engine events for theworkload, which in this case is 19 iterations of the work generated bythe kernel dmatest module.Details describing the events can be found in Appendix D of [1],Performance Monitoring Events, but briefly they are: event 0x1: total input data processed, in 32-byte units event 0x2: total data written, in 32-byte units event 0x4: number of work descriptors that read the source event 0x8: number of work descriptors that write the destination event 0x10: number of work descriptors dispatched from batch descriptors event 0x20: number of work descriptors dispatched from work queues # perf stat -e dsa0/event=0x1,event_category=0x1/, dsa0/event=0x2,event_category=0x1/, dsa0/event=0x4,event_category=0x1/, dsa0/event=0x8,event_category=0x1/, dsa0/event=0x10,event_category=0x1/, dsa0/event=0x20,event_category=0x1/ modprobe dmatest channel=dma0chan0 timeout=2000 iterations=19 run=1 wait=1 Performance counter stats for 'system wide': 5,332 dsa0/event=0x1,event_category=0x1/ 5,327 dsa0/event=0x2,event_category=0x1/ 19 dsa0/event=0x4,event_category=0x1/ 19 dsa0/event=0x8,event_category=0x1/ 0 dsa0/event=0x10,event_category=0x1/ 19 dsa0/event=0x20,event_category=0x1/ 21.977436186 seconds time elapsedThe command below illustrates filter usage with a simple example. Itspecifies that MEM_MOVE operations should be counted for the DSAdevice dsa0 (event 0x8 corresponds to the EV_MEM_MOVE event - Numberof Memory Move Descriptors, which is part of event category 0x3 -Operations. The detailed category and event IDs are available inAppendix D, Performance Monitoring Events, of [1]). In addition tothe event and event category, a number of filters are also specified(the detailed filter values are available in Chapter 6.4 (FilterSupport) of [1]), which will restrict counting to only those eventsthat meet all of the filter criteria. In this case, the filtersspecify that only MEM_MOVE operations that are serviced by work queuewq0 and specifically engine number engine0 and traffic class tc0having sizes between 0 and 4k and page size of between 0 and 1G resultin a counter hit; anything else will be filtered out and not appear inthe final count. Note that filters are optional - any filter notspecified is assumed to be all ones and will pass anything. # perf stat -e dsa0/filter_wq=0x1,filter_tc=0x1,filter_sz=0x7, filter_eng=0x1,event=0x8,event_category=0x3/ modprobe dmatest channel=dma0chan0 timeout=2000 iterations=19 run=1 wait=1 Performance counter stats for 'system wide': 19 dsa0/filter_wq=0x1,filter_tc=0x1,filter_sz=0x7, filter_eng=0x1,event=0x8,event_category=0x3/ 21.865914091 seconds time elapsedThe output above reflects that the unspecified workload resulted inthe counting of 19 MEM_MOVE operation events that met the filtercriteria.[1]: https://software.intel.com/content/www/us/en/develop/download/intel-data-streaming-accelerator-preliminary-architecture-specification.html[ Based on work originally by Jing Lin. ]Reviewed-by: Dave Jiang Reviewed-by: Kan Liang Signed-off-by: Tom Zanussi Link: https://lore.kernel.org/r/0c5080a7d541904c4ad42b848c76a1ce056ddac7.1619276133.git.zanussi@kernel.orgSigned-off-by: Vinod Koul List of files: /linux-6.15/Documentation/ABI/testing/sysfs-bus-event_source-devices-dsa

Changes in sysfs-bus-event_source-devices-dsa

81dd4d4d - dmaengine: idxd: Add IDXD performance monitor support