1Task References 2=============== 3 4Finding the source of task reference count leaks. 5 6Background 7---------- 8 9Tasks in XNU are reference counted. When a task is created it starts with two 10references - one for the caller and one for the task itself. Over the lifetime 11of the task this reference count is modified, for example when a thread is 12created it increments the reference count and when it exits that count drops. 13When a reference count reaches zero, the task is freed. 14 15To grab a reference: 16```c 17task_reference() 18``` 19 20To release a reference: 21```c 22task_deallocate() 23``` 24 25One of the big problems seen with task references is that difficult to debug 26_leaks_ commonly occur. This happens when a reference is taken but never 27released. The task is kept around indefinitely and eventually the system runs 28out of a finite resource (for example ASIDs). At this point there is very little 29information to determine what code was responsible for the leak. 30 31 32Task Reference Groups 33-------------------- 34 35Reference groups are a feature which keep track of statistics (and when 36configured backtrace information) for a set of references. Reference groups are 37hierarchical. To help with debugging the following task reference group 38hierarchy is used: 39 40``` 41task 42 -> task_internal 43 -> task_local_internal 44 -> task_kernel 45 -> task_local_internal 46 -> task_mig 47 -> task_local_internal 48 -> task_external 49 -> task_local_external 50 -> task_com.apple.security.sandbox 51 -> task_com.apple.security.sandbox 52 -> task_com.apple.driver.AppleHV 53 -> task_com.apple.driver.AppleHV 54 ... 55``` 56 57The `task` group contains a count of all task references in the system. The 58first-level groups are static and sub-divide task references based on the 59sub-system they come from. `task_external` is used for kext references and each 60kext will be dynamically assigned a reference group as needed (if there's 61one available). At the bottom level, there's a per-task (local) ref group under 62each global group. 63The exact hierarchy of task references (specifically what per-task reference 64groups are created) changes depending on the 'task_refgrp' boot arg. 65 66Task reference groups can be explored in `lldb` as follows: 67 68``` 69(lldb) showglobaltaskrefgrps 70os_refgrp name count retain release log 710xffffff801ace9250 task_kernel 68 367663 367595 0x0 720xffffff801ace9288 task_internal 974 4953 3979 0x0 730xffffff801ace92c0 task_mig 0 3670 3670 0x0 740xffffff801ace9218 task_external 35 108 73 0x0 750xffffff9369dc7b20 task_com.apple.iokit.IOAcceleratorFamily2 29 77 48 0x0 760xffffff936a3f0a20 task_com.apple.iokit.CoreAnalyticsFamily 1 1 0 0x0 770xffffff936a22cb20 task_com.apple.iokit.EndpointSecurity 0 1 1 0x0 780xffffff936a283f60 task_com.apple.iokit.IOSurface 5 5 0 0x0 790xffffff936a3f08a0 task_com.apple.security.sandbox 0 24 24 0x0 80 81``` 82 83Display a task's reference groups: 84 85``` 86(lldb) showtaskrefgrps kernel_task 87os_refgrp name count retain release log 880xffffff936a4b9200 task_local_kernel 1 6 5 0x0 890xffffff936a4b9238 task_local_internal 132 619 487 0x0 90``` 91 92The reference group hierarchy for a specific group can be displayed as follows: 93 94``` 95(lldb) showosrefgrphierarchy 0xffffff936a3f08a0 960xffffff801ace9988 all 1121 377740 376619 0x0 970xffffff801ace91e0 task 1077 376394 375317 0x0 980xffffff801ace9218 task_external 35 108 73 0x0 990xffffff936a3f08a0 task_com.apple.security.sandbox 0 24 24 0x0 100``` 101 102Reference groups are normally disabled, but task reference group statistics 103*are* enabled by default (for `RELEASE` builds, reference groups are not available 104at all). Backtrace logging for all groups is disabled, including task reference 105groups. To enable backtrace logging and reference group statistics, the `rlog` 106boot-arg must be used. Backtrace logging for task reference groups is only 107enabled when `rlog` has been set to a suitable value. 108 109For example 110 111To enable statistics for all reference groups and backtrace logging for the 112*task_external* reference group in particular: 113 114``` 115nvram boot-args="rlog=task_external ..." 116``` 117 118``` 119(lldb) showglobaltaskrefgrps 120os_refgrp name count retain release log 1210xffffff801e0e9250 task_kernel 1259 132739 131480 0x0 1220xffffff801e0e9218 task_external 35 100 65 0xffffffa05b3fc000 1230xffffff936d117be0 task_com.apple.iokit.IOAcceleratorFamily2 29 77 48 0x0 1240xffffff936db9fa20 task_com.apple.iokit.CoreAnalyticsFamily 1 1 0 0x0 1250xffffff936d9dbb20 task_com.apple.iokit.EndpointSecurity 0 1 1 0x0 1260xffffff936da324e0 task_com.apple.iokit.IOSurface 5 5 0 0x0 1270xffffff936db9f8a0 task_com.apple.security.sandbox 0 16 16 0x0 128 129 130(lldb) showbtlogrecords 0xffffffa05b3fc000 131-------- OP 1 Stack Index 0 with active refs 1 of 165 -------- 1320xffffff801da7c1cb <kernel.development`ref_log_op at refcnt.c:107> 1330xffffff801d27c35d <kernel.development`task_reference_grp at task_ref.c:274> 1340xffffff801ecc014e <EndpointSecurity`VMMap::taskSelf()> 1350xffffff801eccc845 <EndpointSecurity`EndpointSecurityClient::create(ScopedPointer<MachSendWrapper> const&, proc*, ScopedPointer<EndpointSecurityExternalClient> const&, es_client_config_t const&)> 136... 137``` 138