Changes in core_sched.c

f7d2728c - sched/debug: Change SCHED_WARN_ON() to WARN_ON_ONCE()

Mon, 17 Mar 2025 10:42:52 +0000

sched/debug: Change SCHED_WARN_ON() to WARN_ON_ONCE()The scheduler has this special SCHED_WARN() facility thatdepends on CONFIG_SCHED_DEBUG.Since CONFIG_SCHED_DEBUG is getting removed, convertSCHED_WARN() to WARN_ON_ONCE().Note that the warning output isn't 100% equivalent: #define SCHED_WARN_ON(x) WARN_ONCE(x, #x)Because SCHED_WARN_ON() would output the 'x' conditionas well, while WARN_ONCE() will only show a backtrace.Hopefully these are rare enough to not really matter.If it does, we should probably introduce a new WARN_ON()variant that outputs the condition in stringified form,or improve WARN_ON() itself.Signed-off-by: Ingo Molnar Tested-by: Shrikanth Hegde Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Steven Rostedt Cc: Ben Segall Cc: Mel Gorman Cc: Valentin Schneider Cc: Linus Torvalds Link: https://lore.kernel.org/r/20250317104257.3496611-2-mingo@kernel.org List of files: /linux-6.15/kernel/sched/core_sched.c

402de7fc - sched: Fix spelling in comments

Mon, 27 May 2024 14:54:52 +0000

sched: Fix spelling in commentsDo a spell-checking pass.Signed-off-by: Ingo Molnar Cc: Peter Zijlstra Cc: linux-kernel@vger.kernel.orgSigned-off-by: Ingo Molnar List of files: /linux-6.15/kernel/sched/core_sched.c

0b9d46fc - sched: Rename task_running() to task_on_cpu()

Tue, 06 Sep 2022 10:33:04 +0000

sched: Rename task_running() to task_on_cpu()There is some ambiguity about task_running() in that it is unrelatedto TASK_RUNNING but instead tests ->on_cpu. As such, rename the thingtask_on_cpu().Suggested-by: Ingo Molnar Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/Yxhkhn55uHZx+NGl@hirez.programming.kicks-ass.net List of files: /linux-6.15/kernel/sched/core_sched.c

8648f92a - sched/core: Remove superfluous semicolon

Tue, 19 Jul 2022 11:10:44 +0000

sched/core: Remove superfluous semicolonSigned-off-by: Xin Gao Signed-off-by: Ingo Molnar Link: https://lore.kernel.org/r/20220719111044.7095-1-gaoxin@cdjrlc.com List of files: /linux-6.15/kernel/sched/core_sched.c

91caa5ae - sched/core: Fix the bug that task won't enqueue into core tree when update cookie

Tue, 28 Jun 2022 07:57:23 +0000

sched/core: Fix the bug that task won't enqueue into core tree when update cookieIn function sched_core_update_cookie(), a task will enqueue into thecore tree only when it enqueued before, that is, if an uncookied taskis cookied, it will not enqueue into the core tree until it enqueueagain, which will result in unnecessary force idle.Here follows the scenario: CPU x and CPU y are a pair of SMT siblings. 1. Start task a running on CPU x without sleeping, and task b and task c running on CPU y without sleeping. 2. We create a cookie and share it to task a and task b, and then we create another cookie and share it to task c. 3. Simpling core_forceidle_sum of task a and b from /proc/PID/schedAnd we will find out that core_forceidle_sum of task a takes 30%time of the sampling period, which shouldn't happen as task a and bhave the same cookie.Then we migrate task a to CPU x', migrate task b and c to CPU y', whereCPU x' and CPU y' are a pair of SMT siblings, and sampling again, wewill found out that core_forceidle_sum of task a and b are almost zero.To solve this problem, we enqueue the task into the core tree if it'son rq.Fixes: 6e33cad0af49("sched: Trivial core scheduling cookie management")Signed-off-by: Cruz Zhao Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/1656403045-100840-2-git-send-email-CruzZhao@linux.alibaba.com List of files: /linux-6.15/kernel/sched/core_sched.c

1fcf54de - sched/core: add forced idle accounting for cgroups

Wed, 29 Jun 2022 21:14:26 +0000

sched/core: add forced idle accounting for cgroups4feee7d1260 previously added per-task forced idle accounting. This patchextends this to also include cgroups.rstat is used for cgroup accounting, except for the root, which useskcpustat in order to bypass the need for doing an rstat flush whenreading root stats.Only cgroup v2 is supported. Similar to the task accounting, the cgroupaccounting requires that schedstats is enabled.Signed-off-by: Josh Don Signed-off-by: Peter Zijlstra (Intel) Acked-by: Tejun Heo Link: https://lkml.kernel.org/r/20220629211426.3329954-1-joshdon@google.com List of files: /linux-6.15/kernel/sched/core_sched.c

801c1419 - sched/headers: Introduce kernel/sched/build_utility.c and build multiple .c files there

Tue, 22 Feb 2022 12:23:24 +0000

sched/headers: Introduce kernel/sched/build_utility.c and build multiple .c files thereCollect all utility functionality source code files into a single kernel/sched/build_utility.c file,via #include-ing the .c files: kernel/sched/clock.c kernel/sched/completion.c kernel/sched/loadavg.c kernel/sched/swait.c kernel/sched/wait_bit.c kernel/sched/wait.cCONFIG_CPU_FREQ: kernel/sched/cpufreq.cCONFIG_CPU_FREQ_GOV_SCHEDUTIL: kernel/sched/cpufreq_schedutil.cCONFIG_CGROUP_CPUACCT: kernel/sched/cpuacct.cCONFIG_SCHED_DEBUG: kernel/sched/debug.cCONFIG_SCHEDSTATS: kernel/sched/stats.cCONFIG_SMP: kernel/sched/cpupri.c kernel/sched/stop_task.c kernel/sched/topology.cCONFIG_SCHED_CORE: kernel/sched/core_sched.cCONFIG_PSI: kernel/sched/psi.cCONFIG_MEMBARRIER: kernel/sched/membarrier.cCONFIG_CPU_ISOLATION: kernel/sched/isolation.cCONFIG_SCHED_AUTOGROUP: kernel/sched/autogroup.cThe goal is to amortize the 60+ KLOC header bloat from over a dozen build units intoa single build unit.The build time of build_utility.c also roughly matches the build time of core.c andfair.c - allowing better load-balancing of scheduler-only rebuilds.Signed-off-by: Ingo Molnar Reviewed-by: Peter Zijlstra List of files: /linux-6.15/kernel/sched/core_sched.c

b171501f - sched/core: Accounting forceidle time for all tasks except idle task

Tue, 11 Jan 2022 09:55:59 +0000

sched/core: Accounting forceidle time for all tasks except idle taskThere are two types of forced idle time: forced idle time from cookie'dtask and forced idle time form uncookie'd task. The forced idle time fromuncookie'd task is actually caused by the cookie'd task in runqueueindirectly, and it's more accurate to measure the capacity loss with thesum of both.Assuming cpu x and cpu y are a pair of SMT siblings, consider thefollowing scenarios: 1.There's a cookie'd task running on cpu x, and there're 4 uncookie'd tasks running on cpu y. For cpu x, there will be 80% forced idle time (from uncookie'd task); for cpu y, there will be 20% forced idle time (from cookie'd task). 2.There's a uncookie'd task running on cpu x, and there're 4 cookie'd tasks running on cpu y. For cpu x, there will be 80% forced idle time (from cookie'd task); for cpu y, there will be 20% forced idle time (from uncookie'd task).The scenario1 can recurrent by stress-ng(scenario2 can recurrent similary): (cookie'd)taskset -c x stress-ng -c 1 -l 100 (uncookie'd)taskset -c y stress-ng -c 4 -l 100In the above two scenarios, the total capacity loss is 1 cpu, but inscenario1, the cookie'd forced idle time tells us 20% cpu capacity loss, inscenario2, the cookie'd forced idle time tells us 80% cpu capacity loss,which are not accurate. It'll be more accurate to measure with cookie'dforced idle time and uncookie'd forced idle time.Signed-off-by: Cruz Zhao Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Josh Don Link: https://lore.kernel.org/r/1641894961-9241-2-git-send-email-CruzZhao@linux.alibaba.com List of files: /linux-6.15/kernel/sched/core_sched.c

4feee7d1 - sched/core: Forced idle accounting

Mon, 18 Oct 2021 20:34:28 +0000

sched/core: Forced idle accountingAdds accounting for "forced idle" time, which is time where a cookie'dtask forces its SMT sibling to idle, despite the presence of runnabletasks.Forced idle time is one means to measure the cost of enabling corescheduling (ie. the capacity lost due to the need to force idle).Forced idle time is attributed to the thread responsible for causingthe forced idle.A few details: - Forced idle time is displayed via /proc/PID/sched. It also requires that schedstats is enabled. - Forced idle is only accounted when a sibling hyperthread is held idle despite the presence of runnable tasks. No time is charged if a sibling is idle but has no runnable tasks. - Tasks with 0 cookie are never charged forced idle. - For SMT > 2, we scale the amount of forced idle charged based on the number of forced idle siblings. Additionally, we split the time up and evenly charge it to all running tasks, as each is equally responsible for the forced idle.Signed-off-by: Josh Don Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20211018203428.2025792-1-joshdon@google.com List of files: /linux-6.15/kernel/sched/core_sched.c

d07b2eee - sched: Make cookie functions static

Wed, 22 Sep 2021 08:57:35 +0000

sched: Make cookie functions staticMake cookie functions static as these are no longer invoked directlyby other code.No functional change intended.Signed-off-by: Shaokun Zhang Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210922085735.52812-1-zhangshaokun@hisilicon.com List of files: /linux-6.15/kernel/sched/core_sched.c

61bc346c - uapi/linux/prctl: provide macro definitions for the PR_SCHED_CORE type argument

Wed, 25 Aug 2021 17:06:13 +0000

uapi/linux/prctl: provide macro definitions for the PR_SCHED_CORE type argumentCommit 7ac592aa35a684ff ("sched: prctl() core-scheduling interface")made use of enum pid_type in prctl's arg4; this type and the associatedenumeration definitions are not exposed to userspace. Christianhas suggested to provide additional macro definitions that conveythe meaning of the type argument more in alignment with its actualusage, and this patch does exactly that.Link: https://lore.kernel.org/r/20210825170613.GA3884@asgard.redhat.comSuggested-by: Christian Brauner Acked-by: Christian Brauner Signed-off-by: Eugene Syromiatnikov Complements: 7ac592aa35a684ff ("sched: prctl() core-scheduling interface")Signed-off-by: Christian Brauner List of files: /linux-6.15/kernel/sched/core_sched.c

7ac592aa - sched: prctl() core-scheduling interface

Wed, 24 Mar 2021 21:40:15 +0000

sched: prctl() core-scheduling interfaceThis patch provides support for setting and copying core scheduling'task cookies' between threads (PID), processes (TGID), and processgroups (PGID).The value of core scheduling isn't that tasks don't share a core,'nosmt' can do that. The value lies in exploiting all the sharingopportunities that exist to recover possible lost performance and thatrequires a degree of flexibility in the API.From a security perspective (and there are others), the thread,process and process group distinction is an existent hierarchalcategorization of tasks that reflects many of the security concernsabout 'data sharing'. For example, protecting against cache-snoopingby a thread that can just read the memory directly isn't all thatuseful.With this in mind, subcommands to CREATE/SHARE (TO/FROM) provide amechanism to create and share cookies. CREATE/SHARE_TO specify atarget pid with enum pidtype used to specify the scope of the targetedtasks. For example, PIDTYPE_TGID will share the cookie with theprocess and all of it's threads as typically desired in a securityscenario.API: prctl(PR_SCHED_CORE, PR_SCHED_CORE_GET, tgtpid, pidtype, &cookie) prctl(PR_SCHED_CORE, PR_SCHED_CORE_CREATE, tgtpid, pidtype, NULL) prctl(PR_SCHED_CORE, PR_SCHED_CORE_SHARE_TO, tgtpid, pidtype, NULL) prctl(PR_SCHED_CORE, PR_SCHED_CORE_SHARE_FROM, srcpid, pidtype, NULL)where 'tgtpid/srcpid == 0' implies the current process and pidtype iskernel enum pid_type {PIDTYPE_PID, PIDTYPE_TGID, PIDTYPE_PGID, ...}.For return values, EINVAL, ENOMEM are what they say. ESRCH means thetgtpid/srcpid was not found. EPERM indicates lack of PTRACE permissionaccess to tgtpid/srcpid. ENODEV indicates your machines lacks SMT.[peterz: complete rewrite]Signed-off-by: Chris Hyser Signed-off-by: Peter Zijlstra (Intel) Tested-by: Don Hiatt Tested-by: Hongyu Ning Tested-by: Vincent Guittot Link: https://lkml.kernel.org/r/20210422123309.039845339@infradead.org List of files: /linux-6.15/kernel/sched/core_sched.c

85dd3f61 - sched: Inherit task cookie on fork()

Mon, 29 Mar 2021 13:18:35 +0000

sched: Inherit task cookie on fork()Note that sched_core_fork() is called from under tasklist_lock, andnot from sched_fork() earlier. This avoids a few races later.Signed-off-by: Peter Zijlstra (Intel) Tested-by: Don Hiatt Tested-by: Hongyu Ning Tested-by: Vincent Guittot Link: https://lkml.kernel.org/r/20210422123308.980003687@infradead.org List of files: /linux-6.15/kernel/sched/core_sched.c

6e33cad0 - sched: Trivial core scheduling cookie management

Fri, 26 Mar 2021 17:55:06 +0000

sched: Trivial core scheduling cookie managementIn order to not have to use pid_struct, create a new, smaller,structure to manage task cookies for core scheduling.Signed-off-by: Peter Zijlstra (Intel) Tested-by: Don Hiatt Tested-by: Hongyu Ning Tested-by: Vincent Guittot Link: https://lkml.kernel.org/r/20210422123308.919768100@infradead.org List of files: /linux-6.15/kernel/sched/core_sched.c