sched/cpuacct: Optimize away RCU read lock

Since cpuacct_charge() is called from the scheduler update_curr(),
we must already have rq lock held, then the RCU read lock can
be optimized away.

And do the same thing in it's wrapper cgroup_account_cputime(),
but we can't use lockdep_assert_rq_held() there, which defined
in kernel/sched/sched.h.

Suggested-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Chengming Zhou <zhouchengming@bytedance.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/r/20220220051426.5274-2-zhouchengming@bytedance.com

diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
index 75c151413fda8c9ae1436983932ad7b09c3fda23..9a109c6ac0e0f782bc8d71fe0b23ecf687181d96 100644 (file)
--- a/include/linux/cgroup.h
+++ b/include/linux/cgroup.h
@@ -791,11 +791,9 @@ static inline void cgroup_account_cputime(struct task_struct *task,
 
        cpuacct_charge(task, delta_exec);
 
-       rcu_read_lock();
        cgrp = task_dfl_cgroup(task);
        if (cgroup_parent(cgrp))
                __cgroup_account_cputime(cgrp, delta_exec);
-       rcu_read_unlock();
 }
 
 static inline void cgroup_account_cputime_field(struct task_struct *task,

diff --git a/kernel/sched/cpuacct.c b/kernel/sched/cpuacct.c
index 307800586ac81e1bd973445e88412be4cef31905..f79f88456d720ef98434dc5741a4c7962de1ce46 100644 (file)
--- a/kernel/sched/cpuacct.c
+++ b/kernel/sched/cpuacct.c
@@ -337,12 +337,10 @@ void cpuacct_charge(struct task_struct *tsk, u64 cputime)
        unsigned int cpu = task_cpu(tsk);
        struct cpuacct *ca;
 
-       rcu_read_lock();
+       lockdep_assert_rq_held(cpu_rq(cpu));
 
        for (ca = task_ca(tsk); ca; ca = parent_ca(ca))
                *per_cpu_ptr(ca->cpuusage, cpu) += cputime;
-
-       rcu_read_unlock();
 }
 
 /*