rcu/nocb: Check for deferred nocb wakeups before nohz_full early exit
authorPaul E. McKenney <paulmck@linux.ibm.com>
Tue, 16 Apr 2019 21:48:28 +0000 (14:48 -0700)
committerPaul E. McKenney <paulmck@linux.ibm.com>
Tue, 13 Aug 2019 21:35:49 +0000 (14:35 -0700)
commit85f69b32126dcf798f2c8d69a7957ba990bc2e02
tree469f0182ecef1d15e354c8183f0651d483a3748a
parentc00045be32fe13333ba8c62748ba04747c182838
rcu/nocb: Check for deferred nocb wakeups before nohz_full early exit

In theory, a timer is used to defer wakeups of no-CBs grace-period
kthreads when the wakeup cannot be done safely directly from the
call_rcu().  In practice, the one-jiffy delay is not always consistent
with timely callback invocation under heavy call_rcu() loads.  Therefore,
there are a number of checks for a pending deferred wakeup, including
from the scheduling-clock interrupt.  Unfortunately, this check follows
the rcu_nohz_full_cpu() early exit, which renders it useless on such CPUs.

This commit therefore moves the check for the pending deferred no-CB
wakeup to precede the rcu_nohz_full_cpu() early exit.

Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
kernel/rcu/tree.c