rcu/nocb: Reduce __call_rcu_nocb_wake() leaf rcu_node ->lock contention
authorPaul E. McKenney <paulmck@linux.ibm.com>
Mon, 15 Jul 2019 13:06:40 +0000 (06:06 -0700)
committerPaul E. McKenney <paulmck@linux.ibm.com>
Tue, 13 Aug 2019 21:38:24 +0000 (14:38 -0700)
commit296181d78df9892e08e794f2a9a4d2c38f9acedb
treeebec54a8019bec324e60c76ac6980f80d4823f0f
parent1d5a81c18dc68fc38a52e8dab1992a043a358927
rcu/nocb: Reduce __call_rcu_nocb_wake() leaf rcu_node ->lock contention

Currently, __call_rcu_nocb_wake() advances callbacks each time that it
detects excessive numbers of callbacks, though only if it succeeds in
conditionally acquiring its leaf rcu_node structure's ->lock.  Despite
the conditional acquisition of ->lock, this does increase contention.
This commit therefore avoids advancing callbacks unless there are
callbacks in ->cblist whose grace period has completed and advancing
has not yet been done during this jiffy.

Note that this decision does not take the presence of new callbacks
into account.  That is because on this code path, there will always be
at least one new callback, namely the one we just enqueued.

Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
kernel/rcu/tree_plugin.h