bpf: Enable IRQ after irq_work_raise() completes in unit_free{_rcu}()
authorHou Tao <houtao1@huawei.com>
Fri, 1 Sep 2023 11:19:53 +0000 (19:19 +0800)
committerAlexei Starovoitov <ast@kernel.org>
Fri, 8 Sep 2023 15:42:19 +0000 (08:42 -0700)
commit62cf51cb0ebe997a9903208e546755b63eb7ff9d
tree484138c4f3cab4816daacaad98bb76b74b26554f
parent566f6de3cea3482d75d836a2398792a8be32ec26
bpf: Enable IRQ after irq_work_raise() completes in unit_free{_rcu}()

Both unit_free() and unit_free_rcu() invoke irq_work_raise() to free
freed objects back to slab and the invocation may also be preempted by
unit_alloc() and unit_alloc() may return NULL unexpectedly as shown in
the following case:

task A         task B

unit_free()
  // high_watermark = 48
  // free_cnt = 49 after free
  irq_work_raise()
    // mark irq work as IRQ_WORK_PENDING
    irq_work_claim()

               // task B preempts task A
               unit_alloc()
                 // free_cnt = 48 after alloc

               // does unit_alloc() 32-times
       ......
       // free_cnt = 16

       unit_alloc()
         // free_cnt = 15 after alloc
                 // irq work is already PENDING,
                 // so just return
                 irq_work_raise()

       // does unit_alloc() 15-times
               ......
       // free_cnt = 0

               unit_alloc()
                 // free_cnt = 0 before alloc
                 return NULL

Fix it by enabling IRQ after irq_work_raise() completes.

Signed-off-by: Hou Tao <houtao1@huawei.com>
Link: https://lore.kernel.org/r/20230901111954.1804721-3-houtao@huaweicloud.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
kernel/bpf/memalloc.c