block: extend bio caching to task context

bio_put_percpu_cache() puts all non-iopoll bios into the irq-safe list,
which entails disabling irqs. The overhead of that is not that bad when
interrupts are already off but getting worse otherwise. We can optimise
it when we're in the task context by using ->free_list directly just as
the IOPOLL path does.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/4774e1a0f905f96c63174b0f3e4f79f0d9b63246.1707314970.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/block/bio.c b/block/bio.c
index b9642a41f286e5bb52d841255aa9286c1449e83d..8da941974f883c0c2ea0a81b9086556b83ccc205 100644 (file)
--- a/block/bio.c
+++ b/block/bio.c
@@ -770,8 +770,9 @@ static inline void bio_put_percpu_cache(struct bio *bio)
 
        bio_uninit(bio);
 
-       if ((bio->bi_opf & REQ_POLLED) && !WARN_ON_ONCE(in_interrupt())) {
+       if (in_task()) {
                bio->bi_next = cache->free_list;
+               /* Not necessary but helps not to iopoll already freed bios */
                bio->bi_bdev = NULL;
                cache->free_list = bio;
                cache->nr++;