bio: don't rob starving biosets of bios

Biosets keep a mempool, so as long as requests complete we can always
can allocate and have forward progress. Percpu bio caches break that
assumptions as we may complete into the cache of one CPU and after try
and fail to allocate with another CPU. We also can't grab from another
CPU's cache without tricky sync.

If we're allocating with a bio while the mempool is undersaturated,
remove REQ_ALLOC_CACHE flag, so on put it will go straight to mempool.
It might try to free into mempool more requests than required, but
assuming than there is no memory starvation in the system it'll
stabilise and never hit that path.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/aa150caf9c263fa92269e86d7826cc8fa65f38de.1667384020.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/block/bio.c b/block/bio.c
index aa1de6a367f9f308872990183c427a58c7fb2201..91f4a6926a1bade6c8bc5b31c0392b6cebb03949 100644 (file)
--- a/block/bio.c
+++ b/block/bio.c
@@ -526,6 +526,8 @@ struct bio *bio_alloc_bioset(struct block_device *bdev, unsigned short nr_vecs,
        }
        if (unlikely(!p))
                return NULL;
+       if (!mempool_is_saturated(&bs->bio_pool))
+               opf &= ~REQ_ALLOC_CACHE;
 
        bio = p + bs->front_pad;
        if (nr_vecs > BIO_INLINE_VECS) {