io_uring: optimise non-timeout waiting

Unlike the jiffy scheduling version, schedule_hrtimeout() jumps a few
functions before getting into schedule() even if there is no actual
timeout needed. Some tests showed that it takes up to 1% of CPU cycles.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/89f880574eceee6f4899783377ead234df7b3d04.1672916894.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 2ec011f0ba7d11c1c89ecb1860932895f0651322..6229a49c0c33a1051ad5459c7440d969c167cc84 100644 (file)
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2479,7 +2479,9 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx,
                return -EINTR;
        if (unlikely(io_should_wake(iowq)))
                return 0;
-       if (!schedule_hrtimeout(timeout, HRTIMER_MODE_ABS))
+       if (*timeout == KTIME_MAX)
+               schedule();
+       else if (!schedule_hrtimeout(timeout, HRTIMER_MODE_ABS))
                return -ETIME;
        return 0;
 }