mptcp: use copy_from_iter helpers on transmit

The perf traces show an high cost for the MPTCP transmit path memcpy.

It turn out that the helper currently in use carries quite a bit
of unneeded overhead, e.g. to map/unmap the memory pages.

Moving to the 'copy_from_iter' variant removes such overhead and
additionally gains the no-cache support.

Reviewed-by: Mat Martineau <martineau@kernel.org>
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
Signed-off-by: Mat Martineau <martineau@kernel.org>
Link: https://lore.kernel.org/r/20231023-send-net-next-20231023-2-v1-6-9dc60939d371@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
index 7036e30c449fe75d21c509bd7f68aafa0f3768c8..5489f024dd7e6f4891ff8f5b13b91a624f129be0 100644 (file)
--- a/net/mptcp/protocol.c
+++ b/net/mptcp/protocol.c
@@ -1760,6 +1760,18 @@ static int mptcp_sendmsg_fastopen(struct sock *sk, struct msghdr *msg,
        return ret;
 }
 
+static int do_copy_data_nocache(struct sock *sk, int copy,
+                               struct iov_iter *from, char *to)
+{
+       if (sk->sk_route_caps & NETIF_F_NOCACHE_COPY) {
+               if (!copy_from_iter_full_nocache(to, copy, from))
+                       return -EFAULT;
+       } else if (!copy_from_iter_full(to, copy, from)) {
+               return -EFAULT;
+       }
+       return 0;
+}
+
 static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 {
        struct mptcp_sock *msk = mptcp_sk(sk);
@@ -1833,11 +1845,10 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
                if (!sk_wmem_schedule(sk, total_ts))
                        goto wait_for_memory;
 
-               if (copy_page_from_iter(dfrag->page, offset, psize,
-                                       &msg->msg_iter) != psize) {
-                       ret = -EFAULT;
+               ret = do_copy_data_nocache(sk, psize, &msg->msg_iter,
+                                          page_address(dfrag->page) + offset);
+               if (ret)
                        goto do_error;
-               }
 
                /* data successfully copied into the write queue */
                sk_forward_alloc_add(sk, -total_ts);