mm/gup: cache *pudp in follow_pud_mask()

Introduce "pud_t pud" in the function, so the code won't dereference *pudp
multiple time.  Not only because that looks less straightforward, but also
because if the dereference really happened, it's not clear whether there
can be race to see different *pudp values if it's being modified at the
same time.

Link: https://lkml.kernel.org/r/20240327152332.950956-10-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Acked-by: James Houghton <jthoughton@google.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Tested-by: Ryan Roberts <ryan.roberts@arm.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Andrew Jones <andrew.jones@linux.dev>
Cc: Aneesh Kumar K.V (IBM) <aneesh.kumar@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: David Hildenbrand <david@redhat.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Kirill A. Shutemov <kirill@shutemov.name>
Cc: Lorenzo Stoakes <lstoakes@gmail.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: "Mike Rapoport (IBM)" <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Rik van Riel <riel@surriel.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Yang Shi <shy828301@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/mm/gup.c b/mm/gup.c
index 1a1e459fd1382e87e49e0d4e7d76fd808753ec99..39224b5fe62f172fa3d3e709bfb1ed7f281ac678 100644 (file)
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -753,26 +753,27 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma,
                                    unsigned int flags,
                                    struct follow_page_context *ctx)
 {
-       pud_t *pud;
+       pud_t *pudp, pud;
        spinlock_t *ptl;
        struct page *page;
        struct mm_struct *mm = vma->vm_mm;
 
-       pud = pud_offset(p4dp, address);
-       if (pud_none(*pud))
+       pudp = pud_offset(p4dp, address);
+       pud = READ_ONCE(*pudp);
+       if (pud_none(pud))
                return no_page_table(vma, flags, address);
-       if (pud_devmap(*pud)) {
-               ptl = pud_lock(mm, pud);
-               page = follow_devmap_pud(vma, address, pud, flags, &ctx->pgmap);
+       if (pud_devmap(pud)) {
+               ptl = pud_lock(mm, pudp);
+               page = follow_devmap_pud(vma, address, pudp, flags, &ctx->pgmap);
                spin_unlock(ptl);
                if (page)
                        return page;
                return no_page_table(vma, flags, address);
        }
-       if (unlikely(pud_bad(*pud)))
+       if (unlikely(pud_bad(pud)))
                return no_page_table(vma, flags, address);
 
-       return follow_pmd_mask(vma, address, pud, flags, ctx);
+       return follow_pmd_mask(vma, address, pudp, flags, ctx);
 }
 
 static struct page *follow_p4d_mask(struct vm_area_struct *vma,