iommu/amd: Use try_cmpxchg64 in alloc_pte and free_clear_pte

Use try_cmpxchg64 instead of cmpxchg64 (*ptr, old, new) != old in
alloc_pte and free_clear_pte.  cmpxchg returns success in ZF flag, so this
change saves a compare after cmpxchg (and related move instruction
in front of cmpxchg). Also, remove racy explicit assignment to pteval
when cmpxchg fails, this is what try_cmpxchg does implicitly from
*pte in an atomic way.

Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
Cc: Joerg Roedel <joro@8bytes.org>
Cc: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20220525145416.10816-1-ubizjak@gmail.com
Signed-off-by: Joerg Roedel <jroedel@suse.de>

diff --git a/drivers/iommu/amd/io_pgtable.c b/drivers/iommu/amd/io_pgtable.c
index 6608d171757445e3d90030bd04b5c55ec3d4fe1c..7d4b61e5db47b4a457ebb6662fe40de29c61c43f 100644 (file)
--- a/drivers/iommu/amd/io_pgtable.c
+++ b/drivers/iommu/amd/io_pgtable.c
@@ -258,7 +258,7 @@ static u64 *alloc_pte(struct protection_domain *domain,
                        __npte = PM_LEVEL_PDE(level, iommu_virt_to_phys(page));
 
                        /* pte could have been changed somewhere. */
-                       if (cmpxchg64(pte, __pte, __npte) != __pte)
+                       if (!try_cmpxchg64(pte, &__pte, __npte))
                                free_page((unsigned long)page);
                        else if (IOMMU_PTE_PRESENT(__pte))
                                *updated = true;
@@ -341,10 +341,8 @@ static void free_clear_pte(u64 *pte, u64 pteval, struct list_head *freelist)
        u64 *pt;
        int mode;
 
-       while (cmpxchg64(pte, pteval, 0) != pteval) {
+       while (!try_cmpxchg64(pte, &pteval, 0))
                pr_warn("AMD-Vi: IOMMU pte changed since we read it\n");
-               pteval = *pte;
-       }
 
        if (!IOMMU_PTE_PRESENT(pteval))
                return;