x86/mm: Prevent early boot triple-faults with instrumentation

Commit in Fixes added a global TLB flush on the early boot path, after
the kernel switches off of the trampoline page table.

Compiler profiling options enabled with GCOV_PROFILE add additional
measurement code on clang which needs to be initialized prior to
use. The global flush in x86_64_start_kernel() happens before those
initializations can happen, leading to accessing invalid memory.
GCOV_PROFILE builds with gcc are still ok so this is clang-specific.

The second issue this fixes is with KASAN: for a similar reason,
kasan_early_init() needs to have happened before KASAN-instrumented
functions are called.

Therefore, reorder the flush to happen after the KASAN early init
and prevent the compilers from adding profiling instrumentation to
native_write_cr4().

Fixes: f154f290855b ("x86/mm/64: Flush global TLB on boot and AP bringup")
Reported-by: "J. Bruce Fields" <bfields@fieldses.org>
Reported-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Borislav Petkov <bp@suse.de>
Tested-by: Carel Si <beibei.si@intel.com>
Tested-by: "J. Bruce Fields" <bfields@fieldses.org>
Link: https://lore.kernel.org/r/20211209144141.GC25654@xsang-OptiPlex-9020

diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 0083464de5e31e4a8b16a87e3eac02f28e136344..79b3d67addcc3c2503605757ee058e60b6327298 100644 (file)
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -384,7 +384,7 @@ set_register:
 }
 EXPORT_SYMBOL(native_write_cr0);
 
-void native_write_cr4(unsigned long val)
+void __no_profile native_write_cr4(unsigned long val)
 {
        unsigned long bits_changed = 0;
 

diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 75acb6027a87a222c9694b352f09f9c4fc93da43..f5e80a8377ad61c33c2d05f52ec319c2b3c52a97 100644 (file)
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -483,10 +483,12 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
        /* Kill off the identity-map trampoline */
        reset_early_page_tables();
 
-       __native_tlb_flush_global(native_read_cr4());
-
        clear_bss();
 
+       /*
+        * This needs to happen *before* kasan_early_init() because latter maps stuff
+        * into that page.
+        */
        clear_page(init_top_pgt);
 
        /*
@@ -498,6 +500,16 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
 
        kasan_early_init();
 
+       /*
+        * Flush global TLB entries which could be left over from the trampoline page
+        * table.
+        *
+        * This needs to happen *after* kasan_early_init() as KASAN-enabled .configs
+        * instrument native_write_cr4() so KASAN must be initialized for that
+        * instrumentation to work.
+        */
+       __native_tlb_flush_global(this_cpu_read(cpu_tlbstate.cr4));
+
        idt_setup_early_handler();
 
        copy_bootdata(__va(real_mode_data));