bpf: Fix JIT of is_mov_percpu_addr instruction.

The codegen for is_mov_percpu_addr instruction works for rax/r8 registers
only. Fix it to generate proper x86 byte code for other registers.

Fixes: 7bdbf7446305 ("bpf: add special internal-only MOV instruction to resolve per-CPU addrs")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20240417214406.15788-1-alexei.starovoitov@gmail.com

diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 2b5a475c4dd0dfc57f83b9ce916e7af9bc64a6e1..673fdbd765d780d469877f839a7f9fdc3abf7efb 100644 (file)
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -1439,7 +1439,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 #ifdef CONFIG_SMP
                                /* add <dst>, gs:[<off>] */
                                EMIT2(0x65, add_1mod(0x48, dst_reg));
-                               EMIT3(0x03, add_1reg(0x04, dst_reg), 0x25);
+                               EMIT3(0x03, add_2reg(0x04, 0, dst_reg), 0x25);
                                EMIT((u32)(unsigned long)&this_cpu_off, 4);
 #endif
                                break;