Merge remote-tracking branch 'remotes/rth-gitlab/tags/pull-tcg-20210619-2' into staging

TCI cleanup and re-encoding
Fixes for #367 and #390.
Move TCGCond to tcg/tcg-cond.h.
Fix for win32 qemu_try_memalign.

# gpg: Signature made Sun 20 Jun 2021 05:23:53 BST
# gpg:                using RSA key 7A481E78868B4DB6A85A05C064DF38E8AF7E215F
# gpg:                issuer "richard.henderson@linaro.org"
# gpg: Good signature from "Richard Henderson <richard.henderson@linaro.org>" [full]
# Primary key fingerprint: 7A48 1E78 868B 4DB6 A85A  05C0 64DF 38E8 AF7E 215F

* remotes/rth-gitlab/tags/pull-tcg-20210619-2: (33 commits)
  util/oslib-win32: Fix fatal assertion in qemu_try_memalign
  tcg: expose TCGCond manipulation routines
  tcg: Restart when exhausting the stack frame
  tcg: Allocate sufficient storage in temp_allocate_frame
  tcg/sparc: Fix temp_allocate_frame vs sparc stack bias
  accel/tcg: Probe the proper permissions for atomic ops
  tests/tcg: Increase timeout for TCI
  tcg/tci: Use {set,clear}_helper_retaddr
  tcg/tci: Remove the qemu_ld/st_type macros
  Revert "tcg/tci: Use exec/cpu_ldst.h interfaces"
  tcg/tci: Split out tci_qemu_ld, tci_qemu_st
  tcg/tci: Implement add2, sub2
  tcg/tci: Implement mulu2, muls2
  tcg/tci: Implement clz, ctz, ctpop
  tcg/tci: Implement extract, sextract
  tcg/tci: Implement andc, orc, eqv, nand, nor
  tcg/tci: Implement movcond
  tcg/tci: Implement goto_ptr
  tcg/tci: Change encoding to uint32_t units
  tcg/tci: Remove tci_write_reg
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

Merge remote-tracking branch 'remotes/jsnow-gitlab/tags/python-pull-request' into staging

Python Pull request

Moves QMP-related tools not used for build or automatic testing from
scripts/ to python/qemu/qmp/ where they will be protected from bitrot by
the check-python-* CI jobs.

stub forwarders are left in the old locations for now.

# gpg: Signature made Sat 19 Jun 2021 00:02:40 BST
# gpg:                using RSA key F9B7ABDBBCACDF95BE76CBD07DEF8106AAFC390E
# gpg: Good signature from "John Snow (John Huston) <jsnow@redhat.com>" [full]
# Primary key fingerprint: FAEB 9711 A12C F475 812F  18F2 88A9 064D 1835 61EB
#      Subkey fingerprint: F9B7 ABDB BCAC DF95 BE76  CBD0 7DEF 8106 AAFC 390E

* remotes/jsnow-gitlab/tags/python-pull-request: (72 commits)
  scripts/qmp-shell: add redirection shim
  python: add qmp-shell entry point
  scripts/qmp-shell: move to python/qemu/qmp/qmp_shell.py
  scripts/qmp-shell: add docstrings
  scripts/qmp-shell: make QMPShellError inherit QMPError
  scripts/qmp-shell: remove double-underscores
  scripts/qmp-shell: convert usage comment to docstring
  scripts/qmp-shell: Remove too-broad-exception
  scripts/qmp-shell: Fix empty-transaction invocation
  scripts/qmp-shell: remove TODO
  scripts/qmp-shell: use logging to show warnings
  scripts/qmp-shell: Use context manager instead of atexit
  python/qmp: return generic type from context manager
  scripts/qmp-shell: unprivatize 'pretty' property
  scripts/qmp-shell: Accept SocketAddrT instead of string
  scripts/qmp-shell: add mypy types
  python/qmp: add QMPObject type alias
  scripts/qmp-shell: initialize completer early
  scripts/qmp-shell: refactor QMPCompleter
  scripts/qmp-shell: Fix "FuzzyJSON" parser
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

Merge remote-tracking branch 'remotes/ehabkost-gl/tags/x86-next-pull-request' into staging

x86 queue, 2021-06-18

Features:
* Add ratelimit for bus locks acquired in guest (Chenyi Qiang)

Documentation:
* SEV documentation updates (Tom Lendacky)
* Add a table showing x86-64 ABI compatibility levels (Daniel P. Berrangé)

Automated changes:
* Update Linux headers to 5.13-rc4 (Eduardo Habkost)

# gpg: Signature made Fri 18 Jun 2021 20:51:26 BST
# gpg:                using RSA key 5A322FD5ABC4D3DBACCFD1AA2807936F984DC5A6
# gpg:                issuer "ehabkost@redhat.com"
# gpg: Good signature from "Eduardo Habkost <ehabkost@redhat.com>" [full]
# Primary key fingerprint: 5A32 2FD5 ABC4 D3DB ACCF  D1AA 2807 936F 984D C5A6

* remotes/ehabkost-gl/tags/x86-next-pull-request:
  scripts: helper to generate x86_64 CPU ABI compat info
  docs: add a table showing x86-64 ABI compatibility levels
  docs/interop/firmware.json: Add SEV-ES support
  docs: Add SEV-ES documentation to amd-memory-encryption.txt
  doc: Fix some mistakes in the SEV documentation
  i386: Add ratelimit for bus locks acquired in guest
  Update Linux headers to 5.13-rc4

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2021-06-15-v2' into staging

nbd patches for 2021-06-15

- bug fixes in coroutine aio context handling
- rework NBD client connection logic to perform more work in coroutine

# gpg: Signature made Fri 18 Jun 2021 18:29:39 BST
# gpg:                using RSA key 71C2CC22B1C4602927D2F3AAA7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg:                 aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg:                 aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2  F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-nbd-2021-06-15-v2: (34 commits)
  block/nbd: safer transition to receiving request
  block/nbd: add nbd_client_connected() helper
  block/nbd: reuse nbd_co_do_establish_connection() in nbd_open()
  nbd/client-connection: add option for non-blocking connection attempt
  block/nbd: split nbd_co_do_establish_connection out of nbd_reconnect_attempt
  block-coroutine-wrapper: allow non bdrv_ prefix
  nbd/client-connection: return only one io channel
  block/nbd: drop BDRVNBDState::sioc
  block/nbd: don't touch s->sioc in nbd_teardown_connection()
  block/nbd: use negotiation of NBDClientConnection
  block/nbd: split nbd_handle_updated_info out of nbd_client_handshake()
  nbd/client-connection: shutdown connection on release
  nbd/client-connection: implement connection retry
  nbd/client-connection: add possibility of negotiation
  nbd/client-connection: use QEMU_LOCK_GUARD
  nbd: move connection code from block/nbd to nbd/client-connection
  block/nbd: introduce nbd_client_connection_release()
  block/nbd: introduce nbd_client_connection_new()
  block/nbd: rename NBDConnectThread to NBDClientConnection
  block/nbd: make nbd_co_establish_connection_cancel() bs-independent
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

util/oslib-win32: Fix fatal assertion in qemu_try_memalign

The function is called with alignment == 0 which caused an assertion.
Use the code from oslib-posix.c to fix that regression.

Fixes: ed6f53f9ca9
Signed-off-by: Stefan Weil <sw@weilnetz.de>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210611105846.347954-1-sw@weilnetz.de>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: expose TCGCond manipulation routines

This commit moves into a separate file routines used to manipulate
TCGCond. These will be employed by the idef-parser.

Signed-off-by: Alessandro Di Federico <ale@rev.ng>
Signed-off-by: Paolo Montesel <babush@rev.ng>
Message-Id: <20210619093713.1845446-2-ale.qemu@rev.ng>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Restart when exhausting the stack frame

Assume that we'll have fewer temps allocated after
restarting with a fewer number of instructions.

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Allocate sufficient storage in temp_allocate_frame

This function should have been updated for vector types
when they were introduced.

Fixes: d2fd745fe8b
Resolves: https://gitlab.com/qemu-project/qemu/-/issues/367
Cc: qemu-stable@nongnu.org
Tested-by: Stefan Weil <sw@weilnetz.de>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/sparc: Fix temp_allocate_frame vs sparc stack bias

We should not be aligning the offset in temp_allocate_frame,
because the odd offset produces an aligned address in the end.
Instead, pass the logical offset into tcg_set_frame and add
the stack bias last.

Cc: qemu-stable@nongnu.org
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

accel/tcg: Probe the proper permissions for atomic ops

We had a single ATOMIC_MMU_LOOKUP macro that probed for
read+write on all atomic ops.  This is incorrect for
plain atomic load and atomic store.

For user-only, we rely on the host page permissions.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/390
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tests/tcg: Increase timeout for TCI

The longest test at the moment seems to be a (slower)
aarch64 host, for which test-mmap takes 64 seconds.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Acked-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Use {set,clear}_helper_retaddr

Wrap guest memory operations for tci like we do for cpu_ld*_data.

We cannot actually use the cpu_ldst.h interface without duplicating
the memory trace operations performed within, which will already
have been expanded into the tcg opcode stream.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Remove the qemu_ld/st_type macros

These macros are only used in one place.  By expanding,
we get to apply some common-subexpression elimination
and create some local variables.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

Revert "tcg/tci: Use exec/cpu_ldst.h interfaces"

This reverts commit dc09f047eddec8f4a1991c4f5f4a428d7aa3f2c0.

For tcg, tracepoints are expanded inline in tcg opcodes.
Using a helper which generates a second tracepoint is incorrect.

For system mode, the extraction and re-packing of MemOp and mmu_idx
lost the alignment information from MemOp.  So we were no longer
raising alignment exceptions for !TARGET_ALIGNED_ONLY guests.
This can be seen in tests/tcg/xtensa/test_load_store.S.

For user mode, we must update to the new signature of g2h() so that
the revert compiles.  We can leave set_helper_retaddr for later.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Split out tci_qemu_ld, tci_qemu_st

We can share this code between 32-bit and 64-bit loads and stores.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Implement add2, sub2

We already had the 32-bit versions for a 32-bit host; expand this
to 64-bit hosts as well.  The 64-bit opcodes are new.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Implement mulu2, muls2

We already had mulu2_i32 for a 32-bit host; expand this to 64-bit
hosts as well.  The muls2_i32 and the 64-bit opcodes are new.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Implement clz, ctz, ctpop

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Implement extract, sextract

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Implement andc, orc, eqv, nand, nor

These were already present in tcg-target.c.inc,
but not in the interpreter.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Implement movcond

When this opcode is not available in the backend, tcg middle-end
will expand this as a series of 5 opcodes.  So implementing this
saves bytecode space.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Implement goto_ptr

This operation is critical to staying within the interpretation
loop longer, which avoids the overhead of setup and teardown for
many TBs.

The check in tcg_prologue_init is disabled because TCI does
want to use NULL to indicate exit, as opposed to branching to
a real epilogue.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Change encoding to uint32_t units

This removes all of the problems with unaligned accesses
to the bytecode stream.

With an 8-bit opcode at the bottom, we have 24 bits remaining,
which are generally split into 6 4-bit slots.  This fits well
with the maximum length opcodes, e.g. INDEX_op_add2_i32, which
have 6 register operands.

We have, in previous patches, rearranged things such that there
are no operations with a label which have more than one other
operand.  Which leaves us with a 20-bit field in which to encode
a label, giving us a maximum TB size of 512k -- easily large.

Change the INDEX_op_tci_movi_{i32,i64} opcodes to tci_mov[il].
The former puts the immediate in the upper 20 bits of the insn,
like we do for the label displacement.  The later uses a label
to reference an entry in the constant pool.  Thus, in the worst
case we still have a single memory reference for any constant,
but now the constants are out-of-line of the bytecode and can
be shared between different moves saving space.

Change INDEX_op_call to use a label to reference a pair of
pointers in the constant pool.  This removes the only slightly
dodgy link with the layout of struct TCGHelperInfo.

The re-encode cannot be done in pieces.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Remove tci_write_reg

Inline it into its one caller, tci_write_reg64.
Drop the asserts that are redundant with tcg_read_r.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Emit setcond before brcond

The encoding planned for tci does not have enough room for
brcond2, with 4 registers and a condition as input as well
as the label.  Resolve the condition into TCG_REG_TMP, and
relax brcond to one register plus a label, considering the
condition to always be reg != 0.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Reserve r13 for a temporary

We're about to adjust the offset range on host memory ops,
and the format of branches.  Both will require a temporary.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Use ffi for calls

This requires adjusting where arguments are stored.
Place them on the stack at left-aligned positions.
Adjust the stack frame to be at entirely positive offsets.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Move call-return regs to end of tcg_target_reg_alloc_order

As the only call-clobbered regs for TCI, these should
receive the least priority.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg/tci: Improve tcg_target_call_clobber_regs

The current setting is much too pessimistic.  Indicating only
the one or two registers that are actually assigned after a
call should avoid unnecessary movement between the register
array and the stack array.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Build ffi data structures for helpers

Add libffi as a build requirement for TCI.
Add libffi to the dockerfiles to satisfy that requirement.

Construct an ffi_cif structure for each unique typemask.
Record the result in a separate hash table for later lookup;
this allows helper_table to stay const.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Add tcg_call_func

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Store the TCGHelperInfo in the TCGOp for call

This will give us both flags and typemask for use later.

We also fix a dumping bug, wherein calls generated for plugins
fail tcg_find_helper and print (null) instead of either a name
or the raw function pointer.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

accel/tcg: Add tcg call flags to plugins helpers

As noted by qemu-plugins.h, plugins can neither read nor write
guest registers.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

plugins: Drop tcg_flags from struct qemu_plugin_dyn_cb

As noted by qemu-plugins.h, enum qemu_plugin_cb_flags is
currently unused -- plugins can neither read nor write
guest registers.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

accel/tcg/plugin-gen: Drop inline markers

Let the compiler decide on inlining.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Add tcg_call_flags

We're going to change how to look up the call flags from a TCGop,
so extract it as a helper.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

tcg: Combine dh_is_64bit and dh_is_signed to dh_typecode

We will shortly be interested in distinguishing pointers
from integers in the helper's declaration, as well as a
true void return.  We currently have two parallel 1 bit
fields; merge them and expand to a 3 bit field.

Our current maximum is 7 helper arguments, plus the return
makes 8 * 3 = 24 bits used within the uint32_t typemask.

Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

scripts/qmp-shell: add redirection shim

qmp-shell has a new home, add a redirect for a little while as the dust
settles.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-43-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

python: add qmp-shell entry point

now 'qmp-shell' should be available from the command line when
installing the python package.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-42-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: move to python/qemu/qmp/qmp_shell.py

The script will be unavailable for a commit or two, which will help
preserve development history attached to the new file. A forwarder will
be added shortly afterwards.

With qmp_shell in the python qemu.qmp package, now it is fully type
checked, linted, etc. via the Python CI. It will be quite a bit harder
to accidentally break it again in the future.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-41-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: add docstrings

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-40-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: make QMPShellError inherit QMPError

In preparation for moving qmp-shell into the qemu.qmp package, make
QMPShellError inherit from QMPError so that all custom errors in this
package all derive from QMPError.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-39-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: remove double-underscores

They're not needed; single underscore is enough to express intent that
these methods are "internal". double underscore is used as a weak name
mangling, but that isn't beneficial for us here.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-38-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: convert usage comment to docstring

The nice usage comment should be a docstring instead of a comment, so
that it's visible from other python tooling.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-37-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: Remove too-broad-exception

We are only anticipating QMPShellErrors here, for syntax we weren't able
to understand. Other errors, if any, should be allowed to percolate
upwards.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-36-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: Fix empty-transaction invocation

calling "transaction( )" is pointless, but valid. Rework the parser to
allow this kind of invocation. This helps clean up exception handling
later by removing accidental breakages of the parser that aren't
explicitly forbidden.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-35-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: remove TODO

We still want to revamp qmp-shell again, but there's much more to the
idea than the comment now intuits. Remove it.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-34-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: use logging to show warnings

A perfect candidate is non-fatal shell history messages.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-33-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: Use context manager instead of atexit

We can invoke the shell history writing when we leave the QMPShell scope
instead of relying on atexit. Doing so may be preferable to avoid global
state being registered from within a class instead of from the
application logic directly.

Use QMP's context manager to hook this history saving at close time,
which gets invoked when we leave the context block.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-32-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

python/qmp: return generic type from context manager

__enter__ can be invoked from a subclass, so it needs a more flexible
type.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-31-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: unprivatize 'pretty' property

Similar to verbose, there's no reason this needs to be hidden.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-30-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: Accept SocketAddrT instead of string

Don't "extend" QEMUMonitorProtocol by changing the argument types. Move
the string parsing just outside of the class instead.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-29-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: add mypy types

As per my usual, this patch is annotations only. Any changes with side
effects are done elsewhere.

Note: pylint does not understand the subscripts for Collection in Python 3.6,
so use the stronger Sequence type as a workaround.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-28-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

python/qmp: add QMPObject type alias

This is meant to represent any generic object seen in a QMPMessage, not
just the root object itself.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-27-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: initialize completer early

Add an empty completer as a more type-safe placeholder instead of
'None'.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-26-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: refactor QMPCompleter

list is a generic type, but we expect to use strings directly. We could
subclass list[str], but pylint does not presently understand that
invocation.

Change this class to envelop a list instead of *being* a list, for
simpler mypy typing.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-25-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: Fix "FuzzyJSON" parser

I'm not sure when this regressed (Or maybe if it was ever working right
to begin with?), but the Python AST requires you to change "Names" to
"Constants" in order to truly convert `false` to `False`.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-24-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: move the REPL functionality into QMPShell

Instead of doing this in main, move it into the class itself. (This
makes it easier to put into the qemu.qmp package later by removing as
much as we can from the main() function.)

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-23-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: remove prompt argument from read_exec_command

It's only ever used by one caller, we can just absorb that logic.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-22-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: move get_prompt() to prompt property

Small tidying; treat "prompt" like an immutable property instead of
function/method/routine.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-21-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: Make verbose a public attribute

No real reason to hide this behind an underscore; make it part of the
initializer and make it a regular RW attribute.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-20-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: Add pretty attribute to HMP shell

It's less useful, but it makes the initialization methods LSP
consistent, which quiets a mypy complaint.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-19-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: use argparse

Use argparse instead of an open-coded CLI parser, for consistency with
everything else.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-18-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: use isinstance() instead of type()

A bit more idiomatic, and quiets some linter warnings.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-17-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: remove if-raise-else patterns

Shushes pylint. I don't always mind these patterns personally, but I'm
not as sure that I want to remove the warning from pylint's repertoire
entirely. Oh well.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-16-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: fix shell history exception handling

We want to remove exceptions that are too broad here; we only want to
catch IOErrors that get raised as a direct result of the open call.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-15-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: rename one and two-letter variables

A bit of churn and housekeeping for pylint, flake8 et al.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-14-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: make QMPCompleter returns explicit

This function returns None when it doesn't find a match; do that
explicitly.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-13-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: ignore visit_Name name

Not something I control, sorry, pylint.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-12-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: use triple-double-quote docstring style

(2014 me had never written python before.)

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: declare verbose in __init__

Linters get angry when we don't define state at init time.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-10-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: Use python3-style super()

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-9-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: use @classmethod where appropriate

Methods with no self-use should belong to the class.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: remove shadowed variable from _print()

Don't use 'qmp' here, which shadows the qmp module.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-7-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: fix connect method signature

It needs to match the parent's signature -- the negotiate parameter must
be optional.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: fix exception handling

Fixes: 50d189c
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: fix show_banner signature

The signatures need to match.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: Apply flake8 rules

A lot of fiddling around to get us below 80 columns.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qmp-shell: apply isort rules

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qemu-ga-client: Add forwarder shim

Add a little forwarder shim until we are sure that everyone is
comfortable with how to use the tools in their new packaged location.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-12-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

python/qemu-ga-client: add entry point

Remove the shebang, and add a package-defined entry point instead. Now,
it can be accessed using 'qemu-ga-client' from the command line after
installing the package.

The next commit adds a forwarder shim that allows the running of this
script without needing to install the package again.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qemu-ga-client: move to python/qemu/qmp/qemu_ga_client.py

The script itself will be unavailable for a few commits before being
restored, with no way to run it right after this commit. This helps move
git history into the new file. To prevent linter regressions, though, we
do need to immediately touch up the filename to remove dashes (to make
the module importable), and remove the executable bit.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-10-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qemu-ga-client: add mypy type hints

This script is in slightly rough shape, but it still works. A lot of
care went into its initial development. In good faith, I'm updating it
to the latest Python coding standards. If there is in interest in this
script, though, I'll be asking for a contributor to take care of it
further.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-9-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

python/qmp: Correct type of QMPReturnValue

It's only a Dict[str, Any] most of the time. It's not actually
guaranteed to be anything in particular. Fix this type to be
more accurate to the reality we live in.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qemu-ga-client: apply (most) pylint rules

I'm only doing a very quick best-effort to preserve this script, to help
keep it from breaking further. I think there are pending ideas swirling
on the right way to implement better SDKs and better clients, and this
script might be a handy reference for those discussions. It presents
some interesting design problems, like static type safety when using a
dynamic RPC mechanism.

I believe it's worth preserving the effort and care that went into
making this script by updating it to work with our current
infrastructure. However, I am disabling the requirement for docstrings
in this file.

If you would like to help improve this script, please add docstrings
alongside any refactors or rejuvenations you might apply at that time.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-7-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qemu-ga-client: add module docstring

Turn that nice usage comment into a docstring.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qemu-ga-client: replace deprecated optparse with argparse

optparse isn't supported anymore, it's from the python2 days. Replace it
with the mostly similar argparse.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qemu-ga-client: Fix exception handling

Fixes: 50d189c
These error classes aren't available anymore. Fix the bitrot.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qemu-ga-client: apply (most) flake8 rules

- Line length should be < 80
- You shouldn't perform unscoped imports except at the top of the module

Notably, the sys.path hack creates problems with the import rule. This
will be fixed later.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qemu-ga-client: apply isort rules

Hint:
> ln -s scripts/qmp/qemu-ga-client python/qemu/qmp/qemu_ga_client.py
> cd python
> isort qemu

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

python/qmp: add fuse command to 'qom' tools

The 'fuse' command will be unavailable if 'fusepy' is not installed. It
will simply not load and subsequently be unavailable as a subcommand.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-20-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qom-fuse: add redirection shim to python/qemu/qmp/qom-fuse.py

By leaving the script absent for a commit, git-blame travels to the new
file instead of staying on the shim.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-19-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qom-fuse: move to python/qemu/qmp/qom_fuse.py

Move qom-fuse over to the python package now that it passes the
linter. Update the import paradigms so that it continues to pass in the
context of the Python package.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-18-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

python: add optional FUSE dependencies

In preparation for moving qom-fuse over to the python package, we need
some new dependencies to support it.

Add an optional 'fusepy' dependency that users of the package can opt
into with e.g. "pip install qemu[fuse]" which installs the requirements
necessary to obtain the additional functionality.

Add the same fusepy dependency to the 'devel' extras group --
unfortunately I do not see a way for optional groups to imply other
optional groups at present, so the dependency is repeated. The
development group needs to include the full set of dependencies for the
purpose of static analysis of all features offered by this library.

Lastly, add the [fuse] extras group to tox's configuration as a
workaround so that if a stale tox environment is found when running
`make check-tox`, tox will know to rebuild its environments.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-17-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qom-fuse: add static type hints

Because fusepy does not have type hints, add some targeted warning
suppressions.

Namely, we need to allow subclassing something of an unknown type (in
qom_fuse.py), and we need to allow missing imports (recorded against
fuse itself) because mypy will be unable to import fusepy (even when
installed) as it has no types nor type stubs available.

Note: Until now, it was possible to run invocations like 'mypy qemu/'
from ./python and have that work. However, these targeted suppressions
require that you run 'mypy -p qemu/' instead. The correct, canonical
invocation is recorded in ./python/tests/mypy.sh and all of the various
CI invocations always use this correct form.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-16-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qom-fuse: ensure QOMFuse.read always returns bytes

- Use FuseOSError to signal ENOENT instead of returning it
- Wrap qom-get in str(), as we don't always know its type
- The empty return should be b'', not ''.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-15-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qom-fuse: use QOMCommand.qom_list()

the qom_list method provides a type-safe object that's easier to type
check, so switch to using it.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-14-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qom-fuse: Convert to QOMCommand

Move qom-fuse onto the QOMCommand base established in
python/qemu/qmp/qom_common.py. The interface doesn't change
incompatibly, "qom-fuse mountpoint" still works as an invocation, and
QMP_SOCKET is still used as the environment variable.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-13-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qom-fuse: Add docstrings

The methods inherited from fuse don't need docstrings; that's up to
fusepy to handle.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-12-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

scripts/qom-fuse: Apply pylint rules

- Catch specific exceptions from QMP
- Reraise errors with explicit context
- method parameters should match parent's names

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>