high-performance RDMA hardware using the following command:
QEMU Monitor Command:
-$ migrate_set_capability x-rdma-pin-all on # disabled by default
+$ migrate_set_capability rdma-pin-all on # disabled by default
Performing this action will cause all 8GB to be pinned, so if that's
not what you want, then please ignore this step altogether.
Next, on the destination machine, add the following to the QEMU command line:
-qemu ..... -incoming x-rdma:host:port
+qemu ..... -incoming rdma:host:port
Finally, perform the actual migration on the source machine:
QEMU Monitor Command:
-$ migrate -d x-rdma:host:port
+$ migrate -d rdma:host:port
PERFORMANCE
===========
active use and the VM itself is completely idle using the same 40 gbps
infiniband link:
-1. x-rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps
-2. x-rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps
+1. rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps
+2. rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps
These numbers would of course scale up to whatever size virtual machine
you have to migrate using RDMA.
TODO:
=====
-1. 'migrate x-rdma:host:port' and '-incoming x-rdma' options will be
- renamed to 'rdma' after the experimental phase of this work has
- completed upstream.
-2. Currently, 'ulimit -l' mlock() limits as well as cgroups swap limits
+1. Currently, 'ulimit -l' mlock() limits as well as cgroups swap limits
are not compatible with infinband memory pinning and will result in
an aborted migration (but with the source VM left unaffected).
-3. Use of the recent /proc/<pid>/pagemap would likely speed up
+2. Use of the recent /proc/<pid>/pagemap would likely speed up
the use of KSM and ballooning while using RDMA.
-4. Also, some form of balloon-device usage tracking would also
+3. Also, some form of balloon-device usage tracking would also
help alleviate some issues.
-5. Move UNREGISTER requests to a separate thread.
-6. Use LRU to provide more fine-grained direction of UNREGISTER
+4. Use LRU to provide more fine-grained direction of UNREGISTER
requests for unpinning memory in an overcommitted environment.
-7. Expose UNREGISTER support to the user by way of workload-specific
+5. Expose UNREGISTER support to the user by way of workload-specific
hints about application behavior.
if (strstart(uri, "tcp:", &p))
tcp_start_incoming_migration(p, errp);
#ifdef CONFIG_RDMA
- else if (strstart(uri, "x-rdma:", &p))
+ else if (strstart(uri, "rdma:", &p))
rdma_start_incoming_migration(p, errp);
#endif
#if !defined(WIN32)
if (strstart(uri, "tcp:", &p)) {
tcp_start_outgoing_migration(s, p, &local_err);
#ifdef CONFIG_RDMA
- } else if (strstart(uri, "x-rdma:", &p)) {
+ } else if (strstart(uri, "rdma:", &p)) {
rdma_start_outgoing_migration(s, p, &local_err);
#endif
#if !defined(WIN32)
s = migrate_get_current();
- return s->enabled_capabilities[MIGRATION_CAPABILITY_X_RDMA_PIN_ALL];
+ return s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL];
}
bool migrate_auto_converge(void)
# This feature allows us to minimize migration traffic for certain work
# loads, by sending compressed difference of the pages
#
-# @x-rdma-pin-all: Controls whether or not the entire VM memory footprint is
+# @rdma-pin-all: Controls whether or not the entire VM memory footprint is
# mlock()'d on demand or all at once. Refer to docs/rdma.txt for usage.
-# Disabled by default. Experimental: may (or may not) be renamed after
-# further testing is complete. (since 1.6)
+# Disabled by default. (since 2.0)
#
# @zero-blocks: During storage migration encode blocks of zeroes efficiently. This
# essentially saves 1MB of zeroes per block on the wire. Enabling requires
# Since: 1.2
##
{ 'enum': 'MigrationCapability',
- 'data': ['xbzrle', 'x-rdma-pin-all', 'auto-converge', 'zero-blocks'] }
+ 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks'] }
##
# @MigrationCapabilityStatus