2022-03-02 12:05:16 +03:00
|
|
|
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
|
|
|
|
|
From: Fabian Ebner <f.ebner@proxmox.com>
|
|
|
|
|
Date: Wed, 2 Mar 2022 08:35:05 +0100
|
|
|
|
|
Subject: [PATCH] block/backup: move bcs bitmap initialization to job creation
|
|
|
|
|
|
|
|
|
|
For backing up the state of multiple disks from the same time, a job
|
|
|
|
|
for each disk has to be created. It's convenient if the jobs don't
|
|
|
|
|
have to be started at the same time and if operation of the VM can be
|
|
|
|
|
resumed after job creation. This would lead to a window between job
|
|
|
|
|
creation and running the job, where writes can happen. But no writes
|
|
|
|
|
should happen between setting up the copy-before-write filter and
|
|
|
|
|
setting up the block copy state bitmap, because then new writes would
|
|
|
|
|
just pass through.
|
|
|
|
|
|
|
|
|
|
Commit 06e0a9c16405c0a4c1eca33cf286cc04c42066a2 moved initalization of
|
|
|
|
|
the bitmap to setting up the copy-before-write filter when sync_mode
|
|
|
|
|
is not MIRROR_SYNC_MODE_BITMAP. Ensure that the bitmap is initialized
|
|
|
|
|
upon job creation for the remaining case too, by moving the
|
|
|
|
|
backup_init_bcs_bitmap call to backup_job_create.
|
|
|
|
|
|
|
|
|
|
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
|
2022-04-25 11:07:01 +03:00
|
|
|
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
|
2022-03-02 12:05:16 +03:00
|
|
|
---
|
|
|
|
|
block/backup.c | 8 ++++----
|
|
|
|
|
1 file changed, 4 insertions(+), 4 deletions(-)
|
|
|
|
|
|
|
|
|
|
diff --git a/block/backup.c b/block/backup.c
|
update submodule and patches to QEMU 8.0.0
Many changes were necessary this time around:
* QAPI was changed to avoid redundant has_* variables, see commit
44ea9d9be3 ("qapi: Start to elide redundant has_FOO in generated C")
for details. This affected many QMP commands added by Proxmox too.
* Pending querying for migration got split into two functions, one to
estimate, one for exact value, see commit c8df4a7aef ("migration:
Split save_live_pending() into state_pending_*") for details. Relevant
for savevm-async and PBS dirty bitmap.
* Some block (driver) functions got converted to coroutines, so the
Proxmox block drivers needed to be adapted.
* Alloc track auto-detaching during PBS live restore got broken by
AioContext-related changes resulting in a deadlock. The current, hacky
method was replaced by a simpler one. Stefan apparently ran into a
problem with that when he wrote the driver, but there were
improvements in the stream job code since then and I didn't manage to
reproduce the issue. It's a separate patch "alloc-track: fix deadlock
during drop" for now, you can find the details there.
* Async snapshot-related changes:
- The pending querying got adapted to the above-mentioned split and
a patch is added to optimize it/make it more similar to what
upstream code does.
- Added initialization of the compression counters (for
future-proofing).
- It's necessary the hold the BQL (big QEMU lock = iothread mutex)
during the setup phase, because block layer functions are used there
and not doing so leads to racy, hard-to-debug crashes or hangs. It's
necessary to change some upstream code too for this, a version of
the patch "migration: for snapshots, hold the BQL during setup
callbacks" is intended to be upstreamed.
- Need to take the bdrv graph read lock before flushing.
* hmp_info_balloon was moved to a different file.
* Needed to include a new headers from time to time to still get the
correct functions.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-05-15 16:39:53 +03:00
|
|
|
index db3791f4d1..39410dcf8d 100644
|
2022-03-02 12:05:16 +03:00
|
|
|
--- a/block/backup.c
|
|
|
|
|
+++ b/block/backup.c
|
update submodule and patches to 7.1.0
Notable changes:
* The only big change is the switch to using a custom QIOChannel for
savevm-async, because the previously used QEMUFileOps was dropped.
Changes to the current implementation:
* Switch to vector based methods as required for an IO channel. For
short reads the passed-in IO vector is stuffed with zeroes at the
end, just to be sure.
* For reading: The documentation in include/io/channel.h states that
at least one byte should be read, so also error out when whe are
at the very end instead of returning 0.
* For reading: Fix off-by-one error when request goes beyond end.
The wrong code piece was:
if ((pos + size) > maxlen) {
size = maxlen - pos - 1;
}
Previously, the last byte would not be read. It's actually
possible to get a snapshot .raw file that has content all the way
up the final 512 byte (= BDRV_SECTOR_SIZE) boundary without any
trailing zero bytes (I wrote a script to do it).
Luckily, it didn't cause a real issue, because qemu_loadvm_state()
is not interested in the final (i.e. QEMU_VM_VMDESCRIPTION)
section. The buffer for reading it is simply freed up afterwards
and the function will assume that it read the whole section, even
if that's not the case.
* For writing: Make use of the generated blk_pwritev() wrapper
instead of manually wrapping the coroutine to simplify and save a
few lines.
* Adapt to changed interfaces for blk_{pread,pwrite}:
* a9262f551e ("block: Change blk_{pread,pwrite}() param order")
* 3b35d4542c ("block: Add a 'flags' param to blk_pread()")
* bf5b16fa40 ("block: Make blk_{pread,pwrite}() return 0 on success")
Those changes especially affected the qemu-img dd patches, because
the context also changed, but also some of our block drivers used
the functions.
* Drop qemu-common.h include: it got renamed after essentially
everything was moved to other headers. The only remaining user I
could find for things dropped from the header between 7.0 and 7.1
was qemu_get_vm_name() in the iscsi-initiatorname patch, but it
already includes the header to which the function was moved.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2022-10-14 15:07:13 +03:00
|
|
|
@@ -237,8 +237,8 @@ static void backup_init_bcs_bitmap(BackupBlockJob *job)
|
|
|
|
|
true);
|
2022-03-02 12:05:16 +03:00
|
|
|
} else if (job->sync_mode == MIRROR_SYNC_MODE_TOP) {
|
|
|
|
|
/*
|
|
|
|
|
- * We can't hog the coroutine to initialize this thoroughly.
|
|
|
|
|
- * Set a flag and resume work when we are able to yield safely.
|
|
|
|
|
+ * Initialization is costly here. Simply set a flag and let the
|
|
|
|
|
+ * backup_run coroutine resume work once it can yield safely.
|
|
|
|
|
*/
|
|
|
|
|
block_copy_set_skip_unallocated(job->bcs, true);
|
|
|
|
|
}
|
update submodule and patches to 7.1.0
Notable changes:
* The only big change is the switch to using a custom QIOChannel for
savevm-async, because the previously used QEMUFileOps was dropped.
Changes to the current implementation:
* Switch to vector based methods as required for an IO channel. For
short reads the passed-in IO vector is stuffed with zeroes at the
end, just to be sure.
* For reading: The documentation in include/io/channel.h states that
at least one byte should be read, so also error out when whe are
at the very end instead of returning 0.
* For reading: Fix off-by-one error when request goes beyond end.
The wrong code piece was:
if ((pos + size) > maxlen) {
size = maxlen - pos - 1;
}
Previously, the last byte would not be read. It's actually
possible to get a snapshot .raw file that has content all the way
up the final 512 byte (= BDRV_SECTOR_SIZE) boundary without any
trailing zero bytes (I wrote a script to do it).
Luckily, it didn't cause a real issue, because qemu_loadvm_state()
is not interested in the final (i.e. QEMU_VM_VMDESCRIPTION)
section. The buffer for reading it is simply freed up afterwards
and the function will assume that it read the whole section, even
if that's not the case.
* For writing: Make use of the generated blk_pwritev() wrapper
instead of manually wrapping the coroutine to simplify and save a
few lines.
* Adapt to changed interfaces for blk_{pread,pwrite}:
* a9262f551e ("block: Change blk_{pread,pwrite}() param order")
* 3b35d4542c ("block: Add a 'flags' param to blk_pread()")
* bf5b16fa40 ("block: Make blk_{pread,pwrite}() return 0 on success")
Those changes especially affected the qemu-img dd patches, because
the context also changed, but also some of our block drivers used
the functions.
* Drop qemu-common.h include: it got renamed after essentially
everything was moved to other headers. The only remaining user I
could find for things dropped from the header between 7.0 and 7.1
was qemu_get_vm_name() in the iscsi-initiatorname patch, but it
already includes the header to which the function was moved.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2022-10-14 15:07:13 +03:00
|
|
|
@@ -252,8 +252,6 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
|
2022-03-02 12:05:16 +03:00
|
|
|
BackupBlockJob *s = container_of(job, BackupBlockJob, common.job);
|
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
|
|
- backup_init_bcs_bitmap(s);
|
|
|
|
|
-
|
|
|
|
|
if (s->sync_mode == MIRROR_SYNC_MODE_TOP) {
|
|
|
|
|
int64_t offset = 0;
|
|
|
|
|
int64_t count;
|
update submodule and patches to QEMU 8.0.0
Many changes were necessary this time around:
* QAPI was changed to avoid redundant has_* variables, see commit
44ea9d9be3 ("qapi: Start to elide redundant has_FOO in generated C")
for details. This affected many QMP commands added by Proxmox too.
* Pending querying for migration got split into two functions, one to
estimate, one for exact value, see commit c8df4a7aef ("migration:
Split save_live_pending() into state_pending_*") for details. Relevant
for savevm-async and PBS dirty bitmap.
* Some block (driver) functions got converted to coroutines, so the
Proxmox block drivers needed to be adapted.
* Alloc track auto-detaching during PBS live restore got broken by
AioContext-related changes resulting in a deadlock. The current, hacky
method was replaced by a simpler one. Stefan apparently ran into a
problem with that when he wrote the driver, but there were
improvements in the stream job code since then and I didn't manage to
reproduce the issue. It's a separate patch "alloc-track: fix deadlock
during drop" for now, you can find the details there.
* Async snapshot-related changes:
- The pending querying got adapted to the above-mentioned split and
a patch is added to optimize it/make it more similar to what
upstream code does.
- Added initialization of the compression counters (for
future-proofing).
- It's necessary the hold the BQL (big QEMU lock = iothread mutex)
during the setup phase, because block layer functions are used there
and not doing so leads to racy, hard-to-debug crashes or hangs. It's
necessary to change some upstream code too for this, a version of
the patch "migration: for snapshots, hold the BQL during setup
callbacks" is intended to be upstreamed.
- Need to take the bdrv graph read lock before flushing.
* hmp_info_balloon was moved to a different file.
* Needed to include a new headers from time to time to still get the
correct functions.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-05-15 16:39:53 +03:00
|
|
|
@@ -495,6 +493,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
|
2022-03-02 12:05:16 +03:00
|
|
|
block_job_add_bdrv(&job->common, "target", target, 0, BLK_PERM_ALL,
|
|
|
|
|
&error_abort);
|
|
|
|
|
|
|
|
|
|
+ backup_init_bcs_bitmap(job);
|
|
|
|
|
+
|
|
|
|
|
return &job->common;
|
|
|
|
|
|
|
|
|
|
error:
|