pve-qemu/debian/patches/pve/0031-PVE-Backup-pbs-restore...

273 lines
7.6 KiB
Diff
Raw Normal View History

From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: Dietmar Maurer <dietmar@proxmox.com>
Date: Mon, 6 Apr 2020 12:17:01 +0200
Subject: [PATCH] PVE-Backup: pbs-restore - new command to restore from proxmox
backup server
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
squash related patches where there is no good reason to keep them separate. It's a pain during rebase if there are multiple patches changing the same code over and over again. This was especially bad for the backup-related patches. If the history of patches really is needed, it can be extracted via git. Additionally, compilation with partial application of patches was broken since a long time, because one of the master key changes became part of an earlier patch during a past rebase. If only the same files were changed by a subsequent patch and the changes felt to belong together (obvious for later bug fixes, but also done for features e.g. adding master key support for PBS), the patches were squashed together. The PBS namespace support patch was split into the individual parts it changes, i.e. PBS block driver, pbs-restore binary and QMP backup infrastructure, and squashed into the respective patches. No code change is intended, git diff in the submodule should not show any difference between applying all patches before this commit and applying all patches after this commit. The query-proxmox-support QMP function has been left as part of the "PVE-Backup: Proxmox backup patches for QEMU" patch, because it's currently only used there. If it ever is used elsewhere too, it can be split out from there. The recent alloc-track and BQL-related savevm-async changes have been left separate for now, because it's not 100% clear they are the best approach yet. This depends on what upstream decides about the BQL stuff and whether and what kind of issues with the changes pop up. The qemu-img dd snapshot patch has been re-ordered to after the other qemu-img dd patches. Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-05-15 16:39:56 +03:00
[WB: add namespace support]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
meson.build | 4 +
squash related patches where there is no good reason to keep them separate. It's a pain during rebase if there are multiple patches changing the same code over and over again. This was especially bad for the backup-related patches. If the history of patches really is needed, it can be extracted via git. Additionally, compilation with partial application of patches was broken since a long time, because one of the master key changes became part of an earlier patch during a past rebase. If only the same files were changed by a subsequent patch and the changes felt to belong together (obvious for later bug fixes, but also done for features e.g. adding master key support for PBS), the patches were squashed together. The PBS namespace support patch was split into the individual parts it changes, i.e. PBS block driver, pbs-restore binary and QMP backup infrastructure, and squashed into the respective patches. No code change is intended, git diff in the submodule should not show any difference between applying all patches before this commit and applying all patches after this commit. The query-proxmox-support QMP function has been left as part of the "PVE-Backup: Proxmox backup patches for QEMU" patch, because it's currently only used there. If it ever is used elsewhere too, it can be split out from there. The recent alloc-track and BQL-related savevm-async changes have been left separate for now, because it's not 100% clear they are the best approach yet. This depends on what upstream decides about the BQL stuff and whether and what kind of issues with the changes pop up. The qemu-img dd snapshot patch has been re-ordered to after the other qemu-img dd patches. Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-05-15 16:39:56 +03:00
pbs-restore.c | 236 ++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 240 insertions(+)
create mode 100644 pbs-restore.c
diff --git a/meson.build b/meson.build
index d16b97cf3c..6de51c34cb 100644
--- a/meson.build
+++ b/meson.build
@@ -4029,6 +4029,10 @@ if have_tools
vma = executable('vma', files('vma.c', 'vma-reader.c') + genh,
dependencies: [authz, block, crypto, io, qom], install: true)
+ pbs_restore = executable('pbs-restore', files('pbs-restore.c') + genh,
+ dependencies: [authz, block, crypto, io, qom,
+ libproxmox_backup_qemu], install: true)
+
subdir('storage-daemon')
foreach exe: [ 'qemu-img', 'qemu-io', 'qemu-nbd', 'qemu-storage-daemon']
diff --git a/pbs-restore.c b/pbs-restore.c
new file mode 100644
squash related patches where there is no good reason to keep them separate. It's a pain during rebase if there are multiple patches changing the same code over and over again. This was especially bad for the backup-related patches. If the history of patches really is needed, it can be extracted via git. Additionally, compilation with partial application of patches was broken since a long time, because one of the master key changes became part of an earlier patch during a past rebase. If only the same files were changed by a subsequent patch and the changes felt to belong together (obvious for later bug fixes, but also done for features e.g. adding master key support for PBS), the patches were squashed together. The PBS namespace support patch was split into the individual parts it changes, i.e. PBS block driver, pbs-restore binary and QMP backup infrastructure, and squashed into the respective patches. No code change is intended, git diff in the submodule should not show any difference between applying all patches before this commit and applying all patches after this commit. The query-proxmox-support QMP function has been left as part of the "PVE-Backup: Proxmox backup patches for QEMU" patch, because it's currently only used there. If it ever is used elsewhere too, it can be split out from there. The recent alloc-track and BQL-related savevm-async changes have been left separate for now, because it's not 100% clear they are the best approach yet. This depends on what upstream decides about the BQL stuff and whether and what kind of issues with the changes pop up. The qemu-img dd snapshot patch has been re-ordered to after the other qemu-img dd patches. Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-05-15 16:39:56 +03:00
index 0000000000..f03d9bab8d
--- /dev/null
+++ b/pbs-restore.c
squash related patches where there is no good reason to keep them separate. It's a pain during rebase if there are multiple patches changing the same code over and over again. This was especially bad for the backup-related patches. If the history of patches really is needed, it can be extracted via git. Additionally, compilation with partial application of patches was broken since a long time, because one of the master key changes became part of an earlier patch during a past rebase. If only the same files were changed by a subsequent patch and the changes felt to belong together (obvious for later bug fixes, but also done for features e.g. adding master key support for PBS), the patches were squashed together. The PBS namespace support patch was split into the individual parts it changes, i.e. PBS block driver, pbs-restore binary and QMP backup infrastructure, and squashed into the respective patches. No code change is intended, git diff in the submodule should not show any difference between applying all patches before this commit and applying all patches after this commit. The query-proxmox-support QMP function has been left as part of the "PVE-Backup: Proxmox backup patches for QEMU" patch, because it's currently only used there. If it ever is used elsewhere too, it can be split out from there. The recent alloc-track and BQL-related savevm-async changes have been left separate for now, because it's not 100% clear they are the best approach yet. This depends on what upstream decides about the BQL stuff and whether and what kind of issues with the changes pop up. The qemu-img dd snapshot patch has been re-ordered to after the other qemu-img dd patches. Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-05-15 16:39:56 +03:00
@@ -0,0 +1,236 @@
+/*
+ * Qemu image restore helper for Proxmox Backup
+ *
+ * Copyright (C) 2019 Proxmox Server Solutions
+ *
+ * Authors:
+ * Dietmar Maurer (dietmar@proxmox.com)
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include <glib.h>
+#include <getopt.h>
+#include <string.h>
+
+#include "qemu/module.h"
+#include "qemu/error-report.h"
+#include "qemu/main-loop.h"
+#include "qemu/cutils.h"
+#include "qapi/error.h"
+#include "qapi/qmp/qdict.h"
+#include "sysemu/block-backend.h"
+
+#include <proxmox-backup-qemu.h>
+
+static void help(void)
+{
+ const char *help_msg =
squash related patches where there is no good reason to keep them separate. It's a pain during rebase if there are multiple patches changing the same code over and over again. This was especially bad for the backup-related patches. If the history of patches really is needed, it can be extracted via git. Additionally, compilation with partial application of patches was broken since a long time, because one of the master key changes became part of an earlier patch during a past rebase. If only the same files were changed by a subsequent patch and the changes felt to belong together (obvious for later bug fixes, but also done for features e.g. adding master key support for PBS), the patches were squashed together. The PBS namespace support patch was split into the individual parts it changes, i.e. PBS block driver, pbs-restore binary and QMP backup infrastructure, and squashed into the respective patches. No code change is intended, git diff in the submodule should not show any difference between applying all patches before this commit and applying all patches after this commit. The query-proxmox-support QMP function has been left as part of the "PVE-Backup: Proxmox backup patches for QEMU" patch, because it's currently only used there. If it ever is used elsewhere too, it can be split out from there. The recent alloc-track and BQL-related savevm-async changes have been left separate for now, because it's not 100% clear they are the best approach yet. This depends on what upstream decides about the BQL stuff and whether and what kind of issues with the changes pop up. The qemu-img dd snapshot patch has been re-ordered to after the other qemu-img dd patches. Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-05-15 16:39:56 +03:00
+ "usage: pbs-restore [--repository <repo>] [--ns namespace] snapshot archive-name target [command options]\n"
+ ;
+
+ printf("%s", help_msg);
+ exit(1);
+}
+
+typedef struct CallbackData {
+ BlockBackend *target;
+ uint64_t last_offset;
+ bool skip_zero;
+} CallbackData;
+
+static int write_callback(
+ void *callback_data_ptr,
+ uint64_t offset,
+ const unsigned char *data,
+ uint64_t data_len)
+{
+ int res = -1;
+
+ CallbackData *callback_data = (CallbackData *)callback_data_ptr;
+
+ uint64_t last_offset = callback_data->last_offset;
+ if (offset > last_offset) callback_data->last_offset = offset;
+
+ if (data == NULL) {
+ if (callback_data->skip_zero && offset > last_offset) {
+ return 0;
+ }
+ res = blk_pwrite_zeroes(callback_data->target, offset, data_len, 0);
+ } else {
update submodule and patches to 7.1.0 Notable changes: * The only big change is the switch to using a custom QIOChannel for savevm-async, because the previously used QEMUFileOps was dropped. Changes to the current implementation: * Switch to vector based methods as required for an IO channel. For short reads the passed-in IO vector is stuffed with zeroes at the end, just to be sure. * For reading: The documentation in include/io/channel.h states that at least one byte should be read, so also error out when whe are at the very end instead of returning 0. * For reading: Fix off-by-one error when request goes beyond end. The wrong code piece was: if ((pos + size) > maxlen) { size = maxlen - pos - 1; } Previously, the last byte would not be read. It's actually possible to get a snapshot .raw file that has content all the way up the final 512 byte (= BDRV_SECTOR_SIZE) boundary without any trailing zero bytes (I wrote a script to do it). Luckily, it didn't cause a real issue, because qemu_loadvm_state() is not interested in the final (i.e. QEMU_VM_VMDESCRIPTION) section. The buffer for reading it is simply freed up afterwards and the function will assume that it read the whole section, even if that's not the case. * For writing: Make use of the generated blk_pwritev() wrapper instead of manually wrapping the coroutine to simplify and save a few lines. * Adapt to changed interfaces for blk_{pread,pwrite}: * a9262f551e ("block: Change blk_{pread,pwrite}() param order") * 3b35d4542c ("block: Add a 'flags' param to blk_pread()") * bf5b16fa40 ("block: Make blk_{pread,pwrite}() return 0 on success") Those changes especially affected the qemu-img dd patches, because the context also changed, but also some of our block drivers used the functions. * Drop qemu-common.h include: it got renamed after essentially everything was moved to other headers. The only remaining user I could find for things dropped from the header between 7.0 and 7.1 was qemu_get_vm_name() in the iscsi-initiatorname patch, but it already includes the header to which the function was moved. Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2022-10-14 15:07:13 +03:00
+ res = blk_pwrite(callback_data->target, offset, data_len, data, 0);
+ }
+
+ if (res < 0) {
+ fprintf(stderr, "blk_pwrite failed at offset %ld length %ld (%d) - %s\n", offset, data_len, res, strerror(-res));
+ return res;
+ }
+
+ return 0;
+}
+
+int main(int argc, char **argv)
+{
+ Error *main_loop_err = NULL;
+ const char *format = "raw";
+ const char *repository = NULL;
squash related patches where there is no good reason to keep them separate. It's a pain during rebase if there are multiple patches changing the same code over and over again. This was especially bad for the backup-related patches. If the history of patches really is needed, it can be extracted via git. Additionally, compilation with partial application of patches was broken since a long time, because one of the master key changes became part of an earlier patch during a past rebase. If only the same files were changed by a subsequent patch and the changes felt to belong together (obvious for later bug fixes, but also done for features e.g. adding master key support for PBS), the patches were squashed together. The PBS namespace support patch was split into the individual parts it changes, i.e. PBS block driver, pbs-restore binary and QMP backup infrastructure, and squashed into the respective patches. No code change is intended, git diff in the submodule should not show any difference between applying all patches before this commit and applying all patches after this commit. The query-proxmox-support QMP function has been left as part of the "PVE-Backup: Proxmox backup patches for QEMU" patch, because it's currently only used there. If it ever is used elsewhere too, it can be split out from there. The recent alloc-track and BQL-related savevm-async changes have been left separate for now, because it's not 100% clear they are the best approach yet. This depends on what upstream decides about the BQL stuff and whether and what kind of issues with the changes pop up. The qemu-img dd snapshot patch has been re-ordered to after the other qemu-img dd patches. Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-05-15 16:39:56 +03:00
+ const char *backup_ns = NULL;
+ const char *keyfile = NULL;
+ int verbose = false;
+ bool skip_zero = false;
+
+ error_init(argv[0]);
+
+ for (;;) {
+ static const struct option long_options[] = {
+ {"help", no_argument, 0, 'h'},
+ {"skip-zero", no_argument, 0, 'S'},
+ {"verbose", no_argument, 0, 'v'},
+ {"format", required_argument, 0, 'f'},
+ {"repository", required_argument, 0, 'r'},
squash related patches where there is no good reason to keep them separate. It's a pain during rebase if there are multiple patches changing the same code over and over again. This was especially bad for the backup-related patches. If the history of patches really is needed, it can be extracted via git. Additionally, compilation with partial application of patches was broken since a long time, because one of the master key changes became part of an earlier patch during a past rebase. If only the same files were changed by a subsequent patch and the changes felt to belong together (obvious for later bug fixes, but also done for features e.g. adding master key support for PBS), the patches were squashed together. The PBS namespace support patch was split into the individual parts it changes, i.e. PBS block driver, pbs-restore binary and QMP backup infrastructure, and squashed into the respective patches. No code change is intended, git diff in the submodule should not show any difference between applying all patches before this commit and applying all patches after this commit. The query-proxmox-support QMP function has been left as part of the "PVE-Backup: Proxmox backup patches for QEMU" patch, because it's currently only used there. If it ever is used elsewhere too, it can be split out from there. The recent alloc-track and BQL-related savevm-async changes have been left separate for now, because it's not 100% clear they are the best approach yet. This depends on what upstream decides about the BQL stuff and whether and what kind of issues with the changes pop up. The qemu-img dd snapshot patch has been re-ordered to after the other qemu-img dd patches. Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-05-15 16:39:56 +03:00
+ {"ns", required_argument, 0, 'n'},
+ {"keyfile", required_argument, 0, 'k'},
+ {0, 0, 0, 0}
+ };
+ int c = getopt_long(argc, argv, "hvf:r:k:", long_options, NULL);
+ if (c == -1) {
+ break;
+ }
+ switch (c) {
+ case ':':
+ fprintf(stderr, "missing argument for option '%s'\n", argv[optind - 1]);
+ return -1;
+ case '?':
+ fprintf(stderr, "unrecognized option '%s'\n", argv[optind - 1]);
+ return -1;
+ case 'f':
+ format = g_strdup(argv[optind - 1]);
+ break;
+ case 'r':
+ repository = g_strdup(argv[optind - 1]);
+ break;
squash related patches where there is no good reason to keep them separate. It's a pain during rebase if there are multiple patches changing the same code over and over again. This was especially bad for the backup-related patches. If the history of patches really is needed, it can be extracted via git. Additionally, compilation with partial application of patches was broken since a long time, because one of the master key changes became part of an earlier patch during a past rebase. If only the same files were changed by a subsequent patch and the changes felt to belong together (obvious for later bug fixes, but also done for features e.g. adding master key support for PBS), the patches were squashed together. The PBS namespace support patch was split into the individual parts it changes, i.e. PBS block driver, pbs-restore binary and QMP backup infrastructure, and squashed into the respective patches. No code change is intended, git diff in the submodule should not show any difference between applying all patches before this commit and applying all patches after this commit. The query-proxmox-support QMP function has been left as part of the "PVE-Backup: Proxmox backup patches for QEMU" patch, because it's currently only used there. If it ever is used elsewhere too, it can be split out from there. The recent alloc-track and BQL-related savevm-async changes have been left separate for now, because it's not 100% clear they are the best approach yet. This depends on what upstream decides about the BQL stuff and whether and what kind of issues with the changes pop up. The qemu-img dd snapshot patch has been re-ordered to after the other qemu-img dd patches. Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-05-15 16:39:56 +03:00
+ case 'n':
+ backup_ns = g_strdup(argv[optind - 1]);
+ break;
+ case 'k':
+ keyfile = g_strdup(argv[optind - 1]);
+ break;
+ case 'v':
+ verbose = true;
+ break;
+ case 'S':
+ skip_zero = true;
+ break;
+ case 'h':
+ help();
+ return 0;
+ }
+ }
+
+ if (optind >= argc - 2) {
+ fprintf(stderr, "missing arguments\n");
+ help();
+ return -1;
+ }
+
+ if (repository == NULL) {
+ repository = getenv("PBS_REPOSITORY");
+ }
+
+ if (repository == NULL) {
+ fprintf(stderr, "no repository specified\n");
+ help();
+ return -1;
+ }
+
+ char *snapshot = argv[optind++];
+ char *archive_name = argv[optind++];
+ char *target = argv[optind++];
+
+ const char *password = getenv("PBS_PASSWORD");
+ const char *fingerprint = getenv("PBS_FINGERPRINT");
+ const char *key_password = getenv("PBS_ENCRYPTION_PASSWORD");
+
+ if (qemu_init_main_loop(&main_loop_err)) {
+ g_error("%s", error_get_pretty(main_loop_err));
+ }
+
+ bdrv_init();
+ module_call_init(MODULE_INIT_QOM);
+
+ if (verbose) {
+ fprintf(stderr, "connecting to repository '%s'\n", repository);
+ }
+ char *pbs_error = NULL;
squash related patches where there is no good reason to keep them separate. It's a pain during rebase if there are multiple patches changing the same code over and over again. This was especially bad for the backup-related patches. If the history of patches really is needed, it can be extracted via git. Additionally, compilation with partial application of patches was broken since a long time, because one of the master key changes became part of an earlier patch during a past rebase. If only the same files were changed by a subsequent patch and the changes felt to belong together (obvious for later bug fixes, but also done for features e.g. adding master key support for PBS), the patches were squashed together. The PBS namespace support patch was split into the individual parts it changes, i.e. PBS block driver, pbs-restore binary and QMP backup infrastructure, and squashed into the respective patches. No code change is intended, git diff in the submodule should not show any difference between applying all patches before this commit and applying all patches after this commit. The query-proxmox-support QMP function has been left as part of the "PVE-Backup: Proxmox backup patches for QEMU" patch, because it's currently only used there. If it ever is used elsewhere too, it can be split out from there. The recent alloc-track and BQL-related savevm-async changes have been left separate for now, because it's not 100% clear they are the best approach yet. This depends on what upstream decides about the BQL stuff and whether and what kind of issues with the changes pop up. The qemu-img dd snapshot patch has been re-ordered to after the other qemu-img dd patches. Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2023-05-15 16:39:56 +03:00
+ ProxmoxRestoreHandle *conn = proxmox_restore_new_ns(
+ repository,
+ snapshot,
+ backup_ns,
+ password,
+ keyfile,
+ key_password,
+ fingerprint,
+ &pbs_error
+ );
+ if (conn == NULL) {
+ fprintf(stderr, "restore failed: %s\n", pbs_error);
+ return -1;
+ }
+
+ int res = proxmox_restore_connect(conn, &pbs_error);
+ if (res < 0 || pbs_error) {
+ fprintf(stderr, "restore failed (connection error): %s\n", pbs_error);
+ return -1;
+ }
+
+ QDict *options = qdict_new();
+
+ if (format) {
+ qdict_put_str(options, "driver", format);
+ }
+
+
+ if (verbose) {
+ fprintf(stderr, "open block backend for target '%s'\n", target);
+ }
+ Error *local_err = NULL;
+ int flags = BDRV_O_RDWR;
+ BlockBackend *blk = blk_new_open(target, NULL, options, flags, &local_err);
+ if (!blk) {
+ fprintf(stderr, "%s\n", error_get_pretty(local_err));
+ return -1;
+ }
+
+ CallbackData *callback_data = calloc(sizeof(CallbackData), 1);
+
+ callback_data->target = blk;
+ callback_data->skip_zero = skip_zero;
+ callback_data->last_offset = 0;
+
+ // blk_set_enable_write_cache(blk, !writethrough);
+
+ if (verbose) {
+ fprintf(stderr, "starting to restore snapshot '%s'\n", snapshot);
+ fflush(stderr); // ensure we do not get printed after the progress log
+ }
+ res = proxmox_restore_image(
+ conn,
+ archive_name,
+ write_callback,
+ callback_data,
+ &pbs_error,
+ verbose);
+
+ proxmox_restore_disconnect(conn);
+ blk_unref(blk);
+
+ if (res < 0) {
+ fprintf(stderr, "restore failed: %s\n", pbs_error);
+ return -1;
+ }
+
+ return 0;
+}