Commit Graph

6528 Commits (b7719bcad2e92bab5aae3166fa9011f127e6ee49)

Author SHA1 Message Date
Nir Soffer b7719bcad2 iotests: Make qemu_nbd_popen() a contextmanager
Instead of duplicating the code to wait until the server is ready and
remember to terminate the server and wait for it, make it possible to
use like this:

    with qemu_nbd_popen('-k', sock, image):
        # Access image via qemu-nbd socket...

Only test 264 used this helper, but I had to modify the output since it
did not consistently when starting and stopping qemu-nbd.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Message-Id: <20200727215846.395443-3-nsoffer@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2020-07-28 09:54:43 -05:00
Peter Maydell 23ae28783f Block patches for 5.1:
- Coverity fix
 - iotests fix for rx and avr
 - iotests fix for qcow2 -o compat=0.10
 -----BEGIN PGP SIGNATURE-----
 
 iQFGBAABCAAwFiEEkb62CjDbPohX0Rgp9AfbAGHVz0AFAl8e5mcSHG1yZWl0ekBy
 ZWRoYXQuY29tAAoJEPQH2wBh1c9A8WEIAIgYTPqy+BeJ6xaNczH2fsYoYq9uwPfb
 bKkPp5UAaV4krmpDCaddMGK4xu/92t/i5iRTxuhik1CufzUe2rWWO7KNLi5Qpi0X
 9bOCQmlOWpmykyWz5UQ9Vd3W9+rdpyESJ+FCrd3K7DGh13l70vP3f5x/iLFOlnia
 50MBD1Y3+zRqpHcxu8gP5Q+TOlddgRkuuX2STPo6DQnmI3llL3FFWQhcT+OfltJl
 Avc9AF91CoimuDyIDSlONphsx8WWbA/3byYxqINdN0aRy8hrQ+l+BzAMZOGjYxIM
 Sy++rUPHeIQbOlmY06UNA+J6sWcG9bipc1xaSEUr/I8iZ1AbL01O91A=
 =QzwY
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-07-27' into staging

Block patches for 5.1:
- Coverity fix
- iotests fix for rx and avr
- iotests fix for qcow2 -o compat=0.10

# gpg: Signature made Mon 27 Jul 2020 15:36:23 BST
# gpg:                using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg:                issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1  1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-07-27:
  iotests/197: Fix for compat=0.10
  iotests: Select a default machine for the rx and avr targets
  block/amend: Check whether the node exists

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2020-07-28 09:15:44 +01:00
Max Reitz 1855536256 iotests/197: Fix for compat=0.10
Writing zeroes to a qcow2 v2 images without a backing file results in an
unallocated cluster as of 61b3043965.  197 has a test for COR-ing a
cluster on an image without a backing file, which means that the data
will be zero, so now on a v2 image that cluster will just stay
unallocated, and so the test fails.  Just force compat=1.1 for that
particular case to enforce the cluster to get allocated.

Fixes: 61b3043965
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200727135237.1096841-1-mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
2020-07-27 16:35:17 +02:00
Thomas Huth 733dafe491 iotests: Select a default machine for the rx and avr targets
If you are building only with either the new rx-softmmu or avr-softmmu
target, "make check-block" fails a couple of tests since there is no
default machine defined in these new targets. We have to select a machine
in the "check" script for these, just like we already do for the arm- and
tricore-softmmu targets.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20200722161908.25383-1-thuth@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-07-27 12:37:25 +02:00
Robert Foley 4a70232b1d tests/vm: add shutdown timeout in basevm.py
We are adding the shutdown timeout to solve an issue
we now see where the aarch64 VMs timeout on shutdown
under TCG.

There is a new 3 second timeout in machine.py,
which we override in basevm.py when shutting down.

Signed-off-by: Robert Foley <robert.foley@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200717203041.9867-4-robert.foley@linaro.org>
Message-Id: <20200724064509.331-17-alex.bennee@linaro.org>
2020-07-27 09:41:58 +01:00
Alex Bennée 4c5aeb1238 tests/docker: add support for DEB_KEYRING
For installing stuff from sid or ports you may need to manually
specify the location of the keyring. You can even import keys into
your personal keyring and point it there, e.g.:

  gpg --keyserver keyring.debian.org --recv-keys 84C573CD4E1AFD6C
  make docker-binfmt-image-debian-sid-hppa DEB_TYPE=sid DEB_ARCH=hppa \
      DEB_URL=http://ftp.ports.debian.org/debian-ports/ \
      EXECUTABLE=./hppa-linux-user/qemu-hppa V=1 \
      DEB_KEYRING=${HOME}/.gnupg/pubring.kbx

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20200724064509.331-12-alex.bennee@linaro.org>
2020-07-27 09:41:35 +01:00
Alex Bennée 182ec06076 tests/docker: fix binfmt_misc image building
When we updated the arguments for docker.py we missed a bit.

Fixes: dfae628459 ("docker.py/build: support -t and -f arguments")
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200724064509.331-11-alex.bennee@linaro.org>
2020-07-27 09:41:22 +01:00
Alex Bennée e336cec3a5 tests/docker: fix update command due to python3 str/bytes distinction
Does this seem convoluted to you? It feels a little complicated to me.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200724064509.331-10-alex.bennee@linaro.org>
2020-07-27 09:41:18 +01:00
Peter Maydell 0c1fd2f41f Block layer patches:
- file-posix: Handle `EINVAL` fallocate return value
 - qemu-img convert -n: Keep qcow2 v2 target sparse
 -----BEGIN PGP SIGNATURE-----
 
 iQJFBAABCAAvFiEE3D3rFZqa+V09dFb+fwmycsiPL9YFAl8XDZgRHGt3b2xmQHJl
 ZGhhdC5jb20ACgkQfwmycsiPL9bpFQ/9EnS4iV9w0KW/NuJ4FIVdBD/VZFzokDLi
 1vXVVEjoxAxxiP8KlGM9HRi5NtvOMgzKhNGias0wOFiBorx8Ppfc+3sqwygc2dnw
 Vbl/od2D7xQZkddnp4Upo70m+eWRW6xaxX+lAcl6iS3gBPDwExLaYfBN8lFUyRrs
 T4C0miD+abEEyL3C5A4cEZJ7CIs0n7AqZkqgytWA7clwy79VgDSuMOgP6DOP1tGH
 1uK4gMCB0xbn+PHk96lXPORcLwDBOP0PIluo/zBmffzsEZN1Lv5ddVmxMQWSivin
 UmAbpeEtSw9Py5lRVmLSBYvolVOUleE/Rlzad2iue2be5/G8VP8xiRYMp9mUVpLO
 +LPMUd9NRkPx7wjUJMPKF0G9FgVO7R0+9J6rC33aKBj2XAlxY6qQlqUN2Jo11/fK
 2+9AkU7WVqx3vuW2Zz7wjq3Rjvpg/sK+V3P3Cm6HTwwwPbEwv8GcFe6eKdvJrZ9K
 hhwiFSUOd90OUAdKOQXKMFSZ/t1TrZhdX882Hvth11/AlQAUY4cxQbSKcc2nrvLu
 Axk0Va3haOD+ReRTs8W/iYNdrXGZmbr3MCkNiK3QSvnrdj602ompco7xyDTX1/qH
 6Hu28q7jUG3p3cApLQIZVjmogfqcGU7SWIY4lp9HZqtGP0z+pmWg46UNqzlKLyv5
 Y/fVHHshlRU=
 =QhOf
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- file-posix: Handle `EINVAL` fallocate return value
- qemu-img convert -n: Keep qcow2 v2 target sparse

# gpg: Signature made Tue 21 Jul 2020 16:45:28 BST
# gpg:                using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg:                issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74  56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream:
  iotests: Test sparseness for qemu-img convert -n
  qcow2: Implement v2 zero writes with discard if possible
  file-posix: Handle `EINVAL` fallocate return value

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2020-07-21 19:25:48 +01:00
Kevin Wolf 4a01e27ddc iotests: Test sparseness for qemu-img convert -n
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200721135520.72355-3-kwolf@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-21 17:44:35 +02:00
Peter Maydell 98d897eb4b Block patches for 5.1:
- Let LUKS images only be shared between VMs if the guest device was
   configured to allow that
 - Fix abort() from bdrv_aio_cancel() for guest devices without a BDS
 -----BEGIN PGP SIGNATURE-----
 
 iQFGBAABCAAwFiEEkb62CjDbPohX0Rgp9AfbAGHVz0AFAl8W1cUSHG1yZWl0ekBy
 ZWRoYXQuY29tAAoJEPQH2wBh1c9AA7wH/1ckTrSDMroVi1adBrz+KycA3O9kSmzl
 Z4qvLEdj/j7oc3ud96faCguPBv36ogjq/Wu7wl2/5ufNCVtr39LQLi7LeUiuzcuM
 mZaov8BaFPWcVnEyqJKES/VfOB4AbT2LfFhqC+L2VGShsxFDTVOAno6R87Onkkuy
 87qN9gs2b77pyhUQxvgKJzfvjDy0YRDyYn30eBo/WisEjfDfLrf2Fv/wpZze0OC8
 9cqEvczTU2nQzX5k2NnANbf8Vr/U6H3tay/f/C3FZ0lWHcqWEieIKlWp4iYezsTk
 B/LKDMWtvPgrZGxmsHrwOs9Y1Tfre3w86PrLXAC44WpX6OghhXKNKrQ=
 =BV7d
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-07-21' into staging

Block patches for 5.1:
- Let LUKS images only be shared between VMs if the guest device was
  configured to allow that
- Fix abort() from bdrv_aio_cancel() for guest devices without a BDS

# gpg: Signature made Tue 21 Jul 2020 12:47:17 BST
# gpg:                using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg:                issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1  1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-07-21:
  block: fix bdrv_aio_cancel() for ENOMEDIUM requests
  qemu-iotests: add testcase for bz #1857490
  block/crypto: disallow write sharing by default

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2020-07-21 14:03:45 +01:00
Maxim Levitsky 0fca43de1b qemu-iotests: add testcase for bz #1857490
Test that we can't write-share raw luks images by default,
but we still can with share-rw=on

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20200719122059.59843-3-mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-07-21 10:49:02 +02:00
Alexander Bulekov bcbad8b05c fuzz: Fix leak when assembling datadir path string
We freed the string containing the final datadir path, but did not free
the path to the executable's directory that we get from
g_path_get_dirname(). Fix that.

Reported-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <20200717163523.1591-1-alxndr@bu.edu>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2020-07-21 07:21:54 +02:00
Peter Maydell 920d213cb2 Block layer patches:
- file-posix: Fix read-only Linux block devices with auto-read-only
 - Require aligned image size with O_DIRECT to avoid assertion failure
 - Allow byte-aligned direct I/O on NFS instead of guessing 4k alignment
 - Fix nbd_export_close_all() crash
 - Fix race in iotests case 030
 - qemu-img resize: Require --shrink for shrinking all image formats
 - crypto: use a stronger private key for tests
 - Remove VXHS block device
 - MAINTAINERS: vvfat: set status to odd fixes
 -----BEGIN PGP SIGNATURE-----
 
 iQJFBAABCAAvFiEE3D3rFZqa+V09dFb+fwmycsiPL9YFAl8RmhYRHGt3b2xmQHJl
 ZGhhdC5jb20ACgkQfwmycsiPL9adtA//ZfPLfZoNmo/G7tEiiuiVxPfxn3AP50dg
 E5m+3RjyTlw2ss7G9rh09DD1mY2NCO3dPWkTvmYio1mgsYA4FeSsK2Zk4gIQtGEA
 5dKjKVQ7rNqvHp1hqbUWuBF0RoRzRd8shm5y8WZWds/fVWG6gp5+Ydyulb7WqIN1
 moSK/lBAURm18EcGBB1HCkYzd+pp6VgvcpMkeDXr9SYC9JL8eualNgwchvWgl1f0
 sXD3GspKhpDE8rihm4EzQHFe5Q4X35YNlKAZ0gqC5jewlWVLmFJMjaW9icoS5Z9j
 W/Y3gVQ0tSwzXJZrfy9Q8XBJj80lnOntBF3j9xG94J6ubaDRiYdIszTxExomylkF
 wEGwfKJdfKu2DZSrxjBPdtGs1fgf9upm+rR3KKZuT4qc7mVv4xoyEAgzhJg0oPtR
 153jmb3n2SYNJTybqkLA+UIpf4xeHVcJbXy6C7om4mg+UQRzYgqd6jQUNpoT5Eix
 BbnfefsGwATgJJdYW7h5fBim6r93QWaTj2+j19PG6XVAyLhK2aEGrpZveECJm/Wl
 hzc0EaSxCYJB9l0JlHSDJ690tCX0mjh7qcBxNBLXwbp9+BdBmKfTBhaSu0dxSIth
 OW9D0Ky8k5edMeLGoQvophOsa9ktctNbZtFZfWI6QZEvMxzk/+7pNWOdUCsFVLzV
 BsIEN7gOQsE=
 =ijIG
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- file-posix: Fix read-only Linux block devices with auto-read-only
- Require aligned image size with O_DIRECT to avoid assertion failure
- Allow byte-aligned direct I/O on NFS instead of guessing 4k alignment
- Fix nbd_export_close_all() crash
- Fix race in iotests case 030
- qemu-img resize: Require --shrink for shrinking all image formats
- crypto: use a stronger private key for tests
- Remove VXHS block device
- MAINTAINERS: vvfat: set status to odd fixes

# gpg: Signature made Fri 17 Jul 2020 13:31:18 BST
# gpg:                using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg:                issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74  56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream:
  file-posix: Fix leaked fd in raw_open_common() error path
  file-posix: Fix check_hdev_writable() with auto-read-only
  file-posix: Move check_hdev_writable() up
  file-posix: Allow byte-aligned O_DIRECT with NFS
  block: Require aligned image size to avoid assertion failure
  iotests: test shutdown when bitmap is exported through NBD
  nbd: make nbd_export_close_all() synchronous
  iotests/030: Reduce job speed to make race less likely
  crypto: use a stronger private key for tests
  qemu-img resize: Require --shrink for shrinking all image formats
  Remove VXHS block device
  vvfat: set status to odd fixes

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2020-07-17 14:58:13 +01:00
Vladimir Sementsov-Ogievskiy d047cfa78d iotests: test shutdown when bitmap is exported through NBD
Test shutdown when bitmap is exported through NBD and active client
exists. The previous patch fixes a crash, provoked by this scenario.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Tested-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200714162234.13113-3-vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-17 14:20:57 +02:00
Kevin Wolf f21f12936f iotests/030: Reduce job speed to make race less likely
It can happen that the throttling of the stream job doesn't make it slow
enough that we can be sure that it still exists when it is referenced
again. Just use a much smaller speed to make this very unlikely to
happen again.

Reported-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200716132829.20127-1-kwolf@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-17 14:20:57 +02:00
Daniel P. Berrangé 3e018afbfe crypto: use a stronger private key for tests
The unit tests using the x509 crypto functionality have started
failing in Fedora 33 rawhide with a message like

      The certificate uses an insecure algorithm

This is result of Fedora changes to support strong crypto [1]. RSA
with 1024 bit key is viewed as legacy and thus insecure. Generate
a new private key which is 3072 bits long and reasonable future
proof.

[1] https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20200715154701.1041325-1-berrange@redhat.com>
Reviewed-by: Kashyap Chamarthy <kchamart@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-17 14:20:57 +02:00
Marc-André Lureau a08464521c Remove VXHS block device
The vxhs code doesn't compile since v2.12.0. There's no point in fixing
and then adding CI for a config that our users have demonstrated that
they do not use; better to just remove it.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200711065926.2204721-1-marcandre.lureau@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-17 14:20:57 +02:00
Alexander Bulekov f5ec79f5e0 fuzz: Expect the cmdline in a freeable GString
In the initial FuzzTarget, get_init_cmdline returned a char *. With this
API, we had no guarantee about where the string came from. For example,
i440fx-qtest-reboot-fuzz simply returned a pointer to a string literal,
while the QOS-based targets build the arguments out in a GString an
return the gchar *str pointer. Since we did not try to free the cmdline,
we have a leak for any targets that do not simply return string
literals. Clean up this mess by forcing fuzz-targets to return
a GString, that we can free.

Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <20200714174616.20709-1-alxndr@bu.edu>
Reviewed-by: Darren Kenny <darren.kenny@oracle.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2020-07-17 10:44:23 +02:00
Li Qiang 15c51f724e tests: qmp-cmd-test: fix memory leak
Properly free each test response to avoid memory leak and separate
qtest_qmp() calls with spare lines, in a consistent manner.

Fixes: 5b88849e7b9("tests/qmp-cmd-test: Add qmp/object-add-failure-modes")
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Li Qiang <liq3ea@163.com>
Message-Id: <20200715154117.15456-1-liq3ea@163.com>
Fixes: 9fc719b869 ("tests/qmp-cmd-test: Add qmp/object-add-duplicate-id")
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2020-07-17 10:44:23 +02:00
Li Qiang 82e258db15 qtest: bios-tables-test: fix a memory leak
Fixes: 5da7c35e25a("bios-tables-test: Add Q35/TPM-TIS test")
Signed-off-by: Li Qiang <liq3ea@163.com>
Message-Id: <20200714153536.66060-1-liq3ea@163.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Igor Mammedov <imammedo@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2020-07-17 10:44:23 +02:00
Stefan Berger df8a756893 tests: tpm: Skip over pcrUpdateCounter byte in result comparison
The TPM 2 code in libtpms was fixed to handle the PCR 'TCB group' according
to the PCClient profile. The change of the PCRs belonging to the 'TCB group'
now affects the pcrUpdateCounter in the TPM2_PCRRead() responses where its
value is now different (typically lower by '1') than what it was before. To
not fail the tests, we skip the comparison of the 14th byte, which
represents the pcrUpdateCounter.

Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-id: 20200707201625.4177419-3-stefanb@linux.vnet.ibm.com
2020-07-15 14:57:33 -04:00
Peter Maydell dfcade7cef Final fixes for 5.1-rc0
- minor documentation nit
   - docker.py bootstrap fixes
   - tweak containers.yml wildcards
   - fix float16 nan detection
   - conditional use of -Wpsabi
   - fix missing iotlb data for plugins
   - proper locking for helper based bb count
   - drop ppc64abi32 from the plugin check-tcg test
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAl8O4XwACgkQ+9DbCVqe
 KkQacQgAhAFT5KszapViBuKR78KacFWmGbeOYvwwxW/YMmtAJmABaxS6E2mJxUgw
 8Bgc8nHiQ1V3M4EBzOKT+AEdCGoHYAHnpDnuYQ1M9b82Zf5Tmkg9pBL5TvucQ20w
 M7X1Z3UJtau1sMhH+k3Nc3OhZg4dqiOMWven7kefXjA+13kuu/40+hI/bohw58L9
 OAWae2PUevxR2S01q5n6w4frUCnI+IPZoGLQZjSCbcakZqsuCFrc/Z20cfOIQM76
 pe0Rm8VRHQ9XCuPGvZ9OOaCJQ7QF22MHMZmxNnlWJ4+zzsP32RjO273XknfFOmBN
 Af5eTZ8lKLBOxywXTXQ/ehSHf7FQ9Q==
 =rQef
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/stsquad/tags/pull-misc-for-rc0-150720-3' into staging

Final fixes for 5.1-rc0

  - minor documentation nit
  - docker.py bootstrap fixes
  - tweak containers.yml wildcards
  - fix float16 nan detection
  - conditional use of -Wpsabi
  - fix missing iotlb data for plugins
  - proper locking for helper based bb count
  - drop ppc64abi32 from the plugin check-tcg test

# gpg: Signature made Wed 15 Jul 2020 11:59:08 BST
# gpg:                using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* remotes/stsquad/tags/pull-misc-for-rc0-150720-3:
  .travis.yml: skip ppc64abi32-linux-user with plugins
  plugins: expand the bb plugin to be thread safe and track per-cpu
  cputlb: ensure we save the IOTLB data in case of reset
  tests/plugins: don't unconditionally add -Wpsabi
  fpu/softfloat: fix up float16 nan recognition
  gitlab-ci/containers: Add missing wildcard where we should look for changes
  docker.py: fix fetching of FROM layers
  tests/docker: Remove the libssh workaround from the ubuntu 20.04 image
  docs/devel: fix grammar in multi-thread-tcg

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2020-07-15 17:16:39 +01:00
Peter Maydell 673205379f Python patches for 5.1
- Reduce race conditions on QEMUMachine::shutdown()
 
  1. Remove the "bare except" pattern in the existing shutdown code,
     which can mask problems and make debugging difficult.
  2. Ensure that post-shutdown cleanup is always performed, even when
     graceful termination fails.
  3. Unify cleanup paths such that no matter how the VM is terminated,
     the same functions and steps are always taken to reset the object
     state.
  4. Rewrite shutdown() such that any error encountered when attempting
     a graceful shutdown will be raised as an AbnormalShutdown exception.
     The pythonic idiom is to allow the caller to decide if this is a
     problem or not.
 
 - Modify part of the python/qemu library to comply with:
 
   . mypy --strict
   . pylint
   . flake8
 
 - Script for the TCG Continuous Benchmarking project that uses
   callgrind to dissect QEMU execution into three main phases:
 
   . code generation
   . JIT execution
   . helpers execution
 
 CI jobs results:
 . https://cirrus-ci.com/build/5421349961203712
 . https://gitlab.com/philmd/qemu/-/pipelines/166556001
 . https://travis-ci.org/github/philmd/qemu/builds/708102347
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE+qvnXhKRciHc/Wuy4+MsLN6twN4FAl8OGCUACgkQ4+MsLN6t
 wN52ZQ//cB8pRhOeirMzPTamtAePAHhU9kRKSQ4fFzVM/AHxaO3qO3KrMKAv4JnR
 1fwdyAbum0Dv+HrRAXbIVz/lgrlTb6tLqwk20YWlmNUR2CtMmno9fPQzg6LeFWBe
 +qr2/anXWBqRZu5uHzQ3Ly3LxlVXE3qmStxlttNxed9wK9wdhsQ4ijOrExXPmMI5
 IQi9tKijsGvOczcSpIwJET6Px0L/GIYHSFGXfHv4LiJCMQUX4gqY1akr+/2adqlr
 ElcOHlvWG90GHftTC+hq+KbSNmpkwJEnXLHC+HwMaYmQuhH4+U6f9AcrnzYGHjoQ
 84muuYt5CBvT/Q38qO3dUsr42oD8uiDZe2cpD5DmcAHj6U9sza4UwU2Nkd4cNvRD
 OiYRWuHQDzQhwPGbzBym1GRxRlejmPXCpvZ4XqaYODkzhEHi8UVq5AOsNhpTGWVk
 WoDZ3NtnZb9WbU93wDordhWTtHKHOLHNk4CuJLbdRvkW+sg8aFqUObSRvc6RqyU8
 1DXWMMQr9UfCDfMMbp/OCymphn0VrAaixIRIjmYc6nL5mQlyugMwij55jzyugqBH
 rN5bdcsqxAhBTD03lu0lbDn1BYIqbZFXABXeDnJNbxmY/6gD0E1HKWRvIKjxzH+t
 L19QXsOWLoVUpUIjxlOgMlb9QJLPc9m/N3yVOanO0zVrMSBmOKQ=
 =Jp6n
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/philmd-gitlab/tags/python-next-20200714' into staging

Python patches for 5.1

- Reduce race conditions on QEMUMachine::shutdown()

 1. Remove the "bare except" pattern in the existing shutdown code,
    which can mask problems and make debugging difficult.
 2. Ensure that post-shutdown cleanup is always performed, even when
    graceful termination fails.
 3. Unify cleanup paths such that no matter how the VM is terminated,
    the same functions and steps are always taken to reset the object
    state.
 4. Rewrite shutdown() such that any error encountered when attempting
    a graceful shutdown will be raised as an AbnormalShutdown exception.
    The pythonic idiom is to allow the caller to decide if this is a
    problem or not.

- Modify part of the python/qemu library to comply with:

  . mypy --strict
  . pylint
  . flake8

- Script for the TCG Continuous Benchmarking project that uses
  callgrind to dissect QEMU execution into three main phases:

  . code generation
  . JIT execution
  . helpers execution

CI jobs results:
. https://cirrus-ci.com/build/5421349961203712
. https://gitlab.com/philmd/qemu/-/pipelines/166556001
. https://travis-ci.org/github/philmd/qemu/builds/708102347

# gpg: Signature made Tue 14 Jul 2020 21:40:05 BST
# gpg:                using RSA key FAABE75E12917221DCFD6BB2E3E32C2CDEADC0DE
# gpg: Good signature from "Philippe Mathieu-Daudé (F4BUG) <f4bug@amsat.org>" [full]
# Primary key fingerprint: FAAB E75E 1291 7221 DCFD  6BB2 E3E3 2C2C DEAD C0DE

* remotes/philmd-gitlab/tags/python-next-20200714:
  python/qmp.py: add QMPProtocolError
  python/qmp.py: add casts to JSON deserialization
  python/qmp.py: Do not return None from cmd_obj
  python/qmp.py: re-absorb MonitorResponseError
  iotests.py: use qemu.qmp type aliases
  python/qmp.py: Define common types
  python/machine.py: change default wait timeout to 3 seconds
  python/machine.py: re-add sigkill warning suppression
  python/machine.py: split shutdown into hard and soft flavors
  tests/acceptance: Don't test reboot on cubieboard
  tests/acceptance: wait() instead of shutdown() where appropriate
  python/machine.py: Make wait() call shutdown()
  python/machine.py: Add a configurable timeout to shutdown()
  python/machine.py: Prohibit multiple shutdown() calls
  python/machine.py: Perform early cleanup for wait() calls, too
  python/machine.py: Add _early_cleanup hook
  python/machine.py: Close QMP socket in cleanup
  python/machine.py: consolidate _post_shutdown()
  scripts/performance: Add dissect.py script

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2020-07-15 13:04:27 +01:00
Alex Bennée 406b53c9ca plugins: expand the bb plugin to be thread safe and track per-cpu
While there isn't any easy way to make the inline counts thread safe
we can ensure the callback based ones are. While we are at it we can
reduce introduce a new option ("idle") to dump a report of the current
bb and insn count each time a vCPU enters the idle state.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Robert Foley <robert.foley@linaro.org>
Cc: Dave Bort <dbort@dbort.com>

Message-Id: <20200713200415.26214-8-alex.bennee@linaro.org>
2020-07-15 11:52:43 +01:00
Alex Bennée 777dddc501 tests/plugins: don't unconditionally add -Wpsabi
Not all compilers support the -Wpsabi (clang-9 in my case). To handle
this gracefully we pare back the shared build machinery so the
Makefile is relatively "standalone". We still take advantage of
config-host.mak as configure has done a bunch of probing for us but
that is it.

Fixes: bac8d222a
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20200713200415.26214-6-alex.bennee@linaro.org>
2020-07-15 11:52:43 +01:00
Alex Bennée f73e485285 docker.py: fix fetching of FROM layers
This worked on a system that was already bootstrapped because the
stage 2 images already existed even if they wouldn't be used. What we
should have pulled down was the FROM line containers first because
building on gitlab doesn't have the advantage of using our build
system to build the pre-requisite bits.

We still pull the image we want to build just in case we can use the
cached data.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200713200415.26214-4-alex.bennee@linaro.org>
2020-07-15 11:52:43 +01:00
Thomas Huth a77312e77c tests/docker: Remove the libssh workaround from the ubuntu 20.04 image
The libssh problem only exists in Ubuntu 18.04 - we can enable it
in 20.04 again.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200713185237.9419-1-thuth@redhat.com>
Message-Id: <20200713200415.26214-3-alex.bennee@linaro.org>
2020-07-15 11:52:43 +01:00
Peter Maydell 3a9163af4e Fix CVE-2020-13253
By using invalidated address, guest can do out-of-bounds accesses.
 These patches fix the issue by only allowing SD card image sizes
 power of 2, and not switching to SEND_DATA state when the address
 is invalid (out of range).
 
 This issue was found using QEMU fuzzing mode (using --enable-fuzzing,
 see docs/devel/fuzzing.txt) and reported by Alexander Bulekov.
 
 Reproducer:
   https://bugs.launchpad.net/qemu/+bug/1880822/comments/1
 
 CI jobs results:
 . https://cirrus-ci.com/build/5157142548185088
 . https://gitlab.com/philmd/qemu/-/pipelines/166381731
 . https://travis-ci.org/github/philmd/qemu/builds/707956535
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE+qvnXhKRciHc/Wuy4+MsLN6twN4FAl8NuSQACgkQ4+MsLN6t
 wN7MEg/+PER/n+CpmrC2lggQ3WJwNjvY09A4yfPfhKldjOi+25/amf/bQ2Zjmj7m
 HoiiPFu7vz+FugOfGv5YFlTS2+VNmN1UZqGqZRwY/YJJKg9am6TJ8zA4UBf4iegi
 OqNBJOPW/EYsAYdH3jUFmW15zAsRHEM6g2vZ1Z4WwVZqfYHsMB/y2khp9Fr+jGU0
 6wDeG0cdap5QVsamIll4/BoxgBa5UdtBYjzo7QBENs+abvOf56jjUqZx0+AL/Ua/
 IOpZ01mmPZJ4wJxPNT87gfEnHv0MRA7bSpJ7TAC80xVoQjeoK+V2Ohvy+rvYPaqm
 5mR0l4M+GGhglCg44wV3uwNonmltCxvTgGqZrQPsa3WnXMFoXqwGZgwl6XrYdLzV
 hVODJAu/Ivegk9AAbVrZGXg/shQtkB4gyoOaE3Qoraf1az9/XudECIo+zwocP4Ip
 Z0ny8bwQKq2QGYrCU3NWlgWi30sj6PeW5e6Jgq/2b1sUeKuUgNuuBPcRmXQ6kaz5
 vMX7qYsXAxvO7o1QlbASzdvSvOXGx+0J0CJctPnY4jAJ7qjvJTKOb0j+jwMNJy+D
 XFAgB+D0go+UvnaPJn6teIHzaD4NqWE37MaamxsMY6RWjAnoy1+OOvZIZTnq+LnH
 iLbgk2EsxlFyBd3aZ/51ukeTUxpNgu9J6iRcXB3yVNBS4vqlBDw=
 =VZsF
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/philmd-gitlab/tags/sdcard-CVE-2020-13253-pull-request' into staging

Fix CVE-2020-13253

By using invalidated address, guest can do out-of-bounds accesses.
These patches fix the issue by only allowing SD card image sizes
power of 2, and not switching to SEND_DATA state when the address
is invalid (out of range).

This issue was found using QEMU fuzzing mode (using --enable-fuzzing,
see docs/devel/fuzzing.txt) and reported by Alexander Bulekov.

Reproducer:
  https://bugs.launchpad.net/qemu/+bug/1880822/comments/1

CI jobs results:
. https://cirrus-ci.com/build/5157142548185088
. https://gitlab.com/philmd/qemu/-/pipelines/166381731
. https://travis-ci.org/github/philmd/qemu/builds/707956535

# gpg: Signature made Tue 14 Jul 2020 14:54:44 BST
# gpg:                using RSA key FAABE75E12917221DCFD6BB2E3E32C2CDEADC0DE
# gpg: Good signature from "Philippe Mathieu-Daudé (F4BUG) <f4bug@amsat.org>" [full]
# Primary key fingerprint: FAAB E75E 1291 7221 DCFD  6BB2 E3E3 2C2C DEAD C0DE

* remotes/philmd-gitlab/tags/sdcard-CVE-2020-13253-pull-request:
  hw/sd/sdcard: Do not switch to ReceivingData if address is invalid
  hw/sd/sdcard: Update coding style to make checkpatch.pl happy
  hw/sd/sdcard: Do not allow invalid SD card sizes
  hw/sd/sdcard: Simplify realize() a bit
  hw/sd/sdcard: Restrict Class 6 commands to SCSD cards
  tests/acceptance/boot_linux: Expand SD card image to power of 2
  tests/acceptance/boot_linux: Tag tests using a SD card with 'device:sd'
  docs/orangepi: Add instructions for resizing SD image to power of two
  MAINTAINERS: Cc qemu-block mailing list

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2020-07-15 09:06:55 +01:00
John Snow 2012453ddd iotests.py: use qemu.qmp type aliases
iotests.py should use the type definitions from qmp.py instead of its
own.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-3-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2020-07-14 22:22:22 +02:00
John Snow fdb87f0dc2 tests/acceptance: Don't test reboot on cubieboard
cubieboard does not have a functioning reboot, it halts and QEMU does
not exit.

vm.shutdown() is modified in a forthcoming patch that makes it less tolerant
of race conditions on shutdown; tests should consciously decide to WAIT
or to SHUTDOWN qemu.

So long as this test is attempting to reboot, the correct choice would
be to WAIT for the VM to exit. However, since that's broken, we should
SHUTDOWN instead.

SHUTDOWN is indeed what already happens when the test performs teardown,
however, if anyone fixes cubieboard reboot in the future, this test will
develop a new race condition that might be hard to debug.

Therefore: remove the reboot test and make it obvious that the VM is
still running when the test concludes, where the test teardown will do
the right thing.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-10-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2020-07-14 22:22:22 +02:00
John Snow a0690c3900 tests/acceptance: wait() instead of shutdown() where appropriate
When issuing 'reboot' to a VM with the no-reboot option, that VM will
exit. When then issuing a shutdown command, the cleanup may race.

Add calls to vm.wait() which will gracefully mark the VM as having
exited. Subsequent vm.shutdown() calls in generic tearDown code will not
race when called after completion of the call.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-9-jsnow@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2020-07-14 22:22:22 +02:00
Peter Maydell d2628b1eb7 Block layer patches:
- file-posix: Mitigate file fragmentation with extent size hints
 - Tighten qemu-img rules on missing backing format
 - qemu-img map: Don't limit block status request size
 - Fix crash with virtio-scsi and iothreads
 -----BEGIN PGP SIGNATURE-----
 
 iQJFBAABCAAvFiEE3D3rFZqa+V09dFb+fwmycsiPL9YFAl8NsgMRHGt3b2xmQHJl
 ZGhhdC5jb20ACgkQfwmycsiPL9Z0tA//eqauxD7cTEpwrtLNrRtpiBtMG64BBpxz
 QfkURzB38bMVahHlwq3Gt7Zcov8V4V7vxK66h688Z/fhw3vmqIeVe8+P6+Y5s9FL
 jil8lewHuLTa6xELeugoV7SZXH8AAh1W2fQmiR7EPiOmpSE0wf7C5IShVlX8A04E
 r0n09+61qGjRIe1hNTwTtldqQEfx6UGnxQWcQb81JUPA1lZhX3cnPg/j94Bofr+m
 v/DbVTfsmUtTMjc0PdU7n4DKTWu8OS5B/X0unF21rTtO//cYBrhAeY3ax2jbFBWi
 CIZK8HLI5m9/HFyltql1LOsd+B5TtfnXMfSdvDh2jaVUlto7wTeTnWU1fv4wxUB5
 hk7XgJo/y203ebFNHpTmW8tvLfGTP8uqCVfOEFxzjy+JHGrarlbWkwL2LMOFFAZ2
 s2WcwlfqiYGFTG4+OFdhPf9qPWKSqMr+jTdZJTse64/c6+YXWHk+pP9lfYEUOgSi
 OYwdQUY9uiZ1K13q5Tif2TbFvs+c118xdTgVhAV7VtfPnWc3c647dX7iaq8Szknc
 IT93670Iqf/PzEj+L7XUbbLIIsAcmxD0sr7QAQEt7bfiYIDRIQLiVPyzXplETFg2
 SEkvtqBovm84ct7pWQzqA6lFvr3oIFDNquR40XFGozHNnlBeNi5s7pXQnqUBLElr
 wDDuEi+z5QM=
 =DB0q
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- file-posix: Mitigate file fragmentation with extent size hints
- Tighten qemu-img rules on missing backing format
- qemu-img map: Don't limit block status request size
- Fix crash with virtio-scsi and iothreads

# gpg: Signature made Tue 14 Jul 2020 14:24:19 BST
# gpg:                using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg:                issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74  56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream:
  block: Avoid stale pointer dereference in blk_get_aio_context()
  qemu-img: Deprecate use of -b without -F
  block: Add support to warn on backing file change without format
  iotests: Specify explicit backing format where sensible
  qcow2: Deprecate use of qemu-img amend to change backing file
  block: Error if backing file fails during creation without -u
  qcow: Tolerate backing_fmt=
  vmdk: Add trivial backing_fmt support
  sheepdog: Add trivial backing_fmt support
  block: Finish deprecation of 'qemu-img convert -n -o'
  qemu-img: Flush stdout before before potential stderr messages
  file-posix: Mitigate file fragmentation with extent size hints
  iotests/059: Filter out disk size with more standard filter
  qemu-img map: Don't limit block status request size
  iotests: Simplify _filter_img_create() a bit

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2020-07-14 19:39:52 +01:00
Philippe Mathieu-Daudé 6a289a5ba3 tests/acceptance/boot_linux: Expand SD card image to power of 2
In few commits we won't allow SD card images with invalid size
(not aligned to a power of 2). Prepare the tests: add the
pow2ceil() and image_pow2ceil_expand() methods and resize the
images (expanding) of the tests using SD cards.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200713183209.26308-5-f4bug@amsat.org>
2020-07-14 15:38:22 +02:00
Philippe Mathieu-Daudé b7dcbf1395 tests/acceptance/boot_linux: Tag tests using a SD card with 'device:sd'
Avocado tags are handy to automatically select tests matching
the tags. Since these tests use a SD card, tag them.

We can run all the tests using a SD card at once with:

  $ avocado --show=app run -t u-boot tests/acceptance/
  $ AVOCADO_ALLOW_LARGE_STORAGE=ok \
    avocado --show=app \
      run -t device:sd tests/acceptance/
  Fetching asset from tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_sd
  Fetching asset from tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_bionic
  Fetching asset from tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9
   (1/3) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_sd: PASS (19.56 s)
   (2/3) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_bionic: PASS (49.97 s)
   (3/3) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9: PASS (20.06 s)
  RESULTS    : PASS 3 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
  JOB TIME   : 90.02 s

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200713183209.26308-4-f4bug@amsat.org>
2020-07-14 15:38:22 +02:00
Eric Blake d9f059aa6c qemu-img: Deprecate use of -b without -F
Creating an image that requires format probing of the backing image is
potentially unsafe (we've had several CVEs over the years based on
probes leaking information to the guest on a subsequent boot, although
these days tools like libvirt are aware of the issue enough to prevent
the worst effects).  For example, if our probing algorithm ever
changes, or if other tools like libvirt determine a different probe
result than we do, then subsequent use of that backing file under a
different format will present corrupted data to the guest.
Fortunately, the worst effects occur only when the backing image is
originally raw, and we at least prevent commit into a probed raw
backing file that would change its probed type.

Still, it is worth starting a deprecation clock so that future
qemu-img can refuse to create backing chains that would rely on
probing, to encourage clients to avoid unsafe practices.  Most
warnings are intentionally emitted from bdrv_img_create() in the block
layer, but qemu-img convert uses bdrv_create() which cannot emit its
own warning without causing spurious warnings on other code paths.  In
the end, all command-line image creation or backing file rewriting now
performs a check.

Furthermore, if we probe a backing file as non-raw, then it is safe to
explicitly record that result (rather than relying on future probes);
only where we probe a raw image do we care about further warnings to
the user when using such an image (for example, commits into a
probed-raw backing file are prevented), to help them improve their
tooling.  But whether or not we make the probe results explicit, we
still warn the user to remind them to upgrade their workflow to supply
-F always.

iotest 114 specifically wants to create an unsafe image for later
amendment rather than defaulting to our new default of recording a
probed format, so it needs an update.  While touching it, expand it to
cover all of the various warnings enabled by this patch.  iotest 301
also shows a change to qcow messages.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-11-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-14 15:24:05 +02:00
Eric Blake b66ff2c298 iotests: Specify explicit backing format where sensible
There are many existing qcow2 images that specify a backing file but
no format.  This has been the source of CVEs in the past, but has
become more prominent of a problem now that libvirt has switched to
-blockdev.  With older -drive, at least the probing was always done by
qemu (so the only risk of a changed format between successive boots of
a guest was if qemu was upgraded and probed differently).  But with
newer -blockdev, libvirt must specify a format; if libvirt guesses raw
where the image was formatted, this results in data corruption visible
to the guest; conversely, if libvirt guesses qcow2 where qemu was
using raw, this can result in potential security holes, so modern
libvirt instead refuses to use images without explicit backing format.

The change in libvirt to reject images without explicit backing format
has pointed out that a number of tools have been far too reliant on
probing in the past.  It's time to set a better example in our own
iotests of properly setting this parameter.

iotest calls to create, rebase, and convert are all impacted to some
degree.  It's a bit annoying that we are inconsistent on command line
- while all of those accept -o backing_file=...,backing_fmt=..., the
shortcuts are different: create and rebase have -b and -F, while
convert has -B but no -F.  (amend has no shortcuts, but the previous
patch just deprecated the use of amend to change backing chains).

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-9-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-14 15:18:59 +02:00
Eric Blake bc5ee6da71 qcow2: Deprecate use of qemu-img amend to change backing file
The use of 'qemu-img amend' to change qcow2 backing files is not
tested very well.  In particular, our implementation has a bug where
if a new backing file is provided without a format, then the prior
format is blindly reused, even if this results in data corruption, but
this is not caught by iotests.

There are also situations where amending other options needs access to
the original backing file (for example, on a downgrade to a v2 image,
knowing whether a v3 zero cluster must be allocated or may be left
unallocated depends on knowing whether the backing file already reads
as zero), but the command line does not have a nice way to tell us
both the backing file to use for opening the image as well as the
backing file to install after the operation is complete.

Even if we do allow changing the backing file, it is redundant with
the existing ability to change backing files via 'qemu-img rebase -u'.
It is time to deprecate this support (leaving the existing behavior
intact, even if it is buggy), and at a point in the future, require
the use of only 'qemu-img rebase' for adjusting backing chain
relations, saving 'qemu-img amend' for changes unrelated to the
backing chain.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-8-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-14 15:18:59 +02:00
Eric Blake add8200dd1 block: Error if backing file fails during creation without -u
Back in commit 6e6e55f5 (Jul 2017, v2.10), we tweaked the code to warn
if the backing file could not be opened but the user gave a size,
unless the user also passes the -u option to bypass the open of the
backing file.  As one common reason for failure to open the backing
file is when there is mismatch in the requested backing format in
relation to what the backing file actually contains, we actually want
to open the backing file and ensure that it has the right format in as
many cases as possible.  iotest 301 for qcow demonstrates how
detecting explicit format mismatch is useful to prevent the creation
of an image that would probe differently than the user requested.  Now
is the time to finally turn the warning an error, as promised.

Note that the original warning was added prior to our documentation of
an official deprecation policy (eb22aeca, also Jul 2017), and because
the warning didn't mention the word "deprecated", we never actually
remembered to document it as such.  But the warning has been around
long enough that I don't see prolonging it another two releases.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-7-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-14 15:18:59 +02:00
Eric Blake 344acbd62f qcow: Tolerate backing_fmt=
qcow has no space in the metadata to store a backing format, and there
are existing qcow images backed both by raw or by other formats
(usually qcow) images, reliant on probing to tell the difference.  On
the bright side, because we probe every time, raw files are marked as
probed and we thus forbid a commit action into the backing file where
guest-controlled contents could change the result of the probe next
time around (the iotest added here proves that).

Still, allowing the user to specify the backing format during
creation, even if we can't record it, is a good thing.  This patch
blindly allows any value that resolves to a known driver, even if the
user's request is a mismatch from what probing finds; then the next
patch will further enhance things to verify that the user's request
matches what we actually probe.  With this and the next patch in
place, we will finally be ready to deprecate the creation of images
where a backing format was not explicitly specified by the user.

Note that this is only for QemuOpts usage; there is no change to the
QAPI to allow a format through -blockdev.

Add a new iotest 301 just for qcow, to demonstrate the latest
behavior, and to make it easier to show the improvements made in the
next patch.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-6-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-14 15:18:59 +02:00
Eric Blake 25956af3fe block: Finish deprecation of 'qemu-img convert -n -o'
It's been two releases since we started warning; time to make the
combination an error as promised.  There was no iotest coverage, so
add some.

While touching the documentation, tweak another section heading for
consistent style.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-3-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-14 15:18:59 +02:00
Eric Blake 4e2f441878 qemu-img: Flush stdout before before potential stderr messages
During 'qemu-img create ... 2>&1', if --quiet is not in force, we can
end up with buffered I/O in stdout that was produced before failure,
but which appears in output after failure.  This is confusing; the fix
is to flush stdout prior to attempting anything that might produce an
error message.  Several iotests demonstrate the resulting ordering
change now that the merged outputs now reflect chronology.  (An even
better fix would be to avoid printf from within block.c altogether,
but that's much more invasive...)

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-2-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-14 15:18:59 +02:00
Kevin Wolf ffa244c84a file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).

On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.

This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.

For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.

The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.

Without an extent size hint:

    $ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
    Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
    $ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
    Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
    Run completed in 25.848 seconds.
    $ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
    Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
    Run completed in 19.616 seconds.
    $ filefrag ~/tmp/test.raw
    /home/kwolf/tmp/test.raw: 2000000 extents found
    $ time ./qemu-img map ~/tmp/test.raw
    Offset          Length          Mapped to       File
    0               0x1e8480000     0               /home/kwolf/tmp/test.raw

    real    0m1,279s
    user    0m0,043s
    sys     0m1,226s

With the new default extent size hint of 1 MB:

    $ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
    Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
    $ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
    Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
    Run completed in 11.833 seconds.
    $ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
    Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
    Run completed in 10.155 seconds.
    $ filefrag ~/tmp/test.raw
    /home/kwolf/tmp/test.raw: 178 extents found
    $ time ./qemu-img map ~/tmp/test.raw
    Offset          Length          Mapped to       File
    0               0x1e8480000     0               /home/kwolf/tmp/test.raw

    real    0m0,061s
    user    0m0,040s
    sys     0m0,014s

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-14 15:18:59 +02:00
Kevin Wolf 046e07ca55 iotests/059: Filter out disk size with more standard filter
The actual disk space used by an image can vary between filesystems and
depending on other settings like an extent size hint. Replace the one
call of "$QEMU_IMG info" and the associated one-off sed filter with the
more standard "_img_info" and the standard filter from common.filter.

Apart from turning "vmdk" into "IMGFMT" and changing the placeholder for
cid fields, this only removes the "disk size" line.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-14 15:18:59 +02:00
Max Reitz 4b196cd16d iotests: Simplify _filter_img_create() a bit
Not only is it a bit stupid to try to filter multi-line "Formatting"
output (because we only need it for a single test, which can easily be
amended to no longer need it), it is also problematic when there can be
output after a "Formatting" line that we do not want to filter as if it
were part of it.

So rename _filter_img_create to _do_filter_img_create, let it filter
only a single line, and let _filter_img_create loop over all input
lines, calling _do_filter_img_create only on those that match
/^Formatting/ (basically, what _filter_img_create_in_qmp did already).
(And fix 020 to work with that.)

Reported-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200709110205.310942-1-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-14 15:18:59 +02:00
Vladimir Sementsov-Ogievskiy df0e032b61 iotests.py: filter_testfiles(): filter SOCK_DIR too
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200701105331.121670-5-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2020-07-13 09:01:01 -05:00
Vladimir Sementsov-Ogievskiy 1f4b774a64 iotests.py: QemuIoInteractive: print output on failure
Make it simpler to debug when qemu-io fails due to wrong arguments or
environment.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200701105331.121670-3-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2020-07-13 09:01:01 -05:00
Vladimir Sementsov-Ogievskiy a1a7f56cdd iotests: QemuIoInteractive: use qemu_io_args_no_fmt
The only user (iotest 205) of QemuIoInteractive provides -f argument,
so it's a bit inefficient to use qemu_io_args, which contains -f too.
And we are going to add one more test, which wants to specify -f by
hand. Let's use qemu_io_args_no_fmt.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200701105331.121670-2-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
2020-07-13 09:01:01 -05:00
Peter Maydell 00ce6c36b3 * Some fuzzer related fixes
* Fixes / improvements for the "configure" script
 * Doc updates
 * Gitlab pipeline-status script
 -----BEGIN PGP SIGNATURE-----
 
 iQJFBAABCAAvFiEEJ7iIR+7gJQEY8+q5LtnXdP5wLbUFAl8MPAARHHRodXRoQHJl
 ZGhhdC5jb20ACgkQLtnXdP5wLbVLTQ/9H0G+Vo46hqOYpb2+psbfSi29WXAQiyNM
 ZtcloeDW/fwhKUlxtEMTuuAuDNOleGj+UxAsGaodFIEgx5XnJT2MxeqOpE3uE9G0
 9r/potzlcztYgqRHRu+H0keZNYiMY70lZUJy8NSYl+KOnndeRBmSy/GwPb/HOzH5
 2t+hlsCFkbnvm1IOcaGCVKcRNvIlUmqRt5LmqkRVrOxe6u8kfm3lgkZ/CUHzfoDq
 N5S/DuVmy0IWiP25/WnVQiV4XYLUN4jf161XfacwypPmZrK/QT9ScZhaICNuEqIY
 krDH1ZDV2M6a69e1PecJUsPYEVlsfbnmEMb3aM/xEZtrHEm2b14MpQo0CePkWEQc
 BBHd0UkkjbC9K9Gqe8tXG2YC22ZgEi6psqRprdIixNNZptJ5oWYfZ0bAeuT2Bs+7
 VOhtUmRk7ufEu9FwOleSPpHlhkTAwuMh+sYma52wx9an6rB84i6oJPGlAeSje6db
 e2+D5J6z72ZxIxc7wUBbdoL70wM0tpbppYR0QO5TblJA+WJtBupQ6mw71TzeMoOg
 2ZJlwOsnbUy0EbwgKm7huHANAhCFhnbYv70KVseX3agPJ0vSZYDo2PT8vJXTL7Sk
 BNoFn3rI70b7HW40YL8jsMvO6zXrIuofwNkpEIRoNS+gItc2a3SQa/lBosQBQAy3
 5p+qPn/kQ1U=
 =f+Fr
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/huth-gitlab/tags/pull-request-2020-07-13' into staging

* Some fuzzer related fixes
* Fixes / improvements for the "configure" script
* Doc updates
* Gitlab pipeline-status script

# gpg: Signature made Mon 13 Jul 2020 11:48:32 BST
# gpg:                using RSA key 27B88847EEE0250118F3EAB92ED9D774FE702DB5
# gpg:                issuer "thuth@redhat.com"
# gpg: Good signature from "Thomas Huth <th.huth@gmx.de>" [full]
# gpg:                 aka "Thomas Huth <thuth@redhat.com>" [full]
# gpg:                 aka "Thomas Huth <huth@tuxfamily.org>" [full]
# gpg:                 aka "Thomas Huth <th.huth@posteo.de>" [unknown]
# Primary key fingerprint: 27B8 8847 EEE0 2501 18F3  EAB9 2ED9 D774 FE70 2DB5

* remotes/huth-gitlab/tags/pull-request-2020-07-13:
  docs/system/s390x: Improve the 3270 documentation
  GitLab Gating CI: introduce pipeline-status contrib script
  disas/sh4: Add missing fallthrough annotations
  Remove the CONFIG_PRAGMA_DIAGNOSTIC_AVAILABLE switch
  docs/devel/fuzzing: Fix bugs in documentation
  tests/qtest/fuzz: Add missing spaces in description
  fuzz: add missing header for rcu_enable_atfork
  configure: do not clobber CFLAGS with --enable-fuzzing
  configure: fix malloc check

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2020-07-13 13:01:30 +01:00
Thomas Huth 2c65fefa05 tests/qtest/fuzz: Add missing spaces in description
There should be a space between "forking" and "for".

Message-Id: <20200709083719.22221-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2020-07-13 11:40:52 +02:00