From b38a04f71f5bef5cb17c29eada99364b6a017e42 Mon Sep 17 00:00:00 2001 From: Cleber Rosa Date: Fri, 9 Jul 2021 15:29:26 +0100 Subject: [PATCH 01/44] Jobs based on custom runners: documentation and configuration placeholder MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit As described in the included documentation, the "custom runner" jobs extend the GitLab CI jobs already in place. One of their primary goals of catching and preventing regressions on a wider number of host systems than the ones provided by GitLab's shared runners. This sets the stage in which other community members can add their own machine configuration documentation/scripts, and accompanying job definitions. As a general rule, those newly added contributed jobs should run as "non-gating", until their reliability is verified (AKA "allow_failure: true"). Signed-off-by: Cleber Rosa Signed-off-by: Alex Bennée Reviewed-by: Alex Bennée Reviewed-by: Thomas Huth Reviewed-by: Willian Rampazzo Reviewed-by: Wainer dos Santos Moschetta Message-Id: <20210630012619.115262-2-crosa@redhat.com> Message-Id: <20210709143005.1554-2-alex.bennee@linaro.org> --- .gitlab-ci.d/custom-runners.yml | 14 ++++++++++++++ .gitlab-ci.d/qemu-project.yml | 1 + docs/devel/ci.rst | 32 ++++++++++++++++++++++++++++++++ docs/devel/index.rst | 1 + 4 files changed, 48 insertions(+) create mode 100644 .gitlab-ci.d/custom-runners.yml create mode 100644 docs/devel/ci.rst diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml new file mode 100644 index 0000000000..a07b27384c --- /dev/null +++ b/.gitlab-ci.d/custom-runners.yml @@ -0,0 +1,14 @@ +# The CI jobs defined here require GitLab runners installed and +# registered on machines that match their operating system names, +# versions and architectures. This is in contrast to the other CI +# jobs that are intended to run on GitLab's "shared" runners. + +# Different than the default approach on "shared" runners, based on +# containers, the custom runners have no such *requirement*, as those +# jobs should be capable of running on operating systems with no +# compatible container implementation, or no support from +# gitlab-runner. To avoid problems that gitlab-runner can cause while +# reusing the GIT repository, let's enable the clone strategy, which +# guarantees a fresh repository on each job run. +variables: + GIT_STRATEGY: clone diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml index 64cb2ba1da..dde8270301 100644 --- a/.gitlab-ci.d/qemu-project.yml +++ b/.gitlab-ci.d/qemu-project.yml @@ -9,3 +9,4 @@ include: - local: '/.gitlab-ci.d/crossbuilds.yml' - local: '/.gitlab-ci.d/buildtest.yml' - local: '/.gitlab-ci.d/static_checks.yml' + - local: '/.gitlab-ci.d/custom-runners.yml' diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst new file mode 100644 index 0000000000..064ffa9988 --- /dev/null +++ b/docs/devel/ci.rst @@ -0,0 +1,32 @@ +== +CI +== + +QEMU has configurations enabled for a number of different CI services. +The most up to date information about them and their status can be +found at:: + + https://wiki.qemu.org/Testing/CI + +Jobs on Custom Runners +====================== + +Besides the jobs run under the various CI systems listed before, there +are a number additional jobs that will run before an actual merge. +These use the same GitLab CI's service/framework already used for all +other GitLab based CI jobs, but rely on additional systems, not the +ones provided by GitLab as "shared runners". + +The architecture of GitLab's CI service allows different machines to +be set up with GitLab's "agent", called gitlab-runner, which will take +care of running jobs created by events such as a push to a branch. +Here, the combination of a machine, properly configured with GitLab's +gitlab-runner, is called a "custom runner". + +The GitLab CI jobs definition for the custom runners are located under:: + + .gitlab-ci.d/custom-runners.yml + +Custom runners entail custom machines. To see a list of the machines +currently deployed in the QEMU GitLab CI and their maintainers, please +refer to the QEMU `wiki `__. diff --git a/docs/devel/index.rst b/docs/devel/index.rst index ba90badbbd..40a7e2ee6f 100644 --- a/docs/devel/index.rst +++ b/docs/devel/index.rst @@ -27,6 +27,7 @@ Contents: migration atomics stable-process + ci qtest decodetree secure-coding-practices From 159c5d177bf6f0caf1efb85b850b200ac7043c49 Mon Sep 17 00:00:00 2001 From: Cleber Rosa Date: Fri, 9 Jul 2021 15:29:27 +0100 Subject: [PATCH 02/44] Jobs based on custom runners: build environment docs and playbook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit To run basic jobs on custom runners, the environment needs to be properly set up. The most common requirement is having the right packages installed. The playbook introduced here covers the QEMU's project s390x and aarch64 machines. At the time this is being proposed, those machines have already had this playbook applied to them. Signed-off-by: Cleber Rosa Signed-off-by: Alex Bennée Tested-by: Alex Bennée Tested-by: Willian Rampazzo Reviewed-by: Wainer dos Santos Moschetta Reviewed-by: Willian Rampazzo Reviewed-by: Alex Bennée Message-Id: <20210630012619.115262-3-crosa@redhat.com> Message-Id: <20210709143005.1554-3-alex.bennee@linaro.org> --- docs/devel/ci.rst | 40 +++++++++ scripts/ci/setup/.gitignore | 2 + scripts/ci/setup/build-environment.yml | 116 +++++++++++++++++++++++++ scripts/ci/setup/inventory.template | 1 + 4 files changed, 159 insertions(+) create mode 100644 scripts/ci/setup/.gitignore create mode 100644 scripts/ci/setup/build-environment.yml create mode 100644 scripts/ci/setup/inventory.template diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst index 064ffa9988..bfedbb1025 100644 --- a/docs/devel/ci.rst +++ b/docs/devel/ci.rst @@ -30,3 +30,43 @@ The GitLab CI jobs definition for the custom runners are located under:: Custom runners entail custom machines. To see a list of the machines currently deployed in the QEMU GitLab CI and their maintainers, please refer to the QEMU `wiki `__. + +Machine Setup Howto +------------------- + +For all Linux based systems, the setup can be mostly automated by the +execution of two Ansible playbooks. Create an ``inventory`` file +under ``scripts/ci/setup``, such as this:: + + fully.qualified.domain + other.machine.hostname + +You may need to set some variables in the inventory file itself. One +very common need is to tell Ansible to use a Python 3 interpreter on +those hosts. This would look like:: + + fully.qualified.domain ansible_python_interpreter=/usr/bin/python3 + other.machine.hostname ansible_python_interpreter=/usr/bin/python3 + +Build environment +~~~~~~~~~~~~~~~~~ + +The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will +set up machines with the environment needed to perform builds and run +QEMU tests. This playbook consists on the installation of various +required packages (and a general package update while at it). It +currently covers a number of different Linux distributions, but it can +be expanded to cover other systems. + +The minimum required version of Ansible successfully tested in this +playbook is 2.8.0 (a version check is embedded within the playbook +itself). To run the playbook, execute:: + + cd scripts/ci/setup + ansible-playbook -i inventory build-environment.yml + +Please note that most of the tasks in the playbook require superuser +privileges, such as those from the ``root`` account or those obtained +by ``sudo``. If necessary, please refer to ``ansible-playbook`` +options such as ``--become``, ``--become-method``, ``--become-user`` +and ``--ask-become-pass``. diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore new file mode 100644 index 0000000000..ee088604d1 --- /dev/null +++ b/scripts/ci/setup/.gitignore @@ -0,0 +1,2 @@ +inventory + diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml new file mode 100644 index 0000000000..581c1c75d1 --- /dev/null +++ b/scripts/ci/setup/build-environment.yml @@ -0,0 +1,116 @@ +# Copyright (c) 2021 Red Hat, Inc. +# +# Author: +# Cleber Rosa +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. +# +# This is an ansible playbook file. Run it to set up systems with the +# environment needed to build QEMU. +--- +- name: Installation of basic packages to build QEMU + hosts: all + tasks: + - name: Check for suitable ansible version + delegate_to: localhost + assert: + that: + - '((ansible_version.major == 2) and (ansible_version.minor >= 8)) or (ansible_version.major >= 3)' + msg: "Unsuitable ansible version, please use version 2.8.0 or later" + + - name: Update apt cache / upgrade packages via apt + apt: + update_cache: yes + upgrade: yes + when: + - ansible_facts['distribution'] == 'Ubuntu' + + - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04 + package: + name: + # Originally from tests/docker/dockerfiles/ubuntu1804.docker + - ccache + - gcc + - gettext + - git + - glusterfs-common + - libaio-dev + - libattr1-dev + - libbrlapi-dev + - libbz2-dev + - libcacard-dev + - libcap-ng-dev + - libcurl4-gnutls-dev + - libdrm-dev + - libepoxy-dev + - libfdt-dev + - libgbm-dev + - libgtk-3-dev + - libibverbs-dev + - libiscsi-dev + - libjemalloc-dev + - libjpeg-turbo8-dev + - liblzo2-dev + - libncurses5-dev + - libncursesw5-dev + - libnfs-dev + - libnss3-dev + - libnuma-dev + - libpixman-1-dev + - librados-dev + - librbd-dev + - librdmacm-dev + - libsasl2-dev + - libsdl2-dev + - libseccomp-dev + - libsnappy-dev + - libspice-protocol-dev + - libssh-dev + - libusb-1.0-0-dev + - libusbredirhost-dev + - libvdeplug-dev + - libvte-2.91-dev + - libzstd-dev + - make + - python3-yaml + - python3-sphinx + - python3-sphinx-rtd-theme + - ninja-build + - sparse + - xfslibs-dev + state: present + when: + - ansible_facts['distribution'] == 'Ubuntu' + + - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x + package: + name: + - libspice-server-dev + - libxen-dev + state: present + when: + - ansible_facts['distribution'] == 'Ubuntu' + - ansible_facts['architecture'] != 's390x' + + - name: Install basic packages to build QEMU on Ubuntu 18.04 + package: + name: + # Originally from tests/docker/dockerfiles/ubuntu1804.docker + - clang + when: + - ansible_facts['distribution'] == 'Ubuntu' + - ansible_facts['distribution_version'] == '18.04' + + - name: Install basic packages to build QEMU on Ubuntu 20.04 + package: + name: + # Originally from tests/docker/dockerfiles/ubuntu2004.docker + - clang-10 + - genisoimage + - liblttng-ust-dev + - libslirp-dev + - netcat-openbsd + when: + - ansible_facts['distribution'] == 'Ubuntu' + - ansible_facts['distribution_version'] == '20.04' diff --git a/scripts/ci/setup/inventory.template b/scripts/ci/setup/inventory.template new file mode 100644 index 0000000000..2fbb50c4a8 --- /dev/null +++ b/scripts/ci/setup/inventory.template @@ -0,0 +1 @@ +localhost From 40de78c284b14a54fbdde78d588ddb6d766f2a5f Mon Sep 17 00:00:00 2001 From: Cleber Rosa Date: Fri, 9 Jul 2021 15:29:28 +0100 Subject: [PATCH 03/44] Jobs based on custom runners: docs and gitlab-runner setup playbook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit To have the jobs dispatched to custom runners, gitlab-runner must be installed, active as a service and properly configured. The variables file and playbook introduced here should help with those steps. The playbook introduced here covers the Linux distributions and has been primarily tested on OS/machines that the QEMU project has available to act as runners, namely: * Ubuntu 20.04 on aarch64 * Ubuntu 18.04 on s390x But, it should work on all other Linux distributions. Earlier versions were tested on FreeBSD too, so chances of success are high. Signed-off-by: Cleber Rosa Signed-off-by: Alex Bennée Tested-by: Willian Rampazzo Tested-by: Alex Bennée Reviewed-by: Willian Rampazzo Reviewed-by: Wainer dos Santos Moschetta Reviewed-by: Alex Bennée Message-Id: <20210630012619.115262-4-crosa@redhat.com> Message-Id: <20210709143005.1554-4-alex.bennee@linaro.org> --- docs/devel/ci.rst | 55 +++++++++++++++++++++++ scripts/ci/setup/.gitignore | 2 +- scripts/ci/setup/gitlab-runner.yml | 71 ++++++++++++++++++++++++++++++ scripts/ci/setup/vars.yml.template | 12 +++++ 4 files changed, 139 insertions(+), 1 deletion(-) create mode 100644 scripts/ci/setup/gitlab-runner.yml create mode 100644 scripts/ci/setup/vars.yml.template diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst index bfedbb1025..b3bf3ef615 100644 --- a/docs/devel/ci.rst +++ b/docs/devel/ci.rst @@ -70,3 +70,58 @@ privileges, such as those from the ``root`` account or those obtained by ``sudo``. If necessary, please refer to ``ansible-playbook`` options such as ``--become``, ``--become-method``, ``--become-user`` and ``--ask-become-pass``. + +gitlab-runner setup and registration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The gitlab-runner agent needs to be installed on each machine that +will run jobs. The association between a machine and a GitLab project +happens with a registration token. To find the registration token for +your repository/project, navigate on GitLab's web UI to: + + * Settings (the gears-like icon at the bottom of the left hand side + vertical toolbar), then + * CI/CD, then + * Runners, and click on the "Expand" button, then + * Under "Set up a specific Runner manually", look for the value under + "And this registration token:" + +Copy the ``scripts/ci/setup/vars.yml.template`` file to +``scripts/ci/setup/vars.yml``. Then, set the +``gitlab_runner_registration_token`` variable to the value obtained +earlier. + +To run the playbook, execute:: + + cd scripts/ci/setup + ansible-playbook -i inventory gitlab-runner.yml + +Following the registration, it's necessary to configure the runner tags, +and optionally other configurations on the GitLab UI. Navigate to: + + * Settings (the gears like icon), then + * CI/CD, then + * Runners, and click on the "Expand" button, then + * "Runners activated for this project", then + * Click on the "Edit" icon (next to the "Lock" Icon) + +Tags are very important as they are used to route specific jobs to +specific types of runners, so it's a good idea to double check that +the automatically created tags are consistent with the OS and +architecture. For instance, an Ubuntu 20.04 aarch64 system should +have tags set as:: + + ubuntu_20.04,aarch64 + +Because the job definition at ``.gitlab-ci.d/custom-runners.yml`` +would contain:: + + ubuntu-20.04-aarch64-all: + tags: + - ubuntu_20.04 + - aarch64 + +It's also recommended to: + + * increase the "Maximum job timeout" to something like ``2h`` + * give it a better Description diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore index ee088604d1..f4a6183f1f 100644 --- a/scripts/ci/setup/.gitignore +++ b/scripts/ci/setup/.gitignore @@ -1,2 +1,2 @@ inventory - +vars.yml diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml new file mode 100644 index 0000000000..1127db516f --- /dev/null +++ b/scripts/ci/setup/gitlab-runner.yml @@ -0,0 +1,71 @@ +# Copyright (c) 2021 Red Hat, Inc. +# +# Author: +# Cleber Rosa +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. +# +# This is an ansible playbook file. Run it to set up systems with the +# gitlab-runner agent. +--- +- name: Installation of gitlab-runner + hosts: all + vars_files: + - vars.yml + tasks: + - debug: + msg: 'Checking for a valid GitLab registration token' + failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'" + + - name: Create a group for the gitlab-runner service + group: + name: gitlab-runner + + - name: Create a user for the gitlab-runner service + user: + user: gitlab-runner + group: gitlab-runner + comment: GitLab Runner + home: /home/gitlab-runner + shell: /bin/bash + + - name: Remove the .bash_logout file when on Ubuntu systems + file: + path: /home/gitlab-runner/.bash_logout + state: absent + when: "ansible_facts['distribution'] == 'Ubuntu'" + + - name: Set the Operating System for gitlab-runner + set_fact: + gitlab_runner_os: "{{ ansible_facts[\"system\"]|lower }}" + - debug: + msg: gitlab-runner OS is {{ gitlab_runner_os }} + + - name: Set the architecture for gitlab-runner + set_fact: + gitlab_runner_arch: "{{ ansible_to_gitlab_arch[ansible_facts[\"architecture\"]] }}" + - debug: + msg: gitlab-runner arch is {{ gitlab_runner_arch }} + + - name: Download the matching gitlab-runner + get_url: + dest: /usr/local/bin/gitlab-runner + url: "https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version }}/binaries/gitlab-runner-{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}" + owner: gitlab-runner + group: gitlab-runner + mode: u=rwx,g=rwx,o=rx + + - name: Register the gitlab-runner + command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'" + + - name: Install the gitlab-runner service using its own functionality + command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner + register: gitlab_runner_install_service_result + failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr" + + - name: Enable the gitlab-runner service + service: + name: gitlab-runner + state: started + enabled: yes diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template new file mode 100644 index 0000000000..e48089761f --- /dev/null +++ b/scripts/ci/setup/vars.yml.template @@ -0,0 +1,12 @@ +# The version of the gitlab-runner to use +gitlab_runner_version: 13.12.0 +# The URL of the gitlab server to use, usually https://gitlab.com unless you're +# using a private GitLab instance +gitlab_runner_server_url: https://gitlab.com +# A mapping of the ansible to gitlab architecture nomenclature +ansible_to_gitlab_arch: + x86_64: amd64 + aarch64: arm64 + s390x: s390x +# A unique token made available by GitLab to your project for registering runners +gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN From 4799c2102333cee5ddc51051c035b1c6ea83c7f5 Mon Sep 17 00:00:00 2001 From: Cleber Rosa Date: Fri, 9 Jul 2021 15:29:29 +0100 Subject: [PATCH 04/44] Jobs based on custom runners: add job definitions for QEMU's machines MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The QEMU project has two machines (aarch64 and s390x) that can be used for jobs that do build and run tests. This introduces those jobs, which are a mapping of custom scripts used for the same purpose. Signed-off-by: Cleber Rosa Signed-off-by: Alex Bennée Reviewed-by: Willian Rampazzo Reviewed-by: Wainer dos Santos Moschetta Message-Id: <20210630012619.115262-5-crosa@redhat.com> Message-Id: <20210709143005.1554-5-alex.bennee@linaro.org> --- .gitlab-ci.d/custom-runners.yml | 208 ++++++++++++++++++++++++++++++++ 1 file changed, 208 insertions(+) diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml index a07b27384c..061d3cdfed 100644 --- a/.gitlab-ci.d/custom-runners.yml +++ b/.gitlab-ci.d/custom-runners.yml @@ -12,3 +12,211 @@ # guarantees a fresh repository on each job run. variables: GIT_STRATEGY: clone + +# All ubuntu-18.04 jobs should run successfully in an environment +# setup by the scripts/ci/setup/build-environment.yml task +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04" +ubuntu-18.04-s390x-all-linux-static: + allow_failure: true + needs: [] + stage: build + tags: + - ubuntu_18.04 + - s390x + rules: + - if: '$CI_COMMIT_BRANCH =~ /^staging/' + script: + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763 + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages + - mkdir build + - cd build + - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + - make --output-sync -j`nproc` check-tcg V=1 + +ubuntu-18.04-s390x-all: + allow_failure: true + needs: [] + stage: build + tags: + - ubuntu_18.04 + - s390x + rules: + - if: '$CI_COMMIT_BRANCH =~ /^staging/' + script: + - mkdir build + - cd build + - ../configure --disable-libssh + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +ubuntu-18.04-s390x-alldbg: + allow_failure: true + needs: [] + stage: build + tags: + - ubuntu_18.04 + - s390x + rules: + - if: '$CI_COMMIT_BRANCH =~ /^staging/' + script: + - mkdir build + - cd build + - ../configure --enable-debug --disable-libssh + - make clean + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +ubuntu-18.04-s390x-clang: + allow_failure: true + needs: [] + stage: build + tags: + - ubuntu_18.04 + - s390x + rules: + - if: '$CI_COMMIT_BRANCH =~ /^staging/' + when: manual + script: + - mkdir build + - cd build + - ../configure --disable-libssh --cc=clang --cxx=clang++ --enable-sanitizers + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +ubuntu-18.04-s390x-tci: + allow_failure: true + needs: [] + stage: build + tags: + - ubuntu_18.04 + - s390x + rules: + - if: '$CI_COMMIT_BRANCH =~ /^staging/' + script: + - mkdir build + - cd build + - ../configure --disable-libssh --enable-tcg-interpreter + - make --output-sync -j`nproc` + +ubuntu-18.04-s390x-notcg: + allow_failure: true + needs: [] + stage: build + tags: + - ubuntu_18.04 + - s390x + rules: + - if: '$CI_COMMIT_BRANCH =~ /^staging/' + when: manual + script: + - mkdir build + - cd build + - ../configure --disable-libssh --disable-tcg + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +# All ubuntu-20.04 jobs should run successfully in an environment +# setup by the scripts/ci/setup/qemu/build-environment.yml task +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04" +ubuntu-20.04-aarch64-all-linux-static: + allow_failure: true + needs: [] + stage: build + tags: + - ubuntu_20.04 + - aarch64 + rules: + - if: '$CI_COMMIT_BRANCH =~ /^staging/' + script: + # --disable-libssh is needed because of https://bugs.launchpad.net/qemu/+bug/1838763 + # --disable-glusterfs is needed because there's no static version of those libs in distro supplied packages + - mkdir build + - cd build + - ../configure --enable-debug --static --disable-system --disable-glusterfs --disable-libssh + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + - make --output-sync -j`nproc` check-tcg V=1 + +ubuntu-20.04-aarch64-all: + allow_failure: true + needs: [] + stage: build + tags: + - ubuntu_20.04 + - aarch64 + rules: + - if: '$CI_COMMIT_BRANCH =~ /^staging/' + script: + - mkdir build + - cd build + - ../configure --disable-libssh + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +ubuntu-20.04-aarch64-alldbg: + allow_failure: true + needs: [] + stage: build + tags: + - ubuntu_20.04 + - aarch64 + rules: + - if: '$CI_COMMIT_BRANCH =~ /^staging/' + script: + - mkdir build + - cd build + - ../configure --enable-debug --disable-libssh + - make clean + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +ubuntu-20.04-aarch64-clang: + allow_failure: true + needs: [] + stage: build + tags: + - ubuntu_20.04 + - aarch64 + rules: + - if: '$CI_COMMIT_BRANCH =~ /^staging/' + when: manual + script: + - mkdir build + - cd build + - ../configure --disable-libssh --cc=clang-10 --cxx=clang++-10 --enable-sanitizers + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 + +ubuntu-20.04-aarch64-tci: + allow_failure: true + needs: [] + stage: build + tags: + - ubuntu_20.04 + - aarch64 + rules: + - if: '$CI_COMMIT_BRANCH =~ /^staging/' + script: + - mkdir build + - cd build + - ../configure --disable-libssh --enable-tcg-interpreter + - make --output-sync -j`nproc` + +ubuntu-20.04-aarch64-notcg: + allow_failure: true + needs: [] + stage: build + tags: + - ubuntu_20.04 + - aarch64 + rules: + - if: '$CI_COMMIT_BRANCH =~ /^staging/' + when: manual + script: + - mkdir build + - cd build + - ../configure --disable-libssh --disable-tcg + - make --output-sync -j`nproc` + - make --output-sync -j`nproc` check V=1 From 0f1ea9c7a602f0349137672f14c4c2a9671ef28e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 9 Jul 2021 15:29:30 +0100 Subject: [PATCH 05/44] tests/tcg: also disable the signals test for plugins MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This will be more important when plugins is enabled by default. Fixes: eba61056e4 ("tests/tcg: generalise the disabling of the signals test") Signed-off-by: Alex Bennée Reviewed-by: Richard Henderson Message-Id: <20210709143005.1554-6-alex.bennee@linaro.org> --- tests/tcg/i386/Makefile.target | 3 +++ tests/tcg/multiarch/Makefile.target | 2 ++ 2 files changed, 5 insertions(+) diff --git a/tests/tcg/i386/Makefile.target b/tests/tcg/i386/Makefile.target index f7efaab918..b0a2128980 100644 --- a/tests/tcg/i386/Makefile.target +++ b/tests/tcg/i386/Makefile.target @@ -65,6 +65,9 @@ run-plugin-%-with-libinsn.so: -d plugin -D $*-with-libinsn.so.pout $*, \ "$* (inline) on $(TARGET_NAME)") +run-plugin-signals-with-libinsn.so: + $(call skip-test, $<, "BROKEN awaiting sigframe clean-ups and vdso support") + # Update TESTS I386_TESTS:=$(filter-out $(SKIP_I386_TESTS), $(ALL_X86_TESTS)) TESTS=$(MULTIARCH_TESTS) $(I386_TESTS) diff --git a/tests/tcg/multiarch/Makefile.target b/tests/tcg/multiarch/Makefile.target index d57a115873..85a6fb7a2e 100644 --- a/tests/tcg/multiarch/Makefile.target +++ b/tests/tcg/multiarch/Makefile.target @@ -37,6 +37,8 @@ signals: LDFLAGS+=-lrt -lpthread run-signals: signals $(call skip-test, $<, "BROKEN awaiting sigframe clean-ups and vdso support") +run-plugin-signals-with-%: + $(call skip-test, $<, "BROKEN awaiting sigframe clean-ups and vdso support") # We define the runner for test-mmap after the individual # architectures have defined their supported pages sizes. If no From 3cfafd317c97558b66d93b6f338b95c695ad8d39 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Mon, 12 Jul 2021 17:18:10 +0200 Subject: [PATCH 06/44] meson: fix condition for io_uring stubs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CONFIG_LINUX_IO_URING is not included in config-host.mak and therefore is not usable in "when" clauses. Check the availability of the library, which matches the condition for the non-stubbed version block/io_uring.c. At this point, the difference between libraries that have config-host.mak entries and those that do not is quite confusing. The remaining ~dozen should be converted in 6.2. Signed-off-by: Paolo Bonzini Reviewed-by: Philippe Mathieu-Daudé Message-Id: <20210712151810.508249-1-pbonzini@redhat.com> Signed-off-by: Alex Bennée --- stubs/meson.build | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/stubs/meson.build b/stubs/meson.build index 2e79ff9f4d..d3fa8646b3 100644 --- a/stubs/meson.build +++ b/stubs/meson.build @@ -15,7 +15,9 @@ stub_ss.add(files('fdset.c')) stub_ss.add(files('fw_cfg.c')) stub_ss.add(files('gdbstub.c')) stub_ss.add(files('get-vm-name.c')) -stub_ss.add(when: 'CONFIG_LINUX_IO_URING', if_true: files('io_uring.c')) +if linux_io_uring.found() + stub_ss.add(files('io_uring.c')) +endif stub_ss.add(files('iothread-lock.c')) stub_ss.add(files('isa-bus.c')) stub_ss.add(files('is-daemonized.c')) From a1b176f9046a94ae963099a055cde2323dc7c012 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Mon, 12 Jul 2021 14:22:08 +0200 Subject: [PATCH 07/44] disable modular TCG on Darwin MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Accelerator modularity does not work on Darwin: ld: illegal thread local variable reference to regular symbol _current_cpu for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) Fix by avoiding modular TCG builds. Signed-off-by: Paolo Bonzini Message-Id: <20210712122208.456264-1-pbonzini@redhat.com> [AJB: manually merged typo fix] Signed-off-by: Alex Bennée --- meson.build | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/meson.build b/meson.build index b0e2b9a8a0..512ee2e926 100644 --- a/meson.build +++ b/meson.build @@ -92,7 +92,11 @@ if cpu in ['x86', 'x86_64'] } endif -modular_tcg = ['i386-softmmu', 'x86_64-softmmu'] +modular_tcg = [] +# Darwin does not support references to thread-local variables in modules +if targetos != 'darwin' + modular_tcg = ['i386-softmmu', 'x86_64-softmmu'] +endif edk2_targets = [ 'arm-softmmu', 'aarch64-softmmu', 'i386-softmmu', 'x86_64-softmmu' ] install_edk2_blobs = false From 8f4aea712ffc4f2c20c293dac26df8688f497a9f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:31 +0100 Subject: [PATCH 08/44] build: validate that system capstone works before using it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Some versions of capstone have shipped a broken pkg-config file which puts the -I path without the trailing '/capstone' suffix. This breaks the ability to "#include ". Upstream and most distros have fixed this, but a few stragglers remain, notably FreeBSD. Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Thomas Huth Reviewed-by: Willian Rampazzo Reviewed-by: Richard Henderson Message-Id: <20210625172211.451010-2-berrange@redhat.com> Message-Id: <20210709143005.1554-7-alex.bennee@linaro.org> --- meson.build | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/meson.build b/meson.build index 512ee2e926..89c09ed7ea 100644 --- a/meson.build +++ b/meson.build @@ -1657,6 +1657,19 @@ if capstone_opt in ['enabled', 'auto', 'system'] kwargs: static_kwargs, method: 'pkg-config', required: capstone_opt == 'system' or capstone_opt == 'enabled' and not have_internal) + + # Some versions of capstone have broken pkg-config file + # that reports a wrong -I path, causing the #include to + # fail later. If the system has such a broken version + # do not use it. + if capstone.found() and not cc.compiles('#include ', + dependencies: [capstone]) + capstone = not_found + if capstone_opt == 'system' + error('system capstone requested, it does not appear to work') + endif + endif + if capstone.found() capstone_opt = 'system' elif have_internal From 0e103a65ba1c56871cad3ab4be3aee1d984d65f3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:32 +0100 Subject: [PATCH 09/44] gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This adds support for running 4 jobs via Cirrus CI runners: * FreeBSD 12 * FreeBSD 13 * macOS 11 with default XCode * macOS 11 with latest XCode The gitlab job uses a container published by the libvirt-ci project (https://gitlab.com/libvirt/libvirt-ci) that contains the 'cirrus-run' command. This accepts a short yaml file that describes a single Cirrus CI job, runs it using the Cirrus CI REST API, and reports any output to the console. In this way Cirrus CI is effectively working as an indirect custom runner for GitLab CI pipelines. The key benefit is that Cirrus CI job results affect the GitLab CI pipeline result and so the user only has look at one CI dashboard. [AJB: remove $TEMPORARILY_DISABLED condition, s/py37/py38/] Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Message-Id: <20210625172211.451010-3-berrange@redhat.com> Message-Id: <20210709143005.1554-8-alex.bennee@linaro.org> --- .gitlab-ci.d/cirrus.yml | 102 ++++++++++++++++++++++++++++ .gitlab-ci.d/cirrus/README.rst | 54 +++++++++++++++ .gitlab-ci.d/cirrus/build.yml | 35 ++++++++++ .gitlab-ci.d/cirrus/freebsd-12.vars | 13 ++++ .gitlab-ci.d/cirrus/freebsd-13.vars | 13 ++++ .gitlab-ci.d/cirrus/macos-11.vars | 15 ++++ .gitlab-ci.d/qemu-project.yml | 1 + 7 files changed, 233 insertions(+) create mode 100644 .gitlab-ci.d/cirrus.yml create mode 100644 .gitlab-ci.d/cirrus/README.rst create mode 100644 .gitlab-ci.d/cirrus/build.yml create mode 100644 .gitlab-ci.d/cirrus/freebsd-12.vars create mode 100644 .gitlab-ci.d/cirrus/freebsd-13.vars create mode 100644 .gitlab-ci.d/cirrus/macos-11.vars diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml new file mode 100644 index 0000000000..60b13ed83f --- /dev/null +++ b/.gitlab-ci.d/cirrus.yml @@ -0,0 +1,102 @@ +# Jobs that we delegate to Cirrus CI because they require an operating +# system other than Linux. These jobs will only run if the required +# setup has been performed on the GitLab account. +# +# The Cirrus CI configuration is generated by replacing target-specific +# variables in a generic template: some of these variables are provided +# when the GitLab CI job is defined, others are taken from a shell +# snippet generated using lcitool. +# +# Note that the $PATH environment variable has to be treated with +# special care, because we can't just override it at the GitLab CI job +# definition level or we risk breaking it completely. +.cirrus_build_job: + stage: build + image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master + needs: [] + allow_failure: true + script: + - source .gitlab-ci.d/cirrus/$NAME.vars + - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g" + -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g" + -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g" + -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g" + -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g" + -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g" + -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g" + -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g" + -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g" + -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g" + -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g" + -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g" + -e "s|[@]PKGS@|$PKGS|g" + -e "s|[@]MAKE@|$MAKE|g" + -e "s|[@]PYTHON@|$PYTHON|g" + -e "s|[@]PIP3@|$PIP3|g" + -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g" + -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g" + -e "s|[@]TEST_TARGETSS@|$TEST_TARGETSS|g" + <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml + - cat .gitlab-ci.d/cirrus/$NAME.yml + - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml + rules: + - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN" + +x64-freebsd-12-build: + extends: .cirrus_build_job + variables: + NAME: freebsd-12 + CIRRUS_VM_INSTANCE_TYPE: freebsd_instance + CIRRUS_VM_IMAGE_SELECTOR: image_family + CIRRUS_VM_IMAGE_NAME: freebsd-12-2 + CIRRUS_VM_CPUS: 8 + CIRRUS_VM_RAM: 8G + UPDATE_COMMAND: pkg update + INSTALL_COMMAND: pkg install -y + # TODO: Enable gnutls again once FreeBSD's libtasn1 got fixed + # See: https://gitlab.com/gnutls/libtasn1/-/merge_requests/71 + CONFIGURE_ARGS: --disable-gnutls + TEST_TARGETS: check + +x64-freebsd-13-build: + extends: .cirrus_build_job + variables: + NAME: freebsd-13 + CIRRUS_VM_INSTANCE_TYPE: freebsd_instance + CIRRUS_VM_IMAGE_SELECTOR: image_family + CIRRUS_VM_IMAGE_NAME: freebsd-13-0 + CIRRUS_VM_CPUS: 8 + CIRRUS_VM_RAM: 8G + UPDATE_COMMAND: pkg update + INSTALL_COMMAND: pkg install -y + TEST_TARGETS: check + +x64-macos-11-base-build: + extends: .cirrus_build_job + variables: + NAME: macos-11 + CIRRUS_VM_INSTANCE_TYPE: osx_instance + CIRRUS_VM_IMAGE_SELECTOR: image + CIRRUS_VM_IMAGE_NAME: big-sur-base + CIRRUS_VM_CPUS: 12 + CIRRUS_VM_RAM: 24G + UPDATE_COMMAND: brew update + INSTALL_COMMAND: brew install + PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin + PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig + TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64 + +x64-macos-11-xcode-build: + extends: .cirrus_build_job + variables: + NAME: macos-11 + CIRRUS_VM_INSTANCE_TYPE: osx_instance + CIRRUS_VM_IMAGE_SELECTOR: image + CIRRUS_VM_IMAGE_NAME: big-sur-xcode + CIRRUS_VM_CPUS: 12 + CIRRUS_VM_RAM: 24G + UPDATE_COMMAND: brew update + INSTALL_COMMAND: brew install + PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin + PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig + TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64 diff --git a/.gitlab-ci.d/cirrus/README.rst b/.gitlab-ci.d/cirrus/README.rst new file mode 100644 index 0000000000..657b0706d7 --- /dev/null +++ b/.gitlab-ci.d/cirrus/README.rst @@ -0,0 +1,54 @@ +Cirrus CI integration +===================== + +GitLab CI shared runners only provide a docker environment running on Linux. +While it is possible to provide private runners for non-Linux platforms this +is not something most contributors/maintainers will wish to do. + +To work around this limitation, we take advantage of `Cirrus CI`_'s free +offering: more specifically, we use the `cirrus-run`_ script to trigger Cirrus +CI jobs from GitLab CI jobs so that Cirrus CI job output is integrated into +the main GitLab CI pipeline dashboard. + +There is, however, some one-time setup required. If you want FreeBSD and macOS +builds to happen when you push to your GitLab repository, you need to + +* set up a GitHub repository for the project, eg. ``yourusername/qemu``. + This repository needs to exist for cirrus-run to work, but it doesn't need to + be kept up to date, so you can create it and then forget about it; + +* enable the `Cirrus CI GitHub app`_ for your GitHub account; + +* sign up for Cirrus CI. It's enough to log into the website using your GitHub + account; + +* grab an API token from the `Cirrus CI settings`_ page; + +* it may be necessary to push an empty ``.cirrus.yml`` file to your github fork + for Cirrus CI to properly recognize the project. You can check whether + Cirrus CI knows about your project by navigating to: + + ``https://cirrus-ci.com/yourusername/qemu`` + +* in the *CI/CD / Variables* section of the settings page for your GitLab + repository, create two new variables: + + * ``CIRRUS_GITHUB_REPO``, containing the name of the GitHub repository + created earlier, eg. ``yourusername/qemu``; + + * ``CIRRUS_API_TOKEN``, containing the Cirrus CI API token generated earlier. + This variable **must** be marked as *Masked*, because anyone with knowledge + of it can impersonate you as far as Cirrus CI is concerned. + + Neither of these variables should be marked as *Protected*, because in + general you'll want to be able to trigger Cirrus CI builds from non-protected + branches. + +Once this one-time setup is complete, you can just keep pushing to your GitLab +repository as usual and you'll automatically get the additional CI coverage. + + +.. _Cirrus CI GitHub app: https://github.com/marketplace/cirrus-ci +.. _Cirrus CI settings: https://cirrus-ci.com/settings/profile/ +.. _Cirrus CI: https://cirrus-ci.com/ +.. _cirrus-run: https://github.com/sio/cirrus-run/ diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml new file mode 100644 index 0000000000..857bdc5536 --- /dev/null +++ b/.gitlab-ci.d/cirrus/build.yml @@ -0,0 +1,35 @@ +@CIRRUS_VM_INSTANCE_TYPE@: + @CIRRUS_VM_IMAGE_SELECTOR@: @CIRRUS_VM_IMAGE_NAME@ + cpu: @CIRRUS_VM_CPUS@ + memory: @CIRRUS_VM_RAM@ + +env: + CIRRUS_CLONE_DEPTH: 1 + CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@" + CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@" + CI_COMMIT_SHA: "@CI_COMMIT_SHA@" + PATH: "@PATH@" + PKG_CONFIG_PATH: "@PKG_CONFIG_PATH@" + PYTHON: "@PYTHON@" + MAKE: "@MAKE@" + CONFIGURE_ARGS: "@CONFIGURE_ARGS@" + +build_task: + install_script: + - @UPDATE_COMMAND@ + - @INSTALL_COMMAND@ @PKGS@ + - if test -n "@PYPI_PKGS@" ; then @PIP3@ install @PYPI_PKGS@ ; fi + clone_script: + - git clone --depth 100 "$CI_REPOSITORY_URL" . + - git fetch origin "$CI_COMMIT_REF_NAME" + - git reset --hard "$CI_COMMIT_SHA" + build_script: + - mkdir build + - cd build + - ../configure --enable-werror $CONFIGURE_ARGS + || { cat config.log meson-logs/meson-log.txt; exit 1; } + - $MAKE -j$(sysctl -n hw.ncpu) + - for TARGET in $TEST_TARGETS ; + do + $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ; + done diff --git a/.gitlab-ci.d/cirrus/freebsd-12.vars b/.gitlab-ci.d/cirrus/freebsd-12.vars new file mode 100644 index 0000000000..2099b21354 --- /dev/null +++ b/.gitlab-ci.d/cirrus/freebsd-12.vars @@ -0,0 +1,13 @@ +# THIS FILE WAS AUTO-GENERATED +# +# $ lcitool variables freebsd-12 qemu +# +# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1 + +PACKAGING_COMMAND='pkg' +CCACHE='/usr/local/bin/ccache' +MAKE='/usr/local/bin/gmake' +NINJA='/usr/local/bin/ninja' +PYTHON='/usr/local/bin/python3' +PIP3='/usr/local/bin/pip-3.8' +PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 libxml2 llvm lttng-ust lzo2 meson ncurses nettle ninja opencv p5-Test-Harness perl5 pixman pkgconf png py38-numpy py38-pillow py38-pip py38-sphinx py38-sphinx_rtd_theme py38-virtualenv py38-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd' diff --git a/.gitlab-ci.d/cirrus/freebsd-13.vars b/.gitlab-ci.d/cirrus/freebsd-13.vars new file mode 100644 index 0000000000..323fe806d5 --- /dev/null +++ b/.gitlab-ci.d/cirrus/freebsd-13.vars @@ -0,0 +1,13 @@ +# THIS FILE WAS AUTO-GENERATED +# +# $ lcitool variables freebsd-13 qemu +# +# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1 + +PACKAGING_COMMAND='pkg' +CCACHE='/usr/local/bin/ccache' +MAKE='/usr/local/bin/gmake' +NINJA='/usr/local/bin/ninja' +PYTHON='/usr/local/bin/python3' +PIP3='/usr/local/bin/pip-3.8' +PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 libxml2 llvm lttng-ust lzo2 meson ncurses nettle ninja opencv p5-Test-Harness perl5 pixman pkgconf png py38-numpy py38-pillow py38-pip py38-sphinx py38-sphinx_rtd_theme py38-virtualenv py38-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd' diff --git a/.gitlab-ci.d/cirrus/macos-11.vars b/.gitlab-ci.d/cirrus/macos-11.vars new file mode 100644 index 0000000000..cbec8a44a3 --- /dev/null +++ b/.gitlab-ci.d/cirrus/macos-11.vars @@ -0,0 +1,15 @@ +# THIS FILE WAS AUTO-GENERATED +# +# $ lcitool variables macos-11 qemu +# +# https://gitlab.com/libvirt/libvirt-ci/-/commit/c7e275ab27ac0dcd09da290817b9adeea1fd1eb1 + +PACKAGING_COMMAND='brew' +CCACHE='/usr/local/bin/ccache' +MAKE='/usr/local/bin/gmake' +NINJA='/usr/local/bin/ninja' +PYTHON='/usr/local/bin/python3' +PIP3='/usr/local/bin/pip3' +PKGS='bash bc bzip2 capstone ccache cpanminus ctags curl dbus diffutils gcovr gettext git glib gnu-sed gnutls gtk+3 jemalloc jpeg-turbo libepoxy libffi libgcrypt libiscsi libnfs libpng libslirp libssh libtasn1 libusb libxml2 llvm lzo make meson ncurses nettle ninja perl pixman pkg-config python3 rpm2cpio sdl2 sdl2_image snappy sparse spice-protocol tesseract texinfo usbredir vde vte3 zlib zstd' +PYPI_PKGS='PyYAML numpy pillow sphinx sphinx-rtd-theme virtualenv' +CPAN_PKGS='Test::Harness' diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml index dde8270301..b3d79bc429 100644 --- a/.gitlab-ci.d/qemu-project.yml +++ b/.gitlab-ci.d/qemu-project.yml @@ -10,3 +10,4 @@ include: - local: '/.gitlab-ci.d/buildtest.yml' - local: '/.gitlab-ci.d/static_checks.yml' - local: '/.gitlab-ci.d/custom-runners.yml' + - local: '/.gitlab-ci.d/cirrus.yml' From b92da9acb14bd927fbe4c4fd200cf528dc03ecfb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:33 +0100 Subject: [PATCH 10/44] cirrus: delete FreeBSD and macOS jobs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The builds for these two platforms can now be performed from GitLab CI using cirrus-run. Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Thomas Huth Reviewed-by: Willian Rampazzo Reviewed-by: Wainer dos Santos Moschetta Message-Id: <20210625172211.451010-4-berrange@redhat.com> Message-Id: <20210709143005.1554-9-alex.bennee@linaro.org> --- .cirrus.yml | 55 ----------------------------------------------------- 1 file changed, 55 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index f4bf49b704..02c43a074a 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -1,61 +1,6 @@ env: CIRRUS_CLONE_DEPTH: 1 -freebsd_12_task: - freebsd_instance: - image_family: freebsd-12-2 - cpu: 8 - memory: 8G - install_script: - - ASSUME_ALWAYS_YES=yes pkg bootstrap -f ; - - pkg install -y bash curl cyrus-sasl git glib gmake gnutls gsed - nettle perl5 pixman pkgconf png usbredir ninja - script: - - mkdir build - - cd build - # TODO: Enable gnutls again once FreeBSD's libtasn1 got fixed - # See: https://gitlab.com/gnutls/libtasn1/-/merge_requests/71 - - ../configure --enable-werror --disable-gnutls - || { cat config.log meson-logs/meson-log.txt; exit 1; } - - gmake -j$(sysctl -n hw.ncpu) - - gmake -j$(sysctl -n hw.ncpu) check V=1 - -macos_task: - osx_instance: - image: catalina-base - install_script: - - brew install pkg-config python gnu-sed glib pixman make sdl2 bash ninja - script: - - mkdir build - - cd build - - ../configure --python=/usr/local/bin/python3 --enable-werror - --extra-cflags='-Wno-error=deprecated-declarations' - || { cat config.log meson-logs/meson-log.txt; exit 1; } - - gmake -j$(sysctl -n hw.ncpu) - - gmake check-unit V=1 - - gmake check-block V=1 - - gmake check-qapi-schema V=1 - - gmake check-softfloat V=1 - - gmake check-qtest-x86_64 V=1 - -macos_xcode_task: - osx_instance: - # this is an alias for the latest Xcode - image: catalina-xcode - install_script: - - brew install pkg-config gnu-sed glib pixman make sdl2 bash ninja - script: - - mkdir build - - cd build - - ../configure --extra-cflags='-Wno-error=deprecated-declarations' --enable-modules - --enable-werror --cc=clang || { cat config.log meson-logs/meson-log.txt; exit 1; } - - gmake -j$(sysctl -n hw.ncpu) - - gmake check-unit V=1 - - gmake check-block V=1 - - gmake check-qapi-schema V=1 - - gmake check-softfloat V=1 - - gmake check-qtest-x86_64 V=1 - windows_msys2_task: timeout_in: 90m windows_container: From 51f5c849c1036203f5979ea40f61c08f3897908e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:34 +0100 Subject: [PATCH 11/44] hw/usb/ccid: remove references to NSS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The NSS package was previously pre-requisite for building CCID related features, however, this became obsolete when the libcacard library was spun off to a separate project: commit 7b02f5447c64d1854468f758398c9f6fe9e5721f Author: Marc-André Lureau Date: Sun Aug 30 11:48:40 2015 +0200 libcacard: use the standalone project Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Wainer dos Santos Moschetta Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Willian Rampazzo Reviewed-by: Alex Bennée Message-Id: <20210623142245.307776-2-berrange@redhat.com> Message-Id: <20210709143005.1554-10-alex.bennee@linaro.org> --- .travis.yml | 12 ++++++------ docs/ccid.txt | 15 +++++++-------- scripts/coverity-scan/coverity-scan.docker | 1 - tests/docker/dockerfiles/fedora.docker | 2 +- tests/docker/dockerfiles/ubuntu.docker | 1 - tests/docker/dockerfiles/ubuntu1804.docker | 1 - tests/docker/dockerfiles/ubuntu2004.docker | 1 - 7 files changed, 14 insertions(+), 19 deletions(-) diff --git a/.travis.yml b/.travis.yml index 4609240b5a..0faddf7b4e 100644 --- a/.travis.yml +++ b/.travis.yml @@ -27,6 +27,7 @@ addons: - libattr1-dev - libbrlapi-dev - libcap-ng-dev + - libcacard-dev - libgcc-7-dev - libgnutls28-dev - libgtk-3-dev @@ -34,7 +35,6 @@ addons: - liblttng-ust-dev - libncurses5-dev - libnfs-dev - - libnss3-dev - libpixman-1-dev - libpng-dev - librados-dev @@ -129,6 +129,7 @@ jobs: - libaio-dev - libattr1-dev - libbrlapi-dev + - libcacard-dev - libcap-ng-dev - libgcrypt20-dev - libgnutls28-dev @@ -137,7 +138,6 @@ jobs: - liblttng-ust-dev - libncurses5-dev - libnfs-dev - - libnss3-dev - libpixman-1-dev - libpng-dev - librados-dev @@ -163,6 +163,7 @@ jobs: - libaio-dev - libattr1-dev - libbrlapi-dev + - libcacard-dev - libcap-ng-dev - libgcrypt20-dev - libgnutls28-dev @@ -171,7 +172,6 @@ jobs: - liblttng-ust-dev - libncurses5-dev - libnfs-dev - - libnss3-dev - libpixman-1-dev - libpng-dev - librados-dev @@ -196,6 +196,7 @@ jobs: - libaio-dev - libattr1-dev - libbrlapi-dev + - libcacard-dev - libcap-ng-dev - libgcrypt20-dev - libgnutls28-dev @@ -204,7 +205,6 @@ jobs: - liblttng-ust-dev - libncurses5-dev - libnfs-dev - - libnss3-dev - libpixman-1-dev - libpng-dev - librados-dev @@ -238,6 +238,7 @@ jobs: apt_packages: - libaio-dev - libattr1-dev + - libcacard-dev - libcap-ng-dev - libgnutls28-dev - libiscsi-dev @@ -245,7 +246,6 @@ jobs: - liblzo2-dev - libncurses-dev - libnfs-dev - - libnss3-dev - libpixman-1-dev - libsdl2-dev - libsdl2-image-dev @@ -281,6 +281,7 @@ jobs: - libaio-dev - libattr1-dev - libbrlapi-dev + - libcacard-dev - libcap-ng-dev - libgcrypt20-dev - libgnutls28-dev @@ -289,7 +290,6 @@ jobs: - liblttng-ust-dev - libncurses5-dev - libnfs-dev - - libnss3-dev - libpixman-1-dev - libpng-dev - librados-dev diff --git a/docs/ccid.txt b/docs/ccid.txt index c97fbd2de0..2b85b1bd42 100644 --- a/docs/ccid.txt +++ b/docs/ccid.txt @@ -34,15 +34,14 @@ reader and smart card (i.e. not backed by a physical device) using this device. 2. Building -The cryptographic functions and access to the physical card is done via NSS. - -Installing NSS: +The cryptographic functions and access to the physical card is done via the +libcacard library, whose development package must be installed prior to +building QEMU: In redhat/fedora: - yum install nss-devel -In ubuntu/debian: - apt-get install libnss3-dev - (not tested on ubuntu) + yum install libcacard-devel +In ubuntu: + apt-get install libcacard-dev Configuring and building: ./configure --enable-smartcard && make @@ -51,7 +50,7 @@ Configuring and building: 3. Using ccid-card-emulated with hardware Assuming you have a working smartcard on the host with the current -user, using NSS, qemu acts as another NSS client using ccid-card-emulated: +user, using libcacard, QEMU acts as another client using ccid-card-emulated: qemu -usb -device usb-ccid -device ccid-card-emulated diff --git a/scripts/coverity-scan/coverity-scan.docker b/scripts/coverity-scan/coverity-scan.docker index 501ac67233..ecff6ac5b4 100644 --- a/scripts/coverity-scan/coverity-scan.docker +++ b/scripts/coverity-scan/coverity-scan.docker @@ -93,7 +93,6 @@ ENV PACKAGES \ mingw64-SDL2 \ ncurses-devel \ nettle-devel \ - nss-devel \ numactl-devel \ perl \ perl-Test-Harness \ diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker index 00cac5d61c..9dde3f6a78 100644 --- a/tests/docker/dockerfiles/fedora.docker +++ b/tests/docker/dockerfiles/fedora.docker @@ -28,6 +28,7 @@ ENV PACKAGES \ libasan \ libattr-devel \ libblockdev-mpath-devel \ + libcacard-devel \ libcap-ng-devel \ libcurl-devel \ libepoxy-devel \ @@ -82,7 +83,6 @@ ENV PACKAGES \ ncurses-devel \ nettle-devel \ ninja-build \ - nss-devel \ numactl-devel \ perl \ perl-Test-Harness \ diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker index 24d1647a65..100cfa76e3 100644 --- a/tests/docker/dockerfiles/ubuntu.docker +++ b/tests/docker/dockerfiles/ubuntu.docker @@ -40,7 +40,6 @@ ENV PACKAGES \ libncurses5-dev \ libncursesw5-dev \ libnfs-dev \ - libnss3-dev \ libnuma-dev \ libpixman-1-dev \ libpng-dev \ diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker index 2f1ec7c42b..86114be23a 100644 --- a/tests/docker/dockerfiles/ubuntu1804.docker +++ b/tests/docker/dockerfiles/ubuntu1804.docker @@ -27,7 +27,6 @@ ENV PACKAGES \ libncurses5-dev \ libncursesw5-dev \ libnfs-dev \ - libnss3-dev \ libnuma-dev \ libpixman-1-dev \ librados-dev \ diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker index fe993fe2a3..b33ed139d9 100644 --- a/tests/docker/dockerfiles/ubuntu2004.docker +++ b/tests/docker/dockerfiles/ubuntu2004.docker @@ -31,7 +31,6 @@ ENV PACKAGES flex bison \ libncurses5-dev \ libncursesw5-dev \ libnfs-dev \ - libnss3-dev \ libnuma-dev \ libpixman-1-dev \ librados-dev \ From 6ddc3dc7a882f2e7200fa7fecf505a8d0d8bbea9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:35 +0100 Subject: [PATCH 12/44] tests/docker: don't use BUILDKIT in GitLab either MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Using BUILDKIT breaks with certain container registries such as CentOS, with docker build reporting an error such as failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: unexpected status code https://registry.centos.org/v2/centos/manifests/7: 403 Forbidden Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Willian Rampazzo Message-Id: <20210623142245.307776-3-berrange@redhat.com> Message-Id: <20210709143005.1554-11-alex.bennee@linaro.org> --- tests/docker/docker.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/docker/docker.py b/tests/docker/docker.py index 4d9bb7c7ed..78dd13171e 100755 --- a/tests/docker/docker.py +++ b/tests/docker/docker.py @@ -228,7 +228,9 @@ class Docker(object): def __init__(self): self._command = _guess_engine_command() - if "docker" in self._command and "TRAVIS" not in os.environ: + if ("docker" in self._command and + "TRAVIS" not in os.environ and + "GITLAB_CI" not in os.environ): os.environ["DOCKER_BUILDKIT"] = "1" self._buildkit = True else: From 102cd5c294dcb251c814fff65e3024fac503f680 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:36 +0100 Subject: [PATCH 13/44] tests/docker: use project specific container registries MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Since Docker Hub has started to enforce pull rate limits on clients, it is preferrable to use project specific container registries where they are available. Fedora and OpenSUSE projects provide registries. The images in these registries are also refreshed on a more regular basis than the ones in docker hub, so the package update should generally be faster. While CentOS also has a registry it is considerably outdated compared to docker.io, and also only provides x86 images, while docker.io images are multi-arch. Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Wainer dos Santos Moschetta Reviewed-by: Willian Rampazzo Reviewed-by: Alex Bennée Message-Id: <20210623142245.307776-4-berrange@redhat.com> Message-Id: <20210709143005.1554-12-alex.bennee@linaro.org> --- tests/docker/dockerfiles/fedora-cris-cross.docker | 2 +- tests/docker/dockerfiles/fedora-i386-cross.docker | 2 +- tests/docker/dockerfiles/fedora-win32-cross.docker | 2 +- tests/docker/dockerfiles/fedora-win64-cross.docker | 2 +- tests/docker/dockerfiles/fedora.docker | 2 +- tests/docker/dockerfiles/opensuse-leap.docker | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/tests/docker/dockerfiles/fedora-cris-cross.docker b/tests/docker/dockerfiles/fedora-cris-cross.docker index 1dfff6e0b9..91c373fdd3 100644 --- a/tests/docker/dockerfiles/fedora-cris-cross.docker +++ b/tests/docker/dockerfiles/fedora-cris-cross.docker @@ -2,7 +2,7 @@ # Cross compiler for cris system tests # -FROM fedora:33 +FROM registry.fedoraproject.org/fedora:33 ENV PACKAGES gcc-cris-linux-gnu RUN dnf install -y $PACKAGES RUN rpm -q $PACKAGES | sort > /packages.txt diff --git a/tests/docker/dockerfiles/fedora-i386-cross.docker b/tests/docker/dockerfiles/fedora-i386-cross.docker index 8004fd8ee5..dbb8195eb1 100644 --- a/tests/docker/dockerfiles/fedora-i386-cross.docker +++ b/tests/docker/dockerfiles/fedora-i386-cross.docker @@ -1,4 +1,4 @@ -FROM fedora:33 +FROM registry.fedoraproject.org/fedora:33 ENV PACKAGES \ bzip2 \ ccache \ diff --git a/tests/docker/dockerfiles/fedora-win32-cross.docker b/tests/docker/dockerfiles/fedora-win32-cross.docker index a638afb525..9fed35f4e1 100644 --- a/tests/docker/dockerfiles/fedora-win32-cross.docker +++ b/tests/docker/dockerfiles/fedora-win32-cross.docker @@ -1,4 +1,4 @@ -FROM fedora:33 +FROM registry.fedoraproject.org/fedora:33 # Please keep this list sorted alphabetically ENV PACKAGES \ diff --git a/tests/docker/dockerfiles/fedora-win64-cross.docker b/tests/docker/dockerfiles/fedora-win64-cross.docker index f53007ac86..fb641f6104 100644 --- a/tests/docker/dockerfiles/fedora-win64-cross.docker +++ b/tests/docker/dockerfiles/fedora-win64-cross.docker @@ -1,4 +1,4 @@ -FROM fedora:33 +FROM registry.fedoraproject.org/fedora:33 # Please keep this list sorted alphabetically ENV PACKAGES \ diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker index 9dde3f6a78..a506e68d33 100644 --- a/tests/docker/dockerfiles/fedora.docker +++ b/tests/docker/dockerfiles/fedora.docker @@ -1,4 +1,4 @@ -FROM fedora:33 +FROM registry.fedoraproject.org/fedora:33 # Please keep this list sorted alphabetically ENV PACKAGES \ diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker index f7e1cbfbe6..7ebff1b3a8 100644 --- a/tests/docker/dockerfiles/opensuse-leap.docker +++ b/tests/docker/dockerfiles/opensuse-leap.docker @@ -1,4 +1,4 @@ -FROM opensuse/leap:15.2 +FROM registry.opensuse.org/opensuse/leap:15.2 # Please keep this list sorted alphabetically ENV PACKAGES \ From 1ae2786c7c91aadbdc15d4106e951bc15b46b585 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:37 +0100 Subject: [PATCH 14/44] tests/docker: use explicit docker.io registry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit It is good practice to use an explicit registry for referencing the base image. This is because some distros will inject their own registries into the search path. For example registry.fedoraproject.org comes ahead of docker.io. Using an explicit registry avoids wasting time querying multiple registries for images that they won't have. Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Wainer dos Santos Moschetta Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Alex Bennée Message-Id: <20210623142245.307776-5-berrange@redhat.com> Message-Id: <20210709143005.1554-13-alex.bennee@linaro.org> --- tests/docker/dockerfiles/centos8.docker | 2 +- tests/docker/dockerfiles/debian-xtensa-cross.docker | 2 +- tests/docker/dockerfiles/debian10.docker | 2 +- tests/docker/dockerfiles/debian11.docker | 2 +- tests/docker/dockerfiles/ubuntu.docker | 2 +- tests/docker/dockerfiles/ubuntu1804.docker | 2 +- tests/docker/dockerfiles/ubuntu2004.docker | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker index 03e0440e03..8f810810f3 100644 --- a/tests/docker/dockerfiles/centos8.docker +++ b/tests/docker/dockerfiles/centos8.docker @@ -1,4 +1,4 @@ -FROM centos:8.3.2011 +FROM docker.io/centos:8 RUN dnf -y update ENV PACKAGES \ diff --git a/tests/docker/dockerfiles/debian-xtensa-cross.docker b/tests/docker/dockerfiles/debian-xtensa-cross.docker index ba4148299c..2f11b3b7bc 100644 --- a/tests/docker/dockerfiles/debian-xtensa-cross.docker +++ b/tests/docker/dockerfiles/debian-xtensa-cross.docker @@ -5,7 +5,7 @@ # using a prebuilt toolchains for Xtensa cores from: # https://github.com/foss-xtensa/toolchain/releases # -FROM debian:stretch-slim +FROM docker.io/library/debian:stretch-slim RUN apt-get update && \ DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \ diff --git a/tests/docker/dockerfiles/debian10.docker b/tests/docker/dockerfiles/debian10.docker index 4ffe47671e..a27b88df55 100644 --- a/tests/docker/dockerfiles/debian10.docker +++ b/tests/docker/dockerfiles/debian10.docker @@ -7,7 +7,7 @@ # On its own you can't build much but the docker-foo-cross targets # build on top of the base debian image. # -FROM debian:buster-slim +FROM docker.io/library/debian:buster-slim # Duplicate deb line as deb-src RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list diff --git a/tests/docker/dockerfiles/debian11.docker b/tests/docker/dockerfiles/debian11.docker index 5adfd62d55..febf884f8f 100644 --- a/tests/docker/dockerfiles/debian11.docker +++ b/tests/docker/dockerfiles/debian11.docker @@ -8,7 +8,7 @@ # On its own you can't build much but the docker-foo-cross targets # build on top of the base debian image. # -FROM debian:bullseye-slim +FROM docker.io/library/debian:bullseye-slim # Duplicate deb line as deb-src RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker index 100cfa76e3..e0ff425c01 100644 --- a/tests/docker/dockerfiles/ubuntu.docker +++ b/tests/docker/dockerfiles/ubuntu.docker @@ -9,7 +9,7 @@ # system won't pick up that it has changed. # -FROM ubuntu:20.04 +FROM docker.io/library/ubuntu:20.04 ENV PACKAGES \ ccache \ clang \ diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker index 86114be23a..adcdef8244 100644 --- a/tests/docker/dockerfiles/ubuntu1804.docker +++ b/tests/docker/dockerfiles/ubuntu1804.docker @@ -1,4 +1,4 @@ -FROM ubuntu:18.04 +FROM docker.io/library/ubuntu:18.04 ENV PACKAGES \ ccache \ clang \ diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker index b33ed139d9..78755bc2e3 100644 --- a/tests/docker/dockerfiles/ubuntu2004.docker +++ b/tests/docker/dockerfiles/ubuntu2004.docker @@ -1,4 +1,4 @@ -FROM ubuntu:20.04 +FROM docker.io/library/ubuntu:20.04 ENV PACKAGES flex bison \ bsdmainutils \ ccache \ From 888673bbb9f5dc8e35c4747ddd1addba251a9a47 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:38 +0100 Subject: [PATCH 15/44] tests/docker: remove FEATURES env var from templates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In preparation for switching to auto-generated dockerfiles, remove the FEATURES env variable. The equivalent functionality can be achieved in most cases by just looking for existance of a binary. The cases which don't correspond to binaries are simply dropped because configure/meson will probe for any requested feature anyway. Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Willian Rampazzo Reviewed-by: Alex Bennée Message-Id: <20210623142245.307776-6-berrange@redhat.com> Message-Id: <20210709143005.1554-14-alex.bennee@linaro.org> --- tests/docker/common.rc | 19 ++++++++++++++----- tests/docker/dockerfiles/debian10.docker | 2 -- .../dockerfiles/fedora-win32-cross.docker | 1 - .../dockerfiles/fedora-win64-cross.docker | 1 - tests/docker/dockerfiles/fedora.docker | 1 - tests/docker/dockerfiles/ubuntu.docker | 1 - tests/docker/dockerfiles/ubuntu1804.docker | 1 - tests/docker/dockerfiles/ubuntu2004.docker | 1 - tests/docker/run | 3 --- tests/docker/test-clang | 2 +- tests/docker/test-debug | 2 +- tests/docker/test-mingw | 3 ++- tests/docker/test-misc | 2 +- tests/docker/test-tsan | 2 +- 14 files changed, 20 insertions(+), 21 deletions(-) diff --git a/tests/docker/common.rc b/tests/docker/common.rc index ebc5b97ecf..c5cc33d366 100755 --- a/tests/docker/common.rc +++ b/tests/docker/common.rc @@ -15,14 +15,23 @@ # overriden by TARGET_LIST if the user sets it. DEF_TARGET_LIST=${DEF_TARGET_LIST:-"x86_64-softmmu,aarch64-softmmu"} -requires() +requires_binary() { + found=0 for c in $@; do - if ! echo "$FEATURES" | grep -wq -e "$c"; then - echo "Prerequisite '$c' not present, skip" - exit 0 - fi + for d in /bin /usr/bin /usr/local/bin + do + if test -f "$d/$c" + then + found=1 + fi + done done + if test "$found" != "1" + then + echo "Prerequisite '$c' not present, skip" + exit 0 + fi } configure_qemu() diff --git a/tests/docker/dockerfiles/debian10.docker b/tests/docker/dockerfiles/debian10.docker index a27b88df55..b414af1b9f 100644 --- a/tests/docker/dockerfiles/debian10.docker +++ b/tests/docker/dockerfiles/debian10.docker @@ -35,5 +35,3 @@ RUN apt update && \ python3-sphinx \ python3-sphinx-rtd-theme \ $(apt-get -s build-dep --arch-only qemu | egrep ^Inst | fgrep '[all]' | cut -d\ -f2) - -ENV FEATURES docs diff --git a/tests/docker/dockerfiles/fedora-win32-cross.docker b/tests/docker/dockerfiles/fedora-win32-cross.docker index 9fed35f4e1..5a03e1af43 100644 --- a/tests/docker/dockerfiles/fedora-win32-cross.docker +++ b/tests/docker/dockerfiles/fedora-win32-cross.docker @@ -37,7 +37,6 @@ ENV PACKAGES \ RUN dnf install -y $PACKAGES RUN rpm -q $PACKAGES | sort > /packages.txt -ENV FEATURES mingw # Specify the cross prefix for this image (see tests/docker/common.rc) ENV QEMU_CONFIGURE_OPTS --cross-prefix=i686-w64-mingw32- diff --git a/tests/docker/dockerfiles/fedora-win64-cross.docker b/tests/docker/dockerfiles/fedora-win64-cross.docker index fb641f6104..ff706040c4 100644 --- a/tests/docker/dockerfiles/fedora-win64-cross.docker +++ b/tests/docker/dockerfiles/fedora-win64-cross.docker @@ -33,7 +33,6 @@ ENV PACKAGES \ RUN dnf install -y $PACKAGES RUN rpm -q $PACKAGES | sort > /packages.txt -ENV FEATURES mingw # Specify the cross prefix for this image (see tests/docker/common.rc) ENV QEMU_CONFIGURE_OPTS --cross-prefix=x86_64-w64-mingw32- --disable-capstone diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker index a506e68d33..64a413f5e0 100644 --- a/tests/docker/dockerfiles/fedora.docker +++ b/tests/docker/dockerfiles/fedora.docker @@ -117,4 +117,3 @@ ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3 RUN dnf install -y $PACKAGES RUN rpm -q $PACKAGES | sort > /packages.txt ENV PATH $PATH:/usr/libexec/python3-sphinx/ -ENV FEATURES mingw clang pyyaml asan docs diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker index e0ff425c01..f0e0180d21 100644 --- a/tests/docker/dockerfiles/ubuntu.docker +++ b/tests/docker/dockerfiles/ubuntu.docker @@ -69,4 +69,3 @@ ENV PACKAGES \ RUN apt-get update && \ DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES RUN dpkg -l $PACKAGES | sort > /packages.txt -ENV FEATURES clang pyyaml sdl2 docs diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker index adcdef8244..0acdb0d9ad 100644 --- a/tests/docker/dockerfiles/ubuntu1804.docker +++ b/tests/docker/dockerfiles/ubuntu1804.docker @@ -55,7 +55,6 @@ ENV PACKAGES \ RUN apt-get update && \ DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES RUN dpkg -l $PACKAGES | sort > /packages.txt -ENV FEATURES clang pyyaml sdl2 docs # https://bugs.launchpad.net/qemu/+bug/1838763 ENV QEMU_CONFIGURE_OPTS --disable-libssh diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker index 78755bc2e3..88b3cfa136 100644 --- a/tests/docker/dockerfiles/ubuntu2004.docker +++ b/tests/docker/dockerfiles/ubuntu2004.docker @@ -70,7 +70,6 @@ ENV PACKAGES flex bison \ RUN apt-get update && \ DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES RUN dpkg -l $PACKAGES | sort > /packages.txt -ENV FEATURES clang tsan pyyaml sdl2 # Apply patch https://reviews.llvm.org/D75820 # This is required for TSan in clang-10 to compile with QEMU. diff --git a/tests/docker/run b/tests/docker/run index 8edc7026ee..421393046b 100755 --- a/tests/docker/run +++ b/tests/docker/run @@ -30,9 +30,6 @@ mkdir -p $TEST_DIR/{src,build,install} # Extract the source tarballs tar -C $TEST_DIR/src -xf $BASE/qemu.tar || { echo "Failed to untar source"; exit 2; } -if test -f $TEST_DIR/src/Makefile; then - export FEATURES="$FEATURES dtc" -fi if test -n "$SHOW_ENV"; then if test -f /packages.txt; then diff --git a/tests/docker/test-clang b/tests/docker/test-clang index 8c51ead518..b57e0119d9 100755 --- a/tests/docker/test-clang +++ b/tests/docker/test-clang @@ -13,7 +13,7 @@ . common.rc -requires clang +requires_binary clang cd "$BUILD_DIR" diff --git a/tests/docker/test-debug b/tests/docker/test-debug index c050fa0d93..f52f16328c 100755 --- a/tests/docker/test-debug +++ b/tests/docker/test-debug @@ -14,7 +14,7 @@ . common.rc -requires clang asan +requires_binary clang cd "$BUILD_DIR" diff --git a/tests/docker/test-mingw b/tests/docker/test-mingw index c30eb654eb..0bc6d78872 100755 --- a/tests/docker/test-mingw +++ b/tests/docker/test-mingw @@ -13,7 +13,8 @@ . common.rc -requires mingw dtc +requires_binary x86_64-w64-mingw32-gcc +requires_binary i686-w64-mingw32-gcc cd "$BUILD_DIR" diff --git a/tests/docker/test-misc b/tests/docker/test-misc index cc94a738dd..2a3c2c2e1c 100755 --- a/tests/docker/test-misc +++ b/tests/docker/test-misc @@ -14,7 +14,7 @@ . common.rc -requires docs +requires_binary sphinx-build-3 sphinx-build cd "$BUILD_DIR" diff --git a/tests/docker/test-tsan b/tests/docker/test-tsan index eb40ac45b7..53d90d2f79 100755 --- a/tests/docker/test-tsan +++ b/tests/docker/test-tsan @@ -17,7 +17,7 @@ setup_tsan() { - requires clang tsan + requires_binary clang tsan_log_dir="/tmp/qemu-test/build/tsan" mkdir -p $tsan_log_dir > /dev/null || true EXTRA_CONFIGURE_OPTS="${EXTRA_CONFIGURE_OPTS} --enable-tsan \ From 52dab556578eda2ecf6ac1d67ee6149acfb8e832 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:39 +0100 Subject: [PATCH 16/44] tests/docker: fix sorting in package lists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This will make diffs in later patches clearer. Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Wainer dos Santos Moschetta Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Alex Bennée Message-Id: <20210623142245.307776-7-berrange@redhat.com> Message-Id: <20210709143005.1554-15-alex.bennee@linaro.org> --- tests/docker/dockerfiles/centos8.docker | 2 +- tests/docker/dockerfiles/fedora.docker | 4 ++-- tests/docker/dockerfiles/opensuse-leap.docker | 16 ++++++++-------- tests/docker/dockerfiles/ubuntu1804.docker | 4 ++-- tests/docker/dockerfiles/ubuntu2004.docker | 8 +++++--- 5 files changed, 18 insertions(+), 16 deletions(-) diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker index 8f810810f3..ee52305646 100644 --- a/tests/docker/dockerfiles/centos8.docker +++ b/tests/docker/dockerfiles/centos8.docker @@ -22,9 +22,9 @@ ENV PACKAGES \ lzo-devel \ make \ mesa-libEGL-devel \ - nmap-ncat \ nettle-devel \ ninja-build \ + nmap-ncat \ perl-Test-Harness \ pixman-devel \ python36 \ diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker index 64a413f5e0..4a0a84eb43 100644 --- a/tests/docker/dockerfiles/fedora.docker +++ b/tests/docker/dockerfiles/fedora.docker @@ -2,6 +2,7 @@ FROM registry.fedoraproject.org/fedora:33 # Please keep this list sorted alphabetically ENV PACKAGES \ + SDL2-devel \ bc \ brlapi-devel \ bzip2 \ @@ -79,10 +80,10 @@ ENV PACKAGES \ mingw64-pixman \ mingw64-pkg-config \ mingw64-SDL2 \ - nmap-ncat \ ncurses-devel \ nettle-devel \ ninja-build \ + nmap-ncat \ numactl-devel \ perl \ perl-Test-Harness \ @@ -97,7 +98,6 @@ ENV PACKAGES \ python3-sphinx_rtd_theme \ python3-virtualenv \ rdma-core-devel \ - SDL2-devel \ snappy-devel \ sparse \ spice-server-devel \ diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker index 7ebff1b3a8..6011447181 100644 --- a/tests/docker/dockerfiles/opensuse-leap.docker +++ b/tests/docker/dockerfiles/opensuse-leap.docker @@ -9,46 +9,46 @@ ENV PACKAGES \ cyrus-sasl-devel \ gcc \ gcc-c++ \ - mkisofs \ gettext-runtime \ git \ glib2-devel \ glusterfs-devel \ - libgnutls-devel \ gtk3-devel \ + libSDL2-devel \ + libSDL2_image-devel \ libaio-devel \ libattr-devel \ libcap-ng-devel \ libepoxy-devel \ libfdt-devel \ + libgnutls-devel \ libiscsi-devel \ libjpeg8-devel \ + libnuma-devel \ + libpixman-1-0-devel \ libpmem-devel \ libpng16-devel \ librbd-devel \ libseccomp-devel \ + libspice-server-devel \ libssh-devel \ lzo-devel \ make \ - libSDL2_image-devel \ + mkisofs \ ncurses-devel \ ninja \ - libnuma-devel \ perl \ - libpixman-1-0-devel \ python3-base \ python3-virtualenv \ rdma-core-devel \ - libSDL2-devel \ snappy-devel \ - libspice-server-devel \ systemd-devel \ systemtap-sdt-devel \ tar \ usbredir-devel \ virglrenderer-devel \ - xen-devel \ vte-devel \ + xen-devel \ zlib-devel ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3.6 diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker index 0acdb0d9ad..a50a35e6fe 100644 --- a/tests/docker/dockerfiles/ubuntu1804.docker +++ b/tests/docker/dockerfiles/ubuntu1804.docker @@ -46,10 +46,10 @@ ENV PACKAGES \ libxen-dev \ libzstd-dev \ make \ - python3-yaml \ + ninja-build \ python3-sphinx \ python3-sphinx-rtd-theme \ - ninja-build \ + python3-yaml \ sparse \ xfslibs-dev RUN apt-get update && \ diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker index 88b3cfa136..eee2ef3cac 100644 --- a/tests/docker/dockerfiles/ubuntu2004.docker +++ b/tests/docker/dockerfiles/ubuntu2004.docker @@ -1,8 +1,10 @@ FROM docker.io/library/ubuntu:20.04 -ENV PACKAGES flex bison \ +ENV PACKAGES \ + bison \ bsdmainutils \ ccache \ clang-10\ + flex \ gcc \ gcovr \ genisoimage \ @@ -65,8 +67,8 @@ ENV PACKAGES flex bison \ sparse \ tesseract-ocr \ tesseract-ocr-eng \ - xfslibs-dev\ - vim + vim \ + xfslibs-dev RUN apt-get update && \ DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES RUN dpkg -l $PACKAGES | sort > /packages.txt From 1d3a56d7f16020ce00d2c712a03579d4ea33db2d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:40 +0100 Subject: [PATCH 17/44] tests/docker: fix mistakes in centos package lists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit mesa-libEGL-devel is not used in QEMU at all, but mesa-libgbm-devel is. spice-glib-devel is not use in QEMU at all, but spice-protocol is. We also need the -devel package for spice-server, not the runtime. There is no need to specifically refer to python36, we can just use python3 as in other distros. Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Wainer dos Santos Moschetta Reviewed-by: Alex Bennée Message-Id: <20210623142245.307776-8-berrange@redhat.com> Message-Id: <20210709143005.1554-16-alex.bennee@linaro.org> --- tests/docker/dockerfiles/centos8.docker | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker index ee52305646..5f1c57b4ad 100644 --- a/tests/docker/dockerfiles/centos8.docker +++ b/tests/docker/dockerfiles/centos8.docker @@ -21,16 +21,16 @@ ENV PACKAGES \ libgcrypt-devel \ lzo-devel \ make \ - mesa-libEGL-devel \ + mesa-libgbm-devel \ nettle-devel \ ninja-build \ nmap-ncat \ perl-Test-Harness \ pixman-devel \ - python36 \ + python3 \ rdma-core-devel \ - spice-glib-devel \ - spice-server \ + spice-protocol \ + spice-server-devel \ systemtap-sdt-devel \ tar \ zlib-devel From e053de7e63c0b4437ec963bf42caca59b9cf38bc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:41 +0100 Subject: [PATCH 18/44] tests/docker: fix mistakes in fedora package list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit libblockdev-mpath-devel is not used by QEMU, rather it wants device-mapper-multipath-devel. Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Wainer dos Santos Moschetta Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Alex Bennée Message-Id: <20210623142245.307776-9-berrange@redhat.com> Message-Id: <20210709143005.1554-17-alex.bennee@linaro.org> --- tests/docker/dockerfiles/fedora.docker | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker index 4a0a84eb43..f667f03cc5 100644 --- a/tests/docker/dockerfiles/fedora.docker +++ b/tests/docker/dockerfiles/fedora.docker @@ -28,7 +28,6 @@ ENV PACKAGES \ libaio-devel \ libasan \ libattr-devel \ - libblockdev-mpath-devel \ libcacard-devel \ libcap-ng-devel \ libcurl-devel \ From dfaaacc67f82dc660ea59052d0458938a7f8d6f6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:42 +0100 Subject: [PATCH 19/44] tests/docker: fix mistakes in ubuntu package lists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit librados-dev is not required by QEMU directly, only librbd-dev. glusterfs-common is not directly needed by QEMU. QEMU uses ncursesw only on non-Windows hosts. The clang package is clang 10. flex and bison are not required by QEMU. Standardize on nmap ncat implementation to match Fedora/CentOS. Remove vim since it is not a build pre-requisite and no other containers include it. Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Wainer dos Santos Moschetta Reviewed-by: Alex Bennée Message-Id: <20210623142245.307776-10-berrange@redhat.com> Message-Id: <20210709143005.1554-18-alex.bennee@linaro.org> --- tests/docker/dockerfiles/ubuntu1804.docker | 3 --- tests/docker/dockerfiles/ubuntu2004.docker | 10 ++-------- 2 files changed, 2 insertions(+), 11 deletions(-) diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker index a50a35e6fe..ee8545e4b1 100644 --- a/tests/docker/dockerfiles/ubuntu1804.docker +++ b/tests/docker/dockerfiles/ubuntu1804.docker @@ -5,7 +5,6 @@ ENV PACKAGES \ gcc \ gettext \ git \ - glusterfs-common \ libaio-dev \ libattr1-dev \ libbrlapi-dev \ @@ -24,12 +23,10 @@ ENV PACKAGES \ libjemalloc-dev \ libjpeg-turbo8-dev \ liblzo2-dev \ - libncurses5-dev \ libncursesw5-dev \ libnfs-dev \ libnuma-dev \ libpixman-1-dev \ - librados-dev \ librbd-dev \ librdmacm-dev \ libsasl2-dev \ diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker index eee2ef3cac..25f56adfb2 100644 --- a/tests/docker/dockerfiles/ubuntu2004.docker +++ b/tests/docker/dockerfiles/ubuntu2004.docker @@ -1,16 +1,13 @@ FROM docker.io/library/ubuntu:20.04 ENV PACKAGES \ - bison \ bsdmainutils \ ccache \ - clang-10\ - flex \ + clang \ gcc \ gcovr \ genisoimage \ gettext \ git \ - glusterfs-common \ libaio-dev \ libattr1-dev \ libbrlapi-dev \ @@ -30,12 +27,10 @@ ENV PACKAGES \ libjpeg-turbo8-dev \ liblttng-ust-dev \ liblzo2-dev \ - libncurses5-dev \ libncursesw5-dev \ libnfs-dev \ libnuma-dev \ libpixman-1-dev \ - librados-dev \ librbd-dev \ librdmacm-dev \ libsasl2-dev \ @@ -53,7 +48,7 @@ ENV PACKAGES \ libxen-dev \ libzstd-dev \ make \ - netcat-openbsd \ + ncat \ ninja-build \ python3-numpy \ python3-opencv \ @@ -67,7 +62,6 @@ ENV PACKAGES \ sparse \ tesseract-ocr \ tesseract-ocr-eng \ - vim \ xfslibs-dev RUN apt-get update && \ DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES From 52141ab37de49daca3adee7a72065834a696c693 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:43 +0100 Subject: [PATCH 20/44] tests/docker: remove mingw packages from Fedora MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit There are dedicated containers providing mingw packages for Fedora. Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Alex Bennée Message-Id: <20210623142245.307776-11-berrange@redhat.com> Message-Id: <20210709143005.1554-19-alex.bennee@linaro.org> --- tests/docker/dockerfiles/fedora.docker | 27 -------------------------- 1 file changed, 27 deletions(-) diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker index f667f03cc5..5849ea7617 100644 --- a/tests/docker/dockerfiles/fedora.docker +++ b/tests/docker/dockerfiles/fedora.docker @@ -52,33 +52,6 @@ ENV PACKAGES \ lzo-devel \ make \ meson \ - mingw32-bzip2 \ - mingw32-curl \ - mingw32-glib2 \ - mingw32-gmp \ - mingw32-gnutls \ - mingw32-gtk3 \ - mingw32-libjpeg-turbo \ - mingw32-libpng \ - mingw32-libtasn1 \ - mingw32-nettle \ - mingw32-nsis \ - mingw32-pixman \ - mingw32-pkg-config \ - mingw32-SDL2 \ - mingw64-bzip2 \ - mingw64-curl \ - mingw64-glib2 \ - mingw64-gmp \ - mingw64-gnutls \ - mingw64-gtk3 \ - mingw64-libjpeg-turbo \ - mingw64-libpng \ - mingw64-libtasn1 \ - mingw64-nettle \ - mingw64-pixman \ - mingw64-pkg-config \ - mingw64-SDL2 \ ncurses-devel \ nettle-devel \ ninja-build \ From b5883710c4bb0e2181ad30b89888a35ba9073206 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:44 +0100 Subject: [PATCH 21/44] tests/docker: expand centos8 package list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This is the fully expanded list of build pre-requisites QEMU can conceivably use in any scenario. [AJB: added centos-release-advanced-virtualization/epel-release] Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Alex Bennée Message-Id: <20210623142245.307776-12-berrange@redhat.com> Message-Id: <20210709143005.1554-20-alex.bennee@linaro.org> --- tests/docker/dockerfiles/centos8.docker | 70 +++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker index 5f1c57b4ad..46398c61ee 100644 --- a/tests/docker/dockerfiles/centos8.docker +++ b/tests/docker/dockerfiles/centos8.docker @@ -3,39 +3,109 @@ FROM docker.io/centos:8 RUN dnf -y update ENV PACKAGES \ SDL2-devel \ + alsa-lib-devel \ + bc \ + brlapi-devel \ bzip2 \ bzip2-devel \ + ca-certificates \ + capstone-devel \ + ccache \ + clang \ + ctags \ + cyrus-sasl-devel \ + daxctl-devel \ dbus-daemon \ + device-mapper-multipath-devel \ diffutils \ + findutils \ gcc \ gcc-c++ \ genisoimage \ gettext \ git \ glib2-devel \ + glibc-langpack-en \ + glibc-static \ + glusterfs-api-devel \ + gnutls-devel \ + gtk3-devel \ + hostname \ + jemalloc-devel \ libaio-devel \ + libasan \ + libattr-devel \ libbpf-devel \ + libcacard-devel \ + libcap-ng-devel \ + libcurl-devel \ + libdrm-devel \ libepoxy-devel \ libfdt-devel \ libffi-devel \ libgcrypt-devel \ + libiscsi-devel \ + libjpeg-devel \ + libnfs-devel \ + libpmem-devel \ + libpng-devel \ + librbd-devel \ + libseccomp-devel \ + libslirp-devel \ + libssh-devel \ + libtasn1-devel \ + libubsan \ + libudev-devel \ + libusbx-devel \ + libxml2-devel \ + libzstd-devel \ + llvm \ lzo-devel \ make \ mesa-libgbm-devel \ + ncurses-devel \ nettle-devel \ ninja-build \ nmap-ncat \ + numactl-devel \ + openssh-clients \ + pam-devel \ + perl \ perl-Test-Harness \ pixman-devel \ + pkgconfig \ + pulseaudio-libs-devel \ python3 \ + python3-PyYAML \ + python3-numpy \ + python3-pillow \ + python3-pip \ + python3-setuptools \ + python3-sphinx \ + python3-sphinx_rtd_theme \ + python3-virtualenv \ + python3-wheel \ rdma-core-devel \ + rpm \ + sed \ + snappy-devel \ spice-protocol \ spice-server-devel \ + systemd-devel \ systemtap-sdt-devel \ tar \ + texinfo \ + usbredir-devel \ + util-linux \ + virglrenderer-devel \ + vte291-devel \ + which \ + xfsprogs-devel \ zlib-devel RUN dnf install -y dnf-plugins-core && \ dnf config-manager --set-enabled powertools && \ + dnf install -y centos-release-advanced-virtualization && \ + dnf install -y epel-release && \ dnf install -y $PACKAGES RUN rpm -q $PACKAGES | sort > /packages.txt From 5a9c1498e36f32e2e30aeac96991802e8496436d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:45 +0100 Subject: [PATCH 22/44] tests/docker: expand fedora package list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This is the fully expanded list of build pre-requisites QEMU can conceivably use in any scenario. Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Alex Bennée Message-Id: <20210623142245.307776-13-berrange@redhat.com> Message-Id: <20210709143005.1554-21-alex.bennee@linaro.org> --- tests/docker/dockerfiles/fedora.docker | 30 ++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker index 5849ea7617..eec1add7f6 100644 --- a/tests/docker/dockerfiles/fedora.docker +++ b/tests/docker/dockerfiles/fedora.docker @@ -3,63 +3,83 @@ FROM registry.fedoraproject.org/fedora:33 # Please keep this list sorted alphabetically ENV PACKAGES \ SDL2-devel \ + SDL2_image-devel \ + alsa-lib-devel \ bc \ brlapi-devel \ bzip2 \ bzip2-devel \ + ca-certificates \ capstone-devel \ ccache \ clang \ + ctags \ cyrus-sasl-devel \ + daxctl-devel \ dbus-daemon \ device-mapper-multipath-devel \ diffutils \ findutils \ gcc \ gcc-c++ \ + gcovr \ genisoimage \ gettext \ git \ glib2-devel \ + glibc-langpack-en \ + glibc-static \ glusterfs-api-devel \ gnutls-devel \ gtk3-devel \ hostname \ + jemalloc-devel \ libaio-devel \ libasan \ libattr-devel \ + libbpf-devel \ libcacard-devel \ libcap-ng-devel \ libcurl-devel \ + libdrm-devel \ libepoxy-devel \ libfdt-devel \ - libbpf-devel \ libffi-devel \ + libgcrypt-devel \ libiscsi-devel \ libjpeg-devel \ + libnfs-devel \ libpmem-devel \ libpng-devel \ librbd-devel \ libseccomp-devel \ libslirp-devel \ libssh-devel \ + libtasn1-devel \ libubsan \ libudev-devel \ + liburing-devel \ libusbx-devel \ libxml2-devel \ libzstd-devel \ llvm \ + lttng-ust-devel \ lzo-devel \ make \ + mesa-libgbm-devel \ meson \ ncurses-devel \ nettle-devel \ ninja-build \ nmap-ncat \ numactl-devel \ - perl \ + openssh-clients \ + pam-devel \ perl-Test-Harness \ + perl-base \ pixman-devel \ + pkgconfig \ + pulseaudio-libs-devel \ python3 \ python3-PyYAML \ python3-numpy \ @@ -70,19 +90,25 @@ ENV PACKAGES \ python3-sphinx_rtd_theme \ python3-virtualenv \ rdma-core-devel \ + rpm \ + sed \ snappy-devel \ sparse \ + spice-protocol \ spice-server-devel \ systemd-devel \ systemtap-sdt-devel \ tar \ tesseract \ tesseract-langpack-eng \ + texinfo \ usbredir-devel \ + util-linux \ virglrenderer-devel \ vte291-devel \ which \ xen-devel \ + xfsprogs-devel \ zlib-devel ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3 From 0fef572e4343afb4083b3c42bda152beb9ccd883 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:46 +0100 Subject: [PATCH 23/44] tests/docker: expand ubuntu1804 package list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This is the fully expanded list of build pre-requisites QEMU can conceivably use in any scenario. Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Alex Bennée Message-Id: <20210623142245.307776-14-berrange@redhat.com> Message-Id: <20210709143005.1554-22-alex.bennee@linaro.org> --- tests/docker/dockerfiles/ubuntu1804.docker | 61 +++++++++++++++++++++- 1 file changed, 60 insertions(+), 1 deletion(-) diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker index ee8545e4b1..0880bf3e29 100644 --- a/tests/docker/dockerfiles/ubuntu1804.docker +++ b/tests/docker/dockerfiles/ubuntu1804.docker @@ -1,54 +1,113 @@ FROM docker.io/library/ubuntu:18.04 ENV PACKAGES \ + bc \ + bsdmainutils \ + bzip2 \ + ca-certificates \ ccache \ clang \ + dbus \ + debianutils \ + diffutils \ + exuberant-ctags \ + findutils \ + g++ \ gcc \ + gcovr \ + genisoimage \ gettext \ git \ + glusterfs-common \ + hostname \ libaio-dev \ + libasan5 \ + libasound2-dev \ libattr1-dev \ libbrlapi-dev \ libbz2-dev \ + libc6-dev \ libcacard-dev \ libcap-ng-dev \ + libcapstone-dev \ libcurl4-gnutls-dev \ + libdaxctl-dev \ libdrm-dev \ libepoxy-dev \ libfdt-dev \ libffi-dev \ libgbm-dev \ + libgcrypt20-dev \ + libglib2.0-dev \ + libgnutls28-dev \ libgtk-3-dev \ libibverbs-dev \ libiscsi-dev \ libjemalloc-dev \ libjpeg-turbo8-dev \ + liblttng-ust-dev \ liblzo2-dev \ libncursesw5-dev \ libnfs-dev \ libnuma-dev \ + libpam0g-dev \ libpixman-1-dev \ + libpmem-dev \ + libpng-dev \ + libpulse-dev \ librbd-dev \ librdmacm-dev \ libsasl2-dev \ libsdl2-dev \ + libsdl2-image-dev \ libseccomp-dev \ libsnappy-dev \ libspice-protocol-dev \ libspice-server-dev \ libssh-dev \ + libsystemd-dev \ + libtasn1-6-dev \ + libtest-harness-perl \ + libubsan1 \ + libudev-dev \ libusb-1.0-0-dev \ libusbredirhost-dev \ libvdeplug-dev \ + libvirglrenderer-dev \ libvte-2.91-dev \ libxen-dev \ + libxml2-dev \ libzstd-dev \ + llvm \ + locales \ make \ + multipath-tools \ + netcat-openbsd \ + nettle-dev \ ninja-build \ + openssh-client \ + perl-base \ + pkgconf \ + python3 \ + python3-numpy \ + python3-opencv \ + python3-pillow \ + python3-pip \ + python3-setuptools \ python3-sphinx \ python3-sphinx-rtd-theme \ + python3-venv \ + python3-wheel \ python3-yaml \ + rpm2cpio \ + sed \ sparse \ - xfslibs-dev + systemtap-sdt-dev \ + tar \ + tesseract-ocr \ + tesseract-ocr-eng \ + texinfo \ + xfslibs-dev \ + zlib1g-dev RUN apt-get update && \ DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES RUN dpkg -l $PACKAGES | sort > /packages.txt From bc4a117b79b03cbbeea0d22807c2566c579d2b8b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:47 +0100 Subject: [PATCH 24/44] tests/docker: expand ubuntu2004 package list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This is the fully expanded list of build pre-requisites QEMU can conceivably use in any scenario. Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Alex Bennée Message-Id: <20210623142245.307776-15-berrange@redhat.com> Message-Id: <20210709143005.1554-23-alex.bennee@linaro.org> --- tests/docker/dockerfiles/ubuntu2004.docker | 50 +++++++++++++++++++++- 1 file changed, 48 insertions(+), 2 deletions(-) diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker index 25f56adfb2..39de63d012 100644 --- a/tests/docker/dockerfiles/ubuntu2004.docker +++ b/tests/docker/dockerfiles/ubuntu2004.docker @@ -1,25 +1,44 @@ FROM docker.io/library/ubuntu:20.04 ENV PACKAGES \ + bc \ bsdmainutils \ + bzip2 \ + ca-certificates \ ccache \ clang \ + dbus \ + debianutils \ + diffutils \ + exuberant-ctags \ + findutils \ + g++ \ gcc \ gcovr \ genisoimage \ gettext \ git \ + hostname \ libaio-dev \ + libasan5 \ + libasound2-dev \ libattr1-dev \ libbrlapi-dev \ libbz2-dev \ + libc6-dev \ libcacard-dev \ libcap-ng-dev \ + libcapstone-dev \ libcurl4-gnutls-dev \ + libdaxctl-dev \ libdrm-dev \ libepoxy-dev \ libfdt-dev \ libffi-dev \ libgbm-dev \ + libgcrypt20-dev \ + libglib2.0-dev \ + libglusterfs-dev \ + libgnutls28-dev \ libgtk-3-dev \ libibverbs-dev \ libiscsi-dev \ @@ -30,39 +49,66 @@ ENV PACKAGES \ libncursesw5-dev \ libnfs-dev \ libnuma-dev \ + libpam0g-dev \ libpixman-1-dev \ + libpmem-dev \ + libpng-dev \ + libpulse-dev \ librbd-dev \ librdmacm-dev \ libsasl2-dev \ libsdl2-dev \ + libsdl2-image-dev \ libseccomp-dev \ libslirp-dev \ libsnappy-dev \ libspice-protocol-dev \ libspice-server-dev \ libssh-dev \ + libsystemd-dev \ + libtasn1-6-dev \ + libtest-harness-perl \ + libubsan1 \ + libudev-dev \ libusb-1.0-0-dev \ libusbredirhost-dev \ libvdeplug-dev \ + libvirglrenderer-dev \ libvte-2.91-dev \ libxen-dev \ + libxml2-dev \ libzstd-dev \ + llvm \ + locales \ make \ + multipath-tools \ ncat \ + nettle-dev \ ninja-build \ + openssh-client \ + perl-base \ + pkgconf \ + python3 \ python3-numpy \ python3-opencv \ - python3-pil \ + python3-pillow \ python3-pip \ + python3-setuptools \ python3-sphinx \ python3-sphinx-rtd-theme \ python3-venv \ + python3-wheel \ python3-yaml \ rpm2cpio \ + sed \ sparse \ + systemtap-sdt-dev \ + tar \ tesseract-ocr \ tesseract-ocr-eng \ - xfslibs-dev + texinfo \ + xfslibs-dev \ + zlib1g-dev RUN apt-get update && \ DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES RUN dpkg -l $PACKAGES | sort > /packages.txt From 5606ce9ee4db7175cc93ee9335d3c9f9e1a60d00 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Date: Fri, 9 Jul 2021 15:29:48 +0100 Subject: [PATCH 25/44] tests/docker: expand opensuse-leap package list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This is the fully expanded list of build pre-requisites QEMU can conceivably use in any scenario. Signed-off-by: Daniel P. Berrangé Signed-off-by: Alex Bennée Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Alex Bennée Message-Id: <20210623142245.307776-16-berrange@redhat.com> Message-Id: <20210709143005.1554-24-alex.bennee@linaro.org> --- tests/docker/dockerfiles/opensuse-leap.docker | 59 ++++++++++++++++++- 1 file changed, 58 insertions(+), 1 deletion(-) diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker index 6011447181..5a8bee0289 100644 --- a/tests/docker/dockerfiles/opensuse-leap.docker +++ b/tests/docker/dockerfiles/opensuse-leap.docker @@ -2,53 +2,110 @@ FROM registry.opensuse.org/opensuse/leap:15.2 # Please keep this list sorted alphabetically ENV PACKAGES \ + Mesa-devel \ + alsa-lib-devel \ bc \ brlapi-devel \ bzip2 \ + ca-certificates \ ccache \ + clang \ + ctags \ cyrus-sasl-devel \ + dbus-1 \ + diffutils \ + findutils \ gcc \ gcc-c++ \ + gcovr \ gettext-runtime \ git \ glib2-devel \ + glibc-locale \ + glibc-static \ glusterfs-devel \ gtk3-devel \ + hostname \ + jemalloc-devel \ libSDL2-devel \ libSDL2_image-devel \ libaio-devel \ + libasan6 \ libattr-devel \ + libbpf-devel \ + libbz2-devel \ + libcacard-devel \ libcap-ng-devel \ + libcurl-devel \ + libdrm-devel \ libepoxy-devel \ libfdt-devel \ + libffi-devel \ + libgcrypt-devel \ libgnutls-devel \ libiscsi-devel \ libjpeg8-devel \ + libndctl-devel \ + libnettle-devel \ + libnfs-devel \ libnuma-devel \ libpixman-1-0-devel \ libpmem-devel \ libpng16-devel \ + libpulse-devel \ librbd-devel \ libseccomp-devel \ libspice-server-devel \ libssh-devel \ + libtasn1-devel \ + libubsan1 \ + libudev-devel \ + libusb-1_0-devel \ + libxml2-devel \ + libzstd-devel \ + llvm \ + lttng-ust-devel \ lzo-devel \ make \ mkisofs \ + ncat \ ncurses-devel \ ninja \ - perl \ + openssh \ + pam-devel \ + perl-Test-Harness \ + perl-base \ + pkgconfig \ + python3-Pillow \ + python3-PyYAML \ + python3-Sphinx \ python3-base \ + python3-numpy \ + python3-opencv \ + python3-pip \ + python3-setuptools \ + python3-sphinx_rtd_theme \ python3-virtualenv \ + python3-wheel \ rdma-core-devel \ + rpm \ + sed \ snappy-devel \ + sparse \ + spice-protocol-devel \ systemd-devel \ systemtap-sdt-devel \ tar \ + tesseract-ocr \ + tesseract-ocr-traineddata-english \ + texinfo \ usbredir-devel \ + util-linux \ virglrenderer-devel \ vte-devel \ + which \ xen-devel \ + xfsprogs-devel \ zlib-devel ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3.6 From 2b36d741b4f2583c8f45ff96a6223c950f86523b Mon Sep 17 00:00:00 2001 From: Brad Smith Date: Fri, 9 Jul 2021 15:29:49 +0100 Subject: [PATCH 26/44] tests/vm: update NetBSD to 9.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit tests/vm: update NetBSD to 9.2 Signed-off-by: Brad Smith Signed-off-by: Alex Bennée Tested-by: Gerd Hoffmann Reviewed-by: Wainer dos Santos Moschetta Acked-by: Philippe Mathieu-Daudé Message-Id: Message-Id: <20210709143005.1554-25-alex.bennee@linaro.org> --- tests/vm/netbsd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/vm/netbsd b/tests/vm/netbsd index b9efc269d2..4cc58df130 100755 --- a/tests/vm/netbsd +++ b/tests/vm/netbsd @@ -22,8 +22,8 @@ class NetBSDVM(basevm.BaseVM): name = "netbsd" arch = "x86_64" - link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-9.1/images/NetBSD-9.1-amd64.iso" - csum = "65bddc95945991c3b2021f9c8ded7f34c25f0a7611b7aa15a15fe23399e902307e926ae97fcd01dc1662ac67b5f6e4be643c6a2b581692ddcb616d30125066f9" + link = "https://cdn.netbsd.org/pub/NetBSD/NetBSD-9.2/images/NetBSD-9.2-amd64.iso" + csum = "5ee0ea101f73386b9b424f5d1041e371db3c42fdd6f4e4518dc79c4a08f31d43091ebe93425c9f0dcaaed2b51131836fe6774f33f89030b58d64709b35fda72f" size = "20G" pkgs = [ # tools From 396c9984fd18e4243d02f78bc4942a825c78dca2 Mon Sep 17 00:00:00 2001 From: Brad Smith Date: Fri, 9 Jul 2021 15:29:50 +0100 Subject: [PATCH 27/44] tests/vm: update openbsd to release 6.9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit tests/vm: update openbsd to release 6.9 Signed-off-by: Brad Smith Signed-off-by: Alex Bennée Tested-by: Gerd Hoffmann Acked-by: Philippe Mathieu-Daudé Message-Id: Message-Id: <20210709143005.1554-26-alex.bennee@linaro.org> --- tests/vm/openbsd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/vm/openbsd b/tests/vm/openbsd index 4d1399378e..c4c78a80f1 100755 --- a/tests/vm/openbsd +++ b/tests/vm/openbsd @@ -22,8 +22,8 @@ class OpenBSDVM(basevm.BaseVM): name = "openbsd" arch = "x86_64" - link = "https://cdn.openbsd.org/pub/OpenBSD/6.8/amd64/install68.iso" - csum = "47e291fcc2d0c1a8ae0b66329f040b33af755b6adbd21739e20bb5ad56f62b6c" + link = "https://cdn.openbsd.org/pub/OpenBSD/6.9/amd64/install69.iso" + csum = "140d26548aec680e34bb5f82295414228e7f61e4f5e7951af066014fda2d6e43" size = "20G" pkgs = [ # tools From e2bcd3ad093a6bd4a045d5b5d22f0b92243d0da3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 9 Jul 2021 15:29:51 +0100 Subject: [PATCH 28/44] tests/tcg: make test-mmap a little less aggressive MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The check_aligned_anonymous_unfixed_mmaps and check_aligned_anonymous_unfixed_colliding_mmaps do a lot of mmap's and copying of data. This is especially unfriendly to targets like hexagon which have quite large pages and need to do sanity checks on each memory access. While we are at it clean-up the white space and style issues from the legacy code. As we no longer do quite so much needless memory access we can also remove the hexagon timeout hack. Signed-off-by: Alex Bennée Reviewed-by: Richard Henderson Reviewed-by: Taylor Simpson Message-Id: <20210709143005.1554-27-alex.bennee@linaro.org> --- tests/tcg/hexagon/Makefile.target | 9 -- tests/tcg/multiarch/test-mmap.c | 190 +++++++++++++++--------------- 2 files changed, 95 insertions(+), 104 deletions(-) diff --git a/tests/tcg/hexagon/Makefile.target b/tests/tcg/hexagon/Makefile.target index 0992787d50..050cd61c1a 100644 --- a/tests/tcg/hexagon/Makefile.target +++ b/tests/tcg/hexagon/Makefile.target @@ -18,15 +18,6 @@ # Hexagon doesn't support gdb, so skip the EXTRA_RUNS EXTRA_RUNS = -# Hexagon has 64K pages, so increase the timeout to keep -# test-mmap from timing out -ifeq ($(CONFIG_DEBUG_TCG),y) -TIMEOUT=800 -else -TIMEOUT=500 -endif - - CFLAGS += -Wno-incompatible-pointer-types -Wno-undefined-internal CFLAGS += -fno-unroll-loops diff --git a/tests/tcg/multiarch/test-mmap.c b/tests/tcg/multiarch/test-mmap.c index 11d0e777b1..96257f8ebe 100644 --- a/tests/tcg/multiarch/test-mmap.c +++ b/tests/tcg/multiarch/test-mmap.c @@ -49,64 +49,62 @@ size_t test_fsize; void check_aligned_anonymous_unfixed_mmaps(void) { - void *p1; - void *p2; - void *p3; - void *p4; - void *p5; - uintptr_t p; - int i; + void *p1; + void *p2; + void *p3; + void *p4; + void *p5; + uintptr_t p; + int i; + fprintf(stdout, "%s", __func__); + for (i = 0; i < 8; i++) { + size_t len; + len = pagesize + (pagesize * i); + p1 = mmap(NULL, len, PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + p2 = mmap(NULL, len, PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + p3 = mmap(NULL, len, PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + p4 = mmap(NULL, len, PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + p5 = mmap(NULL, len, PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); - fprintf(stdout, "%s", __func__); - for (i = 0; i < 0x1fff; i++) - { - size_t len; + /* + * Make sure we get pages aligned with the pagesize. The + * target expects this. + */ + fail_unless(p1 != MAP_FAILED); + fail_unless(p2 != MAP_FAILED); + fail_unless(p3 != MAP_FAILED); + fail_unless(p4 != MAP_FAILED); + fail_unless(p5 != MAP_FAILED); + p = (uintptr_t) p1; + D(printf("p=%x\n", p)); + fail_unless((p & pagemask) == 0); + p = (uintptr_t) p2; + fail_unless((p & pagemask) == 0); + p = (uintptr_t) p3; + fail_unless((p & pagemask) == 0); + p = (uintptr_t) p4; + fail_unless((p & pagemask) == 0); + p = (uintptr_t) p5; + fail_unless((p & pagemask) == 0); - len = pagesize + (pagesize * i & 7); - p1 = mmap(NULL, len, PROT_READ, - MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); - p2 = mmap(NULL, len, PROT_READ, - MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); - p3 = mmap(NULL, len, PROT_READ, - MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); - p4 = mmap(NULL, len, PROT_READ, - MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); - p5 = mmap(NULL, len, PROT_READ, - MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); - - /* Make sure we get pages aligned with the pagesize. The - target expects this. */ - fail_unless (p1 != MAP_FAILED); - fail_unless (p2 != MAP_FAILED); - fail_unless (p3 != MAP_FAILED); - fail_unless (p4 != MAP_FAILED); - fail_unless (p5 != MAP_FAILED); - p = (uintptr_t) p1; - D(printf ("p=%x\n", p)); - fail_unless ((p & pagemask) == 0); - p = (uintptr_t) p2; - fail_unless ((p & pagemask) == 0); - p = (uintptr_t) p3; - fail_unless ((p & pagemask) == 0); - p = (uintptr_t) p4; - fail_unless ((p & pagemask) == 0); - p = (uintptr_t) p5; - fail_unless ((p & pagemask) == 0); - - /* Make sure we can read from the entire area. */ - memcpy (dummybuf, p1, pagesize); - memcpy (dummybuf, p2, pagesize); - memcpy (dummybuf, p3, pagesize); - memcpy (dummybuf, p4, pagesize); - memcpy (dummybuf, p5, pagesize); - - munmap (p1, len); - munmap (p2, len); - munmap (p3, len); - munmap (p4, len); - munmap (p5, len); - } - fprintf(stdout, " passed\n"); + /* Make sure we can read from the entire area. */ + memcpy(dummybuf, p1, pagesize); + memcpy(dummybuf, p2, pagesize); + memcpy(dummybuf, p3, pagesize); + memcpy(dummybuf, p4, pagesize); + memcpy(dummybuf, p5, pagesize); + munmap(p1, len); + munmap(p2, len); + munmap(p3, len); + munmap(p4, len); + munmap(p5, len); + } + fprintf(stdout, " passed\n"); } void check_large_anonymous_unfixed_mmap(void) @@ -135,52 +133,54 @@ void check_large_anonymous_unfixed_mmap(void) void check_aligned_anonymous_unfixed_colliding_mmaps(void) { - char *p1; - char *p2; - char *p3; - uintptr_t p; - int i; + char *p1; + char *p2; + char *p3; + uintptr_t p; + int i; - fprintf(stdout, "%s", __func__); - for (i = 0; i < 0x2fff; i++) - { - int nlen; - p1 = mmap(NULL, pagesize, PROT_READ, - MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); - fail_unless (p1 != MAP_FAILED); - p = (uintptr_t) p1; - fail_unless ((p & pagemask) == 0); - memcpy (dummybuf, p1, pagesize); + fprintf(stdout, "%s", __func__); + for (i = 0; i < 2; i++) { + int nlen; + p1 = mmap(NULL, pagesize, PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + fail_unless(p1 != MAP_FAILED); + p = (uintptr_t) p1; + fail_unless((p & pagemask) == 0); + memcpy(dummybuf, p1, pagesize); - p2 = mmap(NULL, pagesize, PROT_READ, - MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); - fail_unless (p2 != MAP_FAILED); - p = (uintptr_t) p2; - fail_unless ((p & pagemask) == 0); - memcpy (dummybuf, p2, pagesize); + p2 = mmap(NULL, pagesize, PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + fail_unless(p2 != MAP_FAILED); + p = (uintptr_t) p2; + fail_unless((p & pagemask) == 0); + memcpy(dummybuf, p2, pagesize); - munmap (p1, pagesize); - nlen = pagesize * 8; - p3 = mmap(NULL, nlen, PROT_READ, - MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); - fail_unless (p3 != MAP_FAILED); + munmap(p1, pagesize); + nlen = pagesize * 8; + p3 = mmap(NULL, nlen, PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + fail_unless(p3 != MAP_FAILED); - /* Check if the mmaped areas collide. */ - if (p3 < p2 - && (p3 + nlen) > p2) - fail_unless (0); + /* Check if the mmaped areas collide. */ + if (p3 < p2 + && (p3 + nlen) > p2) { + fail_unless(0); + } - memcpy (dummybuf, p3, pagesize); + memcpy(dummybuf, p3, pagesize); - /* Make sure we get pages aligned with the pagesize. The - target expects this. */ - p = (uintptr_t) p3; - fail_unless ((p & pagemask) == 0); - munmap (p2, pagesize); - munmap (p3, nlen); - } - fprintf(stdout, " passed\n"); + /* + * Make sure we get pages aligned with the pagesize. The + * target expects this. + */ + p = (uintptr_t) p3; + fail_unless((p & pagemask) == 0); + munmap(p2, pagesize); + munmap(p3, nlen); + } + fprintf(stdout, " passed\n"); } void check_aligned_anonymous_fixed_mmaps(void) From 2d932039980ab78b39030a7a80ad5eb7a93c894f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 9 Jul 2021 15:29:52 +0100 Subject: [PATCH 29/44] plugins: fix-up handling of internal hostaddr for 32 bit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The compiler rightly complains when we build on 32 bit that casting uint64_t into a void is a bad idea. We are really dealing with a host pointer at this point so treat it as such. This does involve a uintptr_t cast of the result of the TLB addend as we know that has to point to the host memory. Signed-off-by: Alex Bennée Reviewed-by: Richard Henderson Message-Id: <20210709143005.1554-28-alex.bennee@linaro.org> --- accel/tcg/cputlb.c | 2 +- include/qemu/plugin-memory.h | 2 +- plugins/api.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index b6d5fc6326..b4e15b6aad 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -1728,7 +1728,7 @@ bool tlb_plugin_lookup(CPUState *cpu, target_ulong addr, int mmu_idx, data->v.io.offset = (iotlbentry->addr & TARGET_PAGE_MASK) + addr; } else { data->is_io = false; - data->v.ram.hostaddr = addr + tlbe->addend; + data->v.ram.hostaddr = (void *)((uintptr_t)addr + tlbe->addend); } return true; } else { diff --git a/include/qemu/plugin-memory.h b/include/qemu/plugin-memory.h index b36def27d7..0f59226727 100644 --- a/include/qemu/plugin-memory.h +++ b/include/qemu/plugin-memory.h @@ -18,7 +18,7 @@ struct qemu_plugin_hwaddr { hwaddr offset; } io; struct { - uint64_t hostaddr; + void *hostaddr; } ram; } v; }; diff --git a/plugins/api.c b/plugins/api.c index 332e2c60e2..78b563c5c5 100644 --- a/plugins/api.c +++ b/plugins/api.c @@ -308,11 +308,11 @@ uint64_t qemu_plugin_hwaddr_phys_addr(const struct qemu_plugin_hwaddr *haddr) if (!haddr->is_io) { RAMBlock *block; ram_addr_t offset; - void *hostaddr = (void *) haddr->v.ram.hostaddr; + void *hostaddr = haddr->v.ram.hostaddr; block = qemu_ram_block_from_host(hostaddr, false, &offset); if (!block) { - error_report("Bad ram pointer %"PRIx64"", haddr->v.ram.hostaddr); + error_report("Bad host ram pointer %p", haddr->v.ram.hostaddr); abort(); } From 029aa68fdcc5a5bbf28f0044b9394dabde2b88e3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 9 Jul 2021 15:29:53 +0100 Subject: [PATCH 30/44] meson.build: move TCG plugin summary output MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Let's put it with the rest of the TCG related output with the accelerator. Signed-off-by: Alex Bennée Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Richard Henderson Message-Id: <20210709143005.1554-29-alex.bennee@linaro.org> --- meson.build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meson.build b/meson.build index 89c09ed7ea..c6dd70c400 100644 --- a/meson.build +++ b/meson.build @@ -2833,7 +2833,6 @@ summary_info += {'module support': config_host.has_key('CONFIG_MODULES')} if config_host.has_key('CONFIG_MODULES') summary_info += {'alternative module path': config_host.has_key('CONFIG_MODULE_UPGRADES')} endif -summary_info += {'plugin support': config_host.has_key('CONFIG_PLUGIN')} summary_info += {'fuzzing support': config_host.has_key('CONFIG_FUZZ')} if have_system summary_info += {'Audio drivers': config_host['CONFIG_AUDIO_DRIVERS']} @@ -2949,6 +2948,7 @@ if config_all.has_key('CONFIG_TCG') else summary_info += {'TCG backend': 'native (@0@)'.format(cpu)} endif + summary_info += {'TCG plugins': config_host.has_key('CONFIG_PLUGIN')} summary_info += {'TCG debug enabled': config_host.has_key('CONFIG_DEBUG_TCG')} endif summary_info += {'target list': ' '.join(target_dirs)} From e8575f2532e74cea6d9c750f6b4404071aa47917 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Wed, 14 Jul 2021 11:15:36 +0100 Subject: [PATCH 31/44] meson.build: relax the libdl test to one for the function dlopen MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit For the *BSD family dlopen is already part of libc so it's not a hard dependency to have a libdl.so library. Signed-off-by: Alex Bennée Reviewed-by: Paolo Bonzini Message-Id: <20210714101536.16016-1-alex.bennee@linaro.org> --- meson.build | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/meson.build b/meson.build index c6dd70c400..e5de144233 100644 --- a/meson.build +++ b/meson.build @@ -455,7 +455,10 @@ endif rt = cc.find_library('rt', required: false) libdl = not_found if 'CONFIG_PLUGIN' in config_host - libdl = cc.find_library('dl', required: true) + libdl = cc.find_library('dl', required: false) + if not cc.has_function('dlopen', dependencies: libdl) + error('dlopen not found') + endif endif libiscsi = not_found if not get_option('libiscsi').auto() or have_block From 2572ac1b224359dc2d58543d071ed977ed7c616c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Wed, 14 Jul 2021 11:16:23 +0100 Subject: [PATCH 32/44] tcg/plugins: remove some stale entries from the symbol list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The OSX linker is a lot fussier about these missing symbols. Signed-off-by: Alex Bennée Message-Id: <20210714101623.17046-1-alex.bennee@linaro.org> --- plugins/qemu-plugins.symbols | 3 --- 1 file changed, 3 deletions(-) diff --git a/plugins/qemu-plugins.symbols b/plugins/qemu-plugins.symbols index 4bdb381f48..40b4ff3821 100644 --- a/plugins/qemu-plugins.symbols +++ b/plugins/qemu-plugins.symbols @@ -8,9 +8,7 @@ qemu_plugin_register_vcpu_insn_exec_cb; qemu_plugin_register_vcpu_insn_exec_inline; qemu_plugin_register_vcpu_mem_cb; - qemu_plugin_register_vcpu_mem_haddr_cb; qemu_plugin_register_vcpu_mem_inline; - qemu_plugin_ram_addr_from_host; qemu_plugin_register_vcpu_tb_trans_cb; qemu_plugin_register_vcpu_tb_exec_cb; qemu_plugin_register_vcpu_tb_exec_inline; @@ -32,7 +30,6 @@ qemu_plugin_mem_is_store; qemu_plugin_get_hwaddr; qemu_plugin_hwaddr_is_io; - qemu_plugin_hwaddr_to_raddr; qemu_plugin_vcpu_for_each; qemu_plugin_n_vcpus; qemu_plugin_n_max_vcpus; From d1a142572d8e3cf932647120d0a72c9942667e9c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 9 Jul 2021 15:29:54 +0100 Subject: [PATCH 33/44] configure: don't allow plugins to be enabled for a non-TCG build MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Alex Bennée Reviewed-by: Richard Henderson Message-Id: <20210709143005.1554-30-alex.bennee@linaro.org> --- configure | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/configure b/configure index 4d0a2bfdd8..e2a353ce1c 100755 --- a/configure +++ b/configure @@ -1110,6 +1110,7 @@ for opt do --enable-cap-ng) cap_ng="enabled" ;; --disable-tcg) tcg="disabled" + plugins="no" ;; --enable-tcg) tcg="enabled" ;; @@ -1575,6 +1576,11 @@ for opt do esac done +# test for any invalid configuration combinations +if test "$plugins" = "yes" -a "$tcg" = "disabled"; then + error_exit "Can't enable plugins on non-TCG builds" +fi + case $git_submodules_action in update|validate) if test ! -e "$source_path/.git"; then From 5f2453ac749d2765d81ada38c29d3d10c1ef2bdf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 9 Jul 2021 15:29:55 +0100 Subject: [PATCH 34/44] configure: add an explicit static and plugins check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Moving this check earlier will make the later re-factor for enabling by default a bit neater. Signed-off-by: Alex Bennée Reviewed-by: Richard Henderson Message-Id: <20210709143005.1554-31-alex.bennee@linaro.org> --- configure | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/configure b/configure index e2a353ce1c..2c39ccf8e1 100755 --- a/configure +++ b/configure @@ -2197,11 +2197,14 @@ if test "$modules" = "no" && test "$module_upgrades" = "yes" ; then error_exit "Can't enable module-upgrades as Modules are not enabled" fi -# Static linking is not possible with modules or PIE +# Static linking is not possible with plugins, modules or PIE if test "$static" = "yes" ; then if test "$modules" = "yes" ; then error_exit "static and modules are mutually incompatible" fi + if test "$plugins" = "yes"; then + error_exit "static and plugins are mutually incompatible" + fi fi # Unconditional check for compiler __thread support @@ -3979,7 +3982,7 @@ if test "$plugins" = "yes" && "Plugin support requires dynamic linking and specifying a set of symbols " \ "that are exported to plugins. Unfortunately your linker doesn't " \ "support the flag (--dynamic-list or -exported_symbols_list) used " \ - "for this purpose. You can't build with --static." + "for this purpose." fi ######################################## From 9b8e4298517fc265b7dea93d79eb53a9b3315ddc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 9 Jul 2021 15:29:56 +0100 Subject: [PATCH 35/44] configure: stop user enabling plugins on Windows for now MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit There are some patches on the list that enable plugins on Windows but they still need some changes to be ready: https://patchew.org/QEMU/20201013002806.1447-1-luoyonggang@gmail.com/ In the meantime lets stop the user from being able to configure the support so they don't get confused by the weird linker error messages later. Signed-off-by: Alex Bennée Reviewed-by: Richard Henderson Cc: Yonggang Luo Message-Id: <20210709143005.1554-32-alex.bennee@linaro.org> --- configure | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/configure b/configure index 2c39ccf8e1..e7f2fcd02c 100755 --- a/configure +++ b/configure @@ -708,6 +708,7 @@ MINGW32*) audio_drv_list="" fi supported_os="yes" + plugins="no" pie="no" ;; GNU/kFreeBSD) @@ -1522,7 +1523,11 @@ for opt do ;; --disable-xkbcommon) xkbcommon="disabled" ;; - --enable-plugins) plugins="yes" + --enable-plugins) if test "$mingw32" = "yes"; then + error_exit "TCG plugins not currently supported on Windows platforms" + else + plugins="yes" + fi ;; --disable-plugins) plugins="no" ;; From ba4dd2aabc35bc5385739e13f14e3a10a223ede0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alex=20Benn=C3=A9e?= Date: Fri, 9 Jul 2021 15:29:57 +0100 Subject: [PATCH 36/44] tcg/plugins: enable by default for most TCG builds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Aside from a minor bloat to file size the ability to have TCG plugins has no real impact on performance unless a plugin is actively loaded. Even then the libempty.so plugin shows only a minor degradation in performance caused by the extra book keeping the TCG has to do to keep track of instructions. As it's a useful feature lets just enable it by default and reduce our testing matrix a little. We need to move our linker testing earlier so we can be sure we can enable the loader module required. As we have ruled out static & plugins in an earlier patch we can also reduce the indent a little. Signed-off-by: Alex Bennée Reviewed-by: Thomas Huth Reviewed-by: Richard Henderson Cc: Paolo Bonzini Message-Id: <20210709143005.1554-33-alex.bennee@linaro.org> --- .gitlab-ci.d/buildtest.yml | 23 ------- configure | 124 ++++++++++++++++++++----------------- docs/devel/tcg-plugins.rst | 3 +- 3 files changed, 69 insertions(+), 81 deletions(-) diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml index d9b834c848..89df51517c 100644 --- a/.gitlab-ci.d/buildtest.yml +++ b/.gitlab-ci.d/buildtest.yml @@ -354,29 +354,6 @@ build-some-softmmu: TARGETS: xtensa-softmmu arm-softmmu aarch64-softmmu alpha-softmmu MAKE_CHECK_ARGS: check-tcg -# Run check-tcg against linux-user (with plugins) -# we skip sparc64-linux-user until it has been fixed somewhat -# we skip cris-linux-user as it doesn't use the common run loop -build-user-plugins: - extends: .native_build_job_template - needs: - job: amd64-debian-user-cross-container - variables: - IMAGE: debian-all-test-cross - CONFIGURE_ARGS: --disable-tools --disable-system --enable-plugins --enable-debug-tcg --target-list-exclude=sparc64-linux-user,cris-linux-user - MAKE_CHECK_ARGS: check-tcg - timeout: 1h 30m - -build-some-softmmu-plugins: - extends: .native_build_job_template - needs: - job: amd64-debian-user-cross-container - variables: - IMAGE: debian-all-test-cross - CONFIGURE_ARGS: --disable-tools --disable-user --enable-plugins --enable-debug-tcg - TARGETS: xtensa-softmmu arm-softmmu aarch64-softmmu alpha-softmmu - MAKE_CHECK_ARGS: check-tcg - clang-system: extends: .native_build_job_template needs: diff --git a/configure b/configure index e7f2fcd02c..49b5481139 100755 --- a/configure +++ b/configure @@ -429,7 +429,7 @@ libxml2="auto" debug_mutex="no" libpmem="auto" default_devices="true" -plugins="no" +plugins="$default_feature" fuzzing="no" rng_none="no" secret_keyring="$default_feature" @@ -2209,6 +2209,8 @@ if test "$static" = "yes" ; then fi if test "$plugins" = "yes"; then error_exit "static and plugins are mutually incompatible" + else + plugins="no" fi fi @@ -3102,6 +3104,69 @@ for drv in $audio_drv_list; do esac done +########################################## +# plugin linker support probe + +if test "$plugins" != "no"; then + + ######################################### + # See if --dynamic-list is supported by the linker + + ld_dynamic_list="no" + cat > $TMPTXT < $TMPC < +void foo(void); + +void foo(void) +{ + printf("foo\n"); +} + +int main(void) +{ + foo(); + return 0; +} +EOF + + if compile_prog "" "-Wl,--dynamic-list=$TMPTXT" ; then + ld_dynamic_list="yes" + fi + + ######################################### + # See if -exported_symbols_list is supported by the linker + + ld_exported_symbols_list="no" + cat > $TMPTXT < $TMPTXT < $TMPC < -void foo(void); - -void foo(void) -{ - printf("foo\n"); -} - -int main(void) -{ - foo(); - return 0; -} -EOF - - if compile_prog "" "-Wl,--dynamic-list=$TMPTXT" ; then - ld_dynamic_list="yes" - fi -fi - -######################################### -# See if -exported_symbols_list is supported by the linker - -ld_exported_symbols_list="no" -if test "$static" = "no" ; then - cat > $TMPTXT < Date: Fri, 9 Jul 2021 15:29:58 +0100 Subject: [PATCH 37/44] contrib/plugins: enable -Wall for building plugins MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lets spot the obvious errors. Signed-off-by: Alex Bennée Reviewed-by: Richard Henderson Reviewed-by: Thomas Huth Message-Id: <20210709143005.1554-34-alex.bennee@linaro.org> --- contrib/plugins/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/plugins/Makefile b/contrib/plugins/Makefile index b9d7935e5e..3431bc1ce9 100644 --- a/contrib/plugins/Makefile +++ b/contrib/plugins/Makefile @@ -24,7 +24,7 @@ SONAMES := $(addsuffix .so,$(addprefix lib,$(NAMES))) # The main QEMU uses Glib extensively so it's perfectly fine to use it # in plugins (which many example do). CFLAGS = $(GLIB_CFLAGS) -CFLAGS += -fPIC +CFLAGS += -fPIC -Wall $(filter -W%, $(QEMU_CFLAGS)) CFLAGS += $(if $(findstring no-psabi,$(QEMU_CFLAGS)),-Wpsabi) CFLAGS += -I$(SRC_PATH)/include/qemu From 3d7caf145e69b61d17f7cada8713f37af2aaebf1 Mon Sep 17 00:00:00 2001 From: Alexandre Iooss Date: Fri, 9 Jul 2021 15:29:59 +0100 Subject: [PATCH 38/44] contrib/plugins: add execlog to log instruction execution and memory access MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Log instruction execution and memory access to a file. This plugin can be used for reverse engineering or for side-channel analysis using QEMU. Signed-off-by: Alexandre Iooss Signed-off-by: Alex Bennée Reviewed-by: Alex Bennée Message-Id: <20210702081307.1653644-2-erdnaxe@crans.org> Message-Id: <20210709143005.1554-35-alex.bennee@linaro.org> --- MAINTAINERS | 1 + contrib/plugins/Makefile | 1 + contrib/plugins/execlog.c | 153 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 155 insertions(+) create mode 100644 contrib/plugins/execlog.c diff --git a/MAINTAINERS b/MAINTAINERS index c340bb02b0..83b55030ad 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3017,6 +3017,7 @@ F: include/tcg/ TCG Plugins M: Alex Bennée +R: Alexandre Iooss S: Maintained F: docs/devel/tcg-plugins.rst F: plugins/ diff --git a/contrib/plugins/Makefile b/contrib/plugins/Makefile index 3431bc1ce9..3c9209b6b0 100644 --- a/contrib/plugins/Makefile +++ b/contrib/plugins/Makefile @@ -13,6 +13,7 @@ include $(BUILD_DIR)/config-host.mak VPATH += $(SRC_PATH)/contrib/plugins NAMES := +NAMES += execlog NAMES += hotblocks NAMES += hotpages NAMES += howvec diff --git a/contrib/plugins/execlog.c b/contrib/plugins/execlog.c new file mode 100644 index 0000000000..2de9f0d7d4 --- /dev/null +++ b/contrib/plugins/execlog.c @@ -0,0 +1,153 @@ +/* + * Copyright (C) 2021, Alexandre Iooss + * + * Log instruction execution with memory access. + * + * License: GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + */ +#include +#include +#include +#include +#include +#include + +#include + +QEMU_PLUGIN_EXPORT int qemu_plugin_version = QEMU_PLUGIN_VERSION; + +/* Store last executed instruction on each vCPU as a GString */ +GArray *last_exec; + +/** + * Add memory read or write information to current instruction log + */ +static void vcpu_mem(unsigned int cpu_index, qemu_plugin_meminfo_t info, + uint64_t vaddr, void *udata) +{ + GString *s; + + /* Find vCPU in array */ + g_assert(cpu_index < last_exec->len); + s = g_array_index(last_exec, GString *, cpu_index); + + /* Indicate type of memory access */ + if (qemu_plugin_mem_is_store(info)) { + g_string_append(s, ", store"); + } else { + g_string_append(s, ", load"); + } + + /* If full system emulation log physical address and device name */ + struct qemu_plugin_hwaddr *hwaddr = qemu_plugin_get_hwaddr(info, vaddr); + if (hwaddr) { + uint64_t addr = qemu_plugin_hwaddr_phys_addr(hwaddr); + const char *name = qemu_plugin_hwaddr_device_name(hwaddr); + g_string_append_printf(s, ", 0x%08"PRIx64", %s", addr, name); + } else { + g_string_append_printf(s, ", 0x%08"PRIx64, vaddr); + } +} + +/** + * Log instruction execution + */ +static void vcpu_insn_exec(unsigned int cpu_index, void *udata) +{ + GString *s; + + /* Find or create vCPU in array */ + while (cpu_index >= last_exec->len) { + s = g_string_new(NULL); + g_array_append_val(last_exec, s); + } + s = g_array_index(last_exec, GString *, cpu_index); + + /* Print previous instruction in cache */ + if (s->len) { + qemu_plugin_outs(s->str); + qemu_plugin_outs("s\n"); + } + + /* Store new instruction in cache */ + /* vcpu_mem will add memory access information to last_exec */ + g_string_printf(s, "%u, ", cpu_index); + g_string_append(s, (char *)udata); +} + +/** + * On translation block new translation + * + * QEMU convert code by translation block (TB). By hooking here we can then hook + * a callback on each instruction and memory access. + */ +static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb) +{ + struct qemu_plugin_insn *insn; + uint64_t insn_vaddr; + uint32_t insn_opcode; + char *insn_disas; + + size_t n = qemu_plugin_tb_n_insns(tb); + for (size_t i = 0; i < n; i++) { + /* + * `insn` is shared between translations in QEMU, copy needed data here. + * `output` is never freed as it might be used multiple times during + * the emulation lifetime. + * We only consider the first 32 bits of the instruction, this may be + * a limitation for CISC architectures. + */ + insn = qemu_plugin_tb_get_insn(tb, i); + insn_vaddr = qemu_plugin_insn_vaddr(insn); + insn_opcode = *((uint32_t *)qemu_plugin_insn_data(insn)); + insn_disas = qemu_plugin_insn_disas(insn); + char *output = g_strdup_printf("0x%"PRIx64", 0x%"PRIx32", \"%s\"", + insn_vaddr, insn_opcode, insn_disas); + + /* Register callback on memory read or write */ + qemu_plugin_register_vcpu_mem_cb(insn, vcpu_mem, + QEMU_PLUGIN_CB_NO_REGS, + QEMU_PLUGIN_MEM_RW, NULL); + + /* Register callback on instruction */ + qemu_plugin_register_vcpu_insn_exec_cb(insn, vcpu_insn_exec, + QEMU_PLUGIN_CB_NO_REGS, output); + } +} + +/** + * On plugin exit, print last instruction in cache + */ +static void plugin_exit(qemu_plugin_id_t id, void *p) +{ + guint i; + GString *s; + for (i = 0; i < last_exec->len; i++) { + s = g_array_index(last_exec, GString *, i); + if (s->str) { + qemu_plugin_outs(s->str); + qemu_plugin_outs("\n"); + } + } +} + +/** + * Install the plugin + */ +QEMU_PLUGIN_EXPORT int qemu_plugin_install(qemu_plugin_id_t id, + const qemu_info_t *info, int argc, + char **argv) +{ + /* + * Initialize dynamic array to cache vCPU instruction. In user mode + * we don't know the size before emulation. + */ + last_exec = g_array_new(FALSE, FALSE, sizeof(GString *)); + + /* Register translation block and exit callbacks */ + qemu_plugin_register_vcpu_tb_trans_cb(id, vcpu_tb_trans); + qemu_plugin_register_atexit_cb(id, plugin_exit, NULL); + + return 0; +} From 307ce0aaeb5799d05f63b76a91135466f6b15302 Mon Sep 17 00:00:00 2001 From: Alexandre Iooss Date: Fri, 9 Jul 2021 15:30:00 +0100 Subject: [PATCH 39/44] docs/devel: tcg-plugins: add execlog plugin description MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This adds description of the execlog TCG plugin with an example. Signed-off-by: Alexandre Iooss Signed-off-by: Alex Bennée Message-Id: <20210702081307.1653644-3-erdnaxe@crans.org> Message-Id: <20210709143005.1554-36-alex.bennee@linaro.org> --- docs/devel/tcg-plugins.rst | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/devel/tcg-plugins.rst b/docs/devel/tcg-plugins.rst index 0cd77c77d2..179867e9c1 100644 --- a/docs/devel/tcg-plugins.rst +++ b/docs/devel/tcg-plugins.rst @@ -320,3 +320,27 @@ the user to see what hardware is accessed how often. It has a number of options: off:0000001c, 1, 2 off:00000020, 1, 2 ... + +- contrib/plugins/execlog.c + +The execlog tool traces executed instructions with memory access. It can be used +for debugging and security analysis purposes. +Please be aware that this will generate a lot of output. + +The plugin takes no argument:: + + qemu-system-arm $(QEMU_ARGS) \ + -plugin ./contrib/plugins/libexeclog.so -d plugin + +which will output an execution trace following this structure:: + + # vCPU, vAddr, opcode, disassembly[, load/store, memory addr, device]... + 0, 0xa12, 0xf8012400, "movs r4, #0" + 0, 0xa14, 0xf87f42b4, "cmp r4, r6" + 0, 0xa16, 0xd206, "bhs #0xa26" + 0, 0xa18, 0xfff94803, "ldr r0, [pc, #0xc]", load, 0x00010a28, RAM + 0, 0xa1a, 0xf989f000, "bl #0xd30" + 0, 0xd30, 0xfff9b510, "push {r4, lr}", store, 0x20003ee0, RAM, store, 0x20003ee4, RAM + 0, 0xd32, 0xf9893014, "adds r0, #0x14" + 0, 0xd34, 0xf9c8f000, "bl #0x10c8" + 0, 0x10c8, 0xfff96c43, "ldr r3, [r0, #0x44]", load, 0x200000e4, RAM From e2c5557ce1329f58efd8e1f27c3548acaa82e196 Mon Sep 17 00:00:00 2001 From: Mahmoud Mandour Date: Fri, 9 Jul 2021 15:30:01 +0100 Subject: [PATCH 40/44] plugins: Added a new cache modelling plugin MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Added a cache modelling plugin that uses a static configuration used in many of the commercial microprocessors and uses random eviction policy. The purpose of the plugin is to identify the most cache-thrashing instructions for both instruction cache and data cache. Signed-off-by: Mahmoud Mandour Signed-off-by: Alex Bennée Message-Id: <20210623125458.450462-2-ma.mandourr@gmail.com> Message-Id: <20210709143005.1554-37-alex.bennee@linaro.org> --- contrib/plugins/Makefile | 1 + contrib/plugins/cache.c | 419 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 420 insertions(+) create mode 100644 contrib/plugins/cache.c diff --git a/contrib/plugins/Makefile b/contrib/plugins/Makefile index 3c9209b6b0..54ac5ccd9f 100644 --- a/contrib/plugins/Makefile +++ b/contrib/plugins/Makefile @@ -19,6 +19,7 @@ NAMES += hotpages NAMES += howvec NAMES += lockstep NAMES += hwprofile +NAMES += cache SONAMES := $(addsuffix .so,$(addprefix lib,$(NAMES))) diff --git a/contrib/plugins/cache.c b/contrib/plugins/cache.c new file mode 100644 index 0000000000..e9955cdc3a --- /dev/null +++ b/contrib/plugins/cache.c @@ -0,0 +1,419 @@ +/* + * Copyright (C) 2021, Mahmoud Mandour + * + * License: GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + */ + +#include +#include +#include + +#include + +QEMU_PLUGIN_EXPORT int qemu_plugin_version = QEMU_PLUGIN_VERSION; + +static enum qemu_plugin_mem_rw rw = QEMU_PLUGIN_MEM_RW; + +static GHashTable *miss_ht; + +static GMutex mtx; +static GRand *rng; + +static int limit; +static bool sys; + +static uint64_t dmem_accesses; +static uint64_t dmisses; + +static uint64_t imem_accesses; +static uint64_t imisses; + +/* + * A CacheSet is a set of cache blocks. A memory block that maps to a set can be + * put in any of the blocks inside the set. The number of block per set is + * called the associativity (assoc). + * + * Each block contains the the stored tag and a valid bit. Since this is not + * a functional simulator, the data itself is not stored. We only identify + * whether a block is in the cache or not by searching for its tag. + * + * In order to search for memory data in the cache, the set identifier and tag + * are extracted from the address and the set is probed to see whether a tag + * match occur. + * + * An address is logically divided into three portions: The block offset, + * the set number, and the tag. + * + * The set number is used to identify the set in which the block may exist. + * The tag is compared against all the tags of a set to search for a match. If a + * match is found, then the access is a hit. + */ + +typedef struct { + uint64_t tag; + bool valid; +} CacheBlock; + +typedef struct { + CacheBlock *blocks; +} CacheSet; + +typedef struct { + CacheSet *sets; + int num_sets; + int cachesize; + int assoc; + int blksize_shift; + uint64_t set_mask; + uint64_t tag_mask; +} Cache; + +typedef struct { + char *disas_str; + const char *symbol; + uint64_t addr; + uint64_t dmisses; + uint64_t imisses; +} InsnData; + +Cache *dcache, *icache; + +static int pow_of_two(int num) +{ + g_assert((num & (num - 1)) == 0); + int ret = 0; + while (num /= 2) { + ret++; + } + return ret; +} + +static inline uint64_t extract_tag(Cache *cache, uint64_t addr) +{ + return addr & cache->tag_mask; +} + +static inline uint64_t extract_set(Cache *cache, uint64_t addr) +{ + return (addr & cache->set_mask) >> cache->blksize_shift; +} + +static Cache *cache_init(int blksize, int assoc, int cachesize) +{ + Cache *cache; + int i; + uint64_t blk_mask; + + cache = g_new(Cache, 1); + cache->assoc = assoc; + cache->cachesize = cachesize; + cache->num_sets = cachesize / (blksize * assoc); + cache->sets = g_new(CacheSet, cache->num_sets); + cache->blksize_shift = pow_of_two(blksize); + + for (i = 0; i < cache->num_sets; i++) { + cache->sets[i].blocks = g_new0(CacheBlock, assoc); + } + + blk_mask = blksize - 1; + cache->set_mask = ((cache->num_sets - 1) << cache->blksize_shift); + cache->tag_mask = ~(cache->set_mask | blk_mask); + return cache; +} + +static int get_invalid_block(Cache *cache, uint64_t set) +{ + int i; + + for (i = 0; i < cache->assoc; i++) { + if (!cache->sets[set].blocks[i].valid) { + return i; + } + } + + return -1; +} + +static int get_replaced_block(Cache *cache) +{ + return g_rand_int_range(rng, 0, cache->assoc); +} + +static bool in_cache(Cache *cache, uint64_t addr) +{ + int i; + uint64_t tag, set; + + tag = extract_tag(cache, addr); + set = extract_set(cache, addr); + + for (i = 0; i < cache->assoc; i++) { + if (cache->sets[set].blocks[i].tag == tag && + cache->sets[set].blocks[i].valid) { + return true; + } + } + + return false; +} + +/** + * access_cache(): Simulate a cache access + * @cache: The cache under simulation + * @addr: The address of the requested memory location + * + * Returns true if the requsted data is hit in the cache and false when missed. + * The cache is updated on miss for the next access. + */ +static bool access_cache(Cache *cache, uint64_t addr) +{ + uint64_t tag, set; + int replaced_blk; + + if (in_cache(cache, addr)) { + return true; + } + + tag = extract_tag(cache, addr); + set = extract_set(cache, addr); + + replaced_blk = get_invalid_block(cache, set); + + if (replaced_blk == -1) { + replaced_blk = get_replaced_block(cache); + } + + cache->sets[set].blocks[replaced_blk].tag = tag; + cache->sets[set].blocks[replaced_blk].valid = true; + + return false; +} + +static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info, + uint64_t vaddr, void *userdata) +{ + uint64_t effective_addr; + struct qemu_plugin_hwaddr *hwaddr; + InsnData *insn; + + g_mutex_lock(&mtx); + hwaddr = qemu_plugin_get_hwaddr(info, vaddr); + if (hwaddr && qemu_plugin_hwaddr_is_io(hwaddr)) { + g_mutex_unlock(&mtx); + return; + } + + effective_addr = hwaddr ? qemu_plugin_hwaddr_phys_addr(hwaddr) : vaddr; + + if (!access_cache(dcache, effective_addr)) { + insn = (InsnData *) userdata; + insn->dmisses++; + dmisses++; + } + dmem_accesses++; + g_mutex_unlock(&mtx); +} + +static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata) +{ + uint64_t insn_addr; + InsnData *insn; + + g_mutex_lock(&mtx); + insn_addr = ((InsnData *) userdata)->addr; + + if (!access_cache(icache, insn_addr)) { + insn = (InsnData *) userdata; + insn->imisses++; + imisses++; + } + imem_accesses++; + g_mutex_unlock(&mtx); +} + +static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb) +{ + size_t n_insns; + size_t i; + InsnData *data; + + n_insns = qemu_plugin_tb_n_insns(tb); + for (i = 0; i < n_insns; i++) { + struct qemu_plugin_insn *insn = qemu_plugin_tb_get_insn(tb, i); + uint64_t effective_addr; + + if (sys) { + effective_addr = (uint64_t) qemu_plugin_insn_haddr(insn); + } else { + effective_addr = (uint64_t) qemu_plugin_insn_vaddr(insn); + } + + /* + * Instructions might get translated multiple times, we do not create + * new entries for those instructions. Instead, we fetch the same + * entry from the hash table and register it for the callback again. + */ + g_mutex_lock(&mtx); + data = g_hash_table_lookup(miss_ht, GUINT_TO_POINTER(effective_addr)); + if (data == NULL) { + data = g_new0(InsnData, 1); + data->disas_str = qemu_plugin_insn_disas(insn); + data->symbol = qemu_plugin_insn_symbol(insn); + data->addr = effective_addr; + g_hash_table_insert(miss_ht, GUINT_TO_POINTER(effective_addr), + (gpointer) data); + } + g_mutex_unlock(&mtx); + + qemu_plugin_register_vcpu_mem_cb(insn, vcpu_mem_access, + QEMU_PLUGIN_CB_NO_REGS, + rw, data); + + qemu_plugin_register_vcpu_insn_exec_cb(insn, vcpu_insn_exec, + QEMU_PLUGIN_CB_NO_REGS, data); + } +} + +static void insn_free(gpointer data) +{ + InsnData *insn = (InsnData *) data; + g_free(insn->disas_str); + g_free(insn); +} + +static void cache_free(Cache *cache) +{ + for (int i = 0; i < cache->num_sets; i++) { + g_free(cache->sets[i].blocks); + } + + g_free(cache->sets); + g_free(cache); +} + +static int dcmp(gconstpointer a, gconstpointer b) +{ + InsnData *insn_a = (InsnData *) a; + InsnData *insn_b = (InsnData *) b; + + return insn_a->dmisses < insn_b->dmisses ? 1 : -1; +} + +static int icmp(gconstpointer a, gconstpointer b) +{ + InsnData *insn_a = (InsnData *) a; + InsnData *insn_b = (InsnData *) b; + + return insn_a->imisses < insn_b->imisses ? 1 : -1; +} + +static void log_stats() +{ + g_autoptr(GString) rep = g_string_new(""); + g_string_append_printf(rep, + "Data accesses: %lu, Misses: %lu\nMiss rate: %lf%%\n\n", + dmem_accesses, + dmisses, + ((double) dmisses / (double) dmem_accesses) * 100.0); + + g_string_append_printf(rep, + "Instruction accesses: %lu, Misses: %lu\nMiss rate: %lf%%\n\n", + imem_accesses, + imisses, + ((double) imisses / (double) imem_accesses) * 100.0); + + qemu_plugin_outs(rep->str); +} + +static void log_top_insns() +{ + int i; + GList *curr, *miss_insns; + InsnData *insn; + + miss_insns = g_hash_table_get_values(miss_ht); + miss_insns = g_list_sort(miss_insns, dcmp); + g_autoptr(GString) rep = g_string_new(""); + g_string_append_printf(rep, "%s", "address, data misses, instruction\n"); + + for (curr = miss_insns, i = 0; curr && i < limit; i++, curr = curr->next) { + insn = (InsnData *) curr->data; + g_string_append_printf(rep, "0x%" PRIx64, insn->addr); + if (insn->symbol) { + g_string_append_printf(rep, " (%s)", insn->symbol); + } + g_string_append_printf(rep, ", %ld, %s\n", insn->dmisses, + insn->disas_str); + } + + miss_insns = g_list_sort(miss_insns, icmp); + g_string_append_printf(rep, "%s", "\naddress, fetch misses, instruction\n"); + + for (curr = miss_insns, i = 0; curr && i < limit; i++, curr = curr->next) { + insn = (InsnData *) curr->data; + g_string_append_printf(rep, "0x%" PRIx64, insn->addr); + if (insn->symbol) { + g_string_append_printf(rep, " (%s)", insn->symbol); + } + g_string_append_printf(rep, ", %ld, %s\n", insn->imisses, + insn->disas_str); + } + + qemu_plugin_outs(rep->str); + g_list_free(miss_insns); +} + +static void plugin_exit(qemu_plugin_id_t id, void *p) +{ + log_stats(); + log_top_insns(); + + cache_free(dcache); + cache_free(icache); + + g_hash_table_destroy(miss_ht); +} + +QEMU_PLUGIN_EXPORT +int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info, + int argc, char **argv) +{ + int i; + int iassoc, iblksize, icachesize; + int dassoc, dblksize, dcachesize; + + limit = 32; + sys = info->system_emulation; + + dassoc = 8; + dblksize = 64; + dcachesize = dblksize * dassoc * 32; + + iassoc = 8; + iblksize = 64; + icachesize = iblksize * iassoc * 32; + + + for (i = 0; i < argc; i++) { + char *opt = argv[i]; + if (g_str_has_prefix(opt, "limit=")) { + limit = g_ascii_strtoll(opt + 6, NULL, 10); + } else { + fprintf(stderr, "option parsing failed: %s\n", opt); + return -1; + } + } + + dcache = cache_init(dblksize, dassoc, dcachesize); + icache = cache_init(iblksize, iassoc, icachesize); + + rng = g_rand_new(); + + qemu_plugin_register_vcpu_tb_trans_cb(id, vcpu_tb_trans); + qemu_plugin_register_atexit_cb(id, plugin_exit, NULL); + + miss_ht = g_hash_table_new_full(NULL, g_direct_equal, NULL, insn_free); + + return 0; +} From 86ae3a1daad3c5b7ca039770d505574c08647e07 Mon Sep 17 00:00:00 2001 From: Mahmoud Mandour Date: Fri, 9 Jul 2021 15:30:02 +0100 Subject: [PATCH 41/44] plugins/cache: Enable cache parameterization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enabled configuring both icache and dcache parameters using plugin parameters. Signed-off-by: Mahmoud Mandour Signed-off-by: Alex Bennée Message-Id: <20210623125458.450462-3-ma.mandourr@gmail.com> Message-Id: <20210709143005.1554-38-alex.bennee@linaro.org> --- contrib/plugins/cache.c | 47 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/contrib/plugins/cache.c b/contrib/plugins/cache.c index e9955cdc3a..b550ef31b0 100644 --- a/contrib/plugins/cache.c +++ b/contrib/plugins/cache.c @@ -99,8 +99,28 @@ static inline uint64_t extract_set(Cache *cache, uint64_t addr) return (addr & cache->set_mask) >> cache->blksize_shift; } +static const char *cache_config_error(int blksize, int assoc, int cachesize) +{ + if (cachesize % blksize != 0) { + return "cache size must be divisible by block size"; + } else if (cachesize % (blksize * assoc) != 0) { + return "cache size must be divisible by set size (assoc * block size)"; + } else { + return NULL; + } +} + +static bool bad_cache_params(int blksize, int assoc, int cachesize) +{ + return (cachesize % blksize) != 0 || (cachesize % (blksize * assoc) != 0); +} + static Cache *cache_init(int blksize, int assoc, int cachesize) { + if (bad_cache_params(blksize, assoc, cachesize)) { + return NULL; + } + Cache *cache; int i; uint64_t blk_mask; @@ -397,7 +417,19 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info, for (i = 0; i < argc; i++) { char *opt = argv[i]; - if (g_str_has_prefix(opt, "limit=")) { + if (g_str_has_prefix(opt, "iblksize=")) { + iblksize = g_ascii_strtoll(opt + 9, NULL, 10); + } else if (g_str_has_prefix(opt, "iassoc=")) { + iassoc = g_ascii_strtoll(opt + 7, NULL, 10); + } else if (g_str_has_prefix(opt, "icachesize=")) { + icachesize = g_ascii_strtoll(opt + 11, NULL, 10); + } else if (g_str_has_prefix(opt, "dblksize=")) { + dblksize = g_ascii_strtoll(opt + 9, NULL, 10); + } else if (g_str_has_prefix(opt, "dassoc=")) { + dassoc = g_ascii_strtoll(opt + 7, NULL, 10); + } else if (g_str_has_prefix(opt, "dcachesize=")) { + dcachesize = g_ascii_strtoll(opt + 11, NULL, 10); + } else if (g_str_has_prefix(opt, "limit=")) { limit = g_ascii_strtoll(opt + 6, NULL, 10); } else { fprintf(stderr, "option parsing failed: %s\n", opt); @@ -406,7 +438,20 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info, } dcache = cache_init(dblksize, dassoc, dcachesize); + if (!dcache) { + const char *err = cache_config_error(dblksize, dassoc, dcachesize); + fprintf(stderr, "dcache cannot be constructed from given parameters\n"); + fprintf(stderr, "%s\n", err); + return -1; + } + icache = cache_init(iblksize, iassoc, icachesize); + if (!icache) { + const char *err = cache_config_error(iblksize, iassoc, icachesize); + fprintf(stderr, "icache cannot be constructed from given parameters\n"); + fprintf(stderr, "%s\n", err); + return -1; + } rng = g_rand_new(); From 40c4a5533f2d6f268947acdb3e506d81a53ebe66 Mon Sep 17 00:00:00 2001 From: Mahmoud Mandour Date: Fri, 9 Jul 2021 15:30:03 +0100 Subject: [PATCH 42/44] plugins/cache: Added FIFO and LRU eviction policies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implemented FIFO and LRU eviction policies. Now one of the three eviction policies can be chosen as an argument. On not specifying an argument, LRU is used by default. Signed-off-by: Mahmoud Mandour Signed-off-by: Alex Bennée Message-Id: <20210623125458.450462-4-ma.mandourr@gmail.com> Message-Id: <20210709143005.1554-39-alex.bennee@linaro.org> --- contrib/plugins/cache.c | 203 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 190 insertions(+), 13 deletions(-) diff --git a/contrib/plugins/cache.c b/contrib/plugins/cache.c index b550ef31b0..bf0d2f6097 100644 --- a/contrib/plugins/cache.c +++ b/contrib/plugins/cache.c @@ -29,6 +29,14 @@ static uint64_t dmisses; static uint64_t imem_accesses; static uint64_t imisses; +enum EvictionPolicy { + LRU, + FIFO, + RAND, +}; + +enum EvictionPolicy policy; + /* * A CacheSet is a set of cache blocks. A memory block that maps to a set can be * put in any of the blocks inside the set. The number of block per set is @@ -48,6 +56,8 @@ static uint64_t imisses; * The set number is used to identify the set in which the block may exist. * The tag is compared against all the tags of a set to search for a match. If a * match is found, then the access is a hit. + * + * The CacheSet also contains bookkeaping information about eviction details. */ typedef struct { @@ -57,6 +67,9 @@ typedef struct { typedef struct { CacheBlock *blocks; + uint64_t *lru_priorities; + uint64_t lru_gen_counter; + GQueue *fifo_queue; } CacheSet; typedef struct { @@ -77,6 +90,12 @@ typedef struct { uint64_t imisses; } InsnData; +void (*update_hit)(Cache *cache, int set, int blk); +void (*update_miss)(Cache *cache, int set, int blk); + +void (*metadata_init)(Cache *cache); +void (*metadata_destroy)(Cache *cache); + Cache *dcache, *icache; static int pow_of_two(int num) @@ -89,6 +108,103 @@ static int pow_of_two(int num) return ret; } +/* + * LRU evection policy: For each set, a generation counter is maintained + * alongside a priority array. + * + * On each set access, the generation counter is incremented. + * + * On a cache hit: The hit-block is assigned the current generation counter, + * indicating that it is the most recently used block. + * + * On a cache miss: The block with the least priority is searched and replaced + * with the newly-cached block, of which the priority is set to the current + * generation number. + */ + +static void lru_priorities_init(Cache *cache) +{ + int i; + + for (i = 0; i < cache->num_sets; i++) { + cache->sets[i].lru_priorities = g_new0(uint64_t, cache->assoc); + cache->sets[i].lru_gen_counter = 0; + } +} + +static void lru_update_blk(Cache *cache, int set_idx, int blk_idx) +{ + CacheSet *set = &cache->sets[set_idx]; + set->lru_priorities[blk_idx] = cache->sets[set_idx].lru_gen_counter; + set->lru_gen_counter++; +} + +static int lru_get_lru_block(Cache *cache, int set_idx) +{ + int i, min_idx, min_priority; + + min_priority = cache->sets[set_idx].lru_priorities[0]; + min_idx = 0; + + for (i = 1; i < cache->assoc; i++) { + if (cache->sets[set_idx].lru_priorities[i] < min_priority) { + min_priority = cache->sets[set_idx].lru_priorities[i]; + min_idx = i; + } + } + return min_idx; +} + +static void lru_priorities_destroy(Cache *cache) +{ + int i; + + for (i = 0; i < cache->num_sets; i++) { + g_free(cache->sets[i].lru_priorities); + } +} + +/* + * FIFO eviction policy: a FIFO queue is maintained for each CacheSet that + * stores accesses to the cache. + * + * On a compulsory miss: The block index is enqueued to the fifo_queue to + * indicate that it's the latest cached block. + * + * On a conflict miss: The first-in block is removed from the cache and the new + * block is put in its place and enqueued to the FIFO queue. + */ + +static void fifo_init(Cache *cache) +{ + int i; + + for (i = 0; i < cache->num_sets; i++) { + cache->sets[i].fifo_queue = g_queue_new(); + } +} + +static int fifo_get_first_block(Cache *cache, int set) +{ + GQueue *q = cache->sets[set].fifo_queue; + return GPOINTER_TO_INT(g_queue_pop_tail(q)); +} + +static void fifo_update_on_miss(Cache *cache, int set, int blk_idx) +{ + GQueue *q = cache->sets[set].fifo_queue; + g_queue_push_head(q, GINT_TO_POINTER(blk_idx)); +} + +static void fifo_destroy(Cache *cache) +{ + int i; + + for (i = 0; i < cache->assoc; i++) { + g_queue_free(cache->sets[i].fifo_queue); + } +} + static inline uint64_t extract_tag(Cache *cache, uint64_t addr) { return addr & cache->tag_mask; @@ -139,6 +255,11 @@ static Cache *cache_init(int blksize, int assoc, int cachesize) blk_mask = blksize - 1; cache->set_mask = ((cache->num_sets - 1) << cache->blksize_shift); cache->tag_mask = ~(cache->set_mask | blk_mask); + + if (metadata_init) { + metadata_init(cache); + } + return cache; } @@ -155,12 +276,21 @@ static int get_invalid_block(Cache *cache, uint64_t set) return -1; } -static int get_replaced_block(Cache *cache) +static int get_replaced_block(Cache *cache, int set) { - return g_rand_int_range(rng, 0, cache->assoc); + switch (policy) { + case RAND: + return g_rand_int_range(rng, 0, cache->assoc); + case LRU: + return lru_get_lru_block(cache, set); + case FIFO: + return fifo_get_first_block(cache, set); + default: + g_assert_not_reached(); + } } -static bool in_cache(Cache *cache, uint64_t addr) +static int in_cache(Cache *cache, uint64_t addr) { int i; uint64_t tag, set; @@ -171,11 +301,11 @@ static bool in_cache(Cache *cache, uint64_t addr) for (i = 0; i < cache->assoc; i++) { if (cache->sets[set].blocks[i].tag == tag && cache->sets[set].blocks[i].valid) { - return true; + return i; } } - return false; + return -1; } /** @@ -188,20 +318,28 @@ static bool in_cache(Cache *cache, uint64_t addr) */ static bool access_cache(Cache *cache, uint64_t addr) { + int hit_blk, replaced_blk; uint64_t tag, set; - int replaced_blk; - - if (in_cache(cache, addr)) { - return true; - } tag = extract_tag(cache, addr); set = extract_set(cache, addr); + hit_blk = in_cache(cache, addr); + if (hit_blk != -1) { + if (update_hit) { + update_hit(cache, set, hit_blk); + } + return true; + } + replaced_blk = get_invalid_block(cache, set); if (replaced_blk == -1) { - replaced_blk = get_replaced_block(cache); + replaced_blk = get_replaced_block(cache, set); + } + + if (update_miss) { + update_miss(cache, set, replaced_blk); } cache->sets[set].blocks[replaced_blk].tag = tag; @@ -308,6 +446,10 @@ static void cache_free(Cache *cache) g_free(cache->sets[i].blocks); } + if (metadata_destroy) { + metadata_destroy(cache); + } + g_free(cache->sets); g_free(cache); } @@ -395,6 +537,28 @@ static void plugin_exit(qemu_plugin_id_t id, void *p) g_hash_table_destroy(miss_ht); } +static void policy_init() +{ + switch (policy) { + case LRU: + update_hit = lru_update_blk; + update_miss = lru_update_blk; + metadata_init = lru_priorities_init; + metadata_destroy = lru_priorities_destroy; + break; + case FIFO: + update_miss = fifo_update_on_miss; + metadata_init = fifo_init; + metadata_destroy = fifo_destroy; + break; + case RAND: + rng = g_rand_new(); + break; + default: + g_assert_not_reached(); + } +} + QEMU_PLUGIN_EXPORT int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info, int argc, char **argv) @@ -414,6 +578,7 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info, iblksize = 64; icachesize = iblksize * iassoc * 32; + policy = LRU; for (i = 0; i < argc; i++) { char *opt = argv[i]; @@ -431,12 +596,26 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info, dcachesize = g_ascii_strtoll(opt + 11, NULL, 10); } else if (g_str_has_prefix(opt, "limit=")) { limit = g_ascii_strtoll(opt + 6, NULL, 10); + } else if (g_str_has_prefix(opt, "evict=")) { + gchar *p = opt + 6; + if (g_strcmp0(p, "rand") == 0) { + policy = RAND; + } else if (g_strcmp0(p, "lru") == 0) { + policy = LRU; + } else if (g_strcmp0(p, "fifo") == 0) { + policy = FIFO; + } else { + fprintf(stderr, "invalid eviction policy: %s\n", opt); + return -1; + } } else { fprintf(stderr, "option parsing failed: %s\n", opt); return -1; } } + policy_init(); + dcache = cache_init(dblksize, dassoc, dcachesize); if (!dcache) { const char *err = cache_config_error(dblksize, dassoc, dcachesize); @@ -453,8 +632,6 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info, return -1; } - rng = g_rand_new(); - qemu_plugin_register_vcpu_tb_trans_cb(id, vcpu_tb_trans); qemu_plugin_register_atexit_cb(id, plugin_exit, NULL); From 4c125f3b7591758ebd96c3ac3ff469480b7aaea2 Mon Sep 17 00:00:00 2001 From: Mahmoud Mandour Date: Fri, 9 Jul 2021 15:30:04 +0100 Subject: [PATCH 43/44] docs/devel: Added cache plugin to the plugins docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Mahmoud Mandour Signed-off-by: Alex Bennée Message-Id: <20210628053808.17422-1-ma.mandourr@gmail.com> Message-Id: <20210709143005.1554-40-alex.bennee@linaro.org> --- docs/devel/tcg-plugins.rst | 59 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/docs/devel/tcg-plugins.rst b/docs/devel/tcg-plugins.rst index 179867e9c1..7e54f12837 100644 --- a/docs/devel/tcg-plugins.rst +++ b/docs/devel/tcg-plugins.rst @@ -344,3 +344,62 @@ which will output an execution trace following this structure:: 0, 0xd32, 0xf9893014, "adds r0, #0x14" 0, 0xd34, 0xf9c8f000, "bl #0x10c8" 0, 0x10c8, 0xfff96c43, "ldr r3, [r0, #0x44]", load, 0x200000e4, RAM + +- contrib/plugins/cache + +Cache modelling plugin that measures the performance of a given cache +configuration when a given working set is run:: + + qemu-x86_64 -plugin ./contrib/plugins/libcache.so \ + -d plugin -D cache.log ./tests/tcg/x86_64-linux-user/float_convs + +will report the following:: + + Data accesses: 996479, Misses: 507 + Miss rate: 0.050879% + + Instruction accesses: 2641737, Misses: 18617 + Miss rate: 0.704726% + + address, data misses, instruction + 0x424f1e (_int_malloc), 109, movq %rax, 8(%rcx) + 0x41f395 (_IO_default_xsputn), 49, movb %dl, (%rdi, %rax) + 0x42584d (ptmalloc_init.part.0), 33, movaps %xmm0, (%rax) + 0x454d48 (__tunables_init), 20, cmpb $0, (%r8) + ... + + address, fetch misses, instruction + 0x4160a0 (__vfprintf_internal), 744, movl $1, %ebx + 0x41f0a0 (_IO_setb), 744, endbr64 + 0x415882 (__vfprintf_internal), 744, movq %r12, %rdi + 0x4268a0 (__malloc), 696, andq $0xfffffffffffffff0, %rax + ... + +The plugin has a number of arguments, all of them are optional: + + * arg="limit=N" + + Print top N icache and dcache thrashing instructions along with their + address, number of misses, and its disassembly. (default: 32) + + * arg="icachesize=N" + * arg="iblksize=B" + * arg="iassoc=A" + + Instruction cache configuration arguments. They specify the cache size, block + size, and associativity of the instruction cache, respectively. + (default: N = 16384, B = 64, A = 8) + + * arg="dcachesize=N" + * arg="dblksize=B" + * arg="dassoc=A" + + Data cache configuration arguments. They specify the cache size, block size, + and associativity of the data cache, respectively. + (default: N = 16384, B = 64, A = 8) + + * arg="evict=POLICY" + + Sets the eviction policy to POLICY. Available policies are: :code:`lru`, + :code:`fifo`, and :code:`rand`. The plugin will use the specified policy for + both instruction and data caches. (default: POLICY = :code:`lru`) From a6b95a9733a94f38b289430bf46987809f53ab16 Mon Sep 17 00:00:00 2001 From: Mahmoud Mandour Date: Fri, 9 Jul 2021 15:30:05 +0100 Subject: [PATCH 44/44] MAINTAINERS: Added myself as a reviewer for TCG Plugins MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Mahmoud Mandour Signed-off-by: Alex Bennée Message-Id: <20210623125458.450462-6-ma.mandourr@gmail.com> Message-Id: <20210709143005.1554-41-alex.bennee@linaro.org> --- MAINTAINERS | 1 + 1 file changed, 1 insertion(+) diff --git a/MAINTAINERS b/MAINTAINERS index 83b55030ad..c4439a9488 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3018,6 +3018,7 @@ F: include/tcg/ TCG Plugins M: Alex Bennée R: Alexandre Iooss +R: Mahmoud Mandour S: Maintained F: docs/devel/tcg-plugins.rst F: plugins/