Add inventory

Make mon_host a template
Remove nm earlier
2023-08-29 13:56:20 +03:00 · 2023-08-29 13:56:20 +03:00 · 2023-08-29 13:56:20 +03:00 · 2023-08-29 13:56:20 +03:00 · 2023-08-29 13:56:20 +03:00 · 2023-08-29 13:56:20 +03:00
27 changed files with 669 additions and 115 deletions
--- a/README.md
+++ b/README.md
@ -1,7 +1,50 @@
 "Плейбуки" на баше для конфигурации Ceph + OpenNebula

-Почему на баше?
+Почему на баше? Потому что с ансиблом бесит yaml программирование!

-Потому что с ансиблом бесит yaml программирование!
+Хотя на баше, конечно, тоже не совсем красиво местами.

 Логика та же: можно запускать многократно.
+
+Переменные указываются в файле config/all_vars (предполагается, что он подключается из другого репозитория),
+пример - в all_vars.sample.
+
+В config/inventory/ перечисляются хосты, на которых это выполняется. Имя каждого файла должно быть равно
+соответствующему hostname, внутри файла должны быть две переменные play_host="внешний ip" и int_ip="внутренний ip"
+(пример наполнения в inventory_host.sample).
+
+## Готово
+
+ Вынести в отдельный скрипт настройку ssh-ключей на нодах
+ ethtool -C eth10g rx-usecs 0
+ systemctl mask emergency.service, systemctl mask emergency.target
+ На все ноды поставить qemu 4.x из sid (там virtio-blk discard=unmap умеет)
+ Переименовать сеть в eth1gX + eth10gX (X=1,2), иначе имена сетевых интерфейсов едут при перестановке NVMe-шек
+ Настроить live миграцию при reschedule: https://github.com/OpenNebula/one/blob/612300b191224b172fa4511845f2a17faa5f547c/src/scheduler/etc/sched.conf#L99
+ Везде убрать DHCP
+ Удалять network-manager
+ Везде прописать все ноды в /etc/hosts
+ В known_hosts также добавлять внутренние IP
+ Добавить passwordless ssh от рута между всеми серверами (чисто для удобства, oneadmin и так есть)
+ Шаблонизировать mon_initial_members в ceph.conf
+ Добавить аналог inventory (перечень хостов с их переменными) и на него пересадить генерацию /etc/hosts
+
+## TODO
+
+- sensors & sensors-exporter
+- Убрать quiet из опций GRUB
+- Мониторинг и алерты - пока Prometheus+Grafana настроены вручную на 172.31.1.13 ("подоконник")
+- Написать отдельно скрипты для начальной настройки кластера (создание ceph-deploy, создание пулов в цефе,
+  подключение датастора к opennebula). Оные выполняются только один раз на весь кластер, поэтому отдельно.
+- Сделать на всех хостах LACP из 2x 10GbE сетевых интерфейсов
+- Попробовать технологизировать "баш-плейбуки". Нужно не так много вещей, но некоторые могли
+  бы быть действительно полезны, например:
+  - Функция вида: Скопировать файл на хост и в зависимости от того, отличался ли он до этого, выполнить команду/команды
+  - Аналог ansible lineinfile - проверить наличие строки/строк в файле и добавить, если их там нет.
+    В вариантах "просто слить два файла построчно в любом порядке" и "заменить строчку, удовлетворяющую регэкспу"
+  - Полечить отступы в heredoc (в баше вложенные heredoc ломаются с отступами)
+  - Можно даже попробовать сделать всё это не на баше, но пока что баш выглядит проще всего, на то и шелл - команды дёргать
+  - Придумать, на чём красивее шаблонизировать конфиги (НО ТОЛЬКО НЕ НА JINJA!!!) - пока что это вообще envsubst
+  - И сделать, чтобы при подстановках проверялось, что никто не забыл установить подставляемые переменные
+  - Кстати, ещё были бы полезны функции работы с массивами - вывести через запятую, отфильтровать... - это
+    скорее всего вопрос либо к самому языку написания "плейбуков", либо к тому, что используется как шаблонизатор
--- a/all_vars.sample
+++ b/all_vars.sample
@ -0,0 +1,15 @@
+keepalived_virtual_ip=172.31.1.8
+keepalived_password=
+keepalived_router_id=ONEPROD
+galera_cluster_name=galera_cluster
+galera_password=
+opennebula_db_password=
+init_db=0
+one_key=
+oneadmin_password=
+serveradmin_password=
+one_domain=one.custis.ru
+libvirt_secret_uuid=d46404df-387c-4a3e-877e-1c63955f06d8
+gateway=172.31.1.1
+dns=172.31.1.1
+ntp_server=10.200.20.11
--- a/ceph-deploy/ceph.bootstrap-mds.keyring.example
+++ b/ceph-deploy/ceph.bootstrap-mds.keyring.example
@ -0,0 +1,4 @@
+# все *.keyring нужно взять свои, от ceph-deploy
+[client.bootstrap-mds]
+	key = ...
+	caps mon = "allow profile bootstrap-mds"
--- a/ceph-deploy/ceph.bootstrap-mgr.keyring.example
+++ b/ceph-deploy/ceph.bootstrap-mgr.keyring.example
@ -0,0 +1,3 @@
+[client.bootstrap-mgr]
+	key = ...
+	caps mon = "allow profile bootstrap-mgr"
--- a/ceph-deploy/ceph.bootstrap-osd.keyring.example
+++ b/ceph-deploy/ceph.bootstrap-osd.keyring.example
@ -0,0 +1,3 @@
+[client.bootstrap-osd]
+	key = ...
+	caps mon = "allow profile bootstrap-osd"
--- a/ceph-deploy/ceph.bootstrap-rgw.keyring.example
+++ b/ceph-deploy/ceph.bootstrap-rgw.keyring.example
@ -0,0 +1,3 @@
+[client.bootstrap-rgw]
+	key = ...
+	caps mon = "allow profile bootstrap-rgw"
--- a/ceph-deploy/ceph.client.admin.keyring.example
+++ b/ceph-deploy/ceph.client.admin.keyring.example
@ -0,0 +1,6 @@
+[client.admin]
+	key = ...
+	caps mds = "allow *"
+	caps mgr = "allow *"
+	caps mon = "allow *"
+	caps osd = "allow *"
--- a/ceph-deploy/ceph.client.libvirt.keyring.example
+++ b/ceph-deploy/ceph.client.libvirt.keyring.example
@ -0,0 +1,4 @@
+[client.libvirt]
+	key = ...
+	caps mon = "profile rbd"
+	caps osd = "profile rbd pool=rpool"
--- a/ceph-deploy/ceph.conf
+++ b/ceph-deploy/ceph.conf
@ -0,0 +1,99 @@
+[global]
+# сюда нужно записать uuid сгенерированный ceph-deploy
+fsid = ...
+public network = 192.168.5.0/24
+mon_initial_members = MON_IPS
+mon_host = MON_IPS
+
+rbd cache = false
+# без этой чудо-опции не работает cache=unsafe в QEMU:
+rbd cache writethrough until flush = false
+#rbd cache max dirty age = 5
+#rbd cache max dirty object = 20000
+debug ms = 0/0
+
+# загоняет все образы, клоны и откаты к снапшотам опеннебулы в EC
+rbd_default_data_pool = ecpool
+
+auth_cluster_required = cephx
+auth_service_required = cephx
+auth_client_required = cephx
+
+#ms_async_op_threads=1
+#ms_async_max_op_threads=1
+
+# отключим на хрен подписи
+cephx_require_signatures = false
+cephx_cluster_require_signatures = false
+cephx_sign_messages = false
+
+ms_bind_msgr2 = false
+
+#ms_crc_data = false
+#ms_crc_header = false
+
+debug osd = 0/0
+debug bluefs = 0/0
+debug perfcounter = 0/0
+#debug rbd = 0/0
+#debug rbd = 20/20
+#log file = /var/log/one/rbd.log
+#log file = /root/rbd-fio.log
+#admin socket = /var/run/one/$cluster-$type.$id.$pid.$cctid.asok
+debug rocksdb = 0/0
+debug bluestore = 0/0
+debug tp = 0/0
+debug objecter = 0/0
+debug client = 0/0
+debug crush = 0/0
+
+[osd]
+bluestore_prefer_deferred_size = 0
+bluestore_prefer_deferred_size_hdd = 0
+bluestore_prefer_deferred_size_ssd = 16384
+bluestore_min_alloc_size = 0
+bluestore_min_alloc_size_ssd = 4096
+#bluestore_csum_type = none
+bluestore_max_blob_size = 4194304
+bluestore_max_blob_size_ssd = 4194304
+osd_op_num_threads_per_shard = 8
+osd_op_num_shards = 2
+#osd_op_num_threads_per_shard = 1
+#osd_op_num_shards = 1
+bluestore_throttle_cost_per_io = 4000
+bluestore_sync_submit_transaction = true
+bluestore_compression_mode = passive
+bluestore_compression_min_blob_size_ssd = 131072
+bluestore_compression_max_blob_size_ssd = 4194304
+bdev_enable_discard = true
+bdev_async_discard = true
+osd_skip_data_digest = true
+
+# https://github.com/ceph/ceph/pull/26909, можно юзать с 14.2.4
+bluefs_preextend_wal_files = true
+
+#rocksdb_perf = true
+#rocksdb_collect_compaction_stats = true
+#rocksdb_collect_extended_stats = true
+#rocksdb_collect_memory_stats = true
+
+bluestore_rocksdb_options = compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=8,recycle_log_file_num=32,write_buffer_size=33554432,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
+
+[mon]
+mon allow pool delete = true
+mgr initial modules = dashboard
+ms_bind_msgr2 = true
+
+# SAS SSD Micron S630DC что-то не очень умеют в discard
+
+[osd.0]
+bdev_enable_discard = false
+bdev_async_discard = false
+
+[osd.1]
+bdev_enable_discard = false
+bdev_async_discard = false
+
+[osd.2]
+bdev_enable_discard = false
+bdev_async_discard = false
--- a/ceph-deploy/ceph.mon.keyring.example
+++ b/ceph-deploy/ceph.mon.keyring.example
@ -0,0 +1,3 @@
+[mon.]
+key = ...
+caps mon = allow *
--- a/ceph.sh
+++ b/ceph.sh
@ -0,0 +1,87 @@
+#!/bin/bash
+# Install & configure Ceph (mon+mgr+osds)
+
+# -e = stop on exception, -x = debug, -a = export all variables
+set -e -x -a
+
+# Include config
+. ./load-config.sh
+
+### Check host variables
+if [ -z "$play_host" -o -z "$node_name" -o -z "$int_ip" ]; then
+    echo "play_host/node_name/int_ip not specified"
+    exit 1
+fi
+
+### Configure network
+. ./network.sh
+
+# Setup passwordless self-ssh for root
+ssh root@$play_host <<EOF
+set -e -x
+
+if [ ! -f /root/.ssh/id_rsa.pub ]; then
+    ssh-keygen -t rsa -f /root/.ssh/id_rsa -q -P ""
+fi
+
+> tmp$$
+cat /root/.ssh/known_hosts >> tmp$$ || true
+ssh-keyscan localhost >> tmp$$
+ssh-keyscan $int_ip >> tmp$$
+sort tmp$$ | uniq > /root/.ssh/known_hosts
+rm tmp$$
+
+> tmp$$
+cat /root/.ssh/authorized_keys >> tmp$$ || true
+cat /root/.ssh/id_rsa.pub >> tmp$$
+sort tmp$$ | uniq > /root/.ssh/authorized_keys
+rm tmp$$
+EOF
+
+### Install packages, deploy mon, mgr and osds
+scp -r ceph-deploy root@$play_host:~/
+cat ./ceph-deploy/ceph.conf | perl -pe "s/MON_IPS/$int_ips/" | ssh root@$play_host 'cat > ~/ceph-deploy/ceph.conf'
+
+ssh root@$play_host <<EOF
+set -e -x
+
+DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" \
+    -o Dpkg::Options::="--force-confold" install -y \
+    ceph ceph-mds ceph-deploy jq
+
+cd ~/ceph-deploy
+chmod 600 \$(find ~/ceph-deploy -type f)
+
+cp ~/ceph-deploy/ceph.conf /etc/ceph/
+
+ceph-deploy mon add $int_ip
+ceph-deploy mgr create $node_name
+
+cp ~/ceph-deploy/ceph.bootstrap-osd.keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
+chmod 600 /var/lib/ceph/bootstrap-osd/ceph.keyring
+
+# Don't redeploy anything, just activate existing OSDs, then create new ones
+vgchange -a y
+ceph-volume lvm activate --all
+
+# Select available drives larger than 1.5 TB
+DRIVES=\$(ceph-volume inventory --format json | jq -r '.[] | select(.available == true and .sys_api.size >= 1500000000000) | .sys_api.path')
+
+for DEV in \$DRIVES; do
+    SIZE=\$(blockdev --getsz \$DEV)
+    # Reserve 32 GB partition on each drive for emergency (value is in 512b sectors)
+    RESERVED_SIZE=67108864
+    OSD_SIZE=\$((SIZE-RESERVED_SIZE-2048))
+    RESERVED_START=\$((OSD_SIZE+2048))
+    PREFIX=\$(perl -e "\\\$a = '\$DEV'; \\\$a =~ s/(\d)\\\$/\\\$1p/; print \\\$a;")
+    echo "PREFIX=\$PREFIX"
+    sfdisk \$DEV <<EOD
+label: gpt
+\${PREFIX}1 : start=2048, size=\$OSD_SIZE, type=0FC63DAF-8483-4772-8E79-3D69D8477DE4
+\${PREFIX}2 : start=\$RESERVED_START, type=0FC63DAF-8483-4772-8E79-3D69D8477DE4
+EOD
+    ceph-volume lvm prepare --bluestore --data \${PREFIX}1
+done
+
+ceph-volume lvm activate --all
+EOF
--- a/etc/apt/preferences
+++ b/etc/apt/preferences
@ -0,0 +1,3 @@
+Package: *
+Pin: release a=unstable
+Pin-Priority: 500
--- a/etc/apt/sources.list
+++ b/etc/apt/sources.list
@ -1,14 +1,21 @@
-#deb http://http.debian.net/debian/ sid main contrib non-free
-#deb-src http://http.debian.net/debian/ sid main contrib non-free
+deb http://mirror.yandex.ru/debian/ sid main contrib non-free
+deb-src http://mirror.yandex.ru/debian/ sid main contrib non-free

-deb http://http.debian.net/debian/ buster main contrib non-free
-deb-src http://http.debian.net/debian/ buster main contrib non-free
+deb http://mirror.yandex.ru/debian/ stretch main contrib non-free
+deb-src http://mirror.yandex.ru/debian/ stretch main contrib non-free
+
+deb http://mirror.yandex.ru/debian/ buster main contrib non-free
+deb-src http://mirror.yandex.ru/debian/ buster main contrib non-free

 deb http://security.debian.org/debian-security buster/updates main
 deb-src http://security.debian.org/debian-security buster/updates main

 # buster-updates, previously known as 'volatile'
-deb http://http.debian.net/debian/ buster-updates main
-deb-src http://http.debian.net/debian/ buster-updates main
+deb http://mirror.yandex.ru/debian/ buster-updates main
+deb-src http://mirror.yandex.ru/debian/ buster-updates main

 #deb http://hwraid.le-vert.net/debian stretch main
+
+# Ceph is needed for both OpenNebula nodes and Ceph nodes
+deb http://download.ceph.com/debian-nautilus/ bionic main
+deb-src http://download.ceph.com/debian-nautilus/ bionic main
--- a/etc/apt/sources.list.d/ceph.list
+++ b/etc/apt/sources.list.d/ceph.list
@ -1,2 +0,0 @@
-deb http://download.ceph.com/debian-nautilus/ bionic main
-deb-src http://download.ceph.com/debian-nautilus/ bionic main
--- a/etc/keepalived/keepalived.conf.env
+++ b/etc/keepalived/keepalived.conf.env
@ -1,13 +1,10 @@
 ! Configuration File for keepalived

 global_defs {
-    notification_email {
-        filippov@custis.ru
-    }
-    notification_email_from filippov@custis.ru
-    smtp_server localhost
-    smtp_connect_timeout 30
    router_id $keepalived_router_id
+    script_user root
+    enable_script_security
+    lvs_sync_daemon $eth10g VI_1
 }

 vrrp_instance VI_1 {
@ -16,7 +13,6 @@ vrrp_instance VI_1 {
    garp_master_delay 10
    smtp_alert
    virtual_router_id 51
-    lvs_sync_daemon_interface $eth10g
    priority 100
    advert_int 1
    authentication {
@ -26,4 +22,5 @@ vrrp_instance VI_1 {
    virtual_ipaddress {
        $keepalived_virtual_ip
    }
+    notify /etc/one/one-cluster.sh
 }
--- a/etc/network/interfaces.env
+++ b/etc/network/interfaces.env
@ -6,11 +6,14 @@ iface lo inet loopback
 #allow-hotplug $eth1g
 iface $eth1g inet manual

+# Fucking ifupdown 0.8.35 uses DUID for DHCP4 O_o
+# So we'll use static IPs
 auto br0
-iface br0 inet dhcp
+iface br0 inet static
    bridge_ports $eth1g
-    # Fucking ifupdown 0.8.35 uses DUID for DHCP4 O_o
-    post-up ip addr add $play_host dev br0
+    address $play_host/24
+    gateway $gateway
+    dns-nameservers $dns

 auto $eth10g
 iface $eth10g inet static
--- a/etc/one/one-cluster.sh
+++ b/etc/one/one-cluster.sh
@ -0,0 +1,33 @@
+#!/bin/bash
+
+set -e
+
+TYPE=$1
+NAME=$2
+STATE=$3
+case $STATE in
+    "MASTER")
+        systemctl start opennebula
+        systemctl start opennebula-sunstone
+        systemctl start opennebula-scheduler
+        systemctl start opennebula-novnc
+        exit 0
+        ;;
+    "BACKUP")
+        systemctl stop opennebula
+        systemctl stop opennebula-sunstone
+        systemctl stop opennebula-scheduler
+        systemctl stop opennebula-novnc
+        exit 0
+        ;;
+    "FAULT")
+        systemctl stop opennebula
+        systemctl stop opennebula-sunstone
+        systemctl stop opennebula-scheduler
+        systemctl stop opennebula-novnc
+        exit 0
+        ;;
+    *)  /sbin/logger "opennebula: unknown state"
+        exit 1
+        ;;
+esac
--- a/etc/rc.local.env
+++ b/etc/rc.local.env
@ -1,5 +1,7 @@
 #!/bin/bash

+cpupower frequency-set -g performance
+ethtool -C $eth10g rx-usecs 0
 ethtool -K $eth10g gro off gso off tso off lro off sg off
 ip l set $eth10g mtu 9000
 exit 0
--- a/etc/systemd/timesyncd.conf.env
+++ b/etc/systemd/timesyncd.conf.env
@ -1,5 +1,5 @@
 [Time]
-NTP=10.200.20.11
+NTP=$ntp_server
 #NTP=
 #FallbackNTP=0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org
 #RootDistanceMaxSec=5
--- a/general.sh
+++ b/general.sh
@ -4,7 +4,7 @@
 set -e -x -a

 # Include config
-. all_vars
+. ./load-config.sh

 ### Check host variables
 if [ -z "$play_host" -o -z "$node_name" -o -z "$int_ip" ]; then
@ -13,16 +13,27 @@ if [ -z "$play_host" -o -z "$node_name" -o -z "$int_ip" ]; then
 fi

 scp ./etc/apt/apt.conf root@$play_host:/etc/apt/apt.conf
+scp ./etc/apt/preferences root@$play_host:/etc/apt/preferences
 scp ./etc/apt/sources.list root@$play_host:/etc/apt/sources.list
 scp ./etc/locale.gen root@$play_host:/etc/locale.gen
+
+# Set time sync
+envsubst < ./etc/systemd/timesyncd.conf.env | \
+    ssh root@$play_host 'cat > /etc/systemd/timesyncd.conf'
+
 ssh root@$play_host <<EOF
 set -e -x

 echo $node_name > /etc/hostname

-hostname `cat /etc/hostname`
+hostname \`cat /etc/hostname\`

-apt-get update
+systemctl enable systemd-timesyncd && systemctl restart systemd-timesyncd
+
+systemctl mask emergency.service
+systemctl mask emergency.target
+
+apt-get update || true

 # gpg and friends for apt-key to work correctly
 DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" \
@ -31,8 +42,15 @@ DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" \
    gpg gpg-agent dirmngr apt-transport-https \
    prometheus-node-exporter

+grep -q -P '127.0.1.1\s+$node_name' /etc/hosts || (echo "127.0.1.1 $node_name" >> /etc/hosts)
+
+wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
+
+apt-get update || true
+
+rm /etc/timezone
 echo Europe/Moscow > /etc/timezone
-ln -fs /usr/share/zoneinfo/Europe/Moscow /etc/timezone
+ln -fs /usr/share/zoneinfo/Europe/Moscow /etc/localtime

 if ! grep -q '^PermitRootLogin' /etc/ssh/sshd_config; then
    echo PermitRootLogin without-password >> /etc/ssh/sshd_config
--- a/inventory_host.sample
+++ b/inventory_host.sample
@ -0,0 +1,3 @@
+# Put into config/inventory/host1 (hostname=host1)
+play_host=172.31.1.5
+int_ip=192.168.5.12
--- a/load-config.sh
+++ b/load-config.sh
@ -0,0 +1,19 @@
+#!/bin/bash
+
+set -e -a
+
+old_node_name="$node_name"
+old_int_ip="$int_ip"
+old_play_host="$play_host"
+. config/all_vars
+all_node_names=`ls config/inventory`
+opennebula_hosts=""
+int_ips=""
+for node_name in $all_node_names; do
+    . config/inventory/$node_name
+    opennebula_hosts="$opennebula_hosts $play_host"
+    int_ips="$int_ips $int_ip"
+done
+node_name="$old_node_name"
+int_ip="$old_int_ip"
+play_host="$old_play_host"
--- a/network.sh
+++ b/network.sh
@ -0,0 +1,108 @@
+#!/bin/bash
+
+set -e -x -a
+
+# Run once
+if [ -z "$eth10g" -o -z "$eth1g" ]; then
+
+### Check host variables
+if [ -z "$play_host" -o -z "$node_name" -o -z "$int_ip" ]; then
+    echo "play_host/node_name/int_ip not specified"
+    exit 1
+fi
+
+### Configure network
+ssh root@$play_host <<EOF
+apt-get purge -y network-manager
+EOF
+
+### Rename 1G interfaces to eth1g[1,2,3...], 10G to eth10g[1,2,3...]
+ssh root@$play_host '
+    set -e -x
+    i10g=1
+    i1g=1
+    changed=
+    >tmp$$
+    for i in /sys/class/net/eth1g* /sys/class/net/eth10g* `ls -d /sys/class/net/* | grep -vP "eth10?g"`; do
+        if [ -e "$i/device" ]; then
+            mac=`cat $i/address`
+            oldname=${i##/sys/class/net/}
+            newname=eth10g
+            if ethtool $oldname | grep -q 10000; then
+                newname=eth10g$i10g
+                i10g=$((i10g+1))
+            else
+                newname=eth1g$i1g
+                i1g=$((i1g+1))
+            fi
+            if [ "$newname" != "$oldname" ]; then
+                changed=1
+                ip link set $oldname down
+                ip link set dev $oldname name $newname
+                ip link set $newname up
+                if [ -e /run/network/ifstate.$oldname -a x`cat /run/network/ifstate.$oldname` != "x" ]; then
+                    rm -f /run/network/ifstate.$oldname
+                    echo $newname > /run/network/ifstate.$newname
+                fi
+            fi
+            cat >>tmp$$ <<EOF
+SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="$mac", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="$newname"
+EOF
+        fi
+    done
+    if [ "$changed" = "1" ]; then
+        cp tmp$$ /etc/udev/rules.d/70-persistent-net.rules
+        udevadm control --reload-rules
+        update-initramfs -u -k all
+    fi
+    rm -f tmp$$'
+
+### Find connected 1G and 10G network interfaces (10G is used for keepalived and galera)
+eth10g=
+eth1g=
+export $(ssh root@$play_host '
+    for i in /sys/class/net/*; do
+        ip link set ${i##/sys/class/net/} up
+        if [ x`cat $i/carrier 2>/dev/null` = "x1" ]; then
+            if [ x`cat $i/speed 2>/dev/null` = "x10000" -a "$eth10g" = "" ]; then
+                eth10g=${i##/sys/class/net/}
+                echo eth10g=$eth10g
+            elif [ x`cat $i/speed 2>/dev/null` = "x1000" -a "$eth1g" = "" ]; then
+                eth1g=${i##/sys/class/net/}
+                echo eth1g=$eth1g
+            fi
+        fi
+    done')
+
+if [ -z "$eth10g" ]; then
+    echo "10GbE network not found on $play_host"
+    exit 1
+fi
+
+### Configure network
+envsubst < ./etc/rc.local.env | ssh root@$play_host 'cat > /etc/rc.local'
+envsubst < ./etc/network/interfaces.env | ssh root@$play_host 'cat > /etc/network/interfaces.new'
+ssh root@$play_host <<EOF
+set -e -x
+
+DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" \
+    -o Dpkg::Options::="--force-confold" install -y \
+    bridge-utils resolvconf
+
+chmod 755 /etc/rc.local
+/etc/rc.local
+systemctl enable rc-local
+if ! cmp -s /etc/network/interfaces /etc/network/interfaces.new; then
+    nmcli dev disconnect $eth1g || true
+    mv /etc/network/interfaces.new /etc/network/interfaces
+    ifup br0
+    service networking restart
+fi
+if ! (ip a s | grep $int_ip); then
+    ifdown $eth10g || true
+    ifup $eth10g
+fi
+EOF
+
+# -z eth10g / eth1g
+fi
--- a/opennebula-ceph-cpds-clone.diff
+++ b/opennebula-ceph-cpds-clone.diff
@ -1,3 +1,14 @@
+--- /var/lib/one/remotes/tm/ceph/clone	2019-09-24 16:58:55.000000000 +0300
+++ /var/lib/one/remotes/tm/ceph/clone	2020-04-07 12:56:16.320845677 +0300
+@@ -149,7 +149,7 @@ else
+         set -e -o pipefail
+ 
+         if [ "\$(rbd_format $SRC_PATH)" = "2" ]; then
+-            $RBD ${EC_POOL_OPT} clone "$SRC_PATH@snap" $RBD_DST
+            $RBD ${EC_POOL_OPT} clone --object-size 512K "$SRC_PATH@snap" $RBD_DST
+         else
+             $RBD copy $SRC_PATH $RBD_DST
+         fi
 --- /var/lib/one/remotes/tm/ceph/cpds	2018-11-21 22:48:44.497052898 +0300
 +++ /var/lib/one/remotes/tm/ceph/cpds	2018-11-21 23:17:49.293548923 +0300
@@ -161,11 +161,13 @@ else
@ -5,7 +16,7 @@
 
             RBD_DST=\$RBD_DST@$SNAP_ID
 +
-+            $RBD clone \$RBD_DST $DST
+            $RBD clone --object-size 512K \$RBD_DST $DST
         else
             RBD_DST=$RBD_DST
 -        fi
@ -16,3 +27,14 @@
 EOF
 )
 fi
+--- /var/lib/one/remotes/tm/ceph/snap_revert	2019-09-24 16:58:55.000000000 +0300
+++ /var/lib/one/remotes/tm/ceph/snap_revert	2020-04-07 12:55:52.797266889 +0300
+@@ -151,7 +151,7 @@ SNAP_REVERT_CMD=$(cat <<EOF
+         exit 1
+     fi
+ 
+-    $RBD ${EC_POOL_OPT} clone \${RBD_TGT}@$SNAP_ID $RBD_DST
+    $RBD ${EC_POOL_OPT} clone --object-size 512K \${RBD_TGT}@$SNAP_ID $RBD_DST
+ EOF
+ )
+ 
--- a/opennebula-keys.sh
+++ b/opennebula-keys.sh
@ -0,0 +1,80 @@
+#!/bin/bash
+# Setup passwordless ssh for `oneadmin` (authorized_keys and known_hosts)
+
+set -e -a
+
+# Include config
+. ./load-config.sh
+
+key_hosts=${key_hosts:-$opennebula_hosts}
+
+# Add everyone to /etc/hosts
+add_etc_hosts=""
+for node_name in $all_node_names; do
+    . config/inventory/$node_name
+    add_etc_hosts="$add_etc_hosts""$int_ip $node_name"$'\n'
+done
+
+for node_name in $all_node_names; do
+    . config/inventory/$node_name
+    (ssh root@$play_host 'cat /etc/hosts'; echo -n "$add_etc_hosts" | grep -v $node_name) | sort | uniq > tmp$$
+    scp tmp$$ root@$play_host:/etc/hosts
+    rm tmp$$
+done
+
+# Generate keys for oneadmin if not yet
+for play_host in $key_hosts; do
+    ssh root@$play_host <<EOF
+set -e -x
+if [ ! -f /var/lib/one/.ssh/id_rsa.pub ]; then
+    su - oneadmin -c 'ssh-keygen -t rsa -f /var/lib/one/.ssh/id_rsa -q -P ""'
+fi
+EOF
+done
+
+# Gather & distribute host keys
+ssh-keyscan $opennebula_hosts > tmp$$
+for play_host in $key_hosts; do
+    ssh root@$play_host "ssh-keyscan $int_ips" >> tmp$$
+    break
+done
+for play_host in $key_hosts; do
+    cat tmp$$ > hostkeys$$
+    ssh root@$play_host 'cat /var/lib/one/.ssh/known_hosts || true' >> hostkeys$$
+    ssh root@$play_host 'ssh-keyscan localhost' >> hostkeys$$
+    cat hostkeys$$ | sort | uniq | ssh root@$play_host 'cat > /var/lib/one/.ssh/known_hosts'
+    rm hostkeys$$
+done
+rm tmp$$
+
+# Gather & distribute oneadmin keys
+> tmp$$
+for host in $opennebula_hosts; do
+    ssh root@$host 'cat /var/lib/one/.ssh/id_rsa.pub' >> tmp$$
+done
+for play_host in $key_hosts; do
+    ssh root@$play_host 'cat /var/lib/one/.ssh/authorized_keys || true' >> tmp$$
+    cat tmp$$ | sort | uniq | ssh root@$play_host 'cat > /var/lib/one/.ssh/authorized_keys'
+done
+rm tmp$$
+
+# Generate keys for root if not yet
+for play_host in $key_hosts; do
+    ssh root@$play_host <<EOF
+set -e -x
+if [ ! -f /root/.ssh/id_rsa.pub ]; then
+    ssh-keygen -t rsa -f /root/.ssh/id_rsa -q -P ""
+fi
+EOF
+done
+
+# Gather & distribute root keys
+> tmp$$
+for host in $opennebula_hosts; do
+    ssh root@$host 'cat /root/.ssh/id_rsa.pub' >> tmp$$
+done
+for play_host in $key_hosts; do
+    ssh root@$play_host 'cat /root/.ssh/authorized_keys || true' >> tmp$$
+    cat tmp$$ | sort | uniq | ssh root@$play_host 'cat > /root/.ssh/authorized_keys'
+done
+rm tmp$$
--- a/opennebula.sh
+++ b/opennebula.sh
@ -7,7 +7,7 @@
 set -e -x -a

 # Include config
-. all_vars
+. ./load-config.sh

 ### Check host variables
 if [ -z "$play_host" -o -z "$node_name" -o -z "$int_ip" ]; then
@ -15,84 +15,36 @@ if [ -z "$play_host" -o -z "$node_name" -o -z "$int_ip" ]; then
    exit 1
 fi

-### Find 1G and 10G network interfaces (10G is used for keepalived and galera)
-eth10g=
-eth1g=
-export $(ssh root@$play_host '
-    for i in /sys/class/net/*; do
-        ip link set ${i##/sys/class/net/} up
-        if [ x`cat /sys/class/net/enp4s0/carrier 2>/dev/null` == "x1" ]; then
-            if [ x`cat $i/speed 2>/dev/null` == "x10000" ]; then
-                echo eth10g=${i##/sys/class/net/}
-            elif [ x`cat $i/speed 2>/dev/null` == "x1000" ]; then
-                echo eth1g=${i##/sys/class/net/}
-            fi
-        fi
-    done')
-
-if [ -z "$eth10g" ]; then
-    echo "10GbE network not found on $play_host"
-    exit 1
-fi
-
 ### Configure network
-envsubst < ./etc/rc.local.env | ssh root@$play_host 'cat > /etc/rc.local'
-envsubst < ./etc/network/interfaces.env | ssh root@$play_host 'cat > /etc/network/interfaces.new'
-ssh root@$play_host <<EOF
-set -e
-
-DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" \
-    -o Dpkg::Options::="--force-confold" install -y \
-    bridge-utils
-
-chmod 755 /etc/rc.local
-/etc/rc.local
-systemctl enable rc-local
-if ! cmp -s /etc/network/interfaces /etc/network/interfaces.new; then
-    nmcli dev disconnect $eth1g; true
-    mv /etc/network/interfaces.new /etc/network/interfaces
-    ifup br0
-    service networking restart
-fi
-EOF
+. ./network.sh

 ### Install packages
 scp ./etc/apt/sources.list.d/opennebula.list root@$play_host:/etc/apt/sources.list.d/opennebula.list
 scp ./etc/apt/sources.list.d/mariadb.list root@$play_host:/etc/apt/sources.list.d/mariadb.list

 ssh root@$play_host <<EOF
-set -e
+set -e -x

 wget -q -O - https://downloads.opennebula.org/repo/repo.key | apt-key add -

 apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8

+apt-get update || true
+
 DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" \
    -o Dpkg::Options::="--force-confold" install -y \
-    lsb-release keepalived sudo qemu-kvm qemu-block-extra mariadb-server netcat-openbsd \
-    opennebula opennebula-sunstone opennebula-gate opennebula-flow opennebula-node opennebula-tools
+    lsb-release sudo mariadb-server netcat-openbsd \
+    opennebula opennebula-sunstone opennebula-gate opennebula-flow opennebula-node opennebula-tools ceph ruby-bundler
+
+# Keepalived from buster crashes with libc6 from sid
+DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" \
+    -o Dpkg::Options::="--force-confold" install -y -t sid \
+    qemu qemu-kvm qemu-system-common qemu-system-data qemu-system-x86 qemu-utils qemu-block-extra keepalived

 /usr/share/one/install_gems --yes
 EOF

-### Setup keepalived
-envsubst < ./etc/keepalived/keepalived.conf.env | \
-    ssh root@$play_host 'cat > /etc/keepalived/keepalived.conf'
-ssh root@$play_host 'systemctl restart keepalived'
-
 ### Setup or join MariaDB Galera Cluster
-scp ./etc/mysql/mariadb.conf.d/50-client.cnf root@$play_host:/etc/mysql/mariadb.conf.d/50-client.cnf
-
-# Put all hosts except this one in wsrep_cluster_address
-galera_hosts=
-for host in $opennebula_hosts; do
-    if [ "$host" != "$play_host" ]; then
-        galera_hosts=$galera_hosts,$host
-    fi
-done
-galera_hosts=${galera_hosts##,}
-envsubst < ./etc/mysql/mariadb.conf.d/50-server.cnf.env | \
-    ssh root@$play_host 'cat > /etc/mysql/mariadb.conf.d/50-server.cnf'

 # Create a user for Galera (if not yet)
 ssh root@$play_host <<EOF
@ -107,9 +59,26 @@ if ! (echo 'SELECT 1' | mysql --host=$play_host -u sst_user --password=$galera_p
 EOM
 fi

-[ -h /etc/mysql/my.cnf ] || rm /etc/mysql/my.cnf && ln -fs /etc/mysql/mariadb.cnf /etc/mysql/my.cnf
+if [ -f /etc/mysql/my.cnf ]; then
+    rm /etc/mysql/my.cnf
+fi
+if [ ! -h /etc/mysql/my.cnf ]; then
+    ln -fs /etc/mysql/mariadb.cnf /etc/mysql/my.cnf
+fi
 EOF

+# Copy configs
+scp ./etc/mysql/mariadb.conf.d/50-client.cnf root@$play_host:/etc/mysql/mariadb.conf.d/50-client.cnf
+
+# Put all hosts except this one in wsrep_cluster_address
+galera_hosts=
+for ip in $int_ips; do
+    galera_hosts=$galera_hosts,$ip
+done
+galera_hosts=${galera_hosts##,}
+envsubst < ./etc/mysql/mariadb.conf.d/50-server.cnf.env | \
+    ssh root@$play_host 'cat > /etc/mysql/mariadb.conf.d/50-server.cnf'
+
 if [ "$init_db" -eq 1 ]; then
    # Create a new cluster
    ssh root@$play_host <<EOF
@ -145,16 +114,24 @@ set -e -x

 perl -i -pe 's!^DB\s*=.*!DB = [ backend = "mysql", server = "localhost", port = 0, user = "oneadmin", passwd = "$opennebula_db_password", db_name = "opennebula" ]!' /etc/one/oned.conf

+perl -i -pe 's!^LIVE_RESCHEDS\s*=.*!LIVE_RESCHEDS = 1!' /etc/one/sched.conf
+
 while ! echo SELECT 1 | mysql; do
    echo Waiting for MySQL...
 done

-systemctl enable opennebula
-systemctl enable opennebula-sunstone
-systemctl restart opennebula
-systemctl restart opennebula-sunstone
+systemctl disable opennebula
+systemctl disable opennebula-sunstone
+systemctl stop opennebula
+systemctl stop opennebula-sunstone
 EOF

+### Setup keepalived
+scp etc/one/one-cluster.sh root@$play_host:/etc/one/
+envsubst < ./etc/keepalived/keepalived.conf.env | \
+    ssh root@$play_host 'cat > /etc/keepalived/keepalived.conf'
+ssh root@$play_host 'chmod 755 /etc/one/one-cluster.sh && systemctl restart keepalived'
+
 # Setup onedns
 envsubst < ./etc/systemd/system/onedns.service.env | \
    ssh root@$play_host 'cat > /etc/systemd/system/onedns.service'
@ -173,28 +150,7 @@ systemctl restart onedns
 EOF

 # Setup passwordless ssh for `oneadmin` (authorized_keys and known_hosts)
-ssh root@$play_host <<EOF
-set -e -x
-
-if [ ! -f /var/lib/one/.ssh/id_rsa.pub ]; then
-    su - oneadmin -c 'ssh-keygen -t rsa -f /root/.ssh/id_rsa -q -P ""'
-fi
-
-if [ ! -f /var/lib/one/.ssh/known_hosts ]; then
-    su - oneadmin -c 'ssh-keyscan localhost >> /var/lib/one/.ssh/known_hosts'
-    for host in $opennebula_hosts; do
-        su - oneadmin -c "ssh-keyscan \$host >> /var/lib/one/.ssh/known_hosts"
-    done
-fi
-EOF
-
-> tmp$$
-for host in $opennebula_hosts; do
-    ssh root@$host 'cat /var/lib/one/.ssh/id_rsa.pub' >> tmp$$
-done
-ssh root@$play_host 'cat /var/lib/one/.ssh/authorized_keys; true' >> tmp$$
-cat tmp$$ | sort | uniq | ssh root@$play_host 'cat > /var/lib/one/.ssh/authorized_keys'
-rm tmp$$
+key_hosts=$play_host ./opennebula-keys.sh

 # Add a host to OpenNebula and set reserved memory to 16G
 ssh root@$play_host <<EOF
@ -222,3 +178,27 @@ if ! grep -q clone /var/lib/one/remotes/tm/ceph/cpds; then
    patch -p0 < /root/opennebula-ceph-cpds-clone.diff
 fi
 EOF
+
+# Copy ceph configs
+cat ./ceph-deploy/ceph.conf | perl -pe "s/MON_IPS/$int_ips/" | ssh root@$play_host 'cat > /etc/ceph/ceph.conf'
+scp ./ceph-deploy/ceph.client.admin.keyring root@$play_host:/etc/ceph/
+scp ./ceph-deploy/ceph.client.libvirt.keyring root@$play_host:/etc/ceph/
+ssh root@$play_host 'chmod 600 /etc/ceph/ceph.client.admin.keyring'
+
+# Add Ceph secret
+ssh root@$play_host <<EOF
+set -e -x
+
+if [ ! -f /etc/libvirt/secrets/$libvirt_secret_uuid.base64 ]; then
+    cat > secret.xml <<EOS
+<secret ephemeral='no' private='no'>
+    <uuid>$libvirt_secret_uuid</uuid>
+    <usage type='ceph'>
+        <name>client.libvirt secret</name>
+    </usage>
+</secret>
+EOS
+    KEY=\$(ceph auth get-key client.libvirt)
+    virsh -c qemu:///system secret-define secret.xml; virsh -c qemu:///system secret-set-value --secret $libvirt_secret_uuid --base64 \$KEY
+fi
+EOF
--- a/run.sh
+++ b/run.sh
@ -1,2 +1,13 @@
-play_host=172.31.1.9 node_name=ripper4 int_ip=192.168.5.14 ./general.sh
-play_host=172.31.1.9 node_name=ripper4 int_ip=192.168.5.14 ./opennebula.sh
+#!/bin/bash -ea
+
+. ./load-config.sh
+
+NODES=${NODES:-$all_node_names}
+TAGS=${TAGS:-general opennebula ceph}
+
+for node_name in $NODES; do
+    . config/inventory/$node_name
+    for i in $TAGS; do
+        ./$i.sh
+    done
+done
Author	SHA1	Message	Date
Vitaliy Filippov	c6eeedda52	Add inventory	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	fb3a4f52e7	Make mon_host a template	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	236c10a48c	Remove nm earlier	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	9a436788c1	Fix opennebula installation and key distribution	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	4bbe35b83a	Remove hosts.add from here	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	b82b8c88cd	Add one forgotten step - add localhost into known_hosts	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	8f24e67c35	Fix qemu, add passwordless ssh for root	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	0cbe8f2554	rbd cache writethrough until flush = false	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	5a1bc95da0	Use object-size=512K for RBD clones	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	631a25de83	Fix keepalived scripts	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	dd945044b7	Add internal ips to known_hosts	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	61a1817846	Add all hosts to /etc/hosts on each host	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	3e14430b54	mirror.yandex.ru	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	a7e0ee886d	More qemu packages	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	8c1530e048	scheduler + novnc, too	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	4175da7955	-TODO	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	105767ffd2	Keep opennebula running on one host, fix /etc/timezone	2023-08-29 13:56:20 +03:00
Vitaliy Filippov	5c4576a886	Use apt-get -t instead of preferences to install newer qemu-kvm	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	c281e32f62	Rename network interfaces to eth1g, eth10g	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	95a0041517	Fix bugs	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	6ccc13606c	qemu 4.x	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	f39a2de71b	More optimal settings for Ceph	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	45370fdb45	+TODO	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	fcdc552895	OpenNebula key setup script	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	f39b6269df	Several small fixes - Activate LVM VGs before trying to activate Ceph OSDs (sometimes VGs start as inactive) - Reconfigure 10G network if it doesn't have the required address - Fix /etc/mysql/my.cnf symlink and configs better	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	ad77c551fe	Update ceph.conf before ceph-deploy	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	8982ebf594	Disable discard for osds 0-2 because SAS SSDs Micron S630DC don't handle it well	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	74bc96745b	Always copy osd.bootstrap	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	1dca98e748	Some fixes based on the deployment of another threadripper node :)	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	e048e80b4c	Add governor=performance	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	3c11285ccf	+TODO	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	a479ed08c8	+TODO	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	85d0e1d40b	Use config/all_vars	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	e06bd29c31	+README	2023-08-29 13:56:19 +03:00
Vitaliy Filippov	e6837e4af3	Add Ceph playbook	2023-08-29 13:56:19 +03:00