[DNM] osd: run calc_object_states off the OSD thread #123

Open
lei.xianqiang wants to merge 1 commits from lei.xianqiang/vitastor:calc_object_states_in_worker_threads into master
First-time contributor

Description

Sorting and walking every object version inside the OSD loop used to
stall OSD_OP_PING past osd_ping_timeout, so peers declared the OSD dead.

The calculation now runs on pg-calc threads. It only touches a scratch PG.
The main thread installs the result at the start of the next loop, before
peering, and swaps the object maps so pointers into state_dict stay valid.

stop and repeer bump calc_gen. A job that has not entered
calc_object_states is dropped at once, so the new peering can LIST
without waiting out a stale sort. A job already inside the calculation
finishes, and its result is discarded. calc_inflight_gen names that
in-flight job, so its completion clears only its own calc_running and
cannot drop a calculation submitted later.

relock_pg does not dereference peering_state while the worker owns it.
Tests stay on the synchronous path because the mock ring loop does not
poll eventfd.

Contributor Name

Xianqiang Lei (Datatom, https://www.datatom.com/)

CLA

### Description Sorting and walking every object version inside the OSD loop used to stall OSD_OP_PING past osd_ping_timeout, so peers declared the OSD dead. The calculation now runs on pg-calc threads. It only touches a scratch PG. The main thread installs the result at the start of the next loop, before peering, and swaps the object maps so pointers into state_dict stay valid. stop and repeer bump calc_gen. A job that has not entered calc_object_states is dropped at once, so the new peering can LIST without waiting out a stale sort. A job already inside the calculation finishes, and its result is discarded. calc_inflight_gen names that in-flight job, so its completion clears only its own calc_running and cannot drop a calculation submitted later. relock_pg does not dereference peering_state while the worker owns it. Tests stay on the synchronous path because the mock ring loop does not poll eventfd. ### Contributor Name Xianqiang Lei (Datatom, https://www.datatom.com/) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
lei.xianqiang added 1 commit 2026-11-17 11:11:10 +03:00
Sorting and walking every object version inside the OSD loop used to
stall OSD_OP_PING past osd_ping_timeout, so peers declared the OSD dead.

The calculation now runs on pg-calc threads. It only touches a scratch PG.
The main thread installs the result at the start of the next loop, before
 peering, and swaps the object maps so pointers into state_dict stay valid.

stop and repeer bump calc_gen. A job that has not entered
calc_object_states is dropped at once, so the new peering can LIST
without waiting out a stale sort. A job already inside the calculation
finishes, and its result is discarded. calc_inflight_gen names that
in-flight job, so its completion clears only its own calc_running and
cannot drop a calculation submitted later.

relock_pg does not dereference peering_state while the worker owns it.
Tests stay on the synchronous path because the mock ring loop does not
poll eventfd.
Author
First-time contributor

Do not merge; this is only to discuss the test data for the currently committed code.

  1. Env Info
Node: 3 
Disk: 6 * 7.68TB NVMe SSD, 3 for Ceph 20, 3 for Vitastor 3.2.2 with https://git.yourcmc.ru/vitalif/vitastor/pulls/122
Network: 2×25GbE, Bond 4
Pool: 3 × replication
  1. Test method
a. Pre-create four 1 TB volumes in the pool. 
b. Stop all OSDs on one disk and measure the time from when any PG becomes non-active after the stop until all PGs reach the active state.
c. Start all OSDs on that disk and measure the time from when any PG becomes non-active after the start until all PGs reach the active state. 
d. Run both tests for three rounds in total.
  1. Test data
    Storage Network OSD/Disk PG Num OSD Object Size pg_calc_thread_count peering duration of stop all OSDs on one disk (s) peering duration of start all OSDs on one disk (s)
    1st 2nd 3rd 1st 2nd 3rd
    Ceph TCP 1 64 4M \ 4 2 6 4 4 4
    Vitastor TCP 1 16 128K 0 16 15 16 20 20 20 osd_ping_timeout=100
    Vitastor TCP 1 16 128K 1 15 14 15 19 20 19
    Vitastor TCP 1 16 128K 2 12 11 12 17 17 17
    Vitastor TCP 1 16 128K 3 11 10 10 16 16 15
    Vitastor TCP 1 64 128K 0 14 13 14 18 17 18 osd_ping_timeout=100
    Vitastor TCP 1 16 256K 0 8 8 8 11 11 10 osd_ping_timeout=100
    Vitastor TCP 1 16 256K 1 7 7 7 9 10 10
    Vitastor TCP 1 16 256K 2 6 6 7 8 8 9
    Vitastor TCP 1 16 512K 0 5 5 5 6 6 6 osd_ping_timeout=100
    Vitastor TCP 1 16 512K 1 4 4 4 5 5 5
    Vitastor TCP 1 16 512K 2 3 3 4 4 5 4
    Vitastor TCP 1 16 1M 0 3 3 3 3 4 3
    Vitastor TCP 1 16 1M 1 3 3 3 3 3 4
    Vitastor TCP 1 16 1M 2 3 3 3 3 3 3
    Vitastor TCP 2 32 128K 0 11 12 12 14 13 12 osd_ping_timeout=100
    Vitastor TCP 2 32 128K 1 10 10 10 11 12 11
    Vitastor TCP 2 32 128K 2 7 8 8 11 10 12
    Vitastor TCP 3 32 128K 0 8 7 7 12 14 13 osd_ping_timeout=100
    Vitastor TCP 3 32 128K 1 6 6 6 11 11 12
    Vitastor TCP 3 32 128K 2 6 5 5 9 8 9
    Vitastor RoCEv2 1 16 128K 1 16 15 16 19 19 17

https://github.com/vitalif/vitastor/issues/145#issuecomment-6848289391
increase object count (128k -> 256k), but it requires recreating all OSDs and pools and copying data, it will halve the amount of memory used for metadata and listing sizes too

Increasing the object size can significantly shorten the peering duration. These data can also serve as reference data for peering optimization.

**Do not merge; this is only to discuss the test data for the currently committed code.** 1. Env Info ``` Node: 3 Disk: 6 * 7.68TB NVMe SSD, 3 for Ceph 20, 3 for Vitastor 3.2.2 with https://git.yourcmc.ru/vitalif/vitastor/pulls/122 Network: 2×25GbE, Bond 4 Pool: 3 × replication ``` 2. Test method ``` a. Pre-create four 1 TB volumes in the pool. b. Stop all OSDs on one disk and measure the time from when any PG becomes non-active after the stop until all PGs reach the active state. c. Start all OSDs on that disk and measure the time from when any PG becomes non-active after the start until all PGs reach the active state. d. Run both tests for three rounds in total. ``` 3. Test data | Storage | Network | OSD/Disk | PG Num | OSD Object Size | pg_calc_thread_count | peering duration of stop all OSDs on one disk (s) | | | peering duration of start all OSDs on one disk (s) | | | | | -------- | ------- | -------- | ------ | --------------- | -------------------- | ------------------------------------------------- | ---- | ---- | -------------------------------------------------- | ---- | ---- | -------------------- | | | | | | | | 1st | 2nd | 3rd | 1st | 2nd | 3rd | | | Ceph | TCP | 1 | 64 | 4M | \ | 4 | 2 | 6 | 4 | 4 | 4 | | | Vitastor | TCP | 1 | 16 | 128K | 0 | 16 | 15 | 16 | 20 | 20 | 20 | osd_ping_timeout=100 | | Vitastor | TCP | 1 | 16 | 128K | 1 | 15 | 14 | 15 | 19 | 20 | 19 | | | Vitastor | TCP | 1 | 16 | 128K | 2 | 12 | 11 | 12 | 17 | 17 | 17 | | | Vitastor | TCP | 1 | 16 | 128K | 3 | 11 | 10 | 10 | 16 | 16 | 15 | | | Vitastor | TCP | 1 | 64 | 128K | 0 | 14 | 13 | 14 | 18 | 17 | 18 | osd_ping_timeout=100 | | Vitastor | TCP | 1 | 16 | 256K | 0 | 8 | 8 | 8 | 11 | 11 | 10 | osd_ping_timeout=100 | | Vitastor | TCP | 1 | 16 | 256K | 1 | 7 | 7 | 7 | 9 | 10 | 10 | | | Vitastor | TCP | 1 | 16 | 256K | 2 | 6 | 6 | 7 | 8 | 8 | 9 | | | Vitastor | TCP | 1 | 16 | 512K | 0 | 5 | 5 | 5 | 6 | 6 | 6 | osd_ping_timeout=100 | | Vitastor | TCP | 1 | 16 | 512K | 1 | 4 | 4 | 4 | 5 | 5 | 5 | | | Vitastor | TCP | 1 | 16 | 512K | 2 | 3 | 3 | 4 | 4 | 5 | 4 | | | Vitastor | TCP | 1 | 16 | 1M | 0 | 3 | 3 | 3 | 3 | 4 | 3 | | | Vitastor | TCP | 1 | 16 | 1M | 1 | 3 | 3 | 3 | 3 | 3 | 4 | | | Vitastor | TCP | 1 | 16 | 1M | 2 | 3 | 3 | 3 | 3 | 3 | 3 | | | Vitastor | TCP | 2 | 32 | 128K | 0 | 11 | 12 | 12 | 14 | 13 | 12 | osd_ping_timeout=100 | | Vitastor | TCP | 2 | 32 | 128K | 1 | 10 | 10 | 10 | 11 | 12 | 11 | | | Vitastor | TCP | 2 | 32 | 128K | 2 | 7 | 8 | 8 | 11 | 10 | 12 | | | Vitastor | TCP | 3 | 32 | 128K | 0 | 8 | 7 | 7 | 12 | 14 | 13 | osd_ping_timeout=100 | | Vitastor | TCP | 3 | 32 | 128K | 1 | 6 | 6 | 6 | 11 | 11 | 12 | | | Vitastor | TCP | 3 | 32 | 128K | 2 | 6 | 5 | 5 | 9 | 8 | 9 | | | Vitastor | RoCEv2 | 1 | 16 | 128K | 1 | 16 | 15 | 16 | 19 | 19 | 17 | | > https://github.com/vitalif/vitastor/issues/145#issuecomment-6848289391 > increase object count (128k -> 256k), but it requires recreating all OSDs and pools and copying data, it will halve the amount of memory used for metadata and listing sizes too Increasing the object size can significantly shorten the peering duration. These data can also serve as reference data for peering optimization.
lei.xianqiang force-pushed calc_object_states_in_worker_threads from e5dbfb8a0f to 24d89be551 2026-11-17 11:43:35 +03:00 Compare
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u calc_object_states_in_worker_threads:lei.xianqiang-calc_object_states_in_worker_threads
git checkout lei.xianqiang-calc_object_states_in_worker_threads
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vitalif/vitastor#123
No description provided.