osd: Log only the first ENOSPC write subop failure per slow_log_interval #150

Open
koolkhel wants to merge 1 commits from koolkhel/vitastor:osd-enospc-subop-log into master
Contributor

Description

Clients retry writes that got ENOSPC every client_eio_retry_interval (1 s by default) until space is freed (client_retry_enospc, on by default), and the primary OSD printed a line for every failed write subop of every retry. That is one line per second for every waiting write: one fio with iodepth 16 over a full pool makes ~250 lines/s on the OSD, and many clients waiting for space can fill the disk with logs long before the space is freed.

Print only the first ENOSPC write subop failure in each slow_log_interval and report the number of the others from print_slow():

write_stable subop to 1000000000001:20000 v65537 failed on osd 3: retval = -28 (expected 131072)
5 more write subops failed with ENOSPC in the last 10 s

Other subop errors are still printed one by one. The counter is per OSD, so if several peers are full, each interval names only the first one.

Tested: new tests/test_enospc_retry_log.sh fills the pool, enables client_retry_enospc, runs fio with iodepth 16 for 15 s and counts the failed subop lines: 3840 on master, 6 with the fix (2 per OSD), and the summary line is printed; also with IMMEDIATE_COMMIT=1. test_enospc.sh (replicated, xor, IMMEDIATE_COMMIT=1) and test_recovery_enospc.sh pass.

Contributor Name

Yury Luneff (MIND Software LLC, https://mindsw.io)

CLA

### Description Clients retry writes that got ENOSPC every `client_eio_retry_interval` (1 s by default) until space is freed (`client_retry_enospc`, on by default), and the primary OSD printed a line for every failed write subop of every retry. That is one line per second for every waiting write: one fio with iodepth 16 over a full pool makes ~250 lines/s on the OSD, and many clients waiting for space can fill the disk with logs long before the space is freed. Print only the first ENOSPC write subop failure in each `slow_log_interval` and report the number of the others from `print_slow()`: ``` write_stable subop to 1000000000001:20000 v65537 failed on osd 3: retval = -28 (expected 131072) 5 more write subops failed with ENOSPC in the last 10 s ``` Other subop errors are still printed one by one. The counter is per OSD, so if several peers are full, each interval names only the first one. Tested: new `tests/test_enospc_retry_log.sh` fills the pool, enables `client_retry_enospc`, runs fio with iodepth 16 for 15 s and counts the failed subop lines: 3840 on master, 6 with the fix (2 per OSD), and the summary line is printed; also with `IMMEDIATE_COMMIT=1`. `test_enospc.sh` (replicated, xor, `IMMEDIATE_COMMIT=1`) and `test_recovery_enospc.sh` pass. ### Contributor Name Yury Luneff (MIND Software LLC, https://mindsw.io) ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md
koolkhel added 1 commit 2026-11-22 19:39:37 +03:00
Clients retry writes that got ENOSPC every client_eio_retry_interval until
space is freed (client_retry_enospc), and the primary OSD printed a line for
every failed write subop of every retry. That is one line per second for
every waiting write, so with many writes waiting the logs can fill the disk
long before space is freed.

Print only the first ENOSPC write subop failure in each slow_log_interval
and report the number of the others from print_slow(). Other subop errors
are still printed one by one.
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u osd-enospc-subop-log:koolkhel-osd-enospc-subop-log
git checkout koolkhel-osd-enospc-subop-log
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vitalif/vitastor#150
No description provided.