Fix io_uring fsync race condition causing metadata loss #77

changwei.ye · 2026-08-26T05:54:47+03:00

changwei.ye commented

2026-08-26 05:54:47 +03:00

Description

In the previous implementation, unsynced_meta_write_count was incremented early in prepare_meta_block_write
before the I/O request was even submitted. Because Vitastor does not use IOSQE_IO_LINK for io_uring batch
submissions, the kernel does not guarantee execution order within a batch.

Scenario:
1. Client A initiates a metadata write. `unsynced_meta_write_count` is incremented.
2. Client B sends a global SYNC request. The engine sees `unsynced_meta_write_count > 0`, clears it to 0,
   and prepares an `fsync(meta_fd)` SQE.
3. Both the `writev` and `fsync` SQEs are submitted in the same batch.
4. If the kernel's io_wq schedules the `fsync` before the `writev` (Count Stealing), the `fsync` completes
   instantly without flushing the intended metadata.
5. Client A's write completes and ACKs the client.
6. Client A issues a SYNC to guarantee its write. However, since the counter was already consumed by
   Client B's SYNC, Client A's SYNC sees a count of 0 and returns immediately without flushing.
7. A crash at this point results in metadata loss despite Client A receiving a successful SYNC ACK.

This commit moves the `unsynced_meta_write_count` increment into the io_uring completion callback of
the metadata `writev`. This perfectly mirrors the data area's "count-after-completion" strategy,
ensuring that concurrent SYNC operations cannot steal the count before the metadata is safely in
the OS page cache.

Contributor Name

Changwei Ye

CLA

I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md

### Description In the previous implementation, `unsynced_meta_write_count` was incremented early in `prepare_meta_block_write` before the I/O request was even submitted. Because Vitastor does not use IOSQE_IO_LINK for io_uring batch submissions, the kernel does not guarantee execution order within a batch. Scenario: 1. Client A initiates a metadata write. `unsynced_meta_write_count` is incremented. 2. Client B sends a global SYNC request. The engine sees `unsynced_meta_write_count > 0`, clears it to 0, and prepares an `fsync(meta_fd)` SQE. 3. Both the `writev` and `fsync` SQEs are submitted in the same batch. 4. If the kernel's io_wq schedules the `fsync` before the `writev` (Count Stealing), the `fsync` completes instantly without flushing the intended metadata. 5. Client A's write completes and ACKs the client. 6. Client A issues a SYNC to guarantee its write. However, since the counter was already consumed by Client B's SYNC, Client A's SYNC sees a count of 0 and returns immediately without flushing. 7. A crash at this point results in metadata loss despite Client A receiving a successful SYNC ACK. This commit moves the `unsynced_meta_write_count` increment into the io_uring completion callback of the metadata `writev`. This perfectly mirrors the data area's "count-after-completion" strategy, ensuring that concurrent SYNC operations cannot steal the count before the metadata is safely in the OS page cache. ### Contributor Name Changwei Ye ### CLA - [x] I accept Vitastor CLA agreement: https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/CLA-en.md

changwei.ye added 1 commit 2026-08-26 05:54:48 +03:00

Fix io_uring fsync race condition causing metadata loss 81848432e4

In the previous implementation, `unsynced_meta_write_count` was incremented early in `prepare_meta_block_write`
before the I/O request was even submitted. Because Vitastor does not use IOSQE_IO_LINK for io_uring batch
submissions, the kernel does not guarantee execution order within a batch.

Scenario:
1. Client A initiates a metadata write. `unsynced_meta_write_count` is incremented.
2. Client B sends a global SYNC request. The engine sees `unsynced_meta_write_count > 0`, clears it to 0,
   and prepares an `fsync(meta_fd)` SQE.
3. Both the `writev` and `fsync` SQEs are submitted in the same batch.
4. If the kernel's io_wq schedules the `fsync` before the `writev` (Count Stealing), the `fsync` completes
   instantly without flushing the intended metadata.
5. Client A's write completes and ACKs the client.
6. Client A issues a SYNC to guarantee its write. However, since the counter was already consumed by
   Client B's SYNC, Client A's SYNC sees a count of 0 and returns immediately without flushing.
7. A crash at this point results in metadata loss despite Client A receiving a successful SYNC ACK.

This commit moves the `unsynced_meta_write_count` increment into the io_uring completion callback of
the metadata `writev`. This perfectly mirrors the data area's "count-after-completion" strategy,
ensuring that concurrent SYNC operations cannot steal the count before the metadata is safely in
the OS page cache.

vitalif approved these changes 2026-08-26 11:08:51 +03:00

vitalif changed title from ~~WIP: Fix io_uring fsync race condition causing metadata loss~~ to Fix io_uring fsync race condition causing metadata loss

2026-08-26 11:09:45 +03:00

vitalif merged commit 30dd9869f7 into master

2026-08-26 11:12:55 +03:00

vitalif referenced this issue from a commit

2026-08-26 11:12:56 +03:00

Fix io_uring fsync race condition causing metadata loss (#77)

vitalif referenced this issue from a commit

2026-08-26 11:14:27 +03:00

Fix io_uring fsync race condition causing metadata loss (#77)

changwei.ye deleted branch fix-fsync-race

2026-08-26 12:05:39 +03:00

Sign in to join this conversation.