Wine-NSPA – io_uring I/O Architecture

Wine 11.6 + NSPA RT patchset | Kernel 6.19.x-rt with NTSync PI | 2026-04-15 Author: jordan Johnston

Table of Contents

1. Overview
2. Design Principles
3. I/O Architecture: Before and After
4. Phase 1: Synchronous Poll Replacement
5. Phase 2: Async File I/O Server Bypass
6. Phase 3: Socket I/O Server Bypass
7. File Manifest
8. Status Summary

1. Overview

This document is the deep-dive companion to the Wine-NSPA Architecture Overview. It covers the design decisions, tradeoffs, and implementation details of io_uring integration across all three phases. For background on Wine’s I/O model and why io_uring matters for RT audio, see the architecture page.

The two bottlenecks targeted by this work:

1. Syscall overhead. 4+ kernel transitions per async file read (register + epoll + alert + read). io_uring collapses this to 1 (io_uring_enter).
2. Global lock contention. Every fd in server epoll extends global_lock hold time. Fewer server-monitored fds = shorter hold = less contention for shmem dispatchers.

Relationship to Existing NSPA Infrastructure

Reference: rbernon’s Archived Attempt

Rémi Bernon attempted a full wineserver main-loop replacement with io_uring circa 2021-2022 (gitlab.winehq.org/rbernon/wine, branch archive/iouring). It was abandoned because io_uring was immature at the time – missing features, kernel bugs, API instability. That approach replaced the server’s epoll entirely (~500 LOC across server/fd.c, request.c, thread.c).

Wine-NSPA’s approach is fundamentally different: with the shmem fast path already handling request/reply IPC, the server main loop is no longer the bottleneck. Instead, we target the remaining server-dependent paths – file and socket I/O – from the client side, keeping changes isolated in a new io_uring.c file with minimal modifications to existing code.

2. Design Principles

• Per-thread rings. Each ntdll thread lazily initializes its own io_uring ring with IORING_SETUP_SINGLE_ISSUER | IORING_SETUP_COOP_TASKRUN. No cross-thread submission, no locking.
• RT-safe completions. COOP_TASKRUN ensures CQE processing happens in the submitting thread’s context – preserving SCHED_FIFO priority. No kernel worker threads at default priority.
• Transparent fallback. Every io_uring function returns -ENOSYS if the ring is unavailable. Callers fall back to the existing code path. Wine apps see identical behavior whether io_uring is present or not.
• fd lifetime safety. Async operations dup() the unix fd before submitting to the ring. The duplicate is owned by the in-flight SQE and closed on CQE completion. This prevents use-after-close if the server-side handle table changes during the operation.
• Cooperative completion drain. CQEs are processed at server_select() and server_wait() entry points – the natural places where a thread is about to block. This ensures completions are delivered promptly without adding threads or signals.
• Minimal rebase surface. The bulk of the code lives in a new file (io_uring.c, ~760 lines). Existing files get thin conditionals: if (!ntdll_io_uring_submit_*()) { return STATUS_PENDING; } /* fallback */.
• RT-safe allocation. Pre-allocated TLS pool of 32 uring_async_op structs (matches RING_SIZE). Freelist-based O(1) alloc/free — no malloc/free in the submit path. Initialized once at ring setup.

Per-Thread Ring Architecture

3. I/O Architecture: Before and After

I/O Path Comparison Diagram

Vanilla Wine: Server-Mediated Async File I/O

Client Thread Wineserver ───────────── ────────── NtReadFile(async) server_get_unix_fd() ──→ get_handle_fd register_async() ──→ register_async queue_async(&fd->read_q) set_fd_events(POLLIN) ←── epoll monitors fd return STATUS_PENDING ... main_loop_epoll(): (thread does other work) global_lock.lock() epoll_pwait2() → fd ready fd_poll_event → async_wake_up(STATUS_ALERTED) global_lock.unlock() ... (thread enters alertable wait) async_read_proc(): server_get_unix_fd() ──→ get_handle_fd (again) read(fd, buf, len) set IOSB, signal event

Syscalls per async read: 2 server round-trips (register + get_fd) + epoll_wait + read = 4+ kernel transitions

Wine-NSPA with io_uring: Client-Side Async File I/O

Client Thread Wineserver ───────────── ────────── NtReadFile(async) server_get_unix_fd() ──→ get_handle_fd (cached, usually no round-trip) dup(unix_fd) → ring_fd io_uring_prep_read(ring_fd, buf, len) io_uring_submit() (server never sees this I/O) return STATUS_PENDING ... (thread enters server_wait) ntdll_io_uring_process_completions(): CQE ready → bytes_read file_complete_async() close(ring_fd)

Syscalls per async read: 1 io_uring_enter (submit+wait batched) = 1 kernel transition

The server is completely bypassed for the I/O monitoring and data transfer. It still handles the initial fd lookup (usually cached) and completion port notifications if needed.

Synchronous I/O: poll() Replacement

Before: After: poll(fd, POLLIN, timeout) ntdll_io_uring_poll(fd, POLLIN, timeout) read(fd, buf, len) read(fd, buf, len) ← unchanged

The read()/write() still goes through virtual_locked_read() for write-watch safety. Only the poll wait is replaced.

4. Phase 1: Synchronous Poll Replacement

Status: COMMITTED (2d903dda200)

What Changed

In NtReadFile and NtWriteFile, the synchronous blocking path does poll(fd, events, timeout) to wait for fd readiness, then loops back to read()/write(). Phase 1 replaces the poll() call with ntdll_io_uring_poll().

Integration Points

Fallback

If ntdll_io_uring_poll() returns -ENOSYS (ring unavailable), the code immediately falls back to the original poll() call. Zero behavioral change for non-io_uring systems.

Why virtual_locked_read is Preserved

The actual read()/write() still uses virtual_locked_read() / virtual_locked_pread(), which handles EFAULT from write-watched pages by retrying inside virtual_mutex. io_uring only replaces the poll wait, not the data transfer.

5. Phase 2: Async File I/O Server Bypass

Status: COMMITTED (2d903dda200), pool allocator COMMITTED (82cea7143ff)

What Changed

When NtReadFile or NtWriteFile would register an async operation with the server (register_async / SERVER_START_REQ(register_async)), the io_uring path instead:

1. dup()s the unix fd for lifetime safety
2. Submits IORING_OP_READ or IORING_OP_WRITE to the per-thread ring
3. Returns STATUS_PENDING
4. The CQE is processed later at the next server_wait() or server_select() call

fd Lifetime Safety

The current async model re-fetches the unix fd via server_get_unix_fd() in every callback. io_uring requires the fd to remain valid for the duration of the in-flight SQE. Solution: dup() the fd before submission. The duplicate is owned exclusively by the uring_async_op struct and closed on CQE completion or cancellation.

This is analogous to the thread-map fd reference pattern used elsewhere in NSPA – hold a ref for the lifetime of the operation.

Completion Delivery

CQEs are drained cooperatively:

// server.c — called before any blocking wait
unsigned int server_select(…) {
    ntdll_io_uring_process_completions();  // drain CQ
    …
}
unsigned int server_wait(…) {
    ntdll_io_uring_process_completions();  // drain CQ
    …
}

When a CQE arrives, complete_uring_op() translates the result to NTSTATUS and calls file_complete_async() – the same function used by Wine’s normal async completion path. This handles:

• IO_STATUS_BLOCK update
• Event signaling (NtSetEvent)
• APC queuing (NtQueueApcThread)
• Completion port notification (via add_completion())

EFAULT Handling

If io_uring’s kernel read hits EFAULT (buffer in a write-watched page), the CQE result is -EFAULT. The completion handler detects this and frees the operation – the caller should retry through the server async path, which uses virtual_locked_read() with proper page fault handling. This is the graceful fallback for an edge case that rarely occurs in practice.

Integration Points

6. Phase 3: Socket I/O Server Bypass

Status: COMMITTED — sync + overlapped WORKING (a645adc66ed)

The Challenge

Socket I/O (sock_recv / sock_send) is tightly coupled with the server’s async lifecycle:

• Server creates wait_handle for the async operation
• Server tracks socket state (connect, listen, accept, shutdown)
• Server manages AFD poll state (pending_events, reported_events)
• Client uses set_async_direct_result() to report completion back to server
• sock_get_poll_events() in server decides what to monitor based on queue state

Unlike file I/O (where the server’s only role is fd monitoring), the socket code has the server actively participating in the protocol state machine.

Implemented Approach: E2 Bitmap + ALERTED-State Interception

How It Works

The key innovation is ALERTED-state interception – intercepting in the ALERTED block before set_async_direct_result is called:

Server: recv_socket → STATUS_ALERTED + wait_handle
Client: try_recv(fd) → EAGAIN (not ready)
                                              ← interception point
  BEFORE: set_async_direct_result(PENDING)    ← would restart async on server
  NOW:    set bitmap + io_uring POLL_ADD      ← async stays ALERTED (frozen)
          return STATUS_PENDING
  … io_uring monitors fd …
CQE fires:
  try_recv(fd) → SUCCESS (data available)
  set_async_direct_result(SUCCESS, bytes)     ← server accepts (ALERTED preserved)
  Server: completes async, signals event/IOCP

Why this works: When an async is ALERTED on the server, terminated=1 and async_waiting() returns false. The server does not monitor the fd via epoll. The bitmap provides additional safety (sock_get_poll_events returns -1). Only one call to set_async_direct_result ever happens – from the CQE handler with the final result.

Why previous approaches failed (4 attempts):

1. CQ drain inline NtSetEvent: Signal reentrancy crash – NtSetEvent requires Wine signal manipulation, unsafe from CQ drain context.
2. Deferred completion flush: Deadlock – event must be signaled during the wait to wake inproc_wait, not after.
3. Direct ntsync ioctl: Double completion – set_async_direct_result(PENDING) restarted the async, server monitored via epoll AND io_uring monitored → race.
4. Bitmap after set_async_direct_result: Same race – bitmap set too late, async already restarted.

Sync vs Overlapped Path

Global Lock Impact

Every socket fd removed from server epoll = one fewer event in main_loop_epoll() = shorter global_lock hold per iteration. This benefits ALL shmem dispatchers, not just the thread doing socket I/O.

Before: main_loop_epoll() processes N socket events while holding global_lock After: main_loop_epoll() processes (N - M) events (M = client-monitored sockets)

Test Results (v4-overlapped, 2026-04-15)

Phase A (immediate recv): 2000/2000, avg 95us, p99 162us Phase B (overlapped recv): 2000/2000, avg 113us, p99 189us, 2000 async (PENDING) Full suite: 22/22 PASS (11 tests x baseline + rt)

7. File Manifest

New Files

Modified Files

Build Dependency

liburing.so.2 (system package). Available as liburing in Arch Linux [extra]. Detected at configure time via AC_CHECK_LIB(uring, io_uring_queue_init).

8. Status Summary

Next Actions

1. Profile: measure server epoll fd reduction and global_lock hold time under socket load
2. Test with real socket-heavy applications (networked DAWs, MIDI over network)
3. Investigate multishot recv + provided buffers for streaming socket optimization
4. Investigate K3 (ntsync multishot poll) for event-driven audio loops