Wine-NSPA – Message Bypass Architecture

Wine 11.6 + NSPA RT patchset | Kernel 6.19.x-rt with NTSync PI | 2026-04-18 Author: jordan Johnston

Table of Contents

1. Overview
2. Design Principles
3. Vanilla Wine vs Wine-NSPA Message Ring
4. Ring Layout in Shared Memory
5. Memfd Lifecycle
6. Client Fast Path (POST + SEND)
7. Dispatch Path
8. Opt-In Gating
9. Results & Ballpark Reduction
10. File Manifest
11. Phase History
12. Future Extensions (ranked)

Footnote: Why memfd and not session shmem

1. Overview

Wine’s windowing model routes every PostMessage / SendMessage call through the wineserver: the sender writes a request, the server allocates a struct message, inserts it into the receiver’s queue, the receiver polls via GetMessage / PeekMessage (another wineserver round-trip), and for synchronous sends a reply_message round-trip closes the loop. On a typical RT audio workload this costs hundreds to thousands of wineserver RTTs per second – the NSPA profiler captured 6,239 send_message RTTs / 60 s from Ableton Live’s AudioCalc thread alone during a single adversarial recording session.

Wine-NSPA message bypass replaces that round-trip chain for same-process cross-thread window messages with a direct shared-memory ring:

1. Sender writes the message into the receiver thread’s ring and wakes the receiver via an NTSync event
2. Receiver’s message pump reads the message out of the ring locally (no get_message server request)
3. For synchronous SENDs the receiver writes the reply back into the sender’s reply ring and signals

The feature is invisible to Win32 applications – the same PostMessage / SendMessage API, the same delivery semantics, the same window procedure dispatch. It is same-process-only by design (cross-process messaging continues through the server because ring addresses like HWND / WPARAM / LPARAM only make sense in the sender’s address space and handle table).

Motivating Profile

The bypass targets the AudioCalc + DWM-Sync -> MainThread hot path that dominates this profile.

Relationship to Existing NSPA Infrastructure

2. Design Principles

• Memfd-backed per-queue rings. Each thread queue owns a private memfd_create() region containing its bypass ring. Server allocates on demand; client receives via SCM_RIGHTS and mmaps locally. Rings never live in Wine’s session shmem (§8 explains why that matters).
• Same-process only. Cross-process messages cannot use ring pointers (HWND translation, WPARAM/LPARAM pointer dereferences). The server rejects cross-process nspa_get_thread_queue requests early.
• Transparent fallback. Every bypass operation returns FALSE / NULL on any corner case (bypass disabled, no own ring, lookup failure, cross-process destination, DDE message, thread-message with hwnd == 0). Callers fall back to the legacy wineserver path. Wine apps see identical behavior whether bypass is enabled or not.
• RT-safe fast path. Ring slots use __atomic_* operations only. Shared memory is MAP_POPULATE-prefaulted and mlock-pinned so no demand paging happens on hot access. Warm-up cost (memfd create + map + lock) is paid once per peer, off the RT-critical path.
• Layered opt-in gates. Each tier (POST capture, wake-bit synthesis, SEND reply bootstrap, client-side dispatch) is an independent NSPA_* environment variable. Users can enable the layers they trust and fall back to stricter Phase 3 behavior by unsetting the later gates.
• Minimal protocol surface. Two server requests: nspa_get_thread_queue (peer lookup) and nspa_ensure_own_bypass (own bootstrap). Server handlers reuse a single nspa_alloc_bypass_shm() helper for both.

3. Vanilla Wine vs Wine-NSPA Message Ring

The reduction is not just in RTT count – vanilla Wine’s send_message handler acquires global_lock to insert the new message into the receiver’s queue. Under heavy traffic this contended mutex becomes a serialization point. The memfd ring sidesteps global_lock entirely: slot reservation is a lock-free CAS in shared memory.

4. Ring Layout in Shared Memory

Each msg_queue owns a nspa_queue_bypass_shm_t region consisting of a 64-slot forward-message ring and a 16-slot reply ring. Sizes are compile-time constants chosen to keep the whole region around 10 KB.

The forward ring’s state transitions are CAS-claimed (multi-producer head, single-consumer tail). The reply ring’s generation field discriminates against stale writebacks: a sender that times out marks its slot FREE and a later writer that finds state != PENDING drops the reply without touching user memory.

Forward Slot State Machine

5. Memfd Lifecycle

Every queue’s bypass region is backed by an anonymous memfd_create() file. The fd’s lifetime follows the queue: created on first use, closed on queue destroy. Clients that need to talk to a peer receive the fd over the wineserver socket via SCM_RIGHTS and mmap it locally.

Why memfd, Not Session Shmem

Wine’s session shmem is a single large mmap’d region shared across the entire wineserver + all client processes. alloc_shared_object() sub-allocates from that region via a bump allocator with per-object headers containing an id and a seqlock seq. This is great for small, long-lived shared records (process/thread metadata, desktop state).

The original msg-bypass implementation put the ring inside session shmem too. It produced a reliable, reproducible Ableton regression: library panel would not populate whenever MainThread had a ring allocated – and every runtime gate tested showed the bug persisted independent of ring reads/writes. The allocation machinery itself was the trigger. See §8 for the full story.

memfd gives each queue a private page-aligned region with zero interaction with session shmem’s allocator, seqlock, or layout. The library regression vanished on the redesign.

6. Client Fast Path (POST + SEND)

Warm-cache RT cost estimate (sender side, steady state):

Cold-cache cost (first publish to a peer) adds one SERVER_START_REQ(nspa_get_thread_queue) round-trip + SCM_RIGHTS receive + mmap + mlock – ~50 – 200 µs one-time, paid off the RT path.

Fall-back conditions (fast path returns FALSE -> caller falls back to server):

• NSPA_ENABLE_MSG_RING env var unset (default)
• dest_tid == current tid (same-thread PostMessage)
• hwnd == 0 (thread message; semantics require server handling)
• dest is in a different process
• Message is in WM_DDE_FIRST..WM_DDE_LAST
• MSG_CALLBACK / MSG_HOOK_LL types
• Cache full (32 slot table) – rare with typical thread counts
• For SEND: NSPA_ENABLE_OWN_BOOTSTRAP unset (opt-in gate)

7. Dispatch Path

The receiver side has two concerns: (a) learn that a ring message has arrived, and (b) pull it out and deliver it to the window proc.

Wake-bit Synthesis

NtUserGetQueueStatus, check_queue_bits, and the message pump’s local shmem check all need to report ring-pending activity alongside legacy wake_bits in the queue’s shmem. Phase 4.5 (1d18cb7c4e8) wires this through the Phase 4 TLS-cached own-ring mmap:

c const nspa_queue_bypass_shm_t *queue_bypass = nspa_get_own_bypass_shm_public(); if (queue_bypass) { UINT ring_total = __atomic_load_n(&queue_bypass->nspa_msg_ring.pending_count, ACQUIRE); UINT ring_send = __atomic_load_n(&queue_bypass->nspa_msg_ring.pending_send_count, ACQUIRE); if (ring_total > ring_send) ring_bits |= QS_POSTMESSAGE | QS_ALLPOSTMESSAGE; if (ring_send) ring_bits |= QS_SENDMESSAGE; wake |= ring_bits; }

Without this synthesis, check_queue_bits reports “nothing to do” even after the sender’s ntsync wake – the thread sleeps through ring deliveries until some unrelated event prompts it to call get_message, producing the 5 s dispatch-latency timeout seen pre-4.5.

Client-Side Ring Pop

Phase 4.6 (657c16691f5) adds nspa_try_pop_own_ring_send(), called from peek_message before the SERVER_START_REQ(get_message) block:

c if (signal_bits & QS_SENDMESSAGE && nspa_try_pop_own_ring_send(hwnd, first, last, &pop_type, ..., &pop_win)) { /* Filled info struct directly; skip server RTT entirely */ } else SERVER_START_REQ(get_message) { /* Legacy path: server arbitrates ring + legacy queue */ }

The client pop scans its own ring for a READY SEND-class slot, CAS-claims it (READY -> CONSUMED), decrements pending_count/pending_send_count, advances tail over any leading run of CONSUMED slots, and fills the received_message_info for the message pump. Same-process window filter (hwnd != 0) falls back to server because is_child_window needs server’s window tree.

Reply Path

When the window proc returns for a ring-origin SEND, reply_message() writes the result back:

c if (info->nspa_sender_tid && remove) { if (nspa_write_ring_reply(info->nspa_sender_tid, info->nspa_reply_slot, result, NULL, 0)) return; /* direct ring write + signal -- no server */ /* fall through to server reply_message on stale-slot */ }

nspa_write_ring_reply resolves the sender via the peer cache (nspa_lookup_peer(sender_tid), which mmaps the sender’s ring on first use), writes the LRESULT into the sender’s reply slot, and signals the sender’s queue sync event. The sender’s nspa_wait_ring_reply wakes, reads the result, and returns to the caller.

8. Opt-In Gating

Each capability layers on the previous. Stricter configs trade capture coverage for safety during validation.

Recommended configurations by use case:

9. Results & Ballpark Reduction

Exact A/B on a real DAW isn’t practical without a scripted identical workload. Ballpark from measured data: bypass OFF vs msg-ring full (all four opt-in layers enabled, 90 s active Ableton playback + GUI interaction).

Server message-path load

Ballpark ~200 – 500 wineserver RTTs / sec eliminated on the hot message path during busy playback. Server-side CPU spent in send_message + get_posted_message handlers roughly halves.

Latency-bound benefits (ns vs µs)

The ring also shortens the hot path for latency-sensitive traffic:

For RT-critical senders (AudioCalc, DWM-Sync, winejack JACK callback), removing server round-trips from the message-send path directly shortens the hot loop, not just reduces total throughput.

Qualitative wins

• Library panel populates with ring on (was broken on every earlier ring-in-session-shmem attempt – see footnote at end for the redesign story)
• MIDI / audio / UI interaction all behave normally with all four opt-in layers enabled
• MainThread sleeps properly in ntsync_schedule between events under active GUI stress
• No stale-slot / timeout errors under 90 s of active playback + GUI interaction

10. File Manifest

Total msg-ring feature: ~1500 LOC new code, ~250 LOC edits in existing files.

11. Phase History

What’s next

• Targeted RT profile harness. A scripted workload that exercises the AudioCalc -> MainThread and DWM-Sync -> MainThread SEND paths in isolation would give tight quantified RT-latency numbers (µs-level reply delivery). Today’s measurements are playback-scale.
• Phase 5 cleanup. Retire the Phase-1 bypass_locator.id != 0 sentinel in favor of a proper bypass_fd_valid protocol field. Pure cleanup, no functional change.
• Default-on promotion. After broader validation on non-Ableton workloads (WebView2, MS Office, games), flip the opt-in layers to default-on.
• Wake-bit synthesis extension. Wire ring-synthesized QS_* bits into places other than check_queue_bits if any remaining paths short-circuit without the ring-aware bits (audit pending).

Known scope limits

• Cross-process messaging stays on the server path (ring pointers are intra-process).
• MSG_CALLBACK / MSG_CALLBACK_RESULT stay on server (callback needs the server’s sender-callback state).
• DDE messages stay on server (DDE has separate registered-message handling).
• Packed-data SEND messages (MSG_OTHER_PROCESS) stay on server – ring slots don’t carry variable-length payload today. Expansion is possible with an extra side-channel buffer if needed.

12. Future Extensions (ranked by wineserver-load reduction)

Today’s scope captures roughly 48 % of get_posted_message server calls on busy Ableton playback (the 43,559 / 90,681 ring-arbitrated fraction from §9). The remaining 52 % is split across several message categories the ring doesn’t yet handle. Ranked by expected server-load reduction:

Tier 1 – highest traffic, biggest win

Tier 2 – broader app coverage, medium win

Tier 3 – targeted unlocks, smaller per-extension win

Out-of-scope (architectural mismatches)

• Cross-process messaging – ring pointers (HWND, WPARAM / LPARAM dereferences) are per-address-space. Cross-process messaging must stay on the server path unless we add a full translation layer (effectively a redesign).
• DDE – registered-message handling is separate from the normal message dispatch machinery; not a natural fit for the ring.
• Hardware input queueing – thread_input is a different shared structure with server-owned state; out of scope for the per-thread bypass ring.

Prioritization heuristic

For raw wineserver-load reduction on Ableton-style workloads, Tier 1 (timer + paint) is the clear target. For broadening to non-DAW apps (WebView2, COM-heavy tools, browsers embedded in apps), Tier 2 item 3 (thread messages) is the cheapest high-leverage add. Tiers 2-3 items are best adopted opportunistically as specific apps demand them.

Any of these extensions reuses the existing ring memory layout + cache discipline + fast-path atomics – the extension work is entirely in the “which message types get ring-delivered and how” layer, not in the memfd / seqlock / cross-thread plumbing.

Footnote: why memfd and not session shmem

The memfd design was not the initial plan. The first msg-ring implementation put the per-queue ring inside Wine’s session shmem via alloc_shared_object() – natural given the existing machinery. That produced a reliable Ableton Live regression: the library panel would not populate whenever the ring allocation happened for the process’s first thread (MainThread).

A systematic A/B matrix ruled out every runtime code path that reads or writes the ring. Gates tested (each with bypass on, each in isolation):

Every identifiable runtime side-effect (poison fill, ID bump, locator publish, seqlock ops) was proven innocent. The bug sat in the mere presence of a session_object_t entry + its shared_object_t header inside the shared session for the process’s first thread. The specific mechanism was never isolated further – all named side-effects were ruled out, leaving only a memory-layout / seqlock-interaction class of cause.

Moving the ring to a per-queue memfd eliminates all of it: no session_object_t entry, no shared_object_t header bump, no queue_shm_t locator publish, no interaction with session shmem’s bump allocator. The ring protocol itself (slot layout, state machine, cache discipline, fast paths) was unchanged – only the allocation + discovery layer swapped. Library regression resolved end-to-end.

See also: - nspa/docs/architecture.md – full Wine-NSPA architecture reference - nspa/docs/msg-ring-memfd-redesign.md – implementation plan + phase notes - nspa/docs/io_uring-architecture.md – orthogonal I/O bypass layer