Wine-NSPA – Local-File Bypass Architecture

This page covers the NtCreateFile local-fast-path itself, the local handle table and inode-sharing rules behind it, and the lazy-promotion path back into wineserver when server-owned state becomes necessary.

Table of Contents

1. Overview
2. Motivation
3. Design Principles
4. Vanilla Wine vs Wine-NSPA File Open
5. Handle Range & Per-Process Table
6. Shared Inode Table & Sharing Arbitration
7. Lazy Server-Handle Promotion
8. Dispatch Flow
9. Eligibility Criteria
10. NT API Coverage Matrix
11. File Manifest (post-reorg)
12. Debug Gating
13. Results & Profiler Numbers
14. Known Gaps & Roadmap
15. History

1. Overview

Wine-NSPA’s local-file bypass services a large subset of NtCreateFile calls entirely within the client process. Every eligible open would otherwise cost a full wineserver round-trip: the client builds a create_file request, the server allocates a struct file plus inode tracking plus a handle-table entry, returns a server-visible handle, and later NtReadFile, NtQueryInformationFile, or NtSetInformationFile often pay more server traffic on top of that. For an app like Ableton Live 12 Lite that does roughly 28,500 file opens in a single startup session – DLL manifests, .pyc files, theme resources, Live Library indexes – those round-trips dominate startup profile and show up as real latency on the main thread.

The current path is broader than the earlier public draft. It covers:

• regular-file opens with read-class and common write-class access
• explicit directory opens on the local dir-mint path
• selected downstream metadata, flush, and end-of-file operations without an immediate promote
• client-side file-backed sections built on top of local-file handles

The bypass routes eligible opens to a client-private handle range, maintains a per-process table that owns the unix fd, and exposes the unix fd to every Wine I/O path via a thin fast-path check inside server_get_unix_fd. When a later API genuinely needs server-owned file state, the bypass lazily promotes the local handle to a server-recognized handle on demand. Eligible file-backed sections are the main exception: they can stay local too, and are covered in detail on Local Section Bypass.

The feature is invisible to Win32 applications: same CreateFile semantics, same sharing arbitration, same IO_STATUS_BLOCK results, and the same error codes at the server boundary. Apps see identical functional behavior whether the bypass is enabled or not – the difference is measurable only in profiler output and perceived startup latency.

2. Motivation

Ableton’s startup profile exposed a large population of short-lived file opens:

Each open is cheap on its own (a few µs) but the aggregate is hundreds of millisecond-scale server traffic during startup – and the startup is happening on the main thread, which is where paint and UI dispatch live. Eliminating the server round-trip on these opens directly reduces time-to-first-paint and reduces steady-state priority-inversion risk on the RT audio path (server’s single-threaded main loop services all requests).

Other candidate workloads with similar profiles: plugin scanners (hundreds of VST probe opens), .NET apps (thousands of assembly-manifest reads at JIT time), installers (cache-file probes), and any Windows application using Python or Lua as an embedded runtime.

3. Design Principles

• Client-private handle range. The bypass issues handles from a fixed high range [0x7FFFC000, 0x80000000) that is disjoint from the server’s normal handle allocation (low-to-mid) and from the NTSync client-handle range. Any caller that does nspa_local_file_is_local_handle(h) can cheaply tell whether a handle is ours.
• Per-process table, not process-shared. Local handles are invalid in any other process. The table is a plain linked list under a PI mutex – no shared memory, no cross-process lookups. Cross-process handle duplication falls back to promotion through the server.
• Shared inode table for sharing arbitration. The only thing multiple processes need to agree on is sharing: if another process opens a file with FILE_SHARE_NONE we must honour that. A server-published shmem region carries (dev, inode) -> (aggregate-access, aggregate-sharing, refcount) so client-side arbitration matches what server-side check_sharing would enforce.
• Lazy promotion. On first API that genuinely needs a server-visible file handle, the bypass does a single nspa_create_file_from_unix_fd RPC that hands the unix fd to the server and gets back a real server handle. Subsequent calls on the same local handle reuse the cached promoted handle. Eligible file-backed sections are no longer part of that automatic promote set; they can stay local on their own section table.
• RT-safe fast path. Once the table + inode shmem are warm, the bypass open path is stat() + linked-list-walk-under-lock + open() + list insert – no syscall other than the two that are inherent to the work. No lazy-init remains on the hot path.
• Transparent fallback. Every disqualifier returns STATUS_NOT_SUPPORTED and the caller falls through to the normal server_create_file path. Anything the bypass doesn’t handle is handled by vanilla Wine unchanged.
• Tight correctness envelope. The current envelope is broader than the first public draft but still bounded: local regular-file and explicit-directory opens, synchronous opens, bounded create / overwrite dispositions, common metadata updates, bounded flush / EOF changes, and local file-backed sections. Anything outside that envelope still goes to the server.

4. Vanilla Wine vs Wine-NSPA File Open

5. Handle Range & Per-Process Table

5.1 Handle range

Local handles are allocated from the fixed range [NSPA_LF_HANDLE_BASE, 0x80000000) where NSPA_LF_HANDLE_BASE = 0x80000000 - NSPA_LF_HANDLE_CAP*4 with NSPA_LF_HANDLE_CAP = 4096. That gives an exact 16 KiB handle window disjoint from:

• Server’s normal handle allocation (starts at 0x4, grows up)
• Wine’s pseudo-handles (~0..~5)
• The NTSync client-handle range (near INPROC_SYNC_CACHE_TOTAL)

nspa_local_file_is_local_handle(h) is a constant-time range check: base <= h < 0x80000000 && h != 0x7FFFFFFF (the last exclusion is for the CURRENT_PROCESS pseudo-handle which would otherwise land inside the range). The check is called from every NT-API intercept site to decide whether to take the bypass path or fall through.

5.2 Per-process table

struct nspa_local_open {
    struct list       entry;
    HANDLE            handle;         /* local-range handle returned to app */
    HANDLE            server_handle;  /* lazy-promoted; 0 until first promote */
    int               unix_fd;
    unsigned long long device;
    unsigned long long inode;
    unsigned int      access;
    unsigned int      sharing;
    unsigned int      options;        /* FILE_OPEN options: SYNC_IO_NONALERT, etc */
    unsigned int      attributes;     /* OBJ_INHERIT forwarded on promote */
    WCHAR            *nt_name;        /* original NT path for GetFinalPathNameByHandle */
    USHORT            nt_name_len;
};

Protected by a single process-wide PI mutex (nspa_lf_opens_mutex). Linear list – walk is O(N) per lookup. For Ableton’s typical workload the list reaches a few hundred entries at peak; the walk is in the noise next to a server RTT it avoids.

Table add (on mint) and remove (on close) are the only writers. Every other operation (lookup, promote lookup) is a read under the same lock. The lock is a PI mutex because RT-priority threads occasionally open files at init and we cannot have a low-priority thread holding the lock against the audio callback.

6. Shared Inode Table & Sharing Arbitration

Windows file sharing (FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE) has cross-process semantics: if Process A opens foo with sharing=0, Process B’s open of foo must fail with STATUS_SHARING_VIOLATION. Any pure-local bypass has to see what other processes have done on the same (device, inode).

6.1 Shmem layout

The wineserver publishes a NSPA_INODE_BUCKETS = 1024 bucket hash table as a memfd-backed shmem region. Each bucket has 4 slots of (dev, inode, agg_access, agg_sharing, refcount) + a per-bucket PI mutex. Clients map the region read-only for arbitration lookups, read-write on the mutex word for publishing their own opens.

┌───────────────────────────────────────────────┐
│ nspa_inode_table_shm_t (~160 KB)              │
├───────────────────────────────────────────────┤
│  buckets[0..1023]                             │
│   ├─ lock_storage (pi_mutex_t, 64 B)          │
│   ├─ slot[0] (dev, ino, access, share, ref)   │
│   ├─ slot[1]                                  │
│   ├─ slot[2]                                  │
│   └─ slot[3]                                  │
└───────────────────────────────────────────────┘

Bucket index = hash(dev, inode) mod 1024. Slot selection is linear within the bucket (first free or matching). If all 4 slots are full and none match, the bypass returns STATUS_NOT_SUPPORTED and the open falls back to the server – this is an overflow-safety valve, not a correctness path.

6.2 Arbitration logic

nspa_local_file_check_and_publish_open atomically checks the existing aggregate against the new open’s access/sharing mask, returning STATUS_SHARING_VIOLATION if they conflict. Matching the server’s algorithm exactly:

• Pending opens add access to agg_access and intersect sharing with agg_sharing.
• A new open must satisfy: (agg_access & ~my_sharing) == 0 AND (my_access & ~agg_sharing) == 0.

This lives in nspa_local_file_check_sharing_algorithm(). The server-side publish hooks (nspa_inode_publish_slot) mirror the same rule from the server side whenever a non-bypass open creates or clears an inode entry. Arbitration therefore sees the union of bypass and non-bypass opens.

7. Lazy Server-Handle Promotion

The LF table returns a local-range handle to the application. Most Nt-API intercepts can service the call from the local unix fd directly (NtReadFile, NtWriteFile, NtQueryInformationFile for FileBasicInformation / FilePositionInformation / etc). Some APIs still require a server-visible file handle:

• NtDuplicateObject – dup goes through SERVER_START_REQ(dup_handle)
• NtQueryInformationFile for classes the server handles (e.g. FileNameInformation)
• NtQueryObject – ObjectName / ObjectBasic / ObjectType all server-side
• NtQuerySecurityObject, NtSetSecurityObject
• NtMakePermanentObject, NtMakeTemporaryObject
• NtCompareObjects

For these, the bypass lazily promotes the local handle: on first call needing server state, it issues a single nspa_create_file_from_unix_fd RPC:

1. wine_server_send_fd(unix_fd) – SCM_RIGHTS transfers a dup of the fd to the server
2. Server’s handler wraps the fd in a struct fd + struct file_obj + stores the NT path
3. Server calls alloc_handle and returns a normal server-range handle
4. Client stores the server handle in the LF entry’s server_handle field

Subsequent calls on the same local handle reuse the cached server handle – no second RPC. nspa_promote_if_local(h) is the one-line helper that every intercept site calls:

HANDLE nspa_promote_if_local( HANDLE h );

// Returns `h` unchanged if not local-range.
// Returns the promoted server handle (cached if already promoted) if local-range.
// Returns `h` unchanged if promotion failed (caller falls back to server path).

This is the lazy-promotion path. The alternative – eagerly promoting at mint time – was rejected because most file opens in Ableton’s workload never touch a server-requiring API; they read, maybe query a position, and close. Eager promotion would cost an RPC per open; lazy promotion costs an RPC per distinct file that escapes the read-only happy path.

7.1 attributes plumbing

The promote RPC forwards ObjectAttributes->Attributes (typically OBJ_CASE_INSENSITIVE, plus OBJ_INHERIT when bInheritHandles=TRUE is set on CreateProcess). The server’s alloc_handle_entry translates OBJ_INHERIT to RESERVED_INHERIT on the handle’s access mask, which is how Wine tracks inheritable handles for copy_handle_table during CreateProcess. Without the forwarding, inheritable local-range handles would be silently dropped by the inheritance walk.

7.2 2026-04-30 sync-parity fixes

Three small follow-ons closed correctness gaps in the local-file path without changing the architecture:

• directory-bypass promotion populates fd->nt_name, which fixed the start.exe NULL-Name crash
• directory bypass rejects FILE_NON_DIRECTORY_FILE instead of accepting a shape the server would reject
• the shared-inode check_sharing path arbitrates FILE_MAPPING_WRITE, matching server/fd.c::check_sharing

These are exactly the right kind of follow-on for this stub: preserve the fast path, preserve the fallback discipline, and close any remaining sync-parity gaps at the boundary.

8. Dispatch Flow

9. Eligibility Criteria

The file bypass covers a bounded but materially broader subset than the earlier public draft.

Accepted shapes today:

The important disqualifiers are still the same kind of boundary checks:

FILE_OPEN_REPARSE_POINT, FILE_WRITE_THROUGH, FILE_RANDOM_ACCESS, FILE_SEQUENTIAL_ONLY, FILE_EXECUTE, and FILE_DELETE_CHILD are no longer automatic disqualifiers when the surrounding open shape is otherwise within the local envelope.

When the open carries FILE_SEQUENTIAL_ONLY or FILE_RANDOM_ACCESS, the local path also applies the matching posix_fadvise() hint directly on the bypassed fd. That keeps the bypass path aligned with the server-side open finalization logic instead of silently dropping an advisory access-pattern hint.

10. NT API Coverage Matrix

Every handle-consuming NT API in ntdll/unix and server/ either:

• (intercept) has a local-handle intercept at its top that promotes on demand
• (fast path) services the call from the local unix fd without server contact
• (pass-through) doesn’t care about local handles (e.g. works on SID not handle)

11. File Manifest (post-reorg)

All NSPA-specific source lives under a nspa/ subdirectory in each module. Upstream Wine files carry only single-line intercept hook calls, keeping rebase-against-upstream conflicts minimal.

dlls/ntdll/unix/nspa/
├── local_file.c         -- LF table, bypass dispatch, promote helpers
├── local_timer.c        -- NT timer local dispatcher
└── debug.h              -- NSPA_TRACE macro, compile + runtime gated

dlls/win32u/nspa/
├── msg_ring.c           -- Message bypass (POST/SEND rings)
└── local_wm_timer.c     -- WM_TIMER local dispatcher

server/nspa/
├── local_file.c         -- inode-aggregation shmem + promote handler
├── local_file.h         -- server-side declarations
├── profile.c            -- wineserver per-request-type profiler
└── debug.h              -- server-side NSPA_TRACE macro

Upstream diffs against vanilla Wine are narrow:

• dlls/ntdll/unix/file.c: eligibility gate plus one-line intercept calls for the local-file surface
• dlls/ntdll/unix/server.c: nspa_local_file_try_get_unix_fd(), promote logic in NtDuplicateObject, LF close path, and local section close support
• dlls/ntdll/unix/sync.c: local-file and local-section NtCreateSection handling, plus two promote lines in NtMake{Permanent,Temporary}Object
• dlls/ntdll/unix/virtual.c: local section map / unmap / query hooks
• dlls/ntdll/unix/security.c: two promote lines
• dlls/ntdll/unix/process.c: alloc_handle_list was extended (prong A, currently deferred – see §14) + nspa_local_file_promote_inheritable() call before new_process RPC
• server/file.c: one call to nspa_lf_trace_promote() inside the existing nspa_create_file_from_unix_fd handler

12. Debug Gating

Trace emission is both compile-time gated (NSPA_DEBUG, default on; pass -DNSPA_DEBUG=0 for a release build) and runtime gated via cached env checks.

#if NSPA_DEBUG

#define NSPA_TRACE_ENABLED_FN(name) \
    static inline int nspa_trace_##name##_enabled(void) { \
        static int cache = -1; \
        int v = __atomic_load_n( &cache, __ATOMIC_RELAXED ); \
        if (v < 0) { \
            v = getenv( "NSPA_" #name ) ? 1 : 0; \
            __atomic_store_n( &cache, v, __ATOMIC_RELAXED ); \
        } \
        return v; \
    }

NSPA_TRACE_ENABLED_FN(LF_TRACE)
NSPA_TRACE_ENABLED_FN(LF_TRACE_SRV)
/* ... */

#define NSPA_TRACE(name, ...) \
    do { if (nspa_trace_##name##_enabled()) fprintf( stderr, __VA_ARGS__ ); } while (0)

#else
#define NSPA_TRACE(name, ...) ((void)0)
#endif

• First call per TU does a single getenv(); subsequent calls are a relaxed atomic load + not-taken branch when the env is unset (production default).
• Trace emission lives inside nspa/*.c – upstream Wine files have zero NSPA_TRACE calls. Trace-worthy hooks in upstream code (e.g. the LF fast path in server_get_unix_fd) have been extracted into helpers (nspa_local_file_try_get_unix_fd, nspa_promote_if_local_traced) that the upstream file calls, and all trace logic lives inside those helpers.

13. Results & Profiler Numbers

13.1 Later follow-ons

The original public numbers on this page captured the first local-file bring-up. The current feature set is broader, and the later follow-ons moved more traffic off wineserver:

13.2 Original full-stack playback snapshot

Ableton Live 12 Lite, 95-second playback window, NSPA_PROFILE=1 with all prod gates. Baseline is the pre-LF fullprod run (2026-04-21); “post-LF” is the 2026-04-23 run after the complete stack landed.

The 99% drop on send_message is msg-ring (documented separately) rather than LF – they compose, and the full NSPA bypass stack is what produces the aggregate numbers. LF’s direct contribution shows as the zero rows on create_file / close_handle / get_handle_fd: those are steady-state during playback, but during startup the LF bypass eats roughly 28,500 file opens that would otherwise each cost a server RTT plus a get_handle_fd return-trip.

The bottom-line metric is server handler CPU: 16.8% less server work across the board despite a 10x higher raw request count. The replacement traffic (ring wakeups, hook chain) is ~0.05 µs per request where the replaced traffic was 8+ µs per request.

14. Known Gaps & Roadmap

14.1 Cross-process DuplicateHandle of a local-range source

The same-process path is covered. Cross-process dup where the source lives in another Wine-NSPA process’s local-range is not – the server has no access to the remote’s LF table. Fix would require a cross-process LF promotion RPC. Rare in DAW workloads; parked.

14.2 STARTUPINFOEX PROC_THREAD_ATTRIBUTE_HANDLE_LIST local-range inheritance

The synchronous get_or_promote variant for explicit handle-list inheritance was deferred because the per-handle promote RPC on the CreateProcess-calling thread surfaced as a visible menu-content-paint delay (“black menu flash”). Legacy bInheritHandles=TRUE via nspa_local_file_promote_inheritable is unaffected and covers the common case.

Proper fix options (ranked):

1. Batched promote RPC – single server round-trip that promotes an array of local handles. Caps the CreateProcess cost at one RTT regardless of list length.
2. Async pre-promotion at mint time – if the open carried OBJ_INHERIT, fire the promote RPC off the critical path so the server handle is already cached when alloc_handle_list runs. Lower CreateProcess latency but higher complexity.

Relationship to 14.1

14.1 and 14.2 share the same underlying shape – “an LF handle must become a real server handle before it crosses a process boundary” – but the fix surfaces differ:

• 14.2 is in-process. The CreateProcess caller owns both the LF table and the calling thread, so the unix fds are locally accessible. Option 1 is a pure “batch the existing promote RPC” change.
• 14.1 is cross-process. Process B holds a handle that refers to process A’s LF table. Neither B nor the server has the unix fd. Servicing it requires the server to wake process A and have A run the promote on its own entry before the dup can proceed. That needs new server-to-peer-client signaling, which is not LF infrastructure – it is closer to how the msg-ring wakes peer threads.

The two fixes compose: 14.2’s batched RPC is a prerequisite of 14.1. Once nspa_promote_local_handles exists as a handler, 14.1 can reuse it on process A, driven by a new “remote promote” request where process B asks the server to wake A and invoke it. 14.1’s complexity is then the wake mechanism, not the promote itself. Ship 14.2 first; 14.1 composes on top.

14.3 Remaining boundaries

The current envelope already covers the high-volume surfaces that justified the feature. Anything outside it still falls back cleanly. The remaining boundary cases are the ones that still want an authoritative server path today:

• FILE_DELETE_ON_CLOSE
• open-by-file-id
• full directory enumeration (NtQueryDirectoryFile)
• FILE_DELETE_ON_CLOSE (temp-file semantics)
• cross-process duplication and inheritance corner cases that still require a real server handle at the boundary

None of these has been worth forcing client-side yet; keep them on the server path until a workload demonstrates otherwise.

15. History