[gaborbernat]

Four failure and give-up paths in the read-write locks either left a lock believing in a hold that no longer existed, or kept state for calls that had already left. This PR ties ReadWriteLock's level (its count of nested acquisitions) to its SQLite transaction, makes the async task table trust the sync lock after a failed release, gives every waiting task its own ticket, and turns the async file locks' ticket chain into a queue. It builds on the task-table model from #832.

An interrupted ReadWriteLock release kept a level with no transaction

ReadWriteLock.release() rolls the SQLite transaction back and resets the lock level in two steps. The handler between them cleared the level only for sqlite3.Error, a shape that came in with #652. Python runs a signal handler between any two bytecodes. An exception from one, such as a SIGALRM job timeout or a KeyboardInterrupt that a retry loop catches, left _lock_level at 1 after the rollback had run. The next write_lock() on that instance, the per-path singleton by default, took the re-entrant shortcut and returned with no database lock. Another process acquired the same write lock meanwhile. The object raised and logged nothing until its next release.

stateDiagram-v2
    direction LR
    classDef ok fill:#d8f3dc,stroke:#2d6a4f,color:#1b4332
    classDef bad fill:#ffd6d6,stroke:#c1121f,color:#5c0000
    classDef info fill:#cfe4ff,stroke:#1f6feb,color:#0b1f3a
    Held: level 1, transaction open
    Rolled: level 1, transaction gone
    Free: level 0
    Held --> Rolled: rollback ran
    Rolled --> Free: level reset
    Rolled --> Rolled: interrupt between the two (before)
    Rolled --> Free: next acquire sees no transaction (after)
    class Held info
    class Rolled bad
    class Free ok

Loading

The release now clears the level for any exception once _transaction_ended() reports the rollback done. That check asks the connection's in_transaction, so it also covers an interrupt between the rollback and the flag that records it. An interrupt can still land on the way out of the cleanup, so an acquire that finds a level whose transaction ended drops it and locks again instead of nesting. The async wrappers ask the same question before they let a task join the level taken for tasks, so a phantom level does not admit readers through the table either. An interrupt before the rollback leaves level and transaction together, and the check leaves that alone.

A failed sync release put a hold back that the sync lock had dropped

When the sync release raised, _TaskOwners.release() restored the task's hold so a retried release could find it, which #789 kept on purpose. For AsyncSoftReadWriteLock that assumption does not hold. SoftReadWriteLock.release() clears its hold and stops the heartbeat (the thread that refreshes the holder record) before Participant.leave() commits the successor snapshot. That commit is the step that fails on a full filesystem with ENOSPC. From then on the table admitted every later reader, and the task whose release failed, into a hold with nothing on disk, beside a writer in another process. In-process writers waited forever.

sequenceDiagram
    participant T as task
    participant O as task table
    participant S as SoftReadWriteLock
    participant D as lock files
    T->>O: release()
    O->>S: release the level (executor)
    S->>S: hold = None, heartbeat stopped
    S->>D: commit successor snapshot
    D-->>S: ENOSPC
    S-->>O: OSError
    rect rgb(255, 214, 214)
        Note over O: before: restore the hold, readers join it with no lock behind
    end
    rect rgb(216, 243, 220)
        Note over O: after: sync lock holds nothing, so the table is left empty
    end

Loading

release() asks the wrapper, through its still_held callback, whether the sync lock still has the level. It restores the hold only while it does; otherwise it finishes the transition as released and lets the error propagate. BaseAsyncFileLock already asks its backend before deciding what a failed release means. A release that fails while the sync lock still holds keeps its hold for a retry, as before.

Every give-up behind a writer kept a future alive

All tasks waiting in the table parked on one shared concurrent.futures.Future through asyncio.wrap_future(). That registers a done callback which a concurrent.futures.Future offers no way to remove. A read that gave up through timeout or asyncio.timeout() left its callback and wrapper future reachable from the lock until the table next changed state, about 1.2 KiB each. Clearing a large backlog ran every stale callback on the event loop at once. A service answering requests with read_lock(timeout=...) while one task waited for a long hold grew at its own request rate.

flowchart LR
    W[task waits] --> B[shared future]
    B -. give-up leaves callback .-> B
    W2[task waits] --> T[own ticket]
    T -- give-up --> X[ticket removed]
    classDef ok fill:#d8f3dc,stroke:#2d6a4f,color:#1b4332;
    classDef bad fill:#ffd6d6,stroke:#c1121f,color:#5c0000;
    classDef info fill:#cfe4ff,stroke:#1f6feb,color:#0b1f3a;
    class W,W2 info;
    class B bad;
    class T,X ok;

Loading

Each waiting call now gets its own ticket. _admit() creates the ticket under the table lock, in the same critical section as the admission decision, so no wakeup is lost. A finally around the wait takes it out of the table, which covers Timeout and cancellation alike. _notify() settles only the tickets still registered, in arrival order.

Abandoned acquires settled their tickets by recursion

AsyncFileLock and AsyncSoftFileLock serialize acquire() and release() through a chain of tickets. Since #652 a call that gave up could not leave the chain, so it asked its predecessor to settle its ticket. When the polling task took the lock, the chain unwound as set_result → callback → set_result ..., four frames per abandoned call. Past about 250 give-ups behind one waiting acquire(), at the default recursion limit, concurrent.futures caught a RecursionError inside a callback. The rest of the chain stayed pending. release() did not return, async with lock: did not exit, and the lock file stayed held until the process exited.

flowchart LR
    H[holder] --> A1[abandoned 1] --> A2[abandoned 2] --> AN[abandoned ~250] --> R[RecursionError: tail never settled]
    H2[holder] --> Q[queue] --> N[next ticket]
    classDef ok fill:#d8f3dc,stroke:#2d6a4f,color:#1b4332;
    classDef bad fill:#ffd6d6,stroke:#c1121f,color:#5c0000;
    classDef info fill:#cfe4ff,stroke:#1f6feb,color:#0b1f3a;
    class H,H2 info;
    class A1,A2,AN,R bad;
    class Q,N ok;

Loading

The gate now keeps its tickets in a deque guarded by its lock. A call that gives up removes its own ticket. The holder hands the gate to the next ticket itself, so no ticket settles on another ticket's stack. Admission stays first in, first out. A ticket enters the queue inside the block that removes it, so an exception from a signal handler delivered while a call joins the queue does not leave a ticket behind, and a parked caller that finds itself at the head of the queue takes the gate itself.

Behavior changes

A release() retried after a failure that already dropped the sync-lock hold raises the existing "not held by this task" RuntimeError instead of reporting success. A ReadWriteLock.release() interrupted after its rollback leaves the object unheld. A retried release then raises "not held", as it already did after a failed rollback whose transaction ended.