Repository navigation
[experiment] fully order independent type checking - #65064
Jake Bailey (jakebailey) wants to merge 148 commits into
Conversation
|
TypeScript Bot (@typescript-bot) test it |
|
Hey Jake Bailey (@jakebailey), the results of running the DT tests are ready. There were interesting changes: Branch only errors:
|
|
Jake Bailey (@jakebailey) Here are the results of running the user tests with tsc comparing Something interesting changed - please have a look. Details
|
|
Jake Bailey (@jakebailey) Here they are:
tscComparison Report - baseline..pr
System info unknown
Hosts
Scenarios
Developer Information: |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Well, that's a little bleak. |
7eda73a to
1846356
Compare
|
Not sure why top400 didn't come back either |
Checker results should not depend on which files were checked first, or on which types an editor happened to ask for before diagnostics were requested. Tests had no way to show such dependence, so fixes for it could only be described, not demonstrated. The harness-only checkOrder option queries the checker in a different order: files in reverse, each file in its own program, or every expression's type and symbol before any diagnostics. Listing several orders produces a baseline for each, so order dependence shows up as differing baselines.
The checkOrder option only tries a few fixed orders over whole files, but results can depend on which single query a checker answers first, as when an editor asks for the type of one node before anything else is checked. Such dependence is hard to find by hand and easy to reintroduce. With TS_CHECK_ORDER_FUZZ=n, each test is checked again n times in fresh programs that first answer a seeded random selection of type, symbol, and diagnostics queries. Any change in diagnostics or in the type or symbol of a node fails the test with the seed and a minimized list of queries that reproduces it. The fuzzer is off by default, so it does not slow the normal suite.
Asking the checker for types and symbols before requesting diagnostics changes the diagnostics and types reported for these tests. Recording them in that order too shows which later changes make the results independent of what was asked first.
These tests exercise checker results that can depend on the order in which declarations, files, and relations are checked: variance measurement cycles, relation caching under recursion and depth limits, instantiation limits, contextual re-checks, and deferred type reference cycles. Recording the current baselines makes later changes to make checking order independent show up as reviewable baseline diffs.
Variance measured through a cycle of recursive references used to depend on which type in the cycle was measured first, so reordering declarations or assignments changed which assignments were errors. These cases cover cycles through interfaces, type aliases, callbacks, and nested compositions, alongside annotated and structurally measured types whose results should not change.
The errors for a getter and setter that disagree on abstractness or accessibility were reported on both names by whichever accessor was checked first, guarded by a flag on the getter. When that happened during a query rather than a full check, the flag was set without the errors being reported. Since the errors only depend on the modifiers, each accessor now reports the error on its own name with no state, which yields the same set of errors in any order.
The type parameters of a type alias are resolved along with its declared type, but some readers took them from the links without resolving the declared type first, relying on it having been resolved earlier. Reading them through an accessor that resolves the declared type makes the dependency explicit, so the result does not depend on what was resolved before.
checkAssertion runs in several check modes, such as when a call is first checked without its context sensitive arguments, and each run overwrote the operand type kept for the deferred comparability check. That check used whichever was stored last, so its result depended on the order in which the assertion was checked. The deferred check now uses the cached normal type of the operand. The assertion itself only needs the operand type for `as const`, and the operand of a non-const assertion is contextually typed by the asserted type alone, so in any check mode the assertion computes the cached type the deferred check will read, with inference for an enclosing call hidden from the operand. This is the result the deferred check would produce on its own, and checks each operand once rather than again with fresh flow analysis in the deferred check.
Nothing reads or writes it.
The type of a destructured parameter's binding elements depended on
whether the parameter's type happened to be cached already. For
example, in non-strict mode
(({ [a()?.d]: c = "" }) => {})();
reported TS2537 when 'c' was queried before the IIFE call resolved
the parameter (its declared type is the undefined widening type),
but not otherwise. Conversely, in
f7({ a: {} });
function f7({ a: { x = 0, y = 0 } } = { a: {} }) {}
calling f7 first made 'x' and 'y' errors of type any.
Make the choice depend only on the declaration. Parameters of
context-sensitive functions take their type from contextual
assignment, so ask for the parameter's type. Everything else uses
the type derived from the declaration, as it did when nothing was
cached. Asking for the parameter's type unconditionally instead
introduces circularities in patterns whose initializers reference
sibling elements.
Asking for the type of a context-sensitive function's parameter can
resolve the enclosing call, which assigns the parameter its
contextual type from the inference in progress. The first query
nevertheless returned the type it computed itself from the final
signature, while every later query returned the assigned type. In
declare function fun<T>(g: (x: T) => T, h: (y: T) => T, x: T): T;
fun((Math.random() < 0.5 ? (x => x) : (x => undefined)), x => x, 10);
the last 'x' was number when the call was checked first, but any
when 'x' was queried first and its result was cached.
Return the assigned type whenever one exists, so that the first
query agrees with the rest, as the original comment intended.
When resolving a function expression's members reaches the function again through its enclosing call, the contextual check sees an empty signature list and gives up, yet it left the node marked as context checked. Whichever query got there first therefore decided whether the function's parameters were ever contextually typed, and diagnostics such as implicit any errors could be lost depending on check order. Leave the node unchecked in that case so a later request performs the check, and have the deferred function check request it, so that the check is performed at least once whichever query reached it first.
Generic calls nested in the arguments of other generic calls, or in context sensitive functions passed to them, depend on the enclosing call for their contextual type and parameter types, and the enclosing call infers from them in turn. When the inner call was queried first, it was already marked as resolving by the time its contextual type resolved the enclosing call, so the enclosing inference saw a placeholder and both calls cached results that a check in source order never produces. Computing the contextual type of a call nested in an argument of an enclosing call before resolving it resolves the enclosing call first, as source order does. Other contexts are not resolved ahead of time, since computing them may evaluate expressions that checking does not, such as the target of a destructuring assignment.
Asking for the symbol of a property access that hits an index signature stored the index symbol as the access's resolved symbol. Checking a delete expression reads that link to decide whether the operand must be optional, so a prior symbol query turned `delete a.x` on an index signature into an error that a plain check does not report. The type of the index symbol at such an access, which hovers show, is found by comparing it to the symbol the access resolved to, so that comparison now looks the index symbol up again rather than reading it from links.
A context-sensitive function's signature takes the type parameters of
a generic contextual signature, and its parameters their types, when
the function is contextually checked. Anything reading the signature
before then saw it incomplete and cached what it derived from it. Type
aliases in the body read the type parameters as outer type parameters,
so querying such an alias first resolved its conditional types eagerly.
In
declare function f(fun: <T>(t: T) => void): void
f(function self(t) {
const s = self;
type P = Parameters<typeof s>;
});
P was [t: unknown] when the call was checked first, but [t: T] when P
was asked for first, since s then had the uncontextual signature.
Split out assigning the contextual parameter types, which completes
the signature, from the rest of the contextual check, and have every
way to obtain the signature request it: the type of the function's
symbol, before anything resolves its members, and the signature of its
declaration. The rest of the check, inferring the return type and
checking the signature, remains for checking the function itself.
Contextual typing can reach the enclosing call while the signature is
being created, so keep the signature created there rather than replace
the one that was contextually typed. This also makes it unnecessary to
leave a function unchecked when its members are found being resolved.
A single signature type produced by higher order inference is instantiated with the full mapper combined in, since its signature can reference type parameters other than the outer type parameters of its declaration. The instantiation cache was keyed only by the mapped outer type parameters, so two instantiations differing in another type parameter (here the this type) shared whichever result was computed first. Which property access was checked first therefore decided the type of both, and checking `outer.log` gave it the type of `inner.log` contrary to the test's expectation.
Explaining why a name failed to resolve looks up other symbols, for example to find a spelling suggestion, and that can resolve aliases. When the failed name was the target of an alias, the suggestion lookup re-entered the alias being resolved, or skipped it, depending on which aliases were already in progress. In declarationEmitUnknownImport that decided whether two "Circular definition of import alias" errors were reported: they appeared when the import was checked first, and not when the export specifier was resolved first. Defer the explanation until no resolution is in progress, so it only observes finished results. The circularity errors that were artifacts of the suggestion lookup are no longer reported.
When one reference hit the flow analysis depth limit, flow analysis was disabled for every later reference in the same function or module body. The type of a reference then depended on whether another reference in the body had been analyzed before it. Each reference now gives up only when its own analysis hits the limit, so its type is the one it would get if it were analyzed first.
Its last use was in reportNonexistentProperty, which kept both a set of (name, containing type, JS) keys and the flag on the name's node. The flag already stopped a second report for the node on any containing type, so the keyed set never caught anything the flag did not. Keep a single set of reported name nodes instead. Reports only run while checking, so this stays checking-only state, as the keyed set was.
Measuring variances as a fixpoint changed the language: a type whose
recursive references nest its type parameters, such as
interface Foo1<T> { x: T; y: Foo1<(arg: T) => void> }
was measured invariant instead of covariant. On zod, the stricter
variances of ZodObject failed constraints (TS2344), sent comparisons
into deep structural recursion (TS2589), and made checking 250x slower.
Go back to TypeScript's rule: a recursive reference to a type being
measured has unknown variances, so occurrences nested in it are not
measured. That result depends on where the circularity is entered, so it
is measured from a canonical entry, the type with the smallest symbol,
as before the fixpoint.
The entry was not always found, though. A reference to a type measured
further down the variance stack counted as one of its marker types and
was compared structurally, never reaching its variances, so no
circularity was seen and the result depended on which type was measured
first. Only the measurement at the top of the stack has marker types
now; references to the others find the circularity and restart from the
canonical entry.
The variance tests now match TypeScript, except where TypeScript reports
a circularity that a different check order avoids in TypeScript too.
Each reentry of a computation in progress made every computation above it depend on it. addDep avoids duplicates only for the computation the dependency was most recently added to, so with several cycle heads in progress, as when relating deeply recursive types, every reentry walked the whole stack and appended the head again to each computation on it: on zod v4, 1.5M appends from 114K reentries, a quarter of all memory allocated. Walk down from the top instead, and stop at the first computation that an earlier reentry already made depend on the same head: the computations below it were in progress then too, so they already depend on it. Each computation records which of its dependencies memoCycle added, so other dependencies on the head do not stop the walk early. PERF-PLACEHOLDER
Once a cycle is detected inside a check, the outermost computation in progress switches to recording its effects as a query does, and stays that way until it finishes, even though the cycle usually closes long before. Every read after that pays for dependency accounting. When the last cycle, inner cycle, effects mark and discard log entry are gone, nothing computed since can be discarded anymore, so the effects recorded so far are replayed at the node each computation started at, as they would be when the outermost one finishes, and checking resumes applying effects directly. PERF-PLACEHOLDER
A measurement of variances re-entering one in progress marked the whole measurement circular, so everything computed until it finished was recorded as a query and then thrown away, even though almost none of it depends on variances. On zod, nearly all reads in a check were made under such a cycle, and a third of the instantiations were recomputations. Only comparisons and inferences read variances. As TypeScript does for Ternary.Unknown, a comparison that observed incomplete variances is not cached, nor is any comparison in progress around it. Once the outermost comparison completes, its result escapes to whatever asked, and only then is the measurement re-entered marked circular, so that what reads that result is discarded when it finishes. Comparisons made by a measurement itself are exempt: the variances measured are kept as before, since each is measured from the canonical entry whichever type is entered first. An inference reading incomplete variances marks the measurement circular right away. PERF-PLACEHOLDER
When a circularity among variance measurements restarts from its canonical entry, the restart sets the variances of each type it measures, and the measurements in progress below it only finish to throw their results away. Their comparisons read incomplete variances, so none of them is cached, and on zod's cluster of a dozen mutually recursive types, which restarts from a new canonical entry each time the cycle grows, that redone work was a tenth of all comparisons. A comparison that is part of a measurement whose variances are now set stops at once, as reading the incomplete variances of the measurement, so that what it compared is still not cached and the result is the same. PERF-PLACEHOLDER
Re-entering a memo map entry searched the whole resolution stack for the resolution computing it. Relation and instantiation maps keep hundreds of entries in progress during deep comparisons, so each begin paid for a scan of the stack per matching key. Each entry in progress now records the index of its resolution. PERF-PLACEHOLDER
Once any result has reports to replay, every read in a check looked for the result's memo state, through a filter sized for all states, which mostly misses cache. In a check with no effects pending, only reports are replayed, and few results have them, so a small filter of the keys with reports, and a flag per memo map, rule out nearly every read first. PERF-PLACEHOLDER
Chains that start a new memoized computation at every level, such as conditional types that compare instantiations of themselves, never reach the limit on one computation's depth and only stop at the limit on the whole instantiation. TypeScript stops every instantiation at a depth of 100, so a program it accepts never nests this deeply; the larger limit only needs to leave room for computations started below 100 to reach their own limit. At 1000, each runaway chain in zod v4 nested ten times deeper than TypeScript before stopping, with the same errors at any limit from 150 up. PERF-PLACEHOLDER
Few results have a memo state (about two thousand in zod v4), yet every computation probes the filter of their keys, which at four million bits missed the cache on most probes. Grow the filter with the states, keeping enough bits per state for few false positives. PERF-PLACEHOLDER
Cycle searches visit the resolution of every variance measurement on the stack, and each visit looks up the symbol's variances in a hash map. The symbol link store indexes by symbol id instead, as other symbol links do. PERF-PLACEHOLDER
Most memo frames on large programs wrap a read of a result that is already cached: the type of a type node, a type parameter's constraint or default, a symbol's declared type, members or exports, a module's exports. Beginning and ending the memo costs a hash insert and delete and a stack push for each. When no memo is active, a memo records and replays nothing, and a cached read enters no computation, so skipping the memo changes nothing, unless the computation is in progress: a class or interface publishes its declared type before it is complete, and re-entering it must still find the cycle. A type node's type is only read from its links where getConditionalFlowTypeOfType would leave it unchanged. PERF-PLACEHOLDER
Once any result has reports to replay, which happens early in most programs, every cached read while checking went out of line to the error filter. Only a type alias's declared type and the instantiations of type aliases and conditional types collect reports, so reads of other results now only account for pending effects. The alias's declared type gets its own kind of memo, which still looks for them. PERF-PLACEHOLDER
Whether a member can late-merge depends only on its parent, so keying the dense store by member filled one entry per member and missed for every new member of an already-seen parent. Key it by parent instead, which also makes the lateBindableKinds memo map redundant, so drop it. Members of one parent are usually looked up together, so also keep the last parent and its kinds on the checker. This is per-checker scratch and stays off the shared memo store. PERF-PLACEHOLDER
Beginning a computation in a check looked up the memo state of its result up front, though in the common case nothing reads it: the state is created lazily when the computation has effects to retain, and otherwise only compared against in attachState. For memo maps keyed by relation keys that lookup misses the cache, since the filter of keys with a state is large. Leave the state unset on the fast path, and have attachState peek it when it needs to compare. Peeking later can only find a state created since the computation began, which the computation's own writers already skip, so the comparison gives the same answer. PERF-PLACEHOLDER
Each call of a global type or symbol resolver looked up its result in the store by a key holding the global's name, hashing the string every time. Assigning each key an integer when its resolver is created lets the result live in a memo at that index instead, with no lookup at all. Resolvers are only created as a checker is initialized, in the same order, so checkers sharing a store agree on the integers, and the memos can be allocated once, after the first checker created them. PERF-PLACEHOLDER
Callers passing an error node computed awaited and promised types in their own context, so their errors belong to their node, but this also meant they never reused a cached result. Awaiting in async functions passes an error node, so most awaited types were computed again at every use. A promised type reports errors only when there is none, so a result is shared with every caller. An awaited type can report errors alongside a result, for a union some of whose members cannot be awaited, so only a result whose computation reported nothing is cached. Callers with an error node then reuse it, and compute in their own context only when there is no such result. PERF-PLACEHOLDER
Resolving a call nested in an argument of another call first computes its contextual type, so that the enclosing call is resolved first, as in source order. Checking in source order has nearly always resolved or begun resolving the enclosing call by then, and computing the contextual type again only to find that cost a walk through the enclosing call for every nested call. The enclosing call is now found directly, and the contextual type only computed when it is not yet resolving. PERF-PLACEHOLDER
Interning object literal types by content hashed every property name and type of each literal checked into a key, and looked the key up in a map keyed by that hash. The key starts with the node, so the types created for one literal are only ever compared with each other. Keep the types created for each object literal node instead, and compare their content with the new content directly. Few literals are checked more than once, so this is usually an empty list, and nothing is hashed. Comparing content directly also cannot confuse two contents whose hashes collide. PERF-PLACEHOLDER
Every checked node pushed a resolution so that results depending on errors located at it are discarded when it is left. Most nodes are checked directly within another, with nothing in between and no cycle in progress, and then the enclosing node's resolution can stand in: a cycle through the inner node can only start at that resolution while the inner node is current, and leaving the inner node ends it exactly where popping its own resolution would have. Results are unchanged; three quarters of the pushes go away. PERF-PLACEHOLDER
|
TypeScript Bot (@typescript-bot) test it |
1846356 to
c7ee0b9
Compare
|
Hey Jake Bailey (@jakebailey), this PR is in an unmergable state, so is missing a merge commit to run against; please resolve conflicts and try again. |
|
TypeScript Bot (@typescript-bot) test it |
|
Hey Jake Bailey (@jakebailey), the results of running the DT tests are ready. There were interesting changes: Branch only errors:
|
|
Jake Bailey (@jakebailey) Here are the results of running the user tests with tsc comparing Something interesting changed - please have a look. Details
|
|
Jake Bailey (@jakebailey) Here they are:
tscComparison Report - baseline..pr
System info unknown
Hosts
Scenarios
Developer Information: |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Now we're talking |
|
Jake Bailey (@jakebailey) Here are the results of running the top 400 repos with tsc comparing Something interesting changed - please have a look. Details
|
There's too much here to explain, but at a high level, this is a massive change which aims to make the checker entirely order independent, that is, it doesn't matter when you call the checker, which nodes you walk first, if you do or do not pull something, check some other file, helicopter into the wrong part of a cycle, etc, the behavior, diagnostics, everything, will be the same.
Looking at individual commits is my suggested way to look inside, not the entire diff, which will surely crash your browser.
This is all AI driven, so it's a bit hard to unbox (I plan to break this apart since some stuff can be done early), but some interesting tidbits (handwritten by me!)
There's a new
checkOrdertest option, which makes certain tests variant on interesting orderings, like "what if we visit every single node before checking" or "what if we reverse the order of the files" or even "what if we use PRNG to randomly walk nodes and then check". This up front found on the order of 500 failing tests we already had that were secretly order dependent. These were all fuzzed out and kept.There are a significant number of new tests which all show various ordering bugs never before discovered.
The first chunk of the PR are fixing places where code peeked at caches that "happened to be filled" instead of calling the actual function to pull on them.
Then, all of the ID-using code gets removed. At the end everything now has an ID independent ordering based on content, flags, the works. This caught a bunch of places where we were not caching things and ending up with types that were by all rights identical but had different IDs and then were unstable.
The big and scary change is the introduction of a new state management type dubbed "memo". Every single piece of checker state was moved into memos on a sidecar object on the checker (maybe for future sharing??). This memo system keeps track of the current computation, what depends on what else, eliminating basically all flags/bools that track "computed" or "resolved"-ness. This tracking also records side effects, including diagnostics, "referenced" marks, deferred checks, emit helper checks. The result is that if you query via the API and that calculates some error, those same errors will appear later if the checker runs.
With those data structures in place, now all of the cycles can be handled. Once a cycle is discovered, the checker starts logging what it's doing, and throws all of that info away assuming it's been poisoned.
All of the different recursion stacks get pulled into a single memo stack, which is a cool simplification too.
Variance is now a fixpoint algorithm, proceeding in rounds until stability. In practice, it doesn't iterate much.
Instantiation limits also stop being global numbers (since of course, that is order dependent side effect stuff). Instead, things get limited by the amount of work that's been done, and if you then pull on caches related to work, it tallies that work up to error consistently when needed. (I don't think this is fully done yet, though, since there are depth limits that are really hard to get right. Still thinking about that.)
In some cases, cycles still can't be fixed up, and so they're made consistent by doing what we've done previously for these cases, which is to pick a canonical place to restart to at least get it consistent.
Unfortunately, this all has a cost; the end of the PR tries to claw back perf, but at the end of the day, if we do more work tracking, we spend more CPU and memory tracking. It's still an open question whether this can be cheapened any more. But, my hope is that this idea is minimal enough that we could then optimize TS as a whole assuming this order-independence, e.g. work stealing, checker recycling, shared state, maybe even using LRU caches, or dumping data to disk.
But, it could also well be that this is slop and impossible to manage. But I don't want to sit on it anymore and want to see some real world effects.
Also, this is really meant to be rebased, so expect this all to be nasty nasty force pushes all the time as this gets tweaked. Unfortunately right now that's like 125+ commits, so that sucks.
However, I think that this will fix countless issues on the tracker. I have no idea how to evaluate all 5k issues to see if this PR fixes them. But it's gotta be a lot.