Accuracy benchmark

DYNAMIC_DISPATCH is the edge type where static reading is most likely to be wrong: the call site names an interface method, and the target is whichever implementation happens to be live at that moment. So it is verified against execution. A fixture defines an interface Shape with three implementations (Circle, Square, Triangle); Main builds a Circle and a Square, never a Triangle, and drives all of them through the interface. Every implementation prints a line when its method actually runs, so the log is a real call log rather than a second reading of the source.

npm run bench:runtime compiles the fixture with javac, runs it, crawls the fixture directory, and reconciles the two.

1. Framework wiring is modelled one edge kind at a time, and most of it is left

The benchmark covers four edge kinds. Of the wiring that breaks static-only graphs most badly, only event and messaging publication is modelled so far (EMITS: Spring @EventListener and @TransactionalEventListener, Kafka, JMS, NestJS @OnEvent, with a confidence rule per channel kind). Still producing no edge at all: Pub/Sub and RabbitMQ, proxy boundaries from @Transactional, @Async and @Cacheable, Spring Data repository derivation, @Bean factory methods, @Qualifier/@Primary disambiguation, and @Scheduled/@Async re-entry. EMITS is also not among the 69 labelled cases, and the four benchmark projects turn out to contain no event or messaging wiring at all (0 EMITS edges across all four), so it is verified by fixture tests only rather than measured. Adding a fifth project with published events, and the remaining wiring, is the next step: one edge kind at a time, with a fixture, positive and negative tests, a confidence rule and a doc entry each.

2. INJECTS and ROUTES have no confidence model

Both edge kinds are written without a confidence value, so a consumer cannot tell a proven mapping from a derived guess. The project rule is that no edge type exists without a confidence model; these two are the standing exceptions. Recall has improved enough that this now matters more, not less: a Spring @GetMapping on a bean and a Lombok-derived constructor parameter are different strengths of evidence, and the graph reports both with no confidence at all.

Proxy boundaries took the other route. @Transactional, @Async and @Cacheable facts are categorical rather than scored, because proxyBlocked can name the exact reason a container could never honour an annotation (private, static, final, final-type) and inert marks a self-invocation that bypassed the proxy. That precision is not measured here either: none of the four projects below contains a proxied self-invocation, so proxy facts are fixture-verified only.

3. Dispatch is matched by name and arity, not by proof

A virtual dispatch edge carries a flat 0.80 and is emitted to every in-project subtype with a same-named, same-arity method. It is correct on the sampled labels, but that is partly because spring-boot-realworld's ArticleRepository has exactly one implementation. With several implementations, or overloads differing only by generics, the constant would overstate certainty. There is also no dispatch for lambdas or method references.

4. An ambiguous overload is flagged, not resolved

When a call site matches several overloads equally, the edge keeps the first declaration as its target and lists the rivals in alternatives at confidence 0.6. That is a rank, not a resolution: the graph still does not know which overload ran, and a reader who ignores the field gets the same confident wrong answer as before. Rivals are only detected inside one declaration list, so an inherited overload tie goes unreported, and none of the four benchmark projects contains an ambiguous overload - the behaviour is covered by fixtures only.

5. Foreign calls are unresolved by design

Calls into framework or library types resolve to an ext: node or to an __unknown__ marker, for example Repository<ArticleEntity>.save in the NestJS project. This is the correct trade — the tool does not have TypeORM's bytecode — but it means the graph stops at the framework boundary. Event and messaging wiring that crosses that boundary is only modelled when both the publisher's call and the listener's annotation are in index; proxy wiring still is not.

6. A perfect table on 69 edges is not a proof of the graph

Precision and recall of 1.000 here mean the graph answered all 69 sampled questions correctly. The four graphs contain roughly 12,000 edges, and the labelled cases were chosen to probe specific idioms rather than to sample uniformly: routes, injections, calls and dispatch on four projects. Whole categories — reflection chains, REFERENCES, EXTENDS/IMPLEMENTS hierarchies — carry no labelled cases, so a regression in them would not fail the benchmark. The floors in the manifest are a floor on the sample, not a certificate for the tool.

7. change_impact's ranking is untested against reviewers

change_impact ranks changed declarations with a hand-picked score: risk band, then routes, entry points, injection points, advice, event flows, then raw caller count. No benchmark measures whether that order matches what a reviewer would look at first, so treat the order as a convenience and read every kept entry. What is tested is the mapping, not the ranking: diff hunks onto declarations by line span, a member collapsing into its enclosing type, repo-relative paths resolving against crawl-root-relative ones, and unmatched surviving instead of being dropped. The tool adds no edge kind and no schema field, so its accuracy is exactly the accuracy of the edges underneath it.

Writing the first fixtures exposed a parser crash that silently deleted files. A lookup into a method table was resolving identifiers through Object.prototype, so any file containing a call to toString(), valueOf() or hasOwnProperty() threw during parsing. The crawler records parse failures per file, so those files vanished from the graph with only a log line.

On spring-petclinic this had hidden 7 of 50 files, including the Owner JPA entity: files parsed went from 43 to 50, CALLS edges from 574 to 1,426, and total edges from 1,408 to 2,637. The bug is fixed and covered by regression tests.

The Spring route cases failed on a missing feature rather than a bug. The framework descriptor declared the mapping annotations as entry points, but the Java parser never produced route registrations, so no controller could yield a ROUTES edge at all.

Spring mappings are now extracted: a class-level @RequestMapping prefix is joined with each verb annotation, covering the array form @GetMapping({"/vets"}), a named path = attribute, a bare annotation that inherits the class path, paths without a leading slash, and method = RequestMethod.PUT on the prefix-less form. Spring route recall went from 0.250 to 0.778 on its own; with the FastAPI router prefixes below, ROUTES recall reached 1.000, with no invented edge.

Injection failed the same way: the parser only recorded a parameter when an annotation asked for it. Spring has not needed one since 4.3 — the sole constructor of a bean is autowired silently — and a Lombok @AllArgsConstructor leaves the constructor out of the source entirely, so there was nothing to annotate in the first place.

Bean stereotypes now mark a class as container-managed, and its dependencies follow from that: the sole constructor's parameters when there is exactly one, the promoted fields when a Lombok annotation generates the constructor. The rules stay conservative in the ways that matter. Two unannotated constructors inject nothing, because Spring cannot choose between them either. A field initialised at its declaration is skipped. A class with no stereotype is skipped, since no container constructs it. A class that writes its own constructor alongside a Lombok generator injects nothing, because that is two constructors and the same ambiguity applies. Trusting a generator's field list blindly would have produced 0.95-confidence edges to dependencies the code never receives. Spring constructor injection alone took INJECTS recall from 0.133 to 0.667 and Java recall from 0.778 to 1.000, with precision held at 1.000 and no invented edge. Every remaining miss was FastAPI.

FastAPI failed for two independent reasons, and the benchmark had to be read carefully to separate them. First, APIRouter(prefix=...) was ignored, so every route from a prefixed router was emitted with the wrong path. Second, dependencies declared once as a module-level alias (SessionDep = Annotated[Session, Depends(get_db)]) produced no edge when a handler named the alias in its signature — and neither did a dependency reached only through another dependency function.

Both are now extracted, and writing the cases forced two corrections that were not obvious from the code. An injection belongs to the callable the framework populates, not to the module that declares it, so the edge now starts at read_items rather than at module:...items. And the dependency is the provider the framework calls, not the type it returns: Depends(get_service) is an edge to get_service, not to OrderService. Retargeting to the provider is strictly more accurate — the returned object is produced by that callable at run time — and it changed no property name or edge kind. INJECTS recall went from 0.667 to 1.000 and Python recall from 0.588 to 1.000, with precision held at 1.000.

The work also exposed an incremental-crawl bug in the same area: dependency aliases live in a Map, the parse cache round-trips through JSON, and JSON.stringify turns a Map into {}. Replayed files therefore carried no aliases, and — because an unchanged file's edges were never rebuilt when a different file changed — an edited alias left a stale, confidently wrong injection edge pointing at the previous provider. Alias maps are now packed explicitly, injection edges are rebuilt project-wide whenever anything changes (as DYNAMIC_DISPATCH and ADVISES already were), and both behaviours are covered by tests that crawl twice and then edit a Depends(...) target.