8 Closed-Loop Design Execution
“Beware of bugs in the above code; I have only proved it correct, not tried it.”
— Donald E. Knuth, Notes on the van Emde Boas Construction of Priority Deques (1977) (Knuth 1977)
Knuth, Donald E. 1977. Notes on the van Emde Boas Construction of Priority Deques: An Instructive Use of Recursion. https://cs.stanford.edu/~knuth/faq.html.
Author’s Note. Just as Donald E. Knuth famously separated abstract reasoning about code from the hard evidence of running it, we must bring the same discipline to our architectural studies. A well-defined comparison, a plausible teaching fixture, or a carefully reasoned plan does not transform into execution evidence until we run our tools and retain the resulting artifacts. The prospective cache record and the executed array record we present in this chapter lead us to different conclusions.
North-Star question
What hard conclusions can be drawn when the prospective Lighthouse cache study fails to run, and what distinct insights can the separate, fully executed array study firmly establish?
We drive architectural research questions through a loop of candidate selection, execution, measurement, and decision-making. We present two distinct records that reach different stopping points. We isolate them to highlight distinct phases of our methodology, avoiding an artificial end-to-end narrative.
Our first record, the Lighthouse cache study, distills a broad 3 W RISC-V system objective into a single pre-RTL decision on L2 capacity. We treat this record as prospective, mapping out required comparisons while exposing the uncompleted engineering effort blocking simulation. It never executes, leaving baseline and candidates unresolved. In contrast, our second record is a fully realized retained systolic-array study using SCALE-Sim, a cycle-accurate simulator purpose-built for systolic accelerator architectures, with fidelity scoped to the processing-element array and its on-chip traffic rather than the off-chip memory system (Raj et al. 2025). Across twelve simulator evaluation events, it demonstrates the insights that real execution provides. However, because it evaluates isolated GEMM layers rather than full subsystem pipelines, it cannot resolve our prospective cache evaluation gaps or validate our 3 W target.
Learning objectives
This chapter establishes the following learning objectives:
- Navigate an architectural research question from fixed state to a stopping decision.
- Distinguish prospective experimental plans from executed empirical evidence.
- Isolate the architectural outcome, AI contribution, mechanism result, and actionable next steps.
- Use ties, failed causal explanations, and provenance gaps to scope follow-up studies.
8.1 Prospective Cache Design Study
Every architectural evaluation begins before a single simulation cycle executes, at the moment we lock our experimental parameters and isolate the target trade-off. When we attempt to size an L2 cache for a mobile XR subsystem, the engineering temptation is to launch immediate simulator sweeps and let the tool outputs dictate the decision. Yet without a declared baseline, fixed ISA invariants, and pre-bound stopping criteria, raw simulation numbers yield ambiguous claims rather than actionable architectural decisions. Formally declaring our design space and execution prerequisites up front separates prospective intent from executed, tool-bounded evidence and from later physical or silicon evidence.
8.1.1 Architectural Decision
Balancing memory latency against static leakage power inside a strict 3 W thermal design power (TDP) budget forces us to confront a central hardware trade-off. We must determine whether expanding L2 cache capacity eliminates enough costly DRAM accesses to justify its physical area and leakage overhead. In our Lighthouse mobile extended-reality (XR) subsystem (Chapter 1), mapped to a TSMC N7 or 3 nm-class low-power process, this question determines whether a larger L2 cache should advance to register-transfer level (RTL) evaluation. Rather than deploying AI search heuristics, we frame this choice as a direct, prospective comparison across our 8-layer prompt stack. We hold our RV64GCV ISA contract, vector-capable CPU and NPU compute topologies, extended-reality workload snapshots, and runtime compiler passes fixed, isolating L2 capacity as our single swept architectural variable. Our comparison tests a 2 MiB baseline against 3 MiB and 4 MiB candidates. These capacities define the distinct architectural objects within our prospective study, though they remain specification targets rather than generated RTL instances or measured artifacts in our repository.
While capacity serves as our only swept architecture parameter (limited to the 2, 3, and 4 MiB choices), this does not imply that every underlying hardware request remains identical. A larger cache capacity demands a different set count, bank organization, or SRAM macro composition. Before we dispatch any simulation jobs, we must establish a rule that derives these dependent structural values. At the same time, this rule must hold our core microarchitectural features constant, including associativity, line size, indexing scheme, replacement policy, miss status holding register (MSHR) count, port configurations, hit-latency allocation, and our process, voltage, and temperature (PVT) assumptions. Because our repository currently lacks both this structural derivation rule and the explicitly bound values for our controlled cache and PVT parameters, we cannot yet execute the comparison.
Our fixed architectural state extends far beyond the cache itself. It declares our network-on-chip and memory-system configurations, our 2 GHz host RV64GCV vector-capable CPU microarchitecture, our JIT compiler and runtime lowering environment, our RISC-V software image, and an XR workload snapshot featuring stable frame identifiers and boundaries.
We plan an evaluation path across eight functional tool categories. These comprise full-system CPU and microarchitectural simulation via gem5, a modular full-system architectural simulator (Binkert et al. 2011), and Verilator, an open-source cycle-exact SystemVerilog simulator; gate-level physical synthesis and timing signoff via Yosys, an open-source RTL synthesis framework, OpenROAD, an open-source automated physical design suite, and OpenSTA, a gate-level static timing analyzer; analytical SRAM area and access-time modeling via CACTI; power-domain intent validation using IEEE 1801 Unified Power Format (UPF), an industry standard for declaring multi-voltage power domains; formal property verification using SystemVerilog Assertions (SVA) evaluated through Bounded Model Checking (BMC); thermal integrity signoff using Ansys RedHawk-SC, an industry-standard power grid and thermal solver; off-chip DRAM memory modeling using Ramulator, a fast cycle-accurate DRAM simulator; and multi-chiplet die-to-die interconnect interfaces compliant with Universal Chiplet Interconnect Express (UCIe) and Compute Express Link (CXL) standards for coherent memory expansion. However, we have not yet bound these conceptual tool roles to executable scripts or retained measurement pipelines in our repository.
Binkert, Nathan, Bradford Beckmann, Gabriel Black, et al. 2011. “The gem5 Simulator.” ACM SIGARCH Computer Architecture News 39 (2): 1–7. https://doi.org/10.1145/2024716.2024718.
At its core, our architecture claim asserts that a larger L2 cache will reduce 99th-percentile frame times by eliminating capacity misses, all while satisfying our power, area, access-time, functional correctness, and coherence constraints. We cannot validate this claim simply by showing a reduction in miss rates. Instead, our matched frame observations must prove that resolving these capacity misses directly improves our tail-latency outcomes under the fixed workload and software conditions.
Because our legal design space contains only the baseline and two discrete alternatives, our study relies on direct evaluation, as discussed in Chapter 5. We avoid introducing generators, predictors, optimizers, or learned search methodologies. Our SRAM model acts as a component of the declared tool path rather than as a newly integrated AI module. Omitting these advanced methods ensures our cache study carries no extraneous AI-contribution claims to test. Instead, direct evaluation provides the most streamlined approach capable of answering our targeted architecture question, assuming our tool path is operational.
We expect a focused result. Our study can, at most, recommend a single cache capacity to advance toward RTL evaluation. It cannot authorize a full implementation, final signoff, tapeout, or any binding product commitment. We leave broader architectural explorations (such as cache replacement policies, prefetching mechanisms, floorplanning, and thermal signoffs) outside the scope of this study.
8.1.2 Execution Prerequisites
Translating an abstract architectural proposal into a reproducible simulation job requires converting high-level capacity targets into concrete, hardware-derived configuration structures. A candidate capacity of 3 MiB or 4 MiB cannot simply be passed to a cycle-level simulator as an arbitrary integer; it dictates physical SRAM set counts, bank topologies, array geometries, and hit-latency profiles that must obey invariant physical rules.
Translating our high-level L2 capacity targets into executable simulation jobs takes a complete operational declaration (Table 8.1). Each parameter category fixes the physical and procedural invariants required before dispatch, and together they expose the missing structural derivations and unbound policy settings that currently block simulation. Framing an evaluation plan requires pre-binding every hardware parameter and stopping threshold long before launching a single simulation job.
| Run parameter | Declared Lighthouse cache study work |
|---|---|
| Candidate requests | We must prepare one complete request to the SRAM model under our declared PVT conditions for the 2 MiB baseline, 3 MiB candidate, and 4 MiB candidate. We will treat l2.capacity as the sole swept architecture parameter while strictly holding associativity, line size, indexing, replacement, MSHR count, ports, hit-latency allocation, and PVT fixed. A prospectively bound rule must derive set count, bank count, layout geometry, modeled periphery, and SRAM macro composition directly from this capacity. We will retain every controlled and derived value alongside our SRAM-model and underlying technology identities. Because our repository currently lacks these explicit bindings, our requests remain unfinished. |
| Cycle-level requests | We must rerun the 2 MiB baseline alongside our eligible candidates through our cycle-level simulator. This execution includes our cache and power models while holding the declared XR workload snapshot, the software image, and all other hardware and software identities strictly fixed. |
| Dispatch condition | We consider no request eligible until we fully bind the controlled policy parameters alongside all capacity-derived set, bank, geometry, periphery, and macro choices. Furthermore, a cycle-level request becomes eligible for execution only when (return_complete == true) && (schema_valid == true) && (area_mm2 <= 1.5) && (access_time_ns <= 2.5). We use this condition strictly to authorize a simulation request; it does not independently validate a measurement or definitively reject an architecture candidate. |
| Required returns | We must retain SRAM area, access time, cache-local dynamic energy, leakage, matched frame records, frame-time inputs, and the critical 99th-percentile frame time. We also require a detailed breakdown of cache misses by classification, overall subsystem dynamic and leakage power, tool versions, model identities, translated inputs, warnings, failures, cryptographic hashes, data lineage, and precise accounting records covering queue wait times, run times, compute resource use, license checkout, storage footprints, and human setup time. However, we have not yet defined the operational procedure that will generate these cost records. |
| Checks | We require identical ISA-visible outputs across our declared tests and mandate the same rigorous functional regression suite for every evaluated capacity. We demand zero failures from the simulator’s built-in coherence-invariant checks, modeled subsystem power (dynamic plus leakage) capped at 3 W, total cache macro area at or below 1.5 mm2, and a modeled access time not exceeding 2.5 ns. Ultimately, we require an uncertainty-qualified frame-time comparison against our rerun baseline. We treat detailed thermal and IR-drop signoff as independent requirements reserved for later design stages. |
| Failure handling | We will actively refuse any invalid evaluation request prior to simulation invocation. When failures occur, we must cleanly distinguish among timeouts, job cancellations, underlying infrastructure failures, stale state corruption, missing outputs, and parser errors. If we authorize a retry, we must meticulously link it to its parent attempt, deduct it from our cycle-level budget, and launch it from a freshly generated clean state without altering the underlying architecture candidate. |
| Budget | We permit three total SRAM model evaluations and restrict our cycle-level executions to a maximum of four, explicitly accounting for any infrastructure-driven reruns. We do not include any RTL synthesis or physical signoff in this exploratory budget. Before executing the study, we must explicitly define how we account for queue and run times, compute consumption, license constraints, tool utilization, storage costs, and human engineering effort. This accounting procedure remains unfinished, and we must explicitly define the specific tool returns that will supply each cost metric. |
| Stopping | We stop any simulation attempt the moment its queue-wait or run-time deadline expires. We will halt the entire study when we resolve the intended architectural decision, exhaust all eligible capacities, consume our cycle-level budget, breach the absolute project deadline, lack a required input or tool, or discover that no alternative meets our stringent SRAM physical limits. Currently, the numeric queue-wait, run-time, and absolute deadlines remain unresolved pre-execution tasks that actively block our dispatch. |
| Decision | Our baseline direction rule requires an uncertainty-qualified frame-time improvement strictly greater than zero. Before initiating execution, we must prospectively define our statistical estimator, interval or confidence procedure, and any practical performance margins. Once we fix those criteria, we will recommend at most one capacity for subsequent RTL evaluation, and only if its required checks pass cleanly and its qualified comparison against the rerun baseline proves the declared performance improvement. |
While our declaration explicitly fixes the required improvement direction, it does not mandate the specific statistical procedure for judging it. We must define our estimator, interval or confidence procedure, and any practical performance margin prior to simulation dispatch. We enforce this constraint to guarantee that our architecture team cannot retroactively choose a favorable statistical interpretation after observing the returns.
We specify all declared capacities and serialized values in mebibytes (MiB), where 1 MiB equals \(2^{20}\) bytes (National Institute of Standards and Technology n.d.). Throughout our study, these three capacity labels function as stable, immutable specification names. To maintain experimental hygiene, an executable request must cleanly propagate its declared MiB token and exact byte count into every dependent downstream request long before any baseline comparison begins. Because our repository currently lacks these parameter bindings, derived structural rules, and statistical procedures, this prospective study cannot dispatch, halting our first design loop directly at the specification boundary.
National Institute of Standards and Technology. n.d. Definitions of the SI Units: The Binary Prefixes. NIST reference documentation. https://physics.nist.gov/cuu/Units/binary.html.
8.2 Halting Design Loops Before Dispatch
When an architectural evaluation halts prior to simulation dispatch, the resulting record is often mischaracterized as a complete failure. In practice, an unexecuted study establishes a methodological boundary. It preserves our architectural intent while preventing unverified assumptions from being presented as empirical evidence. Scrutinizing the artifacts retained in our repository allows us to separate prospective specification identities from true execution traces, clarifying why our L2 cache decision remains open and defining the prerequisite infrastructure required for future simulation runs.
8.2.1 Specification Audit
Distinguishing between an unexecuted architectural plan and an empirical simulation result requires an audit of repository artifacts. Prospective specifications declare what a study intends to compare, while teaching fixtures illustrate how measurement pipelines ought to behave; neither class of artifact provides empirical evidence of hardware performance. Independently replayable execution evidence requires retained invocations, serialized state, outputs, identities, and lineage sufficient for another reviewer to reconstruct what ran. A narrower local result can still support a bounded claim when some provenance is missing, provided the record names those gaps and does not claim independent replay, causal attribution, or physical validation.
To audit our repository state, we categorize our retained artifacts in Table 8.2. Our repository contents fall into prospective specification identities (such as our controlled cache parameters and declared tool roles), author-constructed teaching fixtures (such as illustrative SRAM and simulator entries), and executed evidence packages. It preserves substantial design intent and pedagogical structure yet retains zero executed cache binaries, simulator logs, or parsed measurements. Specification documents and teaching fixtures clarify methodology, but they cannot support a measured cache result.
| Retained item | Evidence class | What it supports | What remains absent |
|---|---|---|---|
| Lighthouse cache study, current design revision, controlled-state identities, and the three cache candidates | Specification identity | The cache question, comparator, intended controlled state, permitted capacity change, required measurements, call-count limits, stopping categories, and intended pre-RTL decision. | Bound policy parameters, capacity-derived cache structure, retained candidate artifacts, executable requests, runs, measurements, and a cache disposition. |
| Run declaration and its declared tool roles | Specification identity | The intended request translation, dispatch condition, return schema, failure handling, call-count limits, and tool responsibilities. | Numeric deadlines, a cost-accounting procedure, bound executables, wrapper and parser versions, exact inputs, host state, commands, logs, outputs, and realized cost. |
| Constructed SRAM and simulator entries plus a design-loop card that summarizes the proposed question, state, returns, checks, and decision | Author-constructed teaching fixture | Illustrative area, access-time, power, failure, return, and summary properties that a study record might retain. | Terminal and parser status, exact requests, raw artifacts, state identities, dispatch evidence, lineage, realized cost, and any measured cache result. |
| Retained cache execution package | Executed evidence | None. No such package is present. | Candidate artifacts, tool requests, raw outputs, parsed measurements, hashes, lineage, costs, qualified comparisons, and a decision record. |
These specification entries and teaching fixtures clarify our design intent and operational structure, yet they leave the physical feasibility and performance of each capacity candidate unmeasured.
Design principle: Prospective intent versus executed evidence
The principle: Architectural intent requires absolute separation from executed proof. A prospective specification frames a search space and fixes what a study would compare; it establishes nothing about how a candidate behaves.
The application: Only an executed run returns evidence, and that evidence is bounded by what the executing tool actually models. Conflating an unexecuted plan with a modeled outcome, or a modeled outcome with a physical one, destroys empirical validity.
8.2.2 Analytical Judgments
When an architectural study encounters an infrastructure block before dispatching simulation jobs, our responsibility as architects is to decouple the study’s structural status from its technical conclusions. Modern hardware literature frequently conflates tool availability with architectural validity, mistaking a working script for a proven hardware insight.
To prevent confusing an unexecuted script with a proven hardware insight, we present our analytical framework in Table 8.3. Judged along four axes, our architecture outcome remains unresolved, our AI contribution and hardware mechanism remain untested, and our stop is dictated entirely by missing execution prerequisites rather than budget exhaustion or candidate failure. Decoupling these four axes (architecture outcome, AI contribution, hardware mechanism, and stopping condition) prevents us from misinterpreting an unexecuted tool path as a completed architectural evaluation. This framework does not replace our comprehensive evaluation map in Chapter 10, which remains central for comparing entire agentic workflows across various hardware tasks, costs, and operating conditions.
| Judgment | Cache conclusion | Evidence boundary |
|---|---|---|
| Architecture outcome | Unresolved. No capacity advances, and none is rejected. | No retained cache candidate was executed, the baseline was not rerun, and no required comparison was qualified. |
| AI contribution | Not tested. | Direct evaluation with no added learned method was the declared approach. No AI-versus-conventional comparison exists. |
| Mechanism | Untested. | No retained miss classifications or matched frame observations test whether reduced capacity misses change 99th-percentile frame time. |
| Stopping | Stop before execution because the implementation, measurement, and dispatch prerequisites are unavailable. | The declared budget was not consumed. If the cache decision remains worth pursuing, a new run requires bound policy and derived cache choices, numeric deadlines, a cost-accounting procedure, candidate artifacts, tool bindings, request translation, and an executable measurement path. Deferral, cancellation, or reformulation remain legitimate. |
Our stop belongs before execution. We did not halt due to budget exhaustion, nor did we suffer a failed candidate or a stagnant “no-change” architectural result. Instead, this premature stop dictates that we cannot authorize any of our three proposed cache capacities for downstream RTL implementation.
8.3 Executed Systolic-Array Study
In contrast to our unexecuted cache study, our separate systolic-array experiment begins with concrete cycle counts and bandwidth traces returned by an operational simulator. Yet obtaining raw execution numbers is only the first step in architectural evaluation. Interpreting these results requires measuring them against a frozen research question, ensuring that unexpected simulator returns do not prompt retrospective changes to our scoring metrics or candidate filters. As we evaluate our retained NPU accelerator traces, we demonstrate how empirical data informs our design decisions while exposing the subtle boundaries of local accelerator proxies.
8.3.1 Freezing Candidates
Preregistering an architectural experiment locks our hypothesis, baseline configuration, and interpretation rules before simulation tools return a single performance cycle (Nosek et al. 2018). In our retained systolic-array package (arch2_scale_ai_study_v1), we freeze our legal processing-element geometries, evaluation metrics, simulation budgets, rejection checks, and decision margins prior to candidate selection. This prospective commitment makes post-hoc changes visible rather than preventing them by itself.
Our study evaluates systolic-array geometries across three synthetic general matrix multiplication (GEMM) layers. As is standard in accelerator design, a GEMM operation multiplies an \(M \times K\) activation matrix by a \(K \times N\) weight matrix to yield an \(M \times N\) output matrix. We record three distinct \((M, N, K)\) dimension tuples of 128 by 128 by 256, 64 by 192 by 128, and 32 by 128 by 192. Our architectural purpose here is to compare various systolic-array dimensions under a single, fixed workload and simulator configuration. Consequently, we treat these layers as isolated matrix multiplications rather than complete execution traces drawn from an identified spatial-tracking pipeline or a full XR application.
For this NPU accelerator evaluation, we employ SCALE-Sim (Raj et al. 2025). SCALE-Sim models processing-element (PE) array mechanics, line buffers, and SRAM traffic under a weight-stationary dataflow. SCALE-Sim v3 can integrate Ramulator for a more detailed memory trace path, but our retained configurations explicitly set UseRamulatorTrace = False. The study therefore does not model memory-channel contention, row-buffer hits, bank conflicts, or DRAM command timing.
In our execution setup, we systematically vary array height and width in SCALE-Sim 3.0.0, holding our architecture to a weight-stationary dataflow, a fixed 128 KiB capacity for each of the three on-chip SRAMs, and configured interface bandwidth values of 64. Our execution pipeline dispatches SCALE-Sim via Python wrapper scripts (run_scalesim.py), ingesting CSV workload files that define matrix dimensions for each GEMM layer and outputting structured performance summaries alongside DRAM bandwidth trace logs (BANDWIDTH_REPORT.csv). We derive these specific design choices directly from the package’s frozen preregistration and our retained per-run configurations, rather than relying on default parameters from the original simulator paper.
Selecting weight-stationary execution ensures that weight tiles remain stationary at the processing elements while activation values stream through the array and partial sums accumulate, reducing repeated weight data movement (Raj et al. 2025). Our retained study enforces a 1,024-processing-element hardware limit, a 90,000-cycle performance cutoff, and a mandatory 32 by 32 baseline. We allocate exactly four simulator evaluation events per proposal arm, alongside four shared mechanism-probe events. Although we record an 800 MHz clock frequency to support optional derived latency calculations, we base every architectural judgment in this section on cycle counts rather than wall-clock time.
We establish a fixed heuristic to serve as the budget-matched method comparator for our frozen study. To evaluate candidates intuitively, an effective accelerator must complete matrix operations in few clock cycles while keeping its physical processing elements heavily utilized. Dividing execution latency by fractional layer utilization combines execution speed and hardware efficiency into a single metric, where a lower score indicates a more effective candidate.
Our local, lower-is-better architectural score is
\[ S = \frac{\text{total cycles}}{\bar{U}_{\text{frac}}} \]
where \(\bar{U}_{\text{frac}} = \bar{U}_{\%} / 100 \in (0, 1]\) represents the arithmetic mean fractional layer utilization.
Our evaluation runner computes this score directly from the retained SCALE-Sim simulation outputs. This score operates as a study-specific proxy; it is not a direct physical measurement, a classical area-delay product, or a universal accelerator metric. The runner calculates total cycles by summing the Total Cycles column, excluding the separate prefetch-inclusive column. To determine average layer utilization, we compute the arithmetic mean of the three per-layer SCALE-Sim Overall Util % values (rather than the Compute Util %), ensuring that each workload layer carries equal weight in our analysis. Consequently, our score does not explicitly penalize processing-element count, physical die area, or modeled memory traffic.
Following our frozen decision rule, a lower score indicates a superior architecture. We determine that a legal nonbaseline candidate clears our 1 percent support margin when \(S_{\text{candidate}} \leq 0.99 \times S_{32\times32}\); achieving exact equality at this threshold constitutes a passing result.
Our runner constructs both evaluation arms from the identical three-row CSV workload, tracking a layer name alongside the M, N, and K dimensions for each dense GEMM. We restrict our systolic-array rows and columns to the set of 8, 16, 32, 64, and 128, mandating that their product never exceed our 1,024 processing-element budget. After inserting the mandatory 32 by 32 baseline, our runner holds the workload, layout, SCALE-Sim configuration, dataflow, SRAM capacities, bandwidth parameters, performance deadlines, legality checks, and local score fixed for every candidate architecture.
We constrain our AI model’s role to be equally narrow. It is permitted to return three unique, legal nonbaseline shapes and a supporting mechanism hypothesis. However, we do not allow it to invoke SCALE-Sim, waive any hardware checks, formulate the final recommendation, or autonomously make the architecture decision.
Conversely, our deterministic heuristic filters out the baseline and orders every legal systolic shape first by decreasing processing-element count, then by the absolute log2 distance between the candidate’s columns-to-rows ratio and the aggregate workload N-to-M ratio, and finally by increasing row and column counts. We then programmatically select the first three shapes from this sorted order. Inserting the baseline guarantees that each proposal arm receives exactly four simulator events.
8.3.2 Execution Chronology
Even when an evaluation methodology is prospectively locked, real-world tool execution introduces interface errors, schema mismatches, and infrastructure retries that alter the chronological path to evidence. Our retained execution history captures these operational realities, documenting an initial schema validation failure followed by a corrected model invocation and twelve successful simulator runs.
The executed systolic-array study runs from initial plan freezing to final analytical judgments through one operational workflow (Figure 8.1). We start by freezing our research question, legal design space, scoring metric, and budget, then proceed to proposal schema validation. When our initial JSON schema fails validation, the workflow routes through an explicit repair and revalidation loop before candidate selection and legality checks proceed. Once cleared, the pipeline dispatches twelve cycle-accurate SCALE-Sim runs, feeding an interpreted study record that branches into four independent judgment outputs, comprising architecture outcome, study-local AI contribution, mechanism status, and stopping point. Incorporating explicit error-recovery loops within our workflow preserves prospective experimental integrity when interface failures occur.
Table 8.4 preserves the exact chronological sequence of these execution events, from preregistration freeze through schema rejection, schema amendment, proposal generation, simulator execution, and technical handoff. The sequence exposes both what our execution logs verify and what provenance data remains missing, including omitted random seeds, unrecorded currency costs, and absent repository commit hashes. A complete execution chronology reveals real-world tool recovery costs while keeping the provenance limitations transparent.
| Recorded event | Action and return | Retained support and limit |
|---|---|---|
| Freeze, 2026-07-14 18:19:48 UTC | Our preregistration records the workload, legal shapes, heuristic candidates, score, margin, budgets, rejection checks, mechanism contrast, and decision boundary. | We rely on an internal timestamp to record this order. No external timestamp or commit binding independently proves our precedence. |
| Rejected schema request, 18:21:00 UTC | The response-format service rejects our original JSON Schema (the machine-readable contract for structured JSON output) before inference because two constant-string properties lack explicit types. We produce no model output or usage here. | We retain the old and new schema hashes alongside a prose error description. Because we do not retain the rejected schema body and service response, we cannot independently check the semantic equivalence of our amendment. |
| Schema amendment, 18:21:41 UTC | Our record successfully adds explicit types and marks the change as non-substantive. | The package records our amendment before inference, though we fail to retain both schema bodies. |
| Proposal call, 18:21:51-18:22:24 UTC | One valid call returns the 16 by 64, 8 by 128, and 32 by 16 dimensions alongside a directional mechanism prediction. We execute no subsequent repair calls. | We retain the full response, 32.90 seconds of wall time, and our token usage. However, the provider revision, seed, temperature, top-p, host runtime, currency cost, and an exact matching prompt copy remain unavailable to us. |
| Simulator window, 18:31:23-18:31:42 UTC | Our record contains eight proposal-arm evaluation events and four shared mechanism-probe events. All twelve carry status: ok; every recorded array shape satisfies our 1,024-processing-element legality limit, and every result remains comfortably below the 90,000-cycle cutoff. |
We measure an aggregate runner window of 19.05 seconds. Unfortunately, we do not retain per-evaluation timestamps, commands, exit codes, logs, queue time, resource use, and human effort. We also exercise no invalid-candidate, rejection, timeout, retry, or simulator-recovery paths. |
| Technical stop and handoff | No evaluated nonbaseline shape clears our one-percent margin. The proposal arms tie, and our predicted winner reversal fails. | Our machine-readable recommendation retains the 32 by 32 baseline for the evaluated set and concludes the recorded search. Its accountable decision status rightfully remains awaiting_author_confirmation. |
Because the response-format service rejects our schema prior to model inference, our record treats this failure as a routine issue in request translation and interface validation. Throughout this process, our core architecture question, legal shape set, hardware score, and evaluation budget remain unchanged. We consequently treat the amendment as a necessary request repair within the boundaries of the study. However, because we lack the original schema body and the explicit service response, we cannot confirm that our amendment preserved the request’s intended architectural meaning.
Our model provenance records invocation parameters and interface metadata, but omits the final serving state and sampling settings. In addition, a prompt hash mismatch between preregistration and surviving logs prevents cryptographic verification of the exact input context. Finally, our simulator provenance records the software stack but fails to bind the runner to a specific repository commit, container image, or dependency lock.1 Extensive full-system simulation studies emphasize that unrecorded configuration differences can alter hardware performance outcomes (Gutierrez et al. 2014). While this context does not diagnose an error in our run, it prevents us from confirming which missing identity might have altered the final execution result.
1 Provenance tracking details: Invocations accessed OpenAI via Codex CLI 0.144.4 (alias gpt-5.4), consuming 13.2k input tokens and 1.4k output tokens without recorded seed or temperature. Preregistered prompt hash (ec12df...) differs from the surviving prompt file hash (36e016...). Simulator runs executed on macOS 26.4 (arm64) using Python 3.11.15 and SCALE-Sim 3.0.0.
Gutierrez, Anthony, Joseph Pusdesris, Ronald G. Dreslinski, et al. 2014. “Sources of Error in Full-System Simulation.” 2014 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 13–22. https://doi.org/10.1109/ISPASS.2014.6844457.
Within our retained array directory, we lack any independently preserved validation receipt artifact. Without this verification, an external architectural reviewer cannot treat our reference integrity, execution replay, or focused hardware tests as independently verified. Although our retained files and the current replay pathways remain inspectable, any subsequent replay inherits the same workload, simulation runner, scoring mechanism, and architectural assumptions of our originally recorded computation.
Our executed record also lacks any study-local independent analytical models, register-transfer level (RTL) validation, or cross-simulator checks to anchor our software modeling against cycle-exact FPGA RTL behavior using platforms such as FireSim, an FPGA-accelerated simulation framework (Karandikar et al. 2018). We also neglected to execute a basic analytical fold-count verification check. While a newly frozen mechanism study could introduce that check as future work, our mapping discussion below serves as our expert interpretation of the tagged simulator implementation rather than executed evidence.
Karandikar, Sagar, Howard Mao, Donggyu Kim, et al. 2018. “FireSim: FPGA-Accelerated Cycle-Exact Scale-Out System Simulation in the Public Cloud.” 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), 29–42. https://doi.org/10.1109/ISCA.2018.00014.
These provenance gaps prevent us from claiming independent replay or strong causal attribution. The retained outputs can still support the narrower comparison actually computed under the recorded workload, simulator configuration, local score, and checks, provided we carry the gaps into that claim.
8.3.3 Local Simulator Outputs
Evaluating candidate hardware shapes under equalized simulator-event budgets provides a budget-matched comparison between model-generated proposals and deterministic search heuristics. In our experiment, the model-driven arm and the fixed aspect-ratio heuristic arm each evaluate the 32 by 32 baseline alongside three candidate array geometries across our synthetic matrix multiplication workload.
The empirical simulator returns from our matched evaluation arms tell the whole story compactly (Table 8.5). We evaluate five unique array geometries across eight total simulation runs, comparing our model-generated proposals against our deterministic aspect-ratio heuristic. The empirical returns confirm that the 32 by 32 baseline and the 16 by 64 candidate achieve identical execution latencies of 13,917 cycles and matching layer utilization of 41.196 percent, yielding an identical local score of 33,782.40. Both arms tie with this best-evaluated score, while extreme aspect ratios (such as 8 by 128 and 32 by 16) suffer performance penalties due to lower array utilization. Under equalized simulator-event budgets, generative proposals provide no local performance advantage over a simple analytical heuristic.
| Arm | Shape and source | PE count (author-derived) | Cycles | Average layer utilization | Local score | Result within arm |
|---|---|---|---|---|---|---|
| Model | 32 by 32 baseline | 1,024 | 13,917 | 41.196% | 33,782.40 | Tied best |
| Model | 16 by 64 proposal | 1,024 | 13,917 | 41.196% | 33,782.40 | Tied best |
| Model | 8 by 128 proposal | 1,024 | 19,405 | 29.704% | 65,327.45 | Ranked worse |
| Model | 32 by 16 proposal | 512 | 25,277 | 45.438% | 55,629.61 | Ranked worse |
| Heuristic | 32 by 32 baseline | 1,024 | 13,917 | 41.196% | 33,782.40 | Tied best |
| Heuristic | 16 by 64 proposal | 1,024 | 13,917 | 41.196% | 33,782.40 | Tied best |
| Heuristic | 8 by 128 proposal | 1,024 | 19,405 | 29.704% | 65,327.45 | Ranked worse |
| Heuristic | 64 by 16 proposal | 1,024 | 17,757 | 32.294% | 54,985.25 | Ranked worse |
Our 32 by 16 proposal utilizes 512 processing elements, whereas the 32 by 32, 16 by 64, 8 by 128, and 64 by 16 configurations each deploy exactly 1,024. Consequently, array shape and overall processing-element count vary simultaneously during our 32 by 16 comparison. We expect that a hardware design with 512 processing elements will exhibit a different resource trade-off profile compared to our standard shapes leveraging 1,024 elements. However, because our study neglects to measure the explicit area or energy consequences of this architectural discrepancy, our retained result inherently remains a score-only local hardware outcome.
Our best evaluated nonbaseline configuration, the 16 by 64 array, exactly matches the 32 by 32 baseline in total execution cycles, average layer utilization, and our final local score. It fails to overcome our predefined architectural change margin. Furthermore, the 8 by 128, 32 by 16, and 64 by 16 designs all rank worse under our frozen scoring metric. We interpret this as a no-change outcome across five uniquely evaluated hardware shapes. We caution that this result alone does not provide sufficient evidence to prove that the 32 by 32 layout is globally optimal among all legally permitted dimensions.
While our proposal ranking yields a functional tie, our targeted mechanism test fails its declared performance reversal. Because these two results investigate different architectural questions, we cannot responsibly use the tied evaluation score as supporting evidence for our model’s directional explanation.
Figure 8.2 holds the experiment’s two separable findings: the local candidate scores and the directional mechanism probe that failed.
- Equal simulator-event allocation: Across every evaluated array shape, scored for both the model proposals and the fixed heuristic (lower is better), both the 32 by 32 baseline and the 16 by 64 shape achieve the lowest score of 33.8 thousand in both arms, establishing a functional tie.
- Mechanism probe: Tracking local scores across the original workload and the transposed workload (where matrix dimensions M and N are swapped) for the 16 by 64 and 64 by 16 arrays, the 16 by 64 geometry maintains its lead over 64 by 16 across both orientations rather than exhibiting the predicted reversal under transposition.
A local score tie between evaluation arms does not validate the model’s underlying physical explanation: ranking ties and mechanistic predictions address different architectural questions, requiring independent empirical probes to verify.
This architectural tie demonstrates no study-local AI advantage over our deterministic, fixed heuristic when we enforce equal simulator-event budgets. We must not mistake this tie for evidence of equal capability, identical total design cost, or performance parity with exhaustive enumeration, an intensive expert study, specialized ILP solvers, or alternative hardware search methodologies. Executing just one proposal call supplies no statistically valid estimate of variation or broader reliability for this specific generative model family.
While our retained simulation results establish that the 32 by 32 and 16 by 64 arrays yield strictly equal cycle counts and average layer utilization under our frozen scoring criteria, they do not illuminate the underlying architectural root cause of that equality. Consequently, we introduce the subsequent transposition probe to test a separate mechanistic hypothesis by directly contrasting the 16 by 64 array against its 64 by 16 transpose.
8.3.4 Failed Directional Explanation
Our generative model predicted that a wide 16 by 64 array would outperform a tall 64 by 16 layout on our baseline workload, asserting that swapping matrix dimensions M and N would reverse their relative performance. However, our retained simulator records contradict this mechanistic explanation. In our retained study results, the original local scores of 33,782.40 versus 54,985.25 compare against transposed scores of 23,530.65 versus 25,458.11; the predicted architectural reversal does not occur. Observing a narrower performance gap under transposition fails to validate the proposed physical mechanism.
Our surviving prompt copy also omits the dataflow mapping semantics necessary to interpret this hardware contrast. While our retained execution logs identify SCALE-Sim 3.0.0, they omit any dependency hash or source commit tracking. Consequently, our assumption of a K-height, N-width, and M-temporal mapping directly follows the publicly tagged v3.0.0 implementation (SCALE-Sim Project 2025).
The simulator’s physical hardware mapping semantics decide what our array comparison actually varied (Figure 8.3). SCALE-Sim’s weight-stationary GEMM dataflow maps matrix dimensions onto physical array structures, so a resident weight tile lands differently on the wide 16 by 64 and tall 64 by 16 processing-element layouts.
- Workload mapping: The GEMM weight matrix dimensions map spatially onto the hardware grid, with dimension K assigned to array height and dimension N assigned to array width, while matrix dimension M streams temporally through the array over execution cycles.
- Transposition mechanics: Swapping workload dimensions M and N while dimension K remains fixed mirrors the array geometry from 16 by 64 to 64 by 16, altering both the vertical K-fold count (\(\lceil K/\text{height}\rceil\)) and the horizontal N-fold count (\(\lceil N/\text{width}\rceil\)) simultaneously.
This spatial mapping explains why our directional probe failed to isolate array orientation. Exchanging M and N mutates multiple spatial tiling factors at once rather than isolating a single physical dimension. Diagnostic probes must align precisely with simulator dataflow semantics to avoid confounding multiple hardware parameters.
SCALE-Sim Project. 2025. SCALE-Sim Source Code, Version 3.0.0. https://github.com/scalesim-project/SCALE-Sim/tree/v3.0.0.
While our model’s directional prediction fails its publicly declared test, this invalidated contrast does not establish a viable replacement mechanism. It does not prove that systolic array orientation is structurally irrelevant to overall performance. Instead, we recognize that launching a new mechanism study would require crafting a specialized prompt with pinned mapping semantics, alongside a contrast methodology that isolates and varies exactly one relevant fold count at a time.
Our retained report exposes a fundamental limit regarding our simulator’s internal memory semantics. Specifically, the transposed 16 by 64 input configuration declares its parameters as Bandwidth = 64,64,64. Yet the corresponding retained bandwidth report outputs an Avg OFMAP DRAM BW of 79.073 in one of its rows. Our historical record fails to explain the scaling relationship between these disjoint parameters. Because their underlying bandwidth semantics remain unresolved, we cannot treat the configured bandwidth value as valid evidence of any demonstrated physical or shared-system bottleneck.
Our retained run uses SCALE-Sim’s user-specified interface-bandwidth path with UseRamulatorTrace = False. Its reported word traffic is therefore not a contention-aware DRAM timing result. Dedicated memory-system models such as Ramulator and Ramulator 2.0 (Kim et al. 2015; Luo et al. 2024), or a suitably configured gem5 memory hierarchy, can represent channels, ranks, bank groups, banks, row buffers, controller scheduling, and associated timing. A follow-up study would need such a path before attributing latency to bank conflicts or row-buffer locality.
Kim, Yoongu, Weikun Yang, and Onur Mutlu. 2015. “Ramulator: A Fast and Extensible DRAM Simulator.” IEEE Computer Architecture Letters 15 (1): 45–49. https://doi.org/10.1109/LCA.2015.2414456.
Luo, Haocong, Yahya Can Tuğrul, F. Nisa Bostancı, Ataberk Olgun, A. Giray Yağlıkçı, and Onur Mutlu. 2024. “Ramulator 2.0: A Modern, Modular, and Extensible DRAM Simulator.” IEEE Computer Architecture Letters 23 (1): 112–16. https://doi.org/10.1109/LCA.2023.3333759.
Under an open-page memory-controller policy, requests to an already open row can avoid the precharge and activation work required for a conflicting row in the same bank. Whether either array geometry produces those access patterns depends on the address mapping, request order, controller policy, and memory organization, none of which this retained run establishes.
SCALE-Sim reports 409,726 DRAM-word writes for our 16 by 64 array and 204,863 for the 32 by 32 baseline. That exact difference is a modeled traffic count, not a measured energy or latency penalty. A contention-aware follow-up could test whether the extra traffic changes command-queue delay, arbitration, bank conflicts, row-buffer behavior, or channel utilization. The current record supports none of those downstream magnitudes.
Stepping back to the broader subsystem context, our retained SCALE-Sim configuration and corresponding outputs establish none of the 8 prompt stack layers required by our overarching Lighthouse system decision. Synthetic GEMM workload configurations bypass high-level Abstract Syntax Tree (AST) parsing, Control-Data Flow Graph (CDFG) scheduling, and High-Level / Dialect Intermediate Representation (IR) lowering passes inside Triton, a domain-specific compiler framework for AI workloads, when compiling kernels for vector RV64GCV host cores.
This execution run also preserves no usable data concerning RV64GCV vector ISA compliance, Ramulator DRAM timing delays, TSMC N7 physical layout constraints, gate-level netlists synthesized via Yosys, physical placement from OpenROAD, static timing signoff from OpenSTA, dynamic energy, peak power, die area, thermal dissipation modeled in Ansys RedHawk-SC, power domain isolation specified by IEEE 1801 UPF, or formal safety properties verified through SVA and BMC. This omission is notable given that the broader SCALE-Sim v3 framework supports detailed memory and energy analyses (Raj et al. 2025).
Raj, Ritik, Sarbartha Banerjee, Nikhil Chandra, et al. 2025. “SCALE-Sim V3: A Modular Cycle-Accurate Systolic Accelerator Simulator for End-to-End System Analysis.” 2025 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), May, 186–200. https://doi.org/10.1109/ISPASS64960.2025.00026.
Kwon, Hyoukjun, Krishnakumar Nair, Jamin Seo, et al. 2023. “XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse.” Proceedings of Machine Learning and Systems 5: 1–20. https://proceedings.mlsys.org/paper_files/paper/2023/hash/07b7c2f8a6978e376f9656d69619bfbb-Abstract-mlsys2023.html.
Standard Performance Evaluation Corporation. 2017. SPEC CPU 2017 Benchmark. https://www.spec.org/cpu2017/.
Reddi, Vijay Janapa, Christine Cheng, David Kanter, et al. 2020. “MLPerf Inference Benchmark.” 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 446–59. https://doi.org/10.1109/ISCA45697.2020.00045.
Our configured bandwidth parameters and the simulator’s reported operand traffic also do not model a verified, contention-aware shared-system interface incorporating die-to-die UCIe or CXL chiplet interconnects. This synthetic GEMM workload and our localized array result cannot resolve the system-level cache sizing decision for XRBench, an extended-reality benchmark suite (Kwon et al. 2023), SPEC CPU, a standardized computer architecture benchmark suite (Standard Performance Evaluation Corporation 2017), or MLPerf, an industry-standard AI inference benchmark family (Reddi et al. 2020), satisfy our 3 W TDP thermal envelope, or close out the Lighthouse accelerator design. Yet synthetic workloads represent only the first proxy boundary of our array evaluation. Even within the simulator’s output data, our choice of local scoring metric introduces a second proxy mismatch.
8.4 Metric Mismatches and Proxy Discrepancies
Beyond the limitations of synthetic workloads, automated architectural design loops risk relying on narrow proxy metrics that mask underlying hardware trade-offs. While our local array scoring metric tracked total execution cycles and average processing-element utilization, it omitted modeled DRAM memory traffic, concealing a major discrepancy in off-chip data movement.
We evaluate the design-space coverage of our array experiment in Table 8.6. Out of fifteen legal processing-element geometries satisfying our 1,024-element budget, our budget-constrained proposal arms evaluated only five unique shapes (8 by 128, 16 by 64, 32 by 16, 32 by 32, and 64 by 16), leaving ten legal geometries unexecuted. Repeating candidate evaluations across proposal arms consumes simulation events without expanding architectural coverage: budget-limited evaluation loops must balance candidate exploitation against design-space exploration to avoid leaving major architectural regions uncharacterized.
| Status | Exact legal shapes | Count |
|---|---|---|
| Evaluated as unique original-workload geometries | 8 by 128; 16 by 64; 32 by 16; 32 by 32; 64 by 16 | 5 |
| Not executed | 8 by 8; 8 by 16; 8 by 32; 8 by 64; 16 by 8; 16 by 16; 16 by 32; 32 by 8; 64 by 8; 128 by 8 | 10 |
If we intend to freeze a follow-up study, we could achieve a comprehensive full-space reference through exhaustive enumeration (evaluating every single legal shape under a fixed workload, simulator configuration, scoring metric, and set of checks). However, such a brute-force approach does not match the computation budget of our current four-event proposal arms. In our next architectural iteration, we must budget for this greater simulation cost or use it as a foundational coverage baseline. Since we did not perform that enumeration in our current study, we cannot infer any performance outcomes for the ten unexecuted shapes. We also cannot retroactively simulate these missing shapes just to claim our recorded model tied or outperformed the complete architectural design space.
Our frozen score also omits a reported performance axis that could influence our future system-level decisions. For instance, the 16 by 64 shape ties our baseline performance at exactly 13,917 cycles and 41.196 percent average layer utilization. Yet, when we inspect our retained study results, we discover 409,726 modeled DRAM writes for the 16 by 64 geometry compared to just 204,863 for our 32 by 32 baseline. In another notable contrast, our 64 by 16 shape reports 102,400 modeled writes and fewer total modeled accesses than the baseline, despite requiring more cycles to execute. These numbers represent abstract simulator outputs, rather than true physical DRAM transactions or concrete energy measurements.
Relying on incomplete proxy metrics is dangerous, and our evaluated array geometries make that plain across two distinct simulator metrics, total execution cycles and modeled DRAM-word writes, where lower is better on both (Figure 8.4).
- Baseline dominance: Our 32 by 32 baseline sits at 13,917 cycles and 204,863 DRAM-word writes.
- Shared candidate discrepancy: The shared 16 by 64 candidate matches the baseline at 13,917 cycles but requires 409,726 DRAM-word writes, exactly double the off-chip memory traffic.
- Secondary candidate trade-offs: The remaining shapes trade the two axes off differently, such as 64 by 16, which reduces DRAM writes to 102,400 but increases latency to 17,757 cycles.
Although 16 by 64 tied the baseline under our single cycle-based score, it suffers a severe memory traffic penalty: single-dimensional proxy scores can mask secondary bottlenecks, demanding multi-metric evaluation to capture complete hardware behavior. However, we cannot let this post-run observation retroactively alter our frozen score, nor does it establish actual physical memory energy consumption or authorize us to change our prior design decision.
Should we launch a new enumeration study, we will need to freeze our decision quantities before execution begins. At that point, we could choose to retain our original local score for strict comparability, introduce a separately justified memory-aware metric, or replace the scoring function entirely. Nevertheless, we cannot use any of these future choices to rescore our currently retained and executed study. Our unresolved memory axis illustrates why neither exhaustive architectural coverage nor our current local ranking alone provides enough evidence to establish energy consumption, power draw, or overall Lighthouse suitability. Our tied ranking, the failure of our directional explanation, and our omission of modeled memory traffic from the frozen score mandate a shift in how we approach our next study.
8.5 Technical Boundaries of Architectural Loop Execution
Comparing our unexecuted L2 cache plan with our executed systolic-array experiment illustrates the distinct technical boundaries that govern architectural evaluation loops. Every study reaches a stopping point determined by tool availability, artifact retention, and validation checks. Disentangling these stopping boundaries keeps prospective specifications, executed simulator evidence, and physical or silicon evidence from collapsing into one claim.
When we evaluate the Lighthouse cache path alongside our separate executed array study, we see that they hit different technical stopping points, driving distinct architectural next steps. Looking closely at the executed array case, every returned signal forces us to re-evaluate a distinct architectural claim and redirect our subsequent actions; while we can handle some of these within our current record, others demand that we freeze a new architectural study.
- Tie: When our returned scores challenge the claim that model-guided proposals improve local architecture outcomes, we must adjust our expectations. Because these proposals demonstrate no study-local advantage over our fixed heuristic under equal simulator-event budgets, we retain 32 by 32 for our evaluated set. If we want to claim broader selection benefits, we must run a newly frozen, separately budgeted full-space reference.
- Failed mechanism: Our transposition results alter our directional explanation rather than our measured ranking. They squarely reject our predicted reversal, yet they fail to offer a replacement mechanism. To isolate what is actually happening in the hardware, we need a newly frozen study built on pinned mapping semantics.
- Omitted metric: Once we look at our modeled traffic counts, we expose a critical quantity absent from the frozen score. These counts neither change our current ranking nor allow us to perform post-run rescoring. Instead, they remind us that if we rely on a memory-aware decision quantity, we must build it into a newly frozen study.
- Provenance gap: Because we lack prompt identity and full simulator provenance, we cannot make a strong claim for architectural reproduction. While we can still use our current record within these constraints, any new run must capture the exact prompt, model revision (when available), runner revision, dependencies, command sequences, and complete outputs.
- Pending authority: Although our machine-readable technical recommendation retains 32 by 32 for the evaluated set, we have not recorded any authorized acceptance or rejection. Therefore, our immediate next action within the current record must be a peer review rather than a firm architectural commitment.
To synthesize the technical stopping boundaries of our evaluation loops, we compare our prospective and executed studies in Table 8.7. Our cache path remains stopped before execution with an unresolved outcome and untested mechanisms, whereas our array study completes twelve simulator evaluations across five unique geometries to reach a tied ranking, a failed directional mechanism, and a technical stop that retains the 32 by 32 baseline. Different evaluation loops reach distinct stopping boundaries, requiring architects to tailor follow-up actions to the specific evidence class and technical limits of each record.
| Record | Architecture outcome | AI contribution | Mechanism | Stopping and action |
|---|---|---|---|---|
| Lighthouse cache path | Unresolved. We neither evaluated, rejected, nor advanced any cache candidate. | Not tested. We selected direct evaluation without any added learned methods. | Untested. We retained no result connecting capacity misses to 99th-percentile frame time. | Stop before execution. We lack the cache implementation, measurement, and dispatch prerequisites. If we believe this decision remains worth pursuing, we must launch a new fully retained run that supplies all missing bindings, procedures, and artifacts. Otherwise, we can legitimately defer, cancel, or reformulate. |
| Separate executed array study | No evaluated nonbaseline design clears our local margin across five unique shapes. We do not claim this as an optimality result. | We demonstrate no study-local advantage over the fixed heuristic under equal simulator-event budgets. We also failed to measure equal total cost and method parity. | Our directional reversal prediction fails, and we establish no replacement explanation. | Our machine-readable recommendation retains 32 by 32 for the evaluated set, stopping our local technical search. We recorded no authorized acceptance or rejection. |
Our tie, our failed mechanism, our omitted metric, and our provenance gap all point toward necessary follow-up studies. We did not run any of these as part of our retained execution, and each will require its own separately frozen architectural comparison. Even if we completed those studies, we still would not fully establish XR behavior or hit our 3 W system target, because our retained configurations and outputs lack the necessary system-level checks.
When we step back, both records leave our core system-level questions open. Neither trace explicitly identifies an XR workload, nor does either validate our tail deadlines across actual deployed conditions. We never execute the compiler, the runtime, the RISC-V interface, our host cores, the shared memory path, our sensors, or the display engine. Neither record provides a complete verification plan or fully validates the function, security, or reliability of our overall subsystem.
Our physical design questions also remain unanswered. We retain no concrete evidence that closes our timing, routing, area, or power delivery constraints. We have yet to prove our 3 W subsystem budget, characterize our thermal behavior, or verify our final silicon performance. While our array reports do include modeled local traffic, we lack a shared-memory contention analysis or a physical energy model to connect that traffic back to the Lighthouse platform. Our cache plan might name some of these physical checks, but naming a check in architecture is never a substitute for running it.
Design principle: Claim only what the checked record supports
The principle: An architecture evaluation loop must bound its claims strictly to what its executed, checked tool records demonstrate.
The application: Unexecuted plans do not provide measured results. Partial simulation logs and budget-limited ties can support only the local claims their retained checks establish; they do not substitute for physical or silicon evidence.
8.6 Common Pitfalls
The contrast between our unexecuted cache specifications and executed array proxies highlights how easily automated evaluation loops can corrupt empirical validity. When evaluation pipelines generate vast quantities of simulation logs, unverified declarations, and synthetic candidate shapes, subtle methodological errors can silently invalidate baseline comparisons. Examining common pitfalls in parameter binding, evidence retention, and study boundaries establishes the safeguards required to preserve empirical integrity in automated design workflows.
Conflating prospective plans with executed evidence. We must never treat prospective trial specifications, generator prompts, or configuration scripts as if they were execution records. Elaborate plans and candidate configurations establish setup intent. A candidate remains unmeasured until the relevant tools execute and retain checked outputs; simulator, formal, physical-design, and silicon results then support different classes of claim.
Violating prospective binding rules through retrospective tuning. We must avoid adjusting evaluation parameters, simulation seeds, workload trace windows, or EDA tool constraints after observing interim candidate scores. Prospective binding rules require locking experimental parameters prior to execution. Modifying evaluation conditions mid-loop to favor specific candidates introduces outcome bias and destroys the comparative integrity of our baseline study.2
2 Clinical trial preregistration parallel: Clinical-trial registration records a protocol and its primary and secondary outcome variables before a trial begins, a practice that prevents outcome switching after results are seen (Nosek et al. 2018). In computer architecture, locking trace windows, workload seeds, and EDA synthesis constraints prior to execution plays an identical role, preserving baseline comparative integrity.
Nosek, Brian A., Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. 2018. “The Preregistration Revolution.” Proceedings of the National Academy of Sciences 115 (11): 2600–2606. https://doi.org/10.1073/pnas.1708274114.
Blurring array versus cache study boundaries in subsystem loops. We cannot conflate the execution boundaries of an isolated spatial GEMM array study (evaluating local PE utilization in SCALE-Sim)—an instance of point AI assistance bolted onto a single accelerator task—with full subsystem platform requirements (such as our prospective 3 W L2 cache study under its declared XR workload snapshot). Claiming subsystem power signoff or memory-wall closure from narrow accelerator loop iterations ignores system interconnect contention, host-side synchronization, and cache-hierarchy latency bottlenecks.
Misinterpreting a local tie as architectural equivalence. Equal scores establish no qualifying difference among the evaluated candidates under the frozen workload, configuration, metric, and checks. They do not establish physical parity, an optimal design, insufficient compute, or what unexecuted candidates would have done.
Coupling microarchitectural mechanisms with unverified system outcomes. We must avoid attributing a candidate’s high-level score improvement to a hypothesized microarchitectural mechanism without executing isolating diagnostic checks. Candidate hardware often outperforms a baseline due to secondary effects like altered compiler code alignment or cache line prefetching shifts. We must decouple the true cause from the aggregate score to protect our architectural conclusions.
8.7 Open Questions
Comparing an unexecuted prospective cache plan against a constrained systolic-array experiment reveals the boundaries of current architecture automation frameworks. Stalled workflows, schema rejections, and missing provenance logs represent open methodological challenges where formal evaluation principles intersect tool failures and finite compute budgets. Transforming ad-hoc automation scripts into reliable architectural workflows requires principled stopping rules, dynamic capacity allocation, and unbiased adaptation protocols.
Action selection and capacity allocation. Orchestrating budget-limited evaluation loops requires principled stopping rules and dynamic resource allocation strategies.
When an automated workflow encounters a missing prerequisite or an interface failure, how should it decide its next move? We must develop measurable ways to evaluate different action-selection policies, judging them by their total cost to reach a fully supported architectural conclusion while avoiding invalid tool dispatches and unproductive reformulations.
How should a study ration finite simulation, implementation, and review budgets across competing candidates and checks? We need to compare fixed, cheapest-first, and decision-aware allocation policies under identical budgets, defining stopping rules that halt evaluation the moment a decision margin is resolved or a prerequisite fails.
Experimental binding and cross-study synthesis. Preserving experimental validity demands strict prospective binding contracts and formal cross-study evidence synthesis.
At what point does adapting an experimental plan to fresh data blur into outcome bias? We must establish methodologies to evaluate these trade-offs, distinguishing between intelligently steering subsequent simulation runs toward promising design spaces and compromising the validity of our prospective binding contracts under matched tool budgets.
How do we synthesize stopping criteria and evidence boundaries across heterogeneous, disjoint sub-studies? We must develop formal methods to prevent local simulator outcomes from being misconstrued as comprehensive full-subsystem design decisions.
8.8 Summary
Synthesizing our findings across both prospective plans and executed simulator traces underscores a foundational principle of architectural evaluation. A claim can reach only as far as its retained evidence and checks.
In this chapter, we brought the architecture evaluation cycle together through two records that deliberately do not compose. Prospective plans, executed tool runs, qualified feedback, and stopping rules remain separate obligations. Prospective binding records the intended baseline, trace windows, and interpretation rules before execution, making later changes auditable and reducing the opportunity for undisclosed retrospective tuning.
The technical boundary of loop execution is the checked record itself. Prospective specifications do not substitute for measured results, while a budget-limited simulator tie remains a local result rather than evidence of physical hardware parity. Four takeaways govern this phase of AI-native co-design.
Key Takeaways: Bounding Claims by Evidence Class
- Claims bounded by evidence class. Prospective specifications, executed simulator records, physical-design results, and silicon measurements support different claims and must remain distinguishable.
- Prospective parameter binding. Design parameters, workload trace windows, evaluation seeds, and interpretation rules must be recorded before loop execution so that later changes are visible and baseline comparisons remain interpretable.
- Declared budgets and stopping rules. Finite simulation and synthesis budgets, together with explicit stopping rules, must be declared before execution begins, halting a study once a decision margin is resolved, a budget is exhausted, or a prerequisite fails.
- Enforcing array vs cache study boundaries. Maintain strict evidence boundaries between isolated accelerator module loops (such as SCALE-Sim GEMM studies) and full-subsystem platform evaluations (such as 3 W L2 cache TDP signoff), refusing to conflate local PE utilization ties with system-level closure.
In the next chapter (Chapter 9), we ask which parts of our executed array study and its retained record can actually be reused when our workload, objective, simulation tool, implementation target, or overall system context changes.