Appendix A — The Architecture 2.0 Reviewer’s Checklist

Author

Affiliation

Harvard John A. Paulson School of Engineering and Applied Sciences

Published

August 11, 2026

We provide this appendix as a portable external review instrument for evaluating papers, artifact packages, design reports, and study records from AI-assisted and AI-native architecture work, without requiring authors to adopt the design-loop card from Chapter 3 or any other schema. Our questions trace the evidence from a bounded claim and its executed environment through comparisons, independent checks, replayability, realized costs, claim limits, and technical responsibility.

For each item, we recommend recording one of four statuses.

Supported. The paper and its available artifacts answer the question with evidence appropriate to the claim.
Conditional. The answer holds only within a named limitation or condition that must accompany the review conclusion.
Unresolved. Missing, inaccessible, inconsistent, or insufficient evidence prevents the reviewer from answering the question.
Not applicable. The item does not apply to the stated contribution, and the review records why.

We intend these as item-level review notes, distinct from the program-claim statuses defined in Chapter 10. They do not produce a pass/fail score, nor do they unilaterally decide whether a paper should be accepted.

A.1 Claim Scope

We cannot properly evaluate a claim if it fails to establish clear boundaries (Chapter 3).

Bounded task. Do the authors state the design task alongside its workload, design space, objectives, and constraints, ensuring readers can tell which results the claim covers and which it does not?
Validity region. Do the authors state where the claim stops, detailing the workloads, process assumptions, and operating conditions under which the result was never tested?
Benchmark status. If the authors call the contribution a benchmark, do they define the target task population and its development, holdout, and confirmation splits? Do they specify evaluator-owned checks and any hidden tests, matched baselines and a total-cost rule, and benchmark versioning, maintenance, and governance (Chapter 10; Chapter 12)?
Claim scope. For a paper making a general claim about AI capabilities in hardware design, do the authors state the workload, tool path, and physical consequences the claim covers, and do they name the comparison that would falsify it?

A.2 Environment, State, and Provenance

We require access to the claim-relevant environment and its represented state for external review, even when certain artifacts remain protected or inaccessible. Model weights alone do not adequately preserve the data, constraints, and tool states that bound a result.

Evaluation artifact. Do the authors identify and preserve the claim-relevant environment state, including wrappers, state representations, objectives or rewards, and exact tool versions? Do they clearly state what is publicly released, available only for protected review, or entirely inaccessible?
Constraints. Do the authors preserve baseline heuristics, design limits, and constraints alongside the environment, clearly identifying private or unavailable states rather than hiding them as tacit knowledge in scripts?
Representation. Do the authors source, version, and scope the design-state representation, feature set, or trace corpus, and does its coverage match the design space over which the claim ranges (Chapter 4)?
Source permission. For every project-supplied training or retrieval source, do the authors record its identity, permission for the intended use, license and attribution information, and applied transformations?
Generated-artifact provenance. Do the authors retain source links, retrieved context, model identity, and the construction and transformation history sufficient to assess each generated artifact’s provenance? Do they explicitly mark unavailable pretrained-corpus details as an evidence limit?
Confidentiality and egress. Do the authors distinguish the permission to use a source from the permission to send proprietary project material beyond authorized boundaries? Do they record the execution location and declared egress routes? Do they record the permission and confidentiality scope of workspaces, retrieval indexes, caches or checkpoints, model context, logs, and credentials (Chapter 6)?
Contamination check. Did the authors screen the evaluation for exact or near-duplicate overlap where possible, and do they disclose known or possible overlap with public Hardware Description Language (HDL) benchmark tasks (Wang et al. 2025)?
Transfer claim. If the method is learned rather than searched per instance, do the authors maintain held-out barriers, provide comparison arms with matched pretraining and retrieval access (unless that access is under test), and state the residual exposure limit? (When only one arm uses pretraining, the result supports the complete pretrained package rather than the intrinsic superiority of the method.)
Proprietary data navigation. Do the authors explain how they transition from open-source (GPL/viral) contexts into proprietary Performance, Power, and Area (PPA) / Process Design Kit (PDK) environments, and do they maintain strict state isolation to prevent intellectual property cross-contamination?

Wang, Zeng, Minghao Shao, Jitendra Bhandari, et al. 2025. VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contamination. https://doi.org/10.48550/arXiv.2503.13572.

A.3 Engineering Roles and Method Families

As reviewers, we cannot judge the validity of feedback without knowing which component produced it (Chapter 5).

Roles and methods. Do the authors name the engineering work each component performs, such as proposing, screening, searching, critiquing, repairing, checking, explaining, or coordinating? Do they separately identify the method family and concrete technique rather than describing a monolithic “AI” or “agent”?
Decomposition. If the system composes several roles, do the authors state each role’s input, output, limit, and next check, allowing reviewers to see which component’s error would corrupt which result?

A.4 Proxy Metrics and Claim-Specific Checks

Reward hacking through imperfect proxies is an established concern in AI safety (Amodei et al. 2016), and it transfers to hardware evaluation. When we rely on proxy metrics, we risk allowing an automated search to exploit simulator inaccuracies rather than improve the design.

Amodei, Dario, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. “Concrete Problems in AI Safety.” arXiv Preprint arXiv:1606.06565.

Proxy boundary. Do the authors distinguish the fast surrogate metric used for search from the stronger claim-specific check used to evaluate the final result?
Independence check. For large claimed gains, do the authors state which disputed property the stronger check observes? (A later RTL, synthesis, or timing stage does not automatically close a simulator loophole.)
Proxy calibration. Do the authors report the proxy’s rank fidelity against a stronger reference specifically in the region where the search actually concentrated, rather than averaged over the entire space (Chapter 10)?
Latent space grounding. For methods relying on learned latent embeddings or surrogate manifolds (e.g. GNN floorplan embeddings, Transformer code tokens, BO surrogate models), do the authors define the physical decoding and grounding mechanism that projects latent points back to executable electronic design automation (EDA) and RTL signoff checks (Appendix B)?
Correctness check. Did the authors complete a claim-appropriate functional check before using power, performance, or area metrics to support a valid-design claim? (Quality-of-results measurements from invalid candidates should only serve as diagnostic or cost evidence, and equivalence applies only when the intended reference relation is valid.)

A.5 Matched Baselines and Declared Budgets

We expect any reported gains to emerge from the method or complete workflow itself, rather than from an uneven comparison.

Baseline control. Do the authors compare their method or complete workflow against the strongest practical alternative for the claimed contribution within the same constrained environment? (Random search is a relevant comparator only when the question is whether guided search beats sampling.)
Other axes. Do the authors report every claimed or materially affected objective and constraint at the same operating point, using matched constraint files, tool versions, and conditions for every arm? If the claim focuses on a single axis, does the comparison hold all other material objectives and constraints equal?
Budget parity. Did the authors allocate comparable declared run budgets, review budgets, and tuning effort to every arm? Do they report the resources actually consumed by each arm? (Testing a heavily tuned method against default-configuration baselines does not constitute a fair comparison.)
Evaluation capacity. Do the authors test for a candidate-production/evaluation-capacity mismatch by stating the candidate count or arrival rate alongside the measured or budgeted service capacity of the tool, check, and reviewer stages (including failures and retries)? If the path cannot process arrivals within budget, do they report the backlog and associated costs rather than claiming a throughput improvement (Section 2.6)?
Exploratory split. Do the authors strictly separate development and tuning from a locked, predeclared confirmation evaluation on held-out tasks and conditions appropriate to the claim (Chapter 10)?

A.6 Variation and Statistical Support

Learning-based results, particularly in deep reinforcement learning, can be sensitive to seeds, hyperparameters, and implementation details (Henderson et al. 2018).

Henderson, Peter, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. “Deep Reinforcement Learning That Matters.” Proceedings of the AAAI Conference on Artificial Intelligence 32 (1): 3207–14. https://doi.org/10.1609/aaai.v32i1.11694.

Sample size. Do the authors state the number of runs, seeds, and model invocations, providing confidence intervals on headline numbers rather than merely highlighting single best runs?
Distribution. Do the authors report distribution-level or category-level outcomes, per-task results, failures, and uncertainties using summaries appropriate to the observed shape, including heavy tails or bimodality?
Noise floor. Did the authors measure apparatus variation for their study through fixed-setup repeats and sensitivity to seeds in tools like Quartus Fitter, an FPGA place-and-route tool, where applicable (Altera 2026)? Can we reliably distinguish the claimed gain from that measured variation at a relevant practical margin?
Aggregation. For positive normalized ratios with a common direction, do the authors use the geometric mean rather than a reference-dependent arithmetic mean (Fleming and Wallace 1986)? Do they explicitly declare how they handle zeros, signed slack, failures, and heterogeneous quantities (Chapter 10)?

Altera. 2026. SEED: Quartus Prime Pro Edition Settings File Reference Manual. https://docs.altera.com/r/docs/683296/25.3/quartus-prime-pro-edition-settings-file-reference-manual/seed.

Fleming, Philip J., and John J. Wallace. 1986. “How Not to Lie with Statistics: The Correct Way to Summarize Benchmark Results.” Communications of the ACM 29 (3): 218–21. https://doi.org/10.1145/5666.5673.

A.7 Independent Checks and Failure Records

When we evaluate verification and trust, our central review question applies across submissions (Chapter 7).

Independent check. Does a claim-specific observation path differ enough to change the disputed failure source, and do the authors record the assumptions that remain shared with the producer? Do they declare whether each check is advisory or blocking? (A blocking failure prevents the result from advancing, while an advisory result informs later review. Consensus among correlated models does not constitute independence.)
Failure record. Do the authors report rejected candidates, failed runs, and out-of-distribution failures so that the true difficulty of the search remains fully visible?
Challenge program. Do the authors state which attacks or stress tests they attempted against their own assumptions, checks, and unrewarded constraints, and do they report the tests that succeeded rather than only those that failed (Chapter 10)?
Interruption and recovery. If the workflow ran unattended, do the authors report its behavior under tool timeouts, crashes, and incomplete reports? Do they state whether they excluded results produced after a detected failure from the reported evaluation?

A.8 Execution Provenance and Replay

When we attempt to replay an evaluation, we require the exact recorded candidate, tool path, and behavior-affecting state. We use replay to test that specific retained execution path, which is distinct from an independent reproduction derived from the stated specification.

Replay path. Can a reader identify the exact executed candidate and tool path, clearly seeing what was actually replayed and which required states were unavailable (Chapter 6)? Do the authors distinguish a prospective replay path, an executed replay receipt, and an independent reproduction?
Model identity. Do the authors record model revisions, sampling settings, serving or runtime identities, and access dates when exposed? Do they mark unexposed details as unavailable, leaving exact model identity and replay unresolved rather than inferring them from a floating alias?

A.9 Realized Human and Machine Cost

While a declared budget limits our work before execution, we expect the realized cost to report the resources the study consumed.

Search cost. Do the authors account for the simulator hours, model or API calls, licenses, and compute consumed to find the result, comparing it against a credible alternative complete workflow?
Amortized cost. For learned methods, do the authors report pretraining, fine-tuning, and adaptation costs, and do they apply any allocation based on a declared reuse horizon consistently across comparison arms?
Human cost. Do the authors report the human effort required for setup, steering, discarded prompts, diagnosis, review, repair, and decision work rather than hiding it?
Cost asymmetry. Do the authors explicitly account for the economic imbalance between cheap, high-volume large language model (LLM) generation and the expensive EDA verification required to validate those candidates?

A.10 Claim Support and Reopening Conditions

We must evaluate every claim within the limits of what its underlying observation source can reliably establish.

Observation source. Do the authors bind each claim to the analytical or trace-driven model, simulator, RTL or gate-level flow, emulator, fabricated silicon, or other source that returned the evidence? Do they record that source’s represented state, version, conditions, cost, scope, and blind spots (Chapter 4; Chapter 6; Chapter 7)? (These sources do not form a universal ranking.)
Physical signoff. If a claim depends on physical signoff, do the authors name the specific stage, constraints, modes, and corners?
Reopening conditions. Do the authors state which later workload, software, tool, process, or silicon observation would invalidate or narrow their recommendation?

A.11 Technical Ownership and Decision Boundaries

In our external review process, we identify who qualifies each result, who integrates the system recommendation, and who holds the authority to approve a consequential design transition (Chapter 11).

Domain results. Do the authors identify the domain specialists responsible for qualifying the relevant verification, timing, physical-design, power, safety, security, or reliability results and their scope?
Architecture synthesis. Do the authors identify the architect or architecture team that set the system scope, bounded automated actions, integrated the domain results, and stated exactly what the cross-layer evidence supports?
Commitment and accountability. If the work informs a consequential design transition, do the authors distinguish the named commitment authority from the accountable organization that assigns and resources the technical roles and remains answerable for product consequences?
Challenge capacity. Can the responsible specialists and architect inspect decisive artifacts and failures, commission or reproduce selected checks, and stop or reopen an unsupported transition?

A.12 The Limits of the Checklist

While we provide this instrument to support transparent external review, it does not establish a valid design or authorize a transition. Property-specific verification, domain signoffs, a supported cross-layer architecture recommendation, and an authorized commitment by the named commitment authority remain separate requirements (Chapter 11). Conditional and unresolved items must remain visible in the final review rather than being obscured by averaging them into a single score.