Computational Reproducibility

Introduction: The Computational Result as a Scientific Claim

A computational result enters science as a claim about a relation between inputs, procedures, and outputs. A graph in a journal article, a confidence interval in a clinical study, and the predicted trajectory of a physical system may all be produced by software. Each can be inspected as a displayed result, but the image or number alone says little about the sequence of decisions through which it was obtained. The relevant scientific object includes transformations, assumptions, exceptions, programs, parameters, and the conditions under which the computation took place. The computational reproducibility argument advanced by Peng (2011) captures the significance of this shift: the ability to repeat an analysis gives the community a practical minimum for evaluating a computational finding when the original empirical event cannot readily be repeated.

The decisive epistemic question is whether a competent investigator, acting independently of the original researcher's memory, can trace the result back through a sufficiently specified process. An answer requires more than possessing a file named code or an archive named data. It requires correspondence between the version used and the version described, between the published figure and the command that produced it, and between a model's declared assumptions and the decisions embedded in its implementation. The National Academies report (2019) locates computational reproducibility in precisely this continuity of input data, computational steps, methods, code, and analysis conditions.

This monograph investigates that continuity as an attribute of the scientific record. It argues that reproducibility is valuable because the record becomes independently interrogable: errors can be located, alternative interpretations separated, and later investigators can determine which changes are evidentially relevant. The analysis also identifies circumstances in which successful recomputation supplies little support for the empirical truth of a conclusion. These conclusions are philosophical interpretations of the research literature; they are not presented as a new named theory or an experimentally demonstrated universal law.

Scope, Research Question, and Method of Inquiry

The central question is what makes a computational result independently checkable. The answer will depend on the object of inquiry. In a statistical analysis, the relevant result may be an estimated coefficient and associated uncertainty. In a simulation, it may be a numerical trajectory under specified initial conditions. In a learned model, it may be a distribution of evaluation metrics over repeated training runs. In a data-processing pipeline, it may be a transformed dataset and a documented mapping from source records to final variables. The problem is methodological because these tasks demand different kinds of reconstruction, and philosophical because each reconstruction supports particular claims about evidence and justification.

The method is a critical narrative synthesis of selected primary scientific literature, institutional reports, and specification texts, supported by close reading of accessible claim-bearing passages. It compares the National Academies terminology (2019) with the ACM artifact policy (2020), examines empirical studies by Stodden, Seiler, and Ma (2018) and Pineau et al. (2021), and follows the practical requirements described by Sandve et al. (2013), Wilson et al. (2017), and the standards communities behind FAIR and W3C provenance. The review selects sources for direct relevance and documentary authority. It does not claim to search every database, measure publication prevalence, or perform a quantitative meta-analysis.

The original analytical contribution is a connected account of which dependency must be preserved to justify which scientific assertion. Constructed cases are explicitly distinguished from historical evidence. Technical practices are evaluated as instruments of inquiry, while the judgment that a computation provides evidence remains tied to design, data quality, mathematics, and interpretation. Bibliographic metadata and accessible primary claims were checked; where subscription barriers restrict reading, conclusions are confined to what those sources publicly establish.

Historical Origins: Executable Scientific Documents

The early computational reproducibility movement was an attempt to change the relationship between scientific prose and its figures. Claerbout and Karrenbach's 1992 paper, Electronic Documents Give Reproducible Research a New Meaning, described a technical arrangement in which a reader could recalculate the results associated with an electronic document. Their intervention matters because it treated a publication as an executable intellectual object. Its graphs were connected to data and programs through an operational sequence rather than preserved as detached illustrations.

Subsequent computing practice expanded the object that needed to be preserved. Research analyses began to depend on modular packages, interactive environments, remote services, complex numerical libraries, and parallel machines. Under these conditions, a file containing a command could preserve an intention while leaving the execution itself indeterminate. Peng (2011) located recomputation within scientific evaluation, while Sandve et al. (2013) translated such ambitions into reproducible workflow habits. The historical progression was from connecting figures to commands toward preserving the entire account of how commands acquire their inputs, dependencies, and meaning.

The deeper continuity is that scientific publication has always needed enough detail to permit critical examination of method. Computational technologies alter the kinds of detail required. A measurement instrument historically called for calibration records; an algorithm calls for implementation and dependency records; a distributed workflow calls for a trace of the stages that generated and consumed data. The historical lesson is that a publication's claim to be examinable depends on what the publication makes accessible. The exact mechanism changes with the technology, while the underlying demand for independent criticism remains stable.

Defining Reproducibility, Replicability, and Repeatability

Terminological confusion has practical consequences because the same reported success can describe very different operations. The 2019 National Academies definitions identify reproducibility with obtaining consistent results from the same input data, computational steps, methods, code, and analytic conditions. Replicability refers to agreement across studies directed at the same scientific question but using newly obtained data. The report also distinguishes generalizability, which concerns whether findings apply beyond the initial context. These definitions organize this monograph.

Some computing associations historically interchanged the words, and the ACM artifact badging policy records a 2020 harmonization of its terminology with wider scientific usage. Under the ACM scheme, results reproduced with author-supplied artifacts and results replicated without author-supplied artifacts receive different validation badges. Rather than adopting dictionary meanings without qualification, a scholarly writer should state the governing vocabulary and the operational test. The substantive distinction is between retracing a provided analytical pathway and independently creating new evidentiary material or an independent implementation.

Repeatability is best treated here as the ability to rerun a defined operation under stated conditions, often within the same system or organizational setting; the term has different formal meanings across fields. A repeated computation can be reproducible without an empirical study being replicable. A replication may vindicate a conclusion even if the original scripts no longer run. Computational reproducibility therefore has its own scope and must not inherit every expectation associated with general research replication.

The Object of Reproduction: Claims, Tables, Figures, and Files

Scientific papers contain heterogeneous results. A reported figure might be reproducible from a matrix that was itself generated by undocumented preprocessing; a table might be reconstructed exactly while the underlying sampling frame remains disputed; a statistical significance statement might depend on a correction omitted from a caption. An evaluation must first specify the target: a file checksum, numerical values to a tolerance, a visual figure, a statistical estimate, or the inferential conclusion built from them. The label reproducible becomes meaningful only when coupled with a target and a criterion for agreement.

The ACM badge definitions explicitly acknowledge tolerances appropriate to the experiment and do not demand universal exact identity of results. The National Academies discussion likewise frames consistent outcomes in terms of the same analysis conditions, while recognizing discipline-specific interpretations. This is a substantive correction to the widespread temptation to equate reproduction with equality of every byte. A simulation may be scientifically stable despite minor roundoff differences; a public-health model may change its estimated effect if a revised data-cleaning rule is used. Both events require interpretation against the claim originally advanced.

A responsible report should enumerate the named endpoints and their acceptance criteria before conducting a rerun. If a paper contains thirty figures, verifying only one must be described as a partial result. If a notebook executes without error but its output differs from the published values, successful execution and result reproduction must be recorded separately. Granularity protects the integrity of the verdict.

Inputs: Raw Data, Transformations, and Selection

A computational pipeline begins with an account of what entered it. The phrase original data may refer to files downloaded from a repository, recordings made by an instrument, a snapshot of an administrative database, or a curated dataset produced through many earlier transformations. Retaining the last table fed to a statistical package may make that final calculation repeatable while concealing selection rules, exclusions, and corrections that shaped the table. Accordingly, an input archive must say what is raw relative to each stage and document the links between these stages.

The FAIR Guiding Principles (Wilkinson et al., 2016) address discoverability, accessibility, interoperability, and reuse through persistent identifiers and rich metadata. They improve the conditions for finding and interpreting data, but FAIR status is not a certification of measurement validity or inferential correctness. The W3C provenance ontology (2013) supports another task: representing the entities, activities, and agents associated with derived artifacts. Used together, these resources help describe both the data objects and the transformations by which a later object came into being.

The philosophical consequence is that data lineage belongs to the evidential account of a result. Suppose a survey table contains a variable named exposure. If its derivation from source fields and missing-value rules cannot be recovered, reproducing the regression does not resolve what the coefficient estimates. Reproducible computation requires an identified path from research observations to the symbols manipulated by the final algorithm.

Source Code, Programs, and the Executed Version

Source code is indispensable to many reproducibility projects, but code availability alone supplies an incomplete record. The program a reviewer sees may differ from the one that generated the published result. A last-minute correction can be committed after figures were exported; a collaborator may run an uncommitted local change; a notebook may retain an output generated before the latest cell edits. Version-control history and a tagged or content-addressed snapshot reduce such ambiguity, while a release note must identify the executed revision rather than simply the latest revision.

Wilson et al. (2017) recommend organized projects, version control, and automation in their good-enough computing practices. The software citation principles (Smith et al., 2016) make specificity and persistence central to recognizing software used in scientific work. In this setting, citation does more than distribute credit. It allows investigators to distinguish the software that produced a result from a later product bearing the same name. An exact software revision can be material to the outcome of a numerical method or preprocessing routine.

A Git commit identifier is useful evidence of repository state, provided the repository contents and relevant external resources are available. It is not an automatic snapshot of data stored elsewhere, secret configuration, external APIs, or dynamically downloaded dependencies. The meaning of a source revision must therefore be supplemented with a manifest of the resources and environment actually invoked during the original computation.

Parameters, Random Seeds, and Unrecorded Decisions

Many published results depend on values that lie outside the main algorithm: parameter choices, stopping rules, initialization, thresholds, optimization settings, model-selection procedures, or data partitioning. A function's source code may be identical across runs while its behavior changes through a configuration file or command-line option. Such choices deserve first-class status in the provenance record. Sandve et al. (2013) emphasize the habit of recording how a computation was carried out in their ten rules for reproducible research.

Random seeds can help reproduce a particular pseudorandom sequence, but they are not universal guarantees of deterministic outcomes. A seed may control one library while parallel execution, accelerator kernels, asynchronous operations, or data-loader scheduling introduce additional variation. The problem is especially visible in machine learning, where deep reinforcement learning experiments examined by Henderson et al. (2018) exhibit variability across runs and baselines. The remedy is an explicit description of stochastic sources, not a promise that adding one integer will freeze an entire system.

Parameter provenance also demands a record of decisions made before the reported run. If twenty thresholds were tested and only the most striking was disclosed, the selected calculation might reproduce exactly yet support an overstated inference. Reconstructing the computation is a means of detecting this question; answering it requires a transparent analytic history and appropriate inferential treatment.

Software Dependencies, Environments, and Compatibility

A program runs within a stack of dependencies. Packages may call numerical libraries, operating-system components, compilers, interpreters, system utilities, data format readers, and external network services. Their versions can affect behavior even when the visible script never changes. The Turing Way environment guide explains how differences in operating systems, software packages, and related system features can prevent a transferred project from running or yielding consistent outputs. Environment capture is therefore integral to an independently repeatable result.

Dependencies must be resolved as actually installed rather than described only by permissive ranges. A package manifest that requests any compatible release may recreate a different environment months later. A lockfile, image digest, or equivalent immutable reference narrows this ambiguity. Even a successful rebuild of a container image may depend on mutable registries or packages that were replaced, so long-term strategies require archived artifacts and recorded content digests alongside instructions.

An environment is not simply a list of brand names. It is the set of operational conditions that the result materially depends on. A statistical script may tolerate several library versions; an old climate model may require a particular compiler and numerical routine; a machine-learning experiment may need an accelerator architecture and driver combination. The proper level of preservation follows the sensitivity of the claim, rather than a universal ritual of recording every trivial machine characteristic.

Reconstructing Workflows Rather Than Isolated Commands

Complex results emerge through ordered and conditional workflows. Data may be fetched, cleaned, joined, analyzed, rendered, and summarized through separate programs, each generating inputs for the next. Without explicit dependencies, a reviewer must infer execution order from filenames or the memory of the authors. A reliable workflow description identifies which stage consumes which artifact, the criteria for stage completion, and the outputs expected at each boundary.

The research compendium analysis by Nüst and Pebesma (2021) describes a package of code, data, environments, and script-based workflows as a practical means of scholarly communication. Their account suggests that the unit to preserve is a connected research process rather than an accidental directory copied from a working computer. A directed dependency graph or equivalent executable specification helps readers determine which changes require downstream recalculation.

A workflow graph, however, can conceal manual interventions. If a technician removed an outlier by hand or edited a spreadsheet cell outside the scripted pipeline, the automated trace begins after an already significant decision. Documentation must preserve such interventions or redesign the workflow to encode them transparently. The practical goal is an auditable sequence in which every material transformation has an identified input, agent, operation, and output.

Notebooks, Literate Programming, and Hidden Execution State

Interactive notebooks combine exposition, code, plots, and exploratory interaction in a single document. This makes them persuasive vehicles for explaining analyses, and it also creates a risk: the visible order of cells may not match the order in which the author executed them. A figure can depend on a variable created by a deleted cell, a cached object from a previous experiment, or a file that exists only on the originator's machine. A notebook that displays a finished figure therefore gives evidence of a result but does not, by itself, document a reconstructible route to that result.

The historical aspiration of executable research documents in Claerbout and Karrenbach (1992) connects directly to this modern environment. The link between prose and computation becomes scientifically effective when the document can be executed from a clean state and its dependencies are visible. The practical guidance in Wilson et al. (2017) favors scripted and documented work over reliance on fragile manual sequences. Notebook execution from a clean kernel, ideally within a fresh provisioned environment, tests whether a stated order is sufficient to produce the intended results.

Literate presentation and full reproducibility serve different purposes. A notebook may explain a model beautifully while omitting a proprietary input. A separate pipeline may reproduce every figure while offering little intellectual account of the assumptions. The best scholarly object brings the two functions into correspondence: explanation identifies why the steps matter, and the executable record shows precisely which steps were performed.

Containers, Virtual Machines, and the Limits of Encapsulation

Containers and virtual machines can package substantial portions of a computational environment. A container image may capture versions of user-space libraries and executables; a virtual machine can represent a broader operating-system context. Such techniques often reduce setup effort for independent investigators, especially when combined with a run script and integrity-checked data. The geoscience compendium literature treats containerization as one contribution to practical reuse of computational investigations, and the environment preservation guide addresses the wider dependencies involved.

Encapsulation is powerful because it moves the burden of reconstruction away from undocumented installation memories. It is still conditional on what the package captures. Host operating systems, processors, graphical drivers, GPU libraries, kernel behavior, network access, licensing servers, and external datasets can remain outside a container. An image digest ensures that the same image bytes are referenced; it does not prove that the image can run indefinitely or that an image running on different hardware will give mathematically identical results.

A durable package consequently distinguishes what is inside the boundary from what must be supplied by the reviewer. It should declare architecture, required resources, expected runtime class, external endpoints, and permissible differences in outputs. The relevant test is whether the preserved configuration supports the scientific verification task, rather than whether the package uses fashionable infrastructure.

Numerical Reproducibility and Acceptable Tolerances

Numerical algorithms produce approximations within the constraints of finite-precision representation, iterative stopping rules, and computational implementation. Floating-point arithmetic is sensitive to operation order; parallel reduction can change that order, and a small intermediate difference may grow under an unstable transformation. Exact checksum identity of result files is an appropriate standard for some deterministic pipelines, but it can be inappropriate for a chaotic numerical simulation or a stochastic optimization process. Scientific evaluation must specify an agreement criterion meaningful for the quantity at issue.

The ACM validation guidance recognizes experiment-appropriate tolerance and states that exact numerical equality is generally neither necessary nor expected for validating research outcomes. The National Academies definitions require consistency under the relevant conditions of analysis. These principles allow an important distinction: a different final decimal place may be harmless if a result remains within justified precision, whereas a modest change can be decisive when it alters a classification close to a diagnostic threshold or reverses a policy decision.

Tolerance cannot be selected after seeing a disappointing rerun merely to turn failure into success. It should follow a prior account of numerical error, stochastic variation, measurement resolution, and the substantive hypothesis. Reproducibility consequently requires explicit numerical criteria and a defensible explanation of how those criteria relate to the scientific claim.

Determinism, Stochasticity, and Ensembles

Some computations define a single output for a specified input and execution environment. Others describe random processes whose outcomes must be assessed as distributions. For stochastic workflows, requiring the same one-off realization may test the random-number machinery rather than the study's scientific conclusion. The proper object of reproduction can be a confidence interval, distribution of estimates, rate of failure, calibration measure, or relationship stable across a prespecified ensemble of runs.

The findings described in Deep Reinforcement Learning That Matters (Henderson et al., 2018) demonstrate why performance scores from only one training run are precarious evidence for comparative improvement. Pineau et al. (2021), reviewing a large machine-learning reproducibility initiative, discuss checklists, code submission, and community reproduction as means of improving the record available for scrutiny. Their NeurIPS reproducibility study treats openness and careful reporting as mutually reinforcing features of research practice.

The philosophical task is to match the unit of repeatability to the kind of knowledge being claimed. When an experiment reports an expectation over randomly initialized models, its supporting record must enable scrutiny of the repeated-run design. When it reports a particular checkpoint as a deployable artifact, the precise checkpoint and associated preprocessing become salient. Both may deserve preservation, but they test different propositions.

Parallelism, High-Performance Computing, and Hardware

Distributed and high-performance computation adds dependencies that cannot always be embedded in a portable package. Parallel work partitions data, schedules processes, exchanges intermediate messages, and sometimes uses hardware-specific algorithms. Result identity can depend on the order in which partial computations are combined, on nondeterministic scheduling, and on differences in numerical libraries compiled for particular architectures. A computational result generated on a large cluster may also be difficult to reproduce because the original resource allocation is unavailable to many investigators.

The National Academies account (2019) establishes reproducibility in relation to conditions of analysis; the Turing Way guidance makes the computational environment part of those conditions. From these premises follows a practical expectation: archives should disclose machine architecture, resource characteristics, software builds, execution flags, and any known sources of nondeterminism when these may change the outcome. Resource consumption and execution time should be reported with enough context to distinguish an inaccessible computation from a broken workflow.

An independent check need not always duplicate the complete original infrastructure. A reduced test case can validate a numerical kernel, unit tests can establish behavior of individual modules, and a subset of results may be reproduced on alternative hardware. Such work offers partial evidence and should be labeled as such. The full scientific claim remains subject to the scope of the computational work actually inspected.

Verification, Validation, and Reproduction

A computation can be reproducible while solving the wrong mathematical problem, applying an inappropriate statistical method, or relying on a defective model of its target system. Software verification asks whether an implementation conforms to its specification or intended computational method. Model validation concerns whether assumptions and outputs are adequate for the scientific purpose. Reproduction concerns whether an identified computation can be run again with materially consistent results. These inquiries overlap operationally but differ in what they establish.

The ISO/IEC 25010 product quality model (2023) supplies a broad vocabulary for evaluating software product quality, but it is a quality model rather than a direct proof of scientific validity. The artifact evaluation policy explicitly separates artifact functionality and availability from independently reproduced or replicated results. This separation is analytically valuable: an executable artifact may be well documented and operational while the proposed explanation of a natural phenomenon is poorly supported.

A strong review therefore asks successive questions. Did the intended code run? Did it generate the claimed numbers? Were the numbers obtained from appropriate data and methods? Do the numerical results warrant the interpretation? The answers can diverge. Reproduction provides access to the reasoning chain and a way to test its consistency; scientific justification depends on the chain's content.

Statistical Inference, Researcher Degrees of Freedom, and Reproducibility

In statistical workflows, an analyst may choose exclusions, transformations, model formulas, covariates, stopping decisions, and presentation formats. Recording only the final selected script can permit perfect recalculation of a result while obscuring the search that made the result appear notable. This is why computing practices supporting analytical transparency should include, where relevant, a decision record, prespecified analysis plan, or transparent account of analytic alternatives.

Peng's minimum-standard argument (2011) acknowledges that reproducible analysis is a practical baseline. The baseline is useful precisely because it permits others to check calculations and examine hidden assumptions. But it is a baseline. The National Academies report (2019) distinguishes the possibility of repeating a computation from the broader question of whether a scientific result survives new evidence. Statistical inference additionally depends on sampling, modeling, measurement, and the multiplicity of choices explored.

Consequently, a computational archive should preserve more than the successful branch of an analysis when the unreported branches bear on uncertainty. The extent of that record is contextual: preliminary exploration can legitimately be open-ended, while confirmatory claims demand transparent treatment of selection and uncertainty. A reproducible p-value is a reproducible calculation; its evidential force must still be assessed.

Machine Learning: Datasets, Splits, and Model Artifacts

Machine-learning research often combines a training dataset, data preprocessing, train-validation-test partitions, a model architecture, optimizer settings, initialized weights, and evaluation code. A replication package can contain the architecture without the final trained weights, the weights without the preprocessing, or the evaluation script without the data split. Each omission blocks a different form of inspection. Exact reproduction of a benchmark score requires attention to dataset version and selection rules as well as program implementation.

The practical taxonomy by Tatman et al. (2018) describes increasing levels of machine-learning reproducibility associated with disclosure of algorithms, code, data, and environments. The NeurIPS 2019 program report by Pineau et al. (2021) records policy interventions spanning code submission, a community challenge, and a paper checklist. Such interventions improve the inspectability of machine-learning claims, while their own success should be assessed against documented uptake and actual reproduction outcomes rather than inferred from the existence of a checklist.

In applications where model training is prohibitively expensive, a released checkpoint can permit evaluation but cannot establish how training produced that checkpoint. The manuscript should distinguish reproduction of predictions, reproduction of training, and independent verification of the underlying scientific claim. Computational reproducibility becomes a set of explicitly scoped tests that address different stages of the model lifecycle.

Automated Analysis, Generative Systems, and Hidden Services

Machine-mediated research increasingly uses hosted APIs, proprietary models, retrieval systems, and generative processes whose internal weights, service versions, or sampling regimes are unavailable to the investigator. A saved prompt may be only one component of the operation that yielded a result. The response can also depend on retrieval indexes, moderation layers, hidden system instructions, remote model updates, and parameters controlled by a provider. These conditions complicate claims that another researcher can reproduce the calculation from its visible interface alone.

The principle emerging from W3C provenance descriptions and the computational reproducibility definition is to identify both observable process elements and dependencies that remain outside the researcher's control. Record the service identifier and date, parameters actually exposed, input and output hashes, retrieved sources when lawful, and a contemporaneous execution trace. These measures establish a documentary record of a particular operation. They do not create access to undisclosed internals or guarantee future output identity.

Researchers must state whether the intended test is recovery of one archived output, repeat execution of a live service, or independent corroboration of a scientific inference. The first may be possible after the service changes; the second may be impossible; the third can remain meaningful if designed around independently collected evidence. Clear statements of controllability protect computational research from fictional guarantees of repeatability.

Data Lineage, Provenance, and Scientific Responsibility

Provenance links an output to its production history. The PROV-O specification formalizes relationships among entities, activities, and agents so that such histories can be exchanged among systems. In a reproducible study, a provenance statement can show that a data-cleaning activity used a dated dataset and generated a transformed file, or that a statistical analysis used that transformed file and generated a plot. A machine-readable statement is especially valuable when hundreds of artifacts must be tracked across many stages.

Provenance is also interpretive. A declared activity named validated-data is not automatically an adequate validation procedure, and a recorded author's identity says nothing by itself about whether a conclusion is correct. The value of lineage lies in enabling questions about credibility to be asked at the relevant stage. If a surprising output resulted from a newly substituted input file, the trace can reveal the substitution and direct attention to its evidential effects.

The relation to Aisentica's account of artificial provenance is conceptual and separately attributed. That philosophical project treats provenance as an object of public meaning and attribution. Scientific computational provenance supplies a more operational account of the paths by which artifacts and results are produced. Both make origin visible, while the evidential assessment of a calculation still depends on whether the originating steps were appropriate. Their juxtaposition here is a comparative interpretation rather than an equation of the two conceptual systems.

Persistent Identifiers, Archives, and Long-Term Reuse

A working repository is usually a dynamic environment. Branches move, external dependencies disappear, institutions change hosts, and files are corrected after their first dissemination. A reproducibility record requires stable identification of the exact materials used, with durable access to them as far as ethically and technically possible. The Software Citation Principles (Smith et al., 2016) emphasize specificity and persistence, including appropriate reference to the version of software used. A study that cites only a project homepage may become impossible to reconstruct after a major update.

The SWHID specification offers intrinsic identifiers intended to pinpoint source-code objects by their content. The Citation File Format offers machine- and human-readable software citation metadata. These instruments solve different parts of the problem: one identifies a specific software object; the other communicates citation information about an artifact. Neither, by itself, archives all input data, confirms legal access, or proves that a historical environment remains runnable.

Preservation must distinguish bit-level integrity from operational usability. An archive may retain every byte of a program while losing the interpreter needed to run it. A durable scholarly record should preserve source and binary references where justified, execution instructions, dependency manifests, expected outputs, and metadata explaining why each object matters. Long-term reproducibility is an organized maintenance problem as well as an act of initial disclosure.

Research Compendia as Scholarly Publications

A research compendium is a collection of materials assembled to support the reconstruction, evaluation, and reuse of a study. Its purpose is to unite what a traditional article typically disperses: the prose statement, data, computational code, environmental assumptions, parameters, and scripts generating tables and figures. Nüst and Pebesma (2021) describe this orientation in their practical reproducibility analysis of geography and geosciences. The idea is transferable beyond geography because the dependency between data transformations and published conclusions appears in many fields.

A useful compendium should give an unfamiliar investigator an entry point: what claim is being tested; what files matter; how to acquire allowed inputs; which command to run; how long it should take; and which outputs count as evidence of success. It should state access licenses and limits, distinguish raw from derived data, and carry an inventory of archived versions. The scientific computing recommendations of Wilson et al. (2017) provide pragmatic habits for constructing such a package during ordinary research, rather than attempting retrospective reconstruction months after publication.

A compendium's existence is not equivalent to independent reproduction. Its scientific quality must be tested by somebody who can follow its instructions without unrecorded help. The document can also contain material whose reuse is restricted or whose legal retention is time-limited. The complete publication object therefore includes a transparent account of what is packaged, what is externally referenced, and what independent reviewers can actually verify.

Metadata Standards: FAIR, RO-Crate, and Interoperability

Machine-readable metadata makes a scientific object intelligible beyond its author's local directory conventions. The FAIR principles articulated by Wilkinson et al. (2016) focus on findability, accessibility, interoperability, and reusability, emphasizing the ability of machines to discover and use data. The FAIR source paper addresses stewardship rather than promising that a dataset is correct. Rich descriptions support responsible inspection by identifying objects, formats, licenses, provenance, and relations among materials.

The RO-Crate 1.2 specification (2025) defines a way of packaging and describing research objects through JSON-LD metadata. A research crate can associate files, datasets, software, authors, and contextual information within a portable structure. This offers a common descriptive layer connecting content that would otherwise depend on site-specific folder names or ad hoc text. The RO-Crate version index also illustrates why editorial archives should identify the specific standard version followed: a live project can introduce newer recommendations while earlier research packages remain valid under an identified version.

Standards improve interoperability when they document concrete relationships and identifiers. A metadata record that merely claims reproducible=true without a tested procedure conveys a conclusion without its evidence. The defensible use of these standards is to make artifacts discoverable and traceable, then expose the tests by which a reviewer can assess the particular research result.

Empirical Evidence: What Journal Policies Accomplish

Advocacy for data and code sharing is strongest when accompanied by measurements of actual access and reproducibility. Stodden, Seiler, and Ma (2018) selected 204 Science articles from a period following the journal's data-and-code availability policy and attempted to obtain artifacts and reproduce findings. Their empirical policy assessment reported obtaining artifacts for 44% of the sample and reproducing findings for 26%. These are sample- and protocol-specific outcomes, not a universal rate for science or a direct estimate of research fraud.

The study provides evidence that publication policies relying on supplying materials upon request can leave substantial gaps between a requirement and an independently actionable record. Some missing artifacts may reflect failed correspondence, technical obsolescence, restrictions, or incomplete preservation; an unsuccessful reproduction does not identify a single cause without further investigation. A policy should therefore distinguish a written promise to share materials from verified availability and successful re-execution by a suitably qualified reviewer.

The broader analytical lesson concerns the structure of incentives. Journal language can signal a norm, but compliance depends on what is deposited, how it is described, who checks it, and whether it remains accessible. Public and time-stable research packages reduce dependence on one author's email address. Actual independent reproduction tests provide stronger evidence of practical usability than a declaration of availability alone.

Discipline-Specific Tests and Geographic Research

Reproducibility practices must respond to the sources and computations typical of a discipline. Geographical work may depend on spatial reference systems, raster and vector data formats, map projections, coordinate transformations, specialized geospatial libraries, and licensed base maps. A workflow that reproduces statistical coefficients but transforms coordinates differently can change the spatial substance of a result. Data licensing and platform versions therefore become scientific concerns rather than peripheral implementation choices.

Nüst et al. (2018) studied reproducibility and material availability in AGILE conference papers, while Konkol, Kray, and Pfeiffer examined researcher practices and reconstruction through a mixed-method investigation reported in geoscientific reproducibility research (2019). These studies support the claim that documentation and executable environments are central to checking geographical analyses. Their samples and disciplinary settings limit the generality of any prevalence claim drawn from them.

The value of case-specific analysis is that it identifies material dependencies invisible in generic checklists. An Earth-science model may need calibration records; a digital humanities analysis may need an identified text edition and tokenization rules; a genetics pipeline may need genome reference versions. A universal account of scientific accountability can coexist with locally specific reproduction requirements.

Artifact Evaluation and the Difference Between Access and Function

An artifact may be published in a repository without ever being exercised by an external investigator. Conversely, a package may function for an evaluator under controlled access without being publicly downloadable. The ACM artifact review scheme makes these distinctions explicit through badges for evaluated artifacts, available artifacts, and validated results. Within evaluation it distinguishes functionality and reusability; within validation it differentiates reproduction using author-supplied artifacts and replication without them.

These badges are informative because their criteria describe different acts of checking. Availability concerns retrieval. Functionality concerns whether documentation and artifacts support execution. Reusability adds a more demanding assessment of quality and transfer. Reproduced results concern the main findings obtained by another investigator using supplied materials. Replicated results concern independent derivation without those author-supplied materials. A badge should be read as evidence of the named review activity, not as a certificate that every claim in a paper is true.

The distinction also defines an editorial duty: conclusions about a reproducibility package must report what was actually run. An internal check of syntax and links is narrower than independent artifact evaluation, and a successful artifact test is narrower than external empirical replication. Accurate scope declarations protect scientific credibility because they state the relation between a documented procedure and the conclusion drawn from it.

Accessibility, Restricted Data, and Confidential Research

Computational reproducibility is frequently described as a demand to publish all data and code openly. Scientific practice operates within legal duties, informed consent, intellectual-property rights, community interests, and legitimate confidentiality requirements. A dataset containing identifiable health records cannot simply be posted for unrestricted download. The relevant question becomes how to preserve meaningful independent scrutiny while maintaining authorized control over protected material.

The FAIR principles include accessibility under well-described conditions and do not define accessibility as unrestricted public release. The CARE Principles for Indigenous Data Governance (Carroll et al., 2020) foreground collective benefit, authority to control, responsibility, and ethics in data governance. They expose a limit of purely technical understandings of access: data reuse can affect people and communities whose rights exceed the convenience of a subsequent analyst.

Practical alternatives include qualified secure enclaves, data access committees, disclosure-reviewed outputs, synthetic datasets for code demonstrations, and trusted independent auditors with recorded authority. A synthetic dataset can show that a script is executable; it generally cannot reproduce the original substantive findings unless its evidential properties are specifically established. The publication must distinguish reproducible software behavior from results checked against confidential source data. Transparent constraints and independently governed access are stronger scholarship than an impossible promise of universal openness.

Open Research and the Economics of Maintenance

Reproducibility imposes costs: time to clean code, write documentation, resolve licenses, preserve dependencies, and answer auditors' questions. Some work is a one-time task; other work requires continuing infrastructure. These costs should be understood alongside the cost of studies that cannot be verified, papers whose figures cannot be regenerated, and errors discovered only after substantial follow-up investment. The good-enough practices offer attainable starting habits designed for ordinary researchers, while Stodden and colleagues reveal limits of relying on postpublication requests for material.

Institutional policy shapes whether such labor receives recognition. A data curator or research software engineer may contribute decisively to the quality of a study while remaining invisible in its conventional author list. The software citation principles create a path for crediting software as a legitimate research output. Preservation and verification should likewise be eligible for documented scholarly credit. These are normative editorial recommendations supported by the observed roles and obstacles discussed in the source literature.

A sustainable policy must distinguish minimal evidence-bearing records from expensive optional refinements. The goal is not to force every small analysis into a complex infrastructure. It is to ensure that the cost of preserving a result's material dependencies is paid while the knowledge of those dependencies is still available. Reproduction becomes an ordinary part of knowledge production when institutions reward the work and make it technically feasible.

The Role of Negative Results, Error Reports, and Divergence

A failed reproduction attempt can indicate many distinct circumstances: an archive may be incomplete, a command may not execute, the environment may have changed, the investigator may have misread instructions, or the calculation may produce values inconsistent with those reported. These possibilities require separate reporting. A single label of irreproducible can conceal both a trivial installation problem and a substantial mismatch affecting a scientific claim.

The National Academies report highlights why different kinds of reproduction and replication failure demand interpretation. An unsuccessful attempt to reproduce an analysis from the same data can point to missing computational detail or inconsistency. Failure of a new-data replication can reflect sampling variation, contextual change, measurement differences, or an unsupported original inference. The events may have different epistemic significance and should not be combined into an undifferentiated count.

A useful discrepancy report names the attempted target, exact materials, environment, command sequence, observed behavior, expected behavior, and the claim affected. It records any assistance obtained from the authors and whether that assistance changed the effective protocol. Negative results can then become cumulative knowledge about the limits of a computational claim, rather than private episodes of frustrated debugging.

Reproduction Reports and Calibrated Editorial Judgments

An evaluator should describe the scope of a reproduction attempt before interpreting its outcome. The main questions are whether the required artifacts were obtainable, whether the documented process executed as specified, which outputs were generated, how those outputs were compared, and whether the principal interpretation remained supported. Partial coverage requires an explicit boundary. If only two figures among twelve were checked, the published report should identify those two and avoid a whole-paper success claim.

The ACM artifact framework separates several levels of evaluation, while Pineau et al. (2021) document community practices for reproducibility review in machine learning. From these sources follows a defensible editorial sequence: establish the intended object, record material versions and conditions, perform the check, preserve logs and output checksums, compare by prespecified criteria, and report limitations. No single badge or threshold can replace the written account of what was tested.

Editorial judgment is a claim about evidence gathered through a procedure. It must therefore remain corrigible and bounded. The review may show that a provided package is runnable on one operating system and that one estimated effect matches within rounding. Such a report is valuable in precisely those terms. Stronger conclusions need stronger tests, independent analyses, or replication with new evidence.

Constructed Case I: Exact Computation of an Incorrect Variable

Consider a hypothetical environmental study reporting a relationship between particulate exposure and hospital visits. The authors supply a complete analysis script, an environment image, a fixed seed, a data table, and an automated command that rebuilds every published figure exactly. An independent evaluator obtains the same coefficients to all displayed digits. The computational reproduction is successful under the National Academies definition because the specified inputs and calculations yield consistent results.

During a later code review, the evaluator discovers that a column named daily_exposure was constructed from a weekly aggregate and assigned to each day in the reporting period. The transformation is deterministic and fully reproducible, yet it changes the unit represented by the predictor. The published model may answer a different scientific question from the one described. The mathematical computation has been recovered; the interpretation of its variables is defective.

This example is constructed for analysis and describes no actual study. It demonstrates why scientific warrant has several layers. Re-execution supports confidence that a reported computation corresponds to an identified sequence of operations. A substantive audit must additionally examine measurement and the mapping between data variables and research concepts. Publishing the script makes the problem visible and correctable, which is an epistemic achievement even when the original inference must be revised.

Constructed Case II: A Different Numerical Output, the Same Supported Claim

Consider a hypothetical turbulence simulation run on two computing systems. Both use the same recorded initial conditions, numerical method, and physical parameters. Their floating-point reductions occur in different parallel orders. One system reports a peak value of 1.0048 and the other 1.0051. The study's claim concerns whether the peak stays within a prespecified physically meaningful interval of 0.98 to 1.03 after documented discretization and numerical uncertainty have been assessed. In this constructed situation both outputs support the same bounded conclusion.

A demand for bitwise equality would report a discrepancy even though the physical interpretation remains stable. The ACM policy on acceptable tolerance allows assessments of consistent results appropriate to the experiment, while the 2019 committee definitions organize the question around agreement under specified analysis conditions. The evaluator should still investigate unexpected drift; a tolerant criterion is justified only through the scientific and numerical context that makes it meaningful.

This case is illustrative, not an executed simulation or empirical result. It shows why the standard of agreement must be declared at the level of the claim. If the published claim instead asserted that the peak exceeded 1.0050, the two values would sit on opposite sides of the decision boundary, and the same numerical discrepancy would become scientifically decisive. Reproducibility is a relation between results and justified expectations, not a mechanical preference for exact numbers.

Constructed Case III: Restricted Clinical Data and Independent Checking

Consider a hypothetical multicenter clinical study whose patient-level data cannot lawfully be released to the public. The research team deposits a versioned analysis package with test data, a documented container image, input schemas, and a manifest of expected outputs. A separately governed accredited auditor obtains authorized access to the original protected dataset in a secure environment, runs the exact analysis, and records whether the prespecified tables and effect estimates match. The public can inspect the computational specification and the auditor's report, while the underlying patient records remain protected.

Under the FAIR guidance and the governance principles, scientifically meaningful access can be conditional and accountable. The public package demonstrates execution on test inputs; the controlled independent run can provide evidence about the original substantive calculation. These should be reported as different verification activities. The reviewer also needs to disclose whether the secure environment permitted meaningful inspection of the code, data schema, and outputs rather than only displaying a preselected success message.

This example is hypothetical and supplies no claim about an existing clinical program. Its implication is that confidentiality and reproducibility can be addressed together through appropriate institutional arrangements. The most defensible record states who was permitted to verify what, under which controls, and which parts of the claim remain unexamined by the general reader.

Reproducibility Across Time: Software Decay and Historical Authenticity

A computation that was executable in the year of publication can become unusable because of obsolete libraries, withdrawn datasets, incompatible compilers, or discontinued services. This creates a tension between preserving the historical environment and adapting a method for current platforms. A faithful archive of original code and dependencies is evidence about what was executed then. A ported workflow is an additional interpretive act whose modifications must be disclosed and tested.

The software identifier approach supports precision about source-code objects, while the RO-Crate standard offers structured descriptions connecting artifacts and contexts. The software citation principles strengthen the obligation to identify the exact version of software used. Together they support a distinction between preserving the identity of historical materials and establishing that the method remains usable on future infrastructure. These are complementary scholarly goals, not identical accomplishments.

A migration log should specify which components were substituted, why substitution was necessary, what tests demonstrate behavioral agreement, and which conclusions are sensitive to the changes. If a historical output can be regenerated only through a modified implementation, the publication should say that the original result was reconstructed through a validated port rather than claiming that the old environment was run unchanged. Provenance over time is as important as provenance within a single workflow.

The Epistemic Status of a Successful Reproduction

A successful computational reproduction is evidence about the relationship between a reported result and a specified computational process. Its strength depends on who conducted the examination, which original artifacts were used, which choices were independently reconstructed, what was compared, and how discrepancies were handled. An original investigator who reruns an unchanged notebook has performed a useful integrity check. An independent team that reconstructs execution from a deposited package can test additional assumptions about disclosure, documentation, and dependence on tacit knowledge. The National Academies account and the artifact assessment vocabulary supply different but compatible ways to articulate these distinctions.

The result of a reproduction also interacts with the prior credibility of a claim. If a disputed statistical association is reproduced exactly, the association's calculation acquires a stronger traceable basis, while concerns about confounding or selection remain to be addressed. If a result cannot be recovered, the cause must be investigated: corrupted input, inconsistent environment, missing parameters, unstable randomization, or an actual analytical mistake have different implications. A failure report that identifies a harmless compatibility error offers different evidence from a report discovering a wrong unit conversion. Distinguishing these cases is part of scientific judgment.

Computational reproducibility should therefore be recorded as a qualified finding with an explicit object, procedure, assessor, and comparison criterion. It can make a paper corrigible by exposing a specific faulty assumption and enabling a transparent correction. It cannot, by itself, determine whether the world conforms to the assumptions represented in that paper. That distinction explains both why reproduction matters and why scientific assessment must continue after a program runs.

Research Software as a Scholarly Object

Research software carries methodological choices within abstractions, interfaces, numerical routines, and default values. Those choices shape which questions can be asked, which operations are available, and how uncertainty is propagated. The scholarly object for examination may therefore include a particular source revision, its build scripts, test suite, documentation, dependencies, and defined inputs. Citing only the package's generic name fails to identify the operation used in an argument.

The software citation principles advocate precise identification, attribution, and persistence for software as a first-class research product. A citable version allows readers to establish whether two papers invoked the same algorithmic implementation, while intrinsic software identifiers provide another mechanism for referencing stable software objects. Such references are meaningful when their relation to execution is stated: citing a source revision alone does not prove that binaries were compiled from it or that a particular run used that executable. Provenance must link the source object to build and execution records.

This analysis favors evaluations that treat software documentation and artifact availability as integral to the publication's evidentiary claims. When a method relies on privately developed code, an examiner may need independent tests, controlled inspection, or an auditable technical description. The crucial issue is what questions can be checked and by whom. The scholarly value of software lies in making computational decisions inspectable and attributable across institutional and temporal boundaries.

A Practical Audit of an Executable Scientific Claim

An evaluator can begin by identifying the smallest complete claim whose computational support is being tested: a coefficient in a table, a simulated trajectory, a trained model's score, or an uncertainty interval. The evaluator then records the exact named input objects, access conditions, checksums, units, data exclusions, preprocessing choices, source revision, dependency manifest, interpreter or compiler, hardware-sensitive features, execution order, randomization procedure, and expected output. This sequence draws on the ten rules for computational work and the good enough computing guidance while remaining a documentary audit procedure rather than a newly proposed theoretical model.

The next question concerns independence. Did the evaluator obtain the deposited artifacts without personal guidance from the authors? Were any missing parameters inferred from manuscript prose? Did execution require modifying the source, relaxing error handling, substituting a dataset or changing operating-system interfaces? Each intervention should be recorded in an audit log. Independently checking a numerical comparison additionally requires a declared tolerance and a justification for it. Exact file identity is appropriate for immutable artifacts, while approximate numerical correspondence may be appropriate for certain algorithms, as the ACM assessment guidance recognizes.

Finally the reproduction report should identify the assertions actually supported, open discrepancies, and the conditions under which the finding might change. A pass means the specified computational claim was recovered under the stated conditions; it does not certify the full paper. A fail means a defined claim was not recovered under stated conditions; a diagnosis is needed to determine why. This makes technical checks intelligible to scientific readers and avoids transforming a complex judgment into a decorative green status icon.

Computational Reproducibility and Scientific Institutions

Scientific publication disperses responsibility across authors, funders, institutions, journals, infrastructure providers, software maintainers, and readers. Reproducibility depends on the combined actions of these participants. The authors can preserve executable materials; journals can require precise availability statements and facilitate artifact review; universities can maintain secure compute facilities; funders can support curation; archives can preserve persistent references. The FAIR principles and the ACM badging policy show why institutional arrangements influence what is made findable, accessible, and reviewable.

Policies based only on the presence of a repository link are vulnerable to superficial compliance. A repository may contain a detached notebook, undocumented input dependencies, obsolete packages, or prohibited sensitive data. Conversely, a study with restricted data may have an independently audited analysis with carefully documented access and evaluation procedures. Institutional review should therefore consider substantive accessibility, testing, and governance rather than equating openness with one form of hosting. Preservation also requires funding and maintenance, which a one-time submission checklist cannot guarantee.

The scholarly incentive problem is structural: the costs of documentation and maintenance occur before publication, while many benefits accrue later to researchers other than the original team. Practices that recognize software authorship, reward well-documented artifacts, and train researchers to produce maintainable code can rebalance these incentives. This is an institutional inference from the evidence on software citation and reproducibility programs, not a quantified estimate of the cost or effect of any particular reform.

Artificially Mediated Research and Attribution

Computational research increasingly includes automated code generation, machine learning, data cleaning assistants, search agents, and systems that propose or evaluate experimental pipelines. These tools can accelerate an analysis and also multiply the places where undocumented choices enter a workflow. A machine-generated script is a computational artifact with a traceable history. Its validity depends on its behavior, inputs, assumptions, and testing, just as any other script does. A description such as "AI-assisted" alone contains too little information to enable another researcher to reconstruct the operation.

An adequate record should preserve the generated code actually executed, relevant model and tool versions when known, consequential prompts or configuration, human editorial interventions, permitted external inputs, and independent tests of critical transformations. Where inference depends on nondeterministic external services, a claim of deterministic re-execution may be impossible; the author must preserve observed outputs and specify which parts can be rerun, which parts can only be independently reimplemented, and which require renewed empirical validation. The NeurIPS reproducibility report underscores the value of structured reporting and independently available artifacts in machine-learning research.

The Aisentica project develops a broader philosophical account of Artificial and provenance, as its canonical philosophical corpus records. Here that canon establishes attributional context, while the scientific question remains whether a particular computational claim can be independently evaluated. A source can have precisely documented origin and still contain a scientific mistake. This distinction preserves the independence of philosophy of science from declarations of author identity while recognizing the importance of authorship, provenance, and accountability in machine-mediated inquiry.

Limits, Uncertainty, and the Future of Reproducibility

No finite computational archive can guarantee that every future investigator will reproduce a result without adaptation. Hardware changes, license restrictions, policy changes, the disappearance of online services, and shifts in scientific concepts constrain accessibility through time. Even an exact virtual-machine image must run on some physical and organizational infrastructure. Long-term reproducibility is consequently a maintained relation between preserved evidence, interpretable documentation, and evolving technical capability. The research object packaging standard and software preservation identifiers address components of this challenge without claiming to freeze the history of computation.

The most difficult failures often arise at interfaces between domains of expertise: a scientist assumes a data field means one thing while a programmer encodes another; a statistical analyst uses an undocumented default; a reviewer recognizes a mathematical discrepancy but lacks access to its confidential input. Reproducible research becomes more robust when claims are decomposed into examinable operations and relevant specialists can inspect the transformations they know best. Transparency enables these judgments, while interpretation and testing give them content.

Future infrastructures may automatically assemble executable packages, capture machine-readable provenance, compare reruns across platforms, and generate audit reports. Their success should be assessed by whether they expose meaningful choices and improve independent scrutiny. A dashboard that verifies hashes but ignores scientific measurement will produce impressive-looking but incomplete assurance. Technological progress is valuable when it gives the community clearer reasons to believe, revise, or reject computational claims.

Conclusion

Computational reproducibility makes the computational path from specified inputs to published outputs available for independent examination. Its central achievement is the recovery, explanation, and scrutiny of transformations that otherwise remain hidden inside software, human routines, and temporary computing environments. The history of executable papers, empirical artifact evaluations, scientific computing recommendations, software citation, provenance standards, and institutional policies establishes this work as a durable requirement of evidence-based inquiry. The conceptual precision supplied by the 2019 National Academies assessment remains essential: recovery of a result using specified data and code serves a different purpose from testing a finding using newly collected data.

The research establishes six closely connected conclusions. A reproducible artifact must identify its object and intended claim; every consequential input and transformation needs sufficient provenance; execution environments are part of the evidentiary record; output agreement must be interpreted through appropriate numerical and scientific tolerances; independent checking gains meaning from explicit disclosure of its scope; and scientific validity always requires additional scrutiny of measurement, model assumptions, design, and interpretation. These conclusions synthesize established evidence and documented analytical comparisons. They are not offered as a new named theory or an empirical discovery.

The three constructed cases clarify the stakes. Reproducing an erroneous variable construction reveals the analysis without validating its meaning. Small numerical differences can preserve or reverse a conclusion depending on a predeclared decision boundary. Protected data can support independent computational audit through governed access if the limits of public inspection are reported. Together they demonstrate that reproducibility should be understood as a qualified, transparent scholarly achievement.

Computational research becomes scientifically accountable when another competent investigator can discover what was done, identify the artifacts that mattered, reconstruct the method or precisely state why this is impossible, and judge the relation between the recovered computation and the published scientific claim. That standard remains applicable across conventional programming, large-scale simulations, data science, and machine-mediated research. It is a practical expression of public scientific reason: the grounds for a claim are preserved in a form that others can inspect, contest, improve, and extend.