Disclaimer & Assessment Methodology
This score was produced via LLM-assisted assessment of the repository structure and the supplied evidence bundle. These findings should be treated as a directional pilot signal rather than a verified rating. For more details on our pilot objectives and the development path toward rules-based human review, see the Development Roadmap. Raw files are accessible at the assessment report source.
PyPSA-Earth Assessment
"Open energy-system optimisation workflows can support transparent power/sector-coupled planning."
PyPSA-Earth shows strong open-source and reproducibility infrastructure: it is a public, version-controlled energy-system modelling repository with AGPL/CC SPDX licensing statements, citation metadata linking a software release to DOI publications, documented dependencies, Docker and conda-lock environments, multi-platform CI testing, objective-value regression checks, documentation builds, and visible contribution/community channels. Its weakest traceability area in the supplied evidence is open input-data evidence: the README states that data are automatically retrieved from open sources, but the bundle does not include the actual source catalogue, dataset provenance, licences, measurement methods, uncertainty, quality controls, or versioned input-data artifacts. Overall, the project is highly open as software and moderately-to-strongly traceable as a modelling workflow, but the supplied materials do not establish a complete end-to-end, claim-specific linkage from exact data inputs through code version, configuration, run outputs, review record, and publication.
Claim Traceability Pathway
Inspection trail mapping the connections from raw inputs to the published claim.
Traceability Health Summary
Scan diagnosis of where the evidence chain stands and what requires attention first.
Open-Source Models, Methods, and Software
This stage provides solid evidence, making calculations and documentation highly inspectable and reusable for external actors.
Open Input Data and Measurement Evidence
This is the weakest segment of the evidence chain. Gaps in records or lack of linkage break the evidence chain here.
- Unique dataset DOIs & licenses
- Scrutiny logs & replication results
- Dynamic provenance graph mapping
Strengthen Open Input Data and Measurement Evidence
Register raw dataset bundles with persistent repositories (e.g. Zenodo) and define licensing metadata.
Evidence Linkage Summary
Open Linkage is not a standalone category; it is the connectivity of the evidence chain. Below is the connection status of the connection links between each step in this project's journey.
Are data input paths and schemas programmatically defined inside the code files or configurations?
Is code execution managed through versioned workflow scripts (e.g. Snakemake pipelines) rather than manual scripts?
Are execution outputs and logs tested automatically (CI/CD) and verified inside public pull request reviews?
Are peer reviews, community issues, and discussions linked directly to published methodology reports?
Does the final published paper cite the exact dataset identifiers, model hashes, and run logs that produced the results?
There is moderate-to-strong evidence of linkage across some parts of the chain: CITATION.cff links a named paper, DOI, repository, version, and release date; the README links the model to documentation, publications, data/status resources, and community channels; workflows link tests to lock files, logs, objective values, and reference objective CSVs; and documentation snippets are explicitly derived from config.default.yaml. These are meaningful traceability links. However, the supplied evidence does not provide a complete explicit chain from a specific environmental claim to exact input datasets, input versions, preprocessing transformations, code commit or release, model configuration, workflow run, outputs, review record, and publication text. The data-source linkage is especially incomplete, and CI/test linkage does not necessarily correspond to the published claims. The score is therefore above moderate but below the high range required for an explicit, versioned, externally verifiable end-to-end evidence chain.
Detailed Dimension Inspection
Granular findings for each processing step in the Open Traceability framework.
Open Input Data and Measurement Evidence
Evidence Found
There is moderate public evidence that PyPSA-Earth is designed around open and externally retrievable input data: the README states that all data needed for simulations are automatically and flexibly retrieved from open sources, including sectoral demand, generation capacities, networks, and renewable potentials, and the contribution guide describes public dataset contributions and bundle specifications. This supports a score above the fragmentary range.
Gaps Identified
However, the supplied evidence bundle does not include the actual data-source catalogue, bundle_config.yaml, dataset licences, provenance for each source, primary-data links, measurement methods, quality-control procedures, quantified uncertainty, or versioned input-data releases. Because Open Traceability requires input origins, transformations, uncertainty, reuse conditions, and verification to be inspectable, the score is limited to the mid range rather than high.
"The README says: “All the data needed for a simulation are automatically and flexibly retrieved from open sources,” including energy demand, generation capacities, networks, and renewable potentials."
Relevance: Shows that the project intends to use identifiable open input sources, but does not itself document the exact source list, provenance, uncertainties, or versions."The guide describes contributing datasets such as cutouts and hourly electricity demand time series, uploaded to public storage such as Zenodo and specified in configs/bundle_config.yaml."
Relevance: Indicates a mechanism for public data contribution and dataset registration, but the referenced bundle configuration and dataset metadata are not included in the evidence bundle."The README notes that custom datasets can be integrated and kept private if required."
Relevance: Demonstrates flexibility, but private/custom datasets could weaken external verifiability unless separately documented for a given claim.Open-Source Models, Methods, and Software
Evidence Found
The supplied evidence gives strong public support for open-source software traceability. The project is hosted in a public GitHub repository, describes itself as an open optimisation model, includes SPDX licensing headers and an AGPL-3.0-or-later citation/README licence statement, provides dependency specifications in environment.yaml and documentation requirements, and includes Docker infrastructure for building a reproducible software environment. CI workflows also show linting, type checking, CodeQL, and documentation builds. The score is not higher because the GitHub metadata reports license as null, the supplied evidence does not include a full LICENSE file or complete source tree, and some execution infrastructure may depend on external services or solvers such as Gurobi. Still, the model and software evidence is strong, version-controlled, inspectable, and reusable.
Gaps Identified
No significant gaps identified. This stage is fully documented and verifiable.
"Repository description: “PyPSA-Earth: A flexible Python-based open optimisation model to study energy system futures around the world”; default branch is main and repository is public in the evidence bundle."
Relevance: Establishes that the model software is publicly hosted and version-controlled."README includes SPDX-License-Identifier: AGPL-3.0-or-later, a License: AGPL v3 badge, and describes PyPSA-Earth as an open-source global energy system model."
Relevance: Supports open-source licensing and public inspectability of model code and methods."CITATION.cff lists repository-code as https://github.com/pypsa-meets-earth/pypsa-earth, version 1.0.0, date-released 2023-04-18, and SPDX-License-Identifier: AGPL-3.0-or-later."
Relevance: Provides citable, versioned software metadata connecting the model to a release and publication."environment.yaml specifies the pypsa-earth environment, channels, Python version, PyPSA, atlite, powerplantmatching, earth-osm, Snakemake, solvers, geospatial, plotting, and other dependencies."
Relevance: Documents dependencies and configuration needed to understand and run the model."Dockerfile builds from condaforge/mambaforge, creates the pypsa-earth environment from envs/linux-64.lock.yaml, installs OpenJDK, and activates the environment."
Relevance: Provides open infrastructure for packaging and running the software environment.Open Execution and Reproducibility
Evidence Found
There is strong evidence for reproducible execution infrastructure: the project uses Snakemake-related dependencies, locked environments for multiple platforms, a Dockerfile, multi-platform CI tests, artifacts containing logs/results, objective extraction from solver logs, reference objective comparison, and workflows to update locked environments and reference objectives. This makes the computational path more inspectable than simply releasing code. The score is not higher because the supplied evidence does not show the exact published-paper run, exact input data versions, complete scenario parameters, archived outputs, or a re-executable workflow record linking a specific environmental claim to specific data and code versions. CI artifacts also have short retention in the evidence shown. Therefore reproducibility is strong at the software/test-workflow level
Gaps Identified
but incomplete for exact claim reproduction.
"The test workflow runs on ubuntu, macos, and windows, uses platform lock files, executes make test, uploads logs/results artifacts, and extracts objective function values from solve_network logs."
Relevance: Shows automated execution checks and inspection of computational outputs across platforms."Workflow generates conda-lock files for linux-64, osx-64, win-64, and osx-arm64 and creates a pull request with updated locked environments."
Relevance: Supports reproducible computational environments through generated platform-specific lock files."Manual workflow runs tests, extracts objective values from logs, writes updated_obj_ref.csv, and creates a pull request updating test/utils/obj_ref.csv."
Relevance: Shows that numerical model outputs are checked against reference objective values, improving execution traceability."Utility reads test/utils/obj_ref.csv and returns reference objective values by folder and log-file name."
Relevance: Provides concrete evidence of output comparison infrastructure for reproducibility tests."Docker image creation uses envs/linux-64.lock.yaml to create and activate a pypsa-earth environment."
Relevance: Helps external users recreate the software environment for execution.Open Community and Review
Evidence Found
The project has strong public evidence of open community processes: a public GitHub repository with hundreds of open issues, a contri
Gaps Identified
bution guide inviting questions, feedback, bugs, features, documentation, regional expertise, and datasets, CI checks on pull requests, Discord/community links, and peer-reviewed Applied Energy publications referenced in project metadata. These features make scrutiny and contri
"Repository metadata reports open_issues_count: 298."
Relevance: Shows a public issue-tracking channel with visible ongoing discussion or problem reporting, though issue contents were not supplied."The guide invites users to submit questions and feedback, propose features, report bugs, submit fixes, improve documentation, share regional modelling insights, and contribute datasets."
Relevance: Demonstrates open channels for critique, improvement, and community participation."README includes a Discord badge/link and says the initiative works on open data, open source modelling, open source solver support, and open communities."
Relevance: Supports the existence of public community interaction channels."Pull-request workflow runs pre-commit checks, docstring coverage, and mypy type checking, with summaries posted to GitHub step summaries."
Relevance: Shows automated review gates that contribute to visible quality-control processes for contributions."README cites Applied Energy papers for the power model and sector-coupled model with DOIs 10.1016/j.apenergy.2023.121096 and 10.1016/j.apenergy.2025.125316."
Relevance: Indicates that at least some model claims are associated with formal journal publication, although peer-review reports are not provided.Open Publications and Communication
Evidence Found
The project communicates its claims and methods through a public README, ReadTheDocs documentation link, citation metadata, DOI-linked Applied Energy publications, documentation CI, and generated configuration documentation snippets that are intended to stay synchronized with config.default.yaml. The README clearly states the model purpose, scope, sectors, and intended use, and CITATION.cff provides citation metadata for a specific release and paper. This merits a strong score. The score is not higher because the supplied evidence does not show the full documentation pages, publication accessibility/licensing terms, version history for dynamic outputs, detailed update cycles, or all supporting artifacts cited from the publications. Open communication is strong,
Gaps Identified
but complete publication traceability for each claim is not fully established in the bundle.
"README describes PyPSA-Earth as a global sector-coupled open-source multi-energy system model and explains its scope across electricity, heating, transport, industry, hydrogen, and more."
Relevance: Provides accessible public explanation of the project’s environmental/energy-system modelling claims and scope."README lists academic publications for the power model and sector-coupled model, including DOI links."
Relevance: Connects project communication to formal publications supporting the model."CITATION.cff gives title, abstract, authors, ORCIDs for several authors, DOI 10.1016/j.apenergy.2023.121096, repository-code URL, version 1.0.0, and release date."
Relevance: Provides standardized citation and publication metadata that improves publication traceability."Documentation stack includes mkdocs, mkdocs-material, mkdocstrings, mkdocs-git-revision-date-localized-plugin, and related packages."
Relevance: Shows tooling for generating public documentation, including revision-date support."Docs CI builds documentation with mkdocs build --strict and runs spelling checks on pull requests affecting documentation files."
Relevance: Supports quality control for public documentation."Snippet README explains that YAML snippets are extracted from config.default.yaml so documentation pages stay in sync with the actual config file."
Relevance: Shows an effort to keep explanatory materials aligned with model configuration.Open Linkage
Evidence Found
There is moderate-to-strong evidence of linkage across some parts of the chain: CITATION.cff links a named paper, DOI, repository, version, and release date; the README links the model to documentation, publications, data/status resources, and community channels; workflows link tests to lock files, logs, objective values, and reference objective CSVs; and documentation snippets are explicitly derived from config.default.yaml. These are meaningful traceability links.
Gaps Identified
However, the supplied evidence does not provide a complete explicit chain from a specific environmental claim to exact input datasets, input versions, preprocessing transformations, code commit or release, model configuration, workflow run, outputs, review record, and publication text. The data-source linkage is especially incomplete, and CI/test linkage does not necessarily correspond to the published claims. The score is therefore above moderate but below the high range required for an explicit, versioned, externally verifiable end-to-end evidence chain.
"CITATION.cff links the repository code to the Applied Energy paper DOI, software version 1.0.0, and release date 2023-04-18."
Relevance: Provides a specific link between publication, repository, and software version."README links to ReadTheDocs, DOI publications, PyPSA-Earth-Status, Google Drive, Discord, and the project repository context."
Relevance: Shows public cross-linking among communication, documentation, community, and supporting resources."The test workflow extracts objective function values from solver log files and compares or reports them with reference objective values."
Relevance: Links execution artifacts, logs, and numerical outputs for reproducibility testing."extract_obj_for_path maps a run folder and log file to an objective value in test/utils/obj_ref.csv."
Relevance: Provides a concrete linkage mechanism between model-run artifacts and reference outputs."Documentation snippets are extracted from config.default.yaml and version-controlled to keep configuration examples synchronized with the actual config file."
Relevance: Links documentation to configuration artifacts, improving traceability of methods communication."Workflow generates platform-specific lock files from envs/environment.yaml and creates pull requests for updates."
Relevance: Links dependency specifications to versioned reproducible environments, though not directly to a specific published claim.Critical Traceability Gaps
Inspection details of where the evidence chain breaks down and what that implies for public trust.
| Identified Gap | Impact on Traceability | Suggested Remediation |
|---|---|---|
| Dataset not linked or versioned | High ‐ Blocks inspection of inputs | Register raw datasets in Zenodo and cite unique DOIs in config files. |
| Review and replication logs hidden | Medium ‐ Hides validation gate process | Maintain an open-source technical review tracker in public channels. |
Remediation & Improvements Roadmap
Actionable roadmap prioritized by implementation complexity.
Quick Wins
Low effort, high immediate return.
- Add CITATION.cff file
Enables clear publication-to-repository citation schema.
- Commit top-level LICENSE
Resolves legal terms of software and dataset inspectability.
- Link communication channels
Add status/discord badges to repository README files.
Medium Effort
Standard engineering practices.
- Specify configurations in config files
Provide explicit parameters and dataset configurations in repository.
- Lock computational environments
Lock dependencies using Conda-lock or Docker image digests.
- Archive automated workflow logs
Publish CI test summaries and solver objective check values.
Strategic Upgrades
Highest traceability return.
- Publish workflow runs on Zenodo
Create persistent, immutable archives of the full execution outputs.
- Build claims-to-evidence provenance graphs
Establish a machine-readable map of the end-to-end evidence trail.
Framework Dimension Definitions
- Open Input Data and Measurement Evidence: Identifiable, documented, attributable, versioned raw measurement evidence inputs.
- Open-Source Models, Methods, and Software: Inspectability of Analytical logic via code repositories, configurations, and license files.
- Open Execution and Reproducibility: Computational environments, workflows, run logs, and outputs make rerun processes inspectable.
- Open Community and Review: Scrutiny, peer review, issue tracking, and responses to replication challenges are visible.
- Open Publications and Communication: Resulting reports, papers, and dashboards are openly accessible and reference source materials.
- Open Linkage: The versioned, explicit connectivity linking data, models, execution, review, and publication steps.
Human Review Notes
Human review of this assessment report is pending. An independent reviewer must inspect the repository references to verify the findings.