DPGA Blog

Navigating the Paradox of Open: New Resources for Defending Data and Content DPGs in the Age of AI

The open web is undergoing a systemic structural transformation. Today, digital public goods (DPGs) and the broader knowledge commons face a critical challenge: large-scale, automated extraction by AI crawlers and scrapers operating without reciprocity. This leads to ballooning infrastructure costs for often volunteer-run, underresourced open projects, and the erosion of trust in open knowledge, among other challenges. DPG product owners have to make delicate decisions to protect their resources while keeping them as open as possible. To help stewards of open resources navigate this complex landscape, the DPGA Secretariat is proud to share two complementary assets developed in consultation with our community:Best Practices for Preventing AI Exploitation of Open Content and Open Data DPGsWhat it is: A position paper and set of recommendations compiled directly by DPG product owners across major open initiatives like Wikipedia, Open Food Facts, Storyweaver, Govdirectory, The Turing Way, among others. Core Focus: This note describes the problems faced by DPGs firsthand and outlines practical steps for engaging with commercial AI entities to ensure data users respect licenses, robot policies, and terms of service, and contribute back to the ecosystem, while also detailing technical mechanisms to protect against AI exploitation.Navigating the Paradox of Open: The DPG Defence PlaybookWhat it is: An overview of available mechanisms to protect against AI exploitation and how they map against the current open definition, and, by extension, the DPG Standard. The playbook’s aim is to deepen understanding and conceptual clarity around the tension between maintaining open access and ensuring the survival of an open resource.Core Focus: The playbook helps understand the different layers of interventions available to DPG product owners, ultimately showing that code-level mechanisms are a first line of defence, but solving the challenge of AI exploitation requires ecosystem-level solutions at the policy, legal, and governance levels. Layers of Defence: It introduces a 6-layer "defence and reciprocity pyramid" classifying technical, legal, regulatory and institutional measures:Layer 1: Macro Environment (Legal, Regulatory, Institutional) – Market rules, statutory taxes, and licensing frameworks that operate outside website code (e.g., data excise taxes, copyleft data-to-model licenses).Layer 2: Legal and Economic – Server-level financial or authentication gates that control entry (e.g., commercial API access deals, paywalls).Layer 3: Public Governance – Non-coercive, machine-readable signs broadcasting usage preferences (e.g., robots.txt policies, machine-readable protocol signals).Layer 4: Network & Infrastructure – Systems evaluating how traffic connects to filter out blunt-force bots (e.g., rate limiting, firewall bot blocking).Layer 5: Application Security & Client Verification – Interrogating browser environments and mutating code structure in real-time (e.g., CAPTCHA, proof-of-work challenges).Layer 6: Data Integrity – Structural transformations of the data asset itself, either to poison it or isolate it (e.g., structural perturbations, data poisoning).Evolving the DPG Standard and Our Guidance At the DPGA Secretariat, we remain firmly grounded in the DPG Standard—a globally recognised foundation for openness, responsibility and community-driven innovation. However, the "paradox of openness" can inadvertently empower well-resourced actors at the expense of historically disadvantaged communities, ultimately threatening the open nature of DPGs. As the ecosystem evolves, we recognise that product owners are navigating increasingly complex challenges around sustainability, governance, and equitable participation. Moving forward, the goal is not a retreat from openness, but to further strengthen its practical implementation, supporting approaches that are resilient, reciprocal, and community-centred while remaining firmly anchored in the principles of the DPG Standard. We invite DPG product owners, legal scholars, and ecosystem partners to explore the playbook and emerging best practices. Together, we can continue to evolve the guidance provided through the DPG Wiki and related resources, ensuring the DPG Standard remains both robust and responsive to real-world challenges while preserving its core commitment to openness.Have feedback or want to add a defensive mechanism? Reach out to us at hello@digitalpublicgoods.net.