Embracing openness in delivering an API for digital public goods
Since day one, the Digital Public Goods Alliance (DPGA), has embraced open source values including transparency, openness, and community. The DPGA seeks to advance open source software, open data, open AI models, open standards and open content as solutions to create a more equitable world; and, simply put, we couldn’t do it any other way than the open source way. Today, we are excited to announce the release of the Digital Public Goods API. This article recounts the journey of how we got here.
One of the first actions we embarked on was to build a registry of digital public goods to aid their discovery. Doing so was partly a landscaping effort to know what projects already exist that are working towards a more just world, and partly a community-building effort, welcoming input by a broad set of interested parties and contributors.
Architecturally, the industry-standard would have been to create a database, build a well-polished front-end website, and offer an open API that welcomed others to contribute. However, that approach was neither quite right, nor good enough. This is in part because whoever runs the database would ultimately own the data, as it would inevitably be hosted on closed infrastructure (insert here any cloud hosting provider or in-premises computing). This approach is simply not transparent enough. As a technical co-lead for the DPGA, I knew we could do better.
Enter the beauty and simplicity of a collection of text files hosted on a version control system. The format of text files was JSON, a lightweight data-interchange format. This makes it easy for humans to read and write. It is also easy for machines to parse and generate. You have the best of both worlds. 🙌
Due to its popularity and common use throughout the global developer community, the version control system of choice was git with hosting on GitHub. While we didn’t invest much initially on the “well-polished website” aspect of our infrastructure, we reaped the benefits from using GitHub early on. On GitHub, any code repository comes by default with the functionality of issues and pull requests, providing a transparent communication channel to discuss potential changes with collaborators. Plus, all the data is easily accessible to read, inspect and modify. That was indeed embracing the open source way.
If you wonder about data consistency, conformance, and integrity, you may be concerned at the thought of managing a large dataset with plain text files. So was I. That’s where having a data schema in the form of a JSON schema ensures data validation and checks all of the above dimensions of data quality. Pair such a JSON schema with a continuous integration (CI) pipeline to automate the data workflow, and the result is a very strong and fully transparent backend.
As all this data is hosted on a public code repository, we postponed the offering of a dedicated API, since the data is already publicly available on the DPG repository. Also, GitHub already offers their own API, providing endpoints to access the raw contents of what is available through the website. Yet, we have been longing to complement our offerings with a data API that could enhance our collaborative efforts and make it easier for others to build on this growing dataset. The beauty of storing all our data in JSON is that it is the same format that REST APIs use. Thus, little work was required to repackage this data and serve it through an API. In a clever twist of repurposing the infrastructure that GitHub provides for free through their GitHub Pages (initially conceived as a hosting for static websites—primarily documentation sites), one can pre-generate all possible API endpoints, create the underlying folder structure to host the content for all these endpoints as JSON files, and have a very robust hosting for an API that leverages GitHub’s massive Content Delivery Network (CDN) to ensure high availability and low latency all over the world.
This brings us to this week, when the DPGA releases its API, with its accompanying public repository. We welcome your integrations while continuing to build together on this dataset for a more equitable world.
Are you a software developer, product manager, UI/UX designer, or community manager that enjoys a creative exploration of innovation and technology while contributing to digital public goods for a more equitable world, including the benefit of children worldwide? Stay tuned because DPGA Co-Host UNICEF’s Office of Innovation will be opening positions in the coming weeks, keep an eye on their Twitter and LinkedIn feeds for more information.
Follow our blog, or join our mailing list.
Learn more about the Digital Public Goods Alliance on our website.
Nominate digital public goods through this form.
Cover Image: “Embracing openness at Bryce Canyon National Park in winter” by Victor Grau Serrat is licensed under CC-BY-SA 4.0.