Library APIs and Open Catalogue Data

A catalogue record is data before it is a page. That is easy to forget, because a reader only ever sees the page, but underneath is a structured description — title, creator, date, language, format, subject, identifier — in a format designed to be read by software.

Making those descriptions available as data, rather than only as pages, is what library APIs and open catalogue data are for. It is the reason a university reading list can show live availability, a phone application can tell you which branch holds a book, and a research tool can pull citations without anyone retyping them.

MARC, and why it is still here

Machine-Readable Cataloguing was developed at the Library of Congress in the 1960s, led by Henriette Avram, to let catalogue records be produced once and shared between institutions. It worked, comprehensively, and it is why libraries around the world can exchange descriptions at all — the arrangement described in WorldCat and Union Catalogues.

Its design assumptions are also sixty years old. A MARC record is a flat file of numbered fields intended to travel between library systems. It has no native notion of a web address, no straightforward way to point at another record, and a field structure that software outside libraries finds close to unreadable.

So it persists because it works and because replacing it means re-tooling every library system in the world at once, and it constrains everything built on top of it.

Metadata quality decides what is possible

Before any of the technical layer matters, the description has to be good. Consistent names, correct dates, real subject terms, identifiers that actually identify something.

This is the unglamorous foundation, and the reason library data is worth reusing at all: it was produced deliberately, by people following rules, and it says where it came from. The rules themselves are set out in How Does a Library Catalogue Work?.

A poorly described collection cannot be rescued by an API. It can only be exposed faster.

Identifiers, and why they carry the weight

Matching records by comparing text does not work reliably. Two records for the same book differ in punctuation, in how a name is written, in whether a subtitle is included. Matching by identifier does work.

So the important layer is the set of stable identifiers now in general use: ISBN and ISSN for publications, DOI for articles and datasets, ORCID for researchers, VIAF and ISNI for names of people and organisations, and Wikidata items as a general-purpose hub that connects them to each other.

Wikidata has become unexpectedly important here. It is open, it is editable, and it holds cross-references between identifier systems that no single library maintains, which makes it the practical bridge between library data and everything else on the web.

Linked data, and what it proposes

Linked data is the idea that each thing described should have its own web address, and that descriptions should point at each other using those addresses rather than repeating text.

Applied to a catalogue: the author is not the string “Woolf, Virginia, 1882-1941” repeated in ten thousand records, but one addressable entity that those records refer to. Update it once and every reference follows. Ask what else that entity is connected to and the answer comes back as data.

BIBFRAME, developed by the Library of Congress from 2011 onwards, is the main attempt to rebuild bibliographic description on that basis, and adoption is gradual and partial. The promise is real and the migration is the hard part, which is the normal shape of infrastructure change and a recurring theme in Digital Library.

What is actually available now

Several kinds of thing, at different levels of maturity.

National library open data: many national libraries publish their full bibliographic files for download under open licences, which is the largest and most reliable source of reusable library description.

Discovery APIs from library system suppliers, which let an institution’s own applications query its catalogue, availability and account data. These are normally institutional rather than public.

Public bibliographic APIs, including Open Library and the interfaces published by several national libraries, which anyone can query without an agreement.

Digital collection APIs, most importantly IIIF, which is how museums and libraries publish scanned images so that a viewer built anywhere can display material held anywhere.

What it is used for

Reading lists that show live availability. Discovery layers that search several collections at once. Citation tools that fetch a record rather than asking you to type it. Digital humanities research that treats a catalogue as a dataset — publication patterns, translation flows, what a collection reveals about what was collected. Library holdings surfaced in Wikipedia and Wikidata, which is where a great many readers actually begin.

The through-line from the previous generation is direct. Library 2.0 argued that library data should leave the library’s own website; feeds were the first crude version of that, and this is the same argument with better plumbing.

The obstacles that have not gone away

Licensing, first. Who may reuse a record created inside a cooperative, and on what terms, is unsettled and consequential.

Then capacity: publishing an API is a commitment to maintaining it, and a library that cannot staff that is right not to start.

Then consistency, since an aggregation of five collections described to four standards is a search that works badly in ways nobody can see. And privacy, which is the one that stops the discussion: circulation data would be enormously useful and is a record of what individuals read, so the default answer is no. That constraint, and its consequences for what libraries can automate, runs through Future of Knowledge.

Sources and further reading

The Library of Congress publishes the MARC standards and the BIBFRAME specification. Open Library documents a public bibliographic API. Wikidata holds the cross-identifier links described above, and IIIF publishes the image and presentation specifications used across library and museum digital collections.