Resources, Versions, and URIs

Greg Janée > Documents > Resources, Versions, and URIs

Resources, Versions, and URIs

Summary recommendation
Introduction
Requirements
Apache implementation
Persistent identifier implementations
Examples
Further reading

Summary recommendation

Embed version indicators in URIs consistently across versions, e.g., http://host/path/file.version.suffix.
Support versionless URIs that either redirect to the current version (e.g., file.suffix ⇒ file.currentversion.suffix) or return metadata that includes information on version availability.
Redirect URIs of versions no longer extant to metadata, e.g., file.outdatedversion.suffix ⇒ metadata.html.

Maintaining persistent associations between URIs and web resources becomes increasingly challenging over time as resources get moved, renamed, and reorganized. Unfortunately, there appears to be no simple technical solution to this problem, for persistence is an outcome, a byproduct of awareness of the problem and commitment to honoring old URIs more than anything else. Nevertheless, choices about how resources are named and how URIs are assigned can make the associations more resilient to change.

For some types of resources that themselves change over time, maintenance of distinct, named resource versions and version history is necessary. Versioning is particularly important for science data, where knowledge of data versions is necessary to support provenance, and accessibility of specific versions is necessary to support reprocessing. The presence of resource versions, particularly if it is possible that older versions may cease to exist, compounds the general problem of persistence.

Two traditions, predating the Web, have emerged in the management of file-oriented science datasets such as those found in the Earth sciences: embedding version indicators in filenames, and putting files online via FTP servers. The filenames, being unique within the containing dataset and program and provider, and typically being derived from intrinsic properties of the data and metadata, persist quite well, but the locations of the files inevitably change over time. These traditions support an informal kind of provenance: given a filename and knowledge of the containing dataset and provider, it is generally possible, with some web searches and detective work, to locate the file and/or related versions and relevant metadata. Provenance tracking in this environment is thus akin to locating a scholarly article given a textual citation found at the end of a printed publication: it can be done, but not automatically or reliably.

This document gives requirements for managing versioned resources and assigning URIs to support resource persistence and persistent, online provenance. If implemented for file-oriented science data as recommended below, science data citations would resemble DOI identifiers, which have supplanted textual citations as the means of easily, unambiguously, and persistently citing scholarly literature.

Caveat: there is no universally agreed-upon definition of what constitutes a "version" of a resource, or if and when two resources represent two versions of the same conceptual resource, nor is there a standard data model for versions. As a consequence, the requirements given here can be viewed equally well as defining a model of versions.

Requirements

Given a web resource having distinct, named versions, here are the minimum requirements to support persistence and provenance:

[selectability] Each version of the resource must be independently citable. That is, each version must have a URI that identifies that version specifically, at which that version of the resource may be accessed (modulo the persistence requirement, below).
[apparency] The version of the resource must be apparent in the URI. A client must be able to determine, by observing the URI alone, which version is being cited. Furthermore, a client must be able to select a different version by replacing the version indicator in the URI with another.
[aggregation] A URI must exist that identifies the aggregation of all versions of the resource (or what might be called the resource in the abstract). Furthermore, accessing this URI must yield something: it may yield the most recent version of the resource (a form of defaulting), or it may yield resource metadata (see the metadata requirement, next). It is recommended that the resource's aggregate URI have the syntax of a version-specific URI sans version indicator.
- HTTP implementation: Defaulting to the most recent version may be transparent to clients or exposed as any kind of non-permanent redirection such as a 302 Found or 307 Temporary Redirect status code. In returning metadata, the 300 Multiple Choices status code should not be used as its intention is to support negotiation over representations and formats, not version selection.
[metadata] Metadata must exist describing what versions of the resource ever publicly existed; what they mean and what the differences are; and which versions are still accessible. This metadata may be embedded in the resource itself, or it may reside in a separate resource that represents the resource aggregation. In any case, it must be possible, given either a version-specific URI or an aggregate URI, for a client to find the metadata.
[persistence] Accessing a URI for a resource version that no longer exists must redirect to the resource's metadata. It is not permissible for a nonexistent version to redirect to the most recent version of the resource, or any other version.
- HTTP implementation: It is recommended that the redirection not be transparent to clients, but exposed as any kind of non-temporary redirection such as a 301 Moved Permanently or 404 Not Found status code (the latter response including the metadata location).

The rationale for these requirements is as follows.

Selectability: if versions cannot be independently cited and accessed, they effectively don't exist, at least not in the sense of this document.
Apparency: version apparency gives clients both awareness of multiple versions and control over the selection of the version. The ability to change versions is particularly useful when encountering a nonexistent version.
Aggregation: an aggregate URI for a resource allows the resource to be cited abstractly, i.e., in any situation in which the particular version is unimportant. Furthermore, if the aggregate URI is actionable and accesses resource metadata (either directly or, by returning the most recent version of the resource, indirectly), then clients coming across the URI (e.g., in printed literature) can come to understand the resource and ultimately access appropriate versions of it.
Metadata: the existence of metadata is basic to provenance. Without it, clients cannot make meaningful distinctions between versions, nor can they migrate to a new version from a nonexistent version.
Persistence: persistence requires either that resources always exist, or that URIs remain actionable to a limited extent when the associated resources no longer exist. Returning metadata in place of a nonexistent version allows clients to recognize and cope with the loss of a version. But any kind of automated redirection from a nonexistent version to another version, transparent or exposed, may cause clients, particularly batch processing clients, to unwittingly access an unanticipated version.

Apache implementation

The simplest way to publish web resources is to expose a filesystem hierarchy to an HTTP server such as Apache, letting the server map filesystem directories to URL path components and file suffixes to MIME types. To this basic technical approach must be added two types of URL rewriting rules: one to redirect versionless URLs to the most recent version, and another to redirect URLs of nonexistent resource versions to a metadata page.

For example, suppose we are publishing a collection of HDF files with URLs of the form http://host/path/granule.date.version.hdf, where date is an 8-digit integer that changes between granules and version is a single-digit integer. Suppose further that the current version is 5, and that versions 1-3 no longer exist. Then the following rewrite rules suffice:

RewriteEngine on RewriteBase /path RewriteRule ^(granule\.\d{8})\.hdf$ $1.5.hdf RewriteRule ^granule\.\d{8}\.[1-3]\.hdf$ \ http://otherpath/metadata.html [redirect=permanent]

The first rule adds the most recent version number (5) to a versionless URL in a manner that is entirely transparent to the client. Alternatively, the URL rewriting could be exposed as a 302 Found or 307 Temporary Redirect status code. The second rule specifies that clients requesting versions 1-3 be permanently redirected to the web resource metadata.html.

Persistent identifier implementations

The previous section described a redirection implementation in which the resource URIs were simple URLs. For greater persistence it is often desirable to draw URIs from some kind of persistent identifier scheme that employs lookups and/or redirections to achieve resilience to location changes. The ability to add version-related redirections to such URIs is dependent on the identifier system and its capabilities and restrictions.

Resolving systems that operate on identifier prefixes only, leaving suffixes intact and unchanged, provide no support for version-related redirections, but neither do they hinder them. The Name-to-Thing (N2T) Resolver is an exemplar of this type of service. It redirects naming authorities to hosts, e.g., redirecting http://n2t.info/13030/anything ⇒ http://www.cdlib.org/anything. Thus any additional redirections can be implemented by the local server or the next resolving system down the line.

Persistent Uniform Resource Locators, or PURLs, similarly resolve prefixes only; specifically, the PURL resolving system operates on the longest matching prefix in its URL database. The HTTP status code to be returned during resolution can be specified as well, and hence our version-related redirections can be directly implemented within the PURL system. Creating PURLs for a collection of files as in the example in the previous section would require that each file (and perhaps each version) be individually registered (if the version-related redirections are to be performed by the PURL system and not a downstream local server). While not necessarily a performance issue, this may prove to be a significant management burden. However, we note that a regular-expression-based registration system for PURLs (i.e., the ability to define PURLs for URI patterns) has recently been proposed.

Digital Object Identifiers, or DOIs, resolve entire identifiers only. (The Handle System, on which the DOI system is based, does define a mechanism for passing along URI suffixes, but it is rather funky, and it is not clear that the DOI system has implemented it.) Nor is it possible to specify the HTTP status code to be returned during URI resolution. As a result, it is likely to be difficult to implement our version-related redirections using DOIs.

The ARK identifier system supports the apparency and aggregation requirements by defining a syntactic marker for version indicators and by mandating implicit relationships between versioned and versionless identifiers. The Noid software, which can be used as a resolver for ARK identifiers (and other types of identifiers as well) does not, as of this writing, support version-related redirections as described above, but such redirections can be set up as part of installing Noid as an Apache external redirection service.

Examples

Managing versioned resources is a common problem. Here's how it's been handled by others.

arXiv.org, an e-print service, satisfies all of the above requirements. The most recent version of a paper is cited using a standard URI, e.g., arXiv:hep-th/9710065 (thus satisfying the aggregation requirement). A specific version is cited by appending a version indicator to the paper's base URI, as in arXiv:hep-th/9710065v2 (selectability requirement). This practice is part of arXiv's documented behavior (apparency requirement). The metadata requirement is satisfied by virtue of the fact that arXiv identifiers do not cite or access papers themselves, but rather "splash" pages that contain the paper abstract, available versions and formats, and other metadata. As far as is known, arXiv never has and never will delete papers, so the persistence requirement is moot, but it should be noted that citations to nonexistent versions redirect to the splash page of the most recent version (i.e., to the metadata).
W3C specifications are handled similarly to arXiv papers, though they're not quite as structured because they tend to take the form of collections of related hypertextual documents. Nevertheless, a specification such as HTML has an overall aggregate URI that yields the most recent version, http://www.w3.org/TR/html; aggregate URIs for numbered versions, e.g., http://www.w3.org/TR/html4 which yields the most recent version of HTML 4; and URIs for individual datestamped versions, e.g., http://www.w3.org/TR/1998/REC-html40-19980424. Hyperlinks embedded in the specifications almost, but not quite completely link the versions together. The apparency requirement is only partially satisfied because it is not possible to replace, say, the 4 in the URI for HTML 4 with a 3 to locate the HTML 3.x specification, as a client reasonably expects to be able to do. (The failure to satisfy this requirement may be due in part to the fact that it is not clear whether HTML 3 and HTML 4 are two versions of the same resource, or just two, related resources.) As with arXiv, W3C specifications will presumably never be deleted, but it should be noted that the persistence requirement is met only if all versions exist, for the resource metadata is spread out across all the versions.
Typical of many Earth science data resources, MODIS product granules have processing version numbers embedded in filenames per a standard naming convention, e.g., MOD021KM.A2009213.0000.005.2009213083936.hdf is the name of a version 5 granule produced on the 213th day of 2009. However, granules do not have persistent URIs or reside at persistent URLs, and there are no aggregate URIs. Metadata is available, but tends to be scattered across multiple documents and websites. Hence MODIS does not satisfy the requirements listed above, and as a consequence it is not possible to persistently cite MODIS granules in a way that supports online provenance.