OGC® EO Data Access Best Practice

Suggested additions, changes, and comments on this draft document are welcome and encouraged. Such suggestions may be submitted by email message to the document editor or contributors named above, by creating an issue or a pull request at the GitHub repository, or by making suggested changes in an edited copy of this document.

Presently, this document is a living document until officially submitted to OGC seeking formal adoption. Below is a list of items that are currently worked on or that could be worked on in future activities.

digital representation of a spatio-temporally varying phenomenon as defined in

→ EO Coverage

collection of → EO Coverages

Rectified Grid → Coverage or Referenceable Grid → Coverage having an → EO Metadata record and a WGS84 footprint

→ EO Coverage’s metadata record

→ EO Coverage composed from subsets of one or more co-referenced → Datasets

An EO Product contains one or more related → EO Product Datasets plus metadata and optionally auxiliary data like → EO Product Quicklooks.

One or more files each containing one or more → EO Coverages.

A visual representation of a usually reduced → EO Product Dataset encoded in an image format. The → EO Product Dataset may combine different bands.

Data structure documenting an operation that has been applied to the → Coverage it is part of

contains, in its → EO Metadata element as defined in [OGC 10-157r4], the → EO Metadata element of

Unified Modeling Language (UML) static structure diagrams appearing in this specification are used as described in Subclause 5.2 of OGC Web Services Common [OGC 06-121r9].

The UML model data dictionary is specified herein in a series of tables. The contents of the columns in these tables are described in Subclause 5.5 of [OGC 06-121r9]. The contents of these data dictionary tables are normative, including any table footnotes.

The following namespaces are used in this document. The prefix abbreviations used constitute conventions used here, but are not normative. The namespaces to which the prefixes refer are normative, however.

When multiple representations of the same information are given in a specification document these are consistent. Should this not be the case then this is considered an error, and the XML Schema shall take precedence.

The WCS 2.0 Earth Observation Application Profile (EO-WCS) [OGC 10-140r2] has been adopted by OGC defining a profile of WCS 2.0 for use on Earth Observation data. Naturally the present document focuses on EO-WCS but makes sure not to forget the surroundings to put it into context.

EO-WCS adds some concepts relevant for the EO domain on one side and some constraints on the other to the generic WCS:

One of the motivations to introduce the concept of Dataset Series is to limit the size of capabilities responses. They only need to report the identifiers of Dataset Series instead of summaries for all coverages available. Metadata for individual coverages is obtained using the new DescribeEOCoverageSet operation. This operation supports basic spatial and temporal searching as well as paging and thus stepping through the available data. Figure 1 shows the sequence to use EO-WCS.

Before downloading data it is typically desired or at least useful to view the data and metadata. The OGC service standardized for data viewing is the Web Map Service (WMS). For the EO domain the Web Map Services - Profile for EO Products (EO-WMS) [07-063r1] has been designed. This WMS profile allows to browse through collections using the time parameter on one layer. Provided that the same data is available from the EO-WCS Dataset Series and the EO-WMS collection layer a combined exploitation of EO-WCS and EO-WMS as shown in Figure 2 is feasible.

Of course the WMS in this scenario can easily be replaced by the Web Map Tile Service (WMTS) specifically designed to improve the performance of WMS.

Further OGC standardized services to take into account are the Catalog Service (CSW, OpenSearch) for discovery, the Download Service for Earth Observation Products Best Practice (DS-EO) [13-043] as well as the Web Processing Service (WPS).

Centered around EO-WCS this Best Practice document defines conventions and makes recommendations for the following topics:

In a number of scenarios and use cases it is necessary to be able to link from one service to another, holding e.g. different views on the same data. For example a user might first be presented with an RGB view on some data via WMS before downloading the actual data, that is, full bands via WCS. The question is, how is a client supposed to construct WCS requests for the same product just seen via WMS? Currently there is no standard way defined how to provide such links.

We propose to address the cross service interaction item by giving recommendations on how to structure the data offered in the various services as well as how to explicitly link between them. These explicit link recommendations include these services: OpenSearch, WMS, WMTS, WCS, and DS-EO.

An example for a data structuring recommendation is that for a collection there should be an EO-WMS layer and an EO-WCS DatasetSeries provided both reusing the ID of the collection. Thus clients know that a product viewed via a GetMap request with TIME parameter can be downloaded via a GetEOCoverageSet request using the same TIME parameter value.

First explicit linking recommendations are given in the "EVO-ODAS Recommendation on building a discovery interface using OpenSearch Technical Note (EVOODAS-TN-GEO-4112 version 1.3.0 from 2017-10-24)" by GeoSolutions. The recommendation there is to use OWC ATOM offering elements linking to the EO Product in the respective service. Note that it also includes recommendations on how to structure these links in order to link to a single product. In short the recommendation is to link to a tailored Capabilities document like e.g., http://u.rl?service=WMS&acceptVersions=1.3.0&request=GetCapabilities&layers=S2. This covers all links from OpenSearch to any other service.

In the same way recommendations will be given from (EO-)WMS, WMTS, (EO-)WCS, and DS-EO to the respective four other services using for example wms:DataURL, in the layer specification of the WMS Capabilities linking to the corresponding EO-WCS DatasetSeries.

TODO

The rangeType component of a coverage describes the common data type all this coverage’s range values (such as pixels) share. In programming languages, this typically consists of differentiating between signed integer, unsigned integer, float, etc.

The rangeType concept, however, goes far beyond such classical data typing; adopting OGC O&M definitions, a comprehensive semantics description can be expressed indicating the range value type (which can be a classical data type such as "non-negative integer", but can also denote application semantics such as "radiation", "SAR") given by a URL, a definition of the unit of measure of the values, null values, and several more can be expressed.

In the basic, domain-agnostic OGC Coverage Implementation Schema (CIS, formerly known as GML Application Schema, GMLCOV) only these general items are defined. EO-WCS 1.1 extends this range type description, as it is used in WCS 2, with EO-specific detail information.

The extension includes elements to specify the measured physical properties (wcseo:dataSemantics), the data types of stored numbers (wcseo:dataType), the conversion from stored numbers to physical properties (wcseo:dataType2dataSemantics), as well as a hint for how to generate a RGB version (wcseo:RGBgenerationHint).

The additional range type information is provided via the wcseo:rangeTypeExtension element which is either included once for the whole range type under the swe:DataRecord element or separately for each channel, often referred to as band, under each swe:DataRecord/swe:field/swe:Quantity element. It may also be included in both locations for example when there is one common RGB generation hint but the data conversion is specific for each band.

The new elements are introduced one by one in the following sections and extensive examples are given below.

The wcseo:rangeTypeExtension element first includes the wcseo:dataSemantics element of type anyURI. This element holds an URI preferably resolving to a description of the observed physical property like http://sweet.jpl.nasa.gov/2.3/stateSpectralBand.owl#Visible.

This element needs to be synchronized with the definition attribute of each swe:Quantity element as well as the unit of measure defined via the code attribute of the swe:uom element again of each swe:Quantity element.

XML instance examples included with the OGC schemas make use of http://www.opengis.net/def/property/OGC/0/Radiance for the definition attribute which doesn’t resolve to something useful as expected. Another URI used in OGC examples is http://sweet.jpl.nasa.gov/2.0/physRadiation.owl#Radiance. The latest version of this at the time of writing is http://sweet.jpl.nasa.gov/2.3/propEnergyFlux.owl#Radiance.

It is suspected that the ESA funded projects RARE, SMAAD, OBEOS, and/or PRODTREES define URIs to describe physical properties as well. However, a web research didn’t bring up anything useful in this direction. Thus, for the time being, the examples given use the SWEET ontologies defined by the NASA Jet Propulsion Laboratory (http://sweet.jpl.nasa.gov).

An example for a unit of measure code is W.m-2.sr-1 as defined by http://sweet.jpl.nasa.gov/2.3/reprSciUnits.owl#wattPerMeterSquaredPerSteradian for radiance as used above.

SWE Common mandates the usage of units as defined by http://aurora.regenstrief.org/UCUM. However, this server is not accessible anymore and seems to be moved to http://unitsofmeasure.org/ucum.html.

Another physical property example is spectral radiance with URI http://sweet.jpl.nasa.gov/2.3/propEnergyFlux.owl#SpectralRadiance and unit of measure code W.m-2.sr-1.nm-1 as defined by http://sweet.jpl.nasa.gov/2.3/reprSciUnits.owl#wattPerMeterSquaredPerSteradianPerWavelength.

The wcseo:rangeTypeExtension element further includes the wcseo:dataType element, again of type anyURI. This element again holds an URI preferably resolving to a description of the data type. Examples of such URIs are http://www.opengis.net/def/dataType/OGC/1.1/nonNegativeInteger, http://www.opengis.net/def/dataType/OGC/0/unsignedInt, or http://www.opengis.net/def/property/netcdf/1.0/unsignedShort.

The data type is also implicitly provided via the actual coverage encoding. However, to describe it explicitly in the wcseo:rangeTypeExtension element allows clients to retrieve it also in coverage descriptions and without need to understand and parse the actual coverage encoding format.

In order to be able to convert the stored numbers to the value of the actual measured physical property the wcseo:dataType2dataSemantics element is added to the wcseo:rangeTypeExtension. It describes the conversion via two real number intervals and a type.

wcseo:intervalFrom gives the interval of values stored in the coverage, wcseo:intervalTo specifies the interval the stored values are converted to, and wcseo:type defines which conversion method to use. Both intervals are given via two real numbers and the type via anyURI.

The example below describes a linear transformation, as typically used for optical data, from \$[1,4095\$] to \$[390.0000,780.0000\$] i.e. for a value \$x\$ between \$1\$ and \$4095\$ the actual measured value \$y\$ is calculated as: \$y = 390 + (x-1) * (780-390) / (4095-1)\$

Another example, given below, describes the inverse to a logarithmic transformation as for example sometimes used for radar data. The transformation of stored values \$x\$ in the interval \$[1,65535\$] to observed values \$y\$ in the interval \$[2,1000000000\$] is given by \$y = 2 * e^(((x-1)*(ln(1000000000)-ln(2)))/(65535-1))\$.

The last element in the wcseo:rangeTypeExtension element is the wcseo:RGBgenerationHint element. It is meant to provide a hint for clients wanting to visualize the data. It includes the elements wcseo:bandSequence, wcseo:intervalFrom, wcseo:intervalTo, and wcseo:type. The first is a list of three band names or band arithmetic instructions delimited by spaces used for the three bands to generate the RGB version. The names used shall be equal to name attributes of the respective swe:field element. The other three elements are comparable to the ones used in the data conversion above.

The example below describes the RGB generation from a single band product by reusing the single band three times and logarithmically stretching the interval \$[100,10000000\$] to \$[1,255\$] i.e. value \$x\$ is converted to \$y\$ using \$y = ((ln(x)-ln(100))*(255-1))/(ln(10000000)-ln(100))+1\$.

This section details our recommendations for the most commonly used data as well as for data not covered here. Of course data providers are free to choose any definitions, it’s just highly recommended to use resolvable URIs providing meaningful descriptions ideally machine as well as human readable.

The following provides an example gmlcov:rangeType element including additional range type information for RGB generation on swe:DataRecord level as well as data conversion information on swe:Quantity level.

The following is an example of a multispectral range type.

WCS originally was perceived to give access to single coverages taken from a flat set of coverage offerings on the server.

This approach was based on the classical subdivision into metadata services (i.e., catalogs) and data services, connected through some loose coupling (which effectively was never standardized beyond the recommendation to use URLs).

With the progress of requirements and technology it is more and more considered important to merge data and metadata perspectives with the vision of a common integrated information space.

For WCS, this means in particular to further structure a server’s offering of coverages along various criteria.

Early on EO-WCS has provided a lead in this by introducing both uniform and non-uniform coverage groupings together with dedicated search functionality which is able to perform a focused search within coverage subsets.

With OGC Coverage Implementation Schema (CIS) 1.1 comes massively enhanced support for structuring coverages. Therefore, a harmonization task has to be accomplished to relate both concepts, which may well lead to an extension of functionality for fully exploiting all benefits. One standardization opportunity might be to generalize EO groupings, phase in CIS 1.1 concepts, and establish a general WCS grouping extension.

A separate issue is how far recursion should be allowed: can a coverage grouping be part of another coverage grouping, with unlimited nesting? While the generality may be appealing it might pose severe extra complexity on implementations.

In the following subsections, we discuss EO-relevant grouping concepts, based on CIS and EO-WCS.

First, we address the special case of grouping coverages that share some critical common properties. Next, we generalize this to arbitrary groupings of coverages. Finally, we discuss registration issues.

So far, access to coverages and groupings has been discussed. However, with WCS-T Web-based maintenance of coverage offerings is possible, including insertion, update, and deletion of coverages. This functionality might get extended with maintenance operations on coverage collections.

Essentially, this would include creation and removal of collections, inclusion of coverages into an existing collection as well as excluding selected coverages, and maybe more.

Relevant existing approaches should be considered appropriately, such as the GeoServer REST API 2.0.

TODO

OpenSearch is the designated EVO-ODAS endorsed search service. Thus we propose to extend OpenSearch in a way to allow clients to specify the verbosity of the answer.

Below we draft the usage of a new parameter named view. Alternatively the suitability of the OpenSearch SRU (Search and Retrieval via URL) extension should be investigated.

Several allowed values are defined in that proposal ranging from full for everything to geotime for a very limited view only including id, name, bbox, start, and end.

Additionally, there is the idea of adding histogram like functionality i.e., requesting summary information for defined buckets like months.

It should be noted that there exists an alternative approach to searching which is based on XPath enabling traversal and subsetting of a server’s XML information hierarchy.

Although XPath looks like a promising approach there are some difficulties particularly from an implementation point of view. In general, an XML document can only be subsetted using XPath once it has been generated. This approach wouldn’t scale well on server implementations. Of course particular queries can be computed on the fly but to allow generic XPath queries is quite challenging.

Another issue is, that XPath returns subsets of XML documents without maintaining overlying hierarchy. While it is simple to retrieve a list of coverage IDs for certain search criteria it is difficult if not impossible to retrieve for the same list of coverages tailored coverage descriptions only containing ID, phenomenon time, and footprint.

Further challenges arise when the query is supposed to contain time or spatial subsetting.

OpenSearch and XPath are not competing, but complementary approaches: OpenSearch empowers users to do human-centric search through keywords. XPath, conversely, enables fine-grain drilling into the server’s information structure and, hence, is suitable in particular for machine-to-machine communication.

Both search types, therefore, are important for versatile coverage services. In EVO-ODAS, the OpenSearch avenue will be elaborated.

TODO

We propose to extend the GetCoverage request with parameters to request a masking via a mask coverage or polygons e.g. given as GeoJSON or shapefile. The coverage of polygons can be given by reference or, if small enough, included directly in the request in case of polygons as WKT.

The response shall use a defined NoData value for areas outside of the mask.

This will be described first in the EO Data Access Best Practice at hand. Later on this might be promoted to an actual WCS extension.

TODO