Earth Observation Data Cubes as a Service

Stefan Brand
earth-observation data-cubes cloud-processing

Editor’s Note: The information in this blog post is part of Stefan’s master’s thesis (opens new window), that he wrote as an intern at EOX.

Have you wondered how to extract meaningful information from loads of satellite imagery spanning several months? You are not alone! Satellite imagery is considered Big Earth Data and comes with a specific set of challenges. Earth Observation Data Cubes have set out to tackle these challenges. They allow for easy information extraction, for example in the agricultural domain.

# Checks by Monitoring

In an effort to modernize the Common Agricultural Policy (CAP) (opens new window) the European Union is introducing a new concept called Checks by Monitoring (CbM). The idea is to automatically check all farmers’ subsidy requests using satellite imagery. To this end time stacks of satellite data are overlain by agricultural parcels and typical vegetation "signals" are modeled for each crop.

Typical vegetation index time series of autumn barley and sunflower deduced from satellite observations. Source: Koetz et al., 2019, p. 20, cropped

Then the actual signals of all agricultural parcels are compared to the typical signal models corresponding to the farmers’ declarations. If the declaration and the actual signal agree, the parcel is flagged as compliant. Else, further investigations are triggered, and the farmer is given the chance to review their declaration.

Working on putting the CbM concept into practice EOX has partnered with the Austrian paying agency Agrarmarkt Austria (AMA). Together we are exploring the various steps of the CbM workflow in a hands-on fashion.

# Earth Observation Data Cube

At the beginning of the CbM workflow satellite imagery has to be pre-processed into a form that is more suitable for time series analysis. Earth Observation Data Cubes are an emerging paradigm in the remote sensing realm. Over an area of interest (AOI) images are mosaicked together and then these mosaics are stacked on top of each other. This way we receive a continuous, gridded data structure with at least three dimensions (time, x, y).

Workflow from image files to a pixel-aligned image cube. Source: Kopp et al., 2019, p. 8 cropped

In the context of CbM we are using Sentinel-1 backscatter plus derived polarisation ratios and multiple bands of Sentinel-2 plus derived vegetation, water and salinity indices. Satellite data is processed using the EOX-developed mapchete Hub cloud service. A Sentinel-2 data cube schema that includes half-monthly data from the start of the winter season up to the first checks date is detailed below.

Schema for a four-dimensional (time, x, y, variable) Sentinel-2 Earth observation data cube

# Tiles Processing

Since the project AOI covers about 17,000 km² we tiled the input data into smaller chunks to speed-up the processing. We created a custom WMTS tile pyramid and selected six process zones corresponding to the project AOI. These zones were in turn chunked again into smaller process tiles. While process zones are used to parallelize processing on multiple machines, process tiles are distributed among the available processor cores on each machine.

Process zones (red) and process tiles (pink) covering the 17,000 km² CbM project AOI. Basemap: OpenStreetMap { Data © OpenStreetMap contributors, Rendering © MapServer and EOX }

This setup allowed us to process 8.5 months of Sentinel-2 satellite imagery within 1:24 hours, including calculations of half-monthly averaged time slices and band indices. The resulting data cube with 17 time slices covering 17,000 km² is 48.37 GiB in size. Find the used processing script on GitHub (opens new window).

# Conclusion

The Earth observation data cube experiment presented above shows that EOX is perfectly equipped to deal with the satellite data pre-processing requirements for the upcoming CAP reform. Contact us and we will be glad to discuss your custom processing needs together.

# References

Sentinel-2 cloudless 2020 Cloud Workspaces for Earth Observation data processing - from Ideas to Solutions