Sentinel-2 cloudless 2021

2022-05-18

This year we challenged ourselves by overhauling our processing infrastructure in order to give us more room for improvements in the future. This caused a delay of our usual release interval, but we think it was well worth the effort! With a bit of sweat and blood we can finally announce the latest release of our annual EOxCloudless (opens new window) satellite map Sentinel-2 cloudless (opens new window).

# Layer comparison from 2016 to 2021

Apart from major technical improvements of the data foundation the website Sentinel-2 cloudless (opens new window) has got a new coat as well. You can now browse all years of our global satellite layers by clicking the clock symbol top left. The latest visual change of the map is darker color rendering over the Sahara, Northern Africa, Near East, and Arabian Peninsula.

Animation of the global evolution from 2016 to 2021

Can you help us to find more interesting areas that changed over the past years in our Sentinel-2 cloudless (opens new window)? Feel free to share with us any beautiful examples on Twitter (opens new window).

# Processing development

This technology chapter tries to illustrate how we have re-implemented our whole processing deployment infrastructure. We are using an in-house open source solution for geoprocessing (mapchete (opens new window)) since the first iteration in 2016. However, in scaling it up to be able to process the global dataset on a large computing cluster always required some additional technologies.

Up until this campaign Celery (opens new window) and RabbitMQ (opens new window) were used to distribute jobs over a manually managed set of compute instances. Lacking some desired features we have reevaluated the deployment procedure to run on a Kubernetes (opens new window) cluster on AWS. This allowed us to make use of autoscaling and took the burden of managing instances from the operator's shoulders.

To handle the distributed computing part Celery was replaced by dask (opens new window) and dask-distributed (opens new window) mainly because of the simpler API as well as the large momentum dask gained in recent years. Also, it supports more complex computation graphs but this topic is worth a separate blog post.

Dask also allowed us to make use of other packages in the ecosystem such as dask-gateway (opens new window) and dask-kubernetes (opens new window) which can handle requesting, provisioning, and down-scaling a compute cluster on demand.

To round it up the exposed REST API was built using FastAPI (opens new window) which we use in other projects in EOX as well.

mapchete Hub architecture for the 2021 campaign

We now have a full processing ecosystem in our hands which can be triggered by just interacting with the exposed REST interface. To facilitate usage we developed a command line utility (mhub) which allows job submission and cancellation as well as some additional productivity tools such as showing the progress of a running job and querying jobs by their metadata. The internal metadata representation is based on GeoJSON so all jobs can be dumped and visualized in a GIS if need be.

Once a job is submitted, mapchete determines the processing chunks and requests a dask cluster with certain specifications to dask-gateway. The gateway then passes on the request to Kubernetes and if there are resources available (e.g. AWS spot instances), Kubernetes will start nodes and the desired dask scheduler and worker pods which we refer to as dask cluster. As soon as the cluster is up and running, mapchete will submit its tasks and observe the job progress while keeping the job metadata database up to date.

These developments now bring us one step further to achieving full automation where a full process can be triggered by an event (e.g. a custom mosaic request) rather than an operator triggering and observing the process. Also this flexibility allows us to use the platform in other projects.

# EO-Guesser

Have you heard of GeoGuesser and are a fan of satellite imagery or the world from space? We have developed a neat new game and at EOX we are playing it with great enthusiasm since weeks.

If you want to try your luck or want to put your Earth Observation knowledge to test you have to try our EO-Guesser (opens new window). Being zoomed in on a random place on Earth just able to see the satellite map, can you guess where you are? How well do you know our planet?

# Beyond True Color

Our data can be used to generate incredible map and visualisation products e. g. the image in the top banner of this blogpost. Most of our customers are interested in the True Color imagery, but recently we had a positive feedback on alternatives such as False Color Infrared (Sentinel-2 band combination: B08 - NIR, B04 - red, B03 - green) as False Color RGB or NDVI RGB render. So if you would like to see more of these or would like to see those in our default product offer, please let us know.

# Following next...

The next blogpost for under the EOxCloudless and EOxMaps topic will be introducing the latest version of mapchete-hub the software that handles the processing of our products. Other topics and updates will be arriving as well so stay tuned.

For more information download our EOxCludless brochure.

Finally, if you are looking for a nice screensaver to watch while on a coffee break click the "Tour" button at the bottom left of https://s2maps.eu (opens new window) and enjoy the ride. Also please note that our maps are free to use non commercially with proper attribution. If you aim to use our maps in your business please head to https://cloudless.eox.at (opens new window) or contact us directly via cloudless@eox.at.

# License

For commercial usage we offer the data (opens new window) under the EOX Commercial Attribution-RestrictedUse 1.1 License (opens new window) which is based on CC BY. The main change over CC BY is that EOX reserves the right to provide the data with this license. You're entirely free to use it in your services and products, you just can't transfer this license to your users. Usage examples include a WMTS you provide under your name or a product you sell that bundles the data for offline usage.

The EOX WMTS (opens new window) and EOX WMS (opens new window) are free and open to use for non-commercial applications given that when publishing the corresponding attribution (opens new window) is provided in the publication.

Petr Ševčík, EOX IT Services GmbH Email: petr.sevcik@eox.at

Contains Modified Copernicus Sentinel-2 Data 2016, 2017, 2019, 2020, and 2021.