How it happened and what we learned.

On Friday, August 18, thanks to Joachim's great work, we announced our worldwide Sentinel-2 cloudless map (opens new window) and were quite happy about a couple of hundreds of visitors over the next days.

Please allow cookies and disable adblock to load this Tweet.
You can change your cookie settings at a later point in time here.

We maintained a steady stream of about 300-1000 visitors a day mainly because of tweets from the European Space Agency (ESA).

Please allow cookies and disable adblock to load this Tweet.
You can change your cookie settings at a later point in time here.

Please allow cookies and disable adblock to load this Tweet.
You can change your cookie settings at a later point in time here.

On Monday, August 28 ESA decided to announce Sentinel-2 cloudless on their EO-Portal and send the link to some 60.000 subscribers of their EO newsletter and share it on Facebook to some 800.000 followers.

ESA EO Portal

We were celebrating and prepared our servers for a couple of thousands visits which we expected based on experience from the release of our Europe version back in March. Personally, I was prepared to monitor our servers for a couple of days and then take off for some long awaited family vacations over a long weekend. Then it went crazy.

On Wednesday, the day before my vacations were supposed to start, the German IT news portal heise online posted a blog post about our map (opens new window). I was jokingly asking my colleagues, will this be a hug of death for our service? It really boosted visitor numbers as the graph below shows but I couldn't have been more wrong about traffic peaks.

On Friday, I woke up and checked our servers and saw some crazy numbers which I didn't believe at first. At second look I found the Daily Mail in the referrers (opens new window), holy moly. The number of visitors peaked at 35.000 on Friday. My family wouldn't allow me to spend more than 2 hours trying to increase server capacity for our tile service which of course was not nearly enough as we hadn't planned for anything near these numbers. Nevertheless, Cloudflare, which we were already using for the website, proved to be really great and Hetzner proved to be reliable with network bandwidth, at least to the extent our server was able to fill it.

HTTP response times recovered a little on Saturday and I thought, well this wasn't too bad after all. Again, I couldn't have been more wrong as Sunday morning showed. Numbers were again an order of magnitude higher than on Friday. What had happened this time? I Fucking Love Science had written a piece about us (opens new window) and shared it around midnight to their 2 million followers on Facebook. Perfect timing for me being on vacations and, well, sleeping. Sorry for the performance that night.

The post was trending and got over 38.000 shares and resulted in more than 140.000 visits on Sunday alone with a peak hour at midnight of 25.000 visits.

We quickly added a banner to apologize about not having planned for such a traffic, many thanks Silvester. Again I got 2 hours from my family in which I managed to upgrade and improve our server to serve 65GB of tiles per hour during Sunday afternoon. Still not enough, but way better than before.

Back home on Sunday evening, I bought and configured the new domain s2maps-tiles.eu to serve tiles via Cloudflare. Luckily s2maps.eu had always been behind Cloudflare. On Monday I added load balancing to the Cloudflare configuration using Amazon EC2 instances. These have a very basic configuration where MapCache is serving map tiles from SQLite tile caches loaded to Amazon EFS.

On Tuesday alone more than 120.000.000 tile requests have been made to the new domain. I was expecting Cloudflare to save maybe around 50% of bandwidth but statistics show it served 1.63TB cached and only 179.16GB un-cached on Tuesday alone which corresponds to a huge saving of 85% of bandwidth. Since enabling Cloudflare 360.000.000 tiles have been requested and served successfully in the sense that the HTTP response time has never gone beyond five seconds again.

The graph below shows the number of unique visitors of s2maps.eu since our announcement. Can you still spot the traffic spike when ESA announced it on their EO-Portal on Monday, August 28th?

Let me drop some further statistics here: We see 90 referrers with more than 100 visits and 440 in total. The average visit duration is 1:52 minutes with 3.6 actions average and a bounce rate of 16%. Our most interested visitor made a stunning 155 actions on the page. Direct entries are at 34% but that's mostly due to websites including rel="noopener noreferrer" in their links to s2maps.eu. Interest was worldwide, with at least 10 visits from 161 countries.

On a different note, we added a link to a 3D widget (opens new window) using our mapalupa service (opens new window) which got over 100.000 visits as well. Fairly big news pages like galileo.tv (opens new window) integrated it directly. Quite a validation of our idea, isn't it Daniel and Silvester?

What did we learn stumbling along the way?

• Earth Observation isn't that boring after all and does get attention from the general public.
• Preparations for increased traffic should start at the first indication. In our case that would have been heise online which would have saved us some rather poor performance impressions on my vacation weekend.
• Cloudflare really takes away a lot of traffic, 97% of bandwidth for s2maps.eu, even for an interactive map, 85% of bandwidth for s2maps-tiles.eu.
• Open Source software like MapCache, Apache, etc. is great.
• (Media) people don't give a damn about attribution requirements.
• Certainly we'll make sure to have spare server capacities for future announcements.

Many thanks for your interest and please check back if you encountered a poor performance, it's better now, promised.