We'd like to keep Spreaker up 100% of the time. When that doesn't happen, we write about it here.
|Website||So far so good|
|Api||So far so good|
|Streaming||So far so good|
|Mobile apps||So far so good|
We know how important reliability is to you, and so in these past weeks we worked to provide you a high-available and fault-tolerant recording infrastructure.
We’re progressively rolling out this new infrastructure to all users. Currently all PRO users that broadcast with 3rd party applications are routed to this new infrastructure; in the next weeks we’ll open it up to all users and apps.
In this post, we’d love to share some tech details about it with you, in order to show you how it works and how we handle interruptions.
How it works
The image below shows the big picture.
When an application starts live broadcasting, it connects to icecast.spreaker.com. This DNS entry is resolved to the load-balancer closest to you (latency-based routing), and then the connection is routed to an available server inside that datacenter.
This design guarantees that:
Spreaker Recording’s infrastructure is currently deployed in 3 datacenters: Europe (Ireland), US East (Virginia), US West (Oregon).
What if the connection between the client and the balancer drops?
If the connection between the client and the load balancer drops, the client will automatically retry to connect to icecast.spreaker.com. Once the connection is re-established, the balancer will route the connection to the same exact server where the client was connected before, so that it can continue to broadcast.
What if a balancer is down?
The DNS icecast.spreaker.com is managed by AWS Route 53. It constantly checks the health status of each balancer and, if a balancer is down, it temporarily removes the affected balancer from the pool of available ones.
So, when a balancer goes down:
What if a server is down?
The infrastructure constantly monitors the health of each server. When a server is down, it’s temporarily removed from the pool of available servers. The balancer will route new requests (or reconnection requests) to other available servers in the same datacenter.
The worst case scenario is when all servers in a datacenter are down. In this case, the balancer will route new requests (or reconnection requests) to available servers in other datacenters.