We'd like to keep Spreaker up 100% of the time. When that doesn't happen, we write about it here.

Website So far so good
Api So far so good
Streaming So far so good
Mobile apps So far so good

02 October 2014

12:00 CEST

Resolved an issue with Tube

We just resolved an issue with our Tube service that affected many US customers. We’re really sorry for the inconvenience and we’re working hard to avoid the same issue in the future.

Details

Few months ago we switched our traffic to a new high-available and fault-tolerant recording infrastructure. Once of the core components of this infrastructure is the so called “balancer”. The balancer is the entry point for each recording connection and routes the traffic to the nearest available recording server.

Unfortunately, last night one of these balancers stopped to work (after 91 days of uptime) in a bad way, so that the health check was OK, but balancer was unable to route requests.

We’re currently working to investigate the root cause of the issue and detect such condition during the health checks, so that if it happens again in the future we’ll be able to automatically recover it.

Sorry again for the inconveniente. If you have any further question, don’t hesitate to contact us at http://help.spreaker.com

UPDATE (Oct, 3rd)

We investigated the issue and it looks was a bug in a client library we use to connect to Redis. We just pushed a new balancer version to production, that includes two changes:

  • Updated the Redis client library (should fix the “stale reconnection issue”)
  • Improved the balancer’s health check in order to detect such issues and automatically remove it from the pool of available balancers
Looking for help?

If you need any assistance, please contact us via our customer support service or drop us an email.