We'd like to keep Spreaker up 100% of the time. When that doesn't happen, we write about it here.

Website So far so good
Api So far so good
Streaming So far so good
Mobile apps So far so good

28 October 2014

07:01 CET

Issue with our emailing system

Due to a severe issue with our emailing system, we sent out blank emails in the last hours. We’ve just pushed a fix to production and we also introduced more checks in order to avoid such issue in the future.

We’re really sorry for the inconvenience.

15 October 2014

06:55 CEST

Spreaker disabled SSL v.3 support

Spreaker disabled #SSLv3 protocol support in response to vulnerability published today. Update your browser if you’ve any issue navigating Spreaker via HTTPS.

02 October 2014

12:00 CEST

Resolved an issue with Tube

We just resolved an issue with our Tube service that affected many US customers. We’re really sorry for the inconvenience and we’re working hard to avoid the same issue in the future.

Details

Few months ago we switched our traffic to a new high-available and fault-tolerant recording infrastructure. Once of the core components of this infrastructure is the so called “balancer”. The balancer is the entry point for each recording connection and routes the traffic to the nearest available recording server.

Unfortunately, last night one of these balancers stopped to work (after 91 days of uptime) in a bad way, so that the health check was OK, but balancer was unable to route requests.

We’re currently working to investigate the root cause of the issue and detect such condition during the health checks, so that if it happens again in the future we’ll be able to automatically recover it.

Sorry again for the inconveniente. If you have any further question, don’t hesitate to contact us at http://help.spreaker.com

UPDATE (Oct, 3rd)

We investigated the issue and it looks was a bug in a client library we use to connect to Redis. We just pushed a new balancer version to production, that includes two changes:

  • Updated the Redis client library (should fix the “stale reconnection issue”)
  • Improved the balancer’s health check in order to detect such issues and automatically remove it from the pool of available balancers

20 August 2014

19:57 CEST

High publishing time

Issue at 17:30 UTC: We’re currently experiencing some networking issues while publishing your on-demand tracks at the end of your live broadcast. No worry! Your tracks are not lost: it will just take more time to get published.

We’re working to fix it as soon as possible. New recording connections will not be affected by the issue (we currently disabled affected servers).

Resolved at 21:00 UTC: The network issue is now fixed and all on-demand tracks have been published. We’re really sorry for the inconvenience.

08 July 2014

10:43 CEST

High-Available and Fault-Tolerant Recording Infrastructure

We know how important reliability is to you, and so in these past weeks we worked to provide you a high-available and fault-tolerant recording infrastructure.

We’re progressively rolling out this new infrastructure to all users. Currently all PRO users that broadcast with 3rd party applications are routed to this new infrastructure; in the next weeks we’ll open it up to all users and apps.

In this post, we’d love to share some tech details about it with you, in order to show you how it works and how we handle interruptions.

How it works

The image below shows the big picture.

image

When an application starts live broadcasting, it connects to icecast.spreaker.com. This DNS entry is resolved to the load-balancer closest to you (latency-based routing), and then the connection is routed to an available server inside that datacenter.

This design guarantees that:

  1. You connect to the nearest datacenter.
  2. Each datacenter has multiple servers, and your connection is routed to an available server. A server is considered available when it passes all health checks and it still has some capacity.

Spreaker Recording’s infrastructure is currently deployed in 3 datacenters: Europe (Ireland), US East (Virginia), US West (Oregon).

What if the connection between the client and the balancer drops?

If the connection between the client and the load balancer drops, the client will automatically retry to connect to icecast.spreaker.com. Once the connection is re-established, the balancer will route the connection to the same exact server where the client was connected before, so that it can continue to broadcast.

image

What if a balancer is down?

The DNS icecast.spreaker.com is managed by AWS Route 53. It constantly checks the health status of each balancer and, if a balancer is down, it temporarily removes the affected balancer from the pool of available ones.

image

So, when a balancer goes down:

  1. The connection between the client and the balancer drops.
  2. Route 53 removes the balancer from the pool.
  3. The client attempts to reconnect to icecast.spreaker.com - since the DNS TTL is low, the client will resolve the DNS again and will be routed to another balancer (because the affected one has been removed from the pool). The client automatically attempts reconnection multiple times, and so to smoothly handle the case, the DNS entry will not be updated yet at the time of the first reconnection .
  4. The new balancer will route the connection to the same exact server where the client was connected before.

image

What if a server is down?

The infrastructure constantly monitors the health of each server. When a server is down, it’s temporarily removed from the pool of available servers. The balancer will route new requests (or reconnection requests) to other available servers in the same datacenter.

image

The worst case scenario is when all servers in a datacenter are down. In this case, the balancer will route new requests (or reconnection requests) to available servers in other datacenters.

image

30 June 2014

13:34 CEST

Networking issues - Broadcasting in Europe affected

We’re experiencing some networking issues between two datacenters. Some of you could be temporarily unable to broadcast or once your live broadcast ends, the recorded track could take more time than usual to get ready. We’re working to fix it as soon as possible.

07 June 2014

04:00 CEST

Networking issues

Issue opened at 1:40 UTC

We’re experiencing networking issues. The issue The issue looks related internal our provider (Amazon Web Services - also confirmed by many other customers) and we’ve already alerted them. We’re waiting for a fix.

We apologize for any inconvenience.

Update at 2.10 UTC

The issue has been confirmed by AWS and engineers are working on that.

image

As a side node, the chart below looks how the networking issues affected Spreaker users. Looks that **about 30% **of users are affected.

image

Update at 2.15 UTC

Many users are reporting that networking issues have been fixed now, but still didn’t receive any official bulletin from AWS.

Resolved at 2.28 UTC

AWS confirms the issue has been fixed.

19 May 2014

22:42 CEST

Networking issues from/to US

We’re currently experiencing some networking issues from / to US. Spreaker web servers are currently hosted at AWS (Amazon Web Services) data centers in Europe, and there’re some networking failures between AWS Europe and some US nodes.

We’re really sorry for the inconvenience.


UPDATE at 21.12 UTC

Networking issues are caused by a truncated Trans-Atlantic link. Telia advised they’re working to resolve a major network issue.

Despite other cloud providers, AWS didn’t disclosure any action yet to route the network traffic through other network providers, so some of you may still experience networking issues.


UPDATE at 21.53 UTC

The following map from Akamai shows the affected area.

image


UPDATE at 22.06 UTC

According to DigitalOcean, Telia has repaired their issue.

08 May 2014

13:50 CEST

Issue: high failure rate

We’re experiencing an high failure rate on our API. We’re investing it. More updates will be published here.

UPDATE: the issue was caused by an high load on our RabbitMQ servers (we currently have 2 masters). This caused a chain effect that led to a temporary service failure in our API. Since most of our applications (both web and mobile) are based upon our API, most of you were unable to use Spreaker. We’re really sorry for that and we’ve already planned some improvements for the next week, in order to avoid such issue again.

23 April 2014

09:30 CEST

Spreaker API: removed XML/JSONP support

Dear developers,

as you may know, 2 years ago we deprecated XML and JSONP support, keeping JSON as the only officially supported format.

In the last 30 days we finally didn’t get any request with XML and JSONP response formats, so today we completely removed its support. You should notice no difference, since all of you are now using JSON.

Thanks for your help to make it happen!

Looking for help?

If you need any assistance, please contact us via our customer support service or drop us an email.