Spreaker Status

Maintenance mode at 9.15 UTC (will last 15 minutes)

Yesterday night our web servers had slow response times for few hours. Yesterday night and this morning we analyzed the root cause, that looks related to an hard limit we had on our database servers disks.

We’re going to migrate our database servers to new (and better) machines at 9.15 UTC. The maintenance window should last 15 minutes. More updates will be published here.

11.14 UTC: Spreaker entered maintenance mode.

11.16 UTC: Spreaker database servers have been migrated. Doing few tests.

11.17 UTC: Spreaker exited maintenance mode.

Slow response times

ISSUE at 18.00 UTC: We’re experiencing slow response times. The issue started at 18.00 UTC are we’re still investigating the root cause (seems related to the primary database).

UPDATE at 20.04 UTC: All team has been alerted and working of fixing the issue. We’re experiencing degraded performances on our primary database and we’re now working on different sides to reduce the impact.

UPDATE at 20.10 UTC: Spreaker is now in maintenance mode and both website / api are currently down. It should help us investigate the root cause…

UPDATE at 20.20 UTC: We’re turning on selected services again. API should start serve requests again in few minutes.

UPDATE at 22.45 UTC: Now website is back again and response times are much better (still not fast as usual). Keep working on…

UPDATE at 23.15 UTC: Looks like the root cause was related to the RSS feed generation. It has been been currently disabled, so RSS feeds are now down.

CLOSED at 23.45 UTC: RSS feeds are up again and that’s a good news. We’ll keep going on analyzing the issue, to avoid this problem will arise again. Thanks for your patience!

Issues with the latest version of Adobe Flash Player

In last week, some of our users reported us issues related to the Deejay Console. One of this issue is the difficult to allow the microphone access during the console initialization and to edit the Flash player settings.

After some analysis we’ve seen the the latest version of Adobe Flash Player (11.6) suffers those problems but the Flash plugin embedded in Google Chrome works much better. This is because Google Chrome is shipped with an embedded Flash plugin  different from which you’ve installed on your computer.

If you use Google Chrome, ensure that the embedded Flash plugin is enabled (should be by default):

1. Open the ”about:plugins” page (without quotes)
2. Click on the “Details” link in the upper right corner to expand all the items in the list
3. Look for “Adobe Flash Player” and ensure that all items of the list related to Flash is enabled (white background and link with the “Disable” action). If you see an item with gray backgound and an “Enable” link, this item is disabled. Click on the link to enable it.

If you don’t use Google Chrome, we’ll reccomend to install it to have the best user experience on Spreaker

Networking issues

We’re investigating networking issues in our core datacenter, located at Amazon Web Services eu-west-1.

UPDATE at 13.10 CET: we’re working with Amazon engineers to investigate the issue, working on a quick fix. More updates will be posted here.

FIXED AT 14.00 CET: the networking issue has been fixed now. After digging into with some Amazon engineers, it seems has been caused by a bug in the EC2 security group software.

Slow encoding

ISSUE at 18.21 CET: we’re experiencing network issues with some of our encoding servers. Encoding of uploaded files could be slower. We’re turning on more servers to quickly fix the issue.

UPDATE at 19.24 CET: the queue of files waiting for encoding is now empty (all files have been encoded). We still have network issues between some servers and it looks like a problem within Amazon WebService. More updates will be published later.

Issues with the latest Google Chrome Flash player on Mac

Our users reported to us an issue with the Deejay Console after upgrading to the latest versions of Google Chrome on Mac.

Even if the Microphone settings popup (see screenshot below) shows more than one available microphone input, the setting change will not be reflected and the “active” microphone still remains the system default microphone.

In order to change the settings, you need to change the system default microphone from the “Sound” options in the “System Preferences” menum, by clicking on the “Input” tab and selecting your preferred input source from the list (see screenshot below)

Issue: Amazon AWS connectivity issues

ISSUE at 07.46 CET: we’re experiencing connectivity issues in one of Amazon AWS datacenters where we’re located. We’re keeping monitored the status, but unfortunately we’ve just to wait for a fix from AWS (http://status.aws.amazon.com/).

FIXED at 07.55 CET: as stated by AWS the connectivity issue has been resolved and the service is currently operating normally.

Issue: API currently DOWN

ISSUE at 18:25 CET: public API are currently DOWN. We’re investigating the root cause.

UPDATE at 18.56 CET: public API are now UP, but we’re still investigating the cause of the issue. More updates will come later.

CLOSED at 19.28 CET: we’ve identified and fixed the root cause of the issue. API down has been caused by a memory issue on our servers that raised a chain reaction. Sorry for that.

Issue: delay on files waiting for encoding

ISSUE at 16.30 CET: We’re investigating an issue related to encoding of uploaded files. More updated will be posted later in this thread.

UPDATE at 17.00 CET: We’ve spotted the root cause and fixed it. The queue length is pretty high now, so we turned on more servers to speed up the encoding of your files.

FIXED at 17.20 CET: All the files in the queue are now encoded. The issue has been fixed. Thank you for your patience!

Issue: delay on files waiting for encoding

ISSUE at 21.30 CET: We’re experiencing an high-load and there’re a lot of files in queue for encoding. Don’t worry: we’re turning on more servers and your files will be encoded soon. Sorry for the delay.

FIXED at 22.10 CET: All the enqueued media have been encoded. We’ll keep monitoring the health of our system, ready to react (as usual) in case of any issue ;)