We experienced a storage outage last night forcing us to switch to our standby infrastructure. Although no transactional data was lost, our image replication could not complete prior to the outage, resulting in some images uploaded during the course of 2nd September not being available.
Although our standby system is now fully functional, we are still in "recovery" mode in order to restore storage from our primary servers. Once this is achieved, we will be able to re-sync missing images. We have no ETA for this action as we are waiting for an external vendor to assist us with the storage recovery.
Should some of your images not display, the quickest resolution would be to edit the trade (if you are unable to do this, please upload the images and then contact our customer support who will be able to attach the missing images for you) yourself.
We do hope that we should recover images in the during the course of today and we will provide an update on progress.
We apologise for the inconvenience caused.
For the technically inclined: Our storage array had 3 hard-drives fail at the same time - resulting in the storage array going into recovery mode, downing most of our web-servers. We still seek clarity from our vendor support why fault-tolerance mechanism did not kick in and how 3 drives could fail at the same time.
bidorbuy outage @ 1:35am, 3rd September 2013
Posted by Cuan Akal on 03 September 2013 11:06 AM