Application crash - summary of activities

During the night of Tuesday 9 of March and Wednesday 10 of March, a fire broke out in the OVHCloud server room, resulting in damage to two data centers, which directly caused the inability to send notifications, collect subscribers and the complete suspension of the PushPushGo application. 

The situation has now been brought under control and full operation of the app has been restored. As PushPushGo, we have made every effort to restore functionality as quickly as possible despite the serious and unforeseen situation. 

Thanks to the hard work of our IT team, we were able to do this in under 24 hours, which as you can see from the current market situation in relation to the outage, is an unexpectedly good result. 

I would like to thank all our customers for their patience and understanding in the face of the situation we found ourselves in. Your words of support have been invaluable to us, thank you.

Below are some highlights from yesterday's outage and answers to the most frequently asked questions.

Timeline

00:30 CET

The first minor incident occurred - we received a notification from Page Duty - Several servers were down. Our Administrators have started working to fix the infrastructure.

00:47 CET

First notification of fire in SGB2 - as published by OVH. Disaster recovery plan implemented.

04:10 CET

90% of the infrastructure has stopped working - (firefighting action) data center SGB1, SGB2, SGB3, SGB4, where we had a large part of the repositories. The other data centers were not affected. 

06:00 - 18:00 CET

Full infrastructure restoration and application source code recovery which consisted of preparing and running a copy of the database (to accommodate full load traffic),

Around 14:00 CET first update:

• Ability to log into the application,
• Collection of subscribers has resumed,
• We do not process dispatches (intentional),
• Infrastructure ready.

18:30 CET

Sending campaigns available again - senders/requesters started running again.

FAQ

• During what period new subscriptions were not collected?
  
  Subscriptions for notifications were not possible from 00:30 to 13:00 CET.

• Have the planned campaigns from yesterday been sent?
  
  According to our announcements yesterday, no campaigns scheduled between 00:30 and 18:00 CET were sent.

• Has any data been lost?
  
  No historical data regarding subscribers, mailings nor billing has been lost.