Belgium / Info / Netherlands / outage /
Dear customers and partners,
As you all know, edpnet has gone through a number of serious outages lately. The initial power outage exerted a huge impact on our core infrastructure which caused, unfortunately, much tension to our customers and ourselves. We understand that it is hard to justify the confidence in such circumstances and want to apologize for all inconvenience. Strong measures have already been taken to reduce a risk of further outages and guarantee the stability of our services.
As open communication is very important for us, we would like to provide you with a detailed overview of what exactly happened and what we have done/are going to do to prevent these situations in future and improve our service. Please, find the overview below.
Monday 26/09/2016 4:49: the data center of edpnet Sint-Niklaas lost power, UPS system and power generator failed.
Monday 26/09/2016 10:19: the power was restored, systems were back online. Due to the many issues to get the power online, we provided a backup power feed from our office power feed to the most critical systems
Monday 26/09/2016 11:00: mail load balancers were dead, probably due to intermediate power cuts. These devices were replaced.
Monday 26/09/2016 20:00: the power was lost again in the data center, critical applications such as voice services and main router were still running. The interruption caused an issue on incoming calls.
Monday 26/09/2016 20:15: The root cause of the power issue was found, the UPS itself caused the power cuts. We bypassed the UPS and restored power.
Monday 26/09/2016 21:30: incoming calls were running fine again.
After the power cut, we noticed some strange behavior on our Voice switch 2, several maintenances were performed to have the system stable again, no more issues were noticed after these maintenances.
Sunday 2/10/2016 23:30: a DDoS attack starts which causes connectivity problems.
Monday 3/10/2016 9:30: The DDoS target is found and disabled. Connectivity restored.
Tuesday 4/10/2016 0:00: Planned maintenance to replace the broken UPS, and installing an external bypass system so future UPS maintenance shouldn’t give an interruption.
Wednesday 5/10/2016 7:44: A database issue occurred on Voice switch 2.
Wednesday 5/10/2016 8:06: The database is restored, voice is operational.
Thursday 6/10/2016 1:46: Voice switch 2 crashes, is completely unavailable. Voice switch 1 isn’t taking over causing voice calls not to arrive, and outgoing calls not to be made.
Thursday 6/10/2016 6:43: manually switched all voice traffic towards Voice switch 1, incoming and outgoing calls seem to work fine.
Thursday 6/10/2016 9:00: complaints arrive that no incoming calls are possible again, it seems Voice Switch 1 only receives 50% of the incoming calls
Thursday 6/10/2016 10:57: restored Voice switch 2 on spare hardware, all calls are arriving no more voice issues.
Thursday 6/10/2016 11:30: Proximus and supplier found the issue with the failing redundancy of the voice switches, a new maintenance will be planned soon to make the necessary improvements.
List of improvements to be made:
- UPS replacement (done)
- External bypass system to avoid outage due to UPS works (done)
- Move certain critical services to our data center in Interxion Brussels (ongoing)
- Install a permanent B-power feed in our data center in Sint-Niklaas (ongoing)
- Adapt voice service for correct redundancy (done)
If you have any questions or remarks left, please, do not hesitate to contact us.
Best regards
The edpnet team
Belgium / Closed / Netherlands / outage /
Dear customers,
We have experienced an outage with incoming voice traffic. The outage has been solved.
Please, find the details below.
| Start: |
2016-10-06 01:46* |
| End: |
2016-10-06 10:57* |
| Duration: |
9:11 |
| Impacted services: |
All voice traffic |
| Affected Areas: |
all areas |
Update 11:26: all voice services are restored and are fully operational. There are still issues with redundancy, but we are working on this with the highest priority. No further unannounced voice issues should occur.
We apologize for the inconveniences.
Best regards
The edpnet team
*All times are listed in CET, Central European Time
Closed / Mail / outage /
Dear customers,
We have experienced an outage with our webmail service. The outage is currently under investigation by our NOC team. Further updates will be released as soon as possible.
Please find all details below:
| Start: |
2016-10-05 05:30* |
| End: |
2016-10-05 09:25* |
| Duration: |
Approx. 4 hours |
| Impacted services: |
Webmail |
| Affected Areas: |
All |
We apologize for the inconveniences.
Best regards
The edpnet team
*All times are listed in CET, Central European Time
Closed / Inbound Telephony / outage /
Dear customers,
We are experienced an outage with some voice services. A database issue occurred probably due to the power issues last Monday. The database has been restored.
Please find all details below:
| Start: |
2016-10-05 7:44* |
| End: |
2016-10-05 8:06* |
| Duration: |
22 min |
| Impacted services: |
Voice services |
| Affected Areas: |
All Areas |
We apologize for the inconveniences.
Best regards
The edpnet team
*All times are listed in CET, Central European Time
Belgium / Closed / Issue /
Dear customers,
We have experienced a DDoS attack. The attack is under investigation by our NOC team. The DDoS target has been found and disabled. The issue is solved. We are researching how to mitigate future attacks.
Please find all details below:
| Start: |
2016-10-02 23:30* |
| End: |
2016-10-03 09:30* |
| Duration: |
10 hours, intermittent disconnects |
| Impacted services: |
All DSL services |
| Affected Areas: |
03WIB, 03TEM, 03KAP, 03CEN, 11NER, 11TRU, 11HSW, 16HSR, 89GEN, 03WIL, 13TES, 14BAL, 11HSZ, 03BOO, 03KON, 14HER, 14TUR, 16AAR |
We apologize for the inconveniences.
Best regards
The edpnet team
*All times are listed in CET, Central European Time
Belgium / Closed / Maintenance / Netherlands /
Dear customers,
Edpnet has performed a maintenance last night in order to install new UPS-device in our data-center in Sint-Niklaas. This has been done to avoid power loss in the future.
This maintenance is finished successfully.
Please find all details below:
| Start: |
2016-10-04 00:00 AM* |
| End: |
2016-10-04 02:00 AM* |
| Duration: |
2 hours |
| Impacted services: |
Cloud telephony and colocation service |
| Affected Areas: |
all |
We apologize for the inconveniences.
Best regards
The edpnet team
*All times are listed in CET, Central European Time
Belgium / Closed / Issue / Mail / outage /
Dear customers,
We have experienced an outage with our mail server. For now all mail systems are running again.
Please find all details below:
| Start: |
2016-09-27 15:00* |
| End: |
2016-09-27 15:42* |
| Duration: |
42 minutes |
| Impacted services: |
mail |
| Affected Areas: |
all |
Update 27/09/2016 – 16-00: Mail service is fully restored.
We apologize for the inconveniences.
Best regards
The edpnet team
*All times are listed in CET, Central European Time
Belgium / Issue / Mail / Netherlands / outage / Solved / Telephony / Webservices /
Dear customers,
We have experienced an outage at our colocation in Sint-Niklaas. For now all systems are running again.
Please find all details below:
| Start: |
2016-09-26 07:00* |
| End: |
2016-09-26 16:50* |
| Duration: |
Telephony and web services: 3 hours
Mail: 9 hours 50 min |
| Impacted services: |
All telephony services, mail, web services |
| Affected Areas: |
All areas |
Update 26/09/2016 – 21:30: Incoming voice should be fine now as well.
Update 26/09/2016 – 20:00: We hit the same issue as this morning, power outage of the collocation in Sint-Niklaas, probably due to a faulty UPS, we now bypassed the UPS and are starting all services again.
We added extra protection for the main router and our telephony system so no impact was expected there, but due to a strange coincidence our incoming voice also appears to be impacted by a fiber cut in Brussels that forced a reroute.
Update 26/09/16 – 16:50: High mail latency is solved.
Update 26/09/16 – 14-00: Webmail service is restored, but some customers still could face mail latency about 1 hour due to the huge amount of mails to be delivered on our servers.
Update 26/09/2016 – 12:30: Webmail is currently down in order to avoid too much delay and heavy traffic towards our backup mail system.
Update 26/09/2016 – 10:19: All impacted services have been restored and are running again. Our mail relays are clearing out their backlogs. We expect normal mail traffic flow to be resumed soon.
We apologize for the inconveniences.
Best regards
The edpnet team
*All times are listed in CET, Central European Time
Closed / Maintenance / Online Backup / Scheduled /
Dear customers,
The sheduled maintenance has been done.
Our Acronis Cloud Backup server would be unavailable for an extended period due to maintenance.
Impact: customers using backups will not be able to save / restore backups during this time.
This maintenance may affect your services.
Please find all details below:
| Start: |
2016-08-25 10:00* |
| End: |
2016-08-25 16:00* |
| Duration: |
3-4 hours |
| Impacted services: |
Online backup |
| Affected Areas: |
Online backup |
We apologize for the inconveniences.
Best regards
The edpnet team
*All times are listed in CET, Central European Time
Closed / Info / outage /
Dear customers,
We are currently experiencing an outage with internet services, due to paths cut between Brussels and Amsterdam. Traffic is rerouted but customers may notice a service degradation and high latency. The outage is currently under investigation by our NOC team. Further updates will be released as soon as possible.
Please find all details below:
| Start: |
2016-08-11 11:46 AM |
| End: |
n/a |
| Duration: |
n/a |
| Impacted services: |
all internet services: possible quality degradation, high latency |
| Affected Areas: |
all zones |
We apologize for the inconveniences.
Best regards
The edpnet team
*All times are listed in CET, Central European Time
Update 22:45 – Issue solved. Traffic is taking the normal path again.