This was a Microsoft incident and as such the RCA will be published by then in due cause.
Please follow Microsoft ID: EX680695.
Posted Oct 16, 2023 - 01:21 UTC
Resolved
Hello Everyone,
We are posting this 'Past Incident' status page to address a reported email delivery issues between 9th and 11th of October.
Incident: Email delivery was delayed impacting some features and functionality within uniFLOW Online. On investigation this was found to be caused by an upstream issue with the Microsoft email handling. Microsoft have confirmed this in 'EX680695' and an extract of this is provided below.
NT-ware were waiting on the RCA from Microsoft to provide a full status page response but due to queries from the field have opted to release this notification now. We will update the postmortem when the full Microsoft RCA is release.
Extract from Microsoft Incident Feed: ======================== Start Microsoft Update ======================= Now Resolved:
Some users may encounter delays receiving external email messages in Exchange Online ID: EX680695 Issue type: Incident
Status Service Restored
Impacted Services Exchange Online
Details Title: Some users may encounter delays sending and receiving external email messages in Exchange Online
User impact: Users may have encountered delays sending and receiving external email messages in Exchange Online.
More info: Impact was specific to Simple Mail Transfer Protocol (SMTP). Most users would have experienced up to a 1-hour delay; however, in some cases, users may have encountered delays of up to 24 hours. Additionally, the email sender may have received a Non-Delivery Report (NDR) with a "451 4.7.500 Server busy" error message, and their attempt to retry delivery of the email may have been delivered up to 24 hours later.
Final status: We’ve completed our deployment of the change, which was a targeted reversion of the code issue causing impact. Users should now be experiencing mitigation, though some residual delays may still occur over the next hour as the change completes.
Scope of impact: Impact was specific to some users who were served through the affected infrastructure. The issue affected users Worldwide and only for a portion of IP addresses, therefore not all users experienced impact.
Start time: Monday, October 9, 2023, at 9:00 PM UTC End time: Wednesday, October 11, 2023, at 5:00 PM UTC
Preliminary root cause: A recent service update that was applied to a section of infrastructure responsible for enforcing IP address anti-spam rules contained a code issue, which was resulting in the delays with SMTP email delivery.
Next Steps: - We're further reviewing the recent update to understand how the code issue was introduced and to understand what prevented it from being detected in our update testing and validation procedures, which will allow us to prevent similar issues in future updates.
We’ll publish a Post-Incident Report for this event within five business days.
======================== End Microsoft Update =======================