UK: Notification of new incident. Azure IoT Hub incident
Incident Report for uniFLOW Online
Postmortem

No PM will be published on this event, please see details in the initial status page.

Posted Mar 17, 2024 - 21:52 UTC

Resolved
Update: Incident resolved

Date/Time:
21 February 2024
16:14 UTC

We can confirm that starting at 16:10 UTC the uniFLOW SmartClients started reconnecting to the IoT hub successfully following the failover. At 16:14 UTC the uniFLOW SmartClients were reconnected successfully and normal operations have since resumed.

We will review the findings and collected information from this incident to further improve our online services.

We are sorry for the inconvenience this has caused.

Kind Regards
Online Operations Team

Note:
This incident was caused by an outage on Azure's side (Tracking ID 7_NQ-5PZ) impacting Azure IoT hub availability.
The following was received from Azure:
"Our internal monitoring detected a performance drop, and after investigation we identified that this was due to a configuration change. The change resulted in an unexpected mismatch in our backend, causing a subset of customers to experience availability issues."
Posted Feb 21, 2024 - 16:45 UTC
Monitoring
Update:
The IoT hub failover has completed and we can see that uniFLOW SmartClients are now reconnecting successfully. We will continue to monitor the recovery of services.

Next update:
The next update will be in 1hour.
Posted Feb 21, 2024 - 16:16 UTC
Identified
Update:
The NT-ware Operations Team have initiated a failover of the UK deployment's IoT hub to an Azure region that is not impacted by this Azure incident. The NT-ware Operations Team is monitoring the failover and will further monitor the recovery of services once the failover is completed.

Next Update:
The next update will be in 1 hour.
Posted Feb 21, 2024 - 15:55 UTC
Investigating
Incident details:

Start Time:
21 February 2024
12:50 UTC

Incident Scope:
UK Deployment

Description:
Microsoft Azure is currently investigating an issue impacting the IoT hub which is in use by uniFLOW Online. This is affecting the UK deployment.

Next Update:
The next update will be in 1 hour.
Posted Feb 21, 2024 - 15:34 UTC
This incident affected: UK Deployment (Identification, Printing, Email print, Scanning, Reporting, Other services).