Description 

On the 8th of February the uniFLOW Online system was seriously impacted by critical system failure within the Southeast Asia Microsoft Azure Data Centre.  

User Impact 

During the affected period users of uniFLOW Online will have experienced delays and timeouts of core functionality. Print and scanning services will have been impacted resulting in print and scan functions failing or needing to be performed multiple time before the action would complete. 

Scope of Impact 

This incident was limited to our Singapore deployment. 

Incident Start Date and Time 

• Microsoft incident start: Feb 7th, 2023 – 20:30 UTC 

Incident Resolved Date and Time 

• uniFLOW Online incident end: Feb 8th, 2023 – 07:30 UTC 

  • uniFLOW Online was operational 21 hours before the datacentre incident was resolved due to mitigation actions. 

• Microsoft incident end: Feb 9th, 2023 – 04:30 UTC 

Root Cause 

Microsoft: 

(This information is derived from the Microsoft incident, “Datacentre Cooling Event – Southeast Asia, Tracking ID: VN11-JD8), preliminary report. 

• A power surge impacted some of the datacentre cooling systems. 
• As a result of the cooling system issue infrastructure was taken offline limiting available resources. 
• This resource exhaustion caused significant issues across multiple Azure services affecting all tenants on the degraded infrastructure. 

NOTE: At the time of writing this PM (Post-Mortem) Microsoft has yet to release their own full incident report and NT-ware recommends you follow official Microsoft status updated for a full account of the incident. 

NT-ware: 

While multiple Azure resources were impacted across the affected infrastructure it was the Azure Service Bus component that NT-ware utilises that affected uniFLOW Online. The Azure Service Bus is vital for the queueing and management of events within uniFLOW Online. These events can be a user requesting a print job to be released or a mobile print job submission, etc.  

In the degraded state the service was not able to handle the number of requests which resulted in delays and timeouts. It is important to note that the service was NOT offline, and many requests were being processed and if a user experienced a timeout, it was very likely that subsequent attempts would work. For example, if a print job was not released at the device a second or third attempt may have. This was evident in our testing and could be seen in the azure metrics. 

‌

uniFLOW Online Emergency Mode: 

The uniflow Online emergency was designed to allow local printing when the desktop SmartClient was not able to contact the cloud service for job submission. As the SmartClient was connecting to unaffected azure services it did not meet the fail-over condition, hence not enabled automatically by the system. 

Manually enforcing Emergency mode was considered by the NT-ware Operations teams and not actioned. It was determined based on available metrices that there was an equal number of successful queued tasks getting through as there were errors. Enforcing this action may well have further impacted already working tenants depending on their configuration. The option was designed as a failsafe if the automated detection mentioned above was not triggered, however print submission by SmartClient was not available due to a cloud service failure. To additionally meet the required criteria ALL printing would been to be affected and during this incident this was not the case. 

Chronological Events: 

Feb 7th, 2023 – 20:30 UTC 

• Microsoft Incident Start. 

Feb 7th, 2023 – 21:03 UTC 

• Microsoft first notification was received by email by NT-ware Operations informing us of the service degradation. NT-ware Operations reviewed the report and begin active monitoring of the uniFLOW Online Singapore deployment. It was observed that there was a minimal number of Service Bus errors at this time. Testing by operations team found there was no functional impact to uniFLOW Online and our customers. 

Feb 08, 2023 - 01:21 UTC 

• NT-ware Operations raised support request to Microsoft as per agreed process to ensure we have a direct and indirect source of information in the incident. 

Feb 08, 2023 - 02:00 UTC (Approx) 

• NT-ware testing, and field escalation confirmed that from approximately 02:00 UTC the strain on Azures was now impacting uniFLOW Online and causing the above-mentioned delays and timeouts.  
• It was at this time we evaluated mitigation options such as manually enforcing Emergency Mode. This was not actioned as detailed above. 
• Operations continues to monitor the situation and followed up with Microsoft for an ETA on their incident resolution. 

Feb 08, 2023 - 05:30 UTC 

• Microsoft were not able to provide an ETA and the Azure Status Page information made it clear the incident was going into an extended recovery period. 
• NT-ware Operations and the Dev Operations began mitigation actions to migrate the Azure Service Bus to new Azure infrastructure.  
• Preparing and performing the required tasks through the Azure UI were very slow. This was very likely down to the amount of people connecting to Azure administration UI trying to recover their systems and perform similar actions. 

Feb 08, 2023 - 07:00 UTC 

• Dev Operations redeployed a new Azure Service Bus into an alternate (in region) data centre to bring the system fully online. 
• Service recovery started immediately but took approximately 1 hour before full operational functionality was returned, 08:00 UTC. 
• NT-ware Operations continues to monitor for several hours as the system stabilised. 

Feb 8th, 2023 – 12:00 UTC 

• After 5 hours of monitoring NT-ware Operations confirmed and closed the incident as resolved. This was also reflected on our public status page: uniFLOW Online Status - Advisory: Microsoft Azure Singapore Service Bus resource alert. 

Next Steps 

We apologize for the impact to affected customers. We are continuously taking steps to improve the uniFLOW Online Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to): 

• We will review the fail-over procedure for single component failure to reduce the time required to action in the future. 
• The information and metrices gathered during this incident will be reviewed. Any improvements to availability and recovery will be ticketed and reviewed.