Slow performance and responsiveness issue, US Deployment
Incident Report for uniFLOW Online
Postmortem

User Impact 

Cloud services were degraded, user operations and general usage was delayed or timed out. 

Scope of Impact 

Only impacted the US (United States) Deployment 

Incident Start Date and Time 

Jul 25, 2022, 12:10 PM 

Incident End Date and Time 

Jul 25, 2022, 2:10 PM 

Root Cause 

Our telemetry alerted us to an issue where our cloud service was not scaling out in line with the incoming load. On investigation it was identified that the available resources where not made available fast enough. 

This was acknowledged also by Microsoft, so we manually intervened to ensure the service returned to full operations as quickly as possible.  

Next Steps 

We apologize for the impact to affected customers. We are continuously taking steps to improve the uniFLOW Online Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to): 

  • We will take the lessons learnt from this incident to improve our service resilience and operational capacity.
Posted Jul 28, 2022 - 06:08 UTC

Resolved
This incident is now closed as resolved.
Posted Jul 25, 2022 - 16:55 UTC
Monitoring
Hello Everyone,

Our telemetry and field reports have confirmed the system has recovered. Initial finding identified an issues in the Azure resource cluster our cloud services was running on which has been mitigated. We will publish a Postmortem no later then 20 days from this incident following a detailed investigation and review.

We are sorry for the inconvenience.
uniFLOW Operations Team.
Posted Jul 25, 2022 - 14:20 UTC
Identified
Hello Everyone,

We have identified the cause of this issue and putting mitigation controls in place now. We will move this ticket to monitoring once we are sure the configuration changes are stable. Current estimates show that it could be up to 30 minutes before we see a full recovery of service. We are monitor this closely.

Next update at 14:00 UTC or as further information is available.

uniFLOW Operations Team
Posted Jul 25, 2022 - 13:27 UTC
Update
We are continuing to investigate this issue.
Posted Jul 25, 2022 - 13:02 UTC
Update
We are continuing to investigate this issue.
Posted Jul 25, 2022 - 12:32 UTC
Investigating
Identified: 12:00 UTC 25-07-2022

Incident Scope: US (United States) deployment.


Description:

This is a preliminary notification based on our alerting and monitoring, which has indicated to us a performance degradation within our US uniFLOW Online Cloud Service. NT-ware Operations team is already investigating and working with technical specialists on this.


Next Update: The next update will be in 13:00UTC, or as information becomes available.


Please note: You can change your email notification subscription to only receive notifications affecting deployments you’d like to watch. To do this, click ‘Manage your subscription’ in the email footer of the status page email notification.
Posted Jul 25, 2022 - 12:24 UTC
This incident affected: US Deployment (Identification, General Printing, Mobile Printing, Scanning, Reporting).