UK Deployment connection timeouts observed

Incident Report for uniFLOW Online

Postmortem

User Impact

During the incident users will have experienced access to the uniFLOW Online service degraded and, in some cases, not accessible.

Scope of Impact

Only our UK Deployment was impacted.

Incident Start Date and Time

Jan 09, 2023, 9:03 AM UTC

Incident End Date and Time

Jan 09, 2023, 10:45 AM UTC

Root Cause

The Azure Storage Account operational limit was reached delaying or blocking access to resources due to throttling. This was down to a misconfigured alert resulting in the required storage account configuration not taking place early enough.

The alert has been corrected, configurations reviewed and aligned to the required operational levels.

Next Steps

We have collected all information during the cause of this incident and will use this to improve our procedures and practices.

Posted Jan 11, 2023 - 02:44 UTC

Resolved

Hello everyone,

We are now moving this incident to resolved.
After a full investigation, we will prepare a post-mortem for this incident which will be published in no longer than 20 days.

Kind regards,
The uniFLOW Online team

Posted Jan 09, 2023 - 13:14 UTC

Update

We have positive feedback from our monitoring and telemetry that services have returned to normal operation. We will continue to watch this closely and perform an investigation into the incident.

Next update will be when we move the status to resolved.

Posted Jan 09, 2023 - 11:21 UTC

Update

System recovery is ongoing and most requests are already handled normally.

Posted Jan 09, 2023 - 10:41 UTC

Update

We are continuing to monitor for any further issues.

Posted Jan 09, 2023 - 10:09 UTC

Monitoring

Incident details:

Identified: 09-01-2023 09:45 UTC
Resolved: 09-01-2023 10:00 UTC

Incident Scope:

Deployment(s): UK
Components affected: All uniFLOW Online components.

Description:

Our UK deployment experienced a degradation in performance today. Upon detection of this issue, corrective actions were put in place and the service is now returning to normal operation.

We will post a new status update in about 60 minutes.

Sorry for any impact felt by this issue.

Kind regards,
The uniFLOW Online team

Posted Jan 09, 2023 - 10:07 UTC

This incident affected: UK Deployment (Identification, Printing, Email print, Scanning, Reporting, Other services).