UK: Notification of new incident. Performance and connections issues.

Incident Report for uniFLOW Online

Postmortem

User Impact

Connecting to your uniFLOW Online tenant would be slow for fail (time out).

Scope of Impact

UK Deployment

Incident Start Date and Time

Sept 11, 2023 - 08:30 UTC

Incident End Date and Time

Sept 11, 2023 - 12:30 UTC

Root Cause

It was identified that a code update in 2023.3 specific to the caching mechanism was the cause. Under certain conditions this would perform in a sub optimal state and impact the system. Regression actions were taken immediately on all deployments to the previous state.

Next Steps

We apologize for the impact on affected customers. We are continuously taking steps to improve the uniFLOW Online Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to):

Logs and diagnostic information have been collected to review and improve this component in the future.

Posted Sep 15, 2023 - 23:34 UTC

Resolved

Hello Everyone,

Update: Incident Resolved.

Date/Time: September 11th, 12:40 pm UTC

We will review the findings and collected information from this incident to further improve our online services. There will be a postmortem published for Major (max 20 business days) and Critical (max 10 business days) incidents once a thorough investigation has been completed. For 'Critical incidents a preliminary postmortem will be published within 48 hours with initial findings).

We are sorry for the inconvenience this has caused.

Kind Regards
Online Operations Team

Posted Sep 11, 2023 - 12:41 UTC

Monitoring

Update:
We are seeing that services are now recovering across the deployment.
Recovery of all services may take some time although we can already see a vast improvement.
We are now monitoring the recovery.

Next Update:
The next update will be in 1 hour.

Posted Sep 11, 2023 - 11:41 UTC

Update

Update:
Mitigation controls are still being put in place.

Next Update:
The next update will be in 1 hour or as new information becomes available.

Posted Sep 11, 2023 - 11:10 UTC

Identified

Update:
We have identified a possible cause and mitigation controls are currently being put in place.

Next Update:
The next update will be in 1 hour or as new information becomes available.

Posted Sep 11, 2023 - 10:09 UTC

Investigating

Incident details:
We are investigating a performance issue which is reported from the field as tenants are slow or offline.

Start Time:
11th September 2023
8:30am (UTC) Approximately based on field reports and early telemetry review.

Incident Scope:
UK Deployment

Description:
Customers may have issues with access to their tenant and general service performance.

Next Update:
The next update will be in 1 hour.

Posted Sep 11, 2023 - 09:10 UTC

This incident affected: UK Deployment (Identification, Printing, Email print, Scanning, Reporting, Other services).