User Impact

Any user on the EU deployment will have noticed severe service degradation throughout the system. Users were unable to log into their uniFLOW Online tenant as well as devices connected to uniFLOW Online. Due to the nature of the incident the Emergency Mode on MEAP devices did not kick in where configured/available.

Scope of Impact

• Device login was not possible
• Log in to the tenant website was not possible
• Submission and release of print jobs were heavily impacted
• Important: No jobs were lost as our telemetry indicates jobs sent prior to and during the outage were processed and delivered for release.

Incident Start Date and Time

Feb 02, 2021 - 09:04 UTC

Incident End Date and Time

Feb 02, 2021 - 19:31 UTC

Root Cause

The uniFLOW Online data storage was receiving an abnormal amount of requests pushing it beyond its predefined operational working limits. While the limits under normal conditions are never reached we saw a series of events that encountered a ‘runaway’ condition that consumed the defined max. throughput. A combination of the number of uniFLOW Online SmartClients in the field, load on the system, and a registration retry mechanism saw uniFLOW Online flooded by registration requests. Important to note is that the number of SmartClients on their own is not an issue but the registration process under this fail condition.

This high load subsequently slowed down the web roles causing the uniFLOW SmartClients to not receive a successful response and causing a retry. Additional load was created by the uniFLOW Online SmartClient retrying until the data storage hit its limit subsequently affecting other services such as device login etc.

While protective measures, backend configuration improvements as well as scaling adjustments have been put in place within the first hour of the incident being reported, the system took a long time to recover due to the still high amount of requests coming in. All requests had to be processed and handled by the system.

Incident Response

Feb 02, 2021 - 09:04 UTC

The issue was reported from the field via our support channel and confirmed via our telemetry.

Feb 02, 2021 - 09:30 UTC

NT-ware Operations Team confirmed field reports, collected metrics and escalated to DevOps.

Feb 02, 2021 – 10:00 UTC

First corrective actions were taken to manage the load and number of requests. These initial actions did not correct the issues as expected.

A deeper investigation highlighted the nature of the issue and the combination of events that were leading the outage. The initial action to scale up our capacity to accept SmartClient and Device requests just placed more strain on the data store.

Feb 02, 2021 – 12:00 UTC

The balance of the system had been restored but, by this stage, we had a large number of requests to process that continued till early evening. Due to the nature of the issue, we could not risk dropping these requests and unbalancing the system further and decided to allow our web service to recover naturally. This lengthened the recovery but was the safest option available at the time.

Feb 02, 2021 - 19:31 UTC

Full operation control was restored and, with mitigations in place, we saw no report of the event in the following days of the incident.

Lessons Learnt

1. Extend alerting metrics to be able to react faster and avoid a situation where the system enters a state of abnormal high load for the affected components.
2. 2021.2 will introduce a mechanism allowing us to throttle incoming requests to ensure the backend remains responsive.
3. The uniFLOW Online SmartClient and device applets will receive further improvements to reduce the overall communication footprint.
4. Emergency Mode behavior will be reviewed to more reliably detect performance degradation and treat it as an emergency.
5. Devices connected to a local uniFLOW Server in combination with uniFLOW Online (hybrid setup) will receive an improvement for better fall back handling in case uniFLOW Online is unresponsive.

Customer Post Actions

1. Although not a cause for this outage, we would like to generally encourage customers to update their on-premise components for uniFLOW Online such as the uniFLOW Online SmartClient to benefit from the most recent and future improvements.