Notification of new incident. Delays in print and scan job processing : EU
Incident Report for uniFLOW Online
Postmortem

User Impact 

Processing delay or print and scan jobs submitted to uniFLOW Online. 

Scope of Impact 

Europe (EU) Deployment 

Incident Start Date and Time 

August 22, 2024, 9:30 UTC 

Incident End Date and Time 

August 22, 2024, 12:35 UTC 

Root Cause 

There was performance issues identified in our malware scanner. This was validated and confirmed by our security vendor as a regional issue impacting all endpoints hosted off the EU deployment. 

This was a performance issue and was not a security incident. At no time was the security posture impacted, and all processed files were still scanned. 

 

How did we respond 

9:30 UTC: Telemetry and alerting identified a growing number of unprocessed jobs. Actions were taken to mitigate this issue by provisioning resources to process the growing number of jobs. 

9:45 UTC: The Operations team learnt of the wider outage by our security malware vendor. This validated the delays we were seeing and explained why the earlier resource provisioning was not helping as expected. Jobs were still being processed but there were growing delays impacting end users. 

11:13 UTC: Operations saw a sudden recovery of the system with all resources returning to 100% efficiently. This was shortly after aligned to the recovery of our malware security vendor. There was a backlog of jobs, but they were processed through very quickly. 

12:35 UTC: Full system recovery was confirmed. 

Next Steps 

We apologize for the impact on affected customers. We are continuously taking steps to improve the uniFLOW Online Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to): 

  • We have reviewed this incident across the global operations team and key management members. 

  • Response and recovery efforts will be reviewed for future improvements to our response times and actions.

Posted Sep 01, 2024 - 00:59 UTC

Resolved
Hello Everyone,

Update: Incident Resolved.

Date/Time: August 22nd, 12:35 pm UTC

We will review the findings and collected information from this incident to further improve our online services. There will be a postmortem published for Major (max 20 business days) incidents once a thorough investigation has been completed.

We are sorry for the inconvenience this has caused.

Kind Regards
Online Operations Team
Posted Aug 22, 2024 - 12:41 UTC
Update
Update:
We are continuing to monitor the recovery of services.

Next Update:
The next update will be in 30 minutes.
Posted Aug 22, 2024 - 12:33 UTC
Update
Update:
We are continuing to monitor the recovery of services.

Next Update:
The next update will be in 30 minutes.
Posted Aug 22, 2024 - 12:04 UTC
Monitoring
Update:
We have identified a cause for this issue which has now been resolved and we are currently monitoring the recovery of services.

Next Update:
The next update will be in 30 minutes.
Posted Aug 22, 2024 - 11:36 UTC
Update
Update:
We are still investigating the cause for the delay in processing of print and scan jobs.

Next Update:
The next update will be in 30 minutes.
Posted Aug 22, 2024 - 11:10 UTC
Update
Update:
We are still investigating the cause for the delay in processing of print and scan jobs.

Next Update:
The next update will be in 30 minutes.
Posted Aug 22, 2024 - 10:31 UTC
Investigating
Incident details:
We are currently investigating delays in processing print and scan jobs as reported by our telemetry.

Start Time:
22 August 2024
09:30 am UTC

Incident Scope: 
EU Deployment

Description:
Print and scan jobs are slow in processing.
This is being treated with the highest priority by the NT-ware Operations Team.

Next Update:
The next update will be in 30 minutes.
Posted Aug 22, 2024 - 10:02 UTC
This incident affected: EU Deployment (Identification, Printing, Email print, Scanning, Reporting, Other services).