Notification of new incident. Issue printing and releasing jobs: EU

Incident Report for uniFLOW Online

Postmortem

User Impact 

Users could not submit print jobs from uniFLOW SmartClients where the print job information is stored in uniFLOW Online or release print jobs on the device. Some jobs could be seen in the print queue but on release, an error message would be visible stating that the ‘Selected Print job cannot be released’.

Scope of Impact 

Europe (EU) Deployment 

Incident Start Date and Time 

September 5, 2024, 11:02 AM UTC 

Incident End Date and Time 

September 5, 2024, 12:06 PM UTC

Root Cause

A specific component reached a configured limit as a result of a prolonged period of higher-than-normal load. This caused an issue with processing certain requests.

How did we respond 

11:10 AM UTC: The Operations team received an alert and an investigation was performed to determine the cause while validating if there is any impact to end-users.

11:59 AM UTC: The root cause has been identified. Log analysis showed that a specific component reached a configured limit. 

12:01 PM UTC: A change was made which resolved the root cause and recovery started.

12:06 PM UTC: The system had fully recovered.

Next Steps 

We apologize for the impact on affected customers. We are continuously taking steps to improve the uniFLOW Online Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to): 

We have pre-emptively modified this limit on all of these components across all deployments to prevent such an issue from occurring again.
We have created additional monitoring alerts related to this component.

Was this incident related to previous incidents?

No, this incident is not related to previous incidents.

This was an isolated incident, and it was unrelated to previous observations. We have experienced periods of high load before, and queue limits were not an area of concern.

Reducing the impact of cloud service issues.

For this incident, we have confirmation that customers having ‘Emergency Mode’ configured or where job information was stored locally on their SmartClients were not impacted. NT-ware recommends customers discuss uniFLOW Online ‘Emergency Mode’ with their Canon salesperson / partner to understand the possibilities in their environment.

Emergency Mode can also provide redundancy of printing for issues unrelated to uniFLOW Online, for example internet outage, infrastructure issues impacting access to the uniFLOW Online cloud services.

Posted Sep 16, 2024 - 22:27 UTC

Resolved

Hello Everyone,

Update: Incident Resolved.

Date/Time: 12:20pm UTC Thursday, September 5th

We will review the findings and collected information from this incident to further improve our online services. There will be a postmortem published within 20 business days once a thorough investigation has been completed.

We are sorry for the inconvenience this has caused.

Kind Regards
Online Operations Team

Posted Sep 05, 2024 - 12:21 UTC

Monitoring

Hello Everyone,

We have identified the problem and taken steps to mitigate. Our telemetry confirms the printing services are recovering.

The operation team will continue to monitor the progress and reports from the field.

Regards
Online Operations Team

Posted Sep 05, 2024 - 12:09 UTC

Investigating

Incident details:
We have multiple field reports and confirmation via our telemetry that customers are having issues with printing.

Start Time:
11:30am UTC Thursday, September 5th

Incident Scope:
EU (European Deployment)

Impact:
Users will get issues with Job Submission for most submission types.
Releasing print jobs at the device will result in an error stating the 'Selected Print job cannot be release'.

We are investigating and will update shortly.

Next Update:
The next update will be in 30 minutes.

Posted Sep 05, 2024 - 12:02 UTC

This incident affected: EU Deployment (Printing).