User Impact
Tenants making use of the Universal Print by Microsoft Extension would have had their Universal Print by Microsoft jobs not processed. Print jobs submitted to Universal Print by Microsoft print queues would remain in a Pending state in the print queue and not be picked up by uniFLOW Online for further processing.
Scope of Impact
Incident Start Date and Time
Apr 7, 2024 - 10:30 UTC
Incident End Date and Time
Apr 9, 2024 - 15:55 UTC
Root Cause
When a new print job enters the Universal Print by Microsoft print queue a notification is sent to uniFLOW Online to retrieve the print job and process it further within uniFLOW Online. These notifications are received through a notification subscription that uniFLOW Online creates when Universal Print by Microsoft is configured. This subscription is valid for 70 hours and is periodically renewed by uniFLOW Online on a per-tenant basis to continue receiving that tenant’s print job notifications. This subscription renewal process makes use of an NT-ware managed application secret.
On 6 April 2024 the application secret expired. This was caused by a previous deployment update to the Universal Print by Microsoft components that set a shorter (default) expiry then was previously being monitored for.
This meant that notification subscriptions were no longer being successfully renewed for tenants. Tenants would however have continued to function normally without any impact until the existing notification subscription expires. Once a tenant’s notification subscription expires it would no longer have been possible for uniFLOW Online to be notified of new print jobs from Universal Print by Microsoft and the print jobs would remain in a Pending state in the Universal Print by Microsoft queue.
On 8 April 2024 the application secret was renewed and applied to all uniFLOW Online deployments to allow the notification subscriptions to be renewed as before. This did require a gradual rollout which started at 12:45 UTC and finished globally at 17:00 UTC. From 12:45 UTC tenant’s notification subscription renewals may have succeeded, and normal functionality would continue for these tenants.
There were, however, tenants where the notification subscription expired could not be renewed. For these tenants a new notification subscription had to be created. On 9 April 2024 at 15:30 UTC a fix was rolled out targeting these tenants to create a new notification subscription. At 15:55 UTC these tenants had new notification subscriptions created which restored Universal Print by Microsoft functionality for these tenants.
Next Steps
We apologize for the impact to affected customers. We are continuously taking steps to improve the uniFLOW Online Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to):