Azure Queue Storage
How Azure Queue Storage decouples pipeline stages and drives the event-driven workflow across the Digital Archive.
Overview
Azure Queue Storage is the messaging backbone of the pipeline. It decouples each processing stage from the next, ensuring that workflows are non-blocking — each service processes at its own pace without holding up upstream stages.
Queues
| Queue | Triggers | Description |
|---|---|---|
document-split-classification-queue | Document Classification service | Pushed by ingestion after a file is uploaded to Blob Storage. Triggers document splitting and classification. |
document-extraction-queue | Data Extraction service | Pushed by classification after a document is classified and routed to its container. Triggers field extraction via Azure Document Intelligence. |
document-cleansing-queue | Data Cleansing service | Pushed by data extraction after extracted data is persisted to Azure SQL. Triggers business rule cleansing. |
Poison Queues — Failure Handling
When a message fails processing and exhausts all retry attempts, Azure automatically moves it to a corresponding poison queue:
| Failed Queue | Poison Queue |
|---|---|
document-split-classification-queue | document-split-classification-queue-poison |
document-extraction-queue | document-extraction-queue-poison |
document-cleansing-queue | document-cleansing-queue-poison |
Poison queue messages represent documents that could not be processed due to an exception — for example, a corrupt file, a model failure, or an unexpected data format. These messages are held for manual review and reprocessing and are surfaced in PowerApps under the exceptions view.
Why Queue Storage
Without queuing, each workflow would need to call the next stage directly — creating tight coupling and blocking behaviour under load. Queue Storage allows each Azure Function to:
- Scale independently based on queue depth
- Retry failed messages without affecting upstream stages
- Process documents in order without blocking new ingestion