[EAAPL-WRK010] Streaming Progressive Output

Category: Agentic Workflows Sub-category: Real-Time Delivery Architecture Version: 1.0 Maturity: Proven Tags: streaming, progressive-output, server-sent-events, partial-results, cancellation, real-time Regulatory Relevance: ISO 42001 §8.4, EU AI Act (Art. 13)

1. Executive Summary

The Streaming Progressive Output Pattern defines how agentic workflows deliver partial, incremental results to end-users or downstream systems as processing continues, rather than waiting for complete execution before returning any output. This pattern addresses the fundamental tension between agent workflow latency (which may be 30–120 seconds for complex tasks) and user experience expectations (which are calibrated to sub-second interactive response). By streaming intermediate events — reasoning steps, partial completions, tool call notifications, intermediate results — users see visible progress, can make informed decisions about whether to wait, and can cancel tasks that are heading in an unwanted direction.

For CIO/CTO audiences: streaming is the difference between a user watching a progress bar that actually moves (building trust, enabling early feedback) and staring at a spinner for 90 seconds with no indication of what is happening (building frustration, abandoning the session). For long-running agentic workflows, streaming is not an optional UX enhancement — it is a core capability that determines whether complex AI workflows are usable in production. It also enables a feedback loop that is otherwise impossible: the user can see reasoning steps mid-execution, identify when the agent is heading in the wrong direction, and cancel before wasting the full task cost.

2. Problem Statement

Business Problem

Complex agentic workflows take 30–180 seconds to complete. Without streaming, users receive no feedback during this time: they cannot see whether the agent is making progress, cannot identify if it has misunderstood the task, and cannot cancel if it is heading in an unwanted direction. Abandonment rates for non-streaming long-running tasks are significantly higher than for streaming equivalents.

Technical Problem

Standard request-response architecture requires the server to hold a connection open for the full task duration before returning any data. This creates: (a) connection timeout risks for long tasks, (b) no partial result availability for downstream pipelines that could begin processing early results, (c) no cancellation mechanism — the caller cannot abort an in-flight task.

Symptoms of Absence

• Users abandon long-running agent tasks due to no visible progress feedback
• No ability to cancel a task mid-execution when the agent misunderstands the request
• Downstream pipelines cannot begin processing results until the entire task is complete
• Long agent workflows frequently hit HTTP timeout limits in standard request-response proxies

Cost of Inaction

• UX: High abandonment rates for long-running tasks directly reduce AI workflow adoption
• Cost: Users who cannot cancel misunderstood tasks incur full task cost even when output is useless
• Architecture: Timeout issues require workarounds (polling, webhooks) that add complexity without the UX benefit of true streaming

3. Context

When to Apply

• Agent workflow latency consistently exceeds 5 seconds
• End-users interact with the workflow in real time (not batch processing)
• Intermediate results have value before completion (partial document drafts, preliminary findings)
• Cancellation based on early output is a required capability
• Downstream systems can begin processing partial results (pipeline streaming)

When NOT to Apply

• Batch processing where latency is not a user-facing concern
• Tasks where partial outputs are misleading and must not be shown before completion
• Downstream systems cannot handle partial/incremental data
• Security constraints prohibit streaming (e.g., outputs must be validated before transmission)

Prerequisites

• Streaming-capable LLM API (server-sent events or streaming SDK)
• Streaming-capable transport layer (SSE, WebSocket, or gRPC streaming)
• Client-side streaming consumer implementation
• Cancellation token or signal mechanism
• Backpressure handling for slow consumers

Industry Applicability

4. Architecture Overview

Streaming progressive output is implemented at three layers: the LLM inference layer (token streaming), the workflow event layer (intermediate result events), and the transport layer (SSE/WebSocket).

Token Streaming Modern LLM APIs support token-level streaming: each generated token is delivered to the caller as it is produced, rather than waiting for the full completion. This is the foundation of real-time text generation feedback. Token streaming is straightforward to implement with modern SDK support but only provides the raw token stream — it does not provide structured intermediate results or workflow-level events.

Workflow Event Streaming For agentic workflows with tool calls, reasoning steps, and intermediate results, token-level streaming is insufficient. The user needs structured events: "Starting tool call: search_regulatory_db", "Tool result received: 3 relevant clauses found", "Intermediate finding: Clause 4.1 appears to create an obligation under CPS 234". The Workflow Event Emitter generates these structured events at each significant workflow step and writes them to an event stream.

Progressive Disclosure Strategy Not all intermediate content is appropriate to stream. The progressive disclosure policy defines: (a) what event types are streamed (reasoning steps, tool calls, partial results, progress indicators), (b) what event types are withheld until completion (sensitive intermediate data, unverified claims), (c) how events are formatted for the client (structured JSON events vs. human-readable text). The policy is configurable per task type and user role.

Transport Layer Streaming events are delivered to clients via: (a) Server-Sent Events (SSE) for unidirectional server-to-client streaming over HTTP (simplest, widely supported), (b) WebSocket for bidirectional streaming (enables mid-stream cancellation and user feedback), or (c) gRPC streaming for high-throughput, microservice-internal streaming.

Cancellation The client can send a cancellation signal at any time via the bidirectional transport (WebSocket or HTTP DELETE on the task resource). The Cancellation Handler in the agent workflow checks for cancellation signals between iterations and gracefully stops execution, returning the best partial result accumulated to that point with a status: cancelled metadata flag.

Backpressure If the client consumes events slower than the workflow produces them, the event buffer must be managed. Backpressure strategy: buffer up to N events; if buffer is full, pause workflow event production (let the LLM inference continue but hold events); resume when buffer clears. For fast workflows with slow clients, drop non-critical progress events but always deliver result events.

5. Architecture Diagram

6. Components

7. Data Flow

Error Flow

8. Security Considerations

Information Disclosure via Intermediate Events

• Intermediate events may expose internal reasoning, tool parameters, or intermediate data that should not be visible to the end user
• Mitigation: Progressive disclosure filter enforces role-based event visibility; never stream raw tool parameters containing secrets or PII

OWASP LLM Top 10

9. Governance Considerations

Streaming Event Audit

• For regulated workflows, streaming events constitute a real-time activity log and must be captured in the audit record even if not persisted in the client view
• The full event sequence must be reconstructable from the audit log

Governance Artefacts

10. Operational Considerations

SLOs

11. Cost Considerations

12. Trade-Off Analysis

13. Failure Modes

14. Regulatory Considerations

EU AI Act

• Art. 13 (Transparency): For high-risk AI systems, the streaming event sequence provides real-time transparency of the system's reasoning and tool use. The event archive must be retained as part of the system's transparency documentation.

ISO 42001

• §8.4: Progressive disclosure policy determines what information is made available to users at each stage of a decision-making workflow; this policy must be documented and aligned with information governance requirements.

Australian Context

• For financial services AI providing advice or decision support via streaming, intermediate findings streamed before completion must not be mistakenly treated as completed advice; clear labelling of event finality is required for compliance with ASIC RG 263 guidance on digital advice.

15. Reference Implementations

AWS

Azure

On-Premises

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Proven

18. Revision History