diff --git a/AGENTS.md b/AGENTS.md index 5d2f0dd..24f9604 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -93,6 +93,7 @@ Scheduler → SessionManager scheduled execution → AgentLoop → Scheduler del - **WorkManager** owns the single active plan per session, item state transitions, plan versions, and plan-change events; plans are optional and absent from ordinary chat context - **Scheduler** supports legacy direct-delivery jobs and managed `task`/`monitor` jobs; managed agents cannot call `send_message`, and `on_alert` suppresses only healthy informational results - **AgentLoop** is stateless across turns; it receives prepared history, drains same-Turn steering only at safe model boundaries, calls LLM providers, executes tools, and returns one result +- **AgentCatalog** is immutable per runtime generation; when orchestration is enabled, candidate preparation strictly validates trusted Markdown definitions, Provider profiles, delegated tools, Skill allowlists, and delegation edges before activation. Named Agents currently support foreground execution; durable named background delivery is not available until the run/inbox phase lands - **WebUI management APIs** only expose allowlisted config/profile/log/storage operations; keep response limits, secret redaction, atomic config writes, and profile path allowlists intact - **WebUI styling** uses the local Fluent 2 semantic tokens in `webui/src/styles.css`; page and component styles should consume the aliases instead of introducing independent hard-coded palettes, and must preserve selectable light/dark and brand-color themes, keyboard focus, responsive layout, and reduced-motion behavior. Browser-only appearance settings belong in `lib/theme.js`/`localStorage`, and `public/theme-init.js` must restore them before Svelte mounts - **Gateway config reload** validates and prepares a complete next runtime generation before retiring the old one; close runtime admission before draining inbound lanes, Sessions, Scheduler jobs, and background sub-agents; MCP connects only during activation; host/port, workspace, and effective storage path remain restart-only, and runtime reload must never mutate process environment variables @@ -105,6 +106,7 @@ Scheduler → SessionManager scheduled execution → AgentLoop → Scheduler del - **Providers** are pure HTTP clients; no bus/session/channel awareness - **Provider reasoning state** is private replay data: persist it, replay it only to the matching provider, and never expose it to clients, channels, or logs - **Tools** are executed by `AgentLoop`; every invocation is normalized to `ToolOutput` and passes through `ToolOutputProcessor`. Plain `ToolResult` implementations use the default conversion, while artifact-producing tools declare model/user audience explicitly; model capability checks, final-reply attachment, channel delivery, and Provider serialization stay outside tools +- **Delegated tool access** fails closed: new tools default to `RootOnly`; only code-reviewed `Delegatable` tools may appear in Agent Markdown, while `delegate` and scoped `get_skill` are runtime-injected. Call parameters cannot expand a named Agent's tool set - **Sleep tool** is a cancellable foreground wait bounded to 24 hours; longer or durable delays belong to Scheduler/background work, and cancelling a Turn must normalize active tool blocks to `Cancelled` - **Stateful tools** receive `ToolExecutionContext`; browser calls without `persistent_id` map each PicoBot dialog to an opaque transient agent-browser session. For long-running work an Agent may autonomously create a persistent identity and must pass its validated ID on every related action; the same ID shares one agent-browser session and serialization gate across dialogs, while different IDs have independent sessions/gates. `browser_profiles` may create IDs, persist bounded semantic labels, list, or delete only validated IDs beneath the configured profile root. Do not introduce a global persistence mode switch, a default persistent ID, automatic per-dialog persistent Profiles, Fantoccini, ChromeDriver, WebDriver, model-controlled raw agent-browser session IDs, or arbitrary Profile paths - **Health diagnostics** are read-only and share `HealthService` across `picobot health`, the `health` tool, and `/health`; checks must not install/fix dependencies, call Provider APIs, or expose secrets diff --git a/Cargo.toml b/Cargo.toml index 819acc9..c2439be 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "picobot" -version = "1.5.1" +version = "1.7.0" edition = "2024" [dependencies] @@ -8,6 +8,7 @@ reqwest = { version = "0.13.4", default-features = false, features = ["json", "r serde = { version = "1.0", features = ["derive"] } regex = "1.13" serde_json = "1.0" +serde_yaml = "0.9" async-trait = "0.1" thiserror = "2.0.19" tokio = { version = "1.53", features = ["full"] } diff --git a/README.md b/README.md index 2e41f33..9bad75b 100644 --- a/README.md +++ b/README.md @@ -343,7 +343,7 @@ PicoBot 有两类记忆： | `get_skill` | 列出或读取本地 Skill | | `memory_store` / `memory_recall` / `timeline_recall` / `memory_forget` | 长期记忆操作 | | `reload_config` | 在用户明确要求时校验并重新加载 Gateway 配置 | -| `delegate` | 启动 inline、background 或 parallel 子 Agent | +| `delegate` | 向具名 Agent 委托单个或批量任务；`foreground` 等待结果，`background` 异步执行。批量 foreground 会并发运行并按请求顺序聚合 | | `todo` | 为复杂、多轮任务创建并更新当前 session 的持久化计划 | | `send_message` | 向指定渠道或当前会话发送消息，可附带文件/截图；WebUI/TUI 当前 Turn 的附件并入最终回复 | | `chat_manager` | 查看渠道、会话和历史消息 | @@ -373,6 +373,7 @@ Skill 是包含 `SKILL.md` 的目录。加载优先级从高到低： | `providers` | LLM Provider 配置 | | `models` | 模型参数与输入能力 | | `agents` | Agent 使用哪个 provider/model | +| `agent_orchestration` | 具名子 Agent 定义目录、Root 委托白名单和编排上限；默认关闭 | | `gateway` | HTTP/WebSocket、数据库、调度器、后台任务限制 | | `client` | CLI 客户端默认 Gateway URL | | `channels` | 渠道配置，目前主要是飞书/Lark | @@ -403,6 +404,32 @@ Skill 是包含 `SKILL.md` 的目录。加载优先级从高到低： | `channels.feishu.media_dir_max_bytes` | `536870912` | | `channels.feishu.request_timeout_secs` | `30` | +### 具名子 Agent（Phase 1） + +启用 `agent_orchestration.enabled` 后，PicoBot 会在 Gateway 候选运行代构造时从配置目录下的 `definitions_dir` 加载 `*.md`。相对路径按 `config.json` 所在目录解析，且 canonical path 不得逃逸该目录；角色文件、Provider profile、工具、Skill 和委托边任一无效都会拒绝启动或热重载。第一阶段只开放具名 `foreground` 执行；具名 `background` 和 background 批量接纳会明确拒绝，直到 durable run/inbox 阶段完成。未启用时，旧 general 单任务 background 仍作为兼容路径存在。 + +```md +--- +id: researcher +description: 搜索、阅读并整理技术资料 +llm_profile: research +tools: + - file_read + - file_search + - content_search +delegates: + - reviewer +limits: + timeout_secs: 900 + max_iterations: 24 +--- +# Role + +你是一名严谨的研究 Agent，只返回与任务有关的结论和证据。 +``` + +`llm_profile` 引用顶层 `agents` 的 key，因此每个具名 Agent 可以使用不同 Provider/Model。工具权限由 Markdown 固定，并与代码内 `DelegationPolicy` 取交集；调用时的 `allowed_tools` 只能收窄旧 general Agent，不能给具名 Agent 扩权。当前可委托工具包括只读文件/内容搜索、`web_fetch`、calculator、普通 browser 动作与 sleep；写文件、Shell、HTTP 写请求、外部发送、计划和管理工具默认仅 Root 可用。`get_skill` 只读取 Definition 声明的 Skill allowlist。 + 更完整的配置字段说明见 [resources/skills/about-picobot/references/config.md](resources/skills/about-picobot/references/config.md)。 ## agent-browser 安装与使用 diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 794dda8..c4a182f 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -206,7 +206,7 @@ Session ID 格式为： SessionManager 负责组装会话上下文：系统提示、Skills、召回的 Knowledge、压缩后的 Timeline、可选的 active plan 摘要和当前消息历史。`session::turn_input` 在 Session 锁外并行读取 Knowledge、active plan 并压缩历史，然后通过同一个 assembly 路径生成首次请求和 context-overflow 重试输入；重试不得复制系统提示或 runtime context 拼接逻辑，也不得丢失已经从 mailbox 取出的 steering。普通闲聊 session 没有 plan 摘要；计划状态由 `WorkManager` 从 SQLite 读取，因此不以自然语言摘要作为权威来源。`AgentLoop` 接收完整输入执行一次模型/工具循环，本身不拥有会话状态，但通过本 Turn 的 mailbox 在安全边界接收追加用户输入。执行工具时额外传递只包含 session/turn 身份的 `ToolExecutionContext`；无状态工具使用默认实现忽略它，有状态外部适配器用它路由资源，但可按明确的单用户配置跨 dialog 共享，且不能自行反向查询 SessionManager。 -当前 Turn 的工具进度只从 `AgentLoop` 的结构化 `TurnEvent` 进入 `TurnController`，不能另建字符串 notification 通道重复投递。后台子 Agent 的 `TaskNotification` 表达跨 Turn 的任务完成，仍由独立的受监督消费者投递。自动标题属于非关键派生工作：Turn 持久化完成后由 `TaskSupervisor` 调度，Session worker 不等待模型生成；同一 Session 同时最多有一个标题任务，提交时仍校验标题保持默认值，避免覆盖用户改名。 +当前 Turn 的工具进度只从 `AgentLoop` 的结构化 `TurnEvent` 进入 `TurnController`，不能另建字符串 notification 通道重复投递。具名子 Agent 的 Phase 1 基础已接入：候选运行代从受信任配置目录严格加载不可变 `AgentCatalog`，Definition 固定 Provider profile、工具/Skill allowlist、委托边和执行限制；`delegate` 使用 `foreground/background` 两个 canonical 生命周期词，批量并发与生命周期正交。具名 foreground 支持不同 Provider、批量并发、显式 `AgentExecutionContext`、祖先环路和委托边校验。具名 background 在 durable run/inbox 完成前明确拒绝；旧 general background 的 `TaskNotification` 兼容路径仍由独立受监督消费者直接投递，不提供新设计的可靠收件箱语义。自动标题属于非关键派生工作：Turn 持久化完成后由 `TaskSupervisor` 调度，Session worker 不等待模型生成；同一 Session 同时最多有一个标题任务，提交时仍校验标题保持默认值，避免覆盖用户改名。 每个 session 最多有一个 active plan，但 plan 内多个 item 可以分别绑定不同子 Agent 并行执行。主 Agent 通过 `todo` 创建和维护计划，通过 `delegate.plan_item_id` 委托；子 Agent 不获得 `todo` 或 `delegate`。领取 item 使用条件更新防止重复执行，迟到结果只有在 plan 仍 active 且 execution ID 匹配时才允许提交。 diff --git a/docs/SUB_AGENT_ORCHESTRATION_DESIGN.md b/docs/SUB_AGENT_ORCHESTRATION_DESIGN.md index dabaf3f..6ab48b7 100644 --- a/docs/SUB_AGENT_ORCHESTRATION_DESIGN.md +++ b/docs/SUB_AGENT_ORCHESTRATION_DESIGN.md @@ -1,8 +1,10 @@ # PicoBot 子 Agent 编排与信号投递架构升级设计 -> 状态：提案（2026-08）。 +> 状态：分阶段实施中（2026-08）。Phase 1 的具名 Definition/Catalog、Provider profile、工具与 Skill 裁剪、显式执行上下文、委托图校验和批量 foreground 已落地；durable run/inbox、signal/steer 与可唤醒 sleep 仍按本文后续阶段实施。 > -> 本文定义具名子 Agent、委托图、多 Provider、`delegate`、`emit_signal`、后台结果收件箱、`queue`/`steer` 投递以及可唤醒 `sleep` 的目标架构。本文描述待实现设计；在实现完成并通过测试前，当前行为仍以代码和 `docs/ARCHITECTURE.md` 为准。 +> 本文定义具名子 Agent、委托图、多 Provider、`delegate`、`emit_signal`、后台结果收件箱、`queue`/`steer` 投递以及可唤醒 `sleep` 的目标架构。各阶段是否已经实现以代码、测试和 `docs/ARCHITECTURE.md` 为准；未落地章节仍是目标设计。 +> +> 2026-08 评审提出的 A1–A5、B1–B6、C1–C4 已纳入本文；逐项决策与理由见 [`SUB_AGENT_ORCHESTRATION_REVIEW_RESPONSE.md`](SUB_AGENT_ORCHESTRATION_REVIEW_RESPONSE.md)，代码级落地方案与验收门槛见 [`SUB_AGENT_ORCHESTRATION_IMPLEMENTATION.md`](SUB_AGENT_ORCHESTRATION_IMPLEMENTATION.md)。 ## 1. 背景与现状 @@ -71,7 +73,7 @@ PicoBot 已经具备一版子 Agent 能力：根交互 Agent 通过 `delegate` | Queue | 输入属于后续 Turn；不改变当前 Turn 的模型上下文 | | Steer | 输入尝试进入当前 Turn，并在最近安全边界注入；失败时可靠退化为 queue | | Agent Signal | Background Agent 在运行中主动发出的非终态重要事件 | -| Agent Completion | Agent Run 进入 completed/failed/timed_out/cancelled/interrupted 时由运行时自动产生的终态事件 | +| Agent Completion | Agent Run 进入 completed/failed/timed_out/cancelled/interrupted 时由运行时自动产生的终态事实；background 按 group policy 投影为 run/group inbox event | | Agent Inbox | 持久化的主 Agent 内部收件箱，是 Background 结果与信号的权威来源 | | Turn Mailbox | 当前 Turn 接受 steer 输入的有界内存邮箱，保留来源、顺序和 durable event ID | @@ -132,11 +134,17 @@ tools: delegates: - reviewer +skills: + - technical-research + limits: timeout_secs: 900 max_iterations: 24 max_children: 4 max_depth: 3 + max_concurrent_runs: 2 + max_concurrent_provider_steps: 1 + max_concurrent_tool_steps: 4 max_result_chars: 16000 --- @@ -152,28 +160,39 @@ limits: Frontmatter 只保存非秘密引用和限制；API key、base URL、headers 继续保存在 `config.json`/`.env`。`llm_profile` 引用现有 `config.agents` key，由 `Config::get_provider_config()` 解析 Provider 与 Model。 +`skills` 是可选的受信任 allowlist；只有 Definition 工具集包含 `get_skill` 时才向子 Agent注入这些 Skill。子 Agent不继承主会话临时启用的 Skill、完整历史或 memory recall。调用方需要传递的事实必须进入显式 task/context；未来若支持 memory，只能通过管理员配置的只读 scope 开放。 + ### 5.3 配置扩展 ```json { "agent_orchestration": { - "definitions_dir": "~/.picobot/agents", + "definitions_dir": "agents", "root_delegates": ["researcher", "coder", "reviewer"], "max_tree_depth": 4, "max_runs_per_tree": 16, "max_concurrent_runs": 6, "max_concurrent_runs_per_session": 4, + "max_concurrent_provider_steps": 8, + "max_concurrent_provider_steps_per_session": 4, + "max_concurrent_tool_steps": 16, + "max_concurrent_tool_steps_per_session": 8, "max_pending_inbox_events_per_session": 128, - "inbox_event_ttl_hours": 168 + "inbox_event_ttl_hours": 168, + "max_inbox_delivery_attempts": 8, + "max_user_turn_burst_before_inbox": 4, + "max_inbox_wait_secs": 30 } } ``` `root_delegates` 是 Root Agent 的出边白名单。Root 不需要也不允许出现在 definitions 目录中。 +`definitions_dir` 的相对路径按实际加载的 `config.json` 所在目录解析；第一版要求 canonical path 保持在该受信任配置目录内。默认值 `agents` 对应 `~/.picobot/config.json` 旁的 `~/.picobot/agents/`，也能让仓库内 fallback `./config.json` 使用同仓库配置目录而不跨越信任边界。 + ### 5.4 加载与校验 -AgentCatalog 在 Gateway 候选运行代准备阶段完成全部校验： +AgentCatalog 在 Gateway 候选运行代准备阶段完成全部校验，但候选代不得在此时扫描或修改 inbox/run 恢复状态；恢复扫描、旧 run 收敛和 Router 启动只能在候选代成为活动运行代后的 activation 阶段执行： - 文件大小、UTF-8、frontmatter 格式和必填字段。 - ID 格式、重复 ID、保留 ID（`ROOT`、`main` 等）。 @@ -182,10 +201,13 @@ AgentCatalog 在 Gateway 候选运行代准备阶段完成全部校验： - 每个 delegates 目标存在且不是 Root。 - 限制值在系统硬上限内。 - 角色正文和描述长度有界。 +- Skill allowlist 中的每个 ID 均存在，且只有允许 `get_skill` 的角色可以声明。 - canonical path 位于允许目录，拒绝越界 symlink。 任一引用错误应拒绝候选运行代激活，而不是静默删除工具或委托边。AgentCatalog 以 `Arc` 固定在运行代中，已启动任务不读取修改后的文件。 +第一版 Definition 只允许引用候选代准备阶段已经注册的 built-in 工具。MCP 连接按现有架构只能在 activation 阶段发生，无法在不产生外部副作用的候选代准备阶段完成严格校验，因此 MCP 工具委托暂不开放；未来需要先增加可离线校验的 MCP tool manifest，再扩展 Catalog。 + ## 6. 总体组件设计 ```mermaid @@ -203,7 +225,7 @@ flowchart LR ES --> DB[(agent_runs / agent_inbox_events)] DB --> RR[AgentResultRouter] RR --> SM[SessionManager] - SM --> TM[TurnMailbox / Session Queue] + SM --> TM[TurnMailbox / Inbox Wake + Worker Claim] TM --> Root SM --> DC[DeliveryCoordinator] ``` @@ -220,8 +242,8 @@ flowchart LR - 创建 run/group ID、父子关系和预算。 - 持久化接纳状态后启动 AgentRunner。 - 管理 foreground await、background spawn、取消和超时。 -- 控制全局、session、Agent 与任务树并发。 -- 生成自动 completion event。 +- 控制全局、session、Agent 与任务树的 run admission quota，以及 Provider/普通工具步骤的 execution permit；两类配额不共用生命周期。 +- 生成自动 completion terminal outcome，并按 group policy 物化 inbox event。 - 向 WorkManager 条件提交计划子项结果。 ### 6.3 AgentRunner @@ -237,11 +259,11 @@ flowchart LR ### 6.4 AgentEventSink / AgentResultRouter -`AgentEventSink` 负责持久化 signal/completion；`AgentResultRouter` 负责把 pending inbox event 送到原 root session。Router 的内存 wakeup 是加速器，SQLite inbox 才是权威来源。 +`AgentEventSink` 负责持久化 signal 和按 policy 生成的 run/group completion event；`AgentResultRouter` 负责把 pending inbox event 送到原 root session。Router 的内存 wakeup 是加速器，SQLite inbox 才是权威来源。 ### 6.5 ProviderFactory -根据 `llm_profile` 创建 Provider，注入 Storage/Observer，并复用当前运行代的 workspace、input types、token limit 和价格信息。Provider 仍是纯 HTTP client，不感知 Session、Channel 或 Agent 图。 +根据 `llm_profile` 创建 Provider，注入 Storage/Observer，并复用当前运行代的 workspace、input types 和 token limit。Provider 仍是纯 HTTP client，不感知 Session、Channel 或 Agent 图。`cost` 只有在 profile 明确配置 input/output/cache 价格后才计算；当前没有价格来源时持久化为 `NULL`，不能根据模型名猜测价格。 ## 7. 委托图与权限模型 @@ -252,7 +274,8 @@ flowchart LR ```text target != ROOT target in allowed_targets(caller) -depth < effective_max_depth +next_depth <= global max_tree_depth +remaining_delegation_depth > 0 tree_run_count < max_runs_per_tree target not in current_agent_ancestry remaining budget > 0 @@ -263,6 +286,8 @@ Root 的 allowed targets 来自 `root_delegates`；子 Agent 来自自身 Markdo 配置可以出现 A→B 和 B→A，允许两者在不同任务树中互相委托，但一个执行链默认禁止再次出现同一 Agent ID，从而拒绝 A→B→A 的递归乒乓。未来若需要受控返工循环，应设计显式 iteration workflow，而不是放开隐式递归。 +全局 `max_tree_depth` 是 root-relative 硬上限；Definition 的 `limits.max_depth` 表示该 Agent可继续创建的最大相对后代深度。child context 的 remaining depth 取 `min(parent_remaining - 1, target_definition.max_depth)`，任一限制耗尽即拒绝继续委托。 + ### 7.2 工具权限 调用参数不再提供 `allowed_tools` 扩权。有效工具集为： @@ -312,7 +337,7 @@ pub struct AgentExecutionContext { } ``` -`ToolExecutionContext` 扩展为包含可选 `AgentExecutionContext`、Turn wakeup handle 和资源 scope。`DelegateTool`、`EmitSignalTool` 必须实现 `execute_with_context`；权限判断不能依赖模型参数或仅依赖 Tokio task-local。 +`ToolExecutionContext` 扩展为包含可选 `AgentExecutionContext`、CancellationToken、execution gate、Turn wakeup handle 和资源 scope。Root interactive Agent 没有伪造的 run ID，其 `agent` 字段为 `None`，由 session/turn context 明确识别为 `ROOT`；sub-run 的 `agent` 字段必须为 `Some` 且 run ID 非空。Turn wakeup handle 只为 root interactive Turn 提供，sub-run 中为 `None`。`DelegateTool`、`EmitSignalTool` 必须实现 `execute_with_context`；权限判断不能依赖模型参数或仅依赖 Tokio task-local。 task-local 可以继续作为同一调用栈的便利桥接，但不是授权事实来源。Background spawn 必须显式复制所需上下文，不能假设 task-local 跨 `tokio::spawn` 传播。 @@ -329,6 +354,8 @@ agent_task → get / list / cancel / get_result 较小、单一的 schema 能减少模型错误调用，也便于分别授权。 +`agent_task` 的每次操作都必须同时校验 `root_session_id` 和调用者在任务树中的位置。Root 只能操作当前 session 的任务树；子 Agent只能读取自身及后代、取消未终态后代，不能访问祖先、sibling 或其他 session。run ID 不是授权凭证。 + ### 9.2 单任务请求 ```json @@ -409,7 +436,7 @@ Foreground 请求直接把 completion 作为 tool result 返回，因此不接 ### 9.6 幂等与接纳 -Background delegate 只有在 run/group 记录持久化成功、运行代 admission 成功且执行任务已经被 TaskSupervisor 接纳后才返回成功。可选 `idempotency_key` 在 `(root_session_id, caller_run_id, key)` 范围唯一，用于 Provider 重试时避免重复创建任务。 +Background delegate 只有在 run/group 记录持久化成功、运行代 admission 成功、completion inbox 容量已经预留且执行任务已经被 TaskSupervisor 接纳后才返回成功。可选 `idempotency_key` 在 `(root_session_id, caller_scope_id, key)` 范围唯一，用于 Provider 重试时避免重复创建任务；Root 的 `caller_scope_id` 固定为非空字面量 `ROOT`，数据库使用仅覆盖非空 key 的 partial unique index，避免 SQLite `NULL` 破坏去重。 ## 10. Prompt 与上下文隔离 @@ -434,7 +461,9 @@ PicoBot 基础运行规则 resource_scope_id = root_session_id + run_id ``` -并行子 Agent 不默认共享 browser/session 等有状态外部资源；确需共享必须由工具定义显式支持。 +并行子 Agent 不默认共享 browser/session 等有状态外部资源。共享 browser 登录态的唯一正式路径是：Root 通过 `browser_profiles` 创建/选择经过校验的 persistent ID，把它作为显式 task/artifact reference 交给获准使用 browser 的子 Agent，子 Agent在每次相关调用中显式传入该 ID；瞬时父 session browser scope 不能隐式继承。 + +无 target 的旧委托映射出的内置 `general` 兼容 Agent可在弃用期保留父 session transient browser scope，并给出迁移提示；具名 Agent不继承这一兼容例外。 ## 11. Agent Run 状态机 @@ -475,9 +504,9 @@ stateDiagram-v2 | 事件 | 产生者 | 是否终止 run | 用途 | |------|--------|--------------|------| | Signal | 子 Agent 主动调用 `emit_signal` | 否 | 重要中间状态、监控告警 | -| Completion | AgentCoordinator 自动生成 | 是 | completed/failed/timed_out/cancelled/interrupted | +| Completion outcome | AgentCoordinator 自动生成 | 是 | completed/failed/timed_out/cancelled/interrupted | -最终结果不能依赖模型记得调用工具。即使 Provider 异常、超时或任务被取消，Coordinator 也必须产生终态事件。 +最终结果不能依赖模型记得调用工具。即使 Provider 异常、超时或任务被取消，Coordinator 也必须持久化 run 的终态 outcome。Foreground 将其返回为 tool result；background `each` 将每个 outcome 物化为 run completion inbox event，`all` 只在 group 终态时物化一个 group completion inbox event。 ### 12.2 EmitSignalTool @@ -515,11 +544,13 @@ Coordinator 强制执行： - summary/details 大小与 JSON 深度限制。 - 只能投递到创建该 run 的 root session。 +未提供 `dedupe_key` 时，每次调用使用 `signal:` 作为非空 event key；提供 key 时使用 `signal::`，只在冷却窗口内去重，不能因数据库唯一约束永久压制同类告警。 + 普通进度不应滥用 signal。工具调用进度继续通过内部 Observer/TurnEvent 投影到 UI；只有需要主 Agent采取行动的事件才使用 `emit_signal`。 ### 12.3 Completion 去重 -Completion 包含本 run 已发出的 signal IDs。若最终总结重复某个信号，主 Agent可以识别并避免再次报告。正常 completion 可以配置 queue；关键 failure 可以配置 steer。禁止完全静默丢弃失败，`silent` 若未来开放也只能用于正常 completion。 +run completion payload 包含本 run 已发出的 signal IDs；group completion 则按 run 分组携带这些 IDs。若最终总结重复某个信号，主 Agent可以识别并避免再次报告。正常 completion 可以配置 queue；关键 failure 可以配置 steer。禁止完全静默丢弃失败，`silent` 若未来开放也只能用于正常 completion。 ## 13. SendMessage、EmitSignal 与附件职责 @@ -545,7 +576,7 @@ attach_artifact 工具产物 → 当前 Turn → DeliveryCoordinator（当前回 ### 13.4 自动 completion -Completion 不是工具。AgentRunner 的终结路径统一保存结果并创建 event，避免模型遗漏。 +Completion 不是工具。AgentRunner 的终结路径统一返回 terminal outcome，Coordinator 保存结果并按 foreground/background 与 group policy 创建相应投递 event，避免模型遗漏或重复。 ## 14. 持久化模型 @@ -560,6 +591,8 @@ root_session_id TEXT NOT NULL root_turn_id TEXT parent_run_id TEXT caller_agent_id TEXT NOT NULL +caller_scope_id TEXT NOT NULL +idempotency_key TEXT agent_id TEXT NOT NULL definition_hash TEXT NOT NULL provider_profile TEXT NOT NULL @@ -568,8 +601,15 @@ model_id TEXT NOT NULL mode TEXT NOT NULL depth INTEGER NOT NULL plan_item_id TEXT +execution_id TEXT NOT NULL task TEXT NOT NULL context_json TEXT +budget_json TEXT NOT NULL +signal_contract_json TEXT +signal_delivery TEXT +completion_delivery TEXT +failure_delivery TEXT +deadline_at INTEGER NOT NULL status TEXT NOT NULL result TEXT error TEXT @@ -578,14 +618,21 @@ completion_tokens INTEGER cost REAL tool_calls_count INTEGER NOT NULL DEFAULT 0 iterations INTEGER NOT NULL DEFAULT 0 -runtime_generation TEXT NOT NULL +runtime_generation INTEGER NOT NULL attempt INTEGER NOT NULL DEFAULT 1 +completion_slot_reserved INTEGER NOT NULL DEFAULT 0 started_at INTEGER finished_at INTEGER created_at INTEGER NOT NULL ``` -不保存 API key、Authorization header、Provider 私有 reasoning state 或完整 connection URL。 +```sql +CREATE UNIQUE INDEX agent_runs_idempotency +ON agent_runs(root_session_id, caller_scope_id, idempotency_key) +WHERE idempotency_key IS NOT NULL; +``` + +不保存 API key、Authorization header、Provider 私有 reasoning state 或完整 connection URL。`cost` 是 nullable projection：Provider profile 未配置价格时必须为 `NULL`，usage token 不受影响。 ### 14.2 agent_run_groups @@ -595,17 +642,26 @@ agent_run_groups id root_session_id caller_run_id +caller_scope_id +idempotency_key mode completion_policy all | each expected_runs terminal_runs +completion_slot_reserved deadline_at status created_at finished_at ``` -批量 background 默认 `completion_policy=all`，等全部 run 进入终态或 group deadline 后只唤醒主 Agent一次。独立 run 可通过 300–500ms debounce 合并，避免连续启动多个内部 Turn。 +批量请求的 `idempotency_key` 绑定 group；其 child run 的 key 为 `NULL`。单任务请求没有 group 时，key 绑定 run。两者分别使用 `(root_session_id, caller_scope_id, idempotency_key)` partial unique index，避免批量 children 互相冲突。 + +批量 background 默认 `completion_policy=all`：单个 run 终态只更新 group 计数，全部 run 终态或 group deadline 到达后创建唯一 `group_completion` event；deadline 到达时先把未终态 child 条件更新为 `timed_out`。`completion_policy=each` 则在每个 run 终态时立即创建独立 completion event，不等待 sibling；Router 可通过 300–500ms debounce 把已经到达的多个 event 合并为一次 continuation，但不能用 debounce 改变 deadline 或确认语义。 + +接纳 background run/group 时按 policy 在 session inbox 配额中预留 completion slot：`each` 在每个 run 的 `completion_slot_reserved` 记一个，`all` 在 group 字段记一个。Storage 用同一写事务统计该 session 的 `pending/leased/admitted` 事件和有效 reservation，避免并发接纳越过上限；`consumed/dead_letter` 受 TTL 清理但不占 pending 配额。容量不足在创建 run 前拒绝；signal 只能使用未预留容量。预留在 completion 事务落库或接纳回滚时释放。 + +容量判断不能在每次接纳时通过无锁 `COUNT(*)` 推断。新增每 root session 一行的 `agent_session_state`，在同一 SQLite 写事务中以条件 `UPDATE` 维护 `pending_event_count`、`reserved_completion_slots` 和单调 `revision`。background 接纳先增加 reservation；signal 只有在 `pending + reserved < limit` 时增加 pending；completion 将 reservation 原子转换为 pending；consume/dead-letter 减少 pending。启动恢复会以事件与 run/group 事实重算计数，发现差异时修复并记录告警。 ### 14.3 agent_inbox_events @@ -614,26 +670,40 @@ agent_inbox_events ------------------ id TEXT PRIMARY KEY root_session_id TEXT NOT NULL -run_id TEXT NOT NULL +scope_kind run | group +scope_id TEXT NOT NULL +run_id TEXT group_id TEXT -event_type signal | completion +event_type signal | completion | group_completion event_key TEXT NOT NULL delivery queue | steer +requires_continuation BOOLEAN NOT NULL DEFAULT TRUE severity TEXT payload_json TEXT NOT NULL -status pending | leased | admitted | consumed | dead_letter +status pending | leased | admitted | consumed | superseded | dead_letter attempt_count INTEGER NOT NULL DEFAULT 0 lease_token TEXT lease_until INTEGER +next_attempt_at INTEGER admitted_turn_id TEXT +last_error TEXT created_at INTEGER NOT NULL consumed_at INTEGER +dead_lettered_at INTEGER +fallback_notified_at INTEGER +revision INTEGER NOT NULL -UNIQUE(run_id, event_type, event_key) +UNIQUE(scope_kind, scope_id, event_type, event_key) +CHECK( + (scope_kind = 'run' AND run_id IS NOT NULL AND group_id IS NULL AND scope_id = run_id) OR + (scope_kind = 'group' AND group_id IS NOT NULL AND run_id IS NULL AND scope_id = group_id) +) ``` 完整结果保存在 `agent_runs.result`，inbox payload 默认只放有界摘要、元数据和 result reference，避免复制大文本。 +event key 始终非空：无 dedupe key 的 signal 用 `signal:`，有 dedupe key 的 signal 加冷却窗口 ID；run completion 固定为 `completion:terminal-v1`，group completion 固定为 `group-completion:terminal-v1`。由同一次 `/stop` 产生、无需主 Agent再次解释的 cancelled completion 使用 `requires_continuation=false`，在终态事务中直接记为 consumed，但仍保留事件审计和客户端投影。 + ### 14.4 原子事务 Agent completion 必须在一个 Storage 事务中： @@ -641,13 +711,31 @@ Agent completion 必须在一个 Storage 事务中： ```text UPDATE agent_runs terminal state/result/usage UPDATE agent_run_groups terminal count/status -INSERT agent_inbox_events ... ON CONFLICT DO NOTHING +CONSUME reserved completion capacity +INSERT run completion OR group completion ... ON CONFLICT DO NOTHING UPDATE bound task item by execution_id COMMIT ``` 事务失败时不能对外宣称任务完成。内存 wakeup 只有在 commit 成功后发送。 +`completion_policy=all` 只有把 group 从 non-terminal 条件更新为 terminal 的事务赢家可以插入 group completion；其他 sibling 的迟到终态只完成自己的 run 条件更新，不能重复生成 event。 + +### 14.5 continuation 消息与投递绑定 + +现有 message 持久化需要增加两个可向后兼容字段： + +```text +client_visibility visible | hidden（默认 visible） +turn_origin user | agent_continuation | scheduled（默认 user） +``` + +hidden message 参与 Provider replay 和事务回滚，但 `SessionHistory`、`TurnCommitted.messages` 与 Channel 投递只投影 visible message。Storage 的 continuation commit API 必须在一个事务中写 hidden trigger、可见 assistant/tool 消息、usage 和 event consumption。 + +内部历史读取与客户端历史读取必须拆开：Session 恢复和 Provider replay 读取 visible+hidden；WebSocket/HTTP 历史、管理面消息查询和 committed delta 默认只读 visible。hidden `role=user` 不增加面向用户的 `message_count`，也不参与自动标题生成阈值；上下文压缩和 Provider token 占用仍必须统计它。 + +root session 还需持久化最近一次有效 delivery binding：`channel`、`chat_id` 和 Channel 明确标为 durable 的 opaque context。`ChannelContext` 必须把 `durable_private` 与当前仅用于一次回复的 `reply_to`/`private` 分开；核心只持久化 `durable_private`，不通过猜测 key 名过滤现有 `private`。binding 在成功接纳外部用户输入时更新；平台 thread/root 等稳定字段保持 Channel 私有，核心只存取和回传，不解释。binding 不得包含 message/reaction ID、token、临时上传 ID 或其他短期 credential。 + ## 15. Background 事件投递 ### 15.1 统一输入类型 @@ -670,6 +758,7 @@ pub enum TurnInputSource { User, AgentSignal { run_id: String, agent_id: String }, AgentCompletion { run_id: String, agent_id: String }, + AgentGroupCompletion { group_id: String }, } pub enum InputDelivery { @@ -684,9 +773,9 @@ pub enum InputDelivery { | 主 Agent 状态 | queue | steer | |----------------|-------|-------| -| 无活动 Turn | 入 session queue，启动内部 Turn | 退化为 queue，启动内部 Turn | +| 无活动 Turn | 保持 pending、唤醒 worker claim，启动内部 Turn | 退化为 queue，同左 | | 活动 Turn 接受输入 | 入下一 Turn | 入当前 TurnMailbox | -| TurnMailbox 满/已关闭 | 保持 durable pending 后排队 | 可靠退化为 queue | +| TurnMailbox 满/已关闭 | 保持 durable pending，唤醒 worker | 可靠退化为 queue | | Provider 请求进行中 | 等下一 Turn | 等请求结束后的安全边界 | | 普通工具批次进行中 | 等下一 Turn | 等完整工具批次结束 | | sleep 进行中 | 唤醒 sleep，内容仍留在 queue | 唤醒 sleep，并在工具批次后注入当前 Turn | @@ -695,15 +784,26 @@ Steer 不承诺硬实时抢占。最迟可见时间由当前不可分割 Provide ### 15.3 原子 admission 与 fallback -AgentResultRouter 对 steer 事件执行与用户 steering 相同级别的原子判定： +AgentResultRouter 对 steer 事件使用不跨 Session 锁做 SQLite I/O 的两阶段 admission： -1. 获取目标 Session。 -2. 在 Session 状态锁内分配单调 sequence。 -3. 若 active Turn 正在 accepting，尝试 push TurnMailbox。 -4. 若 closed/full/不存在，创建内部 AgentTask 放入有界 session queue。 -5. 内存 admission 结果与 durable event lease 关联。 +1. 在锁外以条件更新把 event 从 `pending` claim 为 `leased`。 +2. 在 Session 状态锁内为当前 accepting Turn 分配 sequence 和不可排空的 mailbox reservation；closed/full/不存在则不创建 reservation。 +3. 在锁外以 lease token 把 event 条件更新为 `admitted` 并写 `admitted_turn_id`。 +4. 重新取得 Session 锁；只有同一 Turn/generation 仍 accepting 时才把 reservation 激活为 AgentLoop 可见输入。 +5. 任一步失败都删除 reservation，并以 lease token 把 event 恢复 `pending`；若恰逢 `/stop`，由 `/stop` 的 admitted-turn 条件释放和 lease expiry 兜底。 -任何竞态下事件只能属于当前 Turn 或后续 Turn之一，不能同时进入两者，也不能两者都不进入。Session queue 饱和时事件继续保持 durable pending，由 Router 重试；不能像普通瞬时通知一样丢弃。 +AgentLoop 只能排空已经激活的 reservation，因此不会在 durable admission 成功前看到事件。若事件不适合当前 Turn，Router 释放 lease、保持 `pending`，只递增该 session 的 inbox wake revision。 + +任何竞态下事件只能属于当前 Turn 或后续 Turn之一，不能同时进入两者，也不能两者都不进入。durable event payload 不进入保存用户 `AgentTask` 的 mpsc，因此普通 session queue 饱和不影响它。worker 在准备运行 continuation 时才从 SQLite claim lease；内存 wake 丢失由 pending/expired lease 扫描恢复。 + +Session worker 的接收面分为： + +```text +user task lane bounded mpsc(32)，保存 payload，满时明确拒绝 +agent inbox wake lane watch revision，只合并“SQLite 有待处理事件”的提示 +``` + +watch revision 是延迟优化而不是事实来源；不为每个 event 建立另一个可饱和 payload 队列。Router/worker 遇到瞬时错误按 `next_attempt_at` 退避重试，默认最多 8 次并受 event TTL 限制；永久错误立即进入 dead-letter。 ### 15.4 Mailbox 容量与公平性 @@ -716,6 +816,8 @@ agent event lane: 8 messages / 32 KiB 排空时按 session sequence 合并。重要 AgentSignal 可以保留专用容量，但不默认越过更早已接受的用户输入。信号洪泛由 emit_signal rate limit 和 inbox 上限共同控制。 +queue continuation 在 Turn 调度边界采用有界公平，而不是依赖 UI 保证可见：通常先处理用户任务；连续处理 `max_user_turn_burst_before_inbox`（默认 4）个用户 Turn，或最老 pending event 等待达到 `max_inbox_wait_secs`（默认 30 秒）后，下一个调度项必须是一个有界 event batch。当前活动 Turn 从不被 queue event 抢占，因此等待上限从下一个调度边界计算。UI 未读状态只是投影，不参与正确性。 + ### 15.5 安全边界注入 AgentLoop 只在以下边界排空 steer： @@ -743,12 +845,12 @@ Session worker 调度优先级： ```text 当前活动 Turn -> 已排队用户输入 +> 已排队用户输入（受 burst/age 公平上限约束） > queue background result continuation > 等待新事件 ``` -Steer event 在没有活动 Turn 时按 queue 处理。用户输入优先避免后台总结打断新请求；UI 未读状态避免持续用户流量下结果不可见。 +Steer event 在没有活动 Turn 时按 queue 处理。用户输入通常优先以避免后台总结打断新请求，但 §15.4 的 burst/age 规则保证结果不会在持续用户流量下无限饥饿。UI 未读状态只展示 pending/dead-letter 数量，不承担调度正确性。 ### 16.3 内部 continuation Turn @@ -764,6 +866,14 @@ enum AgentTaskSource { SessionManager 直接把领取的结果构造成 bounded runtime context，并加入一个内部触发语义：“检查这些后台结果，结合原始目标验证和汇总，再向用户报告。”内部输入不显示用户气泡；主 Agent输出按普通 assistant Turn 持久化和投递。 +continuation 必须创建一条 durable hidden trigger message，而不是只在内存临时拼 prompt： + +- 数据库 role 使用 Provider-compatible `user`，source 为 `agent_signal`/`agent_result`，并标记 `client_visibility=hidden`、`turn_origin=agent_continuation`。 +- hidden content 是有界 runtime envelope，包含 event/run references 和不可信数据边界；Provider history replay 会保留它，普通 history/WebSocket 投影会过滤它。 +- assistant/tool 消息照常可见。`turn_updated`/`turn_committed` 携带 turn origin，客户端可以显示“后台结果处理”标签，但不创建伪用户气泡。 + +内部 Turn 的 delivery target 来自 root session 的 durable binding：`channel`、`chat_id` 和可复用的 thread/root context。一次性 `reply_to` 不得复用。没有可用外部 binding 时仍提交历史并等待 WebUI/TUI 读取，不能猜测或改投其他 chat。 + ### 16.4 消费确认 领取流程： @@ -772,7 +882,7 @@ SessionManager 直接把领取的结果构造成 bounded runtime context，并 pending → leased → admitted → consumed ``` -`lease_token` 防止重复 worker 处理。同一事务必须保存主 Agent Turn 和把对应 inbox events 标记 consumed。若 AgentLoop 失败、Turn 取消或 Gateway 崩溃，lease 到期后事件恢复 pending。 +`lease_token` 防止重复 worker 处理。同一事务必须保存 hidden trigger、主 Agent Turn、usage，并把对应 inbox events 标记 consumed。continuation 持有 `InboxLeaseGuard`：AgentLoop 失败、Turn 取消、generation stale 等正常退出会以 token 显式 release；只有进程崩溃或强制 abort 才依赖 lease 到期恢复 pending。 外部 LLM 调用无法严格 exactly-once。为降低恢复重跑的副作用，background result continuation 默认只开放只读/汇总工具；需要外部写操作时由主 Agent向用户确认，或工具自身使用幂等键。 @@ -790,6 +900,8 @@ pending → leased → admitted → consumed Sleep 只负责唤醒，不负责消费输入。queue 内容仍属于下一 Turn；steer 内容仍由 TurnMailbox 在工具批次后注入。 +上述“当前 session 输入”只适用于 root interactive Turn。sub-run 没有独立 session input lane，`ToolExecutionContext.turn_wakeup=None`，其 sleep 只响应 timer、run cancellation、timeout 或 shutdown；不会因 root session 用户输入或 sibling signal 被唤醒。Root 如需停止 sleeping sub-run，应调用 `agent_task.cancel`。 + ### 17.2 Wakeup handle `ToolExecutionContext` 增加： @@ -864,6 +976,10 @@ Foreground 子 run 是父 run 的结构化子任务： - run 进入 waiting_children 时仍保留所有权，但不能长期占用模型执行 permit。 - 父取消后迟到结果不能提交为 completed。 +这是 AgentLoop 的显式接口约束：root Turn、foreground child、run timeout 和 runtime shutdown 的 `CancellationToken` 必须贯穿 Provider stream、可取消等待和工具批次外层，不能只依赖调用 future 被 drop。结构化取消先作为 Phase 2A 独立落点实现并回归现有 root Turn 行为，再接入嵌套 run。 + +“不默认硬中断 Provider/副作用工具”只约束普通 `steer`；`steer` 等待安全边界。`/stop` 保持现有强停止语义：取消 token 并使 root Turn future 失效，独立 child 在有界宽限期后仍未退出则由 Coordinator/Supervisor abort。Coordinator 的 terminal condition update 始终阻止取消后的迟到完成提交。 + ### 18.2 Background 所有权 Background run 归 root session 所有，不归发起它的模型 future 所有。Root Turn结束不会自动取消它。 @@ -874,8 +990,9 @@ Background run 归 root session 所有，不归发起它的模型 future 所有 用户 steering 按现有语义可被 `/stop` 丢弃；已经持久化的 AgentSignal/Completion 不能静默消失： -- 已进入 current Turn 但尚未提交的 durable event 恢复 pending。 -- 被取消 background run 产生 cancelled completion，供 UI/主 Agent获知。 +- `/stop` 关闭 TurnMailbox 时收集尚未提交的 event ID，并在 generation 失效后按 `lease_token`/`admitted_turn_id` 条件更新立即恢复 pending；lease expiry 只是崩溃兜底。 +- continuation 的 `InboxLeaseGuard` 在 worker 正常退出、失败或取消时显式 release；durable payload 不进入会被 `agent_tx.take()` 丢弃的普通 mpsc。 +- 被取消 background run 仍由 Coordinator 条件事务写 cancelled completion。由本次 `/stop` 自身造成的 completion 使用 `requires_continuation=false`，保留审计和 UI 状态但不反向启动新 Turn；`/stop` 前已经存在的其他 durable event 恢复 pending 后继续投递。 - 用户显式执行 `agent_task.cancel` 后，可以将该 run 未消费的普通 signal 标记 superseded，但保留审计记录。 ### 18.4 Gateway reload @@ -892,6 +1009,12 @@ AgentCatalog、ProviderFactory、Coordinator 和 inbox router 属于 Gateway run 进程退出后无法恢复正在进行的 LLM stream。启动恢复将旧 `running/waiting_children` 标记 `interrupted` 并生成 completion。普通有副作用 Agent run 不自动重试；只读、显式配置 idempotency/restart policy 的监控任务可以创建新 attempt，并保留原 run 的中断记录。 +### 18.6 Session 归档与删除 + +- session 被归档后不再启动内部 continuation；未消费事件进入 `dead_letter(session_archived)` 并继续在管理面可见，非 Scheduler 所有的未终态 run 被取消。该生命周期原因不发送 system fallback。 +- session 被软删除后同样取消未终态 run，释放 reservation，并将未消费事件收敛为 `dead_letter(session_deleted)`；不得猜测其他 session 或 Channel 作为替代目标，也不发送 system fallback。 +- `/stop` 不是归档或删除：它取消当前 Turn 和该 session 的 active background run，但 `/stop` 前已经存在的 durable event 仍恢复为 pending；由本次停止产生的取消 completion 仅做 status-only 审计。 + ## 19. 并发、预算与死锁避免 ### 19.1 限制层次 @@ -915,6 +1038,13 @@ Gateway 全局 active provider/tool permits permit 应限制活跃 Provider/工具步骤，而不是整个 Agent Run 生命周期。父 run 进入 `waiting_children` 前释放执行 permit，子 run 完成后父 run 再竞争 permit 继续模型迭代。Task tree ownership、timeout 和 cancellation 不随 permit 释放而消失。 +具体归属如下： + +- Coordinator 的 run admission quota 统计已接纳且未终态的 run，可以跨 `waiting_children` 持有。 +- AgentLoop 在每次 Provider 请求前按 global → session → agent 的固定顺序获取 provider step permits，stream 结束/取消即释放。 +- tool executor 只为普通工具调用获取 tool step permit；`delegate`、`agent_task`、`emit_signal` 等 runtime-control 工具不占这种 permit。 +- foreground delegate 通过状态 guard 在等待前条件更新 `running → waiting_children`，返回/取消时再条件更新；等待动作本身不持有 provider/tool permit。 + ### 19.3 预算传播 每次子委托从父 budget 派生硬上限： @@ -922,10 +1052,11 @@ permit 应限制活跃 Provider/工具步骤，而不是整个 Agent Run 生命 ```text child deadline <= parent deadline child max depth <= remaining depth -sum child token/cost reservation <= remaining tree budget +sum child token reservation <= remaining tree budget +sum child cost reservation <= remaining tree budget（仅有价格配置时） ``` -调用方可以收紧 timeout/结果大小，但不能超过 Agent Definition 和系统上限。 +调用方可以收紧 timeout/结果大小，但不能超过 Agent Definition 和系统上限。Provider profile 没有价格信息时不启用 cost reservation，只执行 token、迭代、deadline 和 run-count 硬预算。 ## 20. 可观测性与客户端表现 @@ -948,6 +1079,19 @@ WebUI 管理面展示： - queue completion 在主 Agent内部 continuation 后只显示主 Agent汇总回复。 - steer 信号可以在当前 Turn 工具状态中显示“已接纳”，最终历史由 Turn commit 校准。 +WebSocket 协议使用通用 run/event 投影，不为 Signal 复制一套状态机： + +```text +WsInbound::GetAgentRuns { session_id, cursor, limit } +WsInbound::GetAgentRun { session_id, run_id } + +WsOutbound::SessionAgentRuns { session_id, revision, runs, next_cursor } +WsOutbound::AgentRunUpdated { session_id, revision, run } +WsOutbound::AgentEventUpdated { session_id, revision, event } +``` + +`AgentEventUpdated` 覆盖 accepted/admitted/consumed/dead-letter 状态，客户端按 `(session_id, revision, event_id)` 幂等合并；重连后用 `GetAgentRuns` 全量校准。`TurnSnapshot` 和 `TurnCommitted` 增加 `turn_origin = user | agent_continuation | scheduled`。第一版取消入口继续使用 `/stop`、`agent_task.cancel` 或受保护管理 API，不增加缺少任务树授权上下文的裸 WebSocket cancel 帧。 + ### 20.3 隐私与日志 - 不显示/记录 Agent reasoning 和 Provider 私有 state。 @@ -962,16 +1106,21 @@ WebUI 管理面展示： | Agent Definition 无效 | 拒绝候选运行代；旧代继续服务 | | 委托边不允许 | delegate 立即返回 permission denied，不创建 run | | Background 持久化失败 | delegate 返回失败，不报告 run ID | +| Completion capacity 无法预留 | delegate 在创建 run 前返回 inbox capacity exceeded | | TaskSupervisor 拒绝 spawn | run 条件更新 cancelled/failed，再返回失败 | | Provider 创建失败 | run failed，foreground 返回错误；background 生成 failure event | +| Signal inbox 无可用容量 | emit_signal 返回 inbox_full；run 继续执行 | | Inbox wakeup 丢失 | pending event 由恢复扫描重新唤醒 | | TurnMailbox closed/full | steer 可靠退化 queue | -| Session queue 满 | durable event 保持 pending，Router 有界重试 | -| Main continuation Provider 失败 | event lease 到期并重试；不标 consumed | +| Session user queue 满 | 用户输入明确拒绝；durable event 不经过该队列 | +| Main continuation Provider 失败 | lease guard 显式 release 并退避重试；崩溃时才等 lease 到期 | | 主 Agent回复持久化失败 | event 不确认，避免结果消失 | +| Event 超过 attempts/TTL | 标 dead_letter，并幂等尝试一次 system fallback | | Channel 最终投递失败 | assistant history已持久化；沿用 DeliveryCoordinator terminal fallback | -所有重试必须有次数、退避、deadline 和分类；永久错误立即终态化，不能无界重试。 +所有重试必须有次数、退避、deadline 和分类；永久错误立即终态化，不能无界重试。默认最多 8 次，退避为 `1s/5s/30s/2m/10m` 后封顶 10 分钟，并同时受 inbox event TTL 限制。 + +dead-letter 记录最终原因和时间，并通过 OutboundDispatcher 最多发送一次有界 system fallback，只包含 run/group ID、终态和查询提示；`fallback_notified_at` 保证幂等。fallback 渠道失败时，SQLite run/event 记录和管理 UI 是最终诊断出口，不能把 dead-letter 伪装成已交付。 ## 22. 兼容迁移 @@ -983,10 +1132,9 @@ WebUI 管理面展示： inline → foreground parallel → foreground + tasks[] background → background -async → background（若曾接受该别名） ``` -过渡期解析旧参数并在 tool result/日志中给出弃用提示；新 system prompt 只描述 canonical 值 `foreground/background`。 +过渡期只解析代码中确实存在的旧值并在 tool result/日志中给出弃用提示；`async` 从未是有效值，不新增该别名。新 system prompt 只描述 canonical 值 `foreground/background`。 ### 22.2 allowed_tools @@ -1014,28 +1162,37 @@ async → background（若曾接受该别名） - Delegate schema 使用 target + foreground/background canonical modes。 - 批量 foreground 并发执行并聚合。 - 显式 AgentExecutionContext 和委托图授权。 +- 明确 skills/memory 不继承、具名 Agent browser scope 隔离和 legacy general scope 兼容。 - 保持旧 background 通知路径作为兼容，但不开放嵌套 background。 -### Phase 2：统一 Agent Run 持久化 +### Phase 2A：AgentLoop 结构化取消 + +- CancellationToken 贯穿 root Turn、Provider stream、工具批次和 AgentRunner。 +- 保持 `/stop` 强停止、普通 `steer` 安全边界语义。 +- 用现有 sleep cancellation、Provider stream 和 steering recovery tests 锁定回归基线。 + +### Phase 2B：统一 Agent Run 持久化 - 新增 `agent_runs`、`agent_run_groups`、Storage transaction API。 - 拆分 `delegate` 与 `agent_task`。 - Foreground 结果也持久化，修复截断结果不可查询。 -- 实现结构化取消、预算与 permit 释放。 +- 实现预算、run admission quota 与 step execution permit 释放。 ### Phase 3：Agent Inbox 与 Queue Completion - 新增 `agent_inbox_events`、lease、恢复扫描。 +- 新增 completion capacity reservation、合并式 wake lane和 worker 有界公平。 - Background completion 从 Channel direct notification 改为主 Agent内部 continuation。 -- 增加 SourceKind::AgentResult、内部 AgentTask 和 WebUI 投影。 +- 增加 hidden continuation trigger、SourceKind::AgentResult、Turn origin 和 WebSocket run/event 投影。 - 批次 completion 合并与 debounce。 +- 增加 dead-letter system fallback、UI 未读计数和 reconnect 全量校准。 ### Phase 4：Emit Signal 与 Steer - 新增 EmitSignalTool、SignalContract 和 rate/dedupe。 - SteeringMailbox 泛化为来源感知 TurnMailbox。 - 实现 steer admission、queue fallback、durable ack/recovery。 -- 添加 AgentSignal UI 和任务树。 +- 添加 AgentSignal UI；任务树复用 Phase 3 的 run/event 协议。 ### Phase 5：可唤醒 Sleep 与工具中断元数据 @@ -1043,6 +1200,7 @@ async → background（若曾接受该别名） - SleepTool 使用 watch revision + timer + cancellation select。 - queue/steer 唤醒内容边界和测试。 - 为工具增加 InputInterruptPolicy，默认 Never。 +- sub-run 保持 timer/cancellation-only，不获得 root TurnWakeupHandle。 ## 24. 预计代码边界 @@ -1066,7 +1224,7 @@ src/tools/ src/session/ ├── turn_mailbox.rs typed steer inputs -├── agent_inbox.rs claim/admit/ack orchestration +├── agent_inbox.rs claim/admit/ack、lease guard、coalesced wake └── session.rs typed AgentTask scheduling src/storage/ @@ -1095,6 +1253,7 @@ src/storage/ - 单项失败不丢其他项结果。 - Background 只有持久化并成功 spawn 后才返回 run ID。 - 同 idempotency key 不重复创建 run。 +- Root caller scope 的 idempotency key 在 SQLite 中同样去重。 - Background completion 不直接伪装为用户消息。 ### 25.3 Provider 与工具 @@ -1104,6 +1263,8 @@ src/storage/ - RootOnly 工具不能通过 Markdown 或兼容 allowed_tools 获得。 - runtime-injected delegate/emit_signal 只在上下文允许时存在。 - 并行 run 的 browser/resource scope 隔离。 +- persistent browser profile 可显式共享；具名 Agent不继承 transient parent scope。 +- 子 Agent不隐式继承主会话 history/memory/临时 Skill。 ### 25.4 Inbox 与投递竞态 @@ -1115,7 +1276,12 @@ src/storage/ - 用户队列优先于 queue completion。 - Turn persist 失败时 event 不 consumed。 - lease 超时后可重领，旧 lease token 不能提交。 -- 多 run group 只触发一次汇总 Turn。 +- 正常取消/worker 退出由 lease guard 立即 release，不等待 lease timeout。 +- user mpsc 满不影响 durable wake；wake revision 丢失后扫描可恢复。 +- 用户持续输入时 burst/age 公平上限仍调度 continuation。 +- completion capacity 在 background 接纳时预留，signal 不能抢占。 +- `each` 逐 run 提前投递；`all` 只触发一次 group 汇总 Turn。 +- retries/TTL 耗尽进入 dead-letter，system fallback 最多发送一次。 ### 25.5 Signal @@ -1135,12 +1301,14 @@ src/storage/ - 多条输入只消费一次且顺序稳定。 - `/stop`、parent cancellation、shutdown 取消 sleep 并终态化工具块。 - wakeup 与 timer 同时发生时不丢输入；输入若未入当前 Turn则可靠排队。 +- sub-run sleep 不被 root session 输入唤醒，只响应 timer/cancellation。 ### 25.7 取消、并发与恢复 - 父 foreground 取消级联后代。 - 父 waiting_children 不持有唯一 permit，无死锁。 - `/stop` 取消 session background runs，并恢复未提交 durable events。 +- `/stop` 产生的 cancelled completion 只投影状态，不启动新的 continuation。 - 迟到结果不能覆盖 cancelled/interrupted。 - reload 关闭 admission 后拒绝新 run/signal，pending inbox 由新代恢复。 - Gateway 重启把 running 标记 interrupted 并生成 completion。 @@ -1150,9 +1318,12 @@ src/storage/ - AgentSignal 不渲染为用户气泡。 - Internal continuation 输入不出现在普通历史，assistant 汇总正常持久化。 +- hidden trigger 会参与 Provider replay，并与 assistant/usage/event consumed 原子提交。 +- continuation 使用稳定 delivery binding，绝不复用一次性 reply_to。 - reasoning/provider state 不进入信号、API、客户端和日志。 - send_message 仍走外部投递确认；emit_signal 不走 OutboundDispatcher。 - 同 Turn 附件兼容路径与未来 attach_artifact 不产生重复历史。 +- run/event 增量按 revision 幂等，断线重连后可全量校准。 ## 26. 必须保持的架构不变量 @@ -1161,7 +1332,7 @@ src/storage/ 3. Foreground/Background 只描述委托方等待行为；并发是独立调度维度。 4. Queue 输入永不泄漏正文到当前 Turn；Steer 只在安全边界注入。 5. Signal 先持久化后唤醒；内存通知不是事实来源。 -6. Signal 是非终态事件，Completion 由运行时自动生成且恰好对应一个 run 终态。 +6. Signal 是非终态事件；run Completion 由运行时自动生成且恰好对应一个 run 终态，`all` policy 的 Group Completion 恰好对应一个 group 终态。 7. SendMessage 是外部输出，EmitSignal 是内部输入，不能用一个公开万能工具混合权限。 8. Durable Agent event 在 `/stop`、Turn 失败或 Gateway 崩溃时不能静默丢失。 9. Agent Definition 和 Provider 绑定 runtime generation；运行中不热切换。 @@ -1169,6 +1340,9 @@ src/storage/ 11. 父 run 等待子 run 时不持有会造成递归死锁的执行 permit。 12. 完成状态、结果、usage、计划子项和 inbox event 使用事务/条件更新提交。 13. 客户端、Channel 和日志永不暴露 Provider 私有 reasoning state、secret 或本地内部路径。 +14. Durable event payload 只以 SQLite inbox 为权威来源；普通 user task mpsc 和合并式 wake lane 都不能成为确认点。 +15. Continuation 的 hidden trigger 必须可供 Provider replay，但不得投影为用户消息；trigger、回复和 event consumption 原子提交。 +16. 已接纳 background run 的 terminal completion 容量已经预留；运行结束不能因 signal 洪泛丢失 completion。 ## 27. 设计结论 @@ -1185,7 +1359,7 @@ AgentCoordinator ↓ queue | steer ↓ - Session queue | current TurnMailbox + inbox wake/worker claim | current TurnMailbox ↓ Root Agent ``` diff --git a/docs/SUB_AGENT_ORCHESTRATION_IMPLEMENTATION.md b/docs/SUB_AGENT_ORCHESTRATION_IMPLEMENTATION.md new file mode 100644 index 0000000..4d806b3 --- /dev/null +++ b/docs/SUB_AGENT_ORCHESTRATION_IMPLEMENTATION.md @@ -0,0 +1,1113 @@ +# PicoBot 子 Agent 编排实施细节与可实施性审查 + +> 状态：实施基线（2026-08）。 +> +> 本文以 [`SUB_AGENT_ORCHESTRATION_DESIGN.md`](SUB_AGENT_ORCHESTRATION_DESIGN.md) 为产品与架构规范，以 [`SUB_AGENT_ORCHESTRATION_REVIEW.md`](SUB_AGENT_ORCHESTRATION_REVIEW.md) 和 [`SUB_AGENT_ORCHESTRATION_REVIEW_RESPONSE.md`](SUB_AGENT_ORCHESTRATION_REVIEW_RESPONSE.md) 为评审记录，并逐项对照当前代码给出可以直接拆分为开发任务的实现方案。 +> +> 本文不是“代码已经实现”的声明。实现完成前，运行时事实仍以当前代码、测试和 [`ARCHITECTURE.md`](ARCHITECTURE.md) 为准。 + +> 实施进度（2026-08）：Phase 1 已落地具名 Definition/Catalog、不同 Provider profile、工具/Skill fail-closed 裁剪、显式 `AgentExecutionContext`、父子委托边与 ancestry 校验、canonical `foreground/background` schema 以及批量 foreground 并发。旧 general background 仅作为兼容路径保留。Phase 2A 已落地结构化取消：`AgentError::Cancelled/TimedOut`、CancellationToken 贯穿 root Turn、Provider 连接/stream、并行与串行工具批次以及 sleep；`/stop` 保留 oneshot 兼容桥接并同时取消 Turn token，协作式与强制路径提交同一 Cancelled 终态；树级 `max_runs_per_tree` 由共享原子计数在 foreground 委托接纳时强制。Phase 2B 已落地 durable foreground 编排：schema v6（`agent_run_groups`/`agent_runs`/`agent_session_state`/`agent_inbox_events` 与 messages/sessions 扩展列）在单一迁移事务中原子创建；Storage 领域 API（接纳、running/waiting_children 条件转换、execution-ID 条件 terminal commit、plan item 原子领取/完成、游标分页）；`ExecutionGate` 分离 run quota 与 provider/tool step gate（global→session 顺序获取、弱引用键控回收、取消可中断等待），root Turn 步骤同样占用 step gate；`AgentCoordinator` 持久化全部具名 foreground run（单任务不建 group、批量建 group 并按请求顺序返回）、父 run `waiting_children` 转换、迟到结果丢弃与树位置授权；`agent_task` 工具提供 scoped get/list/get_result/cancel。foreground 不占用 run quota，嵌套 foreground 在并发上限为 1 时不死锁。Phase 3 已落地 durable inbox 与 queue continuation：具名 background 单任务经 Coordinator 接纳（completion slot 预留 → 持久化 queued → TaskSupervisor 托管 runner → terminal commit 原子转换 reservation 为 completion 事件 → notifier wake），spawn 拒绝执行补偿事务；`agent_inbox_events`/`agent_session_state` 容量条件更新与 claim/lease/admit/release/supersede/dead-letter API；`AgentInboxNotifier`（弱引用 late-bound wake）与 Session 双 lane worker（user mpsc + inbox watch 合并 wake），公平调度按 `max_user_turn_burst_before_inbox` 与 `max_inbox_wait_secs` 强制 continuation；continuation Turn 使用 hidden trigger、只读工具集，`commit_continuation_turn` 同事务提交 hidden trigger + assistant/tool/usage + event consume，失败显式 release lease；`client_visibility`/`turn_origin` 贯穿 ChatMessage/MessageMeta/协议 DTO，客户端历史查询默认过滤 hidden，`message_count` 只统计可见用户输入；activation recovery 收敛旧代 run（interrupted + failure completion）、过期 lease、group counter 与容量计数；`/stop`/archive/delete 走 `cancel_session`（suppress_continuation 写 consumed completion）并 dead-letter。background completion 不再直接通知 Channel。Phase 3 审查修复：worker 通过 `next_pending_due_at` 定时器在 release backoff 到期后重新 claim（不再依赖 wake）；达到 `max_inbox_delivery_attempts` 的事件在正常运行中即 dead-letter 而非无限重试；`recover_on_activation` 对含 due 事件的在内存 session 发送合并 wake；修正 MIN 聚合无行时 NULL 被解码为 0 导致 worker 空转的缺陷（`oldest_pending_due`/`next_pending_due_at` 用 `Option>` 显式解码）。Phase 4 已落地 emit_signal 与 steer：Agent Definition frontmatter 新增 `signal:` 块（`SignalContract`：delivery queue/steer、总数/字节/间隔/burst/severity allowlist/dedupe 冷却窗/JSON 深度，全部由工具与 Coordinator 强制，模型只提供 key/severity/summary/details/dedupe_key）；`EmitSignalTool` 仅在带 contract 的 run 注册（fail-closed），`insert_agent_signal` 支持同冷却窗 dedupe（返回原 event + deduplicated）；Coordinator `emit_signal` 校验 run 活跃与 execution ID、容量条件插入、投影+notifier wake，取消 run 时未消费 signal 自动 supersede；SteeringMailbox 泛化为来源感知 TurnMailbox（user lane 32/64KiB 与 agent lane 8/32KiB 独立容量，`TurnInput{source,delivery,durable_event_id,lease_token}`，agent steer 投影为 hidden user 消息保留 source 元数据）；steer 两阶段 admission（claim → mailbox 预留 → DB admit(turn_id) → 同 Turn/generation 激活），任何失败 release lease 并 wake queue lane；`/stop`/generation 变更时已 admit 的 steer 事件按 lease token 条件释放回 pending（绝不静默丢弃），用户 Turn commit 与 steer 事件 consume 同事务（`persist_turn_batch_with_steer_consumption`）。Phase 3 收尾已落地：`ChannelContext.durable_private`（Feishu 仅 thread/root/chat_type 进入）持久化到 `sessions.delivery_context` 并被 continuation 投递复用；WS 协议 `GetAgentRuns`/`GetAgentRun` 与 `SessionAgentRuns`/`AgentRunUpdated`/`AgentEventUpdated`（有界 `AgentRunView`/`AgentEventView`，不暴露 budget/contract/delivery context/execution id）；`AgentProjectionHub` broadcast（复用 plan-change 模式，lag 由客户端 GetAgentRuns 校准）；HTTP `/api/agent-runs*`（列表游标分页/详情/events/cancel）+ `/api/tasks` legacy/new union；WebUI TasksPage 后台 tab 渲染 run tree（group 折叠、agent 标签、深度缩进），ChatPage 显示 continuation 标签与 Signal 卡片（投影事件，不插入 history）。Phase 5 尚未完成：wake-aware sleep 与工具中断策略。 + +## 1. 审查结论 + +结论为：**设计可实施，没有需要推翻总体方案的阻断项，但必须按依赖顺序实施，不能在当前 `SubAgentManager` 上直接追加 durable inbox 或 steer signal。** + +当前代码已经提供以下可复用基础： + +- `AgentLoop` 已经是跨 Turn 无状态执行器，工具执行统一经过 `ToolExecutionContext` 和 `ToolOutputProcessor`。 +- Session 已有单 worker、用户输入有界队列、活动 Turn steering、generation/state version 和 persistence lock。 +- `TurnController`、`DeliveryCoordinator` 已建立“持久化成功后才 Completed”的边界。 +- Storage 已有单事务迁移、批量消息与 usage 原子提交模式。 +- `TaskSupervisor`、`RuntimeAdmission` 和 Gateway 候选运行代已经提供重载与有界关闭框架。 +- WorkManager 已有 `execution_id` 条件更新和计划变更广播。 +- WebSocket 已有 request/event 路由与断线后全量校准的现成模式。 + +必须先解决的结构性差距如下： + +| 领域 | 当前实现 | 实施要求 | 风险等级 | +|------|----------|----------|----------| +| Agent 身份 | 一个共享 Provider 的临时子 Agent | 不可变 AgentCatalog、具名 definition、显式 caller/run context | 高 | +| 委托生命周期 | `inline/background/parallel` 混合建模 | `foreground/background` 与批量并发正交 | 中 | +| 取消 | Session 在 AgentLoop 外 drop future | CancellationToken 贯穿 Provider stream、工具批次和 child tree | 高 | +| 并发 | background run 持有一个 Semaphore permit | run admission 与 provider/tool step permit 分离 | 高 | +| 结果投递 | background 终态直接发 Channel | SQLite inbox 为事实源，Session wake 仅为加速器 | 高 | +| steering | mailbox 只接受用户 ChatMessage | 来源感知 TurnInput、durable admission 和可靠 queue fallback | 高 | +| continuation | 用户任务在模型调用前落库 | hidden trigger 与 assistant/tool/usage/event consume 同事务提交 | 高 | +| history | 所有 role=user 都作为用户消息 | internal replay 读 hidden，客户端默认过滤 hidden | 高 | +| Channel context | `reply_to` 与 opaque `private` 未区分稳定性 | Channel 显式提供 `durable_private`，核心不得猜 key | 高 | +| 重载恢复 | 候选代创建完整 SessionManager | Catalog 在 prepare 校验，run/inbox 恢复只能 activation 后执行 | 高 | +| 客户端 | `/api/tasks` 只读旧 `background_tasks` | 新 run/event 投影、revision、旧表 union 过渡 | 中 | + +实现的关键路径为： + +```text +Catalog/Tool policy + ↓ +AgentLoop cancellation + execution gate + ↓ +Run/group durable lifecycle + ↓ +Inbox state/capacity + Session wake + queue continuation + ↓ +Typed TurnMailbox + emit_signal/steer + ↓ +Wake-aware sleep + UI/legacy cleanup +``` + +Phase 3 依赖前面全部基础。若跳过 cancellation、持久化状态机或 hidden message 分层，结果会在 `/stop`、reload、SQLite 失败或进程重启时出现无法修复的丢失与重复。 + +## 2. 实施中固定的架构决策 + +以下决策在编码前固定，不留给各模块自行解释： + +1. Root Agent 是运行时身份，不创建虚假的 root run。`ToolExecutionContext.agent=None` 表示 Root；child 必须携带完整 `AgentExecutionContext`。 +2. `foreground/background` 只描述调用方是否等待。批量是否并发由 `tasks[]` 和调度器决定；不再存在 canonical `parallel` 模式。 +3. 每个 Agent Run 都落 `agent_runs`，包括 foreground。这样截断结果、失败审计和 `agent_task.get_result` 对两种模式一致。 +4. Background 接纳时预留 completion slot。signal 可以因 inbox 满而失败，已经接纳的 run completion 不得因容量耗尽而丢失。 +5. SQLite 是 run、event、容量计数和消费状态的唯一事实源。watch/broadcast/内存 map 只负责降低延迟或缓存投影。 +6. queue continuation 不伪造 `InboundMessage`。它通过 Session 内部 typed task 启动，并在成功时原子保存 hidden trigger、可见结果和 event consumption。 +7. `steer` 不取消 Provider 或普通工具；它只在安全边界注入。`/stop` 和显式 cancel 才触发 cancellation token。 +8. 候选运行代只解析和校验 Catalog，不扫描或修改数据库运行状态。恢复动作仅在新代 activation 后执行。 +9. 新工具默认 `RootOnly`。只有经过逐项审计的工具可以标记 `Delegatable`；runtime control 工具由 Coordinator 注入。 +10. session 归档/删除、run 取消、event supersede 和 dead-letter 都保留审计事实，不通过物理删除表达状态变化。 + +## 3. 目标运行时组件与依赖装配 + +### 3.1 GatewayState 新增成员 + +建议增加： + +```rust +pub struct GatewayState { + // existing fields ... + agent_catalog: Arc, + agent_coordinator: Arc, + agent_result_router: Arc, + agent_projection_hub: Arc, +} +``` + +装配顺序必须明确： + +```text +Config + registered built-in tools + SkillsLoader + → parse/validate AgentCatalog + → create AgentProjectionHub + → create late-bound AgentInboxNotifier + → create AgentCoordinator(Storage, Catalog, ProviderFactory, tools, Supervisor, Admission) + → create SessionManager(..., notifier) + → bind notifier to Weak + → create AgentResultRouter(Storage, notifier, projection hub) +``` + +`AgentInboxNotifier` 使用 late-bound weak target 解决 Coordinator/SessionManager 的环形依赖： + +```rust +#[async_trait] +pub trait AgentInboxWakeTarget: Send + Sync { + async fn wake_agent_inbox(&self, session_id: &str, revision: i64); +} + +pub struct AgentInboxNotifier { + target: RwLock>, +} +``` + +事件提交后 notifier 失败不回滚事务；周期扫描会重新唤醒。SessionManager 也不能反向持有强 `Arc` 形成释放环。 + +### 3.2 prepare 与 activation 分离 + +`GatewayState::from_config()` 是候选运行代准备阶段，只允许： + +- 解析配置和 Agent Markdown。 +- 校验 Provider profile、工具、skill 和委托图。 +- 创建无外部副作用的内存对象。 +- 运行 Storage schema migration。 + +`start_message_processing()` 成为 activation 边界，按顺序执行： + +1. 连接 MCP；第一版 Agent Definition 不允许引用 MCP 工具，因此此步不改变已经校验完成的 Catalog。 +2. 启动 Session worker/router、projection relay 和 inbox recovery scanner。 +3. 调用 `AgentCoordinator::recover_on_activation(runtime_generation)`，收敛旧代 queued/running/waiting runs。 +4. 扫描 pending/expired leases，修复 `agent_session_state` 计数并发出合并式 wake。 +5. 最后打开新代的 Agent run admission。 + +为避免候选代构造期间与旧代同时修改 run 状态，Coordinator 初始处于 closed 状态，activation 成功后才 `open()`。 + +## 4. 配置与 Agent Definition + +### 4.1 配置类型 + +在 `src/config/mod.rs` 增加： + +```rust +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(default, deny_unknown_fields)] +pub struct AgentOrchestrationConfig { + pub enabled: bool, + pub definitions_dir: String, + pub root_delegates: Vec, + pub max_tree_depth: u16, + pub max_runs_per_tree: usize, + pub max_concurrent_runs: usize, + pub max_concurrent_runs_per_session: usize, + pub max_concurrent_provider_steps: usize, + pub max_concurrent_provider_steps_per_session: usize, + pub max_concurrent_tool_steps: usize, + pub max_concurrent_tool_steps_per_session: usize, + pub max_pending_inbox_events_per_session: usize, + pub inbox_event_ttl_hours: u64, + pub max_inbox_delivery_attempts: u32, + pub max_user_turn_burst_before_inbox: usize, + pub max_inbox_wait_secs: u64, +} +``` + +`Config` 增加 `#[serde(default)] pub agent_orchestration: AgentOrchestrationConfig`。所有默认值必须保持现有配置可加载。旧 `gateway.max_concurrent_background_tasks` 在迁移期只控制 legacy adapter；新实现不复用该字段表达三类不同配额。 + +配置校验需要拒绝 0 容量、session 上限大于 global 上限、TTL/timeout 超过硬上限，以及 definitions 目录越界。配置示例、README 和运行时 config reference 在功能合并时同步更新。 + +热重载允许把 inbox 上限降低到当前占用以下：已有 pending 和 reservation 仍受保护，不删除也不拒绝其 completion；新 background 接纳和 signal 在计数回落到新上限前返回 capacity exceeded。 + +### 4.2 Markdown parser + +新增 `serde_yaml` 依赖，禁止用现有 Skill frontmatter 的宽松字符串 parser 解析安全配置。建议类型： + +```rust +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct AgentFrontmatter { + id: AgentId, + description: String, + llm_profile: String, + #[serde(default)] tools: Vec, + #[serde(default)] delegates: Vec, + #[serde(default)] skills: Vec, + #[serde(default)] limits: AgentLimits, +} +``` + +加载算法： + +1. canonicalize definitions root，确认它是受信任目录。 +2. 只读取 root 第一层的 `*.md`，按文件名排序，保证错误顺序和 hash 稳定。 +3. 对每个文件先用 metadata 检查上限，再做有界读取；拒绝非 UTF-8、越界 symlink 和非普通文件。 +4. frontmatter 必须以第一行 `---` 开始并有独立结束 `---`；正文为空、重复 key、未知 key 均报错。 +5. `id` 使用 `[a-z][a-z0-9_-]{0,63}`，拒绝大小写折叠冲突和保留名。 +6. 对规范化 frontmatter JSON 与原始 role body 计算 SHA-256 `definition_hash`。 +7. 第一遍建立 ID map，第二遍解析 delegate target、Provider、工具和 skill 引用。 + +Catalog 最终类型不可变： + +```rust +pub struct AgentCatalog { + definitions: BTreeMap>, + root_delegates: BTreeSet, + runtime_generation: u64, +} +``` + +第一版 Catalog 只接受候选代准备阶段已经注册的 built-in 工具。现有 MCP 连接只允许在 activation 发生，为维持候选代无外部副作用和“引用错误拒绝整代”的不变量，MCP 工具不得出现在 Agent Definition；未来只有在 MCP 提供可离线校验的 tool manifest 后才能开放。 + +### 4.3 legacy general + +当 orchestration 未配置或旧调用没有 `target` 时，兼容层提供代码内置 `general` definition： + +- Provider 使用当前 root agent profile。 +- 工具只取旧 default list 与 `Delegatable` 的交集。 +- 不能继续 delegate，不能 emit signal。 +- 保留旧 transient browser scope 的兼容行为并返回弃用提示。 + +显式配置的具名 Agent 永远不继承该例外。 + +## 5. ToolRegistry 与执行上下文改造 + +### 5.1 Tool 元数据 + +在 `Tool` trait 增加默认安全元数据： + +```rust +fn delegation_policy(&self) -> DelegationPolicy { + DelegationPolicy::RootOnly +} + +fn input_interrupt_policy(&self) -> InputInterruptPolicy { + InputInterruptPolicy::Never +} +``` + +`DelegationPolicy` 为 `RootOnly | Delegatable | RuntimeInjected`；`InputInterruptPolicy` 为 `Never | WakeOnly | CancelSafe`。二者不能从 Markdown 覆盖。 + +第一轮审计建议： + +| 工具类型 | 初始策略 | 说明 | +|----------|----------|------| +| file/content search、file read | Delegatable | 仍受现有进程文件权限约束，不代表硬 sandbox | +| browser 普通动作 | Delegatable | 具名 run 使用独立 transient resource scope | +| `get_skill` | RuntimeInjected wrapper | 只能读取 definition 的 skill allowlist | +| HTTP request | RootOnly | 当前实现支持写方法；以后可增加 delegated GET-only wrapper | +| bash、file write/edit | RootOnly | 需单独威胁建模后才开放 | +| send_message、todo、cron、reload、browser_profiles、管理工具 | RootOnly | 有外部或全局状态副作用 | +| delegate、agent_task、emit_signal | RuntimeInjected | 依据 caller context 动态注入 | +| sleep | Delegatable + WakeOnly | child 无 TurnWakeupHandle，只响应 timer/cancel | + +`ToolRegistry` 增加只读构建方法，不在共享 registry 上删除工具： + +```rust +pub fn scoped_for_agent( + &self, + definition: &AgentDefinition, + runtime_tools: Vec>, +) -> Result, AgentCatalogError>; +``` + +### 5.2 ToolExecutionContext + +目标类型： + +```rust +#[derive(Clone)] +pub struct ToolExecutionContext { + pub session_id: Option, + pub turn_id: Option, + pub agent: Option>, + pub cancellation: CancellationToken, + pub execution_gate: Option>, + pub turn_wakeup: Option, + pub resource_scope_id: Option, + pub turn_origin: TurnOrigin, +} +``` + +Root interactive context 的 `agent=None`；Coordinator 只允许在 session/turn 身份完整时把它解释成 ROOT。缺少两者的 context 不能调用 delegate。child 的 `agent=Some`，其中 run、parent、ancestry、budget 和 signal contract 是授权事实。 + +不要再依赖 `DELEGATE_CONTEXT` task-local 作授权。它可以暂时保留为旧 adapter 桥接，但所有新工具必须从 `ToolExecutionContext` 读取身份。 + +## 6. AgentLoop：取消、许可与 typed input + +### 6.1 CancellationContext + +给 AgentLoop 的所有 process 入口增加 `AgentLoopExecution`： + +```rust +pub struct AgentLoopExecution { + pub cancellation: CancellationToken, + pub gate: Arc, + pub tool_context: ToolExecutionContext, +} +``` + +Provider 请求必须在获取 provider-step permit 后执行： + +```rust +let _permit = execution.gate.acquire_provider(&execution.cancellation).await?; +let response = tokio::select! { + _ = execution.cancellation.cancelled() => Err(AgentError::Cancelled), + result = self.stream_completion(...) => result, +}; +``` + +普通工具调用同样取得 tool-step permit。runtime-control 工具不取得普通 tool permit，防止父 run 等 child 时占住唯一许可。工具 future 被取消只表示 PicoBot 不再等待；对 bash、HTTP 写入等外部副作用不能宣称已回滚，因此迟到结果必须被 execution ID 条件提交挡住。 + +工具批次当前使用 `join_all`。改造后每项 future 自行获取 permit，批次仍可并发；取消时等待一个很短的 cooperative grace，然后由外层 task abort。`AgentError` 增加结构化 `Cancelled` 和 `TimedOut`，不要用字符串判断终态。 + +### 6.2 ExecutionGate + +配额分成两组： + +- `RunQuota`：统计 queued/running/waiting_children 的非终态 run，直到 terminal commit 才释放。 +- `StepGate`：provider/tool 步骤执行期间持有，步骤结束立即释放。 + +permit 固定按 global → session → agent 顺序获取；失败或取消逆序释放。为了避免动态 semaphore 缓存泄漏，session/agent gate 使用带弱引用的 keyed registry，并在没有 run/permit 时清理。 + +Root Turn 不是 Agent Run，不占 run quota，但它的 Provider/tool 步骤占 global/session step gate，这样 background run 不会绕过整个 Gateway 的资源上限。 + +### 6.3 waiting_children + +`delegate` foreground 执行前创建 `WaitingChildrenGuard`： + +1. child 接纳成功后，条件更新 parent `running → waiting_children`。 +2. 等待期间父没有 provider/tool step permit。 +3. 全部 child 终态、父取消或错误退出时 guard 尝试 `waiting_children → running`；父已经 terminal 时不覆盖。 +4. 父取消递归取消未终态 foreground descendants；background run 归 root session token 所有，不因创建它的 Turn 正常结束而取消。 + +### 6.4 typed Turn input + +Phase 4 将 `SteeringMailbox` 替换为 `TurnMailbox`。在此之前 Phase 2A 只做 cancellation，不改变用户 steering 行为，降低一次改动的回归面。 + +`AgentLoop::append_steering_messages` 不再强制构造 user source；改为由 serializer 把 typed source 转成 Provider-compatible role，同时保留 durable source metadata。Agent event envelope 明确标注为不可信数据，不能改变 system/tool 权限。 + +## 7. AgentCoordinator API 与状态机 + +建议公开最小接口： + +```rust +pub struct DelegateRequest { + pub mode: ExecutionMode, + pub tasks: Vec, + pub completion_policy: CompletionPolicy, + pub idempotency_key: Option, +} + +impl AgentCoordinator { + pub async fn delegate(&self, caller: CallerContext, request: DelegateRequest) + -> Result; + pub async fn get_run(&self, caller: CallerContext, run_id: &RunId) -> Result; + pub async fn list_runs(&self, caller: CallerContext, query: AgentRunQuery) -> Result<_, _>; + pub async fn get_result(&self, caller: CallerContext, run_id: &RunId) -> Result<_, _>; + pub async fn cancel_run(&self, caller: CallerContext, run_id: &RunId) -> Result<_, _>; + pub async fn emit_signal(&self, context: &AgentExecutionContext, signal: SignalInput) + -> Result; + pub async fn cancel_session(&self, session_id: &str, reason: CancelReason) -> Result<(), _>; + pub async fn recover_on_activation(&self, generation: u64) -> Result; +} +``` + +### 7.1 Background 接纳顺序 + +严格顺序： + +1. 从 context 解析 caller，校验 root session、委托边、ancestry、depth 和 budget。 +2. 解析全部 target definition，派生 deadline、token/run reservation 和 delivery contract。 +3. 取得 RuntimeAdmission activity guard 和内存 run quota reservation；尚未写库前任何失败均直接释放。 +4. Storage 事务条件增加 completion reservation、领取 plan item、插入 group/run queued 记录。 +5. 把 cancellation token 和 execution ID 注册到 Coordinator active map。 +6. 通过 `TaskSupervisor::spawn_graceful` 接纳 runner。 +7. spawn 被拒绝时执行补偿事务：queued → cancelled、释放 completion reservation、回滚/阻塞已领取 plan item；不得向模型返回可用 run ID。 +8. spawn 成功后返回 run/group ID。 + +第一版在步骤 1 强制 `background caller == ROOT`；child 只能 foreground 委托。该限制属于 Coordinator policy，不仅是 delegate schema 提示。以后开放 nested background 时仍必须把 run 归属到原 root session，并重新审查取消所有权和 completion reservation。 + +SQLite commit 与 task spawn 无法成为同一原子操作。崩溃发生在步骤 4 与 6 之间时，会留下旧 generation 的 queued run；activation recovery 必须将其收敛为 `interrupted` 并生成预留过的 completion，不能无限保持 queued。 + +ActivityGuard 必须移动进已接纳的 background runner，直到 terminal transaction 完成后才 drop；不能在 `delegate` 返回时提前释放。Runner 等待 run quota/provider permit 时同样受 deadline 和 cancellation 约束，排队时间计入 run timeout。 + +### 7.2 Foreground 执行 + +Foreground 同样先持久化 run，但不预留 inbox slot，也不创建 completion event。单任务直接 await；批量使用 `FuturesUnordered` 并保存原 request index，最终按请求顺序组装结果。每个 child 有独立取消 token；父取消时全部 token 同时取消。 + +完整结果先写 `agent_runs.result`，tool result 只返回有界 projection。这样无论是否截断，`agent_task.get_result` 都能读取相同事实。 + +### 7.3 terminal commit + +Runner 只返回 `AgentTerminalOutcome`，不能自己发 Channel 或更新 WorkManager： + +```rust +pub enum AgentTerminalOutcome { + Completed { result: String, usage: Usage, tool_calls: u32, iterations: u32 }, + Failed { error: AgentRunError, usage: Option }, + TimedOut { deadline_at: i64 }, + Cancelled { reason: CancelReason }, + Interrupted { reason: String }, +} +``` + +Coordinator 用 `(run_id, execution_id, runtime_generation, nonterminal status)` 条件提交。只有更新行数为 1 的赢家可以： + +- 写终态和完整 result/error。 +- 更新 group terminal counter。 +- 转换或释放 completion reservation。 +- 插入 run/group inbox event。 +- 更新绑定的 task item。 +- 递增 session agent revision。 + +迟到结果返回 `StaleExecution` 并丢弃，不改变已保存终态。 + +Coordinator 还需启动一个由 TaskSupervisor 管理、观察 shutdown token 的 deadline reaper。正常 runner 使用 `timeout_at(deadline)`；reaper 负责没有活跃内存 task 的 queued run 和 group deadline，分批条件终态化未完成 children，再走同一个 terminal transaction，不能维护第二套完成逻辑。 + +## 8. Storage schema v6 + +实现时把 `SCHEMA_VERSION` 从 5 升到 6。表创建、旧列增加、索引和 `PRAGMA user_version=6` 必须在当前 `migrate_schema()` 的同一事务中完成。下面的 `ALTER TABLE` 表达目标列；实际代码必须同时更新 fresh-schema 的 `CREATE TABLE`，迁移时沿用现有 `PRAGMA table_info` 检查，仅对缺失列执行 ALTER，不能在 fresh database 上重复加列。 + +SQLite 使用 INTEGER 表示布尔值；应用层枚举仍用 Rust enum，读到未知值要报 corruption，不能静默当成 completed。 + +### 8.1 session 与 message 扩展 + +```sql +ALTER TABLE sessions ADD COLUMN delivery_context TEXT; +ALTER TABLE sessions ADD COLUMN delivery_context_updated_at INTEGER; + +ALTER TABLE messages +ADD COLUMN client_visibility TEXT NOT NULL DEFAULT 'visible'; +ALTER TABLE messages +ADD COLUMN turn_origin TEXT NOT NULL DEFAULT 'user'; + +CREATE INDEX IF NOT EXISTS idx_messages_session_visibility_seq +ON messages(session_id, client_visibility, seq); +``` + +`delivery_context` 只序列化 `ChannelContext.durable_private`。现有 `reply_to` 和 `private` 永不写入该字段。 + +### 8.2 agent_session_state + +```sql +CREATE TABLE IF NOT EXISTS agent_session_state ( + root_session_id TEXT PRIMARY KEY, + revision INTEGER NOT NULL DEFAULT 0, + pending_event_count INTEGER NOT NULL DEFAULT 0, + reserved_completion_slots INTEGER NOT NULL DEFAULT 0, + updated_at INTEGER NOT NULL, + CHECK (revision >= 0), + CHECK (pending_event_count >= 0), + CHECK (reserved_completion_slots >= 0) +); +``` + +这是 inbox 容量和客户端 revision 的权威计数行。每个改变客户端 run/group/event 投影的事务只分配一个新 revision，并把同一 revision 写回本事务改变的所有行；仅内部读取不递增。所有容量增加操作使用条件更新： + +```sql +UPDATE agent_session_state +SET reserved_completion_slots = reserved_completion_slots + ?, + revision = revision + 1, + updated_at = ? +WHERE root_session_id = ? + AND pending_event_count + reserved_completion_slots + ? <= ? +RETURNING revision; +``` + +若 session 尚无行，先 `INSERT ... ON CONFLICT DO NOTHING`，仍在同一写事务中执行条件更新。 + +### 8.3 agent_run_groups + +```sql +CREATE TABLE IF NOT EXISTS agent_run_groups ( + id TEXT PRIMARY KEY, + root_session_id TEXT NOT NULL, + caller_run_id TEXT, + caller_scope_id TEXT NOT NULL, + idempotency_key TEXT, + mode TEXT NOT NULL, + completion_policy TEXT NOT NULL, + expected_runs INTEGER NOT NULL, + terminal_runs INTEGER NOT NULL DEFAULT 0, + abnormal_runs INTEGER NOT NULL DEFAULT 0, + completion_slot_reserved INTEGER NOT NULL DEFAULT 0, + completion_delivery TEXT, + failure_delivery TEXT, + deadline_at INTEGER NOT NULL, + status TEXT NOT NULL, + runtime_generation INTEGER NOT NULL, + revision INTEGER NOT NULL, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + finished_at INTEGER, + CHECK (mode IN ('foreground', 'background')), + CHECK (completion_policy IN ('all', 'each')), + CHECK (status IN ('queued', 'running', 'completed', 'partial', 'failed', + 'timed_out', 'cancelled', 'interrupted')), + CHECK (expected_runs > 0), + CHECK (terminal_runs >= 0 AND terminal_runs <= expected_runs), + CHECK (completion_slot_reserved IN (0, 1)) +); + +CREATE INDEX IF NOT EXISTS idx_agent_groups_session_created +ON agent_run_groups(root_session_id, created_at DESC); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_agent_groups_idempotency +ON agent_run_groups(root_session_id, caller_scope_id, idempotency_key) +WHERE idempotency_key IS NOT NULL; +``` + +批量请求把 request idempotency key 写在 group，所有 child run 的 `idempotency_key` 为 NULL；单任务请求不创建只含一个 child 的 group，key 直接写在 run。这样 retry 能返回原 group/run，又不会让同一批 children 触发唯一索引冲突。 + +### 8.4 agent_runs + +```sql +CREATE TABLE IF NOT EXISTS agent_runs ( + id TEXT PRIMARY KEY, + group_id TEXT, + root_session_id TEXT NOT NULL, + root_turn_id TEXT, + parent_run_id TEXT, + caller_agent_id TEXT NOT NULL, + caller_scope_id TEXT NOT NULL, + idempotency_key TEXT, + agent_id TEXT NOT NULL, + definition_hash TEXT NOT NULL, + provider_profile TEXT NOT NULL, + provider_name TEXT NOT NULL, + model_id TEXT NOT NULL, + mode TEXT NOT NULL, + depth INTEGER NOT NULL, + plan_item_id TEXT, + execution_id TEXT NOT NULL, + task TEXT NOT NULL, + context_json TEXT, + budget_json TEXT NOT NULL, + signal_contract_json TEXT, + signal_delivery TEXT, + completion_delivery TEXT, + failure_delivery TEXT, + status TEXT NOT NULL, + result TEXT, + error TEXT, + prompt_tokens INTEGER, + completion_tokens INTEGER, + cost REAL, + tool_calls_count INTEGER NOT NULL DEFAULT 0, + iterations INTEGER NOT NULL DEFAULT 0, + runtime_generation INTEGER NOT NULL, + attempt INTEGER NOT NULL DEFAULT 1, + completion_slot_reserved INTEGER NOT NULL DEFAULT 0, + deadline_at INTEGER NOT NULL, + revision INTEGER NOT NULL, + started_at INTEGER, + finished_at INTEGER, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + CHECK (mode IN ('foreground', 'background')), + CHECK (status IN ('queued', 'running', 'waiting_children', 'completed', + 'failed', 'timed_out', 'cancelled', 'interrupted')), + CHECK (depth >= 1), + CHECK (completion_slot_reserved IN (0, 1)), + FOREIGN KEY (group_id) REFERENCES agent_run_groups(id) ON DELETE RESTRICT, + FOREIGN KEY (parent_run_id) REFERENCES agent_runs(id) ON DELETE RESTRICT +); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_agent_runs_execution +ON agent_runs(execution_id); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_agent_runs_idempotency +ON agent_runs(root_session_id, caller_scope_id, idempotency_key) +WHERE idempotency_key IS NOT NULL; + +CREATE INDEX IF NOT EXISTS idx_agent_runs_session_created +ON agent_runs(root_session_id, created_at DESC); + +CREATE INDEX IF NOT EXISTS idx_agent_runs_parent +ON agent_runs(parent_run_id, created_at); + +CREATE INDEX IF NOT EXISTS idx_agent_runs_recovery +ON agent_runs(runtime_generation, status, deadline_at); +``` + +Session 使用软删除，agent 表不对 `root_session_id` 建级联外键；生命周期由 `cancel_session` 和恢复扫描显式收敛。 + +### 8.5 agent_inbox_events + +```sql +CREATE TABLE IF NOT EXISTS agent_inbox_events ( + id TEXT PRIMARY KEY, + root_session_id TEXT NOT NULL, + scope_kind TEXT NOT NULL, + scope_id TEXT NOT NULL, + run_id TEXT, + group_id TEXT, + event_type TEXT NOT NULL, + event_key TEXT NOT NULL, + delivery TEXT NOT NULL, + requires_continuation INTEGER NOT NULL DEFAULT 1, + severity TEXT, + payload_json TEXT NOT NULL, + status TEXT NOT NULL, + attempt_count INTEGER NOT NULL DEFAULT 0, + lease_token TEXT, + lease_until INTEGER, + next_attempt_at INTEGER, + admitted_turn_id TEXT, + last_error TEXT, + revision INTEGER NOT NULL, + created_at INTEGER NOT NULL, + consumed_at INTEGER, + superseded_at INTEGER, + dead_lettered_at INTEGER, + fallback_notified_at INTEGER, + fallback_suppressed_reason TEXT, + CHECK (scope_kind IN ('run', 'group')), + CHECK (event_type IN ('signal', 'completion', 'group_completion')), + CHECK (delivery IN ('queue', 'steer')), + CHECK (requires_continuation IN (0, 1)), + CHECK (status IN ('pending', 'leased', 'admitted', 'consumed', + 'superseded', 'dead_letter')), + CHECK ( + (scope_kind = 'run' AND run_id IS NOT NULL AND group_id IS NULL + AND scope_id = run_id) OR + (scope_kind = 'group' AND group_id IS NOT NULL AND run_id IS NULL + AND scope_id = group_id) + ), + UNIQUE(scope_kind, scope_id, event_type, event_key), + FOREIGN KEY (run_id) REFERENCES agent_runs(id) ON DELETE RESTRICT, + FOREIGN KEY (group_id) REFERENCES agent_run_groups(id) ON DELETE RESTRICT +); + +CREATE INDEX IF NOT EXISTS idx_agent_inbox_claim +ON agent_inbox_events(root_session_id, status, next_attempt_at, created_at); + +CREATE INDEX IF NOT EXISTS idx_agent_inbox_lease +ON agent_inbox_events(status, lease_until); + +CREATE INDEX IF NOT EXISTS idx_agent_inbox_revision +ON agent_inbox_events(root_session_id, revision); +``` + +run/group 外键使用 RESTRICT。清理默认只清理大正文或归档整组记录；若未来需要物理删除，必须先按明确保留策略删除 terminal inbox event，再删除 run/group，不能留下失去审计来源的事件。 + +### 8.6 事务 API + +不要把 SQL 分散在 Coordinator、Router 和 WorkManager。Storage 增加领域 API： + +```rust +accept_agent_group(request) -> AcceptedGroup +mark_agent_run_running(run_id, execution_id) +mark_agent_run_waiting_children(run_id, execution_id, expected_status) +commit_agent_terminal(run_id, execution_id, outcome) -> TerminalCommit +insert_agent_signal(run_id, execution_id, signal) -> EventCommit +claim_inbox_batch(session_id, now, lease) -> InboxLease +admit_inbox_event(event_id, lease_token, turn_id) +release_inbox_lease(event_ids, lease_token, retry) +commit_continuation_turn(batch) -> CommittedTurnDelta +recover_agent_state(active_generation, now) -> RecoveryReport +``` + +`commit_agent_terminal` 在一个事务内更新 run/group、容量计数、event、plan item 和 revision。提交后 Coordinator 调用 `WorkManager::refresh_after_external_commit()` 刷新 cache 并广播；WorkManager 不再为这条路径另开事务。 + +## 9. Agent Inbox、唤醒与公平调度 + +### 9.1 Session worker 接收面 + +Session 增加： + +```rust +agent_tx: Option>, +agent_inbox_wake: watch::Sender, +consecutive_user_turns: usize, +``` + +把 worker 创建逻辑抽为 `ensure_agent_worker_locked()`，用户 enqueue 与 `wake_agent_inbox()` 共用。worker 同时持有 user receiver 和 wake receiver： + +```rust +tokio::select! { + biased; + user = task_rx.recv(), if should_take_user => { ... } + changed = inbox_wake.changed() => { ... } +} +``` + +不能仅靠 `biased` 实现公平。每个 Turn 结束后查询最老 due pending event；满足以下任一条件时下一项必须是 continuation： + +- `consecutive_user_turns >= max_user_turn_burst_before_inbox`； +- oldest pending age >= `max_inbox_wait_secs`。 + +watch value 是最新 durable revision，只合并 wake，不携带 payload。worker 收到 wake 后从 SQLite claim；发送方从不等待 Session user mpsc 容量。 + +### 9.2 queue continuation + +claim batch 必须有大小与总字节上限，例如 8 events/32 KiB envelope。`completion_policy=each` 可在 300–500ms 内 debounce 已经 pending 的 siblings，但不能等待未终态 sibling。 + +执行过程： + +1. Storage 把 due events `pending → leased`，返回 lease token。 +2. worker 构造 `AgentTaskSource::BackgroundAgentResults` 和 hidden trigger，但暂不落 messages。 +3. root Agent 使用只读 continuation ToolRegistry 执行。默认允许内容读取、检索和 scoped `agent_task.get/get_result`，不允许 send_message、todo、写文件、delegate 或其他外部副作用。 +4. 成功后 `commit_continuation_turn` 原子插入 hidden trigger、tool/assistant messages、usage，并把同 token events `leased/admitted → consumed`。 +5. commit 后更新 Session 内存、Turn Completed、projection 和 Channel delivery。 +6. 执行失败、取消或 stale generation 时 `InboxLeaseGuard` 显式 release 到 pending 并写 `next_attempt_at`。 + +这一流程与当前“先持久化用户消息再调用模型”不同，必须走独立分支。不能先写 hidden trigger，否则失败重试会在 history 中累积无 assistant 配对的内部输入。 + +### 9.3 hidden history + +`ChatMessage` 和 `MessageMeta` 增加： + +```rust +pub client_visibility: ClientVisibility, +pub turn_origin: TurnOrigin, +``` + +默认构造器生成 `Visible/User`。需要拆分查询： + +- `load_messages_for_replay(session_id)`：visible + hidden。 +- `load_visible_messages(session_id, ...)`：WebSocket/HTTP/管理历史。 +- `persist_message_batch_inner()`：支持两字段并保持向后兼容默认值。 +- `CommittedTurnDelta`：过滤 hidden，并携带 turn origin。 + +Session 的 `message_count` 只统计 visible external user input；`total_message_count` 和压缩/token 估算统计全部 replay messages。自动标题只由 visible user input 触发。 + +### 9.4 steer admission + +Router 的两阶段流程必须由测试锁定： + +```text +pending --claim(token)--> leased +leased --reserve current Turn--> memory reservation (not drainable) +leased --DB admit(token, turn)--> admitted +admitted --activate same turn/generation--> drainable TurnInput +``` + +任何失败都取消 reservation，并以 token 把 event 恢复 pending。若 active Turn 已关闭、mailbox 满或 generation 改变，直接释放 lease并 wake queue；这就是可靠 steer→queue fallback。 + +TurnMailbox 分 user lane 与 agent lane 容量，最终按 session sequence 合并排空。durable agent reservation 在 DB admit 前不能被 AgentLoop 看见；`/stop` 关闭 mailbox 时返回尚未提交的 event IDs/tokens，由 SessionManager 立即 release。 + +## 10. emit_signal 与 completion 投影 + +`EmitSignalTool` 只注册到携带 background signal contract 的 runner。foreground run、Root 或没有 contract 的 child 看不到该工具。 + +`emit_signal` 流程： + +1. 从 AgentExecutionContext 取得唯一 root session、run、execution ID 和 delivery。 +2. 校验 run 仍为 running/waiting_children 且 execution ID 匹配。 +3. 校验 severity、JSON 深度、summary/details 字节、信号总数、burst、最小间隔和 dedupe window。 +4. 条件增加 session pending count；容量必须扣除 reserved completion slots。 +5. 插入 event 和新 revision；commit 后发布 projection + wake。 +6. 返回 accepted event ID。事件已存在时返回同一 ID 和 `deduplicated=true`。 + +Completion 永远由 Coordinator 生成。`completion_policy=all` 时每个 run terminal outcome 只存在于 `agent_runs`，不生成 per-run inbox completion；最后一个 terminal child 或 group deadline 的事务赢家生成唯一 group completion。group 有任一 failed/timed_out/interrupted/cancelled 时使用 `failure_delivery`，否则使用 `completion_delivery`。 + +显式 `agent_task.cancel` 可以把该 run 尚未消费的普通 signal 更新为 `superseded` 并减少 pending count；completion 事实不能 supersede。 + +## 11. Session 生命周期、停止与恢复 + +### 11.1 `/stop` + +执行顺序： + +1. 在 Session 锁内关闭 active Turn admission、取出 mailbox durable reservations、递增 worker generation、取出 root Turn cancellation token。 +2. 锁外取消 root token，条件 release admitted/leased inbox events。 +3. 调用 Coordinator cancel 当前 session 的 active background runs。 +4. cancellation terminal event 使用 `requires_continuation=false` 并直接 consumed，防止 stop 后又自动启动“已取消”Turn。 +5. stop 前已经 pending 的其他事件仍保持 pending；Session 下次可调度时处理。 + +现有 oneshot 可在过渡期由 token adapter 驱动，最终移除 `current_cancel: Option>`，统一为 `CancellationToken`。 + +### 11.2 archive/delete + +Session 归档或软删除必须调用 `AgentCoordinator::cancel_session`。提交事务： + +- 非 Scheduler 所有且未终态 run → cancelled。 +- 释放所有 completion reservation。 +- pending/leased/admitted events → `dead_letter(session_archived|session_deleted)`。 +- 清理 mailbox reservation，不启动 continuation。 + +客户端仍可通过管理面看到 run/event 审计。当前没有可靠 unarchive 投递语义，因此归档事件不保留 pending。归档/删除属于用户生命周期操作，写 `fallback_suppressed_reason` 并禁止 system fallback，避免用户删除对话后又收到后台通知。 + +### 11.3 restart/reload recovery + +activation recovery 分批执行，避免长事务： + +1. 将旧 generation 的 queued/running/waiting_children 条件更新为 interrupted。 +2. 为 background interrupted outcome 转换预留并插入 failure completion；foreground 只保存终态。 +3. group counter 收敛并生成必要的 group completion。 +4. expired leased/admitted events 恢复 pending，attempt +1，写 next retry。 +5. 按每 session 重算 `pending_event_count` 和有效 reservation；差异修复并记录 structured warning。 +6. 对有 pending due events 的 session 只发一次合并 wake。 + +旧代排空期间每个 run 持有 RuntimeAdmission activity guard。超出 reload grace 后 cancellation token 先触发；Supervisor 强制 abort 后留下的非终态记录由新代 recovery 标 interrupted。 + +## 12. ChannelContext 与结果投递 + +当前 `ChannelContext.private` 同时可能含稳定 thread/root ID 和一次性 message/reaction ID，核心无法安全猜测。类型改为： + +```rust +pub struct ChannelContext { + pub reply_to: Option, + pub private: HashMap, + pub durable_private: HashMap, +} +``` + +Channel adapter 负责分类： + +- CLI/WebUI：通常为空。 +- Feishu：thread/root/chat type 等是否可复用由 Feishu adapter 决定；message ID、reaction ID 和本次 reply target 留在 `private/reply_to`。 + +Session 在成功接纳外部用户输入时保存 `durable_private`。continuation 的 `TurnTarget` 使用 session 自身 channel/chat ID 和 durable context，`reply_to=None`。没有 binding 时照常提交历史和 WebSocket 投影，只跳过外部 Channel delivery。 + +## 13. WorkManager 原子性 + +当前 `assign_sub_agent` 和 `finish_sub_agent` 各自开事务，无法与 run 接纳/终态保持原子。实施方案： + +- 把 plan item 条件 SQL 移入 Storage 的 agent run transaction。 +- run 接纳时按 `(plan_id, item_id, status=pending)` 领取，并把 `execution_id` 写为 run execution ID。 +- terminal commit 时按同 execution ID 更新 completed/blocked 和 plan version。 +- WorkManager 新增 `refresh_after_external_commit(session_id, reason, item_ids)`，只负责重新查询、刷新 cache 和广播，不写数据库。 +- 若 plan item 已被其他执行领取，整个 run 接纳事务回滚；不得创建一个与计划脱节的 run。 + +## 14. 协议、管理 API 与 WebUI + +### 14.1 Rust protocol + +增加 DTO 而不直接序列化 Storage row： + +```text +WsInbound::GetAgentRuns { session_id, cursor, limit } +WsInbound::GetAgentRun { session_id, run_id } + +WsOutbound::SessionAgentRuns { session_id, revision, runs, next_cursor } +WsOutbound::AgentRunUpdated { session_id, revision, run } +WsOutbound::AgentEventUpdated { session_id, revision, event } +``` + +`TurnSnapshot`、`CommittedTurnDelta`、`turn_updated` 和 `turn_committed` 增加 `turn_origin`。DTO 不暴露 task/result 全文以外的敏感 context、reasoning、Provider state、budget internals 或 delivery context。 + +`AgentProjectionHub` 复用 WorkManager 的 broadcast 模式。广播 lag 不补逐条事件，客户端收到 lag 或重连后调用 `GetAgentRuns` 校准。revision 来自 `agent_session_state`，客户端只接受更大的 revision；分页 cursor 使用 `(created_at,id)`，不能用 offset。 + +### 14.2 HTTP 管理 API + +过渡期 `/api/tasks` 返回统一 projection： + +- 新 `agent_runs` 映射为 `source=agent_run`。 +- 旧 `background_tasks` 映射为 `source=legacy_background_task`。 +- 按 created_at 合并排序，旧记录只读。 + +新增受设备鉴权保护的： + +```text +GET /api/agent-runs?session_id=&cursor=&limit= +GET /api/agent-runs/:id +GET /api/agent-runs/:id/events +POST /api/agent-runs/:id/cancel +``` + +cancel endpoint 仍通过 Coordinator 做 root session/task-tree 授权，不直接执行 SQL。 + +### 14.3 WebUI + +现有 `TasksPage.svelte` 的“后台任务”tab 演进为运行树： + +- group 可折叠展示 children。 +- status、Agent、Provider/model、duration、usage、工具次数。 +- signal 数量、最近 severity、pending/dead-letter 标记。 +- terminal result 有界预览，完整内容按需加载。 +- cancel 只对允许取消的非终态 run 显示。 + +ChatPage 对 `turn_origin=agent_continuation` 显示轻量标签，不创建 user bubble；Signal 卡片来自 `AgentEventUpdated`，不能把 payload 插入普通聊天 history。所有新 Markdown/result 仍经过现有 sanitize 流程。 + +## 15. 可唤醒 sleep 的代码实现 + +`TurnWakeupHandle` 只由 root interactive Turn 创建并放入 ToolExecutionContext。它持有 `watch::Receiver`；state 使用独立于 durable inbox revision 的 session-local 单调 revision，因为用户 queue/steer 同样需要唤醒： + +```rust +pub struct TurnWakeupState { + pub revision: u64, + pub pending_user_steer: usize, + pub pending_user_queue: usize, + pub pending_agent_steer: usize, + pub pending_agent_queue: usize, + pub latest_source: WakeupSource, + pub latest_safe_preview: Option, +} +``` + +以下 admission 成功后递增 revision 并 `send_replace`：用户进入 TurnMailbox、用户进入 next-turn mpsc、Agent event durable admitted、Agent event 保持 pending 并成功 wake Session。先更新事实状态再发 wake，不能让 sleep 醒来却查询不到输入。 + +`SleepTool::execute_with_context`： + +1. 解析并限制 duration 到 24 小时。 +2. child context 的 `turn_wakeup=None`，只 select timer 与 cancellation。 +3. root context 先读取 `receiver.borrow_and_update()`；若 pending 总数已大于 0，立即返回，不进入等待。 +4. 否则同时等待 timer、`receiver.changed()` 和 cancellation。watch 保留最新 revision，因此 input 在检查与 select 之间到达也不会丢。 +5. wake 后再次读取 state 并构造有界结果。steer 可以返回来源、run ID 和安全摘要；queue 只返回类型/数量，不返回正文。 +6. cancellation 映射为统一 `AgentError::Cancelled`，不能让模型把它当普通 sleep 完成后继续执行。 + +Sleep 只提前结束工具 future，不消费 mailbox/inbox，也不自行改变 queue/steer。工具批次返回后，AgentLoop 在既定安全边界排空 steer；queue 留给下一个 Turn。 + +第一版 `InputInterruptPolicy` 只驱动 sleep 的 `WakeOnly`。不要顺带让 bash、browser、HTTP 或文件工具响应 input wake；以后开放 `CancelSafe` 必须逐工具证明取消、重试和副作用语义。 + +## 16. 分阶段实施清单 + +### 文件级落点 + +| 文件/模块 | 主要改动 | +|-----------|----------| +| `src/config/mod.rs` | orchestration config、默认值、边界校验 | +| `src/agent/definition.rs`（新） | Markdown/frontmatter 类型、严格 parser、definition hash | +| `src/agent/catalog.rs`（新） | immutable catalog、委托图与引用校验 | +| `src/agent/run.rs`（新） | run/group/outcome DTO、AgentRunner、ProviderFactory | +| `src/agent/coordinator.rs`（新） | 授权、预算、接纳、取消、deadline、terminal commit | +| `src/agent/inbox.rs`（新） | event/lease/projection/notifier contracts | +| `src/agent/agent_loop.rs` | cancellation、step gate、typed input safe boundaries | +| `src/agent/steering.rs` | 迁移为 typed TurnMailbox 与 reservation | +| `src/agent/sub_agent.rs` | legacy adapter，逐阶段缩减并最终删除旧 manager | +| `src/tools/traits.rs` | ToolExecutionContext、DelegationPolicy、interrupt policy | +| `src/tools/delegate.rs` | 仅 run/run_many 和兼容参数转换 | +| `src/tools/agent_task.rs`（新） | scoped get/list/cancel/get_result | +| `src/tools/emit_signal.rs`（新） | contract-bound signal | +| `src/tools/sleep.rs` | watch-aware wait 和 cancellation | +| `src/session/session.rs` | dual lane worker、internal task、atomic continuation、stop release | +| `src/session/agent_inbox.rs`（新） | claim/admit/release、fair scheduler、lease guard | +| `src/storage/mod.rs` | schema v6 migration 和领域 API re-export | +| `src/storage/agent_run.rs`（新） | run/group transaction SQL | +| `src/storage/agent_inbox.rs`（新） | capacity/event/lease/recovery SQL | +| `src/storage/message.rs`、`session.rs` | visibility/origin/durable delivery context | +| `src/gateway/mod.rs`、`reload.rs` | prepare/activation、recovery task、projection relay | +| `src/protocol.rs`、`src/session/commands.rs` | run query/event DTO 和 Turn origin | +| `src/gateway/http.rs` | 管理 API、legacy/new projection union | +| `webui/src/pages/TasksPage.svelte` | run tree、event/dead-letter/cancel UI | +| `webui/src/pages/ChatPage.svelte` | continuation origin 和 Signal 卡片 | + +不要让 `session.rs` 继续吸收所有 inbox SQL 和状态机；先抽 Storage 领域 API 与 `session/agent_inbox.rs`，再接 worker，能显著降低竞态代码的审查难度。 + +### Phase 0：契约与 schema 冻结 + +- 固定本文 enums、状态转换、DDL 和默认值。 +- 给现有 delegate、background notification、Session stop/reload 行为补基线测试。 +- 为 config 示例准备 backward-compatible fixture。 + +完成条件：所有后续 PR 可以只引用固定 DTO/Storage contract，不再各自发明状态名。 + +### Phase 1：Catalog 与安全裁剪 + +- `definition.rs`、`catalog.rs`、严格 YAML parser。 +- Config 扩展与候选代校验。 +- Tool delegation metadata、scoped registry、skill wrapper。 +- `foreground/background` schema 和 legacy parameter adapter。 +- built-in general compatibility。 + +完成条件：具名 foreground Agent 能使用不同 Provider 和固定工具集；尚不开放 background 新路径。 + +### Phase 2A：AgentLoop 结构化取消 + +- CancellationToken 进入 root Turn 和 AgentLoop。 +- Provider stream、工具批次、sleep 观察 token。 +- `AgentError::Cancelled/TimedOut`。 +- `/stop` oneshot 兼容桥接和迟到结果测试。 + +完成条件：root 行为不变；Provider、tool、preparation 各阶段 stop 均得到一致 Cancelled 终态。 + +### Phase 2B：Coordinator、run persistence 与 execution gate + +- schema v6 的 run/group 部分。 +- ProviderFactory、AgentRunner、Coordinator。 +- run quota/provider gate/tool gate。 +- foreground 单/批量、parent waiting_children、agent_task。 +- WorkManager 原子绑定。 + +完成条件：foreground 全量持久化、截断结果可查询、并发上限为 1 时嵌套 foreground 不死锁。 + +### Phase 3：durable inbox 与 queue continuation + +- `agent_session_state`、inbox Storage API、容量 reservation。 +- late-bound notifier、Session watch wake、公平调度。 +- hidden messages、turn origin、atomic continuation commit。 +- recovery/dead-letter/fallback。 +- WebSocket/HTTP run projection 和 WebUI 基础运行树。 + +完成条件：background completion 不直接通知 Channel；空闲/忙碌/队列满/重启/reload 下最终都能被主 Agent处理或进入可诊断 dead-letter。 + +### Phase 4：signal 与 steer + +- EmitSignalTool、rate/dedupe/capacity。 +- typed TurnMailbox 和两阶段 steer admission。 +- `/stop` 立即 release、supersede 和 Signal UI。 + +完成条件：steer 在所有 admission 竞态中严格属于当前 Turn 或未来 queue 之一，不能重复或消失。 + +### Phase 5：wake-aware sleep 与清理 + +- TurnWakeupHandle watch revision。 +- root sleep 响应 user/agent queue/steer，child sleep 仅 timer/cancel。 +- 工具 interrupt policy。 +- 移除 direct background notification 默认路径；观察一个版本后删除旧 adapter 和旧表写入。 + +完成条件：sleep 不丢 pre-listen wake，queue 内容不泄漏进当前 Turn，steer 在下一个安全边界可见。 + +## 17. 测试与故障注入 + +除主设计测试矩阵外，实施必须增加以下代码级场景。 + +### 17.1 Storage + +- fresh database 创建 schema v6。 +- v5 fixture 升级到 v6，原消息默认 visible/user。 +- migration 任一步失败时 `user_version` 和全部表结构回滚。 +- 两个并发 background 接纳只有一个能占最后 completion slot。 +- completion 把 reservation 转 pending，不出现中间负数或超限。 +- terminal transaction 在 plan update/event insert/usage 任一点失败时全回滚。 +- execution ID 不匹配的迟到结果更新 0 行。 +- startup reconciliation 能修复故意破坏的 state counters。 + +### 17.2 AgentLoop/Coordinator + +- cancellation 发生在 permit wait、Provider connect、stream、工具批次、child wait。 +- parent 等 child 时不持 step permit；global provider permit=1 仍能完成。 +- foreground batch 并发执行但结果按 request index 返回。 +- parent cancel 取消 foreground descendants，不取消已独立接纳的 background run。 +- TaskSupervisor spawn reject 执行补偿事务且 delegate 不返回可用 ID。 +- crash window 留下 queued old-generation run，activation 恢复为 interrupted。 + +### 17.3 Inbox/Session + +- watch wake 在 worker 开始等待前发生也不会丢。 +- user mpsc 满不影响 durable event。 +- 连续用户输入达到 burst 上限后强制执行一个 event batch。 +- continuation Provider 失败显式 release；进程模拟崩溃后 lease expiry 恢复。 +- hidden trigger 与 assistant/event consume 原子；客户端 history 永远不返回 hidden。 +- hidden user 不增加 title threshold/message_count。 +- `/stop` 与 admit 的每个交错点都只产生 pending 或 consumed/admitted ownership之一。 +- archived/deleted session 不启动 continuation且 reservation 归零。 + +### 17.4 Channel/protocol/UI + +- durable context round trip 不包含 reply_to、message/reaction ID。 +- 无 delivery binding 时 history commit 成功且不发送其他 chat。 +- event broadcast lag/reconnect 后 full query 校准 revision。 +- legacy/new tasks union 不重复并稳定分页。 +- Signal payload、result Markdown 和错误信息通过 sanitize/redaction。 + +### 17.5 Sleep + +- input 在 sleep 读取 revision 前、读取后但 select 前、select 后三个时点到达都能提前结束。 +- user/agent 的 queue wake 只返回数量，不泄漏正文；对应输入仍由下一 Turn处理。 +- steer wake 返回后由 AgentLoop 安全边界注入，SleepTool 自身不消费输入。 +- child sleep 不因 root user input 或 sibling signal 唤醒，但 run cancel/timeout 会立即结束。 +- duration 超过 24 小时拒绝；Gateway shutdown 不留下未托管等待任务。 + +建议把 SQLite failpoint、barrier 和暂停钩子限制在 `#[cfg(test)]`，用于精确制造 admission、stop、commit 与 reload 竞态，避免依赖随机 sleep。 + +## 18. 验证命令与合并门槛 + +每个 Rust 阶段至少执行： + +```bash +cargo fmt --check +cargo test --lib +cargo test --test test_scheduler +cargo test --test test_request_format +cargo clippy --all-targets --all-features -- -D warnings +cargo build +``` + +涉及 WebUI 时另外执行： + +```bash +cd webui +npm ci +npm run check +npm run build +cd .. +cargo build +``` + +文档或配置变更执行 `git diff --check`，并核对 README、配置模板、ARCHITECTURE、AGENTS 和 runtime knowledge 是否需要同步。API-backed ignored tests 只在配置了真实凭据时运行。 + +功能首次合并时按仓库规则提升产品中段版本；本文档本身不改变行为，不单独修改 `1.5.1`（当前功能合并版本 1.7.0）。 + +## 19. 主要风险与回滚 + +| 风险 | 预防 | 回滚方式 | +|------|------|----------| +| hidden history 过滤错误泄漏内部输入 | 查询 API 分层和协议测试 | 禁用 continuation worker，保留事件 pending | +| completion 容量计数漂移 | 单行条件更新 + 启动重算 | reconcile 修复后重启 Router | +| steer admission 竞态重复 | lease token + 不可排空 reservation | 配置强制所有 Agent delivery=queue | +| background 新路径影响现有用户 | legacy adapter 与新表分离 | 一个版本内启用 direct-notification rollback flag，二者互斥 | +| reload 双代同时恢复 | prepare/activation 硬边界 | 关闭新代 admission，旧代继续服务 | +| 工具权限误放大 | default RootOnly + Catalog fail closed | 从 definition 移除工具并 reload；已启动 run 固定旧快照 | +| continuation 重试重复外部副作用 | 默认只读 registry | dead-letter，要求用户显式处理 | + +rollback flag 只能在 Phase 3 过渡期存在，并保证新 inbox continuation 与旧 direct notification 互斥。即使回滚投递方式，新 `agent_runs`/inbox 表仍保留审计事实，不能降 schema 或删除记录。 + +## 20. 最终实施判据 + +只有同时满足下列条件，才能把主设计状态从“提案”改为“已实现”： + +- 具名 definition、不同 Provider、委托图和工具裁剪全部由运行时代码强制。 +- foreground/background 都有 durable run，batch foreground 真正并发且父同步等待。 +- run quota 与 provider/tool step gate 分离，嵌套委托无 permit 死锁。 +- background completion 和 signal 先落 SQLite，丢 wake、queue 满、重启和 reload 均可恢复。 +- queue continuation 使用 hidden trigger 原子提交；客户端历史没有伪用户消息。 +- steer admission 与 `/stop` 竞态经过故障注入证明无丢失、无双重归属。 +- sleep 的 wake 只改变等待，不破坏 queue/steer 内容边界。 +- session archive/delete、dead-letter、fallback 和 management UI 均可诊断。 +- 全部离线测试、Clippy、build、WebUI check/build 通过，并同步公开文档和产品版本。 + +在达到这些判据前，可以按 Phase 逐步合并内部基础，但不能提前对外宣称已经具备可靠的 background Agent orchestration。 diff --git a/docs/SUB_AGENT_ORCHESTRATION_REVIEW.md b/docs/SUB_AGENT_ORCHESTRATION_REVIEW.md new file mode 100644 index 0000000..985644f --- /dev/null +++ b/docs/SUB_AGENT_ORCHESTRATION_REVIEW.md @@ -0,0 +1,153 @@ +# 子 Agent 编排与信号投递设计审核报告 + +> 状态：审核完成（2026-08）。审核对象为设计提案 `docs/SUB_AGENT_ORCHESTRATION_DESIGN.md`，该设计尚未实现；本文所有"现状"描述以当前代码和测试为准。 +> +> 本文结合现有实现逐条核实设计的现状诊断，评估架构合理性，并按严重程度列出缺陷与落地前必须补齐的定义。行号基于审核时的代码快照，后续实现合并后可能过时。 +> +> 设计方逐项答复见 [`SUB_AGENT_ORCHESTRATION_REVIEW_RESPONSE.md`](SUB_AGENT_ORCHESTRATION_REVIEW_RESPONSE.md)；已接受结论同步写入设计文档。 + +## 1. 审核范围与依据 + +### 1.1 审核对象 + +- 设计文档：`docs/SUB_AGENT_ORCHESTRATION_DESIGN.md`（提案，未实现；`rg` 确认 `src/` 与 `webui/` 中无任何 `AgentCatalog`/`AgentCoordinator`/`agent_inbox`/`emit_signal` 相关实现） +- 对照实现：`src/agent/sub_agent.rs`、`src/agent/agent_loop.rs`、`src/agent/steering.rs`、`src/session/session.rs`、`src/session/turn_input.rs`、`src/session/persistence.rs`、`src/tools/delegate.rs`、`src/tools/sleep.rs`、`src/tools/send_message.rs`、`src/tools/traits.rs`、`src/storage/`、`src/work/mod.rs`、`src/scheduler/mod.rs`、`src/task_supervisor.rs`、`src/config/mod.rs`、`src/gateway/reload.rs` + +### 1.2 审核依据 + +- `docs/ARCHITECTURE.md` 与 AGENTS.md 中的架构边界和并发不变量 +- 现有相似机制：Scheduler durable lease（`src/storage/scheduler.rs:310-348`）、WorkManager 乐观并发（`src/work/mod.rs:320-342`）、TurnController"持久化后才 Completed"（`src/session/persistence.rs:147-167`）、RuntimeAdmission（`src/gateway/reload.rs`） + +## 2. 总体结论 + +**设计方向合理，可以按分期推进；但存在 5 处与现有代码强耦合的接缝缺口（A1–A5），落地前必须先补齐定义，否则 Phase 2/3 会被迫返工。** + +设计的核心决策——`foreground/background` 与 `queue/steer` 两个正交维度、先持久化后唤醒、SQLite inbox 为权威来源、lease/consumed 事务提交、ancestry 环检查、AgentCatalog 绑定运行代——与 PicoBot 既有不变量一致，且现状诊断（设计 §1 的 9 条）逐条属实（见第 3 节）。主要问题不在方向，而在设计与现有 session worker、`/stop`、AgentLoop 取消机制的衔接处留白过多。 + +## 3. 现状诊断核实 + +设计 §1 的 9 条诊断全部与代码一致： + +| # | 设计诊断 | 代码证据 | 结论 | +|---|----------|----------|------| +| 1 | 所有子 Agent 复用同一 `LLMProviderConfig` | `SubAgentManager.provider_config` 单实例（`src/agent/sub_agent.rs:119`），inline/background 均用它创建 Provider（:204、:477） | 属实 | +| 2 | 无具名角色文件，工具权限由 `allowed_tools` 临时决定 | `delegate` schema 的 `allowed_tools` 数组（`src/tools/delegate.rs:50-54`）；未填时用默认只读集（`sub_agent.rs:34-41`） | 属实 | +| 3 | 子 Agent 被统一移除 `delegate` | `filter_tools` 硬编码排除 `delegate`/`todo`/`reload_config`（`sub_agent.rs:177-181`） | 属实 | +| 4 | `DelegateContext` 只有 session/channel/chat | `sub_agent.rs:90-95`；无 caller/parent/depth/ancestry | 属实 | +| 5 | `parallel` 混淆"委托方是否等待"与"是否并发" | `run_parallel` 就是 `join_all(run_inline)`（`sub_agent.rs:332-346`） | 属实 | +| 6 | 后台完成通知直发 `MessageBus.outbound`，不成为主 Agent 输入 | `background-task-notifications` 任务格式化后 `publish_outbound` fire-and-forget（`src/session/session.rs:1757-1776`），不写会话历史、不触发 Turn | 属实 | +| 7 | Steering mailbox 只建模用户输入，且继承 `/stop` 丢弃语义 | admission 只推 `SourceKind::UserInput`（`session.rs:2910-2938`）；AgentLoop 防御性归一 `role=user`（`src/agent/agent_loop.rs:624-628`）；`/stop` 调 `close_and_take_pending` 主动丢弃（`session.rs:2120-2128`、`src/agent/steering.rs:215-229`） | 属实 | +| 8 | `SleepTool` 只等定时器 | `tokio::time::sleep` 单一路径（`src/tools/sleep.rs:72`），无 wakeup/cancel 分支 | 属实 | +| 9 | `send_message` 同时覆盖跨 Channel、目标会话写入和同 Turn 附件暂存 | `src/tools/send_message.rs` 的 target/content/origin/files 参数；`OutboundDelivery::AttachedToCurrentTurn` 同 Turn 分支（`src/tools/traits.rs:127-131`） | 属实 | + +**补充：设计隐含覆盖了一个现存 bug。** inline 结果截断时提示"完整结果请使用 check_task 查看"（`sub_agent.rs:809`），但 `run_inline` 从不写 `background_tasks` 表（只有 `run_background` 写，`sub_agent.rs:373-398`），`check_task`（`sub_agent.rs:707-713`）查不到 inline 结果。设计 Phase 2"Foreground 结果也持久化"（§23）修复此问题，分期安排正确。 + +## 4. 设计合理性评估 + +以下决策予以肯定： + +| 设计点 | 评估 | +|--------|------| +| 执行生命周期与投递方式正交化（§4.1） | ✅ 干净消除 `parallel` 的语义混淆；批量 foreground 并发等价旧 parallel 但不作为第三种模式 | +| 先持久化后唤醒、内存 wakeup 仅为加速器（§6.4、§14.4） | ✅ 与"持久化后才 Completed"既有不变量（`persistence.rs:147-167`）同构 | +| lease/consumed 与条件更新（§14、§16.4） | ✅ 复用 Scheduler durable lease 与 WorkManager 乐观并发的成熟模式 | +| `target not in ancestry` 拒绝 A→B→A（§7.1） | ✅ 以显式 iteration workflow 替代隐式递归，边界正确 | +| 授权不依赖 task-local（§8） | ✅ 正确诊断现状 `DELEGATE_CONTEXT` task-local（`sub_agent.rs:19-28`）不是授权事实来源；`tokio::spawn` 不传播 task-local | +| AgentCatalog 以 Arc 固定运行代（§5.4、§18.4） | ✅ 与 RuntimeAdmission 现有集成一致（`SubAgentManager` 已接 admission，`sub_agent.rs:157-163、353-358`） | +| `llm_profile` 引用现有 `config.agents`（§5.2） | ✅ `Config.agents` 与 `get_provider_config(agent_name)` 已存在（`src/config/mod.rs:45、712-745`），无需配置重构 | +| Completion 由运行时自动生成、不依赖模型记得调工具（§12.1） | ✅ 正确；`emit_signal` 无任意目标参数，收敛了权限面 | +| 可唤醒 sleep 用 watch revision 而非裸 Notify（§17.2） | ✅ 正确规避"输入先于订阅到达"的丢失唤醒竞态 | +| §26 不变量清单 | ✅ 与 ARCHITECTURE.md 一致，可作为实现验收标准 | +| 分期顺序（§23） | ✅ Phase 1 纯增量；Phase 2 顺带修复 inline 截断 bug；依赖方向正确 | + +## 5. 缺陷清单 + +严重程度：A=主要（落地前必须补齐定义）；B=中等（实现对应 Phase 前补齐）；C=次要（修订文档即可）。 + +### 5.1 A 级：主要缺陷 + +**A1 — Session 队列饱和语义与现状冲突（设计 §15.3）** + +现状：session 队列容量 32（`session.rs:27`），满时 `try_send` 失败直接丢弃输入并回复"队列已满"（`session.rs:3001-3007`）。设计要求 durable event 在队列饱和时"保持 durable pending，由 Router 有界重试，不能丢弃"，但未定义： + +- agent 事件与用户输入是否共用同一 mpsc（共用则用户流量可长期占满队列，事件重试无收敛界）； +- Router 重试的退避、deadline 与最终处置； +- §15.4 只拆分了 TurnMailbox 的 lane（user 32/64KiB、agent 8/32KiB），session 级队列的 agent lane 容量与优先级未定义。 + +**A2 — `/stop` 与 worker 退出时 durable event 的恢复机制缺失（设计 §18.3）** + +现状 `/stop`：`current_cancel.take()`（`session.rs:2117`）→ `close_and_take_pending` 丢弃 steering（:2120-2128）→ `agent_tx.take()` 丢弃全部排队任务（:2136）→ bump generation/state_version（:2139-2140）→ 取消后台子任务（:2144-2148）。mpsc 被 drop 时没有逐项回调。设计未定义： + +- 被丢弃的内部 `AgentTask`（含 `BackgroundAgentResults`）如何触发 inbox lease 释放——只能靠 lease 超时被动收敛（应明说延迟界），或为 AgentTask 增加 Drop guard（未提）； +- `/stop` 后 worker 退出（`task_rx.recv()` 返回 None 即 break，`session.rs:3150-3152`），被取消 run 异步生成的 cancelled completion 由谁、何时消费，完整链路未写。 + +**A3 — `waiting_children` permit 释放在现有结构中无落点（设计 §19.2）** + +AgentLoop 目前没有任何 permit/cancellation 原语（`agent_loop.rs` 无 CancellationToken/select；取消靠 worker 整体 drop future，`session.rs:3778-3804`）；TaskSupervisor 也没有并发上限，只在 stopping 时拒绝 spawn（`task_supervisor.rs:60-89`）。设计只给出原则"permit 限制活跃 Provider/工具步骤"，未定义： + +- permit 由谁持有与获取/释放（AgentLoop？AgentRunner？Coordinator？）； +- delegate 在父 run 的 tool batch 内执行（`agent_loop.rs:1187-1242`），父 run 进入等待时释放 permit 的钩子如何嵌入现有批处理流程。 + +这是 Phase 2 复杂度最高的部分，只给原则不够。 + +**A4 — 结构化取消是前置条件，但 AgentLoop 当前零支持（设计 §18.1）** + +新架构中子 run 是 Coordinator spawn 的独立任务，父 future 被 drop 不再传播取消，必须用显式 CancellationToken 树贯穿 AgentLoop——这是横切重构，设计只在 Phase 2 列了一行"实现结构化取消"。另有一处表述需要澄清：§3 非目标"不默认硬中断正在进行的 Provider 请求"只约束 `steer`；现有 `/stop` 恰是硬 drop（drop `process_future` 连带中断 provider 流，`session.rs:3778-3804`）。文档应显式声明 `/stop` 保持硬语义，避免实现时误读为 `/stop` 也要走安全边界。 + +**A5 — 内部 continuation Turn 的消息/持久化/渲染模型未定义（设计 §16.3）** + +现有 Turn 以用户消息为起点：先持久化用户消息（`session.rs:3191`）；`prepare_turn_input` 把 runtime context 附加到最后一条 user message（`src/session/turn_input.rs:19-25`）；WebUI/TUI 按 user/assistant 交替渲染。设计说"内部输入不显示用户气泡"，但未定义： + +- continuation Turn 写什么消息行（无 user 行？系统行？）、历史 replay 给 provider 时的形态； +- 客户端如何渲染无用户消息的 Turn（§15.1 的 SourceKind 扩展只解决标记问题）； +- continuation 输出的投递目标：`AgentTask` 的 channel/chat_id/channel_context 来自 InboundMessage（`session.rs:631-641`），内部任务没有 channel 上下文，应显式规定投递到 session 最近的 channel/chat。 + +### 5.2 B 级：中等缺陷 + +**B1 — browser/resource scope 隔离是行为破坏，且无共享出口（设计 §10）。** 现状子 Agent 复用父对话的 browser session（`browser_session_id` 回退到 delegate context 的 session_id，`sub_agent.rs:264-279`；background 路径 :505-508 同）。改为 `root_session_id + run_id` 隔离会破坏依赖父会话登录态/cookie 的场景。"确需共享必须由工具定义显式支持"没有给出机制，应指明 `browser_profiles` persistent ID 为官方共享路径。 + +**B2 — dead_letter 与重试上限策略空缺（设计 §14.3、§21）。** Phase 3 移除直发通知后，continuation 反复失败转 dead_letter 时结果对用户彻底不可见，文档未定义 dead-letter 后的用户可见行为（如回退一条系统通知）。`max_pending_inbox_events_per_session=128` 打满后新事件的行为同样未定义。 + +**B3 — `completion_policy=each` 只有字段没有语义（设计 §14.2）。** 正文只描述了 `all`：一个 run 超时会把全组结果交付拖到 group deadline，缺少 per-run 提前交付或分组拆分策略。 + +**B4 — "UI 未读状态"是防饥饿的关键依赖，但不存在且未立项（设计 §16.2）。** 调度优先级把 queue completion 排在用户输入之后，持续用户流量下后台结果会被无限推迟，设计靠"UI 未读状态"兜底；该 WebUI 功能当前不存在，Phase 3 只写了"WebUI 投影"，未列为明确工作项。 + +**B5 — `cost` 字段假设了不存在的定价配置（设计 §6.5、§14.1）。** `agent_runs.cost` 与 ProviderFactory"复用价格信息"的前提不成立：config 从不填 `price_input/output_per_million`（`src/config/mod.rs:742-743` 硬编码 None，无配置键解析）。要么补定价配置，要么注明 cost 暂为 NULL。 + +**B6 — 子 Agent run 内 sleep 的唤醒语义未定义（设计 §17）。** 全章隐含 root session 的 Turn；sub-run 没有"当前 session 用户输入"概念。应显式规定 sub-run 内 sleep 只响应自身 cancellation/timeout，否则 `TurnWakeupHandle` 的来源不明。 + +### 5.3 C 级：次要问题 + +**C1 — SQLite UNIQUE 与 NULL 语义（设计 §14.3、§9.6）。** `UNIQUE(run_id, event_type, event_key)`：`emit_signal` 未提供 `dedupe_key` 时 `event_key` 的生成规则未定义。`idempotency_key` 的唯一范围 `(root_session_id, caller_run_id, key)` 在 ROOT 调用时 `caller_run_id` 为 NULL，SQLite 中 NULL≠NULL 会导致去重失效，需要哨兵值（如 `root`）。 + +**C2 — 授权与上下文的细节留白（设计 §7.1、§9.1、§10）。** definition `max_depth` 与全局 `max_tree_depth` 是否取 min 未明说；`agent_task` 工具能否操作本 root session 任务树之外的 run（跨 session 越权）未明说；skills/memory 是否进入子 Agent 上下文未提（现状子 Agent 可带 skills prompt，`sub_agent.rs:188-197`；memory recall 只在 session Turn，`turn_input.rs:47`）。 + +**C3 — `async` 别名是多余假设（设计 §22.1）。** 现代码从未接受 `async`（`delegate.rs:151-165` 只解析 inline/background/parallel），该迁移条目可删。 + +**C4 — 客户端协议变更未枚举（设计 §20.2、Phase 4）。** AgentSignal 卡片、"已接纳"状态、continuation Turn 都需要新的 `WsOutbound` 消息类型（`src/protocol.rs`），Phase 4 只写"添加 AgentSignal UI 和任务树"，未列协议变更清单。 + +## 6. 修订建议 + +实现启动前，建议在设计文档中补充五个专项定义（对应 A 级缺陷）： + +1. **Durable event 与 session 队列的 lane 划分**：agent 事件是否独立队列、饱和时的重试退避/deadline/dead-letter 策略、与用户输入的优先级关系（A1）。 +2. **`/stop`、worker 退出与 lease 释放的衔接**：被丢弃内部任务的 lease 释放路径（Drop guard 或明确依赖 lease 超时及延迟界）、`/stop` 后生成的 cancelled completion 的消费链路（A2）。 +3. **Permit 归属**：执行 permit 在 AgentLoop/AgentRunner/Coordinator 之间的获取与释放点，特别是 `waiting_children` 前后的钩子位置（A3）。 +4. **Continuation Turn 模型**：消息行写入形态、provider replay 形态、客户端渲染契约、输出投递目标（A5）。 +5. **取消横切方案**：CancellationToken 贯穿 AgentLoop 的接口设计，并显式声明 `/stop` 保持硬 drop 语义、安全边界注入只约束 `steer`（A4）。 + +B 级问题建议在对应 Phase 实现前补齐：B1/B6 在 Phase 1，B5 在 Phase 2，B2/B3/B4 在 Phase 3。 + +## 7. 分期实施意见 + +| Phase | 风险 | 意见 | +|-------|------|------| +| 1 具名 Agent 与 Foreground | 低 | 纯增量。`llm_profile` 直接复用 `Config::get_provider_config`（`config/mod.rs:712-745`），无配置重构。注意 B1：browser scope 隔离会改变现有子 Agent 共享父会话 browser 的行为，需要迁移说明 | +| 2 统一 Run 持久化 | 高 | 改动面最大：AgentLoop 取消与 permit 均为横切变更。建议先独立原型"CancellationToken 贯穿 AgentLoop"，用现有 sleep 取消测试（`sleep.rs:194-237`）与 steering 恢复测试锁定回归基线，再叠加 permit | +| 3 Inbox 与 Queue Completion | 中 | UX 拐点：完成通知从直发 Channel 改为主 Agent continuation。建议保留配置开关回退直发通知，覆盖 B2 的 dead-letter 空窗；§25.4 测试矩阵是本 Phase 验收关键 | +| 4 Emit Signal 与 Steer | 中 | TurnMailbox 泛化触及 `handle_message` 核心 admission 路径（`session.rs:2815-2953`），与 `/stop` 的原子性必须沿用现有同锁判定模式；先补 C4 协议清单 | +| 5 可唤醒 Sleep | 低 | 相对独立。watch revision 方案正确；先补 B6 的 sub-run 语义 | + +## 8. 结论 + +设计的现状诊断准确、核心决策与既有架构不变量兼容、分期依赖方向正确，**审核结论为"方向通过，需修订后实现"**。A1–A5 五个接缝缺口不是方向错误，而是设计与 `session worker`/`/stop`/`AgentLoop` 取消机制的衔接定义不足；按第 6 节补齐专项定义后，可按第 7 节顺序分期实施。 diff --git a/docs/SUB_AGENT_ORCHESTRATION_REVIEW_RESPONSE.md b/docs/SUB_AGENT_ORCHESTRATION_REVIEW_RESPONSE.md new file mode 100644 index 0000000..6950afe --- /dev/null +++ b/docs/SUB_AGENT_ORCHESTRATION_REVIEW_RESPONSE.md @@ -0,0 +1,222 @@ +# 子 Agent 编排与信号投递设计评审答复 + +> 状态：设计方答复（2026-08）。 +> +> 本文逐项回应 `docs/SUB_AGENT_ORCHESTRATION_REVIEW.md`。评审原文作为审核记录保留；已接受的结论同时回写到 `docs/SUB_AGENT_ORCHESTRATION_DESIGN.md`，后者仍是后续实现的规范来源。代码级实施方案见 [`SUB_AGENT_ORCHESTRATION_IMPLEMENTATION.md`](SUB_AGENT_ORCHESTRATION_IMPLEMENTATION.md)。 + +## 1. 总体答复 + +接受评审的总体结论：方案方向成立，但 A1–A5 必须在实现前成为明确契约。全部 A、B、C 项均采纳；其中 A2、B2 和 B4 不只补充文字，还调整了原方案的数据流： + +- durable Agent event 不进入普通 session 消息队列，而由 SQLite inbox 保存 payload、独立的合并式 wake lane 只传递“有待处理事件”的提示。 +- `/stop` 清理瞬时用户工作，但不通过丢弃内存队列来确认 durable event；事件由条件更新立即释放，lease expiry 只作为崩溃兜底。 +- 后台结果调度采用有界公平策略，UI 未读状态降为可观察性能力，不再承担防饥饿正确性。 +- inbox 容量在接纳 background run 时预留终态事件空间；signal 可以因容量不足被拒绝，completion 不能在 run 结束时才发现无处落库。 +- continuation 使用“持久化但对客户端隐藏”的内部触发消息，保证 Provider replay、事务提交和客户端渲染三者一致。 + +| 评审项 | 答复 | 设计处理 | +|--------|------|----------| +| A1 | 接受 | 独立 durable wake lane、claim-on-run、公平调度和有界重试 | +| A2 | 接受 | `/stop` 条件释放、lease guard、取消 completion 的 status-only 语义 | +| A3 | 接受 | run admission 与 step execution permit 分离 | +| A4 | 接受 | CancellationToken 贯穿 AgentLoop；明确 `/stop` 与 `steer` 不同 | +| A5 | 接受 | hidden trigger message、Turn origin、稳定 delivery binding | +| B1 | 接受 | 默认隔离；persistent browser profile 是唯一显式共享路径 | +| B2 | 接受 | 容量预留、dead-letter fallback、重试边界 | +| B3 | 接受 | `all` 与 `each` 的事件生成和 deadline 语义 | +| B4 | 接受 | worker 有界公平；UI 未读只负责呈现 | +| B5 | 接受 | 未配置价格时 `cost=NULL` | +| B6 | 接受 | sub-run sleep 只响应 timer/cancellation | +| C1 | 接受 | 非空 event key、非空 caller scope、partial unique index | +| C2 | 接受 | 深度、task-tree 授权、skills/memory 继承规则 | +| C3 | 接受 | 删除不存在的 `async` 迁移别名 | +| C4 | 接受 | 枚举 WebSocket 请求、投影和 Turn origin 变更 | + +## 2. A 级意见答复 + +### A1 — Session 队列饱和语义 + +**答复：接受。durable event 不与用户 `AgentTask` 共用 payload mpsc。** + +实现采用两条不同语义的 lane： + +```text +user task lane bounded mpsc(32)，保存任务；满时明确拒绝新用户输入 +agent inbox wake lane watch revision，合并通知；payload 始终留在 SQLite +``` + +Router 对活动 Turn 的 `steer` 使用 lease → mailbox reservation → durable admitted → activate 的两阶段 admission；reservation 在 durable 更新成功前不可被 AgentLoop 排空，且 SQLite I/O 不跨 Session 锁。失败或 `queue` 事件都恢复/保持 `pending`，只递增 wake revision。worker 在真正准备执行 continuation 时才领取 lease，不先把已 leased 的事件塞进可能被丢弃的 mpsc。 + +`watch` 只负责降低延迟：发送失败、revision 被合并或进程退出都不影响事实状态。Gateway 启动、reload 激活和周期恢复扫描会重新发现 `pending`/expired lease。普通 session 队列满不再构成 durable event 丢失或无限重试问题。 + +worker 在每个 Turn 调度边界执行有界公平：通常先处理用户任务；连续处理 4 个用户 Turn，或最老 pending event 已等待 30 秒后，必须先领取一批 continuation。它仍不能中断当前不可分割的 Turn，因此时限从下一个调度边界计算。 + +### A2 — `/stop`、worker 退出与事件恢复 + +**答复：接受。lease expiry 只能是崩溃兜底，不能是正常 `/stop` 的唯一恢复路径。** + +调整后的链路为： + +1. `/stop` 在关闭 TurnMailbox 时取回尚未提交的 durable event IDs。 +2. 在 session generation 失效后，以 `lease_token`/`admitted_turn_id` 条件更新把这些事件立即恢复为 `pending`。 +3. continuation 执行持有 `InboxLeaseGuard`；正常失败、取消或 stale generation 会显式 release，只有进程崩溃或任务被强制 abort 才等待 `lease_until` 到期。 +4. 普通内部 continuation 不作为 payload 存在 session mpsc 中，因此 `agent_tx.take()` 不会吞掉 leased event；worker 领取后才在本地构造 typed task source。 +5. Coordinator 独立拥有 background run。`/stop` 取消 run token 后，Coordinator 仍负责用条件事务写入 `cancelled` 终态，迟到的 completed 结果不能覆盖它。 + +由本次 `/stop` 自身造成的 cancelled completion 设为 `requires_continuation=false`：事件和 run 状态会持久化并投影到任务树，但事件在同一事务中记为 status-only consumed，不会在 `/stop` 后反向启动一个“任务已取消”的主 Agent Turn。`/stop` 前已经存在、尚未处理的 signal/completion 不被确认或删除，恢复为 pending 后仍可继续投递。 + +### A3 — `waiting_children` permit 归属 + +**答复：接受。permit 不由整个 Agent Run 持有，也不由 `delegate` 等编排工具持有。** + +Coordinator 管理两类限制： + +- **run admission quota**：限制树、session、Agent 的已接纳/未终态 run 数量；可以跨 `waiting_children` 持有。 +- **step execution permit**：限制当前正在占用 Provider 或普通工具执行资源的步骤；只在一个步骤期间持有。 + +AgentLoop 在每次 Provider 请求前按固定顺序获取 global → session → agent provider permits，流结束或取消后立即释放。普通工具调用由 tool executor 获取 tool permit；`delegate`、`agent_task`、`emit_signal` 等 runtime-control 工具不获取这种稀缺执行 permit。 + +foreground `delegate` 在创建 child 前通过状态 guard 把父 run 从 `running` 条件更新为 `waiting_children`，等待期间没有 provider/tool permit。children 终态后 guard 把父状态恢复为 `running`；父 Agent 的下一次模型迭代重新竞争 permit。这样即使 provider 并发上限为 1，父等待 child 也不会死锁。 + +### A4 — AgentLoop 结构化取消 + +**答复：接受，并把它提升为 Phase 2 的独立前置里程碑。** + +`AgentLoop` 的执行入口将显式接收 cancellation context，而不是只依赖父 future 被 drop： + +```text +root Turn token +└── foreground run token + └── descendant foreground run token + +root session token ── independently owns background run tokens +``` + +Provider stream、可取消等待和工具批次外层都观察 token；AgentRunner 的终结路径把取消归一为类型化 `cancelled`，Coordinator 再用 execution ID 条件提交。父取消、run timeout、reload/shutdown 可以组合为任一触发即取消。 + +“不默认硬中断”只约束普通 `steer`：它等待安全边界，不取消 Provider 或副作用工具。`/stop` 保持现有强停止语义，会取消 token 并使 root Turn future 失效；未在宽限期内自行退出的独立 child task 由 Coordinator/Supervisor abort。无论 future 如何结束，terminal condition update 都阻止迟到结果提交。 + +### A5 — continuation Turn 模型 + +**答复：接受。continuation 需要同时满足 durable replay、无伪用户气泡和正常 Turn 投递。** + +每个 continuation 生成一条内部触发消息： + +- 数据库 role 使用 Provider 可兼容的 `user`，source 为 `agent_signal`/`agent_result`。 +- 增加 `client_visibility=hidden` 和 `turn_origin=agent_continuation`；普通历史 API 和 `turn_committed.messages` 不投影这条消息。 +- 内容是有界、带 event/run reference 的 runtime envelope，不伪造外部 sender，也不直接采用子 Agent 输出中的指令优先级。 +- Provider 历史 replay 会包含该隐藏消息，因此后续 assistant 回复不会成为无来源的悬空历史。 +- 隐藏触发消息、assistant/tool 结果、usage 和 inbox `consumed` 在同一事务中提交；失败时全部不确认。 + +客户端继续使用 `turn_updated`/`turn_committed` 展示 assistant Turn，但帧增加 `turn_origin`，从而可以显示“后台结果处理”标记且不创建用户气泡。 + +内部 Turn 的出站目标来自 root session 的 durable delivery binding：`channel`、`chat_id` 以及可复用的 thread/root 上下文。一次性 `reply_to` 不得复用。若没有可用的外部 binding，结果仍持久化并供 WebUI/TUI 历史读取，不猜测其他目标。 + +## 3. B 级意见答复 + +### B1 — browser/resource scope 隔离 + +**答复：接受。** 新具名 Agent 默认使用 `root_session_id + run_id` 的瞬时资源 scope，不继承父会话 browser cookie。需要共享登录态时，官方路径是由 Root 创建/选择经过校验的 `browser_profiles` persistent ID，并把该 ID 作为显式 task/artifact reference 交给获准使用 browser 的子 Agent；子 Agent必须在每次相关调用中显式传入该 ID。 + +为平滑迁移，由无 target 的旧调用映射出的内置 `general` 兼容 Agent 可在弃用期保留父 session transient scope；具名 Agent不继承这个例外。文档和 tool result 会明确提示两种 scope 的差异。 + +### B2 — dead letter、重试和 inbox 上限 + +**答复：接受。** 接纳 background run 时按 completion policy 预留不可抢占的终态 event slot：`each` 每个 run 一个，`all` 每个 group 一个。容量不足时 `delegate` 在创建 run 前拒绝。signal 只使用未预留容量，满时 `emit_signal` 返回 `inbox_full`，但不终止 run。因此已接纳 run 的 completion 永远不会在终结时因 inbox 满而丢失。 + +暂定恢复策略为 8 次可配置尝试，退避 `1s/5s/30s/2m/10m` 后封顶 10 分钟，并同时受 event TTL 限制。瞬时 Storage/worker/Provider continuation 失败可重试；session 已删除、授权事实失效或 payload 永久损坏立即 dead-letter。lease timeout 不单独计作永久错误，但会记录 attempt 和原因。 + +事件进入 dead-letter 后： + +1. 保存最终原因和 `dead_lettered_at`，在任务树/API 中持续可见。 +2. 通过 OutboundDispatcher 最多发送一次有界 system fallback，内容只包含 run/group ID、终态和查询提示，不复制大结果。 +3. 用 `fallback_notified_at` 保证 fallback 幂等；渠道也失败时仍以 SQLite 记录和管理 UI 为最终可诊断出口。 + +### B3 — `completion_policy=each` + +**答复：接受。** 语义修订为： + +- `each`：每个 run 进入终态即创建独立 completion event；Router 可在 300–500ms debounce 窗口合并一次 continuation，但不能等待其他 sibling。 +- `all`：单 run 终态只更新 group 计数，不创建可投递 completion；全部终态或 group deadline 到达后创建一个 `group_completion` event，包含全部逐项状态和 result references。 +- group deadline 到达时，未终态 children 被取消并条件更新为 `timed_out`，随后生成唯一 group completion。 + +因此 inbox schema 允许 run-scoped 或 group-scoped event 二选一，而不是强制 `run_id NOT NULL`。 + +### B4 — UI 未读状态与防饥饿 + +**答复：接受。** 正确性由 A1 的 worker 有界公平策略保证；UI 未读只呈现尚未汇总/已 dead-letter 的事件数量，不参与调度。Phase 3 明确包含未读计数、event revision 和 reconnect 后全量校准。 + +### B5 — `cost` 与定价配置 + +**答复：接受。** Phase 2 保留 nullable `cost` 字段，但只有 Provider profile 明确提供 input/output/cache 价格时才计算；当前配置没有价格来源，因此写 `NULL`。usage token 仍照常持久化。价格配置和历史重算不属于本次编排功能的前置条件。 + +### B6 — sub-run 内 sleep + +**答复：接受。** `TurnWakeupHandle` 只存在于 root interactive Turn。sub-run 的 `ToolExecutionContext.turn_wakeup=None`，其 `sleep` 只等待 timer、run cancellation、timeout 或 shutdown；不会监听 root session 用户输入或其他 Agent signal。需要被主 Agent立即控制时使用 `agent_task.cancel`，由 cancellation token 唤醒。 + +## 4. C 级意见答复 + +### C1 — SQLite NULL 与事件去重 + +**答复：接受。** 所有参与唯一约束的 scope/key 都规范化为非空值： + +- background idempotency 使用 `caller_scope_id TEXT NOT NULL`；Root 固定为字面量 `ROOT`。 +- `idempotency_key` 仍可为空，但使用 `CREATE UNIQUE INDEX ... WHERE idempotency_key IS NOT NULL` 的 partial unique index。 +- 无 `dedupe_key` 的 signal 使用 `signal:`。 +- 有 `dedupe_key` 的 signal 使用 `signal::`，只在冷却窗口内去重，不会永久压制同类告警。 +- run completion 使用固定 `completion:terminal-v1`；group completion 使用 `group-completion:terminal-v1`。 + +### C2 — 深度、task-tree 授权和上下文继承 + +**答复：接受。** + +- 全局 `max_tree_depth` 是 root-relative 硬上限；Definition `limits.max_depth` 是该 Agent可继续创建的最大相对后代深度。child 的 remaining depth 为 `min(parent_remaining - 1, target_definition.max_depth)`，任何一项为 0 都不能继续委托。 +- `agent_task` 查询必须匹配当前 `root_session_id`。Root 可操作本 session 的整棵树；子 Agent只能读取自身与后代，只能取消其未终态后代，不能通过猜测 run ID 跨 session 或操作祖先/sibling。 +- 子 Agent不继承主会话 history、memory recall 或临时 activated skills。第一版仅在 Definition 工具集中包含 `get_skill` 时注入受信任 Skill catalog；调用方只能通过显式 task/context 传递事实。若未来开放 memory，必须新增管理员配置的只读 scope，不能默认继承。 + +### C3 — `async` 别名 + +**答复：接受。** 删除 `async → background`。迁移只接受代码中确实存在的 `inline`、`parallel`、`background`；新 prompt/schema 只公布 `foreground`、`background`。 + +### C4 — 客户端协议清单 + +**答复：接受。** 协议按通用运行投影设计，不为 Signal 单独复制一套模型： + +- `WsInbound::GetAgentRuns { session_id, cursor, limit }` +- `WsInbound::GetAgentRun { session_id, run_id }` +- `WsOutbound::SessionAgentRuns { session_id, revision, runs, next_cursor }` +- `WsOutbound::AgentRunUpdated { session_id, revision, run }` +- `WsOutbound::AgentEventUpdated { session_id, revision, event }` +- `TurnSnapshot` 与 `WsOutbound::TurnCommitted` 增加 `turn_origin = user | agent_continuation | scheduled` + +`AgentEventUpdated` 同时承载 accepted、admitted、consumed、dead-letter 等状态，按 `(session_id, revision, event_id)` 幂等合并。断线重连后客户端用 `GetAgentRuns` 全量校准，实时帧只是增量。取消操作第一版继续通过 `/stop`、`agent_task.cancel` 或受保护管理 API，不额外开放一个缺少权限上下文的裸 WebSocket cancel 帧。 + +## 5. 对分期的调整 + +| Phase | 调整后的完成条件 | +|-------|------------------| +| 1 | 除原内容外，明确 browser 兼容 scope、skills/memory 规则和 sub-run sleep 行为 | +| 2A | CancellationToken 贯穿 AgentLoop，先以现有 root Turn/sleep/Provider tests 锁定取消语义 | +| 2B | run/group 持久化、step execution gate、foreground child cancellation 和结果查询 | +| 3 | durable wake lane、capacity reservation、hidden continuation trigger、bounded fairness、dead-letter fallback 与 WebSocket run/event projection | +| 4 | typed TurnMailbox、emit_signal、steer admission，以及同一 `AgentEventUpdated` 的 Signal 卡片呈现 | +| 5 | root Turn wake-aware sleep；sub-run 保持 timer/cancellation-only | + +Phase 3 上线时保留旧 direct notification 的受控兼容开关，但仅作为 rollback 手段，默认路径必须是 inbox/continuation；不能同时投递两条用户通知。开关移除前必须验证 dead-letter fallback、重启恢复和 reconnect 校准。 + +## 6. 最终结论 + +评审结论“方向通过，需修订后实现”成立。修订后的关键边界是： + +```text +SQLite inbox 保存事实 + ├── direct steer admission → 当前 TurnMailbox + └── coalesced wake → worker claim → hidden continuation Turn + +普通 user mpsc 满 ≠ durable event 丢失 +/stop 丢弃瞬时用户工作 ≠ 确认 durable event +run 存活配额 ≠ Provider/tool step permit +UI 未读提示 ≠ 调度正确性 +``` + +在 A1–A5 的专项契约和上述 schema/protocol 调整落地前，不应开始 Phase 3/4 的生产实现。 diff --git a/resources/skills/about-picobot/references/architecture.md b/resources/skills/about-picobot/references/architecture.md index 9f252b0..395517e 100644 --- a/resources/skills/about-picobot/references/architecture.md +++ b/resources/skills/about-picobot/references/architecture.md @@ -50,7 +50,7 @@ Scheduler → SessionManager.handle_cron_message → AgentLoop → send_message - 每个活动 Turn 独占一个 TurnSink；平台 message ID 和 reaction 清理状态只存在于 sink 内 - Tools 接收原始参数，通常返回字符串结果；有状态适配器额外接收 session/turn `ToolExecutionContext` - MCP 工具在 Gateway 初始化时连接服务器、发现工具，并包装成普通 Tool 注册到 ToolRegistry -- 子 Agent 由 `delegate` 工具创建，复用 provider 配置和按需过滤后的工具集；后台任务结果通过 MessageBus 发回原会话 +- 具名子 Agent 从运行代不可变 `AgentCatalog` 加载，Definition 固定 Provider profile、工具/Skill allowlist、委托边与限制；新工具默认 RootOnly。Phase 1 支持单个/批量 foreground（批量并发、按请求顺序返回）和显式父子授权；具名 background 在 durable run/inbox 完成前拒绝。禁用编排时旧 general background 仍通过 MessageBus 直接通知原会话 - 复杂任务可使用 `todo` 创建 session 级计划；多个子项可通过 `delegate.plan_item_id` 并行委托，子 Agent 不能修改计划 - WebUI 聊天复用 `/ws` 与 `cli_chat`；同源管理 API 只读取受限的日志、任务、记忆，并对白名单配置文件做原子写入 - WebUI 使用 Svelte 5 + Vite，Bits UI 提供无样式可访问组件；`cargo build` 增量生成前端到 Cargo `OUT_DIR`，再嵌入单二进制，仓库不保存生成产物 @@ -258,11 +258,12 @@ Gateway 初始化时读取 `config.mcp.servers`： | 模式 | 行为 | |------|------| -| `inline` | 当前轮阻塞等待子 Agent 返回 | -| `background` | 后台运行，完成后通过原 channel/chat 通知 | -| `parallel` | 多个子 Agent 并发执行并聚合结果 | +| `foreground` | 当前轮等待一个或多个子 Agent；批量任务并发执行并按请求顺序聚合 | +| `background` | 异步执行并返回 run ID；当前只有旧 general 兼容路径可用 | -默认工具集是只读工具：`file_read`、`file_search`、`content_search`、`web_fetch`、`http_request`、`calculator`。调用时可通过 `allowed_tools` 显式放开其他工具。后台任务会写入 `background_tasks` 表，默认 24 小时后清理。 +启用 `agent_orchestration` 后，具名 Definition 固定角色、Provider profile、工具/Skill allowlist、委托边和限制。`allowed_tools` 只能收窄 Definition，不能扩权；新工具默认 RootOnly，当前明确可委托的工具包括 `file_read`、`file_search`、`content_search`、`web_fetch`、`calculator`、普通 `browser` 动作和 `sleep`。具名 Agent可按委托图继续 foreground 委托，但 ancestry 重复、越深度或不在白名单的目标会拒绝。具名 background 需要后续 durable run/inbox，当前明确拒绝。 + +未启用编排或省略 target 时使用旧 general 兼容路径。其工具也只能取旧默认集合与 Delegatable 策略的交集；旧后台任务写入 `background_tasks` 表，完成后通过原 channel/chat 直接通知，默认 24 小时后清理，不具备 durable inbox 语义。 后台子 Agent 通过 `TaskSupervisor::spawn_graceful` 注册，受 `gateway.max_concurrent_background_tasks` 限制；Gateway 关停时先收到取消信号，再在总宽限期内清理。 diff --git a/resources/skills/about-picobot/references/config.md b/resources/skills/about-picobot/references/config.md index 7319e60..9c60add 100644 --- a/resources/skills/about-picobot/references/config.md +++ b/resources/skills/about-picobot/references/config.md @@ -11,7 +11,8 @@ Gateway WebUI 的“配置”页可以编辑实际加载的配置文件。读取 { "providers": {}, // LLM 提供商配置 "models": {}, // 模型配置 - "agents": {}, // agent 配置 + "agents": {}, // Provider/Model profile + "agent_orchestration": {}, // 具名子 Agent Definition 与编排上限 "gateway": {}, // 网关配置 "client": {}, // 客户端配置 "channels": {}, // 渠道配置 @@ -51,6 +52,28 @@ Gateway WebUI 的“配置”页可以编辑实际加载的配置文件。读取 | `max_tool_iterations` | int | 99 | 最大工具调用轮数 | | `token_limit` | int | 128000 | 上下文 token 限制 | +## agent_orchestration 字段 + +默认 `enabled=false`。启用后，`definitions_dir` 相对 `config.json` 所在目录解析，且不得通过绝对路径或 symlink 逃逸该受信任配置目录。Gateway 启动和热重载会严格校验全部 Markdown Definition；任一无效 Provider profile、工具、Skill 或委托目标会拒绝整个候选运行代。 + +| 字段 | 默认 | 说明 | +|------|------|------| +| `enabled` | false | 是否启用具名 Agent Catalog | +| `definitions_dir` | agents | 第一层 `*.md` Definition 目录 | +| `root_delegates` | [] | Root 可委托的具名 Agent ID | +| `max_tree_depth` | 4 | Root-relative 委托深度硬上限 | +| `max_runs_per_tree` | 16 | 单任务树 run 预算（Phase 2 持久 Coordinator 强制完整树计数） | +| `max_concurrent_runs` / `max_concurrent_runs_per_session` | 6 / 4 | run admission 上限（Phase 2） | +| `max_concurrent_provider_steps` / `..._per_session` | 8 / 4 | Provider step 上限（Phase 2） | +| `max_concurrent_tool_steps` / `..._per_session` | 16 / 8 | 普通工具 step 上限（Phase 2） | +| `max_pending_inbox_events_per_session` | 128 | durable inbox 容量（Phase 3） | +| `inbox_event_ttl_hours` | 168 | inbox event TTL（Phase 3） | +| `max_inbox_delivery_attempts` | 8 | inbox 最大投递次数（Phase 3） | +| `max_user_turn_burst_before_inbox` | 4 | 用户 Turn 公平调度阈值（Phase 3） | +| `max_inbox_wait_secs` | 30 | inbox 最大等待阈值（Phase 3） | + +当前已实现具名 foreground Agent、不同 `llm_profile`、固定工具/Skill allowlist、批量并发和父子委托边校验。具名 background 会明确拒绝，直到 durable run/inbox 实现；未启用时旧 general background 兼容路径保持可用。 + ## gateway 字段 | 字段 | 类型 | 默认 | 说明 | diff --git a/resources/skills/about-picobot/references/tools.md b/resources/skills/about-picobot/references/tools.md index 3eb6239..7beefa6 100644 --- a/resources/skills/about-picobot/references/tools.md +++ b/resources/skills/about-picobot/references/tools.md @@ -137,21 +137,34 @@ Cron 不是一个带 `action` 的统一工具，而是六个独立工具；仅 | 参数 | 必填 | 说明 | |------|------|------| -| `action` | 是 | `run`, `check_task`, `cancel_task`, `list_tasks` | -| `prompt` | run 必填 | 子任务描述 | -| `mode` | 否 | `inline`, `background`, `parallel`，默认 `inline` | -| `allowed_tools` | 否 | 子 Agent 可用工具列表；默认只读工具集 | -| `max_iterations` | 否 | 最大迭代次数，默认 99 | -| `timeout_secs` | 否 | 超时秒数，默认 3600 | -| `tasks` | parallel 必填 | 并行子任务数组 | +| `action` | 否 | 默认 `run`；迁移期仍支持 `check_task`, `cancel_task`, `list_tasks` | +| `target` | 具名 Agent 必填 | `root_delegates` 或当前 Agent Definition 允许的目标 ID | +| `task` | 单任务必填 | 明确、独立、可验收的子任务；旧 `prompt` 仅兼容解析 | +| `context` | 否 | 子 Agent 所需的显式事实；不会继承完整主会话历史 | +| `mode` | 否 | `foreground`（默认）或 `background`；批量并发不是第三种 mode | +| `tasks` | 批量必填 | 子任务数组；foreground 并发执行、结果保持请求顺序 | +| `allowed_tools` | 否 | 迁移字段，只能收窄具名 Definition 或旧 general 默认集，不能扩权 | +| `max_iterations` | 否 | 旧 general 兼容限制；具名 Agent使用 Definition limits | +| `timeout_secs` | 否 | 旧 general 兼容限制；具名 Agent使用 Definition limits | | `task_id` | 查询/取消必填 | 后台任务 ID | -| `plan_item_id` | 否 | 将 inline/background 子 Agent 绑定到当前计划子项；parallel 数组中的每项也可分别绑定 | +| `plan_item_id` | 否 | 绑定当前计划子项；批量数组中的每项可分别绑定 | -默认只读工具集：`file_read`、`file_search`、`content_search`、`web_fetch`、`http_request`、`calculator`。 +旧 `inline` 映射到 foreground，旧 `parallel` 映射到 foreground + `tasks[]`，但不再出现在 tool schema。具名 background 单任务（`target` + `mode=background`）经 durable run/inbox 接纳：先落 `agent_runs` 并预留 inbox completion slot，完成后由主 Agent 的 continuation Turn 处理结果，不再直接发 Channel 通知；background 批量、子 Agent 发起的 background 以及 general background 的新 durable 路径尚未开放（general 仍走旧 direct notification 兼容路径）。后台运行可以在任务中调用 `emit_signal` 发送结构化内部信号（队列或 steer 投递），Steer 信号会在当前 Turn 的安全边界注入主 Agent；`agent_task.cancel` 会把该 run 未消费的普通信号标记 superseded。 + +## agent_task — 具名 Agent Run 查询与控制 + +仅在 `agent_orchestration` 启用时注册。查询或控制已持久化的具名 foreground run；授权来自调用者身份（ROOT 限本 session，具名 Agent 限自身子树），run ID 本身不是凭证。 + +| 参数 | 必填 | 说明 | +|------|------|------| +| `action` | 是 | `get` 查询单个 run；`list` 列出 session 的 run；`get_result` 读取终态完整结果；`cancel` 取消未终态 run | +| `run_id` | get/get_result/cancel 必填 | 目标 run ID | +| `cursor_created_at` / `cursor_id` | 否 | list 分页游标，必须成对出现 | +| `limit` | 否 | list 上限，默认 20 | ## todo — Session 任务计划 -仅用于明确的复杂、多轮或并行任务。`create` 创建当前 session 唯一的 active plan；`view` 查看；`append` 增加子项；`update` 修改子项状态；`close` 完成或取消计划。普通闲聊和单步操作不应创建计划。子 Agent 始终被过滤掉 `todo` 和 `delegate`，计划结构只由主 Agent 管理。 +仅用于明确的复杂、多轮或并行任务。`create` 创建当前 session 唯一的 active plan；`view` 查看；`append` 增加子项；`update` 修改子项状态；`close` 完成或取消计划。普通闲聊和单步操作不应创建计划。子 Agent 始终不能使用 `todo`；只有 Definition 声明委托边的具名 Agent 会获得运行时注入的 `delegate`，且只能 foreground 委托允许目标。 --- diff --git a/resources/templates/config.example.json b/resources/templates/config.example.json index 9c36030..a202b84 100644 --- a/resources/templates/config.example.json +++ b/resources/templates/config.example.json @@ -46,6 +46,24 @@ "token_limit": 128000 } }, + "agent_orchestration": { + "enabled": false, + "definitions_dir": "agents", + "root_delegates": [], + "max_tree_depth": 4, + "max_runs_per_tree": 16, + "max_concurrent_runs": 6, + "max_concurrent_runs_per_session": 4, + "max_concurrent_provider_steps": 8, + "max_concurrent_provider_steps_per_session": 4, + "max_concurrent_tool_steps": 16, + "max_concurrent_tool_steps_per_session": 8, + "max_pending_inbox_events_per_session": 128, + "inbox_event_ttl_hours": 168, + "max_inbox_delivery_attempts": 8, + "max_user_turn_burst_before_inbox": 4, + "max_inbox_wait_secs": 30 + }, "gateway": { "host": "127.0.0.1", "port": 19876, diff --git a/src/agent/agent_loop.rs b/src/agent/agent_loop.rs index f8cf58b..53b8291 100644 --- a/src/agent/agent_loop.rs +++ b/src/agent/agent_loop.rs @@ -1,6 +1,6 @@ use crate::agent::context_compressor::estimate_tokens; use crate::agent::media_handler::MediaHandlerRegistry; -use crate::agent::steering::SteeringDrain; +use crate::agent::steering::{SteeringDrain, TurnInput}; use crate::agent::system_prompt::build_system_prompt; use crate::agent::turn_event::{AgentTurnContext, TurnEvent}; use crate::bus::message::ContentBlock; @@ -526,18 +526,63 @@ impl AgentLoop { &self.tools } + /// Acquire the provider-step permit for this session when an execution + /// gate is installed. Root Turns are not Agent Runs but still share the + /// gateway-wide step caps, so background runs cannot bypass them. + async fn acquire_provider_permit( + context: &ToolExecutionContext, + cancellation: &tokio_util::sync::CancellationToken, + ) -> Result, AgentError> { + match context.execution_gate.as_ref() { + Some(gate) => gate + .acquire_provider(context.session_id.as_deref(), cancellation) + .await + .map(Some) + .map_err(|_| AgentError::Cancelled), + None => Ok(None), + } + } + + async fn acquire_tool_permit( + context: &ToolExecutionContext, + ) -> Result, AgentError> { + match context.execution_gate.as_ref() { + Some(gate) => gate + .acquire_tool(context.session_id.as_deref(), &context.cancellation) + .await + .map(Some) + .map_err(|_| AgentError::Cancelled), + None => Ok(None), + } + } + async fn stream_completion( &self, request: ChatCompletionRequest, iteration: u32, turn: Option<&AgentTurnContext>, + cancellation: &tokio_util::sync::CancellationToken, ) -> Result { let metrics = crate::observability::metrics::global_metrics(); let provider_name = self.provider.name().to_string(); let provider_model = self.provider.model_id().to_string(); let start = Instant::now(); - let mut provider_stream = match self.provider.stream(request).await { + let stream_result = tokio::select! { + biased; + _ = cancellation.cancelled() => { + metrics.record_provider( + &provider_name, + &provider_model, + None, + start.elapsed().as_millis() as u64, + true, + ); + return Err(AgentError::Cancelled); + } + result = self.provider.stream(request) => result, + }; + let mut provider_stream = match stream_result { Ok(stream) => stream, Err(error) => { tracing::error!(error = %error, "LLM request failed"); @@ -552,7 +597,22 @@ impl AgentLoop { } }; let mut accumulator = ProviderResponseAccumulator::default(); - while let Some(chunk) = provider_stream.next().await { + loop { + let next = tokio::select! { + biased; + _ = cancellation.cancelled() => { + metrics.record_provider( + &provider_name, + &provider_model, + None, + start.elapsed().as_millis() as u64, + true, + ); + return Err(AgentError::Cancelled); + } + chunk = provider_stream.next() => chunk, + }; + let Some(chunk) = next else { break }; let chunk = match chunk { Ok(chunk) => chunk, Err(error) => { @@ -609,30 +669,25 @@ impl AgentLoop { } } - /// Add steering messages to the in-memory transcript in receive order. - /// They remain ordinary `role=user` messages so every provider sees the - /// same conversation semantics and persistence can commit them alongside - /// the rest of this turn. + /// Add steering inputs to the in-memory transcript in receive order. + /// User inputs remain ordinary `role=user` messages so every provider + /// sees the same conversation semantics and persistence can commit them + /// alongside the rest of this turn. Agent steer inputs project to hidden + /// user messages: the client renders the durable Signal projection, not + /// a user bubble, while the model still sees the envelope. fn append_steering_messages( messages: &mut Vec, emitted_messages: &mut Vec, - consumed_steering: &mut Vec, - steering_messages: Vec, + consumed_steering: &mut Vec, + steering_inputs: Vec, turn: &AgentTurnContext, iteration: u32, ) { - for mut message in steering_messages { - // Session routes only ordinary user input to the mailbox. Keep a - // defensive normalization here because the mailbox is public and - // can also be used by embedders/tests. - message.role = "user".to_string(); - if message.turn_id.is_none() { - message.turn_id = Some(turn.turn_id.clone()); - } - if message.iteration.is_none() { - message.iteration = Some(iteration); - } - consumed_steering.push(message.clone()); + for input in steering_inputs { + let message = input + .clone() + .into_chat_message(turn.turn_id.clone(), iteration); + consumed_steering.push(input); messages.push(message.clone()); emitted_messages.push(message); } @@ -644,7 +699,7 @@ impl AgentLoop { } } - fn restore_steering(turn: Option<&AgentTurnContext>, consumed_steering: Vec) { + fn restore_steering(turn: Option<&AgentTurnContext>, consumed_steering: Vec) { if consumed_steering.is_empty() { return; } @@ -753,6 +808,10 @@ impl AgentLoop { tool_context: ToolExecutionContext, ) -> Result { let turn_start = Instant::now(); + let cancellation = tool_context.cancellation.clone(); + if cancellation.is_cancelled() { + return Err(AgentError::Cancelled); + } #[cfg(debug_assertions)] tracing::debug!( @@ -785,6 +844,10 @@ impl AgentLoop { let mut last_request_usage = None; for iteration in 0..self.max_iterations { + if cancellation.is_cancelled() { + Self::restore_steering(turn.as_ref(), consumed_steering); + return Err(AgentError::Cancelled); + } #[cfg(debug_assertions)] tracing::debug!(iteration, "Agent iteration started"); let last_iteration = iteration.saturating_add(1) >= self.max_iterations; @@ -833,18 +896,22 @@ impl AgentLoop { return Err(AgentError::Other("tool iteration exceeds u32".to_string())); } }; - let response = match self - .stream_completion(request, iteration, turn.as_ref()) - .await - { - Ok(response) => response, - Err(error) => { - // The invocation may be retried from persisted history. - // Restore every steering message consumed by an earlier - // boundary; Session decides whether to retry or close - // and queue them after receiving this error. - Self::restore_steering(turn.as_ref(), consumed_steering); - return Err(error); + let response = { + let _provider_permit = + Self::acquire_provider_permit(&tool_context, &cancellation).await?; + match self + .stream_completion(request, iteration, turn.as_ref(), &cancellation) + .await + { + Ok(response) => response, + Err(error) => { + // The invocation may be retried from persisted history. + // Restore every steering message consumed by an earlier + // boundary; Session decides whether to retry or close + // and queue them after receiving this error. + Self::restore_steering(turn.as_ref(), consumed_steering); + return Err(error); + } } }; @@ -1083,10 +1150,13 @@ impl AgentLoop { return Err(AgentError::Other("tool iteration exceeds u32".to_string())); } }; - match self - .stream_completion(request, summary_iteration, turn.as_ref()) - .await - { + let summary_result = { + let _provider_permit = + Self::acquire_provider_permit(&tool_context, &cancellation).await?; + self.stream_completion(request, summary_iteration, turn.as_ref(), &cancellation) + .await + }; + match summary_result { Ok(response) => { accumulated_tokens = accumulated_tokens.saturating_add(response.usage.total_tokens); merge_usage(&mut accumulated_usage, &response.usage); @@ -1202,7 +1272,11 @@ impl AgentLoop { } } - /// Execute tools in parallel using join_all. + /// Execute tools in parallel using join_all. Cancellation wins over a + /// completed batch: once the token fires, PicoBot no longer owns the + /// results. Dropping the tool futures only means PicoBot stops waiting; + /// external side effects are not rolled back and late results must be + /// blocked by execution-ID conditional commits downstream. async fn execute_tools_parallel( &self, tool_calls: &[ToolCall], @@ -1215,10 +1289,13 @@ impl AgentLoop { .map(|tool_call| self.execute_one_tool(tool_call, iteration, turn, context)) .collect(); - futures_util::future::join_all(futures) - .await - .into_iter() - .collect() + tokio::select! { + biased; + _ = context.cancellation.cancelled() => Err(AgentError::Cancelled), + results = futures_util::future::join_all(futures) => { + results.into_iter().collect() + } + } } /// Execute tools sequentially. @@ -1232,10 +1309,15 @@ impl AgentLoop { let mut outcomes = Vec::with_capacity(tool_calls.len()); for tool_call in tool_calls { - outcomes.push( - self.execute_one_tool(tool_call, iteration, turn, context) - .await?, - ); + if context.cancellation.is_cancelled() { + return Err(AgentError::Cancelled); + } + let outcome = tokio::select! { + biased; + _ = context.cancellation.cancelled() => return Err(AgentError::Cancelled), + result = self.execute_one_tool(tool_call, iteration, turn, context) => result?, + }; + outcomes.push(outcome); } Ok(outcomes) @@ -1269,7 +1351,9 @@ impl AgentLoop { }); } + let tool_permit = Self::acquire_tool_permit(context).await?; let result = self.execute_tool_internal(tool_call, context).await; + drop(tool_permit); let duration = start.elapsed(); if let Some(turn) = turn { @@ -1345,7 +1429,9 @@ impl AgentLoop { #[cfg(test)] mod tests { use super::*; - use crate::agent::SteeringMailbox; + use std::time::Duration; + + use crate::agent::TurnMailbox; use crate::observability::{MultiObserver, Observer}; use crate::providers::{ ChatCompletionResponse, FinishReason, ProviderChunk, ProviderStream, Usage, @@ -1517,9 +1603,15 @@ mod tests { PathBuf::from("."), Vec::new(), ); - let mailbox = Arc::new(SteeringMailbox::new()); + let mailbox = Arc::new(TurnMailbox::new()); mailbox - .try_push(ChatMessage::user("please include the log summary")) + .try_push_user(crate::agent::TurnInput::user( + "steer-1", + "please include the log summary", + Vec::new(), + None, + 1_000, + )) .unwrap(); let (controller, emitter, _) = TurnController::start("session", "assistant-id"); let initial = controller.snapshot(); @@ -1565,8 +1657,16 @@ mod tests { PathBuf::from("."), Vec::new(), ); - let mailbox = Arc::new(SteeringMailbox::new()); - mailbox.try_push(ChatMessage::user("retry me")).unwrap(); + let mailbox = Arc::new(TurnMailbox::new()); + mailbox + .try_push_user(crate::agent::TurnInput::user( + "steer-1", + "retry me", + Vec::new(), + None, + 1_000, + )) + .unwrap(); let (controller, emitter, _) = TurnController::start("session", "assistant-id"); let initial = controller.snapshot(); let context = AgentTurnContext::new_with_steering( @@ -1586,7 +1686,7 @@ mod tests { assert!(!mailbox.is_closed()); let restored = mailbox.take_pending(); assert_eq!(restored.len(), 1); - assert_eq!(restored[0].content, "retry me"); + assert_eq!(restored.user_inputs[0].content, "retry me"); } impl TestObserver { @@ -1858,9 +1958,15 @@ mod tests { vec!["text".to_string(), "image".to_string()], ); - let mailbox = Arc::new(SteeringMailbox::new()); + let mailbox = Arc::new(TurnMailbox::new()); mailbox - .try_push(ChatMessage::user("also explain what you found")) + .try_push_user(crate::agent::TurnInput::user( + "steer-2", + "also explain what you found", + Vec::new(), + None, + 1_000, + )) .unwrap(); let (controller, emitter, _) = TurnController::start("session", "assistant-id"); let turn = controller.snapshot(); @@ -2146,20 +2252,301 @@ mod tests { messages.push(ChatMessage::assistant("final")); assert!(!should_include_message_media(&messages, 4)); } + + struct HangingStreamProvider; + + #[async_trait::async_trait] + impl LLMProvider for HangingStreamProvider { + async fn stream( + &self, + _request: ChatCompletionRequest, + ) -> Result { + Ok(Box::pin(futures_util::stream::pending())) + } + + fn ptype(&self) -> &str { + "test" + } + + fn name(&self) -> &str { + "hanging-stream" + } + + fn model_id(&self) -> &str { + "hanging-stream" + } + } + + struct HangingConnectProvider { + connected: std::sync::atomic::AtomicBool, + } + + #[async_trait::async_trait] + impl LLMProvider for HangingConnectProvider { + async fn stream( + &self, + _request: ChatCompletionRequest, + ) -> Result { + self.connected + .store(true, std::sync::atomic::Ordering::SeqCst); + futures_util::future::pending::<()>().await; + unreachable!("connect must be cancelled before completing") + } + + fn ptype(&self) -> &str { + "test" + } + + fn name(&self) -> &str { + "hanging-connect" + } + + fn model_id(&self) -> &str { + "hanging-connect" + } + } + + struct BarrierTool { + barrier: Arc, + read_only: bool, + } + + #[async_trait::async_trait] + impl Tool for BarrierTool { + fn name(&self) -> &str { + "hang" + } + + fn description(&self) -> &str { + "waits until the test releases the barrier" + } + + fn parameters_schema(&self) -> serde_json::Value { + serde_json::json!({ "type": "object" }) + } + + fn read_only(&self) -> bool { + self.read_only + } + + async fn execute(&self, _args: serde_json::Value) -> anyhow::Result { + self.barrier.notified().await; + Ok(ToolResult { + success: true, + output: "late result".to_string(), + error: None, + }) + } + } + + struct ToolCallThenHangProvider { + tool_calls: usize, + } + + #[async_trait::async_trait] + impl LLMProvider for ToolCallThenHangProvider { + async fn stream( + &self, + _request: ChatCompletionRequest, + ) -> Result { + let mut chunks = vec![ProviderChunk::Metadata { + id: "tool-call".into(), + model: "tool-call-hang".into(), + }]; + for index in 0..self.tool_calls { + chunks.push(ProviderChunk::ToolCallStart { + index, + id: Some(format!("call-{index}")), + name: Some("hang".to_string()), + }); + chunks.push(ProviderChunk::ToolCallArguments { + index, + delta: "{}".to_string(), + }); + } + chunks.push(ProviderChunk::Done(FinishReason::ToolCalls)); + Ok(Box::pin(futures_util::stream::iter( + chunks.into_iter().map(Ok), + ))) + } + + fn ptype(&self) -> &str { + "test" + } + + fn name(&self) -> &str { + "tool-call-hang" + } + + fn model_id(&self) -> &str { + "tool-call-hang" + } + } + + fn cancellation_agent(provider: Arc, tools: Arc) -> AgentLoop { + AgentLoop::with_provider_and_tools( + provider, + tools, + 4, + "cancel-test".to_string(), + std::env::temp_dir(), + vec!["text".to_string()], + ) + } + + #[tokio::test] + async fn pre_cancelled_token_returns_cancelled_before_provider() { + let provider = Arc::new(HangingStreamProvider); + let agent = cancellation_agent(provider.clone(), Arc::new(ToolRegistry::new())); + let context = ToolExecutionContext::default(); + context.cancellation.cancel(); + + let error = agent + .process_with_context(vec![ChatMessage::user("work")], context) + .await + .unwrap_err(); + assert!(matches!(error, AgentError::Cancelled)); + } + + #[tokio::test] + async fn cancellation_during_provider_stream_returns_cancelled() { + let agent = cancellation_agent( + Arc::new(HangingStreamProvider), + Arc::new(ToolRegistry::new()), + ); + let context = ToolExecutionContext::default(); + let token = context.cancellation.clone(); + + let process = tokio::spawn(async move { + agent + .process_with_context(vec![ChatMessage::user("work")], context) + .await + }); + tokio::time::sleep(Duration::from_millis(50)).await; + token.cancel(); + + let error = tokio::time::timeout(Duration::from_secs(5), process) + .await + .expect("cancellation must end the provider stream promptly") + .unwrap() + .unwrap_err(); + assert!(matches!(error, AgentError::Cancelled)); + } + + #[tokio::test] + async fn cancellation_during_provider_connect_returns_cancelled() { + let provider = Arc::new(HangingConnectProvider { + connected: std::sync::atomic::AtomicBool::new(false), + }); + let agent = cancellation_agent(provider.clone(), Arc::new(ToolRegistry::new())); + let context = ToolExecutionContext::default(); + let token = context.cancellation.clone(); + + let process = tokio::spawn(async move { + agent + .process_with_context(vec![ChatMessage::user("work")], context) + .await + }); + tokio::time::sleep(Duration::from_millis(50)).await; + token.cancel(); + + let error = tokio::time::timeout(Duration::from_secs(5), process) + .await + .expect("cancellation must interrupt the provider connect") + .unwrap() + .unwrap_err(); + assert!(matches!(error, AgentError::Cancelled)); + assert!(provider.connected.load(std::sync::atomic::Ordering::SeqCst)); + } + + #[tokio::test] + async fn cancellation_during_tool_batch_returns_cancelled_and_discards_late_result() { + let barrier = Arc::new(tokio::sync::Notify::new()); + let tools = Arc::new(ToolRegistry::new()); + tools.register(BarrierTool { + barrier: barrier.clone(), + read_only: false, + }); + let agent = cancellation_agent(Arc::new(ToolCallThenHangProvider { tool_calls: 1 }), tools); + let context = ToolExecutionContext::default(); + let token = context.cancellation.clone(); + + let process = tokio::spawn(async move { + agent + .process_with_context(vec![ChatMessage::user("work")], context) + .await + }); + tokio::time::sleep(Duration::from_millis(100)).await; + token.cancel(); + + let error = tokio::time::timeout(Duration::from_secs(5), process) + .await + .expect("cancellation must end the tool batch promptly") + .unwrap() + .unwrap_err(); + assert!(matches!(error, AgentError::Cancelled)); + + // The cancelled batch dropped its tool futures. Releasing the + // barrier afterwards must not surface the late result anywhere. + barrier.notify_waiters(); + tokio::time::sleep(Duration::from_millis(20)).await; + } + + #[tokio::test] + async fn cancellation_during_parallel_tool_batch_returns_cancelled() { + let barrier = Arc::new(tokio::sync::Notify::new()); + let tools = Arc::new(ToolRegistry::new()); + tools.register(BarrierTool { + barrier: barrier.clone(), + read_only: true, + }); + let agent = cancellation_agent(Arc::new(ToolCallThenHangProvider { tool_calls: 2 }), tools); + let context = ToolExecutionContext::default(); + let token = context.cancellation.clone(); + + let process = tokio::spawn(async move { + agent + .process_with_context(vec![ChatMessage::user("work")], context) + .await + }); + tokio::time::sleep(Duration::from_millis(100)).await; + token.cancel(); + + let error = tokio::time::timeout(Duration::from_secs(5), process) + .await + .expect("cancellation must end the parallel tool batch promptly") + .unwrap() + .unwrap_err(); + assert!(matches!(error, AgentError::Cancelled)); + } } #[derive(Debug)] pub enum AgentError { ProviderCreation(String), LlmError(String), + /// The run was cancelled by `/stop`, a parent run, timeout ownership or + /// shutdown. Terminal state must be decided by this variant, never by + /// matching error strings. + Cancelled, + /// The run exceeded its deadline. + TimedOut, Other(String), } +impl AgentError { + pub fn is_cancelled(&self) -> bool { + matches!(self, AgentError::Cancelled) + } +} + impl std::fmt::Display for AgentError { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { match self { AgentError::ProviderCreation(e) => write!(f, "Provider creation error: {}", e), AgentError::LlmError(e) => write!(f, "LLM error: {}", e), + AgentError::Cancelled => write!(f, "agent run cancelled"), + AgentError::TimedOut => write!(f, "agent run timed out"), AgentError::Other(e) => write!(f, "{}", e), } } diff --git a/src/agent/catalog.rs b/src/agent/catalog.rs new file mode 100644 index 0000000..3fd6acb --- /dev/null +++ b/src/agent/catalog.rs @@ -0,0 +1,440 @@ +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; +use std::path::{Path, PathBuf}; +use std::sync::Arc; + +use crate::config::{AgentOrchestrationConfig, LLMProviderConfig, expand_path}; +use crate::skills::SkillsLoader; +use crate::tools::{DelegationPolicy, ToolRegistry}; + +use super::definition::{AgentDefinition, AgentDefinitionError, parse_definition}; + +#[derive(Debug, thiserror::Error)] +pub enum AgentCatalogError { + #[error("invalid Agent orchestration config: {0}")] + Config(String), + #[error(transparent)] + Definition(#[from] AgentDefinitionError), + #[error("Agent definition directory error: {0}")] + Directory(String), + #[error("Agent '{agent}' references unknown Provider profile '{profile}'")] + UnknownProfile { agent: String, profile: String }, + #[error("Agent '{agent}' references invalid tool '{tool}': {reason}")] + InvalidTool { + agent: String, + tool: String, + reason: String, + }, + #[error("Agent '{agent}' references unknown delegate '{target}'")] + UnknownDelegate { agent: String, target: String }, + #[error("Agent '{agent}' references unknown skill '{skill}'")] + UnknownSkill { agent: String, skill: String }, +} + +#[derive(Debug)] +pub struct AgentCatalog { + definitions: BTreeMap>, + root_delegates: BTreeSet, + runtime_generation: u64, + enabled: bool, + max_tree_depth: u16, + max_runs_per_tree: usize, +} + +impl AgentCatalog { + pub fn legacy() -> Self { + Self { + definitions: BTreeMap::new(), + root_delegates: BTreeSet::new(), + runtime_generation: 0, + enabled: false, + max_tree_depth: 4, + max_runs_per_tree: 16, + } + } + + pub fn load( + config: &AgentOrchestrationConfig, + config_dir: &Path, + provider_profiles: &HashMap, + tools: &ToolRegistry, + skills_loader: &SkillsLoader, + runtime_generation: u64, + ) -> Result { + config.validate().map_err(AgentCatalogError::Config)?; + if !config.enabled { + return Ok(Self::legacy()); + } + + let trusted_root = config_dir.canonicalize().map_err(|error| { + AgentCatalogError::Directory(format!("{}: {error}", config_dir.display())) + })?; + let configured = expand_path(&config.definitions_dir); + let definitions_dir = if configured.is_absolute() { + configured + } else { + trusted_root.join(configured) + }; + let definitions_dir = definitions_dir.canonicalize().map_err(|error| { + AgentCatalogError::Directory(format!("{}: {error}", definitions_dir.display())) + })?; + if !definitions_dir.starts_with(&trusted_root) { + return Err(AgentCatalogError::Directory(format!( + "{} escapes trusted config directory {}", + definitions_dir.display(), + trusted_root.display() + ))); + } + + let mut paths = definition_paths(&definitions_dir)?; + paths.sort(); + let loaded_skills: HashSet = skills_loader + .list_skills() + .into_iter() + .map(|(name, _)| name) + .collect(); + let mut definitions = BTreeMap::new(); + + for path in paths { + let yaml = read_profile_name(&path)?; + let provider = provider_profiles.get(&yaml).cloned().ok_or_else(|| { + AgentCatalogError::UnknownProfile { + agent: path.display().to_string(), + profile: yaml.clone(), + } + })?; + let definition = Arc::new(parse_definition(&path, Arc::new(provider))?); + if definitions.contains_key(&definition.id) { + return Err(AgentCatalogError::Config(format!( + "duplicate Agent id '{}'", + definition.id + ))); + } + validate_definition_tools(&definition, tools)?; + for skill in &definition.skills { + if !loaded_skills.contains(skill) { + return Err(AgentCatalogError::UnknownSkill { + agent: definition.id.clone(), + skill: skill.clone(), + }); + } + } + if !definition.skills.is_empty() + && !definition.tools.iter().any(|tool| tool == "get_skill") + { + return Err(AgentCatalogError::InvalidTool { + agent: definition.id.clone(), + tool: "get_skill".to_string(), + reason: "skills require get_skill in the definition tool list".to_string(), + }); + } + definitions.insert(definition.id.clone(), definition); + } + + for definition in definitions.values() { + for target in &definition.delegates { + if !definitions.contains_key(target) { + return Err(AgentCatalogError::UnknownDelegate { + agent: definition.id.clone(), + target: target.clone(), + }); + } + } + } + + let root_delegates: BTreeSet<_> = config.root_delegates.iter().cloned().collect(); + if root_delegates.len() != config.root_delegates.len() { + return Err(AgentCatalogError::Config( + "root_delegates contains duplicates".to_string(), + )); + } + for target in &root_delegates { + if !definitions.contains_key(target) { + return Err(AgentCatalogError::UnknownDelegate { + agent: "ROOT".to_string(), + target: target.clone(), + }); + } + } + + Ok(Self { + definitions, + root_delegates, + runtime_generation, + enabled: true, + max_tree_depth: config.max_tree_depth, + max_runs_per_tree: config.max_runs_per_tree, + }) + } + + pub fn enabled(&self) -> bool { + self.enabled + } + + pub fn runtime_generation(&self) -> u64 { + self.runtime_generation + } + + pub fn max_tree_depth(&self) -> u16 { + self.max_tree_depth + } + + pub fn max_runs_per_tree(&self) -> usize { + self.max_runs_per_tree + } + + pub fn get(&self, id: &str) -> Option> { + self.definitions.get(id).cloned() + } + + pub fn root_can_delegate(&self, target: &str) -> bool { + self.root_delegates.contains(target) + } + + pub fn can_delegate(&self, caller: &str, target: &str) -> bool { + self.definitions + .get(caller) + .is_some_and(|definition| definition.delegates.iter().any(|id| id == target)) + } + + pub fn root_targets(&self) -> Vec> { + self.root_delegates + .iter() + .filter_map(|id| self.get(id)) + .collect() + } +} + +fn definition_paths(directory: &Path) -> Result, AgentCatalogError> { + let mut paths = Vec::new(); + let entries = std::fs::read_dir(directory).map_err(|error| { + AgentCatalogError::Directory(format!("{}: {error}", directory.display())) + })?; + for entry in entries { + let entry = entry.map_err(|error| AgentCatalogError::Directory(error.to_string()))?; + let path = entry.path(); + if path.extension().and_then(|value| value.to_str()) != Some("md") { + continue; + } + let canonical = path.canonicalize().map_err(|error| { + AgentCatalogError::Directory(format!("{}: {error}", path.display())) + })?; + if !canonical.starts_with(directory) { + return Err(AgentCatalogError::Directory(format!( + "{} escapes definitions directory", + path.display() + ))); + } + paths.push(path); + } + Ok(paths) +} + +fn read_profile_name(path: &Path) -> Result { + let metadata = std::fs::symlink_metadata(path).map_err(|source| AgentDefinitionError::Io { + path: path.display().to_string(), + source, + })?; + if !metadata.file_type().is_file() || metadata.file_type().is_symlink() { + return Err(AgentDefinitionError::Invalid(format!( + "{} must be a regular non-symlink file", + path.display() + )) + .into()); + } + if metadata.len() > super::definition::MAX_DEFINITION_FILE_BYTES { + return Err(AgentDefinitionError::Invalid(format!( + "{} exceeds the {} byte limit", + path.display(), + super::definition::MAX_DEFINITION_FILE_BYTES + )) + .into()); + } + let content = std::fs::read_to_string(path).map_err(|source| AgentDefinitionError::Io { + path: path.display().to_string(), + source, + })?; + let normalized = content.replace("\r\n", "\n"); + let rest = normalized.strip_prefix("---\n").ok_or_else(|| { + AgentDefinitionError::Invalid(format!( + "{} must start with a standalone --- line", + path.display() + )) + })?; + let (yaml, _) = rest.split_once("\n---\n").ok_or_else(|| { + AgentDefinitionError::Invalid(format!( + "{} has no closing frontmatter delimiter", + path.display() + )) + })?; + #[derive(serde::Deserialize)] + struct ProfileOnly { + llm_profile: String, + } + let parsed: ProfileOnly = serde_yaml::from_str(yaml).map_err(AgentDefinitionError::Yaml)?; + Ok(parsed.llm_profile) +} + +fn validate_definition_tools( + definition: &AgentDefinition, + tools: &ToolRegistry, +) -> Result<(), AgentCatalogError> { + for name in &definition.tools { + let tool = tools + .get(name) + .ok_or_else(|| AgentCatalogError::InvalidTool { + agent: definition.id.clone(), + tool: name.clone(), + reason: "tool is not registered in the prepared runtime".to_string(), + })?; + let allowed = tool.delegation_policy() == DelegationPolicy::Delegatable + || (name == "get_skill" + && tool.delegation_policy() == DelegationPolicy::RuntimeInjected); + if !allowed { + return Err(AgentCatalogError::InvalidTool { + agent: definition.id.clone(), + tool: name.clone(), + reason: format!("policy is {:?}", tool.delegation_policy()), + }); + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::tools::{CalculatorTool, FileWriteTool}; + + fn provider() -> LLMProviderConfig { + LLMProviderConfig { + provider_type: "openai".to_string(), + name: "test".to_string(), + base_url: "https://example.invalid/v1".to_string(), + api_key: "test".to_string(), + extra_headers: HashMap::new(), + model_id: "test-model".to_string(), + temperature: None, + max_tokens: None, + model_extra: HashMap::new(), + max_tool_iterations: 99, + token_limit: 4096, + workspace_dir: std::env::temp_dir(), + input_types: vec!["text".to_string()], + price_input_per_million: None, + price_output_per_million: None, + } + } + + fn config() -> AgentOrchestrationConfig { + AgentOrchestrationConfig { + enabled: true, + definitions_dir: "agents".to_string(), + root_delegates: vec!["researcher".to_string()], + ..Default::default() + } + } + + fn write_agent(root: &Path, id: &str, tools: &[&str], delegates: &[&str]) { + let tools = (!tools.is_empty()).then(|| { + format!( + "tools:\n{}\n", + tools + .iter() + .map(|name| format!(" - {name}")) + .collect::>() + .join("\n") + ) + }); + let delegates = (!delegates.is_empty()).then(|| { + format!( + "delegates:\n{}\n", + delegates + .iter() + .map(|name| format!(" - {name}")) + .collect::>() + .join("\n") + ) + }); + std::fs::write( + root.join("agents").join(format!("{id}.md")), + format!( + "---\nid: {id}\ndescription: {id} role\nllm_profile: research\n{}{}---\n# Role\n\nDo the assigned work.\n", + tools.unwrap_or_default(), + delegates.unwrap_or_default() + ), + ) + .unwrap(); + } + + #[test] + fn catalog_loads_provider_tools_and_delegation_graph() { + let root = tempfile::tempdir().unwrap(); + std::fs::create_dir(root.path().join("agents")).unwrap(); + write_agent(root.path(), "researcher", &["calculator"], &["reviewer"]); + write_agent(root.path(), "reviewer", &["calculator"], &[]); + let tools = ToolRegistry::new(); + tools.register(CalculatorTool::new()); + let loader = SkillsLoader::new_for_testing( + root.path().join("skills"), + root.path().join("external-skills"), + ); + let profiles = HashMap::from([("research".to_string(), provider())]); + + let catalog = + AgentCatalog::load(&config(), root.path(), &profiles, &tools, &loader, 7).unwrap(); + + assert!(catalog.root_can_delegate("researcher")); + assert!(catalog.can_delegate("researcher", "reviewer")); + assert_eq!( + catalog.get("researcher").unwrap().provider_config.model_id, + "test-model" + ); + assert_eq!(catalog.runtime_generation(), 7); + } + + #[test] + fn catalog_rejects_root_only_tool() { + let root = tempfile::tempdir().unwrap(); + std::fs::create_dir(root.path().join("agents")).unwrap(); + write_agent(root.path(), "researcher", &["file_write"], &[]); + let tools = ToolRegistry::new(); + tools.register(FileWriteTool::new()); + let loader = SkillsLoader::new_for_testing( + root.path().join("skills"), + root.path().join("external-skills"), + ); + let profiles = HashMap::from([("research".to_string(), provider())]); + + let error = + AgentCatalog::load(&config(), root.path(), &profiles, &tools, &loader, 1).unwrap_err(); + + assert!(matches!(error, AgentCatalogError::InvalidTool { .. })); + } + + #[cfg(unix)] + #[test] + fn catalog_rejects_definition_symlink_escape() { + use std::os::unix::fs::symlink; + + let root = tempfile::tempdir().unwrap(); + let outside = tempfile::tempdir().unwrap(); + std::fs::create_dir(root.path().join("agents")).unwrap(); + let outside_file = outside.path().join("researcher.md"); + std::fs::write( + &outside_file, + "---\nid: researcher\ndescription: test\nllm_profile: research\n---\n# Role\n", + ) + .unwrap(); + symlink(&outside_file, root.path().join("agents/researcher.md")).unwrap(); + let tools = ToolRegistry::new(); + let loader = SkillsLoader::new_for_testing( + root.path().join("skills"), + root.path().join("external-skills"), + ); + let profiles = HashMap::from([("research".to_string(), provider())]); + + assert!( + AgentCatalog::load(&config(), root.path(), &profiles, &tools, &loader, 1,).is_err() + ); + } +} diff --git a/src/agent/coordinator.rs b/src/agent/coordinator.rs new file mode 100644 index 0000000..81bb855 --- /dev/null +++ b/src/agent/coordinator.rs @@ -0,0 +1,1497 @@ +use std::sync::Arc; + +use dashmap::DashMap; +use tokio_util::sync::CancellationToken; +use uuid::Uuid; + +use crate::agent::SubAgentManager; +use crate::agent::inbox::AgentInboxNotifier; +use crate::agent::projection::{AgentProjection, AgentProjectionHub}; +use crate::agent::run::AgentExecutionContext; +use crate::agent::sub_agent::{ + ExecutionMode, ResolvedAgentRun, SubAgentConfig, SubAgentError, SubAgentResult, TaskStatus, +}; +use crate::storage::Storage; +use crate::storage::agent_inbox::AgentEventType; +use crate::storage::agent_run::{ + AcceptAgentRequest, AcceptedAgentRuns, AgentCompletionPolicy, AgentRunMode, AgentRunRecord, + AgentRunStatus, AgentTerminalOutcome, NewAgentGroup, NewAgentRun, +}; +use crate::tools::ToolExecutionContext; +use crate::tools::emit_signal::{SignalAccepted, SignalAcceptedStatus, SignalInput}; + +/// Durable Agent orchestration. Every named run is persisted in +/// `agent_runs` before execution, transitions are execution-ID conditional, +/// and the terminal commit is the single writer of final state. Background +/// runs reserve an inbox completion slot at admission; their completion event +/// is materialized by the terminal commit and delivered through the Session +/// continuation lane instead of a direct channel notification. +pub struct AgentCoordinator { + storage: Arc, + manager: Arc, + work_manager: Option>, + notifier: Arc, + projection: Arc, + task_supervisor: crate::task_supervisor::TaskSupervisor, + runtime_generation: i64, + max_pending_inbox_events_per_session: i64, + max_inbox_delivery_attempts: i64, + active_tokens: DashMap, +} + +#[derive(Debug, thiserror::Error)] +pub enum CoordinatorError { + #[error("agent orchestration rejected the request: {0}")] + Rejected(String), + #[error("agent run storage error: {0}")] + Storage(#[from] crate::storage::StorageError), + #[error(transparent)] + SubAgent(#[from] SubAgentError), +} + +impl AgentCoordinator { + #[allow(clippy::too_many_arguments)] + pub fn new( + storage: Arc, + manager: Arc, + work_manager: Arc, + notifier: Arc, + projection: Arc, + task_supervisor: crate::task_supervisor::TaskSupervisor, + runtime_generation: u64, + orchestration: &crate::config::AgentOrchestrationConfig, + ) -> Arc { + Arc::new(Self { + storage, + manager, + work_manager: Some(work_manager), + notifier, + projection, + task_supervisor, + runtime_generation: runtime_generation as i64, + max_pending_inbox_events_per_session: orchestration.max_pending_inbox_events_per_session + as i64, + max_inbox_delivery_attempts: i64::from(orchestration.max_inbox_delivery_attempts), + active_tokens: DashMap::new(), + }) + } + + /// Admit a named background run for the root caller and spawn its runner. + /// Completion is guaranteed by the reserved inbox slot; the returned ID + /// is only valid when every durable step succeeded. + pub async fn delegate_background( + self: &Arc, + caller: &ToolExecutionContext, + config: SubAgentConfig, + ) -> Result { + if caller.agent.is_some() { + return Err(CoordinatorError::Rejected( + "nested background runs are not available yet; only the root Agent may delegate background work".to_string(), + )); + } + if config.target.is_none() { + return Err(CoordinatorError::Rejected( + "legacy general Agent is not persisted; named background targets only".to_string(), + )); + } + let run_id = Uuid::new_v4().to_string(); + let resolved = self.manager.resolve_agent(&config, caller, &run_id)?; + let root_session_id = caller + .session_id + .clone() + .or_else(|| { + caller + .agent + .as_ref() + .map(|agent| agent.root_session_id.clone()) + }) + .ok_or_else(|| { + CoordinatorError::Rejected("delegate requires a session-bound context".to_string()) + })?; + let now = chrono::Utc::now().timestamp_millis(); + + // 1. Reserve the completion slot; failure means the inbox is full and + // nothing is admitted. + if self + .storage + .reserve_completion_slots( + &root_session_id, + 1, + self.max_pending_inbox_events_per_session, + now, + ) + .await? + .is_none() + { + return Err(CoordinatorError::Rejected( + "inbox capacity exceeded; cannot accept another background run".to_string(), + )); + } + + // 2. Persist the queued run atomically with the reservation. + let accept = NewAgentRun { + id: run_id.clone(), + root_session_id: root_session_id.clone(), + root_turn_id: caller.turn_id.clone(), + parent_run_id: None, + caller_agent_id: "ROOT".to_string(), + caller_scope_id: caller.turn_id.clone().unwrap_or_else(|| "root".to_string()), + idempotency_key: None, + agent_id: resolved.agent_id.clone().unwrap_or_default(), + definition_hash: resolved.definition_hash.clone().unwrap_or_default(), + provider_profile: resolved.llm_profile.clone().unwrap_or_default(), + provider_name: resolved.provider_config.name.clone(), + model_id: resolved.provider_config.model_id.clone(), + mode: AgentRunMode::Background, + depth: 1, + plan_item_id: config.plan_item_id.clone(), + execution_id: run_id.clone(), + task: config.prompt.clone(), + context_json: config.context.clone(), + budget_json: serde_json::json!({ + "remaining_runs": self.manager.catalog().max_runs_per_tree(), + "remaining_depth": self.manager.catalog().max_tree_depth(), + }) + .to_string(), + deadline_at: now + (resolved.timeout_secs * 1000) as i64, + runtime_generation: self.runtime_generation, + completion_slot_reserved: true, + }; + match self + .storage + .accept_agent_runs(AcceptAgentRequest { + group: None, + runs: vec![accept], + now, + }) + .await? + { + AcceptedAgentRuns::Accepted { .. } => {} + AcceptedAgentRuns::Existing { .. } => { + self.storage + .release_completion_slots(&root_session_id, 1, now) + .await?; + return Err(CoordinatorError::Rejected( + "background admission conflicted with an existing run id".to_string(), + )); + } + } + + // 3. Register the cancellation token and spawn the runner. The + // activity guard and run quota are held for the whole run, not + // released when `delegate_background` returns. + let token = CancellationToken::new(); + self.active_tokens.insert(run_id.clone(), token.clone()); + let coordinator = self.clone(); + let config = config.clone(); + let spawned = self + .task_supervisor + .spawn_graceful(format!("agent-run:{run_id}"), { + let run_id = run_id.clone(); + async move { + coordinator + .run_background_runner(&run_id, &config, resolved, token) + .await; + } + }); + if !spawned { + // Compensation: undo the durable admission before returning. + self.active_tokens.remove(&run_id); + let _ = self + .storage + .cancel_agent_run_with_completion(&run_id, "gateway shutdown", true, now) + .await; + return Err(CoordinatorError::Rejected( + "gateway is shutting down and cannot accept background tasks".to_string(), + )); + } + Ok(run_id) + } + + async fn run_background_runner( + self: &Arc, + run_id: &str, + config: &SubAgentConfig, + resolved: ResolvedAgentRun, + token: CancellationToken, + ) { + let now = chrono::Utc::now().timestamp_millis(); + let execution_id = run_id.to_string(); + if !self + .storage + .mark_agent_run_running(run_id, &execution_id, now) + .await + .unwrap_or(false) + { + // Cancelled before start; the canceller already resolved the + // reservation and completion. + self.active_tokens.remove(run_id); + return; + } + + let result = self + .manager + .execute_resolved(config, resolved, run_id) + .await; + + let outcome = match &result { + Ok(result) => match &result.status { + TaskStatus::Completed => AgentTerminalOutcome::Completed { + result: result.full_content.clone(), + prompt_tokens: None, + completion_tokens: None, + cost: None, + tool_calls: result.tool_calls_count as i64, + iterations: result.iterations as i64, + }, + TaskStatus::Failed(error) => AgentTerminalOutcome::Failed { + error: error.clone(), + prompt_tokens: None, + completion_tokens: None, + cost: None, + }, + TaskStatus::TimedOut => AgentTerminalOutcome::TimedOut { + deadline_at: chrono::Utc::now().timestamp_millis(), + }, + TaskStatus::Cancelled => AgentTerminalOutcome::Cancelled { + reason: "cancelled by user, parent or shutdown".to_string(), + }, + }, + Err(error) => AgentTerminalOutcome::Failed { + error: error.to_string(), + prompt_tokens: None, + completion_tokens: None, + cost: None, + }, + }; + + let summary = result + .as_ref() + .ok() + .map(|result| truncate_summary(&result.full_content)); + match self + .storage + .commit_agent_terminal( + run_id, + &execution_id, + self.runtime_generation, + &outcome, + summary.as_deref(), + chrono::Utc::now().timestamp_millis(), + ) + .await + { + Ok(Some(commit)) => { + if let Some(plan_item_id) = commit.run.plan_item_id.clone() { + self.refresh_work_plan(&commit.run.root_session_id, Some(plan_item_id)); + } + // The completion event (if any) was committed; wake the + // session so it claims the inbox soon. A lost wake is not + // fatal — the next claim pass finds the event anyway. + self.projection.publish(AgentProjection { + session_id: commit.run.root_session_id.clone(), + revision: commit.run.revision, + run: Some(crate::protocol::AgentRunView::from_record( + &commit.run, + 2000, + )), + event: None, + }); + self.notifier + .notify(&commit.run.root_session_id, commit.run.revision) + .await; + } + Ok(None) => { + tracing::warn!( + run_id, + "late background result discarded by terminal commit" + ); + } + Err(error) => { + tracing::error!(run_id, error = %error, "background terminal commit failed"); + } + } + self.active_tokens.remove(run_id); + drop(token); + } + + /// Cancel every nonterminal run of a session (archive/delete or `/stop`). + /// Completion events are written consumed so no continuation starts after + /// the session was closed. + pub async fn cancel_session( + &self, + session_id: &str, + reason: &str, + ) -> Result { + let runs = self.storage.list_agent_runs(session_id, None, 200).await?; + let mut count = 0; + for run in runs { + if run.status.is_terminal() { + continue; + } + if self + .storage + .cancel_agent_run_with_completion( + &run.id, + reason, + true, + chrono::Utc::now().timestamp_millis(), + ) + .await? + { + if let Some((_, token)) = self.active_tokens.remove(&run.id) { + token.cancel(); + } + count += 1; + } + } + Ok(count) + } + + /// Startup/activation recovery: interrupt runs of older generations, + /// expire stale leases, converge group counters and reconcile the + /// per-session capacity rows. Safe to call once per activation. + pub async fn recover_on_activation( + &self, + ) -> Result { + let now = chrono::Utc::now().timestamp_millis(); + let report = self + .storage + .recover_agent_state( + self.runtime_generation, + now, + self.max_inbox_delivery_attempts, + 60_000, + ) + .await?; + // One merged wake per session with due events; the notifier is the + // accelerator and a dead target is not an error (the periodic timer + // inside each live worker re-claims anyway). + if report.interrupted_runs > 0 || report.leases_expired > 0 { + for session_id in self.storage.sessions_with_due_events(now).await? { + self.notifier.notify(&session_id, now).await; + } + } + Ok(report) + } + + /// Execute a foreground delegation batch with durable run persistence. + /// Results keep request order even though runs execute concurrently. + pub async fn delegate_foreground( + self: &Arc, + caller: &ToolExecutionContext, + configs: Vec, + ) -> Result, CoordinatorError> { + if configs.is_empty() { + return Err(CoordinatorError::Rejected( + "foreground delegation requires at least one task".to_string(), + )); + } + if configs + .iter() + .any(|config| config.mode != ExecutionMode::Foreground) + { + return Err(CoordinatorError::Rejected( + "coordinator foreground path received a non-foreground request".to_string(), + )); + } + + // Resolve every target before persisting anything so a bad request + // fails closed without leaving orphan rows. + let mut run_ids = Vec::with_capacity(configs.len()); + let mut resolved: Vec = Vec::with_capacity(configs.len()); + for config in &configs { + if config.target.is_none() { + return Err(CoordinatorError::Rejected( + "legacy general Agent is not persisted; named targets only".to_string(), + )); + } + let run_id = Uuid::new_v4().to_string(); + resolved.push(self.manager.resolve_agent(config, caller, &run_id)?); + run_ids.push(run_id); + } + + let root_session_id = caller + .agent + .as_ref() + .map(|agent| agent.root_session_id.clone()) + .or_else(|| caller.session_id.clone()) + .ok_or_else(|| { + CoordinatorError::Rejected("delegate requires a session-bound context".to_string()) + })?; + let now = chrono::Utc::now().timestamp_millis(); + let caller_scope_id = caller + .turn_id + .clone() + .or_else(|| caller.agent.as_ref().map(|agent| agent.run_id.clone())) + .unwrap_or_else(|| "root".to_string()); + + let group = (configs.len() > 1).then(|| NewAgentGroup { + id: Uuid::new_v4().to_string(), + root_session_id: root_session_id.clone(), + caller_run_id: caller.agent.as_ref().map(|agent| agent.run_id.clone()), + caller_scope_id: caller_scope_id.clone(), + idempotency_key: None, + mode: AgentRunMode::Foreground, + completion_policy: AgentCompletionPolicy::All, + deadline_at: now + + resolved + .iter() + .map(|run| (run.timeout_secs * 1000) as i64) + .max() + .unwrap_or(0), + runtime_generation: self.runtime_generation, + }); + + let mut runs = Vec::with_capacity(configs.len()); + for (index, config) in configs.iter().enumerate() { + let resolution = &resolved[index]; + let deadline_at = now + (resolution.timeout_secs * 1000) as i64; + runs.push(NewAgentRun { + id: run_ids[index].clone(), + root_session_id: root_session_id.clone(), + root_turn_id: caller.turn_id.clone(), + parent_run_id: caller.agent.as_ref().map(|agent| agent.run_id.clone()), + caller_agent_id: caller + .agent + .as_ref() + .map(|agent| agent.current_agent_id.clone()) + .unwrap_or_else(|| "ROOT".to_string()), + caller_scope_id: caller_scope_id.clone(), + idempotency_key: None, + agent_id: resolution.agent_id.clone().unwrap_or_default(), + definition_hash: resolution.definition_hash.clone().unwrap_or_default(), + provider_profile: resolution.llm_profile.clone().unwrap_or_default(), + provider_name: resolution.provider_config.name.clone(), + model_id: resolution.provider_config.model_id.clone(), + mode: AgentRunMode::Foreground, + depth: caller + .agent + .as_ref() + .map_or(1, |agent| agent.depth.saturating_add(1) as i64), + plan_item_id: config.plan_item_id.clone(), + execution_id: run_ids[index].clone(), + task: config.prompt.clone(), + context_json: config.context.clone(), + budget_json: serde_json::to_string(&serde_json::json!({ + "remaining_runs": caller.agent.as_ref().map(|agent| agent.budget.remaining_runs), + "remaining_depth": caller.agent.as_ref().map(|agent| agent.budget.remaining_depth), + })) + .unwrap_or_default(), + deadline_at, + runtime_generation: self.runtime_generation, + completion_slot_reserved: false, + }); + } + + match self + .storage + .accept_agent_runs(AcceptAgentRequest { group, runs, now }) + .await? + { + AcceptedAgentRuns::Accepted { .. } => {} + AcceptedAgentRuns::Existing { .. } => { + return Err(CoordinatorError::Rejected( + "foreground delegation conflicted with an existing run id".to_string(), + )); + } + } + + // A named parent waits structurally for its children: it moves to + // waiting_children and holds no step permits while waiting. + let parent = caller.agent.clone(); + if let Some(parent) = parent.as_ref() { + let _ = self + .storage + .mark_agent_run_waiting_children( + &parent.run_id, + &parent.execution_id, + AgentRunStatus::Running, + chrono::Utc::now().timestamp_millis(), + ) + .await; + } + + let futures: Vec<_> = configs + .iter() + .enumerate() + .map(|(index, config)| { + let coordinator = self.clone(); + let run_id = run_ids[index].clone(); + let resolution = resolved[index].clone(); + let config = config.clone(); + async move { coordinator.execute_run(&run_id, &config, resolution).await } + }) + .collect(); + let results = futures_util::future::join_all(futures) + .await + .into_iter() + .enumerate() + .map(|(index, result)| { + result.unwrap_or_else(|error| SubAgentResult { + task_id: run_ids[index].clone(), + content: String::new(), + content_truncated: false, + full_content: String::new(), + status: TaskStatus::Failed(error.to_string()), + tool_calls_count: 0, + iterations: 0, + duration_ms: 0, + }) + }) + .collect(); + + if let Some(parent) = parent.as_ref() { + let _ = self + .storage + .restore_agent_run_running( + &parent.run_id, + &parent.execution_id, + chrono::Utc::now().timestamp_millis(), + ) + .await; + } + + Ok(results) + } + + async fn execute_run( + self: &Arc, + run_id: &str, + config: &SubAgentConfig, + resolution: ResolvedAgentRun, + ) -> Result { + let execution_id = run_id.to_string(); + let token = resolution.tool_context.cancellation.clone(); + self.active_tokens.insert(run_id.to_string(), token); + + let started = self + .storage + .mark_agent_run_running(run_id, &execution_id, chrono::Utc::now().timestamp_millis()) + .await?; + if !started { + self.active_tokens.remove(run_id); + let status = self + .storage + .get_agent_run(run_id) + .await? + .map(|run| run.status) + .unwrap_or(AgentRunStatus::Cancelled); + return Ok(SubAgentResult { + task_id: run_id.to_string(), + content: String::new(), + content_truncated: false, + full_content: String::new(), + status: match status { + AgentRunStatus::Cancelled => TaskStatus::Cancelled, + AgentRunStatus::TimedOut => TaskStatus::TimedOut, + AgentRunStatus::Interrupted => { + TaskStatus::Failed("run interrupted before start".to_string()) + } + _ => TaskStatus::Failed("run was closed before execution started".to_string()), + }, + tool_calls_count: 0, + iterations: 0, + duration_ms: 0, + }); + } + + let result = self + .manager + .execute_resolved(config, resolution, run_id) + .await; + + let outcome_result = match &result { + Ok(result) => match &result.status { + TaskStatus::Completed => AgentTerminalOutcome::Completed { + result: result.full_content.clone(), + prompt_tokens: None, + completion_tokens: None, + cost: None, + tool_calls: result.tool_calls_count as i64, + iterations: result.iterations as i64, + }, + TaskStatus::Failed(error) => AgentTerminalOutcome::Failed { + error: error.clone(), + prompt_tokens: None, + completion_tokens: None, + cost: None, + }, + TaskStatus::TimedOut => AgentTerminalOutcome::TimedOut { + deadline_at: chrono::Utc::now().timestamp_millis(), + }, + TaskStatus::Cancelled => AgentTerminalOutcome::Cancelled { + reason: "cancelled by user, parent or shutdown".to_string(), + }, + }, + Err(error) => AgentTerminalOutcome::Failed { + error: error.to_string(), + prompt_tokens: None, + completion_tokens: None, + cost: None, + }, + }; + + let summary = result + .as_ref() + .ok() + .map(|result| truncate_summary(&result.full_content)); + let commit = self + .storage + .commit_agent_terminal( + run_id, + &execution_id, + self.runtime_generation, + &outcome_result, + summary.as_deref(), + chrono::Utc::now().timestamp_millis(), + ) + .await?; + self.active_tokens.remove(run_id); + match commit { + Some(commit) => { + if commit.run.plan_item_id.is_some() { + self.refresh_work_plan( + &commit.run.root_session_id, + commit.run.plan_item_id.clone(), + ); + } + } + None => { + tracing::warn!(run_id, "late agent run result discarded by terminal commit"); + } + } + + result.map_err(CoordinatorError::SubAgent) + } + + /// The plan item was already mutated inside the Storage transaction; this + /// only re-reads, refreshes the WorkManager cache and broadcasts. + fn refresh_work_plan(&self, session_id: &str, item_id: Option) { + let Some(work_manager) = self.work_manager.as_ref() else { + return; + }; + let Some(item_id) = item_id else { + return; + }; + let work_manager = work_manager.clone(); + let session_id = session_id.to_string(); + tokio::spawn(async move { + if let Err(error) = work_manager + .refresh_after_external_commit(&session_id, "agent_run", vec![item_id]) + .await + { + tracing::warn!(error = %error, "failed to refresh plan after agent run commit"); + } + }); + } + + /// Cancel a nonterminal run owned by the caller's session/tree. + pub async fn cancel_run( + &self, + caller: &ToolExecutionContext, + run_id: &str, + reason: &str, + ) -> Result { + self.cancel_run_inner(caller, run_id, reason, false).await + } + + /// `suppress_continuation` writes the completion event consumed so no + /// continuation Turn restarts after lifecycle cancellation (`/stop`). + pub(crate) async fn cancel_run_inner( + &self, + caller: &ToolExecutionContext, + run_id: &str, + reason: &str, + suppress_continuation: bool, + ) -> Result { + let Some(run) = self.storage.get_agent_run(run_id).await? else { + return Ok(false); + }; + self.authorize_access(caller, &run).await?; + if run.status.is_terminal() { + return Ok(false); + } + let cancelled = self + .storage + .cancel_agent_run_with_completion( + run_id, + reason, + suppress_continuation, + chrono::Utc::now().timestamp_millis(), + ) + .await?; + if cancelled { + if let Some((_, token)) = self.active_tokens.remove(run_id) { + token.cancel(); + } + // An explicit cancel supersedes the run's unconsumed signals: + // they remain as audit facts but no continuation will report + // them. Completions are never superseded. + if let Err(error) = self + .storage + .supersede_agent_events( + run_id, + AgentEventType::Signal, + chrono::Utc::now().timestamp_millis(), + ) + .await + { + tracing::warn!(run_id, error = %error, "failed to supersede signals on cancel"); + } + } + Ok(cancelled) + } + + /// Persist one signal from a running Agent into its root session inbox. + /// The tool only exists for runs with a signal contract; this method is + /// the single writer that enforces capacity and generates the wake. + pub async fn emit_signal( + &self, + context: &AgentExecutionContext, + input: SignalInput, + ) -> Result { + let Some(run) = self.storage.get_agent_run(&context.run_id).await? else { + return Err(CoordinatorError::Rejected( + "signal rejected: run no longer exists".to_string(), + )); + }; + if run.execution_id != context.execution_id { + return Err(CoordinatorError::Rejected( + "signal rejected: stale execution".to_string(), + )); + } + if run.status.is_terminal() { + return Err(CoordinatorError::Rejected( + "signal rejected: run is no longer active".to_string(), + )); + } + let now = chrono::Utc::now().timestamp_millis(); + let event = crate::tools::emit_signal::build_signal_event( + context, + &input, + Uuid::new_v4().to_string(), + run.signal_delivery + .as_deref() + .map(crate::storage::agent_inbox::AgentEventDelivery::parse) + .transpose() + .map_err(CoordinatorError::from)? + .unwrap_or(crate::storage::agent_inbox::AgentEventDelivery::Queue), + ); + match self + .storage + .insert_agent_signal(&event, self.max_pending_inbox_events_per_session, now) + .await? + { + Some((record, deduplicated)) => { + if deduplicated { + return Ok(SignalAccepted { + signal_id: record.id, + status: SignalAcceptedStatus::Deduplicated, + delivery: event.delivery, + }); + } + self.projection.publish(AgentProjection { + session_id: run.root_session_id.clone(), + revision: record.revision, + run: None, + event: Some(crate::protocol::AgentEventView::from_record(&record)), + }); + self.notifier + .notify(&run.root_session_id, record.revision) + .await; + Ok(SignalAccepted { + signal_id: record.id, + status: SignalAcceptedStatus::Accepted, + delivery: event.delivery, + }) + } + None => Err(CoordinatorError::Rejected( + "inbox capacity exceeded; signal rejected".to_string(), + )), + } + } + + pub async fn get_run( + &self, + caller: &ToolExecutionContext, + run_id: &str, + ) -> Result, CoordinatorError> { + let Some(run) = self.storage.get_agent_run(run_id).await? else { + return Ok(None); + }; + self.authorize_access(caller, &run).await?; + Ok(Some(run)) + } + + pub async fn list_runs( + &self, + caller: &ToolExecutionContext, + cursor: Option<(i64, String)>, + limit: i64, + ) -> Result, CoordinatorError> { + let session_id = caller_session(caller)?; + self.storage + .list_agent_runs(&session_id, cursor, limit) + .await + .map_err(CoordinatorError::Storage) + } + + /// Session-scoped run projection for management/WebSocket clients. + pub async fn list_runs_for_session( + &self, + session_id: &str, + cursor: Option<(i64, String)>, + limit: i64, + ) -> Result<(i64, Vec, Option), CoordinatorError> { + let runs = self + .storage + .list_agent_runs(session_id, cursor, limit) + .await?; + let next_cursor = runs + .last() + .map(|run| format!("{}:{}", run.created_at, run.id)); + let views = runs + .iter() + .map(|run| crate::protocol::AgentRunView::from_record(run, 2_000)) + .collect(); + let revision = self.storage.get_session_agent_revision(session_id).await?; + Ok((revision, views, next_cursor)) + } + + /// Session-scoped single-run projection. + pub async fn get_run_for_session( + &self, + session_id: &str, + run_id: &str, + ) -> Result<(i64, Option), CoordinatorError> { + let revision = self.storage.get_session_agent_revision(session_id).await?; + let run = self.storage.get_agent_run(run_id).await?; + let run = run.filter(|run| run.root_session_id == session_id); + Ok(( + revision, + run.map(|run| crate::protocol::AgentRunView::from_record(&run, 2_000)), + )) + } + + /// Inbox events of one run (audit/projection). + pub async fn list_run_events( + &self, + run_id: &str, + limit: i64, + ) -> Result, CoordinatorError> { + self.storage + .list_agent_inbox_events_for_run(run_id, limit) + .await + .map_err(CoordinatorError::Storage) + } + + /// Management-API cancel: session-scoped, suppresses continuation so a + /// cancelled run never restarts a background Turn by itself. + pub async fn cancel_run_for_session( + &self, + session_id: &str, + run_id: &str, + reason: &str, + ) -> Result { + let Some(run) = self.storage.get_agent_run(run_id).await? else { + return Ok(false); + }; + if run.root_session_id != session_id { + return Err(CoordinatorError::Rejected( + "run does not belong to this session".to_string(), + )); + } + if run.status.is_terminal() { + return Ok(false); + } + let caller = ToolExecutionContext::for_session(session_id); + self.cancel_run_inner(&caller, run_id, reason, true).await + } + + /// Full durable result for a finished run; `None` while nonterminal. + pub async fn get_result( + &self, + caller: &ToolExecutionContext, + run_id: &str, + ) -> Result, CoordinatorError> { + let Some(run) = self.get_run(caller, run_id).await? else { + return Ok(None); + }; + if run.status.is_terminal() { + Ok(Some(run)) + } else { + Ok(None) + } + } + + /// Root may access every run of its session; a named Agent may only + /// access its own run and descendants. Run IDs are never credentials. + async fn authorize_access( + &self, + caller: &ToolExecutionContext, + run: &AgentRunRecord, + ) -> Result<(), CoordinatorError> { + let Some(agent) = caller.agent.as_ref() else { + let session_id = caller_session(caller)?; + if run.root_session_id != session_id { + return Err(CoordinatorError::Rejected( + "run belongs to a different session".to_string(), + )); + } + return Ok(()); + }; + if run.root_session_id != agent.root_session_id { + return Err(CoordinatorError::Rejected( + "run belongs to a different session".to_string(), + )); + } + let mut current = run.clone(); + loop { + if current.id == agent.run_id { + return Ok(()); + } + let Some(parent_id) = current.parent_run_id.clone() else { + return Err(CoordinatorError::Rejected( + "run is not part of the caller's delegation tree".to_string(), + )); + }; + let Some(parent) = self.storage.get_agent_run(&parent_id).await? else { + return Err(CoordinatorError::Rejected( + "run ancestry is corrupt".to_string(), + )); + }; + current = parent; + } + } +} + +fn caller_session(caller: &ToolExecutionContext) -> Result { + caller + .agent + .as_ref() + .map(|agent| agent.root_session_id.clone()) + .or_else(|| caller.session_id.clone()) + .ok_or_else(|| { + CoordinatorError::Rejected("caller context is not session-bound".to_string()) + }) +} + +fn truncate_summary(content: &str) -> String { + const MAX: usize = 500; + if content.len() <= MAX { + content.to_string() + } else { + let cut = content.floor_char_boundary(MAX); + format!("{}...", &content[..cut]) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::agent::AgentCatalog; + use crate::config::LLMProviderConfig; + use crate::tools::ToolRegistry; + use std::collections::HashMap; + + fn provider_config() -> LLMProviderConfig { + LLMProviderConfig { + provider_type: "openai".into(), + name: "test".into(), + base_url: "http://localhost".into(), + api_key: "test".into(), + extra_headers: HashMap::new(), + model_id: "test".into(), + temperature: None, + max_tokens: None, + model_extra: HashMap::new(), + max_tool_iterations: 1, + token_limit: 4096, + workspace_dir: std::env::temp_dir(), + input_types: vec!["text".into()], + price_input_per_million: None, + price_output_per_million: None, + } + } + + fn write_catalog(root: &std::path::Path) -> AgentCatalog { + std::fs::create_dir_all(root.join("agents")).unwrap(); + std::fs::write( + root.join("agents/researcher.md"), + "---\nid: researcher\ndescription: research role\nllm_profile: research\n---\n# Role\n\nDo the assigned work.\n", + ) + .unwrap(); + let tools = ToolRegistry::new(); + let loader = crate::skills::SkillsLoader::new_for_testing( + root.join("skills"), + root.join("external-skills"), + ); + let profiles = HashMap::from([("research".to_string(), provider_config())]); + let config = crate::config::AgentOrchestrationConfig { + enabled: true, + definitions_dir: "agents".to_string(), + root_delegates: vec!["researcher".to_string()], + ..Default::default() + }; + AgentCatalog::load(&config, root, &profiles, &tools, &loader, 1).unwrap() + } + + async fn coordinator() -> (Arc, tempfile::TempDir) { + let dir = tempfile::tempdir().unwrap(); + let storage = Arc::new(Storage::new(&dir.path().join("coord.db")).await.unwrap()); + let (notify_tx, _notify_rx) = tokio::sync::mpsc::unbounded_channel(); + let catalog = write_catalog(dir.path()); + let manager = Arc::new( + SubAgentManager::new( + provider_config(), + Arc::new(ToolRegistry::new()), + Some(storage.clone()), + notify_tx, + 1, + None, + crate::task_supervisor::TaskSupervisor::new(), + ) + .with_catalog(Arc::new(catalog)), + ); + let work_manager = Arc::new(crate::work::WorkManager::new(storage.clone())); + let notifier = crate::agent::AgentInboxNotifier::new(); + let supervisor = crate::task_supervisor::TaskSupervisor::new(); + let orchestration = crate::config::AgentOrchestrationConfig { + enabled: true, + max_pending_inbox_events_per_session: 1, + ..Default::default() + }; + ( + AgentCoordinator::new( + storage, + manager, + work_manager, + notifier, + Arc::new(crate::agent::AgentProjectionHub::new()), + supervisor, + 1, + &orchestration, + ), + dir, + ) + } + + fn foreground_config(target: &str) -> SubAgentConfig { + SubAgentConfig { + target: Some(target.to_string()), + prompt: "work".to_string(), + context: None, + mode: ExecutionMode::Foreground, + allowed_tools: None, + max_iterations: Some(1), + timeout_secs: Some(5), + plan_item_id: None, + session_id: Some("cli:test:dialog".to_string()), + } + } + + #[tokio::test] + async fn foreground_run_is_persisted_with_terminal_state() { + let (coordinator, _dir) = coordinator().await; + let caller = ToolExecutionContext::for_session("cli:test:dialog"); + + let results = coordinator + .delegate_foreground(&caller, vec![foreground_config("researcher")]) + .await + .unwrap(); + assert_eq!(results.len(), 1); + + let run_id = &results[0].task_id; + let run = coordinator.get_run(&caller, run_id).await.unwrap().unwrap(); + assert_eq!(run.agent_id, "researcher"); + assert_eq!(run.mode, AgentRunMode::Foreground); + // Provider is unreachable in this test, so the run must fail — but it + // must still be persisted with a terminal status and full audit row. + assert!(run.status.is_terminal()); + assert!(matches!(run.status, AgentRunStatus::Failed)); + assert!(run.finished_at.is_some()); + } + + #[tokio::test] + async fn batch_creates_group_and_keeps_request_order() { + let (coordinator, _dir) = coordinator().await; + let caller = ToolExecutionContext::for_session("cli:test:dialog"); + + let results = coordinator + .delegate_foreground( + &caller, + vec![ + foreground_config("researcher"), + foreground_config("researcher"), + ], + ) + .await + .unwrap(); + assert_eq!(results.len(), 2); + + let first = coordinator + .get_run(&caller, &results[0].task_id) + .await + .unwrap() + .unwrap(); + let second = coordinator + .get_run(&caller, &results[1].task_id) + .await + .unwrap() + .unwrap(); + assert!(first.group_id.is_some()); + assert_eq!(first.group_id, second.group_id); + + let group = coordinator + .storage + .get_agent_run_group(first.group_id.as_ref().unwrap()) + .await + .unwrap() + .unwrap(); + assert_eq!(group.expected_runs, 2); + assert_eq!(group.terminal_runs, 2); + assert!(group.status.is_terminal()); + } + + #[tokio::test] + async fn other_session_cannot_read_or_cancel_runs() { + let (coordinator, _dir) = coordinator().await; + let caller = ToolExecutionContext::for_session("cli:test:dialog"); + let results = coordinator + .delegate_foreground(&caller, vec![foreground_config("researcher")]) + .await + .unwrap(); + let run_id = results[0].task_id.clone(); + + let stranger = ToolExecutionContext::for_session("cli:other:dialog"); + let error = coordinator.get_run(&stranger, &run_id).await.unwrap_err(); + assert!(matches!(error, CoordinatorError::Rejected(_))); + let error = coordinator + .cancel_run(&stranger, &run_id, "x") + .await + .unwrap_err(); + assert!(matches!(error, CoordinatorError::Rejected(_))); + } + + #[tokio::test] + async fn unknown_target_is_rejected_before_persistence() { + let (coordinator, _dir) = coordinator().await; + let caller = ToolExecutionContext::for_session("cli:test:dialog"); + let error = coordinator + .delegate_foreground(&caller, vec![foreground_config("missing")]) + .await + .unwrap_err(); + assert!(matches!(error, CoordinatorError::SubAgent(_))); + assert!( + coordinator + .list_runs(&caller, None, 10) + .await + .unwrap() + .is_empty() + ); + } + + #[tokio::test] + async fn background_admission_persists_and_reserves_completion() { + let (coordinator, _dir) = coordinator().await; + let caller = ToolExecutionContext::for_session("cli:test:dialog"); + + let run_id = coordinator + .delegate_background(&caller, foreground_config("researcher")) + .await + .unwrap(); + + let run = coordinator + .get_run(&caller, &run_id) + .await + .unwrap() + .unwrap(); + assert_eq!(run.mode, AgentRunMode::Background); + assert!(run.completion_slot_reserved); + + // A second background run while the inbox is at capacity must be + // rejected, not admitted silently. + let error = coordinator + .delegate_background(&caller, foreground_config("researcher")) + .await + .unwrap_err(); + assert!(matches!(error, CoordinatorError::Rejected(_))); + + // Wait for the spawned runner to finish (provider is unreachable, so + // it fails fast) and verify the terminal commit converted the + // reservation into a durable completion event. + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10); + loop { + let run = coordinator + .get_run(&caller, &run_id) + .await + .unwrap() + .unwrap(); + if run.status.is_terminal() { + break; + } + assert!( + std::time::Instant::now() < deadline, + "background runner did not finish in time" + ); + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + } + let events = coordinator + .storage + .list_agent_inbox_events("cli:test:dialog", 10) + .await + .unwrap(); + assert_eq!(events.len(), 1); + assert_eq!( + events[0].status, + crate::storage::agent_inbox::AgentEventStatus::Pending + ); + assert!(events[0].requires_continuation); + let state: (i64, i64) = sqlx::query_as( + "SELECT pending_event_count, reserved_completion_slots FROM agent_session_state \ + WHERE root_session_id = 'cli:test:dialog'", + ) + .fetch_one(coordinator.storage.pool()) + .await + .unwrap(); + // Reservation converted: nothing reserved, one event pending. + assert_eq!(state, (1, 0)); + } + + async fn storage_accept_run( + storage: &Arc, + run_id: &str, + session: &str, + now: i64, + slot_reserved: bool, + ) { + let _ = storage.ensure_agent_session_state(session, now).await; + if slot_reserved { + let _ = storage.reserve_completion_slots(session, 1, 1, now).await; + } + let run = NewAgentRun { + id: run_id.to_string(), + root_session_id: session.to_string(), + root_turn_id: None, + parent_run_id: None, + caller_agent_id: "ROOT".to_string(), + caller_scope_id: "turn-1".to_string(), + idempotency_key: None, + agent_id: "researcher".to_string(), + definition_hash: "hash".to_string(), + provider_profile: "research".to_string(), + provider_name: "test".to_string(), + model_id: "test-model".to_string(), + mode: AgentRunMode::Background, + depth: 1, + plan_item_id: None, + execution_id: run_id.to_string(), + task: "work".to_string(), + context_json: None, + budget_json: "{}".to_string(), + deadline_at: now + 100_000, + runtime_generation: 1, + completion_slot_reserved: slot_reserved, + }; + let _ = storage + .accept_agent_runs(AcceptAgentRequest { + group: None, + runs: vec![run], + now, + }) + .await + .unwrap(); + } + + #[tokio::test] + async fn emit_signal_persists_wakes_and_respects_capacity_and_dedupe() { + let (coordinator, _dir) = coordinator().await; + let run_id = "run-sig-1"; + let session = "cli:test:dialog"; + storage_accept_run( + &coordinator.storage, + run_id, + session, + chrono::Utc::now().timestamp_millis(), + false, + ) + .await; + let context = crate::agent::AgentExecutionContext { + root_session_id: session.to_string(), + root_turn_id: None, + run_id: run_id.to_string(), + execution_id: run_id.to_string(), + group_id: None, + parent_run_id: None, + caller_agent_id: "ROOT".to_string(), + current_agent_id: "researcher".to_string(), + ancestry: vec!["researcher".to_string()], + depth: 1, + plan_item_id: None, + cancellation: tokio_util::sync::CancellationToken::new(), + budget: crate::agent::AgentBudget { + remaining_runs: 15, + remaining_depth: 3, + }, + tree_runs: Arc::new(std::sync::atomic::AtomicUsize::new(1)), + signal_contract: None, + emitted_signals: Arc::new(std::sync::Mutex::new(Vec::new())), + }; + + let signal = SignalInput { + key: "err-rate".to_string(), + severity: "warning".to_string(), + summary: "error rate above threshold".to_string(), + details: Some(serde_json::json!({ "current": 0.071 })), + dedupe_key: Some("svc-a:err".to_string()), + event_key: format!("signal:svc-a:err:{}", 0), + }; + let accepted = coordinator + .emit_signal(&context, signal.clone()) + .await + .unwrap(); + assert_eq!(accepted.status, SignalAcceptedStatus::Accepted); + assert!(matches!( + accepted.delivery, + crate::storage::agent_inbox::AgentEventDelivery::Queue + )); + + // Same dedupe key in the same window collapses to the same event. + let mut duplicate = signal.clone(); + duplicate.event_key = format!("signal:svc-a:err:{}", 0); + let deduplicated = coordinator.emit_signal(&context, duplicate).await.unwrap(); + assert_eq!(deduplicated.status, SignalAcceptedStatus::Deduplicated); + assert_eq!(deduplicated.signal_id, accepted.signal_id); + + // A stale execution id is rejected. + let mut stale = context.clone(); + stale.execution_id = "other-exec".to_string(); + assert!(matches!( + coordinator.emit_signal(&stale, signal.clone()).await, + Err(CoordinatorError::Rejected(_)) + )); + } + + #[tokio::test] + async fn emit_signal_rejects_when_run_is_terminal_or_inbox_is_full() { + let (coordinator, _dir) = coordinator().await; + let run_id = "run-sig-2"; + let session = "cli:test:dialog"; + let now = chrono::Utc::now().timestamp_millis(); + storage_accept_run(&coordinator.storage, run_id, session, now, false).await; + let context = crate::agent::AgentExecutionContext { + root_session_id: session.to_string(), + root_turn_id: None, + run_id: run_id.to_string(), + execution_id: run_id.to_string(), + group_id: None, + parent_run_id: None, + caller_agent_id: "ROOT".to_string(), + current_agent_id: "researcher".to_string(), + ancestry: vec!["researcher".to_string()], + depth: 1, + plan_item_id: None, + cancellation: tokio_util::sync::CancellationToken::new(), + budget: crate::agent::AgentBudget { + remaining_runs: 15, + remaining_depth: 3, + }, + tree_runs: Arc::new(std::sync::atomic::AtomicUsize::new(1)), + signal_contract: None, + emitted_signals: Arc::new(std::sync::Mutex::new(Vec::new())), + }; + let signal = SignalInput { + key: "k".to_string(), + severity: "info".to_string(), + summary: "s".to_string(), + details: None, + dedupe_key: None, + event_key: format!("signal:{}", uuid::Uuid::new_v4()), + }; + + // Terminal run: rejected. + coordinator + .storage + .commit_agent_terminal( + run_id, + run_id, + 1, + &AgentTerminalOutcome::Cancelled { + reason: "test".to_string(), + }, + None, + now + 1, + ) + .await + .unwrap(); + assert!(matches!( + coordinator.emit_signal(&context, signal.clone()).await, + Err(CoordinatorError::Rejected(_)) + )); + + // Capacity: a reserved completion slot exhausts the session limit. + let coordinator2 = { + let dir = tempfile::tempdir().unwrap(); + let storage2 = Arc::new(Storage::new(&dir.path().join("c2.db")).await.unwrap()); + let (notify_tx, _) = tokio::sync::mpsc::unbounded_channel(); + let catalog = write_catalog(dir.path()); + let manager = Arc::new( + SubAgentManager::new( + provider_config(), + Arc::new(ToolRegistry::new()), + Some(storage2.clone()), + notify_tx, + 1, + None, + crate::task_supervisor::TaskSupervisor::new(), + ) + .with_catalog(Arc::new(catalog)), + ); + let work_manager = Arc::new(crate::work::WorkManager::new(storage2.clone())); + let notifier = crate::agent::AgentInboxNotifier::new(); + let supervisor = crate::task_supervisor::TaskSupervisor::new(); + let orchestration = crate::config::AgentOrchestrationConfig { + enabled: true, + max_pending_inbox_events_per_session: 1, + ..Default::default() + }; + AgentCoordinator::new( + storage2, + manager, + work_manager, + notifier, + Arc::new(crate::agent::AgentProjectionHub::new()), + supervisor, + 1, + &orchestration, + ) + }; + storage_accept_run(&coordinator2.storage, "run-sig-3", session, now, true).await; + let context2 = crate::agent::AgentExecutionContext { + root_session_id: session.to_string(), + root_turn_id: None, + run_id: "run-sig-3".to_string(), + execution_id: "run-sig-3".to_string(), + group_id: None, + parent_run_id: None, + caller_agent_id: "ROOT".to_string(), + current_agent_id: "researcher".to_string(), + ancestry: vec!["researcher".to_string()], + depth: 1, + plan_item_id: None, + cancellation: tokio_util::sync::CancellationToken::new(), + budget: crate::agent::AgentBudget { + remaining_runs: 15, + remaining_depth: 3, + }, + tree_runs: Arc::new(std::sync::atomic::AtomicUsize::new(1)), + signal_contract: None, + emitted_signals: Arc::new(std::sync::Mutex::new(Vec::new())), + }; + assert!(matches!( + coordinator2.emit_signal(&context2, signal).await, + Err(CoordinatorError::Rejected(_)) + )); + } +} diff --git a/src/agent/definition.rs b/src/agent/definition.rs new file mode 100644 index 0000000..5652a9d --- /dev/null +++ b/src/agent/definition.rs @@ -0,0 +1,485 @@ +use std::path::{Path, PathBuf}; +use std::sync::Arc; + +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; + +use crate::config::LLMProviderConfig; + +pub const MAX_DEFINITION_FILE_BYTES: u64 = 256 * 1024; +const MAX_DESCRIPTION_CHARS: usize = 4_096; +const MAX_ROLE_CHARS: usize = 65_536; + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(default, deny_unknown_fields)] +pub struct AgentLimits { + pub timeout_secs: u64, + pub max_iterations: usize, + pub max_children: usize, + pub max_depth: u16, + pub max_concurrent_runs: usize, + pub max_concurrent_provider_steps: usize, + pub max_concurrent_tool_steps: usize, + pub max_result_chars: usize, +} + +impl Default for AgentLimits { + fn default() -> Self { + Self { + timeout_secs: 900, + max_iterations: 24, + max_children: 4, + max_depth: 3, + max_concurrent_runs: 2, + max_concurrent_provider_steps: 1, + max_concurrent_tool_steps: 4, + max_result_chars: 16_000, + } + } +} + +/// Signal delivery lane for background Agent events. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize, Serialize, Default)] +#[serde(rename_all = "snake_case")] +pub enum SignalDelivery { + #[default] + Queue, + Steer, +} + +impl SignalDelivery { + pub fn as_str(&self) -> &'static str { + match self { + Self::Queue => "queue", + Self::Steer => "steer", + } + } +} + +/// Durable emit_signal contract. An Agent only sees the `emit_signal` tool +/// when this block is present; every limit below is enforced by the tool and +/// the Coordinator, not by the model. +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(default, deny_unknown_fields)] +pub struct SignalContract { + pub delivery: SignalDelivery, + /// Total number of signals one run may emit. + pub max_total: u32, + /// Upper bound for the serialized `details` JSON payload of one signal. + pub max_details_bytes: usize, + /// Minimum wall time between two signals of the same run. + pub min_interval_ms: u64, + /// Maximum signals emitted within `burst_window_ms`. + pub max_burst: u32, + pub burst_window_ms: u64, + /// Allowlisted severities; anything else is rejected. + pub severity_allowlist: Vec, + /// Dedupe key cooldown window; repeated keys inside the window collapse + /// to the same event, keys outside it emit again. + pub dedupe_cooldown_ms: u64, + /// Maximum JSON nesting depth of `details`. + pub max_payload_depth: usize, +} + +impl Default for SignalContract { + fn default() -> Self { + Self { + delivery: SignalDelivery::Queue, + max_total: 64, + max_details_bytes: 8 * 1024, + min_interval_ms: 500, + max_burst: 5, + burst_window_ms: 10_000, + severity_allowlist: vec![ + "info".to_string(), + "warning".to_string(), + "critical".to_string(), + ], + dedupe_cooldown_ms: 60_000, + max_payload_depth: 16, + } + } +} + +impl SignalContract { + fn validate(&self) -> Result<(), AgentDefinitionError> { + if self.max_total == 0 || self.max_total > 1024 { + return Err(AgentDefinitionError::Invalid( + "signal.max_total must be between 1 and 1024".to_string(), + )); + } + if self.max_details_bytes == 0 || self.max_details_bytes > 64 * 1024 { + return Err(AgentDefinitionError::Invalid( + "signal.max_details_bytes must be between 1 and 65536".to_string(), + )); + } + if self.max_burst == 0 || self.max_burst > self.max_total { + return Err(AgentDefinitionError::Invalid( + "signal.max_burst must be between 1 and max_total".to_string(), + )); + } + if self.max_payload_depth == 0 || self.max_payload_depth > 64 { + return Err(AgentDefinitionError::Invalid( + "signal.max_payload_depth must be between 1 and 64".to_string(), + )); + } + if self.severity_allowlist.is_empty() || self.severity_allowlist.len() > 16 { + return Err(AgentDefinitionError::Invalid( + "signal.severity_allowlist must contain 1..=16 severities".to_string(), + )); + } + let mut seen = std::collections::HashSet::new(); + if let Some(duplicate) = self + .severity_allowlist + .iter() + .find(|severity| !seen.insert(severity.as_str())) + { + return Err(AgentDefinitionError::Invalid(format!( + "duplicate signal severity '{duplicate}'" + ))); + } + if self + .severity_allowlist + .iter() + .any(|severity| severity.trim().is_empty() || severity.len() > 64) + { + return Err(AgentDefinitionError::Invalid( + "signal severities must be non-empty and at most 64 characters".to_string(), + )); + } + Ok(()) + } +} + +impl AgentLimits { + fn validate(&self) -> Result<(), AgentDefinitionError> { + if self.timeout_secs == 0 || self.timeout_secs > 86_400 { + return Err(AgentDefinitionError::Invalid( + "limits.timeout_secs must be between 1 and 86400".to_string(), + )); + } + if self.max_iterations == 0 || self.max_iterations > 256 { + return Err(AgentDefinitionError::Invalid( + "limits.max_iterations must be between 1 and 256".to_string(), + )); + } + if self.max_children == 0 || self.max_children > 128 { + return Err(AgentDefinitionError::Invalid( + "limits.max_children must be between 1 and 128".to_string(), + )); + } + if self.max_depth == 0 || self.max_depth > 32 { + return Err(AgentDefinitionError::Invalid( + "limits.max_depth must be between 1 and 32".to_string(), + )); + } + for (name, value, hard_max) in [ + ("max_concurrent_runs", self.max_concurrent_runs, 128), + ( + "max_concurrent_provider_steps", + self.max_concurrent_provider_steps, + 128, + ), + ( + "max_concurrent_tool_steps", + self.max_concurrent_tool_steps, + 512, + ), + ] { + if value == 0 || value > hard_max { + return Err(AgentDefinitionError::Invalid(format!( + "limits.{name} must be between 1 and {hard_max}" + ))); + } + } + if self.max_result_chars == 0 || self.max_result_chars > 1_000_000 { + return Err(AgentDefinitionError::Invalid( + "limits.max_result_chars must be between 1 and 1000000".to_string(), + )); + } + Ok(()) + } +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct AgentFrontmatter { + id: String, + description: String, + llm_profile: String, + #[serde(default)] + tools: Vec, + #[serde(default)] + delegates: Vec, + #[serde(default)] + skills: Vec, + #[serde(default)] + limits: AgentLimits, + #[serde(default)] + signal: Option, +} + +#[derive(Debug, Clone)] +pub struct AgentDefinition { + pub id: String, + pub description: String, + pub llm_profile: String, + pub provider_config: Arc, + pub tools: Vec, + pub delegates: Vec, + pub skills: Vec, + pub limits: AgentLimits, + pub signal_contract: Option, + pub role_prompt: String, + pub definition_hash: String, + pub source_path: PathBuf, +} + +#[derive(Debug, thiserror::Error)] +pub enum AgentDefinitionError { + #[error("failed to read Agent definition {path}: {source}")] + Io { + path: String, + #[source] + source: std::io::Error, + }, + #[error("invalid Agent definition: {0}")] + Invalid(String), + #[error("invalid YAML frontmatter: {0}")] + Yaml(#[from] serde_yaml::Error), +} + +pub(crate) fn parse_definition( + path: &Path, + provider_config: Arc, +) -> Result { + let metadata = std::fs::symlink_metadata(path).map_err(|source| AgentDefinitionError::Io { + path: path.display().to_string(), + source, + })?; + if !metadata.file_type().is_file() || metadata.file_type().is_symlink() { + return Err(AgentDefinitionError::Invalid(format!( + "{} must be a regular non-symlink file", + path.display() + ))); + } + if metadata.len() > MAX_DEFINITION_FILE_BYTES { + return Err(AgentDefinitionError::Invalid(format!( + "{} exceeds the {} byte limit", + path.display(), + MAX_DEFINITION_FILE_BYTES + ))); + } + let content = std::fs::read_to_string(path).map_err(|source| AgentDefinitionError::Io { + path: path.display().to_string(), + source, + })?; + let normalized = content.replace("\r\n", "\n"); + let mut lines = normalized.lines(); + if lines.next() != Some("---") { + return Err(AgentDefinitionError::Invalid(format!( + "{} must start with a standalone --- line", + path.display() + ))); + } + let mut yaml_lines = Vec::new(); + let mut found_end = false; + for line in &mut lines { + if line == "---" { + found_end = true; + break; + } + yaml_lines.push(line); + } + if !found_end { + return Err(AgentDefinitionError::Invalid(format!( + "{} has no closing frontmatter delimiter", + path.display() + ))); + } + let role_prompt = lines.collect::>().join("\n").trim().to_string(); + if role_prompt.is_empty() { + return Err(AgentDefinitionError::Invalid(format!( + "{} has an empty role body", + path.display() + ))); + } + if role_prompt.chars().count() > MAX_ROLE_CHARS { + return Err(AgentDefinitionError::Invalid(format!( + "{} role body exceeds {MAX_ROLE_CHARS} characters", + path.display() + ))); + } + let frontmatter: AgentFrontmatter = serde_yaml::from_str(&yaml_lines.join("\n"))?; + validate_agent_id(&frontmatter.id)?; + if frontmatter.description.trim().is_empty() + || frontmatter.description.chars().count() > MAX_DESCRIPTION_CHARS + { + return Err(AgentDefinitionError::Invalid(format!( + "Agent '{}' description must contain 1..={MAX_DESCRIPTION_CHARS} characters", + frontmatter.id + ))); + } + if frontmatter.llm_profile.trim().is_empty() { + return Err(AgentDefinitionError::Invalid(format!( + "Agent '{}' has an empty llm_profile", + frontmatter.id + ))); + } + frontmatter.limits.validate()?; + if let Some(signal) = frontmatter.signal.as_ref() { + signal.validate()?; + } + reject_duplicates("tools", &frontmatter.tools)?; + reject_duplicates("delegates", &frontmatter.delegates)?; + reject_duplicates("skills", &frontmatter.skills)?; + + let file_stem = path.file_stem().and_then(|value| value.to_str()); + if file_stem != Some(frontmatter.id.as_str()) { + return Err(AgentDefinitionError::Invalid(format!( + "Agent id '{}' must match file name {}", + frontmatter.id, + path.display() + ))); + } + + let canonical_frontmatter = serde_json::to_vec(&frontmatter) + .map_err(|error| AgentDefinitionError::Invalid(error.to_string()))?; + let mut hasher = Sha256::new(); + hasher.update(canonical_frontmatter); + hasher.update(b"\n---\n"); + hasher.update(role_prompt.as_bytes()); + let definition_hash = hasher + .finalize() + .iter() + .map(|byte| format!("{byte:02x}")) + .collect(); + + Ok(AgentDefinition { + id: frontmatter.id, + description: frontmatter.description.trim().to_string(), + llm_profile: frontmatter.llm_profile, + provider_config, + tools: frontmatter.tools, + delegates: frontmatter.delegates, + skills: frontmatter.skills, + limits: frontmatter.limits, + signal_contract: frontmatter.signal, + role_prompt, + definition_hash, + source_path: path.to_path_buf(), + }) +} + +pub fn validate_agent_id(id: &str) -> Result<(), AgentDefinitionError> { + let valid = !id.is_empty() + && id.len() <= 64 + && id + .bytes() + .next() + .is_some_and(|value| value.is_ascii_lowercase()) + && id.bytes().all(|value| { + value.is_ascii_lowercase() || value.is_ascii_digit() || value == b'_' || value == b'-' + }); + if !valid || matches!(id, "root" | "main" | "default" | "general") { + return Err(AgentDefinitionError::Invalid(format!( + "invalid or reserved Agent id '{id}'" + ))); + } + Ok(()) +} + +fn reject_duplicates(field: &str, values: &[String]) -> Result<(), AgentDefinitionError> { + let mut seen = std::collections::HashSet::new(); + if let Some(value) = values.iter().find(|value| !seen.insert(value.as_str())) { + return Err(AgentDefinitionError::Invalid(format!( + "duplicate {field} entry '{value}'" + ))); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::collections::HashMap; + + fn provider() -> Arc { + Arc::new(LLMProviderConfig { + provider_type: "openai".to_string(), + name: "test".to_string(), + base_url: "https://example.invalid/v1".to_string(), + api_key: "test".to_string(), + extra_headers: HashMap::new(), + model_id: "test-model".to_string(), + temperature: None, + max_tokens: None, + model_extra: HashMap::new(), + max_tool_iterations: 99, + token_limit: 4096, + workspace_dir: std::env::temp_dir(), + input_types: vec!["text".to_string()], + price_input_per_million: None, + price_output_per_million: None, + }) + } + + #[test] + fn strict_definition_parses_and_hashes_stably() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("researcher.md"); + std::fs::write( + &path, + concat!( + "---\n", + "id: researcher\n", + "description: Research primary sources\n", + "llm_profile: research\n", + "tools:\n", + " - calculator\n", + "limits:\n", + " max_iterations: 12\n", + "---\n", + "# Role\n\n", + "Return evidence and uncertainty.\n" + ), + ) + .unwrap(); + + let first = parse_definition(&path, provider()).unwrap(); + let second = parse_definition(&path, provider()).unwrap(); + + assert_eq!(first.id, "researcher"); + assert_eq!(first.limits.max_iterations, 12); + assert_eq!(first.definition_hash, second.definition_hash); + assert_eq!(first.definition_hash.len(), 64); + } + + #[test] + fn unknown_frontmatter_key_is_rejected() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("researcher.md"); + std::fs::write( + &path, + "---\nid: researcher\ndescription: test\nllm_profile: research\nunsafe_tools: true\n---\n# Role\n", + ) + .unwrap(); + + assert!(parse_definition(&path, provider()).is_err()); + } + + #[test] + fn reserved_and_mismatched_ids_are_rejected() { + assert!(validate_agent_id("root").is_err()); + assert!(validate_agent_id("Uppercase").is_err()); + + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("researcher.md"); + std::fs::write( + &path, + "---\nid: reviewer\ndescription: test\nllm_profile: research\n---\n# Role\n", + ) + .unwrap(); + assert!(parse_definition(&path, provider()).is_err()); + } +} diff --git a/src/agent/gate.rs b/src/agent/gate.rs new file mode 100644 index 0000000..b5b2d03 --- /dev/null +++ b/src/agent/gate.rs @@ -0,0 +1,339 @@ +use std::sync::{Arc, Weak}; + +use dashmap::DashMap; +use tokio::sync::{OwnedSemaphorePermit, Semaphore}; +use tokio_util::sync::CancellationToken; + +/// Step permits are held while a provider request or tool invocation is in +/// flight and released as soon as the step finishes. Session permits are +/// released before global permits (reverse acquisition order). The fields +/// are never read; ownership alone keeps the quotas reserved (RAII). +#[derive(Debug)] +pub struct StepPermit { + #[allow(dead_code)] + session: Option, + #[allow(dead_code)] + global: OwnedSemaphorePermit, +} + +/// Run quota permits cover a whole Agent Run from admission until its +/// terminal commit. Foreground delegation does not take run permits; the +/// quota gates background admission so a waiting parent can never deadlock +/// nested foreground children. +#[derive(Debug)] +pub struct RunPermit { + #[allow(dead_code)] + session: Option, + #[allow(dead_code)] + global: OwnedSemaphorePermit, +} + +#[derive(Debug, thiserror::Error)] +pub enum GateError { + #[error("execution gate wait cancelled")] + Cancelled, +} + +#[derive(Default)] +struct KeyedSemaphores { + permits: usize, + map: DashMap>, +} + +impl KeyedSemaphores { + fn new(permits: usize) -> Self { + Self { + permits, + map: DashMap::new(), + } + } + + fn semaphore(self: &Arc, key: &str) -> Arc { + if let Some(existing) = self.map.get(key) + && let Some(semaphore) = existing.upgrade() + { + return semaphore; + } + let semaphore = Arc::new(Semaphore::new(self.permits)); + self.map.insert(key.to_string(), Arc::downgrade(&semaphore)); + // Drop registry entries whose owners are gone so long-lived gateways + // do not accumulate one semaphore per historical session. + self.map.retain(|_, weak| weak.strong_count() > 0); + semaphore + } +} + +/// Concurrency gates shared by one runtime generation. Run quota and +/// provider/tool step permits have independent lifecycles; acquisition order +/// is always global -> session and release order is reversed. +pub struct ExecutionGate { + run_global: Arc, + run_session: Arc, + provider_global: Arc, + provider_session: Arc, + tool_global: Arc, + tool_session: Arc, +} + +impl std::fmt::Debug for ExecutionGate { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("ExecutionGate") + .field("run_available", &self.run_global.available_permits()) + .field( + "provider_available", + &self.provider_global.available_permits(), + ) + .field("tool_available", &self.tool_global.available_permits()) + .finish() + } +} + +impl ExecutionGate { + pub fn new(config: &crate::config::AgentOrchestrationConfig) -> Arc { + Arc::new(Self { + run_global: Arc::new(Semaphore::new(config.max_concurrent_runs)), + run_session: Arc::new(KeyedSemaphores::new(config.max_concurrent_runs_per_session)), + provider_global: Arc::new(Semaphore::new(config.max_concurrent_provider_steps)), + provider_session: Arc::new(KeyedSemaphores::new( + config.max_concurrent_provider_steps_per_session, + )), + tool_global: Arc::new(Semaphore::new(config.max_concurrent_tool_steps)), + tool_session: Arc::new(KeyedSemaphores::new( + config.max_concurrent_tool_steps_per_session, + )), + }) + } + + /// Unlimited gate used when orchestration is disabled; root Turns still + /// route through it so the code path stays uniform. + pub fn unbounded() -> Arc { + Arc::new(Self { + run_global: Arc::new(Semaphore::new(Semaphore::MAX_PERMITS)), + run_session: Arc::new(KeyedSemaphores::new(Semaphore::MAX_PERMITS)), + provider_global: Arc::new(Semaphore::new(Semaphore::MAX_PERMITS)), + provider_session: Arc::new(KeyedSemaphores::new(Semaphore::MAX_PERMITS)), + tool_global: Arc::new(Semaphore::new(Semaphore::MAX_PERMITS)), + tool_session: Arc::new(KeyedSemaphores::new(Semaphore::MAX_PERMITS)), + }) + } + + pub async fn acquire_run( + self: &Arc, + session_id: &str, + cancellation: &CancellationToken, + ) -> Result { + let global = acquire_owned(self.run_global.clone(), cancellation).await?; + let session = acquire_keyed(&self.run_session, session_id, cancellation).await?; + Ok(RunPermit { + session: Some(session), + global, + }) + } + + pub async fn acquire_provider( + self: &Arc, + session_id: Option<&str>, + cancellation: &CancellationToken, + ) -> Result { + let global = acquire_owned(self.provider_global.clone(), cancellation).await?; + let session = match session_id { + Some(session_id) => { + Some(acquire_keyed(&self.provider_session, session_id, cancellation).await?) + } + None => None, + }; + Ok(StepPermit { session, global }) + } + + pub async fn acquire_tool( + self: &Arc, + session_id: Option<&str>, + cancellation: &CancellationToken, + ) -> Result { + let global = acquire_owned(self.tool_global.clone(), cancellation).await?; + let session = match session_id { + Some(session_id) => { + Some(acquire_keyed(&self.tool_session, session_id, cancellation).await?) + } + None => None, + }; + Ok(StepPermit { session, global }) + } + + pub fn available_provider_permits(&self) -> usize { + self.provider_global.available_permits() + } + + pub fn available_tool_permits(&self) -> usize { + self.tool_global.available_permits() + } +} + +async fn acquire_owned( + semaphore: Arc, + cancellation: &CancellationToken, +) -> Result { + tokio::select! { + biased; + _ = cancellation.cancelled() => Err(GateError::Cancelled), + result = semaphore.acquire_owned() => { + result.map_err(|_| GateError::Cancelled) + } + } +} + +async fn acquire_keyed( + keyed: &Arc, + key: &str, + cancellation: &CancellationToken, +) -> Result { + acquire_owned(keyed.semaphore(key), cancellation).await +} + +/// Test helper: wait until a predicate holds or fail after the timeout. +#[cfg(test)] +async fn eventually bool>(predicate: F) { + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5); + while std::time::Instant::now() < deadline { + if predicate() { + return; + } + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + panic!("condition did not hold within timeout"); +} + +#[cfg(test)] +mod tests { + use super::*; + use std::time::Duration; + + use crate::config::AgentOrchestrationConfig; + + /// Global caps stay larger than session caps so a session-level waiter + /// can never exhaust the global permits while blocked (acquisition order + /// is global -> session by design). + fn gate(provider_session: usize, tool_session: usize) -> Arc { + let config = AgentOrchestrationConfig { + enabled: true, + max_concurrent_runs: 8, + max_concurrent_runs_per_session: 1, + max_concurrent_provider_steps: 8, + max_concurrent_provider_steps_per_session: provider_session, + max_concurrent_tool_steps: 8, + max_concurrent_tool_steps_per_session: tool_session, + ..Default::default() + }; + ExecutionGate::new(&config) + } + + #[tokio::test] + async fn provider_permits_are_released_after_drop() { + let gate = gate(1, 1); + let token = CancellationToken::new(); + let permit = gate.acquire_provider(None, &token).await.unwrap(); + assert_eq!(gate.available_provider_permits(), 7); + drop(permit); + assert_eq!(gate.available_provider_permits(), 8); + } + + #[tokio::test] + async fn session_quota_blocks_second_session_step_until_release() { + let gate = gate(1, 8); + let token = CancellationToken::new(); + let first = gate + .acquire_provider(Some("cli:a:d1"), &token) + .await + .unwrap(); + // Global capacity remains, but the same session is capped at one. + let blocked = tokio::spawn({ + let gate = gate.clone(); + let token = token.clone(); + async move { gate.acquire_provider(Some("cli:a:d1"), &token).await } + }); + tokio::time::sleep(Duration::from_millis(50)).await; + assert!(!blocked.is_finished()); + // A different session is unaffected. + let other = gate + .acquire_provider(Some("cli:b:d2"), &token) + .await + .unwrap(); + drop(other); + drop(first); + let second = blocked.await.unwrap().unwrap(); + drop(second); + assert_eq!(gate.available_provider_permits(), 8); + } + + #[tokio::test] + async fn cancellation_aborts_permit_wait() { + let gate = gate(8, 1); + let token = CancellationToken::new(); + let held = gate.acquire_tool(Some("cli:a:d1"), &token).await.unwrap(); + + let waiter_token = CancellationToken::new(); + let waiter = tokio::spawn({ + let gate = gate.clone(); + let waiter_token = waiter_token.clone(); + async move { gate.acquire_tool(Some("cli:a:d1"), &waiter_token).await } + }); + tokio::time::sleep(Duration::from_millis(50)).await; + waiter_token.cancel(); + let error = waiter.await.unwrap().unwrap_err(); + assert!(matches!(error, GateError::Cancelled)); + drop(held); + assert_eq!(gate.available_tool_permits(), 8); + } + + #[tokio::test] + async fn run_quota_is_scoped_per_session() { + let gate = gate(8, 8); + let token = CancellationToken::new(); + let held = gate.acquire_run("cli:a:d1", &token).await.unwrap(); + + let blocked = tokio::spawn({ + let gate = gate.clone(); + let token = token.clone(); + async move { gate.acquire_run("cli:a:d1", &token).await } + }); + tokio::time::sleep(Duration::from_millis(50)).await; + assert!(!blocked.is_finished()); + + let other = gate.acquire_run("cli:b:d2", &token).await.unwrap(); + drop(other); + drop(held); + blocked.await.unwrap().unwrap(); + } + + #[tokio::test] + async fn keyed_semaphores_are_reclaimed_when_idle() { + let keyed = Arc::new(KeyedSemaphores::new(1)); + let semaphore = keyed.semaphore("cli:a:d1"); + assert_eq!(keyed.map.len(), 1); + drop(semaphore); + keyed.semaphore("cli:b:d2"); + assert_eq!(keyed.map.len(), 1, "idle session entry must be reclaimed"); + } + + #[tokio::test] + async fn waiting_parent_holds_no_step_permits() { + // With a session cap of 1, a parent that merely waits for its child + // must leave the session's single provider permit free; otherwise + // nested foreground delegation would deadlock. + let gate = gate(1, 1); + let token = CancellationToken::new(); + let parent_step = gate + .acquire_provider(Some("cli:a:d1"), &token) + .await + .unwrap(); + drop(parent_step); + // Parent is now in the waiting_children state: no permits held. + eventually(|| gate.available_provider_permits() == 8).await; + let child = gate + .acquire_provider(Some("cli:a:d1"), &token) + .await + .unwrap(); + drop(child); + assert_eq!(gate.available_provider_permits(), 8); + } +} diff --git a/src/agent/inbox.rs b/src/agent/inbox.rs new file mode 100644 index 0000000..91915b1 --- /dev/null +++ b/src/agent/inbox.rs @@ -0,0 +1,52 @@ +use std::sync::{Arc, RwLock, Weak}; + +/// Wake contract implemented by the SessionManager. The notifier holds only +/// a weak reference so the coordinator can be dropped independently and no +/// release cycle is formed. +#[async_trait::async_trait] +pub trait AgentInboxWakeTarget: Send + Sync { + async fn wake_agent_inbox(&self, session_id: &str, revision: i64); +} + +/// Late-bound wake bus between the AgentCoordinator and Session workers. +/// Event commits never wait for a wake; the durable inbox is the source of +/// truth and a lost wake only delays the next claim until the periodic +/// re-check. +pub struct AgentInboxNotifier { + target: RwLock>>, +} + +impl Default for AgentInboxNotifier { + fn default() -> Self { + Self { + target: RwLock::new(None), + } + } +} + +impl AgentInboxNotifier { + pub fn new() -> Arc { + Arc::new(Self::default()) + } + + pub fn bind(&self, target: Weak) { + *self + .target + .write() + .unwrap_or_else(|poisoned| poisoned.into_inner()) = Some(target); + } + + /// Best-effort wake. A missing or dead target is not an error; the + /// periodic scanner will claim the event later. + pub async fn notify(&self, session_id: &str, revision: i64) { + let target = self + .target + .read() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .as_ref() + .and_then(|weak| weak.upgrade()); + if let Some(target) = target { + target.wake_agent_inbox(session_id, revision).await; + } + } +} diff --git a/src/agent/mod.rs b/src/agent/mod.rs index 1ed26e5..2b7e504 100644 --- a/src/agent/mod.rs +++ b/src/agent/mod.rs @@ -1,14 +1,28 @@ pub mod agent_loop; +pub mod catalog; pub mod context_compressor; +pub mod coordinator; +pub mod definition; +pub mod gate; +pub mod inbox; pub mod media_handler; +pub mod projection; +pub mod run; pub mod steering; pub mod sub_agent; pub mod system_prompt; pub mod turn_event; pub use agent_loop::{AgentError, AgentLoop, AgentProcessResult}; +pub use catalog::{AgentCatalog, AgentCatalogError}; pub use context_compressor::{ContextCompressor, estimate_tokens}; -pub use steering::{SteeringDrain, SteeringMailbox, SteeringPushError}; +pub use coordinator::{AgentCoordinator, CoordinatorError}; +pub use definition::{AgentDefinition, AgentLimits}; +pub use gate::ExecutionGate; +pub use inbox::{AgentInboxNotifier, AgentInboxWakeTarget}; +pub use projection::AgentProjectionHub; +pub use run::{AgentBudget, AgentCaller, AgentExecutionContext}; +pub use steering::{SteeringDrain, SteeringPushError, TurnInput, TurnInputSource, TurnMailbox}; pub use sub_agent::{ DelegateContext, ExecutionMode, SubAgentConfig, SubAgentError, SubAgentManager, SubAgentResult, TaskNotification, TaskStatus, diff --git a/src/agent/projection.rs b/src/agent/projection.rs new file mode 100644 index 0000000..7d8e410 --- /dev/null +++ b/src/agent/projection.rs @@ -0,0 +1,74 @@ +//! Broadcast projection for durable Agent run/event updates. +//! +//! The hub follows the WorkManager plan-change broadcast pattern: the +//! Coordinator publishes a bounded view after each commit; the gateway +//! relays it to WebSocket clients. A lost broadcast is never fatal — the +//! client recalibrates with `GetAgentRuns`, whose `revision` comes from +//! `agent_session_state`. + +use crate::protocol::{AgentEventView, AgentRunView}; + +/// One projected change for a root session. Exactly one of `run`/`event` is +/// present. +#[derive(Debug, Clone)] +pub struct AgentProjection { + pub session_id: String, + pub revision: i64, + pub run: Option, + pub event: Option, +} + +#[derive(Debug, Clone)] +pub struct AgentProjectionHub { + tx: tokio::sync::broadcast::Sender, +} + +impl Default for AgentProjectionHub { + fn default() -> Self { + Self::new() + } +} + +impl AgentProjectionHub { + pub fn new() -> Self { + let (tx, _) = tokio::sync::broadcast::channel(256); + Self { tx } + } + + pub fn subscribe(&self) -> tokio::sync::broadcast::Receiver { + self.tx.subscribe() + } + + /// Publish a run update. Best-effort: relays lag and drop the event, + /// clients recalibrate with a full query. + pub fn publish(&self, projection: AgentProjection) { + let _ = self.tx.send(projection); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn publish_reaches_subscribers_and_drop_is_nonfatal() { + let hub = AgentProjectionHub::new(); + let mut rx = hub.subscribe(); + hub.publish(AgentProjection { + session_id: "cli:test:d".to_string(), + revision: 1, + run: None, + event: None, + }); + let received = rx.try_recv().unwrap(); + assert_eq!(received.session_id, "cli:test:d"); + // No subscribers after the first is dropped; publish must not panic. + drop(rx); + hub.publish(AgentProjection { + session_id: "cli:test:d".to_string(), + revision: 2, + run: None, + event: None, + }); + } +} diff --git a/src/agent/run.rs b/src/agent/run.rs new file mode 100644 index 0000000..c1ce8f8 --- /dev/null +++ b/src/agent/run.rs @@ -0,0 +1,160 @@ +use std::sync::Arc; +use std::sync::Mutex; +use std::sync::atomic::{AtomicUsize, Ordering}; + +use tokio_util::sync::CancellationToken; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AgentCaller { + Root, + Agent, +} + +#[derive(Debug, Clone)] +pub struct AgentBudget { + pub remaining_runs: usize, + pub remaining_depth: u16, +} + +/// A signal this run emitted, preserved so the terminal completion payload +/// can reference the signal IDs for cross-checking. +#[derive(Debug, Clone)] +pub struct EmittedSignal { + pub signal_id: String, + pub severity: String, + pub summary: String, +} + +#[derive(Debug, Clone)] +pub struct AgentExecutionContext { + pub root_session_id: String, + pub root_turn_id: Option, + pub run_id: String, + /// Execution attempt identifier owning conditional state transitions in + /// Storage. Equal to `run_id` for the first attempt. + pub execution_id: String, + pub group_id: Option, + pub parent_run_id: Option, + pub caller_agent_id: String, + pub current_agent_id: String, + /// Agent IDs already present in this execution chain, including current. + pub ancestry: Vec, + pub depth: u16, + pub plan_item_id: Option, + pub cancellation: CancellationToken, + pub budget: AgentBudget, + /// Shared admission counter across the whole delegation tree, including + /// the root run that owns this context. Durable run accounting replaces + /// it once the Coordinator persists runs; until then it is the only + /// tree-wide enforcement of `max_runs_per_tree`. + pub tree_runs: Arc, + /// Durable emit_signal contract; `None` means this run has no signal + /// capability and must not see the `emit_signal` tool. + pub signal_contract: Option>, + /// Signals accepted by this run so far, in emit order. + pub emitted_signals: Arc>>, +} + +impl AgentExecutionContext { + pub fn child( + parent: &Arc, + run_id: String, + target: String, + plan_item_id: Option, + cancellation: CancellationToken, + ) -> Self { + let mut ancestry = parent.ancestry.clone(); + ancestry.push(target.clone()); + Self { + root_session_id: parent.root_session_id.clone(), + root_turn_id: parent.root_turn_id.clone(), + run_id: run_id.clone(), + execution_id: run_id, + group_id: None, + parent_run_id: Some(parent.run_id.clone()), + caller_agent_id: parent.current_agent_id.clone(), + current_agent_id: target, + ancestry, + depth: parent.depth.saturating_add(1), + plan_item_id, + cancellation, + budget: AgentBudget { + remaining_runs: parent.budget.remaining_runs.saturating_sub(1), + remaining_depth: parent.budget.remaining_depth.saturating_sub(1), + }, + tree_runs: parent.tree_runs.clone(), + signal_contract: parent.signal_contract.clone(), + emitted_signals: Arc::new(Mutex::new(Vec::new())), + } + } + + /// Reserve one tree-run slot for a child that has already passed the + /// caller's budget checks. Returns the total number of runs in the tree + /// after reservation, or `None` (with the counter restored) when the + /// reservation would exceed `max_runs_per_tree`. + pub fn reserve_tree_run(&self, max_runs_per_tree: usize) -> Option { + let previous = self.tree_runs.fetch_add(1, Ordering::SeqCst); + let total = previous.saturating_add(1); + if total > max_runs_per_tree { + self.tree_runs.fetch_sub(1, Ordering::SeqCst); + return None; + } + Some(total) + } + + /// Remaining tree capacity for batch admission checks. + pub fn remaining_tree_runs(&self, max_runs_per_tree: usize) -> usize { + max_runs_per_tree.saturating_sub(self.tree_runs.load(Ordering::SeqCst)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn root_context() -> AgentExecutionContext { + AgentExecutionContext { + root_session_id: "cli:test:dialog".to_string(), + root_turn_id: None, + run_id: "run-root".to_string(), + execution_id: "run-root".to_string(), + group_id: None, + parent_run_id: None, + caller_agent_id: "ROOT".to_string(), + current_agent_id: "researcher".to_string(), + ancestry: vec!["researcher".to_string()], + depth: 1, + plan_item_id: None, + cancellation: CancellationToken::new(), + budget: AgentBudget { + remaining_runs: 15, + remaining_depth: 3, + }, + tree_runs: Arc::new(AtomicUsize::new(1)), + signal_contract: None, + emitted_signals: Arc::new(Mutex::new(Vec::new())), + } + } + + #[test] + fn tree_run_reservation_is_shared_and_bounded() { + let parent = Arc::new(root_context()); + assert_eq!(parent.remaining_tree_runs(3), 2); + + assert_eq!(parent.reserve_tree_run(3), Some(2)); + assert_eq!(parent.reserve_tree_run(3), Some(3)); + assert_eq!(parent.reserve_tree_run(3), None); + assert_eq!(parent.tree_runs.load(Ordering::SeqCst), 3); + assert_eq!(parent.remaining_tree_runs(3), 0); + + let child = AgentExecutionContext::child( + &parent, + "run-child".to_string(), + "reviewer".to_string(), + None, + CancellationToken::new(), + ); + assert!(Arc::ptr_eq(&parent.tree_runs, &child.tree_runs)); + assert_eq!(child.remaining_tree_runs(3), 0); + } +} diff --git a/src/agent/steering.rs b/src/agent/steering.rs index e9b30b1..0aeeba5 100644 --- a/src/agent/steering.rs +++ b/src/agent/steering.rs @@ -1,23 +1,166 @@ -//! Bounded, session-owned mailbox for same-turn user steering. +//! Bounded, session-owned mailbox for same-turn steering input. //! //! A mailbox is intentionally separate from the session work queue. The -//! gateway can accept a normal user message while a turn is running and place -//! it here; [`AgentLoop`](super::AgentLoop) drains it only at safe model -//! boundaries (after a complete tool batch, or before deciding that a -//! response is final). The state transition performed by -//! [`SteeringMailbox::drain_or_close`] is atomic with respect to producers, +//! gateway can accept a normal user message or a durable Agent steer event +//! while a turn is running and place it here; +//! [`AgentLoop`](super::AgentLoop) drains it only at safe model boundaries +//! (after a complete tool batch, or before deciding that a response is +//! final). The state transition performed by +//! [`TurnMailbox::drain_or_close`] is atomic with respect to producers, //! which means an input is either accepted by the active turn or rejected so //! the caller can put it on the next-turn queue -- never both and never //! neither. +//! +//! Durable steer events use a two-phase admission: the session first +//! reserves an agent-lane entry, persists the `leased → admitted` transition +//! (with the turn id), and only then activates the reservation into the +//! drainable queue. Reservations and drained-but-uncommitted entries carry +//! their storage lease token so an abandoned turn can return every admitted +//! event to `pending` instead of losing it. -use crate::bus::ChatMessage; +use crate::bus::{ChatMessage, MediaRef, MessageSource}; use std::collections::VecDeque; use std::sync::{Arc, Mutex}; -/// Default maximum number of steering messages accepted by one active turn. -pub const DEFAULT_MAX_STEERING_MESSAGES: usize = 32; -/// Default aggregate UTF-8 byte budget for pending steering messages. -pub const DEFAULT_MAX_STEERING_BYTES: usize = 64 * 1024; +/// Default maximum number of user steering messages accepted by one active +/// turn. +pub const DEFAULT_MAX_USER_STEERING_MESSAGES: usize = 32; +/// Default aggregate UTF-8 byte budget for pending user steering messages. +pub const DEFAULT_MAX_USER_STEERING_BYTES: usize = 64 * 1024; +/// Default maximum number of Agent steer events accepted by one active turn. +pub const DEFAULT_MAX_AGENT_STEERING_MESSAGES: usize = 8; +/// Default aggregate UTF-8 byte budget for pending Agent steer events. +pub const DEFAULT_MAX_AGENT_STEERING_BYTES: usize = 32 * 1024; + +/// Origin of one mailbox input. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum TurnInputSource { + User, + AgentSignal { run_id: String, agent_id: String }, + AgentCompletion { run_id: String, agent_id: String }, + AgentGroupCompletion { group_id: String }, +} + +impl TurnInputSource { + pub fn is_agent(&self) -> bool { + !matches!(self, Self::User) + } +} + +/// How the input reached the mailbox. Queue inputs belong to the next Turn; +/// only Steer entries are drained by the active Turn. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum InputDelivery { + Queue, + Steer, +} + +/// One steering input retained by the active Turn. +#[derive(Debug, Clone)] +pub struct TurnInput { + pub id: String, + pub sequence: u64, + pub source: TurnInputSource, + pub delivery: InputDelivery, + pub content: String, + pub media_refs: Vec, + /// Durable inbox event id; `Some` only for Agent steer events. + pub durable_event_id: Option, + pub received_at: i64, + /// Channel attribution for user inputs, preserved through the turn. + pub message_source: Option, + /// Storage lease token of a durable Agent event; the mailbox keeps it so + /// an abandoned turn can release the event back to `pending`. + pub(crate) lease_token: Option, +} + +impl TurnInput { + pub fn user( + id: impl Into, + content: impl Into, + media_refs: Vec, + message_source: Option, + received_at: i64, + ) -> Self { + Self { + id: id.into(), + sequence: 0, + source: TurnInputSource::User, + delivery: InputDelivery::Steer, + content: content.into(), + media_refs, + durable_event_id: None, + received_at, + message_source, + lease_token: None, + } + } + + /// Project this input into a provider-compatible user message. Agent + /// inputs stay hidden from client history (the Signal UI comes from the + /// durable event projection) but keep their typed source for rendering + /// and cancellation recovery. + pub fn into_chat_message(self, turn_id: String, iteration: u32) -> ChatMessage { + let (client_visibility, source) = match &self.source { + TurnInputSource::User => { + let visibility = crate::bus::ClientVisibility::Visible; + let source = self.message_source.clone(); + (visibility, source) + } + TurnInputSource::AgentSignal { run_id, agent_id } => { + let source = MessageSource { + kind: crate::bus::SourceKind::AgentSignal, + from_channel: None, + from_session: None, + from_user_id: None, + system_name: None, + task_id: self.durable_event_id.clone(), + from_run_id: Some(run_id.clone()), + from_agent_id: Some(agent_id.clone()), + group_id: None, + }; + (crate::bus::ClientVisibility::Hidden, Some(source)) + } + TurnInputSource::AgentCompletion { run_id, agent_id } => { + let source = MessageSource { + kind: crate::bus::SourceKind::AgentCompletion, + from_channel: None, + from_session: None, + from_user_id: None, + system_name: None, + task_id: self.durable_event_id.clone(), + from_run_id: Some(run_id.clone()), + from_agent_id: Some(agent_id.clone()), + group_id: None, + }; + (crate::bus::ClientVisibility::Hidden, Some(source)) + } + TurnInputSource::AgentGroupCompletion { group_id } => { + let source = MessageSource { + kind: crate::bus::SourceKind::AgentGroupCompletion, + from_channel: None, + from_session: None, + from_user_id: None, + system_name: None, + task_id: self.durable_event_id.clone(), + from_run_id: None, + from_agent_id: None, + group_id: Some(group_id.clone()), + }; + (crate::bus::ClientVisibility::Hidden, Some(source)) + } + }; + let mut message = ChatMessage::user(self.content); + message.id = self.id; + message.turn_id = Some(turn_id); + message.iteration = Some(iteration); + message.media_refs = self.media_refs; + message.timestamp = self.received_at; + message.client_visibility = client_visibility; + message.source = source; + message + } +} #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum MailboxPhase { @@ -28,49 +171,54 @@ enum MailboxPhase { #[derive(Debug)] struct MailboxState { phase: MailboxPhase, - pending: VecDeque, - pending_bytes: usize, - /// Messages drained at a safe boundary but not yet committed to durable - /// history. Keeping their count/size reserved prevents concurrent - /// producers from filling the capacity that an error retry may need to - /// restore. - in_flight_messages: usize, - in_flight_bytes: usize, - /// Exact drained messages retained until commit. This lets Session - /// recover a successful AgentLoop result if its subsequent persistence - /// transaction fails. - in_flight: VecDeque, + pending: VecDeque, + reserved: VecDeque, + /// Drained at a safe boundary but not yet committed to durable history. + /// Keeping their count/size reserved prevents concurrent producers from + /// filling the capacity that an error retry may need to restore. + in_flight: VecDeque, } -/// Error returned when the active turn cannot accept a steering message. -/// -/// The original message is returned in the error so the caller can enqueue it -/// as a normal next-turn task without cloning or losing media metadata. +impl MailboxState { + fn user_lane_used(&self) -> (usize, usize) { + let mut count = 0usize; + let mut bytes = 0usize; + for input in self.pending.iter().chain(self.in_flight.iter()) { + if !input.source.is_agent() { + count = count.saturating_add(1); + bytes = bytes.saturating_add(input_size_bytes(input)); + } + } + (count, bytes) + } + + fn agent_lane_used(&self) -> (usize, usize) { + let mut count = 0usize; + let mut bytes = 0usize; + for input in self + .pending + .iter() + .chain(self.in_flight.iter()) + .chain(self.reserved.iter()) + { + if input.source.is_agent() { + count = count.saturating_add(1); + bytes = bytes.saturating_add(input_size_bytes(input)); + } + } + (count, bytes) + } +} + +/// Error returned when the active turn cannot accept a steering input. #[derive(Debug, Clone)] pub enum SteeringPushError { - /// The turn has reached a terminal boundary. Route the message to the - /// session's ordinary queue. - Closed(Box), - /// The mailbox is accepting input, but its bounded capacity is exhausted. - /// Route the message to the ordinary queue (and normally notify the user). - Full(Box), -} - -impl SteeringPushError { - /// Recover the message that was rejected by [`SteeringMailbox::try_push`]. - pub fn into_message(self) -> ChatMessage { - match self { - Self::Closed(message) | Self::Full(message) => *message, - } - } - - pub fn is_closed(&self) -> bool { - matches!(self, Self::Closed(_)) - } - - pub fn is_full(&self) -> bool { - matches!(self, Self::Full(_)) - } + /// The turn has reached a terminal boundary. Route the input to the + /// session's ordinary queue (or, for durable events, keep it pending). + Closed, + /// The mailbox is accepting input, but its bounded lane capacity is + /// exhausted. Route the input to the ordinary queue. + Full, } /// Result of the atomic final-response boundary operation. @@ -78,61 +226,100 @@ impl SteeringPushError { pub enum SteeringDrain { /// One or more inputs were accepted and removed from the mailbox. The /// mailbox remains open for a subsequent safe boundary. - Messages(Vec), + Messages(Vec), /// No pending input existed. The mailbox is now closed; later producers /// receive [`SteeringPushError::Closed`]. Closed, } -/// Shared state for user steering during one active AgentLoop execution. -/// -/// Cloning a mailbox is cheap and shares the same mutex-protected state. In -/// practice the session stores an `Arc` in its active-turn -/// handle and gives another clone to [`AgentTurnContext`](super::AgentTurnContext). -#[derive(Clone)] -pub struct SteeringMailbox { - state: Arc>, - max_messages: usize, - max_bytes: usize, +/// What an abandoned turn returns for requeue/release handling. +#[derive(Debug, Default)] +pub struct MailboxTake { + /// Pending non-durable user inputs. + pub user_inputs: Vec, + /// Durable Agent entries activated and drained or still pending: + /// `(event_id, lease_token)` of `admitted` events. + pub admitted_leases: Vec<(String, String)>, + /// Reserved-but-not-activated Agent entries: `(event_id, lease_token)` + /// of `leased` (or admitted) events. + pub reserved_leases: Vec<(String, String)>, } -impl std::fmt::Debug for SteeringMailbox { +impl MailboxTake { + pub fn is_empty(&self) -> bool { + self.user_inputs.is_empty() + && self.admitted_leases.is_empty() + && self.reserved_leases.is_empty() + } + + pub fn len(&self) -> usize { + self.user_inputs.len() + self.admitted_leases.len() + self.reserved_leases.len() + } +} + +/// Shared state for same-turn steering during one active AgentLoop +/// execution. +/// +/// Cloning a mailbox is cheap and shares the same mutex-protected state. In +/// practice the session stores an `Arc` in its active-turn +/// handle and gives another clone to [`AgentTurnContext`](super::AgentTurnContext). +#[derive(Clone)] +pub struct TurnMailbox { + state: Arc>, + max_user_messages: usize, + max_user_bytes: usize, + max_agent_messages: usize, + max_agent_bytes: usize, +} + +impl std::fmt::Debug for TurnMailbox { fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let state = self .state .lock() .unwrap_or_else(|poisoned| poisoned.into_inner()); formatter - .debug_struct("SteeringMailbox") + .debug_struct("TurnMailbox") .field("phase", &state.phase) - .field("pending_messages", &state.pending.len()) - .field("pending_bytes", &state.pending_bytes) - .field("max_messages", &self.max_messages) - .field("max_bytes", &self.max_bytes) + .field("pending", &state.pending.len()) + .field("reserved", &state.reserved.len()) + .field("in_flight", &state.in_flight.len()) + .field("max_user_messages", &self.max_user_messages) + .field("max_agent_messages", &self.max_agent_messages) .finish() } } -impl SteeringMailbox { - /// Construct a mailbox using the product defaults (32 messages/64 KiB). +impl TurnMailbox { + /// Construct a mailbox using the product defaults (32 user/8 agent). pub fn new() -> Self { - Self::with_limits(DEFAULT_MAX_STEERING_MESSAGES, DEFAULT_MAX_STEERING_BYTES) + Self::with_limits( + DEFAULT_MAX_USER_STEERING_MESSAGES, + DEFAULT_MAX_USER_STEERING_BYTES, + DEFAULT_MAX_AGENT_STEERING_MESSAGES, + DEFAULT_MAX_AGENT_STEERING_BYTES, + ) } - /// Construct a mailbox with explicit bounded capacities. Zero limits are - /// allowed and make every push return [`SteeringPushError::Full`]. - pub fn with_limits(max_messages: usize, max_bytes: usize) -> Self { + /// Construct a mailbox with explicit bounded capacities. Zero limits + /// are allowed and make every push return [`SteeringPushError::Full`]. + pub fn with_limits( + max_user_messages: usize, + max_user_bytes: usize, + max_agent_messages: usize, + max_agent_bytes: usize, + ) -> Self { Self { state: Arc::new(Mutex::new(MailboxState { phase: MailboxPhase::Accepting, pending: VecDeque::new(), - pending_bytes: 0, - in_flight_messages: 0, - in_flight_bytes: 0, + reserved: VecDeque::new(), in_flight: VecDeque::new(), })), - max_messages, - max_bytes, + max_user_messages, + max_user_bytes, + max_agent_messages, + max_agent_bytes, } } @@ -141,44 +328,99 @@ impl SteeringMailbox { Arc::new(Self::new()) } - /// Return an `Arc` suitable for storing in Session with explicit limits. - pub fn shared_with_limits(max_messages: usize, max_bytes: usize) -> Arc { - Arc::new(Self::with_limits(max_messages, max_bytes)) - } - - /// Try to accept one real user [`ChatMessage`]. + /// Try to accept one real user steering input. /// /// This operation and the final close operation use the same mutex. A - /// producer racing with `drain_or_close` therefore receives a deterministic - /// result and can route a rejected message to the ordinary queue. - pub fn try_push(&self, message: ChatMessage) -> Result<(), SteeringPushError> { - let message_bytes = message_size_bytes(&message); + /// producer racing with `drain_or_close` therefore receives a + /// deterministic result and can route a rejected input to the ordinary + /// queue. + pub fn try_push_user(&self, input: TurnInput) -> Result<(), SteeringPushError> { + debug_assert!(!input.source.is_agent()); + let input_bytes = input_size_bytes(&input); let mut state = self .state .lock() .unwrap_or_else(|poisoned| poisoned.into_inner()); if state.phase == MailboxPhase::Closed { - return Err(SteeringPushError::Closed(Box::new(message))); + return Err(SteeringPushError::Closed); } - if state.pending.len().saturating_add(state.in_flight_messages) >= self.max_messages - || state - .pending_bytes - .saturating_add(state.in_flight_bytes) - .saturating_add(message_bytes) - > self.max_bytes + let (count, bytes) = state.user_lane_used(); + if count >= self.max_user_messages + || bytes.saturating_add(input_bytes) > self.max_user_bytes { - return Err(SteeringPushError::Full(Box::new(message))); + return Err(SteeringPushError::Full); } - state.pending_bytes = state.pending_bytes.saturating_add(message_bytes); - state.pending.push_back(message); + state.pending.push_back(input); Ok(()) } + /// Reserve an Agent steer event for the active Turn (two-phase admission, + /// step 2). The entry is not drainable until + /// [`activate_reserved`](Self::activate_reserved) succeeds for the same + /// Turn. `lease_token` is the storage token of the `leased` event. + pub fn try_reserve_steer( + &self, + input: TurnInput, + lease_token: String, + ) -> Result<(), SteeringPushError> { + debug_assert!(input.source.is_agent()); + let input_bytes = input_size_bytes(&input); + let mut state = self + .state + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + if state.phase == MailboxPhase::Closed { + return Err(SteeringPushError::Closed); + } + let (count, bytes) = state.agent_lane_used(); + if count >= self.max_agent_messages + || bytes.saturating_add(input_bytes) > self.max_agent_bytes + { + return Err(SteeringPushError::Full); + } + let mut input = input; + input.lease_token = Some(lease_token); + state.reserved.push_back(input); + Ok(()) + } + + /// Activate all reservations into the drainable queue. Returns the + /// number activated. Callers invoke this only after the durable + /// `leased → admitted` transition succeeded for the same Turn. + pub fn activate_reserved(&self) -> usize { + let mut state = self + .state + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + let count = state.reserved.len(); + let reserved: Vec<_> = state.reserved.drain(..).collect(); + state.pending.extend(reserved); + count + } + + /// Remove all reservations without activating them. Returns + /// `(event_id, lease_token)` pairs so the caller can release the + /// still-leased events back to `pending`. + pub fn cancel_reserved(&self) -> Vec<(String, String)> { + let mut state = self + .state + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + state + .reserved + .drain(..) + .filter_map(|input| { + let token = input.lease_token.clone()?; + Some((input.durable_event_id.unwrap_or_default(), token)) + }) + .collect() + } + /// Drain currently pending inputs while leaving the mailbox open. /// /// This is used after a complete tool-call batch. It intentionally does /// not close the mailbox: another input may steer a later iteration. - pub fn drain(&self) -> Vec { + pub fn drain(&self) -> Vec { let mut state = self .state .lock() @@ -212,26 +454,36 @@ impl SteeringMailbox { state.phase = MailboxPhase::Closed; } - /// Close acceptance and return all pending messages. This is convenient - /// for cancellation/error paths where the caller immediately owns the - /// rejected messages. Any in-flight batch is intentionally discarded; - /// `/stop` uses this method to preserve its queue-clearing semantics. - pub fn close_and_take_pending(&self) -> Vec { + /// Close acceptance and collect everything for requeue/release handling. + /// Any in-flight batch is returned through `admitted_leases`; `/stop` + /// uses this method to preserve its queue-clearing semantics for user + /// input while still releasing durable events back to `pending`. + pub fn close_and_take_pending(&self) -> MailboxTake { let mut state = self .state .lock() .unwrap_or_else(|poisoned| poisoned.into_inner()); state.phase = MailboxPhase::Closed; - // Cancellation is an explicit discard boundary. Any in-flight - // messages that were already drained belong to this cancelled turn - // and must not reserve capacity forever. - take_pending_locked(&mut state) + take_all_locked(&mut state) } - /// Restore all messages drained by AgentLoop since the last commit. This - /// is useful when the AgentLoop completed but Session's durable write then - /// failed: the next retry/queue operation can replay the exact accepted - /// inputs instead of silently losing them. + /// Take pending inputs without changing whether producers may still push. + /// + /// Normally used after `close()`; keeping this method explicit makes it + /// possible for Session to transfer accepted-but-unprocessed input to its + /// FIFO queue without opening a race with a new turn. + pub fn take_pending(&self) -> MailboxTake { + let mut state = self + .state + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + take_all_locked(&mut state) + } + + /// Restore all inputs drained by AgentLoop since the last commit. This + /// is useful when the AgentLoop completed but Session's durable write + /// then failed: the next retry/queue operation can replay the exact + /// accepted inputs instead of silently losing them. pub fn restore_drained(&self) { let mut state = self .state @@ -240,33 +492,29 @@ impl SteeringMailbox { restore_in_flight_locked(&mut state); } - /// Restore messages drained by AgentLoop when a provider/tool error makes - /// the current invocation retry from persisted history. The messages are + /// Restore inputs drained by AgentLoop when a provider/tool error makes + /// the current invocation retry from persisted history. The inputs are /// prepended in their original order and their reserved capacity is - /// released. `drain()`/`drain_or_close()` reserve capacity while a batch is - /// in-flight, so this operation cannot overflow a bounded mailbox due to a - /// racing producer. - pub fn restore_front(&self, messages: Vec) { - if messages.is_empty() { + /// released. `drain()`/`drain_or_close()` reserve capacity while a batch + /// is in-flight, so this operation cannot overflow a bounded mailbox due + /// to a racing producer. + pub fn restore_front(&self, inputs: Vec) { + if inputs.is_empty() { return; } - let restored_bytes = messages.iter().map(message_size_bytes).sum::(); let mut state = self .state .lock() .unwrap_or_else(|poisoned| poisoned.into_inner()); - state.in_flight_messages = state.in_flight_messages.saturating_sub(messages.len()); - state.in_flight_bytes = state.in_flight_bytes.saturating_sub(restored_bytes); - for _ in 0..messages.len() { - state.in_flight.pop_front(); + for _ in 0..inputs.len() { + state.in_flight.pop_back(); } - for message in messages.into_iter().rev() { - state.pending.push_front(message); + for input in inputs.into_iter().rev() { + state.pending.push_front(input); } - state.pending_bytes = state.pending_bytes.saturating_add(restored_bytes); } - /// Mark all previously drained messages as durably committed. Session + /// Mark all previously drained inputs as durably committed. Session /// calls this only after the complete Turn persistence transaction /// succeeds. It is a no-op when no steering batch was consumed. pub fn commit_drained(&self) { @@ -274,22 +522,21 @@ impl SteeringMailbox { .state .lock() .unwrap_or_else(|poisoned| poisoned.into_inner()); - state.in_flight_messages = 0; - state.in_flight_bytes = 0; state.in_flight.clear(); } - /// Take pending inputs without changing whether producers may still push. - /// - /// Normally used after `close()`; keeping this method explicit makes it - /// possible for Session to transfer accepted-but-unprocessed input to its - /// FIFO queue without opening a race with a new turn. - pub fn take_pending(&self) -> Vec { - let mut state = self + /// Durable event ids currently drained into this Turn but not yet + /// committed. Session consumes them atomically with the Turn commit. + pub fn durable_in_flight_ids(&self) -> Vec { + let state = self .state .lock() .unwrap_or_else(|poisoned| poisoned.into_inner()); - take_pending_locked(&mut state) + state + .in_flight + .iter() + .filter_map(|input| input.durable_event_id.clone()) + .collect() } pub fn is_closed(&self) -> bool { @@ -312,77 +559,78 @@ impl SteeringMailbox { self.len() == 0 } - pub fn max_messages(&self) -> usize { - self.max_messages + pub fn max_user_messages(&self) -> usize { + self.max_user_messages } - pub fn max_bytes(&self) -> usize { - self.max_bytes + pub fn max_agent_messages(&self) -> usize { + self.max_agent_messages } } -impl Default for SteeringMailbox { +impl Default for TurnMailbox { fn default() -> Self { Self::new() } } /// Approximate the bounded payload size without serializing the complete -/// message. Content, media paths/types, tool metadata and source fields are -/// all untrusted input; counting their UTF-8 bytes gives a conservative enough -/// guard while retaining the original message losslessly. -fn message_size_bytes(message: &ChatMessage) -> usize { - let mut bytes = message.id.len() - + message.role.len() - + message.content.len() - + message.reasoning_content.as_deref().map_or(0, str::len) - + message.turn_id.as_deref().map_or(0, str::len) - + message.tool_call_id.as_deref().map_or(0, str::len) - + message.tool_name.as_deref().map_or(0, str::len); - for media in &message.media_refs { +/// input. Content, media paths/types and source fields are all untrusted +/// input; counting their UTF-8 bytes gives a conservative enough guard while +/// retaining the original input losslessly. +fn input_size_bytes(input: &TurnInput) -> usize { + let mut bytes = input.id.len() + + input.content.len() + + input.durable_event_id.as_deref().map_or(0, str::len) + + input.lease_token.as_deref().map_or(0, str::len); + for media in &input.media_refs { bytes = bytes.saturating_add(media.path.len() + media.media_type.len()); } - if let Some(tool_calls) = &message.tool_calls { - for call in tool_calls { - bytes = bytes - .saturating_add(call.id.len()) - .saturating_add(call.name.len()) - .saturating_add(call.arguments.to_string().len()); - } + if let Some(source) = input.message_source.as_ref() { + bytes = bytes + .saturating_add(source.from_channel.as_deref().map_or(0, str::len)) + .saturating_add(source.from_user_id.as_deref().map_or(0, str::len)); } bytes } -fn drain_pending_locked(state: &mut MailboxState) -> Vec { - let messages: Vec<_> = state.pending.drain(..).collect(); - let bytes = messages.iter().map(message_size_bytes).sum::(); - state.pending_bytes = state.pending_bytes.saturating_sub(bytes); - state.in_flight_messages = state.in_flight_messages.saturating_add(messages.len()); - state.in_flight_bytes = state.in_flight_bytes.saturating_add(bytes); - state.in_flight.extend(messages.iter().cloned()); - messages +fn drain_pending_locked(state: &mut MailboxState) -> Vec { + let inputs: Vec<_> = state.pending.drain(..).collect(); + state.in_flight.extend(inputs.iter().cloned()); + inputs } -fn take_pending_locked(state: &mut MailboxState) -> Vec { - state.pending_bytes = 0; - state.in_flight_messages = 0; - state.in_flight_bytes = 0; - state.in_flight.clear(); - state.pending.drain(..).collect() +fn take_all_locked(state: &mut MailboxState) -> MailboxTake { + let mut take = MailboxTake::default(); + for input in state.pending.drain(..).chain(state.in_flight.drain(..)) { + if input.source.is_agent() { + if let Some(token) = input.lease_token.clone() + && let Some(event_id) = input.durable_event_id.clone() + { + take.admitted_leases.push((event_id, token)); + } + } else { + take.user_inputs.push(input); + } + } + for input in state.reserved.drain(..) { + if let Some(token) = input.lease_token.clone() + && let Some(event_id) = input.durable_event_id.clone() + { + take.reserved_leases.push((event_id, token)); + } + } + take } fn restore_in_flight_locked(state: &mut MailboxState) { if state.in_flight.is_empty() { return; } - let messages: Vec<_> = state.in_flight.drain(..).collect(); - let bytes = messages.iter().map(message_size_bytes).sum::(); - state.in_flight_messages = 0; - state.in_flight_bytes = 0; - for message in messages.into_iter().rev() { - state.pending.push_front(message); + let inputs: Vec<_> = state.in_flight.drain(..).collect(); + for input in inputs.into_iter().rev() { + state.pending.push_front(input); } - state.pending_bytes = state.pending_bytes.saturating_add(bytes); } #[cfg(test)] @@ -391,99 +639,165 @@ mod tests { use std::sync::Arc; use std::thread; + fn user_input(id: &str, content: &str) -> TurnInput { + TurnInput::user(id, content, Vec::new(), None, 1_000) + } + + fn agent_signal_input(id: &str, event_id: &str) -> TurnInput { + TurnInput { + id: id.to_string(), + sequence: 0, + source: TurnInputSource::AgentSignal { + run_id: "run-1".to_string(), + agent_id: "researcher".to_string(), + }, + delivery: InputDelivery::Steer, + content: "signal summary".to_string(), + media_refs: Vec::new(), + durable_event_id: Some(event_id.to_string()), + received_at: 1_000, + message_source: None, + lease_token: None, + } + } + #[test] - fn accepts_fifo_messages_and_clone_shares_state() { - let mailbox = SteeringMailbox::with_limits(2, 100); + fn accepts_fifo_inputs_and_clone_shares_state() { + let mailbox = TurnMailbox::with_limits(2, 100, 8, 100); let clone = mailbox.clone(); - mailbox.try_push(ChatMessage::user("one")).unwrap(); - clone.try_push(ChatMessage::user("two")).unwrap(); + mailbox.try_push_user(user_input("a", "one")).unwrap(); + clone.try_push_user(user_input("b", "two")).unwrap(); assert_eq!(mailbox.len(), 2); - let messages = mailbox.drain(); - assert_eq!( - messages - .iter() - .map(|m| m.content.as_str()) - .collect::>(), - ["one", "two"] - ); + let inputs = mailbox.drain(); + assert_eq!(inputs.len(), 2); assert!(!mailbox.is_closed()); } #[test] - fn rejects_full_message_without_losing_it() { - let mailbox = SteeringMailbox::with_limits(1, 10_000); - mailbox.try_push(ChatMessage::user("first")).unwrap(); - let second = ChatMessage::user("second"); - let error = mailbox.try_push(second.clone()).unwrap_err(); - assert!(error.is_full()); - assert_eq!(error.into_message().content, second.content); + fn user_and_agent_lanes_have_independent_capacity() { + let mailbox = TurnMailbox::with_limits(1, 100_000, 1, 100_000); + mailbox.try_push_user(user_input("a", "one")).unwrap(); + assert!(matches!( + mailbox.try_push_user(user_input("b", "two")), + Err(SteeringPushError::Full) + )); + mailbox + .try_reserve_steer(agent_signal_input("s1", "evt-1"), "token-1".to_string()) + .unwrap(); + assert!(matches!( + mailbox.try_reserve_steer(agent_signal_input("s2", "evt-2"), "token-2".to_string()), + Err(SteeringPushError::Full) + )); assert_eq!(mailbox.len(), 1); } #[test] - fn byte_limit_is_bounded() { - let mailbox = SteeringMailbox::with_limits(8, 3); - let message = ChatMessage::user("four"); + fn reserved_entries_are_not_drainable_until_activation() { + let mailbox = TurnMailbox::new(); + mailbox + .try_reserve_steer(agent_signal_input("s1", "evt-1"), "token-1".to_string()) + .unwrap(); + assert!(mailbox.drain().is_empty()); + assert_eq!(mailbox.activate_reserved(), 1); + assert_eq!(mailbox.drain().len(), 1); + } + + #[test] + fn cancel_reserved_returns_leases() { + let mailbox = TurnMailbox::new(); + mailbox + .try_reserve_steer(agent_signal_input("s1", "evt-1"), "token-1".to_string()) + .unwrap(); + let leases = mailbox.cancel_reserved(); + assert_eq!(leases, vec![("evt-1".to_string(), "token-1".to_string())]); + assert!(mailbox.drain().is_empty()); + } + + #[test] + fn close_take_releases_admitted_and_reserved_durable_events() { + let mailbox = TurnMailbox::new(); + mailbox + .try_reserve_steer(agent_signal_input("s1", "evt-1"), "token-1".to_string()) + .unwrap(); + mailbox.activate_reserved(); + mailbox + .try_reserve_steer(agent_signal_input("s2", "evt-2"), "token-2".to_string()) + .unwrap(); + let drained = mailbox.drain(); + assert_eq!(drained.len(), 1); + mailbox.try_push_user(user_input("u", "hello")).unwrap(); + + let take = mailbox.close_and_take_pending(); + // evt-1 was drained (admitted), evt-2 still reserved (leased), the + // user input is returned separately. + assert_eq!(take.user_inputs.len(), 1); + assert_eq!( + take.admitted_leases, + vec![("evt-1".to_string(), "token-1".to_string())] + ); + assert_eq!( + take.reserved_leases, + vec![("evt-2".to_string(), "token-2".to_string())] + ); + assert!(mailbox.try_push_user(user_input("late", "x")).is_err()); + } + + #[test] + fn drained_capacity_is_reserved_until_commit_or_restore() { + let mailbox = TurnMailbox::with_limits(1, 100_000, 8, 100_000); + mailbox.try_push_user(user_input("a", "first")).unwrap(); + let drained = mailbox.drain(); + assert_eq!(drained.len(), 1); assert!(matches!( - mailbox.try_push(message), - Err(SteeringPushError::Full(_)) + mailbox.try_push_user(user_input("b", "second")), + Err(SteeringPushError::Full) )); + mailbox.restore_front(drained); + let take = mailbox.take_pending(); + assert_eq!(take.user_inputs[0].content, "first"); + + mailbox.try_push_user(user_input("c", "committed")).unwrap(); + let _ = mailbox.drain(); + mailbox.commit_drained(); + mailbox + .try_push_user(user_input("d", "after commit")) + .unwrap(); + + let drained = mailbox.drain(); + assert_eq!(drained[0].content, "after commit"); + mailbox.restore_drained(); + let take = mailbox.take_pending(); + assert_eq!(take.user_inputs[0].content, "after commit"); } #[test] fn drain_or_close_is_atomic_and_preserves_close_race_semantics() { - let mailbox = Arc::new(SteeringMailbox::new()); + let mailbox = Arc::new(TurnMailbox::new()); let producer = mailbox.clone(); let close_result = thread::spawn(move || producer.drain_or_close()) .join() .unwrap(); assert!(matches!(close_result, SteeringDrain::Closed)); - let message = ChatMessage::user("late"); assert!(matches!( - mailbox.try_push(message), - Err(SteeringPushError::Closed(_)) + mailbox.try_push_user(user_input("late", "x")), + Err(SteeringPushError::Closed) )); } #[test] - fn drain_or_close_drains_but_keeps_accepting_when_non_empty() { - let mailbox = SteeringMailbox::new(); - mailbox.try_push(ChatMessage::user("first")).unwrap(); - let result = mailbox.drain_or_close(); - assert!(matches!(result, SteeringDrain::Messages(_))); - assert!(!mailbox.is_closed()); - mailbox.try_push(ChatMessage::user("second")).unwrap(); - assert_eq!(mailbox.drain()[0].content, "second"); - } - - #[test] - fn close_keeps_pending_for_next_turn_fallback() { - let mailbox = SteeringMailbox::new(); - mailbox.try_push(ChatMessage::user("defer")).unwrap(); - mailbox.close(); - assert!(mailbox.try_push(ChatMessage::user("late")).is_err()); - assert_eq!(mailbox.take_pending()[0].content, "defer"); - assert!(mailbox.is_empty()); - } - - #[test] - fn drained_capacity_is_reserved_until_commit_or_restore() { - let mailbox = SteeringMailbox::with_limits(1, 10_000); - mailbox.try_push(ChatMessage::user("first")).unwrap(); - let drained = mailbox.drain(); - assert_eq!(drained.len(), 1); - assert!(mailbox.try_push(ChatMessage::user("second")).is_err()); - mailbox.restore_front(drained); - assert_eq!(mailbox.take_pending()[0].content, "first"); - - mailbox.try_push(ChatMessage::user("committed")).unwrap(); - let _ = mailbox.drain(); - mailbox.commit_drained(); - mailbox.try_push(ChatMessage::user("after commit")).unwrap(); - - let drained = mailbox.drain(); - assert_eq!(drained[0].content, "after commit"); - mailbox.restore_drained(); - assert_eq!(mailbox.take_pending()[0].content, "after commit"); + fn turn_input_projects_to_hidden_agent_message() { + let input = agent_signal_input("id", "evt-1"); + let message = input.into_chat_message("turn-1".to_string(), 2); + assert_eq!(message.role, "user"); + assert_eq!(message.turn_id.as_deref(), Some("turn-1")); + assert_eq!(message.iteration, Some(2)); + assert_eq!( + message.client_visibility, + crate::bus::ClientVisibility::Hidden + ); + let source = message.source.unwrap(); + assert!(matches!(source.kind, crate::bus::SourceKind::AgentSignal)); + assert_eq!(source.from_run_id.as_deref(), Some("run-1")); + assert_eq!(source.task_id.as_deref(), Some("evt-1")); } } diff --git a/src/agent/sub_agent.rs b/src/agent/sub_agent.rs index ededb50..476f01e 100644 --- a/src/agent/sub_agent.rs +++ b/src/agent/sub_agent.rs @@ -42,7 +42,9 @@ const DEFAULT_READONLY_TOOLS: &[&str] = &[ #[derive(Debug, Clone)] pub struct SubAgentConfig { + pub target: Option, pub prompt: String, + pub context: Option, pub mode: ExecutionMode, pub allowed_tools: Option>, pub max_iterations: Option, @@ -53,16 +55,18 @@ pub struct SubAgentConfig { #[derive(Debug, Clone, PartialEq, Eq)] pub enum ExecutionMode { - Inline, + Foreground, Background, - Parallel, } #[derive(Debug, Clone)] pub struct SubAgentResult { pub task_id: String, + /// Bounded projection returned to the model; may carry a truncation note. pub content: String, pub content_truncated: bool, + /// Untruncated final text, persisted as the durable run result. + pub full_content: String, pub status: TaskStatus, pub tool_calls_count: usize, pub iterations: usize, @@ -127,6 +131,29 @@ pub struct SubAgentManager { work_manager: Option>, task_supervisor: crate::task_supervisor::TaskSupervisor, admission: crate::gateway::reload::RuntimeAdmission, + catalog: Arc, + execution_gate: Arc, + /// Late-bound durable Coordinator. Signals are only available to runs + /// whose definition carries a signal contract AND the runtime has an + /// active Coordinator; resolution happens at delegate time. + coordinator: std::sync::RwLock>>, +} + +#[derive(Clone)] +pub(crate) struct ResolvedAgentRun { + pub provider_config: Arc, + pub tools: Arc, + pub timeout_secs: u64, + pub max_iterations: usize, + pub max_result_chars: usize, + pub role_prompt: Option, + pub skills_prompt: Option, + pub tool_context: ToolExecutionContext, + /// Named-definition metadata used by the durable Coordinator; `None` for + /// the legacy transient general Agent. + pub agent_id: Option, + pub definition_hash: Option, + pub llm_profile: Option, } impl SubAgentManager { @@ -151,9 +178,27 @@ impl SubAgentManager { work_manager: None, task_supervisor, admission: crate::gateway::reload::RuntimeAdmission::open(), + catalog: Arc::new(crate::agent::AgentCatalog::legacy()), + execution_gate: crate::agent::gate::ExecutionGate::unbounded(), + coordinator: std::sync::RwLock::new(None), } } + /// Bind the durable Coordinator so named runs can resolve the + /// contract-bound `emit_signal` tool. Kept as a weak reference: the + /// Coordinator owns this manager, so a strong cycle must never exist. + pub fn bind_coordinator(&self, coordinator: &Arc) { + *self.coordinator.write().unwrap() = Some(Arc::downgrade(coordinator)); + } + + fn coordinator(&self) -> Option> { + self.coordinator + .read() + .unwrap() + .as_ref() + .and_then(std::sync::Weak::upgrade) + } + pub(crate) fn with_admission( mut self, admission: crate::gateway::reload::RuntimeAdmission, @@ -162,6 +207,23 @@ impl SubAgentManager { self } + pub fn with_catalog(mut self, catalog: Arc) -> Self { + self.catalog = catalog; + self + } + + pub fn with_execution_gate( + mut self, + execution_gate: Arc, + ) -> Self { + self.execution_gate = execution_gate; + self + } + + pub fn catalog(&self) -> Arc { + self.catalog.clone() + } + pub fn with_work_manager(mut self, work_manager: Arc) -> Self { self.work_manager = Some(work_manager); self @@ -175,9 +237,7 @@ impl SubAgentManager { let filtered = ToolRegistry::new(); for (name, tool) in self.full_tools.iter() { if allowed_set.contains(name.as_str()) - && name != "delegate" - && name != "todo" - && name != "reload_config" + && tool.delegation_policy() == crate::tools::DelegationPolicy::Delegatable { filtered.register_raw(name, tool); } @@ -196,12 +256,247 @@ impl SubAgentManager { None } + pub(crate) fn resolve_agent( + &self, + config: &SubAgentConfig, + caller: &ToolExecutionContext, + task_id: &str, + ) -> Result { + let Some(target) = config.target.as_deref() else { + if caller.agent.is_some() { + return Err(SubAgentError::Other( + "named child Agents cannot use the legacy general Agent".to_string(), + )); + } + let browser_session_id = config + .session_id + .clone() + .or_else(|| caller.session_id.clone()) + .or_else(|| { + get_delegate_context() + .ok() + .map(|context| context.session_id) + }) + .unwrap_or_else(|| format!("sub-agent:{task_id}")); + let tools = self.filter_tools(&config.allowed_tools); + return Ok(ResolvedAgentRun { + provider_config: Arc::new(self.provider_config.clone()), + skills_prompt: self.get_skills_prompt(&tools), + tools, + timeout_secs: config.timeout_secs.unwrap_or(DEFAULT_TIMEOUT_SECS), + max_iterations: config.max_iterations.unwrap_or(DEFAULT_MAX_ITERATIONS), + max_result_chars: MAX_INLINE_RESULT_CHARS, + role_prompt: None, + tool_context: ToolExecutionContext::for_session(browser_session_id) + .with_cancellation(caller.cancellation.child_token()) + .with_execution_gate(self.execution_gate.clone()), + agent_id: None, + definition_hash: None, + llm_profile: None, + }); + }; + + if !self.catalog.enabled() { + return Err(SubAgentError::Other(format!( + "named Agent '{target}' requested while agent_orchestration is disabled" + ))); + } + let definition = self + .catalog + .get(target) + .ok_or_else(|| SubAgentError::Other(format!("unknown Agent target '{target}'")))?; + let root_session_id = caller + .agent + .as_ref() + .map(|context| context.root_session_id.clone()) + .or_else(|| caller.session_id.clone()) + .ok_or_else(|| { + SubAgentError::Other( + "delegate requires a session-bound ToolExecutionContext".to_string(), + ) + })?; + let cancellation = caller.cancellation.child_token(); + let execution = if let Some(parent) = caller.agent.as_ref() { + if !self.catalog.can_delegate(&parent.current_agent_id, target) { + return Err(SubAgentError::Other(format!( + "Agent '{}' is not allowed to delegate to '{target}'", + parent.current_agent_id + ))); + } + if parent.ancestry.iter().any(|agent| agent == target) { + return Err(SubAgentError::Other(format!( + "delegation cycle rejected: '{target}' is already in the current ancestry" + ))); + } + if parent.budget.remaining_runs == 0 || parent.budget.remaining_depth == 0 { + return Err(SubAgentError::Other( + "delegation budget exhausted".to_string(), + )); + } + let next_depth = parent.depth.saturating_add(1); + if next_depth > self.catalog.max_tree_depth() { + return Err(SubAgentError::Other(format!( + "delegation depth {next_depth} exceeds global limit {}", + self.catalog.max_tree_depth() + ))); + } + if parent + .reserve_tree_run(self.catalog.max_runs_per_tree()) + .is_none() + { + return Err(SubAgentError::Other(format!( + "delegation tree already uses {} runs; max_runs_per_tree is {}", + self.catalog.max_runs_per_tree(), + self.catalog.max_runs_per_tree() + ))); + } + let mut child = crate::agent::AgentExecutionContext::child( + parent, + task_id.to_string(), + target.to_string(), + config.plan_item_id.clone(), + cancellation.clone(), + ); + child.budget.remaining_depth = child + .budget + .remaining_depth + .min(definition.limits.max_depth); + child.signal_contract = definition + .signal_contract + .as_ref() + .map(|contract| Arc::new(contract.clone())); + Arc::new(child) + } else { + if !self.catalog.root_can_delegate(target) { + return Err(SubAgentError::Other(format!( + "ROOT is not allowed to delegate to '{target}'" + ))); + } + Arc::new(crate::agent::AgentExecutionContext { + root_session_id: root_session_id.clone(), + root_turn_id: caller.turn_id.clone(), + run_id: task_id.to_string(), + execution_id: task_id.to_string(), + group_id: None, + parent_run_id: None, + caller_agent_id: "ROOT".to_string(), + current_agent_id: target.to_string(), + ancestry: vec![target.to_string()], + depth: 1, + plan_item_id: config.plan_item_id.clone(), + cancellation: cancellation.clone(), + budget: crate::agent::AgentBudget { + remaining_runs: self.catalog.max_runs_per_tree().saturating_sub(1), + remaining_depth: self + .catalog + .max_tree_depth() + .saturating_sub(1) + .min(definition.limits.max_depth), + }, + tree_runs: Arc::new(std::sync::atomic::AtomicUsize::new(1)), + signal_contract: definition + .signal_contract + .as_ref() + .map(|contract| Arc::new(contract.clone())), + emitted_signals: Arc::new(std::sync::Mutex::new(Vec::new())), + }) + }; + + let mut effective_names = definition.tools.clone(); + if let Some(allowed) = config.allowed_tools.as_ref() { + effective_names.retain(|name| allowed.iter().any(|allowed| allowed == name)); + } + let has_get_skill = effective_names.iter().any(|name| name == "get_skill"); + let mut names = effective_names; + names.retain(|name| name != "get_skill"); + let mut runtime_tools = Vec::new(); + let skills_prompt = if has_get_skill { + let loader = self + .skills_loader + .as_ref() + .ok_or_else(|| SubAgentError::Other("skills loader is unavailable".to_string()))?; + runtime_tools.push(Arc::new(crate::tools::GetSkillTool::scoped( + loader.clone(), + &definition.skills, + )) as Arc); + let prompt = loader.build_scoped_skills_prompt(&definition.skills); + (!prompt.is_empty()).then_some(prompt) + } else { + None + }; + if !definition.delegates.is_empty() { + let delegate = self.full_tools.get("delegate").ok_or_else(|| { + SubAgentError::Other("delegate runtime tool is unavailable".to_string()) + })?; + runtime_tools.push(Arc::new(crate::tools::delegate::ScopedDelegateTool::new( + delegate, + definition.delegates.clone(), + )) as Arc); + } + // The signal tool is contract-bound: it exists only when the + // definition declares a signal block and the durable Coordinator is + // live. If either is missing the run cannot emit signals. + if definition.signal_contract.is_some() { + match self.coordinator() { + Some(coordinator) => { + let contract = definition.signal_contract.clone().unwrap(); + runtime_tools.push(Arc::new(crate::tools::EmitSignalTool::new( + coordinator, + Arc::new(contract), + )) as Arc); + } + None => { + return Err(SubAgentError::Other( + "Agent '{}' declares a signal contract but the durable Coordinator is unavailable" + .replace("{}", target), + )); + } + } + } + let tools = self + .full_tools + .scoped_for_agent(&names, runtime_tools) + .map_err(SubAgentError::Other)?; + + Ok(ResolvedAgentRun { + provider_config: definition.provider_config.clone(), + tools, + timeout_secs: definition.limits.timeout_secs, + max_iterations: definition.limits.max_iterations, + max_result_chars: definition.limits.max_result_chars, + role_prompt: Some(definition.role_prompt.clone()), + skills_prompt, + agent_id: Some(target.to_string()), + definition_hash: Some(definition.definition_hash.clone()), + llm_profile: Some(definition.llm_profile.clone()), + tool_context: ToolExecutionContext::for_session(format!("agent-run:{task_id}")) + .with_turn_id( + caller + .turn_id + .clone() + .unwrap_or_else(|| task_id.to_string()), + ) + .with_agent(execution) + .with_cancellation(cancellation) + .with_execution_gate(self.execution_gate.clone()), + }) + } + pub fn build_sub_agent( &self, config: &SubAgentConfig, tools: Arc, ) -> Result { - let mut provider = create_provider(self.provider_config.clone()) + self.build_sub_agent_with_provider(config, tools, &self.provider_config) + } + + fn build_sub_agent_with_provider( + &self, + config: &SubAgentConfig, + tools: Arc, + provider_config: &LLMProviderConfig, + ) -> Result { + let mut provider = create_provider(provider_config.clone()) .map_err(|e| AgentError::ProviderCreation(e.to_string()))?; if let Some(ref s) = self.storage { provider.set_storage(s.clone()); @@ -209,9 +504,9 @@ impl SubAgentManager { let provider: Arc = Arc::from(provider); let max_iterations = config.max_iterations.unwrap_or(DEFAULT_MAX_ITERATIONS); - let workspace_dir = self.provider_config.workspace_dir.clone(); - let model_name = self.provider_config.model_id.clone(); - let input_types = self.provider_config.input_types.clone(); + let workspace_dir = provider_config.workspace_dir.clone(); + let model_name = provider_config.model_id.clone(); + let input_types = provider_config.input_types.clone(); let agent = AgentLoop::with_provider_and_tools( provider, @@ -221,7 +516,7 @@ impl SubAgentManager { workspace_dir, input_types, ) - .with_context_window(self.provider_config.token_limit); + .with_context_window(provider_config.token_limit); Ok(agent) } @@ -229,31 +524,67 @@ impl SubAgentManager { pub async fn run_inline( &self, config: SubAgentConfig, + ) -> Result { + let mut caller = ToolExecutionContext::default(); + if let Some(session_id) = config.session_id.clone() { + caller.session_id = Some(session_id); + } + self.run_foreground(config, &caller).await + } + + pub async fn run_foreground( + &self, + config: SubAgentConfig, + caller: &ToolExecutionContext, ) -> Result { let task_id = generate_task_id(); - let tools = self.filter_tools(&config.allowed_tools); - let timeout_secs = config.timeout_secs.unwrap_or(DEFAULT_TIMEOUT_SECS); + let resolved = self.resolve_agent(&config, caller, &task_id)?; + self.assign_work_item(&config, &task_id).await?; + let result = self.execute_resolved(&config, resolved, &task_id).await?; + self.finish_work_item(&config, &result).await; + Ok(result) + } + + /// Execute an already-resolved Agent without any plan-item side effects. + /// The durable Coordinator owns plan admission/terminal updates itself; + /// the legacy path wraps this with assign/finish hooks. + pub(crate) async fn execute_resolved( + &self, + config: &SubAgentConfig, + resolved: ResolvedAgentRun, + task_id: &str, + ) -> Result { + let tools = resolved.tools; + let timeout_secs = resolved.timeout_secs; let timeout_human = format_duration(timeout_secs); - let http_get_only = config.allowed_tools.is_none() - || config - .allowed_tools - .as_ref() - .is_some_and(|v| v.iter().any(|t| t == "http_request")); - let skills_prompt = self.get_skills_prompt(&tools); - let system_prompt = build_sub_agent_system_prompt( + let mut system_prompt = build_sub_agent_system_prompt( &config.prompt, &timeout_human, &tools, - &self.provider_config.workspace_dir, - &self.provider_config.model_id, - skills_prompt, - http_get_only, + &resolved.provider_config.workspace_dir, + &resolved.provider_config.model_id, + resolved.skills_prompt, + false, ); + if let Some(role_prompt) = resolved.role_prompt { + system_prompt.push_str("\n\n## Agent Definition\n\n"); + system_prompt.push_str(&role_prompt); + } + if let Some(context) = config + .context + .as_deref() + .filter(|value| !value.trim().is_empty()) + { + system_prompt.push_str("\n\n## 调用方提供的任务上下文\n\n"); + system_prompt.push_str(context); + } + let mut effective_config = config.clone(); + effective_config.max_iterations = Some(resolved.max_iterations); + let max_result_chars = resolved.max_result_chars; let agent = self - .build_sub_agent(&config, tools) + .build_sub_agent_with_provider(&effective_config, tools, &resolved.provider_config) .map_err(|e| SubAgentError::ProviderCreation(e.to_string()))?; - self.assign_work_item(&config, &task_id).await?; let history = vec![ ChatMessage::system(system_prompt), @@ -261,31 +592,35 @@ impl SubAgentManager { ]; let start = Instant::now(); - let browser_session_id = config - .session_id - .clone() - .or_else(|| { - get_delegate_context() - .ok() - .map(|context| context.session_id) - }) - .unwrap_or_else(|| format!("sub-agent:{task_id}")); + let tool_context = resolved.tool_context; - let result = tokio::time::timeout( - std::time::Duration::from_secs(timeout_secs), - agent.process_with_context( - history, - ToolExecutionContext::for_session(browser_session_id), - ), - ) - .await; + let result = tokio::select! { + result = tokio::time::timeout( + std::time::Duration::from_secs(timeout_secs), + agent.process_with_context(history, tool_context.clone()), + ) => result, + _ = tool_context.cancellation.cancelled() => { + return Ok(SubAgentResult { + task_id: task_id.to_string(), + content: String::new(), + content_truncated: false, + full_content: String::new(), + status: TaskStatus::Cancelled, + tool_calls_count: 0, + iterations: 0, + duration_ms: start.elapsed().as_millis() as u64, + }); + } + }; let duration_ms = start.elapsed().as_millis() as u64; - let result = match result { + Ok(match result { Ok(Ok(agent_result)) => { - let (content, truncated) = - truncate_sub_agent_result(&agent_result.final_response.content); + let (content, truncated) = truncate_sub_agent_result_at( + &agent_result.final_response.content, + max_result_chars, + ); let tool_calls_count = agent_result .emitted_messages .iter() @@ -297,52 +632,111 @@ impl SubAgentManager { .filter(|m| m.role == "assistant" && m.tool_calls.is_some()) .count(); SubAgentResult { - task_id: task_id.clone(), + task_id: task_id.to_string(), content, content_truncated: truncated, + full_content: agent_result.final_response.content, status: TaskStatus::Completed, tool_calls_count, iterations, duration_ms, } } - Ok(Err(e)) => SubAgentResult { - task_id: task_id.clone(), + Ok(Err(error)) => SubAgentResult { + task_id: task_id.to_string(), content: String::new(), content_truncated: false, - status: TaskStatus::Failed(e.to_string()), + full_content: String::new(), + status: terminal_status_from_error(error), tool_calls_count: 0, iterations: 0, duration_ms, }, Err(_elapsed) => SubAgentResult { - task_id: task_id.clone(), + task_id: task_id.to_string(), content: String::new(), content_truncated: false, + full_content: String::new(), status: TaskStatus::TimedOut, tool_calls_count: 0, iterations: 0, duration_ms, }, - }; - self.finish_work_item(&config, &result).await; - Ok(result) + }) } pub async fn run_parallel( &self, configs: Vec, ) -> Result, SubAgentError> { + self.run_foreground_batch(configs, &ToolExecutionContext::default()) + .await + } + + pub async fn run_foreground_batch( + &self, + configs: Vec, + caller: &ToolExecutionContext, + ) -> Result, SubAgentError> { + if configs.is_empty() { + return Err(SubAgentError::Other( + "foreground batch must contain at least one task".to_string(), + )); + } + if let Some(parent) = caller.agent.as_ref() { + let definition = self.catalog.get(&parent.current_agent_id).ok_or_else(|| { + SubAgentError::Other(format!( + "caller Agent '{}' is not present in the active catalog", + parent.current_agent_id + )) + })?; + if configs.len() > definition.limits.max_children { + return Err(SubAgentError::Other(format!( + "Agent '{}' may create at most {} children per delegate call", + parent.current_agent_id, definition.limits.max_children + ))); + } + if configs.len() > parent.budget.remaining_runs { + return Err(SubAgentError::Other( + "delegation run budget is smaller than the requested batch".to_string(), + )); + } + if configs.len() > parent.remaining_tree_runs(self.catalog.max_runs_per_tree()) { + return Err(SubAgentError::Other( + "delegation tree capacity is smaller than the requested batch".to_string(), + )); + } + } else if self.catalog.enabled() && configs.len() > self.catalog.max_runs_per_tree() { + return Err(SubAgentError::Other(format!( + "ROOT batch exceeds max_runs_per_tree ({})", + self.catalog.max_runs_per_tree() + ))); + } let futures: Vec<_> = configs .into_iter() .map(|config| { - let mgr = self; // &self borrow, all tasks share the same manager - async move { mgr.run_inline(config).await } + let caller = caller.clone(); + async move { self.run_foreground(config, &caller).await } }) .collect(); let results = futures_util::future::join_all(futures).await; - results.into_iter().collect::, _>>() + Ok(results + .into_iter() + .enumerate() + .map(|(index, result)| { + result.unwrap_or_else(|error| SubAgentResult { + task_id: format!("rejected-{}-{}", index + 1, generate_task_id()), + content: String::new(), + content_truncated: false, + full_content: String::new(), + status: TaskStatus::Failed(error.to_string()), + tool_calls_count: 0, + iterations: 0, + duration_ms: 0, + }) + }) + .collect()) } pub async fn run_background( @@ -442,6 +836,7 @@ impl SubAgentManager { let notify_tx = self.notify_tx.clone(); let active_tasks = Arc::clone(&self.active_tasks); let shutdown = self.task_supervisor.cancellation_token(); + let execution_gate = self.execution_gate.clone(); let tid = task_id.clone(); let sess_id = ctx.session_id.clone(); @@ -499,13 +894,13 @@ impl SubAgentManager { ChatMessage::user(&prompt), ]; - tokio::select! { + let tool_context = ToolExecutionContext::for_session(&sess_id) + .with_cancellation(cancel_token.clone()) + .with_execution_gate(execution_gate.clone()); + tokio::select! { r = tokio::time::timeout( std::time::Duration::from_secs(timeout_secs), - agent.process_with_context( - history, - ToolExecutionContext::for_session(&sess_id), - ), + agent.process_with_context(history, tool_context), ) => { match r { Ok(Ok(agent_result)) => { @@ -515,19 +910,21 @@ impl SubAgentManager { .iter().filter(|m| m.role == "assistant" && m.tool_calls.is_some()).count(); SubAgentResult { task_id: tid.clone(), - content: agent_result.final_response.content, + content: agent_result.final_response.content.clone(), content_truncated: false, + full_content: agent_result.final_response.content, status: TaskStatus::Completed, tool_calls_count, iterations, duration_ms: 0, } }, - Ok(Err(e)) => SubAgentResult { + Ok(Err(error)) => SubAgentResult { task_id: tid.clone(), content: String::new(), content_truncated: false, - status: TaskStatus::Failed(e.to_string()), + full_content: String::new(), + status: terminal_status_from_error(error), tool_calls_count: 0, iterations: 0, duration_ms: 0, @@ -536,6 +933,7 @@ impl SubAgentManager { task_id: tid.clone(), content: String::new(), content_truncated: false, + full_content: String::new(), status: TaskStatus::TimedOut, tool_calls_count: 0, iterations: 0, @@ -547,6 +945,7 @@ impl SubAgentManager { task_id: tid.clone(), content: String::new(), content_truncated: false, + full_content: String::new(), status: TaskStatus::Cancelled, tool_calls_count: 0, iterations: 0, @@ -556,6 +955,7 @@ impl SubAgentManager { task_id: tid.clone(), content: String::new(), content_truncated: false, + full_content: String::new(), status: TaskStatus::Cancelled, tool_calls_count: 0, iterations: 0, @@ -567,6 +967,7 @@ impl SubAgentManager { task_id: tid.clone(), content: String::new(), content_truncated: false, + full_content: String::new(), status: TaskStatus::Failed("provider creation failed".into()), tool_calls_count: 0, iterations: 0, @@ -785,8 +1186,16 @@ impl SubAgentManager { } } +fn terminal_status_from_error(error: AgentError) -> TaskStatus { + match error { + AgentError::Cancelled => TaskStatus::Cancelled, + AgentError::TimedOut => TaskStatus::TimedOut, + other => TaskStatus::Failed(other.to_string()), + } +} + fn generate_task_id() -> String { - Uuid::new_v4().to_string()[..8].to_string() + Uuid::new_v4().to_string() } fn format_duration(seconds: u64) -> String { @@ -799,11 +1208,11 @@ fn format_duration(seconds: u64) -> String { } } -fn truncate_sub_agent_result(content: &str) -> (String, bool) { - if content.len() <= MAX_INLINE_RESULT_CHARS { +fn truncate_sub_agent_result_at(content: &str, max_chars: usize) -> (String, bool) { + if content.len() <= max_chars { (content.to_string(), false) } else { - let truncate_at = content.floor_char_boundary(MAX_INLINE_RESULT_CHARS); + let truncate_at = content.floor_char_boundary(max_chars); ( format!( "{}\n\n[... 结果已截断，共 {} 字符，完整结果请使用 check_task 查看 ...]", @@ -870,7 +1279,9 @@ mod tests { let error = manager .run_background( SubAgentConfig { + target: None, prompt: "test".into(), + context: None, mode: ExecutionMode::Background, allowed_tools: None, max_iterations: None, @@ -902,4 +1313,149 @@ mod tests { let filtered = manager.filter_tools(&Some(vec!["reload_config".to_string()])); assert!(filtered.get("reload_config").is_none()); } + + fn catalog_with_agents( + root: &std::path::Path, + max_runs_per_tree: usize, + ) -> crate::agent::AgentCatalog { + std::fs::create_dir_all(root.join("agents")).unwrap(); + let write = |id: &str, delegates: &[&str]| { + let delegates = (!delegates.is_empty()).then(|| { + format!( + "delegates:\n{}\n", + delegates + .iter() + .map(|name| format!(" - {name}")) + .collect::>() + .join("\n") + ) + }); + std::fs::write( + root.join("agents").join(format!("{id}.md")), + format!( + "---\nid: {id}\ndescription: {id} role\nllm_profile: research\n{}---\n# Role\n\nDo the assigned work.\n", + delegates.unwrap_or_default() + ), + ) + .unwrap(); + }; + write("researcher", &["reviewer"]); + write("reviewer", &[]); + let tools = ToolRegistry::new(); + let loader = crate::skills::SkillsLoader::new_for_testing( + root.join("skills"), + root.join("external-skills"), + ); + let profiles = HashMap::from([( + "research".to_string(), + LLMProviderConfig { + provider_type: "openai".into(), + name: "test".into(), + base_url: "http://localhost".into(), + api_key: "test".into(), + extra_headers: HashMap::new(), + model_id: "test".into(), + temperature: None, + max_tokens: None, + model_extra: HashMap::new(), + max_tool_iterations: 1, + token_limit: 4096, + workspace_dir: std::env::temp_dir(), + input_types: vec!["text".into()], + price_input_per_million: None, + price_output_per_million: None, + }, + )]); + let config = crate::config::AgentOrchestrationConfig { + enabled: true, + definitions_dir: "agents".to_string(), + root_delegates: vec!["researcher".to_string()], + max_runs_per_tree, + ..Default::default() + }; + crate::agent::AgentCatalog::load(&config, root, &profiles, &tools, &loader, 1).unwrap() + } + + #[tokio::test] + async fn foreground_batch_rejects_when_tree_capacity_exhausted() { + let root = tempfile::tempdir().unwrap(); + let catalog = catalog_with_agents(root.path(), 2); + let manager = manager(1).with_catalog(Arc::new(catalog)); + + let caller_context = Arc::new(crate::agent::AgentExecutionContext { + root_session_id: "cli:test:dialog".to_string(), + root_turn_id: None, + run_id: "run-root".to_string(), + execution_id: "run-root".to_string(), + group_id: None, + parent_run_id: None, + caller_agent_id: "ROOT".to_string(), + current_agent_id: "researcher".to_string(), + ancestry: vec!["researcher".to_string()], + depth: 1, + plan_item_id: None, + cancellation: CancellationToken::new(), + budget: crate::agent::AgentBudget { + remaining_runs: 15, + remaining_depth: 3, + }, + tree_runs: Arc::new(std::sync::atomic::AtomicUsize::new(2)), + signal_contract: None, + emitted_signals: Arc::new(std::sync::Mutex::new(Vec::new())), + }); + let caller = + ToolExecutionContext::for_session("cli:test:dialog").with_agent(caller_context); + + let error = manager + .run_foreground_batch( + vec![SubAgentConfig { + target: Some("reviewer".to_string()), + prompt: "test".to_string(), + context: None, + mode: ExecutionMode::Foreground, + allowed_tools: None, + max_iterations: None, + timeout_secs: None, + plan_item_id: None, + session_id: Some("cli:test:dialog".to_string()), + }], + &caller, + ) + .await + .unwrap_err(); + assert!( + matches!(error, SubAgentError::Other(message) if message.contains("tree capacity")) + ); + } + + #[tokio::test] + async fn foreground_batch_preserves_per_task_rejections() { + let manager = manager(1); + let config = |target: &str| SubAgentConfig { + target: Some(target.to_string()), + prompt: "test".to_string(), + context: None, + mode: ExecutionMode::Foreground, + allowed_tools: None, + max_iterations: None, + timeout_secs: None, + plan_item_id: None, + session_id: Some("cli:test:dialog".to_string()), + }; + + let results = manager + .run_foreground_batch( + vec![config("missing-a"), config("missing-b")], + &ToolExecutionContext::for_session("cli:test:dialog"), + ) + .await + .unwrap(); + + assert_eq!(results.len(), 2); + assert!( + results + .iter() + .all(|result| matches!(result.status, TaskStatus::Failed(_))) + ); + } } diff --git a/src/agent/system_prompt.rs b/src/agent/system_prompt.rs index 8231240..ac0b253 100644 --- a/src/agent/system_prompt.rs +++ b/src/agent/system_prompt.rs @@ -380,7 +380,7 @@ impl PromptSection for SubAgentIdentitySection { ## 规则\n\ - 只专注于这个任务，不要扩展到无关话题\n\ - 只在必要时使用工具\n\ - - 不要使用 delegate 工具\n\ + - 只有运行时明确提供 delegate 工具时才可继续委托，并遵守已配置的目标白名单\n\ - 无法完成时，直接说明原因\n\ - 只返回最终结果，不要描述过程\n\ - 超时：{}，接近时限时返回部分结果", diff --git a/src/agent/turn_event.rs b/src/agent/turn_event.rs index 5ee8095..e3cf557 100644 --- a/src/agent/turn_event.rs +++ b/src/agent/turn_event.rs @@ -2,7 +2,7 @@ use std::sync::{Arc, Mutex}; use thiserror::Error; -use crate::agent::steering::SteeringMailbox; +use crate::agent::steering::TurnMailbox; use crate::providers::ToolCall; /// Presentation facts emitted while AgentLoop processes one model turn. @@ -61,7 +61,7 @@ pub struct AgentTurnContext { pub emitter: TurnEmitter, /// Same-turn user input accepted while this turn is active. Session owns /// the mailbox lifecycle; AgentLoop only drains it at safe boundaries. - pub steering: Option>, + pub steering: Option>, } impl AgentTurnContext { @@ -83,7 +83,7 @@ impl AgentTurnContext { turn_id: impl Into, message_id: impl Into, emitter: TurnEmitter, - steering: Arc, + steering: Arc, ) -> Self { Self { turn_id: turn_id.into(), @@ -95,13 +95,13 @@ impl AgentTurnContext { /// Attach a mailbox to an existing context. This builder keeps the old /// `AgentTurnContext::new` call sites source-compatible. - pub fn with_steering(mut self, steering: Arc) -> Self { + pub fn with_steering(mut self, steering: Arc) -> Self { self.steering = Some(steering); self } /// Return a clone of the shared mailbox, if steering is enabled. - pub fn steering(&self) -> Option> { + pub fn steering(&self) -> Option> { self.steering.clone() } } diff --git a/src/bus/message.rs b/src/bus/message.rs index 9975e50..218fc34 100644 --- a/src/bus/message.rs +++ b/src/bus/message.rs @@ -129,6 +129,48 @@ impl MediaItem { // ChatMessage - Used by AgentLoop for LLM conversation history // ============================================================================ +/// Whether a message may be surfaced to clients. Hidden messages exist only +/// for model replay (internal triggers) and must never appear in history, +/// projections or delivery. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)] +#[serde(rename_all = "snake_case")] +pub enum ClientVisibility { + #[default] + Visible, + Hidden, +} + +impl ClientVisibility { + pub fn as_str(&self) -> &'static str { + match self { + Self::Visible => "visible", + Self::Hidden => "hidden", + } + } +} + +/// Where a message Turn originated. Persisted alongside the message so +/// clients can render agent-driven continuation output without treating it as +/// a user bubble. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)] +#[serde(rename_all = "snake_case")] +pub enum TurnOrigin { + #[default] + User, + AgentContinuation, + Scheduled, +} + +impl TurnOrigin { + pub fn as_str(&self) -> &'static str { + match self { + Self::User => "user", + Self::AgentContinuation => "agent_continuation", + Self::Scheduled => "scheduled", + } + } +} + #[derive(Debug, Clone, Serialize, Deserialize)] pub struct ChatMessage { pub id: String, @@ -144,6 +186,10 @@ pub struct ChatMessage { pub iteration: Option, #[serde(default)] pub completion_status: CompletionStatus, + #[serde(default)] + pub client_visibility: ClientVisibility, + #[serde(default)] + pub turn_origin: TurnOrigin, pub media_refs: Vec, pub timestamp: i64, #[serde(skip_serializing_if = "Option::is_none")] @@ -166,6 +212,15 @@ pub enum SourceKind { CrossChannel, #[serde(rename = "external_trigger")] ExternalTrigger, + /// A durable signal emitted by a background Agent via `emit_signal`. + #[serde(rename = "agent_signal")] + AgentSignal, + /// A durable background run completion outcome. + #[serde(rename = "agent_result")] + AgentCompletion, + /// A durable background group completion outcome. + #[serde(rename = "agent_group_result")] + AgentGroupCompletion, } #[derive(Debug, Clone, Serialize, Deserialize)] @@ -176,6 +231,15 @@ pub struct MessageSource { pub from_user_id: Option, pub system_name: Option, pub task_id: Option, + /// Durable Agent run identity for `agent_signal`/`agent_result` sources. + #[serde(default)] + pub from_run_id: Option, + /// Agent definition id for `agent_signal`/`agent_result` sources. + #[serde(default)] + pub from_agent_id: Option, + /// Durable group identity for `agent_group_result` sources. + #[serde(default)] + pub group_id: Option, } impl ChatMessage { @@ -195,6 +259,8 @@ impl ChatMessage { tool_name: None, tool_calls: None, source: None, + client_visibility: ClientVisibility::Visible, + turn_origin: TurnOrigin::User, } } @@ -214,6 +280,8 @@ impl ChatMessage { tool_name: None, tool_calls: None, source: None, + client_visibility: ClientVisibility::Visible, + turn_origin: TurnOrigin::User, } } @@ -233,6 +301,8 @@ impl ChatMessage { tool_name: None, tool_calls: None, source: None, + client_visibility: ClientVisibility::Visible, + turn_origin: TurnOrigin::User, } } @@ -255,6 +325,8 @@ impl ChatMessage { tool_name: None, tool_calls: Some(tool_calls), source: None, + client_visibility: ClientVisibility::Visible, + turn_origin: TurnOrigin::User, } } @@ -274,6 +346,8 @@ impl ChatMessage { tool_name: None, tool_calls: None, source: Some(source), + client_visibility: ClientVisibility::Visible, + turn_origin: TurnOrigin::User, } } @@ -293,6 +367,8 @@ impl ChatMessage { tool_name: None, tool_calls: None, source: None, + client_visibility: ClientVisibility::Visible, + turn_origin: TurnOrigin::User, } } @@ -325,6 +401,8 @@ impl ChatMessage { tool_name: Some(tool_name.into()), tool_calls: None, source: None, + client_visibility: ClientVisibility::Visible, + turn_origin: TurnOrigin::User, } } @@ -344,6 +422,8 @@ impl ChatMessage { tool_name: None, tool_calls: None, source: Some(source), + client_visibility: ClientVisibility::Visible, + turn_origin: TurnOrigin::User, } } } @@ -371,11 +451,15 @@ mod conversation_message_tests { // ============================================================================ /// Opaque channel-owned context that may be carried to the corresponding reply. -/// Core routing understands `reply_to`; all other platform data remains private. +/// Core routing understands `reply_to`; all other platform data remains +/// private. `durable_private` holds only values the channel declares safe to +/// reuse across turns (thread/root identity); one-shot message/reaction ids +/// belong in `private`. #[derive(Debug, Clone, Default)] pub struct ChannelContext { pub reply_to: Option, pub private: HashMap, + pub durable_private: HashMap, } /// Public, durable projection of a newly committed conversation message. @@ -393,6 +477,7 @@ pub struct CommittedMessage { pub tool_call_id: Option, pub tool_name: Option, pub tool_calls: Option>, + pub turn_origin: TurnOrigin, } #[derive(Debug, Clone)] diff --git a/src/bus/mod.rs b/src/bus/mod.rs index 9259288..6b00771 100644 --- a/src/bus/mod.rs +++ b/src/bus/mod.rs @@ -3,9 +3,9 @@ pub mod message; pub use dispatcher::OutboundDispatcher; pub use message::{ - ChannelContext, ChatMessage, CommittedMessage, CommittedTurnDelta, CompletionStatus, - ContentBlock, ControlMessage, InboundMessage, MediaItem, MediaRef, MessageSource, - OutboundMessage, ProviderReasoningState, SourceKind, + ChannelContext, ChatMessage, ClientVisibility, CommittedMessage, CommittedTurnDelta, + CompletionStatus, ContentBlock, ControlMessage, InboundMessage, MediaItem, MediaRef, + MessageSource, OutboundMessage, ProviderReasoningState, SourceKind, TurnOrigin, }; use std::sync::Arc; diff --git a/src/channels/cli_chat.rs b/src/channels/cli_chat.rs index 220be4f..811643f 100644 --- a/src/channels/cli_chat.rs +++ b/src/channels/cli_chat.rs @@ -132,6 +132,40 @@ impl CliChatChannel { } } + /// Push a durable Agent run/event projection update to the owning + /// WebSocket client. Clients recalibrate with `GetAgentRuns` when a + /// broadcast is lagged or lost. + pub async fn publish_agent_projection( + &self, + projection: crate::agent::projection::AgentProjection, + ) { + let Some(session_id) = UnifiedSessionId::parse(&projection.session_id) else { + return; + }; + if session_id.channel != "cli_chat" { + return; + } + let client = self.clients.lock().await.get(&session_id.chat_id).cloned(); + if let Some(client) = client { + let message = if let Some(run) = projection.run { + WsOutbound::AgentRunUpdated { + session_id: projection.session_id, + revision: projection.revision, + run, + } + } else if let Some(event) = projection.event { + WsOutbound::AgentEventUpdated { + session_id: projection.session_id, + revision: projection.revision, + event, + } + } else { + return; + }; + let _ = client.sender.send(message).await; + } + } + /// Handle an inbound message from a client pub(crate) async fn handle_inbound(&self, client: Arc, raw_msg: &str) { match parse_inbound(raw_msg) { @@ -515,6 +549,77 @@ impl CliChatChannel { None => return Err(ChannelError::Other("Control channel closed".to_string())), } } + WsInbound::GetAgentRuns { + session_id, + cursor, + limit, + } => { + let unified_id = Self::parse_client_session(&client, &session_id)?; + let (reply_tx, mut reply_rx) = mpsc::channel(1); + bus.publish_control(ControlMessage { + op: SessionCommand::GetAgentRuns { + session_id: unified_id, + cursor, + limit: limit.unwrap_or(100).clamp(1, 200), + }, + reply_tx, + }) + .await?; + match reply_rx.recv().await { + Some(Ok(SessionEvent::AgentRuns { + session_id, + revision, + runs, + next_cursor, + })) => { + let _ = client + .sender + .send(WsOutbound::SessionAgentRuns { + session_id: session_id.to_string(), + revision, + runs, + next_cursor, + }) + .await; + } + Some(Ok(_)) => {} + Some(Err(error)) => return Err(error), + None => return Err(ChannelError::Other("Control channel closed".to_string())), + } + } + WsInbound::GetAgentRun { session_id, run_id } => { + let unified_id = Self::parse_client_session(&client, &session_id)?; + let (reply_tx, mut reply_rx) = mpsc::channel(1); + bus.publish_control(ControlMessage { + op: SessionCommand::GetAgentRun { + session_id: unified_id, + run_id, + }, + reply_tx, + }) + .await?; + match reply_rx.recv().await { + Some(Ok(SessionEvent::AgentRun { + session_id, + revision, + run, + })) => { + let _ = client + .sender + .send(WsOutbound::AgentRunUpdated { + session_id: session_id.to_string(), + revision, + run: run.ok_or_else(|| { + ChannelError::Other("run not found".to_string()) + })?, + }) + .await; + } + Some(Ok(_)) => {} + Some(Err(error)) => return Err(error), + None => return Err(ChannelError::Other("Control channel closed".to_string())), + } + } WsInbound::RenameSession { session_id, title } => { let target = session_id .or(current_session_guard.clone()) @@ -1255,6 +1360,7 @@ mod tests { tool_call_id: None, tool_name: None, tool_calls: None, + turn_origin: crate::bus::TurnOrigin::User, }], }, ) diff --git a/src/channels/feishu.rs b/src/channels/feishu.rs index f1ed04a..162f7f2 100644 --- a/src/channels/feishu.rs +++ b/src/channels/feishu.rs @@ -1415,6 +1415,16 @@ impl FeishuChannel { private_context.insert("feishu.reaction_id".to_string(), reaction_id); } + // Durable context only carries values safe to reuse across Turns: + // thread/root identity. Message and reaction ids are one-shot and + // stay in `private`/`reply_to`. + let mut durable_context = HashMap::new(); + for key in ["feishu.chat_type", "feishu.thread_id", "feishu.root_id"] { + if let Some(value) = private_context.get(key) { + durable_context.insert(key.to_string(), value.clone()); + } + } + let msg = crate::bus::InboundMessage { channel: "feishu".to_string(), sender_id: parsed.open_id.clone(), @@ -1426,6 +1436,7 @@ impl FeishuChannel { channel_context: crate::bus::ChannelContext { reply_to: Some(message_id), private: private_context, + durable_private: durable_context, }, }; if let Err(error) = self.handle_and_publish(bus, &msg).await { diff --git a/src/client/mod.rs b/src/client/mod.rs index 14846b4..a59675f 100644 --- a/src/client/mod.rs +++ b/src/client/mod.rs @@ -382,7 +382,10 @@ async fn handle_ws_message(app: &mut App, outbound: WsOutbound) { // notifications and may inspect plans through /todo. WsOutbound::SessionPlan { .. } | WsOutbound::SessionStats { .. } - | WsOutbound::PlanUpdated { .. } => {} + | WsOutbound::PlanUpdated { .. } + | WsOutbound::SessionAgentRuns { .. } + | WsOutbound::AgentRunUpdated { .. } + | WsOutbound::AgentEventUpdated { .. } => {} WsOutbound::SessionRenamed { session_id, title } => { if let Some(session) = app .sessions diff --git a/src/client/tui/app.rs b/src/client/tui/app.rs index 26c0c28..1fd1b88 100644 --- a/src/client/tui/app.rs +++ b/src/client/tui/app.rs @@ -552,6 +552,7 @@ mod tests { tool_name: None, tool_calls: None, attachments: Vec::new(), + turn_origin: crate::bus::TurnOrigin::User, }], ); assert!(app.active_turn.is_none()); @@ -578,6 +579,7 @@ mod tests { tool_name: None, tool_calls: None, attachments: Vec::new(), + turn_origin: crate::bus::TurnOrigin::User, }], )); assert!(app.active_turn.is_none()); @@ -604,6 +606,7 @@ mod tests { tool_name: None, tool_calls: None, attachments: Vec::new(), + turn_origin: crate::bus::TurnOrigin::User, }], )); diff --git a/src/config/mod.rs b/src/config/mod.rs index eae7793..afc1c92 100644 --- a/src/config/mod.rs +++ b/src/config/mod.rs @@ -57,6 +57,8 @@ pub struct Config { pub mcp: McpConfig, #[serde(default)] pub browser: BrowserConfig, + #[serde(default)] + pub agent_orchestration: AgentOrchestrationConfig, } fn default_workspace_dir() -> String { @@ -176,6 +178,126 @@ fn default_token_limit() -> usize { 128_000 } +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(default, deny_unknown_fields)] +pub struct AgentOrchestrationConfig { + pub enabled: bool, + pub definitions_dir: String, + pub root_delegates: Vec, + pub max_tree_depth: u16, + pub max_runs_per_tree: usize, + pub max_concurrent_runs: usize, + pub max_concurrent_runs_per_session: usize, + pub max_concurrent_provider_steps: usize, + pub max_concurrent_provider_steps_per_session: usize, + pub max_concurrent_tool_steps: usize, + pub max_concurrent_tool_steps_per_session: usize, + pub max_pending_inbox_events_per_session: usize, + pub inbox_event_ttl_hours: u64, + pub max_inbox_delivery_attempts: u32, + pub max_user_turn_burst_before_inbox: usize, + pub max_inbox_wait_secs: u64, +} + +impl Default for AgentOrchestrationConfig { + fn default() -> Self { + Self { + enabled: false, + definitions_dir: "agents".to_string(), + root_delegates: Vec::new(), + max_tree_depth: 4, + max_runs_per_tree: 16, + max_concurrent_runs: 6, + max_concurrent_runs_per_session: 4, + max_concurrent_provider_steps: 8, + max_concurrent_provider_steps_per_session: 4, + max_concurrent_tool_steps: 16, + max_concurrent_tool_steps_per_session: 8, + max_pending_inbox_events_per_session: 128, + inbox_event_ttl_hours: 168, + max_inbox_delivery_attempts: 8, + max_user_turn_burst_before_inbox: 4, + max_inbox_wait_secs: 30, + } + } +} + +impl AgentOrchestrationConfig { + pub fn validate(&self) -> Result<(), String> { + if !self.enabled { + return Ok(()); + } + let positive = [ + ("max_tree_depth", usize::from(self.max_tree_depth)), + ("max_runs_per_tree", self.max_runs_per_tree), + ("max_concurrent_runs", self.max_concurrent_runs), + ( + "max_concurrent_runs_per_session", + self.max_concurrent_runs_per_session, + ), + ( + "max_concurrent_provider_steps", + self.max_concurrent_provider_steps, + ), + ( + "max_concurrent_provider_steps_per_session", + self.max_concurrent_provider_steps_per_session, + ), + ("max_concurrent_tool_steps", self.max_concurrent_tool_steps), + ( + "max_concurrent_tool_steps_per_session", + self.max_concurrent_tool_steps_per_session, + ), + ( + "max_pending_inbox_events_per_session", + self.max_pending_inbox_events_per_session, + ), + ( + "max_inbox_delivery_attempts", + self.max_inbox_delivery_attempts as usize, + ), + ( + "max_user_turn_burst_before_inbox", + self.max_user_turn_burst_before_inbox, + ), + ]; + if let Some((name, _)) = positive.into_iter().find(|(_, value)| *value == 0) { + return Err(format!( + "agent_orchestration.{name} must be greater than zero" + )); + } + if self.definitions_dir.trim().is_empty() { + return Err("agent_orchestration.definitions_dir must not be empty".to_string()); + } + if self.max_concurrent_runs_per_session > self.max_concurrent_runs { + return Err("agent_orchestration.max_concurrent_runs_per_session cannot exceed max_concurrent_runs".to_string()); + } + if self.max_concurrent_provider_steps_per_session > self.max_concurrent_provider_steps { + return Err("agent_orchestration.max_concurrent_provider_steps_per_session cannot exceed max_concurrent_provider_steps".to_string()); + } + if self.max_concurrent_tool_steps_per_session > self.max_concurrent_tool_steps { + return Err("agent_orchestration.max_concurrent_tool_steps_per_session cannot exceed max_concurrent_tool_steps".to_string()); + } + if self.max_tree_depth > 32 { + return Err("agent_orchestration.max_tree_depth cannot exceed 32".to_string()); + } + if self.max_runs_per_tree > 1024 { + return Err("agent_orchestration.max_runs_per_tree cannot exceed 1024".to_string()); + } + if self.inbox_event_ttl_hours == 0 || self.inbox_event_ttl_hours > 24 * 365 { + return Err( + "agent_orchestration.inbox_event_ttl_hours must be between 1 and 8760".to_string(), + ); + } + if self.max_inbox_wait_secs == 0 || self.max_inbox_wait_secs > 3600 { + return Err( + "agent_orchestration.max_inbox_wait_secs must be between 1 and 3600".to_string(), + ); + } + Ok(()) + } +} + #[derive(Debug, Clone, Deserialize, Serialize)] pub struct GatewayConfig { #[serde(default = "default_gateway_host")] @@ -1029,6 +1151,7 @@ mod tests { 25 * 1024 * 1024 ); assert!(config.browser.enabled); + assert!(!config.agent_orchestration.enabled); let browser: BrowserConfig = serde_json::from_str("{}").unwrap(); assert!(browser.enabled); assert!( @@ -1039,6 +1162,23 @@ mod tests { ); } + #[test] + fn orchestration_config_enforces_hierarchical_limits() { + let valid = AgentOrchestrationConfig { + enabled: true, + ..Default::default() + }; + assert!(valid.validate().is_ok()); + + let invalid = AgentOrchestrationConfig { + enabled: true, + max_concurrent_runs: 1, + max_concurrent_runs_per_session: 2, + ..Default::default() + }; + assert!(invalid.validate().is_err()); + } + #[test] fn browser_persistence_config_is_strict_and_explicit() { let browser: BrowserConfig = serde_json::from_str( diff --git a/src/gateway/http.rs b/src/gateway/http.rs index 7319968..71bbb03 100644 --- a/src/gateway/http.rs +++ b/src/gateway/http.rs @@ -699,6 +699,13 @@ pub struct LimitQuery { limit: Option, } +#[derive(serde::Deserialize)] +pub struct AgentRunsQuery { + session_id: String, + cursor: Option, + limit: Option, +} + fn scheduler_snapshot(jobs: &[crate::storage::ScheduledJob]) -> Value { let enabled = jobs.iter().filter(|job| job.enabled).count(); let failed_jobs = jobs @@ -872,14 +879,159 @@ pub async fn get_tasks( Query(query): Query, ) -> Result, ApiError> { let limit = query.limit.unwrap_or(100).clamp(1, 500); - let tasks = state + // Unified projection: new durable agent_runs plus the legacy + // background_tasks records (read-only). Merged by created_at. + let runs = state + .storage + .list_all_agent_runs(None, limit as i64) + .await + .map_err(ApiError::internal)?; + let legacy = state .storage .list_recent_background_tasks(limit) .await .map_err(ApiError::internal)?; + let mut tasks: Vec = Vec::with_capacity(runs.len() + legacy.len()); + for run in runs { + tasks.push(json!({ + "source": "agent_run", + "id": run.id, + "group_id": run.group_id, + "parent_run_id": run.parent_run_id, + "session_id": run.root_session_id, + "channel": null, + "chat_id": null, + "agent_id": run.agent_id, + "mode": run.mode.as_str(), + "depth": run.depth, + "prompt": run.task, + "status": run.status.as_str(), + "result": run.result, + "error": run.error, + "tool_calls_count": run.tool_calls_count, + "iterations": run.iterations, + "started_at": run.started_at, + "finished_at": run.finished_at, + "created_at": run.created_at, + })); + } + for task in legacy { + tasks.push(json!({ + "source": "legacy_background_task", + "id": task.id, + "session_id": task.session_id, + "channel": task.channel, + "chat_id": task.chat_id, + "prompt": task.prompt, + "status": task.status, + "result": task.result, + "error": task.error, + "tool_calls_count": task.tool_calls_count, + "iterations": task.iterations, + "started_at": task.started_at, + "finished_at": task.finished_at, + "created_at": task.created_at, + })); + } + tasks.sort_by(|left, right| { + right + .get("created_at") + .and_then(Value::as_i64) + .cmp(&left.get("created_at").and_then(Value::as_i64)) + }); + tasks.truncate(limit); Ok(Json(json!({ "tasks": tasks }))) } +/// Session-scoped durable run listing with `(created_at,id)` cursor paging. +pub async fn get_agent_runs( + State(state): State>, + Query(query): Query, +) -> Result, ApiError> { + let Some(coordinator) = state.session_manager.agent_coordinator() else { + return Ok(Json(json!({ + "revision": 0, + "runs": [], + "next_cursor": Value::Null, + }))); + }; + let cursor = query.cursor.as_deref().and_then(|cursor| { + let (created_at, id) = cursor.split_once(':')?; + Some((created_at.parse::().ok()?, id.to_string())) + }); + let limit = query.limit.unwrap_or(100).clamp(1, 200) as i64; + let (revision, runs, next_cursor) = coordinator + .list_runs_for_session(&query.session_id, cursor, limit) + .await + .map_err(ApiError::internal)?; + Ok(Json(json!({ + "revision": revision, + "runs": runs, + "next_cursor": next_cursor, + }))) +} + +pub async fn get_agent_run( + State(state): State>, + Path(id): Path, +) -> Result, ApiError> { + let Some(_coordinator) = state.session_manager.agent_coordinator() else { + return Err(ApiError::not_found("run not found".to_string())); + }; + let run = state + .storage + .get_agent_run(&id) + .await + .map_err(ApiError::internal)?; + let Some(run) = run else { + return Err(ApiError::not_found(format!("run {id} not found"))); + }; + Ok(Json( + json!({ "run": crate::protocol::AgentRunView::from_record(&run, 100_000) }), + )) +} + +pub async fn get_agent_run_events( + State(state): State>, + Path(id): Path, + Query(query): Query, +) -> Result, ApiError> { + let Some(coordinator) = state.session_manager.agent_coordinator() else { + return Ok(Json(json!({ "events": [] }))); + }; + let limit = query.limit.unwrap_or(100).clamp(1, 200) as i64; + let events = coordinator + .list_run_events(&id, limit) + .await + .map_err(ApiError::internal)? + .iter() + .map(crate::protocol::AgentEventView::from_record) + .collect::>(); + Ok(Json(json!({ "events": events }))) +} + +pub async fn cancel_agent_run( + State(state): State>, + Path(id): Path, +) -> Result, ApiError> { + let Some(coordinator) = state.session_manager.agent_coordinator() else { + return Err(ApiError::not_found("run not found".to_string())); + }; + let Some(run) = state + .storage + .get_agent_run(&id) + .await + .map_err(ApiError::internal)? + else { + return Err(ApiError::not_found(format!("run {id} not found"))); + }; + let cancelled = coordinator + .cancel_run_for_session(&run.root_session_id, &id, "cancelled from management UI") + .await + .map_err(ApiError::internal)?; + Ok(Json(json!({ "cancelled": cancelled, "run_id": id }))) +} + pub async fn get_jobs(State(state): State>) -> Result, ApiError> { let jobs = state .storage diff --git a/src/gateway/mod.rs b/src/gateway/mod.rs index ce92081..929f449 100644 --- a/src/gateway/mod.rs +++ b/src/gateway/mod.rs @@ -19,7 +19,7 @@ use crate::logging; use crate::mcp; use crate::memory::MemoryManager; use crate::scheduler::Scheduler; -use crate::session::{SessionManager, SessionManagerServices}; +use crate::session::{AgentCatalogPreparation, SessionManager, SessionManagerServices}; use crate::task_supervisor::TaskSupervisor; /// Process boot clock. A process-level static so uptime survives config reload, @@ -52,6 +52,7 @@ pub struct GatewayState { pub outbound_lanes: Arc, pub(crate) reload: reload::ReloadHandle, pub(crate) admission: reload::RuntimeAdmission, + pub agent_catalog: Arc, } impl GatewayState { @@ -65,6 +66,7 @@ impl GatewayState { config_path, reload::ReloadHandle::unavailable(), true, + 1, ) .await } @@ -74,6 +76,7 @@ impl GatewayState { config_path: std::path::PathBuf, reload: reload::ReloadHandle, initialize_process: bool, + runtime_generation: u64, ) -> Result> { let task_supervisor = TaskSupervisor::new(); let admission = reload::RuntimeAdmission::open(); @@ -168,10 +171,34 @@ impl GatewayState { None }; let health = Arc::new(crate::health::HealthService::new(config.clone())); + let provider_profiles = if config.agent_orchestration.enabled { + config + .agents + .keys() + .filter_map(|name| { + config + .get_provider_config(name) + .ok() + .map(|profile| (name.clone(), profile)) + }) + .collect() + } else { + std::collections::HashMap::new() + }; + let config_dir = config_path + .parent() + .unwrap_or_else(|| std::path::Path::new(".")) + .to_path_buf(); // Create SessionManager with bus injection let session_manager = SessionManager::new( provider_config.clone(), + AgentCatalogPreparation { + provider_profiles, + config: config.agent_orchestration.clone(), + config_dir, + runtime_generation, + }, storage.clone(), SessionManagerServices::new( bus.clone(), @@ -186,6 +213,8 @@ impl GatewayState { config.gateway.max_concurrent_background_tasks, )?; let session_manager = Arc::new(session_manager); + session_manager.bind_inbox_wake(); + let agent_catalog = session_manager.agent_catalog(); // Register send_message tool with available channel names let available_channels = channel_manager.list_channel_names().await; @@ -254,6 +283,7 @@ impl GatewayState { outbound_lanes: Arc::new(AtomicUsize::new(0)), reload, admission, + agent_catalog, }) } @@ -269,6 +299,31 @@ impl GatewayState { /// Start the message processing loops pub async fn start_message_processing(&self) { + // Recover durable Agent state for this runtime generation: interrupt + // runs of older generations, expire stale inbox leases, converge + // group counters and reconcile capacity rows. Runs never recover + // while the generation is still a candidate. + if let Some(coordinator) = self.session_manager.agent_coordinator() { + match coordinator.recover_on_activation().await { + Ok(report) => { + if report.interrupted_runs > 0 || report.leases_expired > 0 { + tracing::warn!( + interrupted = report.interrupted_runs, + completion_events = report.completion_events_generated, + leases_expired = report.leases_expired, + dead_lettered = report.dead_lettered, + sessions_reconciled = report.sessions_reconciled, + groups_converged = report.groups_converged, + "Agent state recovered on activation" + ); + } + } + Err(error) => { + tracing::error!(error = %error, "Agent state recovery failed on activation"); + } + } + } + // MCP connections have external/process-wide side effects. Activate // them only after this generation becomes current, never while it is // merely a reload candidate. @@ -320,6 +375,23 @@ impl GatewayState { } }); + // Relay durable Agent run/event projections. A lagged or lost + // broadcast is non-fatal: the client recalibrates with GetAgentRuns. + let mut agent_projection_events = self.session_manager.projection_hub().subscribe(); + let cli_chat = self.cli_chat_channel(); + self.task_supervisor + .spawn("agent-projection-events", async move { + loop { + match agent_projection_events.recv().await { + Ok(event) => cli_chat.publish_agent_projection(event).await, + Err(tokio::sync::broadcast::error::RecvError::Lagged(skipped)) => { + tracing::warn!(skipped, "Agent projection relay lagged"); + } + Err(tokio::sync::broadcast::error::RecvError::Closed) => break, + } + } + }); + router::spawn_message_routers( bus.clone(), session_manager, @@ -380,6 +452,7 @@ pub async fn run( config_path.clone(), reload_controller.handle.clone(), true, + 1, ) .await?, ); @@ -460,6 +533,7 @@ pub async fn run( config_path.clone(), reload_controller.handle.clone(), false, + requested_generation, ); tokio::pin!(preparation); let prepared = match tokio::select! { @@ -600,6 +674,15 @@ fn build_router(state: Arc) -> Router { .route("/api/skills", routing::get(http::get_skills)) .route("/api/jobs", routing::get(http::get_jobs)) .route("/api/jobs/{id}/runs", routing::get(http::get_job_runs)) + .route("/api/agent-runs", routing::get(http::get_agent_runs)) + .route( + "/api/agent-runs/{id}", + routing::get(http::get_agent_run).post(http::cancel_agent_run), + ) + .route( + "/api/agent-runs/{id}/events", + routing::get(http::get_agent_run_events), + ) .route("/api/memories", routing::get(http::get_memories)) .route( "/api/memories/{key}", diff --git a/src/gateway/router.rs b/src/gateway/router.rs index bc7829b..0f9dd87 100644 --- a/src/gateway/router.rs +++ b/src/gateway/router.rs @@ -330,6 +330,18 @@ async fn handle_control_message(session_manager: &SessionManager, message: Contr .await .map(|stats| SessionEvent::SessionStats { stats }) .map_err(|error| ChannelError::Other(error.to_string())), + GetAgentRuns { + session_id, + cursor, + limit, + } => session_manager + .get_agent_runs(&session_id, cursor, limit) + .await + .map_err(|error| ChannelError::Other(error.to_string())), + GetAgentRun { session_id, run_id } => session_manager + .get_agent_run(&session_id, &run_id) + .await + .map_err(|error| ChannelError::Other(error.to_string())), RenameDialog { session_id, title } => session_manager .rename_dialog(&session_id, &title) .await @@ -427,6 +439,7 @@ mod tests { channel_context: ChannelContext { reply_to: Some("parent".to_string()), private: HashMap::from([("opaque".to_string(), "value".to_string())]), + durable_private: HashMap::new(), }, }; diff --git a/src/protocol.rs b/src/protocol.rs index d8f470a..72d60f3 100644 --- a/src/protocol.rs +++ b/src/protocol.rs @@ -37,6 +37,137 @@ pub struct MessageAttachment { pub mime_type: String, } +/// Bounded client projection of a durable Agent run. Sensitive internals +/// (budget, signal contract, delivery context, provider state) never leave +/// the gateway; the full result is available through the HTTP API. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct AgentRunView { + pub id: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub group_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub parent_run_id: Option, + pub agent_id: String, + pub provider_name: String, + pub model_id: String, + pub mode: String, + pub depth: u16, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub plan_item_id: Option, + pub status: String, + /// Task prompt (bounded for client delivery). + pub task: String, + /// Bounded result excerpt; full content is loaded on demand. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub result: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub error: Option, + pub tool_calls_count: i64, + pub iterations: i64, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub deadline_at: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub started_at: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub finished_at: Option, + pub created_at: i64, + pub updated_at: i64, +} + +/// Bounded client projection of a durable inbox event. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct AgentEventView { + pub id: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub run_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub group_id: Option, + pub event_type: String, + pub delivery: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub severity: Option, + /// Structured payload, bounded; signal summaries and completion status + /// only. The client must treat the payload as untrusted data. + pub payload_json: String, + pub status: String, + pub attempt_count: i64, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_error: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub consumed_at: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub superseded_at: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub dead_lettered_at: Option, + pub created_at: i64, +} + +impl AgentRunView { + pub fn from_record( + record: &crate::storage::agent_run::AgentRunRecord, + max_result_chars: usize, + ) -> Self { + let task = truncate(&record.task, 2_000); + let result = record + .result + .as_deref() + .map(|result| truncate(result, max_result_chars)); + Self { + id: record.id.clone(), + group_id: record.group_id.clone(), + parent_run_id: record.parent_run_id.clone(), + agent_id: record.agent_id.clone(), + provider_name: record.provider_name.clone(), + model_id: record.model_id.clone(), + mode: record.mode.as_str().to_string(), + depth: u16::try_from(record.depth).unwrap_or(u16::MAX), + plan_item_id: record.plan_item_id.clone(), + status: record.status.as_str().to_string(), + task, + result, + error: record.error.clone(), + tool_calls_count: record.tool_calls_count, + iterations: record.iterations, + deadline_at: Some(record.deadline_at), + started_at: record.started_at, + finished_at: record.finished_at, + created_at: record.created_at, + updated_at: record.updated_at, + } + } +} + +impl AgentEventView { + pub fn from_record(record: &crate::storage::agent_inbox::AgentInboxEventRecord) -> Self { + Self { + id: record.id.clone(), + run_id: record.run_id.clone(), + group_id: record.group_id.clone(), + event_type: record.event_type.as_str().to_string(), + delivery: record.delivery.as_str().to_string(), + severity: record.severity.clone(), + payload_json: record.payload_json.clone(), + status: record.status.as_str().to_string(), + attempt_count: record.attempt_count, + last_error: record.last_error.clone(), + consumed_at: record.consumed_at, + superseded_at: record.superseded_at, + dead_lettered_at: record.dead_lettered_at, + created_at: record.created_at, + } + } +} + +fn truncate(value: &str, max: usize) -> String { + if value.chars().count() <= max { + value.to_string() + } else { + let mut truncated: String = value.chars().take(max).collect(); + truncated.push('…'); + truncated + } +} + impl MessageAttachment { pub fn from_media_ref(index: usize, media_ref: &crate::bus::MediaRef) -> Self { let name = std::path::Path::new(&media_ref.path) @@ -75,6 +206,8 @@ pub struct HistoryMessage { pub tool_calls: Option>, #[serde(default, skip_serializing_if = "Vec::is_empty")] pub attachments: Vec, + #[serde(default)] + pub turn_origin: crate::bus::TurnOrigin, } impl From for HistoryMessage { @@ -97,6 +230,7 @@ impl From for HistoryMessage { tool_name: message.tool_name, tool_calls: message.tool_calls, attachments, + turn_origin: message.turn_origin, } } } @@ -126,6 +260,7 @@ impl HistoryMessage { .tool_calls .and_then(|calls| serde_json::from_str(&calls).ok()), attachments, + turn_origin: message.turn_origin, } } } @@ -196,6 +331,16 @@ pub enum WsInbound { }, #[serde(rename = "get_slash_commands")] GetSlashCommands, + #[serde(rename = "get_agent_runs")] + GetAgentRuns { + session_id: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + cursor: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + limit: Option, + }, + #[serde(rename = "get_agent_run")] + GetAgentRun { session_id: String, run_id: String }, #[serde(rename = "ping")] Ping, } @@ -274,6 +419,26 @@ pub enum WsOutbound { HistoryCleared { session_id: String }, #[serde(rename = "slash_commands_list")] SlashCommandsList { commands: Vec }, + #[serde(rename = "session_agent_runs")] + SessionAgentRuns { + session_id: String, + revision: i64, + runs: Vec, + #[serde(default, skip_serializing_if = "Option::is_none")] + next_cursor: Option, + }, + #[serde(rename = "agent_run_updated")] + AgentRunUpdated { + session_id: String, + revision: i64, + run: AgentRunView, + }, + #[serde(rename = "agent_event_updated")] + AgentEventUpdated { + session_id: String, + revision: i64, + event: AgentEventView, + }, #[serde(rename = "pong")] Pong, #[serde(rename = "command_executed")] @@ -343,6 +508,7 @@ mod tests { tool_name: None, tool_calls: None, attachments: Vec::new(), + turn_origin: crate::bus::TurnOrigin::User, }], }; let value = serde_json::to_value(frame).unwrap(); @@ -443,4 +609,86 @@ mod tests { assert_eq!(value["stats"]["context"]["source"], "hybrid"); assert_eq!(value["stats"]["lifetime_usage"]["input_tokens"], 100); } + + #[test] + fn agent_run_and_event_views_serialize_without_sensitive_fields() { + let run = crate::storage::agent_run::AgentRunRecord { + id: "run-1".to_string(), + group_id: None, + root_session_id: "cli:test:d1".to_string(), + root_turn_id: None, + parent_run_id: None, + caller_agent_id: "ROOT".to_string(), + caller_scope_id: "turn-1".to_string(), + idempotency_key: None, + agent_id: "researcher".to_string(), + definition_hash: "hash".to_string(), + provider_profile: "research".to_string(), + provider_name: "openai-test".to_string(), + model_id: "model-x".to_string(), + mode: crate::storage::agent_run::AgentRunMode::Background, + depth: 1, + plan_item_id: None, + execution_id: "run-1".to_string(), + task: "analyze the logs".to_string(), + context_json: Some("sensitive caller context".to_string()), + budget_json: r#"{"remaining_runs":3}"#.to_string(), + signal_contract_json: Some("secret contract".to_string()), + signal_delivery: None, + completion_delivery: None, + failure_delivery: None, + status: crate::storage::agent_run::AgentRunStatus::Completed, + result: Some("r".repeat(10_000)), + error: None, + prompt_tokens: None, + completion_tokens: None, + cost: None, + tool_calls_count: 3, + iterations: 2, + runtime_generation: 1, + attempt: 1, + completion_slot_reserved: false, + deadline_at: 1000, + revision: 7, + started_at: Some(10), + finished_at: Some(20), + created_at: 5, + updated_at: 20, + }; + let view = AgentRunView::from_record(&run, 2_000); + let value = serde_json::to_value(&view).unwrap(); + assert_eq!(value["id"], "run-1"); + assert_eq!(value["status"], "completed"); + assert!(!value.as_object().unwrap().contains_key("context_json")); + assert!(!value.as_object().unwrap().contains_key("budget_json")); + assert!( + !value + .as_object() + .unwrap() + .contains_key("signal_contract_json") + ); + assert!(!value.as_object().unwrap().contains_key("execution_id")); + // The result is bounded for client delivery. + assert!(value["result"].as_str().unwrap().chars().count() <= 2_001); + + let message = WsOutbound::AgentRunUpdated { + session_id: "cli:test:d1".to_string(), + revision: 7, + run: view, + }; + let value = serde_json::to_value(message).unwrap(); + assert_eq!(value["type"], "agent_run_updated"); + assert_eq!(value["revision"], 7); + + let inbound: WsInbound = + parse_inbound(r#"{"type":"get_agent_runs","session_id":"cli:test:d1","limit":10}"#) + .unwrap(); + assert!(matches!( + inbound, + WsInbound::GetAgentRuns { + limit: Some(10), + .. + } + )); + } } diff --git a/src/session/commands.rs b/src/session/commands.rs index 6ec9cc0..00d8857 100644 --- a/src/session/commands.rs +++ b/src/session/commands.rs @@ -30,6 +30,17 @@ pub enum SessionCommand { GetTaskPlan { session_id: UnifiedSessionId }, /// Load token totals and context-window state for a dialog. GetSessionStats { session_id: UnifiedSessionId }, + /// Load the durable Agent run projection for a dialog. + GetAgentRuns { + session_id: UnifiedSessionId, + cursor: Option, + limit: u32, + }, + /// Load one durable Agent run projection. + GetAgentRun { + session_id: UnifiedSessionId, + run_id: String, + }, /// Get the current dialog for a chat GetCurrentDialog { channel: String, chat_id: String }, /// Rename a dialog diff --git a/src/session/events.rs b/src/session/events.rs index dfc791e..0e0bc97 100644 --- a/src/session/events.rs +++ b/src/session/events.rs @@ -43,6 +43,19 @@ pub enum SessionEvent { }, /// Provider usage totals and current context-window state. SessionStats { stats: crate::session::SessionStats }, + /// Durable Agent run projection page for a dialog. + AgentRuns { + session_id: UnifiedSessionId, + revision: i64, + runs: Vec, + next_cursor: Option, + }, + /// One durable Agent run projection. + AgentRun { + session_id: UnifiedSessionId, + revision: i64, + run: Option, + }, /// Dialog renamed DialogRenamed { session_id: UnifiedSessionId, diff --git a/src/session/messenger.rs b/src/session/messenger.rs index 2c6212a..9445001 100644 --- a/src/session/messenger.rs +++ b/src/session/messenger.rs @@ -139,6 +139,9 @@ impl SessionManager { from_user_id: None, system_name: Some(job_name.to_string()), task_id: Some(job_id.to_string()), + from_run_id: None, + from_agent_id: None, + group_id: None, }, Vec::new(), ) @@ -171,6 +174,9 @@ mod tests { from_user_id: None, system_name: None, task_id: None, + from_run_id: None, + from_agent_id: None, + group_id: None, }; let media = vec![MediaItem::new("/tmp/report.pdf", "file")]; diff --git a/src/session/mod.rs b/src/session/mod.rs index cbf2a70..011a921 100644 --- a/src/session/mod.rs +++ b/src/session/mod.rs @@ -14,7 +14,10 @@ pub mod turn; pub use commands::SessionCommand; pub use error::SessionError; pub use events::{DialogInfo, SessionEvent}; -pub use session::{SLASH_COMMANDS, Session, SessionManager, SessionManagerServices, SlashCommand}; +pub use session::{ + AgentCatalogPreparation, SLASH_COMMANDS, Session, SessionManager, SessionManagerServices, + SlashCommand, +}; pub use session_id::UnifiedSessionId; pub use stats::{ContextUsage, ContextUsageSource, LifetimeUsage, SessionStats}; pub use turn::{ diff --git a/src/session/persistence.rs b/src/session/persistence.rs index 7596283..68df48b 100644 --- a/src/session/persistence.rs +++ b/src/session/persistence.rs @@ -11,6 +11,7 @@ use crate::{providers::Usage, session::TurnController}; async fn persist_added_messages( snapshots: Vec>, usage: Option<&crate::storage::TurnUsageRecord>, + steer: Option<&crate::storage::agent_inbox::SteerConsumption>, ) -> Result<(), StorageError> { let mut storage = None; let mut session_id = None; @@ -36,7 +37,17 @@ async fn persist_added_messages( else { return Ok(()); }; - if let Some(usage) = usage { + if let Some(steer) = steer { + storage + .persist_turn_with_steer_with_retry( + &session_id, + &messages, + &final_meta, + usage.unwrap(), + steer, + ) + .await + } else if let Some(usage) = usage { storage .persist_turn_batch_with_retry(&session_id, &messages, &final_meta, usage) .await @@ -77,6 +88,7 @@ pub(super) async fn append_active_turn_message( vec![message], VersionPolicy::PreserveForOwnedTurn(turn_id), None, + None, ) .await .map(|_| ()) @@ -86,7 +98,7 @@ pub(super) async fn append_persisted_messages_with_meta( session: &Arc>, messages: Vec, ) -> Result, StorageError> { - append_persisted_messages_inner(session, messages, VersionPolicy::Advance, None).await + append_persisted_messages_inner(session, messages, VersionPolicy::Advance, None, None).await } pub(super) async fn append_persisted_turn_messages( @@ -94,7 +106,25 @@ pub(super) async fn append_persisted_turn_messages( messages: Vec, usage: crate::storage::TurnUsageRecord, ) -> Result, StorageError> { - append_persisted_messages_inner(session, messages, VersionPolicy::Advance, Some(usage)).await + append_persisted_messages_inner(session, messages, VersionPolicy::Advance, Some(usage), None) + .await +} + +/// Persist a Turn and consume its admitted steer events atomically. +pub(super) async fn append_persisted_turn_messages_with_steer( + session: &Arc>, + messages: Vec, + usage: crate::storage::TurnUsageRecord, + steer: crate::storage::agent_inbox::SteerConsumption, +) -> Result, StorageError> { + append_persisted_messages_inner( + session, + messages, + VersionPolicy::Advance, + Some(usage), + Some(steer), + ) + .await } async fn append_persisted_messages_inner( @@ -102,6 +132,7 @@ async fn append_persisted_messages_inner( messages: Vec, version_policy: VersionPolicy, usage: Option, + steer: Option, ) -> Result, StorageError> { if messages.is_empty() { return Ok(Vec::new()); @@ -130,7 +161,7 @@ async fn append_persisted_messages_inner( .map(|(_, _, message, _)| message.clone()) .collect(); - if let Err(error) = persist_added_messages(snapshots, usage.as_ref()).await { + if let Err(error) = persist_added_messages(snapshots, usage.as_ref(), steer.as_ref()).await { session .lock() .await diff --git a/src/session/session.rs b/src/session/session.rs index 4f4d232..1995248 100644 --- a/src/session/session.rs +++ b/src/session/session.rs @@ -1,7 +1,8 @@ use std::collections::{HashMap, VecDeque}; use std::sync::{Arc, Mutex as StdMutex}; -use tokio::sync::{Mutex, mpsc, oneshot}; +use tokio::sync::{Mutex, mpsc, oneshot, watch}; +use tokio_util::sync::CancellationToken; use super::persistence::{ append_persisted_messages, append_persisted_messages_with_meta, append_persisted_turn_messages, @@ -46,6 +47,7 @@ fn committed_turn_delta( let history_revision = messages.last().map_or(0, |message| message.seq); let messages = messages .into_iter() + .filter(|message| message.client_visibility == crate::bus::ClientVisibility::Visible) .map(|message| crate::bus::CommittedMessage { id: message.id, seq: message.seq, @@ -63,6 +65,7 @@ fn committed_turn_delta( tool_calls: message .tool_calls .and_then(|calls| serde_json::from_str(&calls).ok()), + turn_origin: message.turn_origin, }) .collect(); crate::bus::CommittedTurnDelta { @@ -203,8 +206,8 @@ pub enum HandleResult { use crate::agent::context_compressor::ContextCompressionConfig; use crate::agent::system_prompt::build_system_prompt; use crate::agent::{ - AgentError, AgentLoop, AgentTurnContext, ContextCompressor, TurnEmitter, - steering::SteeringMailbox, + AgentError, AgentLoop, AgentTurnContext, ContextCompressor, TurnEmitter, TurnInput, + TurnInputSource, TurnMailbox, }; use crate::channels::slash_command::parse_slash_command; use crate::config::BrowserConfig; @@ -318,6 +321,36 @@ async fn fail_turn_with_partial( controller.fail(error); } +/// Persist the partial assistant output of a cancelled Turn and close it as +/// `Cancelled`. Shared by the `/stop` select branch and the cooperative +/// `AgentError::Cancelled` return so both paths commit the same terminal +/// state without emitting a processing-error outbound message. +async fn persist_cancelled_turn( + controller: &TurnController, + session: &Arc>, + pending_deliveries: Vec, +) { + let snapshot = controller.snapshot(); + if let Some(partial) = partial_assistant_with_pending_deliveries( + &snapshot, + CompletionStatus::Cancelled, + pending_deliveries, + ) { + controller.begin_finalizing(); + match append_persisted_messages(session, vec![partial]).await { + Ok(()) => { + controller.cancel(Some("stopped by user".to_string())); + } + Err(error) => { + tracing::error!(error = %error, "Failed to persist cancelled partial turn"); + controller.fail(format!("failed to persist cancelled turn: {error}")); + } + } + } else { + controller.cancel(Some("stopped by user".to_string())); + } +} + #[cfg(test)] mod cancelled_partial_tests { use super::*; @@ -593,6 +626,15 @@ pub struct Session { next_task_sequence: u64, /// Cancel signal for the currently executing agent task current_cancel: Option>, + /// Structured cancellation for the active Turn. `/stop` cancels it and + /// the token propagates through AgentLoop provider streams and tool + /// batches. The oneshot above remains the busy/stop compatibility + /// marker until it is removed together with the legacy adapter. + current_turn_token: Option, + /// Latest durable inbox revision for this session. The worker watches + /// this to claim agent events; the value only merges wakes, the payload + /// always comes from SQLite. + agent_inbox_wake: watch::Sender, active_turn_emitter: Option, /// Monotonic counter to detect stale workers worker_generation: u64, @@ -618,7 +660,7 @@ struct ActiveTurnEmitter { /// safe agent-loop boundary. The mailbox is shared with AgentLoop via /// `AgentTurnContext`; keeping it on the session handle makes admission /// atomic with `/stop` and worker cleanup. - steering: Arc, + steering: Arc, /// Original inbound tasks for accepted steering messages. ChatMessage /// intentionally carries only durable history fields, so this side map /// preserves channel context and rich MediaItem metadata if a terminal @@ -640,51 +682,126 @@ struct AgentTask { channel_context: ChannelContext, } +/// Build a typed steer input from a claimed inbox event. The content is a +/// bounded, readable envelope for the model; the typed source and durable +/// event id are preserved for rendering and cancellation recovery. +fn steer_input_from_event( + event: &crate::storage::agent_inbox::AgentInboxEventRecord, + now: i64, +) -> TurnInput { + use crate::agent::steering::InputDelivery; + use crate::storage::agent_inbox::AgentEventType; + let payload: serde_json::Value = + serde_json::from_str(&event.payload_json).unwrap_or(serde_json::Value::Null); + let run_id = event.run_id.clone().unwrap_or_default(); + let agent_id = payload + .get("agent_id") + .and_then(serde_json::Value::as_str) + .unwrap_or("unknown") + .to_string(); + let (source, content) = match event.event_type { + AgentEventType::Signal => { + let severity = event.severity.clone().unwrap_or_else(|| "info".to_string()); + let summary = payload + .get("summary") + .and_then(serde_json::Value::as_str) + .unwrap_or_default(); + let mut content = format!( + "[后台 Agent 信号] severity={severity}, agent={agent_id}, run={run_id}\n{summary}" + ); + if let Some(details) = payload.get("details") + && !details.is_null() + { + content.push_str("\n详情: "); + content.push_str(&details.to_string()); + } + (TurnInputSource::AgentSignal { run_id, agent_id }, content) + } + AgentEventType::Completion => { + let status = payload + .get("status") + .and_then(serde_json::Value::as_str) + .unwrap_or("completed"); + let mut content = + format!("[后台 Agent 任务结果] status={status}, agent={agent_id}, run={run_id}"); + if let Some(error) = payload.get("error").and_then(serde_json::Value::as_str) { + content.push_str(&format!("\n错误: {error}")); + } + ( + TurnInputSource::AgentCompletion { run_id, agent_id }, + content, + ) + } + AgentEventType::GroupCompletion => { + let group_id = event.group_id.clone().unwrap_or_default(); + let content = format!("[后台 Agent 任务组结果] group={group_id}"); + (TurnInputSource::AgentGroupCompletion { group_id }, content) + } + }; + TurnInput { + id: format!("steer:{}", event.id), + sequence: 0, + source, + delivery: InputDelivery::Steer, + content, + media_refs: Vec::new(), + durable_event_id: Some(event.id.clone()), + received_at: now, + message_source: None, + lease_token: Some(event.lease_token.clone().unwrap_or_default()), + } +} + /// Move terminally pending steering into the worker's local FIFO. The /// recovery map is consulted first so the original channel context and rich /// media metadata survive a same-turn fallback; only legacy/test producers /// need the ChatMessage-derived fallback. +/// +/// Durable Agent steer entries are NOT requeued as user tasks: they are +/// returned as `(event_id, lease_token)` pairs so the caller releases them +/// back to the durable inbox (`pending`), where the queue lane will deliver +/// them as a continuation. fn prepend_pending_steering( - mailbox: &SteeringMailbox, + mailbox: &TurnMailbox, recovery: &StdArc>>, local_tasks: &mut VecDeque, fallback_channel: &str, fallback_chat_id: &str, fallback_reply_to: &Option, fallback_private: &HashMap, -) { - let pending = mailbox.close_and_take_pending(); - for message in pending.into_iter().rev() { +) -> Vec<(String, String)> { + let take = mailbox.close_and_take_pending(); + for message in take.user_inputs.into_iter().rev() { let recovered = recovery .lock() .unwrap_or_else(|poisoned| poisoned.into_inner()) .remove(&message.id); - let task = recovered.unwrap_or_else(|| { - let source = message.source.clone(); - AgentTask { - channel: source - .as_ref() - .and_then(|source| source.from_channel.clone()) - .unwrap_or_else(|| fallback_channel.to_string()), - sender_id: source - .as_ref() - .and_then(|source| source.from_user_id.clone()) - .unwrap_or_else(|| "unknown".to_string()), - chat_id: fallback_chat_id.to_string(), - sequence: 0, - client_message_id: Some(message.id.clone()), - content: message.content, - received_at: message.timestamp, - media: message - .media_refs - .into_iter() - .map(|media| MediaItem::new(media.path, media.media_type)) - .collect(), - channel_context: ChannelContext { - reply_to: fallback_reply_to.clone(), - private: fallback_private.clone(), - }, - } + let task = recovered.unwrap_or_else(|| AgentTask { + channel: message + .message_source + .as_ref() + .and_then(|source| source.from_channel.clone()) + .unwrap_or_else(|| fallback_channel.to_string()), + sender_id: message + .message_source + .as_ref() + .and_then(|source| source.from_user_id.clone()) + .unwrap_or_else(|| "unknown".to_string()), + chat_id: fallback_chat_id.to_string(), + sequence: 0, + client_message_id: Some(message.id.clone()), + content: message.content, + received_at: message.received_at, + media: message + .media_refs + .into_iter() + .map(|media| MediaItem::new(media.path, media.media_type)) + .collect(), + channel_context: ChannelContext { + reply_to: fallback_reply_to.clone(), + private: fallback_private.clone(), + durable_private: HashMap::new(), + }, }); local_tasks.push_front(task); } @@ -692,6 +809,32 @@ fn prepend_pending_steering( .lock() .unwrap_or_else(|poisoned| poisoned.into_inner()) .clear(); + let mut leases = take.admitted_leases; + leases.extend(take.reserved_leases); + leases +} + +/// Return admitted/leased Agent steer events to the durable inbox. They +/// become `pending` again with an immediate retry so the queue lane picks +/// them up (steer reliably degrades to queue when the active Turn closes). +async fn release_steer_leases(storage: &Storage, leases: Vec<(String, String)>) { + if leases.is_empty() { + return; + } + let mut by_token: std::collections::HashMap> = + std::collections::HashMap::new(); + for (event_id, token) in leases { + by_token.entry(token).or_default().push(event_id); + } + let now = chrono::Utc::now().timestamp_millis(); + for (token, ids) in by_token { + if let Err(error) = storage + .release_inbox_lease(&token, &ids, 0, Some("steer turn closed"), now) + .await + { + tracing::warn!(error = %error, "failed to release steer events to pending"); + } + } } fn pop_lowest_sequence(local_tasks: &mut VecDeque) -> Option { @@ -711,6 +854,15 @@ struct AgentWorkerDeps { skills_loader: Arc, task_supervisor: crate::task_supervisor::TaskSupervisor, turn_delivery: TurnDeliveryService, + execution_gate: Arc, + /// Fairness policy: after this many consecutive user Turns a due inbox + /// event must be processed before another user input. + inbox_burst: usize, + /// Fairness policy: a pending event older than this is forced before the + /// next user Turn. + inbox_wait_ms: i64, + /// Delivery attempts before an inbox event is dead-lettered. + inbox_max_attempts: i64, } impl Session { @@ -767,6 +919,8 @@ impl Session { agent_tx: None, next_task_sequence: 1, current_cancel: None, + current_turn_token: None, + agent_inbox_wake: watch::channel(0).0, active_turn_emitter: None, worker_generation: 0, title_generation_in_flight: false, @@ -855,6 +1009,8 @@ impl Session { turn_id: m.turn_id, iteration: m.iteration.and_then(|value| u32::try_from(value).ok()), completion_status: m.completion_status, + client_visibility: m.client_visibility, + turn_origin: m.turn_origin, media_refs: m .media_refs .map(|refs| serde_json::from_str(&refs).unwrap_or_default()) @@ -896,6 +1052,8 @@ impl Session { turn_id: m.turn_id, iteration: m.iteration.and_then(|value| u32::try_from(value).ok()), completion_status: m.completion_status, + client_visibility: m.client_visibility, + turn_origin: m.turn_origin, media_refs: m .media_refs .map(|refs| serde_json::from_str(&refs).unwrap_or_default()) @@ -958,6 +1116,8 @@ impl Session { agent_tx: None, next_task_sequence: 1, current_cancel: None, + current_turn_token: None, + agent_inbox_wake: watch::channel(0).0, active_turn_emitter: None, worker_generation: 0, title_generation_in_flight: false, @@ -978,6 +1138,8 @@ impl Session { advance_state_version: bool, ) -> Option { let is_user = message.role == "user"; + let counts_as_user_input = + is_user && message.client_visibility == crate::bus::ClientVisibility::Visible; let now = chrono::Utc::now().timestamp_millis(); // Assign seq @@ -1000,6 +1162,8 @@ impl Session { turn_id: message.turn_id.clone(), iteration: message.iteration.map(i64::from), completion_status: message.completion_status, + client_visibility: message.client_visibility, + turn_origin: message.turn_origin, media_refs: if message.media_refs.is_empty() { None } else { @@ -1026,7 +1190,7 @@ impl Session { // Update in-memory state self.messages.push(message); self.total_message_count += 1; - if is_user { + if counts_as_user_input { self.message_count += 1; } self.last_active_at = now; @@ -1115,7 +1279,7 @@ impl Session { self.active_turn_emitter = Some(ActiveTurnEmitter { turn_id: turn_id.to_string(), emitter, - steering: SteeringMailbox::new_shared(), + steering: TurnMailbox::new_shared(), recovery: StdArc::new(StdMutex::new(HashMap::new())), }); } @@ -1526,6 +1690,14 @@ pub struct SessionManager { memory_manager: Arc, work_manager: Arc, sub_agent_manager: Arc, + agent_catalog: Arc, + execution_gate: Arc, + agent_coordinator: Option>, + inbox_notifier: Arc, + agent_projection_hub: Arc, + inbox_burst: usize, + inbox_wait_ms: i64, + inbox_max_attempts: i64, task_supervisor: crate::task_supervisor::TaskSupervisor, turn_delivery: TurnDeliveryService, reload: crate::gateway::reload::ReloadHandle, @@ -1542,6 +1714,13 @@ pub struct SessionManagerServices { admission: crate::gateway::reload::RuntimeAdmission, } +pub struct AgentCatalogPreparation { + pub provider_profiles: HashMap, + pub config: crate::config::AgentOrchestrationConfig, + pub config_dir: std::path::PathBuf, + pub runtime_generation: u64, +} + impl SessionManagerServices { pub fn new( bus: Arc, @@ -1696,11 +1875,16 @@ impl SessionManager { skills_loader: self.skills_loader.clone(), task_supervisor: self.task_supervisor.clone(), turn_delivery: self.turn_delivery.clone(), + execution_gate: self.execution_gate.clone(), + inbox_burst: self.inbox_burst, + inbox_wait_ms: self.inbox_wait_ms, + inbox_max_attempts: self.inbox_max_attempts, } } pub fn new( provider_config: LLMProviderConfig, + catalog_preparation: AgentCatalogPreparation, storage: Arc, services: SessionManagerServices, browser_config: Option, @@ -1734,6 +1918,23 @@ impl SessionManager { .map_err(|error| AgentError::Other(format!("failed to create tools: {error}")))?, ); + let agent_catalog = Arc::new( + crate::agent::AgentCatalog::load( + &catalog_preparation.config, + &catalog_preparation.config_dir, + &catalog_preparation.provider_profiles, + &tools, + &skills_loader, + catalog_preparation.runtime_generation, + ) + .map_err(|error| AgentError::Other(format!("failed to load Agent catalog: {error}")))?, + ); + let execution_gate = if catalog_preparation.config.enabled { + crate::agent::gate::ExecutionGate::new(&catalog_preparation.config) + } else { + crate::agent::gate::ExecutionGate::unbounded() + }; + // Create SubAgentManager and register DelegateTool let (notify_tx, mut notify_rx) = tokio::sync::mpsc::unbounded_channel(); let sub_agent_manager = Arc::new( @@ -1747,9 +1948,32 @@ impl SessionManager { task_supervisor.clone(), ) .with_admission(admission) + .with_catalog(agent_catalog.clone()) + .with_execution_gate(execution_gate.clone()) .with_work_manager(work_manager.clone()), ); - tools.register(crate::tools::DelegateTool::new(sub_agent_manager.clone())); + let mut delegate_tool = crate::tools::DelegateTool::new(sub_agent_manager.clone()); + let inbox_notifier = crate::agent::AgentInboxNotifier::new(); + let agent_projection_hub = Arc::new(crate::agent::AgentProjectionHub::new()); + let agent_coordinator = if agent_catalog.enabled() { + let coordinator = crate::agent::AgentCoordinator::new( + storage.clone(), + sub_agent_manager.clone(), + work_manager.clone(), + inbox_notifier.clone(), + agent_projection_hub.clone(), + task_supervisor.clone(), + catalog_preparation.runtime_generation, + &catalog_preparation.config, + ); + tools.register(crate::tools::AgentTaskTool::new(coordinator.clone())); + delegate_tool = delegate_tool.with_coordinator(coordinator.clone()); + sub_agent_manager.bind_coordinator(&coordinator); + Some(coordinator) + } else { + None + }; + tools.register(delegate_tool); tools.register(crate::tools::ReloadConfigTool::new(reload.clone())); // Start background task notification consumer @@ -1807,6 +2031,14 @@ impl SessionManager { memory_manager, work_manager, sub_agent_manager, + agent_catalog, + execution_gate, + agent_coordinator, + inbox_notifier, + agent_projection_hub, + inbox_burst: catalog_preparation.config.max_user_turn_burst_before_inbox, + inbox_wait_ms: (catalog_preparation.config.max_inbox_wait_secs * 1000) as i64, + inbox_max_attempts: i64::from(catalog_preparation.config.max_inbox_delivery_attempts), task_supervisor, turn_delivery, reload, @@ -1825,10 +2057,31 @@ impl SessionManager { self.tools.clone() } + pub fn agent_catalog(&self) -> Arc { + self.agent_catalog.clone() + } + + /// The durable coordinator, present only when orchestration is enabled. + pub fn agent_coordinator(&self) -> Option> { + self.agent_coordinator.clone() + } + + /// Bind this SessionManager as the late-bound wake target of the inbox + /// notifier. Called once after the manager is placed in an Arc. + pub fn bind_inbox_wake(self: &Arc) { + let weak: std::sync::Weak = + Arc::downgrade(self) as std::sync::Weak; + self.inbox_notifier.bind(weak); + } + pub fn skills_loader(&self) -> Arc { self.skills_loader.clone() } + pub fn projection_hub(&self) -> Arc { + self.agent_projection_hub.clone() + } + pub fn work_manager(&self) -> Arc { self.work_manager.clone() } @@ -2111,21 +2364,27 @@ impl SessionManager { let sid = current_session_id .ok_or_else(|| AgentError::Other("no active session".to_string()))?; let session = self.get_or_create_session(sid).await?; + let mut stop_leases: Option = None; let msgs = { let mut guard = session.lock().await; let mut msgs: Vec = Vec::new(); if guard.current_cancel.take().is_some() { msgs.push("当前任务已发送停止信号。".to_string()); } + if let Some(token) = guard.current_turn_token.take() { + token.cancel(); + } if let Some(active_turn) = guard.active_turn_emitter.take() { // Closing the mailbox makes an in-flight ordinary // input race resolve to the next queue (or an // explicit queue-full response), rather than being // accepted after `/stop` has invalidated the turn. // `/stop` intentionally discards accepted-but-not-yet - // injected steering, matching its queue-clearing - // semantics. - let _ = active_turn.steering.close_and_take_pending(); + // injected user steering, matching its queue-clearing + // semantics. Durable Agent steer events are NOT + // discarded: they are released back to `pending` so + // the queue lane can still deliver them. + stop_leases = Some(active_turn.steering.close_and_take_pending()); active_turn .recovery .lock() @@ -2142,10 +2401,26 @@ impl SessionManager { }; // Cancel all running background sub-agent tasks for this session - // after releasing the session lock. + // after releasing the session lock. Named durable runs are + // cancelled with suppress_continuation so no continuation Turn + // restarts after an explicit stop. self.sub_agent_manager .cancel_by_session(&sid.to_string()) .await; + if let Some(coordinator) = self.agent_coordinator.as_ref() + && let Err(error) = coordinator + .cancel_session(&sid.to_string(), "stopped by user") + .await + { + tracing::warn!(error = %error, "Failed to cancel durable agent runs on /stop"); + } + // Durable steer events admitted to the stopped Turn return to + // `pending`; the queue lane delivers them later. They are + // never silently discarded by `/stop`. + let stop_leases = stop_leases.unwrap_or_default(); + let mut leases = stop_leases.admitted_leases; + leases.extend(stop_leases.reserved_leases); + release_steer_leases(&self.storage, leases).await; let resp = if msgs.is_empty() { "没有正在执行的任务或队列。".to_string() } else { @@ -2572,6 +2847,52 @@ impl SessionManager { }) } + /// Durable Agent run projection page for management/WebSocket clients. + pub async fn get_agent_runs( + &self, + session_id: &UnifiedSessionId, + cursor: Option, + limit: u32, + ) -> Result { + let cursor = cursor.as_deref().and_then(|cursor| { + let (created_at, id) = cursor.split_once(':')?; + Some((created_at.parse::().ok()?, id.to_string())) + }); + let (revision, runs, next_cursor) = match self.agent_coordinator.as_ref() { + Some(coordinator) => coordinator + .list_runs_for_session(&session_id.to_string(), cursor, i64::from(limit)) + .await + .map_err(|error| AgentError::Other(error.to_string()))?, + None => (0, Vec::new(), None), + }; + Ok(crate::session::SessionEvent::AgentRuns { + session_id: session_id.clone(), + revision, + runs, + next_cursor, + }) + } + + /// One durable Agent run projection. + pub async fn get_agent_run( + &self, + session_id: &UnifiedSessionId, + run_id: &str, + ) -> Result { + let (revision, run) = match self.agent_coordinator.as_ref() { + Some(coordinator) => coordinator + .get_run_for_session(&session_id.to_string(), run_id) + .await + .map_err(|error| AgentError::Other(error.to_string()))?, + None => (0, None), + }; + Ok(crate::session::SessionEvent::AgentRun { + session_id: session_id.clone(), + revision, + run, + }) + } + pub async fn list_dialogs( &self, channel: &str, @@ -2633,6 +2954,22 @@ impl SessionManager { let persistence_lock = { session.lock().await.persistence_lock.clone() }; let _persistence_guard = persistence_lock.lock().await; + // Cancel durable Agent runs and dead-letter inbox events first; the + // audit rows remain visible on the management surface. + if let Some(coordinator) = self.agent_coordinator.as_ref() { + let _ = coordinator + .cancel_session(&session_id_str, "session deleted") + .await; + let _ = self + .storage + .dead_letter_agent_events( + &session_id_str, + "session_deleted", + chrono::Utc::now().timestamp_millis(), + ) + .await; + } + // Soft delete from Storage self.storage .soft_delete_session(&session_id_str) @@ -2653,6 +2990,19 @@ impl SessionManager { let session = self.get_or_create_session(session_id).await?; let persistence_lock = { session.lock().await.persistence_lock.clone() }; let _persistence_guard = persistence_lock.lock().await; + if let Some(coordinator) = self.agent_coordinator.as_ref() { + let _ = coordinator + .cancel_session(&session_id_str, "session archived") + .await; + let _ = self + .storage + .dead_letter_agent_events( + &session_id_str, + "session_archived", + chrono::Utc::now().timestamp_millis(), + ) + .await; + } self.storage .archive_session(&session_id_str) .await @@ -2789,6 +3139,9 @@ impl SessionManager { from_user_id: None, system_name: Some(system_name.to_string()), task_id: task_id.map(|s| s.to_string()), + from_run_id: None, + from_agent_id: None, + group_id: None, }; let msg = ChatMessage::assistant_with_source(content, source); append_persisted_messages(&session, vec![msg]) @@ -2916,6 +3269,9 @@ impl SessionManager { from_user_id: Some(sender_id.to_string()), system_name: None, task_id: None, + from_run_id: None, + from_agent_id: None, + group_id: None, }; let mut message = guard.create_user_message_with_source(content, media_refs, source); @@ -2926,6 +3282,13 @@ impl SessionManager { { message.id = id.to_string(); } + if let Some(id) = inbound + .client_message_id + .as_deref() + .filter(|id| !id.trim().is_empty()) + { + message.id = id.to_string(); + } message.timestamp = inbound.received_at; let message_id = message.id.clone(); let mut recovery_task = task.clone(); @@ -2935,7 +3298,14 @@ impl SessionManager { .lock() .unwrap_or_else(|poisoned| poisoned.into_inner()) .insert(message_id.clone(), recovery_task); - match active.steering.try_push(message) { + let input = TurnInput::user( + message_id.clone(), + message.content.clone(), + message.media_refs.clone(), + message.source.clone(), + message.timestamp, + ); + match active.steering.try_push_user(input) { Ok(()) => return Ok(HandleResult::AgentProcessing), Err(_) => { active @@ -2981,6 +3351,7 @@ impl SessionManager { if needs_spawn { guard.agent_tx = None; guard.current_cancel = None; + guard.current_turn_token = None; guard.worker_generation = guard.worker_generation.wrapping_add(1); let generation = guard.worker_generation; let (tx, rx) = mpsc::channel(SESSION_QUEUE_CAPACITY); @@ -3011,6 +3382,7 @@ impl SessionManager { let task = error.into_inner(); guard.agent_tx = None; guard.current_cancel = None; + guard.current_turn_token = None; guard.worker_generation = guard.worker_generation.wrapping_add(1); let generation = guard.worker_generation; let (tx, rx) = mpsc::channel(SESSION_QUEUE_CAPACITY); @@ -3125,6 +3497,10 @@ fn spawn_agent_worker( skills_loader, task_supervisor, turn_delivery, + execution_gate, + inbox_burst, + inbox_wait_ms, + inbox_max_attempts, } = deps; let worker_supervisor = task_supervisor.clone(); task_supervisor.spawn(format!("session-worker:{unified_str}"), async move { @@ -3136,7 +3512,72 @@ fn spawn_agent_worker( // not overtake an earlier accepted steering message, and moving // pending inputs here cannot fail due to channel capacity. let mut local_tasks = VecDeque::new(); + // Durable inbox wake lane: the value only merges wakes; event + // payloads always come from SQLite claims. + let mut inbox_wake_rx = { + let session_guard = session.lock().await; + session_guard.agent_inbox_wake.subscribe() + }; + let mut consecutive_user_turns = 0usize; 'tasks: loop { + // Fairness: a due inbox event must be processed before the + // next user Turn once the user burst budget is exhausted or + // the oldest pending event has waited too long. The next + // pending due time also arms a timer so a released event is + // re-claimed after its retry backoff without needing a wake. + let mut next_due_at = None; + let storage = { + let guard = session.lock().await; + guard.storage.clone() + }; + if let Some(storage) = storage { + let now = chrono::Utc::now().timestamp_millis(); + let oldest_due = crate::storage::Storage::oldest_pending_due( + &storage, + &unified_str, + now, + ) + .await + .unwrap_or(None); + next_due_at = crate::storage::Storage::next_pending_due_at( + &storage, + &unified_str, + ) + .await + .unwrap_or(None); + let force = consecutive_user_turns >= inbox_burst + || oldest_due.is_some_and(|created_at| now - created_at >= inbox_wait_ms); + if force + && let Ok(Some(lease)) = crate::storage::Storage::claim_inbox_batch( + &storage, + &unified_str, + now, + 60_000, + 8, + 32 * 1024, + None, + ) + .await + { + let claimed = run_inbox_continuation( + &session, + &turn_delivery, + &execution_gate, + lease, + worker_gen, + &unified_str, + inbox_max_attempts, + ) + .await; + if claimed { + consecutive_user_turns = 0; + continue 'tasks; + } + // Claim raced with a user message; fall through to the + // user lane below. + } + } + // Admission sequence numbers are allocated while holding the // Session lock. Drain everything currently visible on the // channel before selecting the smallest sequence, so a @@ -3147,7 +3588,27 @@ fn spawn_agent_worker( let task = if let Some(task) = pop_lowest_sequence(&mut local_tasks) { task } else { - match task_rx.recv().await { + let has_due = next_due_at.is_some(); + let due_sleep = async move { + if let Some(due) = next_due_at { + let now = chrono::Utc::now().timestamp_millis(); + let remaining = (due - now).max(0) as u64; + tokio::time::sleep(tokio::time::Duration::from_millis(remaining)).await; + } + }; + match tokio::select! { + task = task_rx.recv() => task, + _ = inbox_wake_rx.changed() => { + // A durable event landed while idle; loop to + // re-check the inbox before waiting for a user. + continue 'tasks; + } + _ = due_sleep, if has_due => { + // A released event's retry backoff elapsed; loop + // to re-claim it. + continue 'tasks; + } + } { Some(task) => task, None => break, } @@ -3175,6 +3636,9 @@ fn spawn_agent_worker( from_user_id: Some(task.sender_id.clone()), system_name: None, task_id: None, + from_run_id: None, + from_agent_id: None, + group_id: None, }; let mut message = guard.create_user_message_with_source(&task.content, media_refs, source); @@ -3202,6 +3666,25 @@ fn spawn_agent_worker( let _ = bus.publish_outbound(err_outbound).await; continue 'tasks; } + // Remember the channel's durable delivery context (thread/ + // root identity) so later continuation Turns can reuse it. + // One-shot reply/reaction ids never reach this column. + if !task.channel_context.durable_private.is_empty() + && let Some(storage) = session.lock().await.storage.clone() + { + let durable_json = serde_json::json!(task.channel_context.durable_private) + .to_string(); + if let Err(e) = storage + .update_session_delivery_context( + &unified_str, + &durable_json, + chrono::Utc::now().timestamp_millis(), + ) + .await + { + tracing::warn!(error = %e, "Failed to persist durable delivery context"); + } + } let ( agent, @@ -3210,6 +3693,7 @@ fn spawn_agent_worker( system_prompt_out, base_version, cancel_rx, + turn_token, turn_controller, turn_emitter, turn_receiver, @@ -3245,11 +3729,13 @@ fn spawn_agent_worker( }; let (cancel_tx, cancel_rx) = oneshot::channel(); + let turn_token = CancellationToken::new(); if guard.worker_generation != worker_gen { return; // /stop replaced us } guard.current_cancel = Some(cancel_tx); + guard.current_turn_token = Some(turn_token.clone()); // Install the active-turn handle before memory recall and // context preparation. This closes the historical race @@ -3262,7 +3748,7 @@ fn spawn_agent_worker( uuid::Uuid::new_v4().to_string(), ); let initial_turn = turn_controller.snapshot(); - let steering = SteeringMailbox::new_shared(); + let steering = TurnMailbox::new_shared(); let recovery = StdArc::new(StdMutex::new(HashMap::new())); guard.active_turn_emitter = Some(ActiveTurnEmitter { turn_id: initial_turn.id.0.clone(), @@ -3278,6 +3764,7 @@ fn spawn_agent_worker( guard.build_system_prompt(&skills_prompt), guard.state_version, cancel_rx, + turn_token, turn_controller, turn_emitter, turn_receiver, @@ -3299,8 +3786,9 @@ fn spawn_agent_worker( .await; let meta_snapshot = { let mut guard = session.lock().await; + let storage = guard.storage.clone(); if guard.worker_generation != worker_gen { - steering.close_and_take_pending(); + let mut take = steering.close_and_take_pending(); recovery .lock() .unwrap_or_else(|poisoned| poisoned.into_inner()) @@ -3311,7 +3799,9 @@ fn spawn_agent_worker( .is_some_and(|active| active.turn_id == initial_turn.id.0) && let Some(active) = guard.active_turn_emitter.take() { - let _ = active.steering.close_and_take_pending(); + let active_take = active.steering.close_and_take_pending(); + take.admitted_leases.extend(active_take.admitted_leases); + take.reserved_leases.extend(active_take.reserved_leases); active .recovery .lock() @@ -3319,6 +3809,12 @@ fn spawn_agent_worker( .clear(); active.emitter.deactivate(); } + drop(guard); + let mut leases = take.admitted_leases; + leases.extend(take.reserved_leases); + if let Some(storage) = storage { + release_steer_leases(&storage, leases).await; + } turn_emitter.deactivate(); turn_controller.cancel(Some( "session changed before turn preparation completed".to_string(), @@ -3331,13 +3827,15 @@ fn spawn_agent_worker( "Session changed while preparing agent history; dropping stale task" ); guard.current_cancel = None; + guard.current_turn_token = None; + let mut released = Vec::new(); if guard .active_turn_emitter .as_ref() .is_some_and(|active| active.turn_id == initial_turn.id.0) && let Some(active) = guard.active_turn_emitter.take() { - prepend_pending_steering( + let leases = prepend_pending_steering( &active.steering, &active.recovery, &mut local_tasks, @@ -3346,8 +3844,13 @@ fn spawn_agent_worker( &task_reply_to, &task_metadata, ); + released.extend(leases); active.emitter.deactivate(); } + drop(guard); + if let Some(storage) = storage.clone() { + release_steer_leases(&storage, released).await; + } turn_emitter.deactivate(); turn_controller.cancel(Some( "session changed while preparing agent history".to_string(), @@ -3394,14 +3897,16 @@ fn spawn_agent_worker( let live_delivery_started = delivery_handle.is_some(); { let mut guard = session.lock().await; + let storage = guard.storage.clone(); if guard.worker_generation != worker_gen || guard.state_version != base_version { + let mut released = Vec::new(); if guard .active_turn_emitter .as_ref() .is_some_and(|active| active.turn_id == active_turn_id) && let Some(active) = guard.active_turn_emitter.take() { - prepend_pending_steering( + let leases = prepend_pending_steering( &active.steering, &active.recovery, &mut local_tasks, @@ -3410,6 +3915,7 @@ fn spawn_agent_worker( &task_reply_to, &task_metadata, ); + released.extend(leases); active.emitter.deactivate(); } turn_emitter.deactivate(); @@ -3417,6 +3923,11 @@ fn spawn_agent_worker( "session changed before model execution".to_string(), )); guard.current_cancel = None; + guard.current_turn_token = None; + drop(guard); + if let Some(storage) = storage { + release_steer_leases(&storage, released).await; + } continue 'tasks; } } @@ -3442,10 +3953,13 @@ fn spawn_agent_worker( let pending_turn_deliveries = Arc::new(std::sync::Mutex::new(Vec::new())); let scoped_turn_deliveries = pending_turn_deliveries.clone(); let steering_for_process = steering.clone(); + let process_gate = execution_gate.clone(); let process_future = async move { let response_session_id = unified_str2.clone(); let tool_context = ToolExecutionContext::for_session(&response_session_id) - .with_turn_id(agent_turn.turn_id.clone()); + .with_turn_id(agent_turn.turn_id.clone()) + .with_cancellation(turn_token.clone()) + .with_execution_gate(process_gate.clone()); let process_result = crate::agent::sub_agent::DELEGATE_CONTEXT.scope( crate::agent::DelegateContext { session_id: unified_str2, @@ -3549,6 +4063,15 @@ fn spawn_agent_worker( .await { Ok(r) => r, + Err(AgentError::Cancelled) => { + persist_cancelled_turn( + turn_lifecycle, + &session2, + take_current_turn_deliveries(), + ) + .await; + return; + } Err(e) => { tracing::error!( error = %e, @@ -3579,6 +4102,15 @@ fn spawn_agent_worker( } } } + Err(AgentError::Cancelled) => { + persist_cancelled_turn( + turn_lifecycle, + &session2, + take_current_turn_deliveries(), + ) + .await; + return; + } Err(e) => { tracing::error!(error = %e, "Agent processing error"); fail_turn_with_partial( @@ -3618,12 +4150,19 @@ fn spawn_agent_worker( .map(|value| value.prompt_tokens); let (provider_name, model_name) = { let guard = session2.lock().await; + let storage = guard.storage.clone(); if guard.worker_generation != worker_gen { // A generation change is the explicit `/stop` / - // worker-replacement boundary; discard pending - // and in-flight steering just like the command - // path does. - steering_for_process.close_and_take_pending(); + // worker-replacement boundary. User steering is + // discarded like the command path does, but + // durable Agent steer events return to `pending`. + let take = steering_for_process.close_and_take_pending(); + drop(guard); + let mut leases = take.admitted_leases; + leases.extend(take.reserved_leases); + if let Some(storage) = storage { + release_steer_leases(&storage, leases).await; + } turn_lifecycle.cancel(Some( "session changed before turn commit".to_string(), )); @@ -3657,11 +4196,36 @@ fn spawn_agent_worker( } }); let emitted_messages = result.emitted_messages; + // Steer events admitted to this Turn are consumed in the + // same transaction as the Turn commit: the client never + // observes history without the event consumption. + let steer_consumption = { + let event_ids = steering_for_process.durable_in_flight_ids(); + if event_ids.is_empty() { + None + } else { + Some(crate::storage::agent_inbox::SteerConsumption { + event_ids, + admitted_turn_id: agent_turn.turn_id.clone(), + }) + } + }; let response = match finalize_turn_after_persistence( turn_lifecycle, usage.clone(), async { - if let Some(usage_record) = usage_record { + if let Some(steer) = steer_consumption.as_ref() + && let Some(usage_record) = usage_record.clone() + { + let steer = steer.clone(); + crate::session::persistence::append_persisted_turn_messages_with_steer( + &session2, + emitted_messages, + usage_record, + steer, + ) + .await + } else if let Some(usage_record) = usage_record { append_persisted_turn_messages( &session2, emitted_messages, @@ -3779,27 +4343,12 @@ fn spawn_agent_worker( () = process_future => {} _ = cancel_rx => { // cancelled — current_cancel already taken by /stop - let snapshot = turn_controller.snapshot(); - if let Some(partial) = partial_assistant_with_pending_deliveries( - &snapshot, - CompletionStatus::Cancelled, + persist_cancelled_turn( + &turn_controller, + &session, take_pending_turn_deliveries(&pending_turn_deliveries), - ) { - turn_controller.begin_finalizing(); - match append_persisted_messages(&session, vec![partial]).await { - Ok(()) => { - turn_controller.cancel(Some("stopped by user".to_string())); - } - Err(error) => { - tracing::error!(error = %error, "Failed to persist cancelled partial turn"); - turn_controller.fail(format!( - "failed to persist cancelled turn: {error}" - )); - } - } - } else { - turn_controller.cancel(Some("stopped by user".to_string())); - } + ) + .await; } } @@ -3819,7 +4368,7 @@ fn spawn_agent_worker( // processed by a subsequent Turn instead of being lost. // `/stop` calls `close_and_take_pending` itself, therefore // its intentionally discarded inputs do not reach here. - prepend_pending_steering( + let released = prepend_pending_steering( &steering, &recovery, &mut local_tasks, @@ -3830,6 +4379,12 @@ fn spawn_agent_worker( ); // Clean up + if !released.is_empty() { + let storage = { session.lock().await.storage.clone() }; + if let Some(storage) = storage { + release_steer_leases(&storage, released).await; + } + } let mut guard = session.lock().await; if guard .active_turn_emitter @@ -3846,6 +4401,8 @@ fn spawn_agent_worker( active.emitter.deactivate(); } if guard.worker_generation == worker_gen { + guard.current_turn_token = None; + consecutive_user_turns = consecutive_user_turns.saturating_add(1); if local_tasks.is_empty() { guard.current_cancel = None; } else { @@ -3862,6 +4419,404 @@ fn spawn_agent_worker( }); } +/// Execute one durable inbox batch as a continuation Turn of the root Agent. +/// +/// The trigger message is hidden (never shown to clients), the tool set is +/// read-only, and the terminal persistence is atomic with event consumption: +/// a crash or failure releases the lease instead of losing the event. +/// Returns true when a continuation was processed (events consumed or +/// released for retry). +#[allow(clippy::too_many_arguments)] +async fn run_inbox_continuation( + session: &Arc>, + turn_delivery: &TurnDeliveryService, + execution_gate: &Arc, + lease: crate::storage::agent_inbox::InboxLease, + worker_gen: u64, + unified_str: &str, + inbox_max_attempts: i64, +) -> bool { + let event_ids: Vec = lease.events.iter().map(|event| event.id.clone()).collect(); + let exhausted = lease + .events + .iter() + .any(|event| event.attempt_count >= inbox_max_attempts); + + /// Release the lease for retry, or dead-letter events that exhausted + /// their delivery attempts so they become diagnosable instead of looping + /// forever. + async fn release_or_dead_letter( + storage: &StdArc, + token: &str, + event_ids: &[String], + exhausted: bool, + error: &str, + now: i64, + ) { + if exhausted { + let dead = storage + .dead_letter_leased_events(token, event_ids, error, now) + .await + .unwrap_or(0); + tracing::warn!( + dead_lettered = dead, + error, + "Inbox event dead-lettered after exhausting delivery attempts" + ); + } else { + let _ = storage + .release_inbox_lease(token, event_ids, 30_000, Some(error), now) + .await; + } + } + let token = lease.token.clone(); + let now = chrono::Utc::now().timestamp_millis(); + + // Validate the worker generation before doing any work; `/stop` may have + // replaced this worker while the lease was being claimed. + let (provider_config, provider, full_tools, history_base) = { + let guard = session.lock().await; + if guard.worker_generation != worker_gen || guard.archived_at.is_some() { + return false; + } + ( + guard.provider_config.clone(), + guard.provider.clone(), + guard.tools.clone(), + guard.get_history().to_vec(), + ) + }; + let Some(storage) = ({ session.lock().await.storage.clone() }) else { + return false; + }; + + // Read-only continuation registry: inspect results and read files, but + // never write, message, delegate or schedule. + let tools = { + let scoped = crate::tools::ToolRegistry::new(); + for name in crate::storage::agent_inbox::continuation_tool_names() { + if let Some(tool) = full_tools.get(name) { + scoped.register_raw(name.to_string(), tool); + } + } + Arc::new(scoped) + }; + let (controller, _emitter, _receiver) = + TurnController::start(unified_str.to_string(), uuid::Uuid::new_v4().to_string()); + let turn_id = controller.snapshot().id.0.clone(); + let turn_token = CancellationToken::new(); + // Make `/stop` cancel this continuation like any active Turn. + session.lock().await.current_turn_token = Some(turn_token.clone()); + let tool_context = ToolExecutionContext::for_session(unified_str) + .with_turn_id(turn_id.clone()) + .with_cancellation(turn_token.clone()) + .with_execution_gate(execution_gate.clone()); + let trigger = crate::storage::agent_inbox::build_continuation_trigger(&lease.events, now); + + let agent = AgentLoop::with_provider_and_tools( + provider.clone(), + tools.clone(), + 12, + provider_config.model_id.clone(), + provider_config.workspace_dir.clone(), + provider_config.input_types.clone(), + ) + .with_context_window(provider_config.token_limit); + + let mut history = history_base; + history.insert( + 0, + ChatMessage::system(crate::agent::system_prompt::build_system_prompt( + &provider_config.workspace_dir, + &provider_config.model_id, + &tools, + )), + ); + history.push(trigger.clone()); + + let result = tokio::select! { + result = agent.process_with_context(history, tool_context.clone()) => result, + _ = turn_token.cancelled() => Err(crate::agent::AgentError::Cancelled), + }; + + let usage = result.as_ref().ok().and_then(|result| result.usage.clone()); + let last_prompt_tokens = result.as_ref().ok().and_then(|result| { + result + .last_request_usage + .as_ref() + .map(|value| value.prompt_tokens) + }); + let emitted_messages = match result { + Ok(result) => result.emitted_messages, + Err(error) => { + tracing::warn!(session_id = %unified_str, error = %error, "Continuation agent failed; releasing lease"); + controller.fail(format!("continuation failed: {error}")); + release_or_dead_letter( + &storage, + &token, + &event_ids, + exhausted, + &error.to_string(), + now, + ) + .await; + return true; + } + }; + + // Persist atomically: hidden trigger + assistant/tool messages + usage + + // event consumption in one transaction. Failure rolls back the in-memory + // suffix and releases the lease for a retry. + let committed = { + let mut guard = session.lock().await; + if guard.worker_generation != worker_gen || guard.archived_at.is_some() { + drop(guard); + let _ = storage + .release_inbox_lease(&token, &event_ids, 5_000, Some("worker replaced"), now) + .await; + controller.cancel(Some( + "worker replaced before continuation commit".to_string(), + )); + return true; + } + let mut all_metas: Vec = Vec::new(); + let mut session_meta = None; + let mut added_ids = Vec::new(); + for message in std::iter::once(trigger).chain(emitted_messages) { + if let Some((_, _, meta, meta_session)) = + guard.add_message_in_memory_with_version(message, true, true) + { + added_ids.push(meta.id.clone()); + session_meta = Some(meta_session); + all_metas.push(meta); + } + } + let Some(session_meta) = session_meta else { + return true; + }; + let Some(trigger_meta) = all_metas.first().cloned() else { + return true; + }; + let usage_record = usage + .as_ref() + .map(|turn_usage| crate::storage::TurnUsageRecord { + session_id: unified_str.to_string(), + turn_id: turn_id.clone(), + provider: provider.name().to_string(), + model: provider.model_id().to_string(), + usage: turn_usage.clone(), + last_prompt_tokens: last_prompt_tokens.unwrap_or_default(), + created_at: now, + }); + match storage + .commit_continuation_turn( + unified_str, + &trigger_meta, + &all_metas[1..], + &session_meta, + usage_record.as_ref(), + &token, + &event_ids, + now, + ) + .await + { + Ok(committed) => committed, + Err(error) => { + guard.rollback_message_suffix_with_version(&added_ids, true); + controller.fail(format!("continuation persistence failed: {error}")); + release_or_dead_letter( + &storage, + &token, + &event_ids, + exhausted, + &error.to_string(), + now, + ) + .await; + return true; + } + } + }; + + controller.complete(usage); + session.lock().await.current_turn_token = None; + let delta = committed_turn_delta(unified_str, committed); + // Deliver through the session's own channel binding: the WebSocket/WebUI + // projection receives the committed history while external channel + // delivery stays out of the continuation path. When the channel + // declared a durable delivery context (thread/root identity), reuse it + // so a continuation lands in the same conversation thread; one-shot + // `reply_to` values are never reused. + if let Some(session_id) = UnifiedSessionId::parse(unified_str) { + let durable_context = storage + .get_session_delivery_context(unified_str) + .await + .unwrap_or(None); + let mut metadata = outbound_turn_metadata(unified_str, &HashMap::new()); + if let Some(context_json) = durable_context + && let Ok(context) = serde_json::from_str::>(&context_json) + { + metadata.extend(context); + } + let target = crate::channels::TurnTarget { + channel: session_id.channel, + chat_id: session_id.chat_id, + session_id: unified_str.to_string(), + reply_to: None, + metadata, + }; + let _ = turn_delivery.commit(&target, delta).await; + } + tracing::info!(session_id = %unified_str, events = event_ids.len(), "Continuation Turn committed"); + true +} + +#[async_trait::async_trait] +impl crate::agent::AgentInboxWakeTarget for SessionManager { + async fn wake_agent_inbox(&self, session_id: &str, revision: i64) { + let session = { self.inner.lock().await.sessions.get(session_id).cloned() }; + let Some(session) = session else { + // No live session: the event stays pending in SQLite and is + // claimed when the session is next opened. + return; + }; + { + let guard = session.lock().await; + if guard.archived_at.is_some() { + return; + } + let has_active_turn = guard.active_turn_emitter.is_some(); + drop(guard); + // A live Turn accepts steer events through the two-phase + // admission; everything that cannot be admitted stays pending + // for the queue lane. + if has_active_turn { + self.try_steer_inbox_events(&session, session_id, revision) + .await; + } + } + let mut guard = session.lock().await; + // Share the worker creation path with user enqueue so a wake can + // start a worker for an idle session. + if guard.agent_tx.is_none() || guard.agent_tx.as_ref().is_some_and(|tx| tx.is_closed()) { + guard.agent_tx = None; + guard.worker_generation = guard.worker_generation.wrapping_add(1); + let generation = guard.worker_generation; + let (tx, rx) = mpsc::channel(SESSION_QUEUE_CAPACITY); + guard.agent_tx = Some(tx); + drop(guard); + spawn_agent_worker( + rx, + session.clone(), + self.worker_deps(), + generation, + session_id.to_string(), + ); + let _ = session.lock().await.agent_inbox_wake.send_replace(revision); + return; + } + let _ = guard.agent_inbox_wake.send_replace(revision); + } +} + +impl SessionManager { + /// Two-phase steer admission (design §15.3): claim `pending → leased` + /// without the Session lock, reserve agent-lane capacity in the active + /// Turn mailbox, persist `leased → admitted` with the Turn id, and only + /// then activate the reservation into the drainable queue. Any failure + /// releases the lease so the event degrades reliably to the queue lane; + /// a lost release is covered by lease expiry. + async fn try_steer_inbox_events( + &self, + session: &Arc>, + session_id: &str, + revision: i64, + ) { + let storage = { session.lock().await.storage.clone() }; + let Some(storage) = storage else { return }; + let now = chrono::Utc::now().timestamp_millis(); + let Ok(Some(lease)) = crate::storage::Storage::claim_inbox_batch( + &storage, + session_id, + now, + 60_000, + 8, + 32 * 1024, + Some(crate::storage::agent_inbox::AgentEventDelivery::Steer), + ) + .await + else { + return; + }; + let token = lease.token.clone(); + + // Step 2: reserve under the Session lock for the current accepting + // Turn. A closed/full mailbox rejects the whole batch; the events + // stay pending for the queue lane. + let (turn_id, mailbox) = { + let guard = session.lock().await; + let Some(active) = guard.active_turn_emitter.as_ref() else { + drop(guard); + let leases: Vec<_> = lease + .events + .iter() + .map(|event| (event.id.clone(), token.clone())) + .collect(); + release_steer_leases(&storage, leases).await; + let _ = session.lock().await.agent_inbox_wake.send_replace(revision); + return; + }; + (active.turn_id.clone(), active.steering.clone()) + }; + let mut rejected = Vec::new(); + let mut reserved = Vec::new(); + for event in &lease.events { + let input = steer_input_from_event(event, now); + match mailbox.try_reserve_steer(input, token.clone()) { + Ok(()) => reserved.push(event.id.clone()), + Err(_) => rejected.push((event.id.clone(), token.clone())), + } + } + if !reserved.is_empty() { + release_steer_leases(&storage, rejected).await; + rejected = Vec::new(); + // Step 3: durable admission without the Session lock. + let mut admitted = Vec::new(); + for event_id in &reserved { + if storage + .admit_inbox_event(event_id, &token, &turn_id, now) + .await + .unwrap_or(false) + { + admitted.push((event_id.clone(), token.clone())); + } else { + rejected.push((event_id.clone(), token.clone())); + } + } + // Step 4: activate only when the same Turn/generation is still + // accepting; otherwise release the admitted events back to + // `pending` and wake the queue lane. + let still_active = { + let guard = session.lock().await; + guard + .active_turn_emitter + .as_ref() + .is_some_and(|active| active.turn_id == turn_id && !active.steering.is_closed()) + }; + if still_active { + mailbox.activate_reserved(); + } else { + rejected.extend(admitted); + release_steer_leases(&storage, rejected).await; + let _ = session.lock().await.agent_inbox_wake.send_replace(revision); + } + } else { + release_steer_leases(&storage, rejected).await; + } + } +} + impl SessionManager { /// /// Runs in a stateless manner: no session creation, no history persistence. @@ -4023,7 +4978,7 @@ mod slash_command_tests { AgentTask, SLASH_COMMANDS, pop_lowest_sequence, prepend_pending_steering, resolve_slash_command, }; - use crate::agent::steering::{SteeringMailbox, SteeringPushError}; + use crate::agent::steering::{SteeringPushError, TurnInput, TurnMailbox}; use crate::bus::{ChannelContext, ChatMessage, MediaItem}; use std::collections::{HashMap, VecDeque}; use std::sync::{Arc, Mutex}; @@ -4064,17 +5019,20 @@ mod slash_command_tests { #[test] fn steering_capacity_error_is_reliable_fallback_signal() { - let mailbox = SteeringMailbox::with_limits(1, usize::MAX); - mailbox.try_push(ChatMessage::user("first")).unwrap(); - let error = mailbox.try_push(ChatMessage::user("second")).unwrap_err(); - assert!(error.is_full()); - assert!(matches!(error, SteeringPushError::Full(_))); + let mailbox = TurnMailbox::with_limits(1, usize::MAX, 8, usize::MAX); + mailbox + .try_push_user(TurnInput::user("first", "first", Vec::new(), None, 1_000)) + .unwrap(); + let error = mailbox + .try_push_user(TurnInput::user("second", "second", Vec::new(), None, 1_000)) + .unwrap_err(); + assert!(matches!(error, SteeringPushError::Full)); assert_eq!(mailbox.len(), 1); } #[test] fn terminal_pending_steering_keeps_original_route_and_media() { - let mailbox = SteeringMailbox::new_shared(); + let mailbox = TurnMailbox::new_shared(); let recovery = Arc::new(Mutex::new(HashMap::new())); let mut message = ChatMessage::user("inspect the upload"); message.id = "client-steer-1".to_string(); @@ -4094,13 +5052,22 @@ mod slash_command_tests { channel_context: ChannelContext { reply_to: Some("reply-1".to_string()), private: HashMap::from([(String::from("thread"), String::from("root-1"))]), + durable_private: HashMap::new(), }, }; recovery .lock() .unwrap() .insert(message.id.clone(), original); - mailbox.try_push(message).unwrap(); + mailbox + .try_push_user(TurnInput::user( + message.id.clone(), + message.content.clone(), + message.media_refs.clone(), + message.source.clone(), + message.timestamp, + )) + .unwrap(); let mut local_tasks = VecDeque::new(); prepend_pending_steering( @@ -4131,7 +5098,7 @@ mod slash_command_tests { #[test] fn pending_and_channel_tasks_are_selected_by_admission_sequence() { - let mailbox = SteeringMailbox::new_shared(); + let mailbox = TurnMailbox::new_shared(); let recovery = Arc::new(Mutex::new(HashMap::new())); let mut pending = ChatMessage::user("pending"); pending.id = "pending-id".to_string(); @@ -4150,7 +5117,15 @@ mod slash_command_tests { .lock() .unwrap() .insert(pending.id.clone(), template.clone()); - mailbox.try_push(pending).unwrap(); + mailbox + .try_push_user(TurnInput::user( + pending.id.clone(), + pending.content.clone(), + Vec::new(), + None, + pending.timestamp, + )) + .unwrap(); let mut local_tasks = VecDeque::from([ AgentTask { @@ -4178,8 +5153,10 @@ mod slash_command_tests { #[test] fn stop_style_close_discards_pending_steering() { - let mailbox = SteeringMailbox::new_shared(); - mailbox.try_push(ChatMessage::user("discard me")).unwrap(); + let mailbox = TurnMailbox::new_shared(); + mailbox + .try_push_user(TurnInput::user("d", "discard me", Vec::new(), None, 1_000)) + .unwrap(); assert_eq!(mailbox.close_and_take_pending().len(), 1); assert!(mailbox.is_closed()); assert!(mailbox.take_pending().is_empty()); diff --git a/src/skills/mod.rs b/src/skills/mod.rs index 8f352a4..2d0f2d5 100644 --- a/src/skills/mod.rs +++ b/src/skills/mod.rs @@ -368,6 +368,29 @@ impl SkillsLoader { prompt } + /// Build the minimal prompt exposed to a named Agent. The Agent can only + /// discover the allowlisted skills through its scoped get_skill tool. + pub fn build_scoped_skills_prompt(&self, allowed: &[String]) -> String { + if allowed.is_empty() { + return String::new(); + } + let allowed: std::collections::HashSet<_> = allowed.iter().map(String::as_str).collect(); + let skills: Vec<_> = self + .get_loaded_skills() + .into_iter() + .filter(|skill| allowed.contains(skill.name.as_str())) + .collect(); + if skills.is_empty() { + return String::new(); + } + let mut prompt = String::from("## 可用 Skills\n\n"); + for skill in skills { + prompt.push_str(&format!("- **{}**: {}\n", skill.name, skill.description)); + } + prompt.push_str("\n需要详细说明时，使用 `get_skill` 读取上述 allowlist 中的 skill。"); + prompt + } + fn sort_skills(mut skills: Vec) -> Vec { skills.sort_by(|a, b| { b.always diff --git a/src/storage/agent_inbox.rs b/src/storage/agent_inbox.rs new file mode 100644 index 0000000..9337e41 --- /dev/null +++ b/src/storage/agent_inbox.rs @@ -0,0 +1,2031 @@ +use sqlx::Row; + +use super::StorageError; +use crate::bus::{ClientVisibility, TurnOrigin}; + +/// Event kinds materialized in the durable inbox. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AgentEventType { + Signal, + Completion, + GroupCompletion, +} + +impl AgentEventType { + pub fn as_str(&self) -> &'static str { + match self { + Self::Signal => "signal", + Self::Completion => "completion", + Self::GroupCompletion => "group_completion", + } + } + + pub fn parse(value: &str) -> Result { + match value { + "signal" => Ok(Self::Signal), + "completion" => Ok(Self::Completion), + "group_completion" => Ok(Self::GroupCompletion), + other => Err(StorageError::Migration(format!( + "corrupt agent event type '{other}'" + ))), + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AgentEventDelivery { + Queue, + Steer, +} + +impl AgentEventDelivery { + pub fn as_str(&self) -> &'static str { + match self { + Self::Queue => "queue", + Self::Steer => "steer", + } + } + + pub fn parse(value: &str) -> Result { + match value { + "queue" => Ok(Self::Queue), + "steer" => Ok(Self::Steer), + other => Err(StorageError::Migration(format!( + "corrupt agent event delivery '{other}'" + ))), + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AgentEventStatus { + Pending, + Leased, + Admitted, + Consumed, + Superseded, + DeadLetter, +} + +impl AgentEventStatus { + pub fn as_str(&self) -> &'static str { + match self { + Self::Pending => "pending", + Self::Leased => "leased", + Self::Admitted => "admitted", + Self::Consumed => "consumed", + Self::Superseded => "superseded", + Self::DeadLetter => "dead_letter", + } + } + + pub fn parse(value: &str) -> Result { + match value { + "pending" => Ok(Self::Pending), + "leased" => Ok(Self::Leased), + "admitted" => Ok(Self::Admitted), + "consumed" => Ok(Self::Consumed), + "superseded" => Ok(Self::Superseded), + "dead_letter" => Ok(Self::DeadLetter), + other => Err(StorageError::Migration(format!( + "corrupt agent event status '{other}'" + ))), + } + } +} + +#[derive(Debug, Clone)] +pub struct AgentInboxEventRecord { + pub id: String, + pub root_session_id: String, + pub scope_kind: String, + pub scope_id: String, + pub run_id: Option, + pub group_id: Option, + pub event_type: AgentEventType, + pub event_key: String, + pub delivery: AgentEventDelivery, + pub requires_continuation: bool, + pub severity: Option, + pub payload_json: String, + pub status: AgentEventStatus, + pub attempt_count: i64, + pub lease_token: Option, + pub lease_until: Option, + pub next_attempt_at: Option, + pub admitted_turn_id: Option, + pub last_error: Option, + pub revision: i64, + pub created_at: i64, + pub consumed_at: Option, + pub superseded_at: Option, + pub dead_lettered_at: Option, + pub fallback_notified_at: Option, + pub fallback_suppressed_reason: Option, +} + +#[derive(Debug, Clone)] +pub struct NewInboxEvent { + pub id: String, + pub root_session_id: String, + pub scope_kind: String, + pub scope_id: String, + pub run_id: Option, + pub group_id: Option, + pub event_type: AgentEventType, + pub event_key: String, + pub delivery: AgentEventDelivery, + pub requires_continuation: bool, + pub severity: Option, + pub payload_json: String, +} + +/// A claimed batch of due events. The lease token owns the transition back +/// to `pending` on failure; only process crashes rely on lease expiry. +#[derive(Debug)] +pub struct InboxLease { + pub token: String, + pub events: Vec, +} + +/// Steer events admitted to a user Turn; consumed atomically with the Turn's +/// durable commit. +#[derive(Debug, Clone, Default)] +pub struct SteerConsumption { + pub event_ids: Vec, + pub admitted_turn_id: String, +} + +#[derive(Debug, Default)] +pub struct RecoveryReport { + pub interrupted_runs: usize, + pub completion_events_generated: usize, + pub leases_expired: usize, + pub dead_lettered: usize, + pub sessions_reconciled: usize, + pub groups_converged: usize, +} + +const EVENT_COLUMNS: &str = "id, root_session_id, scope_kind, scope_id, run_id, group_id, \ + event_type, event_key, delivery, requires_continuation, severity, payload_json, status, \ + attempt_count, lease_token, lease_until, next_attempt_at, admitted_turn_id, last_error, \ + revision, created_at, consumed_at, superseded_at, dead_lettered_at, fallback_notified_at, \ + fallback_suppressed_reason"; + +fn event_record_from_row( + row: &sqlx::sqlite::SqliteRow, +) -> Result { + Ok(AgentInboxEventRecord { + id: row.get("id"), + root_session_id: row.get("root_session_id"), + scope_kind: row.get("scope_kind"), + scope_id: row.get("scope_id"), + run_id: row.get("run_id"), + group_id: row.get("group_id"), + event_type: AgentEventType::parse(row.get::<&str, _>("event_type"))?, + event_key: row.get("event_key"), + delivery: AgentEventDelivery::parse(row.get::<&str, _>("delivery"))?, + requires_continuation: row.get::("requires_continuation") != 0, + severity: row.get("severity"), + payload_json: row.get("payload_json"), + status: AgentEventStatus::parse(row.get::<&str, _>("status"))?, + attempt_count: row.get("attempt_count"), + lease_token: row.get("lease_token"), + lease_until: row.get("lease_until"), + next_attempt_at: row.get("next_attempt_at"), + admitted_turn_id: row.get("admitted_turn_id"), + last_error: row.get("last_error"), + revision: row.get("revision"), + created_at: row.get("created_at"), + consumed_at: row.get("consumed_at"), + superseded_at: row.get("superseded_at"), + dead_lettered_at: row.get("dead_lettered_at"), + fallback_notified_at: row.get("fallback_notified_at"), + fallback_suppressed_reason: row.get("fallback_suppressed_reason"), + }) +} + +impl super::Storage { + /// Ensure the authoritative inbox capacity/revision row exists. + pub async fn ensure_agent_session_state( + &self, + root_session_id: &str, + now: i64, + ) -> Result<(), StorageError> { + sqlx::query( + "INSERT INTO agent_session_state (root_session_id, revision, pending_event_count, \ + reserved_completion_slots, updated_at) VALUES (?, 0, 0, 0, ?) \ + ON CONFLICT(root_session_id) DO NOTHING", + ) + .bind(root_session_id) + .bind(now) + .execute(self.pool()) + .await?; + Ok(()) + } + + /// Reserve `count` completion slots for background runs whose completion + /// is guaranteed even when the inbox is full. Returns the new revision, + /// or `None` when the capacity check fails (nothing is reserved). + pub async fn reserve_completion_slots( + &self, + root_session_id: &str, + count: i64, + max_pending_and_reserved: i64, + now: i64, + ) -> Result, StorageError> { + let mut tx = self.pool.begin().await?; + ensure_agent_session_state_tx(&mut tx, root_session_id, now).await?; + let revision: Option = sqlx::query_scalar( + "UPDATE agent_session_state \ + SET reserved_completion_slots = reserved_completion_slots + ?, \ + revision = revision + 1, updated_at = ? \ + WHERE root_session_id = ? \ + AND pending_event_count + reserved_completion_slots + ? <= ? \ + RETURNING revision", + ) + .bind(count) + .bind(now) + .bind(root_session_id) + .bind(count) + .bind(max_pending_and_reserved) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + Ok(revision) + } + + /// Release reservation slots that are no longer owned (compensation after + /// spawn rejection, session archive, recovery). Never negative. + pub async fn release_completion_slots( + &self, + root_session_id: &str, + count: i64, + now: i64, + ) -> Result<(), StorageError> { + sqlx::query( + "UPDATE agent_session_state \ + SET reserved_completion_slots = MAX(reserved_completion_slots - ?, 0), \ + revision = revision + 1, updated_at = ? \ + WHERE root_session_id = ?", + ) + .bind(count) + .bind(now) + .bind(root_session_id) + .execute(self.pool()) + .await?; + Ok(()) + } + + /// Insert a signal event. Capacity is enforced conditionally; the slot + /// is accounted against `pending + reserved`, so background completion + /// reservations are never displaced by signals. Returns `None` when the + /// inbox is full. + /// Persist a `signal` event under the session inbox capacity. A signal + /// whose dedupe key already exists (same scope/type/key, i.e. the same + /// cooldown window) returns the existing event with `deduplicated=true` + /// instead of consuming capacity or emitting a second event. + pub async fn insert_agent_signal( + &self, + event: &NewInboxEvent, + max_pending_and_reserved: i64, + now: i64, + ) -> Result, StorageError> { + let mut tx = self.pool.begin().await?; + ensure_agent_session_state_tx(&mut tx, &event.root_session_id, now).await?; + if let Some(existing_id) = sqlx::query_scalar::<_, String>( + "SELECT id FROM agent_inbox_events \ + WHERE scope_kind = ? AND scope_id = ? AND event_type = ? AND event_key = ?", + ) + .bind(&event.scope_kind) + .bind(&event.scope_id) + .bind(event.event_type.as_str()) + .bind(&event.event_key) + .fetch_optional(&mut *tx) + .await? + { + tx.commit().await?; + let record = self.get_agent_inbox_event(&existing_id).await?; + return Ok(record.map(|record| (record, true))); + } + let revision: Option = sqlx::query_scalar( + "UPDATE agent_session_state \ + SET pending_event_count = pending_event_count + 1, \ + revision = revision + 1, updated_at = ? \ + WHERE root_session_id = ? \ + AND pending_event_count + reserved_completion_slots + 1 <= ? \ + RETURNING revision", + ) + .bind(now) + .bind(&event.root_session_id) + .bind(max_pending_and_reserved) + .fetch_optional(&mut *tx) + .await?; + let Some(revision) = revision else { + return Ok(None); + }; + insert_event_tx(&mut tx, event, revision, now).await?; + tx.commit().await?; + let record = self.get_agent_inbox_event(&event.id).await?; + Ok(record.map(|record| (record, false))) + } + + pub async fn get_agent_inbox_event( + &self, + event_id: &str, + ) -> Result, StorageError> { + let row = sqlx::query(sqlx::AssertSqlSafe(format!( + "SELECT {EVENT_COLUMNS} FROM agent_inbox_events WHERE id = ?" + ))) + .bind(event_id) + .fetch_optional(self.pool()) + .await?; + match row { + Some(row) => Ok(Some(event_record_from_row(&row)?)), + None => Ok(None), + } + } + + pub async fn list_agent_inbox_events( + &self, + root_session_id: &str, + limit: i64, + ) -> Result, StorageError> { + let rows = sqlx::query(sqlx::AssertSqlSafe(format!( + "SELECT {EVENT_COLUMNS} FROM agent_inbox_events \ + WHERE root_session_id = ? ORDER BY created_at DESC, id DESC LIMIT ?" + ))) + .bind(root_session_id) + .bind(limit.clamp(1, 200)) + .fetch_all(self.pool()) + .await?; + rows.iter().map(event_record_from_row).collect() + } + + /// Inbox events of one run, newest first (audit/projection queries). + pub async fn list_agent_inbox_events_for_run( + &self, + run_id: &str, + limit: i64, + ) -> Result, StorageError> { + let rows = sqlx::query(sqlx::AssertSqlSafe(format!( + "SELECT {EVENT_COLUMNS} FROM agent_inbox_events \ + WHERE run_id = ? ORDER BY created_at DESC, id DESC LIMIT ?" + ))) + .bind(run_id) + .bind(limit.clamp(1, 200)) + .fetch_all(self.pool()) + .await?; + rows.iter().map(event_record_from_row).collect() + } + + /// Current client-visible revision of a session's Agent projection. + /// `0` when the session never had Agent state. + pub async fn get_session_agent_revision( + &self, + root_session_id: &str, + ) -> Result { + let revision: Option = sqlx::query_scalar( + "SELECT revision FROM agent_session_state WHERE root_session_id = ?", + ) + .bind(root_session_id) + .fetch_optional(self.pool()) + .await?; + Ok(revision.unwrap_or(0)) + } + + /// Oldest `created_at` among currently due pending events, used by the + /// fairness policy to bound how long an event can wait behind user input. + /// `None` when no event is currently due. + pub async fn oldest_pending_due( + &self, + root_session_id: &str, + now: i64, + ) -> Result, StorageError> { + let value: Option> = sqlx::query_scalar::<_, Option>( + "SELECT MIN(created_at) FROM agent_inbox_events \ + WHERE root_session_id = ? AND status = 'pending' AND next_attempt_at <= ?", + ) + .bind(root_session_id) + .bind(now) + .fetch_optional(self.pool()) + .await?; + Ok(value.flatten()) + } + + /// Earliest `next_attempt_at` among pending events — the deadline at + /// which the worker must re-claim after a release backoff. `None` when + /// nothing is pending. + pub async fn next_pending_due_at( + &self, + root_session_id: &str, + ) -> Result, StorageError> { + let value: Option> = sqlx::query_scalar::<_, Option>( + "SELECT MIN(next_attempt_at) FROM agent_inbox_events \ + WHERE root_session_id = ? AND status = 'pending'", + ) + .bind(root_session_id) + .fetch_optional(self.pool()) + .await?; + Ok(value.flatten()) + } + + /// Dead-letter leased/admitted events that exhausted their delivery + /// attempts during normal operation (not only at recovery), reducing the + /// pending count by the number actually dead-lettered. + pub async fn dead_letter_leased_events( + &self, + lease_token: &str, + event_ids: &[String], + reason: &str, + now: i64, + ) -> Result { + if event_ids.is_empty() { + return Ok(0); + } + let mut tx = self.pool.begin().await?; + let mut count = 0i64; + for id in event_ids { + count += sqlx::query( + "UPDATE agent_inbox_events \ + SET status = 'dead_letter', dead_lettered_at = ?, last_error = ?, \ + lease_token = NULL, lease_until = NULL, updated_at = ? \ + WHERE id = ? AND lease_token = ? AND status IN ('leased', 'admitted')", + ) + .bind(now) + .bind(reason) + .bind(now) + .bind(id) + .bind(lease_token) + .execute(&mut *tx) + .await? + .rows_affected() as i64; + } + if count > 0 { + let session: Option = + sqlx::query_scalar("SELECT root_session_id FROM agent_inbox_events WHERE id = ?") + .bind(&event_ids[0]) + .fetch_optional(&mut *tx) + .await?; + if let Some(session) = session { + sqlx::query( + "UPDATE agent_session_state \ + SET pending_event_count = MAX(pending_event_count - ?, 0), \ + revision = revision + 1, updated_at = ? \ + WHERE root_session_id = ?", + ) + .bind(count) + .bind(now) + .bind(session) + .execute(&mut *tx) + .await?; + } + } + tx.commit().await?; + Ok(count as usize) + } + + /// Sessions that currently have due pending events, for the merged + /// post-recovery wake. + pub async fn sessions_with_due_events(&self, now: i64) -> Result, StorageError> { + let ids: Vec = sqlx::query_scalar( + "SELECT DISTINCT root_session_id FROM agent_inbox_events \ + WHERE status = 'pending' AND next_attempt_at <= ?", + ) + .bind(now) + .fetch_all(self.pool()) + .await?; + Ok(ids) + } + + /// Claim the oldest due events for a session as one leased batch, bounded + /// by `max_events` and `max_payload_bytes`. Returns `None` when nothing + /// is due. The lease token must be presented to release or consume. + pub async fn claim_inbox_batch( + &self, + root_session_id: &str, + now: i64, + lease_ms: i64, + max_events: usize, + max_payload_bytes: usize, + delivery: Option, + ) -> Result, StorageError> { + let candidates = match delivery { + None => { + sqlx::query( + "SELECT id, payload_json FROM agent_inbox_events \ + WHERE root_session_id = ? AND status = 'pending' AND next_attempt_at <= ? \ + ORDER BY created_at ASC, id ASC LIMIT ?", + ) + .bind(root_session_id) + .bind(now) + .bind(max_events as i64) + .fetch_all(self.pool()) + .await? + } + Some(delivery) => { + sqlx::query( + "SELECT id, payload_json FROM agent_inbox_events \ + WHERE root_session_id = ? AND status = 'pending' AND next_attempt_at <= ? \ + AND delivery = ? \ + ORDER BY created_at ASC, id ASC LIMIT ?", + ) + .bind(root_session_id) + .bind(now) + .bind(delivery.as_str()) + .bind(max_events as i64) + .fetch_all(self.pool()) + .await? + } + }; + + let mut chosen = Vec::new(); + let mut bytes = 0usize; + for row in candidates { + let id: String = row.get("id"); + let payload: String = row.get("payload_json"); + if !chosen.is_empty() && bytes.saturating_add(payload.len()) > max_payload_bytes { + break; + } + bytes = bytes.saturating_add(payload.len()); + chosen.push(id); + } + if chosen.is_empty() { + return Ok(None); + } + + let token = uuid::Uuid::new_v4().to_string(); + let lease_until = now + lease_ms; + let mut tx = self.pool.begin().await?; + let mut events = Vec::new(); + for id in &chosen { + let updated = sqlx::query( + "UPDATE agent_inbox_events \ + SET status = 'leased', lease_token = ?, lease_until = ?, \ + attempt_count = attempt_count + 1, updated_at = ? \ + WHERE id = ? AND status = 'pending'", + ) + .bind(&token) + .bind(lease_until) + .bind(now) + .bind(id) + .execute(&mut *tx) + .await? + .rows_affected(); + if updated == 1 { + let row = sqlx::query(sqlx::AssertSqlSafe(format!( + "SELECT {EVENT_COLUMNS} FROM agent_inbox_events WHERE id = ?" + ))) + .bind(id) + .fetch_one(&mut *tx) + .await?; + events.push(event_record_from_row(&row)?); + } + } + tx.commit().await?; + if events.is_empty() { + return Ok(None); + } + Ok(Some(InboxLease { token, events })) + } + + /// Transition a leased event into `admitted` for a specific Turn. Used + /// by the steer lane; the queue lane consumes directly. + pub async fn admit_inbox_event( + &self, + event_id: &str, + lease_token: &str, + turn_id: &str, + now: i64, + ) -> Result { + let updated = sqlx::query( + "UPDATE agent_inbox_events SET status = 'admitted', admitted_turn_id = ?, updated_at = ? \ + WHERE id = ? AND lease_token = ? AND status = 'leased'", + ) + .bind(turn_id) + .bind(now) + .bind(event_id) + .bind(lease_token) + .execute(self.pool()) + .await? + .rows_affected(); + Ok(updated == 1) + } + + /// Return leased/admitted events to `pending` with a retry backoff. The + /// lease token prevents double release of events owned by another worker. + pub async fn release_inbox_lease( + &self, + lease_token: &str, + event_ids: &[String], + retry_after_ms: i64, + error: Option<&str>, + now: i64, + ) -> Result<(), StorageError> { + if event_ids.is_empty() { + return Ok(()); + } + let mut tx = self.pool.begin().await?; + for id in event_ids { + sqlx::query( + "UPDATE agent_inbox_events \ + SET status = 'pending', lease_token = NULL, lease_until = NULL, \ + next_attempt_at = ?, last_error = ?, updated_at = ? \ + WHERE id = ? AND lease_token = ? AND status IN ('leased', 'admitted')", + ) + .bind(now + retry_after_ms) + .bind(error) + .bind(now) + .bind(id) + .bind(lease_token) + .execute(&mut *tx) + .await?; + } + tx.commit().await?; + Ok(()) + } + + /// Supersede unconsumed signal events of a run (explicit cancel). The + /// pending count is reduced by the number of events actually superseded. + pub async fn supersede_agent_events( + &self, + run_id: &str, + event_type: AgentEventType, + now: i64, + ) -> Result { + let mut tx = self.pool.begin().await?; + let ids: Vec = sqlx::query_scalar( + "SELECT id FROM agent_inbox_events \ + WHERE run_id = ? AND event_type = ? AND status IN ('pending', 'leased', 'admitted')", + ) + .bind(run_id) + .bind(event_type.as_str()) + .fetch_all(&mut *tx) + .await?; + if ids.is_empty() { + return Ok(0); + } + let mut count = 0i64; + for id in &ids { + count += sqlx::query( + "UPDATE agent_inbox_events SET status = 'superseded', superseded_at = ?, updated_at = ? \ + WHERE id = ? AND status IN ('pending', 'leased', 'admitted')", + ) + .bind(now) + .bind(now) + .bind(id) + .execute(&mut *tx) + .await? + .rows_affected() as i64; + } + if count > 0 { + let session: Option = + sqlx::query_scalar("SELECT root_session_id FROM agent_inbox_events WHERE id = ?") + .bind(&ids[0]) + .fetch_optional(&mut *tx) + .await?; + if let Some(session) = session { + sqlx::query( + "UPDATE agent_session_state \ + SET pending_event_count = MAX(pending_event_count - ?, 0), \ + revision = revision + 1, updated_at = ? \ + WHERE root_session_id = ?", + ) + .bind(count) + .bind(now) + .bind(session) + .execute(&mut *tx) + .await?; + } + } + tx.commit().await?; + Ok(count as usize) + } + + /// Dead-letter every unconsumed event of a session (archive/delete) and + /// zero the pending count. No continuation is started afterwards. + pub async fn dead_letter_agent_events( + &self, + root_session_id: &str, + reason: &str, + now: i64, + ) -> Result { + let mut tx = self.pool.begin().await?; + let updated = sqlx::query( + "UPDATE agent_inbox_events \ + SET status = 'dead_letter', dead_lettered_at = ?, \ + fallback_suppressed_reason = ?, updated_at = ? \ + WHERE root_session_id = ? AND status IN ('pending', 'leased', 'admitted')", + ) + .bind(now) + .bind(reason) + .bind(now) + .bind(root_session_id) + .execute(&mut *tx) + .await? + .rows_affected(); + sqlx::query( + "UPDATE agent_session_state SET pending_event_count = 0, \ + revision = revision + 1, updated_at = ? WHERE root_session_id = ?", + ) + .bind(now) + .bind(root_session_id) + .execute(&mut *tx) + .await?; + tx.commit().await?; + Ok(updated as usize) + } + + /// Atomically commit a continuation Turn: hidden trigger, assistant/tool + /// messages, usage, and consumption of the leased events in one + /// transaction. The `sessions` row and `agent_session_state` counts are + /// updated in the same commit so a client never observes a partially + /// materialized continuation. + #[allow(clippy::too_many_arguments)] + pub async fn commit_continuation_turn( + &self, + session_id: &str, + trigger: &crate::storage::message::MessageMeta, + messages: &[crate::storage::message::MessageMeta], + meta: &crate::storage::session::SessionMeta, + usage: Option<&crate::storage::TurnUsageRecord>, + lease_token: &str, + event_ids: &[String], + now: i64, + ) -> Result, StorageError> { + let mut tx = self.pool.begin().await?; + super::insert_message_query(session_id, trigger) + .execute(&mut *tx) + .await?; + for message in messages { + super::insert_message_query(session_id, message) + .execute(&mut *tx) + .await?; + } + sqlx::query( + r#" + INSERT INTO sessions (id, channel, chat_id, dialog_id, title, created_at, last_active_at, message_count, routing_info, archived_at, deleted_at, last_consolidated_at, last_compressed_message_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(id) DO UPDATE SET + title = excluded.title, + last_active_at = excluded.last_active_at, + message_count = excluded.message_count, + routing_info = excluded.routing_info, + archived_at = excluded.archived_at, + deleted_at = excluded.deleted_at, + last_consolidated_at = excluded.last_consolidated_at, + last_compressed_message_at = excluded.last_compressed_message_at + "#, + ) + .bind(&meta.id) + .bind(&meta.channel) + .bind(&meta.chat_id) + .bind(&meta.dialog_id) + .bind(&meta.title) + .bind(meta.created_at) + .bind(meta.last_active_at) + .bind(meta.message_count) + .bind(&meta.routing_info) + .bind(meta.archived_at) + .bind(meta.deleted_at) + .bind(meta.last_consolidated_at) + .bind(meta.last_compressed_message_at) + .execute(&mut *tx) + .await?; + + if let Some(usage) = usage { + let request_count = messages + .iter() + .filter_map(|message| message.iteration) + .max() + .map_or(1_i64, |iteration| iteration.saturating_add(1)); + sqlx::query( + r#" + INSERT INTO session_turn_usage ( + turn_id, session_id, provider, model, prompt_tokens, + completion_tokens, total_tokens, cached_input_tokens, + request_count, last_prompt_tokens, created_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(turn_id) DO NOTHING + "#, + ) + .bind(&usage.turn_id) + .bind(&usage.session_id) + .bind(&usage.provider) + .bind(&usage.model) + .bind(i64::from(usage.usage.prompt_tokens)) + .bind(i64::from(usage.usage.completion_tokens)) + .bind(i64::from(usage.usage.total_tokens)) + .bind(usage.usage.cached_tokens.map(i64::from)) + .bind(request_count) + .bind(i64::from(usage.last_prompt_tokens)) + .bind(usage.created_at) + .execute(&mut *tx) + .await?; + } + + let consumed = event_ids.len() as i64; + for event_id in event_ids { + sqlx::query( + "UPDATE agent_inbox_events SET status = 'consumed', consumed_at = ?, updated_at = ? \ + WHERE id = ? AND lease_token = ? AND status IN ('leased', 'admitted')", + ) + .bind(now) + .bind(now) + .bind(event_id) + .bind(lease_token) + .execute(&mut *tx) + .await?; + } + sqlx::query( + "UPDATE agent_session_state \ + SET pending_event_count = MAX(pending_event_count - ?, 0), \ + revision = revision + 1, updated_at = ? \ + WHERE root_session_id = ?", + ) + .bind(consumed) + .bind(now) + .bind(session_id) + .execute(&mut *tx) + .await?; + + tx.commit().await?; + let mut committed = Vec::with_capacity(messages.len()); + for message in messages { + committed.push(message.clone()); + } + Ok(committed) + } + + /// Reconcile agent state after a restart or new runtime generation. + /// + /// 1. Old-generation queued/running/waiting_children runs become + /// `interrupted`; background runs with reserved completion slots get a + /// failure completion event (the reservation is converted). + /// 2. Expired leases return to `pending` with a backoff; attempts beyond + /// the maximum become `dead_letter`. + /// 3. Group counters are recomputed from their runs and finalized. + /// 4. Per-session capacity counters are reconciled with the rows. + pub async fn recover_agent_state( + &self, + active_generation: i64, + now: i64, + max_delivery_attempts: i64, + retry_backoff_ms: i64, + ) -> Result { + let mut report = RecoveryReport::default(); + let mut tx = self.pool.begin().await?; + + // 1. Interrupt runs of previous generations. + let interrupted: Vec<(String, String, i64)> = sqlx::query_as( + "SELECT id, root_session_id, completion_slot_reserved FROM agent_runs \ + WHERE runtime_generation != ? AND status IN ('queued', 'running', 'waiting_children')", + ) + .bind(active_generation) + .fetch_all(&mut *tx) + .await?; + for (run_id, session_id, reserved) in &interrupted { + let updated = sqlx::query( + "UPDATE agent_runs SET status = 'interrupted', error = ?, finished_at = ?, updated_at = ? \ + WHERE id = ? AND status IN ('queued', 'running', 'waiting_children')", + ) + .bind("interrupted by runtime generation handover") + .bind(now) + .bind(now) + .bind(run_id) + .execute(&mut *tx) + .await? + .rows_affected(); + if updated != 1 { + continue; + } + report.interrupted_runs += 1; + if *reserved != 0 + && let Some(revision) = convert_reservation_tx(&mut tx, session_id, now).await? + { + let event = NewInboxEvent { + id: uuid::Uuid::new_v4().to_string(), + root_session_id: session_id.clone(), + scope_kind: "run".to_string(), + scope_id: run_id.clone(), + run_id: Some(run_id.clone()), + group_id: None, + event_type: AgentEventType::Completion, + event_key: format!("interrupted:{run_id}"), + delivery: AgentEventDelivery::Queue, + requires_continuation: true, + severity: Some("error".to_string()), + payload_json: serde_json::json!({ + "status": "interrupted", + "error": "interrupted by runtime generation handover", + }) + .to_string(), + }; + insert_event_tx(&mut tx, &event, revision, now).await?; + report.completion_events_generated += 1; + } + } + + // 2. Expire stale leases. + let expired: Vec = sqlx::query_scalar( + "SELECT id FROM agent_inbox_events WHERE status = 'leased' AND lease_until <= ?", + ) + .bind(now) + .fetch_all(&mut *tx) + .await?; + for id in &expired { + let updated = sqlx::query( + "UPDATE agent_inbox_events SET attempt_count = attempt_count + 1, last_error = ?, updated_at = ? \ + WHERE id = ? AND status = 'leased'", + ) + .bind("lease expired") + .bind(now) + .bind(id) + .execute(&mut *tx) + .await? + .rows_affected(); + if updated != 1 { + continue; + } + report.leases_expired += 1; + let attempts: i64 = + sqlx::query_scalar("SELECT attempt_count FROM agent_inbox_events WHERE id = ?") + .bind(id) + .fetch_one(&mut *tx) + .await?; + if attempts >= max_delivery_attempts { + sqlx::query( + "UPDATE agent_inbox_events SET status = 'dead_letter', dead_lettered_at = ?, updated_at = ? \ + WHERE id = ? AND status = 'leased'", + ) + .bind(now) + .bind(now) + .bind(id) + .execute(&mut *tx) + .await?; + report.dead_lettered += 1; + } else { + sqlx::query( + "UPDATE agent_inbox_events SET status = 'pending', lease_token = NULL, lease_until = NULL, \ + next_attempt_at = ?, updated_at = ? \ + WHERE id = ? AND status = 'leased'", + ) + .bind(now + retry_backoff_ms * attempts) + .bind(now) + .bind(id) + .execute(&mut *tx) + .await?; + } + } + + // 3. Converge group counters from their runs. + let group_ids: Vec = sqlx::query_scalar( + "SELECT id FROM agent_run_groups WHERE status IN ('queued', 'running')", + ) + .fetch_all(&mut *tx) + .await?; + for group_id in &group_ids { + let (terminal, abnormal): (i64, i64) = sqlx::query_as( + "SELECT \ + COUNT(*) FILTER (WHERE status IN ('completed','failed','timed_out','cancelled','interrupted')), \ + COUNT(*) FILTER (WHERE status IN ('failed','timed_out','cancelled','interrupted')) \ + FROM agent_runs WHERE group_id = ?", + ) + .bind(group_id) + .fetch_one(&mut *tx) + .await?; + let expected: i64 = + sqlx::query_scalar("SELECT expected_runs FROM agent_run_groups WHERE id = ?") + .bind(group_id) + .fetch_one(&mut *tx) + .await?; + let status = if terminal >= expected && terminal > 0 { + if abnormal == 0 { + "completed" + } else if abnormal < expected { + "partial" + } else { + "failed" + } + } else { + "running" + }; + sqlx::query( + "UPDATE agent_run_groups SET terminal_runs = ?, abnormal_runs = ?, status = ?, \ + finished_at = CASE WHEN status IN ('completed','partial','failed','timed_out','cancelled','interrupted') THEN ? ELSE NULL END, \ + updated_at = ? WHERE id = ?", + ) + .bind(terminal) + .bind(abnormal) + .bind(status) + .bind(now) + .bind(now) + .bind(group_id) + .execute(&mut *tx) + .await?; + if status != "running" { + report.groups_converged += 1; + } + } + + // 4. Reconcile per-session capacity counters. + let sessions: Vec<(String, i64, i64)> = sqlx::query_as( + "SELECT root_session_id, \ + (SELECT COUNT(*) FROM agent_runs r WHERE r.root_session_id = s.root_session_id AND r.completion_slot_reserved = 1) \ + + (SELECT COUNT(*) FROM agent_run_groups g WHERE g.root_session_id = s.root_session_id AND g.completion_slot_reserved = 1), \ + (SELECT COUNT(*) FROM agent_inbox_events e WHERE e.root_session_id = s.root_session_id AND e.status IN ('pending','leased','admitted')) \ + FROM agent_session_state s", + ) + .fetch_all(&mut *tx) + .await?; + for (session_id, expected_reserved, expected_pending) in sessions { + let actual: (i64, i64) = sqlx::query_as( + "SELECT reserved_completion_slots, pending_event_count FROM agent_session_state WHERE root_session_id = ?", + ) + .bind(&session_id) + .fetch_one(&mut *tx) + .await?; + if actual.0 != expected_reserved || actual.1 != expected_pending { + sqlx::query( + "UPDATE agent_session_state SET reserved_completion_slots = ?, pending_event_count = ?, \ + revision = revision + 1, updated_at = ? WHERE root_session_id = ?", + ) + .bind(expected_reserved) + .bind(expected_pending) + .bind(now) + .bind(&session_id) + .execute(&mut *tx) + .await?; + report.sessions_reconciled += 1; + tracing::warn!( + session_id = %session_id, + before_reserved = actual.0, + before_pending = actual.1, + "agent session state counters diverged; reconciled" + ); + } + } + + tx.commit().await?; + Ok(report) + } +} + +async fn ensure_agent_session_state_tx( + tx: &mut sqlx::SqliteConnection, + root_session_id: &str, + now: i64, +) -> Result<(), StorageError> { + sqlx::query( + "INSERT INTO agent_session_state (root_session_id, revision, pending_event_count, \ + reserved_completion_slots, updated_at) VALUES (?, 0, 0, 0, ?) \ + ON CONFLICT(root_session_id) DO NOTHING", + ) + .bind(root_session_id) + .bind(now) + .execute(&mut *tx) + .await?; + Ok(()) +} + +/// Convert one reserved completion slot into a pending event slot. Returns +/// the new revision to stamp the event with. +async fn convert_reservation_tx( + tx: &mut sqlx::SqliteConnection, + root_session_id: &str, + now: i64, +) -> Result, StorageError> { + let revision: Option = sqlx::query_scalar( + "UPDATE agent_session_state \ + SET reserved_completion_slots = MAX(reserved_completion_slots - 1, 0), \ + pending_event_count = pending_event_count + 1, \ + revision = revision + 1, updated_at = ? \ + WHERE root_session_id = ? RETURNING revision", + ) + .bind(now) + .bind(root_session_id) + .fetch_optional(&mut *tx) + .await?; + Ok(revision) +} + +pub(crate) async fn insert_event_tx( + tx: &mut sqlx::SqliteConnection, + event: &NewInboxEvent, + revision: i64, + now: i64, +) -> Result<(), StorageError> { + sqlx::query( + "INSERT INTO agent_inbox_events (id, root_session_id, scope_kind, scope_id, run_id, \ + group_id, event_type, event_key, delivery, requires_continuation, severity, \ + payload_json, status, attempt_count, revision, next_attempt_at, created_at, updated_at) \ + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending', 0, ?, ?, ?, ?)", + ) + .bind(&event.id) + .bind(&event.root_session_id) + .bind(&event.scope_kind) + .bind(&event.scope_id) + .bind(&event.run_id) + .bind(&event.group_id) + .bind(event.event_type.as_str()) + .bind(&event.event_key) + .bind(event.delivery.as_str()) + .bind(i64::from(event.requires_continuation)) + .bind(&event.severity) + .bind(&event.payload_json) + .bind(revision) + .bind(now) + .bind(now) + .bind(now) + .execute(&mut *tx) + .await?; + Ok(()) +} + +/// Helper used by `commit_agent_terminal` to materialize a completion event +/// for a background run that reserved a slot. +pub(crate) async fn insert_completion_event_tx( + tx: &mut sqlx::SqliteConnection, + run_id: &str, + session_id: &str, + status: &str, + error: Option<&str>, + now: i64, +) -> Result<(), StorageError> { + let Some(revision) = convert_reservation_tx(tx, session_id, now).await? else { + return Err(StorageError::Conflict( + "background completion lost its reserved slot".to_string(), + )); + }; + let event = NewInboxEvent { + id: uuid::Uuid::new_v4().to_string(), + root_session_id: session_id.to_string(), + scope_kind: "run".to_string(), + scope_id: run_id.to_string(), + run_id: Some(run_id.to_string()), + group_id: None, + event_type: AgentEventType::Completion, + event_key: format!("completion:{run_id}"), + delivery: AgentEventDelivery::Queue, + requires_continuation: true, + severity: if error.is_some() { + Some("error".to_string()) + } else { + None + }, + payload_json: serde_json::json!({ + "status": status, + "error": error, + }) + .to_string(), + }; + insert_event_tx(tx, &event, revision, now).await +} + +/// Default trigger content for a continuation Turn. +pub fn build_continuation_trigger( + events: &[AgentInboxEventRecord], + now: i64, +) -> crate::bus::ChatMessage { + let mut content = String::from( + "后台 Agent 任务已经完成。请结合以下结果继续当前对话，向用户呈现最相关的部分；\ + 不要重复执行已经完成的工作。", + ); + for event in events { + content.push_str("\n\n- "); + content.push_str(&event.payload_json); + } + let mut message = crate::bus::ChatMessage::user(content); + message.client_visibility = ClientVisibility::Hidden; + message.turn_origin = TurnOrigin::AgentContinuation; + message.timestamp = now; + message +} + +/// Read-only tool set allowed during continuation Turns. The root Agent may +/// inspect results and read files but cannot write, message, delegate or +/// schedule anything. +pub fn continuation_tool_names() -> &'static [&'static str] { + &[ + "file_read", + "file_search", + "content_search", + "web_fetch", + "calculator", + "agent_task", + ] +} + +#[cfg(test)] +mod tests { + use super::*; + use tempfile::TempDir; + + async fn create_test_storage() -> (super::super::Storage, TempDir) { + let dir = tempfile::tempdir().unwrap(); + let db_path = dir.path().join("inbox.db"); + let storage = super::super::Storage::new(&db_path).await.unwrap(); + (storage, dir) + } + + fn completion_event(session: &str, run_id: &str) -> NewInboxEvent { + NewInboxEvent { + id: uuid::Uuid::new_v4().to_string(), + root_session_id: session.to_string(), + scope_kind: "run".to_string(), + scope_id: run_id.to_string(), + run_id: Some(run_id.to_string()), + group_id: None, + event_type: AgentEventType::Completion, + event_key: format!("completion:{run_id}"), + delivery: AgentEventDelivery::Queue, + requires_continuation: true, + severity: None, + payload_json: serde_json::json!({ "status": "completed" }).to_string(), + } + } + + async fn seed_run(storage: &super::super::Storage, run_id: &str, session: &str) { + use crate::storage::agent_run::{AcceptAgentRequest, AgentRunMode, NewAgentRun}; + storage + .accept_agent_runs(AcceptAgentRequest { + group: None, + runs: vec![NewAgentRun { + id: run_id.to_string(), + root_session_id: session.to_string(), + root_turn_id: None, + parent_run_id: None, + caller_agent_id: "ROOT".to_string(), + caller_scope_id: "turn-1".to_string(), + idempotency_key: None, + agent_id: "researcher".to_string(), + definition_hash: "hash".to_string(), + provider_profile: "research".to_string(), + provider_name: "test".to_string(), + model_id: "test-model".to_string(), + mode: AgentRunMode::Foreground, + depth: 1, + plan_item_id: None, + execution_id: format!("exec-{run_id}"), + task: "work".to_string(), + context_json: None, + budget_json: "{}".to_string(), + deadline_at: 1000, + runtime_generation: 1, + completion_slot_reserved: false, + }], + now: 5, + }) + .await + .unwrap(); + } + + #[tokio::test] + async fn reservation_and_completion_never_exceed_capacity() { + let (storage, _dir) = create_test_storage().await; + storage + .ensure_agent_session_state("cli:test:d1", 1) + .await + .unwrap(); + seed_run(&storage, "run-1", "cli:test:d1").await; + seed_run(&storage, "run-2", "cli:test:d1").await; + + // Reserve both slots. + assert!( + storage + .reserve_completion_slots("cli:test:d1", 2, 2, 10) + .await + .unwrap() + .is_some() + ); + assert!( + storage + .reserve_completion_slots("cli:test:d1", 1, 2, 20) + .await + .unwrap() + .is_none() + ); + + // A signal must not displace reserved completions, and a completion + // event cannot be inserted over the reservation cap either (the + // terminal-commit path converts a reservation instead). + let mut event = completion_event("cli:test:d1", "run-1"); + event.event_type = AgentEventType::Signal; + assert!( + storage + .insert_agent_signal(&event, 2, 30) + .await + .unwrap() + .is_none() + ); + assert!( + storage + .insert_agent_signal(&completion_event("cli:test:d1", "run-2"), 2, 40) + .await + .unwrap() + .is_none() + ); + } + + #[tokio::test] + async fn claim_release_cycle_preserves_events() { + let (storage, _dir) = create_test_storage().await; + seed_run(&storage, "run-1", "cli:test:d1").await; + let event = completion_event("cli:test:d1", "run-1"); + storage + .insert_agent_signal(&event, 16, 10) + .await + .unwrap() + .unwrap(); + + let lease = storage + .claim_inbox_batch("cli:test:d1", 100, 60_000, 8, 32 * 1024, None) + .await + .unwrap() + .expect("due event"); + assert_eq!(lease.events.len(), 1); + assert_eq!(lease.events[0].status, AgentEventStatus::Leased); + + // Wrong token cannot release. + storage + .release_inbox_lease( + "wrong-token", + std::slice::from_ref(&event.id), + 5000, + Some("x"), + 110, + ) + .await + .unwrap(); + let still_leased = storage + .get_agent_inbox_event(&event.id) + .await + .unwrap() + .unwrap(); + assert_eq!(still_leased.status, AgentEventStatus::Leased); + + storage + .release_inbox_lease( + &lease.token, + std::slice::from_ref(&event.id), + 5000, + Some("failed"), + 120, + ) + .await + .unwrap(); + let pending = storage + .get_agent_inbox_event(&event.id) + .await + .unwrap() + .unwrap(); + assert_eq!(pending.status, AgentEventStatus::Pending); + assert_eq!(pending.attempt_count, 1); + assert_eq!(pending.last_error.as_deref(), Some("failed")); + assert_eq!(pending.next_attempt_at, Some(120 + 5000)); + } + + #[tokio::test] + async fn batch_claim_respects_size_and_payload_bounds() { + let (storage, _dir) = create_test_storage().await; + for index in 0..4 { + seed_run(&storage, &format!("run-{index}"), "cli:test:d1").await; + + storage + .insert_agent_signal( + &completion_event("cli:test:d1", &format!("run-{index}")), + 16, + 10, + ) + .await + .unwrap() + .unwrap(); + } + let lease = storage + .claim_inbox_batch("cli:test:d1", 100, 60_000, 2, 32 * 1024, None) + .await + .unwrap() + .expect("due events"); + assert_eq!(lease.events.len(), 2); + // Remaining events stay pending for the next claim. + let lease2 = storage + .claim_inbox_batch("cli:test:d1", 100, 60_000, 2, 32 * 1024, None) + .await + .unwrap() + .expect("remaining events"); + assert_eq!(lease2.events.len(), 2); + } + + #[tokio::test] + async fn supersede_reduces_pending_count() { + let (storage, _dir) = create_test_storage().await; + seed_run(&storage, "run-1", "cli:test:d1").await; + let mut event = completion_event("cli:test:d1", "run-1"); + event.event_type = AgentEventType::Signal; + storage + .insert_agent_signal(&event, 16, 10) + .await + .unwrap() + .unwrap(); + + let count = storage + .supersede_agent_events("run-1", AgentEventType::Signal, 20) + .await + .unwrap(); + assert_eq!(count, 1); + let record = storage + .get_agent_inbox_event(&event.id) + .await + .unwrap() + .unwrap(); + assert_eq!(record.status, AgentEventStatus::Superseded); + } + + #[tokio::test] + async fn continuation_commit_is_atomic_with_event_consumption() { + use crate::storage::message::MessageMeta; + use crate::storage::session::SessionMeta; + + let (storage, _dir) = create_test_storage().await; + sqlx::query( + "INSERT INTO sessions (id, channel, chat_id, dialog_id, created_at, last_active_at) \ + VALUES ('cli:test:d1', 'cli', 'test', 'd1', 1, 1)", + ) + .execute(storage.pool()) + .await + .unwrap(); + seed_run(&storage, "run-1", "cli:test:d1").await; + storage + .insert_agent_signal(&completion_event("cli:test:d1", "run-1"), 16, 10) + .await + .unwrap() + .unwrap(); + let lease = storage + .claim_inbox_batch("cli:test:d1", 100, 60_000, 8, 32 * 1024, None) + .await + .unwrap() + .unwrap(); + + let session_meta = SessionMeta { + id: "cli:test:d1".to_string(), + channel: "cli".to_string(), + chat_id: "test".to_string(), + dialog_id: "d1".to_string(), + title: "test".to_string(), + created_at: 1, + last_active_at: 200, + message_count: 1, + routing_info: None, + archived_at: None, + deleted_at: None, + last_consolidated_at: None, + last_compressed_message_at: None, + }; + let trigger = MessageMeta { + id: "trigger-1".to_string(), + session_id: "cli:test:d1".to_string(), + seq: 1, + role: "user".to_string(), + content: "hidden trigger".to_string(), + reasoning_content: None, + provider_state: None, + turn_id: Some("turn-1".to_string()), + iteration: None, + completion_status: crate::bus::CompletionStatus::Completed, + client_visibility: crate::bus::ClientVisibility::Hidden, + turn_origin: crate::bus::TurnOrigin::AgentContinuation, + media_refs: None, + tool_call_id: None, + tool_name: None, + tool_calls: None, + source: None, + created_at: 200, + }; + let assistant = MessageMeta { + id: "assistant-1".to_string(), + session_id: "cli:test:d1".to_string(), + seq: 2, + role: "assistant".to_string(), + content: "task finished".to_string(), + reasoning_content: None, + provider_state: None, + turn_id: Some("turn-1".to_string()), + iteration: Some(1), + completion_status: crate::bus::CompletionStatus::Completed, + client_visibility: crate::bus::ClientVisibility::Visible, + turn_origin: crate::bus::TurnOrigin::AgentContinuation, + media_refs: None, + tool_call_id: None, + tool_name: None, + tool_calls: None, + source: None, + created_at: 200, + }; + + let committed = storage + .commit_continuation_turn( + "cli:test:d1", + &trigger, + &[assistant], + &session_meta, + None, + &lease.token, + &[lease.events[0].id.clone()], + 200, + ) + .await + .unwrap(); + assert_eq!(committed.len(), 1); + + // Messages persisted; hidden trigger is invisible to client queries. + let replay = storage.load_messages("cli:test:d1", 0).await.unwrap(); + assert_eq!(replay.len(), 2); + let visible = storage + .load_recent_session_messages("cli:test:d1", 100) + .await + .unwrap(); + assert!( + visible.iter().all(|message| { + message.client_visibility == crate::bus::ClientVisibility::Visible + }) + ); + assert!(visible.iter().any(|message| message.id == "assistant-1")); + assert!(visible.iter().all(|message| message.id != "trigger-1")); + assert!( + visible + .iter() + .all(|message| message.turn_origin == crate::bus::TurnOrigin::AgentContinuation) + ); + + // The event was consumed; the lease token now owns nothing. + let record = storage + .get_agent_inbox_event(&lease.events[0].id) + .await + .unwrap() + .unwrap(); + assert_eq!(record.status, AgentEventStatus::Consumed); + assert_eq!(record.consumed_at, Some(200)); + let state: i64 = sqlx::query_scalar( + "SELECT pending_event_count FROM agent_session_state WHERE root_session_id = 'cli:test:d1'", + ) + .fetch_one(storage.pool()) + .await + .unwrap(); + assert_eq!(state, 0); + } + + #[tokio::test] + async fn dead_letter_leased_events_reduces_pending_and_requires_token() { + let (storage, _dir) = create_test_storage().await; + seed_run(&storage, "run-1", "cli:test:d1").await; + storage + .insert_agent_signal(&completion_event("cli:test:d1", "run-1"), 16, 10) + .await + .unwrap() + .unwrap(); + let lease = storage + .claim_inbox_batch("cli:test:d1", 100, 60_000, 8, 32 * 1024, None) + .await + .unwrap() + .unwrap(); + + // Wrong token cannot dead-letter. + assert_eq!( + storage + .dead_letter_leased_events("wrong", &[lease.events[0].id.clone()], "boom", 110) + .await + .unwrap(), + 0 + ); + assert_eq!( + storage + .dead_letter_leased_events(&lease.token, &[lease.events[0].id.clone()], "boom", 120) + .await + .unwrap(), + 1 + ); + let record = storage + .get_agent_inbox_event(&lease.events[0].id) + .await + .unwrap() + .unwrap(); + assert_eq!(record.status, AgentEventStatus::DeadLetter); + assert_eq!(record.dead_lettered_at, Some(120)); + let state: i64 = sqlx::query_scalar( + "SELECT pending_event_count FROM agent_session_state WHERE root_session_id = 'cli:test:d1'", + ) + .fetch_one(storage.pool()) + .await + .unwrap(); + assert_eq!(state, 0); + } + + #[tokio::test] + async fn next_pending_due_at_tracks_retry_backoff() { + let (storage, _dir) = create_test_storage().await; + seed_run(&storage, "run-1", "cli:test:d1").await; + storage + .insert_agent_signal(&completion_event("cli:test:d1", "run-1"), 16, 10) + .await + .unwrap() + .unwrap(); + assert_eq!( + storage.next_pending_due_at("cli:test:d1").await.unwrap(), + Some(10) + ); + let lease = storage + .claim_inbox_batch("cli:test:d1", 100, 60_000, 8, 32 * 1024, None) + .await + .unwrap() + .unwrap(); + // Leased events are not pending anymore. + assert_eq!( + storage.next_pending_due_at("cli:test:d1").await.unwrap(), + None + ); + storage + .release_inbox_lease( + &lease.token, + &[lease.events[0].id.clone()], + 5000, + Some("x"), + 200, + ) + .await + .unwrap(); + assert_eq!( + storage.next_pending_due_at("cli:test:d1").await.unwrap(), + Some(200 + 5000) + ); + } + + #[tokio::test] + async fn recovery_interrupts_old_generation_and_expires_leases() { + use crate::storage::agent_run::{AcceptAgentRequest, AgentRunMode, NewAgentRun}; + + let (storage, _dir) = create_test_storage().await; + // A running old-generation background run with a reserved slot. + let run = NewAgentRun { + id: "run-1".to_string(), + root_session_id: "cli:test:d1".to_string(), + root_turn_id: None, + parent_run_id: None, + caller_agent_id: "ROOT".to_string(), + caller_scope_id: "turn-1".to_string(), + idempotency_key: None, + agent_id: "researcher".to_string(), + definition_hash: "hash".to_string(), + provider_profile: "research".to_string(), + provider_name: "test".to_string(), + model_id: "test-model".to_string(), + mode: AgentRunMode::Background, + depth: 1, + plan_item_id: None, + execution_id: "exec-1".to_string(), + task: "work".to_string(), + context_json: None, + budget_json: "{}".to_string(), + deadline_at: 1000, + runtime_generation: 1, + completion_slot_reserved: false, + }; + storage + .accept_agent_runs(AcceptAgentRequest { + group: None, + runs: vec![run], + now: 10, + }) + .await + .unwrap(); + storage + .reserve_completion_slots("cli:test:d1", 1, 16, 10) + .await + .unwrap(); + storage + .mark_agent_run_running("run-1", "exec-1", 10) + .await + .unwrap(); + // Reservation is now reflected on the run row. + sqlx::query("UPDATE agent_runs SET completion_slot_reserved = 1 WHERE id = 'run-1'") + .execute(storage.pool()) + .await + .unwrap(); + + // An old lease that expires during recovery. + seed_run(&storage, "run-x", "cli:test:d1").await; + let event = completion_event("cli:test:d1", "run-x"); + storage + .insert_agent_signal(&event, 16, 10) + .await + .unwrap() + .unwrap(); + let lease = storage + .claim_inbox_batch("cli:test:d1", 100, 10, 8, 32 * 1024, None) + .await + .unwrap() + .unwrap(); + + let report = storage.recover_agent_state(2, 200, 8, 5000).await.unwrap(); + // run-1 (running) and run-x (queued) both belong to generation 1 and + // are interrupted; only run-1 held a reserved completion slot. + assert_eq!(report.interrupted_runs, 2); + assert_eq!(report.completion_events_generated, 1); + assert_eq!(report.leases_expired, 1); + + let run = storage.get_agent_run("run-1").await.unwrap().unwrap(); + assert_eq!( + run.status, + crate::storage::agent_run::AgentRunStatus::Interrupted + ); + + // The expired lease returned to pending with a backoff. + let record = storage + .get_agent_inbox_event(&event.id) + .await + .unwrap() + .unwrap(); + assert_eq!(record.status, AgentEventStatus::Pending); + assert_eq!(record.attempt_count, 2); + assert!(record.lease_token.is_none()); + drop(lease); + } + + fn signal_event(session: &str, run_id: &str, dedupe_key: &str, window: i64) -> NewInboxEvent { + NewInboxEvent { + id: uuid::Uuid::new_v4().to_string(), + root_session_id: session.to_string(), + scope_kind: "run".to_string(), + scope_id: run_id.to_string(), + run_id: Some(run_id.to_string()), + group_id: None, + event_type: AgentEventType::Signal, + event_key: format!("signal:{dedupe_key}:{window}"), + delivery: AgentEventDelivery::Steer, + requires_continuation: true, + severity: Some("warning".to_string()), + payload_json: serde_json::json!({ "kind": "signal", "summary": "s" }).to_string(), + } + } + + #[tokio::test] + async fn steer_claim_only_claims_steer_events() { + let (storage, _dir) = create_test_storage().await; + seed_run(&storage, "run-1", "cli:test:d1").await; + seed_run(&storage, "run-2", "cli:test:d1").await; + let mut queue_event = completion_event("cli:test:d1", "run-1"); + queue_event.delivery = AgentEventDelivery::Queue; + storage + .insert_agent_signal(&queue_event, 16, 10) + .await + .unwrap() + .unwrap(); + storage + .insert_agent_signal(&signal_event("cli:test:d1", "run-2", "k", 0), 16, 10) + .await + .unwrap() + .unwrap(); + + // The steer lane claims only steer events; the queue lane claims all. + let steer_lease = storage + .claim_inbox_batch( + "cli:test:d1", + 100, + 60_000, + 8, + 32 * 1024, + Some(AgentEventDelivery::Steer), + ) + .await + .unwrap() + .unwrap(); + assert_eq!(steer_lease.events.len(), 1); + assert_eq!(steer_lease.events[0].event_type, AgentEventType::Signal); + let all_lease = storage + .claim_inbox_batch("cli:test:d1", 100, 60_000, 8, 32 * 1024, None) + .await + .unwrap() + .unwrap(); + assert_eq!(all_lease.events.len(), 1); + assert_eq!(all_lease.events[0].event_type, AgentEventType::Completion); + } + + #[tokio::test] + async fn signal_dedupe_collapses_same_window_and_reemits_after_cooldown() { + let (storage, _dir) = create_test_storage().await; + seed_run(&storage, "run-1", "cli:test:d1").await; + let first = signal_event("cli:test:d1", "run-1", "svc-a:err", 7); + let (record, deduplicated) = storage + .insert_agent_signal(&first, 16, 10) + .await + .unwrap() + .unwrap(); + assert!(!deduplicated); + let first_id = record.id.clone(); + + // Same key inside the cooldown window returns the same event. + let second = signal_event("cli:test:d1", "run-1", "svc-a:err", 7); + let (record, deduplicated) = storage + .insert_agent_signal(&second, 16, 11) + .await + .unwrap() + .unwrap(); + assert!(deduplicated); + assert_eq!(record.id, first_id); + + // A new cooldown window emits a fresh event. + let third = signal_event("cli:test:d1", "run-1", "svc-a:err", 8); + let (record, deduplicated) = storage + .insert_agent_signal(&third, 16, 12) + .await + .unwrap() + .unwrap(); + assert!(!deduplicated); + assert_ne!(record.id, first_id); + + // Only one event occupies pending capacity. + let state: (i64,) = sqlx::query_as( + "SELECT pending_event_count FROM agent_session_state WHERE root_session_id = 'cli:test:d1'", + ) + .fetch_one(storage.pool()) + .await + .unwrap(); + assert_eq!(state.0, 2); + } + + #[tokio::test] + async fn steer_admit_then_consume_at_user_turn_commit() { + use crate::storage::message::MessageMeta; + use crate::storage::session::SessionMeta; + + let (storage, _dir) = create_test_storage().await; + sqlx::query( + "INSERT INTO sessions (id, channel, chat_id, dialog_id, created_at, last_active_at) \ + VALUES ('cli:test:d1', 'cli', 'test', 'd1', 1, 1)", + ) + .execute(storage.pool()) + .await + .unwrap(); + seed_run(&storage, "run-1", "cli:test:d1").await; + storage + .insert_agent_signal(&signal_event("cli:test:d1", "run-1", "k", 0), 16, 10) + .await + .unwrap() + .unwrap(); + let lease = storage + .claim_inbox_batch( + "cli:test:d1", + 100, + 60_000, + 8, + 32 * 1024, + Some(AgentEventDelivery::Steer), + ) + .await + .unwrap() + .unwrap(); + let event_id = lease.events[0].id.clone(); + + // Wrong token cannot admit; the turn id is recorded (not a guard). + assert!( + !storage + .admit_inbox_event(&event_id, "wrong-token", "turn-1", 110) + .await + .unwrap() + ); + assert!( + storage + .admit_inbox_event(&event_id, &lease.token, "turn-1", 110) + .await + .unwrap() + ); + let admitted = storage + .get_agent_inbox_event(&event_id) + .await + .unwrap() + .unwrap(); + assert_eq!(admitted.status, AgentEventStatus::Admitted); + assert_eq!(admitted.admitted_turn_id.as_deref(), Some("turn-1")); + + // The Turn commit consumes the admitted steer event atomically. + let session_meta = SessionMeta { + id: "cli:test:d1".to_string(), + channel: "cli".to_string(), + chat_id: "test".to_string(), + dialog_id: "d1".to_string(), + title: "test".to_string(), + created_at: 1, + last_active_at: 200, + message_count: 1, + routing_info: None, + archived_at: None, + deleted_at: None, + last_consolidated_at: None, + last_compressed_message_at: None, + }; + let user = MessageMeta { + id: "user-1".to_string(), + session_id: "cli:test:d1".to_string(), + seq: 1, + role: "user".to_string(), + content: "steer envelope".to_string(), + reasoning_content: None, + provider_state: None, + turn_id: Some("turn-1".to_string()), + iteration: Some(1), + completion_status: crate::bus::CompletionStatus::Completed, + client_visibility: crate::bus::ClientVisibility::Hidden, + turn_origin: crate::bus::TurnOrigin::User, + media_refs: None, + tool_call_id: None, + tool_name: None, + tool_calls: None, + source: None, + created_at: 200, + }; + let steer = SteerConsumption { + event_ids: vec![event_id.clone()], + admitted_turn_id: "turn-1".to_string(), + }; + storage + .persist_turn_batch_with_steer_consumption( + "cli:test:d1", + &[user], + &session_meta, + &crate::storage::TurnUsageRecord { + session_id: "cli:test:d1".to_string(), + turn_id: "turn-1".to_string(), + provider: "test".to_string(), + model: "test-model".to_string(), + usage: crate::providers::Usage::default(), + last_prompt_tokens: 0, + created_at: 200, + }, + &steer, + ) + .await + .unwrap(); + + let record = storage + .get_agent_inbox_event(&event_id) + .await + .unwrap() + .unwrap(); + assert_eq!(record.status, AgentEventStatus::Consumed); + let state: (i64, i64) = sqlx::query_as( + "SELECT pending_event_count, revision FROM agent_session_state \ + WHERE root_session_id = 'cli:test:d1'", + ) + .fetch_one(storage.pool()) + .await + .unwrap(); + assert_eq!(state.0, 0); + assert!(state.1 >= 2); + } + + #[tokio::test] + async fn admitted_steer_event_is_released_back_to_pending() { + let (storage, _dir) = create_test_storage().await; + seed_run(&storage, "run-1", "cli:test:d1").await; + storage + .insert_agent_signal(&signal_event("cli:test:d1", "run-1", "k", 0), 16, 10) + .await + .unwrap() + .unwrap(); + let lease = storage + .claim_inbox_batch( + "cli:test:d1", + 100, + 60_000, + 8, + 32 * 1024, + Some(AgentEventDelivery::Steer), + ) + .await + .unwrap() + .unwrap(); + let event_id = lease.events[0].id.clone(); + storage + .admit_inbox_event(&event_id, &lease.token, "turn-1", 110) + .await + .unwrap(); + + // `/stop`-style release returns the admitted event to pending. + storage + .release_inbox_lease( + &lease.token, + std::slice::from_ref(&event_id), + 0, + Some("stopped"), + 120, + ) + .await + .unwrap(); + let record = storage + .get_agent_inbox_event(&event_id) + .await + .unwrap() + .unwrap(); + assert_eq!(record.status, AgentEventStatus::Pending); + assert_eq!(record.next_attempt_at, Some(120)); + } +} diff --git a/src/storage/agent_run.rs b/src/storage/agent_run.rs new file mode 100644 index 0000000..58d088b --- /dev/null +++ b/src/storage/agent_run.rs @@ -0,0 +1,1743 @@ +use sqlx::{Row, SqliteConnection}; + +use super::StorageError; + +/// Frozen schema v6 DDL for the Agent orchestration tables. Executed inside +/// the single migration transaction so table creation, column additions and +/// `user_version` advance atomically. The inbox tables belong to Phase 3 +/// behavior but their shape is frozen together with the run tables. +pub const AGENT_SCHEMA_STATEMENTS: &[&str] = &[ + r#" + CREATE TABLE IF NOT EXISTS agent_run_groups ( + id TEXT PRIMARY KEY, + root_session_id TEXT NOT NULL, + caller_run_id TEXT, + caller_scope_id TEXT NOT NULL, + idempotency_key TEXT, + mode TEXT NOT NULL, + completion_policy TEXT NOT NULL, + expected_runs INTEGER NOT NULL, + terminal_runs INTEGER NOT NULL DEFAULT 0, + abnormal_runs INTEGER NOT NULL DEFAULT 0, + completion_slot_reserved INTEGER NOT NULL DEFAULT 0, + completion_delivery TEXT, + failure_delivery TEXT, + deadline_at INTEGER NOT NULL, + status TEXT NOT NULL, + runtime_generation INTEGER NOT NULL, + revision INTEGER NOT NULL, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + finished_at INTEGER, + CHECK (mode IN ('foreground', 'background')), + CHECK (completion_policy IN ('all', 'each')), + CHECK (status IN ('queued', 'running', 'completed', 'partial', 'failed', + 'timed_out', 'cancelled', 'interrupted')), + CHECK (expected_runs > 0), + CHECK (terminal_runs >= 0 AND terminal_runs <= expected_runs), + CHECK (completion_slot_reserved IN (0, 1)) + ) + "#, + "CREATE INDEX IF NOT EXISTS idx_agent_groups_session_created ON agent_run_groups(root_session_id, created_at DESC)", + "CREATE UNIQUE INDEX IF NOT EXISTS idx_agent_groups_idempotency ON agent_run_groups(root_session_id, caller_scope_id, idempotency_key) WHERE idempotency_key IS NOT NULL", + r#" + CREATE TABLE IF NOT EXISTS agent_runs ( + id TEXT PRIMARY KEY, + group_id TEXT, + root_session_id TEXT NOT NULL, + root_turn_id TEXT, + parent_run_id TEXT, + caller_agent_id TEXT NOT NULL, + caller_scope_id TEXT NOT NULL, + idempotency_key TEXT, + agent_id TEXT NOT NULL, + definition_hash TEXT NOT NULL, + provider_profile TEXT NOT NULL, + provider_name TEXT NOT NULL, + model_id TEXT NOT NULL, + mode TEXT NOT NULL, + depth INTEGER NOT NULL, + plan_item_id TEXT, + execution_id TEXT NOT NULL, + task TEXT NOT NULL, + context_json TEXT, + budget_json TEXT NOT NULL, + signal_contract_json TEXT, + signal_delivery TEXT, + completion_delivery TEXT, + failure_delivery TEXT, + status TEXT NOT NULL, + result TEXT, + error TEXT, + prompt_tokens INTEGER, + completion_tokens INTEGER, + cost REAL, + tool_calls_count INTEGER NOT NULL DEFAULT 0, + iterations INTEGER NOT NULL DEFAULT 0, + runtime_generation INTEGER NOT NULL, + attempt INTEGER NOT NULL DEFAULT 1, + completion_slot_reserved INTEGER NOT NULL DEFAULT 0, + deadline_at INTEGER NOT NULL, + revision INTEGER NOT NULL, + started_at INTEGER, + finished_at INTEGER, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + CHECK (mode IN ('foreground', 'background')), + CHECK (status IN ('queued', 'running', 'waiting_children', 'completed', + 'failed', 'timed_out', 'cancelled', 'interrupted')), + CHECK (depth >= 1), + CHECK (completion_slot_reserved IN (0, 1)), + FOREIGN KEY (group_id) REFERENCES agent_run_groups(id) ON DELETE RESTRICT, + FOREIGN KEY (parent_run_id) REFERENCES agent_runs(id) ON DELETE RESTRICT + ) + "#, + "CREATE UNIQUE INDEX IF NOT EXISTS idx_agent_runs_execution ON agent_runs(execution_id)", + "CREATE UNIQUE INDEX IF NOT EXISTS idx_agent_runs_idempotency ON agent_runs(root_session_id, caller_scope_id, idempotency_key) WHERE idempotency_key IS NOT NULL", + "CREATE INDEX IF NOT EXISTS idx_agent_runs_session_created ON agent_runs(root_session_id, created_at DESC)", + "CREATE INDEX IF NOT EXISTS idx_agent_runs_parent ON agent_runs(parent_run_id, created_at)", + "CREATE INDEX IF NOT EXISTS idx_agent_runs_recovery ON agent_runs(runtime_generation, status, deadline_at)", + r#" + CREATE TABLE IF NOT EXISTS agent_session_state ( + root_session_id TEXT PRIMARY KEY, + revision INTEGER NOT NULL DEFAULT 0, + pending_event_count INTEGER NOT NULL DEFAULT 0, + reserved_completion_slots INTEGER NOT NULL DEFAULT 0, + updated_at INTEGER NOT NULL, + CHECK (revision >= 0), + CHECK (pending_event_count >= 0), + CHECK (reserved_completion_slots >= 0) + ) + "#, + r#" + CREATE TABLE IF NOT EXISTS agent_inbox_events ( + id TEXT PRIMARY KEY, + root_session_id TEXT NOT NULL, + scope_kind TEXT NOT NULL, + scope_id TEXT NOT NULL, + run_id TEXT, + group_id TEXT, + event_type TEXT NOT NULL, + event_key TEXT NOT NULL, + delivery TEXT NOT NULL, + requires_continuation INTEGER NOT NULL DEFAULT 1, + severity TEXT, + payload_json TEXT NOT NULL, + status TEXT NOT NULL, + attempt_count INTEGER NOT NULL DEFAULT 0, + lease_token TEXT, + lease_until INTEGER, + next_attempt_at INTEGER, + admitted_turn_id TEXT, + last_error TEXT, + revision INTEGER NOT NULL, + created_at INTEGER NOT NULL, + consumed_at INTEGER, + superseded_at INTEGER, + dead_lettered_at INTEGER, + fallback_notified_at INTEGER, + fallback_suppressed_reason TEXT, + updated_at INTEGER NOT NULL, + CHECK (scope_kind IN ('run', 'group')), + CHECK (event_type IN ('signal', 'completion', 'group_completion')), + CHECK (delivery IN ('queue', 'steer')), + CHECK (requires_continuation IN (0, 1)), + CHECK (status IN ('pending', 'leased', 'admitted', 'consumed', + 'superseded', 'dead_letter')), + CHECK ( + (scope_kind = 'run' AND run_id IS NOT NULL AND group_id IS NULL + AND scope_id = run_id) OR + (scope_kind = 'group' AND group_id IS NOT NULL AND run_id IS NULL + AND scope_id = group_id) + ), + UNIQUE(scope_kind, scope_id, event_type, event_key), + FOREIGN KEY (run_id) REFERENCES agent_runs(id) ON DELETE RESTRICT, + FOREIGN KEY (group_id) REFERENCES agent_run_groups(id) ON DELETE RESTRICT + ) + "#, + "CREATE INDEX IF NOT EXISTS idx_agent_inbox_claim ON agent_inbox_events(root_session_id, status, next_attempt_at, created_at)", + "CREATE INDEX IF NOT EXISTS idx_agent_inbox_lease ON agent_inbox_events(status, lease_until)", + "CREATE INDEX IF NOT EXISTS idx_agent_inbox_revision ON agent_inbox_events(root_session_id, revision)", +]; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AgentRunMode { + Foreground, + Background, +} + +impl AgentRunMode { + pub fn as_str(&self) -> &'static str { + match self { + Self::Foreground => "foreground", + Self::Background => "background", + } + } + + pub fn parse(value: &str) -> Result { + match value { + "foreground" => Ok(Self::Foreground), + "background" => Ok(Self::Background), + other => Err(StorageError::Migration(format!( + "corrupt agent run mode '{other}'" + ))), + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AgentCompletionPolicy { + All, + Each, +} + +impl AgentCompletionPolicy { + pub fn as_str(&self) -> &'static str { + match self { + Self::All => "all", + Self::Each => "each", + } + } + + pub fn parse(value: &str) -> Result { + match value { + "all" => Ok(Self::All), + "each" => Ok(Self::Each), + other => Err(StorageError::Migration(format!( + "corrupt agent completion policy '{other}'" + ))), + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AgentRunStatus { + Queued, + Running, + WaitingChildren, + Completed, + Failed, + TimedOut, + Cancelled, + Interrupted, +} + +impl AgentRunStatus { + pub fn as_str(&self) -> &'static str { + match self { + Self::Queued => "queued", + Self::Running => "running", + Self::WaitingChildren => "waiting_children", + Self::Completed => "completed", + Self::Failed => "failed", + Self::TimedOut => "timed_out", + Self::Cancelled => "cancelled", + Self::Interrupted => "interrupted", + } + } + + pub fn parse(value: &str) -> Result { + match value { + "queued" => Ok(Self::Queued), + "running" => Ok(Self::Running), + "waiting_children" => Ok(Self::WaitingChildren), + "completed" => Ok(Self::Completed), + "failed" => Ok(Self::Failed), + "timed_out" => Ok(Self::TimedOut), + "cancelled" => Ok(Self::Cancelled), + "interrupted" => Ok(Self::Interrupted), + other => Err(StorageError::Migration(format!( + "corrupt agent run status '{other}'" + ))), + } + } + + pub fn is_terminal(self) -> bool { + !matches!(self, Self::Queued | Self::Running | Self::WaitingChildren) + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AgentGroupStatus { + Queued, + Running, + Completed, + Partial, + Failed, + TimedOut, + Cancelled, + Interrupted, +} + +impl AgentGroupStatus { + pub fn as_str(&self) -> &'static str { + match self { + Self::Queued => "queued", + Self::Running => "running", + Self::Completed => "completed", + Self::Partial => "partial", + Self::Failed => "failed", + Self::TimedOut => "timed_out", + Self::Cancelled => "cancelled", + Self::Interrupted => "interrupted", + } + } + + pub fn parse(value: &str) -> Result { + match value { + "queued" => Ok(Self::Queued), + "running" => Ok(Self::Running), + "completed" => Ok(Self::Completed), + "partial" => Ok(Self::Partial), + "failed" => Ok(Self::Failed), + "timed_out" => Ok(Self::TimedOut), + "cancelled" => Ok(Self::Cancelled), + "interrupted" => Ok(Self::Interrupted), + other => Err(StorageError::Migration(format!( + "corrupt agent group status '{other}'" + ))), + } + } + + pub fn is_terminal(self) -> bool { + !matches!(self, Self::Queued | Self::Running) + } +} + +#[derive(Debug, Clone)] +pub struct AgentRunGroupRecord { + pub id: String, + pub root_session_id: String, + pub caller_run_id: Option, + pub caller_scope_id: String, + pub idempotency_key: Option, + pub mode: AgentRunMode, + pub completion_policy: AgentCompletionPolicy, + pub expected_runs: i64, + pub terminal_runs: i64, + pub abnormal_runs: i64, + pub completion_slot_reserved: bool, + pub completion_delivery: Option, + pub failure_delivery: Option, + pub deadline_at: i64, + pub status: AgentGroupStatus, + pub runtime_generation: i64, + pub revision: i64, + pub created_at: i64, + pub updated_at: i64, + pub finished_at: Option, +} + +#[derive(Debug, Clone)] +pub struct AgentRunRecord { + pub id: String, + pub group_id: Option, + pub root_session_id: String, + pub root_turn_id: Option, + pub parent_run_id: Option, + pub caller_agent_id: String, + pub caller_scope_id: String, + pub idempotency_key: Option, + pub agent_id: String, + pub definition_hash: String, + pub provider_profile: String, + pub provider_name: String, + pub model_id: String, + pub mode: AgentRunMode, + pub depth: i64, + pub plan_item_id: Option, + pub execution_id: String, + pub task: String, + pub context_json: Option, + pub budget_json: String, + pub signal_contract_json: Option, + pub signal_delivery: Option, + pub completion_delivery: Option, + pub failure_delivery: Option, + pub status: AgentRunStatus, + pub result: Option, + pub error: Option, + pub prompt_tokens: Option, + pub completion_tokens: Option, + pub cost: Option, + pub tool_calls_count: i64, + pub iterations: i64, + pub runtime_generation: i64, + pub attempt: i64, + pub completion_slot_reserved: bool, + pub deadline_at: i64, + pub revision: i64, + pub started_at: Option, + pub finished_at: Option, + pub created_at: i64, + pub updated_at: i64, +} + +/// One run to admit inside `accept_agent_runs`. +#[derive(Debug, Clone)] +pub struct NewAgentRun { + pub id: String, + pub root_session_id: String, + pub root_turn_id: Option, + pub parent_run_id: Option, + pub caller_agent_id: String, + pub caller_scope_id: String, + pub idempotency_key: Option, + pub agent_id: String, + pub definition_hash: String, + pub provider_profile: String, + pub provider_name: String, + pub model_id: String, + pub mode: AgentRunMode, + pub depth: i64, + pub plan_item_id: Option, + pub execution_id: String, + pub task: String, + pub context_json: Option, + pub budget_json: String, + pub deadline_at: i64, + pub runtime_generation: i64, + /// Background runs reserve a completion slot at admission so their + /// completion can never be lost to inbox capacity exhaustion. + pub completion_slot_reserved: bool, +} + +/// Group header for batch admission. Single-task requests must not create a +/// group; their idempotency key lives on the run row instead. +#[derive(Debug, Clone)] +pub struct NewAgentGroup { + pub id: String, + pub root_session_id: String, + pub caller_run_id: Option, + pub caller_scope_id: String, + pub idempotency_key: Option, + pub mode: AgentRunMode, + pub completion_policy: AgentCompletionPolicy, + pub deadline_at: i64, + pub runtime_generation: i64, +} + +#[derive(Debug, Clone)] +pub struct AcceptAgentRequest { + pub group: Option, + pub runs: Vec, + pub now: i64, +} + +#[derive(Debug)] +pub enum AcceptedAgentRuns { + Accepted { + group: Option, + runs: Vec, + }, + /// Idempotent retry: the group/run already existed for this key. + Existing { + group: Option, + runs: Vec, + }, +} + +/// Terminal outcome produced by a runner. The Coordinator persists it; the +/// runner itself never writes channels or plan state. +#[derive(Debug, Clone)] +pub enum AgentTerminalOutcome { + Completed { + result: String, + prompt_tokens: Option, + completion_tokens: Option, + cost: Option, + tool_calls: i64, + iterations: i64, + }, + Failed { + error: String, + prompt_tokens: Option, + completion_tokens: Option, + cost: Option, + }, + TimedOut { + deadline_at: i64, + }, + Cancelled { + reason: String, + }, + Interrupted { + reason: String, + }, +} + +impl AgentTerminalOutcome { + pub fn status(&self) -> AgentRunStatus { + match self { + Self::Completed { .. } => AgentRunStatus::Completed, + Self::Failed { .. } => AgentRunStatus::Failed, + Self::TimedOut { .. } => AgentRunStatus::TimedOut, + Self::Cancelled { .. } => AgentRunStatus::Cancelled, + Self::Interrupted { .. } => AgentRunStatus::Interrupted, + } + } + + pub fn is_abnormal(&self) -> bool { + !matches!(self, Self::Completed { .. }) + } +} + +#[derive(Debug, Clone)] +pub struct TerminalCommit { + pub run: AgentRunRecord, + pub group: Option, + pub group_finished: bool, +} + +const RUN_COLUMNS: &str = "id, group_id, root_session_id, root_turn_id, parent_run_id, \ + caller_agent_id, caller_scope_id, idempotency_key, agent_id, definition_hash, \ + provider_profile, provider_name, model_id, mode, depth, plan_item_id, execution_id, \ + task, context_json, budget_json, signal_contract_json, signal_delivery, \ + completion_delivery, failure_delivery, status, result, error, prompt_tokens, \ + completion_tokens, cost, tool_calls_count, iterations, runtime_generation, attempt, \ + completion_slot_reserved, deadline_at, revision, started_at, finished_at, \ + created_at, updated_at"; + +const GROUP_COLUMNS: &str = "id, root_session_id, caller_run_id, caller_scope_id, \ + idempotency_key, mode, completion_policy, expected_runs, terminal_runs, \ + abnormal_runs, completion_slot_reserved, completion_delivery, failure_delivery, \ + deadline_at, status, runtime_generation, revision, created_at, updated_at, finished_at"; + +fn run_record_from_row(row: &sqlx::sqlite::SqliteRow) -> Result { + Ok(AgentRunRecord { + id: row.get("id"), + group_id: row.get("group_id"), + root_session_id: row.get("root_session_id"), + root_turn_id: row.get("root_turn_id"), + parent_run_id: row.get("parent_run_id"), + caller_agent_id: row.get("caller_agent_id"), + caller_scope_id: row.get("caller_scope_id"), + idempotency_key: row.get("idempotency_key"), + agent_id: row.get("agent_id"), + definition_hash: row.get("definition_hash"), + provider_profile: row.get("provider_profile"), + provider_name: row.get("provider_name"), + model_id: row.get("model_id"), + mode: AgentRunMode::parse(row.get::<&str, _>("mode"))?, + depth: row.get("depth"), + plan_item_id: row.get("plan_item_id"), + execution_id: row.get("execution_id"), + task: row.get("task"), + context_json: row.get("context_json"), + budget_json: row.get("budget_json"), + signal_contract_json: row.get("signal_contract_json"), + signal_delivery: row.get("signal_delivery"), + completion_delivery: row.get("completion_delivery"), + failure_delivery: row.get("failure_delivery"), + status: AgentRunStatus::parse(row.get::<&str, _>("status"))?, + result: row.get("result"), + error: row.get("error"), + prompt_tokens: row.get("prompt_tokens"), + completion_tokens: row.get("completion_tokens"), + cost: row.get("cost"), + tool_calls_count: row.get("tool_calls_count"), + iterations: row.get("iterations"), + runtime_generation: row.get("runtime_generation"), + attempt: row.get("attempt"), + completion_slot_reserved: row.get::("completion_slot_reserved") != 0, + deadline_at: row.get("deadline_at"), + revision: row.get("revision"), + started_at: row.get("started_at"), + finished_at: row.get("finished_at"), + created_at: row.get("created_at"), + updated_at: row.get("updated_at"), + }) +} + +fn group_record_from_row( + row: &sqlx::sqlite::SqliteRow, +) -> Result { + Ok(AgentRunGroupRecord { + id: row.get("id"), + root_session_id: row.get("root_session_id"), + caller_run_id: row.get("caller_run_id"), + caller_scope_id: row.get("caller_scope_id"), + idempotency_key: row.get("idempotency_key"), + mode: AgentRunMode::parse(row.get::<&str, _>("mode"))?, + completion_policy: AgentCompletionPolicy::parse(row.get::<&str, _>("completion_policy"))?, + expected_runs: row.get("expected_runs"), + terminal_runs: row.get("terminal_runs"), + abnormal_runs: row.get("abnormal_runs"), + completion_slot_reserved: row.get::("completion_slot_reserved") != 0, + completion_delivery: row.get("completion_delivery"), + failure_delivery: row.get("failure_delivery"), + deadline_at: row.get("deadline_at"), + status: AgentGroupStatus::parse(row.get::<&str, _>("status"))?, + runtime_generation: row.get("runtime_generation"), + revision: row.get("revision"), + created_at: row.get("created_at"), + updated_at: row.get("updated_at"), + finished_at: row.get("finished_at"), + }) +} + +impl super::Storage { + /// Admit a group (optional) and its runs in one transaction, claiming any + /// referenced plan items atomically. If any plan item was already taken + /// the whole admission rolls back so a run can never diverge from the + /// plan it claims to execute. + pub async fn accept_agent_runs( + &self, + request: AcceptAgentRequest, + ) -> Result { + if request.runs.is_empty() { + return Err(StorageError::Conflict( + "agent admission requires at least one run".to_string(), + )); + } + let mut tx = self.pool.begin().await?; + + if let Some(group) = request.group.as_ref() { + let inserted = sqlx::query( + "INSERT INTO agent_run_groups (id, root_session_id, caller_run_id, \ + caller_scope_id, idempotency_key, mode, completion_policy, \ + expected_runs, terminal_runs, abnormal_runs, completion_slot_reserved, \ + deadline_at, status, runtime_generation, revision, created_at, updated_at) \ + VALUES (?, ?, ?, ?, ?, ?, ?, ?, 0, 0, 0, ?, 'queued', ?, 0, ?, ?)", + ) + .bind(&group.id) + .bind(&group.root_session_id) + .bind(&group.caller_run_id) + .bind(&group.caller_scope_id) + .bind(&group.idempotency_key) + .bind(group.mode.as_str()) + .bind(group.completion_policy.as_str()) + .bind(request.runs.len() as i64) + .bind(group.deadline_at) + .bind(group.runtime_generation) + .bind(request.now) + .bind(request.now) + .execute(&mut *tx) + .await? + .rows_affected() + == 1; + if !inserted { + drop(tx); + return self.existing_agent_admission(request).await; + } + } + + for run in &request.runs { + let inserted = sqlx::query( + "INSERT INTO agent_runs (id, group_id, root_session_id, root_turn_id, \ + parent_run_id, caller_agent_id, caller_scope_id, idempotency_key, \ + agent_id, definition_hash, provider_profile, provider_name, model_id, \ + mode, depth, plan_item_id, execution_id, task, context_json, budget_json, \ + status, runtime_generation, attempt, completion_slot_reserved, deadline_at, \ + revision, created_at, updated_at) \ + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, \ + 'queued', ?, 1, ?, ?, 0, ?, ?)", + ) + .bind(&run.id) + .bind(request.group.as_ref().map(|group| group.id.clone())) + .bind(&run.root_session_id) + .bind(&run.root_turn_id) + .bind(&run.parent_run_id) + .bind(&run.caller_agent_id) + .bind(&run.caller_scope_id) + .bind(&run.idempotency_key) + .bind(&run.agent_id) + .bind(&run.definition_hash) + .bind(&run.provider_profile) + .bind(&run.provider_name) + .bind(&run.model_id) + .bind(run.mode.as_str()) + .bind(run.depth) + .bind(&run.plan_item_id) + .bind(&run.execution_id) + .bind(&run.task) + .bind(&run.context_json) + .bind(&run.budget_json) + .bind(run.runtime_generation) + .bind(i64::from(run.completion_slot_reserved)) + .bind(run.deadline_at) + .bind(request.now) + .bind(request.now) + .execute(&mut *tx) + .await? + .rows_affected() + == 1; + if !inserted { + drop(tx); + return self.existing_agent_admission(request).await; + } + + if let Some(item_id) = run.plan_item_id.as_deref() { + claim_plan_item( + &mut tx, + &run.root_session_id, + item_id, + &run.execution_id, + request.now, + ) + .await?; + } + } + + tx.commit().await?; + + let mut runs = Vec::with_capacity(request.runs.len()); + for run in &request.runs { + runs.push(self.get_agent_run(&run.id).await?.ok_or_else(|| { + StorageError::NotFound(format!("agent run {} vanished after admission", run.id)) + })?); + } + let group = match request.group.as_ref() { + Some(group) => Some(self.get_agent_run_group(&group.id).await?.ok_or_else(|| { + StorageError::NotFound(format!("agent group {} vanished after admission", group.id)) + })?), + None => None, + }; + Ok(AcceptedAgentRuns::Accepted { group, runs }) + } + + async fn existing_agent_admission( + &self, + request: AcceptAgentRequest, + ) -> Result { + let mut runs = Vec::new(); + for run in &request.runs { + if let Some(record) = self.get_agent_run(&run.id).await? { + runs.push(record); + } + } + let group = match request.group.as_ref() { + Some(group) => self.get_agent_run_group(&group.id).await?, + None => None, + }; + if runs.is_empty() && group.is_none() { + return Err(StorageError::Conflict( + "agent admission conflicted but no existing rows were found".to_string(), + )); + } + Ok(AcceptedAgentRuns::Existing { group, runs }) + } + + pub async fn get_agent_run( + &self, + run_id: &str, + ) -> Result, StorageError> { + let row = sqlx::query(sqlx::AssertSqlSafe(format!( + "SELECT {RUN_COLUMNS} FROM agent_runs WHERE id = ?" + ))) + .bind(run_id) + .fetch_optional(&self.pool) + .await?; + match row { + Some(row) => Ok(Some(run_record_from_row(&row)?)), + None => Ok(None), + } + } + + pub async fn get_agent_run_group( + &self, + group_id: &str, + ) -> Result, StorageError> { + let row = sqlx::query(sqlx::AssertSqlSafe(format!( + "SELECT {GROUP_COLUMNS} FROM agent_run_groups WHERE id = ?" + ))) + .bind(group_id) + .fetch_optional(&self.pool) + .await?; + match row { + Some(row) => Ok(Some(group_record_from_row(&row)?)), + None => Ok(None), + } + } + + /// List runs for a session ordered by `(created_at DESC, id DESC)`. + /// The cursor is the pair of the last row the client has seen. + pub async fn list_agent_runs( + &self, + root_session_id: &str, + cursor: Option<(i64, String)>, + limit: i64, + ) -> Result, StorageError> { + let limit = limit.clamp(1, 200); + let rows = match cursor { + Some((created_at, id)) => { + sqlx::query(sqlx::AssertSqlSafe(format!( + "SELECT {RUN_COLUMNS} FROM agent_runs \ + WHERE root_session_id = ? AND (created_at < ? OR (created_at = ? AND id < ?)) \ + ORDER BY created_at DESC, id DESC LIMIT ?" + ))) + .bind(root_session_id) + .bind(created_at) + .bind(created_at) + .bind(id) + .bind(limit) + .fetch_all(&self.pool) + .await? + } + None => { + sqlx::query(sqlx::AssertSqlSafe(format!( + "SELECT {RUN_COLUMNS} FROM agent_runs \ + WHERE root_session_id = ? ORDER BY created_at DESC, id DESC LIMIT ?" + ))) + .bind(root_session_id) + .bind(limit) + .fetch_all(&self.pool) + .await? + } + }; + rows.iter().map(run_record_from_row).collect() + } + + /// All durable runs across sessions, newest first (management union). + pub async fn list_all_agent_runs( + &self, + cursor: Option<(i64, String)>, + limit: i64, + ) -> Result, StorageError> { + let limit = limit.clamp(1, 200); + let rows = match cursor { + Some((created_at, id)) => { + sqlx::query(sqlx::AssertSqlSafe(format!( + "SELECT {RUN_COLUMNS} FROM agent_runs \ + WHERE (created_at < ? OR (created_at = ? AND id < ?)) \ + ORDER BY created_at DESC, id DESC LIMIT ?" + ))) + .bind(created_at) + .bind(created_at) + .bind(id) + .bind(limit) + .fetch_all(&self.pool) + .await? + } + None => { + sqlx::query(sqlx::AssertSqlSafe(format!( + "SELECT {RUN_COLUMNS} FROM agent_runs \ + ORDER BY created_at DESC, id DESC LIMIT ?" + ))) + .bind(limit) + .fetch_all(&self.pool) + .await? + } + }; + rows.iter().map(run_record_from_row).collect() + } + + pub async fn list_agent_group_runs( + &self, + group_id: &str, + ) -> Result, StorageError> { + let rows = sqlx::query(sqlx::AssertSqlSafe(format!( + "SELECT {RUN_COLUMNS} FROM agent_runs WHERE group_id = ? ORDER BY created_at ASC, id ASC" + ))) + .bind(group_id) + .fetch_all(&self.pool) + .await?; + rows.iter().map(run_record_from_row).collect() + } + + /// Conditional `queued -> running` transition owned by this execution. + pub async fn mark_agent_run_running( + &self, + run_id: &str, + execution_id: &str, + now: i64, + ) -> Result { + let rows = sqlx::query( + "UPDATE agent_runs SET status = 'running', started_at = ?, updated_at = ? \ + WHERE id = ? AND execution_id = ? AND status = 'queued'", + ) + .bind(now) + .bind(now) + .bind(run_id) + .bind(execution_id) + .execute(&self.pool) + .await? + .rows_affected(); + Ok(rows == 1) + } + + /// Conditional transition into `waiting_children` from the expected + /// nonterminal status while this execution still owns the run. + pub async fn mark_agent_run_waiting_children( + &self, + run_id: &str, + execution_id: &str, + expected: AgentRunStatus, + now: i64, + ) -> Result { + if expected.is_terminal() { + return Err(StorageError::Conflict(format!( + "cannot wait on children from terminal status {}", + expected.as_str() + ))); + } + let rows = sqlx::query( + "UPDATE agent_runs SET status = 'waiting_children', updated_at = ? \ + WHERE id = ? AND execution_id = ? AND status = ?", + ) + .bind(now) + .bind(run_id) + .bind(execution_id) + .bind(expected.as_str()) + .execute(&self.pool) + .await? + .rows_affected(); + Ok(rows == 1) + } + + /// Restore a waiting parent to `running` once its children settled. A + /// run that was cancelled/timed out in the meantime keeps its terminal + /// state. + pub async fn restore_agent_run_running( + &self, + run_id: &str, + execution_id: &str, + now: i64, + ) -> Result { + let rows = sqlx::query( + "UPDATE agent_runs SET status = 'running', updated_at = ? \ + WHERE id = ? AND execution_id = ? AND status = 'waiting_children'", + ) + .bind(now) + .bind(run_id) + .bind(execution_id) + .execute(&self.pool) + .await? + .rows_affected(); + Ok(rows == 1) + } + + /// Cancel a nonterminal run. Returns true when this call owned the + /// transition. + pub async fn cancel_agent_run( + &self, + run_id: &str, + reason: &str, + now: i64, + ) -> Result { + let rows = sqlx::query( + "UPDATE agent_runs SET status = 'cancelled', error = ?, finished_at = ?, updated_at = ? \ + WHERE id = ? AND status IN ('queued', 'running', 'waiting_children')", + ) + .bind(reason) + .bind(now) + .bind(now) + .bind(run_id) + .execute(&self.pool) + .await? + .rows_affected(); + Ok(rows == 1) + } + + /// Cancel a nonterminal run and resolve its completion reservation in the + /// same transaction. With `suppress_continuation` (explicit `/stop` or + /// lifecycle cancellation) the completion event is written directly as + /// `consumed` so no continuation Turn restarts after cancellation; the + /// audit fact is preserved either way. + pub async fn cancel_agent_run_with_completion( + &self, + run_id: &str, + reason: &str, + suppress_continuation: bool, + now: i64, + ) -> Result { + let mut tx = self.pool.begin().await?; + let row = sqlx::query( + "SELECT completion_slot_reserved FROM agent_runs \ + WHERE id = ? AND status IN ('queued', 'running', 'waiting_children')", + ) + .bind(run_id) + .fetch_optional(&mut *tx) + .await?; + let Some(row) = row else { + return Ok(false); + }; + let reserved: bool = row.get::("completion_slot_reserved") != 0; + sqlx::query( + "UPDATE agent_runs SET status = 'cancelled', error = ?, finished_at = ?, updated_at = ? \ + WHERE id = ? AND status IN ('queued', 'running', 'waiting_children')", + ) + .bind(reason) + .bind(now) + .bind(now) + .bind(run_id) + .execute(&mut *tx) + .await?; + if reserved { + let session: String = + sqlx::query_scalar("SELECT root_session_id FROM agent_runs WHERE id = ?") + .bind(run_id) + .fetch_one(&mut *tx) + .await?; + let revision: i64 = sqlx::query_scalar( + "UPDATE agent_session_state \ + SET reserved_completion_slots = MAX(reserved_completion_slots - 1, 0), \ + revision = revision + 1, updated_at = ? \ + WHERE root_session_id = ? RETURNING revision", + ) + .bind(now) + .bind(&session) + .fetch_one(&mut *tx) + .await?; + let event = super::agent_inbox::NewInboxEvent { + id: uuid::Uuid::new_v4().to_string(), + root_session_id: session, + scope_kind: "run".to_string(), + scope_id: run_id.to_string(), + run_id: Some(run_id.to_string()), + group_id: None, + event_type: super::agent_inbox::AgentEventType::Completion, + event_key: format!("completion:{run_id}"), + delivery: super::agent_inbox::AgentEventDelivery::Queue, + requires_continuation: !suppress_continuation, + severity: Some("warning".to_string()), + payload_json: serde_json::json!({ "status": "cancelled", "error": reason }) + .to_string(), + }; + if suppress_continuation { + // Directly consumed: pending count never grows. + sqlx::query( + "INSERT INTO agent_inbox_events (id, root_session_id, scope_kind, scope_id, \ + run_id, group_id, event_type, event_key, delivery, requires_continuation, \ + severity, payload_json, status, attempt_count, revision, next_attempt_at, \ + created_at, updated_at, consumed_at) \ + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'consumed', 0, ?, NULL, ?, ?, ?)", + ) + .bind(&event.id) + .bind(&event.root_session_id) + .bind(&event.scope_kind) + .bind(&event.scope_id) + .bind(&event.run_id) + .bind(&event.group_id) + .bind(event.event_type.as_str()) + .bind(&event.event_key) + .bind(event.delivery.as_str()) + .bind(i64::from(event.requires_continuation)) + .bind(&event.severity) + .bind(&event.payload_json) + .bind(revision) + .bind(now) + .bind(now) + .bind(now) + .execute(&mut *tx) + .await?; + } else { + super::agent_inbox::insert_event_tx(&mut tx, &event, revision, now).await?; + } + } + tx.commit().await?; + Ok(true) + } + + /// Atomically commit a terminal outcome. The conditional update makes + /// exactly one writer the owner; late results from stale executions + /// update zero rows and return `None`. + pub async fn commit_agent_terminal( + &self, + run_id: &str, + execution_id: &str, + runtime_generation: i64, + outcome: &AgentTerminalOutcome, + plan_summary: Option<&str>, + now: i64, + ) -> Result, StorageError> { + let mut tx = self.pool.begin().await?; + + let (result, error, prompt_tokens, completion_tokens, cost, tool_calls, iterations) = + match outcome { + AgentTerminalOutcome::Completed { + result, + prompt_tokens, + completion_tokens, + cost, + tool_calls, + iterations, + } => ( + Some(result.as_str()), + None, + *prompt_tokens, + *completion_tokens, + *cost, + *tool_calls, + *iterations, + ), + AgentTerminalOutcome::Failed { + error, + prompt_tokens, + completion_tokens, + cost, + } => ( + None, + Some(error.as_str()), + *prompt_tokens, + *completion_tokens, + *cost, + 0, + 0, + ), + AgentTerminalOutcome::TimedOut { .. } => { + (None, Some("deadline exceeded"), None, None, None, 0, 0) + } + AgentTerminalOutcome::Cancelled { reason } => { + (None, Some(reason.as_str()), None, None, None, 0, 0) + } + AgentTerminalOutcome::Interrupted { reason } => { + (None, Some(reason.as_str()), None, None, None, 0, 0) + } + }; + + let updated = sqlx::query( + "UPDATE agent_runs SET status = ?, result = ?, error = ?, prompt_tokens = ?, \ + completion_tokens = ?, cost = ?, tool_calls_count = ?, iterations = ?, \ + finished_at = ?, updated_at = ? \ + WHERE id = ? AND execution_id = ? AND runtime_generation = ? \ + AND status IN ('queued', 'running', 'waiting_children')", + ) + .bind(outcome.status().as_str()) + .bind(result) + .bind(error) + .bind(prompt_tokens) + .bind(completion_tokens) + .bind(cost) + .bind(tool_calls) + .bind(iterations) + .bind(now) + .bind(now) + .bind(run_id) + .bind(execution_id) + .bind(runtime_generation) + .execute(&mut *tx) + .await? + .rows_affected(); + if updated != 1 { + return Ok(None); + } + + let run_row = sqlx::query(sqlx::AssertSqlSafe(format!( + "SELECT {RUN_COLUMNS} FROM agent_runs WHERE id = ?" + ))) + .bind(run_id) + .fetch_one(&mut *tx) + .await?; + let run = run_record_from_row(&run_row)?; + + let mut group = None; + let mut group_finished = false; + if let Some(group_id) = run.group_id.as_deref() { + sqlx::query( + "UPDATE agent_run_groups SET terminal_runs = terminal_runs + 1, \ + abnormal_runs = abnormal_runs + ?, updated_at = ? WHERE id = ?", + ) + .bind(i64::from(outcome.is_abnormal())) + .bind(now) + .bind(group_id) + .execute(&mut *tx) + .await?; + let group_row = sqlx::query(sqlx::AssertSqlSafe(format!( + "SELECT {GROUP_COLUMNS} FROM agent_run_groups WHERE id = ?" + ))) + .bind(group_id) + .fetch_one(&mut *tx) + .await?; + let mut record = group_record_from_row(&group_row)?; + if record.terminal_runs >= record.expected_runs && !record.status.is_terminal() { + let final_status = if record.abnormal_runs == 0 { + AgentGroupStatus::Completed + } else if record.abnormal_runs < record.expected_runs { + AgentGroupStatus::Partial + } else { + AgentGroupStatus::Failed + }; + sqlx::query( + "UPDATE agent_run_groups SET status = ?, finished_at = ?, updated_at = ? \ + WHERE id = ? AND status IN ('queued', 'running')", + ) + .bind(final_status.as_str()) + .bind(now) + .bind(now) + .bind(group_id) + .execute(&mut *tx) + .await?; + record.status = final_status; + record.finished_at = Some(now); + group_finished = true; + } + group = Some(record); + } + + if let Some(item_id) = run.plan_item_id.as_deref() { + finish_plan_item( + &mut tx, + &run.root_session_id, + item_id, + &run.execution_id, + matches!(outcome, AgentTerminalOutcome::Completed { .. }), + plan_summary, + now, + ) + .await?; + } + + // Background runs that reserved a completion slot convert the + // reservation into a durable completion event in the same commit. + // The event survives restarts, queue-full conditions and lost wakes. + if run.completion_slot_reserved { + let (status, error) = match outcome { + AgentTerminalOutcome::Completed { .. } => ("completed", None), + AgentTerminalOutcome::Failed { error, .. } => ("failed", Some(error.as_str())), + AgentTerminalOutcome::TimedOut { .. } => ("timed_out", Some("deadline exceeded")), + AgentTerminalOutcome::Cancelled { reason } => ("cancelled", Some(reason.as_str())), + AgentTerminalOutcome::Interrupted { reason } => { + ("interrupted", Some(reason.as_str())) + } + }; + super::agent_inbox::insert_completion_event_tx( + &mut tx, + &run.id, + &run.root_session_id, + status, + error, + now, + ) + .await?; + } + + tx.commit().await?; + Ok(Some(TerminalCommit { + run, + group, + group_finished, + })) + } +} + +async fn claim_plan_item( + tx: &mut SqliteConnection, + session_id: &str, + item_id: &str, + execution_id: &str, + now: i64, +) -> Result<(), StorageError> { + let plan_id: Option = sqlx::query_scalar( + "SELECT id FROM task_plans WHERE session_id = ? AND status = 'active' LIMIT 1", + ) + .bind(session_id) + .fetch_optional(&mut *tx) + .await?; + let Some(plan_id) = plan_id else { + return Err(StorageError::Conflict(format!( + "plan item {item_id} cannot be claimed without an active plan" + ))); + }; + let rows = sqlx::query( + "UPDATE task_items SET status = 'in_progress', executor_kind = 'sub_agent', \ + execution_id = ?, error = NULL, version = version + 1, updated_at = ? \ + WHERE plan_id = ? AND id = ? AND status = 'pending'", + ) + .bind(execution_id) + .bind(now) + .bind(&plan_id) + .bind(item_id) + .execute(&mut *tx) + .await? + .rows_affected(); + if rows != 1 { + return Err(StorageError::Conflict(format!( + "plan item {item_id} was already claimed by another execution" + ))); + } + bump_plan_version(tx, &plan_id, now).await +} + +async fn finish_plan_item( + tx: &mut SqliteConnection, + session_id: &str, + item_id: &str, + execution_id: &str, + completed: bool, + summary: Option<&str>, + now: i64, +) -> Result<(), StorageError> { + let plan_id: Option = sqlx::query_scalar( + "SELECT id FROM task_plans WHERE session_id = ? AND status = 'active' LIMIT 1", + ) + .bind(session_id) + .fetch_optional(&mut *tx) + .await?; + let Some(plan_id) = plan_id else { + return Ok(()); + }; + let status = if completed { "completed" } else { "blocked" }; + let rows = sqlx::query( + "UPDATE task_items SET status = ?, result_summary = ?, error = ?, \ + version = version + 1, updated_at = ? \ + WHERE plan_id = ? AND id = ? AND execution_id = ? AND status = 'in_progress'", + ) + .bind(status) + .bind(completed.then_some(summary).flatten()) + .bind((!completed).then_some(summary).flatten()) + .bind(now) + .bind(&plan_id) + .bind(item_id) + .bind(execution_id) + .execute(&mut *tx) + .await? + .rows_affected(); + if rows != 1 { + return Ok(()); + } + bump_plan_version(tx, &plan_id, now).await +} + +async fn bump_plan_version( + tx: &mut SqliteConnection, + plan_id: &str, + now: i64, +) -> Result<(), StorageError> { + sqlx::query("UPDATE task_plans SET version = version + 1, updated_at = ? WHERE id = ?") + .bind(now) + .bind(plan_id) + .execute(&mut *tx) + .await?; + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use tempfile::TempDir; + + async fn create_test_storage() -> (super::super::Storage, TempDir) { + let dir = tempfile::tempdir().unwrap(); + let db_path = dir.path().join("agent.db"); + let storage = super::super::Storage::new(&db_path).await.unwrap(); + (storage, dir) + } + + fn new_run(id: &str, execution_id: &str, session: &str) -> NewAgentRun { + NewAgentRun { + id: id.to_string(), + root_session_id: session.to_string(), + root_turn_id: None, + parent_run_id: None, + caller_agent_id: "ROOT".to_string(), + caller_scope_id: "turn-1".to_string(), + idempotency_key: None, + agent_id: "researcher".to_string(), + definition_hash: "hash".to_string(), + provider_profile: "research".to_string(), + provider_name: "test".to_string(), + model_id: "test-model".to_string(), + mode: AgentRunMode::Foreground, + depth: 1, + plan_item_id: None, + execution_id: execution_id.to_string(), + task: "do the work".to_string(), + context_json: None, + budget_json: "{\"remaining_runs\":15}".to_string(), + deadline_at: 1_000, + runtime_generation: 1, + completion_slot_reserved: false, + } + } + + #[tokio::test] + async fn fresh_database_creates_schema_v6_agent_tables() { + let (storage, _dir) = create_test_storage().await; + let version: i64 = sqlx::query_scalar("PRAGMA user_version") + .fetch_one(storage.pool()) + .await + .unwrap(); + assert_eq!(version, 6); + for table in [ + "agent_run_groups", + "agent_runs", + "agent_session_state", + "agent_inbox_events", + ] { + let exists: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = ?", + ) + .bind(table) + .fetch_one(storage.pool()) + .await + .unwrap(); + assert_eq!(exists, 1, "missing table {table}"); + } + } + + #[tokio::test] + async fn single_run_admission_persists_queued_without_group() { + let (storage, _dir) = create_test_storage().await; + let accepted = storage + .accept_agent_runs(AcceptAgentRequest { + group: None, + runs: vec![new_run("run-1", "exec-1", "cli:test:d1")], + now: 100, + }) + .await + .unwrap(); + assert!(matches!(accepted, AcceptedAgentRuns::Accepted { .. })); + + let run = storage.get_agent_run("run-1").await.unwrap().unwrap(); + assert_eq!(run.status, AgentRunStatus::Queued); + assert!(run.group_id.is_none()); + assert_eq!(run.execution_id, "exec-1"); + } + + #[tokio::test] + async fn batch_admission_tracks_group_terminal_counters() { + let (storage, _dir) = create_test_storage().await; + let group = NewAgentGroup { + id: "group-1".to_string(), + root_session_id: "cli:test:d1".to_string(), + caller_run_id: None, + caller_scope_id: "turn-1".to_string(), + idempotency_key: Some("batch-key".to_string()), + mode: AgentRunMode::Foreground, + completion_policy: AgentCompletionPolicy::All, + deadline_at: 2_000, + runtime_generation: 1, + }; + storage + .accept_agent_runs(AcceptAgentRequest { + group: Some(group), + runs: vec![ + new_run("run-a", "exec-a", "cli:test:d1"), + new_run("run-b", "exec-b", "cli:test:d1"), + ], + now: 100, + }) + .await + .unwrap(); + + assert!( + storage + .mark_agent_run_running("run-a", "exec-a", 110) + .await + .unwrap() + ); + let first = storage + .commit_agent_terminal( + "run-a", + "exec-a", + 1, + &AgentTerminalOutcome::Completed { + result: "done".to_string(), + prompt_tokens: Some(2), + completion_tokens: Some(3), + cost: None, + tool_calls: 1, + iterations: 2, + }, + None, + 120, + ) + .await + .unwrap() + .unwrap(); + assert!(!first.group_finished); + assert_eq!(first.group.as_ref().unwrap().terminal_runs, 1); + + let second = storage + .commit_agent_terminal( + "run-b", + "exec-b", + 1, + &AgentTerminalOutcome::Failed { + error: "boom".to_string(), + prompt_tokens: None, + completion_tokens: None, + cost: None, + }, + None, + 130, + ) + .await + .unwrap() + .unwrap(); + assert!(second.group_finished); + let group = second.group.unwrap(); + assert_eq!(group.status, AgentGroupStatus::Partial); + assert_eq!(group.finished_at, Some(130)); + + let runs = storage.list_agent_group_runs("group-1").await.unwrap(); + assert_eq!(runs.len(), 2); + } + + #[tokio::test] + async fn stale_execution_cannot_commit_terminal() { + let (storage, _dir) = create_test_storage().await; + storage + .accept_agent_runs(AcceptAgentRequest { + group: None, + runs: vec![new_run("run-1", "exec-1", "cli:test:d1")], + now: 100, + }) + .await + .unwrap(); + + let stale = storage + .commit_agent_terminal( + "run-1", + "exec-other", + 1, + &AgentTerminalOutcome::Completed { + result: "late".to_string(), + prompt_tokens: None, + completion_tokens: None, + cost: None, + tool_calls: 0, + iterations: 0, + }, + None, + 120, + ) + .await + .unwrap(); + assert!(stale.is_none()); + + let run = storage.get_agent_run("run-1").await.unwrap().unwrap(); + assert_eq!(run.status, AgentRunStatus::Queued); + assert!(run.result.is_none()); + } + + #[tokio::test] + async fn waiting_children_transitions_are_conditional() { + let (storage, _dir) = create_test_storage().await; + storage + .accept_agent_runs(AcceptAgentRequest { + group: None, + runs: vec![new_run("run-1", "exec-1", "cli:test:d1")], + now: 100, + }) + .await + .unwrap(); + storage + .mark_agent_run_running("run-1", "exec-1", 110) + .await + .unwrap(); + + assert!( + storage + .mark_agent_run_waiting_children("run-1", "exec-1", AgentRunStatus::Running, 120) + .await + .unwrap() + ); + let run = storage.get_agent_run("run-1").await.unwrap().unwrap(); + assert_eq!(run.status, AgentRunStatus::WaitingChildren); + + assert!( + storage + .restore_agent_run_running("run-1", "exec-1", 130) + .await + .unwrap() + ); + let run = storage.get_agent_run("run-1").await.unwrap().unwrap(); + assert_eq!(run.status, AgentRunStatus::Running); + } + + #[tokio::test] + async fn plan_item_claim_is_atomic_with_run_admission() { + let (storage, _dir) = create_test_storage().await; + sqlx::query( + "INSERT INTO sessions (id, channel, chat_id, dialog_id, created_at, last_active_at) VALUES ('cli:test:d1', 'cli', 'test', 'd1', 1, 1)", + ) + .execute(storage.pool()) + .await + .unwrap(); + sqlx::query( + "INSERT INTO task_plans (id, session_id, objective, status, version, created_at, updated_at) VALUES ('plan-1', 'cli:test:d1', 'obj', 'active', 1, 1, 1)", + ) + .execute(storage.pool()) + .await + .unwrap(); + sqlx::query( + "INSERT INTO task_items (id, plan_id, ordinal, title, status, version, created_at, updated_at) VALUES ('T1', 'plan-1', 1, 'work', 'pending', 1, 1, 1)", + ) + .execute(storage.pool()) + .await + .unwrap(); + + let mut run = new_run("run-1", "exec-1", "cli:test:d1"); + run.plan_item_id = Some("T1".to_string()); + storage + .accept_agent_runs(AcceptAgentRequest { + group: None, + runs: vec![run.clone()], + now: 100, + }) + .await + .unwrap(); + let status: String = sqlx::query_scalar("SELECT status FROM task_items WHERE id = 'T1'") + .fetch_one(storage.pool()) + .await + .unwrap(); + assert_eq!(status, "in_progress"); + + // A second admission for the same item must roll back entirely. + run.id = "run-2".to_string(); + run.execution_id = "exec-2".to_string(); + let error = storage + .accept_agent_runs(AcceptAgentRequest { + group: None, + runs: vec![run], + now: 110, + }) + .await + .unwrap_err(); + assert!(matches!(error, StorageError::Conflict(_))); + assert!(storage.get_agent_run("run-2").await.unwrap().is_none()); + + // Terminal commit releases the item as completed with the summary. + storage + .commit_agent_terminal( + "run-1", + "exec-1", + 1, + &AgentTerminalOutcome::Completed { + result: "done".to_string(), + prompt_tokens: None, + completion_tokens: None, + cost: None, + tool_calls: 0, + iterations: 0, + }, + Some("finished the work"), + 120, + ) + .await + .unwrap(); + let (status, summary): (String, Option) = + sqlx::query_as("SELECT status, result_summary FROM task_items WHERE id = 'T1'") + .fetch_one(storage.pool()) + .await + .unwrap(); + assert_eq!(status, "completed"); + assert_eq!(summary.as_deref(), Some("finished the work")); + } + + #[tokio::test] + async fn list_agent_runs_paginates_with_created_at_cursor() { + let (storage, _dir) = create_test_storage().await; + let mut runs = Vec::new(); + for index in 0..5 { + runs.push(new_run( + &format!("run-{index}"), + &format!("exec-{index}"), + "cli:test:d1", + )); + } + storage + .accept_agent_runs(AcceptAgentRequest { + group: None, + runs, + now: 100, + }) + .await + .unwrap(); + + let first_page = storage + .list_agent_runs("cli:test:d1", None, 2) + .await + .unwrap(); + assert_eq!(first_page.len(), 2); + let last = first_page.last().unwrap(); + let second_page = storage + .list_agent_runs("cli:test:d1", Some((last.created_at, last.id.clone())), 10) + .await + .unwrap(); + assert_eq!(second_page.len(), 3); + let seen: std::collections::HashSet<_> = first_page + .iter() + .chain(second_page.iter()) + .map(|run| run.id.clone()) + .collect(); + assert_eq!(seen.len(), 5); + } + + #[tokio::test] + async fn cancel_agent_run_only_transitions_nonterminal_rows() { + let (storage, _dir) = create_test_storage().await; + storage + .accept_agent_runs(AcceptAgentRequest { + group: None, + runs: vec![new_run("run-1", "exec-1", "cli:test:d1")], + now: 100, + }) + .await + .unwrap(); + assert!( + storage + .cancel_agent_run("run-1", "stopped", 110) + .await + .unwrap() + ); + assert!( + !storage + .cancel_agent_run("run-1", "stopped", 120) + .await + .unwrap() + ); + let run = storage.get_agent_run("run-1").await.unwrap().unwrap(); + assert_eq!(run.status, AgentRunStatus::Cancelled); + assert_eq!(run.error.as_deref(), Some("stopped")); + } + + #[tokio::test] + async fn suppress_cancel_converts_reservation_to_consumed_completion() { + let (storage, _dir) = create_test_storage().await; + let mut run = new_run("run-1", "exec-1", "cli:test:d1"); + run.mode = AgentRunMode::Background; + run.completion_slot_reserved = true; + storage + .accept_agent_runs(AcceptAgentRequest { + group: None, + runs: vec![run], + now: 100, + }) + .await + .unwrap(); + storage + .reserve_completion_slots("cli:test:d1", 1, 16, 100) + .await + .unwrap(); + + assert!( + storage + .cancel_agent_run_with_completion("run-1", "stopped", true, 110) + .await + .unwrap() + ); + let run = storage.get_agent_run("run-1").await.unwrap().unwrap(); + assert_eq!(run.status, AgentRunStatus::Cancelled); + + let events = storage + .list_agent_inbox_events("cli:test:d1", 10) + .await + .unwrap(); + assert_eq!(events.len(), 1); + assert_eq!( + events[0].status, + crate::storage::agent_inbox::AgentEventStatus::Consumed + ); + assert!(!events[0].requires_continuation); + + let state: (i64, i64) = sqlx::query_as( + "SELECT pending_event_count, reserved_completion_slots FROM agent_session_state \ + WHERE root_session_id = 'cli:test:d1'", + ) + .fetch_one(storage.pool()) + .await + .unwrap(); + assert_eq!(state, (0, 0)); + + // A second cancel is a no-op. + assert!( + !storage + .cancel_agent_run_with_completion("run-1", "again", true, 120) + .await + .unwrap() + ); + } +} diff --git a/src/storage/message.rs b/src/storage/message.rs index 11b3123..57ef4cc 100644 --- a/src/storage/message.rs +++ b/src/storage/message.rs @@ -1,6 +1,6 @@ use serde::{Deserialize, Serialize}; -use crate::bus::CompletionStatus; +use crate::bus::{ClientVisibility, CompletionStatus, TurnOrigin}; #[derive(Debug, Clone, Serialize, Deserialize)] pub struct MessageMeta { @@ -14,6 +14,8 @@ pub struct MessageMeta { pub turn_id: Option, pub iteration: Option, pub completion_status: CompletionStatus, + pub client_visibility: ClientVisibility, + pub turn_origin: TurnOrigin, pub media_refs: Option, pub tool_call_id: Option, pub tool_name: Option, diff --git a/src/storage/mod.rs b/src/storage/mod.rs index 37d4f79..0020ca1 100644 --- a/src/storage/mod.rs +++ b/src/storage/mod.rs @@ -1,3 +1,5 @@ +pub mod agent_inbox; +pub mod agent_run; pub mod background_task; pub mod error; pub mod memory; @@ -18,17 +20,17 @@ use sqlx::{Pool, Row, Sqlite}; use std::path::Path; use tokio::time::{Duration, sleep}; -const SCHEMA_VERSION: i64 = 5; +const SCHEMA_VERSION: i64 = 6; const INSERT_MESSAGE_SQL: &str = r#" INSERT INTO messages ( id, session_id, seq, role, content, reasoning_content, provider_state, - turn_id, iteration, completion_status, media_refs, tool_call_id, - tool_name, tool_calls, source, created_at + turn_id, iteration, completion_status, client_visibility, turn_origin, + media_refs, tool_call_id, tool_name, tool_calls, source, created_at ) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) "#; -fn insert_message_query<'a>( +pub(crate) fn insert_message_query<'a>( session_id: &'a str, msg: &'a crate::storage::message::MessageMeta, ) -> sqlx::query::Query<'a, Sqlite, sqlx::sqlite::SqliteArguments> { @@ -43,6 +45,8 @@ fn insert_message_query<'a>( .bind(&msg.turn_id) .bind(msg.iteration) .bind(msg.completion_status.as_str()) + .bind(msg.client_visibility.as_str()) + .bind(msg.turn_origin.as_str()) .bind(&msg.media_refs) .bind(&msg.tool_call_id) .bind(&msg.tool_name) @@ -64,6 +68,8 @@ fn message_meta_from_row(row: SqliteRow) -> crate::storage::message::MessageMeta turn_id: row.get("turn_id"), iteration: row.get("iteration"), completion_status: crate::bus::CompletionStatus::from_storage(&completion_status), + client_visibility: parse_visibility(row.get("client_visibility")), + turn_origin: parse_turn_origin(row.get("turn_origin")), media_refs: row.get("media_refs"), tool_call_id: row.get("tool_call_id"), tool_name: row.get("tool_name"), @@ -73,6 +79,22 @@ fn message_meta_from_row(row: SqliteRow) -> crate::storage::message::MessageMeta } } +fn parse_visibility(value: String) -> crate::bus::ClientVisibility { + if value == "hidden" { + crate::bus::ClientVisibility::Hidden + } else { + crate::bus::ClientVisibility::Visible + } +} + +fn parse_turn_origin(value: String) -> crate::bus::TurnOrigin { + match value.as_str() { + "agent_continuation" => crate::bus::TurnOrigin::AgentContinuation, + "scheduled" => crate::bus::TurnOrigin::Scheduled, + _ => crate::bus::TurnOrigin::User, + } +} + pub struct Storage { pub(crate) pool: Pool, } @@ -115,6 +137,8 @@ impl Storage { deleted_at INTEGER, last_consolidated_at INTEGER, last_compressed_message_at INTEGER, + delivery_context TEXT, + delivery_context_updated_at INTEGER, UNIQUE(channel, chat_id, dialog_id) ) "#, @@ -149,6 +173,8 @@ impl Storage { turn_id TEXT, iteration INTEGER, completion_status TEXT NOT NULL DEFAULT 'completed', + client_visibility TEXT NOT NULL DEFAULT 'visible', + turn_origin TEXT NOT NULL DEFAULT 'user', created_at INTEGER NOT NULL, FOREIGN KEY (session_id) REFERENCES sessions(id) ON DELETE CASCADE ) @@ -424,6 +450,16 @@ impl Storage { "completion_status", "completion_status TEXT NOT NULL DEFAULT 'completed'", ), + ( + "messages", + "client_visibility", + "client_visibility TEXT NOT NULL DEFAULT 'visible'", + ), + ( + "messages", + "turn_origin", + "turn_origin TEXT NOT NULL DEFAULT 'user'", + ), ("sessions", "archived_at", "archived_at INTEGER"), ( "sessions", @@ -435,6 +471,12 @@ impl Storage { "last_compressed_message_at", "last_compressed_message_at INTEGER", ), + ("sessions", "delivery_context", "delivery_context TEXT"), + ( + "sessions", + "delivery_context_updated_at", + "delivery_context_updated_at INTEGER", + ), ("scheduled_jobs", "locked_at", "locked_at INTEGER"), ("scheduled_jobs", "lock_owner", "lock_owner TEXT"), ("scheduled_jobs", "lease_until", "lease_until INTEGER"), @@ -521,6 +563,14 @@ impl Storage { ) .execute(&mut *tx) .await?; + sqlx::query( + "CREATE INDEX IF NOT EXISTS idx_messages_session_visibility_seq ON messages(session_id, client_visibility, seq)", + ) + .execute(&mut *tx) + .await?; + for statement in agent_run::AGENT_SCHEMA_STATEMENTS { + sqlx::query(*statement).execute(&mut *tx).await?; + } sqlx::query(sqlx::AssertSqlSafe(format!( "PRAGMA user_version = {SCHEMA_VERSION}" ))) @@ -863,7 +913,9 @@ impl Storage { msgs: &[crate::storage::message::MessageMeta], meta: &crate::storage::session::SessionMeta, usage: Option<&crate::storage::TurnUsageRecord>, + steer: Option<&crate::storage::agent_inbox::SteerConsumption>, ) -> Result<(), StorageError> { + let now = chrono::Utc::now().timestamp_millis(); let mut tx = self.pool.begin().await?; for msg in msgs { @@ -935,6 +987,38 @@ impl Storage { .await?; } + if let Some(steer) = steer { + let mut consumed = 0i64; + for event_id in &steer.event_ids { + let rows = sqlx::query( + "UPDATE agent_inbox_events SET status = 'consumed', consumed_at = ?, \ + updated_at = ? \ + WHERE id = ? AND status = 'admitted' AND admitted_turn_id = ?", + ) + .bind(now) + .bind(now) + .bind(event_id) + .bind(&steer.admitted_turn_id) + .execute(&mut *tx) + .await? + .rows_affected(); + consumed += rows as i64; + } + if consumed > 0 { + sqlx::query( + "UPDATE agent_session_state \ + SET pending_event_count = MAX(pending_event_count - ?, 0), \ + revision = revision + 1, updated_at = ? \ + WHERE root_session_id = ?", + ) + .bind(consumed) + .bind(now) + .bind(session_id) + .execute(&mut *tx) + .await?; + } + } + tx.commit().await?; Ok(()) } @@ -945,7 +1029,7 @@ impl Storage { msgs: &[crate::storage::message::MessageMeta], meta: &crate::storage::session::SessionMeta, ) -> Result<(), StorageError> { - self.persist_message_batch_inner(session_id, msgs, meta, None) + self.persist_message_batch_inner(session_id, msgs, meta, None, None) .await } @@ -956,7 +1040,22 @@ impl Storage { meta: &crate::storage::session::SessionMeta, usage: &crate::storage::TurnUsageRecord, ) -> Result<(), StorageError> { - self.persist_message_batch_inner(session_id, msgs, meta, Some(usage)) + self.persist_message_batch_inner(session_id, msgs, meta, Some(usage), None) + .await + } + + /// Persist a Turn and consume the admitted steer events of the same Turn + /// in one transaction: the steer inputs appear in history exactly when + /// their durable events become `consumed`. + pub async fn persist_turn_batch_with_steer_consumption( + &self, + session_id: &str, + msgs: &[crate::storage::message::MessageMeta], + meta: &crate::storage::session::SessionMeta, + usage: &crate::storage::TurnUsageRecord, + steer: &crate::storage::agent_inbox::SteerConsumption, + ) -> Result<(), StorageError> { + self.persist_message_batch_inner(session_id, msgs, meta, Some(usage), Some(steer)) .await } @@ -1003,6 +1102,32 @@ impl Storage { unreachable!() } + /// Persist a Turn with steer consumption and bounded retry. + pub async fn persist_turn_with_steer_with_retry( + &self, + session_id: &str, + msgs: &[crate::storage::message::MessageMeta], + meta: &crate::storage::session::SessionMeta, + usage: &crate::storage::TurnUsageRecord, + steer: &crate::storage::agent_inbox::SteerConsumption, + ) -> Result<(), StorageError> { + let delays = [100, 200, 300]; + for (attempt, delay) in delays.iter().enumerate() { + match self + .persist_turn_batch_with_steer_consumption(session_id, msgs, meta, usage, steer) + .await + { + Ok(()) => return Ok(()), + Err(error) if attempt < delays.len() - 1 && error.is_transient() => { + tracing::warn!(attempt = attempt + 1, error = %error, "Turn persistence failed; retrying"); + sleep(Duration::from_millis(*delay)).await; + } + Err(error) => return Err(error), + } + } + unreachable!() + } + pub async fn get_session_usage_totals( &self, session_id: &str, @@ -1064,8 +1189,8 @@ impl Storage { let rows = sqlx::query( r#" SELECT id, session_id, seq, role, content, reasoning_content, provider_state, - turn_id, iteration, completion_status, media_refs, tool_call_id, - tool_name, tool_calls, source, created_at + turn_id, iteration, completion_status, client_visibility, turn_origin, + media_refs, tool_call_id, tool_name, tool_calls, source, created_at FROM messages WHERE session_id = ? AND seq >= ? ORDER BY seq ASC @@ -1087,8 +1212,8 @@ impl Storage { let row = sqlx::query( r#" SELECT id, session_id, seq, role, content, reasoning_content, provider_state, - turn_id, iteration, completion_status, media_refs, tool_call_id, - tool_name, tool_calls, source, created_at + turn_id, iteration, completion_status, client_visibility, turn_origin, + media_refs, tool_call_id, tool_name, tool_calls, source, created_at FROM messages WHERE session_id = ? AND id = ? "#, @@ -1121,14 +1246,14 @@ impl Storage { let rows = sqlx::query( r#" SELECT id, session_id, seq, role, content, reasoning_content, provider_state, - turn_id, iteration, completion_status, media_refs, tool_call_id, - tool_name, tool_calls, source, created_at + turn_id, iteration, completion_status, client_visibility, turn_origin, + media_refs, tool_call_id, tool_name, tool_calls, source, created_at FROM ( SELECT id, session_id, seq, role, content, reasoning_content, provider_state, - turn_id, iteration, completion_status, media_refs, tool_call_id, - tool_name, tool_calls, source, created_at + turn_id, iteration, completion_status, client_visibility, turn_origin, + media_refs, tool_call_id, tool_name, tool_calls, source, created_at FROM messages - WHERE session_id = ? + WHERE session_id = ? AND client_visibility = 'visible' ORDER BY seq DESC LIMIT ? ) @@ -1151,8 +1276,8 @@ impl Storage { let rows = sqlx::query( r#" SELECT id, session_id, seq, role, content, reasoning_content, provider_state, - turn_id, iteration, completion_status, media_refs, tool_call_id, - tool_name, tool_calls, source, created_at + turn_id, iteration, completion_status, client_visibility, turn_origin, + media_refs, tool_call_id, tool_name, tool_calls, source, created_at FROM messages WHERE session_id = ? AND created_at > ? ORDER BY seq ASC @@ -1221,8 +1346,8 @@ impl Storage { let rows = sqlx::query( r#" SELECT id, session_id, seq, role, content, reasoning_content, provider_state, - turn_id, iteration, completion_status, media_refs, tool_call_id, - tool_name, tool_calls, source, created_at + turn_id, iteration, completion_status, client_visibility, turn_origin, + media_refs, tool_call_id, tool_name, tool_calls, source, created_at FROM messages WHERE session_id = ? ORDER BY seq DESC @@ -1256,16 +1381,16 @@ impl Storage { } let count_sql = format!( - "SELECT COUNT(*) as total FROM messages WHERE session_id = ?{}", + "SELECT COUNT(*) as total FROM messages WHERE session_id = ? AND client_visibility = 'visible'{}", where_extra ); let select_sql = format!( r#" SELECT id, session_id, seq, role, content, reasoning_content, provider_state, - turn_id, iteration, completion_status, media_refs, tool_call_id, - tool_name, tool_calls, source, created_at + turn_id, iteration, completion_status, client_visibility, turn_origin, + media_refs, tool_call_id, tool_name, tool_calls, source, created_at FROM messages - WHERE session_id = ?{} + WHERE session_id = ? AND client_visibility = 'visible'{} ORDER BY seq ASC LIMIT ? OFFSET ? "#, @@ -1504,6 +1629,42 @@ impl Storage { .collect()) } + /// Persist the channel's durable delivery context for a session. Only + /// channel-declared reusable values (thread/root identity) ever reach + /// this column; one-shot reply/reaction ids never do. + pub async fn update_session_delivery_context( + &self, + session_id: &str, + context_json: &str, + now: i64, + ) -> Result<(), StorageError> { + sqlx::query( + "UPDATE sessions SET delivery_context = ?, delivery_context_updated_at = ? \ + WHERE id = ?", + ) + .bind(context_json) + .bind(now) + .bind(session_id) + .execute(self.pool()) + .await?; + Ok(()) + } + + /// Durable delivery context previously saved for the session. + pub async fn get_session_delivery_context( + &self, + session_id: &str, + ) -> Result, StorageError> { + let context: Option = sqlx::query_scalar( + "SELECT delivery_context FROM sessions \ + WHERE id = ? AND delivery_context IS NOT NULL", + ) + .bind(session_id) + .fetch_optional(self.pool()) + .await?; + Ok(context) + } + pub async fn cleanup_old_tasks(&self, ttl_ms: i64) -> Result { let cutoff = chrono::Utc::now().timestamp_millis() - ttl_ms; let result = sqlx::query( @@ -1839,6 +2000,18 @@ mod tests { .execute(&pool) .await .unwrap(); + sqlx::query( + "INSERT INTO sessions (id, channel, chat_id, dialog_id, created_at, last_active_at) VALUES ('cli:c:d', 'cli', 'c', 'd', 1, 1)", + ) + .execute(&pool) + .await + .unwrap(); + sqlx::query( + "INSERT INTO messages (id, session_id, seq, role, content, created_at) VALUES ('m1', 'cli:c:d', 1, 'user', 'legacy message', 1)", + ) + .execute(&pool) + .await + .unwrap(); drop(pool); let storage = Storage::new(&db_path).await.unwrap(); @@ -1852,6 +2025,8 @@ mod tests { "turn_id", "iteration", "completion_status", + "client_visibility", + "turn_origin", ], ), ( @@ -1860,6 +2035,8 @@ mod tests { "archived_at", "last_consolidated_at", "last_compressed_message_at", + "delivery_context", + "delivery_context_updated_at", ], ), ( @@ -1885,7 +2062,15 @@ mod tests { .await .unwrap(); assert_eq!(schema_version, SCHEMA_VERSION); - for table in ["task_plans", "task_items", "session_turn_usage"] { + for table in [ + "task_plans", + "task_items", + "session_turn_usage", + "agent_run_groups", + "agent_runs", + "agent_session_state", + "agent_inbox_events", + ] { let exists: i64 = sqlx::query_scalar( "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = ?", ) @@ -1895,6 +2080,19 @@ mod tests { .unwrap(); assert_eq!(exists, 1, "missing migrated table {table}"); } + + let visibility: String = + sqlx::query_scalar("SELECT client_visibility FROM messages ORDER BY seq LIMIT 1") + .fetch_one(storage.pool()) + .await + .unwrap_or_default(); + assert_eq!(visibility, "visible"); + let origin: String = + sqlx::query_scalar("SELECT turn_origin FROM messages ORDER BY seq LIMIT 1") + .fetch_one(storage.pool()) + .await + .unwrap_or_default(); + assert_eq!(origin, "user"); } #[tokio::test] @@ -2100,6 +2298,8 @@ mod tests { turn_id: None, iteration: None, completion_status: crate::bus::CompletionStatus::Completed, + client_visibility: crate::bus::ClientVisibility::Visible, + turn_origin: crate::bus::TurnOrigin::User, media_refs: None, tool_call_id: None, tool_name: None, @@ -2171,6 +2371,8 @@ mod tests { turn_id: Some("turn-1".to_string()), iteration: Some(2), completion_status: crate::bus::CompletionStatus::Interrupted, + client_visibility: crate::bus::ClientVisibility::Visible, + turn_origin: crate::bus::TurnOrigin::User, media_refs: None, tool_call_id: None, tool_name: None, @@ -2229,6 +2431,8 @@ mod tests { turn_id: None, iteration: None, completion_status: crate::bus::CompletionStatus::Completed, + client_visibility: crate::bus::ClientVisibility::Visible, + turn_origin: crate::bus::TurnOrigin::User, media_refs: None, tool_call_id: None, tool_name: None, @@ -2277,4 +2481,51 @@ mod tests { assert_eq!(loaded.message_count, 5); assert_eq!(loaded.last_active_at, 2000); } + + #[tokio::test] + async fn durable_delivery_context_round_trips() { + let (storage, _dir) = create_test_storage().await; + let meta = crate::storage::session::SessionMeta { + id: "feishu:chat-1:dialog".to_string(), + channel: "feishu".to_string(), + chat_id: "chat-1".to_string(), + dialog_id: "dialog".to_string(), + title: "t".to_string(), + created_at: 1, + last_active_at: 2, + message_count: 0, + routing_info: None, + archived_at: None, + deleted_at: None, + last_consolidated_at: None, + last_compressed_message_at: None, + }; + storage.upsert_session(&meta).await.unwrap(); + + assert!( + storage + .get_session_delivery_context("feishu:chat-1:dialog") + .await + .unwrap() + .is_none() + ); + + let context = serde_json::json!({ + "feishu.thread_id": "thread-9", + "feishu.chat_type": "group", + }) + .to_string(); + storage + .update_session_delivery_context("feishu:chat-1:dialog", &context, 50) + .await + .unwrap(); + let loaded = storage + .get_session_delivery_context("feishu:chat-1:dialog") + .await + .unwrap() + .unwrap(); + let parsed: std::collections::HashMap = + serde_json::from_str(&loaded).unwrap(); + assert_eq!(parsed.get("feishu.thread_id").unwrap(), "thread-9"); + } } diff --git a/src/tools/agent_task.rs b/src/tools/agent_task.rs new file mode 100644 index 0000000..57ff613 --- /dev/null +++ b/src/tools/agent_task.rs @@ -0,0 +1,251 @@ +use std::sync::Arc; + +use async_trait::async_trait; +use serde_json::{Value, json}; + +use crate::agent::AgentCoordinator; +use crate::storage::agent_run::AgentRunRecord; +use crate::tools::traits::{DelegationPolicy, Tool, ToolExecutionContext, ToolOutput, ToolResult}; + +const RESULT_PREVIEW_CHARS: usize = 2_000; + +/// Scoped inspection and control of durable Agent runs. Authorization is +/// derived from the caller's ToolExecutionContext (session for ROOT, tree +/// position for named Agents); run IDs are never credentials. +pub struct AgentTaskTool { + coordinator: Arc, +} + +impl AgentTaskTool { + pub fn new(coordinator: Arc) -> Self { + Self { coordinator } + } +} + +#[async_trait] +impl Tool for AgentTaskTool { + fn name(&self) -> &str { + "agent_task" + } + + fn description(&self) -> &str { + "Inspect or control delegated Agent runs: get reads one run, list shows the session's runs, get_result returns the full terminal result, cancel stops a non-terminal run." + } + + fn delegation_policy(&self) -> DelegationPolicy { + DelegationPolicy::RuntimeInjected + } + + fn parameters_schema(&self) -> Value { + json!({ + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["get", "list", "get_result", "cancel"], + "description": "Operation to perform on Agent runs" + }, + "run_id": { + "type": "string", + "description": "Target run identifier for get/get_result/cancel" + }, + "cursor_created_at": { + "type": "integer", + "description": "Pagination cursor: created_at of the last run seen" + }, + "cursor_id": { + "type": "string", + "description": "Pagination cursor: id of the last run seen" + }, + "limit": { + "type": "integer", + "minimum": 1, + "maximum": 100, + "description": "Maximum number of runs to list (default 20)" + } + }, + "required": ["action"] + }) + } + + fn read_only(&self) -> bool { + false + } + + async fn execute(&self, args: Value) -> anyhow::Result { + self.execute_with_context(&ToolExecutionContext::default(), args) + .await + .map(|output| output.result) + } + + async fn execute_with_context( + &self, + context: &ToolExecutionContext, + args: Value, + ) -> anyhow::Result { + let action = args + .get("action") + .and_then(Value::as_str) + .unwrap_or_default(); + let result = match action { + "get" => self.handle_get(context, &args).await, + "list" => self.handle_list(context, &args).await, + "get_result" => self.handle_get_result(context, &args).await, + "cancel" => self.handle_cancel(context, &args).await, + other => Ok(ToolResult { + success: false, + output: String::new(), + error: Some(format!( + "unknown agent_task action '{other}'; supported: get, list, get_result, cancel" + )), + }), + }; + Ok(result?.into()) + } +} + +impl AgentTaskTool { + fn run_id<'a>(&self, args: &'a Value) -> anyhow::Result<&'a str> { + args.get("run_id") + .and_then(Value::as_str) + .filter(|value| !value.trim().is_empty()) + .ok_or_else(|| anyhow::anyhow!("missing required parameter: run_id")) + } + + async fn handle_get( + &self, + context: &ToolExecutionContext, + args: &Value, + ) -> anyhow::Result { + let run_id = self.run_id(args)?; + match self.coordinator.get_run(context, run_id).await { + Ok(Some(run)) => Ok(success(run_projection(&run, true))), + Ok(None) => Ok(failure(format!("run not found: {run_id}"))), + Err(error) => Ok(failure(error.to_string())), + } + } + + async fn handle_list( + &self, + context: &ToolExecutionContext, + args: &Value, + ) -> anyhow::Result { + let cursor = match ( + args.get("cursor_created_at").and_then(Value::as_i64), + args.get("cursor_id").and_then(Value::as_str), + ) { + (Some(created_at), Some(id)) => Some((created_at, id.to_string())), + (None, None) => None, + _ => { + return Ok(failure( + "cursor requires both cursor_created_at and cursor_id", + )); + } + }; + let limit = args.get("limit").and_then(Value::as_i64).unwrap_or(20); + match self.coordinator.list_runs(context, cursor, limit).await { + Ok(runs) => { + let payload: Vec = + runs.iter().map(|run| run_projection(run, false)).collect(); + Ok(success(json!({ "runs": payload }))) + } + Err(error) => Ok(failure(error.to_string())), + } + } + + async fn handle_get_result( + &self, + context: &ToolExecutionContext, + args: &Value, + ) -> anyhow::Result { + let run_id = self.run_id(args)?; + match self.coordinator.get_result(context, run_id).await { + Ok(Some(run)) => Ok(success(json!({ + "run_id": run.id, + "status": run.status.as_str(), + "result": run.result, + "error": run.error, + "tool_calls": run.tool_calls_count, + "iterations": run.iterations, + "finished_at": run.finished_at + }))), + Ok(None) => Ok(failure(format!( + "run {run_id} is not terminal or does not exist" + ))), + Err(error) => Ok(failure(error.to_string())), + } + } + + async fn handle_cancel( + &self, + context: &ToolExecutionContext, + args: &Value, + ) -> anyhow::Result { + let run_id = self.run_id(args)?; + match self + .coordinator + .cancel_run(context, run_id, "cancelled via agent_task") + .await + { + Ok(true) => Ok(success(json!({ "run_id": run_id, "status": "cancelled" }))), + Ok(false) => Ok(failure(format!( + "cannot cancel run {run_id}; it is terminal or does not exist" + ))), + Err(error) => Ok(failure(error.to_string())), + } + } +} + +fn run_projection(run: &AgentRunRecord, include_result_preview: bool) -> Value { + let mut value = json!({ + "run_id": run.id, + "group_id": run.group_id, + "agent_id": run.agent_id, + "status": run.status.as_str(), + "mode": run.mode.as_str(), + "depth": run.depth, + "parent_run_id": run.parent_run_id, + "provider_profile": run.provider_profile, + "model_id": run.model_id, + "tool_calls": run.tool_calls_count, + "iterations": run.iterations, + "created_at": run.created_at, + "started_at": run.started_at, + "finished_at": run.finished_at, + "error": run.error, + }); + if include_result_preview { + value["result_preview"] = json!(run.result.as_deref().map(preview)); + value["result_truncated"] = json!( + run.result + .as_deref() + .is_some_and(|result| result.chars().count() > RESULT_PREVIEW_CHARS) + ); + } + value +} + +fn preview(value: &str) -> String { + if value.chars().count() <= RESULT_PREVIEW_CHARS { + value.to_string() + } else { + let cut = value.floor_char_boundary(RESULT_PREVIEW_CHARS); + format!("{}...", &value[..cut]) + } +} + +fn success(value: Value) -> ToolResult { + ToolResult { + success: true, + output: value.to_string(), + error: None, + } +} + +fn failure(error: impl Into) -> ToolResult { + ToolResult { + success: false, + output: String::new(), + error: Some(error.into()), + } +} diff --git a/src/tools/browser/mod.rs b/src/tools/browser/mod.rs index ceca425..b27e356 100644 --- a/src/tools/browser/mod.rs +++ b/src/tools/browser/mod.rs @@ -51,6 +51,10 @@ impl BrowserTool { #[async_trait] impl Tool for BrowserTool { + fn delegation_policy(&self) -> crate::tools::DelegationPolicy { + crate::tools::DelegationPolicy::Delegatable + } + fn name(&self) -> &str { "browser" } diff --git a/src/tools/calculator.rs b/src/tools/calculator.rs index a11ef1d..e4bdae0 100644 --- a/src/tools/calculator.rs +++ b/src/tools/calculator.rs @@ -18,6 +18,10 @@ impl Default for CalculatorTool { #[async_trait] impl Tool for CalculatorTool { + fn delegation_policy(&self) -> crate::tools::DelegationPolicy { + crate::tools::DelegationPolicy::Delegatable + } + fn name(&self) -> &str { "calculator" } diff --git a/src/tools/chat_manager.rs b/src/tools/chat_manager.rs index 39d6e8e..3cff6ed 100644 --- a/src/tools/chat_manager.rs +++ b/src/tools/chat_manager.rs @@ -362,6 +362,8 @@ mod tests { turn_id: None, iteration: None, completion_status: crate::bus::CompletionStatus::Completed, + client_visibility: crate::bus::ClientVisibility::Visible, + turn_origin: crate::bus::TurnOrigin::User, media_refs: None, tool_call_id: None, tool_name: None, @@ -428,6 +430,8 @@ mod tests { turn_id: None, iteration: None, completion_status: crate::bus::CompletionStatus::Completed, + client_visibility: crate::bus::ClientVisibility::Visible, + turn_origin: crate::bus::TurnOrigin::User, media_refs: None, tool_call_id: None, tool_name: None, @@ -488,6 +492,8 @@ mod tests { turn_id: None, iteration: None, completion_status: crate::bus::CompletionStatus::Completed, + client_visibility: crate::bus::ClientVisibility::Visible, + turn_origin: crate::bus::TurnOrigin::User, media_refs: None, tool_call_id: None, tool_name: None, diff --git a/src/tools/content_search.rs b/src/tools/content_search.rs index 86152b6..f536081 100644 --- a/src/tools/content_search.rs +++ b/src/tools/content_search.rs @@ -51,6 +51,10 @@ impl Default for ContentSearchTool { #[async_trait] impl Tool for ContentSearchTool { + fn delegation_policy(&self) -> crate::tools::DelegationPolicy { + crate::tools::DelegationPolicy::Delegatable + } + fn name(&self) -> &str { "content_search" } diff --git a/src/tools/delegate.rs b/src/tools/delegate.rs index 5baea17..adcc3e5 100644 --- a/src/tools/delegate.rs +++ b/src/tools/delegate.rs @@ -1,18 +1,371 @@ use std::sync::Arc; use async_trait::async_trait; -use serde_json::json; +use serde_json::{Value, json}; -use crate::agent::{ExecutionMode, SubAgentConfig, SubAgentManager, TaskStatus}; -use crate::tools::traits::{Tool, ToolResult}; +use crate::agent::{AgentCoordinator, ExecutionMode, SubAgentConfig, SubAgentManager, TaskStatus}; +use crate::tools::traits::{Tool, ToolExecutionContext, ToolOutput, ToolResult}; pub struct DelegateTool { sub_agent_manager: Arc, + coordinator: Option>, +} + +/// Per-run schema view for a child Agent. Authorization still happens in the +/// manager from ToolExecutionContext; this wrapper keeps the model-visible +/// target enum aligned with that Agent's configured outgoing edges. +pub(crate) struct ScopedDelegateTool { + inner: Arc, + targets: Vec, +} + +impl ScopedDelegateTool { + pub(crate) fn new(inner: Arc, targets: Vec) -> Self { + Self { inner, targets } + } +} + +#[async_trait] +impl Tool for ScopedDelegateTool { + fn name(&self) -> &str { + self.inner.name() + } + + fn description(&self) -> &str { + self.inner.description() + } + + fn parameters_schema(&self) -> Value { + let mut schema = self.inner.parameters_schema(); + schema["properties"]["target"]["enum"] = json!(self.targets); + schema["properties"]["tasks"]["items"]["properties"]["target"]["enum"] = + json!(self.targets); + schema + } + + fn read_only(&self) -> bool { + self.inner.read_only() + } + + fn delegation_policy(&self) -> crate::tools::DelegationPolicy { + crate::tools::DelegationPolicy::RuntimeInjected + } + + async fn execute(&self, args: Value) -> anyhow::Result { + self.inner.execute(args).await + } + + async fn execute_with_context( + &self, + context: &ToolExecutionContext, + args: Value, + ) -> anyhow::Result { + self.inner.execute_with_context(context, args).await + } } impl DelegateTool { pub fn new(sub_agent_manager: Arc) -> Self { - Self { sub_agent_manager } + Self { + sub_agent_manager, + coordinator: None, + } + } + + pub fn with_coordinator(mut self, coordinator: Arc) -> Self { + self.coordinator = Some(coordinator); + self + } + + fn task_schema(&self) -> Value { + let targets: Vec<_> = self + .sub_agent_manager + .catalog() + .root_targets() + .into_iter() + .map(|definition| definition.id.clone()) + .collect(); + let mut target = json!({ + "type": "string", + "description": "目标 Agent ID。省略时仅使用兼容 general Agent" + }); + if !targets.is_empty() { + target["enum"] = json!(targets); + } + json!({ + "type": "object", + "properties": { + "target": target, + "task": { "type": "string", "description": "明确、独立、可验收的子任务" }, + "context": { "type": "string", "description": "完成任务所需的显式上下文；不会继承主会话历史" }, + "plan_item_id": { "type": "string", "description": "可选的当前计划子项 ID" } + }, + "required": ["task"] + }) + } + + fn parse_config(&self, value: &Value, mode: ExecutionMode) -> anyhow::Result { + let prompt = value + .get("task") + .or_else(|| value.get("prompt")) + .and_then(Value::as_str) + .ok_or_else(|| anyhow::anyhow!("missing required parameter: task"))? + .trim() + .to_string(); + if prompt.is_empty() { + anyhow::bail!("task must not be empty"); + } + let allowed_tools = value + .get("allowed_tools") + .and_then(Value::as_array) + .map(|items| { + items + .iter() + .filter_map(Value::as_str) + .map(str::to_string) + .collect() + }); + Ok(SubAgentConfig { + target: value + .get("target") + .and_then(Value::as_str) + .map(str::to_string), + prompt, + context: value + .get("context") + .and_then(Value::as_str) + .map(str::to_string), + mode, + allowed_tools, + max_iterations: value + .get("max_iterations") + .and_then(Value::as_u64) + .map(|v| v as usize), + timeout_secs: value.get("timeout_secs").and_then(Value::as_u64), + plan_item_id: value + .get("plan_item_id") + .and_then(Value::as_str) + .map(str::to_string), + session_id: None, + }) + } + + async fn handle_run( + &self, + args: &Value, + context: &ToolExecutionContext, + ) -> anyhow::Result { + let requested_mode = args + .get("mode") + .and_then(Value::as_str) + .unwrap_or("foreground"); + if matches!(requested_mode, "inline" | "parallel") { + tracing::warn!( + mode = requested_mode, + "deprecated delegate mode used; migrate to foreground with an optional tasks array" + ); + } + let (mode, legacy_parallel) = match requested_mode { + "foreground" | "inline" => (ExecutionMode::Foreground, false), + "background" => (ExecutionMode::Background, false), + "parallel" => (ExecutionMode::Foreground, true), + other => { + return Ok(failure(format!( + "unknown mode '{other}'; supported modes are foreground and background" + ))); + } + }; + let task_values: Vec<&Value> = match args.get("tasks").and_then(Value::as_array) { + Some(tasks) if !tasks.is_empty() => tasks.iter().collect(), + Some(_) => return Ok(failure("tasks must not be empty")), + None if legacy_parallel => { + return Ok(failure("legacy parallel mode requires a tasks array")); + } + None => vec![args], + }; + let mut configs = Vec::with_capacity(task_values.len()); + for task in task_values { + let mut config = self.parse_config(task, mode.clone())?; + if config.target.is_none() { + config.target = args + .get("target") + .and_then(Value::as_str) + .map(str::to_string); + } + if config.context.is_none() { + config.context = args + .get("context") + .and_then(Value::as_str) + .map(str::to_string); + } + if config.allowed_tools.is_none() { + config.allowed_tools = + args.get("allowed_tools") + .and_then(Value::as_array) + .map(|items| { + items + .iter() + .filter_map(Value::as_str) + .map(str::to_string) + .collect() + }); + } + config.max_iterations = config.max_iterations.or_else(|| { + args.get("max_iterations") + .and_then(Value::as_u64) + .map(|v| v as usize) + }); + config.timeout_secs = config + .timeout_secs + .or_else(|| args.get("timeout_secs").and_then(Value::as_u64)); + config.session_id = context + .agent + .as_ref() + .map(|agent| agent.root_session_id.clone()) + .or_else(|| context.session_id.clone()); + configs.push(config); + } + + match mode { + ExecutionMode::Foreground => { + let all_named = configs.iter().all(|config| config.target.is_some()); + let results = if all_named && let Some(coordinator) = self.coordinator.as_ref() { + coordinator + .delegate_foreground(context, configs) + .await + .map_err(|error| anyhow::anyhow!(error.to_string()))? + } else { + if configs.iter().any(|config| config.target.is_some()) { + return Ok(failure( + "mixed named and legacy general batches are not supported", + )); + } + self.sub_agent_manager + .run_foreground_batch(configs, context) + .await + .map_err(|error| anyhow::anyhow!(error.to_string()))? + }; + let payload: Vec<_> = results + .into_iter() + .map(|result| { + let (status, error) = status_projection(&result.status); + json!({ + "run_id": result.task_id, + "status": status, + "result": result.content, + "result_truncated": result.content_truncated, + "error": error, + "tool_calls": result.tool_calls_count, + "iterations": result.iterations, + "duration_ms": result.duration_ms + }) + }) + .collect(); + let all_completed = payload.iter().all(|value| value["status"] == "completed"); + Ok(ToolResult { + success: all_completed, + output: serde_json::to_string(&json!({ + "status": if all_completed { "completed" } else { "partial" }, + "results": payload + }))?, + error: None, + }) + } + ExecutionMode::Background => { + if configs.len() != 1 { + return Ok(failure( + "background batches require durable group admission and are not available yet", + )); + } + if context.agent.is_some() { + return Ok(failure( + "child Agents cannot create background runs in the current implementation", + )); + } + let mut config = configs.into_iter().next().expect("checked non-empty"); + if config.target.is_some() { + let Some(coordinator) = self.coordinator.as_ref() else { + return Ok(failure( + "named background Agents require agent_orchestration to be enabled", + )); + }; + config.session_id = context + .agent + .as_ref() + .map(|agent| agent.root_session_id.clone()) + .or_else(|| context.session_id.clone()); + return match coordinator.delegate_background(context, config).await { + Ok(run_id) => Ok(success(json!({ + "status": "accepted", + "runs": vec![json!({ "run_id": run_id, "status": "queued" })] + }))), + Err(error) => Ok(failure(error.to_string())), + }; + } + let routing = crate::agent::sub_agent::get_delegate_context().map_err(|_| { + anyhow::anyhow!("background delegate requires an active root Agent worker") + })?; + let task_id = self + .sub_agent_manager + .run_background(config, routing) + .await + .map_err(|error| anyhow::anyhow!(error.to_string()))?; + Ok(success(json!({ + "status": "accepted", + "runs": vec![json!({ "run_id": task_id, "status": "queued" })] + }))) + } + } + } + + async fn handle_check_task(&self, args: &Value) -> anyhow::Result { + let task_id = required_task_id(args)?; + let Some(task) = self.sub_agent_manager.check_task(task_id).await else { + return Ok(failure(format!("task not found: {task_id}"))); + }; + Ok(success(json!({ + "task_id": task.id, + "status": task.status, + "task": task.prompt, + "result": task.result, + "error": task.error, + "started_at": task.started_at, + "finished_at": task.finished_at + }))) + } + + async fn handle_cancel_task(&self, args: &Value) -> anyhow::Result { + let task_id = required_task_id(args)?; + match self.sub_agent_manager.cancel_task(task_id).await { + Ok(true) => Ok(success( + json!({ "task_id": task_id, "status": "cancelled" }), + )), + Ok(false) => Ok(failure(format!( + "cannot cancel task {task_id}; it is terminal or does not exist" + ))), + Err(error) => Ok(failure(format!("cancel failed: {error}"))), + } + } + + async fn handle_list_tasks( + &self, + context: &ToolExecutionContext, + ) -> anyhow::Result { + let session_id = context + .agent + .as_ref() + .map(|agent| agent.root_session_id.as_str()) + .or(context.session_id.as_deref()) + .ok_or_else(|| anyhow::anyhow!("delegate context is not session-bound"))?; + let tasks = self.sub_agent_manager.list_tasks(session_id).await; + Ok(success( + json!({ "tasks": tasks.into_iter().map(|task| json!({ + "task_id": task.id, + "status": task.status, + "task": task.prompt, + "created_at": task.created_at + })).collect::>() }), + )) } } @@ -23,73 +376,47 @@ impl Tool for DelegateTool { } fn description(&self) -> &str { - "子任务委托工具。创建子 Agent 处理独立任务，支持三种模式：\ - inline (阻塞返回结果)、background (异步执行，完成后通知)、\ - parallel (多个子 Agent 并发执行，聚合结果)。\ - 也可用于查询、取消和列出后台任务。" + "Delegate one or more independent tasks to configured Agents. foreground waits for all results; background returns accepted run IDs. Multiple tasks execute concurrently." } - fn parameters_schema(&self) -> serde_json::Value { + fn parameters_schema(&self) -> Value { json!({ "type": "object", "properties": { "action": { "type": "string", "enum": ["run", "check_task", "cancel_task", "list_tasks"], - "description": "操作类型: run 创建子Agent执行任务, check_task 查询后台任务, cancel_task 取消后台任务, list_tasks 列出后台任务" - }, - "prompt": { - "type": "string", - "description": "子任务描述（可内含额外约束，如：跳过 .tmp 文件）。action=run 时必填" + "description": "Compatibility task-management actions remain available during migration; omit for run" }, + "target": self.task_schema()["properties"]["target"].clone(), + "task": { "type": "string", "description": "Single delegated task" }, + "context": { "type": "string", "description": "Explicit context for the child Agent" }, "mode": { "type": "string", - "enum": ["inline", "background", "parallel"], - "description": "执行模式: inline=阻塞返回结果, background=异步执行+通知, parallel=多子Agent并发。默认 inline" - }, - "allowed_tools": { - "type": "array", - "items": { "type": "string" }, - "description": "允许子Agent使用的工具列表。不填使用默认只读集: file_read,file_search,content_search,web_fetch,http_request,calculator" - }, - "max_iterations": { - "type": "integer", - "description": "最大迭代次数，默认 99" - }, - "timeout_secs": { - "type": "integer", - "description": "超时秒数，默认 3600（1小时）" + "enum": ["foreground", "background"], + "description": "foreground waits; background returns after acceptance" }, "tasks": { "type": "array", - "description": "并行模式下的多个子任务（仅 mode=parallel 时使用）", - "items": { - "type": "object", - "properties": { - "prompt": { "type": "string", "description": "子任务描述" }, - "allowed_tools": { - "type": "array", - "items": { "type": "string" }, - "description": "该子任务的工具列表" - }, - "plan_item_id": { - "type": "string", - "description": "可选，绑定当前计划中的子项 ID（如 T2）" - } - }, - "required": ["prompt"] - } + "minItems": 1, + "items": self.task_schema(), + "description": "Independent tasks; execution is concurrent and results preserve request order" }, - "task_id": { - "type": "string", - "description": "后台任务ID（action=check_task/cancel_task 时必填）" + "plan_item_id": { "type": "string" }, + "task_id": { "type": "string", "description": "Legacy task-management action target" }, + "allowed_tools": { + "type": "array", + "items": { "type": "string" }, + "description": "Deprecated; only narrows the legacy general Agent and never expands named Agent permissions" }, - "plan_item_id": { - "type": "string", - "description": "inline/background 模式可选，绑定当前计划中的子项 ID" - } + "max_iterations": { "type": "integer", "minimum": 1 }, + "timeout_secs": { "type": "integer", "minimum": 1 } }, - "required": ["action"] + "anyOf": [ + { "required": ["task"] }, + { "required": ["tasks"] }, + { "required": ["action"] } + ] }) } @@ -97,315 +424,116 @@ impl Tool for DelegateTool { false } - async fn execute(&self, args: serde_json::Value) -> anyhow::Result { - let action = args["action"] - .as_str() - .ok_or_else(|| anyhow::anyhow!("missing required parameter: action"))?; - - match action { - "run" => self.handle_run(&args).await, - "check_task" => self.handle_check_task(&args).await, - "cancel_task" => self.handle_cancel_task(&args).await, - "list_tasks" => self.handle_list_tasks(&args).await, - _ => Ok(ToolResult { - success: false, - output: String::new(), - error: Some(format!( - "Unknown action: {}. Supported: run, check_task, cancel_task, list_tasks", - action - )), - }), - } - } -} - -impl DelegateTool { - fn parse_config_from_args(&self, args: &serde_json::Value) -> anyhow::Result { - let prompt = args["prompt"] - .as_str() - .ok_or_else(|| anyhow::anyhow!("missing required parameter: prompt"))? - .to_string(); - - let allowed_tools: Option> = args["allowed_tools"].as_array().map(|arr| { - arr.iter() - .filter_map(|v| v.as_str().map(|s| s.to_string())) - .collect() - }); - - let max_iterations = args["max_iterations"].as_u64().map(|v| v as usize); - let timeout_secs = args["timeout_secs"].as_u64(); - - Ok(SubAgentConfig { - prompt, - mode: ExecutionMode::Inline, - allowed_tools, - max_iterations, - timeout_secs, - plan_item_id: args["plan_item_id"].as_str().map(str::to_string), - session_id: None, - }) + fn delegation_policy(&self) -> crate::tools::DelegationPolicy { + crate::tools::DelegationPolicy::RuntimeInjected } - async fn handle_run(&self, args: &serde_json::Value) -> anyhow::Result { - let mode_str = args["mode"].as_str().unwrap_or("inline"); - let mode = match mode_str { - "inline" => ExecutionMode::Inline, - "background" => ExecutionMode::Background, - "parallel" => ExecutionMode::Parallel, - _ => { - return Ok(ToolResult { - success: false, - output: String::new(), - error: Some(format!( - "unknown mode: {}. Supported: inline, background, parallel", - mode_str - )), - }); - } + async fn execute(&self, args: Value) -> anyhow::Result { + self.execute_with_context(&ToolExecutionContext::default(), args) + .await + .map(|output| output.result) + } + + async fn execute_with_context( + &self, + context: &ToolExecutionContext, + args: Value, + ) -> anyhow::Result { + let action = args.get("action").and_then(Value::as_str).unwrap_or("run"); + let result = match action { + "run" => self.handle_run(&args, context).await?, + "check_task" => self.handle_check_task(&args).await?, + "cancel_task" => self.handle_cancel_task(&args).await?, + "list_tasks" => self.handle_list_tasks(context).await?, + other => failure(format!("unknown delegate action '{other}'")), }; - - match mode { - ExecutionMode::Inline => { - let mut config = self.parse_config_from_args(args)?; - if config.plan_item_id.is_some() { - config.session_id = Some( - crate::agent::sub_agent::get_delegate_context() - .map_err(anyhow::Error::msg)? - .session_id, - ); - } - let result = self - .sub_agent_manager - .run_inline(config) - .await - .map_err(|e| anyhow::anyhow!("{}", e))?; - - match result.status { - TaskStatus::Completed => Ok(ToolResult { - success: true, - output: result.content, - error: None, - }), - TaskStatus::Failed(err) => Ok(ToolResult { - success: false, - output: result.content, - error: Some(err), - }), - TaskStatus::TimedOut => Ok(ToolResult { - success: false, - output: result.content, - error: Some("sub-agent timed out".into()), - }), - TaskStatus::Cancelled => Ok(ToolResult { - success: false, - output: result.content, - error: Some("sub-agent cancelled".into()), - }), - } - } - ExecutionMode::Background => { - let mut config = self.parse_config_from_args(args)?; - let ctx = crate::agent::sub_agent::get_delegate_context().map_err(|_| { - anyhow::anyhow!("delegate context not available: not in an agent worker") - })?; - config.session_id = Some(ctx.session_id.clone()); - - let task_id = self - .sub_agent_manager - .run_background(config, ctx) - .await - .map_err(|e| anyhow::anyhow!("{}", e))?; - - Ok(ToolResult { - success: true, - output: format!("后台任务已启动。\ntask_id: {}", task_id), - error: None, - }) - } - ExecutionMode::Parallel => { - let tasks = args["tasks"] - .as_array() - .ok_or_else(|| anyhow::anyhow!("parallel mode requires 'tasks' array"))?; - - let ctx = crate::agent::sub_agent::get_delegate_context().ok(); - let mut configs = Vec::new(); - for task in tasks { - let prompt = task["prompt"] - .as_str() - .ok_or_else(|| anyhow::anyhow!("each parallel task requires 'prompt'"))? - .to_string(); - let allowed_tools: Option> = - task["allowed_tools"].as_array().map(|arr| { - arr.iter() - .filter_map(|v| v.as_str().map(|s| s.to_string())) - .collect() - }); - - configs.push(SubAgentConfig { - prompt, - mode: ExecutionMode::Inline, - allowed_tools, - max_iterations: args["max_iterations"].as_u64().map(|v| v as usize), - timeout_secs: args["timeout_secs"].as_u64(), - plan_item_id: task["plan_item_id"].as_str().map(str::to_string), - session_id: ctx.as_ref().map(|ctx| ctx.session_id.clone()), - }); - } - - let has_args_allowed = args["allowed_tools"].as_array().is_some(); - for c in &mut configs { - if c.allowed_tools.is_none() && has_args_allowed { - c.allowed_tools = args["allowed_tools"].as_array().map(|arr| { - arr.iter() - .filter_map(|v| v.as_str().map(|s| s.to_string())) - .collect() - }); - } - } - - let results = self - .sub_agent_manager - .run_parallel(configs) - .await - .map_err(|e| anyhow::anyhow!("{}", e))?; - - let mut output = String::new(); - for (i, r) in results.iter().enumerate() { - let status_icon = match r.status { - TaskStatus::Completed => "✅", - TaskStatus::Failed(_) => "❌", - TaskStatus::TimedOut => "⏱️ 超时", - TaskStatus::Cancelled => "🚫 已取消", - }; - output.push_str(&format!("[task_{}] {}\n", i + 1, status_icon)); - if !r.content.is_empty() { - output.push_str(&r.content); - output.push_str("\n\n"); - } - if let TaskStatus::Failed(ref err) = r.status { - output.push_str(&format!("错误: {}\n\n", err)); - } - } - - let all_success = results - .iter() - .all(|r| matches!(r.status, TaskStatus::Completed)); - Ok(ToolResult { - success: all_success, - output: output.trim().to_string(), - error: None, - }) - } - } - } - - async fn handle_check_task(&self, args: &serde_json::Value) -> anyhow::Result { - let task_id = args["task_id"] - .as_str() - .ok_or_else(|| anyhow::anyhow!("missing required parameter: task_id"))?; - - match self.sub_agent_manager.check_task(task_id).await { - Some(task) => { - let status_icon = match task.status.as_str() { - "completed" => "✅ 已完成", - "failed" => "❌ 失败", - "cancelled" => "🚫 已取消", - "running" => "🔄 运行中", - "pending" => "⏳ 等待中", - _ => task.status.as_str(), - }; - let mut output = format!( - "任务 ID: {}\n状态: {}\n任务: {}", - task.id, status_icon, task.prompt - ); - if let Some(ref result) = task.result { - output.push_str(&format!("\n\n结果:\n{}", result)); - } - if let Some(ref error) = task.error { - output.push_str(&format!("\n错误: {}", error)); - } - if let Some(started) = task.started_at - && let Some(finished) = task.finished_at - { - let duration = (finished - started) as f64 / 1000.0; - output.push_str(&format!("\n耗时: {:.1}s", duration)); - } - Ok(ToolResult { - success: true, - output, - error: None, - }) - } - None => Ok(ToolResult { - success: false, - output: String::new(), - error: Some(format!("task not found: {}", task_id)), - }), - } - } - - async fn handle_cancel_task(&self, args: &serde_json::Value) -> anyhow::Result { - let task_id = args["task_id"] - .as_str() - .ok_or_else(|| anyhow::anyhow!("missing required parameter: task_id"))?; - - match self.sub_agent_manager.cancel_task(task_id).await { - Ok(true) => Ok(ToolResult { - success: true, - output: format!("后台任务 {} 已取消", task_id), - error: None, - }), - Ok(false) => Ok(ToolResult { - success: false, - output: String::new(), - error: Some(format!("无法取消任务 {}（可能已完成或不存在）", task_id)), - }), - Err(e) => Ok(ToolResult { - success: false, - output: String::new(), - error: Some(format!("取消失败: {}", e)), - }), - } - } - - async fn handle_list_tasks(&self, _args: &serde_json::Value) -> anyhow::Result { - let ctx = crate::agent::sub_agent::get_delegate_context() - .map_err(|_| anyhow::anyhow!("delegate context not available"))?; - let tasks = self.sub_agent_manager.list_tasks(&ctx.session_id).await; - - if tasks.is_empty() { - return Ok(ToolResult { - success: true, - output: "没有后台任务".to_string(), - error: None, - }); - } - - let mut output = String::from("后台任务列表:\n\n"); - for task in &tasks { - let status_icon = match task.status.as_str() { - "completed" => "✅", - "failed" => "❌", - "cancelled" => "🚫", - "running" => "🔄", - "pending" => "⏳", - _ => "❓", - }; - output.push_str(&format!( - "{} {} - {} - {} (created: {})\n", - status_icon, - &task.id[..std::cmp::min(8, task.id.len())], - task.prompt.chars().take(60).collect::(), - task.status, - task.created_at, - )); - } - output.push_str(&format!("\n共 {} 个任务", tasks.len())); - - Ok(ToolResult { - success: true, - output, - error: None, - }) + Ok(result.into()) + } +} + +fn required_task_id(args: &Value) -> anyhow::Result<&str> { + args.get("task_id") + .and_then(Value::as_str) + .ok_or_else(|| anyhow::anyhow!("missing required parameter: task_id")) +} + +fn status_projection(status: &TaskStatus) -> (&'static str, Option<&str>) { + match status { + TaskStatus::Completed => ("completed", None), + TaskStatus::Failed(error) => ("failed", Some(error.as_str())), + TaskStatus::Cancelled => ("cancelled", None), + TaskStatus::TimedOut => ("timed_out", None), + } +} + +fn success(value: Value) -> ToolResult { + ToolResult { + success: true, + output: value.to_string(), + error: None, + } +} + +fn failure(error: impl Into) -> ToolResult { + ToolResult { + success: false, + output: String::new(), + error: Some(error.into()), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::collections::HashMap; + + fn manager() -> Arc { + let (notify_tx, _) = tokio::sync::mpsc::unbounded_channel(); + Arc::new(SubAgentManager::new( + crate::config::LLMProviderConfig { + provider_type: "openai".to_string(), + name: "test".to_string(), + base_url: "https://example.invalid/v1".to_string(), + api_key: "test".to_string(), + extra_headers: HashMap::new(), + model_id: "test-model".to_string(), + temperature: None, + max_tokens: None, + model_extra: HashMap::new(), + max_tool_iterations: 1, + token_limit: 4096, + workspace_dir: std::env::temp_dir(), + input_types: vec!["text".to_string()], + price_input_per_million: None, + price_output_per_million: None, + }, + Arc::new(crate::tools::ToolRegistry::new()), + None, + notify_tx, + 1, + None, + crate::task_supervisor::TaskSupervisor::new(), + )) + } + + #[test] + fn schema_exposes_only_canonical_lifecycle_modes() { + let schema = DelegateTool::new(manager()).parameters_schema(); + assert_eq!( + schema["properties"]["mode"]["enum"], + json!(["foreground", "background"]) + ); + } + + #[test] + fn scoped_schema_replaces_root_targets_for_child() { + let inner: Arc = Arc::new(DelegateTool::new(manager())); + let scoped = ScopedDelegateTool::new(inner, vec!["reviewer".to_string()]); + let schema = scoped.parameters_schema(); + assert_eq!(schema["properties"]["target"]["enum"], json!(["reviewer"])); + assert_eq!( + schema["properties"]["tasks"]["items"]["properties"]["target"]["enum"], + json!(["reviewer"]) + ); } } diff --git a/src/tools/emit_signal.rs b/src/tools/emit_signal.rs new file mode 100644 index 0000000..0cac9d6 --- /dev/null +++ b/src/tools/emit_signal.rs @@ -0,0 +1,354 @@ +//! Contract-bound `emit_signal` tool for background Agents. +//! +//! The tool only exists inside a run whose definition declares a `signal` +//! contract. Every limit (total count, rate, burst, severity allowlist, +//! payload size/depth, dedupe cooldown) is enforced here and in the +//! Coordinator; the model only supplies `key`, `severity`, `summary`, +//! `details` and an optional `dedupe_key`. Signals are durable inbox events +//! delivered to the run's root session lane (queue or steer per contract). +//! The tool returns only after the event is persisted. + +use std::collections::VecDeque; +use std::sync::Mutex; + +use serde::Deserialize; +use serde_json::Value; + +use crate::agent::coordinator::AgentCoordinator; +use crate::agent::definition::SignalContract; +use crate::agent::run::{AgentExecutionContext, EmittedSignal}; +use crate::storage::agent_inbox::{AgentEventDelivery, AgentEventType, NewInboxEvent}; +use crate::tools::{DelegationPolicy, Tool, ToolExecutionContext, ToolOutput, ToolResult}; + +#[derive(Debug, Clone)] +pub struct SignalInput { + pub key: String, + pub severity: String, + pub summary: String, + pub details: Option, + pub dedupe_key: Option, + /// Durable event key computed by the tool (dedupe window included). + pub event_key: String, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum SignalAcceptedStatus { + Accepted, + Deduplicated, +} + +#[derive(Debug, Clone)] +pub struct SignalAccepted { + pub signal_id: String, + pub status: SignalAcceptedStatus, + pub delivery: AgentEventDelivery, +} + +#[derive(Debug, Default)] +struct SignalRateState { + total: u32, + last_at_ms: i64, + burst_times: VecDeque, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct EmitSignalArgs { + pub key: String, + pub severity: String, + pub summary: String, + #[serde(default)] + pub details: Option, + #[serde(default)] + pub dedupe_key: Option, +} + +use std::sync::Arc; + +pub struct EmitSignalTool { + coordinator: Arc, + contract: Arc, + rate: Mutex, +} + +impl EmitSignalTool { + pub fn new(coordinator: Arc, contract: Arc) -> Self { + Self { + coordinator, + contract, + rate: Mutex::new(SignalRateState::default()), + } + } +} + +const MAX_KEY_CHARS: usize = 128; +const MAX_DEDUPE_KEY_CHARS: usize = 128; +const MAX_SUMMARY_CHARS: usize = 1024; + +#[async_trait::async_trait] +impl Tool for EmitSignalTool { + fn name(&self) -> &str { + "emit_signal" + } + + fn description(&self) -> &str { + "向主 Agent 发送一条结构化内部信号，用于重要中间状态或监控告警；普通进度请留在工具调用记录里，不要滥用信号。" + } + + fn parameters_schema(&self) -> serde_json::Value { + serde_json::json!({ + "type": "object", + "properties": { + "key": { "type": "string", "description": "信号键，用于区分不同信号" }, + "severity": { "type": "string", "description": "严重级别" }, + "summary": { "type": "string", "description": "简短摘要" }, + "details": { "type": "object", "description": "结构化详情" }, + "dedupe_key": { "type": "string", "description": "去重键（可选）" } + }, + "required": ["key", "severity", "summary"] + }) + } + + async fn execute(&self, _args: serde_json::Value) -> anyhow::Result { + Ok(ToolResult { + success: false, + output: String::new(), + error: Some("emit_signal requires a run-bound context".to_string()), + }) + } + + fn delegation_policy(&self) -> DelegationPolicy { + DelegationPolicy::RuntimeInjected + } + + async fn execute_with_context( + &self, + context: &ToolExecutionContext, + args: Value, + ) -> anyhow::Result { + let args: EmitSignalArgs = match serde_json::from_value(args) { + Ok(args) => args, + Err(error) => { + return Ok(ToolResult { + success: false, + output: String::new(), + error: Some(error.to_string()), + } + .into()); + } + }; + let contract = self.contract.clone(); + let coordinator = self.coordinator.clone(); + let rate = &self.rate; + let context = context.clone(); + let result: Result = async move { + let Some(agent) = context.agent.as_deref() else { + return Err("emit_signal requires a run-bound Agent context".to_string()); + }; + let now_ms = chrono::Utc::now().timestamp_millis(); + + // Structural validation against the contract, independent of the + // model's cooperation. + if args.key.trim().is_empty() || args.key.len() > MAX_KEY_CHARS { + return Err(format!("key must contain 1..={MAX_KEY_CHARS} characters")); + } + if let Some(dedupe) = args.dedupe_key.as_deref() + && (dedupe.trim().is_empty() || dedupe.len() > MAX_DEDUPE_KEY_CHARS) + { + return Err(format!( + "dedupe_key must contain 1..={MAX_DEDUPE_KEY_CHARS} characters" + )); + } + if !contract + .severity_allowlist + .iter() + .any(|allowed| allowed == &args.severity) + { + return Err(format!( + "severity '{0}' is not allowed; allowlist: {1}", + args.severity, + contract.severity_allowlist.join(", ") + )); + } + if args.summary.trim().is_empty() || args.summary.chars().count() > MAX_SUMMARY_CHARS { + return Err(format!( + "summary must contain 1..={MAX_SUMMARY_CHARS} characters" + )); + } + if let Some(details) = args.details.as_ref() { + let bytes = details.to_string().len(); + if bytes > contract.max_details_bytes { + return Err(format!( + "details exceed the {} byte contract limit", + contract.max_details_bytes + )); + } + if json_depth(details) > contract.max_payload_depth { + return Err(format!( + "details exceed the {} level depth limit", + contract.max_payload_depth + )); + } + } + + // Per-run rate limits. Rate state is in-memory and per-run: the + // tool instance is created for exactly one run's registry. + { + let mut state = rate.lock().unwrap_or_else(|poisoned| poisoned.into_inner()); + if state.total >= contract.max_total { + return Err(format!( + "signal limit reached: {} signals for this run", + contract.max_total + )); + } + if state.total > 0 && now_ms - state.last_at_ms < contract.min_interval_ms as i64 { + return Err(format!( + "signal rate limited: wait at least {}ms between signals", + contract.min_interval_ms + )); + } + let window_start = now_ms - contract.burst_window_ms as i64; + while state + .burst_times + .front() + .is_some_and(|time| *time < window_start) + { + state.burst_times.pop_front(); + } + if state.burst_times.len() as u32 >= contract.max_burst { + return Err(format!( + "signal burst limited: at most {} signals per {}ms", + contract.max_burst, contract.burst_window_ms + )); + } + } + + let event_key = match args.dedupe_key.as_deref() { + Some(key) => { + let window = now_ms / contract.dedupe_cooldown_ms as i64; + format!("signal:{key}:{window}") + } + None => format!("signal:{}", uuid::Uuid::new_v4()), + }; + let accepted = coordinator + .emit_signal( + agent, + SignalInput { + key: args.key.clone(), + severity: args.severity.clone(), + summary: args.summary.clone(), + details: args.details.clone(), + dedupe_key: args.dedupe_key.clone(), + event_key, + }, + ) + .await + .map_err(|error| error.to_string())?; + + if accepted.status == SignalAcceptedStatus::Accepted { + if let Ok(mut state) = rate.lock() { + state.total = state.total.saturating_add(1); + state.last_at_ms = now_ms; + state.burst_times.push_back(now_ms); + } + if let Ok(mut signals) = agent.emitted_signals.lock() { + signals.push(EmittedSignal { + signal_id: accepted.signal_id.clone(), + severity: args.severity.clone(), + summary: args.summary.clone(), + }); + } + } + + let output = serde_json::json!({ + "signal_id": accepted.signal_id, + "status": if accepted.status == SignalAcceptedStatus::Accepted { + "accepted" + } else { + "deduplicated" + }, + "delivery": accepted.delivery.as_str(), + }) + .to_string(); + Ok(ToolOutput { + result: ToolResult { + success: true, + output, + error: None, + }, + artifacts: Vec::new(), + }) + } + .await; + result.map_err(|error| anyhow::anyhow!(error)) + } +} + +/// Maximum JSON nesting depth, counting objects/arrays. +fn json_depth(value: &Value) -> usize { + match value { + Value::Object(map) => 1 + map.values().map(json_depth).max().unwrap_or(0), + Value::Array(items) => 1 + items.iter().map(json_depth).max().unwrap_or(0), + _ => 0, + } +} + +/// Helper for tests and non-coordinator embedders: build the durable inbox +/// event descriptor for a signal without enforcing runtime state. +pub fn build_signal_event( + context: &AgentExecutionContext, + input: &SignalInput, + event_id: String, + delivery: AgentEventDelivery, +) -> NewInboxEvent { + NewInboxEvent { + id: event_id, + root_session_id: context.root_session_id.clone(), + scope_kind: "run".to_string(), + scope_id: context.run_id.clone(), + run_id: Some(context.run_id.clone()), + group_id: context.group_id.clone(), + event_type: AgentEventType::Signal, + event_key: input.event_key.clone(), + delivery, + requires_continuation: true, + severity: Some(input.severity.clone()), + payload_json: serde_json::json!({ + "kind": "signal", + "severity": input.severity, + "summary": input.summary, + "details": input.details, + "dedupe_key": input.dedupe_key, + "key": input.key, + "run_id": context.run_id, + "agent_id": context.current_agent_id, + }) + .to_string(), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn details_depth_is_measured() { + assert_eq!(json_depth(&Value::Null), 0); + assert_eq!(json_depth(&Value::String("x".into())), 0); + assert_eq!( + json_depth(&serde_json::json!({"a": {"b": [1, {"c": 2}]}})), + 4 + ); + } + + #[test] + fn default_contract_limits_are_sane() { + let contract = SignalContract::default(); + assert_eq!(contract.max_burst, 5); + assert!( + contract + .severity_allowlist + .contains(&"critical".to_string()) + ); + } +} diff --git a/src/tools/file_read.rs b/src/tools/file_read.rs index 55cb90f..a4d0304 100644 --- a/src/tools/file_read.rs +++ b/src/tools/file_read.rs @@ -36,6 +36,10 @@ impl Default for FileReadTool { #[async_trait] impl Tool for FileReadTool { + fn delegation_policy(&self) -> crate::tools::DelegationPolicy { + crate::tools::DelegationPolicy::Delegatable + } + fn name(&self) -> &str { "file_read" } diff --git a/src/tools/file_search.rs b/src/tools/file_search.rs index b9544a4..18d6fd4 100644 --- a/src/tools/file_search.rs +++ b/src/tools/file_search.rs @@ -51,6 +51,10 @@ impl Default for FileSearchTool { #[async_trait] impl Tool for FileSearchTool { + fn delegation_policy(&self) -> crate::tools::DelegationPolicy { + crate::tools::DelegationPolicy::Delegatable + } + fn name(&self) -> &str { "file_search" } diff --git a/src/tools/get_skill.rs b/src/tools/get_skill.rs index 6076dca..48713e6 100644 --- a/src/tools/get_skill.rs +++ b/src/tools/get_skill.rs @@ -1,3 +1,4 @@ +use std::collections::HashSet; use std::sync::Arc; use async_trait::async_trait; @@ -8,11 +9,22 @@ use crate::tools::traits::{Tool, ToolResult}; pub struct GetSkillTool { skills_loader: Arc, + allowed: Option>, } impl GetSkillTool { pub fn new(skills_loader: Arc) -> Self { - Self { skills_loader } + Self { + skills_loader, + allowed: None, + } + } + + pub fn scoped(skills_loader: Arc, allowed: &[String]) -> Self { + Self { + skills_loader, + allowed: Some(allowed.iter().cloned().collect()), + } } fn format_skill(&self, skill: &Skill) -> String { @@ -32,6 +44,10 @@ impl GetSkillTool { #[async_trait] impl Tool for GetSkillTool { + fn delegation_policy(&self) -> crate::tools::DelegationPolicy { + crate::tools::DelegationPolicy::RuntimeInjected + } + fn name(&self) -> &str { "get_skill" } @@ -85,6 +101,20 @@ impl GetSkillTool { } }; + if self + .allowed + .as_ref() + .is_some_and(|allowed| !allowed.contains(skill_name)) + { + return Ok(ToolResult { + success: false, + output: String::new(), + error: Some(format!( + "Skill '{skill_name}' is not allowed for this Agent" + )), + }); + } + match self.skills_loader.get_skill(skill_name) { Some(skill) => { let formatted = self.format_skill(&skill); @@ -118,7 +148,16 @@ impl GetSkillTool { } fn list_skills_full(&self) -> anyhow::Result { - let skills = self.skills_loader.get_loaded_skills(); + let skills: Vec<_> = self + .skills_loader + .get_loaded_skills() + .into_iter() + .filter(|skill| { + self.allowed + .as_ref() + .is_none_or(|allowed| allowed.contains(&skill.name)) + }) + .collect(); if skills.is_empty() { return Ok(ToolResult { success: true, diff --git a/src/tools/mod.rs b/src/tools/mod.rs index d121f55..f87effc 100644 --- a/src/tools/mod.rs +++ b/src/tools/mod.rs @@ -1,3 +1,4 @@ +pub mod agent_task; pub mod bash; pub mod browser; pub mod calculator; @@ -5,6 +6,7 @@ pub mod chat_manager; pub mod content_search; pub mod cron; pub mod delegate; +pub mod emit_signal; mod expression; pub mod file_edit; pub mod file_read; @@ -26,12 +28,14 @@ pub mod todo; pub mod traits; pub mod web_fetch; +pub use agent_task::AgentTaskTool; pub use bash::BashTool; pub use browser::{BrowserProfilesTool, BrowserTool}; pub use calculator::CalculatorTool; pub use chat_manager::ChatManagerTool; pub use content_search::ContentSearchTool; pub use delegate::DelegateTool; +pub use emit_signal::EmitSignalTool; pub use file_edit::FileEditTool; pub use file_read::FileReadTool; pub use file_search::FileSearchTool; @@ -48,8 +52,9 @@ pub use send_message::SendMessageTool; pub use sleep::SleepTool; pub use todo::TodoTool; pub use traits::{ - OutboundDelivery, OutboundMessenger, ProcessedToolOutput, Tool, ToolArtifact, - ToolArtifactAudience, ToolExecutionContext, ToolOutput, ToolOutputProcessor, ToolResult, + DelegationPolicy, InputInterruptPolicy, OutboundDelivery, OutboundMessenger, + ProcessedToolOutput, Tool, ToolArtifact, ToolArtifactAudience, ToolExecutionContext, + ToolOutput, ToolOutputProcessor, ToolResult, }; pub use web_fetch::WebFetchTool; diff --git a/src/tools/registry.rs b/src/tools/registry.rs index 9a07390..62fade7 100644 --- a/src/tools/registry.rs +++ b/src/tools/registry.rs @@ -3,6 +3,7 @@ use std::sync::{Arc, Mutex}; use crate::providers::{Tool, ToolFunction}; +use super::traits::DelegationPolicy; use super::traits::Tool as ToolTrait; pub struct ToolRegistry { @@ -87,6 +88,36 @@ impl ToolRegistry { Arc::new(filtered) } + /// Build an immutable execution view for a named Agent. Definition tools + /// must be explicitly delegatable; runtime tools are passed separately and + /// can only carry the RuntimeInjected policy. + pub fn scoped_for_agent( + &self, + tool_names: &[String], + runtime_tools: Vec>, + ) -> Result, String> { + let scoped = Self::new(); + for name in tool_names { + let tool = self + .get(name) + .ok_or_else(|| format!("tool '{name}' is not registered"))?; + if tool.delegation_policy() != DelegationPolicy::Delegatable { + return Err(format!("tool '{name}' is not delegatable")); + } + scoped.register_raw(name.clone(), tool); + } + for tool in runtime_tools { + if tool.delegation_policy() != DelegationPolicy::RuntimeInjected { + return Err(format!( + "runtime tool '{}' is missing RuntimeInjected policy", + tool.name() + )); + } + scoped.register_raw(tool.name().to_string(), tool); + } + Ok(Arc::new(scoped)) + } + /// 生成工具列表描述，用于子 Agent 系统提示词 pub fn describe_for_prompt(&self) -> String { let mut entries: Vec = self diff --git a/src/tools/send_message.rs b/src/tools/send_message.rs index f3e1624..a1d1391 100644 --- a/src/tools/send_message.rs +++ b/src/tools/send_message.rs @@ -133,6 +133,9 @@ target_chat_id 支持两种格式：:（发送到该聊天下 from_user_id: None, system_name: None, task_id: None, + from_run_id: None, + from_agent_id: None, + group_id: None, }; // 3. Parse files into MediaItems diff --git a/src/tools/sleep.rs b/src/tools/sleep.rs index dffd6a4..789bef9 100644 --- a/src/tools/sleep.rs +++ b/src/tools/sleep.rs @@ -34,6 +34,14 @@ fn parse_seconds(args: &serde_json::Value) -> Result { #[async_trait] impl Tool for SleepTool { + fn delegation_policy(&self) -> crate::tools::DelegationPolicy { + crate::tools::DelegationPolicy::Delegatable + } + + fn input_interrupt_policy(&self) -> crate::tools::InputInterruptPolicy { + crate::tools::InputInterruptPolicy::WakeOnly + } + fn name(&self) -> &str { "sleep" } @@ -58,6 +66,16 @@ impl Tool for SleepTool { } async fn execute(&self, args: serde_json::Value) -> anyhow::Result { + self.execute_with_context(&crate::tools::ToolExecutionContext::default(), args) + .await + .map(|output| output.result) + } + + async fn execute_with_context( + &self, + context: &crate::tools::ToolExecutionContext, + args: serde_json::Value, + ) -> anyhow::Result { let seconds = match parse_seconds(&args) { Ok(seconds) => seconds, Err(error) => { @@ -65,17 +83,25 @@ impl Tool for SleepTool { success: false, output: String::new(), error: Some(error), - }); + } + .into()); } }; - tokio::time::sleep(Duration::from_secs(seconds)).await; + tokio::select! { + biased; + _ = context.cancellation.cancelled() => { + anyhow::bail!("sleep cancelled"); + } + _ = tokio::time::sleep(Duration::from_secs(seconds)) => {} + } Ok(ToolResult { success: true, output: format!("Slept for {seconds} second(s)."), error: None, - }) + } + .into()) } } @@ -235,4 +261,32 @@ mod tests { } if id == "sleep-call" )); } + + #[tokio::test(start_paused = true)] + async fn cancellation_token_ends_sleep_before_timer() { + let context = crate::tools::ToolExecutionContext::default(); + let token = context.cancellation.clone(); + let handle = tokio::spawn(async move { + SleepTool::new() + .execute_with_context(&context, json!({"seconds": MAX_SLEEP_SECONDS})) + .await + }); + tokio::task::yield_now().await; + assert!(!handle.is_finished()); + token.cancel(); + tokio::task::yield_now().await; + let error = handle.await.unwrap().unwrap_err(); + assert!(error.to_string().contains("cancelled")); + } + + #[tokio::test(start_paused = true)] + async fn pre_cancelled_context_never_enters_sleep() { + let context = crate::tools::ToolExecutionContext::default(); + context.cancellation.cancel(); + let error = SleepTool::new() + .execute_with_context(&context, json!({"seconds": 60})) + .await + .unwrap_err(); + assert!(error.to_string().contains("cancelled")); + } } diff --git a/src/tools/traits.rs b/src/tools/traits.rs index 7ce3512..9c3e7a1 100644 --- a/src/tools/traits.rs +++ b/src/tools/traits.rs @@ -3,10 +3,25 @@ use async_trait::async_trait; /// Session identity supplied by the runtime for tools that own external state. /// Ordinary stateless tools can ignore it through the default trait method. -#[derive(Debug, Clone, Default)] +#[derive(Debug, Clone)] pub struct ToolExecutionContext { pub session_id: Option, pub turn_id: Option, + pub agent: Option>, + pub cancellation: tokio_util::sync::CancellationToken, + pub execution_gate: Option>, +} + +impl Default for ToolExecutionContext { + fn default() -> Self { + Self { + session_id: None, + turn_id: None, + agent: None, + cancellation: tokio_util::sync::CancellationToken::new(), + execution_gate: None, + } + } } impl ToolExecutionContext { @@ -14,6 +29,9 @@ impl ToolExecutionContext { Self { session_id: Some(session_id.into()), turn_id: None, + agent: None, + cancellation: tokio_util::sync::CancellationToken::new(), + execution_gate: None, } } @@ -21,6 +39,41 @@ impl ToolExecutionContext { self.turn_id = Some(turn_id.into()); self } + + pub fn with_agent( + mut self, + agent: std::sync::Arc, + ) -> Self { + self.agent = Some(agent); + self + } + + pub fn with_cancellation(mut self, cancellation: tokio_util::sync::CancellationToken) -> Self { + self.cancellation = cancellation; + self + } + + pub fn with_execution_gate( + mut self, + gate: std::sync::Arc, + ) -> Self { + self.execution_gate = Some(gate); + self + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum DelegationPolicy { + RootOnly, + Delegatable, + RuntimeInjected, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum InputInterruptPolicy { + Never, + WakeOnly, + CancelSafe, } #[derive(Debug, Clone)] @@ -154,6 +207,17 @@ pub trait Tool: Send + Sync + 'static { fn parameters_schema(&self) -> serde_json::Value; async fn execute(&self, args: serde_json::Value) -> anyhow::Result; + /// Whether a named Agent definition may receive this tool. New tools fail + /// closed until their delegated behavior has been reviewed explicitly. + fn delegation_policy(&self) -> DelegationPolicy { + DelegationPolicy::RootOnly + } + + /// Whether new Turn input may interrupt an in-flight invocation. + fn input_interrupt_policy(&self) -> InputInterruptPolicy { + InputInterruptPolicy::Never + } + /// Execute the tool through the unified output envelope. Most tools return /// only text and use this default conversion. async fn execute_output(&self, args: serde_json::Value) -> anyhow::Result { diff --git a/src/tools/web_fetch.rs b/src/tools/web_fetch.rs index afe7bda..22e08b9 100644 --- a/src/tools/web_fetch.rs +++ b/src/tools/web_fetch.rs @@ -331,6 +331,10 @@ fn is_private_ip(ip: &std::net::IpAddr) -> bool { #[async_trait] impl Tool for WebFetchTool { + fn delegation_policy(&self) -> crate::tools::DelegationPolicy { + crate::tools::DelegationPolicy::Delegatable + } + fn name(&self) -> &str { "web_fetch" } diff --git a/src/work/mod.rs b/src/work/mod.rs index 55ea1bf..7b28963 100644 --- a/src/work/mod.rs +++ b/src/work/mod.rs @@ -455,6 +455,32 @@ impl WorkManager { Ok(Some(updated)) } + /// Re-read the active plan after an external transaction (agent run + /// admission or terminal commit) already mutated plan items, then refresh + /// the cache and broadcast. This path never writes to the database. + pub async fn refresh_after_external_commit( + &self, + session_id: &str, + reason: &str, + item_ids: Vec, + ) -> Result, StorageError> { + let row = sqlx::query( + "SELECT id, session_id, objective, status, version, created_at, updated_at, closed_at \ + FROM task_plans WHERE session_id = ? AND status = 'active' LIMIT 1", + ) + .bind(session_id) + .fetch_optional(self.storage.pool()) + .await?; + let Some(row) = row else { + self.active_cache.insert(session_id.to_string(), None); + return Ok(None); + }; + let plan = self.plan_from_row(row).await?; + self.cache_active_plan(&plan); + self.emit(reason, item_ids, Some(plan.clone())); + Ok(Some(plan)) + } + pub async fn close_plan( &self, session_id: &str, diff --git a/tests/test_request_format.rs b/tests/test_request_format.rs index 9d9394f..7525304 100644 --- a/tests/test_request_format.rs +++ b/tests/test_request_format.rs @@ -142,6 +142,7 @@ fn test_bounded_session_history_protocol() { tool_name: None, tool_calls: None, attachments: Vec::new(), + turn_origin: picobot::bus::TurnOrigin::User, }], }; let decoded: WsOutbound = @@ -196,6 +197,7 @@ fn test_session_history_preserves_tool_call_metadata() { arguments: serde_json::json!({ "path": "README.md" }), }]), attachments: Vec::new(), + turn_origin: picobot::bus::TurnOrigin::User, }], }; diff --git a/webui/package-lock.json b/webui/package-lock.json index dc619b9..5a245bd 100644 --- a/webui/package-lock.json +++ b/webui/package-lock.json @@ -1,12 +1,12 @@ { "name": "picobot-webui", - "version": "1.5.1", + "version": "1.7.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "picobot-webui", - "version": "1.5.1", + "version": "1.7.0", "dependencies": { "bits-ui": "^2.0.0", "dompurify": "^3.4.12", diff --git a/webui/package.json b/webui/package.json index 9c87864..095b7ed 100644 --- a/webui/package.json +++ b/webui/package.json @@ -1,7 +1,7 @@ { "name": "picobot-webui", "private": true, - "version": "1.5.1", + "version": "1.7.0", "type": "module", "engines": { "node": ">=20" diff --git a/webui/src/pages/ChatPage.svelte b/webui/src/pages/ChatPage.svelte index 8b98229..edf8abb 100644 --- a/webui/src/pages/ChatPage.svelte +++ b/webui/src/pages/ChatPage.svelte @@ -12,6 +12,7 @@ let sessions = $state([]); let currentId = $state(null); let messages = $state([]); + let agentEvents = $state([]); let search = $state(""); let draft = $state(""); let commands = $state([]); @@ -178,6 +179,16 @@ break; } case "session_stats": break; + case "agent_event_updated": { + // Durable Agent events are projections, never part of chat history. + // Signal/completion cards render separately so the main Agent's + // final wording stays the only history authority. + if (frame.session_id !== currentId) break; + agentEvents = [frame.event, ...agentEvents.filter((event) => event.id !== frame.event.id)].slice(0, 20); + break; + } + case "agent_run_updated": + break; case "system_notification": if (!frame.session_id || frame.session_id === currentId) appendMessage("assistant", frame.content); break; @@ -221,6 +232,15 @@ return "○"; } + function signalSummary(event) { + try { + const payload = JSON.parse(event.payload_json); + return payload.summary || payload.status || event.payload_json.slice(0, 200); + } catch { + return event.payload_json.slice(0, 200); + } + } + function toolResult(callId) { return messages.find((message) => message.role === "tool" && message.tool_call_id === callId) || null; } @@ -444,6 +464,28 @@
+ {#if agentEvents.length} +
+ {#each agentEvents as event (event.id)} +
+ +
+
+ {event.event_type === "signal" ? `后台信号 · ${event.severity || "info"}` : `后台任务${event.status === "consumed" ? "已处理" : "完成"}`} + {event.delivery === "steer" ? "steer" : "queue"} +
+ {#if event.event_type === "signal"} + {#if event.payload_json} +

{signalSummary(event)}

+ {/if} + {:else} +

{event.status}{event.last_error ? ` · ${event.last_error}` : ""}

+ {/if} +
+
+ {/each} +
+ {/if} {#if messages.length === 0 && !activeTurn}

今天想做些什么？

消息与 CLI 客户端使用同一套会话、记忆和工具能力。

{/if} @@ -452,6 +494,9 @@
{#if message.role === "user"}你{:else}{/if}
+ {#if message.turn_origin === "agent_continuation" && message.role !== "user"} + 后台结果处理 + {/if} {#if message.reasoning_content}
思考过程 diff --git a/webui/src/pages/TasksPage.svelte b/webui/src/pages/TasksPage.svelte index 3bec2f0..f01f705 100644 --- a/webui/src/pages/TasksPage.svelte +++ b/webui/src/pages/TasksPage.svelte @@ -70,6 +70,32 @@ return "var(--danger)"; } + function groupTasks(all) { + const groups = []; + const byGroup = new Map(); + const roots = []; + for (const task of all) { + if (task.source === "legacy_background_task") { + groups.push({ key: `legacy-${task.id}`, title: task.prompt.slice(0, 100), status: task.status, runs: [task] }); + } else if (task.group_id) { + if (!byGroup.has(task.group_id)) { + byGroup.set(task.group_id, { key: `group-${task.group_id}`, title: `任务组 ${task.group_id.slice(0, 8)}`, status: task.status, runs: [] }); + groups.push(byGroup.get(task.group_id)); + } + byGroup.get(task.group_id).runs.push(task); + } else { + roots.push({ key: `run-${task.id}`, title: task.prompt.slice(0, 100), status: task.status, runs: [task] }); + } + } + for (const group of byGroup.values()) { + group.runs.sort((a, b) => (a.created_at || 0) - (b.created_at || 0)); + group.status = group.runs.find((task) => task.status === "running")?.status || group.runs[0]?.status || "pending"; + } + groups.push(...roots); + groups.sort((a, b) => (b.runs[0]?.created_at || 0) - (a.runs[0]?.created_at || 0)); + return groups; + } + onMount(() => { load(); const timer = setInterval(() => { tick += 1; }, 30000); @@ -91,27 +117,40 @@ {#if loading}
加载中…
{:else if error}
{error}
{:else if tab === "background"} - {#each tasks as task (task.id)} -
-
-
-

- {#if task.status === "running"}{/if} - {task.prompt.slice(0, 100)} -

-
- {task.session_id} - {formatTime(task.created_at)} - {#if task.status === "running"}{elapsed(task.created_at)}{/if} - {task.tool_calls_count} 次工具调用 · {task.iterations} 轮 + {#each groupTasks(tasks) as group (group.key)} + {#if group.runs.length === 0} +
暂无后台任务
+ {:else} +
+
+
+

{group.title}

+
+ {group.runs[0].session_id} + {formatTime(group.runs[0].created_at)} + {#if group.runs[0].status === "running"}{elapsed(group.runs[0].created_at)}{/if} +
+
- -
- {#if task.result}

{task.result}

{/if} - {#if task.error}

{task.error}

{/if} -
- {:else}
暂无后台任务
{/each} +
+ {#each group.runs as task} +
+
+ + {task.agent_id || "general"} + {task.prompt.slice(0, 80)} + + {task.tool_calls_count} 次工具调用 · {task.iterations} 轮 +
+ {#if task.result}

{task.result.slice(0, 300)}

{/if} + {#if task.error}

{task.error.slice(0, 200)}

{/if} +
+ {/each} +
+ + {/if} + {/each} {:else} {#each jobs as job (job.id)}
@@ -147,4 +186,11 @@ .cron { font-size: 12px; color: var(--text-soft); background: var(--code-bg); padding: 1px 6px; border-radius: 4px; border: 1px solid var(--line); font-variant-numeric: tabular-nums; } .status-dots { display: flex; gap: 4px; margin-top: 6px; align-items: center; } .elapsed { color: var(--info); font-family: var(--font-mono); font-size: 11px; font-variant-numeric: tabular-nums; } + .run-tree { margin-top: 8px; display: flex; flex-direction: column; gap: 6px; } + .run-node { margin-left: calc(var(--depth) * 18px); border-left: 1px solid var(--line); padding-left: 10px; } + .run-row { display: flex; align-items: center; gap: 8px; flex-wrap: wrap; } + .agent-tag { font-size: 11px; color: var(--accent); background: var(--code-bg); border: 1px solid var(--line); border-radius: 4px; padding: 0 6px; font-family: var(--font-mono); } + .run-prompt { flex: 1 1 200px; min-width: 120px; } + .pulse { display: none; } + .pulse.visible { display: inline-block; } diff --git a/webui/src/styles.css b/webui/src/styles.css index 5824d13..6faf365 100644 --- a/webui/src/styles.css +++ b/webui/src/styles.css @@ -328,6 +328,19 @@ button:disabled { cursor: not-allowed; opacity: .45; } .attachment-card strong, .attachment-meta strong { font-size: 12px; } .attachment-card small, .attachment-meta small { margin-top: 2px; color: var(--muted); font-size: 10px; } .attachment-card a, .attachment-meta > a { width: 28px; height: 28px; display: grid; place-items: center; border: 1px solid var(--line); border-radius: 4px; color: var(--accent); text-decoration: none; } +.continuation-tag { justify-self: start; font-size: 10px; color: var(--muted); border: 1px solid var(--line); border-radius: 999px; padding: 1px 8px; background: var(--code-bg); } +.agent-event-cards { display: grid; gap: 7px; margin: 14px 0; } +.agent-event-card { display: flex; gap: 9px; align-items: flex-start; padding: 8px 11px; border: 1px solid var(--line); border-left-width: 3px; border-left-color: var(--info); border-radius: 8px; background: var(--panel); box-shadow: var(--shadow-2); } +.agent-event-card.warning { border-left-color: var(--accent); } +.agent-event-card.critical { border-left-color: var(--danger); } +.agent-event-icon { width: 24px; height: 24px; flex: 0 0 24px; display: grid; place-items: center; border-radius: 5px; color: var(--info); background: var(--code-bg); border: 1px solid var(--line); } +.agent-event-card.warning .agent-event-icon { color: var(--accent); } +.agent-event-card.critical .agent-event-icon { color: var(--danger); } +.agent-event-body { min-width: 0; } +.agent-event-title { display: flex; align-items: center; gap: 7px; font-size: 12px; font-weight: 600; color: var(--text); } +.agent-event-delivery { font-family: var(--font-mono); font-size: 10px; color: var(--muted); border: 1px solid var(--line); border-radius: 4px; padding: 0 5px; } +.agent-event-body p { margin: 2px 0 0; font-size: 12px; color: var(--text-soft); overflow-wrap: anywhere; } +.agent-event-status { font-family: var(--font-mono); } .image-attachment { width: min(100%, 680px); overflow: hidden; border: 1px solid var(--line); border-radius: 8px; background: var(--panel); box-shadow: var(--shadow-2); } .image-preview-link { max-height: 520px; display: grid; overflow: hidden; place-items: center; background: var(--panel-2); } .image-preview { display: block; max-width: 100%; max-height: 520px; object-fit: contain; }