feat(scheduler): unify scheduled task execution and delivery
Replace the dual task/monitor model, NO_REPLY string protocol, and Agent self-delivery with a single Scheduled Run path: claim-time JobRun snapshots, isolated Root/named Agent execution, exactly-once complete_scheduled_run termination, and Scheduler-owned policy delivery through a persistent outbox. - SQLite v11: drop job_kind/model/delete_after_run, add job_runs with status/outcome joint constraints and delivery lease columns; one-shot BEGIN IMMEDIATE migration with atomic rollback. - Non-blocking JoinSet event loop with bounded run/delivery concurrency; terminal commit before any channel I/O; recover unfinished runs as unknown. - ExecutionOrigin::Scheduled propagates to descendants, completion sink is top-level only, background delegation downgrades to foreground. - Typed delivery receipts, fixed target_session_id, idempotent scheduled:<job_run_id> history insert. - New cron_runs read-only tool; cron_add/update drop kind/model; WebUI and Health consume the same JobRun projection. - Bump version to 1.22.0.
This commit is contained in:
xiaoxixi
parent
ddd5efc11b
commit
d9ad58b84b
@ -92,7 +92,7 @@ Scheduler → SessionManager scheduled execution → AgentLoop → Scheduler del
|
||||
- **TurnController** is the only owner of active Turn state; snapshots are complete latest-wins values, not token queues, and `Completed` is published only after atomic persistence succeeds
|
||||
- **DeliveryCoordinator** projects active Turn snapshots without mutating history; it owns `TurnSink` lifecycle but no platform message IDs, which remain private to each sink
|
||||
- **WorkManager** owns the single active plan per session, item state transitions, plan versions, and plan-change events; plans are optional and absent from ordinary chat context
|
||||
- **Scheduler** supports legacy direct-delivery jobs and managed `task`/`monitor` jobs; managed agents cannot call `send_message`, and `on_alert` suppresses only healthy informational results
|
||||
- **Scheduler** owns one unified Scheduled Run path: claim-time JobRun snapshots, isolated Root/named Agent execution, exactly-once `complete_scheduled_run`, structured outcome, and policy-driven outbox delivery. Scheduled origin propagates to descendants, forces background delegation to foreground, and disables direct messaging, signals, Inbox completion slots, and cron/config management tools; `on_alert` suppresses only structured `ok`
|
||||
- **AgentLoop** is stateless across turns; it receives prepared history, drains same-Turn steering only at safe model boundaries, calls LLM providers, executes tools, and returns one result
|
||||
- **Context overflow recovery** is type-driven: before tool progress Session may commit one checkpoint and retry once; after any tool batch AgentLoop may retry the current Provider step once from its in-memory transcript, preserving current tool calls/results, and Session must never restart that Turn from durable history
|
||||
- **Context compaction** keeps `messages` append-only and uses one active checkpoint per Session (`summary + first_retained_seq`) for deterministic Provider projection; `/compact`, Turn-boundary auto compaction, and overflow share the same compactor/CAS commit path, Session restoration never derives context from Timeline or calls a Provider, the Model `token_limit` (default 128K) is the hard window ceiling and an optional Agent `token_limit` can only narrow it via `min(agent, model)`, summary input is bounded from that effective window rather than a fixed cap, and the only automatic threshold is `context_tokens > context_window - effective_reserve`
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "picobot"
|
||||
version = "1.21.0"
|
||||
version = "1.22.0"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
|
||||
11
README.md
11
README.md
@ -17,7 +17,7 @@ PicoBot 是一个用 Rust 编写的个人 AI 助手运行时。它在本地启
|
||||
- 将同一套 Agent 能力接入飞书/Lark，并可选用单张卡片实时更新回复。
|
||||
- 让 Agent 使用本地文件、Shell、搜索、HTTP、浏览器、MCP 工具完成任务。
|
||||
- 把长期偏好、事实和历史摘要存成可检索记忆。
|
||||
- 用 Cron 定时执行任务，并把结果发回目标渠道。
|
||||
- 用 Cron 运行隔离的 Root 或命名 Agent，以结构化结果决定始终通知、异常通知或静默记录。
|
||||
- 通过 Skills 为 Agent 注入项目知识和专用操作指南。
|
||||
|
||||
## 快速开始
|
||||
@ -152,7 +152,7 @@ picobot health --json
|
||||
|
||||
缺少核心或当前配置要求的依赖时退出码为 `1`；`rg` / `fd` 等有回退实现的加速项只会标记为 `DEGRADED`。运行中的 Gateway 也提供 `/health` 斜杠命令，Agent 可调用同名 `health` 工具，WebUI 的“配置 → 健康检查”可显示相同的结构化结果并手动复查；这些入口共享同一套只读检查逻辑。
|
||||
|
||||
Debian/Ubuntu 将同一个 fd 程序安装为 `fdfind`，两者都视为首选文件搜索后端；只有退回传统 `find` 时才提示性能警告。启用浏览器工具后，Health 除了检查 agent-browser 版本和浏览器路径，还会在隔离的临时 socket namespace 中执行完整离线 doctor，分别报告浏览器安装、真实 headless 启动和运行环境，因此可发现“文件存在但 Chrome 无法启动”或缺少 Linux 共享库等问题。
|
||||
Debian/Ubuntu 将同一个 fd 程序安装为 `fdfind`，两者都视为首选文件搜索后端；只有退回传统 `find` 时才提示性能警告。启用浏览器工具后，Health 除了检查 agent-browser 版本和浏览器路径，还会在隔离的临时 socket namespace 中执行完整离线 doctor，分别报告浏览器安装、真实 headless 启动和运行环境，因此可发现“文件存在但 Chrome 无法启动”或缺少 Linux 共享库等问题。Gateway 内按需检查还会报告定时任务的无效 Agent/渠道引用、投递积压、最近失败/超时/unknown、静默 unknown 和执行周期覆盖；Health 不会触发任务或连接 Provider。
|
||||
|
||||
### 5.3 使用 WebUI
|
||||
|
||||
@ -273,7 +273,7 @@ WebUI/TUI 上传文件默认保存到 `~/.picobot/media/cli_chat`，单文件上
|
||||
| `delivery` | 活动 Turn 的展示过滤、latest-wins 节流、终态投递与 TurnSink 生命周期 |
|
||||
| `tools` | Agent 可调用工具集合 |
|
||||
| `storage` | SQLite schema、CRUD、消息和任务持久化 |
|
||||
| `scheduler` | 领取定时任务，执行普通/巡检 Agent，并按投递策略记录或发送结果 |
|
||||
| `scheduler` | 原子领取 occurrence，运行隔离的 Scheduled Agent，并通过持久化 outbox 按策略投递结构化结果 |
|
||||
| `work` | 管理 session 级单 active plan、并行子项状态和 WebSocket 变更事件 |
|
||||
| `skills` | 加载 Skill，并把 Skill 指南注入系统提示 |
|
||||
| `mcp` | 连接 MCP Server，将远端工具包装成普通 Tool |
|
||||
@ -329,7 +329,7 @@ PicoBot 有两类记忆：
|
||||
| Knowledge | 偏好、事实、项目规则、长期可复用信息 | 长期保留，手动删除 |
|
||||
| Timeline | 长对话压缩后的历史摘要 | 默认保留 90 天 |
|
||||
|
||||
每轮处理用户消息时，MemoryManager 会按用户输入召回 Knowledge，并作为运行时上下文附加到本轮用户消息。当前召回上限固定为 5；`memory.recall_limit` 已支持解析但尚未接入 worker。长会话使用一个活动 checkpoint：累计摘要加 `first_retained_seq` 之后的原始消息尾部构成模型上下文，原始消息、工具调用结果、ID 和 seq 均不会被压缩改写。旧工具结果会保留在原始历史中，但 checkpoint 边界推进后不再永久占用 Provider 上下文。成功的语义摘要还会 best-effort 保存为 Timeline，供 `timeline_recall` 检索；Timeline 不参与会话恢复正确性。Scheduler 默认创建一个每日维护巡检，按 `memory.timeline_retention_days` 清理过期 Timeline；Knowledge 不会被自动删除。
|
||||
每轮处理用户消息时，MemoryManager 会按用户输入召回 Knowledge，并作为运行时上下文附加到本轮用户消息。当前召回上限固定为 5；`memory.recall_limit` 已支持解析但尚未接入 worker。长会话使用一个活动 checkpoint：累计摘要加 `first_retained_seq` 之后的原始消息尾部构成模型上下文，原始消息、工具调用结果、ID 和 seq 均不会被压缩改写。旧工具结果会保留在原始历史中，但 checkpoint 边界推进后不再永久占用 Provider 上下文。成功的语义摘要还会 best-effort 保存为 Timeline，供 `timeline_recall` 检索；Timeline 不参与会话恢复正确性。Scheduler 默认创建一个每日维护任务，按 `memory.timeline_retention_days` 清理过期 Timeline；结果通过 `complete_scheduled_run` 结构化提交，Knowledge 不会被自动删除。
|
||||
|
||||
模型的 `models.<name>.token_limit` 给出上下文窗口上限，未配置时默认为 128,000；Agent 的 `agents.<name>.token_limit` 是可选的收紧上限，两者都有配置时有效窗口取二者最小值，因此 Agent 不能扩大模型窗口。自动压缩使用保留量阈值 `context_tokens > context_window - effective_reserve`，默认 reserve 为 16,384 tokens，并尽量原样保留最近 20,000 tokens。小窗口会自动把 reserve 限制为窗口的一半、把近期保留量限制为有效阈值的一半。摘要请求不使用固定 32K 输入上限，而是按有效窗口扣除摘要输出、提示词和安全余量；超大历史只在摘要请求副本中按“已有 checkpoint + 最新消息优先”生成有界 head/tail 转录，SQLite 原文不变。手动 `/compact` 跳过自动阈值；换成小模型后若发送前预检已发现硬超限，或首次请求返回真实 context overflow，语义摘要不可用时才使用明确标记的确定性降级裁剪，正式请求最多重试一次。若 overflow 发生在工具已经执行之后，AgentLoop 只在当前内存转录上裁掉旧完整 Turn 并重试当前模型步骤一次，不会从数据库历史重跑工具。
|
||||
|
||||
@ -353,7 +353,8 @@ PicoBot 有两类记忆：
|
||||
| `todo` | 为复杂、多轮任务创建并更新当前 session 的持久化计划 |
|
||||
| `send_message` | 向指定渠道或当前会话发送消息，可附带文件/截图；WebUI/TUI 当前 Turn 的附件并入最终回复 |
|
||||
| `chat_manager` | 查看渠道、会话和历史消息 |
|
||||
| `cron_add/list/remove/enable/disable/update` | 管理定时任务 |
|
||||
| `cron_add/list/remove/enable/disable/update` | 管理定时任务；`agent_id` 选择 Root/命名 Agent，`delivery_policy` 支持 `always/on_alert/never` |
|
||||
| `cron_runs` | 查询定时任务的结构化执行结果、诊断和投递状态，包括静默任务 |
|
||||
| `routine_maintenance` | 安全清理超过保留期的 Timeline，不删除 Knowledge |
|
||||
| `health` | 检查核心、配置相关和可选运行依赖 |
|
||||
| `browser` | 可选 agent-browser 浏览器自动化；默认按 dialog 临时使用，长期任务可用 `persistent_id` 复用个人 Profile |
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
本文档描述 PicoBot 当前实现的运行时边界、数据流、并发模型和演进约束。它面向维护者和后续参与改进的 Agent，是代码架构的主入口；行为细节仍以代码和测试为最终依据。
|
||||
|
||||
流式模型输出、reasoning 展示、活动 Turn 快照和 Channel 实时投递的详细设计与取舍见 [STREAMING_TURN_DESIGN.md](STREAMING_TURN_DESIGN.md)。用户输入路由、Session 执行拆分、终态投递确认和历史增量校准的重构方案见 [MESSAGE_FLOW_REFACTOR_DESIGN.md](MESSAGE_FLOW_REFACTOR_DESIGN.md)。已实施的 checkpoint 上下文压缩、pi 风格 reserve 阈值、统一编排和 overflow 失败语义见 [CONTEXT_COMPACTION_DESIGN.md](CONTEXT_COMPACTION_DESIGN.md)。配置运行代、重载边界和失败语义见 [CONFIG_HOT_RELOAD_DESIGN.md](CONFIG_HOT_RELOAD_DESIGN.md)。具名子 Agent、委托图、后台收件箱、结果传递机制与 `queue`/`steer` 信号的设计见 [SUB_AGENT_DESIGN.md](SUB_AGENT_DESIGN.md)。计划中的统一 Scheduled Run、结构化终结协议、中央投递和 v11 数据库迁移见 [SCHEDULED_RUN_DESIGN.md](SCHEDULED_RUN_DESIGN.md)。
|
||||
流式模型输出、reasoning 展示、活动 Turn 快照和 Channel 实时投递的详细设计与取舍见 [STREAMING_TURN_DESIGN.md](STREAMING_TURN_DESIGN.md)。用户输入路由、Session 执行拆分、终态投递确认和历史增量校准的重构方案见 [MESSAGE_FLOW_REFACTOR_DESIGN.md](MESSAGE_FLOW_REFACTOR_DESIGN.md)。已实施的 checkpoint 上下文压缩、pi 风格 reserve 阈值、统一编排和 overflow 失败语义见 [CONTEXT_COMPACTION_DESIGN.md](CONTEXT_COMPACTION_DESIGN.md)。配置运行代、重载边界和失败语义见 [CONFIG_HOT_RELOAD_DESIGN.md](CONFIG_HOT_RELOAD_DESIGN.md)。具名子 Agent、委托图、后台收件箱、结果传递机制与 `queue`/`steer` 信号的设计见 [SUB_AGENT_DESIGN.md](SUB_AGENT_DESIGN.md)。已实施的统一 Scheduled Run、结构化终结协议、中央投递和 v11 数据库迁移见 [SCHEDULED_RUN_DESIGN.md](SCHEDULED_RUN_DESIGN.md)。
|
||||
|
||||
## 1. 设计目标
|
||||
|
||||
@ -78,7 +78,7 @@ flowchart LR
|
||||
| `health` | 聚合只读依赖检查，供 CLI、Tool 与 slash command 复用 | 安装、修复或连接 Provider |
|
||||
| `storage` | SQLite schema、迁移、原子 CRUD | 运行时调度策略 |
|
||||
| `memory` | Knowledge/Timeline 的存取与召回 | 直接驱动消息发送 |
|
||||
| `scheduler` | 领取到期任务、执行普通/巡检 Agent、应用投递策略、原子记录结果 | 复用聊天会话历史、直接感知 Channel |
|
||||
| `scheduler` | 原子领取 occurrence、运行隔离的 Root/命名 Agent、提交结构化结果并 drain 持久化投递 outbox | 解析模型自然语言、绕过 Bus 直接调用 Channel |
|
||||
| `work` | session 级单 active plan、并行子项状态机、版本和变更事件 | 执行模型调用、持有 Channel/WebSocket |
|
||||
| `task_supervisor` | 后台任务注册、取消、限时回收 | 业务级重试和结果语义 |
|
||||
|
||||
@ -171,6 +171,18 @@ sequenceDiagram
|
||||
|
||||
不要把“已进入 Bus”误认为“外部渠道已收到”。需要确认语义时必须使用 `deliver_outbound`。
|
||||
|
||||
### Scheduled Run
|
||||
|
||||
Scheduler 不复用聊天历史，也不根据模型正文猜测是否通知。一次到期状态在同一 SQLite 事务中取得 Job 租约、插入 `job_runs(status=claimed)`、快照 Agent/投递目标/策略，并提前推进 recurring `next_run_at`；`At` 在 claim 时立即禁用。事件循环以独立有界 JoinSet 执行 Run 和 drain delivery，长任务不阻塞其他领取或通知。
|
||||
|
||||
每次 Run 通过 `AgentCoordinator` 建立顶层 `agent_runs` 审计记录并执行隔离的 Root 或命名 Agent。Scheduled origin 贯穿所有后代，但只有顶层获得 exactly-once `complete_scheduled_run` sink；后代不继承 sink。Scheduled Agent 不获得 `send_message`、cron/config 管理或 `emit_signal`，不创建 Inbox completion slot，任何 background 委托都收敛为 foreground。普通最终文本不代表成功；没有提交 `ok/alert/failed/refused` 之一即 fail-closed。
|
||||
|
||||
顶层 AgentRun 终态、JobRun 的 lifecycle/outcome/message/diagnostic、Job 最近摘要、初始 delivery status 和租约释放在一个事务中提交。`always` 投递所有 outcome，`on_alert` 只抑制结构化 `ok`，`never` 始终只记录。`job_runs` 同时作为轻量 outbox：`pending → delivering → delivered/failed`，瞬态错误最多进行三次持久化尝试；OutboundDispatcher 返回清洗后的类型化回执，Scheduler 不解析错误字符串。
|
||||
|
||||
首次投递把目标 dialog 固定到 `target_session_id`，并先用稳定消息 ID `scheduled:<job_run_id>` 幂等写入本地历史，再调用 `MessageBus::deliver_outbound`。发送成功但 ack 提交前崩溃允许带稳定 metadata 的重复通知，不能为避免重复而丢失告警。启动恢复把遗留 claimed/running JobRun 原子收敛为 `unknown+unknown`、关联非终态 AgentRun 收敛为 interrupted，并按 claim-time 策略决定是否进入 outbox；已推进的 occurrence 不自动重跑。完整状态矩阵、v11 schema 和迁移规则见 [SCHEDULED_RUN_DESIGN.md](SCHEDULED_RUN_DESIGN.md)。
|
||||
|
||||
按需 Health 检查只读查询任务引用、投递积压、最近失败/超时/unknown、`never+unknown`、Every 周期被执行时长覆盖以及不可计算的 next run；它不执行任务、不连接 Provider，也不修改数据。
|
||||
|
||||
### Control 消息
|
||||
|
||||
WebSocket dialog 操作通过 `ControlMessage` 携带一次性回复通道。Gateway control router 以最多 64 个并发受监督任务调用 `SessionManager`，再将 `SessionEvent` 回传给发起者；慢 control 不阻塞其他聊天的 inbound 路由。Bus 只承载消息，不解释操作。
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# 统一定时任务执行与投递设计
|
||||
|
||||
状态：设计完成，尚未实施
|
||||
状态：已实施
|
||||
|
||||
目标数据库版本：v11
|
||||
|
||||
@ -30,6 +30,7 @@
|
||||
7. 执行结果先持久化，再投递；进程重启后可以恢复未完成投递。
|
||||
8. 无法确认是否完成的执行标记为 `unknown`，不盲目重跑同一 occurrence。
|
||||
9. 使用一次性、原子、可失败回滚的 SQLite v11 自动迁移；迁移后运行时代码只认识新 schema 和新枚举。
|
||||
10. `never` 等静默任务的每次运行仍可通过管理 API、WebUI 和只读工具审计。
|
||||
|
||||
## 3. 非目标
|
||||
|
||||
@ -39,6 +40,7 @@
|
||||
- 不自动把上一次执行的工具结果带到下一次执行。
|
||||
- 不增加 `shell`、Webhook 等第二种 Job 执行类型；本期所有任务仍是 Agent 任务。
|
||||
- 不保留旧字段、旧枚举、旧工具参数或旧字符串协议的运行时兼容分支。
|
||||
- `never` 不提供“失败时例外通知”；选择该策略意味着 `failed`、`refused`、`timed_out` 和 `unknown` 也只进入审计、WebUI 与 Health。
|
||||
|
||||
## 4. 参考项目取舍
|
||||
|
||||
@ -99,7 +101,7 @@ pub struct ScheduledJob {
|
||||
|
||||
- `job_kind`：与 `delivery_policy` 重复；
|
||||
- `model`：当前并未实际应用，且会绕开命名 Agent 的 Provider/Model 定义；
|
||||
- `delete_after_run`：当前工具不开放且始终写 `false`；`Schedule::At` 完成领取后统一禁用，用户显式删除即可。
|
||||
- `delete_after_run`：当前工具不开放且始终写 `false`；`Schedule::At` 在 claim 事务内立即禁用，用户显式删除即可。
|
||||
|
||||
### 5.2 DeliveryPolicy
|
||||
|
||||
@ -172,21 +174,36 @@ pub enum ScheduledRunStatus {
|
||||
|
||||
成功调用终结工具后，生命周期状态是 `completed`，业务结果由 `outcome` 表达；例如 `completed + failed` 表示 Agent 正常结束并明确报告检查失败。Provider 错误或结果协议缺失则是 `failed + failed`。
|
||||
|
||||
两者必须满足以下组合约束，不能独立随意取值：
|
||||
|
||||
| Run status | 合法 Outcome |
|
||||
|---|---|
|
||||
| `claimed` / `running` | `NULL` |
|
||||
| `completed` | `ok` / `alert` / `failed` / `refused` |
|
||||
| `failed` / `timed_out` / `cancelled` / `interrupted` | `failed` |
|
||||
| `unknown` | `unknown` |
|
||||
|
||||
因此 `unknown` status 与 `unknown` outcome 恒同现，且只能由平台恢复路径生成：包括启动时恢复未完成运行，以及迁移无法映射的旧状态；模型不能提交它。历史迁移必须先按生命周期状态归一化，再决定 Outcome，不能产生 `unknown + ok` 等非法组合。
|
||||
|
||||
## 6. 总体架构
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Tick[Scheduler tick] --> Claim[Storage claim occurrence]
|
||||
EventLoop[Scheduler event loop] -->|tick + free slots| Claim[Storage claim occurrence]
|
||||
EventLoop -->|run completion| Reap[reap JoinSet]
|
||||
EventLoop -->|pending due| Delivery[bounded delivery drain]
|
||||
Claim --> RunRow[(job_runs: claimed)]
|
||||
Claim --> Advance[提前推进 next_run / 禁用 At]
|
||||
RunRow --> Resolve[ScheduledAgentRunner]
|
||||
RunRow --> JoinSet[bounded JoinSet]
|
||||
JoinSet --> Resolve[ScheduledAgentRunner]
|
||||
Resolve --> Catalog[Root config / AgentCatalog]
|
||||
Resolve --> Agent[AgentLoop]
|
||||
Agent --> Tools[受限工具 + complete_scheduled_run]
|
||||
Tools --> Outcome[ScheduledOutcome]
|
||||
Outcome --> Commit[原子提交 Run 终态与 delivery_status]
|
||||
Outcome --> Reap
|
||||
Reap --> Commit[原子提交 Run 终态与 delivery_status]
|
||||
Commit -->|suppressed / not_requested| Done[完成]
|
||||
Commit -->|pending| Delivery[Scheduler delivery drain]
|
||||
Commit -->|pending| Delivery
|
||||
Delivery --> Bus[MessageBus / OutboundDispatcher]
|
||||
Bus --> Channel[Channel]
|
||||
Channel --> Ack[delivery_status = delivered]
|
||||
@ -196,13 +213,23 @@ flowchart TD
|
||||
|
||||
| 组件 | 职责 | 明确不负责 |
|
||||
|---|---|---|
|
||||
| `Scheduler` | tick、领取、并发上限、执行超时、恢复、投递 drain | 解释模型自然语言 |
|
||||
| `Scheduler` | 非阻塞事件循环、领取、并发上限、执行超时、恢复、投递 drain | 解释模型自然语言 |
|
||||
| `ScheduledAgentRunner` | 解析 Root/命名 Agent、构造隔离上下文、运行 Agent | 渠道发送、next-run 计算 |
|
||||
| `AgentLoop` | 模型/工具循环、识别终结工具已提交 | Job 状态或渠道策略 |
|
||||
| `complete_scheduled_run` | Schema 校验并提交一次结构化 Outcome | 消息发送、数据库写入 |
|
||||
| `Storage` | occurrence、租约、终态、投递状态的原子转换 | Provider 和 Channel I/O |
|
||||
| `OutboundDispatcher` | 目标有序、瞬态错误重试、渠道调用、类型化投递回执 | Scheduled Run 业务分类 |
|
||||
|
||||
Scheduler 主循环维护一个 `JoinSet` 和固定并发上限，不再像旧实现一样等待整批 Job 全部结束后才进入下一次 poll：
|
||||
|
||||
1. tick 到达时先回收已完成 Run，并按空闲槽位领取新 occurrence；
|
||||
2. 每个 Run 终态提交后立即尝试认领并投递自己的 pending 结果；
|
||||
3. 启动和后续 tick 另行有界回收遗留 pending delivery；
|
||||
4. 长任务只占用一个执行槽，不阻塞其他到期 Job 的领取或已完成结果的投递；
|
||||
5. Run task、delivery task 和主循环都由 TaskSupervisor 拥有，关停时停止 admission、取消并有界 join。
|
||||
|
||||
这仍是一个 Scheduler、一种 Job 和一条投递路径；`JoinSet` 只解决生命周期阻塞，不引入新的任务类型或第二套执行器。
|
||||
|
||||
## 7. 结构化终结协议
|
||||
|
||||
### 7.1 工具定义
|
||||
@ -242,7 +269,25 @@ flowchart TD
|
||||
|
||||
### 7.2 AgentLoop 终结行为
|
||||
|
||||
`ToolExecutionContext` 增加可选的 `ScheduledCompletionSink`。终结工具用一次性 compare-and-set 写入 Outcome。AgentLoop 在每个工具调用之后检查 sink：
|
||||
运行上下文增加两个用途不同的字段：
|
||||
|
||||
```rust
|
||||
pub enum ExecutionOrigin {
|
||||
Interactive,
|
||||
Scheduled { job_run_id: i64 },
|
||||
}
|
||||
|
||||
pub struct ToolExecutionContext {
|
||||
// existing fields...
|
||||
pub execution_origin: ExecutionOrigin,
|
||||
pub scheduled_completion: Option<Arc<ScheduledCompletionSink>>,
|
||||
}
|
||||
```
|
||||
|
||||
- `execution_origin` 是事实标记，普通构造器缺省为 `Interactive`，`execute_scheduled()` 显式设为 Scheduled；每次 Coordinator 构造子 Agent 的 ToolExecutionContext 时从父 context 复制，因此所有子孙都继承。它用来统一禁止 signal/inbox、把 background 委托降级为 foreground；不得通过字符串形式的 session ID 或 Agent ID 推断来源。
|
||||
- `scheduled_completion` 是顶层独占能力，只存在于 Scheduled 顶层 Agent，不得传给任何子 Agent。
|
||||
|
||||
终结工具用一次性 compare-and-set 写入 `ScheduledCompletionSink`。AgentLoop 在每个工具调用之后检查 sink：
|
||||
|
||||
1. 未提交：继续普通工具循环；
|
||||
2. 已提交：把本次终结工具调用和结果写入 Agent transcript；
|
||||
@ -276,11 +321,11 @@ flowchart TD
|
||||
`agent_id = NULL` 表示使用当前 Root Provider、模型、上下文窗口和 Skills。工具从 Root registry 派生，但移除：
|
||||
|
||||
- `send_message`；
|
||||
- `cron_add`、`cron_update`、`cron_remove`、`cron_enable`、`cron_disable`；
|
||||
- `cron_add`、`cron_update`、`cron_remove`、`cron_enable`、`cron_disable`、`cron_list`、`cron_runs`；
|
||||
- `reload_config`；
|
||||
- 只服务于交互会话或后台收件箱控制的工具。
|
||||
|
||||
再注入 `complete_scheduled_run`。Root Scheduled Run 不加载用户聊天历史。
|
||||
再注入 `complete_scheduled_run` 和 `delegate`。`delegate` 的目标仍受 AgentCatalog `root_can_delegate()` 约束，其 background 参数在 Scheduled origin 下统一降级为 foreground。Root Scheduled Run 不加载用户聊天历史。
|
||||
|
||||
### 8.2 命名 Agent 任务
|
||||
|
||||
@ -302,11 +347,16 @@ Scheduled 顶层执行仍复用 `agent_runs` 审计和 transcript，但不增加
|
||||
- `caller_agent_id = SCHEDULER`；
|
||||
- `caller_scope_id = scheduled:<job_id>`；
|
||||
- `root_session_id = scheduled-run:<job_run_id>`，它是审计 scope，不是可接收 Inbox 的 Session；
|
||||
- 顶层 `ToolExecutionContext.session_id = scheduled-run:<job_run_id>`，供 browser 等 session-scoped 状态工具隔离使用；
|
||||
- 顶层 `ToolExecutionContext.agent` 必须指向该 AgentRun 的完整 AgentExecutionContext；Agent 访问授权以其中的 `root_session_id` 和 ancestry 为准，不以 `session_id` 字符串代替；
|
||||
- `execution_origin = Scheduled { job_run_id }`；
|
||||
- `completion_slot_reserved = false`；
|
||||
- `job_runs.agent_run_id` 关联顶层 AgentRun。
|
||||
|
||||
这是执行来源的区别，不是第三种并发模式，因此不扩展 `foreground/background` 枚举。
|
||||
|
||||
Scheduled 顶层及其后代不得创建 `agent_session_state`，不得调用 `reserve_completion_slots`，也不得插入 signal/completion inbox event。`recover_agent_state()` 的通用容量对账不应看到合成 Scheduled scope。
|
||||
|
||||
## 9. 子 Agent 语义
|
||||
|
||||
Scheduled Agent 可以调用 `delegate`，但所有子任务必须在本次 Scheduled Run 内收敛：
|
||||
@ -323,9 +373,9 @@ delegate(mode=background) → 自动改为 foreground，并在工具结果中说
|
||||
- 避免创建无法投递的 `cron:<id>` completion；
|
||||
- 前台子 Agent 仍可并行批量执行并由父 Agent 汇总。
|
||||
|
||||
Scheduled Agent 不能使用 `emit_signal` 向用户 Session 建立旁路。子 Agent 的最终结果作为普通工具结果返回父 Scheduled Agent，只有父 Agent 可以调用 `complete_scheduled_run`。
|
||||
Scheduled Agent 不能使用 `emit_signal` 向用户 Session 建立旁路。即使命名 Agent definition 声明了 signal contract，Scheduled origin 也不得注入 `emit_signal`，不得预留 completion slot。子 Agent 的最终结果作为普通工具结果返回父 Scheduled Agent，只有父 Agent 可以调用 `complete_scheduled_run`。
|
||||
|
||||
`ScheduledCompletionSink` 是顶层运行能力，不随 `ToolExecutionContext` 克隆给子 Agent。Coordinator 构造子 Agent context 时必须显式清空该能力，子 Agent registry 也不得注入终结工具；否则子 Agent 会越过父 Agent 汇总直接结束整次任务。
|
||||
`ExecutionOrigin::Scheduled` 必须随每一层子 Agent 继承，使嵌套委托也保持同步收敛语义；`ScheduledCompletionSink` 则是顶层运行能力，Coordinator 构造子 Agent context 时必须显式清空，子 Agent registry 也不得注入终结工具。二者不能合并为同一个可选字段，否则清空 sink 后嵌套子 Agent 会丢失 Scheduled 限制。
|
||||
|
||||
Root Scheduled Agent 的第一次委托必须走 AgentCatalog 的 `root_can_delegate()` 规则，不能把合成的 `SCHEDULER` 或 `ROOT` 身份误当作命名 Agent 传给 `can_delegate()`。命名 Scheduled Agent 的后续委托仍走 `can_delegate(caller, target)`。这一区分只影响委托边校验，不创建第二条 Scheduled 执行路径。
|
||||
|
||||
@ -338,11 +388,15 @@ Root Scheduled Agent 的第一次委托必须走 AgentCatalog 的 `root_can_dele
|
||||
1. 选择 `enabled=1 AND next_run_at<=now` 且租约为空/过期的 Job；
|
||||
2. 条件更新租约；
|
||||
3. 插入唯一的 `job_runs(status=claimed, delivery_status=awaiting_result)` 行，并把其自增 `id` 作为本次 occurrence 的稳定 ID，同时快照 Agent、投递策略和目标；
|
||||
5. 对 `Every/Cron` 把 `next_run_at` 推进到领取时刻之后的第一个未来时间；
|
||||
6. 对 `At` 立即设置 `enabled=0`；
|
||||
7. 提交后返回 `ClaimedScheduledRun { job_snapshot, run_id, owner }`。
|
||||
4. 对 `Every/Cron` 把 `next_run_at` 推进到领取时刻之后的第一个未来时间；
|
||||
5. 对 `At` 在本次 claim 事务内立即设置 `enabled=0`；
|
||||
6. 提交后返回 `ClaimedScheduledRun { job_snapshot, run_id, owner }`。
|
||||
|
||||
领取条件、Run 插入和 Job 推进处于同一写事务；SQLite 的写串行化和条件租约保证一次到期状态只能提交一个 Run。无需额外维护 occurrence key 或 schedule revision。推进下次时间发生在执行之前，因此进程崩溃不会使本次 occurrence 被自动重放；用户之后重新启用一次性任务会建立一个新的 Run，不会与历史运行冲突。错过的多个历史 tick 不逐个补跑，只执行当前到期 occurrence，并计算下一个未来时间。
|
||||
领取条件、Run 插入和 Job 推进处于同一写事务；SQLite 的写串行化和条件租约保证一次到期状态只能提交一个 Run。无需额外维护 occurrence key 或 schedule revision。推进下次时间发生在执行之前，因此进程崩溃不会使本次 occurrence 被自动重放。错过的多个历史 tick 不逐个补跑，只执行当前到期 occurrence，并计算下一个未来时间。
|
||||
|
||||
`Every` 的间隔从领取/开始时刻计算，而不是从完成时刻计算；例如每小时任务执行 55 分钟，下一次约 5 分钟后到期。`Cron` 始终表达绝对日历时间。任务仍受 Job 租约约束，不允许重叠；如果执行时间持续超过调度间隔，Health 应报告“执行时长覆盖调度间隔”，由管理员调整周期或拆分任务。
|
||||
|
||||
过去时间的 `At` 不能通过创建、更新或重新启用隐式重跑：`cron_add` / `cron_update` 拒绝 `at <= now`，`cron_enable` 拒绝启用已经过期的 At，并要求先把 schedule 更新到未来时间。迁移前已经 enabled 且到期的 At 仍保留为一次合法待执行 occurrence。
|
||||
|
||||
Job 执行租约固定为 `execution_timeout + shutdown_grace`，执行本身必须在 `execution_timeout` 内终止，因此不增加心跳续租任务。终态提交同时校验 `job_run_id + lock_owner + 非终态 status`；租约过期后迟到的旧执行不能修改新 Run 或 Job 摘要。
|
||||
|
||||
@ -362,14 +416,17 @@ claimed/running --process restart--> unknown
|
||||
|
||||
### 10.3 启动恢复
|
||||
|
||||
Gateway 启动、Scheduler admission 尚未开放前：
|
||||
Gateway 启动、Scheduler admission 尚未开放前，Storage 先执行一次 `recover_scheduled_runs(active_generation, now)` 原子事务：
|
||||
|
||||
1. 找到所有 `job_runs.status IN ('claimed','running')`；
|
||||
2. 标记为 `unknown`，写入固定诊断；
|
||||
3. 根据该 run 快照的 delivery policy 设置 `pending` 或 `not_requested`；
|
||||
4. 清除关联 Job 的旧租约；
|
||||
5. 不回退已推进的 `next_run_at`，也不重跑 occurrence；
|
||||
6. 开放 Scheduler admission 后先 drain pending delivery，再领取新任务。
|
||||
2. 把 JobRun 标记为 `status=unknown/outcome=unknown`，写入固定诊断；
|
||||
3. 若已关联非终态 AgentRun，把 AgentRun 标记为 `interrupted`，说明其执行生命周期随旧进程结束；
|
||||
4. 根据 JobRun 快照的 delivery policy 设置 `pending` 或 `not_requested`；
|
||||
5. 清除关联 Job 的旧租约；
|
||||
6. 不回退已推进的 `next_run_at`，也不重跑 occurrence；
|
||||
7. 在同一事务内提交以上跨表变化。
|
||||
|
||||
随后再执行通用 `AgentCoordinator::recover_on_activation()`：已由 Scheduled 恢复事务终结的 AgentRun 不会再次处理。`job_runs.status/outcome` 是业务结果和投递决策的唯一权威；`agent_runs.status=interrupted` 只表示编排执行被进程切断，不得反向覆盖 JobRun 的 `unknown`。开放 Scheduler admission 后先 drain pending delivery，再领取新任务。
|
||||
|
||||
优雅关停由 TaskSupervisor 先停止新领取，再取消/限时等待运行；能够得到明确取消结果时记录 `interrupted`，只有硬崩溃才在下次启动归为 `unknown`。
|
||||
|
||||
@ -401,7 +458,7 @@ pending → delivering → delivered
|
||||
└→ failed（永久失败或次数耗尽）
|
||||
```
|
||||
|
||||
Scheduler 在每次 tick 前后以及启动恢复后有界 drain：
|
||||
Scheduler 在 Run 终态提交后立即尝试该 Run，并在每次 tick 和启动恢复后有界 drain 遗留项：
|
||||
|
||||
- 原子认领 `pending` 或租约过期的 `delivering` 行；
|
||||
- 最多 3 次持久化尝试；
|
||||
@ -420,11 +477,36 @@ Scheduler 在每次 tick 前后以及启动恢复后有界 drain：
|
||||
|
||||
OutboundDispatcher 保留现有单次调用内的短暂、内存级瞬态重试；该调用最终返回的回执算一次持久化 delivery attempt。`TransientFailure`、`TimedOut` 和 `DispatcherClosed` 在未达到持久化尝试上限时回到 `pending`，`PermanentFailure` 直接进入 `failed`。Scheduler 不根据错误字符串猜测是否可重试，也不把“成功写入 outbound 队列”当作渠道送达。
|
||||
|
||||
类型映射固定如下，Channel adapter 必须先把平台错误归一化为正确的 ChannelError：
|
||||
|
||||
| 来源 | Delivery receipt | 行为 |
|
||||
|---|---|---|
|
||||
| Channel 成功返回 | `Delivered` | 标记 delivered |
|
||||
| `ConnectionError` / `SendError` 经 Dispatcher 内部重试耗尽 | `TransientFailure` | 持久化退避后重试 |
|
||||
| Dispatcher 单次发送最终超时 | `TimedOut` | 持久化退避后重试 |
|
||||
| Bus/Dispatcher 关停、lane 暂时饱和 | `DispatcherClosed` / `TransientFailure` | 保留 pending，等待当前或下一运行代 |
|
||||
| channel 不存在、`ConfigError`、明确的永久平台错误 | `PermanentFailure` | 立即标记 failed |
|
||||
|
||||
未知 `Other` 默认按永久失败处理，除非产生它的调用点明确证明可以安全重试。回执只持久化经过清洗和长度限制的 summary，不保存平台响应正文、凭据或临时 URL。Dispatcher 内部的秒级重试与 Scheduler 的持久化分钟级退避职责不同，不得互相递归调用或把每次内部尝试计入 `delivery_attempts`。
|
||||
|
||||
外部发送成功但本地 `delivered` 提交前崩溃时可能重复发送。渠道支持幂等键时传递稳定 ID；不支持时允许“至少一次”并在重试消息 metadata 中标出可能重复。不得为了避免重复而丢失告警。
|
||||
|
||||
### 11.3 会话历史
|
||||
|
||||
通知使用确定性的本地 message ID，例如 `scheduled:<job_run_id>`。向目标会话写历史时执行幂等插入，同一个 JobRun 的重试不会产生多条本地历史。消息来源为：
|
||||
需要投递时先解析一次目标会话，并用 `UPDATE ... WHERE target_session_id IS NULL` 把结果固定到 JobRun。后续重试不得重新选择“当前最近 dialog”，避免用户切换会话后同一个通知写入不同历史。目标会话 ID 同时用于本地历史和 OutboundMessage `_session_id` metadata。
|
||||
|
||||
会话解析或幂等历史事务失败时不得绕过历史直接发送：明确的无效目标进入永久 delivery failure；SQLite busy、运行代关停等瞬态错误回到 pending，并计为本次持久化 delivery attempt。
|
||||
|
||||
通知使用确定性的本地 message ID `scheduled:<job_run_id>`。现有 `messages.id` 已是主键，不增加第二个唯一索引；SessionManager 新增专用 `append_scheduled_notification_if_absent()`，在现有 session persistence lock 下协调内存，Storage 提供对应的原子事务：
|
||||
|
||||
1. 使用固定 message ID 执行 `INSERT ... ON CONFLICT(id) DO NOTHING`；
|
||||
2. 只有实际插入时才更新持久化 Session metadata，并由 SessionManager 把同一消息加入内存、推进 state_version；
|
||||
3. 已存在时 SessionManager 直接复用该消息和固定 target_session_id，不重复修改内存或 Session metadata；
|
||||
4. 历史写入成功后才把 OutboundMessage 交给 `deliver_outbound()`。
|
||||
|
||||
SessionManager 可以使用专用 persistence lock 串行化该流程，但不得在 SQLite I/O 期间持有 Session mutex；数据库提交后重新取得 Session mutex，并以固定 message ID 检查内存后再应用一次变更。
|
||||
|
||||
同一个 JobRun 的重试因此不会产生多条本地历史。消息来源为：
|
||||
|
||||
```text
|
||||
SourceKind::ExternalTrigger
|
||||
@ -475,15 +557,36 @@ from_run_id = agent_run_id
|
||||
- 删除 `model`；
|
||||
- 不接受 `direct`。
|
||||
|
||||
`cron_update` 支持更新 prompt、schedule、channel、chat_id、agent_id 和 delivery_policy。`agent_id: null` 明确切回 Root；字段缺失表示不修改。
|
||||
`cron_add` 和 `cron_update` 共用一个 ScheduledJob validator：channel 必须来自当前 ChannelManager allowlist，命名 `agent_id` 必须存在于当前不可变 AgentCatalog；Storage 只接受已验证的类型，不自行读取运行代配置。重载后 Agent 消失时保留 Job 并在运行时明确失败，见 §8.2。
|
||||
|
||||
`cron_update` 支持更新 prompt、schedule、channel、chat_id、agent_id 和 delivery_policy。`agent_id: null` 明确切回 Root；字段缺失表示不修改。新建或更新 `At` 时要求 `at > now`；`cron_enable` 对已过期 At 返回失败并要求先更新 schedule。
|
||||
|
||||
`cron_disable` 只阻止后续 occurrence，不取消已经 claimed/running 的 Run。`cron_remove` 在存在非终态 JobRun 或 Job 租约时返回 conflict，要求先 disable 并等待当前 Run 到达终态；不得依赖 `ON DELETE CASCADE` 静默删除正在执行的 occurrence。
|
||||
|
||||
`cron_list` 展示：
|
||||
|
||||
```text
|
||||
enabled · agent=web-monitor · delivery=on_alert · next=... · last=ok
|
||||
enabled · agent=web-monitor · delivery=on_alert · next=... · last=alert
|
||||
```
|
||||
|
||||
不再展示 kind 或 model。
|
||||
不再展示 kind、model 或截断的 `last_error`；`last` 只显示 `last_outcome`，详细信息由 `cron_runs` 查询。
|
||||
|
||||
新增只读 `cron_runs` 工具并注册到普通全局 registry；Root 交互 Agent 可用，命名交互 Agent 仍须由 definition allowlist 授权，Scheduled Agent 的收窄规则一律移除它：
|
||||
|
||||
```json
|
||||
{
|
||||
"job_id": "job-id",
|
||||
"limit": 20,
|
||||
"run_id": 123
|
||||
}
|
||||
```
|
||||
|
||||
- `job_id` 必填；
|
||||
- 不传 `run_id` 时列出最近记录，`limit` 缺省 20、范围 1–100，每条返回 run_id、时间、status、outcome、delivery status/attempts、duration 及有界 message/diagnostic 摘要；
|
||||
- 传 `run_id` 时校验它属于该 Job，并返回该次运行的完整结构化字段；message 上限沿用 16 KiB，diagnostic 经过清洗并执行独立上限；
|
||||
- `read_only() == true`，不修改 Job/Run、不触发重投；
|
||||
- `never`、`suppressed` 和投递失败记录与其他策略一样可查询；
|
||||
- 输出不包含 Provider 私有状态、reasoning、凭据或 Agent transcript。
|
||||
|
||||
### 13.2 WebUI
|
||||
|
||||
@ -493,7 +596,10 @@ enabled · agent=web-monitor · delivery=on_alert · next=... · last=ok
|
||||
- 投递：始终通知 / 异常通知 / 从不通知；
|
||||
- 最近 Outcome；
|
||||
- 最近运行生命周期状态；
|
||||
- 投递状态及失败原因。
|
||||
- 投递状态、尝试次数及失败原因；
|
||||
- 最近运行的 message/diagnostic 摘要，并可展开单次详情。
|
||||
|
||||
`GET /api/jobs/{id}/runs` 明确切换到 v11 shape：`id/job_id/scheduled_for/started_at/finished_at/status/outcome/message/diagnostic/duration_ms/delivery_status/delivery_attempts/delivery_error`；删除 `output/error/result_kind`。API 继续受管理认证保护并保持 limit 上限。
|
||||
|
||||
创建表单可以提供三个用户友好模板，但模板不进入后端模型：
|
||||
|
||||
@ -509,6 +615,8 @@ HealthService 增加只读 Scheduler 配置检查：
|
||||
- 非法 channel；
|
||||
- 长时间停留在 pending/delivering 的投递；
|
||||
- 最近一次 `unknown`、`failed` 或 `timed_out`；
|
||||
- `never` Job 最近的 unknown（明确提示其不会自动通知）；
|
||||
- 最近执行时长持续覆盖 `Every` 周期；
|
||||
- enabled Job 无法计算 next run。
|
||||
|
||||
Health 不执行 Job、不连接 Provider、不修复数据。
|
||||
@ -573,6 +681,8 @@ CREATE TABLE job_runs (
|
||||
CHECK (delivery_policy IN ('always','on_alert','never')),
|
||||
target_channel TEXT NOT NULL,
|
||||
target_chat_id TEXT NOT NULL,
|
||||
-- resolved once when delivery is first prepared; stable across retries
|
||||
target_session_id TEXT,
|
||||
|
||||
started_at INTEGER,
|
||||
finished_at INTEGER,
|
||||
@ -603,8 +713,21 @@ CREATE TABLE job_runs (
|
||||
CHECK (agent_id IS NULL OR length(trim(agent_id)) > 0),
|
||||
CHECK (length(trim(target_channel)) > 0),
|
||||
CHECK (length(trim(target_chat_id)) > 0),
|
||||
CHECK (target_session_id IS NULL OR length(trim(target_session_id)) > 0),
|
||||
CHECK (delivery_attempts >= 0),
|
||||
CHECK (duration_ms IS NULL OR duration_ms >= 0),
|
||||
CHECK (
|
||||
(status IN ('claimed','running') AND outcome IS NULL
|
||||
AND finished_at IS NULL AND delivery_status = 'awaiting_result') OR
|
||||
(status = 'completed' AND outcome IS NOT NULL
|
||||
AND outcome IN ('ok','alert','failed','refused')
|
||||
AND finished_at IS NOT NULL AND delivery_status != 'awaiting_result') OR
|
||||
(status IN ('failed','timed_out','cancelled','interrupted')
|
||||
AND outcome IS NOT NULL AND outcome = 'failed'
|
||||
AND finished_at IS NOT NULL AND delivery_status != 'awaiting_result') OR
|
||||
(status = 'unknown' AND outcome IS NOT NULL AND outcome = 'unknown'
|
||||
AND finished_at IS NOT NULL AND delivery_status != 'awaiting_result')
|
||||
),
|
||||
CHECK (
|
||||
(delivery_lease_owner IS NULL AND delivery_lease_until IS NULL) OR
|
||||
(delivery_lease_owner IS NOT NULL AND delivery_lease_until IS NOT NULL)
|
||||
@ -621,7 +744,7 @@ CREATE INDEX idx_job_runs_delivery
|
||||
ON job_runs(delivery_status, delivery_next_attempt_at, delivery_lease_until);
|
||||
```
|
||||
|
||||
不为 AgentCatalog 建数据库外键；Agent 定义是配置运行代资源，不是 SQLite 行。`agent_id` 和 definition hash 的最终审计值保存在关联 AgentRun 中。
|
||||
不为 AgentCatalog 建数据库外键；Agent 定义是配置运行代资源，不是 SQLite 行。`agent_id` 和 definition hash 的最终审计值保存在关联 AgentRun 中。`target_session_id` 也不建 Session 外键：它是投递时固定的路由快照，Session 使用软删除且重试不能因当前 dialog 变化而自动改投；写入历史时仍由 SessionManager 验证目标行和作用域。
|
||||
|
||||
## 15. 自动迁移设计
|
||||
|
||||
@ -632,7 +755,7 @@ ON job_runs(delivery_status, delivery_next_attempt_at, delivery_lease_until);
|
||||
- migration 使用一个专用连接和 `BEGIN IMMEDIATE`，在单个 SQLite 事务中完成；
|
||||
- 任一步失败则整体回滚，`user_version` 保持原值，Gateway 拒绝启动；
|
||||
- v11 运行时代码不读取旧列、不解析旧枚举、不调用旧函数；
|
||||
- 旧 schema 知识只存在于 `storage/migrations/v11_scheduled.rs`；
|
||||
- 旧版本形状探测只存在于冻结的 `storage/migrations/legacy_to_v10.rs` 和 `storage/migrations/v11_scheduled.rs`；
|
||||
- v11 数据库由旧版本二进制打开时，沿用现有“数据库版本过新”拒绝策略；
|
||||
- 不支持自动降级。
|
||||
|
||||
@ -640,12 +763,19 @@ ON job_runs(delivery_status, delivery_next_attempt_at, delivery_lease_until);
|
||||
|
||||
### 15.2 初始化顺序调整
|
||||
|
||||
当前 `init_scheduler_schema()` 在 `migrate_schema()` 之前执行，会让新旧 DDL 的所有权混乱。实施后：
|
||||
当前 `migrate_schema()` 是一段按实际 table shape 累积修补的迁移，并不存在现成的逐版本 `v10::up()`；同时 `init_scheduler_schema()` 在 migration 前创建旧 Scheduler DDL。实施不重写全部 v1–v10 历史，而采用最小版本化：
|
||||
|
||||
1. `migrate_schema()` 成为 Scheduler 表 DDL 的唯一入口；
|
||||
2. 全新数据库直接创建 v11 表；
|
||||
3. `user_version=0` 但已有旧表的数据库先执行现有 legacy→v10 规范化，再执行 v10→v11；
|
||||
4. 测试不再直接调用可绕过 migration 的 `init_scheduler_schema()`，统一通过 `Storage::new` 或专用 latest-schema fixture。
|
||||
1. 删除 migration 前的 `init_scheduler_schema()` 调用，`migrate_schema()` 成为 Scheduler 表 DDL 的唯一入口；
|
||||
2. 把当前累积逻辑冻结、抽取为 `normalize_legacy_to_v10(conn, detected_shape)`，只供 `current < 10` 的启动迁移调用；
|
||||
3. 新增 `migrate_v10_to_v11(conn)`，只接受经过预检的 canonical v10 Scheduler 表；
|
||||
4. 全新数据库直接创建 latest base/Agent/Scheduler v11 schema，不先建立再重建 v10 Scheduler 表；
|
||||
5. `user_version=0` 且已有历史应用表时先运行 legacy normalizer，再运行 v11；若旧数据库从未有 Scheduler 表，normalizer 只处理其他历史表，随后直接创建 v11 Scheduler 表；
|
||||
6. `current=10` 只执行 v11，`current=11` 不执行迁移，`current>11` 拒绝；
|
||||
7. Agent schema 必须在创建带 `agent_runs` 外键的 v11 `job_runs` 之前就绪；
|
||||
8. 整个所需步骤共享同一个专用连接和 `BEGIN IMMEDIATE` 事务，只有最后写 `user_version=11`；
|
||||
9. 测试不再直接调用可绕过 migration 的旧初始化函数，统一通过 `Storage::new`、fresh-schema fixture 或明确的 v10 fixture。
|
||||
|
||||
这只是启动 migration 的版本兼容，不是运行时双路径。无需把历史逻辑拆成 `v1→v2→...→v10`，也不得在 v11 Scheduler 查询中保留 shape probing。
|
||||
|
||||
### 15.3 v10 预检
|
||||
|
||||
@ -666,29 +796,42 @@ ON job_runs(delivery_status, delivery_next_attempt_at, delivery_lease_until);
|
||||
| `delivery_policy=direct` | `always` | 新代码统一中央投递 |
|
||||
| `always/on_alert/never` | 原值 | 保留 |
|
||||
| `model` | 删除 | 当前执行路径未应用该值；迁移日志报告被删除的非空数量 |
|
||||
| `delete_after_run` | 删除 | `At` 任务改为完成领取后禁用 |
|
||||
| `delete_after_run` | 删除 | `At` 任务改为 claim 事务内立即禁用 |
|
||||
| 无 `agent_id` | `NULL` | 使用 Root Scheduled Agent |
|
||||
| `last_status=ok` | `last_outcome=ok` | 仅作列表摘要 |
|
||||
| 其他非空 `last_status` | `last_outcome=failed` | 详细旧信息仍在历史 run |
|
||||
| `last_status` | `last_outcome` | 优先取迁移后最新 JobRun.outcome；没有历史 Run 时 `ok→ok`、其他非空值→failed |
|
||||
| `last_error` | 删除 | 作为最新旧 JobRun diagnostic 的 fallback 后删除，不保留 Job 级兼容列 |
|
||||
|
||||
内置 `picobot-routine-maintenance` 的 prompt 在迁移中按固定 Job ID 改成不含 `NO_REPLY` 的任务描述。用户自定义 prompt 不做不安全的字符串替换；新的系统级 Scheduled 契约明确忽略其中关于 `NO_REPLY`、`send_message` 和旧输出格式的指令。
|
||||
|
||||
### 15.5 历史 JobRun 映射
|
||||
|
||||
历史行只用于审计，绝不因升级重新投递：
|
||||
历史行只用于审计，绝不因升级重新投递。status 与 outcome 必须联合映射：
|
||||
|
||||
| v10 值 | v11 值 |
|
||||
| v10 条件 | v11 status | v11 outcome |
|
||||
|---|---|---|
|
||||
| `status=ok` | `completed` | 按下表 result_kind 映射 |
|
||||
| `status=delivery_error` 且已有模型结果/result_kind | `completed` | 按下表 result_kind 映射 |
|
||||
| `status=delivery_error` 且执行 error 非空、无结果 | `failed` | `failed` |
|
||||
| `status=timeout` | `timed_out` | `failed` |
|
||||
| `status=error` | `failed` | `failed` |
|
||||
| 其他状态 | `unknown` | `unknown` |
|
||||
|
||||
仅在新 status 为 `completed` 时读取 result_kind：
|
||||
|
||||
| v10 result_kind | v11 outcome |
|
||||
|---|---|
|
||||
| `quiet` | `ok` |
|
||||
| `content` 且 Job 为 `on_alert` | `alert` |
|
||||
| `content` 其他情况 | `ok` |
|
||||
| `reported_failure` | `failed` |
|
||||
| `refused` | `refused` |
|
||||
| `NULL`（旧 Direct） | `ok` |
|
||||
| 其他无法映射值 | migration error，不猜测 |
|
||||
|
||||
delivery 单独映射，不反向改变执行 status/outcome：
|
||||
|
||||
| v10 delivery_status | v11 delivery_status |
|
||||
|---|---|
|
||||
| `status=ok` | `status=completed` |
|
||||
| `status=timeout` | `status=timed_out` |
|
||||
| `status=error/delivery_error` | `status=failed` |
|
||||
| 其他状态 | `status=unknown` |
|
||||
| `result_kind=quiet` | `outcome=ok` |
|
||||
| `result_kind=content` 且 Job 为 `on_alert` | `outcome=alert` |
|
||||
| `result_kind=content` 其他情况 | `outcome=ok` |
|
||||
| `result_kind=reported_failure` | `outcome=failed` |
|
||||
| `result_kind=refused` | `outcome=refused` |
|
||||
| 无法映射 | `outcome=unknown` |
|
||||
| `delivery_status=direct/delivered` | `delivered` |
|
||||
| `suppressed/skipped` | `suppressed` |
|
||||
| `failed` 或存在 `delivery_error` | `failed` |
|
||||
@ -698,11 +841,14 @@ ON job_runs(delivery_status, delivery_next_attempt_at, delivery_lease_until);
|
||||
|
||||
- `scheduled_for = started_at`；
|
||||
- `message = output`；
|
||||
- `diagnostic = error`；
|
||||
- `diagnostic = error`；最新历史 Run 的 error 为空时可以用同 Job 的 `last_error` 补全；
|
||||
- `target_*`、`delivery_policy` 从迁移时的 Job 行快照；
|
||||
- `target_session_id = NULL`，历史行不重新投递，因此不解析会话；
|
||||
- `agent_run_id = NULL`；
|
||||
- 所有 delivery lease 字段为空。
|
||||
|
||||
若 Job 存在非空 `last_error` 但没有任何可承载它的历史 Run，migration 记录 Job ID 和计数后丢弃该冗余摘要，不为兼容旧投影伪造一条执行记录。
|
||||
|
||||
### 15.6 迁移时发现旧执行锁
|
||||
|
||||
v10 只有 Job lease，没有预先建立的 JobRun。若迁移时发现 `lock_owner` 非空或 lease 尚未清理，说明旧进程可能在终态提交前退出：
|
||||
@ -720,16 +866,17 @@ v10 只有 Job lease，没有预先建立的 JobRun。若迁移时发现 `lock_o
|
||||
|
||||
SQLite 删除列和修改 CHECK 约束采用 canonical table rebuild：
|
||||
|
||||
1. `CREATE TABLE scheduled_jobs_v11 ...`；
|
||||
2. 写入转换后的 Job；
|
||||
3. `CREATE TABLE job_runs_v11 ... REFERENCES scheduled_jobs_v11`；
|
||||
4. 写入转换后的历史 Run 和旧锁恢复 Run；
|
||||
5. 删除旧 `job_runs`；
|
||||
6. 删除旧 `scheduled_jobs`；
|
||||
7. rename `scheduled_jobs_v11`、`job_runs_v11`；
|
||||
8. 重建索引；
|
||||
9. 执行 `PRAGMA foreign_key_check` 并要求零行；
|
||||
10. 最后设置 `PRAGMA user_version = 11` 并提交。
|
||||
1. 确保 latest `agent_runs` schema 已创建；
|
||||
2. 把旧 `job_runs`、`scheduled_jobs` 依次 rename 为内部 legacy 临时表；
|
||||
3. 使用 fresh database 相同的 v11 DDL 常量创建 canonical `scheduled_jobs` 和 `job_runs`；
|
||||
4. 写入转换后的 Job、历史 Run 和旧锁恢复 Run；
|
||||
5. 删除 legacy `job_runs`；
|
||||
6. 删除 legacy `scheduled_jobs`；
|
||||
7. 重建索引；
|
||||
8. 执行 `PRAGMA foreign_key_check` 并要求零行；
|
||||
9. 最后设置 `PRAGMA user_version = 11` 并提交。
|
||||
|
||||
全新数据库走 latest schema creation，不经过临时 v10 表和上述数据搬迁；但使用同一组 v11 DDL 常量，避免 fresh schema 与 migrated schema 漂移。
|
||||
|
||||
表重建必须在同一连接、同一事务中完成。不得用多个 pool connection 分散 DDL，也不得在事务提交前启动 Scheduler。
|
||||
|
||||
@ -766,7 +913,7 @@ SQLite 删除列和修改 CHECK 约束采用 canonical table rebuild：
|
||||
- Agent 返回普通文本后再分类的代码；
|
||||
- 先发送、后写 JobRun 的顺序。
|
||||
|
||||
替换为：claim occurrence → `ScheduledAgentRunner` → typed outcome → terminal commit → delivery drain。
|
||||
替换为：非阻塞 tick/JoinSet 事件循环 → claim occurrence → `ScheduledAgentRunner` → typed outcome → terminal commit → immediate/bounded delivery drain。删除等待整批 Run 完成后才继续 poll 的 `for_each_concurrent(...).await` 结构。
|
||||
|
||||
### `src/session/session.rs`
|
||||
|
||||
@ -790,15 +937,15 @@ Scheduled Agent 构造迁移到 Agent/Coordinator 边界，SessionManager 不再
|
||||
- `set_scheduled_job_behavior()`；
|
||||
- 完成时才插入 JobRun 的旧事务。
|
||||
|
||||
新增严格 typed parser、occurrence claim、终态提交、未知恢复和持久化 delivery claim。
|
||||
新增严格 typed parser、occurrence claim、status/outcome 联合约束、跨 JobRun/AgentRun 的原子未知恢复、终态提交、固定 target_session_id 和持久化 delivery claim。
|
||||
|
||||
### `src/storage/mod.rs` 与 `src/storage/migrations/`
|
||||
|
||||
把 `SCHEMA_VERSION` 一次提升到 11，删除 migration 之前调用 Scheduler 旧 DDL 初始化函数的顺序。新增唯一的 v11 Scheduler migration 模块和跨表终态事务 API；迁移完成后通用 Storage 查询不得包含任何 v10 列名。
|
||||
把 `SCHEMA_VERSION` 一次提升到 11，删除 migration 之前调用 Scheduler 旧 DDL 初始化函数的顺序。把现有累积迁移冻结为 `legacy_to_v10` normalizer，新增唯一的 v11 Scheduler migration、fresh v11 DDL 和跨表终态/恢复事务 API；迁移完成后通用 Storage 查询不得包含任何 v10 列名。
|
||||
|
||||
### `src/tools/cron.rs`
|
||||
|
||||
删除 `kind`、`model` 参数与所有默认联动；默认 `delivery_policy=always`。新增可选 `agent_id`，更新 list 输出和测试。
|
||||
删除 `kind`、`model` 参数与所有默认联动；默认 `delivery_policy=always`。新增可选 `agent_id` 和共享 Agent/Channel validator，拒绝过去的 At；更新 list 输出，并新增只读 `CronRunsTool`。
|
||||
|
||||
### `src/tools`
|
||||
|
||||
@ -806,22 +953,23 @@ Scheduled Agent 构造迁移到 Agent/Coordinator 边界，SessionManager 不再
|
||||
|
||||
### `src/agent`
|
||||
|
||||
新增 `ScheduledCompletionSink` 和 `AgentCoordinator::execute_scheduled()`；复用 foreground AgentRun 持久化，不创建 inbox completion。AgentLoop 在终结工具成功后停止。
|
||||
新增可继承的 `ExecutionOrigin::Scheduled`、顶层独占的 `ScheduledCompletionSink` 和 `AgentCoordinator::execute_scheduled()`；复用 foreground AgentRun 持久化，不创建 agent_session_state、signal 或 inbox completion。AgentLoop 在终结工具成功后停止。
|
||||
|
||||
### `src/session/messenger.rs`
|
||||
|
||||
把 Scheduled 通知改成稳定 message ID 的幂等写入；删除任何依赖 `cron:<id>` 作为可消费会话的行为。
|
||||
第一次投递时固定 target_session_id；把 Scheduled 通知改成利用现有 `messages.id` 主键的稳定 message ID/原子 insert-if-absent 写入；只有实际插入时更新 Session 内存和 metadata。删除任何依赖 `cron:<id>` 作为可消费会话的行为。
|
||||
|
||||
### `src/bus`
|
||||
|
||||
保留 `MessageBus::deliver_outbound()` 的等待回执入口，把 `OutboundMessage.delivery` 和 `BusError` 的字符串结果改为类型化、可判定 retry class 的安全回执。OutboundDispatcher 仍是唯一 Channel 调用方；普通无回执消息继续使用 `publish_outbound()`。
|
||||
保留 `MessageBus::deliver_outbound()` 的等待回执入口，把 `OutboundMessage.delivery` 和 `BusError` 的字符串结果改为类型化、可判定 retry class 的安全回执，并补齐 ChannelError→receipt 映射。OutboundDispatcher 仍是唯一 Channel 调用方；普通无回执消息继续使用 `publish_outbound()`。
|
||||
|
||||
### Gateway / WebUI / Protocol
|
||||
|
||||
- API JSON 删除 `job_kind`、`model`、`delete_after_run`、`result_kind`；
|
||||
- 增加 `agent_id`、`outcome`、delivery attempts/状态；
|
||||
- Gateway 激活时先执行原子 Scheduled 恢复，再执行通用 Agent recovery，最后开放 Scheduler admission；
|
||||
- API JSON 删除 `job_kind`、`model`、`delete_after_run`、`output/error/result_kind`；
|
||||
- 增加 `agent_id`、`outcome/message/diagnostic` 和 delivery attempts/状态；JobRun 内部另存固定 target Session 路由快照，不向非管理客户端暴露；
|
||||
- `webui/src/pages/TasksPage.svelte` 删除“巡检/任务”判断；
|
||||
- Health 与任务详情展示新的结构化状态。
|
||||
- `/api/jobs/{id}/runs`、TasksPage、Health 与 `cron_runs` 消费同一 JobRun projection。
|
||||
|
||||
### 文档与内置知识
|
||||
|
||||
@ -843,19 +991,26 @@ Scheduled Agent 构造迁移到 Agent/Coordinator 边界，SessionManager 不再
|
||||
1. 一个 Job 同时最多有一个持租约 occurrence；短周期任务不会重叠执行。
|
||||
2. occurrence 的 JobRun 在 Agent 启动前持久化；同一个到期状态只能提交一个 Run。
|
||||
3. `next_run_at` 在 claim 事务中推进；执行失败和进程崩溃不会重放同一 occurrence。
|
||||
4. 只有 lease owner 可以把 run 从非终态推进到终态。
|
||||
5. 所有终态不可重写；迟到 Agent 结果必须丢弃并记录日志。
|
||||
6. Outcome 和投递策略都使用 claim-time snapshot；运行过程中编辑 Job 只影响后续 occurrence。
|
||||
7. Outcome、Job 摘要和初始 delivery status 在同一事务提交。
|
||||
8. 渠道 I/O 不发生在 SQLite 事务或 Session mutex 内。
|
||||
9. Scheduled Run 不预留 Agent Inbox slot，也不向合成 session 投递 completion。
|
||||
10. 后台委托在模型可见参数不变，但 Scheduled context 强制变为 foreground。
|
||||
11. `never` 严格不通知，包括失败、拒绝、超时和 unknown；这些状态仍可通过管理页面和 Health 查看。
|
||||
12. 静默只来源于结构化 `ok` 与策略矩阵，不能来源于普通文本。
|
||||
4. `At` 在 claim 事务内禁用；过期 At 不能通过 enable 隐式重跑。
|
||||
5. disable 只影响未来 occurrence；存在非终态 Run 时禁止删除 Job。
|
||||
6. Scheduler 主循环不等待整批执行结束；长任务只占并发槽，不能阻塞其他 Job 或 pending delivery。
|
||||
7. 只有 `job_run_id + lease owner + 非终态 status` 同时匹配才可以提交终态。
|
||||
8. 所有终态不可重写；迟到 Agent 结果必须丢弃并记录日志。
|
||||
9. status/outcome 必须满足 §5.4 联合约束；`unknown` 只能由运行时生成。
|
||||
10. Outcome 和投递策略都使用 claim-time snapshot；运行过程中编辑 Job 只影响后续 occurrence。
|
||||
11. Outcome、Job 摘要和初始 delivery status 在同一事务提交。
|
||||
12. 渠道 I/O 不发生在 SQLite 事务或 Session mutex 内。
|
||||
13. Scheduled origin 传递给所有后代；ScheduledCompletionSink 只属于顶层。
|
||||
14. Scheduled Run 及其后代不创建 agent_session_state、不预留 Inbox slot、不发 signal/completion event。
|
||||
15. 后台委托在模型可见参数不变，但 Scheduled origin 强制变为 foreground。
|
||||
16. JobRun 是 Outcome/投递的唯一权威；关联 AgentRun 只提供编排审计，不能反向改写 JobRun。
|
||||
17. 第一次投递解析出的 target_session_id、message ID 和目标 metadata 在所有重试中保持不变。
|
||||
18. `never` 严格不通知，包括失败、拒绝、超时和 unknown；这些状态仍可通过 `cron_runs`、管理页面和 Health 查看。
|
||||
19. 静默只来源于结构化 `ok` 与策略矩阵，不能来源于普通文本。
|
||||
|
||||
## 18. 错误处理矩阵
|
||||
|
||||
| 场景 | Run status | Outcome | 投递内容来源 |
|
||||
| 场景 | Scheduled JobRun status | Outcome | 投递内容来源 |
|
||||
|---|---|---|---|
|
||||
| Agent 提交 `ok` | completed | ok | Agent message |
|
||||
| Agent 提交 `alert` | completed | alert | Agent message |
|
||||
@ -865,7 +1020,7 @@ Scheduled Agent 构造迁移到 Agent/Coordinator 边界，SessionManager 不再
|
||||
| Provider 失败 | failed | failed | 安全归一化错误，不含响应正文 |
|
||||
| 运行超时 | timed_out | failed | Scheduler 固定超时说明 |
|
||||
| Gateway 优雅取消 | interrupted | failed | Scheduler 固定中断说明 |
|
||||
| 硬崩溃恢复 | unknown | unknown | Scheduler 固定 unknown 说明 |
|
||||
| 硬崩溃恢复 | unknown | unknown | Scheduler 固定 unknown 说明；关联 AgentRun 审计为 interrupted |
|
||||
| Agent 定义不存在 | failed | failed | Agent ID 和修复建议 |
|
||||
| 渠道发送永久失败 | 原 run 不变 | 原 outcome | delivery_status=failed |
|
||||
|
||||
@ -875,14 +1030,15 @@ Scheduled Agent 构造迁移到 Agent/Coordinator 边界，SessionManager 不再
|
||||
|
||||
本设计应在一个功能版本中完成，不能长期保留两套路径：
|
||||
|
||||
1. 增加 v11 migration 和新 Storage 类型、测试；
|
||||
2. 增加 Scheduled completion sink、终结工具和 AgentLoop 终止语义；
|
||||
3. 增加 `AgentCoordinator::execute_scheduled()`；
|
||||
4. 改写 Scheduler claim、执行、恢复和 delivery drain；
|
||||
5. 切换 Cron tools/API/WebUI；
|
||||
6. 更新内置维护任务和文档；
|
||||
7. 删除所有旧类型、旧函数和魔法字符串；
|
||||
8. 运行全量验证后再提交，并按功能变化增加产品中段版本号一次。
|
||||
1. 抽取 legacy→v10 normalizer，增加 fresh v11/v10→v11 migration、新 Storage 类型与迁移测试；
|
||||
2. 增加 `ExecutionOrigin::Scheduled`、Scheduled completion sink、终结工具和 AgentLoop 终止语义；
|
||||
3. 增加 `AgentCoordinator::execute_scheduled()`，落实同步委托、signal/inbox 禁止和顶层 AgentRun；
|
||||
4. 改写 Scheduler 为非阻塞 JoinSet 事件循环，实现 occurrence claim、跨表恢复、终态提交和持久化 delivery drain；
|
||||
5. 实现固定 target_session_id、幂等历史插入和类型化 Bus 回执；
|
||||
6. 切换 Cron tools（含 `cron_runs`）、管理 API、WebUI 与 Health；
|
||||
7. 更新内置维护任务和文档；
|
||||
8. 删除所有旧类型、旧函数、旧字段读取和魔法字符串；
|
||||
9. 运行全量验证后再提交，并按功能变化增加产品中段版本号一次。
|
||||
|
||||
代码合并点只允许新路径。迁移代码可以先写，但最终提交中不允许 Scheduler 通过 feature flag 或 schema 判断走旧路径。
|
||||
|
||||
@ -895,25 +1051,40 @@ Scheduled Agent 构造迁移到 Agent/Coordinator 边界，SessionManager 不再
|
||||
- 终结工具之后的同批工具归约为 Cancelled；
|
||||
- 普通文本、所有 `NO_REPLY` 变体都不能产生 `ok`；
|
||||
- named/root Agent 解析和工具收窄；
|
||||
- Scheduled background delegate 自动前台化；
|
||||
- Scheduled origin 继承到多层子 Agent，而 completion sink 只在顶层；
|
||||
- Scheduled background delegate 在任意嵌套深度自动前台化；
|
||||
- Scheduled Agent/子 Agent 不注入 emit_signal、不预留 slot、不创建 agent_session_state；
|
||||
- 两个并发 Scheduler 对同一到期 Job 只有一个 claim 成功；
|
||||
- At claim 后禁用；
|
||||
- 新建/更新过去 At 被拒绝，过期 At 无法直接 enable；
|
||||
- disable 不影响活动 Run，remove 在活动 Run/租约存在时返回 conflict；
|
||||
- recurring claim 时推进到未来；
|
||||
- 长 Run 不阻塞其他到期 Job 的 claim 或已完成 Run 的投递；
|
||||
- lease owner 条件提交；
|
||||
- 迟到结果不能覆盖 unknown/timeout；
|
||||
- status/outcome 非法组合被数据库约束拒绝；
|
||||
- Scheduled 恢复在一个事务中完成 JobRun unknown、AgentRun interrupted、delivery 决策和租约清理；
|
||||
- delivery claim、瞬态重试、永久失败和尝试上限；
|
||||
- ChannelError 到类型化 delivery receipt 的完整映射；
|
||||
- `deliver_outbound` 只有收到 Channel 成功回执才返回 Delivered，入队成功不算送达；
|
||||
- 回执保留 transient/permanent 分类且不泄露渠道敏感响应；
|
||||
- 本地历史稳定 message ID 去重。
|
||||
- 第一次投递后 target_session_id 固定，切换当前 dialog 不改变重试目标；
|
||||
- 本地历史稳定 message ID 去重，重复调用不重复推进 Session metadata；
|
||||
- `cron_runs` 摘要/单条详情、limit 边界、Job 所属校验和 read_only 声明。
|
||||
|
||||
### Migration 测试
|
||||
|
||||
- 空数据库直接得到 v11；
|
||||
- fresh v11 与迁移所得 v11 的 `sqlite_master` schema 等价；
|
||||
- `user_version=0` 的历史数据库先规范化再升级，缺少旧 Scheduler 表时直接创建 v11 Scheduler；
|
||||
- 真实 v10 fixture 保留所有 Job 和历史 Run；
|
||||
- `task/monitor` 列被物理删除；
|
||||
- `direct` 全部转为 `always`，运行时无法解析 direct；
|
||||
- 非空 model 计数被记录且列被删除；
|
||||
- delete_after_run 列被删除；
|
||||
- `last_error` 只补入最近历史 Run diagnostic，v11 Job 表不保留该列；
|
||||
- 旧 delivery_error 按“执行是否已有结果”分别映射 completed/failed，且 delivery 始终为 failed；
|
||||
- 迁移不会生成非法的 status/outcome 组合；
|
||||
- 历史 Run 不产生 pending delivery；
|
||||
- 旧锁生成 unknown run 并按策略决定 pending/not_requested；
|
||||
- 内置维护 prompt 不含 `NO_REPLY`；
|
||||
@ -928,12 +1099,16 @@ Scheduled Agent 构造迁移到 Agent/Coordinator 边界，SessionManager 不再
|
||||
- `on_alert + ok` 静默但 JobRun 可查询；
|
||||
- `on_alert + alert/failed/refused` 实际投递；
|
||||
- `never` 在所有 Outcome 下均不投递；
|
||||
- `never` 的完整结果可以通过 `cron_runs` 和管理 API 查询；
|
||||
- Provider 普通文本完成被转为协议失败通知；
|
||||
- Gateway 在结果提交后、发送前重启，pending 在启动后恢复；
|
||||
- Gateway 在发送后、ack 前重启，允许有标识的重复但不丢失；
|
||||
- Gateway 在 Agent 执行中退出，恢复为 unknown 且不重跑同一 occurrence；
|
||||
- 恢复后关联 AgentRun 为 interrupted，但任何消费者都不能用它覆盖 JobRun unknown；
|
||||
- 命名 Agent 被删除后 Job 明确失败且 Gateway 仍能启动；
|
||||
- Cron 内 foreground 子 Agent 汇总成功，background 参数不会产生 Inbox 事件。
|
||||
- Cron 内 foreground 子 Agent 汇总成功，嵌套 background 参数不会产生 Inbox/agent_session_state；
|
||||
- 投递失败后切换当前 dialog，重试仍写入并指向首次固定的 target_session_id；
|
||||
- 一个执行到超时上限的 Job 不延迟其他 Job 和 pending delivery。
|
||||
|
||||
### 必跑验证
|
||||
|
||||
@ -958,4 +1133,10 @@ git diff --check
|
||||
7. `on_alert + ok` 不投递，其余矩阵行为与本设计一致。
|
||||
8. 结果提交后崩溃不会丢失待投递消息。
|
||||
9. 执行中崩溃产生 `unknown`，同一 occurrence 不自动重跑。
|
||||
10. v10 数据库首次启动自动迁移到 v11，失败时完整回滚；迁移后没有运行时兼容代码。
|
||||
10. Scheduled 恢复原子地产生 JobRun unknown 和 AgentRun interrupted，JobRun 始终是投递权威。
|
||||
11. Scheduled origin 贯穿所有后代，但 completion sink 只属于顶层；数据库中不产生合成 agent_session_state 或 Inbox event。
|
||||
12. Scheduler 的长 Run 不阻塞其他到期 Job 或 pending delivery。
|
||||
13. 过期 At 不能直接新建、更新或重新启用。
|
||||
14. 每次投递重试复用固定 target_session_id 和 `scheduled:<job_run_id>` message ID，本地历史和 Session metadata 均不重复。
|
||||
15. `cron_runs`、管理 API 和 WebUI 都能查询 `never` 等静默任务的结构化结果。
|
||||
16. v10 数据库首次启动自动迁移到 v11，失败时完整回滚；fresh/migrated v11 schema 一致，迁移后没有运行时兼容代码。
|
||||
|
||||
@ -54,7 +54,13 @@
|
||||
"gateway": {
|
||||
"host": "127.0.0.1",
|
||||
"port": 19876,
|
||||
"require_pairing": true
|
||||
"require_pairing": true,
|
||||
"scheduler": {
|
||||
"enabled": true,
|
||||
"poll_interval_secs": 60,
|
||||
"max_concurrent": 1,
|
||||
"execution_timeout_secs": 900
|
||||
}
|
||||
},
|
||||
"client": {
|
||||
"gateway_url": "ws://127.0.0.1:19876/ws"
|
||||
|
||||
@ -10,7 +10,7 @@ Channel → MessageBus.inbound → Gateway processor → SessionManager → per-
|
||||
AgentLoop → TurnEvent → TurnController → latest TurnSnapshot → DeliveryCoordinator → TurnSink → Channel
|
||||
|
||||
WebSocket/Channel → MessageBus.control → Gateway processor → SessionManager (dialog 操作)
|
||||
Scheduler → SessionManager.handle_cron_message → AgentLoop → send_message
|
||||
Scheduler → occurrence/JobRun claim → AgentCoordinator → isolated Scheduled Agent → complete_scheduled_run → JobRun outbox → SessionManager/MessageBus
|
||||
```
|
||||
|
||||
## 模块职责
|
||||
|
||||
@ -102,8 +102,8 @@ Gateway WebUI 的“配置”页可以编辑实际加载的配置文件。读取
|
||||
|------|------|------|------|
|
||||
| `enabled` | bool | true | 是否启动调度器并注册 cron 工具 |
|
||||
| `poll_interval_secs` | int | 60 | 检查到期任务的轮询间隔 |
|
||||
| `max_concurrent` | int | 1 | 每批到期任务的最大并发数，运行时限制在 1–256 |
|
||||
| `execution_timeout_secs` | int | 900 | 单个定时任务 Agent 执行的硬超时；租约会覆盖执行和托管投递等待 |
|
||||
| `max_concurrent` | int | 1 | 同时执行的 Scheduled Run 上限，运行时限制在 1–256；投递使用独立有界并发 |
|
||||
| `execution_timeout_secs` | int | 900 | 单个定时任务 Agent 执行的硬超时；Job 执行租约额外覆盖关停宽限，投递由持久化 outbox 独立恢复 |
|
||||
|
||||
## memory 字段
|
||||
|
||||
@ -116,7 +116,7 @@ Gateway WebUI 的“配置”页可以编辑实际加载的配置文件。读取
|
||||
| `timeline_retention_days` | int | 90 | 默认日常维护巡检删除超过该期限的 Timeline；Knowledge 不受影响 |
|
||||
| `max_failures_before_degrade` | int | 3 | 预留的归并失败阈值；当前无失败降级循环 |
|
||||
|
||||
注意：当前 worker 的 Knowledge 召回数量仍固定为 5；idle consolidation 和失败降级循环尚未接入。Timeline 清理由默认启用的 `picobot-routine-maintenance` 定时巡检执行。
|
||||
注意：当前 worker 的 Knowledge 召回数量仍固定为 5；idle consolidation 和失败降级循环尚未接入。Timeline 清理由默认启用的 `picobot-routine-maintenance` Scheduled Run 执行；该任务使用 `never` 策略，结构化结果只进入运行审计和 Health。
|
||||
|
||||
## channels.feishu 字段
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
数据库为 SQLite，默认位于配置目录（`~/.picobot`）`data/` 下的 `picobot.db`，与 workspace 相互独立。
|
||||
|
||||
连接启用 WAL、`synchronous=NORMAL`、foreign keys、5 秒 busy timeout，连接池最多 8 个连接。当前 `PRAGMA user_version=10`；启动时会在事务内补齐旧库字段和索引，遇到比程序更新的 schema version 会拒绝启动。
|
||||
连接启用 WAL、`synchronous=NORMAL`、foreign keys、5 秒 busy timeout，连接池最多 8 个连接。当前 `PRAGMA user_version=11`；启动时会在单个事务内迁移旧库，遇到比程序更新的 schema version 会拒绝启动。
|
||||
|
||||
## sessions 表
|
||||
|
||||
@ -176,22 +176,21 @@ background 完成/信号投递的唯一事实源：`pending → leased → admit
|
||||
| `name` | TEXT | 任务名称 |
|
||||
| `schedule` | TEXT | 调度规则 JSON（at/every/cron） |
|
||||
| `prompt` | TEXT | 任务提示词 |
|
||||
| `channel` | TEXT | 执行渠道 |
|
||||
| `agent_id` | TEXT | 可选命名 Agent；NULL 表示 Root |
|
||||
| `channel` | TEXT | 目标渠道 |
|
||||
| `chat_id` | TEXT | 目标对话 |
|
||||
| `model` | TEXT | 可选模型标记；当前会存储/展示，但 Scheduler 执行仍使用默认 Agent 模型 |
|
||||
| `delivery_policy` | TEXT | `always` / `on_alert` / `never` |
|
||||
| `enabled` | INTEGER | 是否启用 (1/0) |
|
||||
| `delete_after_run` | INTEGER | 执行后自动删除 (1/0) |
|
||||
| `next_run_at` | INTEGER | 下次执行时间 |
|
||||
| `last_run_at` | INTEGER | 上次执行时间 |
|
||||
| `last_status` | TEXT | 上次执行状态 |
|
||||
| `last_error` | TEXT | 上次错误信息 |
|
||||
| `last_outcome` | TEXT | 最近结构化结果：ok/alert/failed/refused/unknown |
|
||||
| `locked_at` | INTEGER | 本次领取时间 |
|
||||
| `lock_owner` | TEXT | 领取任务的 Scheduler owner UUID |
|
||||
| `lease_until` | INTEGER | 租约到期时间；进程崩溃后允许其他实例重新领取 |
|
||||
| `lock_owner` | TEXT | 本次 occurrence 的唯一 owner token |
|
||||
| `lease_until` | INTEGER | 租约到期时间 |
|
||||
| `created_at` | INTEGER | 创建时间（Unix 毫秒） |
|
||||
| `updated_at` | INTEGER | 更新时间（Unix 毫秒） |
|
||||
|
||||
Scheduler 使用原子 `UPDATE ... RETURNING` 领取到期任务。任务结果、下次运行时间和租约释放在同一事务中提交，并校验 owner，防止过期 worker 覆盖已恢复的任务。
|
||||
Scheduler 在领取事务中插入 JobRun、快照执行/投递字段并推进下次时间。`At` 在领取时立即禁用；执行失败或崩溃不重放同一个 occurrence。
|
||||
|
||||
## job_runs 表
|
||||
|
||||
@ -199,12 +198,26 @@ Scheduler 使用原子 `UPDATE ... RETURNING` 领取到期任务。任务结果
|
||||
|------|------|------|
|
||||
| `id` | INTEGER PK | 自增 ID |
|
||||
| `job_id` | TEXT FK | 关联任务，外键关联 scheduled_jobs(id) |
|
||||
| `scheduled_for` | INTEGER | 本 occurrence 原计划时间 |
|
||||
| `agent_run_id` | TEXT FK | 顶层 AgentRun 审计记录 |
|
||||
| `agent_id` | TEXT | claim-time Agent 快照 |
|
||||
| `delivery_policy` | TEXT | claim-time 投递策略快照 |
|
||||
| `target_channel` / `target_chat_id` | TEXT | claim-time 目标快照 |
|
||||
| `target_session_id` | TEXT | 首次投递时固定的目标 dialog |
|
||||
| `started_at` | INTEGER | 开始时间 |
|
||||
| `finished_at` | INTEGER | 结束时间 |
|
||||
| `status` | TEXT | 执行状态 |
|
||||
| `output` | TEXT | 执行输出 |
|
||||
| `error` | TEXT | 错误信息 |
|
||||
| `status` | TEXT | claimed/running/completed/failed/timed_out/cancelled/interrupted/unknown |
|
||||
| `outcome` | TEXT | ok/alert/failed/refused/unknown；与 status 有联合约束 |
|
||||
| `message` | TEXT | 面向用户的结构化结果 |
|
||||
| `diagnostic` | TEXT | 有界内部诊断 |
|
||||
| `duration_ms` | INTEGER | 耗时（毫秒） |
|
||||
| `delivery_status` | TEXT | awaiting_result/not_requested/suppressed/pending/delivering/delivered/failed |
|
||||
| `delivery_attempts` | INTEGER | 持久化投递尝试次数，最多 3 次 |
|
||||
| `delivery_next_attempt_at` | INTEGER | 瞬态失败后的退避时间 |
|
||||
| `delivery_lease_owner` / `delivery_lease_until` | TEXT / INTEGER | outbox 领取租约 |
|
||||
| `delivery_error` | TEXT | 清洗后的投递失败摘要 |
|
||||
|
||||
JobRun 是执行结果和投递状态的唯一权威。顶层 AgentRun 与 JobRun 终态、Job 最近摘要和租约释放原子提交；启动恢复将遗留运行归为 `unknown`，不会自动重跑。
|
||||
|
||||
## llm_calls 表
|
||||
|
||||
|
||||
@ -50,14 +50,15 @@
|
||||
|
||||
## Cron 定时任务工具
|
||||
|
||||
Cron 不是一个带 `action` 的统一工具，而是六个独立工具；仅在 `gateway.scheduler.enabled=true` 时注册。
|
||||
Cron 不是一个带 `action` 的统一工具，而是七个独立工具；仅在 `gateway.scheduler.enabled=true` 时注册。
|
||||
|
||||
| 工具 | 主要参数 | 说明 |
|
||||
|------|----------|------|
|
||||
| `cron_add` | `schedule`, `prompt`, `channel`, `chat_id`; 可选 `name`, `model` | 创建任务 |
|
||||
| `cron_add` | `schedule`, `prompt`, `channel`, `chat_id`; 可选 `name`, `agent_id`, `delivery_policy` | 创建任务 |
|
||||
| `cron_list` | 可选 `status=all|enabled|disabled` | 列出任务 |
|
||||
| `cron_update` | `job_id`; 可选 `prompt`, `schedule`, `channel`, `chat_id`, `model` | 更新指定字段 |
|
||||
| `cron_remove` | `job_id` | 永久删除任务和关联 job runs |
|
||||
| `cron_runs` | `job_id`; 可选 `run_id`, `limit` | 查询结构化运行和投递记录，包括静默结果 |
|
||||
| `cron_update` | `job_id`; 可选 `name`, `prompt`, `schedule`, `channel`, `chat_id`, `agent_id`, `delivery_policy` | 更新指定字段；`agent_id:null` 切回 Root |
|
||||
| `cron_remove` | `job_id` | 无活动 Run 或 pending delivery 时永久删除任务 |
|
||||
| `cron_enable` | `job_id` | 启用并重新计算下次运行时间 |
|
||||
| `cron_disable` | `job_id` | 禁用但保留任务 |
|
||||
|
||||
@ -69,7 +70,9 @@ Cron 不是一个带 `action` 的统一工具，而是六个独立工具；仅
|
||||
{"type":"cron","expr":"0 0 9 * * *","tz":"Asia/Shanghai"}
|
||||
```
|
||||
|
||||
时间戳和间隔单位为毫秒；Cron 表达式为 6 段（秒、分、时、日、月、周）。定时 Agent 不复用聊天历史，`prompt` 必须包含完整上下文。`kind` 可为 `task` 或 `monitor`；`delivery_policy` 可为 `always`、`on_alert` 或 `never`。托管任务由 Scheduler 投递，巡检返回 `NO_REPLY[INFO]` 时静默，`NO_REPLY[FAIL]`/`NO_REPLY[REFUSE]` 仍视为需关注结果。升级前创建的任务保留 Agent 直接调用 `send_message` 的兼容行为。`model` 当前会持久化和展示，但执行仍使用默认 Agent Provider/Model，不能依赖它实现模型覆盖。
|
||||
时间戳和间隔单位为毫秒；Cron 表达式为 6 段（秒、分、时、日、月、周）。过去时间的 At 不能创建、更新或直接重新启用。定时 Agent 不复用聊天历史，`prompt` 必须包含完整上下文；`agent_id` 省略时使用 Root，否则使用当前 AgentCatalog 中的命名 Agent。
|
||||
|
||||
每次运行必须恰好一次调用 `complete_scheduled_run(outcome,message)`，outcome 只能是 `ok`、`alert`、`failed`、`refused`。普通最终文本不会被解释为结果，缺少结构化终结会 fail-closed。投递完全由 Scheduler 决定：`always` 投递所有结果，`on_alert` 只抑制 `ok`，`never` 只保留记录。Scheduled Agent 不能自行发送最终通知；子 Agent 委托会同步完成，也不会产生后台 Inbox/Signal。
|
||||
|
||||
---
|
||||
|
||||
|
||||
@ -78,6 +78,12 @@
|
||||
"max_files_per_message": 8,
|
||||
"max_message_bytes": 67108864,
|
||||
"pending_ttl_seconds": 3600
|
||||
},
|
||||
"scheduler": {
|
||||
"enabled": true,
|
||||
"poll_interval_secs": 60,
|
||||
"max_concurrent": 1,
|
||||
"execution_timeout_secs": 900
|
||||
}
|
||||
},
|
||||
"client": {
|
||||
|
||||
@ -1240,6 +1240,31 @@ impl AgentLoop {
|
||||
}
|
||||
completed_tool_batches = completed_tool_batches.saturating_add(1);
|
||||
|
||||
if let Some(outcome) = tool_context
|
||||
.scheduled_completion
|
||||
.as_ref()
|
||||
.and_then(|sink| sink.outcome())
|
||||
{
|
||||
Self::close_steering(turn.as_ref());
|
||||
let mut final_message = ChatMessage::assistant(outcome.message);
|
||||
attach_reply_media(&mut final_message, &reply_media_refs);
|
||||
Self::annotate_message(&mut final_message, turn.as_ref(), iteration, true);
|
||||
emitted_messages.push(final_message.clone());
|
||||
self.forward_to_transcript_sink(&final_message);
|
||||
crate::observability::metrics::global_metrics().record_turn(
|
||||
Some(&accumulated_usage),
|
||||
turn_start.elapsed().as_millis() as u64,
|
||||
);
|
||||
return Ok(AgentProcessResult {
|
||||
final_response: final_message,
|
||||
emitted_messages,
|
||||
total_tokens: Some(accumulated_tokens),
|
||||
usage: Some(accumulated_usage),
|
||||
last_request_usage,
|
||||
last_request_digest,
|
||||
});
|
||||
}
|
||||
|
||||
// A complete tool batch is the first safe steering boundary. Do
|
||||
// not drain at the final available iteration: those inputs must
|
||||
// remain in the closed mailbox for Session to queue after this
|
||||
@ -1517,6 +1542,17 @@ impl AgentLoop {
|
||||
let mut outcomes = Vec::with_capacity(tool_calls.len());
|
||||
|
||||
for tool_call in tool_calls {
|
||||
if context
|
||||
.scheduled_completion
|
||||
.as_ref()
|
||||
.is_some_and(|sink| sink.is_completed())
|
||||
{
|
||||
outcomes.push(ToolExecutionOutcome::failure(
|
||||
"Cancelled: scheduled run was already completed".to_string(),
|
||||
Some("scheduled run was already completed".to_string()),
|
||||
));
|
||||
continue;
|
||||
}
|
||||
if context.cancellation.is_cancelled() {
|
||||
return Err(AgentError::Cancelled);
|
||||
}
|
||||
@ -1666,6 +1702,58 @@ mod tests {
|
||||
requests: std::sync::atomic::AtomicUsize,
|
||||
}
|
||||
|
||||
struct ScheduledCompletionProvider {
|
||||
requests: std::sync::atomic::AtomicUsize,
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl LLMProvider for ScheduledCompletionProvider {
|
||||
async fn stream(
|
||||
&self,
|
||||
_request: ChatCompletionRequest,
|
||||
) -> Result<ProviderStream, crate::providers::DynProviderError> {
|
||||
self.requests
|
||||
.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
|
||||
Ok(crate::providers::provider_stream_for_test(
|
||||
ChatCompletionResponse {
|
||||
id: "scheduled-complete".to_string(),
|
||||
model: "scheduled-complete".to_string(),
|
||||
content: String::new(),
|
||||
reasoning_content: None,
|
||||
provider_state: None,
|
||||
tool_calls: vec![
|
||||
ToolCall {
|
||||
id: "complete".to_string(),
|
||||
name: "complete_scheduled_run".to_string(),
|
||||
arguments: serde_json::json!({
|
||||
"outcome": "ok",
|
||||
"message": "healthy"
|
||||
}),
|
||||
},
|
||||
ToolCall {
|
||||
id: "late-side-effect".to_string(),
|
||||
name: "side_effect".to_string(),
|
||||
arguments: serde_json::json!({}),
|
||||
},
|
||||
],
|
||||
usage: Usage::default(),
|
||||
},
|
||||
))
|
||||
}
|
||||
|
||||
fn ptype(&self) -> &str {
|
||||
"test"
|
||||
}
|
||||
|
||||
fn name(&self) -> &str {
|
||||
"scheduled-complete"
|
||||
}
|
||||
|
||||
fn model_id(&self) -> &str {
|
||||
"scheduled-complete"
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl LLMProvider for AlwaysOverflowProvider {
|
||||
async fn stream(
|
||||
@ -1725,6 +1813,53 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn scheduled_completion_ends_the_loop_and_cancels_later_batch_calls() {
|
||||
let provider = Arc::new(ScheduledCompletionProvider {
|
||||
requests: std::sync::atomic::AtomicUsize::new(0),
|
||||
});
|
||||
let executions = Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||
let tools = Arc::new(ToolRegistry::new());
|
||||
tools.register(crate::tools::CompleteScheduledRunTool::new());
|
||||
tools.register(CountingSideEffectTool {
|
||||
executions: executions.clone(),
|
||||
});
|
||||
let agent = AgentLoop::with_provider_and_tools(
|
||||
provider.clone(),
|
||||
tools,
|
||||
3,
|
||||
"scheduled-complete".to_string(),
|
||||
PathBuf::from("."),
|
||||
Vec::new(),
|
||||
);
|
||||
let sink = Arc::new(crate::tools::ScheduledCompletionSink::default());
|
||||
let context = ToolExecutionContext::for_session("scheduled-run:1")
|
||||
.with_execution_origin(crate::tools::ExecutionOrigin::Scheduled { job_run_id: 1 })
|
||||
.with_scheduled_completion(sink.clone());
|
||||
|
||||
let result = agent
|
||||
.process_with_context(vec![ChatMessage::user("check")], context)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(result.final_response.content, "healthy");
|
||||
assert_eq!(
|
||||
sink.outcome().unwrap().kind,
|
||||
crate::storage::ScheduledOutcomeKind::Ok
|
||||
);
|
||||
assert_eq!(executions.load(std::sync::atomic::Ordering::SeqCst), 0);
|
||||
assert_eq!(
|
||||
provider.requests.load(std::sync::atomic::Ordering::SeqCst),
|
||||
1
|
||||
);
|
||||
assert!(result.emitted_messages.iter().any(|message| {
|
||||
message.role == "tool"
|
||||
&& message
|
||||
.content
|
||||
.contains("scheduled run was already completed")
|
||||
}));
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl LLMProvider for OverflowAfterToolProvider {
|
||||
async fn stream(
|
||||
|
||||
@ -32,6 +32,16 @@ pub struct BackgroundAdmission {
|
||||
pub run_ids: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ScheduledAgentExecution {
|
||||
pub agent_run_id: String,
|
||||
pub status: crate::storage::ScheduledRunStatus,
|
||||
pub outcome: Option<crate::tools::ScheduledOutcome>,
|
||||
pub error: Option<String>,
|
||||
pub agent_terminal: AgentTerminalOutcome,
|
||||
pub runtime_generation: i64,
|
||||
}
|
||||
|
||||
pub struct AgentCoordinator {
|
||||
storage: Arc<Storage>,
|
||||
manager: Arc<SubAgentManager>,
|
||||
@ -88,6 +98,263 @@ impl AgentCoordinator {
|
||||
})
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub async fn execute_scheduled(
|
||||
self: &Arc<Self>,
|
||||
job_run_id: i64,
|
||||
lease_owner: &str,
|
||||
job_id: &str,
|
||||
job_name: &str,
|
||||
agent_id: Option<&str>,
|
||||
prompt: &str,
|
||||
timeout_secs: u64,
|
||||
) -> Result<ScheduledAgentExecution, CoordinatorError> {
|
||||
let run_id = Uuid::new_v4().to_string();
|
||||
let root_session_id = format!("scheduled-run:{job_run_id}");
|
||||
let sink = Arc::new(crate::tools::ScheduledCompletionSink::default());
|
||||
let caller = ToolExecutionContext::for_session(root_session_id.clone())
|
||||
.with_turn_id(format!("scheduled:{job_run_id}"))
|
||||
.with_execution_origin(crate::tools::ExecutionOrigin::Scheduled { job_run_id });
|
||||
let contract = format!(
|
||||
"## Unattended Scheduled Run\n\nYou are executing scheduled task “{job_name}” ({job_id}). The user will not see ordinary final text. After completing all necessary work, you must call complete_scheduled_run exactly once. Use ok only when the task completed and found nothing requiring attention; use alert for actionable findings; use failed when the task did not complete reliably; use refused for a permission or safety refusal. Legacy textual suppression and direct-messaging instructions are obsolete."
|
||||
);
|
||||
let config = SubAgentConfig {
|
||||
target: agent_id.map(str::to_string),
|
||||
prompt: prompt.to_string(),
|
||||
context: Some(contract.clone()),
|
||||
mode: ExecutionMode::Foreground,
|
||||
allowed_tools: None,
|
||||
max_iterations: None,
|
||||
timeout_secs: Some(timeout_secs),
|
||||
plan_item_id: None,
|
||||
session_id: Some(root_session_id.clone()),
|
||||
};
|
||||
let mut resolution = if agent_id.is_some() {
|
||||
let mut resolution = self.manager.resolve_agent(&config, &caller, &run_id)?;
|
||||
resolution
|
||||
.tools
|
||||
.register(crate::tools::CompleteScheduledRunTool::new());
|
||||
resolution.tool_context.session_id = Some(root_session_id.clone());
|
||||
resolution.tool_context.scheduled_completion = Some(sink.clone());
|
||||
resolution.timeout_secs = resolution.timeout_secs.min(timeout_secs);
|
||||
resolution.signal_contract = None;
|
||||
resolution
|
||||
} else {
|
||||
self.manager
|
||||
.resolve_scheduled_root(&caller, &run_id, timeout_secs, sink.clone())?
|
||||
};
|
||||
resolution.tool_context.execution_origin =
|
||||
crate::tools::ExecutionOrigin::Scheduled { job_run_id };
|
||||
|
||||
let now = chrono::Utc::now().timestamp_millis();
|
||||
let deadline_at = now.saturating_add((resolution.timeout_secs * 1000) as i64);
|
||||
let new_run = NewAgentRun {
|
||||
id: run_id.clone(),
|
||||
root_session_id,
|
||||
root_turn_id: None,
|
||||
parent_run_id: None,
|
||||
caller_agent_id: "SCHEDULER".to_string(),
|
||||
caller_scope_id: format!("scheduled:{job_id}"),
|
||||
idempotency_key: Some(format!("scheduled:{job_run_id}")),
|
||||
agent_id: agent_id.unwrap_or("ROOT").to_string(),
|
||||
definition_hash: resolution.definition_hash.clone().unwrap_or_default(),
|
||||
provider_profile: resolution.llm_profile.clone().unwrap_or_default(),
|
||||
provider_name: resolution.provider_config.name.clone(),
|
||||
model_id: resolution.provider_config.model_id.clone(),
|
||||
mode: AgentRunMode::Foreground,
|
||||
depth: 1,
|
||||
plan_item_id: None,
|
||||
execution_id: run_id.clone(),
|
||||
task: prompt.to_string(),
|
||||
context_json: None,
|
||||
budget_json: serde_json::json!({
|
||||
"remaining_runs": self.manager.catalog().max_runs_per_tree().saturating_sub(1),
|
||||
"remaining_depth": self.manager.catalog().max_tree_depth().saturating_sub(1),
|
||||
})
|
||||
.to_string(),
|
||||
signal_contract_json: None,
|
||||
signal_delivery: None,
|
||||
deadline_at,
|
||||
runtime_generation: self.runtime_generation,
|
||||
completion_slot_reserved: false,
|
||||
};
|
||||
match self
|
||||
.storage
|
||||
.accept_agent_runs(AcceptAgentRequest {
|
||||
runs: vec![new_run],
|
||||
now,
|
||||
})
|
||||
.await?
|
||||
{
|
||||
AcceptedAgentRuns::Accepted { .. } => {}
|
||||
AcceptedAgentRuns::Existing { .. } => {
|
||||
return Err(CoordinatorError::Rejected(format!(
|
||||
"scheduled occurrence {job_run_id} already has an Agent run"
|
||||
)));
|
||||
}
|
||||
}
|
||||
let mut mark_attempt = 0_u64;
|
||||
let marked = match loop {
|
||||
match self
|
||||
.storage
|
||||
.mark_scheduled_run_running(job_run_id, lease_owner, Some(&run_id), now)
|
||||
.await
|
||||
{
|
||||
Err(error) if error.is_transient() && mark_attempt < 2 => {
|
||||
mark_attempt += 1;
|
||||
tokio::time::sleep(std::time::Duration::from_millis(50 * mark_attempt)).await;
|
||||
}
|
||||
result => break result,
|
||||
}
|
||||
} {
|
||||
Ok(marked) => marked,
|
||||
Err(error) => {
|
||||
let _ = self
|
||||
.storage
|
||||
.cancel_agent_run_with_completion(
|
||||
&run_id,
|
||||
"scheduled occurrence could not enter running state",
|
||||
true,
|
||||
now,
|
||||
)
|
||||
.await;
|
||||
return Err(error.into());
|
||||
}
|
||||
};
|
||||
if !marked {
|
||||
let _ = self
|
||||
.storage
|
||||
.cancel_agent_run_with_completion(
|
||||
&run_id,
|
||||
"scheduled occurrence was no longer active",
|
||||
true,
|
||||
now,
|
||||
)
|
||||
.await;
|
||||
return Err(CoordinatorError::Rejected(format!(
|
||||
"scheduled occurrence {job_run_id} lost its lease"
|
||||
)));
|
||||
}
|
||||
|
||||
let (execution, agent_terminal) = match self
|
||||
.execute_scheduled_agent_run(&run_id, &config, resolution)
|
||||
.await
|
||||
{
|
||||
Ok(execution) => execution,
|
||||
Err(error) => (
|
||||
Err(error),
|
||||
AgentTerminalOutcome::Failed {
|
||||
error: "scheduled Agent could not enter its execution lifecycle".to_string(),
|
||||
prompt_tokens: None,
|
||||
completion_tokens: None,
|
||||
cost: None,
|
||||
signal_ids: Vec::new(),
|
||||
},
|
||||
),
|
||||
};
|
||||
let status = match &execution {
|
||||
Ok(result) => match &result.status {
|
||||
TaskStatus::Completed => crate::storage::ScheduledRunStatus::Completed,
|
||||
TaskStatus::Failed(_) => crate::storage::ScheduledRunStatus::Failed,
|
||||
TaskStatus::TimedOut => crate::storage::ScheduledRunStatus::TimedOut,
|
||||
TaskStatus::Cancelled => crate::storage::ScheduledRunStatus::Interrupted,
|
||||
},
|
||||
Err(_) => crate::storage::ScheduledRunStatus::Failed,
|
||||
};
|
||||
let error = match &execution {
|
||||
Ok(result) => match &result.status {
|
||||
TaskStatus::Completed => None,
|
||||
TaskStatus::Failed(_) => Some(
|
||||
"scheduled Agent execution failed; inspect Gateway logs for details"
|
||||
.to_string(),
|
||||
),
|
||||
TaskStatus::TimedOut => Some("scheduled Agent timed out".to_string()),
|
||||
TaskStatus::Cancelled => Some("scheduled Agent was cancelled".to_string()),
|
||||
},
|
||||
Err(_) => Some(
|
||||
"scheduled Agent execution failed; inspect Gateway logs for details".to_string(),
|
||||
),
|
||||
};
|
||||
Ok(ScheduledAgentExecution {
|
||||
agent_run_id: run_id,
|
||||
status,
|
||||
outcome: sink.outcome(),
|
||||
error,
|
||||
agent_terminal,
|
||||
runtime_generation: self.runtime_generation,
|
||||
})
|
||||
}
|
||||
|
||||
async fn execute_scheduled_agent_run(
|
||||
self: &Arc<Self>,
|
||||
run_id: &str,
|
||||
config: &SubAgentConfig,
|
||||
resolution: ResolvedAgentRun,
|
||||
) -> Result<
|
||||
(
|
||||
Result<SubAgentResult, CoordinatorError>,
|
||||
AgentTerminalOutcome,
|
||||
),
|
||||
CoordinatorError,
|
||||
> {
|
||||
let execution_id = run_id.to_string();
|
||||
let token = resolution.tool_context.cancellation.clone();
|
||||
self.active_tokens.insert(run_id.to_string(), token);
|
||||
let started = self
|
||||
.storage
|
||||
.mark_agent_run_running(run_id, &execution_id, chrono::Utc::now().timestamp_millis())
|
||||
.await?;
|
||||
if !started {
|
||||
self.active_tokens.remove(run_id);
|
||||
return Err(CoordinatorError::Rejected(format!(
|
||||
"scheduled Agent run {run_id} was closed before execution started"
|
||||
)));
|
||||
}
|
||||
|
||||
let result = self
|
||||
.manager
|
||||
.execute_resolved(config, resolution, run_id)
|
||||
.await
|
||||
.map_err(CoordinatorError::SubAgent);
|
||||
self.active_tokens.remove(run_id);
|
||||
let terminal = match &result {
|
||||
Ok(result) => match &result.status {
|
||||
TaskStatus::Completed => AgentTerminalOutcome::Completed {
|
||||
result: result.full_content.clone(),
|
||||
prompt_tokens: None,
|
||||
completion_tokens: None,
|
||||
cost: None,
|
||||
tool_calls: result.tool_calls_count as i64,
|
||||
iterations: result.iterations as i64,
|
||||
signal_ids: Vec::new(),
|
||||
},
|
||||
TaskStatus::Failed(error) => AgentTerminalOutcome::Failed {
|
||||
error: error.clone(),
|
||||
prompt_tokens: None,
|
||||
completion_tokens: None,
|
||||
cost: None,
|
||||
signal_ids: Vec::new(),
|
||||
},
|
||||
TaskStatus::TimedOut => AgentTerminalOutcome::TimedOut {
|
||||
deadline_at: chrono::Utc::now().timestamp_millis(),
|
||||
signal_ids: Vec::new(),
|
||||
},
|
||||
TaskStatus::Cancelled => AgentTerminalOutcome::Interrupted {
|
||||
reason: "scheduled Agent interrupted by shutdown".to_string(),
|
||||
signal_ids: Vec::new(),
|
||||
},
|
||||
},
|
||||
Err(error) => AgentTerminalOutcome::Failed {
|
||||
error: error.to_string(),
|
||||
prompt_tokens: None,
|
||||
completion_tokens: None,
|
||||
cost: None,
|
||||
signal_ids: Vec::new(),
|
||||
},
|
||||
};
|
||||
Ok((result, terminal))
|
||||
}
|
||||
|
||||
/// Admit a named background run for the root caller and spawn its runner.
|
||||
/// Completion is guaranteed by the reserved inbox slot; the returned ID
|
||||
/// is only valid when every durable step succeeded.
|
||||
@ -100,6 +367,11 @@ impl AgentCoordinator {
|
||||
caller: &ToolExecutionContext,
|
||||
configs: Vec<SubAgentConfig>,
|
||||
) -> Result<BackgroundAdmission, CoordinatorError> {
|
||||
if caller.execution_origin.is_scheduled() {
|
||||
return Err(CoordinatorError::Rejected(
|
||||
"scheduled Agents cannot create background runs".to_string(),
|
||||
));
|
||||
}
|
||||
if caller.agent.is_some() {
|
||||
return Err(CoordinatorError::Rejected(
|
||||
"nested background runs are not available yet; only the root Agent may delegate background work".to_string(),
|
||||
|
||||
@ -21,7 +21,7 @@ pub use context_compaction::{
|
||||
ContextRequestKey, ContextUsageTracker, PreviousCheckpoint, SequencedMessage,
|
||||
context_request_digest, estimate_tokens,
|
||||
};
|
||||
pub use coordinator::{AgentCoordinator, CoordinatorError};
|
||||
pub use coordinator::{AgentCoordinator, CoordinatorError, ScheduledAgentExecution};
|
||||
pub use definition::{AgentDefinition, AgentLimits};
|
||||
pub use gate::ExecutionGate;
|
||||
pub use inbox::{AgentInboxNotifier, AgentInboxWakeTarget};
|
||||
|
||||
@ -10,6 +10,22 @@ use crate::providers::{LLMProvider, create_provider};
|
||||
use crate::skills::SkillsLoader;
|
||||
use crate::tools::{ToolExecutionContext, ToolRegistry};
|
||||
|
||||
const SCHEDULED_DISABLED_TOOLS: &[&str] = &[
|
||||
"send_message",
|
||||
"cron_add",
|
||||
"cron_update",
|
||||
"cron_remove",
|
||||
"cron_enable",
|
||||
"cron_disable",
|
||||
"cron_list",
|
||||
"cron_runs",
|
||||
"reload_config",
|
||||
"agent_task",
|
||||
"chat_manager",
|
||||
"todo",
|
||||
"emit_signal",
|
||||
];
|
||||
|
||||
const DEFAULT_MAX_ITERATIONS: usize = 99;
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
@ -191,7 +207,13 @@ impl SubAgentManager {
|
||||
})?;
|
||||
let cancellation = caller.cancellation.child_token();
|
||||
let execution = if let Some(parent) = caller.agent.as_ref() {
|
||||
if !self.catalog.can_delegate(&parent.current_agent_id, target) {
|
||||
let allowed =
|
||||
if caller.execution_origin.is_scheduled() && parent.current_agent_id == "ROOT" {
|
||||
self.catalog.root_can_delegate(target)
|
||||
} else {
|
||||
self.catalog.can_delegate(&parent.current_agent_id, target)
|
||||
};
|
||||
if !allowed {
|
||||
return Err(SubAgentError::Other(format!(
|
||||
"Agent '{}' is not allowed to delegate to '{target}'",
|
||||
parent.current_agent_id
|
||||
@ -235,10 +257,14 @@ impl SubAgentManager {
|
||||
.budget
|
||||
.remaining_depth
|
||||
.min(definition.limits.max_depth);
|
||||
child.signal_contract = definition
|
||||
child.signal_contract = if caller.execution_origin.is_scheduled() {
|
||||
None
|
||||
} else {
|
||||
definition
|
||||
.signal_contract
|
||||
.as_ref()
|
||||
.map(|contract| Arc::new(contract.clone()));
|
||||
.map(|contract| Arc::new(contract.clone()))
|
||||
};
|
||||
Arc::new(child)
|
||||
} else {
|
||||
if !self.catalog.root_can_delegate(target) {
|
||||
@ -252,7 +278,11 @@ impl SubAgentManager {
|
||||
run_id: task_id.to_string(),
|
||||
execution_id: task_id.to_string(),
|
||||
parent_run_id: None,
|
||||
caller_agent_id: "ROOT".to_string(),
|
||||
caller_agent_id: if caller.execution_origin.is_scheduled() {
|
||||
"SCHEDULER".to_string()
|
||||
} else {
|
||||
"ROOT".to_string()
|
||||
},
|
||||
current_agent_id: target.to_string(),
|
||||
ancestry: vec![target.to_string()],
|
||||
depth: 1,
|
||||
@ -267,10 +297,14 @@ impl SubAgentManager {
|
||||
.min(definition.limits.max_depth),
|
||||
},
|
||||
tree_runs: Arc::new(std::sync::atomic::AtomicUsize::new(1)),
|
||||
signal_contract: definition
|
||||
signal_contract: if caller.execution_origin.is_scheduled() {
|
||||
None
|
||||
} else {
|
||||
definition
|
||||
.signal_contract
|
||||
.as_ref()
|
||||
.map(|contract| Arc::new(contract.clone())),
|
||||
.map(|contract| Arc::new(contract.clone()))
|
||||
},
|
||||
emitted_signals: Arc::new(std::sync::Mutex::new(Vec::new())),
|
||||
})
|
||||
};
|
||||
@ -279,6 +313,9 @@ impl SubAgentManager {
|
||||
if let Some(allowed) = config.allowed_tools.as_ref() {
|
||||
effective_names.retain(|name| allowed.iter().any(|allowed| allowed == name));
|
||||
}
|
||||
if caller.execution_origin.is_scheduled() {
|
||||
effective_names.retain(|name| !SCHEDULED_DISABLED_TOOLS.contains(&name.as_str()));
|
||||
}
|
||||
let has_get_skill = effective_names.iter().any(|name| name == "get_skill");
|
||||
let mut names = effective_names;
|
||||
names.retain(|name| name != "get_skill");
|
||||
@ -310,7 +347,7 @@ impl SubAgentManager {
|
||||
// The signal tool is contract-bound: it exists only when the
|
||||
// definition declares a signal block and the durable Coordinator is
|
||||
// live. If either is missing the run cannot emit signals.
|
||||
if definition.signal_contract.is_some() {
|
||||
if definition.signal_contract.is_some() && !caller.execution_origin.is_scheduled() {
|
||||
match self.coordinator() {
|
||||
Some(coordinator) => {
|
||||
let contract = definition.signal_contract.clone().unwrap();
|
||||
@ -343,8 +380,16 @@ impl SubAgentManager {
|
||||
agent_id: Some(target.to_string()),
|
||||
definition_hash: Some(definition.definition_hash.clone()),
|
||||
llm_profile: definition.llm_profile.clone(),
|
||||
signal_contract: definition.signal_contract.clone(),
|
||||
tool_context: ToolExecutionContext::for_session(format!("agent-run:{task_id}"))
|
||||
signal_contract: (!caller.execution_origin.is_scheduled())
|
||||
.then(|| definition.signal_contract.clone())
|
||||
.flatten(),
|
||||
tool_context: ToolExecutionContext::for_session(
|
||||
if caller.execution_origin.is_scheduled() {
|
||||
root_session_id
|
||||
} else {
|
||||
format!("agent-run:{task_id}")
|
||||
},
|
||||
)
|
||||
.with_turn_id(
|
||||
caller
|
||||
.turn_id
|
||||
@ -353,7 +398,73 @@ impl SubAgentManager {
|
||||
)
|
||||
.with_agent(execution)
|
||||
.with_cancellation(cancellation)
|
||||
.with_execution_gate(self.execution_gate.clone()),
|
||||
.with_execution_gate(self.execution_gate.clone())
|
||||
.with_execution_origin(caller.execution_origin),
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn resolve_scheduled_root(
|
||||
&self,
|
||||
caller: &ToolExecutionContext,
|
||||
task_id: &str,
|
||||
timeout_secs: u64,
|
||||
sink: Arc<crate::tools::ScheduledCompletionSink>,
|
||||
) -> Result<ResolvedAgentRun, SubAgentError> {
|
||||
if !caller.execution_origin.is_scheduled() {
|
||||
return Err(SubAgentError::Other(
|
||||
"scheduled root resolution requires Scheduled execution origin".to_string(),
|
||||
));
|
||||
}
|
||||
let root_session_id = caller.session_id.clone().ok_or_else(|| {
|
||||
SubAgentError::Other("scheduled root requires an execution scope".to_string())
|
||||
})?;
|
||||
let run_id = task_id.to_string();
|
||||
let cancellation = caller.cancellation.child_token();
|
||||
let execution = Arc::new(crate::agent::AgentExecutionContext {
|
||||
root_session_id: root_session_id.clone(),
|
||||
root_turn_id: None,
|
||||
run_id: run_id.clone(),
|
||||
execution_id: run_id,
|
||||
parent_run_id: None,
|
||||
caller_agent_id: "SCHEDULER".to_string(),
|
||||
current_agent_id: "ROOT".to_string(),
|
||||
ancestry: vec!["ROOT".to_string()],
|
||||
depth: 1,
|
||||
plan_item_id: None,
|
||||
cancellation: cancellation.clone(),
|
||||
budget: crate::agent::AgentBudget {
|
||||
remaining_runs: self.catalog.max_runs_per_tree().saturating_sub(1),
|
||||
remaining_depth: self.catalog.max_tree_depth().saturating_sub(1),
|
||||
},
|
||||
tree_runs: Arc::new(std::sync::atomic::AtomicUsize::new(1)),
|
||||
signal_contract: None,
|
||||
emitted_signals: Arc::new(std::sync::Mutex::new(Vec::new())),
|
||||
});
|
||||
let tools = self.full_tools.without(SCHEDULED_DISABLED_TOOLS);
|
||||
tools.register(crate::tools::CompleteScheduledRunTool::new());
|
||||
let skills_prompt = self
|
||||
.skills_loader
|
||||
.as_ref()
|
||||
.map(|loader| loader.build_skills_prompt())
|
||||
.filter(|prompt| !prompt.is_empty());
|
||||
Ok(ResolvedAgentRun {
|
||||
provider_config: Arc::new(self.provider_config.clone()),
|
||||
tools,
|
||||
timeout_secs,
|
||||
max_iterations: self.provider_config.max_tool_iterations,
|
||||
max_result_chars: 16_384,
|
||||
role_prompt: None,
|
||||
skills_prompt,
|
||||
tool_context: ToolExecutionContext::for_session(root_session_id)
|
||||
.with_agent(execution)
|
||||
.with_cancellation(cancellation)
|
||||
.with_execution_gate(self.execution_gate.clone())
|
||||
.with_execution_origin(caller.execution_origin)
|
||||
.with_scheduled_completion(sink),
|
||||
agent_id: None,
|
||||
definition_hash: None,
|
||||
llm_profile: None,
|
||||
signal_contract: None,
|
||||
})
|
||||
}
|
||||
|
||||
@ -664,6 +775,25 @@ mod tests {
|
||||
assert!(!crate::tools::Tool::runtime_injected(&reload));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scheduled_runs_remove_direct_and_interactive_control_tools() {
|
||||
for name in [
|
||||
"send_message",
|
||||
"cron_add",
|
||||
"cron_runs",
|
||||
"reload_config",
|
||||
"agent_task",
|
||||
"chat_manager",
|
||||
"todo",
|
||||
"emit_signal",
|
||||
] {
|
||||
assert!(SCHEDULED_DISABLED_TOOLS.contains(&name));
|
||||
}
|
||||
assert!(!SCHEDULED_DISABLED_TOOLS.contains(&"bash"));
|
||||
assert!(!SCHEDULED_DISABLED_TOOLS.contains(&"delegate"));
|
||||
assert!(!SCHEDULED_DISABLED_TOOLS.contains(&"complete_scheduled_run"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_agent_rejects_missing_target() {
|
||||
let manager = manager();
|
||||
|
||||
@ -5,7 +5,7 @@ use std::time::Duration;
|
||||
|
||||
use tokio::sync::mpsc;
|
||||
|
||||
use crate::bus::{MessageBus, OutboundMessage};
|
||||
use crate::bus::{DeliveryReceipt, MessageBus, OutboundMessage};
|
||||
use crate::channels::ChannelManager;
|
||||
use crate::channels::base::{Channel, ChannelError};
|
||||
use crate::delivery::ConversationWriteLocks;
|
||||
@ -64,12 +64,14 @@ impl OutboundDispatcher {
|
||||
if sender.as_ref().is_none_or(mpsc::Sender::is_closed) {
|
||||
let Some(channel) = self.channel_manager.get_channel(&msg.channel).await else {
|
||||
tracing::warn!(channel = %msg.channel, "No channel found for message");
|
||||
msg.complete_delivery(Err(format!("channel not found: {}", msg.channel)));
|
||||
msg.complete_delivery(DeliveryReceipt::PermanentFailure {
|
||||
summary: format!("channel not found: {}", msg.channel),
|
||||
});
|
||||
continue;
|
||||
};
|
||||
let (new_sender, receiver) = mpsc::channel(LANE_CAPACITY);
|
||||
if !self.spawn_lane(channel, receiver, msg.channel.clone(), msg.chat_id.clone()) {
|
||||
msg.complete_delivery(Err("dispatcher is shutting down".to_string()));
|
||||
msg.complete_delivery(DeliveryReceipt::DispatcherClosed);
|
||||
continue;
|
||||
}
|
||||
lanes.insert(lane_key.clone(), new_sender.clone());
|
||||
@ -89,7 +91,9 @@ impl OutboundDispatcher {
|
||||
capacity = LANE_CAPACITY,
|
||||
"Outbound lane full; rejecting message instead of blocking other destinations"
|
||||
);
|
||||
msg.complete_delivery(Err("outbound lane is full".to_string()));
|
||||
msg.complete_delivery(DeliveryReceipt::TransientFailure {
|
||||
summary: "outbound lane is full".to_string(),
|
||||
});
|
||||
}
|
||||
Err(mpsc::error::TrySendError::Closed(msg)) => {
|
||||
// The lane may have expired between the closed check and
|
||||
@ -97,13 +101,15 @@ impl OutboundDispatcher {
|
||||
lanes.remove(&lane_key);
|
||||
let Some(channel) = self.channel_manager.get_channel(&msg.channel).await else {
|
||||
tracing::warn!(channel = %msg.channel, "No channel found for message");
|
||||
msg.complete_delivery(Err(format!("channel not found: {}", msg.channel)));
|
||||
msg.complete_delivery(DeliveryReceipt::PermanentFailure {
|
||||
summary: format!("channel not found: {}", msg.channel),
|
||||
});
|
||||
continue;
|
||||
};
|
||||
let (new_sender, receiver) = mpsc::channel(LANE_CAPACITY);
|
||||
if !self.spawn_lane(channel, receiver, msg.channel.clone(), msg.chat_id.clone())
|
||||
{
|
||||
msg.complete_delivery(Err("dispatcher is shutting down".to_string()));
|
||||
msg.complete_delivery(DeliveryReceipt::DispatcherClosed);
|
||||
continue;
|
||||
}
|
||||
match new_sender.try_send(msg) {
|
||||
@ -111,9 +117,9 @@ impl OutboundDispatcher {
|
||||
lanes.insert(lane_key, new_sender);
|
||||
}
|
||||
Err(error) => {
|
||||
error.into_inner().complete_delivery(Err(
|
||||
"outbound lane could not be restarted during shutdown".to_string(),
|
||||
));
|
||||
error
|
||||
.into_inner()
|
||||
.complete_delivery(DeliveryReceipt::DispatcherClosed);
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -143,15 +149,15 @@ impl OutboundDispatcher {
|
||||
Ok(None) | Err(_) => break,
|
||||
};
|
||||
let result = Self::send_with_retry(&*channel, &msg, &target_lock).await;
|
||||
if let Err(error) = &result {
|
||||
if result != DeliveryReceipt::Delivered {
|
||||
tracing::error!(
|
||||
channel = %channel_name,
|
||||
chat_id = %chat_id,
|
||||
error = %error,
|
||||
result = ?result,
|
||||
"Failed to send message after retries"
|
||||
);
|
||||
}
|
||||
msg.complete_delivery(result.map_err(|error| error.to_string()));
|
||||
msg.complete_delivery(result);
|
||||
}
|
||||
},
|
||||
)
|
||||
@ -161,26 +167,28 @@ impl OutboundDispatcher {
|
||||
channel: &dyn Channel,
|
||||
msg: &OutboundMessage,
|
||||
target_lock: &tokio::sync::Mutex<()>,
|
||||
) -> Result<(), ChannelError> {
|
||||
) -> DeliveryReceipt {
|
||||
let _guard = target_lock.lock().await;
|
||||
const DELAYS: &[u64] = &[1, 2, 4];
|
||||
|
||||
for (attempt, &delay) in DELAYS.iter().enumerate() {
|
||||
let result = tokio::time::timeout(SEND_TIMEOUT, channel.send(msg.clone())).await;
|
||||
match result {
|
||||
Ok(Ok(())) => return Ok(()),
|
||||
Ok(Ok(())) => return DeliveryReceipt::Delivered,
|
||||
Ok(Err(error)) if attempt < DELAYS.len() - 1 && error.is_transient() => {
|
||||
tracing::warn!(attempt = attempt + 1, delay, error = %error, "Send failed, retrying");
|
||||
tracing::warn!(
|
||||
attempt = attempt + 1,
|
||||
delay,
|
||||
error_class = channel_error_class(&error),
|
||||
"Send failed, retrying"
|
||||
);
|
||||
}
|
||||
Ok(Err(error)) => return Err(error),
|
||||
Ok(Err(error)) => return receipt_from_channel_error(error),
|
||||
Err(_) if attempt < DELAYS.len() - 1 => {
|
||||
tracing::warn!(attempt = attempt + 1, delay, "Send timed out, retrying");
|
||||
}
|
||||
Err(_) => {
|
||||
return Err(ChannelError::Other(format!(
|
||||
"send timed out after {} seconds",
|
||||
SEND_TIMEOUT.as_secs()
|
||||
)));
|
||||
return DeliveryReceipt::TimedOut;
|
||||
}
|
||||
}
|
||||
tokio::time::sleep(Duration::from_secs(delay)).await;
|
||||
@ -189,6 +197,36 @@ impl OutboundDispatcher {
|
||||
}
|
||||
}
|
||||
|
||||
fn channel_error_class(error: &ChannelError) -> &'static str {
|
||||
match error {
|
||||
ChannelError::ConnectionError(_) => "connection",
|
||||
ChannelError::SendError(_) => "send",
|
||||
ChannelError::BusError(_) => "bus",
|
||||
ChannelError::ConfigError(_) => "config",
|
||||
ChannelError::Other(_) => "other",
|
||||
}
|
||||
}
|
||||
|
||||
fn receipt_from_channel_error(error: ChannelError) -> DeliveryReceipt {
|
||||
match error {
|
||||
ChannelError::ConnectionError(_) => DeliveryReceipt::TransientFailure {
|
||||
summary: "channel connection failed after retries".to_string(),
|
||||
},
|
||||
ChannelError::SendError(_) => DeliveryReceipt::TransientFailure {
|
||||
summary: "channel send failed after retries".to_string(),
|
||||
},
|
||||
ChannelError::BusError(_) => DeliveryReceipt::TransientFailure {
|
||||
summary: "channel bus was unavailable".to_string(),
|
||||
},
|
||||
ChannelError::ConfigError(_) => DeliveryReceipt::PermanentFailure {
|
||||
summary: "channel configuration rejected delivery".to_string(),
|
||||
},
|
||||
ChannelError::Other(_) => DeliveryReceipt::PermanentFailure {
|
||||
summary: "channel rejected delivery".to_string(),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/// Decrements the active-lane counter exactly once when a lane task ends,
|
||||
/// whether it exits normally, is cancelled, or is aborted.
|
||||
struct LaneGuard {
|
||||
@ -348,7 +386,7 @@ mod tests {
|
||||
message.channel = "missing".to_string();
|
||||
let error = bus.deliver_outbound(message).await.unwrap_err();
|
||||
|
||||
assert!(matches!(error, crate::bus::BusError::DeliveryFailed(_)));
|
||||
assert!(matches!(error, crate::bus::BusError::DeliveryPermanent(_)));
|
||||
task.abort();
|
||||
supervisor.shutdown(Duration::from_secs(1)).await;
|
||||
}
|
||||
@ -435,18 +473,40 @@ mod tests {
|
||||
};
|
||||
|
||||
let target_lock = tokio::sync::Mutex::new(());
|
||||
let error = OutboundDispatcher::send_with_retry(
|
||||
let receipt = OutboundDispatcher::send_with_retry(
|
||||
&channel,
|
||||
&outbound("invalid", "message"),
|
||||
&target_lock,
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
.await;
|
||||
|
||||
assert!(matches!(error, ChannelError::Other(_)));
|
||||
assert!(matches!(receipt, DeliveryReceipt::PermanentFailure { .. }));
|
||||
assert_eq!(channel.attempts.load(Ordering::SeqCst), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn channel_errors_map_to_typed_sanitized_receipts() {
|
||||
for error in [
|
||||
ChannelError::ConnectionError("https://secret.example/?token=x".to_string()),
|
||||
ChannelError::SendError("private response body".to_string()),
|
||||
ChannelError::BusError("private queue detail".to_string()),
|
||||
] {
|
||||
let receipt = receipt_from_channel_error(error);
|
||||
assert!(matches!(receipt, DeliveryReceipt::TransientFailure { .. }));
|
||||
assert!(!format!("{receipt:?}").contains("private"));
|
||||
assert!(!format!("{receipt:?}").contains("secret"));
|
||||
}
|
||||
for error in [
|
||||
ChannelError::ConfigError("api_key=x".to_string()),
|
||||
ChannelError::Other("private platform payload".to_string()),
|
||||
] {
|
||||
let receipt = receipt_from_channel_error(error);
|
||||
assert!(matches!(receipt, DeliveryReceipt::PermanentFailure { .. }));
|
||||
assert!(!format!("{receipt:?}").contains("private"));
|
||||
assert!(!format!("{receipt:?}").contains("api_key"));
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn shared_write_lock_orders_dispatcher_with_live_turn_writes() {
|
||||
let channel = RecordingChannel {
|
||||
@ -468,7 +528,7 @@ mod tests {
|
||||
assert!(channel.sent.lock().await.is_empty());
|
||||
|
||||
drop(live_write);
|
||||
send.await.unwrap();
|
||||
assert_eq!(send.await, DeliveryReceipt::Delivered);
|
||||
assert_eq!(channel.sent.lock().await.as_slice(), &["after-live-update"]);
|
||||
}
|
||||
}
|
||||
|
||||
@ -509,17 +509,26 @@ pub struct OutboundMessage {
|
||||
pub reply_to: Option<String>,
|
||||
pub media: Vec<MediaItem>,
|
||||
pub metadata: HashMap<String, String>,
|
||||
pub(crate) delivery: Option<tokio::sync::watch::Sender<Option<Result<(), String>>>>,
|
||||
pub(crate) delivery: Option<tokio::sync::watch::Sender<Option<DeliveryReceipt>>>,
|
||||
}
|
||||
|
||||
impl OutboundMessage {
|
||||
pub(crate) fn complete_delivery(&self, result: Result<(), String>) {
|
||||
pub(crate) fn complete_delivery(&self, result: DeliveryReceipt) {
|
||||
if let Some(delivery) = &self.delivery {
|
||||
delivery.send_replace(Some(result));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum DeliveryReceipt {
|
||||
Delivered,
|
||||
TransientFailure { summary: String },
|
||||
PermanentFailure { summary: String },
|
||||
TimedOut,
|
||||
DispatcherClosed,
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// ControlMessage - Message for control channel (session management)
|
||||
// Uses SessionCommand from session module
|
||||
|
||||
@ -4,8 +4,8 @@ pub mod message;
|
||||
pub use dispatcher::OutboundDispatcher;
|
||||
pub use message::{
|
||||
ChannelContext, ChatMessage, ClientVisibility, CommittedMessage, CommittedTurnDelta,
|
||||
CompletionStatus, ContentBlock, ControlMessage, InboundMessage, MediaItem, MediaRef,
|
||||
MessageSource, OutboundMessage, ProviderReasoningState, SourceKind, TurnOrigin,
|
||||
CompletionStatus, ContentBlock, ControlMessage, DeliveryReceipt, InboundMessage, MediaItem,
|
||||
MediaRef, MessageSource, OutboundMessage, ProviderReasoningState, SourceKind, TurnOrigin,
|
||||
};
|
||||
|
||||
use std::sync::Arc;
|
||||
@ -80,7 +80,17 @@ impl MessageBus {
|
||||
loop {
|
||||
delivery_rx.changed().await.map_err(|_| BusError::Closed)?;
|
||||
if let Some(result) = delivery_rx.borrow().clone() {
|
||||
return result.map_err(BusError::DeliveryFailed);
|
||||
return match result {
|
||||
DeliveryReceipt::Delivered => Ok(()),
|
||||
DeliveryReceipt::TransientFailure { summary } => {
|
||||
Err(BusError::DeliveryTransient(summary))
|
||||
}
|
||||
DeliveryReceipt::PermanentFailure { summary } => {
|
||||
Err(BusError::DeliveryPermanent(summary))
|
||||
}
|
||||
DeliveryReceipt::TimedOut => Err(BusError::DeliveryTimedOut),
|
||||
DeliveryReceipt::DispatcherClosed => Err(BusError::Closed),
|
||||
};
|
||||
}
|
||||
}
|
||||
})
|
||||
@ -138,7 +148,8 @@ pub struct QueueDepths {
|
||||
#[derive(Debug)]
|
||||
pub enum BusError {
|
||||
Closed,
|
||||
DeliveryFailed(String),
|
||||
DeliveryTransient(String),
|
||||
DeliveryPermanent(String),
|
||||
DeliveryTimedOut,
|
||||
}
|
||||
|
||||
@ -146,7 +157,12 @@ impl std::fmt::Display for BusError {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
match self {
|
||||
BusError::Closed => write!(f, "Bus channel closed"),
|
||||
BusError::DeliveryFailed(error) => write!(f, "Outbound delivery failed: {error}"),
|
||||
BusError::DeliveryTransient(error) => {
|
||||
write!(f, "Transient outbound delivery failure: {error}")
|
||||
}
|
||||
BusError::DeliveryPermanent(error) => {
|
||||
write!(f, "Permanent outbound delivery failure: {error}")
|
||||
}
|
||||
BusError::DeliveryTimedOut => write!(f, "Outbound delivery confirmation timed out"),
|
||||
}
|
||||
}
|
||||
|
||||
@ -812,8 +812,12 @@ fn scheduler_snapshot(jobs: &[crate::storage::ScheduledJob]) -> Value {
|
||||
.iter()
|
||||
.filter(|job| {
|
||||
matches!(
|
||||
job.last_status.as_deref(),
|
||||
Some("error" | "timeout" | "delivery_error")
|
||||
job.last_outcome,
|
||||
Some(
|
||||
crate::storage::ScheduledOutcomeKind::Failed
|
||||
| crate::storage::ScheduledOutcomeKind::Refused
|
||||
| crate::storage::ScheduledOutcomeKind::Unknown
|
||||
)
|
||||
)
|
||||
})
|
||||
.count();
|
||||
@ -1465,6 +1469,26 @@ pub async fn get_job_runs(
|
||||
.list_scheduled_job_runs(&id, limit)
|
||||
.await
|
||||
.map_err(ApiError::internal)?;
|
||||
let runs = runs
|
||||
.into_iter()
|
||||
.map(|run| {
|
||||
json!({
|
||||
"id": run.id,
|
||||
"job_id": run.job_id,
|
||||
"scheduled_for": run.scheduled_for,
|
||||
"started_at": run.started_at,
|
||||
"finished_at": run.finished_at,
|
||||
"status": run.status,
|
||||
"outcome": run.outcome,
|
||||
"message": run.message,
|
||||
"diagnostic": run.diagnostic,
|
||||
"duration_ms": run.duration_ms,
|
||||
"delivery_status": run.delivery_status,
|
||||
"delivery_attempts": run.delivery_attempts,
|
||||
"delivery_error": run.delivery_error,
|
||||
})
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
Ok(Json(json!({ "runs": runs })))
|
||||
}
|
||||
|
||||
@ -1564,37 +1588,57 @@ mod tests {
|
||||
id: &str,
|
||||
enabled: bool,
|
||||
next_run_at: i64,
|
||||
last_status: Option<&str>,
|
||||
last_outcome: Option<crate::storage::ScheduledOutcomeKind>,
|
||||
) -> crate::storage::ScheduledJob {
|
||||
crate::storage::ScheduledJob {
|
||||
id: id.to_string(),
|
||||
name: id.to_string(),
|
||||
schedule: crate::scheduler::Schedule::Every { every_ms: 60_000 },
|
||||
prompt: String::new(),
|
||||
agent_id: None,
|
||||
channel: "cli_chat".to_string(),
|
||||
chat_id: "test".to_string(),
|
||||
model: None,
|
||||
job_kind: crate::storage::JobKind::Task,
|
||||
delivery_policy: crate::storage::DeliveryPolicy::Never,
|
||||
enabled,
|
||||
delete_after_run: false,
|
||||
next_run_at,
|
||||
last_run_at: None,
|
||||
last_status: last_status.map(str::to_string),
|
||||
last_error: None,
|
||||
last_outcome,
|
||||
created_at: 0,
|
||||
updated_at: 0,
|
||||
locked_at: None,
|
||||
lock_owner: None,
|
||||
lease_until: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scheduler_snapshot_classifies_failures_and_next_enabled_run() {
|
||||
let jobs = vec![
|
||||
scheduled_job("healthy", true, 300, Some("ok")),
|
||||
scheduled_job("error", true, 200, Some("error")),
|
||||
scheduled_job("timeout", false, 100, Some("timeout")),
|
||||
scheduled_job("delivery", true, 400, Some("delivery_error")),
|
||||
scheduled_job("other", false, 50, Some("cancelled")),
|
||||
scheduled_job(
|
||||
"healthy",
|
||||
true,
|
||||
300,
|
||||
Some(crate::storage::ScheduledOutcomeKind::Ok),
|
||||
),
|
||||
scheduled_job(
|
||||
"error",
|
||||
true,
|
||||
200,
|
||||
Some(crate::storage::ScheduledOutcomeKind::Failed),
|
||||
),
|
||||
scheduled_job(
|
||||
"refused",
|
||||
false,
|
||||
100,
|
||||
Some(crate::storage::ScheduledOutcomeKind::Refused),
|
||||
),
|
||||
scheduled_job(
|
||||
"unknown",
|
||||
true,
|
||||
400,
|
||||
Some(crate::storage::ScheduledOutcomeKind::Unknown),
|
||||
),
|
||||
scheduled_job("other", false, 50, None),
|
||||
];
|
||||
|
||||
assert_eq!(
|
||||
|
||||
@ -178,6 +178,7 @@ impl GatewayState {
|
||||
.init(&config, workspace_path.clone())
|
||||
.await
|
||||
.map_err(|e| format!("Failed to init channels: {}", e))?;
|
||||
let available_channels = channel_manager.list_channel_names().await;
|
||||
let turn_delivery = TurnDeliveryService::new(
|
||||
delivery_coordinator.clone(),
|
||||
channel_manager.clone(),
|
||||
@ -189,7 +190,10 @@ impl GatewayState {
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let health = Arc::new(crate::health::HealthService::new(config.clone()));
|
||||
let health = Arc::new(
|
||||
crate::health::HealthService::new(config.clone())
|
||||
.with_scheduler_runtime(storage.clone(), available_channels.clone()),
|
||||
);
|
||||
let provider_profiles: std::collections::HashMap<String, _> = config
|
||||
.agents
|
||||
.keys()
|
||||
@ -245,9 +249,9 @@ impl GatewayState {
|
||||
let session_manager = Arc::new(session_manager);
|
||||
session_manager.bind_inbox_wake();
|
||||
let agent_catalog = session_manager.agent_catalog();
|
||||
health.bind_agent_catalog(agent_catalog.clone());
|
||||
|
||||
// Register send_message tool with available channel names
|
||||
let available_channels = channel_manager.list_channel_names().await;
|
||||
let valid_channels = available_channels.clone();
|
||||
session_manager.register_outbound_tool(available_channels);
|
||||
|
||||
@ -277,11 +281,15 @@ impl GatewayState {
|
||||
.tools()
|
||||
.register(crate::tools::cron::CronAddTool::new(
|
||||
storage.clone(),
|
||||
valid_channels,
|
||||
valid_channels.clone(),
|
||||
agent_catalog.clone(),
|
||||
));
|
||||
session_manager
|
||||
.tools()
|
||||
.register(crate::tools::cron::CronListTool::new(storage.clone()));
|
||||
session_manager
|
||||
.tools()
|
||||
.register(crate::tools::cron::CronRunsTool::new(storage.clone()));
|
||||
session_manager
|
||||
.tools()
|
||||
.register(crate::tools::cron::CronRemoveTool::new(storage.clone()));
|
||||
@ -293,7 +301,11 @@ impl GatewayState {
|
||||
.register(crate::tools::cron::CronDisableTool::new(storage.clone()));
|
||||
session_manager
|
||||
.tools()
|
||||
.register(crate::tools::cron::CronUpdateTool::new(storage.clone()));
|
||||
.register(crate::tools::cron::CronUpdateTool::new(
|
||||
storage.clone(),
|
||||
valid_channels,
|
||||
agent_catalog.clone(),
|
||||
));
|
||||
tracing::info!("Cron tools registered");
|
||||
}
|
||||
|
||||
@ -332,7 +344,25 @@ impl GatewayState {
|
||||
}
|
||||
|
||||
/// Start the message processing loops
|
||||
pub async fn start_message_processing(&self) {
|
||||
pub async fn start_message_processing(&self) -> Result<(), String> {
|
||||
match self
|
||||
.storage
|
||||
.recover_scheduled_runs(chrono::Utc::now().timestamp_millis())
|
||||
.await
|
||||
{
|
||||
Ok(recovered) if recovered > 0 => {
|
||||
tracing::warn!(
|
||||
recovered,
|
||||
"Scheduled runs recovered as unknown on activation"
|
||||
);
|
||||
}
|
||||
Ok(_) => {}
|
||||
Err(error) => {
|
||||
return Err(format!(
|
||||
"Scheduled run recovery failed on activation: {error}"
|
||||
));
|
||||
}
|
||||
}
|
||||
// Recover durable Agent state for this runtime generation: interrupt
|
||||
// runs of older generations, expire stale inbox leases and reconcile
|
||||
// capacity rows. Runs never recover while the generation is still a candidate.
|
||||
@ -351,7 +381,9 @@ impl GatewayState {
|
||||
}
|
||||
}
|
||||
Err(error) => {
|
||||
tracing::error!(error = %error, "Agent state recovery failed on activation");
|
||||
return Err(format!(
|
||||
"Agent state recovery failed on activation: {error}"
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -461,6 +493,7 @@ impl GatewayState {
|
||||
});
|
||||
tracing::info!("Scheduler background task spawned");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
@ -525,7 +558,7 @@ pub async fn run(
|
||||
reload_controller.set_failed(current_generation, error.to_string());
|
||||
return Err(error.into());
|
||||
}
|
||||
state.start_message_processing().await;
|
||||
state.start_message_processing().await?;
|
||||
reload_controller.set_phase(current_generation, reload::ReloadPhase::Active);
|
||||
let app = build_router(state.clone());
|
||||
let generation_listener = TcpListener::from_std(listener.try_clone()?)?;
|
||||
|
||||
@ -459,7 +459,7 @@ mod tests {
|
||||
Some("command")
|
||||
);
|
||||
assert!(!publish_task.is_finished());
|
||||
output.complete_delivery(Ok(()));
|
||||
output.complete_delivery(crate::bus::DeliveryReceipt::Delivered);
|
||||
publish_task.await.unwrap();
|
||||
}
|
||||
|
||||
|
||||
221
src/health.rs
221
src/health.rs
@ -1,6 +1,7 @@
|
||||
use std::collections::HashSet;
|
||||
use std::path::Path;
|
||||
use std::process::{Output, Stdio};
|
||||
use std::sync::{Arc, RwLock};
|
||||
use std::time::Duration;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
@ -117,11 +118,41 @@ impl HealthReport {
|
||||
#[derive(Clone)]
|
||||
pub struct HealthService {
|
||||
config: Config,
|
||||
scheduler_runtime: Arc<RwLock<Option<SchedulerHealthRuntime>>>,
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
struct SchedulerHealthRuntime {
|
||||
storage: Arc<crate::storage::Storage>,
|
||||
channels: HashSet<String>,
|
||||
catalog: Option<Arc<crate::agent::AgentCatalog>>,
|
||||
}
|
||||
|
||||
impl HealthService {
|
||||
pub fn new(config: Config) -> Self {
|
||||
Self { config }
|
||||
Self {
|
||||
config,
|
||||
scheduler_runtime: Arc::new(RwLock::new(None)),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn with_scheduler_runtime(
|
||||
self,
|
||||
storage: Arc<crate::storage::Storage>,
|
||||
channels: Vec<String>,
|
||||
) -> Self {
|
||||
*self.scheduler_runtime.write().unwrap() = Some(SchedulerHealthRuntime {
|
||||
storage,
|
||||
channels: channels.into_iter().collect(),
|
||||
catalog: None,
|
||||
});
|
||||
self
|
||||
}
|
||||
|
||||
pub fn bind_agent_catalog(&self, catalog: Arc<crate::agent::AgentCatalog>) {
|
||||
if let Some(runtime) = self.scheduler_runtime.write().unwrap().as_mut() {
|
||||
runtime.catalog = Some(catalog);
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn check(&self) -> HealthReport {
|
||||
@ -139,9 +170,165 @@ impl HealthService {
|
||||
];
|
||||
checks.extend(self.check_mcp_commands());
|
||||
checks.extend(self.check_browser().await);
|
||||
checks.extend(self.check_scheduler().await);
|
||||
HealthReport::from_checks(checks)
|
||||
}
|
||||
|
||||
async fn check_scheduler(&self) -> Vec<HealthCheck> {
|
||||
let scheduler_config = self.config.gateway.scheduler.clone().unwrap_or_default();
|
||||
if !scheduler_config.enabled {
|
||||
return vec![HealthCheck {
|
||||
name: "scheduled runs".to_string(),
|
||||
category: "configured".to_string(),
|
||||
required: false,
|
||||
status: HealthStatus::Pass,
|
||||
detail: "scheduler disabled; persisted jobs are not executed".to_string(),
|
||||
remediation: None,
|
||||
}];
|
||||
}
|
||||
let runtime = self.scheduler_runtime.read().unwrap().clone();
|
||||
let Some(runtime) = runtime else {
|
||||
return vec![HealthCheck {
|
||||
name: "scheduled runs".to_string(),
|
||||
category: "runtime".to_string(),
|
||||
required: false,
|
||||
status: HealthStatus::Pass,
|
||||
detail: "runtime scheduler state is checked by the running Gateway".to_string(),
|
||||
remediation: None,
|
||||
}];
|
||||
};
|
||||
let jobs = match runtime.storage.list_scheduled_jobs().await {
|
||||
Ok(jobs) => jobs,
|
||||
Err(error) => {
|
||||
return vec![HealthCheck {
|
||||
name: "scheduled run storage".to_string(),
|
||||
category: "runtime".to_string(),
|
||||
required: true,
|
||||
status: HealthStatus::Fail,
|
||||
detail: format!("cannot read scheduled jobs: {error}"),
|
||||
remediation: Some(
|
||||
"Check the configured SQLite database and Gateway logs.".to_string(),
|
||||
),
|
||||
}];
|
||||
}
|
||||
};
|
||||
let now = chrono::Utc::now().timestamp_millis();
|
||||
let stale_after = scheduler_config
|
||||
.poll_interval_secs
|
||||
.max(60)
|
||||
.saturating_mul(10_000)
|
||||
.min(i64::MAX as u64) as i64;
|
||||
let mut bad_references = Vec::new();
|
||||
let mut stale_deliveries = Vec::new();
|
||||
let mut unhealthy_latest = Vec::new();
|
||||
let mut silent_unknown = Vec::new();
|
||||
let mut covered_intervals = Vec::new();
|
||||
let mut invalid_next = Vec::new();
|
||||
|
||||
match runtime
|
||||
.storage
|
||||
.list_stale_scheduled_deliveries(now.saturating_sub(stale_after), 100)
|
||||
.await
|
||||
{
|
||||
Ok(runs) => {
|
||||
stale_deliveries.extend(
|
||||
runs.into_iter()
|
||||
.map(|run| format!("{}#{}", run.job_id, run.id)),
|
||||
);
|
||||
}
|
||||
Err(error) => bad_references.push(format!("delivery backlog unavailable ({error})")),
|
||||
}
|
||||
|
||||
for job in &jobs {
|
||||
if !runtime.channels.contains(&job.channel) {
|
||||
bad_references.push(format!("{}: channel {}", job.id, job.channel));
|
||||
}
|
||||
if let Some(agent_id) = job.agent_id.as_deref()
|
||||
&& runtime
|
||||
.catalog
|
||||
.as_ref()
|
||||
.is_none_or(|catalog| catalog.get(agent_id).is_none())
|
||||
{
|
||||
bad_references.push(format!("{}: agent {}", job.id, agent_id));
|
||||
}
|
||||
if job.enabled && crate::scheduler::next_run_for_schedule(&job.schedule, now).is_none()
|
||||
{
|
||||
invalid_next.push(job.id.clone());
|
||||
}
|
||||
|
||||
let runs = match runtime.storage.list_scheduled_job_runs(&job.id, 20).await {
|
||||
Ok(runs) => runs,
|
||||
Err(error) => {
|
||||
bad_references.push(format!("{}: run history unavailable ({error})", job.id));
|
||||
continue;
|
||||
}
|
||||
};
|
||||
if let Some(latest) = runs.first() {
|
||||
if matches!(
|
||||
latest.status,
|
||||
crate::storage::ScheduledRunStatus::Unknown
|
||||
| crate::storage::ScheduledRunStatus::Failed
|
||||
| crate::storage::ScheduledRunStatus::TimedOut
|
||||
) {
|
||||
unhealthy_latest.push(format!("{}#{}", job.id, latest.id));
|
||||
}
|
||||
if job.delivery_policy == crate::storage::DeliveryPolicy::Never
|
||||
&& latest.outcome == Some(crate::storage::ScheduledOutcomeKind::Unknown)
|
||||
{
|
||||
silent_unknown.push(format!("{}#{}", job.id, latest.id));
|
||||
}
|
||||
if let crate::scheduler::Schedule::Every { every_ms } = job.schedule
|
||||
&& i64::try_from(every_ms).ok().is_some_and(|interval| {
|
||||
latest
|
||||
.duration_ms
|
||||
.is_some_and(|duration| duration >= interval)
|
||||
})
|
||||
{
|
||||
covered_intervals.push(format!("{}#{}", job.id, latest.id));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
vec![
|
||||
scheduler_health_check(
|
||||
"scheduled references",
|
||||
bad_references,
|
||||
"all Agent and channel references are available",
|
||||
"Update or disable jobs that reference missing Agents or channels.",
|
||||
),
|
||||
scheduler_health_check(
|
||||
"scheduled delivery backlog",
|
||||
stale_deliveries,
|
||||
"no stale pending or delivering notifications",
|
||||
"Inspect cron_runs and the target channel configuration.",
|
||||
),
|
||||
scheduler_health_check(
|
||||
"scheduled latest outcomes",
|
||||
unhealthy_latest,
|
||||
"no latest run is failed, timed out, or unknown",
|
||||
"Inspect cron_runs for the diagnostic and assess external side effects before retrying.",
|
||||
),
|
||||
scheduler_health_check(
|
||||
"silent unknown scheduled runs",
|
||||
silent_unknown,
|
||||
"no never-delivery job has an unknown latest outcome",
|
||||
"Inspect the run manually; delivery_policy=never prevents automatic notification.",
|
||||
),
|
||||
scheduler_health_check(
|
||||
"scheduled execution intervals",
|
||||
covered_intervals,
|
||||
"recent execution durations fit their Every intervals",
|
||||
"Increase the interval or split long-running jobs.",
|
||||
),
|
||||
scheduler_health_check(
|
||||
"scheduled next runs",
|
||||
invalid_next,
|
||||
"all enabled schedules can compute a next run",
|
||||
"Correct the schedule expression or disable the job.",
|
||||
),
|
||||
]
|
||||
}
|
||||
|
||||
fn check_mcp_commands(&self) -> Vec<HealthCheck> {
|
||||
let mut seen = HashSet::new();
|
||||
let mut checks = Vec::new();
|
||||
@ -365,6 +552,38 @@ impl HealthService {
|
||||
}
|
||||
}
|
||||
|
||||
fn scheduler_health_check(
|
||||
name: &str,
|
||||
findings: Vec<String>,
|
||||
healthy_detail: &str,
|
||||
remediation: &str,
|
||||
) -> HealthCheck {
|
||||
if findings.is_empty() {
|
||||
HealthCheck {
|
||||
name: name.to_string(),
|
||||
category: "runtime".to_string(),
|
||||
required: false,
|
||||
status: HealthStatus::Pass,
|
||||
detail: healthy_detail.to_string(),
|
||||
remediation: None,
|
||||
}
|
||||
} else {
|
||||
let total = findings.len();
|
||||
let mut sample = findings.into_iter().take(5).collect::<Vec<_>>().join(", ");
|
||||
if total > 5 {
|
||||
sample.push_str(&format!(", and {} more", total - 5));
|
||||
}
|
||||
HealthCheck {
|
||||
name: name.to_string(),
|
||||
category: "runtime".to_string(),
|
||||
required: false,
|
||||
status: HealthStatus::Warning,
|
||||
detail: format!("{total} finding(s): {sample}"),
|
||||
remediation: Some(remediation.to_string()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn check_configuration_recovery(config: &Config) -> HealthCheck {
|
||||
if config.diagnostics.is_empty() {
|
||||
return HealthCheck {
|
||||
|
||||
@ -3,81 +3,41 @@ pub mod types;
|
||||
use std::sync::Arc;
|
||||
use std::time::Instant;
|
||||
|
||||
use futures_util::stream::{self, StreamExt};
|
||||
use tokio::task::JoinSet;
|
||||
use tokio::time;
|
||||
|
||||
use crate::config::SchedulerConfig;
|
||||
use crate::session::SessionManager;
|
||||
use crate::session::session::HandleResult;
|
||||
use crate::storage::ScheduledJob;
|
||||
use crate::storage::Storage;
|
||||
use crate::storage::{DeliveryPolicy, JobKind, JobRun};
|
||||
use crate::session::{ScheduledDeliveryError, SessionManager};
|
||||
use crate::storage::{
|
||||
ClaimedScheduledRun, JobRun, ScheduledOutcomeKind, ScheduledRunCompletion, ScheduledRunStatus,
|
||||
Storage,
|
||||
};
|
||||
|
||||
pub use types::Schedule;
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
enum ScheduledDisposition {
|
||||
Content(String),
|
||||
Quiet(String),
|
||||
ReportedFailure(String),
|
||||
Refused(String),
|
||||
}
|
||||
|
||||
fn parse_scheduled_disposition(output: &str) -> ScheduledDisposition {
|
||||
let trimmed = output.trim();
|
||||
if trimmed.eq_ignore_ascii_case("NO_REPLY") {
|
||||
return ScheduledDisposition::Quiet(String::new());
|
||||
}
|
||||
let upper = trimmed.to_ascii_uppercase();
|
||||
for (prefix, kind) in [
|
||||
("NO_REPLY[INFO]", "info"),
|
||||
("NO_REPLY[FAIL]", "fail"),
|
||||
("NO_REPLY[REFUSE]", "refuse"),
|
||||
] {
|
||||
if upper.starts_with(prefix) {
|
||||
let suffix = &trimmed[prefix.len()..];
|
||||
if !suffix.is_empty() && !suffix.trim_start().starts_with(':') {
|
||||
continue;
|
||||
}
|
||||
let reason = suffix.trim().trim_start_matches(':').trim().to_string();
|
||||
return match kind {
|
||||
"info" => ScheduledDisposition::Quiet(reason),
|
||||
"fail" => ScheduledDisposition::ReportedFailure(reason),
|
||||
_ => ScheduledDisposition::Refused(reason),
|
||||
};
|
||||
}
|
||||
}
|
||||
if trimmed.is_empty() {
|
||||
ScheduledDisposition::ReportedFailure("scheduled agent returned empty output".into())
|
||||
} else {
|
||||
ScheduledDisposition::Content(trimmed.to_string())
|
||||
}
|
||||
}
|
||||
|
||||
/// Compute the next execution time (Unix ms) for a schedule, given `from` (Unix ms).
|
||||
/// Returns `None` if no next time can be determined (e.g., invalid cron expression).
|
||||
/// Returns `None` if no next time can be determined (e.g. an invalid cron expression).
|
||||
pub fn next_run_for_schedule(schedule: &Schedule, from: i64) -> Option<i64> {
|
||||
use chrono::{TimeZone, Utc};
|
||||
use std::str::FromStr;
|
||||
|
||||
match schedule {
|
||||
Schedule::At { at } => Some(*at),
|
||||
Schedule::Every { every_ms } => Some(from + *every_ms as i64),
|
||||
Schedule::Every { every_ms } => Some(from.saturating_add(i64::try_from(*every_ms).ok()?)),
|
||||
Schedule::Cron { expr, tz } => {
|
||||
let cron_schedule = cron::Schedule::from_str(expr.as_str()).ok()?;
|
||||
let from_secs = from / 1000;
|
||||
let from_nanos = ((from % 1000) * 1_000_000) as u32;
|
||||
let from_dt = Utc.timestamp_opt(from_secs, from_nanos).single()?;
|
||||
|
||||
let next_utc = if let Some(tz_str) = tz {
|
||||
let tz: chrono_tz::Tz = tz_str.parse().ok()?;
|
||||
let from_local = from_dt.with_timezone(&tz);
|
||||
let next_local = cron_schedule.after(&from_local).next()?;
|
||||
next_local.with_timezone(&Utc)
|
||||
cron_schedule
|
||||
.after(&from_dt.with_timezone(&tz))
|
||||
.next()?
|
||||
.with_timezone(&Utc)
|
||||
} else {
|
||||
cron_schedule.after(&from_dt).next()?
|
||||
};
|
||||
|
||||
Some(next_utc.timestamp_millis())
|
||||
}
|
||||
}
|
||||
@ -90,8 +50,6 @@ fn now_ms() -> i64 {
|
||||
.as_millis() as i64
|
||||
}
|
||||
|
||||
/// The scheduler runs as a background tokio task, periodically checking for due jobs
|
||||
/// and executing them via `SessionManager::handle_cron_message`.
|
||||
pub struct Scheduler {
|
||||
storage: Arc<Storage>,
|
||||
session_manager: Arc<SessionManager>,
|
||||
@ -129,15 +87,16 @@ impl Scheduler {
|
||||
}
|
||||
}
|
||||
|
||||
/// Claim due jobs with a durable lease, then execute the claimed batch with
|
||||
/// bounded concurrency.
|
||||
/// Non-blocking event loop. Execution and delivery use separate bounded
|
||||
/// JoinSets so one long Agent run cannot delay other claims or outbox work.
|
||||
pub async fn run(self: Arc<Self>) {
|
||||
let poll_duration = time::Duration::from_secs(self.config.poll_interval_secs.max(1));
|
||||
let mut interval = time::interval(poll_duration);
|
||||
interval.set_missed_tick_behavior(time::MissedTickBehavior::Skip);
|
||||
// Keep accidental configuration values from claiming an unbounded
|
||||
// batch and overwhelming the runtime or SQLite parameter conversion.
|
||||
let max_concurrent = self.config.max_concurrent.clamp(1, 256);
|
||||
let max_delivery = max_concurrent.clamp(1, 16);
|
||||
let mut runs = JoinSet::new();
|
||||
let mut deliveries = JoinSet::new();
|
||||
|
||||
tracing::info!(
|
||||
poll_interval_secs = self.config.poll_interval_secs,
|
||||
@ -147,375 +106,340 @@ impl Scheduler {
|
||||
);
|
||||
|
||||
loop {
|
||||
interval.tick().await;
|
||||
tokio::select! {
|
||||
_ = interval.tick() => {}
|
||||
Some(result) = runs.join_next(), if !runs.is_empty() => {
|
||||
if let Err(error) = result {
|
||||
tracing::error!(error = %error, "scheduled run task panicked");
|
||||
}
|
||||
}
|
||||
Some(result) = deliveries.join_next(), if !deliveries.is_empty() => {
|
||||
if let Err(error) = result {
|
||||
tracing::error!(error = %error, "scheduled delivery task panicked");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
while let Some(result) = runs.try_join_next() {
|
||||
if let Err(error) = result {
|
||||
tracing::error!(error = %error, "scheduled run task panicked");
|
||||
}
|
||||
}
|
||||
while let Some(result) = deliveries.try_join_next() {
|
||||
if let Err(error) = result {
|
||||
tracing::error!(error = %error, "scheduled delivery task panicked");
|
||||
}
|
||||
}
|
||||
if !self.admission.is_accepting() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let now = now_ms();
|
||||
let delivery_slots = max_delivery.saturating_sub(deliveries.len());
|
||||
if delivery_slots > 0 {
|
||||
let lease_until = now.saturating_add(180_000);
|
||||
let delivery_owner = format!("{}:delivery:{}", self.owner, uuid::Uuid::new_v4());
|
||||
match self
|
||||
.storage
|
||||
.claim_scheduled_deliveries(now, lease_until, &delivery_owner, delivery_slots)
|
||||
.await
|
||||
{
|
||||
Ok(claimed) => {
|
||||
for run in claimed {
|
||||
let scheduler = self.clone();
|
||||
deliveries.spawn(async move {
|
||||
scheduler.deliver_claimed_run(run).await;
|
||||
});
|
||||
}
|
||||
}
|
||||
Err(error) => {
|
||||
tracing::error!(error = %error, "scheduler: failed to claim deliveries");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let run_slots = max_concurrent.saturating_sub(runs.len());
|
||||
if run_slots == 0 {
|
||||
continue;
|
||||
}
|
||||
let lease_ms = self
|
||||
.config
|
||||
.execution_timeout_secs
|
||||
.saturating_add(150)
|
||||
.saturating_mul(1000)
|
||||
.min(i64::MAX as u64) as i64;
|
||||
let lease_until = now.saturating_add(lease_ms);
|
||||
let jobs = match self
|
||||
let run_owner = format!("{}:run:{}", self.owner, uuid::Uuid::new_v4());
|
||||
match self
|
||||
.storage
|
||||
.claim_due_scheduled_jobs(now, lease_until, &self.owner, max_concurrent)
|
||||
.claim_due_scheduled_runs(now, now.saturating_add(lease_ms), &run_owner, run_slots)
|
||||
.await
|
||||
{
|
||||
Ok(jobs) => jobs,
|
||||
Err(error) => {
|
||||
tracing::error!(error = %error, "scheduler: failed to claim due jobs");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
|
||||
if jobs.is_empty() {
|
||||
continue;
|
||||
}
|
||||
tracing::info!(count = jobs.len(), "scheduler: claimed due jobs");
|
||||
|
||||
stream::iter(jobs)
|
||||
.for_each_concurrent(max_concurrent, |job| {
|
||||
Ok(claimed) => {
|
||||
for run in claimed {
|
||||
let scheduler = self.clone();
|
||||
async move { scheduler.execute_claimed_job(job).await }
|
||||
})
|
||||
.await;
|
||||
runs.spawn(async move {
|
||||
scheduler.execute_claimed_run(run).await;
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
async fn execute_claimed_job(self: Arc<Self>, job: ScheduledJob) {
|
||||
let Some(_activity) = self.admission.try_enter() else {
|
||||
if let Err(error) = self
|
||||
.storage
|
||||
.release_scheduled_job_lease(&job.id, &self.owner)
|
||||
.await
|
||||
{
|
||||
tracing::error!(job_id = %job.id, error = %error, "scheduler: failed to release job claimed during reload drain");
|
||||
}
|
||||
return;
|
||||
};
|
||||
let start = Instant::now();
|
||||
let started_at = now_ms();
|
||||
tracing::info!(job_id = %job.id, job_name = %job.name, "scheduler: executing claimed job");
|
||||
|
||||
let managed = job.delivery_policy != DeliveryPolicy::Direct;
|
||||
let execution = async {
|
||||
if managed {
|
||||
self.session_manager
|
||||
.handle_managed_scheduled_message(
|
||||
&job.prompt,
|
||||
&job.id,
|
||||
&job.name,
|
||||
job.job_kind == JobKind::Monitor,
|
||||
)
|
||||
.await
|
||||
.map(HandleResult::AgentResponse)
|
||||
} else {
|
||||
self.session_manager
|
||||
.handle_cron_message(
|
||||
&job.channel,
|
||||
&job.chat_id,
|
||||
&job.prompt,
|
||||
&job.id,
|
||||
&job.name,
|
||||
)
|
||||
.await
|
||||
}
|
||||
};
|
||||
let result = time::timeout(
|
||||
time::Duration::from_secs(self.config.execution_timeout_secs.max(1)),
|
||||
execution,
|
||||
)
|
||||
.await;
|
||||
let finished_at = now_ms();
|
||||
let duration_ms = start.elapsed().as_millis() as i64;
|
||||
|
||||
let (mut status, output, error, result_kind, mut delivery_status, mut delivery_error) =
|
||||
match result {
|
||||
Ok(Ok(
|
||||
HandleResult::AgentResponse(output) | HandleResult::CommandOutput(output),
|
||||
)) => {
|
||||
let output = if output.len() > 8000 {
|
||||
format!(
|
||||
"{}...[truncated]",
|
||||
&output[..output.ceil_char_boundary(8000)]
|
||||
)
|
||||
} else {
|
||||
output
|
||||
};
|
||||
if !managed {
|
||||
(
|
||||
"ok".into(),
|
||||
Some(output),
|
||||
None,
|
||||
None,
|
||||
Some("direct".into()),
|
||||
None,
|
||||
)
|
||||
} else {
|
||||
let disposition = parse_scheduled_disposition(&output);
|
||||
let (kind, content, alert) = match &disposition {
|
||||
ScheduledDisposition::Content(value) => {
|
||||
("content", Some(value.as_str()), true)
|
||||
}
|
||||
ScheduledDisposition::Quiet(_) => ("quiet", None, false),
|
||||
ScheduledDisposition::ReportedFailure(value) => {
|
||||
("reported_failure", Some(value.as_str()), true)
|
||||
}
|
||||
ScheduledDisposition::Refused(value) => {
|
||||
("refused", Some(value.as_str()), true)
|
||||
}
|
||||
};
|
||||
let should_deliver = match job.delivery_policy {
|
||||
DeliveryPolicy::Always => true,
|
||||
DeliveryPolicy::OnAlert => alert,
|
||||
DeliveryPolicy::Never => false,
|
||||
DeliveryPolicy::Direct => false,
|
||||
};
|
||||
if should_deliver {
|
||||
let message = content.unwrap_or("巡检完成，未发现需要关注的问题。");
|
||||
match self
|
||||
.session_manager
|
||||
.deliver_scheduled_message(
|
||||
&job.channel,
|
||||
&job.chat_id,
|
||||
&job.id,
|
||||
&job.name,
|
||||
message,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(()) => (
|
||||
"ok".into(),
|
||||
Some(output),
|
||||
None,
|
||||
Some(kind.into()),
|
||||
Some("delivered".into()),
|
||||
None,
|
||||
),
|
||||
Err(delivery_error) => (
|
||||
"delivery_error".into(),
|
||||
Some(output),
|
||||
None,
|
||||
Some(kind.into()),
|
||||
Some("failed".into()),
|
||||
Some(delivery_error),
|
||||
),
|
||||
}
|
||||
} else {
|
||||
let delivery = if job.delivery_policy == DeliveryPolicy::Never {
|
||||
"skipped"
|
||||
} else {
|
||||
"suppressed"
|
||||
};
|
||||
(
|
||||
"ok".into(),
|
||||
Some(output),
|
||||
None,
|
||||
Some(kind.into()),
|
||||
Some(delivery.into()),
|
||||
None,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(Ok(HandleResult::AgentProcessing)) => (
|
||||
"error".to_string(),
|
||||
None,
|
||||
Some("cron execution returned asynchronous processing".to_string()),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
),
|
||||
Ok(Err(error)) => (
|
||||
"error".to_string(),
|
||||
None,
|
||||
Some(error.to_string()),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
),
|
||||
Err(_) => (
|
||||
"timeout".to_string(),
|
||||
None,
|
||||
Some(format!(
|
||||
"execution exceeded {} seconds",
|
||||
self.config.execution_timeout_secs.max(1)
|
||||
)),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
),
|
||||
};
|
||||
|
||||
if managed
|
||||
&& delivery_status.is_none()
|
||||
&& job.delivery_policy != DeliveryPolicy::Never
|
||||
&& let Some(message) = error.as_deref()
|
||||
{
|
||||
let notice = format!("定时任务「{}」执行失败：{}", job.name, message);
|
||||
match self
|
||||
.session_manager
|
||||
.deliver_scheduled_message(&job.channel, &job.chat_id, &job.id, &job.name, ¬ice)
|
||||
.await
|
||||
{
|
||||
Ok(()) => delivery_status = Some("delivered".into()),
|
||||
Err(error) => {
|
||||
status = "delivery_error".into();
|
||||
delivery_status = Some("failed".into());
|
||||
delivery_error = Some(error);
|
||||
tracing::error!(error = %error, "scheduler: failed to claim due runs");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let (next_run_at, disable, delete) = match &job.schedule {
|
||||
Schedule::At { .. } => (None, !job.delete_after_run, job.delete_after_run),
|
||||
Schedule::Every { .. } | Schedule::Cron { .. } => {
|
||||
match next_run_for_schedule(&job.schedule, finished_at) {
|
||||
Some(next) => (Some(next), false, false),
|
||||
None => (None, true, false),
|
||||
}
|
||||
}
|
||||
};
|
||||
let run = JobRun {
|
||||
id: 0,
|
||||
job_id: job.id.clone(),
|
||||
started_at,
|
||||
finished_at,
|
||||
status,
|
||||
output,
|
||||
error,
|
||||
duration_ms,
|
||||
result_kind,
|
||||
delivery_status,
|
||||
delivery_error,
|
||||
};
|
||||
|
||||
if let Err(error) = self
|
||||
.storage
|
||||
.complete_scheduled_job(&run, &self.owner, next_run_at, disable, delete)
|
||||
async fn execute_claimed_run(self: Arc<Self>, claimed: ClaimedScheduledRun) {
|
||||
let start = Instant::now();
|
||||
let job = &claimed.job;
|
||||
let (completion, agent_execution) = if let Some(_activity) = self.admission.try_enter() {
|
||||
match self.session_manager.agent_coordinator() {
|
||||
Some(coordinator) => match coordinator
|
||||
.execute_scheduled(
|
||||
claimed.run_id,
|
||||
&claimed.owner,
|
||||
&job.id,
|
||||
&job.name,
|
||||
job.agent_id.as_deref(),
|
||||
&job.prompt,
|
||||
self.config.execution_timeout_secs.max(1),
|
||||
)
|
||||
.await
|
||||
{
|
||||
tracing::error!(job_id = %job.id, error = %error, "scheduler: failed to commit job completion");
|
||||
let _ = self
|
||||
.storage
|
||||
.release_scheduled_job_lease(&job.id, &self.owner)
|
||||
.await;
|
||||
return;
|
||||
Ok(execution) => {
|
||||
if execution.status == ScheduledRunStatus::Completed
|
||||
&& let Some(outcome) = execution.outcome.clone()
|
||||
{
|
||||
(
|
||||
ScheduledRunCompletion {
|
||||
status: ScheduledRunStatus::Completed,
|
||||
outcome: outcome.kind,
|
||||
message: outcome.message,
|
||||
diagnostic: execution.error.clone(),
|
||||
duration_ms: start.elapsed().as_millis() as i64,
|
||||
},
|
||||
Some(execution),
|
||||
)
|
||||
} else {
|
||||
let diagnostic = execution.error.clone().unwrap_or_else(|| {
|
||||
"scheduled Agent returned ordinary text without calling complete_scheduled_run"
|
||||
.to_string()
|
||||
});
|
||||
let status = match execution.status {
|
||||
ScheduledRunStatus::TimedOut => ScheduledRunStatus::TimedOut,
|
||||
ScheduledRunStatus::Interrupted | ScheduledRunStatus::Cancelled => {
|
||||
ScheduledRunStatus::Interrupted
|
||||
}
|
||||
_ => ScheduledRunStatus::Failed,
|
||||
};
|
||||
(
|
||||
ScheduledRunCompletion {
|
||||
status,
|
||||
outcome: ScheduledOutcomeKind::Failed,
|
||||
message: if status == ScheduledRunStatus::TimedOut {
|
||||
format!("定时任务「{}」执行超时。", job.name)
|
||||
} else if status == ScheduledRunStatus::Interrupted {
|
||||
format!("定时任务「{}」在系统关停时被中断。", job.name)
|
||||
} else {
|
||||
format!(
|
||||
"定时任务「{}」未可靠完成：Agent 未提交结构化运行结果。",
|
||||
job.name
|
||||
)
|
||||
},
|
||||
diagnostic: Some(diagnostic),
|
||||
duration_ms: start.elapsed().as_millis() as i64,
|
||||
},
|
||||
Some(execution),
|
||||
)
|
||||
}
|
||||
}
|
||||
Err(error) => (
|
||||
ScheduledRunCompletion {
|
||||
status: ScheduledRunStatus::Failed,
|
||||
outcome: ScheduledOutcomeKind::Failed,
|
||||
message: job.agent_id.as_deref().map_or_else(
|
||||
|| format!("定时任务「{}」未能启动 Root Agent 执行。", job.name),
|
||||
|agent_id| {
|
||||
format!(
|
||||
"定时任务「{}」无法使用 Agent「{}」执行。请恢复该 Agent 定义，或更新任务的 agent_id。",
|
||||
job.name, agent_id
|
||||
)
|
||||
},
|
||||
),
|
||||
diagnostic: Some(error.to_string()),
|
||||
duration_ms: start.elapsed().as_millis() as i64,
|
||||
},
|
||||
None,
|
||||
),
|
||||
},
|
||||
None => (
|
||||
ScheduledRunCompletion {
|
||||
status: ScheduledRunStatus::Failed,
|
||||
outcome: ScheduledOutcomeKind::Failed,
|
||||
message: format!("定时任务「{}」未能启动执行。", job.name),
|
||||
diagnostic: Some("AgentCoordinator is unavailable".to_string()),
|
||||
duration_ms: start.elapsed().as_millis() as i64,
|
||||
},
|
||||
None,
|
||||
),
|
||||
}
|
||||
} else {
|
||||
(
|
||||
ScheduledRunCompletion {
|
||||
status: ScheduledRunStatus::Interrupted,
|
||||
outcome: ScheduledOutcomeKind::Failed,
|
||||
message: format!("定时任务「{}」因 Gateway 重载而中断。", job.name),
|
||||
diagnostic: Some("runtime admission closed".to_string()),
|
||||
duration_ms: start.elapsed().as_millis() as i64,
|
||||
},
|
||||
None,
|
||||
)
|
||||
};
|
||||
let finished_at = now_ms();
|
||||
let mut commit_attempt = 0_u64;
|
||||
let commit = loop {
|
||||
let result = match agent_execution.as_ref() {
|
||||
Some(execution) => {
|
||||
self.storage
|
||||
.finish_scheduled_run_with_agent(
|
||||
claimed.run_id,
|
||||
&claimed.owner,
|
||||
&completion,
|
||||
&execution.agent_run_id,
|
||||
&execution.agent_run_id,
|
||||
execution.runtime_generation,
|
||||
&execution.agent_terminal,
|
||||
finished_at,
|
||||
)
|
||||
.await
|
||||
}
|
||||
None => {
|
||||
self.storage
|
||||
.finish_scheduled_run(
|
||||
claimed.run_id,
|
||||
&claimed.owner,
|
||||
&completion,
|
||||
finished_at,
|
||||
)
|
||||
.await
|
||||
}
|
||||
};
|
||||
match result {
|
||||
Err(error) if error.is_transient() && commit_attempt < 2 => {
|
||||
commit_attempt += 1;
|
||||
tracing::warn!(
|
||||
job_id = %job.id,
|
||||
run_id = claimed.run_id,
|
||||
attempt = commit_attempt + 1,
|
||||
error = %error,
|
||||
"scheduler: retrying transient run completion commit"
|
||||
);
|
||||
time::sleep(time::Duration::from_millis(50 * commit_attempt)).await;
|
||||
}
|
||||
result => break result,
|
||||
}
|
||||
};
|
||||
match commit {
|
||||
Ok(true) => tracing::info!(
|
||||
job_id = %job.id,
|
||||
run_id = claimed.run_id,
|
||||
status = completion.status.as_str(),
|
||||
outcome = completion.outcome.as_str(),
|
||||
duration_ms = completion.duration_ms,
|
||||
"scheduler: run completed"
|
||||
),
|
||||
Ok(false) => tracing::warn!(
|
||||
job_id = %job.id,
|
||||
run_id = claimed.run_id,
|
||||
"scheduler: late run result discarded"
|
||||
),
|
||||
Err(error) => tracing::error!(
|
||||
job_id = %job.id,
|
||||
run_id = claimed.run_id,
|
||||
error = %error,
|
||||
"scheduler: failed to commit run completion"
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
tracing::info!(
|
||||
job_id = %job.id,
|
||||
status = %run.status,
|
||||
duration_ms,
|
||||
"scheduler: job completed"
|
||||
async fn deliver_claimed_run(self: Arc<Self>, run: JobRun) {
|
||||
let Some(delivery_owner) = run.delivery_lease_owner.clone() else {
|
||||
tracing::error!(
|
||||
run_id = run.id,
|
||||
"scheduler: claimed delivery has no lease owner"
|
||||
);
|
||||
return;
|
||||
};
|
||||
let result = self
|
||||
.session_manager
|
||||
.deliver_scheduled_run(&run, &delivery_owner)
|
||||
.await;
|
||||
let (delivered, permanent, error) = match result {
|
||||
Ok(()) => (true, false, None),
|
||||
Err(ScheduledDeliveryError::Transient(error)) => {
|
||||
(false, false, Some(sanitize_error(&error)))
|
||||
}
|
||||
Err(ScheduledDeliveryError::Permanent(error)) => {
|
||||
(false, true, Some(sanitize_error(&error)))
|
||||
}
|
||||
};
|
||||
if let Err(commit_error) = self
|
||||
.storage
|
||||
.complete_scheduled_delivery(
|
||||
run.id,
|
||||
&delivery_owner,
|
||||
delivered,
|
||||
permanent,
|
||||
error.as_deref(),
|
||||
now_ms(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
tracing::error!(
|
||||
run_id = run.id,
|
||||
error = %commit_error,
|
||||
"scheduler: failed to commit delivery receipt"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn sanitize_error(error: &str) -> String {
|
||||
error.chars().take(1_024).collect()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_next_run_at_schedule() {
|
||||
let now = 1000000;
|
||||
let next = next_run_for_schedule(&Schedule::At { at: 2000000 }, now);
|
||||
assert_eq!(next, Some(2000000));
|
||||
fn next_run_for_every_uses_claim_time() {
|
||||
assert_eq!(
|
||||
next_run_for_schedule(&Schedule::Every { every_ms: 5_000 }, 1_000),
|
||||
Some(6_000)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_next_run_every_schedule() {
|
||||
let now = 1000000;
|
||||
let next = next_run_for_schedule(&Schedule::Every { every_ms: 5000 }, now);
|
||||
assert_eq!(next, Some(1005000));
|
||||
fn next_run_for_at_keeps_absolute_timestamp() {
|
||||
assert_eq!(
|
||||
next_run_for_schedule(&Schedule::At { at: 2_000 }, 1_000),
|
||||
Some(2_000)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_next_run_cron_every_minute() {
|
||||
let expr = "0 * * * * *".to_string();
|
||||
let schedule = Schedule::Cron { expr, tz: None };
|
||||
let now = 1000000;
|
||||
let next = next_run_for_schedule(&schedule, now);
|
||||
assert!(next.is_some());
|
||||
assert!(next.unwrap() > now);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_next_run_cron_every_day_at_9am() {
|
||||
let expr = "0 0 9 * * *".to_string();
|
||||
let schedule = Schedule::Cron { expr, tz: None };
|
||||
let now = 1000000;
|
||||
let next = next_run_for_schedule(&schedule, now);
|
||||
assert!(next.is_some());
|
||||
let next_ms = next.unwrap();
|
||||
assert!(next_ms > now);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_next_run_cron_uses_from_argument() {
|
||||
let expr = "0 * * * * *".to_string();
|
||||
let schedule = Schedule::Cron { expr, tz: None };
|
||||
let from = chrono::DateTime::parse_from_rfc3339("2026-06-16T12:34:20Z")
|
||||
.unwrap()
|
||||
.timestamp_millis();
|
||||
|
||||
let next = next_run_for_schedule(&schedule, from).unwrap();
|
||||
let expected = chrono::DateTime::parse_from_rfc3339("2026-06-16T12:35:00Z")
|
||||
.unwrap()
|
||||
.timestamp_millis();
|
||||
assert_eq!(next, expected);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scheduled_disposition_is_fail_safe() {
|
||||
assert!(matches!(
|
||||
parse_scheduled_disposition("NO_REPLY"),
|
||||
ScheduledDisposition::Quiet(_)
|
||||
));
|
||||
assert!(matches!(
|
||||
parse_scheduled_disposition("NO_REPLY[INFO]: healthy"),
|
||||
ScheduledDisposition::Quiet(_)
|
||||
));
|
||||
assert!(matches!(
|
||||
parse_scheduled_disposition("NO_REPLY[FAIL]: timeout"),
|
||||
ScheduledDisposition::ReportedFailure(_)
|
||||
));
|
||||
assert!(matches!(
|
||||
parse_scheduled_disposition("NO_REPLY[REFUSE]: denied"),
|
||||
ScheduledDisposition::Refused(_)
|
||||
));
|
||||
assert!(matches!(
|
||||
parse_scheduled_disposition("text mentioning NO_REPLY"),
|
||||
ScheduledDisposition::Content(_)
|
||||
));
|
||||
assert!(matches!(
|
||||
parse_scheduled_disposition("NO_REPLY[INFO] but this is content"),
|
||||
ScheduledDisposition::Content(_)
|
||||
));
|
||||
assert!(matches!(
|
||||
parse_scheduled_disposition(""),
|
||||
ScheduledDisposition::ReportedFailure(_)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_next_run_cron_timezone_uses_from_argument() {
|
||||
let expr = "0 0 9 * * *".to_string();
|
||||
fn cron_timezone_uses_from_argument() {
|
||||
let schedule = Schedule::Cron {
|
||||
expr,
|
||||
expr: "0 0 9 * * *".to_string(),
|
||||
tz: Some("Asia/Shanghai".to_string()),
|
||||
};
|
||||
let from = chrono::DateTime::parse_from_rfc3339("2026-06-16T00:30:00Z")
|
||||
.unwrap()
|
||||
.timestamp_millis();
|
||||
|
||||
let next = next_run_for_schedule(&schedule, from).unwrap();
|
||||
let expected = chrono::DateTime::parse_from_rfc3339("2026-06-16T01:00:00Z")
|
||||
.unwrap()
|
||||
.timestamp_millis();
|
||||
assert_eq!(next, expected);
|
||||
assert_eq!(next_run_for_schedule(&schedule, from), Some(expected));
|
||||
}
|
||||
}
|
||||
|
||||
@ -4,7 +4,9 @@ use crate::bus::{ChatMessage, MediaItem, MessageSource, OutboundMessage, SourceK
|
||||
use crate::session::UnifiedSessionId;
|
||||
use crate::tools::{OutboundDelivery, OutboundMessenger};
|
||||
|
||||
use super::persistence::{append_active_turn_message, append_persisted_messages};
|
||||
use super::persistence::{
|
||||
append_active_turn_message, append_persisted_message_if_absent, append_persisted_messages,
|
||||
};
|
||||
use super::session::{
|
||||
CURRENT_SOURCE_SESSION, CURRENT_TURN_DELIVERIES, CURRENT_TURN_ID, PendingTurnDelivery,
|
||||
SessionManager,
|
||||
@ -118,35 +120,106 @@ impl OutboundMessenger for SessionManager {
|
||||
}
|
||||
|
||||
impl SessionManager {
|
||||
pub async fn deliver_scheduled_message(
|
||||
pub async fn deliver_scheduled_run(
|
||||
&self,
|
||||
channel: &str,
|
||||
chat_id: &str,
|
||||
job_id: &str,
|
||||
job_name: &str,
|
||||
content: &str,
|
||||
) -> Result<(), String> {
|
||||
<Self as OutboundMessenger>::send_message(
|
||||
self,
|
||||
channel,
|
||||
chat_id,
|
||||
None,
|
||||
content,
|
||||
MessageSource {
|
||||
kind: SourceKind::ExternalTrigger,
|
||||
from_channel: Some("scheduler".to_string()),
|
||||
from_session: Some(format!("cron:{job_id}")),
|
||||
from_user_id: None,
|
||||
system_name: Some(job_name.to_string()),
|
||||
task_id: Some(job_id.to_string()),
|
||||
from_run_id: None,
|
||||
from_agent_id: None,
|
||||
},
|
||||
Vec::new(),
|
||||
run: &crate::storage::JobRun,
|
||||
delivery_owner: &str,
|
||||
) -> Result<(), ScheduledDeliveryError> {
|
||||
let content = run
|
||||
.message
|
||||
.as_deref()
|
||||
.unwrap_or("定时任务已结束，但没有生成可投递的结果。请在任务运行记录中查看诊断信息。");
|
||||
let target_sid = if let Some(session_id) = run.target_session_id.as_deref() {
|
||||
UnifiedSessionId::parse(session_id).ok_or_else(|| {
|
||||
ScheduledDeliveryError::Permanent("stored target session is invalid".to_string())
|
||||
})?
|
||||
} else {
|
||||
let resolved = self
|
||||
.resolve_dialog_id(&run.target_channel, &run.target_chat_id)
|
||||
.await
|
||||
.map_err(|error| ScheduledDeliveryError::Transient(error.to_string()))?;
|
||||
let fixed = self
|
||||
.storage
|
||||
.set_scheduled_delivery_target_session(
|
||||
run.id,
|
||||
delivery_owner,
|
||||
&resolved.to_string(),
|
||||
chrono::Utc::now().timestamp_millis(),
|
||||
)
|
||||
.await
|
||||
.map(|_| ())
|
||||
.map_err(|error| ScheduledDeliveryError::Transient(error.to_string()))?
|
||||
.ok_or_else(|| {
|
||||
ScheduledDeliveryError::Transient(
|
||||
"scheduled delivery lost its claim before fixing target session"
|
||||
.to_string(),
|
||||
)
|
||||
})?;
|
||||
UnifiedSessionId::parse(&fixed).ok_or_else(|| {
|
||||
ScheduledDeliveryError::Permanent("fixed target session is invalid".to_string())
|
||||
})?
|
||||
};
|
||||
if target_sid.channel != run.target_channel || target_sid.chat_id != run.target_chat_id {
|
||||
return Err(ScheduledDeliveryError::Permanent(
|
||||
"fixed target session does not belong to the scheduled destination".to_string(),
|
||||
));
|
||||
}
|
||||
let session = self
|
||||
.get_or_activate_session(&target_sid)
|
||||
.await
|
||||
.map_err(|error| ScheduledDeliveryError::Transient(error.to_string()))?;
|
||||
let source = MessageSource {
|
||||
kind: SourceKind::ExternalTrigger,
|
||||
from_channel: Some("scheduler".to_string()),
|
||||
from_session: Some(format!("scheduled-run:{}", run.id)),
|
||||
from_user_id: None,
|
||||
system_name: Some("scheduled task".to_string()),
|
||||
task_id: Some(run.job_id.clone()),
|
||||
from_run_id: run.agent_run_id.clone(),
|
||||
from_agent_id: run.agent_id.clone(),
|
||||
};
|
||||
let mut message = outbound_history_message(content, source, &[]);
|
||||
message.id = format!("scheduled:{}", run.id);
|
||||
let message_id = message.id.clone();
|
||||
append_persisted_message_if_absent(&session, message)
|
||||
.await
|
||||
.map_err(|error| ScheduledDeliveryError::Transient(error.to_string()))?;
|
||||
|
||||
let metadata = HashMap::from([
|
||||
("_session_id".to_string(), target_sid.to_string()),
|
||||
("_message_id".to_string(), message_id),
|
||||
("scheduled_delivery_id".to_string(), run.id.to_string()),
|
||||
]);
|
||||
self.bus
|
||||
.deliver_outbound(OutboundMessage {
|
||||
channel: run.target_channel.clone(),
|
||||
chat_id: run.target_chat_id.clone(),
|
||||
content: content.to_string(),
|
||||
reply_to: None,
|
||||
media: Vec::new(),
|
||||
metadata,
|
||||
delivery: None,
|
||||
})
|
||||
.await
|
||||
.map_err(|error| match error {
|
||||
crate::bus::BusError::Closed
|
||||
| crate::bus::BusError::DeliveryTimedOut
|
||||
| crate::bus::BusError::DeliveryTransient(_) => {
|
||||
ScheduledDeliveryError::Transient(error.to_string())
|
||||
}
|
||||
crate::bus::BusError::DeliveryPermanent(summary) => {
|
||||
ScheduledDeliveryError::Permanent(summary)
|
||||
}
|
||||
})?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum ScheduledDeliveryError {
|
||||
#[error("transient scheduled delivery failure: {0}")]
|
||||
Transient(String),
|
||||
#[error("permanent scheduled delivery failure: {0}")]
|
||||
Permanent(String),
|
||||
}
|
||||
|
||||
fn outbound_history_message(
|
||||
|
||||
@ -14,6 +14,7 @@ pub mod turn;
|
||||
pub use commands::SessionCommand;
|
||||
pub use error::SessionError;
|
||||
pub use events::{DialogInfo, SessionEvent};
|
||||
pub use messenger::ScheduledDeliveryError;
|
||||
pub use session::{
|
||||
AgentCatalogPreparation, SLASH_COMMANDS, Session, SessionManager, SessionManagerServices,
|
||||
SlashCommand,
|
||||
|
||||
@ -101,6 +101,41 @@ pub(super) async fn append_persisted_messages_with_meta(
|
||||
append_persisted_messages_inner(session, messages, VersionPolicy::Advance, None, None).await
|
||||
}
|
||||
|
||||
pub(super) async fn append_persisted_message_if_absent(
|
||||
session: &Arc<Mutex<Session>>,
|
||||
message: ChatMessage,
|
||||
) -> Result<bool, StorageError> {
|
||||
let persistence_lock = { session.lock().await.persistence_lock.clone() };
|
||||
let _persistence_guard = persistence_lock.lock().await;
|
||||
let message_id = message.id.clone();
|
||||
let snapshot = {
|
||||
let guard = session.lock().await;
|
||||
if guard.contains_message_id(&message_id) {
|
||||
return Ok(false);
|
||||
}
|
||||
guard.prepare_message_persist_snapshot(&message)
|
||||
};
|
||||
let Some((storage, session_id, persisted, meta)) = snapshot else {
|
||||
return Ok(false);
|
||||
};
|
||||
let inserted = storage
|
||||
.persist_message_if_absent(&session_id, &persisted, &meta)
|
||||
.await?;
|
||||
if !inserted {
|
||||
return Ok(false);
|
||||
}
|
||||
let mut guard = session.lock().await;
|
||||
if guard.contains_message_id(&message_id) {
|
||||
return Ok(true);
|
||||
}
|
||||
if !guard.apply_prepared_message_in_memory(message, persisted.seq, persisted.created_at, true) {
|
||||
return Err(StorageError::Conflict(format!(
|
||||
"session changed while committing idempotent message {message_id}"
|
||||
)));
|
||||
}
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
pub(super) async fn append_persisted_turn_messages(
|
||||
session: &Arc<Mutex<Session>>,
|
||||
messages: Vec<ChatMessage>,
|
||||
@ -247,7 +282,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn active_turn_side_effect_does_not_invalidate_its_session_version() {
|
||||
async fn side_effect_messages_apply_the_expected_session_version_once() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let storage = Arc::new(
|
||||
crate::storage::Storage::new(&dir.path().join("memory.db"))
|
||||
@ -255,7 +290,7 @@ mod tests {
|
||||
.unwrap(),
|
||||
);
|
||||
let memory_manager = Arc::new(MemoryManager::new(
|
||||
storage,
|
||||
storage.clone(),
|
||||
"test".to_string(),
|
||||
"test".to_string(),
|
||||
));
|
||||
@ -276,12 +311,32 @@ mod tests {
|
||||
price_input_per_million: None,
|
||||
price_output_per_million: None,
|
||||
};
|
||||
let unified_id = crate::session::UnifiedSessionId::new("cli_chat", "chat", "dialog");
|
||||
let now = chrono::Utc::now().timestamp_millis();
|
||||
storage
|
||||
.upsert_session(&crate::storage::session::SessionMeta {
|
||||
id: unified_id.to_string(),
|
||||
channel: "cli_chat".to_string(),
|
||||
chat_id: "chat".to_string(),
|
||||
dialog_id: "dialog".to_string(),
|
||||
title: "test".to_string(),
|
||||
created_at: now,
|
||||
last_active_at: now,
|
||||
message_count: 0,
|
||||
routing_info: None,
|
||||
archived_at: None,
|
||||
deleted_at: None,
|
||||
last_consolidated_at: None,
|
||||
last_compressed_message_at: None,
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
let session = Arc::new(Mutex::new(
|
||||
Session::new(
|
||||
crate::session::UnifiedSessionId::new("cli_chat", "chat", "dialog"),
|
||||
unified_id,
|
||||
config,
|
||||
Arc::new(ToolRegistry::new()),
|
||||
None,
|
||||
Some(storage.clone()),
|
||||
String::new(),
|
||||
"test".to_string(),
|
||||
super::super::session::SessionContextServices {
|
||||
@ -320,5 +375,37 @@ mod tests {
|
||||
let guard = session.lock().await;
|
||||
assert_eq!(guard.state_version_for_test(), base_version + 1);
|
||||
assert_eq!(guard.get_history().len(), 2);
|
||||
drop(guard);
|
||||
|
||||
let idempotent_base_version = session.lock().await.state_version_for_test();
|
||||
let mut scheduled = ChatMessage::assistant("scheduled result");
|
||||
scheduled.id = "scheduled:42".to_string();
|
||||
assert!(
|
||||
append_persisted_message_if_absent(&session, scheduled.clone())
|
||||
.await
|
||||
.unwrap()
|
||||
);
|
||||
assert!(
|
||||
!append_persisted_message_if_absent(&session, scheduled)
|
||||
.await
|
||||
.unwrap()
|
||||
);
|
||||
let guard = session.lock().await;
|
||||
assert_eq!(guard.state_version_for_test(), idempotent_base_version + 1);
|
||||
assert_eq!(
|
||||
guard
|
||||
.get_history()
|
||||
.iter()
|
||||
.filter(|message| message.id == "scheduled:42")
|
||||
.count(),
|
||||
1
|
||||
);
|
||||
drop(guard);
|
||||
let persisted: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM messages WHERE id = ?")
|
||||
.bind("scheduled:42")
|
||||
.fetch_one(storage.pool())
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(persisted, 1);
|
||||
}
|
||||
}
|
||||
|
||||
@ -1083,17 +1083,29 @@ impl Session {
|
||||
persist: bool,
|
||||
advance_state_version: bool,
|
||||
) -> Option<MessagePersistSnapshot> {
|
||||
let persist_snapshot = persist
|
||||
.then(|| self.prepare_message_persist_snapshot(&message))
|
||||
.flatten();
|
||||
let (seq, now) = persist_snapshot
|
||||
.as_ref()
|
||||
.map(|(_, _, message, _)| (message.seq, message.created_at))
|
||||
.unwrap_or_else(|| (self.seq_counter, chrono::Utc::now().timestamp_millis()));
|
||||
let applied =
|
||||
self.apply_prepared_message_in_memory(message, seq, now, advance_state_version);
|
||||
debug_assert!(applied, "fresh in-memory message snapshot must apply");
|
||||
persist_snapshot
|
||||
}
|
||||
|
||||
pub(super) fn prepare_message_persist_snapshot(
|
||||
&self,
|
||||
message: &ChatMessage,
|
||||
) -> Option<MessagePersistSnapshot> {
|
||||
let storage = self.storage.clone()?;
|
||||
let is_user = message.role == "user";
|
||||
let counts_as_user_input =
|
||||
is_user && message.client_visibility == crate::bus::ClientVisibility::Visible;
|
||||
let now = chrono::Utc::now().timestamp_millis();
|
||||
|
||||
// Assign seq
|
||||
let seq = self.seq_counter;
|
||||
self.seq_counter += 1;
|
||||
|
||||
let persist_snapshot = if persist {
|
||||
self.storage.clone().map(|storage| {
|
||||
let msg_meta = crate::storage::message::MessageMeta {
|
||||
id: message.id.clone(),
|
||||
session_id: self.id.to_string(),
|
||||
@ -1127,25 +1139,7 @@ impl Session {
|
||||
.map(|s| serde_json::to_string(s).unwrap_or_default()),
|
||||
created_at: now,
|
||||
};
|
||||
(storage, self.id.to_string(), msg_meta)
|
||||
})
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
// Update in-memory state
|
||||
self.message_seqs.insert(message.id.clone(), seq);
|
||||
self.messages.push(message);
|
||||
self.total_message_count += 1;
|
||||
if counts_as_user_input {
|
||||
self.message_count += 1;
|
||||
}
|
||||
self.last_active_at = now;
|
||||
if advance_state_version {
|
||||
self.state_version = self.state_version.wrapping_add(1);
|
||||
}
|
||||
|
||||
persist_snapshot.map(|(storage, session_id, msg_meta)| {
|
||||
let session_id = self.id.to_string();
|
||||
let session_meta = crate::storage::session::SessionMeta {
|
||||
id: session_id.clone(),
|
||||
channel: self.id.channel.clone(),
|
||||
@ -1153,8 +1147,8 @@ impl Session {
|
||||
dialog_id: self.id.dialog_id.clone(),
|
||||
title: self.title.clone(),
|
||||
created_at: self.created_at,
|
||||
last_active_at: self.last_active_at,
|
||||
message_count: self.message_count,
|
||||
last_active_at: now,
|
||||
message_count: self.message_count + i64::from(counts_as_user_input),
|
||||
routing_info: if self.routing_info.is_empty() {
|
||||
None
|
||||
} else {
|
||||
@ -1165,8 +1159,38 @@ impl Session {
|
||||
last_consolidated_at: self.last_consolidated_at,
|
||||
last_compressed_message_at: self.last_compressed_message_at,
|
||||
};
|
||||
(storage, session_id, msg_meta, session_meta)
|
||||
})
|
||||
Some((storage, session_id, msg_meta, session_meta))
|
||||
}
|
||||
|
||||
pub(super) fn apply_prepared_message_in_memory(
|
||||
&mut self,
|
||||
message: ChatMessage,
|
||||
seq: i64,
|
||||
now: i64,
|
||||
advance_state_version: bool,
|
||||
) -> bool {
|
||||
if self.seq_counter != seq || self.contains_message_id(&message.id) {
|
||||
return false;
|
||||
}
|
||||
let is_user = message.role == "user";
|
||||
let counts_as_user_input =
|
||||
is_user && message.client_visibility == crate::bus::ClientVisibility::Visible;
|
||||
self.message_seqs.insert(message.id.clone(), seq);
|
||||
self.messages.push(message);
|
||||
self.seq_counter += 1;
|
||||
self.total_message_count += 1;
|
||||
if counts_as_user_input {
|
||||
self.message_count += 1;
|
||||
}
|
||||
self.last_active_at = now;
|
||||
if advance_state_version {
|
||||
self.state_version = self.state_version.wrapping_add(1);
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
pub(super) fn contains_message_id(&self, message_id: &str) -> bool {
|
||||
self.message_seqs.contains_key(message_id)
|
||||
}
|
||||
|
||||
/// Roll back messages that were appended in memory but whose atomic
|
||||
@ -1905,7 +1929,7 @@ pub struct SessionManager {
|
||||
provider_config: LLMProviderConfig,
|
||||
tools: Arc<ToolRegistry>,
|
||||
skills_loader: Arc<SkillsLoader>,
|
||||
storage: Arc<Storage>,
|
||||
pub(super) storage: Arc<Storage>,
|
||||
pub(super) bus: Arc<MessageBus>,
|
||||
memory_manager: Arc<crate::memory::MemoryManager>,
|
||||
work_manager: Arc<crate::work::WorkManager>,
|
||||
@ -2276,46 +2300,6 @@ impl SessionManager {
|
||||
self.work_manager.clone()
|
||||
}
|
||||
|
||||
/// 为定时任务创建一个无 session 绑定的 AgentLoop
|
||||
pub fn create_cron_agent(&self) -> Result<AgentLoop, AgentError> {
|
||||
let tools = self.tools.without(&["reload_config"]);
|
||||
let provider = create_provider(self.provider_config.clone())
|
||||
.map_err(|e| AgentError::Other(format!("failed to create cron provider: {}", e)))?;
|
||||
Ok(AgentLoop::with_provider_and_tools(
|
||||
Arc::from(provider),
|
||||
tools,
|
||||
self.provider_config.max_tool_iterations,
|
||||
self.provider_config.model_id.clone(),
|
||||
self.provider_config.workspace_dir.clone(),
|
||||
self.provider_config.input_types.clone(),
|
||||
)
|
||||
.with_context_window(self.provider_config.token_limit))
|
||||
}
|
||||
|
||||
fn create_managed_scheduled_agent(&self) -> Result<(AgentLoop, Arc<ToolRegistry>), AgentError> {
|
||||
let tools = self.tools.without(&[
|
||||
"send_message",
|
||||
"cron_add",
|
||||
"cron_update",
|
||||
"cron_remove",
|
||||
"cron_enable",
|
||||
"cron_disable",
|
||||
"reload_config",
|
||||
]);
|
||||
let provider = create_provider(self.provider_config.clone())
|
||||
.map_err(|e| AgentError::Other(format!("failed to create scheduled provider: {e}")))?;
|
||||
let agent = AgentLoop::with_provider_and_tools(
|
||||
Arc::from(provider),
|
||||
tools.clone(),
|
||||
self.provider_config.max_tool_iterations,
|
||||
self.provider_config.model_id.clone(),
|
||||
self.provider_config.workspace_dir.clone(),
|
||||
self.provider_config.input_types.clone(),
|
||||
)
|
||||
.with_context_window(self.provider_config.token_limit);
|
||||
Ok((agent, tools))
|
||||
}
|
||||
|
||||
/// 获取所有可用的斜杠命令
|
||||
pub fn get_slash_commands(&self) -> &[SlashCommand] {
|
||||
SLASH_COMMANDS
|
||||
@ -5098,103 +5082,6 @@ impl SessionManager {
|
||||
}
|
||||
|
||||
impl SessionManager {
|
||||
///
|
||||
/// Runs in a stateless manner: no session creation, no history persistence.
|
||||
/// The cron system prompt instructs the LLM to deliver results via the
|
||||
/// `send_message` tool, which handles both delivery and history writing
|
||||
/// on the target session.
|
||||
pub async fn handle_cron_message(
|
||||
&self,
|
||||
channel: &str,
|
||||
chat_id: &str,
|
||||
prompt: &str,
|
||||
job_id: &str,
|
||||
job_name: &str,
|
||||
) -> Result<HandleResult, AgentError> {
|
||||
let skills_prompt = self.skills_loader.build_skills_prompt();
|
||||
|
||||
let base_prompt = build_system_prompt(
|
||||
&self.provider_config.workspace_dir,
|
||||
&self.provider_config.model_id,
|
||||
&self.tools,
|
||||
);
|
||||
let cron_context = format!(
|
||||
"## 定时任务执行\n\n\
|
||||
你正在执行定时任务「{job_name}」({job_id})。\n\
|
||||
目标渠道: {channel}:{chat_id}\n\n\
|
||||
规则:\n\
|
||||
- 这不是聊天对话，没有用户会直接看到你的输出\n\
|
||||
- 你必须使用 send_message 工具将最终结果发送到目标渠道\n\
|
||||
- send_message 格式: target_chat_id=\"{channel}:{chat_id}\", content=\"消息内容\"\n\
|
||||
- 可以调用其他工具收集信息、处理任务，但最终消息必须通过 send_message 发送\n\
|
||||
- 只输出最终消息内容，不要输出中间思考过程或分析！"
|
||||
);
|
||||
let full_system_prompt =
|
||||
format!("{}\n\n{}\n\n{}", base_prompt, skills_prompt, cron_context);
|
||||
|
||||
let history = vec![
|
||||
ChatMessage::system(full_system_prompt),
|
||||
ChatMessage::user(prompt),
|
||||
];
|
||||
|
||||
let agent = self.create_cron_agent()?;
|
||||
let source_session = format!("cron:{}", job_name);
|
||||
let result = CURRENT_SOURCE_SESSION
|
||||
.scope(Some(source_session.clone()), async {
|
||||
agent
|
||||
.process_with_context(
|
||||
history,
|
||||
ToolExecutionContext::for_session(source_session),
|
||||
)
|
||||
.await
|
||||
})
|
||||
.await
|
||||
.inspect_err(|e| {
|
||||
tracing::error!(error = %e, job_id = %job_id, "Cron agent processing error");
|
||||
})?;
|
||||
|
||||
Ok(HandleResult::AgentResponse(result.final_response.content))
|
||||
}
|
||||
|
||||
/// Execute a scheduler-managed task. The agent returns a result but cannot
|
||||
/// deliver it itself; Scheduler applies the configured delivery policy.
|
||||
pub async fn handle_managed_scheduled_message(
|
||||
&self,
|
||||
prompt: &str,
|
||||
job_id: &str,
|
||||
job_name: &str,
|
||||
monitor: bool,
|
||||
) -> Result<String, AgentError> {
|
||||
let (agent, tools) = self.create_managed_scheduled_agent()?;
|
||||
let base_prompt = build_system_prompt(
|
||||
&self.provider_config.workspace_dir,
|
||||
&self.provider_config.model_id,
|
||||
&tools,
|
||||
);
|
||||
let skills_prompt = self.skills_loader.build_skills_prompt();
|
||||
let result_contract = if monitor {
|
||||
"这是无人值守巡检。完成必要检查后：一切正常且无需用户关注时，只返回 NO_REPLY[INFO]: <简短原因>；发现问题时返回简洁、可操作的告警；无法完成时返回 NO_REPLY[FAIL]: <原因>；因安全或权限拒绝时返回 NO_REPLY[REFUSE]: <原因>。不要调用 send_message，不要把不确定当作正常。"
|
||||
} else {
|
||||
"这是 Scheduler 托管投递的定时任务。完成任务后只返回应交付给用户的最终内容，不要调用 send_message。"
|
||||
};
|
||||
let system = format!(
|
||||
"{base_prompt}\n\n{skills_prompt}\n\n## 定时任务执行\n任务「{job_name}」({job_id})。\n{result_contract}"
|
||||
);
|
||||
let history = vec![ChatMessage::system(system), ChatMessage::user(prompt)];
|
||||
let source_session = format!("cron:{job_id}");
|
||||
let result = CURRENT_SOURCE_SESSION
|
||||
.scope(Some(source_session.clone()), async {
|
||||
agent
|
||||
.process_with_context(
|
||||
history,
|
||||
ToolExecutionContext::for_session(source_session),
|
||||
)
|
||||
.await
|
||||
})
|
||||
.await?;
|
||||
Ok(result.final_response.content)
|
||||
}
|
||||
|
||||
pub async fn clear_session_history(
|
||||
&self,
|
||||
unified_id: &UnifiedSessionId,
|
||||
|
||||
@ -1175,7 +1175,7 @@ mod tests {
|
||||
.fetch_one(storage.pool())
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(version, 10);
|
||||
assert_eq!(version, crate::storage::SCHEMA_VERSION);
|
||||
for table in [
|
||||
"agent_runs",
|
||||
"agent_session_state",
|
||||
|
||||
@ -10,7 +10,10 @@ pub mod usage;
|
||||
|
||||
pub use context_checkpoint::{ContextCheckpoint, ContextCheckpointState, NewContextCheckpoint};
|
||||
pub use error::StorageError;
|
||||
pub use scheduler::{DeliveryPolicy, JobKind, JobRun, ScheduledJob};
|
||||
pub use scheduler::{
|
||||
ClaimedScheduledRun, DeliveryPolicy, JobRun, ScheduledDeliveryStatus, ScheduledJob,
|
||||
ScheduledJobUpdate, ScheduledOutcomeKind, ScheduledRunCompletion, ScheduledRunStatus,
|
||||
};
|
||||
pub use usage::{SessionUsageTotals, TurnUsageRecord};
|
||||
|
||||
use sqlx::sqlite::{
|
||||
@ -20,7 +23,7 @@ use sqlx::{Pool, Row, Sqlite};
|
||||
use std::path::Path;
|
||||
use tokio::time::{Duration, sleep};
|
||||
|
||||
const SCHEMA_VERSION: i64 = 10;
|
||||
const SCHEMA_VERSION: i64 = 11;
|
||||
const INSERT_MESSAGE_SQL: &str = r#"
|
||||
INSERT INTO messages (
|
||||
id, session_id, seq, role, content, reasoning_content, provider_state,
|
||||
@ -29,6 +32,15 @@ const INSERT_MESSAGE_SQL: &str = r#"
|
||||
)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
"#;
|
||||
const INSERT_MESSAGE_IF_ABSENT_SQL: &str = r#"
|
||||
INSERT INTO messages (
|
||||
id, session_id, seq, role, content, reasoning_content, provider_state,
|
||||
turn_id, iteration, completion_status, client_visibility, turn_origin,
|
||||
media_refs, tool_call_id, tool_name, tool_calls, source, created_at
|
||||
)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(id) DO NOTHING
|
||||
"#;
|
||||
|
||||
pub(crate) fn insert_message_query<'a>(
|
||||
session_id: &'a str,
|
||||
@ -55,6 +67,31 @@ pub(crate) fn insert_message_query<'a>(
|
||||
.bind(msg.created_at)
|
||||
}
|
||||
|
||||
pub(crate) fn insert_message_if_absent_query<'a>(
|
||||
session_id: &'a str,
|
||||
msg: &'a crate::storage::message::MessageMeta,
|
||||
) -> sqlx::query::Query<'a, Sqlite, sqlx::sqlite::SqliteArguments> {
|
||||
sqlx::query(INSERT_MESSAGE_IF_ABSENT_SQL)
|
||||
.bind(&msg.id)
|
||||
.bind(session_id)
|
||||
.bind(msg.seq)
|
||||
.bind(&msg.role)
|
||||
.bind(&msg.content)
|
||||
.bind(&msg.reasoning_content)
|
||||
.bind(&msg.provider_state)
|
||||
.bind(&msg.turn_id)
|
||||
.bind(msg.iteration)
|
||||
.bind(msg.completion_status.as_str())
|
||||
.bind(msg.client_visibility.as_str())
|
||||
.bind(msg.turn_origin.as_str())
|
||||
.bind(&msg.media_refs)
|
||||
.bind(&msg.tool_call_id)
|
||||
.bind(&msg.tool_name)
|
||||
.bind(&msg.tool_calls)
|
||||
.bind(&msg.source)
|
||||
.bind(msg.created_at)
|
||||
}
|
||||
|
||||
fn message_meta_from_row(row: SqliteRow) -> crate::storage::message::MessageMeta {
|
||||
let completion_status: String = row.get("completion_status");
|
||||
crate::storage::message::MessageMeta {
|
||||
@ -407,7 +444,6 @@ impl Storage {
|
||||
.execute(&self.pool)
|
||||
.await?;
|
||||
|
||||
Self::init_scheduler_schema(&self.pool).await?;
|
||||
self.migrate_schema().await?;
|
||||
|
||||
Ok(())
|
||||
@ -428,7 +464,11 @@ impl Storage {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let mut tx = self.pool.begin().await?;
|
||||
// Acquire the migration write lock before inspecting or modifying any
|
||||
// legacy shape. This keeps the v11 rebuild on one connection and makes
|
||||
// a concurrently running older Gateway fail startup cleanly instead of
|
||||
// partially racing the schema migration.
|
||||
let mut tx = self.pool.begin_with("BEGIN IMMEDIATE").await?;
|
||||
// The legacy drops below are a pre-v8 rebuild concern: the batch
|
||||
// "group" concept was removed in v8 and the old `background_tasks`
|
||||
// table in v7. Gate them on `current < 8` so a v8 -> v9 upgrade only
|
||||
@ -458,6 +498,33 @@ impl Storage {
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
}
|
||||
let legacy_scheduler_exists: i64 = sqlx::query_scalar(
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'scheduled_jobs'",
|
||||
)
|
||||
.fetch_one(&mut *tx)
|
||||
.await?;
|
||||
let legacy_scheduler_exists = legacy_scheduler_exists == 1;
|
||||
if legacy_scheduler_exists {
|
||||
sqlx::query(
|
||||
r#"
|
||||
CREATE TABLE IF NOT EXISTS job_runs (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
job_id TEXT NOT NULL REFERENCES scheduled_jobs(id) ON DELETE CASCADE,
|
||||
started_at INTEGER NOT NULL,
|
||||
finished_at INTEGER NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
output TEXT,
|
||||
error TEXT,
|
||||
duration_ms INTEGER NOT NULL,
|
||||
result_kind TEXT,
|
||||
delivery_status TEXT,
|
||||
delivery_error TEXT
|
||||
)
|
||||
"#,
|
||||
)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
}
|
||||
for (table, column, definition) in [
|
||||
("messages", "source", "source TEXT"),
|
||||
("messages", "reasoning_content", "reasoning_content TEXT"),
|
||||
@ -528,7 +595,8 @@ impl Storage {
|
||||
let columns = sqlx::query(sqlx::AssertSqlSafe(pragma))
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
if !columns
|
||||
if !columns.is_empty()
|
||||
&& !columns
|
||||
.iter()
|
||||
.any(|row| row.get::<String, _>("name") == column)
|
||||
{
|
||||
@ -562,11 +630,6 @@ impl Storage {
|
||||
)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
sqlx::query(
|
||||
"CREATE INDEX IF NOT EXISTS idx_jobs_claimable ON scheduled_jobs(enabled, next_run_at, lease_until)",
|
||||
)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
sqlx::query(
|
||||
r#"
|
||||
CREATE TABLE IF NOT EXISTS session_turn_usage (
|
||||
@ -629,6 +692,7 @@ impl Storage {
|
||||
for statement in agent_run::AGENT_SCHEMA_STATEMENTS {
|
||||
sqlx::query(*statement).execute(&mut *tx).await?;
|
||||
}
|
||||
scheduler::migrate_scheduler_v11(&mut tx, legacy_scheduler_exists).await?;
|
||||
sqlx::query(sqlx::AssertSqlSafe(format!(
|
||||
"PRAGMA user_version = {SCHEMA_VERSION}"
|
||||
)))
|
||||
@ -638,70 +702,6 @@ impl Storage {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Initialize the scheduler tables (idempotent).
|
||||
pub(crate) async fn init_scheduler_schema(pool: &Pool<Sqlite>) -> Result<(), StorageError> {
|
||||
sqlx::query(
|
||||
r#"
|
||||
CREATE TABLE IF NOT EXISTS scheduled_jobs (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
schedule TEXT NOT NULL,
|
||||
prompt TEXT NOT NULL,
|
||||
channel TEXT NOT NULL,
|
||||
chat_id TEXT NOT NULL,
|
||||
model TEXT,
|
||||
job_kind TEXT NOT NULL DEFAULT 'task',
|
||||
delivery_policy TEXT NOT NULL DEFAULT 'direct',
|
||||
enabled INTEGER NOT NULL DEFAULT 1,
|
||||
delete_after_run INTEGER NOT NULL DEFAULT 0,
|
||||
next_run_at INTEGER NOT NULL,
|
||||
last_run_at INTEGER,
|
||||
last_status TEXT,
|
||||
last_error TEXT,
|
||||
locked_at INTEGER,
|
||||
lock_owner TEXT,
|
||||
lease_until INTEGER,
|
||||
created_at INTEGER NOT NULL,
|
||||
updated_at INTEGER NOT NULL
|
||||
)
|
||||
"#,
|
||||
)
|
||||
.execute(pool)
|
||||
.await?;
|
||||
|
||||
sqlx::query(
|
||||
r#"
|
||||
CREATE TABLE IF NOT EXISTS job_runs (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
job_id TEXT NOT NULL REFERENCES scheduled_jobs(id) ON DELETE CASCADE,
|
||||
started_at INTEGER NOT NULL,
|
||||
finished_at INTEGER NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
output TEXT,
|
||||
error TEXT,
|
||||
duration_ms INTEGER NOT NULL,
|
||||
result_kind TEXT,
|
||||
delivery_status TEXT,
|
||||
delivery_error TEXT
|
||||
)
|
||||
"#,
|
||||
)
|
||||
.execute(pool)
|
||||
.await?;
|
||||
|
||||
sqlx::query(
|
||||
"CREATE INDEX IF NOT EXISTS idx_jobs_next_run ON scheduled_jobs(enabled, next_run_at)",
|
||||
)
|
||||
.execute(pool)
|
||||
.await?;
|
||||
|
||||
sqlx::query("CREATE INDEX IF NOT EXISTS idx_runs_job_id ON job_runs(job_id)")
|
||||
.execute(pool)
|
||||
.await?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub async fn append_llm_call(
|
||||
&self,
|
||||
provider: &str,
|
||||
@ -962,6 +962,56 @@ impl Storage {
|
||||
Ok(msg.seq)
|
||||
}
|
||||
|
||||
pub async fn persist_message_if_absent(
|
||||
&self,
|
||||
session_id: &str,
|
||||
msg: &crate::storage::message::MessageMeta,
|
||||
meta: &crate::storage::session::SessionMeta,
|
||||
) -> Result<bool, StorageError> {
|
||||
let mut tx = self.pool.begin().await?;
|
||||
let inserted = insert_message_if_absent_query(session_id, msg)
|
||||
.execute(&mut *tx)
|
||||
.await?
|
||||
.rows_affected()
|
||||
== 1;
|
||||
if inserted {
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO sessions (id, channel, chat_id, dialog_id, title, created_at,
|
||||
last_active_at, message_count, routing_info, archived_at, deleted_at,
|
||||
last_consolidated_at, last_compressed_message_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(id) DO UPDATE SET
|
||||
title = excluded.title,
|
||||
last_active_at = excluded.last_active_at,
|
||||
message_count = excluded.message_count,
|
||||
routing_info = excluded.routing_info,
|
||||
archived_at = excluded.archived_at,
|
||||
deleted_at = excluded.deleted_at,
|
||||
last_consolidated_at = excluded.last_consolidated_at,
|
||||
last_compressed_message_at = excluded.last_compressed_message_at
|
||||
"#,
|
||||
)
|
||||
.bind(&meta.id)
|
||||
.bind(&meta.channel)
|
||||
.bind(&meta.chat_id)
|
||||
.bind(&meta.dialog_id)
|
||||
.bind(&meta.title)
|
||||
.bind(meta.created_at)
|
||||
.bind(meta.last_active_at)
|
||||
.bind(meta.message_count)
|
||||
.bind(&meta.routing_info)
|
||||
.bind(meta.archived_at)
|
||||
.bind(meta.deleted_at)
|
||||
.bind(meta.last_consolidated_at)
|
||||
.bind(meta.last_compressed_message_at)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
}
|
||||
tx.commit().await?;
|
||||
Ok(inserted)
|
||||
}
|
||||
|
||||
/// Atomically persist all messages produced by one logical turn together
|
||||
/// with the resulting session metadata. A turn is either fully visible
|
||||
/// after restart or not visible at all.
|
||||
@ -1584,6 +1634,59 @@ mod tests {
|
||||
(storage, dir)
|
||||
}
|
||||
|
||||
async fn scheduler_schema_sql(storage: &Storage) -> Vec<(String, String, Option<String>)> {
|
||||
sqlx::query_as::<_, (String, String, Option<String>)>(
|
||||
r#"
|
||||
SELECT type, name, sql FROM sqlite_master
|
||||
WHERE name IN (
|
||||
'scheduled_jobs', 'job_runs', 'idx_jobs_claimable',
|
||||
'idx_job_runs_job_finished', 'idx_job_runs_recovery',
|
||||
'idx_job_runs_delivery'
|
||||
)
|
||||
ORDER BY type, name
|
||||
"#,
|
||||
)
|
||||
.fetch_all(storage.pool())
|
||||
.await
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
async fn create_v10_scheduler_schema(pool: &Pool<Sqlite>) {
|
||||
sqlx::query(
|
||||
r#"
|
||||
CREATE TABLE scheduled_jobs (
|
||||
id TEXT PRIMARY KEY, name TEXT NOT NULL, schedule TEXT NOT NULL,
|
||||
prompt TEXT NOT NULL, channel TEXT NOT NULL, chat_id TEXT NOT NULL,
|
||||
model TEXT, job_kind TEXT NOT NULL DEFAULT 'task',
|
||||
delivery_policy TEXT NOT NULL DEFAULT 'direct',
|
||||
enabled INTEGER NOT NULL DEFAULT 1,
|
||||
delete_after_run INTEGER NOT NULL DEFAULT 0, next_run_at INTEGER NOT NULL,
|
||||
last_run_at INTEGER, last_status TEXT, last_error TEXT,
|
||||
locked_at INTEGER, lock_owner TEXT, lease_until INTEGER,
|
||||
created_at INTEGER NOT NULL, updated_at INTEGER NOT NULL
|
||||
)
|
||||
"#,
|
||||
)
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
sqlx::query(
|
||||
r#"
|
||||
CREATE TABLE job_runs (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
job_id TEXT NOT NULL REFERENCES scheduled_jobs(id) ON DELETE CASCADE,
|
||||
started_at INTEGER NOT NULL, finished_at INTEGER NOT NULL,
|
||||
status TEXT NOT NULL, output TEXT, error TEXT,
|
||||
duration_ms INTEGER NOT NULL, result_kind TEXT,
|
||||
delivery_status TEXT, delivery_error TEXT
|
||||
)
|
||||
"#,
|
||||
)
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn sqlite_runtime_guards_are_enabled() {
|
||||
let (storage, _dir) = create_test_storage().await;
|
||||
@ -1764,7 +1867,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn legacy_schema_is_migrated_without_rebuild() {
|
||||
async fn legacy_schema_is_migrated_to_canonical_v11() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let db_path = dir.path().join("legacy.db");
|
||||
let pool = SqlitePoolOptions::new()
|
||||
@ -1860,7 +1963,14 @@ mod tests {
|
||||
),
|
||||
(
|
||||
"scheduled_jobs",
|
||||
vec!["locked_at", "lock_owner", "lease_until"],
|
||||
vec![
|
||||
"agent_id",
|
||||
"delivery_policy",
|
||||
"last_outcome",
|
||||
"locked_at",
|
||||
"lock_owner",
|
||||
"lease_until",
|
||||
],
|
||||
),
|
||||
] {
|
||||
let columns = sqlx::query(sqlx::AssertSqlSafe(format!("PRAGMA table_info({table})")))
|
||||
@ -1876,6 +1986,25 @@ mod tests {
|
||||
);
|
||||
}
|
||||
}
|
||||
let scheduler_columns = sqlx::query("PRAGMA table_info(scheduled_jobs)")
|
||||
.fetch_all(storage.pool())
|
||||
.await
|
||||
.unwrap()
|
||||
.into_iter()
|
||||
.map(|row| row.get::<String, _>("name"))
|
||||
.collect::<Vec<_>>();
|
||||
for removed in [
|
||||
"model",
|
||||
"job_kind",
|
||||
"delete_after_run",
|
||||
"last_status",
|
||||
"last_error",
|
||||
] {
|
||||
assert!(
|
||||
!scheduler_columns.iter().any(|column| column == removed),
|
||||
"legacy scheduler column survived v11: {removed}"
|
||||
);
|
||||
}
|
||||
let schema_version: i64 = sqlx::query_scalar("PRAGMA user_version")
|
||||
.fetch_one(storage.pool())
|
||||
.await
|
||||
@ -1914,6 +2043,250 @@ mod tests {
|
||||
assert_eq!(origin, "user");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn v10_scheduler_history_and_active_lease_migrate_to_v11() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let db_path = dir.path().join("v10-scheduler.db");
|
||||
let pool = SqlitePoolOptions::new()
|
||||
.connect_with(
|
||||
SqliteConnectOptions::new()
|
||||
.filename(&db_path)
|
||||
.create_if_missing(true),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
create_v10_scheduler_schema(&pool).await;
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO scheduled_jobs
|
||||
(id, name, schedule, prompt, channel, chat_id, model, job_kind,
|
||||
delivery_policy, enabled, next_run_at, last_run_at, last_status,
|
||||
last_error, created_at, updated_at)
|
||||
VALUES ('legacy-direct', 'Legacy direct', '{"type":"every","every_ms":60000}',
|
||||
'check it', 'cli_chat', 'chat', 'obsolete-model', 'monitor',
|
||||
'direct', 1, 2000, 1000, 'delivery_error', 'legacy diagnostic', 1, 2)
|
||||
"#,
|
||||
)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO job_runs
|
||||
(job_id, started_at, finished_at, status, output, error, duration_ms,
|
||||
result_kind, delivery_status, delivery_error)
|
||||
VALUES ('legacy-direct', 1000, 1100, 'delivery_error', 'legacy result', NULL,
|
||||
100, NULL, 'failed', 'channel rejected target')
|
||||
"#,
|
||||
)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO scheduled_jobs
|
||||
(id, name, schedule, prompt, channel, chat_id, job_kind,
|
||||
delivery_policy, enabled, next_run_at, last_run_at, last_status,
|
||||
locked_at, lock_owner, lease_until, created_at, updated_at)
|
||||
VALUES ('legacy-locked', 'Legacy locked', '{"type":"every","every_ms":60000}',
|
||||
'check lock', 'cli_chat', 'chat', 'task', 'never', 1, 2000,
|
||||
1500, NULL, 1500, 'old-owner', 900000, 1, 2)
|
||||
"#,
|
||||
)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
sqlx::query("PRAGMA user_version = 10")
|
||||
.execute(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
drop(pool);
|
||||
|
||||
let storage = Storage::new(&db_path).await.unwrap();
|
||||
let migrated_columns = sqlx::query("PRAGMA table_info(scheduled_jobs)")
|
||||
.fetch_all(storage.pool())
|
||||
.await
|
||||
.unwrap()
|
||||
.into_iter()
|
||||
.map(|row| row.get::<String, _>("name"))
|
||||
.collect::<Vec<_>>();
|
||||
assert!(
|
||||
migrated_columns
|
||||
.iter()
|
||||
.any(|column| column == "last_outcome"),
|
||||
"unexpected post-migration scheduler schema: {migrated_columns:?}"
|
||||
);
|
||||
let direct = storage.get_scheduled_job("legacy-direct").await.unwrap();
|
||||
assert_eq!(
|
||||
direct.delivery_policy,
|
||||
crate::storage::DeliveryPolicy::Always
|
||||
);
|
||||
assert_eq!(
|
||||
direct.last_outcome,
|
||||
Some(crate::storage::ScheduledOutcomeKind::Ok)
|
||||
);
|
||||
let historical = storage
|
||||
.list_scheduled_job_runs("legacy-direct", 10)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(historical.len(), 1);
|
||||
assert_eq!(
|
||||
historical[0].status,
|
||||
crate::storage::ScheduledRunStatus::Completed
|
||||
);
|
||||
assert_eq!(
|
||||
historical[0].outcome,
|
||||
Some(crate::storage::ScheduledOutcomeKind::Ok)
|
||||
);
|
||||
assert_eq!(
|
||||
historical[0].delivery_status,
|
||||
crate::storage::ScheduledDeliveryStatus::Failed
|
||||
);
|
||||
assert_eq!(
|
||||
historical[0].diagnostic.as_deref(),
|
||||
Some("legacy diagnostic")
|
||||
);
|
||||
|
||||
let recovered = storage
|
||||
.list_scheduled_job_runs("legacy-locked", 10)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(recovered.len(), 1);
|
||||
assert_eq!(
|
||||
recovered[0].status,
|
||||
crate::storage::ScheduledRunStatus::Unknown
|
||||
);
|
||||
assert_eq!(
|
||||
recovered[0].delivery_status,
|
||||
crate::storage::ScheduledDeliveryStatus::NotRequested
|
||||
);
|
||||
let locked = storage.get_scheduled_job("legacy-locked").await.unwrap();
|
||||
assert!(locked.lock_owner.is_none());
|
||||
|
||||
let columns = sqlx::query("PRAGMA table_info(scheduled_jobs)")
|
||||
.fetch_all(storage.pool())
|
||||
.await
|
||||
.unwrap()
|
||||
.into_iter()
|
||||
.map(|row| row.get::<String, _>("name"))
|
||||
.collect::<Vec<_>>();
|
||||
for removed in [
|
||||
"model",
|
||||
"job_kind",
|
||||
"delete_after_run",
|
||||
"last_status",
|
||||
"last_error",
|
||||
] {
|
||||
assert!(!columns.iter().any(|column| column == removed));
|
||||
}
|
||||
let violations = sqlx::query("PRAGMA foreign_key_check")
|
||||
.fetch_all(storage.pool())
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(violations.is_empty());
|
||||
|
||||
let fresh_dir = tempfile::tempdir().unwrap();
|
||||
let fresh = Storage::new(&fresh_dir.path().join("fresh.db"))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
scheduler_schema_sql(&storage).await,
|
||||
scheduler_schema_sql(&fresh).await
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn invalid_v10_scheduler_data_rolls_back_v11_migration() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let db_path = dir.path().join("invalid-v10-scheduler.db");
|
||||
let pool = SqlitePoolOptions::new()
|
||||
.connect_with(
|
||||
SqliteConnectOptions::new()
|
||||
.filename(&db_path)
|
||||
.create_if_missing(true),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
create_v10_scheduler_schema(&pool).await;
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO scheduled_jobs
|
||||
(id, name, schedule, prompt, channel, chat_id, delivery_policy,
|
||||
enabled, next_run_at, created_at, updated_at)
|
||||
VALUES ('broken', 'Broken', '{not-json}', 'check', 'cli_chat', 'chat',
|
||||
'always', 1, 1000, 1, 1)
|
||||
"#,
|
||||
)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
sqlx::query("PRAGMA user_version = 10")
|
||||
.execute(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
drop(pool);
|
||||
|
||||
let error = match Storage::new(&db_path).await {
|
||||
Ok(_) => panic!("invalid v10 scheduler data unexpectedly migrated"),
|
||||
Err(error) => error,
|
||||
};
|
||||
assert!(matches!(error, StorageError::Migration(_)));
|
||||
|
||||
let pool = SqlitePoolOptions::new()
|
||||
.connect_with(SqliteConnectOptions::new().filename(&db_path))
|
||||
.await
|
||||
.unwrap();
|
||||
let version: i64 = sqlx::query_scalar("PRAGMA user_version")
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(version, 10);
|
||||
let columns = sqlx::query("PRAGMA table_info(scheduled_jobs)")
|
||||
.fetch_all(&pool)
|
||||
.await
|
||||
.unwrap()
|
||||
.into_iter()
|
||||
.map(|row| row.get::<String, _>("name"))
|
||||
.collect::<Vec<_>>();
|
||||
assert!(columns.iter().any(|column| column == "job_kind"));
|
||||
assert!(!columns.iter().any(|column| column == "agent_id"));
|
||||
for table in ["scheduled_jobs_v10_legacy", "job_runs_v10_legacy"] {
|
||||
let exists: i64 = sqlx::query_scalar(
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = ?",
|
||||
)
|
||||
.bind(table)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(exists, 0);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn newer_schema_version_is_rejected() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let db_path = dir.path().join("future.db");
|
||||
let pool = SqlitePoolOptions::new()
|
||||
.connect_with(
|
||||
SqliteConnectOptions::new()
|
||||
.filename(&db_path)
|
||||
.create_if_missing(true),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
sqlx::query("PRAGMA user_version = 12")
|
||||
.execute(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
drop(pool);
|
||||
|
||||
let error = match Storage::new(&db_path).await {
|
||||
Ok(_) => panic!("newer schema version unexpectedly opened"),
|
||||
Err(error) => error,
|
||||
};
|
||||
assert!(matches!(error, StorageError::Migration(_)));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn v3_migration_preserves_existing_reasoning_and_defaults_completion() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
191
src/tools/complete_scheduled_run.rs
Normal file
191
src/tools/complete_scheduled_run.rs
Normal file
@ -0,0 +1,191 @@
|
||||
use async_trait::async_trait;
|
||||
use serde_json::{Value, json};
|
||||
|
||||
use crate::storage::ScheduledOutcomeKind;
|
||||
use crate::tools::{ScheduledOutcome, Tool, ToolExecutionContext, ToolOutput, ToolResult};
|
||||
|
||||
pub struct CompleteScheduledRunTool;
|
||||
|
||||
impl CompleteScheduledRunTool {
|
||||
pub fn new() -> Self {
|
||||
Self
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for CompleteScheduledRunTool {
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl Tool for CompleteScheduledRunTool {
|
||||
fn name(&self) -> &str {
|
||||
"complete_scheduled_run"
|
||||
}
|
||||
|
||||
fn description(&self) -> &str {
|
||||
"Submit the single structured final outcome (ok, alert, failed, or refused) of an unattended scheduled run and end the run immediately."
|
||||
}
|
||||
|
||||
fn parameters_schema(&self) -> Value {
|
||||
json!({
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"outcome": {
|
||||
"type": "string",
|
||||
"enum": ["ok", "alert", "failed", "refused"],
|
||||
"description": "ok: completed with nothing needing attention; alert: completed with actionable findings; failed: did not complete reliably; refused: denied for permission or safety reasons"
|
||||
},
|
||||
"message": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"maxLength": 16384,
|
||||
"description": "The user-facing result or notification body"
|
||||
}
|
||||
},
|
||||
"required": ["outcome", "message"],
|
||||
"additionalProperties": false
|
||||
})
|
||||
}
|
||||
|
||||
fn runtime_injected(&self) -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
fn exclusive(&self) -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
async fn execute(&self, args: Value) -> anyhow::Result<ToolResult> {
|
||||
self.execute_with_context(&ToolExecutionContext::default(), args)
|
||||
.await
|
||||
.map(|output| output.result)
|
||||
}
|
||||
|
||||
async fn execute_with_context(
|
||||
&self,
|
||||
context: &ToolExecutionContext,
|
||||
args: Value,
|
||||
) -> anyhow::Result<ToolOutput> {
|
||||
let Some(sink) = context.scheduled_completion.as_ref() else {
|
||||
return Ok(failure(
|
||||
"complete_scheduled_run is only available to the top-level scheduled Agent",
|
||||
)
|
||||
.into());
|
||||
};
|
||||
if !context.execution_origin.is_scheduled() {
|
||||
return Ok(failure("scheduled completion context is invalid").into());
|
||||
}
|
||||
let Some(object) = args.as_object() else {
|
||||
return Ok(failure("arguments must be an object").into());
|
||||
};
|
||||
if object
|
||||
.keys()
|
||||
.any(|key| key != "outcome" && key != "message")
|
||||
{
|
||||
return Ok(failure("unknown complete_scheduled_run argument").into());
|
||||
}
|
||||
let kind = match args.get("outcome").and_then(Value::as_str) {
|
||||
Some("ok") => ScheduledOutcomeKind::Ok,
|
||||
Some("alert") => ScheduledOutcomeKind::Alert,
|
||||
Some("failed") => ScheduledOutcomeKind::Failed,
|
||||
Some("refused") => ScheduledOutcomeKind::Refused,
|
||||
Some(other) => return Ok(failure(format!("invalid scheduled outcome: {other}")).into()),
|
||||
None => return Ok(failure("outcome is required").into()),
|
||||
};
|
||||
let message = args
|
||||
.get("message")
|
||||
.and_then(Value::as_str)
|
||||
.unwrap_or_default()
|
||||
.trim();
|
||||
if message.is_empty() {
|
||||
return Ok(failure("message must not be empty").into());
|
||||
}
|
||||
if message.chars().count() > 16_384 {
|
||||
return Ok(failure("message exceeds 16384 characters").into());
|
||||
}
|
||||
let message = message.to_string();
|
||||
if let Err(error) = sink.submit(ScheduledOutcome {
|
||||
kind,
|
||||
message: message.clone(),
|
||||
}) {
|
||||
return Ok(failure(error).into());
|
||||
}
|
||||
Ok(ToolResult {
|
||||
success: true,
|
||||
output: format!("scheduled run completed with outcome={}", kind.as_str()),
|
||||
error: None,
|
||||
}
|
||||
.into())
|
||||
}
|
||||
}
|
||||
|
||||
fn failure(error: impl Into<String>) -> ToolResult {
|
||||
ToolResult {
|
||||
success: false,
|
||||
output: String::new(),
|
||||
error: Some(error.into()),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::tools::{ExecutionOrigin, ScheduledCompletionSink};
|
||||
use std::sync::Arc;
|
||||
|
||||
#[tokio::test]
|
||||
async fn submits_exactly_once_in_scheduled_context() {
|
||||
let sink = Arc::new(ScheduledCompletionSink::default());
|
||||
let context = ToolExecutionContext::for_session("scheduled-run:1")
|
||||
.with_execution_origin(ExecutionOrigin::Scheduled { job_run_id: 1 })
|
||||
.with_scheduled_completion(sink.clone());
|
||||
let tool = CompleteScheduledRunTool::new();
|
||||
let first = tool
|
||||
.execute_with_context(&context, json!({"outcome":"ok","message":"healthy"}))
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(first.result.success);
|
||||
assert_eq!(sink.outcome().unwrap().message, "healthy");
|
||||
let second = tool
|
||||
.execute_with_context(&context, json!({"outcome":"alert","message":"again"}))
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(!second.result.success);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn rejects_empty_extra_and_oversized_arguments() {
|
||||
let sink = Arc::new(ScheduledCompletionSink::default());
|
||||
let context = ToolExecutionContext::for_session("scheduled-run:1")
|
||||
.with_execution_origin(ExecutionOrigin::Scheduled { job_run_id: 1 })
|
||||
.with_scheduled_completion(sink);
|
||||
let tool = CompleteScheduledRunTool::new();
|
||||
for args in [
|
||||
json!({"outcome":"ok","message":" "}),
|
||||
json!({"outcome":"ok","message":"fine","notify":false}),
|
||||
json!({"outcome":"unknown","message":"fine"}),
|
||||
] {
|
||||
assert!(
|
||||
!tool
|
||||
.execute_with_context(&context, args)
|
||||
.await
|
||||
.unwrap()
|
||||
.result
|
||||
.success
|
||||
);
|
||||
}
|
||||
assert!(
|
||||
!tool
|
||||
.execute_with_context(
|
||||
&context,
|
||||
json!({"outcome":"ok","message":"x".repeat(16_385)}),
|
||||
)
|
||||
.await
|
||||
.unwrap()
|
||||
.result
|
||||
.success
|
||||
);
|
||||
}
|
||||
}
|
||||
1083
src/tools/cron.rs
1083
src/tools/cron.rs
File diff suppressed because it is too large
Load Diff
@ -155,7 +155,7 @@ impl DelegateTool {
|
||||
args: &Value,
|
||||
context: &ToolExecutionContext,
|
||||
) -> anyhow::Result<ToolResult> {
|
||||
let mode = match args
|
||||
let requested_mode = match args
|
||||
.get("mode")
|
||||
.and_then(Value::as_str)
|
||||
.unwrap_or("foreground")
|
||||
@ -168,6 +168,13 @@ impl DelegateTool {
|
||||
)));
|
||||
}
|
||||
};
|
||||
let background_downgraded =
|
||||
requested_mode == ExecutionMode::Background && context.execution_origin.is_scheduled();
|
||||
let mode = if background_downgraded {
|
||||
ExecutionMode::Foreground
|
||||
} else {
|
||||
requested_mode
|
||||
};
|
||||
let task_values: Vec<&Value> = match args.get("tasks").and_then(Value::as_array) {
|
||||
Some(tasks) if !tasks.is_empty() => tasks.iter().collect(),
|
||||
Some(_) => return Ok(failure("tasks must not be empty")),
|
||||
@ -248,6 +255,7 @@ impl DelegateTool {
|
||||
success: all_completed,
|
||||
output: serde_json::to_string(&json!({
|
||||
"status": if all_completed { "completed" } else { "partial" },
|
||||
"background_downgraded": background_downgraded,
|
||||
"results": payload
|
||||
}))?,
|
||||
error: None,
|
||||
|
||||
@ -3,6 +3,7 @@ pub mod bash;
|
||||
pub mod browser;
|
||||
pub mod calculator;
|
||||
pub mod chat_manager;
|
||||
pub mod complete_scheduled_run;
|
||||
pub mod content_search;
|
||||
pub mod cron;
|
||||
pub mod delegate;
|
||||
@ -32,6 +33,7 @@ pub use bash::BashTool;
|
||||
pub use browser::{BrowserProfilesTool, BrowserTool};
|
||||
pub use calculator::CalculatorTool;
|
||||
pub use chat_manager::ChatManagerTool;
|
||||
pub use complete_scheduled_run::CompleteScheduledRunTool;
|
||||
pub use content_search::ContentSearchTool;
|
||||
pub use delegate::DelegateTool;
|
||||
pub use emit_signal::EmitSignalTool;
|
||||
@ -50,8 +52,9 @@ pub use reload_config::ReloadConfigTool;
|
||||
pub use send_message::SendMessageTool;
|
||||
pub use todo::TodoTool;
|
||||
pub use traits::{
|
||||
OutboundDelivery, OutboundMessenger, ProcessedToolOutput, Tool, ToolArtifact,
|
||||
ToolArtifactAudience, ToolExecutionContext, ToolOutput, ToolOutputProcessor, ToolResult,
|
||||
ExecutionOrigin, OutboundDelivery, OutboundMessenger, ProcessedToolOutput,
|
||||
ScheduledCompletionSink, ScheduledOutcome, Tool, ToolArtifact, ToolArtifactAudience,
|
||||
ToolExecutionContext, ToolOutput, ToolOutputProcessor, ToolResult,
|
||||
};
|
||||
pub use web_fetch::WebFetchTool;
|
||||
|
||||
|
||||
@ -1,6 +1,60 @@
|
||||
use crate::bus::{MediaItem, MediaRef, MessageSource};
|
||||
use async_trait::async_trait;
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
||||
pub enum ExecutionOrigin {
|
||||
#[default]
|
||||
Interactive,
|
||||
Scheduled {
|
||||
job_run_id: i64,
|
||||
},
|
||||
}
|
||||
|
||||
impl ExecutionOrigin {
|
||||
pub fn is_scheduled(self) -> bool {
|
||||
matches!(self, Self::Scheduled { .. })
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct ScheduledOutcome {
|
||||
pub kind: crate::storage::ScheduledOutcomeKind,
|
||||
pub message: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
pub struct ScheduledCompletionSink {
|
||||
outcome: std::sync::Mutex<Option<ScheduledOutcome>>,
|
||||
}
|
||||
|
||||
impl ScheduledCompletionSink {
|
||||
pub fn submit(&self, outcome: ScheduledOutcome) -> Result<(), &'static str> {
|
||||
let mut guard = self
|
||||
.outcome
|
||||
.lock()
|
||||
.unwrap_or_else(|poisoned| poisoned.into_inner());
|
||||
if guard.is_some() {
|
||||
return Err("scheduled result was already submitted");
|
||||
}
|
||||
*guard = Some(outcome);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn outcome(&self) -> Option<ScheduledOutcome> {
|
||||
self.outcome
|
||||
.lock()
|
||||
.unwrap_or_else(|poisoned| poisoned.into_inner())
|
||||
.clone()
|
||||
}
|
||||
|
||||
pub fn is_completed(&self) -> bool {
|
||||
self.outcome
|
||||
.lock()
|
||||
.unwrap_or_else(|poisoned| poisoned.into_inner())
|
||||
.is_some()
|
||||
}
|
||||
}
|
||||
|
||||
/// Session identity supplied by the runtime for tools that own external state.
|
||||
/// Ordinary stateless tools can ignore it through the default trait method.
|
||||
#[derive(Debug, Clone)]
|
||||
@ -10,6 +64,8 @@ pub struct ToolExecutionContext {
|
||||
pub agent: Option<std::sync::Arc<crate::agent::AgentExecutionContext>>,
|
||||
pub cancellation: tokio_util::sync::CancellationToken,
|
||||
pub execution_gate: Option<std::sync::Arc<crate::agent::gate::ExecutionGate>>,
|
||||
pub execution_origin: ExecutionOrigin,
|
||||
pub scheduled_completion: Option<std::sync::Arc<ScheduledCompletionSink>>,
|
||||
}
|
||||
|
||||
impl Default for ToolExecutionContext {
|
||||
@ -20,6 +76,8 @@ impl Default for ToolExecutionContext {
|
||||
agent: None,
|
||||
cancellation: tokio_util::sync::CancellationToken::new(),
|
||||
execution_gate: None,
|
||||
execution_origin: ExecutionOrigin::Interactive,
|
||||
scheduled_completion: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -32,6 +90,8 @@ impl ToolExecutionContext {
|
||||
agent: None,
|
||||
cancellation: tokio_util::sync::CancellationToken::new(),
|
||||
execution_gate: None,
|
||||
execution_origin: ExecutionOrigin::Interactive,
|
||||
scheduled_completion: None,
|
||||
}
|
||||
}
|
||||
|
||||
@ -60,6 +120,19 @@ impl ToolExecutionContext {
|
||||
self.execution_gate = Some(gate);
|
||||
self
|
||||
}
|
||||
|
||||
pub fn with_execution_origin(mut self, execution_origin: ExecutionOrigin) -> Self {
|
||||
self.execution_origin = execution_origin;
|
||||
self
|
||||
}
|
||||
|
||||
pub fn with_scheduled_completion(
|
||||
mut self,
|
||||
sink: std::sync::Arc<ScheduledCompletionSink>,
|
||||
) -> Self {
|
||||
self.scheduled_completion = Some(sink);
|
||||
self
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
|
||||
4
webui/package-lock.json
generated
4
webui/package-lock.json
generated
@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "picobot-webui",
|
||||
"version": "1.21.0",
|
||||
"version": "1.22.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "picobot-webui",
|
||||
"version": "1.21.0",
|
||||
"version": "1.22.0",
|
||||
"dependencies": {
|
||||
"bits-ui": "^2.0.0",
|
||||
"dompurify": "^3.4.12",
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "picobot-webui",
|
||||
"private": true,
|
||||
"version": "1.21.0",
|
||||
"version": "1.22.0",
|
||||
"type": "module",
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
|
||||
@ -2,11 +2,11 @@
|
||||
let { status = "unknown" } = $props();
|
||||
const normalized = $derived(String(status).toLowerCase());
|
||||
const tone = $derived(
|
||||
["completed", "success", "ok", "enabled"].includes(normalized)
|
||||
["completed", "success", "ok", "enabled", "delivered"].includes(normalized)
|
||||
? "ok"
|
||||
: ["failed", "error", "cancelled", "disabled"].includes(normalized)
|
||||
: ["failed", "error", "cancelled", "disabled", "refused", "timed_out"].includes(normalized)
|
||||
? "fail"
|
||||
: ["running", "pending"].includes(normalized) ? "run" : ""
|
||||
: ["running", "pending", "delivering", "alert", "unknown", "interrupted"].includes(normalized) ? "run" : ""
|
||||
);
|
||||
</script>
|
||||
|
||||
|
||||
@ -42,11 +42,23 @@
|
||||
|
||||
function dotColor(status) {
|
||||
if (status === "completed" || status === "success" || status === "ok") return "var(--signal)";
|
||||
if (status === "timeout") return "var(--accent)";
|
||||
if (status === "timed_out" || status === "unknown" || status === "interrupted") return "var(--accent)";
|
||||
if (status === "running") return "var(--info)";
|
||||
return "var(--danger)";
|
||||
}
|
||||
|
||||
function scheduleLabel(schedule) {
|
||||
if (!schedule) return "—";
|
||||
if (schedule.type === "at") return `单次 · ${formatTime(schedule.at)}`;
|
||||
if (schedule.type === "every") return `每 ${Math.max(1, Math.round(schedule.every_ms / 60000))} 分钟`;
|
||||
if (schedule.type === "cron") return `${schedule.expr}${schedule.tz ? ` · ${schedule.tz}` : ""}`;
|
||||
return schedule.type || "—";
|
||||
}
|
||||
|
||||
function deliveryLabel(policy) {
|
||||
return { always: "始终通知", on_alert: "异常通知", never: "从不通知" }[policy] || policy;
|
||||
}
|
||||
|
||||
onMount(() => {
|
||||
load();
|
||||
const timer = setInterval(() => { tick += 1; }, 30000);
|
||||
@ -58,7 +70,7 @@
|
||||
<div class="toolbar">
|
||||
<div>
|
||||
<h2 style="margin:0">定时任务</h2>
|
||||
<p style="margin:2px 0 0;color:var(--muted);font-size:12px">管理定时任务与巡检；后台子代理运行请到「子代理」页面查看。</p>
|
||||
<p style="margin:2px 0 0;color:var(--muted);font-size:12px">查看统一的定时执行、结构化结果与通知状态；Agent 运行审计请到「子代理」页面。</p>
|
||||
</div>
|
||||
<button class="secondary" onclick={load}><Icon name="refresh" size={16} />刷新</button>
|
||||
</div>
|
||||
@ -73,8 +85,9 @@
|
||||
<h3>{job.name}</h3>
|
||||
<p>{job.prompt}</p>
|
||||
<div class="meta">
|
||||
<span class="mono cron">{job.cron}</span>
|
||||
<span>{job.job_kind === "monitor" ? "巡检" : "任务"} · {job.delivery_policy}</span>
|
||||
<span class="mono cron">{scheduleLabel(job.schedule)}</span>
|
||||
<span>Agent：{job.agent_id || "Root"}</span>
|
||||
<span>投递：{deliveryLabel(job.delivery_policy)}</span>
|
||||
<span>{job.channel} · {job.chat_id}</span>
|
||||
<span>下次 {countdown(job.next_run_at)}</span>
|
||||
<span>上次 {formatTime(job.last_run_at)}</span>
|
||||
@ -87,9 +100,23 @@
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
<StatusBadge status={job.enabled ? (job.last_status || "enabled") : "disabled"} />
|
||||
<StatusBadge status={job.enabled ? (job.last_outcome || "enabled") : "disabled"} />
|
||||
</div>
|
||||
{#if runs[job.id]?.length}<div class="details">{#each runs[job.id] as run}<div class="card-row run-row"><span class="meta">{formatTime(run.finished_at)} · {run.duration_ms}ms{run.result_kind ? ` · ${run.result_kind}` : ""}{run.delivery_status ? ` · ${run.delivery_status}` : ""}</span><StatusBadge status={run.status} /></div>{/each}</div>{/if}
|
||||
{#if runs[job.id]?.length}
|
||||
<div class="details">
|
||||
{#each runs[job.id] as run}
|
||||
<details class="run-detail">
|
||||
<summary class="run-row">
|
||||
<span class="meta">{formatTime(run.finished_at || run.started_at || run.scheduled_for)} · {run.duration_ms ?? "—"}ms · outcome={run.outcome || "—"} · delivery={run.delivery_status} ({run.delivery_attempts})</span>
|
||||
<StatusBadge status={run.status} />
|
||||
</summary>
|
||||
{#if run.message}<p><b>结果</b><br />{run.message}</p>{/if}
|
||||
{#if run.diagnostic}<p class="diagnostic"><b>诊断</b><br />{run.diagnostic}</p>{/if}
|
||||
{#if run.delivery_error}<p class="diagnostic"><b>投递错误</b><br />{run.delivery_error}</p>{/if}
|
||||
</details>
|
||||
{/each}
|
||||
</div>
|
||||
{/if}
|
||||
</article>
|
||||
{:else}<div class="empty-card">暂无定时任务</div>{/each}
|
||||
{/if}
|
||||
@ -100,4 +127,9 @@
|
||||
.cron { font-size: 12px; color: var(--text-soft); background: var(--code-bg); padding: 1px 6px; border-radius: 4px; border: 1px solid var(--line); font-variant-numeric: tabular-nums; }
|
||||
.status-dots { display: flex; gap: 4px; margin-top: 6px; align-items: center; }
|
||||
.run-row { display: flex; align-items: center; gap: 8px; flex-wrap: wrap; }
|
||||
.run-detail { padding: 7px 0; border-top: 1px solid var(--line); }
|
||||
.run-detail:first-child { border-top: 0; }
|
||||
.run-detail summary { cursor: pointer; justify-content: space-between; }
|
||||
.run-detail p { margin: 8px 0 0; white-space: pre-wrap; overflow-wrap: anywhere; font-size: 12px; color: var(--text-soft); }
|
||||
.run-detail .diagnostic { color: var(--danger); }
|
||||
</style>
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user