diff --git a/AGENTS.md b/AGENTS.md index 3255cd5..4b3bef8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -92,7 +92,7 @@ Scheduler → SessionManager scheduled execution → AgentLoop → Scheduler del - **TurnController** is the only owner of active Turn state; snapshots are complete latest-wins values, not token queues, and `Completed` is published only after atomic persistence succeeds - **DeliveryCoordinator** projects active Turn snapshots without mutating history; it owns `TurnSink` lifecycle but no platform message IDs, which remain private to each sink - **WorkManager** owns the single active plan per session, item state transitions, plan versions, and plan-change events; plans are optional and absent from ordinary chat context -- **Scheduler** supports legacy direct-delivery jobs and managed `task`/`monitor` jobs; managed agents cannot call `send_message`, and `on_alert` suppresses only healthy informational results +- **Scheduler** owns one unified Scheduled Run path: claim-time JobRun snapshots, isolated Root/named Agent execution, exactly-once `complete_scheduled_run`, structured outcome, and policy-driven outbox delivery. Scheduled origin propagates to descendants, forces background delegation to foreground, and disables direct messaging, signals, Inbox completion slots, and cron/config management tools; `on_alert` suppresses only structured `ok` - **AgentLoop** is stateless across turns; it receives prepared history, drains same-Turn steering only at safe model boundaries, calls LLM providers, executes tools, and returns one result - **Context overflow recovery** is type-driven: before tool progress Session may commit one checkpoint and retry once; after any tool batch AgentLoop may retry the current Provider step once from its in-memory transcript, preserving current tool calls/results, and Session must never restart that Turn from durable history - **Context compaction** keeps `messages` append-only and uses one active checkpoint per Session (`summary + first_retained_seq`) for deterministic Provider projection; `/compact`, Turn-boundary auto compaction, and overflow share the same compactor/CAS commit path, Session restoration never derives context from Timeline or calls a Provider, the Model `token_limit` (default 128K) is the hard window ceiling and an optional Agent `token_limit` can only narrow it via `min(agent, model)`, summary input is bounded from that effective window rather than a fixed cap, and the only automatic threshold is `context_tokens > context_window - effective_reserve` diff --git a/Cargo.toml b/Cargo.toml index 46807d8..550069a 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "picobot" -version = "1.21.0" +version = "1.22.0" edition = "2024" [dependencies] diff --git a/README.md b/README.md index ef5d757..927d53a 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ PicoBot 是一个用 Rust 编写的个人 AI 助手运行时。它在本地启 - 将同一套 Agent 能力接入飞书/Lark，并可选用单张卡片实时更新回复。 - 让 Agent 使用本地文件、Shell、搜索、HTTP、浏览器、MCP 工具完成任务。 - 把长期偏好、事实和历史摘要存成可检索记忆。 -- 用 Cron 定时执行任务，并把结果发回目标渠道。 +- 用 Cron 运行隔离的 Root 或命名 Agent，以结构化结果决定始终通知、异常通知或静默记录。 - 通过 Skills 为 Agent 注入项目知识和专用操作指南。 ## 快速开始 @@ -152,7 +152,7 @@ picobot health --json 缺少核心或当前配置要求的依赖时退出码为 `1`；`rg` / `fd` 等有回退实现的加速项只会标记为 `DEGRADED`。运行中的 Gateway 也提供 `/health` 斜杠命令，Agent 可调用同名 `health` 工具，WebUI 的“配置 → 健康检查”可显示相同的结构化结果并手动复查；这些入口共享同一套只读检查逻辑。 -Debian/Ubuntu 将同一个 fd 程序安装为 `fdfind`，两者都视为首选文件搜索后端；只有退回传统 `find` 时才提示性能警告。启用浏览器工具后，Health 除了检查 agent-browser 版本和浏览器路径，还会在隔离的临时 socket namespace 中执行完整离线 doctor，分别报告浏览器安装、真实 headless 启动和运行环境，因此可发现“文件存在但 Chrome 无法启动”或缺少 Linux 共享库等问题。 +Debian/Ubuntu 将同一个 fd 程序安装为 `fdfind`，两者都视为首选文件搜索后端；只有退回传统 `find` 时才提示性能警告。启用浏览器工具后，Health 除了检查 agent-browser 版本和浏览器路径，还会在隔离的临时 socket namespace 中执行完整离线 doctor，分别报告浏览器安装、真实 headless 启动和运行环境，因此可发现“文件存在但 Chrome 无法启动”或缺少 Linux 共享库等问题。Gateway 内按需检查还会报告定时任务的无效 Agent/渠道引用、投递积压、最近失败/超时/unknown、静默 unknown 和执行周期覆盖；Health 不会触发任务或连接 Provider。 ### 5.3 使用 WebUI @@ -273,7 +273,7 @@ WebUI/TUI 上传文件默认保存到 `~/.picobot/media/cli_chat`，单文件上 | `delivery` | 活动 Turn 的展示过滤、latest-wins 节流、终态投递与 TurnSink 生命周期 | | `tools` | Agent 可调用工具集合 | | `storage` | SQLite schema、CRUD、消息和任务持久化 | -| `scheduler` | 领取定时任务，执行普通/巡检 Agent，并按投递策略记录或发送结果 | +| `scheduler` | 原子领取 occurrence，运行隔离的 Scheduled Agent，并通过持久化 outbox 按策略投递结构化结果 | | `work` | 管理 session 级单 active plan、并行子项状态和 WebSocket 变更事件 | | `skills` | 加载 Skill，并把 Skill 指南注入系统提示 | | `mcp` | 连接 MCP Server，将远端工具包装成普通 Tool | @@ -329,7 +329,7 @@ PicoBot 有两类记忆： | Knowledge | 偏好、事实、项目规则、长期可复用信息 | 长期保留，手动删除 | | Timeline | 长对话压缩后的历史摘要 | 默认保留 90 天 | -每轮处理用户消息时，MemoryManager 会按用户输入召回 Knowledge，并作为运行时上下文附加到本轮用户消息。当前召回上限固定为 5；`memory.recall_limit` 已支持解析但尚未接入 worker。长会话使用一个活动 checkpoint：累计摘要加 `first_retained_seq` 之后的原始消息尾部构成模型上下文，原始消息、工具调用结果、ID 和 seq 均不会被压缩改写。旧工具结果会保留在原始历史中，但 checkpoint 边界推进后不再永久占用 Provider 上下文。成功的语义摘要还会 best-effort 保存为 Timeline，供 `timeline_recall` 检索；Timeline 不参与会话恢复正确性。Scheduler 默认创建一个每日维护巡检，按 `memory.timeline_retention_days` 清理过期 Timeline；Knowledge 不会被自动删除。 +每轮处理用户消息时，MemoryManager 会按用户输入召回 Knowledge，并作为运行时上下文附加到本轮用户消息。当前召回上限固定为 5；`memory.recall_limit` 已支持解析但尚未接入 worker。长会话使用一个活动 checkpoint：累计摘要加 `first_retained_seq` 之后的原始消息尾部构成模型上下文，原始消息、工具调用结果、ID 和 seq 均不会被压缩改写。旧工具结果会保留在原始历史中，但 checkpoint 边界推进后不再永久占用 Provider 上下文。成功的语义摘要还会 best-effort 保存为 Timeline，供 `timeline_recall` 检索；Timeline 不参与会话恢复正确性。Scheduler 默认创建一个每日维护任务，按 `memory.timeline_retention_days` 清理过期 Timeline；结果通过 `complete_scheduled_run` 结构化提交，Knowledge 不会被自动删除。 模型的 `models..token_limit` 给出上下文窗口上限，未配置时默认为 128,000；Agent 的 `agents..token_limit` 是可选的收紧上限，两者都有配置时有效窗口取二者最小值，因此 Agent 不能扩大模型窗口。自动压缩使用保留量阈值 `context_tokens > context_window - effective_reserve`，默认 reserve 为 16,384 tokens，并尽量原样保留最近 20,000 tokens。小窗口会自动把 reserve 限制为窗口的一半、把近期保留量限制为有效阈值的一半。摘要请求不使用固定 32K 输入上限，而是按有效窗口扣除摘要输出、提示词和安全余量；超大历史只在摘要请求副本中按“已有 checkpoint + 最新消息优先”生成有界 head/tail 转录，SQLite 原文不变。手动 `/compact` 跳过自动阈值；换成小模型后若发送前预检已发现硬超限，或首次请求返回真实 context overflow，语义摘要不可用时才使用明确标记的确定性降级裁剪，正式请求最多重试一次。若 overflow 发生在工具已经执行之后，AgentLoop 只在当前内存转录上裁掉旧完整 Turn 并重试当前模型步骤一次，不会从数据库历史重跑工具。 @@ -353,7 +353,8 @@ PicoBot 有两类记忆： | `todo` | 为复杂、多轮任务创建并更新当前 session 的持久化计划 | | `send_message` | 向指定渠道或当前会话发送消息，可附带文件/截图；WebUI/TUI 当前 Turn 的附件并入最终回复 | | `chat_manager` | 查看渠道、会话和历史消息 | -| `cron_add/list/remove/enable/disable/update` | 管理定时任务 | +| `cron_add/list/remove/enable/disable/update` | 管理定时任务；`agent_id` 选择 Root/命名 Agent，`delivery_policy` 支持 `always/on_alert/never` | +| `cron_runs` | 查询定时任务的结构化执行结果、诊断和投递状态，包括静默任务 | | `routine_maintenance` | 安全清理超过保留期的 Timeline，不删除 Knowledge | | `health` | 检查核心、配置相关和可选运行依赖 | | `browser` | 可选 agent-browser 浏览器自动化；默认按 dialog 临时使用，长期任务可用 `persistent_id` 复用个人 Profile | diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index fe4e9ad..c3d0dff 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -2,7 +2,7 @@ 本文档描述 PicoBot 当前实现的运行时边界、数据流、并发模型和演进约束。它面向维护者和后续参与改进的 Agent，是代码架构的主入口；行为细节仍以代码和测试为最终依据。 -流式模型输出、reasoning 展示、活动 Turn 快照和 Channel 实时投递的详细设计与取舍见 [STREAMING_TURN_DESIGN.md](STREAMING_TURN_DESIGN.md)。用户输入路由、Session 执行拆分、终态投递确认和历史增量校准的重构方案见 [MESSAGE_FLOW_REFACTOR_DESIGN.md](MESSAGE_FLOW_REFACTOR_DESIGN.md)。已实施的 checkpoint 上下文压缩、pi 风格 reserve 阈值、统一编排和 overflow 失败语义见 [CONTEXT_COMPACTION_DESIGN.md](CONTEXT_COMPACTION_DESIGN.md)。配置运行代、重载边界和失败语义见 [CONFIG_HOT_RELOAD_DESIGN.md](CONFIG_HOT_RELOAD_DESIGN.md)。具名子 Agent、委托图、后台收件箱、结果传递机制与 `queue`/`steer` 信号的设计见 [SUB_AGENT_DESIGN.md](SUB_AGENT_DESIGN.md)。 +流式模型输出、reasoning 展示、活动 Turn 快照和 Channel 实时投递的详细设计与取舍见 [STREAMING_TURN_DESIGN.md](STREAMING_TURN_DESIGN.md)。用户输入路由、Session 执行拆分、终态投递确认和历史增量校准的重构方案见 [MESSAGE_FLOW_REFACTOR_DESIGN.md](MESSAGE_FLOW_REFACTOR_DESIGN.md)。已实施的 checkpoint 上下文压缩、pi 风格 reserve 阈值、统一编排和 overflow 失败语义见 [CONTEXT_COMPACTION_DESIGN.md](CONTEXT_COMPACTION_DESIGN.md)。配置运行代、重载边界和失败语义见 [CONFIG_HOT_RELOAD_DESIGN.md](CONFIG_HOT_RELOAD_DESIGN.md)。具名子 Agent、委托图、后台收件箱、结果传递机制与 `queue`/`steer` 信号的设计见 [SUB_AGENT_DESIGN.md](SUB_AGENT_DESIGN.md)。已实施的统一 Scheduled Run、结构化终结协议、中央投递和 v11 数据库迁移见 [SCHEDULED_RUN_DESIGN.md](SCHEDULED_RUN_DESIGN.md)。 ## 1. 设计目标 @@ -78,7 +78,7 @@ flowchart LR | `health` | 聚合只读依赖检查，供 CLI、Tool 与 slash command 复用 | 安装、修复或连接 Provider | | `storage` | SQLite schema、迁移、原子 CRUD | 运行时调度策略 | | `memory` | Knowledge/Timeline 的存取与召回 | 直接驱动消息发送 | -| `scheduler` | 领取到期任务、执行普通/巡检 Agent、应用投递策略、原子记录结果 | 复用聊天会话历史、直接感知 Channel | +| `scheduler` | 原子领取 occurrence、运行隔离的 Root/命名 Agent、提交结构化结果并 drain 持久化投递 outbox | 解析模型自然语言、绕过 Bus 直接调用 Channel | | `work` | session 级单 active plan、并行子项状态机、版本和变更事件 | 执行模型调用、持有 Channel/WebSocket | | `task_supervisor` | 后台任务注册、取消、限时回收 | 业务级重试和结果语义 | @@ -171,6 +171,18 @@ sequenceDiagram 不要把“已进入 Bus”误认为“外部渠道已收到”。需要确认语义时必须使用 `deliver_outbound`。 +### Scheduled Run + +Scheduler 不复用聊天历史，也不根据模型正文猜测是否通知。一次到期状态在同一 SQLite 事务中取得 Job 租约、插入 `job_runs(status=claimed)`、快照 Agent/投递目标/策略，并提前推进 recurring `next_run_at`；`At` 在 claim 时立即禁用。事件循环以独立有界 JoinSet 执行 Run 和 drain delivery，长任务不阻塞其他领取或通知。 + +每次 Run 通过 `AgentCoordinator` 建立顶层 `agent_runs` 审计记录并执行隔离的 Root 或命名 Agent。Scheduled origin 贯穿所有后代，但只有顶层获得 exactly-once `complete_scheduled_run` sink；后代不继承 sink。Scheduled Agent 不获得 `send_message`、cron/config 管理或 `emit_signal`，不创建 Inbox completion slot，任何 background 委托都收敛为 foreground。普通最终文本不代表成功；没有提交 `ok/alert/failed/refused` 之一即 fail-closed。 + +顶层 AgentRun 终态、JobRun 的 lifecycle/outcome/message/diagnostic、Job 最近摘要、初始 delivery status 和租约释放在一个事务中提交。`always` 投递所有 outcome，`on_alert` 只抑制结构化 `ok`，`never` 始终只记录。`job_runs` 同时作为轻量 outbox：`pending → delivering → delivered/failed`，瞬态错误最多进行三次持久化尝试；OutboundDispatcher 返回清洗后的类型化回执，Scheduler 不解析错误字符串。 + +首次投递把目标 dialog 固定到 `target_session_id`，并先用稳定消息 ID `scheduled:` 幂等写入本地历史，再调用 `MessageBus::deliver_outbound`。发送成功但 ack 提交前崩溃允许带稳定 metadata 的重复通知，不能为避免重复而丢失告警。启动恢复把遗留 claimed/running JobRun 原子收敛为 `unknown+unknown`、关联非终态 AgentRun 收敛为 interrupted，并按 claim-time 策略决定是否进入 outbox；已推进的 occurrence 不自动重跑。完整状态矩阵、v11 schema 和迁移规则见 [SCHEDULED_RUN_DESIGN.md](SCHEDULED_RUN_DESIGN.md)。 + +按需 Health 检查只读查询任务引用、投递积压、最近失败/超时/unknown、`never+unknown`、Every 周期被执行时长覆盖以及不可计算的 next run；它不执行任务、不连接 Provider，也不修改数据。 + ### Control 消息 WebSocket dialog 操作通过 `ControlMessage` 携带一次性回复通道。Gateway control router 以最多 64 个并发受监督任务调用 `SessionManager`，再将 `SessionEvent` 回传给发起者；慢 control 不阻塞其他聊天的 inbound 路由。Bus 只承载消息，不解释操作。 diff --git a/docs/SCHEDULED_RUN_DESIGN.md b/docs/SCHEDULED_RUN_DESIGN.md new file mode 100644 index 0000000..63b3371 --- /dev/null +++ b/docs/SCHEDULED_RUN_DESIGN.md @@ -0,0 +1,1142 @@ +# 统一定时任务执行与投递设计 + +状态：已实施 + +目标数据库版本：v11 + +适用范围：`scheduler`、Scheduled Agent、Cron tools、SQLite、WebUI 任务页 + +## 1. 背景 + +当前定时任务同时存在四组互相耦合的概念： + +- `JobKind::Task` / `JobKind::Monitor`； +- `DeliveryPolicy::Direct` / `Always` / `OnAlert` / `Never`； +- Agent 自己调用 `send_message` 的旧执行路径，以及 Scheduler 托管投递的新执行路径； +- 通过 `NO_REPLY`、`NO_REPLY[INFO]`、`NO_REPLY[FAIL]`、`NO_REPLY[REFUSE]` 等字符串猜测运行结果的协议。 + +这导致任务“做什么”和“是否投递”没有形成正交模型。特别是 `NO_REPLY[INFO] XXXX`、Markdown 包裹、缺少冒号或模型附加解释时，字符串解析会把本应静默的结果当作普通内容投递。继续放宽正则只会扩大不确定协议，无法从根本上解决问题。 + +本设计把所有 AI 定时任务统一成一种 **Scheduled Run**：任务配置只声明调度、执行 Agent、完整 prompt、目标和投递策略；每次运行必须通过运行时注入的终结工具提交结构化结果；Scheduler 根据结构化结果和投递策略做确定性决策。 + +## 2. 设计目标 + +1. 删除 `task` / `monitor` 运行类型，任务语义只由 prompt 表达。 +2. 删除所有 `NO_REPLY[...]` 魔法字符串和自然语言结果解析。 +3. 删除 Agent 自行投递的 `Direct` 路径，所有投递由 Scheduler 拥有。 +4. 统一普通通知、异常巡检和后台维护的执行路径。 +5. 定时任务可选择 Root 或一个命名 Agent；命名 Agent 的 Provider、模型、工具、Skills 和委托边以 AgentCatalog 为准。 +6. Scheduled Run 不产生脱离当前执行的后台子 Agent；请求后台委托时自动按前台委托执行。 +7. 执行结果先持久化，再投递；进程重启后可以恢复未完成投递。 +8. 无法确认是否完成的执行标记为 `unknown`，不盲目重跑同一 occurrence。 +9. 使用一次性、原子、可失败回滚的 SQLite v11 自动迁移；迁移后运行时代码只认识新 schema 和新枚举。 +10. `never` 等静默任务的每次运行仍可通过管理 API、WebUI 和只读工具审计。 + +## 3. 非目标 + +- 不提供分布式 Scheduler 或跨节点共识。 +- 不承诺外部渠道 exactly-once。渠道发送成功但本地确认前进程崩溃时，仍可能重复投递。 +- 不保存 Scheduled Run 的用户聊天历史；每次执行默认是隔离上下文。 +- 不自动把上一次执行的工具结果带到下一次执行。 +- 不增加 `shell`、Webhook 等第二种 Job 执行类型；本期所有任务仍是 Agent 任务。 +- 不保留旧字段、旧枚举、旧工具参数或旧字符串协议的运行时兼容分支。 +- `never` 不提供“失败时例外通知”；选择该策略意味着 `failed`、`refused`、`timed_out` 和 `unknown` 也只进入审计、WebUI 与 Health。 + +## 4. 参考项目取舍 + +本设计采用以下参考经验： + +- ZeroClaw：Cron 明确归属于某个 Agent，复用该 Agent 的身份、模型和权限；Cron 本身是一项顶层运行，而不是子 Agent 完成消息。 +- Hermes：有限生命周期调用方不能接收后台完成结果时，委托回退为同步；进程崩溃后把副作用不确定的执行标记为 `unknown`，不自动重放。 +- PicoBot：保留现有 SQLite 租约、Scheduler 集中投递、MessageBus 目标有序发送和命名 AgentCatalog。 + +明确不采用： + +- Hermes 大量平台特例、JSON Job 主存储和多套 Scheduler provider； +- ZeroClaw 仅清除过期锁后重新执行的恢复方式； +- pi 的子进程式、无持久化 Subagent 示例； +- 把 Cron 结果伪装成后台 Agent Inbox 事件。 + +## 5. 核心模型 + +### 5.1 ScheduledJob + +ScheduledJob 只回答五个问题： + +```text +何时执行：schedule +由谁执行：agent_id +执行什么：prompt +发到哪里：channel + chat_id +何时投递：delivery_policy +``` + +目标 Rust 类型： + +```rust +pub struct ScheduledJob { + pub id: String, + pub name: String, + pub schedule: Schedule, + pub prompt: String, + /// None 表示 Root；Some(id) 表示当前 AgentCatalog 中的命名 Agent。 + pub agent_id: Option, + pub channel: String, + pub chat_id: String, + pub delivery_policy: DeliveryPolicy, + pub enabled: bool, + pub next_run_at: i64, + pub last_run_at: Option, + pub last_outcome: Option, + pub created_at: i64, + pub updated_at: i64, + // durable claim + pub locked_at: Option, + pub lock_owner: Option, + pub lease_until: Option, +} +``` + +删除以下字段： + +- `job_kind`：与 `delivery_policy` 重复； +- `model`：当前并未实际应用，且会绕开命名 Agent 的 Provider/Model 定义； +- `delete_after_run`：当前工具不开放且始终写 `false`；`Schedule::At` 在 claim 事务内立即禁用，用户显式删除即可。 + +### 5.2 DeliveryPolicy + +只保留三个值： + +```rust +pub enum DeliveryPolicy { + Always, + OnAlert, + Never, +} +``` + +- `always`：任何终态都投递； +- `on_alert`：仅 `alert`、`failed`、`refused`、`unknown` 投递； +- `never`：任何终态都不投递，只保留执行记录。 + +删除 `Direct`。Agent 不再有自行完成最终投递的职责。 + +### 5.3 ScheduledOutcome + +Agent 可以主动提交四种结果，运行时恢复还可以产生 `unknown`： + +```rust +pub enum ScheduledOutcome { + Ok { message: String }, + Alert { message: String }, + Failed { message: String }, + Refused { message: String }, + Unknown { message: String }, // 仅运行时生成，模型不能提交 +} +``` + +语义： + +- `ok`：任务成功完成，没有需要用户关注的异常； +- `alert`：任务成功完成并发现需要用户关注的事实； +- `failed`：检查或任务没有可靠完成； +- `refused`：安全策略、授权或 Agent 自身约束拒绝执行； +- `unknown`：进程在持久化终态前退出，无法判断外部副作用是否发生。 + +投递矩阵： + +| Outcome | `always` | `on_alert` | `never` | +|---|---:|---:|---:| +| `ok` | 投递 | 静默 | 静默 | +| `alert` | 投递 | 投递 | 静默 | +| `failed` | 投递 | 投递 | 静默 | +| `refused` | 投递 | 投递 | 静默 | +| `unknown` | 投递 | 投递 | 静默 | + +Agent 只报告事实，不能通过工具参数提供 `notify=true/false`。投递权始终属于 Scheduler。 + +### 5.4 ScheduledRunStatus + +运行状态只描述生命周期，不表达业务是否正常： + +```rust +pub enum ScheduledRunStatus { + Claimed, + Running, + Completed, + Failed, + TimedOut, + Cancelled, + Interrupted, + Unknown, +} +``` + +成功调用终结工具后，生命周期状态是 `completed`，业务结果由 `outcome` 表达；例如 `completed + failed` 表示 Agent 正常结束并明确报告检查失败。Provider 错误或结果协议缺失则是 `failed + failed`。 + +两者必须满足以下组合约束，不能独立随意取值： + +| Run status | 合法 Outcome | +|---|---| +| `claimed` / `running` | `NULL` | +| `completed` | `ok` / `alert` / `failed` / `refused` | +| `failed` / `timed_out` / `cancelled` / `interrupted` | `failed` | +| `unknown` | `unknown` | + +因此 `unknown` status 与 `unknown` outcome 恒同现，且只能由平台恢复路径生成：包括启动时恢复未完成运行，以及迁移无法映射的旧状态；模型不能提交它。历史迁移必须先按生命周期状态归一化，再决定 Outcome，不能产生 `unknown + ok` 等非法组合。 + +## 6. 总体架构 + +```mermaid +flowchart TD + EventLoop[Scheduler event loop] -->|tick + free slots| Claim[Storage claim occurrence] + EventLoop -->|run completion| Reap[reap JoinSet] + EventLoop -->|pending due| Delivery[bounded delivery drain] + Claim --> RunRow[(job_runs: claimed)] + Claim --> Advance[提前推进 next_run / 禁用 At] + RunRow --> JoinSet[bounded JoinSet] + JoinSet --> Resolve[ScheduledAgentRunner] + Resolve --> Catalog[Root config / AgentCatalog] + Resolve --> Agent[AgentLoop] + Agent --> Tools[受限工具 + complete_scheduled_run] + Tools --> Outcome[ScheduledOutcome] + Outcome --> Reap + Reap --> Commit[原子提交 Run 终态与 delivery_status] + Commit -->|suppressed / not_requested| Done[完成] + Commit -->|pending| Delivery + Delivery --> Bus[MessageBus / OutboundDispatcher] + Bus --> Channel[Channel] + Channel --> Ack[delivery_status = delivered] +``` + +组件职责： + +| 组件 | 职责 | 明确不负责 | +|---|---|---| +| `Scheduler` | 非阻塞事件循环、领取、并发上限、执行超时、恢复、投递 drain | 解释模型自然语言 | +| `ScheduledAgentRunner` | 解析 Root/命名 Agent、构造隔离上下文、运行 Agent | 渠道发送、next-run 计算 | +| `AgentLoop` | 模型/工具循环、识别终结工具已提交 | Job 状态或渠道策略 | +| `complete_scheduled_run` | Schema 校验并提交一次结构化 Outcome | 消息发送、数据库写入 | +| `Storage` | occurrence、租约、终态、投递状态的原子转换 | Provider 和 Channel I/O | +| `OutboundDispatcher` | 目标有序、瞬态错误重试、渠道调用、类型化投递回执 | Scheduled Run 业务分类 | + +Scheduler 主循环维护一个 `JoinSet` 和固定并发上限，不再像旧实现一样等待整批 Job 全部结束后才进入下一次 poll： + +1. tick 到达时先回收已完成 Run，并按空闲槽位领取新 occurrence； +2. 每个 Run 终态提交后立即尝试认领并投递自己的 pending 结果； +3. 启动和后续 tick 另行有界回收遗留 pending delivery； +4. 长任务只占用一个执行槽，不阻塞其他到期 Job 的领取或已完成结果的投递； +5. Run task、delivery task 和主循环都由 TaskSupervisor 拥有，关停时停止 admission、取消并有界 join。 + +这仍是一个 Scheduler、一种 Job 和一条投递路径；`JoinSet` 只解决生命周期阻塞，不引入新的任务类型或第二套执行器。 + +## 7. 结构化终结协议 + +### 7.1 工具定义 + +所有 Scheduled Run 都运行时注入同一个工具： + +```json +{ + "name": "complete_scheduled_run", + "parameters": { + "type": "object", + "properties": { + "outcome": { + "type": "string", + "enum": ["ok", "alert", "failed", "refused"] + }, + "message": { + "type": "string", + "minLength": 1, + "maxLength": 16384 + } + }, + "required": ["outcome", "message"], + "additionalProperties": false + } +} +``` + +约束： + +- `runtime_injected() == true`，Agent definition 的 `tools` 不得声明它； +- 只在 `ToolExecutionContext` 带 Scheduled completion sink 时可用； +- `exclusive() == true`，不与其他工具并行； +- 每次运行只接受一次； +- `unknown` 不出现在模型可见 schema 中； +- message 去除首尾空白后必须非空，并在字符边界安全截断到上限。 + +### 7.2 AgentLoop 终结行为 + +运行上下文增加两个用途不同的字段： + +```rust +pub enum ExecutionOrigin { + Interactive, + Scheduled { job_run_id: i64 }, +} + +pub struct ToolExecutionContext { + // existing fields... + pub execution_origin: ExecutionOrigin, + pub scheduled_completion: Option>, +} +``` + +- `execution_origin` 是事实标记，普通构造器缺省为 `Interactive`，`execute_scheduled()` 显式设为 Scheduled；每次 Coordinator 构造子 Agent 的 ToolExecutionContext 时从父 context 复制，因此所有子孙都继承。它用来统一禁止 signal/inbox、把 background 委托降级为 foreground；不得通过字符串形式的 session ID 或 Agent ID 推断来源。 +- `scheduled_completion` 是顶层独占能力，只存在于 Scheduled 顶层 Agent，不得传给任何子 Agent。 + +终结工具用一次性 compare-and-set 写入 `ScheduledCompletionSink`。AgentLoop 在每个工具调用之后检查 sink： + +1. 未提交：继续普通工具循环； +2. 已提交：把本次终结工具调用和结果写入 Agent transcript； +3. 同一 Provider 消息中排在终结工具之后的工具调用统一归约为 `Cancelled`，原因是 Scheduled Run 已完成； +4. 不再调用 Provider，立即返回结构化 Scheduled 结果。 + +终结工具必须是最后的语义动作，但运行时不依赖模型遵守这一提示来保证结束。 + +### 7.3 Fail-closed + +只有合法的结构化提交才能得到 `ok` 并可能静默。以下情况统一生成 `failed`，不读取文本猜测： + +- Agent 输出普通最终文本但没有调用终结工具； +- 输出任何 `NO_REPLY` 变体； +- 工具参数不合法且模型未修正； +- 达到最大工具迭代次数； +- Provider 错误； +- Agent 返回空结果； +- 执行超时。 + +普通最终文本可以截断后保存在 `diagnostic`，但不得作为通知正文。用户通知由 Scheduler 生成，例如： + +```text +定时任务「生产站点巡检」未能完成：Agent 未提交结构化运行结果。 +``` + +## 8. Agent 解析与权限 + +### 8.1 Root 任务 + +`agent_id = NULL` 表示使用当前 Root Provider、模型、上下文窗口和 Skills。工具从 Root registry 派生，但移除： + +- `send_message`； +- `cron_add`、`cron_update`、`cron_remove`、`cron_enable`、`cron_disable`、`cron_list`、`cron_runs`； +- `reload_config`； +- 只服务于交互会话或后台收件箱控制的工具。 + +再注入 `complete_scheduled_run` 和 `delegate`。`delegate` 的目标仍受 AgentCatalog `root_can_delegate()` 约束，其 background 参数在 Scheduled origin 下统一降级为 foreground。Root Scheduled Run 不加载用户聊天历史。 + +### 8.2 命名 Agent 任务 + +`agent_id = Some(id)` 必须从当前不可变 AgentCatalog 解析： + +- Provider profile、模型、上下文窗口来自 Agent definition； +- 工具和 Skills 完全按 definition allowlist； +- `complete_scheduled_run` 作为固有运行时工具额外注入，不构成权限扩张； +- 委托边按 definition 的 `delegates`； +- Job 不提供 per-job tools 或 model override。 + +创建和更新任务时校验当前 `agent_id`。如果之后重载删除或禁用了该 Agent，Gateway 不因一个 Job 无法启动；该 occurrence 记录为 `failed`，并按投递策略通知。Health 页面同时报告悬空 Agent 引用。 + +### 8.3 AgentRun 记录 + +Scheduled 顶层执行仍复用 `agent_runs` 审计和 transcript，但不增加新的 `AgentRunMode`： + +- `mode = foreground`，因为 Scheduler 同步等待它； +- `caller_agent_id = SCHEDULER`； +- `caller_scope_id = scheduled:`； +- `root_session_id = scheduled-run:`，它是审计 scope，不是可接收 Inbox 的 Session； +- 顶层 `ToolExecutionContext.session_id = scheduled-run:`，供 browser 等 session-scoped 状态工具隔离使用； +- 顶层 `ToolExecutionContext.agent` 必须指向该 AgentRun 的完整 AgentExecutionContext；Agent 访问授权以其中的 `root_session_id` 和 ancestry 为准，不以 `session_id` 字符串代替； +- `execution_origin = Scheduled { job_run_id }`； +- `completion_slot_reserved = false`； +- `job_runs.agent_run_id` 关联顶层 AgentRun。 + +这是执行来源的区别，不是第三种并发模式，因此不扩展 `foreground/background` 枚举。 + +Scheduled 顶层及其后代不得创建 `agent_session_state`，不得调用 `reserve_completion_slots`，也不得插入 signal/completion inbox event。`recover_agent_state()` 的通用容量对账不应看到合成 Scheduled scope。 + +## 9. 子 Agent 语义 + +Scheduled Agent 可以调用 `delegate`，但所有子任务必须在本次 Scheduled Run 内收敛： + +```text +delegate(mode=foreground) → 正常执行 +delegate(mode=background) → 自动改为 foreground，并在工具结果中说明降级 +``` + +理由： + +- `scheduled-run:` 不是用户 Session，不能消费 durable inbox continuation； +- Scheduler 必须在一次 run 内得到完整 Outcome； +- 避免创建无法投递的 `cron:` completion； +- 前台子 Agent 仍可并行批量执行并由父 Agent 汇总。 + +Scheduled Agent 不能使用 `emit_signal` 向用户 Session 建立旁路。即使命名 Agent definition 声明了 signal contract，Scheduled origin 也不得注入 `emit_signal`，不得预留 completion slot。子 Agent 的最终结果作为普通工具结果返回父 Scheduled Agent，只有父 Agent 可以调用 `complete_scheduled_run`。 + +`ExecutionOrigin::Scheduled` 必须随每一层子 Agent 继承，使嵌套委托也保持同步收敛语义；`ScheduledCompletionSink` 则是顶层运行能力，Coordinator 构造子 Agent context 时必须显式清空，子 Agent registry 也不得注入终结工具。二者不能合并为同一个可选字段，否则清空 sink 后嵌套子 Agent 会丢失 Scheduled 限制。 + +Root Scheduled Agent 的第一次委托必须走 AgentCatalog 的 `root_can_delegate()` 规则，不能把合成的 `SCHEDULER` 或 `ROOT` 身份误当作命名 Agent 传给 `can_delegate()`。命名 Scheduled Agent 的后续委托仍走 `can_delegate(caller, target)`。这一区分只影响委托边校验，不创建第二条 Scheduled 执行路径。 + +## 10. occurrence、领取和崩溃恢复 + +### 10.1 领取事务 + +`claim_due_scheduled_jobs` 改为 `claim_due_scheduled_runs`。一次领取在同一事务中： + +1. 选择 `enabled=1 AND next_run_at<=now` 且租约为空/过期的 Job； +2. 条件更新租约； +3. 插入唯一的 `job_runs(status=claimed, delivery_status=awaiting_result)` 行，并把其自增 `id` 作为本次 occurrence 的稳定 ID，同时快照 Agent、投递策略和目标； +4. 对 `Every/Cron` 把 `next_run_at` 推进到领取时刻之后的第一个未来时间； +5. 对 `At` 在本次 claim 事务内立即设置 `enabled=0`； +6. 提交后返回 `ClaimedScheduledRun { job_snapshot, run_id, owner }`。 + +领取条件、Run 插入和 Job 推进处于同一写事务；SQLite 的写串行化和条件租约保证一次到期状态只能提交一个 Run。无需额外维护 occurrence key 或 schedule revision。推进下次时间发生在执行之前，因此进程崩溃不会使本次 occurrence 被自动重放。错过的多个历史 tick 不逐个补跑，只执行当前到期 occurrence，并计算下一个未来时间。 + +`Every` 的间隔从领取/开始时刻计算，而不是从完成时刻计算；例如每小时任务执行 55 分钟，下一次约 5 分钟后到期。`Cron` 始终表达绝对日历时间。任务仍受 Job 租约约束，不允许重叠；如果执行时间持续超过调度间隔，Health 应报告“执行时长覆盖调度间隔”，由管理员调整周期或拆分任务。 + +过去时间的 `At` 不能通过创建、更新或重新启用隐式重跑：`cron_add` / `cron_update` 拒绝 `at <= now`，`cron_enable` 拒绝启用已经过期的 At，并要求先把 schedule 更新到未来时间。迁移前已经 enabled 且到期的 At 仍保留为一次合法待执行 occurrence。 + +Job 执行租约固定为 `execution_timeout + shutdown_grace`，执行本身必须在 `execution_timeout` 内终止，因此不增加心跳续租任务。终态提交同时校验 `job_run_id + lock_owner + 非终态 status`；租约过期后迟到的旧执行不能修改新 Run 或 Job 摘要。 + +### 10.2 状态转换 + +```text +claimed → running → completed + ├→ failed + ├→ timed_out + ├→ cancelled + └→ interrupted + +claimed/running --process restart--> unknown +``` + +所有终态不可重写。Job 的租约 owner 必须匹配才能提交终态，旧执行不得覆盖新领取。 + +### 10.3 启动恢复 + +Gateway 启动、Scheduler admission 尚未开放前，Storage 先执行一次 `recover_scheduled_runs(active_generation, now)` 原子事务： + +1. 找到所有 `job_runs.status IN ('claimed','running')`； +2. 把 JobRun 标记为 `status=unknown/outcome=unknown`，写入固定诊断； +3. 若已关联非终态 AgentRun，把 AgentRun 标记为 `interrupted`，说明其执行生命周期随旧进程结束； +4. 根据 JobRun 快照的 delivery policy 设置 `pending` 或 `not_requested`； +5. 清除关联 Job 的旧租约； +6. 不回退已推进的 `next_run_at`，也不重跑 occurrence； +7. 在同一事务内提交以上跨表变化。 + +随后再执行通用 `AgentCoordinator::recover_on_activation()`：已由 Scheduled 恢复事务终结的 AgentRun 不会再次处理。`job_runs.status/outcome` 是业务结果和投递决策的唯一权威；`agent_runs.status=interrupted` 只表示编排执行被进程切断，不得反向覆盖 JobRun 的 `unknown`。开放 Scheduler admission 后先 drain pending delivery，再领取新任务。 + +优雅关停由 TaskSupervisor 先停止新领取，再取消/限时等待运行；能够得到明确取消结果时记录 `interrupted`，只有硬崩溃才在下次启动归为 `unknown`。 + +## 11. 投递设计 + +### 11.1 先提交再发送 + +执行完成事务负责： + +1. 提交 AgentRun 终态； +2. 提交 JobRun status、outcome、message、diagnostic 和 duration； +3. 更新 ScheduledJob 的 `last_run_at`、`last_outcome`； +4. 按矩阵把 delivery status 设置为： + - `pending`：需要投递； + - `suppressed`：`on_alert + ok`； + - `not_requested`：`never`。 +5. 释放 Job 租约。 + +任何渠道 I/O 都发生在事务之后。 + +### 11.2 复用 job_runs 作为轻量 outbox + +不新增通用消息队列表。JobRun 自带目标快照和投递状态： + +```text +awaiting_result → pending / suppressed / not_requested +pending → delivering → delivered + ├→ pending（瞬态失败、退避后重试） + └→ failed（永久失败或次数耗尽） +``` + +Scheduler 在 Run 终态提交后立即尝试该 Run，并在每次 tick 和启动恢复后有界 drain 遗留项： + +- 原子认领 `pending` 或租约过期的 `delivering` 行； +- 最多 3 次持久化尝试； +- 仅对 Channel 明确分类为瞬态的错误重试； +- 重试间隔使用有上限的指数退避； +- 复用 OutboundDispatcher 的 `(channel, chat_id)` 顺序锁； +- 使用稳定 `scheduled_delivery_id = job_run_id` 写入 metadata。 + +具体发送复用现有 `MessageBus::deliver_outbound()` 和 OutboundDispatcher，不允许 Scheduler 绕过 Bus 直接调用 Channel。现有 delivery watch 回执从 `Result<(), String>` 收紧成不含敏感信息的类型化结果，至少区分： + +- `Delivered`； +- `TransientFailure { summary }`； +- `PermanentFailure { summary }`； +- `TimedOut`； +- `DispatcherClosed`。 + +OutboundDispatcher 保留现有单次调用内的短暂、内存级瞬态重试；该调用最终返回的回执算一次持久化 delivery attempt。`TransientFailure`、`TimedOut` 和 `DispatcherClosed` 在未达到持久化尝试上限时回到 `pending`，`PermanentFailure` 直接进入 `failed`。Scheduler 不根据错误字符串猜测是否可重试，也不把“成功写入 outbound 队列”当作渠道送达。 + +类型映射固定如下，Channel adapter 必须先把平台错误归一化为正确的 ChannelError： + +| 来源 | Delivery receipt | 行为 | +|---|---|---| +| Channel 成功返回 | `Delivered` | 标记 delivered | +| `ConnectionError` / `SendError` 经 Dispatcher 内部重试耗尽 | `TransientFailure` | 持久化退避后重试 | +| Dispatcher 单次发送最终超时 | `TimedOut` | 持久化退避后重试 | +| Bus/Dispatcher 关停、lane 暂时饱和 | `DispatcherClosed` / `TransientFailure` | 保留 pending，等待当前或下一运行代 | +| channel 不存在、`ConfigError`、明确的永久平台错误 | `PermanentFailure` | 立即标记 failed | + +未知 `Other` 默认按永久失败处理，除非产生它的调用点明确证明可以安全重试。回执只持久化经过清洗和长度限制的 summary，不保存平台响应正文、凭据或临时 URL。Dispatcher 内部的秒级重试与 Scheduler 的持久化分钟级退避职责不同，不得互相递归调用或把每次内部尝试计入 `delivery_attempts`。 + +外部发送成功但本地 `delivered` 提交前崩溃时可能重复发送。渠道支持幂等键时传递稳定 ID；不支持时允许“至少一次”并在重试消息 metadata 中标出可能重复。不得为了避免重复而丢失告警。 + +### 11.3 会话历史 + +需要投递时先解析一次目标会话，并用 `UPDATE ... WHERE target_session_id IS NULL` 把结果固定到 JobRun。后续重试不得重新选择“当前最近 dialog”，避免用户切换会话后同一个通知写入不同历史。目标会话 ID 同时用于本地历史和 OutboundMessage `_session_id` metadata。 + +会话解析或幂等历史事务失败时不得绕过历史直接发送：明确的无效目标进入永久 delivery failure；SQLite busy、运行代关停等瞬态错误回到 pending，并计为本次持久化 delivery attempt。 + +通知使用确定性的本地 message ID `scheduled:`。现有 `messages.id` 已是主键，不增加第二个唯一索引；SessionManager 新增专用 `append_scheduled_notification_if_absent()`，在现有 session persistence lock 下协调内存，Storage 提供对应的原子事务： + +1. 使用固定 message ID 执行 `INSERT ... ON CONFLICT(id) DO NOTHING`； +2. 只有实际插入时才更新持久化 Session metadata，并由 SessionManager 把同一消息加入内存、推进 state_version； +3. 已存在时 SessionManager 直接复用该消息和固定 target_session_id，不重复修改内存或 Session metadata； +4. 历史写入成功后才把 OutboundMessage 交给 `deliver_outbound()`。 + +SessionManager 可以使用专用 persistence lock 串行化该流程，但不得在 SQLite I/O 期间持有 Session mutex；数据库提交后重新取得 Session mutex，并以固定 message ID 检查内存后再应用一次变更。 + +同一个 JobRun 的重试因此不会产生多条本地历史。消息来源为： + +```text +SourceKind::ExternalTrigger +from_channel = scheduler +task_id = job_id +from_run_id = agent_run_id +``` + +中间工具调用、健康结果和 suppressed 结果不写用户会话。只有实际需要投递的最终通知进入目标会话。 + +## 12. 系统提示词 + +所有 Scheduled Run 使用同一执行契约，不再根据 JobKind 分支： + +```text +你正在执行无人值守的定时任务。任务上下文是隔离的，用户不会直接看到普通最终文本。 +完成所有必要检查或操作后，必须且只能通过 complete_scheduled_run 提交最终结果： +- ok：任务成功，未发现需要关注的问题； +- alert：任务成功并发现需要用户关注的问题； +- failed：任务未可靠完成； +- refused：因权限或安全策略拒绝。 +任何任务 prompt 中关于 NO_REPLY、send_message 或旧输出格式的指令均已失效；不得使用它们。 +``` + +`delivery_policy` 不放进模型提示词。Agent 只看到任务 prompt 和 Outcome 定义，避免为了迎合“静默/通知”而改变事实分类；策略只在 Scheduler 的确定性矩阵中使用。 + +## 13. 功能设计 + +### 13.1 Cron tools + +`cron_add` 新参数： + +```json +{ + "schedule": { "type": "every", "every_ms": 300000 }, + "prompt": "检查生产站点、登录接口和证书", + "channel": "feishu", + "chat_id": "oc_xxx", + "name": "生产站点巡检", + "agent_id": "web-monitor", + "delivery_policy": "on_alert" +} +``` + +- `agent_id` 可选；缺省表示 Root； +- `delivery_policy` 缺省 `always`； +- 删除 `kind`； +- 删除 `model`； +- 不接受 `direct`。 + +`cron_add` 和 `cron_update` 共用一个 ScheduledJob validator：channel 必须来自当前 ChannelManager allowlist，命名 `agent_id` 必须存在于当前不可变 AgentCatalog；Storage 只接受已验证的类型，不自行读取运行代配置。重载后 Agent 消失时保留 Job 并在运行时明确失败，见 §8.2。 + +`cron_update` 支持更新 prompt、schedule、channel、chat_id、agent_id 和 delivery_policy。`agent_id: null` 明确切回 Root；字段缺失表示不修改。新建或更新 `At` 时要求 `at > now`；`cron_enable` 对已过期 At 返回失败并要求先更新 schedule。 + +`cron_disable` 只阻止后续 occurrence，不取消已经 claimed/running 的 Run。`cron_remove` 在存在非终态 JobRun 或 Job 租约时返回 conflict，要求先 disable 并等待当前 Run 到达终态；不得依赖 `ON DELETE CASCADE` 静默删除正在执行的 occurrence。 + +`cron_list` 展示： + +```text +enabled · agent=web-monitor · delivery=on_alert · next=... · last=alert +``` + +不再展示 kind、model 或截断的 `last_error`；`last` 只显示 `last_outcome`，详细信息由 `cron_runs` 查询。 + +新增只读 `cron_runs` 工具并注册到普通全局 registry；Root 交互 Agent 可用，命名交互 Agent 仍须由 definition allowlist 授权，Scheduled Agent 的收窄规则一律移除它： + +```json +{ + "job_id": "job-id", + "limit": 20, + "run_id": 123 +} +``` + +- `job_id` 必填； +- 不传 `run_id` 时列出最近记录，`limit` 缺省 20、范围 1–100，每条返回 run_id、时间、status、outcome、delivery status/attempts、duration 及有界 message/diagnostic 摘要； +- 传 `run_id` 时校验它属于该 Job，并返回该次运行的完整结构化字段；message 上限沿用 16 KiB，diagnostic 经过清洗并执行独立上限； +- `read_only() == true`，不修改 Job/Run、不触发重投； +- `never`、`suppressed` 和投递失败记录与其他策略一样可查询； +- 输出不包含 Provider 私有状态、reasoning、凭据或 Agent transcript。 + +### 13.2 WebUI + +任务页删除“任务/巡检”徽标，展示： + +- Agent：Root 或命名 Agent； +- 投递：始终通知 / 异常通知 / 从不通知； +- 最近 Outcome； +- 最近运行生命周期状态； +- 投递状态、尝试次数及失败原因； +- 最近运行的 message/diagnostic 摘要，并可展开单次详情。 + +`GET /api/jobs/{id}/runs` 明确切换到 v11 shape：`id/job_id/scheduled_for/started_at/finished_at/status/outcome/message/diagnostic/duration_ms/delivery_status/delivery_attempts/delivery_error`；删除 `output/error/result_kind`。API 继续受管理认证保护并保持 limit 上限。 + +创建表单可以提供三个用户友好模板，但模板不进入后端模型： + +- 定期通知 → `always`； +- 异常巡检 → `on_alert`； +- 后台维护 → `never`。 + +### 13.3 Health + +HealthService 增加只读 Scheduler 配置检查： + +- ScheduledJob 引用不存在或 disabled 的 Agent； +- 非法 channel； +- 长时间停留在 pending/delivering 的投递； +- 最近一次 `unknown`、`failed` 或 `timed_out`； +- `never` Job 最近的 unknown（明确提示其不会自动通知）； +- 最近执行时长持续覆盖 `Every` 周期； +- enabled Job 无法计算 next run。 + +Health 不执行 Job、不连接 Provider、不修复数据。 + +## 14. SQLite v11 schema + +### 14.1 scheduled_jobs + +```sql +CREATE TABLE scheduled_jobs ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + schedule TEXT NOT NULL, + prompt TEXT NOT NULL, + agent_id TEXT, + channel TEXT NOT NULL, + chat_id TEXT NOT NULL, + delivery_policy TEXT NOT NULL + CHECK (delivery_policy IN ('always','on_alert','never')), + enabled INTEGER NOT NULL DEFAULT 1 CHECK (enabled IN (0,1)), + next_run_at INTEGER NOT NULL, + last_run_at INTEGER, + last_outcome TEXT CHECK ( + last_outcome IS NULL OR + last_outcome IN ('ok','alert','failed','refused','unknown') + ), + locked_at INTEGER, + lock_owner TEXT, + lease_until INTEGER, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + CHECK (length(trim(id)) > 0), + CHECK (length(trim(name)) > 0), + CHECK (length(trim(prompt)) > 0), + CHECK (length(trim(channel)) > 0), + CHECK (length(trim(chat_id)) > 0), + CHECK (agent_id IS NULL OR length(trim(agent_id)) > 0), + CHECK ( + (locked_at IS NULL AND lock_owner IS NULL AND lease_until IS NULL) OR + (locked_at IS NOT NULL AND lock_owner IS NOT NULL AND lease_until IS NOT NULL) + ) +); + +CREATE INDEX idx_jobs_claimable +ON scheduled_jobs(enabled, next_run_at, lease_until); +``` + +### 14.2 job_runs + +```sql +CREATE TABLE job_runs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + job_id TEXT NOT NULL + REFERENCES scheduled_jobs(id) ON DELETE CASCADE, + scheduled_for INTEGER NOT NULL, + agent_run_id TEXT UNIQUE + REFERENCES agent_runs(id) ON DELETE SET NULL, + + -- claim-time immutable snapshot + agent_id TEXT, + delivery_policy TEXT NOT NULL + CHECK (delivery_policy IN ('always','on_alert','never')), + target_channel TEXT NOT NULL, + target_chat_id TEXT NOT NULL, + -- resolved once when delivery is first prepared; stable across retries + target_session_id TEXT, + + started_at INTEGER, + finished_at INTEGER, + status TEXT NOT NULL CHECK (status IN ( + 'claimed','running','completed','failed', + 'timed_out','cancelled','interrupted','unknown' + )), + outcome TEXT CHECK ( + outcome IS NULL OR + outcome IN ('ok','alert','failed','refused','unknown') + ), + message TEXT, + diagnostic TEXT, + duration_ms INTEGER, + + delivery_status TEXT NOT NULL CHECK (delivery_status IN ( + 'awaiting_result','not_requested','suppressed','pending', + 'delivering','delivered','failed' + )), + delivery_attempts INTEGER NOT NULL DEFAULT 0, + delivery_next_attempt_at INTEGER, + delivery_lease_owner TEXT, + delivery_lease_until INTEGER, + delivery_error TEXT, + + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + CHECK (agent_id IS NULL OR length(trim(agent_id)) > 0), + CHECK (length(trim(target_channel)) > 0), + CHECK (length(trim(target_chat_id)) > 0), + CHECK (target_session_id IS NULL OR length(trim(target_session_id)) > 0), + CHECK (delivery_attempts >= 0), + CHECK (duration_ms IS NULL OR duration_ms >= 0), + CHECK ( + (status IN ('claimed','running') AND outcome IS NULL + AND finished_at IS NULL AND delivery_status = 'awaiting_result') OR + (status = 'completed' AND outcome IS NOT NULL + AND outcome IN ('ok','alert','failed','refused') + AND finished_at IS NOT NULL AND delivery_status != 'awaiting_result') OR + (status IN ('failed','timed_out','cancelled','interrupted') + AND outcome IS NOT NULL AND outcome = 'failed' + AND finished_at IS NOT NULL AND delivery_status != 'awaiting_result') OR + (status = 'unknown' AND outcome IS NOT NULL AND outcome = 'unknown' + AND finished_at IS NOT NULL AND delivery_status != 'awaiting_result') + ), + CHECK ( + (delivery_lease_owner IS NULL AND delivery_lease_until IS NULL) OR + (delivery_lease_owner IS NOT NULL AND delivery_lease_until IS NOT NULL) + ) +); + +CREATE INDEX idx_job_runs_job_finished +ON job_runs(job_id, finished_at DESC); + +CREATE INDEX idx_job_runs_recovery +ON job_runs(status, updated_at); + +CREATE INDEX idx_job_runs_delivery +ON job_runs(delivery_status, delivery_next_attempt_at, delivery_lease_until); +``` + +不为 AgentCatalog 建数据库外键；Agent 定义是配置运行代资源，不是 SQLite 行。`agent_id` 和 definition hash 的最终审计值保存在关联 AgentRun 中。`target_session_id` 也不建 Session 外键：它是投递时固定的路由快照，Session 使用软删除且重试不能因当前 dialog 变化而自动改投；写入历史时仍由 SessionManager 验证目标行和作用域。 + +## 15. 自动迁移设计 + +### 15.1 总体原则 + +- `SCHEMA_VERSION` 从 10 增加到 11； +- Storage 在 Gateway 启动后台任务之前执行迁移； +- migration 使用一个专用连接和 `BEGIN IMMEDIATE`，在单个 SQLite 事务中完成； +- 任一步失败则整体回滚，`user_version` 保持原值，Gateway 拒绝启动； +- v11 运行时代码不读取旧列、不解析旧枚举、不调用旧函数； +- 旧版本形状探测只存在于冻结的 `storage/migrations/legacy_to_v10.rs` 和 `storage/migrations/v11_scheduled.rs`； +- v11 数据库由旧版本二进制打开时，沿用现有“数据库版本过新”拒绝策略； +- 不支持自动降级。 + +`BEGIN IMMEDIATE` 必须在配置的 SQLite busy timeout 内取得写锁；若另一个 PicoBot 进程仍在使用同一数据库且无法取得锁，启动直接报错，不等待后台任务运行后再迁移。部署流程必须先停旧进程再启动新版本。 + +### 15.2 初始化顺序调整 + +当前 `migrate_schema()` 是一段按实际 table shape 累积修补的迁移，并不存在现成的逐版本 `v10::up()`；同时 `init_scheduler_schema()` 在 migration 前创建旧 Scheduler DDL。实施不重写全部 v1–v10 历史，而采用最小版本化： + +1. 删除 migration 前的 `init_scheduler_schema()` 调用，`migrate_schema()` 成为 Scheduler 表 DDL 的唯一入口； +2. 把当前累积逻辑冻结、抽取为 `normalize_legacy_to_v10(conn, detected_shape)`，只供 `current < 10` 的启动迁移调用； +3. 新增 `migrate_v10_to_v11(conn)`，只接受经过预检的 canonical v10 Scheduler 表； +4. 全新数据库直接创建 latest base/Agent/Scheduler v11 schema，不先建立再重建 v10 Scheduler 表； +5. `user_version=0` 且已有历史应用表时先运行 legacy normalizer，再运行 v11；若旧数据库从未有 Scheduler 表，normalizer 只处理其他历史表，随后直接创建 v11 Scheduler 表； +6. `current=10` 只执行 v11，`current=11` 不执行迁移，`current>11` 拒绝； +7. Agent schema 必须在创建带 `agent_runs` 外键的 v11 `job_runs` 之前就绪； +8. 整个所需步骤共享同一个专用连接和 `BEGIN IMMEDIATE` 事务，只有最后写 `user_version=11`； +9. 测试不再直接调用可绕过 migration 的旧初始化函数，统一通过 `Storage::new`、fresh-schema fixture 或明确的 v10 fixture。 + +这只是启动 migration 的版本兼容，不是运行时双路径。无需把历史逻辑拆成 `v1→v2→...→v10`，也不得在 v11 Scheduler 查询中保留 shape probing。 + +### 15.3 v10 预检 + +在创建新表前读取并验证全部旧行： + +- schedule JSON 必须能反序列化； +- delivery policy 必须是 `direct/always/on_alert/never`； +- Job ID、channel、chat_id 和 prompt 必须满足非空约束； +- 旧 JobRun 外键必须能找到 Job； +- Job lease 字段必须全部为空或全部非空； +- 发现损坏数据时返回包含 Job ID 的 migration error，不做默认修复。 + +### 15.4 Job 字段映射 + +| v10 | v11 | 规则 | +|---|---|---| +| `job_kind` | 删除 | 不读取其运行语义 | +| `delivery_policy=direct` | `always` | 新代码统一中央投递 | +| `always/on_alert/never` | 原值 | 保留 | +| `model` | 删除 | 当前执行路径未应用该值；迁移日志报告被删除的非空数量 | +| `delete_after_run` | 删除 | `At` 任务改为 claim 事务内立即禁用 | +| 无 `agent_id` | `NULL` | 使用 Root Scheduled Agent | +| `last_status` | `last_outcome` | 优先取迁移后最新 JobRun.outcome；没有历史 Run 时 `ok→ok`、其他非空值→failed | +| `last_error` | 删除 | 作为最新旧 JobRun diagnostic 的 fallback 后删除，不保留 Job 级兼容列 | + +内置 `picobot-routine-maintenance` 的 prompt 在迁移中按固定 Job ID 改成不含 `NO_REPLY` 的任务描述。用户自定义 prompt 不做不安全的字符串替换；新的系统级 Scheduled 契约明确忽略其中关于 `NO_REPLY`、`send_message` 和旧输出格式的指令。 + +### 15.5 历史 JobRun 映射 + +历史行只用于审计，绝不因升级重新投递。status 与 outcome 必须联合映射： + +| v10 条件 | v11 status | v11 outcome | +|---|---|---| +| `status=ok` | `completed` | 按下表 result_kind 映射 | +| `status=delivery_error` 且已有模型结果/result_kind | `completed` | 按下表 result_kind 映射 | +| `status=delivery_error` 且执行 error 非空、无结果 | `failed` | `failed` | +| `status=timeout` | `timed_out` | `failed` | +| `status=error` | `failed` | `failed` | +| 其他状态 | `unknown` | `unknown` | + +仅在新 status 为 `completed` 时读取 result_kind： + +| v10 result_kind | v11 outcome | +|---|---| +| `quiet` | `ok` | +| `content` 且 Job 为 `on_alert` | `alert` | +| `content` 其他情况 | `ok` | +| `reported_failure` | `failed` | +| `refused` | `refused` | +| `NULL`（旧 Direct） | `ok` | +| 其他无法映射值 | migration error，不猜测 | + +delivery 单独映射，不反向改变执行 status/outcome： + +| v10 delivery_status | v11 delivery_status | +|---|---| +| `delivery_status=direct/delivered` | `delivered` | +| `suppressed/skipped` | `suppressed` | +| `failed` 或存在 `delivery_error` | `failed` | +| 其他 | `not_requested` | + +其他字段： + +- `scheduled_for = started_at`； +- `message = output`； +- `diagnostic = error`；最新历史 Run 的 error 为空时可以用同 Job 的 `last_error` 补全； +- `target_*`、`delivery_policy` 从迁移时的 Job 行快照； +- `target_session_id = NULL`，历史行不重新投递，因此不解析会话； +- `agent_run_id = NULL`； +- 所有 delivery lease 字段为空。 + +若 Job 存在非空 `last_error` 但没有任何可承载它的历史 Run，migration 记录 Job ID 和计数后丢弃该冗余摘要，不为兼容旧投影伪造一条执行记录。 + +### 15.6 迁移时发现旧执行锁 + +v10 只有 Job lease，没有预先建立的 JobRun。若迁移时发现 `lock_owner` 非空或 lease 尚未清理，说明旧进程可能在终态提交前退出： + +1. 为它创建一条 `status=unknown/outcome=unknown` 的 JobRun； +2. message 固定说明升级时发现未完成执行； +3. `always/on_alert` 设置 delivery `pending`，`never` 设置 `not_requested`； +4. recurring Job 的 `next_run_at` 推进到迁移时刻之后； +5. `At` Job 设为 disabled； +6. 清除旧租约。 + +这条未知通知可能与崩溃前已经送达但未提交的通知重复，但不会静默掩盖不确定状态。 + +### 15.7 SQLite 表重建 + +SQLite 删除列和修改 CHECK 约束采用 canonical table rebuild： + +1. 确保 latest `agent_runs` schema 已创建； +2. 把旧 `job_runs`、`scheduled_jobs` 依次 rename 为内部 legacy 临时表； +3. 使用 fresh database 相同的 v11 DDL 常量创建 canonical `scheduled_jobs` 和 `job_runs`； +4. 写入转换后的 Job、历史 Run 和旧锁恢复 Run； +5. 删除 legacy `job_runs`； +6. 删除 legacy `scheduled_jobs`； +7. 重建索引； +8. 执行 `PRAGMA foreign_key_check` 并要求零行； +9. 最后设置 `PRAGMA user_version = 11` 并提交。 + +全新数据库走 latest schema creation，不经过临时 v10 表和上述数据搬迁；但使用同一组 v11 DDL 常量，避免 fresh schema 与 migrated schema 漂移。 + +表重建必须在同一连接、同一事务中完成。不得用多个 pool connection 分散 DDL，也不得在事务提交前启动 Scheduler。 + +### 15.8 “不在代码层面兼容过去”的准确含义 + +允许且必须存在： + +- 一次性 migration 对 v10 表和旧枚举的读取与转换； +- migration tests 的 v10 fixture； +- 迁移日志和损坏数据诊断。 + +明确禁止： + +- `row_to_job` 同时尝试新旧列； +- `DeliveryPolicy::parse("direct")`； +- 保留 `JobKind` 但在新路径忽略； +- 保留 `handle_cron_message` 作为 fallback； +- 继续解析任意 `NO_REPLY` 文本； +- 新旧工具参数并存； +- 根据 `user_version` 在 Scheduler 运行期分支。 + +迁移成功后，进程内只有 v11 类型和 v11 SQL。 + +## 16. 旧代码清理清单 + +### `src/scheduler/mod.rs` + +删除： + +- `ScheduledDisposition`； +- `parse_scheduled_disposition()`； +- `managed` / `Direct` 双路径； +- `job_kind == Monitor` 分支； +- Agent 返回普通文本后再分类的代码； +- 先发送、后写 JobRun 的顺序。 + +替换为：非阻塞 tick/JoinSet 事件循环 → claim occurrence → `ScheduledAgentRunner` → typed outcome → terminal commit → immediate/bounded delivery drain。删除等待整批 Run 完成后才继续 poll 的 `for_each_concurrent(...).await` 结构。 + +### `src/session/session.rs` + +删除： + +- `create_cron_agent()`； +- `create_managed_scheduled_agent()`； +- `handle_cron_message()`； +- `handle_managed_scheduled_message()`； +- Cron 专用 `NO_REPLY` / `send_message` prompt。 + +Scheduled Agent 构造迁移到 Agent/Coordinator 边界，SessionManager 不再执行 Cron Agent。 + +### `src/storage/scheduler.rs` + +删除： + +- `JobKind` 及 parser； +- `DeliveryPolicy::Direct`； +- `model`、`job_kind`、`delete_after_run` 映射； +- `set_scheduled_job_behavior()`； +- 完成时才插入 JobRun 的旧事务。 + +新增严格 typed parser、occurrence claim、status/outcome 联合约束、跨 JobRun/AgentRun 的原子未知恢复、终态提交、固定 target_session_id 和持久化 delivery claim。 + +### `src/storage/mod.rs` 与 `src/storage/migrations/` + +把 `SCHEMA_VERSION` 一次提升到 11，删除 migration 之前调用 Scheduler 旧 DDL 初始化函数的顺序。把现有累积迁移冻结为 `legacy_to_v10` normalizer，新增唯一的 v11 Scheduler migration、fresh v11 DDL 和跨表终态/恢复事务 API；迁移完成后通用 Storage 查询不得包含任何 v10 列名。 + +### `src/tools/cron.rs` + +删除 `kind`、`model` 参数与所有默认联动；默认 `delivery_policy=always`。新增可选 `agent_id` 和共享 Agent/Channel validator，拒绝过去的 At；更新 list 输出，并新增只读 `CronRunsTool`。 + +### `src/tools` + +新增 `complete_scheduled_run.rs`。它是 runtime-injected、exclusive、Scheduled context-only 的无外部副作用控制工具。 + +### `src/agent` + +新增可继承的 `ExecutionOrigin::Scheduled`、顶层独占的 `ScheduledCompletionSink` 和 `AgentCoordinator::execute_scheduled()`；复用 foreground AgentRun 持久化，不创建 agent_session_state、signal 或 inbox completion。AgentLoop 在终结工具成功后停止。 + +### `src/session/messenger.rs` + +第一次投递时固定 target_session_id；把 Scheduled 通知改成利用现有 `messages.id` 主键的稳定 message ID/原子 insert-if-absent 写入；只有实际插入时更新 Session 内存和 metadata。删除任何依赖 `cron:` 作为可消费会话的行为。 + +### `src/bus` + +保留 `MessageBus::deliver_outbound()` 的等待回执入口，把 `OutboundMessage.delivery` 和 `BusError` 的字符串结果改为类型化、可判定 retry class 的安全回执，并补齐 ChannelError→receipt 映射。OutboundDispatcher 仍是唯一 Channel 调用方；普通无回执消息继续使用 `publish_outbound()`。 + +### Gateway / WebUI / Protocol + +- Gateway 激活时先执行原子 Scheduled 恢复，再执行通用 Agent recovery，最后开放 Scheduler admission； +- API JSON 删除 `job_kind`、`model`、`delete_after_run`、`output/error/result_kind`； +- 增加 `agent_id`、`outcome/message/diagnostic` 和 delivery attempts/状态；JobRun 内部另存固定 target Session 路由快照，不向非管理客户端暴露； +- `webui/src/pages/TasksPage.svelte` 删除“巡检/任务”判断； +- `/api/jobs/{id}/runs`、TasksPage、Health 与 `cron_runs` 消费同一 JobRun projection。 + +### 文档与内置知识 + +实施时同步更新： + +- `README.md`； +- `docs/ARCHITECTURE.md`； +- `AGENTS.md`； +- `resources/skills/about-picobot/references/architecture.md`； +- `resources/skills/about-picobot/references/config.md`； +- `resources/skills/about-picobot/references/db-schema.md`； +- `resources/skills/about-picobot/references/tools.md`； +- 内置维护任务 prompt。 + +全仓库应不存在运行时 `NO_REPLY`、`JobKind`、`DeliveryPolicy::Direct` 或 `handle_cron_message` 引用。 + +## 17. 并发与生命周期不变量 + +1. 一个 Job 同时最多有一个持租约 occurrence；短周期任务不会重叠执行。 +2. occurrence 的 JobRun 在 Agent 启动前持久化；同一个到期状态只能提交一个 Run。 +3. `next_run_at` 在 claim 事务中推进；执行失败和进程崩溃不会重放同一 occurrence。 +4. `At` 在 claim 事务内禁用；过期 At 不能通过 enable 隐式重跑。 +5. disable 只影响未来 occurrence；存在非终态 Run 时禁止删除 Job。 +6. Scheduler 主循环不等待整批执行结束；长任务只占并发槽，不能阻塞其他 Job 或 pending delivery。 +7. 只有 `job_run_id + lease owner + 非终态 status` 同时匹配才可以提交终态。 +8. 所有终态不可重写；迟到 Agent 结果必须丢弃并记录日志。 +9. status/outcome 必须满足 §5.4 联合约束；`unknown` 只能由运行时生成。 +10. Outcome 和投递策略都使用 claim-time snapshot；运行过程中编辑 Job 只影响后续 occurrence。 +11. Outcome、Job 摘要和初始 delivery status 在同一事务提交。 +12. 渠道 I/O 不发生在 SQLite 事务或 Session mutex 内。 +13. Scheduled origin 传递给所有后代；ScheduledCompletionSink 只属于顶层。 +14. Scheduled Run 及其后代不创建 agent_session_state、不预留 Inbox slot、不发 signal/completion event。 +15. 后台委托在模型可见参数不变，但 Scheduled origin 强制变为 foreground。 +16. JobRun 是 Outcome/投递的唯一权威；关联 AgentRun 只提供编排审计，不能反向改写 JobRun。 +17. 第一次投递解析出的 target_session_id、message ID 和目标 metadata 在所有重试中保持不变。 +18. `never` 严格不通知，包括失败、拒绝、超时和 unknown；这些状态仍可通过 `cron_runs`、管理页面和 Health 查看。 +19. 静默只来源于结构化 `ok` 与策略矩阵，不能来源于普通文本。 + +## 18. 错误处理矩阵 + +| 场景 | Scheduled JobRun status | Outcome | 投递内容来源 | +|---|---|---|---| +| Agent 提交 `ok` | completed | ok | Agent message | +| Agent 提交 `alert` | completed | alert | Agent message | +| Agent 提交 `failed` | completed | failed | Agent message | +| Agent 提交 `refused` | completed | refused | Agent message | +| Agent 未调用终结工具 | failed | failed | Scheduler 固定协议错误 | +| Provider 失败 | failed | failed | 安全归一化错误，不含响应正文 | +| 运行超时 | timed_out | failed | Scheduler 固定超时说明 | +| Gateway 优雅取消 | interrupted | failed | Scheduler 固定中断说明 | +| 硬崩溃恢复 | unknown | unknown | Scheduler 固定 unknown 说明；关联 AgentRun 审计为 interrupted | +| Agent 定义不存在 | failed | failed | Agent ID 和修复建议 | +| 渠道发送永久失败 | 原 run 不变 | 原 outcome | delivery_status=failed | + +投递失败不改变已经确定的执行 Outcome；它只改变 delivery status。 + +## 19. 实施顺序 + +本设计应在一个功能版本中完成，不能长期保留两套路径： + +1. 抽取 legacy→v10 normalizer，增加 fresh v11/v10→v11 migration、新 Storage 类型与迁移测试； +2. 增加 `ExecutionOrigin::Scheduled`、Scheduled completion sink、终结工具和 AgentLoop 终止语义； +3. 增加 `AgentCoordinator::execute_scheduled()`，落实同步委托、signal/inbox 禁止和顶层 AgentRun； +4. 改写 Scheduler 为非阻塞 JoinSet 事件循环，实现 occurrence claim、跨表恢复、终态提交和持久化 delivery drain； +5. 实现固定 target_session_id、幂等历史插入和类型化 Bus 回执； +6. 切换 Cron tools（含 `cron_runs`）、管理 API、WebUI 与 Health； +7. 更新内置维护任务和文档； +8. 删除所有旧类型、旧函数、旧字段读取和魔法字符串； +9. 运行全量验证后再提交，并按功能变化增加产品中段版本号一次。 + +代码合并点只允许新路径。迁移代码可以先写，但最终提交中不允许 Scheduler 通过 feature flag 或 schema 判断走旧路径。 + +## 20. 测试设计 + +### 单元测试 + +- `DeliveryPolicy × ScheduledOutcome` 全矩阵； +- 终结工具 schema、空 message、额外字段、重复调用； +- 终结工具之后的同批工具归约为 Cancelled； +- 普通文本、所有 `NO_REPLY` 变体都不能产生 `ok`； +- named/root Agent 解析和工具收窄； +- Scheduled origin 继承到多层子 Agent，而 completion sink 只在顶层； +- Scheduled background delegate 在任意嵌套深度自动前台化； +- Scheduled Agent/子 Agent 不注入 emit_signal、不预留 slot、不创建 agent_session_state； +- 两个并发 Scheduler 对同一到期 Job 只有一个 claim 成功； +- At claim 后禁用； +- 新建/更新过去 At 被拒绝，过期 At 无法直接 enable； +- disable 不影响活动 Run，remove 在活动 Run/租约存在时返回 conflict； +- recurring claim 时推进到未来； +- 长 Run 不阻塞其他到期 Job 的 claim 或已完成 Run 的投递； +- lease owner 条件提交； +- 迟到结果不能覆盖 unknown/timeout； +- status/outcome 非法组合被数据库约束拒绝； +- Scheduled 恢复在一个事务中完成 JobRun unknown、AgentRun interrupted、delivery 决策和租约清理； +- delivery claim、瞬态重试、永久失败和尝试上限； +- ChannelError 到类型化 delivery receipt 的完整映射； +- `deliver_outbound` 只有收到 Channel 成功回执才返回 Delivered，入队成功不算送达； +- 回执保留 transient/permanent 分类且不泄露渠道敏感响应； +- 第一次投递后 target_session_id 固定，切换当前 dialog 不改变重试目标； +- 本地历史稳定 message ID 去重，重复调用不重复推进 Session metadata； +- `cron_runs` 摘要/单条详情、limit 边界、Job 所属校验和 read_only 声明。 + +### Migration 测试 + +- 空数据库直接得到 v11； +- fresh v11 与迁移所得 v11 的 `sqlite_master` schema 等价； +- `user_version=0` 的历史数据库先规范化再升级，缺少旧 Scheduler 表时直接创建 v11 Scheduler； +- 真实 v10 fixture 保留所有 Job 和历史 Run； +- `task/monitor` 列被物理删除； +- `direct` 全部转为 `always`，运行时无法解析 direct； +- 非空 model 计数被记录且列被删除； +- delete_after_run 列被删除； +- `last_error` 只补入最近历史 Run diagnostic，v11 Job 表不保留该列； +- 旧 delivery_error 按“执行是否已有结果”分别映射 completed/failed，且 delivery 始终为 failed； +- 迁移不会生成非法的 status/outcome 组合； +- 历史 Run 不产生 pending delivery； +- 旧锁生成 unknown run 并按策略决定 pending/not_requested； +- 内置维护 prompt 不含 `NO_REPLY`； +- 损坏 schedule/policy 使迁移原子失败且 `user_version` 不变； +- `PRAGMA foreign_key_check` 为空； +- v11 重启迁移幂等； +- v12+ 数据库被拒绝。 + +### 集成测试 + +- `always + ok` 实际投递； +- `on_alert + ok` 静默但 JobRun 可查询； +- `on_alert + alert/failed/refused` 实际投递； +- `never` 在所有 Outcome 下均不投递； +- `never` 的完整结果可以通过 `cron_runs` 和管理 API 查询； +- Provider 普通文本完成被转为协议失败通知； +- Gateway 在结果提交后、发送前重启，pending 在启动后恢复； +- Gateway 在发送后、ack 前重启，允许有标识的重复但不丢失； +- Gateway 在 Agent 执行中退出，恢复为 unknown 且不重跑同一 occurrence； +- 恢复后关联 AgentRun 为 interrupted，但任何消费者都不能用它覆盖 JobRun unknown； +- 命名 Agent 被删除后 Job 明确失败且 Gateway 仍能启动； +- Cron 内 foreground 子 Agent 汇总成功，嵌套 background 参数不会产生 Inbox/agent_session_state； +- 投递失败后切换当前 dialog，重试仍写入并指向首次固定的 target_session_id； +- 一个执行到超时上限的 Job 不延迟其他 Job 和 pending delivery。 + +### 必跑验证 + +```bash +cargo test --lib +cargo test --test test_scheduler +cargo test --test test_request_format +cargo clippy --all-targets --all-features -- -D warnings +cd webui && npm run check && npm run build +cargo build +git diff --check +``` + +## 21. 验收标准 + +1. 新建和更新任务的 API/工具中没有 `kind`、`monitor` 或 `model` 字段。 +2. 数据库 `scheduled_jobs` 不存在 `job_kind`、`model`、`delete_after_run`。 +3. `DeliveryPolicy` 只有 `always/on_alert/never`。 +4. 所有 Scheduled Run 都必须调用 `complete_scheduled_run`；无调用时 fail-closed。 +5. 全仓库运行时代码不再出现 `NO_REPLY` 结果协议。 +6. Scheduler 只有一条 Agent 执行和一条中央投递路径。 +7. `on_alert + ok` 不投递，其余矩阵行为与本设计一致。 +8. 结果提交后崩溃不会丢失待投递消息。 +9. 执行中崩溃产生 `unknown`，同一 occurrence 不自动重跑。 +10. Scheduled 恢复原子地产生 JobRun unknown 和 AgentRun interrupted，JobRun 始终是投递权威。 +11. Scheduled origin 贯穿所有后代，但 completion sink 只属于顶层；数据库中不产生合成 agent_session_state 或 Inbox event。 +12. Scheduler 的长 Run 不阻塞其他到期 Job 或 pending delivery。 +13. 过期 At 不能直接新建、更新或重新启用。 +14. 每次投递重试复用固定 target_session_id 和 `scheduled:` message ID，本地历史和 Session metadata 均不重复。 +15. `cron_runs`、管理 API 和 WebUI 都能查询 `never` 等静默任务的结构化结果。 +16. v10 数据库首次启动自动迁移到 v11，失败时完整回滚；fresh/migrated v11 schema 一致，迁移后没有运行时兼容代码。 diff --git a/resources/skills/about-picobot/assets/config.example.json b/resources/skills/about-picobot/assets/config.example.json index 31ad56e..f734c3b 100644 --- a/resources/skills/about-picobot/assets/config.example.json +++ b/resources/skills/about-picobot/assets/config.example.json @@ -54,7 +54,13 @@ "gateway": { "host": "127.0.0.1", "port": 19876, - "require_pairing": true + "require_pairing": true, + "scheduler": { + "enabled": true, + "poll_interval_secs": 60, + "max_concurrent": 1, + "execution_timeout_secs": 900 + } }, "client": { "gateway_url": "ws://127.0.0.1:19876/ws" diff --git a/resources/skills/about-picobot/references/architecture.md b/resources/skills/about-picobot/references/architecture.md index 6610143..186f586 100644 --- a/resources/skills/about-picobot/references/architecture.md +++ b/resources/skills/about-picobot/references/architecture.md @@ -10,7 +10,7 @@ Channel → MessageBus.inbound → Gateway processor → SessionManager → per- AgentLoop → TurnEvent → TurnController → latest TurnSnapshot → DeliveryCoordinator → TurnSink → Channel WebSocket/Channel → MessageBus.control → Gateway processor → SessionManager (dialog 操作) -Scheduler → SessionManager.handle_cron_message → AgentLoop → send_message +Scheduler → occurrence/JobRun claim → AgentCoordinator → isolated Scheduled Agent → complete_scheduled_run → JobRun outbox → SessionManager/MessageBus ``` ## 模块职责 diff --git a/resources/skills/about-picobot/references/config.md b/resources/skills/about-picobot/references/config.md index 092becc..0263e6c 100644 --- a/resources/skills/about-picobot/references/config.md +++ b/resources/skills/about-picobot/references/config.md @@ -102,8 +102,8 @@ Gateway WebUI 的“配置”页可以编辑实际加载的配置文件。读取 |------|------|------|------| | `enabled` | bool | true | 是否启动调度器并注册 cron 工具 | | `poll_interval_secs` | int | 60 | 检查到期任务的轮询间隔 | -| `max_concurrent` | int | 1 | 每批到期任务的最大并发数，运行时限制在 1–256 | -| `execution_timeout_secs` | int | 900 | 单个定时任务 Agent 执行的硬超时；租约会覆盖执行和托管投递等待 | +| `max_concurrent` | int | 1 | 同时执行的 Scheduled Run 上限，运行时限制在 1–256；投递使用独立有界并发 | +| `execution_timeout_secs` | int | 900 | 单个定时任务 Agent 执行的硬超时；Job 执行租约额外覆盖关停宽限，投递由持久化 outbox 独立恢复 | ## memory 字段 @@ -116,7 +116,7 @@ Gateway WebUI 的“配置”页可以编辑实际加载的配置文件。读取 | `timeline_retention_days` | int | 90 | 默认日常维护巡检删除超过该期限的 Timeline；Knowledge 不受影响 | | `max_failures_before_degrade` | int | 3 | 预留的归并失败阈值；当前无失败降级循环 | -注意：当前 worker 的 Knowledge 召回数量仍固定为 5；idle consolidation 和失败降级循环尚未接入。Timeline 清理由默认启用的 `picobot-routine-maintenance` 定时巡检执行。 +注意：当前 worker 的 Knowledge 召回数量仍固定为 5；idle consolidation 和失败降级循环尚未接入。Timeline 清理由默认启用的 `picobot-routine-maintenance` Scheduled Run 执行；该任务使用 `never` 策略，结构化结果只进入运行审计和 Health。 ## channels.feishu 字段 diff --git a/resources/skills/about-picobot/references/db-schema.md b/resources/skills/about-picobot/references/db-schema.md index 0b8789c..9a1342a 100644 --- a/resources/skills/about-picobot/references/db-schema.md +++ b/resources/skills/about-picobot/references/db-schema.md @@ -2,7 +2,7 @@ 数据库为 SQLite，默认位于配置目录（`~/.picobot`）`data/` 下的 `picobot.db`，与 workspace 相互独立。 -连接启用 WAL、`synchronous=NORMAL`、foreign keys、5 秒 busy timeout，连接池最多 8 个连接。当前 `PRAGMA user_version=10`；启动时会在事务内补齐旧库字段和索引，遇到比程序更新的 schema version 会拒绝启动。 +连接启用 WAL、`synchronous=NORMAL`、foreign keys、5 秒 busy timeout，连接池最多 8 个连接。当前 `PRAGMA user_version=11`；启动时会在单个事务内迁移旧库，遇到比程序更新的 schema version 会拒绝启动。 ## sessions 表 @@ -176,22 +176,21 @@ background 完成/信号投递的唯一事实源：`pending → leased → admit | `name` | TEXT | 任务名称 | | `schedule` | TEXT | 调度规则 JSON（at/every/cron） | | `prompt` | TEXT | 任务提示词 | -| `channel` | TEXT | 执行渠道 | +| `agent_id` | TEXT | 可选命名 Agent；NULL 表示 Root | +| `channel` | TEXT | 目标渠道 | | `chat_id` | TEXT | 目标对话 | -| `model` | TEXT | 可选模型标记；当前会存储/展示，但 Scheduler 执行仍使用默认 Agent 模型 | +| `delivery_policy` | TEXT | `always` / `on_alert` / `never` | | `enabled` | INTEGER | 是否启用 (1/0) | -| `delete_after_run` | INTEGER | 执行后自动删除 (1/0) | | `next_run_at` | INTEGER | 下次执行时间 | | `last_run_at` | INTEGER | 上次执行时间 | -| `last_status` | TEXT | 上次执行状态 | -| `last_error` | TEXT | 上次错误信息 | +| `last_outcome` | TEXT | 最近结构化结果：ok/alert/failed/refused/unknown | | `locked_at` | INTEGER | 本次领取时间 | -| `lock_owner` | TEXT | 领取任务的 Scheduler owner UUID | -| `lease_until` | INTEGER | 租约到期时间；进程崩溃后允许其他实例重新领取 | +| `lock_owner` | TEXT | 本次 occurrence 的唯一 owner token | +| `lease_until` | INTEGER | 租约到期时间 | | `created_at` | INTEGER | 创建时间（Unix 毫秒） | | `updated_at` | INTEGER | 更新时间（Unix 毫秒） | -Scheduler 使用原子 `UPDATE ... RETURNING` 领取到期任务。任务结果、下次运行时间和租约释放在同一事务中提交，并校验 owner，防止过期 worker 覆盖已恢复的任务。 +Scheduler 在领取事务中插入 JobRun、快照执行/投递字段并推进下次时间。`At` 在领取时立即禁用；执行失败或崩溃不重放同一个 occurrence。 ## job_runs 表 @@ -199,12 +198,26 @@ Scheduler 使用原子 `UPDATE ... RETURNING` 领取到期任务。任务结果 |------|------|------| | `id` | INTEGER PK | 自增 ID | | `job_id` | TEXT FK | 关联任务，外键关联 scheduled_jobs(id) | +| `scheduled_for` | INTEGER | 本 occurrence 原计划时间 | +| `agent_run_id` | TEXT FK | 顶层 AgentRun 审计记录 | +| `agent_id` | TEXT | claim-time Agent 快照 | +| `delivery_policy` | TEXT | claim-time 投递策略快照 | +| `target_channel` / `target_chat_id` | TEXT | claim-time 目标快照 | +| `target_session_id` | TEXT | 首次投递时固定的目标 dialog | | `started_at` | INTEGER | 开始时间 | | `finished_at` | INTEGER | 结束时间 | -| `status` | TEXT | 执行状态 | -| `output` | TEXT | 执行输出 | -| `error` | TEXT | 错误信息 | +| `status` | TEXT | claimed/running/completed/failed/timed_out/cancelled/interrupted/unknown | +| `outcome` | TEXT | ok/alert/failed/refused/unknown；与 status 有联合约束 | +| `message` | TEXT | 面向用户的结构化结果 | +| `diagnostic` | TEXT | 有界内部诊断 | | `duration_ms` | INTEGER | 耗时（毫秒） | +| `delivery_status` | TEXT | awaiting_result/not_requested/suppressed/pending/delivering/delivered/failed | +| `delivery_attempts` | INTEGER | 持久化投递尝试次数，最多 3 次 | +| `delivery_next_attempt_at` | INTEGER | 瞬态失败后的退避时间 | +| `delivery_lease_owner` / `delivery_lease_until` | TEXT / INTEGER | outbox 领取租约 | +| `delivery_error` | TEXT | 清洗后的投递失败摘要 | + +JobRun 是执行结果和投递状态的唯一权威。顶层 AgentRun 与 JobRun 终态、Job 最近摘要和租约释放原子提交；启动恢复将遗留运行归为 `unknown`，不会自动重跑。 ## llm_calls 表 diff --git a/resources/skills/about-picobot/references/tools.md b/resources/skills/about-picobot/references/tools.md index 52da78c..0b9ebab 100644 --- a/resources/skills/about-picobot/references/tools.md +++ b/resources/skills/about-picobot/references/tools.md @@ -50,14 +50,15 @@ ## Cron 定时任务工具 -Cron 不是一个带 `action` 的统一工具，而是六个独立工具；仅在 `gateway.scheduler.enabled=true` 时注册。 +Cron 不是一个带 `action` 的统一工具，而是七个独立工具；仅在 `gateway.scheduler.enabled=true` 时注册。 | 工具 | 主要参数 | 说明 | |------|----------|------| -| `cron_add` | `schedule`, `prompt`, `channel`, `chat_id`; 可选 `name`, `model` | 创建任务 | +| `cron_add` | `schedule`, `prompt`, `channel`, `chat_id`; 可选 `name`, `agent_id`, `delivery_policy` | 创建任务 | | `cron_list` | 可选 `status=all|enabled|disabled` | 列出任务 | -| `cron_update` | `job_id`; 可选 `prompt`, `schedule`, `channel`, `chat_id`, `model` | 更新指定字段 | -| `cron_remove` | `job_id` | 永久删除任务和关联 job runs | +| `cron_runs` | `job_id`; 可选 `run_id`, `limit` | 查询结构化运行和投递记录，包括静默结果 | +| `cron_update` | `job_id`; 可选 `name`, `prompt`, `schedule`, `channel`, `chat_id`, `agent_id`, `delivery_policy` | 更新指定字段；`agent_id:null` 切回 Root | +| `cron_remove` | `job_id` | 无活动 Run 或 pending delivery 时永久删除任务 | | `cron_enable` | `job_id` | 启用并重新计算下次运行时间 | | `cron_disable` | `job_id` | 禁用但保留任务 | @@ -69,7 +70,9 @@ Cron 不是一个带 `action` 的统一工具，而是六个独立工具；仅 {"type":"cron","expr":"0 0 9 * * *","tz":"Asia/Shanghai"} ``` -时间戳和间隔单位为毫秒；Cron 表达式为 6 段（秒、分、时、日、月、周）。定时 Agent 不复用聊天历史，`prompt` 必须包含完整上下文。`kind` 可为 `task` 或 `monitor`；`delivery_policy` 可为 `always`、`on_alert` 或 `never`。托管任务由 Scheduler 投递，巡检返回 `NO_REPLY[INFO]` 时静默，`NO_REPLY[FAIL]`/`NO_REPLY[REFUSE]` 仍视为需关注结果。升级前创建的任务保留 Agent 直接调用 `send_message` 的兼容行为。`model` 当前会持久化和展示，但执行仍使用默认 Agent Provider/Model，不能依赖它实现模型覆盖。 +时间戳和间隔单位为毫秒；Cron 表达式为 6 段（秒、分、时、日、月、周）。过去时间的 At 不能创建、更新或直接重新启用。定时 Agent 不复用聊天历史，`prompt` 必须包含完整上下文；`agent_id` 省略时使用 Root，否则使用当前 AgentCatalog 中的命名 Agent。 + +每次运行必须恰好一次调用 `complete_scheduled_run(outcome,message)`，outcome 只能是 `ok`、`alert`、`failed`、`refused`。普通最终文本不会被解释为结果，缺少结构化终结会 fail-closed。投递完全由 Scheduler 决定：`always` 投递所有结果，`on_alert` 只抑制 `ok`，`never` 只保留记录。Scheduled Agent 不能自行发送最终通知；子 Agent 委托会同步完成，也不会产生后台 Inbox/Signal。 --- diff --git a/resources/templates/config.example.json b/resources/templates/config.example.json index a7419c7..0566a4f 100644 --- a/resources/templates/config.example.json +++ b/resources/templates/config.example.json @@ -78,6 +78,12 @@ "max_files_per_message": 8, "max_message_bytes": 67108864, "pending_ttl_seconds": 3600 + }, + "scheduler": { + "enabled": true, + "poll_interval_secs": 60, + "max_concurrent": 1, + "execution_timeout_secs": 900 } }, "client": { diff --git a/src/agent/agent_loop.rs b/src/agent/agent_loop.rs index 1db0a5a..2c81166 100644 --- a/src/agent/agent_loop.rs +++ b/src/agent/agent_loop.rs @@ -1240,6 +1240,31 @@ impl AgentLoop { } completed_tool_batches = completed_tool_batches.saturating_add(1); + if let Some(outcome) = tool_context + .scheduled_completion + .as_ref() + .and_then(|sink| sink.outcome()) + { + Self::close_steering(turn.as_ref()); + let mut final_message = ChatMessage::assistant(outcome.message); + attach_reply_media(&mut final_message, &reply_media_refs); + Self::annotate_message(&mut final_message, turn.as_ref(), iteration, true); + emitted_messages.push(final_message.clone()); + self.forward_to_transcript_sink(&final_message); + crate::observability::metrics::global_metrics().record_turn( + Some(&accumulated_usage), + turn_start.elapsed().as_millis() as u64, + ); + return Ok(AgentProcessResult { + final_response: final_message, + emitted_messages, + total_tokens: Some(accumulated_tokens), + usage: Some(accumulated_usage), + last_request_usage, + last_request_digest, + }); + } + // A complete tool batch is the first safe steering boundary. Do // not drain at the final available iteration: those inputs must // remain in the closed mailbox for Session to queue after this @@ -1517,6 +1542,17 @@ impl AgentLoop { let mut outcomes = Vec::with_capacity(tool_calls.len()); for tool_call in tool_calls { + if context + .scheduled_completion + .as_ref() + .is_some_and(|sink| sink.is_completed()) + { + outcomes.push(ToolExecutionOutcome::failure( + "Cancelled: scheduled run was already completed".to_string(), + Some("scheduled run was already completed".to_string()), + )); + continue; + } if context.cancellation.is_cancelled() { return Err(AgentError::Cancelled); } @@ -1666,6 +1702,58 @@ mod tests { requests: std::sync::atomic::AtomicUsize, } + struct ScheduledCompletionProvider { + requests: std::sync::atomic::AtomicUsize, + } + + #[async_trait::async_trait] + impl LLMProvider for ScheduledCompletionProvider { + async fn stream( + &self, + _request: ChatCompletionRequest, + ) -> Result { + self.requests + .fetch_add(1, std::sync::atomic::Ordering::SeqCst); + Ok(crate::providers::provider_stream_for_test( + ChatCompletionResponse { + id: "scheduled-complete".to_string(), + model: "scheduled-complete".to_string(), + content: String::new(), + reasoning_content: None, + provider_state: None, + tool_calls: vec![ + ToolCall { + id: "complete".to_string(), + name: "complete_scheduled_run".to_string(), + arguments: serde_json::json!({ + "outcome": "ok", + "message": "healthy" + }), + }, + ToolCall { + id: "late-side-effect".to_string(), + name: "side_effect".to_string(), + arguments: serde_json::json!({}), + }, + ], + usage: Usage::default(), + }, + )) + } + + fn ptype(&self) -> &str { + "test" + } + + fn name(&self) -> &str { + "scheduled-complete" + } + + fn model_id(&self) -> &str { + "scheduled-complete" + } + } + #[async_trait::async_trait] impl LLMProvider for AlwaysOverflowProvider { async fn stream( @@ -1725,6 +1813,53 @@ mod tests { } } + #[tokio::test] + async fn scheduled_completion_ends_the_loop_and_cancels_later_batch_calls() { + let provider = Arc::new(ScheduledCompletionProvider { + requests: std::sync::atomic::AtomicUsize::new(0), + }); + let executions = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let tools = Arc::new(ToolRegistry::new()); + tools.register(crate::tools::CompleteScheduledRunTool::new()); + tools.register(CountingSideEffectTool { + executions: executions.clone(), + }); + let agent = AgentLoop::with_provider_and_tools( + provider.clone(), + tools, + 3, + "scheduled-complete".to_string(), + PathBuf::from("."), + Vec::new(), + ); + let sink = Arc::new(crate::tools::ScheduledCompletionSink::default()); + let context = ToolExecutionContext::for_session("scheduled-run:1") + .with_execution_origin(crate::tools::ExecutionOrigin::Scheduled { job_run_id: 1 }) + .with_scheduled_completion(sink.clone()); + + let result = agent + .process_with_context(vec![ChatMessage::user("check")], context) + .await + .unwrap(); + + assert_eq!(result.final_response.content, "healthy"); + assert_eq!( + sink.outcome().unwrap().kind, + crate::storage::ScheduledOutcomeKind::Ok + ); + assert_eq!(executions.load(std::sync::atomic::Ordering::SeqCst), 0); + assert_eq!( + provider.requests.load(std::sync::atomic::Ordering::SeqCst), + 1 + ); + assert!(result.emitted_messages.iter().any(|message| { + message.role == "tool" + && message + .content + .contains("scheduled run was already completed") + })); + } + #[async_trait::async_trait] impl LLMProvider for OverflowAfterToolProvider { async fn stream( diff --git a/src/agent/coordinator.rs b/src/agent/coordinator.rs index 7d3a559..e0cdcdd 100644 --- a/src/agent/coordinator.rs +++ b/src/agent/coordinator.rs @@ -32,6 +32,16 @@ pub struct BackgroundAdmission { pub run_ids: Vec, } +#[derive(Debug, Clone)] +pub struct ScheduledAgentExecution { + pub agent_run_id: String, + pub status: crate::storage::ScheduledRunStatus, + pub outcome: Option, + pub error: Option, + pub agent_terminal: AgentTerminalOutcome, + pub runtime_generation: i64, +} + pub struct AgentCoordinator { storage: Arc, manager: Arc, @@ -88,6 +98,263 @@ impl AgentCoordinator { }) } + #[allow(clippy::too_many_arguments)] + pub async fn execute_scheduled( + self: &Arc, + job_run_id: i64, + lease_owner: &str, + job_id: &str, + job_name: &str, + agent_id: Option<&str>, + prompt: &str, + timeout_secs: u64, + ) -> Result { + let run_id = Uuid::new_v4().to_string(); + let root_session_id = format!("scheduled-run:{job_run_id}"); + let sink = Arc::new(crate::tools::ScheduledCompletionSink::default()); + let caller = ToolExecutionContext::for_session(root_session_id.clone()) + .with_turn_id(format!("scheduled:{job_run_id}")) + .with_execution_origin(crate::tools::ExecutionOrigin::Scheduled { job_run_id }); + let contract = format!( + "## Unattended Scheduled Run\n\nYou are executing scheduled task “{job_name}” ({job_id}). The user will not see ordinary final text. After completing all necessary work, you must call complete_scheduled_run exactly once. Use ok only when the task completed and found nothing requiring attention; use alert for actionable findings; use failed when the task did not complete reliably; use refused for a permission or safety refusal. Legacy textual suppression and direct-messaging instructions are obsolete." + ); + let config = SubAgentConfig { + target: agent_id.map(str::to_string), + prompt: prompt.to_string(), + context: Some(contract.clone()), + mode: ExecutionMode::Foreground, + allowed_tools: None, + max_iterations: None, + timeout_secs: Some(timeout_secs), + plan_item_id: None, + session_id: Some(root_session_id.clone()), + }; + let mut resolution = if agent_id.is_some() { + let mut resolution = self.manager.resolve_agent(&config, &caller, &run_id)?; + resolution + .tools + .register(crate::tools::CompleteScheduledRunTool::new()); + resolution.tool_context.session_id = Some(root_session_id.clone()); + resolution.tool_context.scheduled_completion = Some(sink.clone()); + resolution.timeout_secs = resolution.timeout_secs.min(timeout_secs); + resolution.signal_contract = None; + resolution + } else { + self.manager + .resolve_scheduled_root(&caller, &run_id, timeout_secs, sink.clone())? + }; + resolution.tool_context.execution_origin = + crate::tools::ExecutionOrigin::Scheduled { job_run_id }; + + let now = chrono::Utc::now().timestamp_millis(); + let deadline_at = now.saturating_add((resolution.timeout_secs * 1000) as i64); + let new_run = NewAgentRun { + id: run_id.clone(), + root_session_id, + root_turn_id: None, + parent_run_id: None, + caller_agent_id: "SCHEDULER".to_string(), + caller_scope_id: format!("scheduled:{job_id}"), + idempotency_key: Some(format!("scheduled:{job_run_id}")), + agent_id: agent_id.unwrap_or("ROOT").to_string(), + definition_hash: resolution.definition_hash.clone().unwrap_or_default(), + provider_profile: resolution.llm_profile.clone().unwrap_or_default(), + provider_name: resolution.provider_config.name.clone(), + model_id: resolution.provider_config.model_id.clone(), + mode: AgentRunMode::Foreground, + depth: 1, + plan_item_id: None, + execution_id: run_id.clone(), + task: prompt.to_string(), + context_json: None, + budget_json: serde_json::json!({ + "remaining_runs": self.manager.catalog().max_runs_per_tree().saturating_sub(1), + "remaining_depth": self.manager.catalog().max_tree_depth().saturating_sub(1), + }) + .to_string(), + signal_contract_json: None, + signal_delivery: None, + deadline_at, + runtime_generation: self.runtime_generation, + completion_slot_reserved: false, + }; + match self + .storage + .accept_agent_runs(AcceptAgentRequest { + runs: vec![new_run], + now, + }) + .await? + { + AcceptedAgentRuns::Accepted { .. } => {} + AcceptedAgentRuns::Existing { .. } => { + return Err(CoordinatorError::Rejected(format!( + "scheduled occurrence {job_run_id} already has an Agent run" + ))); + } + } + let mut mark_attempt = 0_u64; + let marked = match loop { + match self + .storage + .mark_scheduled_run_running(job_run_id, lease_owner, Some(&run_id), now) + .await + { + Err(error) if error.is_transient() && mark_attempt < 2 => { + mark_attempt += 1; + tokio::time::sleep(std::time::Duration::from_millis(50 * mark_attempt)).await; + } + result => break result, + } + } { + Ok(marked) => marked, + Err(error) => { + let _ = self + .storage + .cancel_agent_run_with_completion( + &run_id, + "scheduled occurrence could not enter running state", + true, + now, + ) + .await; + return Err(error.into()); + } + }; + if !marked { + let _ = self + .storage + .cancel_agent_run_with_completion( + &run_id, + "scheduled occurrence was no longer active", + true, + now, + ) + .await; + return Err(CoordinatorError::Rejected(format!( + "scheduled occurrence {job_run_id} lost its lease" + ))); + } + + let (execution, agent_terminal) = match self + .execute_scheduled_agent_run(&run_id, &config, resolution) + .await + { + Ok(execution) => execution, + Err(error) => ( + Err(error), + AgentTerminalOutcome::Failed { + error: "scheduled Agent could not enter its execution lifecycle".to_string(), + prompt_tokens: None, + completion_tokens: None, + cost: None, + signal_ids: Vec::new(), + }, + ), + }; + let status = match &execution { + Ok(result) => match &result.status { + TaskStatus::Completed => crate::storage::ScheduledRunStatus::Completed, + TaskStatus::Failed(_) => crate::storage::ScheduledRunStatus::Failed, + TaskStatus::TimedOut => crate::storage::ScheduledRunStatus::TimedOut, + TaskStatus::Cancelled => crate::storage::ScheduledRunStatus::Interrupted, + }, + Err(_) => crate::storage::ScheduledRunStatus::Failed, + }; + let error = match &execution { + Ok(result) => match &result.status { + TaskStatus::Completed => None, + TaskStatus::Failed(_) => Some( + "scheduled Agent execution failed; inspect Gateway logs for details" + .to_string(), + ), + TaskStatus::TimedOut => Some("scheduled Agent timed out".to_string()), + TaskStatus::Cancelled => Some("scheduled Agent was cancelled".to_string()), + }, + Err(_) => Some( + "scheduled Agent execution failed; inspect Gateway logs for details".to_string(), + ), + }; + Ok(ScheduledAgentExecution { + agent_run_id: run_id, + status, + outcome: sink.outcome(), + error, + agent_terminal, + runtime_generation: self.runtime_generation, + }) + } + + async fn execute_scheduled_agent_run( + self: &Arc, + run_id: &str, + config: &SubAgentConfig, + resolution: ResolvedAgentRun, + ) -> Result< + ( + Result, + AgentTerminalOutcome, + ), + CoordinatorError, + > { + let execution_id = run_id.to_string(); + let token = resolution.tool_context.cancellation.clone(); + self.active_tokens.insert(run_id.to_string(), token); + let started = self + .storage + .mark_agent_run_running(run_id, &execution_id, chrono::Utc::now().timestamp_millis()) + .await?; + if !started { + self.active_tokens.remove(run_id); + return Err(CoordinatorError::Rejected(format!( + "scheduled Agent run {run_id} was closed before execution started" + ))); + } + + let result = self + .manager + .execute_resolved(config, resolution, run_id) + .await + .map_err(CoordinatorError::SubAgent); + self.active_tokens.remove(run_id); + let terminal = match &result { + Ok(result) => match &result.status { + TaskStatus::Completed => AgentTerminalOutcome::Completed { + result: result.full_content.clone(), + prompt_tokens: None, + completion_tokens: None, + cost: None, + tool_calls: result.tool_calls_count as i64, + iterations: result.iterations as i64, + signal_ids: Vec::new(), + }, + TaskStatus::Failed(error) => AgentTerminalOutcome::Failed { + error: error.clone(), + prompt_tokens: None, + completion_tokens: None, + cost: None, + signal_ids: Vec::new(), + }, + TaskStatus::TimedOut => AgentTerminalOutcome::TimedOut { + deadline_at: chrono::Utc::now().timestamp_millis(), + signal_ids: Vec::new(), + }, + TaskStatus::Cancelled => AgentTerminalOutcome::Interrupted { + reason: "scheduled Agent interrupted by shutdown".to_string(), + signal_ids: Vec::new(), + }, + }, + Err(error) => AgentTerminalOutcome::Failed { + error: error.to_string(), + prompt_tokens: None, + completion_tokens: None, + cost: None, + signal_ids: Vec::new(), + }, + }; + Ok((result, terminal)) + } + /// Admit a named background run for the root caller and spawn its runner. /// Completion is guaranteed by the reserved inbox slot; the returned ID /// is only valid when every durable step succeeded. @@ -100,6 +367,11 @@ impl AgentCoordinator { caller: &ToolExecutionContext, configs: Vec, ) -> Result { + if caller.execution_origin.is_scheduled() { + return Err(CoordinatorError::Rejected( + "scheduled Agents cannot create background runs".to_string(), + )); + } if caller.agent.is_some() { return Err(CoordinatorError::Rejected( "nested background runs are not available yet; only the root Agent may delegate background work".to_string(), diff --git a/src/agent/mod.rs b/src/agent/mod.rs index 16929b6..e9c4007 100644 --- a/src/agent/mod.rs +++ b/src/agent/mod.rs @@ -21,7 +21,7 @@ pub use context_compaction::{ ContextRequestKey, ContextUsageTracker, PreviousCheckpoint, SequencedMessage, context_request_digest, estimate_tokens, }; -pub use coordinator::{AgentCoordinator, CoordinatorError}; +pub use coordinator::{AgentCoordinator, CoordinatorError, ScheduledAgentExecution}; pub use definition::{AgentDefinition, AgentLimits}; pub use gate::ExecutionGate; pub use inbox::{AgentInboxNotifier, AgentInboxWakeTarget}; diff --git a/src/agent/sub_agent.rs b/src/agent/sub_agent.rs index 500b825..a7dfeb1 100644 --- a/src/agent/sub_agent.rs +++ b/src/agent/sub_agent.rs @@ -10,6 +10,22 @@ use crate::providers::{LLMProvider, create_provider}; use crate::skills::SkillsLoader; use crate::tools::{ToolExecutionContext, ToolRegistry}; +const SCHEDULED_DISABLED_TOOLS: &[&str] = &[ + "send_message", + "cron_add", + "cron_update", + "cron_remove", + "cron_enable", + "cron_disable", + "cron_list", + "cron_runs", + "reload_config", + "agent_task", + "chat_manager", + "todo", + "emit_signal", +]; + const DEFAULT_MAX_ITERATIONS: usize = 99; #[derive(Debug, Clone)] @@ -191,7 +207,13 @@ impl SubAgentManager { })?; let cancellation = caller.cancellation.child_token(); let execution = if let Some(parent) = caller.agent.as_ref() { - if !self.catalog.can_delegate(&parent.current_agent_id, target) { + let allowed = + if caller.execution_origin.is_scheduled() && parent.current_agent_id == "ROOT" { + self.catalog.root_can_delegate(target) + } else { + self.catalog.can_delegate(&parent.current_agent_id, target) + }; + if !allowed { return Err(SubAgentError::Other(format!( "Agent '{}' is not allowed to delegate to '{target}'", parent.current_agent_id @@ -235,10 +257,14 @@ impl SubAgentManager { .budget .remaining_depth .min(definition.limits.max_depth); - child.signal_contract = definition - .signal_contract - .as_ref() - .map(|contract| Arc::new(contract.clone())); + child.signal_contract = if caller.execution_origin.is_scheduled() { + None + } else { + definition + .signal_contract + .as_ref() + .map(|contract| Arc::new(contract.clone())) + }; Arc::new(child) } else { if !self.catalog.root_can_delegate(target) { @@ -252,7 +278,11 @@ impl SubAgentManager { run_id: task_id.to_string(), execution_id: task_id.to_string(), parent_run_id: None, - caller_agent_id: "ROOT".to_string(), + caller_agent_id: if caller.execution_origin.is_scheduled() { + "SCHEDULER".to_string() + } else { + "ROOT".to_string() + }, current_agent_id: target.to_string(), ancestry: vec![target.to_string()], depth: 1, @@ -267,10 +297,14 @@ impl SubAgentManager { .min(definition.limits.max_depth), }, tree_runs: Arc::new(std::sync::atomic::AtomicUsize::new(1)), - signal_contract: definition - .signal_contract - .as_ref() - .map(|contract| Arc::new(contract.clone())), + signal_contract: if caller.execution_origin.is_scheduled() { + None + } else { + definition + .signal_contract + .as_ref() + .map(|contract| Arc::new(contract.clone())) + }, emitted_signals: Arc::new(std::sync::Mutex::new(Vec::new())), }) }; @@ -279,6 +313,9 @@ impl SubAgentManager { if let Some(allowed) = config.allowed_tools.as_ref() { effective_names.retain(|name| allowed.iter().any(|allowed| allowed == name)); } + if caller.execution_origin.is_scheduled() { + effective_names.retain(|name| !SCHEDULED_DISABLED_TOOLS.contains(&name.as_str())); + } let has_get_skill = effective_names.iter().any(|name| name == "get_skill"); let mut names = effective_names; names.retain(|name| name != "get_skill"); @@ -310,7 +347,7 @@ impl SubAgentManager { // The signal tool is contract-bound: it exists only when the // definition declares a signal block and the durable Coordinator is // live. If either is missing the run cannot emit signals. - if definition.signal_contract.is_some() { + if definition.signal_contract.is_some() && !caller.execution_origin.is_scheduled() { match self.coordinator() { Some(coordinator) => { let contract = definition.signal_contract.clone().unwrap(); @@ -343,17 +380,91 @@ impl SubAgentManager { agent_id: Some(target.to_string()), definition_hash: Some(definition.definition_hash.clone()), llm_profile: definition.llm_profile.clone(), - signal_contract: definition.signal_contract.clone(), - tool_context: ToolExecutionContext::for_session(format!("agent-run:{task_id}")) - .with_turn_id( - caller - .turn_id - .clone() - .unwrap_or_else(|| task_id.to_string()), - ) + signal_contract: (!caller.execution_origin.is_scheduled()) + .then(|| definition.signal_contract.clone()) + .flatten(), + tool_context: ToolExecutionContext::for_session( + if caller.execution_origin.is_scheduled() { + root_session_id + } else { + format!("agent-run:{task_id}") + }, + ) + .with_turn_id( + caller + .turn_id + .clone() + .unwrap_or_else(|| task_id.to_string()), + ) + .with_agent(execution) + .with_cancellation(cancellation) + .with_execution_gate(self.execution_gate.clone()) + .with_execution_origin(caller.execution_origin), + }) + } + + pub(crate) fn resolve_scheduled_root( + &self, + caller: &ToolExecutionContext, + task_id: &str, + timeout_secs: u64, + sink: Arc, + ) -> Result { + if !caller.execution_origin.is_scheduled() { + return Err(SubAgentError::Other( + "scheduled root resolution requires Scheduled execution origin".to_string(), + )); + } + let root_session_id = caller.session_id.clone().ok_or_else(|| { + SubAgentError::Other("scheduled root requires an execution scope".to_string()) + })?; + let run_id = task_id.to_string(); + let cancellation = caller.cancellation.child_token(); + let execution = Arc::new(crate::agent::AgentExecutionContext { + root_session_id: root_session_id.clone(), + root_turn_id: None, + run_id: run_id.clone(), + execution_id: run_id, + parent_run_id: None, + caller_agent_id: "SCHEDULER".to_string(), + current_agent_id: "ROOT".to_string(), + ancestry: vec!["ROOT".to_string()], + depth: 1, + plan_item_id: None, + cancellation: cancellation.clone(), + budget: crate::agent::AgentBudget { + remaining_runs: self.catalog.max_runs_per_tree().saturating_sub(1), + remaining_depth: self.catalog.max_tree_depth().saturating_sub(1), + }, + tree_runs: Arc::new(std::sync::atomic::AtomicUsize::new(1)), + signal_contract: None, + emitted_signals: Arc::new(std::sync::Mutex::new(Vec::new())), + }); + let tools = self.full_tools.without(SCHEDULED_DISABLED_TOOLS); + tools.register(crate::tools::CompleteScheduledRunTool::new()); + let skills_prompt = self + .skills_loader + .as_ref() + .map(|loader| loader.build_skills_prompt()) + .filter(|prompt| !prompt.is_empty()); + Ok(ResolvedAgentRun { + provider_config: Arc::new(self.provider_config.clone()), + tools, + timeout_secs, + max_iterations: self.provider_config.max_tool_iterations, + max_result_chars: 16_384, + role_prompt: None, + skills_prompt, + tool_context: ToolExecutionContext::for_session(root_session_id) .with_agent(execution) .with_cancellation(cancellation) - .with_execution_gate(self.execution_gate.clone()), + .with_execution_gate(self.execution_gate.clone()) + .with_execution_origin(caller.execution_origin) + .with_scheduled_completion(sink), + agent_id: None, + definition_hash: None, + llm_profile: None, + signal_contract: None, }) } @@ -664,6 +775,25 @@ mod tests { assert!(!crate::tools::Tool::runtime_injected(&reload)); } + #[test] + fn scheduled_runs_remove_direct_and_interactive_control_tools() { + for name in [ + "send_message", + "cron_add", + "cron_runs", + "reload_config", + "agent_task", + "chat_manager", + "todo", + "emit_signal", + ] { + assert!(SCHEDULED_DISABLED_TOOLS.contains(&name)); + } + assert!(!SCHEDULED_DISABLED_TOOLS.contains(&"bash")); + assert!(!SCHEDULED_DISABLED_TOOLS.contains(&"delegate")); + assert!(!SCHEDULED_DISABLED_TOOLS.contains(&"complete_scheduled_run")); + } + #[test] fn resolve_agent_rejects_missing_target() { let manager = manager(); diff --git a/src/bus/dispatcher.rs b/src/bus/dispatcher.rs index 0ae9db2..8dd62be 100644 --- a/src/bus/dispatcher.rs +++ b/src/bus/dispatcher.rs @@ -5,7 +5,7 @@ use std::time::Duration; use tokio::sync::mpsc; -use crate::bus::{MessageBus, OutboundMessage}; +use crate::bus::{DeliveryReceipt, MessageBus, OutboundMessage}; use crate::channels::ChannelManager; use crate::channels::base::{Channel, ChannelError}; use crate::delivery::ConversationWriteLocks; @@ -64,12 +64,14 @@ impl OutboundDispatcher { if sender.as_ref().is_none_or(mpsc::Sender::is_closed) { let Some(channel) = self.channel_manager.get_channel(&msg.channel).await else { tracing::warn!(channel = %msg.channel, "No channel found for message"); - msg.complete_delivery(Err(format!("channel not found: {}", msg.channel))); + msg.complete_delivery(DeliveryReceipt::PermanentFailure { + summary: format!("channel not found: {}", msg.channel), + }); continue; }; let (new_sender, receiver) = mpsc::channel(LANE_CAPACITY); if !self.spawn_lane(channel, receiver, msg.channel.clone(), msg.chat_id.clone()) { - msg.complete_delivery(Err("dispatcher is shutting down".to_string())); + msg.complete_delivery(DeliveryReceipt::DispatcherClosed); continue; } lanes.insert(lane_key.clone(), new_sender.clone()); @@ -89,7 +91,9 @@ impl OutboundDispatcher { capacity = LANE_CAPACITY, "Outbound lane full; rejecting message instead of blocking other destinations" ); - msg.complete_delivery(Err("outbound lane is full".to_string())); + msg.complete_delivery(DeliveryReceipt::TransientFailure { + summary: "outbound lane is full".to_string(), + }); } Err(mpsc::error::TrySendError::Closed(msg)) => { // The lane may have expired between the closed check and @@ -97,13 +101,15 @@ impl OutboundDispatcher { lanes.remove(&lane_key); let Some(channel) = self.channel_manager.get_channel(&msg.channel).await else { tracing::warn!(channel = %msg.channel, "No channel found for message"); - msg.complete_delivery(Err(format!("channel not found: {}", msg.channel))); + msg.complete_delivery(DeliveryReceipt::PermanentFailure { + summary: format!("channel not found: {}", msg.channel), + }); continue; }; let (new_sender, receiver) = mpsc::channel(LANE_CAPACITY); if !self.spawn_lane(channel, receiver, msg.channel.clone(), msg.chat_id.clone()) { - msg.complete_delivery(Err("dispatcher is shutting down".to_string())); + msg.complete_delivery(DeliveryReceipt::DispatcherClosed); continue; } match new_sender.try_send(msg) { @@ -111,9 +117,9 @@ impl OutboundDispatcher { lanes.insert(lane_key, new_sender); } Err(error) => { - error.into_inner().complete_delivery(Err( - "outbound lane could not be restarted during shutdown".to_string(), - )); + error + .into_inner() + .complete_delivery(DeliveryReceipt::DispatcherClosed); } } } @@ -143,15 +149,15 @@ impl OutboundDispatcher { Ok(None) | Err(_) => break, }; let result = Self::send_with_retry(&*channel, &msg, &target_lock).await; - if let Err(error) = &result { + if result != DeliveryReceipt::Delivered { tracing::error!( channel = %channel_name, chat_id = %chat_id, - error = %error, + result = ?result, "Failed to send message after retries" ); } - msg.complete_delivery(result.map_err(|error| error.to_string())); + msg.complete_delivery(result); } }, ) @@ -161,26 +167,28 @@ impl OutboundDispatcher { channel: &dyn Channel, msg: &OutboundMessage, target_lock: &tokio::sync::Mutex<()>, - ) -> Result<(), ChannelError> { + ) -> DeliveryReceipt { let _guard = target_lock.lock().await; const DELAYS: &[u64] = &[1, 2, 4]; for (attempt, &delay) in DELAYS.iter().enumerate() { let result = tokio::time::timeout(SEND_TIMEOUT, channel.send(msg.clone())).await; match result { - Ok(Ok(())) => return Ok(()), + Ok(Ok(())) => return DeliveryReceipt::Delivered, Ok(Err(error)) if attempt < DELAYS.len() - 1 && error.is_transient() => { - tracing::warn!(attempt = attempt + 1, delay, error = %error, "Send failed, retrying"); + tracing::warn!( + attempt = attempt + 1, + delay, + error_class = channel_error_class(&error), + "Send failed, retrying" + ); } - Ok(Err(error)) => return Err(error), + Ok(Err(error)) => return receipt_from_channel_error(error), Err(_) if attempt < DELAYS.len() - 1 => { tracing::warn!(attempt = attempt + 1, delay, "Send timed out, retrying"); } Err(_) => { - return Err(ChannelError::Other(format!( - "send timed out after {} seconds", - SEND_TIMEOUT.as_secs() - ))); + return DeliveryReceipt::TimedOut; } } tokio::time::sleep(Duration::from_secs(delay)).await; @@ -189,6 +197,36 @@ impl OutboundDispatcher { } } +fn channel_error_class(error: &ChannelError) -> &'static str { + match error { + ChannelError::ConnectionError(_) => "connection", + ChannelError::SendError(_) => "send", + ChannelError::BusError(_) => "bus", + ChannelError::ConfigError(_) => "config", + ChannelError::Other(_) => "other", + } +} + +fn receipt_from_channel_error(error: ChannelError) -> DeliveryReceipt { + match error { + ChannelError::ConnectionError(_) => DeliveryReceipt::TransientFailure { + summary: "channel connection failed after retries".to_string(), + }, + ChannelError::SendError(_) => DeliveryReceipt::TransientFailure { + summary: "channel send failed after retries".to_string(), + }, + ChannelError::BusError(_) => DeliveryReceipt::TransientFailure { + summary: "channel bus was unavailable".to_string(), + }, + ChannelError::ConfigError(_) => DeliveryReceipt::PermanentFailure { + summary: "channel configuration rejected delivery".to_string(), + }, + ChannelError::Other(_) => DeliveryReceipt::PermanentFailure { + summary: "channel rejected delivery".to_string(), + }, + } +} + /// Decrements the active-lane counter exactly once when a lane task ends, /// whether it exits normally, is cancelled, or is aborted. struct LaneGuard { @@ -348,7 +386,7 @@ mod tests { message.channel = "missing".to_string(); let error = bus.deliver_outbound(message).await.unwrap_err(); - assert!(matches!(error, crate::bus::BusError::DeliveryFailed(_))); + assert!(matches!(error, crate::bus::BusError::DeliveryPermanent(_))); task.abort(); supervisor.shutdown(Duration::from_secs(1)).await; } @@ -435,18 +473,40 @@ mod tests { }; let target_lock = tokio::sync::Mutex::new(()); - let error = OutboundDispatcher::send_with_retry( + let receipt = OutboundDispatcher::send_with_retry( &channel, &outbound("invalid", "message"), &target_lock, ) - .await - .unwrap_err(); + .await; - assert!(matches!(error, ChannelError::Other(_))); + assert!(matches!(receipt, DeliveryReceipt::PermanentFailure { .. })); assert_eq!(channel.attempts.load(Ordering::SeqCst), 1); } + #[test] + fn channel_errors_map_to_typed_sanitized_receipts() { + for error in [ + ChannelError::ConnectionError("https://secret.example/?token=x".to_string()), + ChannelError::SendError("private response body".to_string()), + ChannelError::BusError("private queue detail".to_string()), + ] { + let receipt = receipt_from_channel_error(error); + assert!(matches!(receipt, DeliveryReceipt::TransientFailure { .. })); + assert!(!format!("{receipt:?}").contains("private")); + assert!(!format!("{receipt:?}").contains("secret")); + } + for error in [ + ChannelError::ConfigError("api_key=x".to_string()), + ChannelError::Other("private platform payload".to_string()), + ] { + let receipt = receipt_from_channel_error(error); + assert!(matches!(receipt, DeliveryReceipt::PermanentFailure { .. })); + assert!(!format!("{receipt:?}").contains("private")); + assert!(!format!("{receipt:?}").contains("api_key")); + } + } + #[tokio::test] async fn shared_write_lock_orders_dispatcher_with_live_turn_writes() { let channel = RecordingChannel { @@ -468,7 +528,7 @@ mod tests { assert!(channel.sent.lock().await.is_empty()); drop(live_write); - send.await.unwrap(); + assert_eq!(send.await, DeliveryReceipt::Delivered); assert_eq!(channel.sent.lock().await.as_slice(), &["after-live-update"]); } } diff --git a/src/bus/message.rs b/src/bus/message.rs index 4a2e82c..547d216 100644 --- a/src/bus/message.rs +++ b/src/bus/message.rs @@ -509,17 +509,26 @@ pub struct OutboundMessage { pub reply_to: Option, pub media: Vec, pub metadata: HashMap, - pub(crate) delivery: Option>>>, + pub(crate) delivery: Option>>, } impl OutboundMessage { - pub(crate) fn complete_delivery(&self, result: Result<(), String>) { + pub(crate) fn complete_delivery(&self, result: DeliveryReceipt) { if let Some(delivery) = &self.delivery { delivery.send_replace(Some(result)); } } } +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum DeliveryReceipt { + Delivered, + TransientFailure { summary: String }, + PermanentFailure { summary: String }, + TimedOut, + DispatcherClosed, +} + // ============================================================================ // ControlMessage - Message for control channel (session management) // Uses SessionCommand from session module diff --git a/src/bus/mod.rs b/src/bus/mod.rs index 6b00771..c485dc8 100644 --- a/src/bus/mod.rs +++ b/src/bus/mod.rs @@ -4,8 +4,8 @@ pub mod message; pub use dispatcher::OutboundDispatcher; pub use message::{ ChannelContext, ChatMessage, ClientVisibility, CommittedMessage, CommittedTurnDelta, - CompletionStatus, ContentBlock, ControlMessage, InboundMessage, MediaItem, MediaRef, - MessageSource, OutboundMessage, ProviderReasoningState, SourceKind, TurnOrigin, + CompletionStatus, ContentBlock, ControlMessage, DeliveryReceipt, InboundMessage, MediaItem, + MediaRef, MessageSource, OutboundMessage, ProviderReasoningState, SourceKind, TurnOrigin, }; use std::sync::Arc; @@ -80,7 +80,17 @@ impl MessageBus { loop { delivery_rx.changed().await.map_err(|_| BusError::Closed)?; if let Some(result) = delivery_rx.borrow().clone() { - return result.map_err(BusError::DeliveryFailed); + return match result { + DeliveryReceipt::Delivered => Ok(()), + DeliveryReceipt::TransientFailure { summary } => { + Err(BusError::DeliveryTransient(summary)) + } + DeliveryReceipt::PermanentFailure { summary } => { + Err(BusError::DeliveryPermanent(summary)) + } + DeliveryReceipt::TimedOut => Err(BusError::DeliveryTimedOut), + DeliveryReceipt::DispatcherClosed => Err(BusError::Closed), + }; } } }) @@ -138,7 +148,8 @@ pub struct QueueDepths { #[derive(Debug)] pub enum BusError { Closed, - DeliveryFailed(String), + DeliveryTransient(String), + DeliveryPermanent(String), DeliveryTimedOut, } @@ -146,7 +157,12 @@ impl std::fmt::Display for BusError { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { match self { BusError::Closed => write!(f, "Bus channel closed"), - BusError::DeliveryFailed(error) => write!(f, "Outbound delivery failed: {error}"), + BusError::DeliveryTransient(error) => { + write!(f, "Transient outbound delivery failure: {error}") + } + BusError::DeliveryPermanent(error) => { + write!(f, "Permanent outbound delivery failure: {error}") + } BusError::DeliveryTimedOut => write!(f, "Outbound delivery confirmation timed out"), } } diff --git a/src/gateway/http.rs b/src/gateway/http.rs index 5153a8c..5a22c91 100644 --- a/src/gateway/http.rs +++ b/src/gateway/http.rs @@ -812,8 +812,12 @@ fn scheduler_snapshot(jobs: &[crate::storage::ScheduledJob]) -> Value { .iter() .filter(|job| { matches!( - job.last_status.as_deref(), - Some("error" | "timeout" | "delivery_error") + job.last_outcome, + Some( + crate::storage::ScheduledOutcomeKind::Failed + | crate::storage::ScheduledOutcomeKind::Refused + | crate::storage::ScheduledOutcomeKind::Unknown + ) ) }) .count(); @@ -1465,6 +1469,26 @@ pub async fn get_job_runs( .list_scheduled_job_runs(&id, limit) .await .map_err(ApiError::internal)?; + let runs = runs + .into_iter() + .map(|run| { + json!({ + "id": run.id, + "job_id": run.job_id, + "scheduled_for": run.scheduled_for, + "started_at": run.started_at, + "finished_at": run.finished_at, + "status": run.status, + "outcome": run.outcome, + "message": run.message, + "diagnostic": run.diagnostic, + "duration_ms": run.duration_ms, + "delivery_status": run.delivery_status, + "delivery_attempts": run.delivery_attempts, + "delivery_error": run.delivery_error, + }) + }) + .collect::>(); Ok(Json(json!({ "runs": runs }))) } @@ -1564,37 +1588,57 @@ mod tests { id: &str, enabled: bool, next_run_at: i64, - last_status: Option<&str>, + last_outcome: Option, ) -> crate::storage::ScheduledJob { crate::storage::ScheduledJob { id: id.to_string(), name: id.to_string(), schedule: crate::scheduler::Schedule::Every { every_ms: 60_000 }, prompt: String::new(), + agent_id: None, channel: "cli_chat".to_string(), chat_id: "test".to_string(), - model: None, - job_kind: crate::storage::JobKind::Task, delivery_policy: crate::storage::DeliveryPolicy::Never, enabled, - delete_after_run: false, next_run_at, last_run_at: None, - last_status: last_status.map(str::to_string), - last_error: None, + last_outcome, created_at: 0, updated_at: 0, + locked_at: None, + lock_owner: None, + lease_until: None, } } #[test] fn scheduler_snapshot_classifies_failures_and_next_enabled_run() { let jobs = vec![ - scheduled_job("healthy", true, 300, Some("ok")), - scheduled_job("error", true, 200, Some("error")), - scheduled_job("timeout", false, 100, Some("timeout")), - scheduled_job("delivery", true, 400, Some("delivery_error")), - scheduled_job("other", false, 50, Some("cancelled")), + scheduled_job( + "healthy", + true, + 300, + Some(crate::storage::ScheduledOutcomeKind::Ok), + ), + scheduled_job( + "error", + true, + 200, + Some(crate::storage::ScheduledOutcomeKind::Failed), + ), + scheduled_job( + "refused", + false, + 100, + Some(crate::storage::ScheduledOutcomeKind::Refused), + ), + scheduled_job( + "unknown", + true, + 400, + Some(crate::storage::ScheduledOutcomeKind::Unknown), + ), + scheduled_job("other", false, 50, None), ]; assert_eq!( diff --git a/src/gateway/mod.rs b/src/gateway/mod.rs index c2ea2cb..50f4d69 100644 --- a/src/gateway/mod.rs +++ b/src/gateway/mod.rs @@ -178,6 +178,7 @@ impl GatewayState { .init(&config, workspace_path.clone()) .await .map_err(|e| format!("Failed to init channels: {}", e))?; + let available_channels = channel_manager.list_channel_names().await; let turn_delivery = TurnDeliveryService::new( delivery_coordinator.clone(), channel_manager.clone(), @@ -189,7 +190,10 @@ impl GatewayState { } else { None }; - let health = Arc::new(crate::health::HealthService::new(config.clone())); + let health = Arc::new( + crate::health::HealthService::new(config.clone()) + .with_scheduler_runtime(storage.clone(), available_channels.clone()), + ); let provider_profiles: std::collections::HashMap = config .agents .keys() @@ -245,9 +249,9 @@ impl GatewayState { let session_manager = Arc::new(session_manager); session_manager.bind_inbox_wake(); let agent_catalog = session_manager.agent_catalog(); + health.bind_agent_catalog(agent_catalog.clone()); // Register send_message tool with available channel names - let available_channels = channel_manager.list_channel_names().await; let valid_channels = available_channels.clone(); session_manager.register_outbound_tool(available_channels); @@ -277,11 +281,15 @@ impl GatewayState { .tools() .register(crate::tools::cron::CronAddTool::new( storage.clone(), - valid_channels, + valid_channels.clone(), + agent_catalog.clone(), )); session_manager .tools() .register(crate::tools::cron::CronListTool::new(storage.clone())); + session_manager + .tools() + .register(crate::tools::cron::CronRunsTool::new(storage.clone())); session_manager .tools() .register(crate::tools::cron::CronRemoveTool::new(storage.clone())); @@ -293,7 +301,11 @@ impl GatewayState { .register(crate::tools::cron::CronDisableTool::new(storage.clone())); session_manager .tools() - .register(crate::tools::cron::CronUpdateTool::new(storage.clone())); + .register(crate::tools::cron::CronUpdateTool::new( + storage.clone(), + valid_channels, + agent_catalog.clone(), + )); tracing::info!("Cron tools registered"); } @@ -332,7 +344,25 @@ impl GatewayState { } /// Start the message processing loops - pub async fn start_message_processing(&self) { + pub async fn start_message_processing(&self) -> Result<(), String> { + match self + .storage + .recover_scheduled_runs(chrono::Utc::now().timestamp_millis()) + .await + { + Ok(recovered) if recovered > 0 => { + tracing::warn!( + recovered, + "Scheduled runs recovered as unknown on activation" + ); + } + Ok(_) => {} + Err(error) => { + return Err(format!( + "Scheduled run recovery failed on activation: {error}" + )); + } + } // Recover durable Agent state for this runtime generation: interrupt // runs of older generations, expire stale inbox leases and reconcile // capacity rows. Runs never recover while the generation is still a candidate. @@ -351,7 +381,9 @@ impl GatewayState { } } Err(error) => { - tracing::error!(error = %error, "Agent state recovery failed on activation"); + return Err(format!( + "Agent state recovery failed on activation: {error}" + )); } } } @@ -461,6 +493,7 @@ impl GatewayState { }); tracing::info!("Scheduler background task spawned"); } + Ok(()) } } @@ -525,7 +558,7 @@ pub async fn run( reload_controller.set_failed(current_generation, error.to_string()); return Err(error.into()); } - state.start_message_processing().await; + state.start_message_processing().await?; reload_controller.set_phase(current_generation, reload::ReloadPhase::Active); let app = build_router(state.clone()); let generation_listener = TcpListener::from_std(listener.try_clone()?)?; diff --git a/src/gateway/router.rs b/src/gateway/router.rs index 0f9dd87..e514958 100644 --- a/src/gateway/router.rs +++ b/src/gateway/router.rs @@ -459,7 +459,7 @@ mod tests { Some("command") ); assert!(!publish_task.is_finished()); - output.complete_delivery(Ok(())); + output.complete_delivery(crate::bus::DeliveryReceipt::Delivered); publish_task.await.unwrap(); } diff --git a/src/health.rs b/src/health.rs index b76a8de..1b88a0d 100644 --- a/src/health.rs +++ b/src/health.rs @@ -1,6 +1,7 @@ use std::collections::HashSet; use std::path::Path; use std::process::{Output, Stdio}; +use std::sync::{Arc, RwLock}; use std::time::Duration; use serde::{Deserialize, Serialize}; @@ -117,11 +118,41 @@ impl HealthReport { #[derive(Clone)] pub struct HealthService { config: Config, + scheduler_runtime: Arc>>, +} + +#[derive(Clone)] +struct SchedulerHealthRuntime { + storage: Arc, + channels: HashSet, + catalog: Option>, } impl HealthService { pub fn new(config: Config) -> Self { - Self { config } + Self { + config, + scheduler_runtime: Arc::new(RwLock::new(None)), + } + } + + pub fn with_scheduler_runtime( + self, + storage: Arc, + channels: Vec, + ) -> Self { + *self.scheduler_runtime.write().unwrap() = Some(SchedulerHealthRuntime { + storage, + channels: channels.into_iter().collect(), + catalog: None, + }); + self + } + + pub fn bind_agent_catalog(&self, catalog: Arc) { + if let Some(runtime) = self.scheduler_runtime.write().unwrap().as_mut() { + runtime.catalog = Some(catalog); + } } pub async fn check(&self) -> HealthReport { @@ -139,9 +170,165 @@ impl HealthService { ]; checks.extend(self.check_mcp_commands()); checks.extend(self.check_browser().await); + checks.extend(self.check_scheduler().await); HealthReport::from_checks(checks) } + async fn check_scheduler(&self) -> Vec { + let scheduler_config = self.config.gateway.scheduler.clone().unwrap_or_default(); + if !scheduler_config.enabled { + return vec![HealthCheck { + name: "scheduled runs".to_string(), + category: "configured".to_string(), + required: false, + status: HealthStatus::Pass, + detail: "scheduler disabled; persisted jobs are not executed".to_string(), + remediation: None, + }]; + } + let runtime = self.scheduler_runtime.read().unwrap().clone(); + let Some(runtime) = runtime else { + return vec![HealthCheck { + name: "scheduled runs".to_string(), + category: "runtime".to_string(), + required: false, + status: HealthStatus::Pass, + detail: "runtime scheduler state is checked by the running Gateway".to_string(), + remediation: None, + }]; + }; + let jobs = match runtime.storage.list_scheduled_jobs().await { + Ok(jobs) => jobs, + Err(error) => { + return vec![HealthCheck { + name: "scheduled run storage".to_string(), + category: "runtime".to_string(), + required: true, + status: HealthStatus::Fail, + detail: format!("cannot read scheduled jobs: {error}"), + remediation: Some( + "Check the configured SQLite database and Gateway logs.".to_string(), + ), + }]; + } + }; + let now = chrono::Utc::now().timestamp_millis(); + let stale_after = scheduler_config + .poll_interval_secs + .max(60) + .saturating_mul(10_000) + .min(i64::MAX as u64) as i64; + let mut bad_references = Vec::new(); + let mut stale_deliveries = Vec::new(); + let mut unhealthy_latest = Vec::new(); + let mut silent_unknown = Vec::new(); + let mut covered_intervals = Vec::new(); + let mut invalid_next = Vec::new(); + + match runtime + .storage + .list_stale_scheduled_deliveries(now.saturating_sub(stale_after), 100) + .await + { + Ok(runs) => { + stale_deliveries.extend( + runs.into_iter() + .map(|run| format!("{}#{}", run.job_id, run.id)), + ); + } + Err(error) => bad_references.push(format!("delivery backlog unavailable ({error})")), + } + + for job in &jobs { + if !runtime.channels.contains(&job.channel) { + bad_references.push(format!("{}: channel {}", job.id, job.channel)); + } + if let Some(agent_id) = job.agent_id.as_deref() + && runtime + .catalog + .as_ref() + .is_none_or(|catalog| catalog.get(agent_id).is_none()) + { + bad_references.push(format!("{}: agent {}", job.id, agent_id)); + } + if job.enabled && crate::scheduler::next_run_for_schedule(&job.schedule, now).is_none() + { + invalid_next.push(job.id.clone()); + } + + let runs = match runtime.storage.list_scheduled_job_runs(&job.id, 20).await { + Ok(runs) => runs, + Err(error) => { + bad_references.push(format!("{}: run history unavailable ({error})", job.id)); + continue; + } + }; + if let Some(latest) = runs.first() { + if matches!( + latest.status, + crate::storage::ScheduledRunStatus::Unknown + | crate::storage::ScheduledRunStatus::Failed + | crate::storage::ScheduledRunStatus::TimedOut + ) { + unhealthy_latest.push(format!("{}#{}", job.id, latest.id)); + } + if job.delivery_policy == crate::storage::DeliveryPolicy::Never + && latest.outcome == Some(crate::storage::ScheduledOutcomeKind::Unknown) + { + silent_unknown.push(format!("{}#{}", job.id, latest.id)); + } + if let crate::scheduler::Schedule::Every { every_ms } = job.schedule + && i64::try_from(every_ms).ok().is_some_and(|interval| { + latest + .duration_ms + .is_some_and(|duration| duration >= interval) + }) + { + covered_intervals.push(format!("{}#{}", job.id, latest.id)); + } + } + } + + vec![ + scheduler_health_check( + "scheduled references", + bad_references, + "all Agent and channel references are available", + "Update or disable jobs that reference missing Agents or channels.", + ), + scheduler_health_check( + "scheduled delivery backlog", + stale_deliveries, + "no stale pending or delivering notifications", + "Inspect cron_runs and the target channel configuration.", + ), + scheduler_health_check( + "scheduled latest outcomes", + unhealthy_latest, + "no latest run is failed, timed out, or unknown", + "Inspect cron_runs for the diagnostic and assess external side effects before retrying.", + ), + scheduler_health_check( + "silent unknown scheduled runs", + silent_unknown, + "no never-delivery job has an unknown latest outcome", + "Inspect the run manually; delivery_policy=never prevents automatic notification.", + ), + scheduler_health_check( + "scheduled execution intervals", + covered_intervals, + "recent execution durations fit their Every intervals", + "Increase the interval or split long-running jobs.", + ), + scheduler_health_check( + "scheduled next runs", + invalid_next, + "all enabled schedules can compute a next run", + "Correct the schedule expression or disable the job.", + ), + ] + } + fn check_mcp_commands(&self) -> Vec { let mut seen = HashSet::new(); let mut checks = Vec::new(); @@ -365,6 +552,38 @@ impl HealthService { } } +fn scheduler_health_check( + name: &str, + findings: Vec, + healthy_detail: &str, + remediation: &str, +) -> HealthCheck { + if findings.is_empty() { + HealthCheck { + name: name.to_string(), + category: "runtime".to_string(), + required: false, + status: HealthStatus::Pass, + detail: healthy_detail.to_string(), + remediation: None, + } + } else { + let total = findings.len(); + let mut sample = findings.into_iter().take(5).collect::>().join(", "); + if total > 5 { + sample.push_str(&format!(", and {} more", total - 5)); + } + HealthCheck { + name: name.to_string(), + category: "runtime".to_string(), + required: false, + status: HealthStatus::Warning, + detail: format!("{total} finding(s): {sample}"), + remediation: Some(remediation.to_string()), + } + } +} + fn check_configuration_recovery(config: &Config) -> HealthCheck { if config.diagnostics.is_empty() { return HealthCheck { diff --git a/src/scheduler/mod.rs b/src/scheduler/mod.rs index cffccd9..bdd953c 100644 --- a/src/scheduler/mod.rs +++ b/src/scheduler/mod.rs @@ -3,81 +3,41 @@ pub mod types; use std::sync::Arc; use std::time::Instant; -use futures_util::stream::{self, StreamExt}; +use tokio::task::JoinSet; use tokio::time; use crate::config::SchedulerConfig; -use crate::session::SessionManager; -use crate::session::session::HandleResult; -use crate::storage::ScheduledJob; -use crate::storage::Storage; -use crate::storage::{DeliveryPolicy, JobKind, JobRun}; +use crate::session::{ScheduledDeliveryError, SessionManager}; +use crate::storage::{ + ClaimedScheduledRun, JobRun, ScheduledOutcomeKind, ScheduledRunCompletion, ScheduledRunStatus, + Storage, +}; pub use types::Schedule; -#[derive(Debug, Clone, PartialEq, Eq)] -enum ScheduledDisposition { - Content(String), - Quiet(String), - ReportedFailure(String), - Refused(String), -} - -fn parse_scheduled_disposition(output: &str) -> ScheduledDisposition { - let trimmed = output.trim(); - if trimmed.eq_ignore_ascii_case("NO_REPLY") { - return ScheduledDisposition::Quiet(String::new()); - } - let upper = trimmed.to_ascii_uppercase(); - for (prefix, kind) in [ - ("NO_REPLY[INFO]", "info"), - ("NO_REPLY[FAIL]", "fail"), - ("NO_REPLY[REFUSE]", "refuse"), - ] { - if upper.starts_with(prefix) { - let suffix = &trimmed[prefix.len()..]; - if !suffix.is_empty() && !suffix.trim_start().starts_with(':') { - continue; - } - let reason = suffix.trim().trim_start_matches(':').trim().to_string(); - return match kind { - "info" => ScheduledDisposition::Quiet(reason), - "fail" => ScheduledDisposition::ReportedFailure(reason), - _ => ScheduledDisposition::Refused(reason), - }; - } - } - if trimmed.is_empty() { - ScheduledDisposition::ReportedFailure("scheduled agent returned empty output".into()) - } else { - ScheduledDisposition::Content(trimmed.to_string()) - } -} - /// Compute the next execution time (Unix ms) for a schedule, given `from` (Unix ms). -/// Returns `None` if no next time can be determined (e.g., invalid cron expression). +/// Returns `None` if no next time can be determined (e.g. an invalid cron expression). pub fn next_run_for_schedule(schedule: &Schedule, from: i64) -> Option { use chrono::{TimeZone, Utc}; use std::str::FromStr; match schedule { Schedule::At { at } => Some(*at), - Schedule::Every { every_ms } => Some(from + *every_ms as i64), + Schedule::Every { every_ms } => Some(from.saturating_add(i64::try_from(*every_ms).ok()?)), Schedule::Cron { expr, tz } => { let cron_schedule = cron::Schedule::from_str(expr.as_str()).ok()?; let from_secs = from / 1000; let from_nanos = ((from % 1000) * 1_000_000) as u32; let from_dt = Utc.timestamp_opt(from_secs, from_nanos).single()?; - let next_utc = if let Some(tz_str) = tz { let tz: chrono_tz::Tz = tz_str.parse().ok()?; - let from_local = from_dt.with_timezone(&tz); - let next_local = cron_schedule.after(&from_local).next()?; - next_local.with_timezone(&Utc) + cron_schedule + .after(&from_dt.with_timezone(&tz)) + .next()? + .with_timezone(&Utc) } else { cron_schedule.after(&from_dt).next()? }; - Some(next_utc.timestamp_millis()) } } @@ -90,8 +50,6 @@ fn now_ms() -> i64 { .as_millis() as i64 } -/// The scheduler runs as a background tokio task, periodically checking for due jobs -/// and executing them via `SessionManager::handle_cron_message`. pub struct Scheduler { storage: Arc, session_manager: Arc, @@ -129,15 +87,16 @@ impl Scheduler { } } - /// Claim due jobs with a durable lease, then execute the claimed batch with - /// bounded concurrency. + /// Non-blocking event loop. Execution and delivery use separate bounded + /// JoinSets so one long Agent run cannot delay other claims or outbox work. pub async fn run(self: Arc) { let poll_duration = time::Duration::from_secs(self.config.poll_interval_secs.max(1)); let mut interval = time::interval(poll_duration); interval.set_missed_tick_behavior(time::MissedTickBehavior::Skip); - // Keep accidental configuration values from claiming an unbounded - // batch and overwhelming the runtime or SQLite parameter conversion. let max_concurrent = self.config.max_concurrent.clamp(1, 256); + let max_delivery = max_concurrent.clamp(1, 16); + let mut runs = JoinSet::new(); + let mut deliveries = JoinSet::new(); tracing::info!( poll_interval_secs = self.config.poll_interval_secs, @@ -147,375 +106,340 @@ impl Scheduler { ); loop { - interval.tick().await; + tokio::select! { + _ = interval.tick() => {} + Some(result) = runs.join_next(), if !runs.is_empty() => { + if let Err(error) = result { + tracing::error!(error = %error, "scheduled run task panicked"); + } + } + Some(result) = deliveries.join_next(), if !deliveries.is_empty() => { + if let Err(error) = result { + tracing::error!(error = %error, "scheduled delivery task panicked"); + } + } + } + + while let Some(result) = runs.try_join_next() { + if let Err(error) = result { + tracing::error!(error = %error, "scheduled run task panicked"); + } + } + while let Some(result) = deliveries.try_join_next() { + if let Err(error) = result { + tracing::error!(error = %error, "scheduled delivery task panicked"); + } + } if !self.admission.is_accepting() { continue; } + let now = now_ms(); + let delivery_slots = max_delivery.saturating_sub(deliveries.len()); + if delivery_slots > 0 { + let lease_until = now.saturating_add(180_000); + let delivery_owner = format!("{}:delivery:{}", self.owner, uuid::Uuid::new_v4()); + match self + .storage + .claim_scheduled_deliveries(now, lease_until, &delivery_owner, delivery_slots) + .await + { + Ok(claimed) => { + for run in claimed { + let scheduler = self.clone(); + deliveries.spawn(async move { + scheduler.deliver_claimed_run(run).await; + }); + } + } + Err(error) => { + tracing::error!(error = %error, "scheduler: failed to claim deliveries"); + } + } + } + + let run_slots = max_concurrent.saturating_sub(runs.len()); + if run_slots == 0 { + continue; + } let lease_ms = self .config .execution_timeout_secs .saturating_add(150) .saturating_mul(1000) .min(i64::MAX as u64) as i64; - let lease_until = now.saturating_add(lease_ms); - let jobs = match self + let run_owner = format!("{}:run:{}", self.owner, uuid::Uuid::new_v4()); + match self .storage - .claim_due_scheduled_jobs(now, lease_until, &self.owner, max_concurrent) + .claim_due_scheduled_runs(now, now.saturating_add(lease_ms), &run_owner, run_slots) .await { - Ok(jobs) => jobs, - Err(error) => { - tracing::error!(error = %error, "scheduler: failed to claim due jobs"); - continue; + Ok(claimed) => { + for run in claimed { + let scheduler = self.clone(); + runs.spawn(async move { + scheduler.execute_claimed_run(run).await; + }); + } + } + Err(error) => { + tracing::error!(error = %error, "scheduler: failed to claim due runs"); } - }; - - if jobs.is_empty() { - continue; } - tracing::info!(count = jobs.len(), "scheduler: claimed due jobs"); - - stream::iter(jobs) - .for_each_concurrent(max_concurrent, |job| { - let scheduler = self.clone(); - async move { scheduler.execute_claimed_job(job).await } - }) - .await; } } - async fn execute_claimed_job(self: Arc, job: ScheduledJob) { - let Some(_activity) = self.admission.try_enter() else { - if let Err(error) = self - .storage - .release_scheduled_job_lease(&job.id, &self.owner) - .await - { - tracing::error!(job_id = %job.id, error = %error, "scheduler: failed to release job claimed during reload drain"); - } - return; - }; + async fn execute_claimed_run(self: Arc, claimed: ClaimedScheduledRun) { let start = Instant::now(); - let started_at = now_ms(); - tracing::info!(job_id = %job.id, job_name = %job.name, "scheduler: executing claimed job"); - - let managed = job.delivery_policy != DeliveryPolicy::Direct; - let execution = async { - if managed { - self.session_manager - .handle_managed_scheduled_message( - &job.prompt, + let job = &claimed.job; + let (completion, agent_execution) = if let Some(_activity) = self.admission.try_enter() { + match self.session_manager.agent_coordinator() { + Some(coordinator) => match coordinator + .execute_scheduled( + claimed.run_id, + &claimed.owner, &job.id, &job.name, - job.job_kind == JobKind::Monitor, - ) - .await - .map(HandleResult::AgentResponse) - } else { - self.session_manager - .handle_cron_message( - &job.channel, - &job.chat_id, + job.agent_id.as_deref(), &job.prompt, - &job.id, - &job.name, + self.config.execution_timeout_secs.max(1), ) .await - } - }; - let result = time::timeout( - time::Duration::from_secs(self.config.execution_timeout_secs.max(1)), - execution, - ) - .await; - let finished_at = now_ms(); - let duration_ms = start.elapsed().as_millis() as i64; - - let (mut status, output, error, result_kind, mut delivery_status, mut delivery_error) = - match result { - Ok(Ok( - HandleResult::AgentResponse(output) | HandleResult::CommandOutput(output), - )) => { - let output = if output.len() > 8000 { - format!( - "{}...[truncated]", - &output[..output.ceil_char_boundary(8000)] - ) - } else { - output - }; - if !managed { - ( - "ok".into(), - Some(output), - None, - None, - Some("direct".into()), - None, - ) - } else { - let disposition = parse_scheduled_disposition(&output); - let (kind, content, alert) = match &disposition { - ScheduledDisposition::Content(value) => { - ("content", Some(value.as_str()), true) - } - ScheduledDisposition::Quiet(_) => ("quiet", None, false), - ScheduledDisposition::ReportedFailure(value) => { - ("reported_failure", Some(value.as_str()), true) - } - ScheduledDisposition::Refused(value) => { - ("refused", Some(value.as_str()), true) - } - }; - let should_deliver = match job.delivery_policy { - DeliveryPolicy::Always => true, - DeliveryPolicy::OnAlert => alert, - DeliveryPolicy::Never => false, - DeliveryPolicy::Direct => false, - }; - if should_deliver { - let message = content.unwrap_or("巡检完成，未发现需要关注的问题。"); - match self - .session_manager - .deliver_scheduled_message( - &job.channel, - &job.chat_id, - &job.id, - &job.name, - message, - ) - .await - { - Ok(()) => ( - "ok".into(), - Some(output), - None, - Some(kind.into()), - Some("delivered".into()), - None, - ), - Err(delivery_error) => ( - "delivery_error".into(), - Some(output), - None, - Some(kind.into()), - Some("failed".into()), - Some(delivery_error), - ), - } + { + Ok(execution) => { + if execution.status == ScheduledRunStatus::Completed + && let Some(outcome) = execution.outcome.clone() + { + ( + ScheduledRunCompletion { + status: ScheduledRunStatus::Completed, + outcome: outcome.kind, + message: outcome.message, + diagnostic: execution.error.clone(), + duration_ms: start.elapsed().as_millis() as i64, + }, + Some(execution), + ) } else { - let delivery = if job.delivery_policy == DeliveryPolicy::Never { - "skipped" - } else { - "suppressed" + let diagnostic = execution.error.clone().unwrap_or_else(|| { + "scheduled Agent returned ordinary text without calling complete_scheduled_run" + .to_string() + }); + let status = match execution.status { + ScheduledRunStatus::TimedOut => ScheduledRunStatus::TimedOut, + ScheduledRunStatus::Interrupted | ScheduledRunStatus::Cancelled => { + ScheduledRunStatus::Interrupted + } + _ => ScheduledRunStatus::Failed, }; ( - "ok".into(), - Some(output), - None, - Some(kind.into()), - Some(delivery.into()), - None, + ScheduledRunCompletion { + status, + outcome: ScheduledOutcomeKind::Failed, + message: if status == ScheduledRunStatus::TimedOut { + format!("定时任务「{}」执行超时。", job.name) + } else if status == ScheduledRunStatus::Interrupted { + format!("定时任务「{}」在系统关停时被中断。", job.name) + } else { + format!( + "定时任务「{}」未可靠完成：Agent 未提交结构化运行结果。", + job.name + ) + }, + diagnostic: Some(diagnostic), + duration_ms: start.elapsed().as_millis() as i64, + }, + Some(execution), ) } } + Err(error) => ( + ScheduledRunCompletion { + status: ScheduledRunStatus::Failed, + outcome: ScheduledOutcomeKind::Failed, + message: job.agent_id.as_deref().map_or_else( + || format!("定时任务「{}」未能启动 Root Agent 执行。", job.name), + |agent_id| { + format!( + "定时任务「{}」无法使用 Agent「{}」执行。请恢复该 Agent 定义，或更新任务的 agent_id。", + job.name, agent_id + ) + }, + ), + diagnostic: Some(error.to_string()), + duration_ms: start.elapsed().as_millis() as i64, + }, + None, + ), + }, + None => ( + ScheduledRunCompletion { + status: ScheduledRunStatus::Failed, + outcome: ScheduledOutcomeKind::Failed, + message: format!("定时任务「{}」未能启动执行。", job.name), + diagnostic: Some("AgentCoordinator is unavailable".to_string()), + duration_ms: start.elapsed().as_millis() as i64, + }, + None, + ), + } + } else { + ( + ScheduledRunCompletion { + status: ScheduledRunStatus::Interrupted, + outcome: ScheduledOutcomeKind::Failed, + message: format!("定时任务「{}」因 Gateway 重载而中断。", job.name), + diagnostic: Some("runtime admission closed".to_string()), + duration_ms: start.elapsed().as_millis() as i64, + }, + None, + ) + }; + let finished_at = now_ms(); + let mut commit_attempt = 0_u64; + let commit = loop { + let result = match agent_execution.as_ref() { + Some(execution) => { + self.storage + .finish_scheduled_run_with_agent( + claimed.run_id, + &claimed.owner, + &completion, + &execution.agent_run_id, + &execution.agent_run_id, + execution.runtime_generation, + &execution.agent_terminal, + finished_at, + ) + .await + } + None => { + self.storage + .finish_scheduled_run( + claimed.run_id, + &claimed.owner, + &completion, + finished_at, + ) + .await } - Ok(Ok(HandleResult::AgentProcessing)) => ( - "error".to_string(), - None, - Some("cron execution returned asynchronous processing".to_string()), - None, - None, - None, - ), - Ok(Err(error)) => ( - "error".to_string(), - None, - Some(error.to_string()), - None, - None, - None, - ), - Err(_) => ( - "timeout".to_string(), - None, - Some(format!( - "execution exceeded {} seconds", - self.config.execution_timeout_secs.max(1) - )), - None, - None, - None, - ), }; - - if managed - && delivery_status.is_none() - && job.delivery_policy != DeliveryPolicy::Never - && let Some(message) = error.as_deref() - { - let notice = format!("定时任务「{}」执行失败：{}", job.name, message); - match self - .session_manager - .deliver_scheduled_message(&job.channel, &job.chat_id, &job.id, &job.name, ¬ice) - .await - { - Ok(()) => delivery_status = Some("delivered".into()), - Err(error) => { - status = "delivery_error".into(); - delivery_status = Some("failed".into()); - delivery_error = Some(error); + match result { + Err(error) if error.is_transient() && commit_attempt < 2 => { + commit_attempt += 1; + tracing::warn!( + job_id = %job.id, + run_id = claimed.run_id, + attempt = commit_attempt + 1, + error = %error, + "scheduler: retrying transient run completion commit" + ); + time::sleep(time::Duration::from_millis(50 * commit_attempt)).await; } + result => break result, } + }; + match commit { + Ok(true) => tracing::info!( + job_id = %job.id, + run_id = claimed.run_id, + status = completion.status.as_str(), + outcome = completion.outcome.as_str(), + duration_ms = completion.duration_ms, + "scheduler: run completed" + ), + Ok(false) => tracing::warn!( + job_id = %job.id, + run_id = claimed.run_id, + "scheduler: late run result discarded" + ), + Err(error) => tracing::error!( + job_id = %job.id, + run_id = claimed.run_id, + error = %error, + "scheduler: failed to commit run completion" + ), } + } - let (next_run_at, disable, delete) = match &job.schedule { - Schedule::At { .. } => (None, !job.delete_after_run, job.delete_after_run), - Schedule::Every { .. } | Schedule::Cron { .. } => { - match next_run_for_schedule(&job.schedule, finished_at) { - Some(next) => (Some(next), false, false), - None => (None, true, false), - } + async fn deliver_claimed_run(self: Arc, run: JobRun) { + let Some(delivery_owner) = run.delivery_lease_owner.clone() else { + tracing::error!( + run_id = run.id, + "scheduler: claimed delivery has no lease owner" + ); + return; + }; + let result = self + .session_manager + .deliver_scheduled_run(&run, &delivery_owner) + .await; + let (delivered, permanent, error) = match result { + Ok(()) => (true, false, None), + Err(ScheduledDeliveryError::Transient(error)) => { + (false, false, Some(sanitize_error(&error))) + } + Err(ScheduledDeliveryError::Permanent(error)) => { + (false, true, Some(sanitize_error(&error))) } }; - let run = JobRun { - id: 0, - job_id: job.id.clone(), - started_at, - finished_at, - status, - output, - error, - duration_ms, - result_kind, - delivery_status, - delivery_error, - }; - - if let Err(error) = self + if let Err(commit_error) = self .storage - .complete_scheduled_job(&run, &self.owner, next_run_at, disable, delete) + .complete_scheduled_delivery( + run.id, + &delivery_owner, + delivered, + permanent, + error.as_deref(), + now_ms(), + ) .await { - tracing::error!(job_id = %job.id, error = %error, "scheduler: failed to commit job completion"); - let _ = self - .storage - .release_scheduled_job_lease(&job.id, &self.owner) - .await; - return; + tracing::error!( + run_id = run.id, + error = %commit_error, + "scheduler: failed to commit delivery receipt" + ); } - - tracing::info!( - job_id = %job.id, - status = %run.status, - duration_ms, - "scheduler: job completed" - ); } } +fn sanitize_error(error: &str) -> String { + error.chars().take(1_024).collect() +} + #[cfg(test)] mod tests { use super::*; #[test] - fn test_next_run_at_schedule() { - let now = 1000000; - let next = next_run_for_schedule(&Schedule::At { at: 2000000 }, now); - assert_eq!(next, Some(2000000)); + fn next_run_for_every_uses_claim_time() { + assert_eq!( + next_run_for_schedule(&Schedule::Every { every_ms: 5_000 }, 1_000), + Some(6_000) + ); } #[test] - fn test_next_run_every_schedule() { - let now = 1000000; - let next = next_run_for_schedule(&Schedule::Every { every_ms: 5000 }, now); - assert_eq!(next, Some(1005000)); + fn next_run_for_at_keeps_absolute_timestamp() { + assert_eq!( + next_run_for_schedule(&Schedule::At { at: 2_000 }, 1_000), + Some(2_000) + ); } #[test] - fn test_next_run_cron_every_minute() { - let expr = "0 * * * * *".to_string(); - let schedule = Schedule::Cron { expr, tz: None }; - let now = 1000000; - let next = next_run_for_schedule(&schedule, now); - assert!(next.is_some()); - assert!(next.unwrap() > now); - } - - #[test] - fn test_next_run_cron_every_day_at_9am() { - let expr = "0 0 9 * * *".to_string(); - let schedule = Schedule::Cron { expr, tz: None }; - let now = 1000000; - let next = next_run_for_schedule(&schedule, now); - assert!(next.is_some()); - let next_ms = next.unwrap(); - assert!(next_ms > now); - } - - #[test] - fn test_next_run_cron_uses_from_argument() { - let expr = "0 * * * * *".to_string(); - let schedule = Schedule::Cron { expr, tz: None }; - let from = chrono::DateTime::parse_from_rfc3339("2026-06-16T12:34:20Z") - .unwrap() - .timestamp_millis(); - - let next = next_run_for_schedule(&schedule, from).unwrap(); - let expected = chrono::DateTime::parse_from_rfc3339("2026-06-16T12:35:00Z") - .unwrap() - .timestamp_millis(); - assert_eq!(next, expected); - } - - #[test] - fn scheduled_disposition_is_fail_safe() { - assert!(matches!( - parse_scheduled_disposition("NO_REPLY"), - ScheduledDisposition::Quiet(_) - )); - assert!(matches!( - parse_scheduled_disposition("NO_REPLY[INFO]: healthy"), - ScheduledDisposition::Quiet(_) - )); - assert!(matches!( - parse_scheduled_disposition("NO_REPLY[FAIL]: timeout"), - ScheduledDisposition::ReportedFailure(_) - )); - assert!(matches!( - parse_scheduled_disposition("NO_REPLY[REFUSE]: denied"), - ScheduledDisposition::Refused(_) - )); - assert!(matches!( - parse_scheduled_disposition("text mentioning NO_REPLY"), - ScheduledDisposition::Content(_) - )); - assert!(matches!( - parse_scheduled_disposition("NO_REPLY[INFO] but this is content"), - ScheduledDisposition::Content(_) - )); - assert!(matches!( - parse_scheduled_disposition(""), - ScheduledDisposition::ReportedFailure(_) - )); - } - - #[test] - fn test_next_run_cron_timezone_uses_from_argument() { - let expr = "0 0 9 * * *".to_string(); + fn cron_timezone_uses_from_argument() { let schedule = Schedule::Cron { - expr, + expr: "0 0 9 * * *".to_string(), tz: Some("Asia/Shanghai".to_string()), }; let from = chrono::DateTime::parse_from_rfc3339("2026-06-16T00:30:00Z") .unwrap() .timestamp_millis(); - - let next = next_run_for_schedule(&schedule, from).unwrap(); let expected = chrono::DateTime::parse_from_rfc3339("2026-06-16T01:00:00Z") .unwrap() .timestamp_millis(); - assert_eq!(next, expected); + assert_eq!(next_run_for_schedule(&schedule, from), Some(expected)); } } diff --git a/src/session/messenger.rs b/src/session/messenger.rs index c0fd0cd..5cb1bdd 100644 --- a/src/session/messenger.rs +++ b/src/session/messenger.rs @@ -4,7 +4,9 @@ use crate::bus::{ChatMessage, MediaItem, MessageSource, OutboundMessage, SourceK use crate::session::UnifiedSessionId; use crate::tools::{OutboundDelivery, OutboundMessenger}; -use super::persistence::{append_active_turn_message, append_persisted_messages}; +use super::persistence::{ + append_active_turn_message, append_persisted_message_if_absent, append_persisted_messages, +}; use super::session::{ CURRENT_SOURCE_SESSION, CURRENT_TURN_DELIVERIES, CURRENT_TURN_ID, PendingTurnDelivery, SessionManager, @@ -118,37 +120,108 @@ impl OutboundMessenger for SessionManager { } impl SessionManager { - pub async fn deliver_scheduled_message( + pub async fn deliver_scheduled_run( &self, - channel: &str, - chat_id: &str, - job_id: &str, - job_name: &str, - content: &str, - ) -> Result<(), String> { - ::send_message( - self, - channel, - chat_id, - None, - content, - MessageSource { - kind: SourceKind::ExternalTrigger, - from_channel: Some("scheduler".to_string()), - from_session: Some(format!("cron:{job_id}")), - from_user_id: None, - system_name: Some(job_name.to_string()), - task_id: Some(job_id.to_string()), - from_run_id: None, - from_agent_id: None, - }, - Vec::new(), - ) - .await - .map(|_| ()) + run: &crate::storage::JobRun, + delivery_owner: &str, + ) -> Result<(), ScheduledDeliveryError> { + let content = run + .message + .as_deref() + .unwrap_or("定时任务已结束，但没有生成可投递的结果。请在任务运行记录中查看诊断信息。"); + let target_sid = if let Some(session_id) = run.target_session_id.as_deref() { + UnifiedSessionId::parse(session_id).ok_or_else(|| { + ScheduledDeliveryError::Permanent("stored target session is invalid".to_string()) + })? + } else { + let resolved = self + .resolve_dialog_id(&run.target_channel, &run.target_chat_id) + .await + .map_err(|error| ScheduledDeliveryError::Transient(error.to_string()))?; + let fixed = self + .storage + .set_scheduled_delivery_target_session( + run.id, + delivery_owner, + &resolved.to_string(), + chrono::Utc::now().timestamp_millis(), + ) + .await + .map_err(|error| ScheduledDeliveryError::Transient(error.to_string()))? + .ok_or_else(|| { + ScheduledDeliveryError::Transient( + "scheduled delivery lost its claim before fixing target session" + .to_string(), + ) + })?; + UnifiedSessionId::parse(&fixed).ok_or_else(|| { + ScheduledDeliveryError::Permanent("fixed target session is invalid".to_string()) + })? + }; + if target_sid.channel != run.target_channel || target_sid.chat_id != run.target_chat_id { + return Err(ScheduledDeliveryError::Permanent( + "fixed target session does not belong to the scheduled destination".to_string(), + )); + } + let session = self + .get_or_activate_session(&target_sid) + .await + .map_err(|error| ScheduledDeliveryError::Transient(error.to_string()))?; + let source = MessageSource { + kind: SourceKind::ExternalTrigger, + from_channel: Some("scheduler".to_string()), + from_session: Some(format!("scheduled-run:{}", run.id)), + from_user_id: None, + system_name: Some("scheduled task".to_string()), + task_id: Some(run.job_id.clone()), + from_run_id: run.agent_run_id.clone(), + from_agent_id: run.agent_id.clone(), + }; + let mut message = outbound_history_message(content, source, &[]); + message.id = format!("scheduled:{}", run.id); + let message_id = message.id.clone(); + append_persisted_message_if_absent(&session, message) + .await + .map_err(|error| ScheduledDeliveryError::Transient(error.to_string()))?; + + let metadata = HashMap::from([ + ("_session_id".to_string(), target_sid.to_string()), + ("_message_id".to_string(), message_id), + ("scheduled_delivery_id".to_string(), run.id.to_string()), + ]); + self.bus + .deliver_outbound(OutboundMessage { + channel: run.target_channel.clone(), + chat_id: run.target_chat_id.clone(), + content: content.to_string(), + reply_to: None, + media: Vec::new(), + metadata, + delivery: None, + }) + .await + .map_err(|error| match error { + crate::bus::BusError::Closed + | crate::bus::BusError::DeliveryTimedOut + | crate::bus::BusError::DeliveryTransient(_) => { + ScheduledDeliveryError::Transient(error.to_string()) + } + crate::bus::BusError::DeliveryPermanent(summary) => { + ScheduledDeliveryError::Permanent(summary) + } + })?; + Ok(()) } } +#[derive(Debug, thiserror::Error)] +pub enum ScheduledDeliveryError { + #[error("transient scheduled delivery failure: {0}")] + Transient(String), + #[error("permanent scheduled delivery failure: {0}")] + Permanent(String), +} + fn outbound_history_message( content: impl Into, source: MessageSource, diff --git a/src/session/mod.rs b/src/session/mod.rs index 011a921..0c4b0cc 100644 --- a/src/session/mod.rs +++ b/src/session/mod.rs @@ -14,6 +14,7 @@ pub mod turn; pub use commands::SessionCommand; pub use error::SessionError; pub use events::{DialogInfo, SessionEvent}; +pub use messenger::ScheduledDeliveryError; pub use session::{ AgentCatalogPreparation, SLASH_COMMANDS, Session, SessionManager, SessionManagerServices, SlashCommand, diff --git a/src/session/persistence.rs b/src/session/persistence.rs index d50181d..2362faa 100644 --- a/src/session/persistence.rs +++ b/src/session/persistence.rs @@ -101,6 +101,41 @@ pub(super) async fn append_persisted_messages_with_meta( append_persisted_messages_inner(session, messages, VersionPolicy::Advance, None, None).await } +pub(super) async fn append_persisted_message_if_absent( + session: &Arc>, + message: ChatMessage, +) -> Result { + let persistence_lock = { session.lock().await.persistence_lock.clone() }; + let _persistence_guard = persistence_lock.lock().await; + let message_id = message.id.clone(); + let snapshot = { + let guard = session.lock().await; + if guard.contains_message_id(&message_id) { + return Ok(false); + } + guard.prepare_message_persist_snapshot(&message) + }; + let Some((storage, session_id, persisted, meta)) = snapshot else { + return Ok(false); + }; + let inserted = storage + .persist_message_if_absent(&session_id, &persisted, &meta) + .await?; + if !inserted { + return Ok(false); + } + let mut guard = session.lock().await; + if guard.contains_message_id(&message_id) { + return Ok(true); + } + if !guard.apply_prepared_message_in_memory(message, persisted.seq, persisted.created_at, true) { + return Err(StorageError::Conflict(format!( + "session changed while committing idempotent message {message_id}" + ))); + } + Ok(true) +} + pub(super) async fn append_persisted_turn_messages( session: &Arc>, messages: Vec, @@ -247,7 +282,7 @@ mod tests { } #[tokio::test] - async fn active_turn_side_effect_does_not_invalidate_its_session_version() { + async fn side_effect_messages_apply_the_expected_session_version_once() { let dir = tempfile::tempdir().unwrap(); let storage = Arc::new( crate::storage::Storage::new(&dir.path().join("memory.db")) @@ -255,7 +290,7 @@ mod tests { .unwrap(), ); let memory_manager = Arc::new(MemoryManager::new( - storage, + storage.clone(), "test".to_string(), "test".to_string(), )); @@ -276,12 +311,32 @@ mod tests { price_input_per_million: None, price_output_per_million: None, }; + let unified_id = crate::session::UnifiedSessionId::new("cli_chat", "chat", "dialog"); + let now = chrono::Utc::now().timestamp_millis(); + storage + .upsert_session(&crate::storage::session::SessionMeta { + id: unified_id.to_string(), + channel: "cli_chat".to_string(), + chat_id: "chat".to_string(), + dialog_id: "dialog".to_string(), + title: "test".to_string(), + created_at: now, + last_active_at: now, + message_count: 0, + routing_info: None, + archived_at: None, + deleted_at: None, + last_consolidated_at: None, + last_compressed_message_at: None, + }) + .await + .unwrap(); let session = Arc::new(Mutex::new( Session::new( - crate::session::UnifiedSessionId::new("cli_chat", "chat", "dialog"), + unified_id, config, Arc::new(ToolRegistry::new()), - None, + Some(storage.clone()), String::new(), "test".to_string(), super::super::session::SessionContextServices { @@ -320,5 +375,37 @@ mod tests { let guard = session.lock().await; assert_eq!(guard.state_version_for_test(), base_version + 1); assert_eq!(guard.get_history().len(), 2); + drop(guard); + + let idempotent_base_version = session.lock().await.state_version_for_test(); + let mut scheduled = ChatMessage::assistant("scheduled result"); + scheduled.id = "scheduled:42".to_string(); + assert!( + append_persisted_message_if_absent(&session, scheduled.clone()) + .await + .unwrap() + ); + assert!( + !append_persisted_message_if_absent(&session, scheduled) + .await + .unwrap() + ); + let guard = session.lock().await; + assert_eq!(guard.state_version_for_test(), idempotent_base_version + 1); + assert_eq!( + guard + .get_history() + .iter() + .filter(|message| message.id == "scheduled:42") + .count(), + 1 + ); + drop(guard); + let persisted: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM messages WHERE id = ?") + .bind("scheduled:42") + .fetch_one(storage.pool()) + .await + .unwrap(); + assert_eq!(persisted, 1); } } diff --git a/src/session/session.rs b/src/session/session.rs index 83977aa..60c6ab1 100644 --- a/src/session/session.rs +++ b/src/session/session.rs @@ -1083,59 +1083,101 @@ impl Session { persist: bool, advance_state_version: bool, ) -> Option { + let persist_snapshot = persist + .then(|| self.prepare_message_persist_snapshot(&message)) + .flatten(); + let (seq, now) = persist_snapshot + .as_ref() + .map(|(_, _, message, _)| (message.seq, message.created_at)) + .unwrap_or_else(|| (self.seq_counter, chrono::Utc::now().timestamp_millis())); + let applied = + self.apply_prepared_message_in_memory(message, seq, now, advance_state_version); + debug_assert!(applied, "fresh in-memory message snapshot must apply"); + persist_snapshot + } + + pub(super) fn prepare_message_persist_snapshot( + &self, + message: &ChatMessage, + ) -> Option { + let storage = self.storage.clone()?; let is_user = message.role == "user"; let counts_as_user_input = is_user && message.client_visibility == crate::bus::ClientVisibility::Visible; let now = chrono::Utc::now().timestamp_millis(); - - // Assign seq let seq = self.seq_counter; - self.seq_counter += 1; - - let persist_snapshot = if persist { - self.storage.clone().map(|storage| { - let msg_meta = crate::storage::message::MessageMeta { - id: message.id.clone(), - session_id: self.id.to_string(), - seq, - role: message.role.clone(), - content: message.content.clone(), - reasoning_content: message.reasoning_content.clone(), - provider_state: message - .provider_state - .as_ref() - .and_then(|state| serde_json::to_string(state).ok()), - turn_id: message.turn_id.clone(), - iteration: message.iteration.map(i64::from), - completion_status: message.completion_status, - client_visibility: message.client_visibility, - turn_origin: message.turn_origin, - media_refs: if message.media_refs.is_empty() { - None - } else { - Some(serde_json::to_string(&message.media_refs).unwrap_or_default()) - }, - tool_call_id: message.tool_call_id.clone(), - tool_name: message.tool_name.clone(), - tool_calls: message - .tool_calls - .as_ref() - .and_then(|tc| serde_json::to_string(tc).ok()), - source: message - .source - .as_ref() - .map(|s| serde_json::to_string(s).unwrap_or_default()), - created_at: now, - }; - (storage, self.id.to_string(), msg_meta) - }) - } else { - None + let msg_meta = crate::storage::message::MessageMeta { + id: message.id.clone(), + session_id: self.id.to_string(), + seq, + role: message.role.clone(), + content: message.content.clone(), + reasoning_content: message.reasoning_content.clone(), + provider_state: message + .provider_state + .as_ref() + .and_then(|state| serde_json::to_string(state).ok()), + turn_id: message.turn_id.clone(), + iteration: message.iteration.map(i64::from), + completion_status: message.completion_status, + client_visibility: message.client_visibility, + turn_origin: message.turn_origin, + media_refs: if message.media_refs.is_empty() { + None + } else { + Some(serde_json::to_string(&message.media_refs).unwrap_or_default()) + }, + tool_call_id: message.tool_call_id.clone(), + tool_name: message.tool_name.clone(), + tool_calls: message + .tool_calls + .as_ref() + .and_then(|tc| serde_json::to_string(tc).ok()), + source: message + .source + .as_ref() + .map(|s| serde_json::to_string(s).unwrap_or_default()), + created_at: now, }; + let session_id = self.id.to_string(); + let session_meta = crate::storage::session::SessionMeta { + id: session_id.clone(), + channel: self.id.channel.clone(), + chat_id: self.id.chat_id.clone(), + dialog_id: self.id.dialog_id.clone(), + title: self.title.clone(), + created_at: self.created_at, + last_active_at: now, + message_count: self.message_count + i64::from(counts_as_user_input), + routing_info: if self.routing_info.is_empty() { + None + } else { + Some(self.routing_info.clone()) + }, + archived_at: self.archived_at, + deleted_at: None, + last_consolidated_at: self.last_consolidated_at, + last_compressed_message_at: self.last_compressed_message_at, + }; + Some((storage, session_id, msg_meta, session_meta)) + } - // Update in-memory state + pub(super) fn apply_prepared_message_in_memory( + &mut self, + message: ChatMessage, + seq: i64, + now: i64, + advance_state_version: bool, + ) -> bool { + if self.seq_counter != seq || self.contains_message_id(&message.id) { + return false; + } + let is_user = message.role == "user"; + let counts_as_user_input = + is_user && message.client_visibility == crate::bus::ClientVisibility::Visible; self.message_seqs.insert(message.id.clone(), seq); self.messages.push(message); + self.seq_counter += 1; self.total_message_count += 1; if counts_as_user_input { self.message_count += 1; @@ -1144,29 +1186,11 @@ impl Session { if advance_state_version { self.state_version = self.state_version.wrapping_add(1); } + true + } - persist_snapshot.map(|(storage, session_id, msg_meta)| { - let session_meta = crate::storage::session::SessionMeta { - id: session_id.clone(), - channel: self.id.channel.clone(), - chat_id: self.id.chat_id.clone(), - dialog_id: self.id.dialog_id.clone(), - title: self.title.clone(), - created_at: self.created_at, - last_active_at: self.last_active_at, - message_count: self.message_count, - routing_info: if self.routing_info.is_empty() { - None - } else { - Some(self.routing_info.clone()) - }, - archived_at: self.archived_at, - deleted_at: None, - last_consolidated_at: self.last_consolidated_at, - last_compressed_message_at: self.last_compressed_message_at, - }; - (storage, session_id, msg_meta, session_meta) - }) + pub(super) fn contains_message_id(&self, message_id: &str) -> bool { + self.message_seqs.contains_key(message_id) } /// Roll back messages that were appended in memory but whose atomic @@ -1905,7 +1929,7 @@ pub struct SessionManager { provider_config: LLMProviderConfig, tools: Arc, skills_loader: Arc, - storage: Arc, + pub(super) storage: Arc, pub(super) bus: Arc, memory_manager: Arc, work_manager: Arc, @@ -2276,46 +2300,6 @@ impl SessionManager { self.work_manager.clone() } - /// 为定时任务创建一个无 session 绑定的 AgentLoop - pub fn create_cron_agent(&self) -> Result { - let tools = self.tools.without(&["reload_config"]); - let provider = create_provider(self.provider_config.clone()) - .map_err(|e| AgentError::Other(format!("failed to create cron provider: {}", e)))?; - Ok(AgentLoop::with_provider_and_tools( - Arc::from(provider), - tools, - self.provider_config.max_tool_iterations, - self.provider_config.model_id.clone(), - self.provider_config.workspace_dir.clone(), - self.provider_config.input_types.clone(), - ) - .with_context_window(self.provider_config.token_limit)) - } - - fn create_managed_scheduled_agent(&self) -> Result<(AgentLoop, Arc), AgentError> { - let tools = self.tools.without(&[ - "send_message", - "cron_add", - "cron_update", - "cron_remove", - "cron_enable", - "cron_disable", - "reload_config", - ]); - let provider = create_provider(self.provider_config.clone()) - .map_err(|e| AgentError::Other(format!("failed to create scheduled provider: {e}")))?; - let agent = AgentLoop::with_provider_and_tools( - Arc::from(provider), - tools.clone(), - self.provider_config.max_tool_iterations, - self.provider_config.model_id.clone(), - self.provider_config.workspace_dir.clone(), - self.provider_config.input_types.clone(), - ) - .with_context_window(self.provider_config.token_limit); - Ok((agent, tools)) - } - /// 获取所有可用的斜杠命令 pub fn get_slash_commands(&self) -> &[SlashCommand] { SLASH_COMMANDS @@ -5098,103 +5082,6 @@ impl SessionManager { } impl SessionManager { - /// - /// Runs in a stateless manner: no session creation, no history persistence. - /// The cron system prompt instructs the LLM to deliver results via the - /// `send_message` tool, which handles both delivery and history writing - /// on the target session. - pub async fn handle_cron_message( - &self, - channel: &str, - chat_id: &str, - prompt: &str, - job_id: &str, - job_name: &str, - ) -> Result { - let skills_prompt = self.skills_loader.build_skills_prompt(); - - let base_prompt = build_system_prompt( - &self.provider_config.workspace_dir, - &self.provider_config.model_id, - &self.tools, - ); - let cron_context = format!( - "## 定时任务执行\n\n\ - 你正在执行定时任务「{job_name}」({job_id})。\n\ - 目标渠道: {channel}:{chat_id}\n\n\ - 规则:\n\ - - 这不是聊天对话，没有用户会直接看到你的输出\n\ - - 你必须使用 send_message 工具将最终结果发送到目标渠道\n\ - - send_message 格式: target_chat_id=\"{channel}:{chat_id}\", content=\"消息内容\"\n\ - - 可以调用其他工具收集信息、处理任务，但最终消息必须通过 send_message 发送\n\ - - 只输出最终消息内容，不要输出中间思考过程或分析！" - ); - let full_system_prompt = - format!("{}\n\n{}\n\n{}", base_prompt, skills_prompt, cron_context); - - let history = vec![ - ChatMessage::system(full_system_prompt), - ChatMessage::user(prompt), - ]; - - let agent = self.create_cron_agent()?; - let source_session = format!("cron:{}", job_name); - let result = CURRENT_SOURCE_SESSION - .scope(Some(source_session.clone()), async { - agent - .process_with_context( - history, - ToolExecutionContext::for_session(source_session), - ) - .await - }) - .await - .inspect_err(|e| { - tracing::error!(error = %e, job_id = %job_id, "Cron agent processing error"); - })?; - - Ok(HandleResult::AgentResponse(result.final_response.content)) - } - - /// Execute a scheduler-managed task. The agent returns a result but cannot - /// deliver it itself; Scheduler applies the configured delivery policy. - pub async fn handle_managed_scheduled_message( - &self, - prompt: &str, - job_id: &str, - job_name: &str, - monitor: bool, - ) -> Result { - let (agent, tools) = self.create_managed_scheduled_agent()?; - let base_prompt = build_system_prompt( - &self.provider_config.workspace_dir, - &self.provider_config.model_id, - &tools, - ); - let skills_prompt = self.skills_loader.build_skills_prompt(); - let result_contract = if monitor { - "这是无人值守巡检。完成必要检查后：一切正常且无需用户关注时，只返回 NO_REPLY[INFO]: <简短原因>；发现问题时返回简洁、可操作的告警；无法完成时返回 NO_REPLY[FAIL]: <原因>；因安全或权限拒绝时返回 NO_REPLY[REFUSE]: <原因>。不要调用 send_message，不要把不确定当作正常。" - } else { - "这是 Scheduler 托管投递的定时任务。完成任务后只返回应交付给用户的最终内容，不要调用 send_message。" - }; - let system = format!( - "{base_prompt}\n\n{skills_prompt}\n\n## 定时任务执行\n任务「{job_name}」({job_id})。\n{result_contract}" - ); - let history = vec![ChatMessage::system(system), ChatMessage::user(prompt)]; - let source_session = format!("cron:{job_id}"); - let result = CURRENT_SOURCE_SESSION - .scope(Some(source_session.clone()), async { - agent - .process_with_context( - history, - ToolExecutionContext::for_session(source_session), - ) - .await - }) - .await?; - Ok(result.final_response.content) - } - pub async fn clear_session_history( &self, unified_id: &UnifiedSessionId, diff --git a/src/storage/agent_run.rs b/src/storage/agent_run.rs index b14e8af..7fcae51 100644 --- a/src/storage/agent_run.rs +++ b/src/storage/agent_run.rs @@ -1175,7 +1175,7 @@ mod tests { .fetch_one(storage.pool()) .await .unwrap(); - assert_eq!(version, 10); + assert_eq!(version, crate::storage::SCHEMA_VERSION); for table in [ "agent_runs", "agent_session_state", diff --git a/src/storage/mod.rs b/src/storage/mod.rs index 666d375..cec2be1 100644 --- a/src/storage/mod.rs +++ b/src/storage/mod.rs @@ -10,7 +10,10 @@ pub mod usage; pub use context_checkpoint::{ContextCheckpoint, ContextCheckpointState, NewContextCheckpoint}; pub use error::StorageError; -pub use scheduler::{DeliveryPolicy, JobKind, JobRun, ScheduledJob}; +pub use scheduler::{ + ClaimedScheduledRun, DeliveryPolicy, JobRun, ScheduledDeliveryStatus, ScheduledJob, + ScheduledJobUpdate, ScheduledOutcomeKind, ScheduledRunCompletion, ScheduledRunStatus, +}; pub use usage::{SessionUsageTotals, TurnUsageRecord}; use sqlx::sqlite::{ @@ -20,7 +23,7 @@ use sqlx::{Pool, Row, Sqlite}; use std::path::Path; use tokio::time::{Duration, sleep}; -const SCHEMA_VERSION: i64 = 10; +const SCHEMA_VERSION: i64 = 11; const INSERT_MESSAGE_SQL: &str = r#" INSERT INTO messages ( id, session_id, seq, role, content, reasoning_content, provider_state, @@ -29,6 +32,15 @@ const INSERT_MESSAGE_SQL: &str = r#" ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) "#; +const INSERT_MESSAGE_IF_ABSENT_SQL: &str = r#" + INSERT INTO messages ( + id, session_id, seq, role, content, reasoning_content, provider_state, + turn_id, iteration, completion_status, client_visibility, turn_origin, + media_refs, tool_call_id, tool_name, tool_calls, source, created_at + ) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(id) DO NOTHING +"#; pub(crate) fn insert_message_query<'a>( session_id: &'a str, @@ -55,6 +67,31 @@ pub(crate) fn insert_message_query<'a>( .bind(msg.created_at) } +pub(crate) fn insert_message_if_absent_query<'a>( + session_id: &'a str, + msg: &'a crate::storage::message::MessageMeta, +) -> sqlx::query::Query<'a, Sqlite, sqlx::sqlite::SqliteArguments> { + sqlx::query(INSERT_MESSAGE_IF_ABSENT_SQL) + .bind(&msg.id) + .bind(session_id) + .bind(msg.seq) + .bind(&msg.role) + .bind(&msg.content) + .bind(&msg.reasoning_content) + .bind(&msg.provider_state) + .bind(&msg.turn_id) + .bind(msg.iteration) + .bind(msg.completion_status.as_str()) + .bind(msg.client_visibility.as_str()) + .bind(msg.turn_origin.as_str()) + .bind(&msg.media_refs) + .bind(&msg.tool_call_id) + .bind(&msg.tool_name) + .bind(&msg.tool_calls) + .bind(&msg.source) + .bind(msg.created_at) +} + fn message_meta_from_row(row: SqliteRow) -> crate::storage::message::MessageMeta { let completion_status: String = row.get("completion_status"); crate::storage::message::MessageMeta { @@ -407,7 +444,6 @@ impl Storage { .execute(&self.pool) .await?; - Self::init_scheduler_schema(&self.pool).await?; self.migrate_schema().await?; Ok(()) @@ -428,7 +464,11 @@ impl Storage { return Ok(()); } - let mut tx = self.pool.begin().await?; + // Acquire the migration write lock before inspecting or modifying any + // legacy shape. This keeps the v11 rebuild on one connection and makes + // a concurrently running older Gateway fail startup cleanly instead of + // partially racing the schema migration. + let mut tx = self.pool.begin_with("BEGIN IMMEDIATE").await?; // The legacy drops below are a pre-v8 rebuild concern: the batch // "group" concept was removed in v8 and the old `background_tasks` // table in v7. Gate them on `current < 8` so a v8 -> v9 upgrade only @@ -458,6 +498,33 @@ impl Storage { .execute(&mut *tx) .await?; } + let legacy_scheduler_exists: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'scheduled_jobs'", + ) + .fetch_one(&mut *tx) + .await?; + let legacy_scheduler_exists = legacy_scheduler_exists == 1; + if legacy_scheduler_exists { + sqlx::query( + r#" + CREATE TABLE IF NOT EXISTS job_runs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + job_id TEXT NOT NULL REFERENCES scheduled_jobs(id) ON DELETE CASCADE, + started_at INTEGER NOT NULL, + finished_at INTEGER NOT NULL, + status TEXT NOT NULL, + output TEXT, + error TEXT, + duration_ms INTEGER NOT NULL, + result_kind TEXT, + delivery_status TEXT, + delivery_error TEXT + ) + "#, + ) + .execute(&mut *tx) + .await?; + } for (table, column, definition) in [ ("messages", "source", "source TEXT"), ("messages", "reasoning_content", "reasoning_content TEXT"), @@ -528,9 +595,10 @@ impl Storage { let columns = sqlx::query(sqlx::AssertSqlSafe(pragma)) .fetch_all(&mut *tx) .await?; - if !columns - .iter() - .any(|row| row.get::("name") == column) + if !columns.is_empty() + && !columns + .iter() + .any(|row| row.get::("name") == column) { let alter = format!("ALTER TABLE {table} ADD COLUMN {definition}"); // All identifiers and definitions come from the fixed migration list above. @@ -562,11 +630,6 @@ impl Storage { ) .execute(&mut *tx) .await?; - sqlx::query( - "CREATE INDEX IF NOT EXISTS idx_jobs_claimable ON scheduled_jobs(enabled, next_run_at, lease_until)", - ) - .execute(&mut *tx) - .await?; sqlx::query( r#" CREATE TABLE IF NOT EXISTS session_turn_usage ( @@ -629,6 +692,7 @@ impl Storage { for statement in agent_run::AGENT_SCHEMA_STATEMENTS { sqlx::query(*statement).execute(&mut *tx).await?; } + scheduler::migrate_scheduler_v11(&mut tx, legacy_scheduler_exists).await?; sqlx::query(sqlx::AssertSqlSafe(format!( "PRAGMA user_version = {SCHEMA_VERSION}" ))) @@ -638,70 +702,6 @@ impl Storage { Ok(()) } - /// Initialize the scheduler tables (idempotent). - pub(crate) async fn init_scheduler_schema(pool: &Pool) -> Result<(), StorageError> { - sqlx::query( - r#" - CREATE TABLE IF NOT EXISTS scheduled_jobs ( - id TEXT PRIMARY KEY, - name TEXT NOT NULL, - schedule TEXT NOT NULL, - prompt TEXT NOT NULL, - channel TEXT NOT NULL, - chat_id TEXT NOT NULL, - model TEXT, - job_kind TEXT NOT NULL DEFAULT 'task', - delivery_policy TEXT NOT NULL DEFAULT 'direct', - enabled INTEGER NOT NULL DEFAULT 1, - delete_after_run INTEGER NOT NULL DEFAULT 0, - next_run_at INTEGER NOT NULL, - last_run_at INTEGER, - last_status TEXT, - last_error TEXT, - locked_at INTEGER, - lock_owner TEXT, - lease_until INTEGER, - created_at INTEGER NOT NULL, - updated_at INTEGER NOT NULL - ) - "#, - ) - .execute(pool) - .await?; - - sqlx::query( - r#" - CREATE TABLE IF NOT EXISTS job_runs ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - job_id TEXT NOT NULL REFERENCES scheduled_jobs(id) ON DELETE CASCADE, - started_at INTEGER NOT NULL, - finished_at INTEGER NOT NULL, - status TEXT NOT NULL, - output TEXT, - error TEXT, - duration_ms INTEGER NOT NULL, - result_kind TEXT, - delivery_status TEXT, - delivery_error TEXT - ) - "#, - ) - .execute(pool) - .await?; - - sqlx::query( - "CREATE INDEX IF NOT EXISTS idx_jobs_next_run ON scheduled_jobs(enabled, next_run_at)", - ) - .execute(pool) - .await?; - - sqlx::query("CREATE INDEX IF NOT EXISTS idx_runs_job_id ON job_runs(job_id)") - .execute(pool) - .await?; - - Ok(()) - } - pub async fn append_llm_call( &self, provider: &str, @@ -962,6 +962,56 @@ impl Storage { Ok(msg.seq) } + pub async fn persist_message_if_absent( + &self, + session_id: &str, + msg: &crate::storage::message::MessageMeta, + meta: &crate::storage::session::SessionMeta, + ) -> Result { + let mut tx = self.pool.begin().await?; + let inserted = insert_message_if_absent_query(session_id, msg) + .execute(&mut *tx) + .await? + .rows_affected() + == 1; + if inserted { + sqlx::query( + r#" + INSERT INTO sessions (id, channel, chat_id, dialog_id, title, created_at, + last_active_at, message_count, routing_info, archived_at, deleted_at, + last_consolidated_at, last_compressed_message_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(id) DO UPDATE SET + title = excluded.title, + last_active_at = excluded.last_active_at, + message_count = excluded.message_count, + routing_info = excluded.routing_info, + archived_at = excluded.archived_at, + deleted_at = excluded.deleted_at, + last_consolidated_at = excluded.last_consolidated_at, + last_compressed_message_at = excluded.last_compressed_message_at + "#, + ) + .bind(&meta.id) + .bind(&meta.channel) + .bind(&meta.chat_id) + .bind(&meta.dialog_id) + .bind(&meta.title) + .bind(meta.created_at) + .bind(meta.last_active_at) + .bind(meta.message_count) + .bind(&meta.routing_info) + .bind(meta.archived_at) + .bind(meta.deleted_at) + .bind(meta.last_consolidated_at) + .bind(meta.last_compressed_message_at) + .execute(&mut *tx) + .await?; + } + tx.commit().await?; + Ok(inserted) + } + /// Atomically persist all messages produced by one logical turn together /// with the resulting session metadata. A turn is either fully visible /// after restart or not visible at all. @@ -1584,6 +1634,59 @@ mod tests { (storage, dir) } + async fn scheduler_schema_sql(storage: &Storage) -> Vec<(String, String, Option)> { + sqlx::query_as::<_, (String, String, Option)>( + r#" + SELECT type, name, sql FROM sqlite_master + WHERE name IN ( + 'scheduled_jobs', 'job_runs', 'idx_jobs_claimable', + 'idx_job_runs_job_finished', 'idx_job_runs_recovery', + 'idx_job_runs_delivery' + ) + ORDER BY type, name + "#, + ) + .fetch_all(storage.pool()) + .await + .unwrap() + } + + async fn create_v10_scheduler_schema(pool: &Pool) { + sqlx::query( + r#" + CREATE TABLE scheduled_jobs ( + id TEXT PRIMARY KEY, name TEXT NOT NULL, schedule TEXT NOT NULL, + prompt TEXT NOT NULL, channel TEXT NOT NULL, chat_id TEXT NOT NULL, + model TEXT, job_kind TEXT NOT NULL DEFAULT 'task', + delivery_policy TEXT NOT NULL DEFAULT 'direct', + enabled INTEGER NOT NULL DEFAULT 1, + delete_after_run INTEGER NOT NULL DEFAULT 0, next_run_at INTEGER NOT NULL, + last_run_at INTEGER, last_status TEXT, last_error TEXT, + locked_at INTEGER, lock_owner TEXT, lease_until INTEGER, + created_at INTEGER NOT NULL, updated_at INTEGER NOT NULL + ) + "#, + ) + .execute(pool) + .await + .unwrap(); + sqlx::query( + r#" + CREATE TABLE job_runs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + job_id TEXT NOT NULL REFERENCES scheduled_jobs(id) ON DELETE CASCADE, + started_at INTEGER NOT NULL, finished_at INTEGER NOT NULL, + status TEXT NOT NULL, output TEXT, error TEXT, + duration_ms INTEGER NOT NULL, result_kind TEXT, + delivery_status TEXT, delivery_error TEXT + ) + "#, + ) + .execute(pool) + .await + .unwrap(); + } + #[tokio::test] async fn sqlite_runtime_guards_are_enabled() { let (storage, _dir) = create_test_storage().await; @@ -1764,7 +1867,7 @@ mod tests { } #[tokio::test] - async fn legacy_schema_is_migrated_without_rebuild() { + async fn legacy_schema_is_migrated_to_canonical_v11() { let dir = tempfile::tempdir().unwrap(); let db_path = dir.path().join("legacy.db"); let pool = SqlitePoolOptions::new() @@ -1860,7 +1963,14 @@ mod tests { ), ( "scheduled_jobs", - vec!["locked_at", "lock_owner", "lease_until"], + vec![ + "agent_id", + "delivery_policy", + "last_outcome", + "locked_at", + "lock_owner", + "lease_until", + ], ), ] { let columns = sqlx::query(sqlx::AssertSqlSafe(format!("PRAGMA table_info({table})"))) @@ -1876,6 +1986,25 @@ mod tests { ); } } + let scheduler_columns = sqlx::query("PRAGMA table_info(scheduled_jobs)") + .fetch_all(storage.pool()) + .await + .unwrap() + .into_iter() + .map(|row| row.get::("name")) + .collect::>(); + for removed in [ + "model", + "job_kind", + "delete_after_run", + "last_status", + "last_error", + ] { + assert!( + !scheduler_columns.iter().any(|column| column == removed), + "legacy scheduler column survived v11: {removed}" + ); + } let schema_version: i64 = sqlx::query_scalar("PRAGMA user_version") .fetch_one(storage.pool()) .await @@ -1914,6 +2043,250 @@ mod tests { assert_eq!(origin, "user"); } + #[tokio::test] + async fn v10_scheduler_history_and_active_lease_migrate_to_v11() { + let dir = tempfile::tempdir().unwrap(); + let db_path = dir.path().join("v10-scheduler.db"); + let pool = SqlitePoolOptions::new() + .connect_with( + SqliteConnectOptions::new() + .filename(&db_path) + .create_if_missing(true), + ) + .await + .unwrap(); + create_v10_scheduler_schema(&pool).await; + sqlx::query( + r#" + INSERT INTO scheduled_jobs + (id, name, schedule, prompt, channel, chat_id, model, job_kind, + delivery_policy, enabled, next_run_at, last_run_at, last_status, + last_error, created_at, updated_at) + VALUES ('legacy-direct', 'Legacy direct', '{"type":"every","every_ms":60000}', + 'check it', 'cli_chat', 'chat', 'obsolete-model', 'monitor', + 'direct', 1, 2000, 1000, 'delivery_error', 'legacy diagnostic', 1, 2) + "#, + ) + .execute(&pool) + .await + .unwrap(); + sqlx::query( + r#" + INSERT INTO job_runs + (job_id, started_at, finished_at, status, output, error, duration_ms, + result_kind, delivery_status, delivery_error) + VALUES ('legacy-direct', 1000, 1100, 'delivery_error', 'legacy result', NULL, + 100, NULL, 'failed', 'channel rejected target') + "#, + ) + .execute(&pool) + .await + .unwrap(); + sqlx::query( + r#" + INSERT INTO scheduled_jobs + (id, name, schedule, prompt, channel, chat_id, job_kind, + delivery_policy, enabled, next_run_at, last_run_at, last_status, + locked_at, lock_owner, lease_until, created_at, updated_at) + VALUES ('legacy-locked', 'Legacy locked', '{"type":"every","every_ms":60000}', + 'check lock', 'cli_chat', 'chat', 'task', 'never', 1, 2000, + 1500, NULL, 1500, 'old-owner', 900000, 1, 2) + "#, + ) + .execute(&pool) + .await + .unwrap(); + sqlx::query("PRAGMA user_version = 10") + .execute(&pool) + .await + .unwrap(); + drop(pool); + + let storage = Storage::new(&db_path).await.unwrap(); + let migrated_columns = sqlx::query("PRAGMA table_info(scheduled_jobs)") + .fetch_all(storage.pool()) + .await + .unwrap() + .into_iter() + .map(|row| row.get::("name")) + .collect::>(); + assert!( + migrated_columns + .iter() + .any(|column| column == "last_outcome"), + "unexpected post-migration scheduler schema: {migrated_columns:?}" + ); + let direct = storage.get_scheduled_job("legacy-direct").await.unwrap(); + assert_eq!( + direct.delivery_policy, + crate::storage::DeliveryPolicy::Always + ); + assert_eq!( + direct.last_outcome, + Some(crate::storage::ScheduledOutcomeKind::Ok) + ); + let historical = storage + .list_scheduled_job_runs("legacy-direct", 10) + .await + .unwrap(); + assert_eq!(historical.len(), 1); + assert_eq!( + historical[0].status, + crate::storage::ScheduledRunStatus::Completed + ); + assert_eq!( + historical[0].outcome, + Some(crate::storage::ScheduledOutcomeKind::Ok) + ); + assert_eq!( + historical[0].delivery_status, + crate::storage::ScheduledDeliveryStatus::Failed + ); + assert_eq!( + historical[0].diagnostic.as_deref(), + Some("legacy diagnostic") + ); + + let recovered = storage + .list_scheduled_job_runs("legacy-locked", 10) + .await + .unwrap(); + assert_eq!(recovered.len(), 1); + assert_eq!( + recovered[0].status, + crate::storage::ScheduledRunStatus::Unknown + ); + assert_eq!( + recovered[0].delivery_status, + crate::storage::ScheduledDeliveryStatus::NotRequested + ); + let locked = storage.get_scheduled_job("legacy-locked").await.unwrap(); + assert!(locked.lock_owner.is_none()); + + let columns = sqlx::query("PRAGMA table_info(scheduled_jobs)") + .fetch_all(storage.pool()) + .await + .unwrap() + .into_iter() + .map(|row| row.get::("name")) + .collect::>(); + for removed in [ + "model", + "job_kind", + "delete_after_run", + "last_status", + "last_error", + ] { + assert!(!columns.iter().any(|column| column == removed)); + } + let violations = sqlx::query("PRAGMA foreign_key_check") + .fetch_all(storage.pool()) + .await + .unwrap(); + assert!(violations.is_empty()); + + let fresh_dir = tempfile::tempdir().unwrap(); + let fresh = Storage::new(&fresh_dir.path().join("fresh.db")) + .await + .unwrap(); + assert_eq!( + scheduler_schema_sql(&storage).await, + scheduler_schema_sql(&fresh).await + ); + } + + #[tokio::test] + async fn invalid_v10_scheduler_data_rolls_back_v11_migration() { + let dir = tempfile::tempdir().unwrap(); + let db_path = dir.path().join("invalid-v10-scheduler.db"); + let pool = SqlitePoolOptions::new() + .connect_with( + SqliteConnectOptions::new() + .filename(&db_path) + .create_if_missing(true), + ) + .await + .unwrap(); + create_v10_scheduler_schema(&pool).await; + sqlx::query( + r#" + INSERT INTO scheduled_jobs + (id, name, schedule, prompt, channel, chat_id, delivery_policy, + enabled, next_run_at, created_at, updated_at) + VALUES ('broken', 'Broken', '{not-json}', 'check', 'cli_chat', 'chat', + 'always', 1, 1000, 1, 1) + "#, + ) + .execute(&pool) + .await + .unwrap(); + sqlx::query("PRAGMA user_version = 10") + .execute(&pool) + .await + .unwrap(); + drop(pool); + + let error = match Storage::new(&db_path).await { + Ok(_) => panic!("invalid v10 scheduler data unexpectedly migrated"), + Err(error) => error, + }; + assert!(matches!(error, StorageError::Migration(_))); + + let pool = SqlitePoolOptions::new() + .connect_with(SqliteConnectOptions::new().filename(&db_path)) + .await + .unwrap(); + let version: i64 = sqlx::query_scalar("PRAGMA user_version") + .fetch_one(&pool) + .await + .unwrap(); + assert_eq!(version, 10); + let columns = sqlx::query("PRAGMA table_info(scheduled_jobs)") + .fetch_all(&pool) + .await + .unwrap() + .into_iter() + .map(|row| row.get::("name")) + .collect::>(); + assert!(columns.iter().any(|column| column == "job_kind")); + assert!(!columns.iter().any(|column| column == "agent_id")); + for table in ["scheduled_jobs_v10_legacy", "job_runs_v10_legacy"] { + let exists: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = ?", + ) + .bind(table) + .fetch_one(&pool) + .await + .unwrap(); + assert_eq!(exists, 0); + } + } + + #[tokio::test] + async fn newer_schema_version_is_rejected() { + let dir = tempfile::tempdir().unwrap(); + let db_path = dir.path().join("future.db"); + let pool = SqlitePoolOptions::new() + .connect_with( + SqliteConnectOptions::new() + .filename(&db_path) + .create_if_missing(true), + ) + .await + .unwrap(); + sqlx::query("PRAGMA user_version = 12") + .execute(&pool) + .await + .unwrap(); + drop(pool); + + let error = match Storage::new(&db_path).await { + Ok(_) => panic!("newer schema version unexpectedly opened"), + Err(error) => error, + }; + assert!(matches!(error, StorageError::Migration(_))); + } + #[tokio::test] async fn v3_migration_preserves_existing_reasoning_and_defaults_completion() { let dir = tempfile::tempdir().unwrap(); diff --git a/src/storage/scheduler.rs b/src/storage/scheduler.rs index a294046..0ff7e83 100644 --- a/src/storage/scheduler.rs +++ b/src/storage/scheduler.rs @@ -1,29 +1,13 @@ use serde::{Deserialize, Serialize}; -use sqlx::Row; +use sqlx::{Row, Sqlite, Transaction}; -use crate::scheduler::Schedule; -use crate::storage::StorageError; - -#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] -#[serde(rename_all = "snake_case")] -pub enum JobKind { - Task, - Monitor, -} - -impl JobKind { - pub fn as_str(self) -> &'static str { - match self { - Self::Task => "task", - Self::Monitor => "monitor", - } - } -} +use crate::scheduler::{Schedule, next_run_for_schedule}; +use crate::storage::agent_run::AgentTerminalOutcome; +use crate::storage::{Storage, StorageError}; #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "snake_case")] pub enum DeliveryPolicy { - Direct, Always, OnAlert, Never, @@ -32,84 +16,353 @@ pub enum DeliveryPolicy { impl DeliveryPolicy { pub fn as_str(self) -> &'static str { match self { - Self::Direct => "direct", Self::Always => "always", Self::OnAlert => "on_alert", Self::Never => "never", } } + + pub fn parse(value: &str) -> Result { + match value { + "always" => Ok(Self::Always), + "on_alert" => Ok(Self::OnAlert), + "never" => Ok(Self::Never), + other => Err(StorageError::Serialization(format!( + "unknown delivery policy: {other}" + ))), + } + } + + pub fn should_deliver(self, outcome: ScheduledOutcomeKind) -> bool { + match self { + Self::Always => true, + Self::OnAlert => outcome != ScheduledOutcomeKind::Ok, + Self::Never => false, + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum ScheduledOutcomeKind { + Ok, + Alert, + Failed, + Refused, + Unknown, +} + +impl ScheduledOutcomeKind { + pub fn as_str(self) -> &'static str { + match self { + Self::Ok => "ok", + Self::Alert => "alert", + Self::Failed => "failed", + Self::Refused => "refused", + Self::Unknown => "unknown", + } + } + + pub fn parse(value: &str) -> Result { + match value { + "ok" => Ok(Self::Ok), + "alert" => Ok(Self::Alert), + "failed" => Ok(Self::Failed), + "refused" => Ok(Self::Refused), + "unknown" => Ok(Self::Unknown), + other => Err(StorageError::Serialization(format!( + "unknown scheduled outcome: {other}" + ))), + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum ScheduledRunStatus { + Claimed, + Running, + Completed, + Failed, + TimedOut, + Cancelled, + Interrupted, + Unknown, +} + +impl ScheduledRunStatus { + pub fn as_str(self) -> &'static str { + match self { + Self::Claimed => "claimed", + Self::Running => "running", + Self::Completed => "completed", + Self::Failed => "failed", + Self::TimedOut => "timed_out", + Self::Cancelled => "cancelled", + Self::Interrupted => "interrupted", + Self::Unknown => "unknown", + } + } + + pub fn parse(value: &str) -> Result { + match value { + "claimed" => Ok(Self::Claimed), + "running" => Ok(Self::Running), + "completed" => Ok(Self::Completed), + "failed" => Ok(Self::Failed), + "timed_out" => Ok(Self::TimedOut), + "cancelled" => Ok(Self::Cancelled), + "interrupted" => Ok(Self::Interrupted), + "unknown" => Ok(Self::Unknown), + other => Err(StorageError::Serialization(format!( + "unknown scheduled run status: {other}" + ))), + } + } + + pub fn is_terminal(self) -> bool { + !matches!(self, Self::Claimed | Self::Running) + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum ScheduledDeliveryStatus { + AwaitingResult, + NotRequested, + Suppressed, + Pending, + Delivering, + Delivered, + Failed, +} + +impl ScheduledDeliveryStatus { + pub fn as_str(self) -> &'static str { + match self { + Self::AwaitingResult => "awaiting_result", + Self::NotRequested => "not_requested", + Self::Suppressed => "suppressed", + Self::Pending => "pending", + Self::Delivering => "delivering", + Self::Delivered => "delivered", + Self::Failed => "failed", + } + } + + pub fn parse(value: &str) -> Result { + match value { + "awaiting_result" => Ok(Self::AwaitingResult), + "not_requested" => Ok(Self::NotRequested), + "suppressed" => Ok(Self::Suppressed), + "pending" => Ok(Self::Pending), + "delivering" => Ok(Self::Delivering), + "delivered" => Ok(Self::Delivered), + "failed" => Ok(Self::Failed), + other => Err(StorageError::Serialization(format!( + "unknown scheduled delivery status: {other}" + ))), + } + } } -/// A scheduled job stored in the database. #[derive(Debug, Clone, Serialize, Deserialize)] pub struct ScheduledJob { pub id: String, pub name: String, - /// JSON-serialized `Schedule` stored as TEXT in SQLite. pub schedule: Schedule, pub prompt: String, + pub agent_id: Option, pub channel: String, pub chat_id: String, - pub model: Option, - pub job_kind: JobKind, pub delivery_policy: DeliveryPolicy, pub enabled: bool, - pub delete_after_run: bool, pub next_run_at: i64, pub last_run_at: Option, - pub last_status: Option, - pub last_error: Option, + pub last_outcome: Option, pub created_at: i64, pub updated_at: i64, + pub locked_at: Option, + pub lock_owner: Option, + pub lease_until: Option, } -/// A single execution record for a job. #[derive(Debug, Clone, Serialize, Deserialize)] pub struct JobRun { pub id: i64, pub job_id: String, - pub started_at: i64, - pub finished_at: i64, - pub status: String, - pub output: Option, - pub error: Option, - pub duration_ms: i64, - pub result_kind: Option, - pub delivery_status: Option, + pub scheduled_for: i64, + pub agent_run_id: Option, + pub agent_id: Option, + pub delivery_policy: DeliveryPolicy, + pub target_channel: String, + pub target_chat_id: String, + #[serde(skip_serializing)] + pub target_session_id: Option, + pub started_at: Option, + pub finished_at: Option, + pub status: ScheduledRunStatus, + pub outcome: Option, + pub message: Option, + pub diagnostic: Option, + pub duration_ms: Option, + pub delivery_status: ScheduledDeliveryStatus, + pub delivery_attempts: i64, + pub delivery_next_attempt_at: Option, + #[serde(skip_serializing)] + pub delivery_lease_owner: Option, + #[serde(skip_serializing)] + pub delivery_lease_until: Option, pub delivery_error: Option, + pub created_at: i64, + pub updated_at: i64, } -impl crate::storage::Storage { - /// Seed the built-in maintenance monitor once. `INSERT OR IGNORE` preserves - /// user changes such as disabling or editing the task. +#[derive(Debug, Clone)] +pub struct ClaimedScheduledRun { + pub job: ScheduledJob, + pub run_id: i64, + pub scheduled_for: i64, + pub owner: String, +} + +#[derive(Debug, Clone)] +pub struct ScheduledRunCompletion { + pub status: ScheduledRunStatus, + pub outcome: ScheduledOutcomeKind, + pub message: String, + pub diagnostic: Option, + pub duration_ms: i64, +} + +#[derive(Debug, Clone, Default)] +pub struct ScheduledJobUpdate { + pub name: Option, + pub prompt: Option, + pub schedule: Option, + pub agent_id: Option>, + pub channel: Option, + pub chat_id: Option, + pub delivery_policy: Option, +} + +pub(crate) const SCHEDULER_V11_SCHEMA_STATEMENTS: &[&str] = &[ + r#" + CREATE TABLE IF NOT EXISTS scheduled_jobs ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + schedule TEXT NOT NULL, + prompt TEXT NOT NULL, + agent_id TEXT, + channel TEXT NOT NULL, + chat_id TEXT NOT NULL, + delivery_policy TEXT NOT NULL CHECK (delivery_policy IN ('always','on_alert','never')), + enabled INTEGER NOT NULL DEFAULT 1 CHECK (enabled IN (0, 1)), + next_run_at INTEGER NOT NULL, + last_run_at INTEGER, + last_outcome TEXT CHECK (last_outcome IS NULL OR last_outcome IN ('ok','alert','failed','refused','unknown')), + locked_at INTEGER, + lock_owner TEXT, + lease_until INTEGER, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + CHECK (length(trim(id)) > 0), + CHECK (length(trim(name)) > 0), + CHECK (length(trim(prompt)) > 0), + CHECK (length(trim(channel)) > 0), + CHECK (length(trim(chat_id)) > 0), + CHECK (agent_id IS NULL OR length(trim(agent_id)) > 0), + CHECK ((locked_at IS NULL AND lock_owner IS NULL AND lease_until IS NULL) OR + (locked_at IS NOT NULL AND lock_owner IS NOT NULL AND lease_until IS NOT NULL)) + ) + "#, + "CREATE INDEX IF NOT EXISTS idx_jobs_claimable ON scheduled_jobs(enabled, next_run_at, lease_until)", + r#" + CREATE TABLE IF NOT EXISTS job_runs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + job_id TEXT NOT NULL REFERENCES scheduled_jobs(id) ON DELETE CASCADE, + scheduled_for INTEGER NOT NULL, + agent_run_id TEXT UNIQUE REFERENCES agent_runs(id) ON DELETE SET NULL, + agent_id TEXT, + delivery_policy TEXT NOT NULL CHECK (delivery_policy IN ('always','on_alert','never')), + target_channel TEXT NOT NULL, + target_chat_id TEXT NOT NULL, + target_session_id TEXT, + started_at INTEGER, + finished_at INTEGER, + status TEXT NOT NULL CHECK (status IN ('claimed','running','completed','failed','timed_out','cancelled','interrupted','unknown')), + outcome TEXT CHECK (outcome IS NULL OR outcome IN ('ok','alert','failed','refused','unknown')), + message TEXT, + diagnostic TEXT, + duration_ms INTEGER, + delivery_status TEXT NOT NULL CHECK (delivery_status IN ('awaiting_result','not_requested','suppressed','pending','delivering','delivered','failed')), + delivery_attempts INTEGER NOT NULL DEFAULT 0, + delivery_next_attempt_at INTEGER, + delivery_lease_owner TEXT, + delivery_lease_until INTEGER, + delivery_error TEXT, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + CHECK (agent_id IS NULL OR length(trim(agent_id)) > 0), + CHECK (length(trim(target_channel)) > 0), + CHECK (length(trim(target_chat_id)) > 0), + CHECK (target_session_id IS NULL OR length(trim(target_session_id)) > 0), + CHECK (delivery_attempts >= 0), + CHECK (duration_ms IS NULL OR duration_ms >= 0), + CHECK ( + (status IN ('claimed','running') AND outcome IS NULL AND finished_at IS NULL AND delivery_status = 'awaiting_result') OR + (status = 'completed' AND outcome IS NOT NULL AND outcome IN ('ok','alert','failed','refused') AND finished_at IS NOT NULL AND delivery_status != 'awaiting_result') OR + (status IN ('failed','timed_out','cancelled','interrupted') AND outcome IS NOT NULL AND outcome = 'failed' AND finished_at IS NOT NULL AND delivery_status != 'awaiting_result') OR + (status = 'unknown' AND outcome IS NOT NULL AND outcome = 'unknown' AND finished_at IS NOT NULL AND delivery_status != 'awaiting_result') + ), + CHECK ((delivery_lease_owner IS NULL AND delivery_lease_until IS NULL) OR + (delivery_lease_owner IS NOT NULL AND delivery_lease_until IS NOT NULL)) + ) + "#, + "CREATE INDEX IF NOT EXISTS idx_job_runs_job_finished ON job_runs(job_id, finished_at DESC)", + "CREATE INDEX IF NOT EXISTS idx_job_runs_recovery ON job_runs(status, updated_at)", + "CREATE INDEX IF NOT EXISTS idx_job_runs_delivery ON job_runs(delivery_status, delivery_next_attempt_at, delivery_lease_until)", +]; + +impl Storage { + #[cfg(test)] + pub(crate) async fn init_scheduler_schema( + pool: &sqlx::Pool, + ) -> Result<(), StorageError> { + for statement in SCHEDULER_V11_SCHEMA_STATEMENTS { + sqlx::query(*statement).execute(pool).await?; + } + Ok(()) + } + pub async fn ensure_default_maintenance_job(&self) -> Result<(), StorageError> { let now = now_ms(); let job = ScheduledJob { id: "picobot-routine-maintenance".to_string(), name: "PicoBot 日常维护巡检".to_string(), schedule: Schedule::Every { every_ms: 86_400_000 }, - prompt: "调用 routine_maintenance 工具恰好一次。检查工具结果；成功时返回 NO_REPLY[INFO]: 日常维护完成；工具失败或结果不完整时返回 NO_REPLY[FAIL]: <原因>。不要删除 knowledge 类型的长期记忆。".to_string(), + prompt: "调用 routine_maintenance 工具恰好一次。检查工具结果并通过 complete_scheduled_run 提交：成功且无需关注时 outcome=ok；工具失败、结果不完整或需要人工处理时 outcome=failed 或 alert。不要删除 knowledge 类型的长期记忆。".to_string(), + agent_id: None, channel: "cli_chat".to_string(), chat_id: "maintenance".to_string(), - model: None, - job_kind: JobKind::Monitor, delivery_policy: DeliveryPolicy::Never, enabled: true, - delete_after_run: false, next_run_at: now.saturating_add(30 * 60 * 1000), last_run_at: None, - last_status: None, - last_error: None, + last_outcome: None, created_at: now, updated_at: now, + locked_at: None, + lock_owner: None, + lease_until: None, }; let schedule_json = serialize_schedule(&job.schedule)?; sqlx::query( r#" INSERT OR IGNORE INTO scheduled_jobs - (id, name, schedule, prompt, channel, chat_id, model, job_kind, - delivery_policy, enabled, delete_after_run, next_run_at, - last_run_at, last_status, last_error, created_at, updated_at) + (id, name, schedule, prompt, agent_id, channel, chat_id, delivery_policy, + enabled, next_run_at, last_run_at, last_outcome, locked_at, lock_owner, + lease_until, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) "#, ) @@ -117,17 +370,17 @@ impl crate::storage::Storage { .bind(&job.name) .bind(schedule_json) .bind(&job.prompt) + .bind(&job.agent_id) .bind(&job.channel) .bind(&job.chat_id) - .bind(&job.model) - .bind(job.job_kind.as_str()) .bind(job.delivery_policy.as_str()) .bind(job.enabled as i32) - .bind(job.delete_after_run as i32) .bind(job.next_run_at) .bind(job.last_run_at) - .bind(&job.last_status) - .bind(&job.last_error) + .bind(job.last_outcome.map(ScheduledOutcomeKind::as_str)) + .bind(job.locked_at) + .bind(&job.lock_owner) + .bind(job.lease_until) .bind(job.created_at) .bind(job.updated_at) .execute(self.pool()) @@ -135,15 +388,15 @@ impl crate::storage::Storage { Ok(()) } - /// Insert a new scheduled job. pub async fn add_scheduled_job(&self, job: &ScheduledJob) -> Result<(), StorageError> { + validate_job(job)?; let schedule_json = serialize_schedule(&job.schedule)?; sqlx::query( r#" INSERT INTO scheduled_jobs - (id, name, schedule, prompt, channel, chat_id, model, - job_kind, delivery_policy, enabled, delete_after_run, next_run_at, last_run_at, - last_status, last_error, created_at, updated_at) + (id, name, schedule, prompt, agent_id, channel, chat_id, delivery_policy, + enabled, next_run_at, last_run_at, last_outcome, locked_at, lock_owner, + lease_until, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) "#, ) @@ -151,17 +404,17 @@ impl crate::storage::Storage { .bind(&job.name) .bind(&schedule_json) .bind(&job.prompt) + .bind(&job.agent_id) .bind(&job.channel) .bind(&job.chat_id) - .bind(&job.model) - .bind(job.job_kind.as_str()) .bind(job.delivery_policy.as_str()) .bind(job.enabled as i32) - .bind(job.delete_after_run as i32) .bind(job.next_run_at) .bind(job.last_run_at) - .bind(&job.last_status) - .bind(&job.last_error) + .bind(job.last_outcome.map(ScheduledOutcomeKind::as_str)) + .bind(job.locked_at) + .bind(&job.lock_owner) + .bind(job.lease_until) .bind(job.created_at) .bind(job.updated_at) .execute(self.pool()) @@ -169,7 +422,6 @@ impl crate::storage::Storage { Ok(()) } - /// Fetch a single scheduled job by ID. pub async fn get_scheduled_job(&self, id: &str) -> Result { let row = sqlx::query("SELECT * FROM scheduled_jobs WHERE id = ?") .bind(id) @@ -179,7 +431,6 @@ impl crate::storage::Storage { row_to_job(&row) } - /// List all scheduled jobs, ordered by next_run_at ascending. pub async fn list_scheduled_jobs(&self) -> Result, StorageError> { let rows = sqlx::query("SELECT * FROM scheduled_jobs ORDER BY next_run_at ASC") .fetch_all(self.pool()) @@ -187,301 +438,1319 @@ impl crate::storage::Storage { rows.iter().map(row_to_job).collect() } - /// Delete a scheduled job (cascades to job_runs). pub async fn remove_scheduled_job(&self, id: &str) -> Result<(), StorageError> { - sqlx::query("DELETE FROM scheduled_jobs WHERE id = ?") - .bind(id) - .execute(self.pool()) - .await?; + let result = sqlx::query( + r#" + DELETE FROM scheduled_jobs + WHERE id = ? AND lock_owner IS NULL + AND NOT EXISTS ( + SELECT 1 FROM job_runs + WHERE job_id = scheduled_jobs.id + AND (status IN ('claimed','running') + OR delivery_status IN ('pending','delivering')) + ) + "#, + ) + .bind(id) + .execute(self.pool()) + .await?; + if result.rows_affected() == 0 { + if self.get_scheduled_job(id).await.is_ok() { + return Err(StorageError::Conflict(format!( + "scheduled job {id} has an active run or pending delivery" + ))); + } + return Err(StorageError::NotFound(format!("scheduled job {id}"))); + } Ok(()) } - /// Enable or disable a scheduled job. pub async fn set_scheduled_job_enabled( &self, id: &str, enabled: bool, ) -> Result<(), StorageError> { - sqlx::query("UPDATE scheduled_jobs SET enabled = ?, updated_at = ? WHERE id = ?") - .bind(enabled as i32) - .bind(now_ms()) - .bind(id) - .execute(self.pool()) - .await?; + let result = + sqlx::query("UPDATE scheduled_jobs SET enabled = ?, updated_at = ? WHERE id = ?") + .bind(enabled as i32) + .bind(now_ms()) + .bind(id) + .execute(self.pool()) + .await?; + if result.rows_affected() == 0 { + return Err(StorageError::NotFound(format!("scheduled job {id}"))); + } Ok(()) } - pub async fn set_scheduled_job_behavior( - &self, - id: &str, - job_kind: Option, - delivery_policy: Option, - ) -> Result<(), StorageError> { - let current = self.get_scheduled_job(id).await?; - sqlx::query( - "UPDATE scheduled_jobs SET job_kind = ?, delivery_policy = ?, updated_at = ? WHERE id = ?", - ) - .bind(job_kind.unwrap_or(current.job_kind).as_str()) - .bind(delivery_policy.unwrap_or(current.delivery_policy).as_str()) - .bind(now_ms()) - .bind(id) - .execute(self.pool()) - .await?; - Ok(()) - } - - /// Update selective fields on a scheduled job. pub async fn update_scheduled_job( &self, id: &str, - prompt: Option, - schedule: Option, - channel: Option, - chat_id: Option, - model: Option, + update: ScheduledJobUpdate, + next_run_at: Option, ) -> Result<(), StorageError> { - let now = now_ms(); - - if let Some(p) = prompt { - sqlx::query("UPDATE scheduled_jobs SET prompt = ?, updated_at = ? WHERE id = ?") - .bind(&p) - .bind(now) - .bind(id) - .execute(self.pool()) - .await?; - } - if let Some(s) = schedule { - let json = serialize_schedule(&s)?; - sqlx::query("UPDATE scheduled_jobs SET schedule = ?, updated_at = ? WHERE id = ?") - .bind(&json) - .bind(now) - .bind(id) - .execute(self.pool()) - .await?; - } - if let Some(c) = channel { - sqlx::query("UPDATE scheduled_jobs SET channel = ?, updated_at = ? WHERE id = ?") - .bind(&c) - .bind(now) - .bind(id) - .execute(self.pool()) - .await?; - } - if let Some(c) = chat_id { - sqlx::query("UPDATE scheduled_jobs SET chat_id = ?, updated_at = ? WHERE id = ?") - .bind(&c) - .bind(now) - .bind(id) - .execute(self.pool()) - .await?; - } - if let Some(m) = model { - sqlx::query("UPDATE scheduled_jobs SET model = ?, updated_at = ? WHERE id = ?") - .bind(&m) - .bind(now) - .bind(id) - .execute(self.pool()) - .await?; - } - Ok(()) - } - - /// Update next_run_at and last_run_at for a job. - pub async fn set_scheduled_job_next_run( - &self, - id: &str, - next_run_at: i64, - ) -> Result<(), StorageError> { - let now = now_ms(); + let current = self.get_scheduled_job(id).await?; + let schedule = update.schedule.unwrap_or(current.schedule); + let schedule_json = serialize_schedule(&schedule)?; + let name = update.name.unwrap_or(current.name); + let prompt = update.prompt.unwrap_or(current.prompt); + let agent_id = update.agent_id.unwrap_or(current.agent_id); + let channel = update.channel.unwrap_or(current.channel); + let chat_id = update.chat_id.unwrap_or(current.chat_id); + let delivery_policy = update.delivery_policy.unwrap_or(current.delivery_policy); + let next_run_at = next_run_at.unwrap_or(current.next_run_at); + let candidate = ScheduledJob { + id: id.to_string(), + name: name.clone(), + schedule, + prompt: prompt.clone(), + agent_id: agent_id.clone(), + channel: channel.clone(), + chat_id: chat_id.clone(), + delivery_policy, + enabled: current.enabled, + next_run_at, + last_run_at: current.last_run_at, + last_outcome: current.last_outcome, + created_at: current.created_at, + updated_at: now_ms(), + locked_at: current.locked_at, + lock_owner: current.lock_owner, + lease_until: current.lease_until, + }; + validate_job(&candidate)?; sqlx::query( - "UPDATE scheduled_jobs SET next_run_at = ?, last_run_at = ?, updated_at = ? WHERE id = ?", + r#" + UPDATE scheduled_jobs + SET name = ?, prompt = ?, schedule = ?, agent_id = ?, channel = ?, chat_id = ?, + delivery_policy = ?, next_run_at = ?, updated_at = ? + WHERE id = ? + "#, ) + .bind(name) + .bind(prompt) + .bind(schedule_json) + .bind(agent_id) + .bind(channel) + .bind(chat_id) + .bind(delivery_policy.as_str()) .bind(next_run_at) - .bind(now) - .bind(now) + .bind(candidate.updated_at) .bind(id) .execute(self.pool()) .await?; Ok(()) } - /// Atomically claim due jobs for one scheduler instance. A crashed worker's - /// claims become eligible again after `lease_until`. - pub async fn claim_due_scheduled_jobs( + pub async fn claim_due_scheduled_runs( &self, now: i64, lease_until: i64, owner: &str, limit: usize, - ) -> Result, StorageError> { + ) -> Result, StorageError> { + const RETRY_DELAYS_MS: &[u64] = &[25, 50, 100]; + for (attempt, delay) in RETRY_DELAYS_MS.iter().enumerate() { + match self + .claim_due_scheduled_runs_once(now, lease_until, owner, limit) + .await + { + Ok(claimed) => return Ok(claimed), + Err(error) if attempt < RETRY_DELAYS_MS.len() - 1 && error.is_transient() => { + tokio::time::sleep(std::time::Duration::from_millis(*delay)).await; + } + Err(error) => return Err(error), + } + } + unreachable!() + } + + async fn claim_due_scheduled_runs_once( + &self, + now: i64, + lease_until: i64, + owner: &str, + limit: usize, + ) -> Result, StorageError> { if limit == 0 { return Ok(Vec::new()); } + let mut tx = self.pool().begin().await?; let rows = sqlx::query( r#" - UPDATE scheduled_jobs - SET locked_at = ?, lock_owner = ?, lease_until = ?, last_run_at = ?, updated_at = ? - WHERE id IN ( - SELECT id FROM scheduled_jobs - WHERE enabled = 1 - AND next_run_at <= ? - AND (lease_until IS NULL OR lease_until <= ?) - ORDER BY next_run_at ASC - LIMIT ? - ) + SELECT * FROM scheduled_jobs + WHERE enabled = 1 AND next_run_at <= ? AND (lease_until IS NULL OR lease_until <= ?) - RETURNING * + ORDER BY next_run_at ASC LIMIT ? "#, ) .bind(now) + .bind(now) + .bind(limit.min(256) as i64) + .fetch_all(&mut *tx) + .await?; + let mut claimed = Vec::with_capacity(rows.len()); + for row in rows { + let mut job = row_to_job(&row)?; + let scheduled_for = job.next_run_at; + let (next_run_at, enabled) = match &job.schedule { + Schedule::At { .. } => (job.next_run_at, false), + Schedule::Every { .. } | Schedule::Cron { .. } => ( + next_run_for_schedule(&job.schedule, now).ok_or_else(|| { + StorageError::Serialization(format!( + "cannot compute next run for scheduled job {}", + job.id + )) + })?, + true, + ), + }; + let updated = sqlx::query( + r#" + UPDATE scheduled_jobs + SET locked_at = ?, lock_owner = ?, lease_until = ?, next_run_at = ?, + enabled = ?, last_run_at = ?, updated_at = ? + WHERE id = ? AND enabled = 1 AND next_run_at = ? + AND (lease_until IS NULL OR lease_until <= ?) + "#, + ) + .bind(now) + .bind(owner) + .bind(lease_until) + .bind(next_run_at) + .bind(enabled as i32) + .bind(now) + .bind(now) + .bind(&job.id) + .bind(scheduled_for) + .bind(now) + .execute(&mut *tx) + .await?; + if updated.rows_affected() != 1 { + continue; + } + let run_id: i64 = sqlx::query_scalar( + r#" + INSERT INTO job_runs + (job_id, scheduled_for, agent_id, delivery_policy, target_channel, + target_chat_id, status, delivery_status, created_at, updated_at) + VALUES (?, ?, ?, ?, ?, ?, 'claimed', 'awaiting_result', ?, ?) + RETURNING id + "#, + ) + .bind(&job.id) + .bind(scheduled_for) + .bind(&job.agent_id) + .bind(job.delivery_policy.as_str()) + .bind(&job.channel) + .bind(&job.chat_id) + .bind(now) + .bind(now) + .fetch_one(&mut *tx) + .await?; + job.locked_at = Some(now); + job.lock_owner = Some(owner.to_string()); + job.lease_until = Some(lease_until); + job.next_run_at = next_run_at; + job.enabled = enabled; + job.last_run_at = Some(now); + job.updated_at = now; + claimed.push(ClaimedScheduledRun { + job, + run_id, + scheduled_for, + owner: owner.to_string(), + }); + } + tx.commit().await?; + Ok(claimed) + } + + pub async fn mark_scheduled_run_running( + &self, + run_id: i64, + owner: &str, + agent_run_id: Option<&str>, + started_at: i64, + ) -> Result { + let result = sqlx::query( + r#" + UPDATE job_runs + SET status = 'running', agent_run_id = ?, started_at = ?, updated_at = ? + WHERE id = ? AND status = 'claimed' + AND EXISTS ( + SELECT 1 FROM scheduled_jobs j + WHERE j.id = job_runs.job_id AND j.lock_owner = ? + ) + "#, + ) + .bind(agent_run_id) + .bind(started_at) + .bind(started_at) + .bind(run_id) + .bind(owner) + .execute(self.pool()) + .await?; + Ok(result.rows_affected() == 1) + } + + pub async fn finish_scheduled_run( + &self, + run_id: i64, + owner: &str, + completion: &ScheduledRunCompletion, + finished_at: i64, + ) -> Result { + validate_completion(completion)?; + let mut tx = self.pool().begin().await?; + let row = sqlx::query( + "SELECT job_id, delivery_policy FROM job_runs WHERE id = ? AND status IN ('claimed','running')", + ) + .bind(run_id) + .fetch_optional(&mut *tx) + .await?; + let Some(row) = row else { + return Ok(false); + }; + let job_id: String = row.try_get("job_id")?; + let policy = DeliveryPolicy::parse(&row.try_get::("delivery_policy")?)?; + let delivery = initial_delivery_status(policy, completion.outcome); + let updated = sqlx::query( + r#" + UPDATE job_runs + SET status = ?, outcome = ?, message = ?, diagnostic = ?, duration_ms = ?, + finished_at = ?, delivery_status = ?, delivery_next_attempt_at = ?, updated_at = ? + WHERE id = ? AND status IN ('claimed','running') + AND EXISTS ( + SELECT 1 FROM scheduled_jobs j + WHERE j.id = job_runs.job_id AND j.lock_owner = ? + ) + "#, + ) + .bind(completion.status.as_str()) + .bind(completion.outcome.as_str()) + .bind(truncate_text(&completion.message, 16_384)) + .bind( + completion + .diagnostic + .as_deref() + .map(|value| truncate_text(value, 8_192)), + ) + .bind(completion.duration_ms) + .bind(finished_at) + .bind(delivery.as_str()) + .bind((delivery == ScheduledDeliveryStatus::Pending).then_some(finished_at)) + .bind(finished_at) + .bind(run_id) + .bind(owner) + .execute(&mut *tx) + .await?; + if updated.rows_affected() != 1 { + return Ok(false); + } + let released = sqlx::query( + r#" + UPDATE scheduled_jobs + SET last_outcome = ?, locked_at = NULL, lock_owner = NULL, lease_until = NULL, + updated_at = ? + WHERE id = ? AND lock_owner = ? + "#, + ) + .bind(completion.outcome.as_str()) + .bind(finished_at) + .bind(job_id) + .bind(owner) + .execute(&mut *tx) + .await?; + if released.rows_affected() != 1 { + return Err(StorageError::Conflict(format!( + "scheduled job lease lost before run {run_id} completion" + ))); + } + tx.commit().await?; + Ok(true) + } + + #[allow(clippy::too_many_arguments)] + pub async fn finish_scheduled_run_with_agent( + &self, + run_id: i64, + owner: &str, + completion: &ScheduledRunCompletion, + agent_run_id: &str, + agent_execution_id: &str, + runtime_generation: i64, + agent_outcome: &AgentTerminalOutcome, + finished_at: i64, + ) -> Result { + validate_completion(completion)?; + let mut tx = self.pool().begin().await?; + let row = sqlx::query( + "SELECT job_id, delivery_policy FROM job_runs \ + WHERE id = ? AND agent_run_id = ? AND status IN ('claimed','running')", + ) + .bind(run_id) + .bind(agent_run_id) + .fetch_optional(&mut *tx) + .await?; + let Some(row) = row else { + return Ok(false); + }; + let job_id: String = row.try_get("job_id")?; + let policy = DeliveryPolicy::parse(&row.try_get::("delivery_policy")?)?; + let delivery = initial_delivery_status(policy, completion.outcome); + let run_updated = sqlx::query( + r#" + UPDATE job_runs + SET status = ?, outcome = ?, message = ?, diagnostic = ?, duration_ms = ?, + finished_at = ?, delivery_status = ?, delivery_next_attempt_at = ?, updated_at = ? + WHERE id = ? AND agent_run_id = ? AND status IN ('claimed','running') + AND EXISTS ( + SELECT 1 FROM scheduled_jobs j + WHERE j.id = job_runs.job_id AND j.lock_owner = ? + ) + "#, + ) + .bind(completion.status.as_str()) + .bind(completion.outcome.as_str()) + .bind(truncate_text(&completion.message, 16_384)) + .bind( + completion + .diagnostic + .as_deref() + .map(|value| truncate_text(value, 8_192)), + ) + .bind(completion.duration_ms) + .bind(finished_at) + .bind(delivery.as_str()) + .bind((delivery == ScheduledDeliveryStatus::Pending).then_some(finished_at)) + .bind(finished_at) + .bind(run_id) + .bind(agent_run_id) + .bind(owner) + .execute(&mut *tx) + .await?; + if run_updated.rows_affected() != 1 { + return Ok(false); + } + + let (result, error, prompt_tokens, completion_tokens, cost, tool_calls, iterations) = + match agent_outcome { + AgentTerminalOutcome::Completed { + result, + prompt_tokens, + completion_tokens, + cost, + tool_calls, + iterations, + .. + } => ( + Some(result.as_str()), + None, + *prompt_tokens, + *completion_tokens, + *cost, + *tool_calls, + *iterations, + ), + AgentTerminalOutcome::Failed { + error, + prompt_tokens, + completion_tokens, + cost, + .. + } => ( + None, + Some(error.as_str()), + *prompt_tokens, + *completion_tokens, + *cost, + 0, + 0, + ), + AgentTerminalOutcome::TimedOut { .. } => { + (None, Some("deadline exceeded"), None, None, None, 0, 0) + } + AgentTerminalOutcome::Cancelled { reason, .. } + | AgentTerminalOutcome::Interrupted { reason, .. } => { + (None, Some(reason.as_str()), None, None, None, 0, 0) + } + }; + let agent_updated = sqlx::query( + r#" + UPDATE agent_runs + SET status = ?, result = ?, error = ?, prompt_tokens = ?, completion_tokens = ?, + cost = ?, tool_calls_count = ?, iterations = ?, finished_at = ?, updated_at = ? + WHERE id = ? AND execution_id = ? AND runtime_generation = ? + AND completion_slot_reserved = 0 AND plan_item_id IS NULL + AND status IN ('queued','running','waiting_children') + "#, + ) + .bind(agent_outcome.status().as_str()) + .bind(result) + .bind(error) + .bind(prompt_tokens) + .bind(completion_tokens) + .bind(cost) + .bind(tool_calls) + .bind(iterations) + .bind(finished_at) + .bind(finished_at) + .bind(agent_run_id) + .bind(agent_execution_id) + .bind(runtime_generation) + .execute(&mut *tx) + .await?; + if agent_updated.rows_affected() != 1 { + return Err(StorageError::Conflict(format!( + "scheduled Agent run {agent_run_id} rejected its terminal commit" + ))); + } + let released = sqlx::query( + r#" + UPDATE scheduled_jobs + SET last_outcome = ?, locked_at = NULL, lock_owner = NULL, lease_until = NULL, + updated_at = ? + WHERE id = ? AND lock_owner = ? + "#, + ) + .bind(completion.outcome.as_str()) + .bind(finished_at) + .bind(job_id) + .bind(owner) + .execute(&mut *tx) + .await?; + if released.rows_affected() != 1 { + return Err(StorageError::Conflict(format!( + "scheduled job lease lost before run {run_id} completion" + ))); + } + tx.commit().await?; + Ok(true) + } + + pub async fn recover_scheduled_runs(&self, now: i64) -> Result { + let mut tx = self.pool().begin().await?; + let rows = sqlx::query( + "SELECT id, job_id, agent_run_id, delivery_policy FROM job_runs WHERE status IN ('claimed','running')", + ) + .fetch_all(&mut *tx) + .await?; + for row in &rows { + let run_id: i64 = row.try_get("id")?; + let job_id: String = row.try_get("job_id")?; + let agent_run_id: Option = row.try_get("agent_run_id")?; + let policy = DeliveryPolicy::parse(&row.try_get::("delivery_policy")?)?; + let delivery = if policy == DeliveryPolicy::Never { + ScheduledDeliveryStatus::NotRequested + } else { + ScheduledDeliveryStatus::Pending + }; + sqlx::query( + r#" + UPDATE job_runs + SET status = 'unknown', outcome = 'unknown', + message = 'PicoBot 重启时发现该定时任务尚未提交终态，无法确认外部副作用是否发生。', + diagnostic = 'recovered unfinished scheduled run', finished_at = ?, + delivery_status = ?, delivery_next_attempt_at = ?, updated_at = ? + WHERE id = ? AND status IN ('claimed','running') + "#, + ) + .bind(now) + .bind(delivery.as_str()) + .bind((delivery == ScheduledDeliveryStatus::Pending).then_some(now)) + .bind(now) + .bind(run_id) + .execute(&mut *tx) + .await?; + if let Some(agent_run_id) = agent_run_id { + sqlx::query( + r#" + UPDATE agent_runs + SET status = 'interrupted', error = 'scheduled run interrupted by process restart', + finished_at = ?, updated_at = ?, revision = revision + 1 + WHERE id = ? AND status IN ('queued','running','waiting_children') + "#, + ) + .bind(now) + .bind(now) + .bind(agent_run_id) + .execute(&mut *tx) + .await?; + } + sqlx::query( + r#" + UPDATE scheduled_jobs + SET last_outcome = 'unknown', locked_at = NULL, lock_owner = NULL, + lease_until = NULL, updated_at = ? + WHERE id = ? + "#, + ) + .bind(now) + .bind(job_id) + .execute(&mut *tx) + .await?; + } + tx.commit().await?; + Ok(rows.len() as u64) + } + + pub async fn claim_scheduled_deliveries( + &self, + now: i64, + lease_until: i64, + owner: &str, + limit: usize, + ) -> Result, StorageError> { + if limit == 0 { + return Ok(Vec::new()); + } + let mut tx = self.pool().begin().await?; + sqlx::query( + r#" + UPDATE job_runs + SET delivery_status = 'failed', + delivery_error = COALESCE(delivery_error, 'delivery attempt limit exhausted'), + delivery_next_attempt_at = NULL, delivery_lease_owner = NULL, + delivery_lease_until = NULL, updated_at = ? + WHERE delivery_status = 'delivering' AND delivery_attempts >= 3 + AND delivery_lease_until IS NOT NULL AND delivery_lease_until <= ? + "#, + ) + .bind(now) + .bind(now) + .execute(&mut *tx) + .await?; + let rows = sqlx::query( + r#" + UPDATE job_runs + SET delivery_status = 'delivering', delivery_lease_owner = ?, + delivery_lease_until = ?, delivery_attempts = delivery_attempts + 1, + updated_at = ? + WHERE id IN ( + SELECT id FROM job_runs + WHERE delivery_status IN ('pending','delivering') + AND (delivery_next_attempt_at IS NULL OR delivery_next_attempt_at <= ?) + AND (delivery_lease_until IS NULL OR delivery_lease_until <= ?) + AND delivery_attempts < 3 + ORDER BY COALESCE(delivery_next_attempt_at, created_at), id + LIMIT ? + ) + RETURNING * + "#, + ) .bind(owner) .bind(lease_until) .bind(now) .bind(now) .bind(now) - .bind(now) - .bind(limit as i64) - .bind(now) - .fetch_all(self.pool()) + .bind(limit.min(100) as i64) + .fetch_all(&mut *tx) .await?; - rows.iter().map(row_to_job).collect() - } - - /// Persist the run result, reschedule/disable the job, and release its - /// lease in one transaction. The owner check prevents a stale worker from - /// completing a claim that has already been recovered elsewhere. - pub async fn complete_scheduled_job( - &self, - run: &JobRun, - owner: &str, - next_run_at: Option, - disable: bool, - delete: bool, - ) -> Result<(), StorageError> { - let mut tx = self.pool().begin().await?; - - if delete { - let result = sqlx::query("DELETE FROM scheduled_jobs WHERE id = ? AND lock_owner = ?") - .bind(&run.job_id) - .bind(owner) - .execute(&mut *tx) - .await?; - if result.rows_affected() != 1 { - return Err(StorageError::Conflict(format!( - "scheduled job lease lost before delete: {}", - run.job_id - ))); - } - } else { - sqlx::query( - r#" - INSERT INTO job_runs (job_id, started_at, finished_at, status, output, error, duration_ms, - result_kind, delivery_status, delivery_error) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - "#, - ) - .bind(&run.job_id) - .bind(run.started_at) - .bind(run.finished_at) - .bind(&run.status) - .bind(&run.output) - .bind(&run.error) - .bind(run.duration_ms) - .bind(&run.result_kind) - .bind(&run.delivery_status) - .bind(&run.delivery_error) - .execute(&mut *tx) - .await?; - - let result = sqlx::query( - r#" - UPDATE scheduled_jobs - SET next_run_at = COALESCE(?, next_run_at), - enabled = CASE WHEN ? THEN 0 ELSE enabled END, - last_status = ?, last_error = ?, - locked_at = NULL, lock_owner = NULL, lease_until = NULL, - updated_at = ? - WHERE id = ? AND lock_owner = ? - "#, - ) - .bind(next_run_at) - .bind(disable) - .bind(&run.status) - .bind(&run.error) - .bind(run.finished_at) - .bind(&run.job_id) - .bind(owner) - .execute(&mut *tx) - .await?; - if result.rows_affected() != 1 { - return Err(StorageError::Conflict(format!( - "scheduled job lease lost before completion: {}", - run.job_id - ))); - } - } - tx.commit().await?; - Ok(()) + rows.iter().map(row_to_run).collect() } - pub async fn release_scheduled_job_lease( + pub async fn set_scheduled_delivery_target_session( &self, - job_id: &str, + run_id: i64, owner: &str, - ) -> Result<(), StorageError> { + session_id: &str, + now: i64, + ) -> Result, StorageError> { sqlx::query( - "UPDATE scheduled_jobs SET locked_at = NULL, lock_owner = NULL, lease_until = NULL WHERE id = ? AND lock_owner = ?", + r#" + UPDATE job_runs SET target_session_id = ?, updated_at = ? + WHERE id = ? AND delivery_status = 'delivering' AND delivery_lease_owner = ? + AND target_session_id IS NULL + "#, ) - .bind(job_id) + .bind(session_id) + .bind(now) + .bind(run_id) .bind(owner) .execute(self.pool()) .await?; - Ok(()) + let value: Option = sqlx::query_scalar( + "SELECT target_session_id FROM job_runs \ + WHERE id = ? AND delivery_status = 'delivering' AND delivery_lease_owner = ?", + ) + .bind(run_id) + .bind(owner) + .fetch_optional(self.pool()) + .await? + .flatten(); + Ok(value) + } + + pub async fn complete_scheduled_delivery( + &self, + run_id: i64, + owner: &str, + delivered: bool, + permanent: bool, + error: Option<&str>, + now: i64, + ) -> Result { + let row = sqlx::query("SELECT delivery_attempts FROM job_runs WHERE id = ?") + .bind(run_id) + .fetch_optional(self.pool()) + .await?; + let attempts = row + .as_ref() + .map(|row| row.get::("delivery_attempts")) + .unwrap_or(3); + let terminal_failure = permanent || attempts >= 3; + let status = if delivered { + ScheduledDeliveryStatus::Delivered + } else if terminal_failure { + ScheduledDeliveryStatus::Failed + } else { + ScheduledDeliveryStatus::Pending + }; + let backoff_ms = 30_000_i64.saturating_mul(1_i64 << attempts.saturating_sub(1).min(4)); + let next_attempt = + (status == ScheduledDeliveryStatus::Pending).then_some(now.saturating_add(backoff_ms)); + let result = sqlx::query( + r#" + UPDATE job_runs + SET delivery_status = ?, delivery_next_attempt_at = ?, delivery_error = ?, + delivery_lease_owner = NULL, delivery_lease_until = NULL, updated_at = ? + WHERE id = ? AND delivery_status = 'delivering' AND delivery_lease_owner = ? + "#, + ) + .bind(status.as_str()) + .bind(next_attempt) + .bind(error.map(|value| truncate_text(value, 1_024))) + .bind(now) + .bind(run_id) + .bind(owner) + .execute(self.pool()) + .await?; + Ok(result.rows_affected() == 1) + } + + pub async fn get_scheduled_job_run(&self, id: i64) -> Result { + let row = sqlx::query("SELECT * FROM job_runs WHERE id = ?") + .bind(id) + .fetch_optional(self.pool()) + .await? + .ok_or_else(|| StorageError::NotFound(format!("scheduled run {id}")))?; + row_to_run(&row) } - /// List recent runs for a job, newest first. pub async fn list_scheduled_job_runs( &self, job_id: &str, limit: usize, ) -> Result, StorageError> { - let rows = sqlx::query( - "SELECT * FROM job_runs WHERE job_id = ? ORDER BY finished_at DESC LIMIT ?", - ) - .bind(job_id) - .bind(limit as i64) - .fetch_all(self.pool()) - .await?; - rows.iter() - .map(|r| { - Ok(JobRun { - id: r.try_get("id")?, - job_id: r.try_get("job_id")?, - started_at: r.try_get("started_at")?, - finished_at: r.try_get("finished_at")?, - status: r.try_get("status")?, - output: r.try_get("output")?, - error: r.try_get("error")?, - duration_ms: r.try_get("duration_ms")?, - result_kind: r.try_get("result_kind")?, - delivery_status: r.try_get("delivery_status")?, - delivery_error: r.try_get("delivery_error")?, - }) - }) - .collect() + let rows = sqlx::query("SELECT * FROM job_runs WHERE job_id = ? ORDER BY id DESC LIMIT ?") + .bind(job_id) + .bind(limit.clamp(1, 500) as i64) + .fetch_all(self.pool()) + .await?; + rows.iter().map(row_to_run).collect() + } + + pub async fn list_stale_scheduled_deliveries( + &self, + updated_before: i64, + limit: usize, + ) -> Result, StorageError> { + let rows = sqlx::query( + r#" + SELECT * FROM job_runs + WHERE delivery_status IN ('pending','delivering') AND updated_at < ? + ORDER BY updated_at ASC, id ASC LIMIT ? + "#, + ) + .bind(updated_before) + .bind(limit.clamp(1, 500) as i64) + .fetch_all(self.pool()) + .await?; + rows.iter().map(row_to_run).collect() } - /// Delete disabled jobs whose updated_at is before `before`. pub async fn cleanup_disabled_scheduled_jobs(&self, before: i64) -> Result<(), StorageError> { - sqlx::query("DELETE FROM scheduled_jobs WHERE enabled = 0 AND updated_at < ?") - .bind(before) - .execute(self.pool()) - .await?; + sqlx::query( + r#" + DELETE FROM scheduled_jobs + WHERE enabled = 0 AND updated_at < ? AND lock_owner IS NULL + AND NOT EXISTS ( + SELECT 1 FROM job_runs + WHERE job_id = scheduled_jobs.id + AND (status IN ('claimed','running') + OR delivery_status IN ('pending','delivering')) + ) + "#, + ) + .bind(before) + .execute(self.pool()) + .await?; Ok(()) } } +pub(crate) async fn create_scheduler_v11( + tx: &mut Transaction<'_, Sqlite>, +) -> Result<(), StorageError> { + for statement in SCHEDULER_V11_SCHEMA_STATEMENTS { + sqlx::query(*statement).execute(&mut **tx).await?; + } + Ok(()) +} + +#[derive(Debug)] +struct LegacyJob { + id: String, + name: String, + schedule: Schedule, + prompt: String, + channel: String, + chat_id: String, + policy: DeliveryPolicy, + was_direct: bool, + enabled: bool, + next_run_at: i64, + last_run_at: Option, + last_status: Option, + last_error: Option, + locked_at: Option, + lock_owner: Option, + created_at: i64, + updated_at: i64, +} + +#[derive(Debug)] +struct MigratedRun { + id: i64, + job_id: String, + scheduled_for: i64, + started_at: i64, + finished_at: i64, + status: ScheduledRunStatus, + outcome: ScheduledOutcomeKind, + message: Option, + diagnostic: Option, + duration_ms: i64, + delivery_status: ScheduledDeliveryStatus, + delivery_error: Option, +} + +pub(crate) async fn migrate_scheduler_v11( + tx: &mut Transaction<'_, Sqlite>, + legacy_exists: bool, +) -> Result<(), StorageError> { + if !legacy_exists { + return create_scheduler_v11(tx).await; + } + + let now = now_ms(); + let dropped_model_count: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM scheduled_jobs WHERE model IS NOT NULL AND length(trim(model)) > 0", + ) + .fetch_one(&mut **tx) + .await?; + let job_rows = sqlx::query("SELECT * FROM scheduled_jobs ORDER BY id") + .fetch_all(&mut **tx) + .await?; + let mut jobs = Vec::with_capacity(job_rows.len()); + for row in job_rows { + let id: String = row.try_get("id")?; + let name: String = row.try_get("name")?; + let prompt: String = row.try_get("prompt")?; + let channel: String = row.try_get("channel")?; + let chat_id: String = row.try_get("chat_id")?; + for (field, value) in [ + ("id", id.as_str()), + ("name", name.as_str()), + ("prompt", prompt.as_str()), + ("channel", channel.as_str()), + ("chat_id", chat_id.as_str()), + ] { + if value.trim().is_empty() { + return Err(StorageError::Migration(format!( + "scheduled job {id} has empty {field}" + ))); + } + } + let schedule_json: String = row.try_get("schedule")?; + let schedule: Schedule = serde_json::from_str(&schedule_json).map_err(|error| { + StorageError::Migration(format!("scheduled job {id} has invalid schedule: {error}")) + })?; + let raw_policy: String = row.try_get("delivery_policy")?; + let (policy, was_direct) = match raw_policy.as_str() { + "direct" => (DeliveryPolicy::Always, true), + "always" => (DeliveryPolicy::Always, false), + "on_alert" => (DeliveryPolicy::OnAlert, false), + "never" => (DeliveryPolicy::Never, false), + other => { + return Err(StorageError::Migration(format!( + "scheduled job {id} has invalid delivery policy {other}" + ))); + } + }; + let locked_at: Option = row.try_get("locked_at")?; + let lock_owner: Option = row.try_get("lock_owner")?; + let lease_until: Option = row.try_get("lease_until")?; + if !matches!( + ( + locked_at.is_some(), + lock_owner.is_some(), + lease_until.is_some() + ), + (false, false, false) | (true, true, true) + ) { + return Err(StorageError::Migration(format!( + "scheduled job {id} has a partial lease" + ))); + } + let prompt = if id == "picobot-routine-maintenance" { + "调用 routine_maintenance 工具恰好一次。检查工具结果并通过 complete_scheduled_run 提交：成功且无需关注时 outcome=ok；工具失败、结果不完整或需要人工处理时 outcome=failed 或 alert。不要删除 knowledge 类型的长期记忆。".to_string() + } else { + prompt + }; + jobs.push(LegacyJob { + id, + name, + schedule, + prompt, + channel, + chat_id, + policy, + was_direct, + enabled: row.try_get::("enabled")? != 0, + next_run_at: row.try_get("next_run_at")?, + last_run_at: row.try_get("last_run_at")?, + last_status: row.try_get("last_status")?, + last_error: row.try_get("last_error")?, + locked_at, + lock_owner, + created_at: row.try_get("created_at")?, + updated_at: row.try_get("updated_at")?, + }); + } + + let run_rows = sqlx::query("SELECT * FROM job_runs ORDER BY id") + .fetch_all(&mut **tx) + .await?; + let mut runs = Vec::with_capacity(run_rows.len()); + for row in run_rows { + let id: i64 = row.try_get("id")?; + let job_id: String = row.try_get("job_id")?; + let Some(job) = jobs.iter().find(|job| job.id == job_id) else { + return Err(StorageError::Migration(format!( + "scheduled run {id} references missing job {job_id}" + ))); + }; + let raw_status: String = row.try_get("status")?; + let output: Option = row.try_get("output")?; + let error: Option = row.try_get("error")?; + let result_kind: Option = row.try_get("result_kind")?; + let has_result = output.is_some() || result_kind.is_some(); + let (status, outcome) = match raw_status.as_str() { + "ok" => ( + ScheduledRunStatus::Completed, + migrate_result_kind(id, result_kind.as_deref(), job)?, + ), + "delivery_error" if has_result => ( + ScheduledRunStatus::Completed, + migrate_result_kind(id, result_kind.as_deref(), job)?, + ), + "delivery_error" | "error" => { + (ScheduledRunStatus::Failed, ScheduledOutcomeKind::Failed) + } + "timeout" => (ScheduledRunStatus::TimedOut, ScheduledOutcomeKind::Failed), + _ => (ScheduledRunStatus::Unknown, ScheduledOutcomeKind::Unknown), + }; + let raw_delivery: Option = row.try_get("delivery_status")?; + let delivery_error: Option = row.try_get("delivery_error")?; + let delivery_status = + if delivery_error.is_some() || raw_delivery.as_deref() == Some("failed") { + ScheduledDeliveryStatus::Failed + } else { + match raw_delivery.as_deref() { + Some("direct" | "delivered") => ScheduledDeliveryStatus::Delivered, + Some("suppressed" | "skipped") => ScheduledDeliveryStatus::Suppressed, + _ => ScheduledDeliveryStatus::NotRequested, + } + }; + let started_at: i64 = row.try_get("started_at")?; + let finished_at: i64 = row.try_get("finished_at")?; + runs.push(MigratedRun { + id, + job_id, + scheduled_for: started_at, + started_at, + finished_at, + status, + outcome, + message: output.map(|value| truncate_text(&value, 16_384)), + diagnostic: error.map(|value| truncate_text(&value, 8_192)), + duration_ms: row.try_get("duration_ms")?, + delivery_status, + delivery_error: delivery_error.map(|value| truncate_text(&value, 1_024)), + }); + } + + let mut orphan_last_error_count = 0_usize; + for job in &jobs { + if let Some(last_error) = job.last_error.as_ref() + && let Some(latest) = runs + .iter_mut() + .filter(|run| run.job_id == job.id) + .max_by_key(|run| run.id) + && latest.diagnostic.is_none() + { + latest.diagnostic = Some(truncate_text(last_error, 8_192)); + } else if job.last_error.is_some() && !runs.iter().any(|run| run.job_id == job.id) { + orphan_last_error_count += 1; + } + } + if dropped_model_count > 0 || orphan_last_error_count > 0 { + tracing::warn!( + dropped_model_count, + orphan_last_error_count, + "v11 scheduler migration removed obsolete per-job fields" + ); + } + + sqlx::query("DROP TABLE IF EXISTS job_runs_v10_legacy") + .execute(&mut **tx) + .await?; + sqlx::query("DROP TABLE IF EXISTS scheduled_jobs_v10_legacy") + .execute(&mut **tx) + .await?; + sqlx::query("ALTER TABLE job_runs RENAME TO job_runs_v10_legacy") + .execute(&mut **tx) + .await?; + sqlx::query("ALTER TABLE scheduled_jobs RENAME TO scheduled_jobs_v10_legacy") + .execute(&mut **tx) + .await?; + // Reuse the exact fresh-v11 DDL so migrated and newly-created databases + // have one canonical schema rather than two near-equivalent definitions. + sqlx::query(SCHEDULER_V11_SCHEMA_STATEMENTS[0]) + .execute(&mut **tx) + .await?; + sqlx::query(SCHEDULER_V11_SCHEMA_STATEMENTS[2]) + .execute(&mut **tx) + .await?; + + for job in &jobs { + let latest_outcome = runs + .iter() + .filter(|run| run.job_id == job.id) + .max_by_key(|run| run.id) + .map(|run| run.outcome) + .or_else(|| { + job.last_status.as_deref().map(|status| { + if status == "ok" { + ScheduledOutcomeKind::Ok + } else { + ScheduledOutcomeKind::Failed + } + }) + }); + let had_active_lease = job.lock_owner.is_some(); + let (enabled, next_run_at, last_outcome) = if had_active_lease { + let next_run_at = match &job.schedule { + Schedule::At { .. } => job.next_run_at, + Schedule::Every { .. } | Schedule::Cron { .. } => { + next_run_for_schedule(&job.schedule, now).ok_or_else(|| { + StorageError::Migration(format!( + "cannot advance locked scheduled job {}", + job.id + )) + })? + } + }; + ( + !matches!(job.schedule, Schedule::At { .. }) && job.enabled, + next_run_at, + Some(ScheduledOutcomeKind::Unknown), + ) + } else { + (job.enabled, job.next_run_at, latest_outcome) + }; + sqlx::query( + r#" + INSERT INTO scheduled_jobs + (id, name, schedule, prompt, agent_id, channel, chat_id, delivery_policy, + enabled, next_run_at, last_run_at, last_outcome, locked_at, lock_owner, + lease_until, created_at, updated_at) + VALUES (?, ?, ?, ?, NULL, ?, ?, ?, ?, ?, ?, ?, NULL, NULL, NULL, ?, ?) + "#, + ) + .bind(&job.id) + .bind(&job.name) + .bind(serde_json::to_string(&job.schedule).map_err(|error| { + StorageError::Migration(format!("cannot serialize job {}: {error}", job.id)) + })?) + .bind(&job.prompt) + .bind(&job.channel) + .bind(&job.chat_id) + .bind(job.policy.as_str()) + .bind(enabled as i32) + .bind(next_run_at) + .bind(job.last_run_at) + .bind(last_outcome.map(ScheduledOutcomeKind::as_str)) + .bind(job.created_at) + .bind(job.updated_at.max(now)) + .execute(&mut **tx) + .await?; + } + + for run in &runs { + let job = jobs.iter().find(|job| job.id == run.job_id).unwrap(); + sqlx::query( + r#" + INSERT INTO job_runs + (id, job_id, scheduled_for, agent_run_id, agent_id, delivery_policy, + target_channel, target_chat_id, target_session_id, started_at, finished_at, + status, outcome, message, diagnostic, duration_ms, delivery_status, + delivery_attempts, delivery_next_attempt_at, delivery_lease_owner, + delivery_lease_until, delivery_error, created_at, updated_at) + VALUES (?, ?, ?, NULL, NULL, ?, ?, ?, NULL, ?, ?, ?, ?, ?, ?, ?, ?, 0, + NULL, NULL, NULL, ?, ?, ?) + "#, + ) + .bind(run.id) + .bind(&run.job_id) + .bind(run.scheduled_for) + .bind(job.policy.as_str()) + .bind(&job.channel) + .bind(&job.chat_id) + .bind(run.started_at) + .bind(run.finished_at) + .bind(run.status.as_str()) + .bind(run.outcome.as_str()) + .bind(&run.message) + .bind(&run.diagnostic) + .bind(run.duration_ms.max(0)) + .bind(run.delivery_status.as_str()) + .bind(&run.delivery_error) + .bind(run.started_at) + .bind(run.finished_at) + .execute(&mut **tx) + .await?; + } + + for job in jobs.iter().filter(|job| job.lock_owner.is_some()) { + let delivery = if job.policy == DeliveryPolicy::Never { + ScheduledDeliveryStatus::NotRequested + } else { + ScheduledDeliveryStatus::Pending + }; + sqlx::query( + r#" + INSERT INTO job_runs + (job_id, scheduled_for, agent_id, delivery_policy, target_channel, + target_chat_id, started_at, finished_at, status, outcome, message, + diagnostic, duration_ms, delivery_status, delivery_attempts, + delivery_next_attempt_at, created_at, updated_at) + VALUES (?, ?, NULL, ?, ?, ?, ?, ?, 'unknown', 'unknown', ?, ?, NULL, ?, 0, ?, ?, ?) + "#, + ) + .bind(&job.id) + .bind(job.last_run_at.unwrap_or(job.next_run_at)) + .bind(job.policy.as_str()) + .bind(&job.channel) + .bind(&job.chat_id) + .bind(job.locked_at) + .bind(now) + .bind("升级数据库时发现尚未提交终态的定时任务，无法确认外部副作用是否发生。") + .bind("v10 lease recovered during v11 migration") + .bind(delivery.as_str()) + .bind((delivery == ScheduledDeliveryStatus::Pending).then_some(now)) + .bind(job.locked_at.unwrap_or(now)) + .bind(now) + .execute(&mut **tx) + .await?; + } + + sqlx::query("DROP TABLE job_runs_v10_legacy") + .execute(&mut **tx) + .await?; + sqlx::query("DROP TABLE scheduled_jobs_v10_legacy") + .execute(&mut **tx) + .await?; + for statement in &SCHEDULER_V11_SCHEMA_STATEMENTS[1..] { + if statement.trim_start().starts_with("CREATE INDEX") { + sqlx::query(*statement).execute(&mut **tx).await?; + } + } + let foreign_key_errors = sqlx::query("PRAGMA foreign_key_check") + .fetch_all(&mut **tx) + .await?; + if !foreign_key_errors.is_empty() { + return Err(StorageError::Migration(format!( + "v11 scheduler migration produced {} foreign-key violations", + foreign_key_errors.len() + ))); + } + Ok(()) +} + +fn migrate_result_kind( + run_id: i64, + result_kind: Option<&str>, + job: &LegacyJob, +) -> Result { + match result_kind { + Some("quiet") => Ok(ScheduledOutcomeKind::Ok), + Some("content") if job.policy == DeliveryPolicy::OnAlert => Ok(ScheduledOutcomeKind::Alert), + Some("content") => Ok(ScheduledOutcomeKind::Ok), + Some("reported_failure") => Ok(ScheduledOutcomeKind::Failed), + Some("refused") => Ok(ScheduledOutcomeKind::Refused), + None if job.was_direct => Ok(ScheduledOutcomeKind::Ok), + other => Err(StorageError::Migration(format!( + "scheduled run {run_id} has unmappable result kind {other:?}" + ))), + } +} + +fn validate_job(job: &ScheduledJob) -> Result<(), StorageError> { + for (field, value) in [ + ("id", job.id.as_str()), + ("name", job.name.as_str()), + ("prompt", job.prompt.as_str()), + ("channel", job.channel.as_str()), + ("chat_id", job.chat_id.as_str()), + ] { + if value.trim().is_empty() { + return Err(StorageError::Serialization(format!( + "scheduled job {field} must not be empty" + ))); + } + } + if job + .agent_id + .as_deref() + .is_some_and(|value| value.trim().is_empty()) + { + return Err(StorageError::Serialization( + "scheduled job agent_id must not be empty".to_string(), + )); + } + Ok(()) +} + +fn validate_completion(completion: &ScheduledRunCompletion) -> Result<(), StorageError> { + let valid = match completion.status { + ScheduledRunStatus::Completed => matches!( + completion.outcome, + ScheduledOutcomeKind::Ok + | ScheduledOutcomeKind::Alert + | ScheduledOutcomeKind::Failed + | ScheduledOutcomeKind::Refused + ), + ScheduledRunStatus::Failed + | ScheduledRunStatus::TimedOut + | ScheduledRunStatus::Cancelled + | ScheduledRunStatus::Interrupted => completion.outcome == ScheduledOutcomeKind::Failed, + ScheduledRunStatus::Unknown => completion.outcome == ScheduledOutcomeKind::Unknown, + ScheduledRunStatus::Claimed | ScheduledRunStatus::Running => false, + }; + if !valid { + return Err(StorageError::Serialization(format!( + "invalid scheduled completion pair: {}/{}", + completion.status.as_str(), + completion.outcome.as_str() + ))); + } + if completion.message.trim().is_empty() { + return Err(StorageError::Serialization( + "scheduled completion message must not be empty".to_string(), + )); + } + Ok(()) +} + +fn initial_delivery_status( + policy: DeliveryPolicy, + outcome: ScheduledOutcomeKind, +) -> ScheduledDeliveryStatus { + if policy == DeliveryPolicy::Never { + ScheduledDeliveryStatus::NotRequested + } else if policy == DeliveryPolicy::OnAlert && outcome == ScheduledOutcomeKind::Ok { + ScheduledDeliveryStatus::Suppressed + } else { + ScheduledDeliveryStatus::Pending + } +} + +fn serialize_schedule(schedule: &Schedule) -> Result { + serde_json::to_string(schedule).map_err(|error| StorageError::Serialization(error.to_string())) +} + +fn row_to_job(row: &sqlx::sqlite::SqliteRow) -> Result { + let schedule_json: String = row.try_get("schedule")?; + let schedule = serde_json::from_str(&schedule_json) + .map_err(|error| StorageError::Serialization(error.to_string()))?; + let last_outcome: Option = row.try_get("last_outcome")?; + Ok(ScheduledJob { + id: row.try_get("id")?, + name: row.try_get("name")?, + schedule, + prompt: row.try_get("prompt")?, + agent_id: row.try_get("agent_id")?, + channel: row.try_get("channel")?, + chat_id: row.try_get("chat_id")?, + delivery_policy: DeliveryPolicy::parse(&row.try_get::("delivery_policy")?)?, + enabled: row.try_get::("enabled")? != 0, + next_run_at: row.try_get("next_run_at")?, + last_run_at: row.try_get("last_run_at")?, + last_outcome: last_outcome + .as_deref() + .map(ScheduledOutcomeKind::parse) + .transpose()?, + created_at: row.try_get("created_at")?, + updated_at: row.try_get("updated_at")?, + locked_at: row.try_get("locked_at")?, + lock_owner: row.try_get("lock_owner")?, + lease_until: row.try_get("lease_until")?, + }) +} + +fn row_to_run(row: &sqlx::sqlite::SqliteRow) -> Result { + let outcome: Option = row.try_get("outcome")?; + Ok(JobRun { + id: row.try_get("id")?, + job_id: row.try_get("job_id")?, + scheduled_for: row.try_get("scheduled_for")?, + agent_run_id: row.try_get("agent_run_id")?, + agent_id: row.try_get("agent_id")?, + delivery_policy: DeliveryPolicy::parse(&row.try_get::("delivery_policy")?)?, + target_channel: row.try_get("target_channel")?, + target_chat_id: row.try_get("target_chat_id")?, + target_session_id: row.try_get("target_session_id")?, + started_at: row.try_get("started_at")?, + finished_at: row.try_get("finished_at")?, + status: ScheduledRunStatus::parse(&row.try_get::("status")?)?, + outcome: outcome + .as_deref() + .map(ScheduledOutcomeKind::parse) + .transpose()?, + message: row.try_get("message")?, + diagnostic: row.try_get("diagnostic")?, + duration_ms: row.try_get("duration_ms")?, + delivery_status: ScheduledDeliveryStatus::parse( + &row.try_get::("delivery_status")?, + )?, + delivery_attempts: row.try_get("delivery_attempts")?, + delivery_next_attempt_at: row.try_get("delivery_next_attempt_at")?, + delivery_lease_owner: row.try_get("delivery_lease_owner")?, + delivery_lease_until: row.try_get("delivery_lease_until")?, + delivery_error: row.try_get("delivery_error")?, + created_at: row.try_get("created_at")?, + updated_at: row.try_get("updated_at")?, + }) +} + +fn truncate_text(value: &str, max_chars: usize) -> String { + value.chars().take(max_chars).collect() +} + fn now_ms() -> i64 { std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH) @@ -489,376 +1758,479 @@ fn now_ms() -> i64 { .as_millis() as i64 } -fn serialize_schedule(schedule: &Schedule) -> Result { - serde_json::to_string(schedule).map_err(|error| StorageError::Serialization(error.to_string())) -} - -fn row_to_job(row: &sqlx::sqlite::SqliteRow) -> Result { - let schedule_json: String = row.try_get("schedule")?; - let schedule: Schedule = serde_json::from_str(&schedule_json) - .map_err(|error| StorageError::Serialization(error.to_string()))?; - Ok(ScheduledJob { - id: row.try_get("id")?, - name: row.try_get("name")?, - schedule, - prompt: row.try_get("prompt")?, - channel: row.try_get("channel")?, - chat_id: row.try_get("chat_id")?, - model: row.try_get("model")?, - job_kind: parse_job_kind(&row.try_get::("job_kind")?)?, - delivery_policy: parse_delivery_policy(&row.try_get::("delivery_policy")?)?, - enabled: row.try_get::("enabled")? != 0, - delete_after_run: row.try_get::("delete_after_run")? != 0, - next_run_at: row.try_get("next_run_at")?, - last_run_at: row.try_get("last_run_at")?, - last_status: row.try_get("last_status")?, - last_error: row.try_get("last_error")?, - created_at: row.try_get("created_at")?, - updated_at: row.try_get("updated_at")?, - }) -} - -fn parse_job_kind(value: &str) -> Result { - match value { - "task" => Ok(JobKind::Task), - "monitor" => Ok(JobKind::Monitor), - other => Err(StorageError::Serialization(format!( - "unknown job kind: {other}" - ))), - } -} - -fn parse_delivery_policy(value: &str) -> Result { - match value { - "direct" => Ok(DeliveryPolicy::Direct), - "always" => Ok(DeliveryPolicy::Always), - "on_alert" => Ok(DeliveryPolicy::OnAlert), - "never" => Ok(DeliveryPolicy::Never), - other => Err(StorageError::Serialization(format!( - "unknown delivery policy: {other}" - ))), - } -} - #[cfg(test)] mod tests { - use super::{DeliveryPolicy, JobKind, ScheduledJob}; - use crate::scheduler::Schedule; - use crate::storage::Storage; + use super::*; use sqlx::SqlitePool; - - fn now() -> i64 { - std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap() - .as_millis() as i64 - } + use std::sync::Arc; async fn setup_storage() -> Storage { let pool = SqlitePool::connect("sqlite::memory:").await.unwrap(); - let storage = Storage { pool }; - Storage::init_scheduler_schema(storage.pool()) - .await - .unwrap(); - storage + for statement in crate::storage::agent_run::AGENT_SCHEMA_STATEMENTS { + sqlx::query(*statement).execute(&pool).await.unwrap(); + } + Storage::init_scheduler_schema(&pool).await.unwrap(); + Storage { pool } + } + + fn job(now: i64, schedule: Schedule, policy: DeliveryPolicy) -> ScheduledJob { + ScheduledJob { + id: "job-1".to_string(), + name: "test job".to_string(), + schedule, + prompt: "check status".to_string(), + agent_id: None, + channel: "cli_chat".to_string(), + chat_id: "chat".to_string(), + delivery_policy: policy, + enabled: true, + next_run_at: now, + last_run_at: None, + last_outcome: None, + created_at: now, + updated_at: now, + locked_at: None, + lock_owner: None, + lease_until: None, + } + } + + #[test] + fn delivery_matrix_is_deterministic() { + assert!(DeliveryPolicy::Always.should_deliver(ScheduledOutcomeKind::Ok)); + assert!(!DeliveryPolicy::OnAlert.should_deliver(ScheduledOutcomeKind::Ok)); + assert!(DeliveryPolicy::OnAlert.should_deliver(ScheduledOutcomeKind::Alert)); + assert!(!DeliveryPolicy::Never.should_deliver(ScheduledOutcomeKind::Failed)); } #[tokio::test] - async fn test_init_creates_tables() { + async fn claim_creates_run_and_advances_recurring_job() { let storage = setup_storage().await; - let row: (i64,) = sqlx::query_as("SELECT COUNT(*) FROM scheduled_jobs") - .fetch_one(storage.pool()) - .await - .unwrap(); - assert_eq!(row.0, 0); - } - - #[tokio::test] - async fn default_maintenance_job_is_idempotent_and_preserves_user_state() { - let storage = setup_storage().await; - storage.ensure_default_maintenance_job().await.unwrap(); - let job = storage - .get_scheduled_job("picobot-routine-maintenance") - .await - .unwrap(); - assert_eq!(job.job_kind, JobKind::Monitor); - assert_eq!(job.delivery_policy, DeliveryPolicy::Never); - assert!(job.enabled); - storage - .set_scheduled_job_enabled("picobot-routine-maintenance", false) + .add_scheduled_job(&job( + 1_000, + Schedule::Every { every_ms: 5_000 }, + DeliveryPolicy::Always, + )) .await .unwrap(); - storage.ensure_default_maintenance_job().await.unwrap(); - assert!( - !storage - .get_scheduled_job("picobot-routine-maintenance") + let claimed = storage + .claim_due_scheduled_runs(1_000, 10_000, "owner", 1) + .await + .unwrap(); + assert_eq!(claimed.len(), 1); + assert_eq!(claimed[0].scheduled_for, 1_000); + assert_eq!( + storage + .get_scheduled_job("job-1") .await .unwrap() - .enabled + .next_run_at, + 6_000 + ); + assert_eq!( + storage + .get_scheduled_job_run(claimed[0].run_id) + .await + .unwrap() + .status, + ScheduledRunStatus::Claimed ); } #[tokio::test] - async fn test_add_and_get_job() { + async fn at_claim_disables_job_atomically() { let storage = setup_storage().await; - let t = now(); - let job = ScheduledJob { - id: "job-1".into(), - name: "test job".into(), - schedule: Schedule::Every { every_ms: 3600000 }, - prompt: "say hello".into(), - channel: "cli_chat".into(), - chat_id: "conn-1".into(), - model: None, - job_kind: JobKind::Task, - delivery_policy: DeliveryPolicy::Direct, - enabled: true, - delete_after_run: false, - next_run_at: t + 3600000, - last_run_at: None, - last_status: None, - last_error: None, - created_at: t, - updated_at: t, - }; - storage.add_scheduled_job(&job).await.unwrap(); - let got = storage.get_scheduled_job("job-1").await.unwrap(); - assert_eq!(got.id, "job-1"); - assert_eq!(got.name, "test job"); - assert_eq!(got.prompt, "say hello"); - } - - #[tokio::test] - async fn test_list_jobs() { - let storage = setup_storage().await; - let t = now(); - for i in 0..3 { - let job = ScheduledJob { - id: format!("job-{}", i), - name: format!("job {}", i), - schedule: Schedule::Every { every_ms: 3600000 }, - prompt: "ping".into(), - channel: "cli_chat".into(), - chat_id: "conn-1".into(), - model: None, - job_kind: JobKind::Task, - delivery_policy: DeliveryPolicy::Direct, - enabled: true, - delete_after_run: false, - next_run_at: t + 1000, - last_run_at: None, - last_status: None, - last_error: None, - created_at: t, - updated_at: t, - }; - storage.add_scheduled_job(&job).await.unwrap(); - } - let jobs = storage.list_scheduled_jobs().await.unwrap(); - assert_eq!(jobs.len(), 3); - } - - #[tokio::test] - async fn test_remove_job() { - let storage = setup_storage().await; - let t = now(); - let job = ScheduledJob { - id: "job-rm".into(), - name: "remove me".into(), - schedule: Schedule::Every { every_ms: 1000 }, - prompt: "hi".into(), - channel: "cli_chat".into(), - chat_id: "c".into(), - model: None, - job_kind: JobKind::Task, - delivery_policy: DeliveryPolicy::Direct, - enabled: true, - delete_after_run: false, - next_run_at: t, - last_run_at: None, - last_status: None, - last_error: None, - created_at: t, - updated_at: t, - }; - storage.add_scheduled_job(&job).await.unwrap(); - storage.remove_scheduled_job("job-rm").await.unwrap(); - let result = storage.get_scheduled_job("job-rm").await; - assert!(result.is_err()); - } - - #[tokio::test] - async fn test_set_enabled() { - let storage = setup_storage().await; - let t = now(); - let job = ScheduledJob { - id: "job-toggle".into(), - name: "toggle".into(), - schedule: Schedule::Every { every_ms: 1000 }, - prompt: "hi".into(), - channel: "cli_chat".into(), - chat_id: "c".into(), - model: None, - job_kind: JobKind::Task, - delivery_policy: DeliveryPolicy::Direct, - enabled: true, - delete_after_run: false, - next_run_at: t, - last_run_at: None, - last_status: None, - last_error: None, - created_at: t, - updated_at: t, - }; - storage.add_scheduled_job(&job).await.unwrap(); storage - .set_scheduled_job_enabled("job-toggle", false) + .add_scheduled_job(&job( + 1_000, + Schedule::At { at: 1_000 }, + DeliveryPolicy::Never, + )) .await .unwrap(); - let got = storage.get_scheduled_job("job-toggle").await.unwrap(); - assert!(!got.enabled); + storage + .claim_due_scheduled_runs(1_000, 10_000, "owner", 1) + .await + .unwrap(); + assert!(!storage.get_scheduled_job("job-1").await.unwrap().enabled); } #[tokio::test] - async fn test_update_job() { + async fn completion_applies_alert_policy_and_releases_lease() { let storage = setup_storage().await; - let t = now(); - let job = ScheduledJob { - id: "job-update".into(), - name: "old name".into(), - schedule: Schedule::Every { every_ms: 1000 }, - prompt: "old prompt".into(), - channel: "feishu".into(), - chat_id: "oc_1".into(), - model: None, - job_kind: JobKind::Task, - delivery_policy: DeliveryPolicy::Direct, - enabled: true, - delete_after_run: false, - next_run_at: t, - last_run_at: None, - last_status: None, - last_error: None, - created_at: t, - updated_at: t, - }; - storage.add_scheduled_job(&job).await.unwrap(); storage - .update_scheduled_job( - "job-update", - Some("new prompt".into()), - Some(Schedule::Every { every_ms: 60000 }), - None, - None, - None, + .add_scheduled_job(&job( + 1_000, + Schedule::Every { every_ms: 5_000 }, + DeliveryPolicy::OnAlert, + )) + .await + .unwrap(); + let claimed = storage + .claim_due_scheduled_runs(1_000, 10_000, "owner", 1) + .await + .unwrap() + .remove(0); + assert!( + storage + .mark_scheduled_run_running(claimed.run_id, "owner", None, 1_001) + .await + .unwrap() + ); + assert!( + storage + .finish_scheduled_run( + claimed.run_id, + "owner", + &ScheduledRunCompletion { + status: ScheduledRunStatus::Completed, + outcome: ScheduledOutcomeKind::Ok, + message: "healthy".to_string(), + diagnostic: None, + duration_ms: 10, + }, + 1_011, + ) + .await + .unwrap() + ); + let run = storage.get_scheduled_job_run(claimed.run_id).await.unwrap(); + assert_eq!(run.delivery_status, ScheduledDeliveryStatus::Suppressed); + assert!( + storage + .get_scheduled_job("job-1") + .await + .unwrap() + .lock_owner + .is_none() + ); + } + + #[tokio::test] + async fn remove_rejects_active_run() { + let storage = setup_storage().await; + storage + .add_scheduled_job(&job( + 1_000, + Schedule::At { at: 1_000 }, + DeliveryPolicy::Never, + )) + .await + .unwrap(); + storage + .claim_due_scheduled_runs(1_000, 10_000, "owner", 1) + .await + .unwrap(); + assert!(matches!( + storage.remove_scheduled_job("job-1").await, + Err(StorageError::Conflict(_)) + )); + } + + #[tokio::test] + async fn concurrent_claimers_create_only_one_occurrence() { + let storage = Arc::new(setup_storage().await); + storage + .add_scheduled_job(&job( + 1_000, + Schedule::Every { every_ms: 5_000 }, + DeliveryPolicy::Always, + )) + .await + .unwrap(); + let left = { + let storage = storage.clone(); + async move { + storage + .claim_due_scheduled_runs(1_000, 10_000, "owner-a", 1) + .await + .unwrap() + } + }; + let right = { + let storage = storage.clone(); + async move { + storage + .claim_due_scheduled_runs(1_000, 10_000, "owner-b", 1) + .await + .unwrap() + } + }; + let (left, right) = tokio::join!(left, right); + assert_eq!(left.len() + right.len(), 1); + assert_eq!( + storage + .list_scheduled_job_runs("job-1", 10) + .await + .unwrap() + .len(), + 1 + ); + } + + #[tokio::test] + async fn lease_owner_and_database_constraints_reject_late_or_invalid_completion() { + let storage = setup_storage().await; + storage + .add_scheduled_job(&job( + 1_000, + Schedule::Every { every_ms: 5_000 }, + DeliveryPolicy::Always, + )) + .await + .unwrap(); + let claimed = storage + .claim_due_scheduled_runs(1_000, 10_000, "owner", 1) + .await + .unwrap() + .remove(0); + assert!( + !storage + .finish_scheduled_run( + claimed.run_id, + "stale-owner", + &ScheduledRunCompletion { + status: ScheduledRunStatus::Completed, + outcome: ScheduledOutcomeKind::Ok, + message: "healthy".to_string(), + diagnostic: None, + duration_ms: 1, + }, + 1_001, + ) + .await + .unwrap() + ); + let invalid = sqlx::query( + "UPDATE job_runs SET status = 'completed', outcome = NULL, finished_at = 1001, \ + delivery_status = 'not_requested' WHERE id = ?", + ) + .bind(claimed.run_id) + .execute(storage.pool()) + .await; + assert!(invalid.is_err()); + } + + #[tokio::test] + async fn delivery_retries_are_persistent_and_bounded() { + let storage = setup_storage().await; + storage + .add_scheduled_job(&job( + 1_000, + Schedule::At { at: 1_000 }, + DeliveryPolicy::Always, + )) + .await + .unwrap(); + let claimed = storage + .claim_due_scheduled_runs(1_000, 10_000, "run-owner", 1) + .await + .unwrap() + .remove(0); + storage + .finish_scheduled_run( + claimed.run_id, + "run-owner", + &ScheduledRunCompletion { + status: ScheduledRunStatus::Completed, + outcome: ScheduledOutcomeKind::Alert, + message: "attention".to_string(), + diagnostic: None, + duration_ms: 10, + }, + 1_010, ) .await .unwrap(); - let got = storage.get_scheduled_job("job-update").await.unwrap(); - assert_eq!(got.prompt, "new prompt"); - } - - #[tokio::test] - async fn claim_is_exclusive_until_lease_expires() { - let storage = setup_storage().await; - let t = now(); - let job = ScheduledJob { - id: "leased-job".into(), - name: "leased".into(), - schedule: Schedule::Every { every_ms: 1000 }, - prompt: "run".into(), - channel: "cli_chat".into(), - chat_id: "c".into(), - model: None, - job_kind: JobKind::Task, - delivery_policy: DeliveryPolicy::Direct, - enabled: true, - delete_after_run: false, - next_run_at: t, - last_run_at: None, - last_status: None, - last_error: None, - created_at: t, - updated_at: t, - }; - storage.add_scheduled_job(&job).await.unwrap(); - - let first = storage - .claim_due_scheduled_jobs(t, t + 100, "owner-1", 1) - .await - .unwrap(); - let duplicate = storage - .claim_due_scheduled_jobs(t, t + 100, "owner-2", 1) - .await - .unwrap(); - let recovered = storage - .claim_due_scheduled_jobs(t + 101, t + 201, "owner-2", 1) - .await - .unwrap(); - - assert_eq!(first.len(), 1); - assert!(duplicate.is_empty()); - assert_eq!(recovered.len(), 1); - } - - #[tokio::test] - async fn completion_is_atomic_and_releases_lease() { - let storage = setup_storage().await; - let t = now(); - let job = ScheduledJob { - id: "complete-job".into(), - name: "complete".into(), - schedule: Schedule::Every { every_ms: 1000 }, - prompt: "run".into(), - channel: "cli_chat".into(), - chat_id: "c".into(), - model: None, - job_kind: JobKind::Task, - delivery_policy: DeliveryPolicy::Direct, - enabled: true, - delete_after_run: false, - next_run_at: t, - last_run_at: None, - last_status: None, - last_error: None, - created_at: t, - updated_at: t, - }; - storage.add_scheduled_job(&job).await.unwrap(); - storage - .claim_due_scheduled_jobs(t, t + 1000, "owner", 1) - .await - .unwrap(); - let run = super::JobRun { - id: 0, - job_id: job.id.clone(), - started_at: t, - finished_at: t + 10, - status: "ok".into(), - output: Some("done".into()), - error: None, - duration_ms: 10, - result_kind: Some("content".into()), - delivery_status: Some("direct".into()), - delivery_error: None, - }; - storage - .complete_scheduled_job(&run, "owner", Some(t + 2000), false, false) - .await - .unwrap(); - - let completed = storage.get_scheduled_job(&job.id).await.unwrap(); - let runs = storage.list_scheduled_job_runs(&job.id, 10).await.unwrap(); - let lease: (Option, Option) = - sqlx::query_as("SELECT lock_owner, lease_until FROM scheduled_jobs WHERE id = ?") - .bind(&job.id) - .fetch_one(storage.pool()) + for attempt in 1..=3 { + let now = 100_000 * attempt; + let runs = storage + .claim_scheduled_deliveries(now, now + 1_000, "delivery-owner", 1) .await .unwrap(); - assert_eq!(completed.next_run_at, t + 2000); - assert_eq!(completed.last_status.as_deref(), Some("ok")); - assert_eq!(runs.len(), 1); - assert_eq!(lease, (None, None)); + assert_eq!(runs.len(), 1); + if attempt == 1 { + let fixed = storage + .set_scheduled_delivery_target_session( + claimed.run_id, + "delivery-owner", + "cli_chat:chat:first", + now, + ) + .await + .unwrap(); + assert_eq!(fixed.as_deref(), Some("cli_chat:chat:first")); + let unchanged = storage + .set_scheduled_delivery_target_session( + claimed.run_id, + "delivery-owner", + "cli_chat:chat:second", + now, + ) + .await + .unwrap(); + assert_eq!(unchanged.as_deref(), Some("cli_chat:chat:first")); + assert!(matches!( + storage.remove_scheduled_job("job-1").await, + Err(StorageError::Conflict(_)) + )); + } + storage + .complete_scheduled_delivery( + claimed.run_id, + "delivery-owner", + false, + false, + Some("temporary"), + now, + ) + .await + .unwrap(); + } + let run = storage.get_scheduled_job_run(claimed.run_id).await.unwrap(); + assert_eq!(run.delivery_attempts, 3); + assert_eq!(run.delivery_status, ScheduledDeliveryStatus::Failed); + assert!( + storage + .claim_scheduled_deliveries(1_000_000, 1_001_000, "other", 1) + .await + .unwrap() + .is_empty() + ); + } + + #[tokio::test] + async fn scheduled_and_agent_terminal_states_commit_atomically() { + let storage = setup_storage().await; + storage + .add_scheduled_job(&job( + 1_000, + Schedule::At { at: 1_000 }, + DeliveryPolicy::Never, + )) + .await + .unwrap(); + let claimed = storage + .claim_due_scheduled_runs(1_000, 10_000, "run-owner", 1) + .await + .unwrap() + .remove(0); + let agent_run_id = "scheduled-agent-1"; + storage + .accept_agent_runs(crate::storage::agent_run::AcceptAgentRequest { + runs: vec![crate::storage::agent_run::NewAgentRun { + id: agent_run_id.to_string(), + root_session_id: format!("scheduled-run:{}", claimed.run_id), + root_turn_id: None, + parent_run_id: None, + caller_agent_id: "SCHEDULER".to_string(), + caller_scope_id: "scheduled:job-1".to_string(), + idempotency_key: Some(format!("scheduled:{}", claimed.run_id)), + agent_id: "ROOT".to_string(), + definition_hash: String::new(), + provider_profile: String::new(), + provider_name: "test".to_string(), + model_id: "test".to_string(), + mode: crate::storage::agent_run::AgentRunMode::Foreground, + depth: 1, + plan_item_id: None, + execution_id: agent_run_id.to_string(), + task: "check".to_string(), + context_json: None, + budget_json: "{}".to_string(), + signal_contract_json: None, + signal_delivery: None, + deadline_at: 10_000, + runtime_generation: 1, + completion_slot_reserved: false, + }], + now: 1_000, + }) + .await + .unwrap(); + assert!( + storage + .mark_scheduled_run_running(claimed.run_id, "run-owner", Some(agent_run_id), 1_001,) + .await + .unwrap() + ); + assert!( + storage + .mark_agent_run_running(agent_run_id, agent_run_id, 1_001) + .await + .unwrap() + ); + let completion = ScheduledRunCompletion { + status: ScheduledRunStatus::Completed, + outcome: ScheduledOutcomeKind::Ok, + message: "healthy".to_string(), + diagnostic: None, + duration_ms: 10, + }; + let agent_outcome = AgentTerminalOutcome::Completed { + result: "healthy".to_string(), + prompt_tokens: None, + completion_tokens: None, + cost: None, + tool_calls: 1, + iterations: 1, + signal_ids: Vec::new(), + }; + assert!( + storage + .finish_scheduled_run_with_agent( + claimed.run_id, + "run-owner", + &completion, + agent_run_id, + agent_run_id, + 99, + &agent_outcome, + 1_011, + ) + .await + .is_err() + ); + assert_eq!( + storage + .get_scheduled_job_run(claimed.run_id) + .await + .unwrap() + .status, + ScheduledRunStatus::Running + ); + assert!( + storage + .finish_scheduled_run_with_agent( + claimed.run_id, + "run-owner", + &completion, + agent_run_id, + agent_run_id, + 1, + &agent_outcome, + 1_011, + ) + .await + .unwrap() + ); + assert_eq!( + storage + .get_scheduled_job_run(claimed.run_id) + .await + .unwrap() + .status, + ScheduledRunStatus::Completed + ); + assert_eq!( + storage + .get_agent_run(agent_run_id) + .await + .unwrap() + .unwrap() + .status, + crate::storage::agent_run::AgentRunStatus::Completed + ); } } diff --git a/src/tools/complete_scheduled_run.rs b/src/tools/complete_scheduled_run.rs new file mode 100644 index 0000000..194d7b0 --- /dev/null +++ b/src/tools/complete_scheduled_run.rs @@ -0,0 +1,191 @@ +use async_trait::async_trait; +use serde_json::{Value, json}; + +use crate::storage::ScheduledOutcomeKind; +use crate::tools::{ScheduledOutcome, Tool, ToolExecutionContext, ToolOutput, ToolResult}; + +pub struct CompleteScheduledRunTool; + +impl CompleteScheduledRunTool { + pub fn new() -> Self { + Self + } +} + +impl Default for CompleteScheduledRunTool { + fn default() -> Self { + Self::new() + } +} + +#[async_trait] +impl Tool for CompleteScheduledRunTool { + fn name(&self) -> &str { + "complete_scheduled_run" + } + + fn description(&self) -> &str { + "Submit the single structured final outcome (ok, alert, failed, or refused) of an unattended scheduled run and end the run immediately." + } + + fn parameters_schema(&self) -> Value { + json!({ + "type": "object", + "properties": { + "outcome": { + "type": "string", + "enum": ["ok", "alert", "failed", "refused"], + "description": "ok: completed with nothing needing attention; alert: completed with actionable findings; failed: did not complete reliably; refused: denied for permission or safety reasons" + }, + "message": { + "type": "string", + "minLength": 1, + "maxLength": 16384, + "description": "The user-facing result or notification body" + } + }, + "required": ["outcome", "message"], + "additionalProperties": false + }) + } + + fn runtime_injected(&self) -> bool { + true + } + + fn exclusive(&self) -> bool { + true + } + + async fn execute(&self, args: Value) -> anyhow::Result { + self.execute_with_context(&ToolExecutionContext::default(), args) + .await + .map(|output| output.result) + } + + async fn execute_with_context( + &self, + context: &ToolExecutionContext, + args: Value, + ) -> anyhow::Result { + let Some(sink) = context.scheduled_completion.as_ref() else { + return Ok(failure( + "complete_scheduled_run is only available to the top-level scheduled Agent", + ) + .into()); + }; + if !context.execution_origin.is_scheduled() { + return Ok(failure("scheduled completion context is invalid").into()); + } + let Some(object) = args.as_object() else { + return Ok(failure("arguments must be an object").into()); + }; + if object + .keys() + .any(|key| key != "outcome" && key != "message") + { + return Ok(failure("unknown complete_scheduled_run argument").into()); + } + let kind = match args.get("outcome").and_then(Value::as_str) { + Some("ok") => ScheduledOutcomeKind::Ok, + Some("alert") => ScheduledOutcomeKind::Alert, + Some("failed") => ScheduledOutcomeKind::Failed, + Some("refused") => ScheduledOutcomeKind::Refused, + Some(other) => return Ok(failure(format!("invalid scheduled outcome: {other}")).into()), + None => return Ok(failure("outcome is required").into()), + }; + let message = args + .get("message") + .and_then(Value::as_str) + .unwrap_or_default() + .trim(); + if message.is_empty() { + return Ok(failure("message must not be empty").into()); + } + if message.chars().count() > 16_384 { + return Ok(failure("message exceeds 16384 characters").into()); + } + let message = message.to_string(); + if let Err(error) = sink.submit(ScheduledOutcome { + kind, + message: message.clone(), + }) { + return Ok(failure(error).into()); + } + Ok(ToolResult { + success: true, + output: format!("scheduled run completed with outcome={}", kind.as_str()), + error: None, + } + .into()) + } +} + +fn failure(error: impl Into) -> ToolResult { + ToolResult { + success: false, + output: String::new(), + error: Some(error.into()), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::tools::{ExecutionOrigin, ScheduledCompletionSink}; + use std::sync::Arc; + + #[tokio::test] + async fn submits_exactly_once_in_scheduled_context() { + let sink = Arc::new(ScheduledCompletionSink::default()); + let context = ToolExecutionContext::for_session("scheduled-run:1") + .with_execution_origin(ExecutionOrigin::Scheduled { job_run_id: 1 }) + .with_scheduled_completion(sink.clone()); + let tool = CompleteScheduledRunTool::new(); + let first = tool + .execute_with_context(&context, json!({"outcome":"ok","message":"healthy"})) + .await + .unwrap(); + assert!(first.result.success); + assert_eq!(sink.outcome().unwrap().message, "healthy"); + let second = tool + .execute_with_context(&context, json!({"outcome":"alert","message":"again"})) + .await + .unwrap(); + assert!(!second.result.success); + } + + #[tokio::test] + async fn rejects_empty_extra_and_oversized_arguments() { + let sink = Arc::new(ScheduledCompletionSink::default()); + let context = ToolExecutionContext::for_session("scheduled-run:1") + .with_execution_origin(ExecutionOrigin::Scheduled { job_run_id: 1 }) + .with_scheduled_completion(sink); + let tool = CompleteScheduledRunTool::new(); + for args in [ + json!({"outcome":"ok","message":" "}), + json!({"outcome":"ok","message":"fine","notify":false}), + json!({"outcome":"unknown","message":"fine"}), + ] { + assert!( + !tool + .execute_with_context(&context, args) + .await + .unwrap() + .result + .success + ); + } + assert!( + !tool + .execute_with_context( + &context, + json!({"outcome":"ok","message":"x".repeat(16_385)}), + ) + .await + .unwrap() + .result + .success + ); + } +} diff --git a/src/tools/cron.rs b/src/tools/cron.rs index 78ea5c9..fa0237e 100644 --- a/src/tools/cron.rs +++ b/src/tools/cron.rs @@ -4,8 +4,9 @@ use async_trait::async_trait; use serde_json::{Value, json}; use uuid::Uuid; +use crate::agent::AgentCatalog; use crate::scheduler::{Schedule, next_run_for_schedule}; -use crate::storage::{DeliveryPolicy, JobKind, ScheduledJob, Storage}; +use crate::storage::{DeliveryPolicy, ScheduledJob, ScheduledJobUpdate, Storage}; use crate::tools::traits::{Tool, ToolResult}; fn now_ms() -> i64 { @@ -15,16 +16,52 @@ fn now_ms() -> i64 { .as_millis() as i64 } +#[derive(Clone)] +struct ScheduledJobValidator { + valid_channels: Vec, + catalog: Arc, +} + +impl ScheduledJobValidator { + fn validate_channel(&self, channel: &str) -> Result<(), String> { + if self.valid_channels.iter().any(|value| value == channel) { + Ok(()) + } else { + Err(format!( + "unknown channel '{channel}'; available: {}", + self.valid_channels.join(", ") + )) + } + } + + fn validate_agent(&self, agent_id: Option<&str>) -> Result<(), String> { + match agent_id { + Some("") => Err("agent_id must not be empty".to_string()), + Some(agent_id) if self.catalog.get(agent_id).is_none() => { + Err(format!("unknown or disabled Agent '{agent_id}'")) + } + _ => Ok(()), + } + } +} + pub struct CronAddTool { storage: Arc, - valid_channels: Vec, + validator: ScheduledJobValidator, } impl CronAddTool { - pub fn new(storage: Arc, valid_channels: Vec) -> Self { + pub fn new( + storage: Arc, + valid_channels: Vec, + catalog: Arc, + ) -> Self { Self { storage, - valid_channels, + validator: ScheduledJobValidator { + valid_channels, + catalog, + }, } } } @@ -36,205 +73,109 @@ impl Tool for CronAddTool { } fn description(&self) -> &str { - "Create a new scheduled task (cron job). The task will execute an AI prompt on a schedule \ - and deliver the result to the specified channel/chat. \ - Important: the execution environment is a fresh session with no access to your current \ - conversation history. The prompt parameter MUST include all necessary context: \ - what to do, the target audience, required output format, and any background information. \ - Schedule formats: \ - - 'every': {\"type\":\"every\",\"every_ms\":3600000} for every hour, \ - - 'at': {\"type\":\"at\",\"at\":} for one-shot, \ - - 'cron': {\"type\":\"cron\",\"expr\":\"0 0 9 * * *\"} for cron expressions (6-field: sec min hour dom month dow)." + "Create an unattended AI scheduled task executed by the Root Agent or a named Agent. Each run is isolated (no chat history), so the prompt must carry all required context. Every run must finish by calling complete_scheduled_run exactly once; delivery_policy (always/on_alert/never) alone decides whether the structured result is delivered." } fn parameters_schema(&self) -> Value { json!({ "type": "object", "properties": { - "schedule": { - "type": "object", - "description": "Schedule definition. One of: {\"type\":\"every\",\"every_ms\":}, {\"type\":\"at\",\"at\":}, or {\"type\":\"cron\",\"expr\":\"\",\"tz\":\"\"}", - "required": ["type"] - }, - "prompt": { - "type": "string", - "description": "The AI prompt to execute on each trigger" - }, - "channel": { - "type": "string", - "description": "Target channel for delivering results (e.g., 'feishu', 'cli_chat')" - }, - "chat_id": { - "type": "string", - "description": "Target chat ID within the channel" - }, - "name": { - "type": "string", - "description": "Human-readable name for the job (optional, defaults to truncated prompt)" - }, - "model": { - "type": "string", - "description": "Optional model override for this job" - }, - "kind": { - "type": "string", - "enum": ["task", "monitor"], - "description": "task always reports a result; monitor is quiet when healthy" - }, + "schedule": {"type":"object", "description":"at/every/cron schedule", "required":["type"]}, + "prompt": {"type":"string", "minLength":1}, + "channel": {"type":"string", "minLength":1}, + "chat_id": {"type":"string", "minLength":1}, + "name": {"type":"string", "minLength":1}, + "agent_id": {"type":"string", "description":"Optional named Agent; omit for Root"}, "delivery_policy": { - "type": "string", - "enum": ["always", "on_alert", "never"], - "description": "Scheduler-managed delivery policy. Defaults to always for task and on_alert for monitor." + "type":"string", + "enum":["always","on_alert","never"], + "description":"always sends every result; on_alert suppresses only ok; never sends nothing" } }, - "required": ["schedule", "prompt", "channel", "chat_id"] + "required":["schedule","prompt","channel","chat_id"], + "additionalProperties": false }) } async fn execute(&self, args: Value) -> anyhow::Result { - let schedule_json = args - .get("schedule") - .ok_or_else(|| anyhow::anyhow!("missing 'schedule'"))?; - let schedule: Schedule = serde_json::from_value(schedule_json.clone()) - .map_err(|e| anyhow::anyhow!("invalid schedule: {}", e))?; - + let schedule = match parse_schedule(args.get("schedule"), now_ms()) { + Ok(schedule) => schedule, + Err(error) => return Ok(failure(error)), + }; let prompt = args .get("prompt") - .and_then(|v| v.as_str()) - .unwrap_or("") + .and_then(Value::as_str) + .unwrap_or_default() + .trim() .to_string(); - if prompt.is_empty() { - return Ok(ToolResult { - success: false, - output: String::new(), - error: Some("prompt is required".into()), - }); - } - let channel = args .get("channel") - .and_then(|v| v.as_str()) - .unwrap_or("") + .and_then(Value::as_str) + .unwrap_or_default() + .trim() .to_string(); - if !self.valid_channels.contains(&channel) { - return Ok(ToolResult { - success: false, - output: format!( - "Unknown channel '{}'. Available: {}", - channel, - self.valid_channels.join(", ") - ), - error: Some(format!("Unknown channel: {}", channel)), - }); - } - let chat_id = args .get("chat_id") - .and_then(|v| v.as_str()) - .unwrap_or("") + .and_then(Value::as_str) + .unwrap_or_default() + .trim() .to_string(); - if chat_id.is_empty() { - return Ok(ToolResult { - success: false, - output: String::new(), - error: Some("chat_id is required".into()), - }); + if prompt.is_empty() || chat_id.is_empty() { + return Ok(failure("prompt and chat_id are required")); } - + if let Err(error) = self.validator.validate_channel(&channel) { + return Ok(failure(error)); + } + let agent_id = args + .get("agent_id") + .and_then(Value::as_str) + .map(str::trim) + .map(str::to_string); + if let Err(error) = self.validator.validate_agent(agent_id.as_deref()) { + return Ok(failure(error)); + } + let delivery_policy = match parse_policy(args.get("delivery_policy")) { + Ok(policy) => policy.unwrap_or(DeliveryPolicy::Always), + Err(error) => return Ok(failure(error)), + }; let name = args .get("name") - .and_then(|v| v.as_str()) - .unwrap_or_else(|| { - // char-boundary-safe truncation to 50 bytes - let limit = 50; - if prompt.len() <= limit { - prompt.as_str() - } else { - let mut end = limit; - while !prompt.is_char_boundary(end) { - end -= 1; - } - &prompt[..end] - } - }) - .to_string(); - let model = args - .get("model") - .and_then(|v| v.as_str()) - .map(|s| s.to_string()); - let job_kind = match args.get("kind").and_then(|v| v.as_str()).unwrap_or("task") { - "task" => JobKind::Task, - "monitor" => JobKind::Monitor, - value => { - return Ok(ToolResult { - success: false, - output: String::new(), - error: Some(format!("invalid kind: {value}")), - }); - } - }; - let default_policy = if job_kind == JobKind::Monitor { - "on_alert" - } else { - "always" - }; - let delivery_policy = match args - .get("delivery_policy") - .and_then(|v| v.as_str()) - .unwrap_or(default_policy) - { - "always" => DeliveryPolicy::Always, - "on_alert" => DeliveryPolicy::OnAlert, - "never" => DeliveryPolicy::Never, - value => { - return Ok(ToolResult { - success: false, - output: String::new(), - error: Some(format!("invalid delivery_policy: {value}")), - }); - } - }; - + .and_then(Value::as_str) + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(str::to_string) + .unwrap_or_else(|| prompt.chars().take(50).collect()); let now = now_ms(); let next_run_at = next_run_for_schedule(&schedule, now) - .ok_or_else(|| anyhow::anyhow!("could not compute next run time from schedule"))?; - + .ok_or_else(|| anyhow::anyhow!("could not compute next run"))?; let id = Uuid::new_v4().to_string()[..8].to_string(); - let job = ScheduledJob { - id: id.clone(), - name: name.clone(), - schedule, - prompt, - channel, - chat_id, - model, - job_kind, - delivery_policy, - enabled: true, - delete_after_run: false, - next_run_at, - last_run_at: None, - last_status: None, - last_error: None, - created_at: now, - updated_at: now, - }; - - self.storage.add_scheduled_job(&job).await?; - - Ok(ToolResult { - success: true, - output: format!( - "Scheduled job created: id={}, name=\"{}\", next_run_at={}", - id, name, next_run_at - ), - error: None, - }) + self.storage + .add_scheduled_job(&ScheduledJob { + id: id.clone(), + name: name.clone(), + schedule, + prompt, + agent_id, + channel, + chat_id, + delivery_policy, + enabled: true, + next_run_at, + last_run_at: None, + last_outcome: None, + created_at: now, + updated_at: now, + locked_at: None, + lock_owner: None, + lease_until: None, + }) + .await?; + Ok(success(format!( + "Scheduled job created: id={id}, name=\"{name}\", next_run_at={next_run_at}" + ))) } } -// ── CronListTool ───────────────────────────────────────────────────────────── - pub struct CronListTool { storage: Arc, } @@ -252,7 +193,7 @@ impl Tool for CronListTool { } fn description(&self) -> &str { - "List all scheduled tasks (cron jobs) with their status and next run time." + "List scheduled tasks and their latest structured outcome." } fn read_only(&self) -> bool { @@ -261,68 +202,116 @@ impl Tool for CronListTool { fn parameters_schema(&self) -> Value { json!({ - "type": "object", - "properties": { - "status": { - "type": "string", - "enum": ["all", "enabled", "disabled"], - "description": "Filter by job status (default: all)" - } - } + "type":"object", + "properties":{"status":{"type":"string","enum":["all","enabled","disabled"]}} }) } async fn execute(&self, args: Value) -> anyhow::Result { - let filter = args.get("status").and_then(|v| v.as_str()).unwrap_or("all"); + let filter = args.get("status").and_then(Value::as_str).unwrap_or("all"); let jobs = self.storage.list_scheduled_jobs().await?; - - let filtered: Vec<&ScheduledJob> = match filter { - "enabled" => jobs.iter().filter(|j| j.enabled).collect(), - "disabled" => jobs.iter().filter(|j| !j.enabled).collect(), - _ => jobs.iter().collect(), - }; - - if filtered.is_empty() { - return Ok(ToolResult { - success: true, - output: "No scheduled jobs found.".into(), - error: None, - }); - } - - let mut lines = Vec::new(); - for j in &filtered { - let status = if j.enabled { "enabled" } else { "disabled" }; - let last = match (&j.last_status, &j.last_error) { - (Some(s), _) if s == "ok" => " last:ok".to_string(), - (Some(_), Some(e)) => format!(" last:err({})", &e[..e.len().min(40)]), - _ => String::new(), - }; - let model = j.model.as_deref().unwrap_or("default"); - lines.push(format!( - "[{}] id={} name=\"{}\" kind={} delivery={} channel={} chat={} model={} next={}{}", - status, - j.id, - j.name, - j.job_kind.as_str(), - j.delivery_policy.as_str(), - j.channel, - j.chat_id, - model, - j.next_run_at, - last - )); - } - - Ok(ToolResult { - success: true, - output: lines.join("\n"), - error: None, - }) + let jobs: Vec<_> = jobs + .into_iter() + .filter(|job| match filter { + "enabled" => job.enabled, + "disabled" => !job.enabled, + _ => true, + }) + .map(|job| { + json!({ + "id": job.id, + "name": job.name, + "enabled": job.enabled, + "agent_id": job.agent_id, + "delivery_policy": job.delivery_policy, + "channel": job.channel, + "chat_id": job.chat_id, + "next_run_at": job.next_run_at, + "last_run_at": job.last_run_at, + "last_outcome": job.last_outcome, + }) + }) + .collect(); + Ok(success(serde_json::to_string(&json!({"jobs":jobs}))?)) } } -// ── CronRemoveTool ─────────────────────────────────────────────────────────── +pub struct CronRunsTool { + storage: Arc, +} + +impl CronRunsTool { + pub fn new(storage: Arc) -> Self { + Self { storage } + } +} + +#[async_trait] +impl Tool for CronRunsTool { + fn name(&self) -> &str { + "cron_runs" + } + + fn description(&self) -> &str { + "Read structured execution and delivery results for one scheduled task, including silent runs." + } + + fn read_only(&self) -> bool { + true + } + + fn parameters_schema(&self) -> Value { + json!({ + "type":"object", + "properties":{ + "job_id":{"type":"string","minLength":1}, + "run_id":{"type":"integer","minimum":1}, + "limit":{"type":"integer","minimum":1,"maximum":100} + }, + "required":["job_id"], + "additionalProperties":false + }) + } + + async fn execute(&self, args: Value) -> anyhow::Result { + let job_id = args + .get("job_id") + .and_then(Value::as_str) + .unwrap_or_default(); + if job_id.is_empty() { + return Ok(failure("job_id is required")); + } + self.storage.get_scheduled_job(job_id).await?; + if let Some(value) = args.get("run_id") { + let Some(run_id) = value.as_i64() else { + return Ok(failure("run_id must be an integer greater than zero")); + }; + if run_id < 1 { + return Ok(failure("run_id must be greater than zero")); + } + let run = self.storage.get_scheduled_job_run(run_id).await?; + if run.job_id != job_id { + return Ok(failure("run does not belong to the requested job")); + } + return Ok(success(serde_json::to_string( + &json!({"run":scheduled_run_json(&run, true)}), + )?)); + } + let limit = match args.get("limit") { + Some(value) => match value.as_u64() { + Some(limit @ 1..=100) => limit as usize, + _ => return Ok(failure("limit must be an integer from 1 to 100")), + }, + None => 20, + }; + let runs = self.storage.list_scheduled_job_runs(job_id, limit).await?; + let runs = runs + .iter() + .map(|run| scheduled_run_json(run, false)) + .collect::>(); + Ok(success(serde_json::to_string(&json!({"runs":runs}))?)) + } +} pub struct CronRemoveTool { storage: Arc, @@ -341,58 +330,20 @@ impl Tool for CronRemoveTool { } fn description(&self) -> &str { - "Delete a scheduled task permanently by its job ID. Use cron_list first to find the ID." + "Permanently delete a scheduled task that has no active occurrence or pending delivery." } fn parameters_schema(&self) -> Value { - json!({ - "type": "object", - "properties": { - "job_id": { - "type": "string", - "description": "The ID of the job to delete" - } - }, - "required": ["job_id"] - }) + job_id_schema() } async fn execute(&self, args: Value) -> anyhow::Result { - let job_id = args - .get("job_id") - .and_then(|v| v.as_str()) - .unwrap_or("") - .to_string(); - if job_id.is_empty() { - return Ok(ToolResult { - success: false, - output: String::new(), - error: Some("job_id is required".into()), - }); - } - - match self.storage.get_scheduled_job(&job_id).await { - Ok(_) => {} - Err(_) => { - return Ok(ToolResult { - success: false, - output: format!("Job {} not found.", job_id), - error: Some("not found".into()), - }); - } - } - - self.storage.remove_scheduled_job(&job_id).await?; - Ok(ToolResult { - success: true, - output: format!("Job {} deleted.", job_id), - error: None, - }) + let job_id = required_job_id(&args)?; + self.storage.remove_scheduled_job(job_id).await?; + Ok(success(format!("Job {job_id} deleted."))) } } -// ── CronEnableTool ─────────────────────────────────────────────────────────── - pub struct CronEnableTool { storage: Arc, } @@ -410,60 +361,32 @@ impl Tool for CronEnableTool { } fn description(&self) -> &str { - "Enable a disabled scheduled task by its job ID." + "Enable a scheduled task and recompute its next run time. An expired one-shot At schedule must be updated to a future time before it can be enabled." } fn parameters_schema(&self) -> Value { - json!({ - "type": "object", - "properties": { - "job_id": { - "type": "string", - "description": "The ID of the job to enable" - } - }, - "required": ["job_id"] - }) + job_id_schema() } async fn execute(&self, args: Value) -> anyhow::Result { - let job_id = args - .get("job_id") - .and_then(|v| v.as_str()) - .unwrap_or("") - .to_string(); - if job_id.is_empty() { - return Ok(ToolResult { - success: false, - output: String::new(), - error: Some("job_id is required".into()), - }); + let job_id = required_job_id(&args)?; + let job = self.storage.get_scheduled_job(job_id).await?; + let now = now_ms(); + if matches!(job.schedule, Schedule::At { at } if at <= now) { + return Ok(failure( + "expired At task cannot be enabled; update its schedule to a future time", + )); } - - let job = self - .storage - .get_scheduled_job(&job_id) - .await - .map_err(|e| anyhow::anyhow!("Job {} not found: {}", job_id, e))?; - - let next = next_run_for_schedule(&job.schedule, now_ms()); + let next = next_run_for_schedule(&job.schedule, now) + .ok_or_else(|| anyhow::anyhow!("could not compute next run"))?; self.storage - .set_scheduled_job_enabled(&job_id, true) + .update_scheduled_job(job_id, ScheduledJobUpdate::default(), Some(next)) .await?; - if let Some(n) = next { - self.storage.set_scheduled_job_next_run(&job_id, n).await?; - } - - Ok(ToolResult { - success: true, - output: format!("Job {} enabled.", job_id), - error: None, - }) + self.storage.set_scheduled_job_enabled(job_id, true).await?; + Ok(success(format!("Job {job_id} enabled."))) } } -// ── CronDisableTool ────────────────────────────────────────────────────────── - pub struct CronDisableTool { storage: Arc, } @@ -481,62 +404,41 @@ impl Tool for CronDisableTool { } fn description(&self) -> &str { - "Disable a scheduled task by its job ID without deleting it." + "Disable future occurrences of a scheduled task without cancelling its current run." } fn parameters_schema(&self) -> Value { - json!({ - "type": "object", - "properties": { - "job_id": { - "type": "string", - "description": "The ID of the job to disable" - } - }, - "required": ["job_id"] - }) + job_id_schema() } async fn execute(&self, args: Value) -> anyhow::Result { - let job_id = args - .get("job_id") - .and_then(|v| v.as_str()) - .unwrap_or("") - .to_string(); - if job_id.is_empty() { - return Ok(ToolResult { - success: false, - output: String::new(), - error: Some("job_id is required".into()), - }); - } - - let _ = self - .storage - .get_scheduled_job(&job_id) - .await - .map_err(|e| anyhow::anyhow!("Job {} not found: {}", job_id, e))?; + let job_id = required_job_id(&args)?; + self.storage.get_scheduled_job(job_id).await?; self.storage - .set_scheduled_job_enabled(&job_id, false) + .set_scheduled_job_enabled(job_id, false) .await?; - - Ok(ToolResult { - success: true, - output: format!("Job {} disabled.", job_id), - error: None, - }) + Ok(success(format!("Job {job_id} disabled."))) } } -// ── CronUpdateTool ─────────────────────────────────────────────────────────── - pub struct CronUpdateTool { storage: Arc, + validator: ScheduledJobValidator, } impl CronUpdateTool { - pub fn new(storage: Arc) -> Self { - Self { storage } + pub fn new( + storage: Arc, + valid_channels: Vec, + catalog: Arc, + ) -> Self { + Self { + storage, + validator: ScheduledJobValidator { + valid_channels, + catalog, + }, + } } } @@ -547,374 +449,227 @@ impl Tool for CronUpdateTool { } fn description(&self) -> &str { - "Update fields of an existing scheduled task. Only specified fields are changed." + "Update selected fields of a scheduled task: name, prompt, schedule, channel, chat_id, agent_id, or delivery_policy. Omit a field to leave it unchanged; pass agent_id null to switch back to the Root Agent. An At schedule must be in the future." } fn parameters_schema(&self) -> Value { json!({ - "type": "object", - "properties": { - "job_id": { - "type": "string", - "description": "The ID of the job to update" - }, - "prompt": { - "type": "string", - "description": "New AI prompt" - }, - "schedule": { - "type": "object", - "description": "New schedule definition" - }, - "channel": { - "type": "string", - "description": "New target channel" - }, - "chat_id": { - "type": "string", - "description": "New target chat ID" - }, - "model": { - "type": "string", - "description": "New model override" - }, - "kind": { - "type": "string", - "enum": ["task", "monitor"] - }, - "delivery_policy": { - "type": "string", - "enum": ["always", "on_alert", "never"] - } + "type":"object", + "properties":{ + "job_id":{"type":"string","minLength":1}, + "name":{"type":"string","minLength":1}, + "prompt":{"type":"string","minLength":1}, + "schedule":{"type":"object"}, + "agent_id":{"type":["string","null"]}, + "channel":{"type":"string","minLength":1}, + "chat_id":{"type":"string","minLength":1}, + "delivery_policy":{"type":"string","enum":["always","on_alert","never"]} }, - "required": ["job_id"] + "required":["job_id"], + "additionalProperties":false }) } async fn execute(&self, args: Value) -> anyhow::Result { - let job_id = args - .get("job_id") - .and_then(|v| v.as_str()) - .unwrap_or("") - .to_string(); - if job_id.is_empty() { - return Ok(ToolResult { - success: false, - output: String::new(), - error: Some("job_id is required".into()), - }); - } - - let _ = self - .storage - .get_scheduled_job(&job_id) - .await - .map_err(|e| anyhow::anyhow!("Job {} not found: {}", job_id, e))?; - - let prompt = args - .get("prompt") - .and_then(|v| v.as_str()) - .map(|s| s.to_string()); - let schedule: Option = match args.get("schedule") { - Some(s) => Some( - serde_json::from_value(s.clone()) - .map_err(|e| anyhow::anyhow!("invalid schedule: {}", e))?, - ), + let job_id = required_job_id(&args)?; + self.storage.get_scheduled_job(job_id).await?; + let now = now_ms(); + let schedule = match args.get("schedule") { + Some(value) => Some(match parse_schedule(Some(value), now) { + Ok(schedule) => schedule, + Err(error) => return Ok(failure(error)), + }), None => None, }; + let next_run_at = schedule + .as_ref() + .and_then(|schedule| next_run_for_schedule(schedule, now)); let channel = args .get("channel") - .and_then(|v| v.as_str()) - .map(|s| s.to_string()); - let chat_id = args - .get("chat_id") - .and_then(|v| v.as_str()) - .map(|s| s.to_string()); - let model = args - .get("model") - .and_then(|v| v.as_str()) - .map(|s| s.to_string()); - let job_kind = match args.get("kind").and_then(|v| v.as_str()) { - Some("task") => Some(JobKind::Task), - Some("monitor") => Some(JobKind::Monitor), - Some(value) => { - return Ok(ToolResult { - success: false, - output: String::new(), - error: Some(format!("invalid kind: {value}")), - }); - } - None => None, + .and_then(Value::as_str) + .map(str::trim) + .map(str::to_string); + if let Some(channel) = channel.as_deref() + && let Err(error) = self.validator.validate_channel(channel) + { + return Ok(failure(error)); + } + let agent_id = if args.get("agent_id").is_some() { + Some( + args.get("agent_id") + .and_then(Value::as_str) + .map(str::trim) + .map(str::to_string), + ) + } else { + None }; - let delivery_policy = match args.get("delivery_policy").and_then(|v| v.as_str()) { - Some("always") => Some(DeliveryPolicy::Always), - Some("on_alert") => Some(DeliveryPolicy::OnAlert), - Some("never") => Some(DeliveryPolicy::Never), - Some(value) => { - return Ok(ToolResult { - success: false, - output: String::new(), - error: Some(format!("invalid delivery_policy: {value}")), - }); - } - None => None, + if let Some(value) = agent_id.as_ref() + && let Err(error) = self.validator.validate_agent(value.as_deref()) + { + return Ok(failure(error)); + } + let delivery_policy = match parse_policy(args.get("delivery_policy")) { + Ok(policy) => policy, + Err(error) => return Ok(failure(error)), }; - - self.storage - .update_scheduled_job(&job_id, prompt, schedule, channel, chat_id, model) - .await?; - self.storage - .set_scheduled_job_behavior(&job_id, job_kind, delivery_policy) - .await?; - - if args.get("schedule").is_some() { - let job = self.storage.get_scheduled_job(&job_id).await?; - if let Some(next) = next_run_for_schedule(&job.schedule, now_ms()) { - self.storage - .set_scheduled_job_next_run(&job_id, next) - .await?; + let name = string_field(&args, "name"); + let prompt = string_field(&args, "prompt"); + let chat_id = string_field(&args, "chat_id"); + for (field, value) in [ + ("name", name.as_deref()), + ("prompt", prompt.as_deref()), + ("chat_id", chat_id.as_deref()), + ] { + if value.is_some_and(str::is_empty) { + return Ok(failure(format!("{field} must not be empty"))); } } + let update = ScheduledJobUpdate { + name, + prompt, + schedule, + agent_id, + channel, + chat_id, + delivery_policy, + }; + self.storage + .update_scheduled_job(job_id, update, next_run_at) + .await?; + Ok(success(format!("Job {job_id} updated."))) + } +} - Ok(ToolResult { - success: true, - output: format!("Job {} updated.", job_id), - error: None, +fn parse_schedule(value: Option<&Value>, now: i64) -> Result { + let value = value.ok_or_else(|| "schedule is required".to_string())?; + let schedule: Schedule = serde_json::from_value(value.clone()) + .map_err(|error| format!("invalid schedule: {error}"))?; + match &schedule { + Schedule::At { at } if *at <= now => Err("At schedule must be in the future".to_string()), + Schedule::Every { every_ms: 0 } => Err("every_ms must be greater than zero".to_string()), + _ if next_run_for_schedule(&schedule, now).is_none() => { + Err("schedule has no computable next run".to_string()) + } + _ => Ok(schedule), + } +} + +fn parse_policy(value: Option<&Value>) -> Result, String> { + match value.and_then(Value::as_str) { + None if value.is_none() => Ok(None), + Some("always") => Ok(Some(DeliveryPolicy::Always)), + Some("on_alert") => Ok(Some(DeliveryPolicy::OnAlert)), + Some("never") => Ok(Some(DeliveryPolicy::Never)), + Some(other) => Err(format!("invalid delivery_policy: {other}")), + None => Err("delivery_policy must be a string".to_string()), + } +} + +fn string_field(args: &Value, name: &str) -> Option { + args.get(name) + .and_then(Value::as_str) + .map(str::trim) + .map(str::to_string) +} + +fn scheduled_run_json(run: &crate::storage::JobRun, detail: bool) -> Value { + let bounded = |value: Option<&str>, limit: usize| { + value.map(|text| { + if detail { + text.to_string() + } else { + let mut output = text.chars().take(limit).collect::(); + if text.chars().count() > limit { + output.push('…'); + } + output + } }) + }; + json!({ + "run_id": run.id, + "job_id": run.job_id, + "scheduled_for": run.scheduled_for, + "agent_run_id": run.agent_run_id, + "agent_id": run.agent_id, + "started_at": run.started_at, + "finished_at": run.finished_at, + "status": run.status, + "outcome": run.outcome, + "message": bounded(run.message.as_deref(), 512), + "diagnostic": bounded(run.diagnostic.as_deref(), 512), + "duration_ms": run.duration_ms, + "delivery_status": run.delivery_status, + "delivery_attempts": run.delivery_attempts, + "delivery_error": bounded(run.delivery_error.as_deref(), 512), + }) +} + +fn required_job_id(args: &Value) -> anyhow::Result<&str> { + let value = args + .get("job_id") + .and_then(Value::as_str) + .unwrap_or_default() + .trim(); + if value.is_empty() { + anyhow::bail!("job_id is required"); + } + Ok(value) +} + +fn job_id_schema() -> Value { + json!({ + "type":"object", + "properties":{"job_id":{"type":"string","minLength":1}}, + "required":["job_id"], + "additionalProperties":false + }) +} + +fn success(output: String) -> ToolResult { + ToolResult { + success: true, + output, + error: None, + } +} + +fn failure(error: impl Into) -> ToolResult { + ToolResult { + success: false, + output: String::new(), + error: Some(error.into()), } } #[cfg(test)] mod tests { use super::*; - use crate::scheduler::Schedule; - use crate::storage::{ScheduledJob, Storage}; - use serde_json::json; - use sqlx::SqlitePool; - async fn setup_storage() -> Arc { - let pool = SqlitePool::connect("sqlite::memory:").await.unwrap(); - Storage::init_scheduler_schema(&pool).await.unwrap(); - Arc::new(Storage { pool }) + #[test] + fn expired_at_is_rejected() { + assert!(parse_schedule(Some(&json!({"type":"at","at":10})), 10).is_err()); } - fn now() -> i64 { - std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap() - .as_millis() as i64 + #[test] + fn policy_has_only_three_values() { + assert_eq!( + parse_policy(Some(&json!("on_alert"))).unwrap(), + Some(DeliveryPolicy::OnAlert) + ); + assert!(parse_policy(Some(&json!("direct"))).is_err()); } #[tokio::test] - async fn test_cron_add_tool() { - let storage = setup_storage().await; - let tool = CronAddTool::new(storage.clone(), vec!["cli_chat".to_string()]); - let result = tool - .execute(json!({ - "schedule": {"type": "every", "every_ms": 3600000}, - "prompt": "report status", - "channel": "cli_chat", - "chat_id": "test-chat-1", - "name": "hourly report" - })) - .await - .unwrap(); - assert!(result.success); - assert!(result.output.contains("hourly report")); - - let jobs = storage.list_scheduled_jobs().await.unwrap(); - assert_eq!(jobs.len(), 1); - assert_eq!(jobs[0].name, "hourly report"); - assert_eq!(jobs[0].job_kind, JobKind::Task); - assert_eq!(jobs[0].delivery_policy, DeliveryPolicy::Always); - } - - #[tokio::test] - async fn monitor_defaults_to_alert_only_delivery() { - let storage = setup_storage().await; - let tool = CronAddTool::new(storage.clone(), vec!["cli_chat".to_string()]); - let result = tool - .execute(json!({ - "schedule": {"type": "every", "every_ms": 3600000}, - "prompt": "check health", - "channel": "cli_chat", - "chat_id": "test-chat-1", - "kind": "monitor" - })) - .await - .unwrap(); - assert!(result.success); - let jobs = storage.list_scheduled_jobs().await.unwrap(); - assert_eq!(jobs[0].job_kind, JobKind::Monitor); - assert_eq!(jobs[0].delivery_policy, DeliveryPolicy::OnAlert); - } - - #[tokio::test] - async fn test_cron_add_invalid_channel() { - let storage = setup_storage().await; - let tool = CronAddTool::new(storage.clone(), vec!["cli_chat".to_string()]); - let result = tool - .execute(json!({ - "schedule": {"type": "every", "every_ms": 3600000}, - "prompt": "test", - "channel": "nonexistent", - "chat_id": "x", - "name": "test" - })) - .await - .unwrap(); - assert!(!result.success); - assert!(result.error.as_ref().unwrap().contains("Unknown channel")); - } - - #[tokio::test] - async fn test_cron_list_tool() { - let storage = setup_storage().await; - let t = now(); - let job = ScheduledJob { - id: uuid::Uuid::new_v4().to_string(), - name: "list-test".into(), - schedule: Schedule::Every { every_ms: 1000 }, - prompt: "hi".into(), - channel: "cli_chat".into(), - chat_id: "c".into(), - model: None, - job_kind: JobKind::Task, - delivery_policy: DeliveryPolicy::Direct, - enabled: true, - delete_after_run: false, - next_run_at: t + 1000, - last_run_at: None, - last_status: None, - last_error: None, - created_at: t, - updated_at: t, - }; - storage.add_scheduled_job(&job).await.unwrap(); - - let tool = CronListTool::new(storage.clone()); - let result = tool.execute(json!({})).await.unwrap(); - assert!(result.success); - assert!(result.output.contains("list-test")); - } - - #[tokio::test] - async fn test_cron_remove_tool() { - let storage = setup_storage().await; - let t = now(); - let job = ScheduledJob { - id: "job-rm-tool".into(), - name: "rm me".into(), - schedule: Schedule::Every { every_ms: 1000 }, - prompt: "hi".into(), - channel: "cli_chat".into(), - chat_id: "c".into(), - model: None, - job_kind: JobKind::Task, - delivery_policy: DeliveryPolicy::Direct, - enabled: true, - delete_after_run: false, - next_run_at: t, - last_run_at: None, - last_status: None, - last_error: None, - created_at: t, - updated_at: t, - }; - storage.add_scheduled_job(&job).await.unwrap(); - - let tool = CronRemoveTool::new(storage.clone()); - let result = tool - .execute(json!({"job_id": "job-rm-tool"})) - .await - .unwrap(); - assert!(result.success); - assert!(storage.get_scheduled_job("job-rm-tool").await.is_err()); - } - - #[tokio::test] - async fn test_cron_enable_disable_tools() { - let storage = setup_storage().await; - let t = now(); - let job = ScheduledJob { - id: "job-toggle-tool".into(), - name: "toggle".into(), - schedule: Schedule::Every { every_ms: 1000 }, - prompt: "hi".into(), - channel: "cli_chat".into(), - chat_id: "c".into(), - model: None, - job_kind: JobKind::Task, - delivery_policy: DeliveryPolicy::Direct, - enabled: true, - delete_after_run: false, - next_run_at: t, - last_run_at: None, - last_status: None, - last_error: None, - created_at: t, - updated_at: t, - }; - storage.add_scheduled_job(&job).await.unwrap(); - - let disable_tool = CronDisableTool::new(storage.clone()); - let result = disable_tool - .execute(json!({"job_id": "job-toggle-tool"})) - .await - .unwrap(); - assert!(result.success); - - let got = storage.get_scheduled_job("job-toggle-tool").await.unwrap(); - assert!(!got.enabled); - - let enable_tool = CronEnableTool::new(storage.clone()); - let result = enable_tool - .execute(json!({"job_id": "job-toggle-tool"})) - .await - .unwrap(); - assert!(result.success); - - let got = storage.get_scheduled_job("job-toggle-tool").await.unwrap(); - assert!(got.enabled); - } - - #[tokio::test] - async fn test_cron_update_tool() { - let storage = setup_storage().await; - let t = now(); - let job = ScheduledJob { - id: "job-update-tool".into(), - name: "old".into(), - schedule: Schedule::Every { every_ms: 3600000 }, - prompt: "old prompt".into(), - channel: "feishu".into(), - chat_id: "oc_1".into(), - model: None, - job_kind: JobKind::Task, - delivery_policy: DeliveryPolicy::Direct, - enabled: true, - delete_after_run: false, - next_run_at: t + 1000, - last_run_at: None, - last_status: None, - last_error: None, - created_at: t, - updated_at: t, - }; - storage.add_scheduled_job(&job).await.unwrap(); - - let tool = CronUpdateTool::new(storage.clone()); - let result = tool - .execute(json!({ - "job_id": "job-update-tool", - "prompt": "new prompt", - "schedule": {"type": "every", "every_ms": 60000} - })) - .await - .unwrap(); - assert!(result.success); - - let got = storage.get_scheduled_job("job-update-tool").await.unwrap(); - assert_eq!(got.prompt, "new prompt"); + async fn cron_runs_is_read_only() { + let dir = tempfile::tempdir().unwrap(); + let storage = Arc::new( + Storage::new(&dir.path().join("cron-runs.db")) + .await + .unwrap(), + ); + let tool = CronRunsTool::new(storage); + assert!(tool.read_only()); } } diff --git a/src/tools/delegate.rs b/src/tools/delegate.rs index f629c63..f10ee71 100644 --- a/src/tools/delegate.rs +++ b/src/tools/delegate.rs @@ -155,7 +155,7 @@ impl DelegateTool { args: &Value, context: &ToolExecutionContext, ) -> anyhow::Result { - let mode = match args + let requested_mode = match args .get("mode") .and_then(Value::as_str) .unwrap_or("foreground") @@ -168,6 +168,13 @@ impl DelegateTool { ))); } }; + let background_downgraded = + requested_mode == ExecutionMode::Background && context.execution_origin.is_scheduled(); + let mode = if background_downgraded { + ExecutionMode::Foreground + } else { + requested_mode + }; let task_values: Vec<&Value> = match args.get("tasks").and_then(Value::as_array) { Some(tasks) if !tasks.is_empty() => tasks.iter().collect(), Some(_) => return Ok(failure("tasks must not be empty")), @@ -248,6 +255,7 @@ impl DelegateTool { success: all_completed, output: serde_json::to_string(&json!({ "status": if all_completed { "completed" } else { "partial" }, + "background_downgraded": background_downgraded, "results": payload }))?, error: None, diff --git a/src/tools/mod.rs b/src/tools/mod.rs index 1d9ee84..b04a494 100644 --- a/src/tools/mod.rs +++ b/src/tools/mod.rs @@ -3,6 +3,7 @@ pub mod bash; pub mod browser; pub mod calculator; pub mod chat_manager; +pub mod complete_scheduled_run; pub mod content_search; pub mod cron; pub mod delegate; @@ -32,6 +33,7 @@ pub use bash::BashTool; pub use browser::{BrowserProfilesTool, BrowserTool}; pub use calculator::CalculatorTool; pub use chat_manager::ChatManagerTool; +pub use complete_scheduled_run::CompleteScheduledRunTool; pub use content_search::ContentSearchTool; pub use delegate::DelegateTool; pub use emit_signal::EmitSignalTool; @@ -50,8 +52,9 @@ pub use reload_config::ReloadConfigTool; pub use send_message::SendMessageTool; pub use todo::TodoTool; pub use traits::{ - OutboundDelivery, OutboundMessenger, ProcessedToolOutput, Tool, ToolArtifact, - ToolArtifactAudience, ToolExecutionContext, ToolOutput, ToolOutputProcessor, ToolResult, + ExecutionOrigin, OutboundDelivery, OutboundMessenger, ProcessedToolOutput, + ScheduledCompletionSink, ScheduledOutcome, Tool, ToolArtifact, ToolArtifactAudience, + ToolExecutionContext, ToolOutput, ToolOutputProcessor, ToolResult, }; pub use web_fetch::WebFetchTool; diff --git a/src/tools/traits.rs b/src/tools/traits.rs index dc9bd96..be6dac6 100644 --- a/src/tools/traits.rs +++ b/src/tools/traits.rs @@ -1,6 +1,60 @@ use crate::bus::{MediaItem, MediaRef, MessageSource}; use async_trait::async_trait; +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub enum ExecutionOrigin { + #[default] + Interactive, + Scheduled { + job_run_id: i64, + }, +} + +impl ExecutionOrigin { + pub fn is_scheduled(self) -> bool { + matches!(self, Self::Scheduled { .. }) + } +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ScheduledOutcome { + pub kind: crate::storage::ScheduledOutcomeKind, + pub message: String, +} + +#[derive(Debug, Default)] +pub struct ScheduledCompletionSink { + outcome: std::sync::Mutex>, +} + +impl ScheduledCompletionSink { + pub fn submit(&self, outcome: ScheduledOutcome) -> Result<(), &'static str> { + let mut guard = self + .outcome + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + if guard.is_some() { + return Err("scheduled result was already submitted"); + } + *guard = Some(outcome); + Ok(()) + } + + pub fn outcome(&self) -> Option { + self.outcome + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .clone() + } + + pub fn is_completed(&self) -> bool { + self.outcome + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .is_some() + } +} + /// Session identity supplied by the runtime for tools that own external state. /// Ordinary stateless tools can ignore it through the default trait method. #[derive(Debug, Clone)] @@ -10,6 +64,8 @@ pub struct ToolExecutionContext { pub agent: Option>, pub cancellation: tokio_util::sync::CancellationToken, pub execution_gate: Option>, + pub execution_origin: ExecutionOrigin, + pub scheduled_completion: Option>, } impl Default for ToolExecutionContext { @@ -20,6 +76,8 @@ impl Default for ToolExecutionContext { agent: None, cancellation: tokio_util::sync::CancellationToken::new(), execution_gate: None, + execution_origin: ExecutionOrigin::Interactive, + scheduled_completion: None, } } } @@ -32,6 +90,8 @@ impl ToolExecutionContext { agent: None, cancellation: tokio_util::sync::CancellationToken::new(), execution_gate: None, + execution_origin: ExecutionOrigin::Interactive, + scheduled_completion: None, } } @@ -60,6 +120,19 @@ impl ToolExecutionContext { self.execution_gate = Some(gate); self } + + pub fn with_execution_origin(mut self, execution_origin: ExecutionOrigin) -> Self { + self.execution_origin = execution_origin; + self + } + + pub fn with_scheduled_completion( + mut self, + sink: std::sync::Arc, + ) -> Self { + self.scheduled_completion = Some(sink); + self + } } #[derive(Debug, Clone)] diff --git a/webui/package-lock.json b/webui/package-lock.json index 1320a3c..fdff2ea 100644 --- a/webui/package-lock.json +++ b/webui/package-lock.json @@ -1,12 +1,12 @@ { "name": "picobot-webui", - "version": "1.21.0", + "version": "1.22.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "picobot-webui", - "version": "1.21.0", + "version": "1.22.0", "dependencies": { "bits-ui": "^2.0.0", "dompurify": "^3.4.12", diff --git a/webui/package.json b/webui/package.json index a05b1d4..41c48b2 100644 --- a/webui/package.json +++ b/webui/package.json @@ -1,7 +1,7 @@ { "name": "picobot-webui", "private": true, - "version": "1.21.0", + "version": "1.22.0", "type": "module", "engines": { "node": ">=20" diff --git a/webui/src/lib/StatusBadge.svelte b/webui/src/lib/StatusBadge.svelte index 54f749f..e2a8a38 100644 --- a/webui/src/lib/StatusBadge.svelte +++ b/webui/src/lib/StatusBadge.svelte @@ -2,11 +2,11 @@ let { status = "unknown" } = $props(); const normalized = $derived(String(status).toLowerCase()); const tone = $derived( - ["completed", "success", "ok", "enabled"].includes(normalized) + ["completed", "success", "ok", "enabled", "delivered"].includes(normalized) ? "ok" - : ["failed", "error", "cancelled", "disabled"].includes(normalized) + : ["failed", "error", "cancelled", "disabled", "refused", "timed_out"].includes(normalized) ? "fail" - : ["running", "pending"].includes(normalized) ? "run" : "" + : ["running", "pending", "delivering", "alert", "unknown", "interrupted"].includes(normalized) ? "run" : "" ); diff --git a/webui/src/pages/TasksPage.svelte b/webui/src/pages/TasksPage.svelte index 94b2ef8..e9f0655 100644 --- a/webui/src/pages/TasksPage.svelte +++ b/webui/src/pages/TasksPage.svelte @@ -42,11 +42,23 @@ function dotColor(status) { if (status === "completed" || status === "success" || status === "ok") return "var(--signal)"; - if (status === "timeout") return "var(--accent)"; + if (status === "timed_out" || status === "unknown" || status === "interrupted") return "var(--accent)"; if (status === "running") return "var(--info)"; return "var(--danger)"; } + function scheduleLabel(schedule) { + if (!schedule) return "—"; + if (schedule.type === "at") return `单次 · ${formatTime(schedule.at)}`; + if (schedule.type === "every") return `每 ${Math.max(1, Math.round(schedule.every_ms / 60000))} 分钟`; + if (schedule.type === "cron") return `${schedule.expr}${schedule.tz ? ` · ${schedule.tz}` : ""}`; + return schedule.type || "—"; + } + + function deliveryLabel(policy) { + return { always: "始终通知", on_alert: "异常通知", never: "从不通知" }[policy] || policy; + } + onMount(() => { load(); const timer = setInterval(() => { tick += 1; }, 30000); @@ -58,7 +70,7 @@

定时任务

-

管理定时任务与巡检；后台子代理运行请到「子代理」页面查看。

+

查看统一的定时执行、结构化结果与通知状态；Agent 运行审计请到「子代理」页面。

@@ -73,8 +85,9 @@

{job.name}

{job.prompt}

- {job.cron} - {job.job_kind === "monitor" ? "巡检" : "任务"} · {job.delivery_policy} + {scheduleLabel(job.schedule)} + Agent：{job.agent_id || "Root"} + 投递：{deliveryLabel(job.delivery_policy)} {job.channel} · {job.chat_id} 下次 {countdown(job.next_run_at)} 上次 {formatTime(job.last_run_at)} @@ -87,9 +100,23 @@
{/if} - + - {#if runs[job.id]?.length}
{#each runs[job.id] as run}
{formatTime(run.finished_at)} · {run.duration_ms}ms{run.result_kind ? ` · ${run.result_kind}` : ""}{run.delivery_status ? ` · ${run.delivery_status}` : ""}
{/each}
{/if} + {#if runs[job.id]?.length} +
+ {#each runs[job.id] as run} +
+ + {formatTime(run.finished_at || run.started_at || run.scheduled_for)} · {run.duration_ms ?? "—"}ms · outcome={run.outcome || "—"} · delivery={run.delivery_status} ({run.delivery_attempts}) + + + {#if run.message}

结果
{run.message}

{/if} + {#if run.diagnostic}

诊断
{run.diagnostic}

{/if} + {#if run.delivery_error}

投递错误
{run.delivery_error}

{/if} +
+ {/each} +
+ {/if} {:else}
暂无定时任务
{/each} {/if} @@ -100,4 +127,9 @@ .cron { font-size: 12px; color: var(--text-soft); background: var(--code-bg); padding: 1px 6px; border-radius: 4px; border: 1px solid var(--line); font-variant-numeric: tabular-nums; } .status-dots { display: flex; gap: 4px; margin-top: 6px; align-items: center; } .run-row { display: flex; align-items: center; gap: 8px; flex-wrap: wrap; } + .run-detail { padding: 7px 0; border-top: 1px solid var(--line); } + .run-detail:first-child { border-top: 0; } + .run-detail summary { cursor: pointer; justify-content: space-between; } + .run-detail p { margin: 8px 0 0; white-space: pre-wrap; overflow-wrap: anywhere; font-size: 12px; color: var(--text-soft); } + .run-detail .diagnostic { color: var(--danger); }