Compare commits

...

3 Commits

Author SHA1 Message Date
xiaoxixi
334e98d894 Merge branch 'scheduled-run-design' 2026-08-21 14:59:22 +08:00
xiaoxixi
d9ad58b84b feat(scheduler): unify scheduled task execution and delivery
Replace the dual task/monitor model, NO_REPLY string protocol, and Agent
self-delivery with a single Scheduled Run path: claim-time JobRun snapshots,
isolated Root/named Agent execution, exactly-once complete_scheduled_run
termination, and Scheduler-owned policy delivery through a persistent outbox.

- SQLite v11: drop job_kind/model/delete_after_run, add job_runs with
  status/outcome joint constraints and delivery lease columns; one-shot
  BEGIN IMMEDIATE migration with atomic rollback.
- Non-blocking JoinSet event loop with bounded run/delivery concurrency;
  terminal commit before any channel I/O; recover unfinished runs as unknown.
- ExecutionOrigin::Scheduled propagates to descendants, completion sink is
  top-level only, background delegation downgrades to foreground.
- Typed delivery receipts, fixed target_session_id, idempotent
  scheduled:<job_run_id> history insert.
- New cron_runs read-only tool; cron_add/update drop kind/model; WebUI and
  Health consume the same JobRun projection.
- Bump version to 1.22.0.
2026-08-21 14:59:02 +08:00
xiaoxixi
ddd5efc11b docs: design unified scheduled run delivery 2026-08-20 20:15:13 +08:00
39 changed files with 5999 additions and 2119 deletions

View File

@ -92,7 +92,7 @@ Scheduler → SessionManager scheduled execution → AgentLoop → Scheduler del
- **TurnController** is the only owner of active Turn state; snapshots are complete latest-wins values, not token queues, and `Completed` is published only after atomic persistence succeeds - **TurnController** is the only owner of active Turn state; snapshots are complete latest-wins values, not token queues, and `Completed` is published only after atomic persistence succeeds
- **DeliveryCoordinator** projects active Turn snapshots without mutating history; it owns `TurnSink` lifecycle but no platform message IDs, which remain private to each sink - **DeliveryCoordinator** projects active Turn snapshots without mutating history; it owns `TurnSink` lifecycle but no platform message IDs, which remain private to each sink
- **WorkManager** owns the single active plan per session, item state transitions, plan versions, and plan-change events; plans are optional and absent from ordinary chat context - **WorkManager** owns the single active plan per session, item state transitions, plan versions, and plan-change events; plans are optional and absent from ordinary chat context
- **Scheduler** supports legacy direct-delivery jobs and managed `task`/`monitor` jobs; managed agents cannot call `send_message`, and `on_alert` suppresses only healthy informational results - **Scheduler** owns one unified Scheduled Run path: claim-time JobRun snapshots, isolated Root/named Agent execution, exactly-once `complete_scheduled_run`, structured outcome, and policy-driven outbox delivery. Scheduled origin propagates to descendants, forces background delegation to foreground, and disables direct messaging, signals, Inbox completion slots, and cron/config management tools; `on_alert` suppresses only structured `ok`
- **AgentLoop** is stateless across turns; it receives prepared history, drains same-Turn steering only at safe model boundaries, calls LLM providers, executes tools, and returns one result - **AgentLoop** is stateless across turns; it receives prepared history, drains same-Turn steering only at safe model boundaries, calls LLM providers, executes tools, and returns one result
- **Context overflow recovery** is type-driven: before tool progress Session may commit one checkpoint and retry once; after any tool batch AgentLoop may retry the current Provider step once from its in-memory transcript, preserving current tool calls/results, and Session must never restart that Turn from durable history - **Context overflow recovery** is type-driven: before tool progress Session may commit one checkpoint and retry once; after any tool batch AgentLoop may retry the current Provider step once from its in-memory transcript, preserving current tool calls/results, and Session must never restart that Turn from durable history
- **Context compaction** keeps `messages` append-only and uses one active checkpoint per Session (`summary + first_retained_seq`) for deterministic Provider projection; `/compact`, Turn-boundary auto compaction, and overflow share the same compactor/CAS commit path, Session restoration never derives context from Timeline or calls a Provider, the Model `token_limit` (default 128K) is the hard window ceiling and an optional Agent `token_limit` can only narrow it via `min(agent, model)`, summary input is bounded from that effective window rather than a fixed cap, and the only automatic threshold is `context_tokens > context_window - effective_reserve` - **Context compaction** keeps `messages` append-only and uses one active checkpoint per Session (`summary + first_retained_seq`) for deterministic Provider projection; `/compact`, Turn-boundary auto compaction, and overflow share the same compactor/CAS commit path, Session restoration never derives context from Timeline or calls a Provider, the Model `token_limit` (default 128K) is the hard window ceiling and an optional Agent `token_limit` can only narrow it via `min(agent, model)`, summary input is bounded from that effective window rather than a fixed cap, and the only automatic threshold is `context_tokens > context_window - effective_reserve`

View File

@ -1,6 +1,6 @@
[package] [package]
name = "picobot" name = "picobot"
version = "1.21.0" version = "1.22.0"
edition = "2024" edition = "2024"
[dependencies] [dependencies]

View File

@ -17,7 +17,7 @@ PicoBot 是一个用 Rust 编写的个人 AI 助手运行时。它在本地启
- 将同一套 Agent 能力接入飞书/Lark并可选用单张卡片实时更新回复。 - 将同一套 Agent 能力接入飞书/Lark并可选用单张卡片实时更新回复。
- 让 Agent 使用本地文件、Shell、搜索、HTTP、浏览器、MCP 工具完成任务。 - 让 Agent 使用本地文件、Shell、搜索、HTTP、浏览器、MCP 工具完成任务。
- 把长期偏好、事实和历史摘要存成可检索记忆。 - 把长期偏好、事实和历史摘要存成可检索记忆。
- 用 Cron 定时执行任务,并把结果发回目标渠道 - 用 Cron 运行隔离的 Root 或命名 Agent以结构化结果决定始终通知、异常通知或静默记录
- 通过 Skills 为 Agent 注入项目知识和专用操作指南。 - 通过 Skills 为 Agent 注入项目知识和专用操作指南。
## 快速开始 ## 快速开始
@ -152,7 +152,7 @@ picobot health --json
缺少核心或当前配置要求的依赖时退出码为 `1``rg` / `fd` 等有回退实现的加速项只会标记为 `DEGRADED`。运行中的 Gateway 也提供 `/health` 斜杠命令Agent 可调用同名 `health` 工具WebUI 的“配置 → 健康检查”可显示相同的结构化结果并手动复查;这些入口共享同一套只读检查逻辑。 缺少核心或当前配置要求的依赖时退出码为 `1``rg` / `fd` 等有回退实现的加速项只会标记为 `DEGRADED`。运行中的 Gateway 也提供 `/health` 斜杠命令Agent 可调用同名 `health` 工具WebUI 的“配置 → 健康检查”可显示相同的结构化结果并手动复查;这些入口共享同一套只读检查逻辑。
Debian/Ubuntu 将同一个 fd 程序安装为 `fdfind`,两者都视为首选文件搜索后端;只有退回传统 `find` 时才提示性能警告。启用浏览器工具后Health 除了检查 agent-browser 版本和浏览器路径,还会在隔离的临时 socket namespace 中执行完整离线 doctor分别报告浏览器安装、真实 headless 启动和运行环境,因此可发现“文件存在但 Chrome 无法启动”或缺少 Linux 共享库等问题。 Debian/Ubuntu 将同一个 fd 程序安装为 `fdfind`,两者都视为首选文件搜索后端;只有退回传统 `find` 时才提示性能警告。启用浏览器工具后Health 除了检查 agent-browser 版本和浏览器路径,还会在隔离的临时 socket namespace 中执行完整离线 doctor分别报告浏览器安装、真实 headless 启动和运行环境,因此可发现“文件存在但 Chrome 无法启动”或缺少 Linux 共享库等问题。Gateway 内按需检查还会报告定时任务的无效 Agent/渠道引用、投递积压、最近失败/超时/unknown、静默 unknown 和执行周期覆盖Health 不会触发任务或连接 Provider。
### 5.3 使用 WebUI ### 5.3 使用 WebUI
@ -273,7 +273,7 @@ WebUI/TUI 上传文件默认保存到 `~/.picobot/media/cli_chat`,单文件上
| `delivery` | 活动 Turn 的展示过滤、latest-wins 节流、终态投递与 TurnSink 生命周期 | | `delivery` | 活动 Turn 的展示过滤、latest-wins 节流、终态投递与 TurnSink 生命周期 |
| `tools` | Agent 可调用工具集合 | | `tools` | Agent 可调用工具集合 |
| `storage` | SQLite schema、CRUD、消息和任务持久化 | | `storage` | SQLite schema、CRUD、消息和任务持久化 |
| `scheduler` | 领取定时任务,执行普通/巡检 Agent并按投递策略记录或发送结果 | | `scheduler` | 原子领取 occurrence运行隔离的 Scheduled Agent并通过持久化 outbox 按策略投递结构化结果 |
| `work` | 管理 session 级单 active plan、并行子项状态和 WebSocket 变更事件 | | `work` | 管理 session 级单 active plan、并行子项状态和 WebSocket 变更事件 |
| `skills` | 加载 Skill并把 Skill 指南注入系统提示 | | `skills` | 加载 Skill并把 Skill 指南注入系统提示 |
| `mcp` | 连接 MCP Server将远端工具包装成普通 Tool | | `mcp` | 连接 MCP Server将远端工具包装成普通 Tool |
@ -329,7 +329,7 @@ PicoBot 有两类记忆:
| Knowledge | 偏好、事实、项目规则、长期可复用信息 | 长期保留,手动删除 | | Knowledge | 偏好、事实、项目规则、长期可复用信息 | 长期保留,手动删除 |
| Timeline | 长对话压缩后的历史摘要 | 默认保留 90 天 | | Timeline | 长对话压缩后的历史摘要 | 默认保留 90 天 |
每轮处理用户消息时MemoryManager 会按用户输入召回 Knowledge并作为运行时上下文附加到本轮用户消息。当前召回上限固定为 5`memory.recall_limit` 已支持解析但尚未接入 worker。长会话使用一个活动 checkpoint累计摘要加 `first_retained_seq` 之后的原始消息尾部构成模型上下文原始消息、工具调用结果、ID 和 seq 均不会被压缩改写。旧工具结果会保留在原始历史中,但 checkpoint 边界推进后不再永久占用 Provider 上下文。成功的语义摘要还会 best-effort 保存为 Timeline`timeline_recall` 检索Timeline 不参与会话恢复正确性。Scheduler 默认创建一个每日维护巡检,按 `memory.timeline_retention_days` 清理过期 TimelineKnowledge 不会被自动删除。 每轮处理用户消息时MemoryManager 会按用户输入召回 Knowledge并作为运行时上下文附加到本轮用户消息。当前召回上限固定为 5`memory.recall_limit` 已支持解析但尚未接入 worker。长会话使用一个活动 checkpoint累计摘要加 `first_retained_seq` 之后的原始消息尾部构成模型上下文原始消息、工具调用结果、ID 和 seq 均不会被压缩改写。旧工具结果会保留在原始历史中,但 checkpoint 边界推进后不再永久占用 Provider 上下文。成功的语义摘要还会 best-effort 保存为 Timeline`timeline_recall` 检索Timeline 不参与会话恢复正确性。Scheduler 默认创建一个每日维护任务,按 `memory.timeline_retention_days` 清理过期 Timeline结果通过 `complete_scheduled_run` 结构化提交,Knowledge 不会被自动删除。
模型的 `models.<name>.token_limit` 给出上下文窗口上限,未配置时默认为 128,000Agent 的 `agents.<name>.token_limit` 是可选的收紧上限,两者都有配置时有效窗口取二者最小值,因此 Agent 不能扩大模型窗口。自动压缩使用保留量阈值 `context_tokens > context_window - effective_reserve`,默认 reserve 为 16,384 tokens并尽量原样保留最近 20,000 tokens。小窗口会自动把 reserve 限制为窗口的一半、把近期保留量限制为有效阈值的一半。摘要请求不使用固定 32K 输入上限,而是按有效窗口扣除摘要输出、提示词和安全余量;超大历史只在摘要请求副本中按“已有 checkpoint + 最新消息优先”生成有界 head/tail 转录SQLite 原文不变。手动 `/compact` 跳过自动阈值;换成小模型后若发送前预检已发现硬超限,或首次请求返回真实 context overflow语义摘要不可用时才使用明确标记的确定性降级裁剪正式请求最多重试一次。若 overflow 发生在工具已经执行之后AgentLoop 只在当前内存转录上裁掉旧完整 Turn 并重试当前模型步骤一次,不会从数据库历史重跑工具。 模型的 `models.<name>.token_limit` 给出上下文窗口上限,未配置时默认为 128,000Agent 的 `agents.<name>.token_limit` 是可选的收紧上限,两者都有配置时有效窗口取二者最小值,因此 Agent 不能扩大模型窗口。自动压缩使用保留量阈值 `context_tokens > context_window - effective_reserve`,默认 reserve 为 16,384 tokens并尽量原样保留最近 20,000 tokens。小窗口会自动把 reserve 限制为窗口的一半、把近期保留量限制为有效阈值的一半。摘要请求不使用固定 32K 输入上限,而是按有效窗口扣除摘要输出、提示词和安全余量;超大历史只在摘要请求副本中按“已有 checkpoint + 最新消息优先”生成有界 head/tail 转录SQLite 原文不变。手动 `/compact` 跳过自动阈值;换成小模型后若发送前预检已发现硬超限,或首次请求返回真实 context overflow语义摘要不可用时才使用明确标记的确定性降级裁剪正式请求最多重试一次。若 overflow 发生在工具已经执行之后AgentLoop 只在当前内存转录上裁掉旧完整 Turn 并重试当前模型步骤一次,不会从数据库历史重跑工具。
@ -353,7 +353,8 @@ PicoBot 有两类记忆:
| `todo` | 为复杂、多轮任务创建并更新当前 session 的持久化计划 | | `todo` | 为复杂、多轮任务创建并更新当前 session 的持久化计划 |
| `send_message` | 向指定渠道或当前会话发送消息,可附带文件/截图WebUI/TUI 当前 Turn 的附件并入最终回复 | | `send_message` | 向指定渠道或当前会话发送消息,可附带文件/截图WebUI/TUI 当前 Turn 的附件并入最终回复 |
| `chat_manager` | 查看渠道、会话和历史消息 | | `chat_manager` | 查看渠道、会话和历史消息 |
| `cron_add/list/remove/enable/disable/update` | 管理定时任务 | | `cron_add/list/remove/enable/disable/update` | 管理定时任务;`agent_id` 选择 Root/命名 Agent`delivery_policy` 支持 `always/on_alert/never` |
| `cron_runs` | 查询定时任务的结构化执行结果、诊断和投递状态,包括静默任务 |
| `routine_maintenance` | 安全清理超过保留期的 Timeline不删除 Knowledge | | `routine_maintenance` | 安全清理超过保留期的 Timeline不删除 Knowledge |
| `health` | 检查核心、配置相关和可选运行依赖 | | `health` | 检查核心、配置相关和可选运行依赖 |
| `browser` | 可选 agent-browser 浏览器自动化;默认按 dialog 临时使用,长期任务可用 `persistent_id` 复用个人 Profile | | `browser` | 可选 agent-browser 浏览器自动化;默认按 dialog 临时使用,长期任务可用 `persistent_id` 复用个人 Profile |

View File

@ -2,7 +2,7 @@
本文档描述 PicoBot 当前实现的运行时边界、数据流、并发模型和演进约束。它面向维护者和后续参与改进的 Agent是代码架构的主入口行为细节仍以代码和测试为最终依据。 本文档描述 PicoBot 当前实现的运行时边界、数据流、并发模型和演进约束。它面向维护者和后续参与改进的 Agent是代码架构的主入口行为细节仍以代码和测试为最终依据。
流式模型输出、reasoning 展示、活动 Turn 快照和 Channel 实时投递的详细设计与取舍见 [STREAMING_TURN_DESIGN.md](STREAMING_TURN_DESIGN.md)。用户输入路由、Session 执行拆分、终态投递确认和历史增量校准的重构方案见 [MESSAGE_FLOW_REFACTOR_DESIGN.md](MESSAGE_FLOW_REFACTOR_DESIGN.md)。已实施的 checkpoint 上下文压缩、pi 风格 reserve 阈值、统一编排和 overflow 失败语义见 [CONTEXT_COMPACTION_DESIGN.md](CONTEXT_COMPACTION_DESIGN.md)。配置运行代、重载边界和失败语义见 [CONFIG_HOT_RELOAD_DESIGN.md](CONFIG_HOT_RELOAD_DESIGN.md)。具名子 Agent、委托图、后台收件箱、结果传递机制与 `queue`/`steer` 信号的设计见 [SUB_AGENT_DESIGN.md](SUB_AGENT_DESIGN.md)。 流式模型输出、reasoning 展示、活动 Turn 快照和 Channel 实时投递的详细设计与取舍见 [STREAMING_TURN_DESIGN.md](STREAMING_TURN_DESIGN.md)。用户输入路由、Session 执行拆分、终态投递确认和历史增量校准的重构方案见 [MESSAGE_FLOW_REFACTOR_DESIGN.md](MESSAGE_FLOW_REFACTOR_DESIGN.md)。已实施的 checkpoint 上下文压缩、pi 风格 reserve 阈值、统一编排和 overflow 失败语义见 [CONTEXT_COMPACTION_DESIGN.md](CONTEXT_COMPACTION_DESIGN.md)。配置运行代、重载边界和失败语义见 [CONFIG_HOT_RELOAD_DESIGN.md](CONFIG_HOT_RELOAD_DESIGN.md)。具名子 Agent、委托图、后台收件箱、结果传递机制与 `queue`/`steer` 信号的设计见 [SUB_AGENT_DESIGN.md](SUB_AGENT_DESIGN.md)。已实施的统一 Scheduled Run、结构化终结协议、中央投递和 v11 数据库迁移见 [SCHEDULED_RUN_DESIGN.md](SCHEDULED_RUN_DESIGN.md)。
## 1. 设计目标 ## 1. 设计目标
@ -78,7 +78,7 @@ flowchart LR
| `health` | 聚合只读依赖检查,供 CLI、Tool 与 slash command 复用 | 安装、修复或连接 Provider | | `health` | 聚合只读依赖检查,供 CLI、Tool 与 slash command 复用 | 安装、修复或连接 Provider |
| `storage` | SQLite schema、迁移、原子 CRUD | 运行时调度策略 | | `storage` | SQLite schema、迁移、原子 CRUD | 运行时调度策略 |
| `memory` | Knowledge/Timeline 的存取与召回 | 直接驱动消息发送 | | `memory` | Knowledge/Timeline 的存取与召回 | 直接驱动消息发送 |
| `scheduler` | 领取到期任务、执行普通/巡检 Agent、应用投递策略、原子记录结果 | 复用聊天会话历史、直接感知 Channel | | `scheduler` | 原子领取 occurrence、运行隔离的 Root/命名 Agent、提交结构化结果并 drain 持久化投递 outbox | 解析模型自然语言、绕过 Bus 直接调用 Channel |
| `work` | session 级单 active plan、并行子项状态机、版本和变更事件 | 执行模型调用、持有 Channel/WebSocket | | `work` | session 级单 active plan、并行子项状态机、版本和变更事件 | 执行模型调用、持有 Channel/WebSocket |
| `task_supervisor` | 后台任务注册、取消、限时回收 | 业务级重试和结果语义 | | `task_supervisor` | 后台任务注册、取消、限时回收 | 业务级重试和结果语义 |
@ -171,6 +171,18 @@ sequenceDiagram
不要把“已进入 Bus”误认为“外部渠道已收到”。需要确认语义时必须使用 `deliver_outbound` 不要把“已进入 Bus”误认为“外部渠道已收到”。需要确认语义时必须使用 `deliver_outbound`
### Scheduled Run
Scheduler 不复用聊天历史,也不根据模型正文猜测是否通知。一次到期状态在同一 SQLite 事务中取得 Job 租约、插入 `job_runs(status=claimed)`、快照 Agent/投递目标/策略,并提前推进 recurring `next_run_at``At` 在 claim 时立即禁用。事件循环以独立有界 JoinSet 执行 Run 和 drain delivery长任务不阻塞其他领取或通知。
每次 Run 通过 `AgentCoordinator` 建立顶层 `agent_runs` 审计记录并执行隔离的 Root 或命名 Agent。Scheduled origin 贯穿所有后代,但只有顶层获得 exactly-once `complete_scheduled_run` sink后代不继承 sink。Scheduled Agent 不获得 `send_message`、cron/config 管理或 `emit_signal`,不创建 Inbox completion slot任何 background 委托都收敛为 foreground。普通最终文本不代表成功没有提交 `ok/alert/failed/refused` 之一即 fail-closed。
顶层 AgentRun 终态、JobRun 的 lifecycle/outcome/message/diagnostic、Job 最近摘要、初始 delivery status 和租约释放在一个事务中提交。`always` 投递所有 outcome`on_alert` 只抑制结构化 `ok``never` 始终只记录。`job_runs` 同时作为轻量 outbox`pending → delivering → delivered/failed`瞬态错误最多进行三次持久化尝试OutboundDispatcher 返回清洗后的类型化回执Scheduler 不解析错误字符串。
首次投递把目标 dialog 固定到 `target_session_id`,并先用稳定消息 ID `scheduled:<job_run_id>` 幂等写入本地历史,再调用 `MessageBus::deliver_outbound`。发送成功但 ack 提交前崩溃允许带稳定 metadata 的重复通知,不能为避免重复而丢失告警。启动恢复把遗留 claimed/running JobRun 原子收敛为 `unknown+unknown`、关联非终态 AgentRun 收敛为 interrupted并按 claim-time 策略决定是否进入 outbox已推进的 occurrence 不自动重跑。完整状态矩阵、v11 schema 和迁移规则见 [SCHEDULED_RUN_DESIGN.md](SCHEDULED_RUN_DESIGN.md)。
按需 Health 检查只读查询任务引用、投递积压、最近失败/超时/unknown、`never+unknown`、Every 周期被执行时长覆盖以及不可计算的 next run它不执行任务、不连接 Provider也不修改数据。
### Control 消息 ### Control 消息
WebSocket dialog 操作通过 `ControlMessage` 携带一次性回复通道。Gateway control router 以最多 64 个并发受监督任务调用 `SessionManager`,再将 `SessionEvent` 回传给发起者;慢 control 不阻塞其他聊天的 inbound 路由。Bus 只承载消息,不解释操作。 WebSocket dialog 操作通过 `ControlMessage` 携带一次性回复通道。Gateway control router 以最多 64 个并发受监督任务调用 `SessionManager`,再将 `SessionEvent` 回传给发起者;慢 control 不阻塞其他聊天的 inbound 路由。Bus 只承载消息,不解释操作。

1142
docs/SCHEDULED_RUN_DESIGN.md Normal file

File diff suppressed because it is too large Load Diff

View File

@ -54,7 +54,13 @@
"gateway": { "gateway": {
"host": "127.0.0.1", "host": "127.0.0.1",
"port": 19876, "port": 19876,
"require_pairing": true "require_pairing": true,
"scheduler": {
"enabled": true,
"poll_interval_secs": 60,
"max_concurrent": 1,
"execution_timeout_secs": 900
}
}, },
"client": { "client": {
"gateway_url": "ws://127.0.0.1:19876/ws" "gateway_url": "ws://127.0.0.1:19876/ws"

View File

@ -10,7 +10,7 @@ Channel → MessageBus.inbound → Gateway processor → SessionManager → per-
AgentLoop → TurnEvent → TurnController → latest TurnSnapshot → DeliveryCoordinator → TurnSink → Channel AgentLoop → TurnEvent → TurnController → latest TurnSnapshot → DeliveryCoordinator → TurnSink → Channel
WebSocket/Channel → MessageBus.control → Gateway processor → SessionManager (dialog 操作) WebSocket/Channel → MessageBus.control → Gateway processor → SessionManager (dialog 操作)
Scheduler → SessionManager.handle_cron_message → AgentLoop → send_message Scheduler → occurrence/JobRun claim → AgentCoordinator → isolated Scheduled Agent → complete_scheduled_run → JobRun outbox → SessionManager/MessageBus
``` ```
## 模块职责 ## 模块职责

View File

@ -102,8 +102,8 @@ Gateway WebUI 的“配置”页可以编辑实际加载的配置文件。读取
|------|------|------|------| |------|------|------|------|
| `enabled` | bool | true | 是否启动调度器并注册 cron 工具 | | `enabled` | bool | true | 是否启动调度器并注册 cron 工具 |
| `poll_interval_secs` | int | 60 | 检查到期任务的轮询间隔 | | `poll_interval_secs` | int | 60 | 检查到期任务的轮询间隔 |
| `max_concurrent` | int | 1 | 每批到期任务的最大并发数,运行时限制在 1256 | | `max_concurrent` | int | 1 | 同时执行的 Scheduled Run 上限,运行时限制在 1256投递使用独立有界并发 |
| `execution_timeout_secs` | int | 900 | 单个定时任务 Agent 执行的硬超时;租约会覆盖执行和托管投递等待 | | `execution_timeout_secs` | int | 900 | 单个定时任务 Agent 执行的硬超时;Job 执行租约额外覆盖关停宽限,投递由持久化 outbox 独立恢复 |
## memory 字段 ## memory 字段
@ -116,7 +116,7 @@ Gateway WebUI 的“配置”页可以编辑实际加载的配置文件。读取
| `timeline_retention_days` | int | 90 | 默认日常维护巡检删除超过该期限的 TimelineKnowledge 不受影响 | | `timeline_retention_days` | int | 90 | 默认日常维护巡检删除超过该期限的 TimelineKnowledge 不受影响 |
| `max_failures_before_degrade` | int | 3 | 预留的归并失败阈值;当前无失败降级循环 | | `max_failures_before_degrade` | int | 3 | 预留的归并失败阈值;当前无失败降级循环 |
注意:当前 worker 的 Knowledge 召回数量仍固定为 5idle consolidation 和失败降级循环尚未接入。Timeline 清理由默认启用的 `picobot-routine-maintenance` 定时巡检执行 注意:当前 worker 的 Knowledge 召回数量仍固定为 5idle consolidation 和失败降级循环尚未接入。Timeline 清理由默认启用的 `picobot-routine-maintenance` Scheduled Run 执行;该任务使用 `never` 策略,结构化结果只进入运行审计和 Health
## channels.feishu 字段 ## channels.feishu 字段

View File

@ -2,7 +2,7 @@
数据库为 SQLite默认位于配置目录`~/.picobot``data/` 下的 `picobot.db`,与 workspace 相互独立。 数据库为 SQLite默认位于配置目录`~/.picobot``data/` 下的 `picobot.db`,与 workspace 相互独立。
连接启用 WAL、`synchronous=NORMAL`、foreign keys、5 秒 busy timeout连接池最多 8 个连接。当前 `PRAGMA user_version=10`;启动时会在事务内补齐旧库字段和索引,遇到比程序更新的 schema version 会拒绝启动。 连接启用 WAL、`synchronous=NORMAL`、foreign keys、5 秒 busy timeout连接池最多 8 个连接。当前 `PRAGMA user_version=11`;启动时会在单个事务内迁移旧库,遇到比程序更新的 schema version 会拒绝启动。
## sessions 表 ## sessions 表
@ -176,22 +176,21 @@ background 完成/信号投递的唯一事实源:`pending → leased → admit
| `name` | TEXT | 任务名称 | | `name` | TEXT | 任务名称 |
| `schedule` | TEXT | 调度规则 JSONat/every/cron | | `schedule` | TEXT | 调度规则 JSONat/every/cron |
| `prompt` | TEXT | 任务提示词 | | `prompt` | TEXT | 任务提示词 |
| `channel` | TEXT | 执行渠道 | | `agent_id` | TEXT | 可选命名 AgentNULL 表示 Root |
| `channel` | TEXT | 目标渠道 |
| `chat_id` | TEXT | 目标对话 | | `chat_id` | TEXT | 目标对话 |
| `model` | TEXT | 可选模型标记;当前会存储/展示,但 Scheduler 执行仍使用默认 Agent 模型 | | `delivery_policy` | TEXT | `always` / `on_alert` / `never` |
| `enabled` | INTEGER | 是否启用 (1/0) | | `enabled` | INTEGER | 是否启用 (1/0) |
| `delete_after_run` | INTEGER | 执行后自动删除 (1/0) |
| `next_run_at` | INTEGER | 下次执行时间 | | `next_run_at` | INTEGER | 下次执行时间 |
| `last_run_at` | INTEGER | 上次执行时间 | | `last_run_at` | INTEGER | 上次执行时间 |
| `last_status` | TEXT | 上次执行状态 | | `last_outcome` | TEXT | 最近结构化结果ok/alert/failed/refused/unknown |
| `last_error` | TEXT | 上次错误信息 |
| `locked_at` | INTEGER | 本次领取时间 | | `locked_at` | INTEGER | 本次领取时间 |
| `lock_owner` | TEXT | 领取任务的 Scheduler owner UUID | | `lock_owner` | TEXT | 本次 occurrence 的唯一 owner token |
| `lease_until` | INTEGER | 租约到期时间;进程崩溃后允许其他实例重新领取 | | `lease_until` | INTEGER | 租约到期时间 |
| `created_at` | INTEGER | 创建时间Unix 毫秒) | | `created_at` | INTEGER | 创建时间Unix 毫秒) |
| `updated_at` | INTEGER | 更新时间Unix 毫秒) | | `updated_at` | INTEGER | 更新时间Unix 毫秒) |
Scheduler 使用原子 `UPDATE ... RETURNING` 领取到期任务。任务结果、下次运行时间和租约释放在同一事务中提交,并校验 owner防止过期 worker 覆盖已恢复的任务 Scheduler 在领取事务中插入 JobRun、快照执行/投递字段并推进下次时间。`At` 在领取时立即禁用;执行失败或崩溃不重放同一个 occurrence
## job_runs 表 ## job_runs 表
@ -199,12 +198,26 @@ Scheduler 使用原子 `UPDATE ... RETURNING` 领取到期任务。任务结果
|------|------|------| |------|------|------|
| `id` | INTEGER PK | 自增 ID | | `id` | INTEGER PK | 自增 ID |
| `job_id` | TEXT FK | 关联任务,外键关联 scheduled_jobs(id) | | `job_id` | TEXT FK | 关联任务,外键关联 scheduled_jobs(id) |
| `scheduled_for` | INTEGER | 本 occurrence 原计划时间 |
| `agent_run_id` | TEXT FK | 顶层 AgentRun 审计记录 |
| `agent_id` | TEXT | claim-time Agent 快照 |
| `delivery_policy` | TEXT | claim-time 投递策略快照 |
| `target_channel` / `target_chat_id` | TEXT | claim-time 目标快照 |
| `target_session_id` | TEXT | 首次投递时固定的目标 dialog |
| `started_at` | INTEGER | 开始时间 | | `started_at` | INTEGER | 开始时间 |
| `finished_at` | INTEGER | 结束时间 | | `finished_at` | INTEGER | 结束时间 |
| `status` | TEXT | 执行状态 | | `status` | TEXT | claimed/running/completed/failed/timed_out/cancelled/interrupted/unknown |
| `output` | TEXT | 执行输出 | | `outcome` | TEXT | ok/alert/failed/refused/unknown与 status 有联合约束 |
| `error` | TEXT | 错误信息 | | `message` | TEXT | 面向用户的结构化结果 |
| `diagnostic` | TEXT | 有界内部诊断 |
| `duration_ms` | INTEGER | 耗时(毫秒) | | `duration_ms` | INTEGER | 耗时(毫秒) |
| `delivery_status` | TEXT | awaiting_result/not_requested/suppressed/pending/delivering/delivered/failed |
| `delivery_attempts` | INTEGER | 持久化投递尝试次数,最多 3 次 |
| `delivery_next_attempt_at` | INTEGER | 瞬态失败后的退避时间 |
| `delivery_lease_owner` / `delivery_lease_until` | TEXT / INTEGER | outbox 领取租约 |
| `delivery_error` | TEXT | 清洗后的投递失败摘要 |
JobRun 是执行结果和投递状态的唯一权威。顶层 AgentRun 与 JobRun 终态、Job 最近摘要和租约释放原子提交;启动恢复将遗留运行归为 `unknown`,不会自动重跑。
## llm_calls 表 ## llm_calls 表

View File

@ -50,14 +50,15 @@
## Cron 定时任务工具 ## Cron 定时任务工具
Cron 不是一个带 `action` 的统一工具,而是个独立工具;仅在 `gateway.scheduler.enabled=true` 时注册。 Cron 不是一个带 `action` 的统一工具,而是个独立工具;仅在 `gateway.scheduler.enabled=true` 时注册。
| 工具 | 主要参数 | 说明 | | 工具 | 主要参数 | 说明 |
|------|----------|------| |------|----------|------|
| `cron_add` | `schedule`, `prompt`, `channel`, `chat_id`; 可选 `name`, `model` | 创建任务 | | `cron_add` | `schedule`, `prompt`, `channel`, `chat_id`; 可选 `name`, `agent_id`, `delivery_policy` | 创建任务 |
| `cron_list` | 可选 `status=all|enabled|disabled` | 列出任务 | | `cron_list` | 可选 `status=all|enabled|disabled` | 列出任务 |
| `cron_update` | `job_id`; 可选 `prompt`, `schedule`, `channel`, `chat_id`, `model` | 更新指定字段 | | `cron_runs` | `job_id`; 可选 `run_id`, `limit` | 查询结构化运行和投递记录,包括静默结果 |
| `cron_remove` | `job_id` | 永久删除任务和关联 job runs | | `cron_update` | `job_id`; 可选 `name`, `prompt`, `schedule`, `channel`, `chat_id`, `agent_id`, `delivery_policy` | 更新指定字段;`agent_id:null` 切回 Root |
| `cron_remove` | `job_id` | 无活动 Run 或 pending delivery 时永久删除任务 |
| `cron_enable` | `job_id` | 启用并重新计算下次运行时间 | | `cron_enable` | `job_id` | 启用并重新计算下次运行时间 |
| `cron_disable` | `job_id` | 禁用但保留任务 | | `cron_disable` | `job_id` | 禁用但保留任务 |
@ -69,7 +70,9 @@ Cron 不是一个带 `action` 的统一工具,而是六个独立工具;仅
{"type":"cron","expr":"0 0 9 * * *","tz":"Asia/Shanghai"} {"type":"cron","expr":"0 0 9 * * *","tz":"Asia/Shanghai"}
``` ```
时间戳和间隔单位为毫秒Cron 表达式为 6 段(秒、分、时、日、月、周)。定时 Agent 不复用聊天历史,`prompt` 必须包含完整上下文。`kind` 可为 `task``monitor``delivery_policy` 可为 `always``on_alert``never`。托管任务由 Scheduler 投递,巡检返回 `NO_REPLY[INFO]` 时静默,`NO_REPLY[FAIL]`/`NO_REPLY[REFUSE]` 仍视为需关注结果。升级前创建的任务保留 Agent 直接调用 `send_message` 的兼容行为。`model` 当前会持久化和展示,但执行仍使用默认 Agent Provider/Model不能依赖它实现模型覆盖。 时间戳和间隔单位为毫秒Cron 表达式为 6 段(秒、分、时、日、月、周)。过去时间的 At 不能创建、更新或直接重新启用。定时 Agent 不复用聊天历史,`prompt` 必须包含完整上下文;`agent_id` 省略时使用 Root否则使用当前 AgentCatalog 中的命名 Agent。
每次运行必须恰好一次调用 `complete_scheduled_run(outcome,message)`outcome 只能是 `ok``alert``failed``refused`。普通最终文本不会被解释为结果,缺少结构化终结会 fail-closed。投递完全由 Scheduler 决定:`always` 投递所有结果,`on_alert` 只抑制 `ok``never` 只保留记录。Scheduled Agent 不能自行发送最终通知;子 Agent 委托会同步完成,也不会产生后台 Inbox/Signal。
--- ---

View File

@ -78,6 +78,12 @@
"max_files_per_message": 8, "max_files_per_message": 8,
"max_message_bytes": 67108864, "max_message_bytes": 67108864,
"pending_ttl_seconds": 3600 "pending_ttl_seconds": 3600
},
"scheduler": {
"enabled": true,
"poll_interval_secs": 60,
"max_concurrent": 1,
"execution_timeout_secs": 900
} }
}, },
"client": { "client": {

View File

@ -1240,6 +1240,31 @@ impl AgentLoop {
} }
completed_tool_batches = completed_tool_batches.saturating_add(1); completed_tool_batches = completed_tool_batches.saturating_add(1);
if let Some(outcome) = tool_context
.scheduled_completion
.as_ref()
.and_then(|sink| sink.outcome())
{
Self::close_steering(turn.as_ref());
let mut final_message = ChatMessage::assistant(outcome.message);
attach_reply_media(&mut final_message, &reply_media_refs);
Self::annotate_message(&mut final_message, turn.as_ref(), iteration, true);
emitted_messages.push(final_message.clone());
self.forward_to_transcript_sink(&final_message);
crate::observability::metrics::global_metrics().record_turn(
Some(&accumulated_usage),
turn_start.elapsed().as_millis() as u64,
);
return Ok(AgentProcessResult {
final_response: final_message,
emitted_messages,
total_tokens: Some(accumulated_tokens),
usage: Some(accumulated_usage),
last_request_usage,
last_request_digest,
});
}
// A complete tool batch is the first safe steering boundary. Do // A complete tool batch is the first safe steering boundary. Do
// not drain at the final available iteration: those inputs must // not drain at the final available iteration: those inputs must
// remain in the closed mailbox for Session to queue after this // remain in the closed mailbox for Session to queue after this
@ -1517,6 +1542,17 @@ impl AgentLoop {
let mut outcomes = Vec::with_capacity(tool_calls.len()); let mut outcomes = Vec::with_capacity(tool_calls.len());
for tool_call in tool_calls { for tool_call in tool_calls {
if context
.scheduled_completion
.as_ref()
.is_some_and(|sink| sink.is_completed())
{
outcomes.push(ToolExecutionOutcome::failure(
"Cancelled: scheduled run was already completed".to_string(),
Some("scheduled run was already completed".to_string()),
));
continue;
}
if context.cancellation.is_cancelled() { if context.cancellation.is_cancelled() {
return Err(AgentError::Cancelled); return Err(AgentError::Cancelled);
} }
@ -1666,6 +1702,58 @@ mod tests {
requests: std::sync::atomic::AtomicUsize, requests: std::sync::atomic::AtomicUsize,
} }
struct ScheduledCompletionProvider {
requests: std::sync::atomic::AtomicUsize,
}
#[async_trait::async_trait]
impl LLMProvider for ScheduledCompletionProvider {
async fn stream(
&self,
_request: ChatCompletionRequest,
) -> Result<ProviderStream, crate::providers::DynProviderError> {
self.requests
.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
Ok(crate::providers::provider_stream_for_test(
ChatCompletionResponse {
id: "scheduled-complete".to_string(),
model: "scheduled-complete".to_string(),
content: String::new(),
reasoning_content: None,
provider_state: None,
tool_calls: vec![
ToolCall {
id: "complete".to_string(),
name: "complete_scheduled_run".to_string(),
arguments: serde_json::json!({
"outcome": "ok",
"message": "healthy"
}),
},
ToolCall {
id: "late-side-effect".to_string(),
name: "side_effect".to_string(),
arguments: serde_json::json!({}),
},
],
usage: Usage::default(),
},
))
}
fn ptype(&self) -> &str {
"test"
}
fn name(&self) -> &str {
"scheduled-complete"
}
fn model_id(&self) -> &str {
"scheduled-complete"
}
}
#[async_trait::async_trait] #[async_trait::async_trait]
impl LLMProvider for AlwaysOverflowProvider { impl LLMProvider for AlwaysOverflowProvider {
async fn stream( async fn stream(
@ -1725,6 +1813,53 @@ mod tests {
} }
} }
#[tokio::test]
async fn scheduled_completion_ends_the_loop_and_cancels_later_batch_calls() {
let provider = Arc::new(ScheduledCompletionProvider {
requests: std::sync::atomic::AtomicUsize::new(0),
});
let executions = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let tools = Arc::new(ToolRegistry::new());
tools.register(crate::tools::CompleteScheduledRunTool::new());
tools.register(CountingSideEffectTool {
executions: executions.clone(),
});
let agent = AgentLoop::with_provider_and_tools(
provider.clone(),
tools,
3,
"scheduled-complete".to_string(),
PathBuf::from("."),
Vec::new(),
);
let sink = Arc::new(crate::tools::ScheduledCompletionSink::default());
let context = ToolExecutionContext::for_session("scheduled-run:1")
.with_execution_origin(crate::tools::ExecutionOrigin::Scheduled { job_run_id: 1 })
.with_scheduled_completion(sink.clone());
let result = agent
.process_with_context(vec![ChatMessage::user("check")], context)
.await
.unwrap();
assert_eq!(result.final_response.content, "healthy");
assert_eq!(
sink.outcome().unwrap().kind,
crate::storage::ScheduledOutcomeKind::Ok
);
assert_eq!(executions.load(std::sync::atomic::Ordering::SeqCst), 0);
assert_eq!(
provider.requests.load(std::sync::atomic::Ordering::SeqCst),
1
);
assert!(result.emitted_messages.iter().any(|message| {
message.role == "tool"
&& message
.content
.contains("scheduled run was already completed")
}));
}
#[async_trait::async_trait] #[async_trait::async_trait]
impl LLMProvider for OverflowAfterToolProvider { impl LLMProvider for OverflowAfterToolProvider {
async fn stream( async fn stream(

View File

@ -32,6 +32,16 @@ pub struct BackgroundAdmission {
pub run_ids: Vec<String>, pub run_ids: Vec<String>,
} }
#[derive(Debug, Clone)]
pub struct ScheduledAgentExecution {
pub agent_run_id: String,
pub status: crate::storage::ScheduledRunStatus,
pub outcome: Option<crate::tools::ScheduledOutcome>,
pub error: Option<String>,
pub agent_terminal: AgentTerminalOutcome,
pub runtime_generation: i64,
}
pub struct AgentCoordinator { pub struct AgentCoordinator {
storage: Arc<Storage>, storage: Arc<Storage>,
manager: Arc<SubAgentManager>, manager: Arc<SubAgentManager>,
@ -88,6 +98,263 @@ impl AgentCoordinator {
}) })
} }
#[allow(clippy::too_many_arguments)]
pub async fn execute_scheduled(
self: &Arc<Self>,
job_run_id: i64,
lease_owner: &str,
job_id: &str,
job_name: &str,
agent_id: Option<&str>,
prompt: &str,
timeout_secs: u64,
) -> Result<ScheduledAgentExecution, CoordinatorError> {
let run_id = Uuid::new_v4().to_string();
let root_session_id = format!("scheduled-run:{job_run_id}");
let sink = Arc::new(crate::tools::ScheduledCompletionSink::default());
let caller = ToolExecutionContext::for_session(root_session_id.clone())
.with_turn_id(format!("scheduled:{job_run_id}"))
.with_execution_origin(crate::tools::ExecutionOrigin::Scheduled { job_run_id });
let contract = format!(
"## Unattended Scheduled Run\n\nYou are executing scheduled task “{job_name}” ({job_id}). The user will not see ordinary final text. After completing all necessary work, you must call complete_scheduled_run exactly once. Use ok only when the task completed and found nothing requiring attention; use alert for actionable findings; use failed when the task did not complete reliably; use refused for a permission or safety refusal. Legacy textual suppression and direct-messaging instructions are obsolete."
);
let config = SubAgentConfig {
target: agent_id.map(str::to_string),
prompt: prompt.to_string(),
context: Some(contract.clone()),
mode: ExecutionMode::Foreground,
allowed_tools: None,
max_iterations: None,
timeout_secs: Some(timeout_secs),
plan_item_id: None,
session_id: Some(root_session_id.clone()),
};
let mut resolution = if agent_id.is_some() {
let mut resolution = self.manager.resolve_agent(&config, &caller, &run_id)?;
resolution
.tools
.register(crate::tools::CompleteScheduledRunTool::new());
resolution.tool_context.session_id = Some(root_session_id.clone());
resolution.tool_context.scheduled_completion = Some(sink.clone());
resolution.timeout_secs = resolution.timeout_secs.min(timeout_secs);
resolution.signal_contract = None;
resolution
} else {
self.manager
.resolve_scheduled_root(&caller, &run_id, timeout_secs, sink.clone())?
};
resolution.tool_context.execution_origin =
crate::tools::ExecutionOrigin::Scheduled { job_run_id };
let now = chrono::Utc::now().timestamp_millis();
let deadline_at = now.saturating_add((resolution.timeout_secs * 1000) as i64);
let new_run = NewAgentRun {
id: run_id.clone(),
root_session_id,
root_turn_id: None,
parent_run_id: None,
caller_agent_id: "SCHEDULER".to_string(),
caller_scope_id: format!("scheduled:{job_id}"),
idempotency_key: Some(format!("scheduled:{job_run_id}")),
agent_id: agent_id.unwrap_or("ROOT").to_string(),
definition_hash: resolution.definition_hash.clone().unwrap_or_default(),
provider_profile: resolution.llm_profile.clone().unwrap_or_default(),
provider_name: resolution.provider_config.name.clone(),
model_id: resolution.provider_config.model_id.clone(),
mode: AgentRunMode::Foreground,
depth: 1,
plan_item_id: None,
execution_id: run_id.clone(),
task: prompt.to_string(),
context_json: None,
budget_json: serde_json::json!({
"remaining_runs": self.manager.catalog().max_runs_per_tree().saturating_sub(1),
"remaining_depth": self.manager.catalog().max_tree_depth().saturating_sub(1),
})
.to_string(),
signal_contract_json: None,
signal_delivery: None,
deadline_at,
runtime_generation: self.runtime_generation,
completion_slot_reserved: false,
};
match self
.storage
.accept_agent_runs(AcceptAgentRequest {
runs: vec![new_run],
now,
})
.await?
{
AcceptedAgentRuns::Accepted { .. } => {}
AcceptedAgentRuns::Existing { .. } => {
return Err(CoordinatorError::Rejected(format!(
"scheduled occurrence {job_run_id} already has an Agent run"
)));
}
}
let mut mark_attempt = 0_u64;
let marked = match loop {
match self
.storage
.mark_scheduled_run_running(job_run_id, lease_owner, Some(&run_id), now)
.await
{
Err(error) if error.is_transient() && mark_attempt < 2 => {
mark_attempt += 1;
tokio::time::sleep(std::time::Duration::from_millis(50 * mark_attempt)).await;
}
result => break result,
}
} {
Ok(marked) => marked,
Err(error) => {
let _ = self
.storage
.cancel_agent_run_with_completion(
&run_id,
"scheduled occurrence could not enter running state",
true,
now,
)
.await;
return Err(error.into());
}
};
if !marked {
let _ = self
.storage
.cancel_agent_run_with_completion(
&run_id,
"scheduled occurrence was no longer active",
true,
now,
)
.await;
return Err(CoordinatorError::Rejected(format!(
"scheduled occurrence {job_run_id} lost its lease"
)));
}
let (execution, agent_terminal) = match self
.execute_scheduled_agent_run(&run_id, &config, resolution)
.await
{
Ok(execution) => execution,
Err(error) => (
Err(error),
AgentTerminalOutcome::Failed {
error: "scheduled Agent could not enter its execution lifecycle".to_string(),
prompt_tokens: None,
completion_tokens: None,
cost: None,
signal_ids: Vec::new(),
},
),
};
let status = match &execution {
Ok(result) => match &result.status {
TaskStatus::Completed => crate::storage::ScheduledRunStatus::Completed,
TaskStatus::Failed(_) => crate::storage::ScheduledRunStatus::Failed,
TaskStatus::TimedOut => crate::storage::ScheduledRunStatus::TimedOut,
TaskStatus::Cancelled => crate::storage::ScheduledRunStatus::Interrupted,
},
Err(_) => crate::storage::ScheduledRunStatus::Failed,
};
let error = match &execution {
Ok(result) => match &result.status {
TaskStatus::Completed => None,
TaskStatus::Failed(_) => Some(
"scheduled Agent execution failed; inspect Gateway logs for details"
.to_string(),
),
TaskStatus::TimedOut => Some("scheduled Agent timed out".to_string()),
TaskStatus::Cancelled => Some("scheduled Agent was cancelled".to_string()),
},
Err(_) => Some(
"scheduled Agent execution failed; inspect Gateway logs for details".to_string(),
),
};
Ok(ScheduledAgentExecution {
agent_run_id: run_id,
status,
outcome: sink.outcome(),
error,
agent_terminal,
runtime_generation: self.runtime_generation,
})
}
async fn execute_scheduled_agent_run(
self: &Arc<Self>,
run_id: &str,
config: &SubAgentConfig,
resolution: ResolvedAgentRun,
) -> Result<
(
Result<SubAgentResult, CoordinatorError>,
AgentTerminalOutcome,
),
CoordinatorError,
> {
let execution_id = run_id.to_string();
let token = resolution.tool_context.cancellation.clone();
self.active_tokens.insert(run_id.to_string(), token);
let started = self
.storage
.mark_agent_run_running(run_id, &execution_id, chrono::Utc::now().timestamp_millis())
.await?;
if !started {
self.active_tokens.remove(run_id);
return Err(CoordinatorError::Rejected(format!(
"scheduled Agent run {run_id} was closed before execution started"
)));
}
let result = self
.manager
.execute_resolved(config, resolution, run_id)
.await
.map_err(CoordinatorError::SubAgent);
self.active_tokens.remove(run_id);
let terminal = match &result {
Ok(result) => match &result.status {
TaskStatus::Completed => AgentTerminalOutcome::Completed {
result: result.full_content.clone(),
prompt_tokens: None,
completion_tokens: None,
cost: None,
tool_calls: result.tool_calls_count as i64,
iterations: result.iterations as i64,
signal_ids: Vec::new(),
},
TaskStatus::Failed(error) => AgentTerminalOutcome::Failed {
error: error.clone(),
prompt_tokens: None,
completion_tokens: None,
cost: None,
signal_ids: Vec::new(),
},
TaskStatus::TimedOut => AgentTerminalOutcome::TimedOut {
deadline_at: chrono::Utc::now().timestamp_millis(),
signal_ids: Vec::new(),
},
TaskStatus::Cancelled => AgentTerminalOutcome::Interrupted {
reason: "scheduled Agent interrupted by shutdown".to_string(),
signal_ids: Vec::new(),
},
},
Err(error) => AgentTerminalOutcome::Failed {
error: error.to_string(),
prompt_tokens: None,
completion_tokens: None,
cost: None,
signal_ids: Vec::new(),
},
};
Ok((result, terminal))
}
/// Admit a named background run for the root caller and spawn its runner. /// Admit a named background run for the root caller and spawn its runner.
/// Completion is guaranteed by the reserved inbox slot; the returned ID /// Completion is guaranteed by the reserved inbox slot; the returned ID
/// is only valid when every durable step succeeded. /// is only valid when every durable step succeeded.
@ -100,6 +367,11 @@ impl AgentCoordinator {
caller: &ToolExecutionContext, caller: &ToolExecutionContext,
configs: Vec<SubAgentConfig>, configs: Vec<SubAgentConfig>,
) -> Result<BackgroundAdmission, CoordinatorError> { ) -> Result<BackgroundAdmission, CoordinatorError> {
if caller.execution_origin.is_scheduled() {
return Err(CoordinatorError::Rejected(
"scheduled Agents cannot create background runs".to_string(),
));
}
if caller.agent.is_some() { if caller.agent.is_some() {
return Err(CoordinatorError::Rejected( return Err(CoordinatorError::Rejected(
"nested background runs are not available yet; only the root Agent may delegate background work".to_string(), "nested background runs are not available yet; only the root Agent may delegate background work".to_string(),

View File

@ -21,7 +21,7 @@ pub use context_compaction::{
ContextRequestKey, ContextUsageTracker, PreviousCheckpoint, SequencedMessage, ContextRequestKey, ContextUsageTracker, PreviousCheckpoint, SequencedMessage,
context_request_digest, estimate_tokens, context_request_digest, estimate_tokens,
}; };
pub use coordinator::{AgentCoordinator, CoordinatorError}; pub use coordinator::{AgentCoordinator, CoordinatorError, ScheduledAgentExecution};
pub use definition::{AgentDefinition, AgentLimits}; pub use definition::{AgentDefinition, AgentLimits};
pub use gate::ExecutionGate; pub use gate::ExecutionGate;
pub use inbox::{AgentInboxNotifier, AgentInboxWakeTarget}; pub use inbox::{AgentInboxNotifier, AgentInboxWakeTarget};

View File

@ -10,6 +10,22 @@ use crate::providers::{LLMProvider, create_provider};
use crate::skills::SkillsLoader; use crate::skills::SkillsLoader;
use crate::tools::{ToolExecutionContext, ToolRegistry}; use crate::tools::{ToolExecutionContext, ToolRegistry};
const SCHEDULED_DISABLED_TOOLS: &[&str] = &[
"send_message",
"cron_add",
"cron_update",
"cron_remove",
"cron_enable",
"cron_disable",
"cron_list",
"cron_runs",
"reload_config",
"agent_task",
"chat_manager",
"todo",
"emit_signal",
];
const DEFAULT_MAX_ITERATIONS: usize = 99; const DEFAULT_MAX_ITERATIONS: usize = 99;
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
@ -191,7 +207,13 @@ impl SubAgentManager {
})?; })?;
let cancellation = caller.cancellation.child_token(); let cancellation = caller.cancellation.child_token();
let execution = if let Some(parent) = caller.agent.as_ref() { let execution = if let Some(parent) = caller.agent.as_ref() {
if !self.catalog.can_delegate(&parent.current_agent_id, target) { let allowed =
if caller.execution_origin.is_scheduled() && parent.current_agent_id == "ROOT" {
self.catalog.root_can_delegate(target)
} else {
self.catalog.can_delegate(&parent.current_agent_id, target)
};
if !allowed {
return Err(SubAgentError::Other(format!( return Err(SubAgentError::Other(format!(
"Agent '{}' is not allowed to delegate to '{target}'", "Agent '{}' is not allowed to delegate to '{target}'",
parent.current_agent_id parent.current_agent_id
@ -235,10 +257,14 @@ impl SubAgentManager {
.budget .budget
.remaining_depth .remaining_depth
.min(definition.limits.max_depth); .min(definition.limits.max_depth);
child.signal_contract = definition child.signal_contract = if caller.execution_origin.is_scheduled() {
.signal_contract None
.as_ref() } else {
.map(|contract| Arc::new(contract.clone())); definition
.signal_contract
.as_ref()
.map(|contract| Arc::new(contract.clone()))
};
Arc::new(child) Arc::new(child)
} else { } else {
if !self.catalog.root_can_delegate(target) { if !self.catalog.root_can_delegate(target) {
@ -252,7 +278,11 @@ impl SubAgentManager {
run_id: task_id.to_string(), run_id: task_id.to_string(),
execution_id: task_id.to_string(), execution_id: task_id.to_string(),
parent_run_id: None, parent_run_id: None,
caller_agent_id: "ROOT".to_string(), caller_agent_id: if caller.execution_origin.is_scheduled() {
"SCHEDULER".to_string()
} else {
"ROOT".to_string()
},
current_agent_id: target.to_string(), current_agent_id: target.to_string(),
ancestry: vec![target.to_string()], ancestry: vec![target.to_string()],
depth: 1, depth: 1,
@ -267,10 +297,14 @@ impl SubAgentManager {
.min(definition.limits.max_depth), .min(definition.limits.max_depth),
}, },
tree_runs: Arc::new(std::sync::atomic::AtomicUsize::new(1)), tree_runs: Arc::new(std::sync::atomic::AtomicUsize::new(1)),
signal_contract: definition signal_contract: if caller.execution_origin.is_scheduled() {
.signal_contract None
.as_ref() } else {
.map(|contract| Arc::new(contract.clone())), definition
.signal_contract
.as_ref()
.map(|contract| Arc::new(contract.clone()))
},
emitted_signals: Arc::new(std::sync::Mutex::new(Vec::new())), emitted_signals: Arc::new(std::sync::Mutex::new(Vec::new())),
}) })
}; };
@ -279,6 +313,9 @@ impl SubAgentManager {
if let Some(allowed) = config.allowed_tools.as_ref() { if let Some(allowed) = config.allowed_tools.as_ref() {
effective_names.retain(|name| allowed.iter().any(|allowed| allowed == name)); effective_names.retain(|name| allowed.iter().any(|allowed| allowed == name));
} }
if caller.execution_origin.is_scheduled() {
effective_names.retain(|name| !SCHEDULED_DISABLED_TOOLS.contains(&name.as_str()));
}
let has_get_skill = effective_names.iter().any(|name| name == "get_skill"); let has_get_skill = effective_names.iter().any(|name| name == "get_skill");
let mut names = effective_names; let mut names = effective_names;
names.retain(|name| name != "get_skill"); names.retain(|name| name != "get_skill");
@ -310,7 +347,7 @@ impl SubAgentManager {
// The signal tool is contract-bound: it exists only when the // The signal tool is contract-bound: it exists only when the
// definition declares a signal block and the durable Coordinator is // definition declares a signal block and the durable Coordinator is
// live. If either is missing the run cannot emit signals. // live. If either is missing the run cannot emit signals.
if definition.signal_contract.is_some() { if definition.signal_contract.is_some() && !caller.execution_origin.is_scheduled() {
match self.coordinator() { match self.coordinator() {
Some(coordinator) => { Some(coordinator) => {
let contract = definition.signal_contract.clone().unwrap(); let contract = definition.signal_contract.clone().unwrap();
@ -343,17 +380,91 @@ impl SubAgentManager {
agent_id: Some(target.to_string()), agent_id: Some(target.to_string()),
definition_hash: Some(definition.definition_hash.clone()), definition_hash: Some(definition.definition_hash.clone()),
llm_profile: definition.llm_profile.clone(), llm_profile: definition.llm_profile.clone(),
signal_contract: definition.signal_contract.clone(), signal_contract: (!caller.execution_origin.is_scheduled())
tool_context: ToolExecutionContext::for_session(format!("agent-run:{task_id}")) .then(|| definition.signal_contract.clone())
.with_turn_id( .flatten(),
caller tool_context: ToolExecutionContext::for_session(
.turn_id if caller.execution_origin.is_scheduled() {
.clone() root_session_id
.unwrap_or_else(|| task_id.to_string()), } else {
) format!("agent-run:{task_id}")
},
)
.with_turn_id(
caller
.turn_id
.clone()
.unwrap_or_else(|| task_id.to_string()),
)
.with_agent(execution)
.with_cancellation(cancellation)
.with_execution_gate(self.execution_gate.clone())
.with_execution_origin(caller.execution_origin),
})
}
pub(crate) fn resolve_scheduled_root(
&self,
caller: &ToolExecutionContext,
task_id: &str,
timeout_secs: u64,
sink: Arc<crate::tools::ScheduledCompletionSink>,
) -> Result<ResolvedAgentRun, SubAgentError> {
if !caller.execution_origin.is_scheduled() {
return Err(SubAgentError::Other(
"scheduled root resolution requires Scheduled execution origin".to_string(),
));
}
let root_session_id = caller.session_id.clone().ok_or_else(|| {
SubAgentError::Other("scheduled root requires an execution scope".to_string())
})?;
let run_id = task_id.to_string();
let cancellation = caller.cancellation.child_token();
let execution = Arc::new(crate::agent::AgentExecutionContext {
root_session_id: root_session_id.clone(),
root_turn_id: None,
run_id: run_id.clone(),
execution_id: run_id,
parent_run_id: None,
caller_agent_id: "SCHEDULER".to_string(),
current_agent_id: "ROOT".to_string(),
ancestry: vec!["ROOT".to_string()],
depth: 1,
plan_item_id: None,
cancellation: cancellation.clone(),
budget: crate::agent::AgentBudget {
remaining_runs: self.catalog.max_runs_per_tree().saturating_sub(1),
remaining_depth: self.catalog.max_tree_depth().saturating_sub(1),
},
tree_runs: Arc::new(std::sync::atomic::AtomicUsize::new(1)),
signal_contract: None,
emitted_signals: Arc::new(std::sync::Mutex::new(Vec::new())),
});
let tools = self.full_tools.without(SCHEDULED_DISABLED_TOOLS);
tools.register(crate::tools::CompleteScheduledRunTool::new());
let skills_prompt = self
.skills_loader
.as_ref()
.map(|loader| loader.build_skills_prompt())
.filter(|prompt| !prompt.is_empty());
Ok(ResolvedAgentRun {
provider_config: Arc::new(self.provider_config.clone()),
tools,
timeout_secs,
max_iterations: self.provider_config.max_tool_iterations,
max_result_chars: 16_384,
role_prompt: None,
skills_prompt,
tool_context: ToolExecutionContext::for_session(root_session_id)
.with_agent(execution) .with_agent(execution)
.with_cancellation(cancellation) .with_cancellation(cancellation)
.with_execution_gate(self.execution_gate.clone()), .with_execution_gate(self.execution_gate.clone())
.with_execution_origin(caller.execution_origin)
.with_scheduled_completion(sink),
agent_id: None,
definition_hash: None,
llm_profile: None,
signal_contract: None,
}) })
} }
@ -664,6 +775,25 @@ mod tests {
assert!(!crate::tools::Tool::runtime_injected(&reload)); assert!(!crate::tools::Tool::runtime_injected(&reload));
} }
#[test]
fn scheduled_runs_remove_direct_and_interactive_control_tools() {
for name in [
"send_message",
"cron_add",
"cron_runs",
"reload_config",
"agent_task",
"chat_manager",
"todo",
"emit_signal",
] {
assert!(SCHEDULED_DISABLED_TOOLS.contains(&name));
}
assert!(!SCHEDULED_DISABLED_TOOLS.contains(&"bash"));
assert!(!SCHEDULED_DISABLED_TOOLS.contains(&"delegate"));
assert!(!SCHEDULED_DISABLED_TOOLS.contains(&"complete_scheduled_run"));
}
#[test] #[test]
fn resolve_agent_rejects_missing_target() { fn resolve_agent_rejects_missing_target() {
let manager = manager(); let manager = manager();

View File

@ -5,7 +5,7 @@ use std::time::Duration;
use tokio::sync::mpsc; use tokio::sync::mpsc;
use crate::bus::{MessageBus, OutboundMessage}; use crate::bus::{DeliveryReceipt, MessageBus, OutboundMessage};
use crate::channels::ChannelManager; use crate::channels::ChannelManager;
use crate::channels::base::{Channel, ChannelError}; use crate::channels::base::{Channel, ChannelError};
use crate::delivery::ConversationWriteLocks; use crate::delivery::ConversationWriteLocks;
@ -64,12 +64,14 @@ impl OutboundDispatcher {
if sender.as_ref().is_none_or(mpsc::Sender::is_closed) { if sender.as_ref().is_none_or(mpsc::Sender::is_closed) {
let Some(channel) = self.channel_manager.get_channel(&msg.channel).await else { let Some(channel) = self.channel_manager.get_channel(&msg.channel).await else {
tracing::warn!(channel = %msg.channel, "No channel found for message"); tracing::warn!(channel = %msg.channel, "No channel found for message");
msg.complete_delivery(Err(format!("channel not found: {}", msg.channel))); msg.complete_delivery(DeliveryReceipt::PermanentFailure {
summary: format!("channel not found: {}", msg.channel),
});
continue; continue;
}; };
let (new_sender, receiver) = mpsc::channel(LANE_CAPACITY); let (new_sender, receiver) = mpsc::channel(LANE_CAPACITY);
if !self.spawn_lane(channel, receiver, msg.channel.clone(), msg.chat_id.clone()) { if !self.spawn_lane(channel, receiver, msg.channel.clone(), msg.chat_id.clone()) {
msg.complete_delivery(Err("dispatcher is shutting down".to_string())); msg.complete_delivery(DeliveryReceipt::DispatcherClosed);
continue; continue;
} }
lanes.insert(lane_key.clone(), new_sender.clone()); lanes.insert(lane_key.clone(), new_sender.clone());
@ -89,7 +91,9 @@ impl OutboundDispatcher {
capacity = LANE_CAPACITY, capacity = LANE_CAPACITY,
"Outbound lane full; rejecting message instead of blocking other destinations" "Outbound lane full; rejecting message instead of blocking other destinations"
); );
msg.complete_delivery(Err("outbound lane is full".to_string())); msg.complete_delivery(DeliveryReceipt::TransientFailure {
summary: "outbound lane is full".to_string(),
});
} }
Err(mpsc::error::TrySendError::Closed(msg)) => { Err(mpsc::error::TrySendError::Closed(msg)) => {
// The lane may have expired between the closed check and // The lane may have expired between the closed check and
@ -97,13 +101,15 @@ impl OutboundDispatcher {
lanes.remove(&lane_key); lanes.remove(&lane_key);
let Some(channel) = self.channel_manager.get_channel(&msg.channel).await else { let Some(channel) = self.channel_manager.get_channel(&msg.channel).await else {
tracing::warn!(channel = %msg.channel, "No channel found for message"); tracing::warn!(channel = %msg.channel, "No channel found for message");
msg.complete_delivery(Err(format!("channel not found: {}", msg.channel))); msg.complete_delivery(DeliveryReceipt::PermanentFailure {
summary: format!("channel not found: {}", msg.channel),
});
continue; continue;
}; };
let (new_sender, receiver) = mpsc::channel(LANE_CAPACITY); let (new_sender, receiver) = mpsc::channel(LANE_CAPACITY);
if !self.spawn_lane(channel, receiver, msg.channel.clone(), msg.chat_id.clone()) if !self.spawn_lane(channel, receiver, msg.channel.clone(), msg.chat_id.clone())
{ {
msg.complete_delivery(Err("dispatcher is shutting down".to_string())); msg.complete_delivery(DeliveryReceipt::DispatcherClosed);
continue; continue;
} }
match new_sender.try_send(msg) { match new_sender.try_send(msg) {
@ -111,9 +117,9 @@ impl OutboundDispatcher {
lanes.insert(lane_key, new_sender); lanes.insert(lane_key, new_sender);
} }
Err(error) => { Err(error) => {
error.into_inner().complete_delivery(Err( error
"outbound lane could not be restarted during shutdown".to_string(), .into_inner()
)); .complete_delivery(DeliveryReceipt::DispatcherClosed);
} }
} }
} }
@ -143,15 +149,15 @@ impl OutboundDispatcher {
Ok(None) | Err(_) => break, Ok(None) | Err(_) => break,
}; };
let result = Self::send_with_retry(&*channel, &msg, &target_lock).await; let result = Self::send_with_retry(&*channel, &msg, &target_lock).await;
if let Err(error) = &result { if result != DeliveryReceipt::Delivered {
tracing::error!( tracing::error!(
channel = %channel_name, channel = %channel_name,
chat_id = %chat_id, chat_id = %chat_id,
error = %error, result = ?result,
"Failed to send message after retries" "Failed to send message after retries"
); );
} }
msg.complete_delivery(result.map_err(|error| error.to_string())); msg.complete_delivery(result);
} }
}, },
) )
@ -161,26 +167,28 @@ impl OutboundDispatcher {
channel: &dyn Channel, channel: &dyn Channel,
msg: &OutboundMessage, msg: &OutboundMessage,
target_lock: &tokio::sync::Mutex<()>, target_lock: &tokio::sync::Mutex<()>,
) -> Result<(), ChannelError> { ) -> DeliveryReceipt {
let _guard = target_lock.lock().await; let _guard = target_lock.lock().await;
const DELAYS: &[u64] = &[1, 2, 4]; const DELAYS: &[u64] = &[1, 2, 4];
for (attempt, &delay) in DELAYS.iter().enumerate() { for (attempt, &delay) in DELAYS.iter().enumerate() {
let result = tokio::time::timeout(SEND_TIMEOUT, channel.send(msg.clone())).await; let result = tokio::time::timeout(SEND_TIMEOUT, channel.send(msg.clone())).await;
match result { match result {
Ok(Ok(())) => return Ok(()), Ok(Ok(())) => return DeliveryReceipt::Delivered,
Ok(Err(error)) if attempt < DELAYS.len() - 1 && error.is_transient() => { Ok(Err(error)) if attempt < DELAYS.len() - 1 && error.is_transient() => {
tracing::warn!(attempt = attempt + 1, delay, error = %error, "Send failed, retrying"); tracing::warn!(
attempt = attempt + 1,
delay,
error_class = channel_error_class(&error),
"Send failed, retrying"
);
} }
Ok(Err(error)) => return Err(error), Ok(Err(error)) => return receipt_from_channel_error(error),
Err(_) if attempt < DELAYS.len() - 1 => { Err(_) if attempt < DELAYS.len() - 1 => {
tracing::warn!(attempt = attempt + 1, delay, "Send timed out, retrying"); tracing::warn!(attempt = attempt + 1, delay, "Send timed out, retrying");
} }
Err(_) => { Err(_) => {
return Err(ChannelError::Other(format!( return DeliveryReceipt::TimedOut;
"send timed out after {} seconds",
SEND_TIMEOUT.as_secs()
)));
} }
} }
tokio::time::sleep(Duration::from_secs(delay)).await; tokio::time::sleep(Duration::from_secs(delay)).await;
@ -189,6 +197,36 @@ impl OutboundDispatcher {
} }
} }
fn channel_error_class(error: &ChannelError) -> &'static str {
match error {
ChannelError::ConnectionError(_) => "connection",
ChannelError::SendError(_) => "send",
ChannelError::BusError(_) => "bus",
ChannelError::ConfigError(_) => "config",
ChannelError::Other(_) => "other",
}
}
fn receipt_from_channel_error(error: ChannelError) -> DeliveryReceipt {
match error {
ChannelError::ConnectionError(_) => DeliveryReceipt::TransientFailure {
summary: "channel connection failed after retries".to_string(),
},
ChannelError::SendError(_) => DeliveryReceipt::TransientFailure {
summary: "channel send failed after retries".to_string(),
},
ChannelError::BusError(_) => DeliveryReceipt::TransientFailure {
summary: "channel bus was unavailable".to_string(),
},
ChannelError::ConfigError(_) => DeliveryReceipt::PermanentFailure {
summary: "channel configuration rejected delivery".to_string(),
},
ChannelError::Other(_) => DeliveryReceipt::PermanentFailure {
summary: "channel rejected delivery".to_string(),
},
}
}
/// Decrements the active-lane counter exactly once when a lane task ends, /// Decrements the active-lane counter exactly once when a lane task ends,
/// whether it exits normally, is cancelled, or is aborted. /// whether it exits normally, is cancelled, or is aborted.
struct LaneGuard { struct LaneGuard {
@ -348,7 +386,7 @@ mod tests {
message.channel = "missing".to_string(); message.channel = "missing".to_string();
let error = bus.deliver_outbound(message).await.unwrap_err(); let error = bus.deliver_outbound(message).await.unwrap_err();
assert!(matches!(error, crate::bus::BusError::DeliveryFailed(_))); assert!(matches!(error, crate::bus::BusError::DeliveryPermanent(_)));
task.abort(); task.abort();
supervisor.shutdown(Duration::from_secs(1)).await; supervisor.shutdown(Duration::from_secs(1)).await;
} }
@ -435,18 +473,40 @@ mod tests {
}; };
let target_lock = tokio::sync::Mutex::new(()); let target_lock = tokio::sync::Mutex::new(());
let error = OutboundDispatcher::send_with_retry( let receipt = OutboundDispatcher::send_with_retry(
&channel, &channel,
&outbound("invalid", "message"), &outbound("invalid", "message"),
&target_lock, &target_lock,
) )
.await .await;
.unwrap_err();
assert!(matches!(error, ChannelError::Other(_))); assert!(matches!(receipt, DeliveryReceipt::PermanentFailure { .. }));
assert_eq!(channel.attempts.load(Ordering::SeqCst), 1); assert_eq!(channel.attempts.load(Ordering::SeqCst), 1);
} }
#[test]
fn channel_errors_map_to_typed_sanitized_receipts() {
for error in [
ChannelError::ConnectionError("https://secret.example/?token=x".to_string()),
ChannelError::SendError("private response body".to_string()),
ChannelError::BusError("private queue detail".to_string()),
] {
let receipt = receipt_from_channel_error(error);
assert!(matches!(receipt, DeliveryReceipt::TransientFailure { .. }));
assert!(!format!("{receipt:?}").contains("private"));
assert!(!format!("{receipt:?}").contains("secret"));
}
for error in [
ChannelError::ConfigError("api_key=x".to_string()),
ChannelError::Other("private platform payload".to_string()),
] {
let receipt = receipt_from_channel_error(error);
assert!(matches!(receipt, DeliveryReceipt::PermanentFailure { .. }));
assert!(!format!("{receipt:?}").contains("private"));
assert!(!format!("{receipt:?}").contains("api_key"));
}
}
#[tokio::test] #[tokio::test]
async fn shared_write_lock_orders_dispatcher_with_live_turn_writes() { async fn shared_write_lock_orders_dispatcher_with_live_turn_writes() {
let channel = RecordingChannel { let channel = RecordingChannel {
@ -468,7 +528,7 @@ mod tests {
assert!(channel.sent.lock().await.is_empty()); assert!(channel.sent.lock().await.is_empty());
drop(live_write); drop(live_write);
send.await.unwrap(); assert_eq!(send.await, DeliveryReceipt::Delivered);
assert_eq!(channel.sent.lock().await.as_slice(), &["after-live-update"]); assert_eq!(channel.sent.lock().await.as_slice(), &["after-live-update"]);
} }
} }

View File

@ -509,17 +509,26 @@ pub struct OutboundMessage {
pub reply_to: Option<String>, pub reply_to: Option<String>,
pub media: Vec<MediaItem>, pub media: Vec<MediaItem>,
pub metadata: HashMap<String, String>, pub metadata: HashMap<String, String>,
pub(crate) delivery: Option<tokio::sync::watch::Sender<Option<Result<(), String>>>>, pub(crate) delivery: Option<tokio::sync::watch::Sender<Option<DeliveryReceipt>>>,
} }
impl OutboundMessage { impl OutboundMessage {
pub(crate) fn complete_delivery(&self, result: Result<(), String>) { pub(crate) fn complete_delivery(&self, result: DeliveryReceipt) {
if let Some(delivery) = &self.delivery { if let Some(delivery) = &self.delivery {
delivery.send_replace(Some(result)); delivery.send_replace(Some(result));
} }
} }
} }
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum DeliveryReceipt {
Delivered,
TransientFailure { summary: String },
PermanentFailure { summary: String },
TimedOut,
DispatcherClosed,
}
// ============================================================================ // ============================================================================
// ControlMessage - Message for control channel (session management) // ControlMessage - Message for control channel (session management)
// Uses SessionCommand from session module // Uses SessionCommand from session module

View File

@ -4,8 +4,8 @@ pub mod message;
pub use dispatcher::OutboundDispatcher; pub use dispatcher::OutboundDispatcher;
pub use message::{ pub use message::{
ChannelContext, ChatMessage, ClientVisibility, CommittedMessage, CommittedTurnDelta, ChannelContext, ChatMessage, ClientVisibility, CommittedMessage, CommittedTurnDelta,
CompletionStatus, ContentBlock, ControlMessage, InboundMessage, MediaItem, MediaRef, CompletionStatus, ContentBlock, ControlMessage, DeliveryReceipt, InboundMessage, MediaItem,
MessageSource, OutboundMessage, ProviderReasoningState, SourceKind, TurnOrigin, MediaRef, MessageSource, OutboundMessage, ProviderReasoningState, SourceKind, TurnOrigin,
}; };
use std::sync::Arc; use std::sync::Arc;
@ -80,7 +80,17 @@ impl MessageBus {
loop { loop {
delivery_rx.changed().await.map_err(|_| BusError::Closed)?; delivery_rx.changed().await.map_err(|_| BusError::Closed)?;
if let Some(result) = delivery_rx.borrow().clone() { if let Some(result) = delivery_rx.borrow().clone() {
return result.map_err(BusError::DeliveryFailed); return match result {
DeliveryReceipt::Delivered => Ok(()),
DeliveryReceipt::TransientFailure { summary } => {
Err(BusError::DeliveryTransient(summary))
}
DeliveryReceipt::PermanentFailure { summary } => {
Err(BusError::DeliveryPermanent(summary))
}
DeliveryReceipt::TimedOut => Err(BusError::DeliveryTimedOut),
DeliveryReceipt::DispatcherClosed => Err(BusError::Closed),
};
} }
} }
}) })
@ -138,7 +148,8 @@ pub struct QueueDepths {
#[derive(Debug)] #[derive(Debug)]
pub enum BusError { pub enum BusError {
Closed, Closed,
DeliveryFailed(String), DeliveryTransient(String),
DeliveryPermanent(String),
DeliveryTimedOut, DeliveryTimedOut,
} }
@ -146,7 +157,12 @@ impl std::fmt::Display for BusError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self { match self {
BusError::Closed => write!(f, "Bus channel closed"), BusError::Closed => write!(f, "Bus channel closed"),
BusError::DeliveryFailed(error) => write!(f, "Outbound delivery failed: {error}"), BusError::DeliveryTransient(error) => {
write!(f, "Transient outbound delivery failure: {error}")
}
BusError::DeliveryPermanent(error) => {
write!(f, "Permanent outbound delivery failure: {error}")
}
BusError::DeliveryTimedOut => write!(f, "Outbound delivery confirmation timed out"), BusError::DeliveryTimedOut => write!(f, "Outbound delivery confirmation timed out"),
} }
} }

View File

@ -812,8 +812,12 @@ fn scheduler_snapshot(jobs: &[crate::storage::ScheduledJob]) -> Value {
.iter() .iter()
.filter(|job| { .filter(|job| {
matches!( matches!(
job.last_status.as_deref(), job.last_outcome,
Some("error" | "timeout" | "delivery_error") Some(
crate::storage::ScheduledOutcomeKind::Failed
| crate::storage::ScheduledOutcomeKind::Refused
| crate::storage::ScheduledOutcomeKind::Unknown
)
) )
}) })
.count(); .count();
@ -1465,6 +1469,26 @@ pub async fn get_job_runs(
.list_scheduled_job_runs(&id, limit) .list_scheduled_job_runs(&id, limit)
.await .await
.map_err(ApiError::internal)?; .map_err(ApiError::internal)?;
let runs = runs
.into_iter()
.map(|run| {
json!({
"id": run.id,
"job_id": run.job_id,
"scheduled_for": run.scheduled_for,
"started_at": run.started_at,
"finished_at": run.finished_at,
"status": run.status,
"outcome": run.outcome,
"message": run.message,
"diagnostic": run.diagnostic,
"duration_ms": run.duration_ms,
"delivery_status": run.delivery_status,
"delivery_attempts": run.delivery_attempts,
"delivery_error": run.delivery_error,
})
})
.collect::<Vec<_>>();
Ok(Json(json!({ "runs": runs }))) Ok(Json(json!({ "runs": runs })))
} }
@ -1564,37 +1588,57 @@ mod tests {
id: &str, id: &str,
enabled: bool, enabled: bool,
next_run_at: i64, next_run_at: i64,
last_status: Option<&str>, last_outcome: Option<crate::storage::ScheduledOutcomeKind>,
) -> crate::storage::ScheduledJob { ) -> crate::storage::ScheduledJob {
crate::storage::ScheduledJob { crate::storage::ScheduledJob {
id: id.to_string(), id: id.to_string(),
name: id.to_string(), name: id.to_string(),
schedule: crate::scheduler::Schedule::Every { every_ms: 60_000 }, schedule: crate::scheduler::Schedule::Every { every_ms: 60_000 },
prompt: String::new(), prompt: String::new(),
agent_id: None,
channel: "cli_chat".to_string(), channel: "cli_chat".to_string(),
chat_id: "test".to_string(), chat_id: "test".to_string(),
model: None,
job_kind: crate::storage::JobKind::Task,
delivery_policy: crate::storage::DeliveryPolicy::Never, delivery_policy: crate::storage::DeliveryPolicy::Never,
enabled, enabled,
delete_after_run: false,
next_run_at, next_run_at,
last_run_at: None, last_run_at: None,
last_status: last_status.map(str::to_string), last_outcome,
last_error: None,
created_at: 0, created_at: 0,
updated_at: 0, updated_at: 0,
locked_at: None,
lock_owner: None,
lease_until: None,
} }
} }
#[test] #[test]
fn scheduler_snapshot_classifies_failures_and_next_enabled_run() { fn scheduler_snapshot_classifies_failures_and_next_enabled_run() {
let jobs = vec![ let jobs = vec![
scheduled_job("healthy", true, 300, Some("ok")), scheduled_job(
scheduled_job("error", true, 200, Some("error")), "healthy",
scheduled_job("timeout", false, 100, Some("timeout")), true,
scheduled_job("delivery", true, 400, Some("delivery_error")), 300,
scheduled_job("other", false, 50, Some("cancelled")), Some(crate::storage::ScheduledOutcomeKind::Ok),
),
scheduled_job(
"error",
true,
200,
Some(crate::storage::ScheduledOutcomeKind::Failed),
),
scheduled_job(
"refused",
false,
100,
Some(crate::storage::ScheduledOutcomeKind::Refused),
),
scheduled_job(
"unknown",
true,
400,
Some(crate::storage::ScheduledOutcomeKind::Unknown),
),
scheduled_job("other", false, 50, None),
]; ];
assert_eq!( assert_eq!(

View File

@ -178,6 +178,7 @@ impl GatewayState {
.init(&config, workspace_path.clone()) .init(&config, workspace_path.clone())
.await .await
.map_err(|e| format!("Failed to init channels: {}", e))?; .map_err(|e| format!("Failed to init channels: {}", e))?;
let available_channels = channel_manager.list_channel_names().await;
let turn_delivery = TurnDeliveryService::new( let turn_delivery = TurnDeliveryService::new(
delivery_coordinator.clone(), delivery_coordinator.clone(),
channel_manager.clone(), channel_manager.clone(),
@ -189,7 +190,10 @@ impl GatewayState {
} else { } else {
None None
}; };
let health = Arc::new(crate::health::HealthService::new(config.clone())); let health = Arc::new(
crate::health::HealthService::new(config.clone())
.with_scheduler_runtime(storage.clone(), available_channels.clone()),
);
let provider_profiles: std::collections::HashMap<String, _> = config let provider_profiles: std::collections::HashMap<String, _> = config
.agents .agents
.keys() .keys()
@ -245,9 +249,9 @@ impl GatewayState {
let session_manager = Arc::new(session_manager); let session_manager = Arc::new(session_manager);
session_manager.bind_inbox_wake(); session_manager.bind_inbox_wake();
let agent_catalog = session_manager.agent_catalog(); let agent_catalog = session_manager.agent_catalog();
health.bind_agent_catalog(agent_catalog.clone());
// Register send_message tool with available channel names // Register send_message tool with available channel names
let available_channels = channel_manager.list_channel_names().await;
let valid_channels = available_channels.clone(); let valid_channels = available_channels.clone();
session_manager.register_outbound_tool(available_channels); session_manager.register_outbound_tool(available_channels);
@ -277,11 +281,15 @@ impl GatewayState {
.tools() .tools()
.register(crate::tools::cron::CronAddTool::new( .register(crate::tools::cron::CronAddTool::new(
storage.clone(), storage.clone(),
valid_channels, valid_channels.clone(),
agent_catalog.clone(),
)); ));
session_manager session_manager
.tools() .tools()
.register(crate::tools::cron::CronListTool::new(storage.clone())); .register(crate::tools::cron::CronListTool::new(storage.clone()));
session_manager
.tools()
.register(crate::tools::cron::CronRunsTool::new(storage.clone()));
session_manager session_manager
.tools() .tools()
.register(crate::tools::cron::CronRemoveTool::new(storage.clone())); .register(crate::tools::cron::CronRemoveTool::new(storage.clone()));
@ -293,7 +301,11 @@ impl GatewayState {
.register(crate::tools::cron::CronDisableTool::new(storage.clone())); .register(crate::tools::cron::CronDisableTool::new(storage.clone()));
session_manager session_manager
.tools() .tools()
.register(crate::tools::cron::CronUpdateTool::new(storage.clone())); .register(crate::tools::cron::CronUpdateTool::new(
storage.clone(),
valid_channels,
agent_catalog.clone(),
));
tracing::info!("Cron tools registered"); tracing::info!("Cron tools registered");
} }
@ -332,7 +344,25 @@ impl GatewayState {
} }
/// Start the message processing loops /// Start the message processing loops
pub async fn start_message_processing(&self) { pub async fn start_message_processing(&self) -> Result<(), String> {
match self
.storage
.recover_scheduled_runs(chrono::Utc::now().timestamp_millis())
.await
{
Ok(recovered) if recovered > 0 => {
tracing::warn!(
recovered,
"Scheduled runs recovered as unknown on activation"
);
}
Ok(_) => {}
Err(error) => {
return Err(format!(
"Scheduled run recovery failed on activation: {error}"
));
}
}
// Recover durable Agent state for this runtime generation: interrupt // Recover durable Agent state for this runtime generation: interrupt
// runs of older generations, expire stale inbox leases and reconcile // runs of older generations, expire stale inbox leases and reconcile
// capacity rows. Runs never recover while the generation is still a candidate. // capacity rows. Runs never recover while the generation is still a candidate.
@ -351,7 +381,9 @@ impl GatewayState {
} }
} }
Err(error) => { Err(error) => {
tracing::error!(error = %error, "Agent state recovery failed on activation"); return Err(format!(
"Agent state recovery failed on activation: {error}"
));
} }
} }
} }
@ -461,6 +493,7 @@ impl GatewayState {
}); });
tracing::info!("Scheduler background task spawned"); tracing::info!("Scheduler background task spawned");
} }
Ok(())
} }
} }
@ -525,7 +558,7 @@ pub async fn run(
reload_controller.set_failed(current_generation, error.to_string()); reload_controller.set_failed(current_generation, error.to_string());
return Err(error.into()); return Err(error.into());
} }
state.start_message_processing().await; state.start_message_processing().await?;
reload_controller.set_phase(current_generation, reload::ReloadPhase::Active); reload_controller.set_phase(current_generation, reload::ReloadPhase::Active);
let app = build_router(state.clone()); let app = build_router(state.clone());
let generation_listener = TcpListener::from_std(listener.try_clone()?)?; let generation_listener = TcpListener::from_std(listener.try_clone()?)?;

View File

@ -459,7 +459,7 @@ mod tests {
Some("command") Some("command")
); );
assert!(!publish_task.is_finished()); assert!(!publish_task.is_finished());
output.complete_delivery(Ok(())); output.complete_delivery(crate::bus::DeliveryReceipt::Delivered);
publish_task.await.unwrap(); publish_task.await.unwrap();
} }

View File

@ -1,6 +1,7 @@
use std::collections::HashSet; use std::collections::HashSet;
use std::path::Path; use std::path::Path;
use std::process::{Output, Stdio}; use std::process::{Output, Stdio};
use std::sync::{Arc, RwLock};
use std::time::Duration; use std::time::Duration;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
@ -117,11 +118,41 @@ impl HealthReport {
#[derive(Clone)] #[derive(Clone)]
pub struct HealthService { pub struct HealthService {
config: Config, config: Config,
scheduler_runtime: Arc<RwLock<Option<SchedulerHealthRuntime>>>,
}
#[derive(Clone)]
struct SchedulerHealthRuntime {
storage: Arc<crate::storage::Storage>,
channels: HashSet<String>,
catalog: Option<Arc<crate::agent::AgentCatalog>>,
} }
impl HealthService { impl HealthService {
pub fn new(config: Config) -> Self { pub fn new(config: Config) -> Self {
Self { config } Self {
config,
scheduler_runtime: Arc::new(RwLock::new(None)),
}
}
pub fn with_scheduler_runtime(
self,
storage: Arc<crate::storage::Storage>,
channels: Vec<String>,
) -> Self {
*self.scheduler_runtime.write().unwrap() = Some(SchedulerHealthRuntime {
storage,
channels: channels.into_iter().collect(),
catalog: None,
});
self
}
pub fn bind_agent_catalog(&self, catalog: Arc<crate::agent::AgentCatalog>) {
if let Some(runtime) = self.scheduler_runtime.write().unwrap().as_mut() {
runtime.catalog = Some(catalog);
}
} }
pub async fn check(&self) -> HealthReport { pub async fn check(&self) -> HealthReport {
@ -139,9 +170,165 @@ impl HealthService {
]; ];
checks.extend(self.check_mcp_commands()); checks.extend(self.check_mcp_commands());
checks.extend(self.check_browser().await); checks.extend(self.check_browser().await);
checks.extend(self.check_scheduler().await);
HealthReport::from_checks(checks) HealthReport::from_checks(checks)
} }
async fn check_scheduler(&self) -> Vec<HealthCheck> {
let scheduler_config = self.config.gateway.scheduler.clone().unwrap_or_default();
if !scheduler_config.enabled {
return vec![HealthCheck {
name: "scheduled runs".to_string(),
category: "configured".to_string(),
required: false,
status: HealthStatus::Pass,
detail: "scheduler disabled; persisted jobs are not executed".to_string(),
remediation: None,
}];
}
let runtime = self.scheduler_runtime.read().unwrap().clone();
let Some(runtime) = runtime else {
return vec![HealthCheck {
name: "scheduled runs".to_string(),
category: "runtime".to_string(),
required: false,
status: HealthStatus::Pass,
detail: "runtime scheduler state is checked by the running Gateway".to_string(),
remediation: None,
}];
};
let jobs = match runtime.storage.list_scheduled_jobs().await {
Ok(jobs) => jobs,
Err(error) => {
return vec![HealthCheck {
name: "scheduled run storage".to_string(),
category: "runtime".to_string(),
required: true,
status: HealthStatus::Fail,
detail: format!("cannot read scheduled jobs: {error}"),
remediation: Some(
"Check the configured SQLite database and Gateway logs.".to_string(),
),
}];
}
};
let now = chrono::Utc::now().timestamp_millis();
let stale_after = scheduler_config
.poll_interval_secs
.max(60)
.saturating_mul(10_000)
.min(i64::MAX as u64) as i64;
let mut bad_references = Vec::new();
let mut stale_deliveries = Vec::new();
let mut unhealthy_latest = Vec::new();
let mut silent_unknown = Vec::new();
let mut covered_intervals = Vec::new();
let mut invalid_next = Vec::new();
match runtime
.storage
.list_stale_scheduled_deliveries(now.saturating_sub(stale_after), 100)
.await
{
Ok(runs) => {
stale_deliveries.extend(
runs.into_iter()
.map(|run| format!("{}#{}", run.job_id, run.id)),
);
}
Err(error) => bad_references.push(format!("delivery backlog unavailable ({error})")),
}
for job in &jobs {
if !runtime.channels.contains(&job.channel) {
bad_references.push(format!("{}: channel {}", job.id, job.channel));
}
if let Some(agent_id) = job.agent_id.as_deref()
&& runtime
.catalog
.as_ref()
.is_none_or(|catalog| catalog.get(agent_id).is_none())
{
bad_references.push(format!("{}: agent {}", job.id, agent_id));
}
if job.enabled && crate::scheduler::next_run_for_schedule(&job.schedule, now).is_none()
{
invalid_next.push(job.id.clone());
}
let runs = match runtime.storage.list_scheduled_job_runs(&job.id, 20).await {
Ok(runs) => runs,
Err(error) => {
bad_references.push(format!("{}: run history unavailable ({error})", job.id));
continue;
}
};
if let Some(latest) = runs.first() {
if matches!(
latest.status,
crate::storage::ScheduledRunStatus::Unknown
| crate::storage::ScheduledRunStatus::Failed
| crate::storage::ScheduledRunStatus::TimedOut
) {
unhealthy_latest.push(format!("{}#{}", job.id, latest.id));
}
if job.delivery_policy == crate::storage::DeliveryPolicy::Never
&& latest.outcome == Some(crate::storage::ScheduledOutcomeKind::Unknown)
{
silent_unknown.push(format!("{}#{}", job.id, latest.id));
}
if let crate::scheduler::Schedule::Every { every_ms } = job.schedule
&& i64::try_from(every_ms).ok().is_some_and(|interval| {
latest
.duration_ms
.is_some_and(|duration| duration >= interval)
})
{
covered_intervals.push(format!("{}#{}", job.id, latest.id));
}
}
}
vec![
scheduler_health_check(
"scheduled references",
bad_references,
"all Agent and channel references are available",
"Update or disable jobs that reference missing Agents or channels.",
),
scheduler_health_check(
"scheduled delivery backlog",
stale_deliveries,
"no stale pending or delivering notifications",
"Inspect cron_runs and the target channel configuration.",
),
scheduler_health_check(
"scheduled latest outcomes",
unhealthy_latest,
"no latest run is failed, timed out, or unknown",
"Inspect cron_runs for the diagnostic and assess external side effects before retrying.",
),
scheduler_health_check(
"silent unknown scheduled runs",
silent_unknown,
"no never-delivery job has an unknown latest outcome",
"Inspect the run manually; delivery_policy=never prevents automatic notification.",
),
scheduler_health_check(
"scheduled execution intervals",
covered_intervals,
"recent execution durations fit their Every intervals",
"Increase the interval or split long-running jobs.",
),
scheduler_health_check(
"scheduled next runs",
invalid_next,
"all enabled schedules can compute a next run",
"Correct the schedule expression or disable the job.",
),
]
}
fn check_mcp_commands(&self) -> Vec<HealthCheck> { fn check_mcp_commands(&self) -> Vec<HealthCheck> {
let mut seen = HashSet::new(); let mut seen = HashSet::new();
let mut checks = Vec::new(); let mut checks = Vec::new();
@ -365,6 +552,38 @@ impl HealthService {
} }
} }
fn scheduler_health_check(
name: &str,
findings: Vec<String>,
healthy_detail: &str,
remediation: &str,
) -> HealthCheck {
if findings.is_empty() {
HealthCheck {
name: name.to_string(),
category: "runtime".to_string(),
required: false,
status: HealthStatus::Pass,
detail: healthy_detail.to_string(),
remediation: None,
}
} else {
let total = findings.len();
let mut sample = findings.into_iter().take(5).collect::<Vec<_>>().join(", ");
if total > 5 {
sample.push_str(&format!(", and {} more", total - 5));
}
HealthCheck {
name: name.to_string(),
category: "runtime".to_string(),
required: false,
status: HealthStatus::Warning,
detail: format!("{total} finding(s): {sample}"),
remediation: Some(remediation.to_string()),
}
}
}
fn check_configuration_recovery(config: &Config) -> HealthCheck { fn check_configuration_recovery(config: &Config) -> HealthCheck {
if config.diagnostics.is_empty() { if config.diagnostics.is_empty() {
return HealthCheck { return HealthCheck {

View File

@ -3,81 +3,41 @@ pub mod types;
use std::sync::Arc; use std::sync::Arc;
use std::time::Instant; use std::time::Instant;
use futures_util::stream::{self, StreamExt}; use tokio::task::JoinSet;
use tokio::time; use tokio::time;
use crate::config::SchedulerConfig; use crate::config::SchedulerConfig;
use crate::session::SessionManager; use crate::session::{ScheduledDeliveryError, SessionManager};
use crate::session::session::HandleResult; use crate::storage::{
use crate::storage::ScheduledJob; ClaimedScheduledRun, JobRun, ScheduledOutcomeKind, ScheduledRunCompletion, ScheduledRunStatus,
use crate::storage::Storage; Storage,
use crate::storage::{DeliveryPolicy, JobKind, JobRun}; };
pub use types::Schedule; pub use types::Schedule;
#[derive(Debug, Clone, PartialEq, Eq)]
enum ScheduledDisposition {
Content(String),
Quiet(String),
ReportedFailure(String),
Refused(String),
}
fn parse_scheduled_disposition(output: &str) -> ScheduledDisposition {
let trimmed = output.trim();
if trimmed.eq_ignore_ascii_case("NO_REPLY") {
return ScheduledDisposition::Quiet(String::new());
}
let upper = trimmed.to_ascii_uppercase();
for (prefix, kind) in [
("NO_REPLY[INFO]", "info"),
("NO_REPLY[FAIL]", "fail"),
("NO_REPLY[REFUSE]", "refuse"),
] {
if upper.starts_with(prefix) {
let suffix = &trimmed[prefix.len()..];
if !suffix.is_empty() && !suffix.trim_start().starts_with(':') {
continue;
}
let reason = suffix.trim().trim_start_matches(':').trim().to_string();
return match kind {
"info" => ScheduledDisposition::Quiet(reason),
"fail" => ScheduledDisposition::ReportedFailure(reason),
_ => ScheduledDisposition::Refused(reason),
};
}
}
if trimmed.is_empty() {
ScheduledDisposition::ReportedFailure("scheduled agent returned empty output".into())
} else {
ScheduledDisposition::Content(trimmed.to_string())
}
}
/// Compute the next execution time (Unix ms) for a schedule, given `from` (Unix ms). /// Compute the next execution time (Unix ms) for a schedule, given `from` (Unix ms).
/// Returns `None` if no next time can be determined (e.g., invalid cron expression). /// Returns `None` if no next time can be determined (e.g. an invalid cron expression).
pub fn next_run_for_schedule(schedule: &Schedule, from: i64) -> Option<i64> { pub fn next_run_for_schedule(schedule: &Schedule, from: i64) -> Option<i64> {
use chrono::{TimeZone, Utc}; use chrono::{TimeZone, Utc};
use std::str::FromStr; use std::str::FromStr;
match schedule { match schedule {
Schedule::At { at } => Some(*at), Schedule::At { at } => Some(*at),
Schedule::Every { every_ms } => Some(from + *every_ms as i64), Schedule::Every { every_ms } => Some(from.saturating_add(i64::try_from(*every_ms).ok()?)),
Schedule::Cron { expr, tz } => { Schedule::Cron { expr, tz } => {
let cron_schedule = cron::Schedule::from_str(expr.as_str()).ok()?; let cron_schedule = cron::Schedule::from_str(expr.as_str()).ok()?;
let from_secs = from / 1000; let from_secs = from / 1000;
let from_nanos = ((from % 1000) * 1_000_000) as u32; let from_nanos = ((from % 1000) * 1_000_000) as u32;
let from_dt = Utc.timestamp_opt(from_secs, from_nanos).single()?; let from_dt = Utc.timestamp_opt(from_secs, from_nanos).single()?;
let next_utc = if let Some(tz_str) = tz { let next_utc = if let Some(tz_str) = tz {
let tz: chrono_tz::Tz = tz_str.parse().ok()?; let tz: chrono_tz::Tz = tz_str.parse().ok()?;
let from_local = from_dt.with_timezone(&tz); cron_schedule
let next_local = cron_schedule.after(&from_local).next()?; .after(&from_dt.with_timezone(&tz))
next_local.with_timezone(&Utc) .next()?
.with_timezone(&Utc)
} else { } else {
cron_schedule.after(&from_dt).next()? cron_schedule.after(&from_dt).next()?
}; };
Some(next_utc.timestamp_millis()) Some(next_utc.timestamp_millis())
} }
} }
@ -90,8 +50,6 @@ fn now_ms() -> i64 {
.as_millis() as i64 .as_millis() as i64
} }
/// The scheduler runs as a background tokio task, periodically checking for due jobs
/// and executing them via `SessionManager::handle_cron_message`.
pub struct Scheduler { pub struct Scheduler {
storage: Arc<Storage>, storage: Arc<Storage>,
session_manager: Arc<SessionManager>, session_manager: Arc<SessionManager>,
@ -129,15 +87,16 @@ impl Scheduler {
} }
} }
/// Claim due jobs with a durable lease, then execute the claimed batch with /// Non-blocking event loop. Execution and delivery use separate bounded
/// bounded concurrency. /// JoinSets so one long Agent run cannot delay other claims or outbox work.
pub async fn run(self: Arc<Self>) { pub async fn run(self: Arc<Self>) {
let poll_duration = time::Duration::from_secs(self.config.poll_interval_secs.max(1)); let poll_duration = time::Duration::from_secs(self.config.poll_interval_secs.max(1));
let mut interval = time::interval(poll_duration); let mut interval = time::interval(poll_duration);
interval.set_missed_tick_behavior(time::MissedTickBehavior::Skip); interval.set_missed_tick_behavior(time::MissedTickBehavior::Skip);
// Keep accidental configuration values from claiming an unbounded
// batch and overwhelming the runtime or SQLite parameter conversion.
let max_concurrent = self.config.max_concurrent.clamp(1, 256); let max_concurrent = self.config.max_concurrent.clamp(1, 256);
let max_delivery = max_concurrent.clamp(1, 16);
let mut runs = JoinSet::new();
let mut deliveries = JoinSet::new();
tracing::info!( tracing::info!(
poll_interval_secs = self.config.poll_interval_secs, poll_interval_secs = self.config.poll_interval_secs,
@ -147,375 +106,340 @@ impl Scheduler {
); );
loop { loop {
interval.tick().await; tokio::select! {
_ = interval.tick() => {}
Some(result) = runs.join_next(), if !runs.is_empty() => {
if let Err(error) = result {
tracing::error!(error = %error, "scheduled run task panicked");
}
}
Some(result) = deliveries.join_next(), if !deliveries.is_empty() => {
if let Err(error) = result {
tracing::error!(error = %error, "scheduled delivery task panicked");
}
}
}
while let Some(result) = runs.try_join_next() {
if let Err(error) = result {
tracing::error!(error = %error, "scheduled run task panicked");
}
}
while let Some(result) = deliveries.try_join_next() {
if let Err(error) = result {
tracing::error!(error = %error, "scheduled delivery task panicked");
}
}
if !self.admission.is_accepting() { if !self.admission.is_accepting() {
continue; continue;
} }
let now = now_ms(); let now = now_ms();
let delivery_slots = max_delivery.saturating_sub(deliveries.len());
if delivery_slots > 0 {
let lease_until = now.saturating_add(180_000);
let delivery_owner = format!("{}:delivery:{}", self.owner, uuid::Uuid::new_v4());
match self
.storage
.claim_scheduled_deliveries(now, lease_until, &delivery_owner, delivery_slots)
.await
{
Ok(claimed) => {
for run in claimed {
let scheduler = self.clone();
deliveries.spawn(async move {
scheduler.deliver_claimed_run(run).await;
});
}
}
Err(error) => {
tracing::error!(error = %error, "scheduler: failed to claim deliveries");
}
}
}
let run_slots = max_concurrent.saturating_sub(runs.len());
if run_slots == 0 {
continue;
}
let lease_ms = self let lease_ms = self
.config .config
.execution_timeout_secs .execution_timeout_secs
.saturating_add(150) .saturating_add(150)
.saturating_mul(1000) .saturating_mul(1000)
.min(i64::MAX as u64) as i64; .min(i64::MAX as u64) as i64;
let lease_until = now.saturating_add(lease_ms); let run_owner = format!("{}:run:{}", self.owner, uuid::Uuid::new_v4());
let jobs = match self match self
.storage .storage
.claim_due_scheduled_jobs(now, lease_until, &self.owner, max_concurrent) .claim_due_scheduled_runs(now, now.saturating_add(lease_ms), &run_owner, run_slots)
.await .await
{ {
Ok(jobs) => jobs, Ok(claimed) => {
Err(error) => { for run in claimed {
tracing::error!(error = %error, "scheduler: failed to claim due jobs"); let scheduler = self.clone();
continue; runs.spawn(async move {
scheduler.execute_claimed_run(run).await;
});
}
}
Err(error) => {
tracing::error!(error = %error, "scheduler: failed to claim due runs");
} }
};
if jobs.is_empty() {
continue;
} }
tracing::info!(count = jobs.len(), "scheduler: claimed due jobs");
stream::iter(jobs)
.for_each_concurrent(max_concurrent, |job| {
let scheduler = self.clone();
async move { scheduler.execute_claimed_job(job).await }
})
.await;
} }
} }
async fn execute_claimed_job(self: Arc<Self>, job: ScheduledJob) { async fn execute_claimed_run(self: Arc<Self>, claimed: ClaimedScheduledRun) {
let Some(_activity) = self.admission.try_enter() else {
if let Err(error) = self
.storage
.release_scheduled_job_lease(&job.id, &self.owner)
.await
{
tracing::error!(job_id = %job.id, error = %error, "scheduler: failed to release job claimed during reload drain");
}
return;
};
let start = Instant::now(); let start = Instant::now();
let started_at = now_ms(); let job = &claimed.job;
tracing::info!(job_id = %job.id, job_name = %job.name, "scheduler: executing claimed job"); let (completion, agent_execution) = if let Some(_activity) = self.admission.try_enter() {
match self.session_manager.agent_coordinator() {
let managed = job.delivery_policy != DeliveryPolicy::Direct; Some(coordinator) => match coordinator
let execution = async { .execute_scheduled(
if managed { claimed.run_id,
self.session_manager &claimed.owner,
.handle_managed_scheduled_message(
&job.prompt,
&job.id, &job.id,
&job.name, &job.name,
job.job_kind == JobKind::Monitor, job.agent_id.as_deref(),
)
.await
.map(HandleResult::AgentResponse)
} else {
self.session_manager
.handle_cron_message(
&job.channel,
&job.chat_id,
&job.prompt, &job.prompt,
&job.id, self.config.execution_timeout_secs.max(1),
&job.name,
) )
.await .await
} {
}; Ok(execution) => {
let result = time::timeout( if execution.status == ScheduledRunStatus::Completed
time::Duration::from_secs(self.config.execution_timeout_secs.max(1)), && let Some(outcome) = execution.outcome.clone()
execution, {
) (
.await; ScheduledRunCompletion {
let finished_at = now_ms(); status: ScheduledRunStatus::Completed,
let duration_ms = start.elapsed().as_millis() as i64; outcome: outcome.kind,
message: outcome.message,
let (mut status, output, error, result_kind, mut delivery_status, mut delivery_error) = diagnostic: execution.error.clone(),
match result { duration_ms: start.elapsed().as_millis() as i64,
Ok(Ok( },
HandleResult::AgentResponse(output) | HandleResult::CommandOutput(output), Some(execution),
)) => { )
let output = if output.len() > 8000 {
format!(
"{}...[truncated]",
&output[..output.ceil_char_boundary(8000)]
)
} else {
output
};
if !managed {
(
"ok".into(),
Some(output),
None,
None,
Some("direct".into()),
None,
)
} else {
let disposition = parse_scheduled_disposition(&output);
let (kind, content, alert) = match &disposition {
ScheduledDisposition::Content(value) => {
("content", Some(value.as_str()), true)
}
ScheduledDisposition::Quiet(_) => ("quiet", None, false),
ScheduledDisposition::ReportedFailure(value) => {
("reported_failure", Some(value.as_str()), true)
}
ScheduledDisposition::Refused(value) => {
("refused", Some(value.as_str()), true)
}
};
let should_deliver = match job.delivery_policy {
DeliveryPolicy::Always => true,
DeliveryPolicy::OnAlert => alert,
DeliveryPolicy::Never => false,
DeliveryPolicy::Direct => false,
};
if should_deliver {
let message = content.unwrap_or("巡检完成,未发现需要关注的问题。");
match self
.session_manager
.deliver_scheduled_message(
&job.channel,
&job.chat_id,
&job.id,
&job.name,
message,
)
.await
{
Ok(()) => (
"ok".into(),
Some(output),
None,
Some(kind.into()),
Some("delivered".into()),
None,
),
Err(delivery_error) => (
"delivery_error".into(),
Some(output),
None,
Some(kind.into()),
Some("failed".into()),
Some(delivery_error),
),
}
} else { } else {
let delivery = if job.delivery_policy == DeliveryPolicy::Never { let diagnostic = execution.error.clone().unwrap_or_else(|| {
"skipped" "scheduled Agent returned ordinary text without calling complete_scheduled_run"
} else { .to_string()
"suppressed" });
let status = match execution.status {
ScheduledRunStatus::TimedOut => ScheduledRunStatus::TimedOut,
ScheduledRunStatus::Interrupted | ScheduledRunStatus::Cancelled => {
ScheduledRunStatus::Interrupted
}
_ => ScheduledRunStatus::Failed,
}; };
( (
"ok".into(), ScheduledRunCompletion {
Some(output), status,
None, outcome: ScheduledOutcomeKind::Failed,
Some(kind.into()), message: if status == ScheduledRunStatus::TimedOut {
Some(delivery.into()), format!("定时任务「{}」执行超时。", job.name)
None, } else if status == ScheduledRunStatus::Interrupted {
format!("定时任务「{}」在系统关停时被中断。", job.name)
} else {
format!(
"定时任务「{}」未可靠完成Agent 未提交结构化运行结果。",
job.name
)
},
diagnostic: Some(diagnostic),
duration_ms: start.elapsed().as_millis() as i64,
},
Some(execution),
) )
} }
} }
Err(error) => (
ScheduledRunCompletion {
status: ScheduledRunStatus::Failed,
outcome: ScheduledOutcomeKind::Failed,
message: job.agent_id.as_deref().map_or_else(
|| format!("定时任务「{}」未能启动 Root Agent 执行。", job.name),
|agent_id| {
format!(
"定时任务「{}」无法使用 Agent「{}」执行。请恢复该 Agent 定义,或更新任务的 agent_id。",
job.name, agent_id
)
},
),
diagnostic: Some(error.to_string()),
duration_ms: start.elapsed().as_millis() as i64,
},
None,
),
},
None => (
ScheduledRunCompletion {
status: ScheduledRunStatus::Failed,
outcome: ScheduledOutcomeKind::Failed,
message: format!("定时任务「{}」未能启动执行。", job.name),
diagnostic: Some("AgentCoordinator is unavailable".to_string()),
duration_ms: start.elapsed().as_millis() as i64,
},
None,
),
}
} else {
(
ScheduledRunCompletion {
status: ScheduledRunStatus::Interrupted,
outcome: ScheduledOutcomeKind::Failed,
message: format!("定时任务「{}」因 Gateway 重载而中断。", job.name),
diagnostic: Some("runtime admission closed".to_string()),
duration_ms: start.elapsed().as_millis() as i64,
},
None,
)
};
let finished_at = now_ms();
let mut commit_attempt = 0_u64;
let commit = loop {
let result = match agent_execution.as_ref() {
Some(execution) => {
self.storage
.finish_scheduled_run_with_agent(
claimed.run_id,
&claimed.owner,
&completion,
&execution.agent_run_id,
&execution.agent_run_id,
execution.runtime_generation,
&execution.agent_terminal,
finished_at,
)
.await
}
None => {
self.storage
.finish_scheduled_run(
claimed.run_id,
&claimed.owner,
&completion,
finished_at,
)
.await
} }
Ok(Ok(HandleResult::AgentProcessing)) => (
"error".to_string(),
None,
Some("cron execution returned asynchronous processing".to_string()),
None,
None,
None,
),
Ok(Err(error)) => (
"error".to_string(),
None,
Some(error.to_string()),
None,
None,
None,
),
Err(_) => (
"timeout".to_string(),
None,
Some(format!(
"execution exceeded {} seconds",
self.config.execution_timeout_secs.max(1)
)),
None,
None,
None,
),
}; };
match result {
if managed Err(error) if error.is_transient() && commit_attempt < 2 => {
&& delivery_status.is_none() commit_attempt += 1;
&& job.delivery_policy != DeliveryPolicy::Never tracing::warn!(
&& let Some(message) = error.as_deref() job_id = %job.id,
{ run_id = claimed.run_id,
let notice = format!("定时任务「{}」执行失败:{}", job.name, message); attempt = commit_attempt + 1,
match self error = %error,
.session_manager "scheduler: retrying transient run completion commit"
.deliver_scheduled_message(&job.channel, &job.chat_id, &job.id, &job.name, &notice) );
.await time::sleep(time::Duration::from_millis(50 * commit_attempt)).await;
{
Ok(()) => delivery_status = Some("delivered".into()),
Err(error) => {
status = "delivery_error".into();
delivery_status = Some("failed".into());
delivery_error = Some(error);
} }
result => break result,
} }
};
match commit {
Ok(true) => tracing::info!(
job_id = %job.id,
run_id = claimed.run_id,
status = completion.status.as_str(),
outcome = completion.outcome.as_str(),
duration_ms = completion.duration_ms,
"scheduler: run completed"
),
Ok(false) => tracing::warn!(
job_id = %job.id,
run_id = claimed.run_id,
"scheduler: late run result discarded"
),
Err(error) => tracing::error!(
job_id = %job.id,
run_id = claimed.run_id,
error = %error,
"scheduler: failed to commit run completion"
),
} }
}
let (next_run_at, disable, delete) = match &job.schedule { async fn deliver_claimed_run(self: Arc<Self>, run: JobRun) {
Schedule::At { .. } => (None, !job.delete_after_run, job.delete_after_run), let Some(delivery_owner) = run.delivery_lease_owner.clone() else {
Schedule::Every { .. } | Schedule::Cron { .. } => { tracing::error!(
match next_run_for_schedule(&job.schedule, finished_at) { run_id = run.id,
Some(next) => (Some(next), false, false), "scheduler: claimed delivery has no lease owner"
None => (None, true, false), );
} return;
};
let result = self
.session_manager
.deliver_scheduled_run(&run, &delivery_owner)
.await;
let (delivered, permanent, error) = match result {
Ok(()) => (true, false, None),
Err(ScheduledDeliveryError::Transient(error)) => {
(false, false, Some(sanitize_error(&error)))
}
Err(ScheduledDeliveryError::Permanent(error)) => {
(false, true, Some(sanitize_error(&error)))
} }
}; };
let run = JobRun { if let Err(commit_error) = self
id: 0,
job_id: job.id.clone(),
started_at,
finished_at,
status,
output,
error,
duration_ms,
result_kind,
delivery_status,
delivery_error,
};
if let Err(error) = self
.storage .storage
.complete_scheduled_job(&run, &self.owner, next_run_at, disable, delete) .complete_scheduled_delivery(
run.id,
&delivery_owner,
delivered,
permanent,
error.as_deref(),
now_ms(),
)
.await .await
{ {
tracing::error!(job_id = %job.id, error = %error, "scheduler: failed to commit job completion"); tracing::error!(
let _ = self run_id = run.id,
.storage error = %commit_error,
.release_scheduled_job_lease(&job.id, &self.owner) "scheduler: failed to commit delivery receipt"
.await; );
return;
} }
tracing::info!(
job_id = %job.id,
status = %run.status,
duration_ms,
"scheduler: job completed"
);
} }
} }
fn sanitize_error(error: &str) -> String {
error.chars().take(1_024).collect()
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
#[test] #[test]
fn test_next_run_at_schedule() { fn next_run_for_every_uses_claim_time() {
let now = 1000000; assert_eq!(
let next = next_run_for_schedule(&Schedule::At { at: 2000000 }, now); next_run_for_schedule(&Schedule::Every { every_ms: 5_000 }, 1_000),
assert_eq!(next, Some(2000000)); Some(6_000)
);
} }
#[test] #[test]
fn test_next_run_every_schedule() { fn next_run_for_at_keeps_absolute_timestamp() {
let now = 1000000; assert_eq!(
let next = next_run_for_schedule(&Schedule::Every { every_ms: 5000 }, now); next_run_for_schedule(&Schedule::At { at: 2_000 }, 1_000),
assert_eq!(next, Some(1005000)); Some(2_000)
);
} }
#[test] #[test]
fn test_next_run_cron_every_minute() { fn cron_timezone_uses_from_argument() {
let expr = "0 * * * * *".to_string();
let schedule = Schedule::Cron { expr, tz: None };
let now = 1000000;
let next = next_run_for_schedule(&schedule, now);
assert!(next.is_some());
assert!(next.unwrap() > now);
}
#[test]
fn test_next_run_cron_every_day_at_9am() {
let expr = "0 0 9 * * *".to_string();
let schedule = Schedule::Cron { expr, tz: None };
let now = 1000000;
let next = next_run_for_schedule(&schedule, now);
assert!(next.is_some());
let next_ms = next.unwrap();
assert!(next_ms > now);
}
#[test]
fn test_next_run_cron_uses_from_argument() {
let expr = "0 * * * * *".to_string();
let schedule = Schedule::Cron { expr, tz: None };
let from = chrono::DateTime::parse_from_rfc3339("2026-06-16T12:34:20Z")
.unwrap()
.timestamp_millis();
let next = next_run_for_schedule(&schedule, from).unwrap();
let expected = chrono::DateTime::parse_from_rfc3339("2026-06-16T12:35:00Z")
.unwrap()
.timestamp_millis();
assert_eq!(next, expected);
}
#[test]
fn scheduled_disposition_is_fail_safe() {
assert!(matches!(
parse_scheduled_disposition("NO_REPLY"),
ScheduledDisposition::Quiet(_)
));
assert!(matches!(
parse_scheduled_disposition("NO_REPLY[INFO]: healthy"),
ScheduledDisposition::Quiet(_)
));
assert!(matches!(
parse_scheduled_disposition("NO_REPLY[FAIL]: timeout"),
ScheduledDisposition::ReportedFailure(_)
));
assert!(matches!(
parse_scheduled_disposition("NO_REPLY[REFUSE]: denied"),
ScheduledDisposition::Refused(_)
));
assert!(matches!(
parse_scheduled_disposition("text mentioning NO_REPLY"),
ScheduledDisposition::Content(_)
));
assert!(matches!(
parse_scheduled_disposition("NO_REPLY[INFO] but this is content"),
ScheduledDisposition::Content(_)
));
assert!(matches!(
parse_scheduled_disposition(""),
ScheduledDisposition::ReportedFailure(_)
));
}
#[test]
fn test_next_run_cron_timezone_uses_from_argument() {
let expr = "0 0 9 * * *".to_string();
let schedule = Schedule::Cron { let schedule = Schedule::Cron {
expr, expr: "0 0 9 * * *".to_string(),
tz: Some("Asia/Shanghai".to_string()), tz: Some("Asia/Shanghai".to_string()),
}; };
let from = chrono::DateTime::parse_from_rfc3339("2026-06-16T00:30:00Z") let from = chrono::DateTime::parse_from_rfc3339("2026-06-16T00:30:00Z")
.unwrap() .unwrap()
.timestamp_millis(); .timestamp_millis();
let next = next_run_for_schedule(&schedule, from).unwrap();
let expected = chrono::DateTime::parse_from_rfc3339("2026-06-16T01:00:00Z") let expected = chrono::DateTime::parse_from_rfc3339("2026-06-16T01:00:00Z")
.unwrap() .unwrap()
.timestamp_millis(); .timestamp_millis();
assert_eq!(next, expected); assert_eq!(next_run_for_schedule(&schedule, from), Some(expected));
} }
} }

View File

@ -4,7 +4,9 @@ use crate::bus::{ChatMessage, MediaItem, MessageSource, OutboundMessage, SourceK
use crate::session::UnifiedSessionId; use crate::session::UnifiedSessionId;
use crate::tools::{OutboundDelivery, OutboundMessenger}; use crate::tools::{OutboundDelivery, OutboundMessenger};
use super::persistence::{append_active_turn_message, append_persisted_messages}; use super::persistence::{
append_active_turn_message, append_persisted_message_if_absent, append_persisted_messages,
};
use super::session::{ use super::session::{
CURRENT_SOURCE_SESSION, CURRENT_TURN_DELIVERIES, CURRENT_TURN_ID, PendingTurnDelivery, CURRENT_SOURCE_SESSION, CURRENT_TURN_DELIVERIES, CURRENT_TURN_ID, PendingTurnDelivery,
SessionManager, SessionManager,
@ -118,37 +120,108 @@ impl OutboundMessenger for SessionManager {
} }
impl SessionManager { impl SessionManager {
pub async fn deliver_scheduled_message( pub async fn deliver_scheduled_run(
&self, &self,
channel: &str, run: &crate::storage::JobRun,
chat_id: &str, delivery_owner: &str,
job_id: &str, ) -> Result<(), ScheduledDeliveryError> {
job_name: &str, let content = run
content: &str, .message
) -> Result<(), String> { .as_deref()
<Self as OutboundMessenger>::send_message( .unwrap_or("定时任务已结束,但没有生成可投递的结果。请在任务运行记录中查看诊断信息。");
self, let target_sid = if let Some(session_id) = run.target_session_id.as_deref() {
channel, UnifiedSessionId::parse(session_id).ok_or_else(|| {
chat_id, ScheduledDeliveryError::Permanent("stored target session is invalid".to_string())
None, })?
content, } else {
MessageSource { let resolved = self
kind: SourceKind::ExternalTrigger, .resolve_dialog_id(&run.target_channel, &run.target_chat_id)
from_channel: Some("scheduler".to_string()), .await
from_session: Some(format!("cron:{job_id}")), .map_err(|error| ScheduledDeliveryError::Transient(error.to_string()))?;
from_user_id: None, let fixed = self
system_name: Some(job_name.to_string()), .storage
task_id: Some(job_id.to_string()), .set_scheduled_delivery_target_session(
from_run_id: None, run.id,
from_agent_id: None, delivery_owner,
}, &resolved.to_string(),
Vec::new(), chrono::Utc::now().timestamp_millis(),
) )
.await .await
.map(|_| ()) .map_err(|error| ScheduledDeliveryError::Transient(error.to_string()))?
.ok_or_else(|| {
ScheduledDeliveryError::Transient(
"scheduled delivery lost its claim before fixing target session"
.to_string(),
)
})?;
UnifiedSessionId::parse(&fixed).ok_or_else(|| {
ScheduledDeliveryError::Permanent("fixed target session is invalid".to_string())
})?
};
if target_sid.channel != run.target_channel || target_sid.chat_id != run.target_chat_id {
return Err(ScheduledDeliveryError::Permanent(
"fixed target session does not belong to the scheduled destination".to_string(),
));
}
let session = self
.get_or_activate_session(&target_sid)
.await
.map_err(|error| ScheduledDeliveryError::Transient(error.to_string()))?;
let source = MessageSource {
kind: SourceKind::ExternalTrigger,
from_channel: Some("scheduler".to_string()),
from_session: Some(format!("scheduled-run:{}", run.id)),
from_user_id: None,
system_name: Some("scheduled task".to_string()),
task_id: Some(run.job_id.clone()),
from_run_id: run.agent_run_id.clone(),
from_agent_id: run.agent_id.clone(),
};
let mut message = outbound_history_message(content, source, &[]);
message.id = format!("scheduled:{}", run.id);
let message_id = message.id.clone();
append_persisted_message_if_absent(&session, message)
.await
.map_err(|error| ScheduledDeliveryError::Transient(error.to_string()))?;
let metadata = HashMap::from([
("_session_id".to_string(), target_sid.to_string()),
("_message_id".to_string(), message_id),
("scheduled_delivery_id".to_string(), run.id.to_string()),
]);
self.bus
.deliver_outbound(OutboundMessage {
channel: run.target_channel.clone(),
chat_id: run.target_chat_id.clone(),
content: content.to_string(),
reply_to: None,
media: Vec::new(),
metadata,
delivery: None,
})
.await
.map_err(|error| match error {
crate::bus::BusError::Closed
| crate::bus::BusError::DeliveryTimedOut
| crate::bus::BusError::DeliveryTransient(_) => {
ScheduledDeliveryError::Transient(error.to_string())
}
crate::bus::BusError::DeliveryPermanent(summary) => {
ScheduledDeliveryError::Permanent(summary)
}
})?;
Ok(())
} }
} }
#[derive(Debug, thiserror::Error)]
pub enum ScheduledDeliveryError {
#[error("transient scheduled delivery failure: {0}")]
Transient(String),
#[error("permanent scheduled delivery failure: {0}")]
Permanent(String),
}
fn outbound_history_message( fn outbound_history_message(
content: impl Into<String>, content: impl Into<String>,
source: MessageSource, source: MessageSource,

View File

@ -14,6 +14,7 @@ pub mod turn;
pub use commands::SessionCommand; pub use commands::SessionCommand;
pub use error::SessionError; pub use error::SessionError;
pub use events::{DialogInfo, SessionEvent}; pub use events::{DialogInfo, SessionEvent};
pub use messenger::ScheduledDeliveryError;
pub use session::{ pub use session::{
AgentCatalogPreparation, SLASH_COMMANDS, Session, SessionManager, SessionManagerServices, AgentCatalogPreparation, SLASH_COMMANDS, Session, SessionManager, SessionManagerServices,
SlashCommand, SlashCommand,

View File

@ -101,6 +101,41 @@ pub(super) async fn append_persisted_messages_with_meta(
append_persisted_messages_inner(session, messages, VersionPolicy::Advance, None, None).await append_persisted_messages_inner(session, messages, VersionPolicy::Advance, None, None).await
} }
pub(super) async fn append_persisted_message_if_absent(
session: &Arc<Mutex<Session>>,
message: ChatMessage,
) -> Result<bool, StorageError> {
let persistence_lock = { session.lock().await.persistence_lock.clone() };
let _persistence_guard = persistence_lock.lock().await;
let message_id = message.id.clone();
let snapshot = {
let guard = session.lock().await;
if guard.contains_message_id(&message_id) {
return Ok(false);
}
guard.prepare_message_persist_snapshot(&message)
};
let Some((storage, session_id, persisted, meta)) = snapshot else {
return Ok(false);
};
let inserted = storage
.persist_message_if_absent(&session_id, &persisted, &meta)
.await?;
if !inserted {
return Ok(false);
}
let mut guard = session.lock().await;
if guard.contains_message_id(&message_id) {
return Ok(true);
}
if !guard.apply_prepared_message_in_memory(message, persisted.seq, persisted.created_at, true) {
return Err(StorageError::Conflict(format!(
"session changed while committing idempotent message {message_id}"
)));
}
Ok(true)
}
pub(super) async fn append_persisted_turn_messages( pub(super) async fn append_persisted_turn_messages(
session: &Arc<Mutex<Session>>, session: &Arc<Mutex<Session>>,
messages: Vec<ChatMessage>, messages: Vec<ChatMessage>,
@ -247,7 +282,7 @@ mod tests {
} }
#[tokio::test] #[tokio::test]
async fn active_turn_side_effect_does_not_invalidate_its_session_version() { async fn side_effect_messages_apply_the_expected_session_version_once() {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
let storage = Arc::new( let storage = Arc::new(
crate::storage::Storage::new(&dir.path().join("memory.db")) crate::storage::Storage::new(&dir.path().join("memory.db"))
@ -255,7 +290,7 @@ mod tests {
.unwrap(), .unwrap(),
); );
let memory_manager = Arc::new(MemoryManager::new( let memory_manager = Arc::new(MemoryManager::new(
storage, storage.clone(),
"test".to_string(), "test".to_string(),
"test".to_string(), "test".to_string(),
)); ));
@ -276,12 +311,32 @@ mod tests {
price_input_per_million: None, price_input_per_million: None,
price_output_per_million: None, price_output_per_million: None,
}; };
let unified_id = crate::session::UnifiedSessionId::new("cli_chat", "chat", "dialog");
let now = chrono::Utc::now().timestamp_millis();
storage
.upsert_session(&crate::storage::session::SessionMeta {
id: unified_id.to_string(),
channel: "cli_chat".to_string(),
chat_id: "chat".to_string(),
dialog_id: "dialog".to_string(),
title: "test".to_string(),
created_at: now,
last_active_at: now,
message_count: 0,
routing_info: None,
archived_at: None,
deleted_at: None,
last_consolidated_at: None,
last_compressed_message_at: None,
})
.await
.unwrap();
let session = Arc::new(Mutex::new( let session = Arc::new(Mutex::new(
Session::new( Session::new(
crate::session::UnifiedSessionId::new("cli_chat", "chat", "dialog"), unified_id,
config, config,
Arc::new(ToolRegistry::new()), Arc::new(ToolRegistry::new()),
None, Some(storage.clone()),
String::new(), String::new(),
"test".to_string(), "test".to_string(),
super::super::session::SessionContextServices { super::super::session::SessionContextServices {
@ -320,5 +375,37 @@ mod tests {
let guard = session.lock().await; let guard = session.lock().await;
assert_eq!(guard.state_version_for_test(), base_version + 1); assert_eq!(guard.state_version_for_test(), base_version + 1);
assert_eq!(guard.get_history().len(), 2); assert_eq!(guard.get_history().len(), 2);
drop(guard);
let idempotent_base_version = session.lock().await.state_version_for_test();
let mut scheduled = ChatMessage::assistant("scheduled result");
scheduled.id = "scheduled:42".to_string();
assert!(
append_persisted_message_if_absent(&session, scheduled.clone())
.await
.unwrap()
);
assert!(
!append_persisted_message_if_absent(&session, scheduled)
.await
.unwrap()
);
let guard = session.lock().await;
assert_eq!(guard.state_version_for_test(), idempotent_base_version + 1);
assert_eq!(
guard
.get_history()
.iter()
.filter(|message| message.id == "scheduled:42")
.count(),
1
);
drop(guard);
let persisted: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM messages WHERE id = ?")
.bind("scheduled:42")
.fetch_one(storage.pool())
.await
.unwrap();
assert_eq!(persisted, 1);
} }
} }

View File

@ -1083,59 +1083,101 @@ impl Session {
persist: bool, persist: bool,
advance_state_version: bool, advance_state_version: bool,
) -> Option<MessagePersistSnapshot> { ) -> Option<MessagePersistSnapshot> {
let persist_snapshot = persist
.then(|| self.prepare_message_persist_snapshot(&message))
.flatten();
let (seq, now) = persist_snapshot
.as_ref()
.map(|(_, _, message, _)| (message.seq, message.created_at))
.unwrap_or_else(|| (self.seq_counter, chrono::Utc::now().timestamp_millis()));
let applied =
self.apply_prepared_message_in_memory(message, seq, now, advance_state_version);
debug_assert!(applied, "fresh in-memory message snapshot must apply");
persist_snapshot
}
pub(super) fn prepare_message_persist_snapshot(
&self,
message: &ChatMessage,
) -> Option<MessagePersistSnapshot> {
let storage = self.storage.clone()?;
let is_user = message.role == "user"; let is_user = message.role == "user";
let counts_as_user_input = let counts_as_user_input =
is_user && message.client_visibility == crate::bus::ClientVisibility::Visible; is_user && message.client_visibility == crate::bus::ClientVisibility::Visible;
let now = chrono::Utc::now().timestamp_millis(); let now = chrono::Utc::now().timestamp_millis();
// Assign seq
let seq = self.seq_counter; let seq = self.seq_counter;
self.seq_counter += 1; let msg_meta = crate::storage::message::MessageMeta {
id: message.id.clone(),
let persist_snapshot = if persist { session_id: self.id.to_string(),
self.storage.clone().map(|storage| { seq,
let msg_meta = crate::storage::message::MessageMeta { role: message.role.clone(),
id: message.id.clone(), content: message.content.clone(),
session_id: self.id.to_string(), reasoning_content: message.reasoning_content.clone(),
seq, provider_state: message
role: message.role.clone(), .provider_state
content: message.content.clone(), .as_ref()
reasoning_content: message.reasoning_content.clone(), .and_then(|state| serde_json::to_string(state).ok()),
provider_state: message turn_id: message.turn_id.clone(),
.provider_state iteration: message.iteration.map(i64::from),
.as_ref() completion_status: message.completion_status,
.and_then(|state| serde_json::to_string(state).ok()), client_visibility: message.client_visibility,
turn_id: message.turn_id.clone(), turn_origin: message.turn_origin,
iteration: message.iteration.map(i64::from), media_refs: if message.media_refs.is_empty() {
completion_status: message.completion_status, None
client_visibility: message.client_visibility, } else {
turn_origin: message.turn_origin, Some(serde_json::to_string(&message.media_refs).unwrap_or_default())
media_refs: if message.media_refs.is_empty() { },
None tool_call_id: message.tool_call_id.clone(),
} else { tool_name: message.tool_name.clone(),
Some(serde_json::to_string(&message.media_refs).unwrap_or_default()) tool_calls: message
}, .tool_calls
tool_call_id: message.tool_call_id.clone(), .as_ref()
tool_name: message.tool_name.clone(), .and_then(|tc| serde_json::to_string(tc).ok()),
tool_calls: message source: message
.tool_calls .source
.as_ref() .as_ref()
.and_then(|tc| serde_json::to_string(tc).ok()), .map(|s| serde_json::to_string(s).unwrap_or_default()),
source: message created_at: now,
.source
.as_ref()
.map(|s| serde_json::to_string(s).unwrap_or_default()),
created_at: now,
};
(storage, self.id.to_string(), msg_meta)
})
} else {
None
}; };
let session_id = self.id.to_string();
let session_meta = crate::storage::session::SessionMeta {
id: session_id.clone(),
channel: self.id.channel.clone(),
chat_id: self.id.chat_id.clone(),
dialog_id: self.id.dialog_id.clone(),
title: self.title.clone(),
created_at: self.created_at,
last_active_at: now,
message_count: self.message_count + i64::from(counts_as_user_input),
routing_info: if self.routing_info.is_empty() {
None
} else {
Some(self.routing_info.clone())
},
archived_at: self.archived_at,
deleted_at: None,
last_consolidated_at: self.last_consolidated_at,
last_compressed_message_at: self.last_compressed_message_at,
};
Some((storage, session_id, msg_meta, session_meta))
}
// Update in-memory state pub(super) fn apply_prepared_message_in_memory(
&mut self,
message: ChatMessage,
seq: i64,
now: i64,
advance_state_version: bool,
) -> bool {
if self.seq_counter != seq || self.contains_message_id(&message.id) {
return false;
}
let is_user = message.role == "user";
let counts_as_user_input =
is_user && message.client_visibility == crate::bus::ClientVisibility::Visible;
self.message_seqs.insert(message.id.clone(), seq); self.message_seqs.insert(message.id.clone(), seq);
self.messages.push(message); self.messages.push(message);
self.seq_counter += 1;
self.total_message_count += 1; self.total_message_count += 1;
if counts_as_user_input { if counts_as_user_input {
self.message_count += 1; self.message_count += 1;
@ -1144,29 +1186,11 @@ impl Session {
if advance_state_version { if advance_state_version {
self.state_version = self.state_version.wrapping_add(1); self.state_version = self.state_version.wrapping_add(1);
} }
true
}
persist_snapshot.map(|(storage, session_id, msg_meta)| { pub(super) fn contains_message_id(&self, message_id: &str) -> bool {
let session_meta = crate::storage::session::SessionMeta { self.message_seqs.contains_key(message_id)
id: session_id.clone(),
channel: self.id.channel.clone(),
chat_id: self.id.chat_id.clone(),
dialog_id: self.id.dialog_id.clone(),
title: self.title.clone(),
created_at: self.created_at,
last_active_at: self.last_active_at,
message_count: self.message_count,
routing_info: if self.routing_info.is_empty() {
None
} else {
Some(self.routing_info.clone())
},
archived_at: self.archived_at,
deleted_at: None,
last_consolidated_at: self.last_consolidated_at,
last_compressed_message_at: self.last_compressed_message_at,
};
(storage, session_id, msg_meta, session_meta)
})
} }
/// Roll back messages that were appended in memory but whose atomic /// Roll back messages that were appended in memory but whose atomic
@ -1905,7 +1929,7 @@ pub struct SessionManager {
provider_config: LLMProviderConfig, provider_config: LLMProviderConfig,
tools: Arc<ToolRegistry>, tools: Arc<ToolRegistry>,
skills_loader: Arc<SkillsLoader>, skills_loader: Arc<SkillsLoader>,
storage: Arc<Storage>, pub(super) storage: Arc<Storage>,
pub(super) bus: Arc<MessageBus>, pub(super) bus: Arc<MessageBus>,
memory_manager: Arc<crate::memory::MemoryManager>, memory_manager: Arc<crate::memory::MemoryManager>,
work_manager: Arc<crate::work::WorkManager>, work_manager: Arc<crate::work::WorkManager>,
@ -2276,46 +2300,6 @@ impl SessionManager {
self.work_manager.clone() self.work_manager.clone()
} }
/// 为定时任务创建一个无 session 绑定的 AgentLoop
pub fn create_cron_agent(&self) -> Result<AgentLoop, AgentError> {
let tools = self.tools.without(&["reload_config"]);
let provider = create_provider(self.provider_config.clone())
.map_err(|e| AgentError::Other(format!("failed to create cron provider: {}", e)))?;
Ok(AgentLoop::with_provider_and_tools(
Arc::from(provider),
tools,
self.provider_config.max_tool_iterations,
self.provider_config.model_id.clone(),
self.provider_config.workspace_dir.clone(),
self.provider_config.input_types.clone(),
)
.with_context_window(self.provider_config.token_limit))
}
fn create_managed_scheduled_agent(&self) -> Result<(AgentLoop, Arc<ToolRegistry>), AgentError> {
let tools = self.tools.without(&[
"send_message",
"cron_add",
"cron_update",
"cron_remove",
"cron_enable",
"cron_disable",
"reload_config",
]);
let provider = create_provider(self.provider_config.clone())
.map_err(|e| AgentError::Other(format!("failed to create scheduled provider: {e}")))?;
let agent = AgentLoop::with_provider_and_tools(
Arc::from(provider),
tools.clone(),
self.provider_config.max_tool_iterations,
self.provider_config.model_id.clone(),
self.provider_config.workspace_dir.clone(),
self.provider_config.input_types.clone(),
)
.with_context_window(self.provider_config.token_limit);
Ok((agent, tools))
}
/// 获取所有可用的斜杠命令 /// 获取所有可用的斜杠命令
pub fn get_slash_commands(&self) -> &[SlashCommand] { pub fn get_slash_commands(&self) -> &[SlashCommand] {
SLASH_COMMANDS SLASH_COMMANDS
@ -5098,103 +5082,6 @@ impl SessionManager {
} }
impl SessionManager { impl SessionManager {
///
/// Runs in a stateless manner: no session creation, no history persistence.
/// The cron system prompt instructs the LLM to deliver results via the
/// `send_message` tool, which handles both delivery and history writing
/// on the target session.
pub async fn handle_cron_message(
&self,
channel: &str,
chat_id: &str,
prompt: &str,
job_id: &str,
job_name: &str,
) -> Result<HandleResult, AgentError> {
let skills_prompt = self.skills_loader.build_skills_prompt();
let base_prompt = build_system_prompt(
&self.provider_config.workspace_dir,
&self.provider_config.model_id,
&self.tools,
);
let cron_context = format!(
"## 定时任务执行\n\n\
{job_name}({job_id})\n\
: {channel}:{chat_id}\n\n\
:\n\
- \n\
- 使 send_message \n\
- send_message : target_chat_id=\"{channel}:{chat_id}\", content=\"消息内容\"\n\
- send_message \n\
- "
);
let full_system_prompt =
format!("{}\n\n{}\n\n{}", base_prompt, skills_prompt, cron_context);
let history = vec![
ChatMessage::system(full_system_prompt),
ChatMessage::user(prompt),
];
let agent = self.create_cron_agent()?;
let source_session = format!("cron:{}", job_name);
let result = CURRENT_SOURCE_SESSION
.scope(Some(source_session.clone()), async {
agent
.process_with_context(
history,
ToolExecutionContext::for_session(source_session),
)
.await
})
.await
.inspect_err(|e| {
tracing::error!(error = %e, job_id = %job_id, "Cron agent processing error");
})?;
Ok(HandleResult::AgentResponse(result.final_response.content))
}
/// Execute a scheduler-managed task. The agent returns a result but cannot
/// deliver it itself; Scheduler applies the configured delivery policy.
pub async fn handle_managed_scheduled_message(
&self,
prompt: &str,
job_id: &str,
job_name: &str,
monitor: bool,
) -> Result<String, AgentError> {
let (agent, tools) = self.create_managed_scheduled_agent()?;
let base_prompt = build_system_prompt(
&self.provider_config.workspace_dir,
&self.provider_config.model_id,
&tools,
);
let skills_prompt = self.skills_loader.build_skills_prompt();
let result_contract = if monitor {
"这是无人值守巡检。完成必要检查后:一切正常且无需用户关注时,只返回 NO_REPLY[INFO]: <简短原因>;发现问题时返回简洁、可操作的告警;无法完成时返回 NO_REPLY[FAIL]: <原因>;因安全或权限拒绝时返回 NO_REPLY[REFUSE]: <原因>。不要调用 send_message不要把不确定当作正常。"
} else {
"这是 Scheduler 托管投递的定时任务。完成任务后只返回应交付给用户的最终内容,不要调用 send_message。"
};
let system = format!(
"{base_prompt}\n\n{skills_prompt}\n\n## 定时任务执行\n任务「{job_name}」({job_id})。\n{result_contract}"
);
let history = vec![ChatMessage::system(system), ChatMessage::user(prompt)];
let source_session = format!("cron:{job_id}");
let result = CURRENT_SOURCE_SESSION
.scope(Some(source_session.clone()), async {
agent
.process_with_context(
history,
ToolExecutionContext::for_session(source_session),
)
.await
})
.await?;
Ok(result.final_response.content)
}
pub async fn clear_session_history( pub async fn clear_session_history(
&self, &self,
unified_id: &UnifiedSessionId, unified_id: &UnifiedSessionId,

View File

@ -1175,7 +1175,7 @@ mod tests {
.fetch_one(storage.pool()) .fetch_one(storage.pool())
.await .await
.unwrap(); .unwrap();
assert_eq!(version, 10); assert_eq!(version, crate::storage::SCHEMA_VERSION);
for table in [ for table in [
"agent_runs", "agent_runs",
"agent_session_state", "agent_session_state",

View File

@ -10,7 +10,10 @@ pub mod usage;
pub use context_checkpoint::{ContextCheckpoint, ContextCheckpointState, NewContextCheckpoint}; pub use context_checkpoint::{ContextCheckpoint, ContextCheckpointState, NewContextCheckpoint};
pub use error::StorageError; pub use error::StorageError;
pub use scheduler::{DeliveryPolicy, JobKind, JobRun, ScheduledJob}; pub use scheduler::{
ClaimedScheduledRun, DeliveryPolicy, JobRun, ScheduledDeliveryStatus, ScheduledJob,
ScheduledJobUpdate, ScheduledOutcomeKind, ScheduledRunCompletion, ScheduledRunStatus,
};
pub use usage::{SessionUsageTotals, TurnUsageRecord}; pub use usage::{SessionUsageTotals, TurnUsageRecord};
use sqlx::sqlite::{ use sqlx::sqlite::{
@ -20,7 +23,7 @@ use sqlx::{Pool, Row, Sqlite};
use std::path::Path; use std::path::Path;
use tokio::time::{Duration, sleep}; use tokio::time::{Duration, sleep};
const SCHEMA_VERSION: i64 = 10; const SCHEMA_VERSION: i64 = 11;
const INSERT_MESSAGE_SQL: &str = r#" const INSERT_MESSAGE_SQL: &str = r#"
INSERT INTO messages ( INSERT INTO messages (
id, session_id, seq, role, content, reasoning_content, provider_state, id, session_id, seq, role, content, reasoning_content, provider_state,
@ -29,6 +32,15 @@ const INSERT_MESSAGE_SQL: &str = r#"
) )
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
"#; "#;
const INSERT_MESSAGE_IF_ABSENT_SQL: &str = r#"
INSERT INTO messages (
id, session_id, seq, role, content, reasoning_content, provider_state,
turn_id, iteration, completion_status, client_visibility, turn_origin,
media_refs, tool_call_id, tool_name, tool_calls, source, created_at
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(id) DO NOTHING
"#;
pub(crate) fn insert_message_query<'a>( pub(crate) fn insert_message_query<'a>(
session_id: &'a str, session_id: &'a str,
@ -55,6 +67,31 @@ pub(crate) fn insert_message_query<'a>(
.bind(msg.created_at) .bind(msg.created_at)
} }
pub(crate) fn insert_message_if_absent_query<'a>(
session_id: &'a str,
msg: &'a crate::storage::message::MessageMeta,
) -> sqlx::query::Query<'a, Sqlite, sqlx::sqlite::SqliteArguments> {
sqlx::query(INSERT_MESSAGE_IF_ABSENT_SQL)
.bind(&msg.id)
.bind(session_id)
.bind(msg.seq)
.bind(&msg.role)
.bind(&msg.content)
.bind(&msg.reasoning_content)
.bind(&msg.provider_state)
.bind(&msg.turn_id)
.bind(msg.iteration)
.bind(msg.completion_status.as_str())
.bind(msg.client_visibility.as_str())
.bind(msg.turn_origin.as_str())
.bind(&msg.media_refs)
.bind(&msg.tool_call_id)
.bind(&msg.tool_name)
.bind(&msg.tool_calls)
.bind(&msg.source)
.bind(msg.created_at)
}
fn message_meta_from_row(row: SqliteRow) -> crate::storage::message::MessageMeta { fn message_meta_from_row(row: SqliteRow) -> crate::storage::message::MessageMeta {
let completion_status: String = row.get("completion_status"); let completion_status: String = row.get("completion_status");
crate::storage::message::MessageMeta { crate::storage::message::MessageMeta {
@ -407,7 +444,6 @@ impl Storage {
.execute(&self.pool) .execute(&self.pool)
.await?; .await?;
Self::init_scheduler_schema(&self.pool).await?;
self.migrate_schema().await?; self.migrate_schema().await?;
Ok(()) Ok(())
@ -428,7 +464,11 @@ impl Storage {
return Ok(()); return Ok(());
} }
let mut tx = self.pool.begin().await?; // Acquire the migration write lock before inspecting or modifying any
// legacy shape. This keeps the v11 rebuild on one connection and makes
// a concurrently running older Gateway fail startup cleanly instead of
// partially racing the schema migration.
let mut tx = self.pool.begin_with("BEGIN IMMEDIATE").await?;
// The legacy drops below are a pre-v8 rebuild concern: the batch // The legacy drops below are a pre-v8 rebuild concern: the batch
// "group" concept was removed in v8 and the old `background_tasks` // "group" concept was removed in v8 and the old `background_tasks`
// table in v7. Gate them on `current < 8` so a v8 -> v9 upgrade only // table in v7. Gate them on `current < 8` so a v8 -> v9 upgrade only
@ -458,6 +498,33 @@ impl Storage {
.execute(&mut *tx) .execute(&mut *tx)
.await?; .await?;
} }
let legacy_scheduler_exists: i64 = sqlx::query_scalar(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'scheduled_jobs'",
)
.fetch_one(&mut *tx)
.await?;
let legacy_scheduler_exists = legacy_scheduler_exists == 1;
if legacy_scheduler_exists {
sqlx::query(
r#"
CREATE TABLE IF NOT EXISTS job_runs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
job_id TEXT NOT NULL REFERENCES scheduled_jobs(id) ON DELETE CASCADE,
started_at INTEGER NOT NULL,
finished_at INTEGER NOT NULL,
status TEXT NOT NULL,
output TEXT,
error TEXT,
duration_ms INTEGER NOT NULL,
result_kind TEXT,
delivery_status TEXT,
delivery_error TEXT
)
"#,
)
.execute(&mut *tx)
.await?;
}
for (table, column, definition) in [ for (table, column, definition) in [
("messages", "source", "source TEXT"), ("messages", "source", "source TEXT"),
("messages", "reasoning_content", "reasoning_content TEXT"), ("messages", "reasoning_content", "reasoning_content TEXT"),
@ -528,9 +595,10 @@ impl Storage {
let columns = sqlx::query(sqlx::AssertSqlSafe(pragma)) let columns = sqlx::query(sqlx::AssertSqlSafe(pragma))
.fetch_all(&mut *tx) .fetch_all(&mut *tx)
.await?; .await?;
if !columns if !columns.is_empty()
.iter() && !columns
.any(|row| row.get::<String, _>("name") == column) .iter()
.any(|row| row.get::<String, _>("name") == column)
{ {
let alter = format!("ALTER TABLE {table} ADD COLUMN {definition}"); let alter = format!("ALTER TABLE {table} ADD COLUMN {definition}");
// All identifiers and definitions come from the fixed migration list above. // All identifiers and definitions come from the fixed migration list above.
@ -562,11 +630,6 @@ impl Storage {
) )
.execute(&mut *tx) .execute(&mut *tx)
.await?; .await?;
sqlx::query(
"CREATE INDEX IF NOT EXISTS idx_jobs_claimable ON scheduled_jobs(enabled, next_run_at, lease_until)",
)
.execute(&mut *tx)
.await?;
sqlx::query( sqlx::query(
r#" r#"
CREATE TABLE IF NOT EXISTS session_turn_usage ( CREATE TABLE IF NOT EXISTS session_turn_usage (
@ -629,6 +692,7 @@ impl Storage {
for statement in agent_run::AGENT_SCHEMA_STATEMENTS { for statement in agent_run::AGENT_SCHEMA_STATEMENTS {
sqlx::query(*statement).execute(&mut *tx).await?; sqlx::query(*statement).execute(&mut *tx).await?;
} }
scheduler::migrate_scheduler_v11(&mut tx, legacy_scheduler_exists).await?;
sqlx::query(sqlx::AssertSqlSafe(format!( sqlx::query(sqlx::AssertSqlSafe(format!(
"PRAGMA user_version = {SCHEMA_VERSION}" "PRAGMA user_version = {SCHEMA_VERSION}"
))) )))
@ -638,70 +702,6 @@ impl Storage {
Ok(()) Ok(())
} }
/// Initialize the scheduler tables (idempotent).
pub(crate) async fn init_scheduler_schema(pool: &Pool<Sqlite>) -> Result<(), StorageError> {
sqlx::query(
r#"
CREATE TABLE IF NOT EXISTS scheduled_jobs (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
schedule TEXT NOT NULL,
prompt TEXT NOT NULL,
channel TEXT NOT NULL,
chat_id TEXT NOT NULL,
model TEXT,
job_kind TEXT NOT NULL DEFAULT 'task',
delivery_policy TEXT NOT NULL DEFAULT 'direct',
enabled INTEGER NOT NULL DEFAULT 1,
delete_after_run INTEGER NOT NULL DEFAULT 0,
next_run_at INTEGER NOT NULL,
last_run_at INTEGER,
last_status TEXT,
last_error TEXT,
locked_at INTEGER,
lock_owner TEXT,
lease_until INTEGER,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL
)
"#,
)
.execute(pool)
.await?;
sqlx::query(
r#"
CREATE TABLE IF NOT EXISTS job_runs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
job_id TEXT NOT NULL REFERENCES scheduled_jobs(id) ON DELETE CASCADE,
started_at INTEGER NOT NULL,
finished_at INTEGER NOT NULL,
status TEXT NOT NULL,
output TEXT,
error TEXT,
duration_ms INTEGER NOT NULL,
result_kind TEXT,
delivery_status TEXT,
delivery_error TEXT
)
"#,
)
.execute(pool)
.await?;
sqlx::query(
"CREATE INDEX IF NOT EXISTS idx_jobs_next_run ON scheduled_jobs(enabled, next_run_at)",
)
.execute(pool)
.await?;
sqlx::query("CREATE INDEX IF NOT EXISTS idx_runs_job_id ON job_runs(job_id)")
.execute(pool)
.await?;
Ok(())
}
pub async fn append_llm_call( pub async fn append_llm_call(
&self, &self,
provider: &str, provider: &str,
@ -962,6 +962,56 @@ impl Storage {
Ok(msg.seq) Ok(msg.seq)
} }
pub async fn persist_message_if_absent(
&self,
session_id: &str,
msg: &crate::storage::message::MessageMeta,
meta: &crate::storage::session::SessionMeta,
) -> Result<bool, StorageError> {
let mut tx = self.pool.begin().await?;
let inserted = insert_message_if_absent_query(session_id, msg)
.execute(&mut *tx)
.await?
.rows_affected()
== 1;
if inserted {
sqlx::query(
r#"
INSERT INTO sessions (id, channel, chat_id, dialog_id, title, created_at,
last_active_at, message_count, routing_info, archived_at, deleted_at,
last_consolidated_at, last_compressed_message_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(id) DO UPDATE SET
title = excluded.title,
last_active_at = excluded.last_active_at,
message_count = excluded.message_count,
routing_info = excluded.routing_info,
archived_at = excluded.archived_at,
deleted_at = excluded.deleted_at,
last_consolidated_at = excluded.last_consolidated_at,
last_compressed_message_at = excluded.last_compressed_message_at
"#,
)
.bind(&meta.id)
.bind(&meta.channel)
.bind(&meta.chat_id)
.bind(&meta.dialog_id)
.bind(&meta.title)
.bind(meta.created_at)
.bind(meta.last_active_at)
.bind(meta.message_count)
.bind(&meta.routing_info)
.bind(meta.archived_at)
.bind(meta.deleted_at)
.bind(meta.last_consolidated_at)
.bind(meta.last_compressed_message_at)
.execute(&mut *tx)
.await?;
}
tx.commit().await?;
Ok(inserted)
}
/// Atomically persist all messages produced by one logical turn together /// Atomically persist all messages produced by one logical turn together
/// with the resulting session metadata. A turn is either fully visible /// with the resulting session metadata. A turn is either fully visible
/// after restart or not visible at all. /// after restart or not visible at all.
@ -1584,6 +1634,59 @@ mod tests {
(storage, dir) (storage, dir)
} }
async fn scheduler_schema_sql(storage: &Storage) -> Vec<(String, String, Option<String>)> {
sqlx::query_as::<_, (String, String, Option<String>)>(
r#"
SELECT type, name, sql FROM sqlite_master
WHERE name IN (
'scheduled_jobs', 'job_runs', 'idx_jobs_claimable',
'idx_job_runs_job_finished', 'idx_job_runs_recovery',
'idx_job_runs_delivery'
)
ORDER BY type, name
"#,
)
.fetch_all(storage.pool())
.await
.unwrap()
}
async fn create_v10_scheduler_schema(pool: &Pool<Sqlite>) {
sqlx::query(
r#"
CREATE TABLE scheduled_jobs (
id TEXT PRIMARY KEY, name TEXT NOT NULL, schedule TEXT NOT NULL,
prompt TEXT NOT NULL, channel TEXT NOT NULL, chat_id TEXT NOT NULL,
model TEXT, job_kind TEXT NOT NULL DEFAULT 'task',
delivery_policy TEXT NOT NULL DEFAULT 'direct',
enabled INTEGER NOT NULL DEFAULT 1,
delete_after_run INTEGER NOT NULL DEFAULT 0, next_run_at INTEGER NOT NULL,
last_run_at INTEGER, last_status TEXT, last_error TEXT,
locked_at INTEGER, lock_owner TEXT, lease_until INTEGER,
created_at INTEGER NOT NULL, updated_at INTEGER NOT NULL
)
"#,
)
.execute(pool)
.await
.unwrap();
sqlx::query(
r#"
CREATE TABLE job_runs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
job_id TEXT NOT NULL REFERENCES scheduled_jobs(id) ON DELETE CASCADE,
started_at INTEGER NOT NULL, finished_at INTEGER NOT NULL,
status TEXT NOT NULL, output TEXT, error TEXT,
duration_ms INTEGER NOT NULL, result_kind TEXT,
delivery_status TEXT, delivery_error TEXT
)
"#,
)
.execute(pool)
.await
.unwrap();
}
#[tokio::test] #[tokio::test]
async fn sqlite_runtime_guards_are_enabled() { async fn sqlite_runtime_guards_are_enabled() {
let (storage, _dir) = create_test_storage().await; let (storage, _dir) = create_test_storage().await;
@ -1764,7 +1867,7 @@ mod tests {
} }
#[tokio::test] #[tokio::test]
async fn legacy_schema_is_migrated_without_rebuild() { async fn legacy_schema_is_migrated_to_canonical_v11() {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
let db_path = dir.path().join("legacy.db"); let db_path = dir.path().join("legacy.db");
let pool = SqlitePoolOptions::new() let pool = SqlitePoolOptions::new()
@ -1860,7 +1963,14 @@ mod tests {
), ),
( (
"scheduled_jobs", "scheduled_jobs",
vec!["locked_at", "lock_owner", "lease_until"], vec![
"agent_id",
"delivery_policy",
"last_outcome",
"locked_at",
"lock_owner",
"lease_until",
],
), ),
] { ] {
let columns = sqlx::query(sqlx::AssertSqlSafe(format!("PRAGMA table_info({table})"))) let columns = sqlx::query(sqlx::AssertSqlSafe(format!("PRAGMA table_info({table})")))
@ -1876,6 +1986,25 @@ mod tests {
); );
} }
} }
let scheduler_columns = sqlx::query("PRAGMA table_info(scheduled_jobs)")
.fetch_all(storage.pool())
.await
.unwrap()
.into_iter()
.map(|row| row.get::<String, _>("name"))
.collect::<Vec<_>>();
for removed in [
"model",
"job_kind",
"delete_after_run",
"last_status",
"last_error",
] {
assert!(
!scheduler_columns.iter().any(|column| column == removed),
"legacy scheduler column survived v11: {removed}"
);
}
let schema_version: i64 = sqlx::query_scalar("PRAGMA user_version") let schema_version: i64 = sqlx::query_scalar("PRAGMA user_version")
.fetch_one(storage.pool()) .fetch_one(storage.pool())
.await .await
@ -1914,6 +2043,250 @@ mod tests {
assert_eq!(origin, "user"); assert_eq!(origin, "user");
} }
#[tokio::test]
async fn v10_scheduler_history_and_active_lease_migrate_to_v11() {
let dir = tempfile::tempdir().unwrap();
let db_path = dir.path().join("v10-scheduler.db");
let pool = SqlitePoolOptions::new()
.connect_with(
SqliteConnectOptions::new()
.filename(&db_path)
.create_if_missing(true),
)
.await
.unwrap();
create_v10_scheduler_schema(&pool).await;
sqlx::query(
r#"
INSERT INTO scheduled_jobs
(id, name, schedule, prompt, channel, chat_id, model, job_kind,
delivery_policy, enabled, next_run_at, last_run_at, last_status,
last_error, created_at, updated_at)
VALUES ('legacy-direct', 'Legacy direct', '{"type":"every","every_ms":60000}',
'check it', 'cli_chat', 'chat', 'obsolete-model', 'monitor',
'direct', 1, 2000, 1000, 'delivery_error', 'legacy diagnostic', 1, 2)
"#,
)
.execute(&pool)
.await
.unwrap();
sqlx::query(
r#"
INSERT INTO job_runs
(job_id, started_at, finished_at, status, output, error, duration_ms,
result_kind, delivery_status, delivery_error)
VALUES ('legacy-direct', 1000, 1100, 'delivery_error', 'legacy result', NULL,
100, NULL, 'failed', 'channel rejected target')
"#,
)
.execute(&pool)
.await
.unwrap();
sqlx::query(
r#"
INSERT INTO scheduled_jobs
(id, name, schedule, prompt, channel, chat_id, job_kind,
delivery_policy, enabled, next_run_at, last_run_at, last_status,
locked_at, lock_owner, lease_until, created_at, updated_at)
VALUES ('legacy-locked', 'Legacy locked', '{"type":"every","every_ms":60000}',
'check lock', 'cli_chat', 'chat', 'task', 'never', 1, 2000,
1500, NULL, 1500, 'old-owner', 900000, 1, 2)
"#,
)
.execute(&pool)
.await
.unwrap();
sqlx::query("PRAGMA user_version = 10")
.execute(&pool)
.await
.unwrap();
drop(pool);
let storage = Storage::new(&db_path).await.unwrap();
let migrated_columns = sqlx::query("PRAGMA table_info(scheduled_jobs)")
.fetch_all(storage.pool())
.await
.unwrap()
.into_iter()
.map(|row| row.get::<String, _>("name"))
.collect::<Vec<_>>();
assert!(
migrated_columns
.iter()
.any(|column| column == "last_outcome"),
"unexpected post-migration scheduler schema: {migrated_columns:?}"
);
let direct = storage.get_scheduled_job("legacy-direct").await.unwrap();
assert_eq!(
direct.delivery_policy,
crate::storage::DeliveryPolicy::Always
);
assert_eq!(
direct.last_outcome,
Some(crate::storage::ScheduledOutcomeKind::Ok)
);
let historical = storage
.list_scheduled_job_runs("legacy-direct", 10)
.await
.unwrap();
assert_eq!(historical.len(), 1);
assert_eq!(
historical[0].status,
crate::storage::ScheduledRunStatus::Completed
);
assert_eq!(
historical[0].outcome,
Some(crate::storage::ScheduledOutcomeKind::Ok)
);
assert_eq!(
historical[0].delivery_status,
crate::storage::ScheduledDeliveryStatus::Failed
);
assert_eq!(
historical[0].diagnostic.as_deref(),
Some("legacy diagnostic")
);
let recovered = storage
.list_scheduled_job_runs("legacy-locked", 10)
.await
.unwrap();
assert_eq!(recovered.len(), 1);
assert_eq!(
recovered[0].status,
crate::storage::ScheduledRunStatus::Unknown
);
assert_eq!(
recovered[0].delivery_status,
crate::storage::ScheduledDeliveryStatus::NotRequested
);
let locked = storage.get_scheduled_job("legacy-locked").await.unwrap();
assert!(locked.lock_owner.is_none());
let columns = sqlx::query("PRAGMA table_info(scheduled_jobs)")
.fetch_all(storage.pool())
.await
.unwrap()
.into_iter()
.map(|row| row.get::<String, _>("name"))
.collect::<Vec<_>>();
for removed in [
"model",
"job_kind",
"delete_after_run",
"last_status",
"last_error",
] {
assert!(!columns.iter().any(|column| column == removed));
}
let violations = sqlx::query("PRAGMA foreign_key_check")
.fetch_all(storage.pool())
.await
.unwrap();
assert!(violations.is_empty());
let fresh_dir = tempfile::tempdir().unwrap();
let fresh = Storage::new(&fresh_dir.path().join("fresh.db"))
.await
.unwrap();
assert_eq!(
scheduler_schema_sql(&storage).await,
scheduler_schema_sql(&fresh).await
);
}
#[tokio::test]
async fn invalid_v10_scheduler_data_rolls_back_v11_migration() {
let dir = tempfile::tempdir().unwrap();
let db_path = dir.path().join("invalid-v10-scheduler.db");
let pool = SqlitePoolOptions::new()
.connect_with(
SqliteConnectOptions::new()
.filename(&db_path)
.create_if_missing(true),
)
.await
.unwrap();
create_v10_scheduler_schema(&pool).await;
sqlx::query(
r#"
INSERT INTO scheduled_jobs
(id, name, schedule, prompt, channel, chat_id, delivery_policy,
enabled, next_run_at, created_at, updated_at)
VALUES ('broken', 'Broken', '{not-json}', 'check', 'cli_chat', 'chat',
'always', 1, 1000, 1, 1)
"#,
)
.execute(&pool)
.await
.unwrap();
sqlx::query("PRAGMA user_version = 10")
.execute(&pool)
.await
.unwrap();
drop(pool);
let error = match Storage::new(&db_path).await {
Ok(_) => panic!("invalid v10 scheduler data unexpectedly migrated"),
Err(error) => error,
};
assert!(matches!(error, StorageError::Migration(_)));
let pool = SqlitePoolOptions::new()
.connect_with(SqliteConnectOptions::new().filename(&db_path))
.await
.unwrap();
let version: i64 = sqlx::query_scalar("PRAGMA user_version")
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(version, 10);
let columns = sqlx::query("PRAGMA table_info(scheduled_jobs)")
.fetch_all(&pool)
.await
.unwrap()
.into_iter()
.map(|row| row.get::<String, _>("name"))
.collect::<Vec<_>>();
assert!(columns.iter().any(|column| column == "job_kind"));
assert!(!columns.iter().any(|column| column == "agent_id"));
for table in ["scheduled_jobs_v10_legacy", "job_runs_v10_legacy"] {
let exists: i64 = sqlx::query_scalar(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = ?",
)
.bind(table)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(exists, 0);
}
}
#[tokio::test]
async fn newer_schema_version_is_rejected() {
let dir = tempfile::tempdir().unwrap();
let db_path = dir.path().join("future.db");
let pool = SqlitePoolOptions::new()
.connect_with(
SqliteConnectOptions::new()
.filename(&db_path)
.create_if_missing(true),
)
.await
.unwrap();
sqlx::query("PRAGMA user_version = 12")
.execute(&pool)
.await
.unwrap();
drop(pool);
let error = match Storage::new(&db_path).await {
Ok(_) => panic!("newer schema version unexpectedly opened"),
Err(error) => error,
};
assert!(matches!(error, StorageError::Migration(_)));
}
#[tokio::test] #[tokio::test]
async fn v3_migration_preserves_existing_reasoning_and_defaults_completion() { async fn v3_migration_preserves_existing_reasoning_and_defaults_completion() {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,191 @@
use async_trait::async_trait;
use serde_json::{Value, json};
use crate::storage::ScheduledOutcomeKind;
use crate::tools::{ScheduledOutcome, Tool, ToolExecutionContext, ToolOutput, ToolResult};
pub struct CompleteScheduledRunTool;
impl CompleteScheduledRunTool {
pub fn new() -> Self {
Self
}
}
impl Default for CompleteScheduledRunTool {
fn default() -> Self {
Self::new()
}
}
#[async_trait]
impl Tool for CompleteScheduledRunTool {
fn name(&self) -> &str {
"complete_scheduled_run"
}
fn description(&self) -> &str {
"Submit the single structured final outcome (ok, alert, failed, or refused) of an unattended scheduled run and end the run immediately."
}
fn parameters_schema(&self) -> Value {
json!({
"type": "object",
"properties": {
"outcome": {
"type": "string",
"enum": ["ok", "alert", "failed", "refused"],
"description": "ok: completed with nothing needing attention; alert: completed with actionable findings; failed: did not complete reliably; refused: denied for permission or safety reasons"
},
"message": {
"type": "string",
"minLength": 1,
"maxLength": 16384,
"description": "The user-facing result or notification body"
}
},
"required": ["outcome", "message"],
"additionalProperties": false
})
}
fn runtime_injected(&self) -> bool {
true
}
fn exclusive(&self) -> bool {
true
}
async fn execute(&self, args: Value) -> anyhow::Result<ToolResult> {
self.execute_with_context(&ToolExecutionContext::default(), args)
.await
.map(|output| output.result)
}
async fn execute_with_context(
&self,
context: &ToolExecutionContext,
args: Value,
) -> anyhow::Result<ToolOutput> {
let Some(sink) = context.scheduled_completion.as_ref() else {
return Ok(failure(
"complete_scheduled_run is only available to the top-level scheduled Agent",
)
.into());
};
if !context.execution_origin.is_scheduled() {
return Ok(failure("scheduled completion context is invalid").into());
}
let Some(object) = args.as_object() else {
return Ok(failure("arguments must be an object").into());
};
if object
.keys()
.any(|key| key != "outcome" && key != "message")
{
return Ok(failure("unknown complete_scheduled_run argument").into());
}
let kind = match args.get("outcome").and_then(Value::as_str) {
Some("ok") => ScheduledOutcomeKind::Ok,
Some("alert") => ScheduledOutcomeKind::Alert,
Some("failed") => ScheduledOutcomeKind::Failed,
Some("refused") => ScheduledOutcomeKind::Refused,
Some(other) => return Ok(failure(format!("invalid scheduled outcome: {other}")).into()),
None => return Ok(failure("outcome is required").into()),
};
let message = args
.get("message")
.and_then(Value::as_str)
.unwrap_or_default()
.trim();
if message.is_empty() {
return Ok(failure("message must not be empty").into());
}
if message.chars().count() > 16_384 {
return Ok(failure("message exceeds 16384 characters").into());
}
let message = message.to_string();
if let Err(error) = sink.submit(ScheduledOutcome {
kind,
message: message.clone(),
}) {
return Ok(failure(error).into());
}
Ok(ToolResult {
success: true,
output: format!("scheduled run completed with outcome={}", kind.as_str()),
error: None,
}
.into())
}
}
fn failure(error: impl Into<String>) -> ToolResult {
ToolResult {
success: false,
output: String::new(),
error: Some(error.into()),
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::tools::{ExecutionOrigin, ScheduledCompletionSink};
use std::sync::Arc;
#[tokio::test]
async fn submits_exactly_once_in_scheduled_context() {
let sink = Arc::new(ScheduledCompletionSink::default());
let context = ToolExecutionContext::for_session("scheduled-run:1")
.with_execution_origin(ExecutionOrigin::Scheduled { job_run_id: 1 })
.with_scheduled_completion(sink.clone());
let tool = CompleteScheduledRunTool::new();
let first = tool
.execute_with_context(&context, json!({"outcome":"ok","message":"healthy"}))
.await
.unwrap();
assert!(first.result.success);
assert_eq!(sink.outcome().unwrap().message, "healthy");
let second = tool
.execute_with_context(&context, json!({"outcome":"alert","message":"again"}))
.await
.unwrap();
assert!(!second.result.success);
}
#[tokio::test]
async fn rejects_empty_extra_and_oversized_arguments() {
let sink = Arc::new(ScheduledCompletionSink::default());
let context = ToolExecutionContext::for_session("scheduled-run:1")
.with_execution_origin(ExecutionOrigin::Scheduled { job_run_id: 1 })
.with_scheduled_completion(sink);
let tool = CompleteScheduledRunTool::new();
for args in [
json!({"outcome":"ok","message":" "}),
json!({"outcome":"ok","message":"fine","notify":false}),
json!({"outcome":"unknown","message":"fine"}),
] {
assert!(
!tool
.execute_with_context(&context, args)
.await
.unwrap()
.result
.success
);
}
assert!(
!tool
.execute_with_context(
&context,
json!({"outcome":"ok","message":"x".repeat(16_385)}),
)
.await
.unwrap()
.result
.success
);
}
}

File diff suppressed because it is too large Load Diff

View File

@ -155,7 +155,7 @@ impl DelegateTool {
args: &Value, args: &Value,
context: &ToolExecutionContext, context: &ToolExecutionContext,
) -> anyhow::Result<ToolResult> { ) -> anyhow::Result<ToolResult> {
let mode = match args let requested_mode = match args
.get("mode") .get("mode")
.and_then(Value::as_str) .and_then(Value::as_str)
.unwrap_or("foreground") .unwrap_or("foreground")
@ -168,6 +168,13 @@ impl DelegateTool {
))); )));
} }
}; };
let background_downgraded =
requested_mode == ExecutionMode::Background && context.execution_origin.is_scheduled();
let mode = if background_downgraded {
ExecutionMode::Foreground
} else {
requested_mode
};
let task_values: Vec<&Value> = match args.get("tasks").and_then(Value::as_array) { let task_values: Vec<&Value> = match args.get("tasks").and_then(Value::as_array) {
Some(tasks) if !tasks.is_empty() => tasks.iter().collect(), Some(tasks) if !tasks.is_empty() => tasks.iter().collect(),
Some(_) => return Ok(failure("tasks must not be empty")), Some(_) => return Ok(failure("tasks must not be empty")),
@ -248,6 +255,7 @@ impl DelegateTool {
success: all_completed, success: all_completed,
output: serde_json::to_string(&json!({ output: serde_json::to_string(&json!({
"status": if all_completed { "completed" } else { "partial" }, "status": if all_completed { "completed" } else { "partial" },
"background_downgraded": background_downgraded,
"results": payload "results": payload
}))?, }))?,
error: None, error: None,

View File

@ -3,6 +3,7 @@ pub mod bash;
pub mod browser; pub mod browser;
pub mod calculator; pub mod calculator;
pub mod chat_manager; pub mod chat_manager;
pub mod complete_scheduled_run;
pub mod content_search; pub mod content_search;
pub mod cron; pub mod cron;
pub mod delegate; pub mod delegate;
@ -32,6 +33,7 @@ pub use bash::BashTool;
pub use browser::{BrowserProfilesTool, BrowserTool}; pub use browser::{BrowserProfilesTool, BrowserTool};
pub use calculator::CalculatorTool; pub use calculator::CalculatorTool;
pub use chat_manager::ChatManagerTool; pub use chat_manager::ChatManagerTool;
pub use complete_scheduled_run::CompleteScheduledRunTool;
pub use content_search::ContentSearchTool; pub use content_search::ContentSearchTool;
pub use delegate::DelegateTool; pub use delegate::DelegateTool;
pub use emit_signal::EmitSignalTool; pub use emit_signal::EmitSignalTool;
@ -50,8 +52,9 @@ pub use reload_config::ReloadConfigTool;
pub use send_message::SendMessageTool; pub use send_message::SendMessageTool;
pub use todo::TodoTool; pub use todo::TodoTool;
pub use traits::{ pub use traits::{
OutboundDelivery, OutboundMessenger, ProcessedToolOutput, Tool, ToolArtifact, ExecutionOrigin, OutboundDelivery, OutboundMessenger, ProcessedToolOutput,
ToolArtifactAudience, ToolExecutionContext, ToolOutput, ToolOutputProcessor, ToolResult, ScheduledCompletionSink, ScheduledOutcome, Tool, ToolArtifact, ToolArtifactAudience,
ToolExecutionContext, ToolOutput, ToolOutputProcessor, ToolResult,
}; };
pub use web_fetch::WebFetchTool; pub use web_fetch::WebFetchTool;

View File

@ -1,6 +1,60 @@
use crate::bus::{MediaItem, MediaRef, MessageSource}; use crate::bus::{MediaItem, MediaRef, MessageSource};
use async_trait::async_trait; use async_trait::async_trait;
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum ExecutionOrigin {
#[default]
Interactive,
Scheduled {
job_run_id: i64,
},
}
impl ExecutionOrigin {
pub fn is_scheduled(self) -> bool {
matches!(self, Self::Scheduled { .. })
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ScheduledOutcome {
pub kind: crate::storage::ScheduledOutcomeKind,
pub message: String,
}
#[derive(Debug, Default)]
pub struct ScheduledCompletionSink {
outcome: std::sync::Mutex<Option<ScheduledOutcome>>,
}
impl ScheduledCompletionSink {
pub fn submit(&self, outcome: ScheduledOutcome) -> Result<(), &'static str> {
let mut guard = self
.outcome
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner());
if guard.is_some() {
return Err("scheduled result was already submitted");
}
*guard = Some(outcome);
Ok(())
}
pub fn outcome(&self) -> Option<ScheduledOutcome> {
self.outcome
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.clone()
}
pub fn is_completed(&self) -> bool {
self.outcome
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.is_some()
}
}
/// Session identity supplied by the runtime for tools that own external state. /// Session identity supplied by the runtime for tools that own external state.
/// Ordinary stateless tools can ignore it through the default trait method. /// Ordinary stateless tools can ignore it through the default trait method.
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
@ -10,6 +64,8 @@ pub struct ToolExecutionContext {
pub agent: Option<std::sync::Arc<crate::agent::AgentExecutionContext>>, pub agent: Option<std::sync::Arc<crate::agent::AgentExecutionContext>>,
pub cancellation: tokio_util::sync::CancellationToken, pub cancellation: tokio_util::sync::CancellationToken,
pub execution_gate: Option<std::sync::Arc<crate::agent::gate::ExecutionGate>>, pub execution_gate: Option<std::sync::Arc<crate::agent::gate::ExecutionGate>>,
pub execution_origin: ExecutionOrigin,
pub scheduled_completion: Option<std::sync::Arc<ScheduledCompletionSink>>,
} }
impl Default for ToolExecutionContext { impl Default for ToolExecutionContext {
@ -20,6 +76,8 @@ impl Default for ToolExecutionContext {
agent: None, agent: None,
cancellation: tokio_util::sync::CancellationToken::new(), cancellation: tokio_util::sync::CancellationToken::new(),
execution_gate: None, execution_gate: None,
execution_origin: ExecutionOrigin::Interactive,
scheduled_completion: None,
} }
} }
} }
@ -32,6 +90,8 @@ impl ToolExecutionContext {
agent: None, agent: None,
cancellation: tokio_util::sync::CancellationToken::new(), cancellation: tokio_util::sync::CancellationToken::new(),
execution_gate: None, execution_gate: None,
execution_origin: ExecutionOrigin::Interactive,
scheduled_completion: None,
} }
} }
@ -60,6 +120,19 @@ impl ToolExecutionContext {
self.execution_gate = Some(gate); self.execution_gate = Some(gate);
self self
} }
pub fn with_execution_origin(mut self, execution_origin: ExecutionOrigin) -> Self {
self.execution_origin = execution_origin;
self
}
pub fn with_scheduled_completion(
mut self,
sink: std::sync::Arc<ScheduledCompletionSink>,
) -> Self {
self.scheduled_completion = Some(sink);
self
}
} }
#[derive(Debug, Clone)] #[derive(Debug, Clone)]

View File

@ -1,12 +1,12 @@
{ {
"name": "picobot-webui", "name": "picobot-webui",
"version": "1.21.0", "version": "1.22.0",
"lockfileVersion": 3, "lockfileVersion": 3,
"requires": true, "requires": true,
"packages": { "packages": {
"": { "": {
"name": "picobot-webui", "name": "picobot-webui",
"version": "1.21.0", "version": "1.22.0",
"dependencies": { "dependencies": {
"bits-ui": "^2.0.0", "bits-ui": "^2.0.0",
"dompurify": "^3.4.12", "dompurify": "^3.4.12",

View File

@ -1,7 +1,7 @@
{ {
"name": "picobot-webui", "name": "picobot-webui",
"private": true, "private": true,
"version": "1.21.0", "version": "1.22.0",
"type": "module", "type": "module",
"engines": { "engines": {
"node": ">=20" "node": ">=20"

View File

@ -2,11 +2,11 @@
let { status = "unknown" } = $props(); let { status = "unknown" } = $props();
const normalized = $derived(String(status).toLowerCase()); const normalized = $derived(String(status).toLowerCase());
const tone = $derived( const tone = $derived(
["completed", "success", "ok", "enabled"].includes(normalized) ["completed", "success", "ok", "enabled", "delivered"].includes(normalized)
? "ok" ? "ok"
: ["failed", "error", "cancelled", "disabled"].includes(normalized) : ["failed", "error", "cancelled", "disabled", "refused", "timed_out"].includes(normalized)
? "fail" ? "fail"
: ["running", "pending"].includes(normalized) ? "run" : "" : ["running", "pending", "delivering", "alert", "unknown", "interrupted"].includes(normalized) ? "run" : ""
); );
</script> </script>

View File

@ -42,11 +42,23 @@
function dotColor(status) { function dotColor(status) {
if (status === "completed" || status === "success" || status === "ok") return "var(--signal)"; if (status === "completed" || status === "success" || status === "ok") return "var(--signal)";
if (status === "timeout") return "var(--accent)"; if (status === "timed_out" || status === "unknown" || status === "interrupted") return "var(--accent)";
if (status === "running") return "var(--info)"; if (status === "running") return "var(--info)";
return "var(--danger)"; return "var(--danger)";
} }
function scheduleLabel(schedule) {
if (!schedule) return "—";
if (schedule.type === "at") return `单次 · ${formatTime(schedule.at)}`;
if (schedule.type === "every") return `每 ${Math.max(1, Math.round(schedule.every_ms / 60000))} 分钟`;
if (schedule.type === "cron") return `${schedule.expr}${schedule.tz ? ` · ${schedule.tz}` : ""}`;
return schedule.type || "—";
}
function deliveryLabel(policy) {
return { always: "始终通知", on_alert: "异常通知", never: "从不通知" }[policy] || policy;
}
onMount(() => { onMount(() => {
load(); load();
const timer = setInterval(() => { tick += 1; }, 30000); const timer = setInterval(() => { tick += 1; }, 30000);
@ -58,7 +70,7 @@
<div class="toolbar"> <div class="toolbar">
<div> <div>
<h2 style="margin:0">定时任务</h2> <h2 style="margin:0">定时任务</h2>
<p style="margin:2px 0 0;color:var(--muted);font-size:12px">管理定时任务与巡检;后台子代理运行请到「子代理」页面查看</p> <p style="margin:2px 0 0;color:var(--muted);font-size:12px">查看统一的定时执行、结构化结果与通知状态Agent 运行审计请到「子代理」页面</p>
</div> </div>
<button class="secondary" onclick={load}><Icon name="refresh" size={16} />刷新</button> <button class="secondary" onclick={load}><Icon name="refresh" size={16} />刷新</button>
</div> </div>
@ -73,8 +85,9 @@
<h3>{job.name}</h3> <h3>{job.name}</h3>
<p>{job.prompt}</p> <p>{job.prompt}</p>
<div class="meta"> <div class="meta">
<span class="mono cron">{job.cron}</span> <span class="mono cron">{scheduleLabel(job.schedule)}</span>
<span>{job.job_kind === "monitor" ? "巡检" : "任务"} · {job.delivery_policy}</span> <span>Agent{job.agent_id || "Root"}</span>
<span>投递:{deliveryLabel(job.delivery_policy)}</span>
<span>{job.channel} · {job.chat_id}</span> <span>{job.channel} · {job.chat_id}</span>
<span>下次 {countdown(job.next_run_at)}</span> <span>下次 {countdown(job.next_run_at)}</span>
<span>上次 {formatTime(job.last_run_at)}</span> <span>上次 {formatTime(job.last_run_at)}</span>
@ -87,9 +100,23 @@
</div> </div>
{/if} {/if}
</div> </div>
<StatusBadge status={job.enabled ? (job.last_status || "enabled") : "disabled"} /> <StatusBadge status={job.enabled ? (job.last_outcome || "enabled") : "disabled"} />
</div> </div>
{#if runs[job.id]?.length}<div class="details">{#each runs[job.id] as run}<div class="card-row run-row"><span class="meta">{formatTime(run.finished_at)} · {run.duration_ms}ms{run.result_kind ? ` · ${run.result_kind}` : ""}{run.delivery_status ? ` · ${run.delivery_status}` : ""}</span><StatusBadge status={run.status} /></div>{/each}</div>{/if} {#if runs[job.id]?.length}
<div class="details">
{#each runs[job.id] as run}
<details class="run-detail">
<summary class="run-row">
<span class="meta">{formatTime(run.finished_at || run.started_at || run.scheduled_for)} · {run.duration_ms ?? "—"}ms · outcome={run.outcome || "—"} · delivery={run.delivery_status} ({run.delivery_attempts})</span>
<StatusBadge status={run.status} />
</summary>
{#if run.message}<p><b>结果</b><br />{run.message}</p>{/if}
{#if run.diagnostic}<p class="diagnostic"><b>诊断</b><br />{run.diagnostic}</p>{/if}
{#if run.delivery_error}<p class="diagnostic"><b>投递错误</b><br />{run.delivery_error}</p>{/if}
</details>
{/each}
</div>
{/if}
</article> </article>
{:else}<div class="empty-card">暂无定时任务</div>{/each} {:else}<div class="empty-card">暂无定时任务</div>{/each}
{/if} {/if}
@ -100,4 +127,9 @@
.cron { font-size: 12px; color: var(--text-soft); background: var(--code-bg); padding: 1px 6px; border-radius: 4px; border: 1px solid var(--line); font-variant-numeric: tabular-nums; } .cron { font-size: 12px; color: var(--text-soft); background: var(--code-bg); padding: 1px 6px; border-radius: 4px; border: 1px solid var(--line); font-variant-numeric: tabular-nums; }
.status-dots { display: flex; gap: 4px; margin-top: 6px; align-items: center; } .status-dots { display: flex; gap: 4px; margin-top: 6px; align-items: center; }
.run-row { display: flex; align-items: center; gap: 8px; flex-wrap: wrap; } .run-row { display: flex; align-items: center; gap: 8px; flex-wrap: wrap; }
.run-detail { padding: 7px 0; border-top: 1px solid var(--line); }
.run-detail:first-child { border-top: 0; }
.run-detail summary { cursor: pointer; justify-content: space-between; }
.run-detail p { margin: 8px 0 0; white-space: pre-wrap; overflow-wrap: anywhere; font-size: 12px; color: var(--text-soft); }
.run-detail .diagnostic { color: var(--danger); }
</style> </style>