Compare commits

...

11 Commits

Author SHA1 Message Date
oudecheng
f05f09b636 docs: 异步子代理设计文档
记录 v8 设计方案的三个核心不变量、wait 模式、取消机制与崩溃恢复策略
2026-08-13 22:13:00 +08:00
oudecheng
daf973a7dc fix(web): 子代理状态显示与渲染崩溃修复
MessageBubble:
- taskStatusConfig 补全 running/completed/cancelled/interrupted 状态配色,
  修复未知状态访问 undefined.borderColor 导致的白屏崩溃
- 添加安全回退到 failed 状态

main.tsx:
- 添加 ErrorBoundary 捕获 React 渲染错误,防止白屏并提供错误详情与恢复入口

useMessages:
- 收到子代理 execution_completed 事件后,更新主视图中 task tool result 占位消息的
  status 字段(running->completed/failed/cancelled),消除永久黄色转圈

protocol.ts:
- ExecutionCompleted 接口添加 subagent_status 和 subagent_summary 字段
2026-08-13 22:12:43 +08:00
oudecheng
fc3a95b152 feat: 异步子代理 v8 实现与 /stop 取消流程修复
后端核心实现:
- 新增 wait_coordinator:释放/重获取 serial_lock 的 in-tool waiting 模式,替代旧的 break-exit 方案
- 新增 wait_for_subagents 工具:select! 等待子代理完成/用户消息/超时,支持 try_drain 批量消费
- task 工具异步 spawn 路径:CancellationToken 注册表 + RAII guard + Semaphore 并发限流
- process exit 安全检查:pending 子代理存在时抑制 ExecutionCompleted
- 崩溃恢复:启动时标记 running->interrupted,history 加载时对账占位

/stop 取消流程修复:
- wait_coordinator select! 添加 cancel 分支,完整清理状态
- ToolContext 注入 cancel_rx(watch::Receiver clone)
- agent_loop 工具执行 select! 对 wait 工具跳过竞速,防止 drop coordinator 清理逻辑
- 子代理完成状态通过 execution_completed metadata 传播

存储层:
- pending_subagents 表 + 条件 UPDATE
- mark_all_running_as_interrupted 崩溃恢复
2026-08-13 22:12:20 +08:00
oudecheng
e0313ab8f3 fix: 增强 panic 安全与计算器健壮性
- agent_loop/processor: catch_unwind 隔离 panic,防止消息静默丢失
- calculator: 拒绝 NaN/Infinity 输入,修复阶乘溢出(上限 34),拒绝非有限表达式结果
- message: 修复 sanitize 两阶段删除索引未排序导致的越界 panic
- utils: 新增 panic_payload_message 提取可读 panic 消息
- .gitignore: 忽略 artifacts/ 测试产物目录
2026-08-13 08:38:36 +08:00
oudecheng
f65fb0167f chore: 忽略 artifacts 构建产物目录 2026-08-12 10:27:12 +08:00
oudecheng
a9ce308c05 fix(web): 移除 ChatContainer 的 key remount,修复新建话题列表刷新延迟
话题切换时 key={selectedTopic} 导致整个 ChatContainer 子树卸载重建,
触发 ExpertSelector/ModelSelector 重新挂载并发起 3 次冗余 HTTP 请求
(getSelectedExpert/listModelOptions/模型选择刷新),引入 200-500ms 延迟。

改用 topicId prop 透传话题 ID:
- App.tsx: 移除 key,viewKey 纳入 selectedTopic 保持各话题独立滚动位置
- ChatContainer: 新增 topicId prop 透传给 MessageInput
- MessageInput: 监听 topicId 变化清空草稿,替代原 key remount 重置机制
2026-08-12 09:16:38 +08:00
oudecheng
3910bc324b chore: 升级版本号至 0.3.5 并更新 CHANGELOG 2026-08-12 08:39:40 +08:00
oudecheng
29060315a3 feat(observability): 端到端可观测性整改,修复 trace_id 断链与指标配对
- 传播 trace_id:BusToolCallEmitter/SubAgentEmitter/processor 全链路设置

- AgentEnd 配对:补发 5 个 cancel/defensive 路径,闭合 AgentStart 指标

- LLM 计时修正:attempt_start 移入 retry 循环,排除退避等待时间

- /metrics auth:非 loopback 部署时纳入 Bearer token 校验

- recorder 复用:OnceLock 缓存 PrometheusHandle,热重启后不再返回 503

- 结构化日志:新增 tracing_ctx + JSON 日志格式支持
2026-08-12 08:26:42 +08:00
oudecheng
7cb170e0c2 fix(gateway): 压缩期间释放 session 锁,修复新建话题列表刷新延迟
压缩任务在 LLM 调用(2-5s)期间持有 session 锁,阻塞 create_session
命令获取锁执行 switch_topic,导致新建话题后列表刷新延迟。

将压缩重构为 4 阶段:
1. 短暂持锁:读取 history/compressor/store/session_id/provider_config
2. 释放锁:LLM 压缩调用(2-5s)
3. 释放锁:DB 写入(store 为 Arc,DB 层自带事务保护)
4. 短暂持锁:reload 内存历史

并发安全:同 topic 由调用方的 per-topic serial lock 保证串行;
不同 topic 完全并行不受影响。
2026-08-12 08:24:46 +08:00
oudecheng
77a0eac2c8 fix(mcp): 修复前端空输入重置为不超时,增加超时路径单元测试
P3: 清空输入框时回退到默认值 300 而非 0;新增两个 tokio 测试覆盖有超时和无超时(error pass-through)路径
2026-08-11 23:07:00 +08:00
oudecheng
3e97ed903c feat(mcp): 为 MCP 工具调用增加超时保护,默认 5 分钟
在 McpToolWrapper 适配层用 tokio::time::timeout 包裹 call_tool,防止外部 MCP server 挂起导致 agent loop 无限阻塞。超时时间通过 config.mcp_tool_timeout_secs 配置(默认 300 秒,0=不超时),前端 McpTab 设置页提供输入框。
2026-08-11 21:53:07 +08:00
62 changed files with 4239 additions and 286 deletions

2
.gitignore vendored
View File

@ -40,3 +40,5 @@ node_modules
logs logs
dist dist
.trae .trae
.opencode/
artifacts/

133
Cargo.lock generated
View File

@ -466,6 +466,15 @@ dependencies = [
"crossbeam-utils", "crossbeam-utils",
] ]
[[package]]
name = "crossbeam-epoch"
version = "0.9.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2d6914041f254d6e9176c01941b21115dcfb7089e55135a35411081bd106ef3f"
dependencies = [
"crossbeam-utils",
]
[[package]] [[package]]
name = "crossbeam-utils" name = "crossbeam-utils"
version = "0.8.21" version = "0.8.21"
@ -879,6 +888,25 @@ dependencies = [
"yaml-rust2", "yaml-rust2",
] ]
[[package]]
name = "h2"
version = "0.4.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6cb093c84e8bd9b188d4c4a8cb6579fc016968d14c99882163cd3ff402a4f155"
dependencies = [
"atomic-waker",
"bytes",
"fnv",
"futures-core",
"futures-sink",
"http",
"indexmap",
"slab",
"tokio",
"tokio-util",
"tracing",
]
[[package]] [[package]]
name = "hashbrown" name = "hashbrown"
version = "0.14.5" version = "0.14.5"
@ -937,6 +965,12 @@ version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
[[package]]
name = "hermit-abi"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c"
[[package]] [[package]]
name = "hex" name = "hex"
version = "0.4.3" version = "0.4.3"
@ -1004,6 +1038,7 @@ dependencies = [
"bytes", "bytes",
"futures-channel", "futures-channel",
"futures-core", "futures-core",
"h2",
"http", "http",
"http-body", "http-body",
"httparse", "httparse",
@ -1024,7 +1059,9 @@ dependencies = [
"http", "http",
"hyper", "hyper",
"hyper-util", "hyper-util",
"log",
"rustls", "rustls",
"rustls-native-certs",
"tokio", "tokio",
"tokio-rustls", "tokio-rustls",
"tower-service", "tower-service",
@ -1441,6 +1478,52 @@ version = "2.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79" checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79"
[[package]]
name = "metrics"
version = "0.23.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3045b4193fbdc5b5681f32f11070da9be3609f189a79f3390706d42587f46bb5"
dependencies = [
"ahash",
"portable-atomic",
]
[[package]]
name = "metrics-exporter-prometheus"
version = "0.15.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4f0c8427b39666bf970460908b213ec09b3b350f20c0c2eabcbba51704a08e6"
dependencies = [
"base64",
"http-body-util",
"hyper",
"hyper-rustls",
"hyper-util",
"indexmap",
"ipnet",
"metrics",
"metrics-util",
"quanta",
"thiserror 1.0.69",
"tokio",
"tracing",
]
[[package]]
name = "metrics-util"
version = "0.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4259040465c955f9f2f1a4a8a16dc46726169bca0f88e8fb2dbeced487c3e828"
dependencies = [
"crossbeam-epoch",
"crossbeam-utils",
"hashbrown 0.14.5",
"metrics",
"num_cpus",
"quanta",
"sketches-ddsketch",
]
[[package]] [[package]]
name = "meval" name = "meval"
version = "0.2.0" version = "0.2.0"
@ -1556,6 +1639,16 @@ dependencies = [
"autocfg", "autocfg",
] ]
[[package]]
name = "num_cpus"
version = "1.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b"
dependencies = [
"hermit-abi",
"libc",
]
[[package]] [[package]]
name = "once_cell" name = "once_cell"
version = "1.21.4" version = "1.21.4"
@ -1635,7 +1728,7 @@ dependencies = [
[[package]] [[package]]
name = "picobot" name = "picobot"
version = "0.3.3" version = "0.3.5"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"async-trait", "async-trait",
@ -1654,6 +1747,8 @@ dependencies = [
"iana-time-zone", "iana-time-zone",
"image", "image",
"libc", "libc",
"metrics",
"metrics-exporter-prometheus",
"meval", "meval",
"mime_guess", "mime_guess",
"parking_lot", "parking_lot",
@ -1710,6 +1805,12 @@ dependencies = [
"miniz_oxide", "miniz_oxide",
] ]
[[package]]
name = "portable-atomic"
version = "1.15.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "05c8b63e8d9609db387f0324918f81d68fe27748f084ef092fb35954d0539a85"
[[package]] [[package]]
name = "potential_utf" name = "potential_utf"
version = "0.1.5" version = "0.1.5"
@ -1796,6 +1897,21 @@ version = "0.1.29"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e0c5ccf5294c6ccd63a74f1565028353830a9c2f5eb0c682c355c471726a6e3f" checksum = "e0c5ccf5294c6ccd63a74f1565028353830a9c2f5eb0c682c355c471726a6e3f"
[[package]]
name = "quanta"
version = "0.12.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f3ab5a9d756f0d97bdc89019bd2e4ea098cf9cde50ee7564dde6b81ccc8f06c7"
dependencies = [
"crossbeam-utils",
"libc",
"once_cell",
"raw-cpuid",
"wasi",
"web-sys",
"winapi",
]
[[package]] [[package]]
name = "quick-error" name = "quick-error"
version = "2.0.1" version = "2.0.1"
@ -1947,6 +2063,15 @@ version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69"
[[package]]
name = "raw-cpuid"
version = "11.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "498cd0dc59d73224351ee52a95fee0f1a617a2eae0e7d9d720cc622c73a54186"
dependencies = [
"bitflags",
]
[[package]] [[package]]
name = "redox_syscall" name = "redox_syscall"
version = "0.5.18" version = "0.5.18"
@ -2562,6 +2687,12 @@ version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649" checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649"
[[package]]
name = "sketches-ddsketch"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85636c14b73d81f541e525f585c0a2109e6744e1565b5c1668e31c70c10ed65c"
[[package]] [[package]]
name = "slab" name = "slab"
version = "0.4.12" version = "0.4.12"

View File

@ -1,6 +1,6 @@
[package] [package]
name = "picobot" name = "picobot"
version = "0.3.3" version = "0.3.5"
edition = "2024" edition = "2024"
[lints.rust] [lints.rust]
@ -41,6 +41,8 @@ prost = "0.14"
tracing = "0.1" tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] } tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
tracing-appender = "0.2" tracing-appender = "0.2"
metrics = "0.23"
metrics-exporter-prometheus = "0.15"
anyhow = "1.0" anyhow = "1.0"
chrono = { version = "0.4", features = ["serde"] } chrono = { version = "0.4", features = ["serde"] }
chrono-tz = "0.10" chrono-tz = "0.10"

View File

@ -0,0 +1,588 @@
# 异步子 Agent 方案设计
> 版本: v8 (最终版)
> 日期: 2026-08-12
> 状态: 设计定稿
## 一、核心思路
子 agent 执行与父 agent 流程解耦,通过**双通道**实现异步通信:
- **子代理结果**走独立队列 `sub_done_q`(直达 wait不绕 SQLite → bus → reload
- **用户消息**走现有 `bus`(保持现有机制,零 channel 适配器改动)
wait 工具通过 **释放锁 + select! 等待 + 重新获取锁** 实现真等待bus 消息能在 wait 期间通过 `is_waiting` 分流注入 history 并唤醒 wait。
### 核心机制
```
sub_done_q独立队列 ← 后台子代理完成时注入 → wait select! 直达消费
bus现有机制 ← ws.rs 用户消息注入 → process_one 分流注入 + wakeup 唤醒 wait
```
## 二、关键设计决策
| 决策点 | 选择 | 理由 |
|---|---|---|
| 子 agent 执行方式 | 异步tokio::spawn | 消除最大阻塞源 |
| 子代理结果通道 | sub_done_q 独立队列 | 直达 wait不绕 SQLite → bus → reload |
| 用户消息通道 | 复用 bus | 现有机制channel 零改动 |
| wait 语义 | 释放锁 + select! + 重获取锁 | 真等待,无死锁,无 TOCTOU |
| is_waiting 判断 | 持锁后判断 | 原子操作,无 TOCTOU |
| 完成回调内容 | 当前结果 + 未完成子代理 id 列表 | LLM 据此决策下一步 |
| task tool_result | running + 引导提示(只提 wait | LLM 知道应调 wait |
| 嵌套层 | 仅顶层异步 | 避免复杂度爆炸 |
| 模式1中断 | 不做 | 复杂度高,收益低 |
## 三、数据结构
### 1. SessionHistory 扩展(核心)
```rust
// src/gateway/session_history.rs
pub(crate) struct SessionHistory {
// 现有字段
topic_histories: HashMap<String, Vec<ChatMessage>>,
chat_topic_ids: HashMap<String, String>,
compression_in_flight: HashSet<String>,
topic_serial_locks: HashMap<String, Arc<tokio::sync::Mutex<()>>>,
conversations: Arc<dyn ConversationRepository>,
skill_events: Arc<dyn SkillEventRepository>,
// 新增per-topic 子代理完成队列
sub_done_queues: HashMap<String, mpsc::Sender<SubagentResult>>,
// 新增per-topic wait 唤醒信号
wait_wakeups: HashMap<String, Arc<tokio::sync::Notify>>,
// 新增per-topic 等待状态
waiting_flags: HashMap<String, bool>,
}
```
**生命周期与 topic_serial_locks 完全一致**
- 创建:`topic_serial_lock(topic_id)` 时同步创建
- 驱逐:`evict_inactive_if_needed` 时同步移除
- 清理topic 删除时同步清理
### 2. SubagentResult 结构
```rust
struct SubagentResult {
task_id: String,
status: SubagentStatus, // completed/failed/timeout/cancelled
output: String, // 当前子代理的输出(与 task 工具返回格式一致)
pending_task_ids: Vec<String>, // 未完成的子代理 id 列表
}
```
**关键设计**:每个完成回调都带上 `pending_task_ids`LLM 据此判断全局进度。
### 3. pending_subagents 表(新增)
```sql
CREATE TABLE pending_subagents (
task_id TEXT PRIMARY KEY,
parent_session_id TEXT NOT NULL,
parent_topic_id TEXT NOT NULL,
parent_chat_id TEXT NOT NULL,
parent_channel TEXT NOT NULL,
def_name TEXT,
spawned_at INTEGER NOT NULL,
status TEXT NOT NULL DEFAULT 'running' -- running/completed/failed/interrupted/cancelled
);
```
## 四、执行流程
### 阶段 ① 发起 task
```
用户消息 → bus → process_one → lock(topic) → agent.process()
→ LLM 调用 task 工具
→ TaskTool::execute():
① INSERT pending_subagents (status='running')
② tokio::spawn 子 agent后台执行
③ 立即返回 tool_result:
"running, task_id=t1。
调用 wait_for_subagents(timeout_secs) 等待子代理完成并获取结果。"
→ LLM 看到 running + 引导,自主决策下一步
```
**LLM 的 3 种可能行为**
- 调用 wait_for_subagents主动等待— 主路径
- 调用其他工具(继续工作)— 罕见
- 生成最终回答(不调 wait— 后端兜底
### 阶段 ② 后台子 agent 执行
```
tokio::spawn 独立运行:
→ SubAgentEmitter 实时推消息流给前端
→ Semaphore 限流(默认 8 并发)
→ 执行完成或失败或超时
```
### 阶段 ③ 子 agent 完成回调(直达 sub_done_q
```
后台 task 结束:
① 查询 pending_subagents WHERE parent_topic_id=? AND status='running'
→ 得到未完成列表 ["t2", "t3"]
② sub_done_q.send(SubagentResult{
task_id: "t1",
status: "completed",
output: "{与 task 工具一致的输出格式}",
pending_task_ids: ["t2", "t3"]
})
③ UPDATE pending_subagents SET status='completed' WHERE task_id='t1'
```
**关键**:子代理结果**不写 SQLite、不入 bus**,直接通过 sub_done_q 传递给 wait。
**顺序保证**:查询未完成 → send → UPDATE
### 阶段 ④ wait 工具(释放锁 + select! + 重获取锁)
```
LLM 调用 wait_for_subagents(timeout_secs):
① 查询 pending_subagents
→ 无 running → 返回 "无需等待",继续循环
→ 有 running → 进入等待
② 释放锁,标记等待
drop(lock_guard)
session.set_waiting(topic_id, true)
③ select! 等待(不持锁)
select! {
result = sub_done_q.recv() => Event::Subagent(result),
_ = wakeup.notified() => Event::UserMessage,
_ = timeout(60s) => Event::Timeout,
}
④ 重新获取锁,清除标记(顺序重要:先获取锁,再清除)
let lock_guard = lock(topic_id).await
session.set_waiting(topic_id, false)
⑤ 返回 tool_result
Subagent(r) => "子代理 {task_id} 完成: {output}。未完成: {pending}"
UserMessage => "有新用户消息到达"
Timeout => "等待超时。未完成子代理: {pending}"
```
### 阶段 ⑤ process_one 分流wait 期间注入用户消息)
```rust
async fn process_one(&self, inbound: InboundMessage) {
let guard = lock(topic_id).await;
if session.is_waiting(topic_id) {
// wait 期间:只注入消息到 history不启动 process
session.inject_to_history(inbound).await; // INSERT + 更新内存 history
drop(guard);
session.wakeup(topic_id); // 唤醒 wait
return;
}
// 正常处理topic 空闲或 agent 已退出)
agent.process().await;
}
```
### 阶段 ⑥ process 退出兜底
```
process 退出前:
if waited=true:
→ 不发 ExecutionCompleted等 bus 消息触发新轮)
else if pending_subagents 有 running:
→ 不发 ExecutionCompletedLLM 没调 wait 但有 pending
else:
→ 发 ExecutionCompleted
```
## 五、wait 工具的完整设计
### 工具定义
```rust
// 工具名wait_for_subagents
// 参数timeout_secs可选默认 60s
// 描述:等待子代理完成或用户消息。用于有 pending 子代理时进入等待状态。
wait_for_subagents(timeout_secs: Option<u64>) -> ToolResult {
let timeout = Duration::from_secs(timeout_secs.unwrap_or(60));
let pending = query_pending_task_ids(topic_id);
if pending.is_empty() {
return ToolResult {
output: "无 pending 子代理,无需等待".to_string(),
metadata: { wait_marker: false }
};
}
// 释放锁,标记等待
drop(lock_guard);
session.set_waiting(topic_id, true);
// 等待(不持锁)
let event = select! {
result = sub_done_q.recv() => Event::Subagent(result),
_ = wakeup.notified() => Event::UserMessage,
_ = sleep(timeout) => Event::Timeout,
};
// 重新获取锁,清除标记(顺序重要)
let lock_guard = lock(topic_id).await;
session.set_waiting(topic_id, false);
// 先 drain sub_done_q 积压(处理多子代理同时完成)
match event {
Event::Subagent(result) => {
format_subagent_result(result)
}
Event::UserMessage => {
"有新用户消息到达".to_string()
}
Event::Timeout => {
let pending = query_pending_task_ids(topic_id);
format!("等待超时。未完成子代理: {:?}", pending)
}
}
}
```
### wait 的三种返回场景
| 事件 | 返回内容 | LLM 行为 |
|---|---|---|
| 子代理完成 | "子代理 t1 完成: {output}。未完成: [t2, t3]" | 处理结果,看 pending 决定继续 wait 或综合 |
| 用户消息到达 | "有新用户消息到达" | LLM 下一轮看到注入的 user 消息 |
| 超时 | "等待超时。未完成子代理: [t1, t2]" | 决定继续等或放弃 |
## 六、两个通道的职责(核心设计)
### sub_done_qwait 专属消费
```
后台子代理完成 → sub_done_q.send(result) ← 不获取锁,直达 wait
wait select! → 立即收到 → 返回给 LLM
```
**不经过 SQLite、不经过 bus**,直接队列传递。
### bus用户消息wait 期间分流注入
```
用户消息 → bus.publish_inbound(UserMessage)
→ process_one → lock(topic)
├─ is_waiting=true → 注入 history + wakeup 唤醒 wait
└─ is_waiting=false → 正常启动 process
```
**wait 期间**:用户消息持锁注入 historywakeup 唤醒 wait。
**非 wait 期间**:正常走 process_one。
## 七、LLM 看到的信息流
| 时机 | LLM 看到的内容 |
|---|---|
| task 调用后 | `tool_result("running, task_id=t1。调用 wait_for_subagents 等待...")` |
| wait 调用后 | `tool_result("已进入等待2 个子代理运行中: [t1,t2]")` |
| t1 完成wait 返回) | `tool_result("子代理 t1 完成: {output}。未完成: [t2]")` |
| 用户消息wait 返回) | `tool_result("有新用户消息到达")` + 下轮 `user("用户消息")` |
| t2 完成wait 返回) | `tool_result("子代理 t2 完成: {output}。未完成: []")` |
LLM 通过 `pending_task_ids` 能判断:
- `[]` 空列表 → 全部完成,综合回答
- `["t3"]` 非空 → 还有未完成的,继续调 wait
## 八、消息的完整路径
### 子代理结果
```
后台 task 完成 → sub_done_q.send(SubagentResult)
→ wait select! 立即收到 → 返回 tool_result 给 LLM
```
**直达,不绕路**。
### 用户消息
```
所有 channel 消息 → bus.publish_inbound(UserMessage)
→ process_one → lock(topic)
├─ is_waiting=true → 注入 history + wakeup 唤醒 wait
│ → wait 返回 "有新用户消息到达"
│ → LLM 下一轮看到注入的 user 消息
└─ is_waiting=false → 正常启动 process
```
**两条路径职责清晰,互不干扰**。
## 九、三个边界问题的解法
### 边界 1wakeup vs timeout 竞态
```
timeout 和 wakeup 同时触发 → select! 随机选一个
如果 timeout 赢:
- 用户消息已注入 historyprocess_one 在锁内完成注入)
- wait 返回 "超时"
- LLM 下一轮 reload history 会看到用户消息
- 不会丢失
```
**解法**无需特殊处理消息已持久化reload 能读到。
### 边界 2is_waiting 清除时机
```
错误顺序:清除 is_waiting → 获取锁
→ 中间窗口 process_one 看到 is_waiting=false → 启动 process → 死锁
正确顺序:获取锁 → 清除 is_waiting
→ 持锁后才清除process_one 在锁外看到 is_waiting=true → 走注入路径
```
**解法**`let guard = lock().await; set_waiting(false);` 顺序保证。
### 边界 3多条用户消息
```
wait 释放锁后3 条用户消息依次到达:
msg1 → lock → 注入 → drop → wakeup
msg2 → lock → 注入 → drop → wakeup
msg3 → lock → 注入 → drop → wakeup
wakeup 是 Notify3 次 notify 只存储 1 个
wait 被唤醒 1 次,但 history 已有 3 条消息
wait 返回 "有新用户消息到达"
LLM 下一轮看到 3 条 user 消息
```
**解法**Notify 合并是正确行为history 完整。
## 十、时序场景表现
### 场景 1多子代理并发完成
```
① task(t1, t2, t3) → running
② wait → 释放锁 → select!
③ t1, t2, t3 几乎同时完成 → sub_done_q: [t1, t2, t3]
④ wait select 收到 t1 → 返回 "t1 完成。未完成: [t2, t3]"
⑤ LLM 处理 t1 → 再调 wait
wait 先 try_recv → 立即拿到 t2 → 返回 "t2 完成。未完成: [t3]"
⑥ LLM 处理 t2 → 再调 wait
wait 先 try_recv → 立即拿到 t3 → 返回 "t3 完成。未完成: []"
⑦ LLM 综合 t1+t2+t3 回答
```
**3 轮 wait但无延迟try_recv 立即返回积压)**。
### 场景 2wait 期间用户发消息
```
① task(t1) → running → wait → 释放锁 → select!
② 用户发消息 → bus → process_one → lockwait 已释放,获取成功)
→ is_waiting=true → 注入 history + wakeup
③ wait 被 wakeup 唤醒 → 重新获取锁 → 返回 "有新用户消息到达"
④ LLM 看到 [wait_result, user("用户消息")] → 处理用户消息
```
**用户消息即时响应wait 被唤醒**。
### 场景 3子代理完成 + 用户消息同时到达
```
① task(t1) → running → wait → 释放锁 → select!
② t1 完成 → sub_done_q
用户发消息 → bus → process_one → 注入 + wakeup
③ select! 随机选一个:
- 选 sub_done_q → 返回 "t1 完成" → LLM 下一轮看到 user 消息
- 选 wakeup → 返回 "有新用户消息" → LLM 下一轮看到 t1 结果try_recv
```
**两种情况都不丢失消息**。
### 场景 4父 agent 生成最终回答(不调 wait
```
① task(t1) → running
② LLM 生成最终回答(无 tool_calls→ process 退出
③ 退出前检查: pending 有 running → 不发 ExecutionCompleted
④ t1 完成 → sub_done_q.send无人消费留在队列
→ 触发新轮 process_one通过 bus 或定期检查)
⑤ 新轮 LLM 看到 [自己之前的回答] → try_recv sub_done_q → 处理 t1 结果
```
**注意**:此场景需要额外机制触发新轮 processsub_done_q 有残留时)。可通过:
- 后台 task 完成后同时 publish_inbound(bus) 作为触发信号
- 或 wait 退出兜底检查 sub_done_q
### 场景 5wait 超时
```
① task(t1) → running → wait(60s) → 释放锁 → select!
② 60s 内无事件 → timeout 触发
③ wait 重新获取锁 → 返回 "等待超时。未完成: [t1]"
④ LLM 决定继续等(再调 wait或放弃
```
## 十一、关键机制
### 1. 队列与状态生命周期管理
```rust
// session_history.rs
/// 获取或创建该 topic 的 sub_done_q sender
pub(crate) fn sub_done_queue(&mut self, topic_id: &str) -> mpsc::Sender<SubagentResult> {
self.sub_done_queues
.entry(topic_id.to_string())
.or_insert_with(|| {
let (tx, _rx) = mpsc::channel(32);
tx
})
.clone()
}
/// 获取或创建该 topic 的 wakeup
pub(crate) fn wait_wakeup(&mut self, topic_id: &str) -> Arc<Notify> {
self.wait_wakeups
.entry(topic_id.to_string())
.or_insert_with(|| Arc::new(Notify::new()))
.clone()
}
/// 设置等待状态
pub(crate) fn set_waiting(&mut self, topic_id: &str, waiting: bool) {
self.waiting_flags.insert(topic_id.to_string(), waiting);
}
/// 检查等待状态
pub(crate) fn is_waiting(&self, topic_id: &str) -> bool {
self.waiting_flags.get(topic_id).copied().unwrap_or(false)
}
// evict_inactive_if_needed 中同步清理
fn evict_inactive_if_needed(&mut self) {
// ... 现有逻辑
if let Some(tid) = to_evict.cloned() {
self.topic_histories.remove(&tid);
self.topic_serial_locks.remove(&tid);
self.sub_done_queues.remove(&tid); // 新增
self.wait_wakeups.remove(&tid); // 新增
self.waiting_flags.remove(&tid); // 新增
}
}
```
### 2. pending_subagents 状态管理
| 状态 | 含义 |
|---|---|
| running | 后台执行中 |
| completed | 正常完成 |
| failed | 执行失败 |
| interrupted | 进程崩溃,启动扫描标记 |
| cancelled | 用户取消 |
### 3. 崩溃恢复
**启动扫描**
```sql
UPDATE pending_subagents SET status='interrupted' WHERE status='running';
```
**history 加载时**
- 遇到 `tool_result("running, task_id=...")` → 查 pending_subagents
- status=interrupted → 替换为"子代理因重启中断,请决定是否重新发起"
**两步操作之间崩溃**(查询未完成 → send → UPDATE
- 崩溃在任何点:启动扫描标记 interruptedhistory 加载时替换占位
- 最坏情况sub_done_q 消息丢失wait 未消费),但 pending 表状态可恢复
### 4. 取消传播
- `task_id → CancellationToken` 映射
- 用户取消 → 遍历 pending → 触发 cancel_token
- 子代理退出 → sub_done_q.send(SubagentResult{status:"cancelled"})
### 5. 去重
- DefaultSubAgentRuntime 维护 `completed_tasks: HashSet<task_id>`
- 入队前检查,已入队的不再入
### 6. topic_serial_lock 一致性
- 用户消息走 bus用 topic_id 作 lock_key
- 子代理结果走 sub_done_q不需要 lock直达 wait
- wait 释放锁后bus 的 process_one 能获取锁注入消息
## 十二、改动范围
| 文件 | 改动 |
|---|---|
| `session_history.rs` | 新增 sub_done_q + wait_wakeup + waiting_flagsper-topic+ 生命周期 |
| `runtime.rs` | spawn 异步化 + 完成回调(含 pending_task_ids注入 sub_done_q |
| `task_tool.rs`(新) | TaskTool 返回 running + 引导提示 + spawn |
| `wait_tool.rs`(新) | wait 释放锁 + select! + 重获取锁 + 返回 pending 信息 |
| `agent_loop.rs` | 传递 lock_guard 给 wait 工具(支持释放/重获取) |
| `processor.rs` | process_one 分流is_waiting 检查 + 注入 + wakeup |
| `db.rs` | pending_subagents 表 + 启动扫描 |
| `history.rs` | running 占位替换 |
| `cancel.rs` | 取消传播 |
**预估总改动:~800 行**
## 十三、与之前方案的对比
| 维度 | v6bus + break | v8双通道 + wait 释放锁) |
|---|---|---|
| 子代理结果通道 | bus绕 SQLite → reload | **sub_done_q 直达** ✅ |
| 用户消息通道 | bus | bus |
| wait 语义 | break 退出 | **select! 真等待** ✅ |
| wait 期间用户消息 | 等锁agent 退出后) | **即时注入 + wakeup** ✅ |
| is_waiting 判断 | 不需要 | 需要(持锁后判断,无 TOCTOU |
| TOCTOU 风险 | 无 | **无**(持锁后判断) |
| 死锁风险 | 无 | **无**wait 释放锁) |
| LLM 调用次数 | 2batch/ N+2 | **2-N** |
| process 退出 | 每次都退出 reload | wait 不退出,连续推理 |
| 上下文连贯 | 每轮 reload | **同 process 内连续** ✅ |
| 实现复杂度 | 低 | 中(~20 行新增 + 边界解法) |
| channel 适配器改动 | 零 | **零** ✅ |
## 十四、不做的事
- ❌ 模式1中断插入
- ❌ user_msg_q用户消息走 bus
- ❌ 嵌套层异步
- ❌ 子代理结果走 bus改回 sub_done_q
- ❌ wait 不持锁(改为释放锁 + select + 重获取)
## 十五、方案定位总结
| 机制 | 定位 | 通道 |
|---|---|---|
| wait 释放锁 + select! | **主路径**(真等待,子代理直达) | sub_done_q |
| process_one 分流 + wakeup | **用户消息路径**(注入 + 唤醒) | bus |
| process 退出兜底pending 检查) | **安全网**LLM 不调 wait 时) | - |
三机制分工清晰:
- wait select! 是核心(子代理结果直达 + 用户消息 wakeup
- process_one 分流是用户消息路径wait 期间注入 + 唤醒)
- 退出兜底是安全网(防止 LLM 不调 wait 时丢消息)
## 十六、配置说明
```json
{
"subagent": {
"max_concurrent": 8,
"spawn_timeout_secs": 300,
"wait_default_timeout_secs": 60
}
}
```
| 配置项 | 默认值 | 说明 |
|---|---|---|
| max_concurrent | 8 | 并发子 agent 数量Semaphore |
| spawn_timeout_secs | 300 | 子 agent 执行超时 |
| wait_default_timeout_secs | 60 | wait 工具默认超时 |

View File

@ -2,6 +2,142 @@
本文件记录 Picobot 各版本的显著变更,遵循 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/) 风格。 本文件记录 Picobot 各版本的显著变更,遵循 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/) 风格。
## [0.3.5] - 2026-08-12
较 [0.3.4] 的 7 个 commit 迭代,聚焦 **可观测性**、**并发性能** 与 **外部集成健壮性** 三大方向。
### 新增功能
#### MCP 工具调用超时保护
- 在 `McpToolWrapper` 适配层用 `tokio::time::timeout` 包裹 `call_tool`,防止外部 MCP server 挂起导致 agent loop 无限阻塞。
- 超时时间通过 `config.mcp_tool_timeout_secs` 配置(默认 300 秒0=不超时)。
- 前端 McpTab 设置页提供输入框。
#### 端到端可观测性整改
- **trace_id 全链路传播**`BusToolCallEmitter` / `SubAgentEmitter` / `processor` 全链路设置 trace_id。
- **AgentEnd 指标配对**:补发 5 个 cancel/defensive 路径的 AgentEnd 事件,闭合 AgentStart 指标。
- **LLM 计时修正**`attempt_start` 移入 retry 循环,排除退避等待时间。
- **`/metrics` auth**:非 loopback 部署时纳入 Bearer token 校验。
- **recorder 复用**`OnceLock` 缓存 `PrometheusHandle`,热重启后不再返回 503。
- **结构化日志**:新增 `tracing_ctx` + JSON 日志格式支持。
### 修复
#### 压缩期间 session 锁阻塞新建话题
- 压缩任务在 LLM 调用2-5s期间持有 session 锁,阻塞 `create_session` 命令获取锁执行 `switch_topic`,导致新建话题后列表刷新延迟。
- 将压缩重构为 4 阶段:
1. 短暂持锁:读取 history/compressor/store/session_id/provider_config
2. 释放锁LLM 压缩调用2-5s
3. 释放锁DB 写入store 为 ArcDB 层自带事务保护)
4. 短暂持锁reload 内存历史
- 并发安全:同 topic 由调用方的 per-topic serial lock 保证串行;不同 topic 完全并行不受影响。
#### MCP 前端空输入重置为不超时
- 清空输入框时回退到默认值 300 而非 00 表示不超时,与用户预期不符)。
- 新增两个 tokio 测试覆盖有超时和无超时error pass-through路径。
#### 移除 ChatContainer key remount 修复新建话题列表刷新延迟
- 话题切换时 `key={selectedTopic}` 导致整个 `ChatContainer` 子树卸载重建,触发 `ExpertSelector` / `ModelSelector` 重新挂载并发起 3 次冗余 HTTP 请求(`getSelectedExpert` / `listModelOptions` / 模型选择刷新),引入 200-500ms 延迟。
- 改用 `topicId` prop 透传话题 ID
- `App.tsx`:移除 key`viewKey` 纳入 `selectedTopic` 保持各话题独立滚动位置。
- `ChatContainer`:新增 `topicId` prop 透传给 `MessageInput`
- `MessageInput`:监听 `topicId` 变化清空草稿,替代原 key remount 重置机制。
#### Panic 安全增强
- `agent_loop.rs` / `processor.rs`:用 `catch_unwind` 隔离工具执行和消息处理的 panic归一化为错误返回LLM 可见错误并自我纠正,防止用户消息被静默丢弃。
- `utils.rs`:新增 `panic_payload_message()` 从 panic payload 中提取可读消息,支持 `&str` / `String` / 其他类型降级。
- `bus/message.rs`:修复 `sanitize_incomplete_tool_call_sequences` 两阶段删除索引未全局排序导致的越界 panicPhase 1 降序 + Phase 1.5 升序 → 合并后必须重新排序)。
#### 计算器健壮性
- 拒绝 `"NaN"` / `"inf"` 等非有限输入,防止 `sort_by``partial_cmp().unwrap()` panic。
- 修复阶乘溢出:上限从 170 降至 3435! 超出 `u128::MAX`),改用 `checked_mul` 替代 unchecked 乘法。
- `evaluate` 表达式拒绝非有限结果(如 `1/0` → inf、`0/0` → NaN
- 新增 7 个 tokio 测试覆盖所有边界场景。
### 测试
- MCP 超时路径新增 2 个 tokio 单元测试。
---
## [0.3.4] - 2026-08-10
较 [0.3.3] 的 9 个 commit 迭代,聚焦 **安全加固**、**健壮性防御** 与 **代码质量优化** 三大方向。
### 新增功能
#### 顶栏显示后端版本号
- 复用现有 `/health` 端点返回的 `CARGO_PKG_VERSION`,避免在 `package.json` 重复维护版本号。
- 前端挂载时拉取一次,失败则不显示徽标。
#### 文件工具路径限制(最小权限原则)
- 默认行为:文件读写编辑限制在当前工作目录内。
- 新增配置项:`tools.allowed_dirs`(白名单目录列表)+ `tools.file_access_unrestricted`opt-in 全局放开)。
- 共享 `resolve_file_path()` 函数,使用 `canonicalize()` 防御符号链接与 `..` 路径遍历。
- 前端:设置页 > 工具标签新增"文件访问"卡片,含开关与目录标签编辑器。
- 向后兼容:旧配置默认 cwd 限制(`serde(default)`)。
### 安全修复
#### Shell 会话生命周期管理
- `GatewayState` 持有 `Arc<ShellSessionManager>` 引用,网关关闭时(`ctrl_c` 和重启两条路径)调用 `shutdown()`
- 新增防御性 `Drop` impl泄露会话时打 warn 日志。
- 子进程 kill 添加 5s 超时,防止无限阻塞。
#### HTTP 工具 OOM 防护
- `WebFetchTool` / `HttpRequestTool` 实现流式读取响应体(`futures_util::StreamExt`),防止大响应撑爆内存。
- 检查 `Content-Length` 头,分块读取并累计校验大小。
- 默认限制:`WebFetch` 200KB`HttpRequest` 4MB。
#### TimeTool 重复类型转换修复
- 移除 `time.rs` 中重复的 `u32::try_from` 调用(复制粘贴错误)。
#### 消除 16 处 unreachable! panic 风险
- 15 个 command handler 文件中 `_ => unreachable!()` 替换为 `_ => Err(CommandError::new(...))`,防止 `Command` 枚举新增变体时运行时 panic。
#### 配置占位符校验
- 新增 `validate_no_unresolved_placeholders()`,拒绝包含未解析 `<ENV_VAR>``\` 占位符的配置。
- 防止字面占位符字符串被当作真实 API key 发送到 LLM provider难以诊断的 401 根因)。
- 首次运行跳过校验(`create_default_config` 模板场景)。
### 修复
#### 切换话题后滚动到最新消息
- 修复切换话题后不滚动到最新消息的问题。
- 跟随意图仅由用户输入wheel/touch/key推翻程序触发的滚动不会打断用户浏览。
- 将虚拟化 `totalSize` 纳入滚动依赖,使测量收敛到真实底部。
#### WebSocket 重连竞态修复
- 修复 URL 切换时的 WebSocket 重连竞态问题。
- 事件处理引入代际同一性守卫,孤儿 socket 事件整体忽略。
- disconnect 恰好触发一次 `onDisconnect`,避免重复清理。
#### 删除话题竞态修复
- 修复删除选中话题时的竞态问题。
- 乐观移除本地话题并以 `null` 作为唯一无选择哨兵。
- 杜绝基于过期列表重选已删除话题。
### 重构
#### 8 处 quick wins
- `shared.rs` 提取 channel/cli 适配器公共逻辑,消除 124 行重复代码。
- `http_utils.rs` 提取 HTTP 响应读取公共逻辑,`http_request.rs``web_fetch.rs` 共享实现。
- `bash.rs` / `shell_session.rs` 简化冗余逻辑。
- `gateway/mod.rs` 优化路由分支。
- `storage/mod.rs` 精简查询逻辑。
- `mcp/client.rs` 增强错误处理。
### 内部改进
- 修复 release 构建编译器警告:补全 `regex::Captures` 显式生命周期标注,`cfg(debug_assertions)` 条件化仅 debug 使用的变量,`AnthropicProvider::llm_timeout_secs` 添加 `cfg_attr allow(dead_code)`
- 飞书 channel 时间戳去重改用 `crate::utils::current_timestamp()`,消除 `unwrap()` panic 点并统一日志格式。
- 批量更新依赖 patch 版本chrono, clap, anyhow, bytes 等),无 breaking change。
### 测试
- `cancel_manager.rs` 新增 8 个单元测试,覆盖 register/cancel/remove/overwrite/list/cancel_all/drop-safety 核心路径。
- `session_pool.rs` 新增 3 个单元测试,覆盖 `is_scheduler_chat_id` 路由逻辑。
- 配置占位符校验新增 11 个单元测试,覆盖空值、小写、数字、去重、多占位符、首次运行、加载集成等边缘场景。
---
## [0.3.3] - 2026-08-07 ## [0.3.3] - 2026-08-07
较 [0.3.2] 的 12 个 commit 迭代,聚焦 **前端渲染性能飞跃**、**流式通信健壮性** 与 **代码质量治理** 三大方向。 较 [0.3.2] 的 12 个 commit 迭代,聚焦 **前端渲染性能飞跃**、**流式通信健壮性** 与 **代码质量治理** 三大方向。
@ -380,6 +516,8 @@
- 前端静态文件嵌入二进制。 - 前端静态文件嵌入二进制。
- React Web UI 前端界面。 - React Web UI 前端界面。
[0.3.5]: https://github.com/picobot/picobot/compare/v0.3.4...v0.3.5
[0.3.4]: https://github.com/picobot/picobot/compare/v0.3.3...v0.3.4
[0.3.3]: https://github.com/picobot/picobot/compare/v0.3.2...v0.3.3 [0.3.3]: https://github.com/picobot/picobot/compare/v0.3.2...v0.3.3
[0.3.2]: https://github.com/picobot/picobot/compare/v0.3.1...v0.3.2 [0.3.2]: https://github.com/picobot/picobot/compare/v0.3.1...v0.3.2
[0.3.1]: https://github.com/picobot/picobot/compare/v0.3.0...v0.3.1 [0.3.1]: https://github.com/picobot/picobot/compare/v0.3.0...v0.3.1

View File

@ -15,6 +15,7 @@ use crate::text::{char_count, take_prefix_chars, take_suffix_chars};
use crate::tools::{ToolContext, ToolRegistry}; use crate::tools::{ToolContext, ToolRegistry};
use crate::utils::format_error_chain; use crate::utils::format_error_chain;
use async_trait::async_trait; use async_trait::async_trait;
use futures_util::FutureExt;
use std::borrow::Cow; use std::borrow::Cow;
use std::collections::{HashMap, VecDeque}; use std::collections::{HashMap, VecDeque};
use std::hash::{Hash, Hasher}; use std::hash::{Hash, Hasher};
@ -1068,13 +1069,13 @@ impl AgentLoop {
/// - `compaction_sink`: 压缩结果回写端(可选)。配置 `compressor` 后, /// - `compaction_sink`: 压缩结果回写端(可选)。配置 `compressor` 后,
/// 当 LLM 压缩被触发时通过此 sink 把压缩结果持久化到 DB。 /// 当 LLM 压缩被触发时通过此 sink 把压缩结果持久化到 DB。
/// 传 None 则即使配置了 compressor 也只改内存不回写。 /// 传 None 则即使配置了 compressor 也只改内存不回写。
#[tracing::instrument(skip(self, messages, system_prompt_context, compaction_sink), fields(history_len = messages.len(), max_iterations = self.max_iterations))]
pub async fn process( pub async fn process(
&self, &self,
mut messages: Vec<ChatMessage>, mut messages: Vec<ChatMessage>,
system_prompt_context: Option<&SystemPromptContext>, system_prompt_context: Option<&SystemPromptContext>,
compaction_sink: Option<&dyn CompactionSink>, compaction_sink: Option<&dyn CompactionSink>,
) -> Result<AgentProcessResult, AgentError> { ) -> Result<AgentProcessResult, AgentError> {
#[cfg(debug_assertions)]
tracing::debug!( tracing::debug!(
history_len = messages.len(), history_len = messages.len(),
max_iterations = self.max_iterations, max_iterations = self.max_iterations,
@ -1179,6 +1180,14 @@ impl AgentLoop {
) )
.await; .await;
// Emit AgentStart event for metrics (LLM 请求耗时/token 指标)
if let Some(ref observer) = self.observer {
observer.record_event(&ObserverEvent::AgentStart {
provider: self.provider.name().to_string(),
model: self.provider.model_id().to_string(),
});
}
// Set up streaming delta consumer // Set up streaming delta consumer
// Pre-generate the message ID so stream deltas and the final assistant // Pre-generate the message ID so stream deltas and the final assistant
// message share the same ID — this lets the front-end replace the // message share the same ID — this lets the front-end replace the
@ -1190,9 +1199,12 @@ impl AgentLoop {
let max_retries = self.runtime_config.max_retries as usize; let max_retries = self.runtime_config.max_retries as usize;
let mut response: Option<crate::providers::ChatCompletionResponse> = None; let mut response: Option<crate::providers::ChatCompletionResponse> = None;
// 记录最后一次尝试的耗时(不含重试退避),用于 AgentEnd 指标
let mut last_attempt_duration = std::time::Duration::ZERO;
'retry: for attempt in 0..=max_retries { 'retry: for attempt in 0..=max_retries {
// 每次重试重建 channel + consumer上次失败的 channel 可能已关闭。 // 每次重试重建 channel + consumer上次失败的 channel 可能已关闭。
let attempt_start = std::time::Instant::now();
let (delta_tx, mut delta_rx) = tokio::sync::mpsc::channel::<StreamDelta>(256); let (delta_tx, mut delta_rx) = tokio::sync::mpsc::channel::<StreamDelta>(256);
let consumer_handler = self.emitted_message_handler.clone(); let consumer_handler = self.emitted_message_handler.clone();
let consumer_task = tokio::spawn(async move { let consumer_task = tokio::spawn(async move {
@ -1222,6 +1234,16 @@ impl AgentLoop {
_ = self.cancel_signal() => { _ = self.cancel_signal() => {
drop(stream_callback); drop(stream_callback);
let _ = consumer_task.await; let _ = consumer_task.await;
// cancel 路径补发 AgentEnd保证指标配对闭合
last_attempt_duration = attempt_start.elapsed();
if let Some(ref observer) = self.observer {
observer.record_event(&ObserverEvent::AgentEnd {
provider: self.provider.name().to_string(),
model: self.provider.model_id().to_string(),
duration: last_attempt_duration,
tokens_used: None,
});
}
let cancel = Self::build_cancel_result(iteration, emitted_messages); let cancel = Self::build_cancel_result(iteration, emitted_messages);
self.emit_live_tool_call_message(cancel.final_response.clone()).await; self.emit_live_tool_call_message(cancel.final_response.clone()).await;
return Ok(cancel); return Ok(cancel);
@ -1242,10 +1264,12 @@ impl AgentLoop {
match llm_result { match llm_result {
Ok(resp) => { Ok(resp) => {
last_attempt_duration = attempt_start.elapsed();
response = Some(resp); response = Some(resp);
break 'retry; break 'retry;
} }
Err(e) => { Err(e) => {
last_attempt_duration = attempt_start.elapsed();
let error_text = e.to_string(); let error_text = e.to_string();
let can_retry = attempt < max_retries let can_retry = attempt < max_retries
&& !emitted.load(Ordering::SeqCst) && !emitted.load(Ordering::SeqCst)
@ -1265,6 +1289,16 @@ impl AgentLoop {
if self.cancel_token.is_some() { if self.cancel_token.is_some() {
tokio::select! { tokio::select! {
_ = self.cancel_signal() => { _ = self.cancel_signal() => {
// cancel 路径补发 AgentEnd保证指标配对闭合
last_attempt_duration = attempt_start.elapsed();
if let Some(ref observer) = self.observer {
observer.record_event(&ObserverEvent::AgentEnd {
provider: self.provider.name().to_string(),
model: self.provider.model_id().to_string(),
duration: last_attempt_duration,
tokens_used: None,
});
}
let cancel = Self::build_cancel_result(iteration, emitted_messages); let cancel = Self::build_cancel_result(iteration, emitted_messages);
self.emit_live_tool_call_message(cancel.final_response.clone()).await; self.emit_live_tool_call_message(cancel.final_response.clone()).await;
return Ok(cancel); return Ok(cancel);
@ -1283,6 +1317,15 @@ impl AgentLoop {
error_details = %format_error_chain(e.as_ref()), error_details = %format_error_chain(e.as_ref()),
"LLM request failed" "LLM request failed"
); );
// 错误分支补发 AgentEnd保证指标配对闭合
if let Some(ref observer) = self.observer {
observer.record_event(&ObserverEvent::AgentEnd {
provider: self.provider.name().to_string(),
model: self.provider.model_id().to_string(),
duration: last_attempt_duration,
tokens_used: None,
});
}
let assistant_message = let assistant_message =
ChatMessage::assistant(recoverable_llm_message(&error_text)); ChatMessage::assistant(recoverable_llm_message(&error_text));
emitted_messages.push(assistant_message.clone()); emitted_messages.push(assistant_message.clone());
@ -1298,11 +1341,34 @@ impl AgentLoop {
} }
} }
let response = response.ok_or_else(|| { // 防御性兜底retry 循环异常退出(正常不应发生)。
AgentError::Other( // 补发 AgentEnd 保证指标配对闭合,再返回错误。
"retry loop exited without setting response or returning".to_string(), let response = match response {
) Some(resp) => resp,
})?; None => {
if let Some(ref observer) = self.observer {
observer.record_event(&ObserverEvent::AgentEnd {
provider: self.provider.name().to_string(),
model: self.provider.model_id().to_string(),
duration: last_attempt_duration,
tokens_used: None,
});
}
return Err(AgentError::Other(
"retry loop exited without setting response or returning".to_string(),
));
}
};
// Emit AgentEnd event for metrics (LLM 请求耗时/token 指标)
if let Some(ref observer) = self.observer {
observer.record_event(&ObserverEvent::AgentEnd {
provider: self.provider.name().to_string(),
model: self.provider.model_id().to_string(),
duration: last_attempt_duration,
tokens_used: Some(response.usage.total_tokens as u64),
});
}
// Signal stream end if handler exists // Signal stream end if handler exists
let had_streaming = self.emitted_message_handler.is_some(); let had_streaming = self.emitted_message_handler.is_some();
@ -1381,10 +1447,29 @@ impl AgentLoop {
.await; .await;
// Execute tools and add results to messages // Execute tools and add results to messages
// 工具执行与取消信号竞速:取消时 drop join_all 或 sequential future //
// 未完成的工具调用被丢弃。 // 取消竞速策略:
let tool_results = if self.cancel_token.is_some() { // - 包含 wait_for_subagents 时:不使用 select!,直接 await execute_tools。
// 原因coordinator.wait() 在 select! 返回后有不可中断的清理逻辑
// (步骤 6-8重获取 serial_lock、回填 guard_slot、清除 is_waiting
// 若 agent_loop 的 select! 在清理期间 drop execute_tools
// 会导致 is_waiting=true 永久残留、guard_slot 为空、serial_lock 未持有,
// 后续所有用户消息走注入路径但 wakeup 无接收者 → 系统永久卡死。
// cancel 由 coordinator 内部 select! 的 cancel 分支处理,清理不会被打断。
//
// - 不含 wait_for_subagents 时:保留 select! 竞速,允许 /stop 中断
// 长时间运行的工具(如 MCP HTTP 请求)。
let has_wait_tool = response
.tool_calls
.iter()
.any(|tc| tc.name == "wait_for_subagents");
let tool_results = if self.cancel_token.is_some() && !has_wait_tool {
tokio::select! { tokio::select! {
biased;
results = self.execute_tools(&response.tool_calls) => {
results
}
_ = self.cancel_signal() => { _ = self.cancel_signal() => {
// 为所有 tool_calls 补充取消结果,避免孤立 assistant(tool_calls) // 为所有 tool_calls 补充取消结果,避免孤立 assistant(tool_calls)
for tool_call in &response.tool_calls { for tool_call in &response.tool_calls {
@ -1402,9 +1487,6 @@ impl AgentLoop {
self.emit_live_tool_call_message(cancel.final_response.clone()).await; self.emit_live_tool_call_message(cancel.final_response.clone()).await;
return Ok(cancel); return Ok(cancel);
} }
results = self.execute_tools(&response.tool_calls) => {
results
}
} }
} else { } else {
self.execute_tools(&response.tool_calls).await self.execute_tools(&response.tool_calls).await
@ -1715,8 +1797,20 @@ impl AgentLoop {
.await; .await;
let max_retries = self.runtime_config.max_retries as usize; let max_retries = self.runtime_config.max_retries as usize;
// Emit AgentStart for the summary LLM call
if let Some(ref observer) = self.observer {
observer.record_event(&ObserverEvent::AgentStart {
provider: self.provider.name().to_string(),
model: self.provider.model_id().to_string(),
});
}
// 记录最后一次尝试的耗时(不含重试退避)
let mut summary_last_attempt_duration = std::time::Duration::ZERO;
for attempt in 0..=max_retries { for attempt in 0..=max_retries {
// 最终 summary 调用也与取消信号竞速 // 最终 summary 调用也与取消信号竞速
let attempt_start = std::time::Instant::now();
let final_result: Result< let final_result: Result<
crate::providers::ChatCompletionResponse, crate::providers::ChatCompletionResponse,
Box<dyn std::error::Error + Send + Sync>, Box<dyn std::error::Error + Send + Sync>,
@ -1724,6 +1818,16 @@ impl AgentLoop {
if self.cancel_token.is_some() { if self.cancel_token.is_some() {
tokio::select! { tokio::select! {
_ = self.cancel_signal() => { _ = self.cancel_signal() => {
// cancel 路径补发 AgentEnd保证指标配对闭合
summary_last_attempt_duration = attempt_start.elapsed();
if let Some(ref observer) = self.observer {
observer.record_event(&ObserverEvent::AgentEnd {
provider: self.provider.name().to_string(),
model: self.provider.model_id().to_string(),
duration: summary_last_attempt_duration,
tokens_used: None,
});
}
let cancel = Self::build_cancel_result(self.max_iterations, std::mem::take(emitted_messages)); let cancel = Self::build_cancel_result(self.max_iterations, std::mem::take(emitted_messages));
self.emit_live_tool_call_message(cancel.final_response.clone()).await; self.emit_live_tool_call_message(cancel.final_response.clone()).await;
return cancel; return cancel;
@ -1738,6 +1842,16 @@ impl AgentLoop {
match final_result { match final_result {
Ok(response) => { Ok(response) => {
summary_last_attempt_duration = attempt_start.elapsed();
// Emit AgentEnd for the summary LLM call
if let Some(ref observer) = self.observer {
observer.record_event(&ObserverEvent::AgentEnd {
provider: self.provider.name().to_string(),
model: self.provider.model_id().to_string(),
duration: summary_last_attempt_duration,
tokens_used: Some(response.usage.total_tokens as u64),
});
}
let mut assistant_message = if let Some(reasoning_content) = let mut assistant_message = if let Some(reasoning_content) =
response.reasoning_content response.reasoning_content
{ {
@ -1761,6 +1875,7 @@ impl AgentLoop {
}; };
} }
Err(e) => { Err(e) => {
summary_last_attempt_duration = attempt_start.elapsed();
let error_text = e.to_string(); let error_text = e.to_string();
let can_retry = attempt < max_retries && is_recoverable_llm_error(&error_text); let can_retry = attempt < max_retries && is_recoverable_llm_error(&error_text);
if can_retry { if can_retry {
@ -1776,6 +1891,16 @@ impl AgentLoop {
if self.cancel_token.is_some() { if self.cancel_token.is_some() {
tokio::select! { tokio::select! {
_ = self.cancel_signal() => { _ = self.cancel_signal() => {
// cancel 路径补发 AgentEnd保证指标配对闭合
summary_last_attempt_duration = attempt_start.elapsed();
if let Some(ref observer) = self.observer {
observer.record_event(&ObserverEvent::AgentEnd {
provider: self.provider.name().to_string(),
model: self.provider.model_id().to_string(),
duration: summary_last_attempt_duration,
tokens_used: None,
});
}
let cancel = Self::build_cancel_result(self.max_iterations, std::mem::take(emitted_messages)); let cancel = Self::build_cancel_result(self.max_iterations, std::mem::take(emitted_messages));
self.emit_live_tool_call_message(cancel.final_response.clone()).await; self.emit_live_tool_call_message(cancel.final_response.clone()).await;
return cancel; return cancel;
@ -1794,6 +1919,15 @@ impl AgentLoop {
error_details = %format_error_chain(e.as_ref()), error_details = %format_error_chain(e.as_ref()),
"Failed to get summary from LLM" "Failed to get summary from LLM"
); );
// 错误分支补发 AgentEnd保证指标配对闭合
if let Some(ref observer) = self.observer {
observer.record_event(&ObserverEvent::AgentEnd {
provider: self.provider.name().to_string(),
model: self.provider.model_id().to_string(),
duration: summary_last_attempt_duration,
tokens_used: None,
});
}
let final_message = let final_message =
ChatMessage::assistant(recoverable_llm_message(&error_text)); ChatMessage::assistant(recoverable_llm_message(&error_text));
emitted_messages.push(final_message.clone()); emitted_messages.push(final_message.clone());
@ -1818,6 +1952,15 @@ impl AgentLoop {
model = %self.provider.model_id(), model = %self.provider.model_id(),
"run_final_summary retry loop exited without returning" "run_final_summary retry loop exited without returning"
); );
// 兜底也补发 AgentEnd
if let Some(ref observer) = self.observer {
observer.record_event(&ObserverEvent::AgentEnd {
provider: self.provider.name().to_string(),
model: self.provider.model_id().to_string(),
duration: summary_last_attempt_duration,
tokens_used: None,
});
}
let final_message = ChatMessage::assistant( let final_message = ChatMessage::assistant(
"Failed to generate final summary: retry loop exited unexpectedly.", "Failed to generate final summary: retry loop exited unexpectedly.",
); );
@ -1935,6 +2078,7 @@ impl AgentLoop {
} }
/// Execute a single tool and return the outcome with event tracking. /// Execute a single tool and return the outcome with event tracking.
#[tracing::instrument(skip(self, tool_call), fields(tool = %tool_call.name))]
async fn execute_one_tool(&self, tool_call: &ToolCall) -> ToolExecutionOutcome { async fn execute_one_tool(&self, tool_call: &ToolCall) -> ToolExecutionOutcome {
let start = Instant::now(); let start = Instant::now();
let tool_name = tool_call.name.clone(); let tool_name = tool_call.name.clone();
@ -1993,18 +2137,36 @@ impl AgentLoop {
} }
}; };
match tool let tool_context = {
.execute_with_context( let mut ctx = self.tool_context.clone();
&{ ctx.tool_call_id = Some(tool_call.id.clone());
let mut ctx = self.tool_context.clone(); ctx
ctx.tool_call_id = Some(tool_call.id.clone()); };
ctx // catch_unwind 隔离单个工具的 panic否则一个工具崩溃会终止整个 turn
}, // 用户消息被静默丢弃。归一化为工具级失败后 LLM 还能看到错误并自我纠正。
normalized_arguments.clone(), let execution = std::panic::AssertUnwindSafe(
) tool.execute_with_context(&tool_context, normalized_arguments.clone()),
.await )
{ .catch_unwind()
Ok(result) => { .await;
match execution {
Err(payload) => {
let error = format!(
"Tool '{}' panicked: {}",
tool_call.name,
crate::utils::panic_payload_message(&payload)
);
tracing::error!(
tool = %tool_call.name,
args = %truncate_args(&tool_call.arguments, 4_000),
normalized_args = %truncate_args(&normalized_arguments, 4_000),
error = %error,
"Tool execution panicked"
);
ToolExecutionOutcome::failure(format!("Error: {}", error), Some(error))
}
Ok(Ok(result)) => {
if result.success { if result.success {
if let Some(pending_output) = parse_pending_tool_output(&result.output) { if let Some(pending_output) = parse_pending_tool_output(&result.output) {
ToolExecutionOutcome::pending(pending_output) ToolExecutionOutcome::pending(pending_output)
@ -2031,7 +2193,7 @@ impl AgentLoop {
ToolExecutionOutcome::failure(failure_output, Some(error)) ToolExecutionOutcome::failure(failure_output, Some(error))
} }
} }
Err(e) => { Ok(Err(e)) => {
tracing::error!( tracing::error!(
tool = %tool_call.name, tool = %tool_call.name,
args = %truncate_args(&tool_call.arguments, 4_000), args = %truncate_args(&tool_call.arguments, 4_000),
@ -2984,6 +3146,122 @@ mod tests {
assert_eq!(messages.len(), 3); assert_eq!(messages.len(), 3);
} }
/// 良构不变量校验sanitize 的输出必须满足
/// 1. 每个带 tool_calls 的 assistant 之后紧邻其全部 tool 结果(无其他消息隔断);
/// 2. 每个 tool 消息都有存活的父 assistant。
fn assert_well_formed(messages: &[ChatMessage]) {
let mut pending: Vec<String> = Vec::new();
for (i, m) in messages.iter().enumerate() {
if m.role == "assistant" {
if let Some(calls) = m.tool_calls.as_ref().filter(|c| !c.is_empty()) {
assert!(
pending.is_empty(),
"assistant at {i} starts tool_calls while previous results are pending"
);
pending = calls.iter().map(|tc| tc.id.clone()).collect();
}
} else if m.role == "tool" {
let tc_id = m.tool_call_id.clone().unwrap_or_default();
let pos = pending
.iter()
.position(|id| *id == tc_id)
.unwrap_or_else(|| panic!("tool at {i} has no pending parent (id={tc_id})"));
pending.remove(pos);
} else if !pending.is_empty() {
panic!("non-tool message at {i} interrupts pending tool results");
}
}
assert!(
pending.is_empty(),
"trailing assistant tool_calls without results"
);
}
#[test]
fn test_sanitize_mixed_removal_order_does_not_panic_or_corrupt() {
// Phase 1反向扫描按降序收集孤儿 assistant 索引 [2,1,0]
// Phase 1.5(正向扫描)随后按升序追加索引 3 → remove_indices=[2,1,0,3]。
// 若不全局排序就逐个 Vec::remove第 4 次删除时越界 panic。
let mut messages = vec![
ChatMessage::assistant_with_tool_calls(
"orphan 1",
vec![ToolCall {
id: "call_x".to_string(),
name: "bash".to_string(),
arguments: serde_json::json!({}),
}],
),
ChatMessage::assistant_with_tool_calls(
"orphan 2",
vec![ToolCall {
id: "call_y".to_string(),
name: "bash".to_string(),
arguments: serde_json::json!({}),
}],
),
ChatMessage::assistant_with_tool_calls(
"orphan 3",
vec![ToolCall {
id: "call_z".to_string(),
name: "bash".to_string(),
arguments: serde_json::json!({}),
}],
),
ChatMessage::assistant_with_tool_calls(
"resolved but interrupted",
vec![ToolCall {
id: "call_w".to_string(),
name: "bash".to_string(),
arguments: serde_json::json!({}),
}],
),
ChatMessage::user("next question"),
ChatMessage::tool("call_w", "bash", "result"),
];
let removed = crate::bus::message::sanitize_incomplete_tool_call_sequences(&mut messages);
assert_eq!(removed, 5, "4 assistants + 1 orphaned tool result");
assert_eq!(messages.len(), 1);
assert_eq!(messages[0].role, "user");
assert_eq!(messages[0].content, "next question");
assert_well_formed(&messages);
}
#[test]
fn test_sanitize_mixed_removal_order_deletes_correct_messages() {
// remove_indices=[0(Phase 1), 1(Phase 1.5)]:先 remove(0) 后索引漂移,
// 未排序时 remove(1) 会误删 user 消息而非第二个 assistant。
let mut messages = vec![
ChatMessage::assistant_with_tool_calls(
"orphan",
vec![ToolCall {
id: "call_x".to_string(),
name: "bash".to_string(),
arguments: serde_json::json!({}),
}],
),
ChatMessage::assistant_with_tool_calls(
"resolved but interrupted",
vec![ToolCall {
id: "call_y".to_string(),
name: "bash".to_string(),
arguments: serde_json::json!({}),
}],
),
ChatMessage::user("interrupting question"),
ChatMessage::tool("call_y", "bash", "result"),
];
let removed = crate::bus::message::sanitize_incomplete_tool_call_sequences(&mut messages);
assert_eq!(removed, 3, "2 assistants + 1 orphaned tool result");
assert_eq!(messages.len(), 1);
assert_eq!(messages[0].role, "user");
assert_eq!(messages[0].content, "interrupting question");
assert_well_formed(&messages);
}
// ===== LLM 重试机制测试 ===== // ===== LLM 重试机制测试 =====
#[test] #[test]

View File

@ -443,7 +443,12 @@ pub(crate) fn sanitize_incomplete_tool_call_sequences(messages: &mut Vec<ChatMes
} }
} }
// Remove in descending index order to avoid shifting // Remove in descending index order to avoid shifting.
// 两阶段产出的索引并非全局降序Phase 1反向扫描按降序追加
// Phase 1.5(正向扫描)按升序追加。逐个 Vec::remove 前必须全局排序,
// 否则已删除元素会使后续索引漂移(删错消息)甚至越界 panic。
remove_indices.sort_unstable_by(|a, b| b.cmp(a));
remove_indices.dedup();
for &idx in &remove_indices { for &idx in &remove_indices {
messages.remove(idx); messages.remove(idx);
removed += 1; removed += 1;
@ -502,6 +507,9 @@ pub struct InboundMessage {
pub metadata: HashMap<String, String>, pub metadata: HashMap<String, String>,
/// Data forwarded from inbound to outbound (copied to OutboundMessage.metadata by gateway). /// Data forwarded from inbound to outbound (copied to OutboundMessage.metadata by gateway).
pub forwarded_metadata: HashMap<String, String>, pub forwarded_metadata: HashMap<String, String>,
/// 端到端追踪 ID由 channel 在构造消息时生成,贯穿 bus→processor→agent_loop→provider→tool 全链路)。
/// 基础设施层元数据,不进入 domain 层。
pub trace_id: String,
} }
impl InboundMessage { impl InboundMessage {
@ -537,6 +545,9 @@ pub struct OutboundMessage {
/// instead of generating a random UUID. Critical for stream delta → assistant_response /// instead of generating a random UUID. Critical for stream delta → assistant_response
/// ID matching on the front-end. /// ID matching on the front-end.
pub message_id: Option<String>, pub message_id: Option<String>,
/// 端到端追踪 ID从 InboundMessage 继承,用于 outbound dispatcher 日志关联)。
/// 非 agent 执行路径产生的消息(如 scheduler 通知)此字段为空。
pub trace_id: String,
} }
#[derive(Debug, Clone, PartialEq, Eq)] #[derive(Debug, Clone, PartialEq, Eq)]
@ -564,6 +575,12 @@ impl OutboundMessage {
) )
} }
/// 设置 trace_idbuilder 模式,用于 agent 执行路径中从 InboundMessage 继承)。
pub fn with_trace_id(mut self, trace_id: impl Into<String>) -> Self {
self.trace_id = trace_id.into();
self
}
pub fn assistant( pub fn assistant(
channel: impl Into<String>, channel: impl Into<String>,
chat_id: impl Into<String>, chat_id: impl Into<String>,
@ -587,6 +604,7 @@ impl OutboundMessage {
tool_arguments: None, tool_arguments: None,
reasoning_content: None, reasoning_content: None,
message_id: None, message_id: None,
trace_id: String::new(),
} }
} }
@ -645,6 +663,7 @@ impl OutboundMessage {
tool_arguments: Some(tool_arguments), tool_arguments: Some(tool_arguments),
reasoning_content: None, reasoning_content: None,
message_id: None, message_id: None,
trace_id: String::new(),
} }
} }
@ -676,6 +695,7 @@ impl OutboundMessage {
tool_arguments: None, tool_arguments: None,
reasoning_content: None, reasoning_content: None,
message_id: None, message_id: None,
trace_id: String::new(),
} }
} }
@ -707,6 +727,7 @@ impl OutboundMessage {
tool_arguments: None, tool_arguments: None,
reasoning_content: None, reasoning_content: None,
message_id: None, message_id: None,
trace_id: String::new(),
} }
} }
@ -735,6 +756,7 @@ impl OutboundMessage {
tool_arguments: None, tool_arguments: None,
reasoning_content: reasoning_delta, reasoning_content: reasoning_delta,
message_id: None, message_id: None,
trace_id: String::new(),
} }
} }
@ -761,6 +783,7 @@ impl OutboundMessage {
tool_arguments: None, tool_arguments: None,
reasoning_content: None, reasoning_content: None,
message_id: None, message_id: None,
trace_id: String::new(),
} }
} }
@ -786,6 +809,7 @@ impl OutboundMessage {
tool_arguments: None, tool_arguments: None,
reasoning_content: None, reasoning_content: None,
message_id: None, message_id: None,
trace_id: String::new(),
} }
} }

View File

@ -35,8 +35,15 @@ impl MessageBus {
/// Publish a message to the inbound queue /// Publish a message to the inbound queue
pub async fn publish_inbound(&self, msg: InboundMessage) -> Result<(), BusError> { pub async fn publish_inbound(&self, msg: InboundMessage) -> Result<(), BusError> {
#[cfg(debug_assertions)] tracing::debug!(
tracing::debug!(channel = %msg.channel, sender = %msg.sender_id, chat = %msg.chat_id, content_len = %msg.content.len(), media_count = %msg.media.len(), "Bus: publishing inbound message"); channel = %msg.channel,
sender = %msg.sender_id,
chat_id = %msg.chat_id,
trace_id = %msg.trace_id,
content_len = %msg.content.len(),
media_count = %msg.media.len(),
"Bus: publishing inbound message"
);
self.inbound_tx self.inbound_tx
.send(msg) .send(msg)
.await .await
@ -47,8 +54,13 @@ impl MessageBus {
/// Returns `None` when the channel is closed (all senders dropped). /// Returns `None` when the channel is closed (all senders dropped).
pub async fn consume_inbound(&self) -> Option<InboundMessage> { pub async fn consume_inbound(&self) -> Option<InboundMessage> {
let msg = self.inbound_rx.lock().await.recv().await?; let msg = self.inbound_rx.lock().await.recv().await?;
#[cfg(debug_assertions)] tracing::debug!(
tracing::debug!(channel = %msg.channel, sender = %msg.sender_id, chat = %msg.chat_id, "Bus: consuming inbound message"); channel = %msg.channel,
sender = %msg.sender_id,
chat_id = %msg.chat_id,
trace_id = %msg.trace_id,
"Bus: consuming inbound message"
);
Some(msg) Some(msg)
} }
@ -59,13 +71,20 @@ impl MessageBus {
/// blocked by slow or disconnected display consumers. Persistent state is /// blocked by slow or disconnected display consumers. Persistent state is
/// unaffected — messages are stored in SQLite independently. /// unaffected — messages are stored in SQLite independently.
pub async fn publish_outbound(&self, msg: OutboundMessage) -> Result<(), BusError> { pub async fn publish_outbound(&self, msg: OutboundMessage) -> Result<(), BusError> {
#[cfg(debug_assertions)] tracing::debug!(
tracing::debug!(channel = %msg.channel, chat_id = %msg.chat_id, content_len = %msg.content.len(), "Bus: publishing outbound message"); channel = %msg.channel,
chat_id = %msg.chat_id,
trace_id = %msg.trace_id,
content_len = %msg.content.len(),
"Bus: publishing outbound message"
);
match self.outbound_tx.try_send(msg) { match self.outbound_tx.try_send(msg) {
Ok(()) => Ok(()), Ok(()) => Ok(()),
Err(tokio::sync::mpsc::error::TrySendError::Full(msg)) => { Err(tokio::sync::mpsc::error::TrySendError::Full(msg)) => {
tracing::warn!( tracing::warn!(
channel = %msg.channel, channel = %msg.channel,
chat_id = %msg.chat_id,
trace_id = %msg.trace_id,
"Outbound bus full, dropping message" "Outbound bus full, dropping message"
); );
Err(BusError::Dropped) Err(BusError::Dropped)

View File

@ -1327,6 +1327,7 @@ impl FeishuChannel {
media: parsed.media.map(|m| vec![m]).unwrap_or_default(), media: parsed.media.map(|m| vec![m]).unwrap_or_default(),
metadata: std::collections::HashMap::new(), metadata: std::collections::HashMap::new(),
forwarded_metadata, forwarded_metadata,
trace_id: crate::observability::tracing_ctx::new_trace_id(),
}; };
if let Err(e) = channel.handle_and_publish(&bus, &msg).await { if let Err(e) = channel.handle_and_publish(&bus, &msg).await {
tracing::error!(error = %e, open_id = %parsed.open_id, chat_id = %parsed.chat_id, "Failed to publish Feishu message to bus"); tracing::error!(error = %e, open_id = %parsed.open_id, chat_id = %parsed.chat_id, "Failed to publish Feishu message to bus");

View File

@ -247,6 +247,7 @@ impl Channel for WechatChannel {
media, media,
metadata, metadata,
forwarded_metadata: HashMap::new(), forwarded_metadata: HashMap::new(),
trace_id: crate::observability::tracing_ctx::new_trace_id(),
}; };
if let Err(error) = bus.publish_inbound(inbound).await { if let Err(error) = bus.publish_inbound(inbound).await {

View File

@ -78,6 +78,8 @@ impl InitWizard {
tools: crate::config::ToolsConfig::default(), tools: crate::config::ToolsConfig::default(),
memory_maintenance: crate::config::MemoryMaintenanceConfig::default(), memory_maintenance: crate::config::MemoryMaintenanceConfig::default(),
mcp_servers: HashMap::new(), mcp_servers: HashMap::new(),
mcp_tool_timeout_secs: 300,
observability: crate::config::ObservabilityConfig::default(),
image_context: crate::config::ImageContextConfig::default(), image_context: crate::config::ImageContextConfig::default(),
subagents: crate::config::SubagentsConfig::default(), subagents: crate::config::SubagentsConfig::default(),
experts: crate::config::ExpertsConfig::default(), experts: crate::config::ExpertsConfig::default(),
@ -843,6 +845,8 @@ impl InitWizard {
tools: existing.tools.clone(), tools: existing.tools.clone(),
memory_maintenance: existing.memory_maintenance.clone(), memory_maintenance: existing.memory_maintenance.clone(),
mcp_servers: existing.mcp_servers.clone(), mcp_servers: existing.mcp_servers.clone(),
mcp_tool_timeout_secs: existing.mcp_tool_timeout_secs,
observability: existing.observability.clone(),
image_context: existing.image_context.clone(), image_context: existing.image_context.clone(),
subagents: existing.subagents.clone(), subagents: existing.subagents.clone(),
experts: existing.experts.clone(), experts: existing.experts.clone(),

View File

@ -1,4 +1,5 @@
use async_trait::async_trait; use async_trait::async_trait;
use std::sync::Arc;
use crate::command::Command; use crate::command::Command;
use crate::command::context::CommandContext; use crate::command::context::CommandContext;
@ -6,18 +7,27 @@ use crate::command::handler::{CommandHandler, CommandMetadata};
use crate::command::response::{CommandError, CommandResponse, MessageKind}; use crate::command::response::{CommandError, CommandResponse, MessageKind};
use crate::gateway::cancel_manager::CancelManager; use crate::gateway::cancel_manager::CancelManager;
use crate::gateway::session::SessionManager; use crate::gateway::session::SessionManager;
use crate::tools::SubAgentRuntime;
/// 处理 StopExecution 命令:按话题取消当前正在执行的 Agent。 /// 处理 StopExecution 命令:按话题取消当前正在执行的 Agent。
///
/// 取消传播:同时取消该 topic 下所有正在运行的异步子代理(通过 CancellationToken
pub struct StopExecutionCommandHandler { pub struct StopExecutionCommandHandler {
cancel_manager: CancelManager, cancel_manager: CancelManager,
session_manager: SessionManager, session_manager: SessionManager,
subagent_executor: Option<Arc<dyn SubAgentRuntime>>,
} }
impl StopExecutionCommandHandler { impl StopExecutionCommandHandler {
pub fn new(cancel_manager: CancelManager, session_manager: SessionManager) -> Self { pub fn new(
cancel_manager: CancelManager,
session_manager: SessionManager,
subagent_executor: Option<Arc<dyn SubAgentRuntime>>,
) -> Self {
Self { Self {
cancel_manager, cancel_manager,
session_manager, session_manager,
subagent_executor,
} }
} }
} }
@ -89,9 +99,26 @@ impl CommandHandler for StopExecutionCommandHandler {
let cancelled = self.cancel_manager.cancel_by_topic(&topic_id).await; let cancelled = self.cancel_manager.cancel_by_topic(&topic_id).await;
if cancelled { // 取消传播:同时取消该 topic 下所有正在运行的异步子代理
let cancelled_subagents = if let Some(ref executor) = self.subagent_executor {
executor.cancel_pending_for_topic(&topic_id).await
} else {
0
};
if cancelled || cancelled_subagents > 0 {
let msg = if cancelled && cancelled_subagents > 0 {
format!(
"正在停止当前任务及 {} 个后台子代理...",
cancelled_subagents
)
} else if cancelled {
"正在停止当前任务...".to_string()
} else {
format!("正在停止 {} 个后台子代理...", cancelled_subagents)
};
Ok(CommandResponse::success(ctx.request_id) Ok(CommandResponse::success(ctx.request_id)
.with_message(MessageKind::Notification, "正在停止当前任务...")) .with_message(MessageKind::Notification, msg))
} else { } else {
Ok(CommandResponse::success(ctx.request_id) Ok(CommandResponse::success(ctx.request_id)
.with_message(MessageKind::Notification, "当前没有正在执行的任务")) .with_message(MessageKind::Notification, "当前没有正在执行的任务"))

View File

@ -34,6 +34,9 @@ pub struct Config {
pub memory_maintenance: MemoryMaintenanceConfig, pub memory_maintenance: MemoryMaintenanceConfig,
#[serde(default, rename = "mcpServers")] #[serde(default, rename = "mcpServers")]
pub mcp_servers: HashMap<String, crate::mcp::McpServerConfig>, pub mcp_servers: HashMap<String, crate::mcp::McpServerConfig>,
/// MCP 工具调用超时时间。0 表示不超时。默认 3005 分钟)。
#[serde(default = "default_mcp_tool_timeout_secs")]
pub mcp_tool_timeout_secs: u64,
#[serde(default)] #[serde(default)]
pub image_context: ImageContextConfig, pub image_context: ImageContextConfig,
#[serde(default)] #[serde(default)]
@ -42,6 +45,8 @@ pub struct Config {
pub experts: ExpertsConfig, pub experts: ExpertsConfig,
#[serde(default)] #[serde(default)]
pub compaction: CompactionConfig, pub compaction: CompactionConfig,
#[serde(default)]
pub observability: ObservabilityConfig,
} }
/// 图片上下文限制配置 /// 图片上下文限制配置
@ -122,6 +127,34 @@ impl Default for CompactionConfig {
} }
} }
/// 可观测性配置日志格式、metrics 开关等)
#[derive(Debug, Clone, Deserialize, Serialize)]
pub struct ObservabilityConfig {
/// 日志输出格式text默认或 json。
/// json 格式便于接入 ELK/Loki 等日志聚合系统。
#[serde(default)]
pub log_format: LogFormat,
}
impl Default for ObservabilityConfig {
fn default() -> Self {
Self {
log_format: LogFormat::default(),
}
}
}
/// 日志输出格式
#[derive(Debug, Clone, Deserialize, Serialize, Default, PartialEq, Eq)]
#[serde(rename_all = "lowercase")]
pub enum LogFormat {
/// 纯文本格式(默认,便于人读)
#[default]
Text,
/// JSON 格式(便于机器解析和日志聚合)
Json,
}
#[derive(Debug, Clone, Deserialize, Serialize)] #[derive(Debug, Clone, Deserialize, Serialize)]
pub struct TimeConfig { pub struct TimeConfig {
#[serde(default = "default_timezone")] #[serde(default = "default_timezone")]
@ -304,6 +337,12 @@ pub struct TaskConfig {
pub allowed_tools: Vec<String>, pub allowed_tools: Vec<String>,
#[serde(default = "default_task_max_nesting_depth")] #[serde(default = "default_task_max_nesting_depth")]
pub max_nesting_depth: u32, pub max_nesting_depth: u32,
/// 异步子代理最大并发数Semaphore 限流,仅主 agent 顶层 spawn 生效)
#[serde(default = "default_task_max_concurrent")]
pub max_concurrent: usize,
/// wait_for_subagents 工具默认超时LLM 可通过参数覆盖
#[serde(default = "default_task_wait_default_timeout_secs")]
pub wait_default_timeout_secs: u64,
} }
fn default_task_enabled() -> bool { fn default_task_enabled() -> bool {
@ -322,6 +361,14 @@ fn default_task_max_nesting_depth() -> u32 {
2 2
} }
fn default_task_max_concurrent() -> usize {
8
}
fn default_task_wait_default_timeout_secs() -> u64 {
60
}
fn default_task_allowed_tools() -> Vec<String> { fn default_task_allowed_tools() -> Vec<String> {
vec![ vec![
"read".to_string(), "read".to_string(),
@ -347,6 +394,8 @@ impl Default for TaskConfig {
ttl_hours: default_task_ttl_hours(), ttl_hours: default_task_ttl_hours(),
allowed_tools: default_task_allowed_tools(), allowed_tools: default_task_allowed_tools(),
max_nesting_depth: default_task_max_nesting_depth(), max_nesting_depth: default_task_max_nesting_depth(),
max_concurrent: default_task_max_concurrent(),
wait_default_timeout_secs: default_task_wait_default_timeout_secs(),
} }
} }
} }
@ -556,6 +605,10 @@ fn default_max_retries() -> u32 {
3 3
} }
fn default_mcp_tool_timeout_secs() -> u64 {
300
}
#[derive(Debug, Clone, Deserialize, Serialize)] #[derive(Debug, Clone, Deserialize, Serialize)]
pub struct GatewayConfig { pub struct GatewayConfig {
#[serde(default = "default_gateway_host")] #[serde(default = "default_gateway_host")]

View File

@ -1,5 +1,7 @@
use std::sync::Arc; use std::sync::Arc;
use tokio::sync::mpsc;
use crate::agent::context_compressor::ContextCompressor; use crate::agent::context_compressor::ContextCompressor;
use crate::agent::{AgentError, AgentLoop, AgentRuntimeConfig, CompositeSystemPromptProvider, SystemPromptProvider}; use crate::agent::{AgentError, AgentLoop, AgentRuntimeConfig, CompositeSystemPromptProvider, SystemPromptProvider};
use crate::config::{CompactionConfig, LLMProviderConfig, ModelResolver}; use crate::config::{CompactionConfig, LLMProviderConfig, ModelResolver};
@ -9,11 +11,13 @@ use crate::experts::ExpertRuntime;
use crate::gateway::agent_prompt_provider::AgentPromptProvider; use crate::gateway::agent_prompt_provider::AgentPromptProvider;
use crate::gateway::model_selection::ModelSelectionStore; use crate::gateway::model_selection::ModelSelectionStore;
use crate::gateway::tool_prompt_provider::ToolPromptProvider; use crate::gateway::tool_prompt_provider::ToolPromptProvider;
use crate::observability::Observer;
use crate::skills::{SkillPromptProvider, SkillRuntime}; use crate::skills::{SkillPromptProvider, SkillRuntime};
use crate::storage::PromptInjectionRepository; use crate::storage::PromptInjectionRepository;
use crate::storage::persistent_session_id; use crate::storage::persistent_session_id;
use crate::tools::task::runtime::{SubagentPromptProvider, SubagentRuntime}; use crate::tools::task::runtime::{SubagentPromptProvider, SubagentRuntime};
use crate::tools::{ToolContext, ToolRegistry}; use crate::tools::task::SubagentResult;
use crate::tools::{ToolContext, ToolRegistry, WaitCoordinator};
/// 构建与 Agent 实际使用的完全一致的组合系统提示词 Provider。 /// 构建与 Agent 实际使用的完全一致的组合系统提示词 Provider。
/// ///
@ -56,6 +60,8 @@ pub(crate) struct AgentFactory {
model_selections: Arc<ModelSelectionStore>, model_selections: Arc<ModelSelectionStore>,
/// 上下文压缩算法配置(所有 agent 共享) /// 上下文压缩算法配置(所有 agent 共享)
compaction_config: CompactionConfig, compaction_config: CompactionConfig,
/// 可观测性 Observer依赖注入到 AgentLoop业务层不感知具体实现
observer: Option<Arc<dyn Observer>>,
/// 实例创建时间戳(用于区分新旧 AgentFactory 实例) /// 实例创建时间戳(用于区分新旧 AgentFactory 实例)
instance_id: u64, instance_id: u64,
} }
@ -71,6 +77,13 @@ pub(crate) struct AgentBuildRequest<'a> {
pub(crate) topic_id: Option<String>, pub(crate) topic_id: Option<String>,
/// 取消信号接收端可选Agent 在每次迭代时检查是否被取消 /// 取消信号接收端可选Agent 在每次迭代时检查是否被取消
pub(crate) cancel_token: Option<tokio::sync::watch::Receiver<()>>, pub(crate) cancel_token: Option<tokio::sync::watch::Receiver<()>>,
/// 端到端追踪 ID从 InboundMessage 继承,注入 ToolContext 供 tool 执行路径日志关联)
pub(crate) trace_id: Option<String>,
/// 异步子代理完成队列的 sender按 topic 隔离)。
/// 仅主 agent 有值TaskTool 据此在子代理完成时发送 SubagentResult。
pub(crate) sub_done_sender: Option<mpsc::Sender<SubagentResult>>,
/// wait_for_subagents 工具的协调器(仅主 agent 有值)。
pub(crate) wait_coordinator: Option<Arc<dyn WaitCoordinator>>,
} }
impl AgentFactory { impl AgentFactory {
@ -84,6 +97,7 @@ impl AgentFactory {
model_resolver: Arc<ModelResolver>, model_resolver: Arc<ModelResolver>,
model_selections: Arc<ModelSelectionStore>, model_selections: Arc<ModelSelectionStore>,
compaction_config: CompactionConfig, compaction_config: CompactionConfig,
observer: Option<Arc<dyn Observer>>,
) -> Self { ) -> Self {
// 使用 Arc 指针地址作为实例标识符,用于区分新旧 AgentFactory 实例 // 使用 Arc 指针地址作为实例标识符,用于区分新旧 AgentFactory 实例
let instance_id = Arc::as_ptr(&tools) as u64; let instance_id = Arc::as_ptr(&tools) as u64;
@ -102,6 +116,7 @@ impl AgentFactory {
model_resolver, model_resolver,
model_selections, model_selections,
compaction_config, compaction_config,
observer,
instance_id, instance_id,
} }
} }
@ -227,6 +242,12 @@ impl AgentFactory {
.notification_chat_id .notification_chat_id
.unwrap_or(request.session_chat_id); .unwrap_or(request.session_chat_id);
// 构建上下文压缩器(参数内聚到 ContextCompressorCompactionConfig 注入) // 构建上下文压缩器(参数内聚到 ContextCompressorCompactionConfig 注入)
// 注入取消信号 receiver 的 clone 到 ToolContext
// 供 wait_for_subagents 工具传递给 coordinator.wait() 的 select!。
// watch::Receiver::clone() 创建共享同一 sender 的新 receiver
// 各 receiver 的 has_changed()/changed() 状态独立,互不影响。
let cancel_rx_for_context = request.cancel_token.as_ref().map(|rx| rx.clone());
let runtime_config = AgentRuntimeConfig::from(effective_provider_config.clone()); let runtime_config = AgentRuntimeConfig::from(effective_provider_config.clone());
let compressor = Arc::new(self.build_compressor(&runtime_config)); let compressor = Arc::new(self.build_compressor(&runtime_config));
let mut agent = agent let mut agent = agent
@ -245,8 +266,19 @@ impl AgentFactory {
tool_call_id: None, tool_call_id: None,
// 注入专家 capabilityTaskTool 据此强制校验子代理白/黑名单 // 注入专家 capabilityTaskTool 据此强制校验子代理白/黑名单
parent_capability: expert_capability.clone(), parent_capability: expert_capability.clone(),
trace_id: request.trace_id.clone(),
// 注入异步子代理完成队列 sender按 topic 隔离)
sub_done_sender: request.sub_done_sender.clone(),
// 注入 wait 协调器(封装释放/重获取 serial_lock 逻辑)
wait_coordinator: request.wait_coordinator.clone(),
// 注入取消信号 receiver clone供 wait 工具的 cancel 检查)
cancel_rx: cancel_rx_for_context,
}) })
.with_compressor(Some(compressor)); .with_compressor(Some(compressor));
// 注入观测器依赖注入agent_loop 只认 Observer trait
if let Some(ref observer) = self.observer {
agent = agent.with_observer(observer.clone());
}
// 如果有取消信号接收端,注入 Agent // 如果有取消信号接收端,注入 Agent
if let Some(token) = request.cancel_token { if let Some(token) = request.cancel_token {
agent = agent.with_cancel_token(token); agent = agent.with_cancel_token(token);

View File

@ -84,9 +84,10 @@ pub fn extract_bearer_token(headers: &HeaderMap) -> Option<&str> {
}) })
} }
/// axum 中间件:对 `/api/*` 路由强制 Bearer token 校验。 /// axum 中间件:对 `/api/*` 和 `/metrics` 路由强制 Bearer token 校验。
/// 仅在 `requires_auth` 为 true 时挂载。 /// 仅在 `requires_auth` 为 true 时挂载。
/// `/health`、`/ws`、静态资源放行;`/ws` 的 token 校验在 ws_handler 内完成。 /// `/health`、`/ws`、静态资源放行;`/ws` 的 token 校验在 ws_handler 内完成。
/// `/metrics` 包含运行时指标provider/model/耗时/token 用量),远程部署时需保护。
pub async fn require_bearer_auth( pub async fn require_bearer_auth(
headers: HeaderMap, headers: HeaderMap,
request: Request, request: Request,
@ -94,8 +95,9 @@ pub async fn require_bearer_auth(
) -> Response { ) -> Response {
let path = request.uri().path(); let path = request.uri().path();
// 仅对 /api/ 前缀的请求强制认证 // /api/* 和 /metrics 需要认证;其余放行
if !path.starts_with("/api/") { let needs_auth = path.starts_with("/api/") || path == "/metrics";
if !needs_auth {
return next.run(request).await; return next.run(request).await;
} }

View File

@ -6,13 +6,20 @@ use crate::agent::AgentError;
use super::session::Session; use super::session::Session;
/// Run two-segment history compression synchronously. /// Run two-segment history compression.
/// ///
/// Unlike the previous background approach (tokio::spawn), this holds the /// The session lock is held only for the brief data-gathering and
/// session lock during the LLM calls (25 seconds). Since the agent loop /// history-reload phases. The expensive LLM call (25 seconds) and the DB
/// has already finished by this point there is no response-time impact, and /// write happen **without** holding the session lock, so other commands
/// the synchronous guarantee means the next execution always starts with /// (e.g. `create_session`, `list_topics`) are not blocked during compression.
/// freshly compacted history. ///
/// Concurrency safety:
/// - **Same topic**: the caller (`prepare_and_execute_message`) holds the
/// per-topic serial lock (`_serial_guard`) for the entire duration of
/// execution + compaction, so no other message for this topic can modify
/// the in-memory or DB history between phases.
/// - **Different topic**: fully unblocked — the session lock is free during
/// the LLM call.
/// ///
/// 按 topic_id 隔离:压缩只处理指定 topic 的历史DB 替换也只影响该 topic。 /// 按 topic_id 隔离:压缩只处理指定 topic 的历史DB 替换也只影响该 topic。
pub(crate) async fn schedule_background_history_compaction( pub(crate) async fn schedule_background_history_compaction(
@ -23,35 +30,41 @@ pub(crate) async fn schedule_background_history_compaction(
let chat_id = chat_id.into(); let chat_id = chat_id.into();
let topic_id = topic_id.into(); let topic_id = topic_id.into();
let mut session_guard = session.lock().await; // Phase 1: brief session lock to gather compaction inputs.
session_guard.ensure_persistent_session(&chat_id)?; let (history, compressor, store, session_id, provider_config) = {
session_guard.ensure_chat_loaded(&chat_id, Some(&topic_id))?; let mut session_guard = session.lock().await;
session_guard.ensure_persistent_session(&chat_id)?;
session_guard.ensure_chat_loaded(&chat_id, Some(&topic_id))?;
let history = session_guard.get_or_create_history(&topic_id).clone(); let history = session_guard.get_or_create_history(&topic_id).clone();
let compressor = session_guard.compressor().clone(); let compressor = session_guard.compressor().clone();
let store = session_guard.store();
let session_id = session_guard.persistent_session_id(&chat_id);
let provider_config = session_guard.provider_config().clone();
(history, compressor, store, session_id, provider_config)
};
// session lock released here
if !compressor.should_compress(&history) { if !compressor.should_compress(&history) {
return Ok(()); return Ok(());
} }
let store = session_guard.store();
let session_id = session_guard.persistent_session_id(&chat_id);
let provider_config = session_guard.provider_config().clone();
tracing::info!( tracing::info!(
chat_id = %chat_id, chat_id = %chat_id,
topic_id = %topic_id, topic_id = %topic_id,
msg_count = history.len(), msg_count = history.len(),
"Starting synchronous two-segment compression" "Starting two-segment compression (session lock released during LLM call)"
); );
// Synchronous compression — holds lock during LLM calls. // Phase 2: LLM compression WITHOUT holding the session lock.
// compress_two_segment guarantees the result contains no tool_calls, // compress_two_segment guarantees the result contains no tool_calls,
// so there is no risk of orphaned tool call sequences. // so there is no risk of orphaned tool call sequences.
let compressed = compressor let compressed = compressor
.compress_two_segment(&history, &provider_config) .compress_two_segment(&history, &provider_config)
.await?; .await?;
// Phase 3: DB write — store is Arc<dyn ConversationRepository>, no
// session lock needed.
// 保留原始消息(标记 is_compacted=1+ 插入压缩摘要,不删除原消息, // 保留原始消息(标记 is_compacted=1+ 插入压缩摘要,不删除原消息,
// 从而让前端仍能展示完整原始对话LLM 只看压缩后的精简历史。 // 从而让前端仍能展示完整原始对话LLM 只看压缩后的精简历史。
store store
@ -65,7 +78,11 @@ pub(crate) async fn schedule_background_history_compaction(
"Two-segment compression committed (original messages retained)" "Two-segment compression committed (original messages retained)"
); );
session_guard.reload_topic_history(&chat_id, &topic_id)?; // Phase 4: re-acquire session lock to refresh in-memory history.
{
let mut session_guard = session.lock().await;
session_guard.reload_topic_history(&chat_id, &topic_id)?;
}
Ok(()) Ok(())
} }

View File

@ -15,6 +15,8 @@ use tokio::sync::Mutex;
use super::compaction::schedule_background_history_compaction; use super::compaction::schedule_background_history_compaction;
use super::message_prepare::enrich_user_content_with_media_refs; use super::message_prepare::enrich_user_content_with_media_refs;
use super::session::Session; use super::session::Session;
use super::wait_coordinator::SessionWaitCoordinator;
use crate::tools::WaitCoordinator;
/// 空的 EmittedMessageHandler不转发消息仅配合 PersistingEmittedMessageHandler 做持久化。 /// 空的 EmittedMessageHandler不转发消息仅配合 PersistingEmittedMessageHandler 做持久化。
struct NoOpEmittedMessageHandler; struct NoOpEmittedMessageHandler;
@ -114,6 +116,8 @@ pub(crate) struct MessageExecutionRequest<'a> {
pub(crate) live_emitter: Option<Arc<dyn EmittedMessageHandler>>, pub(crate) live_emitter: Option<Arc<dyn EmittedMessageHandler>>,
/// 消息接收时捕获的 topic_id全程显式传递避免从共享状态重复读取竞态 /// 消息接收时捕获的 topic_id全程显式传递避免从共享状态重复读取竞态
pub(crate) topic_id: Option<String>, pub(crate) topic_id: Option<String>,
/// 端到端追踪 ID从 InboundMessage 透传,贯穿 agent → tool → outbound
pub(crate) trace_id: &'a str,
} }
pub(crate) struct ScheduledExecutionRequest<'a> { pub(crate) struct ScheduledExecutionRequest<'a> {
@ -127,6 +131,8 @@ pub(crate) struct ScheduledExecutionRequest<'a> {
pub(crate) system_prompt: Option<&'a str>, pub(crate) system_prompt: Option<&'a str>,
pub(crate) metadata: &'a HashMap<String, String>, pub(crate) metadata: &'a HashMap<String, String>,
pub(crate) fresh_session: bool, pub(crate) fresh_session: bool,
/// 端到端追踪 ID由 ScheduledAgentTaskService 生成)
pub(crate) trace_id: String,
} }
impl AgentExecutionService { impl AgentExecutionService {
@ -280,15 +286,32 @@ impl AgentExecutionService {
// 获取该 topic 的串行锁(通过短暂获取 session 锁) // 获取该 topic 的串行锁(通过短暂获取 session 锁)
// 同一 topic 的消息处理必须串行执行,防止并发 loop 操作同一历史的不同快照 // 同一 topic 的消息处理必须串行执行,防止并发 loop 操作同一历史的不同快照
// 不同 topic 之间互不阻塞,支持多话题并发执行 // 不同 topic 之间互不阻塞,支持多话题并发执行
let serial_lock = { let (serial_lock, store, lock_key) = {
let mut session_guard = request.session.lock().await; let mut session_guard = request.session.lock().await;
let lock_key = request.topic_id.as_deref().unwrap_or(request.chat_id); let lock_key = request
session_guard.topic_serial_lock(lock_key) .topic_id
.as_deref()
.unwrap_or(request.chat_id)
.to_string();
session_guard.ensure_sub_done_channel(&lock_key);
(
session_guard.topic_serial_lock(&lock_key),
session_guard.session_store(),
lock_key,
)
}; };
// 等待该 topic 的前一条消息处理完成(含压缩) // 等待该 topic 的前一条消息处理完成(含压缩)
// await 串行锁时不持有 session 锁,其他 topic 的消息可以正常处理 // await 串行锁时不持有 session 锁,其他 topic 的消息可以正常处理
let _serial_guard = serial_lock.lock().await; // 使用 lock_owned 获取 OwnedMutexGuard存入 guard_slot 供 wait_coordinator 释放/重获取
// 注意lock_owned 消费 Arc<Self>,需 clone 保留 serial_lock 供 coordinator 使用
let serial_guard = serial_lock.clone().lock_owned().await;
// guard_slotwait_coordinator 通过此 slot 释放/重获取 serial_lock。
// 正常执行时 guard 留在 slot 中锁持有wait 工具调用时 take guard 释放锁,
// select! 等待结束后重获取锁并回填新 guard。
// guard_slot 作为 Arc 共享于执行路径与 coordinator二者全部 drop 时 guard 才释放锁。
let guard_slot = Arc::new(Mutex::new(Some(serial_guard)));
let (history, agent, user_message, user_message_count, original_topic_id) = { let (history, agent, user_message, user_message_count, original_topic_id) = {
let mut session_guard = request.session.lock().await; let mut session_guard = request.session.lock().await;
@ -336,11 +359,27 @@ impl AgentExecutionService {
let history = session_guard.get_or_create_history(history_key).clone(); let history = session_guard.get_or_create_history(history_key).clone();
session_guard.record_skill_offer(request.chat_id)?; session_guard.record_skill_offer(request.chat_id)?;
// 创建 wait 协调器(封装释放/重获取 serial_lock + select! 等待逻辑)。
// 仅主 agent 注入coordinator 通过 guard_slot 释放/重获取 serial_lock
// 使 wait_for_subagents 工具能在等待期间让 process_one 注入用户消息。
let wait_coordinator: Option<Arc<dyn WaitCoordinator>> = {
let coordinator = SessionWaitCoordinator::new(
request.session.clone(),
guard_slot.clone(),
serial_lock.clone(),
store.clone(),
lock_key.clone(),
);
Some(Arc::new(coordinator))
};
let mut agent = session_guard.create_agent( let mut agent = session_guard.create_agent(
request.chat_id, request.chat_id,
Some(request.sender_id), Some(request.sender_id),
Some(&user_message.id), Some(&user_message.id),
original_topic_id.as_deref(), original_topic_id.as_deref(),
request.trace_id,
wait_coordinator,
)?; )?;
if let Some(handler) = request.live_emitter.clone() { if let Some(handler) = request.live_emitter.clone() {
agent = agent.with_emitted_message_handler(handler); agent = agent.with_emitted_message_handler(handler);
@ -403,17 +442,26 @@ impl AgentExecutionService {
// 获取该 topic 的串行锁(与普通消息路径共享,保证串行执行) // 获取该 topic 的串行锁(与普通消息路径共享,保证串行执行)
// 定时任务由调度器触发,无用户消息竞态;在锁前一次性捕获 topic_id // 定时任务由调度器触发,无用户消息竞态;在锁前一次性捕获 topic_id
// 锁后复用同一值作为 original_topic_id保证锁键与写入目标一致。 // 锁后复用同一值作为 original_topic_id保证锁键与写入目标一致。
let (serial_lock, lock_time_topic_id) = { let (serial_lock, session_store, lock_key, lock_time_topic_id) = {
let mut session_guard = request.session.lock().await; let mut session_guard = request.session.lock().await;
let tid = session_guard let tid = session_guard
.current_topic(request.chat_id) .current_topic(request.chat_id)
.map(|s| s.to_string()); .map(|s| s.to_string());
let lock_key = tid.as_deref().unwrap_or(request.chat_id); let lock_key = tid.as_deref().unwrap_or(request.chat_id).to_string();
(session_guard.topic_serial_lock(lock_key), tid) session_guard.ensure_sub_done_channel(&lock_key);
(
session_guard.topic_serial_lock(&lock_key),
session_guard.session_store(),
lock_key,
tid,
)
}; };
// 等待该 topic 的前一条消息处理完成(含压缩) // 等待该 topic 的前一条消息处理完成(含压缩)
let _serial_guard = serial_lock.lock().await; // 使用 lock_owned 获取 OwnedMutexGuard存入 guard_slot 供 wait_coordinator 释放/重获取
// 注意lock_owned 消费 Arc<Self>,需 clone 保留 serial_lock 供 coordinator 使用
let serial_guard = serial_lock.clone().lock_owned().await;
let guard_slot = Arc::new(Mutex::new(Some(serial_guard)));
let ( let (
history, history,
@ -470,6 +518,18 @@ impl AgentExecutionService {
let history = session_guard.get_or_create_history(history_key).clone(); let history = session_guard.get_or_create_history(history_key).clone();
session_guard.record_skill_offer(request.chat_id)?; session_guard.record_skill_offer(request.chat_id)?;
// 创建 wait 协调器(与普通消息路径一致,支持定时任务中 spawn 异步子代理)
let wait_coordinator: Option<Arc<dyn WaitCoordinator>> = {
let coordinator = SessionWaitCoordinator::new(
request.session.clone(),
guard_slot.clone(),
serial_lock.clone(),
session_store.clone(),
lock_key.clone(),
);
Some(Arc::new(coordinator))
};
let agent = session_guard.create_agent_with_provider_config( let agent = session_guard.create_agent_with_provider_config(
request.chat_id, request.chat_id,
request.notification_chat_id, // 传入真实 chat_id request.notification_chat_id, // 传入真实 chat_id
@ -477,6 +537,8 @@ impl AgentExecutionService {
Some(&user_message.id), Some(&user_message.id),
request.provider_config.clone(), request.provider_config.clone(),
original_topic_id.as_deref(), original_topic_id.as_deref(),
&request.trace_id,
wait_coordinator,
)?; )?;
// 获取 store 和 session_id用于构造消息持久化 handler // 获取 store 和 session_id用于构造消息持久化 handler

View File

@ -240,6 +240,21 @@ pub async fn list_executions(State(state): State<Arc<GatewayState>>) -> Json<Exe
Json(ExecutionsResponse { topic_ids }) Json(ExecutionsResponse { topic_ids })
} }
/// GET /metrics — Prometheus metrics 端点
///
/// 返回 Prometheus 格式的 metrics 文本。若 recorder 未安装则返回 503。
pub async fn metrics_handler(
State(state): State<Arc<GatewayState>>,
) -> (StatusCode, String) {
match &state.prometheus_handle {
Some(handle) => (StatusCode::OK, handle.render()),
None => (
StatusCode::SERVICE_UNAVAILABLE,
"Metrics recorder not initialized".to_string(),
),
}
}
/// GET /api/mcp/status — Return MCP server connection status /// GET /api/mcp/status — Return MCP server connection status
pub async fn mcp_status( pub async fn mcp_status(
State(state): State<Arc<GatewayState>>, State(state): State<Arc<GatewayState>>,

View File

@ -28,6 +28,7 @@ pub mod session_pool;
pub mod static_files; pub mod static_files;
pub mod tool_prompt_provider; pub mod tool_prompt_provider;
pub mod tool_registry_factory; pub mod tool_registry_factory;
pub mod wait_coordinator;
pub mod ws; pub mod ws;
use axum::{Router, middleware, routing}; use axum::{Router, middleware, routing};
@ -70,8 +71,13 @@ pub struct GatewayState {
pub skills: Arc<SkillRuntime>, pub skills: Arc<SkillRuntime>,
pub experts: Arc<crate::experts::ExpertRuntime>, pub experts: Arc<crate::experts::ExpertRuntime>,
pub subagent_runtime: Arc<SubagentRuntime>, pub subagent_runtime: Arc<SubagentRuntime>,
/// 异步子代理执行器DefaultSubAgentRuntime用于取消传播等操作
pub subagent_executor: Option<Arc<dyn crate::tools::SubAgentRuntime>>,
/// per-session 的用户模型选择(覆盖专家配置) /// per-session 的用户模型选择(覆盖专家配置)
pub model_selections: Arc<model_selection::ModelSelectionStore>, pub model_selections: Arc<model_selection::ModelSelectionStore>,
/// Prometheus metrics handle/metrics 端点渲染用)。
/// None 表示 recorder 安装失败;热重启时从 OnceLock 缓存复用,不会因重复安装而变为 None。
pub prometheus_handle: Option<metrics_exporter_prometheus::PrometheusHandle>,
} }
impl GatewayState { impl GatewayState {
@ -102,7 +108,7 @@ impl GatewayState {
mcp_servers: config.mcp_servers.clone(), mcp_servers: config.mcp_servers.clone(),
}; };
let (session_manager, task_repository, mcp_manager, subagent_runtime, model_selections) = let (session_manager, task_repository, mcp_manager, subagent_runtime, model_selections, subagent_executor) =
build_session_manager_with_sender( build_session_manager_with_sender(
agent_prompt_reinject_every, agent_prompt_reinject_every,
show_tool_results, show_tool_results,
@ -118,6 +124,7 @@ impl GatewayState {
config.memory_maintenance.clone(), config.memory_maintenance.clone(),
session_ttl_hours, session_ttl_hours,
mcp_config, mcp_config,
config.mcp_tool_timeout_secs,
Some(bus.clone()), Some(bus.clone()),
Arc::new(crate::config::ModelResolver::from_config(&config)), Arc::new(crate::config::ModelResolver::from_config(&config)),
config.compaction.clone(), config.compaction.clone(),
@ -131,6 +138,9 @@ impl GatewayState {
let cancel_manager = CancelManager::new(); let cancel_manager = CancelManager::new();
// 安装 Prometheus recorder幂等首次安装并缓存 handle热重启时返回缓存
let prometheus_handle = crate::observability::metrics::init_recorder();
Ok(Self { Ok(Self {
config: Arc::new(RwLock::new(config)), config: Arc::new(RwLock::new(config)),
session_manager, session_manager,
@ -143,7 +153,9 @@ impl GatewayState {
skills, skills,
experts, experts,
subagent_runtime, subagent_runtime,
subagent_executor,
model_selections, model_selections,
prometheus_handle,
}) })
} }
@ -170,6 +182,7 @@ impl GatewayState {
semaphore, semaphore,
provider_config, provider_config,
self.cancel_manager.clone(), self.cancel_manager.clone(),
self.subagent_executor.clone(),
); );
tokio::spawn(inbound_processor.run()); tokio::spawn(inbound_processor.run());
@ -194,9 +207,10 @@ pub async fn run(
) -> Result<bool, Box<dyn std::error::Error>> { ) -> Result<bool, Box<dyn std::error::Error>> {
let config = Config::load_default()?; let config = Config::load_default()?;
let timezone = config.time.parse_timezone()?; let timezone = config.time.parse_timezone()?;
let log_format = config.observability.log_format.clone();
// Initialize logging // Initialize logging
logging::init_logging(timezone); logging::init_logging(timezone, log_format);
tracing::info!("Starting PicoBot Gateway"); tracing::info!("Starting PicoBot Gateway");
// Restart signal channel // Restart signal channel
@ -204,6 +218,30 @@ pub async fn run(
let state = Arc::new(GatewayState::from_config(config, restart_tx)?); let state = Arc::new(GatewayState::from_config(config, restart_tx)?);
// ── 崩溃恢复:标记中断的异步子代理 ──
// 服务器重启后,之前 spawn 的异步子代理进程已不存在,
// 将 pending_subagents 表中所有 status='running' 的记录标记为 'interrupted'。
// 下次 wait_for_subagents 调用时,这些 task_id 不会出现在 pending 列表中,
// agent 可据此判断子代理未正常完成。
match state.session_manager.store().mark_all_running_as_interrupted() {
Ok(0) => {
tracing::info!("Crash recovery: no interrupted subagents to recover");
}
Ok(n) => {
tracing::info!(
recovered_count = n,
"Crash recovery: marked {} running subagents as interrupted (server restarted)",
n
);
}
Err(e) => {
tracing::error!(
error = %e,
"Crash recovery: failed to mark interrupted subagents"
);
}
}
// Get provider config for channels // Get provider config for channels
let cfg = state.config.read().await; let cfg = state.config.read().await;
let provider_config = cfg.get_provider_config("default")?; let provider_config = cfg.get_provider_config("default")?;
@ -331,7 +369,8 @@ pub async fn run(
"/api/session/selected-model", "/api/session/selected-model",
routing::get(http::session_selected_model), routing::get(http::session_selected_model),
) )
.route("/ws", routing::get(ws::ws_handler)); .route("/ws", routing::get(ws::ws_handler))
.route("/metrics", routing::get(http::metrics_handler));
// 仅 fallback 按模式区分:嵌入资源 vs 磁盘目录。 // 仅 fallback 按模式区分:嵌入资源 vs 磁盘目录。
// fallback 必须在 with_state 之前调用,否则 handler 的 State 类型无法推断。 // fallback 必须在 with_state 之前调用,否则 handler 的 State 类型无法推断。

View File

@ -171,6 +171,8 @@ impl OutboundDispatcher {
channel_name: &str, channel_name: &str,
msg: OutboundMessage, msg: OutboundMessage,
) { ) {
let msg_chat_id = msg.chat_id.clone();
let msg_trace_id = msg.trace_id.clone();
match Self::send_with_retry(channel, msg).await { match Self::send_with_retry(channel, msg).await {
Ok(()) => {} Ok(()) => {}
Err(ChannelError::ChannelFull) => { Err(ChannelError::ChannelFull) => {
@ -178,12 +180,16 @@ impl OutboundDispatcher {
// 记 warn 而非 error这是预期的背压丢弃。 // 记 warn 而非 error这是预期的背压丢弃。
tracing::warn!( tracing::warn!(
channel = %channel_name, channel = %channel_name,
chat_id = %msg_chat_id,
trace_id = %msg_trace_id,
"Message dropped: channel queue full" "Message dropped: channel queue full"
); );
} }
Err(error) => { Err(error) => {
tracing::error!( tracing::error!(
channel = %channel_name, channel = %channel_name,
chat_id = %msg_chat_id,
trace_id = %msg_trace_id,
error = %error, error = %error,
"Failed to send message after retries" "Failed to send message after retries"
); );

View File

@ -1,5 +1,6 @@
use std::collections::HashSet; use std::collections::HashSet;
use std::sync::Arc; use std::sync::Arc;
use futures_util::FutureExt;
use parking_lot::Mutex; use parking_lot::Mutex;
use tokio::sync::Semaphore; use tokio::sync::Semaphore;
@ -28,6 +29,7 @@ use crate::storage::persistent_session_id;
use crate::topic_description::generate_topic_description; use crate::topic_description::generate_topic_description;
use super::session::{BusToolCallEmitter, SessionManager}; use super::session::{BusToolCallEmitter, SessionManager};
use super::message_prepare::enrich_user_content_with_media_refs;
#[derive(Clone)] #[derive(Clone)]
pub struct InboundProcessor { pub struct InboundProcessor {
@ -47,6 +49,7 @@ impl InboundProcessor {
semaphore: Arc<Semaphore>, semaphore: Arc<Semaphore>,
provider_config: LLMProviderConfig, provider_config: LLMProviderConfig,
cancel_manager: CancelManager, cancel_manager: CancelManager,
subagent_executor: Option<Arc<dyn crate::tools::SubAgentRuntime>>,
) -> Self { ) -> Self {
// 创建命令路由器并注册处理器 // 创建命令路由器并注册处理器
let mut command_router = CommandRouter::new(); let mut command_router = CommandRouter::new();
@ -117,6 +120,7 @@ impl InboundProcessor {
command_router.register(Box::new(StopExecutionCommandHandler::new( command_router.register(Box::new(StopExecutionCommandHandler::new(
cancel_manager.clone(), cancel_manager.clone(),
session_manager.clone(), session_manager.clone(),
subagent_executor,
))); )));
Self { Self {
@ -147,17 +151,15 @@ impl InboundProcessor {
} }
}; };
#[cfg(debug_assertions)] tracing::debug!(
{ channel = %inbound.channel,
tracing::debug!( chat_id = %inbound.chat_id,
channel = %inbound.channel, trace_id = %inbound.trace_id,
chat_id = %inbound.chat_id, sender = %inbound.sender_id,
sender = %inbound.sender_id, content_len = %inbound.content.len(),
content_len = %inbound.content.len(), media_count = %inbound.media.len(),
media_count = %inbound.media.len(), "Processing inbound message"
"Processing inbound message" );
);
}
// 2. 获取 semaphore permit控制并发 // 2. 获取 semaphore permit控制并发
let permit = match self.semaphore.clone().acquire_owned().await { let permit = match self.semaphore.clone().acquire_owned().await {
@ -172,18 +174,53 @@ impl InboundProcessor {
let processor = self.clone(); let processor = self.clone();
// 4. 独立任务处理(包含 permit任务完成自动释放 // 4. 独立任务处理(包含 permit任务完成自动释放
tokio::spawn(async move { // spawn 不自动传播父 span用 traced() 重建 span 上下文,
let _permit = permit; // 持有 permit 直到任务完成 // 使 process_one 内所有日志携带 trace_id/chat_id/session_id。
if let Err(e) = processor.process_one(inbound).await { let trace_id = inbound.trace_id.clone();
tracing::error!(error = %e, "Message processing failed"); let chat_id_for_span = inbound.chat_id.clone();
} let session_id_for_span =
}); crate::storage::persistent_session_id(&inbound.channel, &inbound.chat_id);
tokio::spawn(
crate::observability::tracing_ctx::traced(
&trace_id,
&chat_id_for_span,
&session_id_for_span,
async move {
let _permit = permit; // 持有 permit 直到任务完成
// catch_unwind 将 panic 归一化为错误:否则工具/历史清理中的
// panic 只会终止任务并打 panic hook 日志,跳过错误日志与指标,
// 用户消息被静默吞掉。参考 channels/wechat.rs 的同类用法。
let result = std::panic::AssertUnwindSafe(processor.process_one(inbound))
.catch_unwind()
.await;
match result {
Ok(Ok(())) => {}
Ok(Err(e)) => {
tracing::error!(
error = %crate::utils::format_error_chain(&e),
"Message processing failed"
);
crate::observability::metrics::record_message_processing_error();
}
Err(payload) => {
tracing::error!(
error = %crate::utils::panic_payload_message(&payload),
"Message processing panicked"
);
crate::observability::metrics::record_message_processing_error();
}
}
},
),
);
} }
} }
#[tracing::instrument(skip(self, inbound), fields(trace_id = %inbound.trace_id, chat_id = %inbound.chat_id, session_id))]
async fn process_one(&self, inbound: InboundMessage) -> Result<(), AgentError> { async fn process_one(&self, inbound: InboundMessage) -> Result<(), AgentError> {
// 计算正确的 session_id根据 channel_name 和 chat_id // 计算正确的 session_id根据 channel_name 和 chat_id
let session_id = persistent_session_id(&inbound.channel, &inbound.chat_id); let session_id = persistent_session_id(&inbound.channel, &inbound.chat_id);
tracing::Span::current().record("session_id", tracing::field::display(&session_id));
// 获取当前话题(封装了 session 创建逻辑) // 获取当前话题(封装了 session 创建逻辑)
let current_topic = self let current_topic = self
@ -220,14 +257,17 @@ impl InboundProcessor {
for msg in &response.messages { for msg in &response.messages {
if let Err(error) = self if let Err(error) = self
.bus .bus
.publish_outbound(OutboundMessage::assistant( .publish_outbound(
inbound.channel.clone(), OutboundMessage::assistant(
inbound.chat_id.clone(), inbound.channel.clone(),
response.metadata.get("session_id").cloned(), inbound.chat_id.clone(),
msg.content.clone(), response.metadata.get("session_id").cloned(),
None, msg.content.clone(),
inbound.forwarded_metadata.clone(), None,
)) inbound.forwarded_metadata.clone(),
)
.with_trace_id(&inbound.trace_id),
)
.await .await
{ {
match error { match error {
@ -243,14 +283,17 @@ impl InboundProcessor {
} else if let Some(error) = response.error { } else if let Some(error) = response.error {
if let Err(e) = self if let Err(e) = self
.bus .bus
.publish_outbound(OutboundMessage::assistant( .publish_outbound(
inbound.channel.clone(), OutboundMessage::assistant(
inbound.chat_id.clone(), inbound.channel.clone(),
response.metadata.get("session_id").cloned(), inbound.chat_id.clone(),
format!("Error [{}]: {}", error.code, error.message), response.metadata.get("session_id").cloned(),
None, format!("Error [{}]: {}", error.code, error.message),
inbound.forwarded_metadata.clone(), None,
)) inbound.forwarded_metadata.clone(),
)
.with_trace_id(&inbound.trace_id),
)
.await .await
{ {
match e { match e {
@ -272,6 +315,86 @@ impl InboundProcessor {
if let Some(ref topic_id) = current_topic { if let Some(ref topic_id) = current_topic {
emitter_metadata.insert("topic_id".to_string(), topic_id.clone()); emitter_metadata.insert("topic_id".to_string(), topic_id.clone());
} }
// ── 异步子代理等待注入路径 ──
// 当主 agent 正在 wait_for_subagents 中等待(已释放 serial_lock、is_waiting=true
// 新用户消息不应启动新的 agent loop而应注入 history 并唤醒等待中的 agent。
//
// 流程:
// 1. 获取 serial_lock若 agent 正常运行则阻塞;若 agent 在 wait 中则立即获取)
// 2. 检查 is_waitingtrue → 注入 + wakeup + returnfalse → 释放锁走正常路径
//
// 安全性is_waiting 在持锁状态下检查wait_coordinator 清除 is_waiting 需先重获取锁,
// 两者互斥,无 TOCTOU。
if let Some(ref topic_id) = current_topic {
if let Some(session) = self.session_manager.get(&inbound.channel).await {
let lock_key = topic_id.clone();
// 获取 serial_lock Arc短暂持有 session 锁)
let serial_lock = {
let mut g = session.lock().await;
g.ensure_sub_done_channel(&lock_key);
g.topic_serial_lock(&lock_key)
};
// 阻塞获取 serial_lock
// - agent 正常运行:阻塞至其完成(天然串行化)
// - agent 在 wait 中wait 已释放锁,可立即获取
let _inject_guard = serial_lock.clone().lock_owned().await;
// 检查 is_waiting持锁状态下安全
let is_waiting = {
let g = session.lock().await;
g.is_waiting(&lock_key)
};
if is_waiting {
// Agent 正在 wait_for_subagents 中等待 → 注入用户消息 + 唤醒
tracing::info!(
topic_id = %lock_key,
"Topic is in waiting state, injecting user message and waking up agent"
);
let wakeup = {
let mut g = session.lock().await;
// 确保 session 和 chat 已加载
g.ensure_persistent_session(&inbound.chat_id)?;
g.ensure_chat_loaded(&inbound.chat_id, Some(&lock_key))?;
// 构造用户消息(与 prepare_and_execute_message 一致的处理流程)
let media_refs: Vec<String> = inbound
.media
.iter()
.map(|m| m.path.clone())
.collect();
let enriched_content =
enrich_user_content_with_media_refs(&inbound.content, &media_refs)?;
let user_message =
g.create_user_message(&enriched_content, media_refs);
g.append_persisted_message(
&inbound.chat_id,
Some(&lock_key),
user_message,
)?;
// 获取 wakeup 信号
g.wait_wakeup(&lock_key)
};
// 唤醒等待中的 agentwait_coordinator 的 select! 会捕获此通知)
wakeup.notify_one();
// _inject_guard 在此处 drop → 释放 serial_lock
// wait_coordinator 重获取锁后继续处理history 已包含新用户消息)
//
// 跳过 handle_message / cancel 注册 / execution_completed
// 因为等待中的 agent 会处理这条消息。
return Ok(());
}
// is_waiting=false_inject_guard drop 释放锁,走正常 handle_message 路径
}
}
let live_emitter = Arc::new(PersistingEmittedMessageHandler::new( let live_emitter = Arc::new(PersistingEmittedMessageHandler::new(
BusToolCallEmitter::new( BusToolCallEmitter::new(
self.bus.clone(), self.bus.clone(),
@ -279,6 +402,7 @@ impl InboundProcessor {
inbound.chat_id.clone(), inbound.chat_id.clone(),
emitter_metadata, emitter_metadata,
self.session_manager.store(), self.session_manager.store(),
inbound.trace_id.clone(),
), ),
self.session_manager.store(), self.session_manager.store(),
&session_id, &session_id,
@ -307,6 +431,7 @@ impl InboundProcessor {
inbound.media, inbound.media,
Some(live_emitter), Some(live_emitter),
current_topic.as_deref(), current_topic.as_deref(),
&inbound.trace_id,
) )
.await .await
{ {
@ -319,6 +444,8 @@ impl InboundProcessor {
.metadata .metadata
.insert("topic_id".to_string(), topic_id.clone()); .insert("topic_id".to_string(), topic_id.clone());
} }
// 透传 trace_id 到出站消息,保持端到端追踪贯通
outbound.trace_id = inbound.trace_id.clone();
if let Err(error) = self.bus.publish_outbound(outbound).await { if let Err(error) = self.bus.publish_outbound(outbound).await {
match error { match error {
crate::bus::BusError::Dropped => { crate::bus::BusError::Dropped => {
@ -414,19 +541,26 @@ impl InboundProcessor {
} }
} }
Err(error) => { Err(error) => {
tracing::error!(error = %error, "Failed to handle message"); tracing::error!(
error = %crate::utils::format_error_chain(&error),
"Failed to handle message"
);
crate::observability::metrics::record_message_processing_error();
let mut metadata = inbound.forwarded_metadata.clone(); let mut metadata = inbound.forwarded_metadata.clone();
metadata.insert("error_kind".to_string(), "agent_execution".to_string()); metadata.insert("error_kind".to_string(), "agent_execution".to_string());
if let Err(publish_error) = self if let Err(publish_error) = self
.bus .bus
.publish_outbound(OutboundMessage::error_notification( .publish_outbound(
inbound.channel, OutboundMessage::error_notification(
inbound.chat_id, inbound.channel,
None, // session_id inbound.chat_id,
error.to_string(), None, // session_id
None, error.to_string(),
metadata, None,
)) metadata,
)
.with_trace_id(&inbound.trace_id),
)
.await .await
{ {
match publish_error { match publish_error {
@ -446,28 +580,58 @@ impl InboundProcessor {
self.cancel_manager.remove_by_topic(topic_id).await; self.cancel_manager.remove_by_topic(topic_id).await;
} }
// 发送执行完成信号,通知前端可以停止 loading 状态 // 发送执行完成信号,通知前端可以停止 loading 状态。
// 无论成功还是失败都发送,确保前端状态正确 //
let mut completion_metadata = inbound.forwarded_metadata.clone(); // 退出兜底safety net如果当前 topic 仍有 running 状态的子代理,
if let Some(ref topic_id) = current_topic { // 不发送 ExecutionCompleted。这防止 LLM 未调用 wait_for_subagents 就退出时,
completion_metadata.insert("topic_id".to_string(), topic_id.clone()); // 前端过早停止 loading 导致子代理结果"丢失"的观感。
} // 恢复路径:下一条用户消息触发新的 process_one → 加载 history →
if let Err(error) = self // LLM 看到 "running" 占位 → 调用 wait_for_subagents → 消费 sub_done_q 结果。
.bus let has_pending_subagents = if let Some(ref topic_id) = current_topic {
.publish_outbound(OutboundMessage::execution_completed( let pending = self
channel, .session_manager
chat_id, .store()
Some(session_id), .list_pending_subagents(topic_id, Some("running"))
completion_metadata, .unwrap_or_default();
)) if !pending.is_empty() {
.await tracing::info!(
{ topic_id = %topic_id,
match error { pending_count = pending.len(),
crate::bus::BusError::Dropped => { "Skipping ExecutionCompleted: pending subagents still running"
tracing::warn!(error = %error, "Outbound dropped (bus full)"); );
} true
crate::bus::BusError::Closed => { } else {
tracing::error!(error = %error, "Failed to publish execution_completed"); false
}
} else {
false
};
if !has_pending_subagents {
let mut completion_metadata = inbound.forwarded_metadata.clone();
if let Some(ref topic_id) = current_topic {
completion_metadata.insert("topic_id".to_string(), topic_id.clone());
}
if let Err(error) = self
.bus
.publish_outbound(
OutboundMessage::execution_completed(
channel,
chat_id,
Some(session_id),
completion_metadata,
)
.with_trace_id(&inbound.trace_id),
)
.await
{
match error {
crate::bus::BusError::Dropped => {
tracing::warn!(error = %error, "Outbound dropped (bus full)");
}
crate::bus::BusError::Closed => {
tracing::error!(error = %error, "Failed to publish execution_completed");
}
} }
} }
} }

View File

@ -26,7 +26,7 @@ use crate::tools::task::runtime::SubagentRuntime;
use crate::tools::todo_write::TodoItem; use crate::tools::todo_write::TodoItem;
use crate::tools::{ use crate::tools::{
DefaultSubAgentRuntime, InMemoryTaskRepository, NoopSessionMessageSender, SessionMessageSender, DefaultSubAgentRuntime, InMemoryTaskRepository, NoopSessionMessageSender, SessionMessageSender,
SubAgentRuntimeConfig, SubagentCatalog, TaskTool, ToolRegistry, SubAgentRuntime, SubAgentRuntimeConfig, SubagentCatalog, TaskTool, ToolRegistry,
}; };
use super::agent_factory::AgentFactory; use super::agent_factory::AgentFactory;
@ -56,6 +56,7 @@ pub(crate) fn build_session_manager(
maintenance_config: MemoryMaintenanceConfig, maintenance_config: MemoryMaintenanceConfig,
session_ttl_hours: Option<u64>, session_ttl_hours: Option<u64>,
mcp_config: crate::mcp::McpConfig, mcp_config: crate::mcp::McpConfig,
mcp_tool_timeout_secs: u64,
bus: Option<Arc<MessageBus>>, bus: Option<Arc<MessageBus>>,
model_resolver: Arc<ModelResolver>, model_resolver: Arc<ModelResolver>,
compaction_config: CompactionConfig, compaction_config: CompactionConfig,
@ -66,6 +67,7 @@ pub(crate) fn build_session_manager(
Option<Arc<McpClientManager>>, Option<Arc<McpClientManager>>,
Arc<SubagentRuntime>, Arc<SubagentRuntime>,
Arc<ModelSelectionStore>, Arc<ModelSelectionStore>,
Option<Arc<dyn SubAgentRuntime>>,
), ),
AgentError, AgentError,
> { > {
@ -84,6 +86,7 @@ pub(crate) fn build_session_manager(
maintenance_config, maintenance_config,
session_ttl_hours, session_ttl_hours,
mcp_config, mcp_config,
mcp_tool_timeout_secs,
bus, bus,
model_resolver, model_resolver,
compaction_config, compaction_config,
@ -106,6 +109,7 @@ pub(crate) fn build_session_manager_with_sender(
maintenance_config: MemoryMaintenanceConfig, maintenance_config: MemoryMaintenanceConfig,
session_ttl_hours: Option<u64>, session_ttl_hours: Option<u64>,
mcp_config: crate::mcp::McpConfig, mcp_config: crate::mcp::McpConfig,
mcp_tool_timeout_secs: u64,
bus: Option<Arc<MessageBus>>, bus: Option<Arc<MessageBus>>,
model_resolver: Arc<ModelResolver>, model_resolver: Arc<ModelResolver>,
compaction_config: CompactionConfig, compaction_config: CompactionConfig,
@ -116,6 +120,7 @@ pub(crate) fn build_session_manager_with_sender(
Option<Arc<McpClientManager>>, Option<Arc<McpClientManager>>,
Arc<SubagentRuntime>, Arc<SubagentRuntime>,
Arc<ModelSelectionStore>, Arc<ModelSelectionStore>,
Option<Arc<dyn SubAgentRuntime>>,
), ),
AgentError, AgentError,
> { > {
@ -192,6 +197,7 @@ pub(crate) fn build_session_manager_with_sender(
manager.clone(), manager.clone(),
server_key.clone(), server_key.clone(),
tool_info, tool_info,
mcp_tool_timeout_secs,
); );
mcp_tools_for_subagents.push(wrapper); mcp_tools_for_subagents.push(wrapper);
} }
@ -205,10 +211,11 @@ pub(crate) fn build_session_manager_with_sender(
} }
// Create SubAgentRuntime (if task tool is enabled) // Create SubAgentRuntime (if task tool is enabled)
let (factory, task_repository, subagent_runtime): ( let (factory, task_repository, subagent_runtime, subagent_executor): (
_, _,
Arc<dyn TaskRepository>, Arc<dyn TaskRepository>,
Arc<SubagentRuntime>, Arc<SubagentRuntime>,
Option<Arc<dyn SubAgentRuntime>>,
) = if task_config.enabled { ) = if task_config.enabled {
let task_repository = Arc::new(InMemoryTaskRepository::new()); let task_repository = Arc::new(InMemoryTaskRepository::new());
// Build subagent tools with MCP tools (task tool registered separately below) // Build subagent tools with MCP tools (task tool registered separately below)
@ -233,6 +240,7 @@ pub(crate) fn build_session_manager_with_sender(
default_max_execution_secs: task_config.max_execution_secs, default_max_execution_secs: task_config.max_execution_secs,
ttl_hours: task_config.ttl_hours, ttl_hours: task_config.ttl_hours,
max_nesting_depth: task_config.max_nesting_depth, max_nesting_depth: task_config.max_nesting_depth,
max_concurrent: task_config.max_concurrent,
}; };
let default_subagent_runtime = Arc::new(DefaultSubAgentRuntime::new( let default_subagent_runtime = Arc::new(DefaultSubAgentRuntime::new(
@ -256,10 +264,14 @@ pub(crate) fn build_session_manager_with_sender(
)); ));
} }
let subagent_executor: Option<Arc<dyn SubAgentRuntime>> =
Some(default_subagent_runtime.clone());
( (
factory.with_subagent_runtime(default_subagent_runtime), factory.with_subagent_runtime(default_subagent_runtime),
task_repository, task_repository,
subagent_runtime, subagent_runtime,
subagent_executor,
) )
} else { } else {
// task_config 未启用时仍创建 subagent_runtime供 API 使用) // task_config 未启用时仍创建 subagent_runtime供 API 使用)
@ -268,6 +280,7 @@ pub(crate) fn build_session_manager_with_sender(
factory, factory,
Arc::new(InMemoryTaskRepository::new()), Arc::new(InMemoryTaskRepository::new()),
subagent_runtime, subagent_runtime,
None,
) )
}; };
@ -308,6 +321,8 @@ pub(crate) fn build_session_manager_with_sender(
let prompt_repository: Arc<dyn PromptInjectionRepository> = store.clone(); let prompt_repository: Arc<dyn PromptInjectionRepository> = store.clone();
let model_selections = Arc::new(ModelSelectionStore::new()); let model_selections = Arc::new(ModelSelectionStore::new());
let observer: Arc<dyn crate::observability::Observer> =
crate::observability::metrics::default_observer();
let agent_factory = AgentFactory::new( let agent_factory = AgentFactory::new(
tools.clone(), tools.clone(),
skills.clone(), skills.clone(),
@ -318,6 +333,7 @@ pub(crate) fn build_session_manager_with_sender(
model_resolver.clone(), model_resolver.clone(),
model_selections.clone(), model_selections.clone(),
compaction_config, compaction_config,
Some(observer),
); );
let session_factory = SessionFactory::new( let session_factory = SessionFactory::new(
provider_config.clone(), provider_config.clone(),
@ -360,5 +376,6 @@ pub(crate) fn build_session_manager_with_sender(
mcp_manager, mcp_manager,
subagent_runtime, subagent_runtime,
model_selections, model_selections,
subagent_executor,
)) ))
} }

View File

@ -47,7 +47,10 @@ impl ScheduledAgentTaskService {
.unwrap_or_else(|| "scheduler".to_string()); .unwrap_or_else(|| "scheduler".to_string());
let provider_config = self.provider_configs.select(options.agent.as_deref())?; let provider_config = self.provider_configs.select(options.agent.as_deref())?;
AgentExecutionService::new(self.show_tool_results) // 定时任务没有入站消息,在此生成独立 trace_id 以贯穿 agent → tool → outbound
let trace_id = crate::observability::tracing_ctx::new_trace_id();
let mut outbound_messages = AgentExecutionService::new(self.show_tool_results)
.prepare_and_execute_scheduled_task(ScheduledExecutionRequest { .prepare_and_execute_scheduled_task(ScheduledExecutionRequest {
session, session,
channel_name, channel_name,
@ -59,7 +62,15 @@ impl ScheduledAgentTaskService {
system_prompt: options.system_prompt.as_deref(), system_prompt: options.system_prompt.as_deref(),
metadata: &options.metadata, metadata: &options.metadata,
fresh_session: options.fresh_session, fresh_session: options.fresh_session,
trace_id: trace_id.clone(),
}) })
.await .await?;
// 将 trace_id 透传到出站消息,保持端到端追踪贯通
for msg in &mut outbound_messages {
msg.trace_id = trace_id.clone();
}
Ok(outbound_messages)
} }
} }

View File

@ -14,10 +14,12 @@ use crate::storage::{
use crate::tools::ToolRegistry; use crate::tools::ToolRegistry;
use crate::tools::task::repository::TaskRepository; use crate::tools::task::repository::TaskRepository;
use crate::tools::task::runtime::SubagentRuntime; use crate::tools::task::runtime::SubagentRuntime;
use crate::tools::task::SubagentResult;
use crate::tools::WaitCoordinator;
use async_trait::async_trait; use async_trait::async_trait;
use std::collections::HashMap; use std::collections::HashMap;
use std::sync::Arc; use std::sync::Arc;
use tokio::sync::{Mutex, mpsc}; use tokio::sync::{Mutex, Notify, mpsc};
use uuid::Uuid; use uuid::Uuid;
use super::agent_factory::{AgentBuildRequest, AgentFactory}; use super::agent_factory::{AgentBuildRequest, AgentFactory};
@ -61,6 +63,7 @@ pub struct BusToolCallEmitter {
metadata: HashMap<String, String>, metadata: HashMap<String, String>,
store: Arc<SessionStore>, store: Arc<SessionStore>,
stream_message_id: parking_lot::Mutex<Option<String>>, stream_message_id: parking_lot::Mutex<Option<String>>,
trace_id: String,
} }
impl BusToolCallEmitter { impl BusToolCallEmitter {
@ -70,6 +73,7 @@ impl BusToolCallEmitter {
chat_id: impl Into<String>, chat_id: impl Into<String>,
metadata: HashMap<String, String>, metadata: HashMap<String, String>,
store: Arc<SessionStore>, store: Arc<SessionStore>,
trace_id: impl Into<String>,
) -> Self { ) -> Self {
Self { Self {
bus, bus,
@ -78,6 +82,7 @@ impl BusToolCallEmitter {
metadata, metadata,
store, store,
stream_message_id: parking_lot::Mutex::new(None), stream_message_id: parking_lot::Mutex::new(None),
trace_id: trace_id.into(),
} }
} }
} }
@ -85,7 +90,7 @@ impl BusToolCallEmitter {
#[async_trait] #[async_trait]
impl EmittedMessageHandler for BusToolCallEmitter { impl EmittedMessageHandler for BusToolCallEmitter {
async fn handle(&self, message: ChatMessage) { async fn handle(&self, message: ChatMessage) {
for outbound in OutboundMessage::from_chat_message( for mut outbound in OutboundMessage::from_chat_message(
&self.channel_name, &self.channel_name,
&self.chat_id, &self.chat_id,
None, // session_id None, // session_id
@ -93,6 +98,7 @@ impl EmittedMessageHandler for BusToolCallEmitter {
&self.metadata, &self.metadata,
&message, &message,
) { ) {
outbound.trace_id = self.trace_id.clone();
if let Err(error) = self.bus.publish_outbound(outbound).await { if let Err(error) = self.bus.publish_outbound(outbound).await {
match error { match error {
crate::bus::BusError::Dropped => { crate::bus::BusError::Dropped => {
@ -111,7 +117,7 @@ impl EmittedMessageHandler for BusToolCallEmitter {
if let Some(ms) = duration_ms { if let Some(ms) = duration_ms {
metadata.insert("tool_duration_ms".to_string(), ms.to_string()); metadata.insert("tool_duration_ms".to_string(), ms.to_string());
} }
for outbound in OutboundMessage::from_chat_message( for mut outbound in OutboundMessage::from_chat_message(
&self.channel_name, &self.channel_name,
&self.chat_id, &self.chat_id,
None, // session_id None, // session_id
@ -119,6 +125,7 @@ impl EmittedMessageHandler for BusToolCallEmitter {
&metadata, &metadata,
&message, &message,
) { ) {
outbound.trace_id = self.trace_id.clone();
if let Err(error) = self.bus.publish_outbound(outbound).await { if let Err(error) = self.bus.publish_outbound(outbound).await {
match error { match error {
crate::bus::BusError::Dropped => { crate::bus::BusError::Dropped => {
@ -147,7 +154,7 @@ impl EmittedMessageHandler for BusToolCallEmitter {
}; };
// Empty content + no reasoning = stream end signal // Empty content + no reasoning = stream end signal
let outbound = if delta.content.is_empty() && delta.reasoning_content.is_none() { let mut outbound = if delta.content.is_empty() && delta.reasoning_content.is_none() {
OutboundMessage::stream_end( OutboundMessage::stream_end(
&self.channel_name, &self.channel_name,
&self.chat_id, &self.chat_id,
@ -166,6 +173,7 @@ impl EmittedMessageHandler for BusToolCallEmitter {
self.metadata.clone(), self.metadata.clone(),
) )
}; };
outbound.trace_id = self.trace_id.clone();
if let Err(error) = self.bus.publish_outbound(outbound).await { if let Err(error) = self.bus.publish_outbound(outbound).await {
match error { match error {
@ -302,6 +310,7 @@ impl Session {
model_resolver, model_resolver,
Arc::new(super::model_selection::ModelSelectionStore::new()), Arc::new(super::model_selection::ModelSelectionStore::new()),
crate::config::CompactionConfig::default(), crate::config::CompactionConfig::default(),
None,
); );
Self::with_factories( Self::with_factories(
channel_name, channel_name,
@ -406,12 +415,17 @@ impl Session {
/// 确保指定 topic 的历史已加载到内存。 /// 确保指定 topic 的历史已加载到内存。
/// 按 topic_id 键化查找,已存在则直接返回,否则从 DB 加载。 /// 按 topic_id 键化查找,已存在则直接返回,否则从 DB 加载。
/// 加载后对账 pending_subagents DB替换过时的 "running" 占位。
pub fn ensure_chat_loaded( pub fn ensure_chat_loaded(
&mut self, &mut self,
chat_id: &str, chat_id: &str,
topic_id: Option<&str>, topic_id: Option<&str>,
) -> Result<(), AgentError> { ) -> Result<(), AgentError> {
self.history.ensure_chat_loaded(chat_id, topic_id) self.history.ensure_chat_loaded(chat_id, topic_id)?;
if let Some(tid) = topic_id {
self.reconcile_running_placeholders(tid);
}
Ok(())
} }
pub fn ensure_agent_prompt_before_user_message( pub fn ensure_agent_prompt_before_user_message(
@ -580,19 +594,147 @@ impl Session {
self.history.topic_serial_lock(topic_id) self.history.topic_serial_lock(topic_id)
} }
/// 确保该 topic 的 sub_done 队列已创建(与 topic_serial_lock 同步初始化)。
pub(crate) fn ensure_sub_done_channel(&mut self, topic_id: &str) {
self.history.ensure_sub_done_channel(topic_id);
}
/// 获取该 topic 的 sub_done 队列 sender用于后台子代理发送结果
#[allow(dead_code)]
pub(crate) fn sub_done_sender(
&mut self,
topic_id: &str,
) -> Option<mpsc::Sender<SubagentResult>> {
self.history.sub_done_sender(topic_id)
}
/// 取出该 topic 的 sub_done 队列 receiverwait 工具进入 select! 前调用)。
pub(crate) fn take_sub_done_receiver(
&mut self,
topic_id: &str,
) -> Option<mpsc::Receiver<SubagentResult>> {
self.history.take_sub_done_receiver(topic_id)
}
/// 归还该 topic 的 sub_done 队列 receiverwait 工具 select! 结束后调用)。
pub(crate) fn restore_sub_done_receiver(
&mut self,
topic_id: &str,
rx: mpsc::Receiver<SubagentResult>,
) {
self.history.restore_sub_done_receiver(topic_id, rx);
}
/// 获取或创建该 topic 的 wait 唤醒信号。
pub(crate) fn wait_wakeup(&mut self, topic_id: &str) -> Arc<Notify> {
self.history.wait_wakeup(topic_id)
}
/// 设置该 topic 的等待状态。
pub(crate) fn set_waiting(&mut self, topic_id: &str, waiting: bool) {
self.history.set_waiting(topic_id, waiting);
}
/// 检查该 topic 是否处于等待状态。
pub(crate) fn is_waiting(&self, topic_id: &str) -> bool {
self.history.is_waiting(topic_id)
}
/// 按 topic_id 从 DB 重新加载历史到内存 /// 按 topic_id 从 DB 重新加载历史到内存
pub(crate) fn reload_topic_history( pub(crate) fn reload_topic_history(
&mut self, &mut self,
chat_id: &str, chat_id: &str,
topic_id: &str, topic_id: &str,
) -> Result<(), AgentError> { ) -> Result<(), AgentError> {
self.history.reload_topic_history(chat_id, topic_id) self.history.reload_topic_history(chat_id, topic_id)?;
self.reconcile_running_placeholders(topic_id);
Ok(())
}
/// 对账 pending_subagents DB将内存 history 中过时的 "running" 占位
/// 替换为 DB 中的实际状态。
///
/// 场景服务器崩溃重启后history 中仍保留 "running" 占位,
/// 但 DB 中该子代理状态可能已被启动扫描标记为 "interrupted"。
/// 若不对账LLM 会看到 "running" → 调用 wait_for_subagents →
/// query_pending_task_ids 返回空DB 已非 running→ 返回 "no pending" →
/// LLM 困惑history 说 running 但 wait 说无 pending。
///
/// 支持两种 content 格式:
/// - JSON: `{"status":"running","task_id":"xxx",...}`(当前 task 工具返回格式)
/// - 纯文本: `running, task_id=xxx. ...`(旧格式,向后兼容)
///
/// 仅修改内存缓存,不持久化到 DB每次从 DB 加载时重新对账,幂等)。
fn reconcile_running_placeholders(&mut self, topic_id: &str) {
let pending = match self.store.list_pending_subagents(topic_id, None) {
Ok(records) => records,
Err(e) => {
tracing::warn!(
error = %e,
topic_id = %topic_id,
"Failed to query pending_subagents for reconciliation"
);
return;
}
};
if pending.is_empty() {
return;
}
let status_map: std::collections::HashMap<&str, &str> = pending
.iter()
.map(|r| (r.task_id.as_str(), r.status.as_str()))
.collect();
let history = self.history.get_or_create_history(topic_id);
let mut reconciled = 0;
for msg in history.iter_mut() {
if msg.role != "tool" {
continue;
}
// 尝试提取 task_id同时支持 JSON 和纯文本格式)
let (task_id, is_json) = match extract_task_id_from_content(&msg.content) {
Some(id) => id,
None => continue,
};
let actual_status = match status_map.get(task_id.as_str()) {
Some(s) => *s,
None => continue, // 记录不存在(已清理),保留原占位
};
if actual_status == "running" {
continue; // 仍在运行,保留占位
}
// 替换为实际状态
msg.content = format_reconciled_content(&task_id, actual_status, is_json);
reconciled += 1;
}
if reconciled > 0 {
tracing::info!(
topic_id = %topic_id,
reconciled_count = reconciled,
"Reconciled stale 'running' placeholders with DB status"
);
}
} }
pub(crate) fn store(&self) -> Arc<dyn ConversationRepository> { pub(crate) fn store(&self) -> Arc<dyn ConversationRepository> {
self.history.conversations() self.history.conversations()
} }
/// 获取底层 SessionStore用于 pending_subagents 查询等)。
/// 与 `store()` 不同:后者返回 ConversationRepository trait object
/// 此方法返回具体的 SessionStore 类型,暴露 pending_subagents 等 CRUD。
pub(crate) fn session_store(&self) -> Arc<SessionStore> {
self.store.clone()
}
pub fn record_skill_offer(&self, chat_id: &str) -> Result<(), AgentError> { pub fn record_skill_offer(&self, chat_id: &str) -> Result<(), AgentError> {
if self.skills.is_empty() { if self.skills.is_empty() {
return Ok(()); return Ok(());
@ -613,6 +755,8 @@ impl Session {
sender_id: Option<&str>, sender_id: Option<&str>,
message_id: Option<&str>, message_id: Option<&str>,
explicit_topic_id: Option<&str>, explicit_topic_id: Option<&str>,
trace_id: &str,
wait_coordinator: Option<Arc<dyn WaitCoordinator>>,
) -> Result<AgentLoop, AgentError> { ) -> Result<AgentLoop, AgentError> {
self.create_agent_with_provider_config( self.create_agent_with_provider_config(
chat_id, chat_id,
@ -621,6 +765,8 @@ impl Session {
message_id, message_id,
self.provider_config.clone(), self.provider_config.clone(),
explicit_topic_id, explicit_topic_id,
trace_id,
wait_coordinator,
) )
} }
@ -632,6 +778,8 @@ impl Session {
message_id: Option<&str>, message_id: Option<&str>,
provider_config: LLMProviderConfig, provider_config: LLMProviderConfig,
explicit_topic_id: Option<&str>, explicit_topic_id: Option<&str>,
trace_id: &str,
wait_coordinator: Option<Arc<dyn WaitCoordinator>>,
) -> Result<AgentLoop, AgentError> { ) -> Result<AgentLoop, AgentError> {
// 优先使用显式传入的 topic_id回退到当前 chat 的活跃 topic // 优先使用显式传入的 topic_id回退到当前 chat 的活跃 topic
let topic_id = explicit_topic_id let topic_id = explicit_topic_id
@ -648,6 +796,12 @@ impl Session {
None => self.pending_cancel_tokens.remove(session_chat_id), None => self.pending_cancel_tokens.remove(session_chat_id),
}; };
// 获取该 topic 的 sub_done 队列 sender用于异步子代理完成回调
// 仅主 agent 注入;无 topic 时为 None走同步路径
let sub_done_sender = topic_id
.as_deref()
.and_then(|tid| self.history.sub_done_sender(tid));
self.agent_factory.create(AgentBuildRequest { self.agent_factory.create(AgentBuildRequest {
channel_name: &self.channel_name, channel_name: &self.channel_name,
session_chat_id, session_chat_id,
@ -657,10 +811,98 @@ impl Session {
message_id, message_id,
provider_config, provider_config,
cancel_token, cancel_token,
trace_id: Some(trace_id.to_string()),
sub_done_sender,
wait_coordinator,
}) })
} }
} }
/// 从 tool result content 中提取 task_id同时支持 JSON 和纯文本格式。
///
/// JSON 格式: `{"status":"running","task_id":"xxx",...}`
/// 纯文本格式: `running, task_id=xxx. ...`
///
/// 返回 (task_id, is_json)。仅当 status=="running" 时才提取(只对账 running 占位)。
fn extract_task_id_from_content(content: &str) -> Option<(String, bool)> {
// 尝试 JSON 格式
let json_start = content.find('{')?;
if let Ok(parsed) = serde_json::from_str::<serde_json::Value>(&content[json_start..]) {
let status = parsed.get("status").and_then(|v| v.as_str())?;
if status != "running" {
return None;
}
let task_id = parsed.get("task_id").and_then(|v| v.as_str())?;
if task_id.is_empty() {
return None;
}
return Some((task_id.to_string(), true));
}
// 回退到纯文本格式(向后兼容)
let prefix = "running, task_id=";
let rest = content.strip_prefix(prefix)?;
let end = rest.find('.').unwrap_or(rest.len());
let task_id = &rest[..end];
if task_id.is_empty() {
None
} else {
Some((task_id.to_string(), false))
}
}
/// 根据子代理的实际状态生成替换内容。
///
/// JSON 格式:更新 JSON 中的 status 字段(保持前端 parseTaskResult 兼容)。
/// 纯文本格式:替换为描述性文本(向后兼容)。
fn format_reconciled_content(task_id: &str, status: &str, is_json: bool) -> String {
if is_json {
// 更新 JSON 中的 status 字段,保持前端 parseTaskResult 能正确解析
let placeholder = match status {
"interrupted" => format!(
"Subagent {} was interrupted (server restart). Result is unavailable.",
task_id
),
"completed" => format!(
"Subagent {} has completed. Call wait_for_subagents to retrieve the result.",
task_id
),
"failed" => format!(
"Subagent {} has failed. Call wait_for_subagents to retrieve error details.",
task_id
),
"timeout" => format!("Subagent {} timed out.", task_id),
"cancelled" => format!("Subagent {} was cancelled.", task_id),
other => format!("Subagent {} status: {}.", task_id, other),
};
serde_json::json!({
"status": status,
"summary": placeholder,
"output": placeholder,
"task_id": task_id,
})
.to_string()
} else {
match status {
"interrupted" => format!(
"Subagent {} was interrupted (server restart). Result is unavailable.",
task_id
),
"completed" => format!(
"Subagent {} has completed. Call wait_for_subagents to retrieve the result.",
task_id
),
"failed" => format!(
"Subagent {} has failed. Call wait_for_subagents to retrieve error details.",
task_id
),
"timeout" => format!("Subagent {} timed out.", task_id),
"cancelled" => format!("Subagent {} was cancelled.", task_id),
other => format!("Subagent {} status: {}.", task_id, other),
}
}
}
/// SessionManager 管理所有 Session按 channel_name 路由 /// SessionManager 管理所有 Session按 channel_name 路由
#[derive(Clone)] #[derive(Clone)]
pub struct SessionManager { pub struct SessionManager {
@ -747,11 +989,12 @@ impl SessionManager {
maintenance_config, maintenance_config,
session_ttl_hours, session_ttl_hours,
mcp_config, mcp_config,
300,
None, None,
model_resolver, model_resolver,
crate::config::CompactionConfig::default(), crate::config::CompactionConfig::default(),
) )
.map(|(session_manager, _, _, _, _)| session_manager) .map(|(session_manager, _, _, _, _, _)| session_manager)
} }
pub fn tools(&self) -> Arc<ToolRegistry> { pub fn tools(&self) -> Arc<ToolRegistry> {
@ -907,6 +1150,7 @@ impl SessionManager {
media: Vec<crate::bus::MediaItem>, media: Vec<crate::bus::MediaItem>,
live_emitter: Option<Arc<dyn EmittedMessageHandler>>, live_emitter: Option<Arc<dyn EmittedMessageHandler>>,
topic_id: Option<&str>, topic_id: Option<&str>,
trace_id: &str,
) -> Result<Vec<OutboundMessage>, AgentError> { ) -> Result<Vec<OutboundMessage>, AgentError> {
self.messages self.messages
.handle_message( .handle_message(
@ -917,6 +1161,7 @@ impl SessionManager {
media, media,
live_emitter, live_emitter,
topic_id, topic_id,
trace_id,
) )
.await .await
} }
@ -1304,6 +1549,7 @@ mod tests {
Vec::new(), Vec::new(),
None, None,
None, None,
"test-trace",
) )
.await .await
.unwrap(); .unwrap();
@ -2094,8 +2340,14 @@ mod tests {
async fn test_bus_tool_call_emitter_emits_completed_tool_results() { async fn test_bus_tool_call_emitter_emits_completed_tool_results() {
let store = Arc::new(SessionStore::in_memory().unwrap()); let store = Arc::new(SessionStore::in_memory().unwrap());
let bus = MessageBus::new(4); let bus = MessageBus::new(4);
let emitter = let emitter = BusToolCallEmitter::new(
BusToolCallEmitter::new(bus.clone(), "test-channel", "chat-1", HashMap::new(), store); bus.clone(),
"test-channel",
"chat-1",
HashMap::new(),
store,
"test-trace-id",
);
emitter emitter
.handle(ChatMessage::tool("call-1", "calculator", "2")) .handle(ChatMessage::tool("call-1", "calculator", "2"))

View File

@ -1,11 +1,14 @@
use std::collections::{HashMap, HashSet}; use std::collections::{HashMap, HashSet};
use std::sync::Arc; use std::sync::Arc;
use tokio::sync::{Notify, mpsc};
use crate::agent::AgentError; use crate::agent::AgentError;
use crate::bus::ChatMessage; use crate::bus::ChatMessage;
use crate::storage::{ use crate::storage::{
ConversationRepository, SessionRecord, SkillEventRepository, persistent_session_id, ConversationRepository, SessionRecord, SkillEventRepository, persistent_session_id,
}; };
use crate::tools::task::SubagentResult;
/// 内存中缓存的 topic 历史上限。 /// 内存中缓存的 topic 历史上限。
/// 超过此值时,驱逐非活跃 topic不在 chat_topic_ids 当前引用中的 topic /// 超过此值时,驱逐非活跃 topic不在 chat_topic_ids 当前引用中的 topic
@ -35,6 +38,23 @@ pub(crate) struct SessionHistory {
/// 防止并发 loop 操作同一历史的不同快照产生交错序列。 /// 防止并发 loop 操作同一历史的不同快照产生交错序列。
/// 不同 topic 之间互不阻塞,支持多话题并发执行。 /// 不同 topic 之间互不阻塞,支持多话题并发执行。
topic_serial_locks: HashMap<String, Arc<tokio::sync::Mutex<()>>>, topic_serial_locks: HashMap<String, Arc<tokio::sync::Mutex<()>>>,
/// per-topic 子代理完成队列的 sender 端。
/// 后台子代理完成后 clone 此 sender 发送 SubagentResult
/// 由 wait_for_subagents 工具的 receiver 端消费。
/// 生命周期与 topic_serial_locks 完全一致。
sub_done_senders: HashMap<String, mpsc::Sender<SubagentResult>>,
/// per-topic 子代理完成队列的 receiver 端。
/// 使用 Option 包装以便 wait_for_subagents 工具在 select! 期间
/// 将其取出(不持有 Session 锁),结束后归还。
sub_done_receivers: HashMap<String, Option<mpsc::Receiver<SubagentResult>>>,
/// per-topic wait 唤醒信号。
/// wait 释放锁进入 select! 后process_one 在注入用户消息到 history 后
/// 调用 notify_one() 唤醒 wait。
wait_wakeups: HashMap<String, Arc<Notify>>,
/// per-topic 等待状态标志。
/// true 表示该 topic 的 agent 正在 wait_for_subagents 中等待(已释放 serial_lock
/// process_one 持锁后检查此标志true 则注入用户消息 + wakeupfalse 则正常处理。
waiting_flags: HashMap<String, bool>,
conversations: Arc<dyn ConversationRepository>, conversations: Arc<dyn ConversationRepository>,
skill_events: Arc<dyn SkillEventRepository>, skill_events: Arc<dyn SkillEventRepository>,
} }
@ -67,6 +87,20 @@ impl SessionHistory {
if active.contains(tid.as_str()) || self.compression_in_flight.contains(*tid) { if active.contains(tid.as_str()) || self.compression_in_flight.contains(*tid) {
return false; return false;
} }
// 不变量 2lock 实例单射性 — 驱逐会移除 topic_serial_locks/wakeup/sender 等 entry
// 下次访问会创建新实例,破坏原 Arc 持有者wait_coordinator与新建者process_one
// 的串行化。因此必须充分覆盖所有"活跃"语义:
// a) waiting_flag=true → wait_coordinator 正在 select! 等待
// b) sub_done_receivers[tid]=None → receiver 被 take 走wait_coordinator 持有)
// c) topic_serial_lock 被持有 → 有 agent 任务正在处理
// 任一为真都不能驱逐。
if self.waiting_flags.get(*tid).copied().unwrap_or(false) {
return false;
}
if matches!(self.sub_done_receivers.get(*tid), Some(None)) {
// receiver 被 take 走 = wait_coordinator 正在 select! 中
return false;
}
// 检查是否有活跃 agent 任务serial lock 被持有) // 检查是否有活跃 agent 任务serial lock 被持有)
// try_lock 成功 = 锁空闲 = 无活跃任务 = 可驱逐 // try_lock 成功 = 锁空闲 = 无活跃任务 = 可驱逐
// try_lock 失败 = 锁被持有 = 有活跃任务 = 不驱逐 // try_lock 失败 = 锁被持有 = 有活跃任务 = 不驱逐
@ -81,6 +115,11 @@ impl SessionHistory {
if let Some(tid) = to_evict.cloned() { if let Some(tid) = to_evict.cloned() {
let msg_count = self.topic_histories.get(&tid).map(|h| h.len()).unwrap_or(0); let msg_count = self.topic_histories.get(&tid).map(|h| h.len()).unwrap_or(0);
self.topic_histories.remove(&tid); self.topic_histories.remove(&tid);
self.topic_serial_locks.remove(&tid);
self.sub_done_senders.remove(&tid);
self.sub_done_receivers.remove(&tid);
self.wait_wakeups.remove(&tid);
self.waiting_flags.remove(&tid);
tracing::info!( tracing::info!(
topic_id = %tid, topic_id = %tid,
evicted_messages = msg_count, evicted_messages = msg_count,
@ -101,6 +140,10 @@ impl SessionHistory {
chat_topic_ids: HashMap::new(), chat_topic_ids: HashMap::new(),
compression_in_flight: HashSet::new(), compression_in_flight: HashSet::new(),
topic_serial_locks: HashMap::new(), topic_serial_locks: HashMap::new(),
sub_done_senders: HashMap::new(),
sub_done_receivers: HashMap::new(),
wait_wakeups: HashMap::new(),
waiting_flags: HashMap::new(),
conversations, conversations,
skill_events, skill_events,
} }
@ -109,6 +152,9 @@ impl SessionHistory {
/// 获取或创建该 topic 的串行化锁。 /// 获取或创建该 topic 的串行化锁。
/// 同一 topic 的所有消息处理共享同一个锁,保证串行执行; /// 同一 topic 的所有消息处理共享同一个锁,保证串行执行;
/// 不同 topic 之间互不阻塞,支持多话题并发执行。 /// 不同 topic 之间互不阻塞,支持多话题并发执行。
///
/// 同时同步创建该 topic 的 sub_done 队列、wait 唤醒信号和等待标志,
/// 生命周期与 serial_lock 完全一致。
pub(crate) fn topic_serial_lock(&mut self, topic_id: &str) -> Arc<tokio::sync::Mutex<()>> { pub(crate) fn topic_serial_lock(&mut self, topic_id: &str) -> Arc<tokio::sync::Mutex<()>> {
self.topic_serial_locks self.topic_serial_locks
.entry(topic_id.to_string()) .entry(topic_id.to_string())
@ -116,6 +162,68 @@ impl SessionHistory {
.clone() .clone()
} }
/// 确保该 topic 的 sub_done 队列已创建。
/// 应在 topic 初始化时(与 topic_serial_lock 同步)调用。
/// 队列容量为 32足够缓存多个子代理同时完成的结果。
pub(crate) fn ensure_sub_done_channel(&mut self, topic_id: &str) {
if !self.sub_done_senders.contains_key(topic_id) {
let (tx, rx) = mpsc::channel::<SubagentResult>(32);
self.sub_done_senders.insert(topic_id.to_string(), tx);
self.sub_done_receivers
.insert(topic_id.to_string(), Some(rx));
}
}
/// 获取该 topic 的 sub_done 队列 sender用于后台子代理发送结果
/// 调用前应已通过 `ensure_sub_done_channel` 创建队列。
pub(crate) fn sub_done_sender(&mut self, topic_id: &str) -> Option<mpsc::Sender<SubagentResult>> {
self.ensure_sub_done_channel(topic_id);
self.sub_done_senders.get(topic_id).cloned()
}
/// 取出该 topic 的 sub_done 队列 receiver。
/// 由 wait_for_subagents 工具在进入 select! 前调用(需释放 Session 锁)。
/// 调用后 receiver 不在 map 中,需通过 `restore_sub_done_receiver` 归还。
pub(crate) fn take_sub_done_receiver(
&mut self,
topic_id: &str,
) -> Option<mpsc::Receiver<SubagentResult>> {
self.sub_done_receivers
.get_mut(topic_id)
.and_then(|opt| opt.take())
}
/// 归还该 topic 的 sub_done 队列 receiver。
/// 由 wait_for_subagents 工具在 select! 结束后调用。
pub(crate) fn restore_sub_done_receiver(
&mut self,
topic_id: &str,
rx: mpsc::Receiver<SubagentResult>,
) {
self.sub_done_receivers
.insert(topic_id.to_string(), Some(rx));
}
/// 获取或创建该 topic 的 wait 唤醒信号。
pub(crate) fn wait_wakeup(&mut self, topic_id: &str) -> Arc<Notify> {
self.wait_wakeups
.entry(topic_id.to_string())
.or_insert_with(|| Arc::new(Notify::new()))
.clone()
}
/// 设置该 topic 的等待状态。
/// true = agent 正在 wait_for_subagents 中等待(已释放 serial_lock
pub(crate) fn set_waiting(&mut self, topic_id: &str, waiting: bool) {
self.waiting_flags.insert(topic_id.to_string(), waiting);
}
/// 检查该 topic 是否处于等待状态。
/// process_one 持锁后调用true 则走注入路径false 则正常处理。
pub(crate) fn is_waiting(&self, topic_id: &str) -> bool {
self.waiting_flags.get(topic_id).copied().unwrap_or(false)
}
pub(crate) fn persistent_session_id(&self, chat_id: &str) -> String { pub(crate) fn persistent_session_id(&self, chat_id: &str) -> String {
persistent_session_id(&self.channel_name, chat_id) persistent_session_id(&self.channel_name, chat_id)
} }
@ -212,6 +320,11 @@ impl SessionHistory {
// (仅在无活跃任务时安全移除;有活跃任务时 lock 被 Arc clone 持有, // (仅在无活跃任务时安全移除;有活跃任务时 lock 被 Arc clone 持有,
// 移除 HashMap entry 不影响正在使用 lock 的任务) // 移除 HashMap entry 不影响正在使用 lock 的任务)
self.topic_serial_locks.remove(topic_id); self.topic_serial_locks.remove(topic_id);
// 同步清理 sub_done 队列、wait 唤醒信号和等待标志
self.sub_done_senders.remove(topic_id);
self.sub_done_receivers.remove(topic_id);
self.wait_wakeups.remove(topic_id);
self.waiting_flags.remove(topic_id);
} }
/// 清空指定 chat/topic 的内存历史和 DB 消息。 /// 清空指定 chat/topic 的内存历史和 DB 消息。

View File

@ -51,7 +51,8 @@ impl SessionMessageSender for BusSessionMessageSender {
text, text,
None, None,
metadata.clone(), metadata.clone(),
); )
.with_trace_id(context.trace_id.as_deref().unwrap_or(""));
if attachment_count > 0 { if attachment_count > 0 {
outbound.media = request.attachments.clone(); outbound.media = request.attachments.clone();
} }
@ -88,7 +89,8 @@ impl SessionMessageSender for BusSessionMessageSender {
String::new(), String::new(),
None, None,
metadata.clone(), metadata.clone(),
); )
.with_trace_id(context.trace_id.as_deref().unwrap_or(""));
outbound.media = vec![attachment]; outbound.media = vec![attachment];
match self.bus.publish_outbound(outbound).await { match self.bus.publish_outbound(outbound).await {
Ok(()) => { Ok(()) => {

View File

@ -29,6 +29,7 @@ impl SessionMessageService {
media: Vec<MediaItem>, media: Vec<MediaItem>,
live_emitter: Option<Arc<dyn EmittedMessageHandler>>, live_emitter: Option<Arc<dyn EmittedMessageHandler>>,
topic_id: Option<&str>, topic_id: Option<&str>,
trace_id: &str,
) -> Result<Vec<OutboundMessage>, AgentError> { ) -> Result<Vec<OutboundMessage>, AgentError> {
#[cfg(debug_assertions)] #[cfg(debug_assertions)]
{ {
@ -56,6 +57,7 @@ impl SessionMessageService {
media, media,
live_emitter, live_emitter,
topic_id: topic_id.map(|s| s.to_string()), topic_id: topic_id.map(|s| s.to_string()),
trace_id,
}) })
.await?; .await?;

View File

@ -14,7 +14,7 @@ use crate::tools::{
BashTool, CalculatorTool, FileEditTool, FileReadTool, FileWriteTool, HttpRequestTool, BashTool, CalculatorTool, FileEditTool, FileReadTool, FileWriteTool, HttpRequestTool,
MemoryManageTool, MemorySearchTool, SchedulerManageTool, SessionMessageSender, SessionSendTool, MemoryManageTool, MemorySearchTool, SchedulerManageTool, SessionMessageSender, SessionSendTool,
ShellSessionManager, SkillActivateTool, SkillManageTool, SubAgentRuntime, TaskTool, TimeTool, ShellSessionManager, SkillActivateTool, SkillManageTool, SubAgentRuntime, TaskTool, TimeTool,
TodoReadTool, TodoWriteTool, ToolRegistry, WebFetchTool, TodoReadTool, TodoWriteTool, ToolRegistry, WaitForSubagentsTool, WebFetchTool,
}; };
pub(crate) struct ToolRegistryFactory { pub(crate) struct ToolRegistryFactory {
@ -160,6 +160,11 @@ impl ToolRegistryFactory {
if self.is_enabled("task") && self.task_config.enabled { if self.is_enabled("task") && self.task_config.enabled {
if let Some(runtime) = &self.subagent_runtime { if let Some(runtime) = &self.subagent_runtime {
registry.register(TaskTool::new(runtime.clone(), None)); registry.register(TaskTool::new(runtime.clone(), None));
// 注册 wait_for_subagents 工具(仅主 agent用于等待异步子代理完成
// 默认超时从配置读取LLM 可通过 timeout_secs 参数覆盖
registry.register(WaitForSubagentsTool::new(
self.task_config.wait_default_timeout_secs,
));
} }
} }

View File

@ -0,0 +1,291 @@
use std::sync::Arc;
use std::time::Duration;
use async_trait::async_trait;
use tokio::sync::{Mutex, OwnedMutexGuard, watch};
use tokio::time::sleep;
use crate::gateway::session::Session;
use crate::storage::SessionStore;
use crate::tools::task::SubagentResult;
use crate::tools::{WaitCoordinator, WaitEvent};
/// 基于 Session 的 wait 协调器实现。
///
/// 持有 serial_lock 的 guard slot可通过 take/drop 释放锁,通过 put 回填新 guard
/// 以及 Session 引用(用于管理 sub_done_receiver、wait_wakeup、waiting_flag
///
/// wait() 流程:
/// 1. 设置 waiting=true让 process_one 走注入路径)
/// 2. 取出 sub_done_receiver从 Session 中 takeselect! 期间不持有 Session 锁)
/// 3. 获取 wait_wakeupArc<Notify>clone 后不持有 Session 锁)
/// 4. 释放 serial_lock从 guard_slot take 并 drop guard
/// 5. select! { sub_done_q.recv(), wakeup.notified(), timeout }
/// 6. 重新获取 serial_lockserial_lock.lock_owned().await
/// 7. 回填 guard 到 guard_slot
/// 8. 设置 waiting=false先获取锁后清除避免 TOCTOU
/// 9. 归还 sub_done_receiver
pub struct SessionWaitCoordinator {
/// Session 引用Arc<Mutex<Session>>),用于访问 SessionHistory 的队列和状态
session: Arc<Mutex<Session>>,
/// serial_lock guard 的存储槽。
/// 执行路径execution.rs获取锁后将 guard 存入此槽;
/// wait() 取出并 drop 以释放锁,重获取后回填新 guard。
guard_slot: Arc<Mutex<Option<OwnedMutexGuard<()>>>>,
/// serial_lock 本体Arc<tokio::sync::Mutex<()>>),用于重获取锁
serial_lock: Arc<tokio::sync::Mutex<()>>,
/// SessionStore 引用,用于查询 pending_subagents
store: Arc<SessionStore>,
/// 当前 topic_id
topic_id: String,
}
impl SessionWaitCoordinator {
pub fn new(
session: Arc<Mutex<Session>>,
guard_slot: Arc<Mutex<Option<OwnedMutexGuard<()>>>>,
serial_lock: Arc<tokio::sync::Mutex<()>>,
store: Arc<SessionStore>,
topic_id: String,
) -> Self {
Self {
session,
guard_slot,
serial_lock,
store,
topic_id,
}
}
}
#[async_trait]
impl WaitCoordinator for SessionWaitCoordinator {
fn query_pending_task_ids(&self) -> Vec<String> {
self.store
.list_pending_subagents(&self.topic_id, Some("running"))
.map(|records| records.into_iter().map(|r| r.task_id).collect())
.unwrap_or_default()
}
async fn try_drain_queued_results(&self) -> Vec<SubagentResult> {
// 取出 receiver → try_recv 排空 → 归还 receiver
// 安全性此方法在执行路径中被调用serial_lock 已持有),
// 无其他代码并发访问 receiver。
let rx = {
let mut session = self.session.lock().await;
session.take_sub_done_receiver(&self.topic_id)
};
let mut results = Vec::new();
if let Some(mut rx) = rx {
while let Ok(result) = rx.try_recv() {
results.push(result);
}
// 归还 receiver即使已排空仍需放回供后续 wait() 使用)
let mut session = self.session.lock().await;
session.restore_sub_done_receiver(&self.topic_id, rx);
}
if !results.is_empty() {
tracing::debug!(
topic_id = %self.topic_id,
drained_count = results.len(),
"Drained buffered subagent results from sub_done_q"
);
}
results
}
async fn wait(
&self,
timeout: Duration,
cancel_rx: Option<watch::Receiver<()>>,
) -> WaitEvent {
// 1. 设置 waiting=true
{
let mut session = self.session.lock().await;
session.set_waiting(&self.topic_id, true);
}
// 2. 取出 sub_done_receiverselect! 期间不持有 Session 锁)
let receiver = {
let mut session = self.session.lock().await;
session.take_sub_done_receiver(&self.topic_id)
};
// 3. 获取 wait_wakeupArc<Notify>clone 后不持有 Session 锁)
let wakeup = {
let mut session = self.session.lock().await;
session.wait_wakeup(&self.topic_id)
};
// 3.5. 记录等待前的用户消息数量(用于 wakeup 后提取新注入的消息)
// 直接从 SQLite 读取,不持有任何锁
let user_msg_count_before = self
.store
.load_messages_for_topic(&self.topic_id, None)
.map(|msgs| msgs.iter().filter(|m| m.role == "user").count())
.unwrap_or(0);
// 4. 释放 serial_lock取出 guard 并 drop
{
let mut slot = self.guard_slot.lock().await;
let _ = slot.take(); // drop guard → 释放 serial_lock
}
tracing::debug!(
topic_id = %self.topic_id,
timeout_secs = timeout.as_secs(),
user_msg_count_before,
has_cancel_rx = cancel_rx.is_some(),
"SessionWaitCoordinator: lock released, entering select!"
);
// 5. select! 等待(不持有任何锁)
//
// cancel 分支放在最后biased 排序中最后被 poll
// 确保子代理结果和用户消息优先于取消信号被处理。
// 场景:/stop 后用户立即发消息 → process_one 注入消息 + wakeup
// select! 优先消费 wakeupUserMessage而非 cancelCancelled
// 使已注入的用户消息能被 Agent 处理而非丢失。
//
// 但 cancel_rx.changed() 不会无限阻塞——若无子代理结果、无用户消息,
// cancel 仍是唯一就绪分支,等待被优雅终止。
let event = if let Some(mut rx) = receiver {
let mut cancel_rx = cancel_rx;
tokio::select! {
biased;
result = rx.recv() => {
match result {
Some(subagent_result) => {
let event = WaitEvent::SubagentResult(subagent_result);
let mut session = self.session.lock().await;
session.restore_sub_done_receiver(&self.topic_id, rx);
event
}
None => {
// sender 全部 drop所有 sub_done_sender 被释放)
WaitEvent::Timeout
}
}
}
_ = wakeup.notified() => {
// 用户消息到达process_one 已注入 history 并 wakeup
let mut session = self.session.lock().await;
session.restore_sub_done_receiver(&self.topic_id, rx);
// 提取等待期间新注入的用户消息内容
let new_messages = self.fetch_new_user_messages(user_msg_count_before);
tracing::info!(
topic_id = %self.topic_id,
new_msg_count = new_messages.len(),
"SessionWaitCoordinator: woke up by user message"
);
WaitEvent::UserMessage(new_messages)
}
_ = async {
if let Some(ref mut crx) = cancel_rx {
let _ = crx.changed().await;
} else {
std::future::pending::<()>().await;
}
} => {
// 取消信号到达(/stop→ 归还 receiver返回 Cancelled
tracing::info!(
topic_id = %self.topic_id,
"SessionWaitCoordinator: cancelled by /stop during wait"
);
let mut session = self.session.lock().await;
session.restore_sub_done_receiver(&self.topic_id, rx);
WaitEvent::Cancelled
}
_ = sleep(timeout) => {
let mut session = self.session.lock().await;
session.restore_sub_done_receiver(&self.topic_id, rx);
WaitEvent::Timeout
}
}
} else {
// 无 receivertopic 无 sub_done 队列),直接等待 timeout 或 wakeup
let mut cancel_rx = cancel_rx;
tokio::select! {
biased;
_ = wakeup.notified() => {
let new_messages = self.fetch_new_user_messages(user_msg_count_before);
tracing::info!(
topic_id = %self.topic_id,
new_msg_count = new_messages.len(),
"SessionWaitCoordinator: woke up by user message (no receiver)"
);
WaitEvent::UserMessage(new_messages)
}
_ = async {
if let Some(ref mut crx) = cancel_rx {
let _ = crx.changed().await;
} else {
std::future::pending::<()>().await;
}
} => {
tracing::info!(
topic_id = %self.topic_id,
"SessionWaitCoordinator: cancelled by /stop during wait (no receiver)"
);
WaitEvent::Cancelled
}
_ = sleep(timeout) => WaitEvent::Timeout,
}
};
// 6. 重新获取 serial_lock
// lock_owned 消费 Arc<Self>,需 clone 保留 self.serial_lock 供后续可能的重入
//
// 取消场景下此处可能阻塞——如果 process_one 正持有锁注入用户消息,
// 需等其释放后才能重获取。这是正确行为:确保 is_waiting 清除与
// process_one 的注入互斥,避免 TOCTOU。
let new_guard = self.serial_lock.clone().lock_owned().await;
// 7. 回填 guard 到 guard_slot
{
let mut slot = self.guard_slot.lock().await;
*slot = Some(new_guard);
}
// 8. 设置 waiting=false先获取锁后清除避免 TOCTOU
// 注意serial_lock 已在步骤 6 获取,此时 process_one 无法获取锁,
// 所以清除 waiting 是安全的。
{
let mut session = self.session.lock().await;
session.set_waiting(&self.topic_id, false);
}
tracing::debug!(
topic_id = %self.topic_id,
"SessionWaitCoordinator: lock reacquired, waiting cleared"
);
event
}
}
impl SessionWaitCoordinator {
/// 提取等待期间新注入的用户消息内容。
/// 通过对比等待前的用户消息数量,从 SQLite 中取出新增的用户消息。
fn fetch_new_user_messages(&self, count_before: usize) -> Vec<String> {
match self.store.load_messages_for_topic(&self.topic_id, None) {
Ok(msgs) => msgs
.iter()
.filter(|m| m.role == "user")
.skip(count_before)
.map(|m| m.content.clone())
.collect(),
Err(e) => {
tracing::warn!(
error = %e,
topic_id = %self.topic_id,
"Failed to load messages for fetching new user messages"
);
Vec::new()
}
}
}
}

View File

@ -397,6 +397,7 @@ async fn handle_inbound(
media, media,
metadata: HashMap::new(), metadata: HashMap::new(),
forwarded_metadata: HashMap::new(), forwarded_metadata: HashMap::new(),
trace_id: crate::observability::tracing_ctx::new_trace_id(),
}) })
.await .await
.map_err(|error| AgentError::Other(error.to_string()))?; .map_err(|error| AgentError::Other(error.to_string()))?;
@ -534,6 +535,7 @@ async fn handle_inbound(
router.register(Box::new(StopExecutionCommandHandler::new( router.register(Box::new(StopExecutionCommandHandler::new(
state.cancel_manager.clone(), state.cancel_manager.clone(),
state.session_manager.clone(), state.session_manager.clone(),
state.subagent_executor.clone(),
))); )));
// 构建命令上下文 // 构建命令上下文

View File

@ -3,9 +3,11 @@ use chrono_tz::Tz;
use std::path::PathBuf; use std::path::PathBuf;
use tracing_appender::rolling::{RollingFileAppender, Rotation}; use tracing_appender::rolling::{RollingFileAppender, Rotation};
use tracing_subscriber::{ use tracing_subscriber::{
fmt, fmt::time::FormatTime, layer::SubscriberExt, util::SubscriberInitExt, EnvFilter, fmt, fmt::time::FormatTime, layer::SubscriberExt, util::SubscriberInitExt, EnvFilter, Layer,
}; };
use crate::config::LogFormat;
#[derive(Clone, Copy, Debug)] #[derive(Clone, Copy, Debug)]
struct ConfiguredTimestamp { struct ConfiguredTimestamp {
timezone: Tz, timezone: Tz,
@ -40,7 +42,10 @@ pub fn get_default_config_path() -> PathBuf {
/// Initialize logging with file appender /// Initialize logging with file appender
/// Logs are written to ~/.picobot/logs/ with daily rotation /// Logs are written to ~/.picobot/logs/ with daily rotation
pub fn init_logging(timezone: Tz) { ///
/// `log_format` 控制文件日志格式Text默认或 Json便于日志聚合
/// 控制台始终使用文本格式(便于人读)。
pub fn init_logging(timezone: Tz, log_format: LogFormat) {
use std::sync::Once; use std::sync::Once;
static INIT: Once = Once::new(); static INIT: Once = Once::new();
@ -72,14 +77,28 @@ pub fn init_logging(timezone: Tz) {
// Build subscriber with both console and file output // Build subscriber with both console and file output
let env_filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")); let env_filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info"));
let file_layer = fmt::layer() // 文件层:根据 log_format 选择 text 或 json
.with_writer(file_appender) let file_layer = match log_format {
.with_timer(ConfiguredTimestamp { timezone }) LogFormat::Json => fmt::layer()
.with_ansi(false) .with_writer(file_appender)
.with_target(true) .with_timer(ConfiguredTimestamp { timezone })
.with_level(true) .with_ansi(false)
.with_thread_ids(true); .with_target(true)
.with_level(true)
.with_thread_ids(true)
.json()
.boxed(),
LogFormat::Text => fmt::layer()
.with_writer(file_appender)
.with_timer(ConfiguredTimestamp { timezone })
.with_ansi(false)
.with_target(true)
.with_level(true)
.with_thread_ids(true)
.boxed(),
};
// 控制台层:始终文本格式
let console_layer = fmt::layer() let console_layer = fmt::layer()
.with_timer(ConfiguredTimestamp { timezone }) .with_timer(ConfiguredTimestamp { timezone })
.with_target(true) .with_target(true)
@ -91,7 +110,11 @@ pub fn init_logging(timezone: Tz) {
.with(file_layer) .with(file_layer)
.init(); .init();
tracing::info!("Logging initialized. Log directory: {}", log_dir.display()); tracing::info!(
log_format = ?log_format,
log_dir = %log_dir.display(),
"Logging initialized"
);
} }
/// Initialize logging without file output (console only) /// Initialize logging without file output (console only)

View File

@ -788,13 +788,14 @@ impl McpInitializer {
pub async fn register_tools( pub async fn register_tools(
&mut self, &mut self,
registry: &mut crate::tools::ToolRegistry, registry: &mut crate::tools::ToolRegistry,
timeout_secs: u64,
) -> anyhow::Result<()> { ) -> anyhow::Result<()> {
if let Some(manager) = self.manager.clone() { if let Some(manager) = self.manager.clone() {
// Wait for connections to complete first // Wait for connections to complete first
self.wait_for_connections().await?; self.wait_for_connections().await?;
tracing::info!("Registering MCP tools after connections completed"); tracing::info!("Registering MCP tools after connections completed");
crate::mcp::register_mcp_tools(manager, registry).await?; crate::mcp::register_mcp_tools(manager, registry, timeout_secs).await?;
} }
Ok(()) Ok(())
} }

View File

@ -37,11 +37,18 @@ pub struct McpToolWrapper {
full_name: String, full_name: String,
/// Tool information from MCP server /// Tool information from MCP server
tool_info: Tool, tool_info: Tool,
/// Tool call timeout in seconds (0 = no timeout)
timeout_secs: u64,
} }
impl McpToolWrapper { impl McpToolWrapper {
/// Create a new tool wrapper /// Create a new tool wrapper
pub fn new(manager: Arc<McpClientManager>, server_key: String, tool_info: Tool) -> Self { pub fn new(
manager: Arc<McpClientManager>,
server_key: String,
tool_info: Tool,
timeout_secs: u64,
) -> Self {
let tool_name = tool_info.name.clone().into_owned(); let tool_name = tool_info.name.clone().into_owned();
let raw_name = format!("mcp_{}_{}", server_key, tool_name); let raw_name = format!("mcp_{}_{}", server_key, tool_name);
let full_name = sanitize_tool_name(&raw_name); let full_name = sanitize_tool_name(&raw_name);
@ -61,6 +68,7 @@ impl McpToolWrapper {
tool_name, tool_name,
full_name, full_name,
tool_info, tool_info,
timeout_secs,
} }
} }
@ -98,10 +106,33 @@ impl PicoBotTool for McpToolWrapper {
"Calling MCP tool" "Calling MCP tool"
); );
let result = self let call = self
.manager .manager
.call_tool(&self.server_key, &self.tool_name, args) .call_tool(&self.server_key, &self.tool_name, args);
.await?;
let result = if self.timeout_secs > 0 {
tokio::time::timeout(
std::time::Duration::from_secs(self.timeout_secs),
call,
)
.await
.map_err(|_| {
tracing::warn!(
server_key = %self.server_key,
tool = %self.tool_name,
timeout_secs = self.timeout_secs,
"MCP tool call timed out"
);
anyhow::anyhow!(
"MCP tool '{}' on server '{}' timed out after {}s",
self.tool_name,
self.server_key,
self.timeout_secs
)
})??
} else {
call.await?
};
// Convert MCP CallToolResult to PicoBot ToolResult // Convert MCP CallToolResult to PicoBot ToolResult
let output = extract_text_content(&result); let output = extract_text_content(&result);
@ -147,11 +178,17 @@ fn extract_text_content(result: &rmcp::model::CallToolResult) -> String {
pub async fn register_mcp_tools( pub async fn register_mcp_tools(
manager: Arc<McpClientManager>, manager: Arc<McpClientManager>,
registry: &mut crate::tools::registry::ToolRegistry, registry: &mut crate::tools::registry::ToolRegistry,
timeout_secs: u64,
) -> anyhow::Result<()> { ) -> anyhow::Result<()> {
let all_tools = manager.all_tools().await; let all_tools = manager.all_tools().await;
for (server_key, tool_info) in all_tools { for (server_key, tool_info) in all_tools {
let wrapper = McpToolWrapper::new(manager.clone(), server_key.clone(), tool_info); let wrapper = McpToolWrapper::new(
manager.clone(),
server_key.clone(),
tool_info,
timeout_secs,
);
tracing::info!( tracing::info!(
name = %wrapper.name(), name = %wrapper.name(),
@ -198,7 +235,7 @@ mod tests {
.clone(); .clone();
let tool_info = Tool::new("echo", "Echo tool", schema); let tool_info = Tool::new("echo", "Echo tool", schema);
let wrapper = McpToolWrapper::new(manager, "filesystem".to_string(), tool_info); let wrapper = McpToolWrapper::new(manager, "filesystem".to_string(), tool_info, 300);
assert_eq!(wrapper.name(), "mcp_filesystem_echo"); assert_eq!(wrapper.name(), "mcp_filesystem_echo");
assert_eq!(wrapper.original_name(), "echo"); assert_eq!(wrapper.original_name(), "echo");
assert_eq!(wrapper.server_key(), "filesystem"); assert_eq!(wrapper.server_key(), "filesystem");
@ -216,7 +253,7 @@ mod tests {
.clone(); .clone();
let tool_info = Tool::new("tools.list:read", "Namespaced tool", schema); let tool_info = Tool::new("tools.list:read", "Namespaced tool", schema);
let wrapper = McpToolWrapper::new(manager, "github.api".to_string(), tool_info); let wrapper = McpToolWrapper::new(manager, "github.api".to_string(), tool_info, 300);
// mcp_github.api_tools.list:read → mcp_github_api_tools_list_read // mcp_github.api_tools.list:read → mcp_github_api_tools_list_read
assert_eq!(wrapper.name(), "mcp_github_api_tools_list_read"); assert_eq!(wrapper.name(), "mcp_github_api_tools_list_read");
// Original identifiers preserved for routing // Original identifiers preserved for routing
@ -244,4 +281,49 @@ mod tests {
// Empty stays empty // Empty stays empty
assert_eq!(sanitize_tool_name(""), ""); assert_eq!(sanitize_tool_name(""), "");
} }
#[tokio::test]
async fn test_execute_returns_error_when_server_not_connected() {
// McpClientManager with no connected servers — call should fail immediately
// with "not connected" error, not hang or timeout.
let manager = Arc::new(McpClientManager::new());
let schema: serde_json::Map<String, serde_json::Value> =
serde_json::json!({"type": "object"})
.as_object()
.unwrap()
.clone();
let tool_info = Tool::new("echo", "Echo tool", schema);
let wrapper = McpToolWrapper::new(manager, "ghost".to_string(), tool_info, 300);
let result = wrapper.execute(serde_json::json!({})).await;
assert!(result.is_err());
let msg = result.unwrap_err().to_string();
assert!(
msg.contains("not connected"),
"expected 'not connected' error, got: {msg}"
);
}
#[tokio::test]
async fn test_timeout_zero_does_not_wrap() {
// With timeout_secs = 0, the wrapper should pass through directly
// without tokio::time::timeout. The call still fails because no server
// is connected, but the error path is the non-timeout branch.
let manager = Arc::new(McpClientManager::new());
let schema: serde_json::Map<String, serde_json::Value> =
serde_json::json!({"type": "object"})
.as_object()
.unwrap()
.clone();
let tool_info = Tool::new("echo", "Echo tool", schema);
let wrapper = McpToolWrapper::new(manager, "ghost".to_string(), tool_info, 0);
let result = wrapper.execute(serde_json::json!({})).await;
assert!(result.is_err());
let msg = result.unwrap_err().to_string();
assert!(
msg.contains("not connected"),
"expected 'not connected' error, got: {msg}"
);
}
} }

View File

@ -0,0 +1,157 @@
//! Metrics 基础设施:基于 `metrics` + `metrics-exporter-prometheus`。
//!
//! 提供 `MetricsObserver`(实现 `Observer` trait桥接 agent_loop 事件到 metrics
//! 以及 Prometheus recorder 初始化。
//!
//! 设计原则:
//! - 业务层agent_loop只认 `dyn Observer` trait不感知 metrics 实现。
//! - 具体指标名和 label 约定集中于此模块。
//! - metrics 后端可替换(换掉 recorder + Observer 实现即可)。
use std::sync::Arc;
use metrics_exporter_prometheus::{PrometheusBuilder, PrometheusHandle};
use super::{Observer, ObserverEvent};
// ============================================================================
// 指标名常量
// ============================================================================
/// LLM 请求耗时直方图。Labels: provider, model
pub const LLM_REQUEST_DURATION: &str = "picobot_llm_request_duration_seconds";
/// LLM token 使用量计数器。Labels: provider, model, type (prompt/completion/total)
pub const LLM_TOKENS_USED: &str = "picobot_llm_tokens_used_total";
/// 工具执行耗时直方图。Labels: tool
pub const TOOL_EXECUTION_DURATION: &str = "picobot_tool_execution_duration_seconds";
/// 工具执行总数计数器。Labels: tool, success (true/false)
pub const TOOL_EXECUTION_TOTAL: &str = "picobot_tool_execution_total";
/// Agent 迭代总数(计数器)
pub const AGENT_ITERATIONS: &str = "picobot_agent_iterations_total";
/// 消息处理错误总数(计数器)
pub const MESSAGE_PROCESSING_ERRORS: &str = "picobot_message_processing_errors_total";
// ============================================================================
// Recorder 初始化
// ============================================================================
/// 安装 Prometheus recorder返回 handle 供 `/metrics` 端点渲染。
///
/// 幂等:首次调用安装 recorder 并缓存 handle后续调用含热重启返回缓存的 handle。
/// 这避免了热重启后 `install_recorder()` 因 recorder 已安装而失败、导致 `/metrics` 返回 503 的问题。
/// 返回 None 表示安装失败非致命metrics 静默降级)。
static PROMETHEUS_HANDLE: std::sync::OnceLock<Option<PrometheusHandle>> = std::sync::OnceLock::new();
pub fn init_recorder() -> Option<PrometheusHandle> {
PROMETHEUS_HANDLE
.get_or_init(|| {
let builder = PrometheusBuilder::new();
match builder.install_recorder() {
Ok(handle) => {
tracing::info!("Prometheus metrics recorder installed");
Some(handle)
}
Err(e) => {
tracing::warn!(error = %e, "Failed to install Prometheus recorder (metrics will be no-op)");
None
}
}
})
.clone()
}
// ============================================================================
// MetricsObserver — 桥接 ObserverEvent 到 metrics 宏
// ============================================================================
/// 将 `ObserverEvent` 转换为 metrics 指标的 Observer 实现。
///
/// 通过 `AgentFactory` 依赖注入到 `AgentLoop`agent_loop 不感知 metrics 实现。
pub struct MetricsObserver;
impl MetricsObserver {
pub fn new() -> Self {
Self
}
}
impl Default for MetricsObserver {
fn default() -> Self {
Self::new()
}
}
impl Observer for MetricsObserver {
fn record_event(&self, event: &ObserverEvent) {
match event {
ObserverEvent::ToolCallStart { tool, .. } => {
// 工具开始:不记录指标,仅 span 日志已覆盖
let _ = tool;
}
ObserverEvent::ToolCall {
tool,
duration,
success,
} => {
let duration_secs = duration.as_secs_f64();
metrics::histogram!(TOOL_EXECUTION_DURATION, "tool" => tool.clone())
.record(duration_secs);
metrics::counter!(
TOOL_EXECUTION_TOTAL,
"tool" => tool.clone(),
"success" => success.to_string()
)
.increment(1);
}
ObserverEvent::AgentStart { provider, model } => {
metrics::counter!(
AGENT_ITERATIONS,
"provider" => provider.clone(),
"model" => model.clone()
)
.increment(1);
}
ObserverEvent::AgentEnd {
provider,
model,
duration,
tokens_used,
} => {
let duration_secs = duration.as_secs_f64();
metrics::histogram!(
LLM_REQUEST_DURATION,
"provider" => provider.clone(),
"model" => model.clone()
)
.record(duration_secs);
if let Some(tokens) = tokens_used {
metrics::counter!(
LLM_TOKENS_USED,
"provider" => provider.clone(),
"model" => model.clone(),
"type" => "total"
)
.increment(*tokens);
}
}
}
}
fn name(&self) -> &str {
"metrics_observer"
}
}
/// 创建默认的 `Arc<dyn Observer>`(供 AgentFactory 注入)。
pub fn default_observer() -> Arc<dyn Observer> {
Arc::new(MetricsObserver::new())
}
// ============================================================================
// 辅助函数:供非 agent_loop 路径直接记录指标
// ============================================================================
/// 记录消息处理错误(供 processor 错误路径调用)。
pub fn record_message_processing_error() {
metrics::counter!(MESSAGE_PROCESSING_ERRORS).increment(1);
}

View File

@ -3,6 +3,9 @@
//! This module provides an Observer pattern for emitting and collecting //! This module provides an Observer pattern for emitting and collecting
//! telemetry events during agent execution. //! telemetry events during agent execution.
pub mod metrics;
pub mod tracing_ctx;
use std::time::Duration; use std::time::Duration;
#[derive(Debug, Clone, Copy, PartialEq, Eq)] #[derive(Debug, Clone, Copy, PartialEq, Eq)]

View File

@ -0,0 +1,43 @@
//! 观测上下文辅助trace_id 生成与 span 创建。
//!
//! 集中管理 trace_id 的生成和 span 字段命名,避免散落在各模块。
//! 业务代码调用此模块的辅助函数,不直接拼 span 字段。
use tracing::Instrument;
use std::future::Future;
/// 生成新的 trace_idUUID v4
pub fn new_trace_id() -> String {
uuid::Uuid::new_v4().to_string()
}
/// 在携带 trace_id/chat_id/session_id 的 span 内执行 future。
///
/// 用于 `tokio::spawn` 边界spawn 不自动传播父 span
/// 调用此函数在 spawn 的 async block 内重建 span 上下文。
///
/// # 示例
/// ```ignore
/// tokio::spawn(
/// traced(&trace_id, &chat_id, &session_id, async move {
/// // 此处所有 tracing 日志自动携带 trace_id/chat_id/session_id
/// process_one(inbound).await
/// })
/// );
/// ```
pub fn traced<F>(trace_id: &str, chat_id: &str, session_id: &str, f: F) -> Instrumented<F>
where
F: Future,
{
let span = tracing::info_span!(
"request",
trace_id = %trace_id,
chat_id = %chat_id,
session_id = %session_id
);
f.instrument(span)
}
/// tracing::Instrumented 的重新导出,便于调用方使用。
pub type Instrumented<F> = tracing::instrument::Instrumented<F>;

View File

@ -330,6 +330,12 @@ pub enum WsOutbound {
timestamp: Option<i64>, timestamp: Option<i64>,
#[serde(default, skip_serializing_if = "Option::is_none")] #[serde(default, skip_serializing_if = "Option::is_none")]
subagent_task_id: Option<String>, subagent_task_id: Option<String>,
/// 子代理最终状态completed/failed/timeout/cancelled/interrupted
/// 供前端更新主视图中 task tool result 占位消息的显示状态。
#[serde(default, skip_serializing_if = "Option::is_none")]
subagent_status: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
subagent_summary: Option<String>,
}, },
#[serde(rename = "todo_list")] #[serde(rename = "todo_list")]
TodoList { TodoList {

View File

@ -221,6 +221,8 @@ pub(crate) fn ws_outbound_from_outbound_message(message: &OutboundMessage) -> Ve
topic_id: message.metadata.get("topic_id").cloned(), topic_id: message.metadata.get("topic_id").cloned(),
timestamp: Some(crate::protocol::now_timestamp()), timestamp: Some(crate::protocol::now_timestamp()),
subagent_task_id: message.metadata.get("subagent_task_id").cloned(), subagent_task_id: message.metadata.get("subagent_task_id").cloned(),
subagent_status: message.metadata.get("subagent_status").cloned(),
subagent_summary: message.metadata.get("subagent_summary").cloned(),
}], }],
} }
} }

View File

@ -252,6 +252,7 @@ struct AnthropicUsage {
#[async_trait] #[async_trait]
impl LLMProvider for AnthropicProvider { impl LLMProvider for AnthropicProvider {
#[tracing::instrument(skip(self, request), fields(provider = %self.name, model = %self.model_id))]
async fn chat( async fn chat(
&self, &self,
request: ChatCompletionRequest, request: ChatCompletionRequest,
@ -259,6 +260,14 @@ impl LLMProvider for AnthropicProvider {
let url = format!("{}/v1/messages", self.base_url); let url = format!("{}/v1/messages", self.base_url);
let max_tokens = request.max_tokens.or(self.max_tokens).unwrap_or(1024); let max_tokens = request.max_tokens.or(self.max_tokens).unwrap_or(1024);
tracing::info!(
provider = %self.name,
model = %self.model_id,
message_count = request.messages.len(),
has_tools = request.tools.is_some(),
"Anthropic: sending chat completion request"
);
let tools = request.tools.map(|tools| { let tools = request.tools.map(|tools| {
tools tools
.iter() .iter()
@ -304,7 +313,16 @@ impl LLMProvider for AnthropicProvider {
req_builder = req_builder.header(key.as_str(), value.as_str()); req_builder = req_builder.header(key.as_str(), value.as_str());
} }
let resp = req_builder.json(&body).send().await?; let resp = req_builder.json(&body).send().await.map_err(|e| {
tracing::error!(
provider = %self.name,
model = %self.model_id,
url = %url,
error = %format_error_chain(&e),
"Anthropic: HTTP request failed"
);
e
})?;
let status = resp.status(); let status = resp.status();
let text = resp.text().await?; let text = resp.text().await?;
@ -321,11 +339,13 @@ impl LLMProvider for AnthropicProvider {
return Err(format!("API error {}: {}", status, text).into()); return Err(format!("API error {}: {}", status, text).into());
} }
#[cfg(debug_assertions)] tracing::debug!(
{ provider = %self.name,
let resp_preview: String = text.chars().take(100).collect(); model = %self.model_id,
tracing::debug!(status = %status, response_preview = %resp_preview, response_len = %text.len(), timeout_secs = self.llm_timeout_secs, "Anthropic response (first 100 chars shown)"); status = %status,
} response_len = text.len(),
"Anthropic response received"
);
let anthropic_resp: AnthropicResponse = serde_json::from_str(&text).map_err(|e| { let anthropic_resp: AnthropicResponse = serde_json::from_str(&text).map_err(|e| {
tracing::error!( tracing::error!(
@ -364,18 +384,29 @@ impl LLMProvider for AnthropicProvider {
} }
} }
let usage = Usage {
prompt_tokens: anthropic_resp.usage.input_tokens,
completion_tokens: anthropic_resp.usage.output_tokens,
total_tokens: anthropic_resp.usage.input_tokens + anthropic_resp.usage.output_tokens,
};
tracing::info!(
provider = %self.name,
model = %self.model_id,
prompt_tokens = usage.prompt_tokens,
completion_tokens = usage.completion_tokens,
total_tokens = usage.total_tokens,
has_tool_calls = !tool_calls.is_empty(),
"Anthropic: chat completion completed"
);
Ok(ChatCompletionResponse { Ok(ChatCompletionResponse {
id: anthropic_resp.id, id: anthropic_resp.id,
model: anthropic_resp.model, model: anthropic_resp.model,
content, content,
reasoning_content: None, reasoning_content: None,
tool_calls, tool_calls,
usage: Usage { usage,
prompt_tokens: anthropic_resp.usage.input_tokens,
completion_tokens: anthropic_resp.usage.output_tokens,
total_tokens: anthropic_resp.usage.input_tokens
+ anthropic_resp.usage.output_tokens,
},
}) })
} }

View File

@ -323,3 +323,28 @@ pub(super) fn add_column_if_missing(conn: &Connection, sql: &str) -> Result<(),
Err(error) => Err(StorageError::Database(error)), Err(error) => Err(StorageError::Database(error)),
} }
} }
/// pending_subagents 表:跟踪异步子代理执行状态,用于崩溃恢复。
pub(super) fn ensure_pending_subagents_schema(conn: &Connection) -> Result<(), StorageError> {
conn.execute_batch(
"
CREATE TABLE IF NOT EXISTS pending_subagents (
task_id TEXT PRIMARY KEY,
parent_session_id TEXT NOT NULL,
parent_topic_id TEXT NOT NULL,
parent_chat_id TEXT NOT NULL,
parent_channel TEXT NOT NULL,
def_name TEXT,
spawned_at INTEGER NOT NULL,
status TEXT NOT NULL DEFAULT 'running'
);
CREATE INDEX IF NOT EXISTS idx_pending_subagents_topic
ON pending_subagents(parent_topic_id, status);
CREATE INDEX IF NOT EXISTS idx_pending_subagents_session
ON pending_subagents(parent_session_id);
",
)?;
Ok(())
}

View File

@ -28,10 +28,10 @@ pub use ports::{
SkillEventRepository, TodoRepository, SkillEventRepository, TodoRepository,
}; };
pub use records::{ pub use records::{
ALLOWED_MEMORY_NAMESPACES, GLOBAL_SCOPE_KEY, MemoryRecord, MemoryUpsert, SchedulerJobRecord, ALLOWED_MEMORY_NAMESPACES, GLOBAL_SCOPE_KEY, MemoryRecord, MemoryUpsert, PendingSubagentRecord,
SchedulerJobState, SchedulerJobStatus, SchedulerJobUpsert, SessionRecord, SessionTokenStats, SchedulerJobRecord, SchedulerJobState, SchedulerJobStatus, SchedulerJobUpsert, SessionRecord,
SkillEventRecord, TodoRecord, TopicRecord, allowed_namespace_names, get_namespace_description, SessionTokenStats, SkillEventRecord, TodoRecord, TopicRecord, allowed_namespace_names,
is_valid_namespace, get_namespace_description, is_valid_namespace,
}; };
#[derive(Clone)] #[derive(Clone)]
@ -234,6 +234,7 @@ impl SessionStore {
ensure_scheduler_schema(&conn)?; ensure_scheduler_schema(&conn)?;
ensure_memory_scope_key_migration(&conn)?; ensure_memory_scope_key_migration(&conn)?;
ensure_todos_schema(&conn)?; ensure_todos_schema(&conn)?;
ensure_pending_subagents_schema(&conn)?;
drop(conn); drop(conn);
@ -2052,6 +2053,140 @@ impl SessionStore {
} }
Ok(todos) Ok(todos)
} }
// ==================== pending_subagents ====================
/// 插入一条 pending_subagent 记录task 工具 spawn 时调用)。
pub fn insert_pending_subagent(
&self,
record: &PendingSubagentRecord,
) -> Result<(), StorageError> {
let conn = self.pool.get()?;
conn.execute(
"INSERT OR REPLACE INTO pending_subagents
(task_id, parent_session_id, parent_topic_id, parent_chat_id, parent_channel, def_name, spawned_at, status)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)",
params![
record.task_id,
record.parent_session_id,
record.parent_topic_id,
record.parent_chat_id,
record.parent_channel,
record.def_name,
record.spawned_at,
record.status,
],
)?;
Ok(())
}
/// 查询指定 topic 下匹配状态的 pending_subagent 记录。
/// `status` 为 None 时查询所有状态。
pub fn list_pending_subagents(
&self,
topic_id: &str,
status: Option<&str>,
) -> Result<Vec<PendingSubagentRecord>, StorageError> {
let conn = self.pool.get()?;
let sql = if status.is_some() {
"SELECT task_id, parent_session_id, parent_topic_id, parent_chat_id, parent_channel, def_name, spawned_at, status
FROM pending_subagents
WHERE parent_topic_id = ?1 AND status = ?2
ORDER BY spawned_at ASC"
} else {
"SELECT task_id, parent_session_id, parent_topic_id, parent_chat_id, parent_channel, def_name, spawned_at, status
FROM pending_subagents
WHERE parent_topic_id = ?1
ORDER BY spawned_at ASC"
};
let mut stmt = conn.prepare(sql)?;
let rows = if let Some(s) = status {
stmt.query_map(params![topic_id, s], map_pending_subagent_record)?
} else {
stmt.query_map(params![topic_id], map_pending_subagent_record)?
};
let mut result = Vec::new();
for row in rows {
result.push(row?);
}
Ok(result)
}
/// 获取指定 task_id 的 pending_subagent 记录。
pub fn get_pending_subagent(
&self,
task_id: &str,
) -> Result<Option<PendingSubagentRecord>, StorageError> {
let conn = self.pool.get()?;
let mut stmt = conn.prepare(
"SELECT task_id, parent_session_id, parent_topic_id, parent_chat_id, parent_channel, def_name, spawned_at, status
FROM pending_subagents
WHERE task_id = ?1",
)?;
let mut rows = stmt.query_map(params![task_id], map_pending_subagent_record)?;
match rows.next() {
Some(row) => Ok(Some(row?)),
None => Ok(None),
}
}
/// 更新指定 task_id 的状态(子代理完成或取消时调用)。
pub fn update_pending_subagent_status(
&self,
task_id: &str,
new_status: &str,
) -> Result<(), StorageError> {
let conn = self.pool.get()?;
conn.execute(
"UPDATE pending_subagents SET status = ?1 WHERE task_id = ?2",
params![new_status, task_id],
)?;
Ok(())
}
/// 条件更新状态:仅在当前状态为 `expected_current` 时才更新为 `new_status`。
///
/// 实现状态机不可逆性不变量:避免 cancel 路径覆盖 spawn 已写入的终态
/// completed → cancelled 是非法转换)。
///
/// 返回是否实际更新affected rows > 0。false 表示状态已被其他路径更新,
/// 调用方应跳过后续基于该假设的操作。
pub fn try_update_pending_subagent_status(
&self,
task_id: &str,
expected_current: &str,
new_status: &str,
) -> Result<bool, StorageError> {
let conn = self.pool.get()?;
let affected = conn.execute(
"UPDATE pending_subagents SET status = ?1 WHERE task_id = ?2 AND status = ?3",
params![new_status, task_id, expected_current],
)?;
Ok(affected > 0)
}
/// 将所有 running 状态的 pending_subagent 标记为 interrupted启动时崩溃恢复调用
pub fn mark_all_running_as_interrupted(&self) -> Result<usize, StorageError> {
let conn = self.pool.get()?;
let affected = conn.execute(
"UPDATE pending_subagents SET status = 'interrupted' WHERE status = 'running'",
[],
)?;
Ok(affected)
}
}
fn map_pending_subagent_record(row: &rusqlite::Row<'_>) -> rusqlite::Result<PendingSubagentRecord> {
Ok(PendingSubagentRecord {
task_id: row.get(0)?,
parent_session_id: row.get(1)?,
parent_topic_id: row.get(2)?,
parent_chat_id: row.get(3)?,
parent_channel: row.get(4)?,
def_name: row.get(5)?,
spawned_at: row.get(6)?,
status: row.get(7)?,
})
} }
pub fn persistent_session_id(channel_name: &str, chat_id: &str) -> String { pub fn persistent_session_id(channel_name: &str, chat_id: &str) -> String {

View File

@ -111,6 +111,31 @@ pub struct TopicRecord {
pub message_count: i64, pub message_count: i64,
} }
/// pending_subagents 表的记录,跟踪异步子代理执行状态。
///
/// 生命周期task 工具 spawn 时插入status=running
/// 子代理完成时更新为 completed/failed/timeout
/// 进程重启时 running 状态被标记为 interrupted。
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PendingSubagentRecord {
/// 子代理 task_id主键与 TaskSession.id 一致)
pub task_id: String,
/// 父会话 session_id
pub parent_session_id: String,
/// 父会话 topic_id用于按 topic 查询未完成子代理)
pub parent_topic_id: String,
/// 父会话 chat_id
pub parent_chat_id: String,
/// 父会话 channel_name
pub parent_channel: String,
/// 子代理定义名称(可选,用于诊断)
pub def_name: Option<String>,
/// 子代理启动时间戳
pub spawned_at: i64,
/// 执行状态running / completed / failed / interrupted / cancelled / timeout
pub status: String,
}
/// 单个 session 的 token 用量统计(聚合结果)。 /// 单个 session 的 token 用量统计(聚合结果)。
#[derive(Debug, Clone, Default, Serialize, Deserialize)] #[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct SessionTokenStats { pub struct SessionTokenStats {

View File

@ -158,14 +158,20 @@ fn extract_f64(args: &serde_json::Value, key: &str, name: &str) -> Result<f64, S
match args.get(key) { match args.get(key) {
None => Err(format!("Missing required parameter: {name}")), None => Err(format!("Missing required parameter: {name}")),
Some(v) => { Some(v) => {
if let Some(n) = v.as_f64() { let n = if let Some(n) = v.as_f64() {
Ok(n) n
} else if let Some(s) = v.as_str() { } else if let Some(s) = v.as_str() {
s.parse::<f64>() s.parse::<f64>()
.map_err(|_| format!("{name} is not a valid number: {s}")) .map_err(|_| format!("{name} is not a valid number: {s}"))?
} else { } else {
Err(format!("{name} must be a number")) return Err(format!("{name} must be a number"));
};
// f64::from_str 接受 "NaN"/"inf";非有限值会使 sort_by 的
// partial_cmp().unwrap() panic且算术结果无意义统一在边界拒绝。
if !n.is_finite() {
return Err(format!("{name} must be a finite number"));
} }
Ok(n)
} }
} }
} }
@ -207,6 +213,11 @@ fn extract_values(args: &serde_json::Value, min_len: usize) -> Result<Vec<f64>,
} else { } else {
return Err(format!("values[{i}] is not a valid number")); return Err(format!("values[{i}] is not a valid number"));
}; };
// f64::from_str 接受 "NaN"/"inf";非有限值会使 sort_by 的
// partial_cmp().unwrap() panic且统计结果无意义统一在边界拒绝。
if !n.is_finite() {
return Err(format!("values[{i}] is not a finite number"));
}
nums.push(n); nums.push(n);
} }
Ok(nums) Ok(nums)
@ -251,12 +262,16 @@ fn calc_factorial(args: &serde_json::Value) -> Result<String, String> {
} }
#[allow(clippy::cast_sign_loss, clippy::cast_possible_truncation)] #[allow(clippy::cast_sign_loss, clippy::cast_possible_truncation)]
let n = x.round() as u128; let n = x.round() as u128;
if n > 170 { // u128::MAX ≈ 3.4e3834! 是最后一个不溢出的阶乘35! ≈ 1.03e40)。
return Err("Factorial result exceeds f64 range (max input: 170)".to_string()); // 修复前按 f64 范围放行到 170实际在 n≥35 时 debug panic / release 静默回绕。
if n > 34 {
return Err("Factorial result exceeds supported integer range (max input: 34)".to_string());
} }
let mut result: u128 = 1; let mut result: u128 = 1;
for i in 2..=n { for i in 2..=n {
result *= i; result = result
.checked_mul(i)
.ok_or_else(|| "Factorial result exceeds supported integer range".to_string())?;
} }
Ok(result.to_string()) Ok(result.to_string())
} }
@ -412,8 +427,15 @@ fn calc_evaluate(args: &serde_json::Value) -> Result<String, String> {
.ok_or_else(|| "Missing required parameter: expression".to_string())?; .ok_or_else(|| "Missing required parameter: expression".to_string())?;
meval::eval_str(expression) meval::eval_str(expression)
.map(format_num)
.map_err(|e| format!("Expression evaluation error: {e}")) .map_err(|e| format!("Expression evaluation error: {e}"))
.and_then(|n| {
// 表达式可产生非有限结果(如 "1/0" → inf、"0/0" → NaN
// 与 extract_values/extract_f64 的边界策略保持一致:拒绝输出。
if !n.is_finite() {
return Err(format!("Expression result is not a finite number: {expression}"));
}
Ok(format_num(n))
})
} }
#[cfg(test)] #[cfg(test)]
@ -779,4 +801,99 @@ mod tests {
.contains("Missing required parameters") .contains("Missing required parameters")
); );
} }
#[tokio::test]
async fn test_median_rejects_nan_string_value() {
// f64::from_str accepts "NaN"; partial_cmp on NaN is None and would
// panic inside sort_by — must surface as a tool error instead.
let tool = CalculatorTool::new();
let result = tool
.execute(json!({"function": "median", "values": ["NaN", 1.0, 2.0]}))
.await
.unwrap();
assert!(!result.success);
assert!(
result.error.as_ref().unwrap().contains("finite"),
"expected finiteness error, got: {:?}",
result.error
);
}
#[tokio::test]
async fn test_percentile_rejects_infinity_string_value() {
let tool = CalculatorTool::new();
let result = tool
.execute(json!({"function": "percentile", "values": ["inf", 1.0], "p": 50}))
.await
.unwrap();
assert!(!result.success);
assert!(result.error.as_ref().unwrap().contains("finite"));
}
#[tokio::test]
async fn test_sum_rejects_nan_instead_of_returning_nan() {
let tool = CalculatorTool::new();
let result = tool
.execute(json!({"function": "sum", "values": ["NaN", 1.0]}))
.await
.unwrap();
assert!(!result.success);
assert!(result.error.as_ref().unwrap().contains("finite"));
}
#[tokio::test]
async fn test_clamp_rejects_non_finite_scalar() {
let tool = CalculatorTool::new();
let result = tool
.execute(json!({"function": "clamp", "x": "NaN", "min_val": 0.0, "max_val": 1.0}))
.await
.unwrap();
assert!(!result.success);
assert!(result.error.as_ref().unwrap().contains("finite"));
}
#[tokio::test]
async fn test_factorial_large_input_returns_error_not_overflow() {
// 35! ≈ 1.03e40 超出 u128::MAX ≈ 3.4e38:修复前 debug 下乘法溢出 panic、
// release 下静默回绕。必须返回工具错误。
let tool = CalculatorTool::new();
let result = tool
.execute(json!({"function": "factorial", "x": 35.0}))
.await
.unwrap();
assert!(!result.success);
assert!(
result.error.as_ref().unwrap().contains("range"),
"expected range error, got: {:?}",
result.error
);
// 34! 仍可精确计算
let ok = tool
.execute(json!({"function": "factorial", "x": 34.0}))
.await
.unwrap();
assert!(ok.success);
assert_eq!(
ok.output,
"295232799039604140847618609643520000000"
);
}
#[tokio::test]
async fn test_evaluate_rejects_non_finite_result() {
let tool = CalculatorTool::new();
let division_by_zero = tool
.execute(json!({"function": "evaluate", "expression": "1/0"}))
.await
.unwrap();
assert!(!division_by_zero.success);
assert!(division_by_zero.error.as_ref().unwrap().contains("finite"));
let nan_result = tool
.execute(json!({"function": "evaluate", "expression": "0/0"}))
.await
.unwrap();
assert!(!nan_result.success);
assert!(nan_result.error.as_ref().unwrap().contains("finite"));
}
} }

View File

@ -18,6 +18,7 @@ pub mod time;
pub mod todo_read; pub mod todo_read;
pub mod todo_write; pub mod todo_write;
pub mod traits; pub mod traits;
pub mod wait_tool;
pub mod web_fetch; pub mod web_fetch;
pub use bash::BashTool; pub use bash::BashTool;
@ -45,7 +46,8 @@ pub use task::{
pub use time::TimeTool; pub use time::TimeTool;
pub use todo_read::TodoReadTool; pub use todo_read::TodoReadTool;
pub use todo_write::TodoWriteTool; pub use todo_write::TodoWriteTool;
pub use traits::{Tool, ToolContext, ToolResult}; pub use traits::{Tool, ToolContext, ToolResult, WaitCoordinator, WaitEvent};
pub use wait_tool::WaitForSubagentsTool;
pub use web_fetch::WebFetchTool; pub use web_fetch::WebFetchTool;
/// Extract a string parameter from JSON args. /// Extract a string parameter from JSON args.

View File

@ -18,6 +18,9 @@ pub enum TaskError {
#[error("Task execution timed out")] #[error("Task execution timed out")]
Timeout, Timeout,
#[error("Task cancelled by user")]
Cancelled,
#[error("Repository error: {0}")] #[error("Repository error: {0}")]
RepositoryError(#[from] StorageError), RepositoryError(#[from] StorageError),
@ -35,6 +38,7 @@ impl TaskError {
pub fn as_status(&self) -> &'static str { pub fn as_status(&self) -> &'static str {
match self { match self {
Self::Timeout => "timeout", Self::Timeout => "timeout",
Self::Cancelled => "cancelled",
Self::SessionNotFound(_) => "failed", Self::SessionNotFound(_) => "failed",
Self::InvalidParentSession => "failed", Self::InvalidParentSession => "failed",
Self::AgentCreationFailed(_) => "failed", Self::AgentCreationFailed(_) => "failed",

View File

@ -14,6 +14,6 @@ pub use runtime::{
}; };
pub use tool::TaskTool; pub use tool::TaskTool;
pub use types::{ pub use types::{
SubagentDef, SubagentSource, SubagentType, TaskDefinition, TaskHandle, TaskSession, SubagentDef, SubagentResult, SubagentSource, SubagentStatus, SubagentType, TaskDefinition,
TaskSessionState, TaskToolArgs, TaskToolResult, TaskHandle, TaskSession, TaskSessionState, TaskToolArgs, TaskToolResult,
}; };

View File

@ -8,6 +8,49 @@ use std::time::Duration;
use async_trait::async_trait; use async_trait::async_trait;
use serde::Deserialize; use serde::Deserialize;
/// RAII guardspawn 任务退出时(正常/early return/panic确定性清理 cancel_registry 条目。
///
/// 实现不变量 3资源生命周期与作用域严格绑定
/// spawn 块末行的手动清理是脆弱的panic/early return 会绕过;
/// 用 Drop impl 把清理封进作用域语义,由编译器保证执行。
struct CancelRegistryGuard {
task_id: String,
registry: Arc<parking_lot::Mutex<HashMap<String, tokio_util::sync::CancellationToken>>>,
/// 标记是否已显式释放(例如 spawn 块成功路径末尾主动 disarm
/// 默认 falsedrop 时执行清理。
disarmed: bool,
}
impl CancelRegistryGuard {
fn new(
task_id: String,
registry: Arc<parking_lot::Mutex<HashMap<String, tokio_util::sync::CancellationToken>>>,
) -> Self {
Self {
task_id,
registry,
disarmed: false,
}
}
/// 显式释放:成功路径末尾调用,避免重复清理。
/// (实际 drop 也会幂等移除,但 disarm 让语义更清晰。)
#[allow(dead_code)]
fn disarm(&mut self) {
self.disarmed = true;
}
}
impl Drop for CancelRegistryGuard {
fn drop(&mut self) {
if self.disarmed {
return;
}
// 幂等:条目可能已被 cancel_pending_for_topic 移除或先前已 drop
self.registry.lock().remove(&self.task_id);
}
}
use crate::agent::{ use crate::agent::{
AgentLoop, AgentRuntimeConfig, EmittedMessageHandler, PersistingEmittedMessageHandler, AgentLoop, AgentRuntimeConfig, EmittedMessageHandler, PersistingEmittedMessageHandler,
SystemPrompt, SystemPromptContext, SystemPromptProvider, SystemPrompt, SystemPromptContext, SystemPromptProvider,
@ -20,14 +63,18 @@ use crate::domain::CapabilityPolicy;
use crate::experts::ExpertRuntime; use crate::experts::ExpertRuntime;
use crate::providers::StreamDelta; use crate::providers::StreamDelta;
use crate::skills::SkillRuntime; use crate::skills::SkillRuntime;
use crate::storage::{ConversationRepository, SessionStore}; use crate::storage::{ConversationRepository, PendingSubagentRecord, SessionStore};
use crate::tools::{ToolContext, ToolRegistry}; use crate::tools::{ToolContext, ToolRegistry};
use crate::utils::current_timestamp;
use super::error::TaskError; use super::error::TaskError;
use super::prompt::{SubagentPromptBuilder, extract_summary}; use super::prompt::{SubagentPromptBuilder, extract_summary};
use super::repository::TaskRepository; use super::repository::TaskRepository;
use super::tool::TaskTool; use super::tool::TaskTool;
use super::types::{SubagentDef, SubagentSource, TaskDefinition, TaskSession, TaskToolResult}; use super::types::{
SubagentDef, SubagentResult, SubagentSource, SubagentStatus, TaskDefinition, TaskSession,
TaskToolResult,
};
/// 子代理运行时配置 /// 子代理运行时配置
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
@ -40,6 +87,8 @@ pub struct SubAgentRuntimeConfig {
pub ttl_hours: u64, pub ttl_hours: u64,
/// 子代理最大嵌套深度0 = 禁止嵌套1 = 允许 1 层孙代理) /// 子代理最大嵌套深度0 = 禁止嵌套1 = 允许 1 层孙代理)
pub max_nesting_depth: u32, pub max_nesting_depth: u32,
/// 异步子代理最大并发数Semaphore 限流)
pub max_concurrent: usize,
} }
impl Default for SubAgentRuntimeConfig { impl Default for SubAgentRuntimeConfig {
@ -62,6 +111,7 @@ impl Default for SubAgentRuntimeConfig {
default_max_execution_secs: 3600, // 60分钟 default_max_execution_secs: 3600, // 60分钟
ttl_hours: 24, ttl_hours: 24,
max_nesting_depth: 1, max_nesting_depth: 1,
max_concurrent: 8,
} }
} }
} }
@ -92,6 +142,12 @@ pub trait SubAgentRuntime: Send + Sync + 'static {
/// 获取可用的子代理类型列表 /// 获取可用的子代理类型列表
fn available_subagent_names(&self) -> Vec<String>; fn available_subagent_names(&self) -> Vec<String>;
/// 取消指定 topic 下所有正在运行的异步子代理。
///
/// 用于 /stop 命令传播:用户取消主 agent 时,同步取消其后台子代理。
/// 返回被触发取消的子代理数量。
async fn cancel_pending_for_topic(&self, topic_id: &str) -> usize;
} }
/// 静态系统提示词提供者(用于子代理) /// 静态系统提示词提供者(用于子代理)
@ -115,12 +171,13 @@ struct SubAgentEmitter {
/// 子/孙智能体自身的 task_id用于持久化时作为 scope_key /// 子/孙智能体自身的 task_id用于持久化时作为 scope_key
task_id: String, task_id: String,
stream_message_id: parking_lot::Mutex<Option<String>>, stream_message_id: parking_lot::Mutex<Option<String>>,
trace_id: Option<String>,
} }
#[async_trait] #[async_trait]
impl EmittedMessageHandler for SubAgentEmitter { impl EmittedMessageHandler for SubAgentEmitter {
async fn handle(&self, message: ChatMessage) { async fn handle(&self, message: ChatMessage) {
for outbound in OutboundMessage::from_chat_message( for mut outbound in OutboundMessage::from_chat_message(
&self.channel_name, &self.channel_name,
&self.chat_id, &self.chat_id,
None, None,
@ -128,6 +185,9 @@ impl EmittedMessageHandler for SubAgentEmitter {
&self.metadata, &self.metadata,
&message, &message,
) { ) {
if let Some(ref tid) = self.trace_id {
outbound.trace_id = tid.clone();
}
if let Err(error) = self.bus.publish_outbound(outbound).await { if let Err(error) = self.bus.publish_outbound(outbound).await {
match error { match error {
crate::bus::BusError::Dropped => { crate::bus::BusError::Dropped => {
@ -146,7 +206,7 @@ impl EmittedMessageHandler for SubAgentEmitter {
if let Some(ms) = duration_ms { if let Some(ms) = duration_ms {
metadata.insert("tool_duration_ms".to_string(), ms.to_string()); metadata.insert("tool_duration_ms".to_string(), ms.to_string());
} }
for outbound in OutboundMessage::from_chat_message( for mut outbound in OutboundMessage::from_chat_message(
&self.channel_name, &self.channel_name,
&self.chat_id, &self.chat_id,
None, None,
@ -154,6 +214,9 @@ impl EmittedMessageHandler for SubAgentEmitter {
&metadata, &metadata,
&message, &message,
) { ) {
if let Some(ref tid) = self.trace_id {
outbound.trace_id = tid.clone();
}
if let Err(error) = self.bus.publish_outbound(outbound).await { if let Err(error) = self.bus.publish_outbound(outbound).await {
match error { match error {
crate::bus::BusError::Dropped => { crate::bus::BusError::Dropped => {
@ -180,7 +243,7 @@ impl EmittedMessageHandler for SubAgentEmitter {
.clone() .clone()
}; };
let outbound = if delta.content.is_empty() && delta.reasoning_content.is_none() { let mut outbound = if delta.content.is_empty() && delta.reasoning_content.is_none() {
OutboundMessage::stream_end( OutboundMessage::stream_end(
&self.channel_name, &self.channel_name,
&self.chat_id, &self.chat_id,
@ -199,6 +262,9 @@ impl EmittedMessageHandler for SubAgentEmitter {
self.metadata.clone(), self.metadata.clone(),
) )
}; };
if let Some(ref tid) = self.trace_id {
outbound.trace_id = tid.clone();
}
if let Err(error) = self.bus.publish_outbound(outbound).await { if let Err(error) = self.bus.publish_outbound(outbound).await {
match error { match error {
@ -302,20 +368,36 @@ fn build_subagent_event_metadata(session: &TaskSession) -> HashMap<String, Strin
"topic_id".to_string(), "topic_id".to_string(),
session.parent_topic_id.clone().unwrap_or_default(), session.parent_topic_id.clone().unwrap_or_default(),
); );
// 子代理最终状态completed/failed/timeout/cancelled/interrupted
// 供前端更新主视图中 task tool result 占位消息的显示状态。
metadata.insert(
"subagent_status".to_string(),
session.state.as_str().to_string(),
);
if let Some(ref summary) = session.summary {
metadata.insert("subagent_summary".to_string(), summary.clone());
}
metadata metadata
} }
/// 发布子智能体执行完成事件ExecutionCompletedmetadata 含 subagent_task_id。 /// 发布子智能体执行完成事件ExecutionCompletedmetadata 含 subagent_task_id。
async fn publish_subagent_completion(bus: &Option<Arc<MessageBus>>, session: &TaskSession) { async fn publish_subagent_completion(
bus: &Option<Arc<MessageBus>>,
session: &TaskSession,
trace_id: &str,
) {
if let Some(bus) = bus { if let Some(bus) = bus {
let metadata = build_subagent_event_metadata(session); let metadata = build_subagent_event_metadata(session);
if let Err(e) = bus if let Err(e) = bus
.publish_outbound(OutboundMessage::execution_completed( .publish_outbound(
session.parent_channel_name.clone(), OutboundMessage::execution_completed(
session.parent_chat_id.clone(), session.parent_channel_name.clone(),
Some(session.parent_session_id.clone()), session.parent_chat_id.clone(),
metadata, Some(session.parent_session_id.clone()),
)) metadata,
)
.with_trace_id(trace_id),
)
.await .await
{ {
tracing::warn!(error = %e, task_id = %session.id, "Failed to publish subagent execution_completed"); tracing::warn!(error = %e, task_id = %session.id, "Failed to publish subagent execution_completed");
@ -328,18 +410,22 @@ async fn publish_subagent_error(
bus: &Option<Arc<MessageBus>>, bus: &Option<Arc<MessageBus>>,
session: &TaskSession, session: &TaskSession,
error_msg: &str, error_msg: &str,
trace_id: &str,
) { ) {
if let Some(bus) = bus { if let Some(bus) = bus {
let metadata = build_subagent_event_metadata(session); let metadata = build_subagent_event_metadata(session);
if let Err(e) = bus if let Err(e) = bus
.publish_outbound(OutboundMessage::error_notification( .publish_outbound(
session.parent_channel_name.clone(), OutboundMessage::error_notification(
session.parent_chat_id.clone(), session.parent_channel_name.clone(),
Some(session.parent_session_id.clone()), session.parent_chat_id.clone(),
error_msg.to_string(), Some(session.parent_session_id.clone()),
None, error_msg.to_string(),
metadata, None,
)) metadata,
)
.with_trace_id(trace_id),
)
.await .await
{ {
tracing::warn!(error = %e, task_id = %session.id, "Failed to publish subagent error notification"); tracing::warn!(error = %e, task_id = %session.id, "Failed to publish subagent error notification");
@ -371,6 +457,11 @@ pub struct DefaultSubAgentRuntime {
store: Arc<SessionStore>, store: Arc<SessionStore>,
/// 技能运行时(实时计算技能索引,替代冻结快照) /// 技能运行时(实时计算技能索引,替代冻结快照)
skills: Arc<SkillRuntime>, skills: Arc<SkillRuntime>,
/// 异步子代理并发限流(按 config.max_concurrent 初始化)
semaphore: Arc<tokio::sync::Semaphore>,
/// task_id → CancellationToken 映射,用于取消传播
/// Arc 包装以便 spawned task 完成后清理自身条目
cancel_registry: Arc<parking_lot::Mutex<HashMap<String, tokio_util::sync::CancellationToken>>>,
} }
impl DefaultSubAgentRuntime { impl DefaultSubAgentRuntime {
@ -386,6 +477,8 @@ impl DefaultSubAgentRuntime {
store: Arc<SessionStore>, store: Arc<SessionStore>,
skills: Arc<SkillRuntime>, skills: Arc<SkillRuntime>,
) -> Self { ) -> Self {
let max_concurrent = config.max_concurrent.max(1);
let semaphore = Arc::new(tokio::sync::Semaphore::new(max_concurrent));
Self { Self {
config, config,
task_repository, task_repository,
@ -397,6 +490,8 @@ impl DefaultSubAgentRuntime {
bus, bus,
store, store,
skills, skills,
semaphore,
cancel_registry: Arc::new(parking_lot::Mutex::new(HashMap::new())),
} }
} }
@ -480,6 +575,7 @@ impl DefaultSubAgentRuntime {
def: Option<&SubagentDef>, def: Option<&SubagentDef>,
parent_nesting_depth: u32, parent_nesting_depth: u32,
parent_task_id: Option<String>, parent_task_id: Option<String>,
trace_id: Option<String>,
) -> Result<AgentLoop, TaskError> { ) -> Result<AgentLoop, TaskError> {
let prompt_provider = Arc::new(StaticSystemPromptProvider::new(system_prompt)); let prompt_provider = Arc::new(StaticSystemPromptProvider::new(system_prompt));
@ -529,6 +625,14 @@ impl DefaultSubAgentRuntime {
// 子代理自身的 capability 作为孙代理的 parent_capability // 子代理自身的 capability 作为孙代理的 parent_capability
// 使孙代理的 TaskTool 能按此策略校验(与主 agent 注入专家 capability 同构) // 使孙代理的 TaskTool 能按此策略校验(与主 agent 注入专家 capability 同构)
parent_capability: def.map(|d| d.capability.clone()), parent_capability: def.map(|d| d.capability.clone()),
// 从父 ToolContext 继承 trace_id保持端到端追踪贯通子代理
trace_id: trace_id.clone(),
// 子代理不注入 sub_done_sender嵌套层不支持异步走同步路径
sub_done_sender: None,
// 子代理不注入 wait_coordinator嵌套层不支持异步 wait
wait_coordinator: None,
// 子代理不注入 cancel_rx嵌套层不支持异步 wait无需 cancel 检查
cancel_rx: None,
}); });
// 如果有 MessageBus附加实时广播 emitter // 如果有 MessageBus附加实时广播 emitter
@ -550,6 +654,7 @@ impl DefaultSubAgentRuntime {
store: self.store.clone(), store: self.store.clone(),
task_id: session.id.clone(), task_id: session.id.clone(),
stream_message_id: parking_lot::Mutex::new(None), stream_message_id: parking_lot::Mutex::new(None),
trace_id: trace_id.clone(),
}, },
self.conversation_repository.clone(), self.conversation_repository.clone(),
session.session_id.clone(), session.session_id.clone(),
@ -571,6 +676,18 @@ impl DefaultSubAgentRuntime {
session: &TaskSession, session: &TaskSession,
def: &SubagentDef, def: &SubagentDef,
prompt: String, prompt: String,
) -> Result<TaskToolResult, TaskError> {
let max_secs = self.effective_max_execution_secs(def);
Self::execute_task_static(agent, session, def, prompt, max_secs).await
}
/// 静态执行任务(供 tokio::spawn 调用,不依赖 &self
async fn execute_task_static(
agent: AgentLoop,
session: &TaskSession,
_def: &SubagentDef,
prompt: String,
max_secs: u64,
) -> Result<TaskToolResult, TaskError> { ) -> Result<TaskToolResult, TaskError> {
// 构建初始消息 // 构建初始消息
let history = vec![ChatMessage::user(prompt)]; let history = vec![ChatMessage::user(prompt)];
@ -581,7 +698,6 @@ impl DefaultSubAgentRuntime {
}; };
// 设置超时 // 设置超时
let max_secs = self.effective_max_execution_secs(def);
let timeout_duration = Duration::from_secs(max_secs); let timeout_duration = Duration::from_secs(max_secs);
let result = tokio::time::timeout( let result = tokio::time::timeout(
@ -658,6 +774,7 @@ impl DefaultSubAgentRuntime {
&self, &self,
session: TaskSession, session: TaskSession,
error: TaskError, error: TaskError,
trace_id: &str,
) -> Result<TaskToolResult, TaskError> { ) -> Result<TaskToolResult, TaskError> {
let status = error.as_status(); let status = error.as_status();
tracing::warn!( tracing::warn!(
@ -674,7 +791,7 @@ impl DefaultSubAgentRuntime {
session.mark_failed(error.to_string()); session.mark_failed(error.to_string());
} }
self.task_repository.save_task_session(&session).await?; self.task_repository.save_task_session(&session).await?;
publish_subagent_error(&self.bus, &session, &error.to_string()).await; publish_subagent_error(&self.bus, &session, &error.to_string(), trace_id).await;
Ok(TaskToolResult { Ok(TaskToolResult {
status: status.to_string(), status: status.to_string(),
summary: error.to_string(), summary: error.to_string(),
@ -791,6 +908,7 @@ impl SubAgentRuntime for DefaultSubAgentRuntime {
tool_arguments: None, tool_arguments: None,
reasoning_content: None, reasoning_content: None,
message_id: None, message_id: None,
trace_id: parent_context.trace_id.clone().unwrap_or_default(),
}; };
if let Err(e) = bus.publish_outbound(event).await { if let Err(e) = bus.publish_outbound(event).await {
@ -798,60 +916,282 @@ impl SubAgentRuntime for DefaultSubAgentRuntime {
} }
} }
// 6-8. 构建提示词、创建子代理、执行任务 // 6. 构建子代理系统提示词
// 统一为单个 Result 表达式model_resolver / create_subagent / execute_task // 实时按 def.capability 过滤技能索引(替代冻结快照,反映运行时技能增删)
// 的任何失败都流入下方 match 的 Err 分支,经 handle_task_failure 返回结构化结果。 let skills_index = if def.capability.has_skill_policy() {
let result: Result<TaskToolResult, TaskError> = { self.skills.system_index_prompt_filtered(
// 6. 构建子代理系统提示词 def.capability.allowed_skills.as_deref(),
// 实时按 def.capability 过滤技能索引(替代冻结快照,反映运行时技能增删) &def.capability.denied_skills,
let skills_index = if def.capability.has_skill_policy() { )
self.skills.system_index_prompt_filtered( } else {
def.capability.allowed_skills.as_deref(), self.skills.system_index_prompt()
&def.capability.denied_skills, };
// 同步解析 def 中的 provider/model 覆盖,保证环境提示中的模型名与实际使用的模型一致
let effective_provider_config = match (def.provider.is_some(), def.model.is_some()) {
(true, _) | (_, true) => self
.model_resolver
.resolve(
def.provider.as_deref(),
def.model.as_deref(),
&self.provider_config,
) )
} else { .map_err(|e| {
self.skills.system_index_prompt() TaskError::AgentCreationFailed(format!(
}; "subagent '{}' model resolution failed: {}",
// 同步解析 def 中的 provider/model 覆盖,保证环境提示中的模型名与实际使用的模型一致 def.name, e
let effective_provider_config = match (def.provider.is_some(), def.model.is_some()) { ))
(true, _) | (_, true) => self })?,
.model_resolver _ => self.provider_config.clone(),
.resolve( };
def.provider.as_deref(), let system_prompt = SubagentPromptBuilder::build(
def.model.as_deref(), &def,
&self.provider_config, &task.description,
) &task.prompt,
.map_err(|e| { &effective_provider_config,
TaskError::AgentCreationFailed(format!( skills_index.as_deref(),
"subagent '{}' model resolution failed: {}", );
def.name, e
))
})?,
_ => self.provider_config.clone(),
};
let system_prompt = SubagentPromptBuilder::build(
&def,
&task.description,
&task.prompt,
&effective_provider_config,
skills_index.as_deref(),
);
// 7. 创建子代理 // 7. 创建子代理
let agent = self.create_subagent( let agent = match self.create_subagent(
&session, &session,
system_prompt, system_prompt,
Some(&def), Some(&def),
parent_context.nesting_depth, parent_context.nesting_depth,
parent_context.task_id.clone(), parent_context.task_id.clone(),
)?; parent_context.trace_id.clone(),
) {
// 8. 执行任务 Ok(agent) => agent,
self.execute_task(agent, &session, &def, task.prompt.clone()) Err(e) => {
.await let trace_id = parent_context.trace_id.as_deref().unwrap_or("");
return self.handle_task_failure(session, e, trace_id).await;
}
}; };
// 9. 更新会话状态并保存 let trace_id = parent_context.trace_id.as_deref().unwrap_or("");
// 8. 判断执行模式:异步(主 agent + 有 sub_done_sender或同步子代理/无 sender
let is_async_mode = parent_context.nesting_depth == 0
&& parent_context.sub_done_sender.is_some()
&& session.parent_topic_id.is_some();
if is_async_mode {
// ===== 异步路径 =====
let topic_id = session.parent_topic_id.clone().unwrap_or_default();
let task_id = session.id.clone();
// 8a. INSERT pending_subagents 记录
let pending_record = PendingSubagentRecord {
task_id: task_id.clone(),
parent_session_id: session.parent_session_id.clone(),
parent_topic_id: topic_id.clone(),
parent_chat_id: session.parent_chat_id.clone(),
parent_channel: session.parent_channel_name.clone(),
def_name: Some(def.name.clone()),
spawned_at: current_timestamp(),
status: "running".to_string(),
};
if let Err(e) = self.store.insert_pending_subagent(&pending_record) {
tracing::warn!(
error = %e,
task_id = %task_id,
"Failed to insert pending_subagent record"
);
}
// 8b. tokio::spawn 后台执行子代理
let store = self.store.clone();
let task_repository = self.task_repository.clone();
let bus = self.bus.clone();
let sub_done_sender = parent_context.sub_done_sender.clone().unwrap();
let session_clone = session.clone();
let def_clone = def.clone();
let prompt = task.prompt.clone();
let trace_id_owned = trace_id.to_string();
let max_secs = self.effective_max_execution_secs(&def);
let task_id_for_spawn = task_id.clone();
let semaphore = self.semaphore.clone();
let cancel_registry = self.cancel_registry.clone();
// 创建 CancellationToken 并注册到 registry供 /stop 取消传播)
let cancel_token = tokio_util::sync::CancellationToken::new();
cancel_registry
.lock()
.insert(task_id_for_spawn.clone(), cancel_token.clone());
// RAII guardspawn 任务退出时(正常/early return/panic确定性清理 registry
// 不变量 3清理与作用域绑定避免末行清理被 panic 绕过
let registry_guard =
CancelRegistryGuard::new(task_id_for_spawn.clone(), cancel_registry.clone());
tokio::spawn(async move {
// guard 在闭包退出时 drop确定性清理 cancel_registry 条目
let _registry_guard = registry_guard;
// 获取并发许可Semaphore 限流)
let _permit = match semaphore.acquire_owned().await {
Ok(p) => p,
Err(e) => {
// 不变量 1状态机收敛性 — early return 也必须收敛终态
// 发送 Failed 结果让 wait 收到,更新 DB 状态,否则系统出现悬空记录
tracing::warn!(
error = %e,
task_id = %task_id_for_spawn,
"Semaphore closed, subagent cannot start; converging state machine to Failed"
);
let result = SubagentResult {
task_id: task_id_for_spawn.clone(),
status: SubagentStatus::Failed,
output: String::new(),
pending_task_ids: store
.list_pending_subagents(&topic_id, Some("running"))
.map(|records| {
records
.into_iter()
.map(|r| r.task_id)
.filter(|id| id != &task_id_for_spawn)
.collect::<Vec<_>>()
})
.unwrap_or_default(),
};
let _ = sub_done_sender.send(result).await;
let _ =
store.update_pending_subagent_status(&task_id_for_spawn, "failed");
// _registry_guard drop 时清理 registry 条目
return;
}
};
// select! 等待执行完成或取消信号
let exec_result = tokio::select! {
biased;
_ = cancel_token.cancelled() => {
tracing::info!(
task_id = %task_id_for_spawn,
"Subagent cancelled by user"
);
Err(TaskError::Cancelled)
}
r = Self::execute_task_static(
agent,
&session_clone,
&def_clone,
prompt,
max_secs,
) => r,
};
// 完成回调:查询未完成 → send SubagentResult → UPDATE status
let (status, output, _summary) = match &exec_result {
Ok(tool_result) => (
SubagentStatus::Completed,
serde_json::to_string(&tool_result).unwrap_or_default(),
tool_result.summary.clone(),
),
Err(TaskError::Timeout) => (
SubagentStatus::Timeout,
String::new(),
"timeout".to_string(),
),
Err(TaskError::Cancelled) => (
SubagentStatus::Cancelled,
String::new(),
"cancelled".to_string(),
),
Err(e) => (
SubagentStatus::Failed,
String::new(),
e.to_string(),
),
};
// 查询同 topic 下仍未完成的子代理列表
let pending_task_ids = store
.list_pending_subagents(&topic_id, Some("running"))
.map(|records| {
records
.into_iter()
.map(|r| r.task_id)
.filter(|id| id != &task_id_for_spawn)
.collect::<Vec<_>>()
})
.unwrap_or_default();
// 发送 SubagentResult 到 sub_done_q
let result = SubagentResult {
task_id: task_id_for_spawn.clone(),
status,
output,
pending_task_ids,
};
if let Err(e) = sub_done_sender.send(result).await {
tracing::warn!(
error = %e,
task_id = %task_id_for_spawn,
"Failed to send SubagentResult to sub_done_q (receiver dropped?)"
);
}
// UPDATE pending_subagents 状态
let status_str = match status {
SubagentStatus::Completed => "completed",
SubagentStatus::Failed => "failed",
SubagentStatus::Timeout => "timeout",
SubagentStatus::Cancelled => "cancelled",
};
if let Err(e) = store.update_pending_subagent_status(&task_id_for_spawn, status_str) {
tracing::warn!(
error = %e,
task_id = %task_id_for_spawn,
"Failed to update pending_subagent status"
);
}
// 更新 TaskSession 状态并发布完成事件
let mut session_done = session_clone;
match exec_result {
Ok(tool_result) => {
session_done.mark_completed(tool_result.summary);
if let Err(e) = task_repository.save_task_session(&session_done).await {
tracing::warn!(error = %e, task_id = %task_id_for_spawn, "Failed to save completed session");
}
publish_subagent_completion(&bus, &session_done, &trace_id_owned).await;
}
Err(e) => {
let err_str = e.to_string();
if matches!(e, TaskError::Timeout) {
session_done.mark_timeout();
} else if matches!(e, TaskError::Cancelled) {
session_done.mark_cancelled();
} else {
session_done.mark_failed(err_str);
}
if let Err(e) = task_repository.save_task_session(&session_done).await {
tracing::warn!(error = %e, task_id = %task_id_for_spawn, "Failed to save failed session");
}
publish_subagent_error(&bus, &session_done, &e.to_string(), &trace_id_owned).await;
}
}
// _registry_guard 在此 drop确定性清理 cancel_registry 条目
// (替代原末行手动 remove覆盖 panic/early return 全路径)
});
// 8c. 立即返回 running 占位结果
// 注意: summary 留空output 只含引导信息。LLM 看到 running 后应调 wait_for_subagents。
return Ok(TaskToolResult {
status: "running".to_string(),
summary: format!("Task {} spawned asynchronously", task_id),
output: format!(
"running, task_id={}. Call wait_for_subagents(timeout_secs) to wait for subagent completion and get results.",
task_id
),
task_id,
});
}
// ===== 同步路径(子代理嵌套或无 sub_done_sender =====
// 9. 执行任务并处理结果
let result = self.execute_task(agent, &session, &def, task.prompt.clone()).await;
match result { match result {
Ok(tool_result) => { Ok(tool_result) => {
let mut session = session; let mut session = session;
@ -863,13 +1203,13 @@ impl SubAgentRuntime for DefaultSubAgentRuntime {
); );
self.task_repository.save_task_session(&session).await?; self.task_repository.save_task_session(&session).await?;
// 发布子智能体 ExecutionCompletedmetadata 注入 subagent_task_id 供前端路由到对应子智能体层 // 发布子智能体 ExecutionCompletedmetadata 注入 subagent_task_id 供前端路由到对应子智能体层
publish_subagent_completion(&self.bus, &session).await; publish_subagent_completion(&self.bus, &session, trace_id).await;
Ok(tool_result) Ok(tool_result)
} }
Err(e) => { Err(e) => {
// 会话创建后的任何失败(含 AgentCreationFailed、Timeout、ExecutionFailed // 会话创建后的任何失败(含 AgentCreationFailed、Timeout、ExecutionFailed
// 统一返回结构化结果,携带 task_id 供前端导航 // 统一返回结构化结果,携带 task_id 供前端导航
self.handle_task_failure(session, e).await self.handle_task_failure(session, e, trace_id).await
} }
} }
} }
@ -939,24 +1279,26 @@ impl SubAgentRuntime for DefaultSubAgentRuntime {
Some(&def), Some(&def),
parent_context.nesting_depth, parent_context.nesting_depth,
parent_context.task_id.clone(), parent_context.task_id.clone(),
parent_context.trace_id.clone(),
)?; )?;
self.execute_task_with_history(agent, &session, additional_prompt) self.execute_task_with_history(agent, &session, additional_prompt)
.await .await
}; };
// 7. 更新会话状态 // 7. 更新会话状态
let trace_id = parent_context.trace_id.as_deref().unwrap_or("");
match result { match result {
Ok(tool_result) => { Ok(tool_result) => {
let mut session = session; let mut session = session;
session.mark_completed(tool_result.summary.clone()); session.mark_completed(tool_result.summary.clone());
self.task_repository.save_task_session(&session).await?; self.task_repository.save_task_session(&session).await?;
// 发布子智能体 ExecutionCompletedmetadata 注入 subagent_task_id 供前端路由到对应子智能体层 // 发布子智能体 ExecutionCompletedmetadata 注入 subagent_task_id 供前端路由到对应子智能体层
publish_subagent_completion(&self.bus, &session).await; publish_subagent_completion(&self.bus, &session, trace_id).await;
Ok(tool_result) Ok(tool_result)
} }
Err(e) => { Err(e) => {
// 修复:原代码一律 mark_failed未处理 timeout现统一走 handle_task_failure // 修复:原代码一律 mark_failed未处理 timeout现统一走 handle_task_failure
self.handle_task_failure(session, e).await self.handle_task_failure(session, e, trace_id).await
} }
} }
} }
@ -979,6 +1321,75 @@ impl SubAgentRuntime for DefaultSubAgentRuntime {
fn available_subagent_names(&self) -> Vec<String> { fn available_subagent_names(&self) -> Vec<String> {
self.subagent_runtime.available_names() self.subagent_runtime.available_names()
} }
async fn cancel_pending_for_topic(&self, topic_id: &str) -> usize {
// 查询该 topic 下所有 running 的子代理
let running = match self.store.list_pending_subagents(topic_id, Some("running")) {
Ok(records) => records,
Err(e) => {
tracing::warn!(
error = %e,
topic_id = %topic_id,
"Failed to list pending subagents for cancellation"
);
return 0;
}
};
let count = running.len();
if count == 0 {
return 0;
}
tracing::info!(
topic_id = %topic_id,
count,
"Cancelling pending subagents for topic"
);
// 触发每个子代理的 CancellationToken
let registry = self.cancel_registry.lock();
for record in &running {
if let Some(token) = registry.get(&record.task_id) {
token.cancel();
tracing::info!(
task_id = %record.task_id,
"Cancelled subagent token"
);
} else {
// token 不在 registry 中(可能已完成但 DB 状态未更新,或进程重启后丢失)
// 不变量 1条件 UPDATE仅在 status='running' 时转为 cancelled
// 避免 spawn 已完成的终态被覆盖completed → cancelled 是非法转换)
match self
.store
.try_update_pending_subagent_status(&record.task_id, "running", "cancelled")
{
Ok(true) => {
tracing::info!(
task_id = %record.task_id,
"Marked subagent as cancelled in DB (token not in registry)"
);
}
Ok(false) => {
tracing::info!(
task_id = %record.task_id,
"Subagent status already updated by another path, skip cancel"
);
}
Err(e) => {
tracing::warn!(
error = %e,
task_id = %record.task_id,
"Failed to mark subagent as cancelled in DB"
);
}
}
}
}
drop(registry);
count
}
} }
/// 子代理定义目录 /// 子代理定义目录

View File

@ -17,6 +17,8 @@ pub enum TaskSessionState {
Failed, Failed,
/// 已超时 /// 已超时
Timeout, Timeout,
/// 已取消(用户 /stop 传播)
Cancelled,
/// 状态未知(如重启后从 DB 重建时无法可靠推断原状态) /// 状态未知(如重启后从 DB 重建时无法可靠推断原状态)
Unknown, Unknown,
} }
@ -27,6 +29,19 @@ impl Default for TaskSessionState {
} }
} }
impl TaskSessionState {
pub fn as_str(&self) -> &'static str {
match self {
Self::Running => "running",
Self::Completed => "completed",
Self::Failed => "failed",
Self::Timeout => "timeout",
Self::Cancelled => "cancelled",
Self::Unknown => "unknown",
}
}
}
/// 子代理来源 /// 子代理来源
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")] #[serde(rename_all = "snake_case")]
@ -223,6 +238,13 @@ impl TaskSession {
self.error = Some("Task execution timed out".to_string()); self.error = Some("Task execution timed out".to_string());
self.updated_at = current_timestamp(); self.updated_at = current_timestamp();
} }
/// 标记取消
pub fn mark_cancelled(&mut self) {
self.state = TaskSessionState::Cancelled;
self.error = Some("Task cancelled by user".to_string());
self.updated_at = current_timestamp();
}
} }
/// 任务工具参数 /// 任务工具参数
@ -271,7 +293,7 @@ pub struct TaskHandle {
/// 任务执行结果 /// 任务执行结果
#[derive(Debug, Clone, Serialize)] #[derive(Debug, Clone, Serialize)]
pub struct TaskToolResult { pub struct TaskToolResult {
/// 状态: success/failed/timeout /// 状态: success/failed/timeout/running
pub status: String, pub status: String,
/// 任务完成总结 /// 任务完成总结
pub summary: String, pub summary: String,
@ -280,3 +302,26 @@ pub struct TaskToolResult {
/// 会话 ID用于恢复 /// 会话 ID用于恢复
pub task_id: String, pub task_id: String,
} }
/// 异步子代理完成状态
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum SubagentStatus {
Completed,
Failed,
Timeout,
Cancelled,
}
/// 异步子代理完成回调内容(通过 sub_done_q 传递给 wait 工具)
#[derive(Debug, Clone)]
pub struct SubagentResult {
/// 完成的子代理 task_id
pub task_id: String,
/// 完成状态
pub status: SubagentStatus,
/// 子代理输出(与 TaskToolResult.output 格式一致)
pub output: String,
/// 仍未完成的子代理 task_id 列表(供 LLM 判断全局进度)
pub pending_task_ids: Vec<String>,
}

View File

@ -184,6 +184,8 @@ mod tests {
parent_task_id: None, parent_task_id: None,
tool_call_id: None, tool_call_id: None,
parent_capability: None, parent_capability: None,
trace_id: None,
..Default::default()
} }
} }

View File

@ -486,6 +486,8 @@ mod tests {
parent_task_id: None, parent_task_id: None,
tool_call_id: None, tool_call_id: None,
parent_capability: None, parent_capability: None,
trace_id: None,
..Default::default()
} }
} }

View File

@ -1,6 +1,11 @@
use std::time::Duration;
use std::sync::Arc;
use async_trait::async_trait; use async_trait::async_trait;
use tokio::sync::{mpsc, watch};
use crate::domain::CapabilityPolicy; use crate::domain::CapabilityPolicy;
use crate::tools::task::SubagentResult;
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct ToolResult { pub struct ToolResult {
@ -9,7 +14,58 @@ pub struct ToolResult {
pub error: Option<String>, pub error: Option<String>,
} }
#[derive(Debug, Clone, Default)] /// wait_for_subagents 工具等待期间的事件。
#[derive(Debug, Clone)]
pub enum WaitEvent {
/// 一个子代理完成,携带其结果
SubagentResult(SubagentResult),
/// wait 期间有新用户消息到达(已注入 history携带新用户消息的内容列表
UserMessage(Vec<String>),
/// 等待超时
Timeout,
/// 收到取消信号(/stop等待已优雅终止状态已清理完毕
Cancelled,
}
/// wait_for_subagents 工具的协调器接口。
///
/// 封装了释放/重获取 serial_lock + select! 等待 + 管理等待状态的逻辑,
/// 使 wait 工具不直接依赖 gateway 模块(避免循环依赖)。
///
/// 具体实现 `SessionWaitCoordinator` 在 gateway 模块中,持有 Session 引用。
#[async_trait]
pub trait WaitCoordinator: Send + Sync + 'static {
/// 查询当前 topic 下仍处于 running 状态的子代理 task_id 列表。
fn query_pending_task_ids(&self) -> Vec<String>;
/// 尝试排空 sub_done_q 中已缓冲的子代理结果(非阻塞 drain但方法本身是 async
/// 因为需要获取 Session 锁)。
///
/// 用于两种场景:
/// 1. wait_for_subagents 入口处:即使 DB 中无 running 子代理,
/// 队列可能仍缓冲了已完成子代理的结果(子代理完成 → send 到队列 →
/// DB 更新为 completed但 LLM 上轮未消费队列)。
/// 2. wait() 进入 select! 前:多个子代理同时完成时,批量消费避免
/// 每个结果各触发一次 LLM 调用。
async fn try_drain_queued_results(&self) -> Vec<SubagentResult>;
/// 进入等待状态:释放 serial_lock → select! → 重获取 serial_lock。
///
/// 调用前提serial_lock 已被执行路径获取guard 存于 coordinator 内部。
/// 返回后serial_lock 已被重新获取waiting 标志已清除。
///
/// `cancel_rx`:可选的取消信号接收端。当收到信号时(/stop 命令),
/// select! 立即返回 `WaitEvent::Cancelled`,并完成完整的状态清理
///(重获取锁、回填 guard、清除 is_waiting、归还 receiver
/// 为 None 时退化为不检查取消(向后兼容,子代理场景)。
async fn wait(
&self,
timeout: Duration,
cancel_rx: Option<watch::Receiver<()>>,
) -> WaitEvent;
}
#[derive(Clone, Default)]
pub struct ToolContext { pub struct ToolContext {
pub channel_name: Option<String>, pub channel_name: Option<String>,
pub sender_id: Option<String>, pub sender_id: Option<String>,
@ -32,6 +88,49 @@ pub struct ToolContext {
/// TaskTool 据此强制校验子代理加载(白/黑名单),与 spawn/resume 安全范式一致。 /// TaskTool 据此强制校验子代理加载(白/黑名单),与 spawn/resume 安全范式一致。
/// 以数据形式传递,避免 task 模块反向依赖 experts 模块。 /// 以数据形式传递,避免 task 模块反向依赖 experts 模块。
pub parent_capability: Option<CapabilityPolicy>, pub parent_capability: Option<CapabilityPolicy>,
/// 端到端追踪 ID从 InboundMessage 继承,用于 tool 执行路径的日志关联)。
/// None 表示无追踪上下文(如子代理独立执行或测试环境)。
pub trace_id: Option<String>,
/// 异步子代理完成队列的 sender按 topic 隔离)。
/// 仅主 agentnesting_depth=0有值agent_factory 构建时从 SessionHistory 注入。
/// TaskTool spawn 异步子代理后,子代理完成时通过此 sender 发送 SubagentResult
/// 由 wait_for_subagents 工具的 receiver 端消费。
/// 子代理自身nesting_depth>0为 None嵌套层不支持异步走同步路径。
pub sub_done_sender: Option<mpsc::Sender<SubagentResult>>,
/// wait_for_subagents 工具的协调器(仅主 agent 有值)。
/// 封装了释放/重获取 serial_lock + select! 等待逻辑。
/// wait 工具通过此接口实现真等待(释放锁让 process_one 注入用户消息)。
pub wait_coordinator: Option<Arc<dyn WaitCoordinator>>,
/// 取消信号接收端(仅主 agent 有值,由 agent_factory 从 cancel_token clone 注入)。
/// wait_for_subagents 工具将其传给 coordinator.wait() 的 select!
/// 使 /stop 命令能立即中断等待并完成状态清理。
/// watch::Receiver 可安全 clone多个 receiver 共享同一 sender
/// 互不影响各自的 has_changed() / changed() 状态。
pub cancel_rx: Option<watch::Receiver<()>>,
}
impl std::fmt::Debug for ToolContext {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("ToolContext")
.field("channel_name", &self.channel_name)
.field("sender_id", &self.sender_id)
.field("chat_id", &self.chat_id)
.field("session_id", &self.session_id)
.field("topic_id", &self.topic_id)
.field("message_id", &self.message_id)
.field("message_seq", &self.message_seq)
.field("subagent_description", &self.subagent_description)
.field("nesting_depth", &self.nesting_depth)
.field("task_id", &self.task_id)
.field("parent_task_id", &self.parent_task_id)
.field("tool_call_id", &self.tool_call_id)
.field("parent_capability", &self.parent_capability)
.field("trace_id", &self.trace_id)
.field("sub_done_sender", &self.sub_done_sender)
.field("wait_coordinator", &self.wait_coordinator.is_some())
.field("cancel_rx", &self.cancel_rx.is_some())
.finish()
}
} }
#[async_trait] #[async_trait]

250
src/tools/wait_tool.rs Normal file
View File

@ -0,0 +1,250 @@
use std::time::Duration;
use async_trait::async_trait;
use serde_json::json;
use crate::tools::{Tool, ToolContext, ToolResult, WaitEvent};
/// wait_for_subagents 工具 — 等待异步子代理完成或用户消息到达。
///
/// 调用后释放 serial_lock进入 select! 等待:
/// - 子代理完成 → 返回结果 + 未完成列表
/// - 用户消息到达 → 返回 "有新用户消息"(消息已注入 history
/// - 超时 → 返回超时 + 未完成列表
///
/// 等待结束后重新获取 serial_lock保证后续工具调用串行。
pub struct WaitForSubagentsTool {
/// 默认超时LLM 未指定时使用
default_timeout_secs: u64,
}
impl WaitForSubagentsTool {
pub const TOOL_NAME: &'static str = "wait_for_subagents";
pub fn new(default_timeout_secs: u64) -> Self {
Self {
default_timeout_secs,
}
}
}
#[async_trait]
impl Tool for WaitForSubagentsTool {
fn name(&self) -> &str {
Self::TOOL_NAME
}
fn description(&self) -> &str {
"Wait for asynchronous subagents to complete, or for new user messages to arrive. \
Use this after launching subagents via the task tool to receive their results. \
Returns the first completed subagent's result and a list of still-pending task IDs. \
If pending_task_ids is non-empty, call this tool again to wait for the next one."
}
fn parameters_schema(&self) -> serde_json::Value {
json!({
"type": "object",
"properties": {
"timeout_secs": {
"type": "integer",
"description": "Maximum seconds to wait. Default 60. The tool returns early if a subagent completes or a user message arrives.",
"default": 60
}
},
"required": []
})
}
fn read_only(&self) -> bool {
false
}
fn exclusive(&self) -> bool {
// wait 工具释放/重获取 serial_lock不应与其他工具并发
true
}
async fn execute(&self, _args: serde_json::Value) -> anyhow::Result<ToolResult> {
Ok(ToolResult {
success: false,
output: String::new(),
error: Some(
"wait_for_subagents requires tool context with wait_coordinator".to_string(),
),
})
}
async fn execute_with_context(
&self,
context: &ToolContext,
args: serde_json::Value,
) -> anyhow::Result<ToolResult> {
// 1. 获取 wait_coordinator仅主 agent 有值)
let coordinator = match &context.wait_coordinator {
Some(c) => c.clone(),
None => {
return Ok(ToolResult {
success: false,
output: String::new(),
error: Some(
"wait_for_subagents is not available in this context (no wait_coordinator)"
.to_string(),
),
});
}
};
// 2. 解析超时参数
let timeout_secs = args
.get("timeout_secs")
.and_then(|v| v.as_u64())
.unwrap_or(self.default_timeout_secs);
let timeout = Duration::from_secs(timeout_secs);
// 3. 先尝试排空队列中已缓冲的结果
// 场景:子代理已完成 → 结果 send 到队列 + DB 更新为 completed
// 但 LLM 上轮未调用 wait或 wait 超时未消费)→ 结果缓冲在队列中。
// 若不排空query_pending_task_ids 返回空DB 已非 running
// 返回 "No pending" → 缓冲结果永远丢失。
let drained = coordinator.try_drain_queued_results().await;
if !drained.is_empty() {
let pending_after = coordinator.query_pending_task_ids();
let pending_str = if pending_after.is_empty() {
"none".to_string()
} else {
pending_after.join(", ")
};
let formatted: Vec<String> = drained
.iter()
.map(|r| {
format!(
"Subagent {} completed (status: {:?}). Output: {}",
r.task_id, r.status, r.output
)
})
.collect();
return Ok(ToolResult {
success: true,
output: format!(
"Retrieved {} buffered subagent result(s):\n{}\nStill pending: [{}]",
drained.len(),
formatted.join("\n"),
pending_str
),
error: None,
});
}
// 4. 查询 pending 子代理
let pending = coordinator.query_pending_task_ids();
if pending.is_empty() {
return Ok(ToolResult {
success: true,
output: "No pending subagents to wait for.".to_string(),
error: None,
});
}
tracing::info!(
topic_id = ?context.topic_id,
pending_count = pending.len(),
timeout_secs,
"wait_for_subagents: entering wait"
);
// 4. 进入等待coordinator 内部:释放锁 → select! → 重获取锁)
// 传入 cancel_rx 使 /stop 命令能立即中断等待。
// coordinator 在 select! 中以 biased 优先级处理:
// 子代理结果 > 用户消息 > 取消信号 > 超时
let event = coordinator
.wait(timeout, context.cancel_rx.clone())
.await;
// 5. 格式化返回结果
let output = match event {
WaitEvent::SubagentResult(result) => {
let pending_str = if result.pending_task_ids.is_empty() {
"none".to_string()
} else {
result.pending_task_ids.join(", ")
};
format!(
"Subagent {} completed (status: {:?}). Output: {}\nStill pending: [{}]",
result.task_id, result.status, result.output, pending_str
)
}
WaitEvent::UserMessage(messages) => {
let pending = coordinator.query_pending_task_ids();
let pending_str = if pending.is_empty() {
"none".to_string()
} else {
pending.join(", ")
};
if messages.is_empty() {
format!(
"A new user message arrived while waiting (content could not be retrieved). \
Still pending subagents: [{}]",
pending_str
)
} else {
let formatted_msgs: Vec<String> = messages
.iter()
.enumerate()
.map(|(i, msg)| format!(" [{}] {}", i + 1, msg))
.collect();
format!(
"New user message(s) arrived while waiting:\n{}\n\
These messages have been added to the conversation history. \
Still pending subagents: [{}]",
formatted_msgs.join("\n"),
pending_str
)
}
}
WaitEvent::Timeout => {
let pending = coordinator.query_pending_task_ids();
format!(
"Wait timed out after {}s. Still pending subagents: [{}]",
timeout_secs,
if pending.is_empty() {
"none".to_string()
} else {
pending.join(", ")
}
)
}
WaitEvent::Cancelled => {
// /stop 命令中断了等待。coordinator 已完成全部状态清理
//(重获取 serial_lock、回填 guard、清除 is_waiting、归还 receiver
// 返回提示性输出Agent 下一轮迭代会检测到 cancel 并退出。
let pending = coordinator.query_pending_task_ids();
tracing::info!(
topic_id = ?context.topic_id,
pending_count = pending.len(),
"wait_for_subagents: cancelled by /stop"
);
format!(
"Wait was cancelled by /stop command. \
Pending subagents (if any) have been cancelled separately. \
Still pending in DB: [{}]",
if pending.is_empty() {
"none".to_string()
} else {
pending.join(", ")
}
)
}
};
tracing::info!(
topic_id = ?context.topic_id,
"wait_for_subagents: returning result"
);
Ok(ToolResult {
success: true,
output,
error: None,
})
}
}

View File

@ -11,6 +11,20 @@ pub fn current_timestamp() -> i64 {
.as_millis() as i64 .as_millis() as i64
} }
/// 从 `catch_unwind` 的 panic payload 中提取可读消息。
///
/// `panic!` 的 payload 通常是 `&str` 或 `String`;其他类型(如直接
/// `panic!(42)`)无法还原原文,返回占位描述。
pub fn panic_payload_message(payload: &(dyn std::any::Any + Send)) -> String {
if let Some(s) = payload.downcast_ref::<&str>() {
return (*s).to_string();
}
if let Some(s) = payload.downcast_ref::<String>() {
return s.clone();
}
"<non-string panic payload>".to_string()
}
/// 递归展开 `error.source()` 链,生成 `"顶层错误\ncaused by: 原因\ncaused by: ..."` 格式的字符串。 /// 递归展开 `error.source()` 链,生成 `"顶层错误\ncaused by: 原因\ncaused by: ..."` 格式的字符串。
pub fn format_error_chain(error: &(dyn std::error::Error + 'static)) -> String { pub fn format_error_chain(error: &(dyn std::error::Error + 'static)) -> String {
let mut details = vec![error.to_string()]; let mut details = vec![error.to_string()];
@ -23,3 +37,27 @@ pub fn format_error_chain(error: &(dyn std::error::Error + 'static)) -> String {
details.join("\ncaused by: ") details.join("\ncaused by: ")
} }
#[cfg(test)]
mod tests {
use super::*;
use std::any::Any;
#[test]
fn test_panic_payload_message_str() {
let payload: Box<dyn Any + Send> = Box::new("boom");
assert_eq!(panic_payload_message(&*payload), "boom");
}
#[test]
fn test_panic_payload_message_string() {
let payload: Box<dyn Any + Send> = Box::new(String::from("boom"));
assert_eq!(panic_payload_message(&*payload), "boom");
}
#[test]
fn test_panic_payload_message_other_type() {
let payload: Box<dyn Any + Send> = Box::new(42u32);
assert!(panic_payload_message(&*payload).contains("non-string"));
}
}

View File

@ -611,8 +611,8 @@ function App() {
const viewKey = useMemo(() => { const viewKey = useMemo(() => {
if (schedulerView) return `scheduler:${schedulerView.jobId}`; if (schedulerView) return `scheduler:${schedulerView.jobId}`;
if (subAgentView) return `subagent:${subAgentView.taskId}`; if (subAgentView) return `subagent:${subAgentView.taskId}`;
return 'main'; return `main:${selectedTopic ?? ''}`;
}, [schedulerView, subAgentView]); }, [schedulerView, subAgentView, selectedTopic]);
return ( return (
<div className="flex h-screen flex-col bg-[var(--bg-primary)] text-[var(--text-primary)] overflow-hidden"> <div className="flex h-screen flex-col bg-[var(--bg-primary)] text-[var(--text-primary)] overflow-hidden">
@ -906,9 +906,9 @@ function App() {
)} )}
<div className="flex-1 min-h-0"> <div className="flex-1 min-h-0">
<ChatContainer <ChatContainer
key={selectedTopic ?? 'no-topic'}
messages={chatMessages} messages={chatMessages}
isLoading={isLoading} isLoading={isLoading}
topicId={selectedTopic}
isReadOnly={subAgentView || schedulerView ? true : isReadOnly} isReadOnly={subAgentView || schedulerView ? true : isReadOnly}
channelName={ channelName={
schedulerView schedulerView

View File

@ -24,6 +24,8 @@ interface ChatContainerProps {
onOpenSettings?: () => void; onOpenSettings?: () => void;
/** 设置弹窗关闭信号(每次关闭递增,用于触发 ExpertSelector 刷新) */ /** 设置弹窗关闭信号(每次关闭递增,用于触发 ExpertSelector 刷新) */
settingsClosedTick?: number; settingsClosedTick?: number;
/** 当前话题 ID用于切换话题时清空输入框草稿 */
topicId?: string | null;
} }
export function ChatContainer({ export function ChatContainer({
@ -40,6 +42,7 @@ export function ChatContainer({
sessionId, sessionId,
onOpenSettings, onOpenSettings,
settingsClosedTick, settingsClosedTick,
topicId,
}: ChatContainerProps) { }: ChatContainerProps) {
const [selectedExpert, setSelectedExpert] = useState<{ const [selectedExpert, setSelectedExpert] = useState<{
name: string; name: string;
@ -74,6 +77,7 @@ export function ChatContainer({
isReadOnly={isReadOnly} isReadOnly={isReadOnly}
channelName={channelName} channelName={channelName}
selectedExpert={selectedExpert} selectedExpert={selectedExpert}
topicId={topicId}
/> />
</div> </div>
); );

View File

@ -34,13 +34,14 @@ function StatusIcon({
status, status,
size = 14, size = 14,
}: { }: {
status: 'calling' | 'result' | 'pending' | 'success' | 'failed' | 'timeout'; status: string;
size?: number; size?: number;
}) { }) {
const iconClass = `transition-all duration-300`; const iconClass = `transition-all duration-300`;
switch (status) { switch (status) {
case 'calling': case 'calling':
case 'running':
return ( return (
<Loader2 <Loader2
className={`${iconClass} animate-spin`} className={`${iconClass} animate-spin`}
@ -50,6 +51,7 @@ function StatusIcon({
); );
case 'result': case 'result':
case 'success': case 'success':
case 'completed':
return ( return (
<CheckCircle <CheckCircle
className={`${iconClass} animate-scale-in`} className={`${iconClass} animate-scale-in`}
@ -74,6 +76,8 @@ function StatusIcon({
/> />
); );
case 'pending': case 'pending':
case 'interrupted':
case 'cancelled':
return ( return (
<Loader <Loader
className={`${iconClass} animate-spin`} className={`${iconClass} animate-spin`}
@ -437,19 +441,49 @@ export const MessageBubble = memo(function MessageBubble({
((message.arguments as Record<string, unknown> | null)?.prompt as string) || ''; ((message.arguments as Record<string, unknown> | null)?.prompt as string) || '';
// task tool 专用的状态配色 // task tool 专用的状态配色
const taskStatusConfig = { // 支持的状态:
// - running: 异步子代理刚 spawn占位结果
// - success/completed: 子代理执行成功
// - failed: 子代理执行失败
// - timeout: 子代理执行超时
// - cancelled: 子代理被用户取消
// - interrupted: 子代理因服务器重启被中断
const taskStatusConfig: Record<string, { dot: string; borderColor: string; iconColor: string }> = {
running: {
dot: 'bg-amber-400 animate-pulse',
borderColor: 'border-amber-500/40',
iconColor: 'text-amber-400',
},
success: { success: {
dot: 'bg-emerald-400', dot: 'bg-emerald-400',
borderColor: 'border-emerald-500/40', borderColor: 'border-emerald-500/40',
iconColor: 'text-emerald-400', iconColor: 'text-emerald-400',
}, },
completed: {
dot: 'bg-emerald-400',
borderColor: 'border-emerald-500/40',
iconColor: 'text-emerald-400',
},
failed: { dot: 'bg-red-400', borderColor: 'border-red-500/40', iconColor: 'text-red-400' }, failed: { dot: 'bg-red-400', borderColor: 'border-red-500/40', iconColor: 'text-red-400' },
timeout: { timeout: {
dot: 'bg-amber-400', dot: 'bg-amber-400',
borderColor: 'border-amber-500/40', borderColor: 'border-amber-500/40',
iconColor: 'text-amber-400', iconColor: 'text-amber-400',
}, },
} as const; cancelled: {
dot: 'bg-zinc-400',
borderColor: 'border-zinc-500/40',
iconColor: 'text-zinc-400',
},
interrupted: {
dot: 'bg-orange-400',
borderColor: 'border-orange-500/40',
iconColor: 'text-orange-400',
},
};
// 安全获取 task 状态配色,未知状态回退到默认(避免 undefined.borderColor 崩溃)
const taskStyle = taskResult ? (taskStatusConfig[taskResult.status] ?? taskStatusConfig.failed) : null;
return ( return (
<div data-message-id={message.id} className="flex gap-3 animate-slide-in"> <div data-message-id={message.id} className="flex gap-3 animate-slide-in">
@ -481,14 +515,14 @@ export const MessageBubble = memo(function MessageBubble({
<div <div
onClick={() => setToolExpanded(!toolExpanded)} onClick={() => setToolExpanded(!toolExpanded)}
className={`cursor-pointer rounded-xl border bg-[var(--bg-tertiary)]/60 w-full transition-all duration-500 hover:bg-[var(--bg-tertiary)]/80 group ${ className={`cursor-pointer rounded-xl border bg-[var(--bg-tertiary)]/60 w-full transition-all duration-500 hover:bg-[var(--bg-tertiary)]/80 group ${
taskResult ? taskStatusConfig[taskResult.status].borderColor : statusConfig.fullBorder taskStyle ? taskStyle.borderColor : statusConfig.fullBorder
}`} }`}
> >
{/* Header row */} {/* Header row */}
<div className="flex items-center gap-2 px-3 py-2"> <div className="flex items-center gap-2 px-3 py-2">
<span <span
className={`inline-block h-2 w-2 rounded-full flex-shrink-0 transition-colors duration-500 ${ className={`inline-block h-2 w-2 rounded-full flex-shrink-0 transition-colors duration-500 ${
taskResult ? taskStatusConfig[taskResult.status].dot : statusConfig.dot taskStyle ? taskStyle.dot : statusConfig.dot
}`} }`}
/> />
<span className="text-sm font-medium text-[var(--text-secondary)] truncate"> <span className="text-sm font-medium text-[var(--text-secondary)] truncate">
@ -496,9 +530,7 @@ export const MessageBubble = memo(function MessageBubble({
</span> </span>
<span <span
className={`flex-shrink-0 transition-all duration-300 ${ className={`flex-shrink-0 transition-all duration-300 ${
taskResult taskStyle ? taskStyle.iconColor : statusConfig.iconColor
? taskStatusConfig[taskResult.status].iconColor
: statusConfig.iconColor
}`} }`}
> >
{taskResult ? ( {taskResult ? (

View File

@ -25,6 +25,8 @@ interface MessageInputProps {
isReadOnly?: boolean; isReadOnly?: boolean;
channelName?: string; channelName?: string;
selectedExpert?: { name: string; description: string } | null; selectedExpert?: { name: string; description: string } | null;
/** 当前话题 ID切换话题时自动清空草稿 */
topicId?: string | null;
} }
interface FileAttachment { interface FileAttachment {
@ -51,6 +53,7 @@ export function MessageInput({
isReadOnly = false, isReadOnly = false,
channelName, channelName,
selectedExpert, selectedExpert,
topicId,
}: MessageInputProps) { }: MessageInputProps) {
const effectivePlaceholder = const effectivePlaceholder =
placeholder ?? placeholder ??
@ -62,6 +65,16 @@ export function MessageInput({
const textareaRef = useRef<HTMLTextAreaElement>(null); const textareaRef = useRef<HTMLTextAreaElement>(null);
const fileInputRef = useRef<HTMLInputElement>(null); const fileInputRef = useRef<HTMLInputElement>(null);
const wasLoadingRef = useRef(false); const wasLoadingRef = useRef(false);
const prevTopicIdRef = useRef<string | null | undefined>(topicId);
// 切换话题时清空草稿(替代原来通过 key remount 的重置机制)
useEffect(() => {
if (prevTopicIdRef.current !== topicId) {
prevTopicIdRef.current = topicId;
setContent('');
setAttachments([]);
}
}, [topicId]);
useEffect(() => { useEffect(() => {
const textarea = textareaRef.current; const textarea = textareaRef.current;

View File

@ -58,6 +58,21 @@ export function McpTab({ config, update, setToast }: Props) {
return ( return (
<div className="space-y-4"> <div className="space-y-4">
<div className="rounded-xl border border-[var(--border-color)] bg-[var(--bg-secondary)]/60 p-4">
<Field label="工具调用超时(秒)" hint="0 = 不超时,默认 3005 分钟)">
<input
type="number"
value={config.mcp_tool_timeout_secs ?? 300}
onChange={(e) => {
const v = e.target.value;
update('mcp_tool_timeout_secs', v === '' ? 300 : Number(v));
}}
className={inputCls}
min={0}
step={30}
/>
</Field>
</div>
{mcpStatus && mcpStatus.enabled && ( {mcpStatus && mcpStatus.enabled && (
<div className="flex items-center gap-3 p-3 rounded-lg bg-[var(--bg-tertiary)] text-xs"> <div className="flex items-center gap-3 p-3 rounded-lg bg-[var(--bg-tertiary)] text-xs">
<div className="flex items-center gap-1.5"> <div className="flex items-center gap-1.5">

View File

@ -257,6 +257,7 @@ export interface AppConfig {
client: ClientConfig; client: ClientConfig;
channels: Record<string, ChannelConfig>; channels: Record<string, ChannelConfig>;
mcpServers: Record<string, McpServerConfig>; mcpServers: Record<string, McpServerConfig>;
mcp_tool_timeout_secs: number;
} }
export type TabId = export type TabId =

View File

@ -314,6 +314,39 @@ export function useMessages(options: UseMessagesOptions): UseMessagesReturn {
if (getSubagentTaskId(message)) { if (getSubagentTaskId(message)) {
// 子代理执行完成bump 统一 triggerApp.tsx 根据 subAgentView 分派 load_task_messages // 子代理执行完成bump 统一 triggerApp.tsx 根据 subAgentView 分派 load_task_messages
bumpTopicRefreshTrigger(); bumpTopicRefreshTrigger();
// 更新主视图中 task tool result 占位消息的状态
// task 工具返回时 status='running'(黄色转圈),子代理完成后需更新为最终状态
if (msg.subagent_task_id && msg.subagent_status) {
const taskId = msg.subagent_task_id;
const newStatus = msg.subagent_status;
const newSummary = msg.subagent_summary;
setMessages((prev) => {
let changed = false;
const updated = prev.map((m) => {
if (m.type !== 'tool_result' || m.toolName !== 'task') return m;
if (!m.content) return m;
// content 是 TaskToolResult JSON可能带 loop_detector 前缀)
const jsonStart = m.content.indexOf('{');
if (jsonStart < 0) return m;
try {
const parsed = JSON.parse(m.content.slice(jsonStart));
if (parsed.task_id !== taskId) return m;
if (parsed.status === newStatus) return m;
parsed.status = newStatus;
if (newSummary !== undefined) parsed.summary = newSummary;
const newJson = JSON.stringify(parsed);
changed = true;
const prefix = jsonStart > 0 ? m.content.slice(0, jsonStart) : '';
return { ...m, content: prefix + newJson };
} catch {
return m;
}
});
return changed ? updated : prev;
});
}
return true; return true;
} }
// 按 topic_id 移除处理状态,不论当前选中哪个话题。 // 按 topic_id 移除处理状态,不论当前选中哪个话题。

View File

@ -3,8 +3,49 @@ import ReactDOM from 'react-dom/client';
import App from './App'; import App from './App';
import './index.css'; import './index.css';
class ErrorBoundary extends React.Component<
{ children: React.ReactNode },
{ hasError: boolean; error: Error | null }
> {
constructor(props: { children: React.ReactNode }) {
super(props);
this.state = { hasError: false, error: null };
}
static getDerivedStateFromError(error: Error) {
return { hasError: true, error };
}
componentDidCatch(error: Error, info: React.ErrorInfo) {
console.error('ErrorBoundary caught:', error, info.componentStack);
}
render() {
if (this.state.hasError) {
return (
<div style={{ padding: '20px', color: '#ff6b6b', background: '#1a1a1a', minHeight: '100vh', fontFamily: 'monospace', whiteSpace: 'pre-wrap' }}>
<h2>React Render Error</h2>
<p><strong>{this.state.error?.name}:</strong> {this.state.error?.message}</p>
<pre>{this.state.error?.stack}</pre>
<hr />
<p>Try clearing browser cache and localStorage, then refresh.</p>
<button
onClick={() => { localStorage.clear(); location.reload(); }}
style={{ marginTop: '10px', padding: '8px 16px', cursor: 'pointer' }}
>
Clear localStorage & Refresh
</button>
</div>
);
}
return this.props.children;
}
}
ReactDOM.createRoot(document.getElementById('root')!).render( ReactDOM.createRoot(document.getElementById('root')!).render(
<React.StrictMode> <React.StrictMode>
<App /> <ErrorBoundary>
<App />
</ErrorBoundary>
</React.StrictMode>, </React.StrictMode>,
); );

View File

@ -303,6 +303,9 @@ export interface ExecutionCompleted {
topic_id?: string; topic_id?: string;
timestamp?: number; timestamp?: number;
subagent_task_id?: string; subagent_task_id?: string;
/** 子代理最终状态completed/failed/timeout/cancelled/interrupted */
subagent_status?: string;
subagent_summary?: string;
} }
export type WsOutbound = export type WsOutbound =
@ -504,7 +507,15 @@ export interface ChatMessage {
/** task 工具返回的 JSON 结构 */ /** task 工具返回的 JSON 结构 */
export interface TaskToolResult { export interface TaskToolResult {
status: 'success' | 'failed' | 'timeout'; // status 值由后端 TaskToolResult.status 决定,包括:
// - running: 异步子代理刚 spawn 的占位结果
// - success: 子代理执行成功(旧)
// - completed: 子代理执行成功(新,与 SubagentStatus 对齐)
// - failed: 子代理执行失败
// - timeout: 子代理执行超时
// - cancelled: 子代理被用户取消
// - interrupted: 子代理因服务器重启被中断
status: string;
summary: string; summary: string;
output: string; output: string;
task_id: string; task_id: string;