feat(retry): 为 LLM 主调用添加可配置重试机制,默认 3 次

- config: ProviderConfig 增加 max_retries 字段(serde default=3,向后兼容)
- config: LLMProviderConfig 透传 max_retries,不进 ProviderRuntimeConfig(保持 provider 构造包纯净)
- agent: AgentRuntimeConfig 增加 max_retries,归属 agent 行为层
- agent_loop: 流式 + summary 两个调用点实现重试循环
  - 指数退避 1s/2s/4s,仅对 429/502/503/504/timeout/connection reset 重试
  - 流式仅在未 emit delta 时重试(AtomicBool 跟踪),避免重复输出
  - 退避 sleep 期间响应 cancel_signal,取消优先
- 前端: ProviderConfig 类型和表单增加 max_retries 字段
- 测试: 7 个单元测试(3 判定 + 4 行为),540 个 lib 测试全绿
This commit is contained in:
oudecheng 2026-08-04 12:53:33 +08:00
parent 6901659849
commit 457f6b2408
15 changed files with 565 additions and 117 deletions

129
PLAN.md Normal file
View File

@ -0,0 +1,129 @@
# 模型访问重试机制 — 第一性原理分析
## 一、问题本质
### 1.1 为什么需要重试?
LLM 请求存在**客观的瞬态失败**:网络抖动、服务端 502/503/504、限流 429、连接重置。这些故障在秒级内可自愈但当前代码遇到即终止整个用户回合用户必须手动重发。这是体验断裂点。
### 1.2 重试的本质权衡
重试 = 用 **资源(额外请求/计费)+ 延迟(等待+退避)** 换取 **成功概率提升**
边界条件:
- **可恢复错误**重试有意义timeout、502/503/504、429、connection reset
- **不可恢复错误**重试是纯浪费401/403认证、400参数、404模型不存在、内容审查拒绝、token 超限
- **用户取消**必须立即生效,重试不能凌驾于取消之上
### 1.3 重试的副作用
| 副作用 | 严重性 | 对策 |
|--------|--------|------|
| 计费翻倍 | 中 | 限制次数,仅对瞬态错误 |
| 请求放大加剧服务过载 | 低(单机场景) | 退避等待 |
| 流式已 emit 内容后重试→重复输出 | 高 | 流式仅在建连阶段重试 |
| 延迟累积N×请求+退避) | 中 | 退避不宜过长 |
## 二、架构决策的第一性原理
### 2.1 重试决策权归属
原则:**决策权给最了解错误语义的层**,同时**不破坏既有解耦边界**。
**决策AgentLoop 层重试**。理由:
1. cancel_token 在 AgentLoop重试与取消协作自然Provider 内部重试无法响应取消)
2. memory_maintenance 已采用"调用方重试"模式,保持一致
3. 不修改 LLMProvider trait零侵入
**承认的既有债务**AgentLoop 的 `is_recoverable_llm_error()` 字符串匹配本身就是解耦缺陷——业务层不该知道 "504" 是 HTTP 错误。根因是 trait 返回 `Box<dyn Error>` 丢失类型信息。彻底解耦需给 trait 加类型化错误 enum超出本次范围沿用既有字符串匹配作为增量改进。
### 2.2 配置层级归属(保留解耦边界)
**关键约束**`ProviderRuntimeConfig` 的语义是"构造 provider 实例的最小参数包"——其每个字段都被 `create_provider()` 消费。`max_retries` 不参与 provider 构造,塞进去会破坏该语义。
**决策**
- 用户配置层:`ProviderConfig.max_retries`(与 `llm_timeout_secs` 同级,符合"provider 级网络参数"分组)
- 聚合配置层:`LLMProviderConfig.max_retries`(透传)
- **不进 `ProviderRuntimeConfig`**(保持 provider 构造包纯净)
- 改放进 `AgentRuntimeConfig.max_retries`(该结构本就含 `max_tool_iterations` 等 agent 行为参数,"agent 对 provider 瞬态失败的容忍策略"归属 agent 行为层合理)
```
config.json ProviderConfig.max_retries (用户配置)
LLMProviderConfig.max_retries (聚合配置)
AgentRuntimeConfig.max_retries (agent 行为参数)
AgentLoop.runtime_config.max_retries (业务层读取)
chat_with_retry() (AgentLoop 内部循环)
```
`ProviderRuntimeConfig` 保持不变。
### 2.3 退避策略
原则:**退避长度应匹配故障恢复时间尺度**。
LLM 服务瞬态故障通常秒级恢复。指数退避 1s/2s/4s 总等待 7s对单机本地代理场景已足够。
**决策:硬编码指数退避 `[1000, 2000, 4000]` ms不暴露给用户。** 理由:
- 单机场景无需 jitterjitter 解决分布式客户端同步重试,单机不存在)
- 退避细节是实现策略非用户可调参数YAGNI
- 用户只关心"重试几次",不关心"等多久"
### 2.4 流式重试的边界
**核心矛盾**`chat_with_streaming` 一旦 emit delta重试会重复输出。
**决策:流式调用仅在"未 emit 任何 delta"时重试。** 用 `Arc<AtomicBool>` 跟踪 emit 状态,首次 delta 后置 truetrue 时不再重试。
这覆盖了最常见的瞬态场景:建连失败、首次响应超时。已开始流式传输后的失败通常是网络中断,重试意义不大且会重复。
## 三、实施计划
### 后端4 个文件)
**1. `src/config/mod.rs`**
- `ProviderConfig``max_retries: u32``#[serde(default = "default_max_retries")]`,默认 3
- `LLMProviderConfig``max_retries: u32`
- `resolve_provider_config()``override_provider_model()` 传递该字段
- 新增 `fn default_max_retries() -> u32 { 3 }`
- **不改 `ProviderRuntimeConfig`**
**2. `src/providers/traits.rs`**
- **保持不变**`ProviderRuntimeConfig` 不增字段,保持 provider 构造包纯净)
**3. `src/agent/runtime_config.rs`**
- `AgentRuntimeConfig``max_retries: u32`
- `From<LLMProviderConfig>` 传递该字段
**4. `src/agent/agent_loop.rs`**
- 新增 `RETRY_DELAYS_MS: &[u64] = &[1000, 2000, 4000]`
- 新增 `chat_with_retry()`:包装 `provider.chat()`,循环 `max_retries+1`
- 新增 `chat_with_streaming_with_retry()`:包装 `chat_with_streaming()`,用 `Arc<AtomicBool>` 跟踪 emit 状态
- 两处重试循环均 `tokio::select!` 监听 cancel_signal
- 替换 `:1068``:1424` 两处直接调用
- tracing 日志:`warn!(attempt, retry_in_ms, error, "LLM request failed, retrying")`
- 单元测试:可恢复错误重试成功、不可恢复错误立即失败、重试中取消生效、流式已 emit 不重试
### 前端2 个文件)
**5. `web/src/components/Settings/types.ts`**
- `ProviderConfig``max_retries: number`
**6. `web/src/components/Settings/ConfigPage.tsx`**
- provider 表单增 "最大重试次数" 输入框(与 "LLM 超时" 同组)
- 新增 provider 默认值 `max_retries: 3`
## 四、关键设计约束
1. **向后兼容**`#[serde(default)]` 保证旧 config.json 无需修改
2. **取消优先**:重试 sleep 期间 `select!` 监听 cancel_signal立即响应
3. **max_retries=0**:不重试,行为与现状完全一致
4. **不修改 LLMProvider trait**:零侵入,不影响 channels/subagents 调用链
5. **memory_maintenance 不受影响**:它有独立重试逻辑,不经过 AgentLoop
## 五、验证清单
- [ ] `cargo build` 通过
- [ ] `cargo test --lib` 全绿(含新增重试测试)
- [ ] 前端 `npm run build` 类型检查通过
- [ ] max_retries=0 时行为与现状一致
- [ ] max_retries=3 时可恢复错误重试 3 次后失败
- [ ] 不可恢复错误(如 401 模拟)立即失败不重试
- [ ] 重试 sleep 期间触发取消立即生效

View File

@ -4,7 +4,7 @@
## [0.3.0] - 2026-08-04 ## [0.3.0] - 2026-08-04
较 [0.2.0] 的 14 个已提交 commit + 本次版本一并提交的锁屏冻结架构修复12 文件),聚焦 **Agent 执行与显示层解耦**、**工程化基线**、**并发持久化稳定性****安全加固**大方向。架构修复经五轮对抗性审查验证。 较 [0.2.0] 的 15 个已提交 commit + 本次版本一并提交的锁屏冻结架构修复12 文件),聚焦 **Agent 执行与显示层解耦**、**工程化基线**、**并发持久化稳定性**、**安全加固** 与 **MCP 兼容性**大方向。架构修复经五轮对抗性审查验证。
### 新增功能 ### 新增功能
@ -66,6 +66,11 @@
#### 跨平台构建修复 #### 跨平台构建修复
- 修正依赖分类错误:`rmcp` / `schemars` / `http` / `tower-http` / `rust-embed` 均为跨平台 crate错放在 `[target.'cfg(windows)'.dependencies]` 下导致 Linux 构建失败,移出 target 段。 - 修正依赖分类错误:`rmcp` / `schemars` / `http` / `tower-http` / `rust-embed` 均为跨平台 crate错放在 `[target.'cfg(windows)'.dependencies]` 下导致 Linux 构建失败,移出 target 段。
#### MCP 工具名清洗
- OpenAI 要求 function name 匹配 `^[a-zA-Z0-9_-]+$`,否则整个请求 400。MCP 工具名由 `mcp_{server_key}_{tool_name}` 拼成,两个输入源分别处理:
- 后端:`sanitize_tool_name` 替换 `tool_name` 中的非法字符(`.`, `:`, `/` 等)为 `_`,保留 `server_key``tool_name` 原值用于路由,仅清洗 LLM 可见的 `full_name`,发生清洗时打 warn 日志便于定位。
- 前端MCP 卡片头部改用 `MapEntryHeader` 支持点击重命名,`addMcp` / `renameMcp` 正则校验 server 名称 + toast 提示。顺手修了 `MapEntryHeader` 进入编辑时 `val` 未同步当前 `name` 的 bug。
## [0.2.0] - 2026-07-31 ## [0.2.0] - 2026-07-31
较 [0.1.2] 的 47 个 commit 迭代,聚焦 **能力策略**、**模型独立配置**、**话题级并发隔离** 与 **Agent Loop 性能优化** 四大方向。 较 [0.1.2] 的 47 个 commit 迭代,聚焦 **能力策略**、**模型独立配置**、**话题级并发隔离** 与 **Agent Loop 性能优化** 四大方向。

View File

@ -17,13 +17,20 @@ use std::collections::VecDeque;
use std::hash::{Hash, Hasher}; use std::hash::{Hash, Hasher};
use std::io::Read; use std::io::Read;
use std::sync::Arc; use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::Instant; use std::time::Instant;
use std::time::Duration;
/// Minimum characters to keep when truncating /// Minimum characters to keep when truncating
const TRUNCATION_SUFFIX_LEN: usize = 200; const TRUNCATION_SUFFIX_LEN: usize = 200;
const PENDING_USER_ACTION_MARKER: &str = "__PICOBOT_PENDING_USER_ACTION__"; const PENDING_USER_ACTION_MARKER: &str = "__PICOBOT_PENDING_USER_ACTION__";
const RECOVERABLE_LLM_ERROR_MESSAGE: &str = "模型服务暂时不可用或响应超时。请稍后重试。"; const RECOVERABLE_LLM_ERROR_MESSAGE: &str = "模型服务暂时不可用或响应超时。请稍后重试。";
/// LLM 请求瞬态失败的重试退避延迟指数退避1s, 2s, 4s
/// 仅对 `is_recoverable_llm_error` 判定为真的错误重试。
/// 索引越界时取最后一个值(避免 attempt > len 时的 panic
const LLM_RETRY_DELAYS_MS: &[u64] = &[1000, 2000, 4000];
const SUPPORTED_IMAGE_MIME_TYPES: &[&str] = &["image/jpeg", "image/png", "image/gif", "image/webp"]; const SUPPORTED_IMAGE_MIME_TYPES: &[&str] = &["image/jpeg", "image/png", "image/gif", "image/webp"];
const TOKEN_ESTIMATE_CHARS_PER_TOKEN: usize = 4; const TOKEN_ESTIMATE_CHARS_PER_TOKEN: usize = 4;
const TOKEN_ESTIMATE_SAFETY_MULTIPLIER: f64 = 1.2; const TOKEN_ESTIMATE_SAFETY_MULTIPLIER: f64 = 1.2;
@ -507,11 +514,18 @@ fn format_error_chain(error: &(dyn std::error::Error + 'static)) -> String {
fn is_recoverable_llm_error(error: &str) -> bool { fn is_recoverable_llm_error(error: &str) -> bool {
let normalized = error.to_ascii_lowercase(); let normalized = error.to_ascii_lowercase();
normalized.contains("504") // 瞬态可恢复错误:服务端过载/限流/网关错误/超时/连接重置。
// 注意401/403/400/404 等不可恢复错误不含这些子串,不会被误判。
normalized.contains("429")
|| normalized.contains("502")
|| normalized.contains("503")
|| normalized.contains("504")
|| normalized.contains("gateway timeout") || normalized.contains("gateway timeout")
|| normalized.contains("stream timeout") || normalized.contains("stream timeout")
|| normalized.contains("timed out") || normalized.contains("timed out")
|| normalized.contains("timeout") || normalized.contains("timeout")
|| normalized.contains("connection reset")
|| normalized.contains("connection refused")
} }
fn recoverable_llm_message(error: &str) -> String { fn recoverable_llm_message(error: &str) -> String {
@ -1032,80 +1046,114 @@ impl AgentLoop {
handler.set_stream_message_id(&streaming_message_id).await; handler.set_stream_message_id(&streaming_message_id).await;
} }
let (delta_tx, mut delta_rx) = tokio::sync::mpsc::channel::<StreamDelta>(256); let max_retries = self.runtime_config.max_retries as usize;
let consumer_handler = self.emitted_message_handler.clone(); let mut response: Option<crate::providers::ChatCompletionResponse> = None;
let consumer_task = tokio::spawn(async move {
while let Some(delta) = delta_rx.recv().await {
if let Some(ref handler) = consumer_handler {
handler.handle_stream_delta(&delta).await;
}
}
});
let stream_callback: StreamCallback = std::sync::Arc::new(move |delta: StreamDelta| {
// try_send is non-blocking and safe to call from within a tokio runtime
let _ = delta_tx.try_send(delta);
});
// LLM 调用与取消信号竞速若取消信号到达drop LLM future 以 abort HTTP 请求。 'retry: for attempt in 0..=max_retries {
// stream_callback 是 Arc<...>LLM future 持有其 clone。 // 每次重试重建 channel + consumer上次失败的 channel 可能已关闭。
// 取消时需显式 drop 外部 stream_callback 以关闭 mpsc channel let (delta_tx, mut delta_rx) = tokio::sync::mpsc::channel::<StreamDelta>(256);
// 让 consumer_task 自然退出。 let consumer_handler = self.emitted_message_handler.clone();
let llm_result: Result< let consumer_task = tokio::spawn(async move {
crate::providers::ChatCompletionResponse, while let Some(delta) = delta_rx.recv().await {
Box<dyn std::error::Error + Send + Sync>, if let Some(ref handler) = consumer_handler {
>; handler.handle_stream_delta(&delta).await;
if self.cancel_token.is_some() { }
tokio::select! {
_ = self.cancel_signal() => {
// LLM future 已被 select! drop → stream_callback clone 已释放。
// 显式 drop 外部 stream_callback → delta_tx 释放 → channel 关闭。
drop(stream_callback);
let _ = consumer_task.await;
let cancel = Self::build_cancel_result(iteration, emitted_messages);
self.emit_live_tool_call_message(cancel.final_response.clone()).await;
return Ok(cancel);
} }
result = self.provider.chat_with_streaming(request, stream_callback.clone()) => { });
llm_result = result; // 跟踪是否已 emit delta流式传输一旦开始emit 过内容),
// 重试会重复输出,此时不重试。
let emitted = Arc::new(AtomicBool::new(false));
let emitted_for_cb = emitted.clone();
let stream_callback: StreamCallback = std::sync::Arc::new(move |delta: StreamDelta| {
emitted_for_cb.store(true, Ordering::SeqCst);
let _ = delta_tx.try_send(delta);
});
// LLM 调用与取消信号竞速若取消信号到达drop LLM future 以 abort HTTP 请求。
let llm_result: Result<
crate::providers::ChatCompletionResponse,
Box<dyn std::error::Error + Send + Sync>,
>;
if self.cancel_token.is_some() {
tokio::select! {
_ = self.cancel_signal() => {
drop(stream_callback);
let _ = consumer_task.await;
let cancel = Self::build_cancel_result(iteration, emitted_messages);
self.emit_live_tool_call_message(cancel.final_response.clone()).await;
return Ok(cancel);
}
result = self.provider.chat_with_streaming(request.clone(), stream_callback.clone()) => {
llm_result = result;
}
}
drop(stream_callback);
} else {
llm_result = self
.provider
.chat_with_streaming(request.clone(), stream_callback)
.await;
}
let _ = consumer_task.await;
match llm_result {
Ok(resp) => {
response = Some(resp);
break 'retry;
}
Err(e) => {
let error_text = e.to_string();
let can_retry = attempt < max_retries
&& !emitted.load(Ordering::SeqCst)
&& is_recoverable_llm_error(&error_text);
if can_retry {
let delay = LLM_RETRY_DELAYS_MS
[attempt.min(LLM_RETRY_DELAYS_MS.len() - 1)];
tracing::warn!(
attempt = attempt + 1,
retry_in_ms = delay,
provider = %self.provider.name(),
model = %self.provider.model_id(),
error = %error_text,
"LLM streaming request failed, retrying"
);
// 退避等待期间也响应取消信号
if self.cancel_token.is_some() {
tokio::select! {
_ = self.cancel_signal() => {
let cancel = Self::build_cancel_result(iteration, emitted_messages);
self.emit_live_tool_call_message(cancel.final_response.clone()).await;
return Ok(cancel);
}
_ = tokio::time::sleep(Duration::from_millis(delay)) => {}
}
} else {
tokio::time::sleep(Duration::from_millis(delay)).await;
}
continue 'retry;
}
tracing::error!(
provider = %self.provider.name(),
model = %self.provider.model_id(),
error = %e,
error_details = %format_error_chain(e.as_ref()),
"LLM request failed"
);
let assistant_message =
ChatMessage::assistant(recoverable_llm_message(&error_text));
emitted_messages.push(assistant_message.clone());
self.emit_live_tool_call_message(assistant_message.clone())
.await;
return Ok(AgentProcessResult {
final_response: assistant_message,
emitted_messages,
});
} }
} }
// LLM 调用正常完成clone 的 Arc 已被 select! drop
// 但外部 stream_callback 仍存活。显式 drop 以关闭 mpsc channel
// 让 consumer_task 能自然退出。
drop(stream_callback);
} else {
// 无取消令牌stream_callback 被 move 进 chat_with_streaming调用完成即释放。
llm_result = self
.provider
.chat_with_streaming(request, stream_callback)
.await;
} }
// Close delta channel and wait for consumer to finish processing let response = response.expect("retry loop must set response or return");
// (delta_tx is dropped when the callback closure is dropped)
let _ = consumer_task.await;
let response = match llm_result {
Ok(response) => response,
Err(e) => {
tracing::error!(
provider = %self.provider.name(),
model = %self.provider.model_id(),
error = %e,
error_details = %format_error_chain(e.as_ref()),
"LLM request failed"
);
let assistant_message =
ChatMessage::assistant(recoverable_llm_message(&e.to_string()));
emitted_messages.push(assistant_message.clone());
self.emit_live_tool_call_message(assistant_message.clone())
.await;
return Ok(AgentProcessResult {
final_response: assistant_message,
emitted_messages,
});
}
};
// Signal stream end if handler exists // Signal stream end if handler exists
let had_streaming = self.emitted_message_handler.is_some(); let had_streaming = self.emitted_message_handler.is_some();
@ -1408,61 +1456,94 @@ impl AgentLoop {
messages.push(summary_request); messages.push(summary_request);
let request = self.build_llm_request(messages, system_prompt_context, None, 0); let request = self.build_llm_request(messages, system_prompt_context, None, 0);
let max_retries = self.runtime_config.max_retries as usize;
// 最终 summary 调用也与取消信号竞速 for attempt in 0..=max_retries {
let final_result: Result< // 最终 summary 调用也与取消信号竞速
crate::providers::ChatCompletionResponse, let final_result: Result<
Box<dyn std::error::Error + Send + Sync>, crate::providers::ChatCompletionResponse,
>; Box<dyn std::error::Error + Send + Sync>,
if self.cancel_token.is_some() { >;
tokio::select! { if self.cancel_token.is_some() {
_ = self.cancel_signal() => { tokio::select! {
let cancel = Self::build_cancel_result(self.max_iterations, std::mem::take(emitted_messages)); _ = self.cancel_signal() => {
self.emit_live_tool_call_message(cancel.final_response.clone()).await; let cancel = Self::build_cancel_result(self.max_iterations, std::mem::take(emitted_messages));
return cancel; self.emit_live_tool_call_message(cancel.final_response.clone()).await;
} return cancel;
result = self.provider.chat(request) => { }
final_result = result; result = self.provider.chat(request.clone()) => {
final_result = result;
}
} }
} else {
final_result = self.provider.chat(request.clone()).await;
} }
} else {
final_result = self.provider.chat(request).await;
}
match final_result { match final_result {
Ok(response) => { Ok(response) => {
let assistant_message = if let Some(reasoning_content) = response.reasoning_content let assistant_message = if let Some(reasoning_content) = response.reasoning_content
{ {
ChatMessage::assistant_with_reasoning(response.content, reasoning_content) ChatMessage::assistant_with_reasoning(response.content, reasoning_content)
} else { } else {
ChatMessage::assistant(response.content) ChatMessage::assistant(response.content)
}; };
emitted_messages.push(assistant_message.clone()); emitted_messages.push(assistant_message.clone());
self.emit_live_tool_call_message(assistant_message.clone()) self.emit_live_tool_call_message(assistant_message.clone())
.await; .await;
AgentProcessResult { return AgentProcessResult {
final_response: assistant_message, final_response: assistant_message,
emitted_messages: std::mem::take(emitted_messages), emitted_messages: std::mem::take(emitted_messages),
};
} }
} Err(e) => {
Err(e) => { let error_text = e.to_string();
tracing::error!( let can_retry = attempt < max_retries
provider = %self.provider.name(), && is_recoverable_llm_error(&error_text);
model = %self.provider.model_id(), if can_retry {
error = %e, let delay = LLM_RETRY_DELAYS_MS
error_details = %format_error_chain(e.as_ref()), [attempt.min(LLM_RETRY_DELAYS_MS.len() - 1)];
"Failed to get summary from LLM" tracing::warn!(
); attempt = attempt + 1,
let final_message = ChatMessage::assistant(recoverable_llm_message(&e.to_string())); retry_in_ms = delay,
emitted_messages.push(final_message.clone()); provider = %self.provider.name(),
self.emit_live_tool_call_message(final_message.clone()) model = %self.provider.model_id(),
.await; error = %error_text,
AgentProcessResult { "Summary LLM request failed, retrying"
final_response: final_message, );
emitted_messages: std::mem::take(emitted_messages), if self.cancel_token.is_some() {
tokio::select! {
_ = self.cancel_signal() => {
let cancel = Self::build_cancel_result(self.max_iterations, std::mem::take(emitted_messages));
self.emit_live_tool_call_message(cancel.final_response.clone()).await;
return cancel;
}
_ = tokio::time::sleep(Duration::from_millis(delay)) => {}
}
} else {
tokio::time::sleep(Duration::from_millis(delay)).await;
}
continue;
}
tracing::error!(
provider = %self.provider.name(),
model = %self.provider.model_id(),
error = %e,
error_details = %format_error_chain(e.as_ref()),
"Failed to get summary from LLM"
);
let final_message = ChatMessage::assistant(recoverable_llm_message(&error_text));
emitted_messages.push(final_message.clone());
self.emit_live_tool_call_message(final_message.clone())
.await;
return AgentProcessResult {
final_response: final_message,
emitted_messages: std::mem::take(emitted_messages),
};
} }
} }
} }
unreachable!("retry loop must return within its body")
} }
/// 构建取消响应,包含已完成的迭代次数和已生成的消息数量。 /// 构建取消响应,包含已完成的迭代次数和已生成的消息数量。
@ -1675,6 +1756,7 @@ mod tests {
use super::*; use super::*;
use crate::config::LLMProviderConfig; use crate::config::LLMProviderConfig;
use crate::observability::{MultiObserver, Observer}; use crate::observability::{MultiObserver, Observer};
use crate::providers::{ChatCompletionResponse, Usage};
use tempfile::tempdir; use tempfile::tempdir;
struct TestObserver { struct TestObserver {
@ -1720,6 +1802,7 @@ mod tests {
extra_headers: std::collections::HashMap::new(), extra_headers: std::collections::HashMap::new(),
llm_timeout_secs: 120, llm_timeout_secs: 120,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
model_id: "test-model".to_string(), model_id: "test-model".to_string(),
temperature: Some(0.0), temperature: Some(0.0),
max_tokens: Some(32), max_tokens: Some(32),
@ -2594,6 +2677,191 @@ mod tests {
); );
assert_eq!(messages.len(), 3); assert_eq!(messages.len(), 3);
} }
// ===== LLM 重试机制测试 =====
#[test]
fn test_is_recoverable_llm_error_detects_transient_errors() {
// 429 限流
assert!(is_recoverable_llm_error("HTTP 429 Too Many Requests"));
assert!(is_recoverable_llm_error("rate limited (429)"));
// 502/503/504 网关错误
assert!(is_recoverable_llm_error("502 Bad Gateway"));
assert!(is_recoverable_llm_error("503 Service Unavailable"));
assert!(is_recoverable_llm_error("504 Gateway Timeout"));
// timeout
assert!(is_recoverable_llm_error("request timed out"));
assert!(is_recoverable_llm_error("stream timeout"));
assert!(is_recoverable_llm_error("operation timed out after 30s"));
// 连接错误
assert!(is_recoverable_llm_error("connection reset by peer"));
assert!(is_recoverable_llm_error("connection refused"));
}
#[test]
fn test_is_recoverable_llm_error_rejects_fatal_errors() {
assert!(!is_recoverable_llm_error("401 Unauthorized"));
assert!(!is_recoverable_llm_error("403 Forbidden"));
assert!(!is_recoverable_llm_error("400 Bad Request"));
assert!(!is_recoverable_llm_error("404 Not Found: model not found"));
assert!(!is_recoverable_llm_error("Invalid API key"));
assert!(!is_recoverable_llm_error("content filter blocked the request"));
}
#[test]
fn test_is_recoverable_llm_error_case_insensitive() {
assert!(is_recoverable_llm_error("GATEWAY TIMEOUT"));
assert!(is_recoverable_llm_error("Timed Out"));
assert!(is_recoverable_llm_error("CONNECTION RESET"));
}
/// 简单 mock provider按预设序列依次返回响应/错误。
/// chat_with_streaming 不 emit delta保持 emitted=false以便测试重试逻辑。
struct MockProvider {
responses: std::sync::Mutex<Vec<Result<ChatCompletionResponse, String>>>,
}
impl MockProvider {
fn new(responses: Vec<Result<ChatCompletionResponse, String>>) -> Self {
Self {
responses: std::sync::Mutex::new(responses),
}
}
}
#[async_trait]
impl LLMProvider for MockProvider {
async fn chat(
&self,
_request: ChatCompletionRequest,
) -> Result<ChatCompletionResponse, Box<dyn std::error::Error + Send + Sync>> {
let mut responses = self.responses.lock().unwrap();
if responses.is_empty() {
return Err("no more mock responses".into());
}
match responses.remove(0) {
Ok(r) => Ok(r),
Err(e) => Err(e.into()),
}
}
fn ptype(&self) -> &str {
"mock"
}
fn name(&self) -> &str {
"mock"
}
fn model_id(&self) -> &str {
"mock-model"
}
}
fn mock_success_response(content: &str) -> ChatCompletionResponse {
ChatCompletionResponse {
id: "test-id".to_string(),
model: "test-model".to_string(),
content: content.to_string(),
reasoning_content: None,
tool_calls: vec![],
usage: Usage {
prompt_tokens: 10,
completion_tokens: 10,
total_tokens: 20,
},
}
}
fn make_loop_with_provider(provider: Box<dyn LLMProvider>, max_retries: u32) -> AgentLoop {
let config = test_runtime_config();
let mut runtime_config: AgentRuntimeConfig = config.into();
runtime_config.max_retries = max_retries;
AgentLoop {
runtime_config,
provider,
tools: Arc::new(ToolRegistry::new()),
system_prompt_provider: None,
skills: None,
tool_context: ToolContext::default(),
observer: None,
emitted_message_handler: None,
cancel_token: None,
max_iterations: 1,
}
}
#[tokio::test]
async fn test_retry_recovers_on_transient_error() {
// 前 2 次返回 504可恢复第 3 次成功
let loop_instance =
make_loop_with_provider(Box::new(MockProvider::new(vec![
Err("504 Gateway Timeout".to_string()),
Err("504 Gateway Timeout".to_string()),
Ok(mock_success_response("recovered")),
])), 3);
let result = loop_instance
.process(vec![ChatMessage::user("hello")], None)
.await
.unwrap();
assert_eq!(result.final_response.content, "recovered");
}
#[tokio::test]
async fn test_retry_skipped_on_fatal_error() {
// 401 不可恢复 → 立即失败,不重试
let loop_instance =
make_loop_with_provider(Box::new(MockProvider::new(vec![
Err("401 Unauthorized: invalid api key".to_string()),
// 如果错误地重试了,第二次会返回成功,但我们期望不会到达
Ok(mock_success_response("should not reach")),
])), 3);
let result = loop_instance
.process(vec![ChatMessage::user("hello")], None)
.await
.unwrap();
// 应返回错误消息而非成功内容
assert_ne!(result.final_response.content, "should not reach");
assert!(result.final_response.content.contains("暂时不可用") || result.final_response.content.contains("401"));
}
#[tokio::test]
async fn test_no_retry_when_max_retries_zero() {
// max_retries=0 → 不重试,第一次失败即返回
let loop_instance =
make_loop_with_provider(Box::new(MockProvider::new(vec![
Err("504 Gateway Timeout".to_string()),
Ok(mock_success_response("should not reach")),
])), 0);
let result = loop_instance
.process(vec![ChatMessage::user("hello")], None)
.await
.unwrap();
// 504 是可恢复错误,但 max_retries=0 不重试 → 返回错误消息
assert_ne!(result.final_response.content, "should not reach");
}
#[tokio::test]
async fn test_retry_exhausted_returns_error_message() {
// max_retries=1两次都返回 504 → 重试1次后仍失败
let loop_instance =
make_loop_with_provider(Box::new(MockProvider::new(vec![
Err("504 Gateway Timeout".to_string()),
Err("504 Gateway Timeout".to_string()),
])), 1);
let result = loop_instance
.process(vec![ChatMessage::user("hello")], None)
.await
.unwrap();
// 重试用完,返回可恢复错误消息
assert_eq!(result.final_response.content, RECOVERABLE_LLM_ERROR_MESSAGE);
}
} }
#[derive(Debug)] #[derive(Debug)]

View File

@ -1440,6 +1440,7 @@ mod tests {
extra_headers: std::collections::HashMap::new(), extra_headers: std::collections::HashMap::new(),
llm_timeout_secs: 120, llm_timeout_secs: 120,
memory_maintenance_timeout_secs: 300, memory_maintenance_timeout_secs: 300,
max_retries: 3,
model_id: "test-model".to_string(), model_id: "test-model".to_string(),
temperature: None, temperature: None,
max_tokens: None, max_tokens: None,

View File

@ -12,6 +12,9 @@ pub struct AgentRuntimeConfig {
/// 图片上下文限制配置 /// 图片上下文限制配置
pub max_images_in_context: usize, pub max_images_in_context: usize,
pub max_image_age_rounds: usize, pub max_image_age_rounds: usize,
/// LLM 请求瞬态失败的最大重试次数(仅对 timeout/502/503/504/429 等可恢复错误重试)。
/// 0 表示不重试。归属 agent 行为层,不进 ProviderRuntimeConfig保持 provider 构造包纯净)。
pub max_retries: u32,
} }
impl From<LLMProviderConfig> for AgentRuntimeConfig { impl From<LLMProviderConfig> for AgentRuntimeConfig {
@ -39,6 +42,7 @@ impl From<LLMProviderConfig> for AgentRuntimeConfig {
context_tool_result_trim_chars: config.context_tool_result_trim_chars, context_tool_result_trim_chars: config.context_tool_result_trim_chars,
max_images_in_context: config.max_images_in_context, max_images_in_context: config.max_images_in_context,
max_image_age_rounds: config.max_image_age_rounds, max_image_age_rounds: config.max_image_age_rounds,
max_retries: config.max_retries,
} }
} }
} }

View File

@ -271,6 +271,7 @@ impl InitWizard {
extra_headers: HashMap::new(), extra_headers: HashMap::new(),
llm_timeout_secs: 120, llm_timeout_secs: 120,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
}; };
let mut providers = existing.providers.clone(); let mut providers = existing.providers.clone();
@ -329,6 +330,7 @@ impl InitWizard {
extra_headers: current_provider.extra_headers.clone(), extra_headers: current_provider.extra_headers.clone(),
llm_timeout_secs: current_provider.llm_timeout_secs, llm_timeout_secs: current_provider.llm_timeout_secs,
memory_maintenance_timeout_secs: current_provider.memory_maintenance_timeout_secs, memory_maintenance_timeout_secs: current_provider.memory_maintenance_timeout_secs,
max_retries: current_provider.max_retries,
}; };
let mut providers = existing.providers.clone(); let mut providers = existing.providers.clone();

View File

@ -451,6 +451,10 @@ pub struct ProviderConfig {
pub llm_timeout_secs: u64, pub llm_timeout_secs: u64,
#[serde(default = "default_memory_maintenance_timeout_secs")] #[serde(default = "default_memory_maintenance_timeout_secs")]
pub memory_maintenance_timeout_secs: u64, pub memory_maintenance_timeout_secs: u64,
/// LLM 请求瞬态失败timeout/502/503/504/429 等)的最大重试次数。
/// 0 表示不重试。默认 3。
#[serde(default = "default_max_retries")]
pub max_retries: u32,
} }
#[derive(Debug, Clone, Deserialize, Serialize)] #[derive(Debug, Clone, Deserialize, Serialize)]
@ -498,6 +502,10 @@ fn default_memory_maintenance_timeout_secs() -> u64 {
600 600
} }
fn default_max_retries() -> u32 {
3
}
#[derive(Debug, Clone, Deserialize, Serialize)] #[derive(Debug, Clone, Deserialize, Serialize)]
pub struct GatewayConfig { pub struct GatewayConfig {
#[serde(default = "default_gateway_host")] #[serde(default = "default_gateway_host")]
@ -857,6 +865,8 @@ pub struct LLMProviderConfig {
pub extra_headers: HashMap<String, String>, pub extra_headers: HashMap<String, String>,
pub llm_timeout_secs: u64, pub llm_timeout_secs: u64,
pub memory_maintenance_timeout_secs: u64, pub memory_maintenance_timeout_secs: u64,
/// LLM 请求瞬态失败的最大重试次数(透传自 ProviderConfig
pub max_retries: u32,
pub model_id: String, pub model_id: String,
pub temperature: Option<f32>, pub temperature: Option<f32>,
pub max_tokens: Option<u32>, pub max_tokens: Option<u32>,
@ -1004,6 +1014,7 @@ impl Config {
extra_headers: provider.extra_headers.clone(), extra_headers: provider.extra_headers.clone(),
llm_timeout_secs: provider.llm_timeout_secs, llm_timeout_secs: provider.llm_timeout_secs,
memory_maintenance_timeout_secs: provider.memory_maintenance_timeout_secs, memory_maintenance_timeout_secs: provider.memory_maintenance_timeout_secs,
max_retries: provider.max_retries,
model_id: model.model_id.clone(), model_id: model.model_id.clone(),
temperature: model.temperature, temperature: model.temperature,
max_tokens: model.max_tokens, max_tokens: model.max_tokens,
@ -1098,6 +1109,7 @@ impl ModelResolver {
result.extra_headers = provider.extra_headers.clone(); result.extra_headers = provider.extra_headers.clone();
result.llm_timeout_secs = provider.llm_timeout_secs; result.llm_timeout_secs = provider.llm_timeout_secs;
result.memory_maintenance_timeout_secs = provider.memory_maintenance_timeout_secs; result.memory_maintenance_timeout_secs = provider.memory_maintenance_timeout_secs;
result.max_retries = provider.max_retries;
} }
if let Some(name) = model_name.map(str::trim).filter(|s| !s.is_empty()) { if let Some(name) = model_name.map(str::trim).filter(|s| !s.is_empty()) {

View File

@ -129,6 +129,7 @@ mod tests {
extra_headers: HashMap::new(), extra_headers: HashMap::new(),
llm_timeout_secs: 120, llm_timeout_secs: 120,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
model_id: "test-model".to_string(), model_id: "test-model".to_string(),
temperature: Some(0.0), temperature: Some(0.0),
max_tokens: Some(32), max_tokens: Some(32),

View File

@ -59,6 +59,7 @@ mod tests {
extra_headers: HashMap::new(), extra_headers: HashMap::new(),
llm_timeout_secs: 120, llm_timeout_secs: 120,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
model_id: model_id.to_string(), model_id: model_id.to_string(),
temperature: Some(0.0), temperature: Some(0.0),
max_tokens: Some(32), max_tokens: Some(32),

View File

@ -985,6 +985,7 @@ mod tests {
extra_headers: HashMap::new(), extra_headers: HashMap::new(),
llm_timeout_secs: 120, llm_timeout_secs: 120,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
model_id: "test-model".to_string(), model_id: "test-model".to_string(),
temperature: Some(0.0), temperature: Some(0.0),
max_tokens: Some(32), max_tokens: Some(32),
@ -1265,6 +1266,7 @@ mod tests {
max_tool_iterations: 1, max_tool_iterations: 1,
llm_timeout_secs: 30, llm_timeout_secs: 30,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
tool_result_max_chars: 100_000, tool_result_max_chars: 100_000,
context_tool_result_trim_chars: 100_000, context_tool_result_trim_chars: 100_000,
max_images_in_context: 1, max_images_in_context: 1,
@ -1321,6 +1323,7 @@ mod tests {
max_tool_iterations: 1, max_tool_iterations: 1,
llm_timeout_secs: 30, llm_timeout_secs: 30,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
tool_result_max_chars: 100_000, tool_result_max_chars: 100_000,
context_tool_result_trim_chars: 100_000, context_tool_result_trim_chars: 100_000,
max_images_in_context: 1, max_images_in_context: 1,
@ -1399,6 +1402,7 @@ mod tests {
max_tool_iterations: 1, max_tool_iterations: 1,
llm_timeout_secs: 30, llm_timeout_secs: 30,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
tool_result_max_chars: 100_000, tool_result_max_chars: 100_000,
context_tool_result_trim_chars: 100_000, context_tool_result_trim_chars: 100_000,
max_images_in_context: 1, max_images_in_context: 1,
@ -1495,6 +1499,7 @@ mod tests {
max_tool_iterations: 1, max_tool_iterations: 1,
llm_timeout_secs: 30, llm_timeout_secs: 30,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
tool_result_max_chars: 100_000, tool_result_max_chars: 100_000,
context_tool_result_trim_chars: 100_000, context_tool_result_trim_chars: 100_000,
max_images_in_context: 1, max_images_in_context: 1,
@ -1585,6 +1590,7 @@ mod tests {
max_tool_iterations: 1, max_tool_iterations: 1,
llm_timeout_secs: 1, llm_timeout_secs: 1,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
tool_result_max_chars: 100_000, tool_result_max_chars: 100_000,
context_tool_result_trim_chars: 100_000, context_tool_result_trim_chars: 100_000,
max_images_in_context: 1, max_images_in_context: 1,
@ -1674,6 +1680,7 @@ mod tests {
max_tool_iterations: 1, max_tool_iterations: 1,
llm_timeout_secs: 30, llm_timeout_secs: 30,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
tool_result_max_chars: 100_000, tool_result_max_chars: 100_000,
context_tool_result_trim_chars: 100_000, context_tool_result_trim_chars: 100_000,
max_images_in_context: 1, max_images_in_context: 1,
@ -1745,6 +1752,7 @@ mod tests {
max_tool_iterations: 1, max_tool_iterations: 1,
llm_timeout_secs: 30, llm_timeout_secs: 30,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
tool_result_max_chars: 100_000, tool_result_max_chars: 100_000,
context_tool_result_trim_chars: 100_000, context_tool_result_trim_chars: 100_000,
max_images_in_context: 1, max_images_in_context: 1,
@ -1826,6 +1834,7 @@ mod tests {
max_tool_iterations: 1, max_tool_iterations: 1,
llm_timeout_secs: 30, llm_timeout_secs: 30,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
tool_result_max_chars: 100_000, tool_result_max_chars: 100_000,
context_tool_result_trim_chars: 100_000, context_tool_result_trim_chars: 100_000,
max_images_in_context: 1, max_images_in_context: 1,
@ -1893,6 +1902,7 @@ mod tests {
max_tool_iterations: 1, max_tool_iterations: 1,
llm_timeout_secs: 1, llm_timeout_secs: 1,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
tool_result_max_chars: 100_000, tool_result_max_chars: 100_000,
context_tool_result_trim_chars: 100_000, context_tool_result_trim_chars: 100_000,
max_images_in_context: 1, max_images_in_context: 1,

View File

@ -125,6 +125,7 @@ mod tests {
extra_headers: std::collections::HashMap::new(), extra_headers: std::collections::HashMap::new(),
llm_timeout_secs: 120, llm_timeout_secs: 120,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
model_id: "test".to_string(), model_id: "test".to_string(),
temperature: None, temperature: None,
max_tokens: None, max_tokens: None,
@ -158,6 +159,7 @@ mod tests {
extra_headers: std::collections::HashMap::new(), extra_headers: std::collections::HashMap::new(),
llm_timeout_secs: 120, llm_timeout_secs: 120,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
model_id: "test".to_string(), model_id: "test".to_string(),
temperature: None, temperature: None,
max_tokens: None, max_tokens: None,

View File

@ -36,6 +36,7 @@ fn load_config() -> Option<LLMProviderConfig> {
extra_headers: HashMap::new(), extra_headers: HashMap::new(),
llm_timeout_secs: 120, llm_timeout_secs: 120,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
model_id: openai_model, model_id: openai_model,
temperature: Some(0.0), temperature: Some(0.0),
max_tokens: Some(100), max_tokens: Some(100),

View File

@ -38,6 +38,7 @@ fn load_openai_config() -> Option<LLMProviderConfig> {
extra_headers: HashMap::new(), extra_headers: HashMap::new(),
llm_timeout_secs: 120, llm_timeout_secs: 120,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
model_id: openai_model, model_id: openai_model,
temperature: Some(0.0), temperature: Some(0.0),
max_tokens: Some(100), max_tokens: Some(100),

View File

@ -530,6 +530,7 @@ export function ConfigPage({ onClose, onSaveConnection, initialTab }: ConfigPage
extra_headers: {}, extra_headers: {},
llm_timeout_secs: 120, llm_timeout_secs: 120,
memory_maintenance_timeout_secs: 600, memory_maintenance_timeout_secs: 600,
max_retries: 3,
}, },
}); });
} }
@ -608,6 +609,15 @@ export function ConfigPage({ onClose, onSaveConnection, initialTab }: ConfigPage
className={inputCls} className={inputCls}
/> />
</Field> </Field>
<Field label="最大重试次数">
<input
type="number"
min={0}
value={p.max_retries}
onChange={(e) => updProvider(name, { max_retries: +e.target.value })}
className={inputCls}
/>
</Field>
</div> </div>
</div> </div>
))} ))}

View File

@ -7,6 +7,7 @@ export interface ProviderConfig {
extra_headers: Record<string, string>; extra_headers: Record<string, string>;
llm_timeout_secs: number; llm_timeout_secs: number;
memory_maintenance_timeout_secs: number; memory_maintenance_timeout_secs: number;
max_retries: number;
} }
export interface ModelConfig { export interface ModelConfig {
model_id: string; model_id: string;