diff --git a/AGENTS.md b/AGENTS.md index 0366a39..7bbb946 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,6 +7,7 @@ This file is the operational contract for coding agents working in this reposito - `cargo build` — build the binary - `cargo run -- gateway` — start gateway server (binds `127.0.0.1:19876` by default) - `cargo run -- chat` — connect to gateway as CLI client (default `ws://127.0.0.1:19876/ws`) +- `docker compose up -d` — start the container with Gateway bound/published on `0.0.0.0:19876`; override `PICOBOT_GATEWAY_HOST`, `PICOBOT_PUBLISH_HOST`, or `PICOBOT_GATEWAY_PORT` as needed - WebUI — start Gateway, then open `http://127.0.0.1:19876/`; no separate frontend build is required - `cd webui && npm ci && npm run check && npm run build` — validate the Svelte WebUI independently (Node.js 20+); its local `dist/` is ignored - `cargo build` automatically runs an incremental WebUI production build into Cargo `OUT_DIR`; it runs `npm ci` only when `package-lock.json` is not represented by the installed dependency stamp @@ -15,7 +16,7 @@ This file is the operational contract for coding agents working in this reposito ## Config - Config load order: `~/.picobot/config.json` then fallback to `./config.json` (`Config::load_default` in `src/config/mod.rs`) -- `.env` (cwd) is loaded with a custom parser, not via dotenv crate; env var placeholders `` in config JSON are substituted +- `.env` files use a custom parser, not dotenv: load `/.env`, then `/.env`, while pre-existing process variables remain highest priority; config placeholders `` use the merged values - Config example: `resources/templates/config.example.json` (released to `~/.picobot/` on first run) - CLI TUI identity is stored in `~/.picobot/tui_client_id`; it is a non-secret stable chat scope used to restore dialogs across reconnects @@ -107,7 +108,7 @@ Scheduler → SessionManager scheduled execution → AgentLoop → Scheduler del - Session/message persistence uses SQLite via `sqlx`; DB stored in workspace as `picobot.db` by default - `ChannelManager` owns the `MessageBus` and all channel instances - `OutboundDispatcher` routes outbound messages to the correct channel via `ChannelManager` -- Config `.env` loading uses `unsafe { env::set_var(...) }` — don't refactor to safer patterns without understanding side effects +- Layered config/workspace `.env` loading uses `unsafe { env::set_var(...) }` during single-threaded startup — don't move it after Gateway tasks are spawned or refactor it without understanding process-wide side effects ## Change Workflow diff --git a/README.md b/README.md index 4e8d3a3..36953c4 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,13 @@ Gateway 首次启动时会把模板释放到 `~/.picobot/config.example.json`。 } ``` -`.env` 会由 PicoBot 自己解析。配置里的 `` 这类占位符会在 `.env` 和系统环境变量加载后替换。 +`.env` 会在启动时由 PicoBot 自己解析，不依赖 dotenv。环境变量按以下顺序分层，越靠后优先级越高： + +1. `config.json` 所在目录的 `.env`，作为所有 workspace 共用的基础配置。 +2. `workspace_dir/.env`，用于当前 workspace 的覆盖值。 +3. 启动 PicoBot 时进程中已有的环境变量，例如 Docker Compose 的 `environment`，优先级最高且不会被文件覆盖。 + +合并后的值既用于替换配置里的 `` 等占位符，也会写入 PicoBot 进程环境，供 MCP Server 和工具子进程继承。`workspace_dir` 的位置由配置目录层和进程环境决定；workspace 自己的 `.env` 不能反过来修改 `workspace_dir`。 ### 4. 启动 Gateway @@ -83,6 +89,32 @@ cargo run -- gateway 默认监听 `127.0.0.1:19876`。Gateway 启动后会把进程工作目录切到 `workspace_dir`，默认 SQLite 数据库也会写到该 workspace 下的 `picobot.db`。 +监听地址可通过配置文件或命令行覆盖。命令行参数优先于 `config.json`： + +```json +{ + "gateway": { + "host": "0.0.0.0", + "port": 19876 + } +} +``` + +```bash +picobot gateway --host 0.0.0.0 --port 19876 +``` + +Docker Compose 默认让容器内 Gateway 监听所有 IPv4 接口。监听地址、宿主机发布地址和端口均可通过环境变量调整： + +```bash +PICOBOT_GATEWAY_HOST=0.0.0.0 \ +PICOBOT_PUBLISH_HOST=192.168.1.10 \ +PICOBOT_GATEWAY_PORT=19876 \ +docker compose up -d +``` + +`PICOBOT_GATEWAY_HOST` 是容器内进程的监听地址；`PICOBOT_PUBLISH_HOST` 是 Docker 在宿主机上发布端口的地址。对局域网开放时应保持 `gateway.require_pairing=true`，并由防火墙限制可信网段。 + ### 5. 启动 CLI 客户端 另开一个终端： diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index ac2dfe6..9f7cc43 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -25,9 +25,11 @@ PicoBot 只有一个二进制，提供两种模式： Linux 上可通过 `picobot service install/start/stop/status/restart/uninstall` 管理 systemd 用户服务。unit 固定为 `picobot.service`，其主进程仍是普通 Gateway 模式，不引入额外 daemon/fork 层；异常退出由 systemd 按 `Restart=on-failure` 拉起。 +原生 Gateway 默认绑定 `127.0.0.1:19876`，`gateway.host`/`gateway.port` 可由命令行 `--host`/`--port` 覆盖。Docker Compose 为保证端口映射可达，默认向容器传入 `0.0.0.0:19876`；`PICOBOT_GATEWAY_HOST` 控制容器内监听地址，`PICOBOT_PUBLISH_HOST` 控制宿主机发布地址，`PICOBOT_GATEWAY_PORT` 同时控制监听与映射端口。 + CLI TUI 在 `~/.picobot/tui_client_id` 保存非敏感客户端标识，并通过 WebSocket 查询参数 `client_id` 传给 Gateway。`cli_chat` 以该标识作为稳定 chat scope；重连时恢复内存中的当前 dialog，Gateway 重启后则恢复该 scope 最近活跃的未归档 dialog。无效或缺失的标识会退化为连接级随机 scope。 -Gateway 启动时会切换进程工作目录到 `workspace_dir`。因此所有相对路径都应按 workspace 解释，不能假设仍位于源码仓库。 +Gateway 启动时先从配置目录 `.env`、workspace `.env` 和既有进程环境合并启动变量，再初始化日志并切换进程工作目录到 `workspace_dir`。优先级为进程环境 > workspace `.env` > 配置目录 `.env`；配置目录层先用于定位 workspace，workspace 层不得重定向自身位置。环境文件只在单线程启动阶段写入进程环境，不能移到后台任务启动之后。切换完成后所有相对路径都应按 workspace 解释，不能假设仍位于源码仓库。 ## 3. 组件关系 @@ -171,7 +173,7 @@ SessionManager 负责组装会话上下文：系统提示、Skills、召回的 K ### 安全边界 -- API Key 和渠道凭据只来自配置占位符、`.env` 或进程环境，不得写入仓库。 +- API Key 和渠道凭据只来自配置占位符、配置目录/workspace 的 `.env` 或进程环境，不得写入仓库。既有进程环境优先级最高，workspace `.env` 可覆盖配置目录 `.env`；日志只能记录所加载的文件路径，不能记录变量值。 - 日志不得输出 token、secret、Authorization header，或包含临时凭据的完整 URL；应记录脱敏后的 host/path 和必要诊断字段。 - Gateway 把 cwd 切到 workspace，因此相对文件路径和 Shell 默认从 workspace 开始；这不是硬沙箱。当前内置文件工具接受绝对路径，Bash 也可访问进程权限允许的位置。若某场景需要硬边界，必须显式配置/实现 allowed directory 和进程隔离。 - `http_request` 和 `web_fetch` 的私网/回环地址校验属于 SSRF 防线，重构网络层时不能绕过。 @@ -220,8 +222,8 @@ WebUI/TUI 文件字节通过受鉴权的 HTTP 接口流式传输，WebSocket 只 ### 启动 -1. 加载配置和 `.env`，初始化 WebUI 配对存储与本机管理密钥。 -2. 创建并切换到 workspace。 +1. 解析配置路径，加载配置目录 `.env`，据此定位 workspace，再加载 workspace `.env`；既有进程环境保持最高优先级，合并后重新解析配置。 +2. 初始化日志，创建并切换到 workspace，初始化 WebUI 配对存储与本机管理密钥。 3. 初始化 SQLite、MemoryManager、MessageBus 和 SessionManager；Scheduler 启用时幂等创建默认日常维护巡检。 4. 注册内置工具、渠道、MCP 工具和 Cron 工具。 5. 启动所有 Channel。 diff --git a/resources/skills/about-picobot/references/architecture.md b/resources/skills/about-picobot/references/architecture.md index 560e41b..646fa36 100644 --- a/resources/skills/about-picobot/references/architecture.md +++ b/resources/skills/about-picobot/references/architecture.md @@ -54,7 +54,7 @@ Scheduler → SessionManager.handle_cron_message → AgentLoop → send_message - SQLite 数据在 `{workspace}/picobot.db` - ChannelManager 持有 MessageBus 和所有 channel - OutboundDispatcher 通过 ChannelManager 路由出站消息 -- Config `.env` 加载使用 `unsafe { env::set_var(...) }` +- 配置目录 `.env` 与 workspace `.env` 仅在单线程启动阶段分层加载，并使用 `unsafe { env::set_var(...) }` 写入进程环境；优先级为既有进程环境 > workspace > 配置目录 - `browser` 工具只有在 `browser.enabled=true` 时注册，依赖 Chrome/Chromium 与 WebDriver - 同一 session 的普通消息串行处理，不同 session 可并发；session 队列容量为 32，满时明确拒绝 - 出站消息按 `(channel, chat_id)` 分 lane 保序；lane 容量为 64，慢目标不阻塞其他目标 diff --git a/resources/skills/about-picobot/references/config.md b/resources/skills/about-picobot/references/config.md index b6c3dfb..b4cbee4 100644 --- a/resources/skills/about-picobot/references/config.md +++ b/resources/skills/about-picobot/references/config.md @@ -1,7 +1,7 @@ # PicoBot 配置说明 配置文件加载顺序：`~/.picobot/config.json` → 当前目录 `./config.json`。 -占位符 `` 从环境变量替换，环境变量从 `.env` 文件或系统环境读取。 +占位符 `` 从启动环境替换。PicoBot 依次加载 `config.json` 同目录的 `.env`、`workspace_dir/.env`，最后保留启动进程已有环境变量作为最高优先级；workspace 层覆盖配置目录层。合并值也会进入进程环境，供 MCP 和工具子进程继承。workspace `.env` 不能修改用于定位自身的 `workspace_dir`。 Gateway WebUI 的“配置”页可以编辑实际加载的配置文件。读取时 API Key、secret、password 和 token 会显示为 `********`，保持掩码不变再保存会保留原值；写入采用同目录临时文件替换。运行配置保存后需要重启 Gateway，`USER.md` 与 `AGENTS.md` 的修改用于后续构建的 Agent 上下文。 diff --git a/src/config/mod.rs b/src/config/mod.rs index f05b366..8ba72b2 100644 --- a/src/config/mod.rs +++ b/src/config/mod.rs @@ -501,25 +501,57 @@ impl Config { Self::load_from(&path) } - fn load_from(path: &Path) -> Result> { - load_env_file()?; - let content = if path.exists() { - tracing::info!(path = %path.display(), "Config loaded"); - fs::read_to_string(path)? + pub(crate) fn load_from(path: &Path) -> Result> { + let config_path = if path.exists() { + path.to_path_buf() } else { - // Fallback to current directory - let fallback = Path::new("config.json"); + let fallback = env::current_dir() + .unwrap_or_else(|_| PathBuf::from(".")) + .join("config.json"); if fallback.exists() { - tracing::info!(path = %fallback.display(), "Config loaded from fallback path"); - fs::read_to_string(fallback)? + fallback } else { return Err(Box::new(ConfigError::ConfigNotFound( path.to_string_lossy().to_string(), ))); } }; - let content = resolve_env_placeholders(&content); - let config: Config = serde_json::from_str(&content)?; + + let content = fs::read_to_string(&config_path)?; + let process_env = collect_process_env(); + let config_env_path = config_path + .parent() + .unwrap_or_else(|| Path::new(".")) + .join(".env"); + let config_env = read_env_file(&config_env_path)?; + + // The config-directory layer selects the workspace. Loading the workspace + // layer first would be circular because its location comes from config.json. + let initial_env = merge_env_layers(&config_env, &HashMap::new(), &process_env); + let initial_content = resolve_env_placeholders(&content, &initial_env); + let initial_config: Config = serde_json::from_str(&initial_content)?; + let workspace_path = expand_path(&initial_config.workspace_dir); + let workspace_env_path = workspace_path.join(".env"); + let workspace_env = read_env_file(&workspace_env_path)?; + + let effective_env = merge_env_layers(&config_env, &workspace_env, &process_env); + let resolved_content = resolve_env_placeholders(&content, &effective_env); + let config: Config = serde_json::from_str(&resolved_content)?; + if config.workspace_dir != initial_config.workspace_dir { + return Err(format!( + "workspace .env cannot change workspace_dir (selected {}, resolved {})", + initial_config.workspace_dir, config.workspace_dir + ) + .into()); + } + + apply_env_layers(&config_env, &workspace_env, &process_env); + tracing::info!( + path = %config_path.display(), + config_env = %config_env_path.display(), + workspace_env = %workspace_env_path.display(), + "Config and layered environment loaded" + ); Ok(config) } @@ -582,34 +614,70 @@ impl std::fmt::Display for ConfigError { impl std::error::Error for ConfigError {} -fn load_env_file() -> Result<(), Box> { - let env_path = Path::new(".env"); - if env_path.exists() { - let content = fs::read_to_string(env_path)?; - for line in content.lines() { - let line = line.trim(); - if line.is_empty() || line.starts_with('#') { - continue; - } - if let Some((key, value)) = line.split_once('=') { - let key = key.trim(); - let value = value.trim().trim_matches('"').trim_matches('\''); - if !value.is_empty() { - // SAFETY: Setting environment variables for the current process - // is safe as we're only modifying our own process state - unsafe { env::set_var(key, value) }; - } +fn read_env_file(path: &Path) -> Result, Box> { + let mut values = HashMap::new(); + if !path.exists() { + return Ok(values); + } + + let content = fs::read_to_string(path)?; + for line in content.lines() { + let line = line.trim(); + if line.is_empty() || line.starts_with('#') { + continue; + } + if let Some((key, value)) = line.split_once('=') { + let key = key.trim(); + let value = value.trim().trim_matches('"').trim_matches('\''); + if !key.is_empty() && !value.is_empty() { + values.insert(key.to_string(), value.to_string()); } } } - Ok(()) + Ok(values) } -fn resolve_env_placeholders(content: &str) -> String { +fn collect_process_env() -> HashMap { + env::vars_os() + .filter_map(|(key, value)| Some((key.into_string().ok()?, value.into_string().ok()?))) + .collect() +} + +fn merge_env_layers( + config_env: &HashMap, + workspace_env: &HashMap, + process_env: &HashMap, +) -> HashMap { + let mut merged = config_env.clone(); + merged.extend(workspace_env.clone()); + merged.extend(process_env.clone()); + merged +} + +fn apply_env_layers( + config_env: &HashMap, + workspace_env: &HashMap, + process_env: &HashMap, +) { + for (key, value) in config_env.iter().chain(workspace_env) { + if !process_env.contains_key(key) { + // SAFETY: Config loading happens during single-threaded startup before + // Gateway background tasks are spawned. Existing process values are + // never modified, and the workspace layer intentionally overwrites the + // lower-priority config-directory layer. + unsafe { env::set_var(key, value) }; + } + } +} + +fn resolve_env_placeholders(content: &str, values: &HashMap) -> String { let re = Regex::new(r"<([A-Z_]+)>").expect("invalid regex"); re.replace_all(content, |caps: ®ex::Captures| { let var_name = &caps[1]; - env::var(var_name).unwrap_or_else(|_| caps[0].to_string()) + values + .get(var_name) + .cloned() + .unwrap_or_else(|| caps[0].to_string()) }) .to_string() } @@ -617,11 +685,25 @@ fn resolve_env_placeholders(content: &str) -> String { #[cfg(test)] mod tests { use super::*; + use std::ffi::OsString; + use std::sync::Mutex; - fn write_test_config() -> tempfile::NamedTempFile { - let file = tempfile::NamedTempFile::new().unwrap(); - std::fs::write( - file.path(), + struct TestConfig { + _dir: tempfile::TempDir, + path: PathBuf, + } + + impl TestConfig { + fn path(&self) -> &Path { + &self.path + } + } + + fn write_test_config() -> TestConfig { + let dir = tempfile::TempDir::new().unwrap(); + let path = dir.path().join("config.json"); + let workspace = dir.path().join("workspace"); + let mut content: serde_json::Value = serde_json::from_str( r#"{ "providers": { "aliyun": { @@ -659,7 +741,94 @@ mod tests { }"#, ) .unwrap(); - file + content["workspace_dir"] = serde_json::json!(workspace); + std::fs::write(&path, serde_json::to_vec_pretty(&content).unwrap()).unwrap(); + TestConfig { _dir: dir, path } + } + + static ENV_TEST_LOCK: Mutex<()> = Mutex::new(()); + + struct EnvRestore(Vec<(&'static str, Option)>); + + impl Drop for EnvRestore { + fn drop(&mut self) { + for (key, value) in self.0.drain(..) { + if let Some(value) = value { + // SAFETY: The test serializes mutations of these unique keys. + unsafe { env::set_var(key, value) }; + } else { + // SAFETY: The test serializes mutations of these unique keys. + unsafe { env::remove_var(key) }; + } + } + } + } + + #[test] + fn layered_env_uses_config_then_workspace_then_process_precedence() { + const VALUE: &str = "PICOBOT_ENV_LAYERING_VALUE"; + const SYSTEM: &str = "PICOBOT_ENV_LAYERING_SYSTEM"; + const CONFIG_ONLY: &str = "PICOBOT_ENV_LAYERING_CONFIG_ONLY"; + + let _lock = ENV_TEST_LOCK.lock().unwrap(); + let _restore = EnvRestore( + [VALUE, SYSTEM, CONFIG_ONLY] + .into_iter() + .map(|key| (key, env::var_os(key))) + .collect(), + ); + // SAFETY: Config loading is the code under test and these unique keys are + // protected by ENV_TEST_LOCK for the duration of the test. + unsafe { + env::remove_var(VALUE); + env::set_var(SYSTEM, "process"); + env::remove_var(CONFIG_ONLY); + } + + let dir = tempfile::TempDir::new().unwrap(); + let config_dir = dir.path().join("config"); + let workspace_dir = dir.path().join("workspace"); + fs::create_dir_all(&config_dir).unwrap(); + fs::create_dir_all(&workspace_dir).unwrap(); + fs::write( + config_dir.join(".env"), + format!("{VALUE}=config\n{SYSTEM}=config\n{CONFIG_ONLY}=config-only\n"), + ) + .unwrap(); + fs::write( + workspace_dir.join(".env"), + format!("{VALUE}=workspace\n{SYSTEM}=workspace\n"), + ) + .unwrap(); + + let config_path = config_dir.join("config.json"); + let config_json = serde_json::json!({ + "providers": { + "default": { + "type": "openai", + "base_url": "https://example.invalid/v1", + "api_key": format!("<{VALUE}>|<{SYSTEM}>|<{CONFIG_ONLY}>") + } + }, + "models": { "default": { "model_id": "test" } }, + "agents": { "default": { "provider": "default", "model": "default" } }, + "workspace_dir": workspace_dir + }); + fs::write( + &config_path, + serde_json::to_vec_pretty(&config_json).unwrap(), + ) + .unwrap(); + + let config = Config::load(config_path.to_str().unwrap()).unwrap(); + + assert_eq!( + config.providers["default"].api_key, + "workspace|process|config-only" + ); + assert_eq!(env::var(VALUE).unwrap(), "workspace"); + assert_eq!(env::var(SYSTEM).unwrap(), "process"); + assert_eq!(env::var(CONFIG_ONLY).unwrap(), "config-only"); } #[test] diff --git a/src/gateway/mod.rs b/src/gateway/mod.rs index 9d40aa1..d60b777 100644 --- a/src/gateway/mod.rs +++ b/src/gateway/mod.rs @@ -35,7 +35,14 @@ pub struct GatewayState { impl GatewayState { pub async fn new() -> Result> { let config_path = crate::config::resolve_default_config_path(); - let config = Config::load_default()?; + let config = Config::load_from(&config_path)?; + Self::from_config(config, config_path).await + } + + async fn from_config( + config: Config, + config_path: std::path::PathBuf, + ) -> Result> { let task_supervisor = TaskSupervisor::new(); let connection_shutdown = tokio_util::sync::CancellationToken::new(); let auth = auth::AuthManager::load( @@ -469,13 +476,16 @@ pub async fn run( host: Option, port: Option, ) -> Result<(), Box> { + let config_path = crate::config::resolve_default_config_path(); + let config = Config::load_from(&config_path)?; + // Initialize logging logging::init_logging(); - tracing::info!("Starting PicoBot Gateway"); + tracing::info!(config_path = %config_path.display(), "Starting PicoBot Gateway"); - let state = Arc::new(GatewayState::new().await?); + let state = Arc::new(GatewayState::from_config(config, config_path).await?); - // Start all channels (init already done in GatewayState::new) + // Start all channels (init already done while constructing GatewayState) state.channel_manager.start_all().await?; // Start message processing (inbound processor + control processor + outbound dispatcher)