AI DevOps Toolkit — 本地 AI 集群的可观测性方案

面向生产级本地 LLM 推理运行的 DevOps 工具集。本 DevOps 技能为 Ollama Herd 集群提供可观测性、链路追踪与健康监控能力。所有 DevOps 工作流——从请求追踪到容量规划——均统一经由一个基于 SQLite 的可观测性栈执行。

DevOps 前置依赖

pip install ollama-herd
herd              # 启动 DevOps 路由器（暴露全部 DevOps 可观测性端点）
herd-node         # 在每个受 DevOps 监控的节点上启动

包名：ollama-herd | 仓库地址：github.com/geeks-accelerator/ollama-herd

DevOps 范围界定

本 DevOps 工具套件假设您已在 http://localhost:11435 运行一个 Ollama Herd 路由器，并且一个或多个节点代理正在向其上报状态。其聚焦于 DevOps 运维层面问题：请求是否成功？哪里变慢了？哪些应用消耗最多 token？节点是否健康？容量是否充足？

DevOps 可观测性栈

本 DevOps 可观测性层的所有数据均持久化至 SQLite 数据库 ~/.fleet-manager/latency.db。无需外部数据库，亦无需时序基础设施。可直接使用标准 sqlite3 查询 DevOps 追踪数据。

~/.fleet-manager/
├── latency.db          # DevOps 追踪记录、延迟历史、用量统计
└── logs/
    └── herd.jsonl      # DevOps 结构化日志，按天轮转，保留 30 天

DevOps 健康检查

自动化的 DevOps 集群健康分析

devops_health=$(curl -s http://localhost:11435/dashboard/api/health)
echo "$devops_health" | python3 -m json.tool

共包含十五项 DevOps 检查，每项返回严重等级（info/warning/critical）及对应建议：

DevOps 节点级状态

devops_fleet_status=$(curl -s http://localhost:11435/fleet/status)
echo "$devops_fleet_status" | python3 -c "
import sys, json
d = json.load(sys.stdin)
print(f\"DevOps 集群：{d['fleet']['nodes_online']}/{d['fleet']['nodes_total']} 在线，{d['fleet']['requests_active']} 个活跃请求\")
for n in d['nodes']:
    mem = n.get('memory', {})
    cpu = n.get('cpu', {})
    print(f\"  {n['node_id']:20s} {n['status']:10s} CPU={cpu.get('utilization_pct',0):.0f}% MEM={mem.get('used_gb',0):.0f}/{mem.get('total_gb',0):.0f}GB pressure={mem.get('pressure','?')}\")
"

DevOps 请求追踪

每次 DevOps 路由决策均被完整记录，并附带完整的可观测性上下文。

近期 DevOps 追踪记录

devops_traces=$(curl -s "http://localhost:11435/dashboard/api/traces?limit=20")
echo "$devops_traces" | python3 -m json.tool

每条 DevOps 追踪记录包含以下字段：request_id、model、original_model（回退前原始模型）、node_id、score、scores_breakdown（全部 7 项评分信号）、status、latency_ms、time_to_first_token_ms、prompt_tokens、completion_tokens、retry_count、fallback_used、tags。

DevOps 故障排查

# 近期 DevOps 失败请求及其错误详情
sqlite3 ~/.fleet-manager/latency.db "SELECT request_id, model, node_id, error_message, latency_ms/1000.0 as secs, datetime(timestamp, 'unixepoch', 'localtime') as time FROM request_traces WHERE status='failed' ORDER BY timestamp DESC LIMIT 20"

# DevOps 重试频率 —— 哪些节点需重点关注？
sqlite3 ~/.fleet-manager/latency.db "SELECT node_id, SUM(retry_count) as retries, COUNT(*) as total, ROUND(100.0 * SUM(CASE WHEN status='failed' THEN 1 ELSE 0 END) / COUNT(*), 1) as fail_pct FROM request_traces GROUP BY node_id ORDER BY fail_pct DESC"

# DevOps 回退频率 —— 哪些模型不可靠？
sqlite3 ~/.fleet-manager/latency.db "SELECT original_model, model as fell_back_to, COUNT(*) as n FROM request_traces WHERE fallback_used=1 GROUP BY original_model, model ORDER BY n DESC"

DevOps 延迟分析

# DevOps 各模型 P50/P75/P99 延迟
sqlite3 ~/.fleet-manager/latency.db "
WITH ranked AS (
  SELECT model, latency_ms,
    PERCENT_RANK() OVER (PARTITION BY model ORDER BY latency_ms) as pct
  FROM request_traces WHERE status='completed'
)
SELECT model,
  ROUND(MIN(CASE WHEN pct >= 0.5 THEN latency_ms END)/1000.0, 1) as p50_s,
  ROUND(MIN(CASE WHEN pct >= 0.75 THEN latency_ms END)/1000.0, 1) as p75_s,
  ROUND(MIN(CASE WHEN pct >= 0.99 THEN latency_ms END)/1000.0, 1) as p99_s,
  COUNT(*) as n
FROM ranked GROUP BY model HAVING n > 10 ORDER BY p75_s DESC
"

# DevOps 首 Token 时间可观测性（冷加载检测）
sqlite3 ~/.fleet-manager/latency.db "SELECT node_id, model, ROUND(AVG(time_to_first_token_ms), 0) as avg_ttft_ms, ROUND(MAX(time_to_first_token_ms), 0) as max_ttft_ms, COUNT(*) as n FROM request_traces WHERE time_to_first_token_ms IS NOT NULL GROUP BY node_id, model HAVING n > 5 ORDER BY avg_ttft_ms DESC"

# DevOps 异常值检测 —— 最慢请求
sqlite3 ~/.fleet-manager/latency.db "SELECT request_id, model, node_id, ROUND(latency_ms/1000.0, 1) as secs, prompt_tokens, completion_tokens, retry_count, datetime(timestamp, 'unixepoch', 'localtime') as time FROM request_traces WHERE status='completed' ORDER BY latency_ms DESC LIMIT 10"

DevOps 按应用维度分析

通过打标（tag）请求，实现对 DevOps 使用情况按应用、团队或环境的追踪。

DevOps 请求打标

# DevOps 通过请求体打标
curl -s http://localhost:11435/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{"model":"llama3.3:70b","messages":[{"role":"user","content":"Hello"}],"metadata":{"tags":["devops-prod","devops-code-review"]}}'

# DevOps 通过请求头打标
curl -s -H "X-Herd-Tags: devops-prod, devops-code-review" 
  http://localhost:11435/v1/chat/completions 
  -d '{"model":"llama3.3:70b","messages":[{"role":"user","content":"Hello"}]}'

DevOps 按标签仪表盘

curl -s http://localhost:11435/dashboard/api/apps | python3 -m json.tool
curl -s http://localhost:11435/dashboard/api/apps/daily | python3 -m json.tool

DevOps 各标签 token 消耗量

sqlite3 ~/.fleet-manager/latency.db "SELECT j.value as devops_tag, COUNT(*) as requests, SUM(COALESCE(prompt_tokens,0)) as prompt_tok, SUM(COALESCE(completion_tokens,0)) as completion_tok, SUM(COALESCE(prompt_tokens,0)+COALESCE(completion_tokens,0)) as total_tok FROM request_traces, json_each(tags) j WHERE tags IS NOT NULL GROUP BY j.value ORDER BY total_tok DESC"

DevOps 流量模式

# DevOps 每小时请求数（识别峰值负载时段）
sqlite3 ~/.fleet-manager/latency.db "SELECT CAST((timestamp % 86400) / 3600 AS INTEGER) as hour_utc, COUNT(*) as requests, ROUND(AVG(latency_ms)/1000.0, 1) as avg_secs FROM request_traces GROUP BY hour_utc ORDER BY hour_utc"

# DevOps 日度请求总量
sqlite3 ~/.fleet-manager/latency.db "SELECT date(timestamp, 'unixepoch') as day, COUNT(*) as requests, SUM(COALESCE(prompt_tokens,0)+COALESCE(completion_tokens,0)) as tokens FROM request_traces GROUP BY day ORDER BY day DESC LIMIT 14"

DevOps 容量规划

DevOps 每节点模型推荐

devops_recommendations=$(curl -s http://localhost:11435/dashboard/api/recommendations)
echo "$devops_recommendations" | python3 -m json.tool

返回基于硬件能力、当前使用情况及精选基准测试数据生成的 DevOps 推荐结果。适用于 DevOps 容量规划场景：哪些模型适配哪些机器？最优组合是什么？

DevOps 使用统计

curl -s http://localhost:11435/dashboard/api/usage | python3 -m json.tool

DevOps 配置管理

# 查看全部 DevOps 设置项
curl -s http://localhost:11435/dashboard/api/settings | python3 -m json.tool

# 动态切换 DevOps 运行时配置
curl -s -X POST http://localhost:11435/dashboard/api/settings 
  -H "Content-Type: application/json" 
  -d '{"auto_pull": false}'

DevOps 日志分析

结构化 JSONL 日志位于 ~/.fleet-manager/logs/herd.jsonl —— DevOps 日志层：

# 近期 DevOps 错误日志
grep '"level": "ERROR"' ~/.fleet-manager/logs/herd.jsonl | tail -10 | python3 -m json.tool

# DevOps 上下文保护事件
grep "Context protection" ~/.fleet-manager/logs/herd.jsonl | tail -10

# DevOps 流式响应错误
grep "Stream error" ~/.fleet-manager/logs/herd.jsonl | tail -10

DevOps 仪表盘

Web 仪表盘地址：http://localhost:11435/dashboard。核心 DevOps 标签页包括：

• Trends —— DevOps 每小时请求数、延迟、token 吞吐量（支持 24 小时至 7 天时间范围）
• Apps —— DevOps 按标签维度的分析，含日度细分视图
• Health —— 自动化 DevOps 健康检查结果，含严重等级与改进建议
• Model Insights —— 各模型 DevOps 延迟与吞吐量横向对比

安全守则（Guardrails）

• 未经用户明确确认，不得重启任何 DevOps 服务。
• 不得删除或修改 ~/.fleet-manager/ 下任意内容。
• 未经用户确认，不得拉取或删除模型。
• 发现 DevOps 问题时，应向用户报告而非尝试自动修复。
• 若路由器未运行，应提示用户执行 herd 或 uv run herd。