8000 perf(storage): 空闲时跳过未变化的报表缓存与账本写入 by InfinityPacer · Pull Request #22 · InfinityPacer/meterleaf · GitHub
[go: up one dir, main page]

Skip to content
8000
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ METERLEAF_HOST=127.0.0.1
METERLEAF_PORT=4318
# 自动同步开启后:有可见页面 / 无可见页面的批次间隔,范围 5 秒至 1 小时。
METERLEAF_SYNC_VISIBLE_INTERVAL_MS=15000
METERLEAF_SYNC_HIDDEN_INTERVAL_MS=60000
METERLEAF_SYNC_HIDDEN_INTERVAL_MS=900000
# 报表缓存后台刷新间隔,范围 30 秒至 24 小时。
METERLEAF_REPORT_REFRESH_INTERVAL_MS=300000
METERLEAF_REPORT_REFRESH_INTERVAL_MS=86400000

# 只影响 USD 估算;credits 和源用量事实不变。
METERLEAF_USD_BASIS=subscription
Expand Down
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@

## [未发布]

## [0.6.4] - 2026-09-30

- 大幅减少空闲时的磁盘写入:报表缓存、账户与额度记录以及同步状态只在内容变化时写入。
- 报表定时刷新默认改为每天一次,无人查看页面时的后台同步默认改为 15 分钟一次;打开页面时报表和同步仍即时刷新。已在 `.env` 中显式设置 `METERLEAF_REPORT_REFRESH_INTERVAL_MS` 或 `METERLEAF_SYNC_HIDDEN_INTERVAL_MS` 的部署不受影响。

## [0.6.3] - 2026-09-30

- 宽屏总览的模型分布里,带日期的长模型名(如 `claude-haiku-4-5-20251001`)改为折行显示,不再压住右侧的用量数字。
Expand Down
4 changes: 2 additions & 2 deletions compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ services:
SUB2API_DATABASE_URL: ${SUB2API_DATABASE_URL:-}
METERLEAF_SOURCE_ID: ${METERLEAF_SOURCE_ID:-sub2api}
METERLEAF_LOG_LEVEL: ${METERLEAF_LOG_LEVEL:-info}
METERLEAF_REPORT_REFRESH_INTERVAL_MS: ${METERLEAF_REPORT_REFRESH_INTERVAL_MS:-300000}
METERLEAF_REPORT_REFRESH_INTERVAL_MS: ${METERLEAF_REPORT_REFRESH_INTERVAL_MS:-86400000}
METERLEAF_SYNC_VISIBLE_INTERVAL_MS: ${METERLEAF_SYNC_VISIBLE_INTERVAL_MS:-15000}
METERLEAF_SYNC_HIDDEN_INTERVAL_MS: ${METERLEAF_SYNC_HIDDEN_INTERVAL_MS:-60000}
METERLEAF_SYNC_HIDDEN_INTERVAL_MS: ${METERLEAF_SYNC_HIDDEN_INTERVAL_MS:-900000}
METERLEAF_USD_BASIS: ${METERLEAF_USD_BASIS:-subscription}
METERLEAF_PRICE_BOOK: ${METERLEAF_PRICE_BOOK:-}
METERLEAF_INGEST_KEYS: ${METERLEAF_INGEST_KEYS:-}
Expand Down
6 changes: 4 additions & 2 deletions docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,15 +57,17 @@ docker compose up -d --no-build --pull never
| `METERLEAF_BIND_ADDRESS` | `127.0.0.1` | Compose 对外绑定地址 |
| `METERLEAF_PUBLISHED_PORT` | `4318` | Compose 对外端口 |
| `METERLEAF_SYNC_VISIBLE_INTERVAL_MS` | `15000` | 页面可见时的自动同步间隔,5 秒至 1 小时 |
| `METERLEAF_SYNC_HIDDEN_INTERVAL_MS` | `60000` | 页面不可见时的自动同步间隔,5 秒至 1 小时 |
| `METERLEAF_REPORT_REFRESH_INTERVAL_MS` | `300000` | 报表后台刷新间隔,30 秒至 24 小时 |
| `METERLEAF_SYNC_HIDDEN_INTERVAL_MS` | `900000` | 页面不可见时的自动同步间隔,5 秒至 1 小时 |
| `METERLEAF_REPORT_REFRESH_INTERVAL_MS` | `86400000` | 报表后台兜底刷新间隔,30 秒至 24 小时 |
| `METERLEAF_USD_BASIS` | `subscription` | 默认 USD 口径,也可选 `api` |
| `METERLEAF_PRICE_BOOK` | 内置价格表 | 自定义完整价格 JSON 路径 |
| `METERLEAF_LOG_LEVEL` | `info` | `debug`、`info`、`warn`、`error`、`silent` |
| `METERLEAF_INGEST_KEYS` | 未启用 | 本机采集器写入密钥,见下文 |

`SUB2API_DATABASE_URL` 与 `METERLEAF_INGEST_KEYS` 至少填写一项,未填写的来源不启用。

打开页面时报表会先显示上次结果并立即在后台刷新,自动同步也按可见间隔运行,因此两个较长的默认间隔只影响无人查看时的后台频率,用来减少 NAS 磁盘写入。

Compose 将容器内数据目录、监听地址和端口固定为 `/app/app_data`、`0.0.0.0` 和 `4318`;`METERLEAF_DATA_DIR`、`METERLEAF_HOST`、`METERLEAF_PORT`、`METERLEAF_DEMO` 仅用于直接以 Bun 启动时的本地配置。

自定义价格文件可放在挂载的 `prices` 目录,设置 `METERLEAF_PRICE_BOOK=/app/prices/custom.json`。修改后更新费率版本并重启,详见[计价说明](pricing.md)。
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "meterleaf",
"version": "0.6.3",
"version": "0.6.4",
"private": true,
"type": "module",
"license": "Apache-2.0",
Expand Down
4 changes: 2 additions & 2 deletions src/server/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,13 +45,13 @@ const envSchema = z.object({
.int()
.min(5_000)
.max(3_600_000)
.default(60_000),
.default(900_000),
METERLEAF_REPORT_REFRESH_INTERVAL_MS: z.coerce
.number()
.int()
.min(30_000)
.max(86_400_000)
.default(300_000),
.default(86_400_000),
METERLEAF_SOURCE_ID: z
.string()
.regex(/^[a-zA-Z0-9_-]+$/)
Expand Down
2 changes: 1 addition & 1 deletion src/server/sync.ts
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ class SyncPhaseError extends Error {

const defaultOptions: ResolvedSyncRunnerOptions = {
intervalMs: 15_000,
hiddenIntervalMs: 60_000,
hiddenIntervalMs: 900_000,
backlogIntervalMs: 1_000,
quotaIntervalMs: 30_000,
sweepMs: 6 * 3_600_000,
Expand Down
31 changes: 30 additions & 1 deletion src/server/view-service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ import {
import type { SyncStatus } from "./sync";

const CACHE_LIMIT = 16;
const DEFAULT_REFRESH_INTERVAL_MS = 300_000;
/** 定时刷新只是兜底;打开页面时读取会先返回缓存再后台刷新。 */
const DEFAULT_REFRESH_INTERVAL_MS = 86_400_000;

/** 报表结果的可见同步状态;账本结果本身仍保持 shared 契约不变。 */
export interface ReportStatus {
Expand Down Expand Up @@ -86,6 +87,27 @@ interface CacheEntry {
refreshing: boolean;
lastError: SafeErrorSummary | null;
sync?: SyncStatus;
/** 已排队或已落盘内容的摘要;未知时为 undefined,下次持久化必定写入。 */
persistedDigest?: string;
}

/**
* 持久化内容摘要。各层 asOf 只是计算时刻,sync 读取时由当前状态覆盖,均不计入摘要,
* 避免账本未变时的刷新反复改写整条缓存;lastUsed 同理只影响重启后的淘汰顺序。
*/
function persistedDigest(entry: {
query: ViewQuery;
basis: UsdBasis;
value: LedgerView;
}): string {
const { sync: _sync, reportStatus: _status, ...value } = entry.value;
return createHash("sha256")
.update(
JSON.stringify([entry.query, entry.basis, value], (key, item) =>
key === "asOf" ? undefined : item,
),
)
.digest("hex");
}

type ReportErrorCode =
Expand Down Expand Up @@ -333,6 +355,7 @@ export class ViewService {
...saved,
refreshing: false,
lastError: null,
persistedDigest: persistedDigest(saved),
});
}
cache.close();
Expand Down Expand Up @@ -433,9 +456,13 @@ export class ViewService {
);
}

/** 内容与已持久化版本相同则不写;worker 仍保持 save → saved 的逐条协议。 */
private persistCache(entry: CacheEntry) {
if (this.closed || this.cachePersistenceDisabled || !this.cacheWriter)
return;
const digest = persistedDigest(entry);
if (digest === entry.persistedDigest) return;
entry.persistedDigest = digest;
for (const key of this.pendingCacheWrites.keys()) {
if (!this.cache.has(key)) this.pendingCacheWrites.delete(key);
}
Expand All @@ -452,6 +479,8 @@ export class ViewService {
.sort((left, right) => left.lastUsed - right.lastUsed)[0];
if (!victim) break;
this.pendingCacheWrites.delete(victim.key);
const dropped = this.cache.get(victim.key);
if (dropped) dropped.persistedDigest = undefined;
}
this.scheduleCacheWrites();
}
Expand Down
9 changes: 5 additions & 4 deletions src/storage/ledger.ts
Original file line number Diff line number Diff line change
Expand Up @@ -281,10 +281,11 @@ export class LedgerStore {
.get(key);
return row ? (JSON.parse(row.value) as T) : null;
}
/** 值未变时不改写行,避免周期性同步状态在空闲时持续产生 WAL 写入。 */
setState(key: string, value: unknown) {
this.db
.query(
"INSERT INTO sync_state VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value=excluded.value",
"INSERT INTO sync_state VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value=excluded.value WHERE sync_state.value IS NOT excluded.value",
)
.run(key, JSON.stringify(value));
}
Expand Down Expand Up @@ -395,7 +396,7 @@ export class LedgerStore {
for (const account of accounts) {
const write = this.db
.query(
"INSERT INTO accounts VALUES (?, ?, ?) ON CONFLICT(source_id, external_id) DO UPDATE SET payload=excluded.payload WHERE accounts.payload<>excluded.payload",
"INSERT INTO accounts VALUES (?, ?, ?) ON CONFLICT(source_id, external_id) DO UPDATE SET payload=excluded.payload WHERE accounts.payload IS NOT excluded.payload",
)
.run(account.sourceId, account.externalId, JSON.stringify(account));
changed ||= write.changes > 0;
Expand All @@ -420,7 +421,7 @@ export class LedgerStore {
for (const account of accounts) {
const write = this.db
.query(
"INSERT INTO accounts VALUES (?, ?, ?) ON CONFLICT(source_id, external_id) DO UPDATE SET payload=excluded.payload WHERE accounts.payload<>excluded.payload",
"INSERT INTO accounts VALUES (?, ?, ?) ON CONFLICT(source_id, external_id) DO UPDATE SET payload=excluded.payload WHERE accounts.payload IS NOT excluded.payload",
)
.run(account.sourceId, account.externalId, JSON.stringify(account));
changed ||= write.changes > 0;
Expand Down Expand Up @@ -497,7 +498,7 @@ export class LedgerStore {
for (const account of batch.accounts) {
const write = this.db
.query(
"INSERT INTO accounts VALUES (?, ?, ?) ON CONFLICT(source_id, external_id) DO UPDATE SET payload=excluded.payload WHERE accounts.payload<>excluded.payload",
"INSERT INTO accounts VALUES (?, ?, ?) ON CONFLICT(source_id, external_id) DO UPDATE SET payload=excluded.payload WHERE accounts.payload IS NOT excluded.payload",
)
.run(sourceId, account.externalId, JSON.stringify(account));
changed ||= write.changes > 0;
Expand Down
68 changes: 68 additions & 0 deletions tests/ledger.test.ts
10BC8
Original file line number Diff line number Diff line change
Expand Up @@ -563,3 +563,71 @@ test("quota amounts and estimates keep priced spend when another request is unpr
expect(unknown.estimate.usd).toBeNull();
expect(unknown.periodCredits).not.toBeNull();
});

test("re-saving identical accounts, quotas and state leaves the ledger untouched; real changes still propagate", async () => {
const dir = await mkdtemp(join(tmpdir(), "meterleaf-ledger-noop-"));
const path = join(dir, "ledger.sqlite");
const store = new LedgerStore(path, book);
// 报表线程用独立只读连接判断账本是否变化,这里按同样方式观察。
const reader = new LedgerStore(path, book, { readonly: true });
try {
const accounts = [account("test", "a")];
const quotas = [quota(40, "2026-09-09T00:00:00Z").fact];
const collectedAt = "2026-09-09T00:01:00Z";
const replay = () => {
store.saveAccountsSnapshot("test", accounts);
store.saveAccounts(accounts);
store.saveQuotas(quotas, collectedAt);
store.saveIngestBatch(
"test",
{
batchId: "batch-1",
collector: { name: "test", version: "1" },
accounts,
usage: [],
quotas,
},
collectedAt,
);
store.savePage(
"test",
"incremental",
{ records: [fact()], nextCursor: "1", hasMore: false },
collectedAt,
);
};
replay();
const revision = store.revision();
const fingerprint = reader.reportFingerprint();
const totalChanges = () =>
store.db
.query<{ count: number }, []>("SELECT total_changes() AS count")
.get()!.count;

const stateChanges = totalChanges();
store.setState("test:incremental:cursor", "1");
store.setState("test:incremental:caughtUp", true);
expect(totalChanges()).toBe(stateChanges);

replay();
expect(store.revision()).toBe(revision);
expect(reader.reportFingerprint()).toBe(fingerprint);
expect(store.quotas()).toHaveLength(1);

const changedQuota = { ...quotas[0]!, percent: 55 };
store.saveQuotas([changedQuota], "2026-09-09T00:02:00Z");
expect(store.revision()).toBe(revision + 1);
const afterQuota = reader.reportFingerprint();
expect(afterQuota).not.toBe(fingerprint);
expect(reader.quotas().map((item) => item.fact.percent)).toEqual([40, 55]);

store.saveAccountsSnapshot("test", [{ ...accounts[0]!, name: "renamed" }]);
expect(store.revision()).toBe(revision + 2);
expect(reader.reportFingerprint()).not.toBe(afterQuota);
expect(reader.accounts()[0]!.name).toBe("renamed");
} finally {
reader.close();
store.close();
await rm(dir, { recursive: true, force: true });
}
});
2 changes: 1 addition & 1 deletion tests/server.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ test("live config is explicit and errors never repeat credentials", () => {
);
expect(readConfig({ METERLEAF_DEMO: "true" })).toMatchObject({
METERLEAF_PORT: 4318,
METERLEAF_REPORT_REFRESH_INTERVAL_MS: 300_000,
METERLEAF_REPORT_REFRESH_INTERVAL_MS: 86_400_000,
});
expect(
readConfig({
Expand Down
4 changes: 2 additions & 2 deletions tests/sync-presence.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ test("page leases expire independently and hiding one page preserves other viewe
expect(presence.hasVisible(135)).toBe(false);
});

test("sync intervals default to 15/60 seconds and accept bounded overrides", () => {
test("sync intervals default to 15 seconds visible / 15 minutes hidden and accept bounded overrides", () => {
const defaults = readConfig({ METERLEAF_DEMO: "true" });
expect(defaults.METERLEAF_SYNC_VISIBLE_INTERVAL_MS).toBe(15000);
expect(defaults.METERLEAF_SYNC_HIDDEN_INTERVAL_MS).toBe(60000);
expect(defaults.METERLEAF_SYNC_HIDDEN_INTERVAL_MS).toBe(900000);
expect(
readConfig({
METERLEAF_DEMO: "true",
Expand Down
Loading
Loading
0