91e5f44cc8
- Implemented MemoryStore class for managing concepts with YAML frontmatter. - Added methods for creating, reading, updating, and deleting concepts. - Introduced locking mechanism for concurrent access. - Developed search and hinting capabilities for concept retrieval. - Created tests for access control and store functionality. - Added long-term memory skill documentation and operational guidelines. - Included package.json and package-lock.json for dependency management.
91 lines
6.1 KiB
Markdown
91 lines
6.1 KiB
Markdown
# Kimiko long-term memory design
|
|
|
|
## Decision
|
|
|
|
Use OKF Markdown as the source of truth and a small Pi extension as the deterministic producer/consumer. Do not add a database, embeddings, autonomous summarization, or automatic Git commits in the first version.
|
|
|
|
```text
|
|
User request
|
|
-> memory_search (metadata-first lexical retrieval)
|
|
-> memory_read (selected bodies)
|
|
-> model chooses semantic content
|
|
-> memory_write / memory_forget
|
|
-> deterministic storage, validation, indexing, and logging
|
|
```
|
|
|
|
This keeps memory inspectable, portable, diffable, and useful without a service.
|
|
|
|
## Responsibility boundary
|
|
|
|
### Model
|
|
|
|
- Decides whether information is durable and relevant.
|
|
- Chooses atomic concepts, stable IDs, types, summaries, tags, Markdown bodies, links, and citations.
|
|
- Resolves semantic contradiction or ambiguity with the user.
|
|
|
|
### Extension
|
|
|
|
- Enforces safe bundle-relative lowercase IDs and reserved filenames.
|
|
- Parses YAML, preserves unknown metadata on valid updates, and requires the local producer profile's `type` and `timestamp`.
|
|
- Generates timestamps from the system clock only after meaningful changes; no-op writes preserve timestamps.
|
|
- Maintains marked generated sections in indexes while retaining human text outside them.
|
|
- Maintains the root log.
|
|
- Serializes writes under a filesystem lock, writes through an atomic rename, reads back and semantically verifies output, then retries once.
|
|
- Repairs missing metadata from deterministic signals. Missing timestamps use filesystem mtime; missing types use `Recovered Concept`.
|
|
- Backs up malformed documents under `.recovery/`, preserves recoverable bodies, and marks repaired concepts for review.
|
|
|
|
The model never writes frontmatter, timestamps, indexes, or logs and is not asked to validate its own formatting.
|
|
|
|
## Retrieval and context budget
|
|
|
|
Each user turn resets access state and removes detailed memory tools. Explicit memory language reveals those tools for that turn so weaker models know the capability exists; otherwise only `memory_search` remains and dynamically enables `memory_read`, `memory_write`, and `memory_forget`. Visibility is not authorization: existing concepts must be returned by search and read in the same turn before update or deletion, and new concepts require a current-turn duplicate search. Search returns metadata rather than bodies; full reads are limited to ten concepts and 45,000 characters.
|
|
|
|
Before the model runs, deterministic lexical retrieval may add at most two conservative metadata candidates to that turn's system prompt. Hints require distinctive metadata matches, contain no bodies, do not authorize mutation, are not session messages, and disappear when the next turn rebuilds the system prompt.
|
|
|
|
Retrieval is weighted deterministic lexical matching over path, title, description, tags, type, and body, with stop-word filtering. This is intentionally simple and adequate for a personal bundle containing hundreds or a few thousand concepts. The root and directory indexes remain useful to humans even though search scans files directly.
|
|
|
|
The always-loaded instruction is one line in `SYSTEM.md`. Behavioral detail lives in the on-demand `long-term-memory` skill, and operational recovery detail is another level down in its reference document.
|
|
|
|
## Failure behavior
|
|
|
|
1. A mutation obtains a bundle-wide lock. Stale locks older than 30 seconds are removed.
|
|
2. The extension writes a same-directory temporary file, flushes it, and atomically renames it.
|
|
3. It reads the file back, parses it, checks required metadata, and compares semantic fingerprints.
|
|
4. Failed verification repeats the complete atomic write once from the structured input.
|
|
5. A second failure is reported as a tool error rather than delegated to the model.
|
|
6. Startup runs deterministic repair and index regeneration. `/memory-check` supports explicit validation; `--repair` applies the same repair path.
|
|
|
|
An interrupted operation can leave a concept updated before its index or log. Startup repair regenerates indexes. The root log is human-oriented, not a transactional journal; use an explicit external backup policy when durability beyond the local filesystem is required.
|
|
|
|
## Safety and privacy
|
|
|
|
- The skill prohibits credentials, tokens, transient chat, and unsupported inference.
|
|
- Forget requires the ID to be repeated exactly, deletes the current file and matching recovery backups, redacts its links from the root log, and records only an anonymous deletion event.
|
|
- No process auto-commits because Git history may retain data a user intended to forget.
|
|
- The bundle inherits host filesystem permissions; OKF itself is not an access-control system.
|
|
- `memory/` is ignored by Git by default to reduce accidental disclosure. Version it only after selecting a private remote, retention, and deletion policy.
|
|
- Repaired malformed source is retained in `.recovery/`; operators must remove those backups when handling a privacy deletion.
|
|
|
|
## Deliberately deferred
|
|
|
|
Add these only when observed scale or quality requires them:
|
|
|
|
1. **SQLite FTS5 cache** keyed by path and content hash for larger bundles. Markdown remains authoritative; the cache is disposable and rebuilt deterministically.
|
|
2. **Embeddings/reranking** only after lexical retrieval has measured misses. Store vectors outside OKF and version the embedding model.
|
|
3. **Contradiction workflow** with explicit supersession links and human confirmation; do not silently merge claims.
|
|
4. **Source freshness policy** using producer-specific `valid_until`, authority, or confidence metadata.
|
|
5. **Cross-process transaction journal** if concurrent writers or strict log durability become real requirements.
|
|
6. **Git automation** only with an explicit privacy, signing, remote, and retention policy.
|
|
|
|
## Production thresholds
|
|
|
|
Revisit the design when any of these occurs:
|
|
|
|
- Search latency is consistently above 100 ms.
|
|
- The bundle exceeds roughly 5,000 concepts or 50 MB.
|
|
- Multiple processes write concurrently.
|
|
- Memory has regulatory retention/deletion requirements.
|
|
- Retrieval quality is measured and lexical search misses important concepts.
|
|
|
|
Until then, files plus deterministic tooling are the smallest production-aware architecture.
|