feat: context engine plugins, scaffolding, and pricing fixes (#6)
* feat: add 4 context engine plugins
- topic-memory: keyword clustering for topic-aware memory recall
- episodic-memory: conversation segmentation and cross-session recall
- user-profile: persistent user profiling from conversation patterns
- context-decay: time-based memory decay with reinforcement dynamics
All plugins use the ingest/after_turn hook protocol with stdin/stdout JSON.
* chore: add plugin scaffolding, update docs and templates
- Add plugin.toml template with {{NAME}} placeholder
- Add new-plugin Makefile target with hooks/ scaffolding
- Update plugins/README.md with all 10 plugins
- Update README.md stats (10 plugins, 220+ models)
- Add Plugin checkbox and checklist to PR template
- Add Plugin to issue template content type dropdown
- Fix CONTRIBUTING.md: last_verified is recommended, not required
* fix: correct model pricing and remove deprecated entries
- openrouter/gemma-2-9b-it: fix pricing from 0.0 to 0.03/0.09 per M tokens
(free variant correctly stays at 0.0)
- github-copilot: remove deprecated copilot/gpt-4 model entry
(GPT-4 retired in favor of GPT-4o for Copilot)
* docs: annotate kimi-coding as membership-gated
Kimi Code CLI uses quota-based membership model (not per-token billing).
Free tier has limited weekly requests; underlying model is K2.5.
Pricing kept at 0.0 consistent with other subscription providers
(chatgpt, github-copilot) but with explanatory comments.
* style: fix trailing newline in github-copilot.toml
* fix: correct Moonshot/Kimi model pricing from official sources
All 5 models had incorrect pricing:
- moonshot-v1-8k: 0.10/0.10 → 0.20/2.00
- moonshot-v1-32k: 0.30/0.30 → 1.00/3.00
- moonshot-v1-128k: 0.80/0.80 → 2.00/5.00
- kimi-k2: 2.00/8.00 → 0.60/2.50
- kimi-k2.5: 2.00/8.00 → 0.45/2.20
Sources: platform.moonshot.ai/docs/pricing/chat, costgoat.com, getmaxim.ai
* feat: add MiniMax M2.7 and M2.7-highspeed models
Released 2026-03-18, MiniMax's latest flagship text model.
10B activated params, 200K context, 128K output, tool use, streaming.
Pricing: $0.30/$1.20 per M tokens (input/output).
Added to both international (minimax.io) and China (minimaxi.com) providers.
This commit is contained in:
33 files changed
+2085
-36
No files matched your search
+6
-2
@@ -6,7 +6,7 @@ Plugin packages for LibreFang. Plugins extend agent behavior through lifecycle h
|
||||
|
||||
```
|
||||
plugins/
|
||||
└── echo-memory/
|
||||
└── <plugin-name>/
|
||||
├── plugin.toml # Plugin manifest
|
||||
├── hooks/
|
||||
│ ├── ingest.py # Called on user message
|
||||
@@ -45,16 +45,20 @@ stdin: {"type": "after_turn", "agent_id": "...", "messages": [...]}
|
||||
stdout: {"type": "ok"}
|
||||
```
|
||||
|
||||
## Current Plugins (6)
|
||||
## Current Plugins (10)
|
||||
|
||||
| Plugin | Hooks | Description |
|
||||
|--------|-------|-------------|
|
||||
| auto-summarizer | ingest, after_turn | Running conversation summary for long context compression |
|
||||
| context-decay | ingest, after_turn | Time-based memory decay with relevance scoring for natural forgetting |
|
||||
| conversation-logger | after_turn | Logs conversations to JSONL files for auditing and analytics |
|
||||
| episodic-memory | ingest, after_turn | Episode-based conversation segmentation and cross-session recall |
|
||||
| guardrails | ingest | Safety filter detecting PII, prompt injection, and credential exposure |
|
||||
| keyword-memory | ingest | Extracts keywords and named entities as contextual memories |
|
||||
| sentiment-tracker | ingest | Analyzes user sentiment and injects emotional context |
|
||||
| todo-tracker | ingest, after_turn | Detects, persists, and recalls action items from conversations |
|
||||
| topic-memory | ingest, after_turn | Topic-aware keyword clustering with cross-conversation context recall |
|
||||
| user-profile | ingest, after_turn | Persistent user profiling from conversation patterns for personalization |
|
||||
|
||||
## Adding a New Plugin
|
||||
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
# context-decay
|
||||
|
||||
Time-based memory decay with relevance scoring. Memories lose confidence over time (5% per day) and are only recalled when they pass both a decay threshold and a relevance check against the current message. Implements "use it or lose it" -- recalled memories get their access timestamps refreshed.
|
||||
|
||||
## How it works
|
||||
|
||||
**After each turn**, the plugin extracts memorable statements from the conversation:
|
||||
|
||||
- User preferences ("I prefer...", "I use...")
|
||||
- Decisions ("let's use...", "we decided...")
|
||||
- Important facts (names, versions, URLs)
|
||||
- Corrections ("no, actually...", "that's wrong...")
|
||||
|
||||
Similar memories are reinforced (confidence +0.1, cap 1.0). All memories receive a decay pass, and those below 0.1 confidence are pruned.
|
||||
|
||||
**On ingest**, each memory's confidence is decayed based on time elapsed, then scored for relevance to the current message via keyword overlap. The composite score `decayed_confidence * (0.5 + 0.5 * relevance)` must exceed 0.3 to be recalled. Recalled memories get their `last_accessed` timestamp updated.
|
||||
|
||||
## Hooks
|
||||
|
||||
| Hook | Script | Description |
|
||||
|------|--------|-------------|
|
||||
| ingest | `hooks/ingest.py` | Applies decay, scores relevance, returns top 5 memories above threshold |
|
||||
| after_turn | `hooks/after_turn.py` | Extracts new memories, reinforces similar ones, prunes decayed entries |
|
||||
|
||||
## Storage
|
||||
|
||||
Memories are stored at `~/.librefang/plugins/context-decay/{agent_id}.json`. Max 100 memories per agent. Decay formula: `confidence * 0.95^(hours / 24)`.
|
||||
|
||||
## Usage
|
||||
|
||||
Installed automatically when enabled in agent configuration.
|
||||
@@ -0,0 +1,370 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Context-decay after_turn hook.
|
||||
|
||||
Extracts memorable statements from the conversation turn, stores new memories
|
||||
or reinforces existing ones, applies time-based decay to all memories, and
|
||||
prunes dead memories.
|
||||
|
||||
Receives via stdin:
|
||||
{"type": "after_turn", "agent_id": "...", "messages": [
|
||||
{"role": "user"|"assistant", "content": "..."}
|
||||
]}
|
||||
|
||||
Prints to stdout:
|
||||
{"type": "ok"}
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Decay: 5% confidence loss per day
|
||||
DECAY_FACTOR = 0.95
|
||||
|
||||
# Confidence below this gets pruned
|
||||
PRUNE_THRESHOLD = 0.1
|
||||
|
||||
# Maximum memories per agent
|
||||
MAX_MEMORIES = 100
|
||||
|
||||
# Initial confidence for new memories
|
||||
INITIAL_CONFIDENCE = 0.8
|
||||
|
||||
# Confidence boost when reinforcing an existing memory
|
||||
REINFORCE_BOOST = 0.1
|
||||
|
||||
# Minimum keyword overlap to consider memories similar
|
||||
SIMILARITY_THRESHOLD = 0.5
|
||||
|
||||
# Maximum content length for a single memory
|
||||
MAX_CONTENT_LEN = 200
|
||||
|
||||
# Minimum keyword length
|
||||
MIN_WORD_LEN = 3
|
||||
|
||||
STOPWORDS = frozenset({
|
||||
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
|
||||
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
|
||||
"being", "have", "has", "had", "do", "does", "did", "will", "would",
|
||||
"could", "should", "may", "might", "shall", "can", "need", "must",
|
||||
"it", "its", "i", "me", "my", "you", "your", "he", "she", "we",
|
||||
"they", "them", "their", "this", "that", "these", "those", "what",
|
||||
"which", "who", "how", "when", "where", "why", "if", "then", "so",
|
||||
"not", "no", "just", "also", "very", "too", "about", "up", "out",
|
||||
"all", "some", "any", "each", "every", "into", "over", "after",
|
||||
})
|
||||
|
||||
# Patterns that indicate a memorable statement
|
||||
PREFERENCE_PATTERNS = [
|
||||
re.compile(r"\bi\s+prefer\b", re.IGNORECASE),
|
||||
re.compile(r"\bi\s+like\b", re.IGNORECASE),
|
||||
re.compile(r"\bi\s+use\b", re.IGNORECASE),
|
||||
re.compile(r"\bi\s+want\b", re.IGNORECASE),
|
||||
re.compile(r"\bi\s+need\b", re.IGNORECASE),
|
||||
re.compile(r"\bi\s+always\b", re.IGNORECASE),
|
||||
]
|
||||
|
||||
DECISION_PATTERNS = [
|
||||
re.compile(r"\blet'?s?\s+use\b", re.IGNORECASE),
|
||||
re.compile(r"\bwe\s+decided\b", re.IGNORECASE),
|
||||
re.compile(r"\bgoing\s+with\b", re.IGNORECASE),
|
||||
re.compile(r"\bwe\s+should\s+use\b", re.IGNORECASE),
|
||||
re.compile(r"\bi'?ll\s+go\s+with\b", re.IGNORECASE),
|
||||
]
|
||||
|
||||
CORRECTION_PATTERNS = [
|
||||
re.compile(r"\bno,?\s+actually\b", re.IGNORECASE),
|
||||
re.compile(r"\bthat'?s?\s+wrong\b", re.IGNORECASE),
|
||||
re.compile(r"\bi\s+meant\b", re.IGNORECASE),
|
||||
re.compile(r"\bactually,?\s+i\b", re.IGNORECASE),
|
||||
re.compile(r"\bnot\s+that,?\s+", re.IGNORECASE),
|
||||
]
|
||||
|
||||
# Pattern for specific facts: contains version numbers, URLs, or proper nouns
|
||||
FACT_PATTERNS = [
|
||||
re.compile(r"\bv?\d+\.\d+(?:\.\d+)?\b"), # version numbers
|
||||
re.compile(r"https?://[^\s]+"), # URLs
|
||||
re.compile(r"\b[A-Z][a-z]+(?:\s+[A-Z][a-z]+)+\b"), # proper noun phrases
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def extract_keywords(text):
|
||||
"""Extract deduplicated lowercase keywords from text."""
|
||||
words = re.findall(r"[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9]|[a-zA-Z]", text)
|
||||
seen = set()
|
||||
keywords = []
|
||||
for w in words:
|
||||
lower = w.lower()
|
||||
if lower not in STOPWORDS and len(lower) >= MIN_WORD_LEN and lower not in seen:
|
||||
seen.add(lower)
|
||||
keywords.append(lower)
|
||||
return keywords
|
||||
|
||||
|
||||
def store_dir():
|
||||
"""Return the storage directory path for context-decay."""
|
||||
return os.path.join(
|
||||
os.path.expanduser("~"), ".librefang", "plugins", "context-decay"
|
||||
)
|
||||
|
||||
|
||||
def store_path(agent_id):
|
||||
"""Return the JSON store file path for a given agent."""
|
||||
return os.path.join(store_dir(), f"{agent_id}.json")
|
||||
|
||||
|
||||
def load_store(agent_id):
|
||||
"""Load the memory store for an agent. Returns default on failure."""
|
||||
path = store_path(agent_id)
|
||||
if not os.path.isfile(path):
|
||||
return {"memories": []}
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
if not isinstance(data, dict) or "memories" not in data:
|
||||
return {"memories": []}
|
||||
return data
|
||||
except (json.JSONDecodeError, OSError):
|
||||
return {"memories": []}
|
||||
|
||||
|
||||
def save_store(agent_id, data):
|
||||
"""Persist the memory store for an agent."""
|
||||
dirpath = store_dir()
|
||||
os.makedirs(dirpath, exist_ok=True)
|
||||
path = store_path(agent_id)
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
|
||||
|
||||
def now_iso():
|
||||
"""Return the current UTC time as an ISO 8601 string."""
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def parse_iso(ts):
|
||||
"""Parse an ISO 8601 timestamp string to a datetime object."""
|
||||
if not ts:
|
||||
return None
|
||||
try:
|
||||
cleaned = ts.replace("Z", "+00:00")
|
||||
return datetime.fromisoformat(cleaned)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def hours_since(ts_str, now):
|
||||
"""Calculate hours elapsed between a timestamp string and now."""
|
||||
dt = parse_iso(ts_str)
|
||||
if dt is None:
|
||||
return 0.0
|
||||
delta = now - dt
|
||||
return max(delta.total_seconds() / 3600.0, 0.0)
|
||||
|
||||
|
||||
def apply_decay(confidence, hours_elapsed):
|
||||
"""Apply exponential decay: confidence * 0.95^(hours / 24)."""
|
||||
if hours_elapsed <= 0:
|
||||
return confidence
|
||||
return confidence * (DECAY_FACTOR ** (hours_elapsed / 24.0))
|
||||
|
||||
|
||||
def keyword_overlap(keywords_a, keywords_b):
|
||||
"""Compute Jaccard similarity between two keyword lists."""
|
||||
if not keywords_a or not keywords_b:
|
||||
return 0.0
|
||||
set_a = set(keywords_a)
|
||||
set_b = set(keywords_b)
|
||||
intersection = set_a & set_b
|
||||
union = set_a | set_b
|
||||
if not union:
|
||||
return 0.0
|
||||
return len(intersection) / len(union)
|
||||
|
||||
|
||||
def next_memory_id(memories):
|
||||
"""Generate the next incrementing memory ID in m_XXX format."""
|
||||
max_num = 0
|
||||
for mem in memories:
|
||||
mid = mem.get("id", "")
|
||||
if mid.startswith("m_"):
|
||||
try:
|
||||
num = int(mid[2:])
|
||||
if num > max_num:
|
||||
max_num = num
|
||||
except ValueError:
|
||||
pass
|
||||
return f"m_{max_num + 1:03d}"
|
||||
|
||||
|
||||
def truncate(text, max_len):
|
||||
"""Truncate text to max_len, appending ... if trimmed."""
|
||||
if len(text) <= max_len:
|
||||
return text
|
||||
return text[: max_len - 3].rstrip() + "..."
|
||||
|
||||
|
||||
def matches_any(text, patterns):
|
||||
"""Return True if text matches any of the compiled regex patterns."""
|
||||
for pat in patterns:
|
||||
if pat.search(text):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def extract_sentences(text):
|
||||
"""Split text into sentences on common boundaries."""
|
||||
# Split on period, exclamation, question mark followed by space or end
|
||||
parts = re.split(r"(?<=[.!?])\s+", text.strip())
|
||||
return [s.strip() for s in parts if s.strip()]
|
||||
|
||||
|
||||
def extract_memorable_statements(messages):
|
||||
"""Extract statements worth remembering from conversation messages.
|
||||
|
||||
Focuses on user messages containing preferences, decisions, corrections,
|
||||
or specific facts.
|
||||
"""
|
||||
statements = []
|
||||
for msg in messages:
|
||||
if msg.get("role") != "user":
|
||||
continue
|
||||
content = msg.get("content", "")
|
||||
if not content.strip():
|
||||
continue
|
||||
|
||||
sentences = extract_sentences(content)
|
||||
for sentence in sentences:
|
||||
# Check if this sentence matches any memorable pattern
|
||||
is_memorable = (
|
||||
matches_any(sentence, PREFERENCE_PATTERNS)
|
||||
or matches_any(sentence, DECISION_PATTERNS)
|
||||
or matches_any(sentence, CORRECTION_PATTERNS)
|
||||
or matches_any(sentence, FACT_PATTERNS)
|
||||
)
|
||||
if is_memorable:
|
||||
trimmed = truncate(sentence.strip(), MAX_CONTENT_LEN)
|
||||
keywords = extract_keywords(trimmed)
|
||||
if keywords:
|
||||
statements.append({
|
||||
"content": trimmed,
|
||||
"keywords": keywords,
|
||||
})
|
||||
|
||||
return statements
|
||||
|
||||
|
||||
def find_similar_memory(memories, keywords):
|
||||
"""Find an existing memory with keyword overlap above the similarity threshold.
|
||||
|
||||
Returns the index of the best match, or -1 if none found.
|
||||
"""
|
||||
best_idx = -1
|
||||
best_overlap = 0.0
|
||||
for i, mem in enumerate(memories):
|
||||
overlap = keyword_overlap(mem.get("keywords", []), keywords)
|
||||
if overlap > SIMILARITY_THRESHOLD and overlap > best_overlap:
|
||||
best_overlap = overlap
|
||||
best_idx = i
|
||||
return best_idx
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main():
|
||||
try:
|
||||
request = json.loads(sys.stdin.read())
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
print(json.dumps({"type": "ok"}))
|
||||
return
|
||||
|
||||
agent_id = request.get("agent_id", "")
|
||||
messages = request.get("messages", [])
|
||||
|
||||
if not agent_id or not messages:
|
||||
print(json.dumps({"type": "ok"}))
|
||||
return
|
||||
|
||||
data = load_store(agent_id)
|
||||
memories = data.get("memories", [])
|
||||
now = datetime.now(timezone.utc)
|
||||
now_str = now_iso()
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# Step 1: Extract memorable statements from the conversation turn
|
||||
# -----------------------------------------------------------------------
|
||||
statements = extract_memorable_statements(messages)
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# Step 2: Store new memories or reinforce existing ones
|
||||
# -----------------------------------------------------------------------
|
||||
for stmt in statements:
|
||||
similar_idx = find_similar_memory(memories, stmt["keywords"])
|
||||
|
||||
if similar_idx >= 0:
|
||||
# Reinforce existing memory
|
||||
existing = memories[similar_idx]
|
||||
existing["confidence"] = min(
|
||||
existing.get("confidence", 0.0) + REINFORCE_BOOST, 1.0
|
||||
)
|
||||
existing["content"] = stmt["content"]
|
||||
existing["keywords"] = stmt["keywords"]
|
||||
existing["last_accessed"] = now_str
|
||||
existing["access_count"] = existing.get("access_count", 0) + 1
|
||||
else:
|
||||
# Add new memory
|
||||
new_mem = {
|
||||
"id": next_memory_id(memories),
|
||||
"content": stmt["content"],
|
||||
"keywords": stmt["keywords"],
|
||||
"confidence": INITIAL_CONFIDENCE,
|
||||
"created": now_str,
|
||||
"last_accessed": now_str,
|
||||
"access_count": 0,
|
||||
}
|
||||
memories.append(new_mem)
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# Step 3: Apply decay pass to ALL memories
|
||||
# -----------------------------------------------------------------------
|
||||
for mem in memories:
|
||||
elapsed = hours_since(mem.get("last_accessed", mem.get("created", "")), now)
|
||||
mem["confidence"] = apply_decay(mem.get("confidence", 0.0), elapsed)
|
||||
# Update last_accessed to now so next decay is relative to this pass
|
||||
# (decay is applied on each hook invocation, not accumulated)
|
||||
mem["last_accessed"] = now_str
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# Step 4: Prune memories below the prune threshold
|
||||
# -----------------------------------------------------------------------
|
||||
memories = [m for m in memories if m.get("confidence", 0.0) >= PRUNE_THRESHOLD]
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# Step 5: Evict lowest-confidence memories if over capacity
|
||||
# -----------------------------------------------------------------------
|
||||
if len(memories) > MAX_MEMORIES:
|
||||
memories.sort(key=lambda m: m.get("confidence", 0.0), reverse=True)
|
||||
memories = memories[:MAX_MEMORIES]
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# Step 6: Save
|
||||
# -----------------------------------------------------------------------
|
||||
data["memories"] = memories
|
||||
save_store(agent_id, data)
|
||||
|
||||
print(json.dumps({"type": "ok"}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,227 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Context-decay ingest hook.
|
||||
|
||||
Loads the memory store for the current agent, applies time-based decay to
|
||||
all memories, scores them for relevance to the incoming message, and returns
|
||||
the top matches above the recall threshold.
|
||||
|
||||
This hook updates last_accessed timestamps for recalled memories to implement
|
||||
"use it or lose it" dynamics -- the one exception where ingest modifies storage.
|
||||
|
||||
Receives via stdin:
|
||||
{"type": "ingest", "agent_id": "...", "message": "user message text"}
|
||||
|
||||
Prints to stdout:
|
||||
{"type": "ingest_result", "memories": [{"content": "..."}]}
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Decay: 5% confidence loss per day
|
||||
DECAY_FACTOR = 0.95
|
||||
|
||||
# Minimum final_score to recall a memory
|
||||
RECALL_THRESHOLD = 0.3
|
||||
|
||||
# Maximum memories to return per ingest
|
||||
MAX_RECALL = 5
|
||||
|
||||
# Minimum keyword length
|
||||
MIN_WORD_LEN = 3
|
||||
|
||||
STOPWORDS = frozenset({
|
||||
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
|
||||
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
|
||||
"being", "have", "has", "had", "do", "does", "did", "will", "would",
|
||||
"could", "should", "may", "might", "shall", "can", "need", "must",
|
||||
"it", "its", "i", "me", "my", "you", "your", "he", "she", "we",
|
||||
"they", "them", "their", "this", "that", "these", "those", "what",
|
||||
"which", "who", "how", "when", "where", "why", "if", "then", "so",
|
||||
"not", "no", "just", "also", "very", "too", "about", "up", "out",
|
||||
"all", "some", "any", "each", "every", "into", "over", "after",
|
||||
})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def extract_keywords(text):
|
||||
"""Extract deduplicated lowercase keywords from text."""
|
||||
words = re.findall(r"[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9]|[a-zA-Z]", text)
|
||||
seen = set()
|
||||
keywords = []
|
||||
for w in words:
|
||||
lower = w.lower()
|
||||
if lower not in STOPWORDS and len(lower) >= MIN_WORD_LEN and lower not in seen:
|
||||
seen.add(lower)
|
||||
keywords.append(lower)
|
||||
return keywords
|
||||
|
||||
|
||||
def store_dir():
|
||||
"""Return the storage directory path for context-decay."""
|
||||
return os.path.join(
|
||||
os.path.expanduser("~"), ".librefang", "plugins", "context-decay"
|
||||
)
|
||||
|
||||
|
||||
def store_path(agent_id):
|
||||
"""Return the JSON store file path for a given agent."""
|
||||
return os.path.join(store_dir(), f"{agent_id}.json")
|
||||
|
||||
|
||||
def load_store(agent_id):
|
||||
"""Load the memory store for an agent. Returns default on failure."""
|
||||
path = store_path(agent_id)
|
||||
if not os.path.isfile(path):
|
||||
return {"memories": []}
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
if not isinstance(data, dict) or "memories" not in data:
|
||||
return {"memories": []}
|
||||
return data
|
||||
except (json.JSONDecodeError, OSError):
|
||||
return {"memories": []}
|
||||
|
||||
|
||||
def save_store(agent_id, data):
|
||||
"""Persist the memory store for an agent."""
|
||||
dirpath = store_dir()
|
||||
os.makedirs(dirpath, exist_ok=True)
|
||||
path = store_path(agent_id)
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
|
||||
|
||||
def now_iso():
|
||||
"""Return the current UTC time as an ISO 8601 string."""
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def parse_iso(ts):
|
||||
"""Parse an ISO 8601 timestamp string to a datetime object.
|
||||
|
||||
Handles both +00:00 and Z suffixes. Returns None on failure.
|
||||
"""
|
||||
if not ts:
|
||||
return None
|
||||
try:
|
||||
# Replace Z suffix for compatibility with fromisoformat on older Python
|
||||
cleaned = ts.replace("Z", "+00:00")
|
||||
return datetime.fromisoformat(cleaned)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def hours_since(ts_str, now):
|
||||
"""Calculate hours elapsed between a timestamp string and now."""
|
||||
dt = parse_iso(ts_str)
|
||||
if dt is None:
|
||||
return 0.0
|
||||
delta = now - dt
|
||||
return max(delta.total_seconds() / 3600.0, 0.0)
|
||||
|
||||
|
||||
def apply_decay(confidence, hours_elapsed):
|
||||
"""Apply exponential decay: confidence * 0.95^(hours / 24)."""
|
||||
if hours_elapsed <= 0:
|
||||
return confidence
|
||||
return confidence * (DECAY_FACTOR ** (hours_elapsed / 24.0))
|
||||
|
||||
|
||||
def keyword_overlap(keywords_a, keywords_b):
|
||||
"""Compute Jaccard similarity between two keyword lists."""
|
||||
if not keywords_a or not keywords_b:
|
||||
return 0.0
|
||||
set_a = set(keywords_a)
|
||||
set_b = set(keywords_b)
|
||||
intersection = set_a & set_b
|
||||
union = set_a | set_b
|
||||
if not union:
|
||||
return 0.0
|
||||
return len(intersection) / len(union)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main():
|
||||
try:
|
||||
request = json.loads(sys.stdin.read())
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||
return
|
||||
|
||||
message = request.get("message", "")
|
||||
agent_id = request.get("agent_id", "")
|
||||
|
||||
if not message.strip() or not agent_id:
|
||||
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||
return
|
||||
|
||||
query_keywords = extract_keywords(message)
|
||||
if not query_keywords:
|
||||
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||
return
|
||||
|
||||
data = load_store(agent_id)
|
||||
memories = data.get("memories", [])
|
||||
|
||||
if not memories:
|
||||
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||
return
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
scored = []
|
||||
store_modified = False
|
||||
|
||||
for mem in memories:
|
||||
# Apply time-based decay
|
||||
elapsed = hours_since(mem.get("last_accessed", mem.get("created", "")), now)
|
||||
decayed = apply_decay(mem.get("confidence", 0.0), elapsed)
|
||||
|
||||
# Score relevance via keyword overlap
|
||||
relevance = keyword_overlap(mem.get("keywords", []), query_keywords)
|
||||
|
||||
# Composite score: decayed confidence weighted with relevance
|
||||
final_score = decayed * (0.5 + 0.5 * relevance)
|
||||
|
||||
if final_score > RECALL_THRESHOLD:
|
||||
scored.append((final_score, decayed, mem))
|
||||
|
||||
# Sort by final_score descending, take top MAX_RECALL
|
||||
scored.sort(key=lambda x: x[0], reverse=True)
|
||||
top = scored[:MAX_RECALL]
|
||||
|
||||
result_memories = []
|
||||
for final_score, decayed, mem in top:
|
||||
confidence_pct = int(round(decayed * 100))
|
||||
content = mem.get("content", "")
|
||||
result_memories.append({
|
||||
"content": f"[context-decay] Recalled ({confidence_pct}%): {content}"
|
||||
})
|
||||
|
||||
# Update last_accessed and save back -- "use it or lose it"
|
||||
mem["last_accessed"] = now_iso()
|
||||
mem["access_count"] = mem.get("access_count", 0) + 1
|
||||
store_modified = True
|
||||
|
||||
# Persist access timestamp updates
|
||||
if store_modified:
|
||||
save_store(agent_id, data)
|
||||
|
||||
print(json.dumps({"type": "ingest_result", "memories": result_memories}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,8 @@
|
||||
name = "context-decay"
|
||||
version = "0.1.0"
|
||||
description = "Time-based memory decay with relevance scoring for natural context forgetting"
|
||||
author = "librefang"
|
||||
|
||||
[hooks]
|
||||
ingest = "hooks/ingest.py"
|
||||
after_turn = "hooks/after_turn.py"
|
||||
Whitespace-only changes.
@@ -0,0 +1,26 @@
|
||||
# episodic-memory
|
||||
|
||||
Episode-based conversation segmentation and cross-session recall. Automatically detects topic shifts to split conversations into discrete episodes, then recalls relevant past episodes when similar topics arise.
|
||||
|
||||
## How it works
|
||||
|
||||
**After each turn**, the plugin tracks a "current episode" with accumulated keywords. When the Jaccard similarity between the current turn's keywords and the running episode keywords drops below 0.1 (and the episode has at least 4 messages), it marks the episode as completed with a summary and starts a new one.
|
||||
|
||||
**On ingest**, the plugin scores all completed episodes against the incoming message keywords and returns the top 2 matches (overlap > 0.2) as contextual memories including the episode timestamp and summary.
|
||||
|
||||
This gives agents episodic recall -- "last time we discussed Docker deployment, we configured nginx as a reverse proxy."
|
||||
|
||||
## Hooks
|
||||
|
||||
| Hook | Script | Description |
|
||||
|------|--------|-------------|
|
||||
| ingest | `hooks/ingest.py` | Scores completed episodes against message keywords, returns top matches |
|
||||
| after_turn | `hooks/after_turn.py` | Tracks current episode, detects topic shifts, segments conversations |
|
||||
|
||||
## Storage
|
||||
|
||||
Episodes are stored at `~/.librefang/plugins/episodic-memory/{agent_id}.json`. Max 30 completed episodes per agent (oldest evicted when full).
|
||||
|
||||
## Usage
|
||||
|
||||
Installed automatically when enabled in agent configuration.
|
||||
@@ -0,0 +1,305 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Episodic memory after_turn hook.
|
||||
|
||||
Maintains the episode store by tracking topic continuity across turns.
|
||||
Detects topic shifts via Jaccard similarity between the current turn's
|
||||
keywords and the running episode's keywords. When a shift is detected
|
||||
(and the current episode has enough messages), the episode is completed
|
||||
and a new one starts.
|
||||
|
||||
This hook is write-only -- it never returns memories.
|
||||
|
||||
Receives via stdin:
|
||||
{"type": "after_turn", "agent_id": "...", "messages": [...]}
|
||||
|
||||
Prints to stdout:
|
||||
{"type": "ok"}
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Stopwords & keyword extraction (identical logic to ingest.py)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
STOPWORDS = frozenset({
|
||||
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
|
||||
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
|
||||
"being", "have", "has", "had", "do", "does", "did", "will", "would",
|
||||
"could", "should", "may", "might", "shall", "can", "need", "must",
|
||||
"it", "its", "i", "me", "my", "you", "your", "he", "she", "we",
|
||||
"they", "them", "their", "this", "that", "these", "those", "what",
|
||||
"which", "who", "how", "when", "where", "why", "if", "then", "so",
|
||||
"not", "no", "just", "also", "very", "too", "about", "up", "out",
|
||||
"all", "some", "any", "each", "every", "into", "over", "after",
|
||||
})
|
||||
|
||||
MIN_WORD_LEN = 3
|
||||
|
||||
# Topic shift detection threshold
|
||||
TOPIC_SHIFT_THRESHOLD = 0.1
|
||||
|
||||
# Minimum messages before allowing topic shift completion
|
||||
MIN_MESSAGES_FOR_COMPLETION = 4
|
||||
|
||||
# Maximum completed episodes to retain
|
||||
MAX_EPISODES = 30
|
||||
|
||||
# Maximum summary length in characters
|
||||
MAX_SUMMARY_LEN = 150
|
||||
|
||||
|
||||
def extract_keywords(text):
|
||||
"""Extract deduplicated keywords from text."""
|
||||
words = re.findall(r"[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9]|[a-zA-Z]", text)
|
||||
seen = set()
|
||||
keywords = []
|
||||
for w in words:
|
||||
lower = w.lower()
|
||||
if lower not in STOPWORDS and len(lower) >= MIN_WORD_LEN and lower not in seen:
|
||||
seen.add(lower)
|
||||
keywords.append(lower)
|
||||
return keywords
|
||||
|
||||
|
||||
def extract_keywords_from_messages(messages):
|
||||
"""Extract combined keywords from all messages in the turn."""
|
||||
all_keywords = []
|
||||
seen = set()
|
||||
for msg in messages:
|
||||
content = msg.get("content", "")
|
||||
for kw in extract_keywords(content):
|
||||
if kw not in seen:
|
||||
seen.add(kw)
|
||||
all_keywords.append(kw)
|
||||
return all_keywords
|
||||
|
||||
|
||||
def extract_first_user_message(messages):
|
||||
"""Return the content of the first user message, or empty string."""
|
||||
for msg in messages:
|
||||
if msg.get("role") == "user":
|
||||
content = msg.get("content", "").strip()
|
||||
if content:
|
||||
return content
|
||||
return ""
|
||||
|
||||
|
||||
def extract_latest_user_message(messages):
|
||||
"""Return the content of the last user message, or empty string."""
|
||||
for msg in reversed(messages):
|
||||
if msg.get("role") == "user":
|
||||
content = msg.get("content", "").strip()
|
||||
if content:
|
||||
return content
|
||||
return ""
|
||||
|
||||
|
||||
def jaccard_similarity(set_a, set_b):
|
||||
"""Compute Jaccard similarity between two sets."""
|
||||
if not set_a and not set_b:
|
||||
return 1.0 # Both empty = identical (no topic)
|
||||
a = set(set_a)
|
||||
b = set(set_b)
|
||||
intersection = a & b
|
||||
union = a | b
|
||||
if not union:
|
||||
return 1.0
|
||||
return len(intersection) / len(union)
|
||||
|
||||
|
||||
def generate_summary(current_episode):
|
||||
"""Generate a short episode summary from stored episode context.
|
||||
|
||||
Uses the first_user_message and latest_user_message fields that are
|
||||
accumulated on the current_episode during normal (non-shift) turns.
|
||||
This avoids the problem of using the wrong turn's messages when a
|
||||
topic shift is detected.
|
||||
"""
|
||||
parts = []
|
||||
|
||||
first_msg = current_episode.get("first_user_message", "")
|
||||
if first_msg:
|
||||
parts.append(first_msg)
|
||||
|
||||
latest_msg = current_episode.get("latest_user_message", "")
|
||||
if latest_msg and latest_msg != first_msg:
|
||||
parts.append(latest_msg)
|
||||
|
||||
if not parts:
|
||||
# Fallback: summarize from keywords
|
||||
keywords = current_episode.get("keywords", [])
|
||||
if keywords:
|
||||
return f"Discussion about: {', '.join(keywords[:8])}"
|
||||
return "No summary available"
|
||||
|
||||
summary = " | ".join(parts)
|
||||
if len(summary) > MAX_SUMMARY_LEN:
|
||||
summary = summary[: MAX_SUMMARY_LEN - 3] + "..."
|
||||
return summary
|
||||
|
||||
|
||||
def now_iso():
|
||||
"""Return current UTC timestamp in ISO 8601 format."""
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def next_episode_id(episodes):
|
||||
"""Generate the next episode ID in ep_XXX format."""
|
||||
max_num = 0
|
||||
for ep in episodes:
|
||||
ep_id = ep.get("id", "")
|
||||
if ep_id.startswith("ep_"):
|
||||
try:
|
||||
num = int(ep_id[3:])
|
||||
if num > max_num:
|
||||
max_num = num
|
||||
except ValueError:
|
||||
pass
|
||||
return f"ep_{max_num + 1:03d}"
|
||||
|
||||
|
||||
def get_store_path(agent_id):
|
||||
"""Return the filesystem path for an agent's episode store."""
|
||||
store_dir = os.path.join(
|
||||
os.path.expanduser("~"), ".librefang", "plugins", "episodic-memory"
|
||||
)
|
||||
os.makedirs(store_dir, exist_ok=True)
|
||||
return os.path.join(store_dir, f"{agent_id}.json")
|
||||
|
||||
|
||||
def load_episode_store(agent_id):
|
||||
"""Load the episode store, returning a default structure on any failure."""
|
||||
store_path = get_store_path(agent_id)
|
||||
if not os.path.isfile(store_path):
|
||||
return {"episodes": [], "current_episode": None}
|
||||
try:
|
||||
with open(store_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
# Ensure expected structure
|
||||
if not isinstance(data, dict):
|
||||
return {"episodes": [], "current_episode": None}
|
||||
if "episodes" not in data:
|
||||
data["episodes"] = []
|
||||
return data
|
||||
except (json.JSONDecodeError, OSError):
|
||||
return {"episodes": [], "current_episode": None}
|
||||
|
||||
|
||||
def save_episode_store(agent_id, store):
|
||||
"""Persist the episode store to disk."""
|
||||
store_path = get_store_path(agent_id)
|
||||
with open(store_path, "w", encoding="utf-8") as f:
|
||||
json.dump(store, f, indent=2, ensure_ascii=False)
|
||||
|
||||
|
||||
def evict_oldest_episodes(episodes):
|
||||
"""Keep only the MAX_EPISODES most recent completed episodes."""
|
||||
if len(episodes) <= MAX_EPISODES:
|
||||
return episodes
|
||||
# Sort by ended date descending, keep newest
|
||||
episodes.sort(
|
||||
key=lambda ep: ep.get("ended", ep.get("started", "")),
|
||||
reverse=True,
|
||||
)
|
||||
return episodes[:MAX_EPISODES]
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
request = json.loads(sys.stdin.read())
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
print(json.dumps({"type": "ok"}))
|
||||
return
|
||||
|
||||
agent_id = request.get("agent_id", "")
|
||||
messages = request.get("messages", [])
|
||||
|
||||
if not agent_id or not messages:
|
||||
print(json.dumps({"type": "ok"}))
|
||||
return
|
||||
|
||||
store = load_episode_store(agent_id)
|
||||
turn_keywords = extract_keywords_from_messages(messages)
|
||||
|
||||
current = store.get("current_episode")
|
||||
|
||||
if current is None:
|
||||
# No active episode -- start one
|
||||
ep_id = next_episode_id(store["episodes"])
|
||||
first_msg = extract_first_user_message(messages)
|
||||
store["current_episode"] = {
|
||||
"id": ep_id,
|
||||
"started": now_iso(),
|
||||
"keywords": turn_keywords,
|
||||
"messages_seen": 1,
|
||||
"last_keywords": turn_keywords,
|
||||
"first_user_message": first_msg,
|
||||
"latest_user_message": first_msg,
|
||||
}
|
||||
save_episode_store(agent_id, store)
|
||||
print(json.dumps({"type": "ok"}))
|
||||
return
|
||||
|
||||
# Compare current turn keywords against running episode keywords
|
||||
episode_keywords = current.get("keywords", [])
|
||||
similarity = jaccard_similarity(turn_keywords, episode_keywords)
|
||||
messages_seen = current.get("messages_seen", 0)
|
||||
|
||||
if similarity < TOPIC_SHIFT_THRESHOLD and messages_seen >= MIN_MESSAGES_FOR_COMPLETION:
|
||||
# Topic shift detected -- complete the current episode
|
||||
completed_episode = {
|
||||
"id": current.get("id", next_episode_id(store["episodes"])),
|
||||
"started": current.get("started", now_iso()),
|
||||
"ended": now_iso(),
|
||||
"keywords": episode_keywords,
|
||||
"summary": generate_summary(current),
|
||||
"message_count": messages_seen,
|
||||
"status": "completed",
|
||||
}
|
||||
store["episodes"].append(completed_episode)
|
||||
store["episodes"] = evict_oldest_episodes(store["episodes"])
|
||||
|
||||
# Start a new episode with current turn's keywords
|
||||
new_id = next_episode_id(store["episodes"])
|
||||
first_msg = extract_first_user_message(messages)
|
||||
store["current_episode"] = {
|
||||
"id": new_id,
|
||||
"started": now_iso(),
|
||||
"keywords": turn_keywords,
|
||||
"messages_seen": 1,
|
||||
"last_keywords": turn_keywords,
|
||||
"first_user_message": first_msg,
|
||||
"latest_user_message": first_msg,
|
||||
}
|
||||
else:
|
||||
# No topic shift -- update the current episode
|
||||
existing_kw_set = set(episode_keywords)
|
||||
merged_keywords = list(episode_keywords)
|
||||
for kw in turn_keywords:
|
||||
if kw not in existing_kw_set:
|
||||
existing_kw_set.add(kw)
|
||||
merged_keywords.append(kw)
|
||||
|
||||
current["keywords"] = merged_keywords
|
||||
current["messages_seen"] = messages_seen + 1
|
||||
current["last_keywords"] = turn_keywords
|
||||
|
||||
# Track user messages for summary generation
|
||||
latest_msg = extract_latest_user_message(messages)
|
||||
if latest_msg:
|
||||
current["latest_user_message"] = latest_msg
|
||||
if not current.get("first_user_message"):
|
||||
current["first_user_message"] = latest_msg
|
||||
|
||||
store["current_episode"] = current
|
||||
|
||||
save_episode_store(agent_id, store)
|
||||
print(json.dumps({"type": "ok"}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,144 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Episodic memory ingest hook.
|
||||
|
||||
Loads the episode store for the current agent, scores completed episodes
|
||||
against the incoming message's keywords using keyword overlap ratio, and
|
||||
returns the top 2 matching episodes as contextual memories.
|
||||
|
||||
This hook is read-only -- it never modifies the episode store.
|
||||
|
||||
Receives via stdin:
|
||||
{"type": "ingest", "agent_id": "...", "message": "user message text"}
|
||||
|
||||
Prints to stdout:
|
||||
{"type": "ingest_result", "memories": [{"content": "..."}]}
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Stopwords & keyword extraction
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
STOPWORDS = frozenset({
|
||||
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
|
||||
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
|
||||
"being", "have", "has", "had", "do", "does", "did", "will", "would",
|
||||
"could", "should", "may", "might", "shall", "can", "need", "must",
|
||||
"it", "its", "i", "me", "my", "you", "your", "he", "she", "we",
|
||||
"they", "them", "their", "this", "that", "these", "those", "what",
|
||||
"which", "who", "how", "when", "where", "why", "if", "then", "so",
|
||||
"not", "no", "just", "also", "very", "too", "about", "up", "out",
|
||||
"all", "some", "any", "each", "every", "into", "over", "after",
|
||||
})
|
||||
|
||||
MIN_WORD_LEN = 3
|
||||
|
||||
# Minimum overlap ratio to consider an episode relevant
|
||||
MIN_OVERLAP = 0.2
|
||||
|
||||
# Maximum episodes to return
|
||||
MAX_RESULTS = 2
|
||||
|
||||
|
||||
def extract_keywords(text):
|
||||
"""Extract deduplicated keywords from text.
|
||||
|
||||
Lowercases, removes stopwords, filters words shorter than MIN_WORD_LEN.
|
||||
"""
|
||||
words = re.findall(r"[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9]|[a-zA-Z]", text)
|
||||
seen = set()
|
||||
keywords = []
|
||||
for w in words:
|
||||
lower = w.lower()
|
||||
if lower not in STOPWORDS and len(lower) >= MIN_WORD_LEN and lower not in seen:
|
||||
seen.add(lower)
|
||||
keywords.append(lower)
|
||||
return keywords
|
||||
|
||||
|
||||
def load_episode_store(agent_id):
|
||||
"""Load the episode store JSON for the given agent. Returns None on failure."""
|
||||
store_dir = os.path.join(
|
||||
os.path.expanduser("~"), ".librefang", "plugins", "episodic-memory"
|
||||
)
|
||||
store_path = os.path.join(store_dir, f"{agent_id}.json")
|
||||
if not os.path.isfile(store_path):
|
||||
return None
|
||||
try:
|
||||
with open(store_path, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except (json.JSONDecodeError, OSError):
|
||||
return None
|
||||
|
||||
|
||||
def score_episode(episode_keywords, query_keywords):
|
||||
"""Compute keyword overlap ratio between an episode and the query.
|
||||
|
||||
overlap_ratio = |intersection| / |union| (Jaccard similarity)
|
||||
"""
|
||||
if not episode_keywords or not query_keywords:
|
||||
return 0.0
|
||||
ep_set = set(episode_keywords)
|
||||
q_set = set(query_keywords)
|
||||
intersection = ep_set & q_set
|
||||
union = ep_set | q_set
|
||||
if not union:
|
||||
return 0.0
|
||||
return len(intersection) / len(union)
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
request = json.loads(sys.stdin.read())
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||
return
|
||||
|
||||
message = request.get("message", "")
|
||||
agent_id = request.get("agent_id", "")
|
||||
|
||||
if not message.strip() or not agent_id:
|
||||
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||
return
|
||||
|
||||
query_keywords = extract_keywords(message)
|
||||
if not query_keywords:
|
||||
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||
return
|
||||
|
||||
store = load_episode_store(agent_id)
|
||||
if not store:
|
||||
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||
return
|
||||
|
||||
episodes = store.get("episodes", [])
|
||||
|
||||
# Score only completed episodes
|
||||
scored = []
|
||||
for ep in episodes:
|
||||
if ep.get("status") != "completed":
|
||||
continue
|
||||
overlap = score_episode(ep.get("keywords", []), query_keywords)
|
||||
if overlap > MIN_OVERLAP:
|
||||
scored.append((overlap, ep))
|
||||
|
||||
# Sort by overlap descending, take top MAX_RESULTS
|
||||
scored.sort(key=lambda x: x[0], reverse=True)
|
||||
top = scored[:MAX_RESULTS]
|
||||
|
||||
memories = []
|
||||
for _score, ep in top:
|
||||
timestamp = ep.get("ended", ep.get("started", "unknown"))
|
||||
summary = ep.get("summary", "no summary")
|
||||
memories.append({
|
||||
"content": f"[episodic-memory] Past episode ({timestamp}): {summary}"
|
||||
})
|
||||
|
||||
print(json.dumps({"type": "ingest_result", "memories": memories}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,8 @@
|
||||
name = "episodic-memory"
|
||||
version = "0.1.0"
|
||||
description = "Episode-based memory segmentation and recall for cross-conversation context continuity"
|
||||
author = "librefang"
|
||||
|
||||
[hooks]
|
||||
ingest = "hooks/ingest.py"
|
||||
after_turn = "hooks/after_turn.py"
|
||||
Whitespace-only changes.
@@ -0,0 +1,26 @@
|
||||
# topic-memory
|
||||
|
||||
Topic-aware memory recall using keyword clustering. Tracks conversation topics per agent and recalls related context when similar topics arise in future conversations.
|
||||
|
||||
## How it works
|
||||
|
||||
**After each turn**, the plugin extracts keywords from user and assistant messages, then either merges them into an existing topic cluster (Jaccard similarity > 0.3) or creates a new one. Each cluster stores a keyword set, a summary, a hit count, and a last-seen timestamp.
|
||||
|
||||
**On ingest**, the plugin scores all stored topic clusters against the incoming message keywords using Jaccard similarity and returns the top 3 matches (threshold > 0.15) as contextual memories.
|
||||
|
||||
This gives agents cross-conversation topic awareness -- if a user discussed Python async patterns last week, bringing up `asyncio` today will recall that context.
|
||||
|
||||
## Hooks
|
||||
|
||||
| Hook | Script | Description |
|
||||
|------|--------|-------------|
|
||||
| ingest | `hooks/ingest.py` | Scores stored topics against message keywords, returns top matches |
|
||||
| after_turn | `hooks/after_turn.py` | Extracts keywords, merges or creates topic clusters |
|
||||
|
||||
## Storage
|
||||
|
||||
Topic clusters are stored at `~/.librefang/plugins/topic-memory/{agent_id}.json`. Max 50 clusters per agent (lowest hit-count evicted when full).
|
||||
|
||||
## Usage
|
||||
|
||||
Installed automatically when enabled in agent configuration.
|
||||
@@ -0,0 +1,226 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Topic-memory after_turn hook.
|
||||
|
||||
After each conversation turn, extracts keywords from the latest exchange,
|
||||
then either merges into an existing topic cluster or creates a new one.
|
||||
|
||||
This hook is WRITE-ONLY -- it never returns memories.
|
||||
|
||||
Receives via stdin:
|
||||
{"type": "after_turn", "agent_id": "...", "messages": [
|
||||
{"role": "user"|"assistant", "content": "..."}
|
||||
]}
|
||||
|
||||
Prints to stdout:
|
||||
{"type": "ok"}
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Stopwords & constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
STOPWORDS = frozenset({
|
||||
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
|
||||
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
|
||||
"being", "have", "has", "had", "do", "does", "did", "will", "would",
|
||||
"could", "should", "may", "might", "shall", "can", "need", "must",
|
||||
"it", "its", "i", "me", "my", "you", "your", "he", "she", "we",
|
||||
"they", "them", "their", "this", "that", "these", "those", "what",
|
||||
"which", "who", "how", "when", "where", "why", "if", "then", "so",
|
||||
"not", "no", "just", "also", "very", "too", "about", "up", "out",
|
||||
"all", "some", "any", "each", "every", "into", "over", "after",
|
||||
})
|
||||
|
||||
MIN_WORD_LEN = 3
|
||||
MAX_TOPICS = 50
|
||||
MERGE_THRESHOLD = 0.3
|
||||
SUMMARY_MAX_LEN = 200
|
||||
|
||||
STORE_DIR = os.path.join(os.path.expanduser("~"), ".librefang", "plugins", "topic-memory")
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def extract_keywords(text):
|
||||
"""Extract lowercase keywords from text, filtering stopwords and short tokens."""
|
||||
words = re.findall(r"[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9]|[a-zA-Z]", text)
|
||||
seen = set()
|
||||
keywords = set()
|
||||
for w in words:
|
||||
lower = w.lower()
|
||||
if lower not in STOPWORDS and len(lower) >= MIN_WORD_LEN and lower not in seen:
|
||||
seen.add(lower)
|
||||
keywords.add(lower)
|
||||
return keywords
|
||||
|
||||
|
||||
def jaccard_similarity(set_a, set_b):
|
||||
"""Compute Jaccard similarity between two sets."""
|
||||
if not set_a or not set_b:
|
||||
return 0.0
|
||||
intersection = len(set_a & set_b)
|
||||
union = len(set_a | set_b)
|
||||
if union == 0:
|
||||
return 0.0
|
||||
return intersection / union
|
||||
|
||||
|
||||
def load_store(agent_id):
|
||||
"""Load the topic store JSON for an agent. Returns empty structure on any error."""
|
||||
path = os.path.join(STORE_DIR, f"{agent_id}.json")
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
if isinstance(data, dict) and isinstance(data.get("topics"), list):
|
||||
return data
|
||||
except (OSError, json.JSONDecodeError, ValueError):
|
||||
pass
|
||||
return {"topics": []}
|
||||
|
||||
|
||||
def save_store(agent_id, store):
|
||||
"""Persist the topic store to disk."""
|
||||
os.makedirs(STORE_DIR, exist_ok=True)
|
||||
path = os.path.join(STORE_DIR, f"{agent_id}.json")
|
||||
tmp_path = path + ".tmp"
|
||||
with open(tmp_path, "w", encoding="utf-8") as f:
|
||||
json.dump(store, f, ensure_ascii=False, indent=2)
|
||||
os.replace(tmp_path, path)
|
||||
|
||||
|
||||
def next_topic_id(topics):
|
||||
"""Generate the next t_XXX topic ID."""
|
||||
max_num = 0
|
||||
for t in topics:
|
||||
tid = t.get("id", "")
|
||||
if tid.startswith("t_"):
|
||||
try:
|
||||
num = int(tid[2:])
|
||||
if num > max_num:
|
||||
max_num = num
|
||||
except ValueError:
|
||||
pass
|
||||
return f"t_{max_num + 1:03d}"
|
||||
|
||||
|
||||
def build_summary(user_content, assistant_content):
|
||||
"""Build a truncated summary from the latest user + assistant exchange."""
|
||||
parts = []
|
||||
if user_content:
|
||||
parts.append(f"User: {user_content.strip()}")
|
||||
if assistant_content:
|
||||
parts.append(f"Assistant: {assistant_content.strip()}")
|
||||
raw = " | ".join(parts)
|
||||
if len(raw) > SUMMARY_MAX_LEN:
|
||||
return raw[: SUMMARY_MAX_LEN - 3] + "..."
|
||||
return raw
|
||||
|
||||
|
||||
def now_iso():
|
||||
"""Return current UTC time as ISO 8601 string."""
|
||||
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def ok_result():
|
||||
"""Return the standard ok response."""
|
||||
return json.dumps({"type": "ok"})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
request = json.loads(sys.stdin.read())
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
print(ok_result())
|
||||
return
|
||||
|
||||
agent_id = request.get("agent_id", "")
|
||||
messages = request.get("messages", [])
|
||||
|
||||
if not agent_id or not isinstance(messages, list) or not messages:
|
||||
print(ok_result())
|
||||
return
|
||||
|
||||
# Extract the latest user and assistant messages
|
||||
latest_user = ""
|
||||
latest_assistant = ""
|
||||
for msg in reversed(messages):
|
||||
role = msg.get("role", "")
|
||||
content = msg.get("content", "")
|
||||
if role == "assistant" and not latest_assistant:
|
||||
latest_assistant = content
|
||||
elif role == "user" and not latest_user:
|
||||
latest_user = content
|
||||
if latest_user and latest_assistant:
|
||||
break
|
||||
|
||||
if not latest_user and not latest_assistant:
|
||||
print(ok_result())
|
||||
return
|
||||
|
||||
# Extract keywords from the combined exchange
|
||||
combined_text = f"{latest_user} {latest_assistant}"
|
||||
current_keywords = extract_keywords(combined_text)
|
||||
|
||||
if not current_keywords:
|
||||
print(ok_result())
|
||||
return
|
||||
|
||||
store = load_store(agent_id)
|
||||
topics = store.get("topics", [])
|
||||
timestamp = now_iso()
|
||||
|
||||
# Find the best matching existing topic cluster
|
||||
best_sim = 0.0
|
||||
best_idx = -1
|
||||
for idx, topic in enumerate(topics):
|
||||
topic_kw = set(topic.get("keywords", []))
|
||||
sim = jaccard_similarity(current_keywords, topic_kw)
|
||||
if sim > best_sim:
|
||||
best_sim = sim
|
||||
best_idx = idx
|
||||
|
||||
if best_sim >= MERGE_THRESHOLD and best_idx >= 0:
|
||||
# Merge into existing topic cluster
|
||||
topic = topics[best_idx]
|
||||
existing_kw = set(topic.get("keywords", []))
|
||||
merged_kw = existing_kw | current_keywords
|
||||
topic["keywords"] = sorted(merged_kw)
|
||||
topic["summary"] = build_summary(latest_user, latest_assistant)
|
||||
topic["last_seen"] = timestamp
|
||||
topic["hit_count"] = topic.get("hit_count", 0) + 1
|
||||
else:
|
||||
# Create a new topic cluster
|
||||
new_topic = {
|
||||
"id": next_topic_id(topics),
|
||||
"keywords": sorted(current_keywords),
|
||||
"summary": build_summary(latest_user, latest_assistant),
|
||||
"last_seen": timestamp,
|
||||
"hit_count": 1,
|
||||
}
|
||||
topics.append(new_topic)
|
||||
|
||||
# Evict lowest hit_count topics if over capacity
|
||||
if len(topics) > MAX_TOPICS:
|
||||
topics.sort(key=lambda t: (t.get("hit_count", 0), t.get("last_seen", "")))
|
||||
topics = topics[len(topics) - MAX_TOPICS:]
|
||||
|
||||
store["topics"] = topics
|
||||
save_store(agent_id, store)
|
||||
|
||||
print(ok_result())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,139 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Topic-memory ingest hook.
|
||||
|
||||
Reads the topic store for the given agent and returns the top matching
|
||||
topic summaries based on Jaccard similarity between the incoming message
|
||||
keywords and each stored topic cluster.
|
||||
|
||||
This hook is READ-ONLY -- it never modifies the topic store.
|
||||
|
||||
Receives via stdin:
|
||||
{"type": "ingest", "agent_id": "...", "message": "user message text"}
|
||||
|
||||
Prints to stdout:
|
||||
{"type": "ingest_result", "memories": [{"content": "..."}]}
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Stopwords & constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
STOPWORDS = frozenset({
|
||||
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
|
||||
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
|
||||
"being", "have", "has", "had", "do", "does", "did", "will", "would",
|
||||
"could", "should", "may", "might", "shall", "can", "need", "must",
|
||||
"it", "its", "i", "me", "my", "you", "your", "he", "she", "we",
|
||||
"they", "them", "their", "this", "that", "these", "those", "what",
|
||||
"which", "who", "how", "when", "where", "why", "if", "then", "so",
|
||||
"not", "no", "just", "also", "very", "too", "about", "up", "out",
|
||||
"all", "some", "any", "each", "every", "into", "over", "after",
|
||||
})
|
||||
|
||||
MIN_WORD_LEN = 3
|
||||
MAX_RESULTS = 3
|
||||
MIN_SIMILARITY = 0.15
|
||||
|
||||
STORE_DIR = os.path.join(os.path.expanduser("~"), ".librefang", "plugins", "topic-memory")
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def extract_keywords(text):
|
||||
"""Extract lowercase keywords from text, filtering stopwords and short tokens."""
|
||||
words = re.findall(r"[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9]|[a-zA-Z]", text)
|
||||
seen = set()
|
||||
keywords = set()
|
||||
for w in words:
|
||||
lower = w.lower()
|
||||
if lower not in STOPWORDS and len(lower) >= MIN_WORD_LEN and lower not in seen:
|
||||
seen.add(lower)
|
||||
keywords.add(lower)
|
||||
return keywords
|
||||
|
||||
|
||||
def jaccard_similarity(set_a, set_b):
|
||||
"""Compute Jaccard similarity between two sets."""
|
||||
if not set_a or not set_b:
|
||||
return 0.0
|
||||
intersection = len(set_a & set_b)
|
||||
union = len(set_a | set_b)
|
||||
if union == 0:
|
||||
return 0.0
|
||||
return intersection / union
|
||||
|
||||
|
||||
def load_store(agent_id):
|
||||
"""Load the topic store JSON for an agent. Returns empty structure on any error."""
|
||||
path = os.path.join(STORE_DIR, f"{agent_id}.json")
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
if isinstance(data, dict) and isinstance(data.get("topics"), list):
|
||||
return data
|
||||
except (OSError, json.JSONDecodeError, ValueError):
|
||||
pass
|
||||
return {"topics": []}
|
||||
|
||||
|
||||
def empty_result():
|
||||
"""Return an empty ingest result."""
|
||||
return json.dumps({"type": "ingest_result", "memories": []})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
request = json.loads(sys.stdin.read())
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
print(empty_result())
|
||||
return
|
||||
|
||||
message = request.get("message", "")
|
||||
agent_id = request.get("agent_id", "")
|
||||
|
||||
if not message.strip() or not agent_id:
|
||||
print(empty_result())
|
||||
return
|
||||
|
||||
msg_keywords = extract_keywords(message)
|
||||
if not msg_keywords:
|
||||
print(empty_result())
|
||||
return
|
||||
|
||||
store = load_store(agent_id)
|
||||
topics = store.get("topics", [])
|
||||
|
||||
# Score each topic cluster against the current message keywords
|
||||
scored = []
|
||||
for topic in topics:
|
||||
topic_kw = set(topic.get("keywords", []))
|
||||
sim = jaccard_similarity(msg_keywords, topic_kw)
|
||||
if sim >= MIN_SIMILARITY:
|
||||
scored.append((sim, topic))
|
||||
|
||||
# Sort descending by similarity, take top N
|
||||
scored.sort(key=lambda x: x[0], reverse=True)
|
||||
top = scored[:MAX_RESULTS]
|
||||
|
||||
memories = []
|
||||
for _sim, topic in top:
|
||||
summary = topic.get("summary", "")
|
||||
if summary:
|
||||
memories.append({"content": f"[topic-memory] Related context: {summary}"})
|
||||
|
||||
print(json.dumps({"type": "ingest_result", "memories": memories}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,8 @@
|
||||
name = "topic-memory"
|
||||
version = "0.1.0"
|
||||
description = "Topic-aware memory recall with keyword clustering for cross-conversation context"
|
||||
author = "librefang"
|
||||
|
||||
[hooks]
|
||||
ingest = "hooks/ingest.py"
|
||||
after_turn = "hooks/after_turn.py"
|
||||
Whitespace-only changes.
@@ -0,0 +1,29 @@
|
||||
# user-profile
|
||||
|
||||
Persistent user profiling from conversation patterns. Builds a profile of user expertise areas, communication style, and technical level, then injects it as context so agents can personalize responses.
|
||||
|
||||
## How it works
|
||||
|
||||
**After each turn**, the plugin analyzes user messages to update the profile:
|
||||
|
||||
- **Expertise areas** -- extracts technical keywords and tracks frequency (top 20 retained)
|
||||
- **Communication style** -- running average of message lengths (brief / moderate / detailed)
|
||||
- **Technical level** -- scored from signals like code blocks, version numbers, tech abbreviations, and question patterns (beginner / intermediate / advanced)
|
||||
- **Question ratio** -- fraction of user messages containing questions
|
||||
|
||||
**On ingest**, once the profile has at least 5 interactions, the plugin returns a compact profile summary as a memory fragment: `expertise=python,devops; style=detailed; level=advanced`.
|
||||
|
||||
## Hooks
|
||||
|
||||
| Hook | Script | Description |
|
||||
|------|--------|-------------|
|
||||
| ingest | `hooks/ingest.py` | Returns the profile summary as a memory fragment (after 5+ interactions) |
|
||||
| after_turn | `hooks/after_turn.py` | Analyzes user messages to update the profile |
|
||||
|
||||
## Storage
|
||||
|
||||
Profiles are stored at `~/.librefang/plugins/user-profile/{agent_id}.json`.
|
||||
|
||||
## Usage
|
||||
|
||||
Installed automatically when enabled in agent configuration.
|
||||
@@ -0,0 +1,282 @@
|
||||
#!/usr/bin/env python3
|
||||
"""User-profile after_turn hook.
|
||||
|
||||
Analyses user messages from the completed turn and updates the persisted
|
||||
user profile with extracted signals: expertise areas, message length
|
||||
statistics, question ratio, and inferred technical level.
|
||||
|
||||
This hook is WRITE-ONLY -- it updates the profile store but never returns
|
||||
memories.
|
||||
|
||||
Receives via stdin:
|
||||
{"type": "after_turn", "agent_id": "...", "messages": [...]}
|
||||
|
||||
Each message: {"role": "user"|"assistant", "content": "..."}
|
||||
|
||||
Prints to stdout:
|
||||
{"type": "ok"}
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
STORE_DIR = os.path.join(
|
||||
os.path.expanduser("~"), ".librefang", "plugins", "user-profile"
|
||||
)
|
||||
|
||||
MAX_EXPERTISE_ENTRIES = 20
|
||||
MIN_KEYWORD_LEN = 3
|
||||
|
||||
# Technical abbreviations that signal intermediate+ level
|
||||
TECH_ABBREVIATIONS = frozenset({
|
||||
"api", "cli", "sdk", "orm", "sql", "css", "html", "http", "https",
|
||||
"jwt", "oauth", "ssr", "csr", "dom", "cdn", "dns", "tcp", "udp",
|
||||
"grpc", "wasm", "yaml", "toml", "json", "xml", "cicd", "gpu",
|
||||
"cpu", "ram", "ssd", "tls", "ssh", "llm", "rag", "mlops", "etl",
|
||||
"crud", "rest", "graphql", "ide", "vcs", "iot", "saas", "paas",
|
||||
})
|
||||
|
||||
STOPWORDS = frozenset({
|
||||
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
|
||||
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
|
||||
"being", "have", "has", "had", "do", "does", "did", "will", "would",
|
||||
"could", "should", "may", "might", "shall", "can", "need", "must",
|
||||
"it", "its", "i", "me", "my", "you", "your", "he", "she", "we",
|
||||
"they", "them", "their", "this", "that", "these", "those", "what",
|
||||
"which", "who", "how", "when", "where", "why", "if", "then", "so",
|
||||
"not", "no", "just", "also", "very", "too", "about", "up", "out",
|
||||
"all", "some", "any", "each", "every", "into", "over", "after",
|
||||
"been", "before", "between", "both", "down", "during", "few", "get",
|
||||
"got", "here", "him", "his", "her", "like", "make", "many", "more",
|
||||
"most", "much", "new", "now", "old", "one", "only", "other", "our",
|
||||
"own", "same", "say", "see", "still", "such", "take", "than",
|
||||
"there", "thing", "think", "time", "use", "used", "using", "want",
|
||||
"way", "well", "work", "know", "really", "right", "going", "back",
|
||||
})
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def ok_result():
|
||||
"""Return an ok result."""
|
||||
return json.dumps({"type": "ok"})
|
||||
|
||||
|
||||
def load_profile(agent_id):
|
||||
"""Load the profile JSON for an agent. Returns default structure on any error."""
|
||||
path = os.path.join(STORE_DIR, f"{agent_id}.json")
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
if isinstance(data, dict) and isinstance(data.get("interaction_count"), int):
|
||||
return data
|
||||
except (OSError, json.JSONDecodeError, ValueError):
|
||||
pass
|
||||
return {
|
||||
"interaction_count": 0,
|
||||
"expertise_areas": {},
|
||||
"avg_message_length": 0.0,
|
||||
"question_ratio": 0.0,
|
||||
"technical_level": "beginner",
|
||||
"last_updated": "",
|
||||
}
|
||||
|
||||
|
||||
def save_profile(agent_id, profile):
|
||||
"""Persist profile to disk."""
|
||||
os.makedirs(STORE_DIR, exist_ok=True)
|
||||
path = os.path.join(STORE_DIR, f"{agent_id}.json")
|
||||
try:
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(profile, f, indent=2, ensure_ascii=False)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def extract_keywords(text):
|
||||
"""Extract meaningful keywords from text, filtering stopwords."""
|
||||
# Handle hyphenated compound terms and regular words
|
||||
words = re.findall(r"[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9]|[a-zA-Z]", text)
|
||||
keywords = []
|
||||
seen = set()
|
||||
for w in words:
|
||||
lower = w.lower()
|
||||
if lower not in STOPWORDS and len(lower) >= MIN_KEYWORD_LEN and lower not in seen:
|
||||
seen.add(lower)
|
||||
keywords.append(lower)
|
||||
return keywords
|
||||
|
||||
|
||||
def has_code_blocks(text):
|
||||
"""Check if text contains code blocks or backtick references."""
|
||||
return bool(re.search(r"```|`[^`]+`", text))
|
||||
|
||||
|
||||
def has_version_numbers(text):
|
||||
"""Check if text contains version references like v3.2, Python 3.12, etc."""
|
||||
return bool(re.search(r"v\d+\.\d+|(?<!\w)\d+\.\d+\.\d+", text))
|
||||
|
||||
|
||||
def has_tech_abbreviations(text):
|
||||
"""Check if text contains known technical abbreviations."""
|
||||
words = set(re.findall(r"\b[a-zA-Z]{2,6}\b", text))
|
||||
lower_words = {w.lower() for w in words}
|
||||
return bool(lower_words & TECH_ABBREVIATIONS)
|
||||
|
||||
|
||||
def has_basic_questions(text):
|
||||
"""Check if text contains beginner-style 'what is' / 'explain' patterns."""
|
||||
lower = text.lower()
|
||||
return bool(re.search(r"\bwhat\s+is\b|\bexplain\b|\bwhat\s+are\b", lower))
|
||||
|
||||
|
||||
def average_word_length(text):
|
||||
"""Compute average word length in the text."""
|
||||
words = re.findall(r"[a-zA-Z]+", text)
|
||||
if not words:
|
||||
return 0.0
|
||||
return sum(len(w) for w in words) / len(words)
|
||||
|
||||
|
||||
def infer_technical_level(text):
|
||||
"""Infer technical level from a single message. Returns a score.
|
||||
|
||||
Score >= 3 -> "advanced"
|
||||
Score 1-2 -> "intermediate"
|
||||
Score <= 0 -> "beginner"
|
||||
"""
|
||||
score = 0
|
||||
|
||||
if has_code_blocks(text):
|
||||
score += 2
|
||||
if has_version_numbers(text):
|
||||
score += 1
|
||||
if has_tech_abbreviations(text):
|
||||
score += 1
|
||||
if has_basic_questions(text):
|
||||
score -= 1
|
||||
if average_word_length(text) > 5.5:
|
||||
score += 1
|
||||
|
||||
return score
|
||||
|
||||
|
||||
def tech_level_from_score(score):
|
||||
"""Map a numeric score to a technical level label."""
|
||||
if score >= 3:
|
||||
return "advanced"
|
||||
elif score >= 1:
|
||||
return "intermediate"
|
||||
else:
|
||||
return "beginner"
|
||||
|
||||
|
||||
def prune_expertise(expertise, max_entries):
|
||||
"""Keep only the top max_entries expertise areas by count."""
|
||||
if len(expertise) <= max_entries:
|
||||
return expertise
|
||||
sorted_items = sorted(expertise.items(), key=lambda x: x[1], reverse=True)
|
||||
return dict(sorted_items[:max_entries])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
request = json.loads(sys.stdin.read())
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
print(ok_result())
|
||||
return
|
||||
|
||||
agent_id = request.get("agent_id", "")
|
||||
messages = request.get("messages", [])
|
||||
|
||||
if not agent_id or not isinstance(messages, list):
|
||||
print(ok_result())
|
||||
return
|
||||
|
||||
# Filter to user messages only
|
||||
user_messages = []
|
||||
for msg in messages:
|
||||
if isinstance(msg, dict) and msg.get("role") == "user":
|
||||
content = msg.get("content", "")
|
||||
if isinstance(content, str) and content.strip():
|
||||
user_messages.append(content)
|
||||
|
||||
if not user_messages:
|
||||
print(ok_result())
|
||||
return
|
||||
|
||||
profile = load_profile(agent_id)
|
||||
|
||||
old_count = profile["interaction_count"]
|
||||
new_count = old_count + len(user_messages)
|
||||
|
||||
# --- Expertise areas ---
|
||||
expertise = profile.get("expertise_areas", {})
|
||||
for msg in user_messages:
|
||||
keywords = extract_keywords(msg)
|
||||
for kw in keywords:
|
||||
expertise[kw] = expertise.get(kw, 0) + 1
|
||||
|
||||
expertise = prune_expertise(expertise, MAX_EXPERTISE_ENTRIES)
|
||||
profile["expertise_areas"] = expertise
|
||||
|
||||
# --- Average message length (running average) ---
|
||||
old_avg = profile.get("avg_message_length", 0.0)
|
||||
total_new_len = sum(len(msg) for msg in user_messages)
|
||||
if old_count == 0:
|
||||
new_avg = total_new_len / len(user_messages)
|
||||
else:
|
||||
# Weighted running average: combine old aggregate with new messages
|
||||
old_total = old_avg * old_count
|
||||
new_avg = (old_total + total_new_len) / new_count
|
||||
profile["avg_message_length"] = round(new_avg, 1)
|
||||
|
||||
# --- Question ratio (running ratio) ---
|
||||
old_ratio = profile.get("question_ratio", 0.0)
|
||||
questions_in_batch = sum(1 for msg in user_messages if "?" in msg)
|
||||
if old_count == 0:
|
||||
new_ratio = questions_in_batch / len(user_messages)
|
||||
else:
|
||||
old_question_count = round(old_ratio * old_count)
|
||||
new_ratio = (old_question_count + questions_in_batch) / new_count
|
||||
profile["question_ratio"] = round(new_ratio, 3)
|
||||
|
||||
# --- Technical level (weighted towards recent) ---
|
||||
total_score = 0
|
||||
for msg in user_messages:
|
||||
total_score += infer_technical_level(msg)
|
||||
avg_score = total_score / len(user_messages)
|
||||
|
||||
# Blend with historical level: map old level to a score, then average
|
||||
level_to_score = {"beginner": 0, "intermediate": 1.5, "advanced": 3}
|
||||
old_level_score = level_to_score.get(profile.get("technical_level", "beginner"), 0)
|
||||
if old_count == 0:
|
||||
blended_score = avg_score
|
||||
else:
|
||||
# Give 70% weight to history, 30% to this batch
|
||||
blended_score = 0.7 * old_level_score + 0.3 * avg_score
|
||||
profile["technical_level"] = tech_level_from_score(blended_score)
|
||||
|
||||
# --- Bookkeeping ---
|
||||
profile["interaction_count"] = new_count
|
||||
profile["last_updated"] = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
save_profile(agent_id, profile)
|
||||
print(ok_result())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,134 @@
|
||||
#!/usr/bin/env python3
|
||||
"""User-profile ingest hook.
|
||||
|
||||
Reads the persisted user profile for the given agent and, when enough
|
||||
interaction data has been collected (>= 5 interactions), returns a
|
||||
compact profile summary as injected memory so the agent can personalise
|
||||
its responses.
|
||||
|
||||
This hook is READ-ONLY -- it never modifies the profile store.
|
||||
|
||||
Receives via stdin:
|
||||
{"type": "ingest", "agent_id": "...", "message": "user message text"}
|
||||
|
||||
Prints to stdout:
|
||||
{"type": "ingest_result", "memories": [{"content": "..."}]}
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
STORE_DIR = os.path.join(
|
||||
os.path.expanduser("~"), ".librefang", "plugins", "user-profile"
|
||||
)
|
||||
|
||||
MIN_INTERACTIONS = 5
|
||||
MAX_SUMMARY_LEN = 200
|
||||
TOP_EXPERTISE = 5
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def empty_result():
|
||||
"""Return an empty ingest result."""
|
||||
return json.dumps({"type": "ingest_result", "memories": []})
|
||||
|
||||
|
||||
def load_profile(agent_id):
|
||||
"""Load the profile JSON for an agent. Returns None on any error."""
|
||||
path = os.path.join(STORE_DIR, f"{agent_id}.json")
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
if isinstance(data, dict) and isinstance(data.get("interaction_count"), int):
|
||||
return data
|
||||
except (OSError, json.JSONDecodeError, ValueError):
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def message_length_bucket(avg_len):
|
||||
"""Classify average message length into a human-readable bucket."""
|
||||
if avg_len < 50:
|
||||
return "brief"
|
||||
elif avg_len <= 200:
|
||||
return "moderate"
|
||||
else:
|
||||
return "detailed"
|
||||
|
||||
|
||||
def build_summary(profile):
|
||||
"""Build a compact profile summary string (max MAX_SUMMARY_LEN chars)."""
|
||||
parts = []
|
||||
|
||||
# Top expertise areas
|
||||
expertise = profile.get("expertise_areas", {})
|
||||
if expertise:
|
||||
sorted_areas = sorted(expertise.items(), key=lambda x: x[1], reverse=True)
|
||||
top = [area for area, _count in sorted_areas[:TOP_EXPERTISE]]
|
||||
parts.append("expertise=" + ",".join(top))
|
||||
|
||||
# Communication style
|
||||
avg_len = profile.get("avg_message_length", 0)
|
||||
parts.append("style=" + message_length_bucket(avg_len))
|
||||
|
||||
# Technical level
|
||||
tech_level = profile.get("technical_level", "")
|
||||
if tech_level:
|
||||
parts.append("level=" + tech_level)
|
||||
|
||||
# Question ratio
|
||||
q_ratio = profile.get("question_ratio", 0.0)
|
||||
if q_ratio > 0.5:
|
||||
parts.append("asks-many-questions")
|
||||
|
||||
summary = "; ".join(parts)
|
||||
if len(summary) > MAX_SUMMARY_LEN:
|
||||
summary = summary[:MAX_SUMMARY_LEN - 3] + "..."
|
||||
return summary
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
request = json.loads(sys.stdin.read())
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
print(empty_result())
|
||||
return
|
||||
|
||||
agent_id = request.get("agent_id", "")
|
||||
if not agent_id:
|
||||
print(empty_result())
|
||||
return
|
||||
|
||||
profile = load_profile(agent_id)
|
||||
if profile is None:
|
||||
print(empty_result())
|
||||
return
|
||||
|
||||
interaction_count = profile.get("interaction_count", 0)
|
||||
if interaction_count < MIN_INTERACTIONS:
|
||||
print(empty_result())
|
||||
return
|
||||
|
||||
summary = build_summary(profile)
|
||||
if not summary:
|
||||
print(empty_result())
|
||||
return
|
||||
|
||||
memory = {"content": f"[user-profile] User context: {summary}"}
|
||||
print(json.dumps({"type": "ingest_result", "memories": [memory]}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,8 @@
|
||||
name = "user-profile"
|
||||
version = "0.1.0"
|
||||
description = "Persistent user profiling from conversation patterns for personalized agent responses"
|
||||
author = "librefang"
|
||||
|
||||
[hooks]
|
||||
ingest = "hooks/ingest.py"
|
||||
after_turn = "hooks/after_turn.py"
|
||||
Whitespace-only changes.
Reference in new issue
Block a user