feat: context engine plugins, scaffolding, and pricing fixes (#6)
* feat: add 4 context engine plugins
- topic-memory: keyword clustering for topic-aware memory recall
- episodic-memory: conversation segmentation and cross-session recall
- user-profile: persistent user profiling from conversation patterns
- context-decay: time-based memory decay with reinforcement dynamics
All plugins use the ingest/after_turn hook protocol with stdin/stdout JSON.
* chore: add plugin scaffolding, update docs and templates
- Add plugin.toml template with {{NAME}} placeholder
- Add new-plugin Makefile target with hooks/ scaffolding
- Update plugins/README.md with all 10 plugins
- Update README.md stats (10 plugins, 220+ models)
- Add Plugin checkbox and checklist to PR template
- Add Plugin to issue template content type dropdown
- Fix CONTRIBUTING.md: last_verified is recommended, not required
* fix: correct model pricing and remove deprecated entries
- openrouter/gemma-2-9b-it: fix pricing from 0.0 to 0.03/0.09 per M tokens
(free variant correctly stays at 0.0)
- github-copilot: remove deprecated copilot/gpt-4 model entry
(GPT-4 retired in favor of GPT-4o for Copilot)
* docs: annotate kimi-coding as membership-gated
Kimi Code CLI uses quota-based membership model (not per-token billing).
Free tier has limited weekly requests; underlying model is K2.5.
Pricing kept at 0.0 consistent with other subscription providers
(chatgpt, github-copilot) but with explanatory comments.
* style: fix trailing newline in github-copilot.toml
* fix: correct Moonshot/Kimi model pricing from official sources
All 5 models had incorrect pricing:
- moonshot-v1-8k: 0.10/0.10 → 0.20/2.00
- moonshot-v1-32k: 0.30/0.30 → 1.00/3.00
- moonshot-v1-128k: 0.80/0.80 → 2.00/5.00
- kimi-k2: 2.00/8.00 → 0.60/2.50
- kimi-k2.5: 2.00/8.00 → 0.45/2.20
Sources: platform.moonshot.ai/docs/pricing/chat, costgoat.com, getmaxim.ai
* feat: add MiniMax M2.7 and M2.7-highspeed models
Released 2026-03-18, MiniMax's latest flagship text model.
10B activated params, 200K context, 128K output, tool use, streaming.
Pricing: $0.30/$1.20 per M tokens (input/output).
Added to both international (minimax.io) and China (minimaxi.com) providers.
This commit is contained in:
33 files changed
+2085
-36
No files matched your search
@@ -1,5 +1,5 @@
|
|||||||
name: New Content Request
|
name: New Content Request
|
||||||
description: Request addition of an agent, hand, integration, or skill
|
description: Request addition of an agent, hand, integration, skill, or plugin
|
||||||
labels: ["new-content"]
|
labels: ["new-content"]
|
||||||
body:
|
body:
|
||||||
- type: dropdown
|
- type: dropdown
|
||||||
@@ -12,6 +12,7 @@ body:
|
|||||||
- Hand
|
- Hand
|
||||||
- Integration (MCP server)
|
- Integration (MCP server)
|
||||||
- Skill
|
- Skill
|
||||||
|
- Plugin
|
||||||
validations:
|
validations:
|
||||||
required: true
|
required: true
|
||||||
|
|
||||||
|
|||||||
@@ -6,6 +6,7 @@
|
|||||||
- [ ] Hand (`hands/`)
|
- [ ] Hand (`hands/`)
|
||||||
- [ ] Integration (`integrations/`)
|
- [ ] Integration (`integrations/`)
|
||||||
- [ ] Skill (`skills/`)
|
- [ ] Skill (`skills/`)
|
||||||
|
- [ ] Plugin (`plugins/`)
|
||||||
- [ ] Provider / Model (`providers/`)
|
- [ ] Provider / Model (`providers/`)
|
||||||
- [ ] Other
|
- [ ] Other
|
||||||
|
|
||||||
@@ -50,3 +51,10 @@
|
|||||||
- [ ] `[runtime].type` is correct
|
- [ ] `[runtime].type` is correct
|
||||||
- [ ] `[input]` documents all parameters
|
- [ ] `[input]` documents all parameters
|
||||||
- [ ] Prompt templates use correct `{{param}}` syntax
|
- [ ] Prompt templates use correct `{{param}}` syntax
|
||||||
|
|
||||||
|
### Plugins (if applicable)
|
||||||
|
- [ ] `name` matches directory name
|
||||||
|
- [ ] `[hooks]` lists at least one hook
|
||||||
|
- [ ] All referenced hook files exist and parse without errors
|
||||||
|
- [ ] Hook scripts read JSON from stdin and write JSON to stdout
|
||||||
|
- [ ] `requirements.txt` present (empty if stdlib-only)
|
||||||
+2
-2
@@ -222,7 +222,7 @@ aliases = ["short-name"]
|
|||||||
- [ ] `context_window` and `max_output_tokens` are positive integers
|
- [ ] `context_window` and `max_output_tokens` are positive integers
|
||||||
- [ ] Boolean capability fields are correct
|
- [ ] Boolean capability fields are correct
|
||||||
- [ ] Pricing verified from official source
|
- [ ] Pricing verified from official source
|
||||||
- [ ] `last_verified` date included (ISO format, e.g. `2025-03-15`)
|
- [ ] `last_verified` date included if possible (ISO format, e.g. `2025-03-15`)
|
||||||
|
|
||||||
### Pricing Verification
|
### Pricing Verification
|
||||||
|
|
||||||
@@ -232,7 +232,7 @@ When adding or updating model pricing:
|
|||||||
|
|
||||||
1. Check the provider's official pricing page (see links below)
|
1. Check the provider's official pricing page (see links below)
|
||||||
2. Record the exact `input_cost_per_m` and `output_cost_per_m` values in USD per million tokens
|
2. Record the exact `input_cost_per_m` and `output_cost_per_m` values in USD per million tokens
|
||||||
3. Include the `last_verified` field with today's date in ISO format (`YYYY-MM-DD`)
|
3. Include the `last_verified` field with today's date in ISO format (`YYYY-MM-DD`) when possible
|
||||||
4. If a model is subscription-based (e.g. GitHub Copilot) or has no public per-token pricing, note this in your PR description
|
4. If a model is subscription-based (e.g. GitHub Copilot) or has no public per-token pricing, note this in your PR description
|
||||||
|
|
||||||
Common official pricing pages:
|
Common official pricing pages:
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ AGENTS_DIR := agents
|
|||||||
HANDS_DIR := hands
|
HANDS_DIR := hands
|
||||||
INTEGRATIONS_DIR := integrations
|
INTEGRATIONS_DIR := integrations
|
||||||
SKILLS_DIR := skills
|
SKILLS_DIR := skills
|
||||||
|
PLUGINS_DIR := plugins
|
||||||
PROVIDERS_DIR := providers
|
PROVIDERS_DIR := providers
|
||||||
|
|
||||||
# ── Validation ────────────────────────────────────────────────────────────────
|
# ── Validation ────────────────────────────────────────────────────────────────
|
||||||
@@ -80,6 +81,18 @@ new-skill: ## Scaffold a new skill (usage: make new-skill NAME=my-skill)
|
|||||||
@echo "Created $(SKILLS_DIR)/$(NAME)/skill.toml"
|
@echo "Created $(SKILLS_DIR)/$(NAME)/skill.toml"
|
||||||
@echo "Next: edit the skill.toml to fill in TODO placeholders."
|
@echo "Next: edit the skill.toml to fill in TODO placeholders."
|
||||||
|
|
||||||
|
.PHONY: new-plugin
|
||||||
|
new-plugin: ## Scaffold a new plugin (usage: make new-plugin NAME=my-plugin)
|
||||||
|
@if [ -z "$(NAME)" ]; then echo "ERROR: NAME is required. Usage: make new-plugin NAME=my-plugin"; exit 1; fi
|
||||||
|
@if [ -d "$(PLUGINS_DIR)/$(NAME)" ]; then echo "ERROR: Plugin '$(NAME)' already exists."; exit 1; fi
|
||||||
|
@mkdir -p "$(PLUGINS_DIR)/$(NAME)/hooks"
|
||||||
|
@sed 's/{{NAME}}/$(NAME)/g' "$(TEMPLATES_DIR)/plugin.toml" > "$(PLUGINS_DIR)/$(NAME)/plugin.toml"
|
||||||
|
@touch "$(PLUGINS_DIR)/$(NAME)/hooks/ingest.py"
|
||||||
|
@touch "$(PLUGINS_DIR)/$(NAME)/hooks/after_turn.py"
|
||||||
|
@touch "$(PLUGINS_DIR)/$(NAME)/requirements.txt"
|
||||||
|
@echo "Created $(PLUGINS_DIR)/$(NAME)/ with plugin.toml and hooks/"
|
||||||
|
@echo "Next: implement the hook scripts and fill in TODO placeholders."
|
||||||
|
|
||||||
.PHONY: new-provider
|
.PHONY: new-provider
|
||||||
new-provider: ## Scaffold a new provider (usage: make new-provider NAME=my-provider)
|
new-provider: ## Scaffold a new provider (usage: make new-provider NAME=my-provider)
|
||||||
@if [ -z "$(NAME)" ]; then echo "ERROR: NAME is required. Usage: make new-provider NAME=my-provider"; exit 1; fi
|
@if [ -z "$(NAME)" ]; then echo "ERROR: NAME is required. Usage: make new-provider NAME=my-provider"; exit 1; fi
|
||||||
@@ -102,4 +115,5 @@ help: ## Show this help
|
|||||||
@echo " make new-hand NAME=my-hand"
|
@echo " make new-hand NAME=my-hand"
|
||||||
@echo " make new-integration NAME=my-service"
|
@echo " make new-integration NAME=my-service"
|
||||||
@echo " make new-skill NAME=my-skill"
|
@echo " make new-skill NAME=my-skill"
|
||||||
|
@echo " make new-plugin NAME=my-plugin"
|
||||||
@echo " make new-provider NAME=my-provider"
|
@echo " make new-provider NAME=my-provider"
|
||||||
@@ -27,7 +27,7 @@ librefang-registry/
|
|||||||
│ ├── anthropic.toml
|
│ ├── anthropic.toml
|
||||||
│ ├── openai.toml
|
│ ├── openai.toml
|
||||||
│ └── ... (46 providers, 190+ models)
|
│ └── ... (46 providers, 190+ models)
|
||||||
├── plugins/ # Plugin packages
|
├── plugins/ # Plugin packages (10 plugins)
|
||||||
├── aliases.toml # Global model alias mappings
|
├── aliases.toml # Global model alias mappings
|
||||||
├── schema.toml # Provider/model schema reference
|
├── schema.toml # Provider/model schema reference
|
||||||
├── scripts/
|
├── scripts/
|
||||||
@@ -169,8 +169,9 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed instructions for each conten
|
|||||||
| Hands | 14 |
|
| Hands | 14 |
|
||||||
| Integrations | 25 |
|
| Integrations | 25 |
|
||||||
| Skills | 2 |
|
| Skills | 2 |
|
||||||
|
| Plugins | 10 |
|
||||||
| Providers | 46 |
|
| Providers | 46 |
|
||||||
| Models | 190+ |
|
| Models | 220+ |
|
||||||
| Aliases | 80+ |
|
| Aliases | 80+ |
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|||||||
+6
-2
@@ -6,7 +6,7 @@ Plugin packages for LibreFang. Plugins extend agent behavior through lifecycle h
|
|||||||
|
|
||||||
```
|
```
|
||||||
plugins/
|
plugins/
|
||||||
└── echo-memory/
|
└── <plugin-name>/
|
||||||
├── plugin.toml # Plugin manifest
|
├── plugin.toml # Plugin manifest
|
||||||
├── hooks/
|
├── hooks/
|
||||||
│ ├── ingest.py # Called on user message
|
│ ├── ingest.py # Called on user message
|
||||||
@@ -45,16 +45,20 @@ stdin: {"type": "after_turn", "agent_id": "...", "messages": [...]}
|
|||||||
stdout: {"type": "ok"}
|
stdout: {"type": "ok"}
|
||||||
```
|
```
|
||||||
|
|
||||||
## Current Plugins (6)
|
## Current Plugins (10)
|
||||||
|
|
||||||
| Plugin | Hooks | Description |
|
| Plugin | Hooks | Description |
|
||||||
|--------|-------|-------------|
|
|--------|-------|-------------|
|
||||||
| auto-summarizer | ingest, after_turn | Running conversation summary for long context compression |
|
| auto-summarizer | ingest, after_turn | Running conversation summary for long context compression |
|
||||||
|
| context-decay | ingest, after_turn | Time-based memory decay with relevance scoring for natural forgetting |
|
||||||
| conversation-logger | after_turn | Logs conversations to JSONL files for auditing and analytics |
|
| conversation-logger | after_turn | Logs conversations to JSONL files for auditing and analytics |
|
||||||
|
| episodic-memory | ingest, after_turn | Episode-based conversation segmentation and cross-session recall |
|
||||||
| guardrails | ingest | Safety filter detecting PII, prompt injection, and credential exposure |
|
| guardrails | ingest | Safety filter detecting PII, prompt injection, and credential exposure |
|
||||||
| keyword-memory | ingest | Extracts keywords and named entities as contextual memories |
|
| keyword-memory | ingest | Extracts keywords and named entities as contextual memories |
|
||||||
| sentiment-tracker | ingest | Analyzes user sentiment and injects emotional context |
|
| sentiment-tracker | ingest | Analyzes user sentiment and injects emotional context |
|
||||||
| todo-tracker | ingest, after_turn | Detects, persists, and recalls action items from conversations |
|
| todo-tracker | ingest, after_turn | Detects, persists, and recalls action items from conversations |
|
||||||
|
| topic-memory | ingest, after_turn | Topic-aware keyword clustering with cross-conversation context recall |
|
||||||
|
| user-profile | ingest, after_turn | Persistent user profiling from conversation patterns for personalization |
|
||||||
|
|
||||||
## Adding a New Plugin
|
## Adding a New Plugin
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# context-decay
|
||||||
|
|
||||||
|
Time-based memory decay with relevance scoring. Memories lose confidence over time (5% per day) and are only recalled when they pass both a decay threshold and a relevance check against the current message. Implements "use it or lose it" -- recalled memories get their access timestamps refreshed.
|
||||||
|
|
||||||
|
## How it works
|
||||||
|
|
||||||
|
**After each turn**, the plugin extracts memorable statements from the conversation:
|
||||||
|
|
||||||
|
- User preferences ("I prefer...", "I use...")
|
||||||
|
- Decisions ("let's use...", "we decided...")
|
||||||
|
- Important facts (names, versions, URLs)
|
||||||
|
- Corrections ("no, actually...", "that's wrong...")
|
||||||
|
|
||||||
|
Similar memories are reinforced (confidence +0.1, cap 1.0). All memories receive a decay pass, and those below 0.1 confidence are pruned.
|
||||||
|
|
||||||
|
**On ingest**, each memory's confidence is decayed based on time elapsed, then scored for relevance to the current message via keyword overlap. The composite score `decayed_confidence * (0.5 + 0.5 * relevance)` must exceed 0.3 to be recalled. Recalled memories get their `last_accessed` timestamp updated.
|
||||||
|
|
||||||
|
## Hooks
|
||||||
|
|
||||||
|
| Hook | Script | Description |
|
||||||
|
|------|--------|-------------|
|
||||||
|
| ingest | `hooks/ingest.py` | Applies decay, scores relevance, returns top 5 memories above threshold |
|
||||||
|
| after_turn | `hooks/after_turn.py` | Extracts new memories, reinforces similar ones, prunes decayed entries |
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
|
||||||
|
Memories are stored at `~/.librefang/plugins/context-decay/{agent_id}.json`. Max 100 memories per agent. Decay formula: `confidence * 0.95^(hours / 24)`.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
Installed automatically when enabled in agent configuration.
|
||||||
@@ -0,0 +1,370 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Context-decay after_turn hook.
|
||||||
|
|
||||||
|
Extracts memorable statements from the conversation turn, stores new memories
|
||||||
|
or reinforces existing ones, applies time-based decay to all memories, and
|
||||||
|
prunes dead memories.
|
||||||
|
|
||||||
|
Receives via stdin:
|
||||||
|
{"type": "after_turn", "agent_id": "...", "messages": [
|
||||||
|
{"role": "user"|"assistant", "content": "..."}
|
||||||
|
]}
|
||||||
|
|
||||||
|
Prints to stdout:
|
||||||
|
{"type": "ok"}
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Constants
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Decay: 5% confidence loss per day
|
||||||
|
DECAY_FACTOR = 0.95
|
||||||
|
|
||||||
|
# Confidence below this gets pruned
|
||||||
|
PRUNE_THRESHOLD = 0.1
|
||||||
|
|
||||||
|
# Maximum memories per agent
|
||||||
|
MAX_MEMORIES = 100
|
||||||
|
|
||||||
|
# Initial confidence for new memories
|
||||||
|
INITIAL_CONFIDENCE = 0.8
|
||||||
|
|
||||||
|
# Confidence boost when reinforcing an existing memory
|
||||||
|
REINFORCE_BOOST = 0.1
|
||||||
|
|
||||||
|
# Minimum keyword overlap to consider memories similar
|
||||||
|
SIMILARITY_THRESHOLD = 0.5
|
||||||
|
|
||||||
|
# Maximum content length for a single memory
|
||||||
|
MAX_CONTENT_LEN = 200
|
||||||
|
|
||||||
|
# Minimum keyword length
|
||||||
|
MIN_WORD_LEN = 3
|
||||||
|
|
||||||
|
STOPWORDS = frozenset({
|
||||||
|
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
|
||||||
|
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
|
||||||
|
"being", "have", "has", "had", "do", "does", "did", "will", "would",
|
||||||
|
"could", "should", "may", "might", "shall", "can", "need", "must",
|
||||||
|
"it", "its", "i", "me", "my", "you", "your", "he", "she", "we",
|
||||||
|
"they", "them", "their", "this", "that", "these", "those", "what",
|
||||||
|
"which", "who", "how", "when", "where", "why", "if", "then", "so",
|
||||||
|
"not", "no", "just", "also", "very", "too", "about", "up", "out",
|
||||||
|
"all", "some", "any", "each", "every", "into", "over", "after",
|
||||||
|
})
|
||||||
|
|
||||||
|
# Patterns that indicate a memorable statement
|
||||||
|
PREFERENCE_PATTERNS = [
|
||||||
|
re.compile(r"\bi\s+prefer\b", re.IGNORECASE),
|
||||||
|
re.compile(r"\bi\s+like\b", re.IGNORECASE),
|
||||||
|
re.compile(r"\bi\s+use\b", re.IGNORECASE),
|
||||||
|
re.compile(r"\bi\s+want\b", re.IGNORECASE),
|
||||||
|
re.compile(r"\bi\s+need\b", re.IGNORECASE),
|
||||||
|
re.compile(r"\bi\s+always\b", re.IGNORECASE),
|
||||||
|
]
|
||||||
|
|
||||||
|
DECISION_PATTERNS = [
|
||||||
|
re.compile(r"\blet'?s?\s+use\b", re.IGNORECASE),
|
||||||
|
re.compile(r"\bwe\s+decided\b", re.IGNORECASE),
|
||||||
|
re.compile(r"\bgoing\s+with\b", re.IGNORECASE),
|
||||||
|
re.compile(r"\bwe\s+should\s+use\b", re.IGNORECASE),
|
||||||
|
re.compile(r"\bi'?ll\s+go\s+with\b", re.IGNORECASE),
|
||||||
|
]
|
||||||
|
|
||||||
|
CORRECTION_PATTERNS = [
|
||||||
|
re.compile(r"\bno,?\s+actually\b", re.IGNORECASE),
|
||||||
|
re.compile(r"\bthat'?s?\s+wrong\b", re.IGNORECASE),
|
||||||
|
re.compile(r"\bi\s+meant\b", re.IGNORECASE),
|
||||||
|
re.compile(r"\bactually,?\s+i\b", re.IGNORECASE),
|
||||||
|
re.compile(r"\bnot\s+that,?\s+", re.IGNORECASE),
|
||||||
|
]
|
||||||
|
|
||||||
|
# Pattern for specific facts: contains version numbers, URLs, or proper nouns
|
||||||
|
FACT_PATTERNS = [
|
||||||
|
re.compile(r"\bv?\d+\.\d+(?:\.\d+)?\b"), # version numbers
|
||||||
|
re.compile(r"https?://[^\s]+"), # URLs
|
||||||
|
re.compile(r"\b[A-Z][a-z]+(?:\s+[A-Z][a-z]+)+\b"), # proper noun phrases
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def extract_keywords(text):
|
||||||
|
"""Extract deduplicated lowercase keywords from text."""
|
||||||
|
words = re.findall(r"[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9]|[a-zA-Z]", text)
|
||||||
|
seen = set()
|
||||||
|
keywords = []
|
||||||
|
for w in words:
|
||||||
|
lower = w.lower()
|
||||||
|
if lower not in STOPWORDS and len(lower) >= MIN_WORD_LEN and lower not in seen:
|
||||||
|
seen.add(lower)
|
||||||
|
keywords.append(lower)
|
||||||
|
return keywords
|
||||||
|
|
||||||
|
|
||||||
|
def store_dir():
|
||||||
|
"""Return the storage directory path for context-decay."""
|
||||||
|
return os.path.join(
|
||||||
|
os.path.expanduser("~"), ".librefang", "plugins", "context-decay"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def store_path(agent_id):
|
||||||
|
"""Return the JSON store file path for a given agent."""
|
||||||
|
return os.path.join(store_dir(), f"{agent_id}.json")
|
||||||
|
|
||||||
|
|
||||||
|
def load_store(agent_id):
|
||||||
|
"""Load the memory store for an agent. Returns default on failure."""
|
||||||
|
path = store_path(agent_id)
|
||||||
|
if not os.path.isfile(path):
|
||||||
|
return {"memories": []}
|
||||||
|
try:
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
if not isinstance(data, dict) or "memories" not in data:
|
||||||
|
return {"memories": []}
|
||||||
|
return data
|
||||||
|
except (json.JSONDecodeError, OSError):
|
||||||
|
return {"memories": []}
|
||||||
|
|
||||||
|
|
||||||
|
def save_store(agent_id, data):
|
||||||
|
"""Persist the memory store for an agent."""
|
||||||
|
dirpath = store_dir()
|
||||||
|
os.makedirs(dirpath, exist_ok=True)
|
||||||
|
path = store_path(agent_id)
|
||||||
|
with open(path, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||||
|
|
||||||
|
|
||||||
|
def now_iso():
|
||||||
|
"""Return the current UTC time as an ISO 8601 string."""
|
||||||
|
return datetime.now(timezone.utc).isoformat()
|
||||||
|
|
||||||
|
|
||||||
|
def parse_iso(ts):
|
||||||
|
"""Parse an ISO 8601 timestamp string to a datetime object."""
|
||||||
|
if not ts:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
cleaned = ts.replace("Z", "+00:00")
|
||||||
|
return datetime.fromisoformat(cleaned)
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def hours_since(ts_str, now):
|
||||||
|
"""Calculate hours elapsed between a timestamp string and now."""
|
||||||
|
dt = parse_iso(ts_str)
|
||||||
|
if dt is None:
|
||||||
|
return 0.0
|
||||||
|
delta = now - dt
|
||||||
|
return max(delta.total_seconds() / 3600.0, 0.0)
|
||||||
|
|
||||||
|
|
||||||
|
def apply_decay(confidence, hours_elapsed):
|
||||||
|
"""Apply exponential decay: confidence * 0.95^(hours / 24)."""
|
||||||
|
if hours_elapsed <= 0:
|
||||||
|
return confidence
|
||||||
|
return confidence * (DECAY_FACTOR ** (hours_elapsed / 24.0))
|
||||||
|
|
||||||
|
|
||||||
|
def keyword_overlap(keywords_a, keywords_b):
|
||||||
|
"""Compute Jaccard similarity between two keyword lists."""
|
||||||
|
if not keywords_a or not keywords_b:
|
||||||
|
return 0.0
|
||||||
|
set_a = set(keywords_a)
|
||||||
|
set_b = set(keywords_b)
|
||||||
|
intersection = set_a & set_b
|
||||||
|
union = set_a | set_b
|
||||||
|
if not union:
|
||||||
|
return 0.0
|
||||||
|
return len(intersection) / len(union)
|
||||||
|
|
||||||
|
|
||||||
|
def next_memory_id(memories):
|
||||||
|
"""Generate the next incrementing memory ID in m_XXX format."""
|
||||||
|
max_num = 0
|
||||||
|
for mem in memories:
|
||||||
|
mid = mem.get("id", "")
|
||||||
|
if mid.startswith("m_"):
|
||||||
|
try:
|
||||||
|
num = int(mid[2:])
|
||||||
|
if num > max_num:
|
||||||
|
max_num = num
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
return f"m_{max_num + 1:03d}"
|
||||||
|
|
||||||
|
|
||||||
|
def truncate(text, max_len):
|
||||||
|
"""Truncate text to max_len, appending ... if trimmed."""
|
||||||
|
if len(text) <= max_len:
|
||||||
|
return text
|
||||||
|
return text[: max_len - 3].rstrip() + "..."
|
||||||
|
|
||||||
|
|
||||||
|
def matches_any(text, patterns):
|
||||||
|
"""Return True if text matches any of the compiled regex patterns."""
|
||||||
|
for pat in patterns:
|
||||||
|
if pat.search(text):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def extract_sentences(text):
|
||||||
|
"""Split text into sentences on common boundaries."""
|
||||||
|
# Split on period, exclamation, question mark followed by space or end
|
||||||
|
parts = re.split(r"(?<=[.!?])\s+", text.strip())
|
||||||
|
return [s.strip() for s in parts if s.strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def extract_memorable_statements(messages):
|
||||||
|
"""Extract statements worth remembering from conversation messages.
|
||||||
|
|
||||||
|
Focuses on user messages containing preferences, decisions, corrections,
|
||||||
|
or specific facts.
|
||||||
|
"""
|
||||||
|
statements = []
|
||||||
|
for msg in messages:
|
||||||
|
if msg.get("role") != "user":
|
||||||
|
continue
|
||||||
|
content = msg.get("content", "")
|
||||||
|
if not content.strip():
|
||||||
|
continue
|
||||||
|
|
||||||
|
sentences = extract_sentences(content)
|
||||||
|
for sentence in sentences:
|
||||||
|
# Check if this sentence matches any memorable pattern
|
||||||
|
is_memorable = (
|
||||||
|
matches_any(sentence, PREFERENCE_PATTERNS)
|
||||||
|
or matches_any(sentence, DECISION_PATTERNS)
|
||||||
|
or matches_any(sentence, CORRECTION_PATTERNS)
|
||||||
|
or matches_any(sentence, FACT_PATTERNS)
|
||||||
|
)
|
||||||
|
if is_memorable:
|
||||||
|
trimmed = truncate(sentence.strip(), MAX_CONTENT_LEN)
|
||||||
|
keywords = extract_keywords(trimmed)
|
||||||
|
if keywords:
|
||||||
|
statements.append({
|
||||||
|
"content": trimmed,
|
||||||
|
"keywords": keywords,
|
||||||
|
})
|
||||||
|
|
||||||
|
return statements
|
||||||
|
|
||||||
|
|
||||||
|
def find_similar_memory(memories, keywords):
|
||||||
|
"""Find an existing memory with keyword overlap above the similarity threshold.
|
||||||
|
|
||||||
|
Returns the index of the best match, or -1 if none found.
|
||||||
|
"""
|
||||||
|
best_idx = -1
|
||||||
|
best_overlap = 0.0
|
||||||
|
for i, mem in enumerate(memories):
|
||||||
|
overlap = keyword_overlap(mem.get("keywords", []), keywords)
|
||||||
|
if overlap > SIMILARITY_THRESHOLD and overlap > best_overlap:
|
||||||
|
best_overlap = overlap
|
||||||
|
best_idx = i
|
||||||
|
return best_idx
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def main():
|
||||||
|
try:
|
||||||
|
request = json.loads(sys.stdin.read())
|
||||||
|
except (json.JSONDecodeError, ValueError):
|
||||||
|
print(json.dumps({"type": "ok"}))
|
||||||
|
return
|
||||||
|
|
||||||
|
agent_id = request.get("agent_id", "")
|
||||||
|
messages = request.get("messages", [])
|
||||||
|
|
||||||
|
if not agent_id or not messages:
|
||||||
|
print(json.dumps({"type": "ok"}))
|
||||||
|
return
|
||||||
|
|
||||||
|
data = load_store(agent_id)
|
||||||
|
memories = data.get("memories", [])
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
now_str = now_iso()
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
# Step 1: Extract memorable statements from the conversation turn
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
statements = extract_memorable_statements(messages)
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
# Step 2: Store new memories or reinforce existing ones
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
for stmt in statements:
|
||||||
|
similar_idx = find_similar_memory(memories, stmt["keywords"])
|
||||||
|
|
||||||
|
if similar_idx >= 0:
|
||||||
|
# Reinforce existing memory
|
||||||
|
existing = memories[similar_idx]
|
||||||
|
existing["confidence"] = min(
|
||||||
|
existing.get("confidence", 0.0) + REINFORCE_BOOST, 1.0
|
||||||
|
)
|
||||||
|
existing["content"] = stmt["content"]
|
||||||
|
existing["keywords"] = stmt["keywords"]
|
||||||
|
existing["last_accessed"] = now_str
|
||||||
|
existing["access_count"] = existing.get("access_count", 0) + 1
|
||||||
|
else:
|
||||||
|
# Add new memory
|
||||||
|
new_mem = {
|
||||||
|
"id": next_memory_id(memories),
|
||||||
|
"content": stmt["content"],
|
||||||
|
"keywords": stmt["keywords"],
|
||||||
|
"confidence": INITIAL_CONFIDENCE,
|
||||||
|
"created": now_str,
|
||||||
|
"last_accessed": now_str,
|
||||||
|
"access_count": 0,
|
||||||
|
}
|
||||||
|
memories.append(new_mem)
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
# Step 3: Apply decay pass to ALL memories
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
for mem in memories:
|
||||||
|
elapsed = hours_since(mem.get("last_accessed", mem.get("created", "")), now)
|
||||||
|
mem["confidence"] = apply_decay(mem.get("confidence", 0.0), elapsed)
|
||||||
|
# Update last_accessed to now so next decay is relative to this pass
|
||||||
|
# (decay is applied on each hook invocation, not accumulated)
|
||||||
|
mem["last_accessed"] = now_str
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
# Step 4: Prune memories below the prune threshold
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
memories = [m for m in memories if m.get("confidence", 0.0) >= PRUNE_THRESHOLD]
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
# Step 5: Evict lowest-confidence memories if over capacity
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
if len(memories) > MAX_MEMORIES:
|
||||||
|
memories.sort(key=lambda m: m.get("confidence", 0.0), reverse=True)
|
||||||
|
memories = memories[:MAX_MEMORIES]
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
# Step 6: Save
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
data["memories"] = memories
|
||||||
|
save_store(agent_id, data)
|
||||||
|
|
||||||
|
print(json.dumps({"type": "ok"}))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,227 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Context-decay ingest hook.
|
||||||
|
|
||||||
|
Loads the memory store for the current agent, applies time-based decay to
|
||||||
|
all memories, scores them for relevance to the incoming message, and returns
|
||||||
|
the top matches above the recall threshold.
|
||||||
|
|
||||||
|
This hook updates last_accessed timestamps for recalled memories to implement
|
||||||
|
"use it or lose it" dynamics -- the one exception where ingest modifies storage.
|
||||||
|
|
||||||
|
Receives via stdin:
|
||||||
|
{"type": "ingest", "agent_id": "...", "message": "user message text"}
|
||||||
|
|
||||||
|
Prints to stdout:
|
||||||
|
{"type": "ingest_result", "memories": [{"content": "..."}]}
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Constants
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Decay: 5% confidence loss per day
|
||||||
|
DECAY_FACTOR = 0.95
|
||||||
|
|
||||||
|
# Minimum final_score to recall a memory
|
||||||
|
RECALL_THRESHOLD = 0.3
|
||||||
|
|
||||||
|
# Maximum memories to return per ingest
|
||||||
|
MAX_RECALL = 5
|
||||||
|
|
||||||
|
# Minimum keyword length
|
||||||
|
MIN_WORD_LEN = 3
|
||||||
|
|
||||||
|
STOPWORDS = frozenset({
|
||||||
|
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
|
||||||
|
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
|
||||||
|
"being", "have", "has", "had", "do", "does", "did", "will", "would",
|
||||||
|
"could", "should", "may", "might", "shall", "can", "need", "must",
|
||||||
|
"it", "its", "i", "me", "my", "you", "your", "he", "she", "we",
|
||||||
|
"they", "them", "their", "this", "that", "these", "those", "what",
|
||||||
|
"which", "who", "how", "when", "where", "why", "if", "then", "so",
|
||||||
|
"not", "no", "just", "also", "very", "too", "about", "up", "out",
|
||||||
|
"all", "some", "any", "each", "every", "into", "over", "after",
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def extract_keywords(text):
|
||||||
|
"""Extract deduplicated lowercase keywords from text."""
|
||||||
|
words = re.findall(r"[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9]|[a-zA-Z]", text)
|
||||||
|
seen = set()
|
||||||
|
keywords = []
|
||||||
|
for w in words:
|
||||||
|
lower = w.lower()
|
||||||
|
if lower not in STOPWORDS and len(lower) >= MIN_WORD_LEN and lower not in seen:
|
||||||
|
seen.add(lower)
|
||||||
|
keywords.append(lower)
|
||||||
|
return keywords
|
||||||
|
|
||||||
|
|
||||||
|
def store_dir():
|
||||||
|
"""Return the storage directory path for context-decay."""
|
||||||
|
return os.path.join(
|
||||||
|
os.path.expanduser("~"), ".librefang", "plugins", "context-decay"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def store_path(agent_id):
|
||||||
|
"""Return the JSON store file path for a given agent."""
|
||||||
|
return os.path.join(store_dir(), f"{agent_id}.json")
|
||||||
|
|
||||||
|
|
||||||
|
def load_store(agent_id):
|
||||||
|
"""Load the memory store for an agent. Returns default on failure."""
|
||||||
|
path = store_path(agent_id)
|
||||||
|
if not os.path.isfile(path):
|
||||||
|
return {"memories": []}
|
||||||
|
try:
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
if not isinstance(data, dict) or "memories" not in data:
|
||||||
|
return {"memories": []}
|
||||||
|
return data
|
||||||
|
except (json.JSONDecodeError, OSError):
|
||||||
|
return {"memories": []}
|
||||||
|
|
||||||
|
|
||||||
|
def save_store(agent_id, data):
|
||||||
|
"""Persist the memory store for an agent."""
|
||||||
|
dirpath = store_dir()
|
||||||
|
os.makedirs(dirpath, exist_ok=True)
|
||||||
|
path = store_path(agent_id)
|
||||||
|
with open(path, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||||
|
|
||||||
|
|
||||||
|
def now_iso():
|
||||||
|
"""Return the current UTC time as an ISO 8601 string."""
|
||||||
|
return datetime.now(timezone.utc).isoformat()
|
||||||
|
|
||||||
|
|
||||||
|
def parse_iso(ts):
|
||||||
|
"""Parse an ISO 8601 timestamp string to a datetime object.
|
||||||
|
|
||||||
|
Handles both +00:00 and Z suffixes. Returns None on failure.
|
||||||
|
"""
|
||||||
|
if not ts:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
# Replace Z suffix for compatibility with fromisoformat on older Python
|
||||||
|
cleaned = ts.replace("Z", "+00:00")
|
||||||
|
return datetime.fromisoformat(cleaned)
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def hours_since(ts_str, now):
|
||||||
|
"""Calculate hours elapsed between a timestamp string and now."""
|
||||||
|
dt = parse_iso(ts_str)
|
||||||
|
if dt is None:
|
||||||
|
return 0.0
|
||||||
|
delta = now - dt
|
||||||
|
return max(delta.total_seconds() / 3600.0, 0.0)
|
||||||
|
|
||||||
|
|
||||||
|
def apply_decay(confidence, hours_elapsed):
|
||||||
|
"""Apply exponential decay: confidence * 0.95^(hours / 24)."""
|
||||||
|
if hours_elapsed <= 0:
|
||||||
|
return confidence
|
||||||
|
return confidence * (DECAY_FACTOR ** (hours_elapsed / 24.0))
|
||||||
|
|
||||||
|
|
||||||
|
def keyword_overlap(keywords_a, keywords_b):
|
||||||
|
"""Compute Jaccard similarity between two keyword lists."""
|
||||||
|
if not keywords_a or not keywords_b:
|
||||||
|
return 0.0
|
||||||
|
set_a = set(keywords_a)
|
||||||
|
set_b = set(keywords_b)
|
||||||
|
intersection = set_a & set_b
|
||||||
|
union = set_a | set_b
|
||||||
|
if not union:
|
||||||
|
return 0.0
|
||||||
|
return len(intersection) / len(union)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def main():
|
||||||
|
try:
|
||||||
|
request = json.loads(sys.stdin.read())
|
||||||
|
except (json.JSONDecodeError, ValueError):
|
||||||
|
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||||
|
return
|
||||||
|
|
||||||
|
message = request.get("message", "")
|
||||||
|
agent_id = request.get("agent_id", "")
|
||||||
|
|
||||||
|
if not message.strip() or not agent_id:
|
||||||
|
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||||
|
return
|
||||||
|
|
||||||
|
query_keywords = extract_keywords(message)
|
||||||
|
if not query_keywords:
|
||||||
|
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||||
|
return
|
||||||
|
|
||||||
|
data = load_store(agent_id)
|
||||||
|
memories = data.get("memories", [])
|
||||||
|
|
||||||
|
if not memories:
|
||||||
|
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||||
|
return
|
||||||
|
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
scored = []
|
||||||
|
store_modified = False
|
||||||
|
|
||||||
|
for mem in memories:
|
||||||
|
# Apply time-based decay
|
||||||
|
elapsed = hours_since(mem.get("last_accessed", mem.get("created", "")), now)
|
||||||
|
decayed = apply_decay(mem.get("confidence", 0.0), elapsed)
|
||||||
|
|
||||||
|
# Score relevance via keyword overlap
|
||||||
|
relevance = keyword_overlap(mem.get("keywords", []), query_keywords)
|
||||||
|
|
||||||
|
# Composite score: decayed confidence weighted with relevance
|
||||||
|
final_score = decayed * (0.5 + 0.5 * relevance)
|
||||||
|
|
||||||
|
if final_score > RECALL_THRESHOLD:
|
||||||
|
scored.append((final_score, decayed, mem))
|
||||||
|
|
||||||
|
# Sort by final_score descending, take top MAX_RECALL
|
||||||
|
scored.sort(key=lambda x: x[0], reverse=True)
|
||||||
|
top = scored[:MAX_RECALL]
|
||||||
|
|
||||||
|
result_memories = []
|
||||||
|
for final_score, decayed, mem in top:
|
||||||
|
confidence_pct = int(round(decayed * 100))
|
||||||
|
content = mem.get("content", "")
|
||||||
|
result_memories.append({
|
||||||
|
"content": f"[context-decay] Recalled ({confidence_pct}%): {content}"
|
||||||
|
})
|
||||||
|
|
||||||
|
# Update last_accessed and save back -- "use it or lose it"
|
||||||
|
mem["last_accessed"] = now_iso()
|
||||||
|
mem["access_count"] = mem.get("access_count", 0) + 1
|
||||||
|
store_modified = True
|
||||||
|
|
||||||
|
# Persist access timestamp updates
|
||||||
|
if store_modified:
|
||||||
|
save_store(agent_id, data)
|
||||||
|
|
||||||
|
print(json.dumps({"type": "ingest_result", "memories": result_memories}))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
name = "context-decay"
|
||||||
|
version = "0.1.0"
|
||||||
|
description = "Time-based memory decay with relevance scoring for natural context forgetting"
|
||||||
|
author = "librefang"
|
||||||
|
|
||||||
|
[hooks]
|
||||||
|
ingest = "hooks/ingest.py"
|
||||||
|
after_turn = "hooks/after_turn.py"
|
||||||
Whitespace-only changes.
@@ -0,0 +1,26 @@
|
|||||||
|
# episodic-memory
|
||||||
|
|
||||||
|
Episode-based conversation segmentation and cross-session recall. Automatically detects topic shifts to split conversations into discrete episodes, then recalls relevant past episodes when similar topics arise.
|
||||||
|
|
||||||
|
## How it works
|
||||||
|
|
||||||
|
**After each turn**, the plugin tracks a "current episode" with accumulated keywords. When the Jaccard similarity between the current turn's keywords and the running episode keywords drops below 0.1 (and the episode has at least 4 messages), it marks the episode as completed with a summary and starts a new one.
|
||||||
|
|
||||||
|
**On ingest**, the plugin scores all completed episodes against the incoming message keywords and returns the top 2 matches (overlap > 0.2) as contextual memories including the episode timestamp and summary.
|
||||||
|
|
||||||
|
This gives agents episodic recall -- "last time we discussed Docker deployment, we configured nginx as a reverse proxy."
|
||||||
|
|
||||||
|
## Hooks
|
||||||
|
|
||||||
|
| Hook | Script | Description |
|
||||||
|
|------|--------|-------------|
|
||||||
|
| ingest | `hooks/ingest.py` | Scores completed episodes against message keywords, returns top matches |
|
||||||
|
| after_turn | `hooks/after_turn.py` | Tracks current episode, detects topic shifts, segments conversations |
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
|
||||||
|
Episodes are stored at `~/.librefang/plugins/episodic-memory/{agent_id}.json`. Max 30 completed episodes per agent (oldest evicted when full).
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
Installed automatically when enabled in agent configuration.
|
||||||
@@ -0,0 +1,305 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Episodic memory after_turn hook.
|
||||||
|
|
||||||
|
Maintains the episode store by tracking topic continuity across turns.
|
||||||
|
Detects topic shifts via Jaccard similarity between the current turn's
|
||||||
|
keywords and the running episode's keywords. When a shift is detected
|
||||||
|
(and the current episode has enough messages), the episode is completed
|
||||||
|
and a new one starts.
|
||||||
|
|
||||||
|
This hook is write-only -- it never returns memories.
|
||||||
|
|
||||||
|
Receives via stdin:
|
||||||
|
{"type": "after_turn", "agent_id": "...", "messages": [...]}
|
||||||
|
|
||||||
|
Prints to stdout:
|
||||||
|
{"type": "ok"}
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Stopwords & keyword extraction (identical logic to ingest.py)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
STOPWORDS = frozenset({
|
||||||
|
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
|
||||||
|
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
|
||||||
|
"being", "have", "has", "had", "do", "does", "did", "will", "would",
|
||||||
|
"could", "should", "may", "might", "shall", "can", "need", "must",
|
||||||
|
"it", "its", "i", "me", "my", "you", "your", "he", "she", "we",
|
||||||
|
"they", "them", "their", "this", "that", "these", "those", "what",
|
||||||
|
"which", "who", "how", "when", "where", "why", "if", "then", "so",
|
||||||
|
"not", "no", "just", "also", "very", "too", "about", "up", "out",
|
||||||
|
"all", "some", "any", "each", "every", "into", "over", "after",
|
||||||
|
})
|
||||||
|
|
||||||
|
MIN_WORD_LEN = 3
|
||||||
|
|
||||||
|
# Topic shift detection threshold
|
||||||
|
TOPIC_SHIFT_THRESHOLD = 0.1
|
||||||
|
|
||||||
|
# Minimum messages before allowing topic shift completion
|
||||||
|
MIN_MESSAGES_FOR_COMPLETION = 4
|
||||||
|
|
||||||
|
# Maximum completed episodes to retain
|
||||||
|
MAX_EPISODES = 30
|
||||||
|
|
||||||
|
# Maximum summary length in characters
|
||||||
|
MAX_SUMMARY_LEN = 150
|
||||||
|
|
||||||
|
|
||||||
|
def extract_keywords(text):
|
||||||
|
"""Extract deduplicated keywords from text."""
|
||||||
|
words = re.findall(r"[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9]|[a-zA-Z]", text)
|
||||||
|
seen = set()
|
||||||
|
keywords = []
|
||||||
|
for w in words:
|
||||||
|
lower = w.lower()
|
||||||
|
if lower not in STOPWORDS and len(lower) >= MIN_WORD_LEN and lower not in seen:
|
||||||
|
seen.add(lower)
|
||||||
|
keywords.append(lower)
|
||||||
|
return keywords
|
||||||
|
|
||||||
|
|
||||||
|
def extract_keywords_from_messages(messages):
|
||||||
|
"""Extract combined keywords from all messages in the turn."""
|
||||||
|
all_keywords = []
|
||||||
|
seen = set()
|
||||||
|
for msg in messages:
|
||||||
|
content = msg.get("content", "")
|
||||||
|
for kw in extract_keywords(content):
|
||||||
|
if kw not in seen:
|
||||||
|
seen.add(kw)
|
||||||
|
all_keywords.append(kw)
|
||||||
|
return all_keywords
|
||||||
|
|
||||||
|
|
||||||
|
def extract_first_user_message(messages):
|
||||||
|
"""Return the content of the first user message, or empty string."""
|
||||||
|
for msg in messages:
|
||||||
|
if msg.get("role") == "user":
|
||||||
|
content = msg.get("content", "").strip()
|
||||||
|
if content:
|
||||||
|
return content
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def extract_latest_user_message(messages):
|
||||||
|
"""Return the content of the last user message, or empty string."""
|
||||||
|
for msg in reversed(messages):
|
||||||
|
if msg.get("role") == "user":
|
||||||
|
content = msg.get("content", "").strip()
|
||||||
|
if content:
|
||||||
|
return content
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def jaccard_similarity(set_a, set_b):
|
||||||
|
"""Compute Jaccard similarity between two sets."""
|
||||||
|
if not set_a and not set_b:
|
||||||
|
return 1.0 # Both empty = identical (no topic)
|
||||||
|
a = set(set_a)
|
||||||
|
b = set(set_b)
|
||||||
|
intersection = a & b
|
||||||
|
union = a | b
|
||||||
|
if not union:
|
||||||
|
return 1.0
|
||||||
|
return len(intersection) / len(union)
|
||||||
|
|
||||||
|
|
||||||
|
def generate_summary(current_episode):
|
||||||
|
"""Generate a short episode summary from stored episode context.
|
||||||
|
|
||||||
|
Uses the first_user_message and latest_user_message fields that are
|
||||||
|
accumulated on the current_episode during normal (non-shift) turns.
|
||||||
|
This avoids the problem of using the wrong turn's messages when a
|
||||||
|
topic shift is detected.
|
||||||
|
"""
|
||||||
|
parts = []
|
||||||
|
|
||||||
|
first_msg = current_episode.get("first_user_message", "")
|
||||||
|
if first_msg:
|
||||||
|
parts.append(first_msg)
|
||||||
|
|
||||||
|
latest_msg = current_episode.get("latest_user_message", "")
|
||||||
|
if latest_msg and latest_msg != first_msg:
|
||||||
|
parts.append(latest_msg)
|
||||||
|
|
||||||
|
if not parts:
|
||||||
|
# Fallback: summarize from keywords
|
||||||
|
keywords = current_episode.get("keywords", [])
|
||||||
|
if keywords:
|
||||||
|
return f"Discussion about: {', '.join(keywords[:8])}"
|
||||||
|
return "No summary available"
|
||||||
|
|
||||||
|
summary = " | ".join(parts)
|
||||||
|
if len(summary) > MAX_SUMMARY_LEN:
|
||||||
|
summary = summary[: MAX_SUMMARY_LEN - 3] + "..."
|
||||||
|
return summary
|
||||||
|
|
||||||
|
|
||||||
|
def now_iso():
|
||||||
|
"""Return current UTC timestamp in ISO 8601 format."""
|
||||||
|
return datetime.now(timezone.utc).isoformat()
|
||||||
|
|
||||||
|
|
||||||
|
def next_episode_id(episodes):
|
||||||
|
"""Generate the next episode ID in ep_XXX format."""
|
||||||
|
max_num = 0
|
||||||
|
for ep in episodes:
|
||||||
|
ep_id = ep.get("id", "")
|
||||||
|
if ep_id.startswith("ep_"):
|
||||||
|
try:
|
||||||
|
num = int(ep_id[3:])
|
||||||
|
if num > max_num:
|
||||||
|
max_num = num
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
return f"ep_{max_num + 1:03d}"
|
||||||
|
|
||||||
|
|
||||||
|
def get_store_path(agent_id):
|
||||||
|
"""Return the filesystem path for an agent's episode store."""
|
||||||
|
store_dir = os.path.join(
|
||||||
|
os.path.expanduser("~"), ".librefang", "plugins", "episodic-memory"
|
||||||
|
)
|
||||||
|
os.makedirs(store_dir, exist_ok=True)
|
||||||
|
return os.path.join(store_dir, f"{agent_id}.json")
|
||||||
|
|
||||||
|
|
||||||
|
def load_episode_store(agent_id):
|
||||||
|
"""Load the episode store, returning a default structure on any failure."""
|
||||||
|
store_path = get_store_path(agent_id)
|
||||||
|
if not os.path.isfile(store_path):
|
||||||
|
return {"episodes": [], "current_episode": None}
|
||||||
|
try:
|
||||||
|
with open(store_path, "r", encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
# Ensure expected structure
|
||||||
|
if not isinstance(data, dict):
|
||||||
|
return {"episodes": [], "current_episode": None}
|
||||||
|
if "episodes" not in data:
|
||||||
|
data["episodes"] = []
|
||||||
|
return data
|
||||||
|
except (json.JSONDecodeError, OSError):
|
||||||
|
return {"episodes": [], "current_episode": None}
|
||||||
|
|
||||||
|
|
||||||
|
def save_episode_store(agent_id, store):
|
||||||
|
"""Persist the episode store to disk."""
|
||||||
|
store_path = get_store_path(agent_id)
|
||||||
|
with open(store_path, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(store, f, indent=2, ensure_ascii=False)
|
||||||
|
|
||||||
|
|
||||||
|
def evict_oldest_episodes(episodes):
|
||||||
|
"""Keep only the MAX_EPISODES most recent completed episodes."""
|
||||||
|
if len(episodes) <= MAX_EPISODES:
|
||||||
|
return episodes
|
||||||
|
# Sort by ended date descending, keep newest
|
||||||
|
episodes.sort(
|
||||||
|
key=lambda ep: ep.get("ended", ep.get("started", "")),
|
||||||
|
reverse=True,
|
||||||
|
)
|
||||||
|
return episodes[:MAX_EPISODES]
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
try:
|
||||||
|
request = json.loads(sys.stdin.read())
|
||||||
|
except (json.JSONDecodeError, ValueError):
|
||||||
|
print(json.dumps({"type": "ok"}))
|
||||||
|
return
|
||||||
|
|
||||||
|
agent_id = request.get("agent_id", "")
|
||||||
|
messages = request.get("messages", [])
|
||||||
|
|
||||||
|
if not agent_id or not messages:
|
||||||
|
print(json.dumps({"type": "ok"}))
|
||||||
|
return
|
||||||
|
|
||||||
|
store = load_episode_store(agent_id)
|
||||||
|
turn_keywords = extract_keywords_from_messages(messages)
|
||||||
|
|
||||||
|
current = store.get("current_episode")
|
||||||
|
|
||||||
|
if current is None:
|
||||||
|
# No active episode -- start one
|
||||||
|
ep_id = next_episode_id(store["episodes"])
|
||||||
|
first_msg = extract_first_user_message(messages)
|
||||||
|
store["current_episode"] = {
|
||||||
|
"id": ep_id,
|
||||||
|
"started": now_iso(),
|
||||||
|
"keywords": turn_keywords,
|
||||||
|
"messages_seen": 1,
|
||||||
|
"last_keywords": turn_keywords,
|
||||||
|
"first_user_message": first_msg,
|
||||||
|
"latest_user_message": first_msg,
|
||||||
|
}
|
||||||
|
save_episode_store(agent_id, store)
|
||||||
|
print(json.dumps({"type": "ok"}))
|
||||||
|
return
|
||||||
|
|
||||||
|
# Compare current turn keywords against running episode keywords
|
||||||
|
episode_keywords = current.get("keywords", [])
|
||||||
|
similarity = jaccard_similarity(turn_keywords, episode_keywords)
|
||||||
|
messages_seen = current.get("messages_seen", 0)
|
||||||
|
|
||||||
|
if similarity < TOPIC_SHIFT_THRESHOLD and messages_seen >= MIN_MESSAGES_FOR_COMPLETION:
|
||||||
|
# Topic shift detected -- complete the current episode
|
||||||
|
completed_episode = {
|
||||||
|
"id": current.get("id", next_episode_id(store["episodes"])),
|
||||||
|
"started": current.get("started", now_iso()),
|
||||||
|
"ended": now_iso(),
|
||||||
|
"keywords": episode_keywords,
|
||||||
|
"summary": generate_summary(current),
|
||||||
|
"message_count": messages_seen,
|
||||||
|
"status": "completed",
|
||||||
|
}
|
||||||
|
store["episodes"].append(completed_episode)
|
||||||
|
store["episodes"] = evict_oldest_episodes(store["episodes"])
|
||||||
|
|
||||||
|
# Start a new episode with current turn's keywords
|
||||||
|
new_id = next_episode_id(store["episodes"])
|
||||||
|
first_msg = extract_first_user_message(messages)
|
||||||
|
store["current_episode"] = {
|
||||||
|
"id": new_id,
|
||||||
|
"started": now_iso(),
|
||||||
|
"keywords": turn_keywords,
|
||||||
|
"messages_seen": 1,
|
||||||
|
"last_keywords": turn_keywords,
|
||||||
|
"first_user_message": first_msg,
|
||||||
|
"latest_user_message": first_msg,
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
# No topic shift -- update the current episode
|
||||||
|
existing_kw_set = set(episode_keywords)
|
||||||
|
merged_keywords = list(episode_keywords)
|
||||||
|
for kw in turn_keywords:
|
||||||
|
if kw not in existing_kw_set:
|
||||||
|
existing_kw_set.add(kw)
|
||||||
|
merged_keywords.append(kw)
|
||||||
|
|
||||||
|
current["keywords"] = merged_keywords
|
||||||
|
current["messages_seen"] = messages_seen + 1
|
||||||
|
current["last_keywords"] = turn_keywords
|
||||||
|
|
||||||
|
# Track user messages for summary generation
|
||||||
|
latest_msg = extract_latest_user_message(messages)
|
||||||
|
if latest_msg:
|
||||||
|
current["latest_user_message"] = latest_msg
|
||||||
|
if not current.get("first_user_message"):
|
||||||
|
current["first_user_message"] = latest_msg
|
||||||
|
|
||||||
|
store["current_episode"] = current
|
||||||
|
|
||||||
|
save_episode_store(agent_id, store)
|
||||||
|
print(json.dumps({"type": "ok"}))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,144 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Episodic memory ingest hook.
|
||||||
|
|
||||||
|
Loads the episode store for the current agent, scores completed episodes
|
||||||
|
against the incoming message's keywords using keyword overlap ratio, and
|
||||||
|
returns the top 2 matching episodes as contextual memories.
|
||||||
|
|
||||||
|
This hook is read-only -- it never modifies the episode store.
|
||||||
|
|
||||||
|
Receives via stdin:
|
||||||
|
{"type": "ingest", "agent_id": "...", "message": "user message text"}
|
||||||
|
|
||||||
|
Prints to stdout:
|
||||||
|
{"type": "ingest_result", "memories": [{"content": "..."}]}
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Stopwords & keyword extraction
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
STOPWORDS = frozenset({
|
||||||
|
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
|
||||||
|
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
|
||||||
|
"being", "have", "has", "had", "do", "does", "did", "will", "would",
|
||||||
|
"could", "should", "may", "might", "shall", "can", "need", "must",
|
||||||
|
"it", "its", "i", "me", "my", "you", "your", "he", "she", "we",
|
||||||
|
"they", "them", "their", "this", "that", "these", "those", "what",
|
||||||
|
"which", "who", "how", "when", "where", "why", "if", "then", "so",
|
||||||
|
"not", "no", "just", "also", "very", "too", "about", "up", "out",
|
||||||
|
"all", "some", "any", "each", "every", "into", "over", "after",
|
||||||
|
})
|
||||||
|
|
||||||
|
MIN_WORD_LEN = 3
|
||||||
|
|
||||||
|
# Minimum overlap ratio to consider an episode relevant
|
||||||
|
MIN_OVERLAP = 0.2
|
||||||
|
|
||||||
|
# Maximum episodes to return
|
||||||
|
MAX_RESULTS = 2
|
||||||
|
|
||||||
|
|
||||||
|
def extract_keywords(text):
|
||||||
|
"""Extract deduplicated keywords from text.
|
||||||
|
|
||||||
|
Lowercases, removes stopwords, filters words shorter than MIN_WORD_LEN.
|
||||||
|
"""
|
||||||
|
words = re.findall(r"[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9]|[a-zA-Z]", text)
|
||||||
|
seen = set()
|
||||||
|
keywords = []
|
||||||
|
for w in words:
|
||||||
|
lower = w.lower()
|
||||||
|
if lower not in STOPWORDS and len(lower) >= MIN_WORD_LEN and lower not in seen:
|
||||||
|
seen.add(lower)
|
||||||
|
keywords.append(lower)
|
||||||
|
return keywords
|
||||||
|
|
||||||
|
|
||||||
|
def load_episode_store(agent_id):
|
||||||
|
"""Load the episode store JSON for the given agent. Returns None on failure."""
|
||||||
|
store_dir = os.path.join(
|
||||||
|
os.path.expanduser("~"), ".librefang", "plugins", "episodic-memory"
|
||||||
|
)
|
||||||
|
store_path = os.path.join(store_dir, f"{agent_id}.json")
|
||||||
|
if not os.path.isfile(store_path):
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
with open(store_path, "r", encoding="utf-8") as f:
|
||||||
|
return json.load(f)
|
||||||
|
except (json.JSONDecodeError, OSError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def score_episode(episode_keywords, query_keywords):
|
||||||
|
"""Compute keyword overlap ratio between an episode and the query.
|
||||||
|
|
||||||
|
overlap_ratio = |intersection| / |union| (Jaccard similarity)
|
||||||
|
"""
|
||||||
|
if not episode_keywords or not query_keywords:
|
||||||
|
return 0.0
|
||||||
|
ep_set = set(episode_keywords)
|
||||||
|
q_set = set(query_keywords)
|
||||||
|
intersection = ep_set & q_set
|
||||||
|
union = ep_set | q_set
|
||||||
|
if not union:
|
||||||
|
return 0.0
|
||||||
|
return len(intersection) / len(union)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
try:
|
||||||
|
request = json.loads(sys.stdin.read())
|
||||||
|
except (json.JSONDecodeError, ValueError):
|
||||||
|
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||||
|
return
|
||||||
|
|
||||||
|
message = request.get("message", "")
|
||||||
|
agent_id = request.get("agent_id", "")
|
||||||
|
|
||||||
|
if not message.strip() or not agent_id:
|
||||||
|
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||||
|
return
|
||||||
|
|
||||||
|
query_keywords = extract_keywords(message)
|
||||||
|
if not query_keywords:
|
||||||
|
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||||
|
return
|
||||||
|
|
||||||
|
store = load_episode_store(agent_id)
|
||||||
|
if not store:
|
||||||
|
print(json.dumps({"type": "ingest_result", "memories": []}))
|
||||||
|
return
|
||||||
|
|
||||||
|
episodes = store.get("episodes", [])
|
||||||
|
|
||||||
|
# Score only completed episodes
|
||||||
|
scored = []
|
||||||
|
for ep in episodes:
|
||||||
|
if ep.get("status") != "completed":
|
||||||
|
continue
|
||||||
|
overlap = score_episode(ep.get("keywords", []), query_keywords)
|
||||||
|
if overlap > MIN_OVERLAP:
|
||||||
|
scored.append((overlap, ep))
|
||||||
|
|
||||||
|
# Sort by overlap descending, take top MAX_RESULTS
|
||||||
|
scored.sort(key=lambda x: x[0], reverse=True)
|
||||||
|
top = scored[:MAX_RESULTS]
|
||||||
|
|
||||||
|
memories = []
|
||||||
|
for _score, ep in top:
|
||||||
|
timestamp = ep.get("ended", ep.get("started", "unknown"))
|
||||||
|
summary = ep.get("summary", "no summary")
|
||||||
|
memories.append({
|
||||||
|
"content": f"[episodic-memory] Past episode ({timestamp}): {summary}"
|
||||||
|
})
|
||||||
|
|
||||||
|
print(json.dumps({"type": "ingest_result", "memories": memories}))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
name = "episodic-memory"
|
||||||
|
version = "0.1.0"
|
||||||
|
description = "Episode-based memory segmentation and recall for cross-conversation context continuity"
|
||||||
|
author = "librefang"
|
||||||
|
|
||||||
|
[hooks]
|
||||||
|
ingest = "hooks/ingest.py"
|
||||||
|
after_turn = "hooks/after_turn.py"
|
||||||
Whitespace-only changes.
@@ -0,0 +1,26 @@
|
|||||||
|
# topic-memory
|
||||||
|
|
||||||
|
Topic-aware memory recall using keyword clustering. Tracks conversation topics per agent and recalls related context when similar topics arise in future conversations.
|
||||||
|
|
||||||
|
## How it works
|
||||||
|
|
||||||
|
**After each turn**, the plugin extracts keywords from user and assistant messages, then either merges them into an existing topic cluster (Jaccard similarity > 0.3) or creates a new one. Each cluster stores a keyword set, a summary, a hit count, and a last-seen timestamp.
|
||||||
|
|
||||||
|
**On ingest**, the plugin scores all stored topic clusters against the incoming message keywords using Jaccard similarity and returns the top 3 matches (threshold > 0.15) as contextual memories.
|
||||||
|
|
||||||
|
This gives agents cross-conversation topic awareness -- if a user discussed Python async patterns last week, bringing up `asyncio` today will recall that context.
|
||||||
|
|
||||||
|
## Hooks
|
||||||
|
|
||||||
|
| Hook | Script | Description |
|
||||||
|
|------|--------|-------------|
|
||||||
|
| ingest | `hooks/ingest.py` | Scores stored topics against message keywords, returns top matches |
|
||||||
|
| after_turn | `hooks/after_turn.py` | Extracts keywords, merges or creates topic clusters |
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
|
||||||
|
Topic clusters are stored at `~/.librefang/plugins/topic-memory/{agent_id}.json`. Max 50 clusters per agent (lowest hit-count evicted when full).
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
Installed automatically when enabled in agent configuration.
|
||||||
@@ -0,0 +1,226 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Topic-memory after_turn hook.
|
||||||
|
|
||||||
|
After each conversation turn, extracts keywords from the latest exchange,
|
||||||
|
then either merges into an existing topic cluster or creates a new one.
|
||||||
|
|
||||||
|
This hook is WRITE-ONLY -- it never returns memories.
|
||||||
|
|
||||||
|
Receives via stdin:
|
||||||
|
{"type": "after_turn", "agent_id": "...", "messages": [
|
||||||
|
{"role": "user"|"assistant", "content": "..."}
|
||||||
|
]}
|
||||||
|
|
||||||
|
Prints to stdout:
|
||||||
|
{"type": "ok"}
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Stopwords & constants
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
STOPWORDS = frozenset({
|
||||||
|
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
|
||||||
|
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
|
||||||
|
"being", "have", "has", "had", "do", "does", "did", "will", "would",
|
||||||
|
"could", "should", "may", "might", "shall", "can", "need", "must",
|
||||||
|
"it", "its", "i", "me", "my", "you", "your", "he", "she", "we",
|
||||||
|
"they", "them", "their", "this", "that", "these", "those", "what",
|
||||||
|
"which", "who", "how", "when", "where", "why", "if", "then", "so",
|
||||||
|
"not", "no", "just", "also", "very", "too", "about", "up", "out",
|
||||||
|
"all", "some", "any", "each", "every", "into", "over", "after",
|
||||||
|
})
|
||||||
|
|
||||||
|
MIN_WORD_LEN = 3
|
||||||
|
MAX_TOPICS = 50
|
||||||
|
MERGE_THRESHOLD = 0.3
|
||||||
|
SUMMARY_MAX_LEN = 200
|
||||||
|
|
||||||
|
STORE_DIR = os.path.join(os.path.expanduser("~"), ".librefang", "plugins", "topic-memory")
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def extract_keywords(text):
|
||||||
|
"""Extract lowercase keywords from text, filtering stopwords and short tokens."""
|
||||||
|
words = re.findall(r"[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9]|[a-zA-Z]", text)
|
||||||
|
seen = set()
|
||||||
|
keywords = set()
|
||||||
|
for w in words:
|
||||||
|
lower = w.lower()
|
||||||
|
if lower not in STOPWORDS and len(lower) >= MIN_WORD_LEN and lower not in seen:
|
||||||
|
seen.add(lower)
|
||||||
|
keywords.add(lower)
|
||||||
|
return keywords
|
||||||
|
|
||||||
|
|
||||||
|
def jaccard_similarity(set_a, set_b):
|
||||||
|
"""Compute Jaccard similarity between two sets."""
|
||||||
|
if not set_a or not set_b:
|
||||||
|
return 0.0
|
||||||
|
intersection = len(set_a & set_b)
|
||||||
|
union = len(set_a | set_b)
|
||||||
|
if union == 0:
|
||||||
|
return 0.0
|
||||||
|
return intersection / union
|
||||||
|
|
||||||
|
|
||||||
|
def load_store(agent_id):
|
||||||
|
"""Load the topic store JSON for an agent. Returns empty structure on any error."""
|
||||||
|
path = os.path.join(STORE_DIR, f"{agent_id}.json")
|
||||||
|
try:
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
if isinstance(data, dict) and isinstance(data.get("topics"), list):
|
||||||
|
return data
|
||||||
|
except (OSError, json.JSONDecodeError, ValueError):
|
||||||
|
pass
|
||||||
|
return {"topics": []}
|
||||||
|
|
||||||
|
|
||||||
|
def save_store(agent_id, store):
|
||||||
|
"""Persist the topic store to disk."""
|
||||||
|
os.makedirs(STORE_DIR, exist_ok=True)
|
||||||
|
path = os.path.join(STORE_DIR, f"{agent_id}.json")
|
||||||
|
tmp_path = path + ".tmp"
|
||||||
|
with open(tmp_path, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(store, f, ensure_ascii=False, indent=2)
|
||||||
|
os.replace(tmp_path, path)
|
||||||
|
|
||||||
|
|
||||||
|
def next_topic_id(topics):
|
||||||
|
"""Generate the next t_XXX topic ID."""
|
||||||
|
max_num = 0
|
||||||
|
for t in topics:
|
||||||
|
tid = t.get("id", "")
|
||||||
|
if tid.startswith("t_"):
|
||||||
|
try:
|
||||||
|
num = int(tid[2:])
|
||||||
|
if num > max_num:
|
||||||
|
max_num = num
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
return f"t_{max_num + 1:03d}"
|
||||||
|
|
||||||
|
|
||||||
|
def build_summary(user_content, assistant_content):
|
||||||
|
"""Build a truncated summary from the latest user + assistant exchange."""
|
||||||
|
parts = []
|
||||||
|
if user_content:
|
||||||
|
parts.append(f"User: {user_content.strip()}")
|
||||||
|
if assistant_content:
|
||||||
|
parts.append(f"Assistant: {assistant_content.strip()}")
|
||||||
|
raw = " | ".join(parts)
|
||||||
|
if len(raw) > SUMMARY_MAX_LEN:
|
||||||
|
return raw[: SUMMARY_MAX_LEN - 3] + "..."
|
||||||
|
return raw
|
||||||
|
|
||||||
|
|
||||||
|
def now_iso():
|
||||||
|
"""Return current UTC time as ISO 8601 string."""
|
||||||
|
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
|
||||||
|
|
||||||
|
def ok_result():
|
||||||
|
"""Return the standard ok response."""
|
||||||
|
return json.dumps({"type": "ok"})
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
try:
|
||||||
|
request = json.loads(sys.stdin.read())
|
||||||
|
except (json.JSONDecodeError, ValueError):
|
||||||
|
print(ok_result())
|
||||||
|
return
|
||||||
|
|
||||||
|
agent_id = request.get("agent_id", "")
|
||||||
|
messages = request.get("messages", [])
|
||||||
|
|
||||||
|
if not agent_id or not isinstance(messages, list) or not messages:
|
||||||
|
print(ok_result())
|
||||||
|
return
|
||||||
|
|
||||||
|
# Extract the latest user and assistant messages
|
||||||
|
latest_user = ""
|
||||||
|
latest_assistant = ""
|
||||||
|
for msg in reversed(messages):
|
||||||
|
role = msg.get("role", "")
|
||||||
|
content = msg.get("content", "")
|
||||||
|
if role == "assistant" and not latest_assistant:
|
||||||
|
latest_assistant = content
|
||||||
|
elif role == "user" and not latest_user:
|
||||||
|
latest_user = content
|
||||||
|
if latest_user and latest_assistant:
|
||||||
|
break
|
||||||
|
|
||||||
|
if not latest_user and not latest_assistant:
|
||||||
|
print(ok_result())
|
||||||
|
return
|
||||||
|
|
||||||
|
# Extract keywords from the combined exchange
|
||||||
|
combined_text = f"{latest_user} {latest_assistant}"
|
||||||
|
current_keywords = extract_keywords(combined_text)
|
||||||
|
|
||||||
|
if not current_keywords:
|
||||||
|
print(ok_result())
|
||||||
|
return
|
||||||
|
|
||||||
|
store = load_store(agent_id)
|
||||||
|
topics = store.get("topics", [])
|
||||||
|
timestamp = now_iso()
|
||||||
|
|
||||||
|
# Find the best matching existing topic cluster
|
||||||
|
best_sim = 0.0
|
||||||
|
best_idx = -1
|
||||||
|
for idx, topic in enumerate(topics):
|
||||||
|
topic_kw = set(topic.get("keywords", []))
|
||||||
|
sim = jaccard_similarity(current_keywords, topic_kw)
|
||||||
|
if sim > best_sim:
|
||||||
|
best_sim = sim
|
||||||
|
best_idx = idx
|
||||||
|
|
||||||
|
if best_sim >= MERGE_THRESHOLD and best_idx >= 0:
|
||||||
|
# Merge into existing topic cluster
|
||||||
|
topic = topics[best_idx]
|
||||||
|
existing_kw = set(topic.get("keywords", []))
|
||||||
|
merged_kw = existing_kw | current_keywords
|
||||||
|
topic["keywords"] = sorted(merged_kw)
|
||||||
|
topic["summary"] = build_summary(latest_user, latest_assistant)
|
||||||
|
topic["last_seen"] = timestamp
|
||||||
|
topic["hit_count"] = topic.get("hit_count", 0) + 1
|
||||||
|
else:
|
||||||
|
# Create a new topic cluster
|
||||||
|
new_topic = {
|
||||||
|
"id": next_topic_id(topics),
|
||||||
|
"keywords": sorted(current_keywords),
|
||||||
|
"summary": build_summary(latest_user, latest_assistant),
|
||||||
|
"last_seen": timestamp,
|
||||||
|
"hit_count": 1,
|
||||||
|
}
|
||||||
|
topics.append(new_topic)
|
||||||
|
|
||||||
|
# Evict lowest hit_count topics if over capacity
|
||||||
|
if len(topics) > MAX_TOPICS:
|
||||||
|
topics.sort(key=lambda t: (t.get("hit_count", 0), t.get("last_seen", "")))
|
||||||
|
topics = topics[len(topics) - MAX_TOPICS:]
|
||||||
|
|
||||||
|
store["topics"] = topics
|
||||||
|
save_store(agent_id, store)
|
||||||
|
|
||||||
|
print(ok_result())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,139 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Topic-memory ingest hook.
|
||||||
|
|
||||||
|
Reads the topic store for the given agent and returns the top matching
|
||||||
|
topic summaries based on Jaccard similarity between the incoming message
|
||||||
|
keywords and each stored topic cluster.
|
||||||
|
|
||||||
|
This hook is READ-ONLY -- it never modifies the topic store.
|
||||||
|
|
||||||
|
Receives via stdin:
|
||||||
|
{"type": "ingest", "agent_id": "...", "message": "user message text"}
|
||||||
|
|
||||||
|
Prints to stdout:
|
||||||
|
{"type": "ingest_result", "memories": [{"content": "..."}]}
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Stopwords & constants
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
STOPWORDS = frozenset({
|
||||||
|
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
|
||||||
|
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
|
||||||
|
"being", "have", "has", "had", "do", "does", "did", "will", "would",
|
||||||
|
"could", "should", "may", "might", "shall", "can", "need", "must",
|
||||||
|
"it", "its", "i", "me", "my", "you", "your", "he", "she", "we",
|
||||||
|
"they", "them", "their", "this", "that", "these", "those", "what",
|
||||||
|
"which", "who", "how", "when", "where", "why", "if", "then", "so",
|
||||||
|
"not", "no", "just", "also", "very", "too", "about", "up", "out",
|
||||||
|
"all", "some", "any", "each", "every", "into", "over", "after",
|
||||||
|
})
|
||||||
|
|
||||||
|
MIN_WORD_LEN = 3
|
||||||
|
MAX_RESULTS = 3
|
||||||
|
MIN_SIMILARITY = 0.15
|
||||||
|
|
||||||
|
STORE_DIR = os.path.join(os.path.expanduser("~"), ".librefang", "plugins", "topic-memory")
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def extract_keywords(text):
|
||||||
|
"""Extract lowercase keywords from text, filtering stopwords and short tokens."""
|
||||||
|
words = re.findall(r"[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9]|[a-zA-Z]", text)
|
||||||
|
seen = set()
|
||||||
|
keywords = set()
|
||||||
|
for w in words:
|
||||||
|
lower = w.lower()
|
||||||
|
if lower not in STOPWORDS and len(lower) >= MIN_WORD_LEN and lower not in seen:
|
||||||
|
seen.add(lower)
|
||||||
|
keywords.add(lower)
|
||||||
|
return keywords
|
||||||
|
|
||||||
|
|
||||||
|
def jaccard_similarity(set_a, set_b):
|
||||||
|
"""Compute Jaccard similarity between two sets."""
|
||||||
|
if not set_a or not set_b:
|
||||||
|
return 0.0
|
||||||
|
intersection = len(set_a & set_b)
|
||||||
|
union = len(set_a | set_b)
|
||||||
|
if union == 0:
|
||||||
|
return 0.0
|
||||||
|
return intersection / union
|
||||||
|
|
||||||
|
|
||||||
|
def load_store(agent_id):
|
||||||
|
"""Load the topic store JSON for an agent. Returns empty structure on any error."""
|
||||||
|
path = os.path.join(STORE_DIR, f"{agent_id}.json")
|
||||||
|
try:
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
if isinstance(data, dict) and isinstance(data.get("topics"), list):
|
||||||
|
return data
|
||||||
|
except (OSError, json.JSONDecodeError, ValueError):
|
||||||
|
pass
|
||||||
|
return {"topics": []}
|
||||||
|
|
||||||
|
|
||||||
|
def empty_result():
|
||||||
|
"""Return an empty ingest result."""
|
||||||
|
return json.dumps({"type": "ingest_result", "memories": []})
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
try:
|
||||||
|
request = json.loads(sys.stdin.read())
|
||||||
|
except (json.JSONDecodeError, ValueError):
|
||||||
|
print(empty_result())
|
||||||
|
return
|
||||||
|
|
||||||
|
message = request.get("message", "")
|
||||||
|
agent_id = request.get("agent_id", "")
|
||||||
|
|
||||||
|
if not message.strip() or not agent_id:
|
||||||
|
print(empty_result())
|
||||||
|
return
|
||||||
|
|
||||||
|
msg_keywords = extract_keywords(message)
|
||||||
|
if not msg_keywords:
|
||||||
|
print(empty_result())
|
||||||
|
return
|
||||||
|
|
||||||
|
store = load_store(agent_id)
|
||||||
|
topics = store.get("topics", [])
|
||||||
|
|
||||||
|
# Score each topic cluster against the current message keywords
|
||||||
|
scored = []
|
||||||
|
for topic in topics:
|
||||||
|
topic_kw = set(topic.get("keywords", []))
|
||||||
|
sim = jaccard_similarity(msg_keywords, topic_kw)
|
||||||
|
if sim >= MIN_SIMILARITY:
|
||||||
|
scored.append((sim, topic))
|
||||||
|
|
||||||
|
# Sort descending by similarity, take top N
|
||||||
|
scored.sort(key=lambda x: x[0], reverse=True)
|
||||||
|
top = scored[:MAX_RESULTS]
|
||||||
|
|
||||||
|
memories = []
|
||||||
|
for _sim, topic in top:
|
||||||
|
summary = topic.get("summary", "")
|
||||||
|
if summary:
|
||||||
|
memories.append({"content": f"[topic-memory] Related context: {summary}"})
|
||||||
|
|
||||||
|
print(json.dumps({"type": "ingest_result", "memories": memories}))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
name = "topic-memory"
|
||||||
|
version = "0.1.0"
|
||||||
|
description = "Topic-aware memory recall with keyword clustering for cross-conversation context"
|
||||||
|
author = "librefang"
|
||||||
|
|
||||||
|
[hooks]
|
||||||
|
ingest = "hooks/ingest.py"
|
||||||
|
after_turn = "hooks/after_turn.py"
|
||||||
Whitespace-only changes.
@@ -0,0 +1,29 @@
|
|||||||
|
# user-profile
|
||||||
|
|
||||||
|
Persistent user profiling from conversation patterns. Builds a profile of user expertise areas, communication style, and technical level, then injects it as context so agents can personalize responses.
|
||||||
|
|
||||||
|
## How it works
|
||||||
|
|
||||||
|
**After each turn**, the plugin analyzes user messages to update the profile:
|
||||||
|
|
||||||
|
- **Expertise areas** -- extracts technical keywords and tracks frequency (top 20 retained)
|
||||||
|
- **Communication style** -- running average of message lengths (brief / moderate / detailed)
|
||||||
|
- **Technical level** -- scored from signals like code blocks, version numbers, tech abbreviations, and question patterns (beginner / intermediate / advanced)
|
||||||
|
- **Question ratio** -- fraction of user messages containing questions
|
||||||
|
|
||||||
|
**On ingest**, once the profile has at least 5 interactions, the plugin returns a compact profile summary as a memory fragment: `expertise=python,devops; style=detailed; level=advanced`.
|
||||||
|
|
||||||
|
## Hooks
|
||||||
|
|
||||||
|
| Hook | Script | Description |
|
||||||
|
|------|--------|-------------|
|
||||||
|
| ingest | `hooks/ingest.py` | Returns the profile summary as a memory fragment (after 5+ interactions) |
|
||||||
|
| after_turn | `hooks/after_turn.py` | Analyzes user messages to update the profile |
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
|
||||||
|
Profiles are stored at `~/.librefang/plugins/user-profile/{agent_id}.json`.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
Installed automatically when enabled in agent configuration.
|
||||||
@@ -0,0 +1,282 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""User-profile after_turn hook.
|
||||||
|
|
||||||
|
Analyses user messages from the completed turn and updates the persisted
|
||||||
|
user profile with extracted signals: expertise areas, message length
|
||||||
|
statistics, question ratio, and inferred technical level.
|
||||||
|
|
||||||
|
This hook is WRITE-ONLY -- it updates the profile store but never returns
|
||||||
|
memories.
|
||||||
|
|
||||||
|
Receives via stdin:
|
||||||
|
{"type": "after_turn", "agent_id": "...", "messages": [...]}
|
||||||
|
|
||||||
|
Each message: {"role": "user"|"assistant", "content": "..."}
|
||||||
|
|
||||||
|
Prints to stdout:
|
||||||
|
{"type": "ok"}
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Constants
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
STORE_DIR = os.path.join(
|
||||||
|
os.path.expanduser("~"), ".librefang", "plugins", "user-profile"
|
||||||
|
)
|
||||||
|
|
||||||
|
MAX_EXPERTISE_ENTRIES = 20
|
||||||
|
MIN_KEYWORD_LEN = 3
|
||||||
|
|
||||||
|
# Technical abbreviations that signal intermediate+ level
|
||||||
|
TECH_ABBREVIATIONS = frozenset({
|
||||||
|
"api", "cli", "sdk", "orm", "sql", "css", "html", "http", "https",
|
||||||
|
"jwt", "oauth", "ssr", "csr", "dom", "cdn", "dns", "tcp", "udp",
|
||||||
|
"grpc", "wasm", "yaml", "toml", "json", "xml", "cicd", "gpu",
|
||||||
|
"cpu", "ram", "ssd", "tls", "ssh", "llm", "rag", "mlops", "etl",
|
||||||
|
"crud", "rest", "graphql", "ide", "vcs", "iot", "saas", "paas",
|
||||||
|
})
|
||||||
|
|
||||||
|
STOPWORDS = frozenset({
|
||||||
|
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
|
||||||
|
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
|
||||||
|
"being", "have", "has", "had", "do", "does", "did", "will", "would",
|
||||||
|
"could", "should", "may", "might", "shall", "can", "need", "must",
|
||||||
|
"it", "its", "i", "me", "my", "you", "your", "he", "she", "we",
|
||||||
|
"they", "them", "their", "this", "that", "these", "those", "what",
|
||||||
|
"which", "who", "how", "when", "where", "why", "if", "then", "so",
|
||||||
|
"not", "no", "just", "also", "very", "too", "about", "up", "out",
|
||||||
|
"all", "some", "any", "each", "every", "into", "over", "after",
|
||||||
|
"been", "before", "between", "both", "down", "during", "few", "get",
|
||||||
|
"got", "here", "him", "his", "her", "like", "make", "many", "more",
|
||||||
|
"most", "much", "new", "now", "old", "one", "only", "other", "our",
|
||||||
|
"own", "same", "say", "see", "still", "such", "take", "than",
|
||||||
|
"there", "thing", "think", "time", "use", "used", "using", "want",
|
||||||
|
"way", "well", "work", "know", "really", "right", "going", "back",
|
||||||
|
})
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def ok_result():
|
||||||
|
"""Return an ok result."""
|
||||||
|
return json.dumps({"type": "ok"})
|
||||||
|
|
||||||
|
|
||||||
|
def load_profile(agent_id):
|
||||||
|
"""Load the profile JSON for an agent. Returns default structure on any error."""
|
||||||
|
path = os.path.join(STORE_DIR, f"{agent_id}.json")
|
||||||
|
try:
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
if isinstance(data, dict) and isinstance(data.get("interaction_count"), int):
|
||||||
|
return data
|
||||||
|
except (OSError, json.JSONDecodeError, ValueError):
|
||||||
|
pass
|
||||||
|
return {
|
||||||
|
"interaction_count": 0,
|
||||||
|
"expertise_areas": {},
|
||||||
|
"avg_message_length": 0.0,
|
||||||
|
"question_ratio": 0.0,
|
||||||
|
"technical_level": "beginner",
|
||||||
|
"last_updated": "",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def save_profile(agent_id, profile):
|
||||||
|
"""Persist profile to disk."""
|
||||||
|
os.makedirs(STORE_DIR, exist_ok=True)
|
||||||
|
path = os.path.join(STORE_DIR, f"{agent_id}.json")
|
||||||
|
try:
|
||||||
|
with open(path, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(profile, f, indent=2, ensure_ascii=False)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def extract_keywords(text):
|
||||||
|
"""Extract meaningful keywords from text, filtering stopwords."""
|
||||||
|
# Handle hyphenated compound terms and regular words
|
||||||
|
words = re.findall(r"[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9]|[a-zA-Z]", text)
|
||||||
|
keywords = []
|
||||||
|
seen = set()
|
||||||
|
for w in words:
|
||||||
|
lower = w.lower()
|
||||||
|
if lower not in STOPWORDS and len(lower) >= MIN_KEYWORD_LEN and lower not in seen:
|
||||||
|
seen.add(lower)
|
||||||
|
keywords.append(lower)
|
||||||
|
return keywords
|
||||||
|
|
||||||
|
|
||||||
|
def has_code_blocks(text):
|
||||||
|
"""Check if text contains code blocks or backtick references."""
|
||||||
|
return bool(re.search(r"```|`[^`]+`", text))
|
||||||
|
|
||||||
|
|
||||||
|
def has_version_numbers(text):
|
||||||
|
"""Check if text contains version references like v3.2, Python 3.12, etc."""
|
||||||
|
return bool(re.search(r"v\d+\.\d+|(?<!\w)\d+\.\d+\.\d+", text))
|
||||||
|
|
||||||
|
|
||||||
|
def has_tech_abbreviations(text):
|
||||||
|
"""Check if text contains known technical abbreviations."""
|
||||||
|
words = set(re.findall(r"\b[a-zA-Z]{2,6}\b", text))
|
||||||
|
lower_words = {w.lower() for w in words}
|
||||||
|
return bool(lower_words & TECH_ABBREVIATIONS)
|
||||||
|
|
||||||
|
|
||||||
|
def has_basic_questions(text):
|
||||||
|
"""Check if text contains beginner-style 'what is' / 'explain' patterns."""
|
||||||
|
lower = text.lower()
|
||||||
|
return bool(re.search(r"\bwhat\s+is\b|\bexplain\b|\bwhat\s+are\b", lower))
|
||||||
|
|
||||||
|
|
||||||
|
def average_word_length(text):
|
||||||
|
"""Compute average word length in the text."""
|
||||||
|
words = re.findall(r"[a-zA-Z]+", text)
|
||||||
|
if not words:
|
||||||
|
return 0.0
|
||||||
|
return sum(len(w) for w in words) / len(words)
|
||||||
|
|
||||||
|
|
||||||
|
def infer_technical_level(text):
|
||||||
|
"""Infer technical level from a single message. Returns a score.
|
||||||
|
|
||||||
|
Score >= 3 -> "advanced"
|
||||||
|
Score 1-2 -> "intermediate"
|
||||||
|
Score <= 0 -> "beginner"
|
||||||
|
"""
|
||||||
|
score = 0
|
||||||
|
|
||||||
|
if has_code_blocks(text):
|
||||||
|
score += 2
|
||||||
|
if has_version_numbers(text):
|
||||||
|
score += 1
|
||||||
|
if has_tech_abbreviations(text):
|
||||||
|
score += 1
|
||||||
|
if has_basic_questions(text):
|
||||||
|
score -= 1
|
||||||
|
if average_word_length(text) > 5.5:
|
||||||
|
score += 1
|
||||||
|
|
||||||
|
return score
|
||||||
|
|
||||||
|
|
||||||
|
def tech_level_from_score(score):
|
||||||
|
"""Map a numeric score to a technical level label."""
|
||||||
|
if score >= 3:
|
||||||
|
return "advanced"
|
||||||
|
elif score >= 1:
|
||||||
|
return "intermediate"
|
||||||
|
else:
|
||||||
|
return "beginner"
|
||||||
|
|
||||||
|
|
||||||
|
def prune_expertise(expertise, max_entries):
|
||||||
|
"""Keep only the top max_entries expertise areas by count."""
|
||||||
|
if len(expertise) <= max_entries:
|
||||||
|
return expertise
|
||||||
|
sorted_items = sorted(expertise.items(), key=lambda x: x[1], reverse=True)
|
||||||
|
return dict(sorted_items[:max_entries])
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
try:
|
||||||
|
request = json.loads(sys.stdin.read())
|
||||||
|
except (json.JSONDecodeError, ValueError):
|
||||||
|
print(ok_result())
|
||||||
|
return
|
||||||
|
|
||||||
|
agent_id = request.get("agent_id", "")
|
||||||
|
messages = request.get("messages", [])
|
||||||
|
|
||||||
|
if not agent_id or not isinstance(messages, list):
|
||||||
|
print(ok_result())
|
||||||
|
return
|
||||||
|
|
||||||
|
# Filter to user messages only
|
||||||
|
user_messages = []
|
||||||
|
for msg in messages:
|
||||||
|
if isinstance(msg, dict) and msg.get("role") == "user":
|
||||||
|
content = msg.get("content", "")
|
||||||
|
if isinstance(content, str) and content.strip():
|
||||||
|
user_messages.append(content)
|
||||||
|
|
||||||
|
if not user_messages:
|
||||||
|
print(ok_result())
|
||||||
|
return
|
||||||
|
|
||||||
|
profile = load_profile(agent_id)
|
||||||
|
|
||||||
|
old_count = profile["interaction_count"]
|
||||||
|
new_count = old_count + len(user_messages)
|
||||||
|
|
||||||
|
# --- Expertise areas ---
|
||||||
|
expertise = profile.get("expertise_areas", {})
|
||||||
|
for msg in user_messages:
|
||||||
|
keywords = extract_keywords(msg)
|
||||||
|
for kw in keywords:
|
||||||
|
expertise[kw] = expertise.get(kw, 0) + 1
|
||||||
|
|
||||||
|
expertise = prune_expertise(expertise, MAX_EXPERTISE_ENTRIES)
|
||||||
|
profile["expertise_areas"] = expertise
|
||||||
|
|
||||||
|
# --- Average message length (running average) ---
|
||||||
|
old_avg = profile.get("avg_message_length", 0.0)
|
||||||
|
total_new_len = sum(len(msg) for msg in user_messages)
|
||||||
|
if old_count == 0:
|
||||||
|
new_avg = total_new_len / len(user_messages)
|
||||||
|
else:
|
||||||
|
# Weighted running average: combine old aggregate with new messages
|
||||||
|
old_total = old_avg * old_count
|
||||||
|
new_avg = (old_total + total_new_len) / new_count
|
||||||
|
profile["avg_message_length"] = round(new_avg, 1)
|
||||||
|
|
||||||
|
# --- Question ratio (running ratio) ---
|
||||||
|
old_ratio = profile.get("question_ratio", 0.0)
|
||||||
|
questions_in_batch = sum(1 for msg in user_messages if "?" in msg)
|
||||||
|
if old_count == 0:
|
||||||
|
new_ratio = questions_in_batch / len(user_messages)
|
||||||
|
else:
|
||||||
|
old_question_count = round(old_ratio * old_count)
|
||||||
|
new_ratio = (old_question_count + questions_in_batch) / new_count
|
||||||
|
profile["question_ratio"] = round(new_ratio, 3)
|
||||||
|
|
||||||
|
# --- Technical level (weighted towards recent) ---
|
||||||
|
total_score = 0
|
||||||
|
for msg in user_messages:
|
||||||
|
total_score += infer_technical_level(msg)
|
||||||
|
avg_score = total_score / len(user_messages)
|
||||||
|
|
||||||
|
# Blend with historical level: map old level to a score, then average
|
||||||
|
level_to_score = {"beginner": 0, "intermediate": 1.5, "advanced": 3}
|
||||||
|
old_level_score = level_to_score.get(profile.get("technical_level", "beginner"), 0)
|
||||||
|
if old_count == 0:
|
||||||
|
blended_score = avg_score
|
||||||
|
else:
|
||||||
|
# Give 70% weight to history, 30% to this batch
|
||||||
|
blended_score = 0.7 * old_level_score + 0.3 * avg_score
|
||||||
|
profile["technical_level"] = tech_level_from_score(blended_score)
|
||||||
|
|
||||||
|
# --- Bookkeeping ---
|
||||||
|
profile["interaction_count"] = new_count
|
||||||
|
profile["last_updated"] = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
|
||||||
|
save_profile(agent_id, profile)
|
||||||
|
print(ok_result())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,134 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""User-profile ingest hook.
|
||||||
|
|
||||||
|
Reads the persisted user profile for the given agent and, when enough
|
||||||
|
interaction data has been collected (>= 5 interactions), returns a
|
||||||
|
compact profile summary as injected memory so the agent can personalise
|
||||||
|
its responses.
|
||||||
|
|
||||||
|
This hook is READ-ONLY -- it never modifies the profile store.
|
||||||
|
|
||||||
|
Receives via stdin:
|
||||||
|
{"type": "ingest", "agent_id": "...", "message": "user message text"}
|
||||||
|
|
||||||
|
Prints to stdout:
|
||||||
|
{"type": "ingest_result", "memories": [{"content": "..."}]}
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Constants
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
STORE_DIR = os.path.join(
|
||||||
|
os.path.expanduser("~"), ".librefang", "plugins", "user-profile"
|
||||||
|
)
|
||||||
|
|
||||||
|
MIN_INTERACTIONS = 5
|
||||||
|
MAX_SUMMARY_LEN = 200
|
||||||
|
TOP_EXPERTISE = 5
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def empty_result():
|
||||||
|
"""Return an empty ingest result."""
|
||||||
|
return json.dumps({"type": "ingest_result", "memories": []})
|
||||||
|
|
||||||
|
|
||||||
|
def load_profile(agent_id):
|
||||||
|
"""Load the profile JSON for an agent. Returns None on any error."""
|
||||||
|
path = os.path.join(STORE_DIR, f"{agent_id}.json")
|
||||||
|
try:
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
if isinstance(data, dict) and isinstance(data.get("interaction_count"), int):
|
||||||
|
return data
|
||||||
|
except (OSError, json.JSONDecodeError, ValueError):
|
||||||
|
pass
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def message_length_bucket(avg_len):
|
||||||
|
"""Classify average message length into a human-readable bucket."""
|
||||||
|
if avg_len < 50:
|
||||||
|
return "brief"
|
||||||
|
elif avg_len <= 200:
|
||||||
|
return "moderate"
|
||||||
|
else:
|
||||||
|
return "detailed"
|
||||||
|
|
||||||
|
|
||||||
|
def build_summary(profile):
|
||||||
|
"""Build a compact profile summary string (max MAX_SUMMARY_LEN chars)."""
|
||||||
|
parts = []
|
||||||
|
|
||||||
|
# Top expertise areas
|
||||||
|
expertise = profile.get("expertise_areas", {})
|
||||||
|
if expertise:
|
||||||
|
sorted_areas = sorted(expertise.items(), key=lambda x: x[1], reverse=True)
|
||||||
|
top = [area for area, _count in sorted_areas[:TOP_EXPERTISE]]
|
||||||
|
parts.append("expertise=" + ",".join(top))
|
||||||
|
|
||||||
|
# Communication style
|
||||||
|
avg_len = profile.get("avg_message_length", 0)
|
||||||
|
parts.append("style=" + message_length_bucket(avg_len))
|
||||||
|
|
||||||
|
# Technical level
|
||||||
|
tech_level = profile.get("technical_level", "")
|
||||||
|
if tech_level:
|
||||||
|
parts.append("level=" + tech_level)
|
||||||
|
|
||||||
|
# Question ratio
|
||||||
|
q_ratio = profile.get("question_ratio", 0.0)
|
||||||
|
if q_ratio > 0.5:
|
||||||
|
parts.append("asks-many-questions")
|
||||||
|
|
||||||
|
summary = "; ".join(parts)
|
||||||
|
if len(summary) > MAX_SUMMARY_LEN:
|
||||||
|
summary = summary[:MAX_SUMMARY_LEN - 3] + "..."
|
||||||
|
return summary
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
try:
|
||||||
|
request = json.loads(sys.stdin.read())
|
||||||
|
except (json.JSONDecodeError, ValueError):
|
||||||
|
print(empty_result())
|
||||||
|
return
|
||||||
|
|
||||||
|
agent_id = request.get("agent_id", "")
|
||||||
|
if not agent_id:
|
||||||
|
print(empty_result())
|
||||||
|
return
|
||||||
|
|
||||||
|
profile = load_profile(agent_id)
|
||||||
|
if profile is None:
|
||||||
|
print(empty_result())
|
||||||
|
return
|
||||||
|
|
||||||
|
interaction_count = profile.get("interaction_count", 0)
|
||||||
|
if interaction_count < MIN_INTERACTIONS:
|
||||||
|
print(empty_result())
|
||||||
|
return
|
||||||
|
|
||||||
|
summary = build_summary(profile)
|
||||||
|
if not summary:
|
||||||
|
print(empty_result())
|
||||||
|
return
|
||||||
|
|
||||||
|
memory = {"content": f"[user-profile] User context: {summary}"}
|
||||||
|
print(json.dumps({"type": "ingest_result", "memories": [memory]}))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
name = "user-profile"
|
||||||
|
version = "0.1.0"
|
||||||
|
description = "Persistent user profiling from conversation patterns for personalized agent responses"
|
||||||
|
author = "librefang"
|
||||||
|
|
||||||
|
[hooks]
|
||||||
|
ingest = "hooks/ingest.py"
|
||||||
|
after_turn = "hooks/after_turn.py"
|
||||||
Whitespace-only changes.
@@ -1,5 +1,6 @@
|
|||||||
# GitHub Copilot — https://github.com/features/copilot
|
# GitHub Copilot — https://github.com/features/copilot
|
||||||
# Models: 2 (free for GitHub Copilot subscribers)
|
# Models: 1 (free for GitHub Copilot subscribers)
|
||||||
|
# Note: copilot/gpt-4 removed — deprecated in favor of GPT-4o
|
||||||
|
|
||||||
[provider]
|
[provider]
|
||||||
id = "github-copilot"
|
id = "github-copilot"
|
||||||
@@ -20,16 +21,3 @@ supports_tools = true
|
|||||||
supports_vision = true
|
supports_vision = true
|
||||||
supports_streaming = true
|
supports_streaming = true
|
||||||
aliases = ["copilot-gpt4o"]
|
aliases = ["copilot-gpt4o"]
|
||||||
|
|
||||||
[[models]]
|
|
||||||
id = "copilot/gpt-4"
|
|
||||||
display_name = "GPT-4 (Copilot)"
|
|
||||||
tier = "frontier"
|
|
||||||
context_window = 128000
|
|
||||||
max_output_tokens = 4096
|
|
||||||
input_cost_per_m = 0.0
|
|
||||||
output_cost_per_m = 0.0
|
|
||||||
supports_tools = true
|
|
||||||
supports_vision = false
|
|
||||||
supports_streaming = true
|
|
||||||
aliases = ["copilot-gpt4"]
|
|
||||||
@@ -1,5 +1,7 @@
|
|||||||
# Kimi for Code — https://api.kimi.com
|
# Kimi for Code — https://api.kimi.com
|
||||||
# Models: 1
|
# Models: 1 (membership-based, not per-token billing)
|
||||||
|
# Note: Free tier has limited weekly quota; paid plans from ~49 RMB/month
|
||||||
|
# Underlying model is Kimi K2.5 (~$0.45/$2.20 per M on standard Moonshot API)
|
||||||
|
|
||||||
[provider]
|
[provider]
|
||||||
id = "kimi_coding"
|
id = "kimi_coding"
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
# MiniMax (China) — https://minimaxi.com
|
# MiniMax (China) — https://minimaxi.com
|
||||||
# Models: 6 (same models as international, different endpoint)
|
# Models: 8 (same models as international, different endpoint)
|
||||||
|
|
||||||
[provider]
|
[provider]
|
||||||
id = "minimax-cn"
|
id = "minimax-cn"
|
||||||
@@ -8,6 +8,32 @@ api_key_env = "MINIMAX_CN_API_KEY"
|
|||||||
base_url = "https://api.minimaxi.com/v1"
|
base_url = "https://api.minimaxi.com/v1"
|
||||||
key_required = true
|
key_required = true
|
||||||
|
|
||||||
|
[[models]]
|
||||||
|
id = "MiniMax-M2.7"
|
||||||
|
display_name = "MiniMax M2.7"
|
||||||
|
tier = "frontier"
|
||||||
|
context_window = 204800
|
||||||
|
max_output_tokens = 131072
|
||||||
|
input_cost_per_m = 0.30
|
||||||
|
output_cost_per_m = 1.20
|
||||||
|
supports_tools = true
|
||||||
|
supports_vision = false
|
||||||
|
supports_streaming = true
|
||||||
|
aliases = []
|
||||||
|
|
||||||
|
[[models]]
|
||||||
|
id = "MiniMax-M2.7-highspeed"
|
||||||
|
display_name = "MiniMax M2.7 Highspeed"
|
||||||
|
tier = "smart"
|
||||||
|
context_window = 204800
|
||||||
|
max_output_tokens = 131072
|
||||||
|
input_cost_per_m = 0.30
|
||||||
|
output_cost_per_m = 1.20
|
||||||
|
supports_tools = true
|
||||||
|
supports_vision = false
|
||||||
|
supports_streaming = true
|
||||||
|
aliases = []
|
||||||
|
|
||||||
[[models]]
|
[[models]]
|
||||||
id = "minimax-text-01"
|
id = "minimax-text-01"
|
||||||
display_name = "MiniMax Text 01"
|
display_name = "MiniMax Text 01"
|
||||||
|
|||||||
+27
-1
@@ -1,5 +1,5 @@
|
|||||||
# MiniMax (International) — https://minimax.io
|
# MiniMax (International) — https://minimax.io
|
||||||
# Models: 6
|
# Models: 8
|
||||||
|
|
||||||
[provider]
|
[provider]
|
||||||
id = "minimax"
|
id = "minimax"
|
||||||
@@ -8,6 +8,32 @@ api_key_env = "MINIMAX_API_KEY"
|
|||||||
base_url = "https://api.minimax.io/v1"
|
base_url = "https://api.minimax.io/v1"
|
||||||
key_required = true
|
key_required = true
|
||||||
|
|
||||||
|
[[models]]
|
||||||
|
id = "MiniMax-M2.7"
|
||||||
|
display_name = "MiniMax M2.7"
|
||||||
|
tier = "frontier"
|
||||||
|
context_window = 204800
|
||||||
|
max_output_tokens = 131072
|
||||||
|
input_cost_per_m = 0.30
|
||||||
|
output_cost_per_m = 1.20
|
||||||
|
supports_tools = true
|
||||||
|
supports_vision = false
|
||||||
|
supports_streaming = true
|
||||||
|
aliases = ["minimax-m2.7"]
|
||||||
|
|
||||||
|
[[models]]
|
||||||
|
id = "MiniMax-M2.7-highspeed"
|
||||||
|
display_name = "MiniMax M2.7 Highspeed"
|
||||||
|
tier = "smart"
|
||||||
|
context_window = 204800
|
||||||
|
max_output_tokens = 131072
|
||||||
|
input_cost_per_m = 0.30
|
||||||
|
output_cost_per_m = 1.20
|
||||||
|
supports_tools = true
|
||||||
|
supports_vision = false
|
||||||
|
supports_streaming = true
|
||||||
|
aliases = ["minimax-m2.7-highspeed", "m2.7-highspeed"]
|
||||||
|
|
||||||
[[models]]
|
[[models]]
|
||||||
id = "minimax-text-01"
|
id = "minimax-text-01"
|
||||||
display_name = "MiniMax Text 01"
|
display_name = "MiniMax Text 01"
|
||||||
|
|||||||
+10
-10
@@ -14,8 +14,8 @@ display_name = "Moonshot V1 128K"
|
|||||||
tier = "smart"
|
tier = "smart"
|
||||||
context_window = 131072
|
context_window = 131072
|
||||||
max_output_tokens = 8192
|
max_output_tokens = 8192
|
||||||
input_cost_per_m = 0.80
|
input_cost_per_m = 2.00
|
||||||
output_cost_per_m = 0.80
|
output_cost_per_m = 5.00
|
||||||
supports_tools = true
|
supports_tools = true
|
||||||
supports_vision = false
|
supports_vision = false
|
||||||
supports_streaming = true
|
supports_streaming = true
|
||||||
@@ -27,8 +27,8 @@ display_name = "Moonshot V1 32K"
|
|||||||
tier = "balanced"
|
tier = "balanced"
|
||||||
context_window = 32768
|
context_window = 32768
|
||||||
max_output_tokens = 8192
|
max_output_tokens = 8192
|
||||||
input_cost_per_m = 0.30
|
input_cost_per_m = 1.00
|
||||||
output_cost_per_m = 0.30
|
output_cost_per_m = 3.00
|
||||||
supports_tools = true
|
supports_tools = true
|
||||||
supports_vision = false
|
supports_vision = false
|
||||||
supports_streaming = true
|
supports_streaming = true
|
||||||
@@ -40,8 +40,8 @@ display_name = "Moonshot V1 8K"
|
|||||||
tier = "fast"
|
tier = "fast"
|
||||||
context_window = 8192
|
context_window = 8192
|
||||||
max_output_tokens = 4096
|
max_output_tokens = 4096
|
||||||
input_cost_per_m = 0.10
|
input_cost_per_m = 0.20
|
||||||
output_cost_per_m = 0.10
|
output_cost_per_m = 2.00
|
||||||
supports_tools = true
|
supports_tools = true
|
||||||
supports_vision = false
|
supports_vision = false
|
||||||
supports_streaming = true
|
supports_streaming = true
|
||||||
@@ -53,8 +53,8 @@ display_name = "Kimi K2"
|
|||||||
tier = "frontier"
|
tier = "frontier"
|
||||||
context_window = 131072
|
context_window = 131072
|
||||||
max_output_tokens = 16384
|
max_output_tokens = 16384
|
||||||
input_cost_per_m = 2.00
|
input_cost_per_m = 0.60
|
||||||
output_cost_per_m = 8.00
|
output_cost_per_m = 2.50
|
||||||
supports_tools = true
|
supports_tools = true
|
||||||
supports_vision = true
|
supports_vision = true
|
||||||
supports_streaming = true
|
supports_streaming = true
|
||||||
@@ -66,8 +66,8 @@ display_name = "Kimi K2.5"
|
|||||||
tier = "frontier"
|
tier = "frontier"
|
||||||
context_window = 131072
|
context_window = 131072
|
||||||
max_output_tokens = 16384
|
max_output_tokens = 16384
|
||||||
input_cost_per_m = 2.00
|
input_cost_per_m = 0.45
|
||||||
output_cost_per_m = 8.00
|
output_cost_per_m = 2.20
|
||||||
supports_tools = true
|
supports_tools = true
|
||||||
supports_vision = true
|
supports_vision = true
|
||||||
supports_streaming = true
|
supports_streaming = true
|
||||||
|
|||||||
@@ -118,8 +118,8 @@ display_name = "Gemma 2 9B (OpenRouter)"
|
|||||||
tier = "fast"
|
tier = "fast"
|
||||||
context_window = 8192
|
context_window = 8192
|
||||||
max_output_tokens = 4096
|
max_output_tokens = 4096
|
||||||
input_cost_per_m = 0.0
|
input_cost_per_m = 0.03
|
||||||
output_cost_per_m = 0.0
|
output_cost_per_m = 0.09
|
||||||
supports_tools = false
|
supports_tools = false
|
||||||
supports_vision = false
|
supports_vision = false
|
||||||
supports_streaming = true
|
supports_streaming = true
|
||||||
|
|||||||
@@ -0,0 +1,8 @@
|
|||||||
|
name = "{{NAME}}"
|
||||||
|
version = "0.1.0"
|
||||||
|
description = "TODO: What this plugin does"
|
||||||
|
author = "TODO: your-name"
|
||||||
|
|
||||||
|
[hooks]
|
||||||
|
ingest = "hooks/ingest.py" # Called when user message is received
|
||||||
|
after_turn = "hooks/after_turn.py" # Called after each conversation turn
|
||||||
Reference in new issue
Block a user