librefang-registry/hands/collector/SKILL.md

name, version, description, runtime

name	version	description	runtime
collector-hand-skill	1.0.0	Expert knowledge for AI intelligence collection — OSINT methodology, entity extraction, knowledge graphs, change detection, and sentiment analysis	prompt_only

Intelligence Collection Expert Knowledge

OSINT Methodology

Collection Cycle

1. Planning: Define target, scope, and collection requirements
2. Collection: Gather raw data from open sources
3. Processing: Extract entities, relationships, and data points
4. Analysis: Synthesize findings, identify patterns, detect changes
5. Dissemination: Generate reports, alerts, and updates
6. Feedback: Refine queries based on what worked and what didn't

Source Categories (by reliability)

Tier	Source Type	Reliability	Examples
1	Official/Primary	Very High	Company filings, government data, press releases
2	Institutional	High	News agencies (Reuters, AP), research institutions
3	Professional	Medium-High	Industry publications, analyst reports, expert blogs
4	Community	Medium	Forums, social media, review sites
5	Anonymous/Unverified	Low	Anonymous posts, rumors, unattributed claims

Search Query Construction by Focus Area

Market Intelligence:

"[target] market share"
"[target] industry report [year]"
"[target] TAM SAM SOM"
"[target] growth rate"
"[target] market analysis"
"[target industry] trends [year]"

Business Intelligence:

"[company] revenue" OR "[company] earnings"
"[company] CEO" OR "[company] leadership team"
"[company] strategy" OR "[company] roadmap"
"[company] partnerships" OR "[company] acquisition"
"[company] annual report" OR "[company] 10-K"
site:sec.gov "[company]"

Competitor Analysis:

"[company] vs [competitor]"
"[company] alternative"
"[company] review" OR "[company] comparison"
"[company] pricing" site:g2.com OR site:capterra.com
"[company] customer reviews" site:trustpilot.com
"switch from [company] to"

Person Tracking:

"[person name]" "[company]"
"[person name]" interview OR podcast OR keynote
"[person name]" site:linkedin.com
"[person name]" publication OR paper
"[person name]" conference OR summit

Technology Monitoring:

"[technology] release" OR "[technology] update"
"[technology] benchmark [year]"
"[technology] adoption" OR "[technology] usage statistics"
"[technology] vs [alternative]"
"[technology]" site:github.com
"[technology] roadmap" OR "[technology] changelog"

---

Entity Extraction Patterns

Named Entity Types

1. Person: Name, title, organization, role
2. Organization: Company name, type, industry, location, size
3. Product: Product name, company, category, version
4. Event: Type, date, participants, location, significance
5. Financial: Amount, currency, type (funding, revenue, valuation)
6. Technology: Name, version, category, vendor
7. Location: City, state, country, region
8. Date/Time: Specific dates, time ranges, deadlines

Extraction Heuristics

• Person detection: Title + Name pattern ("CEO John Smith"), bylines, quoted speakers
• Organization detection: Legal suffixes (Inc, LLC), "at [Company]", domain names
• Financial detection: Currency symbols, "raised $X", "valued at", "revenue of"
• Event detection: Date + verb ("launched on", "announced at", "acquired")
• Technology detection: CamelCase names, version numbers, "built with", "powered by"

---

Knowledge Graph Best Practices

Entity Schema

{
  "entity_id": "unique_id",
  "name": "Entity Name",
  "type": "person|company|product|event|technology",
  "attributes": {
    "key": "value"
  },
  "sources": ["url1", "url2"],
  "first_seen": "timestamp",
  "last_seen": "timestamp",
  "confidence": "high|medium|low"
}

Relation Schema

{
  "source_entity": "entity_id_1",
  "relation": "works_at|founded|competes_with|...",
  "target_entity": "entity_id_2",
  "attributes": {
    "since": "date",
    "context": "description"
  },
  "source": "url",
  "confidence": "high|medium|low"
}

Common Relations

Relation	Between	Example
works_at	Person → Company	"Jane Smith works at Acme"
founded	Person → Company	"John Doe founded StartupX"
invested_in	Company → Company	"VC Fund invested in StartupX"
competes_with	Company → Company	"Acme competes with BetaCo"
partnered_with	Company → Company	"Acme partnered with CloudY"
launched	Company → Product	"Acme launched ProductZ"
acquired	Company → Company	"BigCorp acquired StartupX"
uses	Company → Technology	"Acme uses Kubernetes"
mentioned_in	Entity → Source	"Acme mentioned in TechCrunch"

---

Change Detection Methodology

Snapshot Comparison

1. Store the current state of all entities as a JSON snapshot
2. On next collection cycle, compare new state against previous snapshot
3. Classify changes:

Change Type	Significance	Example
Entity appeared	Varies	New competitor enters market
Entity disappeared	Important	Company goes quiet, product deprecated
Attribute changed	Critical-Minor	CEO changed (critical), address changed (minor)
New relation	Important	New partnership, acquisition, hiring
Relation removed	Important	Person left company, partnership ended
Sentiment shift	Important	Positive→Negative media coverage

Significance Scoring

CRITICAL (immediate alert):
  - Leadership change (CEO, CTO, board)
  - Acquisition or merger
  - Major funding round (>$10M)
  - Product discontinuation
  - Legal action or regulatory issue

IMPORTANT (include in next report):
  - New product launch
  - New partnership or integration
  - Hiring surge (>5 roles)
  - Pricing change
  - Competitor move
  - Major customer win/loss

MINOR (note in report):
  - Blog post or press mention
  - Minor update or patch
  - Social media activity spike
  - Conference appearance
  - Job posting (individual)

---

Sentiment Analysis Heuristics

When track_sentiment is enabled, classify each source's tone:

Classification Rules

• Positive indicators: "growth", "innovation", "breakthrough", "success", "award", "expansion", "praise", "recommend"
• Negative indicators: "lawsuit", "layoffs", "decline", "controversy", "failure", "breach", "criticism", "warning"
• Neutral indicators: factual reporting without strong adjectives, data-only articles, announcements

Sentiment Scoring

Strong positive: +2 (e.g., "Company wins major award")
Mild positive:   +1 (e.g., "Steady growth continues")
Neutral:          0 (e.g., "Company releases Q3 report")
Mild negative:   -1 (e.g., "Faces increased competition")
Strong negative: -2 (e.g., "Major data breach disclosed")

Track rolling average over last 5 collection cycles to detect trends.

---

Report Templates

Intelligence Brief (Markdown)

# Intelligence Report: [Target]
**Date**: YYYY-MM-DD HH:MM UTC
**Collection Cycle**: #N
**Sources Processed**: X
**New Data Points**: Y

## Priority Changes
1. [CRITICAL] [Description + source]
2. [IMPORTANT] [Description + source]

## Executive Summary
[2-3 paragraph synthesis of new intelligence]

## Detailed Findings

### [Category 1]
- Finding with [source](url)
- Data point with confidence: high/medium/low

### [Category 2]
- ...

## Entity Updates
| Entity | Change | Previous | Current | Source |
|--------|--------|----------|---------|--------|

## Sentiment Trend
| Period | Score | Direction | Notable |
|--------|-------|-----------|---------|

## Collection Metadata
- Queries executed: N
- Sources fetched: N
- New entities: N
- Updated entities: N
- Next scheduled collection: [datetime]

---

Source Evaluation Checklist

Before including data in the knowledge graph, evaluate:

1. Recency: Published within relevant timeframe? Stale data can mislead.
2. Primary vs Secondary: Is this the original source, or citing someone else?
3. Corroboration: Do other independent sources confirm this?
4. Bias check: Does the source have a financial or political interest in this claim?
5. Specificity: Does it provide concrete data, or vague assertions?
6. Track record: Has this source been reliable in the past?

If a claim fails 3+ checks, downgrade its confidence to "low".

---

Worked Examples

Example 1: Competitor Monitoring Campaign

Scenario: A B2B SaaS company wants continuous intelligence on three direct competitors: AlphaCloud, BetaStack, and GammaSuite.

Step 1 — Define targets and collection requirements

Configure the hand with:

target_subject: "AlphaCloud, BetaStack, GammaSuite"
focus_area: competitor
collection_depth: deep
update_frequency: daily
alert_on_changes: true
track_sentiment: true
max_sources_per_cycle: 50

Build the initial query set:

"AlphaCloud" pricing OR plans OR tiers
"AlphaCloud" product launch OR release OR update
"AlphaCloud" review site:g2.com OR site:capterra.com
"AlphaCloud" customer case study
"AlphaCloud" hiring site:linkedin.com OR site:greenhouse.io
"switch from AlphaCloud to"
(repeat for BetaStack and GammaSuite)

Step 2 — Run first collection cycle

Execute queries, fetch top results, extract entities:

[
  {"type": "product", "name": "AlphaCloud v4.2", "company": "AlphaCloud", "launch_date": "2025-11-15", "source": "alphacloud.com/blog"},
  {"type": "person", "name": "Sarah Chen", "role": "New VP Engineering", "company": "BetaStack", "source": "linkedin.com/in/sarachen"},
  {"type": "event", "name": "GammaSuite Series C", "amount": "$85M", "date": "2025-11-10", "source": "techcrunch.com/2025/11/10/gammasuite-series-c"}
]

Step 3 — Build knowledge graph entries

knowledge_add_entity  type=company  name="AlphaCloud"   industry="SaaS"  funding_stage="Series B"
knowledge_add_entity  type=product  name="AlphaCloud v4.2"  category="cloud platform"
knowledge_add_entity  type=person   name="Sarah Chen"    role="VP Engineering"  company="BetaStack"
knowledge_add_relation source="AlphaCloud" relation="launched" target="AlphaCloud v4.2"
knowledge_add_relation source="Sarah Chen" relation="works_at" target="BetaStack"

Step 4 — Process findings into change detection

Change	Type	Significance	Action
AlphaCloud released v4.2 with AI features	Product launch	IMPORTANT	Include in report, compare against own roadmap
BetaStack hired VP Engineering from FAANG	Leadership change	IMPORTANT	Track subsequent hiring patterns
GammaSuite raised $85M Series C	Major funding	CRITICAL	Immediate alert, expect aggressive expansion

Step 5 — Generate intelligence brief

# Competitor Intelligence Brief
**Date**: 2025-11-16 | **Cycle**: 1 | **Sources**: 47

## Priority Changes
1. [CRITICAL] GammaSuite closed $85M Series C led by Sequoia (TechCrunch, confirmed via Crunchbase)
2. [IMPORTANT] AlphaCloud shipped v4.2 with AI-assisted workflow builder
3. [IMPORTANT] BetaStack hired Sarah Chen (ex-Google) as VP Engineering

## Executive Summary
GammaSuite's large funding round signals intent to accelerate growth — expect increased
marketing spend and possible M&A activity in the next 6 months. AlphaCloud's v4.2
introduces direct feature overlap with our AI pipeline. BetaStack's engineering
leadership hire suggests a product quality push.

## Recommended Actions
- Review AlphaCloud v4.2 feature parity against our roadmap
- Monitor GammaSuite job postings for expansion signals
- Track BetaStack engineering team growth over next 3 cycles

---

Example 2: Technology Landscape Mapping

Scenario: Map the emerging real-time AI inference landscape — track frameworks, adoption signals, key players, and performance benchmarks.

Step 1 — Define scope and seed entities

target_subject: "real-time AI inference (vLLM, TensorRT-LLM, Triton, Ollama, llama.cpp)"
focus_area: technology
collection_depth: exhaustive
update_frequency: weekly

Initial seed queries:

"real-time AI inference" benchmark 2025
"vLLM" vs "TensorRT-LLM" performance
"llama.cpp" release changelog
"AI inference" startup funding 2025
"edge AI inference" adoption enterprise
"AI inference" tokens per second benchmark
site:github.com "vLLM" stars OR contributors
site:arxiv.org "inference optimization" 2025

Step 2 — Build entity graph from first sweep

Entities collected:

[
  {"type": "technology", "name": "vLLM", "version": "0.6.3", "vendor": "UC Berkeley / community", "category": "inference engine"},
  {"type": "technology", "name": "TensorRT-LLM", "version": "0.15", "vendor": "NVIDIA", "category": "inference engine"},
  {"type": "company", "name": "Groq", "industry": "AI hardware", "product": "LPU Inference Engine"},
  {"type": "number", "metric": "tokens_per_second", "value": 523, "context": "Groq Llama 3 70B", "date": "2025-10"},
  {"type": "number", "metric": "github_stars", "value": 32400, "context": "vLLM", "date": "2025-11"}
]

Relationships:

vLLM        --competes_with-->  TensorRT-LLM
vLLM        --competes_with-->  Ollama
Groq        --launched-->       "LPU Inference Engine"
NVIDIA      --launched-->       TensorRT-LLM
llama.cpp   --uses-->           GGUF format

Step 3 — Track adoption signals across cycles

Signal Type	What to Watch	Detection Method
GitHub velocity	Stars, forks, contributor count week-over-week	Snapshot comparison
Enterprise adoption	Case studies, "we migrated to X" blog posts	Keyword search
Benchmark results	Tokens/sec, latency, cost-per-token comparisons	Structured extraction
Job postings	"Experience with vLLM" in job descriptions	Job board queries
Conference talks	Accepted papers, keynote mentions	Conference program search

Step 4 — Detect trends over 4 weekly cycles

Cycle 1: vLLM 31,800 stars | TensorRT-LLM 9,200 stars | Ollama 98,000 stars
Cycle 2: vLLM 32,400 stars | TensorRT-LLM 9,500 stars | Ollama 101,000 stars
Cycle 3: vLLM 33,500 stars | TensorRT-LLM 9,600 stars | Ollama 103,500 stars
Cycle 4: vLLM 35,200 stars | TensorRT-LLM 9,700 stars | Ollama 105,000 stars

Trend: vLLM accelerating (+1,700/wk avg → +1,700 last week)
       Ollama decelerating (+3,000/wk → +1,500/wk)
       TensorRT-LLM flat (~200/wk)

Step 5 — Produce technology landscape report

Include a positioning summary:

Framework	Strengths	Weaknesses	Momentum	Best For
vLLM	High throughput, PagedAttention	GPU-only, complex setup	Accelerating	Production serving at scale
TensorRT-LLM	NVIDIA optimization, low latency	Vendor lock-in, NVIDIA GPUs only	Flat	NVIDIA-stack deployments
Ollama	Simple UX, local-first	Lower throughput, less tunable	Decelerating	Developer experimentation
llama.cpp	CPU support, portable	Manual optimization needed	Steady	Edge/embedded inference
Groq LPU	Extreme speed, low latency	Limited model support, cloud-only	Growing	Latency-critical applications

---

Example 3: M&A Signal Detection

Scenario: Detect early acquisition indicators for companies in the enterprise observability space (Datadog, Grafana Labs, Chronosphere, Honeycomb).

Step 1 — Define M&A signal categories

Signal Category	Indicators	Weight
Executive changes	CEO/CFO departure, new "Chief Strategy Officer", board additions	High
Hiring patterns	Sudden corporate development/M&A roles, legal team expansion	High
Financial signals	Unusual funding, secondary sales, down round, runway concerns	High
Strategic moves	Exclusive partnerships, technology licensing, IP transfers	Medium
Market behavior	Quiet period (no product updates), website changes, domain changes	Medium
Social signals	Founder tone shifts, "exciting news soon" posts, unusual silence	Low

Step 2 — Build targeted queries

"Chronosphere" AND ("acquisition" OR "acquire" OR "acqui-hire" OR "merger")
"Honeycomb" AND ("strategic alternatives" OR "exploring options" OR "advisors")
"Grafana Labs" AND ("corporate development" OR "M&A" OR "strategic partnership")
site:linkedin.com "Chronosphere" "corporate development" OR "M&A"
site:sec.gov "Honeycomb" OR "Hound Technology"
"[company]" "quiet period" OR "exciting announcement"
"[company]" hiring "corporate development" OR "business development director"
"[company]" board of directors new appointment

Step 3 — Entity and event extraction

From collected sources, extract and classify:

[
  {
    "type": "event",
    "name": "Chronosphere CFO departure",
    "date": "2025-10-28",
    "entities": ["Chronosphere", "Lisa Park"],
    "signal_category": "executive_change",
    "m_and_a_weight": "high",
    "source": "linkedin.com/posts/lisapark-farewell"
  },
  {
    "type": "event",
    "name": "Honeycomb hires Goldman Sachs advisor",
    "date": "2025-11-02",
    "entities": ["Honeycomb", "Goldman Sachs"],
    "signal_category": "financial",
    "m_and_a_weight": "high",
    "source": "theinformation.com/articles/honeycomb-advisors"
  },
  {
    "type": "event",
    "name": "Datadog acquires incident.io",
    "date": "2025-11-08",
    "entities": ["Datadog", "incident.io"],
    "signal_category": "strategic",
    "m_and_a_weight": "confirmed_event",
    "source": "datadog.com/blog/incident-io-acquisition"
  }
]

Step 4 — Score composite M&A probability

Aggregate signals per company over a rolling 90-day window:

Chronosphere:
  - CFO departed (high)         +3
  - 2 corp dev job postings     +2
  - No product release in 90d   +1
  - Composite score: 6/10 → ELEVATED

Honeycomb:
  - Hired investment bank       +4
  - Board added PE partner      +2
  - Founder "grateful" post     +1
  - Composite score: 7/10 → HIGH

Grafana Labs:
  - New enterprise partnerships +1
  - Active hiring across all    -1 (normal growth, reduces M&A signal)
  - Composite score: 0/10 → LOW

Step 5 — Generate M&A signal alert

# M&A Signal Alert: Enterprise Observability Sector
**Date**: 2025-11-10 | **Window**: 90 days

## HIGH probability
- **Honeycomb**: Investment bank engagement + board changes suggest active process.
  Key evidence: Goldman Sachs advisory (The Information), new PE board member.
  Likely acquirers: Datadog, Cisco, ServiceNow.

## ELEVATED probability
- **Chronosphere**: Leadership turnover + hiring freeze + corp dev roles.
  Key evidence: CFO departure, no product releases, corp dev postings on LinkedIn.
  Could indicate: acquisition target OR internal restructuring.

## LOW probability
- **Grafana Labs**: Normal operating patterns, active hiring, regular releases.
- **Datadog**: Active acquirer (incident.io deal closed), not a target.

---

Advanced Entity Extraction

Relationship Mapping from Unstructured Text

Extract relationships by identifying sentence-level patterns that connect two named entities.

Pattern templates:

[Person] joined [Company] as [Role]
  → relation: works_at, attributes: {role: Role, event: "joined"}

[Company] acquired [Company] for [Amount]
  → relation: acquired, attributes: {amount: Amount}

[Person] and [Person] co-founded [Company]
  → relations: founded (x2), co_founded_with (between persons)

[Company] partnered with [Company] to [Purpose]
  → relation: partnered_with, attributes: {purpose: Purpose}

[Person] left [Company] to join [Company]
  → relation: left (old), works_at (new), attributes: {event: "departure"}

Multi-hop relationships: When A relates to B and B relates to C, infer indirect connections:

Sarah Chen works_at BetaStack
BetaStack competes_with AlphaCloud
→ Indirect: Sarah Chen is key_person_at competitor of AlphaCloud

Negation detection: Watch for negated relationships that should NOT be added:

"Company X denied it was in acquisition talks with Company Y"
→ Do NOT add acquired relation. Add entity note: "denied acquisition rumor, [date]"

"Former CEO of Company X" → Person left. Mark works_at as ended.

Temporal Event Extraction (Timeline Construction)

Extract dates and temporal markers to build event timelines.

Explicit dates:

"On March 15, 2025, Acme launched ProductX"
  → event: product_launch, date: 2025-03-15, entities: [Acme, ProductX]

Relative dates (resolve against article publication date):

"last week"     → pub_date - 7 days
"earlier today" → pub_date
"next quarter"  → pub_date + next fiscal quarter boundary
"in Q3"         → July-September of article's year
"recently"      → pub_date - 30 days (approximate, confidence: medium)

Temporal ordering heuristics:

"before the acquisition" → event precedes known acquisition date
"following the launch"   → event follows known launch date
"amid layoffs"          → event concurrent with layoff period

Timeline output format:

{
  "entity": "Acme Corp",
  "timeline": [
    {"date": "2025-01-15", "event": "Series B ($40M)", "type": "funding", "confidence": "high"},
    {"date": "2025-03-20", "event": "Hired new CTO (Jane Lee)", "type": "leadership", "confidence": "high"},
    {"date": "2025-06-01", "event": "Launched v3.0", "type": "product", "confidence": "high"},
    {"date": "2025-08-10", "event": "Partnership with CloudCo", "type": "partnership", "confidence": "medium"},
    {"date": "2025-11-05", "event": "Acquired by BigCorp", "type": "acquisition", "confidence": "high"}
  ]
}

Quantitative Data Extraction

Extract numerical data points with units, context, and time reference.

Financial figures:

Pattern: "[Company] raised $[amount][M/B] in [round]"
Example: "Acme raised $40M in Series B"
  → {metric: "funding", value: 40000000, currency: "USD", context: "Series B", entity: "Acme"}

Pattern: "[Company] revenue of $[amount][M/B]"
Example: "reported annual revenue of $120M"
  → {metric: "revenue", value: 120000000, currency: "USD", period: "annual", entity: subject}

Growth rates:

Pattern: "[metric] grew [X]% [period]"
Example: "ARR grew 45% year-over-year"
  → {metric: "ARR_growth", value: 0.45, period: "YoY", entity: subject}

Pattern: "from [X] to [Y]"
Example: "headcount grew from 200 to 350"
  → {metric: "headcount", previous: 200, current: 350, growth: 0.75, entity: subject}

Headcounts and scale metrics:

"[Company] now has [N] employees"
"[Company] serves [N] customers"
"[Product] has [N] monthly active users"
"[Company] operates in [N] countries"

Extraction validation rules:

• Currency amounts without a clear entity reference: discard or mark confidence "low"
• Growth percentages without a base period: mark confidence "medium"
• Round numbers (e.g., "about 1,000 employees"): flag as approximate
• Conflicting numbers from different sources: record both, note discrepancy

Multi-Source Entity Resolution

When the same entity appears across different sources with variations, deduplicate.

Company name normalization:

"Acme Corp" = "Acme Corporation" = "Acme, Inc." = "ACME" (when context matches)
"Google" = "Alphabet" (parent) — but keep as separate entities with parent_of relation

Resolution rules:

Signal	Match Confidence	Action
Exact name match	High	Merge immediately
Name + same industry + same location	High	Merge
Abbreviated name + same context	Medium	Merge with note
Similar name, different industry	Low	Keep separate, flag for review
Person same name, different company	Low	Keep separate unless linked by career event

Deduplication process:

1. Normalize: lowercase, strip legal suffixes, expand abbreviations
2. Match: compare against existing entity list using normalized form
3. Verify: check at least one corroborating attribute (industry, location, person association)
4. Merge: combine attributes, keep all source references, use highest confidence level
5. Log: record the merge decision for audit

{
  "canonical": "entity_acme_corp",
  "aliases": ["Acme Corp", "Acme Corporation", "Acme, Inc.", "ACME"],
  "merged_from": ["source_techcrunch_entity_12", "source_linkedin_entity_89"],
  "merge_confidence": "high",
  "merge_reason": "exact name + same industry (SaaS) + same HQ (San Francisco)"
}

---

Collection Automation Patterns

Scheduled Collection Workflows

Define collection cadences matched to intelligence needs.

Daily cycle (for active competitive monitoring):

06:00 UTC — Run news queries for all targets (surface scan)
06:15 UTC — Check social media and forums for overnight mentions
06:30 UTC — Compare against yesterday's snapshot, flag changes
06:45 UTC — Generate daily brief, send alerts for CRITICAL items

Weekly cycle (for technology landscape and market mapping):

Monday  — Full source sweep: news, blogs, official sites
Tuesday — Job board scan: new postings, closed postings, pattern analysis
Wednesday — Financial data: funding rounds, SEC filings, earnings
Thursday — Community signals: GitHub activity, forum discussions, reviews
Friday  — Synthesis: generate weekly report, update entity graph, adjust queries

Event-triggered cycle (supplement scheduled runs):

Trigger: CRITICAL change detected in any cycle
  → Immediately run deep collection on the affected entity
  → Expand query set to cover related entities
  → Generate ad-hoc alert report
  → Shorten next scheduled cycle interval (e.g., weekly → daily for 7 days)

Source Prioritization Based on Hit Rate

Track which sources consistently produce actionable intelligence and allocate collection effort accordingly.

Hit rate calculation:

hit_rate = (data_points_extracted / fetches_from_source) over last 10 cycles

Priority tiers:

Hit Rate	Priority	Collection Behavior
> 60%	Tier 1	Always fetch, process first
30-60%	Tier 2	Fetch on every cycle
10-30%	Tier 3	Fetch every other cycle
< 10%	Tier 4	Fetch weekly regardless of cycle frequency
0% for 5+ cycles	Drop	Remove from active source list, log reason

Source performance tracking:

{
  "source": "techcrunch.com",
  "total_fetches": 48,
  "data_points_extracted": 31,
  "hit_rate": 0.65,
  "tier": 1,
  "avg_confidence": "medium-high",
  "last_hit": "2025-11-15",
  "best_queries": ["[company] funding", "[company] acquisition"]
}

Incremental Collection (Only New/Changed Content)

Avoid re-processing unchanged content across cycles.

Techniques:

1. URL deduplication: Maintain a set of already-processed URLs. Skip on subsequent cycles.
2. Content hashing: Hash the extracted text body. If hash matches previous cycle, skip processing.
3. Date filtering: Append date ranges to queries to limit results to new content.
4. Pagination cursors: For APIs and structured sources, store the last-seen ID or timestamp.

Query date narrowing:

Cycle runs daily at 06:00 UTC:
  "AlphaCloud" after:2025-11-15 before:2025-11-16
  "AlphaCloud" news past 24 hours

Cycle runs weekly:
  "AlphaCloud" after:2025-11-08 before:2025-11-15

State tracking for incremental collection:

{
  "processed_urls": ["https://example.com/article-1", "..."],
  "content_hashes": {"url1": "sha256:abc123", "url2": "sha256:def456"},
  "last_collection_time": "2025-11-15T06:00:00Z",
  "query_cursors": {
    "techcrunch_rss": "2025-11-15T05:30:00Z",
    "github_api_events": "event_id_98765"
  }
}

Alert Trigger Conditions and Escalation Rules

Define when and how to escalate detected changes.

Trigger conditions:

IMMEDIATE ALERT (publish event_publish within the cycle):
  - Leadership change at target company (CEO, CTO, CFO)
  - Acquisition or merger announcement
  - Funding round > $10M
  - Product discontinuation or major pivot
  - Regulatory action or legal filing
  - Data breach or security incident

DAILY DIGEST (batch into next daily report):
  - New product feature or version release
  - New partnership announcement
  - Hiring surge (> 5 new roles in a category)
  - Pricing or packaging change
  - Significant sentiment shift (score delta > 2 in one cycle)

WEEKLY SUMMARY (include in weekly report only):
  - Blog posts and thought leadership
  - Conference appearances
  - Minor version updates or patches
  - Individual job postings
  - Social media activity within normal range

Escalation rules:

Level 1 — Auto-include in next scheduled report (default for all changes)
Level 2 — event_publish immediately (for CRITICAL significance changes)
Level 3 — event_publish + re-run deep collection on affected entity (for M&A, major crises)

False positive suppression:

• Require 2+ independent sources before triggering Level 2 alerts
• Ignore "rumor" or "speculation" tagged content for immediate alerts
• If the same alert fired in the previous cycle with no new corroboration, suppress repeat

---

Analysis Techniques

Link Analysis (Connection Mapping)

Map the network of relationships between entities to reveal hidden connections, influence patterns, and structural vulnerabilities.

Building the adjacency map:

From the knowledge graph, extract all relations and build:

Nodes: [Acme, BetaCo, GammaSuite, Jane Lee, CloudCo, InvestorX]
Edges:
  Acme       --competes_with-->    BetaCo
  Acme       --partnered_with-->   CloudCo
  Jane Lee   --works_at-->         Acme
  Jane Lee   --formerly-->         BetaCo
  InvestorX  --invested_in-->      Acme
  InvestorX  --invested_in-->      GammaSuite

Key metrics to compute:

Metric	Meaning	Use
Degree centrality	Number of direct connections	Identifies most-connected entities
Shared connections	Entities with overlapping relationships	Reveals indirect competition or collaboration
Bridge nodes	Entities connecting otherwise separate clusters	Identifies key influencers or gatekeepers
Cluster density	Ratio of actual to possible connections in a group	Measures how tightly coupled a set of entities is

Practical analysis patterns:

Investor overlap:
  InvestorX invested_in Acme AND GammaSuite
  → Potential: board-level information sharing, future merger pressure

Talent flow:
  Jane Lee: BetaCo (2020-2024) → Acme (2024-present)
  3 other engineers: BetaCo → Acme in same period
  → Pattern: talent drain from BetaCo to Acme, possible IP risk

Supply chain dependency:
  Acme uses CloudCo infrastructure
  BetaCo uses CloudCo infrastructure
  → Shared dependency: CloudCo outage affects both competitors

Timeline Analysis (Event Sequencing and Pattern Detection)

Arrange extracted events chronologically to detect causal chains, recurring patterns, and anomalous timing.

Constructing the timeline:

2025-01  Acme raises Series B ($40M)
2025-02  Acme posts 15 engineering roles
2025-03  Acme hires CTO from Google
2025-05  Acme acquires small startup (data pipeline tool)
2025-06  Acme launches v3.0 with data pipeline features
2025-08  Acme announces enterprise pricing tier

Pattern detection rules:

Pattern	Sequence	Interpretation
Build-up to launch	Funding → Hiring surge → Leadership hire → Product release	Normal growth execution
Acquisition integration	Acquire company → Quiet period (2-4 months) → Feature launch using acquired tech	Successful integration
Pre-acquisition signals	Advisor hire → Leadership departures → Quiet period → Announcement	Target company being acquired
Distress pattern	Layoffs → Pricing cuts → Leadership change → Pivot or shutdown	Company in trouble
Expansion play	Funding → New market entry → Localized hiring → Regional partnerships	Geographic or vertical expansion

Anomaly detection:

Expected: Funding round → hiring surge within 60 days
Observed: Funding round → no hiring after 90 days
→ Flag: "Post-funding hiring anomaly — possible pivot, internal issues, or stealth project"

Expected: Product launch → marketing push within 30 days
Observed: Product launch → silence
→ Flag: "Launch without marketing — possible soft launch, or product issues"

Trend Detection (Acceleration, Deceleration, Inflection Points)

Track metrics across collection cycles to identify directional shifts.

Metric tracking format:

{
  "entity": "Acme Corp",
  "metric": "job_postings",
  "series": [
    {"cycle": 1, "date": "2025-09-01", "value": 12},
    {"cycle": 2, "date": "2025-09-08", "value": 18},
    {"cycle": 3, "date": "2025-09-15", "value": 31},
    {"cycle": 4, "date": "2025-09-22", "value": 45},
    {"cycle": 5, "date": "2025-09-29", "value": 42}
  ]
}

Trend classification:

Pattern	Detection Rule	Meaning
Accelerating	Growth rate increasing cycle-over-cycle	Expanding investment in area
Decelerating	Growth rate decreasing but still positive	Approaching saturation or shift in priorities
Inflection point	Direction change (growth → decline or vice versa)	Strategic shift, market event, or external shock
Plateau	Value stable within 10% for 3+ cycles	Steady state, maintenance mode
Spike	Single-cycle jump > 2x previous value	One-time event (launch, announcement, crisis)
Cliff	Single-cycle drop > 50%	Sudden change (layoff, shutdown, policy change)

Multi-metric correlation:

When two metrics move together, the correlation strengthens the signal:

Acme job_postings: accelerating
Acme github_commits: accelerating
→ Corroborated signal: major development push underway

BetaCo job_postings: cliff (-60%)
BetaCo glassdoor_rating: declining
→ Corroborated signal: organizational distress

Competitive Positioning Maps

Synthesize collected intelligence into comparative frameworks.

Feature parity matrix:

Capability	Acme	BetaCo	GammaSuite	Your Product
Real-time dashboards	Yes (v2.0+)	Yes	Limited	Yes
AI-powered alerts	Yes (new in v4.2)	No	Beta	Planned Q1
On-prem deployment	No	Yes	Yes	Yes
SOC2 compliance	Yes	Yes	In progress	Yes
Free tier	No	Yes (limited)	Yes	Yes

Market position quadrant (based on collected metrics):

                    High Market Share
                         |
           Leaders       |      Challengers
          (Acme)         |      (GammaSuite)
                         |
  Low Growth ────────────┼──────────── High Growth
                         |
           Declining     |      Emerging
          (Legacy Co)    |      (BetaCo)
                         |
                    Low Market Share

Inputs for positioning:

• Market share proxy: mention frequency, customer count, job posting volume
• Growth proxy: funding recency, hiring rate, product release velocity, GitHub star velocity

Pricing intelligence table:

Tier	Acme	BetaCo	GammaSuite	Notes
Free	--	5 users	10 users	BetaCo most restrictive
Team	$15/user/mo	$12/user/mo	$20/user/mo	BetaCo cheapest
Enterprise	Custom	$35/user/mo	Custom	BetaCo only one with public enterprise pricing
Notable changes	Raised Team tier 20% in Q3	Unchanged 12 months	New tier added Q4	Acme pricing pressure

Track pricing changes across cycles — pricing increases signal confidence, decreases signal competitive pressure or churn concerns.