feat(hands): complete i18n fixes, SKILL.md enhancements, and README overhaul
- Fix French accent characters (é/è/ê/ç/â/ô) across all 14 HAND.toml files - Fix German special characters (ä/ö/ü/ß) across all 14 HAND.toml files - Add category translations to all 6 i18n language blocks in all 14 hands - Enhance SKILL.md content for 9 hands with practical examples and workflows - Trim bloated SKILL.md files (apitester 1400→892, devops 1301→870) - Rewrite root README.md with accurate stats, complete hand/integration tables - Update hands/README.md with full 14-hand listing and i18n documentation
This commit is contained in:
Evan Hu committed
1 parent
315f955ce2
commit
33d279889c
27 files changed
+10001
-78
No files matched your search
@@ -337,3 +337,702 @@ Level 4: What to do (prescriptive)
|
||||
| Timeliness | Current | Data refreshed daily |
|
||||
| Uniqueness | 99% | 1% duplicate records found |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Worked Examples
|
||||
|
||||
### Example 1: E-commerce Sales Analysis
|
||||
|
||||
**Goal**: Analyze 12 months of order data to identify revenue drivers, customer segments, and growth trends.
|
||||
|
||||
#### Step 1 — Load and clean
|
||||
```python
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
|
||||
df = pd.read_csv('orders.csv', parse_dates=['order_date'])
|
||||
|
||||
# Quick audit
|
||||
print(f"Rows: {len(df):,} Columns: {df.shape[1]}")
|
||||
print(df.isnull().sum()[df.isnull().sum() > 0])
|
||||
|
||||
# Clean
|
||||
df = df.dropna(subset=['customer_id', 'order_total'])
|
||||
df['order_total'] = df['order_total'].clip(lower=0) # Remove negative values
|
||||
df['order_month'] = df['order_date'].dt.to_period('M')
|
||||
```
|
||||
|
||||
#### Step 2 — Revenue trend analysis
|
||||
```python
|
||||
monthly = (
|
||||
df.groupby('order_month')
|
||||
.agg(revenue=('order_total', 'sum'),
|
||||
orders=('order_id', 'nunique'),
|
||||
customers=('customer_id', 'nunique'))
|
||||
.reset_index()
|
||||
)
|
||||
monthly['aov'] = monthly['revenue'] / monthly['orders'] # Average order value
|
||||
monthly['revenue_mom'] = monthly['revenue'].pct_change() # Month-over-month growth
|
||||
|
||||
fig, axes = plt.subplots(2, 1, figsize=(12, 8), sharex=True)
|
||||
axes[0].bar(monthly['order_month'].astype(str), monthly['revenue'], color='steelblue')
|
||||
axes[0].set_title('Monthly Revenue', fontsize=14, fontweight='bold')
|
||||
axes[0].set_ylabel('Revenue ($)')
|
||||
|
||||
axes[1].plot(monthly['order_month'].astype(str), monthly['aov'], marker='o', color='coral')
|
||||
axes[1].set_title('Average Order Value', fontsize=14, fontweight='bold')
|
||||
axes[1].set_ylabel('AOV ($)')
|
||||
plt.xticks(rotation=45, ha='right')
|
||||
plt.tight_layout()
|
||||
plt.savefig('revenue_trend.png', dpi=150, bbox_inches='tight')
|
||||
plt.close()
|
||||
```
|
||||
|
||||
#### Step 3 — Customer segmentation (RFM)
|
||||
```python
|
||||
snapshot_date = df['order_date'].max() + pd.Timedelta(days=1)
|
||||
|
||||
rfm = df.groupby('customer_id').agg(
|
||||
recency=('order_date', lambda x: (snapshot_date - x.max()).days),
|
||||
frequency=('order_id', 'nunique'),
|
||||
monetary=('order_total', 'sum')
|
||||
)
|
||||
|
||||
# Score each dimension 1-4 using quartiles
|
||||
for col in ['recency', 'frequency', 'monetary']:
|
||||
labels = [4, 3, 2, 1] if col == 'recency' else [1, 2, 3, 4]
|
||||
rfm[f'{col}_score'] = pd.qcut(rfm[col], q=4, labels=labels, duplicates='drop')
|
||||
|
||||
rfm['rfm_score'] = (rfm['recency_score'].astype(int)
|
||||
+ rfm['frequency_score'].astype(int)
|
||||
+ rfm['monetary_score'].astype(int))
|
||||
|
||||
# Segment mapping
|
||||
def segment(row):
|
||||
r, f, m = int(row['recency_score']), int(row['frequency_score']), int(row['monetary_score'])
|
||||
if r >= 3 and f >= 3:
|
||||
return 'Champions'
|
||||
elif r >= 3 and f < 3:
|
||||
return 'New / Promising'
|
||||
elif r < 3 and f >= 3:
|
||||
return 'At Risk'
|
||||
else:
|
||||
return 'Needs Attention'
|
||||
|
||||
rfm['segment'] = rfm.apply(segment, axis=1)
|
||||
print(rfm.groupby('segment').agg(
|
||||
count=('monetary', 'size'),
|
||||
avg_revenue=('monetary', 'mean'),
|
||||
avg_frequency=('frequency', 'mean')
|
||||
).sort_values('avg_revenue', ascending=False))
|
||||
```
|
||||
|
||||
#### Step 4 — Cohort retention analysis
|
||||
```python
|
||||
df['cohort'] = df.groupby('customer_id')['order_date'].transform('min').dt.to_period('M')
|
||||
df['order_period'] = df['order_date'].dt.to_period('M')
|
||||
df['cohort_index'] = (df['order_period'] - df['cohort']).apply(lambda x: x.n)
|
||||
|
||||
cohort_table = (
|
||||
df.groupby(['cohort', 'cohort_index'])['customer_id']
|
||||
.nunique()
|
||||
.reset_index()
|
||||
.pivot(index='cohort', columns='cohort_index', values='customer_id')
|
||||
)
|
||||
|
||||
# Convert to retention percentages
|
||||
retention = cohort_table.div(cohort_table[0], axis=0) * 100
|
||||
|
||||
fig, ax = plt.subplots(figsize=(14, 8))
|
||||
sns.heatmap(retention, annot=True, fmt='.0f', cmap='YlOrRd_r', ax=ax)
|
||||
ax.set_title('Cohort Retention (% of original customers)', fontsize=14, fontweight='bold')
|
||||
ax.set_xlabel('Months Since First Purchase')
|
||||
ax.set_ylabel('Cohort')
|
||||
plt.tight_layout()
|
||||
plt.savefig('cohort_retention.png', dpi=150, bbox_inches='tight')
|
||||
plt.close()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Example 2: A/B Test Analysis
|
||||
|
||||
**Goal**: Evaluate whether a new checkout flow (variant B) improves conversion rate over the existing flow (variant A).
|
||||
|
||||
#### Step 1 — Sample size calculation (pre-test)
|
||||
```python
|
||||
from scipy import stats
|
||||
import numpy as np
|
||||
|
||||
baseline_rate = 0.12 # Current conversion rate: 12%
|
||||
mde = 0.02 # Minimum detectable effect: 2 percentage points
|
||||
alpha = 0.05 # Significance level
|
||||
power = 0.80 # Statistical power
|
||||
|
||||
# Using the normal approximation formula
|
||||
p1 = baseline_rate
|
||||
p2 = baseline_rate + mde
|
||||
p_avg = (p1 + p2) / 2
|
||||
|
||||
z_alpha = stats.norm.ppf(1 - alpha / 2) # Two-tailed
|
||||
z_beta = stats.norm.ppf(power)
|
||||
|
||||
n_per_group = ((z_alpha * np.sqrt(2 * p_avg * (1 - p_avg))
|
||||
+ z_beta * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
|
||||
/ (p2 - p1) ** 2)
|
||||
|
||||
print(f"Required sample size per group: {int(np.ceil(n_per_group)):,}")
|
||||
print(f"Total required: {int(np.ceil(n_per_group)) * 2:,}")
|
||||
```
|
||||
|
||||
#### Step 2 — Run the test and collect results
|
||||
```python
|
||||
ab = pd.read_csv('ab_test_results.csv')
|
||||
|
||||
summary = ab.groupby('variant').agg(
|
||||
visitors=('user_id', 'nunique'),
|
||||
conversions=('converted', 'sum')
|
||||
)
|
||||
summary['conversion_rate'] = summary['conversions'] / summary['visitors']
|
||||
print(summary)
|
||||
```
|
||||
|
||||
#### Step 3 — Statistical significance
|
||||
```python
|
||||
a = ab[ab['variant'] == 'A']
|
||||
b = ab[ab['variant'] == 'B']
|
||||
|
||||
# Chi-squared test for proportions
|
||||
contingency = pd.crosstab(ab['variant'], ab['converted'])
|
||||
chi2, p_value, dof, expected = stats.chi2_contingency(contingency)
|
||||
|
||||
# Proportions z-test (more direct)
|
||||
from statsmodels.stats.proportion import proportions_ztest
|
||||
successes = [summary.loc['B', 'conversions'], summary.loc['A', 'conversions']]
|
||||
trials = [summary.loc['B', 'visitors'], summary.loc['A', 'visitors']]
|
||||
z_stat, p_val = proportions_ztest(successes, trials, alternative='larger')
|
||||
|
||||
print(f"Z-statistic: {z_stat:.4f}")
|
||||
print(f"P-value: {p_val:.4f}")
|
||||
print(f"Significant: {'Yes' if p_val < 0.05 else 'No'} (at alpha=0.05)")
|
||||
```
|
||||
|
||||
#### Step 4 — Effect size and confidence interval
|
||||
```python
|
||||
p_a = summary.loc['A', 'conversion_rate']
|
||||
p_b = summary.loc['B', 'conversion_rate']
|
||||
n_a = summary.loc['A', 'visitors']
|
||||
n_b = summary.loc['B', 'visitors']
|
||||
|
||||
lift = (p_b - p_a) / p_a
|
||||
se_diff = np.sqrt(p_a * (1 - p_a) / n_a + p_b * (1 - p_b) / n_b)
|
||||
ci_lower = (p_b - p_a) - 1.96 * se_diff
|
||||
ci_upper = (p_b - p_a) + 1.96 * se_diff
|
||||
|
||||
print(f"Control rate: {p_a:.4f}")
|
||||
print(f"Variant rate: {p_b:.4f}")
|
||||
print(f"Absolute lift: {p_b - p_a:+.4f}")
|
||||
print(f"Relative lift: {lift:+.2%}")
|
||||
print(f"95% CI for diff: [{ci_lower:+.4f}, {ci_upper:+.4f}]")
|
||||
```
|
||||
|
||||
#### Step 5 — Recommendation template
|
||||
```
|
||||
## A/B Test Report: New Checkout Flow
|
||||
|
||||
| Metric | Control (A) | Variant (B) |
|
||||
|---------------------|-------------|-------------|
|
||||
| Visitors | 15,204 | 15,198 |
|
||||
| Conversions | 1,824 | 2,127 |
|
||||
| Conversion Rate | 12.00% | 13.99% |
|
||||
|
||||
**Result**: Statistically significant (p = 0.0003, alpha = 0.05)
|
||||
**Lift**: +1.99pp absolute / +16.6% relative
|
||||
**95% CI**: [+0.90pp, +3.08pp]
|
||||
**Recommendation**: Deploy variant B. The effect is both statistically
|
||||
and practically significant with a lower bound above the +1pp threshold.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Example 3: Customer Churn Analysis
|
||||
|
||||
**Goal**: Identify which factors most strongly predict customer churn and quantify their relative importance.
|
||||
|
||||
#### Step 1 — Feature engineering
|
||||
```python
|
||||
df = pd.read_csv('customers.csv')
|
||||
|
||||
# Create behavioral features from raw data
|
||||
features = df.copy()
|
||||
features['tenure_months'] = (pd.Timestamp.now() - pd.to_datetime(df['signup_date'])).dt.days / 30
|
||||
features['support_tickets_per_month'] = df['total_tickets'] / features['tenure_months'].clip(lower=1)
|
||||
features['avg_session_minutes'] = df['total_session_minutes'] / df['total_sessions'].clip(lower=1)
|
||||
features['days_since_last_login'] = (pd.Timestamp.now() - pd.to_datetime(df['last_login'])).dt.days
|
||||
features['has_premium'] = (df['plan'] == 'premium').astype(int)
|
||||
|
||||
# Drop raw columns, keep engineered features
|
||||
feature_cols = [
|
||||
'tenure_months', 'support_tickets_per_month', 'avg_session_minutes',
|
||||
'days_since_last_login', 'has_premium', 'monthly_spend', 'num_features_used'
|
||||
]
|
||||
```
|
||||
|
||||
#### Step 2 — Correlation analysis
|
||||
```python
|
||||
churn_corr = features[feature_cols + ['churned']].corr()['churned'].drop('churned').sort_values()
|
||||
|
||||
fig, ax = plt.subplots(figsize=(8, 5))
|
||||
churn_corr.plot(kind='barh', ax=ax, color=['coral' if x > 0 else 'steelblue' for x in churn_corr])
|
||||
ax.set_title('Feature Correlation with Churn', fontsize=14, fontweight='bold')
|
||||
ax.set_xlabel('Pearson Correlation')
|
||||
ax.axvline(x=0, color='black', linewidth=0.5)
|
||||
plt.tight_layout()
|
||||
plt.savefig('churn_correlations.png', dpi=150, bbox_inches='tight')
|
||||
plt.close()
|
||||
```
|
||||
|
||||
#### Step 3 — Key driver identification via group comparison
|
||||
```python
|
||||
churned = features[features['churned'] == 1]
|
||||
retained = features[features['churned'] == 0]
|
||||
|
||||
comparison = []
|
||||
for col in feature_cols:
|
||||
t_stat, p_val = stats.ttest_ind(churned[col].dropna(), retained[col].dropna())
|
||||
d = cohens_d(churned[col].dropna(), retained[col].dropna()) # From earlier definition
|
||||
comparison.append({
|
||||
'feature': col,
|
||||
'churned_mean': churned[col].mean(),
|
||||
'retained_mean': retained[col].mean(),
|
||||
'diff_pct': (churned[col].mean() - retained[col].mean()) / retained[col].mean() * 100,
|
||||
'cohens_d': abs(d),
|
||||
'p_value': p_val,
|
||||
'significant': p_val < 0.05
|
||||
})
|
||||
|
||||
result = pd.DataFrame(comparison).sort_values('cohens_d', ascending=False)
|
||||
print(result.to_string(index=False))
|
||||
```
|
||||
|
||||
#### Step 4 — Interpret and report
|
||||
```
|
||||
## Churn Driver Analysis
|
||||
|
||||
**Top 3 factors distinguishing churned vs. retained customers:**
|
||||
|
||||
| Factor | Churned (avg) | Retained (avg) | Diff | Effect Size |
|
||||
|----------------------------|---------------|----------------|----------|-------------|
|
||||
| Days since last login | 34.2 | 8.7 | +293% | Large |
|
||||
| Support tickets per month | 2.8 | 0.9 | +211% | Large |
|
||||
| Number of features used | 3.1 | 7.4 | -58% | Medium |
|
||||
|
||||
**Actionable insights:**
|
||||
1. Customers inactive >14 days are 4x more likely to churn -- trigger re-engagement email at day 10
|
||||
2. High support ticket rate signals frustration -- escalate accounts with >2 tickets/month to success team
|
||||
3. Low feature adoption correlates with churn -- implement onboarding flow targeting unused features
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Advanced pandas Patterns
|
||||
|
||||
### Window Functions
|
||||
|
||||
```python
|
||||
# Expanding window (cumulative statistics)
|
||||
df['cumulative_avg'] = df['value'].expanding().mean()
|
||||
df['cumulative_max'] = df['value'].expanding().max()
|
||||
|
||||
# Exponentially weighted moving average (EWMA) -- emphasizes recent values
|
||||
df['ewma_7'] = df['value'].ewm(span=7).mean() # Span-based decay
|
||||
df['ewma_a'] = df['value'].ewm(alpha=0.3).mean() # Explicit decay factor
|
||||
|
||||
# Comparison: rolling vs. EWMA
|
||||
# - rolling(7).mean() weights all 7 values equally
|
||||
# - ewm(span=7).mean() weights recent values exponentially more
|
||||
# Use EWMA when recent data matters more (stock prices, real-time metrics)
|
||||
|
||||
# Rolling with min_periods (handles early rows with insufficient data)
|
||||
df['rolling_avg'] = df['value'].rolling(window=30, min_periods=5).mean()
|
||||
|
||||
# Rolling rank (percentile within window)
|
||||
df['rolling_pctile'] = df['value'].rolling(90).rank(pct=True)
|
||||
```
|
||||
|
||||
### Multi-Index Operations
|
||||
|
||||
```python
|
||||
# Create multi-index from groupby
|
||||
multi = df.groupby(['region', 'product']).agg(
|
||||
revenue=('amount', 'sum'),
|
||||
units=('quantity', 'sum')
|
||||
)
|
||||
|
||||
# Access levels
|
||||
multi.loc['North'] # All products in North region
|
||||
multi.loc[('North', 'Widget')] # Specific region + product
|
||||
multi.xs('Widget', level='product') # All regions for Widget
|
||||
|
||||
# Swap and sort levels
|
||||
multi = multi.swaplevel().sort_index()
|
||||
|
||||
# Reset to flat columns
|
||||
flat = multi.reset_index()
|
||||
|
||||
# Stack / unstack (reshape between long and wide)
|
||||
wide = multi['revenue'].unstack(level='product') # Products become columns
|
||||
long = wide.stack() # Back to multi-index
|
||||
```
|
||||
|
||||
### Merge and Join Patterns
|
||||
|
||||
```python
|
||||
# Inner join (only matching rows)
|
||||
merged = orders.merge(customers, on='customer_id', how='inner')
|
||||
|
||||
# Left join with indicator (see which rows matched)
|
||||
merged = orders.merge(customers, on='customer_id', how='left', indicator=True)
|
||||
unmatched = merged[merged['_merge'] == 'left_only']
|
||||
|
||||
# Join on multiple keys
|
||||
merged = df1.merge(df2, on=['date', 'region'], how='left')
|
||||
|
||||
# Join with different column names
|
||||
merged = orders.merge(products, left_on='prod_id', right_on='product_id')
|
||||
|
||||
# Anti-join (rows in A that have no match in B)
|
||||
anti = df_a.merge(df_b, on='key', how='left', indicator=True)
|
||||
anti = anti[anti['_merge'] == 'left_only'].drop(columns='_merge')
|
||||
|
||||
# Self-join (compare rows within same table)
|
||||
df_prev = df[['customer_id', 'order_date', 'amount']].rename(
|
||||
columns={'order_date': 'prev_date', 'amount': 'prev_amount'}
|
||||
)
|
||||
df_with_prev = df.merge(df_prev, on='customer_id', how='left')
|
||||
df_with_prev = df_with_prev[df_with_prev['prev_date'] < df_with_prev['order_date']]
|
||||
```
|
||||
|
||||
### Apply and Transform
|
||||
|
||||
```python
|
||||
# transform() returns same-shaped output -- useful for group-level stats on each row
|
||||
df['group_mean'] = df.groupby('category')['value'].transform('mean')
|
||||
df['pct_of_group'] = df['value'] / df.groupby('category')['value'].transform('sum')
|
||||
df['z_within_group'] = df.groupby('category')['value'].transform(
|
||||
lambda x: (x - x.mean()) / x.std()
|
||||
)
|
||||
|
||||
# apply() for multi-column group operations
|
||||
def top_n(group, n=3):
|
||||
return group.nlargest(n, 'value')
|
||||
|
||||
top3_per_category = df.groupby('category', group_keys=False).apply(top_n, n=3)
|
||||
|
||||
# Vectorized operations (prefer these over apply when possible)
|
||||
# Slow:
|
||||
df['result'] = df.apply(lambda row: row['a'] * row['b'] + row['c'], axis=1)
|
||||
# Fast:
|
||||
df['result'] = df['a'] * df['b'] + df['c']
|
||||
|
||||
# np.where for conditional columns (vectorized if/else)
|
||||
df['tier'] = np.where(df['revenue'] > 10000, 'high', 'low')
|
||||
|
||||
# np.select for multiple conditions
|
||||
conditions = [
|
||||
df['revenue'] > 10000,
|
||||
df['revenue'] > 5000,
|
||||
df['revenue'] > 0,
|
||||
]
|
||||
choices = ['high', 'medium', 'low']
|
||||
df['tier'] = np.select(conditions, choices, default='none')
|
||||
```
|
||||
|
||||
### Memory Optimization for Large Datasets
|
||||
|
||||
```python
|
||||
# Check current memory usage
|
||||
print(df.memory_usage(deep=True).sum() / 1024**2, "MB")
|
||||
|
||||
# Downcast numeric types
|
||||
df['int_col'] = pd.to_numeric(df['int_col'], downcast='integer') # int64 -> int8/16/32
|
||||
df['float_col'] = pd.to_numeric(df['float_col'], downcast='float') # float64 -> float32
|
||||
|
||||
# Use category type for low-cardinality strings
|
||||
for col in df.select_dtypes(include='object'):
|
||||
if df[col].nunique() / len(df) < 0.5: # Less than 50% unique values
|
||||
df[col] = df[col].astype('category')
|
||||
|
||||
# Read in chunks for files that exceed memory
|
||||
chunks = pd.read_csv('huge_file.csv', chunksize=100_000)
|
||||
results = []
|
||||
for chunk in chunks:
|
||||
processed = chunk.groupby('category')['value'].sum()
|
||||
results.append(processed)
|
||||
final = pd.concat(results).groupby(level=0).sum()
|
||||
|
||||
# Specify dtypes at load time (avoids loading as float64/object first)
|
||||
dtypes = {
|
||||
'id': 'int32',
|
||||
'category': 'category',
|
||||
'value': 'float32',
|
||||
'flag': 'bool'
|
||||
}
|
||||
df = pd.read_csv('data.csv', dtype=dtypes)
|
||||
|
||||
# Use pyarrow backend for better memory efficiency (pandas 2.0+)
|
||||
df = pd.read_csv('data.csv', engine='pyarrow', dtype_backend='pyarrow')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dashboard and Reporting Patterns
|
||||
|
||||
### Executive Dashboard Template
|
||||
|
||||
```python
|
||||
import matplotlib.pyplot as plt
|
||||
import matplotlib.gridspec as gridspec
|
||||
from matplotlib.patches import FancyBboxPatch
|
||||
|
||||
def executive_dashboard(kpis, trend_df, comparison_df, output='dashboard.png'):
|
||||
"""
|
||||
kpis: dict with keys like {'Revenue': '$1.2M', 'Growth': '+15%', ...}
|
||||
trend_df: DataFrame with 'date' and 'value' columns
|
||||
comparison_df: DataFrame with 'category' and 'current'/'previous' columns
|
||||
"""
|
||||
fig = plt.figure(figsize=(16, 10))
|
||||
gs = gridspec.GridSpec(3, len(kpis), hspace=0.4, wspace=0.3)
|
||||
|
||||
# Row 1: KPI cards
|
||||
for i, (label, value) in enumerate(kpis.items()):
|
||||
ax = fig.add_subplot(gs[0, i])
|
||||
ax.text(0.5, 0.6, value, ha='center', va='center',
|
||||
fontsize=28, fontweight='bold', color='#2c3e50')
|
||||
ax.text(0.5, 0.2, label, ha='center', va='center',
|
||||
fontsize=12, color='#7f8c8d')
|
||||
ax.set_xlim(0, 1)
|
||||
ax.set_ylim(0, 1)
|
||||
ax.axis('off')
|
||||
# Card background
|
||||
rect = FancyBboxPatch((0.05, 0.05), 0.9, 0.9, boxstyle="round,pad=0.05",
|
||||
facecolor='#f8f9fa', edgecolor='#dee2e6')
|
||||
ax.add_patch(rect)
|
||||
|
||||
# Row 2: Trend line
|
||||
ax_trend = fig.add_subplot(gs[1, :])
|
||||
ax_trend.plot(trend_df['date'], trend_df['value'], linewidth=2, color='steelblue')
|
||||
ax_trend.fill_between(trend_df['date'], trend_df['value'], alpha=0.1, color='steelblue')
|
||||
ax_trend.set_title('Trend Over Time', fontsize=13, fontweight='bold')
|
||||
ax_trend.set_ylabel('Value')
|
||||
|
||||
# Row 3: Period comparison (grouped bar)
|
||||
ax_comp = fig.add_subplot(gs[2, :])
|
||||
x = range(len(comparison_df))
|
||||
width = 0.35
|
||||
ax_comp.bar([i - width/2 for i in x], comparison_df['previous'], width,
|
||||
label='Previous', color='#bdc3c7')
|
||||
ax_comp.bar([i + width/2 for i in x], comparison_df['current'], width,
|
||||
label='Current', color='steelblue')
|
||||
ax_comp.set_xticks(list(x))
|
||||
ax_comp.set_xticklabels(comparison_df['category'], rotation=45, ha='right')
|
||||
ax_comp.set_title('Current vs. Previous Period', fontsize=13, fontweight='bold')
|
||||
ax_comp.legend()
|
||||
|
||||
plt.savefig(output, dpi=150, bbox_inches='tight', facecolor='white')
|
||||
plt.close()
|
||||
```
|
||||
|
||||
### Weekly Metrics Report Template
|
||||
|
||||
```python
|
||||
def weekly_report(df, date_col='date', metric_col='value', group_col=None):
|
||||
"""Generate a standard weekly metrics summary."""
|
||||
df[date_col] = pd.to_datetime(df[date_col])
|
||||
df['week'] = df[date_col].dt.isocalendar().week.astype(int)
|
||||
df['year'] = df[date_col].dt.year
|
||||
|
||||
current_week = df['week'].max()
|
||||
prev_week = current_week - 1
|
||||
|
||||
curr = df[df['week'] == current_week]
|
||||
prev = df[df['week'] == prev_week]
|
||||
|
||||
report = {
|
||||
'period': f"Week {current_week}",
|
||||
'total': curr[metric_col].sum(),
|
||||
'mean': curr[metric_col].mean(),
|
||||
'median': curr[metric_col].median(),
|
||||
'wow_change': (curr[metric_col].sum() - prev[metric_col].sum())
|
||||
/ prev[metric_col].sum() * 100
|
||||
if prev[metric_col].sum() != 0 else None,
|
||||
}
|
||||
|
||||
if group_col:
|
||||
report['by_group'] = curr.groupby(group_col)[metric_col].agg(['sum', 'mean', 'count'])
|
||||
|
||||
# Sparkline trend (last 8 weeks)
|
||||
weekly_totals = (
|
||||
df.groupby('week')[metric_col].sum()
|
||||
.tail(8)
|
||||
.reset_index()
|
||||
)
|
||||
|
||||
fig, ax = plt.subplots(figsize=(6, 2))
|
||||
ax.plot(weekly_totals['week'], weekly_totals[metric_col], marker='o',
|
||||
linewidth=2, color='steelblue', markersize=4)
|
||||
ax.fill_between(weekly_totals['week'], weekly_totals[metric_col],
|
||||
alpha=0.1, color='steelblue')
|
||||
ax.set_title(f'{metric_col.title()} — Last 8 Weeks', fontsize=10)
|
||||
ax.tick_params(labelsize=8)
|
||||
plt.tight_layout()
|
||||
plt.savefig('weekly_sparkline.png', dpi=150, bbox_inches='tight')
|
||||
plt.close()
|
||||
|
||||
return report
|
||||
```
|
||||
|
||||
### Anomaly Detection Patterns
|
||||
|
||||
```python
|
||||
def detect_anomalies(series, method='zscore', threshold=3.0, window=30):
|
||||
"""
|
||||
Detect anomalies in a numeric series.
|
||||
|
||||
Methods:
|
||||
- 'zscore': Flag values beyond `threshold` standard deviations from mean
|
||||
- 'iqr': Flag values beyond 1.5x IQR from quartiles
|
||||
- 'rolling': Flag values beyond `threshold` std devs from rolling mean
|
||||
"""
|
||||
anomalies = pd.Series(False, index=series.index)
|
||||
|
||||
if method == 'zscore':
|
||||
z = (series - series.mean()) / series.std()
|
||||
anomalies = z.abs() > threshold
|
||||
|
||||
elif method == 'iqr':
|
||||
q1 = series.quantile(0.25)
|
||||
q3 = series.quantile(0.75)
|
||||
iqr = q3 - q1
|
||||
anomalies = (series < q1 - 1.5 * iqr) | (series > q3 + 1.5 * iqr)
|
||||
|
||||
elif method == 'rolling':
|
||||
rolling_mean = series.rolling(window, min_periods=5).mean()
|
||||
rolling_std = series.rolling(window, min_periods=5).std()
|
||||
anomalies = (series - rolling_mean).abs() > threshold * rolling_std
|
||||
|
||||
return anomalies
|
||||
|
||||
|
||||
# Usage: detect and visualize
|
||||
anomalies = detect_anomalies(df['metric'], method='rolling', threshold=2.5, window=30)
|
||||
|
||||
fig, ax = plt.subplots(figsize=(14, 5))
|
||||
ax.plot(df.index, df['metric'], linewidth=1, color='steelblue', label='Metric')
|
||||
ax.scatter(df.index[anomalies], df['metric'][anomalies],
|
||||
color='red', s=40, zorder=5, label='Anomaly')
|
||||
ax.legend()
|
||||
ax.set_title('Anomaly Detection (Rolling Z-Score)', fontsize=14, fontweight='bold')
|
||||
plt.tight_layout()
|
||||
plt.savefig('anomalies.png', dpi=150, bbox_inches='tight')
|
||||
plt.close()
|
||||
|
||||
print(f"Detected {anomalies.sum()} anomalies out of {len(series):,} data points")
|
||||
```
|
||||
|
||||
**Method selection guide:**
|
||||
|
||||
| Method | Best For | Assumptions | Sensitivity |
|
||||
|--------|----------|-------------|-------------|
|
||||
| Z-score | Stationary data with normal distribution | Constant mean and variance | Low (misses local anomalies) |
|
||||
| IQR | Skewed distributions, outlier screening | None (non-parametric) | Medium |
|
||||
| Rolling z-score | Time series with trends or seasonality | Local stationarity within window | High (adapts to drift) |
|
||||
|
||||
---
|
||||
|
||||
## Common Analytics Pitfalls
|
||||
|
||||
### Simpson's Paradox
|
||||
|
||||
A trend that appears in grouped data reverses when the groups are combined.
|
||||
|
||||
```
|
||||
Department A: Drug works better (80% vs 70%)
|
||||
Department B: Drug works better (50% vs 40%)
|
||||
Combined: Drug appears WORSE (55% vs 60%) <-- paradox
|
||||
```
|
||||
|
||||
**Why it happens**: Unequal group sizes create a confounding effect. Department B (with lower overall rates) sent most patients to the drug group.
|
||||
|
||||
**Prevention**: Always segment data by relevant confounders before drawing conclusions. If aggregate and segmented results disagree, trust the segmented analysis and report the confounding variable.
|
||||
|
||||
### Survivorship Bias
|
||||
|
||||
Analyzing only entities that "survived" a selection process, ignoring those that dropped out.
|
||||
|
||||
**Classic examples:**
|
||||
- Studying only successful companies to find success patterns (ignoring failed companies with the same patterns)
|
||||
- Analyzing only current customers to understand satisfaction (ignoring those who already left)
|
||||
- Looking at fund performance by examining only funds that still exist (dead funds were closed)
|
||||
|
||||
**Prevention**: Always ask "what is missing from this dataset?" before drawing conclusions. If possible, include data from non-survivors. Explicitly note the selection criteria and what it excludes.
|
||||
|
||||
### Correlation vs. Causation
|
||||
|
||||
A statistically significant correlation between X and Y does not mean X causes Y. Possible explanations:
|
||||
|
||||
| Explanation | Example |
|
||||
|-------------|---------|
|
||||
| X causes Y | Exercise reduces blood pressure |
|
||||
| Y causes X | Depression reduces exercise (not exercise causes depression) |
|
||||
| Z causes both | Income drives both education spending AND health outcomes |
|
||||
| Coincidence | Ice cream sales correlate with drowning deaths (both driven by summer) |
|
||||
|
||||
**Prevention**: Establish causation only with randomized controlled experiments (A/B tests). For observational data, state findings as "associated with" not "causes." Look for confounders and test whether the relationship holds when controlling for them.
|
||||
|
||||
### Cherry-Picking Time Windows
|
||||
|
||||
Selecting a start/end date that makes a metric look better or worse than the true trend.
|
||||
|
||||
```python
|
||||
# Example: same data, different conclusions
|
||||
# "Revenue up 40%!" -- comparing Jan (seasonal low) to Dec (seasonal high)
|
||||
# "Revenue flat." -- comparing Dec 2024 to Dec 2025 (year-over-year)
|
||||
|
||||
# Prevention: always use year-over-year comparison for seasonal data
|
||||
df['yoy_change'] = df.groupby(df['date'].dt.month)['revenue'].pct_change(periods=12)
|
||||
```
|
||||
|
||||
**Prevention checklist:**
|
||||
- Compare like-for-like periods (YoY for seasonal businesses)
|
||||
- Show the full time range, not a selected subset
|
||||
- Use multiple time windows (WoW, MoM, QoQ, YoY) and note if they disagree
|
||||
- Include a moving average to show the underlying trend separate from noise
|
||||
|
||||
### Small Sample Size Issues
|
||||
|
||||
Small samples produce unstable statistics that can flip with just a few more observations.
|
||||
|
||||
```python
|
||||
# Illustrate instability: conversion rates with small vs. large samples
|
||||
from scipy.stats import beta
|
||||
|
||||
# Scenario: 3 conversions out of 10 visitors (30%)
|
||||
a_small, b_small = 3 + 1, 10 - 3 + 1 # Beta posterior
|
||||
ci_small = beta.interval(0.95, a_small, b_small)
|
||||
print(f"n=10: 30% conversion, 95% CI: [{ci_small[0]:.1%}, {ci_small[1]:.1%}]")
|
||||
# Output: 95% CI: [9.9%, 56.8%] -- extremely wide, almost useless
|
||||
|
||||
# Scenario: 300 conversions out of 1000 visitors (30%)
|
||||
a_large, b_large = 300 + 1, 1000 - 300 + 1
|
||||
ci_large = beta.interval(0.95, a_large, b_large)
|
||||
print(f"n=1000: 30% conversion, 95% CI: [{ci_large[0]:.1%}, {ci_large[1]:.1%}]")
|
||||
# Output: 95% CI: [27.2%, 32.9%] -- narrow and actionable
|
||||
```
|
||||
|
||||
**Rules of thumb:**
|
||||
- n < 30: Do not draw firm conclusions. Report as directional only.
|
||||
- Conversion rates need hundreds (not dozens) of conversions to stabilize.
|
||||
- Always report confidence intervals alongside point estimates.
|
||||
- If sample size is fixed and small, use exact tests (Fisher's exact) rather than approximations (chi-squared).
|
||||
Reference in new issue
Block a user