---
name: analytics-hand-skill
version: "1.0.0"
description: "Expert knowledge for AI data analytics -- statistical methods, visualization best practices, pandas reference, and reporting patterns"
runtime: prompt_only
---

# Data Analytics Expert Knowledge

## pandas Quick Reference

### Data Loading
```python
import pandas as pd

# CSV
df = pd.read_csv('data.csv')
df = pd.read_csv('data.csv', parse_dates=['date_col'], index_col='id')

# JSON
df = pd.read_json('data.json')
df = pd.read_json('data.json', orient='records')

# Excel
df = pd.read_excel('data.xlsx', sheet_name='Sheet1')

# From dict
df = pd.DataFrame({'col1': [1, 2, 3], 'col2': ['a', 'b', 'c']})
```

### Data Inspection
```python
df.shape              # (rows, columns)
df.dtypes             # Column types
df.info()             # Summary including memory usage
df.describe()         # Statistical summary
df.head(10)           # First 10 rows
df.isnull().sum()     # Missing values per column
df.duplicated().sum() # Number of duplicate rows
df.nunique()          # Unique values per column
```

### Data Cleaning
```python
# Handle missing values
df.dropna()                          # Drop rows with any NaN
df.fillna(0)                         # Fill NaN with 0
df.fillna(df.mean())                 # Fill with column means
df['col'].interpolate()              # Interpolate missing values

# Remove duplicates
df.drop_duplicates()
df.drop_duplicates(subset=['col1', 'col2'])

# Type conversion
df['col'] = df['col'].astype(int)
df['date'] = pd.to_datetime(df['date'])
df['cat'] = df['cat'].astype('category')

# Outlier removal (IQR method)
Q1 = df['col'].quantile(0.25)
Q3 = df['col'].quantile(0.75)
IQR = Q3 - Q1
df = df[(df['col'] >= Q1 - 1.5*IQR) & (df['col'] <= Q3 + 1.5*IQR)]
```

### Aggregation & Grouping
```python
# Group by
df.groupby('category').agg({'value': ['mean', 'sum', 'count']})

# Pivot table
pd.pivot_table(df, values='value', index='row_cat', columns='col_cat', aggfunc='mean')

# Cross tabulation
pd.crosstab(df['cat1'], df['cat2'])

# Rolling statistics
df['rolling_mean'] = df['value'].rolling(window=7).mean()

# Percentage change
df['pct_change'] = df['value'].pct_change()
```

### Time Series
```python
# Set datetime index
df.set_index('date', inplace=True)

# Resample
df.resample('W').mean()   # Weekly average
df.resample('M').sum()    # Monthly sum
df.resample('Q').count()  # Quarterly count

# Date range
pd.date_range(start='2025-01-01', periods=30, freq='D')

# Shift/Lag
df['prev_value'] = df['value'].shift(1)
df['next_value'] = df['value'].shift(-1)
```

---

## Visualization Best Practices

### matplotlib + seaborn Reference

```python
import matplotlib
matplotlib.use('Agg')  # Non-interactive backend
import matplotlib.pyplot as plt
import seaborn as sns

# Set style
sns.set_theme(style='whitegrid')
plt.rcParams['figure.figsize'] = (10, 6)
```

### Chart Selection Guide

| Data Type | Question | Chart Type |
|-----------|----------|------------|
| Categorical | Comparison | Bar chart |
| Categorical | Proportion | Pie chart (if <6 categories) |
| Numerical | Distribution | Histogram / Box plot |
| Two numerical | Relationship | Scatter plot |
| Time series | Trend | Line chart |
| Matrix | Correlation | Heatmap |
| Categories + values | Comparison | Grouped bar / Stacked bar |
| Geographical | Location | Map / Choropleth |

### Chart Templates

**Bar Chart**:
```python
fig, ax = plt.subplots(figsize=(10, 6))
data = df['category'].value_counts()
data.plot(kind='bar', ax=ax, color='steelblue')
ax.set_title('Distribution by Category', fontsize=14, fontweight='bold')
ax.set_xlabel('Category')
ax.set_ylabel('Count')
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.savefig('bar_chart.png', dpi=150, bbox_inches='tight')
plt.close()
```

**Line Chart (Time Series)**:
```python
fig, ax = plt.subplots(figsize=(12, 6))
ax.plot(df.index, df['value'], linewidth=2, color='steelblue')
ax.fill_between(df.index, df['value'], alpha=0.1, color='steelblue')
ax.set_title('Trend Over Time', fontsize=14, fontweight='bold')
ax.set_xlabel('Date')
ax.set_ylabel('Value')
plt.tight_layout()
plt.savefig('line_chart.png', dpi=150, bbox_inches='tight')
plt.close()
```

**Correlation Heatmap**:
```python
fig, ax = plt.subplots(figsize=(10, 8))
corr = df.select_dtypes(include='number').corr()
sns.heatmap(corr, annot=True, fmt='.2f', cmap='RdBu_r', center=0, ax=ax)
ax.set_title('Correlation Matrix', fontsize=14, fontweight='bold')
plt.tight_layout()
plt.savefig('heatmap.png', dpi=150, bbox_inches='tight')
plt.close()
```

**Scatter Plot**:
```python
fig, ax = plt.subplots(figsize=(10, 6))
ax.scatter(df['x'], df['y'], alpha=0.6, edgecolors='black', linewidth=0.5)
ax.set_title('X vs Y', fontsize=14, fontweight='bold')
ax.set_xlabel('X Variable')
ax.set_ylabel('Y Variable')
plt.tight_layout()
plt.savefig('scatter.png', dpi=150, bbox_inches='tight')
plt.close()
```

### Visualization Do's and Don'ts

**Do**:
- Start y-axis at 0 for bar charts
- Use consistent colors across related charts
- Label axes clearly with units
- Add titles that describe the insight, not just the data
- Use appropriate scales (log scale for exponential data)

**Don't**:
- Use 3D charts (distorts perception)
- Use more than 6-7 colors in one chart
- Truncate axes to exaggerate differences
- Use pie charts for more than 5 categories
- Add unnecessary chart junk (borders, backgrounds, grids)

---

## Statistical Methods

### Descriptive Statistics
| Measure | pandas | Purpose |
|---------|--------|---------|
| Mean | `df['col'].mean()` | Central tendency |
| Median | `df['col'].median()` | Robust central tendency |
| Std Dev | `df['col'].std()` | Variability |
| Skewness | `df['col'].skew()` | Distribution symmetry |
| Kurtosis | `df['col'].kurtosis()` | Distribution tails |
| Percentiles | `df['col'].quantile([0.25, 0.5, 0.75])` | Distribution spread |

### Correlation Analysis
```python
# Pearson correlation (linear)
df['col1'].corr(df['col2'])

# Spearman correlation (monotonic)
df['col1'].corr(df['col2'], method='spearman')

# Full correlation matrix
df.select_dtypes(include='number').corr()
```

Interpretation:
- |r| > 0.7: Strong correlation
- 0.4 < |r| < 0.7: Moderate correlation
- |r| < 0.4: Weak correlation
- Correlation != Causation

### Hypothesis Testing (scipy)
```python
from scipy import stats

# T-test (compare two group means)
t_stat, p_value = stats.ttest_ind(group1, group2)

# Chi-squared test (categorical independence)
chi2, p_value, dof, expected = stats.chi2_contingency(contingency_table)

# Significance: p < 0.05 is commonly used threshold

# Mann-Whitney U test (non-parametric alternative to t-test)
u_stat, p_value = stats.mannwhitneyu(group1, group2, alternative='two-sided')

# One-way ANOVA (compare 3+ group means)
f_stat, p_value = stats.f_oneway(group1, group2, group3)

# Normality check (determines which test to use)
shapiro_stat, p_value = stats.shapiro(data)  # p > 0.05 means normal
```

### Statistical Significance Decision Guide

**Test selection flowchart:**
| Data Situation | Normal Distribution? | Test to Use |
|---------------|---------------------|-------------|
| Compare 2 group means | Yes | Independent t-test (`ttest_ind`) |
| Compare 2 group means | No | Mann-Whitney U (`mannwhitneyu`) |
| Compare 3+ group means | Yes | One-way ANOVA (`f_oneway`) |
| Compare 3+ group means | No | Kruskal-Wallis (`kruskal`) |
| Compare paired samples | Yes | Paired t-test (`ttest_rel`) |
| Compare paired samples | No | Wilcoxon signed-rank (`wilcoxon`) |
| Test categorical independence | N/A | Chi-squared (`chi2_contingency`) |
| Test correlation | Yes | Pearson (`pearsonr`) |
| Test correlation | No | Spearman (`spearmanr`) |

**P-value interpretation:**
| p-value | Interpretation | Action |
|---------|---------------|--------|
| p < 0.01 | Strong evidence against null hypothesis | Report as statistically significant |
| 0.01 ≤ p < 0.05 | Moderate evidence | Report as significant with caveat |
| 0.05 ≤ p < 0.10 | Weak evidence | Report as marginally significant |
| p ≥ 0.10 | Insufficient evidence | Do not claim significance |

**Practical significance — always report effect size:**
```python
# Cohen's d for comparing two means
def cohens_d(group1, group2):
    n1, n2 = len(group1), len(group2)
    var1, var2 = group1.var(), group2.var()
    pooled_std = ((n1 - 1) * var1 + (n2 - 1) * var2) / (n1 + n2 - 2)
    return (group1.mean() - group2.mean()) / (pooled_std ** 0.5)

# Interpretation: |d| < 0.2 = negligible, 0.2-0.5 = small, 0.5-0.8 = medium, > 0.8 = large
```

**Sample size awareness:**
- n < 30: Use non-parametric tests; results are exploratory
- 30 ≤ n < 100: Parametric tests OK if normality holds; moderate confidence
- n ≥ 100: Central Limit Theorem applies; high confidence in parametric tests
- Always report sample size alongside p-values

**Confidence threshold mapping:**
| Setting | p-value threshold | Minimum effect size | Minimum sample size |
|---------|------------------|--------------------|--------------------|
| High | p < 0.01 | Cohen's d ≥ 0.5 | n ≥ 100 |
| Medium | p < 0.05 | Cohen's d ≥ 0.3 | n ≥ 30 |
| Low | p < 0.10 | Any | Any |

---

## Report Structure Best Practices

### CRISP-DM Framework
1. **Business Understanding**: What question are we answering?
2. **Data Understanding**: What data do we have? Quality?
3. **Data Preparation**: Cleaning, transformation, feature engineering
4. **Modeling**: Statistical analysis, ML models
5. **Evaluation**: Are results valid and useful?
6. **Deployment**: Reports, dashboards, recommendations

### Insight Hierarchy
```
Level 1: What happened (descriptive)
  "Revenue increased 15% last quarter"

Level 2: Why it happened (diagnostic)
  "Revenue increase driven by 30% growth in enterprise segment"

Level 3: What will happen (predictive)
  "Based on current trends, Q2 revenue projected at $X"

Level 4: What to do (prescriptive)
  "Invest in enterprise sales team to capitalize on growth trajectory"
```

### Data Quality Assessment Template
```
| Dimension | Score | Details |
|-----------|-------|---------|
| Completeness | 85% | 15% missing values in 'email' column |
| Accuracy | High | Validated against source system |
| Consistency | Medium | Date formats vary across sources |
| Timeliness | Current | Data refreshed daily |
| Uniqueness | 99% | 1% duplicate records found |
```

---

## Worked Examples

### Example 1: E-commerce Sales Analysis

**Goal**: Analyze 12 months of order data to identify revenue drivers, customer segments, and growth trends.

#### Step 1 — Load and clean
```python
import pandas as pd
import numpy as np

df = pd.read_csv('orders.csv', parse_dates=['order_date'])

# Quick audit
print(f"Rows: {len(df):,}  Columns: {df.shape[1]}")
print(df.isnull().sum()[df.isnull().sum() > 0])

# Clean
df = df.dropna(subset=['customer_id', 'order_total'])
df['order_total'] = df['order_total'].clip(lower=0)  # Remove negative values
df['order_month'] = df['order_date'].dt.to_period('M')
```

#### Step 2 — Revenue trend analysis
```python
monthly = (
    df.groupby('order_month')
    .agg(revenue=('order_total', 'sum'),
         orders=('order_id', 'nunique'),
         customers=('customer_id', 'nunique'))
    .reset_index()
)
monthly['aov'] = monthly['revenue'] / monthly['orders']  # Average order value
monthly['revenue_mom'] = monthly['revenue'].pct_change()  # Month-over-month growth

fig, axes = plt.subplots(2, 1, figsize=(12, 8), sharex=True)
axes[0].bar(monthly['order_month'].astype(str), monthly['revenue'], color='steelblue')
axes[0].set_title('Monthly Revenue', fontsize=14, fontweight='bold')
axes[0].set_ylabel('Revenue ($)')

axes[1].plot(monthly['order_month'].astype(str), monthly['aov'], marker='o', color='coral')
axes[1].set_title('Average Order Value', fontsize=14, fontweight='bold')
axes[1].set_ylabel('AOV ($)')
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.savefig('revenue_trend.png', dpi=150, bbox_inches='tight')
plt.close()
```

#### Step 3 — Customer segmentation (RFM)
```python
snapshot_date = df['order_date'].max() + pd.Timedelta(days=1)

rfm = df.groupby('customer_id').agg(
    recency=('order_date', lambda x: (snapshot_date - x.max()).days),
    frequency=('order_id', 'nunique'),
    monetary=('order_total', 'sum')
)

# Score each dimension 1-4 using quartiles
for col in ['recency', 'frequency', 'monetary']:
    labels = [4, 3, 2, 1] if col == 'recency' else [1, 2, 3, 4]
    rfm[f'{col}_score'] = pd.qcut(rfm[col], q=4, labels=labels, duplicates='drop')

rfm['rfm_score'] = (rfm['recency_score'].astype(int)
                     + rfm['frequency_score'].astype(int)
                     + rfm['monetary_score'].astype(int))

# Segment mapping
def segment(row):
    r, f, m = int(row['recency_score']), int(row['frequency_score']), int(row['monetary_score'])
    if r >= 3 and f >= 3:
        return 'Champions'
    elif r >= 3 and f < 3:
        return 'New / Promising'
    elif r < 3 and f >= 3:
        return 'At Risk'
    else:
        return 'Needs Attention'

rfm['segment'] = rfm.apply(segment, axis=1)
print(rfm.groupby('segment').agg(
    count=('monetary', 'size'),
    avg_revenue=('monetary', 'mean'),
    avg_frequency=('frequency', 'mean')
).sort_values('avg_revenue', ascending=False))
```

#### Step 4 — Cohort retention analysis
```python
df['cohort'] = df.groupby('customer_id')['order_date'].transform('min').dt.to_period('M')
df['order_period'] = df['order_date'].dt.to_period('M')
df['cohort_index'] = (df['order_period'] - df['cohort']).apply(lambda x: x.n)

cohort_table = (
    df.groupby(['cohort', 'cohort_index'])['customer_id']
    .nunique()
    .reset_index()
    .pivot(index='cohort', columns='cohort_index', values='customer_id')
)

# Convert to retention percentages
retention = cohort_table.div(cohort_table[0], axis=0) * 100

fig, ax = plt.subplots(figsize=(14, 8))
sns.heatmap(retention, annot=True, fmt='.0f', cmap='YlOrRd_r', ax=ax)
ax.set_title('Cohort Retention (% of original customers)', fontsize=14, fontweight='bold')
ax.set_xlabel('Months Since First Purchase')
ax.set_ylabel('Cohort')
plt.tight_layout()
plt.savefig('cohort_retention.png', dpi=150, bbox_inches='tight')
plt.close()
```

---

### Example 2: A/B Test Analysis

**Goal**: Evaluate whether a new checkout flow (variant B) improves conversion rate over the existing flow (variant A).

#### Step 1 — Sample size calculation (pre-test)
```python
from scipy import stats
import numpy as np

baseline_rate = 0.12      # Current conversion rate: 12%
mde = 0.02                # Minimum detectable effect: 2 percentage points
alpha = 0.05              # Significance level
power = 0.80              # Statistical power

# Using the normal approximation formula
p1 = baseline_rate
p2 = baseline_rate + mde
p_avg = (p1 + p2) / 2

z_alpha = stats.norm.ppf(1 - alpha / 2)  # Two-tailed
z_beta = stats.norm.ppf(power)

n_per_group = ((z_alpha * np.sqrt(2 * p_avg * (1 - p_avg))
                + z_beta * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
               / (p2 - p1) ** 2)

print(f"Required sample size per group: {int(np.ceil(n_per_group)):,}")
print(f"Total required: {int(np.ceil(n_per_group)) * 2:,}")
```

#### Step 2 — Run the test and collect results
```python
ab = pd.read_csv('ab_test_results.csv')

summary = ab.groupby('variant').agg(
    visitors=('user_id', 'nunique'),
    conversions=('converted', 'sum')
)
summary['conversion_rate'] = summary['conversions'] / summary['visitors']
print(summary)
```

#### Step 3 — Statistical significance
```python
a = ab[ab['variant'] == 'A']
b = ab[ab['variant'] == 'B']

# Chi-squared test for proportions
contingency = pd.crosstab(ab['variant'], ab['converted'])
chi2, p_value, dof, expected = stats.chi2_contingency(contingency)

# Proportions z-test (more direct)
from statsmodels.stats.proportion import proportions_ztest
successes = [summary.loc['B', 'conversions'], summary.loc['A', 'conversions']]
trials = [summary.loc['B', 'visitors'], summary.loc['A', 'visitors']]
z_stat, p_val = proportions_ztest(successes, trials, alternative='larger')

print(f"Z-statistic: {z_stat:.4f}")
print(f"P-value:     {p_val:.4f}")
print(f"Significant: {'Yes' if p_val < 0.05 else 'No'} (at alpha=0.05)")
```

#### Step 4 — Effect size and confidence interval
```python
p_a = summary.loc['A', 'conversion_rate']
p_b = summary.loc['B', 'conversion_rate']
n_a = summary.loc['A', 'visitors']
n_b = summary.loc['B', 'visitors']

lift = (p_b - p_a) / p_a
se_diff = np.sqrt(p_a * (1 - p_a) / n_a + p_b * (1 - p_b) / n_b)
ci_lower = (p_b - p_a) - 1.96 * se_diff
ci_upper = (p_b - p_a) + 1.96 * se_diff

print(f"Control rate:        {p_a:.4f}")
print(f"Variant rate:        {p_b:.4f}")
print(f"Absolute lift:       {p_b - p_a:+.4f}")
print(f"Relative lift:       {lift:+.2%}")
print(f"95% CI for diff:     [{ci_lower:+.4f}, {ci_upper:+.4f}]")
```

#### Step 5 — Recommendation template
```
## A/B Test Report: New Checkout Flow

| Metric              | Control (A) | Variant (B) |
|---------------------|-------------|-------------|
| Visitors            | 15,204      | 15,198      |
| Conversions         | 1,824       | 2,127       |
| Conversion Rate     | 12.00%      | 13.99%      |

**Result**: Statistically significant (p = 0.0003, alpha = 0.05)
**Lift**: +1.99pp absolute / +16.6% relative
**95% CI**: [+0.90pp, +3.08pp]
**Recommendation**: Deploy variant B. The effect is both statistically
and practically significant with a lower bound above the +1pp threshold.
```

---

### Example 3: Customer Churn Analysis

**Goal**: Identify which factors most strongly predict customer churn and quantify their relative importance.

#### Step 1 — Feature engineering
```python
df = pd.read_csv('customers.csv')

# Create behavioral features from raw data
features = df.copy()
features['tenure_months'] = (pd.Timestamp.now() - pd.to_datetime(df['signup_date'])).dt.days / 30
features['support_tickets_per_month'] = df['total_tickets'] / features['tenure_months'].clip(lower=1)
features['avg_session_minutes'] = df['total_session_minutes'] / df['total_sessions'].clip(lower=1)
features['days_since_last_login'] = (pd.Timestamp.now() - pd.to_datetime(df['last_login'])).dt.days
features['has_premium'] = (df['plan'] == 'premium').astype(int)

# Drop raw columns, keep engineered features
feature_cols = [
    'tenure_months', 'support_tickets_per_month', 'avg_session_minutes',
    'days_since_last_login', 'has_premium', 'monthly_spend', 'num_features_used'
]
```

#### Step 2 — Correlation analysis
```python
churn_corr = features[feature_cols + ['churned']].corr()['churned'].drop('churned').sort_values()

fig, ax = plt.subplots(figsize=(8, 5))
churn_corr.plot(kind='barh', ax=ax, color=['coral' if x > 0 else 'steelblue' for x in churn_corr])
ax.set_title('Feature Correlation with Churn', fontsize=14, fontweight='bold')
ax.set_xlabel('Pearson Correlation')
ax.axvline(x=0, color='black', linewidth=0.5)
plt.tight_layout()
plt.savefig('churn_correlations.png', dpi=150, bbox_inches='tight')
plt.close()
```

#### Step 3 — Key driver identification via group comparison
```python
churned = features[features['churned'] == 1]
retained = features[features['churned'] == 0]

comparison = []
for col in feature_cols:
    t_stat, p_val = stats.ttest_ind(churned[col].dropna(), retained[col].dropna())
    d = cohens_d(churned[col].dropna(), retained[col].dropna())  # From earlier definition
    comparison.append({
        'feature': col,
        'churned_mean': churned[col].mean(),
        'retained_mean': retained[col].mean(),
        'diff_pct': (churned[col].mean() - retained[col].mean()) / retained[col].mean() * 100,
        'cohens_d': abs(d),
        'p_value': p_val,
        'significant': p_val < 0.05
    })

result = pd.DataFrame(comparison).sort_values('cohens_d', ascending=False)
print(result.to_string(index=False))
```

#### Step 4 — Interpret and report
```
## Churn Driver Analysis

**Top 3 factors distinguishing churned vs. retained customers:**

| Factor                     | Churned (avg) | Retained (avg) | Diff     | Effect Size |
|----------------------------|---------------|----------------|----------|-------------|
| Days since last login      | 34.2          | 8.7            | +293%    | Large       |
| Support tickets per month  | 2.8           | 0.9            | +211%    | Large       |
| Number of features used    | 3.1           | 7.4            | -58%     | Medium      |

**Actionable insights:**
1. Customers inactive >14 days are 4x more likely to churn -- trigger re-engagement email at day 10
2. High support ticket rate signals frustration -- escalate accounts with >2 tickets/month to success team
3. Low feature adoption correlates with churn -- implement onboarding flow targeting unused features
```

---

## Advanced pandas Patterns

### Window Functions

```python
# Expanding window (cumulative statistics)
df['cumulative_avg'] = df['value'].expanding().mean()
df['cumulative_max'] = df['value'].expanding().max()

# Exponentially weighted moving average (EWMA) -- emphasizes recent values
df['ewma_7'] = df['value'].ewm(span=7).mean()     # Span-based decay
df['ewma_a'] = df['value'].ewm(alpha=0.3).mean()   # Explicit decay factor

# Comparison: rolling vs. EWMA
# - rolling(7).mean() weights all 7 values equally
# - ewm(span=7).mean() weights recent values exponentially more
# Use EWMA when recent data matters more (stock prices, real-time metrics)

# Rolling with min_periods (handles early rows with insufficient data)
df['rolling_avg'] = df['value'].rolling(window=30, min_periods=5).mean()

# Rolling rank (percentile within window)
df['rolling_pctile'] = df['value'].rolling(90).rank(pct=True)
```

### Multi-Index Operations

```python
# Create multi-index from groupby
multi = df.groupby(['region', 'product']).agg(
    revenue=('amount', 'sum'),
    units=('quantity', 'sum')
)

# Access levels
multi.loc['North']                        # All products in North region
multi.loc[('North', 'Widget')]            # Specific region + product
multi.xs('Widget', level='product')       # All regions for Widget

# Swap and sort levels
multi = multi.swaplevel().sort_index()

# Reset to flat columns
flat = multi.reset_index()

# Stack / unstack (reshape between long and wide)
wide = multi['revenue'].unstack(level='product')    # Products become columns
long = wide.stack()                                  # Back to multi-index
```

### Merge and Join Patterns

```python
# Inner join (only matching rows)
merged = orders.merge(customers, on='customer_id', how='inner')

# Left join with indicator (see which rows matched)
merged = orders.merge(customers, on='customer_id', how='left', indicator=True)
unmatched = merged[merged['_merge'] == 'left_only']

# Join on multiple keys
merged = df1.merge(df2, on=['date', 'region'], how='left')

# Join with different column names
merged = orders.merge(products, left_on='prod_id', right_on='product_id')

# Anti-join (rows in A that have no match in B)
anti = df_a.merge(df_b, on='key', how='left', indicator=True)
anti = anti[anti['_merge'] == 'left_only'].drop(columns='_merge')

# Self-join (compare rows within same table)
df_prev = df[['customer_id', 'order_date', 'amount']].rename(
    columns={'order_date': 'prev_date', 'amount': 'prev_amount'}
)
df_with_prev = df.merge(df_prev, on='customer_id', how='left')
df_with_prev = df_with_prev[df_with_prev['prev_date'] < df_with_prev['order_date']]
```

### Apply and Transform

```python
# transform() returns same-shaped output -- useful for group-level stats on each row
df['group_mean'] = df.groupby('category')['value'].transform('mean')
df['pct_of_group'] = df['value'] / df.groupby('category')['value'].transform('sum')
df['z_within_group'] = df.groupby('category')['value'].transform(
    lambda x: (x - x.mean()) / x.std()
)

# apply() for multi-column group operations
def top_n(group, n=3):
    return group.nlargest(n, 'value')

top3_per_category = df.groupby('category', group_keys=False).apply(top_n, n=3)

# Vectorized operations (prefer these over apply when possible)
# Slow:
df['result'] = df.apply(lambda row: row['a'] * row['b'] + row['c'], axis=1)
# Fast:
df['result'] = df['a'] * df['b'] + df['c']

# np.where for conditional columns (vectorized if/else)
df['tier'] = np.where(df['revenue'] > 10000, 'high', 'low')

# np.select for multiple conditions
conditions = [
    df['revenue'] > 10000,
    df['revenue'] > 5000,
    df['revenue'] > 0,
]
choices = ['high', 'medium', 'low']
df['tier'] = np.select(conditions, choices, default='none')
```

### Memory Optimization for Large Datasets

```python
# Check current memory usage
print(df.memory_usage(deep=True).sum() / 1024**2, "MB")

# Downcast numeric types
df['int_col'] = pd.to_numeric(df['int_col'], downcast='integer')    # int64 -> int8/16/32
df['float_col'] = pd.to_numeric(df['float_col'], downcast='float')  # float64 -> float32

# Use category type for low-cardinality strings
for col in df.select_dtypes(include='object'):
    if df[col].nunique() / len(df) < 0.5:  # Less than 50% unique values
        df[col] = df[col].astype('category')

# Read in chunks for files that exceed memory
chunks = pd.read_csv('huge_file.csv', chunksize=100_000)
results = []
for chunk in chunks:
    processed = chunk.groupby('category')['value'].sum()
    results.append(processed)
final = pd.concat(results).groupby(level=0).sum()

# Specify dtypes at load time (avoids loading as float64/object first)
dtypes = {
    'id': 'int32',
    'category': 'category',
    'value': 'float32',
    'flag': 'bool'
}
df = pd.read_csv('data.csv', dtype=dtypes)

# Use pyarrow backend for better memory efficiency (pandas 2.0+)
df = pd.read_csv('data.csv', engine='pyarrow', dtype_backend='pyarrow')
```

---

## Dashboard and Reporting Patterns

### Executive Dashboard Template

```python
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
from matplotlib.patches import FancyBboxPatch

def executive_dashboard(kpis, trend_df, comparison_df, output='dashboard.png'):
    """
    kpis: dict with keys like {'Revenue': '$1.2M', 'Growth': '+15%', ...}
    trend_df: DataFrame with 'date' and 'value' columns
    comparison_df: DataFrame with 'category' and 'current'/'previous' columns
    """
    fig = plt.figure(figsize=(16, 10))
    gs = gridspec.GridSpec(3, len(kpis), hspace=0.4, wspace=0.3)

    # Row 1: KPI cards
    for i, (label, value) in enumerate(kpis.items()):
        ax = fig.add_subplot(gs[0, i])
        ax.text(0.5, 0.6, value, ha='center', va='center',
                fontsize=28, fontweight='bold', color='#2c3e50')
        ax.text(0.5, 0.2, label, ha='center', va='center',
                fontsize=12, color='#7f8c8d')
        ax.set_xlim(0, 1)
        ax.set_ylim(0, 1)
        ax.axis('off')
        # Card background
        rect = FancyBboxPatch((0.05, 0.05), 0.9, 0.9, boxstyle="round,pad=0.05",
                              facecolor='#f8f9fa', edgecolor='#dee2e6')
        ax.add_patch(rect)

    # Row 2: Trend line
    ax_trend = fig.add_subplot(gs[1, :])
    ax_trend.plot(trend_df['date'], trend_df['value'], linewidth=2, color='steelblue')
    ax_trend.fill_between(trend_df['date'], trend_df['value'], alpha=0.1, color='steelblue')
    ax_trend.set_title('Trend Over Time', fontsize=13, fontweight='bold')
    ax_trend.set_ylabel('Value')

    # Row 3: Period comparison (grouped bar)
    ax_comp = fig.add_subplot(gs[2, :])
    x = range(len(comparison_df))
    width = 0.35
    ax_comp.bar([i - width/2 for i in x], comparison_df['previous'], width,
                label='Previous', color='#bdc3c7')
    ax_comp.bar([i + width/2 for i in x], comparison_df['current'], width,
                label='Current', color='steelblue')
    ax_comp.set_xticks(list(x))
    ax_comp.set_xticklabels(comparison_df['category'], rotation=45, ha='right')
    ax_comp.set_title('Current vs. Previous Period', fontsize=13, fontweight='bold')
    ax_comp.legend()

    plt.savefig(output, dpi=150, bbox_inches='tight', facecolor='white')
    plt.close()
```

### Weekly Metrics Report Template

```python
def weekly_report(df, date_col='date', metric_col='value', group_col=None):
    """Generate a standard weekly metrics summary."""
    df[date_col] = pd.to_datetime(df[date_col])
    df['week'] = df[date_col].dt.isocalendar().week.astype(int)
    df['year'] = df[date_col].dt.year

    current_week = df['week'].max()
    prev_week = current_week - 1

    curr = df[df['week'] == current_week]
    prev = df[df['week'] == prev_week]

    report = {
        'period': f"Week {current_week}",
        'total': curr[metric_col].sum(),
        'mean': curr[metric_col].mean(),
        'median': curr[metric_col].median(),
        'wow_change': (curr[metric_col].sum() - prev[metric_col].sum())
                      / prev[metric_col].sum() * 100
                      if prev[metric_col].sum() != 0 else None,
    }

    if group_col:
        report['by_group'] = curr.groupby(group_col)[metric_col].agg(['sum', 'mean', 'count'])

    # Sparkline trend (last 8 weeks)
    weekly_totals = (
        df.groupby('week')[metric_col].sum()
        .tail(8)
        .reset_index()
    )

    fig, ax = plt.subplots(figsize=(6, 2))
    ax.plot(weekly_totals['week'], weekly_totals[metric_col], marker='o',
            linewidth=2, color='steelblue', markersize=4)
    ax.fill_between(weekly_totals['week'], weekly_totals[metric_col],
                    alpha=0.1, color='steelblue')
    ax.set_title(f'{metric_col.title()} — Last 8 Weeks', fontsize=10)
    ax.tick_params(labelsize=8)
    plt.tight_layout()
    plt.savefig('weekly_sparkline.png', dpi=150, bbox_inches='tight')
    plt.close()

    return report
```

### Anomaly Detection Patterns

```python
def detect_anomalies(series, method='zscore', threshold=3.0, window=30):
    """
    Detect anomalies in a numeric series.

    Methods:
    - 'zscore':  Flag values beyond `threshold` standard deviations from mean
    - 'iqr':     Flag values beyond 1.5x IQR from quartiles
    - 'rolling': Flag values beyond `threshold` std devs from rolling mean
    """
    anomalies = pd.Series(False, index=series.index)

    if method == 'zscore':
        z = (series - series.mean()) / series.std()
        anomalies = z.abs() > threshold

    elif method == 'iqr':
        q1 = series.quantile(0.25)
        q3 = series.quantile(0.75)
        iqr = q3 - q1
        anomalies = (series < q1 - 1.5 * iqr) | (series > q3 + 1.5 * iqr)

    elif method == 'rolling':
        rolling_mean = series.rolling(window, min_periods=5).mean()
        rolling_std = series.rolling(window, min_periods=5).std()
        anomalies = (series - rolling_mean).abs() > threshold * rolling_std

    return anomalies


# Usage: detect and visualize
anomalies = detect_anomalies(df['metric'], method='rolling', threshold=2.5, window=30)

fig, ax = plt.subplots(figsize=(14, 5))
ax.plot(df.index, df['metric'], linewidth=1, color='steelblue', label='Metric')
ax.scatter(df.index[anomalies], df['metric'][anomalies],
           color='red', s=40, zorder=5, label='Anomaly')
ax.legend()
ax.set_title('Anomaly Detection (Rolling Z-Score)', fontsize=14, fontweight='bold')
plt.tight_layout()
plt.savefig('anomalies.png', dpi=150, bbox_inches='tight')
plt.close()

print(f"Detected {anomalies.sum()} anomalies out of {len(series):,} data points")
```

**Method selection guide:**

| Method | Best For | Assumptions | Sensitivity |
|--------|----------|-------------|-------------|
| Z-score | Stationary data with normal distribution | Constant mean and variance | Low (misses local anomalies) |
| IQR | Skewed distributions, outlier screening | None (non-parametric) | Medium |
| Rolling z-score | Time series with trends or seasonality | Local stationarity within window | High (adapts to drift) |

---

## Common Analytics Pitfalls

### Simpson's Paradox

A trend that appears in grouped data reverses when the groups are combined.

```
Department A: Drug works better     (80% vs 70%)
Department B: Drug works better     (50% vs 40%)
Combined:     Drug appears WORSE    (55% vs 60%)  <-- paradox
```

**Why it happens**: Unequal group sizes create a confounding effect. Department B (with lower overall rates) sent most patients to the drug group.

**Prevention**: Always segment data by relevant confounders before drawing conclusions. If aggregate and segmented results disagree, trust the segmented analysis and report the confounding variable.

### Survivorship Bias

Analyzing only entities that "survived" a selection process, ignoring those that dropped out.

**Classic examples:**
- Studying only successful companies to find success patterns (ignoring failed companies with the same patterns)
- Analyzing only current customers to understand satisfaction (ignoring those who already left)
- Looking at fund performance by examining only funds that still exist (dead funds were closed)

**Prevention**: Always ask "what is missing from this dataset?" before drawing conclusions. If possible, include data from non-survivors. Explicitly note the selection criteria and what it excludes.

### Correlation vs. Causation

A statistically significant correlation between X and Y does not mean X causes Y. Possible explanations:

| Explanation | Example |
|-------------|---------|
| X causes Y | Exercise reduces blood pressure |
| Y causes X | Depression reduces exercise (not exercise causes depression) |
| Z causes both | Income drives both education spending AND health outcomes |
| Coincidence | Ice cream sales correlate with drowning deaths (both driven by summer) |

**Prevention**: Establish causation only with randomized controlled experiments (A/B tests). For observational data, state findings as "associated with" not "causes." Look for confounders and test whether the relationship holds when controlling for them.

### Cherry-Picking Time Windows

Selecting a start/end date that makes a metric look better or worse than the true trend.

```python
# Example: same data, different conclusions
# "Revenue up 40%!"    -- comparing Jan (seasonal low) to Dec (seasonal high)
# "Revenue flat."       -- comparing Dec 2024 to Dec 2025 (year-over-year)

# Prevention: always use year-over-year comparison for seasonal data
df['yoy_change'] = df.groupby(df['date'].dt.month)['revenue'].pct_change(periods=12)
```

**Prevention checklist:**
- Compare like-for-like periods (YoY for seasonal businesses)
- Show the full time range, not a selected subset
- Use multiple time windows (WoW, MoM, QoQ, YoY) and note if they disagree
- Include a moving average to show the underlying trend separate from noise

### Small Sample Size Issues

Small samples produce unstable statistics that can flip with just a few more observations.

```python
# Illustrate instability: conversion rates with small vs. large samples
from scipy.stats import beta

# Scenario: 3 conversions out of 10 visitors (30%)
a_small, b_small = 3 + 1, 10 - 3 + 1   # Beta posterior
ci_small = beta.interval(0.95, a_small, b_small)
print(f"n=10:   30% conversion, 95% CI: [{ci_small[0]:.1%}, {ci_small[1]:.1%}]")
# Output: 95% CI: [9.9%, 56.8%]  -- extremely wide, almost useless

# Scenario: 300 conversions out of 1000 visitors (30%)
a_large, b_large = 300 + 1, 1000 - 300 + 1
ci_large = beta.interval(0.95, a_large, b_large)
print(f"n=1000: 30% conversion, 95% CI: [{ci_large[0]:.1%}, {ci_large[1]:.1%}]")
# Output: 95% CI: [27.2%, 32.9%]  -- narrow and actionable
```

**Rules of thumb:**
- n < 30: Do not draw firm conclusions. Report as directional only.
- Conversion rates need hundreds (not dozens) of conversions to stabilize.
- Always report confidence intervals alongside point estimates.
- If sample size is fixed and small, use exact tests (Fisher's exact) rather than approximations (chi-squared).