Anomaly Detector
Audience: Data engineers and analysts detecting outliers in datasets.
Goal: Provide production-ready anomaly detection functions for various data types.
Scripts
Execute detection functions from scripts/anomaly_detection.py:
from scripts.anomaly_detection import (
detect_anomalies_zscore,
detect_anomalies_iqr,
detect_anomalies_modified_zscore,
detect_anomalies_isolation_forest,
detect_anomalies_lof,
detect_anomalies_rolling,
detect_anomalies_stl,
detect_anomalies_ensemble
)Method Selection
| Method | Best For | Limitations |
|---|---|---|
| Z-Score | Normal distributions | Sensitive to outliers |
| IQR | Skewed distributions | Less sensitive overall |
| Modified Z-Score | Robust detection | Slower computation |
| Isolation Forest | High-dimensional data | Requires tuning |
| LOF | Local density anomalies | Computationally expensive |
| Rolling | Time-series with trends | Window size sensitive |
| STL | Seasonal time-series | Requires known period |
Usage Examples
Single Column Detection
import pandas as pd
from scripts.anomaly_detection import detect_anomalies_zscore, detect_anomalies_iqr
df = pd.read_csv('data.csv')
# Z-score method (good for normal distributions)
anomalies_z = detect_anomalies_zscore(df['value'], threshold=3.0)
# IQR method (robust to skewed data)
anomalies_iqr = detect_anomalies_iqr(df['value'], multiplier=1.5)
print(f"Z-score found {anomalies_z.sum()} anomalies")
print(f"IQR found {anomalies_iqr.sum()} anomalies")Multi-Column with Isolation Forest
from scripts.anomaly_detection import detect_anomalies_isolation_forest
numeric_cols = ['revenue', 'quantity', 'price']
anomalies = detect_anomalies_isolation_forest(df, numeric_cols, contamination=0.01)
df_anomalies = df[anomalies]Ensemble Approach (Recommended)
from scripts.anomaly_detection import detect_anomalies_ensemble
results = detect_anomalies_ensemble(
df,
columns=['revenue', 'quantity'],
methods=['zscore', 'iqr', 'isolation_forest'],
min_agreement=2 # Flag if 2+ methods agree
)
confirmed_anomalies = df[results['is_anomaly']]Time-Series Anomalies
from scripts.anomaly_detection import detect_anomalies_rolling, detect_anomalies_stl
# Rolling window (for trending data)
anomalies = detect_anomalies_rolling(df['daily_sales'], window=7, n_std=2.0)
# STL decomposition (for seasonal data)
anomalies = detect_anomalies_stl(df['monthly_revenue'], period=12, threshold=3.0)Dependencies
pandas
numpy
scikit-learn # For Isolation Forest, LOF
statsmodels # For STL decomposition