Correlation Analysis
Overview
Correlation analysis measures the strength and direction of relationships between variables, helping identify which features are related and detect multicollinearity.
When to Use
- Identifying relationships between numerical variables
- Detecting multicollinearity before regression modeling
- Exploratory data analysis to understand feature dependencies
- Feature selection and dimensionality reduction
- Validating assumptions about variable relationships
- Comparing linear and non-linear associations
Correlation Types
- Pearson: Linear correlation (continuous variables)
- Spearman: Rank-based correlation (ordinal/non-linear)
- Kendall: Rank correlation (robust alternative)
- Cramér's V: Association for categorical variables
- Mutual Information: Non-linear dependencies
Key Concepts
- Correlation Coefficient: Ranges from -1 to +1
- Positive Correlation: Variables move together
- Negative Correlation: Variables move oppositely
- Multicollinearity: High correlations between predictors
Implementation with Python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.stats import pearsonr, spearmanr, kendalltau
# Sample data
np.random.seed(42)
n = 200
age = np.random.uniform(20, 70, n)
income = age * 2000 + np.random.normal(0, 10000, n)
education_years = age / 2 + np.random.normal(0, 3, n)
satisfaction = income / 50000 + np.random.normal(0, 0.5, n)
df = pd.DataFrame({
'age': age,
'income': income,
'education_years': education_years,
'satisfaction': satisfaction,
'years_employed': age - education_years - 6
})
# Pearson correlation (linear)
corr_matrix = df.corr(method='pearson')
print("Pearson Correlation Matrix:")
print(corr_matrix)
# Individual correlation with p-value
corr_coef, p_value = pearsonr(df['age'], df['income'])
print(f"\nPearson correlation (age vs income): r={corr_coef:.4f}, p-value={p_value:.4f}")
# Spearman correlation (rank-based)
spearman_matrix = df.corr(method='spearman')
print("\nSpearman Correlation Matrix:")
print(spearman_matrix)
spearman_coef, p_value = spearmanr(df['age'], df['income'])
print(f"Spearman correlation (age vs income): rho={spearman_coef:.4f}, p-value={p_value:.4f}")
# Kendall tau correlation
kendall_coef, p_value = kendalltau(df['age'], df['income'])
print(f"Kendall correlation (age vs income): tau={kendall_coef:.4f}, p-value={p_value:.4f}")
# Correlation heatmap
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# Pearson heatmap
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm', center=0,
square=True, ax=axes[0], vmin=-1, vmax=1)
axes[0].set_title('Pearson Correlation Heatmap')
# Spearman heatmap
sns.heatmap(spearman_matrix, annot=True, cmap='coolwarm', center=0,
square=True, ax=axes[1], vmin=-1, vmax=1)
axes[1].set_title('Spearman Correlation Heatmap')
plt.tight_layout()
plt.show()
# Correlation with significance testing
def correlation_with_pvalue(df):
rows, cols = [], []
for col1 in df.columns:
for col2 in df.columns:
if col1 < col2: # Avoid duplicates
r, p = pearsonr(df[col1], df[col2])
rows.append({
'Variable 1': col1,
'Variable 2': col2,
'Correlation': r,
'P-value': p,
'Significant': 'Yes' if p < 0.05 else 'No'
})
return pd.DataFrame(rows)
corr_table = correlation_with_pvalue(df)
print("\nCorrelation with P-values:")
print(corr_table)
# Scatter plots with regression lines
fig, axes = plt.subplots(2, 2, figsize=(12, 10))
pairs = [('age', 'income'), ('age', 'education_years'),
('income', 'satisfaction'), ('education_years', 'years_employed')]
for idx, (var1, var2) in enumerate(pairs):
ax = axes[idx // 2, idx % 2]
ax.scatter(df[var1], df[var2], alpha=0.5)
# Add regression line
z = np.polyfit(df[var1], df[var2], 1)
p = np.poly1d(z)
x_line = np.linspace(df[var1].min(), df[var1].max(), 100)
ax.plot(x_line, p(x_line), "r--", linewidth=2)
r, p_val = pearsonr(df[var1], df[var2])
ax.set_title(f'{var1} vs {var2}\nr={r:.4f}, p={p_val:.4f}')
ax.set_xlabel(var1)
ax.set_ylabel(var2)
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
# Multicollinearity detection (VIF)
from statsmodels.stats.outliers_influence import variance_inflation_factor
X = df[['age', 'education_years', 'years_employed']]
vif_data = pd.DataFrame()
vif_data['Variable'] = X.columns
vif_data['VIF'] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
print("\nVariance Inflation Factor (VIF):")
print(vif_data)
print("\nVIF > 10: High multicollinearity")
print("VIF > 5: Moderate multicollinearity")
# Partial correlation (controlling for confounding)
def partial_correlation(df, x, y, control_vars):
from scipy.stats import linregress
# Residuals of x after removing control variables
x_residuals = df[x] - np.poly1d(
np.polyfit(df[control_vars].values, df[x], deg=1)
)(df[control_vars].values)
# Residuals of y after removing control variables
y_residuals = df[y] - np.poly1d(
np.polyfit(df[control_vars].values, df[y], deg=1)
)(df[control_vars].values)
return pearsonr(x_residuals, y_residuals)[0]
partial_corr = partial_correlation(df, 'income', 'satisfaction', ['age'])
print(f"\nPartial correlation (income vs satisfaction, controlling for age): {partial_corr:.4f}")
# Distance correlation (non-linear relationships)
try:
from dcor import distance_correlation
dist_corr = distance_correlation(df['age'], df['income'])
print(f"Distance correlation (age vs income): {dist_corr:.4f}")
except ImportError:
print("dcor library not installed for distance correlation")
# Correlation stability over time
fig, ax = plt.subplots(figsize=(12, 5))
rolling_corr = df['age'].rolling(window=50).corr(df['income'])
ax.plot(rolling_corr.index, rolling_corr.values)
ax.set_title('Rolling Correlation (age vs income, window=50)')
ax.set_ylabel('Correlation Coefficient')
ax.grid(True, alpha=0.3)
plt.show()Interpretation Guidelines
- |r| = 0.0-0.3: Weak correlation
- |r| = 0.3-0.7: Moderate correlation
- |r| = 0.7-1.0: Strong correlation
- p < 0.05: Statistically significant
- High VIF (>10): Multicollinearity problem
Important Notes
- Correlation ≠ Causation
- Non-linear relationships missed by Pearson
- Outliers can distort correlations
- Sample size affects significance
- Temporal trends can create spurious correlations
Visualization Strategies
- Heatmaps for overview
- Scatter plots for relationships
- Pair plots for multivariate analysis
- Rolling correlations for time-varying relationships
Deliverables
- Correlation matrices (Pearson, Spearman)
- Correlation heatmaps with annotations
- Statistical significance table
- Scatter plots with regression lines
- Multicollinearity assessment (VIF)
- Partial correlation analysis
- Relationship interpretation report