monitoring-dashboard（监控仪表板）

Monitoring Dashboard

Purpose: Provide complete monitoring dashboard templates and setup scripts for ML training with TensorBoard and Weights & Biases (WandB).

Activation Triggers:

• Setting up training monitoring dashboards
• Tracking experiments and metrics in real-time
• Comparing multiple training runs
• Visualizing hyperparameters and results
• Integrating monitoring into existing training pipelines
• Logging custom metrics, images, and model artifacts

Key Resources:

• scripts/setup-tensorboard.sh - Install and configure TensorBoard
• scripts/setup-wandb.sh - Install and configure Weights & Biases
• scripts/launch-monitoring.sh - Launch monitoring dashboards
• templates/tensorboard-config.yaml - TensorBoard configuration template
• templates/wandb-config.py - WandB integration template
• templates/logging-config.json - Unified logging configuration
• examples/tensorboard-integration.md - Complete TensorBoard integration guide
• examples/wandb-integration.md - Complete WandB integration guide

Quick Start

1. Choose Monitoring Solution

TensorBoard (Local/Open Source):

• Free, runs locally
• Best for: Single-user development, offline work
• Features: Metrics, histograms, graphs, images, embeddings
• Storage: Local filesystem

Weights & Biases (Cloud/Collaboration):

• Free tier available, cloud-hosted
• Best for: Team collaboration, experiment comparison, production
• Features: All TensorBoard features + collaboration, alerts, reports
• Storage: Cloud with unlimited history

Both (Recommended for Production):

• Use TensorBoard for local development
• Use WandB for team collaboration and production tracking

2. Setup TensorBoard

# Install and configure TensorBoard
./scripts/setup-tensorboard.sh

# Launch TensorBoard
./scripts/launch-monitoring.sh tensorboard --logdir ./runs

Access: Open browser to http://localhost:6006

3. Setup Weights & Biases

# Install and configure WandB
./scripts/setup-wandb.sh

# Login with API key
wandb login

# Launch monitoring
./scripts/launch-monitoring.sh wandb

Access: Dashboard at https://wandb.ai/your-username/your-project

TensorBoard Integration

Basic Setup

Template: templates/tensorboard-config.yaml

from torch.utils.tensorboard import SummaryWriter
import datetime

# Create TensorBoard writer
log_dir = f"runs/experiment_{datetime.datetime.now().strftime('%Y%m%d-%H%M%S')}"
writer = SummaryWriter(log_dir=log_dir)

# Log scalar metrics
writer.add_scalar('Loss/train', train_loss, epoch)
writer.add_scalar('Loss/validation', val_loss, epoch)
writer.add_scalar('Accuracy/train', train_acc, epoch)
writer.add_scalar('Accuracy/validation', val_acc, epoch)

# Log learning rate
writer.add_scalar('Learning_Rate', optimizer.param_groups[0]['lr'], epoch)

# Close writer when done
writer.close()

Advanced Logging

Histograms (Weight Distributions):

# Log model weights
for name, param in model.named_parameters():
    writer.add_histogram(f'weights/{name}', param, epoch)
    writer.add_histogram(f'gradients/{name}', param.grad, epoch)

Images:

# Log sample predictions
writer.add_image('predictions', image_grid, epoch)
writer.add_images('batch_samples', image_batch, epoch)

Text:

# Log hyperparameters as text
config_text = '\n'.join([f'{k}: {v}' for k, v in config.items()])
writer.add_text('hyperparameters', config_text, 0)

Model Graph:

# Log model architecture
writer.add_graph(model, input_tensor)

Embeddings (t-SNE, PCA):

# Visualize embeddings
writer.add_embedding(embeddings, metadata=labels, label_img=images)

Launch TensorBoard

# Basic launch
tensorboard --logdir runs

# Specify port
tensorboard --logdir runs --port 6007

# Load faster (sample data)
tensorboard --logdir runs --samples_per_plugin scalars=1000

# Enable reload
tensorboard --logdir runs --reload_interval 5

Weights & Biases Integration

Basic Setup

Template: templates/wandb-config.py

import wandb

# Initialize WandB run
wandb.init(
    project="my-ml-project",
    name=f"experiment-{datetime.now().strftime('%Y%m%d-%H%M%S')}",
    config={
        "learning_rate": 0.001,
        "epochs": 100,
        "batch_size": 32,
        "model": "resnet50",
        "dataset": "imagenet"
    }
)

# Log metrics
wandb.log({
    "train_loss": train_loss,
    "val_loss": val_loss,
    "train_acc": train_acc,
    "val_acc": val_acc,
    "epoch": epoch
})

# Finish run
wandb.finish()

Advanced Features

Log Media:

# Log images
wandb.log({"predictions": [wandb.Image(img, caption=f"Pred: {pred}")]})

# Log tables
table = wandb.Table(columns=["epoch", "loss", "accuracy"], data=data)
wandb.log({"results_table": table})

# Log audio
wandb.log({"audio": wandb.Audio(audio_array, sample_rate=16000)})

# Log videos
wandb.log({"video": wandb.Video(video_path, fps=30)})

Track Model Artifacts:

# Save model checkpoint
artifact = wandb.Artifact('model-checkpoint', type='model')
artifact.add_file('model.pth')
wandb.log_artifact(artifact)

# Load model from artifact
artifact = wandb.use_artifact('model-checkpoint:latest')
model_path = artifact.download()

Hyperparameter Sweeps:

# Define sweep configuration
sweep_config = {
    'method': 'bayes',
    'metric': {'name': 'val_loss', 'goal': 'minimize'},
    'parameters': {
        'learning_rate': {'min': 0.0001, 'max': 0.1},
        'batch_size': {'values': [16, 32, 64]},
        'optimizer': {'values': ['adam', 'sgd', 'adamw']}
    }
}

# Initialize sweep
sweep_id = wandb.sweep(sweep_config, project="my-project")

# Run sweep agent
wandb.agent(sweep_id, function=train_model, count=10)

Custom Charts:

# Create custom plot
data = [[x, y] for (x, y) in zip(x_values, y_values)]
table = wandb.Table(data=data, columns=["x", "y"])
wandb.log({
    "custom_plot": wandb.plot.line(table, "x", "y", title="Custom Plot")
})

Alerts:

# Alert on metric threshold
if val_loss < 0.1:
    wandb.alert(
        title="Low Validation Loss",
        text=f"Validation loss dropped to {val_loss:.4f}",
        level=wandb.AlertLevel.INFO
    )

Unified Logging Configuration

Template: templates/logging-config.json

Use this configuration to log to both TensorBoard and WandB simultaneously:

import wandb
from torch.utils.tensorboard import SummaryWriter

class UnifiedLogger:
    def __init__(self, project_name, experiment_name, config):
        # TensorBoard
        self.tb_writer = SummaryWriter(
            log_dir=f"runs/{experiment_name}"
        )

        # WandB
        wandb.init(
            project=project_name,
            name=experiment_name,
            config=config
        )

    def log_metrics(self, metrics_dict, step):
        """Log to both TensorBoard and WandB"""
        # TensorBoard
        for key, value in metrics_dict.items():
            self.tb_writer.add_scalar(key, value, step)

        # WandB
        wandb.log(metrics_dict, step=step)

    def log_images(self, images_dict, step):
        """Log images to both platforms"""
        for key, image in images_dict.items():
            # TensorBoard
            self.tb_writer.add_image(key, image, step)

            # WandB
            wandb.log({key: wandb.Image(image)}, step=step)

    def log_model(self, model, input_sample):
        """Log model architecture"""
        # TensorBoard graph
        self.tb_writer.add_graph(model, input_sample)

        # WandB watches gradients
        wandb.watch(model, log="all", log_freq=100)

    def close(self):
        """Close both loggers"""
        self.tb_writer.close()
        wandb.finish()

# Usage
logger = UnifiedLogger(
    project_name="my-project",
    experiment_name="exp-001",
    config={"lr": 0.001, "batch_size": 32}
)

logger.log_metrics({
    "train_loss": 0.5,
    "val_loss": 0.6
}, step=epoch)

logger.close()

Common Monitoring Patterns

1. Training Loop Integration

for epoch in range(num_epochs):
    # Training phase
    model.train()
    train_loss = 0
    for batch_idx, (data, target) in enumerate(train_loader):
        loss = train_step(model, data, target, optimizer)
        train_loss += loss.item()

        # Log batch-level metrics
        global_step = epoch * len(train_loader) + batch_idx
        logger.log_metrics({
            "batch_loss": loss.item(),
            "learning_rate": optimizer.param_groups[0]['lr']
        }, step=global_step)

    # Validation phase
    model.eval()
    val_loss, val_acc = validate(model, val_loader)

    # Log epoch-level metrics
    logger.log_metrics({
        "epoch": epoch,
        "train_loss": train_loss / len(train_loader),
        "val_loss": val_loss,
        "val_acc": val_acc
    }, step=epoch)

    # Log model weights distribution
    for name, param in model.named_parameters():
        logger.tb_writer.add_histogram(f'weights/{name}', param, epoch)

2. Experiment Comparison

TensorBoard:

# Compare multiple runs
tensorboard --logdir_spec \
  exp1:runs/experiment_1,\
  exp2:runs/experiment_2,\
  exp3:runs/experiment_3

WandB:

# Automatically compares all runs in project dashboard
# Filter and group runs by tags, config values, or custom fields

3. Real-Time Monitoring

TensorBoard:

# Auto-reload new data
tensorboard --logdir runs --reload_interval 5

WandB:

# Real-time by default
# Enable email/slack alerts for key metrics
wandb.alert(
    title="Training Alert",
    text=f"Accuracy reached {acc:.2%}",
    level=wandb.AlertLevel.INFO
)

Best Practices

1. Metric Naming Conventions

Organize by category:

# Good: Hierarchical naming
"Loss/train"
"Loss/validation"
"Accuracy/train"
"Accuracy/validation"
"Metrics/precision"
"Metrics/recall"

# Bad: Flat naming
"train_loss"
"validation_loss"
"train_accuracy"

2. Logging Frequency

Guidelines:

• Scalars: Every batch or every N batches
• Histograms: Every epoch
• Images: Every epoch or every N epochs
• Model graph: Once at start
• Embeddings: Once per major checkpoint

# Log batch metrics every 10 batches
if batch_idx % 10 == 0:
    logger.log_metrics({"batch_loss": loss}, step)

# Log epoch metrics
if batch_idx == len(train_loader) - 1:
    logger.log_metrics({"epoch_loss": epoch_loss}, epoch)

# Log images every 5 epochs
if epoch % 5 == 0:
    logger.log_images({"samples": sample_images}, epoch)

3. Disk Space Management

TensorBoard:

# Limit log retention
find runs/ -type d -mtime +30 -exec rm -rf {} +

# Compress old logs
tar -czf archive_$(date +%Y%m%d).tar.gz runs/old_experiments/
rm -rf runs/old_experiments/

WandB:

# Cloud storage handles retention
# Configure retention in project settings
# Download important runs for local backup
wandb.restore('model.pth', run_path="user/project/run_id")

4. Security & Privacy

TensorBoard:

# Restrict access to localhost only
tensorboard --logdir runs --host 127.0.0.1

# Or use SSH tunnel for remote access
ssh -L 6006:localhost:6006 user@remote-server

WandB:

# Use private projects
wandb.init(project="my-project", entity="private-team")

# Disable cloud sync for sensitive data
wandb.init(mode="offline")  # Logs locally only

Troubleshooting

TensorBoard Issues

Problem: Dashboard not updating

# Force reload
tensorboard --logdir runs --reload_interval 1

# Clear cache
rm -rf /tmp/.tensorboard-info/

Problem: Port already in use

# Use different port
tensorboard --logdir runs --port 6007

# Or kill existing process
pkill -f tensorboard

WandB Issues

Problem: Login fails

# Re-login with API key
wandb login --relogin

# Or set via environment
export WANDB_API_KEY=your_api_key

Problem: Slow logging

# Reduce logging frequency
wandb.init(settings=wandb.Settings(
    _disable_stats=True,  # Disable system metrics
    _disable_meta=True    # Disable metadata
))

Scripts Usage

Setup TensorBoard

./scripts/setup-tensorboard.sh

# Verifies:
# - Python environment
# - TensorBoard installation
# - Creates default log directory structure

Setup WandB

./scripts/setup-wandb.sh

# Verifies:
# - WandB installation
# - API key configuration
# - Creates wandb config file

Launch Monitoring

# TensorBoard
./scripts/launch-monitoring.sh tensorboard --logdir ./runs --port 6006

# WandB (opens browser to dashboard)
./scripts/launch-monitoring.sh wandb --project my-project

# Both
./scripts/launch-monitoring.sh both --logdir ./runs --project my-project

Resources

Scripts:

• setup-tensorboard.sh - Install and configure TensorBoard
• setup-wandb.sh - Install and configure WandB
• launch-monitoring.sh - Launch monitoring dashboards

Templates:

• tensorboard-config.yaml - TensorBoard setup configuration
• wandb-config.py - WandB integration template
• logging-config.json - Unified logging configuration

Examples:

• tensorboard-integration.md - Complete TensorBoard integration
• wandb-integration.md - Complete WandB integration with sweeps

Supported Frameworks: PyTorch, TensorFlow, JAX, Hugging Face Transformers Python Version: 3.8+ Best Practice: Use both TensorBoard (local dev) and WandB (team collaboration)