Storage Sync
Synchronize memories between Turso (durable) and redb (cache) storage layers.
When to Sync
- On startup - After system initialization
- Periodic - Scheduled background sync
- Cache staleness - When redb appears outdated
- Recovery - After storage failures
- Manual trigger - When explicitly requested
Sync Process
- Check connection health:
turso_client.ping().await?; redb_env.check_integrity()?; - Fetch latest from Turso:
let episodes = turso_client.query("SELECT * FROM episodes ORDER BY timestamp DESC LIMIT?").bind(max_episodes_cache).await?; - Update redb cache:
tokio::task::spawn_blocking(move || {let write_txn = redb.begin_write()?; let mut table = write_txn.open_table(EPISODES_TABLE)?; for episode in episodes {table.insert(episode.id.as_bytes(), episode.to_bytes())?;} write_txn.commit()?;}); - Sync patterns and embeddings if enabled
Configuration
pub struct SyncConfig {
pub max_episodes_cache: usize, // Default: 1000
pub batch_size: usize, // Default: 100
pub sync_patterns: bool, // Default: true
pub sync_embeddings: bool, // Default: true
}Error Handling
| Error | Handling |
|---|---|
| Turso unavailable | Skip sync, log warning, retry later |
| redb corruption | Attempt repair, rebuild from Turso |
| Partial sync | Track progress, resume from last point |
Performance Tips
- Batch small operations
- Incremental sync (only changes)
- Parallel fetch with Tokio
- Write-ahead preparation
Validation
After sync, verify:
- Episode count matches
- Latest episodes present
- Pattern counts consistent
- No orphaned embeddings
Troubleshooting
| Issue | Solution |
|---|---|
| Slow syncs | Reduce cache size, increase batch |
| redb lock errors | Use dedicated write task |
| Memory pressure | Stream large results, smaller batches |