Performance Optimization - Critical Best Practices
NEVER Fetch Full Datasets
CRITICAL RULE: Never fetch entire datasets, especially large ones (100k+ rows). Always use targeted queries with only the columns needed for each visualization.
Performance Impact
// ❌ TERRIBLE - Fetches 300k rows × 30 columns = 9M data points
const allData = await new Query()
.select(['col1', 'col2', 'col3', ...]) // All columns
.fetch('dataset');
// ✅ GOOD - Fetches 300k rows × 2 columns = 600k data points (93% reduction)
const totals = await new Query()
.select(['Transactions', 'Total Amount (USD)'])
.fetch('dataset');
// Then aggregate client-side
// ✅ BEST - Server-side aggregation, returns only aggregated results
const totals = await new Query()
.groupBy('Merchant Category', { 'Total Amount (USD)': 'sum' })
.fetch('dataset');
// Returns only ~10-20 rows (one per category)Query Optimization Strategy
1. Use Server-Side Aggregation When Possible
Priority order:
- Best:
.groupBy()with aggregations (server-side) - Good:
.select()with only needed columns, then aggregate client-side - Acceptable:
.select()with multiple columns if aggregation isn't possible - Never: Fetch all columns or entire dataset
// ✅ BEST - Server aggregates, returns minimal data
const salesByRegion = await new Query()
.groupBy('region', { 'Sales_Amount': 'sum', 'Order_Count': 'count' })
.fetch('sales');
// ✅ GOOD - Only fetch columns needed, aggregate client-side
const salesData = await new Query()
.select(['region', 'Sales_Amount'])
.fetch('sales');
const totalSales = salesData.reduce((sum, row) => sum + row.Sales_Amount, 0);
// ❌ BAD - Fetches all columns
const allData = await new Query().fetch('sales');2. One Query Per Visualization
Each visualization should have its own optimized query:
// ✅ CORRECT - Separate optimized queries
const quickStats = await fetchQuickStats(); // Only Account Key, Account Status
const riskMetrics = await fetchRiskMetrics(); // Only Transactions, KYC Status, etc.
const categories = await fetchCategories(); // Grouped by Merchant Category
// ❌ WRONG - One query for everything
const allData = await fetchAllData();
const quickStats = calculateFromAll(allData);
const riskMetrics = calculateFromAll(allData);
const categories = calculateFromAll(allData);3. Column Selection Strategy
Always specify columns explicitly:
- Only select columns needed for the specific calculation
- Don't use
.select()without arguments (fetches all columns) - For aggregations, only select the grouping column + aggregated columns
// ✅ CORRECT - Only 2 columns
const data = await new Query()
.select(['Account Key', 'Account Status'])
.fetch('dataset');
// ❌ WRONG - Fetches all columns
const data = await new Query().fetch('dataset');Common Patterns
Pattern 1: Totals Without Grouping
When you need totals but no grouping column:
// Option A: Select only needed columns, sum client-side
const totalsData = await new Query()
.select(['Transactions', 'Total Amount (USD)'])
.fetch('dataset');
const totalTransactions = totalsData.reduce((sum, row) => sum + (row.Transactions || 0), 0);
const totalVolume = totalsData.reduce((sum, row) => sum + (row['Total Amount (USD)'] || 0), 0);
// Option B: Use a constant grouping column (if dataset has one)
// Not always possible, but more efficient if availablePattern 2: Counts with Filters
// ✅ CORRECT - Only fetch the column needed for counting
const kycData = await new Query()
.select(['KYC Status'])
.fetch('dataset');
const approvedCount = kycData.filter(row => row['KYC Status'] === 'APPROVED').length;
// Better if possible: Use server-side count aggregation
// (See Query API limitations below)Pattern 3: Unique Values
// ✅ CORRECT - Only fetch the column needed
const statesData = await new Query()
.select(['State'])
.fetch('dataset');
const uniqueStates = new Set(statesData.map(row => row.State)).size;Query API Limitations
.aggregate() Doesn't Work
CRITICAL: The .aggregate() method shown in some documentation does not work in practice. It causes error: DA0057: An alias list was provided but it could not be parsed.
// ❌ DOES NOT WORK - Causes DA0057 error
const totals = await new Query()
.aggregate({ 'Transactions': 'sum', 'Total Amount (USD)': 'sum' })
.fetch('dataset');
// ✅ WORKAROUND - Use .groupBy() with a grouping column
const totals = await new Query()
.groupBy('some_column', { 'Transactions': 'sum', 'Total Amount (USD)': 'sum' })
.fetch('dataset');
// ✅ ALTERNATIVE - Select columns and aggregate client-side
const totals = await new Query()
.select(['Transactions', 'Total Amount (USD)'])
.fetch('dataset');
// Then sum client-side.groupBy() Requires a Grouping Column
You cannot use .groupBy() with only aggregations - you must provide a grouping column:
// ❌ DOES NOT WORK - No grouping column
.groupBy({ 'Transactions': 'sum' })
// ✅ WORKS - Has grouping column
.groupBy('region', { 'Transactions': 'sum' })
// ✅ WORKS - Multiple groupBy calls
.groupBy('region')
.groupBy({ 'Transactions': 'sum' })Performance Monitoring
Always check:
- Network tab - How much data is being transferred?
- Console - Any warnings about large queries?
- Response times - Are queries taking too long?
If a query is slow or returns too much data:
- Reduce columns selected
- Add filters to reduce rows
- Use server-side aggregation instead of client-side
- Consider pagination for large result sets