SQL Query Optimizer
Overview
Rewrite SQL queries for maximum performance by eliminating anti-patterns, restructuring JOINs, leveraging window functions, and applying database-specific optimizations for PostgreSQL and MySQL. This skill takes a slow query and its execution plan as input and produces an optimized version with measurable improvement, along with any supporting index changes needed.
Prerequisites
- The slow SQL query text and its current execution time
EXPLAIN ANALYZEoutput (PostgreSQL) orEXPLAIN FORMAT=JSONoutput (MySQL) for the query- Table row counts and approximate data distribution for involved tables
psqlormysqlCLI for testing rewrites- Knowledge of the application's acceptable result ordering and NULL handling requirements
Instructions
- Examine the original query structure and identify common anti-patterns:
- SELECT * instead of specific columns (forces unnecessary I/O) - WHERE column IN (SELECT...) that can be rewritten as JOIN or EXISTS - DISTINCT used to mask duplicate rows from incorrect JOINs - Functions applied to indexed columns in WHERE clauses (WHERE UPPER(name) = 'FOO') - OR conditions that prevent index usage - NOT IN with nullable columns (produces wrong results and poor plans)
- Analyze the execution plan to identify the most expensive operation nodes. Focus optimization effort on the node consuming the most time or processing the most rows.
- Rewrite subqueries as JOINs where possible. Convert correlated subqueries to lateral joins (PostgreSQL) or derived tables. Replace
IN (SELECT...)withEXISTS (SELECT 1...)for existence checks since EXISTS short-circuits after the first match. - Optimize JOIN ordering for the query planner: place the most selective table (fewest matching rows after WHERE filters) as the driving table. Use
JOINhints only as a last resort since the optimizer usually picks the correct order with accurate statistics. - Replace multiple OR conditions on the same column with
IN (...): changeWHERE status = 'active' OR status = 'pending'toWHERE status IN ('active', 'pending'). For OR across different columns, consider UNION ALL of two simpler queries. - Apply window functions to replace self-joins or correlated subqueries. Use
ROW_NUMBER() OVER (PARTITION BY... ORDER BY...)for top-N-per-group queries instead ofGROUP BYwith subqueries. - Leverage CTEs (Common Table Expressions) for readability but be aware that PostgreSQL versions before 12 materialize all CTEs. For performance-critical queries on older PostgreSQL, inline the CTE as a subquery.
- Optimize aggregation queries by filtering before grouping (
WHEREis more efficient thanHAVINGfor non-aggregate conditions), using partial indexes for filtered aggregates, and considering materialized views for expensive recurring aggregations. - Test the rewritten query with
EXPLAIN ANALYZEand compare execution time, row estimates vs. actuals, and buffer usage against the original. The optimized version should show fewer rows processed, index scans replacing sequential scans, and lower total execution time. - Document each change made, the reason for the change, and the measured impact so the development team understands and can apply similar patterns to future queries.
Output
- Optimized SQL query with comments explaining each structural change
- Before/after execution plans showing performance improvement
- Index recommendations (CREATE INDEX statements) needed to support the optimized query
- Anti-pattern report listing issues found in the original query with explanations
- Performance metrics comparison (execution time, rows scanned, buffer hits)
Error Handling
| Error | Cause | Solution |
|---|---|---|
| Rewritten query returns different results | JOIN type change (INNER vs LEFT) or NULL handling difference | Verify result sets match with EXCEPT query; preserve original JOIN types; handle NULLs explicitly with COALESCE |
| Optimized query slower than original | Statistics outdated causing planner to choose wrong plan | Run ANALYZE on involved tables; compare estimated rows vs actual rows in EXPLAIN; consider SET enable_seqscan = off to test alternative plans |
| CTE materialization hurting performance | PostgreSQL <12 materializes CTEs preventing predicate pushdown | Inline the CTE as a subquery; upgrade PostgreSQL; add AS NOT MATERIALIZED hint in PostgreSQL 12+ |
| Window function query uses excessive memory | Large partition sizes with ORDER BY in window specification | Add LIMIT to outer query; use index matching the PARTITION BY and ORDER BY columns; increase work_mem for the session |
| UNION ALL produces duplicates | Overlapping conditions in constituent queries | Add mutually exclusive WHERE conditions to each branch; or use UNION (with dedup cost) if overlap is unavoidable |
Examples
Converting correlated subquery to JOIN: Original: SELECT * FROM orders WHERE customer_id IN (SELECT id FROM customers WHERE region = 'US') taking 8 seconds with sequential scan on orders. Rewrite: SELECT o.* FROM orders o JOIN customers c ON o.customer_id = c.id WHERE c.region = 'US' using index on orders.customer_id reduces to 120ms.
Top-N per group with window function: Original uses self-join to find the 3 most recent orders per customer (15 seconds). Rewrite: SELECT * FROM (SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY created_at DESC) AS rn FROM orders) sub WHERE rn <= 3 with index on (customer_id, created_at DESC) completes in 400ms.
Eliminating DISTINCT from incorrect JOIN: SELECT DISTINCT o.* FROM orders o JOIN line_items li ON o.id = li.order_id WHERE li.amount > 100 scans all line items. Rewrite: SELECT o.* FROM orders o WHERE EXISTS (SELECT 1 FROM line_items li WHERE li.order_id = o.id AND li.amount > 100) eliminates the deduplication step and halves execution time.
Resources
- PostgreSQL query planning: https://www.postgresql.org/docs/current/planner-optimizer.html
- MySQL query optimization: https://dev.mysql.com/doc/refman/8.0/en/optimization.html
- SQL anti-patterns reference: https://use-the-index-luke.com/sql/where-clause
- Window functions tutorial: https://www.postgresql.org/docs/current/tutorial-window.html
- Modern SQL features: https://modern-sql.com/