Skip to main content
Skip to main content
Edit this page

Performance Optimization: Production-Tested Strategies

This guide is part of a collection of findings gained from community meetups. For more real world solutions and insights you can browse by specific problem. Having trouble with Materialized Views? Check out the Materialized Views community insights guide. If you're experiencing slow queries and want more examples, we also have a Query Optimization guide.

Order by Cardinality (Lowest to Highest)​

Time Granularity Matters​

Focus on Individual Queries, Not Averages​

"The right way is to ask yourself why this particular query was processed in five seconds... I don't care if median and other queries process quickly. I only care about my query"

Instead of looking at average performance, identify specific query patterns that cause problems:

Spot queries with different performance bottlenecks:

The key lesson from production teams: When a query is slow, don't just look at averages. Ask "Why was THIS specific query slow?" and examine the actual resource usage patterns.

Memory vs Row Scanning Trade-offs​

Sentry's key insight: "The cardinality of the grouping key that's going to drive memory in this particular situation" - High cardinality aggregations kill performance through memory exhaustion, not row scanning.

Pattern Recognition: When queries fail, determine if it's a memory problem (too many groups) or scanning problem (too many rows).

Compare granularity impact:

High-cardinality danger pattern:

Sentry's sampling solution for memory problems:

Sentry's Bit Mask Optimization: From Memory Explosion to 8 Bytes​

The Problem: When aggregating by high-cardinality columns (like URLs), each unique value creates a separate aggregation state in memory, leading to memory exhaustion.

Sentry's Solution: Instead of grouping by the actual URL strings, group by boolean expressions that collapse into bit masks.

PROBLEM: Memory explosion with unbounded string arrays

SOLUTION: Boolean expressions that collapse to single integers

Compare the memory efficiency:

The Memory Impact:

  • Before: Each user stores arrays of ALL unique repo names (potentially MBs per user)
  • After: Each user stores fixed number of counters (32 bytes for 4 conditions)
  • Result: Sentry achieved 100x memory reduction for certain query patterns

Production Insight: "Don't look for 10 or 20% improvements, look for orders of magnitude... you want to see like 10x, 100x less memory consumed."

Why This Works: Instead of storing every unique string in memory, you're storing the answer to questions about those strings as integers. The aggregation state becomes bounded and tiny, regardless of data diversity.

Video Sources​

Read Next: