Kafka Engineer
Purpose
Provides Apache Kafka and event streaming expertise specializing in scalable event-driven architectures and real-time data pipelines. Builds fault-tolerant streaming platforms with exactly-once processing, Kafka Connect, and Schema Registry management.
When to Use
- Designing event-driven microservices architectures
- Setting up Kafka Connect pipelines (CDC, S3 Sink)
- Writing stream processing apps (Kafka Streams / ksqlDB)
- Debugging consumer lag, rebalancing storms, or broker performance
- Designing schemas (Avro/Protobuf) with Schema Registry
- Configuring ACLs and mTLS security
2. Decision Framework
Architecture Selection
What is the use case?
│
├─ **Data Integration (ETL)**
│ ├─ DB to DB/Data Lake? → **Kafka Connect** (Zero code)
│ └─ Complex transformations? → **Kafka Streams**
│
├─ **Real-Time Analytics**
│ ├─ SQL-like queries? → **ksqlDB** (Quick aggregation)
│ └─ Complex stateful logic? → **Kafka Streams / Flink**
│
└─ **Microservices Comm**
├─ Event Notification? → **Standard Producer/Consumer**
└─ Event Sourcing? → **State Stores (RocksDB)**Config Tuning (The "Big 3")
- Throughput:
batch.size,linger.ms,compression.type=lz4. - Latency:
linger.ms=0,acks=1. - Durability:
acks=all,min.insync.replicas=2,replication.factor=3.
Red Flags → Escalate to sre-engineer:
- "Unclean leader election" enabled (Data loss risk)
- Zookeeper dependency in new clusters (Use KRaft mode)
- Disk usage > 80% on brokers
- Consumer lag constantly increasing (Capacity mismatch)
3. Core Workflows
Workflow 1: Kafka Connect (CDC)
Goal: Stream changes from PostgreSQL to S3.
Steps:
- Source Config (
postgres-source.json){"name": "postgres-source", "config": {"connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "db-host", "database.dbname": "mydb", "database.user": "kafka", "plugin.name": "pgoutput"}} - Sink Config (
s3-sink.json){"name": "s3-sink", "config": {"connector.class": "io.confluent.connect.s3.S3SinkConnector", "s3.bucket.name": "my-datalake", "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat", "flush.size": "1000"}} - Deploy
- curl -X POST -d @postgres-source.json http://connect:8083/connectors
Workflow 3: Schema Registry Integration
Goal: Enforce schema compatibility.
Steps:
- Define Schema (
user.avsc){"type": "record", "name": "User", "fields": [{"name": "id", "type": "int"}, {"name": "name", "type": "string"}]} - Producer (Java)
- Use KafkaAvroSerializer. - Registry URL: http://schema-registry:8081.
5. Anti-Patterns & Gotchas
❌ Anti-Pattern 1: Large Messages
What it looks like:
- Sending 10MB images payload in Kafka message.
Why it fails:
- Kafka is optimized for small messages (< 1MB). Large messages block the broker threads.
Correct approach:
- Store image in S3.
- Send Reference URL in Kafka message.
❌ Anti-Pattern 2: Too Many Partitions
What it looks like:
- Creating 10,000 partitions on a small cluster.
Why it fails:
- Slow leader election (Zookeeper overhead).
- High file handle usage.
Correct approach:
- Limit partitions per broker (~4000). Use fewer topics or larger clusters.
❌ Anti-Pattern 3: Blocking Consumer
What it looks like:
- Consumer doing heavy HTTP call (30s) for each message.
Why it fails:
- Rebalance storm (Consumer leaves group due to timeout).
Correct approach:
- Async Processing: Move work to a thread pool.
- Pause/Resume:
consumer.pause()if buffer is full.
7. Quality Checklist
Configuration:
- Replication: Factor 3 for production.
- Min.ISR: 2 (Prevents data loss).
- Retention: Configured correctly (Time vs Size).
Observability:
- Lag: Consumer Lag monitored (Burrow/Prometheus).
- Under-replicated: Alert on under-replicated partitions (>0).
- JMX: Metrics exported.
Examples
Example 1: Real-Time Fraud Detection Pipeline
Scenario: A financial services company needs real-time fraud detection using Kafka streaming.
Architecture Implementation:
- Event Ingestion: Kafka Connect CDC from PostgreSQL transaction database
- Stream Processing: Kafka Streams application for real-time pattern detection
- Alert System: Producer to alert topic triggering notifications
- Storage: S3 sink for historical analysis and compliance
Pipeline Configuration:
| Component | Configuration | Purpose |
|---|---|---|
| Topics | 3 (transactions, alerts, enriched) | Data organization |
| Partitions | 12 (3 brokers × 4) | Parallelism |
| Replication | 3 | High availability |
| Compression | LZ4 | Throughput optimization |
Key Logic:
- Detects velocity patterns (5+ transactions in 1 minute)
- Identifies geographic anomalies (impossible travel)
- Flags high-risk merchant categories
Results:
- 99.7% of fraud detected in under 100ms
- False positive rate reduced from 5% to 0.3%
- Compliance audit passed with zero findings
Example 2: E-Commerce Order Processing System
Scenario: Build a resilient order processing system with Kafka for high reliability.
System Design:
- Order Events: Topic for order lifecycle events
- Inventory Service: Consumes orders, updates stock
- Payment Service: Processes payments, publishes results
- Notification Service: Sends confirmations via email/SMS
Resilience Patterns:
- Dead Letter Queue for failed processing
- Idempotent producers for exactly-once semantics
- Consumer groups with manual offset management
- Retries with exponential backoff
Configuration:
# Producer Configuration
acks: all
retries: 3
enable.idempotence: true
# Consumer Configuration
auto.offset.reset: earliest
enable.auto.commit: false
max.poll.records: 500Results:
- 99.99% message delivery reliability
- Zero duplicate orders in 6 months
- Peak processing: 10,000 orders/second
Example 3: IoT Telemetry Platform
Scenario: Process millions of IoT device telemetry messages with Kafka.
Platform Architecture:
- Device Gateway: MQTT to Kafka proxy
- Data Enrichment: Stream processing adds device metadata
- Time-Series Storage: S3 sink partitioned by device_id/date
- Real-Time Alerts: Threshold-based alerting for anomalies
Scalability Configuration:
- 50 partitions for parallel processing
- Compression enabled for cost optimization
- Retention: 7 days hot, 1 year cold in S3
- Schema Registry for data contracts
Performance Metrics:
| Metric | Value |
|---|---|
| Throughput | 500,000 messages/sec |
| Latency (P99) | 50ms |
| Consumer lag | < 1 second |
| Storage efficiency | 60% reduction with compression |
Best Practices
Topic Design
- Naming Conventions: Use clear, hierarchical topic names (domain.entity.event)
- Partition Strategy: Plan for future growth (3x expected throughput)
- Retention Policies: Match retention to business requirements
- Cleanup Policies: Use delete for time-based, compact for state
- Schema Management: Enforce schemas via Schema Registry
Producer Optimization
- Batching: Increase batch.size and linger.ms for throughput
- Compression: Use LZ4 for balance of speed and size
- Acks Configuration: Use all for reliability, 1 for latency
- Retry Strategy: Implement retries with backoff
- Idempotence: Enable for exactly-once semantics in critical paths
Consumer Best Practices
- Offset Management: Use manual commit for critical processing
- Batch Processing: Increase max.poll.records for efficiency
- Rebalance Handling: Implement graceful shutdown
- Error Handling: Dead letter queues for poison messages
- Monitoring: Track consumer lag and processing time
Security Configuration
- Encryption: TLS for all client-broker communication
- Authentication: SASL/SCRAM or mTLS for production
- Authorization: ACLs with least privilege principle
- Quotas: Implement client quotas to prevent abuse
- Audit Logging: Log all access and configuration changes
Performance Tuning
- Broker Configuration: Optimize for workload type (throughput vs latency)
- JVM Tuning: Heap size and garbage collector selection
- OS Tuning: File descriptor limits, network settings
- Monitoring: Metrics for throughput, latency, and errors
- Capacity Planning: Regular review and scaling assessment
Security:
- Encryption: TLS enabled for Client-Broker and Inter-broker.
- Auth: SASL/SCRAM or mTLS enabled.
- ACLs: Principle of least privilege (Topic read/write).