kafka
Purpose
Apache Kafka is a distributed event streaming platform used for building real-time data pipelines and streaming apps, enabling high-throughput, fault-tolerant messaging.
When to Use
Use Kafka for scenarios requiring real-time data ingestion and processing, such as log aggregation, event-driven architectures, or microservices communication; avoid it for simple queueing needs where lighter tools like RabbitMQ suffice.
Key Capabilities
- Supports distributed streaming with topics, partitions, and replicas for scalability and durability.
- Offers exactly-once semantics via transactional APIs to prevent data loss or duplication.
- Handles high volumes with configurable retention policies, e.g., retaining messages for 7 days using
log.retention.hours=168in broker config. - Provides consumer groups for load balancing, where multiple consumers share a group ID to partition topic consumption.
- Integrates streaming processing via Kafka Streams API for stateful transformations, like aggregating events with
KTableobjects.
Usage Patterns
To produce messages, create a topic first, then use a producer client; for consumption, subscribe to a topic and process messages in a loop. Always handle offsets manually or via auto-commit to avoid reprocessing. For batch processing, use Kafka Connect to ingest data from sources like databases. Pattern: Use idempotent producers for at-least-once delivery by setting enable.idempotence=true in producer configs.
Common Commands/API
Use Kafka CLI for quick operations:
- Create a topic:
kafka-topics.sh --create --topic mytopic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 2 - Produce messages:
kafka-console-producer.sh --topic mytopic --bootstrap-server localhost:9092(type messages and press Ctrl+D to send) - Consume messages:
kafka-console-consumer.sh --topic mytopic --from-beginning --bootstrap-server localhost:9092 --group mygroupFor API usage in Java: - Producer example:
Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); KafkaProducer<String, String> producer = new KafkaProducer<>(props); producer.send(new ProducerRecord<>("mytopic", "key", "value")); - Consumer example:
Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("group.id", "mygroup"); props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props); consumer.subscribe(Collections.singletonList("mytopic")); ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
Authenticate with SASL using env var: Set $KAFKA_SASL_JAAS_CONFIG to "org.apache.kafka.common.security.plain.PlainLoginModule required username='$SERVICE_API_KEY';".
Integration Notes
Integrate Kafka with other systems via Kafka Connect for sources/sinks, e.g., JDBC connector for databases. For authentication, use SSL or SASL with keys from env vars like $KAFKA_CLIENT_API_KEY. When linking to Spark, configure Spark Streaming with spark.kafka.bootstrap.servers and include dependencies like spark-sql-kafka-0-10_2.12. For microservices, use Kafka as a backbone with producers sending events to topics and consumers reacting via webhooks. Always specify exact versions, e.g., Kafka 3.4.0 with Confluent Schema Registry at endpoint http://localhost:8081/subjects.
Error Handling
Handle common errors like connection failures by checking broker availability and retrying with exponential backoff; for example, in code, wrap producer.send() in a try-catch and retry up to 3 times. If offsets are out of range, use auto.offset.reset=earliest in consumer configs to start from the beginning. For authentication errors (e.g., 401 Unauthorized), verify env vars like $SERVICE_API_KEY and ensure SASL mechanisms match. Log errors with details, e.g., in Java: catch (KafkaException e) {log.error("Kafka error: {}", e.getMessage());}. Address broker crashes by monitoring replicas and using min.insync.replicas=2 to enforce acknowledgment.
Graph Relationships
- Belongs to cluster: data-engineering
- Related tags: event-streaming, data-pipelines, kafka
- Potential links: integrates with skills in data-engineering cluster, such as spark or hadoop for data processing pipelines.