Apache kafka part 2
Download as pptx, pdf0 likes94 views
The document discusses various producer and consumer configurations in Apache Kafka. It describes producer acknowledgment settings like acks=0, acks=1, and acks=all that control how many replicas must acknowledge a record before the request is considered complete. It also discusses synchronous, asynchronous, and "fire-and-forget" producer sending modes. Consumer configurations covered include fetch size, partition assignment strategies, committing offsets asynchronously or synchronously, and rebalance listeners. The document also briefly mentions log compaction in Kafka.
More Related Content
What's hot (16)
Similar to Apache kafka part 2 (20)
Recently uploaded (20)
Apache kafka part 2
- 1. Producer | Processing data
- 2. Acknowledgments acks The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent. The following settings are allowed: ● acks=0 If set to zero then the producer will not wait for any acknowledgment from the server at all. ● acks=1 This will mean the leader will write the record to its local log but will respond without awaiting full acknowledgement from all followers. ● acks=all This means the leader will wait for the full set of in-sync replicas to acknowledge the record.
- 3. Producer | Sending Messages Fire-and-forget We send a message to the server and don’t really care if it arrives successfully or not. Synchronous send We send a message, the send() method returns a Future object, and we use send().get() to wait on the future Asynchronous send We call the send(ProducerRecord, ProducerCallback) method with a callback function. , which gets triggered when it receives a response from the Kafka broker.
- 4. Producer | Record Size vs Throughput Handle large size message ?
- 5. Consumer | Config fetch.min.bytes This property allows a consumer to specify the minimum amount of data that it wants to receive from the broker when fetching records. max.partition.fetch.bytes This property controls the maximum number of bytes the server will return per partition. enable.auto.commit This parameter controls whether the consumer will commit offsets automatically, and defaults to true. partition.assignment.strategy Range : T1 T2 T3 || C1 C2 → C1 (T1, T2) & C2 (T3) Round Robin : T1 T2 T3 || C1 C2 → C1 (T1, T3) & C2 (T2) max.poll.records The maximum number of records returned in a single call to poll().
- 6. Consumer | Sync or Async Commit consumer.commitSync() Retry the commit consumer.commitAsync() Not Retry the commit → commit the latest offset returned by poll(), so make sure you call commitSync() after you are done processing all the records in the collection.
- 7. Async Commit | CallBack | Retrying Async consumer.commitAsync(new OffsetCommitCallback() { public void onComplete(Map<TopicPartition, OffsetAndMetadata> offsets, Exception exception) { if (e != null) log.error("Commit failed for offsets {}", offsets, e); } });
- 8. Commit while processing the records private Map<TopicPartition, OffsetAndMetadata> currentOffsets = new HashMap<>(); ConsumerRecords<String, String> records = consumer.poll(100); for (ConsumerRecord<String, String> record : records) { //processing if (count % 10 == 0) { consumer.commitAsync(currentOffsets, null); } count++; } Commit Specific Offset
- 9. Rebalance Listeners ConsumerRebalanceListener Allows you to run your own code when partitions are added or removed from the consumer. You do this by passing a ConsumerRebalanceListener when calling the subscribe(). ● onPartitionsRevoked ● onPartitionsAssigned
- 10. Compaction Log.cleaner.enabled Clean : Messages that have been compacted before. Dirty : Messages that were written after the last compaction.
- 11. Compaction
- 13. Thank You Never Stop Learning