Quantcast
Viewing latest article 15
Browse Latest Browse All 87

Kafka consumer seek() losses some data in consmer.pool() with single consumer assign to multiple partitions

I am running one consumer and assigned 4 partitions(0,1,2,3) to it.consumer.assign(0,1,2,3)

I have send A1,B2,C3,D4 data on partition 0,1,2,3 on the topic consumer is running.Now, i pool the data from consumer then process it. if data is process successfully then adjust the offset consumer.commitSync(partitionAndOffset) if data processing is failed then i do consumer.seek(new TopicPartition(record.topic(), record.partition()),record.offset()).

In my case, consumer.poll() is polling data from each partitions (i.e A1,B2,C3,D4) but what is happening some data is lost. let's say if in first poll it's get A1 and B2 then for A1 processing failed then i call seek, then for B2 it also failed then again seek().

In next pool it will get B2 and C3 and so for then at the end it's only process D4 data rest of them (A1,B2,C3 from other partitions) are lost.

My question is :

  1. Is this expected ?
  2. Is there other Kafka API to help me out here ?

below is sample code:

ConsumerRecords<String, String> consumerRecords = consumer.poll(100);if (consumerRecords.count() > 0) {    try {         for (ConsumerRecord<String, String> record : consumerRecords){            if (!Processin Application(record.topic(),record.value(),record.partition())) {                // if processing falied in appliction then seek                consumer.seek(new TopicPartition(record.topic(), record.partition()),record.offset());                break;            }            else            {                //adjust offset manually on successful processing by application                try {                    Map<TopicPartition, OffsetAndMetadata> partitionAndOffset = new HashMap<>();                    partitionAndOffset.put(new TopicPartition(record.topic(), record.partition()), new OffsetAndMetadata(record.offset()));                    consumer.commitSync(partitionAndOffset);                } catch (Exception e) {                }            }        }    } catch (Exception e1) {    }}

This issue is when i run 1 consumer and assigned multiple partitions to it.if i run multiple consumer and assign one partition to each then it's working fine.


Viewing latest article 15
Browse Latest Browse All 87

Trending Articles