I am running 3 red panda's brokers with replication on 3 k8s nodes, this works perfectly fine. now I am connecting with java client and try to compare performance to learn more about configuration. broker has 12 partitions and I have there 80GB of data, each message has 4kb or 200bytes.
1 scenario:1 group consumer can fetch around 400mb per second, which is around 86,419 messages, this grows to around 823,043 messages when payload is smaller (200bytes). consumer fetches all 12 partitions in multiple pulls.
2 scenario:4 consumers run on 4 different threads, each consumer assigns its own, single, unique partition (yes I fetch only 4 partition in this scenario). I expected that I should get at least same throughput for each consumer and in overall improved performance - what I observe is that each consumer can fetch around 100-200MB per second, which sums back to 400-600MB
3 scenario:single pod that starts multiple group consumers. Multiple pods, each starts single group consumer. Again I don't get improved throughput.
consumer pods are run on different node group than red panda. I have 16cpus and 32 Gbps Network Bandwidth on each node, that should be enough to push data between nodes.
my consumer configuration is
props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, 100);props.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, 4 * 1024 * 1_00_000);props.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, 4 * 1024 * 1_00_000);props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 1_000_000);props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 10_000);I haven't seen tcp configuration on consumer side, to experiment, I am still looking what should I set on red panda's side to improve throughput.
each RedPanda s broker is run on k8s in GCP c2-standard-16 which is described as
vCPU 16Memory 64 GBCPU Manufacturer IntelCPU Platform Intel Cascade LakeCPU Base Frequency 3.1 GHzCPU Turbo Frequency 3.8 GHzCPU Max. Turbo Frequency 3.9 GHzNetwork Bandwidth 32 Gbps