Kafka consumers reading too slow
We have a Kafka implementation of around 40 topics and each topic has 40 partitions and 3 replications. We have 7 brokers and 40 consumers in the Kafka setup. All nodes (for brokers and consumers) are of reasonable configurations, hosted and AWS and we hardly see any spikes in any of the machines. But somehow, we are finding that the consumer lag is very high, despite adding 40 consumers to cater the read for the above setup. This is despite the fact that we have only around 215 messages (each message is of around 2KB in size) of ingestion per second, to the above topics. Have tried everything possible, but we are not able to solve the lag issue.
We also see the consumers are most of the times sitting idle and consume the messages once in a while. Are 40 consumers enough to handle the above scenario (40 topics with 40 partitions and each topic has around 215 messages (430 K) of ingestions per second? Please help.
apache-kafka kafka-consumer-api
asked Nov 10 at 5:54
Dheeraj Kalathil
32
add a comment |
We have a Kafka implementation of around 40 topics and each topic has 40 partitions and 3 replications. We have 7 brokers and 40 consumers in the Kafka setup. All nodes (for brokers and consumers) are of reasonable configurations, hosted and AWS and we hardly see any spikes in any of the machines. But somehow, we are finding that the consumer lag is very high, despite adding 40 consumers to cater the read for the above setup. This is despite the fact that we have only around 215 messages (each message is of around 2KB in size) of ingestion per second, to the above topics. Have tried everything possible, but we are not able to solve the lag issue.
We also see the consumers are most of the times sitting idle and consume the messages once in a while. Are 40 consumers enough to handle the above scenario (40 topics with 40 partitions and each topic has around 215 messages (430 K) of ingestions per second? Please help.
apache-kafka kafka-consumer-api
asked Nov 10 at 5:54
Dheeraj Kalathil
32
so you mean each consumer is consuming from 1 partitions of 40 distinct topics, (total 1 consumer = 40 partitions which is huge for 1 consumer)
– Deadpool
Nov 10 at 6:05
try using a consumer group for consumers and keep the number of consumers equal to the number of partitions on that topic. I would separate out the consumers for set of topics into different groups. ex. 40 consumer dedicatedly listening to 10 topics, and 4 such group(totally running 160 consumers). this would need some trial and error to figure out the best combination that works for your SLAs.
– segFaulter
Nov 10 at 6:11
@Deadpool, thank you for the comment. I think, you are spot on the scenario. Yes, each consumer is reading from 1 partition of 40 distinct topics, which I agree can be huge. In that case too, I thought the consumers should be reading continuously, but I can see that the consumers are sitting idle for a long time and consumes once in a while, which is very confusing.
– Dheeraj Kalathil
Nov 11 at 4:52
@segFaulter, thank you for the comment. Each consumer instance uses around 1 GB RAM to run, so adding 160 consumers will incur a lot of cost in terms of setting up the environment.
– Dheeraj Kalathil
Nov 11 at 4:57
consumer can poll from one partition at a time, (so it should poll from 1 to 40 assume each poll will take 1 sec, so 40 polls to complete all partitions assigned to it will take 40 seconds), consumer can sit ideal when there is no enough data to poll in partitions, there are so couple of factors on consumers causing this
– Deadpool
Nov 11 at 4:58
add a comment |
We have a Kafka implementation of around 40 topics and each topic has 40 partitions and 3 replications. We have 7 brokers and 40 consumers in the Kafka setup. All nodes (for brokers and consumers) are of reasonable configurations, hosted and AWS and we hardly see any spikes in any of the machines. But somehow, we are finding that the consumer lag is very high, despite adding 40 consumers to cater the read for the above setup. This is despite the fact that we have only around 215 messages (each message is of around 2KB in size) of ingestion per second, to the above topics. Have tried everything possible, but we are not able to solve the lag issue.
We also see the consumers are most of the times sitting idle and consume the messages once in a while. Are 40 consumers enough to handle the above scenario (40 topics with 40 partitions and each topic has around 215 messages (430 K) of ingestions per second? Please help.
apache-kafka kafka-consumer-api
asked Nov 10 at 5:54
Dheeraj Kalathil
32
We have a Kafka implementation of around 40 topics and each topic has 40 partitions and 3 replications. We have 7 brokers and 40 consumers in the Kafka setup. All nodes (for brokers and consumers) are of reasonable configurations, hosted and AWS and we hardly see any spikes in any of the machines. But somehow, we are finding that the consumer lag is very high, despite adding 40 consumers to cater the read for the above setup. This is despite the fact that we have only around 215 messages (each message is of around 2KB in size) of ingestion per second, to the above topics. Have tried everything possible, but we are not able to solve the lag issue.
We also see the consumers are most of the times sitting idle and consume the messages once in a while. Are 40 consumers enough to handle the above scenario (40 topics with 40 partitions and each topic has around 215 messages (430 K) of ingestions per second? Please help.
apache-kafka kafka-consumer-api
apache-kafka kafka-consumer-api
asked Nov 10 at 5:54
Dheeraj Kalathil
32
asked Nov 10 at 5:54
Dheeraj Kalathil
32
asked Nov 10 at 5:54
Dheeraj Kalathil
32
asked Nov 10 at 5:54
Dheeraj Kalathil
32
asked Nov 10 at 5:54
Dheeraj Kalathil
32
32
so you mean each consumer is consuming from 1 partitions of 40 distinct topics, (total 1 consumer = 40 partitions which is huge for 1 consumer)
– Deadpool
Nov 10 at 6:05
try using a consumer group for consumers and keep the number of consumers equal to the number of partitions on that topic. I would separate out the consumers for set of topics into different groups. ex. 40 consumer dedicatedly listening to 10 topics, and 4 such group(totally running 160 consumers). this would need some trial and error to figure out the best combination that works for your SLAs.
– segFaulter
Nov 10 at 6:11
@Deadpool, thank you for the comment. I think, you are spot on the scenario. Yes, each consumer is reading from 1 partition of 40 distinct topics, which I agree can be huge. In that case too, I thought the consumers should be reading continuously, but I can see that the consumers are sitting idle for a long time and consumes once in a while, which is very confusing.
– Dheeraj Kalathil
Nov 11 at 4:52
@segFaulter, thank you for the comment. Each consumer instance uses around 1 GB RAM to run, so adding 160 consumers will incur a lot of cost in terms of setting up the environment.
– Dheeraj Kalathil
Nov 11 at 4:57
consumer can poll from one partition at a time, (so it should poll from 1 to 40 assume each poll will take 1 sec, so 40 polls to complete all partitions assigned to it will take 40 seconds), consumer can sit ideal when there is no enough data to poll in partitions, there are so couple of factors on consumers causing this
– Deadpool
Nov 11 at 4:58
add a comment |
so you mean each consumer is consuming from 1 partitions of 40 distinct topics, (total 1 consumer = 40 partitions which is huge for 1 consumer)
– Deadpool
Nov 10 at 6:05
try using a consumer group for consumers and keep the number of consumers equal to the number of partitions on that topic. I would separate out the consumers for set of topics into different groups. ex. 40 consumer dedicatedly listening to 10 topics, and 4 such group(totally running 160 consumers). this would need some trial and error to figure out the best combination that works for your SLAs.
– segFaulter
Nov 10 at 6:11
@Deadpool, thank you for the comment. I think, you are spot on the scenario. Yes, each consumer is reading from 1 partition of 40 distinct topics, which I agree can be huge. In that case too, I thought the consumers should be reading continuously, but I can see that the consumers are sitting idle for a long time and consumes once in a while, which is very confusing.
– Dheeraj Kalathil
Nov 11 at 4:52
@segFaulter, thank you for the comment. Each consumer instance uses around 1 GB RAM to run, so adding 160 consumers will incur a lot of cost in terms of setting up the environment.
– Dheeraj Kalathil
Nov 11 at 4:57
consumer can poll from one partition at a time, (so it should poll from 1 to 40 assume each poll will take 1 sec, so 40 polls to complete all partitions assigned to it will take 40 seconds), consumer can sit ideal when there is no enough data to poll in partitions, there are so couple of factors on consumers causing this
– Deadpool
Nov 11 at 4:58
so you mean each consumer is consuming from 1 partitions of 40 distinct topics, (total 1 consumer = 40 partitions which is huge for 1 consumer)
– Deadpool
Nov 10 at 6:05
so you mean each consumer is consuming from 1 partitions of 40 distinct topics, (total 1 consumer = 40 partitions which is huge for 1 consumer)
– Deadpool
Nov 10 at 6:05
try using a consumer group for consumers and keep the number of consumers equal to the number of partitions on that topic. I would separate out the consumers for set of topics into different groups. ex. 40 consumer dedicatedly listening to 10 topics, and 4 such group(totally running 160 consumers). this would need some trial and error to figure out the best combination that works for your SLAs.
– segFaulter
Nov 10 at 6:11
try using a consumer group for consumers and keep the number of consumers equal to the number of partitions on that topic. I would separate out the consumers for set of topics into different groups. ex. 40 consumer dedicatedly listening to 10 topics, and 4 such group(totally running 160 consumers). this would need some trial and error to figure out the best combination that works for your SLAs.
– segFaulter
Nov 10 at 6:11
@Deadpool, thank you for the comment. I think, you are spot on the scenario. Yes, each consumer is reading from 1 partition of 40 distinct topics, which I agree can be huge. In that case too, I thought the consumers should be reading continuously, but I can see that the consumers are sitting idle for a long time and consumes once in a while, which is very confusing.
– Dheeraj Kalathil
Nov 11 at 4:52
@Deadpool, thank you for the comment. I think, you are spot on the scenario. Yes, each consumer is reading from 1 partition of 40 distinct topics, which I agree can be huge. In that case too, I thought the consumers should be reading continuously, but I can see that the consumers are sitting idle for a long time and consumes once in a while, which is very confusing.
– Dheeraj Kalathil
Nov 11 at 4:52
@segFaulter, thank you for the comment. Each consumer instance uses around 1 GB RAM to run, so adding 160 consumers will incur a lot of cost in terms of setting up the environment.
– Dheeraj Kalathil
Nov 11 at 4:57
@segFaulter, thank you for the comment. Each consumer instance uses around 1 GB RAM to run, so adding 160 consumers will incur a lot of cost in terms of setting up the environment.
– Dheeraj Kalathil
Nov 11 at 4:57
consumer can poll from one partition at a time, (so it should poll from 1 to 40 assume each poll will take 1 sec, so 40 polls to complete all partitions assigned to it will take 40 seconds), consumer can sit ideal when there is no enough data to poll in partitions, there are so couple of factors on consumers causing this
– Deadpool
Nov 11 at 4:58
consumer can poll from one partition at a time, (so it should poll from 1 to 40 assume each poll will take 1 sec, so 40 polls to complete all partitions assigned to it will take 40 seconds), consumer can sit ideal when there is no enough data to poll in partitions, there are so couple of factors on consumers causing this
– Deadpool
Nov 11 at 4:58
add a comment |
1 Answer
1
active
oldest
votes
It's not clear what group ID you've specified or what topics you are assigning to which consumer.
Assuming all consumers are reading from all topics (you subscribed to a pattern .*), then you're missing out on 1560 partitions that can have dedicated consumer instances (40*40 total partitions in the cluster - 40 existing "active" consumer threads).
Since a consumer can only read from one partition at a time, not all partitions of a given topic at once, sounds to me like you'll need to add more consumers, ideally spread over several application instances / machines.
answered Nov 10 at 13:08
cricket_007
79.3k1142109
Thank you for the answer. All the consumers are using same group ID. May be, I understand your suggestion of using 1600 consumers to get each consumer read from specifically one partition. Though, that will add up a lot to the cost. Also, what I am not able to understand is why each consumer is waiting a long time between reads. If the consumers were running without idling in between, they would have been able to catch up with the speed since we have only around 215 messages per second getting inserted to all the topics (around 450 KB of data per second, in total).
– Dheeraj Kalathil
Nov 11 at 5:07
I'm not exactly sure how you're measuring idleness, but not every consumer needs to be done on one machine
– cricket_007
Nov 11 at 7:47
Thank you for the response. I have the consumers spread over 10 different nodes, each node running 4 consumers each. The consumers are running using pm2 and the consumer apps are writing to logs upon consumption of messages. This is how I am able to figure out when the consumers are remaining idle.
– Dheeraj Kalathil
Nov 11 at 10:39
4 consumers per machine is a fairy low number.... Threads are cheap... I would definitely suggest making it higher
– cricket_007
Nov 11 at 17:33
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53236368%2fkafka-consumers-reading-too-slow%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
It's not clear what group ID you've specified or what topics you are assigning to which consumer.
Assuming all consumers are reading from all topics (you subscribed to a pattern .*), then you're missing out on 1560 partitions that can have dedicated consumer instances (40*40 total partitions in the cluster - 40 existing "active" consumer threads).
Since a consumer can only read from one partition at a time, not all partitions of a given topic at once, sounds to me like you'll need to add more consumers, ideally spread over several application instances / machines.
answered Nov 10 at 13:08
cricket_007
79.3k1142109
Thank you for the answer. All the consumers are using same group ID. May be, I understand your suggestion of using 1600 consumers to get each consumer read from specifically one partition. Though, that will add up a lot to the cost. Also, what I am not able to understand is why each consumer is waiting a long time between reads. If the consumers were running without idling in between, they would have been able to catch up with the speed since we have only around 215 messages per second getting inserted to all the topics (around 450 KB of data per second, in total).
– Dheeraj Kalathil
Nov 11 at 5:07
I'm not exactly sure how you're measuring idleness, but not every consumer needs to be done on one machine
– cricket_007
Nov 11 at 7:47
Thank you for the response. I have the consumers spread over 10 different nodes, each node running 4 consumers each. The consumers are running using pm2 and the consumer apps are writing to logs upon consumption of messages. This is how I am able to figure out when the consumers are remaining idle.
– Dheeraj Kalathil
Nov 11 at 10:39
4 consumers per machine is a fairy low number.... Threads are cheap... I would definitely suggest making it higher
– cricket_007
Nov 11 at 17:33
add a comment |
It's not clear what group ID you've specified or what topics you are assigning to which consumer.
Assuming all consumers are reading from all topics (you subscribed to a pattern .*), then you're missing out on 1560 partitions that can have dedicated consumer instances (40*40 total partitions in the cluster - 40 existing "active" consumer threads).
Since a consumer can only read from one partition at a time, not all partitions of a given topic at once, sounds to me like you'll need to add more consumers, ideally spread over several application instances / machines.
answered Nov 10 at 13:08
cricket_007
79.3k1142109
Thank you for the answer. All the consumers are using same group ID. May be, I understand your suggestion of using 1600 consumers to get each consumer read from specifically one partition. Though, that will add up a lot to the cost. Also, what I am not able to understand is why each consumer is waiting a long time between reads. If the consumers were running without idling in between, they would have been able to catch up with the speed since we have only around 215 messages per second getting inserted to all the topics (around 450 KB of data per second, in total).
– Dheeraj Kalathil
Nov 11 at 5:07
I'm not exactly sure how you're measuring idleness, but not every consumer needs to be done on one machine
– cricket_007
Nov 11 at 7:47
Thank you for the response. I have the consumers spread over 10 different nodes, each node running 4 consumers each. The consumers are running using pm2 and the consumer apps are writing to logs upon consumption of messages. This is how I am able to figure out when the consumers are remaining idle.
– Dheeraj Kalathil
Nov 11 at 10:39
4 consumers per machine is a fairy low number.... Threads are cheap... I would definitely suggest making it higher
– cricket_007
Nov 11 at 17:33
add a comment |
It's not clear what group ID you've specified or what topics you are assigning to which consumer.
Assuming all consumers are reading from all topics (you subscribed to a pattern .*), then you're missing out on 1560 partitions that can have dedicated consumer instances (40*40 total partitions in the cluster - 40 existing "active" consumer threads).
Since a consumer can only read from one partition at a time, not all partitions of a given topic at once, sounds to me like you'll need to add more consumers, ideally spread over several application instances / machines.
answered Nov 10 at 13:08
cricket_007
79.3k1142109
It's not clear what group ID you've specified or what topics you are assigning to which consumer.
Assuming all consumers are reading from all topics (you subscribed to a pattern .*), then you're missing out on 1560 partitions that can have dedicated consumer instances (40*40 total partitions in the cluster - 40 existing "active" consumer threads).
Since a consumer can only read from one partition at a time, not all partitions of a given topic at once, sounds to me like you'll need to add more consumers, ideally spread over several application instances / machines.
answered Nov 10 at 13:08
cricket_007
79.3k1142109
answered Nov 10 at 13:08
cricket_007
79.3k1142109
answered Nov 10 at 13:08
cricket_007
79.3k1142109
answered Nov 10 at 13:08
cricket_007
79.3k1142109
79.3k1142109
Thank you for the answer. All the consumers are using same group ID. May be, I understand your suggestion of using 1600 consumers to get each consumer read from specifically one partition. Though, that will add up a lot to the cost. Also, what I am not able to understand is why each consumer is waiting a long time between reads. If the consumers were running without idling in between, they would have been able to catch up with the speed since we have only around 215 messages per second getting inserted to all the topics (around 450 KB of data per second, in total).
– Dheeraj Kalathil
Nov 11 at 5:07
I'm not exactly sure how you're measuring idleness, but not every consumer needs to be done on one machine
– cricket_007
Nov 11 at 7:47
Thank you for the response. I have the consumers spread over 10 different nodes, each node running 4 consumers each. The consumers are running using pm2 and the consumer apps are writing to logs upon consumption of messages. This is how I am able to figure out when the consumers are remaining idle.
– Dheeraj Kalathil
Nov 11 at 10:39
4 consumers per machine is a fairy low number.... Threads are cheap... I would definitely suggest making it higher
– cricket_007
Nov 11 at 17:33
add a comment |
Thank you for the answer. All the consumers are using same group ID. May be, I understand your suggestion of using 1600 consumers to get each consumer read from specifically one partition. Though, that will add up a lot to the cost. Also, what I am not able to understand is why each consumer is waiting a long time between reads. If the consumers were running without idling in between, they would have been able to catch up with the speed since we have only around 215 messages per second getting inserted to all the topics (around 450 KB of data per second, in total).
– Dheeraj Kalathil
Nov 11 at 5:07
I'm not exactly sure how you're measuring idleness, but not every consumer needs to be done on one machine
– cricket_007
Nov 11 at 7:47
Thank you for the response. I have the consumers spread over 10 different nodes, each node running 4 consumers each. The consumers are running using pm2 and the consumer apps are writing to logs upon consumption of messages. This is how I am able to figure out when the consumers are remaining idle.
– Dheeraj Kalathil
Nov 11 at 10:39
4 consumers per machine is a fairy low number.... Threads are cheap... I would definitely suggest making it higher
– cricket_007
Nov 11 at 17:33
Thank you for the answer. All the consumers are using same group ID. May be, I understand your suggestion of using 1600 consumers to get each consumer read from specifically one partition. Though, that will add up a lot to the cost. Also, what I am not able to understand is why each consumer is waiting a long time between reads. If the consumers were running without idling in between, they would have been able to catch up with the speed since we have only around 215 messages per second getting inserted to all the topics (around 450 KB of data per second, in total).
– Dheeraj Kalathil
Nov 11 at 5:07
Thank you for the answer. All the consumers are using same group ID. May be, I understand your suggestion of using 1600 consumers to get each consumer read from specifically one partition. Though, that will add up a lot to the cost. Also, what I am not able to understand is why each consumer is waiting a long time between reads. If the consumers were running without idling in between, they would have been able to catch up with the speed since we have only around 215 messages per second getting inserted to all the topics (around 450 KB of data per second, in total).
– Dheeraj Kalathil
Nov 11 at 5:07
I'm not exactly sure how you're measuring idleness, but not every consumer needs to be done on one machine
– cricket_007
Nov 11 at 7:47
I'm not exactly sure how you're measuring idleness, but not every consumer needs to be done on one machine
– cricket_007
Nov 11 at 7:47
Thank you for the response. I have the consumers spread over 10 different nodes, each node running 4 consumers each. The consumers are running using pm2 and the consumer apps are writing to logs upon consumption of messages. This is how I am able to figure out when the consumers are remaining idle.
– Dheeraj Kalathil
Nov 11 at 10:39
Thank you for the response. I have the consumers spread over 10 different nodes, each node running 4 consumers each. The consumers are running using pm2 and the consumer apps are writing to logs upon consumption of messages. This is how I am able to figure out when the consumers are remaining idle.
– Dheeraj Kalathil
Nov 11 at 10:39
4 consumers per machine is a fairy low number.... Threads are cheap... I would definitely suggest making it higher
– cricket_007
Nov 11 at 17:33
4 consumers per machine is a fairy low number.... Threads are cheap... I would definitely suggest making it higher
– cricket_007
Nov 11 at 17:33
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53236368%2fkafka-consumers-reading-too-slow%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
so you mean each consumer is consuming from 1 partitions of 40 distinct topics, (total 1 consumer = 40 partitions which is huge for 1 consumer)
– Deadpool
Nov 10 at 6:05
try using a consumer group for consumers and keep the number of consumers equal to the number of partitions on that topic. I would separate out the consumers for set of topics into different groups. ex. 40 consumer dedicatedly listening to 10 topics, and 4 such group(totally running 160 consumers). this would need some trial and error to figure out the best combination that works for your SLAs.
– segFaulter
Nov 10 at 6:11
@Deadpool, thank you for the comment. I think, you are spot on the scenario. Yes, each consumer is reading from 1 partition of 40 distinct topics, which I agree can be huge. In that case too, I thought the consumers should be reading continuously, but I can see that the consumers are sitting idle for a long time and consumes once in a while, which is very confusing.
– Dheeraj Kalathil
Nov 11 at 4:52
@segFaulter, thank you for the comment. Each consumer instance uses around 1 GB RAM to run, so adding 160 consumers will incur a lot of cost in terms of setting up the environment.
– Dheeraj Kalathil
Nov 11 at 4:57
consumer can poll from one partition at a time, (so it should poll from 1 to 40 assume each poll will take 1 sec, so 40 polls to complete all partitions assigned to it will take 40 seconds), consumer can sit ideal when there is no enough data to poll in partitions, there are so couple of factors on consumers causing this
– Deadpool
Nov 11 at 4:58