kafka python客户端连接风暴_Apache 风暴 kafka spout 只读取主题分区的一半
发布日期:2021-06-24 13:33:19 浏览次数:2 分类:技术文章

本文共 3764 字,大约阅读时间需要 12 分钟。

在我们的生产 Storm 集群上开发的一个问题,我们无法弄清楚或解决。

在某些时候,似乎 kafka 鲸鱼喷水停止从一半的主题分区读取。有 40 个分区,它只读取其中的 20 个。我们无法在发生这种情况时找到我们对风暴集群或 kafka 所做的任何更改。

我们更改了使用者组 ID 并将 spout config startOffsetTime设置为OffsetRequest.LatestTime,以尝试从所有分区读取新数据。它仍然只连接到相同的 20 个分区。我们查看了 Storm zookeeper 中的节点//,看到那里只有 20 个分区。

我们已经验证消息正在发布到所有 40 个分区。

Kafka 版本是 0.9.0.1,风暴版本是 1.1.0.

任何关于如何调试或在哪里看的提示都将非常感激。我是否提到这是在生产中发生的?我是否在一周前开始提到它,我们今天早上才注意到了? :(

附加信息:我们在 Kafka 状态更改日志中发现了一些错误(分区 9 是受影响的分区之一,日志中的时间戳看起来与问题开始的时间有关)

kafka.common.NoReplicaOnlineException: No replica for partition

[transcription-results,9] is alive. Live brokers are: [Set()], Assigned replicas are: [List(1, 4, 0)]

[2018-03-14 03:11:40,863] TRACE Controller 0 epoch 44 changed state of replica 1 for partition [transcription-results,9] from OnlineReplica to OfflineReplica (state.change.logger)

[2018-03-14 03:11:41,141] TRACE Controller 0 epoch 44 sending become-follower LeaderAndIsr request (Leader:-1,ISR:0,4,LeaderEpoch:442,ControllerEpoch:44) to broker 4 for partition [transcription-results,9] (state.change.logger)

[2018-03-14 03:11:41,145] TRACE Controller 0 epoch 44 sending become-follower LeaderAndIsr request (Leader:-1,ISR:0,4,LeaderEpoch:442,ControllerEpoch:44) to broker 0 for partition [transcription-results,9] (state.change.logger)

[2018-03-14 03:11:41,208] TRACE Controller 0 epoch 44 changed state of replica 4 for partition [transcription-results,9] from OnlineReplica to OnlineReplica (state.change.logger)

[2018-03-14 03:11:41,218] TRACE Controller 0 epoch 44 changed state of replica 1 for partition [transcription-results,9] from OfflineReplica to OnlineReplica (state.change.logger)

[2018-03-14 03:11:41,226] TRACE Controller 0 epoch 44 sending become-follower LeaderAndIsr request (Leader:-1,ISR:0,4,LeaderEpoch:442,ControllerEpoch:44) to broker 4 for partition [transcription-results,9] (state.change.logger)

[2018-03-14 03:11:41,230] TRACE Controller 0 epoch 44 sending become-follower LeaderAndIsr request (Leader:-1,ISR:0,4,LeaderEpoch:442,ControllerEpoch:44) to broker 1 for partition [transcription-results,9] (state.change.logger)

[2018-03-14 03:11:41,450] TRACE Broker 0 received LeaderAndIsr request (LeaderAndIsrInfo:Leader:-1,ISR:0,4,LeaderEpoch:442,ControllerEpoch:44),ReplicationFactor:3),AllReplicas:1,4,0) correlation id 158 from controller 0 epoch 44 for partition [transcription-results,9] (state.change.logger)

[2018-03-14 03:11:41,454] TRACE Broker 0 handling LeaderAndIsr request correlationId 158 from controller 0 epoch 44 starting the become-follower transition for partition [transcription-results,9] (state.change.logger)

[2018-03-14 03:11:41,455] ERROR Broker 0 received LeaderAndIsrRequest with correlation id 158 from controller 0 epoch 44 for partition [transcription-results,9] but cannot become follower since the new leader -1 is unavailable. (state.change.logger)

//... removed some TRACE statements here

[2018-03-14 03:11:41,908] WARN Broker 0 ignoring LeaderAndIsr request from controller 1 with correlation id 1 epoch 47 for partition [transcription-results,9] since its associated leader epoch 441 is old. Current leader epoch is 441 (state.change.logger)

[2018-03-14 03:11:41,982] TRACE Broker 0 cached leader info (LeaderAndIsrInfo:Leader:1,ISR:0,1,4,LeaderEpoch:441,ControllerEpoch:44),ReplicationFactor:3),AllReplicas:1,4,0) for partition [transcription-results,9] in response to UpdateMetadata request sent by controller 1 epoch 47 with correlation id 2 (state.change.logger)

[2018-03-22 14:43:36,098] TRACE Broker 0 received LeaderAndIsr request (LeaderAndIsrInfo:Leader:-1,ISR:,LeaderEpoch:444,ControllerEpoch:47),ReplicationFactor:3),AllReplicas:1,4,0) correlation id 679 from controller 1 epoch 47 for partition [transcription-results,9] (state.change.logger)

但是我们怎样才能从中恢复呢?

转载地址:https://blog.csdn.net/weixin_33212263/article/details/113393785 如侵犯您的版权,请留言回复原文章的地址,我们会给您删除此文章,给您带来不便请您谅解!

上一篇:python 硬编码_试试 python-dotenv,避免敏感信息被硬编码到代码中
下一篇:win+d没反应_黑科技!2.5D风格场景插画立体图案一键生成PS插件

发表评论

最新留言

留言是一种美德,欢迎回访!
[***.207.175.100]2024年04月12日 21时52分08秒