kafka消费问题
offset is outside the range
报错内容类似如下
The Requested offset is outside the range of offsets maintained by the server for the given topic/patition.
指定消费位移时，如果对应位移超过了Kafka对应分区的范围，则会出现类似如上报错，一般原因会有以下几种
- 消费能力跟不上，Kafka集群的数据有保护期，过期后的数据会被删除，因此提供的位移可能是低于Kafka服务端的最小位移，一般有以下解决办法
- 调整消费参数，具体可以参考Scheduler配置
- 新增Scheduler节点
提示
scheduler节点数最大不能大于kafka分区数
- Kafka集群有更换主题或者清理主题数据，但是Clustermgr的消费位移未清理，导致指定offset大于Kafka集群对应分区范围
- 清理Clustermgr中保存的消费位移或者重新设置对应消费位移
无论是哪种原因，当前Scheduler服务正常启动，都需要将Clustermgr中的消费位移设置正确才能启动，下面介绍如何通过blobstore-cli工具问题处理步骤
- 查看消费位移
cm kv list --prefix blob_delete-consume_offset # 查看删除消息，默认返回10条，更多可以带上参数--count
Key : blob_delete-consume_offset-blob_delete_10001-0
Value : {"topic":"blob_delete_10001","partition":0,"offset":170446922}
Key : blob_delete-consume_offset-blob_delete_10001-1
Value : {"topic":"blob_delete_10001","partition":1,"offset":170445513}
Key : blob_delete-consume_offset-blob_delete_10001-10
Value : {"topic":"blob_delete_10001","partition":10,"offset":170440299}
Key : blob_delete-consume_offset-blob_delete_10001-11
Value : {"topic":"blob_delete_10001","partition":11,"offset":170465361}
Key : blob_delete-consume_offset-blob_delete_10001-12
Value : {"topic":"blob_delete_10001","partition":12,"offset":170462346}
Key : blob_delete-consume_offset-blob_delete_10001-13
Value : {"topic":"blob_delete_10001","partition":13,"offset":170453103}
Key : blob_delete-consume_offset-blob_delete_10001-14
Value : {"topic":"blob_delete_10001","partition":14,"offset":170463101}
Key : blob_delete-consume_offset-blob_delete_10001-15
Value : {"topic":"blob_delete_10001","partition":15,"offset":170439254}
Key : blob_delete-consume_offset-blob_delete_10001-16
Value : {"topic":"blob_delete_10001","partition":16,"offset":170470119}
Key : blob_delete-consume_offset-blob_delete_10001-17
Value : {"topic":"blob_delete_10001","partition":17,"offset":170463816}
next marker: blob_delete-consume_offset-blob_delete_10001-17
cm kv list --prefix shard_repair-consume_offset
Key : shard_repair-consume_offset-shard_repair_10001-0
Value : {"topic":"shard_repair_10001","partition":0,"offset":1140969}
Key : shard_repair-consume_offset-shard_repair_10001-1
Value : {"topic":"shard_repair_10001","partition":1,"offset":1141604}
Key : shard_repair-consume_offset-shard_repair_10001-10
Value : {"topic":"shard_repair_10001","partition":10,"offset":1141553}
Key : shard_repair-consume_offset-shard_repair_10001-11
Value : {"topic":"shard_repair_10001","partition":11,"offset":1140962}
Key : shard_repair-consume_offset-shard_repair_10001-12
Value : {"topic":"shard_repair_10001","partition":12,"offset":1140533}
Key : shard_repair-consume_offset-shard_repair_10001-13
Value : {"topic":"shard_repair_10001","partition":13,"offset":1143025}
Key : shard_repair-consume_offset-shard_repair_10001-14
Value : {"topic":"shard_repair_10001","partition":14,"offset":1140768}
Key : shard_repair-consume_offset-shard_repair_10001-15
Value : {"topic":"shard_repair_10001","partition":15,"offset":1140299}
Key : shard_repair-consume_offset-shard_repair_10001-16
Value : {"topic":"shard_repair_10001","partition":16,"offset":1141417}
Key : shard_repair-consume_offset-shard_repair_10001-17
Value : {"topic":"shard_repair_10001","partition":17,"offset":1141269}
next marker: shard_repair-consume_offset-shard_repair_10001-17
提示
如果查看某个分区的消费位移，可以cm kv list --prefix <task_type>-consume_offset-<topic>-<partition>，如cm kv list --prefix shard_repair-consume_offset-shard_repair_10001-20
- 比对Kafka服务端分区位移
核对确认Clustermgr保存的消费位移是否不在Kafka集群的对应分区范围内
停止scheduler节点
修正消费位移
# scheduler kafka set --task_type <task_type> --topic <topic> --partition <partition> <offset>
# 示例如下
scheduler kafka set --task_type shard_repair --topic shard_repair_10001 --partition 20 332053
- 启动Scheduler
在github上编辑