ClickHouse Kafka引擎表 Application maximum poll interval … leaving group

线上ClickHouse集群发现Kafka引擎表不能正常工作，物化视图未触发，无法将数据正常的吸取到ClickHouse本地表中,检查ClickHouse日志(clickhouse-server.err.log)发现Kafka引擎表出现poll失利退出的问题,日志中显现如下内容

2022.03.09 16:49:11.188012 [ 538056 ] {de2d06ca-8b96-4ad7-b805-09414f05fdad} <Error> StorageKafka (kafka_engine_table): Error during draining: Local: Maximum application poll interval (max.poll.interval.ms) exceeded
2022.03.09 16:49:11.289315 [ 538056 ] {de2d06ca-8b96-4ad7-b805-09414f05fdad} <Error> StorageKafka (kafka_engine_table): Error during draining: Local: Maximum application poll interval (max.poll.interval.ms) exceeded
2022.03.09 16:55:40.776456 [ 8094 ] {} <Warning> StorageKafka (kafka_engine_table): [rdk:MAXPOLL] [thrd:main]: Application maximum poll interval (300000ms) exceeded by 51ms (adjust max.poll.interval.ms for long-running message processing): leaving group
2022.03.09 16:55:45.277079 [ 8100 ] {} <Warning> StorageKafka (kafka_engine_table): [rdk:MAXPOLL] [thrd:main]: Application maximum poll interval (300000ms) exceeded by 50ms (adjust max.poll.interval.ms for long-running message processing): leaving group

问题排查

Kafka引擎表在poll数据的时候直接超时并离开消费组. 可是在集群中有很多的Kafka引擎表都可以正常吸取数据,咱们首要扫除了这是网络的问题.

期间也尝试过删去Kafka引擎表重建仍是相同的问题, 直接查询Kafka引擎表可以正常获取数据 select * from kafka_engine_table_name limit 100 扫除是Kafka服务的问题.

检查了以下咱们的ClickHouse版别是 20.11.3.3, 通过翻阅ClickHouse源码发现 https://github.com/ClickHouse/ClickHouse/blob/master/src/Core/Settings.h 装备中有一个名为 background_schedule_pool_size 的装备选项.

源码解说如下:

M(UInt64, background_schedule_pool_size, 128, “Number of threads performing background tasks for replicated tables, dns cache updates. Only has meaning at server startup.”, 0)

改参数用于装备表后台运行task的选项,初步判断是这儿的问题,可是看源码这儿显现的是默认值 128 理论上咱们的集群中是不会超越这个值的, 咱们前往ClickHouse服务器中进行了参数的查询发现系统装备的是 16

然后结合该文件的提交前史发现 alesapin 2021/6/8, 18:55 Increase background schedule pool default size 提交前史的commit
然后结合发布前史版别发现改参数是在 21.3.9.83 版别以后才设置的默认值 128

咱们通过以下SQL查询ClickHouse服务器的默认装备和监控指标:

WITH t0 AS (
  SELECT
    value AS defaultValue
  FROM
    `system`.settings
  WHERE
    name = 'background_schedule_pool_size'
),
t1 AS (
  SELECT
    value AS currentValue
  FROM
    `system`.metrics
  WHERE
    metric = 'BackgroundSchedulePoolTask'
)
SELECT
  defaultValue,
  currentValue
FROM
  t0,
  t1

得到以下结果

Query id: 82dfca06-b416-49c8-8a96-5d30b2abebc2
┌─defaultValue─┬─currentValue─┐
│ 16           │          16  │
└──────────────┴──────────────┘
1 rows in set. Elapsed: 0.003 sec.

发现当时值现已达到装备的最大值，需要添加以扩展后台执行的Task使命数, 咱们参照新源码的 128 装备.

根据源码解说说该参数需要重启服务,装备完成后重启ClickHouse集群.

装备方法

修正 users.xml 添加如下装备

<background_schedule_pool_size>128</background_schedule_pool_size>

添加到 profiles ---> default 节点下

重启ClickHouse集群后,Kafka引擎表可以正常吸取数据!!!