June's Studio.

Kafka-压力测试

MQ Kafka
字数统计: 2.3k阅读时长: 12 min
2019/03/20

测试目的

本地单机环境对 Kafka 处理 MQ 消息能力进行压力测试。测试包括对 Kafka 写入 MQ 消息和消费 MQ 消息进行压力测试，根据 10w、100w 和 1000w 级别的消息处理结果，评估 Kafka 的处理性能是否满足项目需求

测试范围及方法

测试使用 Kafka 自带的测试脚本，通过命令对 Kafka 发起写入 MQ 消息和 Kafka 消费 MQ 消息的请求。
模拟不同数量级的 MQ 消息写入和 MQ 消息消费场景，根据 Kafka 的处理结果，评估 Kafka 处理消息的能力

测试场景：

本机信息

操作系统：Windows 7 Enterprise Edition Service Pack 1 (Build 7601), 64-bit
处理器：Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz 四核
内存：16.00 GB
硬盘：西部数据 WDCWD10EZEX-75WN4A0(1000GB)
kafka版本： 2.11-2.1.1

写入压力测试

|测试场景 | MQ 消息数 | 每秒写入消息数 | 记录大小 (单位：字节) | 测试命令
| —— | —– | —— |—— |——
|Kafka 消息写入测试 | 10W | 2000 条 | 1000 | kafka-producer-perf-test.bat –topic test_perf –num-records 100000 –record-size 1000 –throughput 2000 –producer-props bootstrap.servers=127.0.0.1:9092
| | 10W | 5000 条 | 1000 | kafka-producer-perf-test.bat –topic test_perf –num-records 1000000 –record-size 2000 –throughput 5000 –producer-props bootstrap.servers=127.0.0.1:9092
| | 1000W | 5000 条 | 1000 | kafka-producer-perf-test.bat –topic test_perf –num-records 10000000 –record-size 2000 –throughput 5000 –producer-props bootstrap.servers=127.0.0.1:9092

消费压力测试

|测试场景 | MQ 消息数 |测试命令
| :—— : | :—— : | :—— :
|Kafka 消息消费测试 | 10W |kafka-consumer-perf-test.bat –broker-list localhost:9092 –topic test_perf –fetch-size 1048576 –messages 100000 –threads 1
| | 100W |kafka-consumer-perf-test.bat –broker-list localhost:9092 –topic test_perf –fetch-size 1048576 –messages 1000000 –threads 1
| | 1000W |kafka-consumer-perf-test.bat –broker-list localhost:9092 –topic test_perf –fetch-size 1048576 –messages 10000000 –threads 1

测试结果

写入测试

  • 10W写入

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    D:\dev\kafka_2.11-2.1.1\bin\windows
    λ kafka-producer-perf-test.bat --topic test_perf --num-records 100000 --record-size 1000 --throughput 2000 --producer-props bootstrap.servers=127.0.0.1:9092
    10002 records sent, 1999.6 records/sec (1.91 MB/sec), 1.2 ms avg latency, 147.0 max latency.
    10008 records sent, 2001.2 records/sec (1.91 MB/sec), 0.3 ms avg latency, 17.0 max latency.
    10004 records sent, 2000.8 records/sec (1.91 MB/sec), 0.2 ms avg latency, 5.0 max latency.
    10049 records sent, 2009.4 records/sec (1.92 MB/sec), 0.2 ms avg latency, 24.0 max latency.
    10003 records sent, 2000.6 records/sec (1.91 MB/sec), 0.2 ms avg latency, 4.0 max latency.
    10006 records sent, 2000.8 records/sec (1.91 MB/sec), 0.2 ms avg latency, 3.0 max latency.
    10014 records sent, 2002.4 records/sec (1.91 MB/sec), 0.2 ms avg latency, 6.0 max latency.
    10009 records sent, 2001.4 records/sec (1.91 MB/sec), 0.2 ms avg latency, 9.0 max latency.
    10005 records sent, 2000.6 records/sec (1.91 MB/sec), 0.2 ms avg latency, 3.0 max latency.
    100000 records sent, 1999.800020 records/sec (1.91 MB/sec), 0.29 ms avg latency, 147.00 ms max latency, 0 ms 50th, 1 ms 95th, 2 ms 99th, 25 ms 99.9th.

    本例中写入 100w 条 MQ 消息为例，每秒平均向 kafka 写入了 1.91MB 的数据，大概是 1999.800条消息 / 秒，每次写入的平均延迟为 0.29 毫秒，最大的延迟为 147 毫秒。

  • 100w条写入

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    D:\dev\kafka_2.11-2.1.1\bin\windows
    λ ckafka-producer-perf-test.bat --topic test_perf --num-records 1000000 --record-size 1000 --throughput 5000 --producer-props bootstrap.servers=127.0.0.1:9092
    25002 records sent, 4997.4 records/sec (4.77 MB/sec), 1.7 ms avg latency, 155.0 max latency.
    25020 records sent, 5004.0 records/sec (4.77 MB/sec), 0.4 ms avg latency, 12.0 max latency.
    25015 records sent, 5002.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 5.0 max latency.
    25015 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 6.0 max latency.
    25025 records sent, 5002.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 6.0 max latency.
    25020 records sent, 5002.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 12.0 max latency.
    25005 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 8.0 max latency.
    25018 records sent, 5003.6 records/sec (4.77 MB/sec), 0.3 ms avg latency, 5.0 max latency.
    25007 records sent, 4999.4 records/sec (4.77 MB/sec), 0.3 ms avg latency, 13.0 max latency.
    25010 records sent, 5002.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 4.0 max latency.
    25015 records sent, 5001.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 6.0 max latency.
    25020 records sent, 5002.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 5.0 max latency.
    25015 records sent, 5003.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 3.0 max latency.
    25015 records sent, 5002.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 15.0 max latency.
    25015 records sent, 5003.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 3.0 max latency.
    25020 records sent, 5003.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 4.0 max latency.
    25005 records sent, 5001.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 5.0 max latency.
    25010 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 7.0 max latency.
    25010 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 5.0 max latency.
    25020 records sent, 5004.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 4.0 max latency.
    25005 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 5.0 max latency.
    25005 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 14.0 max latency.
    25010 records sent, 5001.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 6.0 max latency.
    25020 records sent, 5003.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 7.0 max latency.
    25025 records sent, 5003.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 4.0 max latency.
    25004 records sent, 5000.8 records/sec (4.77 MB/sec), 0.3 ms avg latency, 4.0 max latency.
    25016 records sent, 5003.2 records/sec (4.77 MB/sec), 0.2 ms avg latency, 5.0 max latency.
    25010 records sent, 5002.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 3.0 max latency.
    25005 records sent, 5001.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 3.0 max latency.
    25010 records sent, 5002.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 13.0 max latency.
    25025 records sent, 5003.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 4.0 max latency.
    25010 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 3.0 max latency.
    25024 records sent, 5004.8 records/sec (4.77 MB/sec), 0.3 ms avg latency, 4.0 max latency.
    25001 records sent, 5000.2 records/sec (4.77 MB/sec), 0.3 ms avg latency, 10.0 max latency.
    25005 records sent, 5000.0 records/sec (4.77 MB/sec), 0.6 ms avg latency, 46.0 max latency.
    25010 records sent, 5002.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 6.0 max latency.
    25020 records sent, 5002.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 5.0 max latency.
    25020 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 7.0 max latency.
    25015 records sent, 5003.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 3.0 max latency.
    1000000 records sent, 4999.875003 records/sec (4.77 MB/sec), 0.31 ms avg latency, 155.00 ms max latency, 0 ms 50th, 1 ms 95th, 2 ms 99th, 19 ms 99.9th.

本例中写入 100w 条 MQ 消息为例，每秒平均向 kafka 写入了 4.77MB 的数据，大概是 4999.875 条消息 / 秒，每次写入的平均延迟为 0.31 毫秒，最大的延迟为 155 毫秒。

  • 1000w条数据写入测试
    1
    10000000 records sent, 4999.982500 records/sec (4.77 MB/sec), 0.93 ms avg latency, 682.00 ms max latency, 0 ms 50th, 1 ms 95th, 3 ms 99th, 209 ms 99.9th.
    本例中写入 1000w 条 MQ 消息为例，每秒平均向 kafka 写入了 4.77MB 的数据，大概是 4999.982 条消息 / 秒，每次写入的平均延迟为 0.93 毫秒，最大的延迟为 682 毫秒。
消费测试
  • 消费 10w 消息压测结果
1
2
3
4
kafka-consumer-perf-test.bat --broker-list localhost:9092 --topic test_perf --fetch-size 1048576 --messages 100000 --threads 1

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2019-03-20 17:32:23:660, 2019-03-20 17:32:36:593, 95.8204, 7.4090, 100475, 7768.8858, 3014, 9919, 9.6603, 10129.5493

以本例中消费 10w 条 MQ 消息为例总共消费了 95.8204M 的数据，每秒消费数据大小为 7.4090M，总共消费了 100475 条消息，每秒消费 7768.8858 条消息。

  • 消费 100w 消息压测结果

    1
    2
    3
    4
    D:\dev\kafka_2.11-2.1.1\bin\windows
    λ kkafka-consumer-perf-test.bat --broker-list localhost:9092 --topic test_perf --fetch-size 1048576 --messages 1000000 --threads 1
    start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
    2019-03-20 17:37:17:229, 2019-03-20 17:37:48:082, 953.8565, 30.9162, 1000191, 32417.9496, 3016, 27837, 34.2658, 35930.2727

    以本例中消费 100w 条 MQ 消息为例总共消费了 953.8565M 的数据，每秒消费数据大小为 30.9162M，总共消费了 1000191 条消息，每秒消费 32417.9496 条消息。

  • 消费 1000w 消息压测结果

1
2
3
4
D:\dev\kafka_2.11-2.1.1\bin\windows
λ kafka-consumer-perf-test.bat --broker-list localhost:9092 --topic test_perf --fetch-size 1048576 --messages 10000000 --threads 1
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2019-03-20 17:41:59:064, 2019-03-20 17:46:47:322, 9536.9205, 33.0847, 10000186, 34691.7900, 3013, 285245, 33.4341, 35058.2341

以本例中消费 1000w 条 MQ 消息为例总共消费了 9536.9205 的数据，每秒消费数据大小为 33.0847M，总共消费了 10000186 条消息，每秒消费 35058.2341 条消息。

总结

一般写入 MQ 消息设置 5000 条 / 秒时，消息延迟时间小于等于 1ms，在可接受范围内，说明消息写入及时。

Kafka 消费 MQ 消息时，1000W 待处理消息的处理能力如果在每秒 20w 条以上，那么处理结果是理想的。

根据 Kafka 处理 10w、100w 和 1000w 级的消息时的处理能力，可以评估出 Kafka 集群服务，是否有能力处理上亿级别的消息。

本次测试是在单台服务器上进行，基本不需要考虑网络带宽的影响。

原文作者：June

原文链接：https://jatshw.com/2019/03/20/MQ-kafka%E5%8E%8B%E5%8A%9B%E6%B5%8B%E8%AF%95/

发表日期：March 20th 2019, 5:52:07 pm

更新日期：March 21st 2023, 9:53:55 pm

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
  1. 1. 测试目的
  2. 2. 测试范围及方法
  3. 3. 测试场景：
    1. 3.1. 本机信息
    2. 3.2. 写入压力测试
    3. 3.3. 消费压力测试
  4. 4. 测试结果
    1. 4.1. 写入测试
    2. 4.2. 消费测试
  5. 5. 总结
Total : 83
2023
2022
2021
2020
2019
2018
2017
PC 3C 工作总结 旅游 球鞋 Interview Algorithms Apm server Docker Golang 面试题 JDBC Java Jvm Log 线程 Jdk 限流 GC 车 ActiveMQ MQ Kafka MySQL Netty Nginx 分布式 Redis ServicemeshMeetUp Kafka - MQ Spring SpringBoot SSL 场景 设计模式
LIFE DEV BLOG