聊聊 HDFS Balancer负载均衡器

大数据 数据分析
平衡指的是 每个DataNode的利用率 与 集群的利用率 之间相差不超过给定的阈值百分比。此处的平衡 指的是各个DataNode之间的平衡,同一个DataNode之间的各个磁盘是不会平衡的。

1、背景

当我们的hadoop集群运行了一段时间之后,各个DataNode上的数据分布并不一定是均匀分布的。比如说: 我们向现有集群中添加了一个新的DataNode。

DataNode数据不均衡

2、什么是平衡

此处是我自己的一个简单的理解

所谓的平衡指的是 每个DataNode的利用率 与 集群的利用率 之间相差不超过给定的阈值百分比。此处的平衡 指的是各个DataNode之间的平衡,同一个DataNode之间的各个磁盘是不会平衡的。

2.1 每个DataNode的利用率计算

每个DataNode的利用率计算

DataNode的利用率= dfs已用的空间 / 分配给dfs的空间。

注意: 分配给dfs的空间 不是磁盘的总空间。

2.2 集群的利用率

集群的利用率

集群的利用率= 各datanode dfs已使用的空间 / 各datanode总空间

2.3 平衡

假设平衡的阈值是 5%,集群的利用率是 37.5,那么每个节点的利用率在32.5%到42.5%之间都认为是均衡的。也就是说,极端情况下,DataNode的利用率最大相差10%。

3、hdfs balancer语法

[hadoopdeploy@hadoop01 ~]$ hdfs balancer --help
Usage: hdfs balancer
[-policy <policy>] the balancing policy: datanode or blockpool
[-threshold <threshold>] Percentage of disk capacity
[-exclude [-f <hosts-file> | <comma-separated list of hosts>]] Excludes the specified datanodes.
[-include [-f <hosts-file> | <comma-separated list of hosts>]] Includes only the specified datanodes.
[-source [-f <hosts-file> | <comma-separated list of hosts>]] Pick only the specified datanodes as source nodes.
[-blockpools <comma-separated list of blockpool ids>] The balancer will only run on blockpools included in this list.
[-idleiterations <idleiterations>] Number of consecutive idle iterations (-1 for Infinite) before exit.
[-runDuringUpgrade] Whether to run the balancer during an ongoing HDFS upgrade.This is usually not desired since it will not affect used space on over-utilized machines.
[-asService] Run as a long running service.

Generic options supported are:
-conf <configuration file> specify an application configuration file
-D <property=value> define a value for a given property
-fs <file:///|hdfs://namenode:port> specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations.
-jt <local|resourcemanager:port> specify a ResourceManager
-files <file1,...> specify a comma-separated list of files to be copied to the map reduce cluster
-libjars <jar1,...> specify a comma-separated list of jar files to be included in the classpath
-archives <archive1,...> specify a comma-separated list of archives to be unarchived on the compute machines

The general command line syntax is:
command [genericOptions] [commandOptions]

参数

描述

threshold

磁盘容量的百分比。默认值为10%,表示上下浮动10%。

policy

平衡策略。
datanode(默认):当每一个DataNode是平衡的时候,集群就是平衡的。
blockpool:当每一个DataNode中的blockpool是平衡的,集群就是平衡的。

exclude

不参与平衡的DataNode节点

include

参与平衡的DataNode节点

source

仅选取指定的数据节点作为源节点

blockpools

Balancer仅在指定的blockpools中运行

idleiterations

退出前的连续空闲迭代次数(-1表示无限)

-runDuringUpgrade

是否在正在进行的HDFS升级过程中运行平衡器。通常不需要这样做,因为这不会影响过度使用的计算机上的已用空间。

-asService

作为长期运行的服务运行


4、运行一个简单的balance案例

4.1 设置平衡数据传输带宽

[hadoopdeploy@hadoop01 ~]$ hdfs dfsadmin  -setBalancerBandwidth 10485760
Balancer bandwidth is set to 10485760
[hadoopdeploy@hadoop01 ~]$

当我们的集群负载需要调低这个值,当我们的集群负载较低时,可以适当调高这个值。

4.2 执行banalce

[hadoopdeploy@hadoop01 ~]$ hdfs balancer -policy datanode -threshold 5
2023-03-26 14:10:09,785 INFO balancer.Balancer: Using a threshold of 5.0
2023-03-26 14:10:09,786 INFO balancer.Balancer: namenodes = [hdfs://hadoop01:8020]
2023-03-26 14:10:09,786 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 5.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false]
2023-03-26 14:10:09,786 INFO balancer.Balancer: included nodes = []
2023-03-26 14:10:09,786 INFO balancer.Balancer: excluded nodes = []
2023-03-26 14:10:09,786 INFO balancer.Balancer: source nodes = []
Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved NameNode
2023-03-26 14:10:09,787 INFO balancer.NameNodeConnector: getBlocks calls for hdfs://hadoop01:8020 will be rate-limited to 20 per second
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.namenode.get-blocks.max-qps = 20 (default=20)
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 100 (default=100)
2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.datanode.balance.bandwidthPerSec = 104857600 (default=104857600)
2023-03-26 14:10:10,395 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)
2023-03-26 14:10:10,395 INFO balancer.Balancer: dfs.blocksize = 134217728 (default=134217728)
2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.141:9866
2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.140:9866
2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.142:9866
2023-03-26 14:10:10,402 INFO balancer.Balancer: 0 over-utilized: []
2023-03-26 14:10:10,402 INFO balancer.Balancer: 0 underutilized: []
2023-3-26 14:10:10 0 0 B 0 B 0 B 0 hdfs://hadoop01:8020
The cluster is balanced. Exiting...
2023-3-26 14:10:10 Balancing took 810.0 milliseconds
[hadoopdeploy@hadoop01 ~]$

5、参考文档

1、https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer
2、https://help.aliyun.com/document_detail/449686.html​

责任编辑:武晓燕 来源: 今日头条
相关推荐

2023-04-03 06:57:38

DataNode工具命令行

2010-05-06 10:14:31

负载均衡器

2024-02-22 10:11:00

负载均衡器反向代理

2023-02-13 16:39:45

Kubernetes容器负载均衡器

2017-05-19 14:45:01

OVN负载均衡器路由器

2022-07-14 08:53:48

MetalLBkubernetes

2010-04-22 10:46:40

Lvs负载均衡故障负载均衡器

2010-07-15 11:16:04

负载均衡

2010-05-10 14:13:26

2010-04-20 10:46:59

什么是负载均衡器

2010-05-10 18:22:51

负载均衡器

2010-04-26 15:04:08

负载均衡器

2010-05-10 14:05:31

负载均衡器

2010-05-04 13:32:37

nginx负载均衡器

2010-04-22 10:36:06

负载均衡器

2010-04-22 10:09:28

负载均衡器

2011-03-17 09:27:07

HAProxy负载均衡

2010-04-28 16:50:46

Apusic负载均衡器

2011-08-24 13:45:49

HAProxy负载均衡负载均衡器

2010-03-24 10:35:02

Nginx负载均衡器
点赞
收藏

51CTO技术栈公众号