diff --git a/notes/installation/基于Zookeeper搭建Hadoop高可用集群.md b/notes/installation/基于Zookeeper搭建Hadoop高可用集群.md new file mode 100644 index 0000000..e324ed4 --- /dev/null +++ b/notes/installation/基于Zookeeper搭建Hadoop高可用集群.md @@ -0,0 +1,423 @@ +# 基于ZooKeeper搭建Hadoop高可用集群 + +## 一、Hadoop高可用 + +![HDFS-HA-Architecture-Edureka](D:\BigData-Notes\pictures\HDFS-HA-Architecture-Edureka.png) + +> 图片引用自：https://www.edureka.co/blog/how-to-set-up-hadoop-cluster-with-hdfs-high-availability/ + +## 二、集群规划 + + + +## 三、前置条件 + ++ 所有服务器都安装有JDK，安装步骤可以参见：[Linux下JDK的安装](https://github.com/heibaiying/BigData-Notes/blob/master/notes/installation/JDK%E5%AE%89%E8%A3%85.md)； ++ 搭建好ZooKeeper集群，搭建步骤可以参见：[Zookeeper单机环境和集群环境搭建](https://github.com/heibaiying/BigData-Notes/blob/master/notes/installation/Zookeeper单机环境和集群环境搭建.md) + ++ 所有服务器之间都配置好SSH免密登录。 + + + +## 四、集群配置 + +### 4.1 下载并解压 + +下载Hadoop。这里我下载的是CDH版本Hadoop，下载地址为：http://archive.cloudera.com/cdh5/cdh/5/ + +```shell +# tar -zvxf hadoop-2.6.0-cdh5.15.2.tar.gz +``` + +### 4.2 配置环境变量 + +编辑`profile`文件： + +```shell +# vim /etc/profile +``` + +增加如下配置： + +``` +export HADOOP_HOME=/usr/app/hadoop-2.6.0-cdh5.15.2 +export PATH=${HADOOP_HOME}/bin:$PATH +``` + +执行`source`命令，使得配置立即生效： + +```shell +# source /etc/profile +``` + +### 4.3 修改配置 + +进入`${HADOOP_HOME}/etc/hadoop`目录下，修改配置文件。各个配置文件内容如下： + +#### 1. hadoop-env.sh + +```shell +# 指定JDK的安装位置 +export JAVA_HOME=/usr/java/jdk1.8.0_201/ +``` + +#### 2. core-site.xml + +```xml + + + + fs.defaultFS + hdfs://hadoop001:8020 + + + + hadoop.tmp.dir + /home/hadoop/tmp + + + ha.zookeeper.quorum + hadoop001:2181,hadoop002:2181,hadoop002:2181 + + + + ha.zookeeper.session-timeout.ms + 1000 + + +``` + +#### 3. hdfs-site.xml + +```xml + + + + dfs.replication + 3 + + + dfs.namenode.name.dir + /home/hadoop/namenode/data + + + dfs.datanode.data.dir + /home/hadoop/datanode/data + + + + dfs.nameservices + mycluster + + + + dfs.ha.namenodes.mycluster + nn1,nn2 + + + + dfs.namenode.rpc-address.mycluster.nn1 + hadoop001:8020 + + + + dfs.namenode.rpc-address.mycluster.nn2 + hadoop002:8020 + + + + dfs.namenode.http-address.mycluster.nn1 + hadoop001:50070 + + + + dfs.namenode.http-address.mycluster.nn2 + hadoop002:50070 + + + + dfs.namenode.shared.edits.dir + qjournal://hadoop001:8485;hadoop002:8485;hadoop003:8485/mycluster + + + + dfs.journalnode.edits.dir + /home/hadoop/journalnode/data + + + + dfs.ha.fencing.methods + sshfence + + + + dfs.ha.fencing.ssh.private-key-files + /root/.ssh/id_rsa + + + + dfs.ha.fencing.ssh.connect-timeout + 30000 + + + + dfs.client.failover.proxy.provider.mycluster + org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider + + + + dfs.ha.automatic-failover.enabled + true + + +``` + +#### 4. yarn-site.xml + +```xml + + + + yarn.nodemanager.aux-services + mapreduce_shuffle + + + + yarn.log-aggregation-enable + true + + + + yarn.log-aggregation.retain-seconds + 86400 + + + + + yarn.resourcemanager.ha.enabled + true + + + + + yarn.resourcemanager.cluster-id + my-yarn-cluster + + + + + yarn.resourcemanager.ha.rm-ids + rm1,rm2 + + + + + yarn.resourcemanager.hostname.rm1 + hadoop002 + + + + yarn.resourcemanager.hostname.rm2 + hadoop003 + + + + yarn.resourcemanager.webapp.address.rm1 + hadoop002:8088 + + + yarn.resourcemanager.webapp.address.rm2 + hadoop003:8088 + + + + + yarn.resourcemanager.zk-address + hadoop001:2181,hadoop002:2181,hadoop003:2181 + + + + + yarn.resourcemanager.recovery.enabled + true + + + + + yarn.resourcemanager.store.class + org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore + + +``` + +#### 5. mapred-site.xml + +```xml + + + + mapreduce.framework.name + yarn + + +``` + +#### 5. slaves + +配置所有从属节点的主机名或IP地址，每行一个。所有从属节点上的`DataNode`服务和`NodeManager`服务都会被启动。 + +```properties +hadoop001 +hadoop002 +hadoop003 +``` + +### 4.4 分发程序 + +将Hadoop安装包分发到其他两台服务器，分发后建议在这两台服务器上也配置一下Hadoop的环境变量。 + +```shell +# 将安装包分发到hadoop002 +scp -r /usr/app/hadoop-2.6.0-cdh5.15.2/ hadoop002:/usr/app/ +# 将安装包分发到hadoop003 +scp -r /usr/app/hadoop-2.6.0-cdh5.15.2/ hadoop003:/usr/app/ +``` + + + +## 五、启动集群 + +### 5.1 启动ZooKeeper + +分别到三台服务器上启动ZooKeeper服务： + +```ssh + zkServer.sh start +``` + +### 5.2 启动Journalnode + +分别到三台服务器的的`${HADOOP_HOME}/sbin`目录下，启动`journalnode`进程： + +```shell +hadoop-daemon.sh start journalnode +``` + +### 5.3 初始化NameNode + +在`hadop001`上执行`NameNode`初始化命令： + +``` +hdfs namenode -format +``` + +执行初始化命令后，需要将`NameNode`元数据目录的内容，复制到其他未格式化的`NameNode`上。元数据存储目录就是我们在`hdfs-site.xml`中使用`dfs.namenode.name.dir`属性指定的目录。这里我们需要将其复制到`hadoop002`上： + +```shell + scp -r /home/hadoop/namenode/data hadoop002:/home/hadoop/namenode/ +``` + +### 5.4 初始化HA状态 + +在任意一台`NameNode`上使用以下命令初始化HA状态。此时程序会在ZooKeeper上创建一个Znode，用于存储自动故障转移系统的相关数据。这里在`hadoop001`或`hadoop002`上执行都可以，但不要两台机器都执行，因为不能创建同名的Znode。 + +```shell +hdfs zkfc -formatZK +``` + +### 5.5 启动HDFS + +进入到`Hadoop001`的`${HADOOP_HOME}/sbin`目录下，启动HDFS。此时`hadoop001`和`hadoop002`上的`NameNode`服务，和三台服务器上的`DataNode`服务都会被启动： + +```shell +start-dfs.sh +``` + +### 5.6 启动YARN + +进入到`Hadoop001`的`${HADOOP_HOME}/sbin`目录下，启动YARN。此时`hadoop002`上的`ResourceManager`服务，和三台服务器上的`NodeManager`服务都会被启动。 + +```SHEll +start-yarn.sh +``` + +需要注意的是，这个时候`hadoop003`上的`ResourceManager`服务通常是没有启动的，需要手动启动： + +```shell +hadoop-daemon.sh start datanode +``` + +## 六、查看集群 + +### 6.1 查看进程 + +成功启动后，每台服务器上的进程应该如下： + +```shell +[root@hadoop001 sbin]# jps +4512 DFSZKFailoverController +3714 JournalNode +4114 NameNode +3668 QuorumPeerMain +5012 DataNode +4639 NodeManager + + +[root@hadoop002 sbin]# jps +4499 ResourceManager +4595 NodeManager +3465 QuorumPeerMain +3705 NameNode +3915 DFSZKFailoverController +5211 DataNode +3533 JournalNode + + +[root@hadoop003 sbin]# jps +3491 JournalNode +3942 NodeManager +4102 ResourceManager +4201 DataNode +3435 QuorumPeerMain +``` + + + +### 6.2 查看Web UI + +HDFS和YARN的端口号分别为`50070`和`8080`，界面应该如下： + +此时hadoop001上的`NameNode`处于可用状态： + +![hadoop高可用集群1](D:\BigData-Notes\pictures\hadoop高可用集群1.png) + +而hadoop002上的`NameNode`则处于备用状态： + +
+ +![hadoop高可用集群1](D:\BigData-Notes\pictures\hadoop高可用集群3.png) + +
+ +hadoop002上的`ResourceManager`处于可用状态： + +
+ +![hadoop高可用集群1](D:\BigData-Notes\pictures\hadoop高可用集群4.png) + +
+ +hadoop003上的`ResourceManager`则处于备用状态： + +
+ +![hadoop高可用集群1](D:\BigData-Notes\pictures\hadoop高可用集群5.png) + +
+ +同时界面上也有`Journal Manager`的相关信息： + +
+ +![hadoop高可用集群1](D:\BigData-Notes\pictures\hadoop高可用集群2.png) + diff --git a/pictures/HDFS-HA-Architecture-Edureka.png b/pictures/HDFS-HA-Architecture-Edureka.png new file mode 100644 index 0000000..9ec4343 Binary files /dev/null and b/pictures/HDFS-HA-Architecture-Edureka.png differ diff --git a/pictures/hadoop高可用集群1.png b/pictures/hadoop高可用集群1.png new file mode 100644 index 0000000..e05b54b Binary files /dev/null and b/pictures/hadoop高可用集群1.png differ diff --git a/pictures/hadoop高可用集群2.png b/pictures/hadoop高可用集群2.png new file mode 100644 index 0000000..d3dfdca Binary files /dev/null and b/pictures/hadoop高可用集群2.png differ diff --git a/pictures/hadoop高可用集群3.png b/pictures/hadoop高可用集群3.png new file mode 100644 index 0000000..7b2d3a9 Binary files /dev/null and b/pictures/hadoop高可用集群3.png differ diff --git a/pictures/hadoop高可用集群4.png b/pictures/hadoop高可用集群4.png new file mode 100644 index 0000000..465f72d Binary files /dev/null and b/pictures/hadoop高可用集群4.png differ diff --git a/pictures/hadoop高可用集群5.png b/pictures/hadoop高可用集群5.png new file mode 100644 index 0000000..da37d29 Binary files /dev/null and b/pictures/hadoop高可用集群5.png differ