本篇介紹在centos7中搭建hadoop2.10高可用集群,首先準備6臺機器:2臺nn(namenode);4臺dn(datanode);3臺jns(journalnodes)
IP | hostname | 進程 |
192.168.30.141 | s141 | nn1(namenode),zkfc(DFSZKFailoverController),zk(QuorumPeerMain) |
192.168.30.142 | s142 | dn(datanode), jn(journalnode),zk(QuorumPeerMain) |
192.168.30.143 | s143 | dn(datanode), jn(journalnode),zk(QuorumPeerMain) |
192.168.30.144 | s144 | dn(datanode), jn(journalnode) |
192.168.30.145 | s145 | dn(datanode) |
192.168.30.146 | s146 | nn2(namenode),zkfc(DFSZKFailoverController) |
各個機器 jps進程:
由于本人使用的是vmware虛擬機,所以在配置好一臺機器后,使用克隆,克隆出剩余機器,并修改hostname和IP,這樣每臺機器配置就都統一了每臺機器配置添加hdfs用戶及用戶組,配置jdk環境,安裝hadoop,本次搭建高可用集群在hdfs用戶下,可以參照:centos7搭建hadoop2.10偽分布模式
下面是安裝高可用集群的一些步驟和細節:
1.設置每臺機器的hostname 和 hosts
修改hosts文件,hosts設置有后可以使用hostname訪問機器,這樣比較方便,修改如下:
127.0.0.1 locahost192.168.30.141 s141192.168.30.142 s142192.168.30.143 s143192.168.30.144 s144192.168.30.145 s145192.168.30.146 s146
2.設置ssh無密登錄,由于s141和s146都為namenode,所以要將這兩臺機器無密登錄到所有機器,最好hdfs用戶和root用戶都設置無密登錄
我們將s141設置為nn1,s146設置為nn2,就需要s141、s146能夠通過ssh無密登錄到其他機器,這樣就需要在s141和s146機器hdfs用戶下生成密鑰對,并將s141和s146公鑰發送到其他機器放到~/.ssh/authorized_keys文件中,更確切的說要將公鑰添加的所有機器上(包括自己)
在s141和s146機器上生成密鑰對:
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
將id_rsa.pub文件內容追加到s141-s146機器的/home/hdfs/.ssh/authorized_keys中,現在其他機器暫時沒有authorized_keys文件,我們就將id_rsa.pub更名為authorized_keys即可,如果其他機器已存在authorized_keys文件可以將id_rsa.pub內容追加到該文件后,遠程復制可以使用scp命令:
s141機器公鑰復制到其他機器
scp id_rsa.pub hdfs@s141:/home/hdfs/.ssh/id_rsa_141.pubscp id_rsa.pub hdfs@s142:/home/hdfs/.ssh/id_rsa_141.pubscp id_rsa.pub hdfs@s143:/home/hdfs/.ssh/id_rsa_141.pubscp id_rsa.pub hdfs@s144:/home/hdfs/.ssh/id_rsa_141.pubscp id_rsa.pub hdfs@s145:/home/hdfs/.ssh/id_rsa_141.pubscp id_rsa.pub hdfs@s146:/home/hdfs/.ssh/id_rsa_141.pub
s146機器公鑰復制到其他機器
scp id_rsa.pub hdfs@s141:/home/hdfs/.ssh/id_rsa_146.pubscp id_rsa.pub hdfs@s142:/home/hdfs/.ssh/id_rsa_146.pubscp id_rsa.pub hdfs@s143:/home/hdfs/.ssh/id_rsa_146.pubscp id_rsa.pub hdfs@s144:/home/hdfs/.ssh/id_rsa_146.pubscp id_rsa.pub hdfs@s145:/home/hdfs/.ssh/id_rsa_146.pubscp id_rsa.pub hdfs@s146:/home/hdfs/.ssh/id_rsa_146.pub
新聞熱點
疑難解答