1. 程式人生 > >hadoop高可用HA的搭建

hadoop高可用HA的搭建

hadoop高可用HA的搭建:

8485:journal node的預設埠號

思路:搭建zookeeper,在hadoop啟動之前zookeeper就已經啟動了。

免祕鑰:就是讓zookeeper能夠在各臺叢集機器之間進行自由的互動。

1、上傳解壓zookeeper

zookeeper在使用的時候,

一種是主從模式

無主模型

在conf下修改檔案:cp zoo.cfg

進入zoo.cfg檔案:

建立相應的文資料夾:mkdir -p /var/sxt/hadoop/zk

將相應的ID追加到相應的檔案中:

echo 1 > /var/sxt/hadoop/zk/myid

依次在node12,node13,node14上邊去執行

啟動:zkServer.sh start

檢視狀態:zkServer.sh status

進入檔案的新用法:

[[email protected] ~]# cd $HADOOP_HOME
[[email protected] hadoop-2.6.5]# cd 
[[email protected] ~]# cd $JAVA_HOME
[[email protected] jdk1.7.0_67]# cd 
[[email protected] ~]#

以上,zookeeper就已經配置好了

2、搭建HA模式:

配置兩個檔案:

檔案的內容:

hdfs-site.xml

<configuration>
	<property>
		<name>dfs.replication</name>
		<value>2</value>
	</property>
<property>
  <name>dfs.nameservices</name>
  <value>mycluster</value>
</property>
<property>
  <name>dfs.ha.namenodes.mycluster</name>
  <value>nn1,nn2</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.mycluster.nn1</name>
  <value>node11:8020</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.mycluster.nn2</name>
  <value>node12:8020</value>
</property>
<property>
  <name>dfs.namenode.http-address.mycluster.nn1</name>
  <value>node11:50070</value>
</property>
<property>
  <name>dfs.namenode.http-address.mycluster.nn2</name>
  <value>node12:50070</value>
</property>


<property>
  <name>dfs.namenode.shared.edits.dir</name>
  <value>qjournal://node11:8485;node12:8485;node13:8485/mycluster</value>
</property>

<property>
  <name>dfs.journalnode.edits.dir</name>
  <value>/var/sxt/hadoop/ha/jn</value>
</property>


<property>
  <name>dfs.client.failover.proxy.provider.mycluster</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
  <name>dfs.ha.fencing.methods</name>
  <value>sshfence</value>
</property>
<property>
  <name>dfs.ha.fencing.ssh.private-key-files</name>
  <value>/root/.ssh/id_dsa</value>
</property>

<property>
   <name>dfs.ha.automatic-failover.enabled</name>
   <value>true</value>
 </property>

</configuration>

core-site.xml

<configuration>
	<property>	
		<name>fs.defaultFS</name>
		<value>hdfs://mycluster</value>
	</property>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>/var/sxt/hadoop/ha</value>
	</property>

<property>
   <name>ha.zookeeper.quorum</name>
   <value>node12:2181,node13:2181,node14:2181</value>
 </property>

</configuration>

然後將這兩個檔案分發給其他節點:

[[email protected] hadoop]# scp core-site.xml hdfs-site.xml node14:`pwd`

3、格式化:

格式化之前必須要先啟動journal node

[[email protected] hadoop]# hadoop-daemon.sh start journalnode

同時要確保zookeeper處於啟動狀態

執行格式化的命令:

[[email protected] hadoop]# hdfs namenode -format

第一臺同步,之前需要qidongnamenode:

[[email protected] hadoop]# hadoop-daemon.sh start namenode

第二臺執行:

[[email protected] sxt]# hdfs namenode -bootstrapStandby

在node14上邊啟動zookeeper的客戶端:

找node11或者node12格式化zookeeper

[[email protected] hadoop]# hdfs zkfc -formatZK

完成之後啟動hdfs:在node11上邊執行:

[[email protected] hadoop]# start-dfs.sh

檢查node14上邊zookeeper的狀態:

單獨啟動:

[[email protected] hadoop]# hadoop-daemon.sh start namenode