Centos7下配置Hadoop偽分散式環境
Centos 版本：7
Hadoop版本：2.7.4
Java版本：1.8
一、安裝JDK
官網下載jdk 1.8
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
下載jdk-8u151-linux-x64.tar.gz並複製到 /usr/lib目錄下
然後解壓
| 1 |
tar -zxvf jdk-8u151-linux-x64.tar.gz |
配置環境變數
| 1 2 3 |
#set java enviroenment export JAVA_HOME=/usr/lib/jdk1.8.0_151 export PATH=$JAVA_HOME/bin:$PATH |
使環境變數生效
| 1 |
. /etc/profile |
驗證jdk
| 1 2 3 4 |
[[email protected] ~]# java -version java version "1.8.0_151" Java(TM) SE Runtime Environment (build 1.8.0_151-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode) |
二、配置ssh無密碼登入
yum安裝openssh-server
| 1 |
yum install openssh-server |
配置無密碼登入
| 1 2 |
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys |
驗證ssh
| 1 |
ssh shirukai.novalocal |
三、關閉防火牆
a. 檢視防火牆狀態
| 1 |
systemctl status firewalld |
b. 檢視開機是否啟動防火牆服務
| 1 |
systemctl is-enabled firewalld |
c. 關閉防火牆
| 1 2 |
systemctl stop firewalld systemctl status firewalld |
d. 警用防火牆（系統啟動時不啟動防火牆）
| 1 2 |
systemctl disable firewalld systemctl is-enabled firewalld |
e. 確保setlinux關閉
| 1 |
setenforce 0 |
四、下載Hadoop2.7.4
在usr目錄下建立hadoop目錄
| 1 2 3 |
cd /usr mkdir hadoop cd hadoop |
下載hadoop
| 1 |
wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.4/hadoop-2.7.4.tar.gz |
地址：
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.4/hadoop-2.7.4.tar.gz
解壓hadoop-2.7.4.tar.gz
| 1 |
tar -zxvf hadoop-2.7.4.tar.gz |
在 hadoop目錄下建立目錄
| 1 2 3 4 |
mkdir /usr/hadoop/tmp mkdir /usr/hadoop/hdfs mkdir /usr/hadoop/hdfs/data mkdir /usr/hadoop/hdfs/name |
五、配置Hadoop
1.配置系統環境變數
| 1 |
vim /etc/profile |
新增內容
| 1 2 3 |
#set hadoop path export HADOOP_HOME=/usr/hadoop/hadoop-2.7.4 export PATH=$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$PATH |
配置hosts
檢視當前的ip
| 1 2 |
ifconfig //記住當前ip：10.110.13.243 |
檢視當前主機名
| 1 2 |
[[email protected] ~]# hostname shirukai.novalocal |
修改hosts
/etc/hosts
| 1 |
vim hosts |
修改內容
| 1 |
10.110.13.243 shirukai.novalocal |
2.配置hadoop-env.sh
/usr/hadoop/hadoop-2.7.4/etc/hadoop
將java的環境變數配置到裡面指定位置(大約在25行)
| 1 2 |
# The java implementation to use. export JAVA_HOME=/usr/lib/jdk1.8.0_151 |
3.配置yarn-env.sh
在大約26行位置
| 1 2 |
25 echo "run java in $JAVA_HOME" 26 JAVA_HOME=$JAVA_HOME |
4.配置core-site.xml
| 1 2 3 4 5 6 7 8 9 10 11 12 13 |
<configuration> <property> <name>fs.default.name</name> <value>hdfs://0.0.0.0:9000</value> <description>HDFS的URI，檔案系統://namenode標識:埠號</description> </property> <property> <name>hadoop.tmp.dir</name> <value>/usr/hadoop/tmp</value> <description>namenode上本地的hadoop臨時資料夾</description> </property> </configuration> |
5.配置hdfs-site.xml
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
<configuration> <property> <name>dfs.name.dir</name> <value>/usr/hadoop/hdfs/name</value> <description>namenode上儲存hdfs名字空間元資料 </description> </property> <property> <name>dfs.data.dir</name> <value>/usr/hadoop/hdfs/data</value> <description>datanode上資料塊的物理儲存位置</description> </property> <property> <name>dfs.replication</name> <value>1</value> <description>副本個數，配置預設是3,應小於datanode機器數量</description> </property> </configuration> |
6.配置mapred-site.xml
| 1 2 3 4 5 6 |
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> |
7.配置yarn-site.xml
| 1 2 3 4 5 6 7 8 9 10 11 |
<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>0.0.0.0:8088</value> </property> </configuration> |
8.修改slaves
| 1 |
shirukai.novalocal |
六、啟動服務
1.格式化namenode
1 |
bin/hdfs namenode -format |
2.啟動namenode和datanode守護程序
1 |
sbin/start-dfs.sh |
3.啟動Resourcemanager和NodeManager守護程序
1 |
sbin/start-yarn.sh |
4.驗證
| 1 2 3 4 5 6 7 |
[[email protected] logs]# jps 3584 SecondaryNameNode 3411 DataNode 3285 NameNode 3753 ResourceManager 5310 Jps 3855 NodeManager |
瀏覽器輸入