1. 程式人生 > >Centos7下配置Hadoop偽分散式環境

Centos7下配置Hadoop偽分散式環境

Centos 版本:7

Hadoop版本:2.7.4

Java版本:1.8

一、安裝JDK

官網下載jdk 1.8

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

下載jdk-8u151-linux-x64.tar.gz並複製到 /usr/lib目錄下

然後解壓

 

1

 

tar -zxvf jdk-8u151-linux-x64.tar.gz

配置環境變數

 

1

2

3

 

#set java enviroenment

export JAVA_HOME=/usr/lib/jdk1.8.0_151

export PATH=$JAVA_HOME/bin:$PATH

使環境變數生效

 

1

 

. /etc/profile

驗證jdk

 

1

2

3

4

 

[[email protected] ~]# java -version

java version "1.8.0_151"

Java(TM) SE Runtime Environment (build 1.8.0_151-b12)

Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

二、配置ssh無密碼登入

yum安裝openssh-server

 

1

 

yum install openssh-server

配置無密碼登入

 

1

2

 

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

驗證ssh

 

1

 

ssh shirukai.novalocal

三、關閉防火牆

a. 檢視防火牆狀態

 

1

 

systemctl status firewalld

b. 檢視開機是否啟動防火牆服務

 

1

 

systemctl is-enabled firewalld

c. 關閉防火牆

 

1

2

 

systemctl stop firewalld

systemctl status firewalld

d. 警用防火牆(系統啟動時不啟動防火牆)

 

1

2

 

systemctl disable firewalld

systemctl is-enabled firewalld

e. 確保setlinux關閉

 

1

 

setenforce 0

四、下載Hadoop2.7.4

在usr目錄下建立hadoop目錄

 

1

2

3

 

cd /usr

mkdir hadoop

cd hadoop

下載hadoop

 

1

 

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.4/hadoop-2.7.4.tar.gz

地址:

http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.4/hadoop-2.7.4.tar.gz

解壓hadoop-2.7.4.tar.gz

 

1

 

tar -zxvf hadoop-2.7.4.tar.gz

在 hadoop目錄下建立目錄

 

1

2

3

4

 

mkdir /usr/hadoop/tmp

mkdir /usr/hadoop/hdfs

mkdir /usr/hadoop/hdfs/data

mkdir /usr/hadoop/hdfs/name

五、配置Hadoop

1.配置系統環境變數

 

1

 

vim /etc/profile

新增內容

 

1

2

3

 

#set hadoop path

export HADOOP_HOME=/usr/hadoop/hadoop-2.7.4

export PATH=$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$PATH

配置hosts

檢視當前的ip

 

1

2

 

ifconfig

//記住當前ip:10.110.13.243

檢視當前主機名

 

1

2

 

[[email protected] ~]# hostname

shirukai.novalocal

修改hosts

/etc/hosts

 

1

 

vim hosts

修改內容

 

1

 

10.110.13.243 shirukai.novalocal

2.配置hadoop-env.sh

/usr/hadoop/hadoop-2.7.4/etc/hadoop

將java的環境變數配置到裡面指定位置(大約在25行)

 

1

2

 

# The java implementation to use.

export JAVA_HOME=/usr/lib/jdk1.8.0_151

3.配置yarn-env.sh

在大約26行位置

 

1

2

 

25 echo "run java in $JAVA_HOME"

26 JAVA_HOME=$JAVA_HOME

4.配置core-site.xml

 

1

2

3

4

5

6

7

8

9

10

11

12

13

 

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://0.0.0.0:9000</value>

<description>HDFS的URI,檔案系統://namenode標識:埠號</description>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/usr/hadoop/tmp</value>

<description>namenode上本地的hadoop臨時資料夾</description>

</property>

</configuration>

5.配置hdfs-site.xml

 

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

 

<configuration>

<property>

<name>dfs.name.dir</name>

<value>/usr/hadoop/hdfs/name</value>

<description>namenode上儲存hdfs名字空間元資料 </description>

</property>

<property>

<name>dfs.data.dir</name>

<value>/usr/hadoop/hdfs/data</value>

<description>datanode上資料塊的物理儲存位置</description>

</property>

<property>

<name>dfs.replication</name>

<value>1</value>

<description>副本個數,配置預設是3,應小於datanode機器數量</description>

</property>

</configuration>

6.配置mapred-site.xml

 

1

2

3

4

5

6

 

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

</configuration>

7.配置yarn-site.xml

 

1

2

3

4

5

6

7

8

9

10

11

 

<configuration>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address</name>

<value>0.0.0.0:8088</value>

</property>

</configuration>

8.修改slaves

 

1

 

shirukai.novalocal

六、啟動服務

1.格式化namenode

1
bin/hdfs namenode -format

2.啟動namenode和datanode守護程序

1
sbin/start-dfs.sh

3.啟動Resourcemanager和NodeManager守護程序

1
sbin/start-yarn.sh

4.驗證

 

1

2

3

4

5

6

7

 

[[email protected] logs]# jps

3584 SecondaryNameNode

3411 DataNode

3285 NameNode

3753 ResourceManager

5310 Jps

3855 NodeManager

瀏覽器輸入

http://10.110.13.243:50070

http://10.110.13.243:8099