1. 程式人生 > >重新編譯並安裝Tez on hive(Tez 0.8.5,Hive 2.3.3)

重新編譯並安裝Tez on hive(Tez 0.8.5,Hive 2.3.3)

author by Fei Joe (Thupdi 技術創新中心)

一,

maven3.3.9

protoc的版本是2.5.0

二,安裝nodejs

yum -y install gcc make gcc-c++ openssl

wget http://nodejs.org/dist/v0.12.2/node-v0.12.2.tar.gz 

解壓後./configure 

如果報錯 沒有 python: ImportError: No module named bz2

解決方法:

yum install bzip2-devel

然後需要重新編譯一下python,之後再import bz2就不會提示錯誤了。

  1. cd Python-2.7.10

  2. ./configure --prefix=/usr/local/python2

  3. make && make install

make && make install    最後檢視是否成功node -v

三,安裝npm

curl -L https://www.npmjs.com/install.sh | sh

如果執行node -v和npm -v能看到版本就說明安裝成功了

四,安裝bower

npm -g install bower

安裝後執行下面bower install命令如果提示瞭如下命令說明bower安裝成功。中這個錯誤提示的是root使用者不能執行bower install 命令

如果需要安裝 需要加 --allow-root命令

進入tez_ui 修改pom.xml

<arguments>
                <argument>node_modules/bower/bin/bower</argument>
                <argument>install</argument>
                <argument>--allow-root</argument> 
                <argument>--remove-unnecessary-resolutions=false</argument>
              </arguments>

五,安裝git

yum install git

 mvn clean package  -Dhadoop.version=2.7.3  -DskipTests=true -Dmaven.javadoc.skip=true

如果tez_ui2報錯 去pom.xml 註釋掉<!-- <module>tez-ui2</module>-->

如下:

編譯成功後:

Now, in the target directory of tez-dist you can see two tar balls are created:

tez-0.8.5.tar.gz

tez-0.8.5-minimal.tar.gz

Next, create a new folder with the name tez in your desired location. We are creating this folder in the home directory and then moving the two tar balls into the folder.

Step 17: Create the directory using the command mkdir tez

hadoop fs -mkdir /tez

Step 17: Copy the tez-0.8.5.tar.gz tar ball into the directory using the below command:

hadoop fs -put tez-0.8.5.tar.gz /tez/

You can check whether the file has copied to HDFS or not by using the below command:

hadoop fs -ls /tez/

Step 18: Now, within the tez folder, which is in your local file system, create two folders with name conf and tez using the below commands:

mkdir conf
mkdir tez

Step 19: Extract the contents of the file tez-0.8.1-alpha-minimal.tar.gz to the newly created tez folder.

Step 20: Now, create a file tez-site.xml in your newly created conf directory and add the below configurations in the tez-site.xml file:

<configuration>
<property>
<name>tez.lib.uris</name>
<value>hdfs://localhost:8020/tez/tez-0.8.5.tar.gz</value>
</property>
</configuration>

修改hive/conf/hive-site.xml 

Now close and save the file. Your Tez is ready! You can now integrate it with Hadoop by adding the following properties in your hadoop-env.sh file of Hadoop.

export TEZ_CONF_DIR=/mnt/moduled/conf/
export TEZ_JARS=/mnt/moduled/tez/tez/
export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:${HADOOP_CLASSPATH}:${MAPREDUCE_LIBS}