CDH 5.7.0離線安裝指南
系統環境
硬體環境:window下VMware 11虛擬機器
作業系統:紅帽6.5伺服器版本
Cloudera Manager:5.7
CDH:CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel
資料庫:Mysql 5.6
參考說明
http://www.cloudera.com/documentation/enterprise/latest/topics/installation_installation.html下載
Java:http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
MySQL:http://dev.mysql.com/downloads/mysql/
CM5:http://archive.cloudera.com/cm5/
CDH5:http://archive.cloudera.com/cdh5/
JDBC:http://dev.mysql.com/downloads/connector/j/
配置環境
1.ssh 2.java 3.mysql 4.ntp 5.network 6.防火牆及SELUNUX 以上略。。安裝
Cloudera Manager 5
1. 首先解壓CM5的cm-5.7.0和cloudera到Opt目錄下。$ sudo tar xzf cloudera-manager*.tar.gz -C /opt/
2. 配置Server地址
$ vi /opt/cm-5.7.0/etc/cloudera-scm-agent/config.ini
server_host=hadoop001
server_port=7183
3. MySql Driver
$ tar zxvf mysql-connector-java-5.1.37.tar.gz $ sudo mkdir -p /usr/share/java/ $ sudo cp mysql-connector-java-5.1.37/mysql-connector-java-5.1.37-bin.jar /usr/share/java/mysql-connector-java.jar4. 因為CM5裡包含了SERVER和Agent所以每個節點上都需要安裝它。
$ sudo scp -r /opt/* agent_host:/opt/
5. Cloudera Manger和服務需要Linux帳戶來進行管理。所以必須建立一個使用者帳戶在每一個節點機器上,因為Cloudera預設使用這個帳戶來管理Cloudera叢集。
$ sudo useradd --system --home=/opt/cm-5.7.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
6. 建立Cloudera Manager Server本地檔案儲存地址
$ sudo mkdir /var/log/cloudera-scm-server $ sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-server
6-1. (可以略過)預設配置使用的路徑
- /var/log/cloudera-scm-headlamp
- /var/log/cloudera-scm-firehose
- /var/log/cloudera-scm-alertpublisher
- /var/log/cloudera-scm-eventserver
- /var/lib/cloudera-scm-headlamp
- /var/lib/cloudera-scm-firehose
- /var/lib/cloudera-scm-alertpublisher
- /var/lib/cloudera-scm-eventserver
- /var/lib/cloudera-scm-server
$ sudo chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-headlamp
-
Use alternate directories:
-
If the directories you plan to use do not exist, create them. For example, to create /var/cm_logs/cloudera-scm-headlamp for use by the cloudera-scm user,
you can use the following commands:
mkdir /var/cm_logs/cloudera-scm-headlamp chown cloudera-scm /var/cm_logs/cloudera-scm-headlamp
- Connect to the Cloudera Manager Admin Console.
- Select
- Select .
- Click the Configuration tab.
- Enter a term in the Search field to find the settings to be changed. For example, you might enter /var or directory.
-
Update each value with the new locations for Cloudera Manager to use.Note: The configuration property for the Cloudera
Manager Server Local Data Storage Directory (default value is: /var/lib/cloudera-scm-server) is located on a different page:
- Select .
- Type directory in the Search box.
- Enter the directory path in the Cloudera Manager Server Local Data Storage Directory property.
- Click Save Changes to commit the changes.
-
If the directories you plan to use do not exist, create them. For example, to create /var/cm_logs/cloudera-scm-headlamp for use by the cloudera-scm user,
you can use the following commands:
7. 初始化資料庫及使用者
/opt/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql cm -uroot -p123456 --scm-host hadoop001 root 123456
7-1. (可以略過)初始化資料庫設定
http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_installing_configuring_dbs.html#cmig_topic_5
vi /usr/my.cnf
[mysqld] transaction-isolation = READ-COMMITTED # Disabling symbolic-links is recommended to prevent assorted security risks; # to do so, uncomment this line: # symbolic-links = 0 key_buffer = 16M key_buffer_size = 32M max_allowed_packet = 32M thread_stack = 256K thread_cache_size = 64 query_cache_limit = 8M query_cache_size = 64M query_cache_type = 1 max_connections = 550 #expire_logs_days = 10 #max_binlog_size = 100M #log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system #and chown the specified folder to the mysql user. log_bin=/var/lib/mysql/mysql_binary_log # For MySQL version 5.1.8 or later. Comment out binlog_format for older versions. binlog_format = mixed read_buffer_size = 2M read_rnd_buffer_size = 16M sort_buffer_size = 8M join_buffer_size = 8M # InnoDB settings innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 64M innodb_buffer_pool_size = 4G innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid sql_mode=STRICT_ALL_TABLES
7-2. (可以略過)建立自定義資料庫及關聯使用者
CREATE USER 'amon'@'*' IDENTIFIED BY 'amon';
CREATE USER 'rman'@'*' IDENTIFIED BY 'rman';
CREATE USER 'hive'@'*' IDENTIFIED BY 'hive';
CREATE USER 'sentry'@'*' IDENTIFIED BY 'sentry';
CREATE USER 'nav'@'*' IDENTIFIED BY 'nav';
CREATE USER 'navms'@'*' IDENTIFIED BY 'navms';
CREATE USER 'hue'@'*' IDENTIFIED BY 'hue';
CREATE USER 'oozie'@'*' IDENTIFIED BY 'oozie';
create database amon DEFAULT CHARACTER SET utf8;
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
create database rman DEFAULT CHARACTER SET utf8;
grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
create database hive DEFAULT CHARACTER SET utf8;
grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';
create database sentry DEFAULT CHARACTER SET utf8;
grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';
create database nav DEFAULT CHARACTER SET utf8;
grant all on nav.* TO 'nav'@'%' IDENTIFIED BY 'nav';
create database navms DEFAULT CHARACTER SET utf8;
grant all on navms.* TO 'navms'@'%' IDENTIFIED BY 'navms';
create database hue DEFAULT CHARACTER SET utf8;
grant all on hue.* to 'hue'@'%' identified by 'hue';
create database oozie DEFAULT CHARACTER SET utf8;
grant all on oozie.* to 'oozie'@'%' identified by 'oozie';
FLUSH PRIVILEGES;
8. 複製CDH安裝包到每個節點
$ sudo chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo $ cp CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel /opt/cloudera/parcel-repo
9. 啟動主節點Clouder Manager Server
$ /opt/cm-5.7.0/etc/init.d/cloudera-scm-server start
10. 啟動每個節點的Clouder Manager Agent
$ mkdir /opt/cm-5.7.0/run/cloudera-scm-agent $ /opt/cm-5.7.0/etc/init.d/cloudera-scm-agent start
設定自動啟動Agent
$ cp /opt/cm-5.7.0/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent $ chkconfig cloudera-scm-agent on
Cloudera Distribution Hadoop 5.7.0
1. 登入localhost:7180
2. 有3個版本可以使用就不解釋了
3. 檢視到可以用到的元件
4. 選擇需要安裝的節點
5. 安裝CDH 5.7.0並根據需求安裝相應元件
6. 分配節點任務