大資料平臺中遇到的實際問題，整理了一下，使用CDH5.8版本，包括Hadoop、Spark、Hive、kafka、Hbase、Phoenix、Impala、Sqoop、CDH等問題，初步整理下最近遇到的問題，不定期更新。

啟動nodemanager失敗

2016-09-07 14:28:46,434 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.BindException: Problem binding to [0.0.0.0:8040] 
java.net.BindException: Address already in use; For more details see:  http://wiki.apache.org/hadoop/BindException
        at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:139)
        at org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
        at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.createServer(ResourceLocalizationService.java:278)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceStart(ResourceLocalizationService.java:258)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart
(ContainerManagerImpl.java:293)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:199)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:339)
 

問題原因：8040埠被佔用，netstat -apn | grep 8040後發現有nodemanager程序再執行。
解決辦法：kill正在執行的nodemanager程序，重新啟動nodemanager。成功。

Spark on yarn部署遇到的問題
命令列啟動spark-shell --master yarn

org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:124)
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:64)
	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:530)
	at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017)
	at $iwC$$iwC.<init>(<console>:15)
	at $iwC.<init>(<console>:24)
	at <init>(<console>:26)
	at .<init>(<console>:30)
	at .<clinit>(<console>)
	at .<init>(<console>:7)
	at .<clinit>(<console>)
	at $print(<console>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 

applicationMaster日誌只有：
/bin/bash: {{JAVA_HOME}}/bin/java: No such file or directory
問題原因：Spark使用的Hadoop版本與與線上Hadoop版本不一致
解決辦法：使用線上Hadoop版本對應的Spark，解決問題

啟動spark sql server

16/09/08 18:45:26 INFO service.AbstractService: Service:HiveServer2 is started.
16/09/08 18:45:26 INFO thriftserver.HiveThriftServer2: HiveThriftServer2 started
16/09/08 18:45:26 ERROR thrift.ThriftCLIService: Error starting HiveServer2: could not start ThriftBinaryCLIService
org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address /172.16.0.43:10002.
        at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:109)
        at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:91)
        at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:87)
        at org.apache.hive.service.auth.HiveAuthFactory.getServerSocket(HiveAuthFactory.java:241)
        at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:66)
        at java.lang.Thread.run(Thread.java:722)
16/09/08 18:45:26 INFO server.HiveServer2: Shutting down HiveServer2
16/09/08 18:45:26 INFO service.AbstractService: Service:ThriftBinaryCLIService is stopped.
16/09/08 18:45:26 INFO service.AbstractService: Service:OperationManager is stopped.
16/09/08 18:45:26 INFO service.AbstractService: Service:SessionManager is stopped.
16/09/08 18:45:26 INFO service.AbstractService: Service:CLIService is stopped.
 

問題原因：與Hive中的HiveServer2 程序埠衝突。
解決辦法：修改spark配置中的hive-site.xml中的<property>
  <name>hive.server2.thrift.bind.host</name>
  <value>172.16.0.46</value>
</property> 將地址改到沒有hiveserver服務的節點。

安裝impala
啟動impala-server服務：[[email protected] impala]$ sudo service impala-server status

Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0913 15:12:06.521616  5070 logging.cc:103] stderr will be logged to this file.
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
E0913 15:12:10.286190  5070 impala-server.cc:210] Could not read the HDFS root directory at hdfs://master:50070. Error was:
Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "master/172.16.0.39"; destination host is: "master":50070;
E0913 15:12:10.286229  5070 impala-server.cc:212] Aborting Impala Server startup due to improper configuration

問題原因：是版本問題
解決辦法：安裝Hadoop2.2.0對應版本CDH版本

使用yum install安裝軟體時，總會找原來的jar包
問題原因：是yum快取的原因
解決辦法：使用yum clean all清除快取即可。

部署線上CDH5.8叢集，Cdh安裝中遇到“正在獲取安裝鎖”，安裝時出錯了，後來重新裝，就遇到了這個問題
問題原因：上次安裝時有scm的一些檔案遺留。
解決辦法：進入/tmp 目錄，ls -a檢視，刪除scm_prepare_node.*的檔案，以及.scm_prepare_node.lock檔案。

DNS反向解析錯誤，不能正確解析Cloudera Manager Server主機名

正在檢測 Cloudera Manager Server...
 BEGIN host -t PTR 172.16.0.240 
 240.0.16.172.in-addr.arpa domain name pointer localhost. 
END (0) 
 using localhost as scm server hostname 
BEGIN which python 
 /usr/bin/python 
END (0) 
BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' localhost 7182 
 Traceback (most recent call last): 
 File "<string>", line 1, in <module> 
 File "<string>", line 1, in connect 
socket.error: [Errno 111] Connection refused 
END (1) 
 could not contact scm server at localhost:7182, giving up 
 waiting for rollback request

解決辦法：將連不上的機器 /usr/bin/host 檔案刪掉。mv /usr/bin/host /usr/bin/host.bak

安裝Hadoop時，啟動失敗

java.io.IOException: the pathcomponent: '/data' is world-writable. Its permissions are 0777. Pleasefix this or select a different socket path

問題原因：DataNode的root根目錄許可權設定為0777太高導致不安全

解決辦法：修改為755或者預設許可權

使用sqoop導msyql資料到hbase中，mysql資料庫欄位 tinyint 型別匯入到hbase中成了true/false
原因：這屬於sqoop的一種錯誤轉換
解決辦法：url後面拼上tinyInt1isBit=false

hive執行join操作，會失敗
語句：use hivedb;insert into table report.sitef2_tmp_payment_hbase select concat_ws('_', member0.site_id,substr(trade0.pay_time, 0, 10),substr(trade0.pay_time, 0, 10)), member0.site_id,substr(trade0.pay_time, 0, 10) pay_time,count(*) trade_num, sum(trade0.curry_amount) trade_volume from trade_trade_payments trade0 left join member_member member0 on (trade0.pay_status = '20' and trade0.sup_id = member0.member_id) group by substr(trade0.pay_time, 0, 10),member0.site_id

return code 3 from MapredLocalTask
Execution failed with exit status: 3
Obtaining error information
Task failed!
Task ID:
  Stage-4
Logs:
/tmp/root/hive.log
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

問題原因：join基於hbase的對映表時，join進行了轉換
解決辦法：set hive.auto.convert.join = false;

hive建表時插入註釋時，中文亂碼。

解決辦法：修改hive元資料表的編碼設定
//修改欄位註釋字符集
alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8;
alter table TABLE_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;
//修改分割槽註釋字符集
alter table PARTITION_KEYS modify column PKEY_COMMENT varchar(4000) character set utf8;
//修改表註釋字符集

CDH叢集總是使用交換區記憶體，有時候impala會使用交換區記憶體比較慢：
解決辦法：設定交換區使用
http://blog.csdn.net/huaishu/article/details/8762957
[[email protected] ~]# sysctl vm.swappiness=10
vm.swappiness = 10
[[email protected] ~]# cat /proc/sys/vm/swappiness
10
這裡我們的修改已經生效，但是如果我們重啟了系統，又會變成60.
--永久修改：
在/etc/sysctl.conf 檔案裡新增如下引數：
vm.swappiness=10
或者：
[[email protected] ~]# echo 'vm.swappiness=10' >>/etc/sysctl.conf

安裝kafka，直接安裝官方地址的kafka（https://archive.cloudera.com/kafka/parcels/latest/），下載，啟用。
安裝好之後啟動失敗。
問題原因：broker_max_heap_size配置太小。這個安裝好之後，cdh配置的大小為256MB，其實預設是1G的。
解決辦法：將broker_max_heap_size調大，我改為了512MB，啟動的成功。根據具體情況可繼續調大該引數。

CDH安裝Phoenix，啟動Phoenix失敗
Apache Phoenix unable to connect to HBase
 ./sqlline.py master1:2181/kafka啟動報錯

6/10/24 18:42:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Error: org.apache.hadoop.hbase.DoNotRetryIOException: Class org.apache.phoenix.coprocessor.MetaDataRegionObserver cannot be loaded Set hbase.table.sanity.checks to false at conf or table descriptor if you want to bypass sanity checks
	at org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1707)
	at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1568)
	at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1497)
	at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:468)
	at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55682)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745) (state=08000,code=101)
org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException: Class org.apache.phoenix.coprocessor.MetaDataRegionObserver cannot be loaded Set hbase.table.sanity.checks to false at conf or table descriptor if you want to bypass sanity checks

原因：安裝Phoenix時，只把Phoenix的包放到hbase的lib中了，但是沒有重啟hbase，hbase沒有載入Phoenix的相關class
解決辦法：重啟hbase叢集

連線遠端kafka時，使用ip地址連線報錯：
命令：kafka-console-consumer --zookeeper 172.16.0.242:2181/kafka --topic wk_wordlibrary_title --from-beginning

2016-10-26 14:39:57,949] WARN Fetching topic metadata with correlation id 12 for topics [Set(wk_wordlibrary_title)] from broker [BrokerEndPoint(92,node1,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
	at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
	at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:76)
	at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:75)
	at kafka.producer.SyncProducer.send(SyncProducer.scala:120)
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
	at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2016-10-26 14:39:57,949] WARN [console-consumer-89764_t2-1477463994654-bcfb2442-leader-finder-thread], Failed to find leader for Set([wk_wordlibrary_title,0]) (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)

原因：在client使用的元資訊是機器名而不是IP。客戶端無法解析這個機器名所以出現了前面的異常。 
解決辦法：1.在客戶端主機配置遠端kafka主機-ip對映
2.也可以在server.properties中配置advertised.host.name（broker ip值）

其他部門使用hdfs儲存，佔用儲存空間較大，將磁碟沾滿。（副本數為3）

解決辦法：先將hdfs副本數配置設為1，然後使用hdfs dfs -setrep -R 1  /tmp1/logs命令將對應目錄的檔案副本數更新為1，騰出空間，然後當再寫hdfs檔案時，設定dfs.replication=1(Hadoop dfs -D dfs.replication=1 -put 70M logs/2)---參考連結：http://blog.csdn.net/lskyne/article/details/8898666
hbase regionserver（master1）啟動失敗,報錯日誌

 ABORTING region server master1,60020,1478833728358: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server master1,60020,1478833728358 has been rejected; Reported time is too far out of sync with master.  Time difference of 36240ms > max allowed of 30000ms
	at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:401)
	at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:267)
	at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:366)
	at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745) 

原因：master1時鐘不同步
解決辦法：同步master1時鐘(ntpdate master1),啟動成功

spark sql server（thriftserver.HiveThriftServer2）執行任務時會停掉

報錯：ERROR executor.CoarseGrainedExecutorBackend: Driver 172.16.0.241:43286 disassociated! Shutting down.
	INFO util.ShutdownHookManager: Shutdown hook called
	INFO util.ShutdownHookManager: Deleting directory /data/cdh/yarn/nm/usercache/root/appcache/application_1478760707461_0459/spark-ab6b5ac2-0d5c-4e10-90a1-75b948596b7c
	INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
	ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM

原因：從資訊上看來不知道是什麼問題，但是歸根結底還是記憶體的問題。
解決辦法：一是，如上面所說，加大excutor-memory的值，減少executor-cores的數量，問題可以解決。二是，加大executor.overhead的值，但是這樣其實並沒有解決掉根本的問題。所以如果叢集的資源是支援的話，就用1的辦法吧。
　　另外，這個錯誤也出現在partitionBy(new HashPartition(partiton-num))時，如果partiton-num太大或者太小的時候會報這種錯誤，說白了也是記憶體的原因，不過這個時候增加記憶體和overhead沒有什麼用，得去調整這個partiton-num的值。