2. 程式人生 > 其它 >【Hadoop報錯】The directory item limit of /tmp/hadoop-yarn/staging/history/done_intermediate/hdfs is exceeded: limit=1048576 items=1048576

【Hadoop報錯】The directory item limit of /tmp/hadoop-yarn/staging/history/done_intermediate/hdfs is exceeded: limit=1048576 items=1048576

阿新 • • 發佈：2021-12-01

問題描述：排程系統執行hive任務失敗，一直執行失敗，報錯如下：
java.io.IOException: java.net.ConnectException: Call From #HostName/#IP to #HostName:10020 failed on connection exception: java.net.ConnectException: 拒絕連線; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

Caused by: java.net.ConnectException: Call From #HostName/#IP to #HostName:10020 failed on connection exception: java.net.ConnectException: 拒絕連線; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

CONSOLE# Ended Job = job_1638255473937_0568 with exception 'java.io.IOException(java.net.ConnectException: Call From #HostName/#IP to #HostName:10020 failed on connection exception: java.net.ConnectException: 拒絕連線; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused)

CONSOLE# FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. java.net.ConnectException: Call From #HostName/#IP to #HostName:10020 failed on connection exception: java.net.ConnectException: 拒絕連線; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

這個資訊還看不出具體的問題所在，檢視伺服器上所有的日誌,也沒有看出問題，最後檢視yarn的日誌，看出問題所在。
根據排程系統，獲取到ApplicationId：application_1638255473937_0568 ， 然後從hdfs上檢視對應的日誌資訊。

關鍵報錯資訊：
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException): The directory item limit of /tmp/hadoop-yarn/staging/history/done_intermediate/hdfs is exceeded: limit=1048576 items=1048576

相關推薦

【Hadoop報錯】The directory item limit of /tmp/hadoop-yarn/staging/history/done_intermediate/hdfs is exceeded: limit=1048576 items=1048576

問題描述：排程系統執行hive任務失敗，一直執行失敗，報錯如下： java.io.IOException: java.net.ConnectException: Call From #HostName/#IP to #HostName:10020 failed on connection exception: java.net.Connec

【Python報錯】ValueError: If using all scalar values, you must pass an index

問題： 1、ValueError: If using all scalar values, you must pass an index。意思是：如果使用所有標量值，則必須傳遞索引

【kubeadm初始化報錯】failed to run Kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

復現場景 環境 系統：Centos7 kubernetes：1.18.5 docker：19.03.9 復現步驟 1、通過 yum 或 rpm 安裝 kubelet kubectl kubeadm，並 systemctl enable --now kubelet

【連線SAP HANA報錯】com.sap.db.jdbc.exceptions.JDBCDriverException: SAP DBTech JDBC: [258]: insufficient privilege: Detailed info for this error can be found with guid

簡記 連線SAP HANA資料庫時報瞭如下錯誤： ERROR org.springframework.boot.SpringApplication - Application run failed

【報錯】flask-socketio failed: Error during WebSocket handshake: Unexpected response code: 500

今天遇到個很奇怪的問題，flask-socketio本地跑的時候，沒有報錯，但是一旦部署到線上，就報了 failed: Error during WebSocket handshake: Unexpected response code: 500 的錯誤。後來檢視伺服器日誌：出

【連線SAP HANA報錯】com.sap.db.jdbc.exceptions.JDBCDriverException: SAP DBTech JDBC: [258]: insufficient p...

技術標籤：問題解決資料庫javamysqlpythonlinux 簡記 連線SAP HANA資料庫時報瞭如下錯誤：

【Python報錯解決方案】TypeError: ‘float‘ object is not callable

技術標籤：Python報錯解決方案 最近又開始寫程式碼了，今天學習神經網路的BP時候發現了一個報錯並且在網站上看了，大部分的解釋都很簡單但是有很難懂，但是人家的方法也是可以用的。其實這個錯誤非常簡單，但如果

【報錯】ORA-02096: specified initialization parameter is not modifiable with this option

　　雖然初始化引數可以修改，但是不能使用指定的命令去修改 SQL> show parameter inmemory_size;

【求助-pytorch執行報錯】CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`

問題如下： 1、利用pytorch復現googlenet網路 2、定義訓練過程函式中 def train(epoch):

Struts報錯：The Struts dispatcher cannot be found.

報錯 錯誤資訊為： The Struts dispatcher cannot be found. This is usually caused by using Struts tags without the associated filter. Struts tags are only usable when the request has passed thr

轉載（總結）Oracle啟動監聽報錯：The listener supports no services解決

PS：前幾天裝了一臺oracle11g 當時的hosts檔案如下 127.0.0.1localhost192.168.153.10oraserver

eclipse執行tomcat報錯：the specified jre does not exists

1.確認是否安裝Java成功 在cmd執行視窗輸入 java在cmd執行視窗輸入 java -version 2.檢視eclipse配置jre路徑是否正確

mmdetection訓練報錯ERROR - The testing results of the whole dataset is empty.

該問題困擾多天，旋轉迷迷糊糊算解決了，看了很多github上的討論，講下排查思路。 1.檢查輸入資料的annotations是否準確，包括但不限於category_id，classes數量是否準確等，在config中的classes是否和_base_中

gulp報錯：The following tasks did not complete: myFirstGulp Did you forget to signal async completion?

技術標籤：問題解決node.js 問題截圖： 程式碼如下： const gulp = require(\'gulp\'); // const htmlmin = require(\'gulp-htmlmin\');

tomcat 啟動報錯 registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister

技術標籤：Webtomcat伺服器 在幫朋友改bug時出現下面錯誤： org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [] registered the JDBC driver [com.alibaba.druid.pr

【2021-03-17】The entity type ‘Video‘ is part of a hierarchy, but does not have a discriminator value

問題 初始化資料庫失敗，錯誤部分原始碼如下： protected override void OnModelCreating(ModelBuilder modelBuilder)

QT外掛vs報錯。 The system cannot find the path specified. 1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V120\Microsoft.CppCommon.targets(170,5): error MSB6006: “cmd.exe”已退出，程式碼為 3

由於裝有多個版本因為空間不夠解除安裝了 又裝了新的版本。好久不碰又被坑了一下。

python 關於request模組post方法上傳檔案報錯"the request was rejected because no multipart boundary was found"

python-關於request模組post方法上傳檔案報錯\"the request was rejected because no multipart boundary was found\"

【並查集】The Suspects POJ-1611

打算嘗試一種新寫法 我願叫它“蒟蒻的level++” 寫下我的心路歷程 Description Severe acute respiratory syndrome (SARS), an atypical pneumonia of unknown aetiology, was recognized as a global threat in

執行報錯：The activity must be exported or contain an intent-filter

在測試一個activity時出現了以下錯誤： (1)這裡一般是因為此activity不是Launcher Activity，也沒有設定為 android:exported=\"true\"