Java API獲取HDFS的檔案資訊

1).獲取檔案屬性

環境：Windows Java API 函式：mkdir、FileStatus[]、listStatus、isDirectory

@Test
public void test1() throws Exception{
    //配置HDFS主節點:NameNode
    Configuration conf = new Configuration();
    conf.set("fs.defaultFS","hdfs://192.168.142.111:9000");
    //得到HDFS客戶端
    FileSystem fs = FileSystem.get(conf);
    //建立資料夾  /tools/foldder
    fs.mkdir(new Path("/tools/folder"));
    
    //獲取檔案狀態
    FileStatus[] list = fs.listStatus(new Path("/tools"));
    for(FileStatus status:list){
        //獲取檔案屬性
        System.out.println(status.isDirectory?"目錄":"檔案")；
    }
}
 

執行結果

目錄

2).獲取資料塊資訊

環境：Windows Java API 函式：getFileStatus、getFileBlockLocations、getHosts、getNames

@Test
public void test2()throws Exception{
	Configuration conf = new Configuration();
	conf.set("fs.defaultFS", "hdfs://192.168.142.111:9000");
	//獲取某個檔案的資料塊資訊
	FileSystem fs = FileSystem.get(conf);
	FileStatus fileStatus = fs.getFileStatus(new Path("/tools/up1.mp4"));
	BlockLocation[] list = fs.getFileBlockLocations(fileStatus,0,fileStatus.getLen());
	for(BlockLocation bl:list){
	    System.out.println(Arrays.toString(bl.getHosts));
	    System.out.println(Arrays.toString(bl.getNames));
	}
 

Java API獲取HDFS的檔案上傳/下載

1.0檔案上傳原理1.1).上傳檔案至HDFS(Java IO)

環境：Windows Java API 函式：InputStream、FileInputStream、OutputStream、create

@Test
public void upload1() throws Exception {
//1.配置HDFS
    Configuration conf = new Configuration();
    conf.set("fs.defaultFS","hdfs://192.168.142.111:9000");
//2.獲取客戶端
    FileSystem fs = FileSystem.get(conf);
//3.開啟輸入流
    InputStream in = new FileInputStream("E:\\java Hadoop\\upfile.mp4");
//4.建立輸出端
    OutputStream out = fs.create(new Path("/tools/up1.mp4"));
//5.建立快取區
    byte[] buffer = new byte[1024];
    int len= 0;
//6.上傳檔案
    while((len = read(buffer))>0){
        out.write(out,0,len);
    }
    flush();
//7.關閉流
    in.close();
    out.close();
}
 

1.2).上傳檔案至HDFS(內建函式)

環境：Windows Java API 函式：InputStream、FileInputStream、OutputStream、create、IOUtils、copyBytes

@Test
public void upload2()throws Exception{
//1.配置HDFS的主節點：NameNode
    Configuration conf = new Configuration();
    conf.set("fs.defaultFS","hdfs://192.168.142.111:9000");
//2.獲取客戶端
    FileSystem fs = FileSystem.get(conf);
//3.開啟輸入流
    InputStream input = new FileInputStream("E:\\java Hadoop\\upload1.mp4");
//4.建立輸出流
    OutputStream output = fs.create(new Path("/tools/up2.mp4"));
//5.copyByte
    IOUtils.copyBytes(input,output,1024);
    
}

2.1).下載檔案至Windows(Java IO) HDFS檔案下載原理 

環境：Windows Java API 函式：InputStream、FileInputStream、OutputStream、create、IOUtils、copyBytes

@Test
public void download1()throws Exception{
//1.配置HDFS的主節點：NameNode
    Configuration conf = new Configuration();
    conf.set("fs.defaultFS","hdfs://192.168.142.111:9000");
//2.獲取客戶端
    FileSystem fs = FileSystem.get(conf);
//3.開啟輸入流(HDFS)
    InputStream in = fs.open(new Path("/tools/up1.mp4"));
//4.建立輸出流
    OutputStream out = new FileOutput("E:\\java Hadoop\\down1.mp4");
//5.建立快取區
    byte[] buffer = new byte[1024];
    int len = 0;
//6.寫入輸出流
    while((len = read(buffer))>0){
        out.write(buffer,0,len);
    }
//7.關閉流   
    in.close();
    out.close();
}

2.2).下載檔案至Windows(內建函式)

環境：Windows Java API 函式：InputStream、FileInputStream、OutputStream、create、IOUtils、copyBytes

@Test
public void download()throws Expection{
//1.配置HDFS的NameNode
    Configuration conf = new Configuration();
    conf.set("fs.defaultFS","hdfs://192.168.142.111:9000");
//2獲取客戶端
    FileSystem fs = FileSystem.get(conf);
//3.開啟輸入流
    InputStream input = fs.open(new Path("/tools/up1.mp4"));
//4.建立輸出流
    OutputStream output = new FileOutput("E:\\java Hadoop\\down2.mp4");
//5.檔案下載
    IOUtils.copyBytes(input,output,1024);
}