Java IO讀寫大檔案的幾種方式及測試
阿新 • • 發佈:2019-01-28
讀取檔案大小:1.45G
第一種,OldIO:
Java程式碼
耗時70.79s
第二種,newIO:
Java程式碼
耗時47.24s
第三種,RandomAccessFile:
Java程式碼
耗時46.65
第四種,MappedByteBuffer:
Java程式碼
耗時:36
前三種讀法對應的資源佔用圖如下:
相對於最後一種記憶體直接對映方式前面的測試其實無意義,基本秒殺。。。。。
對於很大的檔案直接分塊對映時記憶體會不夠,這是因為MappedByteBuffer未被釋放造成的,sun未提供直接回收MappedByteBuffer區域的方法,這個時候有兩種方法解決,第一種比較愚笨的:
Java程式碼
第二種網上找來的,利用反射呼叫clean方法:
Java程式碼
以上兩種方法感覺都彆扭,還有就是可以自己分割成物理檔案再迴圈呼叫,這個也不太美觀。
速度也會減慢好多。
第一種,OldIO:
Java程式碼
- publicstaticvoid oldIOReadFile() throws IOException{
- BufferedReader br = new BufferedReader(new FileReader("G://lily_947.txt"));
- PrintWriter pw = new PrintWriter("G://oldIO.tmp");
- char[] c = newchar[100*1024*1024];
- for(;;){
- if(br.read(c)!=-1){
- pw.print(c);
- }else{
- break;
- }
- }
- pw.close();
- br.close();
- }
public static void oldIOReadFile() throws IOException{ BufferedReader br = new BufferedReader(new FileReader("G://lily_947.txt")); PrintWriter pw = new PrintWriter("G://oldIO.tmp"); char[] c = new char[100*1024*1024]; for(;;){ if(br.read(c)!=-1){ pw.print(c); }else{ break; } } pw.close(); br.close(); }
耗時70.79s
第二種,newIO:
Java程式碼
- publicstaticvoid newIOReadFile() throws IOException{
- FileChannel read = new RandomAccessFile("G://lily_947.txt","r").getChannel();
- FileChannel writer = new RandomAccessFile("G://newIO.tmp","rw").getChannel();
- ByteBuffer bb = ByteBuffer.allocate(200*1024*1024);
- while(read.read(bb)!=-
- bb.flip();
- writer.write(bb);
- bb.clear();
- }
- read.close();
- writer.close();
- }
public static void newIOReadFile() throws IOException{
FileChannel read = new RandomAccessFile("G://lily_947.txt","r").getChannel();
FileChannel writer = new RandomAccessFile("G://newIO.tmp","rw").getChannel();
ByteBuffer bb = ByteBuffer.allocate(200*1024*1024);
while(read.read(bb)!=-1){
bb.flip();
writer.write(bb);
bb.clear();
}
read.close();
writer.close();
}
耗時47.24s
第三種,RandomAccessFile:
Java程式碼
- publicstaticvoid randomReadFile() throws IOException{
- RandomAccessFile read = new RandomAccessFile("G://lily_947.txt","r");
- RandomAccessFile writer = new RandomAccessFile("G://random.tmp","rw");
- byte[] b = newbyte[200*1024*1024];
- while(read.read(b)!=-1){
- writer.write(b);
- }
- writer.close();
- read.close();
- }
public static void randomReadFile() throws IOException{
RandomAccessFile read = new RandomAccessFile("G://lily_947.txt","r");
RandomAccessFile writer = new RandomAccessFile("G://random.tmp","rw");
byte[] b = new byte[200*1024*1024];
while(read.read(b)!=-1){
writer.write(b);
}
writer.close();
read.close();
}
耗時46.65
第四種,MappedByteBuffer:
Java程式碼
- publicstaticvoid mappedBuffer() throws IOException{
- FileChannel read = new FileInputStream("G://lily_947.txt").getChannel();
- FileChannel writer = new RandomAccessFile("G://buffer.tmp","rw").getChannel();
- long i = 0;
- long size = read.size()/30;
- ByteBuffer bb,cc = null;
- while(i<read.size()&&(read.size()-i)>size){
- bb = read.map(FileChannel.MapMode.READ_ONLY, i, size);
- cc = writer.map(FileChannel.MapMode.READ_WRITE, i, size);
- cc.put(bb);
- i+=size;
- bb.clear();
- cc.clear();
- }
- bb = read.map(FileChannel.MapMode.READ_ONLY, i, read.size()-i);
- cc.put(bb);
- bb.clear();
- cc.clear();
- read.close();
- writer.close();
- }
public static void mappedBuffer() throws IOException{
FileChannel read = new FileInputStream("G://lily_947.txt").getChannel();
FileChannel writer = new RandomAccessFile("G://buffer.tmp","rw").getChannel();
long i = 0;
long size = read.size()/30;
ByteBuffer bb,cc = null;
while(i<read.size()&&(read.size()-i)>size){
bb = read.map(FileChannel.MapMode.READ_ONLY, i, size);
cc = writer.map(FileChannel.MapMode.READ_WRITE, i, size);
cc.put(bb);
i+=size;
bb.clear();
cc.clear();
}
bb = read.map(FileChannel.MapMode.READ_ONLY, i, read.size()-i);
cc.put(bb);
bb.clear();
cc.clear();
read.close();
writer.close();
}
耗時:36
前三種讀法對應的資源佔用圖如下:
相對於最後一種記憶體直接對映方式前面的測試其實無意義,基本秒殺。。。。。
對於很大的檔案直接分塊對映時記憶體會不夠,這是因為MappedByteBuffer未被釋放造成的,sun未提供直接回收MappedByteBuffer區域的方法,這個時候有兩種方法解決,第一種比較愚笨的:
Java程式碼
- System.gc();
- System.runFinalization();
- try {
- Thread.sleep(3000);
- } catch (InterruptedException e) {
- e.printStackTrace();
- }
System.gc();
System.runFinalization();
try {
Thread.sleep(3000);
} catch (InterruptedException e) {
e.printStackTrace();
}
第二種網上找來的,利用反射呼叫clean方法:
Java程式碼
- publicstaticvoid unmap(final MappedByteBuffer buffer) {
- if (buffer == null) {
- return;
- }
- AccessController.doPrivileged(new PrivilegedAction<Object>() {
- public Object run() {
- try {
- Method getCleanerMethod = buffer.getClass().getMethod("cleaner", new Class[0]);
- if (getCleanerMethod != null) {
- getCleanerMethod.setAccessible(true);
- Object cleaner = getCleanerMethod.invoke(buffer, new Object[0]);
- Method cleanMethod = cleaner.getClass().getMethod("clean", new Class[0]);
- if (cleanMethod != null) {
- cleanMethod.invoke(cleaner, new Object[0]);
- }
- }
- } catch (Exception e) {
- e.printStackTrace();
- }
- returnnull;
- }
- });
- }
public static void unmap(final MappedByteBuffer buffer) {
if (buffer == null) {
return;
}
AccessController.doPrivileged(new PrivilegedAction<Object>() {
public Object run() {
try {
Method getCleanerMethod = buffer.getClass().getMethod("cleaner", new Class[0]);
if (getCleanerMethod != null) {
getCleanerMethod.setAccessible(true);
Object cleaner = getCleanerMethod.invoke(buffer, new Object[0]);
Method cleanMethod = cleaner.getClass().getMethod("clean", new Class[0]);
if (cleanMethod != null) {
cleanMethod.invoke(cleaner, new Object[0]);
}
}
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
});
}
以上兩種方法感覺都彆扭,還有就是可以自己分割成物理檔案再迴圈呼叫,這個也不太美觀。
速度也會減慢好多。