Hadoop第一個測試例項WordCount的執行
首先確保hadoop已經正確安裝、配置以及執行。
拷貝WordCount.java到我們的資料夾，下載的hadoop裡帶有WordCount.java，路徑為：
hadoop-0.20.203.0/src/examples/org/apache/hadoop/examples/WordCount.java
進行拷貝操作
[[email protected]~]$ cp hadoop-0.20.203.0/src/examples/org/apache/hadoop/examples/WordCount.java ~[[email protected]~]$ lsDesktop hadoop-0.20.203.0 WordCount.java
在當前目錄下建立一個用來存放WordCount.class的資料夾
[[email protected]~]$ mkdir classes編譯WordCount.java
[[email protected]~]$ javac -classpath hadoop-0.20.203.0/hadoop-core-0.20.203.0.jar -d classesWordCount.java上面的方法是按照hadoop自帶的doc文件進行編譯的，如果發現報錯，出現如下異常
WordCount.java:53:error: cannot access Options
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
^
class file for org.apache.commons.cli.Optionsnot found
1 error
則按如下進行編譯
[[email protected]calhost~]$ javac -classpath hadoop-0.20.203.0/hadoop-core-0.20.203.0.jar:hadoop-0.20.203.0/lib/commons-cli-1.2.jar -d classes WordCount.java編譯成功，classes下會出現一個org的資料夾
[ org
對編譯好的class進行打包
[[email protected]~]$ jar -cvf wordcount.jar -C classes/ .added manifest
adding: org/(in =0) (out= 0)(stored 0%)
adding:org/apache/(in = 0) (out= 0)(stored 0%)
adding:org/apache/hadoop/(in = 0) (out= 0)(stored 0%)
adding: org/apache/hadoop/examples/(in= 0) (out= 0)(stored 0%)
adding:org/apache/hadoop/examples/WordCount$TokenizerMapper.class(in = 1790) (out=765)(deflated 57%)
adding:org/apache/hadoop/examples/WordCount$IntSumReducer.class(in = 1793) (out=746)(deflated 58%)
adding:org/apache/hadoop/examples/WordCount.class(in = 1911) (out= 996)(deflated 47%)
[[email protected]~]$ lsclasses Desktop hadoop-0.20.203.0 wordcount.jar WordCount.java
到此為止，java檔案的編譯工作已經完成
現在為測試工作準備需要的測試檔案，建立2個檔案file01、file02，內容如下
[[email protected]~]$ lsclasses Desktop file01 file02 hadoop-0.20.203.0 wordcount.jar WordCount.java
[[email protected]~]$ cat file01Hello World Bye World
[[email protected]~]$ cat file02Hello Hadoop Goodbye Hadoop
啟動hadoop，在hadoop中建立input資料夾
[[email protected]~]$ hadoop-0.20.203.0/bin/hadoop dfs -ls
[[email protected]~]$ hadoop-0.20.203.0/bin/hadoop dfs -mkdir input
[[email protected]~]$ hadoop-0.20.203.0/bin/hadoop dfs -lsFound 1 items
drwxr-xr-x - hadoop supergroup 0 2011-11-23 05:20 /user/hadoop/input
把file01、file02上傳input中
[[email protected]~]$ hadoop-0.20.203.0/bin/hadoop fs -put file01 input
[[email protected]~]$ hadoop-0.20.203.0/bin/hadoop fs -put file02 input
[[email protected]~]$ hadoop-0.20.203.0/bin/hadoop fs -ls inputFound 2 items
-rw-r--r-- 1 hadoop supergroup 22 2011-11-23 05:22/user/hadoop/input/file01
-rw-r--r-- 1 hadoop supergroup 28 2011-11-23 05:22/user/hadoop/input/file02
執行及輸出過程如下
[[email protected]~]$ hadoop-0.20.203.0/bin/hadoop jar wordcount.jar org.apache.hadoop.examples.WordCount input output11/11/23 05:25:11INFO input.FileInputFormat: Total input paths to process : 2
11/11/23 05:25:12INFO mapred.JobClient: Running job: job_201111230519_0001
11/11/23 05:25:13INFO mapred.JobClient: map 0% reduce 0%
11/11/23 05:25:45INFO mapred.JobClient: map 50% reduce 0%
11/11/23 05:25:48INFO mapred.JobClient: map 100% reduce0%
11/11/23 05:26:03INFO mapred.JobClient: map 100% reduce 100%
11/11/23 05:26:07INFO mapred.JobClient: Job complete: job_201111230519_0001
11/11/23 05:26:07INFO mapred.JobClient: Counters: 25
11/11/23 05:26:07INFO mapred.JobClient: Job Counters
11/11/23 05:26:07INFO mapred.JobClient: Launchedreduce tasks=1
11/11/23 05:26:07INFO mapred.JobClient: SLOTS_MILLIS_MAPS=48562
11/11/23 05:26:07INFO mapred.JobClient: Total timespent by all reduces waiting after reserving slots (ms)=0
11/11/23 05:26:07INFO mapred.JobClient: Total timespent by all maps waiting after reserving slots (ms)=0
11/11/23 05:26:07INFO mapred.JobClient: Launched maptasks=2
11/11/23 05:26:07INFO mapred.JobClient: Data-local maptasks=2
11/11/23 05:26:07INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=16678
11/11/23 05:26:07INFO mapred.JobClient: File OutputFormat Counters
11/11/23 05:26:07INFO mapred.JobClient: BytesWritten=41
11/11/23 05:26:07INFO mapred.JobClient: FileSystemCounters
11/11/23 05:26:07INFO mapred.JobClient: FILE_BYTES_READ=79
11/11/23 05:26:07INFO mapred.JobClient: HDFS_BYTES_READ=272
11/11/23 05:26:07INFO mapred.JobClient: FILE_BYTES_WRITTEN=63583
11/11/23 05:26:07INFO mapred.JobClient: HDFS_BYTES_WRITTEN=41
11/11/23 05:26:07INFO mapred.JobClient: File Input FormatCounters
11/11/23 05:26:07INFO mapred.JobClient: Bytes Read=50
11/11/23 05:26:07INFO mapred.JobClient: Map-ReduceFramework
11/11/23 05:26:07INFO mapred.JobClient: Reduce inputgroups=5
11/11/23 05:26:07INFO mapred.JobClient: Map outputmaterialized bytes=85
11/11/23 05:26:07INFO mapred.JobClient: Combine outputrecords=6
11/11/23 05:26:07INFO mapred.JobClient: Map inputrecords=2
11/11/23 05:26:07INFO mapred.JobClient: Reduce shufflebytes=85
11/11/23 05:26:07INFO mapred.JobClient: Reduce outputrecords=5
11/11/23 05:26:07INFO mapred.JobClient: SpilledRecords=12
11/11/23 05:26:07INFO mapred.JobClient: Map outputbytes=82
11/11/23 05:26:07INFO mapred.JobClient: Combine inputrecords=8
11/11/23 05:26:07INFO mapred.JobClient: Map outputrecords=8
11/11/23 05:26:07INFO mapred.JobClient: SPLIT_RAW_BYTES=222
11/11/23 05:26:07INFO mapred.JobClient: Reduce inputrecords=6
進行結果的檢視
[[email protected]~]$ hadoop-0.20.203.0/bin/hadoop fs -ls Found 2 items
drwxr-xr-x - hadoop supergroup 0 2011-11-23 05:22 /user/hadoop/input
drwxr-xr-x - hadoop supergroup 0 2011-11-23 05:26/user/hadoop/output
發現在hadoop中多了一個output資料夾，檢視output中的檔案資訊
[[email protected]~]$ hadoop-0.20.203.0/bin/hadoop fs -ls outputFound 3 items
-rw-r--r-- 1 hadoop supergroup 0 2011-11-23 05:26/user/hadoop/output/_SUCCESS
drwxr-xr-x - hadoop supergroup 0 2011-11-23 05:25/user/hadoop/output/_logs
-rw-r--r-- 1 hadoop supergroup 41 2011-11-23 05:25/user/hadoop/output/part-r-00000
對WordCount的執行結果進行檢視
[[email protected]~]$ hadoop-0.20.203.0/bin/hadoop fs -cat output/part-r-00000Bye 1
Goodbye 1
Hadoop 2
Hello 2
World 2
至此，hadoop下的WordCount例項執行結束，如果還想重新執行一遍，這需把hadoop下的output資料夾刪除，因為hadoop為了保證結果的正確性，存在輸出的資料夾的話，就會報異常，異常如下
[[email protected]~]$ hadoop-0.20.203.0/bin/hadoop jar wordcount.jar org.apache.hadoop.examples.WordCount input output11/11/23 05:33:05INFO mapred.JobClient: Cleaning up the staging areahdfs://localhost:9000/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201111230519_0002
Exception inthread "main" org.apache.hadoop.mapred.FileAlreadyExistsException:Output directory output already exists
atorg.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:134)
atorg.apache.hadoop.mapred.JobClient$2.run(JobClient.java:830)
atorg.apache.hadoop.mapred.JobClient$2.run(JobClient.java:791)
atjava.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
atorg.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:791)
atorg.apache.hadoop.mapreduce.Job.submit(Job.java:465)
atorg.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:494)
atorg.apache.hadoop.examples.WordCount.main(WordCount.java:67)
atsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
atjava.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
刪除output資料夾的操作為
[[email protected]~]$ hadoop-0.20.203.0/bin/hadoop dfs -rmr outputDeletedhdfs://localhost:9000/user/hadoop/output