企业外贸网站推广,网站服务器大小,php网站模板,做旅游网站的yi1.1 文件准备 创建本地目录和创建两个文本文件#xff0c;在两个文件中输入单词#xff0c;用于统计词频。
cd /usr/local/hadoop
mkdir WordFile
cd WordFile
touch wordfile1.txt
touch wordfile2.txt1.2 创建一个HDFS目录#xff0c;在本地上不可见,并将本地文本文件上传…1.1 文件准备 创建本地目录和创建两个文本文件在两个文件中输入单词用于统计词频。
cd /usr/local/hadoop
mkdir WordFile
cd WordFile
touch wordfile1.txt
touch wordfile2.txt1.2 创建一个HDFS目录在本地上不可见,并将本地文本文件上传到HDFS目录。通过如下命令创建。
cd /usr/local/hadoop
./bin/hdfs dfs -mkdir wordfileinput
./bin/hdfs dfs -put ./WordFile/wordfile1.txt wordfileinput
./bin/hdfs dfs -put ./WordFile/wordfile2.txt wordfileinput
1.3 保证HDFS目录不存在output,我们执行如下命令每次运行词频统计都要删除output输出文件,/user/hadoop/是HDFS的用户目录不是本地目录。
./bin/hdfs dfs -rm -r /user/hadoop/output1.4 Eclips编写代码 创建Java project ,项目名称为MapReduceWordCount,右键项目名导入相关Jar包。 1.5 点击Add External Jars,进入目录/usr/local/hadoop/share/hadoop,导入如下包。
“/usr/local/hadoop/share/hadoop/common”目录下的hadoop-common-3.1.3.jar和haoop-nfs-3.1.3.jar“/usr/local/hadoop/share/hadoop/common/lib”目录下的所有JAR包“/usr/local/hadoop/share/hadoop/mapreduce”目录下的所有JAR包但是不包括jdiff、lib、lib-examples和sources目录;“/usr/local/hadoop/share/hadoop/mapreduce/lib”目录下的所有JAR包。
1.6 创建类WordCount.java
import java.io.IOException;
import java.util.Iterator;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
public class WordCount {public WordCount() {}public static void main(String[] args) throws Exception {Configuration conf new Configuration();String[] otherArgs (new GenericOptionsParser(conf, args)).getRemainingArgs();if(otherArgs.length 2) {System.err.println(Usage: wordcount in [in...] out);System.exit(2);}Job job Job.getInstance(conf, word count);job.setJarByClass(WordCount.class);job.setMapperClass(WordCount.TokenizerMapper.class);job.setCombinerClass(WordCount.IntSumReducer.class);job.setReducerClass(WordCount.IntSumReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class); for(int i 0; i otherArgs.length - 1; i) {FileInputFormat.addInputPath(job, new Path(otherArgs[i]));}FileOutputFormat.setOutputPath(job, new Path(otherArgs[otherArgs.length - 1]));System.exit(job.waitForCompletion(true)?0:1);}public static class TokenizerMapper extends MapperObject, Text, Text, IntWritable {private static final IntWritable one new IntWritable(1);private Text word new Text();public TokenizerMapper() {}public void map(Object key, Text value, MapperObject, Text, Text, IntWritable.Context context) throws IOException, InterruptedException {StringTokenizer itr new StringTokenizer(value.toString()); while(itr.hasMoreTokens()) {this.word.set(itr.nextToken());context.write(this.word, one);}}}
public static class IntSumReducer extends ReducerText, IntWritable, Text, IntWritable {private IntWritable result new IntWritable();public IntSumReducer() {}public void reduce(Text key, IterableIntWritable values, ReducerText, IntWritable, Text, IntWritable.Context context) throws IOException, InterruptedException {int sum 0;IntWritable val;for(Iterator i$ values.iterator(); i$.hasNext(); sum val.get()) {val (IntWritable)i$.next();}this.result.set(sum);context.write(key, this.result);}}
}1.7 编译打包程序 将程序打包到 /usr/local/hadoop/myapp目录下
cd /usr/local/hadoop
mkdir myappRun As 运行程序右键工程名ExportJavaRunnable JAR file “Launch configuration”用于设置生成的JAR包被部署启动时运行的主类需要在下拉列表中选择刚才配置的类“WordCount-MapReduceWordCount”。在“Export destination”中需要设置JAR包要输出保存到哪个目录即其名称。点击finish,中间会出现一些信息一直点击即可。
1.8 运行程序 启动hadoop
cd /usr/local/hadoop
./sbin/start-dfs.sh
./bin/hadoop jar ./myapp/WordCount.jar wordfileinput output 1.9 查看结果
cd /usr/local/hadoop
./bin/hdfs dfs -cat output/*1.20 查看HDFS 文件系统 进入/usr/local/hadoop/bin 目录执行相关命令。
./hadoop fs -ls1.21 源文档 http://dblab.xmu.edu.cn/blog/2481-2/#more-2481
