Hadoop源码详解之Job 类
2023-09-14 09:13:17 时间
Hadoop源码详解之Job
类
1. 源码
- 包:
org.apache.hadoop.mapreduce
- 继承的接口有:
AutoCloseable
,JobContext
,org.apache.hadoop.mapreduce.MRJobConfig
The job submitter’s view of the Job.
It allows the user to configure the job, submit it, control its execution, and query the state. The set methods only work until the job is submitted, afterwards they will throw an IllegalStateException.
Normally the user creates the application, describes various facets of the job via Job and then submits the job and monitor its progress.
作业提交者层次上的作业视图。
它允许用户配置job,提交它,并且控制它的运行,然后查询状态。set
方法 仅仅工作直到job被提交,否则会抛出IllegalStateException
。
正常情况下,用户创建一个应用,描述工作的各个方面,通过Job
类,并且提交job,然后监测它的进度。
下面给出一个示例关于如何使用 Job
类去提交一个job。
// Create a new Job
Job job = Job.getInstance();
job.setJarByClass(MyJob.class);
// Specify various job-specific parameters
job.setJobName("myjob");
job.setInputPath(new Path("in"));
job.setOutputPath(new Path("out"));
job.setMapperClass(MyJob.MyMapper.class);
job.setReducerClass(MyJob.MyReducer.class);
// Submit the job, then poll for progress until the job is complete
job.waitForCompletion(true);
2. 方法详解
2.1 构造器
以前的构造器全部建议不再使用,转而使用getInistance(...)
这个方法。
2.2 waitForCompletion
- 方法释义
Submit the job to the cluster and wait for it to finish.
提交job到集群,并且等待它完成。
- 方法源码
/**
* @param verbose print the progress to the user
* @return true if the job succeeded
* @throws IOException thrown if the communication with the JobTracker is lost
*/
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
相关文章
- 手动安装Hadoop集群的过程
- Hadoop源码学习
- 高可用Hadoop平台-Oozie工作流
- Hadoop - Ambari集群管理剖析
- Hadoop参考学习
- Hadoop 启动hdfs和yarn的命令
- Hadoop MapReduce执行过程详解(带hadoop例子)
- Hadoop-2.2.0中国文献—— MapReduce 下一代 -- 公平调度
- Cloudera公司主要提供Apache Hadoop开发工程师认证
- Hadoop项目实战之将MapReduce的结果写入到Mysql
- Hadoop 源码详解之InputFormat类
- Hadoop 源码详解之RecordReader接口
- Hadoop源码详解之FileOutputFormat 类
- Hadoop源码详解之Mapper类
- Hadoop源码阅读之WritableComparator类
- CentOS6安装大数据软件(二):Hadoop分布式集群配置