org.apache.spark.mapred

Class SparkHadoopMapRedUtil

Object

org.apache.spark.mapred.SparkHadoopMapRedUtil

public class SparkHadoopMapRedUtil
extends Object

Constructor Summary

Constructors

Constructor and Description
SparkHadoopMapRedUtil()

Method Summary

Modifier and Type	Method and Description
static void	commitTask(org.apache.hadoop.mapreduce.OutputCommitter committer, org.apache.hadoop.mapreduce.TaskAttemptContext mrTaskContext, int jobId, int splitId) Commits a task output.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkHadoopMapRedUtil

public SparkHadoopMapRedUtil()

Method Detail

commitTask

public static void commitTask(org.apache.hadoop.mapreduce.OutputCommitter committer,
                              org.apache.hadoop.mapreduce.TaskAttemptContext mrTaskContext,
                              int jobId,
                              int splitId)

Commits a task output. Before committing the task output, we need to know whether some other task attempt might be racing to commit the same output partition. Therefore, coordinate with the driver in order to determine whether this attempt can commit (please see SPARK-4879 for details).

Output commit coordinator is only used when spark.hadoop.outputCommitCoordination.enabled is set to true (which is the default).

Parameters:

committer - (undocumented)

mrTaskContext - (undocumented)

jobId - (undocumented)

splitId - (undocumented)