org.apache.spark.internal.io

Class HadoopWriteConfigUtil<K,V>

Object

org.apache.spark.internal.io.HadoopWriteConfigUtil<K,V>

All Implemented Interfaces:

java.io.Serializable

public abstract class HadoopWriteConfigUtil<K,V>
extends Object
implements scala.Serializable

Interface for create output format/committer/writer used during saving an RDD using a Hadoop OutputFormat (both from the old mapred API and the new mapreduce API)

Notes: 1. Implementations should throw IllegalArgumentException when wrong hadoop API is referenced; 2. Implementations must be serializable, as the instance instantiated on the driver will be used for tasks on executors; 3. Implementations should have a constructor with exactly one argument: (conf: SerializableConfiguration) or (conf: SerializableJobConf).

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
HadoopWriteConfigUtil(scala.reflect.ClassTag<V> evidence$1)

Method Summary

Modifier and Type	Method and Description
abstract void	assertConf(org.apache.hadoop.mapreduce.JobContext jobContext, SparkConf conf)
abstract void	closeWriter(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)
abstract HadoopMapReduceCommitProtocol	createCommitter(int jobId)
abstract org.apache.hadoop.mapreduce.JobContext	createJobContext(String jobTrackerId, int jobId)
abstract org.apache.hadoop.mapreduce.TaskAttemptContext	createTaskAttemptContext(String jobTrackerId, int jobId, int splitId, int taskAttemptId)
abstract void	initOutputFormat(org.apache.hadoop.mapreduce.JobContext jobContext)
abstract void	initWriter(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext, int splitId)
abstract void	write(scala.Tuple2<K,V> pair)

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopWriteConfigUtil

public HadoopWriteConfigUtil(scala.reflect.ClassTag<V> evidence$1)

Method Detail

createJobContext

public abstract org.apache.hadoop.mapreduce.JobContext createJobContext(String jobTrackerId,
                                                                        int jobId)

createTaskAttemptContext

public abstract org.apache.hadoop.mapreduce.TaskAttemptContext createTaskAttemptContext(String jobTrackerId,
                                                                                        int jobId,
                                                                                        int splitId,
                                                                                        int taskAttemptId)

createCommitter

public abstract HadoopMapReduceCommitProtocol createCommitter(int jobId)

initWriter

public abstract void initWriter(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext,
                                int splitId)

write

public abstract void write(scala.Tuple2<K,V> pair)

closeWriter

public abstract void closeWriter(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)

initOutputFormat

public abstract void initOutputFormat(org.apache.hadoop.mapreduce.JobContext jobContext)

assertConf

public abstract void assertConf(org.apache.hadoop.mapreduce.JobContext jobContext,
                                SparkConf conf)