org.apache.spark.input

Class FixedLengthBinaryInputFormat

Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.apache.hadoop.mapreduce.lib.input.FileInputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.BytesWritable>

org.apache.spark.input.FixedLengthBinaryInputFormat

public class FixedLengthBinaryInputFormat
extends org.apache.hadoop.mapreduce.lib.input.FileInputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.BytesWritable>

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

org.apache.hadoop.mapreduce.lib.input.FileInputFormat.Counter

Constructor Summary

Constructors

Constructor and Description
FixedLengthBinaryInputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
long	computeSplitSize(long blockSize, long minSize, long maxSize) This input format overrides computeSplitSize() to make sure that each split only contains full records.
org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.BytesWritable>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context) Create a FixedLengthBinaryRecordReader
static int	getRecordLength(org.apache.hadoop.mapreduce.JobContext context) Retrieves the record length property from a Hadoop configuration
boolean	isSplitable(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.fs.Path filename) Override of isSplitable to ensure initial computation of the record length
static String	RECORD_LENGTH_PROPERTY() Property name to set in Hadoop JobConfs for record length

Methods inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

addInputPath, addInputPaths, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, getSplits, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSize

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FixedLengthBinaryInputFormat

public FixedLengthBinaryInputFormat()

Method Detail

RECORD_LENGTH_PROPERTY

public static String RECORD_LENGTH_PROPERTY()

Property name to set in Hadoop JobConfs for record length

getRecordLength

public static int getRecordLength(org.apache.hadoop.mapreduce.JobContext context)

Retrieves the record length property from a Hadoop configuration

isSplitable

public boolean isSplitable(org.apache.hadoop.mapreduce.JobContext context,
                  org.apache.hadoop.fs.Path filename)

Override of isSplitable to ensure initial computation of the record length

Overrides:

isSplitable in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.BytesWritable>

computeSplitSize

public long computeSplitSize(long blockSize,
                    long minSize,
                    long maxSize)

This input format overrides computeSplitSize() to make sure that each split only contains full records. Each InputSplit passed to FixedLengthBinaryRecordReader will start at the first byte of a record, and the last byte will the last byte of a record.

Overrides:

computeSplitSize in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.BytesWritable>

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.BytesWritable> createRecordReader(org.apache.hadoop.mapreduce.InputSplit split,
                                                                                                                                org.apache.hadoop.mapreduce.TaskAttemptContext context)

Create a FixedLengthBinaryRecordReader

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.BytesWritable>