org.apache.spark.rdd

Class SequenceFileRDDFunctions<K,V>

Object

org.apache.spark.rdd.SequenceFileRDDFunctions<K,V>

All Implemented Interfaces:

java.io.Serializable, Logging

public class SequenceFileRDDFunctions<K,V>
extends Object
implements Logging, scala.Serializable

Extra functions available on RDDs of (key, value) pairs to create a Hadoop SequenceFile, through an implicit conversion. Note that this can't be part of PairRDDFunctions because we need more implicit parameters to convert our keys and values to Writable.

Import org.apache.spark.SparkContext._ at the top of their program to use these functions.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
SequenceFileRDDFunctions(RDD<scala.Tuple2<K,V>> self, scala.Function1<K,org.apache.hadoop.io.Writable> evidence$1, scala.reflect.ClassTag<K> evidence$2, scala.Function1<V,org.apache.hadoop.io.Writable> evidence$3, scala.reflect.ClassTag<V> evidence$4)

Method Summary

Methods

Modifier and Type	Method and Description
void	saveAsSequenceFile(String path, scala.Option<Class<? extends org.apache.hadoop.io.compress.CompressionCodec>> codec) Output the RDD as a Hadoop SequenceFile using the Writable types we infer from the RDD's key and value types.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initialized, initializeIfNecessary, initializeLogging, initLock, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

SequenceFileRDDFunctions

public SequenceFileRDDFunctions(RDD<scala.Tuple2<K,V>> self,
                        scala.Function1<K,org.apache.hadoop.io.Writable> evidence$1,
                        scala.reflect.ClassTag<K> evidence$2,
                        scala.Function1<V,org.apache.hadoop.io.Writable> evidence$3,
                        scala.reflect.ClassTag<V> evidence$4)

Method Detail

saveAsSequenceFile

public void saveAsSequenceFile(String path,
                      scala.Option<Class<? extends org.apache.hadoop.io.compress.CompressionCodec>> codec)

Output the RDD as a Hadoop SequenceFile using the Writable types we infer from the RDD's key and value types. If the key or value are Writable, then we use their classes directly; otherwise we map primitive types such as Int and Double to IntWritable, DoubleWritable, etc, byte arrays to BytesWritable, and Strings to Text. The path can be on any Hadoop-supported file system.