org.apache.spark.rdd

Class PipedRDD<T>

Object

org.apache.spark.rdd.RDD<String>

org.apache.spark.rdd.PipedRDD<T>

All Implemented Interfaces:

java.io.Serializable, Logging

public class PipedRDD<T>
extends RDD<String>

An RDD that pipes the contents of each parent partition through an external command (printing them one per line) and returns the output as a collection of strings.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	PipedRDD.NotEqualsFileNameFilter A FilenameFilter that accepts anything that isn't equal to the name passed in.

Constructor Summary

Constructors

Constructor and Description
PipedRDD(RDD<T> prev, scala.collection.Seq<String> command, scala.collection.Map<String,String> envVars, scala.Function1<scala.Function1<String,scala.runtime.BoxedUnit>,scala.runtime.BoxedUnit> printPipeContext, scala.Function2<T,scala.Function1<String,scala.runtime.BoxedUnit>,scala.runtime.BoxedUnit> printRDDElement, boolean separateWorkingDir, scala.reflect.ClassTag<T> evidence$1)
PipedRDD(RDD<T> prev, String command, scala.collection.Map<String,String> envVars, scala.Function1<scala.Function1<String,scala.runtime.BoxedUnit>,scala.runtime.BoxedUnit> printPipeContext, scala.Function2<T,scala.Function1<String,scala.runtime.BoxedUnit>,scala.runtime.BoxedUnit> printRDDElement, boolean separateWorkingDir, scala.reflect.ClassTag<T> evidence$2)

Method Summary

Methods

Modifier and Type	Method and Description
scala.collection.Iterator<String>	compute(Partition split, TaskContext context) :: DeveloperApi :: Implemented by subclasses to compute a given partition.
Partition[]	getPartitions() Implemented by subclasses to return the set of partitions in this RDD.
static scala.collection.Seq<String>	tokenize(String command)

Methods inherited from class org.apache.spark.rdd.RDD

aggregate, cache, cartesian, checkpoint, checkpointData, coalesce, collect, collect, collectPartitions, computeOrReadCheckpoint, conf, context, count, countApprox, countApproxDistinct, countApproxDistinct, countByValue, countByValueApprox, creationSite, dependencies, distinct, distinct, doCheckpoint, elementClassTag, filter, filterWith, first, flatMap, flatMapWith, fold, foreach, foreachPartition, foreachWith, getCheckpointFile, getCreationSite, getNarrowAncestors, getStorageLevel, glom, groupBy, groupBy, groupBy, id, intersection, intersection, intersection, isCheckpointed, iterator, keyBy, map, mapPartitions, mapPartitionsWithContext, mapPartitionsWithIndex, mapPartitionsWithSplit, mapWith, markCheckpointed, max, min, name, partitioner, partitions, persist, persist, pipe, pipe, pipe, preferredLocations, randomSplit, reduce, repartition, retag, retag, sample, saveAsObjectFile, saveAsTextFile, saveAsTextFile, setName, sortBy, sparkContext, subtract, subtract, subtract, take, takeOrdered, takeSample, toArray, toDebugString, toJavaRDD, toLocalIterator, top, toString, union, unpersist, zip, zipPartitions, zipPartitions, zipPartitions, zipPartitions, zipPartitions, zipPartitions, zipWithIndex, zipWithUniqueId

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

PipedRDD

public PipedRDD(RDD<T> prev,
        scala.collection.Seq<String> command,
        scala.collection.Map<String,String> envVars,
        scala.Function1<scala.Function1<String,scala.runtime.BoxedUnit>,scala.runtime.BoxedUnit> printPipeContext,
        scala.Function2<T,scala.Function1<String,scala.runtime.BoxedUnit>,scala.runtime.BoxedUnit> printRDDElement,
        boolean separateWorkingDir,
        scala.reflect.ClassTag<T> evidence$1)

PipedRDD

public PipedRDD(RDD<T> prev,
        String command,
        scala.collection.Map<String,String> envVars,
        scala.Function1<scala.Function1<String,scala.runtime.BoxedUnit>,scala.runtime.BoxedUnit> printPipeContext,
        scala.Function2<T,scala.Function1<String,scala.runtime.BoxedUnit>,scala.runtime.BoxedUnit> printRDDElement,
        boolean separateWorkingDir,
        scala.reflect.ClassTag<T> evidence$2)

Method Detail

tokenize

public static scala.collection.Seq<String> tokenize(String command)

getPartitions

public Partition[] getPartitions()

Description copied from class: RDD

Implemented by subclasses to return the set of partitions in this RDD. This method will only be called once, so it is safe to implement a time-consuming computation in it.

compute

public scala.collection.Iterator<String> compute(Partition split,
                                        TaskContext context)

Description copied from class: RDD

:: DeveloperApi :: Implemented by subclasses to compute a given partition.

Specified by:

compute in class RDD<String>