org.apache.spark.graphx.impl

Class EdgePartition<ED,VD>

Object

org.apache.spark.graphx.impl.EdgePartition<ED,VD>

All Implemented Interfaces:

java.io.Serializable

public class EdgePartition<ED,VD>
extends Object
implements scala.Serializable

A collection of edges, along with referenced vertex attributes and an optional active vertex set for filtering computation on the edges.

The edges are stored in columnar format in localSrcIds, localDstIds, and data. All referenced global vertex ids are mapped to a compact set of local vertex ids according to the global2local map. Each local vertex id is a valid index into vertexAttrs, which stores the corresponding vertex attribute, and local2global, which stores the reverse mapping to global vertex id. The global vertex ids that are active are optionally stored in activeSet.

The edges are clustered by source vertex id, and the mapping from global vertex id to the index of the corresponding edge cluster is stored in index.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description

EdgePartition(int[] localSrcIds, int[] localDstIds, Object data, org.apache.spark.graphx.util.collection.GraphXPrimitiveKeyOpenHashMap<Object,Object> index, org.apache.spark.graphx.util.collection.GraphXPrimitiveKeyOpenHashMap<Object,Object> global2local, long[] local2global, Object vertexAttrs, scala.Option<org.apache.spark.util.collection.OpenHashSet<Object>> activeSet, scala.reflect.ClassTag<ED> evidence$1, scala.reflect.ClassTag<VD> evidence$2)

Method Summary

Methods

Modifier and Type	Method and Description
<A> scala.collection.Iterator<scala.Tuple2<Object,A>>	aggregateMessagesEdgeScan(scala.Function1<EdgeContext<VD,ED,A>,scala.runtime.BoxedUnit> sendMsg, scala.Function2<A,A,A> mergeMsg, TripletFields tripletFields, EdgeActiveness activeness, scala.reflect.ClassTag<A> evidence$11) Send messages along edges and aggregate them at the receiving vertices.
<A> scala.collection.Iterator<scala.Tuple2<Object,A>>	aggregateMessagesIndexScan(scala.Function1<EdgeContext<VD,ED,A>,scala.runtime.BoxedUnit> sendMsg, scala.Function2<A,A,A> mergeMsg, TripletFields tripletFields, EdgeActiveness activeness, scala.reflect.ClassTag<A> evidence$12) Send messages along edges and aggregate them at the receiving vertices.
EdgePartition<ED,VD>	filter(scala.Function1<EdgeTriplet<VD,ED>,Object> epred, scala.Function2<Object,VD,Object> vpred) Construct a new edge partition containing only the edges matching epred and where both vertices match vpred.
void	foreach(scala.Function1<Edge<ED>,scala.runtime.BoxedUnit> f) Apply the function f to all edges in this partition.
EdgePartition<ED,VD>	groupEdges(scala.Function2<ED,ED,ED> merge) Merge all the edges with the same src and dest id into a single edge using the merge function
int	indexSize() The number of unique source vertices in the partition.
<ED2,ED3> EdgePartition<ED3,VD>	innerJoin(EdgePartition<ED2,?> other, scala.Function4<Object,Object,ED,ED2,ED3> f, scala.reflect.ClassTag<ED2> evidence$9, scala.reflect.ClassTag<ED3> evidence$10) Apply f to all edges present in both this and other and return a new EdgePartition containing the resulting edges.
boolean	isActive(long vid) Look up vid in activeSet, throwing an exception if it is None.
Object	iterator() Get an iterator over the edges in this partition.
<ED2> EdgePartition<ED2,VD>	map(scala.Function1<Edge<ED>,ED2> f, scala.reflect.ClassTag<ED2> evidence$7) Construct a new edge partition by applying the function f to all edges in this partition.
<ED2> EdgePartition<ED2,VD>	map(scala.collection.Iterator<ED2> iter, scala.reflect.ClassTag<ED2> evidence$8) Construct a new edge partition by using the edge attributes contained in the iterator.
scala.Option<Object>	numActives() The number of active vertices, if any exist.
EdgePartition<ED,VD>	reverse() Reverse all the edges in this partition.
int	size() The number of edges in this partition
scala.collection.Iterator<EdgeTriplet<VD,ED>>	tripletIterator(boolean includeSrc, boolean includeDst) Get an iterator over the edge triplets in this partition.
EdgePartition<ED,VD>	updateVertices(scala.collection.Iterator<scala.Tuple2<Object,VD>> iter) Return a new `EdgePartition` with updates to vertex attributes specified in `iter`.
EdgePartition<ED,VD>	withActiveSet(scala.collection.Iterator<Object> iter) Return a new `EdgePartition` with the specified active set, provided as an iterator.
<ED2> EdgePartition<ED2,VD>	withData(Object data, scala.reflect.ClassTag<ED2> evidence$5) Return a new `EdgePartition` with the specified edge data.
<VD2> EdgePartition<ED,VD2>	withoutVertexAttributes(scala.reflect.ClassTag<VD2> evidence$6) Return a new `EdgePartition` without any locally cached vertex attributes.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EdgePartition

public EdgePartition(int[] localSrcIds,
             int[] localDstIds,
             Object data,
             org.apache.spark.graphx.util.collection.GraphXPrimitiveKeyOpenHashMap<Object,Object> index,
             org.apache.spark.graphx.util.collection.GraphXPrimitiveKeyOpenHashMap<Object,Object> global2local,
             long[] local2global,
             Object vertexAttrs,
             scala.Option<org.apache.spark.util.collection.OpenHashSet<Object>> activeSet,
             scala.reflect.ClassTag<ED> evidence$1,
             scala.reflect.ClassTag<VD> evidence$2)

Method Detail

withData

public <ED2> EdgePartition<ED2,VD> withData(Object data,
                                   scala.reflect.ClassTag<ED2> evidence$5)

Return a new `EdgePartition` with the specified edge data.

withActiveSet

public EdgePartition<ED,VD> withActiveSet(scala.collection.Iterator<Object> iter)

Return a new `EdgePartition` with the specified active set, provided as an iterator.

updateVertices

public EdgePartition<ED,VD> updateVertices(scala.collection.Iterator<scala.Tuple2<Object,VD>> iter)

Return a new `EdgePartition` with updates to vertex attributes specified in `iter`.

withoutVertexAttributes

public <VD2> EdgePartition<ED,VD2> withoutVertexAttributes(scala.reflect.ClassTag<VD2> evidence$6)

Return a new `EdgePartition` without any locally cached vertex attributes.

isActive

public boolean isActive(long vid)

Look up vid in activeSet, throwing an exception if it is None.

numActives

public scala.Option<Object> numActives()

The number of active vertices, if any exist.

reverse

public EdgePartition<ED,VD> reverse()

Reverse all the edges in this partition.

Returns:

a new edge partition with all edges reversed.

map

public <ED2> EdgePartition<ED2,VD> map(scala.Function1<Edge<ED>,ED2> f,
                              scala.reflect.ClassTag<ED2> evidence$7)

Construct a new edge partition by applying the function f to all edges in this partition.

Be careful not to keep references to the objects passed to f. To improve GC performance the same object is re-used for each call.

Parameters:

f - a function from an edge to a new attribute

Returns:

a new edge partition with the result of the function f applied to each edge

map

public <ED2> EdgePartition<ED2,VD> map(scala.collection.Iterator<ED2> iter,
                              scala.reflect.ClassTag<ED2> evidence$8)

Construct a new edge partition by using the edge attributes contained in the iterator.

Parameters:

iter - an iterator for the new attribute values

Returns:

a new edge partition with the attribute values replaced

filter

public EdgePartition<ED,VD> filter(scala.Function1<EdgeTriplet<VD,ED>,Object> epred,
                          scala.Function2<Object,VD,Object> vpred)

Construct a new edge partition containing only the edges matching epred and where both vertices match vpred.

foreach

public void foreach(scala.Function1<Edge<ED>,scala.runtime.BoxedUnit> f)

Apply the function f to all edges in this partition.

Parameters:

f - an external state mutating user defined function.

groupEdges

public EdgePartition<ED,VD> groupEdges(scala.Function2<ED,ED,ED> merge)

Merge all the edges with the same src and dest id into a single edge using the merge function

Parameters:

merge - a commutative associative merge operation

Returns:

a new edge partition without duplicate edges

innerJoin

public <ED2,ED3> EdgePartition<ED3,VD> innerJoin(EdgePartition<ED2,?> other,
                                        scala.Function4<Object,Object,ED,ED2,ED3> f,
                                        scala.reflect.ClassTag<ED2> evidence$9,
                                        scala.reflect.ClassTag<ED3> evidence$10)

Apply f to all edges present in both this and other and return a new EdgePartition containing the resulting edges.

If there are multiple edges with the same src and dst in this, f will be invoked once for each edge, but each time it may be invoked on any corresponding edge in other.

If there are multiple edges with the same src and dst in other, f will only be invoked once.

size

public int size()

The number of edges in this partition

Returns:

size of the partition

indexSize

public int indexSize()

The number of unique source vertices in the partition.

iterator

public Object iterator()

Get an iterator over the edges in this partition.

Be careful not to keep references to the objects from this iterator. To improve GC performance the same object is re-used in next().

Returns:

an iterator over edges in the partition

tripletIterator

public scala.collection.Iterator<EdgeTriplet<VD,ED>> tripletIterator(boolean includeSrc,
                                                            boolean includeDst)

Get an iterator over the edge triplets in this partition.

It is safe to keep references to the objects from this iterator.

aggregateMessagesEdgeScan

public <A> scala.collection.Iterator<scala.Tuple2<Object,A>> aggregateMessagesEdgeScan(scala.Function1<EdgeContext<VD,ED,A>,scala.runtime.BoxedUnit> sendMsg,
                                                                              scala.Function2<A,A,A> mergeMsg,
                                                                              TripletFields tripletFields,
                                                                              EdgeActiveness activeness,
                                                                              scala.reflect.ClassTag<A> evidence$11)

Send messages along edges and aggregate them at the receiving vertices. Implemented by scanning all edges sequentially.

Parameters:

sendMsg - generates messages to neighboring vertices of an edge

mergeMsg - the combiner applied to messages destined to the same vertex

tripletFields - which triplet fields sendMsg uses

activeness - criteria for filtering edges based on activeness

Returns:

iterator aggregated messages keyed by the receiving vertex id

aggregateMessagesIndexScan

public <A> scala.collection.Iterator<scala.Tuple2<Object,A>> aggregateMessagesIndexScan(scala.Function1<EdgeContext<VD,ED,A>,scala.runtime.BoxedUnit> sendMsg,
                                                                               scala.Function2<A,A,A> mergeMsg,
                                                                               TripletFields tripletFields,
                                                                               EdgeActiveness activeness,
                                                                               scala.reflect.ClassTag<A> evidence$12)

Send messages along edges and aggregate them at the receiving vertices. Implemented by filtering the source vertex index, then scanning each edge cluster.

Parameters:

sendMsg - generates messages to neighboring vertices of an edge

mergeMsg - the combiner applied to messages destined to the same vertex

tripletFields - which triplet fields sendMsg uses

activeness - criteria for filtering edges based on activeness

Returns:

iterator aggregated messages keyed by the receiving vertex id