SparkSessionExtensions

:: Experimental :: Holder for injection points to the SparkSession. We make NO guarantee about the stability regarding binary compatibility and source compatibility of methods here.

This current provides the following extension points: - Analyzer Rules. - Check Analysis Rules - Optimizer Rules. - Planning Strategies. - Customized Parser. - (External) Catalog listeners.

The extensions can be used by calling withExtension on the SparkSession.Builder, for example:

SparkSession.builder()
  .master("...")
  .conf("...", true)
  .withExtensions { extensions =>
    extensions.injectResolutionRule { session =>
      ...
    }
    extensions.injectParser { (session, parser) =>
      ...
    }
  }
  .getOrCreate()

Note that none of the injected builders should assume that the SparkSession is fully initialized and should not touch the session's internals (e.g. the SessionState).

Annotations: @DeveloperApi() @Experimental() @Unstable()
Source: SparkSessionExtensions.scala

Linear Supertypes

AnyRef, Any

Instance Constructors

new SparkSessionExtensions()

Type Members

type CheckRuleBuilder = (SparkSession) ⇒ (LogicalPlan) ⇒ Unit
type ParserBuilder = (SparkSession, ParserInterface) ⇒ ParserInterface
type RuleBuilder = (SparkSession) ⇒ Rule[LogicalPlan]
type StrategyBuilder = (SparkSession) ⇒ Strategy

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
def injectCheckRule(builder: CheckRuleBuilder): Unit

Inject an check analysis Rule builder into the SparkSession.
Inject an check analysis Rule builder into the SparkSession. The injected rules will be executed after the analysis phase. A check analysis rule is used to detect problems with a LogicalPlan and should throw an exception when a problem is found.
def injectOptimizerRule(builder: RuleBuilder): Unit

Inject an optimizer Rule builder into the SparkSession.
Inject an optimizer Rule builder into the SparkSession. The injected rules will be executed during the operator optimization batch. An optimizer rule is used to improve the quality of an analyzed logical plan; these rules should never modify the result of the LogicalPlan.
def injectParser(builder: ParserBuilder): Unit

Inject a custom parser into the SparkSession.
Inject a custom parser into the SparkSession. Note that the builder is passed a session and an initial parser. The latter allows for a user to create a partial parser and to delegate to the underlying parser for completeness. If a user injects more parsers, then the parsers are stacked on top of each other.
def injectPlannerStrategy(builder: StrategyBuilder): Unit

Inject a planner Strategy builder into the SparkSession.
Inject a planner Strategy builder into the SparkSession. The injected strategy will be used to convert a LogicalPlan into a executable org.apache.spark.sql.execution.SparkPlan.
def injectPostHocResolutionRule(builder: RuleBuilder): Unit

Inject an analyzer Rule builder into the SparkSession.
Inject an analyzer Rule builder into the SparkSession. These analyzer rules will be executed after resolution.
def injectResolutionRule(builder: RuleBuilder): Unit

Inject an analyzer resolution Rule builder into the SparkSession.
Inject an analyzer resolution Rule builder into the SparkSession. These analyzer rules will be executed as part of the resolution phase of analysis.
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package sql

class SparkSessionExtensions extends AnyRef

Instance Constructors

new SparkSessionExtensions()

Type Members

type CheckRuleBuilder = (SparkSession) ⇒ (LogicalPlan) ⇒ Unit

type ParserBuilder = (SparkSession, ParserInterface) ⇒ ParserInterface

type RuleBuilder = (SparkSession) ⇒ Rule[LogicalPlan]

type StrategyBuilder = (SparkSession) ⇒ Strategy

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

def injectCheckRule(builder: CheckRuleBuilder): Unit

def injectOptimizerRule(builder: RuleBuilder): Unit

def injectParser(builder: ParserBuilder): Unit

def injectPlannerStrategy(builder: StrategyBuilder): Unit

def injectPostHocResolutionRule(builder: RuleBuilder): Unit

def injectResolutionRule(builder: RuleBuilder): Unit

final def isInstanceOf[T0]: Boolean

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from AnyRef

Inherited from Any

Ungrouped