org.apache.spark.sql

Class SQLContext

java.lang.Object

org.apache.spark.sql.SQLContext

All Implemented Interfaces:

java.io.Serializable, Logging

Direct Known Subclasses:

HiveContext

public class SQLContext
extends java.lang.Object
implements Logging, scala.Serializable

The entry point for working with structured data (rows and columns) in Spark. Allows the creation of DataFrame objects as well as the execution of SQL queries.

Since:

1.0.0

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	SQLContext.implicits$ :: Experimental :: (Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.
protected class	SQLContext.QueryExecution
protected class	SQLContext.SparkPlanner

Constructor Summary

Constructors

Constructor and Description
SQLContext(JavaSparkContext sparkContext)
SQLContext(SparkContext sparkContext)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	addJar(java.lang.String path) Add a jar to SQLContext
protected org.apache.spark.sql.catalyst.analysis.Analyzer	analyzer()
DataFrame	applySchema(JavaRDD<?> rdd, java.lang.Class<?> beanClass)
DataFrame	applySchema(JavaRDD<Row> rowRDD, StructType schema)
DataFrame	applySchema(RDD<?> rdd, java.lang.Class<?> beanClass)
DataFrame	applySchema(RDD<Row> rowRDD, StructType schema)
protected DataFrame	applySchemaToPythonRDD(RDD<java.lang.Object[]> rdd, java.lang.String schemaString)
protected DataFrame	applySchemaToPythonRDD(RDD<java.lang.Object[]> rdd, StructType schema)
DataFrame	baseRelationToDataFrame(BaseRelation baseRelation)
protected org.apache.spark.sql.execution.CacheManager	cacheManager()
void	cacheTable(java.lang.String tableName) Caches the specified table in-memory.
protected org.apache.spark.sql.catalyst.analysis.Catalog	catalog()
static void	clearActive() Clears the active SQLContext for current thread.
void	clearCache() Removes all cached tables from the in-memory cache.
protected org.apache.spark.sql.SQLConf	conf()
DataFrame	createDataFrame(JavaRDD<?> rdd, java.lang.Class<?> beanClass)
DataFrame	createDataFrame(JavaRDD<Row> rowRDD, StructType schema)
DataFrame	createDataFrame(java.util.List<?> data, java.lang.Class<?> beanClass)
DataFrame	createDataFrame(java.util.List<Row> rows, StructType schema)
DataFrame	createDataFrame(RDD<?> rdd, java.lang.Class<?> beanClass)
<A extends scala.Product> DataFrame	createDataFrame(RDD<A> rdd, scala.reflect.api.TypeTags.TypeTag<A> evidence$1)
DataFrame	createDataFrame(RDD<Row> rowRDD, StructType schema)
<A extends scala.Product> DataFrame	createDataFrame(scala.collection.Seq<A> data, scala.reflect.api.TypeTags.TypeTag<A> evidence$2)
<T> Dataset<T>	createDataset(java.util.List<T> data, Encoder<T> evidence$5)
<T> Dataset<T>	createDataset(RDD<T> data, Encoder<T> evidence$4)
<T> Dataset<T>	createDataset(scala.collection.Seq<T> data, Encoder<T> evidence$3)
DataFrame	createExternalTable(java.lang.String tableName, java.lang.String path)
DataFrame	createExternalTable(java.lang.String tableName, java.lang.String source, java.util.Map<java.lang.String,java.lang.String> options)
DataFrame	createExternalTable(java.lang.String tableName, java.lang.String source, scala.collection.immutable.Map<java.lang.String,java.lang.String> options)
DataFrame	createExternalTable(java.lang.String tableName, java.lang.String path, java.lang.String source)
DataFrame	createExternalTable(java.lang.String tableName, java.lang.String source, StructType schema, java.util.Map<java.lang.String,java.lang.String> options)
DataFrame	createExternalTable(java.lang.String tableName, java.lang.String source, StructType schema, scala.collection.immutable.Map<java.lang.String,java.lang.String> options)
protected org.apache.spark.sql.execution.datasources.DDLParser	ddlParser()
protected java.lang.String	dialectClassName()
void	dropTempTable(java.lang.String tableName)
DataFrame	emptyDataFrame() :: Experimental :: Returns a DataFrame with no rows or columns.
protected RDD<org.apache.spark.sql.catalyst.InternalRow>	emptyResult()
protected org.apache.spark.sql.execution.QueryExecution	executePlan(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan)
protected org.apache.spark.sql.execution.QueryExecution	executeSql(java.lang.String sql)
ExperimentalMethods	experimental() :: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.
protected org.apache.spark.sql.catalyst.analysis.FunctionRegistry	functionRegistry()
scala.collection.immutable.Map<java.lang.String,java.lang.String>	getAllConfs() Return all the configuration properties that have been set (i.e.
java.lang.String	getConf(java.lang.String key) Return the value of Spark SQL configuration property for the given key.
java.lang.String	getConf(java.lang.String key, java.lang.String defaultValue) Return the value of Spark SQL configuration property for the given key.
static SQLContext	getOrCreate(SparkContext sparkContext) Get the singleton SQLContext if it exists or create a new one using the given SparkContext.
protected scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.AttributeReference>	getSchema(java.lang.Class<?> beanClass)
protected org.apache.spark.sql.catalyst.ParserDialect	getSQLDialect()
SQLContext.implicits$	implicits() Accessor for nested Scala object
boolean	isCached(java.lang.String tableName) Returns true if the table is currently cached in-memory.
boolean	isRootContext()
DataFrame	jdbc(java.lang.String url, java.lang.String table) Deprecated. As of 1.4.0, replaced by read().jdbc(). This will be removed in Spark 2.0.
DataFrame	jdbc(java.lang.String url, java.lang.String table, java.lang.String[] theParts) Deprecated. As of 1.4.0, replaced by read().jdbc(). This will be removed in Spark 2.0.
DataFrame	jdbc(java.lang.String url, java.lang.String table, java.lang.String columnName, long lowerBound, long upperBound, int numPartitions) Deprecated. As of 1.4.0, replaced by read().jdbc(). This will be removed in Spark 2.0.
DataFrame	jsonFile(java.lang.String path) Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.
DataFrame	jsonFile(java.lang.String path, double samplingRatio) Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.
DataFrame	jsonFile(java.lang.String path, StructType schema) Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.
DataFrame	jsonRDD(JavaRDD<java.lang.String> json) Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.
DataFrame	jsonRDD(JavaRDD<java.lang.String> json, double samplingRatio) Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.
DataFrame	jsonRDD(JavaRDD<java.lang.String> json, StructType schema) Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.
DataFrame	jsonRDD(RDD<java.lang.String> json) Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.
DataFrame	jsonRDD(RDD<java.lang.String> json, double samplingRatio) Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.
DataFrame	jsonRDD(RDD<java.lang.String> json, StructType schema) Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.
org.apache.spark.sql.execution.ui.SQLListener	listener()
ExecutionListenerManager	listenerManager()
DataFrame	load(java.lang.String path) Deprecated. As of 1.4.0, replaced by read().load(path). This will be removed in Spark 2.0.
DataFrame	load(java.lang.String source, java.util.Map<java.lang.String,java.lang.String> options) Deprecated. As of 1.4.0, replaced by read().format(source).options(options).load(). This will be removed in Spark 2.0.
DataFrame	load(java.lang.String source, scala.collection.immutable.Map<java.lang.String,java.lang.String> options) Deprecated. As of 1.4.0, replaced by read().format(source).options(options).load().
DataFrame	load(java.lang.String path, java.lang.String source) Deprecated. As of 1.4.0, replaced by read().format(source).load(path). This will be removed in Spark 2.0.
DataFrame	load(java.lang.String source, StructType schema, java.util.Map<java.lang.String,java.lang.String> options) Deprecated. As of 1.4.0, replaced by read().format(source).schema(schema).options(options).load().
DataFrame	load(java.lang.String source, StructType schema, scala.collection.immutable.Map<java.lang.String,java.lang.String> options) Deprecated. As of 1.4.0, replaced by read().format(source).schema(schema).options(options).load().
SQLContext	newSession() Returns a SQLContext as new session, with separated SQL configurations, temporary tables, registered functions, but sharing the same SparkContext, CacheManager, SQLListener and SQLTab.
protected org.apache.spark.sql.catalyst.optimizer.Optimizer	optimizer()
DataFrame	parquetFile(scala.collection.Seq<java.lang.String> paths)
DataFrame	parquetFile(java.lang.String... paths) Deprecated. As of 1.4.0, replaced by read().parquet(). This will be removed in Spark 2.0.
protected DataType	parseDataType(java.lang.String dataTypeString)
protected org.apache.spark.sql.catalyst.plans.logical.LogicalPlan	parseSql(java.lang.String sql)
protected org.apache.spark.sql.execution.SparkPlanner	planner()
protected org.apache.spark.sql.catalyst.rules.RuleExecutor<org.apache.spark.sql.execution.SparkPlan>	prepareForExecution()
DataFrame	range(long end)
DataFrame	range(long start, long end)
DataFrame	range(long start, long end, long step, int numPartitions)
DataFrameReader	read()
static void	setActive(SQLContext sqlContext) Changes the SQLContext that will be returned in this thread and its children when SQLContext.getOrCreate() is called.
void	setConf(java.util.Properties props) Set Spark SQL configuration properties.
void	setConf(java.lang.String key, java.lang.String value) Set the given Spark SQL configuration property.
SparkContext	sparkContext()
DataFrame	sql(java.lang.String sqlText)
protected org.apache.spark.sql.execution.SparkSQLParser	sqlParser()
DataFrame	table(java.lang.String tableName)
java.lang.String[]	tableNames()
java.lang.String[]	tableNames(java.lang.String databaseName)
DataFrame	tables()
DataFrame	tables(java.lang.String databaseName)
UDFRegistration	udf() A collection of methods for registering user-defined functions (UDF).
void	uncacheTable(java.lang.String tableName) Removes the specified table from the in-memory cache.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

SQLContext

public SQLContext(SparkContext sparkContext)

SQLContext

public SQLContext(JavaSparkContext sparkContext)

Method Detail

getOrCreate

public static SQLContext getOrCreate(SparkContext sparkContext)

Get the singleton SQLContext if it exists or create a new one using the given SparkContext.

This function can be used to create a singleton SQLContext object that can be shared across the JVM.

If there is an active SQLContext for current thread, it will be returned instead of the global one.

Parameters:

sparkContext - (undocumented)

Returns:

(undocumented)

Since:

1.5.0

setActive

public static void setActive(SQLContext sqlContext)

Changes the SQLContext that will be returned in this thread and its children when SQLContext.getOrCreate() is called. This can be used to ensure that a given thread receives a SQLContext with an isolated session, instead of the global (first created) context.

Parameters:

sqlContext - (undocumented)

Since:

1.6.0

clearActive

public static void clearActive()

Clears the active SQLContext for current thread. Subsequent calls to getOrCreate will return the first created context instead of a thread-local override.

Since:

1.6.0

parquetFile

public DataFrame parquetFile(java.lang.String... paths)

Deprecated. As of 1.4.0, replaced by read().parquet(). This will be removed in Spark 2.0.

Loads a Parquet file, returning the result as a DataFrame. This function returns an empty DataFrame if no paths are passed in.

Parameters:

paths - (undocumented)

Returns:

(undocumented)

sparkContext

public SparkContext sparkContext()

cacheManager

protected org.apache.spark.sql.execution.CacheManager cacheManager()

listener

public org.apache.spark.sql.execution.ui.SQLListener listener()

isRootContext

public boolean isRootContext()

newSession

public SQLContext newSession()

Returns a SQLContext as new session, with separated SQL configurations, temporary tables, registered functions, but sharing the same SparkContext, CacheManager, SQLListener and SQLTab.

Returns:

(undocumented)

Since:

1.6.0

conf

protected org.apache.spark.sql.SQLConf conf()

Returns:

Spark SQL configuration

setConf

public void setConf(java.util.Properties props)

Set Spark SQL configuration properties.

Parameters:

props - (undocumented)

Since:

1.0.0

setConf

public void setConf(java.lang.String key,
           java.lang.String value)

Set the given Spark SQL configuration property.

Parameters:

key - (undocumented)

value - (undocumented)

Since:

1.0.0

getConf

public java.lang.String getConf(java.lang.String key)

Return the value of Spark SQL configuration property for the given key.

Parameters:

key - (undocumented)

Returns:

(undocumented)

Since:

1.0.0

getConf

public java.lang.String getConf(java.lang.String key,
                       java.lang.String defaultValue)

Return the value of Spark SQL configuration property for the given key. If the key is not set yet, return defaultValue.

Parameters:

key - (undocumented)

defaultValue - (undocumented)

Returns:

(undocumented)

Since:

1.0.0

getAllConfs

public scala.collection.immutable.Map<java.lang.String,java.lang.String> getAllConfs()

Return all the configuration properties that have been set (i.e. not the default). This creates a new copy of the config properties in the form of a Map.

Returns:

(undocumented)

Since:

1.0.0

listenerManager

public ExecutionListenerManager listenerManager()

catalog

protected org.apache.spark.sql.catalyst.analysis.Catalog catalog()

functionRegistry

protected org.apache.spark.sql.catalyst.analysis.FunctionRegistry functionRegistry()

analyzer

protected org.apache.spark.sql.catalyst.analysis.Analyzer analyzer()

optimizer

protected org.apache.spark.sql.catalyst.optimizer.Optimizer optimizer()

ddlParser

protected org.apache.spark.sql.execution.datasources.DDLParser ddlParser()

sqlParser

protected org.apache.spark.sql.execution.SparkSQLParser sqlParser()

getSQLDialect

protected org.apache.spark.sql.catalyst.ParserDialect getSQLDialect()

parseSql

protected org.apache.spark.sql.catalyst.plans.logical.LogicalPlan parseSql(java.lang.String sql)

executeSql

protected org.apache.spark.sql.execution.QueryExecution executeSql(java.lang.String sql)

executePlan

protected org.apache.spark.sql.execution.QueryExecution executePlan(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan)

dialectClassName

protected java.lang.String dialectClassName()

addJar

protected void addJar(java.lang.String path)

Add a jar to SQLContext

Parameters:

path - (undocumented)

experimental

public ExperimentalMethods experimental()

:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.

Returns:

(undocumented)

Since:

1.3.0

emptyDataFrame

public DataFrame emptyDataFrame()

:: Experimental :: Returns a DataFrame with no rows or columns.

Returns:

(undocumented)

Since:

1.3.0

udf

public UDFRegistration udf()

A collection of methods for registering user-defined functions (UDF).

The following example registers a Scala closure as UDF:

   sqlContext.udf.register("myUDF", (arg1: Int, arg2: String) => arg2 + arg1)
 

The following example registers a UDF in Java:

   sqlContext.udf().register("myUDF",
       new UDF2<Integer, String, String>() {
           @Override
           public String call(Integer arg1, String arg2) {
               return arg2 + arg1;
           }
      }, DataTypes.StringType);
 

Or, to use Java 8 lambda syntax:

   sqlContext.udf().register("myUDF",
       (Integer arg1, String arg2) -> arg2 + arg1,
       DataTypes.StringType);
 

Returns:

(undocumented)

Since:

1.3.0 TODO move to SQLSession?

isCached

public boolean isCached(java.lang.String tableName)

Returns true if the table is currently cached in-memory.

Parameters:

tableName - (undocumented)

Returns:

(undocumented)

Since:

1.3.0

cacheTable

public void cacheTable(java.lang.String tableName)

Caches the specified table in-memory.

Parameters:

tableName - (undocumented)

Since:

1.3.0

uncacheTable

public void uncacheTable(java.lang.String tableName)

Removes the specified table from the in-memory cache.

Parameters:

tableName - (undocumented)

Since:

1.3.0

clearCache

public void clearCache()

Removes all cached tables from the in-memory cache.

Since:

1.3.0

implicits

public SQLContext.implicits$ implicits()

Accessor for nested Scala object

Returns:

(undocumented)

createDataFrame

public <A extends scala.Product> DataFrame createDataFrame(RDD<A> rdd,
                                                  scala.reflect.api.TypeTags.TypeTag<A> evidence$1)

createDataFrame

public <A extends scala.Product> DataFrame createDataFrame(scala.collection.Seq<A> data,
                                                  scala.reflect.api.TypeTags.TypeTag<A> evidence$2)

baseRelationToDataFrame

public DataFrame baseRelationToDataFrame(BaseRelation baseRelation)

createDataFrame

public DataFrame createDataFrame(RDD<Row> rowRDD,
                        StructType schema)

createDataset

public <T> Dataset<T> createDataset(scala.collection.Seq<T> data,
                           Encoder<T> evidence$3)

createDataset

public <T> Dataset<T> createDataset(RDD<T> data,
                           Encoder<T> evidence$4)

createDataset

public <T> Dataset<T> createDataset(java.util.List<T> data,
                           Encoder<T> evidence$5)

createDataFrame

public DataFrame createDataFrame(JavaRDD<Row> rowRDD,
                        StructType schema)

createDataFrame

public DataFrame createDataFrame(java.util.List<Row> rows,
                        StructType schema)

createDataFrame

public DataFrame createDataFrame(RDD<?> rdd,
                        java.lang.Class<?> beanClass)

createDataFrame

public DataFrame createDataFrame(JavaRDD<?> rdd,
                        java.lang.Class<?> beanClass)

createDataFrame

public DataFrame createDataFrame(java.util.List<?> data,
                        java.lang.Class<?> beanClass)

read

public DataFrameReader read()

createExternalTable

public DataFrame createExternalTable(java.lang.String tableName,
                            java.lang.String path)

createExternalTable

public DataFrame createExternalTable(java.lang.String tableName,
                            java.lang.String path,
                            java.lang.String source)

createExternalTable

public DataFrame createExternalTable(java.lang.String tableName,
                            java.lang.String source,
                            java.util.Map<java.lang.String,java.lang.String> options)

createExternalTable

public DataFrame createExternalTable(java.lang.String tableName,
                            java.lang.String source,
                            scala.collection.immutable.Map<java.lang.String,java.lang.String> options)

createExternalTable

public DataFrame createExternalTable(java.lang.String tableName,
                            java.lang.String source,
                            StructType schema,
                            java.util.Map<java.lang.String,java.lang.String> options)

createExternalTable

public DataFrame createExternalTable(java.lang.String tableName,
                            java.lang.String source,
                            StructType schema,
                            scala.collection.immutable.Map<java.lang.String,java.lang.String> options)

dropTempTable

public void dropTempTable(java.lang.String tableName)

range

public DataFrame range(long end)

range

public DataFrame range(long start,
              long end)

range

public DataFrame range(long start,
              long end,
              long step,
              int numPartitions)

sql

public DataFrame sql(java.lang.String sqlText)

table

public DataFrame table(java.lang.String tableName)

tables

public DataFrame tables()

tables

public DataFrame tables(java.lang.String databaseName)

tableNames

public java.lang.String[] tableNames()

tableNames

public java.lang.String[] tableNames(java.lang.String databaseName)

planner

protected org.apache.spark.sql.execution.SparkPlanner planner()

emptyResult

protected RDD<org.apache.spark.sql.catalyst.InternalRow> emptyResult()

prepareForExecution

protected org.apache.spark.sql.catalyst.rules.RuleExecutor<org.apache.spark.sql.execution.SparkPlan> prepareForExecution()

parseDataType

protected DataType parseDataType(java.lang.String dataTypeString)

applySchemaToPythonRDD

protected DataFrame applySchemaToPythonRDD(RDD<java.lang.Object[]> rdd,
                               java.lang.String schemaString)

applySchemaToPythonRDD

protected DataFrame applySchemaToPythonRDD(RDD<java.lang.Object[]> rdd,
                               StructType schema)

getSchema

protected scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.AttributeReference> getSchema(java.lang.Class<?> beanClass)

applySchema

public DataFrame applySchema(RDD<Row> rowRDD,
                    StructType schema)

applySchema

public DataFrame applySchema(JavaRDD<Row> rowRDD,
                    StructType schema)

applySchema

public DataFrame applySchema(RDD<?> rdd,
                    java.lang.Class<?> beanClass)

applySchema

public DataFrame applySchema(JavaRDD<?> rdd,
                    java.lang.Class<?> beanClass)

parquetFile

public DataFrame parquetFile(scala.collection.Seq<java.lang.String> paths)

jsonFile

public DataFrame jsonFile(java.lang.String path)

Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.

Loads a JSON file (one object per line), returning the result as a DataFrame. It goes through the entire dataset once to determine the schema.

Parameters:

path - (undocumented)

Returns:

(undocumented)

jsonFile

public DataFrame jsonFile(java.lang.String path,
                 StructType schema)

Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.

Loads a JSON file (one object per line) and applies the given schema, returning the result as a DataFrame.

Parameters:

path - (undocumented)

schema - (undocumented)

Returns:

(undocumented)

jsonFile

public DataFrame jsonFile(java.lang.String path,
                 double samplingRatio)

Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.

Parameters:

path - (undocumented)

samplingRatio - (undocumented)

Returns:

(undocumented)

jsonRDD

public DataFrame jsonRDD(RDD<java.lang.String> json)

Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.

Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame. It goes through the entire dataset once to determine the schema.

Parameters:

json - (undocumented)

Returns:

(undocumented)

jsonRDD

public DataFrame jsonRDD(JavaRDD<java.lang.String> json)

Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.

Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame. It goes through the entire dataset once to determine the schema.

Parameters:

json - (undocumented)

Returns:

(undocumented)

jsonRDD

public DataFrame jsonRDD(RDD<java.lang.String> json,
                StructType schema)

Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.

Loads an RDD[String] storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.

Parameters:

json - (undocumented)

schema - (undocumented)

Returns:

(undocumented)

jsonRDD

public DataFrame jsonRDD(JavaRDD<java.lang.String> json,
                StructType schema)

Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.

Loads an JavaRDD storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.

Parameters:

json - (undocumented)

schema - (undocumented)

Returns:

(undocumented)

jsonRDD

public DataFrame jsonRDD(RDD<java.lang.String> json,
                double samplingRatio)

Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.

Loads an RDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.

Parameters:

json - (undocumented)

samplingRatio - (undocumented)

Returns:

(undocumented)

jsonRDD

public DataFrame jsonRDD(JavaRDD<java.lang.String> json,
                double samplingRatio)

Deprecated. As of 1.4.0, replaced by read().json(). This will be removed in Spark 2.0.

Loads a JavaRDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.

Parameters:

json - (undocumented)

samplingRatio - (undocumented)

Returns:

(undocumented)

load

public DataFrame load(java.lang.String path)

Deprecated. As of 1.4.0, replaced by read().load(path). This will be removed in Spark 2.0.

Returns the dataset stored at path as a DataFrame, using the default data source configured by spark.sql.sources.default.

Parameters:

path - (undocumented)

Returns:

(undocumented)

load

public DataFrame load(java.lang.String path,
             java.lang.String source)

Deprecated. As of 1.4.0, replaced by read().format(source).load(path). This will be removed in Spark 2.0.

Returns the dataset stored at path as a DataFrame, using the given data source.

Parameters:

path - (undocumented)

source - (undocumented)

Returns:

(undocumented)

load

public DataFrame load(java.lang.String source,
             java.util.Map<java.lang.String,java.lang.String> options)

Deprecated. As of 1.4.0, replaced by read().format(source).options(options).load(). This will be removed in Spark 2.0.

(Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.

Parameters:

source - (undocumented)

options - (undocumented)

Returns:

(undocumented)

load

public DataFrame load(java.lang.String source,
             scala.collection.immutable.Map<java.lang.String,java.lang.String> options)

Deprecated. As of 1.4.0, replaced by read().format(source).options(options).load().

(Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.

Parameters:

source - (undocumented)

options - (undocumented)

Returns:

(undocumented)

load

public DataFrame load(java.lang.String source,
             StructType schema,
             java.util.Map<java.lang.String,java.lang.String> options)

Deprecated. As of 1.4.0, replaced by read().format(source).schema(schema).options(options).load().

(Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.

Parameters:

source - (undocumented)

schema - (undocumented)

options - (undocumented)

Returns:

(undocumented)

load

public DataFrame load(java.lang.String source,
             StructType schema,
             scala.collection.immutable.Map<java.lang.String,java.lang.String> options)

Deprecated. As of 1.4.0, replaced by read().format(source).schema(schema).options(options).load().

(Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.

Parameters:

source - (undocumented)

schema - (undocumented)

options - (undocumented)

Returns:

(undocumented)

jdbc

public DataFrame jdbc(java.lang.String url,
             java.lang.String table)

Deprecated. As of 1.4.0, replaced by read().jdbc(). This will be removed in Spark 2.0.

Construct a DataFrame representing the database table accessible via JDBC URL url named table.

Parameters:

url - (undocumented)

table - (undocumented)

Returns:

(undocumented)

jdbc

public DataFrame jdbc(java.lang.String url,
             java.lang.String table,
             java.lang.String columnName,
             long lowerBound,
             long upperBound,
             int numPartitions)

Deprecated. As of 1.4.0, replaced by read().jdbc(). This will be removed in Spark 2.0.

Construct a DataFrame representing the database table accessible via JDBC URL url named table. Partitions of the table will be retrieved in parallel based on the parameters passed to this function.

Parameters:

columnName - the name of a column of integral type that will be used for partitioning.

lowerBound - the minimum value of columnName used to decide partition stride

upperBound - the maximum value of columnName used to decide partition stride

numPartitions - the number of partitions. the range minValue-maxValue will be split evenly into this many partitions

url - (undocumented)

table - (undocumented)

Returns:

(undocumented)

jdbc

public DataFrame jdbc(java.lang.String url,
             java.lang.String table,
             java.lang.String[] theParts)

Deprecated. As of 1.4.0, replaced by read().jdbc(). This will be removed in Spark 2.0.

Construct a DataFrame representing the database table accessible via JDBC URL url named table. The theParts parameter gives a list expressions suitable for inclusion in WHERE clauses; each one defines one partition of the DataFrame.

Parameters:

url - (undocumented)

table - (undocumented)

theParts - (undocumented)

Returns:

(undocumented)