Class

org.apache.spark.sql.catalog

Catalog

Related Doc: package catalog

Permalink

abstract class Catalog extends AnyRef

Catalog interface for Spark. To access this, use SparkSession.catalog.

Source
Catalog.scala
Since

2.0.0

Linear Supertypes
AnyRef, Any
Known Subclasses
CatalogImpl
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Catalog
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Catalog()

    Permalink

Abstract Value Members

  1. abstract def cacheTable(tableName: String): Unit

    Permalink

    Caches the specified table in-memory.

    Caches the specified table in-memory.

    Since

    2.0.0

  2. abstract def clearCache(): Unit

    Permalink

    Removes all cached tables from the in-memory cache.

    Removes all cached tables from the in-memory cache.

    Since

    2.0.0

  3. abstract def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

    Permalink

    :: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options.

    :: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    Annotations
    @Experimental()
    Since

    2.0.0

  4. abstract def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

    Permalink

    :: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options.

    :: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    Annotations
    @Experimental()
    Since

    2.0.0

  5. abstract def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

    Permalink

    :: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options.

    :: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

    Annotations
    @Experimental()
    Since

    2.0.0

  6. abstract def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

    Permalink

    :: Experimental :: Creates an external table from the given path based on a data source and a set of options.

    :: Experimental :: Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

    Annotations
    @Experimental()
    Since

    2.0.0

  7. abstract def createExternalTable(tableName: String, path: String, source: String): DataFrame

    Permalink

    :: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.

    :: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.

    Annotations
    @Experimental()
    Since

    2.0.0

  8. abstract def createExternalTable(tableName: String, path: String): DataFrame

    Permalink

    :: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame.

    :: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.

    Annotations
    @Experimental()
    Since

    2.0.0

  9. abstract def currentDatabase: String

    Permalink

    Returns the current default database in this session.

    Returns the current default database in this session.

    Since

    2.0.0

  10. abstract def dropTempView(viewName: String): Unit

    Permalink

    Drops the temporary view with the given view name in the catalog.

    Drops the temporary view with the given view name in the catalog. If the view has been cached before, then it will also be uncached.

    viewName

    the name of the view to be dropped.

    Since

    2.0.0

  11. abstract def isCached(tableName: String): Boolean

    Permalink

    Returns true if the table is currently cached in-memory.

    Returns true if the table is currently cached in-memory.

    Since

    2.0.0

  12. abstract def listColumns(dbName: String, tableName: String): Dataset[Column]

    Permalink

    Returns a list of columns for the given table in the specified database.

    Returns a list of columns for the given table in the specified database.

    Annotations
    @throws( "database or table does not exist" )
    Since

    2.0.0

  13. abstract def listColumns(tableName: String): Dataset[Column]

    Permalink

    Returns a list of columns for the given table in the current database or the given temporary table.

    Returns a list of columns for the given table in the current database or the given temporary table.

    Annotations
    @throws( "table does not exist" )
    Since

    2.0.0

  14. abstract def listDatabases(): Dataset[Database]

    Permalink

    Returns a list of databases available across all sessions.

    Returns a list of databases available across all sessions.

    Since

    2.0.0

  15. abstract def listFunctions(dbName: String): Dataset[Function]

    Permalink

    Returns a list of functions registered in the specified database.

    Returns a list of functions registered in the specified database. This includes all temporary functions

    Annotations
    @throws( "database does not exist" )
    Since

    2.0.0

  16. abstract def listFunctions(): Dataset[Function]

    Permalink

    Returns a list of functions registered in the current database.

    Returns a list of functions registered in the current database. This includes all temporary functions

    Since

    2.0.0

  17. abstract def listTables(dbName: String): Dataset[Table]

    Permalink

    Returns a list of tables in the specified database.

    Returns a list of tables in the specified database. This includes all temporary tables.

    Annotations
    @throws( "database does not exist" )
    Since

    2.0.0

  18. abstract def listTables(): Dataset[Table]

    Permalink

    Returns a list of tables in the current database.

    Returns a list of tables in the current database. This includes all temporary tables.

    Since

    2.0.0

  19. abstract def refreshByPath(path: String): Unit

    Permalink

    Invalidate and refresh all the cached data (and the associated metadata) for any dataframe that contains the given data source path.

    Invalidate and refresh all the cached data (and the associated metadata) for any dataframe that contains the given data source path.

    Since

    2.0.0

  20. abstract def refreshTable(tableName: String): Unit

    Permalink

    Invalidate and refresh all the cached metadata of the given table.

    Invalidate and refresh all the cached metadata of the given table. For performance reasons, Spark SQL or the external data source library it uses might cache certain metadata about a table, such as the location of blocks. When those change outside of Spark SQL, users should call this function to invalidate the cache.

    If this table is cached as an InMemoryRelation, drop the original cached version and make the new version cached lazily.

    Since

    2.0.0

  21. abstract def setCurrentDatabase(dbName: String): Unit

    Permalink

    Sets the current default database in this session.

    Sets the current default database in this session.

    Since

    2.0.0

  22. abstract def uncacheTable(tableName: String): Unit

    Permalink

    Removes the specified table from the in-memory cache.

    Removes the specified table from the in-memory cache.

    Since

    2.0.0

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  6. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  7. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  8. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  9. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  10. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  11. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  12. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  13. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  14. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  15. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  16. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  17. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  18. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  19. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped