Convert a SQL datum to the user type
Convert the user type to a SQL datum
Convert the user type to a SQL datum
TODO: Can we make this take obj: UserType? The issue is in CatalystTypeConverters.convertToCatalyst, where we need to convert Any to UserType.
Underlying storage type for this UDT
Class object for the UserType
The default size of a value of the UserDefinedType is 4096 bytes.
The default size of a value of the UserDefinedType is 4096 bytes.
The compact JSON representation of this data type.
The compact JSON representation of this data type.
The pretty (i.
The pretty (i.e. indented) JSON representation of this data type.
Paired Python UDT class, if exists.
Serialized Python UDT class, if exists.
Readable string representation for the type.
Readable string representation for the type.
Name of the type used in JSON serialization.
Name of the type used in JSON serialization.
::DeveloperApi:: The data type for User Defined Types (UDTs).
This interface allows a user to make their own classes more interoperable with SparkSQL; e.g., by creating a UserDefinedType for a class X, it becomes possible to create a
DataFrame
which has class X in the schema.For SparkSQL to recognize UDTs, the UDT must be annotated with SQLUserDefinedType.
The conversion via
serialize
occurs when instantiating aDataFrame
from another RDD. The conversion viadeserialize
occurs when reading from aDataFrame
.