rikai.spark.types package¶

Submodules¶

rikai.spark.types.geometry module¶

Geometry User Defined Types in Spark

class rikai.spark.types.geometry.Box2dType¶

Bases: UserDefinedType

User defined type for the 2D bounding box.

deserialize(datum: Row) → rikai.types.geometry.Box2d¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(obj: rikai.types.geometry.Box2d)¶: Serialize a Box2d into a PySpark Row

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.

class rikai.spark.types.geometry.Box3dType¶

Bases: UserDefinedType

Spark UDT for Box3d class.

deserialize(datum: Row) → Box3d¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(obj: Box3d)¶: Serialize an Box3d into a Spark Row

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.

class rikai.spark.types.geometry.MaskType¶

Bases: UserDefinedType

Spark UDT for 2-D mask

deserialize(datum: Row) → Mask¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(mask: Mask) → Row¶: Converts a user-type object into a SQL datum.

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.

class rikai.spark.types.geometry.PointType¶

Bases: UserDefinedType

Spark UDT for rikai.types.geometry.Point class.

deserialize(datum: Row) → Point¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(obj: Point)¶: Serialize an PointType into Spark Row

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.

rikai.spark.types.video module¶

class rikai.spark.types.video.VideoStreamType¶

Bases: UserDefinedType

VideoStreamType defines the Spark UserDefineType for a given video stream

deserialize(datum) → VideoStream¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(obj: VideoStream)¶: Serialize a VideoStream to a Spark Row

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.

class rikai.spark.types.video.YouTubeVideoType¶

Bases: UserDefinedType

YouTubeVideoType defines the Spark UserDefineType for a piece of YouTube video content (i.e., corresponds to a given youtube id but can have multiple streams)

deserialize(datum) → YouTubeVideo¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(obj: YouTubeVideo)¶: Serialize a YouTubeVideo

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.

rikai.spark.types.vision module¶

class rikai.spark.types.vision.ImageType¶

Bases: UserDefinedType

ImageType defines the Spark UserDefineType for Image type

deserialize(datum) → Image¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(obj: Image)¶: Serialize an Image to a Spark Row?

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.

Module contents¶

Spark User Defined Types

class rikai.spark.types.Box2dType¶

Bases: UserDefinedType

User defined type for the 2D bounding box.

deserialize(datum: Row) → rikai.types.geometry.Box2d¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(obj: rikai.types.geometry.Box2d)¶: Serialize a Box2d into a PySpark Row

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.

class rikai.spark.types.Box3dType¶

Bases: UserDefinedType

Spark UDT for Box3d class.

deserialize(datum: Row) → Box3d¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(obj: Box3d)¶: Serialize an Box3d into a Spark Row

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.

class rikai.spark.types.ImageType¶

Bases: UserDefinedType

ImageType defines the Spark UserDefineType for Image type

deserialize(datum) → Image¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(obj: Image)¶: Serialize an Image to a Spark Row?

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.

class rikai.spark.types.MaskType¶

Bases: UserDefinedType

Spark UDT for 2-D mask

deserialize(datum: Row) → Mask¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(mask: Mask) → Row¶: Converts a user-type object into a SQL datum.

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.

class rikai.spark.types.NDArrayType¶

Bases: UserDefinedType

User define type for an arbitrary numpy.ndarray.

This UDT serializes numpy.ndarray into bytes buffer.

deserialize(datum: Row) → ndarray¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(obj: ndarray)¶: Serialize an numpy.ndarray into Spark Row

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.

class rikai.spark.types.PointType¶

Bases: UserDefinedType

Spark UDT for rikai.types.geometry.Point class.

deserialize(datum: Row) → Point¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(obj: Point)¶: Serialize an PointType into Spark Row

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.

class rikai.spark.types.SegmentType¶

Bases: UserDefinedType

Defines the Spark UDT for a video segment A video segment is defined by the starting frame number (start_fno) and ending frame number (end_fno). fno terminology is chosen to be consistent with the opencv library.

deserialize(datum) → Segment¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(obj: Segment)¶: Converts a user-type object into a SQL datum.

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.

class rikai.spark.types.VideoStreamType¶

Bases: UserDefinedType

VideoStreamType defines the Spark UserDefineType for a given video stream

deserialize(datum) → VideoStream¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(obj: VideoStream)¶: Serialize a VideoStream to a Spark Row

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.

class rikai.spark.types.YouTubeVideoType¶

Bases: UserDefinedType

YouTubeVideoType defines the Spark UserDefineType for a piece of YouTube video content (i.e., corresponds to a given youtube id but can have multiple streams)

deserialize(datum) → YouTubeVideo¶: Converts a SQL datum into a user-type object.

classmethod module() → str¶: The Python module of the UDT.

classmethod scalaUDT() → str¶: The class name of the paired Scala UDT (could be ‘’, if there is no corresponding one).

serialize(obj: YouTubeVideo)¶: Serialize a YouTubeVideo

simpleString() → str¶

classmethod sqlType() → StructType¶: Underlying SQL storage type for this UDT.