rikai.spark.sql package¶
Subpackages¶
- rikai.spark.sql.codegen package
- Submodules
- rikai.spark.sql.codegen.base module
- rikai.spark.sql.codegen.dummy module
- rikai.spark.sql.codegen.fs module
- rikai.spark.sql.codegen.mlflow_logger module
- rikai.spark.sql.codegen.mlflow_registry module
- rikai.spark.sql.codegen.pytorch module
- rikai.spark.sql.codegen.sklearn module
- rikai.spark.sql.codegen.tensorflow module
- rikai.spark.sql.codegen.testing module
- Module contents
- rikai.spark.sql.generated package
- Submodules
- rikai.spark.sql.generated.RikaiModelSchemaLexer module
RikaiModelSchemaLexer
RikaiModelSchemaLexer.ARRAY
RikaiModelSchemaLexer.IDENTIFIER
RikaiModelSchemaLexer.STRUCT
RikaiModelSchemaLexer.T__0
RikaiModelSchemaLexer.T__1
RikaiModelSchemaLexer.T__2
RikaiModelSchemaLexer.T__3
RikaiModelSchemaLexer.WS
RikaiModelSchemaLexer.atn
RikaiModelSchemaLexer.channelNames
RikaiModelSchemaLexer.decisionsToDFA
RikaiModelSchemaLexer.grammarFileName
RikaiModelSchemaLexer.literalNames
RikaiModelSchemaLexer.modeNames
RikaiModelSchemaLexer.ruleNames
RikaiModelSchemaLexer.symbolicNames
serializedATN()
- rikai.spark.sql.generated.RikaiModelSchemaParser module
RikaiModelSchemaParser
RikaiModelSchemaParser.ARRAY
RikaiModelSchemaParser.ArrayContext
RikaiModelSchemaParser.ArrayTypeContext
RikaiModelSchemaParser.EOF
RikaiModelSchemaParser.FieldContext
RikaiModelSchemaParser.FieldTypeContext
RikaiModelSchemaParser.IDENTIFIER
RikaiModelSchemaParser.IdentifierContext
RikaiModelSchemaParser.NestedArrayTypeContext
RikaiModelSchemaParser.NestedStructTypeContext
RikaiModelSchemaParser.PlainFieldTypeContext
RikaiModelSchemaParser.RULE_array
RikaiModelSchemaParser.RULE_field
RikaiModelSchemaParser.RULE_fieldType
RikaiModelSchemaParser.RULE_identifier
RikaiModelSchemaParser.RULE_schema
RikaiModelSchemaParser.RULE_struct
RikaiModelSchemaParser.STRUCT
RikaiModelSchemaParser.SchemaContext
RikaiModelSchemaParser.StructContext
RikaiModelSchemaParser.StructFieldContext
RikaiModelSchemaParser.StructTypeContext
RikaiModelSchemaParser.T__0
RikaiModelSchemaParser.T__1
RikaiModelSchemaParser.T__2
RikaiModelSchemaParser.T__3
RikaiModelSchemaParser.UnquotedIdentifierContext
RikaiModelSchemaParser.WS
RikaiModelSchemaParser.array()
RikaiModelSchemaParser.atn
RikaiModelSchemaParser.buildParseTrees
RikaiModelSchemaParser.decisionsToDFA
RikaiModelSchemaParser.field()
RikaiModelSchemaParser.fieldType()
RikaiModelSchemaParser.grammarFileName
RikaiModelSchemaParser.identifier()
RikaiModelSchemaParser.literalNames
RikaiModelSchemaParser.ruleNames
RikaiModelSchemaParser.schema()
RikaiModelSchemaParser.sharedContextCache
RikaiModelSchemaParser.struct()
RikaiModelSchemaParser.symbolicNames
serializedATN()
- rikai.spark.sql.generated.RikaiModelSchemaVisitor module
RikaiModelSchemaVisitor
RikaiModelSchemaVisitor.visitArrayType()
RikaiModelSchemaVisitor.visitNestedArrayType()
RikaiModelSchemaVisitor.visitNestedStructType()
RikaiModelSchemaVisitor.visitPlainFieldType()
RikaiModelSchemaVisitor.visitSchema()
RikaiModelSchemaVisitor.visitStructField()
RikaiModelSchemaVisitor.visitStructType()
RikaiModelSchemaVisitor.visitUnquotedIdentifier()
- Module contents
Submodules¶
rikai.spark.sql.exceptions module¶
rikai.spark.sql.model module¶
- class rikai.spark.sql.model.ModelSpec(spec: Dict[str, Any], validate: bool = True)¶
Bases:
ABC
Model Spec Payload
- Parameters
- validate(schema={'properties': {'model': {'description': 'model description', 'properties': {'flavor': {'type': 'string'}, 'type': {'type': 'string'}, 'uri': {'type': 'string'}}, 'required': ['uri'], 'type': 'object'}, 'name': {'description': 'Model name', 'type': 'string'}, 'schema': {'type': 'string'}, 'version': {'description': 'Model SPEC format version', 'type': 'string'}}, 'required': ['version', 'model'], 'type': 'object'})¶
Validate model spec
- Raises
SpecError – If the spec is not well-formatted.
- class rikai.spark.sql.model.ModelType¶
Bases:
ABC
Base-class for a Model Type.
A Model Type defines the functionalities which is required to run an arbitrary ML models in SQL ML, including:
Result schema:
schema()
.transform()
, pre-processing routine.predict()
, inference AND post-processing routine.
- dataType() pyspark.sql.types.DataType ¶
Returns schema as
pyspark.sql.types.DataType
.
- abstract predict(*args, **kwargs) Any ¶
Run model inference and convert return types into Rikai-compatible types.
- release()¶
Release underneath resources if applicable.
It will be called after a model runner finishes a partition in Spark.
rikai.spark.sql.schema module¶
- class rikai.spark.sql.schema.SparkDataTypeVisitor¶
Bases:
RikaiModelSchemaVisitor
- visitArrayType(ctx: ArrayTypeContext) ArrayType ¶
- visitPlainFieldType(ctx: PlainFieldTypeContext) DataType ¶
- visitStructField(ctx: StructFieldContext) StructField ¶
- visitStructType(ctx: StructTypeContext) StructType ¶
- visitUnquotedIdentifier(ctx: UnquotedIdentifierContext) str ¶
- rikai.spark.sql.schema.parse_schema(schema_str: str, visitor: ~rikai.spark.sql.generated.RikaiModelSchemaVisitor.RikaiModelSchemaVisitor = <rikai.spark.sql.schema.SparkDataTypeVisitor object>)¶
Parse schema and returns the data type for the runtime