> ## Documentation Index > Fetch the complete documentation index at: https://docs.truestate.io/llms.txt > Use this file to discover all available pages before exploring further. # Release Pipeline ## OpenAPI ````yaml https://api.truestate.io/openapi.json post /pipelines/{pipeline_id}/release/ openapi: 3.1.0 info: title: TrueState Backend API summary: >- TrueState is an AI platform for business users to create high-impact AI solutions. description: > This **[TrueState](https://www.truestate.io/)** backend API allows our clients to automate and more powerfully retrieve and modify data from the TrueState platform. Example use cases include: - Accessing embeddings for a dataset. - Automating the process of dataset creation and model retraining as your data changes. - Much more. Our backend is built using the [FastAPI](https://fastapi.tiangolo.com/) framework in python using [Auth0](https://auth0.com/) and [Client API Keys](https://www.truestate.io/dawn/docs) for internal an external authentication, respectively. Our backend is designed to be secure, scalable, easy to use and well documented _(each router with its own linked docs page)_. Clients can only access data from their own organisation, as setup in Auth0 our the Client API Keys. See how **[here](https://www.truestate.io/dawn/docs)**. **PS**: You can view this API documentation in either **[Swagger](swagger/docs)** or **[Redoc](redoc/)**, but only the swagger lets you test calls. contact: name: True State Support url: https://www.truestate.io/dawn/docs email: support@truestate.io license: name: Private url: https://www.apache.org/licenses/LICENSE-2.0.html version: 0.13.95 servers: [] security: [] paths: /pipelines/{pipeline_id}/release/: post: tags: - pipelines summary: Release Pipeline operationId: release_pipeline_pipelines__pipeline_id__release__post parameters: - name: pipeline_id in: path required: true schema: type: string format: uuid title: Pipeline Id - name: Api-key in: header required: false schema: anyOf: - type: string - type: 'null' title: Api-Key requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ReleasePipelineRequest' responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/PipelineRead' '404': description: Not found '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' security: - HTTPBearer: [] components: schemas: ReleasePipelineRequest: properties: version: type: string title: Version description: type: string title: Description overwrite_secrets: type: boolean title: Overwrite Secrets datasets_to_overwrite: items: type: string type: array title: Datasets To Overwrite type: object required: - version - description - overwrite_secrets - datasets_to_overwrite title: ReleasePipelineRequest PipelineRead: properties: name: type: string title: Name description: type: string title: Description id: type: string format: uuid title: Id organisation_id: type: string title: Organisation Id current_version: anyOf: - $ref: '#/components/schemas/PipelineVersionRead' - type: 'null' reference_notes: anyOf: - type: string - type: 'null' title: Reference Notes schedule_enabled: anyOf: - type: boolean - type: 'null' title: Schedule Enabled cron_schedule: anyOf: - type: string - type: 'null' title: Cron Schedule created_at: type: string format: date-time title: Created At updated_at: type: string format: date-time title: Updated At notification_settings: anyOf: - items: oneOf: - $ref: '#/components/schemas/EmailNotification' - $ref: '#/components/schemas/SlackNotification' discriminator: propertyName: method mapping: EMAIL: $ref: '#/components/schemas/EmailNotification' SLACK: $ref: '#/components/schemas/SlackNotification' type: array - type: 'null' title: Notification Settings lower_env_pipeline_id: anyOf: - type: string format: uuid - type: 'null' title: Lower Env Pipeline Id higher_env_pipeline_id: anyOf: - type: string format: uuid - type: 'null' title: Higher Env Pipeline Id latest_run_status: anyOf: - type: string - type: 'null' title: Latest Run Status description: >- Latest run status for the pipeline's current version (list endpoint; detail uses pipeline_runs). type: object required: - name - description - id - organisation_id - cron_schedule - created_at - updated_at - notification_settings - lower_env_pipeline_id - higher_env_pipeline_id title: PipelineRead HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError PipelineVersionRead: properties: id: type: string format: uuid title: Id pipeline_id: type: string format: uuid title: Pipeline Id steps: items: $ref: '#/components/schemas/PipelineStepRead' type: array title: Steps change_description: anyOf: - type: string - type: 'null' title: Change Description checkpoint: anyOf: - $ref: '#/components/schemas/PipelineVersionCheckpointRead' - type: 'null' release: anyOf: - $ref: '#/components/schemas/PipelineReleaseRead' - type: 'null' created_at: type: string format: date-time title: Created At created_by: anyOf: - $ref: '#/components/schemas/UserRead' - type: 'null' annotations: anyOf: - $ref: '#/components/schemas/PipelineVersionAnnotations' - type: 'null' type: object required: - id - pipeline_id - steps - release - created_at - created_by - annotations title: PipelineVersionRead EmailNotification: properties: method: type: string const: EMAIL title: Method default: EMAIL user_ids: items: type: string type: array title: User Ids member_ids: items: type: string type: array title: Member Ids type: object title: EmailNotification SlackNotification: properties: method: type: string const: SLACK title: Method default: SLACK channel_id: type: string title: Channel Id bot_token_secret_name: type: string title: Bot Token Secret Name type: object required: - channel_id - bot_token_secret_name title: SlackNotification ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type input: title: Input ctx: type: object title: Context type: object required: - loc - msg - type title: ValidationError PipelineStepRead: properties: id: type: string format: uuid title: Id name: type: string title: Name config: anyOf: - $ref: '#/components/schemas/DataTransformConfig' - $ref: '#/components/schemas/EmbeddingConfig' - $ref: '#/components/schemas/BulkPromptConfig' - $ref: '#/components/schemas/ModelInferenceConfig' - $ref: '#/components/schemas/ModelEvaluationConfig' - $ref: '#/components/schemas/ModelExplanationConfig' - $ref: '#/components/schemas/ClusteringTrainingConfig' - $ref: '#/components/schemas/TabularRegressorTrainingConfigV2' - $ref: '#/components/schemas/EnsembleRegressorTrainingConfigV2-Output' - $ref: '#/components/schemas/TabularClassifierTrainingConfigV2-Output' - $ref: '#/components/schemas/EnsembleClassifierTrainingConfigV2-Output' - $ref: '#/components/schemas/LinearRegressorTrainingConfigV2' - $ref: '#/components/schemas/PythonModelTrainingConfig-Output' - $ref: '#/components/schemas/HyperparameterSearchCategoryConfig-Output' - $ref: '#/components/schemas/HyperparameterSearchRegressorConfig-Output' - $ref: '#/components/schemas/CustomIntegrationConfig' - $ref: '#/components/schemas/IntegrationConfig' - $ref: '#/components/schemas/IntegrationExportConfig-Output' - $ref: '#/components/schemas/BatchAutomationConfig' - $ref: '#/components/schemas/OptimisationConfig' - $ref: '#/components/schemas/RunPipelineConfig' - $ref: '#/components/schemas/TabularRegressorTrainingConfig' - $ref: '#/components/schemas/TabularRegressorInferenceConfig' - $ref: '#/components/schemas/EnsembleRegressorTrainingConfig-Output' - $ref: '#/components/schemas/EnsembleRegressorInferenceConfig' - $ref: '#/components/schemas/TabularRegressorShapConfig' - $ref: '#/components/schemas/TabularClassifierTrainingConfig-Output' - $ref: '#/components/schemas/TabularClassifierInferenceConfig' - $ref: '#/components/schemas/EnsembleClassifierTrainingConfig-Output' - $ref: '#/components/schemas/EnsembleClassifierInferenceConfig' - $ref: '#/components/schemas/TabularClassifierInferenceConfigV2' - $ref: '#/components/schemas/LinearRegressorTrainingConfig' - $ref: '#/components/schemas/LinearRegressorInferenceConfig' - $ref: '#/components/schemas/ClusteringInferenceConfig' title: Config created_at: type: string format: date-time title: Created At type: object required: - name - config - created_at title: PipelineStepRead PipelineVersionCheckpointRead: properties: pipeline_version_id: type: string format: uuid title: Pipeline Version Id name: type: string title: Name description: type: string title: Description id: type: string format: uuid title: Id pipeline_id: type: string format: uuid title: Pipeline Id created_at: type: string format: date-time title: Created At type: object required: - pipeline_version_id - name - description - id - pipeline_id - created_at title: PipelineVersionCheckpointRead PipelineReleaseRead: properties: id: type: string format: uuid title: Id version: type: string title: Version description: type: string title: Description created_at: type: string format: date-time title: Created At created_by: $ref: '#/components/schemas/UserRead' type: object required: - id - version - description - created_at - created_by title: PipelineReleaseRead UserRead: properties: id: type: string format: uuid title: Id name: type: string title: Name email: type: string title: Email type: object required: - id - name - email title: UserRead PipelineVersionAnnotations: properties: changed: items: $ref: '#/components/schemas/PipelineVersionStepChange' type: array title: Changed canvas_node_annotations: anyOf: - additionalProperties: type: string type: object - type: 'null' title: Canvas Node Annotations dataset_output_folders: anyOf: - additionalProperties: type: string type: object - type: 'null' title: Dataset Output Folders type: object title: PipelineVersionAnnotations description: Version metadata persisted as JSON on `PipelineVersion.annotations`. DataTransformConfig: properties: job_type: type: string const: data-transform title: Job Type default: data-transform query_type: $ref: '#/components/schemas/DataTransformQueryType' query: type: string title: Query input_datasets: items: type: string type: array title: Input Datasets output_datasets: items: type: string type: array title: Output Datasets type: object required: - query_type - query - input_datasets - output_datasets title: DataTransformConfig description: >- This job type is used to transform data from one dataset to another. This can be done by running a SQL query on Bigquery or by running a pandas script. Always default to Bigquery SQL query unless user explicitly asked for pandas. It is typically used to perform ETL operations including data cleaning and feature engineering. SQL transforms are deliberately limited to exactly one ``CREATE OR REPLACE TABLE ... AS SELECT`` statement and exactly one output dataset. Do not use multi-statement SQL scripts. To produce multiple output tables (for example, train and test splits), create one data-transform step per output table and connect each step to the same upstream dataset. Attributes: query_type (DataTransformQueryType): Indicates whether the query is an SQL query or something else. query (str): The SQL or other query to be executed during data transformation. Include a comment at the beginning of the query explaining the query in a way that is easy to understand. Preface the query with a -- using standard SQL comment syntax. Do not reference BigQuery in the comment. EmbeddingConfig: properties: job_type: type: string const: apply-embeddings title: Job Type default: apply-embeddings column_name: type: string title: Column Name input_dataset: type: string title: Input Dataset output_dataset: type: string title: Output Dataset type: object required: - column_name - input_dataset - output_dataset title: EmbeddingConfig description: >- This job type is used to embed data into a vector space by running an embedding model. It will apply the embedding model to the column specified by the column_name attribute on the input dataset and output the results to the output dataset. A new column will be added to the output dataset with the generated embeddings. This dataset can be searched using a vector search engine. Attributes: column_name (str): The name of the column to embed. BulkPromptConfig: properties: job_type: type: string const: bulk-prompt title: Job Type default: bulk-prompt prompt: type: string title: Prompt input_dataset: type: string title: Input Dataset output_dataset: type: string title: Output Dataset base_model_type: $ref: '#/components/schemas/BulkPromptBaseModelType' max_new_tokens: type: integer title: Max New Tokens default: 4096 output_column_name: type: string title: Output Column Name default: generated_text type: object required: - prompt - input_dataset - output_dataset - base_model_type title: BulkPromptConfig description: >- This job type is used to apply a prompt to a dataset. The prompt is a Python-style f-string which accepts column names as variables within the {} brackets. The formatted prompt is applied to each record of the dataset and fed to an LLM. The LLM's response is then added as a new column to the output dataset. Attributes: prompt (str): The prompt to apply to the dataset. max_new_tokens (int): The maximum number of tokens to generate. ModelInferenceConfig: properties: job_type: type: string const: model-inference title: Job Type default: model-inference input_model: type: string title: Input Model input_dataset: type: string title: Input Dataset output_dataset: type: string title: Output Dataset output_column_name: anyOf: - type: string - type: 'null' title: Output Column Name type: object required: - input_model - input_dataset - output_dataset title: ModelInferenceConfig description: >- This job type applies a trained model to an input dataset to produce an output dataset. The input/output column names and the inference model type (e.g. tabular-classifier, ensemble-regresssor, clustering etc.) are derived from the input_model metadata, so they are not part of this config. Attributes: output_column_name: column name for the model output/prediction. If not specified, it will default to the target column name from the model training. ModelEvaluationConfig: properties: job_type: type: string const: model-evaluation title: Job Type default: model-evaluation input_model: type: string title: Input Model input_dataset: type: string title: Input Dataset output_dataset: type: string title: Output Dataset actuals_column_name: type: string title: Actuals Column Name prediction_column_name: anyOf: - type: string - type: 'null' title: Prediction Column Name metrics_to_monitor: items: $ref: '#/components/schemas/ModelEvaluationMetric' type: array title: Metrics To Monitor evaluation_name: type: string title: Evaluation Name default: evaluation type: object required: - input_model - input_dataset - output_dataset - actuals_column_name title: ModelEvaluationConfig description: >- Apply a trained model to a dataset and compare predictions with an actuals column. This node is for validation, holdout testing, or post-hoc monitoring; use ModelInferenceConfig for batch scoring where no actuals are available. This node is the only supported way to record the model performance metrics shown on the pipeline MLOps page; evaluating a model with SQL/data-transform nodes produces datasets the MLOps page never reads. Version targeting rules — metrics attach to one specific model version: - Place this node in the same pipeline as the training node, downstream of it: input_model must exactly match the training node's output_model (the shared name wires the dependency), so it evaluates the candidate version produced by that run. - If this node is not downstream of a training node in its run (an eval-only pipeline, or a partial run that skips training), it falls back to the most recently updated Ready version — NOT necessarily the current production version — and its metrics can land on the wrong version. ModelExplanationConfig: properties: job_type: type: string const: model-explanation title: Job Type default: model-explanation input_model: type: string title: Input Model input_dataset: type: string title: Input Dataset output_dataset: type: string title: Output Dataset method: $ref: '#/components/schemas/ModelExplanationMethod' default: shap input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_column_prefix: type: string title: Output Column Prefix default: shap sample_size: anyOf: - type: integer minimum: 1 - type: 'null' title: Sample Size default: 500 target_output: anyOf: - type: string - type: 'null' title: Target Output type: object required: - input_model - input_dataset - output_dataset title: ModelExplanationConfig description: |- Explain a trained model on an input dataset. The runtime reads the model metadata and selects the available explainer for that model family. ClusteringTrainingConfig: properties: job_type: type: string const: clustering-training title: Job Type default: clustering-training input_column_names: items: type: string type: array title: Input Column Names input_dataset: type: string title: Input Dataset output_model: type: string title: Output Model clustering_params: $ref: '#/components/schemas/KMeansParams' type: object required: - input_column_names - input_dataset - output_model - clustering_params title: ClusteringTrainingConfig description: >- This job type is used to train an unsupervised clustering model on a dataset. The model is trained using the columns specified by the input_column_names attribute from the input_dataset. No target or label column is required. The default training process will apply 1. median + missing indicator for null numeric values and 2. one hot encoding including the missing value to non numeric columns The following metrics are captured as part of the training code: n_clusters, inertia, silhouette_score TabularRegressorTrainingConfigV2: properties: job_type: type: string const: tabular-regressor-training-v2 title: Job Type default: tabular-regressor-training-v2 input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_dataset: type: string title: Input Dataset output_model: type: string title: Output Model regressor_config: $ref: '#/components/schemas/XGBoostRegressorConfig' metrics_to_monitor: items: $ref: '#/components/schemas/RegresssorMetrics' type: array title: Metrics To Monitor training_split_size: anyOf: - $ref: '#/components/schemas/TrainingSplitSize' - type: 'null' type: object required: - input_feature_column_names - output_target_column_name - input_dataset - output_model - regressor_config title: TabularRegressorTrainingConfigV2 description: >- This job type trains a tabular regressor model on the provided training dataset. Attributes: training_split_size (TrainingSplitSize | None): Deprecated train-time holdout split. Prefer data transforms plus ModelEvaluation nodes for validation/test reporting. metrics_to_monitor (list[RegresssorMetrics]): Deprecated train-time holdout metrics. Prefer ModelEvaluation for reporting. EnsembleRegressorTrainingConfigV2-Output: properties: job_type: type: string const: ensemble-regressor-training-v2 title: Job Type default: ensemble-regressor-training-v2 input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_dataset: type: string title: Input Dataset output_model: type: string title: Output Model ensemble_config: $ref: '#/components/schemas/EnsembleRegressorConfig' metrics_to_monitor: items: $ref: '#/components/schemas/RegresssorMetrics' type: array title: Metrics To Monitor training_split_size: anyOf: - $ref: '#/components/schemas/TrainingSplitSize' - type: 'null' type: object required: - input_feature_column_names - output_target_column_name - input_dataset - output_model - ensemble_config title: EnsembleRegressorTrainingConfigV2 description: >- This job type trains an ensemble of XGBoost regressor models using bagging with dataset split and metric monitoring. Attributes: ensemble_config (EnsembleRegressorConfig): Configuration for the ensemble including size and regressor settings. training_split_size (TrainingSplitSize | None): Deprecated train-time holdout split. Prefer data transforms plus ModelEvaluation nodes for validation/test reporting. metrics_to_monitor (list[RegresssorMetrics]): Deprecated train-time holdout metrics. Prefer ModelEvaluation for reporting. TabularClassifierTrainingConfigV2-Output: properties: job_type: type: string const: tabular-classifier-training-v2 title: Job Type default: tabular-classifier-training-v2 input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_dataset: type: string title: Input Dataset validation_dataset: anyOf: - type: string - type: 'null' title: Validation Dataset test_dataset: anyOf: - type: string - type: 'null' title: Test Dataset output_model: type: string title: Output Model classifier_config: $ref: '#/components/schemas/XGBoostClassifierConfig' metrics_to_monitor: items: $ref: '#/components/schemas/ClassifierMetrics' type: array title: Metrics To Monitor training_split_size: anyOf: - $ref: '#/components/schemas/TrainingSplitSize' - type: 'null' hyper_param_search_config: anyOf: - $ref: '#/components/schemas/HyperparameterSearchConfig' - type: 'null' type: object required: - input_feature_column_names - output_target_column_name - input_dataset - output_model - classifier_config title: TabularClassifierTrainingConfigV2 description: >- Use this job to train an XGBoost classifier on a tabular dataset. The model predicts a categorical target column from one or more feature columns. Dataset rules: - input_dataset is required. - By default, input_dataset is used as the training dataset only. - Use data-transform nodes to prepare training, validation, and test datasets explicitly. For imbalanced classification, balance only the training dataset unless the user asks otherwise. - Use ModelEvaluation nodes for validation/test metrics and actuals. - validation_dataset is a backwards-compatible train-time validation hook for old pipelines. Prefer ModelEvaluation nodes. - validation_dataset and test_dataset default to None. - test_dataset is a backwards-compatible train-time holdout for old pipelines. Prefer ModelEvaluation nodes for test metrics. - training_split_size is a backwards-compatible train-time split for old pipelines. Prefer data transforms plus ModelEvaluation nodes. - Only set validation_dataset, test_dataset, training_split_size, or metrics_to_monitor for legacy compatibility or if the user explicitly asks for train-time holdout behaviour. - validation_dataset and test_dataset may already exist, or may be created by the agent in earlier steps, as long as the user requested this and provided enough instructions. - test_dataset requires validation_dataset, but validation_dataset may be provided without test_dataset. - If validation_dataset is provided, training_split_size should be None. Any provided split size will be ignored. Training/evaluation rules: - Train the model on the training set. - Prefer ModelEvaluation nodes for validation/test metrics, especially when the training dataset has been balanced or sampled. - metrics_to_monitor is deprecated train-time holdout reporting. New pipelines should leave it empty. - Without the legacy holdout fields this job records no metrics itself: unless the pipeline also has a ModelEvaluation node downstream of this node (input_model = this node's output_model), the model version will show "No metrics" on the MLOps page. Computing metrics in SQL nodes does not populate the MLOps page. Hyperparameter search: - hyper_param_search_config should be None by default. - Only set hyper_param_search_config when the user explicitly requests hyperparameter tuning/search. Attributes: metrics_to_monitor: Deprecated holdout metrics. Prefer ModelEvaluation nodes for reporting. hyper_param_search_config: Optional hyperparameter search configuration. EnsembleClassifierTrainingConfigV2-Output: properties: job_type: type: string const: ensemble-classifier-training-v2 title: Job Type default: ensemble-classifier-training-v2 input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_dataset: type: string title: Input Dataset output_model: type: string title: Output Model ensemble_config: $ref: '#/components/schemas/EnsembleClassifierConfig-Output' metrics_to_monitor: items: $ref: '#/components/schemas/ClassifierMetrics' type: array title: Metrics To Monitor training_split_size: anyOf: - $ref: '#/components/schemas/TrainingSplitSize' - type: 'null' type: object required: - input_feature_column_names - output_target_column_name - input_dataset - output_model - ensemble_config title: EnsembleClassifierTrainingConfigV2 description: >- This job type trains an ensemble of XGBoost classifier models using bagging. Attributes: training_split_size (TrainingSplitSize | None): Deprecated train-time holdout split. Prefer data transforms plus ModelEvaluation nodes for validation/test reporting. metrics_to_monitor (list[ClassifierMetrics]): Deprecated train-time holdout metrics. Prefer ModelEvaluation for reporting. LinearRegressorTrainingConfigV2: properties: job_type: type: string const: linear-regressor-training-v2 title: Job Type default: linear-regressor-training-v2 input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_dataset: type: string title: Input Dataset output_model: type: string title: Output Model regressor_config: $ref: '#/components/schemas/LinearRegressorConfig' metrics_to_monitor: items: $ref: '#/components/schemas/RegresssorMetrics' type: array title: Metrics To Monitor training_split_size: anyOf: - $ref: '#/components/schemas/TrainingSplitSize' - type: 'null' type: object required: - input_feature_column_names - output_target_column_name - input_dataset - output_model - regressor_config title: LinearRegressorTrainingConfigV2 description: >- This job type trains a linear regressor model on the provided training dataset. Attributes: training_split_size (TrainingSplitSize | None): Deprecated train-time holdout split. Prefer data transforms plus ModelEvaluation nodes for validation/test reporting. metrics_to_monitor (list[RegresssorMetrics]): Deprecated train-time holdout metrics. Prefer ModelEvaluation for reporting. PythonModelTrainingConfig-Output: properties: job_type: type: string const: python-model-training title: Job Type default: python-model-training input_dataset: type: string title: Input Dataset output_model: type: string title: Output Model runtime_profile: $ref: '#/components/schemas/PythonModelRuntimeProfile' default: ml gpu: anyOf: - $ref: '#/components/schemas/PythonModelTrainingGpu' - type: 'null' description: >- Optional training GPU. Use 't4' only with runtime_profile='dl' when the user requests GPU training; otherwise omit it. execution_target: $ref: '#/components/schemas/PythonModelTrainingExecutionTarget' description: >- Where approved training runs execute. 'remote_workstation' is reserved for a private GPU runner and is mutually exclusive with gpu='t4'. default: kubernetes source: type: string title: Source default: | def train(ctx): df = ctx.input("training_data").load_dataframe() ctx.metrics.summary({"row_count": len(df), "column_count": len(df.columns)}) ctx.metrics.set_primary_metric("row_count", "maximize") ctx.model("model").save_pickle({"columns": list(df.columns)}) entrypoint: type: string title: Entrypoint default: train parameters: additionalProperties: true type: object title: Parameters input_dataset_name: type: string title: Input Dataset Name default: training_data output_model_name: type: string title: Output Model Name default: model bundle: anyOf: - $ref: '#/components/schemas/PythonModelBundleSpec' - type: 'null' config: additionalProperties: true type: object title: Config type: object required: - input_dataset - output_model title: PythonModelTrainingConfig description: >- Train a model using a Python block. The platform owns the dataset/model handles, artifact folder, metrics files, logs, lineage, versioning, and deployment lifecycle. New configs expose one training source block plus parameters. Legacy configs can still pass the lower-level bundle shape used by saved model artifacts. Large BigQuery inputs can use a context-managed reader: `with ctx.input("training_data").open_storage_read_session( selected_fields=[...], row_restriction="split = 'train'", max_stream_count=4, format="arrow") as reader:`. Iterate Arrow RecordBatches or call `reader.stage(new_directory)` for atomic Arrow IPC shards plus a completion manifest. `reader.telemetry()` reports per-stream bytes, rows, retries and elapsed time. Filters support basic column/literal predicates, not arbitrary SQL/functions. Credentials and table scope are platform-owned. Keep DataLoader(num_workers=0); open the reader in the final process, never fork after opening it. Use a caller-owned cancel_event for cooperative stop. Buffer limits cover the queue, not total RSS (one decoded response per worker is additional). Session/offset retry state expires and is NOT a durable checkpoint cursor. A new session has different provenance and cannot silently resume checkpoints from the old one; retain a durable staged dataset for that. Training code can call `ctx.resources.read()` for live pod RAM, CPU, and scratch-disk usage against the cgroup limits. CPU training uses HighMem (15 CPU / 115G). Set gpu='t4' only for a PyTorch `dl` runtime that should use one NVIDIA T4 (3 CPU / 12G); omit it otherwise. HyperparameterSearchCategoryConfig-Output: properties: job_type: type: string const: hyperparameter-search-category title: Job Type default: hyperparameter-search-category input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_dataset: type: string title: Input Dataset output_model: type: string title: Output Model output_dataset: type: string title: Output Dataset search_config: $ref: '#/components/schemas/HyperparameterSearchConfig' type: object required: - input_feature_column_names - output_target_column_name - input_dataset - output_model - output_dataset - search_config title: HyperparameterSearchCategoryConfig description: >- This job type is used to find optimal hyperparameters for an XGBoost classifier using grid search. It accepts a single input dataset which is automatically split into training and test sets (typically 80/20). The model will be trained using the feature columns specified by the input_feature_column_names attribute on the input dataset to predict the label column specified by the output_target_column_name attribute. Grid search is performed using cross-validation on the training set, and the best model (based on the optimization metric) is evaluated on the test set. The job outputs the best model and an evaluation dataset containing one row per trial with metrics and hyperparameters. HyperparameterSearchRegressorConfig-Output: properties: job_type: type: string const: hyperparameter-search-regressor title: Job Type default: hyperparameter-search-regressor input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_dataset: type: string title: Input Dataset output_model: type: string title: Output Model output_dataset: type: string title: Output Dataset search_config: $ref: '#/components/schemas/HyperparameterSearchRegressorSearchConfig' type: object required: - input_feature_column_names - output_target_column_name - input_dataset - output_model - output_dataset - search_config title: HyperparameterSearchRegressorConfig description: >- This job type is used to find optimal hyperparameters for an XGBoost regressor using grid search. It accepts a single input dataset which is automatically split into training and test sets (typically 80/20). The model will be trained using the feature columns specified by the input_feature_column_names attribute on the input dataset to predict the target column specified by the output_target_column_name attribute. Grid search is performed using cross-validation on the training set, and the best model (based on the optimization metric) is evaluated on the test set. The job outputs the best model and an evaluation dataset containing one row per trial with metrics and hyperparameters. CustomIntegrationConfig: properties: job_type: type: string const: custom-integration title: Job Type default: custom-integration secrets: items: type: string type: array title: Secrets output_datasets: items: type: string type: array title: Output Datasets output_dataset_specs: additionalProperties: $ref: '#/components/schemas/OutputDatasetSpec' type: object title: Output Dataset Specs description: >- Optional landing behaviour per output dataset, keyed by output dataset name. Omitted datasets keep the default behaviour: the destination table is fully replaced each run. Set mode='upsert' with merge_keys to MERGE extracted rows into the destination, or mode='window_replace' with window_column to atomically restate the extracted window values. Both are idempotent under retries. mode='append' inserts without any duplicate protection and is not retry-safe. watermark_column (any incremental mode) keys the RUNTIME_SYNC_STATE injection; in upsert it also orders the keep-latest dedupe. Only use a non-default mode when run() extracts a bounded delta rather than a full snapshot. code: type: string title: Code description: >- Python source code for the custom integration. Must define a zero-argument run() function that returns output dataset names mapped to pandas DataFrames. The sandbox has no warehouse or Google Cloud identity, so never build a client from ambient or default credentials such as a bare bigquery.Client() or storage.Client(); reach external sources only through the RUNTIME_* secret globals, parsed per the model description (e.g. json.loads(RUNTIME_ATTIO_API_KEY)). Do not import os or read os.environ/os.getenv in generated code. Available libraries in the custom-integration runtime — these are the exact pinned packages from py_executor/requirements.txt. Generated code may import ONLY these (using each package's standard Python import name, e.g. google-cloud-storage is imported as google.cloud.storage) plus the Python standard library (json, csv, io, datetime, base64, etc.). You cannot pip install anything else — for a source with no SDK listed here, call its HTTP API with httpx. Versions are pinned; use them when debugging version-specific behaviour. - httpx==0.28.* - pandas==2.3.* - pandas-gbq==0.35.* - pyarrow==24.* - pyodbc==5.3.* - lxml==6.1.* - sqlalchemy==2.0.* - google-cloud-storage==3.12.* - google-cloud-bigquery-storage==2.39.* - db-dtypes==1.7.* - boto3==1.43.* - azure-storage-blob==12.30.* - simple-salesforce==1.12.* - clickhouse-connect==1.4.* - redshift-connector==2.1.* - snowflake-connector-python[pandas]==4.* - msal==1.37.* - openpyxl==3.* - PyMySQL==1.2.* - psycopg2-binary==2.9.* - matplotlib==3.10.* - seaborn==0.13.* - statsmodels==0.14.* - numpy==2.* - faker==40.* parameters: additionalProperties: true type: object title: Parameters description: >- Optional JSON-serialisable configuration exposed to the code as the PARAMETERS global (a plain dict). Use for tunables such as row counts, seeds, date ranges, and page sizes. Never put secret values here — it is stored as plain step config. estimated_size_gb: type: integer enum: - 16 - 64 - 128 - 256 - 512 title: Estimated Size Gb default: 128 type: object required: - output_datasets - code title: CustomIntegrationConfig description: >- Use this job type to run custom Python integration code in Google Batch. New imports from external sources should use this job type, including arbitrary SaaS products, REST APIs, databases, spreadsheets, files, and cloud/object storage. Normal `integration` jobs are legacy compatibility for existing pipelines. It is also the right job type for credential-less synthetic/mock data generation: `run()` may construct the DataFrames locally (`random`, `numpy`, `faker`) with no connection and an empty `secrets` list, when the user wants demo, test, or placeholder pipeline data without an external system. The `code` field must define a zero-argument Python function named `run`. The function should import data from external sources and return a dict mapping each output dataset name to a `pandas.DataFrame`, for example: `{"customers": customers_df, "orders": orders_df}`. The generated code should import `pandas as pd` or use the preloaded `pd` module, and it should not write parquet files itself. The orchestrator passes this code inline to Google Batch. The custom integration runner calls `run()` and writes each returned dataframe as a parquet file to `custom_integration///.parquet` in the organisation GCS bucket, then the orchestrator lands each dataframe in its matching BigQuery table — by default a full replace, or incrementally per `output_dataset_specs`. Because that BigQuery load is automatic, do not add a separate data-transform step to move the output into BigQuery. The Batch sandbox has no access to the TrueState warehouse and no Google Cloud permissions of its own: the orchestrator hands the job a pre-authorised upload URL for each output dataset, so its `custom-integration-batch` runtime identity holds no storage or BigQuery roles. Building a client from ambient or default credentials — a bare `bigquery.Client()` or `storage.Client()` — therefore fails with a permission error. Reach external sources only with credentials read from the `RUNTIME_*` globals; that includes Google Cloud sources, which need an explicit client built from a customer credential rather than the default one. Secrets listed in `secrets` are injected into generated code as Python global variables named `RUNTIME_`, with the secret name uppercased and hyphens/spaces converted to underscores. For example, `attio-api-key` is available as the Python global `RUNTIME_ATTIO_API_KEY`. Read those globals directly (never `os`, `os.environ`, or `os.getenv`), and parse each raw Secret Manager value by its `secret_type` — for a token secret, `json.loads(RUNTIME_ATTIO_API_TOKEN)['token']`, not the raw string. Generate code so it never exposes secret values: do not print them, include them in exceptions/logs, post them to unrelated external services, or write them into datasets. Legitimate use for authentication, such as HTTP headers or database clients, is allowed. Review generated code for malicious behavior before creating this config. Incremental (delta) imports: by default run() should extract the full snapshot — full replace is always correct. When the source supports change tracking (a proven modified-since filter, an updated_at column, or date-windowed reports) you may extract only changed rows and declare the landing in `output_dataset_specs`: mode "upsert" with merge_keys (plus watermark_column for keep-latest ordering) for mutable records with a verified-unique key, or mode "window_replace" with window_column for date-windowed facts that restate recent days. The runner injects a RUNTIME_SYNC_STATE global — a JSON object mapping each dataset that has a watermark_column to the destination's current maximum value (ISO-8601 strings for temporal columns), or null on the first run: `since = json.loads(RUNTIME_SYNC_STATE).get("orders")` — treat None as "extract everything". Always re-fetch a small overlap (watermark minus a margin, or the last N report days); upsert and window_replace dedupe it. An empty dataframe with stable columns is a VALID delta result when the source has no new rows. Only use a non-default mode when a probe has proven the source filter narrows results and the merge key is unique; when unsure, keep replace. Temporal dataframe types are part of that stable-schema contract. The runner writes the returned dataframe to Parquet and BigQuery auto-detects the staging schema. For a BigQuery TIMESTAMP column, normalize pandas values to microsecond precision *and dtype* before returning them: `df["event_time"] = pd.to_datetime(df["event_time"], errors="raise").dt.as_unit("us")`; verify the dtype is `datetime64[us]`. Do not leave it as pandas `datetime64[ns]`: Parquet nanosecond timestamps are not a supported BigQuery timestamp input and can arrive as INTEGER. `.dt.floor("us")` changes values but leaves the dtype nanosecond-based, so it is not a fix. Do not format temporal values as text (that lands as STRING). Python `datetime` objects serialize as a Parquet microsecond timestamp and therefore land as BigQuery TIMESTAMP, not DATETIME. For a BigQuery DATE column, return actual `datetime.date` values, not strings. Before enabling an incremental mode on an existing dataset, inspect its schema and match it exactly. Parquet auto-detection cannot preserve an existing BigQuery DATETIME column. If the target is DATETIME, do not guess another pandas representation and do not test a bounded `replace` against the live table: keep it intact and use a separately authorized shadow-table migration to TIMESTAMP, or escalate a runner enhancement that supplies an explicit schema/cast. A one-time replace with microsecond timestamps changes the destination to TIMESTAMP; it does not preserve DATETIME. Generated code should include a short header comment for future agent maintenance: - source and docs checked - secret globals used, such as RUNTIME_ATTIO_API_KEY - output dataset names - pagination, date range, object prefix, or scope assumptions - for delta imports: the landing mode per dataset, the delta filter used, and the probe evidence for filter behaviour and key uniqueness Values in `parameters` are exposed to the code as the `PARAMETERS` global (a plain dict). Put tunables there — row counts, seeds, date ranges, page sizes — instead of burying them as literals, so they can be changed without editing `code`. Choose the smallest `estimated_size_gb` that fits the output; 16 is right for small imports and mock datasets. Prefer clear constants, small helper functions, explicit request timeouts, pagination loops, stable output dataset keys, and conversion to `pandas.DataFrame`. Write to this exact contract: the runner is stricter than any exploratory `run(ctx)` probe and rejects a `ctx` argument, environment-variable reads, and process/filesystem/dynamic-execution escape hatches such as `os`, `subprocess`, `socket`, `open`, `eval`, `getattr`, and dunder access. Do not rely on comments for safety; enforce safety by using the secret globals correctly, not printing secrets, and validating the pipeline run. IntegrationConfig: properties: job_type: type: string const: integration title: Job Type default: integration connection_id: anyOf: - type: string format: uuid - type: 'null' title: Connection Id integration_config: anyOf: - additionalProperties: true type: object - type: 'null' title: Integration Config credential_secret: anyOf: - type: string - type: 'null' title: Credential Secret output_dataset: type: string title: Output Dataset input_dataset: anyOf: - type: string - type: 'null' title: Input Dataset type: object required: - output_dataset title: IntegrationConfig description: >- Legacy external-source integration job. Do not use this job type for new imports. New external-source ingestion should use `CustomIntegrationConfig` (`job_type`: `custom-integration`) so the agent can handle arbitrary APIs, databases, files, spreadsheets, cloud storage, and source-specific logic with custom Python code. IntegrationExportConfig-Output: properties: job_type: type: string const: integration-export title: Job Type default: integration-export integration_config: oneOf: - $ref: '#/components/schemas/GoogleCloudStorageLfsExportParams' - $ref: '#/components/schemas/GoogleCloudStorageIntegrationParams' - $ref: '#/components/schemas/AzureBlobStorageIntegrationParams' - $ref: '#/components/schemas/AwsS3IntegrationParams' - $ref: '#/components/schemas/SqlServerIntegrationExportParams' - $ref: '#/components/schemas/SnowflakeIntegrationExportParams' - $ref: '#/components/schemas/SendEmailExportParams' title: Integration Config discriminator: propertyName: source mapping: aws-s3: $ref: '#/components/schemas/AwsS3IntegrationParams' azure-blob-storage: $ref: '#/components/schemas/AzureBlobStorageIntegrationParams' google-cloud-storage: $ref: '#/components/schemas/GoogleCloudStorageIntegrationParams' google-cloud-storage-lfs: $ref: '#/components/schemas/GoogleCloudStorageLfsExportParams' send-email: $ref: '#/components/schemas/SendEmailExportParams' snowflake: $ref: '#/components/schemas/SnowflakeIntegrationExportParams' sql-server: $ref: '#/components/schemas/SqlServerIntegrationExportParams' credential_secret: type: string title: Credential Secret input_dataset: type: string title: Input Dataset type: object required: - integration_config - credential_secret - input_dataset title: IntegrationExportConfig description: >- Export a Platform Dataset to a configured destination. For a Google Cloud Storage export to the organisation's own bucket, set ``credential_secret`` to ``"null"`` and give ``path`` as a relative object name, for example ``exports/my-table.parquet``. The runner resolves that to the current organisation bucket and uses its workload identity, so never include the environment-specific ``gs://-/`` prefix. Use a fully-qualified ``gs://`` path plus a credential only when exporting to an external GCS bucket. Set ``filetype`` to ``"parquet"`` for Parquet. BatchAutomationConfig: properties: job_type: type: string const: batch-automation title: Job Type default: batch-automation input_column_names: items: type: string type: array title: Input Column Names output_column_name: type: string title: Output Column Name input_dataset: type: string title: Input Dataset output_dataset: type: string title: Output Dataset automation: $ref: '#/components/schemas/AutomationConfig' type: object required: - input_column_names - output_column_name - input_dataset - output_dataset - automation title: BatchAutomationConfig OptimisationConfig: properties: job_type: type: string const: optimisation title: Job Type default: optimisation script: type: string title: Script input_dataset: type: string title: Input Dataset output_dataset: type: string title: Output Dataset type: object required: - script - input_dataset - output_dataset title: OptimisationConfig description: >- This job type is used to run combinatorial optimisation using Or-Tools. 'script' field must have a function `def run():` which will contain your optimisation code and return a pandas dataframe as the output. This this function will be run in an isolated python environment that only allows standard packages, 'pandas' and 'ortools' packages so don't use any others. The input_dataset name can be directly used as a python variable in the script and is a pandas DataFrame. The `run()` function must return a pandas dataframe which will be saved as the output_dataset. Example script: import pandas as pd from ortools.sat.python import cp_model def run(): # replace the input_dataset_name df = model = cp_model.CpModel() n = len(df) # Decision variables: calls per account (0, 1, or 2) calls = { i: model.NewIntVar(0, 2, f"calls_{i}") for i in range(n) } # Constraint: total calls ≤ 50 max_total_calls = 50 model.Add(sum(calls[i] for i in range(n)) <= max_total_calls) # Objective: maximize total net value objective_terms = [] for i in range(n): expected_value = int(df.loc[i, "expected_value"]) per_call_cost = int(df.loc[i, "per_call_cost"]) # expected_value − cost × calls objective_terms.append( expected_value - per_call_cost * calls[i] ) model.Maximize(sum(objective_terms)) # Solve solver = cp_model.CpSolver() solver.parameters.max_time_in_seconds = 10 status = solver.Solve(model) if status not in (cp_model.OPTIMAL, cp_model.FEASIBLE): raise RuntimeError("No solution found") # Build result DataFrame result = df.copy() result["allocated_calls"] = [ solver.Value(calls[i]) for i in range(n) ] result["net_value"] = ( result["expected_value"] - result["per_call_cost"] * result["allocated_calls"] ) return result RunPipelineConfig: properties: job_type: type: string const: run-pipeline title: Job Type default: run-pipeline pipeline_id: type: string format: uuid title: Pipeline Id input_datasets: items: type: string type: array title: Input Datasets default: [] output_datasets: items: type: string type: array title: Output Datasets default: [] input_models: items: type: string type: array title: Input Models default: [] output_models: items: type: string type: array title: Output Models default: [] type: object required: - pipeline_id title: RunPipelineConfig description: >- Use this job when the current pipeline needs to trigger an existing pipeline Guidance: - Do not create a new pipeline here; reference an existing pipeline_id. - Use input datasets/models to make this step wait for upstream steps. - Use output datasets/models to allow downstream steps to depend on this step. used only to define dependencies for the current pipeline step. - Do not assume the listed datasets/models are passed into or returned from the pipeline being run. They are only to define dependencies for the current pipeline step. TabularRegressorTrainingConfig: properties: job_type: type: string const: tabular-regressor-training title: Job Type default: tabular-regressor-training input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_dataset: type: string title: Input Dataset output_model: type: string title: Output Model regressor_config: $ref: '#/components/schemas/XGBoostRegressorConfig' type: object required: - input_feature_column_names - output_target_column_name - input_dataset - output_model - regressor_config title: TabularRegressorTrainingConfig description: Deprecated, use V2. TabularRegressorInferenceConfig: properties: job_type: type: string const: tabular-regressor-inference title: Job Type default: tabular-regressor-inference input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_model: type: string title: Input Model input_dataset: type: string title: Input Dataset output_dataset: type: string title: Output Dataset type: object required: - input_feature_column_names - output_target_column_name - input_model - input_dataset - output_dataset title: TabularRegressorInferenceConfig description: Deprecated, use V2. EnsembleRegressorTrainingConfig-Output: properties: job_type: type: string const: ensemble-regressor-training title: Job Type default: ensemble-regressor-training input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_dataset: type: string title: Input Dataset output_model: type: string title: Output Model ensemble_config: $ref: '#/components/schemas/EnsembleRegressorConfig' type: object required: - input_feature_column_names - output_target_column_name - input_dataset - output_model - ensemble_config title: EnsembleRegressorTrainingConfig description: >- This configuration defines the XGBoost regressor configurations to use for each model in the ensemble. The ensemble size is determined by the number of configurations provided. Attributes: ensemble_configs (list[TabularRegressorConfig]): List of XGBoost regressor configurations to use for each model. Each model in the ensemble will use the corresponding config (cycling through if needed). EnsembleRegressorInferenceConfig: properties: job_type: type: string const: ensemble-regressor-inference title: Job Type default: ensemble-regressor-inference input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_model: type: string title: Input Model input_dataset: type: string title: Input Dataset output_dataset: type: string title: Output Dataset type: object required: - input_feature_column_names - output_target_column_name - input_model - input_dataset - output_dataset title: EnsembleRegressorInferenceConfig description: Deprecated. Use ModelInferenceConfig. TabularRegressorShapConfig: properties: job_type: type: string const: tabular-regressor-shap title: Job Type default: tabular-regressor-shap input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_model: type: string title: Input Model input_dataset: type: string title: Input Dataset output_dataset: type: string title: Output Dataset type: object required: - input_feature_column_names - output_target_column_name - input_model - input_dataset - output_dataset title: TabularRegressorShapConfig description: >- This job type is used to calculate the SHAP values for a tabular regressor or tabular inference model on a dataset. The SHAP values are calculated for each record in the dataset and added as new columns to the output dataset. The input_feature_column_names and output_target_column_name must match the input_feature_column_names and output_target_column_name of the input model. Typically this job is used to explain the predictions of a model on a dataset. TabularClassifierTrainingConfig-Output: properties: job_type: type: string const: tabular-classifier-training title: Job Type default: tabular-classifier-training input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_dataset: type: string title: Input Dataset output_model: type: string title: Output Model classifier_config: $ref: '#/components/schemas/XGBoostClassifierConfig' type: object required: - input_feature_column_names - output_target_column_name - input_dataset - output_model - classifier_config title: TabularClassifierTrainingConfig description: Deprecated, use V2. TabularClassifierInferenceConfig: properties: job_type: type: string const: tabular-classifier-inference title: Job Type default: tabular-classifier-inference input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_model: type: string title: Input Model input_dataset: type: string title: Input Dataset output_dataset: type: string title: Output Dataset type: object required: - input_feature_column_names - output_target_column_name - input_model - input_dataset - output_dataset title: TabularClassifierInferenceConfig description: Deprecated. Use ModelInferenceConfig. EnsembleClassifierTrainingConfig-Output: properties: job_type: type: string const: ensemble-classifier-training title: Job Type default: ensemble-classifier-training input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_dataset: type: string title: Input Dataset output_model: type: string title: Output Model ensemble_config: $ref: '#/components/schemas/EnsembleClassifierConfig-Output' type: object required: - input_feature_column_names - output_target_column_name - input_dataset - output_model - ensemble_config title: EnsembleClassifierTrainingConfig description: Deprecated, use V2. EnsembleClassifierInferenceConfig: properties: job_type: type: string const: ensemble-classifier-inference title: Job Type default: ensemble-classifier-inference input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_model: type: string title: Input Model input_dataset: type: string title: Input Dataset output_dataset: type: string title: Output Dataset type: object required: - input_feature_column_names - output_target_column_name - input_model - input_dataset - output_dataset title: EnsembleClassifierInferenceConfig description: Deprecated. Use ModelInferenceConfig. TabularClassifierInferenceConfigV2: properties: job_type: type: string const: tabular-classifier-inference-v2 title: Job Type default: tabular-classifier-inference-v2 input_model: type: string title: Input Model input_dataset: type: string title: Input Dataset output_dataset: type: string title: Output Dataset type: object required: - input_model - input_dataset - output_dataset title: TabularClassifierInferenceConfigV2 description: Deprecated. Use ModelInferenceConfig. LinearRegressorTrainingConfig: properties: job_type: type: string const: linear-regressor-training title: Job Type default: linear-regressor-training input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_dataset: type: string title: Input Dataset output_model: type: string title: Output Model regressor_config: $ref: '#/components/schemas/LinearRegressorConfig' type: object required: - input_feature_column_names - output_target_column_name - input_dataset - output_model - regressor_config title: LinearRegressorTrainingConfig description: Deprecated, use V2. LinearRegressorInferenceConfig: properties: job_type: type: string const: linear-regressor-inference title: Job Type default: linear-regressor-inference input_feature_column_names: items: type: string type: array title: Input Feature Column Names output_target_column_name: type: string title: Output Target Column Name input_model: type: string title: Input Model input_dataset: type: string title: Input Dataset output_dataset: type: string title: Output Dataset type: object required: - input_feature_column_names - output_target_column_name - input_model - input_dataset - output_dataset title: LinearRegressorInferenceConfig description: Deprecated, use V2. ClusteringInferenceConfig: properties: job_type: type: string const: clustering-inference title: Job Type default: clustering-inference input_dataset: type: string title: Input Dataset input_model: type: string title: Input Model output_dataset: type: string title: Output Dataset type: object required: - input_dataset - input_model - output_dataset title: ClusteringInferenceConfig description: Deprecated. Use ModelInferenceConfig. PipelineVersionStepChange: properties: step_name: type: string title: Step Name change_status: $ref: '#/components/schemas/PipelineVersionStepChangeStatus' before: anyOf: - type: string - type: 'null' title: Before after: anyOf: - type: string - type: 'null' title: After type: object required: - step_name - change_status title: PipelineVersionStepChange DataTransformQueryType: type: string enum: - sql - python - pandas title: DataTransformQueryType BulkPromptBaseModelType: type: string enum: - gemini_2_5_flash title: BulkPromptBaseModelType ModelEvaluationMetric: type: string enum: - accuracy - precision - recall - f1 - logloss - auc - aucpr - mae - rmse - r2 - mape title: ModelEvaluationMetric ModelExplanationMethod: type: string enum: - shap title: ModelExplanationMethod KMeansParams: properties: algorithm: type: string const: K_MEANS title: Algorithm default: K_MEANS n_clusters: type: integer title: N Clusters max_iter: type: integer title: Max Iter default: 300 tol: type: number title: Tol default: 0.001 type: object required: - n_clusters title: KMeansParams XGBoostRegressorConfig: properties: tree_method: type: string title: Tree Method default: auto learning_rate: type: number title: Learning Rate default: 0.3 gamma: type: number title: Gamma default: 0 max_depth: type: integer title: Max Depth default: 6 min_child_weight: type: number title: Min Child Weight default: 1 max_delta_step: type: number title: Max Delta Step default: 0 subsample: type: number title: Subsample default: 1 sampling_method: type: string title: Sampling Method default: uniform colsample_bytree: type: number title: Colsample Bytree default: 1 colsample_bylevel: type: number title: Colsample Bylevel default: 1 colsample_bynode: type: number title: Colsample Bynode default: 1 reg_lambda: type: number title: Reg Lambda default: 1 reg_alpha: type: number title: Reg Alpha default: 0 grow_policy: type: string title: Grow Policy default: depthwise max_bin: type: integer title: Max Bin default: 256 num_parallel_tree: type: integer title: Num Parallel Tree default: 1 num_boost_round: type: integer title: Num Boost Round default: 10 early_stopping_rounds: anyOf: - type: integer - type: 'null' title: Early Stopping Rounds log_steps: type: integer title: Log Steps default: 1000 type: object title: XGBoostRegressorConfig RegresssorMetrics: type: string enum: - mae - rmse - r2 - mape title: RegresssorMetrics TrainingSplitSize: properties: train: type: number maximum: 1 minimum: 0 title: Train default: 1 validation: type: number maximum: 1 minimum: 0 title: Validation default: 0 test: type: number maximum: 1 minimum: 0 title: Test default: 0 type: object title: TrainingSplitSize EnsembleRegressorConfig: properties: regressor_configs: items: $ref: '#/components/schemas/XGBoostRegressorConfig' type: array title: Regressor Configs type: object required: - regressor_configs title: EnsembleRegressorConfig XGBoostClassifierConfig: properties: tree_method: type: string title: Tree Method default: auto learning_rate: type: number title: Learning Rate default: 0.3 gamma: type: number title: Gamma default: 0 max_depth: type: integer title: Max Depth default: 6 min_child_weight: type: number title: Min Child Weight default: 1 max_delta_step: type: number title: Max Delta Step default: 0 subsample: type: number title: Subsample default: 1 sampling_method: type: string title: Sampling Method default: uniform colsample_bytree: type: number title: Colsample Bytree default: 1 colsample_bylevel: type: number title: Colsample Bylevel default: 1 colsample_bynode: type: number title: Colsample Bynode default: 1 reg_lambda: type: number title: Reg Lambda default: 1 reg_alpha: type: number title: Reg Alpha default: 0 grow_policy: type: string title: Grow Policy default: depthwise max_bin: type: integer title: Max Bin default: 256 num_parallel_tree: type: integer title: Num Parallel Tree default: 1 num_boost_round: type: integer title: Num Boost Round default: 10 early_stopping_rounds: anyOf: - type: integer - type: 'null' title: Early Stopping Rounds log_steps: type: integer title: Log Steps default: 1000 scale_pos_weight: anyOf: - type: number - type: 'null' title: Scale Pos Weight eval_metric: $ref: '#/components/schemas/ClassifierMetrics' default: logloss type: object title: XGBoostClassifierConfig ClassifierMetrics: type: string enum: - accuracy - precision - recall - f1 - logloss - auc - aucpr title: ClassifierMetrics HyperparameterSearchConfig: properties: search_space: additionalProperties: items: {} type: array propertyNames: $ref: '#/components/schemas/SearchableHyperparameter' type: object title: Search Space default: {} optimization_metric: $ref: '#/components/schemas/ClassifierMetrics' default: f1 metric_direction: type: string title: Metric Direction default: maximize cv_folds: type: integer title: Cv Folds default: 5 fixed_params: additionalProperties: true type: object title: Fixed Params default: {} type: object title: HyperparameterSearchConfig description: >- Configuration for hyperparameter search for classifiers. This configuration defines the search space, optimization metric, and other settings for finding optimal hyperparameters using grid search on an XGBoost classifier. Attributes: search_space (Dict[SearchableHyperparameter, list]): Dictionary mapping hyperparameters to lists of discrete values to search. Example: {SearchableHyperparameter.learning_rate: [0.01, 0.1, 0.3], SearchableHyperparameter.max_depth: [3, 5, 7]}. optimization_metric: The metric to optimize. metric_direction (str): Whether to maximize or minimize the metric. Options: "maximize" or "minimize". Default: "maximize". cv_folds (int): Number of cross-validation folds to use on the training set. Default: 5. fixed_params (dict): Fixed hyperparameters that are not searched, applied to all trials. These should match XGBoostClassifierConfig structure. EnsembleClassifierConfig-Output: properties: classifier_configs: items: $ref: '#/components/schemas/XGBoostClassifierConfig' type: array title: Classifier Configs type: object required: - classifier_configs title: EnsembleClassifierConfig description: >- Configuration for an ensemble classifier. This configuration defines the XGBoost classifier configurations to use for each model in the ensemble. The ensemble size is determined by the number of configurations provided. Attributes: classifier_configs (list[TabularClassifierConfig]): List of XGBoost classifier configurations to use for each model. Each model in the ensemble will use the corresponding config (cycling through if needed). LinearRegressorConfig: properties: fit_intercept: type: boolean title: Fit Intercept default: true type: object title: LinearRegressorConfig PythonModelRuntimeProfile: type: string enum: - ml - timeseries - dl title: PythonModelRuntimeProfile PythonModelTrainingGpu: type: string enum: - t4 title: PythonModelTrainingGpu PythonModelTrainingExecutionTarget: type: string enum: - kubernetes - remote_workstation title: PythonModelTrainingExecutionTarget PythonModelBundleSpec: properties: schema_version: type: string const: python-model.v1 title: Schema Version default: python-model.v1 runtime_profile: $ref: '#/components/schemas/PythonModelRuntimeProfile' description: >- Dependency runtime for this Python model bundle. Use 'ml' for scikit-learn, XGBoost, and LightGBM; 'timeseries' for the ML stack plus Prophet/CmdStan and statsmodels; or 'dl' for PyTorch. Training compute is selected separately on PythonModelTrainingConfig. Legacy values like 'python-ml-cpu' and 'python-dl-cpu' are accepted for backwards compatibility but new configs should emit these short values. default: ml packages: additionalProperties: type: string type: object title: Packages description: >- Optional package pins available to the user code. Pins should match packages installed in the selected runtime profile image. files: items: $ref: '#/components/schemas/PythonModelSourceFile' type: array minItems: 1 title: Files entrypoints: $ref: '#/components/schemas/PythonModelEntrypoints' config: additionalProperties: true type: object title: Config input_dataset_name: type: string title: Input Dataset Name default: training_data output_model_name: type: string title: Output Model Name default: model batch_input_dataset_name: type: string title: Batch Input Dataset Name default: input_data batch_output_dataset_name: type: string title: Batch Output Dataset Name default: predictions additionalProperties: false type: object required: - files title: PythonModelBundleSpec HyperparameterSearchRegressorSearchConfig: properties: search_space: additionalProperties: items: {} type: array propertyNames: $ref: '#/components/schemas/SearchableHyperparameter' type: object title: Search Space default: {} optimization_metric: type: string title: Optimization Metric default: rmse metric_direction: type: string title: Metric Direction default: minimize cv_folds: type: integer title: Cv Folds default: 5 fixed_params: additionalProperties: true type: object title: Fixed Params default: {} type: object title: HyperparameterSearchRegressorSearchConfig description: >- Configuration for hyperparameter search for regressors. This configuration defines the search space, optimization metric, and other settings for finding optimal hyperparameters using grid search on an XGBoost regressor. Attributes: search_space (Dict[SearchableHyperparameter, list]): Dictionary mapping hyperparameters to lists of discrete values to search. Example: {SearchableHyperparameter.learning_rate: [0.01, 0.1, 0.3], SearchableHyperparameter.max_depth: [3, 5, 7]}. optimization_metric (str): The metric to optimize. Options: "rmse", "mae", "r2". Default: "rmse". metric_direction (str): Whether to maximize or minimize the metric. For RMSE and MAE, use "minimize". For R2, use "maximize". Default: "minimize". cv_folds (int): Number of cross-validation folds to use on the training set. Default: 5. fixed_params (dict): Fixed hyperparameters that are not searched, applied to all trials. These should match XGBoostRegressorConfig structure. OutputDatasetSpec: properties: mode: type: string enum: - replace - append - upsert - window_replace title: Mode default: replace merge_keys: anyOf: - items: type: string type: array - type: 'null' title: Merge Keys window_column: anyOf: - type: string - type: 'null' title: Window Column watermark_column: anyOf: - type: string - type: 'null' title: Watermark Column type: object title: OutputDatasetSpec description: |- Landing behaviour for one custom-integration output dataset. Without a spec (or with mode "replace") the destination table is fully replaced each run — the historical default. The other modes land the extracted rows incrementally via a per-run staging table: - "upsert": MERGE into the destination on merge_keys, keeping the latest row per key (ordered by watermark_column when set). Idempotent — safe under step retries and overlapping extract windows. Requires merge_keys. - "window_replace": atomically deletes the destination rows inside the [min, max] range of window_column present in the extracted rows, then inserts them. Idempotent per window. Requires window_column with no NULL values. - "append": plain insert of the extracted rows. NOT idempotent — a retried run duplicates rows; prefer upsert or window_replace. watermark_column works on ANY incremental mode: it drives the RUNTIME_SYNC_STATE injection (the destination's current MAX of that column), and in upsert additionally orders the keep-latest dedupe and guards against older overlapping runs regressing rows. On append it does NOT dedupe — duplicate rows still land. Column names must be valid BigQuery identifiers after the runner's column-name sanitisation (letters, digits, underscores). GoogleCloudStorageLfsExportParams: properties: source: type: string const: google-cloud-storage-lfs title: Source default: google-cloud-storage-lfs path: type: string maxLength: 512 minLength: 1 title: Path type: object required: - path title: GoogleCloudStorageLfsExportParams description: >- GCS (LFS): server-side sharded Parquet export, without pandas. Use credential_secret='null'. path is a new, versioned relative prefix in the organisation bucket. A successful export publishes path/manifest.json; training must depend on this step and consume that manifest, not a glob. Never reuse a prefix for different data. No external bucket or inline SQL. GoogleCloudStorageIntegrationParams: properties: source: type: string const: google-cloud-storage title: Source default: google-cloud-storage path: type: string title: Path filetype: $ref: '#/components/schemas/IntegrationFileType' type: object required: - path - filetype title: GoogleCloudStorageIntegrationParams AzureBlobStorageIntegrationParams: properties: source: type: string const: azure-blob-storage title: Source default: azure-blob-storage path: type: string title: Path filetype: $ref: '#/components/schemas/IntegrationFileType' type: object required: - path - filetype title: AzureBlobStorageIntegrationParams AwsS3IntegrationParams: properties: source: type: string const: aws-s3 title: Source default: aws-s3 path: type: string title: Path filetype: $ref: '#/components/schemas/IntegrationFileType' type: object required: - path - filetype title: AwsS3IntegrationParams SqlServerIntegrationExportParams: properties: source: type: string const: sql-server title: Source default: sql-server server: type: string title: Server database: type: string title: Database table_name: type: string title: Table Name database_schema: type: string title: Database Schema default: dbo type: object required: - server - database - table_name title: SqlServerIntegrationExportParams SnowflakeIntegrationExportParams: properties: source: type: string const: snowflake title: Source default: snowflake account_identifier: type: string title: Account Identifier database: type: string title: Database table_name: type: string title: Table Name database_schema: type: string title: Database Schema default: public warehouse: anyOf: - type: string - type: 'null' title: Warehouse role: anyOf: - type: string - type: 'null' title: Role type: object required: - account_identifier - database - table_name title: SnowflakeIntegrationExportParams SendEmailExportParams: properties: source: type: string const: send-email title: Source default: send-email subject: type: string title: Subject body_html_template: type: string title: Body Html Template to_emails: items: type: string type: array title: To Emails type: object required: - subject - body_html_template - to_emails title: SendEmailExportParams AutomationConfig: properties: id: type: string format: uuid title: Id type: object required: - id title: AutomationConfig PipelineVersionStepChangeStatus: type: string enum: - added - changed - deleted title: PipelineVersionStepChangeStatus PythonModelSourceFile: properties: path: type: string title: Path content: type: string title: Content type: object required: - path - content title: PythonModelSourceFile PythonModelEntrypoints: properties: train: type: string title: Train default: train.py:train batch_predict: type: string title: Batch Predict default: predict.py:batch_predict serve_setup: anyOf: - type: string - type: 'null' title: Serve Setup default: serve.py:setup serve_predict: type: string title: Serve Predict default: serve.py:predict type: object title: PythonModelEntrypoints IntegrationFileType: type: string enum: - csv - parquet title: IntegrationFileType securitySchemes: HTTPBearer: type: http scheme: bearer ````