Evaluate and use models

A training run finishing successfully tells you that the computation completed. Evaluation tells you how well the model solves the intended problem.

Use data the model did not learn from

Keep a holdout set that reflects how predictions will be made. For future predictions, respect time ordering. For repeated customers or devices, avoid an accidental split that makes evaluation unrealistically easy. Fit learned preparation steps on training data only. Ask Dawn to explain how missing-value handling, encoding, and feature selection are applied to the holdout set.

Match metrics to the decision

Task	Useful checks	What to watch for
Classification	Precision, recall, confusion matrix, and threshold-dependent results	Accuracy can conceal poor performance on a rare class.
Regression	MAE or RMSE, error distribution, and comparison with a simple baseline	A single average can hide large errors for important groups.
Clustering	Group sizes, representative records, stability, and interpretability	Clusters need a useful interpretation; their existence alone is not success.

Available evaluation outputs depend on the model and configured steps. If a needed metric is absent, ask Dawn to create an evaluation output from the predictions and actual outcomes.

Evaluate these predictions against the holdout outcomes. Compare with a simple
baseline, show the largest errors, and break results down by relevant groups.
Explain the sample size and limitations. Do not select a threshold using the
final test set and then report that same set as an unbiased evaluation.

Inspect errors and explanations

Look at false positives, false negatives, or large residuals. Check whether errors concentrate in a particular period, category, or data-quality issue. Feature importance and model explanations can help investigate behaviour, but they do not establish causation. For a model driving a limited-capacity process, evaluate the actual decision rule. For example, if a team can review 100 records per week, inspect the quality of that ranked subset.

Use a model on new data

Prepare the new data with the same feature definitions used for training.
Select the intended trained model in the inference step.
Run predictions into a named output dataset.
Check row counts, identifiers, missing predictions, and output columns.
Validate a small sample before connecting the output to a downstream workflow.

A prediction is an estimate, not an observed outcome. Label prediction fields and reporting clearly.

Monitor after release

Track input completeness, changes in feature distributions, prediction distributions, and performance once actual outcomes arrive. Keep the model version associated with each scoring workflow so changes can be investigated. Before replacing a model, compare it with the existing one on a consistent evaluation set. Record why the replacement is useful and which workflows need the new version.

Get started

Connect your data

Prepare and model data

Understand and manage context

Share and repeat work

Manage your workspace

Use data the model did not learn from

Match metrics to the decision

Inspect errors and explanations

Use a model on new data

Monitor after release

​Use data the model did not learn from

​Match metrics to the decision

​Inspect errors and explanations

​Use a model on new data

​Monitor after release

Use data the model did not learn from

Match metrics to the decision

Inspect errors and explanations

Use a model on new data

Monitor after release