> ## Documentation Index
> Fetch the complete documentation index at: https://docs.truestate.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate and use models

> Check whether a model is useful before relying on its predictions.

A training run finishing successfully tells you that the computation completed. Evaluation tells you how well the model solves the intended problem.

## Use data the model did not learn from

Keep a holdout set that reflects how predictions will be made. For future predictions, respect time ordering. For repeated customers or devices, avoid an accidental split that makes evaluation unrealistically easy.

Fit learned preparation steps on training data only. Ask Dawn to explain how missing-value handling, encoding, and feature selection are applied to the holdout set.

## Match metrics to the decision

| Task           | Useful checks                                                          | What to watch for                                                            |
| -------------- | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| Classification | Precision, recall, confusion matrix, and threshold-dependent results   | Accuracy can conceal poor performance on a rare class.                       |
| Regression     | MAE or RMSE, error distribution, and comparison with a simple baseline | A single average can hide large errors for important groups.                 |
| Clustering     | Group sizes, representative records, stability, and interpretability   | Clusters need a useful interpretation; their existence alone is not success. |

Available evaluation outputs depend on the model and configured steps. If a needed metric is absent, ask Dawn to create an evaluation output from the predictions and actual outcomes.

```text theme={null}
Evaluate these predictions against the holdout outcomes. Compare with a simple
baseline, show the largest errors, and break results down by relevant groups.
Explain the sample size and limitations. Do not select a threshold using the
final test set and then report that same set as an unbiased evaluation.
```

## Inspect errors and explanations

Look at false positives, false negatives, or large residuals. Check whether errors concentrate in a particular period, category, or data-quality issue. Feature importance and model explanations can help investigate behaviour, but they do not establish causation.

For a model driving a limited-capacity process, evaluate the actual decision rule. For example, if a team can review 100 records per week, inspect the quality of that ranked subset.

## Use a model on new data

1. Prepare the new data with the same feature definitions used for training.
2. Select the intended trained model in the inference step.
3. Run predictions into a named output dataset.
4. Check row counts, identifiers, missing predictions, and output columns.
5. Validate a small sample before connecting the output to a downstream workflow.

A prediction is an estimate, not an observed outcome. Label prediction fields and reporting clearly.

## Monitor after release

Track input completeness, changes in feature distributions, prediction distributions, and performance once actual outcomes arrive. Keep the model version associated with each scoring workflow so changes can be investigated.

Before replacing a model, compare it with the existing one on a consistent evaluation set. Record why the replacement is useful and which workflows need the new version.