> ## Documentation Index
> Fetch the complete documentation index at: https://docs.truestate.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Data engineering with Dawn

> Build a repeatable workflow to clean, join, and validate data.

Use a pipeline when data preparation needs to be reproducible. Start with one useful output, such as a clean orders table, and add complexity after you can explain and verify that output.

## Before you start

You need one or more datasets, or a connection from which you can import them. Know what a row represents in each source and which fields can join the sources correctly.

For this example, assume an orders table and a customers table. Replace these names and rules with your own business definitions.

## 1. Profile the inputs

Attach the datasets to a conversation and ask Dawn to inspect them before changing anything:

```text theme={null}
Profile these orders and customers datasets. Report row counts, date ranges,
missing keys, duplicate keys, and the relationship between customer_id in
each table. Explain what one row represents in each source.
```

Check the findings. Two rows with the same order ID could be a duplicate or two legitimate order lines. Establish the correct rule before deduplicating.

## 2. Specify the output

Describe the desired table in terms a colleague could verify:

```text theme={null}
Create a pipeline that produces clean_order_lines. Keep one row per order
line. Standardise order dates, preserve customer IDs as text, exclude
cancelled orders, and join customer region using customer_id. Keep unmatched
customers and flag them. Write rejected records to a separate dataset.
Do not overwrite the source datasets. Show the plan before running it.
```

Make ambiguous choices explicit: timezone, currency, null handling, duplicate selection, and treatment of returns. Save durable business definitions in [Context](/context/overview).

## 3. Review the pipeline

Open the generated pipeline. Check its inputs, transformation steps, and output dataset names. Inspect generated SQL or Python, especially join conditions and filters.

For every join, establish whether the relationship is one-to-one, one-to-many, or many-to-many. An unexpected many-to-many join can multiply rows and inflate totals without producing a technical error.

Dawn supports SQL and Python preparation workflows. Use SQL for table operations such as filters, joins, and aggregations; use Python where the preparation requires logic or libraries better expressed in code.

## 4. Run and reconcile

Run the pipeline and inspect the result. Ask Dawn to create validation outputs when useful; checks described in a prompt should be verified in the resulting pipeline rather than assumed to exist.

| Check                 | Example expectation                                                               |
| --------------------- | --------------------------------------------------------------------------------- |
| Row count             | Every source order line is either retained or explicitly rejected.                |
| Key uniqueness        | The chosen order-line key is unique in the output.                                |
| Join completeness     | Unmatched customers are counted and visible.                                      |
| Amount reconciliation | Differences from the source total are explained by exclusions or transformations. |
| Date coverage         | The intended reporting period is present and timestamps use the agreed timezone.  |
| Missing values        | Required fields are populated or exceptions are reported.                         |

## 5. Test a second run

Run the same workflow again and compare the results. This catches accidental appending or duplicate ingestion. Then test the intended refresh behaviour with updated source data when you have an appropriate test dataset.

Specify how to handle late-arriving records and corrections. A recurring schedule repeats a workflow; it does not decide these data rules for you.

## 6. Schedule and maintain

Save the pipeline and configure its schedule when the output is reliable. Record the owner, expected refresh frequency, and downstream dashboards, reports, or models.

When a run fails, inspect the first failed step and its logs. Check for expired credentials, renamed columns, changed types, or missing source files before rerunning the entire pipeline. See [pipeline troubleshooting](/guides/pipelines#troubleshoot-a-failed-run).

## Keep raw and prepared data understandable

Use names that distinguish source snapshots from prepared outputs. Describe each dataset's row meaning, key fields, refresh expectations, and known limitations. Keep a rejected-records output when records need manual investigation.

For organisation and eventual retirement, see [workspace cleanup](/workspace/cleanup).