Skip to main content
Use a pipeline when data preparation needs to be reproducible. Start with one useful output, such as a clean orders table, and add complexity after you can explain and verify that output.

​
Before you start

You need one or more datasets, or a connection from which you can import them. Know what a row represents in each source and which fields can join the sources correctly. For this example, assume an orders table and a customers table. Replace these names and rules with your own business definitions.

​
1. Profile the inputs

Attach the datasets to a conversation and ask Dawn to inspect them before changing anything:
Check the findings. Two rows with the same order ID could be a duplicate or two legitimate order lines. Establish the correct rule before deduplicating.

​
2. Specify the output

Describe the desired table in terms a colleague could verify:
Make ambiguous choices explicit: timezone, currency, null handling, duplicate selection, and treatment of returns. Save durable business definitions in Context.

​
3. Review the pipeline

Open the generated pipeline. Check its inputs, transformation steps, and output dataset names. Inspect generated SQL or Python, especially join conditions and filters. For every join, establish whether the relationship is one-to-one, one-to-many, or many-to-many. An unexpected many-to-many join can multiply rows and inflate totals without producing a technical error. Dawn supports SQL and Python preparation workflows. Use SQL for table operations such as filters, joins, and aggregations; use Python where the preparation requires logic or libraries better expressed in code.

​
4. Run and reconcile

Run the pipeline and inspect the result. Ask Dawn to create validation outputs when useful; checks described in a prompt should be verified in the resulting pipeline rather than assumed to exist.

​
5. Test a second run

Run the same workflow again and compare the results. This catches accidental appending or duplicate ingestion. Then test the intended refresh behaviour with updated source data when you have an appropriate test dataset. Specify how to handle late-arriving records and corrections. A recurring schedule repeats a workflow; it does not decide these data rules for you.

​
6. Schedule and maintain

Save the pipeline and configure its schedule when the output is reliable. Record the owner, expected refresh frequency, and downstream dashboards, reports, or models. When a run fails, inspect the first failed step and its logs. Check for expired credentials, renamed columns, changed types, or missing source files before rerunning the entire pipeline. See pipeline troubleshooting.

​
Keep raw and prepared data understandable

Use names that distinguish source snapshots from prepared outputs. Describe each dataset’s row meaning, key fields, refresh expectations, and known limitations. Keep a rejected-records output when records need manual investigation. For organisation and eventual retirement, see workspace cleanup.