Loading Data

​

Prerequisites

• An active TrueState account
• Access to your data source (files or access credential)

• in Datasets page -> ’+ create new dataset’
• in the Chat -> press ’+’ -> press top right ’+’ in the new pop-up window

​

Loading CSV files

• Maximum supported upload: 10 GB. Contact us if you need to ingest files larger than this.
• Column types are inferred automatically during import; you do not need to pre-format numbers or dates.

​

What works best

• Use UTF-8 encoding
• Include a single header row with column names
• Use a comma (,) as the separator

​

If your CSV upload fails (quick fixes)

• Microsoft Excel: File → Save As → “CSV UTF-8 (Comma delimited) (*.csv)”
• Google Sheets: File → Download → “Comma-separated values (.csv)”
• Apple Numbers: File → Export To → CSV → Advanced Options → Text Encoding = “Unicode (UTF-8)”, Field Delimiters = “Comma”
• LibreOffice: File → Save As → Select “Text CSV (.csv)” → Check “Edit filter settings” → Character set = “Unicode (UTF-8)”, Field delimiter = “,”

​

Common reasons a CSV fails and how to resolve

• Unusual text encoding (most common): Re-save as “CSV UTF-8 (Comma delimited)”.
• Non-comma separator: Re-export with commas. Auto-detection usually works, but commas are the most reliable.
• Generated by a proprietary tool: Open the file in Excel/Sheets and re-save as “CSV UTF-8 (Comma delimited)”.
• Extra empty columns: We automatically drop unnamed columns when they are entirely empty.

​

Loading Excel files

• Reads the first sheet in the workbook
• Uses the first row as column names
• Ingests the displayed cell values (as they appear in Excel)

​

Tips for Excel uploads

• If your workbook has multiple sheets, move the data you want to upload to the first sheet.
• If a workbook fails to upload, try “Save As → CSV UTF-8 (Comma delimited)” and upload the CSV.
• Maximum supported upload: 10 GB. Contact us if you need to ingest files larger than this.

​

Importing from external platforms

• Manually refresh by clicking the Refresh icon on the Datasets page, or
• Set up a cron schedule to refresh automatically.

​

Amazon S3

• path: S3 URI in the form s3://bucket/key-or-prefix; single object or a common prefix for many objects
• Credential: aws_access_key_id, aws_secret_access_key
• Behavior: Paginates keys under the prefix and concatenates them.

​

Azure Blob Storage

• path: HTTPS URL https://<account>.blob.core.windows.net/<container>/<file-or-prefix>; single blob or a prefix
• Credential: storage_account_name, storage_account_key
• Behavior: Lists blobs starting with the given prefix and concatenates them.

​

Google BigQuery

• query: Standard SQL statement to execute in BigQuery
• bq_location (optional): Region for your BigQuery data (default to ‘us-east1’, if your data is somewhere else, please specify)
• Credential: Google service account JSON (roles: BigQuery Job User, Read Session User, Table Viewer)
• Behavior: Executes the query and fetches results into a dataset.

​

Google Cloud Storage (GCS)

• path: GCS URI in the form gs://bucket/path-or-prefix; can point to a single file or a prefix to include multiple files
• Credential: Google service account JSON with Storage access
• Behavior: Lists objects under the prefix and concatenates all CSVs into one dataset.

​

Microsoft Fabric Warehouse

• sql_endpoint: Fabric SQL endpoint host
• warehouse_name: Target Fabric warehouse name
• query: SQL statement to run; results are ingested
• Credential: client_id, client_secret
• Behavior: Connects via ODBC (Driver 18), streams results, concatenates to a dataset.

​

Microsoft SQL Server

• server: Hostname or address of the SQL Server (e.g., ‘server.database.windows.net’)
• database: Database name to connect to (e.g., ‘sales’)
• query: T-SQL SELECT statement returning the rows to ingest
• Credential: username, password
• Behavior: Streams results in chunks for large datasets; outputs a single dataset.

​

Salesforce (SOQL)

• domain: Salesforce sub-domain (e.g., ‘yourcompany.my’)
• query: SOQL string to select the records to ingest
• Credential: client_id, client_secret (i.e. Consumer Key and Consumer Secret)
• Behavior: Executes the SOQL and flattens nested objects using underscores.
• Example: SELECT Id, Name, CreatedDate FROM Account WHERE IsDeleted = false

​

Snowflake

• account_identifier: Your Snowflake account identifier (e.g., ‘myorg-myaccount’)
• database: Database name to query (schema defaults to ‘public’ if not specified)
• query: SQL statement to execute
• warehouse (optional): Virtual warehouse to run the query
• role (optional): Snowflake role to use for execution
• Credential: username, password
• Behavior: Streams result batches and concatenates into one dataset.

​

What happens to your column names

• Completely empty columns that are entirely blank are dropped
• All characters that are not letters or numbers are replaced with underscores

"Order Date (US$)"   →  "Order_Date__US__"
"customer-id"         →  "customer_id"
"Region & Country"    →  "Region___Country"
"π_value"             →  "__value"

​

Next steps

• Add your data to the chat context and start working with the TrueState AI agent