Back to Utils
dbt macro
Logging
Automatically track when a dbt run starts and finishes, directly in Snowflake.
Goal
In production environments you want to know when a dbt run started, how long it took and whether it succeeded. The Logging macro automatically writes this information to a table in Snowflake via dbt's hook mechanisms. This way you build a complete audit trail of all your dbt runs, without extra tooling.
What to expect
After every dbt run, a row is added to a log table in Snowflake with the following fields:
| Column | Type | Description |
|---|---|---|
| invocation_id | TEXT | Unique id of the dbt run |
| project_name | TEXT | Name of the dbt project |
| target_name | TEXT | dbt target (e.g. dev or prod) |
| run_started_at | TIMESTAMP_TZ | Start timestamp of the run |
| run_ended_at | TIMESTAMP_TZ | End timestamp of the run |
| stat | TEXT | Status (e.g. success or error) |
Usage
1. Add hooks to dbt_project.yml
The macros are called via the on-run-start and on-run-end hooks of dbt:
on-run-start:
- "{{ log_dbt_start() }}"
on-run-end:
- "{{ log_dbt_end() }}"2. Create the log table in Snowflake
The macro writes to a table you create first. By default datamodder.runtimes is used:
CREATE TABLE <database>.datamodder.runtimes ( invocation_id TEXT, project_name TEXT, target_name TEXT, run_started_at TIMESTAMP_TZ, run_ended_at TIMESTAMP_TZ, stat TEXT );
3. Optional: change schema and table
Use variables in dbt_project.yml to change the location:
vars:
management_config:
schema: datamodder
table: runtimes