dbt-config v1.0.0

The dbt project for managing data transformations for LDT's organiser-facing reporting.

Setting up the project for local development

1. Create a copy of the .env.sample file and name it .env. Delete any comments (noted by #...).

2. Set the environment variables in .env according to the commented instructions.

3. To run dbt commands locally, you'll need to setup a virtual environment:

• Check that you have Python3 installed on your Mac by running which python3. If not, install Python3 using run install-python or via the Python downloads page.
• Create your virtual environment, by running:

    yarn create-virtual-env

• Activate your virtual environment, by running:

    source dbt_venv/bin/activate

4. Install dbt (with the postgres adapter), as specified in the requirements.txt file, by running:

    yarn install-dbt

5. Setup, and install any external dbt package dependencies that this project requires, by running:

    yarn setup-dbt

6. If this outputs "All checks passed!" you're now ready to run some dbt commands! Here are some examples:

• To run all the models, run:

    yarn run

• To run one model, use the --select flag (or -s flag), followed by the name of the model:

    doppler run -- dbt run --select <MODEL NAME>

• To run a model and all of its downstream children:

    doppler run -- dbt run --select <MODEL NAME>+

• To run a model and all of its upstream parents:

    doppler run -- dbt run --select +<MODEL NAME>

• To run just the models in one subdirectory of the models folder, use the --select flag (or -s flag), followed by the name of the subdirectory:

    doppler run -- dbt run --select models.<SUBDIRECTORY NAME>

• To run all the tests, run:

    doppler run -- dbt test

• To run a macro, run:

    doppler run -- dbt run-operation <MACRO NAME>

• To run a macro with arguments, run:

    doppler run -- dbt run-operation <MACRO NAME> --args '{<ARG 1>: <VALUE 1>, <ARG 2>: <VALUE 2>}'

Creating a new dbt model

dbt models are all stored in the models subfolder in src. To create a new dbt model:

1. Create a .sql file for your model in a folder under the models directory. For operational reports, create the model in the operational subfolder. Aggregate analytic models should be created in the analytics subfolder.

2. Add a description of the model and columns in schema.yml. Here you can also define tests to run on your model. For more details about writing tests in dbt, see this link.

3. Run dbt run -m <MODEL NAME> to create the model locally.

4. If you have specified tests on your model, run dbt test -m <MODEL NAME> to test your model. Note that you need to build the model using dbt run before you test it.

5. Push your changes and raise a PR.

Sharing DBT Models with wider code-base

1. To re-generate the typescript schemas and view creation commands run:

    yarn gen

2. Copy package to where it needs to be used

Resources:

• Learn more about dbt in the docs
• Check out Discourse for commonly asked questions and answers
• Join the chat on Slack for live discussions and support
• Find dbt events near you
• Check out the blog for the latest news on dbt's development and best practices