Autogenerate Lightdash-ready .yml files for your models

The beauty of Lightdash is that we're pretty well synced with your dbt project. So, in Lightdash, Tables actually come from dbt models that have been defined in your dbt project's .yml files.

If your dbt model has been defined in a .yml file, it will appear in Lightdash as a Table.

info

Not too sure what a .yml file or a Table is? Be sure to check out our tutorial on Adding Tables to Lightdash for more details.

Auto-generate schema.yml files

To make it faster to write and maintain your schema.yml files, the Lightdash CLI will automatically generate and sync your schema.yml files.

Inside your dbt project, open a terminal and run this command to generate the schema.yml file for a model called mymodel:

git commit # make sure to commit all your existing project first!
lightdash generate -s mymodel

This will create or update an existing schema.yml file to add any new columns that have appeared in the database.

Auto-sync schema.yml files after dbt run

Since the generate command relies on your database schema to detect which columns are required, you'll often want to run dbt run before running generate. You can combine dbt run and lighdash generate into one command which will run models and then regenerate the schema.yml files for any models than changed:

git commit
lightdash dbt run  # Run dbt AND re-sync schema.yml files

Generating schema.yml for multiple models

lightdash generate supports dbt model selection syntax to generate files for a group of models:

lightdash generate # all models
lightdash generate -s payments  # just payments
lightdash generate -s payments+ # payments and all children
lightdash generate -s +payments # payments and all parents
lightdash generate -s +payments+ # payments and all children and parents
lightdash generate -s tag:prod # all models with the prod tag
lightdash generate -s payments customers # two specific models
lightdash generate -s payments+ +customers tag:prod # mix and match

Using doc blocks to build better .yml files

Doc blocks are markdown files defined in your dbt project that are useful for creating consistent documentation in your dbt project.

For example, if I have a field called product_category that's used a bunch of times throughout my project, I can create a doc block for this field:

{% docs product_category %}

This is a string field that tells us the category of the product that was sold. This is a higher level grouping than the `product_type`. Every `product_type` maps to a single `product_category`.

The valid product categories are:

- clothing
- home
- accessories
- beauty

{% enddocs %}

I can then reference this doc block in my .yml files like so:

version: 2

models:
  - name: products
    description: 'one row per product ID'

    columns:
      - name: product_category
        description: '{{ doc("product_category") }}'

The docs for product_category (in Lightdash and in your dbt documentation) will become the doc block I've written above.

The Lightdash CLI will automatically add doc blocks to your .yml files

The nice thing about doc blocks is that Lightdash automatically adds these to the .yml files it generates. For example, if I had a .yml file for my product_sales model that looked like this:

version: 2

models:
  - name: product_sales

    columns:
      - name: product_id
      - name: product_value
      - name: product_category

Then, I did:

lightdash dbt run -s product_sales # or lightdash generate -s product_sales

Then my product_sales.yml file would be updated to include the doc block for product_category:

version: 2

models:
  - name: product_sales

    columns:
      - name: product_id
      - name: product_value
      - name: product_category
        description: '{{ doc("product_category") }}'

We recommend using doc blocks whenever you can! They help with keeping the definitions of fields across your project consistent, and they also allow you to get great docs, completely automatically with the Lightdash CLI.