dbt freshness

In dbt we can check and assess our data to see if it is stale or has not been updated for some time. 

We can use the following configuration here to do it - config-> freshness.

# dbt test --select "test_type:data"

version: 2

sources:

  - name: SNOWFLAKE_SAMPLE_DATA

    database: SNOWFLAKE_SAMPLE_DATA  

    schema: TPCDS_SF10TCL

    config:

      freshness:

        error_after:

          count: 2

          period: day

    tables:

      - name: store_sales

We can run "dbt source freshness". 

In the first section, it error out because the day last update is 1 year 8 moths ago 

We can easily bump up the number here to 3285 (9 years). 

# dbt test --select "test_type:data"

version: 2

sources:

  - name: SNOWFLAKE_SAMPLE_DATA

    database: SNOWFLAKE_SAMPLE_DATA  

    schema: TPCDS_SF10TCL

    config:

      freshness:

        error_after:

          count: 3285

          period: day

    tables:

      - name: store_sales

And then re-run it again "dbt source freshness".