sources

dbt-coves generate sources

Overview

This command will generate the dbt source configuration as well as the initial dbt staging model(s). It will look in the database defined in your profiles.yml file or you can pass the --database argument or set up default configuration options (see below)

dbt-coves generate sources --database raw

Supports Jinja templates to adjust how the resources are generated. See below for examples.

dbt-coves can be used to create the initial staging models. It will do the following:

1. Create / Update the source yml file
2. Create the initial staging model(sql) file and offer to flatten VARIANT(JSON) fields
3. Create the staging model's property(yml) file.

NOTE: While there is no current option to skip source or staging model generation, if you don't want the source.yml or staging models, you can update the path in the dbt-coves config file to point to a static location such as /tmp/not_needed.sql and /tmp/not_needed.yml

Arguments

dbt-coves generate sources supports the following args:

See full list in help

dbt-coves generate sources -h

--database
# Database to inspect

--schemas
# Schema(s) to inspect. Accepts wildcards (must be enclosed in quotes if used)

--select-relations
# List of relations where raw data resides. The parameter must be enclosed in quotes. Accepts wildcards.

--exclude-relations
# Filter relation(s) to exclude from source file(s) generation. The parameter must be enclosed in quotes. Accepts wildcards.

--sources-destination
# Where sources yml files will be generated, default: 'models/staging/{{schema}}/sources.yml'
# Accepts the '{{database}}', '{{schema}}' and '{{relation}}' template variables.

--models-destination
# Where models sql files will be generated, default: 'models/staging/{{schema}}/{{relation}}.sql'
# Accepts the '{{database}}', '{{schema}}' and '{{relation}}' template variables.

--model-props-destination
# Where models yml files will be generated, default: 'models/staging/{{schema}}/{{relation}}.yml'
# Accepts the '{{database}}', '{{schema}}' and '{{relation}}' template variables.

--update-strategy
# Action to perform when a file already exists: 'update', 'recreate', 'fail', 'ask' (per file)

--templates-folder
# Folder with jinja templates that override default sources generation templates, i.e. 'templates'

--metadata
# Path to csv file containing metadata, i.e. 'metadata.csv'

--flatten-json-fields
# Action to perform when JSON fields exist: 'yes', 'no', 'ask' (per file)

--overwrite-staging-models
# Flag: overwrite existing staging (SQL) files

--skip-model-props
# Flag: don't create model's property (yml) files

--no-prompt
# Silently generate source dbt models

Metadata

dbt-coves supports the argument --metadata which allows users to specify a csv file containing field types and descriptions to be used when creating the staging models and property files.

dbt-coves generate sources --metadata metadata.csv

Metadata format: You can download a sample csv file as reference

database	schema	relation	column	key	type	description
raw	raw	_airbyte_raw_country_populations	_airbyte_data	Year	integer	Year of country population measurement
raw	raw	_airbyte_raw_country_populations	_airbyte_data		variant	Airbyte data columns (VARIANT) in Snowflake
raw	raw	_airbyte_raw_country_populations	_airbyte_ab_id		varchar	Airbyte unique identifier used during data load