dbt_table_diff

此存储库旨在比较BigQuery models在dbt公开 PR 期间发生更改的内容。

用法

该存储库已作为Github Actionand发布PyPi Package，这意味着可以通过多种方式利用它：

• 直接在 Python 中通过run_dbt_table_diff.
• 通过直接在终端中python3 -m dbt_table_diff。
• 在 Github 工作流文件中，通过自动在 Open PR 上Github Actions添加评论。

快速开始：

pip3 install dbt_table_diff

示例代码用法：

from dbt_table_diff import run_dbt_table_diff

run_dbt_table_diff(
        project_id="ultimate-bit-359101",
        keyfile_path="secrets/bq_keyfile.json",
        manifest_file="target/manifest.json",
        dev_prefix="dev_",
        prod_prefix="prod_",
        fallback_prefix="fb_",
        custom_checks_path="",
        ignored_schemas=[],
        irregular_schemas=[],
        org_name="org-not-included",
        repo_name="dbt_example",
        pr_id="2",
        auth_token="my_github_pat",
)

示例 CLI 用法：

python3 -m dbt_table_diff -t $GH_TOKEN -o org-not-included -r dbt_example -l 2 \
--manifest_file 'target/manifest.json' --project_id 'ultimate-bit-359101' \
--keyfile_path 'secrets/bq_keyfile.json' --dev_prefix 'dev_' --prod_prefix 'prod_' --fallback_prefix 'fb_'

示例 Github 操作用法：

• Github 操作概述
• 打开 PR展示如何dbt_table_diff作为 Github Action 使用。

Github 操作输入参数：

逐步分解过程：

• 获取在拉取请求中修改的文件列表
  • 通过卷曲github.api.com/repos/{organization}/{repository}/pulls/{pull_request_id}/files
• 过滤器开启relevant_files
  • 哪些是匹配的文件models/*.sql
• 构建manifest.json
  • 通过运行dbt deps; dbt compile
• 解析manifest.json为relevant_models
  • 使用清单属性original_file_path匹配relevant_files
• 运行所有 SQL 文件dbt_table_diff/sql_checks
  • 对于每一个relevant_models，比较两个 dbt 目标 ( dev_prefixvs prod_prefix)
• 将输出保存到文件
  • 采用 Github 评论支持的格式
• 对公开公关发表评论
  • 利用dbt_table_diffPyPi 包

文档

python3 -m dbt_table_diff --help

usage: dbt_table_diff [-h] [-o ORG_NAME] [-r REPO_NAME] [-t AUTH_TOKEN] [-l PR_ID] [--manifest_file MANIFEST_FILE] [--project_id PROJECT_ID] [--keyfile_path KEYFILE_PATH] [--ignored_schemas IGNORED_SCHEMAS]
                      [--irregular_schemas IRREGULAR_SCHEMAS] [--dev_prefix DEV_PREFIX] [--prod_prefix PROD_PREFIX] [--fallback_prefix FALLBACK_PREFIX] [--custom_checks_path CUSTOM_CHECKS_PATH]

optional arguments:
  -h, --help            show this help message and exit
  -o ORG_NAME, --org_name ORG_NAME
                        Owner of GitHub repository.
  -r REPO_NAME, --repo_name REPO_NAME
                        Name of the GitHub repository.
  -t AUTH_TOKEN, --auth_token AUTH_TOKEN
                        User's GitHub Personal Access Token.
  -l PR_ID, --pr_id PR_ID
                        The issue # of the Pull Request.
  --manifest_file MANIFEST_FILE
                        The path to dbt's manifest file.
  --project_id PROJECT_ID
                        The BigQuery Project ID to leverage.
  --keyfile_path KEYFILE_PATH
                        The path to the keyfile to use during BQ calls.
  --ignored_schemas IGNORED_SCHEMAS
                        Folders in models/ to always ignore during row/col checks.
  --irregular_schemas IRREGULAR_SCHEMAS
                        Folders in models/ which use 'fallback_prefix' in prod.
  --dev_prefix DEV_PREFIX
                        Prefix used by development datasets in dbt.
  --prod_prefix PROD_PREFIX
                        Prefix used by production datasets in dbt.
  --fallback_prefix FALLBACK_PREFIX
                        Uncommon prefix used by only some production datasets in dbt.
  --custom_checks_path CUSTOM_CHECKS_PATH
                        A local folder containing any custom SQL to run.