datacatalog-tag-manager

A Python package to manage Google Cloud Data Catalog tags, loading metadata from external sources. Currently supports the CSV file format.

Table of Contents

• 1. Environment setup
  • 1.1. Python + virtualenv
    • 1.1.1. Install Python 3.6+
    • 1.1.2. Create a folder
    • 1.1.3. Create and activate an isolated Python environment
    • 1.1.4. Install the package
  • 1.2. Docker
    • 1.2.1. Get the source code
  • 1.3. Auth credentials
    • 1.3.1. Create a service account and grant it below roles
    • 1.3.2. Download a JSON key and save it as
    • 1.3.3. Set the environment variables
• 2. Manage Tags
  • 2.1. Create or Update
    • 2.1.1. From a CSV file
  • 2.2. Delete
    • 2.2.1. From a CSV file
• 3. How to contribute
  • 3.1. Report issues
  • 3.2. Contribute code

1. Environment setup

1.1. Python + virtualenv

Using virtualenv is optional, but strongly recommended unless you use Docker.

1.1.1. Install Python 3.6+

1.1.2. Create a folder

This is recommended so all related stuff will reside at the same place, making it easier to follow the next instructions.

mkdir ./datacatalog-tag-manager
cd ./datacatalog-tag-manager

All paths starting with ./ in the next steps are relative to the datacatalog-tag-manager folder.

1.1.3. Create and activate an isolated Python environment

pip install --upgrade virtualenv
python3 -m virtualenv --python python3 env
source ./env/bin/activate

1.1.4. Install the package

pip install --upgrade datacatalog-tag-manager

1.2. Docker

Docker may be used as an option to run datacatalog-tag-manager. In this case, please disregard the above virtualenv setup instructions.

1.2.1. Get the source code

git clone https://github.com/ricardolsmendes/datacatalog-tag-manager
cd ./datacatalog-tag-manager

1.3. Auth credentials

1.3.1. Create a service account and grant it below roles

• BigQuery Metadata Viewer
• Data Catalog TagTemplate User
• A custom role with bigquery.datasets.updateTag and bigquery.tables.updateTag permissions

1.3.2. Download a JSON key and save it as

• ./credentials/datacatalog-tag-manager.json

1.3.3. Set the environment variables

This step may be skipped if you're using Docker.

export GOOGLE_APPLICATION_CREDENTIALS=./credentials/datacatalog-tag-manager.json

2. Manage Tags

2.1. Create or Update

2.1.1. From a CSV file

• SCHEMA

The metadata schema to create or update Tags is presented below. Use as many lines as needed to describe all the Tags and Fields you need.

• SAMPLE INPUT

1. sample-input/upsert-tags for reference;
2. Data Catalog Sample Tags (Google Sheets) might help to create/export a CSV file.

• COMMANDS

Python + virtualenv

datacatalog-tags upsert --csv-file <CSV-FILE-PATH>

Docker

docker build --rm --tag datacatalog-tag-manager .
docker run --rm --tty \
  --volume <CREDENTIALS-FILE-DIR>:/credentials --volume <CSV-FILE-DIR>:/data \
  datacatalog-tag-manager upsert --csv-file /data/<CSV-FILE-PATH>

2.2. Delete

2.2.1. From a CSV file

• SCHEMA

The metadata schema to delete Tags is presented below. Use as many lines as needed to delete all the Tags you want.

• SAMPLE INPUT

1. sample-input/delete-tags for reference;
2. Data Catalog Sample Tags (Google Sheets) might help to create/export a CSV file.

• COMMANDS

Python + virtualenv

datacatalog-tags delete --csv-file <CSV-FILE-PATH>

Docker

docker build --rm --tag datacatalog-tag-manager .
docker run --rm --tty \
  --volume <CREDENTIALS-FILE-DIR>:/credentials --volume <CSV-FILE-DIR>:/data \
  datacatalog-tag-manager delete --csv-file /data/<CSV-FILE-PATH>

3. How to contribute

Please make sure to take a moment and read the Code of Conduct.

3.1. Report issues

Please report bugs and suggest features via the GitHub Issues.

Before opening an issue, search the tracker for possible duplicates. If you find a duplicate, please add a comment saying that you encountered the problem as well.

3.2. Contribute code

Please make sure to read the Contributing Guide before making a pull request.