Programmatically bulk-upload existing metadata using Atlan's APIs
🀸🏼

Programmatically bulk-upload existing metadata using Atlan's APIs

Join the Weekly Digest
image

πŸ€“ Why?

When a team starts their journey towards building a modern data workspace like Atlan, one of the first action items after making all their data assets discoverable and searchable πŸ”Ž is to upload all available existing documentation into Atlan. This is often spread across scattered sources like Google Sheets, Excel files, and Confluence pages!

πŸ’‘
Many DataOps champions use Atlan's Bulk Metadata APIs to upload metadata....in bulk!

πŸ“„ What are Atlan's Bulk Metadata APIs?

Atlan is built on extensive open APIs that make it really easy to upload any and all kinds of metadata programatically. If your data asset's documentation is available in another tool like Confluence, you can easily use Atlan's Bulk Metadata APIs to push metadata into Atlan.

Here are some of the key APIs that can be used for uploading metadata in bulk:

πŸ‘‰ TL;DR: Upload your existing metadata in bulk using Atlan's APIs

1️⃣ Start by defining the metadata that's available.

List all the different metadata that is already available in different tools and services. These could be Confluence pages, Google Sheets, or any other knowledge bases.

Here is some of the metadata that many organizations already have:

Owners and Experts
Table/dashboard description
Column descriptions
Long text descriptions

Glossary terms
Classifications
Popular queries

2️⃣ Bring all the information into a tabular format.

Put all your available metadata into a tabular format. The base columns should help uniquely identify a data asset.

Here is a sample template that you can use:

Bulk Upload Template

data_sourcetypeNamedatabaseschemanamedescriptionClassificationstatus_tagteamOwner
glue
AtlanTable
express_dwh
default
sales_2m
PII|Protected
VERIFIED
Sales & Marketing
glue
AtlanTable
express_dwh
default
purchase_order_registry
Sales
VERIFIED
Sales & Marketing

3️⃣ Use the APIs to push the metadata into Atlan.

Once the table is ready with the information, you have done most of the work! Now, using the Atlan APIs, this information can be pushed and made available to all the users in the organization.

Here's the API Documentation to help you get started. If you prefer using Python, we have you covered β€”Β here's a sample script.

πŸ“” Resources

🌐
Quick Start: Reusable Bulk Upload Python Script

πŸ”— Related reads