Atlan natively supports the AWS Glue catalog which allows you to seamlessly integrate your Glue catalog with your Atlan Workspace.
You can set up an AWS Glue integration with your Atlan workspace in 4 easy steps:
- Select the Source a.k.a Glue 😉
- Provide your credentials ✍️
- Set up your configuration 🗄️
- Schedule automatic updates 🕑
📜 Pre-Requisites for AWS Glue Integration
Before you get started with integrating your AWS Glue with Atlan, you'll need some pre-requisite information which will help establish a connection between Atlan and your Glue Account:
- Access Key - Access keys consist of an access key ID and a secret access key, which are used to sign programmatic requests that you make to AWS. Visit AWS documentation around Access keys to know more about access keys and how to create them.
- Catalog ID - Each AWS account has a Data Catalog in an AWS Region with the 12-digit account ID known as the catalog ID.
- Region - The region your AWS Glue is hosted in.
- Database - A database in the AWS Glue is a container that holds tables. Databases are used to organize your tables into separate categories.
- Table - The table you want to integrate with your Atlan workspace
- Use Case - You have an option to either pick Development or Production. As the name suggests, the option refers to whether the data asset getting created would be used for Production or further Development purpose. This is metadata information and will not have any impact on the process of integration.
Pro-tip: If you don't have this information handy, reach out to your cloud admin or data lake administrator to get these details before you get started!
🚀 The step by step guide to integrate AWS Glue with Atlan
Once you have the pre-requisite information listed in the section above, please follow the steps below to establish a connection and integrate Atlan with your AWS Glue catalog.
Step 1: Selecting the Source
- Login to your Atlan Workspace
- On the Home Screen, click on the 'New Integration' button on the top right corner. You will see a Dialogue box with the list of sources available on your workspace
- Select 'Glue' from the list of options and click on 'Next'
Step 2: Providing Credentials
- You will see an option to either select a pre-configured credential from the dropdown or to create a credential. To set up a new connection, click on the 'Create Credential' button.
- You will be required to fill in your AWS Glue credentials. Below is an example of the credentials required:
Credential Name - Glue
Access Key - AKIA5XXXXXXXXXXWIJUS
Secret Key - R1xXXXXXXXXX5PEdHOUXXXXXXXX7Ooz47
Catalog ID - 902919795975
Region - US West (N. California)
Use case - development
- Once you have filled in the details, click on 'Next'.
Step 3: Setting up Configuration
- You will now be asked to fill in the details of your database and table. Below is an example -
Add Database - Sales Master
Add Table - Daily Sales
- Chose whether to run the crawler once or schedule it for a Daily, a Weekly, or a Monthly run. You would be asked to specify the timezone to trigger the run.
- Click on 'Create'. Your connection is now created.
Congratulations, you have now integrated Atlan with your AWS Glue Catalog! 🎉
🏁 Monitoring your AWS Glue Integration
Once the integration setup is completed, you will be redirected to the Monitor tab for your AWS Glue asset where you can monitor the progress.