> For the complete documentation index, see [llms.txt](https://mamawhocode.gitbook.io/aws/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://mamawhocode.gitbook.io/aws/services/analytics/data-processing/glue.md).

# Glue

serverless ETL service

## Overview

* A fully managed <mark style="color:red;">**ETL**</mark> (<mark style="color:red;">**E**</mark>xtract, <mark style="color:red;">**T**</mark>ransform, <mark style="color:red;">**L**</mark>oad) service that can extract data from various sources, transform it into the required format, and load it into a target data store.

### Use cases

* Prepare & transform data for analytic.

<figure><img src="https://2259236002-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuh9xZDZ53qGqmMCM44PU%2Fuploads%2FV7FxBUzrUCaBMbvdHWkD%2Fimage.png?alt=media&amp;token=90d0ba42-43f5-454d-b3e1-82e415ce99ae" alt=""><figcaption><p>ETL workload using Glue</p></figcaption></figure>

## Features

### Glue - Convert data into Parquet format

<figure><img src="https://2259236002-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuh9xZDZ53qGqmMCM44PU%2Fuploads%2FZ1xTjt45vQ77uvUb2khk%2Fimage.png?alt=media&amp;token=0dfc7485-f4b0-4f0d-8cf0-8881016bb8c0" alt=""><figcaption><p>convert to Parquet format</p></figcaption></figure>

### Glue - Data Crawler: Catalog a dataset

<figure><img src="https://2259236002-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuh9xZDZ53qGqmMCM44PU%2Fuploads%2Fn3fwWmCnSkStQ6veHJsO%2Fimage.png?alt=media&amp;token=524ebfb1-ed22-49b6-a59f-fd60e3506a1a" alt=""><figcaption><p>Catalog using Glue Data Crawler</p></figcaption></figure>

### Glue - Job bookmark

Prevent processing *<mark style="background-color:yellow;">old data</mark>* -> Glue can resume a job from where it left off.

### Glue - Studio

GUI for create, run and monitor ETL jobs.

### Glue - DataBrew

* Glue DataBrew is a <mark style="color:red;">**visual data preparation tool**</mark> that allows you to clean, transform, and enrich data without writing code.
* **Features**:
  * Data profiling: automatically analyze datasets to provide insights like missing values, outliers, and data distributions.
  * Pre-built transformation: offer over 250 transformations, such as filtering, normalizing, and deduplication.
  * Custom rules: allows you to define custom data quality rules to enforce specific standards (e.g., detecting PII or ensuring data completeness).
  * Integration: work with S3, Redshift, and Glue ETL.
* **Use cases**:
  * Cleaning raw data from sources like S3 or databases.
  * Detecting and handling **PII (Personally Identifiable Information)**.
  * Preparing data for machine learning or analytics in tools like Amazon Athena or Redshift.

### Glue - Streaming ETL

* Built on Apache Spark Structured Streaming
* Compatible with Kinesis Data Streaming, Kafka, MSK (managed Kafka)

## Best practices

## Trivia

* Data transformation = AWS Glue.

## Concepts

* [Parquet format](#convert-data-into-parquet-format): is a *columnar storage* file format optimized for use with *<mark style="color:red;">**big data**</mark>* processing frameworks.&#x20;