Glue

serverless ETL service

Overview

• A fully managed ETL (Extract, Transform, Load) service that can extract data from various sources, transform it into the required format, and load it into a target data store.

Use cases

• Prepare & transform data for analytic.

ETL workload using Glue

Features

Glue - Convert data into Parquet format

convert to Parquet format

Glue - Data Crawler: Catalog a dataset

Catalog using Glue Data Crawler

Glue - Job bookmark

Prevent processing old data -> Glue can resume a job from where it left off.

Glue - Studio

GUI for create, run and monitor ETL jobs.

Glue - DataBrew

Clean & normalize data using pre-built transformation.

Glue - Streaming ETL

• Built on Apache Spark Structured Streaming

• Compatible with Kinesis Data Streaming, Kafka, MSK (managed Kafka)

Best practices

Trivia

• Data transformation = AWS Glue.

Concepts

• Parquet format: is a columnar storage file format optimized for use with big data processing frameworks.