Transunion
|Data Engineer
Highlights
Enhanced software platform for CCPA compliance, ensuring robust data privacy and automated user data management
Design and manage Apache Airflow DAGs to automate, schedule, and monitor ETL workflows
Working on more than 3 petabytes of data in GCS using Scala and Spark
Building a graph database for U.S. households, tracking IP counts and unique person identifiers (eKey) for analysis.
Leverage Redis Queue to continuously update paths and perform pre-processing for efficient data pipeline execution.
Utilized Google Cloud buckets and implemented effective data purging strategies, reducing unnecessary storage costs and enhancing overall data management efficiency.