ML data platform: GA4 to a Medallion lakehouse
Data platform under both recommenders: GA4 events into S3, Bronze/Silver/Gold on Spark + Iceberg, Airflow 3 and Prefect on Kubernetes, on-prem Linux servers.
- Status
- Production
- Period
- Mar 2025 – Feb 2026
- Role
- Data Scientist (AI & Automation), Nam Viet Media
- Code
- Proprietary
- Stack
- Apache Spark
- Apache Iceberg
- Amazon S3
- Airflow 3
- Prefect
- Kubernetes
- Docker
- Linux
- GA4 events
- Amazon S3
- Bronze
- Silver
- Gold
- Features
- Training + batch inference
Problem
The recommenders needed clean training data, but the source was unstructured Google Analytics 4 event data. There was no platform between raw events and the models.
Approach
I built the data platform from storage up to the recommender.
- Ingestion. Unstructured GA4 event data is ingested into Amazon S3.
- Medallion lakehouse. Data is refined through a Medallion architecture (Bronze → Silver → Gold) on Apache Spark and Apache Iceberg.
- Interaction tables. Raw GA4 events (page views, plays, skips, clicks, sessions) become clean user-item interaction tables and features for training and analytics.
- Orchestration. Ingestion, feature, training, and batch inference pipelines run on Apache Airflow 3 and Prefect, as containerized workloads with Docker on Kubernetes.
- Hardware. The stack is deployed as microservices on on-premise Linux servers. I configured the hardware layer (server chassis, mainboards, and attached physical storage) to connect storage directly with the compute serving the recommender.
- Retraining. Reproducible experimentation and automated retraining keep models current with new content and user behavior.
Architecture
GA4 → S3 → Bronze → Silver → Gold → features → training and batch inference. The same Gold tables serve analytics. Orchestration is split across Airflow 3 and Prefect, and everything runs in containers on Kubernetes over the company's own servers.
Results
This platform is the data layer under both production recommenders (news and music streaming), which together contributed to an approximately 40% increase in customer retention.