Migrating from BigQuery to AWS: Yggdrasil Gaming’s Modern Lakehouse Journey

Intro

Yggdrasil Gaming develops and publishes casino games globally, processing massive amounts of real-time gaming data for game performance analytics, player behavior insights, and industry intelligence. As the company grew, managing a dual-cloud environment across AWS and Google Cloud created significant operational overhead and limited their ability to implement advanced analytics. This challenge became critical ahead of the launch of their "Game in a Box" solution on AWS Marketplace, which was projected to dramatically increase data volume and complexity.

Previously using Google BigQuery for their data warehousing, Yggdrasil engaged GOStack to consolidate their data infrastructure on AWS. The core objective was to build a modern, open, and cost-efficient lakehouse architecture that could handle real-time gaming analytics and support future machine learning use cases

The Challenge

Yggdrasil’s data architecture was facing several critical challenges that prompted the migration to a unified AWS platform.

Multi-Cloud Operational Complexity: Managing infrastructure across both AWS and Google Cloud created significant operational overhead, reducing agility and increasing maintenance costs. The data team had to maintain expertise in both environments and coordinate complex data movements between clouds.

Proprietary System Limitations: The existing setup on BigQuery, while powerful, created a dependency on a proprietary system. Yggdrasil sought to move to an open-standard architecture to avoid vendor lock-in and gain greater flexibility for future innovation.

Cost Inefficiency: The provisioned, always-on compute model of their existing data warehouse was not cost-effective for their bursty, event-driven workloads. Costs were incurred even during off-peak periods, while the architecture struggled to scale efficiently for game launches and tournaments.

High Latency for Analytics: With data freshness cycles taking up to two hours, the business lacked the real-time insights needed to react quickly to player behavior, game performance, or operational issues.

Our Solution

GOStack designed and implemented a modern lakehouse architecture on AWS, migrating Yggdrasil from BigQuery and establishing a scalable, open, and cost-efficient foundation for their entire data ecosystem.

Open Lakehouse Foundation on Amazon S3: The solution centers on a data lake built on Amazon S3, providing durable and cost-efficient storage. Data is stored in Apache Iceberg table format, which delivers ACID transactions, schema evolution, and time-travel capabilities, all while maintaining an open standard. AWS Glue Data Catalog serves as the central metadata repository, with Amazon Athena acting as the serverless query engine.

Real-Time Data Ingestion with Debezium: To capture transactional data in real-time, GOStack deployed Debezium Server Iceberg on Amazon EKS using a GitOps model with Argo CD. This streams change data capture (CDC) events directly from operational databases into the Iceberg tables on S3, bypassing the need for intermediate brokers and ensuring data is available for querying in near real-time.

Modern, Modular Data Transformation with dbt: The transformation layer was completely redesigned using dbt with the dbt-athena adapter. Legacy stored procedures and scheduled queries from BigQuery were rebuilt as modular, version-controlled dbt models. This shift made the transformation logic more transparent, maintainable, and easier to govern. Orchestration was consolidated on Argo Workflows running on Amazon EKS.

Phased Migration Strategy: The migration from BigQuery to the AWS lakehouse followed a structured, four-phased approach to minimize risk and ensure business continuity:

Establish Lakehouse Foundation: Set up the core architecture with S3, Iceberg, Glue, and Athena.
Implement Real-Time Ingestion: Deploy Debezium for real-time CDC from source systems.
Migrate Processing Pipelines: Re-platform legacy data applications and ETL jobs on AWS Lambda and Amazon EMR.
Modernize Transformations: Rebuild the transformation layer using dbt and Argo Workflows.

Centralized Governance with AWS Lake Formation: AWS Lake Formation was implemented as the primary governance layer, providing fine-grained access control at the database, table, column, and row levels. This enabled Yggdrasil to establish a robust security posture for their data lake, balancing strong security with operational flexibility.

Results and Benefits

The migration to a modern AWS lakehouse delivered significant and measurable improvements across the board.

• 60% Reduction in Data Processing Costs: The move to a serverless, pay-per-query model with Amazon Athena, combined with the open architecture, dramatically reduced costs compared to the previous provisioned data warehouse.

• 75% Improvement in Data Freshness: Analytics latency was reduced from 2 hours to just 30 minutes, providing the business with much faster access to critical insights.

• Eliminated Multi-Cloud Complexity: Consolidating on AWS removed the operational overhead of managing a dual-cloud environment, freeing up the data team to focus on value-added activities.

• Future-Ready, Open Architecture: The use of open formats like Apache Iceberg and Parquet provides flexibility and avoids vendor lock-in, ensuring the platform can evolve with Yggdrasil’s needs.

Partnership for Years

GOStack runs our AWS infrastructure and data stack. We've scaled a lot and it all just works. That's what you want from a partner.

James Curwen

CEO

Yggdrasil Gaming