Data Evolution with Real-time Sync & UpgradeData Evolution with Real-time Sync & Upgrade

MarTech

BUSINESS CHALLENGE

Revitalizing outdated database with seamless sync and real-time migration for the client's software upgrade using innovative scripts, Lambda functions, and modularized JSON approach.

• Our client's outdated and poorly designed database posed significant obstacles in syncing with a new, stable, and standardized system. To ensure a seamless transition during their software upgrade, they sought a real-time data sync solution.
• Our data engineering team was entrusted with developing a data pipeline that replicated data from the previous system with near real-time synchronization, targeting a latency of approximately 15 minutes.
• Transitioning from batched pipelines to real-time synchronization presented a transformative challenge.

OUR SOLUTION

Leveraging AWS with Glue and Lambda, we achieved seamless communication, scalability, and efficient data syncing for sustained operational excellence. We modularized our approach, integrating JSON files into the pipeline and enabling adaptability in table structures, columns, and data types.

• Given that the client's infrastructure and databases were hosted on AWS, we harnessed the power of AWS to ensure seamless communication between databases and back-end systems.
• Glue, being able to handle substantial loads, was employed for the one-time bulk load. For incremental data updates, we leveraged the cost-effective and scalable nature of AWS Lambda functions.
• We optimized performance by redesigning the database with a normalized architecture. Data syncing was achieved using AWS Lambda functions executed every 15 minutes, ensuring efficient retrieval and seamless operation since its production deployment months ago.

Amazon Web Services | MySQL | Glue | Lambda Functions

Lambda migrates complex structures and completes tasks in <1 min for 40+ tables, ensuring reliable data migration.

• We carefully crafted the incremental load Lambda function, utilizing customizable inputs for mapping source and destination tables. Despite the complexity of migrating over 40 tables and transforming their structures according to the new architecture, the Lambda function consistently completes its task in less than 1 minute. This achievement underscores the remarkable success of this project, offering our client a reliable and efficient data migration solution.