Optimizing Data Infrastructure for a Fintech LeaderOptimizing Data Infrastructure for a Fintech Leader

Fintech

BUSINESS CHALLENGE

The client is cited as the industry’s largest commercially available database of securitized mortgages.

• Since they collect data from various providers, they needed a scalable infrastructure to process these data streams to create client-based products that could serve their customers through API and BI Tools.
• This called for a scalable platform to store all the structured and unstructured data.
• The client was looking for a cost-effective solution for the implementation of a new pipeline.

OUR SOLUTION

We made quality deployments in time while monitoring and maintaining the running services and tackled the use cases with innovative workflows.

• We adopted Data Lake with AWS Lake Formation and Glue to eliminate the need to model data into an enterprise-wide schema. With the increase in data volume, data quality, and metadata, the quality of analyses also increases.
• Many existing processes were run through spark jobs on AWS EMR, leading to increased server costs. So, we switched to a newer graviton-based instance type which uses 64-bit Arm Neoverse cores to provide the highest price performance and scalability with minimal latency and cost.
• With the introduction of new processes, we had to keep up with more Jenkins job creations and executions. This created a cluttered Jenkins Server and became a bottleneck for multiple urgent deployments due to the server’s limited computing resources. So, we completely migrated all the existing Jenkins jobs into Github action workflows, automating the deployments after every commit. This made deployment, monitoring and debugging a lot easier and this reduced the time for making quality deployment releases by 40-50%.
• To maintain data security, alerts are monitored and resolved by Trusted Advisor, and sensitive information is stored in AWS Parameter Store.

We built a scalable infrastructure to process huge volumes of structured and unstructured data, empowering the client to provide primary and secondary market participants with the necessary solutions and analytics.

• Implementation of Data Lake gave a 360-degree view of customers and made analysis processes more robust while minimizing cost since all data could be stored in one place.
• Adding load balancer config increased the desired number of tasks, reducing downtime of the services hosted on ECS by around 85-90%.
• Using Github Actions workflows instead of Jenkins servers hosted on EC2 servers eliminated the overhead cost of the EC2 Jenkins instances.
• Migration from m5.xlarge instance type to m6g.xlarge helped improve performance by 40%.
• In the next phase, we will be upgrading to AWS EMR Serverless to further reduce the cost while maintaining the required performance.