Multinational credit reporting company optimizes their API integrations to local government systems
Our client is a multinational data analytics and consumer credit reporting company that operates in over 30 countries, harnessing the power of extensive data analytics to create opportunities for consumers worldwide. In Colombia, the client is embarking on a strategic migration journey to the cloud, with a specific focus on integrating Amazon Web Services (AWS) to enhance their operational capabilities.
Our client employs a product suite that relies on government data about its citizens, which is sourced from two different providers. This data is integral to their operations, as some products require real-time data queries, while others are dependent on batch processing. Recently, their primary data provider implemented stringent API usage regulations, significantly limiting the number of permissible requests per second. These new constraints present a unique challenge: optimizing the query limit to maintain the efficiency of real-time services without compromising the performance of batch processing tasks.
Additionally, given that some components of both the real-time and batch solutions are situated on-premises, there is a critical need to manage the data transactions seamlessly between the on-premises environment and the cloud, ensuring both real-time and batch processes are optimized within the imposed limits.
MacondoTek was engaged by the client to provide the architecture and design of a cloud base solution to allow millisecond response times by using cloud-first and serverless technologies to scale and control the rate of calls made to third party providers.
The solution was designed with efficiency and security in mind and cleverly addresses batch and online processing requirements. Ensures urgent operations are handled with due diligence while maintaining the highest security standards.
The solution included:
- Serverless components: Lambda functions, Amazon Simple Notification Service (SNS), and Amazon Simple Queue Service (SQS) for a scalable, event-driven architecture.
- Connectivity to OnPremise infrastructure, enabling secure communication and processing between cloud and local systems.
- DynamoDB database, leveraging a fast and flexible NoSQL database to handle high volumes of traffic and data.
- Transaction flow control using Concurrency-Adjust and alarms in CloudWatch, ensuring workloads are managed efficiently and within desired operational limits.
- Amazon S3 object storage for batch processing storage, providing a durable and accessible storage solution for batch data management.
- Infrastructure deployment and management automation using Terraform, ensuring consistency and ease of maintenance.
- Robust security policies implemented to safeguard cloud resources and data, aligning with corporate security standards.
With this solution, the client was able optimize data queries to their providers, maximizing efficiency across their various product lines without batch processes interfering with the response times of online products.
Third party limitations were resolved, while ensuring the architecture can be scaled to support other solutions requiring similar considerations, enhancing overall operational adaptability.
