How a global credit reporting company resolved government API rate-limit conflicts between real-time and batch workloads — using a serverless, event-driven architecture that enforces call rates automatically.
A global credit reporting company operating in 30+ countries relies on government citizen data from two providers to power a suite of real-time and batch data products. When the primary provider imposed strict new API rate limits, real-time queries began competing with batch jobs for the same restricted quota — threatening SLAs and product performance. MacondoTek designed a serverless, event-driven architecture on AWS that separates traffic lanes, prioritizes time-sensitive requests, and enforces rate limits automatically across both cloud and on-premises workloads.
Experian is part of a global conglomerate present in 37 countries with over 16,000 employees. Its local platform allows approximately 8 million financial consumers to query, correct, and improve their credit history, access personalized financial offers, and manage payment agreements. The platform supports multiple credit bureau products and processes critical financial data daily.
MacondoTek is a cloud consulting and professional services firm headquartered in Atlanta, USA, with nearshore delivery teams across Latin America, specializing in cloud architecture, data platforms, and application modernization on AWS. As an AWS Advanced Tier Partner with Service Delivery Program designations for Amazon RDS and Amazon DynamoDB, MacondoTek supports organizations across the full cloud lifecycle, from migration through cloud-native development to 24/7 managed operations. Its proprietary MTK CloudOpps platform powers FinOps and cost governance engagements with enterprise-wide visibility across large multi-account AWS environments, including deep experience serving financial services organizations that must balance reliability, security, and compliance with cost optimization.
The client’s product suite depends on government citizen data sourced from two providers — some products require real-time sub-second queries, while others rely on scheduled batch processing. When the primary data provider imposed strict new API rate limits, the two workloads began competing for the same restricted request quota. Without intervention, batch jobs would consume capacity needed for real-time products, risking SLA violations and degraded service response times.
The additional complexity was that parts of both the real-time and batch workflows ran on-premises, requiring any solution to bridge a hybrid cloud boundary while enforcing rate controls consistently across both environments. The architecture also had to accommodate two separate government data providers, each with different query patterns and availability characteristics.
MacondoTek designed a serverless, event-driven architecture on AWS that intelligently manages API call rates across both real-time and batch workloads. The design separates traffic lanes, prioritizes time-sensitive requests, and enforces rate limits automatically using cloud-native controls — with no servers to provision or manage.
The solution included:
The solution resolved the API throttling conflict without requiring changes to the underlying products or data provider contracts. Real-time queries now meet millisecond response targets while batch processing runs on a controlled lane that never competes with online traffic — maximizing efficiency across all product lines within the imposed limits.
The event-driven architecture is reusable and extensible: additional data providers or products requiring similar rate-limit management can be onboarded into the same framework, enhancing the client’s overall operational adaptability for future integrations.
The architecture separates real-time and batch workloads into distinct processing lanes — ensuring time-sensitive online queries always have priority within the rate limit envelope, without starving batch jobs or requiring manual intervention.
Lambda, SQS, and SNS form an event-driven pipeline that scales automatically with request volume. No servers to provision, no idle capacity to pay for, and built-in fault tolerance for transient provider failures.
Parts of both the real-time and batch workflows ran on-premises. MacondoTek bridged the hybrid boundary to ensure cloud and on-prem workloads operate within the same rate-limit controls — consistently enforced across the full data path.
Concurrency adjustment policies and CloudWatch alarms provide real-time visibility and automatic enforcement of API call rates — preventing throttling violations while maximizing throughput within the provider's allowed limits.
