A global hotel-booking SaaS connecting hotels and online travel aggregators was ingesting up to half a gigabyte every 30 seconds — overwhelming its database with IOPS and connection drops. We replaced it with a near real-time, pay-per-use streaming pipeline on AWS.
The client's interface enables two-way data exchange between hotels and OTAs, with ingestion ranging from a few megabytes to half a gigabyte every 30 seconds — roughly 3.3 billion records a month. The resulting IOPS and frequent connection disruptions were hammering database performance and cost.
Up to 0.5 GB every 30 seconds — ~3.3 billion records monthly — far beyond what the existing database was built to absorb.
Exceptionally high input/output operations and frequent connection disruptions degraded performance across the application.
Those same pressures drove database costs up, with capacity provisioned for peaks that sat idle the rest of the time.
We built a near real-time solution on AWS using cloud-native services that expands effortlessly as data volume grows and is billed solely on actual usage — no charges for idle capacity. The application streams into a data lake via Kinesis Firehose, with transactional tables kept fresh from SQL Server and Spark-based reporting jobs on Glue and Athena.
Provisioned with Terraform · CI/CD via Jenkins · one-click deployment · validated to 5× current volume (~100M)
The pipeline absorbs sudden volume surges with no data loss or record throttling.
End-to-end automation enables effortless one-click deployment across environments.
Cost-optimized per service and billed on actual usage — keeping monthly spend below $1,000.
When ingestion outgrows your database, a streaming lake bills by usage and scales on demand. We'll design one that fits your volume — and your budget.
Book a consultation →